In the first blog of a special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice, Cochrane UK’s Director, Professor Martin Burton, takes us from experts to evidence.
Page last reviewed: 5 April 2022.
In January 2022, a BBC news item reported that NHS Trusts use more than 100 different types of mesh in the repair of hernias, one of the most common operations. That must mean at least 100 different surgeons – probably many more – are using their preferred type of mesh, each of them convinced that the one they are using is best.
I wonder how often they have each been asked the question, “why do you use that particular sort of mesh?” I presume the answer would be: “Because I think this mesh is best for my patient”. “How do you know?”. “Because in my experience, having done 47 of these operations over the last……” That is a typical exchange. Believe me. I’m an expert.
Actually, don’t please believe me simply “because I am an expert”. One thing that people need to understand if they are to weigh up claims made about the effectivenessThe ability of an intervention (for example a drug, surgery, or exercise) to produce a desired effect, such as reduce symptoms. or otherwise of treatments is this: “Expert opinion is not always right”.
Experts don’t always agree
Many people – including the experts – often disagree about the benefits of different treatments and the harms they may cause. You see this in many ways. One sign is the wide variation across the country in the use of some surgical procedures for the treatmentSomething done with the aim of improving health or relieving suffering. For example, medicines, surgery, psychological and physical therapies, diet and exercise changes. of common conditions.
If operation A is better than operation B for treating people with a particular disease, why is it that in some places everyone has operation B and not A? It is often because the experts who do operation B think – and say – that it is better than A. Perhaps there really is no difference and they are equally effective. Or equally ineffective. Or perhaps they have never really thought about the alternatives.
There could be several other reasons. Be that as it may: experts are not necessarily right all the time. Sometimes they are. I’d like to say that for the best of them, they often are. But beware of putting too much weight on a person’s so-called expertise. Instead, consider challenging them to provide the evidence for their recommendations.
Challenging the expert, asking for evidence
But challenging experts may be easier said than done.
More years ago than I care to remember, when I was a junior doctor, I remember learning how to treat patients with different medical problems. After a few months, I would have a clear idea which tests to order for a patient, and what treatment to start. All this under the close, expert eye of the consultant and her team. Imagine my surprise (and some indignation and discomfort) in discovering, when I moved on to the next hospital, that my new consultant wanted his patients, with exactly the same condition, to be treated completely differently. How could that be? Surely they were both experts. How could both their opinions be right? These events occurred such a long time ago that “evidence-based medicine” was unheard of. So asking the question “what’s the evidence” would have been unheard of. It was something you simply never asked.
Times have changed. Thankfully. Nobody should be afraid to challenge the experts, not least a patient. But here’s a ‘top tip’ – consider the framing of the question. “Thank you for that recommendation. I am really pleased to have your expert advice. But I’d like to understand a little more about the sort of evidence that underpins it…”
Experts (and all of us) need evidence!
Early in my career I came across another ‘expert’ activity – expert ‘reviews’. These were, I thought, comprehensive, up-to-date articles, published in various journals, bringing together experts’ views on various topics. For example, describing the various treatment options for a particular condition and drawing conclusions about which one was most effective and safe. I was never very sure what the criteria were that led to someone being asked to write such a review. Just being ‘an expert’ and willing and able to write it I imagined. So I was surprised that whilst I was still a trainee surgeon (albeit, quite a senior one) I was asked to write one. I threw myself into it. I spent ages finding papers by famous and distinguished people, describing how they treated the condition. In those days, getting hold of the papers was a real effort; traipsing to libraries, finding bound volumes of journals, waiting in line for a photocopier. So it went on. The upshot of this effort was a strong desire to make sure that no effort was wasted. In deciding which papers to cite in the final article, there was a general feeling of – “include them all, if they’re at all relevant, because after all that effort to get them and read them…”.
Looking back on this process it was all rather random. Locating the papers to include involved a certain amount of serendipity combined with respect for those with big names in the discipline. Appraising the papers rested almost exclusively on the status of the authors and the journals they were published in; big names writing in big journals must be right. And bringing the results together in the review article was a form of writing in which information was brought together and combined in a way that did not follow any particular rules. All in all, although this process was undertaken conscientiously and carefully, respectful of the expertise of my peers, it was very unscientific.
From ‘expert’ reviews to systematic reviews
I now know that this was far from ideal. Far better to undertake the proper, rigorous process of “locating, appraising, and synthesising evidence from scientific studies according to a strict protocolThe plan or set of steps to be followed in a study. A protocol for a systematic review should describe the rationale for the review, the objectives, and the methods that will be used to locate, select, and critically appraise studies, and to collect and analyse data from the included studies. The protocols for Cochrane Reviews are available in the Cochrane Library.” – in other words, to do a “systematic review”. High-quality systematic reviewsIn systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. of randomisedRandomization is the process of randomly dividing into groups the people taking part in a trial. One group (the intervention group) will be given the intervention being tested (for example a drug, surgery, or exercise) and compared with a group which does not receive the intervention (the control group). controlled trialsA trial in which a group (the ‘intervention group’) is given a intervention being tested (for example a drug, surgery, or exercise) is compared with a group which does not receive the intervention (the ‘control group’). are at the top of the “evidence hierarchy” and provide much more reliable evidence on the effectiveness or otherwise of a specific treatment. Cochrane specialises in producing reviews like this, called Cochrane ReviewsCochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research., and Cochrane’s standards in producing these are said to be the highest in the world.
Let’s return to the issue of asking an expert for the evidence underpinning their advice. What might a really good answer look like? How about this: “Well, there are several good systematic reviews of trialsClinical trials are research studies involving people who use healthcare services. They often compare a new or different treatment with the best treatment currently available. This is to test whether the new or different treatment is safe, effective and any better than what is currently used. No matter how promising a new treatment may appear during tests in a laboratory, it must go through clinical trials before its benefits and risks can really be known. that allow us to conclude with a high degree of certainty, that in 75% of patients like you, this treatment will cure the problem. At the same time, there is a 5% chance that you may experience some minor side-effects of the treatment”.
Experts’ advice isn’t always right. But it is more likely to be right when it is based on solid, rigorous, scientific evidence.
- Expert advice isn’t always right or based on careful consideration of the best evidence.
- Always feel able to ask “What is the evidence? How certain can we be?”
- Systematic reviews in general, and Cochrane Reviews in particular, are a good source of evidence.
Join in the conversation on Twitter with @CochraneUK #OhReally or leave a comment on the blog.
References and further reading (pdf)
Visit the Teachers of Evidence-Based Health Care website, where you can find resources which explain and illustrate why expert opinion is not always right.
This series of blogs is inspired by a list of ‘Key Concepts’ developed by the Informed Health Choices.
Although this comment is a couple of years late and from across the pond, I congratulate Dr Burton on his observations. As a US patient advocate and healthcare writer, I see “the tyranny of the expert” operating in the US to a horrendous degree. Nowhere is this more evident than in the recently released 2022 updates to our US CDC and Veterans’ Administration guidelines on prescription of opioids in severe pain. The guidelines are substantially in error on both science and medical ethics. Along with US State government legislation and the ongoing US Drug Enforcement Agency’s witch hunt against doctors who prescribe opioid therapy for their patients, the guidelines are arguably responsible for the destruction of the practice of pain medicine in the US. Possibly thousands of patients have died of suicide due to forced tapers and refusal of care by clinicians afraid of sanctions that might destroy their livelihoods if not put them in prison.
Thus this footnote: before we consider anyone an “expert”, we should first ask “what is their background?” Public health specialists and academics such as the US CDC hired to write its guidelines are NOT competently trained or experienced in community or hospital practice of actual pain care. Likewise, specialists trained in addiction medicine are not necessarily competent in pain management practice. And all specialists suffer from common debilitating errors of process — notably confirmation bias. Sometimes “experts” see what they expect to see or what most benefits their professional stature or incomes, instead of what is actually happening.
I have had a lot of problems with doctors making what I have to consider to be mistakes, 7 doctors didn’t notice that the non generic form of Lisinopril, zestril, warns it can cause liver and kidney problems, 5.6 and 5.3 in the PDR. After 7 months, I found it.
I have been told, those are not your symptoms, and then eight months later had those symptoms photographed. One symptom is still being denied, that the condition goes away with antibiotics. That after it was not able to be seen in my eye until I stop taking oral antibiotics for dental issues. I still lack a diagnosis over a year later, having only a description.
Neurologist as patient: A missed diagnosis, poor communication, and incompetent care could have led to quadriplegia, November 12, 2020, at https://www.mdedge.com/neurology/article/231721/business-medicine/neurologist-patient-missed-diagnosis-poor-communication/page/0/
If he hadn’t himself in a neurologist, he would be a quadriplegic and probably on lifelong ventilation. When this was brought to the attention of the medical board, they said that the neurologist he consulted when out of state did the best he could.
I call this twisted logic, because if that’s the best he could do, he’s in the wrong job. But they used it to say that he did the best he could do so he should be allowed to continue to do it. This is described as normalization of deviation.
Medical billing code E71.43, ICD-10 but unchanged since 2007, iatrogenic carnitine deficiency from hemodialysis and valproic acid, plus PMID: 24282920 was one proof that Depakote was increasing my mental illness, elsewhere it showed L carnitine deficiency causing brain damage to the hypothalamus, which increased my diabetes, raised my blood pressure, and caused my temperature to go down to 95.6 F with L-Carnitine to 97.4 F, except it goes up easily. Either the oxytocin production or receptors in the hypothalamus or both are damaged, but I’m told I need to get a companion at all, not medication. I can pay for my own naturopath, and there’s a compounding pharmacy nearby, but why am I paying for healthcare, being in America. This was caused by medications, why can’t the medications to treat it be covered?
One-place the expert doctors might not know anything about is the administration side. I was told horizant wasn’t covered but that an exception could be made in early January. In late February I found out it was denied. In March after I changed health plans, I found out that my Humulin n was not covered and I only received it as an exception, the doctor didn’t know and the agent who sold me the health plan claimed that both were covered.
Human error: I was told that my liver and kidney problems from Lisinopril, non-traumatic acute kidney injury and over 10 times normal liver enzymes, 7 numbers significantly off, would magically go away if I lost 10 lb. That’s not on my chart though. I told the gastroenterologist that I’d lost 46 Pounds already and gain nothing so how could losing 10 more fix a problem that gaining haven’t caused. My chart showed that I lost a hundred pounds, because she heard 46 lb and thought big number. And didn’t remember the exact number.
9 out of 11 test showed my pH was acidic, despite the fact I was drinking 7 Gatorades a day, and that the medication finally given me to see whether it would help or not by the fifth nephrologist had the active ingredients only sodium citrate and citric acid, the same as in Gatorade. Self-medicating, I still showed acidic. I am still told it wasn’t significantly enough out of range, even though my numbers now are normal and my sciatica has gone away. I had been told that it getting worse and then better with lemon juice, better with grapes, and worse with acidic foods showed that I had psychiatric problems. My psychiatrist sent me to the fifth nephrologist.
Doctors don’t always look at your chart before treating you. I was told I had never had bad enough cataracts to require surgery, after missing two surgeries in the same Health Clinic. That embarrassment to the head of the department has cost me treatment by denying me care due to supposed psychiatric conditions I don’t have, and he broke the law to the point that when I told him I’d found it in my chart, a referral I’d been denied 3 times 6 months before suddenly showed up on my doorstep without my asking.
Making complaints with my health plan is a waste of time. The Grievances come back with such Insanity as putting me in two cities 20 minutes apart at the same time, one of which is not listed on my chart. I was given Bactrim although I was allergic to Septra, both sulfameth trimethoprim, and two separate complaints from supposedly the head and the assistant of the women’s building explained that they are not the same medication but are in the same family and therefore would not have the same reaction, and that they were not the same medication and not in the same family. The H pylori test internal document to that Health Clinic, the place they send ones fecal matter to test, and the manufacturer of the antigen test, Meridian bioscience, all show 3 false negatives including antibiotics and proton pump inhibitors, but I was told, “The test is very accurate. False negatives is not something that the department sees or expects.” It should be. Lab tests for Depakote and lithium are not being done under the correct protocol, just before the next dose. The psychiatrist don’t tell anyone, probably assuming that this is taken care of by the lab, but this is the only lab of four I have ever dealt with that does not ask how long it is been since our last dose, including the company that processes this clinics Depakote and lithium lab work. I’ve contacted their legal department to put pressure on the clinic but nothing happened. I’ve attempted to make 7 complaints, the third I was told they would not take because the problem had been fixed. A friend still in the clinic taking Lithium was not asked when he last took his medication, which was the morning of the lab test, when he had it drawn late last year.
Researchers are also human. There is discussion of putting lithium, I believe lithium carbonate, in the water supply to reduce suicides and mental illness. However, most of the studies do not mention what type of lithium is in the water in locations where it seems to be part of reducing mental illness and suicides. It turns out that lithium carbonate is not one of these natural lithiums in the few studies that actually list what type of lithium is involved, and it does make a difference. There is twice as much lithium to orotate as there is lithium to carbonate in lithium orotate. Though equally bioavailable, lithium orotate seems to stay longer in the brain, requiring doses that are suggested at more than a dozen times smaller for the same effect, having supposedly less effect on the thyroid and causing less balance issues. I am using lithium aspartate, which affects the thyroid somewhat, but I walk now without a walker.
The doctors in my clinic are not aware of the lab index which tells them the patient preparation and protocols, they go instead to the government medical publication websites, which do not specify that there are more than one type of test, and that they need to know which one is being used.
Having been told that beta blockers were well tolerated, I found out that they in fact inhibit peripheral T4 to T3 conversion. I was told that the effect is very rarely a problem, which is why my next door neighbor, having gotten off the beta blocker, found that his blood pressure was now controllable only with a calcium channel blocker we’re as it had gotten better and then worse during the beta blocker, which is normal for low thyroid. His doctor would not run a T3 test, and the TSH will show normal when there is inhibition of peripheral T4 to T3 conversion. The symptoms, constipation, memory loss, inability to function, weight gain, hair loss, and night blindness which may be unrelated to the thyroid condition, and many more, I was told was old age. My old age reversed within two weeks, taking that long because I had to go back on the beta blocker due to sudden diarrhea as my body compensated. Next door to each other does not imply rare to me. It implies that because the inhibition can take months that the doctors don’t connect the two together, although there is medical research which is how I figured out what was going on, other than the hair loss, proving that beta blockers inhibit peripheral T4 to T3 conversion, some faster and some slower. They’re used in the ER for thyroid storms for this purpose, but if this purpose is not required, the doctors don’t realize that it happens anyway.
The reason I’m allergic or reactive to sulfameth trimethoprim is because I was on Depakote and Metformin at the same time when I was given Septra. Up until then I had no reaction, but being simultaneously on for incidental antifolates at the same time caused a problem. No one told me that I should be taking folic acid with them. My psoriasis did go away, but that was not why I was given the four medications. The doctors assumed, if they even knew that these were antifolates, that since the purpose was not to reduce folic acid or folate from the body that it wouldn’t matter whether it was removed. Compounding even Depakote with metformin was causing problems, making me crave lettuce to the tune of two and a half pounds a day. Adding Septra, aka sulfameth and trimethoprim, at the same time made my body react because I could not keep the folate levels up. I had used it previously with no ill effect. I got severe arthralgia, and was put back on the medication by a doctor who did not connect the problems.
An idiopathic problem of my own, taking Kombucha Tea caused my thyroid to get stronger or my immune system to stop attacking it, from Hashimoto’s thyroiditis. My doctor didn’t tell me that my thyroid had gotten out of range, simply ordered the medication. I ran out and missed a dose, I picked up the new bottle and the next day realized it was the wrong color and missed another dose, and she tested less than a week later. I tested normal. Almost five months later, with heart palpitations and pounding and inability to think correctly during a move which ended me homeless due to poor choices, and was found a year-and-a-half later after I did research showing that this was likely, that I had bone density loss to the tune of 13.7% after being told that there would be no ramifications from the period of high thyroid, my TSH was 0.02 and a range of 5.5 to 0.1, and I suffered chest pain repeatedly. The test didn’t come back while I was in the emergency room, so they told me there was nothing wrong with me. When I got that lab results the next day, the emergency doctor told me to stay on the same dose of medication until I saw my doctor, who is out of town for 5 days. I stopped cold turkey and saved my life, but when I spoke to doctors for the next two days I was told by one that you cannot die from high thyroid. She hung up when I tried to tell her that heart attacks can be fatal.
I was given the H pylori cure, but not told that a prescription given in March, colchicine, would have a potentially fatal interaction with the Clarithromycin in in April. However, the pharmacy didn’t tell me because the doctor had put an override in, which implied that she had talked to me about it. Both warned me of a potential interaction with amlodipine, which is what my health plan said wasn’t fatal twice in two different complaints. They change the name of the medication I was complaining about twice, just as previously they changed the name of the doctor I was complaining about who told me that there was no false negative on the H pylori test.
These days, I do my own research. When I have a good doctor, I present that research to him or her, and they are able to use it. But they simply do not have the time required to do the 20 to 60 hours a week of research I have been doing because they work for a living. The good doctor’s tell me keywords and I do my own research and find things that have been published in the last year to 20 years which never made it to my Doctors Medical books and training. Even the best doctors simply don’t have an extra 20 to 60 hours a week if they expect to sleep.
Human error creeps into Data added to medical records. Some poor person I was waiting with crossed knees for his or her size small incontinence briefs which were sent to me, I am not size small. There have been wrong diagnosis, either a mistyped patient number or a mistyped medical billing code. These are small things which is why everyone should look at their chart every two to five years, they need correction but don’t require a complaint. This is just people being human. I make mistakes, I have to make sure that others mistakes are caught. That is the patient responsibility, although the doctor should check as well on a regular but not necessarily constant basis to confirm, or anytime they are using a diagnosis, the same way that my doctors traditionally check my medications every single visit. However at the health plan by was with in January and February, when I made a correction and requested medication that have been promised me as covered, the nurse did nothing, I had to contact the health plan directly and in one case more than six times before the medication I was told I would receive was denied.
The definition of an expert is someone who has enough familiarity with a topic to know when they need to do further checking, which was new research is basically all the time, and who knows where to look for it. And who is given time by their employer to be able to look for the information. In America, was 5 to 15 minutes per visit, that latter part hamstrings even the best of doctors unless they let me do the research for them on my own time.
HealthWatch, and independent UK charity that has been promoting science and integrity in healthcare since 1991, held a special symposium to consider the type of evidence required for surgical procedures such as mesh implantation and the use of implants and devices. We concluded:
• Approval has been a technical rather than a medical process.
• The ‘equivalence’ system using Notified Bodies has failed.
• Using and recording device serial numbers would be a simple first step.
• The IDEAL-D framework provides for evidence-based implant development.
• Adequately funded registries are needed with compliance monitoring.
• Political action will be required to influence the developing rules and to draw agencies together.
• There are academic responsibilities: early reporting; development of evidential standards; guidelines for data reporting and appropriate data amalgamation procedures.
• Putting the issues into simple statements will be a powerful aid to progress.
When it comes to medical implants, HealthWatch says:
Implant approval should be graduated and supported by step-by-step evidence. This should replace the ‘equivalence’ system of approval using Notified Bodies which has failed.
Those who implant a device must know (and be able to explain to their patient):
o What it is and what its constituents are
o How it is identified and tracked
o How the evidence shows that it works
o What risks are involved
o What to do if things go wrong
Regulators, academics and professional bodies should work together to achieve these aims
The full symposium report is available at: https://www.healthwatch-uk.org/projects/medical-devices.html
Good afternoon! Good article, it was interesting to read this. There were situations when I did not agree with the doctor, in order to know what I really needed to do, I always told them about my doubts if they could not tell me clearly their position and why they prescribed me this particular drug and not another one, then I go to another doctor. Unfortunately, this happens often. Good luck!
Recognition of experts by their droppings (guano-based eminence-based medicine). https://www.bmj.com/content/329/7480/1460.full.pdf+html
.Oxman AD, Chalmers I, Liberati A. A field guide to experts. BMJ 2004;329:1460-1463.
Thank you for your comment. In my experience – both as a clinician and a patient – when an expert sets out the evidence and the options clearly, the shared decision-making process can be relatively straightforward. But at times, an expert will admit to a significant degree of uncertainty. The best will then point patients to places and people to help them come to a decision that is right for them. At times this will be to another expert for a ‘second opinion’. Even in the NHS you are ‘entitled’ to a second opinion if you ask for one. And again, the best experts will be happy to facilitate this, often with the help of your GP. We all provide second opinions probably more often than you might imagine. But there seems to be a tacit assumption that this opportunity will not be ‘abused’; the NHS simply could not cope with everybody seeking a second opinion all the time. It is worth saying that if an expert is uncertain about the advice they are offering a patient, she or he will almost invariably discuss this with a colleague – a different sort of ‘second opinion’.
Curious to hear your views on the practical application of this advice by members of public in the UK. We all know that getting an appointment with a specialist on the NHS takes time and persistence. One can and should ask that expert for the evidence, but you do not have a choice of expert opinions and it’s between you and this one expert to agree on the way forward. So practically, within a state health system you follow the best advice of the expert you are referred to.