In the first blog of our new special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice, Cochrane UK’s Director, Professor Martin Burton, takes us from experts to evidence.
A BBC news item reported recently that NHS Trusts use more than 100 different types of mesh in the repair of hernias, one of the most common operations. That must mean at least 100 different surgeons – probably many more – are using their preferred type of mesh, each of them convinced that the one they are using is best. I wonder how often they have each been asked the question, “why do you use that particular sort of mesh?” I presume the answer would be: “Because I think this mesh is best for my patient”. “How do you know?”. “Because in my experience, having done 47 of these operations over the last……” That is a typical exchange. Believe me. I’m an expert.
Actually, don’t please believe me simply “because I am an expert”. One thing that people need to understand if they are to weigh up claims made about the The ability of an intervention (for example a drug, surgery, or exercise) to produce a desired effect, such as reduce symptoms. or otherwise of treatments is this: “Expert opinion is not always right”.
Experts don’t always agree
Many people – including the experts – often disagree about the benefits of different treatments and the harms they may cause. You see this in many ways. One sign is the wide variation across the country in the use of some surgical procedures for the Something done with the aim of improving health or relieving suffering. For example, medicines, surgery, psychological and physical therapies, diet and exercise changes. of common conditions. If operation A is better than operation B for treating people with a particular disease, why is it that in some places everyone has operation B and not A? It is often because the experts who do operation B think – and say – that it is better than A. Perhaps there really is no difference and they are equally effective. Or equally ineffective. Or perhaps they have never really thought about the alternatives.
There could be several other reasons. Be that as it may: experts are not necessarily right all the time. Sometimes they are. I’d like to say that for the best of them, they often are. But beware of putting too much weight on a person’s so-called expertise. Instead, consider challenging them to provide the evidence for their recommendations.
Challenging the expert, asking for evidence
But challenging experts may be easier said than done. More years ago than I care to remember, when I was a junior doctor, I remember learning how to treat patients with different medical problems. After a few months I would have a clear idea which tests to order for a patient, and what treatment to start. All this under the close, expert eye of the consultant and her team. Imagine my surprise (and some indignation and discomfort) in discovering, when I moved on to the next hospital, that my new consultant wanted his patients, with exactly the same condition, to be treated completely differently. How could that be? Surely they were both experts. How could both their opinions be right? These events occurred such a long time ago that “evidence-based medicine” was unheard of. So asking the question “what’s the evidence” would have been unheard of. It was something you simply never asked.
Times have changed. Thankfully. Nobody should be afraid to challenge the experts, not least a patient. But here’s a ‘top tip’ – consider the framing of the question. “Thank you for that recommendation. I am really pleased to have your expert advice. But I’d like to understand a little more about the sort of evidence that underpins it…”
Experts (and all of us) need evidence!
Early in my career I came across another ‘expert’ activity – expert ‘reviews’. These were, I thought, comprehensive, up-to-date articles, published in various journals, bringing together experts’ views on various topics. For example, describing the various treatment options for a particular condition and drawing conclusions about which one was most effective and safe. I was never very sure what the criteria were that led to someone being asked to write such a review. Just being ‘an expert’ and willing and able to write it I imagined. So I was surprised that whilst I was still a trainee surgeon (albeit, quite a senior one) I was asked to write one. I threw myself into it. I spent ages finding papers by famous and distinguished people, describing how they treated the condition. In those days, getting hold of the papers was a real effort; traipsing to libraries, finding bound volumes of journals, waiting in line for a photocopier. So it went on. The upshot of this effort was a strong desire to make sure that no effort was wasted. In deciding which papers to cite in the final article, there was a general feeling of – “include them all, if they’re at all relevant, because after all that effort to get them and read them…”.
Looking back on this process it was all rather random. Locating the papers to include involved a certain amount of serendipity combined with respect for those with big names in the discipline. Appraising the papers rested almost exclusively on the status of the authors and the journals they were published in; big names writing in big journals must be right. And bringing the results together in the review article was a form of writing in which information was brought together and combined in a way that did not follow any particular rules. All in all, although this process was undertaken conscientiously and carefully, respectful of the expertise of my peers, it was very unscientific.
From ‘expert’ reviews to systematic reviews
I now know that this was far from ideal. Far better to undertake the proper, rigorous process of “locating, appraising, and synthesising evidence from scientific studies according to a strict The plan or set of steps to be followed in a study. A protocol for a systematic review should describe the rationale for the review, the objectives, and the methods that will be used to locate, select, and critically appraise studies, and to collect and analyse data from the included studies. The protocols for Cochrane Reviews are available in the Cochrane Library.” – in other words, to do a “systematic review”. High-quality In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. of Randomization is the process of randomly dividing into groups the people taking part in a trial. One group (the intervention group) will be given the intervention being tested (for example a drug, surgery, or exercise) and compared with a group which does not receive the intervention (the control group). A trial in which a group (the ‘intervention group’) is given a intervention being tested (for example a drug, surgery, or exercise) is compared with a group which does not receive the intervention (the ‘control group’). are at the top of the “evidence hierarchy” and provide much more reliable evidence on the effectiveness or otherwise of a specific treatment. Cochrane specialises in producing reviews like this, called Cochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research., and Cochrane’s standards in producing these are said to be the highest in the world.
Let’s return to the issue of asking an expert for the evidence underpinning their advice. What might a really good answer look like? How about this: “Well, there are several good systematic reviews of Clinical trials are research studies involving people who use healthcare services. They often compare a new or different treatment with the best treatment currently available. This is to test whether the new or different treatment is safe, effective and any better than what is currently used. No matter how promising a new treatment may appear during tests in a laboratory, it must go through clinical trials before its benefits and risks can really be known. that allow us to conclude with a high degree of certainty, that in 75% of patients like you, this treatment will cure the problem. At the same time, there is a 5% chance that you may experience some minor side-effects of the treatment”.
Experts’ advice isn’t always right. But it is more likely to be right when it is based on solid, rigorous, scientific evidence.
- Expert advice isn’t always right or based on careful consideration of the best evidence.
- Always feel able to ask “What is the evidence? How certain can we be?”
- Systematic reviews in general, and Cochrane Reviews in particular, are a good source of evidence.