In the third blog of our special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice, Selena Ryan-Vig, Cochrane UK’s Communication and Engagement Officer, explores some cautionary examples which remind us that all treatments have potential harms.
“Dangerous” and “irresponsible”. That’s how Donald Trump’s comments about treating coronavirus have been described by doctors and scientists . Last week the US president suggested that injecting disinfectant might be a possible treatment for the virus. Warnings from health professionals, scientists and product manufacturers in response were emphatic: ingesting or injecting disinfectant could do serious harm. Nonetheless, in the wake of Trump’s comments, emergency hotlines in the United States reported a spike in calls about dangerous disinfectant use. Trump said he “can’t imagine why” his comments could have sparked this response . In doing so, he failed to acknowledge that people in need or desperation may hope that – even the most outlandish – treatments will work, while downplaying, ignoring or failing to understand potential harms.
Aside from Trump’s suggestion, disinfectant was never intended as a treatment and so this may be a particularly extreme example. However, Trump’s ill-founded comments are a timely reminder that any treatment – from the most unusual to those that are widely used – may be harmful. Few treatments are 100% safe. Making ill-informed choices could mean that, at best, we may waste time and money on treatments that have never been shown to be useful. At worst, we may opt for treatments that harm.
Let’s, in turn, consider four key issues:
- Just because a treatment is widely used, doesn’t mean it’s entirely safe and without potential harms.
- Some harms may seem counterintuitive or unexpected; we can’t simply assume a treatment to be safe and should consider the evidence.
- Decisions about treatments may involve weighing up whether the (potential) benefits outweigh the (potential) harms.
- It is essential that information about harms is clearly reported in studies, yet reporting is often poor and inconsistent.
1. Just because a treatment is widely used, doesn’t mean it’s entirely safe and without potential harms
As the most-commonly used painkiller in the world, paracetamol is our ‘go-to’ for many acute and chronic pain conditions, yet concerns about its harms have grown . The harmful effects of over-dosing on paracetamol, whether accidentally or intentionally, are widely known , but what about when we use it at recommended doses, over a long period of time?
Andrew Moore, who has authored over 200 systematic reviews, many on pain, explored the evidence on paracetamol in this blog . He outlined evidence linking paracetamol with an increased risk of a number of, relatively rare but important, harms. These include cardiovascular problems; gastrointestinal problems (such as stomach ulcers); as well as kidney and liver impairment. It is important to point out that the evidence base for the harms of long-term paracetamol use consists of many observational studies, which must be interpreted with caution .
Nonetheless, we have reason to take these concerns seriously. Not least because the benefits of paracetamol are increasingly being called into question. There is high-certainty Cochrane evidence, for example, that there is little or no difference between paracetamol and placebo for improving outcomes in people with acute low back pain . Paracetamol is also unlikely to provide meaningful levels of pain-relief for the majority of people with a range of other pain conditions too, including osteoarthritis , cancer pain , acute postoperative pain  and migraine . The widespread use of paracetamol might lead us to presume it is both effective and safe; the evidence challenges us on both fronts.
2. Some harms may seem counterintuitive or unexpected; we can’t simply assume a treatment to be safe and should consider the evidence
Example: electric fans during heatwaves
Making assumptions about safety can be dangerous. Take the example of electric fans. Isn’t it obvious that electric fans are useful when the weather gets hot? Would you question whether they could actually do more harm than good during a heatwave?
A Cochrane Review on the use of electric fans during heatwaves painted a mixed picture . Some studies suggested that fans may reduce health problems, others suggested that fans might actually make things worse. It is possible that, at extreme temperatures, fans may actually heat us up, by blowing hot air onto us, in turn exacerbating problems such as dehydration. Uncertainties about the health effects of electric fans remain. Nonetheless, this example is a reminder that harms may feel unexpected or unlikely. All interventions should be scrutinised: both their effectiveness and their safety.
3. Decisions about treatments may involve weighing up whether the (potential) benefits outweigh the (potential) harms
Example: Nicotine Replacement Therapy (NRT) for helping people quit smoking
Side-effects can be associated with any treatment. As such, making decisions about treatments may involve balancing the potential benefits against the potential harms. You might take into account, for example, the severity of the side-effects and how common they are. A person may be willing to accept some level of unpleasant effects if these are relatively mild and/or if the risk of more severe harms is relatively low.
An illustration of this comes from a Cochrane Review looking at whether Nicotine Replacement Therapy (NRT) is an effective and safe way to help people quit smoking . NRT was compared with placebo or no NRT in 136 studies and 64,640 people. The review found high-certainty evidence that NRT in all licensed forms (gum, lozenges, patches, sprays, inhalers and tablets/lozenges) increases someone’s chances of successfully quitting smoking. Chest pains and palpitations were rare and serious adverse events, such as stroke and heart attack, were extremely rare. Nonetheless, less severe side-effects were common. For example, up to 54% of people using a patch reported skin sensitivity and irritation. The reactions were usually not severe enough to prompt people to stop taking the treatment.
4. It is essential that information about treatment harms is clearly reported in studies, yet reporting is poor and inconsistent
An interesting example here is macrolide antibiotics (macrolides), which are among the most commonly used antibiotics worldwide. A Cochrane Review of 183 studies with a total of 252,886 people assessed the adverse effects of macrolides when used for the prevention and treatment of a wide range of infections . It found moderate-certainty evidence that people treated with macrolide antibiotics are probably more likely to experience nausea, vomiting and abdominal pain and there was low-certainty evidence that macrolides may carry a higher risk of diarrhoea, hearing loss and taste disturbances. However, 1 in 10 of the studies provided no information about harms; leaving us uncertain about whether or not, and to what extent, the people in those studies experienced side effects. However, even in those studies that did provide this information, the review authors highlighted the poor and inconsistent reporting about one very important adverse event in particular: antimicrobial resistance.
Antimicrobial resistance is one of the biggest global health problems of our time, with antibiotic (mis)use being a key driver. Resistance could see people developing infections which cannot be easily treated with antibiotics, meaning that even common infections or simple wounds could become fatal . Yet despite the importance of the problem, and despite including a large number of studies (183), the authors stressed that “very limited information was available to assess if people treated with a macrolide antibiotic were at greater risk of developing resistant bacteria than those treated with placebo”. This leaves patients and healthcare professionals in the dark; the full picture regarding the potential risks of using this type of antibiotic is unclear.
Concerns about poor reporting of harms are not confined to the issue of antibiotic resistance, or this clinical topic. Across a range of conditions – from hypertension to epilepsy – the standard of reporting of harms-related information has been found to be inadequate . When we do not have enough, good-quality information about possible adverse effects of treatments, this compromises our ability to balance the trade-offs between benefits and harms.
Questions about safety should be asked about all interventions
Trump’s latest suggestion about treating coronavirus prompted such outrage and concern because disinfectant is a product with such obvious, severe harms. Yet, particularly when treatments are widely established or seemingly innocuous, it can be easy to forget that any intervention might have harmful effects to some degree. From the paracetamol in our cupboards to the electric fans in our homes, we should consider the evidence, and question not only whether something works, but whether it is safe.
Join in the conversation on Twitter with @CochraneUK and #OhReally or leave a comment on the blog. Please note, we will not publish comments that link to commercial sites or appear to endorse commercial products.
*Photo credit: Michelle Tribe (2007). Paracetamol/acetaminophen pills, 500 mg. Licensed under the Creative Commons Attribution 2.0 Generic license.
Selena Ryan-Vig has nothing to disclose.
About point 3, “Decisions about treatments may involve weighing up whether the (potential) benefits outweigh the (potential) harms,” there are new demographics which if included in studies might be able to pinpoint better who will have more benefit, and who will have more harm.
Of course, there is the MTHFR gene mutations. If untreated, people with these mutations may have intractable conditions which could skew results, making treatments for people without this condition look as if they are less beneficial.
Then there are the conditions described on the Walsh Research Institute’s page on Biochemical Individuality. (The Walsh Institute
Although the description is aimed at mental illness, there are medical comorbidities with each of these. Copper overload patients may have a number of Wilson’s disease comorbidities. Treatments which affect copper metabolism and therefore zinc and iron will affect people with copper overload differently. Likewise, Pyrrole syndrome sufferers will be affected differently than those without this condition, which also usually goes undiagnosed.
People with undermethylation and overmethylation, the latter of which often causes the former as well, should be treated before they get cancer, and also will not respond to treatments the way the common populace would.
For that matter, people with pernicious anemia dump zinc in the fecal matter when they are given folic acid. Since folic acid is given in the form of enriched bread, rice, noodles and breakfast cereal in hospitals, people with undiagnosed pernicious anemia suffering from Coronavirus will skew results, and will have erratic benefits depending upon what they are fed.
If these conditions were included along with age, gender, race, location, smoking status, etc., studies would be able to target which of these conditions would be better benefited, and which would be more likely to have dangerous side effects.
On top of that, since these conditions often go with nutritional deficiencies, the cause of the side effect may be ameliorated by filling in particular nutrients, making that treatment safer and more effective, not just for people in these demographics but for people who have those deficiencies who do not fit the particular demographic as well.
Without doubt, there are genetic mutations and conditions of which I am unaware which should also be included.
A final note: Most studies, in the effort of being very clear about cause of problems and benefits make a point to avoid people who have comorbidities. The conditions mentioned above usually go with comorbidities, and therefore these populations do not get included in many studies, so the studies will be missing whether a specific treatment would be particularly beneficial or particularly detrimental to people who suffer from any of these conditions which often cause multiple comorbidities.
And by being left out of studies, it is as if it is assumed that people with these conditions will respond the same way as people without them. Thus, we are guaranteeing that we do not know enough about how the people who need the treatments the most are going to respond to them.
The recent Cumberlege report “First Do No Harm” (https://www.immdsreview.org.uk/downloads/IMMDSReview_Web.pdf) has far-reaching and welcome implications highlighting harms of drugs and devices. It revealed shocking failings of the drug and device makers and the regulators. I would say the failings extend to academia, scientific publishing, and the medical establishment.
The report did not consider potential harms to patients from psychological and behavioural interventions which are still widely recommended for people with ME, and for an ever-expanding range of patients chucked into the (as yet) “medically unexplained symptoms” (MUS) swamp. There is currently no UK mechanism for recording or monitoring harms from these types of interventions, or for challenging the assumption that they are an appropriate first-line treatment option for ME or for other long-term conditions where the cause and perpetuating factors are unknown.
In clinical research, investigators often have a vested interest in the treatment they are investigating. They have already convinced themselves, their peers, and if higher up the food chain, politicians, funders, regulators, policy makers, insurance companies, that a treatment has merit. With non-drug interventions where trials are usually impossible to blind, investigators without the necessary “equipoise” can design and manipulate a trial to flatter a favoured treatment. For example, a trial can rely on subjective outcomes, have overly vague and wide participant eligibility criteria, or overly relaxed recovery criteria. Trials, even of long-term conditions, are usually of relatively short duration, so it’s also easy to gloss over a lack of lasting benefit and downplay the possibility of harm.
There is also no incentive for anyone “invested” financially or reputationally in a behavioural or psychological therapy to carry out post-marketing surveillance, and there is no Yellow Card system for patients to raise the alarm. All we have are systematic reviews, Cochrane being the most trusted brand, at least by patients and the public.
The merit and safety of psychological and behavioural interventions is generally backed up by poor science which escapes the greater scrutiny given to research on drugs and medical devices. It is currently not prohibited by Cochrane to have authors of included trials on review teams or acting as advisors. Cochrane does nothing proactively to identify where this has caused a problem, and has refused to withdraw reviews affected when the problem is identified by others. This refusal to withdraw out of date and misleading publications, such as the Exercise for Chronic Fatigue Syndrome and the CBT for Chronic Fatigue Syndrome reviews, is protecting the reputation of Cochrane contributors, rather than putting the safety of patients first.
Dear Selena, I realise that you in a difficult position, but I hope that deep down you know that I am not that dumb. And I have read the relevant email transcripts.
It is good to see public education about the potential harms of treatments but I think for Cochrane to put out such advice carries an irony. There is a standing Cochrane review on exercise treatments for chronic fatigue syndrome that fails on just about all the issues you raise. People at Cochrane know that it is seriously flawed but do nothing about it because of a strange rule that means reviews cannot be withdrawn if the authors do not want to. The interests of Cochrane contributors come above transparency.
What is even more worrying is that this review is only up because an even more problematic review was withdrawn. That one was co-authored by a very prominent figure in Cochrane who writes major review articles on quality of evidence.
I personally have no competing interest in this area, being a retired physician who writes philosophy and goes birdwatching, but who also dislikes hypocrisy.
The whole problem boils down to vested interest. Cochrane gives the impression of being King Solomon but in reality was set up by people with a very clear agenda based towards the interest of low tech primary care-based medicine. High tech gets the Cochrane workover. Low tech, accessible by primary care, like therapist-based treatments, seems to be let through on the nod. I asked Iain Chalmers if he was not concerned about vested interest with regard to therapists writing reviews about treatments that provide their livelihood. He seemed to think raising vested interest was an insult. Yet surely the whole point of Cochrane was to address vested interest? The playing field does not seem to be level.
In your position I would be concerned that I was being used as a squeaky clean window dressing for an organisation that is not quite what it presents itself as. Cochrane is just a group of people with opinions, and those opinions are far from unbiased. I used to think otherwise (or it never crossed my mind) but various recent events have made it clear that it is much like any other organisation that makes its living out of providing opinions people want to hear.
Thanks for your comment.
You may have seen Cochrane’s response to the issues raised with the review you discuss. If not, the full response can be found here: https://www.cochrane.org/news/publication-cochrane-review-exercise-therapy-chronic-fatigue-syndrome
In sum, an amended version of the review was published in October 2019, evaluated by independent peer reviewers. It now places more emphasis on the limited applicability of the evidence to definitions of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) used in the included studies, the long-term effects of exercise on symptoms of fatigue, and acknowledges the limitations of the evidence about harms that may occur.
Nonetheless, Cochrane recognises that review is still based on a research question and a set of methods from 2002, and reflects evidence from studies that applied definitions of ME/CFS from the 1990s. Having heard different views expressed about the evidence base for this condition, Cochrane has acknowledged that the publication of this amended review will not resolve all the ongoing questions and has committed to a full update of the review.
Work has begun with a comprehensive review of the protocol, developed in consultation with an independent advisory group. This group will involve partners from patient-advocacy groups from different parts of the world who will help to embed a patient-focused, contemporary perspective on the review question, methods and findings.
You may also be interested in reading about plans to make Cochrane’s conflict of interest policy more rigorous. This new conflict of interest policy will be implemented this year: https://www.cochrane.org/news/more-rigorous-conflict-interest-policy-coming-cochrane
I hope this answers some of your concerns.
Regarding the issues with the review on exercise for ‘CFS’, I was disappointed with the response to Jonathan Edwards. It circumvents the question how it was possible that updates of that review have been repeatedly published that didn’t address the rigorous criticism which had been submitted. That this criticism was dismissed happened in line with Cochrane policy and the latest Cochrane tools to assess evidence and risks of bias.
Patients, clinicians and scientists also submitted criticism on the Risk of Bias tool but never received an answer.
Setting up an IAG is a nice idea and could be helpful. However, if it is clear from the beginning that patients’ criticism of the Cochrane methodology of assessing evidence and risk of bias won’t be taken into account, this is not very encouraging.
As a person with ME/CFS my experience that graded exercise and the cognitive behavioral approach to ME/CFS are not helpful but repeatedly caused severe relapses was not taken seriously by health care professionals. It was easy for my doctors, my health insurance etc. to dismiss my experience as a distortion of reality — referencing my country’s national treatment guidelines and other publications that are based, among others, on the Cochrane review on exercise.
Those in charge at Cochrane know some of the many testimonies of patients who have been harmed by this neglect. They know that one of the people who suffered from that neglect was Robert Courtney, a civil scientist who repeatedly submitted comments on the Cochrane exercise review. They know that this civil scientist and person with ME deteriorated with his efforts to be taken seriously, that, bedbound and not believed by his doctors, he took his own life.
I am not completely bedbound but after 10 years of being only mildly affected now am in the ‘moderate’ stage of ME/CFS. I am mostly housebound and need to lie in bed 70% of the time.
I was lucky though that after 20 years with ME/CFS, some NHS doctors finally believed me, ignored diverse guidelines on how to deal with people with ‘CFS’, assessed me as severely disabled and also as permanently incapacitated for work. That means that I now receive social benefits due to ill health and don’t have to pretend anymore that I’m still able to work and write job applications every week in order to qualify for unemployment benefits.
Consequently, I can now manage my condition more easily by pacing. It also means that I can report (sometimes new) health issues to my doctors without adding ‘evidence’ that I am merely having false illness beliefs thus adding evidence that I was still fit for work if I only acknowledged that my illness was due to my false beliefs. It means that doctors take symptoms seriously, don’t automatically associate them with ME/CFS but actually investigate them. (In the meantime, doctors repeatedly found things that needed treatment).
Other people with ME aren’t that lucky and Cochrane continues to enable that they are harmed by wrong claims of evidence. Cochrane allows reviewers to assess merely subjective/ self-reported outcomes from unblinded trials as moderate quality evidence. The updated review on exercise for ‘CFS’ still states that, compared to treatment as usual, “participants who have exercise therapy probably have less fatigue at the end of treatment, and they may have moderately better physical functioning”, despite the fact that both outcomes (fatigue and physical function) are solely self-reported, subjective data that were collected in unblinded trials and not backed up by any objective outcomes.
I would appreciate it if Cochrane offered an apology to patients. That would also help encourage involvement with the advisory group. It would also help if Cochrane was able to demonstrate that it is receptive to independent criticism and is willing to answer questions about its assessment tools.
I have certainly been harmed by the radiotherapy I received 33 years ago at Guildford. I was given large doses every other day instead of small doses every day, leading to fibrotic lumps in the breast and armpit, which became progressively worse, until 18 months ago I suffered a mild stroke, which has further damaged my right side. The carotid artery in my neck was affected. Originally I had a poor prognosis, which may be why my oncologist didn’t expect me to live long enough to experience any long-term effects. But I attended the Bristol Cancer Help Centre (now Penny Brohn UK), changed my diet, received counselling. then trained as a counsellor myself. I came to Scotland to join family members in 2003. You still haven’t heard the last of me: my book ‘Foiled Creative Fire: a study of remarkable
women with breast cancer’, is due to be published on 1st July 2020. It traces the mental health of 12 women through the ages, from Anne of Austria to Audre Lorde, showing the influence of their anxiety and depression on their physical health.
Thank you for sharing your story Heather. I’m sorry to hear of your stroke but it is encouraging to read about the positives that have come out of your experiences. Take care, Selena