Personal experiences or anecdotes (stories) are an unreliable basis for assessing the effects of most treatments

In this blog, Elaine Finucane who works with iHealthFacts – an online resource where the public can check the reliability of a health claim circulated by social media – explains why anecdotes are unreliable evidence. This is the fifth blog of our special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice.

‘It ain’t so much the things we don’t know that get us into trouble. It’s the things we know that just ain’t so.’ – Anonymous

We have all grown up with old wives’ tales advising us to use specific treatments to cure common and not so common ailments.  Stories and treatments that have been passed down through centuries and generations.

How many of us have rubbed dock leaves on nettle stings to relieve the painful sting? Or, maybe drank flat soda to help relieve tummy pain?

We now live in a society where health claims, some reliable and some unreliable, are created and spread at the touch of a button. During the COVID-19 pandemic, we have experienced a rapid increase in these health claims, with The World Health Organisation labelling it an ‘infodemic’ of information1 (where we are overwhelmed with information – some accurate and some not).

In response, iHealthFacts, an online resource where the public can quickly and easily check the reliability of a health claim, has been ‘fact checking’ claims submitted by the public.  These claims have varied wildly, from advising members of the public to take the drug Chloroquine to prevent or treat COVID-19, to a claim telling the public that being unable to hold one’s breath for ten seconds, without coughing, is a way of diagnosing COVID-192.  Often, these claims were supported by personal stories and individual experiences – but were not supported by evidence. While it can be very tempting to believe these claims, especially during these uncertain times, it is important to make well-informed decisions about your health.

So what? What’s wrong with believing what we are told?

If your friend Tom tells you, that after being stung by a nettle, he rubbed a dock leaf on his skin and the pain vanished, it must be true, right? Indeed if it worked for Tom, it has to work for you?

Well… not necessarily.

If dock leaves were a reliable treatment for nettle stings, then we would expect Tom’s experience to be typical, but is Tom’s experience on its own enough evidence to support this claim?2

Just because Tom felt better after using a dock leaf does not mean that the dock leaf is responsible for Tom feeling better.  Rubbing dock leaves onto the nettle sting may have distracted Tom from the pain of the sting.  A placebo effect may be responsible, where Tom believed the dock leaf would help, and this belief provided a beneficial effect3. Would he have felt better if he did nothing?

In other words, personal experiences – or even a series of personal experiences – can sometimes be misleading. Sir Francis Galton coined the term ‘regression to the mean’ when he noted that extreme outcomes tend to be followed by a more normal or average one4. Like tossing a coin several times in a row and it always lands on heads – it is unusual and would be very difficult to repeat5. When ‘regression to the mean’ is applied to symptoms such as Tom’s pain, it tells us that the initial extreme pain he felt may have improved without treatment.

The power of many

When talking about dock leaves and nettle stings, it may seem harmless to use personal experiences or stories to inform your healthcare decisions. However, when you apply the same logic to the claim that the drug chloroquine prevents or treats COVID-19, a drug used to treat malaria and rheumatoid conditions, such as arthritis, with its potentially serious side effects,  the stakes become higher6.

Rather than relying on Tom’s personal experience, or a series of personal experiences to tell you whether dock leaves relieve nettle stings, a better option would be to look at all the available evidence on this intervention. To examine the experiences of people of different ages, gender and ethnicities to see what effect, if any, dock leaves had on their nettle stings.

We call this process a systematic review of the evidence7. A systematic review looks at all the current evidence available to answer a question. Combining all the evidence, systematically, is likely to provide a more reliable, robust answer on which to base your healthcare decisions.

For example, a recent Cochrane systematic review, which included evidence from 16 individual studies and 7706 participants, examined signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID‐19 disease8. Despite online claims that being unable to hold one’s breath for ten seconds, without coughing, is a way of diagnosing COVID-19, a claim supported by personal experiences, this review found that a single symptom or sign, including a cough or respiratory symptom, cannot accurately diagnose COVID‐19.

Think critically about health claims!

It is unwise to base healthcare decisions on personal experiences – or even a series of personal experiences. Personal experiences or anecdotes do not provide robust evidence.  Systematic reviews of all the available evidence on an intervention help minimise bias and produce more reliable evidence to support informed healthcare decisions.

Take-home points

Personal experience or a series of personal experiences can be misleading. Just because an individual got better after using a treatment does not mean that other people who receive the same treatment will also improve, or that the treatment is responsible – ‘regression to the mean’ tells us that experiences such as pain may improve anyway without treatment. Systematic reviews - which bring together all of the best available evidence to answer a question - provide more reliable, robust evidence to inform decisions about your health.

Join in the conversation on Twitter with @CochraneUK, @iHealthFacts1 and @elainefin4 or leave a comment on the blog. Please note, we cannot give medical advice and we will not publish comments that link to commercial sites or appear to endorse commercial products.

References (pdf)

Visit the iHealthFacts website to submit a health claim to be fact checked, or search for previously answered questions.

Visit the Teachers of Evidence-Based Health Care website, where you can find resources which explain and illustrate why anecdotes are unreliable evidence.

This series of blogs is inspired by a list of ‘Key Concepts’ developed by the Informed Health Choices.

Elaine Finucane reports that she is a Research Associate with the HRB-Trials Methodology Research Network, NUI Galway and an evidence researcher for iHealthFacts. iHealthFacts (www.ihealthfacts.ie) is funded by the Health Research Board, Ireland.



Personal experiences or anecdotes (stories) are an unreliable basis for assessing the effects of most treatments by Elaine Finucane

is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International

Leave a Reply

Your email address will not be published. Required fields are marked *

*

UA-49496932-1