In the fourth blog of our special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice, Lynda Ware, a Senior Fellow in General Practice at Cochrane UK, explains that absence of evidence is not evidence of absence and why it’s important not to mistake one for the other, with some examples from Cochrane ReviewsCochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. and the media.
Page last updated 3 February 2023.
Take-home points
Absence of evidence is not evidence of absence: no one seems to be sure where this phrase came from, although it is commonly attributed to the Astronomer Royal, Martin Rees, a British astrophysicist and cosmologist.
It captures in eight words the important message that it is pretty much impossible to prove a negative – that is, that something doesn’t exist.
The philosopher and mathematician Betrand Russell waded in on the argument. He used the example of a ‘cosmic teapot’ to illustrate his point. He said that, even though it could not be refuted, no one should believe him if he claimed, without proof, that a tiny teapot was orbiting the sun between Earth and Mars (Boncheck, 2016).
In similar vein, Martin Burton, Director of Cochrane UK, wrote that an invisible unicorn had been grazing in his office and invited readers to prove him wrong!
An important concept
In the realms of medical science the concept is important. It is claimed, not infrequently, that a treatmentSomething done with the aim of improving health or relieving suffering. For example, medicines, surgery, psychological and physical therapies, diet and exercise changes. ‘has no effect’ or that ‘there is no difference’ between two treatments. These claims can be found not only in the media but in the scientific papers from which the headlines derive. We are led to believe that if there is no reliable evidence proving an effect or a difference then they do not exist. This is wrong.
Evidence may be unreliable for several reasons such as:
- The studies may be too small to detect an effect or a difference.
- The effect or difference may be very, very small.
- There may be too few dataData is the information collected through research. or not enough studies.
- The evidence may be of very low quality.
- The studies may have been poorly designed.
Lost in translation: some examples from the media
“Giving children full-fat milk will not make them obese, studyAn investigation of a healthcare problem. There are different types of studies used to answer research questions, for example randomised controlled trials or observational studies. finds.” (Telegraph, March 2020)
A bold, contradict-me-if-you-dare, statement that full-fat milk is not linked to childhood obesity. Now this may be true, but the research stated that there was no evidence that full-fat milk and dairy products lead to overweight children. There is always the possibility that an effect has not been detected or is very small. It’s not possible to conclusively prove that something doesn’t exist.
And another….
“Using talcum powder does NOT raise the riskA way of expressing the chance of an event taking place, expressed as the number of events divided by the total number of observations or people. It can be stated as ‘the chance of falling were one in four’ (1/4 = 25%). This measure is good no matter the incidence of events i.e. common or infrequent. of ovarian cancer as study of 250000 women debunks fears after decades of uncertainty.” (Mail Online, January 2020)
This headline comes from a review of four large clinical studies looking at whether using talcum powder on the vulva is linked to an increase in ovarian tumours. Although the findings are reassuring, the researchers point out that they are unable to exclude the possibility of a very small increase in ovarian cancer. They explain that ovarian cancer is a relatively rare finding and that the studies may have been flawed.
Absence of evidence: some Cochrane Reviews
Let’s look now at some Cochrane reviews to give two more examples.
Interventions for managing halitosis (bad breath)
This review considers different ways to help halitosis (bad breath). These include mechanical cleaning (for example, tongue cleaners and toothbrushes), chewing gums, systemic deodorising agents (for example, mushroom extract that you eat), topical agents (such as gel that you apply), toothpastes, mouthrinse/mouthwash, tablets, and combination of different treatments. It is uncertain which, if any, work best at controlling bad breath. This is not to say none works but that there is insufficient, good quality evidence to guide us.
Convalescent plasma or hyperimmune immunoglobulin for people with COVID‐19: a rapid review
This important and topical systematic reviewIn systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. considers whether giving plasma from people who have recovered from COVID-19, and which is full of antibodies to the virus, helps those who are critically ill with the infection. It is one of Cochrane’s rapid reviews, produced in response to the pandemic.
In the first couple of versions of this review, the authors concluded that they were uncertain whether convalescent plasma is beneficial and safe for people admitted to hospital with COVID-19.
Although there were trialsClinical trials are research studies involving people who use healthcare services. They often compare a new or different treatment with the best treatment currently available. This is to test whether the new or different treatment is safe, effective and any better than what is currently used. No matter how promising a new treatment may appear during tests in a laboratory, it must go through clinical trials before its benefits and risks can really be known. available, there was no reliable evidence that convalescent plasma helps or harms. However, the authors identified many ongoing studies and since then the picture has become clearer.
Emerging evidence has helped to address this evidence gap. In February 2023, the fifth update of this Cochrane Review was published. There is now high-certaintyThe certainty (or quality) of evidence is the extent to which we can be confident that what the research tells us about a particular treatment effect is likely to be accurate. Concerns about factors such as bias can reduce the certainty of the evidence. Evidence may be of high certainty; moderate certainty; low certainty or very-low certainty. Cochrane has adopted the GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) for assessing certainty (or quality) of evidence. Find out more here: https://training.cochrane.org/grade-approach evidence that convalescent plasma has little or no benefit in the treatment of people with moderate to severe COVID-19, and there may be little or no benefit for people with mild COVID-19 either.
Beware bold statements
Beware bold statements that something is ineffective or is no different to another treatment. That there is no evidence of an effect or difference is quite different to saying that there is no effect or difference.
Just remember that cosmic teapot and invisible unicorn….
Join in the conversation on Twitter with @CochraneUK @lynda_ware or leave a comment on the blog. Please note, we cannot give medical advice and we will not publish comments that link to commercial sites or appear to endorse commercial products.
Lynda Ware has nothing to disclose.
If something does not exist , there will be no evidence of it’s existence. But there can be Evidence of it’s absence, and hence can be proved it doesn’t exist.
Very good topic and a good read.
In a Bayesian setting, absence of evidence for a hypothesis H does typically imply evidence against H. For let E be the claim that there is evidence for H. Then, typically (i.e., assuming some technical assumptions) we have P(H | E) > P(H), i.e., the hypothesis H becomes more likely given E — the presence of evidence would be evidence for H. But then it follows mathematically that P(not-H | not-E) > P(not-H), i.e., the absence of E raises the probability of not-H, so the claim not-E that there is no evidence for H does indeed provide evidence against H.
For a concrete example, let U be the hypothesis that an invisible unicorn has been grazing in Dr. Burton’s office. Let E1 be this possible piece of evidence: No animal was seen but some of Dr. Burton’s books have been chewed on. Then, the presence of E1 would be evidence for U. It would, of course, be far from conclusive evidence — there are many more likely hypotheses than U to explain E1 — but it would be *some* evidence for U. Why? Because E1 is more likely to occur if there is an invisible unicorn grazing than if there isn’t, so P(E1|U)>P(E1|not-U), and by Bayes’ Theorem (assuming all the relevant hypotheses have probabilities other than 0 and 1) it follows that P(U|E1)>P(U). But then it also follows mathematically that P(not-U|not-E1)>P(not-U). Thus the absence of the evidence E1 for a unicorn raises the probability that there is no unicorn. And we can run this argument for every possible piece of evidence for U, and together the absence of all these pieces raises the probability that there is no unicorn even more.
None of that need be a *proof* that there is no invisible unicorn. But it is some *evidence* against the invisible unicorn. And the original maxim only say that “Absence of evidence is not *evidence* of absence.”
If something does not exist, there will be no evidence for its existence, i.e. an absence of evidence.
So why is this absence of evidence not evidence for absence?
Because our base of evidence will always be flawed. We never know everything, hence the lack of knowing is not evidence that there is nothing to know.
We incorporate this in practice within business and economics, using proxy or interim measures to guide decision making in the absence of full evidence. If we didn’t take this approach then we would only perform actions which we could measure the success of (e.g. what gets measured gets done).
Try reading about the johari window….. I found it very useful for understanding different levels of knowledge
Because it wasn’t reported, doesn’t mean it doesn’t exist. The small sample size is another reason absence of evidence is not evidence of absence.
Great article, but, still, we have to give merits to the ones that are trying to find a solution.