Teapots and unicorns: absence of evidence is not evidence of absence

In the fourth blog of our special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice, Lynda Ware, a Senior Fellow in General Practice at Cochrane UK, explains that absence of evidence is not evidence of absence and why it’s important not to mistake one for the other, with some examples from Cochrane Reviews and the media.

Absence of evidence is not evidence of absence: no one seems to be sure where this phrase came from, although it is commonly attributed to the Astronomer Royal, Martin Rees, a British astrophysicist and cosmologist.

It captures in eight words the important message that it is pretty much impossible to prove a negative – that is, that something doesn’t exist.

The philosopher and mathematician Betrand Russell waded in on the argument. He used the example of a ‘cosmic teapot’ to illustrate his point. He said that, even though it could not be refuted, no one should believe him if he claimed, without proof, that a tiny teapot was orbiting the sun between Earth and Mars (Boncheck, 2016).

In similar vein, Martin Burton, Director of Cochrane UK, wrote that an invisible unicorn had been grazing in his office and invited readers to prove him wrong!

It is difficult to prove that an invisible unicorn doesn't exist

It is pretty much impossible to prove a negative – that is, that something doesn’t exist

An important concept

In the realms of medical science the concept is important. It is claimed, not infrequently, that a treatment ‘has no effect’ or that ‘there is no difference’ between two treatments. These claims can be found not only in the media but in the scientific papers from which the headlines derive. We are led to believe that if there is no reliable evidence proving an effect or a difference then they do not exist. This is wrong.

Evidence may be unreliable for several reasons such as:

  • The studies may be too small to detect an effect or a difference.
  • The effect or difference may be very, very small.
  • There may be too few data or not enough studies.
  • The evidence may be of very low quality.
  • The studies may have been poorly designed.

Lost in translation: some examples from the media

    “Giving children full-fat milk will not make them obese, study finds.” (Telegraph, March 2020)

A bold, contradict-me-if-you-dare, statement that full-fat milk is not linked to childhood obesity. Now this may be true, but the research stated that there was no evidence that full-fat milk and dairy products lead to overweight children. There is always the possibility that an effect has not been detected or is very small. It’s not possible to conclusively prove that something doesn’t exist.

And another….

       “Using talcum powder does NOT raise the risk of ovarian cancer as study of 250000 women debunks fears after decades of uncertainty.” (Mail Online, January 2020)

This headline comes from a review of four large clinical studies looking at whether using talcum powder on the vulva is linked to an increase in ovarian tumours. Although the findings are reassuring, the researchers point out that they are unable to exclude the possibility of a very small increase in ovarian cancer. They explain that ovarian cancer is a relatively rare finding and that the studies may have been flawed.

Absence of evidence: some Cochrane Reviews

Let’s look now at some Cochrane reviews to give two more examples.

Convalescent plasma or hyperimmune immunoglobulin for people with COVID‐19: a rapid review

This important and topical systematic review considers whether giving plasma from people who have recovered from COVID-19, and which is full of antibodies to the virus, helps those who are critically ill with the infection. It is one of Cochrane’s rapid reviews, which have been produced in response to the pandemic.

In October 2020 the second update of the review was published analysing data from 19 trials. Despite including over 38000 participants, more than 36000 of whom received convalescent plasma, the authors conclude that they are uncertain whether it is beneficial to people admitted to hospital with COVID-19. Most of the studies did not include a control group and so they are also unable to assess the relative safety of convalescent plasma therapy.

 As yet there is insufficient high certainty evidence to know the answer to the question. In other words, there is no reliable evidence that convalescent plasma helps or harms. There are 138 ongoing studies that will be assessed as they are published and their data will be included in the review. 73 are randomized controlled trials, so there is hope that more robust evidence will soon be available.

""

Convalescent plasma for treating COVID-19: as yet there is no evidence that it helps or harms.

Interventions for managing halitosis (bad breath)

This review considers different ways to help halitosis (bad breath). These include mechanical cleaning (for example, tongue cleaners and toothbrushes), chewing gums, systemic deodorising agents (for example, mushroom extract that you eat), topical agents (such as gel that you apply), toothpastes, mouthrinse/mouthwash, tablets, and combination of different treatments. It is uncertain which, if any, work best at controlling bad breath. This is not to say none works but that there is insufficient, good quality evidence to guide us.

Beware bold statements

Beware bold statements that something is ineffective or is no different to another treatment. That there is no evidence of an effect or difference is quite different to saying that there is no effect or difference.

Just remember that cosmic teapot and invisible unicorn….

Take-home points

Take-home points: it is pretty much impossible to prove a negative - that is, that something doesn’t exist. 10:35 Beware bold statements that something is ineffective or is no different to another treatment. There are many reasons why evidence may be unreliable.

Join in the conversation on Twitter with @CochraneUK @lynda_ware or leave a comment on the blog. Please note, we cannot give medical advice and we will not publish comments that link to commercial sites or appear to endorse commercial products.

References (pdf)

Lynda Ware has nothing to disclose.

Editor’s note: this blog was updated in November 2020, to take account of the latest update of the review ‘Convalescent plasma or hyperimmune immunoglobulin for people with COVID‐19: a living systematic review‘.



Teapots and unicorns: absence of evidence is not evidence of absence by Lynda Ware

is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International

4 Comments on this post

  1. Avatar

    If something does not exist, there will be no evidence for its existence, i.e. an absence of evidence.
    So why is this absence of evidence not evidence for absence?

    John Rudkin / Reply
    • Avatar

      Because our base of evidence will always be flawed. We never know everything, hence the lack of knowing is not evidence that there is nothing to know.

      We incorporate this in practice within business and economics, using proxy or interim measures to guide decision making in the absence of full evidence. If we didn’t take this approach then we would only perform actions which we could measure the success of (e.g. what gets measured gets done).

    • Avatar

      Try reading about the johari window….. I found it very useful for understanding different levels of knowledge

      Annabel Dawson / (in reply to John Rudkin) Reply
  2. Avatar

    Great article, but, still, we have to give merits to the ones that are trying to find a solution.

Leave a Reply

Your email address will not be published. Required fields are marked *

*