Publication bias: a problem that leaves us without the full picture on the benefits and harms of treatments

In the final blog of our special series on Evidently Cochrane: “Oh, really?” 12 things to help you question health advice, Selena Ryan-Vig, one of Cochrane UK’s Knowledge Brokers, discusses the issue of ‘publication bias’ and why it is so problematic for people making decisions about health care.

Page last checked 28 June 2023

Take-home points

You are buying a new car. To understand how good and how safe the car is, you carry out an online search and find 20 reviews, including user feedback and tests. Many of these show the car in a favourable light. You are impressed and keen to buy it.

But what if you discovered that over 20 more reviews existed? Only you hadn’t been able to find them because they had never been made readily available. Even more concerning, what if the hidden reviews were more likely to be negative than those you found, showing the car in a less favourable light? Your opinion of the car could be biased. You might overestimate how good it is and underestimate any potential safety issues.

Now what if the thing you’d like to be informed about isn’t a car, but a medical treatment. Unfortunately, this is far from an imaginary issue with medical research. This was demonstrated in a Cochrane Review which set out to explore just how much biomedical research goes unpublished…

The studies that go unpublished

Researchers often present abstracts or summaries of their research at scientific conferences and meetings. They should then go on to publish their findings, in full, as articles in journals. In part, this is because abstracts presented at conferences are not particularly easy for people searching for evidence to find. In contrast, published journal articles contain more information about the study and are much easier to find.

So, in order to investigate just how much medical research is ‘hidden from view’, the authors of the Cochrane Review ‘Full publication of results initially presented in abstracts’ found 425 reports which had explored how many abstracts presented at conferences go on to be published in full within the following 10 years. These reports included a large sample of 307,028 abstracts presented at conferences.

The review authors observed that less than half of all studies, and around two‐thirds of randomized trials, initially presented as abstracts at scientific meetings, went on to be published as journal articles.

There were a number of factors associated with studies being more likely to be published. Critically, one factor was that those published were more likely to contain positive results than those that weren’t. In other words, the studies ‘hidden from view’ are less likely to show the treatment of interest (whether that is a medication, therapy or so on) in a favourable light. This is known as ‘publication bias’.

Why does publication bias occur?

There are many possible reasons for publication bias.

Researchers may simply not submit their research for publication, perhaps because they have competing commitments or feel that ‘negative’ results are uninteresting. Journal editors, who decide which studies to publish, may also reject submitted research because the results are not seen as ‘exciting’.

Conflicts of interest may also play a role; some researchers may have a vested interest in presenting a particular medication or therapy positively and wish to avoid publishing findings that suggest otherwise. Whatever the reason or motivation behind a failure to publish – publication bias is a problem. It leaves us without the full picture of the benefits and harms of treatments.

Publication bias: what impact does it have? 

It is difficult to know how large an impact publication bias has. Large-scale research has tended to focus on documenting how common the issue is, rather than its impact. Nonetheless, the Catalogue of bias collaboration state that the impact may “vary from the non-publication of a single notable study, to affecting the complete assessment of a therapeutic area”.

Regardless of the scale of its effect on any given therapeutic area, publication bias is not just a scientific problem, but an ethical wrongdoing. People who participate in research, and the bodies that fund it, do so with the understanding that the research will add to knowledge. Failing to report research is a waste of time and resources and it undermines the ability to make truly evidence-informed decisions about health care.

Further, much research is publicly funded, making it even more important that it is publicly accessible. The Department of Health and Social Care (DHSC) and the National Institute for Health Research (NIHR), who fund much health research in the UK, stress that: “ideas and knowledge derived from publicly-funded research must be made available and accessible for public use, interrogation and scrutiny as widely, rapidly and effectively as possible”.

Dealing with, and preventing, publication bias

There are a number of ways that publication bias can be – and is – tackled and prevented.

Taking account of publication bias: the role of systematic reviewers

Authors of systematic reviews, such as Cochrane Reviews, can take steps to reduce the impact of publication bias. A systematic review summarises evidence from multiple studies to answer a specific question. (For example, what are the benefits and harms of a particular treatment for a particular health condition?). In order to give a complete picture of the benefits and safety of any given treatment, it is important that systematic reviews include all of the studies the authors can find that meet certain criteria (to do with the research question and study design for example). Studies should be included whatever their results are –  regardless of whether they show the treatment in a positive light.

Authors of systematic reviews should make efforts to search for unpublished research, in trial registries for example. They can also contact researchers who they know (or suspect) have not published their work and request that they share their findings.

However, even comprehensive efforts may fail to identify all unpublished studies. So systematic reviewers can also use statistical methods to estimate whether their analyses are likely to be affected by publication bias. They can then take this into account when making judgements about what the evidence tells them about the intervention.

An example comes from a Cochrane Review on topical benzoyl peroxide (BPO), a widely used acne treatment. The authors set out to find all the trials exploring the effects and safety of this treatment. For some of the comparisons and outcomes, the authors suspected publication bias. This was because, when they conducted their search for studies in 2019, they found some trials which had started before 2015 that were still shown as “ongoing” in the trial registries.

As a result – and due to concerns about multiple other types of bias affecting the studies they had found – the authors ‘downgraded’ the certainty of the evidence. In other words, they lowered their confidence in their estimates of the effects that BPO has. For example, regarding safety, the authors concluded: “Very low‐certainty evidence means that we are uncertain if BPO leads to more side effects among participants receiving medium‐ to long‐term BPO than among those given no treatment/placebo”.

It is important that systematic reviewers are able to take account of bias (including publication bias) in this way. Yet this is no real consolation for patients, and anyone else making treatment decisions, when they are left with uncertainty about the potential benefits and harms of treatments. Ultimately, the best cure for publication bias is prevention.

Preventing publication bias: the role of journals and the law

Preventative efforts take many forms.

In recent times, there have been prominent campaigns to stamp out publication bias (such as the AllTrials campaign). Some journals (such as ‘Trials’) have made it a core part of their mission to publish studies which report ‘non-positive’ results.

Additionally, guidelines and laws have been introduced. These dictate that the pharmaceutical companies or universities conducting research must register their study before it can begin, in searchable clinical trial registries such as clinicaltrials.gov. Then, when the study is completed, researchers should report their findings directly on the registries in a timely manner (within 12 months, for example). Failure to do so could result in legal action, including hefty fines.

However, there are concerns about how effectively such laws are enforced. Efforts may also be undermined by loopholes and poor compliance. So, although recent years have seen steps in the right direction, there is still a long way to go.

References (pdf)

Join in the conversation on Twitter with @CochraneUK or leave a comment on the blog. Please note, we cannot give medical advice and we will not publish comments that link to commercial sites or appear to endorse commercial products. We welcome diverse views and encourage discussion. However we ask that comments are respectful and reserve the right to not publish comments we consider offensive.

Visit the Teachers of Evidence-Based Health Care website, where you can find resources which explain and illustrate why all fair comparisons and outcomes should be reported.

This series of blogs is inspired by a list of ‘Key Concepts’ developed by the Informed Health Choices.

Read the rest of the blogs in the series: “Oh, really?” 12 things to help you question health advice.

Selena Ryan-Vig has nothing to disclose.