Lessons in disappointment and some New Year’s resolutions

Broken your New Year’s resolutions yet? Research from the US suggests just 8% of people achieve theirs and we know that if you’re hanging onto them into February you’re doing really well. When I thought about making a list of resolutions which clinical trialists and systematic reviewers might like to follow (the cheek of it!), it looked like a welcome distraction from deciding I’m going to take up running (again), eat only healthy snacks and get 10 hours sleep a night. But I find I am coming to this with a rising sense of sadness, disappointment and, well, outrage about some of the evidence gaps. Gaps which leave people struggling with health problems, and those trying to help them, without clear guidance on which interventions might help; and, indeed, on which may make things worse.

I saw the titles of several newly published or updated Cochrane reviewsCochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. More at the tail end of the year and thought, great, these look important. But most of them were ’empty reviews’, with no evidence at all, or fell into that all too familiar category of reviews where the reviewers were unable to come to clear conclusions because of problems with the evidence.

Pass the parcel – what no present?!

Many of us will remember this game from our childhood. A parcel is handed round and when the music stops the person holding it gets to unwrap a layer of paper. The last one finds a present, but only after lots of people have had their hopes raised only to find nothing under the wrapper. It’s a bit like this with ’empty reviews’. The review team sets out to answer an important clinical question by devising complicated search strategies and using them to hunt through vast databases of clinical trialsClinical trials are research studies involving people who use healthcare services. They often compare a new or different treatment with the best treatment currently available. This is to test whether the new or different treatment is safe, effective and any better than what is currently used. No matter how promising a new treatment may appear during tests in a laboratory, it must go through clinical trials before its benefits and risks can really be known. More, looking for some relevant studies to include. Then find – none.

Here are the pressing needs addressed in those recent reviews:

• alternatives to hospital follow-up after treatmentSomething done with the aim of improving health or relieving suffering. For example, medicines, surgery, psychological and physical therapies, diet and exercise changes. More for gynaecological cancer
• effective ways to manage faecal incontinence, constipation and urinary catheters in people with neurological diseases
• guidance on the use of prophylactic antibiotics for people with penetrating abdominal injuries
• preventing the spread of MRSA among older people in nursing homes
• whether end-of-life care pathways are an effective means of providing high quality care

This is really important stuff, right? Take the first one, follow-up after gynaecological cancer treatment to detect recurrences, a new review published in November. This usually involves multiple hospital visits that cause anxiety among the women, its cost effectivenessThe ability of an intervention (for example a drug, surgery, or exercise) to produce a desired effect, such as reduce symptoms. More is questionable, and there’s no evidence that this method identifies recurrences earlier or improves overall survival. Patient groups have called for a consideration of alternatives, such as using patient-reported outcome measures (PROMs) questionnaires through which people can give detailed information about their health and associated quality of life. These are increasingly being used as part of routine follow-up. Bring on a systematic review. The review team didn’t expect to get randomizedRandomization is the process of randomly dividing into groups the people taking part in a trial. One group (the intervention group) will be given the intervention being tested (for example a drug, surgery, or exercise) and compared with a group which does not receive the intervention (the control group).   More controlled trialsA trial in which a group (the ‘intervention group’) is given a intervention being tested (for example a drug, surgery, or exercise) is compared with a group which does not receive the intervention (the ‘control group’). More (RCTs) so looked for studies of other designs too. They found none.

You might think that it’s just a matter of research catching up with changes in practice. But we have been doing things for a long time for which there’s still no research evidence. Since World War II antibiotics have been given to people with penetrating abdominal injuries in the hope of preventing wound infection. But is this best practice? Problems of antibiotic resistance, side effects and costs are reasons not to use antibiotics unnecessarily and then there’s debate about timing. A Cochrane review on this has recently been updated for the second time and still there were no trials to include. Guidelines, of necessity, remain expert-based not evidence-based.

Then there’s bladder and bowel business. Can there be many things which would make more difference to the quality of life of people who have severe problems with these than things to help manage them effectively and alleviate problems? For the person with nerve damage affecting their bladder, leaking urine or problems with passing it, and a riskA way of expressing the chance of an event taking place, expressed as the number of events divided by the total number of observations or people. It can be stated as ‘the chance of falling were one in four’ (1/4 = 25%). This measure is good no matter the incidence of events i.e. common or infrequent. More of infection and kidney damage, have to be managed, but how best to do it? How about a permanent catheter, or perhaps use one intermittently, or forget the catheter and pass urine at timed intervals? It’s important to know how effective these are and also how easy to use, how comfortable and, for the NHS purse, how they compare in terms of costs. A review addressing this was first published a decade ago and has recently been updated again, but still found no trials and can’t offer any evidence to guide practice.

A review on managing faecal incontinence and constipation in people with neurological diseases has just been updated for the third time and despite the addition of 15 trials, bringing the total to 20, the reviewers are still unable to draw any reliable conclusions due to the limitations of the evidence. These displayed some all too familiar problems, not least poor reporting. ‘Usual care’ is a common comparator in trials but utterly meaningless unless you explain what the ‘usual care’ was. In most of the trials in this review, the authors left us to guess.

Small sample sizes in the trials included in this review meant that confidence intervals were wide, so we can’t be very sure that the findings are reliable. Then there was such variety in the choice of outcome measures that putting together the results was rendered inappropriate. Even identifying where future research activity should be focused was made difficult for the reviewers because of the lack of condition-specific, validated outcome measures, little evidence to support any interventionA treatment, procedure or programme of health care that has the potential to change the course of events of a healthcare condition. Examples include  a drug, surgery, exercise or counselling.   More and, crucially, “an absence of reliable information about what is important to patients regarding the outcomesOutcomes are measures of health (for example quality of life, pain, blood sugar levels) that can be used to assess the effectiveness and safety of a treatment or other intervention (for example a drug, surgery, or exercise). In research, the outcomes considered most important are ‘primary outcomes’ and those considered less important are ‘secondary outcomes’. More of bowel management”. Which brings me to the New Year’s resolutions list. The high quality research we need demands attention to all aspects of the way it’s designed and carried out, but here are the things at the top of my list. I hope we might have a blog on each of these points over the coming months, but right now I’m going to point you to some places you should go if you’re interested.

• Identify priorities for research (both primary research and systematic reviewsIn systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. More). The James Lind Alliance brings patients and clinicians together in Priority Setting Partnerships to prioritise topics for research and you can find them here
• Get the outcomes right. This should include the development of core outcome sets for specific populations. Validated measures that focus on things that matter to patients and are common to research investigating the same area will really help the evaluation of interventions and the pooling of results in systematic reviews. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative brings together people interested in the development and application of agreed standardised sets of outcomes. Trialists wouldn’t be restricted to that set; rather, it would be expected that these outcomes would be collected and reported, along with others that the researchers wish to explore. Read more here
• Better reporting (poor reporting is the killer for much research that may be good but is rendered unusable by all that’s left out at the reporting stage). You need to dash over to The EQUATOR network, which “works to improve the reliability and value of medical research literature by promoting transparent and accurate reporting of research studies” and has links to so many useful resources on this topic  http://www.equator-network.org/

‘Empty’ things sometimes contain something valuable!

Empty reviews lack the stuff we hoped for – relevant trials to help answer the question – but they do contain other valuable things. As Joanne Yaffe and colleagues pointed out in their study of empty Cochrane reviews, the absence of studies might reflect a topic area that is too new to be ready for review, or perhaps priorities of health care decision makers and researchers don’t much overlap, but either way the absence of evidence might help stimulate appropriate research. The hope is that the review will then be updated with eligible studies. Good reporting is vital here too; you can see a poster on this and more at the Cochrane Empty Reviews Project website.

Happy New Year!

Links:

Nama V, Nordin A, Bryant A. Patient-reported outcome measures for follow-up after gynaecological cancer treatment. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD010299. DOI: 10.1002/14651858.CD010299.pub2.

Brand M, Grieve A. Prophylactic antibiotics for penetrating abdominal trauma. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD007370. DOI: 10.1002/14651858.CD007370.pub3

Jamison J, Maguire S, McCann J. Catheter policies for management of long term voiding problems in adults with neurogenic bladder disorders. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD004375. DOI: 10.1002/14651858.CD004375.pub4.

Coggrave M, Norton C, Cody JD. Management of faecal incontinence and constipation in adults with central neurological diseases. Cochrane Database of Systematic Reviews 2014, Issue 1. Art. No.: CD002115. DOI: 10.1002/14651858.CD002115.pub5.

Hughes C, Tunney M, Bradley MC. Infection control strategies for preventing the transmission of meticillin-resistant Staphylococcus aureus (MRSA) in nursing homes for older people. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD006354. DOI: 10.1002/14651858.CD006354.pub4.

Chan RJ, Webster J. End-of-life care pathways for improving outcomes in caring for the dying. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD008006. DOI: 10.1002/14651858.CD008006.pub3.

Yaffe J, Montgomery P, Hopewell S, Shepard LD (2012) Empty Reviews: A Description and Consideration of Cochrane Systematic Reviews with No Included Studies. PLoS ONE 7(5): e36626, doi:10.1371/journal.pone.0036626

This page was last updated: 31 March 2014