Shaun Treweek, Professor of Health Services Research at the University of Aberdeen, points a finger at the thin evidence base for trial process decision-making and highlights a new funding initiative from the National Institute for Health Research that will help.
There are many things that need to be considered when making a decision about treatment but one them is definitely evidence. We want health professionals and others to be thinking about systematic review evidence (or guidelines that use it) when discussing our treatment options. What none of us want is for our health professionals to take a punt on something for which there is no evidence to speak of but hey, it seemed to work last time. Those of us who work in trials think this is why we do them: to make those treatment decisions evidence-informed decisions.
So, let’s imagine we’re going to do a trial. How do people like me, trialists and methodologists, make our decisions about, say, recruitment, retention, training, site selection, data collection, or dissemination of results to participants?
Well, much of the time we take a punt on something for which there is no evidence to speak of but hey, it seemed to work last time.
To put it charitably, this is odd. Given that hundreds of thousands of trials have been done and tens of thousands are ongoing right now, you’d be forgiven for thinking that there’d be plenty of research to support the process choices made in every trial. Sadly, you’d be wrong. There is in fact remarkably little evidence available to support the doing of trials, and trials are much less efficient than they could be as a result.
For example, the 2018 update of the Cochrane Review of interventions to improve trial recruitment found high certainty evidence for just three things and even one of those has a caveat. It is no surprise that poor recruitment terrifies everyone who works in trials: it can and does kill trials. We ought to be equally terrified about retention because the Cochrane Review of interventions to improve retention finds little evidence and nothing at all to help with face-to-face visits, a feature of many if not most trials. There is almost no evidence to support trial management decisions.
Evidence for trial process choices: introducing SWAT
This is what makes the National Institute for Heath Research (NIHR) new funding stream for ‘Studies Within A Trial (SWATs)’ in the Health Technology Assessment (HTA) program so fabulous. A SWAT is a self-contained research study that is embedded in a host trial in order to evaluate or explore alternative ways of delivering or organising a particular trial process. SWATs can answer questions such as ‘Will 6-monthly newsletters help keep patients in the trial?’, ‘Will face-to-face site initiation visits improve data quality?’, or ‘How much more accurate is it to use specially collected data rather than routine data for follow up?’
These are the sorts of question asked every day in trials but the evidence to support their answers and the decisions that are made is equivocal at best. There might be as much uncertainty about these issues as there is about differences between the treatments being tested in the trial and a decision one way or the other has consequences. Newsletters do not write themselves. Trial managers might have to spend two days out of the office to set up each site. Funders will have to pay for the special data collection. SWATs can provide the evidence on whether these activities are worth it.
The new NIHR funding makes it possible to build a SWAT costing up to £10,000 into every HTA trial. There is some guidance on how to design and run SWATs and for recruitment there is a James Lind Alliance Priority Setting Partnership Top 10 list of unanswered research questions that are prime targets for SWATs. A similar priority setting process for retention is underway right now (a blog on this, PRIORITY 2, will be published on Evidently Cochrane tomorrow). The Cochrane recruitment review not only says what research is needed but gives the protocols for three priority SWATs. The SWAT repository will also help because it provides outlines for more than 50 existing SWATs. What would help even more is if SWATs are coordinated, so that we get evidence from several evaluations quickly. This body of evidence can then be used to inform trial process decisions and researchers can move on to other SWATs, rather than have single SWAT evaluations floating around for years but unable to really help decision making.
Sources of support
There’s support too. Trial Forge ([email protected]) can help and so can a group led by David Torgerson and Adwoa Parker at the York Trials Unit ([email protected]). In addition to having done many SWATs, they lead an MRC-funded project called PROMETHEUS on recruitment and retention SWATs in ongoing trials. They can advise on SWATs to build into your new HTA proposal, have a funding scheme for putting SWATs into ongoing trials, can coordinate with Trial Forge and others as well as provide practical advice about how to do the SWAT.
This is a good time for trial process evidence in the UK. This new NIHR funding and several other initiatives are all pushing in the same direction. Although SWATs will not solve all our problems, the results from bundles of them will improve decisions and choices about trial processes. In short, if you’re doing a trial, do a SWAT.
In the video below, hear Shaun’s thoughts on why we should “all celebrate and support randomized trials”.
Shaun Treweek has nothing to disclose.
References may be found here.