Writing a Cochrane Review: my experiences of a diagnostic test accuracy review

In a blog for anyone interested in dementia diagnosis or the process of conducting a Cochrane diagnostic test accuracy review, Dr Lucy Beishon from University of Leicester discusses her recently published Cochrane Review and her experiences of working with the Cochrane Dementia and Cognitive Improvement Group.

How it began

I’ve been using the dementia screening test called Addenbrooke’s Cognitive Examination-III (or ACE-III for short!) in both my research studies, and in memory clinic.  I use the test to help decide whether a patient I’m seeing might have a diagnosis of dementia. I must have used the ACE-III so many times that I could complete the test blindfolded by now! But despite this I’ve never asked where did this test come from and what is the evidence to say it is good enough to be supporting us in making a diagnosis of dementia?

I had never asked where did this test come from and what is the evidence to say it is good enough to be supporting us in making a diagnosis of dementia?

A question that needs an answer

Giving a diagnosis of dementia is not something any clinician would do lightly. The diagnosis can have important consequences for people in terms of occupation, driving and insurance, not to mention the stress and stigma that is often attached to a diagnosis. So having the right tool to help us determine as accurately as possible if someone has dementia is crucial. It’s important to say at this point we would never completely base a diagnosis of dementia on the results of one of these tests alone.  Rather these tests form part of a more comprehensive, multidisciplinary assessment. Using ACE-III Is not unique to my hospital, this is a test used routinely in memory clinics and wards up and down the country to support the assessment of memory and thinking problems. So now I have my first stage in a Cochrane Review – a question that needs an answer!

In this blog I am going to take you through my journey working on this recent Cochrane Review of the test accuracy of ACE-III and its shorter version the mini-ACE.

The proposal

One of the things I didn’t anticipate was the number of stages required in writing a Cochrane Review. Writing the proposal is the first stage for any budding Cochrane reviewer.  This is where I began to frame and develop my initial question about the ACE-III and mini-ACE. Writing the proposal is a bit like a business pitch in the Dragon’s den, but this time you are pitching your idea to Cochrane.  The Cochrane team review the question and determine whether it is suitable for a review, and whether it needs answering at all!

Writing the proposal allowed me to think carefully about the questions I wanted to ask, what did I want to find out by doing this review? I also began to learn and understand more about the background behind these tests and where they came from. I decided to focus on the most recent version of the ACE-III. Although there had been a number of other versions of ACE used in the past, these had largely been replaced by the ACE-III and mini-ACE. We also decided to look at using the test in various healthcare settings to see whether the ACE-III and mini-ACE were better at detecting dementia in a particular area (i.e. in hospitals, clinics, or GP practices).

The protocol

Once Cochrane have accepted your proposal, the next stage is to outline what you are going to do in a protocol. It is good practice in any research to say what you are going to do, how you will do it and then try and stick as closely as possible to this plan. Writing the protocol turned out to be a great learning experience. When I was at medical school we were often taught how to tell a good study from a bad one, but these were nearly always studies looking at medications or other treatments. We didn’t really consider research looking at the accuracy of tests and yet clinicians use multiple tests every day.

After writing this review I have now learnt how to tell a good test accuracy study from a bad one, and which studies might have misleading results because of the methods they used.  For example, some studies will take the test (such as ACE-III) and use it with patients where the researchers already know they have a diagnosis of dementia, and at the same time use the test on a group of healthy people with no memory problems (often called controls). This type of study is commonly referred to as a case-control study. The problem with this design of study is that it often compares the test between patients who can have quite advanced or severe problems and people who have no problems at all.  This can make the test seem much better at diagnosing dementia than it actually is. It also misses out much of the “grey area” in between, that is those people who have milder memory problems where it can be tricky to make a diagnosis of dementia. In my experience these more challenging cases constitute the majority of clinical work.  So because of the concerns around the case-control approach we did not include these types of studies in the review.

The review

This is now the main event and where the majority of the work happens to take the review to completion. Undertaking a Cochrane Review involves quite a number of stages to ensure the review is conducted in line with the best methods that are currently available, and therefore provides the best up-to-date summary of all the available evidence on that topic.

Looking for studies

The first stage is to search all the available databases for studies that might be relevant to your review question, and in our review the search found 5655 studies in total. However, after we removed all the studies which were either duplicated or not relevant to the review question, we had only seven left! This was perhaps the most surprising finding for me, for a test that is so widely used there were actually very few studies that met the criteria to be included in the review. In total across these seven studies there were 1711 patients.

After we removed all the studies which were either duplicated or not relevant to the review question, we had only seven left to include in the Cochrane Review!

Assessing study quality

Now that we had our studies, the next stage was to examine the quality of these studies: how good were the methods they used to look at the accuracy of the ACE-III and the mini-ACE? For this aspect we used two reviewers who completed this work independent of one another and then compared results. This ensured quality control and fairness. On the couple of occasions where I disagreed with my fellow reviewer, one of the Cochrane editors made the final decision. At this stage I was again quite surprised at the quality of the published research. Some of the studies were good, but there were also studies where we had a number of concerns, particularly as to the way in which the ACE-III or mini-ACE was implemented and whether the test was completed masked to the results of the comparison ‘gold standard’ dementia assessment. This is important because if the ACE-III and mini-ACE were carried out by the same person who did the comparison test, this could influence the results of the study and the test may seem better than it really is.

Finding, appraising and summarising the information from studies

Once we had assessed the quality of the seven studies we again used two reviewers to extract all the relevant information from the studies to go into the review.  At this point we realised there would not be enough studies to combine together using statistical methods, known as a meta-analysis. Studies looking at accuracy often only include modest numbers of participants and the results around can be imprecise. A meta-analysis allows us to combine all the results together, increasing the number of people studied and hopefully gives a more robust result.

Although formal meta-analysis was not possible, we were still able to summarise the results of the studies and determine that the ACE-III and mini-ACE were reasonably good at identifying dementia (correct 20-99% of the time) but their ability to rule out a diagnosis of dementia was not as good. It seemed the ACE-III was particularly poor for identifying problems with memory after a stroke. The other surprising result was that all studies had been conducted in a hospital setting and none in community or general practice, so we don’t know whether these tests are useful or work at all outside of a hospital setting.

Peer review then publication!

We’re nearly there! The final stage in producing a Cochrane Review is to publish it so that it can be accessed by others as a source of information. To do this, the review has to be evaluated through a process known as peer review. Peer review means the review is closely examined by a number of experts in the field who check the review methods, consistency and accuracy to make sure it is worthy of publication. This process helps keep the standards high for research in general, but the Cochrane peer review is a particularly rigorous process using experts from a range of areas. The reviewers all provided useful comments that allowed me to revise my review and (hopefully) improve its quality.

Final thoughts

In summary, my experience of working on a Cochrane Review has been overwhelmingly positive. Although at times the work involved and number of steps seems difficult, this is much outweighed by the skills, knowledge, and experience working on a Cochrane Review gives back to you. I think for any budding researcher the opportunity to work on a Cochrane Review is a must-have experience and I look forward to working on my next review with the Cochrane Dementia Group.

Take-home points

Many tests are used to help make a dementia diagnosis, as part of a more comprehensive, multidisciplinary assessment. It is important to consider what is the evidence for a test, to decide if it is good enough to support making a diagnosis. A good Cochrane review always starts with a good question that needs an answer. Undertaking a Cochrane Review is complex but rewarding.

 

What do you think of Lucy’s review or her Cochrane experiences? Join in the conversation on Twitter with @LBeishon @CochraneDCIG @CochraneUK or leave a comment on the blog. Please note, we will not publish comments that link to commercial sites or appear to endorse commercial products.

Click here for the Cochrane ReviewBeishon  LC, Batterham  AP, Quinn  TJ, Nelson  CP, Panerai  RB, Robinson  T, Haunton  VJ. Addenbrooke’s Cognitive Examination III (ACE‐III) and mini‐ACE for the detection of dementia and mild cognitive impairment. Cochrane Database of Systematic Reviews 2019, Issue 12. Art. No.: CD013282. DOI: 10.1002/14651858.CD013282.pub2.

Lucy Beishon has nothing to disclose.


Lucy Beishon

About Lucy Beishon

view all posts

I have been working at the University of Leicester as a Clinical Research Fellow in Geriatric medicine, with an interest in cognitive disorders. I am particularly interested in how brain blood flow is altered or damaged in dementia and how this contributes to the development or progression of different types of dementia. I am currently funded by the Dunhill Medical Trust on my research training fellowship project entitled ‘The Cognition & Flow Study: The effects of brain training on brain blood flow.’ This study aims to investigate the use of cognitive or “brain training “as a treatment for Alzheimer’s disease, and mild cognitive impairment, and as prevention in healthy older adults. In addition to my research work, I am also a specialist trainee in geriatric medicine and my long-term goal is to work as a geriatrician with clinical and research interests in dementia. In the future I plan to investigate the role of cognitive training, and the effects on brain blood flow in dementia, in a larger multi-centre study.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

UA-49496932-1