In this blog for our Understanding Evidence series, Selena Ryan-Vig, Cochrane UK’s Knowledge and Engagement Officer, highlights some important considerations when reading research by looking at Cochrane evidenceCochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. on cocoa and blood pressure.
Page last checked 10 March 2023
At first glance, a recently updated Cochrane ReviewCochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. exploring the Effect of cocoa on blood pressure (published April 2017) may come as happy reading for chocolate lovers (well, lovers of flavanol-rich chocolate, specifically). The review is also an interesting case studyAn investigation of a healthcare problem. There are different types of studies used to answer research questions, for example randomised controlled trials or observational studies.. It provides food for thought about several issues we need to be aware of when we’re making sense of research in general and the types of claims that can be drawn from the evidence available to us. Before we get to that, let’s look at the review.
Interest in exploring the medicinal potential of chocolate comes from the fact that cocoa products contain flavanols. These are plant-based nutrients, thought to lower blood pressure by widening the blood vessels.
This review updated a previous version published in 2012, and included 17 more studies. This gave a total of 35 studies involving 1804 (mainly healthy) participants. Every day for at least two weeks, intervention groupA group of people in a study receiving a particular health care intervention (for example a drug, surgery, or exercise). participants consumed flavanol-rich chocolate (30 to 1218mg of flavanols in 1.4 to 105 grams of chocolate). Control group participants consumed either low-flavanol-containing chocolate or a flavanol-free product.
There was moderate-quality evidence that, compared with control participants, individuals consuming flavanol-rich chocolate experienced a small, statistically significantA statistically significant result is one which is unlikely to have happened by chance. This is not the same thing as clinical significance. reduction in blood pressure (1.8mmHg for both systolic and diastolic blood pressure).
Statistically significant, but is it clinically important?
Before we all dash off to stock up on flavanol-rich chocolate (or indeed any interventionA treatment, procedure or programme of health care that has the potential to change the course of events of a healthcare condition. Examples include a drug, surgery, exercise or counselling. which has been demonstrated to lead to a statistically significant benefit) it’s important to pause and think. The small reduction in blood pressure was statistically significant, but is it clinically importantClinical significance is the practical importance of an effect (e.g. a reduction in symptoms); whether it has a real genuine, palpable, noticeable effect on daily life. It is not the same as statistical significance. For instance, showing that a drug lowered the heart rate by an average of 1 beat per minute would not be clinically significant, as it is unlikely to be a big enough effect to be important to patients and healthcare providers.?
Let’s consider the size of the reduction in blood pressure. For context, NHS Choices (2016) states:
- high blood pressure (systolic blood pressure/diastolic blood pressure) is considered to be 140/90 mmHg or higher
- ideal blood pressure is considered to be between 90/60mmHg and 120/80mmHg
So the observed reduction (1.8mmHg) is indeed modest, but the review authors suggest even this may be clinically important. Consuming flavanol-rich chocolate, they propose, might complement other treatmentSomething done with the aim of improving health or relieving suffering. For example, medicines, surgery, psychological and physical therapies, diet and exercise changes. options and could, at a populationThe group of people being studied. Populations may be defined by any characteristics e.g. where they live, age group, certain diseases. level, lead to fewer cardiovascular events. At this stage however, while this sounds plausible and promising, it remains speculative. This is largely owing to this next point…
The focus, and short-term nature, of the research leaves us with unanswered questions…
As the review authors allude to, what we’d really like to know is whether flavanol-rich chocolate could have beneficial effects not just on blood pressure but on the long-term consequences of high blood pressure. Could it reduce the incidenceThe number of new occurrences of something in a population over a particular period of time, e.g. the number of cases of a disease in a country over one year. of heart attacks or stroke for instance? Unfortunately, we just don’t have the dataData is the information collected through research. to say one way or the other.
This is because the included studies: a) are of short duration (most lasted between two and 12 weeks, with only one lasting 18 weeks) and b) did not look at important outcomesOutcomes are measures of health (for example quality of life, pain, blood sugar levels) that can be used to assess the effectiveness and safety of a treatment or other intervention (for example a drug, surgery, or exercise). In research, the outcomes considered most important are ‘primary outcomes’ and those considered less important are ‘secondary outcomes’. such as heart disease or stroke. So we’re left in the dark about long-term effects.
This speaks to an issue affecting research more generally: often studies do not focus on the outcomes we’re really interested in due to reasons of cost and convenience. Events such as heart attacks and strokes are relatively rare. As such, we’d need studies with very large sample sizes, and/or spanning many years, to capture enough of these events to be able to determine whether there is a meaningful difference in the rateThe speed or frequency of occurrence of an event, usually expressed with respect to time. For instance, a mortality rate might be the number of deaths per year, per 100,000 people. of their occurrence among those receiving intervention X vs. intervention Y. So instead research often focuses on ‘surrogate outcomes’, such as blood pressure, as a substitute for these important clinical outcomes or endpoints. While this is not unreasonable, we can’t just assume that beneficial effects on these surrogate outcomes will necessarily translate into real health benefits felt by patients.
Additionally, without long-term trialsClinical trials are research studies involving people who use healthcare services. They often compare a new or different treatment with the best treatment currently available. This is to test whether the new or different treatment is safe, effective and any better than what is currently used. No matter how promising a new treatment may appear during tests in a laboratory, it must go through clinical trials before its benefits and risks can really be known., we cannot be sure whether there are side effects associated with long-term daily consumption of cocoa products. In the short-term it seems people are happy to take their chocolatey medicine. Side effects, including “dislike of the trial product” and “digestive complaints” were reported by only 1% of the intervention participants. But longer term would individuals get sick of daily chocolate consumption (literally or figuratively)?
When the individual studies paint a varied picture…
Another issue is that the studies were highly statistically heterogeneous. This means that some studies suggested considerable benefits of cocoa on blood pressure, whereas others suggested no benefit.
The authors struggled to account fully for this variation, so much so that they had to rate the overall quality of the evidenceThe certainty (or quality) of evidence is the extent to which we can be confident that what the research tells us about a particular treatment effect is likely to be accurate. Concerns about factors such as bias can reduce the certainty of the evidence. Evidence may be of high certainty; moderate certainty; low certainty or very-low certainty. Cochrane has adopted the GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) for assessing certainty (or quality) of evidence. Find out more here: https://training.cochrane.org/grade-approach as ‘moderate’ rather than ‘high’. Nonetheless, they did identify a number of possible explanations. There were tentative indications that younger individuals (18-49 years) tended to experience greater reductions in blood pressure than older individuals (50-73 years) as did those who had higher blood pressure at baseline.
A couple of other explanations for the heterogeneity are worth highlighting too, as these are issues which crop up time and time again to undermine a variety of healthcare research…
Beware of poorly blinded studies
There was a trend towards participants experiencing greater benefits when they were not “blindedBlinding is the process of preventing those involved in a trial from knowing to which comparison group a particular participant belongs.” i.e. they were aware that they were consuming flavanol-rich chocolate or not. Further research is needed to confirm whether expectations (e.g. “dark chocolate is good for me”) can influence blood pressure. Nonetheless, this serves as an important reminder about the need for caution when drawing inferences from poorly blinded studies.
Beware of industry-sponsored research
It’s known that, across a spectrum of health research, industry-funded studies significantly more often favour the sponsors’ products than non-industry-sponsored studies (Lundh et al. 2017). This can lead us to overestimate the benefits, and underestimate the harms of interventions.
Indeed this may be at play here. The review authors explored what would happen when they removed six studies authored by individuals sponsored by industry. Excluding them “revealed a marked difference in results”, making the benefits of flavanol-rich chocolate seem less promising. So much so that the reduction in systolic blood pressure became non-significant. This is an important reminder to remain sceptical of claims made by those with conflicting interests.
What now?
So it seems that – as is so often the case with Cochrane evidence – the review’s take-home message is more nuanced than we might have hoped. Important questions remain to be answered, ideally in long-term, well-blinded, non-industry sponsored studies. I wonder whether there are any currently recruiting participants. I suppose I’m willing to tuck into some chocolate in the name of medical research. If you’d like to do the same (well, it might not be chocolate), visit Be Part of Research to find trials that are recruiting.
Selena Ryan-Vig has nothing to disclose.
References may be found here.
Excellent commentary and very helpful for the general public.