Teaching kids to assess goopy health claims

In this blog for our #UnderstandingEvidence series, Matt Oxman talks about the Informed Health Choices project, which helps people sort the wheat from the chaff of evidence about treatmentSomething done with the aim of improving health or relieving suffering. For example, medicines, surgery, psychological and physical therapies, diet and exercise changes. effects, starting with school children.

Page checked 4 April 2023

Evidently Cochrane makes evidence accessible, but so does Gwyneth Paltrow on her blog Goop. The difference is in the quality of evidenceThe certainty (or quality) of evidence is the extent to which we can be confident that what the research tells us about a particular treatment effect is likely to be accurate. Concerns about factors such as bias can reduce the certainty of the evidence. Evidence may be of high certainty; moderate certainty; low certainty or very-low certainty.  Cochrane has adopted the GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) for assessing certainty (or quality) of evidence. Find out more here: https://training.cochrane.org/grade-approach. However, if most people cannot tell apart a systematic reviewIn systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. from Paltrow’s anecdote about the benefits of “bee venom therapy”, where do you think they will go for their health advice? Cochrane certainly cannot compete in terms of celebrity.

“I was recently given ‘bee venom therapy’ for an old injury and it disappeared.” ~ Gwyneth Paltrow

Before the will to access high-quality evidence must come appreciation of what makes high-quality evidence. Without this understanding, people are prone to be misled by unreliable health claims and make uninformed choices, leading to waste and unnecessary suffering much more serious than bee stings.

This is the problem that my colleagues and I—mostly based in East Africa and Norway, but also the UK and other countries—are addressing with the Informed Health Choices project. We are developing and evaluating learning resources to help people separate the wheat from the chaff of evidence about treatment effects, starting in primary schools.

In 2014, we began developing the first resources, targeting Ugandan 10-year-olds and their parents, respectively. For the kids, we have made a combination textbook and comic book, with supplemental materials including an exercise book and a guide for teachers using the materials in their classrooms.

For the parents, we have produced a podcast, including a theme song.

The resources cover concepts such as the difference between personal experiences (anecdotal evidence) and a large fair comparison (randomised trialA trial in which the people taking part are randomly divided into groups. A group (the intervention group) is given the intervention being tested (for example a drug, surgery, or exercise) and compared with a group which does not receive the intervention (the control group).). They also include the concept of weighing benefits versus harms—for example, living like a celebrity versus being stung by bees.

We targeted children because they have the time and open minds to learn. Adults, on the other hand are, busy. Moreover, they often have to unlearn misconceptions and give up beliefs tied to their identities.

We targeted Uganda not because Ugandans face more unreliable claims than Norwegians or Brits, and are less able to assess them, but because the less you have, the less you can afford to make uninformed health choices. In fact, science teachers at an international school in Norway have already made using our materials a permanent part of their students’ science education.

Recently, we tested the resources in two randomisedRandomization is the process of randomly dividing into groups the people taking part in a trial. One group (the intervention group) will be given the intervention being tested (for example a drug, surgery, or exercise) and compared with a group which does not receive the intervention (the control group).   trialsClinical trials are research studies involving people who use healthcare services. They often compare a new or different treatment with the best treatment currently available. This is to test whether the new or different treatment is safe, effective and any better than what is currently used. No matter how promising a new treatment may appear during tests in a laboratory, it must go through clinical trials before its benefits and risks can really be known., in Central Uganda. About 15 000 children participated in the trial of the primary school resources. Soon, we will find out whether the resources showed an effect on participants understanding and abilities to apply the concepts, measured by their answers to multiple-choice, scenario-based questions.

While we wait for the results, we have already started planning improvements to the resources and the development of new ones. Many of my colleagues have one foot in the project and another in Cochrane, and we are looking at ways of connecting the resources directly to Cochrane reviewsCochrane Reviews are systematic reviews. In systematic reviews we search for and summarize studies that answer a specific research question (e.g. is paracetamol effective and safe for treating back pain?). The studies are identified, assessed, and summarized by using a systematic and predefined approach. They inform recommendations for healthcare and research. in the form of interactive Summary of Findings tables. For more information about what we have done and plan to do, visit our website, where you can download the resources for free.

Finally, a warning: when I wrote about the project in a guest blog post for HealthNewsReview.com, I also used a celebrity claim as the main example. That celebrity was subsequently elected President of the most powerful country on Earth. Goop, by the way, means “a stupid person”. Paltrow’s blog appears to have been named without an Intention to be Ironic (ITI) analysis.

Matt Oxman has nothing to disclose.

Featured Image:  Children in Uganda show off the original prototype of what would become The Health Choices Book: Learning to think carefully about treatments.  (Matt Oxman/Informed Health Choices project).