Confessions of a rookie consumer peer reviewer

Karen Morley blogs and draws her experiences as a daunted and delighted volunteer peer reviewer for Cochrane Common Mental Disorders.

Page last checked 19 April 2023

Would I, as a new consumer volunteer for Cochrane Common Mental Disorders, have jumped in at the deep end and attempted a peer review if I hadn’t been asked? I don’t think so. But now I’m here, unsteadily dogpaddling, with two protocols and a systematic review under my belt, I thought I’d share some reflections and questions with you.

First, the Fear

First came the Fear. I felt quite tiny in the huge, complex Cochrane organisation. And ‘peer reviewing’ sounded so imposing.  I know there are consumers who are themselves researchers or health professionals, but how could I be considered a ‘peer’?

How can I be a peer reviewer? I haven't even got a lanyard!

Jess Hendon, the Managing Editor, was warm, kind and welcoming and assured me my lived experience was what counted. But did I have the knowledge, understanding and skills for the task? I had some experience of close reading and critical evaluation from my years as a secondary English teacher, and a basic understanding of systematic reviews from contributing to some Cochrane consumer learning resources that are currently in development, but would that be enough? I was afraid of being judged and afraid what I had to say might not be useful. Well, it was too late for dithering; a deadline was approaching: I took the plunge.

Starting out

There was a checklist that gave some focus and structure. There was also some linked online support for peer reviewers. At the moment this is quite brief and packed with detail and assumes some prior knowledge in the reader (for instance, ‘interventions’, ‘outcomes’, ‘comparator’ and ‘settings’ have specialist meanings that might be unfamiliar to a complete newcomer) and the language is rather formal. I was lucky to have already explored the subject a little and I wondered what it must feel like to be starting from scratch.

Despite the checklist I was still full of doubt. What exactly was expected of me? How should I express myself and in how much detail? A worked example would have been helpful. When I later saw the feedback from the authors on all the reviewers’ comments I had a much clearer idea of the process and the kind of contributions that were acceptable. I felt strangely detached and lonely as I looked at my screen. It would have been nice to have the luxury of a mentor, or some buddies to bounce ideas off, or maybe to have had a group practice run. It would also be interesting to hear an author’s view. (I’ve since had the pleasure of meeting Lindsay Robertson, which I found reassuring – thanks, Lindsay!) Last year I took part in a James Lind Alliance Priority Setting Partnership and I found it was illuminating to hear the comments of other patients, carers and clinicians: we have lots to learn from each other.

The protocol I looked at first would certainly be challenging for a consumer to read. I had to check my own understanding of ‘monoamine neurotransmission’, for instance. (That was quite a ride!)

Peer reviewing can feel challenging

I’d be interested to know whether, when consumer reviewers write about language and style, the same issues crop up frequently, and whether it might be desirable or feasible to collect them or to have some sort of glossary. Making suggestions for explanations or alternative expressions throws up a host of knotty problems. Who is the reader I’m writing for? What level am I pitching at? Which consumers read protocols – and systematic reviews – and why? How much can something be simplified without losing important nuances or even completely misrepresenting its meaning? Is there in fact a conflict between several different audiences and purposes? Which ones do the authors have at the forefront of their minds? Why doesn’t a protocol have some kind of Plain Language Summary?

Responsibility and representation

I suppose I felt the most confident about working on the language and less so about commenting on things like objectives and the criteria for considering studies. I knew there would be other contributors but I had a strong sense of responsibility: a real desire to write something that would actually be helpful to the authors. I have enormous respect for them and the huge, intricate, demanding task they have carried out. I don’t want to waste their time.

But now to the heart of the matter: representation. ‘Does the background address the hopes and concerns of people considering the treatment?’ asks one of the prompts. My task is to put myself into the shoes of the most important end users of systematic reviews and protocols: the consumers. This is the biggest responsibility of all. I can do my best but I’m aware that, alone, I’m not up to the task. I am one old, white, middle class woman, with one experience of one manifestation of a wide-ranging mental health condition: one tiny part of a huge jigsaw. Cochrane needs many consumer peer reviewers to represent us in all our diversity. And not just reviewers. I appreciate my opportunity to make changes to the review before it’s published, but this is the end of a very long journey. Have other consumers been involved before I see the review – in setting priorities or determining the research question or outcomes, say? How would I know? Shouldn’t it be mentioned in the review somewhere?

You are needed!

So, consumers, you are needed! If you feel the stirrings of an inclination to get involved, please have a go at something – anything. You might feel reticent, but come and help lead the way until it becomes commonplace for us to be included and to shape research around all our hopes and concerns.

I am sometimes shocked when I think how comparatively young the modern evidence based medicine movement is. Similarly, it’s easy to forget that the meaningful inclusion of consumers is really only beginning. I am delighted that Cochrane’s revised 2018 Peer Review Policy means Cochrane Review Groups must now aim to have at least one consumer peer reviewer for each Cochrane Review. (If there were more of us, fellow consumers, it could be more than an aim!) I’m delighted and daunted in equal measure to be one of those peer reviewers, and enthusiastic about CCMD’s plans to take this forward. It has been a tremendous pleasure to meet Rachel Churchill, Jess Hendon, Lindsay Robertson and Noortje Uphoff (well, I popped into her office!). The reception they have given me has been affirming and encouraging. I feel the burden, but I can see a great opportunity.

So what’s next?

I know I ask a lot of questions. I’ve only just started.

Join in the conversation on Twitter with @KMorley9 @Cochrane_CCMD @CochraneUK or leave a comment on the blog. You might also like to look at getting involved in Cochrane through Cochrane Consumers and Cochrane Crowd.

Karen’s blog and cartoons first appeared on Cochrane Common Mental Disorders website where you can also find out more about their work.

Confessions of a rookie consumer peer reviewer by Karen Morley

is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International

3 Comments on this post

  1. Dear Karen,
    Great blog, and I hope the volunteer peer reviewers join forces and become the mighty power driving Cochrane to better best evidence based research for consumers to read.
    Friendly access and readability two goals that always need that question to be asked,
    How can we make Cochrane better still?.

    I am a volunteer peer reviewer for the breast cancer clinical trial group, based out of The university of Sydney, our managing editor is Melina Willson. Melina has welcomed me with great support and made my journey into peer reviewing very well supported.
    I feel a wonderful sense of being able to have a say, and that freedom of speech is very important and well done Cochrane for nurturing that.
    Look forward to the next rookie volunteer peer reviewer blog.
    When every kitchen first aide kit has Cochrane in it, I will be happy, job done!.
    Best wishes,
    Philippa Hobbs🌎🇳🇿📚📓📖

    Philippa Hobbs / Reply
  2. Thank you so much, Polly, for your lovely, sensitive and encouraging comment. I really appreciate it. It’s good to know someone has shared my experience about the different elements of the role. I’m hoping to attend the Colloquium next year with Cochrane Common Mental Disorders – I’m so pleased you think it would be affirming. Thank you again for taking the time and trouble to respond to and support a newcomer – and other consumers who might be thinking of joining us.

    Karen Morley / Reply
  3. Thank you, Karen, for your eloquent and relevant comments about being a peer reviewer. I worked with the Pregnancy and Childbirth group for over 10 years, and was never completely sure I really knew what I was doing! For me, the main confusion of being a lay reviewer was which role was more important: “editor of language” (I too had been a teacher so was very sensitive to writing that was unclear) or “representative of the interests of the consumer”, i.e. ensuring the questions and issues raised were important to users. I think the lack of feedback was the most frustrating part of the job, never being sure if my comments were useful to anyone. In general, the Cochrane Community has made a lot of progress toward realizing and addressing these problems, but it is still a lonely task. I recommend attending a Colloquium if you have time. Applying for a stipend is not strenuous and the experience of meeting people face-to-face very affirming. Good luck for the future. You are just the kind of person Cochrane needs.

    Polly Griffith / Reply

Leave a Reply

Your email address will not be published. Required fields are marked *