How not to do peer review


Academia may be unique among careers in its lack of standardised processes or training for so many of the common activities that are essential to being an academic. Instead, new researchers have to bumble blindfolded through the dark room of early career researchhood to work out how to literally do the academic parts of their job. Sometimes we’re lucky to have a supervisor, colleague, or mentor who might guide us to a door (but it may not always be the right door).

Publishing and peer review are part of this bumbling process. Publishing our research in peer-reviewed literature is a key part of our job description, to share knowledge with the discipline and beyond.

All researchers know how to do their research, and we’re trained on the basics of how to write a scientific paper from undergraduate years.

But actually getting your paper published in a reputable peer reviewed journal? There is very little training and very few standards across journals or disciplines. So much about the publishing process is a mystery to early career researchers new to the experience.

Many PhD students publish their first paper with their supervisors, so they may learn about the process this way (which is what supervisors are there for). But it might take them a few years to realise that the way their supervisor does it might not be the way everyone else does it, or even the way everyone else expects them to be doing it!

Cover letters for paper submissions are a good example of one of these publishing mysteries. It is a topic of regular disagreement on social media – some editors demand them, some say they never read them (I rarely do), some expect one succinct paragraph, others expect a page of overhyped marketing selling your work before they’ll even consider sending it to peer reviewers.

It’s totally fine for editors to have their preferences; the problem is that very few journals actually reveal this to authors in the submission guidelines. You could waste precious time crafting a mic drop letter that never gets read, or you could throw together a generic token attempt that annoys a letter-loving editor and ends up getting you a desk reject. How would you ever know?

If your paper is lucky enough to get reviewed, you are now faced with another of the publishing mysteries: the Response to Reviewers document.

What is the goal of the response document? Think of it as part of a conversation. It is your chance to respond to valid concerns and questions about clarity that the reviewers (i.e. readers) have raised.

Unfortunately, the bad minority of reviewers seem to get a disproportionate amount of airtime, which can give early career researchers the false impression that peer reviewers are all either horrible or stupid, and don’t need to be listened to.

But, if a genuine reviewer misinterprets something, chances are the author didn’t explain it very well. This matters, because any published paper can now potentially get much more public attention than it would have once upon a time. If a manuscript is easily misrepresented or misinterpreted, things could get messy once it’s published.

So here are five things NOT to do when you write your response document.

1. Do not start writing responses as soon as you read them. It always hurts reading pages of criticism of your work, especially if those criticisms are written in inappropriately condescending tones. Express emotions before putting fingers to keyboard. Read the comments through, get cranky, rant to friends and colleagues about those horrid reviewers, and then put them aside. Start writing when ready to think clearly and accept the flaws in the work.

2. Do not submit an unformatted ‘stream of consciousness’ document. As an editor, the most frustrating thing for me is to spend unnecessary time trying to decipher who said what in a response document. Format the response document and clearly distinguish comments from responses with obvious line breaks and paragraph markers, e.g. different coloured text, a numbering system, or the words ‘COMMENT’ and ‘RESPONSE’ in front of each paragraph.

3. Do not ignore comments or refuse to make any changes in response to a comment. Never respond to a reviewer’s suggestion or request for clarification with ‘yeah nah, we didn’t change anything because we like it how it is’. If the reviewer is legitimately wrong or misguided, clearly explain how they may have misunderstood, citing evidence. In most cases, it’s also good practice to edit the wording of the relevant text to improve clarity and avoid further misunderstanding. If a reviewer has misinterpreted something, they may not be a stupid curmudgeon, it’s more likely that the author didn’t express themself clearly.

4. Do not assume you know what the reviewer meant when they wrote the comment (and that they’re wrong). Some reviewer comments can sound rude or are written ambiguously (sometimes not intentionally, see below) and it can be unclear exactly what their argument or suggestion is. Never write a response telling a reviewer what they’re saying. As an author, if the meaning of a comment is unclear, say so politely, and provide some attempt at a revision that addresses what you think they might mean. Give them the opportunity to clarify their comments if needed, just as they have given the authors an opportunity to do so.

5. Do not expect the editor to dig through messy track changes to find your revised words and sections. Never respond to complex comments with a sentence like “Thanks, we made some relevant changes in the paragraph.” It takes more time for the editor to try and match up the original line numbers with the revised line numbers and work out which changes respond to which comments. Instead, explain exactly what changes were made, and provide exact line numbers from the revised version. (If the reviewer’s comment requires a simple edit, e.g. ‘the scientific name should be italicised’, it’s totally okay to just reply with ‘Done’ or ‘Corrected’).

Peer review is a community service, not a gladiator ring. As an editor, an author, and a fellow peer reivewer, I’ve seen too many reviewer reports that appear unecessarily combative.

So here are five things NOT to do as a reviewer.

1. Do not let your unconscious biases get in the way of a fair peer review.

2. Do not judge conceptual/opinion papers on the same standards as literature reviews or empirical papers. Conceptual papers are called different things at different journals, but basically they are about publishing ideas and insights, calls for new research, and expert scrutiny of popular issues. These are all valuable contributions to the academic literature. And they don’t require data analysis, let alone a meta-analysis, to be a valuable contribution to the discipline. Don’t be that reviewer.

3. Do not ask the authors to get a ‘native English’ speaker to read the paper. If the paper is written in English, but the grammar and syntax need some improvement to increase clarity, state this. Never, ever, suggest the authors should get ‘a native English speaker’ to revise the paper.

4. Do not write in active English voice as a norm. Writing in active voice as a reviewer can sound accusatory, rude or condescending, e.g. ‘You said this..’ ‘You did this…’. Consider how these sentences might sound to readers; especially those from a non-English primary language culture that has different approaches to grammar and syntax. Passive voice is unfairly vilified and has many appropriate uses in scientific writing.

5. Do not push pedantic and unreasonable requests for ‘moar data!’. This is probably especially true for papers that have already gone through at least one peer review revision. But it also applies to any paper – find out what category the paper has been submitted in, and judge it within those boundaries (see point above). Don’t ask for experiments that ‘could’ or ‘would’ have been done. Just judge it on whether the methods match the aims, whether the interpretation matches the results, and whether it’s ethical science. That’s all.

©Manu Saunders 2020

9 thoughts on “How not to do peer review

  1. Marco Mello November 17, 2020 / 7:57 PM

    “Academia may be unique among careers in its lack of standardised processes or training for so many of the common activities that are essential to being an academic.” Couldn’t agree more! That’s why I started by blog in 2012: to help fill this gap. 😉

    Liked by 2 people

  2. Marco Mello November 17, 2020 / 9:15 PM

    Now regarding all those subtleties of peer review you mentioned, imagine how we, from peripheral, non-anglophone countries feel about them? Nobody teaches us how to behave during peer review, and all norms have been established by Anglo-Saxon cultures. We don’t speak English as a mother tongue, but need to communicate in this foreign language and, in addition, follow foreign cultural norms. Our public education systems are much worse than those of central countries, so we start the academic marathon kilometers behind our privileged colleagues. Furthermore, our countries have been invaded and colonized, so from birth we learn to have low self-esteem and worship our colonizers. On top of that, colonialism built racist structures in our societies, so our self-esteem is even lower, when our skin is not white. I guarantee it’s not easy to pursue an academic career under these conditions and compete on an international level.

    Liked by 1 person

    • Manu Saunders November 17, 2020 / 9:40 PM

      Agree, and I understand how frustrating this process would be for you.Hope we can make positive change to the system

      Liked by 1 person

  3. Anarchanthropus crapuloideus November 18, 2020 / 1:27 AM

    Simply fantastic. May I ask your permission to translate this text into Spanish and upload it to one of my blogs for my master and phd students? Thanks in advance for your answer.

    Like

  4. ecoquant November 18, 2020 / 2:52 PM

    This is an excellent assessment of the state of affairs from someone, admittedly, who although technically trained, always observed it from the outside looking in.I have fortunately befriended many in academe, and have a son immersed in academe, who could provide some insights. This all depends to some degree upon the field in question.

    It’s a bit idyllic, but I have always thought the pattern in much of statistics where a breakthrough or controversial paper is accepted for publication, however tortuous the process, and then a discussion is published alongside, some of people invited, some who volunteered. Or a controversial paper gets published, which spurs a countervailing paper, and then the union of the authors get together, do some research, and publish what is effectively a rejoinder paper.

    I think these are scholarship at its best, but I know many cases where this just goes so wrong. Again, it depends upon the field. It’s easier, IMO, in Mathematics to get consensus. Perhaps, because of the similarity in chemistry, Statistics is easy, too,although personalities there do have an effect.

    This all gets heightened if the topic under discussion is,well, topical and connects into social policy of one kind or another. That’s a strange world. Presentation of evidence bends under the weight of motives and purpose.

    While the Renaissance model of peers educating novices which supports even modern academe has many good features, as the post says, it depends upon the quality of the mentor-peer. Bad mentor-peers can strangle a promising career. But there is no system for scoring them. And how is a novice able to properly distinguish between a poor choice of mentor and a good one? And what about competition for attention?

    Having worked for industry for 40+ years, let me say industry, despite the common and careless and casual deprecation of academe you hear there, has nothing on academe. It’s a wonder industry, particularly technology based industry, hasn’t more market failures than it does. That has a lot to do with its admirable PR apparatus. I think, though, to the degree academe wants to succeed, it needs to look at itself harder and come up with better measurement and assessment systems.

    I am a consumer of peer reviewed technical literature. I know that space. I know what’s useful and what’s not. I value it all. I need it. But I suspect the producers behind the closed door need to introspect and figure out a process whereby it could be better. If nothing else, the process is too slow, too reliant upon the good will of reviewers who do not receive compensation, even of in kind, and too opaque, even if that’s motivated by a blinding constraint.

    One way might be to pursue the open literature review, where people comment, but their comments are expected to meet minimum standards of citations substantiating critiques, and so on.

    I think, as well, from what I have seen that even participants in the robust process of publication do not demonstrate as deep a familiarity of their literature as they should and might. There’s a lot to be learned there, and many things we might think of being original actually were known long ago, even if in dark corners, and “new ideas” had precursors. I think it’s worth spending more time looking at those than trying to impress peers with the latest greatest insight.

    Conversations through published literature with long predecessors is a great experience.

    Like

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s