If you believe the hype, peer review is flawed and corrupt, a broken system threatening to undermine the very foundations of academia…particularly science. From fake reviews to biased ones, one of the main arguments for ditching the system is the myth that reviewers can no longer be trusted to give a fair assessment of another scientist’s paper.
But the problem is not with peer review per se, it’s with our expectations of the system.
The academic review system as we know it today began around the 1960s. But the process of peer review has been around for centuries, formally and informally, from the Greek Agora to the first Royal Society meetings.
We need peer review because science (and scholarship generally) is a community endeavour.
Sure, there are some cheaters, but most scientists behave ethically. In my short career, most of the reviews I’ve received have been genuinely helpful and I’ve published better papers because of them.
One solution being argued for in op-eds and on social media is to pay reviewers for the time spent reviewing. There are lots of reasons why this won’t work.
Peer review is a community service, not a commodity. A community service, by definition, is unpaid work with good intentions to help individuals or the community as a whole. As an academic, peer review is a way of giving back to the community that helped you build your own career. Commoditising a voluntary community service is not the best way to change a system that already suffers from ethical issues.
We have a long human history of complicating things once money gets involved…look at why people don’t understand ecosystem services. Placing a value on something, monetary or not, immediately changes how we interact with that system. Non-monetary incentives (e.g. journal subscriptions) might help some people, but probably won’t change a reviewer’s mind if they don’t have time to accept a review request. See here for a thoughtful discussion of credit for peer review schemes. And, more to the point, how does paying a reviewer guarantee that he or she will do a better review?
Paying reviewers won’t reduce bias. Unfair assessment because of personal biases against the author is one of the biggest risks with peer review. Paying reviewers definitely won’t reduce this risk.
Double-blind review is often argued as a way to combat unfairness inherent in the current system (where reviewers know the author’s name but not the other way around). But this will never work, because many authors often need to cite their previous work to build a case for the current study.
Open review (where both author and reviewer know each other’s names) is a better option, as it increases ethical responsibility on both sides. It also works better than incentives to increase the quality of the review.
I’ve reviewed for open-access journal PeerJ, where they offer me the option to sign my review, and also an incentive (a free publication) if I finish the review in 10 days. I have never ‘won’ the free publication, because a 10 day turnaround has never fit my schedule. But I always choose to sign the review…and yes, I spend more time on it!
Paying reviewers won’t fix the novelty problem. One of the main issues with scholarly publishing is journal editors who think they’re publishing popular media instead of knowledge. Paying reviewers won’t stop editors unfairly rejecting a perfectly decent paper because it’s not sexy enough…or because it doesn’t include a meta-analysis (that actually happened to me).
It also won’t fix the issue of overburdened reviewers, an issue that is very hard to define. This fish biologist reviews 100 papers per year, but separate studies have shown that political scientists, meteorologists and nurses review an average 5-8 papers per year. Is this a disciplinary thing, or is it because ‘famous’ or more experienced academics get more review requests than less-experienced academics? This study of 4 ecology journals, found that 20- 52% of reviewers were doing fewer reviews/year than they should be. But it doesn’t clarify whether these reviewers are declining review requests or just not being asked in the first place.
So if paying reviewers doesn’t fix any of the real problems with the system, and just creates more ethical problems, then it’s not really a solution. Also see here and here for why a couple of editors quit Scientific Reports over payment-for-review schemes.
**************
I like peer review because it’s based on a mutual relationship. Sometimes there are glitches and broken links, and sometimes mutualisms turn into antagonisms. But is the whole concept of peer review flawed and unjust, just because some people don’t play nice?
There are very few mutual ‘give-and-take’ relationships where both parties experience both the give and the take in the same way. Bees and flowers are in a mutual relationship. But you don’t see flowers flying around looking for bees to feed on…
This two-way relationship is invaluable to my own research. Reviewing other people’s papers is one of the best motivations to improve my own writing and it also helps me keep abreast of how other researchers are approaching the same problems I’m working on. And I’m a pretty impatient person, so peer review is one of the best forms of instant gratification therapy!
If you’re new to peer review, here are some good resources:
Wiley’s Guide to reviewing a manuscript
British Ecological Society Guide to peer review (pdf)
A quick guide to writing a solid peer review
How to become good at peer review, via @JenniferRaff
© Manu Saunders 2016
Ecology is not a dirty word 
Nice post. I agree with you that paying reviewers is not a good idea (even though I am an economist – or maybe just because I am;-). But some kind of reward would be nice, because this is really hard work and normally you don’t get any credit for that (mainly because mostly you are anonymous and you’re employer does not care about how many papers you have reviewed – what counts is how many you’ve published + the journals’ IF). How to reward it? I have no idea… The idea of a free publication you’ve mentioned is great, but it only works for open-access journals where you have to actually pay a fee for submitting a paper (but maybe this is future?). I’ve heard that some journals publish a list of people who have reviewed for them in the previous year – this is also some way of giving credit.
But in the end, what rewarding might fix is people’s eagerness to write reviews. Not the quality of them. Dealing with biased interviews, for instance, should actually be the job of editors (too bad if they are biased themselves…). But, first, editors are just people themselves, and second, they often don’t have the time to carefully take a look at each review and each submission.
Finally, I don’t see a viable alternative to peer review, despite all its flaws.
LikeLiked by 1 person
Thanks for the great comment. I agree that acknowledgement is important, but not a ‘reward’ per se. I think open review does a good job of covering that, because you are immediately acknowledged for your contribution in building that paper, and your peers (i.e. potential future collaborators or employers) can see that you have been acknowledged. This can either be done via the journal, e.g. PeerJ, or via an external credit system, like Publons.
LikeLike
Well said Manu, I agree with you. Peer review is a service to our shared academic community and an important mutual relationship. We should look after this community of ours and treat it with respect.
Paying wont change problems, but do we even need to? I wonder if the people calling for payment have another agenda. The problems, from my perspective anyway, are small on balance.
Perhaps it is just my sector, but small acknowledgements of the effort taken are already quite common. Free journal access is a common one.
I don’t mind open review, but I especially like zielonygrzyb’s idea of journals publishing a list of the reviewers from the previous year. It could build institutional respect for the time taken reviewing.
LikeLiked by 1 person
Thanks Margi. Agree, acknowledgement makes a difference. And yes, some journals publish lists of reviewers at the end of the year, or even in each issue – as you say, it’s a citation that you can use in a resume or grant app to get institutional acknowledgement.
LikeLike
Great post🌟 You are so right, thanks for sharing this!👍And thank you again for underlining that ‘ecosystem services’ subject too.🙏
LikeLike
Great post Manu – we do indeed need more people willing to review https://simonleather.wordpress.com/2013/11/20/referees-your-journals-need-you/ and like you I don’t think paying people is the way forward.
LikeLiked by 1 person
Great post! I completely agree. It is true as Simon says that we need more people to step up to review (to keep up with the exponentially growing publication rate). But this in no way leads to the idea we should pay for peer review. And you outline beautifully why!
One thing we recently announced at GEB http://onlinelibrary.wiley.com/doi/10.1111/geb.12416/abstract (sorry for the shameless plug) is that not only will we acknowledge authors in the end of the year issue, but we will acknowledge a few people with oustanding reviewer and oustanding associate editor awards. We hope this will look good on a CV and provide some extra acknowledgement for really hard work that creates a race of the top in quality rather than incentivizing just quantity as pay per piece would do. This was a widely supported idea at the EUMacro2015 conference. It seems quite simple to do and to my mind aligns more closely with the kinds of relationships and motivations you describe so well.
In general, the data shows pretty clearly people become increasingly positive about peer review as their career stage advances. You could argue that is because advanced career people are the winners of the peer review system or just because they’re establishment. But I actually think it reflects genuine experience and knowledge. As a graduate student peer review seems so mysterious and threatening, so of course I will question it. But as my statistical sample gets large I come to know that it is not perfect (we all have our scars!) but on average it is quite good and really does improve the vast majority of manuscripts (and can be quite enjoyable and a positive contribution as a reviewer).
LikeLiked by 1 person
Thanks Brian! Yes, reviewer recognition is important, and things like publishing lists of names, open review, or reviewer ‘awards’ is definitely the way to do it.
I like your point about becoming more positive about it with experience – which is even more reason to maintain those mutual linkages!
LikeLike
Interesting arguments. But the one counter argument is this: publishing companies have higher profit margins than Apple, making billions from our papers and free labor. So it’s not quite an altruistic system!
LikeLike
Thanks Trevor. I agree that some (not all) publishing companies are making disproportionate profits from the publishing business. But I don’t think that, in and of itself, is a reason to pay reviewers. That’s like saying that film critics should get a share of the box office $$ for every movie they review.
The act of peer review is a community service, which is incommensurable with labour…trying to standardise all these different value scales is partly why the ecosystem services concept has become so blurry, as per my previous post that I linked to above.
LikeLike
I mostly agree about the need for peer review, but think there’s some room for creativity here.
What if someone could actually monetarily support themselves by being a full-time reviewer? That would make for a new career path: professional reviewer. Clearly that person’s reputation as a reviewer would matter, and one would suppose they might tend to write high-quality reviews with a relatively short turn-around. Perhaps a win-win for everyone.
And what about for-profit journals? I really dislike the idea of doing pro bono work for a journal that is profiteering off of my time. Shouldn’t for-profit journals pay reviewers, simply as a matter of professional ethics? I certainly don’t volunteer my time to help Google or Microsoft. Why should anyone subsidize the profits of journals?
LikeLiked by 1 person
Thanks Margaret, great comment. I have thought about the ‘professional reviewer’ scenario as an option, and I’ve seen a few people call for this system in op-eds. But I think it’s similar to teaching – a university lecturer is expected to be actively participating in research to contribute that contemporary experience to the next gen of scientists/scholars…which is why there are no FT ‘100% teaching, no research at all’ positions in unis. Peer review is a similar situation – I would like to know that people reviewing my papers are really up-to-date with my research topic, current methods etc. And I also wonder how hard it would be to find FT peer reviewers for every discipline, sub-discipline & specialist topic, especially if you think beyond the scientific disciplines. If you’re publishing a paper on an emerging specialist topic, that only a handful of people worldwide are working on, you would want one of those people to review it, not a hired reviewer who once worked in your general subject area!
And yes the ‘for profit’ journals is another interesting issue, with lots of grey areas! I think Trevor was raising a similar point in his comment above. Yes, it’s frustrating that some journals are making ridiculous profits from the publishing business. But I don’t see that in itself as a justification that peer reviewers should be paid. I think a community service was once clearly delineated from employment or commodities, before the business modus operandi became so predominant across all sectors of society (i.e. the ecosystem services issue, my previous post on this linked to above). If you look at peer review from the scholar’s side, it as an act of service that should happen independent of publishing company profits. If you look at it from the other direction, yes it seems unfair that publishing companies are making profits from voluntary service…but where do we draw the line? I think making the choice to not publish/review in those types of journals will do more for the scholarly community than adopting a pay-for-review scheme.
LikeLike
I think your points are interesting to think about, but not necessarily good justifications against paying reviewers.
I actually think that full-time reviewers might be more on top of the literature than current researchers — at least a lot of them. (And there are 100% FT teachers at Unis that don’t do research. Not many, but they do exist.) And I’m not arguing for a switch to only full-time reviewers. Rather, that if you start paying people, some people may migrate to that sort of roll. So I don’t think the ‘we couldn’t find enough full-time reviewers’ is really a good argument against. It doesn’t have to be all-or-nothing.
I’ll point out that editors for books don’t work pro-bono. Aren’t they also performing an act of service? I hire a caregiver to watch my child while I work as an ecologist. That woman is performing a very important act of service that I also perform (for free) when I am not working as an ecologist. But I pay her. So I really don’t buy the “reviewers are peforming an act of service argument” — they are acting on behalf of the publisher, not the authors.
To be a devil’s advocate, we could flip it around and ask: what is the justification for not paying reviewers? What would be the harm in paying reviewers? I can’t think of any other than (1) it’s not traditional, and so there will be pushback; and (2) it will cost (slightly) more to publish.
LikeLike
“So I don’t think the ‘we couldn’t find enough full-time reviewers’ is really a good argument against”
My main argument against FT reviewers was that I would like to know my reviewers are practising researchers, not that we couldn’t find enough to do the job. Perhaps a sabbatical system, where researchers take 6 months off to be paid by the journal to do reviews…but I don’t see how that would work, because reviewers should be someone who knows the topic/study system, not just whoever was available at the time.
I am not trying to argue against paying reviewers, but I think there are plenty of shades of grey to think about first. And I think acknowledgement (e.g. open review system or see Brian’s comment above) is a far better option than $. From a simplistic perspective, if anyone ‘should’ be paid in the current profit-publishing system, I would think it would be the authors, not the reviewers. But I think most people would agree that paying authors raises some ethical concerns, so why is paying reviewers any different?
And I don’t agree that reviewers are acting on behalf of the publisher – they are acting on behalf of the authors and the community of science (or whatever discipline) as a whole. The point of peer review is to provide feedback and suggestions to collaboratively build the body of current knowledge. The publisher is just the mode of printing & disseminating that knowledge. Many authors also seek pre-submission peer review from colleagues & mentors, but I don’t know anyone who would expect payment for this.
LikeLike
Good point. What are the ethical concerns? People who write books expect to get paid. (Maybe a different post for a different day…)
Call me cynical, but publishers engage reviewers to vet the work, so that the journal publishes only valid, important science. The journal cares about its reputation first and foremost. This is happily aligned with “building the body of knowledge,” but I don’t think any (especially for-profit) journal would publish something that “builds the body of knowledge,” but that also tarnishes the journal’s reputation. (But I can’t think of any examples where these two things aren’t aligned.) I really do see reviewers engaged by the journal to be working on behalf of the journal, not the authors. Reviewers engaged directly by the authors are working on behalf of the authors.
LikeLiked by 1 person
I think this is a really interesting discussion, and I guess it all comes down to how people view the act of peer review – are you reviewing as a ‘favour’ to give feedback to an author & the field generally? or are you being ‘hired’ to provide a proofreading/commenting service? Definitely room for plenty more blog posts on these issues!
LikeLike
Manu
I am sorry, but I think you are completely wrong. If you need speed and get more professional reviewer system you do need to pay reviewers. Scientists are professionals, not priest. You can spend hours reviewing a proposal or a paper. Try to ask a lawyer, a vet or a doc. to “have a look” in your dog and see how much he/she will charge you. We should move to a more professional reviewer process. None of your arguments are valid. Particularly now that you pay to pay to publish…
Peer review is a community service, not a commodity. Yes, it is. You spend millions of money from tax payers or yourself to do this job. If you want do it for free, it’s your choice.
Paying reviewers won’t reduce bias. Payed or not payed it’s the same argument. Circular.
It also won’t fix the issue of overburdened reviewers. Yes it will.
Paying reviewers won’t fix the novelty problem. Again, a circular idea. Novelty does not have anything to do with good reviewers.
I think your post is naive and does not bring any advance in the discussion to what is the role of scientists in the Society. Scientists are not apart from any other job.
LikeLike
Thanks for your opinion Mauro, I understand there are lots of different views on this issue. I recommend reading some of the great discussions in the comments above.
I agree we should move to a more professional review process, but I don’t think paying reviewers is the way to do this, for the reasons I outline above. I agree recognition of reviewers is critical, and this is something that can be addressed immediately by journals and editors, e.g. via promoting open review, or publishing names of reviewers…also see Brian’s great comment above. As I explain in the post, I see peer review as a community service, as a contribution to the community of science (or whatever discipline) as a whole.
I guess the debate comes down to how a person defines peer review. Is a community service the same as being hired for your services as a consultant professional or business manager? There are plenty of lawyers, vets and doctors who do pro bono work. It’s a matter of choice how much of this voluntary service we perform. Do we expect payment for other community services? Things like disaster relief, helping the homeless etc….most people would do these things voluntarily because they genuinely wish to see their community improve. Being asked to join a committee at your university to improve student experiences or staff interaction is a service to your university community…Should we be paid to attend meetings? We often review papers for colleagues/mentors before they even submit it to a journal – do we expect to be paid for this service?
LikeLike
Excellent post. Many authors do not understand that the articles written by them have no market value beyond the narrow scholarly community. Is there anyone else who is going to pay to read the papers published other than a limited number of scholars? No. So they should be happy that a publisher is making their work popular and does not charges them for the service. The journals also have massive infrastructure costs which cost a lot. Just like any corporation, they need to maintain a certain level of profit to maintain their operations which obviously means paying the staff and the managers very good salary to run the journal effectively.
The current business model is based to support the authors i.e. they are not charged to publish their work which makes sense as none of them is so famous that people are going to pay to read their papers published just like they pay for buying books written by famous authors
Several academics do not understand that it costs a lot to maintain a journal i.e. thousands of submissions, storing articles in server farms and keeping track of every paper and the reviewers and multiple rounds of revision.
For the authors, getting paid for a peer review would mean that peer review will become a commercial transaction and remove the community aspect i.e. one scholar helping another scholar become successful.
If an author regularly reviews for many journals, it means that their knowledge and expertise is high valued by editors and the scientific community which eventually translates into monetary benefits in the long run i.e. being promoted or receiving other types of recognition. Therefore no one loses any income at is always discussed in the current scenario.
Even in other professions, like music and photography, artists make money by developing an identity or a brand. They also provide lot of free service to develop a brand before they can cash on it. The process takes time and requires patience and being persistent. Even Youtubers do not charge the users of the content directly but make money through brand endorsements once they have a good audience base.
LikeLike