Ecology is not a dirty word

Literature reviews summarise existing knowledge and emerging paths for inquiry. They are essential for most grant applications, undergrad assignments and research projects.

But how do you actually do one? And what should you expect when you read a literature review?

Recently, I’m seeing a lot of papers submitted to or published in reputable journals that claim to be “comprehensive reviews” when they could not be further from the truth. Flawed review methods, no review methods, or a limited/localised observational study dressed up as a global review….

It’s got me wondering, what are the standards for a review paper? If you compare the author guidelines of most peer-reviewed journals, or the PhD guidelines between university websites, there are no standards. Editors and peer reviewers also seem to regularly confuse literature reviews with conceptual papers.

So are there even standards?

Once upon a time, literature reviews were a foundational research component embedded in a larger body of work. As the science disciplines have grown, knowledge has grown, and rigorous literature reviews are now so much more important to every discipline. They justify new research goals, they bring interconnected disciplines together, and they identify key knowledge gaps that we need to pay attention to.

Most supervisors will encourage students to design their research around a publishable lit review. I do this, and I think this is really important for a few reasons: (1) a PhD inherently is about contributing something new to the discipline, so this research must be on the scientific record (deposited at the university isn’t necessarily on the scientific record); (2) good literature reviews are a huge benefit to any discipline, and they are relatively easy to publish; and (3) publications help any post-PhD career path.

However, not all reviews are equal. Essentially, there are three basic categories:

• Narrative reviews are structured like an essay, and most journals do not require Methods or Results sections. They provide some insight into previous research and the current knowledge gaps, but are more like comments/opinions because authors can create a narrative from the research they cite.
• Systematic reviews (or comprehensive review, evidence synthesis etc.) must be presented with repeatable search methods, just like any other empirical scientific research. Just like a lab experiment, you are asking a question of the literature: search methods must be designed to identify what has been done before, and where are the knowledge gaps.
• Meta-analyses are a type of systematic review that also statistically analyse the results from all the studies collated in the review. They can be used to identify overall effects for a particular interaction, but to be rigorous, they need to use the right statistics on the right type and number of papers (e.g. this book by Koricheva, Gurevitch & Mengersen).

As a reader, editor and reviewer, I expect that a review paper should meet its own aims, but also the general aims of whatever type of review it claims to be. If the authors claim it is a “comprehensive” or “systematic” review, I expect to see much more than a random bunch of Google Scholar searches. If the authors present a narrative or opinion-based review on a particular topic to support future research, I expect a solid assessment of published literature and some knowledge gaps and recommendations outlined at the end of the paper.

Review papers can have a huge impact on the discipline through citations, and can also get a lot of damaging media attention. So I think it’s important for Editors and Reviewers to think about the longer term implications and consider a few questions when assessing manuscripts claiming to be reviews:

• Do the results/recommendations match the aims/methods outlined at the start of the paper?
• If the paper claims to be a systematic or comprehensive review, does it provide comprehensive and repeatable search methods? (Google Scholar is not this!)
• Have the authors outlined their aims and limitations clearly? It’s impossible to review every single text published on a topic, and authors must acknowledge this via parameters, search methods, justification, future questions etc.

© Manu Saunders 2020