Literature reviews summarise existing knowledge and emerging paths for inquiry. They are essential for most grant applications, undergrad assignments and research projects.
But how do you actually do one? And what should you expect when you read a literature review?
Recently, I’m seeing a lot of papers submitted to or published in reputable journals that claim to be “comprehensive reviews” when they could not be further from the truth. Flawed review methods, no review methods, or a limited/localised observational study dressed up as a global review….
It’s got me wondering, what are the standards for a review paper? If you compare the author guidelines of most peer-reviewed journals, or the PhD guidelines between university websites, there are no standards. Editors and peer reviewers also seem to regularly confuse literature reviews with conceptual papers.
So are there even standards?
Once upon a time, literature reviews were a foundational research component embedded in a larger body of work. As the science disciplines have grown, knowledge has grown, and rigorous literature reviews are now so much more important to every discipline. They justify new research goals, they bring interconnected disciplines together, and they identify key knowledge gaps that we need to pay attention to.
Most supervisors will encourage students to design their research around a publishable lit review. I do this, and I think this is really important for a few reasons: (1) a PhD inherently is about contributing something new to the discipline, so this research must be on the scientific record (deposited at the university isn’t necessarily on the scientific record); (2) good literature reviews are a huge benefit to any discipline, and they are relatively easy to publish; and (3) publications help any post-PhD career path.
However, not all reviews are equal. Essentially, there are three basic categories:
- Narrative reviews are structured like an essay, and most journals do not require Methods or Results sections. They provide some insight into previous research and the current knowledge gaps, but are more like comments/opinions because authors can create a narrative from the research they cite.
- Systematic reviews (or comprehensive review, evidence synthesis etc.) must be presented with repeatable search methods, just like any other empirical scientific research. Just like a lab experiment, you are asking a question of the literature: search methods must be designed to identify what has been done before, and where are the knowledge gaps.
- Meta-analyses are a type of systematic review that also statistically analyse the results from all the studies collated in the review. They can be used to identify overall effects for a particular interaction, but to be rigorous, they need to use the right statistics on the right type and number of papers (e.g. this book by Koricheva, Gurevitch & Mengersen).
As a reader, editor and reviewer, I expect that a review paper should meet its own aims, but also the general aims of whatever type of review it claims to be. If the authors claim it is a “comprehensive” or “systematic” review, I expect to see much more than a random bunch of Google Scholar searches. If the authors present a narrative or opinion-based review on a particular topic to support future research, I expect a solid assessment of published literature and some knowledge gaps and recommendations outlined at the end of the paper.
Review papers can have a huge impact on the discipline through citations, and can also get a lot of damaging media attention. So I think it’s important for Editors and Reviewers to think about the longer term implications and consider a few questions when assessing manuscripts claiming to be reviews:
- Do the results/recommendations match the aims/methods outlined at the start of the paper?
- If the paper claims to be a systematic or comprehensive review, does it provide comprehensive and repeatable search methods? (Google Scholar is not this!)
- Have the authors outlined their aims and limitations clearly? It’s impossible to review every single text published on a topic, and authors must acknowledge this via parameters, search methods, justification, future questions etc.
© Manu Saunders 2020
I’ve heard an influential representative of the evidence synthesis research say that meta-analysis is not a type of literature review, but rather a specific method of analysis of the literature. Seems plausible, though a bit nitpicking;-) A nice post in any case!
LikeLiked by 1 person
Yes, I’ve heard this and I guess it’s a valid point, even if a bit nitpicky! But I included it here as I have seen some authors claim their review is a meta analysis when they didn’t do any analysis
LikeLiked by 1 person
Yeah, I’ve seen those, too… Or “systematic” reviews that don’t even provide a list of search terms…
Is there an ecological equivalent of the Cochrane Collaboration that is widely accepted as the gold standard systematic review/meta-analysis output of the medical literature? But also an equivalent set of standards for reporting (GRADE, PRISMA)?
There are some standards developing for certain types of systematic reviews or meta-analyses. But I don’t really think that is the answer to the bigger problem, i.e. we still need to value narrative reviews and methodological diversity. I think the issue is more with how we teach and understand the role of literature reviews for the discipline.