week 6 Discussion 1

Editorial

Ten Simple Rules for Writing a Literature Review Marco Pautasso

1,2 *

1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France, 2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

Literature reviews are in great demand

in most scientific fields. Their need stems

from the ever-increasing output of scien-

tific publications [1]. For example, com-

pared to 1991, in 2008 three, eight, and

forty times more papers were indexed in

Web of Science on malaria, obesity, and

biodiversity, respectively [2]. Given such

mountains of papers, scientists cannot be

expected to examine in detail every single

new paper relevant to their interests [3].

Thus, it is both advantageous and neces-

sary to rely on regular summaries of the

recent literature. Although recognition for

scientists mainly comes from primary

research, timely literature reviews can lead

to new synthetic insights and are often

widely read [4]. For such summaries to be

useful, however, they need to be compiled

in a professional way [5].

When starting from scratch, reviewing

the literature can require a titanic amount

of work. That is why researchers who have

spent their career working on a certain

research issue are in a perfect position to

review that literature. Some graduate

schools are now offering courses in

reviewing the literature, given that most

research students start their project by

producing an overview of what has

already been done on their research issue

[6]. However, it is likely that most

scientists have not thought in detail about

how to approach and carry out a literature

review.

Reviewing the literature requires the

ability to juggle multiple tasks, from

finding and evaluating relevant material

to synthesising information from various

sources, from critical thinking to para-

phrasing, evaluating, and citation skills [7].

In this contribution, I share ten simple

rules I learned working on about 25

literature reviews as a PhD and postdoc-

toral student. Ideas and insights also come

from discussions with coauthors and

colleagues, as well as feedback from

reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review?

There are so many issues in contemporary

science that you could spend a lifetime of

attending conferences and reading the

literature just pondering what to review.

On the one hand, if you take several years

to choose, several other people may have

had the same idea in the meantime. On

the other hand, only a well-considered

topic is likely to lead to a brilliant literature

review [8]. The topic must at least be:

(i) interesting to you (ideally, you

should have come across a series of

recent papers related to your line of

work that call for a critical summa-

ry),

(ii) an important aspect of the field (so

that many readers will be interested

in the review and there will be

enough material to write it), and

(iii) a well-defined issue (otherwise you

could potentially include thousands

of publications, which would make

the review unhelpful).

Ideas for potential reviews may come

from papers providing lists of key research

questions to be answered [9], but also

from serendipitous moments during des-

ultory reading and discussions. In addition

to choosing your topic, you should also

select a target audience. In many cases, the

topic (e.g., web services in computational

biology) will automatically define an

audience (e.g., computational biologists),

but that same topic may also be of interest

to neighbouring fields (e.g., computer

science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and

audience, start by checking the literature

and downloading relevant papers. Five

pieces of advice here:

(i) keep track of the search items you

use (so that your search can be

replicated [10]),

(ii) keep a list of papers whose pdfs you

cannot access immediately (so as to

retrieve them later with alternative

strategies),

(iii) use a paper management system

(e.g., Mendeley, Papers, Qiqqa,

Sente),

(iv) define early in the process some

criteria for exclusion of irrelevant

papers (these criteria can then be

described in the review to help

define its scope), and

(v) do not just look for research papers

in the area you wish to review, but

also seek previous reviews.

The chances are high that someone will

already have published a literature review

(Figure 1), if not exactly on the issue you

are planning to tackle, at least on a related

topic. If there are already a few or several

reviews of the literature on your issue, my

advice is not to give up, but to carry on

with your own literature review,

(i) discussing in your review the ap-

proaches, limitations, and conclu-

sions of past reviews,

(ii) trying to find a new angle that has

not been covered adequately in the

previous reviews, and

(iii) incorporating new material that has

inevitably accumulated since their

appearance.

Citation: Pautasso M (2013) Ten Simple Rules for Writing a Literature Review. PLoS Comput Biol 9(7): e1003149. doi:10.1371/journal.pcbi.1003149

Editor: Philip E. Bourne, University of California San Diego, United States of America

Published July 18, 2013

Copyright: � 2013 Marco Pautasso. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

Competing Interests: The author has declared that no competing interests exist.

* E-mail: [email protected]

PLOS Computational Biology | www.ploscompbiol.org 1 July 2013 | Volume 9 | Issue 7 | e1003149

When searching the literature for per-

tinent papers and reviews, the usual rules

apply:

(i) be thorough,

(ii) use different keywords and database

sources (e.g., DBLP, Google Schol-

ar, ISI Proceedings, JSTOR Search,

Medline, Scopus, Web of Science),

and

(iii) look at who has cited past relevant

papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only

afterwards start writing the review, you

will need a very good memory to remem-

ber who wrote what, and what your

impressions and associations were while

reading each single paper. My advice is,

while reading, to start writing down

interesting pieces of information, insights

about how to organize the review, and

thoughts on what to write. This way, by

the time you have read the literature you

selected, you will already have a rough

draft of the review.

Of course, this draft will still need much

rewriting, restructuring, and rethinking to

obtain a text with a coherent argument

[11], but you will have avoided the danger

posed by staring at a blank document. Be

careful when taking notes to use quotation

marks if you are provisionally copying

verbatim from the literature. It is advisable

then to reformulate such quotes with your

own words in the final draft. It is

important to be careful in noting the

references already at this stage, so as to

avoid misattributions. Using referencing

software from the very beginning of your

endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading

the literature, you will have a rough idea

of the amount of material available for the

review. This is probably a good time to

decide whether to go for a mini- or a full

review. Some journals are now favouring

the publication of rather short reviews

focusing on the last few years, with a limit

on the number of words and citations. A

mini-review is not necessarily a minor

review: it may well attract more attention

from busy readers, although it will inevi-

tably simplify some issues and leave out

some relevant material due to space

limitations. A full review will have the

advantage of more freedom to cover in

detail the complexities of a particular

scientific development, but may then be

left in the pile of the very important papers

‘‘to be read’’ by readers with little time to

spare for major monographs.

There is probably a continuum between

mini- and full reviews. The same point

applies to the dichotomy of descriptive vs.

integrative reviews. While descriptive re-

views focus on the methodology, findings,

and interpretation of each reviewed study,

integrative reviews attempt to find com-

mon ideas and concepts from the reviewed

material [12]. A similar distinction exists

between narrative and systematic reviews:

while narrative reviews are qualitative,

systematic reviews attempt to test a

hypothesis based on the published

evidence, which is gathered using a

Figure 1. A conceptual diagram of the need for different types of literature reviews depending on the amount of published research papers and literature reviews. The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33]. doi:10.1371/journal.pcbi.1003149.g001

PLOS Computational Biology | www.ploscompbiol.org 2 July 2013 | Volume 9 | Issue 7 | e1003149

predefined protocol to reduce bias [13,14].

When systematic reviews analyse quanti-

tative results in a quantitative way, they

become meta-analyses. The choice be-

tween different review types will have to be

made on a case-by-case basis, depending

not just on the nature of the material

found and the preferences of the target

journal(s), but also on the time available to

write the review and the number of

coauthors [15].

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or

a full review, it is good advice to keep it

focused [16,17]. Including material just for

the sake of it can easily lead to reviews that

are trying to do too many things at once.

The need to keep a review focused can be

problematic for interdisciplinary reviews,

where the aim is to bridge the gap between

fields [18]. If you are writing a review on,

for example, how epidemiological ap-

proaches are used in modelling the spread

of ideas, you may be inclined to include

material from both parent fields, epidemi-

ology and the study of cultural diffusion.

This may be necessary to some extent, but

in this case a focused review would only

deal in detail with those studies at the

interface between epidemiology and the

spread of ideas.

While focus is an important feature of a

successful review, this requirement has to

be balanced with the need to make the

review relevant to a broad audience. This

square may be circled by discussing the

wider implications of the reviewed topic

for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp

collecting. A good review does not just

summarize the literature, but discusses it

critically, identifies methodological prob-

lems, and points out research gaps [19].

After having read a review of the litera-

ture, a reader should have a rough idea of:

(i) the major achievements in the

reviewed field,

(ii) the main areas of debate, and

(iii) the outstanding research questions.

It is challenging to achieve a successful

review on all these fronts. A solution can

be to involve a set of complementary

coauthors: some people are excellent at

mapping what has been achieved, some

others are very good at identifying dark

clouds on the horizon, and some have

instead a knack at predicting where

solutions are going to come from. If your

journal club has exactly this sort of team,

then you should definitely write a review

of the literature! In addition to critical

thinking, a literature review needs consis-

tency, for example in the choice of passive

vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review

has a number of telling features: it is worth

the reader’s time, timely, systematic, well

written, focused, and critical. It also needs

a good structure. With reviews, the usual

subdivision of research papers into intro-

duction, methods, results, and discussion

does not work or is rarely used. However,

a general introduction of the context and,

toward the end, a recapitulation of the

main points covered and take-home mes-

sages make sense also in the case of

reviews. For systematic reviews, there is a

trend towards including information about

how the literature was searched (database,

keywords, time limits) [20].

How can you organize the flow of the

main body of the review so that the reader

will be drawn into and guided through it?

It is generally helpful to draw a conceptual

scheme of the review, e.g., with mind-

mapping techniques. Such diagrams can

help recognize a logical way to order and

link the various sections of a review [21].

This is the case not just at the writing

stage, but also for readers if the diagram is

included in the review as a figure. A

careful selection of diagrams and figures

relevant to the reviewed topic can be very

helpful to structure the text too [22].

Rule 8: Make Use of Feedback

Reviews of the literature are normally

peer-reviewed in the same way as research

papers, and rightly so [23]. As a rule,

incorporating feedback from reviewers

greatly helps improve a review draft.

Having read the review with a fresh mind,

reviewers may spot inaccuracies, inconsis-

tencies, and ambiguities that had not been

noticed by the writers due to rereading the

typescript too many times. It is however

advisable to reread the draft one more

time before submission, as a last-minute

correction of typos, leaps, and muddled

sentences may enable the reviewers to

focus on providing advice on the content

rather than the form.

Feedback is vital to writing a good

review, and should be sought from a

variety of colleagues, so as to obtain a

diversity of views on the draft. This may

lead in some cases to conflicting views on

the merits of the paper, and on how to

improve it, but such a situation is better

than the absence of feedback. A diversity

of feedback perspectives on a literature

review can help identify where the con-

sensus view stands in the landscape of the

current scientific understanding of an issue

[24].

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the litera-

ture will have published studies relevant to

the review they are writing. This could

create a conflict of interest: how can

reviewers report objectively on their own

work [25]? Some scientists may be overly

enthusiastic about what they have pub-

lished, and thus risk giving too much

importance to their own findings in the

review. However, bias could also occur in

the other direction: some scientists may be

unduly dismissive of their own achieve-

ments, so that they will tend to downplay

their contribution (if any) to a field when

reviewing it.

In general, a review of the literature

should neither be a public relations

brochure nor an exercise in competitive

self-denial. If a reviewer is up to the job of

producing a well-organized and methodi-

cal review, which flows well and provides a

service to the readership, then it should be

possible to be objective in reviewing one’s

own relevant findings. In reviews written

by multiple authors, this may be achieved

by assigning the review of the results of a

coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in

the publication of scientific papers, today’s

reviews of the literature need awareness

not just of the overall direction and

achievements of a field of inquiry, but

also of the latest studies, so as not to

become out-of-date before they have been

published. Ideally, a literature review

should not identify as a major research

gap an issue that has just been addressed

in a series of papers in press (the same

applies, of course, to older, overlooked

studies (‘‘sleeping beauties’’ [26])). This

implies that literature reviewers would do

well to keep an eye on electronic lists of

papers in press, given that it can take

months before these appear in scientific

databases. Some reviews declare that they

PLOS Computational Biology | www.ploscompbiol.org 3 July 2013 | Volume 9 | Issue 7 | e1003149

have scanned the literature up to a certain

point in time, but given that peer review

can be a rather lengthy process, a full

search for newly appeared literature at the

revision stage may be worthwhile. Assess-

ing the contribution of papers that have

just appeared is particularly challenging,

because there is little perspective with

which to gauge their significance and

impact on further research and society.

Inevitably, new papers on the reviewed

topic (including independently written

literature reviews) will appear from all

quarters after the review has been pub-

lished, so that there may soon be the need

for an updated review. But this is the

nature of science [27–32]. I wish every-

body good luck with writing a review of

the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-

Schmutz, T. Döring, D. Fontaneto, M. Garbe-

lotto, O. Holdenrieder, M. Jeger, D. Lonsdale,

A. MacLeod, P. Mills, M. Moslonka-Lefebvre,

G. Stancanelli, P. Weisberg, and X. Xu for

insights and discussions, and to P. Bourne, T.

Matoni, and D. Smith for helpful comments on

a previous draft.

References

1. Rapple C (2011) The role of the critical review

article in alleviating information overload. Annual

Reviews White Paper. Available: http://www.

annualreviews.org/userimages/ContentEditor/

1300384004941/Annual_Reviews_WhitePaper_

Web_2011.pdf. Accessed May 2013.

2. Pautasso M (2010) Worsening file-drawer prob-

lem in the abstracts of natural, medical and social

science databases. Scientometrics 85: 193–202.

doi:10.1007/s11192-010-0233-5.

3. Erren TC, Cullen P, Erren M (2009) How to surf

today’s information tsunami: on the craft of

effective reading. Med Hypotheses 73: 278–279.

doi:10.1016/j.mehy.2009.05.002.

4. Hampton SE, Parker JN (2011) Collaboration

and productivity in scientific synthesis. Bioscience

61: 900–910. doi:10.1525/bio.2011.61.11.9.

5. Ketcham CM, Crawford JM (2007) The impact

of review articles. Lab Invest 87: 1174–1185.

doi:10.1038/labinvest.3700688.

6. Boote DN, Beile P (2005) Scholars before

researchers: on the centrality of the dissertation

literature review in research preparation. Educ

R e s 3 4 : 3 – 1 5 . d o i : 1 0 . 3 1 0 2 /

0013189X034006003.

7. Budgen D, Brereton P (2006) Performing system-

atic literature reviews in software engineering.

Proc 28th Int Conf Software Engineering, ACM

N e w Y o r k , N Y , U S A , p p . 1 0 5 1 – 1 0 5 2 .

doi:10.1145/1134285.1134500.

8. Maier HR (2013) What constitutes a good

literature review and why does its quality matter?

Environ Model Softw 43: 3–4. doi:10.1016/

j.envsoft.2013.02.004.

9. Sutherland WJ, Fleishman E, Mascia MB, Pretty

J, Rudd MA (2011) Methods for collaboratively

identifying research priorities and emerging issues

in science and policy. Methods Ecol Evol 2: 238–

247. doi:10.1111/j.2041-210X.2010.00083.x.

10. Maggio LA, Tannery NH, Kanter SL (2011)

Reproducibility of literature search reporting in

medical education reviews. Acad Med 86: 1049–

1054. doi:10.1097/ACM.0b013e31822221e7.

11. Torraco RJ (2005) Writing integrative literature

reviews: guidelines and examples. Human Res De v e lo p R e v 4: 3 5 6– 36 7 . d oi : 1 0. 1 1 77 /

1534484305278283.

12. Khoo CSG, Na JC, Jaidka K (2011) Analysis of the macro-level discourse structure of literature

reviews. Online Info Rev 35: 255–271. doi:10.1108/14684521111128032.

13. Rosenfeld RM (1996) How to systematically review the medical literature. Otolaryngol Head

Neck Surg 115: 53–63. doi:10.1016/S0194-

5998(96)70137-7. 14. Cook DA, West CP (2012) Conducting systematic

reviews in medical education: a stepwise ap- proach. Med Educ 46: 943–952. doi:10.1111/

j.1365-2923.2012.04328.x.

15. Dijkers M, The Task Force on Systematic Reviews and Guidelines (2009) The value of

‘‘traditional’’ reviews in the era of systematic reviewing. Am J Phys Med Rehabil 88: 423–430.

doi:10.1097/PHM.0b013e31819c59c6.

16. Eco U (1977) Come si fa una tesi di laurea. Milan: Bompiani.

17. Hart C (1998) Doing a literature review: releasing the social science research imagination. London:

SAGE. 18. Wagner CS, Roessner JD, Bobb K, Klein JT,

Boyack KW, et al. (2011) Approaches to under-

standing and measuring interdisciplinary scientific research (IDR): a review of the literature.

J I n f o r m e t r 5 : 1 4 – 2 6 . d o i : 1 0 . 1 0 1 6 / j.joi.2010.06.004.

19. Carnwell R, Daly W (2001) Strategies for the

construction of a critical review of the literature. Nurse Educ Pract 1: 57–63. doi:10.1054/

nepr.2001.0008. 20. Roberts PD, Stewart GB, Pullin AS (2006) Are

review articles a reliable source of evidence to support conservation and environmental man-

agement? A comparison with medicine. Biol

Conserv 132: 409–423. doi:10.1016/j.bio- con.2006.04.034.

21. Ridley D (2008) The literature review: a step-by- step guide for students. London: SAGE.

22. Kelleher C, Wagener T (2011) Ten guidelines for

effective data visualization in scientific publica-

tions. Environ Model Softw 26: 822–827.

doi:10.1016/j.envsoft.2010.12.006.

23. Oxman AD, Guyatt GH (1988) Guidelines for

reading literature reviews. CMAJ 138: 697–703.

24. May RM (2011) Science as organized scepticism.

Philos Trans A Math Phys Eng Sci 369: 4685–

4689. doi:10.1098/rsta.2011.0177.

25. Logan DW, Sandal M, Gardner PP, Manske M,

Bateman A (2010) Ten simple rules for editing

Wikipedia. PLoS Comput Biol 6: e1000941.

doi:10.1371/journal.pcbi.1000941.

26. van Raan AFJ (2004) Sleeping beauties in science.

Scientometrics 59: 467–472. doi:10.1023/

B:SCIE.0000018543.82441.f1.

27. Rosenberg D (2003) Early modern information

overload. J Hist Ideas 64: 1–9. doi:10.1353/

jhi.2003.0017.

28. Bastian H, Glasziou P, Chalmers I (2010)

Seventy-five trials and eleven systematic reviews

a day: how will we ever keep up? PLoS Med 7:

e1000326. doi:10.1371/journal.pmed.1000326.

29. Bertamini M, Munafò MR (2012) Bite-size

science and its undesired side effects. Perspect

P s y c h o l S c i 7 : 6 7 – 7 1 . d o i : 1 0 . 1 1 7 7 /

1745691611429353.

30. Pautasso M (2012) Publication growth in biolog-

ical sub-fields: patterns, predictability and sus-

tai nabi lity. S ustai nabi lity 4: 3 234– 3247 .

doi:10.3390/su4123234.

31. Michels C, Schmoch U (2013) Impact of biblio-

metric studies on the publication behaviour of

authors. Scientometrics. doi:10.1007/s11192-

013-1015-7. In press.

32. Tsafnat G, Dunn A, Glasziou P, Coiera E (2013)

The automation of systematic reviews. BMJ 346:

f139. doi:10.1136/bmj.f139.

33. Pautasso M, Döring TF, Garbelotto M, Pellis L,

Jeger MJ (2012) Impacts of climate change on

plant diseases - opinions and trends. Eur J Plant

Pathol 133: 295–313. doi:10.1007/s10658-012-

9936-1.

PLOS Computational Biology | www.ploscompbiol.org 4 July 2013 | Volume 9 | Issue 7 | e1003149

Copyright of PLoS Computational Biology is the property of Public Library of Science and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.