Project Portfolio – Final Summation IO Psy Capstone
Steadystate12
prs_10_3_133.pdf
Research Note
On the Experience of Conducting a Systematic Review in Industrial,
Work, and Organizational Psychology Yes, It Is Worthwhile
Céline Rojon,1,2 Almuth McDowall,1 and Mark N. K. Saunders2
1Department of Psychology, University of Surrey, Guildford, UK, 2School of Management, University of Surrey, Guildford, UK
Abstract. Systematic Review methodology (SRm) is an increasingly popular choice for literature reviews in the Social Sciences. While compared to traditional narrative reviews SRm appears time-consuming and laborious, transparency and replicability of the methodology is argued to facilitate greater clarity of review. Nevertheless, researchers in Industrial, Work, and Organizational (IWO) Psychology have yet to embrace this methodology. Drawing on experience from conducting a Systematic Review (SR) of individual workplace performance we explore the premise: The advantages of SRm to IWO Psychology researchers outweigh the disadvantages. We offer observations, insights, and potential solutions to challenges faced during the reviewing process, concluding that SRm is worthwhile for IWO Psychology researchers.
Keywords: Systematic Review, literature review, evidence-based research, performance
Prior to embarking on a new research project, an initial explo- ration of what is already known is crucial, supporting an informed decision about the focus and execution of future studies. Usually, this involves a literature review. A range of possible review strategies (cf. Denyer, 2009; Jesson, Matheson, & Lacey, 2011) is available to Industrial, Work, and Organizational (IWO) Psychology researchers, including the Systematic Review methodology (SRm). SRm is argued to offer advantages over traditional literature reviews, allow- ingsenseto bemadeoflargebodiesofinformationwhilemin- imizing bias (Petticrew & Roberts, 2006) and has for some time gained acceptance in the Social Sciences (Harlen & Crick, 2004). Yet, IWO Psychology researchers have still to embrace SRm; a search of 13 relevant journals (e.g., Journal of Occupational and Organizational Psychology, Journal of Personnel Psychology) elicited no published Systematic Reviews (SRs). We therefore examine the premise that the advantages of SRm to IWO Psychology researchers outweigh thedisadvantages,offeringobservationsandinsightsfromour experience of conducting a SR of individual workplace per- formance; addressing the question ‘‘is it worth it?’’
What is Systematic Review Methodology?
SRm is defined as:
A specific methodology that locates existing studies, selects and evaluates contributions, analyses and
synthesizes data, and reports the evidence in such a way that allows reasonably clear conclusions to be reached about what is and is not known (Denyer & Tranfield, 2009, p. 671).
Originating in the medical sciences, SRm has been used widely, often to evaluate the effectiveness of specific thera- pies or treatments (Leucht, Kissling, & Davis, 2009). During the last two decades, the methodology’s importance has been recognized by other disciplines: Within the Social Sciences, the Evidence for Policy and Practice Information and Coordinating Centre was established in 1995, utilizing SRm to provide more evidence-based guidance for policy and practice (Harlen & Crick, 2004). More recently, the potential value of SRm for evidence-based research has been acknowledged by Management and Organization Sciences (MOS) researchers, as a ‘‘key methodology for locating, appraising, synthesizing, and reporting best evidence’’ (Briner, Denyer, & Rousseau, 2009, p. 24), supporting its potential for IWO Psychology.
Reasons for Using SRm
SRm is distinguished from traditional narrative and other forms of literature review in two interrelated ways (Petticrew & Roberts, 2006). Firstly, SRm adheres closely to a set of pro- cesses to limit researcher bias through attempting to identify, appraise, and synthesize all studies relevant to the research
� 2011 Hogrefe Publishing Journal of Personnel Psychology 2011; Vol. 10(3):133–138 DOI: 10.1027/1866-5888/a000041
T hi
s do
cu m
en t i
s co
py ri
gh te
d by
th e
A m
er ic
an P
sy ch
ol og
ic al
A ss
oc ia
tio n
or o
ne o
f i ts
a lli
ed p
ub lis
he rs
. T
hi s
ar tic
le is
in te
nd ed
s ol
el y
fo r t
he p
er so
na l u
se o
f t he
in di
vi du
al u
se r a
nd is
n ot
to b
e di
ss em
in at
ed b
ro ad
ly .
question(s). Secondly, these processes are defined in advance and reported in sufficient detail to enable replication.
However, the key question for IWO Psychology researchers is: What advantages does this methodology for reviewing the literature offer? This is important given the availability of alternatives including meta-narrative approaches (e.g., Greenhalgh et al., 2005), critical appraisals (e.g., Hill & Spittlehouse, 2003), and realist reviews (e.g., Pawson, Greenhalgh, Harvey, & Walshe, 2005). In answer to this, SRm literature highlights distinct advantages over such approaches, summarized in Table 1.
Yet literature also acknowledges SRm as laborious and time-consuming. The average SR requires seven months’ work by a team of reviewers (Allen & Olkin, 1999; cited in Petticrew & Roberts, 2006), emphasizing the need to answer the question ‘‘is it worthwhile?’’ We do this by drawing on our experiences of a transdisciplinary SR of individual workplace performance combining literatures from IWO Psychology and MOS. Our motivation for this SR was to provide an up-to-date, exhaustive integration of the available research evidence to inform researchers and practitioners alike. Previous reviews were undertaken mostly in the 1990s (e.g., Arvey & Murphy, 1998), address- ing the topic less comprehensively than our approach. While we considered the structured and meticulous procedures applied in SRm a particularly suitable aid in consolidating the ample, heterogeneous literature available, we also need to explore to what extent the advantages of SRm outweigh the disadvantages.
Method
The focus and conduct of SRm varies between disciplines. Petticrew and Roberts (2006) have developed guidelines for the Social Sciences, using SRm to address evidence about a variety of questions, rather than focusing solely on cause-effect relationships as emphasized in medical sci- ences. In MOS research, Denyer and Tranfield (2009) have adapted medical sciences’ guidance to suit their discipline, offering four (amended) methodological principles: (1) transparent, (2) inclusive, (3) explanatory, and (4) heuristic. Both sets of guidelines can be integrated as a process con- sisting of a scoping study followed by five discrete review stages. These we now outline and illustrate using the review topic individual workplace performance, reflecting on chal- lenges encountered and offering possible solutions.
Pre-Review Scoping Study
A scoping study typically precedes the actual review to determine the basis of the literature search, ascertaining if a review is actually needed or if it would be mere replica- tion. This is of particular importance for cross-disciplinary topics such as individual workplace performance due to the dispersed spread of evidence. An a-priori search within relevant SR databases (e.g., Cochrane Database of System- atic Reviews) indicated no previous SRm reviews had been Ta
b le
1 . A d v an ta g es
o f S R m
co m p ar ed
to o th er
re v ie w in g ap p ro ac h es
S R m
(e .g ., B ri n er
& R o u ss ea u , 2 0 1 1 ; D en y er
& T ra n fi el d , 2 0 0 9 ; P et ti cr ew
& R o b er ts , 2 0 0 6 )
T ra d it io n al
n ar ra ti v e re v ie w
(J es so n , M at h es o n , &
L ac ey , 2 0 1 1 )
M et a- an al y si s
(H u n te r &
S ch m id t, 2 0 0 4 )
R ig o r, th o ro u g h n es s, an d o b je ct iv it y th ro u g h ad h er en ce
to cl ea r p ri n ci p le s an d p re sc ri b ed
st ag es
o f re v ie w in g
D o es
n o t g en er al ly
h av e a fo rm
al m et h o d o lo g y,
th u s re su lt in g
in la ck in g ri g o r, tr an sp ar en cy , an d re p li ca b il it y b y o th er s;
b u t: al lo w s m o re
fl ex ib il it y an d ex p lo ra ti o n o f re se ar ch er ’s id ea s
O n ce
ap p ro p ri at e m et a- an al y si s m et h o d h as
b ee n
ch o se n (e .g ., st at is ti ca l m et a- an al y si s) , it is cr u ci al
to il lu st ra te
cl ea rl y p ro ce ss
o f lo ca ti n g , ev al u at in g ,
se le ct in g , an d co d in g st u d ie s to
al lo w
re p li ca b il it y
C o n si d er at io n an d re co n ci li at io n o f al l p o te n ti al ly
re le v an t
so u rc es
o f in fo rm
at io n al lo w s co m p re h en si v e co ll at io n o f
al l ex is ti n g ev id en ce
ac ro ss
re le v an t st u d ie s an d in te g ra ti o n
o f d if fe re n t sc h o o ls o f th o u g h t an d re se ar ch
fi n d in g s an d
is p ar ti cu la rl y su it ab le
w h en
aw ar e o f m ai n th em
es co n ce rn in g
th e re v ie w
to p ic , b u t u n su re
o f ac tu al
ev id en ce
R es ea rc h er
ca n fo cu s o n ‘‘ p re fe rr ed ’’ li te ra tu re
so u rc es
(e .g ., fa v o ri te
d at ab as es ) an d b as e re v ie w
o n a p er so n al ,
p u rp o si v e se le ct io n o f m at er ia ls
th ey
b el ie v e to
b e im
p o rt an t,
th u s p o te n ti al ly
in tr o d u ci n g a o n e- si d ed
o r ev en
b ia se d ar g u m en t
R es ea rc h er
ca n b e v er y se le ct iv e as
to w h ic h
st u d ie s to
in cl u d e in
th ei r m et a- an al y se s, th u s
p o te n ti al ly
in tr o d u ci n g re se ar ch er
b ia s;
n o t al w ay s m ad e cl ea r en o u g h w h y so m e
st u d ie s h av e b ee n in cl u d ed
w h il e o th er s h av e n o t
F ac il it at io n o f re v ie w s o f to p ic s w h er e a v as t an d h et er o g en eo u s
b o d y o f li te ra tu re
is av ai la b le
b y fo ll o w in g an
a- p ri o ri d ev el o p ed
p ro to co l th at
cl ea rl y st at es
ta sk s an d st ag es
o f th e re v ie w in g p ro ce ss
(e .g ., se le ct io n an d ev al u at io n o f re fe re n ce s b y m ea n s o f in cl u si o n /
ex cl u si o n cr it er ia )
T h er e is n o t u su al ly
a re v ie w
p ro to co l o r st ra te g y o r a d efi n ed
m et h o d to
fo ll o w , w h ic h ca n m ak e it d if fi cu lt to
id en ti fy
an d
re v ie w
to p ic s w h er e a v as t an d h et er o g en eo u s b o d y o f li te ra tu re
is av ai la b le
C an
fa ci li ta te
th e q u an ti ta ti v e re v ie w
o f ar ea s w it h in
P sy ch o lo g y an d o th er
S o ci al
S ci en ce s in
w h ic h
n u m b er
o f av ai la b le
st u d ie s is la rg e an d fi n d in g s
se em
co n tr ad ic to ry
b y ad h er in g to
st at is ti ca l an d
p sy ch o m et ri c p ri n ci p le s o f d at a an al y si s in
m et a- an al y si s
C o m b in at io n o f an al y si s an d sy n th es is m et h o d s p o ss ib le ,
th at
is , ca n in cl u d e b o th
a n ar ra ti v e an d a m et a- an al y si s
co m p o n en t
W h il e re v ie w
m ay
co n ta in
a m et a- an al y si s, it is ty p ic al ly
fo cu se d o n
th e n ar ra ti v e co m p o n en t (q u al it at iv e sy n th es is )
W h il e re v ie w
m ay
co n ta in
n ar ra ti v e el em
en t,
it is ty p ic al ly
fo cu se d o n th e m et a- an al y si s co m p o n en t
(q u an ti ta ti v e sy n th es is )
134 C. Rojon et al.: Systematic Reviews in IWO Psychology
Journal of Personnel Psychology 2011; Vol. 10(3):133–138 � 2011 Hogrefe Publishing
T hi
s do
cu m
en t i
s co
py ri
gh te
d by
th e
A m
er ic
an P
sy ch
ol og
ic al
A ss
oc ia
tio n
or o
ne o
f i ts
a lli
ed p
ub lis
he rs
. T
hi s
ar tic
le is
in te
nd ed
s ol
el y
fo r t
he p
er so
na l u
se o
f t he
in di
vi du
al u
se r a
nd is
n ot
to b
e di
ss em
in at
ed b
ro ad
ly .
undertaken on this topic, giving us the impetus to pursue our idea further. To identify the precise focus for the review, we undertook an initial, exploratory search for relevant litera- ture. This revealed diverse understandings of the individual workplace performance construct, in particular its definition, conceptualization, and measurement (e.g., is the construct uni- or multi-dimensional?). These issues provided the focus for our subsequent review.
Determination of Specific Review Questions
Clearly framed, answerable research questions provide the basis for selecting potentially relevant studies for a SR. As recommended (Denyer & Tranfield, 2009; Petticrew & Rob- erts, 2006), we consulted an advisory panel of 10 experts (non-probability maximum variation sample). These com- prised Psychology and MOS academics with research foci in workplace performance (e.g., professors of Human Resource Management (HRM) and Occupational Psychol- ogy), chosen on the grounds of having specific expertise on the topic; alongside private and public sector HRM prac- titioners (e.g., an Organizational Development Manager in government administration) to provide a practitioner per- spective, based on real-world experience of performance management.
We conducted broad semi-structured interviews with these heterogeneous stakeholders. This flexible approach ensured that the review questions would remain aligned with our focus, as elicited from the initial scoping stage, but also useful and practically relevant to a wider community.
Qualitative content analysis of these interviews involved examination and interpretation of responses focusing on main themes to firm up review questions from the experts’ suggestions. Reconciling different stakeholder perspectives proved challenging; we resolved this issue by ensuring a balance of academic and practitioner foci in the final review questions. Feedback from the expert panel indicated
that these were sufficiently focused to allow new meaningful theoretical insights, while comprehensive to inform practical performance contexts, namely:
Table 2. Number of references identified in Systematic Review Stages 2 and 3
Stage Source/Process Number of potentially relevant references
2 Searching the literature Web of Science database 35,173 PsycInfo database 11,381 Business Source Complete database 9,079 Medline database 4,145 Emerald Management eJournals database 2,200 IBSS (International Bibliography of the Social Sciences) database
874
Psychology & Behavioral Sciences Collection database
708
AOM (Academy of Management) Conference Proceedings database
301
Psybooks database 54 CIPD (Chartered Institute of Personnel and Development) Research Summaries database
34
Manual searches (three journals, four conferences, scholars’ literature)
29
British Library e-Theses database 23 I&DeA (Improvement and Development Agency for Local Government) database
5
Total (prior to removal of duplicates) 64,006 Total (after removal of duplicates) 59,465
3 Selecting and evaluating references Screening by title alone 3,010 Screening by title and abstract 314 Evaluation by full text 171
Table 3. Sample inclusion/exclusion criteria for study selection
Criterion Explanation of criterion
Does the study address any/all of the review questions? Is it relevant to any/all of the review questions?
Only studies that can contribute to answering any or all of the review questions are useful for the SR; all others are excluded outright
Is the study well informed by existing theory? Outlining of previous findings and existing theory and integration within study (cf. Cassell, 2010; Denyer, 2009)
Are the purpose and aims of the study clearly specified? Clear specification of the research questions and objectives addressed (cf. Denyer, 2009)
Are the methods chosen appropriate to the stated purpose? Clear explanation of what methods of data collection were chosen and why (cf. Cassell, 2010; Denyer, 2009)
Does the study claim a contribution? Study creates, extends, or advances knowledge in a meaningful way. Guidance for future research is provided
Is the study relevant for the practice? Usefulness and applicability of the results for a practitioner, such as an IWO psychologist, a human resources manager, etc. Author comments on how this is the case (cf. Cassell, 2010; Denyer, 2009)
Are the conclusions well linked to the purpose and aims of the research?
Reference back to initially formulated research questions and aims; establishment of clear links (Cassell, 2010)
C. Rojon et al.: Systematic Reviews in IWO Psychology 135
� 2011 Hogrefe Publishing Journal of Personnel Psychology 2011; Vol. 10(3):133–138
T hi
s do
cu m
en t i
s co
py ri
gh te
d by
th e
A m
er ic
an P
sy ch
ol og
ic al
A ss
oc ia
tio n
or o
ne o
f i ts
a lli
ed p
ub lis
he rs
. T
hi s
ar tic
le is
in te
nd ed
s ol
el y
fo r t
he p
er so
na l u
se o
f t he
in di
vi du
al u
se r a
nd is
n ot
to b
e di
ss em
in at
ed b
ro ad
ly .
1. How is individual workplace performance as a criterion defined and conceptualized?
2. How is individual workplace performance measured and why?
3. What are the relationships, if any, between overall ver- sus criterion-specific measures of workplace perfor- mance and established predictors (i.e., ability and personality measures)?
Searching the Literature
We undertook a comprehensive search to locate all poten- tially relevant studies. The challenge was to ensure that potential key references were not excluded. Tailored search strings combined terms relating to the research topic; for example, to find references pertaining to ‘‘measurement,’’ the string ‘‘assess* OR apprais* OR evaluat* OR test OR rating OR review OR measure OR manage*’’ was used, the asterisk enabling searching on truncated word forms. Pilot database searches proved useful in determining the util- ity of such strings and specifying a start date.
To help ensure maximum saturation, 12 databases and proceedings from four conferences (e.g., Society for Indus- trial and Organizational Psychology Conference) were used following a subject librarian’s advice. Further manual searches (for three journals inaccessible through the dat- abases, e.g., ‘‘Assessment & Development Matters’’) were undertaken and requests sent to scholars with relevant research interests. After removal of duplicates, this resulted in a database of 59,465 references (Table 2).
Selecting and Evaluating References
References were screened initially by title for relevance to the three review questions, reducing their number to 3,010. Subsequent screening by title and abstract reduced their number to 314. Having obtained full text copies of all, these were read and evaluated using inclusion/exclusion criteria (Table 3) derived from researchers’ (mostly with experience in SRm) suggestions (Briner et al., 2009; Cassell, 2010; Denyer, 2009) – this was a time-consuming process, taking approximately three months. Despite precisely defined criteria, digression was a potential challenge, being minimized through constant focus on the review questions. The 171 publications that met the inclusion criteria of satis- factory quality and contributing to answering the review questions were our final pool of references (Petticrew & Roberts, 2006). For each study key information (e.g., study context, data collection methods, findings in relation to the review questions) was recorded using a data extraction form.
Analyzing and Synthesizing Findings
SRm literature (e.g., Petticrew & Roberts, 2006; Rousseau, Manning, & Denyer, 2008) indicates a variety of processes for synthesizing evidence recorded on data extraction forms (e.g., synthesis by explanation or aggregation), depending on the type of review questions asked and the available data. T a b le
4 . D es cr ip ti v e st at is ti cs
o f th e p u b li ca ti o n s u se d in
th e re v ie w
(N = 1 7 1 )
T y p e o f d o cu m en t
% /N
P ee r- re v ie w ed
jo u rn al s
% /N
Y ea r o f p u b li ca ti o n
% /N
Q u al it y
% /N
P ee r- re v ie w ed
jo u rn al
ar ti cl es
8 4 .8 /1 4 5
Jo u rn al
o f A p p li ed
P sy ch o lo g y
1 7 .0 /2 9
2 0 0 1 – 2 0 1 0
3 5 .1 /6 0
H ig h o v er al l q u al it y
5 3 .2 /9 1
N o n -p ee r- re v ie w ed
jo u rn al
ar ti cl es
1 .2 /2
P er so n n el
P sy ch o lo g y
9 .4 /1 6
1 9 9 1 – 2 0 0 0
4 0 .9 /7 0
A b o v e av er ag e o v er al l q u al it y
2 5 .1 /4 3
Jo u rn al
ar ti cl es
(w it h p ee r- re v ie w
st at u s u n cl ea r)
1 .2 /2
In te rn at io n al
Jo u rn al
o f
S el ec ti o n an d A ss es sm
en t
7 .6 /1 3
1 9 8 1 – 1 9 9 0
1 0 .5 /1 8
A v er ag e o v er al l q u al it y
2 1 .6 /3 7
B o o k ch ap te rs
9 .4 /1 6
H u m an
P er fo rm
an ce
7 .0 /1 2
1 9 7 1 – 1 9 8 0
9 .4 /1 6
D o ct o ra l th es es
2 .3 /4
Jo u rn al
o f O cc u p at io n al
an d
O rg an iz at io n al
P sy ch o lo g y
4 .7 /8
1 9 6 1 – 1 9 7 0
3 .5 /6
C o n fe re n ce
p ro ce ed in g s
1 .2 /2
Jo u rn al
o f B u si n es s &
P sy ch o lo g y
4 .7 /8
1 9 5 1 – 1 9 6 0
0 .6 /1
O th er
jo u rn al s
4 1 .1 /5 9
N o te s. A n o v er al l q u al it y sc o re
ra n g in g fr o m
1 (l o w
o ve ra ll q u a li ty ) to
5 (h ig h o ve ra ll q u a li ty ) w as
o b ta in ed
fo r ea ch
p u b li ca ti o n – th is w as
b as ed
o n th e ap p li ca ti o n o f p re v io u sl y
d et er m in ed
in cl u si o n /e x cl u si o n cr it er ia
re la ti n g to
q u al it y (T ab le
3 ).
136 C. Rojon et al.: Systematic Reviews in IWO Psychology
Journal of Personnel Psychology 2011; Vol. 10(3):133–138 � 2011 Hogrefe Publishing
T hi
s do
cu m
en t i
s co
py ri
gh te
d by
th e
A m
er ic
an P
sy ch
ol og
ic al
A ss
oc ia
tio n
or o
ne o
f i ts
a lli
ed p
ub lis
he rs
. T
hi s
ar tic
le is
in te
nd ed
s ol
el y
fo r t
he p
er so
na l u
se o
f t he
in di
vi du
al u
se r a
nd is
n ot
to b
e di
ss em
in at
ed b
ro ad
ly .
For the first two review questions, evidence was synthesized qualitatively through narrative integration, involving com- parison and corroboration (Rousseau et al., 2008). The third review question was addressed quantitatively through aggre- gation, using statistical meta-analysis (Hunter & Schmidt, 2004). This combination of analytic methods allowed full integration of evidence considered.
Discussion and Utilization of the Findings
Review findings addressed all three questions accentuating what was known, what was not known, where future research should focus, and how this might inform policy and practice. Overall, the SR process was laborious, taking nine months and offering a potential challenge regarding
loss of motivation. We avoided this through regular discus- sions with other systematic reviewers.
Results
Some 84.8% of the 171 publications used to address the three review questions were peer-reviewed journal articles, having been published in a wide variety of journals (N = 52) (Table 4), over half being published in six journals. Documents included in the SR were published between 1959 and 2010, with more than 75% published in the last 20 years. Over half (53.2%) of publications included were considered of high overall quality as defined through the inclusion/exclusion criteria (Table 3). 19.3% of publications addressed at least two of the review questions. Questions 1,
Table 5. Potential challenges and solutions: SRs in IWO Psychology (#) and other disciplines (*)
Stage Potential challenge Potential solution
1 Expert consultation to reflect on the review topic and to determine review question(s) #
- Involve a heterogeneous sample of stakeholders - Ask open, broad interview questions - Ensure resulting review questions are comprehensive, yet sufficiently focused
- Justify the role of each stakeholder in the expert panel
Analysis and reconciliation of different stakeholder perspectives #
While review questions may have a more academic focus, answers need to be formulated in a way meaningful to both academics/researchers and practical contexts
2 Development of appropriate search terms/strings
- Discuss planned literature searches with librarian - Include any terms in the search strings that are related to the
review question(s) - Conduct pilot searches
3 and 4 Reference management - Accept only relevant studies of a satisfactory quality - Establish well-defined inclusion/exclusion criteria - Focus on review question(s) throughout to avoid digression - Allow sufficient time to acquire and assess references
Stringent inclusion/exclusion criteria (Smith, Devane, Begley, & Clarke, 2011)*
- SRs in health care (such as Smith et al., 2011) may restrict their inclusion criteria to studies that are randomized controlled trials; such criteria are unlikely to be useful to SRs in IWO Psychology as they would delimit the scope of the review too strongly
Across all stages Time management - Expect that most tasks will take longer than anticipated - Acquiring new knowledge and skills for SRm will take time, too
Dwindling motivation - Discuss process and progress with other systematic reviewers
Large number of SRs of variable quality scope on important topics (Smith et al., 2011)*
- SRm being a new approach to reviewing literature in IWO Psychology, this aspect is unlikely to be a problem to researchers in this discipline; in the case of Smith and colleagues (2011), a SR of SRs is suggested to overcome this particular problem
Notes. Challenges experienced by researchers in disciplines other than IWO Psychology have been marked with an asterisk (*), those applicable to SRs in IWO Psychology and neighboring disciplines with a hash (#). Challenges that are unmarked are considered applicable to SRs in any discipline.
C. Rojon et al.: Systematic Reviews in IWO Psychology 137
� 2011 Hogrefe Publishing Journal of Personnel Psychology 2011; Vol. 10(3):133–138
T hi
s do
cu m
en t i
s co
py ri
gh te
d by
th e
A m
er ic
an P
sy ch
ol og
ic al
A ss
oc ia
tio n
or o
ne o
f i ts
a lli
ed p
ub lis
he rs
. T
hi s
ar tic
le is
in te
nd ed
s ol
el y
fo r t
he p
er so
na l u
se o
f t he
in di
vi du
al u
se r a
nd is
n ot
to b
e di
ss em
in at
ed b
ro ad
ly .
2, and 3 were addressed, respectively, by 27.5%, 36.3%, and 49.7% of publications, indicating how individual workplace performance was defined and conceptualized as a criterion was the least researched aspect, analysis of the data extrac- tion forms enabling identification of knowledge gaps.
Discussion
This note has explored the premise that the advantages of SRm to IWO Psychology researchers outweigh the disadvan- tages, addressing the question ‘‘is it worth it?’’ Based on the experiences outlined we consider that, despite SRm being laborious and time-consuming for establishing best evidence, both process and outcome are worthwhile given the effort required. Firstly, the rigor and standardization of SRm results in greater transparency, explicitness, and replicability than may be achieved through traditional narrative reviews. It is suited to topics such as workplace performance, where differ- ent disciplines and theoretical and practical orientations need to be integrated. Secondly, using an expert panel of academ- ics and professionals to help determine review questions facilitates theoretical rigor and practical relevance in the review. Thirdly, using precise inclusion/exclusion criteria alongside continued focus on review questions ensures qual- ity while minimizing digression.
Invariably SRm, regardless of the review context, pre- sents challenges. In our case, the key challenge, which reviewers with narrower or less commonly researched ques- tions might not face, was dealing with the large number of references. Further challenges addressed during our SR, which we believe fellow researchers in IWO Psychology and neighboring disciplines (e.g., HRM) might encounter, alongside solutions and challenges experienced by reviewers in different disciplines, are summarized in Table 5. Finally, individuals planning a SR in similar contexts are advised to ensure stakeholders from constituent disciplines are involved in the scoping study and determination of specific review questions. SRm is not a universal process suitable for any topics. Rather, the purpose is to establish current best evidence regarding specified research questions, thereby allowing more informed decisions about future studies. It is therefore necessary to ensure the review focus is appraisal, synthesis, and reporting of existing evidence. While this focus and thus SRm may not always be appropriate, this note offers a better understanding of the utility of this meth- odology. We have found the experience of conducting a SR rewarding, owing to the structured approach taken, skills acquired, and increased confidence from a sound under- standing of extant literature. We would therefore encourage IWO Psychology researchers to embrace and use SRm to their advantage to review best evidence from existing knowledge.
References
Arvey, R. D., & Murphy, K. R. (1998). Performance evalu- ation in work settings. Annual Review of Psychology, 49, 141–168.
Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence- based management: Concept cleanup time? Academy of Management Perspectives, 23, 19–32.
Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 3–22.
Cassell, C. (2010, January). Qualitative guidelines: Criteria for evaluating papers using qualitative research methods. Journal of Occupational and Organizational Psychology. Retrieved from http://onlinelibrary.wiley.com/journal/10.1111/(ISSN) 2044-8325/homepage/qualitative_guidelines.htm
Denyer, D. (2009, October). Reviewing the literature systemat- ically. Cranfield: Advanced Institute of Management Research (AIM). Retrieved from http://aimexpertresearcher. org
Denyer, D., & Tranfield, D. (2009). Producing a systematic review. In D. Buchanan & A. Bryman (Eds.), The Sage handbook of organizational research methods (pp. 671–689). London, UK: Sage.
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science & Medicine, 61, 417–430.
Harlen, W., & Crick, R. D. (2004). Opportunities and challenges of using systematic reviews of research for evidence-based policy in education. Evaluation & Research in Education, 18, 54–71.
Hill, A., & Spittlehouse, C. (2003, October). What is critical appraisal?. Evidence Based Medicine, 3, 1–8.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta- analysis: Correcting for error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
Jesson, J. K., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: Traditional and systematic techniques. London, UK: Sage.
Leucht, S., Kissling, W., & Davis, J. M. (2009). How to read and understand and use systematic reviews and meta-analyses. Acta Psychiatrica Scandinavica, 119, 443–450.
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review – a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10, 21–34.
Petticrew, M, & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Oxford, UK: Blackwell.
Rousseau, D. M., Manning, J., & Denyer, D. (2008). Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals, 2, 475–515.
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011). Methodology in conducting a systematic review of system- atic reviews of healthcare interventions. BMC Medical Research Methodology, 11, 15–20.
Céline Rojon
Department of Psychology & School of Management University of Surrey Guildford Surrey GU2 7XH UK Tel. +44 1483-686870 Fax +44 1483-689553 E-mail [email protected]
138 C. Rojon et al.: Systematic Reviews in IWO Psychology
Journal of Personnel Psychology 2011; Vol. 10(3):133–138 � 2011 Hogrefe Publishing
T hi
s do
cu m
en t i
s co
py ri
gh te
d by
th e
A m
er ic
an P
sy ch
ol og
ic al
A ss
oc ia
tio n
or o
ne o
f i ts
a lli
ed p
ub lis
he rs
. T
hi s
ar tic
le is
in te
nd ed
s ol
el y
fo r t
he p
er so
na l u
se o
f t he
in di
vi du
al u
se r a
nd is
n ot
to b
e di
ss em
in at
ed b
ro ad
ly .
