DB wk 550

Towards a Methodology for Developing Evidence-Informed Management

Knowledge by Means of Systematic Review*

David Tranfield, David Denyer and Palminder Smart Advanced Management Research Centre (AMRC), Cranfield School of Management,

Cranfield University, Cranfield, MK43 OA1, UK

Corresponding author email: D. [email protected]

Undertaking a review of the literature is an important part of any research project. The researcher both maps and assesses the relevant intellectual territory in order to specify a

research question which will further develop the knowledge base. However, traditional

‘narrative’ reviews frequently lack thoroughness, and in many cases are not undertaken

as genuine pieces of investigatory science. Consequently they can lack a means for making sense of what the collection of studies is saying. These reviews can be biased by

the researcher and often lack rigour. Furthermore, the use of reviews of the available

evidence to provide insights and guidance for intervention into operational needs of practitioners and policymakers has largely been of secondary importance. For

practitioners, making sense of a mass of often-contradictory evidence has become

progressively harder. The quality of evidence underpinning decision-making and action

has been questioned, for inadequate or incomplete evidence seriously impedes policy formulation and implementation. In exploring ways in which evidence-informed

management reviews might be achieved, the authors evaluate the process of systematic

review used in the medical sciences. Over the last fifteen years, medical science has

attempted to improve the review process by synthesizing research in a systematic, transparent, and reproducible manner with the twin aims of enhancing the knowledge base

and informing policymaking and practice. This paper evaluates the extent to which the

process of systematic review can be applied to the management field in order to produce a

reliable knowledge stock and enhanced practice by developing context-sensitive research. The paper highlights the challenges in developing an appropriate methodology.

Introduction: the need for an evidence- informed approach

Undertaking a review of the literature to provide the best evidence for informing policy and

practice in any discipline, is a key research objective for the respective academic and practi- tioner communities. The post-World-War-II era witnessed a sharp

focus of attention by academics and practitioners on the discipline and profession of management (Blake and Mouton, 1976; Tisdall, 1982). The pace of knowledge production in this field has been accelerating ever since and has resulted in a body of knowledge that is increasingly fragmen- ted and transdisciplinary as well as being inter- dependent from advancements in the social

*This paper results from research undertaken in Cran- field IMRC (EPSRC) grant no IMRC19, ‘Developing a methodology for evidence-informed management knowledge using systematic review’, Professor David Tranfield and Dr David Denyer.

British Journal of Management, Vol. 14, 207–222 (2003)

r 2003 British Academy of Management

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

sciences (Friedman, Durkin, Phillips and Volt- singer, 2000). In management research, the literature review

process is a key tool, used to manage the diversity of knowledge for a specific academic inquiry. The aim of conducting a literature review is often to enable the researcher both to map and to assess the existing intellectual territory, and to specify a research question to develop the existing body of knowledge further. Management reviews are usually narrative and have been widely criticized for being singular descriptive accounts of the contributions made by writers in the field, often selected for inclusion on the implicit biases of the researcher (Fink, 1998; Hart, 1998). Not surpris- ingly they have also been condemned for lacking critical assessment. The management-research community perpetuates this type of practice by not actively commissioning infrastructural ar- rangements to ensure previous investments in literature reviews are not lost. This tolerance to loss of knowledge forms a high-risk strategy that will inevitably become unsustainable as organiza- tions endeavour further into the networked and knowledge-based economy. Reviews of the available evidence in manage-

ment to assimilate ‘best evidence’ to provide insights and guidance for intervention into the operational needs of practitioners and policy- makers have largely become a secondary con- sideration. Sufficient momentum from academics, practi-

tioners, and government has stirred an urgent need to re-evaluate the process by which manage- ment researchers conduct literature reviews. Over the last fifteen years, medical science has at- tempted to improve the quality of the review process. This paper proposes the view that applying specific principles of the systematic review methodology used in the medical sciences to management research will help in counteracting bias by making explicit the values and assump- tions underpinning a review. By enhancing the legitimacy and authority of the resultant evidence, systematic reviews could provide practitioners and policy-makers with a reliable basis to formulate decisions and take action. This is particularly sobering if one considers the growing pressures upon practitioners in today’s global trading environments to do this in shorter cycle times. This paper will begin by discussing the

evidence-based approach in medical sciences

through the effective use of systematic reviews. The following sections will compare and contrast the nature of reviews in medical science and management research and evaluate the extent to which the systematic review process can be applied to the management field. Finally this paper will present the challenges in designing an appropriate methodology for management re- search.

The origins of the evidence-based approach

Since the 1980s, the British central government has placed increasing emphasis on ensuring that policy and practice are informed through a more rigorous and challenging evidence base. The ‘three E’ initiatives (economy, efficiency and effectiveness) have focused attention on the delivery of public services and have led to the development of detailed guidance and best- practice manuals in many disciplines. Effective- ness in this context is concerned both with appropriateness and the validity of the methods used by professionals in their day-to-day work to achieve their basic aims and also with the overall ability of agencies to deliver the services they are required to provide (Davies, Nutley and Smith, 2000). The concern for effective service delivery has attracted considerable attention, and has focused interest on basing policy and practice on the best evidence available. Consequently, an evidence-based movement has developed under New Labour, and in May 1997 Tony Blair announced that ‘what counts is what works’, the intention being to signal a new ‘post-ideological’ approach to public policy where evidence would take centre stage in the decision-making process (Davies, Nutley and Smith, 2000).

The evidence-based approach in medical science and healthcare

The evidence-based movement has had a major impact in certain disciplines. Pre-eminent have been applications in medical science, where the pace of knowledge production has meant that making sense of an often-contradictory mass of evidence has become increasingly difficult (Ohls- son, 1994). Specifically in the late 1980s, attention

208 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

was drawn to the comparative lack of rigour in secondary research (Mulrow, 1987). Critics ar- gued that the preparation of reviews of secondary sources were dependent on implicit, idiosyncratic methods of data collection and interpretation (Cook, Mulrow and Haynes, 1997; Greenhalgh, 1997). In addition, practice based on poor-quality evaluations of the literature sometimes had led to inappropriate recommendations (Cook, Green- gold, Ellrodt and Weingarten, 1997). In 1991, Smith questioned the overall wisdom of much of medical science, arguing that only 15–20% of medical interventions were supported by solid medical evidence (Smith, 1991). The result, it was argued, was that patients were being regularly subjected to ineffective treatments and interven- tions, and for many practices there was little or no understanding of whether or not the benefits outweighed the potential harm (Davies, Nutley and Smith, 1999). The National Health Service (NHS) Research

and Development Strategy identified that too little research was being carried out in the important clinical areas and that much of the existing research was ad hoc, piecemeal and poorly conducted (Peckham, 1991). The report also argued that researchers rather than practi- tioners, managers or policymakers drove the research agenda. Furthermore, there was little dissemination, let

alone diffusion, of research findings. The Strategy not only argued for an increase in the level of research conducted but also for systematic re- views of existing research on important clinical or operational questions, assessing the best evidence available, collating the findings and presenting them in a way that was accessible and relevant to decision-makers (Peckham, 1991).

Systematic review – a key tool in developing the evidence base

Over the last decade medical science has made significant strides in attempting to improve the quality of the review process by synthesizing research in a systematic, transparent and repro- ducible manner to inform policy and decision- making about the organization and delivery of health and social care (Cook, Greengold, Ellrodt and Weingarten, 1997; Cook, Mulrow and Haynes, 1997; Wolf, Shea and Albanese, 2001).

Systematic reviews differ from traditional narrative reviews by adopting a replicable, scientific and transparent process, in other words a detailed technology, that aims to minimize bias through exhaustive literature searches of pub- lished and unpublished studies and by providing an audit trail of the reviewers decisions, proce- dures and conclusions (Cook, Mulrow and Haynes, 1997). The process of systematic review and its associated procedure, meta-analysis, has been developed over the last decade and now plays a major role in evidence-based practices. Whereas systematic review identifies key scien-

tific contributions to a field or question, meta- analysis offers a statistical procedure for synthe- sizing findings in order to obtain overall relia- bility unavailable from any single study alone. Indeed, undertaking systematic review is now regarded as a ‘fundamental scientific activity’ (Mulrow, 1994, p. 597). The 1990s saw several organizations formed with the aim of establishing agreed and formalized procedures for systematic review and to undertake systematic reviews to synthesize and disseminate evidence across all areas of healthcare. These organizations included the Cochrane Collaboration (2001), the National Health Science Centre for Reviews and Dissemi- nation (2001) and the National Institute for Clinical Excellence (2001).

Evidence-based approaches in other disciplines

The movement to base practice on the best available evidence has migrated from medicine to other disciplines. In the UK, the Department for Education and Skills (DfES) has established a Centre for Evidence Informed Policy and Prac- tice in Education. Furthermore, a ‘What Works? Programme’ was introduced in the probation service following the Crime Reduction Strategy published by the Home Office in July 1998. The aim of the programme was to develop successful intervention programmes based on hard evidence so that they could be used as models for day-to- day probation practice (HM Inspectorate of Probation, 1998; Home Office, 1998). An Effec- tive Practice Initiative also has sought to address the difficult problem of ensuring that offender supervision changes in line with research evidence on what works (Furniss and Nutley, 2000). The

Developing Evidence-Informed Management Knowledge 209

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

Department for the Environment, Transport, and the Regions (DETR) commissioned a review of the evidence base as it relates to regeneration policy and practice (Dabinett, Lawless, Rhodes and Tyler, 2001). Other disciplines such as nursing (Evans and Pearson, 2001), housing policy (Davies and Nutley, 1999; Maclennan and More, 1999), social care (Macdonald, 1999) and criminal justice (Laycock, 2000) have also adjusted the approach with varying degrees of success. In 2001, the Economic and Social Research Council (ESRC) funded the establish- ment of a network (the Evidence Network) of multi-disciplinary centres dedicated to the im- provement of the evidence base for policy and practice in social sciences. The Evidence Network aims to use systematic review to inform and improve decision-making in government, busi- ness and the voluntary sector. Internationally, in February 2000 the Camp-

bell Collaboration was launched in Philadelphia by about 150 pioneering social scientists. This equivalent of the Cochrane collaboration aims:

‘to help people make well-informed decisions about

the effects of interventions in the social, behavioural

and educational arenas’ (Campbell Collaboration,

2001).

Within the approach taken by the Campbell Collaboration, delegates considered questions such as how practitioners might engage the review process, what makes research useful and useable and what standards and quality criteria distinguished reliable from unreliable research? In this sense, discussions addressed the need for

research to be both well founded and socially robust. This emphasis on producing a science base, which is both rigorous in formulation and relevant to practice, is a key characteristic of an evidence-based approach. The quality of information accepted as evi-

dence in a discipline is dependent on a number of criteria. These include the broad intellectual approach, the value system adopted by research- ers and commissioning bodies and the usual research methods employed (Davies and Nutley, 1999). Medical science has traditionally adopted a ‘normal science’ approach within which double- blinded randomized controlled trials have been widely accepted as the most rigorous method for testing interventions before use. So far, systema- tic reviews have tended to be applied in, and to emanate from, fields and disciplines privileging a positivist tradition, attempting to do for research synthesis what randomized controlled trials aspire to do for single studies (Macdonald, 1999). Systematic reviews entail a series of techni- ques for minimizing bias and error, and as such systematic review and meta-analysis are widely regarded as providing ‘high-quality’ evidence. Figure 1 highlights the hierarchy of evidence in the medical sciences (Davies and Nutley, 1999). In other disciplines such as education, social

services and criminal justice there is often both less consensus regarding the appropriate metho- dology to be used for evaluating the evidence base, and little agreement as to how use research evidence to inform policy and practice (Davies and Nutley, 1999; Laycock, 2000; Macdonald, 1999; Maclennan and More, 1999). Furthermore, policy

Hierarchy of evidence

I-I Systematic review and meta-analysis of two or more double blind randomized controlled trials.

I-2 One or more large double-blind randomized controlled trials.

II-1 One or more well-conducted cohort studies.

II-2 One or more well-conducted case-control studies.

II-3 A dramatic uncontrolled experiment.

III Expert committee sitting in review; peer leader opinion.

IV Personal experience.

Figure 1. Hierarchies of evidence. Source: reproduced by kind permission of the publisher from Davies, H. T. O. and S. M. Nutley

(1999). ‘The Rise and Rise of Evidence in Health Care’, Public Money & Management, 19 (1), pp. 9–16.r 1999 Blackwell Publishing.

210 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

questions are rarely addressed by the use of randomized controlled trials. For example, in social care the nature of evidence is often hotly disputed and there exists strong resistance to privileging one research method over another. Indeed, postmodern perspectives generally mis- trust any notion of objective evidence. Divergences such as these are deeply rooted in

the ontological and epistemological assumptions of specific fields. Despite these difficulties, Davies, Nutley and Smith argue optimistically:

‘The different ontological and epistemological start-

ing points in different professional traditions un-

doubtedly colour the methods and enthusiasm with

which professionals engage with evidence. However,

what is clear is that there remains in all of the areas

examined great potential for research evidence to be

vastly more influential than hitherto’. (2000, p. 4)

The nature of management research

The nature of the field of management research has been subject, over the years, to considerable analysis and discussion. Much of this discussion and debate has focused upon the ontological status of the field, particularly its fragmented and divergent nature. For example, Whitley (1984a, 1984b), in two influential articles, investigated the scientific status of management research as a ‘practically oriented social science’. He identified its fragmented state and argued that the conse- quence of this is a:

‘low degree of reputational control over significance

standards y (which) means that the significance of

problems and preferred ways of formulating them

are unstable, subject to disputes, and are assessed

by diffused and diverse standards.’ (Whitley, 1984a,

p. 343)

Whitley (2000) further refined this position, suggesting that the continued fragmentation of the management field may displace academics as key stakeholders in the research process. In comparing management research with industrial, work and organizational psychology, Hodgkin- son, Herriot and Anderson (2001, s45) also conclude that there is a considerable and widen- ing divide between academics and other stake- holder groups and that ‘this divergence is likely to further proliferate irrelevant theory and untheor- ized and invalid practice’.

Pettigrew (1997, p. 291), in much the same vein as Whitley, emphasized the significance of the social production of knowledge in viewing management research, emphasizing stakeholder perspectives. His influential view was that man- agement research faces a series of challenges:

‘best captured in a series of concurrent double

hurdles, which together raise a wide spectrum of

cognitive, social and political demands on [the]

skills and knowledge of [management] researchers.’

He argued for a thematic approach:

‘to meet the double hurdle of embeddedness in the

social sciences and the worlds of policy and

practice’ (Pettigrew, 1997, p. 292).

Berry (1995) offered a Gallic perspective, arguing strongly the case for the importance of qualitative work. Several writers (Aram and Salipante, 2000; Pfeffer and Sutton, 1999; Van de Ven, 1998; Wind and Nueno, 1998) have argued convincingly for the applied nature of management research. Likewise, Hambrick (1994) and Huff (2000) both used their addresses as President of the Academy of Management to address the ontological status of the field. More recently, Wilcoxson and Fitzgerald (2001) have focused on the nature of management as a discipline and the consequences of this for researchers and practitioners in an Australasian context and Van Aken (2001) has developed a view of management research based as a design science, rather than as a formal or explanatory science. By conceptualizing management research in this way, he identifies the need for a field of study to deliver output not only of high academic quality but also which is practitioner and context-sensitive. He argues that the mission of design sciences is to develop valid and reliable knowledge in the form of ‘field tested and grounded technological rules’ to be used in designing, configuring and implementing solu- tions to specific problems. The 1990s saw an extensive debate concerning

the nature of management research within the British Academy of Management, which focused on the ontological status of the field, and particularly the extent to which academic–practi- tioner relations were to be privileged. The work of Gibbons et al. (1994) on modes of knowledge production has become increasingly influential in such debates. In particular, their notion of mode

Developing Evidence-Informed Management Knowledge 211

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

2 knowledge production, where there is ‘a constant flow back and forth between the theoretical and the practical’ and where ‘knowledge is produced in the context of application’ has been argued to be central to debates about the future of management research (Tranfield and Starkey, 1998). Creating a management research which is both theoretically sound and methodologically rigorous as well as relevant to the practitioner community has been a theme explored by both the British Academy of Management and the Foundation for Manage- ment Education (Starkey and Madan, 2001). This discussion was developed further in a special issue of the British Journal of Management (Hodgkinson, 2001).

Comparing the management and medical fields

Tranfield and Starkey (1998), in an article which both reflected and drove the debate in the British Academy of Management, used Becher’s (1989) dimensions drawn from the sociology of knowl- edge to characterize management research as ‘soft’ rather than ‘hard’ ‘applied’ rather than ‘pure’, rural’ rather than ‘urban’, and ‘divergent’ rather than ‘convergent’. The creation of such a profile, with the use of dimensions drawn from the sociology of knowledge, enabled contrasts to be made with other disciplines, particularly medical science, where systematic review has been applied to considerable effect. Comparison can be made in both epistemological and ontological realms. Whereas medical research enjoys considerable

and extensive epistemological consensus, this is untrue of management research, in general. The consequential difficulties of establishing agreed thresholds for high-quality work result from this lack of consensus. Key ontological differences between manage-

ment research and medical science concern the dimension ‘convergent–divergent’. The extent to which a discipline resides at one end of this dimension or another is purported to depend upon similarities in research ideologies, values and quality judgements which create a shared sense of nationhood amongst researchers within the field (Becher, 1989). Agreements concerning key research questions to be addressed lead to a relatively low tolerance of deviance, but have the

advantage of defining disciplinary boundaries, making them easy to defend. Thus, the extent to which disciplines are opening up research ques- tions, or addressing a previously defined and agreed agenda, dictates positioning on this dimension. Management research is a relatively young

field, far less well developed in terms of agenda and question formulation than much of medical science. As a result there tends to be low consensus concerning key research questions in management research. Studies in the field rarely address identical problems and share a research agenda or, more importantly, ask the same questions. Therefore, it is unlikely that aggrega- tive approaches to research synthesis, such as meta-analysis will be appropriate in management research as the heterogeneity of studies prevents the pooling of results and the measurement of the net effectiveness of interventions. Table 1 outlines the similarities and differences

between medical science as an applied field of study stemming from the biological sciences, and management research as an applied field with strong connections to the social sciences. The main question here is to what extent

review processes developed in fields that are striving to become evidence based, such as the more convergent field of medicine, can inform the review process in the management field to help create rigorous and relevant reviews. As manage- ment research questions need to be clearly specified, either as replication of an existing study, as further development of an existing study, or as a new study to meet a defined ‘gap’ in the literature, a more systematic literature review process can help to justify/qualify the near/final research question which is posed. Furthermore, the process described/proposed in this paper values and takes steps to encourage participation, by both academics and by managers/policy- makers, and is pragmatic in intent. Systematic reviews have traditionally been

applied in fields and disciplines privileging a positivist and quantitative tradition:

‘Positivists seek cause-and-effect laws that are

sufficiently generalizable to ensure that a knowledge

of prior events enables a reasonable predication of

subsequent event y Because positivists see knowl-

edge as accumulating, they have been more inter-

ested in developing approaches to research

212 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

Table 1. Differences between medical research and management research

Medicine Management

Nature of the discipline Convergent Divergent. Research culture Subjected to rigorous scientific evaluation. Split between positivist and phenomenological

perspectives. Research questions High consensus over research questions. Low consensus over research questions. Interventions Can be measured through experiments. Experimentation may or may not be feasible. Research designs Based upon a hierarchy of evidence. Triangulation is recommended. Theory Concerned with what works–did the intervention

offer overall benefits. Concerned with why something works or does not work and the context in which this occurs.

Aims of policy Generally reducing illness and death, and improving health.

Multiple and competing and the balance between them may change over time.

Weight of inputs into policy Scientific evidence. Many extraneous factors. Methods Predominantly quantitative. Quantitative and qualitative. Literature reviews Systematic review and meta-analysis. Largely narrative reviews. The need for a review Reviews of effectiveness are used by clinical

practitioners. To develop a research question and inform empirical research practice.

Preparation of the review A review panel (including practitioners) guides the process. A brief scoping study is conducted to delimit the subject area.

Usually an informal/ad hoc process involving the researcher, peers and supervisor.

Review protocol A plan prior to the review states the criterion for including and excluding studies, the search strategy, description of the methods to be used, coding strategies and the statistical procedures to the employed. Protocols are made available by international bodies to enhance networking the exchange of knowledge.

Level of formality and standardisation in designing/adopting protocols is usually low. Unacceptable to ‘tightly’ plan literature review, as this may inhibit the researchers capacity to explore, discover and develop ideas.

Identifying research A comprehensive, structured search is conducted using predetermined keywords and search strings.

Identifying a field/sub- fields of study generally occurs through informal consultation. Implicit idiosyncratic methods of data collection are used.

Selection of studies Inclusion and exclusion criteria are expressed in the protocol to ensure a review of the best available evidence. Draw upon ‘raw data’ from ‘whole studies’ for analysis to create a study in its own right.

Based on studies that appear relevant or interesting. Researchers bias disables critical appraisal. Decisions regarding choice are not recorded precluding any audit trails ‘Raw data’ is often not available in academic articles, which usually represent ‘partial studies’. Precise inclusion/exclusion criteria are often not formally agreed, applied recorded or monitored.

Study quality assessment Studies are assessed against predetermined criteria. The internal validity of the study is judged Assessing and including qualitative studies is problematic.

Poor evaluating of the fit between research methodology and research questions. Researchers tend to rely on the quality rating of a particular journal, rather than applying quality assessment criteria to individual articles.

Data extraction Data extraction forms are used which act as a historical record for the decisions made during the process and provides the basis on which to conduct data synthesis.

Data extraction is not formally guided by explicitly stated inclusion and exclusion criteria. Data extracted is not comprehensively recorded and monitored.

Data synthesis A qualitative synthesis provides a tabulation of key characteristics and results. Meta-analysis pools the data across studies to increase the power of statistical analysis. Aims to generate ‘best’ evidence.

Generally narrative and qualitative. Higher levels of subjectivity associated with what is taken from an article for analysis and synthesis. Lack explicit descriptive and thematic analysis. Specific tools and techniques from the field of qualitative data analysis are increasingly applied.

Reporting and Dissemination

Standardized reporting structures used Non- explanatory style adopted. Short scripts recorded and made widely avail able through internationally recognized institutions. Comprehensible by practitioners.

Non-standardized reporting structures. Interpretive long scripts. The explanatory power improved through the use of analogy, metaphor and homology. Process of knowledge production, omitted. Sometimes incomprehensible by practitioners lack links between different literature.

Evidence into practice Collaborative process and practice-oriented. Implementation of evidence is often an afterthought.

Developing Evidence-Informed Management Knowledge 213

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

synthesis than have interpretivists.’ (Noblit and

Hare, 1988, p. 12)

Indeed researchers from an interpretivist or phenomenological position may suggest that systematic reviews, with their positivist leanings, should not be adopted in the social sciences. Even within medical research, not everybody accepts that systematic reviews are necessary or desirable (Petticrew, 2001). Petticrew (2001, p. 98) argues that the concern over systematic review has been fuelled by the fact that they are often presented as synonymous with a numerical aggregation of the results of individual studies through a process of meta-analysis and ‘that they are incapable of dealing with other forms of evidence, such as from non-randomized studies or qualitative re- search’. However, meta-analysis ‘is simply one of the tools, albeit a particularly important one, that is used in preparing systematic reviews’ (Mulrow, Cook and Davidoff, 1997, p. 290). In most systematic reviews the heterogeneity of study data prevents the use of meta-analysis. In these cases, synthesis is achieved through summarizing the findings of a group of studies. Alternative methods of research synthesis such as realist synthesis, meta-synthesis and meta-ethnography have also been developed to draw comparisons and conclusions from a collection of studies through interpretative and inductive methods. Whilst there are fundamental differences between meta-analysis and qualitative research synthesis (Campbell, Pound, Pope, Bitten, Pill, Mogan, and Donovan, 2003), both are concerned with ‘putting together’ (Noblit and Hare, 1988, p. 7) findings from a number of empirical studies in some coherent way (Dingwall, Murphy, Watson, Greatbatch and Parker, 1998). The following section of the paper reports the

systematic review methodology used in medical science, seeks to tease out the key characteristics of the approach, highlights the key challenges in transferring the model to the management field and presents a number of recommendations on how these may be addressed.

Conducting a systematic review

Despite the relative infancy of systematic review, a reasonable consensus has emerged as to its desirable methodological characteristics (Davies

and Crombie, 1998). The Cochrane Collabora- tion’s Cochrane Reviewers’ Handbook (Clarke and Oxman, 2001) and the National Health Service Dissemination (2001) provide a list of stages in conducting systematic review (see Figure 2).

Stage I: planning the review

Prior to beginning the review a review panel is formed encompassing a range of experts in the areas of both methodology and theory. Efforts should be made to include practitioners working in the field on the panel. The review panel should help direct the process through regular meetings and resolve any disputes over the inclusion and exclusion of studies. The initial stages of systematic reviews may be an iterative process of definition, clarification, and refinement (Clarke and Oxman, 2001). Within management it will be necessary to conduct scoping studies to assess the relevance and size of the literature and to delimit the subject area or topic. Such studies need to consider cross-disciplinary perspectives and alternative ways in which a research topic has previously been tackled. The scoping study may also include a brief overview of the theoretical, practical and methodological history debates surrounding the field and sub-fields of

Stage I−Planning the review

Stage II−Conducting a review

Stage III−Reporting and dissemination

Phase 0 - Identification for the need for a review Phase 1 - Preparation of a proposal for a review

Phase 2 - Development of a review protocol

Phase 3 - Identification of research

Phase 4 - Selection of studies

Phase 5 - Study quality assessment

Phase 6 - Data extraction and monitoring progress

Phase 7 - Data synthesis

Phase 8 - The report and recommendations Phase 9 - Getting evidence into practice

Figure 2. Stages of a systematic review (Source: adapted by kind

permission of the publisher from NHS Centre for Reviews and

Dissemination (2001). Undertaking Systematic Reviews of

Research on Effectiveness. CRD’s Guidance for those Carrying

Out or Commissioning Reviews. CRD Report Number 4 (2nd

Edition) r 2001 NHS Centre for Reviews and Dissemination,

University of York.

214 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

study. Where fields comprise of semi-independent and autonomous sub-fields, then this process may prove difficult and the researcher is likely to struggle with the volume of information and the creation of transdisciplinary understanding.

Within medical science the researcher will also arrive at a definitive review question. The review question is critical to systematic review as other aspects of the process flow from it. In systematic review the outcome of these decisions is captured through a formal document called a review protocol. The protocol is a plan that helps to protect objectivity by providing explicit descrip- tions of the steps to be taken. The protocol contains information on the specific questions addressed by the study, the population (or sample) that is the focus of the study, the search strategy for identification of relevant studies, and the criteria for inclusion and exclusion of studies in the review (Davies and Crombie, 1998). Once protocols are complete they are registered with the appropriate review-group editors, such as the Cochrane Collaboration. If satisfactory, the review is published to encourage interested parties to contact the reviewers and to avoid duplication of studies. Any management review protocol may contain

a conceptual discussion of the research problem and a statement of the problem’s significance rather than a defined research question. Further- more management reviews are often regarded as a process of exploration, discovery and develop- ment. Therefore, it is generally considered unac- ceptable to plan the literature-review activities closely. A more flexible approach may make explicit what the researcher intends to do a priori but can be modified through the course of the study. The researcher needs to state explicitly what changes have been made and the rationale for doing so. The aim is to produce a protocol that does not compromise the researcher’s ability to be creative in the literature review process, whilst also ensuring reviews be less open to researcher bias than are the more traditional narrative reviews.

Stage II: conducting the review

A comprehensive, unbiased search is one of the fundamental differences between a traditional narrative review and a systematic review. Although sometimes taking considerable time,

and almost always requiring perseverance and attention to detail, systematic review has been argued to provide the most efficient and high- quality method for identifying and evaluating extensive literatures (Mulrow, 1994). A systema- tic search begins with the identification of keywords and search terms, which are built from the scoping study, the literature and discussions within the review team. The reviewer should then decide on the search strings that are most appropriate for the study. The search strategy should be reported in detail sufficient to ensure that the search could be replicated. Searches should not only be conducted in published journals and listed in bibliographic databases, but also comprise unpublished studies, con- ference proceedings, industry trials, the Internet and even personal requests to known investiga- tors. The output of the information search should be a full listing of articles and papers (core contributions) on which the review will be based. Only studies that meet all the inclusion criteria

specified in the review protocol and which manifest none of the exclusion criteria need be incorporated into the review. The strict criteria used in systematic review are linked to the desire to base reviews on the best-quality evidence. As decisions regarding inclusion and exclusion re- main relatively subjective, this stage of the systematic review might be conducted by more than one reviewer. Disagreements can be resolved within the review panel. The process of selecting studies in systematic review involves several stages. The reviewer will initially conduct a review of all potentially relevant citations identi- fied in the search. Relevant sources will be retrieved for a more detailed evaluation of the full text and from these some will be chosen for the systematic review. The number of sources included and excluded at each stage of the review is documented with the reasons for exclusions. Within the medical domain there is a tension

between the statistical benefits of including a large number of primary studies and conducting high-quality reviews of fewer studies with the use of more selective methodological criteria of inclusion and exclusion (Davies, 2000). Quality assessment refers to the appraisal of a study’s internal validity and the degree to which its design, conduct and analysis have minimized biases or errors. Individual studies in systematic review are judged against a set of predetermined

Developing Evidence-Informed Management Knowledge 215

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

criteria and checklists to assist the process (Ox- man, 1994). The relevance of a study to the review depends on the relevance of its research questions and the quality of its methodology. The reviewer should avoid including:

‘all studies that meet broad standards in terms of

independent and dependent variables, avoiding any

judgement of quality.’ (Slavin, 1986, p. 6)

Systematic reviews, due to their positivistic origins, sit comfortably with studies that use quantitative methods such as randomized con- trolled trials, quasi-experimental designs, and cost-benefit and cost-effectiveness studies, there- fore, establishing criteria for ascertaining what is ‘relevant’ or ‘good quality’ in qualitative research provides a further challenge (Engel and Kuzel, 1992). With qualitative studies there is no possi- bility of testing statistically the significance of the results. Qualitative research, by its very nature:

‘is non-standard, unconfined, and dependent on the

subjective experience of both the researcher and the

researched y it is debatable, therefore, whether an

all-encompassing critical appraisal checklist along

the lines of the User’s Guides to the Medical

Literature could ever be developed’ (Greenhalgh

and Taylor, 1997, p. 741).

Several authors have presented a range of criteria that might be used to appraise and evaluate qualitative studies (Blaxter, 1996; Greenhalgh and Taylor, 1997; Mays and Pope, 2000; Popay, Rogers and Williams, 1998). Popay, Rogers and Williams (1998) suggest that a quality assessment would include the following:

� a primary marker: is the research aiming to explore

the subjective meanings that people give to particular

experiences and interventions?;

� context sensitive: has the research been designed in

such a way as to enable it to be sensitive/flexible to

changes occurring during the study?;

� sampling strategy: has the study sample been selected

in a purposeful way shaped by theory and/or

attention given to the diverse contexts and meanings

that the study is aiming to explore?;

� data quality: are different sources of knowledge/under-

standing about the issues being explored or compared?;

� theoretical adequacy: do researchers make explicit the

process by which they move from data to interpreta-

tion?;

� generalizability: if claims are made to generalizability

do these follow logically and/or theoretically from the

data?

Sandelowski, Docherty and Emden (1997) claim that checklists, when applied to qualitative studies, should be used with caution if they are used as a basis on which to exclude studies from a review. They go on to argue that any decisions regarding exclusion must be supported by a detailed explanation of the reviewer’s conception of ‘good’ and ‘bad’ studies and the reasons for exclusion. Whereas systematic reviews draw upon ‘raw

data’, in management research these data are often not made available in articles by authors. In many cases the articles only represent the results of part studies that satisfy the orientation of the editors of a particular journal. Therefore, the decisions regarding the selection of studies actually become decisions about the selection of ‘articles’ based on the more subjective findings and conclusions of the author rather than on the ‘raw’ data:

‘It is highly unlikely that such a synthesis will

involve a re-analysis of primary data which may be

in the form of transcripts from interviews, for field-

notes from studies involving participant observa-

tion. Rather, the data to be analysed are most likely

to be the findings of the studies involved. These

might take the form of substantive themes arising,

for example, from in-depth interviews. Within

qualitative research (and arguably all research)

theory plays a pivotal role in informing the

interpretation of data. Whilst few authors appear

to have considered the role for theory-led synthesis

of findings across studies an argument can be made

for exploring the potential for this approach.’

(Clarke and Oxman, 2001, section 4, p. 20)

Systematic reviews expose studies to rigorous methodological scrutiny. Within the management field it may be possible to conduct a quality assessment of the research articles by evaluating the fit between research methodology and re- search questions. However, management re- searchers usually rely on the implicit quality rating of a particular journal, rather than formally applying any quality assessment criteria to the articles they include in their reviews (i.e. refereed journals are ‘better’ than practitioner journals). The difficulty in specifying and con- ducting quality assessments of studies is a major challenge in developing a systematic review methodology for management research. To reduce human error and bias, systematic

reviews employ data-extraction forms. These

216 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

often contain general information (title, author, publication details), study features and specific information (details and methods) and notes on emerging themes coupled with details of synth- esis. The Cochrane Collaboration states that data-extraction forms serve at least three im- portant functions. First, the form is directly linked to the formulated review question and the planned assessment of the incorporated studies, providing a visual representation of these. Second, the extraction form acts as a historical record of the decisions made during the process. Third, the data-extraction form is the data-repository from which the analysis will emerge (Clarke and Oxman, 2001). The data-extraction process requires a docu-

mentation of all steps taken. In many cases double extraction processes are employed, where two independent assessors analyse a study and their findings are compared and reconciled if required. Data-extraction can be paper based or computer based. The development of the data- extraction sheets is flexible and may depend upon the nature of the study. When devising the form, reviewers should consider the information that will be needed to construct summary tables and to perform data synthesis. Data-extraction forms should include details of the information source (title, authors, journal, publication details) and any other features of the study such as popula- tion characteristics, context of the study and an evaluation of the study’s methodological quality. Links to other concepts, identification of emer- gent themes, and key results and additional notes also need to be included on the data-extraction form. Research synthesis is the collective term for a

family of methods for summarizing, integrating, and, where possible, cumulating the findings of different studies on a topic or research question (Mulrow, 1994). The simplest and best-known form of research synthesis is a narrative review that attempts to identify what has been written on a subject or topic. Such reviews make no attempt to seek generalization or cumulative knowledge from what is reviewed (Greenhalgh, 1997). Meta-analysis is an alternative approach to synthesis, which enables the pooling of data from individual studies to allow for an increase in statistical power and a more precise estimate of effect size (Glass, 1976). Within management research, few studies address the same research

question and measure the phenomenon in the same way. Furthermore, researchers are less concerned with the effectiveness of certain classes of intervention, and rather more concerned with understanding organizations and management processes. Therefore, it is unlikely that meta- analysis will be appropriate in management research. A number of authors have offered interpretive

and inductive approaches to research synthesis, which are more likely to provide a means of drawing insight from studies and for addressing issues pertinent to management research. Some authors contend that there are a number of philosophical and practical problems associated with ‘summing up’ qualitative studies, whilst others argue that attempts to ‘synthesize existing studies are seen as essential to reaching higher analytic goals and also enhancing the general- izability of qualitative research’ (Sandelowski, Docherty and Emden, 1997, p.367). Two inter- pretive and inductive methods, realist synthesis and meta-synthesis, have been developed to fill the gap between narrative reviews and meta- analysis. For Pawson (2001), realist synthesis offers one

technique for producing a synthesis of a range of study types. He argues that in medical research, programmes (such as medical treatments) carry the potential for change. The aim of a systematic review is to classify these programmes and to conduct a meta-analysis to provide a reliable measure of net effect. The practitioner is invited to replicate the treatment that has worked to maximum effect. In contrast, narrative reviews tend to explain the combination of attributes in a programme and generally identify exemplars of best practice. The practitioner is invited to imitate the programmes that are successful. According to Pawson, it is not programmes that work; rather it is the underlying reasons or resources that they offer subjects that generate change. Whether change occurs is also dependent on the nature of the actors and the circumstances of the programme. Realist synthesis captures a list of vital ingredients or mechanisms (positive or negative) that underpin each individual pro- gramme. The researcher then builds theory by accumulating understanding across a range of programmes. Whilst some scholars would ques- tion whether contingency statements could ever be developed, Pawson (2001) argues that a realist

Developing Evidence-Informed Management Knowledge 217

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

synthesis can provide a transferable programme theory in the form of ‘what works for whom in what circumstances’. Meta-synthesis also offers an interpretative

approach to research synthesis which can be used to identify the:

‘theories, grand narratives, generalizations, or

interpretative translations produced from the in-

tegration or comparison of findings from qualita-

tive studies.’ (Sandelowski, Docherty and Emden,

1997, p. 366)

Unlike meta-analysis, meta-synthesis is not lim- ited to synthesizing strictly comparable studies by constructing ‘interpretations, not analyses, and by revealing the analogies between accounts’(No- blit and Hare, 1988, p. 8). Meta-synthesis provides a means of taking into account:

‘all important similarities and differences in lan-

guage, concepts, images, and other ideas around a

target experience.’ (Sandelowski, Docherty and

Emden, 1997, p.669)

Meta-ethnography is a method of meta-synthesis that offers three alternative techniques for synthesising studies. ‘Refutational synthesis’ can be used when reports give conflicting representa- tions of the same phenomenon, ‘reciprocal translations’ can be used where reports address similar issues and ‘lines of argument synthesis’ can be used if different reports examine different aspects of the same phenomenon. A meta- ethnography is analogous with a grounded theory approach for open coding and identifying categories emerging from the data and by making constant comparisons between individual ac- counts (Beck, 2001). The categories are then linked interpretively to provide a holistic account of the whole phenomenon (Suri, 1999). Many of the techniques of meta-synthesis

remain ‘either relatively untried and undeve- loped, and/or difficult to codify and understand’ (Sandelowski, Docherty and Emden, 1997, p. 369). However, both realist synthesis and meta- synthesis challenge the positivistic orthodoxy that surrounds contemporary approaches to research reviews, demonstrating that a synthesis can be an interpretive, inductive, hermeneutic and eclectic process (Jensen and Alien, 1996). Whilst meta- synthesis and realist synthesis approaches are fundamentally different to systematic reviews and in particular meta-analysis, they both share a

desire to improve upon traditional narrative reviews by adopting explicit and rigorous pro- cesses and by:

‘the bringing together of findings on a chosen

theme, the results of which should be to achieve a

greater level of understanding and attain a level of

conceptual or theoretical development beyond that

achieved in any individual empirical study. (Camp-

bell et al., 2002, p. 2)

As in systematic reviews, the aim of realist syntheses and meta-syntheses is to ‘have impact’ by being ‘presented in an accessible and usable form in the real world of practice and policy making’ (Sandelowski, Docherty and Emden, 1971, p. 365).

Stage III: reporting and dissemination

A good systematic review should make it easier for the practitioner to understand the research by synthesizing extensive primary research papers from which it was derived. Within management research a two-stage report might be produced. The first would provide full (rough-cut and detailed) ‘descriptive analysis’ of the field. This is achieved using a very simple set of categories with the use of the extraction forms. For example, who are the authors, how many of the core contributions are from the USA, how many are European? What is the age profile of the articles? Can the fields be divided into epochs in terms of volume of orientation of study? Do simple categories divide up the field? For example, can the field be divided sectorally? By gender? Or simple categories ‘borrowed’ from associated cognate disciplines such as psychology or sociology (interpretivist versus positivistic or behavioural versus cognitive studies, for exam- ple). The researcher should be able to provide a broad ranging descriptive account of the field with specific exemplars and an audit trail, justifying his/her conclusions. Researchers also need to report the findings of

a ‘thematic analysis’, whether or not the results were derived through an aggregative or inter- pretative approach, outlining that which is known and established already from data- extraction forms of the core contributions. They may wish to focus on the extent to which consensus is shared across various themes. They may also want to identify key emerging themes

218 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

and research questions. Whatever overarching categories are chosen for the tabulation, research- ers should again provide a detailed audit trail back to the core contributions to justify and ground their conclusions. Linking themes across the various core contributions wherever possible and highlighting such links is an important part of the reporting process. Systematic review provides a means for practi-

tioners to use the evidence provided by research to inform their decisions. However, turning the conclusions from systematic reviews into guidelines for practice has been a challenge in medicine (Macdonald, 1999), as ‘clinicians reason about individual patients on the basis of analogy, experience, heuristics, and theory, as well as evidence’ (Cook, Mulrow and Haynes, 1997, p. 380). Decision-makers are likely, and should be encouraged, to use personal experience and pro- blem-solving skills rather than relying solely on the results of systematic reviews (Bero and Rennie, 1995; Rosenberg and Donald, 1995). Within management there is a need to recognize that evidence alone is often insufficient and incomplete, only informing decision-making by bounding available options. Therefore, the terms ‘evidence informed’ or even ‘evidence aware’, rather than ‘evidence based’ (Nutley, Davies and Walter, 2002; Nutley and Davies, 2002), may be more appro- priate in the management field, and the former has influenced our choice of title for this paper. Improving the translation of research evidence

into practice is not unproblematic as the ‘rela- tionships between research, knowledge, policy and practice are always likely to remain loose, shifting and contingent’ (Nutley and Davies, 2002, p. 11). For evidence-informed practice to be achieved, strategies need to be developed which encourage the uptake and utilization of evidence that move beyond the simple construc- tion and dissemination of the research base (Nutley and Davies, 2000). Encouraging practi- tioners to set specific questions for reviews and to engage in the process may help in developing a ‘context sensitive’ science (Nowotny, Scott and Gibbons, 2001) which may help to blur the boundaries between science, policy and practice. Increasing the precision of a reliable evidence base in order that policymakers and practitioners can make more sensitive judgements is the ultimate aim of the application of systematic review procedures to management research.

Conclusions

This paper began by arguing that reviews of existing research evidence in the management field lack both rigour and relevance. Anderson, Herriot and Hodgkinson (2001) offer a four-fold characterization of applied social science. They term research that is low on rigour but high on relevance ‘Popularist Science’. In contrast, ‘Ped- antic Science’ is high on rigour but low on relevance, whereas ‘puerile Science’ is neither rigorous nor relevant. Only ‘Pragmatic Science’ balances both rigour and relevance (see Figure 3). They acknowledge that the pursuit of ‘prag-

matic’ research:

‘that genuinely bears the hallmarks of scientific

rigour (irrespective of whether it be quantitative

and/or qualitative in nature), but which also

engages a wider body of stakeholders in the

knowledge production process, presents a set of

formidable challenges for the management research

community at this juncture.’ (Hodgkinson, Herriot

and Anderson, 2001, p. S46)

This paper has outlined the opportunities and challenges in applying ideas and methods devel-

Low

Low

High High

Practical relevance

Theoretical and methodological rigour

Quadrant 1: Quadrant 2:

Quadrant 4:Quadrant 3:

Popularist Pragmatic

PedanticPuerile Science Science

ScienceScience

,,

,,

,

,

,

,

Figure 3. A four-fold typology of research in industrial, work and

organizational psychology. Source: adapted by G. P. Hodgkin-

son, P. Herriot and N. Anderson (2001), British Journal of

Management, 12 (Special Issue), page S42, from N. Anderson,

P. Herriot and G. P. Hodgkinson, ‘The practitioner-researcher

divide in industrial, work and organizational (IWO) psychology:

where are we now, and where do we go from here?’, Journal of

Occupational and Organizational Psychology, 74, pp. 391–411.

r 2001 The British Psychological Society and the British

Academy of Management. Reproduced by kind permission of

both publishers.

Developing Evidence-Informed Management Knowledge 219

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

oped in medical science to the field of manage- ment, with the aim of further developing and enhancing the quality of management reviews and ensuring that they are practitioner and context sensitive. The aim of systematic review is to provide collective insights through theore- tical synthesis into fields and sub-fields. For academics, the reviewing process increases meth- odological rigour. For practitioners/managers, systematic review helps develop a reliable knowl- edge base by accumulating knowledge from a range of studies. In so doing the researcher may be able to develop a set of ‘field tested and grounded technological rules’ (Van Aken, 2001, p. 1). In this sense, systematic review can be argued to lie at the heart of a ‘pragmatic’ management research, which aims to serve both academic and practitioner communities.

References

Anderson, N., P. Herriot and G. P. Hodgkinson (2001). ‘The

Practitioner-Researcher Divide in Industrial, Work and

Organizational (IWO) Psychology: Where Are We Now,

and Where Do We Go From Here?’, Journal of Occupational

and Organizational Psychology, 74 (4), pp. 391–411.

Aram, J. D. and J. P. F. Salipante (2000), ‘Applied Research in

Management: Criteria for Management Educators and for

Practitioner-Scholars’, US Academy of Management Confer-

enceFMultiple Perspectives on Learning in Management

Education, Toronto.

Becher, A. (1989). Academic Tribes and Territories: Intellectual

Enquiry and the Cultures of Disciplines, The Society for

Research into Higher Education and the Open University

Press, Milton Keynes.

Beck, C. T. (2001). ‘Caring with nursing education: A meta-

synthesis’. Journal of Nursing Education, 40 (3), pp. 101–110.

Bero, L. and D. Rennie (1995). ‘The Cochrane Collaboration:

Preparing, Maintaining and Disseminating Systematic Re-

views of the Effects of Health Care’, Journal of the American

Medical Association, 274 (1), pp. 1935–1938.

Berry, M. (1995). ‘Research and the Practice of Management: A

French View’, Organizational Science, 6 (2), pp. 104–116.

Blake, R. R. and J. S. Mouton (1976). Consultation. Addison-

Wesley Publishing, Reading, MA.

Blaxter, M. (1996). ‘Criteria for the Evaluation of Qualitative

Research Papers’, Medical Sociology News, 22 (1), pp. 68–71.

Campbell, R., P. Pound, C. Pope, N. Britten, R. Pill, M. Morgan,

and J. Donovan (2003). ‘Evaluating meta-ethnography:

a synthesis of qualitative research on lay experiences of

diabetes and diabetes care’. Social Science and Medicine,

56 (4), pp. 671–684.

Campbell Collaboration (2001). http://campbell.gse.upenn.edu/

about.htm.

Clarke, M. and A. D. Oxman (Eds) (2001). Cochrane Reviewers’

Handbook 4.1.4 [updated October 2001], The Cochrane

Library, Oxford.

Cochrane Collaboration (2001), The Cochrane Brochure, http://

www.cochrane.org/cochrane/cc-broch.htm#BDL

Cook, D. J., N. L. Greengold, A. G. Ellrodt, and S. R.

Weingarten (1997). ‘The Relation Between Systematic Re-

views and Practice Guidelines’, Annals of Internal Medicine,

127 (3) August, pp. 210–216.

Cook, D. J., C. D. Mulrow, and R. B. Haynes (1997).

‘Systematic Reviews: Synthesis of Best Evidence for Clinical

Decisions’, Annals of Internal Medicine, 126 (5) March, pp.

376–380.

Dabinett, G., P. Lawless, J. Rhodes, and P. Tyler (2001). A

Review of the Evidence Base for Regeneration Policy and

Practice, Department of the Environment Transport and the

Regions.

Davies, H. T. O. and I. K. Crombie (1998). ‘Getting to Grips

with Systematic Reviews and Meta-Analyses’, Hospital

Medicine, 59 (12), pp. 955–958.

Davies, H. T. O. and S. M. Nutley (1999). ‘The Rise and Rise of

Evidence in Health Care’, Public Money & Management, 19

(1), pp. 9–16.

Davies, H. T. O., S. M. Nutley and P. C. Smith (2000).

‘Editorial: What Works? The Role of Evidence in Public

Sector Policy and Practice’, Public Money & Management, 19

(1), pp. 3–5.

Davies, H. T. O., S. M. Nutley and N, Tilley (1999). ‘Editorial:

Getting Research into Practice, Public Money & Manage-

ment, 20(4), pp. 17–22.

Davies, P. (2000). The Relevance of Systematic Reviews to

Educational Policy and Practice, http://www.jiscmail.ac.uk/

files/BEME/oxreview.htm.

Dingwall R., E. A. Murphy, P. Watson, D. Greatbatch and S.

Parker (1998). ‘Catching Goldfish: Quality in Qualitative

Research’, Journal of Health Services Research and Policy, 3

(3), pp. 167–172.

Economic and Social Research Council (1999). A History of the

EPSRC UK Centre for Evidence Based Policy and Practice,

http://www.evidencenetwork.org/home.asp.

Engel, J. D. and A. J. Kuzel (1992). ‘On the idea of what

constitutes good qualitative inquiry’, Qualitative Health

Research, 2, pp. 504–510.

Estabrooks, C. A., P. A. Field, and J. M. Morse (1994).

‘Aggregating qualitative findings: An approach to theory

development’, Qualitative Health Research, 4, pp. 503–511.

Evans, D. and A. Pearson (2001). ‘Systematic Reviews:

Gatekeepers of Nursing Knowledge’, Journal of Clinical

Nursing, 10 (5), pp. 593–599.

Fink, A. (1998). Conducting Research Literature Reviews: From

Paper to the Internet, Sage Publications, London.

Friedman A., C. Durkin, M. Phillips, and E. Voltsinger (2000).

The Future of UK Professional Associations, Proceedings for

5th Annual Professional Association Research Network

Conference.

Furniss, J. S. and M. Nutley (2000). ‘Implementing

What Works with OffendersFThe Effective Practice

Initiative’, Public Money & Management, 20 (4), pp.

23–28.

Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, P.

Scott and M. Trow (1994). The New Production of Knowl-

edge: The Dynamics of Science and Research in Contemporary

Societies, Sage Publications, London.

Glass, G. V. (1976). ‘Primary, Secondary, and Meta-analysis of

Research’, Educational Researcher, 5 (2), pp. 3–8.

220 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

Greenhalgh, T. (1997), ‘Papers that Summarise Other Papers

(Systematic Reviews and Meta-analyses)’, British Medical

Journal, 315 (7109), pp. 672–675.

Greenhalgh, T. and R. Taylor (1997). ‘Papers that go Beyond

Numbers (Qualitative Research)’, British Medical Journal,

315 (7110), pp. 740–743.

Halladay, M. and L. Bero (2000). ‘Implementing Evidence-

based Practice in Health Care’, Public Money &Management,

20 (4), pp. 43–50.

Hambrick, D. C. (1994). ‘What if the Academy Actually

Mattered?’, Academy of Management Review, 19 (1), pp. 11.

Hart, C. (1998). Doing a Literature Review: Releasing the Social

Science Research Imagination, Sage Publications, London.

HM Inspectorate of Probation (1998). Strategies for Effective

Offended Supervision. Report of the HMIP What Works

Project, Home Office, London.

Hodgkinson, G. P. (Ed.) (2001). Facing the future: the nature

and purpose of management research re-assessed, British

Journal of Management, 12 (Special Issue), pp. S1–S80.

Hodgkinson, G. P., P. Herriot and N. Anderson (2001). ‘Re-

aligning the Stakeholders in Management Research: Lessons

from Industrial, Work and Organizational Psychology’,

British Journal of Management, 12 (Special Issue), pp.

S41–S48.

Home Office (1998), Reducing Offending: An Assessment of

Research Evidence on Ways of Dealing with Offending

Behaviour, Home Office Research and Statistics Directorate,

London.

Huff, A. S. (2000). ‘Changes in Organizational Knowledge

Production’, Academy of Management Review, 25 (2), pp.

288–293.

Jensen, L. A., and M. N. Alien (1996). ‘Meta-synthesis of

qualitative findings’. Qualitative Health Research, 6 (4), pp.

553–60.

Laycock, G. (2000), ‘From Central Research to Local Practice:

Identifying and Addressing Repeat Victimization’, Public

Money & Management, 19 (1) pp. 17–22.

Macdonald, G. (1999), ‘Evidence-based Social Care: Wheels off

the Runway?’, Public Money & Management, 19 (1), pp.

25–32.

Maclennan, D. and A. More (1999). ‘Evidence, What Evidence?

The Foundations for Housing Policy’, Public Money &

Management, 19 (1), pp. 17–24.

Mays, N. and C. Pope (2000). ‘Assessing Quality in Quali-

tative Research’, British Medical Journal, 320 (January), pp.

50–52.

Mulrow, C. D. (1987). ‘The Medical Review Article: State of

the Science’, Annual International Medicine, 106, pp. 485–488.

Mulrow, C. D. (1994). ‘Systematic Reviews–Rationale for

Systematic Reviews’, British Medical Journal, 309 (6954), pp.

597–599.

Mulrow, C. D., D. J. Cook, and F. Davidoff (1997). ‘Systematic

reviews: Critical links in the great chain of evidence’. Annals

of Internal Medicine, 126, pp. 389–391.

National Institute for Clinical Excellence (2001), http://

www.nice.org.uk/.

NHS Centre for Reviews and Dissemination (2001), Under-

taking Systematic Reviews of Research on Effectiveness.

CRD’s Guidance for those Carrying Out or Commissioning

Reviews. CRD Report Number 4 (2nd Edition). York.

Noblit, G. W. and R. D. Hare (1988). Meta-ethnography:

Synthesizing Qualitative Studies, Sage Publications, London.

Nowotny, H., P. Scott and M. Gibbons (2001), Rethinking

Science: Knowledge and the Public in an Age of Uncertainty.

Blackwell Publishers, Maiden, MA.

Nutley, S. M. and H. T. O. Davies (2000). ‘Making a Reality

of Evidence Based Practice: Some Lessons from the Diffusion

of Innovations’, Public Money & Management, 20 (4), pp.

35–42.

Nutley S. M., H. T. O. Davies and I. Walter (2002). From

Knowing to Doing: A Framework for Understanding the

Evidence-into-practice Agenda, Discussion Paper 1, Research

Unit for Research Utilisation. http://www.st-and.ac.uk/

Bcppm/KnowDo%20paper.pdf.

Nutley, S. M. and H. T. O. Davies (2002). Evidence-based Policy

& Practice: Moving from Rhetoric to Reality, Discussion Paper

2, Research Unit for Research Utilisation, http://www.st-

and.ac.uk/Bcppm/Rhetoric%20to%20reality%20NF.pdf.

Ohlsson, A. (1994). ‘Systematic ReviewsFTheory and Prac-

tice’, Scandinavian Journal of Clinical & Laboratory Investi-

gation, 54 (219), pp. 25–32.

Oxman, A. D, (1994). ‘Systematic ReviewsFChecklists for

Review Articles’, British Medical Journal, 309 (6955), pp.

648–651.

Pawson, R. (2001). The Promise of a Realist Synthesis,Working

Paper No.4, ESRC Evidence Network, Centre for Evidence

Based Policy and Practice. http://www.evidencenetwork.org/

Documents/wp4.pdf

Peckham, M. (1991). ‘Research and Development for the

National Health Service’, Lancet, 338, pp. 367–371.

Petticrew, M. (2001). ‘Systematic reviews from astronomy to

zoology: myths and misconceptions’, British Medical Journal

322 (13) January, pp. 98–101.

Pettigrew, A. M. (1997). The Double Hurdles for Management

Research, Advancement in Organizational Behaviour, Ash-

gate, Aldershot.

Pfeffer, J. and R. I. Sutton (1999). ‘Knowing ‘‘What’’ to Do is

Not Enough: Turning Knowledge into Action’, California

Management Review, 42 (1), pp. 83–108.

Popay, J., A. Rogers and G. Williams (1998). ’Rationale and

Standards for the Systematic Review of Qualitative Litera-

ture in Health Services Research’, Qualitative Health

Research, 8 (3), pp. 341–351.

Rosenberg, W. and W. Donald (1995). ‘Evidence Based

Medicine: An Approach to Clinical Problem-Solving’, British

Medical Journal, 310 (6987), pp. 1122–1126.

Sandelowski, M., S. Docherty, and C. Emden (1997). ‘Quali-

tative Metasynthesis: Issues and Techniques’. Research in

Nursing and Health, 20 (4), pp. 365–371.

Smith, R. (1991), ‘Where is the Wisdom.? The poverty of

medical evidence’ [editorial], British Medical Journal, 303, pp.

789–799.

Slavin, R. E. (1986). ’Best-evidence Synthesis: An Alternative to

Meta-analytic and Traditional Reviews’, Educational Re-

searcher, 15 (9), pp. 5–11.

Starkey, K. and P. Madan (2001). ‘Bridging the Relevance Gap:

Aligning Stakeholders in the Future of Management

Research’, British Journal of Management, 12 (S1), pp. 3–26.

Suri, H. (1999). ‘The process of synthesising qualitative

research: A case study’, Annual Conference of the Association

for Qualitative Research, Melbourne, http://www.latrobe.e-

du.au/aqr/offer/papers/HSuri.htm

Tisdall, P. (1982). Agents of Change; The Development and

Practice of Management Consultancy, Heinemann, London

Developing Evidence-Informed Management Knowledge 221

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense

Tranfield, D. and K. Starkey (1998). ‘The Nature, Social

Organization and Promotion of Management Research:

Towards Policy’ British Journal of Management, 9 (4), pp.

341–353.

Van de Ven, A. H. (1998). Professional Science for a

Professional School, Breaking the Code of Change Con-

ference, Harvard Business School, Boston, MA.

Van Aken, J. (2001). Management Research Based on the

Paradigm of the Design Sciences: The Quest for Field Tested

and Grounded Technological Rules, Working Paper 01.1,

Eindhoven Centre for Innovation Studies, Eindhoven Uni-

versity of Technology, Eindhoven.

Whitley, R. (1984a), ‘The Fragmented State of Management

Studies: Reasons and Consequences’, Journal of Management

Studies, 21 (3), pp. 331–348.

Whitley, R. (1984b), ‘The Scientific Status of Management

Research as a Practically-oriented Social Science’, Journal of

Management Studies, 21 (4), pp. 369–390.

Whitely, R. (2000), The Intellectual and Social Organization of

the Sciences, Second Edition, Oxford University Press,

Oxford.

Wilcoxson, L. and E. P. Fitzgerald (2001). The Nature and Role

of Management Research in Australia and New Zealand,

Australian and New Zealand Academy Of Management

Conference.

Wind, J. and P. Nueno (1998). The Impact Imperative: Closing

the Relevance Gap of Academic Management Research,

International Academy of Management North America

Meeting, New York.

Wolf, F. M., J. A. Shea and M. A. Albanese (2001), ‘Toward

Setting a Research Agenda for Systematic Reviews of

Evidence of the Effects of Medical Education’, Teaching

and Learning in Medicine, 13 (1), pp. 54–60.

222 D. Tranfield, D. Denyer and P. Smart

14678551, 2003, 3, D ow

nloaded from https://onlinelibrary.w

iley.com /doi/10.1111/1467-8551.00375 by A

rizona State U niversity A

cq & A

nalysis, L ib C

ontinuations, W iley O

nline L ibrary on [21/02/2023]. See the T

erm s and C

onditions (https://onlinelibrary.w iley.com

/term s-and-conditions) on W

iley O nline L

ibrary for rules of use; O A

articles are governed by the applicable C reative C

om m

ons L icense