Exceptional proff only 2

To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty 399

Public Administration Review,

Vol. 76, Iss. 3, pp. 399–402. © 2016 by

The American Society for Public Administration.

DOI: 10.1111/puar.12555.

Adam Wellstead is associate professor

of environmental and energy policy in the

Department of Social Sciences at Michigan

Technological University in Houghton,

Michigan. His research interests include

climate change adaptation, policy capacity,

historical institutionalism, survey design,

and policy causal mechanisms. Before

entering the world of academia in 2011,

he was a civil servant with the Canadian

federal government for 16 years.

E-mail: [email protected]

Kathryn Oliver is departmental lecturer

in evidence-based social intervention and

policy evaluation at University of Oxford,

United Kingdom. She is a social scientist

with an interest in the use of evidence

in policy making, research impact, and

science policy.

E-mail: [email protected]

Paul Cairney is professor of politics and

public policy at the University of Stirling,

United Kingdom. His research interests

include comparing theories on evidence

and policy ( The Politics of Evidence-Based Policymaking, 2016, and Understanding Public Policy, 2012) and the use of evidence to promote tobacco control ( Global Tobacco Control, 2012, with Donley T. Studlar and Hadii M. Mamudu), “prevention” policies,

and fracking. His blog on public policy can

be found at https://paulcairney.wordpress.

com/.

E-mail: [email protected]

Evidence in Public Administration

Abstract : Policy makers cannot consider all evidence relevant to policy. They use two shortcuts—emotions and beliefs to understand problems and “rational” ways of establishing the best evidence on solutions—to act quickly in complex, multilevel policy-making environments. Many studies only address one part of this problem. Improving the supply of evidence helps reduce scientific and policy maker uncertainty. However, policy makers also combine their beliefs with limited evidence to reduce ambiguity by choosing one of several possible ways to understand and solve a problem. We use this insight to consider solutions designed to “close the evidence–policy gap.”

Practitioner Points • The natural science literature on how to maximize the use of evidence in policy should learn from studies

of public policy and administration. • Lesson one: Identify “where the action is” in multilevel policy-making environments. • Lesson two: Supplement the use of evidence to reduce policy maker uncertainty with persuasion to reduce

ambiguity.

Paul Cairney University of Stirling , United Kingdom

Kathryn Oliver University of Oxford , United Kingdom

Adam Wellstead Michigan Technological University

To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty

There is now a large literature on the gaps between the production of scientific evidence and a policy-making response. However, in disciplines such as health and environmental sciences, most studies do not use insights from policy theory, public administration, or organizational studies to explain the gap. Focusing on policy theory, we explain why this matters by identifying the difference between empirical uncertainty and policy ambiguity. Both concepts relate to “bounded rationality”: policy makers do not have the ability to consider all evidence relevant to policy problems. Instead, they employ two shortcuts: “rational,” by pursuing clear goals and prioritizing certain kinds and sources of information, and “irrational,” by drawing on emotions, gut feelings, deeply held beliefs, and habits to make decisions quickly. This takes place in an unpredictable policy- making environment in which attention can lurch from issue to issue, policy is made routinely in networks, and the “rules of the game” take time to learn.

The key problem with many studies is that they focus only on the first shortcut. They only identify the problem of uncertainty and incomplete information and seek to solve it by creating hierarchies of evidence, improving the supply of information to policy makers, and encouraging academic–practitioner workshops. They ignore the wider process of debate, coalition

formation, and persuasion to reduce ambiguity by establishing a dominant way to “frame” policy problems. The latter can determine the demand for scientific evidence. Therefore, an additional solution for advocates of scientific evidence is to engage in a process of persuasion, for example, by forming coalitions with allies with the same aims or beliefs and working with them to combine evidence with simple stories to exploit the emotional and ideological biases of policy makers.

We demonstrate this argument in three steps. First, we outline two systematic reviews of the literature on “barriers” to evidence and policy. Most studies present solutions to reduce uncertainty. Second, we derive key insights from a wide range of policy concepts and theories (hereafter “policy theory”) to show the effect of unpredictable policy environments on the use of evidence and, therefore, the importance of ambiguity and persuasion. Third, our conclusion explores the implications for the role of science and experts in policy making. We identify the practical, normative, and ethical consequences for actors trying to maximize the uptake of scientific evidence within government.

Limited Insights from Existing Studies: Improve the Evidence, Reduce Uncertainty We use the examples of health and environmental sciences, in which scientific advocates express

400 Public Administration Review • May | June 2016

frustration at their limited ability to promote evidence-based solutions to urgent policy problems. The most frequently reported barriers relate to the lack of time, support, resources, and incentives for scientists to disseminate high-quality information effectively (Cairney 2016 ; Oliver et al. 2014). Studies also identify the low policy relevance in academic research and minimal academic knowledge about how to be strategic, such as to identify relevant policy makers and opportunities for timely engagement (Stringer and Dougill 2013 ). They highlight differences in academic and political “cultures”: language and jargon, longer scientific timescales, low incentives to engage, differing perceptions of scientific knowledge, and the relative need for scientists to challenge evidence (to ensure it is robust) but for policy makers to generate an image of policy certainty and reconcile evidence with well-established beliefs. There is also a perception that policy makers rely on personal experience, ad hoc links with experts, people they know and trust, and simple decision-making techniques and stories rather than the state-of-the-art in scientific research and sophisticated modeling systems (Lomas and Brown 2009 , 906).

The proposed solutions to this cultural gap are very similar in health and environmental studies: improve the evidence base on policy solutions; “package” the evidence well to make it easy to understand, framed in a way that is attractive to policy makers, and/ or accompanied by realistic expectations for policy change; foster networks with policy makers and encourage academic–practitioner workshops; use intermediaries such as “knowledge brokers”; recognize the importance of timing; and encourage policy maker skills or better government understanding of problems (Cvitanovic et al. 2015 ).

Very few of these studies draw on policy theory or other sources of knowledge of the policy process such as public administration or organizational studies. Most are based on personal experiences, producing incomplete findings because the evidence is limited to interviewees with minimal knowledge of policy making. Or they draw on outmoded understandings of policy making, including policy cycles, containing a core group of policy makers at the “center” and clearly defined and well-ordered stages, and the sense that advances in scientific evidence gathering can minimize the problem of “bounded rationality” (Cairney 2016 ). Such discussions suggest that scientists can improve the use of evidence in policy in a relatively straightforward way by identifying a core group of policy makers and using the best scientific information to reduce policy maker uncertainty.

Lessons from Policy Theory Identify Multilevel Policy-Making Environments Policy theory helps us fill two key gaps in such analysis. First, policy making takes place in less ordered and predictable policy environments, exhibiting the following:

• A wide range of actors (individuals and organizations) influencing policy at many levels of government

• A proliferation of rules and norms followed by different levels or types of government

• Close relationships (“networks”) between policy makers and powerful actors

• A tendency for certain beliefs or “paradigms” to dominate discussion

• Shifting policy conditions and events that can prompt policy maker attention to lurch at short notice

A focus on this bigger picture shifts our attention from the use of scientific evidence by an elite group of elected policy makers at the “top” to its use by a wide range of influential actors in a multilevel policy process. It shows scientists that they are competing with many actors to present evidence in a particular way to secure a policy maker audience. Support for particular solutions varies according to which organization takes the lead and how it understands the problem. Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favor particular sources of evidence and some participants over others, and there is a language—indicating what ways of thinking are in good “currency” (such as “value for money”)—that takes time to learn. Well-established beliefs provide the context for policy making: new evidence on the effectiveness of a policy solution has to be accompanied by a shift of attention and successful persuasion. In some cases, social or economic “crises” can prompt lurches of attention from one issue to another, and some forms of evidence can be used to encourage that shift.

In this context, too many studies analyze, for example, a singular point of central government decision rather than the longer-term process (Oliver, Lorenc, and Innvær 2014, 10). Overcoming barriers to influence in that small part of the process will not provide an overall solution.

Use Evidence and Persuasion to Reduce Ambiguity Significant advances in scientific practice, knowledge, and systematic review address only one aspect of bounded rationality: the reduction of uncertainty (Botterill and Hindmoor 2012 , 371). They do not address others, such as the inability to completely separate empirical facts from human values and ways of thinking. All individuals combine goal-driven and evidence-informed strategies with moral or emotional judgments based on their well-established beliefs and informational shortcuts based on their familiarity with information and sources of information (Alter and Oppenheimer 2009 ; Haidt 2001 , 818; Kahneman 2012 , 20; Schneider, Ingram, and deLeon 2014 ).

Consequently, the focus of policy theory is on the links between evidence, persuasion, and framing (in the wider context of a tendency for certain beliefs to dominate discussion). Framing refers to the ways in which actors understand, portray, and categorize issues (Cairney 2012 , 185). Problems are multifaceted, but bounded rationality limits the attention of policy makers, and actors compete to highlight one image at the expense of others. The outcome of this process determines who is involved (for example, portraying an issue as technical limits involvement to experts) and responsible for policy, how much attention they pay, and what kind of solution they favor. For example, tobacco control is more likely when policy makers view it primarily as a public health epidemic rather than an economic good, while fracking policy depends on its primary image as a new energy boom or environmental disaster.

Scientific evidence plays a part in this process, but we should not exaggerate the ability of scientists to win the day with reference to evidence. Rather, policy theory signals the strategies that

To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty 401

practitioners may have to adopt to increase demand for their evidence:

• Combine facts with emotional appeals to prompt lurches of policy maker attention from one policy image to another (True, Jones, and Baumgartner 2007 , 161).

• Tell simple stories that are easy to understand, help manipulate people ’ s biases, apportion praise and blame, and highlight the moral and political value of solutions (Jones, Shanahan, and McBeth 2014 ).

• Interpret new evidence through the lens of the preexisting beliefs of actors within coalitions, some of which dominate policy networks (Weible, Heikkila, and Sabatier 2012 ).

• Produce a policy solution that is feasible and exploit a time when policy makers have the opportunity to adopt it (Kingdon 1984 ).

Further, the impact of a framing strategy is not necessarily immediate even if it is successful. Scientific evidence may prompt a lurch of attention to a policy problem, prompting a shift of views in one venue or the new involvement of actors from other venues. However, it can take years to produce support for an “evidence- based” policy solution built on its technical and political feasibility (will it work as intended, and do policy makers have the motive and opportunity to select it?).

Concluding Discussion: Key Strategies for Advocates of Evidence-Based Policy Making Policy making takes place in an often unpredictable environment in which many actors make and influence policy, using “rational” and “irrational” shortcuts to make decisions quickly. From policy theory, we can identify pragmatic ways to adapt to, and engage in, this process: identify where the action is (in subsystems and venues in several levels of government); learn and follow the “rules of the game” of subsystems to improve strategies and help build up trust; form coalitions with actors with similar aims and beliefs; and frame the evidence to appeal to the biases, beliefs, and priorities of policy makers (Cairney 2016 ; Stoker 2010 , 55–57; Weible, Heikkila, and Sabatier 2012 ). Meaningful influence requires a long-term strategy to form alliances and develop knowledge of policy makers and policy making, not a one-off dissemination of state-of-the-art knowledge.

Consequently, scientists have a difficult choice to make between the need to specialize and maintain an image of academic objectivity or to go beyond their comfort zone, and expertise, to engage in an advocacy enterprise to increase impact. Without such advocacy, policy makers will rely on highly biased and persuasive stories from vested interests rather than struggle through inaccessible and equivocal scientific papers.

Of course, there is a huge difference between identifying this advice and expecting practitioners such as scientists to be in a position to follow it. Instead, we offer four pragmatic suggestions. First, the existing literature is correct to promote networks between communities of scientists and policy makers. Networks allow both sides to learn to appreciate the constraints under which they, and the other camp, operate. Academics must be open to conversations about the political value and risks of the information they generate. Policy makers must acknowledge that being explicit about the scientific uncertainty inherent in any study is absolutely essential

to the intellectual credibility of individual academics and academic work. While policy makers can be frustrated by the reluctance of academics to make unequivocal policy recommendations, we need respectful conversation which allows political judgment to be truly informed by evidence, without academics feeling undue pressure to kowtow or be silenced. Further, academics must come to feel comfortable that their advice and expertise is valued even if the knowledge being shared is months or years old; a lifetime of expertise can be as valuable to a policy maker as a brand-new study. For academics, this should soften the apparent choice between advocacy and impact, on the one hand, and isolationist but credible academic activity, on the other. What is required is a thoughtful dialogue about the responsibilities each set of actors have to themselves, their practices, and their constituencies, whether voters or research participants. Both sides have much to learn about and from each other, and working more closely together can help both sides deliver more relevant, useful, and effective policy.

Second, there may be ways to improve these links by overcoming barriers to engagement. There is much to be gained from closer collaboration based on new incentives for both groups. Creating opportunities which are rewarded (e.g., by promotions) for “coproduction” of policy or policy-relevant research would be an obvious way to enable coalition building. Academics would be enabled to do more policy-relevant research if they were more aware of the upcoming policy agendas; they would be able to present their research to the right audiences if this (and training) were enabled by funders and valued by universities and if more time were made available to both camps to allow conversations, even without specific policy targets in mind. Reducing the current waste of time and resources in the scientific funding models we currently use would be one way of resourcing these conversations.

Third, in some fields there is a tendency to use generalist intermediaries or “knowledge brokers,” and this may be an economic solution even if the evidence of their effectiveness is mixed. What seems to matter is the ability to know who to ask which question, and having a sufficiently good and ongoing relationship to be able to call on academic or policy experts when needed (Oliver, Money, and de Vocht 2013 ).

Finally, it is important to draw insights from policy studies to help us recognize the profound limits to each of these strategies and to relate them to nascent policy agendas on measuring the practical value of academic research. There is an increasing focus, particularly in countries such as the United Kingdom, on demonstrating academic “impact” on policy and policy making. This involves a linear and short-term process of engagement and impact that can be described simply enough in University reports to justify economic rewards for academic activity. Yet our discussion shows how shortsighted and box ticking such exercises may become. Meaningful policy impact built on academic–policy maker relationships take time and effort to create and maintain. It cannot simply be bought, outsourced, or produced during ad hoc workshops. Further, in a complex policy-making system, it makes little sense to pinpoint discrete examples of academic influence. There are ways to produce meaningful academic–policy maker engagement, but we should not exaggerate its impact or our ability to measure it in a simple way.

402 Public Administration Review • May | June 2016

References Alter , Adam , and Daniel Oppenheimer . 2009 . Uniting the Tribes of Fluency to

Form a Metacognitive Nation . Personality and Social Psychology Review 13 ( 3 ): 219 – 35 .

Botterill , Linda , and Andrew Hindmoor . 2012 . Turtles All the Way Down: Bounded Rationality in an Evidence-Based Age . Policy Studies 33 ( 5 ): 367 – 79 .

Cairney , Paul . 2012 . Understanding Public Policy: Theory and Issues . Basingstoke, UK : Palgrave Macmillan .

——— . 2016 . The Politics of Evidence-Based Policymaking . London : Palgrave Pivot . Cvitanovic , Christopher , Alistair Hobday , Shaun Wilson , Kirsten Dobbs , and

Nadine Marshall . 2015 . Improving Knowledge Exchange among Scientists and Decision-Makers to Facilitate the Adaptive Governance of Marine Resources: A Review of Knowledge and Research Needs . Ocean and Coastal Management 112 : 25 – 35 .

Haidt , Jonathan . 2001 . The Emotional Dog and Its Rational Tail: A Social Intuitionist Approach to Moral Judgment . Psychological Review 108 ( 4 ): 814 – 34 .

Jones , Michael D. , Elizabeth A. Shanahan , and Mark K. McBeth , eds. 2014 . The Science of Stories: Applications of the Narrative Policy Framework in Public Policy Analysis . New York : Palgrave Macmillan .

Kahneman , Daniel . 2012 . Thinking, Fast and Slow . London : Penguin . Kingdon , John W. 1984 . Agendas, Alternatives, and Public Policies . Boston : Little, Brown . Lomas , Jonathan , and Adalsteinn D. Brown . 2009 . Research and Advice Giving: A

Functional View of Evidence-Informed Policy Advice in a Canadian Ministry of Health . Milbank Quarterly 87 ( 4 ): 903 – 26 .

Oliver , Kathryn , Simon Innvær , Theo Lorenc , Jenny Woodman , and James Thomas . 2014a . A Systematic Review of Barriers to and Facilitators of the Use of Evidence

by Policymakers . BMC Health Services Research 14 : 2 . http://www.biomedcentral. com/1472-6963/14/2 [accessed March 11, 2016].

Oliver , Kathryn , Theo Lorenc , and Simon Innvær . 2014b . New Directions in Evidence-Based Policy Research: A Critical Analysis of the Literature . Health Research Policy and Systems 12 : 34 . http://www.biomedcentral.com/content/ pdf/1478-4505-12-34.pdf [accessed March 11, 2016].

Oliver , Kathryn , Annemarie Money , and Frank de Vocht . 2013 . PP70 Knowledge Brokers or Policy Entrepreneurs? Strategies to Influence the Policy Process. Supplement 1 , Journal of Epidemiology and Community Health 67 : A76.

Schneider , Anne , Helen Ingram , and Peter deLeon . 2014 . Democratic Policy Design: Social Construction of Target Populations . In Theories of the Policy Process , 3rd ed. , edited by Paul A. Sabatier , and Christopher M. Weible , 105 – 50 . Boulder, CO : Westview Press .

Stoker , Gerry . 2010 . Translating Experiments into Policy . Annals of the American Academy of Political and Social Science 628 : 47 – 58 .

Stringer , Lindsay C. , and Andrew J. Dougill . 2013 . Channelling Science into Policy: Enabling Best Practices from Research on Land Degradation and Sustainable Land Management in Dryland Africa . Journal of Environmental Management 114 : 328 – 35 .

True , James L. , Bryan D. Jones , and Frank R. Baumgartner . 2007 . Punctuated Equilibrium Theory: Explaining Stability and Change in Public Policymaking . In Theories of the Policy Process , 2nd ed. , edited by Paul A. Sabatier , 59 – 104 . Boulder, CO : Westview Press .

Weible , Christopher M. , Tanya Heikkila , and Paul A. Sabatier . 2012 . Understanding and Influencing the Policy Process . Policy Sciences 45 ( 1 ): 1 – 21 .

Copyright of Public Administration Review is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.