Exceptional proff only 2

Policy design as craft: teasing out policy design expertise using a semi-experimental approach

Mark Considine • Damon Alexander • Jenny M. Lewis

Published online: 24 November 2013 � Springer Science+Business Media New York 2013

Abstract Public policy research typically neglects the role of the individual policy actor with most accounts of the policy process instead privileging the role of governmental

systems, institutions, processes, organizations; organised interests or networks of multiple

actors. The policy design literature suffers from similar limitations, with very few authors

paying attention to the crucial work of the individual policy designer or considering how

the latter’s skills, expertise and creativity are employed in the design task. This represents a

significant weakness in our understanding of how policy is formulated. This paper outlines

and previews what we believe is a potentially fruitful semi-experimental methodological

tool for exploring how individual policy actors draw on knowledge, expertise, intuition and

creativity in framing and responding to complex policy issues. Real-time scenario-based

problem-solving exercises are used to explore how policy problems and solutions are

framed and articulated by novice (first-term politicians and early career bureaucrats) and

experienced (former cabinet ministers and senior civil servants) policy actors and to

examine the strategies and approaches they employ in response to specific problem cues.

Initial findings are discussed, and we conclude by advancing potential refinements of the

instrument and directions for future research.

Keywords Policy design � Expertise � Decision-making � Scenario-based exercises

M. Considine (&) Faculty of Arts, University of Melbourne, Parkville, Australia e-mail: m.considine@unimelb.edu.au

D. Alexander � J. M. Lewis School of Social and Political Sciences, University of Melbourne, Parkville, Australia e-mail: dta@unimelb.edu.au

J. M. Lewis e-mail: jmlewis@unimelb.edu.au

123

Policy Sci (2014) 47:209–225 DOI 10.1007/s11077-013-9191-0

Introduction

Writing in 1988, Anne Schneider and Helen Ingram remarked that despite major advances

in the study of public policy, policy design remained uncharted and relatively underde-

veloped (1988: 61). Somewhat surprisingly, two decades later their observation remains

valid. Despite promising beginnings in the early 1980s with the work of authors such as

Alexander (1979, 1982), Simon (1981), Dryzek (1983), Linder and Peters (1984) and

Schneider and Ingham (1988), the art of policy design remains an important yet seriously

under-researched field within policy analysis. Persuasive models of the policy process now

abound and account for factors as disparate as the role of historical inheritance, ideas in

good standing, the impact of coalitions of actors and the nature of governance systems.

Few systematic analyses though have focused on policy design as a creative act or have

considered what might be gained by thinking about policy as a design process. Even fewer

have closely analysed the crucial work of the policy designer or considered how the latter’s

skills, expertise and creativity are employed in the design task.

Indeed, for the most part, public policy research has neglected the role of the individual

policy actor. Instead, attention typically focuses on seeking to explain and account for the

work of governmental systems, institutions, processes, like-minded interests and of the

intersecting work of multiple rather than individual players (Allison 1971; Bozeman and

Scott 1992; Considine 2005). This approach, we argue, not only underplays the agency and

influence of individual policy actors, but also the importance of these actor’s skills,

political style, creativity and expertise in shaping policy processes and outcomes. This is a

significant weakness in our understanding of how policy is developed that also potentially

limits our capacity to train and support the development of new policy experts and leaders.

As authors such as Chabal (2003) convincingly argue, individual actors such as min-

isters and senior civil servants are able to influence policy processes and content by dint of

their professional style and capacity. This influence has long been recognised by scholars

in the fields of international relations, particularly within foreign policy analysis where

actor-centric explanations of key events have been common (See for example Hudson

2005; Schafer and Walker 2006 Shannon and Keller 2007). The policy sciences, in con-

trast, have been slow to recognise this point with very few studies systematically exploring

the importance of individual capacities and actions in shaping policy processes or

outcomes.

This paper forms part of a larger emerging research programme that seeks to do just

this. It begins by reviewing the trajectory of policy design research since the early 1980s,

arguing that despite initial promise, research in the field has become increasingly bogged

down with questions of instrumentation. This, the paper argues, has come at the expense of

developing a deeper, more nuanced understanding of the ‘art’ of the individual policy

designer. The paper then reviews the existing literature exploring the skills, attributes and

expertise exhibited by policy-makers, before mapping out and testing an experimental

scenario-based problem-solving method designed to shed light on how these skills and

expertise are brought to bear in response to complex policy challenges.

Policy design: ‘process’ versus ‘instrumentation’

In a recent and comprehensive account, Howlett and Lejano (2012) trace the declining

interest in ‘policy design’ from the mid-1990s onwards. This shift, they argue, can be

attributed to the ‘decentering of policy studies away from the centrality of authority and

state centredness’ as those working in the field looked increasingly towards globalisation

210 Policy Sci (2014) 47:209–225

123

and the shift from ‘government to governance’ to explain policy formulation and instru-

ment choice (Howlett and Lejano 2012: 9). As they argue,

Many existing debates about policy choices are embedded in discussions of these two

processes, and as a result, much of the existing discussions of policy tools and policy

designs is characterized by misinformation, ideological predilection, and unneces-

sarily polarized positions. (2012: 13)

While we agree that this ‘present-day neglect of the existing corpus of studies on policy

design’ (2012: 14) is problematic, it both reflects and overlooks an equally troublesome

and for our purposes, more significant shift in the focus of ‘‘policy design scholarship’’’.

This shift we characterise as a move away from policy design as verb with an emphasis on

design processes towards policy design as noun with an emphasis on design content or

instrumentation (May 1981; Schneider and Ingram 1988; Howlett and Lejano 2012).

Importantly, this shift away from exploring the artistry of the design process itself towards

a greater focus on design outputs leaves us without a clear picture of how policy designs

come to be. This is a significant void in our understanding of the policy process.

As Dryzek argued in the early 1980s, public policy’s capacity to respond effectively to

complex contemporary social problems could be significantly enriched by a shift in policy

analysis ‘…away from methods emphasising the assessment of pre-ordained and well- defined alternatives, and towards policy design’ (Dryzek 1983: 345). This Dryzek defined

as ‘… the process of inventing, developing and fine-tuning [our emphasis] a course of action with the amelioration of some problem or the achievement of some target in mind’

(Dryzek 1983: 346). Interestingly, the creative element of this ‘inventing, developing and

fine-tuning’, what we might call the design component of the design process, has been

consistently underplayed in the literature. Design involves a practice of disciplined crea-

tivity, imaginative prototyping and an achieved form of competence that can be translated

from one project to another (Alexander 1982; Considine 2012). Yet, these important facets

of the design task are largely ignored in the policy analysis literature.

Part of this underplaying no doubt stems from the traditional dominance of ‘rationalism’

within the policy sciences. As Alexander noted in the early 1980s, when pushing for

greater recognition of the role of creativity: ‘those who would like to believe that the

decision process can be expressed in an algorithm, however complex … [n]aturally prefer to disregard a stage involving creativity, with its associations of unpredictability and its

basically irrational nature’ (Alexander 1982:281). Further explanation lies in the previ-

ously alluded to tendency for the policy sciences and the social sciences in general to focus

on the levels of analysis beyond the individual, while the sheer difficulty of unpacking the

policy design ‘black box’ is also a likely contributor. The design process often occurs

beyond public scrutiny and is incredibly complex (Howlett and Lejano 2012), both features

which militate against empirical observation. Moreover, as Schneider and Ingram note ‘… when design includes ideas about strategies to solve problems, it has been viewed as so

creative that it is an art rather than a science, and therefore cannot be captured’ (1988:

61–62). While similar debates in the fields of expertise studies and within the discipline of

design have led to the development of innovative and valuable new empirical methods

such as protocol analysis (see for example Dorst 1995; Valkenburg and Dorst 1998 Jiang

and Yen 2009) and important new theoretical debates and frameworks [see for example

Schon’s (1983) theory of reflective practice], we are yet to see comparable breakthroughs

in the policy sciences.

In this paper, we echo Howlett and Lejano’s (2012) call to rediscover policy design. But

we do so with an interest in policy design that is focused upon exploring and understanding

Policy Sci (2014) 47:209–225 211

123

the application of the skills, expertise, craft and creativity of individual policy actors, rather

than policy design as content or instrumentation (see Weimer 1993, 1998 for an excellent

discussion of policy analysis as craft). We link these ideas by considering one key

dimension of a design approach—the notion that policy making has important biographical

dynamics that inform the skill set of the designer. These mature into high-level abilities to

make judgements and develop sophisticated shortcuts in order to tackle very complex

policy challenges, including the challenge to make decisions under time and information

constraints.

Politicians, bureaucrats and policy skills and expertise

The attributes, skills and expertise that politicians and bureaucrats bring to their role as

policy actors are likely to hold important implications for the performance of government

(Headey 1974; Beckman 2006; Alexander et al. 2013). Unfortunately, despite this sig-

nificance, research in this area remains to be limited. There is a significant body of

literature focusing on the skills and attributes of policy entrepreneurs, policy brokers and

policy innovators. Kingdon’s (1995) classic work on agenda setting, for example, notes the

superior networking and negotiating skills of policy entrepreneurs, their ‘claim to a

hearing’ based on expertise or formal role and their persistence. Mintrom (2000) has

similarly pointed to the creativity and insight, social perceptiveness, social and political

dexterity, persuasiveness and strategic sense of policy entrepreneurs. Kuhnert (2001)

identifies personality, charisma and management skills as important attributes, while

Mintrom and Norman (2009) and Considine et al. (2009) identify social acuity and net-

working skills as key attributes. In terms of methods, insights can be gained from diaries,

biographies, autobiographies and memoirs (Greenstein, 1969) and from the more detailed

anthropological accounts of the working lives of elite-level policy actors provided by

interpretive studies (See for example Rhodes 2005, 2011; Rhodes et al. 2007; Rhodes and

Weller 2001).

There is also a small body of studies focusing on the general role requirements and

performance of elite political actors rather than on more narrow policy-specific skills or

attributes. In this vein, Greenstein’s research into presidential performance identifies

proficiency as a public communicator; organizational capacity; political skill; policy

vision; cognitive style; and emotional intelligence as key markers of elite-level perfor-

mance (cited in Theakston 2006: 3). Similarly, Bakema and Secker (1988) focus on

ministerial performance, noting the importance of technical knowledge; political knowl-

edge and skills and administrative and managerial skills. Headey’s (1974) ambitious

attempt to identify the role requirements of cabinet ministers and to explain cross-national

variations in government performance by differences in ministerial competencies provides

another useful example focusing on core skills with those identified, including specialised

knowledge of their policy field; analytical skills; managerial skills; the ability to ‘organise’;

public relations skills; political weight; political judgement and brokerage skills.

The seminal Australian study: Can Ministers Cope? (Weller and Grattan 1981) and the

more recent follow-up Learning to Be A Minister (Tiernan and Weller 2010) also explore

the nature of ministerial roles in federal politics, and, more obliquely, the skill sets and

individual attributes required. The second work in particular touches on ministerial skills

and qualities and how these develop by exploring the early experiences of ministers in the

Rudd Labor government. Elsewhere, where ministerial expertise has been examined, it has

often simply been assumed based on professional background rather than examined in

empirical terms (see Bakema and Secker 1988; Beckman 2006 for a critique of this

212 Policy Sci (2014) 47:209–225

123

approach). Thus, a lawyer appointed to the position of Attorney General may be regarded

as an ‘expert’, but a lawyer appointed as Treasurer regarded as a generalist.

Beyond this, studies focusing specifically on the skills and expertise of policy-makers

and how they develop and ply their trade are rarely generalised, and lacking an empirical

grounding—or as Theakston puts it ‘strong on sweeping generalisation or anecdote and

weak in terms of systematic comparison and evaluation’ (2006:3). In the next section, we

provide an overview of one approach to examine the art of policy design—scenario-based

problem-solving.

Scenario-based problem-solving

Scenario-based problem exercises provide one potential avenue of empirically exploring

these questions in a systematic fashion. The use of such exercises, simulation or gaming to

understand decision-making processes or to improve decision-making strategies through

training is well established across a wide range of fields. In sport, for example gaming and

simulations of varying degrees of complexity have been widely used to examine and

improve the decision-making strategies of squash players (Allain and Sarrazin 1990),

sailors (Walls et al. 1998), soccer goal-keepers (Williams and Burwitz 1993), tennis

players (Ward et al. 2002), volleyball players (Borgeaud and Abernathy (1987) and rugby

union referees (Mascarenhas et al. 2005). In medicine, similar methods have been used to

examine differences in perceptual-motor skills between trainee and expert surgeons (see

for example Torkington et al. 2001 as cited in Ward et al. 2006) and to train nurses and

surgical teams (Kneebone 2003; Law et al. 2004; Norman et al. 2006). An even more

extensive body of research uses simulation and gaming to explore decision-making in a

military context (for an overview see Smith 2010). Closer to our current subject matter,

proponents of game theory have used simulations of varying degrees of complexity to

model the decision-making processes of voters, public officials and political leaders alike

(see for example Guetzkow et al. 1963; Jervis 1988; Schaph 1991, 1994; Tsebelis 1990).

As Ward et al. (2006) suggest, such approaches offer a potentially valuable compromise

between ‘real-world’ examinations of decision-making expertise, which are often difficult

to capture and generalise from, and more simplistic laboratory-based experimentation

which, though easily replicated in controlled environments, often offers little in terms of

useful explanation for real-world situations. Despite their utility and prevalence in other

fields, the use of such methods in public policy and management research has been rela-

tively restricted. Throughout the 1950s, 1960s and 1970s, influential think tanks such as the

Rand Corporation were amongst the first institutions to extend the use of such approaches

to the public policy field (Mayer 2009). Moving beyond the highly formalised and pri-

marily computer-based operations research and systems research approaches, which had

been used to model military and strategic decision-making, Rand began experimenting

with human-based free-form models of simulation in areas such as urban planning and drug

policy (Kahan et al. 1995) in an attempt to shed light on decision-making in government

and to make the policy process more rational (Mayer 2009).

More recently, similar methods such as multi-actor scenario-based policy exercises (see

Toth 1988a, b for detailed descriptions) have been used to explore alternative approaches

to complex policy issues. These exercises are interactive and flexibly structured processes

designed to act as an interface between academics, experts and policy-makers (Toth

1988a). Typically, individuals or teams are provided with detailed starting scenarios

Policy Sci (2014) 47:209–225 213

123

depicting specific policy problems and environmental conditions. In an iterative process

involving multiple stages and feedback loops, alternative policy responses are tested,

adjusted for predicted consequences, recast and re-evaluated as a means of synthesising

knowledge from a range of perspectives and bringing greater insight to the nature of the

problem and potential solutions. Such exercises have been used to explore a diverse range

of complex policy issues such as likely stakeholder reactions to dramatic climate change

events (Toth and Hizsnyik 2004); problems around the implementation of climate change

mitigation measures (Parson 1996); safeguards around the use of genetically modified

crops (Marris et al. 2003; Oreszczyn and Carr 2008) and decisions around the provision of

healthcare for the aged in the Netherlands (Joldersma et al. 1995). Obviously, the most

potent challenge to using simulations or scenario-based methods such as these is the fact

that they are artificial rather than real-life situations, and knowing this, the decision-makers

may react differently. While true, this is also the case for every educational, training,

internship and apprenticeship environment through which we currently develop high-level

expertise.

Data and research method

Against this background, we trialled the use of a relatively simple scenario-based approach

to test its utility in shedding light on how individual policy actors frame and respond to a

range of complex policy problems. We were particularly interested in whether such

instruments could be usefully employed to tease out how policy actors ‘read’ issues and

navigate their way through complex administrative and political territory. Given the semi-

experimental nature of the method, we were keen to test whether sufficient realism and

‘buy-in’ could be elicited from respondents, so that responses might reflect actual problem-

solving behaviour with some form of external validity.

Forty-six semi-structured interviews were conducted with former Australian state and

federal government ministers, sitting first-term backbenchers, early career state and federal

policy officers and ex-senior government bureaucrats. 1

A description of this current sample

is provided in Table 1. The nine ministers interviewed have on average of almost 18-year

parliamentary service and 8-year ministerial experience, while the senior bureaucrats have

each served lengthy periods at the highest level of their respective state civil services. In

contrast, the backbenchers interviewed are all in their first parliamentary terms with the

early career bureaucrats averaging three years’ service in a policy role. Interviews ranging

1 This constitutes an initial sample which will be extended to approximately 70–80 respondents by the end

of the project. The sample frame includes all ex-state and federal ministers who had served at least two parliamentary terms; senior bureaucrats included all ex department secretaries who had served at least 10 years in a senior leadership position at the state and/or federal level; first-term backbenchers included all federal and Victorian parliamentary members serving their first term in office; early career policy officers included state and federal public officials in the first 5 years of a policy role. Recruitment for ex-ministers, ex-senior bureaucrats and backbenchers was by direct invitation with all those responding favourably to interview. Early career policy officers were recruited by invitation through a professional association, the Institute for Public Administration Australia (IPAA) Victorian branch. This sample is currently heavily skewed as it includes no female ministers and only a small number of senior bureaucrats. The final sample will reflect a quota sampling methodology with approximately equal numbers across all four subpopulations and a proportional representation across genders.

214 Policy Sci (2014) 47:209–225

123

in duration from 45 to 90 min were conducted face-to-face between March 2011 and

December 2012 and were recorded and fully transcribed. 2

In these interviews, respondents were provided with a brief one and a half-page scenario to

read through. This scenario outlined a high-profile policy problem—alcohol-fuelled street

violence in an urban environment—and allocated respondents a specific role to play: either a

cabinet minister with carriage over the policy area (ex-ministers and backbenchers); department

secretary (ex-senior bureaucrats) or executive director with departmental responsibility for the

policy issue (novice bureaucrats). Having read through the problem, respondents were

prompted to ‘talk us through what you see—tell us what jumps off the page’. Similar to practices

employed in methods such as protocol analysis, interaction with the participant was kept to an

absolute minimum while they outlined key elements of the problem as they saw it. Effectively,

the interviewers adopted the role of ‘non-judgmental listener’, and only brief verbal and non-

verbal signals and prompts were used to encourage the participant to fully express their thoughts

(Williamson et al. 2000; Ranyard and Williamson 2005).

Having outlined the nature of the problem as they see it, interviewees were then asked to take

on their allocated role in the scenario and to describe the steps they would take in dealing with

such a problem, again, with minimal interaction with the interviewer. The entire conversation

was digitally recorded and fully transcribed. These transcripts were then subjected to detailed

narrative analysis to assess how actors evaluate and frame the nature and key characteristics of

the problem, and then to evaluate the level of detail and expertise reflected in their response.

The remainder of this paper discusses some of the initial analytical approaches we are

experimenting with and outlines some of our early findings. We emphasise that these

analytical approaches are in their infancy and that further more rigorous testing, analysis

and refinement is required before the data collection method and analytical approaches

used are properly validated. The purpose of outlining them here and flagging the pre-

liminary results of our analysis is to generate discussion about the potential utility of the

approach and to encourage others to test variations of it.

Initial results

‘First response’ classification

At this point in the method’s development, we have restricted our attention to capturing

and analysing the ‘first responses’ of our policy actors to two relatively simple tasks—

Table 1 Current sample characteristics

Role Jurisdiction Total

Federal State

Ex-ministers 5 4 9

Backbenchers 4 8 12

Senior bureaucrats 0 3 3

Early career policy officers 2 20 22

11 35 46

2 The research design incorporates questionnaire-based scale items (administered face-to-face); ‘emblem-

atic’ case study analysis and a series of scenario-based problem-solving exercises.

Policy Sci (2014) 47:209–225 215

123

problem-framing and mapping initial responses. We have done this to make the analytical

task more manageable as we test the efficacy of the method, but also in recognition of the

critical nature, these initial steps in framing and responding to problems play in shaping

both policy design processes and outcomes. The manner in which we define problems sets

boundaries to our attention and imposes a level of coherence on the situation confronted

(Olshfski and Cunningham 2008). Initial descriptions and responses are likely to reflect

these ‘sense-making’ processes and the ‘working assumptions’ used by actors addressing

complex problems (Hajer 1995; Hood and Margetts 2007). As the anchoring literature

suggests (see for example Tversky and Kahneman 1974; Furnham and Boo 2011), they are

also important markers of likely subsequent decisions and responses.

With this in mind, we examined the content of the first substantive thought provided by

interviewees when asked ‘what jumps off the page?’ By substantive thought, we mean the

first meaningful thought expressed relating directly to the problem (See Table 2 for

examples). Thematic analyses of these initial reactions to the scenario revealed six general

types of responses put forward by the interviewees: Diagnosis; Complexity; Practical;

Sceptical; Detail-focused and Value-based. These are described in Table 2 with illustrative

examples of responses from each category also provided.

As the data in Table 3 indicates, more than one in four respondents launched imme-

diately into a diagnosis of the underlying nature of the problem based on their initial

reading of the scenario. Seven of 42 respondents initially noted the complexity of the

problem, with another 7 asking for more detail. Six displayed a sense of scepticism about

how the problem was being framed in the scenario.

These results suggest the method is capable of discerning different problem-framing

strategies adopted by individual actors. More fine-grained analysis also pointed towards

differences across subgroups of policy actors (for example experienced players were much

more likely to proceed straight to diagnosis than novice actors, with the latter more likely

to focus on the actual problem details) although the numbers in the current sample are too

small to draw any definitive conclusions.

We expect similar differences between problem-framing and problem-solving approa-

ches to emerge across subgroups of policy actors (such as novice versus experienced

players; and politicians versus bureaucrats) when we analyse the interviewees’ actual

response strategies. What might these difference look like? In terms of the novice versus

experienced actor category, we advance the following working propositions as examples of

the kinds of markers which analysis of the scenario responses may enable us to identify.

Proposition 1 Firstly, all things being equal, we would expect both the level of analysis and the responses from experienced players to be more extensive (in length) and com-

prehensive in terms of substance than novice actors. They should also contain a noticeably

greater level of overall detail; more sophisticated analysis and explanation; and a greater

number of substantive elements.

Proposition 2 We also posit that novice actors are more likely to focus on data/infor- mation in trying to define/recognise the problem. Experienced actors, on the other hand

might be expected to more immediately recognise the nature of the problem based on prior

experience and to advance to the problem solution without requiring further information.

This being the case we would expect novice players to focus more upon the nature and

validity of the data provided, and to more overtly prioritise the need to supplement the

latter prior to acting.

216 Policy Sci (2014) 47:209–225

123

T a

b le

2 C

la ss

ifi c a ti

o n

o f

re sp

o n se

st ra

te g y

ty p e s

(p ro

m p t

= ‘h

a v

in g

re a d

th ro

u g

h th

a t

w h

a t

st a n

d s

o u

t/ w

h a t

le a p

s o

ff th

e p

a g

e ’)

In it

ia l

st ra

te g

y E

x a m

p le

D ia

g n o

si s:

re sp

o n

se p

ro c e e d

s d

ir e c tl

y to

a d

ia g

n o

si s

o f

th e

p ro

b le

m ‘O

n e

p o

in t

a b

so lu

te ly

u tt

e rl

y ju

m p

s o

u t

o f

th e

p a g

e .

T h

is is

a ll

a b

o u

t p

e rc

e p

ti o

n s’

. M

ic h

a e l,

e x

- m

in is

te r

‘W e ll

, I

p e rs

o n

a ll

y th

in k

it ’s

a p

ro b

le m

o f

p e rc

e p

ti o

n a n

d th

is m

a y

b e

a b

it o

f m

y h

is to

ry w

o rk

in g

in th

is is

su e ,

b e c a u

se I’

v e

w o

rk e d

in th

e a re

a a n

d I

th in

k it

is p

e rc

e p

ti o

n o

f th

e is

su e

b e in

g a c tu

a ll

y a

li tt

le b

it b

lo w

n o

u t

o f

p ro

p o rt

io n

m y se

lf ’.

C la

ir ,

n o

v ic

e b

u re

a u

c ra

t

C o

m p le

x it

y :

fi rs

t re

sp o

n se

is to

a c k

n o

w le

d g

e c o

m p

le x

it y

/ m

u lt

i- fa

c e te

d n

a tu

re o

f th

e p

ro b

le m

‘W e ll

, th

a t

it ’s

c o m

p le

x a n

d th

a t

if th

e re

w a s

a q

u ic

k fi

x so

lu ti

o n

it w

o u

ld h

a v

e …

T h

e re

is n

o e a sy

a n

sw e r’

. Z

o e ,

n o

v ic

e b

u re

a u

c ra

t

‘S o

th a t

th e

st a n

d o

u t

is su

e s

a re

th e re

’s a

g e n

u in

e p

o li

c y

p ro

b le

m , th

e re

’s a

re a l

n e e d

fo r

so m

e c h

a n

g e

in g

o v

e rn

m e n

t p

o li

c ie

s a n

d th

a t

th is

p ro

b le

m is

o c c u

rr in

g fo

r a

w h

o le

ra n g

e o

f c o

m p

le x

re a so

n s

in c lu

d in

g in

c re

a se

d w

e a lt

h , m

o b

il it

y , lo

w e r

p ri

c e

o f

a lc

o h o

l, p

o p

u la

ti o

n g

ro w

th , e c o

n o

m ic

g ro

w th

, p

o p

u la

ri ty

o f

M e lb

o u

rn e ’.

Jo h

n ,

e x

-m in

is te

r

P ra

c ti

c a l:

F ir

st re

sp o

n se

fo c u

se s

o n

p ra

c ti

c a l

re q u

ir e m

e n

ts o

f d

e a li

n g

w it

h th

e p

ro b

le m

. ‘W

e ll

, th

a t

th e re

’s a ,

th e re

’s a

re c o g

n it

io n

o f

a p

ro b

le m

in a

n u

m b

e r

o f

d if

fe re

n t

lo c a ti

o n

s a n

d it

’s re

g a rd

e d

a s

so m

e th

in g

th a t,

so m

e th

in g

n e e d s

to b e

d o n e

a b o u t.

It sh

o u ld

n ’t

, it

’s n o t

a c c e p ta

b le

to a ll

o w

it to

c o n

ti n

u e .

S o

so m

e th

in g

’s g

o t

to b

e d

o n

e ’.

P e te

r, se

n io

r b

u re

a u

c ra

t

‘T h

a t

y o

u ’v

e g

o t

to g

iv e

a re

p o

rt a t

c a b

in e t

n e x

t w

e e k

c o

m in

g u

p w

it h

a so

lu ti

o n

a n d

th a t

th e re

’s n

o si

n g

le e x

p la

in in

g fa

c to

r fo

r th

e v

io le

n c e

w h

ic h

is so

m e th

in g

th a t

re so

n a te

s w

it h

m e ’.

N a ta

li e ,

fi rs

t- te

rm b a c k b e n c h e r

S c e p

ti c a l:

fi rs

t re

sp o

n se

w a s

to q

u e st

io n

e v

id e n

c e

p ro

v id

e d

in th

e sc

e n a ri

o a s

b a si

s fo

r a c ti

o n

‘T h

e re

’s n

o e v

id e n

c e

is w

h a t

le a p

s o

u t

a t

m e …

th a t

it ’s

v e ry

– th

e so

lu ti

o n

s th

a t

h a v

e b

e e n

d e ri

v e d

a re

v e ry

k n e e

je rk

, in

tu it

iv e

a n d

th e re

’s n o

e v id

e n c e ’.

P a t,

se n io

r b u re

a u c ra

t

‘W e ll

, it

’s a n

is su

e th

a t

n e e d

s d

e a li

n g

w it

h [b

u t]

a t

fa c e

v a lu

e I

w o

u ld

n ’t

b e li

e v

e a n

y o

f th

e so

u rc

e s,

th e

m e d ia

o r

th e

p o li

c e ’.

M a rk

, e x -m

in is

te r

D e ta

il -f

o c u se

d :

fi rs

t re

sp o n se

w a s

to re

c o u n t

d e ta

il p

ro v

id e d

in th

e sc

e n

a ri

o ‘T

h a t

it ’s

p re

d o

m in

a n

tl y

in v

o lv

in g

y o

u n g

p e o

p le

a n

d th

a t

y o

u n g

p e o

p le

h a v

e a ls

o m

o st

li k

e ly

b e e n

th e

v ic

ti m

s o

f it

’. A

d a m

, fi

rs t-

te rm

b a c k

b e n

c h e r

‘A lc

o h

o l,

th e

v io

le n

c e ,

th e

c u

lt u

re o

f th

a t

v io

le n

c e

is o

k a y

’. D

a n

, n

o v

ic e

b u

re a u

c ra

t

V a lu

e -b

a se

d :

fi rs

t re

sp o n se

re fl

e c te

d v a lu

e -b

a se

d ju

d g

e m

e n

t o

f n

a tu

re o

f th

e p

ro b

le m

‘T h

e b

a la

n c e

b e tw

e e n

in d

iv id

u a l

re sp

o n

si b

il it

y a n

d g

o v

e rn

m e n

t’ s

in te

rv e n

ti o

n in

th a t

a re

a fo

r, if

y o

u li

k e ,

a m

a rk

e t

fa il

u re

’. M

ic h a e l,

fi rs

t- te

rm b a c k b e n c h e r

‘F o

r m

e I’

d w

o n

d e r

w h

a t

th e

ro le

o f

g o

v e rn

m e n

t is

in fi

x in

g th

is p

ro b

le m

, h

o w

m u

c h

g o

v e rn

m e n

t sh

o u ld

in te

rv e n

e to

c o

rr e c t

in d

iv id

u a l’

s p

o o

r b

e h

a v

io u

r, th

e re

a re

lo ts

o f

li tt

le w

a y

s th

a t

g o

v e rn

m e n t

c a n

tw e a k

th in

g s,

b u

t I

th in

k it

’s th

e in

d iv

id u a ls

th e m

se lv

e s,

w h

ic h

is p

ro b

a b ly

w h

y I

w o

n ’t

g e t

p ro

m o

te d

to th

is ro

le ’.

M e li

ss a ,

n o

v ic

e b

u re

a u

c ra

t

Policy Sci (2014) 47:209–225 217

123

Proposition 3 In line with the literature (see for example Simon 1978, 1981; Kahneman et al. 1982; Polsby 1984; Schneider and Ingram 1988; Ross et al. 2006) we would also

expect the experienced players to demonstrate a greater capacity to ‘pattern match’

problems and solutions against previously encountered issues as well as projecting for-

ward through mental simulation in dealing with the problem. This being the case we would

expect to see overt referrals back to prior experiences as well as evidence of forward

projection through simulation in their responses such as: if A then B type responses.

Proposition 4 Finally, we expect those with experience to demonstrate a superior level of situational awareness. This in turn should allow them to focus beyond the immediate

parameters of the problem to incorporate other important elements such as political

management and the likely reactions of other actors, rather than focusing on the specific

details of the problem itself. This being the case, we would expect experienced actors to

display a more developed and nuanced understanding of their environment and what we

might call ‘second-order’ issues such as political, stakeholder and media management and

how such groups are likely to react or behave in response to policy intervention.

While this part of the research programme is very much a work in progress, a first-cut

analysis of responses from four participants to the ‘Alcohol-fuelled Street Violence’ sce-

nario demonstrates the kinds of analytical approaches that may be usefully employed in

characterising the scenario-based responses of individual actors. It also suggests that such

an approach may be useful in identifying different ways in which actors approach the

policy design task. Table 4 provides a summary analysis of responses from two of the

experienced former ministers interviewed (Jackson and Greg) and from two first-term

backbenchers (Terrence and Martin).

First, in line with Proposition 1 on comprehensiveness, both former ministers were

much more expansive in their replies, in terms of both duration and word count. Jackson, at

7 min 39 s, spoke for between six and seven times longer than both novice respondents,

with Greg’s response taking three and four times that of Terrence and Martin, respectively.

Both former ministers also rated much higher in terms of the level of detail and sophis-

tication provided in their responses and tended to raise a greater number of substantive

elements both in framing the problem and in detailing how they would approach it. 3

There also seems to be some confirmation of our prediction about the likely immediate

focus of novice versus experienced actors on data/detail as outlined in Proposition 2.

Table 3 Classification of initial response to street violence sce- nario: (prompt = having read through that what stands out/what leaps off the page)

Total

Diagnosis 11

Complexity 7

Practical 4

Sceptical 6

Detail-focused 7

Value-based 4

Other 3

Total 42

3 For the purposes of this article, coding and assessment of responses were carried out by one of the authors.

Once the full sample is collected, responses will be blind-coded and categorised by a panel of two researchers independent of the research team.

218 Policy Sci (2014) 47:209–225

123

T a

b le

4 S

u m

m a ry

st a ti

st ic

s fo

r st

re e t

v io

le n c e

sc e n a ri

o re

sp o n se

s

In te

rv ie

w e r:

W h

a t

a re

y o

u r

st e p

s th

e n

fo r

d e a li

n g

w it

h so

m e th

in g

li k

e th

is ?

A s

a m

in is

te r,

if so

m e th

in g

li k

e th

a t

la n

d e d

o n

y o

u r

d e sk

, w

h a t

d o

y o

u d

o , h

o w

d o

y o

u a p

p ro

a c h

it ?

Ja c k

so n

G re

g T

e rr

e n

c e

M a rt

in

S u

m m

a ry

P o

li c y

fi e ld

e x

p e rt

is e

Y e s

(s o

c ia

l p

o li

c y

) N

o N

o N

o

R e a d

in g

ti m

e 2

: 5

8 1

:2 4

2 :3

3 2

:1 0

O v e ra

ll d e ta

il /s

o p h is

ti c a ti

o n

H ig

h M

e d iu

m L

o w

L o w

E x

te n

si v

e n

e ss

o f

re sp

o n

se (w

o rd

s/ ti

m e )

9 4

2 w

o rd

s (7

:3 9 )

5 3

4 w

o rd

s (4

:2 8

) 1

6 1

w o

rd s

(1 :3

0 )

1 0

3 w

o rd

s (1

:0 5 )

F lo

w (w

o rd

s/ s)

2 .1

2 .0

1 .8

1 .6

S u b st

a n ti

v e

e le

m e n ts

id e n ti

fi e d

(n )

[ 1

0 [

1 0

\ 5

[ 1

0

A n

a ly

si s

d e ta

il H

ig h

M e d

iu m

L o

w L

o w

R e sp

o n se

st ra

te g y

S e a rc

h Y

e s

Y e s

Y e s

N o

C o n

su lt

/c o

ll a b

o ra

te Y

e s

Y e s

Y e s

N o

C la

ss ifi

e s/

m a tc

h e s

Y e s

Y e s

N o

N o

P ro

je c ts

fo rw

a rd

Y e s

N o

N o

N o

R e sp

o n

se c o

n te

n t

R e sp

o n

se c o

n te

n t

d e ta

il H

ig h

M e d

iu m

L o

w M

e d

iu m

Id e n

ti fi

e s

k e y

a c to

rs Y

e s

Y e s

N o

N o

P ro

c e ss

d e ta

il H

ig h

L o

w L

o w

L o

w

P o

li ti

c a l

m a n a g

e m

e n

t H

ig h

L o

w L

o w

L o

w

Policy Sci (2014) 47:209–225 219

123

T a

b le

5 In

it ia

l re

sp o

n se

(fi rs

t tw

o se

n te

n c e s)

to st

re e t

v io

le n

c e

sc e n

a ri

o

In te

rv ie

w e r:

W h

a t

a re

y o

u r

st e p

s th

e n

? Y

o u

’r e

th e

m in

is te

r, y

o u

’v e

g o

t to

d o

so m

e th

in g

a b

o u

t it

, w

h a t

a re

th e

st e p

s y

o u

w o

u ld

ta k

e ?

T e rr

e n c e :

‘Y o u ’v

e g o t

to e st

a b li

sh th

e d e p th

o f

w h a t

th e

is su

e is

, w

h e th

e r

it ’s

so m

e —

I m

e a n ,

w e ’r

e in

a w

o rl

d n

o w

th a t

e v

e ry

th in

g is

re p o rt

e d

a n d

o v e r

re p o rt

e d ,

a n d

w e

c o m

p a re

b a c k

to te

n ,

2 0 ,

3 0

y e a rs

a g

o ,

is th

e si

tu a ti

o n

a n

y w

o rs

e n

o w

th a n

w h

a t

it w

a s

b a c k

th e n

? B

e fo

re ,

y o

u ju

st ju

m p

a n

d st

a rt

b o

x in

g a t

sh a d

o w

s, sa

y ,

w e ’v

e g

o t

to c ra

c k

d o

w n

o n

th is

so rt

o f

st u

ff —

b e c a u

se a t

th e

e n

d o

f th

e d

a y

, it

’s re

la ti

v e

to w

h a t’

s h

a p

p e n

e d

o v

e r

ti m

e ’.

G re

g :

‘W e ll

, it

’s a

c la

ss ic

c o

m b

in a ti

o n

o f

a sh

a rp

e n

d ‘h

e re

a n

d n

o w

p ro

b le

m ’,

c o m

b in

e d

w it

h a n

u n

d e rl

y in

g ‘r

o o

t c a u se

p ro

b le

m ’

a n

d d

e a li

n g

e it

h e r

w it

h th

e sh

a rp

e n

d o

r th

e ro

o t

c a u se

to th

e e x

c lu

si o

n o

f th

e o

th e r,

is n

o t

g o

in g

to so

lv e

th e

to ta

l p

ro b

le m

. B

u t

p ro

b a b ly

m y

in st

in c t

w o

u ld

b e

to fo

c u

s o

n th

e sh

a rp

e n

d a n

d h

o p

e th

a t

o v

e r

ti m

e th

e m

e ss

a g

e w

o u

ld se

e p

th ro

u g

h in

to th

e w

id e r

c o

m m

u n

it y

th a t

ir re

sp o n si

b le

b e h a v io

u r

is g o in

g to

b e

p u n is

h e d

a c c o rd

in g ly

’.

M a rt

in :

‘F ir

st ly

id e n ti

fy th

e p

ro b

le m

a n

d te

st th

e p

ro b

le m

b e c a u

se th

a t

w il

l in

d ic

a te

th e

e x te

n t

o f

th e

p ro

b le

m .

B a se

d o n

th is

b e in

g a c c u ra

te a n

d th

e in

fo rm

a ti

o n

in th

e re

, th

e re

is a

p ro

b le

m ’.

Ja c k

so n :

‘W e ll

, in

re la

ti o

n to

th e

b ro

a d

e r

p o

li c y

, w

h ic

h is

th e

re a l

p ro

b le

m ,

I w

o u

ld b

e u

si n

g th

e so

rt o

f a p

p ro

a c h

e d

u se

d in

th e se

o th

e r

th in

g s.

I w

o u

ld b

e se

e k in

g e x p e rt

a d v ic

e a n d

in p u t

a s

to w

h a t

h a s

w o rk

e d

e ls

e w

h e re

, a n d

w h a t

m ig

h t

w o

rk ’.

220 Policy Sci (2014) 47:209–225

123

Table 5 provides the initial response (first two sentences) given by each participant when

asked what steps they would take in response to the problem. As the responses show and as

predicted, both novice players immediately focused in on the need to gather more infor-

mation in order to more accurately gauge the problem. In marked contrast, both ex-

ministers ignored such detail and immediately went about classifying the nature of the

problem and working through potential solutions.

In doing so, their responses also provided evidence of pattern matching and simulation

thereby supporting Proposition 3. Greg, for example, talks about the problem being a

‘classic combination of a sharp end here and now problem, combined with an underlying

root cause problem’ [emphasis added]. In doing so, he relies upon a typology of familiar

problems based on prior experience. Later in his response, he draws a parallel between the

street violence problem and combating the road toll, suggesting advertising campaigns

based on those aimed at changing attitudes towards drink driving and speeding may be one

approach worth exploring. Jackson also overtly matches or compares the nature of the

problem and his likely approach in his opening response (see Table 5) to the scenario

noting that he would be ‘using the sort of approach used in these other things’. 4

There is also some evidence of ‘forward projection’ and mental simulation as he talks

about the need to develop holding strategies capable of relieving the immediate political

pressure while developing and implementing medium and long-term solutions capable of

dealing with what he identifies as the core problem of changing social behaviour and

attitudes around alcohol consumption. In contrast, neither of the novice actors provided

any evidence of problem matching, classifying or projecting in their responses.

Similar distinctions between novice and experienced actors were also evident with

respect to Proposition 4. As expected, both ex-ministers displayed far superior levels of

situational awareness, were more able to identify key actors/stakeholders and how they

might react to various strategies and were able to identify and focus attention on a range of

potentially important second-order issues, beyond the immediate context of the problem.

This was particularly the case for Jackson, who having served in a state-level social policy-

orientated portfolio, had at least some working knowledge of the problem identified in the

scenario and was therefore much better placed than his federal ministerial colleague to

provide a detailed, domain-specific response. 5

Conclusion

In this paper, we have considered some of the early research on policy design noting the

shift away from explorations of creative design processes (policy design as verb) towards a

focus on design content or instrumentation (policy design as noun). This shift we argue not

4 Prior to addressing the scenario, Jackson had outlined his involvement and approach in two other policy

issues. 5

This suggests an important contextual variable which needs to be considered in any analysis of variations in expertise levels displayed by novice and experienced policy-makers—the impact of domain-specific experience either in terms of jurisdiction, portfolio responsibilities, or prior experience. For example, it is possible that a first-term backbencher with a background in policing or the criminal justice system may display superior levels of domain-specific expertise in responding to this scenario than an ex federal minister with years of experience in a completely unrelated port-folio area. To control for this and to identify cases where prior experience may aid the development of domain-specific expertise, we are collecting detailed biographical information on all participants. We are also using two diverse scenario-based exercises in each interview as a way of ensuring at least one response will be drawn from an unfamiliar domain.

Policy Sci (2014) 47:209–225 221

123

only leaves us with a fundamentally incomplete picture of how policy designs come to be

but also further dilutes the already limited attention paid to the creativity, skills and

expertise of individual policy designers. This paper represents a small initial step in our

attempt to construct an approach to understanding public policy making that takes these

individual level factors into account. This focus on individuals is not to deny the impor-

tance of structure. Policy actors do not of course operate in a vacuum, but face all manner

of structural, institutional and organisational constraints, challenges and opportunities in

framing, designing, implementing and evaluating policy. The manner in which they

respond to these influences though, including, ultimately, their success or failure is heavily

conditioned by the skills, expertise, experience and creativity that they as individuals bring

to their role as policy actors. These factors warrant exploring.

In the second half of this paper, we have sketched out and reported our initial findings

from a semi-experimental method designed to shed light on how individual policy actors

frame and respond to a range of complex policy problems. While the approach is still

relatively rudimentary, our initial analysis suggests there may be significant potential in

using such scenario-based problem-solving exercises to tease out how respondents read,

make sense of and respond to problem cues, and in identifying different modes or styles of

thinking and acting. Such methods, as Bozeman and Scott (1992) note, have been widely

used elsewhere in the social sciences for decades, forming a key part of the research toolkit

for disciplines such as psychology, sociology, business, information science and education.

Closer to home, they have also been employed in the field of foreign policy analysis and

international relations. Surprisingly though, such approaches have barely been explored

within the fields of public policy and public administration. Technological development

and burgeoning interest in computer gaming are likely to change this in the near future

with rapid growth in the application of ‘serious computer gaming and simulation’ as a

research, teaching and learning aid in complex decision-making environments. 6

These

developments make the use of realistic, real-time, multi-iterative decision-making exer-

cises possible and have the potential to add greatly to our understanding of how and why

policy actors make the decisions they do.

Even operating at this relatively simple initial level, our approach has been able to

successfully tease out apparent differences in the ways in which actors read cues and

prioritise different problem components, and in the way in which novice and experienced

actors respond to policy problems. Beyond more in-depth analysis of the material we have

collected from these interviews through these scenarios, our next challenge is to introduce

more biographical characteristics such as role; policy field and jurisdiction; and to

incorporate iterative or multi-stage scenarios which provide respondents with opportunities

for information gathering and feedback loops. This will enable us to explore in more detail

how individual problem-solving styles vary across a wider range of biographical factors, as

well as improving the realism of the scenarios employed.

References

Alexander, E. R. (1979). The design of alternatives in organizational contexts: A pilot study. Administrative Science Quarterly (ASQ), 42(3), 382–404.

Alexander, E. R. (1982). Design in the decision making process. Policy Sciences, 14(3), 279–292.

6 See the work of the Delft Centre for Serious Gaming for a leading example of developments in this area.

www.seriousgaming.tudelft.nl.

222 Policy Sci (2014) 47:209–225

123

Alexander, D., Considine, M., & Lewis, J. M. (2013). Solving problems in policy and management: The role of skills, expertise and experience. In Conference paper presented at XVII IRSPM conference, Prague, April 10–12, 2013.

Allain, C., & Sarrazin, C. (1990). Study of decision-making in squash competition: A computer simulation approach. Canadian Journal of Sport Science, 15(3), 193–200.

Allison, G. T. (1971). Essence of decision: Explaining the Cuban missile crisis. Boston: Little Brown. Bakema, W. E., & Secker, I. P. (1988). Ministerial expertise and the Dutch case. European Journal of

Political Research, 16(2), 153–170. Beckman, L. (2006). The competent Cabinet? Ministers in Sweden and the problem of competence and

democracy. Scandanavian Political Studies, 26(2), 111–129. Borgeaud, P., & Abernethy, B. (1987). Skilled perception in volleyball defense. Journal of Sport Psy-

chology, 9, 400–406. Bozeman, B., & Scott, P. (1992). Laboratory experiments in public policy and management. Journal of

Public Administration, Research and Theory, 2(3), 293–313. Chabal, P. M. (2003). Do ministers matter? The individual style of ministers in programmed policy change.

International Review of Administrative Sciences, 69(1), 29–49. Considine, M. (2005). Making public policy: Institutions, actors, strategies. Cambridge: Polity Press. Considine, M. (2012). Thinking outside the box: Applying design theory to public policy. Politics and

Policy, 40(4), 704–724. Considine, M., Lewis, J. M., & Alexander, D. (2009). Networks, innovation and public policy: Politicians,

bureaucrats and the pathways to change inside government. Houndmills, Basinstoke: Palgrave Macmillan. Dorst, K. (1995). ‘Editorial’. Special issue on: Analysing design activity: New directions in protocol

analysis. Design Studies, 16, 139–142. Dryzek, J. S. (1983). Don’t toss coins in garbage cans: A prologue to policy design. Journal of Public

Policy, 3, 345–367. Furnham, A., & Boo, H. C. (2011). A literature review of the anchoring effect. The Journal of Socioeco-

nomics, 40, 35–42. Greenstein, F. (1969). Personality and politics: Problems of evidence, inference and conceptualisation.

Chicago: Markham Pub Co. Guetzkow, H., Alger, C., Brody, R., Noel, R. & Snyder, R. (1963). Simulation in international relations:

developments for research and teaching. Englewood Cliffs, NJ: Prentice-Hall, Inc. Hajer, M. A. (1995). The politics of environmental discourse: Ecological modernization and the policy

process. Oxford: Oxford University Press. Headey, B. W. (1974). The role skills of Cabinet ministers: A cross-national review. Political Studies, 22,

66–85. Hood, C., & Margetts, H. (2007). The tools of government in the digital age. Houndmills, Basinstoke:

Palgrave Macmillan. Howlett, M., & Lejano, R. P. (2012). Tales from the Crypt: The rise and fall (and Rebirth?) of policy design.

Administration and Society, XX(X), 1–25. Hudson, V. M. (2005). Foreign policy analysis: Actor-specific theory and the ground of international

relations. Foreign Policy Analysis, 1(1), 1–30. Jervis, R. (1988). Realism, game theory, and cooperation. World Politics, 40(3), 317–349. Jiang, H., & Yen, C. C. (2009). Protocol analysis in design research: A review. Paper presented at the

‘‘Design Rigor & Relevance’’, International association of societies of design research (IASDR) 2009 Conference, Seoul, Korea.

Joldersma, C., Geurts, J. L., Vermaas, J., & Heyne, G. (1995). A policy exercise for the Dutch health care system for the elderly. In D. Crookall & K. Arai (Eds.), Simulation and gaming across disciplines and cultures: ISAGA at a watershed (pp. 111–121). Thousand Oaks, CA: Sage Publications.

Kahan, J. P., Rydell, C. P., & Setear, J. (1995). A game of urban drug policy. Peace and Conflict: Journal of Peace Psychology, 1(3), 275–290, (RAND Reprint RP-459).

Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.

Kingdon, J. (1995). Agendas, alternatives and public policies (2nd ed.). Boston: Harper Collins. Kneebone, R. (2003). Simulation in surgical training: Educational issues and practical applications. Medical

Education, 37, 267–277. Kuhnert, K. W. (2001). Leadership theory in postmodernist organizations. In R. T. Golembiewski (Ed.), The

nature of organizational leadership. San Francisco: Jossey-Bass. Law, B., Atkins, M., Kirkpatrick, A., Lomax, A., & Mackenzie, C. (2004). Eye gaze patterns differentiate

novice and experts in a MIST VR laparoscopic simulator. Studies in Health Technology and Infor- matics, 50, 124–130.

Policy Sci (2014) 47:209–225 223

123

Linder, S. H., & Peters, B. G. (1984). From social theory top policy design. Journal of Public Policy, 4, 237–249.

Marris, C., Joly, P., Ronda, S., & Bonneuil, C. (2003). Precautionary expertise for GM crops (PEG). National Workshop Report, France.

Mascarenhas, D. R. D., Collins, D., Mortimer, P., & Morris, R. L. (2005). A naturalistic approach to training accurate and coherent decision making in rugby union referees. The Sport Psychologist, 19, 131–147.

May, P. J. (1981). Hints for crafting alternative policies. Policy Analysis, 7, 227–244. Mayer, I. (2009). The gaming of policy and the politics of gaming: A review. Simulation and Gaming, 40(6),

825–862. Mintrom, M. (2000). Policy entrepreneurs and school choice. Washington, DC: Georgetown University

Press. Mintrom, M., & Norman, P. (2009). Policy entrepreneurship and policy change. The Policy Studies Journal,

37(4), 649–667. Norman, E., Brooks, L., & Hamstra, S. (2006). Expertise in medicine and surgery. In E. Ericsson, N.

Charness, P. J. Feltovich, & R. Hoffman (Eds.), Cambridge handbook of expertise and expert per- formance (pp. 339–353). Cambridge: Cambridge University Press.

Olshfski, D., & Cunningham, R. (2008). Agendas and decisions how state government executives and middle managers make and administer policy. Albany: State University of New York Press.

Oreszczyn, S., & Carr, S. (2008). Improving the link between policy research and practice: Using a scenario workshop as a qualitative research tool in the case of genetically modified crops. Qualitative Research, 8(4), 473–497.

Parson, E. (1996). A Global Climate-Change Policy Exercise: Results of a Test Run, July 27–29, 1995. Polsby, N. W. (1984). Political innovation in America: The politics of policy initiation. New Haven, Conn:

Yale University Press. Ranyard, R., & Williamson, J. (2005). Conversation-based process tracing methods for the study of natu-

ralistic decision making: Analysing information search and verbal protocols. In H. Montgomery, R. Lipshitz, & B. Brehmer (Eds.), How professionals make decisions. Mahwah, NJ: LEA Associates.

Rhodes, R. A. W. (2005). Everyday life in a ministry: Public administration as anthropology. American Review of Public Administration, 20(1), 1–23.

Rhodes, R. A. W. (2011). Everyday life in British government. Oxford: Oxford University Press. Rhodes, R. A. W., ‘t Hart, P., & Noordegraaf, M. (Eds.). (2007). Observing government elites: Up close and

personal. Basingstoke: Palgrave Macmillan. Rhodes, R. A. W., & Weller, P. (Eds.). (2001). The changing world of top officials: Mandarins or valets?.

Buckingham, UK: Open University Press. Ross, K. G., Shaffer, J. L., & Klein, G. (2006). Professional judgments and ‘‘naturalistic decision making’’.

In K. A. Ericsson, N. Charness, R. R. Hoffman & P. J. Feltovich (Eds.), The Cambridge handbook of expertise and expert performance. New York: Cambridge University Press.

Schafer, M., & Walker, S. G. (2006). Democratic leaders and the democratic peace: The operational codes of Tony Blair and Bill Clinton. International Studies Quarterly, 50(3), 561–583.

Schaph, F. (1991). Games real actors could play. The challenge of complexity. Journal of Theoretical Politics, 3(3), 277–304.

Schaph, F. (1994). Games real actors could play. Positive and negative coordination in embedded negoti- ations. Journal of Theoretical Politics, 6(1), 27–53.

Schneider, A., & Ingram, H. (1988). Systematically pinching ideas: A comparative approach to policy design. Journal of Public Policy, 8(1), 61–80.

Schon, D. (1983). The reflective practitioner: How professionals think in action. London: Temple Smith. Shannon, V. P., & Keller, J. W. (2007). Leadership style and international norm violation: The case of the

Iraq war. Foreign Policy Analysis, 3(1), 79–104. Simon, H. A. (1978). Rational decision making in business organizations. Nobel Memorial Lecture. Simon, H. A. (1981). The sciences of the artificial. Cambridge, MA: The MIT Press. Smith, R. (2010). The long history of gaming in military training. Simulation and Gaming: An Interdisci-

plinary Journal, 41(1), 6–19. Theakston, K. (2006). What makes for an effective British prime minister? Paper prepared for the 2006

annual meeting of the American Political Science Association. Tiernan, A., & Weller, P. (2010). Learning to be a minister: Heroic expectations, practical realities.

Melbourne: Melbourne University Press. Torkington J, Smith S.G., Rees B.I., & Darzi, A. (2001). ‘Skill transfer from virtual reality to a real

laparoscopic task’, Surgical Endoscopy, 15(10), 1076–1079 Toth, F. L. (1988a). Policy exercises: Objectives and design elements. Simulation and Gaming: An Inter-

disciplinary Journal, 19, 235–255.

224 Policy Sci (2014) 47:209–225

123

Toth, F. L. (1988b). Policy exercises: Procedures and implementation. Simulation and Gaming: An Inter- disciplinary Journal, 19, 256–276.

Toth, F., & Hizsnyik, E. (2004). Managing the inconceivable: Participatory assessments of impacts and responses to extreme climate change. Laxenburg, Austria: International Institute for Applied Systems Analysis.

Tsebelis, G. (1990). Nested games: Rational choice in comparative politics. California: University of California Press.

Tversky, A., & Kahneman, D. (1974). Judgement under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

Valkenburg, R., & Dorst, K. (1998). The reflective practice of design teams. Design Studies, 19(3), 249–271. Walls, J., Bertrand, L., Gale, T., & Saunders, N. (1998). Assessment of upwind dinghy sailing performance

using a virtual reality dinghy sailing simulator. Journal of Science and Medicine in Sport, 1(2), 61–72. Ward, P., Williams, A. M., & Bennett, S. J. (2002). Visual search and biological motion perception in tennis.

Research Quarterly for Exercise and Sport, 73, 107–112. Ward, P., Williams, M., & Hancock, P. (2006). Simulation for performance and training. In E. Ericsson, N.

Charness, P. J. Feltovich, & R. Hoffman (Eds.), Cambridge handbook of expertise and expert per- formance (pp. 243–262). Cambridge: Cambridge University Press.

Weimer, D. (1993). ’The current state of design craft: Borrowing, tinkering, and problem solving. Public Administration Review, 53(2), 110–120.

Weimer, D. (1998). Policy analysis and evidence: A craft perspective. Policy Studies Journal, 26(1), 114–128.

Weller, P., & Grattan, M. (1981). Can ministers cope? Australian federal ministers at work. Richmond, Vic: Hutchinson.

Williams, A. M., & Burwitz, L. (1993). Advance cue utilization in soccer. In T. Reilly, J. Clarys, & A. Stibbe (Eds.), Science and football II (pp. 239–244). London: E. & F. N. Spon.

Williamson, J., Raynard, R., & Cuthbert, L. (2000). A conversation-based process tracing method for use with naturalistic decisions: An evaluation study. British Journal of Psychology, 91, 203–221.

Policy Sci (2014) 47:209–225 225

123

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.