615 forum

Evaluation and Program Planning 58 (2016) 208–215

Original full length article

Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives

Carmen Huckel Schneidera,b,*, Andrew J. Milatc,d, Gabriel Moorea

a Menzies Centre for Health Policy, University of Sydney, Level 6 The Hub, Charles Perkins Centre D17, The University of Sydney NSW 2006, Australia b The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo NSW 2007, Australia c New South Wales Ministry of Health, 73 Miller St, North Sydney NSW 2060, Australia d Sydney Medical School, University of Sydney, Edward Ford Building (A27), Fisher Road, University of Sydney NSW 2006, Australia

A R T I C L E I N F O

Article history: Received 9 December 2015 Received in revised form 28 April 2016 Accepted 17 June 2016 Available online 11 July 2016

Keywords: Policy evaluation Health policy Public health administration Public health

A B S T R A C T

Our research sought to identify the barriers and facilitators experienced by policymakers and evaluation researchers in the critical early stages of establishing an evaluation of a policy or program. We sought to determine the immediate barriers experienced at the point of initiating or commissioning evaluations and how these relate to broader system factors previously identified in the literature. We undertook 17 semi-structured interviews with a purposive sample of senior policymakers (n = 9)

and senior evaluation researchers (n = 8) in Australia. Six themes were consistently raised by participants: political influence, funding, timeframes, a ‘culture

of evaluation’, caution over anticipated results, and skills of policy agency staff. Participants also reflected on the dynamics of policy-researcher relationships including different motivations, physical and conceptual separation of the policy and researcher worlds, intellectual property concerns, and trust. We found that political and system factors act as macro level barriers to good evaluation practice that

are manifested as time and funding constraints and contribute to organisational cultures that can come to fear evaluation. These factors then fed into meso and micro level factors. The dynamics of policy- researcher relationship provide a further challenge to evaluating government policies and programs.

ã 2016 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locat e/e valprogplan

1. Introduction

Previous research suggests that the number of public sector policies and programs that are evaluated in industrialized nations is low (Oxman, Bjorndal, & Becerra-Posada, 2010) and that policy makers, who we define as those working in the public sector to plan, design, implement and oversee public programming, face significant hurdles in establishing and implementing policy and program evaluation (Evans, Snooks, & Howson, 2013). These challenges sit within what is known as the ‘messy’ business of public policy; including political compromise, collaboration and coordination amongst multiple stakeholders and strategic and tactical decision-making, that have long been recognised in the

* Corresponding author. Present address: Menzies Centre for Health Policy, University of Sydney Level 6 The Hub, Charles Perkins Centre D17, The University of Sydney NSW 2006, Australia.

E-mail addresses: [email protected] (C. Huckel Schneider), [email protected] (A.J. Milat), [email protected] (G. Moore).

http://dx.doi.org/10.1016/j.evalprogplan.2016.06.011 0149-7189/ã 2016 Elsevier Ltd. All rights reserved.

evaluation policy literature (Head & Alford, 2015; Parsons, 2002; House, 2004).

There is however notable potential to utilise evaluation to draw lessons about public health policies’ and programs’ effectiveness, applicability and efficiency, all of which can contribute to a better evidence-base for informed decision making. This is particularly the case when implementation of a policy or program is studied as experimentation in practice (Petticrew et al., 2005; Craig, Cooper, Gunnell, Gunnell, & Gunnell, 2012; Banks, 2009; Palfrey, Thomas, & Phillips, 2012; Patton, 1997). There is also a growing expectation that evaluation should be used by policymakers to inform public health policies and programs (Oxman et al., 2010; Chalmers, 2003; NSW Government, 2013; Patton, 1997) and in theory, use of evaluation should have synergies with principles of new public management style public service, such as those widely adopted in countries such as United Kingdom, Australia, Scandinavia and North America (Dahler-Larsen, 2009; Johnston, 2000).

There is a wide range of literature on policy and program evaluation, including considerable debate on appropriate evalua- tion methodologies, (DeGroff & Cargo, 2009; Learmonth, 2000; Rychetnik, 2010; Smith & Petticrew, 2010; White, 2010) and the

C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215 209

challenges of evaluation research uptake (Lomas, 2000; Dobbins et al., 2009; Chelimsky, 1987; Weiss, 1970). Amongst this literature there is awareness that evaluation of public health policies and programs is complex, highly context dependent, plagued by ideological debate and subject to ongoing challenges of research- er—policy collaboration (Flitcroft, Gillespie, Salkeld, Carter, & Trevena, 2011; Ross, Lavis, & Rodriguez, 2003; Haynes et al., 2012; Petticrew, 2013). There are however gaps in the literature about barriers and facilitators to undertaking quality evaluation at the level of public service where evaluations tend to be initiated and commissioned, namely in middle management and by program directors. There are also few studies that deliberately seek to gather, compare and analyse perceived barriers from both policy makers and evaluation researchers operating within the same policy space.

In this paper, we aim to explore the early stages of policy and program evaluation, including the decision to evaluate, preparing for evaluation, commissioning evaluations and how policymakers and evaluation researchers work together for the purpose of high quality evaluation undertaken as part of evaluation research. Our primary aim is to:

� Identify barriers and facilitators to evaluation at the initiation and commissioning stages, from within both the policy making as well as evaluation research environments.

To achieve this overarching aim, we address the following:

� How evaluations of health policies and programs are planned, commissioned and undertaken in Australia;

� The barriers and facilitators experienced by policymakers and evaluation researchers in the early stages of establishing an evaluation of a policy or program and in undertaking commis- sioned evaluation work; and

� Initiatives that may assist government to overcome barriers to effective evaluation.

The study was designed and undertaken at the Sax Institute, an independent research and knowledge translation institute located in Sydney Australia, to inform the development of programs to assist and improve evaluation practice in health policy.

Box 1. Interview schedule questions.

Questions for policy makers

1. Could you please describe how policies and programs are eva 2. What factors influence decisions to undertake evaluations of p 3. What factors act as barriers or facilitators to undertaking qualit 4. What is the best way to engage with evaluators? 5. Now, thinking about commissioning external evaluations, what

evaluator?

Questions for evaluation researchers

1. From your experience could you please describe how governm 2. What is the quality of such evaluations? 3. What are the barriers and facilitators to undertaking good eva

programs? 4. What is the best way for policy makers to engage researchers/ 5. What can policy makers do to effectively commission an evalu

2. Methods

We undertook 17 semi-structured interviews with a purposive sample of policymakers (n = 9) and evaluation researchers (n = 8) in Australia. Policymakers had a minimum of 10 years of experience in a health policy or program delivery role in either state or federal government. They were also currently employed in a health related government department and had previously been involved with the evaluation of government policies and programs. Evaluation researchers had a minimum of eight years of experience in undertaking evaluations of health or health related policies or programs run by state or federal level government agencies in Australia and were senior academics at a university.

Separate but similar interview schedules were developed for policymakers and evaluation researchers. The interviews com- prised five open-ended questions (see Box 1). The study was approved by the Humanities Human Research Ethics Committee, University of Sydney, Australia. Informed written consent was obtained from all participants.

Policymakers were asked to reflect on their experiences of how public health policies and programs are evaluated, decisions about whether or not to evaluate them, ways in which policymakers engage or work with evaluation researchers and what would help them to commission an evaluation with an external evaluator.

Evaluation researchers were asked about their experiences of how government health policies and programs are evaluated in Australia, working with government agencies and asked about what would facilitate the process of undertaking evaluations of public policies and programs.

Sixteen interviews were voice-recorded and transcribed and one was documented with detailed notes. We used thematic analysis to draw conclusions from the texts.

We used thematic analysis as a realist method—assuming that by-and-large the responses of the interviewees to questions posed reflect the experiences, meaning and reality of the participants. Coding and identification of themes broadly followed the method of Braun and Clarke (2006) and proceeded in four steps.

First, texts were de-identified and applied with markers to categorise policymakers (PM1-9) or evaluation researchers (ER1- 8). Second, author CHS inductively coded eight texts (four from each category) using a free coding process. Codes were collated and merged within parent codes where possible. Authors CHS and GM

luated in your agency? olicies or programs (in your experience)? y evaluations?

would help you to commission an evaluation with an external

ent health policies and programs are evaluated?

luations of health or health-related government policies and

evaluators to evaluate government policies and programs? ation?

210 C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215

then grouped parent codes into categories and compiled them in a proto-coding manual. Third, nine transcripts were distributed amongst all authors, CHS, AM, GM to test-code using the proto- coding manual. Where necessary codes were supplemented and/or merged and re-categorised. The coding manual was finalised comprising five main categories, each with several sub-categories of codes.

1) Planning and commissioning evaluations of policies and programs

2) Relationships between policy makers and evaluation research- ers

3) Conducting evaluations of policies and programs 4) Utility of evaluation 5) Current evaluation practice

Fourth, author CHS re-coded all texts using the coding manual. All authors then reviewed each code to interpret meaning and emerging themes.

3. Results

Evaluations of government programs in Australia are generally initiated and commissioned at the level of the program or policy to be evaluated � though this may occur within higher level evaluation frameworks. Program managers or directors of a programmatic unit may take direct responsibility for setting initial scope requirements of an evaluation and establishing a committee structure to oversee competitive or selective–tender- ing to an external evaluator. In some government agencies there may be specific programmatic units that perform internal evaluations or facilitate the process of commissioning evaluations and establishing partnerships with research institutions. Program level managers and staff usually take on the role of facilitating access to data, site visits and other necessary permissions for evaluation work to be undertaken. Models of engagement between evaluation researchers and policy agencies vary from case to case. Respondents in our interviews described a range of ‘contracting out’ models, such as competitive and selective tender, pre-qualification/panel schemes as well as ‘collaborative engagement’ models such as co-production and researcher initiated evaluation. A discussion of these types of models of engagement is reported on elsewhere [Reference removed in blinded manuscript].

Here, we discuss results of the interviews that shed light on our main aim: identifying barriers and facilitators to evaluation from within both the policy making as well as evaluation research environments. A broad array of themes emerged from the interviews that related to experiences that facilitated or acted as a barrier to evaluation. Of these, six were consistently raised as particularly important by a high number of study participants: timeframes (15 participants), political influence (12 participants), funding (10 participants), having a ‘culture of evaluation’ (9 participants), caution over anticipated evaluation outcomes (8 participants) and skills and ability of policy agency staff (8 participants).

3.1. Timeframes

Evaluation timeframes was the issue raised by more study participants than any other. It acted as a barrier or facilitator in three ways:

First, evaluations are often initiated late in the policy or program implementation process. This can have several conse- quences: funding to undertake evaluation can simply run out; or the situation arises that no adequate data has been collected at an

early stage of implementation that can act as a baseline. There may also simply be inadequate time to undertake data collection or conduct analysis for rigorous evaluation. Many interviewees found that the consequence of starting late was to undertake small evaluations limited in scope, often evaluating processes rather than outcome. One interviewee reported on limiting the evalua- tion of a complex intervention on childhood well-being to a review of relationships between stakeholders in the process.

“One of the things that makes it difficult is when something has been implemented and then ‘oh, let’s do an evaluation’. So you don’t have baseline data, it hasn’t been set up to exclude all sorts of other factors that may have brought about change” (ER7)

Second, timeframes are often too short to conduct an evaluation. The vast majority of public health policies and programs will require years to have a measureable impact on health outcomes. For example, one interviewee had been involved in evaluation reporting for obesity prevention programs that had to focus on logic of assumptions to conduct the evaluation.

“But timeframe is another problem I think. Often, the process as a new program gets rolled out and it’s expected to have massive impact somehow within a 12 month period and it’s just not a realistic timeframe to be able to demonstrate change” (ER3)

The time required to design a complex evaluation and undertake complex analysis is also beyond what is feasible in many of the short timeframes of evaluations.

“I mean if you want to do a proper evaluation you can’t have a turnaround time of one month or something like six weeks. You know if you’re going to do proper focus groups or you know, design questionnaires . . . and I notice a lot of times they want this quick turnaround time” (PM1)

Furthermore, timeframes further limit who might be engaged to undertake evaluations. Academic researchers are less likely to be able to dedicate time to an evaluation that needs to be done quickly, especially if requiring the recruitment of staff.

“But one of the barriers for academic groups being part of it is the timeframes. So you know you’ll normally sort of have to put in your tender bid very quickly and then you normally have to start work on it instantaneously, and academic groups are not always able to be quite so nimble as that” (ER5)

Third, policymakers reported general lack of time between their everyday tasks to dedicate to careful consideration of evaluation planning, including program objectives and evaluation questions.

“But because of the kind of immediacy of the job, and even if the intellectual capacity is there to really engage with the process there often isn’t the time. You know the writing of briefs, and advising people, and doing this and that. I know it is about getting the right balance . . . but it limits the time available to sit down and really nut out your questions” (PM4)

3.2. Political influence

Many participants had experienced political imperatives as a major barrier to conducting evaluations; often citing it as a more significant issue than funding, timing or the knowledge and skills of agency staff.

“I think we tend to complain that there aren’t resources and that there isn’t enough people around with the ability to do it. To me those are important but they’re much less significant than the political cloud that surrounds the whole business” (ER1).

Political imperatives included those situations where political preferences influenced whether an evaluation goes ahead; pressure from higher up to have a policy or program succeed or

C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215 211

not; and any decisions based on the sensitivity of a policy or program. Certain issue areas were more likely to be subject to ideological debate and therefore political pressure than others; as was the case for certain approaches. In public health for example, debate between community vs individual responsibility as a basic paradigm that underlies policy decisions can make one issue area or policy response more politically sensitive than another. In some cases public health programs and policies can proceed at arms- length to such debates, in other cases they can be heavily influenced or weighed down by political imperatives and uncertainty (Head & Alford, 2015).

Political influence was seen to be stronger at more senior levels of government agencies. Election cycles reduced timeframes for action and subsequently evaluation and reporting. The time required for a policy or program to actually show an effect may thus lie outside of the available timeframe required.

“Policymakers obviously work with a lot of constraints and political constraints. Even when it is personally recognised that they really need is five years to see if something is making a difference they are working in an environment that doesn’t allow for that” (ER3)

Political ideology and experiences of highest level public servants and elected officials were seen to trickle down into government agencies determining what programs are prioritised or placed under scrutiny.

“I wouldn’t do this—but it’s possible that if something’s coming under attack and . . . or is contentious and either the government of the minister or the department thinks that it’s a valuable program that they’ll construct an evaluation or research around it in such a way that it will deliver the answers they’re looking for” (PM7)

Political imperatives not only influenced if, when and with what funding a policy or program might be evaluated, but indirectly determined the focus of evaluations. For example, there may be a preference for measuring cost-utility rather than population health outcomes. In some situations, it may be more important to be immediately seen to be ‘doing something’ rather than having an actual effect.

“If it is a politically motivated process, then the aim is to be recognised for doing the program or having the policy. The actual ‘Was it really effective?’ . . . I don’t think those questions are always required or wanted” (ER2)

Both evaluation researchers and policymakers recognised the frustration than can result when evaluation results and recom- mendations formulated from them are not acted upon.

3.3. Funding

Inadequate funding for evaluation was generally seen to compromise the quality of evaluations. It was also linked to evaluations that were initiated late in the implementation phase and sometimes after funding had run out.

“So barriers—always money . . . We used to occasionally find out some of the more complex analyses, and if you don’t have money to do that then it obviously stymies your ability to do some of the more complex stuff” (PM6)

Interviewees did not however experience a lack of funding for policies and programs per se, but rather that too little funding was allocated specifically for evaluation in the policy and program design phase.

“They had money for the implementation of it but no money had been set aside for the evaluation for example. It was how the money was packaged” (ER5)

Having evaluation written in to the program design from the beginning was seen to increase the likelihood that adequate funding for the evaluation would be available.

“If it’s written in that the program’s going to have an evaluation, then that would help, and if there’s resources identified to do it, both money and people—and skills so the whole set” (PM9)

Lack of funding also influenced whether an evaluation would be undertaken with external evaluators or undertaken in-house. In situations where agency resources are scarce and there were fewer opportunities for staff to develop skills, the quality of evaluations was seen to be compromised, or at least limited in methodological scope.

3.4. Culture of evaluation

A culture of evaluation with a government agency refers to a value being placed on evaluation in all decision making processes such as priority setting, program design and implementation, and funding (Dobbins et al., 2009). Both policymakers and evaluation researchers connected the extent to which such values were present within a government agency as a key facilitator to good evaluation practice. However, it was an area commonly identified for potential improvement.

“ . . . so I really think it just depends on whether people are focused on doing them or not” (ER1)

A culture of evaluation was seen to require a high degree of motivation to evaluate policies and programs, and opportunities to share experiences, positive and negative, with colleagues. It was also linked with having high level championing of evaluation and a staff mix with skills, experience and motivation to integrate evaluation into standard practice.

“I mean it’s probably something to do with people that are in decision-making roles I expect � how much of an emphasis they place on evaluation and then how much they know about it” (PM9)

A culture of evaluation was seen to have the potential to counteract some of the nervousness surrounding evaluation. A poor evaluation culture was characterised by a lack of support for government agency staff that were confused about how to conduct an evaluation.

3.5. Caution over anticipated outcomes

Several policymakers and evaluation researchers recalled experiences where fear or worry of possible negative evaluation results (ie. results showing that a program failed to have an effect, or had made a situation worse) acted as a barrier to evaluation. This fear appeared to act as a more significant barrier for large scale programs or policies.

“not actually wanting to deal with the unpalatable truth then obviously that’s a major barrier” (ER2)

In many cases, evaluations are undertaken specifically to demonstrate positive results and participants reported that briefs for external evaluators were written in a way that implies a certain result is expected.

“Of course there are some agendas behind evaluations, where the aim is to snow success rather than learn” (PM2)

This can influence the eventual mode of engagement with external researchers; private consulting firms were often seen as more likely to report positively on policies and programs, in a way that is less of a risk in terms of consequences of possible negative evaluation outcomes. It also led to situations where there was an understanding that results of evaluation would not be published; and in such cases there was a concern that quality of evaluations

212 C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215

was generally lower due to the absence of expectations of public scrutiny.

“Within government, and also most organisations, people would like to see very positive stories coming out, and tend to be quite wary of independent groups taking a look at what they are doing. Because you’ll never know what they’ll say” (ER7)

Interviewees postulated that the underlying cause of this degree of caution was a need to demonstrate positive outcomes to ensure ongoing funding, but also a general culture of not being allowed to fail. One policy maker suggested that evaluators can and should take on the task of alleviating this fear of evaluation by engaging in an ongoing dialogue with program staff.

“I think a big barrier to conducting good evaluations is that there is a fear that we will have to say what went wrong, and in the climate (we) operate in, that’s a big no-no” (PM5)

3.6. Skills and ability of policy agency staff

Having policy agency staff with adequate skills and training to undertake evaluation was identified as a key facilitator to evaluation. The skills of policymakers in evaluation research were seen to vary greatly, but to be more often inadequate than adequate. Some policymakers rated their own skills in evaluation research as poor, but generally it was remarked that there was a variety of skillsets within policy agencies.

Poor evaluation skills were considered a major barrier to the quality of evaluation, even for evaluations undertaken externally, because of the high level of understanding required to prepare for commissioning an evaluation. Several evaluation researchers had experienced poorly defined program objectives, confusing evalua- tion questions and over estimations in the amount of evaluation work that can be done within a particular timeframe or setting.

“I think if a group wants to do an evaluation and is going to write a brief it requires great technical skill about evaluation to write the brief in the first place” (ER5)

The skills mix of staff in an agency, or unit within an agency is also seen to have a broader influence on the culture of evaluation; with a lack of knowledge or interest on the part of some staff acting as a barrier to evaluation becoming a consistent practice. There was also a recognition that evaluation is in some cases particularly complex and requires specialist skills that go beyond a basic understanding of evaluation.

Several participants reported that this can be resolved, at least in part, by ensuring all staff within a government agency had access to staff with evaluation expertise.

“At my previous department we had an evaluation manager who worked full time on working through evaluation processes, doing that early thinking. It was very important that person was around and engaged” (PM2)

3.7. Further barriers and facilitators

Additional facilitators identified through the interviews includ- ed:

� Having a standardised–or partly regulated–approach to evalua- tion. This comprised two aspects: firstly integrating evaluation in standard processes was seen as assisting in building a culture of evaluation over time, second, having tools and checklists available to staff to assist in scoping and when applicable designing evaluations was seen to alleviate some apprehension about evaluation and clarify thinking amongst staff.

“I think having templates for contracting, or if not templates just cheat sheets or guidelines or something like that. It would be good to have prompts to consider. Sometimes it’s hard to know what it is you actually want, or what you should do” (PM2)

� Champions and leadership. The presence of a person or persons to drive evaluation thinking and practice was seen as increasing the likelihood of evaluation taking place and of its being of high quality. Having persons at a high level take on this role was considered by one policymaker to be the most important driver of evaluation.

“Support from senior executive staff can make or break an evaluation” (PM8) “Often the important thing there was to have a champion for the work” (ER1)

Additional barriers identified were:

� High turnover in government departments. Evaluation research- ers and policymakers experienced frequent changes in staffing as a barrier to evaluation. Evaluation researchers frequently had contact with several different policy and program managers during the course of an evaluation. This led to changing expectations and confusion about evaluation purpose—exacer- bating the need to clearly and formally document roles, responsibilities, expectations and evaluation parameters for evaluation researchers. Evaluation researchers sometimes felt that they took on a role of holding the institutional memory of a policy or program. “ . . . there’s a big tendency for a turnover of personnel in government agencies and therefore very often the person who receives an evaluation report or receives the evaluation outcomes is not the same person that commissioned it . . . . Then there have been situations where yes, we’ve tendered for something, same sort of situation has occurred, the person who wrote the tender, even by the time the tender is determined, has moved on” (ER1)

� Cumbersome approval processes. Evaluation researchers men- tioned unexpected delays due to several layers of administrative process as a source of frustration and barrier to quality evaluation. Policymakers recognised the importance of bureau- cratic checks, but also found that the time spent on ensuring that due process is followed took away from time to dedicate to designing and preparing for rigorous evaluation. Some policy- makers also found that some processes led to poor decisions � e.g. engaging the wrong evaluation team.

“We work so fast, yet things happen so slow! So it’s kind of odd, you always feel you are in a hurry, and think something like, I’d really like to sit down and give that contract a lot more attention to detail. And yet, for some reason you don’t even through the whole process actually takes a very long time” (PM4)

3.8. The policymaker—researcher gap

Responses on barriers and facilitators to evaluation were quite similar amongst evaluation researchers and policymakers indicat- ing a high level of understanding for both the policy and research environment – or at least the challenges they each face – amongst all study participants. Despite this, 16 of the 17 participants reflected on the dynamics of policy-researcher relationships as particularly challenging part of evaluating government policies and programs. In particular, different career priorities and motivations to undertake evaluation, physical and conceptual

C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215 213

separation of the policy and researcher worlds, intellectual property concerns, and trust were identified as key issues.

3.9. Different career priorities and motivation to undertake evaluation

Participants found that different career imperatives influenced motivations to undertake evaluation. The need to publish; gain recognition amongst the scientific community and contribute to the wider body of knowledge were recognised as key motivations of evaluation researchers.

“I think working with the academic sector, the quality again is good, but I think there can be conflation between our requirements and the requirements of academics to publish. Essentially their KPIs are different to ours” (PM4)

This led to challenges in both finding evaluation researchers willing to undertake evaluations of government programs and to a need to negotiate suitable conditions; such as the degree of autonomy to determine evaluation questions and the right to publish results.

“For researchers it’s usually about having independence and having some sort of right to the intellectual property of what they generate and that they’re able to publish it, and for the (government agency) it’s obviously to have useful and relevant questions answered. But I do think there’s scope for those two to actually align” (ER2) “ . . . then the issue of academic freedom comes up and whether you know, the university can publish it without any sort of interference from whoever’s funding or commissioning the evaluation” (PM3)

3.10. Intellectual property concerns

Any commissioned evaluation requires some form of formal contractual agreement to protect both parties from unreasonable obligations. Several participants noted that rules over intellectual property are a particular challenge during the phase of contractual negotiation. It also influences evaluation researchers’ decisions whether to undertake evaluation of government policies and programs, and for policymakers whether to engage with evalua- tion researchers from the academic sector.

“The IP stuff’s quite complicated . . . Not all researchers are academic and so some of them won’t care that they don’t publish. But certainly for an academic environment it is (an issue).” (ER6)

3.11. Trust

Some evaluation researchers expressed a degree of mistrust with regards to the motivations of policymakers seeking to commission out evaluation. There was a fear that original or creative input might be used without acknowledgement or passed on to others and used for their benefit.

“What’s in it for you to have that conversation with people and provide advice . . . they might just steal your ideas and give it to someone else. There’s always that concern in the academic environment. You have to be a bit wary about that unless you had an expectation of continuing that’s true” (ER6)

3.12. Physical and conceptual separation

Several participants also commented on the separation of the policy and research worlds. This included both a physical separation – rarely coming in contact with each other – as well as basic differences in understanding each other’s work context.

“A lot of people I think work in silos . . . the academics go to conferences, so they just stay in their own sort of groups, the policy makers they do their own different things. So there needs to be a better opportunity or some sort of vehicle where the two can work together” (ER8)

Evaluation researchers experienced difficulties working with government agencies due to a lack of understanding of context and politics that surrounds individual policies and programs.

“But certainly my own experience of my difficulties of doing evaluations for government agencies has been that I didn’t fully appreciate what the situation underlying the problem was . . . and that invariably meant that what I did, however good it was, well, whether it was good or not, didn’t necessarily hit the mark” (ER1)

Several participants however indicated that this separation of worlds was more dependent on institutional constraints rather than fundamentally different views of knowledge generation and the value of evaluation.

“There really is no ‘us’ and ‘them’ there is just one big mushy us. It’s the imperatives, the organisations that we work for that makes the difference. I think we easily forget that we have a shared discourse, we know what to ask for, but we don’t quite do it” (PM4)

4. Discussion

The challenges raised by the participants can be understood as comprising micro (individual behavioural), meso (organisational) and macro (context and system) factors that influence evaluation of public health policies and programs (Fig. 1).

Setting the context, political pressure can stem from ideological preferences of high level decision makers; the nature of the democratic system; as well as personal disputes or strategic manoeuvring (Flitcroft et al., 2011; O’Brien, Payne, Nolan, & Ingleton, 2010; Liverani, Hawkins, & Parkhurst, 2013). Public service in Australia has widely adopted a new public management model in which contract and performance management arrange- ments are used to set outcome targets and management of programs is devolved to lower organisational levels. Decision making within this environment is inherently a mix of strategic choices. This can create a feedback loop whereby political pressures, and even standard reporting requirements, can limit the amount and quality of evaluation, which in turn limits the amount of evidence that can inform, or hold weight, in decision making. In some instances this feedback loop may be intended, in others an unintended consequence (Palfrey et al., 2012; Chelimsky, 1987). Our research indicates that these constraints then manifest as two main system level barriers. First, funding challenges including not having adequate, or any, program specific funding allocated to evaluation, or adequate funding allocated to develop- ing skills or hiring staff with specific roles dedicated to championing evaluation. Second, restrictive timeframes including having to fit in with election and media cycles, or to justify (re-) financing towards the end of finite funding and reporting periods. Short evaluation timeframes limit the feasibility of many study designs and hence can compromise the quality of evaluations (Atienza & King, 2002).

Macro level barriers have a direct influence on several features of government agencies that were identified by the participants on the organisational – or meso – level as well as individual – micro – level. Institutional realities of government agencies such as a high turnover of staff – moving in and out of agencies as well as between roles within agencies – and the need to work within cumbersome approval processes further limit the amount of time that policy- makers were able to dedicate to preparing for evaluations,

Fig. 1. Macro, meso and micro level barriers to evaluation.

214 C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215

including defining program objectives, evaluation scope and developing partnerships with evaluation researchers and exacer- bating time pressures from the macro level (South & Tilford, 2000). Furthermore, short term funding can create an organisational culture that fears evaluation rather than promotes it, reducing the breadth of evaluation within a government agency and the capacity to prepare and undertake quality evaluation (Dobbins et al., 2009).

To overcome these challenges, several approaches directly targeting meso and micro levels may be promising, such as having tools, checklists or cheat-sheets that can increase staff capacity to prepare and undertake evaluation and contribute to an improved culture of evaluation. It was also suggested staff with a good understanding of evaluation will be less likely to fear it, although it is not clear to what extent this would counteract strong political and funding imperatives. It is however known that training to increase skills of agency staff – a micro level factor – can have an effect on changing organisation culture, in particular with regards to “the value placed on using research evidence in decision making, and the expectation to demonstrate use of research evidence in decision making” (Dobbins et al., 2009). The presence of champions that drive evaluation, can serve as key points of contact for staff and external evaluators and high level evaluation support were also identified as important facilitators.

The dynamics of policy-researcher relationship provide a further challenge to evaluating government policies and programs. We recognise that many policymakers have experience in academic research and that many evaluation researchers have worked in government agencies. Indeed, among our interviewees there was a high degree of apparent familiarity with each other’s worlds. Despite this, constraints appear to persist that stem directly from a policy-researcher gap, even if these differences were perceived to comprise meso level constraints at the institutional level such as changing liaison persons and intellectual property disputes, rather than fundamentally different views of knowledge generation and the value of evaluation research (Wingens, 1990; Wehrens, 2013; Johnson, Greenseid, & Toal, 2009). Even so, micro level factors such as different career priorities and motivations to undertake evaluation, physical and abstract separation of the policy and researchers worlds concerns, and trust were identified as important issues.

Our results suggest that good quality preparation of evaluation, including clear program objectives and evaluation questions, expectations regarding the role evaluation researchers, under- standing and flexibility with regards to methodologies, allowing adequate academic freedom and openness to publication can act as facilitators and increase high quality evaluation research. Inter- ventions that either increase skills in these areas, or partly outsource evaluation preparation might therefore be considered. In mirroring the barriers that have been identified, there also needs to be a focus on system and organisational factors that create incentives to improve evaluation practice, including adequate timeframes and financing.

Overcoming barriers therefore requires policy for policy- making, ie. a set of rules and incentives that drive paradigm change within the bureaucracy (Trochim, 2009). In the late 2000s, Trochim and colleagues had identified that while most govern- ments have evaluation policy it usually remains ad hoc and even conflicting across sectors (Cooksy, Mark, & Trochim, 2009; Trochim, 2009). Consistent evaluation policy may be able to advance the facilitators to good evaluation listed above through combinations of carrots, sticks and sermons. In several states in Australia, in particular New South Wales and Queensland, there have been a series of recent attempts to strengthen evaluation with high level evaluation policy. These initiatives include stipulation of the expectation to evaluate all programs across whole of government (sermons); incentives to build capacity in evaluation amongst government workers (carrots) and mandating the requirement to include evaluation plans for new program proposals at the point of expenditure review (sticks) (NSW Government, 2013; Queensland Government, 2014; Treasury, 2013; NSW Government, 2016). The rationale behind these methods appears to be the creation of high level incentives that will filter down through government agencies creating an environment in which upskilling of staff, timely commissioning of evaluations and adequate planning will be enabled (Boyle, Lemaire, & Rist, 1999). Consistent rules of what should be evaluated, and when, are also intended to act as a barrier between higher level strategic and political manoeuvring program level management. Meso and micro level initiatives already mentioned could run parallel to these strategies— strengthening the ‘evalua- tion culture’ within departments and sub-units within

C. Huckel Schneider et al. / Evaluation and Program Planning 58 (2016) 208–215 215

government to develop resilience to barriers to good evaluation practice (Sanderson, 2001).

5. Conclusion and implications

There is a relationship between the political nature of public health policy and programming and barriers to evaluation. Clarifying the specific nature of barriers to evaluation can elucidate appropriate ways through which policy makers can be supported when seeking to evaluate their programs by means of creating a culture positively attuned to evaluation. Responses to can include both high-level evaluation policy to create incentives to change culture as well as direct initiatives aimed at those levels of government where evaluations of policies and programs tend to be initiated and commissioned.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The Sax Institute receives a grant from the NSW Ministry of Health.

Acknowledgements

The authors thank discussants and participants for comments on an earlier draft presented at the International Conference on Public Policy, Grenoble, France, June 2013.

References

Atienza, A. A., & King, A. C. (2002). Community-based health intervention trials: An overview of methodological issues. Epidemiologic Reviews, 24, 72–79.

Banks, G. (2009). Challenges of evidence-Based policy making. Productivity Commision, Australian Government.

Boyle, R., Lemaire, D., & Rist, R. C. (1999). Introduction: Building evaluation capacity. In R. Boyle, & D. Lemaire (Eds.), Building evaluation capacity: Lessons from practiceNew Brunswick: Transaction Publishers.

Braun, V., & Clarke, V. (2006). Using thematic analysis in Psychology. Qualitative Research in Psychology, 3, 77–101.

Chalmers, I. (2003). Trying to do more good than harm in policy and practice: The role of rigorous, transparent, up-to-date evaluations. Annals of American Academy of Political and Social Science, 589, 22–40.

Chelimsky, E. (1987). The politics of program evaluation. Society, 25, 24–32. Cooksy, L. J., Mark, M. M., & Trochim, W. M. K. (2009). Evaluation policy and

evaluation practice: Where do we go from here? New Directions for Evaluation, 2009, 103–109.

Craig, P., Cooper, C., Gunnell, D., Haw, S., Lawson, K., Macintyre, S., et al. (2012). Using natural experiments to evaluate population health interventions: New Medical Research Council guidance. Journal of Epidemiology and Community Health, 66, 1182–1186.

Dahler-Larsen, P. (2009). Evaluation and public management. In E. Ferlie, L. E. Lynn, & C. Pollitt (Eds.), The oxford handbook of public managementOxford: Oxford University Press.

DeGroff, A., & Cargo, M. (2009). Policy implementation: Implications for evaluation. New Directions for Evaluation47–60.

Dobbins, M., Hanna, S. E., Ciliska, D., Manske, D., Cameron, R., Mercer, S. L., et al. (2009). A randomised controlled trial evaluating the impact of knowledge translation and exchange strategies. Implementation Science, 4.

Evans, B. A., Snooks, H., Howson, H., & Davies, M. (2013). How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study. Implementation Science, 8.

Flitcroft, K., Gillespie, J., Salkeld, G., Carter, S., & Trevena, L. (2011). Getting evidence into policy: The need for deliberative strategies? Social Science & Medicine, 72, 1039–1046.

Haynes, A. S., Derrick, G. E., Redman, S., Hall, W. D., Gillespie, J. A., Chapman, S., et al. (2012). Indentifying Trustworth Experts: How do policymakers find and assess public health reserachers worth consulting or collaborating with? PLoS One7.

Head, B. W., & Alford, J. (2015). Wicked problems: Implications for public policy and management. Administration & Society, 47, 711–739.

House, E. R. (2004). The role of the evaluator in a political world. Canadian Journal of Program Evaluation, 19, 1–16.

Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009). Research on evaluation use: A review of the Empirical literature from 1986 to 2005. American Journal of Evaluation, 30, .

Johnston, J. (2000). The new public management in Australia. Administrative Theory & Praxis, 22, 345–368.

Learmonth, A. M. (2000). Utilizing research in practice and generating evidence from practice. Health Education Research, 15, 743–756.

Liverani, M., Hawkins, B., & Parkhurst, J. O. (2013). Political and institutional influences on the use of evidence in public health policy. A systematic review. PLoS One, 8, e77404 [Electronic Resource].

Lomas, J. (2000). Connecting research and policy. Isuma Canadian Journal of Policy Research140–144.

NSW Government (2013). NSW Government Evaluation Framework August 2013. Sydney.

NSW Government (2016). NSW Government Program Evaluation Guidelines. Sydney.

O’Brien, T., Payne, S., Nolan, M., & Ingleton, C. (2010). Unpacking the politics of evaluation: A dramaturgical analysis. Evaluation, 16, 431–444.

Oxman, A., Bjorndal, A., Becerra-Posada, F., et al. (2010). A framework for mandatory impact evaluation to ensure well informed public policy decisions. The Lancet, 375, 427–431.

Palfrey, C., Thomas, P., & Phillips, C. (2012). Evaluation for the real world: The impact of evidence in policy making. Bristol: Policy Press.

Parsons, W. (2002). From muddling through to muddling up—Evidence based policy making and the modernisation of British government. Public Policy and Administration, 17, 43–60.

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text. Thousand Oaks: Sage.

Petticrew, M., Cummins, S., Ferrell, C., Findlay, A., Higgins, C., Hoy, C., et al. (2005). Natural experiments: An underused tool for public health? Public Health, 119, 751–757.

Petticrew, M. (2013). Public health evaluation: Epistemological challenges to evidence production and use. Evidence and Policy, 9, 87–95.

Queensland Government (2014). Program Evaluation Guidelines November 2014. Ross, S., Lavis, J., Rodriguez, C., et al. (2003). Partnership experiences: Involving

decision-makers in the research process. Journal of Health Services Research & Policy, 8(Suppl. 2), 26–34.

Rychetnik, L. (2010). Is an ‘evaluation jigsaw’ a feasible way forward? Journal of Public Health (Oxford), 32, 10–11.

Sanderson, I. (2001). Performance management, evaluation and learning in ‘Modern’ local government. Public Administration, 79, 297–313.

Smith, R. D., & Petticrew, M. (2010). Public health evaluation in the twenty-first century: Time to see the wood as well as the trees. Journal of Public Health (Oxford, England), 32, 2–7.

South, J., & Tilford, S. (2000). Perceptions of research and evaluation in health promotion practice and influences on activity. Health Education Research, 15, 729–741.

Treasury, H.M. (2013). The Green Book: Appraisal and Evaluation in Central Government.

Trochim, W. M. K. (2009). Evaluation policy and evaluation practice. New Directions for Evaluation, 2009, 13–32.

Wehrens, R. (2013). Beyond Two Communities. The co-production or research, policy and practice in collaborative public health settings. Rotterdam: Erasmus Universiteit Rotterdam.

Weiss, C. (1970). The politicization of evaluation research. Journal of Social Issues, 26, 57–68.

White, H. (2010). A contribution to current debates in impact evaluation. Evaluation, 16, 153–164.

Wingens, M. (1990). Towards a general utilization theory. Science and Communication, 12, 27–42.