Project Management

Public Performance & Management Review, Vol. 35, No. 3, March 2012, pp. 550–563. © 2012 M.E. Sharpe, Inc. All rights reserved.

1530-9576/2012 $9.50 + 0.00. DOI 10.2753/PMR1530-9576350309550

RESultS FIRSt

using Evidence-Based Policy Models in State Policymaking

GARy R. VANlANDINGhAM Pew Center on the States

ElIzABEth K. DRAKE Washington State Institute for Public Policy

AbstrAct: Cost-benefit analysis has been used with some success at the federal level for many years, but states have generally made limited use of this approach. They are now showing growing interest in evidence-based policy and budget decision making, due in part to their ongoing deep budget difficulties. Several promising developments have increased the probability of success, including methodological advances and the development of user-friendly cost- benefit analysis models by the Washington State Institute for Public Policy. These have been integrated into Washington State’s policy and budget processes for more than 15 years. The Results First initiative of the Pew Center on the States is helping other states to implement and customize these models. While the effort is just beginning, early outcomes are encouraging.

Keywords: benefit-cost analysis, budgets, cost-benefit analysis, evidence- based programs, states

there is growing interest in using evidence-based policy models to guide state policy and budget decisions. Generally based on cost-benefit or cost-effectiveness analyses that estimate the costs and outcomes of policy alternatives, these models can help policymakers make more informed budget and policy choices. these approaches have been used at the federal level for many years (with mixed suc- cess) but have had more limited use at the state level. In light of their ongoing fiscal difficulties, states are taking another look at this approach. the Washington State Institute for Public Policy (WSIPP) provides a useful case study on the potential impact of cost-benefit analysis in state policymaking. the cutting-edge cost-benefit analysis models developed by WSIPP have been used by the state for more than 15 years to help guide policy and budget choices (Drake, 2010). the Results First Initiative of the Pew Center on the States is helping other states implement WSIPP’s cost-benefit model. this effort is just beginning, but early results are encouraging.

Vanlandingham and Drake / RESultS FIRSt 551

Cost-benefit analysis is an analytical approach that attempts to identify and compare the costs and benefits of one or more policy alternatives. It does so by projecting and assigning current dollar values to the predicted outcomes of the policy choices, ideally including all their direct and indirect effects throughout society. It then compares the summed predicted costs and benefits of each alter- native to determine whether the policy choice(s) would generate a net positive benefit to society. this is typically reported as a cost-benefit ratio. A 2.3:1 ratio, for instance, indicates that an alternative would generate $2.3 in value throughout society for every $1 in costs. the approach is intended to help decision makers ensure that their choices will generate social benefits that outweigh their costs. It also can be used to compare multiple options to identify which of them would generate the greatest efficiency, as measured by the predicted greatest return on investment (Boardman, Greenberg, Vining, & Weimer, 2006; Shaffer, 2010; Vining & Weimer, 2010). Cost-effectiveness analysis follows a similar logic in comparing alternatives but typically does not assign a dollar value to costs and benefits that do not have available market prices, such as the dollar value of a hu- man life. Instead, it often compares alternative means of achieving a desired end by reporting results, such as the average per-person cost of completing different drug treatment programs (Boardman et al., 2006; Brent, 2006).

these approaches have been used at the federal level for many years. In 1920, Congress required the u.S. Army Corps of Engineers to recommend only water projects that would produce benefits exceeding their costs, and in 1936, it required the Corps to evaluate all benefits and costs of such projects (zerbe, Davis, Garland, & Scott, 2010). the Corps subsequently rejected more than half the proposed water projects it assessed using this approach, after the analyses found that the projects’ costs were not justified by their potential benefits (Porter, 1995).

Federal use of cost-benefit analysis approaches substantially expanded during the 1960s, when the studies were mandated by the Department of Defense as part of the Planning and Programming Budgeting System initiative, which was later used throughout the executive branch (Fuchs & Anderson, 1987). Although that effort was unsuccessful and subsequently abandoned, the Nixon administration incorporated the approaches into its review process for federal regulations, and this application was broadened by President Ronald Reagan’s Executive Order 12291, requiring agencies to show that regulations would generate benefits exceeding their potential costs to society. President Bill Clinton’s Executive Order 12866 similarly required agencies to assess the costs and benefits of regulations, using both quantitative and qualitative methods. Circular A-4, issued by the George W. Bush administration, prescribed the methods federal agencies were to use when conducting cost-benefit analyses. the Barack Obama administration has required federal agencies to use cost-benefit analysis and to demonstrate that their funding priorities are based on credible empirical evidence (zerbe et al., 2010).

552 PPMR / March 2012

States have made less use of cost-benefit analysis, although they are showing increasing interest in this approach. A May 2011 search of state statutes identified 207 mandates to conduct these studies (the search of the 50 states’ statutes used Westlaw and variations of the terms “cost-benefit analysis” and “cost-effectiveness analysis”). Studies were most frequently mandated in the areas of economic development policy (required in 23 states), communication/information technol- ogy policy (17 states), and health-care policy (15 states). Almost all states (45) have established one or more such requirements (see Table 1). Many states are considering enacting more requirements to conduct such studies. A search of bills filed during 2010 state legislative sessions identified 136 bills under consideration in 25 states that, if passed, would create new mandates for cost-benefit and cost- effectiveness analysis (National Conference of State legislatures, 2010).

to establish a more comprehensive baseline on state government use of cost- benefit and cost-effectiveness analyses, Pew is conducting a nationwide study of how states use these approaches in their policymaking. to be completed and released in early 2012, it will identify which states are conducting cost-benefit analysis, the policy areas in which they are using this approach, the impact of the analyses in the policymaking process, and lessons learned from the states’ experi- ences (Pew Center on the States, 2011).

A key factor supporting the growing interest in these approaches is the increas- ing budget pressures that are forcing states to make painful choices. Between the start of the Great Recession in Fy 2008 and Fy 2012, state governments collec- tively faced budget shortfalls that exceeded $500 billion. Although state revenues have finally begun to expand, this growth is far outstripped by increased costs of critical programs, particularly Medicaid. the tide of red ink is likely to continue,

table 1. Incidence of state-Mandated cost-benefit/cost-effectiveness Analysis

Policy area Number of states

Budget/Evaluation 7 Communication/Information technology 17 Economic development 23 Education 4 Environment 6 health 15 human services 7 Insurance 7 land/Facilities 9 Procurement 5 Public safety 5 Public utilities 8 Rule making 6 transportation 8 ANy AREA 45

Vanlandingham and Drake / RESultS FIRSt 553

as states are already projecting shortfalls of more than $30 billion for Fy 2013 (Eckl, 2011). the long-term outlook is also problematic, because any action to address the federal budget deficit will probably involve reducing funds currently granted to states. As a result, states will lack the resources needed to sustain their traditional mix of public services and will need to make more strategic choices on how to invest limited resources to achieve key policy goals. Cost-benefit analysis, with its potential for identifying policies that do not generate sufficient benefits to justify their costs, is seen as a way to help eliminate poorly performing programs and thus lessen budget pressures (Graham, 2008).

the renewed academic interest in cost-benefit and cost-effectiveness analyses is helping to expand their theoretical framework to address the challenges of applying the approach to a variety of policy areas. this effort includes the found- ing of the Benefit-Cost Analysis Center at the Evans School of Public Affairs at the university of Washington, which focuses on improving and standardizing methodologies, strengthening relationships between practitioners, disseminating information about the use and misuse of the approaches, and helping to expand their use when appropriate (Evans School, 2011). the Journal of Benefit-Cost Analysis was established in 2010 with the goal of “encouraging and publishing theory, cases, and techniques surrounding this methodology” (Farrow, 2010, p. 1). Recent scholarship has proposed several approaches to deal with critical issues in applying these studies to social policies, including the difficulty of valuing the costs and benefits of policy outcomes that are diffuse and long-term (Vining & Weimer, 2010). For example, it can be challenging to monetize the benefits of early childhood programs intended to generate lifelong outcomes in learning, employment, criminal justice, family, and health.

the John D. and Catherine t. MacArthur Foundation has committed significant resources to supporting these and similar efforts. the foundation has allocated $35 million to its Power of Measuring Social Benefits initiative, which “aims to increase the number of social cost-benefit studies by developing work in at least ten timely and relevant areas of domestic policy, to strengthen methods and to improve measurement of social benefit” (MacArthur Foundation, 2011, p. 1). Its grants have financed cost-benefit studies of specified programs (e.g., home energy conservation investments, health insurance for low-income people), sup- ported academic conferences and research centers, and funded efforts to develop policy analysis models assessing the return on investment that states could achieve through selected policy choices.

barriers to Use of cost-benefit Analysis

As has long been noted, policy research such as cost-benefit and cost-effectiveness analysis faces substantial barriers to use by policymakers. these include political

554 PPMR / March 2012

factors as well as differences between policymakers and researchers in terms of time perspectives, values, acceptance of uncertainty, and interpretation of evidence (Chelimsky, 2009; Radin, 2000; Snow, 1961; Weiss, 1973). Researchers often find it difficult to gain the attention of policymakers because of the amount of compet- ing information provided by other sources, such as lobbyists, think tanks, staff, agencies, and the media. While research typically produces written products, the policymaking environment tends to value oral communication, anecdotal stories, and personal relationships over data (Weiss, 1989; Whiteman, 1995). In addition, because of timing problems, research results often are not available when key decisions must be made (zajano & lochtefeld, 1999).

A wide range of steps have been proposed to address these barriers and promote the utilization of research findings. Much scholarship has identified the importance of researchers seeking stakeholder participation in their studies (e.g., Cousins & Shulha, 2006; Patton, 2008). Researchers are advised to create effective com- munication channels with key stakeholders that facilitate ongoing information exchanges throughout the research process. Exchanges are beneficial because they can help in reaching a consensus about stakeholders’ information needs, the methodologies that can best produce timely and credible results, and how and when results need to be reported; exchanges also promote organizational learn- ing through evaluative inquiry (Preskill & torres, 1999). Such interactions can be particularly important in the policy environment, as decision makers and key staff members tend to rely on insider sources close to the policy process rather than external information sources (Mooney, 1991; Schneider, Scholz, lubell, Mindruta, & Edwardsen, 2003).

wsIPP case study

the Washington State Institute for Public Policy is a useful case study of the im- pact of cost-benefit analysis in state policymaking. Given the limited information about how such research is used by state governments, the case study method is an appropriate choice because it “investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident” (yin, 2003, p. 13). WSIPP, a nonpartisan research unit of Washington’s state legislature, conducts research activities using its own policy analysts and economists, specialists from universities, and consultants. Its staff works closely with legislators, legislative and state agency staff, and experts in the field to ensure that studies answer relevant policy questions. One of WSIPP’s duties is to provide information to the legislature on Washington’s evidence-based initiatives; it has responded to this charge by developing cost-benefit analysis models for key state policy areas. this role has evolved significantly over the past 15 years (Aos et al., 2011; Drake, 2010).

Vanlandingham and Drake / RESultS FIRSt 555

bAcKgroUnd

In the mid-1990s, the state legislature directed WSIPP to identify evidence-based justice programs that could lower the incidence of juvenile crime. In response, WSIPP built its first cost-benefit analytical tool in 1997 to help the legislature select sound investments in Washington’s juvenile justice system (Aos, Barnoski, & lieb, 1998; Aos, Phipps, Barnoski, & lieb, 2001). this initial effort identified several programs—not then operating in Washington—that had potential to reduce crime and save taxpayer money. In subsequent sessions, the legislature used the results to begin a series of policy reforms (Barnoski, 2004). In the early 2000s, following the initial favorable experience in juvenile justice, the legislature directed WSIPP to apply the same cost-benefit approach to other areas of public policy (Aos, lieb, Mayfield, Miller, & Pennucci, 2004; Aos, Miller, & Drake, 2006; Aos, Miller, & Mayfield, 2007; lee, Aos, & Miller, 2008; Pennucci, Aos, & Ngugi, 2008).

Most recently, in 2009 the state legislature directed WSIPP “to calculate the return on investment to taxpayers from evidence-based prevention and intervention programs and policies” (Engrossed Substitute house Bill 1244, State of Washington, 61st legislature, Chapter 564, laws of 2009, Section 610(4), p. 179).1 this assign- ment addressed two fundamental public policy questions: (1) how could the state government better achieve certain public outcomes while providing citizens with a superior return on their tax dollars? and (2) Could the legislature use “evidence” and “costs and benefits” to help craft strategic policies that would lead to measur- able improvements in key statewide outcomes? the legislature asked WSIPP to identify public policies shown to cost-efficiently improve outcomes in crime, K–12 education, child maltreatment, substance abuse, mental health, public health, public assistance, employment, and housing. this effort produced an up-to-date economic analysis of many evidence-based policy choices for the state as well as user-friendly software that will allow Washington legislative and executive staff members and other interested parties easy access to WSIPP’s findings (Aos et al., 2011).

cost-benefIt APProAch

After reviewing national research and drawing conclusions about what works and what does not, WSIPP provides “investment advice” to the legislature. the basic idea is to learn what programs and policies have been successful across the united States and determine whether they would be a good choice for Washington.

Research Evidence on What Works (and What Does Not)

the first step in WSIPP’s process produces estimates of what does and does not work on key topics of legislative interest.2 WSIPP begins by analyzing high-quality research from across the united States to determine which program and policy options have best achieved the desired outcomes and which have not.

556 PPMR / March 2012

Once relevant research questions are established, WSIPP applies a meta-analytic framework to systematically assess the entire research literature on a specific topic. All the credible literature on a topic is reviewed, and effect sizes are screened and adjusted for research-design quality and other factors. this process produces an expected effect, as well as a measure of uncertainty, for the public policy option under study, given the weight of the credible evidence.

Compute the Economics (Costs and Benefits) of Specific Policy Options

the research review reveals whether a given actionable policy option can be expected to achieve desired public outcomes. For example, WSIPP determines whether the weight of the credible evidence indicates that early childhood educa- tion programs improve the academic success of students and, if so, by how much. Once the average effect is estimated, costs and benefits are analyzed by determin- ing how much it costs to produce the effect found in the research review, and how much it is worth to the people in Washington to achieve the outcome.

WSIPP summarizes these economic findings by reporting standard financial statistics: net present value, cost-benefit ratio, and return on investment. Estimates from three distinct perspectives are also presented: the benefits that accrue directly to program participants, the benefits for taxpayers, and benefits from a nonpar- ticipant, nontaxpayer perspective that do not fall into the other categories. the addition of these three perspectives provides a “total state” bottom line.

to continue the previous example, WSIPP’s research review might indicate that an early childhood education program directly benefits participants by increasing their lifetime economic earnings. this research may indicate that the program also directly benefits taxpayers, because some of these earnings will be taxed and the children who attend the programs will have lower rates of criminal behavior later in their lives, which reduces criminal justice system costs. Finally, the research may show that the program benefits nonparticipants in other ways, such as by reducing the costs of criminal victimization. WSIPP adds these three perspectives—participants, taxpayers, and nonparticipants—to produce a total state bottom line; doing so helps answer questions that arise when legislators are considering policy options.

Analyze “Portfolio-Level” Effects

the main products of the first two steps are Consumer Reports–like lists of what works and what does not, ranked by cost-benefit estimates; this information has proven to help Washington legislators as they make decisions. however, WSIPP has also found it useful to estimate how a set of adopted policies is likely to achieve broad public policy goals. In this third analytic step, WSIPP estimates the degree to which a portfolio of adopted policies is likely to affect statewide outcomes, such as reducing crime, while controlling taxpayer spending (Aos, Miller, & Drake, 2006).

Vanlandingham and Drake / RESultS FIRSt 557

to expand this portfolio-analysis approach, the Pew Center on the States contracted in 2010 with WSIPP to develop a sentencing tool that identifies evidence-based policy mixes that can both fight crime and save taxpayer money. It resides in the larger WSIPP cost-benefit model and focuses on the net ef- fects of two types of policies as part of a state’s overall portfolio of evidence- based crime-fighting resources: programming and sentencing. Programming resources refer to governmental efforts aimed at reducing crime, such as drug courts or early childhood education. Sentencing refers broadly to policies that affect the state’s average daily prison population, such as mandates that certain convicted offenders go to prison, requirements specifying the length of stay per sentence, and the discretion granted judges and corrections depart- ments to vary the length of time served in prison for a given sentence. the large research literature on the effect of incarceration rates on crime indicates that certain types of incarceration (the product of sentencing policies) can lower crime rates (Marvell, 2010). Both types of policies influence a state’s incarceration rate—the proportion of its adult population that is behind bars on any average day.

the aim of the sentencing tool is to help states identify reasonable portfolios of evidence-based courses of policy action, given the current state of knowledge about “what works” to reduce crime and save taxpayer money. there are dif- ferential crime-fighting effectiveness and costs of the both programming and sentencing resources. States that wish to fight crime and pinch taxpayer pennies must consider the tradeoffs between sentencing and programming resources as well as the mix of resources that can help lower crime and lower the taxpayer costs of crime.

Uncertainty Analysis to Assess the Riskiness of Bottom-Line Estimates

the final analytical step involves testing the robustness of the results. Single-point bottom lines offer a convenient finding, but because there is a considerable amount of uncertainty in any estimate of benefits and costs, it is important to see how conclusions change when discount rates or other assumptions are altered. to do this, WSIPP uses Monte Carlo simulation to vary the key factors that enter into the calculations, such as treatment size (reflecting the range of standard error of the estimates). uncertainty analysis determines the probability that the estimates would produce a contrary finding—that is, the chance that money would be lost rather than gained if a particular policy were adopted.

the model calculates two bottom-line statistics: the expected value of overall benefits minus costs, and an estimate of the risk that a given strategy could produce negative net benefits. this type of risk and uncertainty analysis is commonly used by many private-sector businesses in decision making.

558 PPMR / March 2012

PolIcy IMPActs

WSIPP’s cost-benefit model has been used in Washington to help guide policy and budget choices for more than 15 years at both the individual program level and the portfolio level with the goal of affecting statewide outcomes. At the program level, WSIPP’s unfavorable-outcome evaluation of an intensive probation program in the juvenile justice system prompted the legislature to cut the program’s fund- ing and redirect the funds into programs the model identified as cost-effective in reducing crime (Drake, 2010). the legislature has also used the model to make budget choices at the portfolio level. During the 2007 session, the legislature invested $48 million in a portfolio of prevention, juvenile justice, and adult correc- tions programs that the model determined to be evidence-based and economically sound, thereby enabling the legislature to remove a $250 million prison from the state’s long-term forecast (laws of 2007, ch. 522 § 203, ShB 1128). this was a major step forward—moving from compiling lists of effective programs to using the model to achieve measurable statewide outcomes.

the 2011 legislative session demonstrated additional policy impacts from WSIPP’s work, but also showed that political factors can affect use of policy research. Washington State faced a projected deficit of $5.3 billion for the 2011 to 2013 period as tax revenues continued to decline (Shannon & Schmidt, 2011). During the 2011 session, legislators introduced a bill intended to make strategic cuts in the correctional system to save money for the state and allow for the re- investment of funds in evidence-based programming identified by WSIPP. the goal was to maintain no net increase in the prison caseload forecast (Senate Bill 5866). the proposed bill shortened the length of stay in prison for some offenders considered at lower risk of committing new offenses, reducing the average daily prison population by 638 offenders.

legislators and legislative staff asked WSIPP to analyze the crime and fiscal impacts of Senate Bill 5866 using the sentencing tool. Estimates showed that, over the long run, the net effect of these two policies would lower crime by about 3,600 incidents in Washington (with a 93% to 99% probability of a net reduction). In terms of fiscal effects, WSIPP estimated that the net effect of the two policies would contribute $6.6 million in savings to the 2011–2013 state biennial budgets, saving state taxpayers an additional $35 million and local taxpayers $44 million in reduced criminal justice system costs over the long run because of the expected reduction in crime.

WSIPP assisted with crime and fiscal-impact statements for various iterations of the bill throughout the legislative process. the Senate version of the bill called for a 60-day early release of nonviolent offenders (Proposed Senate 2011–13 Operating Budget, April 19, 2011). the house version did not specify how early release was to occur but instead required WSIPP to develop a plan for adjusting the sentencing guidelines to save at least $10 million by Fy 2013; these funds were

Vanlandingham and Drake / RESultS FIRSt 559

to be reinvested into evidence-based programs for juveniles and adults proven to reduce recidivism (2011–13 Biennial Operating Budget and 2011 Supplemental As Passed house).

Political factors, noted in the literature cited earlier, have affected some uses of WSIPP’s analysis in Washington. For example, despite the fiscal and crime benefits predicted by WSIPP for the Senate bill, the early-release provision was vetoed by the governor. Nonetheless, WSIPP’s experience over the past 15 years has shown that cost-benefit analysis, when conducted by an entity that works closely with policymakers, can become embedded in the legislative process and regularly affect both state budget and policy decisions.

results first: transferring the wsIPP Approach to other states

Building on Washington’s experience, the Pew Center on the States developed the Results First initiative to implement the approach in other states. the Pew Charitable trusts and the John D. and Catherine t. MacArthur Foundation, with additional support from the Annie E. Casey Foundation, launched Results First in 2010.

Results First has two major components: assessing state use of cost-benefit analysis and helping states adapt the WSIPP cost-benefit model to inform policy decisions. For the first component, Results First is conducting a nationwide study to assess how and to what extent the 50 states and the District of Columbia cur- rently use cost-benefit analysis and its impact. this research will be completed and reported in 2012. the study includes an intensive document-collection effort as well as interviews with a wide range of key staff in each state, including execu- tive, legislative, and judicial branch representatives.

Results First is also supporting other state efforts to implement the WSIPP cost-benefit analysis model. Before beginning this work, Pew conducted an inde- pendent expert validation of the WSIPP model and methodologies. this validation was conducted by an expert panel that included highly regarded academics and practitioners with experience in providing cost-benefit data to policymakers. the panel concluded that the WSIPP model and approach were sound and represented a major advance in the field.

Although efforts to expand the WSIPP model into other states are young, early results are promising. As of January 2012, 12 states (Connecticut, Florida, Idaho, Illinois, Iowa, Kansas, Kentucky, New Mexico, New york, Oregon, texas, and Vermont) in addition to Washington were actively implementing the WSIPP cost-benefit analysis model, and more than 20 other states had expressed interest in the approach.

Pew’s process for implementing Results First is designed to address both the technical and political challenges to using this approach in the policy process.

560 PPMR / March 2012

to ensure that there is critical support for the approach, Pew requires states to submit a letter of invitation signed, typically, by the legislature’s presiding of- ficers or the governor. Pew also requires states to establish a policy workgroup to oversee the implementation effort; this group typically includes representatives from the key stakeholders that will interpret and use the model’s results (e.g., the legislature, governor’s office, department of corrections). this is intended to ensure that policymakers are fully aware of the effort and that the results of the analysis are brought to their attention for consideration in subsequent budget and policy deliberations. to address the technical challenges of implementing the model, Pew requires states to establish a staff group that compiles the program and cost information necessary to operate the model. this group must have access to the data, the technical skills needed to operate the model, regular access to the policymaker group, and status as an honest broker so that political stakeholders will have confidence in the process and its results.

Pew provides technical assistance throughout the implementation process. this includes helping organize the policy and staff workgroups, planning the implementation process, and providing assistance in areas such as identifying data sources, conducting statistical and fiscal analyses, operating the model, and interpreting the results for policymakers. Pew is also working to create a learning community of participating states by convening key staff and conducting seminars for legislative and executive branch staff and officials on cost-benefit analysis approaches; the first convocation was held in July 2011 and included representa- tives from 10 states.

Each participating state was just beginning to implement the process as of January 2012; their progress until then had typically been limited to forming the policy and staff workgroups and beginning to collect and analyze data. Pew anticipates that states normally will require three to six months to get the model operational; customizing the model to include additional programs and research will probably take longer. Pew will issue reports assessing the states’ progress and lessons learned in 2012.

conclusion

Given the severity of the structural budget challenges facing states and the wide- spread skepticism about the merits of many public programs, the time seems ripe for expanding the use of cost-benefit and cost-effectiveness analysis in state policymaking. the WSIPP case study provides evidence that this approach can be deployed in a significant way in a state and can be successful over time in helping the state enact reforms that improve policy effectiveness and reduce taxpayer costs. the WSIPP model is a major technical advance that can enable states to better consider the predicted results of policy choices during their policy and budget

Vanlandingham and Drake / RESultS FIRSt 561

deliberations. Nonetheless, the case study demonstrates that it is important to consider both technical and political challenges in conducting this work. Pew’s Results First initiative is designed to enhance the success of the approach in other states by laying the groundwork for both technical and political acceptance of this analysis. While these efforts are still beginning, they are showing promise. Ongoing analysis of these efforts will be available from the Pew Center on the States.

notes

1. the legislation authorized the institute to receive outside funding for this project; the MacArthur Foundation is supporting 80% of the work, with the legislature funding the other 20%.

2. See the technical appendices of Aos et al. (2011) for more details on WSIPP’s meta- analytic and cost-benefit analysis methods.

references

Aos, S., Barnoski, R., & lieb, R. (1998, January). Watching the bottom line: Cost-effective interventions for reducing crime in Washington [Document no. 98-01-1201]. Olympia: Washington State Institute for Public Policy.

Aos, S., Miller, M., & Drake, E. (2006, October). Evidence-based public policy options to reduce future prison construction, criminal justice costs, and crime rates [Document no. 06-10-1201]. Olympia: Washington State Institute for Public Policy.

Aos, S., Miller, M., & Mayfield, J. (2007, March). Benefits and costs of K-12 educational policies: Evidence-based effects of class size reductions and full-day kindergarten [Document no. 07-03-2201]. Olympia: Washington State Institute for Public Policy.

Aos, S., Miller, M., & Pennucci, A. (2007, December). Report to the Joint Task Force on Basic Education Finance: School employee compensation and student outcomes [Docu- ment no. 07-12-2201]. Olympia: Washington State Institute for Public Policy.

Aos, S., Phipps, P., Barnoski, R., & lieb, R. (2001, May). The comparative costs and benefits of programs to reduce crime, v 4.0 [Document no. 01-05-1201]. Olympia: Washington State Institute for Public Policy.

Aos, S., lieb, R., Mayfield, J., Miller, M., & Pennucci, A. (2004, September). Benefits and costs of prevention and early intervention programs for youth [Document no. 04- 07-3901]. Olympia: Washington State Institute for Public Policy.

Aos, S., lee, S., Drake, E., Pennucci, A., Klima, t., Miller, M., Anderson, l., Mayfield, J., & Burley, M. (2011, July). Return on investment: Evidence-based options to improve statewide outcomes [Document no. 11-07-1201]. Olympia: Washington State Institute for Public Policy.

Barnoski, R. (2004, January). Outcome evaluation of Washington State’s research-based programs for juvenile offenders [Document no. 04-01-1201]. Olympia: Washington State Institute for Public Policy.

Boardman, A.E., Greenberg, D.h., Vining, A.R., & Weimer, D.l. (2006). Cost-benefit analysis—Concepts and practice. upper Saddle River, NJ: Pearson Prentice hall.

Brent, R.J. (2006). Applied cost benefit analysis (2d ed.). Cheltenham, uK: Edward Elgar.

Chelimsky, E. (2009). A clash of cultures—Improving the “fit” between evaluative inde- pendence and the political requirements of a democratic society. American Journal of Evaluation, 29(4), 400–415.

562 PPMR / March 2012

Cousins, J.B., & Shulha, l. (2006). A comparative analysis of evaluation utilization and its cognate fields of inquiry: Current issues and trends. In I.F. Shaw, J.C. Greene, & M. Mark (Eds.), The Sage handbook of evaluation: Policies, programs and practice (pp. 266–291). thousand Oaks, CA: Sage.

Drake, E.K. (2010). Washington State juvenile court funding: Applying research in a public policy setting [Document no. 10-12-1201]. Olympia: Washington State Institute for Public Policy.

Drake, E.K., Aos, S., & Miller, M.G. (2009). Evidence-based public policy options to reduce crime and criminal justice costs: Implications in Washington State. Victims and Offenders, 4(2), 170–196.

Eckl, C. (2011, February). Deep holes, few options. State Legislatures, 18–20. Evans School of Public Affairs. (2011). Benefit-Cost Analysis Center. Available at http://

evans.washington.edu/research/centers/benefit-cost-analysis/. Fuchs, E.P., & J.E. Anderson. (1987). the institutionalization of cost-benefit analysis.

Public Productivity Review, 10(4), 25–33. Graham, J.D. (2008). Saving lives through administrative law and economics. University

of Pennsylvania Law Review, 157(2), 101–245. lee, S., Aos, S., & Miller, M.G. (2008, July). Evidence-based programs to prevent chil-

dren from entering and remaining in the child welfare system: Benefits and costs for Washington [Document no. 08-07-3901]. Olympia: Washington State Institute for Public Policy.

MacArthur Foundation. (2011). Policy research recent grants: the power of measuring social benefits. Available at www.macfound.org/site/c.lklXJ8MCKrh/b.947545/k. D8C1/Domestic_Grantmaking/, accessed April 22, 2011.

Marvell, t.B. (2010). Prison population and crime. In B.l. Benson and P.R. zimmerman (Eds.), Handbook on the economics of crime (pp. 145–183). Cheltenham, uK: Edward Elgar.

Mooney, C. (1991). Information sources in state legislative decision making. Legislative Studies Quarterly, 16(3), 445–455.

National Conference of State legislatures (NCSl). (2010). Bill Information database, accessed August 26, 2010. Denver.

Patton, M. (2008). Utilization: Focused evaluation (4th ed.). los Angeles: Sage. Pennucci, A., Aos, S., & Ngugi, I. (2008, September). Report to the Joint task Force on

Basic Education Finance [Document no. 08-09-2201]. Olympia: Washington State Institute for Public Policy.

Pew Center on the States. (2011). Results First: Helping states assess the costs and benefits of policy options and use that data to make decisions based on results. Washington, DC: Pew Charitable trusts.

Porter, t.M. (1995). Trust in numbers: The pursuit of objectivity in science and public life. Princeton: Princeton university Press.

Preskill, h., & torres, R. (1999). Evaluative inquiry for learning in organizations. thou- sand Oaks, CA: Sage.

Radin, B. (2000). Beyond Machiavelli: Policy analysis comes of age. Washington, DC: Georgetown university Press.

Schneider, M., Scholz, J., lubell, M., Mindruta, D., & Edwardsen, M. (2003). Building consensual institutions: Networks and the national estuary program. American Journal of Political Science, 47(1), 143–158.

Shaffer, M. (2010). Multiple account benefit-cost analysis—A practical guide for the sys- tematic evaluation of project and policy alternatives. toronto: university of toronto Press.

Shannon, B., & Schmidt, K. (2011). State budget shortfall at $5.3 billion. Olympian,

Vanlandingham and Drake / RESultS FIRSt 563

March 19. Available at www.theolympian.com/2011/03/18/1583490/shortfall-at-53- billion.html.

Snow, C. (1961). Science and government. Cambridge: harvard university Press. Vining, A., & Weimer, D. (2010). An assessment of important issues concerning the ap-

plication of benefit-cost analysis to social policy. Journal of Benefit-Cost Analysis, 1(1), article 6.

Weiss, C. (1973). Where politics and evaluation research meet. Evaluation Practice, 14(1), 93–106.

Weiss, C. (1989). Congressional committees as users of analysis. Journal of Policy Analysis and Management, 8(3), 411–431.

Whiteman, D. (1995). Communication in Congress: Members, staff, and the search for information. lawrence: university Press of Kansas.

yin, R. 2003. Case study research (3d ed.). thousand Oaks, CA: Sage. zajano, N., & lochtefeld, S. (1999). the nature of knowledge and language in the legislative

arena. In K. Jonas (Ed.), Legislative program evaluation: Utilization-driven research for decision makers (pp. 3–22). San Francisco: Jossey-Bass.

zerbe, R.O., Jr., Davis, t.B., Garland, N., & Scott, t. (2010). Toward principles and standards in the use of benefit-cost analysis. Seattle: university of Washington Benefit Cost Center.

Gary R. VanLandingham is director of Results First with the Pew Center on the States. Before joining Pew, he led the Florida legislature’s Office of Program Policy Analysis and Government Accountability. He has more than 30 years of experi- ence in program evaluation and policy analysis at the state and local government levels. He has served in leadership positions with the National Conference of State Legislatures, the National Legislative Program Evaluation Society, the Southeast Evaluation Association, and the North Florida Chapter of the American Society for Public Administration. He has a Ph.D. in public administration and policy, has taught undergraduate and graduate seminars at Florida State University and has authored publications on performance budgeting, policy research utilization, and public management.

Elizabeth K. Drake is a senior research associate for the Washington State Institute for Public Policy and before joining the institute was a researcher for the Washing- ton State Department of Corrections. She received her B.S. from Bowling Green State University and her M.A. from Washington State University, both degrees in criminal justice. She has professional experience conducting program evaluations, meta-analysis, and cost-benefit analysis in adult and juvenile justice.

Copyright of Public Performance & Management Review is the property of M.E. Sharpe Inc. and its content

may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express

written permission. However, users may print, download, or email articles for individual use.