Effective Policy Evaluation

8 Policy Evaluation CHAPTER OVERVIEW

Evaluation provides important feedback to policymakers and administrators for improvement to public policy and the achievement of desired policy outcomes. Policy evaluators concentrate on the “does or does not do” elements of public policy. Good policy evaluation is based in purely objective analysis. When the empirical world collides with the normative world, things often become a bit messy. The effective policy evaluator is able to dance through the minefield of politics bearing important messages for policymakers to consider. The specific goals for the chapter are: • Discuss the purpose of policy evaluation and the skills needed to conduct a good evaluation. • Discuss the types and methods of evaluation. SKILLS OF A GOOD POLICY EVALUATOR The late Stuart Nagel (2002) considered the required skills of a good policy evaluator: • Policy expertise Evaluators should have a solid understanding of the policy area being evaluated. Graduate school studies in public policy and/or public affairs help to develop an understanding of policy history, contemporary public policy, and future trends. A good policy evaluator understands multiple policy areas; often, public policies are interrelated. • Public administration and management Policy evaluator must understand how and why public administration works in the manner it does, variations in administrative style and operation, and the types of constraints under which administrators operate. Conceptualizing public administration requires an understanding of the philosophy of government and the dynamic nature of the social contract— the foundation of our liberal democracy. Social contracts evolve and 150 Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.values change. A product of progressivism, public administration values equity, efficiency, accountability, fiscal integrity, effectiveness, and responsiveness (Wilson 1989). • Organizational theory Evaluators must be sensitive to organizational differences and circumstances. Every organization has different standard operating procedures, operates under different leadership, and has different employees. Additionally, organizational culture is a critical element that will shape organizational outcomes. In some instances, organizations compete with other organizations for resources. Alternatively, organizations may develop cooperative relationships with other organizations to meet organizational goals. Public–public and public–private organizational partnerships have become more common in the policy implementation process. An evaluator who understands how organizations work internally and how they relate to their “environment” will have a greater understanding of how policy implementation works, where pitfalls arise, and where policy success is realized. • Research methods Policy evaluation relies on both qualitative and quantitative analysis. Quantitative analysis will likely always play a dominant role in policy evaluation. Effective policy evaluators must develop strong skills in statistical analysis. (See Box 8.1 A Policy Evaluator’s Toolbox; Box 8.2 Policy Sciences and Evaluation; and Box 8.3 Policy Analysis Versus Policy Evaluation.) BOX 8.1 A Policy Evaluator’s Toolbox Knowledge/Skill/Ability Source Policy Area Expertise Experience; policy sciences, political science courses. Public Administration or Management Experience; public administration/ political science courses. Organizational Theory Experience; public administration, organizational psychology, cultural anthropology courses. Research Methods/Statistics Experience; economics, statistics, political science, policy studies, sociology, psychology courses. Ethics and Philosophy Experience; philosophy, comparative religion, cultural diversity, history, political science courses. Chapter 8 • Policy Evaluation 151 Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.BOX 8.3 Policy Analysis Versus Policy Evaluation As discussed in chapter 4, policy analysis often focuses on the questions of what government ought or ought not do and how the normative priorities can become empirical realities. Policy analysis is conducted early on in the policy cycle, particularly during the policy formulation process, and helps guide policy choices and plans. Policy evaluation, however, is very carefully focused on existing public policy—policy that is being implemented or has been implemented. Pointing out this distinction may seem rather pedantic, but it is a very real concern. At the time at which it is conducted, policy evaluation should attempt to measure only the impact, output, efficiency, and effectiveness of public policy. The information generated may, at a later time, serve as the foundation of future policy analysis and policy adjustments, but superior policy evaluations cannot be conducted in a fair and impartial manner if normative issues cloud the evaluation process. Without objective evaluation it is impossible for policymakers and administrators to have a clear sense of what policy is and is not doing. Important empirical findings must be juxtaposed with the goals of public policy and considered in future policy cycles if public policy is to advance and improve. Given the importance of evaluation, is the distinction between policy analysis and policy evaluation understood and observed in the policy process? Frequently, the answer is: no. Policy evaluation and analysis are often confused, assumed to be one BOX 8.2 Policy Sciences and Evaluation The policy sciences are grounded in normatively driven Progressive politics. As the late Charles Merriam argued, there was a need for a more reasoned approach to politics. As Merriam saw it, politics is frequently a highly wasteful process. Ignorant and selfinterested individuals and groups pursue their objectives without any due consideration for the “higher” purposes of government—namely, efficient, effective, and equitable outcomes. The politics of the past, according to Merriam and others, were driven by power and brute force. Issues of justice were grounded in the notion that “might makes right.” Merriam rejected this notion of government, concluding that the modern era would place its faith in grounded principles of general social, economic, and political advancement rather than the interests of a narrow group of very powerful individuals. For the Progressive-minded, policy sciences should focus on the development of reasoned approaches to the study of social, political, and economic phenomena; identify problems that can be solved through social action; and develop acceptable solutions to those problems. Policy science was seen as a liberating force in public policy, removing the influences of political “mumbo-jumbo” and encouraging “the constructive possibilities of human nature.” Policy evaluation—an important part of the policy sciences—was born of this vision and continues to be guided by related principles. 152 Part 2 • Dimensions of Public Policy Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.GENERAL PURPOSES OF EVALUATION Evaluation occurs for several reasons. According to Carol Weiss (1998), these purposes can be divided into two general categories: overt and covert purposes. The overt purposes of policy evaluation are: (1) to determine the process by which a policy is being implemented and develop a better understanding of the progression and pitfalls associated with implementation; and (2) to determine policy impacts and outcomes. Overt evaluation is done “above the board,” so there are fewer ethical dilemmas for the policy evaluator. Evaluation results are usually objective, intended to help policymakers and implementers develop efficient, fair, and effective policy outcomes. Covert evaluation is often used to protect agencies from destructive “political storms.” Covert evaluation is conducted for at least three reasons: (1) postponement, (2) ducking responsibility, and (3) window dressing (Weiss 1989, 22). Postponement means that agencies are trying to fend off external pressures. Elected officials, clientele groups, and policy advocates or critics may be searching for information regarding agency implementation or may be pressuring the agency. An implementing agency or administrator may use evaluation to delay the distribution of information or slow the decision-making process. Ducking responsibility is a classic example of covert evaluation. Evaluation is conducted to shed responsibility for failures and shift blame. Agencies document that they have “just” followed the rules and have not been involved with the policy failure. Ducking responsibility is unconstructive; it does not help policymakers seeking to improve outcomes. Window dressing is a third reason for covert evaluation. The approach “offer[s] legitimacy” (Weiss 1989, 22) to agencies as they go about the implementation process. It is a method of reducing constraints imposed by elected officials via oversight and grassroots efforts to influence agency decision-making in the implementation process. Agencies claim that evaluation was conducted using objective empirical techniques for the purpose of justifying implementation and the same. The result of this confusion is an inability by many policymakers, administrators, and policy evaluators to separate objective conclusions about a public policy from normatively based advocacy for or rejection of public policy. In other words, the results of evaluation study are frequently viewed as either a sign of loyalty to the proponents of particular policy goals or as a serious affront. Policy evaluators face the same constraints as did the legendary Cassandra, daughter of Priam, the king of Troy in the Iliad. Like Cassandra, evaluators speak truths to those who hold positions of power but are often ignored; their “truths”—particularly painful truths—negated. At times, the evaluators involved are castigated for their efforts. Policy analysis is often driven by advocacy and the analysts are often praised by individuals and groups whose ideas they support. The policy evaluator does not have the luxury of working within such an inviting setting. Whether the evaluator’s evidence demonstrates policy success or failure, the evidence will likely draw the attention of policy critics and apologists who either reject or embrace the findings. Chapter 8 • Policy Evaluation 153 Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.decisions. Window dressing evaluations may also be done for public relations purposes to build support for a policy. INTERNAL VERSUS EXTERNAL EVALUATIONS Internal policy evaluators are individuals or teams drawn from an agency’s own personnel. External evaluators are individuals or teams drawn from professional evaluation organizations, either for-profit private firms or not-for-profit organizations such as a research department in a public university. Internal Evaluation Internal evaluations have one immediate advantage, namely, the evaluators are often well-known and trusted members of the organization being evaluated, who understand an implementing agency’s organization culture, processes, and goals. Familiarity reduces the time and effort necessary for an evaluator to learn “how” a policy is implemented. Evaluators must develop a policy narrative, drawing conclusions about (1) how a policy is being implemented, (2) the success of the implementation, and (3) constructive advice on how to improve policy implementation processes and outcomes. Internal evaluations face several challenges. First, there is the potential for subjectivity to cloud the evaluation process and outcomes. Internal evaluators often feel a real or perceived pressure to produce positive evaluation results, aware of possibly uncomfortable relations with colleagues and supervisors if the evaluation is critical. Second, internal evaluators may ignore data damaging to the implementing agency image. Third, policy formulators, pressure groups, and clientele may disregard internal evaluation findings if they do not trust the implementing agency. Finally, internal evaluators may suffer from groupthink—as members of the agency being evaluated, the evaluators might find it difficult to think outside the agency paradigm or may be constrained in their ability to consider the outcomes and impacts of policy implementation. Internal evaluators may consciously or unconsciously introduce bias into evaluation studies and findings. External Evaluation External evaluations are viewed as more accurate and objective. External policy evaluations may be conducted by private firms or by public organizations (e.g. universities’ policy workshops) specializing in evaluation studies. Agencies prefer external evaluators who are objective, but who couch findings in an understanding of policy goals, horizons, and complexity; implementing agency processes and culture; and the nature of the target groups being served. TYPES OF EVALUATIONS Informal Evaluations Sometimes implementing agencies wish to conduct small, unscientific, informal studies of policy implementation processes and outcomes. Through informal 154 Part 2 • Dimensions of Public Policy Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.evaluation, adjustments can be made at various stages in the implementation process. In the case of new policies, particularly policies that are not well defined through statutory guidelines, implementing agencies may find it necessary to build policy processes incrementally and through trial and error determine an optimum process to achieve a satisfactory policy outcome. Public administrators, however, tend to be conservative and do not like to take big risks. Administrators realize that their decisions can impact the provision of resources and could damage their ability to demonstrate fiscal integrity and accountability. Thus, informal evaluations may be conducted to determine if policy-related decisions are producing desirable outcomes. Informal evaluation often relies on the experience of policy administrators rather than solely on scientific objectivity. Through their collective experiences in a particular policy area, administrators often have a good idea about what is feasible and what will result in policy failures. Administrative personnel who have worked for a long period of time with a specific target group develop a sense of policy impacts. Informal evaluations are usually conducted by internal evaluators. In fact, many informal evaluations are conducted by individual personnel who collect and study readily accessible information. The nature of the evaluation usually does not demand the expense of inordinate amounts of resources to collect and analyze data. (See Box 8.4 Data and Developing Confidence in Numerical Measurement.) BOX 8.4 Data and Developing Confidence in Numerical Measurement Collecting data involves more than simply determining its type. Policy evaluators must be aware of the quality of the data. First, data must be shown to have high levels of validity. In other words, the policy evaluator must be able to demonstrate that the measures developed through operationalization accurately represent theoretical concepts. Face validity is a nonstatistical approach to justifying an empirical measure for a concept. This approach appeals to intuitive sense that a measure reflects a theoretical concept. For example, net income is likely to be one way of measuring the concept of socioeconomic status. Other approaches to determining validity involve statistical analysis. Concurrent validity determines if multiple measures of a concept are positively correlated (or vary in the same or very similar way) with one another. For example, if net income, number of years of formal education completed, and occupational prestige were used as measures of socioeconomic status, concurrent validation of the measures would be expected to yield very high positive correlations between the measures. Discriminant validity analysis assumes that measures that do not describe the same concept should not be correlated. For example, in a policy evaluation study of antiterrorism policies related to airport security, it would not be expected that a measure of the concept socioeconomic status (e.g., income) would be correlated with a measure of the concept clientele satisfaction (e.g., feelings toward airport security personnel). The reliability of measures is also very important in policy evaluation. Measurement must be demonstrated to be consistent otherwise the results of policy evaluation may not (Continued) Chapter 8 • Policy Evaluation 155 Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.One problem that emerges from an over-reliance on informal evaluations is that it often tells only a tiny part of the story about policy implementation. Placing great faith on the experience of administrators, the approach may wrongly assume that data collected are valid and contain reliable measures of implementation processes and outcomes. Formal Evaluations Formal evaluations usually require a significant time commitment by evaluators and a substantial financial commitment on the part of a policy-implementing agency. Implementing agencies operating under severe budget constraints may wince when faced with the cost of conducting a formal policy evaluation. Nevertheless, formal evaluation is likely to confirm or disconfirm the success of implementation efforts. Post-policy evaluations are one way of reducing the costs of evaluation, although post-policy evaluations usually occur because evaluation is thought to be something that occurs “at the end.” At other times, post-policy evaluation is a function of timing—agency administrators do not establish evaluation teams or contract out for evaluation services until policy implementation has commenced. Post-policy evaluations are typically used in the case of policies that are supported by grants-in-aid from other levels of government or from private granting institutions. Granting agencies usually require evaluation studies for purposes of accountability. Post-policy evaluations are incapable of demonstrating the impact of a public policy on a target group. As nobody really knows the condition of the target group prior to the implementation of a public policy, it is impossible to use a post-policy evaluation approach to empirically demonstrate a change in the target group’s condition. Pre-/post-policy evaluations are much better at demonstrating the changes brought about by the implementation of a public policy. A target group’s condition prior to the implementation of a public policy is duly noted through prepolicy study. Following the policy influence on the members of the target group, an identical study is conducted to record their post-policy condition. It is very reflect that which is occurring. Reliability analysis requires the use of statistics to determine the consistency of a measure. Test-retest analysis measures the same phenomena repeatedly. If the results of measurement are the same from measure to measure, then the reliability of the measurement can be assumed. Validity and reliability are very sophisticated topics that can be simplified through an analogy. Pretend for a moment that you are shooting a crossbow at a paper target. You have a scope with crosshairs on the crossbow to guide your arrows to the center of the target. If you find all your arrows stuck in the bulls-eye then your scope measure of the center of the target is valid and reliable. If you find that your arrows are clustered together elsewhere on the target, then the scope’s measure of the center of the target is not valid but it is reliable. Finally, if you find that your arrows are randomly distributed on the target, then the scope’s measure of the center of the target is neither valid nor reliable. 156 Part 2 • Dimensions of Public Policy Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.important that identical measures are used in both the pre- and post-policy evaluation. If the measures are not identical, then it is impossible to conduct an accurate comparison. While the pre-/post-policy approach provides greater understanding of the condition of a target group prior to and following interaction with a public policy, it does not show whether impact is a function of public policy or the result of changing societal conditions, unrelated to policy. For example, a welfare-to-work program may find that there was a significant increase in employment among their unemployed clientele, but was that increase in employment a function of the welfare-to-work program or was it a function of changing economic conditions? It would be helpful if a pre-/ post-policy evaluation had at least two groups to study—one group served by a public policy (an experimental group) and another group similar to the target population but not served by the public policy (a control group). If there is no change in the latter group but significant change in the former group (or if the change in the latter group is not as positive as that of the former group), then the policy evaluator could conclude that the public policy, as implemented, is achieving a positive outcome and having an impact on the target population. One problem with the policy experiment approach relates to ethics. In order to develop a control group it is necessary to deny government services to individuals who are very similar to the target population being served by public policy. An individual’s constitutional rights are likely to be impinged, particularly if it is shown that the status of the experimental group is improving in an expected manner. In other words, the control group would be denied the benefits of public policy for purposes of experimentation. In addition to demonstrating the presence or absence of positive policy outcomes, evaluators may also focus on different types or amounts of influence a public policy seeks to exert on a target population. This is very similar to medical studies scrutinizing the differential impact of medications or dosages on patients. In regulatory policy, for instance, it is often not necessary to use maximum penalties to dissuade an individual or organization from doing things deemed illegal or deleterious to the public welfare. This approach to evaluation is especially valuable to policy analysts trying to produce efficient and effective public policy. If it can be shown through policy evaluation that a simpler and more costeffective approach to policy implementation produces the same effect as more expensive and sophisticated approaches, then it is likely that the cheaper, yet equally effective, approach should be adopted. As with the control group scenario, offering different types and levels of service delivery may face scrutiny by target groups and other policy stakeholders. Few people are pleased to discover they were treated like test animals. If individuals or groups have been given lower-quality services that do not produce the desirable effect, then they might feel discriminated against by their government, which has the potential to produce negative and long-lasting impacts on these individuals or groups. (See Box 8.5 Qualitative Approaches to Evaluation.) Chapter 8 • Policy Evaluation 157 Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.BOX 8.5 Qualitative Approaches to Evaluation In some instances a public policy is not conducive to quantitative analysis. The policy may seek to produce qualitative outcomes and must therefore be evaluated on those terms. While a pre-, post-, and interim evaluation design is possible, the analyses cannot be shown to have a significant impact on target populations without a concerted effort on the part of the policy evaluator to weave a narrative demonstrating desired policy impacts. Internal evaluations that are based in qualitative methods are usually highly suspect and are less likely to be accepted by individuals and groups who are skeptical or critical of the public policy. External evaluations using qualitative methods are often seen as potentially misleading, but the fact that these studies are being conducted by individuals and groups from outside the implementing organization tends to lend a higher degree of credibility. The evaluator’s or evaluation team’s track record in conducting objective policy evaluations is also of considerable importance when conducting external policy evaluations based in qualitative analysis. Qualitative analysis requires a high level of creativity on the part of the evaluator or evaluation team. Rather than focus on formal organizational structure, qualitative process analyses focus on the informal relationship between policy actors within and outside the implementing agency. The evaluator uses personal interviews and archival data as the basis of qualitative process evaluations, which study the systematic organizational methods by which policy goals are accomplished. Interviews are difficult to obtain and take a considerable amount of skill on the part of the interviewer. Additionally, interviews require a time commitment on the part of the interviewer and interviewee. It takes time to answer questions and time to develop the level of trust necessary to obtain accurate and complete responses. Finally, the evaluator using the interview technique must sift through hours of transcripts and written notes, trying to develop a narrative or series of narratives that describe the informal processes of policy implementation and then juxtapose that informal process with the formal processes associated with the achievement of policy goals. Personal interviews are frequently used by evaluators conducting outcome evaluations. Again, the evaluator might wish to conduct interim interviews with policy implementers and clientele groups. Following policy implementation, evaluators may wish to meet again with agency personnel and clientele to discuss the outcomes of the policy, outline things that went well and things that could be improved about the process of implementation, and discuss the policy impacts, quality of service, and areas of improvement associated with policy implementation and clientele. In some cases interviews are conducted on an individual or one-on-one basis. As noted previously, interviews are expensive and time-consuming. In order to maximize the use of this qualitative policy evaluation technique, focus groups are frequently used. A group of individuals from the implementing agency or clientele is invited to participate in focus groups. While more cost effective, there is the possibility that individuals will be reluctant to express views that are not consistent with the opinion of the majority of individuals present. Additionally, a form of groupthink may develop—individuals will construct a group narrative rather than offer individual views; the result is that variation is diminished and evaluation results may seem stilted. 158 Part 2 • Dimensions of Public Policy Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.Lessons Learned from Chapter 8 SPECIFICS • Evaluation is a highly technical process of determining if a policy is meeting the goals it was intended to meet. • Evaluations can either be formal or informal. • Evaluations study the impacts of policy as well as the processes by which policy goals are accomplished. • Evaluations can be conducted either internally or by external evaluation teams. • Two major methods of evaluation were discussed: quantitative and qualitative. THE BIG PICTURE The effectiveness of policy evaluation is dependent on the type of evaluation conducted, the skill level of the evaluator or evaluation team, and the level of trust present between evaluator or evaluation team and the individuals or agencies under study. I have outlined two major types of policy evaluation here: internal and external. Both approaches have certain strengths and certain weaknesses. Clearly, internal evaluators are faced with certain constraints associated with being members of the implementing agency. Evaluator bias may consciously or subconsciously affect evaluation processes and outcomes. Nevertheless, internal evaluation may be capable of greater consonance between evaluation process and policy intent as defined by statute and informal policy implementation processes and subgoals. Internal evaluators may be able to gather information with greater ease due to their knowledge of the organization External evaluators are often viewed as being more objective in their analysis. As nonagency personnel, external evaluators may be better able to see policy and agency implementation in a holistic manner; able to compare their analyses with evaluation studies conducted under similar circumstances and of related policy. In that sense, the external evaluator or evaluation team’s experience and collective knowledge covering a wide variety of evaluation studies is highly valuable to implementing agencies. Not being members of the implementing agency, however, makes the collection of relevant information more difficult and requires building trusting relationships with key actors within the implementing agency being evaluated. Additionally, the skill level of the evaluators or evaluation teams is critical to the evaluation process. Internal and external evaluators must be chosen on the basis of their skills in—among other things—objectivity, ethical standards, statistics, quantitative and qualitative methodology, organization theory, personnel administration, and policy expertise. In the case of internal evaluation teams, the implementing agency must be committed to the provision of continual training for the evaluator or evaluation team. External policy evaluators must also be committed to continual training and retraining, but it is difficult for the Chapter 8 • Policy Evaluation 159 Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.implementing agency to know that skill levels are continually being improved— one way of measuring the knowledge, skills, and abilities of an external policy evaluator or evaluation team is through careful study of their past work. An evaluation cannot be conducted in an ad hoc manner. For evaluation to be accurate and complete, it is necessary for evaluators to be active observers of public policy as it is being implemented. Evaluation must parallel the goals and processes of implementation. Goals must be operationalized for implementation and evaluation. Processes must be clearly identified and standards must be established in a manner that can be measured for purposes of evaluation. Benchmarks must be built into the policy evaluation and implementation process, which will serve as instruments of comparison between what an implementing agency’s goals and outcomes. Most public policies are highly complex and require months or years to achieve desired outcomes. In this case, implementing agencies and evaluators must work together to identify distinct policy phases. Each of these phases should have identifiable subgoals that must be met. Evaluators can then conduct analyses with a greater understanding of what should be accomplished by a certain point. If the policy phase goals are not being met, then evaluation can be used to correct or to adjust policy implementation processes and outcomes. An important thing to remember about policy evaluation is that it is as much an art as a science. Many of the principles of policy evaluation discussed emphasize the use of a scientific method of evaluation, but a good policy evaluator is also aware that evaluation must fit policy and agency type. Just as one would not build a skyscraper out of bricks or mud, a policy evaluator must choose his or her media and tools carefully, constructing an accurate story about a policy and how it is being implemented. Key terms analysis operationalization ducking responsibility postponement findings theoretical development formal evaluation window dressing informal evaluation Questions for Study 1. What are three major purposes of evaluation? Identify and discuss. 2. What is the difference between formal and informal evaluation? 3. What are three major challenges to policy evaluation? Identify and discuss. Bibliography Berk, R., and P. Rossi. 1998. Thinking about program evaluation, 2nd ed. Newbury Park, CA: Sage. 160 Part 2 • Dimensions of Public Policy Simon, C. A. (2017). Public policy : Preferences and outcomes. ProQuest Ebook Central http://ebookcentral.proquest.com Created from capella on 2021-06-06 00:16:43. Copyright © 2017. Taylor & Francis Group. All rights reserved.Davidson, E. 2005. Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. Mohr, Lawrence. 1995. Impact analysis for program evaluation, 2nd ed. Newbury Park, CA: Sage. Nagel, S. 2002. Contemporary policy evaluation. New York: Nova Science Publishers. ———, ed. 2002. Handbook of public policy evaluation. Thousand Oaks, CA: Sage. Merriam, C. 1972. New aspects of politics. Chicago: University of Chicago Press. Rogers, P., Hacsi, T., Petrosino, A., and T. Huebner, eds. 2000. Program theory in evaluation challenges and opportunities: New directions for evaluation. San Francisco, CA: Jossey Bass. Stufflebeam, D. 2001. Evaluation models. San Francisco, CA: Jossey Bass. Sylvia, R., K. Sylvia, and E. Gunn. 1997. Program planning and evaluation for the public manager, 2nd ed. Long Grove, IL: Waveland Press. Weiss, C. 1998. Policy evaluation. Upper Saddle River, NJ: Prentice Hall. Wildavsky, A. 1979. Speaking truth to power: The art and craft of policy analysis. Boston: Little, Brown. Wilson, J. 1989. Bureaucracy: What government agencies do and why they do it