Exceptional Proff only 2

530

Introduction

 

Topics to be covered include:

· Evaluative criteria for public policy

· Using criteria

· Pitfalls in evaluating criteria

· Economic criteria

· Equity criteria

· Technical, political, and administrative criteria

· The Government Performance and Results Act (GPRA)

· Cost-benefit analysis

· The Delphi method

· Impact assessments

· Policy implementation

 

 

In Lesson 3 we began our deeper look into the policy analysis process through our discussion on defining and analyzing the problem and constructing alternatives. During this discussion, we briefly touched on the subject of policy evaluation. This week we will consider the next step in the process: developing evaluative criteria. We will also identify some of the laws and methods used in evaluation.

 

Evaluative Criteria for Public Policy

A statement of goals for solving a problem is adopted whenever a policy problem is identified to elucidate what the policy alternative should accomplish. A goal statement is usually very broad and offers a long-term solution (or set of solutions) to solving the problem indefinitely. For example, pollution in local rivers may be identified as a problem. The goal statement can be as simple as “Our goal in adopting this policy is to make sure that all of our rivers are clean and safe.” From this goal statement, several objectives are created. Objectives may include “People should be able to safely swim in the Toole River,” “The fish in the Toole River should be healthy and safe to eat,” and “Keeping the Toole River clean and safe should be a task that does not impede or harm local industry.”

To measure these objectives, criteria are identified and are used to compare how different proposed policy alternatives will meet the goals stated in the goal statement and solve the identified problem. These criteria set the rules to follow in comparing different solutions to the problem (or policy alternatives). The primary evaluative criteria used in policy evaluation are as follows:

· Effectiveness , or a measurement of how well a policy achieves its goal;

· Efficiency , a measurement of the cost of a policy in relation to its expected benefits; and

· Equity , a measurement of the level of fairness in the distribution of costs and benefits in society.

In addition, there are five other additional criteria that might be used:

· Ethics , or a measurement of the impact of the policy on cultural norms;

· Political feasibility , a measurement of the level of acceptance of a proposed policy by elected officials;

· Social acceptability , a measurement of the level of public acceptance and support of a proposed policy;

· Administrative feasibility , a measurement of the ability to implement a proposed policy; and

· Technical feasibility , a measurement of the availability and reliability of the technology needed to implement a proposed policy.

Using Criteria

As an example, consider the goal of keeping the Toole River clean and safe. Criteria for a policy to solve this problem may include asking the following questions:

· Effectiveness : To what extent will this alternative improve the water quality of the river?

· Efficiency : How much will it cost to improve water quality using this alternative?

· Technical feasibility : Does this alternative require a degree of knowledge and expertise or the use of equipment, and if so are those requirements available?

· Political feasibility : Is this alternative politically acceptable? Are people in positions of authority likely to support it?

A full evaluation of these criteria will involve the employment of quantitative data. This information may include the following:

· Effectiveness : This alternative will reduce water pollution by how many milligrams of pollutants per liter of water?

· Efficiency : What exact dollar amount will be needed to implement this alternative?

· Technical feasibility : How much equipment is required for this alternative to be implemented? How much time and expense will be involved in training people how to use it?

· Political feasibility : If a statewide poll were taken of the voting-age population, what percentage of them would be likely to favor this alternative?

Pitfalls in Evaluating Criteria

Specifying criteria is a difficult task when a goal statement involves vague, fuzzy, or conflicting goals. In the example given previously, pollution in the Toole River was the problem, and the goal was to make it clean and safe. If more than one alternative is available for cleaning this river, then what is the most important consideration in choosing the best one? Furthermore, the goal statement does not specify what “clean” and “safe” truly mean. Is it even possible to completely clean the river? If everything possible was done to clean the river, would there be a possibility that the fish in the river were still not safe to eat? What standards should be used to determine which levels of toxicity the policymakers should aim for?

It is imperative that there be no ambiguity present in the criteria and their measures used. All results produced by applying them should be uniform, regardless of who measures them, and if the measurements are repeated the results should be the same. Criteria and measures should also be appropriate to the unit of analysis that is used. That is, if the goal of a proposed policy alternative is to clean a river, the unit of measurement should be the level of toxicity acceptable in that river, and in order to be reliable and valid this unit of measurement should be consistently adhered to throughout the evaluation.

Economic Criteria

Economic criteria of one nature or another are almost universal to policy analysis. They may include how the policy will impact the economy, expected public sector revenues, government spending, and a number of other factors. Costs, the most common economic criteria, may include the costs of borrowing funds, decreases in net worth (assets and/or liabilities), costs to directly implement the policy alternative, costs that can and cannot be counted or quantified, cost that can be expressed in dollar amounts, one-time fixed costs, ongoing operations and maintenance costs, opportunity costs (other things that could have been done with the same resources instead). All costs must be identified and counted in a policy evaluation as completely as possible; otherwise, accountability issues may arise over the course of the policy.

MARGINAL COST

Another type of cost criterion that is often employed is  marginal cost . If a good or service is already being produced, how much more will it cost to produce one additional unit? A number of additional types of costs are considered in marginal analysis include fixed costs, which do not vary in the short run regardless of how many units are produced; variable costs, which vary directly with the volume of output of goods or services; average costs, or the total of units of output divided by the total costs of output; and sunk costs, which have already been spent in the past and therefore can be disregarded.

BENEFITS

Benefits should also be considered with economic criteria. The opposite of costs, benefits can be measured similarly in the forms of those that can be directly attributed to the policy alternative, increases in assets and/or liabilities, additional benefits not included in the goals, interest that will accrue or be paid, benefits that can or cannot be counted or quantified, or benefits that can be expressed in dollar amounts.

Equity Criteria

Who pays for a policy? Who benefits from it? In many cases, those who pay for a policy may not be the ones to directly benefit from it but will in fact benefit from it indirectly. For example, a policy that supports public schools may be financed by taxpayers, many of whom do not have children of their own attending public schools. However, the argument might be made that having a well-educated populace benefits all citizens.

If a policy will change the distribution of burdens and benefits in society, equity is a criterion in its evaluation. How should benefits and burdens be distributed in society? That is a matter of opinion, and one that will likely be debated as heavily in the future as it has been in the past. But few people will disagree that discrimination on the basis of arbitrary factors like race, gender, and disability is wrong, or that people who are equally and unequally situated should be treated the same.

TYPES OF EQUITY

There are three main types of equity that are taken into account in policy evaluation.  Horizontal  equity concerns the question of whether burdens and benefits are being shifted among relatively equal societal groups.  Vertical equity concerns whether burdens and benefits are being shifted among unequal groups.  Inter-generational  equity concerns whether burdens or benefits are being shifted from one generation period to another; for instance if younger generations pay more towards a policy and receive less from it than older ones.

EQUAL AND UNEQUAL GROUPS

Equal and unequal groups may be identified on the basis of residence, income, citizenship, race or ethnicity, age, family status, home ownership, educational status, veteran status, criminal record, substance abuse record, or general health. When assessing equity in policy evaluation, the evaluator may encounter questions of how this population should be sub-divided, how groups should be defined, and whether historical criteria, status quo, or desired states should be used. Burdens and benefits in relation to policies must be defined clearly, as should the degree of need that the policy addresses and the ability for those who finance it to pay.

Technical, Political, and Administrative Criteria

Effectiveness is probably the most common criterion used in policy evaluation, as it measures the extent to which the proposed policy will actually solve the problems at hand. Technical feasibility is also of great importance. If a policy alternative proposes requiring all industrial facilities to treat their waste and cool it to a manageable temperature before disposing of it, that alternative may solve the problem of pollution in the Toole River. But how widely available is the equipment that would be required to do this? If it can be obtained at all, who would be required to pay for it—the industries or the public? Another issue involves the measurement of criteria itself. Would it even be possible to measure pollution levels in the water with any acceptable degree of reliability and validity?

In addition to technical feasibility, let’s take a closer look at two other criteria:

POLITICAL CRITERIA

Policy alternatives often require support from elected or appointed officials in the political arena.  If a policy must be passed or voted upon by lawmakers, political criteria must be included in the evaluation of policy alternatives. To what extent will relevant groups, decision makers, legislators, administrators, citizens, neighborhoods, unions, or other formal and informal policy actors accept a proposed policy alternative? Consider the following questions:

· Will the proposed alternative be politically acceptable?

· Will the proposed alternative be appropriate to the values of the communities, societies, or governments that are affected by it?

· Is the proposed alternative legal under current law? Will statutes have to be amended or enacted? Might the alternative be constitutionally challenged in a court of law?

· Will the proposed alternative meet the real or perceived needs of the people whom it was created to help?

ADMINISTRATIVE OPERABILITY

Government agencies are often charged with implementing public policy, so administrative operability and ease should be used as criteria for judging proposed policies. From an administrative standpoint, the evaluator should answer these questions:

· Does this agency have the authority to implement this policy alternative?

· Are top managers, field staff, and support staff in this agency committed to implementing and maintaining this policy alternative?

· Does this agency have the staff, skills, money, training, expertise, and other resources to implement this policy alternative?

· Does this agency have the facilities, equipment, and other support available for implementing and maintaining this policy alternative?

The Government Performance and Results Act (GPRA)

The Government Performance and Results Act  (GPRA) was signed by President Bill Clinton in 1993, though it was not implemented until six years later. Among the GPRA’s requirements when first enacted were strategic planning, annual performance plans, and annual performance reports on federal agencies that would then be released to the public. The GPRA’s provisions apply to all agencies and programs of the federal government and carry the force of law. Please take some time to review some  recent examples of these reports .

This act was aimed at improving public confidence, program performance and effectiveness, service delivery, Congressional decision-making, and internal management, with measures that focus on outcomes, effectiveness, and results established by individual agencies.

BEFORE GPRA

Before the GPRA was enacted, the Program Planning and Budgeting System attempted to achieve the goal of performance management. Performance-based budgeting, zero-based budgeting, and total quality management were other attempts to accomplish what the GPRA would later achieve, though the bills supporting them were never signed into law. At the time of the GPRA’s introduction, committee findings showed that waste and inefficiency in federal programs had undermined the confidence of the American people. This lack of confidence had resulted in a reduction of the federal government’s ability to address and implement public policy effectively. It was also found that congressional policymaking, spending decisions, and program oversight were seriously handicapped by a lack of attention to program performance and results.

GPRA’S PURPOSES

The GPRA’s stated purposes were as follows:

· Improving the American people’s confidence in the government by holding federal agencies accountable for achieving program results;

· Initiating program performance reform through pilot projects in setting program goals, measuring program performance against those goals, and reporting this progress to the public;

· Improving program effectiveness and public accountability by focusing on results, service quality, and customer satisfaction;

· Requiring federal managers to plan for meeting program objectives and providing them with information about program results and service quality;

· Improving Congressional decision-making by providing more objective information on the effectiveness and efficiency of programs and spending; and

· Improving the internal management of the government.

GOVERNMENT PERFORMANCE AND RESULTS MODERNIZATION ACT OF 2010

The GRPA had its weaknesses, and to address them President Barack Obama signed the Government Performance and Results Modernization Act of 2010 into law on January 4, 2010. This update created a more defined framework by better connecting plans, programs, and performance information.  Quarterly instead of annual reporting is now required to increase the use of performance information in program decision-making. The intent of the GRPA is to change behaviors in the executive branch by creating a more fact-based, decision-making framework to implement programs that will be more results oriented.

Cost-benefit Analysis

Cost benefit analysis is often conducted to determine which policy option to adopt. Essentially, it measures the gains and losses in terms of dollars of each course of action to determine which is most fiscally optimal. It is used most often when a program or project begins and when different options or courses of action are being appraised and compared. It can also be used to evaluate a program’s overall impact in dollar amounts.

A cost benefit analysis will total the costs of a program or activity and compare them against its total benefits. It assumes that these costs and benefits can be expressed in quantitative values, including both tangible and intangible returns. A major advantage of this type of analysis is that it forces policymakers to consider the various factors that should and will influence their choices.

Decisions are made through a cost benefit analysis by comparing the net present value (NPV) of the policy’s costs with the NPV of its benefits. Decisions are based on whether total benefits are less or more than total costs. Costs and benefits that occur in the future are typically not as easy to compute when creating a cost benefit analysis; thus, it is necessary to reduce the value of future costs or benefits to place them on par with those incurred today. Public sector analyses generally use a rate of five to six percent when computing future costs.

A cost benefit analysis should be conducted for any project where policy development is involved in conjunction with capital spending, the use of assets, or the setting of standards or policymaking. It will sometimes be very simple and easy, but it may also require complex economic analysis. Whatever the case, an exhaustive list of all the different costs and benefits that could arise should be made at the commencement of the analysis. Some of these costs and benefits may later be excluded, but it is important that no aspects of the analysis are overlooked. Information on costs, benefits, and risks can usually not be determined with absolute certainty when the future is involved.

The Delphi Method

Developed by the Rand Corporation in the 1950s and named after an ancient Greek oracle that could predict the future, the Delphi technique is designed as a group communication process that aims at conducting detailed examinations and discussions of a specific issue for the purpose of setting goals, evaluating policies, or predicting future events. Instead of trying to identify “what is,” as a typical survey does, the Delphi technique addresses the question of “what could/should be.” The Delphi technique can be used to:

· determine or develop a range of possible program alternatives;

· explore underlying assumptions that may lead to different judgments;

· seek out information for creating a consensus;

· correlating judgments concerning a wide range of disciplines; or

· educate a group about a topic and the wide range of aspects it involves.

The Delphi technique involves the use of a series of questionnaires to collect data from a panel of selected subjects. Unlike other data gathering and analysis techniques, this method repeats information in different manners to develop a consensus on a specific topic. In doing so, it encourages Delphi participants to reassess their initial judgments about the information. Participants are allowed to modify opinions and statements made earlier in the study as the study progresses.

ANONYMITY OF PARTICIPANTS

Very important to the Delphi process is the anonymity of the participants. Confidentiality is ensured by a wide geographic dispersion of the subjects and the use of electronic communication such as e-mail in the exchange of information. Controlled feedback in the Delphi process reduces the effect of noise, or communication that may distort the data and that deals with group or individual interests rather than finding a solution for the problem at hand. The controlled feedback process involves a well-organized summary of the prior information provided distributed to the subjects to allow each participant an opportunity to generate additional insights and more thoroughly clarify their previously held views. Participants are expected to become more concerned with solving problems, to offer more insightful opinions, and to minimize the effects of noise.

STATISTICAL ANALYSIS

The ability to use statistical analysis techniques in a Delphi study further reduces the potential of participants to be pressured into group conformity. Statistical analysis can also ensure that each participant’s opinions in a Delphi study are well represented because even at the end of a study the opinions given may spread over a wide span. Each participant would not be pressured, either in reality or their own perception, to conform to another participant’s responses in an attempt to obey social norms, customs, organizational culture, or their standing within their profession. An objective and impartial analysis and summarization of the collected data is the end result of a Delphi study.

Impact Assessments

Impact assessment measures the effectiveness of organizational activities and judging the significance of changes brought about by those activities. Being able to assess and articulate impact is a powerful means of communicating a policy, internally and externally. Impact is seen as the positive and negative, intended or unintended long-term results produced by a policy either directly or indirectly, and should be seen as the contribution of the intervention to the overall goal.

ROLES IN POLICY EVALUATION

Impact analysis can play two roles in policy evaluation: predicting what the impact of an intervention might be so that policymakers can be informed of the potential risks, and assessing the extent that policy corrects the problem it was intended to address. Typically, impact assessment focuses on the effects of the policy, rather than the cost and efficiency of the intervention, its unintended effects, and how the experience from this policy can be used to improve the design of future policies.

The majority of impact assessments can help address broader evaluation questions, and is often used with other evaluative techniques in order to reach conclusions. However, impact assessment should never be confused with evaluation; an impact assessment is not designed to answer as many questions as an evaluation, and impact assessments tend to focus on a narrow and tightly defined set of impacts, especially those that focus overly on economic impacts.

CAUSES AND EFFECTS

Impact assessment is a theory-based activity in several senses. It involves establishing a “theory of change,” a “program theory,” or “intervention logic.” In other words, it must include a description of the causes and effects leading from a policy’s introduction to its desired effects. The essence of impact assessment is to establish a chain of causation from policymaking to impact and to measure or describe the changes that occur along that chain. This approach has the advantage of specificity and focus but also tends to limit observed effects to categories that have been predicted, by definition omitting unexpected effects, including perverse outcomes or undesirable effects that the policymakers have not anticipated. Impact assessments can focus on one set of potential effects at the cost of others even when this is not explicitly obvious.

ASSESSING IMPACT

There is no single method for assessing impact. Quantitative and qualitative data from a number of sources should be evaluated against a number of fields related to the policy in this process. In assessing this data, judgments then can be made against each field and an overall assessment will indicate the impact of the policy. The actual result of the policy is identified, so that it can be compared with what was intended to happen as a result of the policy.

Assessing impact involves making judgments about how well the policy met the needs of the organization and how well it was implemented. Implementation can have unintended effects (for example, on the local economy or environment) that can be positive or negative. Negative effects are easy to identify, but positive effects typically require sufficient data and a strong evaluation in order to elucidate. Assessing overall impact of a policy requires a degree of judgment not only of the policy itself but also its longer-term consequences. This may involve asking the simple question of what might have happened if the policy had not been made and the previous state of affairs had been allowed to continue as it was.

Policy Implementation

Policy implementation is the final step of the policy analysis process. For this step to be successful, the following criteria must be met:

· Sound theory must underlie the policy and the group of people it targets, and this theory must link the group behavior to the objectives of the policy.

· The objectives that structure the policy must not be ambiguous. Clarity in policy will maximize compliance to the policy among the policy’s subjects. There must be no conflicting goals or incomplete information. Otherwise, the policy may be met with hostility, resistance, or non-compliance.

· Leaders who have the necessary skills, support the policy’s objectives, and have the resources needed must be available throughout the implementation. Agencies involved must be supportive of the policy’s objectives and must place them high on their list of priorities.

· Active support from potentially affected parties must be present. Such parties may include legislators, courts, interest groups, or other units of government. A lack of interest or active opposition to the policy from organized groups who have the resources to combat the policy must not be present for the policy to be successful.

· Conflict with other public policies cannot exist, and the policy must not be undermined by changing socio-economic conditions or competition from other issues and demands. However, priorities can be expected to change over time.

The success of a policy can be measured by how the people affected by the policy change their behavior, or the degree of compliance (or non-compliance) with the policy can be measured. Situations that may result in non-compliance may include the following:

· A breakdown in communication of the policy occurs, the people it is meant to target believe it does not pertain to them, or their expectations under the policy are unclear.

· The people targeted by the policy do not have the time, skill, resources, or mental ability to comply with the policy.

· The people targeted may disagree with the policy, or those charged with implementing the policy did not follow proper procedures.

· The actions required for policy compliance are too difficult, and the people targeted are simply unable to follow them.

· People perceive the government as acting illegally, respect for authority is lacking, or little or no punishment exists for not following the policy.

The policymakers can use negative sanctions to compel those targeted by the policy to comply. These sanctions can include prosecution, fines, imprisonment, unfavorable publicity, revocation of licenses or other privileges, seizure and destruction of goods and property, assessment of damages, denial of services or benefits, cease and desist orders, and other means. Likewise, positive sanctions such as favorable publicity, price supports, tax credits, rewards, voluntary standards, training, contracts, subsidies, informal procedures, and other positive sanctions can be employed to encourage compliance with policies.

Conclusion

This week we discussed evaluation criteria and methods to conduct evaluations. Evaluating a public policy begins with identifying goals and objectives. How well the policy met those goals and objectives is measured using criteria. These criteria can be economic, equitable, technical, political, or administrative in nature. The Government Performance and Results Act (GPRA), enacted in 1993 and updated in 2010, improved the process through the use of strategic planning, annual performance plans, and annual performance reports on federal agencies. Means such as cost benefit analyses, the Delphi technique, and impact assessments are further used to facilitate this process. Finally, a policy’s implementation is evaluated with compliance by the policy’s target population as the primary metric.

References

Government Performance and Results Act (GPRA) Related Materials. (n.d.). Retrieved November 21, 2017, from https://obamawhitehouse.archives.gov/omb/mgmt-gpra/index-gpra

Government Performance and Results Act (GPRA) Reports . (n.d.). Retrieved November 21, 2017, from https://www.federalreserve.gov/publications/gpra.htm

Kaiser, F. M., and McMurtry, V.A. (1998, March 17). Government Performance and Results Act: Proposed Amendments (H.R. 2883). Congressional Research Service. Retrieved from https://digital.library.unt.edu/ark:/67531/metacrs672/m1/1/high_res_d/98-224gov_1998Mar17.pdf.

U.S. Congress. S.20 - Government Performance and Results Act of 1993. Retrieved from https://www.congress.gov/bill/103rd-congress/senate-bill/00020.