Public Administration ‒ The Good, The Bad, The Ugly

PLANNING, IMPLEMENTATION, AND EVALUATION

Developing policies and programs, putting them into operation, and measuring their success or failure constitute an important and recurring cycle for public and nonprofit managers. For example, a new monitoring program is initiated in the Balkans. To support that policy, the U.S. Navy organizes a fleet of vessels to expand our presence in the Persian Gulf, while political and military leaders assess the operation and decide what to do next. Similarly, a new policy involves sending literature on AIDS to all households in the United States. A group in the surgeon general's office convenes to monitor the operation. Both the efficiency of the mailing and its effectiveness as an educational device are discussed. Meanwhile, a local parks and recreation department joins with nonprofit organizations to develop a program for handicapped athletes. After staff and money are acquired to support the program and it begins operations, the department director asks whether the program is worth the time and energy it seems to be taking from other tasks. Repeatedly, plans are made, policies and programs are implemented, and the work of the organization is evaluated (Birkland, 2011; Sabatier, 2007; Stone, 2011).

Recently these issues have taken on increased importance as managers in the public and

nonprofit sectors have been asked to do with less, while at the same time providing more and better services. This situation has led many to call for “managing for results”—that is,

clearly stating goals and objectives in terms of public outcomes, designing and implementing

programs, and then measuring the performance of the government or other agency against

established standards. The idea of “managing for results” or “performance management” suggests the importance of bringing together careful planning, implementation, and

evaluation.

Although planning, implementation, and evaluation all require knowledge of the political and ethical context of public administration and certain personal and interpersonal skills,

various technical aids have been developed during the past decade to assist the manager in

these three areas. These technical aids include Web-based participative planning, strategic

planning, and specific quantitative methods for measuring performance. Moreover, these techniques reflect a change in the way public and nonprofit organizations account for their

actions. Citizens, lawmakers, and other advocacy groups increasingly hold agencies

accountable not only for their efficiency in expending public and charitable resources, but

also for their effectiveness in achieving public outcomes. Techniques such as strategic planning and performance measurement link the actions of public and nonprofit organizations

with specific, measurable results. Through the use of such techniques, public administrators

may account for both their own efforts and the impact those efforts have in each given policy area.

Networking

The Government Accountability Office website at -w-w-w-.-g-a-o-.-g-o-v provides resources

relating to managing for results. For additional resources on productivity and performance

management, see the American Productivity and Quality Center at -w-w-w-.-a-p-q-c-.-o-r-g.

Planning On a daily basis, all managers engage in planning. But organizations, and indeed entire

governments, engage in more formal planning processes, often involving a wide range of

participants and the development of considerable data and other information. Planning

typically leads to the development of alternative courses of action, and each must be

examined to decide which way to go. Depending on the level of the problem, the process of examining and choosing among alternatives may involve the manager in either policy

analysis or program design or both.

In a planning effort that captured the public's attention like few in recent memory, New York

City was faced with the momentous task of deciding how to rebuild the devastated downtown area after the September 11, 2001, terrorist attacks. New York's Regional

Planning Association convened the Civic Alliance to spearhead a participative planning

process (see -w-w-w-.-c-i-v-i-c---a-l-l-i-a-n-c-e-.-o-r-g). The Civic Alliance is a coalition of

approximately eighty-five different civic groups, businesses, universities, foundations, and community groups that joined together to solicit public comment and involvement and to

provide recommendations to the Lower Manhattan Development Corporation, a joint city

and state organization charged with the rebuilding and revitalization effort. Several different approaches were used to provide citizens with the opportunity to become involved and have

their ideas heard in the process. For example, an event called “Listening to the City”

employed both face-to-face dialogue and technology in one-day forums. More than 4,500

people participated in small roundtable discussions while a network of people with laptop computers recorded and immediately transmitted the ideas generated to a “theme team.”

The theme team identified key themes and concepts and reported them back to all of the

participants. Participants also had electronic keypads to record their preferences and

answers to specific questions. Subsequent to these forums, an additional 800 people exchanged approximately 10,000 messages in small virtual discussion groups and

participated in numerous polls in online dialogue over a two-week period to express their

views and preferences.

In the first phase of this project, 230 workshops and an interactive website were used to

solicit answers to three primary questions: What have we lost? How have we changed? What

should be done? In the subsequent phases, participants were asked to react to proposed

designs for rebuilding on the site of the World Trade Center. Although the revitalization project is not completed, the process used in New York City illustrates the potential of using

innovative strategies and approaches to garner public participation in large-scale planning

projects.

Strategic Planning

Planning is not only used when responding to a crisis, but has been increasingly employed

as an ongoing process in the public and nonprofit sectors. A number of writers have

commented on the rapidity of the social and technological changes we are now experiencing

and on the turbulence and complexity that such changes generate. In an effort to recognize and respond to such changes, many private corporations began programs in the 1960s and

1970s to systematically plan for future development. The success of these programs is now

confirmed by the fact that more than half of publicly traded companies use strategic planning

in some form. Strategic planning helps an organization match its objectives and capabilities to the anticipated demands of the environment to produce a plan of action that will ensure

achievement of objectives (Andrews, 2012; Bryson, 2004; Greiner & Cummings, 2009).

We can differentiate strategic planning from more familiar long-range planning activities in

several ways. Long-range planning primarily concerns establishing goals or performance objectives over a period of time; it is less concerned with specific steps that must be

undertaken to achieve those goals. Strategic planning, on the other hand, implies that a series

of action steps will be developed as part of the planning process and that these steps will

guide the organization's activities in the immediate future. Strategic planning takes the

future into account, but in such a way as to improve present decisions.

A second way that strategic planning differs from long-range planning is its special attention

to environmental complexity. The organization is not assumed to exist in a vacuum; rather, both the organization's objectives and the steps to achieve them are seen in the context of

the resources and constraints presented by the organization's environment.

A final distinction between the two types of planning is that strategic planning, especially in

the public sector, is a process that must involve many individuals at many levels. As most managers know quite well, effective changes in organizational practices are most readily

accomplished by involving all those who will be affected by the change. This general rule is

especially applicable to changes generated through a process of strategic planning.

Public organizations undertake strategic planning efforts for many reasons: (1) to give

clarity and direction to the organization, (2) to choose from among competing goals and

activities, (3) to cope with expected shifts in the environment, and (4) to bring together the

thoughts and ideas of all participants in the work of the organization. Most importantly, planning activities provide an opportunity for the widespread involvement of leaders and

citizens in defining the direction of the community or the agency as it moves into the futur e,

thus building trust and commitment.

Planning for Planning

As a manager, you may wonder whether such activities are appropriate for your jurisdiction

or agency. Whatever your work—at any level of government or in a nonprofit organization—

you will find precedents for planning. Many federal, state, and local agencies have begun

strategic planning programs over the past several years, as have voluntary associations,

human service organizations, and job training programs. The key seems to be that any

organization is a candidate for strategic planning if, by allocation of resources, it can

significantly influence either formulation or implementation of public policy.

You may, of course, question whether strategic planning is worth the costs in terms of

consultant fees, research and data analysis, and time away from other duties. The best gauges

for assessing costs are: (1) Is it likely that careful planning will lead to reduced operating

costs or increased productivity over the long run? and (2) What might the organization lose in the absence of a more comprehensive and integrated approach to the future?

The latter question has become increasingly important to those in local governments, who

now realize that they must compete with other communities in attracting industry, providing

amenities, and maintaining the population base. The issue, however, must be treated differently when an administrative agency such as a state government department is

considering planning. Although strategic planning might make the agency more competitive

in attracting resources from the executive or legislature, this clearly should not be the purpose of planning. Rather, the agency should use strategic planning to involve key

stakeholders in assessing the unit's work and the possibilities for improving its services. The

process may indeed lead to requests for further funding, but it may also suggest ways to more

effectively utilize existing resources or even ways to reduce the scope of activities.

What Would You Do?

You are a member of the management team of a medium-sized Florida oceanfront vacation

and retirement community. You have been asked to make recommendations to the city

council concerning elements that should be part of a new vision and an accompanying

strategic plan for the city. What would you do?

Because of budgetary uncertainties, you may also question whether the time is right for

planning activities. Some say that planning can't take place without solid information about

funding levels. But the opposite argument is compelling—that planning is most essential in times of uncertainty, for these are exactly the times when you most need to be in control of

your own destiny. Times of uncertainty do not mitigate the need for planning; they intensify

it.

Managers in the public sector voice a related argument: that periodic changes in political leadership make planning more difficult than in private industry. Again, the opposite

argument is compelling: in times of transition, planning can provide continuity. Even when

new leadership wishes to change the directions specified in an earlier planning effort,

changes can be made with greater clarity and aimed more readily toward critical concerns if a plan is in place.

Finally, you may wonder whether strategic planning efforts are consistent with your

organization's commitment to democratic or participatory processes. Here lies the most significant difference between strategic planning in the public and private sectors. Whereas

planning in the private sector may involve many people throughout an organization, it

remains centered and directed at the top, because that is where the private interests of the

firm are most clearly articulated. In the public sector, however, every effort must be made to

significantly involve all those who play an important role in the jurisdiction or the agency. For example, a local government planning effort should involve not only elected leadership

and city staff, but also many others with a stake in the outcome: unions, neighborhood

associations, chambers of commerce, civic organizations, and so forth. Similarly, a state

government agency's planning effort should involve people from all levels of the organization, members of constituent groups, elected officials, people from other agencies

and other levels of government, and representatives of the general citizenry.

Strategic planning in the public sector must be a highly participatory process; consequently,

this participation opens the possibility of building new understanding among various groups. Many communities that have engaged in strategic planning have found that the

process brought together various groups in a way not previously possible. Strategic planning

may therefore be undertaken to achieve both direction and commitment.

Organizing for Planning

The planning process can proceed in a number of different ways, but the most common

approach is to form a central planning group to work closely with an outside consultant to

obtain information and make commitments to various new directions. In a local community, the group might include the city's political leadership; representatives of the city

administration (for example, the city manager); representatives of business, industry, and

labor; members of neighborhood associations; and so on. For a federal or state agency, on

the other hand, the major planning group might consist of the agency director, managers

from the next organizational level below, and selected program directors. The planning

group in a nonprofit organization might include the executive director, members of the

board, staff members, and representatives of constituent groups.

Steps in Planning

Once it has been brought together, the planning group will want to give its attention to four

primary concerns before developing action strategies (see the box “Exploring Concepts:

Steps in Strategic Planning”). The group should consider (1) the organization's mission or

objectives, (2) an assessment of the environment in terms of both opportunities and

constraints, (3) an examination of the organization's existing strengths and weaknesses, and

(4) the values, interests, and aspirations of those important to the organization's future. Consideration of these issues will lead to several strategic alternatives—perhaps stated as

“scenarios for the future”—and to the choice of a particular direction in which the

organization should move. Finally, a set of action steps or implementation items will be

developed to indicate what must be done immediately to put the organization in the proper position to face the future most effectively.

Exploring Concepts

STEPS IN STRATEGIC PLANNING

1. Statement of mission or objectives

2. Environmental analysis

3. Assessment of strengths and weaknesses 4. Analysis of organizational leaders' values

5. Development of alternative strategies

Statement of Mission or Objectives Arriving at a concise, yet inclusive, mission

statement of the organization is a difficult step in the planning process. Although most organizations have a general sense of their mission, questions often arise that cannot

be readily answered in terms of stated objectives. Having a specific mission statement,

however, provides an identity for the organization, as well as a guideline for future decisions and a standard against which to measure specific actions.

Because arriving at a mission statement may imply certain strategies, care should be taken

to consider alternative approaches to the organization's goals. A mission statement might

indicate, for instance, whether a city wishes to seek a broad industrial base or focus on particular types of businesses, such as tourism or high-tech industries. Similarly, a university

mission statement might indicate whether the institution seeks a broad range of programs

in all areas or a limited number of exceedingly high-quality programs. The mission statement

of a state agency might comment on the desired range of clientele, responsiveness to changes in the environment, or quality of service. If there is doubt or debate about items, they should

be carried forward as elements of strategy for later consideration.

Environmental Analysis After developing a mission statement, the planning group should move to an analysis of the environment within which the organization

operates. This assessment should include legal and political considerations, social and

cultural trends, economic circumstances, technological developments, and, where

appropriate, the organization's competitive or “market” position. Each area should be

examined in terms of the present environment and how it is likely to change in the

future. This assessment leads the group toward identifying possibilities for reducing

constraints and extending opportunities.

Assessment of Strengths and Weaknesses At this point, the planning group can turn its attention toward assessing the organization's existing capabilities—its strengths

and weaknesses. The analysis should be as forthright and inclusive as possible, taking

into account financial resources (including changing patterns of funding), human

resources (including political and managerial strengths and weaknesses), the operation of both technical and organizational systems, and quality of work. This

assessment of capabilities should relate as directly as possible to the stated mission

of the organization. For example, an agency involved in facilities design and

construction might want to consider the age and condition of facilities, the number

and abilities of architects and engineers, the number and frequency of design projects,

and the unit's standing among other similar organizations. Examining strengths and

weaknesses should be accompanied by some attention to programs that might significantly improve capabilities in one or more areas.

Analysis of Organizational Leaders' Values A next step in preparing to develop

strategic alternatives is to take into account the values, interests, and aspirations of

those who will guide the organization into the future. People will respond to the same environmental and organizational analysis in different ways. In business, for example,

some will be perfectly satisfied with the security of a stable market share, whereas

others will be willing to take greater risks in the hope of greater payoffs. Leaders vary

in terms of creativity, energy, and commitment. Yet, if a plan is to be effectively implemented, it must reflect the concerns and interests of those who will play major

roles in shaping the future of the organization.

Development of Alternative Strategies At this point, the planning group can move to formulate alternative strategies. These strategies can take several forms; however,

one useful way to proceed is to draw up alternative “scenarios of the future,”

indicating what the organization might look like five, ten, or twenty years into the

future. The scenarios should indicate new directions the organization might take; pessimistic, realistic, and optimistic interpretations of its future; and factors likely to

influence these future patterns. It is helpful to develop more than one scenario and then use

them as competing viewpoints from which to debate the merits of various alternatives. From

a thorough discussion of the scenarios, one or more strategies will emerge. The strategy should be chosen that most effectively moves the organization toward its mission, given

environmental opportunities and constraints, organizational strengths and weaknesses, and

the values, interests, and aspirations of the leadership. After developing the strategic orientation, the planning group should be pressed to identify specific action steps for

implementing the strategy.

The Logic of Policy Analysis

One possible outcome of a formal planning process is that the need for new policies will be identified. (The need for new policies can be generated in other ways as well, many of which

we discussed in Chapter 2.) A local group considering economic development issues might

recognize the need for new tax incentives for industries interested in locating in the

community. A state welfare department planning group might focus on the relationship between providing day care and job training. Or a nonprofit organization might decide there

is a need for a new publications program. In each case, a problem is identified and the

question arises as to whether a new approach to the problem—a new policy—might help.

Many issues may come up. Exactly what is the nature of the problem? What would we be

trying to achieve with the new policy? What might be alternative approaches? What might

we expect from each alternative? What criteria would we use to evaluate alternatives? Which

alternative would best meet our criteria? Answering questions like these is the basis of

analyzing public policies. We can therefore define policy analysis as the process of researching or analyzing public problems to give policy makers specific information about the range of

available policy options and their advantages and disadvantages. There are several ways you

might become involved in policy analysis. All managers engage almost daily in a sort of informal

analysis of public policies; they encounter new problems and consider alternative policies. Often a more formal review of policy options is called for. Sometimes staff members perform the

analysis; many public organizations employ policy analysts to work on just such problems. In

other cases, another governmental agency may help; for example, the Office of Management

and Budget, as well as its counterparts in many states, develops policy reports. Policy analysis might also be performed by legislative staff or legislative research groups. Finally, many

analyses are performed by consultants, including university consultants, where the public

manager acts as a client, issues the contract, monitors the work, and receives the final report.

Even though, as a manager, you may not perform the analysis yourself, you must be able to

distinguish between high-quality analysis and work of limited usefulness (Fischer, Miller, &

Sidney, 2007; Iris, 2005; Kraft & Furlong, 2009; Sabatier, 2007; Stone, 2011).

Steps in Policy Analysis

Broadly speaking, most policy analyses attempt to follow a “rational” model of decision

making, involving five major steps: (1) formulating the problem, (2) establishing criteria for

evaluation, (3) developing policy alternatives, (4) considering the expected impact of the various alternatives, and (5) ranking the alternatives according to the established criteria.

(See the box “Exploring Concepts: Steps in Policy Analysis.”) As a simple illustration, think

about how you might decide what would be the best route from home to work (Quade, 1989,

pp. 33–34). If you assumed at the outset that the “best” route is the shortest, then you could simply lay out the alternative routes on a map and select the shortest. (Using a map would

in effect create a “model” that would help in your analysis.) As in almost all policy analyses,

however, there may be more than one criterion involved. For example, the shortest route

might involve more traffic and take longer to drive. The shortest travel time might then constitute a second criterion, but it would require a more sophisticated model than a map,

taking into account traffic congestion and perhaps other variables. Just thinking through the

various complications that might arise in this “simple” example, you can get some sense of

the difficulties you might encounter in moving through the five stages of a more comprehensive policy analysis.

Exploring Concepts

STEPS IN POLICY ANALYSIS

6. Defining the problem

7. Setting objectives and criteria

8. Developing alternatives 9. Analyzing various policies

10. Ranking and choosing

Defining the Problem There are obviously many problems facing any public

organization and, correspondingly, many opportunities to analyze policy alternatives.

Someone, however, must decide about the problem and how the analysis will proceed.

This someone—the sponsor of the analysis—may be a legislator, an elected chief executive,

or an agency manager. But, in any case, the one who will perform the analysis—the analyst— should seek as clear a statement of the problem as possible and as much information about

the nature of the problem and the range of solutions. Why has the problem surfaced? Who is

affected? How does this problem relate to similar problems? What policy options have

already been tried? What is the range of policies that would be feasible, both economically and politically? What resources are available to support the analysis?

Obviously, how the question is initially formulated will guide the analyst toward certain

possibilities and away from others, so it is important at the outset to be as clear as possible without unnecessarily cutting off alternatives. The sponsor might ask, for example, “How can

we provide adequate shelter for the homeless in our community this winter?” This statement

permits exploring alternatives ranging from subsidizing existing shelters to building new

shelters. If, however, certain options, such as building new shelters, are clearly out of the question due to time or money, then the analyst should be advised of these limitations.

Sometimes the problem is only vaguely understood at the outset, and part of the analyst's

job is to develop a background statement or issue a paper that outlines the problem. In some

cases, gathering information at the library will be helpful, especially in laying out the history of the problem, discovering approaches used in other jurisdictions, and becoming aware of

technical developments in the field. The analyst may want to talk with other people, perhaps

in other jurisdictions, to see what their experience has taught them. People in other governments, other levels of government, and other agencies at the same level can be helpful.

The analyst can also gather information from those involved. In our example, the analyst

would probably want to talk with those already involved in providing shelter. A statistical

survey might even be possible. Finally, agency records and statistics might be helpful.

Throughout these initial information-gathering efforts, the analyst wants to develop an idea

of how different people and different groups perceive the problem and possible solutions.

Setting Objectives and Criteria As we have seen, establishing objectives for a new

policy or criteria for judging alternatives is often quite difficult. In some rare agencies,

the existing values and preferences are clear enough to guide choices. The manager

might be able to say, “It's worth much more to our agency to achieve Result A than

Result B, C, or D. Therefore, whenever the choice presents itself, choose A.” But in most policy areas, there are likely to be multiple and often conflicting objectives. To route

a highway through an urban area, for example, one must consider factors such as the

cost of the project, how many and who might use the highway, the number of houses

and other properties that might be displaced, and the impact of noise and pollution on adjacent neighborhoods. How does one begin to rank all the factors?

There are other problems in selecting criteria. For example, criteria may differ among

different levels of the organization. A constant problem for decision makers is to be sure that criteria used at one level are consistent with those at another level. A particular course of

action might fit the criteria developed at one level, but so distort the use of resources at the

next higher level as to make the choice inappropriate. Criteria must also be stated as

completely as possible. An analyst might be told to seek a solution that maximizes ou tput at minimum cost and then discover that no single alternative can meet both criteria. Which is

more important?

Finally, choosing criteria depends on individual perspective. Most policy areas have many

different stakeholders—people who are involved in and affected by the policy decision. These may include legislators, agency personnel, client groups, and other interest groups,

and each group may feel quite differently about what is most important. In the design of a

new highway, for example, a neighborhood association might place highest value on

environmental concerns, while someone who lives in the suburbs might be most concerned

with finding the shortest, quickest route to work. Different criteria compete for prominence

in any policy analysis. And, often, which criteria receive greatest prominence is a political

decision of legislators or high-ranking administrators.

Developing Alternatives Developing alternative policies is without question the most

creative phase of policy analysis, for it is here that the analyst must move beyond easy

solutions and develop innovative approaches to public problems. Different

alternatives often derive from different assumptions about the problem. For example, should the welfare system be oriented toward providing support at home for

impoverished mothers or should it enable mothers to work by providing day care?

Should day care be addressed by building new centers or by providing tax credits or vouchers to subsidize attendance at existing centers? Obviously, answers to questions

about alternative approaches to child support depend on interpretation of both the

causes of poverty and the motivations of the mothers. To develop a complete range of

alternatives, the analyst must assume the perspectives of many different

stakeholders.

Another way to develop far-ranging alternatives is to consider the relationship between the

particular problem and other similar issues. For example, adequate care for the homeless

ties to issues of health care, financial support for housing, welfare policy, and perhaps such areas as mental health and Social Security. Again, alternatives that take the various

interrelated concerns into account are likely to be generated if the analyst considers the

views of many different stakeholders. Rather than saying, “How can my organization solve this problem?” the analyst should ask, “How can this problem be solved?”

Analyzing Various Policies Having generated a number of realistic policy alternatives,

the analyst must now assess the likely impact of each alternative. How one analyzes

the impact will vary according to the particular type of policy. In some policy areas, including some of major importance, limited information about possible impacts will

be available. The analyst can only make intuitive judgments based on his or her

experience and the experience of others. In other cases, however, one can gather specific data and analyze it by means of quantitative techniques. In the urban highway

example, data could be gathered and analyzed to determine cost per mile, load-

bearing capabilities, travel time for users, and a variety of other factors.

Occasionally, actual experiments with several policy options may be possible, sometimes with an experimental design similar to that used in the natural sciences. That is, the behavior

of a particular target population may be compared to that of a control group when only one

variable (the policy) is changed. Applied to large-scale social problems, such experiments

may be quite costly, but they may also save considerable time and money in the long run. Sometimes it is appropriate to spend millions to save billions. (We should also note the

ethical problems associated with providing a treatment expected to be beneficial to one

group but intentionally denying it to another “control” group. Is it ethical to deny some

people a treatment you think will be beneficial?) A less formal means of policy

experimentation occurs when one state or locality tries a particular policy approach and

makes the results available to other communities. Sometimes this form of experimentation

is simply the result of different groups trying different programs, but sometimes it is conscious. When state and local groups pressured the Carter administration to move the

management of the small cities' portion of the Community Development Block Grant

program to the states, Wisconsin and Kentucky were asked to run the program on an

experimental basis. Their success in tailoring programs to local needs led to legislation allowing all other states to assume administration of the program (Jennings et al., 1986; for

more recent examples, see Beland & Waddan, 2012).

Ranking and Choosing The final step in the analytic process is to compare the impacts associated with various alternatives and the criteria for evaluation established

earlier. Alternatives can then be ranked in terms of their respective impacts. When

both the criteria and impact levels are fairly straightforward, a simple comparison of

possible effects may readily show which choice should be made; other cases may be

more complex. The highway construction example, for instance, might yield three or

four alternative proposals and as many as twenty criteria by which to evaluate the

alternatives. One way to treat such cases is to simply lay out the expected results of

each alternative in terms of the various criteria, leaving the task of comparing the data

and ranking the alternatives to the decision maker. Sometimes more sophisticated

quantitative techniques are available to the analyst.

Costs and Benefits

One of the most straightforward quantitative techniques is the cost-effectiveness approach,

which compares policies by quantifying their total costs and effects. Costs are usually measured

in monetary terms, but effects may be measured in units of any type.

Typically, the cost-effectiveness approach takes one of two forms. First, the level of

effectiveness can be fixed, and one can search for the alternative that achieves this level at the least cost. If, for example, we want to increase the number of houses in a community that

get tested for radon by 25 percent, would it be cheaper to hire inspectors or to spend money

on advertising so that homeowners would do the inspection themselves? A second approach

fixes the budget amount and then asks which alternative will provide the highest level of effectiveness for that amount. If we want to spend no more than $50,000 a year on radon

inspections, which two approaches will result in a higher number of inspections?

The cost-effectiveness model is widely used because it is quite flexible and does not demand

the same degree of precision as other approaches. Cost-effectiveness is especially useful when the relative merits of competing proposals, such as different child-care delivery

mechanisms, are being debated. It is not as useful in comparing questions of absolute merits,

however, such as whether to allocate resources to early childhood programs or to radon

testing. Moreover, the cost-effectiveness approach may be somewhat limited where criteria

and impacts are more complex.

Closely related to cost-effectiveness is cost-benefit analysis. Essentially, the cost-benefit

approach involves identifying and quantifying both the negative impacts (costs) and positive impacts (benefits) of a proposal, then subtracting one from the other to arrive at a measure of

net benefit. In contrast to cost-effectiveness analysis, the cost-benefit approach seeks to

establish both the total monetary costs and total monetary benefits of a proposal. The logic of

cost-benefit analysis is obvious, but applying it to policy proposals that involve large expenditures and produce difficult-to-measure results can be quite complicated.

There are several advantages to cost-benefit analysis. If programs can be evaluated in terms

of costs and benefits, the approach can result in rather precise recommendations. But even if it is difficult to calculate costs or benefits, focusing on the two areas may help clarify the

manager's thinking about a proposal. Legislation often requires that cost-benefit analysis

precede particular policy changes, especially in environmental or regulatory policy.

Several factors make it difficult to assess the costs and benefits of a particular program. First, the analyst will be asked to come up with measures of both costs and benefits and reduce

them to a common unit of measure (usually money). But in analyzing a proposed new

highway, can we accurately portray the fatality rate for similar highway segments as a

measure of safety? And, if so, how can we translate the rate of fatalities into dollars? Second,

we should always remember that the final calculated cost-benefit ratio is not the only basis

for choosing one alternative over another. Despite the ratio of costs and benefits in our

highway example, a particular level of fatalities may simply be considered too high, either politically or ethically.

Typically, costs are thought of as inputs and benefits as outputs. Costs might include one-

time items such as research and development, buildings and facilities, land acquisition,

equipment purchases, and so on. Costs might also include recurring budgetary items such as personnel, rent, maintenance, administrative overhead, insurance, and so forth. Because

these expenditures take place over time, calculations usually take into account the time value

of money—the fact that people generally are not as willing to pay for something in the future as in the present. Although the particular calculations are beyond the scope of this text,

taking time into account enables us to answer questions such as whether Project A with low

initial cost but high maintenance is better than Project B with high initial cost but low

maintenance.

Benefits, based on outputs, include both positive and negative effects. (The negative effects

of a program obviously might be calculated either as increases in cost or decreases in

benefits. They are usually the latter.) Positive benefits might include reduction in disease or

improved drinking water or increased highway safety. Negative benefits might include increased noise and pollution from constructing a new airport. Again, some effort to

translate positive or negative benefits into monetary terms would have to be made.

Obviously, measuring outputs and translating them into dollars are exceedingly difficult tasks. For example, eliminating a disease might increase productivity, which could be

measured, but also reduce pain and suffering, which would be more difficult to measure.

Omitting these factors because they are hard to measure biases the analysis, but assigning a

dollar value to them might do the same. Consequently, the quantitative presentation of costs and benefits is often accompanied by an explanation of additional qualitative considerations.

Other Quantitative Techniques

In addition to cost-effectiveness and cost-benefit approaches, there are many other

techniques to aid policy analysis. It is not necessary to examine the mathematical formulas,

but it is helpful to understand the logic they depend on. Let us examine the following payoff

matrices with that goal in mind. Assume a simple example: hiring an office worker who will

need proficiency in computer operation and budgeting. After interviewing two applicants, A and B, you feel that A is stronger (rated 1) than B in both areas. Your thoughts might be

modeled as shown in Table 4.1.

TABLE 4.1

Simple Payoff Matrix

Value Measures

Computers Budgeting

Candidate A 1 1 Candidate B 2 2

© Cengage Learning

Your choice here is simple, because one candidate is clearly superior in both respects. But

what if your decision appears to be structured as shown in Table 4.2?

TABLE 4.2

Simple Payoff Matrix

Value Measures

Computers Budgeting

Candidate A 1 2

Candidate B 2 1

© Cengage Learning

Now there is no clear choice. Even if you thought computer skills were more important than

budgeting skills, you couldn't choose, because Candidate A might be a little better with

computers, but Candidate B may be much better in budgeting. To decide, you need either more sophisticated measures of ability or a way to weight the two factors, as we do in the

example in Table 4.3.

TABLE 4.3

Weighted Payoff Matrix

Value Measures

Computers Budgeting Combination Candidate A 9 3 6.3 + 0.9 = 7.2

Candidate B 5 8 3.5 + 2.4 = 5.9

Weight 0.7 0.3

© Cengage Learning

Here we measure the ability of each candidate in the two areas on a 10-point scale, with a higher score indicating higher proficiency. Because we have established that computer skills

are more important than budgeting skills, we give those skills a higher weight (0.7 compared

to 0.3). By multiplying the candidates' scores by the weights, we obtain a combined value measure for the two candidates, thus enabling us to choose the better candidate. (This classic

example is adapted from Latane, 1963.)

We could extend the logic of the payoff matrix even further. One way is to combine scores

under differing working conditions. Indeed, following the logic of the payoff matrix, we could accommodate large numbers of weighted variables, as might be involved in a large-scale

policy analysis; the logic remains much the same. Remember that one can adopt different

decision rules and that the choice of criteria is subjective.

Another tool of policy analysis is decision analysis, a technique for use where decisions are

likely to be made sequentially and with some degree of uncertainty. Decision analysis is

applicable to a variety of complex problems, such as choosing airport sites or developing plans

for commercial breeder reactors, but the underlying logic is fairly straightforward and often quite helpful. Consider another classic illustration.

The officer in charge of a U.S. embassy recreation program has decided to replenish the

employees' club funds by arranging a fund-raising dinner. It rains nine days out of ten at the

post, and he must decide whether to hold the dinner indoors or out. An enclosed pavilion is

available but uncomfortable, and past experience has shown turnout to be low at indoor

functions, resulting in a 60 percent chance of gaining $100 from a dinner held in the pavilion and a 40 percent chance of losing $20. On the other hand, an outdoor dinner could be

expected to earn $500 unless it rains, in which case the dinner would lose about $10. (Stokey

& Zechauser, 1978, p. 202)

Using decision analysis to structure the officer's dilemma involves first constructing a decision tree to show the various possible outcomes, given the risks associated with each (see

Figure 4.1). Obviously, the decision tree lays out the options, probabilities of various

occurrences, and anticipated outcomes in much the same way as a payoff matrix. It is easy to

imagine how much more complicated the situation could become, however, with the addition of other variables or other decision options. Even in this simple case, matters might

be complicated by other variables, such as whether the weather will be hot or cold, whether

there are other ways to increase attendance (advertising and so on), and whether the

commanding officer prefers to be indoors or outdoors. You can imagine how much more

serious are the sequences and variables involved in a decision concerning location of a

nuclear facility.

FIGURE 4.1 Embassy Dinner Decision Tree

© Cengage Learning

FIGURE 4.2

Chess Match Decision Tree

© Cengage Learning

And as if this weren't enough, consider what happens when you take into account competition from others. Let's imagine a chess match in which we have decided on some

evaluation criterion, such as king safety or center control that we can measure. That is, we

have identified a way to place a value on each outcome that might result from a given set of moves. Let's say that White is ready to move and has two options, W1 and W2, leading to the

decision tree in Figure 4.2. (If we move W1, then Black can move either B1 or B2; if we move

W2, then Black can move either B3 or B4; and so on. After evaluating all possible outcomes,

we assigned the values shown across the bottom row.) We would obviously prefer to choose W2, and then have Black choose B4, so we could choose W9, the alternative with the highest

value for us. But taking into account what Black is likely to do, we recognize that if we take

W2, then Black will take B3, leading us to the two lowest payoffs. Recognizing this

probability, we will instead take W1, expecting that Black will take B1, and we will have a satisfactory outcome.

Although our examples have been quite simple, their logic can support far more

sophisticated applications of policy analysis. Moreover, the discipline these techniques impose makes them useful for even relatively simple applications. The models force us to

examine our assumptions, structure the problem clearly and logically, and consider the full

range of available options. The models also allow us to more effectively communicate our

analysis to others.

That brings us to one final point. No matter how sophisticated the analysis and how rational

its conclusions, a policy analysis must be effectively communicated to the actual decision

makers. Communication is often quite difficult, because decision makers are extremely busy

and have a variety of conflicting demands on their time and interests. Sometimes even those who are invited to do a policy analysis find themselves and their analysis swept aside by

political or other considerations, and, indeed, that is the prerogative of major decision

makers. A rational analysis is helpful in the decision process, but political considerations, in

the positive sense, must also be taken into account before any actions are taken.

Bardach (2011) suggests that policy analysis is more art than science and emphasizes the

importance of intuition as a complementary part of the methodology used for a policy

analysis. He develops eight useful steps in the policy analysis process that can be used as a practical guide when constructing policy proposals.

11. Define the problem. A clear operational definition of the problem to be solved is a

crucial step that gives “both a reason for doing all the work necessary to complete the

project and a sense of direction for your evidence-gathering activity” (p. 1). 12. Assemble some evidence. Before gathering evidence, it is important for the analyst to

think carefully about what is relevant to the problem. Pertinent literature review,

applied best practices, and analogies are some of the sources for supporting the argument for the existing policy problem.

13. Construct the alternatives. When looking for alternatives, use a comprehensive and

focused approach. The alternatives should address the problem and should be

feasible.

14. Select the criteria. Commonly used evaluative criteria are efficiency, equality, equity,

fairness, and justice. Commonly used practical criteria are legality, political

acceptability, robustness, and improbability.

15. Project the outcomes. When analyzing the policy along with the policy proposal, think about results that will come in the future. For this step, it is most important to be

realistic (not too optimistic, which is a common pitfall) in one's expectations for the

future. 16. Confront the trade-offs. “As economics teaches us, trade-offs occur at the margin” (p.

48). In other words, it tells us how much extra money we should pay to receive an

extra unit of some service.

17. Decide! With this step, the policy analyst makes a decision on which policy alternative to concentrate his or her attention.

18. Tell your story. This step implies that the intention of the analysis is to suggest a

solution for a certain problem. The policy analyst should be aware that the clarity of the presentation of his or her analysis is very important.

Implementation In the cycle of planning, implementation, and evaluation, implementation is the action phase.

Once plans have been made and policies decided on, you must put them into operation. Financial and human resources must be allocated and mobilized, organizational structures

and systems must be devised, and internal policies and procedures must be developed.

During implementation, you may be involved in issuing and enforcing directives, disbursing

funds, awarding grants and contracts, analyzing programmatic and operational problems, taking corrective action, and negotiating with citizens, business, and those in other public

and nonprofit organizations.

Over the past 25 years, a body of literature dealing with the implementation process has

emerged. Some of the literature merely uses new terms to talk about the general processes of administration in the public sector. Other parts of the literature focus on the relationship

between policy development and program implementation. This literature specifically alerts

us to the difficulty of effective program implementation and to how implementation may distort or even subvert the intent of policy makers. Most pointedly, one commentator has

written, “It is hard enough to design public policies and programs that look good on paper.

It is harder still to formulate them in words and slogans that resonate pleasingly in the ears

of political leaders and the constituencies to which they are responsive. And it is excruciatingly hard to implement them in a way that pleases anyone at all, including the

supposed beneficiaries or clients” (Bardach, 1977, p. 3).

A classic study of the relationship between policy and implementation was suggestively

titled Implementation: How Great Expectations in Washington Are Dashed in Oakland; or Why It's Amazing That Federal Programs Work at All (Pressman & Wildavsky, 1973).

Implementation described a particular economic development program in the Oakland,

California, area that was less than successful. Pressman and Wildavsky concluded that “what seemed to be a simple program turned out to be a very complex one, involving many

participants, a host of different perspectives, and a long and tortuous path of decision points

that had to be cleared” (p. 94). Implementation was characterized by multiple and conflicting

interests trying to influence the program's direction to suit their many and divergent needs. The major recommendation of the study seemed to be that people involved in designing

public policies “pay as much attention to the creation of organizational machinery for

executing a program as for launching one” (pp. 144–145).

This lesson has been clearly recognized in the literature of strategic planning. Plans remain sterile without implementation, so there has always been a close connection between

planning and execution. As noted, planning is most beneficial where it can help make

immediate decisions in light of future impact. Thus, a final step in any planning process is to arrive at a series of specific actions to take in the near future—the next six months, or the

next year or two years—who does what, when, and to what effect. These steps, which may

detail new policy positions or new organizational processes, will form a new action agenda for the community or the agency.

Organizational Design

Some of the classic approaches to implementation, formerly called “organization and

management,” focused on the structure and design of new organizations and their work processes or flows. The traditional organization chart expresses both the division of labor

within an organization and the structure of command or control.

In the late 1930s, Luther Gulick advised managers developing new organizations that there

were several ways they could divide work (Gulick, 1937, pp. 21–29). Among these were (1)

purpose, (2) process, (3) people or things, or (4) place. Dividing work according to purpose

might result in distinctions such as between providing education or controlling crime, while

dividing work according to process might lead to a legal unit, a medical unit, or an engineering unit. One could also divide work according to the people served or the things

being dealt with—for example, the Department of Veterans Affairs deals with all problems

that veterans face, whether legal, medical, and so on. Finally, one may organize according to

geographic area, as would a state welfare department that has regional or county offices.

Networking

Online resources and case studies relating to the planning and implementation of

productivity initiatives can be found at the U.S. Conference of Mayors site at

-w-w-w-.-u-s-m-a-y-o-r-s-.-o-r-g-/-u-s-c-m-/-b-e-s-t-_-p-r-a-c-t-i-c-e-s. For information on scenario planning, go to the Global Business Network at -w-w-w-.-g-b-n-.-c-o-m.

Gulick and his contemporaries also talked about the number of levels that would be

appropriate to an organization. Obviously, many organizations are fairly “tall”—they have

many levels; others are “flat”—they have relatively few levels. The number of levels is guided

to a degree by the type of work and by the number of people who report to any one manager.

The term span of control signifies the number of people that one individual supervises;

although there are significant variations, depending on the type of work, it is generally

considered difficult to supervise more than six to ten people.

In addition to developing organizational structures, early writers urged charting work

processes as an aid to organizational design. Process charting or flowcharting can provide a

graphic demonstration of the various steps in an operation, the people performing each step, and the relationships among these elements. Figure 4.3 shows a simple illustration of process

charting, although charts can become far more sophisticated in actual applications. This

process chart uses a variety of symbols to indicate different activities. The vertical lines set

the basic framework of the chart. The columns show the flow of work from one person to another and vary depending on the complexity of the process and the degree of analysis

desired. The column headings indicate the positions under study. In Figure 4.3, the ovals

indicate a specific task (sorting, conferring, and so on); the circles indicate transportation of work from one person to another. The triangle indicates storage, a period during which

operation is stationary. Finally, the rectangle indicates an inspection, usually to check for

quality or quantity. As illustrated here, one can make notations on the chart to indicate the

nature of particular steps in the process.

FIGURE 4.3

Process Charting

© Cengage Learning

Process charting is most useful where a considerable number of clerical or nonprofessional

employees perform the same general classes of work and follow the same general sequence of operating steps. Although process charting is less useful in analyzing the work of

professionals, there are possible applications here as well. For example, charting a

professional operation may reveal bottlenecks, excessive periods of review, or excessive checkpoints that inhibit the flow of work. As with other techniques, process charting can

become quite complex, but its logic is both simple and compelling. Process charting

simplifies analysis because it sharply points out backtracking, excessive detail, unnecessary

repetition, poor distribution of functions, and other administrative defects. For this reason, process charting has enjoyed a revival of interest by those implementing total quality

management programs.

Systems Analysis

There are many other sophisticated devices that have been developed for analyzing the

design and operations of both public and private organizations. Many of the approaches are

based in systems theory, an effort to identify, in logical fashion, the interactions of various

internal and external elements that impinge on an organization's operations. The systems approach has been used in a variety of fields, including physics, biology, economics, sociology,

and information science, but the basic concepts are much the same regardless of discipline.

Generally speaking, a system is a set of regularized interactions configured or bounded in a way

that differentiates and separates them from other actions that constitute the systems environment. Thus, we can speak of a biological system, a physical system, an economic system,

or a political system. Any such system receives inputs from its environment and then translates

these through some sort of conversion process into outputs that are returned to the environment. These outputs in turn affect future inputs to the system through a feedback

loop. Presumably, if the outputs of a system are valued by the environment, new inputs will

be forthcoming and the organization will survive. A basic systems model is illustrated in

Figure 4.4.

FIGURE 4.4

A Basic Systems Model

© Cengage Learning

Following this model, consider the operation of a thermostat. The thermostat takes in

information about the heat in a room and then measures the heat against some standard. If

the level of heat is below the standard, the thermostat causes more heat to be put out into

the room. The additional heat becomes part of the environment and creates new information (feedback), which becomes part of the next input into the system.

The systems concept works similarly in human organizations. A business might receive input

from its environment that customers are demanding more red shoes. A decision might be

made to produce more red shoes, and those shoes would be part of the organization's output. The new red shoes become part of the environment and affect new inputs into the system,

which might range from comments about the quality of the shoes to information that the

demand has not yet been met. This new information guides the operation of the system in the future.

Like many of the other models we have discussed in this chapter, the systems approach has

been used in highly sophisticated applications ranging from the analysis of organizational

design and processes to the creation and modification of major weapons systems. Indeed, the first major applications of systems analysis occurred in the military during and soon after

World War II. For some time thereafter, the Department of Defense was the major user of

systems analysis, depending on a variety of contractors, most notably the RAND Corporation

to help in applying systems analysis to a variety of problems. Yet, it is possible to apply systems logic to a variety of problems public organizations face, and, as with other

techniques, the systematic discipline that the approach brings to problem solving is perhaps

its greatest strength.

Systems analysis emphasizes the relationship between the organization and its

environment, suggesting that public managers carefully consider factors in the environment

that impinge on their operations. These factors include legal and political matters, support

and opposition generated by interest groups and client organizations, human and financial

resources, and applicable technology. Naturally, the environment also includes a large

number of other organizations with which the agency interacts, such as the chief executive's

office, the legislature, the budget office, related agencies at the same level of government,

parallel agencies at other levels of government, and a variety of private and nonprofit groups and associations.

Many systems analysts tend to ignore what occurs within the system itself, preferring to

think of it as a black box into which inputs go and from which outputs come. Others speak of several different subsystems that carry forward the organization's work. In a classic

formulation, the institutional subsystem is responsible for adapting the organization to its

environment and for anticipating and planning for the future. People involved in this activity generally constitute the organization's leadership cadre. The technical subsystem, on the other

hand, is concerned with the effective performance of the organization's actual work. If the work

of the organization is building rockets, the technical subsystem is the people who actually build

the rockets. Finally, the managerial subsystem is concerned with providing the necessary resources for accomplishing the technical task, as well as mediating between the technical and

institutional subsystems (Thompson, 1967, pp. 10–11).

Outputs of public organizations range from goods (such as highways or buildings) to services

(such as student loans or employment counseling), but they also include regulations, adjudication, and support for other programs. To know the effect of their efforts, managers

need some sort of feedback mechanism. Feedback often occurs naturally: clients write letters

of appreciation; legislators inquire about program operations; a program may even become

an issue during an election campaign. Sometimes, however, you will want to secure more

systematic and accurate feedback, for which you can use devices such as questionnaire

surveys, field testing, or spot checks of service provision. Recall that systems analysis helps

focus on how an organization interacts with its environment; developing effective feedback mechanisms helps the manager in that process.

Reengineering

Over the past twenty years, many public administrators have adopted a more

comprehensive, even radical technique for enhancing organizational performance, called

reengineering. (Although the word reengineering has been used with decreasing frequency, the

basic principles of reengineering are reflected in other approaches such as Six Sigma and have

become a part of the core organizational processes in many agencies.) The central tenet of reengineering centers on redesigning work processes and organizational structures to be in

line with agency outcomes. Through this technique, policy makers attempt to make public

organizations more flexible and capable of responding to the dynamic conditions of

contemporary society.

In some respects, reengineering builds upon systems theory and other analytic techniques

in that it involves the recognition of core processes and the systemic context of staff

behavior. However, the outcome of reengineering goes well beyond simply making

alterations within the existing bureaucratic structure. Its goal is to overhaul rigid government agencies into what one author calls seamless organizations: “In contrast to the

fragmented bureaucracies of the past, seamless organizations provide a smooth,

transparent, almost effortless experience for their customers. Staff in seamless organizations perform the full job, in direct contact with their end users” (Linden, 1994, p. xii).

Implementation of a reengineering process begins with an identification of the

organization's desired outcomes. These include the short- and long-term impacts the agency

wants to achieve. Then the organization is redesigned around the core and support

processes that will produce these outcomes. Given the hierarchical, inflexible nature of many public organizations, though, this is not as simple as it may seem.

Reengineering requires that public administrators change their current assumptions, those

equating organization with traditional bureaucracy. Such a reorientation helps to transform

work processes and agency structures to those driven by meaningful outcomes—a shift from segmentation to integration, from division of labor to seamless work (Linden, 1994).

Reengineering involves enhancing those activities that may be considered value-added—

activities that give customers more of what they are willing to pay for and cutting functions

that merely stand in the way. Of course, some functions remain crucial for the organization's success. Central administration activities such as budgeting, accounting, and quality

inspections cannot simply be removed from the picture. On the other hand, these functions

often hinder the completion of the more value-added activities. The key to successful reengineering is to separate the core processes from the other tasks, enabling the critical

activities to be carried out more effectively (Hammer & Champy, 1993; Linden, 1994).

Evaluation The sequence of planning, implementation, and evaluation is completed by asking whether

the program goals and objectives have been achieved in a way that was both efficient and

effective. Such evaluations may, however, operate at a variety of different levels. Some may

respond to the legislature's interest in knowing whether the intended benefits of legislation were achieved; others may be designed to communicate to the public what is happening in

areas of broad citizen interest; still others may be oriented toward improvements either in

the design of the policy being implemented or in the way it was conducted. An understanding

of contemporary approaches to evaluation requires attention to both the performance measurement movement mentioned earlier and more traditional program evaluation

approaches. Whereas program evaluation offers insight into each policy or program's

direction, effectiveness, and sustainability, performance measurement generates information concerning the organization or network as a whole. When combined within the

framework of evaluation research, these strategies not only assist in the decision-making

process but also improve the overall accountability of public organizations. In turn,

evaluation research enhances legislative oversight and administrative control.

Several legislative groups conduct or sponsor evaluation research at the federal level. These

include the Government Accountability Office (GAO), the Congressional Budget Office, and

various legislative committees, primarily those concerned with the budget and oversight of

specific programs. Executive agencies, such as the Office of Management and Budget and the Executive Office of the President, also conduct evaluation research. Much of this research,

however, is sponsored by the agencies themselves, as managers seek to determine how they

can better manage or generate greater productivity from their organizations.

There is also great interest in program evaluation at the state and local levels, although

resources to support such activities have often been limited. State governments have developed analytic capabilities within the executive branch, often through the budget office.

In recent years, many states have restructured their budgets to be in line with

predetermined performance standards (see Chapter 10). This has enabled state

governments to link fiscal resources to the desired results targeted by each agency. Evaluation research is then used to help a state government determine how successful it has

been in achieving its performance goals. Consequently, state legislatures and citizens can

now see what services and impacts were gained with public resources.

Again, at state and local levels and in most nongovernmental organizations, as at the federal level, a great deal of evaluation research is done as part of the agency's or program manager's

ongoing responsibility. Public administrators increasingly must show not only the efficiency

of their actions, but also the results of their actions within the broader stakeholder

community. Examples of performance measurement and program evaluation range from

complex, detailed, one-time studies to the ongoing, integrated monitoring of performance

goals. Regardless of the level of sophistication, evaluation research offers important details

to support the organization's overall strategic planning and to assist the organization in determining the direction of individual programs.

Program Evaluation

There are a variety of ways to classify the approaches to program evaluation, including

outcome evaluations and process evaluations. Outcome evaluations are closely tied to the type

of assessment in performance measurement; they focus on the results of program activity—

that is, the extent to which a program meets its objectives in terms of impact on the

environment (as described in the discussion of the United Way in Chapter 3). If the work of an organization is to improve adult literacy, then an evaluation might measure the number of

individuals who learned to read. That information would likely then be related to program

inputs to show, in a cost-benefit ratio (in this case, based on allocative efficiency), the number

of individuals learning to read per thousand dollars spent. In general, an outcome evaluation seeks to determine whether X causes Y, where X is the activity of the program and Y is the

desired outcome or goal. As you can imagine, outcome evaluations are particularly valuable

to legislators, grant makers, and others concerned with the performance of various

programs.

In contrast to outcome evaluations, process evaluations focus on ways that program

implementation might be improved to better meet the program's objectives. The question here

is what can be done to X, the program's management, to improve Y, the desired outcome. Whereas an evaluator interested in outcomes might spend a great deal of time developing

systematic measures of program results, someone interested in process evaluation would

analyze the organization and management of the agency's activities, including distribution

of financial and human resources and design of service delivery mechanisms. Process

evaluations also determine if legally prescribed processes are being followed and ensure that individual rights are not violated.

Log in to -w-w-w-.-c-e-n-g-a- g-e-.-c-o-m and open CourseReader to access the reading:

Read “Children in Poverty: Can Public Policy Alleviate the Consequences?” by Aletha C. Huston, and “Prison Math: What Are the Costs and Benefits of Leading the World in Locking Up Human Beings?” by Veronique de Rugy. These two articles discuss the kinds of public policy issues that confront those in the public sector and in related organizations. The first article discusses policy options that might be available to help with the issue of children in poverty. The second article discusses the costs and benefits of a high prison population.

Discuss each case, emphasizing in the first the possible policy solutions that

might be developed. What are the alternative policies, and what are the advantages and disadvantages of each? In the second case, consider the “mathematics” of the situation being evaluated here. What policy conclusions would your analysis of the situation suggest? How much are your recommendations guided by the data and how much by your personal values? How does one affect the other?

Relevant measures here would fall more on the input side and might include such items as workload measures or data on resource allocation. In such studies, it may be important to

distinguish between efficiency and effectiveness. Efficiency is concerned with the relationship

between inputs and outputs, usually expressed in a ratio per unit of input. For example, a

measure of streets paved per thousand dollars spent would be a measure of efficiency. Effectiveness, on the other hand, is concerned with the extent to which a program is achieving

or failing to achieve its stated objectives. Effectiveness measures are outcome oriented; they

focus on the real changes the program produces, such as a decrease in airline deaths.

Sometimes process evaluations occur after the fact—that is, upon completion of the program; but often they occur during program operation. Indeed, some process evaluations

are almost continuous in their ongoing review of program operations. In either case, the

information that emerges in the course of a process evaluation is likely to be of greatest interest to the program manager who hopes to improve his or her organization's

performance.

Program evaluations may therefore be directed toward many different audiences and serve

many different purposes. The specific kinds of information required vary from evaluation to evaluation. Eleanor Chelimsky, former head of the GAO's Program Evaluation and

Methodology Division, lists the following types of information that may be developed

retrospectively:

• Information on program implementation (such as the degree to which the program is

operational, how similar it is across sites, whether it conforms to the policies and expectations formulated, how much it costs, how stakeholders feel about it, whether

there are major problems of service delivery or of error, fraud, and abuse, and so on).

• Ongoing information on the current state of the problem or threat addressed by the

program. (Is the problem growing? Is it diminishing? Is it diminishing enough so that the program is no longer needed? Is it changing in terms of its significant

characteristics?)

• Information on program outcomes. (What happened as a result of program

implementation?) • Information on the degree to which the program made, or is making, a difference.

(That is, what change in the problem or threat occurred that can be directly attributed

to the program?)

• Information on the unexpected (as well as expected) effects of the programs. (For

instance, was a program of drug education accompanied by an increase in the use of

drugs?) (Chelimsky, 1985, pp. 8–9)

Evaluation Designs and Techniques

Approaches to the evaluation of public programs range from historical analysis to

sophisticated experimental designs. Indeed, over the years, there has been a recurring

debate over the proper approach to evaluation. Some argue that such research should be primarily qualitative, concerned with tracking program development and indicating forces

that helped shape the program. Advocates of this approach tend to be most interested in

process questions, such as reasons for success or failure and unanticipated consequences of

the program; they ask, “What happened?” Others argue that program evaluations should, wherever possible, employ the most rigorous scientific methods appropriate to the subject

matter, including the design and execution of formal experiments. These analysts tend to be

more interested in program outcomes; they ask, “Does it work?” (Chelimsky, 1985, p. 14).

Whatever the approach, those involved in program evaluation must confront two challenges to the validity of their work. The first question, concerning internal validity, asks whether the

approach measured what was intended. Was the design consistent with the goals of the

program and the needs of the sponsor? Were the methods most appropriate for answering

the questions that needed to be asked? Were the results as free from bias as possible? A second question, concerning external validity, asks to what extent the findings may be

applicable to more general circumstances. What does the study say about similarly situated

programs? Can the study be replicated and be expected to produce similar results? These and other questions can be directed toward the various techniques employed in evaluation

research.

Qualitative Techniques Many program evaluations depend on qualitative information

derived from reading about the program, from interviewing important actors (including agency personnel, clients, and others), and sometimes from actually

participating in the work of the program. The initial step in a qualitative evaluation

project is usually to read everything available about the program, including

background material on the subject of the program (flu vaccines, child nutrition, rapid transit systems, and so on), agency documents, operating procedures, internal

memoranda, newspaper and magazine articles, articles on similar programs

elsewhere, and reports issued by various concerned groups. The researcher would

also likely make a few phone calls to identify the significant actors in the program and determine where the most important activities are taking place.

Following an initial reconnaissance, the analyst settles on a limited number of sites (schools,

hospitals, highway systems, and so on) as the focus of the investigation. Most qualitative

evaluations are largely exploratory, designed to explore a variety of hunches or intuitions

about the program's operation. For these cases, the analyst will probably try to select sites

that vary widely along several crucial dimensions. Some evaluations, however, are hypothesis

guided, designed to demonstrate the plausibility of a particular hypothesis, so the analyst might choose a limited number of crucial sites that are especially illustrative of the issue

under investigation.

What Would You Do?

You have been asked to assess the effectiveness of a new program aimed at reducing teen

pregnancies by providing birth control information through the high schools. You have heard several suggestions about how to design the study—and some comments that it's simply

impossible to measure something like this. What would you do?? What Would You Do?

Once the research sites have been chosen, the analyst may choose to gather most of his or her information through intensive interviews, detailed information-gathering sessions

involving major actors both inside and outside the agency responsible for the program.

Interviewing skills include establishing the interviewer's credentials, setting the proper

climate, arranging questions effectively, asking reasonable but challenging questions, and keeping a good record of all that is said. Perhaps most important, the interviewer must keep

the discussion on the subject, in a way that is neither obvious nor embarrassing to either

party. Immediately following the interview, the interviewer should review and expand upon

the notes taken during the interview session. These notes will form an important basis for

drawing conclusions about the program.

An alternative means of gathering qualitative information is the use of a participant-observer,

someone involved in either the target population or the agency itself who makes observations and draws conclusions based on firsthand data. For example, an evaluation of an antipoverty

program in eastern Kentucky some years ago employed a participant-observer who lived in the

community, talked daily with others in the community about the program, and reported back

to the overall evaluation staff.

Either technique can be questioned with respect to both internal and external validity.

Biased information and questions about internal validity can arise if the wrong people are chosen to interview or if those interviewed provide misleading information, intentionally or

unintentionally. Participant-observers can affect the program's operation through their own

presence, leading to outcomes far different from what would otherwise have happened.

Questions concerning external validity (or generalizability) might be raised with either technique based on the choice of only a limited number of sites for investigation.

Quantitative Techniques Policy evaluations often endeavor to approximate the

scientific methods of the physical sciences, although such efforts are extremely

difficult. In its classic formulation, an experimental design involves examination of two or more groups under carefully controlled conditions. One group, the experimental group,

receives a treatment or intervention; in the case of program evaluation, members of the

experimental group receive the benefits of the program being evaluated. Another group, the

control group, consists of individuals who are as similar as possible to those in the

experimental group and who act under the same general conditions, yet do not receive the

intervention. Members of both groups are tested before and after the experimental

intervention (pretest and posttest measures), and the results are compared. If the program has had either a positive or negative effect, the differences should show up in the data. We

can illustrate the difficulties in designing a rigorous experimental design with respect to

social programs by imagining that we are interested in analyzing the effectiveness of a new

approach to mathematics education in the fourth grade. One classroom might be designated an experimental group and be taught using the new approach; another classroom might be

designated the control group and be taught using traditional methods. The mathematical

abilities of all students would be measured both before and after the period in which the new program was being taught. If the new technique is indeed more effective in educating

children in mathematics, the posttest scores of the children in the experimental group should

be higher than those of the children in the control group.

In a very general sense, this is an application of an experimental design to a social program,

and you can easily imagine how similar designs might be used to measure other programs,

ranging from immunizations to welfare incentives to highway designs. But we can observe

difficulties in such designs, some of which relate to questions of internal validity. One might

respond to the study by saying that students in the experimental group were smarter to

begin with, or that the absence rate was higher among those in the control group. Or you

might suggest that one teacher was better than the other, and that made the difference. Or

even if the same teacher taught both groups, you might speculate that he or she taught the new material with more enthusiasm. Similar questions might be raised about external

validity. For instance, if the results were obtained in a rural school, would they apply as well

to an urban setting?

Some, if not all, of these questions could be anticipated by slightly altering the research design. For example, students could be randomly assigned to the two groups, thus

eliminating any possibility of bias in the groups' composition. But questions such as these

show the difficulty of achieving true experimental conditions in measuring social programs.

For this reason, most evaluations of social programs are called quasi-experimental.

Quasi-experimental designs retain the requirement for systematic data gathering that should

be part of any quantitative approach, but they free the researcher from some of the

difficulties of developing experimental designs, such as the need for random assignment of

subjects to various groups. Here again, different groups may be compared, but an essential task for the researcher is to separate the effects of a treatment from the effects of other

factors. Only the effects caused by the treatment are of interest.

Quasi-experimental approaches not only are more adaptable to social situations, but they

also better fit the situation in which program evaluators often find themselves—assigned to the evaluation long after the program has begun and having little influence on patterns of

intervention. In such a case, a historical approach may be of special value. For example, one

quasi-experimental design—time series analysis—involves making a number of observations about the target population both before and after the program intervention. (These

observations may even be made retrospectively by gathering historical data.) In one case, basic

information about neighborhood crime was evaluated for a period of years prior to the

introduction of a new patrol pattern; then similar data evaluation followed after the new approach was introduced.

Summary and Action Implications As a public manager, you will become quite familiar with the cycle of planning,

implementation, and evaluation. In practice, the phases of the cycle will rarely appear as distinct as in our discussion, but you will still find that you must devote a portion of your

time to each phase. In middle and upper management, the planning, implementation, and

evaluation cycle will become especially complex because you will find yourself engaged in

all three phases almost simultaneously. That is, you will be planning for one project at the same time that you are implementing a second and evaluating a third, and so on. Obviously,

maintaining a good sense of the timing of the various projects and knowing when and how

to shift from one to the next will be extremely important.

As we have seen, techniques have been developed to help you work through the typical

problems you will encounter in each phase of the cycle. Although many of the techniques can

be elaborated in highly complex ways, the logic on which they are based can be helpful in

dealing even with fairly simple and immediate problems.

Throughout the planning, implementation, and evaluation cycle, you should remember that,

whereas we have focused on technical aids to your administrative work, each of the three

areas will be strongly affected by how you interact with the people in your organization (and

elsewhere). Planning, implementation, and evaluation are human processes and are thus subject to people's shifting values, attitudes, and behaviors. In planning, implementation,

and evaluation, as with budgeting, financial management, and personnel, techniques are

successful only when you use them with full regard for democratic values, clear leadership, and humane management.

STUDY QUESTIONS 19. Planning is one aspect of the policy process. Discuss the various types of planning and

their objectives.

20. In organizing a planning process, what are the primary concerns of the planning

group? 21. Discuss the necessary steps for comprehensive policy analysis.

22. Identify some of the quantitative techniques used for policy analysis.

23. The second phase of the policy process is implementation of plans. Discuss some of

the techniques available to help in the beginning stages of the implementation process.

24. Compare and contrast the several different subsystems that carry forward an

organization's work.

25. What does the phrase “managing for results” mean? How might such a program be

implemented?

26. What are the different types of evaluation approaches? Discuss the distinctions

among them.