PhD doctorate only

dtrischrmen

chapter_1.pdf

Home >Social Science homework help >PhD doctorate only

I PART

Methodology of Policy Analysis

Problem Structuring

Forecasting

Monitoring Prescription

POLICY PROBLEMS

POLICY ACTIONS

Evaluation

EXPECTED OUTCOMES

OBSERVED OUTCOMES

Practical Inference

POLICY PERFORMANCE

IS B

N 1

-2 56

-9 77

08 -X

O B J E C T I V E S

By studying this chapter, you should be able to

The Process of Policy Analysis

1 CHAPTER

� Define and illustrate phases of policy analysis.

� Describe elements of integrated policy analysis.

� Distinguish four strategies of policy analysis.

� Contrast reconstructed logic and logic-in-use.

� Distinguish prospective and retrospective policy analysis.

� Describe the structure of a policy argument and its elements.

� Understand the role of argument mapping in critical thinking.

� Interpret scorecards, spreadsheets, influence diagrams, decision trees, and argument maps.

Policy analysis is a process of multidisciplinary inquiry aiming at thecreation, critical assessment, and communication of policy-relevant infor-mation. As a problem-solving discipline, it draws on social science methods, theories, and substantive findings to solve practical problems.1

1For a sample of alternative definitions see Harold D. Lasswell, A Pre-view of Policy Sciences (New York: American Elsevier Publishing, 1971); Yehezkel Dror, Ventures in Policy Sciences: Concepts and Applications (New York: American Elsevier Publishing, 1971); Edward S. Quade, Analysis for Public Decisions, 3d rev. ed., ed. Grace M. Carter (New York: North Holland Publishing, 1989); David L. Weimer and Aidan R. Vining, Policy Analysis: Concepts and Practice, 2d ed. (Englewood Cliffs, NJ: Prentice Hall, Inc., 1992); Duncan Mac Rae Jr., The Social Function of Social Science (New Haven, CT: Yale University Press, 1976).

IS B

N 1-256-97708-X

Methodology of Policy Analysis 3

METHODOLOGY OF POLICY ANALYSIS As used here, the word methodology refers to a process of reasoned inquiry aimed at finding solutions to practical problems. The aim of methodology is to help us understand not only the products of policy inquiry but also the processes employed to create these products.2 The methodology of policy analysis is not confined to the analytical routines of specialized social science fields—for example, benefit-cost analysis in economics or implementation analysis in political science—because none of these holds a privileged place in policy inquiry. Nor is the methodology of policy analysis constrained by the doctrines and principles of obsolescent philosophies of science such as logical positivism, which mistakenly claimed that scientific knowledge, properly understood, is objective, value free, and quantitative.3 On the contrary, policy analysis is methodologically eclectic; its practitioners are free to choose among a wide range of scientific methods, qualitative as well as quantitative, as long as these yield reliable knowledge. In this context, policy analysis includes art, craft, and reasoned persuasion, all of which are scientific to the extent that they succeed in producing reliable knowledge.4 Ordinary commonsense knowing and well-winnowed practical wisdom—both products of evolutionary learning across generations of

2Abraham Kaplan, The Conduct of Inquiry: Methodology for Behavioral Science (San Francisco, CA: Chandler Publishing Company, 1964), pp. 23–24. 3Logical positivism (or logical empiricism) was abandoned by most philosophers of science more than 50 years ago, although its epistemological pillars—the correspondence theory of truth, the empirical criterion of meaning, and quantificationism—are still venerated by many social scientists. For alterna- tives to logical positivism in economics and political science see Daniel Bromley, Sufficient Reason: Volitional Pragmatism and the Meaning of Economic Institutions (Princeton, NJ: Princeton University Press, 2006); Henry E. Brady and David Collier, eds. Rethinking Social Inquiry: Diverse Tools, Shared Standards (Lanham, MD: Rowman Littlefield, 2004); Deirdre N. McCloskey, The Rhetoric of Economics, 2nd ed., Madison: University of Wisconsin Press, 1998; Stephen Thomas Ziliak and Deirdre N. McCloskey, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor: University of Michigan Press, 2008. Paul Diesing, How Does Social Science Work? Reflections on Practice (Pittsburgh, PA: University of Pittsburgh Press, 1991); and Mary Hawkesworth, Theoretical Issues in Policy Analysis (Albany: State University of New York Press, 1988). 4Larry Laudan has argued that the demarcation between science and non-science, including art and craft, is a pseudo-problem that should be replaced by focusing on the distinction between reliable and unreliable knowledge. It is not necessary to ask whether knowledge is “scientific,” only whether it is reliable. “The Demise of the Demarcation Problem,” in R.S. Cohen and L. Laudan, Physics, Philosophy and Psychoanalysis: Essays in Honor of Adolf Grünbaum. Boston Studies in the Philosophy of Science, Vol.76 (Dordrecht: D. Reidel, 1983), pp. 111–127. Aaron Wildavsky and others have used the terms art and craft to characterize policy analysis. See Aaron Wildavsky, Speaking Truth to Power: The Art and Craft of Policy Analysis (Boston, MA: Little Brown, 1979); and Iris Geva-May and Aaron Wildavsky, An Operational Approach to Policy Analysis: The Craft, Prescriptions for Better Analysis (Boston, MA: Kluwer, 1997). The term policy science(s) is Harold Lasswell’s. See the short methodological history of the policy sciences in Ronald Brunner, “The Policy Movement as a Policy Problem,” in Advances in Policy Studies since 1950, vol. 10, Policy Studies Review Annual, ed. W. N. Dunn and R. M. Kelly (New Brunswick, NJ: Transaction Books, 1992), pp. 155–97 and contributions to Michael Moran, Martin Rein, and Robert E. Goodin, eds. The Oxford Handbook of Public Policy (Oxford: Oxford University Press, 2006).IS

B N

1 -2

56 -9

77 08

-X

4 CHAPTER 1 The Process of Policy Analysis

problem solvers—often permit conclusions that are more trustworthy and reliable than those produced by means of policy analysis and other specialized forms of professional and scientific inquiry.5

The rationale for policy analysis is pragmatic. For this reason, it is unmistakably different from social science disciplines that prize knowledge for its own sake. The policy-relevance of these disciplines depends not on their status as sciences but on the extent to which they are successful in illuminating and alleviating practical problems, problems that come in complex bundles that are at once economic, political, cultural, ethical, and more. Practical problems do not arrive in separate disciplinary packages addressed to departments of economics and political science—to name two of the most important policy disciplines. In today’s world, multidisciplinary policy analysis seems to provide the best fit with the manifold complexity of public policy making.

POLICY ANALYSIS—A MULTIDISCIPLINARY FRAMEWORK Policy analysis is partly descriptive. It relies on traditional social science disciplines to describe and explain the causes and consequences of policies. But it is also normative, a term that refers to value judgments about what ought to be, in con- trast to descriptive statements about what is.6 To investigate problems of efficiency and fairness, policy analysis draws on normative economics and decision analysis as well as ethics and other branches of social and political philosophy—all of which are about what ought to be. This normative orientation stems from the fact that analyzing policies demands that we choose among desired consequences (ends) and preferred courses of action (means). The choice of ends and means requires contin- uing trade-offs among competing values of efficiency, equity, security, liberty, and democracy.7 The importance of normative reasoning in policy analysis was well stated by a former undersecretary in the Department of Housing and Urban Development: “Our problem is not to do what is right. Our problem is to know what is right.”8

5On the contrasts between scientific and professional knowledge on one hand, and ordinary commonsense knowing on the other, see Charles E. Lindblom and David K. Cohen, Usable Knowledge: Social Science and Social Problem Solving (New Haven, CT: Yale University Press, 1979). On the frequent soundness of evolved practical knowledge—but the periodic need for supplemental scientific testing—see Donald T. Campbell, “Evolutionary Epistemology,” in Methodology and Epistemology for Social Science: Selected Papers, ed. E. S. Overman (Chicago: University of Chicago Press, 1989). 6One classic statement of the difference between positive and normative knowledge in economics is Milton Friedman, Essays in Positive Economics (Chicago, IL: University of Chicago Press, 1953). This same positive-normative distinction is present throughout the social sciences. 7Deborah Stone, Policy Paradox: The Art of Political Decision Making, rev ed. (New York: W. W. Norton, 2001). 8Robert C. Wood, “Foreword” to The Study of Policy Formation, ed. Raymond A. Bauer and Kenneth J. Gergen (New York: Free Press, 1968), p. v. Wood is quoting President Lyndon Johnson.

IS B

N 1-256-97708-X

Policy Analysis—A Multidisciplinary Framework 5

Policy-Relevant Information Policy analysis is designed to provide policy-relevant information about five types of questions:

� Policy problems. What is the problem for which a potential solution is sought? Is global warming a human-made consequence of aircraft and motor vehicle emissions? Or is global warming a consequence of periodic fluctuations in the temperature of the atmosphere? What alternatives are available to mitigate global warming? What are the potential outcomes of these alternatives and what is their value or utility?

� Expected policy outcomes. What are the expected outcomes of policies designed to reduce harmful emissions? Because periodic natural fluctuations are difficult or impossible to control, what is the likelihood that emissions can be reduced by raising the price of gasoline and diesel fuel, compared with requiring that aircraft and motor vehicles use biofuels?

� Preferred policies. Which policies should be chosen, considering not only their expected outcomes in reducing harmful emissions, but also the value of reduced emissions in terms of economic costs and benefits? Should distribu- tional criteria involving environmental justice be used along with criteria of economic efficiency?

� Observed policy outcomes. What policy outcomes are observed, as distin- guished from the outcomes expected before a preferred policy is implemented? Did the preferred policy actually result in reduced emissions? Were other factors such as political opposition to governmental regulation responsible for the limited achievement of emissions targets?

� Policy performance. To what extent do observed policy outcomes con- tribute to the reduction of global warming through emissions controls? What are the benefits and costs of government regulation to present and future generations?

Answers to these questions yield five types of information, which are policy- informational components. These components are shown as rectangles in Figure 1.1.9

A policy problem is an unrealized need, value, or opportunity for improvement attainable through public action.10 Knowledge of what problem to solve requires information about a problem’s antecedent conditions (e.g., school dropouts as an antecedent condition of unemployment), as well as information about values (e.g., safe schools or a living wage) whose achievement may lead to the problem’s solution. Information about policy problems plays a critical role in policy analysis,

9The framework was originally suggested by Walter Wallace, The Logic of Science in Sociology (Chicago: Aldine Books, 1971). Wallace’s framework addresses research methodology in sociology, whereas Figure 1.1 addresses the methodology of policy analysis. 10Compare James A. Anderson, Public Policymaking: An Introduction, 7th ed. (Boston, MA: Wadsworth, 2011); Charles O. Jones, An Introduction to the Study of Public Policy, 2d ed. (North Scituate, MA: Duxbury Press, 1977), p. 15; and David Dery, Problem Definition in Policy Analysis (Lawrence: University of Kansas Press, 1984).

IS B

N 1

-2 56

-9 77

08 -X

6 CHAPTER 1 The Process of Policy Analysis

FIGURE 1.1 The process of integrated analysis

because the way a problem is defined shapes the search for available solutions. Inadequate or faulty information may result in a fatal error: defining the wrong problem.11

Expected policy outcomes are likely consequences of one or more policy alterna- tives designed to solve a problem. Information about the circumstances that gave rise to a problem is essential for producing information about expected policy outcomes. Such information is often insufficient, however, because the past does not repeat itself completely, and the values that shape behavior may change in the future. For this reason, information about expected policy outcomes is not “given” by the existing situation. To produce such information may require creativity, insight, and the use of tacit knowledge.12

11Defining the wrong problem is a type III error, as contrasted with type I and type II errors committed when the level of statistical significance (alpha) is set too high or too low in testing the null hypothesis. An early statement of this contrast is Ian I. Mitroff and Thomas R. Featheringham, “On Systematic Problem Solving and the Error of the Third Kind,” Behavioral Sciences 19, no. 6 (1974): 383–93. 12Dror, Ventures in Policy Sciences; Sir Geoffrey Vickers, The Art of Judgment: A Study of Policy Making (New York: Basic Books, 1965); and C. West Churchman, The Design of Inquiring Systems; Basic Concepts of Systems and Organization (New York: Basic Books, 1971).

Problem Structuring

Forecasting

Monitoring Prescription

POLICY PROBLEMS

PREFERRED POLICIES

Evaluation

EXPECTED OUTCOMES

OBSERVED OUTCOMES

Practical Inference

POLICY PERFORMANCE

IS B

N 1-256-97708-X

Policy Analysis—A Multidisciplinary Framework 7

A preferred policy is a potential solution to a problem. To select a preferred policy, it is necessary to have information about expected policy outcomes as well as informa- tion about the value or utility of these expected outcomes. Another way to say this is that factual as well as value premises are required for policy prescriptions. Fact alone— for example, the fact that one policy produces more of some quantity than another—do not justify the choice of a preferred policy. Factual premises must be joined with value premises involving efficiency, equality, security, democracy, or some other value.

An observed policy outcome is a present or past consequence of implementing a preferred policy. It is sometimes unclear whether an outcome is actually an effect of a policy, because some effects are not policy outcomes; many outcomes are the result of other, extra-policy factors. It is important to recognize that the consequences of action cannot be fully stated or known in advance, which means that many conse- quences are neither anticipated nor intended. Fortunately, information about such consequences can be produced ex post (after policies have been implemented), not only ex ante (before policies are implemented).

Policy performance is the degree to which an observed policy outcome con- tributes to the solution of a problem. In practice, policy performance is never perfect. Problems are rarely “solved”; most often, problems are resolved, reformulated, and even “unsolved.”13 To know whether a problem has been solved, resolved, reformu- lated, or unsolved requires information about observed policy outcomes, as well as information about the extent to which these outcomes contribute to the opportunities for improvement that gave rise to a problem.

Policy-Informational Transformations The five types of policy-relevant information are interdependent. The arrows connect- ing each pair of components represent policy-informational transformations, whereby one type of information is changed into another, so that the creation of information at any point depends on information produced in an adjacent phase. Information about policy performance, for example, depends on the transformation of prior information about observed policy outcomes. The reason for this dependence is that any assess- ment of how well a policy achieves its objectives assumes that we already have reliable information about the outcomes of that policy. The other types of policy-relevant information are dependent in the same way.

Information about policy problems is a special case. Information about policy problems usually includes some problem elements—for example, potential solutions or expected outcomes—and excludes others. What is included or excluded affects which policies are eventually prescribed, which values are appropriate as criteria of policy performance, and which potentially predictable outcomes warrant or do not warrant attention. At the risk of being overly repetitious, it is worth stressing again that a fatal error of policy analysis is a type III error—defining the wrong problem.14

13Russell L. Ackoff, “Beyond Problem Solving,” General Systems 19 (1974): 237–39. 14Type I and type II errors are also known as false positives and false negatives. Other sources on type III errors include A. W. Kimball, “Errors of the Third Kind in Statistical Consulting,” Journal of the American Statistical Association 52 (1957): 133–42; Howard Raiffa, Decision Analysis (Reading, MA: Addison- Wesley, 1968), p. 264; and Ian I. Mitroff, The Subjective Side of Science (New York: Elsevier, 1974).

IS B

N 1

-2 56

-9 77

08 -X

8 CHAPTER 1 The Process of Policy Analysis

Policy-Analytic Methods The five types of policy-relevant information are produced and transformed by using policy-analytic methods. All methods involve judgments of different kinds:15 judgments to accept or reject an explanation, to affirm or dispute the rightness of an action, to prescribe or not prescribe a policy, to accept or reject a prediction, and to formulate a problem in one way rather than another.

In policy analysis, these procedures have special names:

� Problem structuring. Problem-structuring methods are employed to produce information about which problem to solve. One example of problem-structuring methods is the influence diagram and decision tree presented in Case 1.3 of this chapter (The Influence Diagram and Decision Tree—Structuring Problems of Energy Policy and International Security). Other examples of problem-structuring methods include critical thinking tools such as argument mapping (Case 1.4: The Argument Map—Problem Structuring in National Defense and Energy Policy). Chapter 3 of this book covers problem-structuring methods and their application.

� Forecasting. Forecasting methods are used to produce information about expected policy outcomes. Although many kinds of forecasting methods are covered in Chapter 4, an example of a simple forecasting tool is the score- card described in Case 1.1 (The Goeller Scorecard—Monitoring and Forecasting Technological Impacts). Scorecards, which are based on the judgments of experts, are particularly useful in identifying expected outcomes of science and technology policies.

� Prescription. Methods of prescription are employed to create information about preferred policies. An example of a prescriptive method is the spread- sheet (Case 1.2: The Spreadsheet—Evaluating the Benefits and Costs of Energy Policies). The spreadsheet goes beyond the identification of expected policy outcomes by expressing consequences in terms of monetary benefits and costs. Benefit-cost analysis and other methods of prescription are presented in Chapter 5.

� Monitoring. Methods of monitoring are employed to produce information about observed policy outcomes. The scorecard (Case 1.1) is a simple method for monitoring observed policy outcomes as well as for forecasting expected policy outcomes. Chapter 6 covers methods of monitoring in detail.

� Evaluation. Evaluation methods are used to produce information about the value or utility of observed policy outcomes and their contributions to policy performance. Although evaluation methods are covered more fully in Chapter 7, the spreadsheet (Case 1.2) may be used for evaluation as well as prescription.

The first method, problem structuring, is about the other methods. For this reason, it is a metamethod (method of methods). In the course of structuring a problem, analysts typically experience a “troubled, perplexed, trying situation, where the difficulty is, as

15John O’Shaughnessy, Inquiry and Decision (London: George Allen & Unwin, 1972).

IS B

N 1-256-97708-X

Policy Analysis—A Multidisciplinary Framework 9

it were, spread throughout the entire situation, infecting it as a whole.”16 Problem situations are not problems; problems are representations of problem situations. Hence, problems are not “out there” in the world, but they stem from the interaction of thought and external environments. Imagine a graph showing the growth of defense expenditures as a percentage of gross domestic product. The graph represents a problem situation, not a problem, because one analyst will see the graph as evidence of increasing national security (more of the budget is allocated to defense), while another interprets the graph as an indication of a declining budget for social welfare (less of the budget can be allocated to social services). Problem structuring, a proce- dure for testing different representations of a problem situation, is the central guidance system of policy analysis.

Policy-analytic methods are interdependent. It is not possible to use one method without first having used others. Thus, although it is possible to monitor past policies without forecasting their future consequences, it is usually not possi- ble to forecast policies without first monitoring them.17 Similarly, analysts can monitor policy outcomes without evaluating them, but it is not possible to evaluate an outcome without first establishing that it is an outcome in the first place. Finally, to select a preferred policy requires that analysts have already monitored, evaluated, and forecasted outcomes.18 This is yet one more way of saying that policy prescription is based on factual as well as value premises.

Figure 1.1 supplied a framework for integrating methods from different policy- relevant disciplines. Some methods are used solely or primarily in some disciplines, and not others. Program evaluation, for example, employs monitoring to investigate whether a policy is causally relevant to an observed policy outcome. Although program evaluation has made extensive use of interrupted time-series analysis, regression discontinuity analysis, causal modeling, and other techniques associated with the design and analysis of field experiments,19 implementation research within political science has not. Instead, implementation researchers have relied mainly on techniques of case study analysis.20 Another example comes from forecasting. Although forecasting is central to both economics and systems analysis, economics has drawn almost exclusively on econometric techniques. Systems analysis has made greater use of qualitative forecasting techniques for synthesizing expert judgment, for example, the Delphi technique.21

16John Dewey, How We Think (Boston, MA: D.C. Heath and Company, 1933), p. 108. The original statement of the difference between a problem and a problem situation is attributable to philosophical pragmatists including Charles Sanders Peirce. 17An exception is predictions made on the basis of expert judgment. The explanation of a policy is not necessary for predicting its future consequences. Strictly speaking, a prediction is a causal inference, whereas a projection, extrapolation, or “rational forecast” is not. 18Causation may be assumed but not understood. Recipes claim only that a desired result is a conse- quence of action. Joseph L. Bower, “Descriptive Decision Theory from the ‘Administrative’ Viewpoint,” in The Study of Policy Formation, ed. Bauer and Gergen, p. 10. 19See, for example, William R. Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Boston, MA: Houghton Mifflin, 2002). 20Paul A. Sabatier and Hank C. Jenkins-Smith, “The Advocacy Coalition Framework: An Assessment,” in Theories of the Policy Process, ed. P. A. Sabatier (Boulder, CO: Westview Press, 1999), pp. 117–66. 21See Chapter 5.

IS B

N 1

-2 56

-9 77

08 -X

10 CHAPTER 1 The Process of Policy Analysis

FIGURE 1.2 Forms strategies of policy analysis

FOUR STRATEGIES OF ANALYSIS Relationships among policy-informational components, policy-analytic methods, and policy-informational transformations provide a basis for contrasting four strategies of policy analysis (Figure 1.2).

Prospective and Retrospective Analysis Prospective policy analysis involves the production and transformation of informa- tion before policy actions are taken. This strategy of ex ante analysis, shown as the right half of Figure 1.2, typifies the operating styles of economists, systems analysts, operations researchers, and decision analysts.

The prospective strategy is what Williams means by policy analysis.22 Policy analysis is “a means of synthesizing information to draw from it policy alternatives and preferences stated in comparable, predicted quantitative and qualitative terms as a basis or guide for policy decisions; conceptually, it does not include the gathering of information [emphasis in original].” Policy research, by contrast,

22Walter Williams, Social Policy Research and Analysis: The Experience in the Federal Social Agencies (New York: American Elsevier, 1971), p. 8.

Problem Structuring

Forecasting

Monitoring Prescription

EXPECTED OUTCOMES

OBSERVED OUTCOMES

POLICY PERFORMANCE

RETROSPECTIVE (ex post): What happened and what difference does it make?

PROSPECTIVE (ex ante): What will happen and what should be done?

PROBLEM FINDING: What problem should be solved?

PROBLEM SOLVING: What is the solution to the problem?

POLICY PROBLEMS

PREFERRED POLICIES

Evaluation

Practical Inference

IS B

N 1-256-97708-X

Four Strategies of Analysis 11

refers to “all studies using scientific methodologies to describe phenomena and/or determine relationships among them.” Prospective analysis often creates wide gaps between preferred solutions and actual efforts to implement them. Perhaps no more than 10 percent of the work actually required to achieve a desired set of policy outcomes is carried out before policies are implemented: “It is not that we have too many good analytic solutions to problems. It is, rather, that we have more good solutions than we have appropriate actions.”23

Retrospective policy analysis is displayed as the left half of Figure 1.2. This strategy of ex post analysis involves the production and transformation of informa- tion after policies have been implemented. Retrospective analysis characterizes the operating styles of three groups of analysts:

� Discipline-oriented analysts. This group, composed mainly of political scientists, economists, and sociologists, seeks to develop and test discipline- based theories that describe the causes and consequences of policies. This group is not concerned with the identification of specific policy goals or with distinctions between “policy” variables that are subject to policy manipula- tion and those that are not.24 For example, the analysis of the effects of party competition on government expenditures provides no information about specific policy goals; nor is party competition a variable that policy makers can manipulate to change public expenditures.

� Problem-oriented analysts. This group, again composed mainly of political scientists, economists, and sociologists, seeks to describe the causes and conse- quences of policies. Problem-oriented analysts, however, are less concerned with the development and testing of theories believed to be important in social science disciplines than with identifying variables that may explain a problem. Problem- oriented analysts are not overly concerned with specific goals and objectives, primarily because the practical problems they analyze are usually general in nature. For example, the analysis of aggregate data on the effects of gender, ethnicity, and social inequality on national achievement test scores provides information that helps explain a problem (e.g., inadequate test performance) but does not provide information about policy variables that can be manipulated.

� Applications-oriented analysts. A third group includes applied economists, applied sociologists, applied psychologists, and applied anthropologists, as well as analysts from professions such as public administration, social work, and evaluation research. This group also seeks to describe the causes and consequences of public policies and programs and is not concerned with the development and testing of discipline-based theories. This group is concerned not only with manipulable policy variables but also with the identification of specific policy goals and objectives. Information about specific goals and objectives provides a basis for monitoring and evaluating outcomes and

23Graham T. Allison, Essence of Decision: Explaining the Cuban Missile Crisis (Boston, MA: Little, Brown, 1971), pp. 267–68. 24James S. Coleman, “Problems of Conceptualization and Measurement in Studying Policy Impacts,” in Public Policy Evaluation, ed. Kenneth M. Dolbeare (Beverly Hills and London: Sage Publications, 1975), p. 25.IS

B N

1 -2

56 -9

77 08

-X

12 CHAPTER 1 The Process of Policy Analysis

impacts of policies. For example, applications-oriented analysts may address early childhood reading readiness programs that can be manipulated in order to achieve higher scores on reading tests.

The operating styles of the three groups reflect their characteristic strengths and limitations. Discipline-oriented as well as problem-oriented analysts seldom produce information that is directly useful to policy makers. Even when problem- oriented analysts investigate important problems such as educational opportunity, energy conservation, crime control, or national security, the resultant information is often macronegative. Macronegative information describes the basic (or “root”) causes and consequences of policies, usually by employing aggregate data to show why policies do not work. By contrast, micropositive information shows what policies and programs do work under specified conditions.25 It is of little practical value to policy makers to know that the crime rate is higher in urban than rural areas, but it is practically important to know that a specific form of gun control reduces the commission of serious crimes or that intensive police patrolling is a deterrent.

Even when applications-oriented analysts provide micropositive information, they may find it difficult to communicate with practitioners of ex ante policy analysis, who in most cases are professional economists. In agency settings, ex ante analysts, whose job it is to find optimally efficient solutions, often have limited access to information about policy outcomes produced through retrospec- tive analysis. For their part, practitioners of ex ante analysis often fail to specify in sufficient detail the kinds of policy-relevant information that will be most useful for monitoring, evaluating, and implementing their recommendations. Often, the intended outcomes of a policy are so vague that “almost any evalua- tion of it may be regarded as irrelevant because it missed the ‘problem’ toward which the policy was directed.”26 Legislators, for example, usually formulate problems in general terms in order to gain acceptance, forestall opposition, or maintain neutrality.

Contrasts among the operating styles of policy analysts suggest that disci- pline-oriented and problem-oriented analysis are inherently less useful than applications-oriented analysis—that retrospective (ex post) analysis as a whole is perhaps less effective in solving problems than prospective (ex ante) analysis. Although this conclusion may have merit from the point of view of policy makers who want advice on what actions to take, it overlooks several important benefits of retrospective analysis. Retrospective analysis, whatever its shortcom- ings, places primary emphasis on the results of action and is not content with information about expected policy outcomes, as is the case with prospective analysis. Discipline-oriented and problem-oriented analysis may offer new frameworks for understanding policy-making processes, challenging conven- tional formulations of problems, questioning social and economic myths, and shaping the climate of opinion in a community or society. Retrospective analysis,

25Williams, Social Policy Research and Analysis, p. 8. 26Ibid. p. 13; and Alice Rivlin, Systematic Thinking for Social Action (Washington, DC: Brookings, 1971).

IS B

N 1-256-97708-X

Four Strategies of Analysis 13

however, “has been most important in its impact on intellectual priorities and understandings, and not nearly so effective in offering solutions for specific political problems.”27

Descriptive and Normative Analysis Figure 1.2 also captures another important contrast, the distinction between descriptive and normative strategies of policy analysis. Descriptive policy analysis parallels descriptive decision theory, which refers to a set of logically consistent propositions that describe or explain action.28 Descriptive decision theories may be tested against observations obtained through monitoring and forecasting. Descriptive theories, models, and conceptual frameworks originate for the most part in political science, sociology, and economics. The main function of these theories, models, and frameworks is to explain, understand, and predict policies by identify- ing patterns of causality. The principal function of approaches to monitoring such as field experimentation is to establish the approximate validity of causal inferences relating policies to their presumed outcomes.29 In Figure 1.2, the descriptive form of policy analysis can be visualized as an axis moving from the lower left (monitoring) to the upper right (forecasting).

Normative policy analysis parallels normative decision theory, which refers to a set of logically consistent propositions that evaluate or prescribe action.30 In Figure 1.2, the normative strategy of policy analysis can be visualized as an axis running from the lower right (prescription) to upper left (evaluation). Different kinds of information are required to test normative and descriptive decision theories. Methods of evaluation and prescription provide information about policy performance and preferred policies, for example, policies that have been or will be optimally efficient because benefits outweigh costs or optimally equitable because those most in need are made better off. One of the most important features of normative policy analysis is that its propositions rest on disagreements about values such as efficiency, equity, responsiveness, liberty, and security.

Problem Finding and Problem Solving The upper and lower halves of Figure 1.2 provide another important distinction. The upper half points to methods that are designed for problem finding, whereas the lower designates methods for problem solving. The problem-finding strategy has to do with the discovery of elements that go into the definition of problems, and not to their solution. How well do we understand the problem? Who are the most impor- tant stakeholders who affect and are affected by the problem? Have the appropriate objectives been identified? Which alternatives are available to achieve objectives?

27Janet A. Weiss, “Using Social Science for Social Policy,” Policy Studies Journal 4, (Spring 1976): 237. 28Bower, “Descriptive Decision Theory,” p. 104. 29See Thomas D. Cook and Donald T. Campbell, Quasi-Experimentation: Design and Analysis Issues for Field Settings (Boston, MA: Houghton Mifflin, 1979); Shadish, Cook, and Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. 30Bower, “Descriptive Decision Theory,” pp. 104–05.

IS B

N 1

-2 56

-9 77

08 -X

14 CHAPTER 1 The Process of Policy Analysis

Which uncertain events should be taken into account? Are we solving the “right” problem rather than the “wrong” one?

Problem-solving methods, located in the lower half of Figure 1.2, are designed to solve rather than find problems. The problem-solving strategy is primarily technical in nature, in contrast to problem finding, which is more conceptual. Problem-solving methods such as econometrics are useful in answering questions about policy causa- tion, statistical estimation, and optimization. How much of the variance in a policy outcome is explained by one or more independent variables? What is the probability of obtaining a coefficient as large as that obtained? Another problem-solving method is benefit-cost analysis. What are the net benefits of different policies? What is their expected utility or payoff?

Segmented and Integrated Analysis Integrated policy analysis links the four strategies of analysis displayed in Figure 1.2. Retrospective and prospective strategies are joined in one continuous process. Descriptive and normative strategies are also linked, as are methods designed to find as well as solve problems. Practically speaking, this means that policy analysts bridge the several main pillars of multidisciplinary policy analysis, especially economics and political science. Today, this need is not being properly met by specialized social science disciplines, which tend to practice segmented policy analysis. The job of bridging segmented disciplines—to convert intellectual knowledge into practical knowledge—is carried out by multidisciplinary professions including public adminis- tration, planning, management, and policy analysis. The American Society for Public Administration (ASPA), the National Association of Schools of Public Affairs and Administration (NASPAA), the American Planning Association (APA), the International Association of Schools and Institutes of Administration (IASIA), the Academy of Management (AM), the Operations Research Society of America (ORSA), and the Association for Public Policy and Management (APPAM) are organizations that represent these professions. So far, these professions have been more open to the disciplines of economics and political science than those disciplines have been open to them, notwithstanding a consensus among policy scholars and practitioners that the substance and methods of these and other disciplines are essential for producing policy-relevant information.

In summary, the framework for integrated policy analysis (Figure 1.1) helps examine the assumptions, strengths, and limitations of methods employed in disci- plines that tend to be overly segmented and excessively specialized to be useful in practical problem solving. The framework identifies and relates major elements of policy analysis—policy-informational components, policy-analytic methods, and policy-informational transformations—enabling us to see the particular roles performed by methods of problem structuring, monitoring, evaluation, forecasting, and prescription. The framework (Figure 1.2) identifies different strategies of policy analysis: prospective (ex ante) and retrospective (ex post), descriptive and normative, and problem finding and problem solving. The framework integrates these strategies of analysis and explains why we have defined policy analysis as a problem-solving discipline that links social science theories, methods, and substantive findings to solve practical problems.

IS B

N 1-256-97708-X

The Practice of Policy Analysis 15

THE PRACTICE OF POLICY ANALYSIS Reconstructed Logic versus Logic-in-Use The process of integrated policy analysis is a logical reconstruction (reconstructed logic). The process of actually doing policy analysis never completely conforms to this reconstruction, because all logical reconstructions are abstract representations of stylized practices endorsed by the scientific community.31 By contrast, the logic-in-use of practicing analysts, as distinguished from the logical reconstruction of their use of reason and evidence to solve practical problems, always varies from methodological “best practices” due to personal characteristics of analysts, their professional social- ization, and the institutional settings in which they work.

� Cognitive styles. The personal cognitive styles of analysts predispose them toward different modes of acquiring, interpreting, and using information.32

Corporations, nonprofit organizations, and public agencies such as the U.S. Department of Corrections and the National Science Foundation use the Myers-Briggs test as a training and personnel selection diagnostic.

� Analytic roles. In agency settings, most analysts are largely insulated from politics. As such, they are primarily “technicians.” Others perform roles that, in addition to technical content, are political. These “politicians” are actively committed to advancing the interests of political leaders or officials to whom they report. Other activist analysts are “entrepreneurs” who seek greater influence in policy making.33

� Institutional incentive systems. Policy “think tanks” encourage different orientations toward analysis, including the “humanistic-value-critical” and the “scientific.”34 Institutional rewards and punishments affect the validity of conclusions and recommendations.35

� Institutional time constraints. Analysts working in governmental settings are often subject to tight institutional time constraints (three to seven days is typical). They work with much greater speed, and perhaps greater efficiency, than analysts in academic settings or think tanks. Understandably, government analysts rarely collect original data; nor do they employ complex and time-consuming techniques.36

31On reconstructed logic and logic-in-use, see Kaplan, Conduct of Inquiry, pp. 3–11. 32Studies using the Myers-Briggs type indicator (Jungian personality types) suggest different cognitive styles among scientists, managers, and analysts. References provided by the Myers and Briggs Foundation at www.myersbriggs.org. See also Ian I. Mitroff and Ralph H. Kilmann, Methodological Approaches to Social Science (San Francisco: Jossey-Bass, 1978). 33Arnold Meltsner, Policy Analysts in the Bureaucracy (Berkeley: University of California Press, 1976); Robert A. Heineman, William T. Bluhm, Steven A. Peterson, and Edward N. Kearney, The World of the Policy Analyst. Chatham, NJ: Chatham House, 1990. 34Pamela Doty, “Values in Policy Research,” in Values, Ethics, and the Practice of Policy Analysis, ed. William N. Dunn (Lexington, MA: D.C. Heath, 1983). 35Donald T. Campbell, “Guidelines for Monitoring the Scientific Competence of Preventive Intervention Research Centers: An Exercise in the Sociology of Scientific Validity,” Knowledge: Creation, Diffusion, Utilization 8, no. 3 (1987): 389–430. 36See P. J. Cook and J. W. Vaupel, “What Policy Analysts Do: Three Research Styles,” Journal of Policy Analysis and Management 4, no. 3 (1985): 427–28.

IS B

N 1

-2 56

-9 77

08 -X

16 CHAPTER 1 The Process of Policy Analysis

� Professional socialization. The different disciplines and professions that make up policy analysis socialize their members into different norms and values. Analyses of published papers suggest that analysts employ formal-quantitative as well as informal-narrative approaches, although sound policy recommendations sometimes require formal-quantitative procedures.37

� Multidisciplinary teamwork. Much of the analysis conducted in public agencies is carried out by multidisciplinary teams. Some members have primary responsibility for the particular types of analysis displayed in Figure 1.2. Team members trained in economics and decision analysis are typically more qualified to perform prospective (ex ante) analysis, whereas team members trained in applied sociology, applied political science, and program evaluation are usually better at retrospective (ex post) analysis. The effectiveness of teams depends on everyone acquiring an operational understanding of analytic methods employed throughout the process of integrated policy analysis.

Methodological Opportunity Costs Integrated analysis has opportunity costs. Given limited time and resources, it is difficult to conduct systematic economic, political, and organizational analyses simultaneously. Multiple triangulation,38 or what Cook calls critical multiplism,39

responds to some of the inadequacies of logical positivism.40 Positivism now appears as a one-sided methodology and epistemology claiming that true statements about the world must be logically and empirically verifiable, expressed in a formal (ideal) language such as mathematical statistics, and confirmed by means of state- ments that correspond to objective reality. Objective reality, rather than a reality constituted by subjective meaningful actions and institutions, is the foundation of true statements. Logical positivism, as Cook argues, was the dominant methodology of policy analysis and program evaluation during the era of President Lyndon Johnson’s War on Poverty. The advantage of critical multiplism over logical positivism is that multiplism provides a better approximation of what is true by employing procedures that triangulate from a variety of perspectives on what is worth knowing and what is known about policies.41

37An early but representative overview of approaches is Janet A. Schneider, Nancy J. Stevens, and Louis G. Tornatzky, “Policy Research and Analysis: An Empirical Profile, 1975–1980,” Policy Sciences 15 (1982): 99–114. 38The methodology of triangulation is analogous to practices employed in geodesic surveys; cartography; navigation; and, more recently, satellite tracking. The position or location of an object is found by means of bearings from two or more fixed points or electronic signals a known distance apart. 39Cook advanced critical multiplism as an alternative to logical positivism. See Thomas D. Cook, “Postpositivist Critical Multiplism,” in Social Science and Social Policy, ed. R. Lane Shotland and Melvin M. Mark (Beverly Hills, CA: Sage Publications, 1985), pp. 21–62. 40A critical assessment of logical positivism is Mary E. Hawkesworth, “Epistemology and Policy Analysis,” in Advances in Policy Studies since 1950, ed. Dunn and Kelly, pp. 293–328; and Hawkesworth, Theoretical Issues in Policy Analysis. 41Cook, “Postpositivist Critical Multiplism,” p. 57.

IS B

N 1-256-97708-X

Critical Thinking and Public Policy 17

A disadvantage of multiplism lies in its costs. Triangulation among multiple disciplinary perspectives, along with the use of multiple methods, measures, and data sources, involves trade-offs and opportunity costs.42 When single methods such as econometric modeling are employed to achieve measurement precision and statistical generalizability, analysts forgo opportunities to acquire a deeper understanding of policies that is possible through ethnographic interviews, case studies, and other qualitative methods. A leading econometrician, noting that economists are unique among social scientists because they are trained only to analyze data, not to collect it, observes that “empirical work can be greatly enhanced by being sensitive to the context of the problem (the data-generating process) and knowing a lot about one’s data.”43 Similar trade-offs apply to methods of research synthesis, or meta-analysis, which purchase measurement precision and generalized policy causation at the expense of a deeper understand- ing of contexts of policy-making.44

Ethnographic interviews, by contrast, involve high information costs because they require the collection of substantial primary data through interviews. However, they also lack precision and seldom permit the generalization of policy causation to other settings. Although greater precision and generalizability can be obtained by means of field studies and field experiments, these are expensive, especially when they are employed in conjunction with mixed (quantitative and qualitative) methods. To be sure, triangulation among convergent (and divergent) perspectives, methods, and measures may enhance the validity of policy analysis and other applied social sciences.45 But the time and financial constraints make trade-offs inevitable.

CRITICAL THINKING AND PUBLIC POLICY The world of the policy analyst is complex. Analysts must sift through and evalu- ate a large volume of available quantitative and qualitative data, make difficult choices among sources of information, select appropriate methods and techniques, and employ effective strategies for communicating the results of analysis through oral briefings and documents (see Chapter 9). These practical challenges place a premium on critical thinking—that is, the capacity to organize, synthesize, and evaluate diverse sources of reasoning and evidence. One method available for this

42See David Brinberg and Joseph E. McGrath, Validity and the Research Process (Beverly Hills, CA: Sage Publications, 1985). For Brinberg and McGrath and other methodological pragmatists, the choice of methods is similar to an optimization problem in decision analysis. See C. West Churchman, Prediction and Optimal Decision: Philosophical Issues of a Science of Values (Englewood Cliffs, NJ: Prentice Hall, 1961); and Russell Ackoff, Scientific Method: Optimizing Applied Research Decisions (New York: John Wiley, 1962). 43Peter Kennedy, A Guide to Econometrics, 4th ed. (Cambridge, MA: MIT Press, 1998), pp. 83–84. 44See Lawrence Rudner, Gene V. Glass, David L. Evartt, and Patrick J. Emery, A User’s Guide to the Meta-Analysis of Research Studies. ERIC Clearinghouse on Assessment and Evaluation, University of Maryland, College Park, 2002. http://echo.edres.org 45The case for triangulation in its many forms is found in Campbell, Methodology and Epistemology for Social Science, ed. Overman.

IS B

N 1

-2 56

-9 77

08 -X

18 CHAPTER 1 The Process of Policy Analysis

purpose is the analysis of policy arguments. By analyzing policy arguments, we are able to identify and probe the assumptions underlying competing policy claims, recognize and evaluate objections to these claims, and synthesize policy-relevant information from different sources.

The Structure of Policy Arguments Policy arguments are the main vehicle carrying debates about public policies.46

Although social scientists may rightly pride themselves on methodological special- ization, they too often forget that “public policy is made of language. Whether in written or oral form, argument is central to all stages of the policy process.”47

The structure of a policy argument can be represented as a set of seven elements (Figure 1.3):48

Policy claim (C). A policy claim is the conclusion of a policy argument. Arguments also include other elements, including policy-relevant infor- mation (I), warrants (W), backings (B), qualifiers (Q), objections (O), and rebuttals (R). The movement from policy-relevant information to claim implies therefore, thus, or so. Policy claims are of different types. Some are normative: “Congress should pass the amendments to the Fair Employment Practices Act.” Some are descriptive: “The use of the Internet will double in the next ten years.”

Policy-relevant information (I). Policy-relevant information provides the grounds for a policy claim. These grounds may be statistical data, experi- mental findings, expert testimony, common sense, or political judgments. Policy-relevant information is a response to the question: What information is relevant to the claim? Information is the starting point of a new argument and the end of a previous one. Policy arguments may lead to complex argument chains, trees, or cycles.

Warrant (W). The warrant is a reason to support a claim. Warrants may be economic theories, ethical principles, political ideas, professional authority, and so forth.49 A warrant answers the question: Why does this reason support the claim? Different types of warrants are related to arguments

46See Frank Fischer and John Forester, ed., The Argumentative Turn in Policy Analysis and Planning (Durham, NC: Duke University Press, 1993). Earlier works on policy argumentation are Ian I. Mitroff and Richard O. Mason, Creating a Dialectical Social Science (Boston: D. Reidel, 1981); William N. Dunn, “Reforms as Arguments,” Knowledge: Creation, Diffusion, Utilization 3 (1982): 293–326; Donald T. Campbell, “Experiments as Arguments,” Knowledge: Creation, Diffusion, Utilization 3 (1982): 327–47; Giandomenico Majone, Evidence, Argument, and Persuasion in the Policy Process (New Haven, CT: Yale University Press, 1989); and Stone, Policy Paradox and Political Reason. 47Majone, Evidence, Argument, and Persuasion, p. 1. 48This structural model of argument is part of the computer software called Rationale 2, which was de- veloped by Tim van Gelder and his colleagues in Australia. URL: www.austhink.com. The classic struc- tural model is presented in Stephen Toulmin, The Uses of Argument (Cambridge: Cambridge University Press, 1958); and Stephen Toulmin, A. Rieke, and A. Janik, An Introduction to Reasoning (New York: Macmillan, 1984). 49Different kinds of warrants yield the “modes” of policy argument presented in Chapter 8.

IS B

N 1-256-97708-X

Critical Thinking and Public Policy 19

FIGURE 1.3 Elements of a policy argument

Source: Created with Rationale 2. Melbourne: Austhink Consulting, 2010. www.austhink.com

made in different disciplines and professions. For example, law uses case comparisons and rules of evidence, whereas economics uses theories and component laws such as the law of diminishing utility of money. Policy makers as well as social scientists employ causal warrants such as “Ethnic cleansing will be deterred by air strikes that establish NATO’s credibility in

A policy claim is the conclusion of a policy argument. There are four types of policy claims: definitional, descriptive, evaluative, and advocative.

CLAIM

Policy-relevant information provides taken-for-granted facts to support a policy claim. Policy-relevant information may be statistical data, experimental findings, expert testimony, common sense, or political judgments.

INFORMATION A warrant is a reason to support a policy claim. Warrants may be economic theories, ethical principles, political ideas, authority, and so forth. Most arguments have multiple warrants.

WARRANT A qualifier expresses the approximate truth of a claim, considering the strength of information, warrants, backings, objections, and rebuttals. Qualifiers may be stated statistically (p < 0.01) or in everyday language (“probably,” “not likely,” “apparently,” “unlikely”).

QUALIFIER

A backing justifies or “backs up” a warrant by providing good reasons for believing the warrant.

BACKING An objection opposes or challenges a qualifier by identifying special conditions or exceptions which reduce confidence in the strength of the qualifier.

OBJECTION

An objection opposes or challenges a backing by identifying special conditions or exceptions that reduce confidence in the truth of the backing.

OBJECTION

A rebuttal opposes or challenges an objection by identifying special conditions or exceptions that reduce confidence in the truth of the objection,

REBUTTAL

An objection opposes or challenges information by identifying special conditions or exceptions that reduce confidence in the truth of the information.

OBJECTION

IS B

N 1

-2 56

-9 77

08 -X

20 CHAPTER 1 The Process of Policy Analysis

the region.” The warrant, which provides a justification for accepting a claim, answers the question: Considering the information, what reasons make the claim true?

Qualifier(Q). The qualifier expresses the degree to which a claim is approxi- mately true, given the strength of the information, warrants, and backings, as well as objections and rebuttals. Although social scientists may state qualifiers in the language of formal probability (p � 0.01 or t � 2.24), ordinary language is the normal mode of qualifying claims with such terms as certainly, absolutely, necessarily, probably, in all likelihood, presumably, apparently, and barring unforeseen circumstances. The qualifier answers the question: How strong or credible is the claim? It is primarily through processes of argumentation and debate that policy makers, policy analysts, and other policy stakeholders adjust or even abandon arguments. Such changes, when they occur, are motivated by the strength of objections and rebuttals offered by those who have a stake in policies.

Backing (B). The backing is an additional reason to support or “back up” the warrant. The backing answers the question: Why does the warrant support the claim? with a more general reason, assumption, or argument that begins with because. Different kinds of backings are characteristically employed by members of different disciplines and professions. Backings may be scientific laws, appeals to the authority of experts, or ethical and moral principles. For example, consider the warrant presented earlier: “Ethnic cleansing will be deterred by air strikes that establish NATO’s credibility in the region.” The backing for warrants advocating the use of coercive force is frequently an informal statement of the law of diminishing utility: “The greater the cost of an alternative, the less likely it will be pursued.”

Objection (O). An objection opposes or challenges the information, warrant, backing, or qualifier by identifying special conditions or exceptions that reduce confidence in the truth of the information, warrant, backing, or qualifier. An objection answers the question: Are there special circum- stances or exceptions that threaten the credibility of the warrant? Analysts who pay attention to objections are more likely to take a critical perspec- tive toward a policy argument, identifying weak or hidden assumptions, anticipating unintended consequences, or questioning possible rebuttals to objections. Thereby, analysts can be self-critical, challenging their own assumptions and arguments.

Rebuttal(R). A rebuttal is an objection to an objection. Rebuttals oppose or challenge objections by identifying special conditions or exceptions that reduce confidence in the truth of the objection. Rebuttals answer the ques- tion: Are there special circumstances or exceptions that threaten the cred- ibility of the objection? Most policy arguments have objections and rebut- tals, because policy making involves bargaining, negotiation, competition, and compromise among opponents and proponents of policies.

The frames of reference, perspectives, and reasons of policy makers and analysts are found in their underlying warrants, backings, objections, and rebuttals. Therefore, identical policy-relevant information is interpreted in distinctly different

IS B

N 1-256-97708-X

Demonstration Exercises 21

ways. A decrease in crime rates in urban areas may be welcomed by the urban poor, viewed with skepticism by owners of central city businesses, rejected by criminolo- gists who attribute urban crime rates to changes in unemployment and homelessness, and hailed as an achievement by elected officials. By examining contending arguments and their underlying assumptions, analysts can uncover and critically assess reasoning and evidence that otherwise goes unnoticed. Equally important is what it brings to analysts themselves—they can probe their own assumptions by examining the objections, qualifications, and exceptions to their own conclusions.

REVIEW QUESTIONS 1. What does it mean to define policy analysis as

a process of inquiry as distinguished from a set of methods?

2. Describe the dynamics of policy-informational components, policy-analytic methods, and policy-informational transformations.

3. Contrast segmented and integrated policy analysis. Give examples.

4. How does normative decision theory differ from descriptive decision theory?

5. List some of the key differences between prob- lem solving and problem finding.

6. Contrast retrospective and prospective analy- sis. Which social science disciplines tend to spe- cialize in prospective analysis? Retrospective analysis?

7. Discuss the strengths and limitations of critical multiplism.

8. Contrast the “logic-in-use” and the “recon- structed logic” of policy analysis. Provide examples.

9. How can argumentation mapping assist analysts to become critical thinkers?

DEMONSTRATION EXERCISES 1. Scorecards provide a useful overview of the

observed and expected outcomes of different policies. When using the scorecard for monitor- ing and forecasting the outcomes of the two speed limits (Case 1.2), the 55 mph speed limit seems preferable to the 65 mph speed limit. Compare the scorecard (Figure C1.2) with the spreadsheet (Figure C1.3). Does the comparison change your conclusions about the performance of the 55 mph speed limit? What does this tell us

about the differences between monitoring and evaluation in policy analysis? What are the implications for the distinctions among different types of policy analysis?

2. Influence diagrams and decision trees are useful methods for structuring policy problems. The diagram and tree displayed in Figure 1.2 help identify policy stakeholders, policy alternatives, uncertain outcomes and events, probabilities of these outcomes and events, and valued

CHAPTER SUMMARY This chapter has provided a framework for policy analysis that identifies the role of policy-analytic methods in creating and trans- forming policy-relevant information. The four regions of this framework call attention to similarities and differences among methods of policy analysis and point to the origins of these methods in different social science disciplines and professions, thus clarifying the meaning of

multidisciplinary inquiry. No one methodol- ogy is appropriate for all or most problems. Given the need to choose among methods, methodological choices can be viewed as an optimization problem involving trade-offs and opportunity costs. The actual work of practic- ing analysts demands critical thinking. The analysis of policy arguments is well suited for this purpose.

IS B

N 1

-2 56

-9 77

08 -X

22 CHAPTER 1 The Process of Policy Analysis

outcomes (valued outcomes are objectives). Consider the influence diagram that represents the problem of energy supply in 1973 and 1974, when the OPEC oil embargo posed sig- nificant challenges both to U.S. energy supply and national security. How does Figure C1.3 help us formulate the problem? What do the ar- rows suggest about the causes of the energy shortage as well as the causes of the decline in traffic fatalities? What does the influence diagram suggest about the conditions that gave rise to the 55 mph speed limit and its effective- ness? After comparing the influence diagram with the decision tree (which is based on the

influence diagram), describe why these two problem representations are good examples of descriptive and normative decision theory.

3. Create an argument map based on the influ- ence diagram presented in Case 1.3. Begin with the following claim: “The United States should return to the 55 mph speed limit in order to conserve fuel and save lives.” Include in your map as many warrants, backings, objections, and rebuttals as you can. Assuming that the original qualifier was certainly, indicate whether the qualifier changes as we move from a simple, static, uncontested argument to a complex, dynamic, and contested argument.

BIBLIOGRAPHY Campbell, Donald T. Methodology and Epistemology

for Social Science: Selected Papers. Edited by E. Samuel Overman. Chicago: University of Chicago Press, 1988.

Diesing, Paul. How Social Science Works: Reflections on Practice. Pittsburgh, PA: Pittsburgh University Press, 1991.

Dunn, William N., and Rita Mae Kelly. Advances in Policy Studies since 1950. New Brunswick, NJ: Transactions Books, 1992.

Fischer, Frank, and John Forester. The Argumentative Turn in Policy Analysis and Planning. Durham, NC: Duke University Press, 1993.

Hawkesworth, Mary E. Theoretical Issues in Policy Analysis. Albany: State University of New York Press, 1988.

Kaplan, Abraham. The Conduct of Inquiry: Methodology for Behavioral Science. San Francisco, CA: Chandler, 1964.

Mac Rae, Duncan Jr. The Social Function of Social Science. New Haven, CT: Yale University Press, 1976.

Stone, Deborah. Policy Paradox: The Art of Political Decision Making. Rev Ed. New York: W. W. Norton, 2001.

Toulmin, Stephen R. Return to Reason. Cambridge, MA: Harvard University Press, 2001.

Van Gelder, Tim. “The Rationale for Rationale.” Law, Probability, and Risk 6 (2007): 23–42.

When advanced technologies are used to achieve policy goals, sociotechnical systems of considerable complexity is created. Although it is analytically tempting to prepare a comprehensive economic analysis of the costs and benefits of such policies, most practicing analysts do not have the time or

the resources to do so. Given the time constraints of policy making, many analyses are completed in a period of several days to a month, and in most cases policy analyses do not involve the collection and analysis of new data. Early on in a project, policy makers and their staffs typically want an

CASE 1.1 THE GOELLER SCORECARD— MONITORING AND FORECASTING TECHNOLOGICAL IMPACTS

IS B

N 1-256-97708-X

Case 1.1 23

50E.S. Quade, Analysis for Public Decisions (New York: American Elsevier, 1975), p. 65.

TABLE C1.1

Scorecard

Social Impacts CTOL VTOL TACV

TRANSPORTATION

Passengers (million miles) 7 4 9 Per trip time (hours) 2 1.5 2.5 Per trip cost ($) $17 $28 $20 Reduced congestion (%) 0% 5% 10%

FINANCIAL

Investment ($ millions) $150 $200 $200 Annual subsidy ($ millions) 0 0 90

ECONOMIC

Added jobs (thousands) 20 25 100 Added sales ($millions) 50 88 500

COMMUNITY

Noise (households) 10 1 20 Added air pollution (%) 3% 9% 1% Petroleum savings (%) 0% –20% 30% Displaced households 0 20 500 Taxes lost ($millions) 0 0.2 2 Landmarks destroyed None None Fort X

DISTRIBUTIONAL

Low-income trips (%) 7% 1% 20% Low-income household Noise annoyance (%) 2% 16% 40%

SOurce: Goeller (1974); Quade, Analysis for Public Decisions (1975), p. 60.

NOte: Conventional takeoff and landing aircraft (CTOL); vertical takeoff and landing aircraft (VTOL); tracked air-cushion vehicle (TACV).

overview of the problem situation and the potential impacts of alternative policies. Under these circumstances, the scorecard is appropriate.

The Goeller scorecard, named after Bruce Goeller of the RAND Corporation, is appropriate for this purpose.Table C1.1 shows the impacts of alternative transportation systems. Some of the impacts involve transportation services used by members of the community, whereas others involve

impacts on low-income groups. In this case, as Quade observes, the large number of diverse impacts are difficult to value in dollar terms, making a benefit-cost analysis impractical and even impossible.50 Other impacts involve financial and economic questions such as investments, jobs created, sales, and tax revenues. Other impacts are distributional because they involve the differential effects of transportation. �

IS B

N 1

-2 56

-9 77

08 -X

In 1972 and 1973, the United States and other petroleum-dependent countries experienced the first of several oil crises precipitated by a dramatic increase in the price of crude oil by the Organization of Petroleum Exporting Countries (OPEC).The response of American and European leaders was to adopt maximum speed limits of 55 mph and 90 kph, respectively. In the United States, the National Maximum Speed Limit (NMSL) was designed to reduce the consumption of gasoline by requiring that all vehicles on interstate highways travel at a maximum of 55 mph, a speed that would maximize fuel efficiency for most vehicles.

Soon after the implementation of the 55 mph speed limit, it was discovered that the new policy not only reduced fuel consumption, but apparently caused a dramatic decline in traffic fatalities and injuries as well. Therefore, long after the OPEC oil crisis was over, the speed limit was retained, although it was no longer needed to respond to the energy crisis that prompted its passage in 1973. Indeed, the 55 mph speed limit was retained for more than 20 years until it was officially repealed in November 1995.51

Heated debates preceded the repeal. Senator John C. Danforth of Missouri, an influential advocate of the policy, argued that the repeal would save one minute per day per driver but result in an

additional 600 to 1,000 deaths. The Washington Post and the New York Times joined the opposition, reporting that, although fatalities would surely rise, the savings in time was trivial. Later, Secretary of Transportation Pena announced that the Clinton administration was firmly opposed to abandoning the speed limit.

This was the right moment for an evaluation of the benefits and costs of the NMSL. A spreadsheet is a simple but powerful tool for doing so. The scorecard, as we saw in Case 1.1, is a useful tool for monitoring and forecasting impacts when benefit- cost analysis is not feasible or desirable. On the scorecard, policy alternatives are arrayed in columns along the top of the matrix and policy impacts are listed in each row. Spreadsheets, by contrast, are appropriate and useful for prescribing preferred policies and evaluating their outcomes. Spreadsheets display the benefits and costs of observed or expected policy outcomes, creating information about policy performance as well as preferred policies (see Figure 1.1).

Table C1.2 displays a spreadsheet used to evaluate the effects of the 55 mph speed limit at the end of 1974, one year after the policy was implemented. To show the differences between the spreadsheet and the scorecard, Table C1.2 also displays the same information as a scorecard. �

51On April 2, 1987, Congress enacted the Surface Transportation and Uniform Relocation Assistance Act, permitting 40 states to experiment with speed limits up to 65 mph.

CASE 1.2 THE SPREADSHEET—EVALUATING THE BENEFITS AND COSTS OF ENERGY POLICIES

IS B

N 1-256-97708-X

Case 1.3 25

(continued)

TABLE C1.2

Scorecard and Spreadsheet

(a) Scorecard

65 MPH OUTCOMES (Base Case)* 55 MPH

Fatalities 54,052 45,196 Miles traveled (billions) 1,313 1,281 Hours driving (billions) 20.2 21.9 Gallons fuel consumed (billions) 46.8 43.3 Fuel efficiency (mpg) 14.9 16.1 Traffic citations (millions) 5,711 7,425 Property damage (million cases) 25.8 23.1

*The base case is the policy against which the new policy is compared.

(b) Spreadsheet

OBJECTIVES 65MPH 55MPH Difference Value $ Billions

I Fatalities (000s) 54.1 45.2 8.856 $240,000.00 $ 2.13

II Hours driving (billions) 20.2 21.9 �1.7 5.05 �8.59 III Gallons fuel consumed

(billions) 46.8 43.3 3.5 0.53 1.86

IV Traffic citations (000s) 5,711 7,425 �1,714 3.94 �0.0068 V Property damage

cases (000s) 25,800 23,100 2,700 363.00 0.98

Benefits (I � III � V) 4.97

Costs (II � IV) �8.60 Net Benefits (B–C) $ �3.63

Along with other policy-analytic methods discussed earlier in this chapter (Figure 1.1), the influence diagram and decision tree are useful tools for structuring policy problems.52 The influence

diagram (Figure C1.3) displays the policy, the National Maximum Speed Limit, as a rectangle. A rectangle always refers to a policy choice or decision node, which in this case is the choice

52The diagram and tree were created with the Decision Programming Language (DPL), which is available from Syncopation Software at http://www.syncopation.com. Educational, professional, and commercial versions of DPL 7.0 are available.

CASE 1.3 THE INFLUENCE DIAGRAM AND DECISION TREE—STRUCTURING PROBLEMS OF ENERGY POLICY AND INTERNATIONAL SECURITY

IS B

N 1

-2 56

-9 77

08 -X

26 CHAPTER 1 The Process of Policy Analysis

Travel Time

Fuel Used

Net Benefits

Injuries

Fatalities

Miles Traveled

Employment

Recession OPEC

Oil Crisis

NMSL

FIGURE C1.3 Influence diagram and decision tree

between adopting and not adopting the national maximum speed limit of 55 mph. To the right and above the decision node are uncertain events, represented as ovals, which are connected to the decision node with arrows showing how the speed limit affects or is affected by them. The rectangles with shaved corners represent valued policy outcomes or objectives. The objectives are to lower fuel consumption, reduce travel time, reduce injuries, and avert traffic fatalities. To the right of the objectives is another shaved rectangle, which designates the net benefits (benefits less costs) of the four objectives.The surprising result of using the influence diagram for problem structuring is the discovery of causally relevant economic events,

such as the recession and unemployment, which affect miles driven, which in turn affect all four objectives. The “root cause” appears to be the OPEC oil embargo.

The decision tree is another representation of the influence diagram. Whereas the influence diagram shows how policy choices and uncertain events affect the achievement of objectives, the decision tree displays the monetary value of these objectives. In this abridged and simplified decision tree, there are two branches that represent the alternatives but also the OPEC oil embargo, the recession, the costs of miles traveled, and the dollar benefits of reducing fatalities. The bolded branches show the events with the greatest likelihood of occurring or that already have occurred. �

Yes Yes

OPEC Oil Crisis

Renew NMSL

Recession FatalitiesMiles Traveled

Yes

Down 10% (32 billion miles)

$2.77 per 100 Miles Traveled

$2,77 per 100 Miles Traveled Down 3%

$2.77 per 100 Miles Traveled Unchanged

Down 20% (8,900 fatalities)

$240,000 per Fatality Averted

$240,000 per Fatality Averted Down 5%

$240,000 per Fatality Averted Down 10%

No No No

(b) Decision Tree

(a) Influence Diagram

IS B

N 1-256-97708-X

CASE 1.4 THE ARGUMENT MAP—PROBLEM STRUCTURING IN NATIONAL DEFENSE AND TRANSPORTATION POLICY

The role of causal arguments in transforming policy- relevant information into policy claims may be illustrated by Allison’s well-known study of foreign policy decision making during the Cuban missile crisis of October 1962.53 Showing how different explanatory models yield different conclusions, Allison argues that government policy analysts think about problems of foreign policy in terms of implicit conceptual models that shape their thought; most analysts explain the behavior of governments in terms of a model that assumes the rationality of political choices (rational actor model); alternative models, including those that emphasize organizational processes (organizational process model) and bureaucratic politics (bureaucratic politics model), provide a basis for improved explanations.

In 1962, the policy alternatives open to the United States ranged from no action and diplomatic pressure to secret negotiations, invasion, surgical air strikes, and blockade. Among the several claims made at the time of the Cuban missile crisis, let us consider the policy actually adopted by the United States: “The United States should blockade Cuba.” In this case, the policy-relevant information (I) is “The Soviet Union is placing offensive missiles in Cuba.” The warrant states that “the blockade will force the withdrawal of missiles by showing the Russians that the United States is determined to use force.” In providing reasons to accept the warrant, the backing (B) supports the warrant by stating that “an increase in the cost of an alternative reduces the likelihood of that alternative being chosen.”54 The backing (B) represents a general theoretical

proposition, or law, within the rational policy model. After the objection (O) has successfully challenged the warrant, the qualifier (Q) changes from absolutely to doubtful.

Allison’s account shows how the use of multiple competing explanations can facilitate critical thinking. The use of multiple competing models moves the analysis from a simple uncontested argument (Figure C1.4.1) to a new argument that is complex, contested, and dynamic (Figure C1.4.2). This change occurs because a serious objection has been raised about the warrant and the backing of the claim. The objection states: “But Soviet leaders may fail to convince their naval units to depart from established organizational routines.” The warrant for this objection is: “The bulk of research on organizations shows that major lines of organizational behavior tend to be straight. Behavior at time t+1 differs little from behavior at time t.55

The blockade will not work.” The warrant for the objection is again a general proposition or law within the organizational process model, otherwise known as the disjointed incremental theory of policy change.

Simple uncontested maps of arguments about the 55 mph speed limit are presented alongside of the arguments about the Cuban missile crisis (Figures C1.4.1a and C1.4.1b).The comparisons show that the simple argument maps represent uncritical thinking. By contrast, the complex, dynamic, and contested maps of the same crises (Figures C1.4.2a and C1.4.2b) illustrate what is meant by critical thinking. �

53Graham T. Allison, “Conceptual Models and the Cuban Missile Crisis,” American Political Science Review 3002, no. 3 (1969): 689–718. 54Allison, “Conceptual Models,” p. 694. 55Ibid., p. 702.

(continued)

27IS B

N 1

-2 56

-9 77

08 -X

28 CHAPTER 1 The Process of Policy Analysis

FIGURE C1.4.1 Simple argument maps are static and uncontested

Reliable intelligence reports confirm that the Soviet Union is placing offensive missiles in Cuba.

I A blockade will show Soviet leaders that the States means business and is prepared to use force.

W Very probably

Qsupports supports

The United States should force the Soviet Union to withdraw the missiles by blockading Cuba.

(a) The Cuban Missile Crisis—Simple Uncontested Argument

There was a decline of 8,300 traffic fatalities in the year following the implementation of the 55 MPH speed limit.

I The speed limit was responsible for the decline in traffic fatalities.

W Certainly Q

support

Congress should reinstate the 55 MPH speed limit.

(b) The 55 mph Speed Limit—Simple Uncontested Argument

IS B

N 1-256-97708-X

Case 1.4 29

The United States should force the Soviet Union to withdraw the missiles by blockading Cuba.

Because an increase in the cost of an action reduces the likelihood that it will be taken.

B But considering the objections, the claim is doubtful.

Given that reliable intelli- gence reports confirm that the Soviet Union is placing offensive missiles in Cuba.

I Because a blockade will show Soviet leaders that the States means business and is prepared to use force.

W Absolutely Q weaklysupport

weakly support

strongly opposes

But Soviet leaders may fail to convince their naval units to depart from established routines.

O strongly opposes

Because the bulk of research on organizations shows that major lines of behavior tend to be straight: Behavior at time t + 1 differs little from behavior at time t. The blockade will not work.

W strongly supports

(a) The Cuban Missile Crisis—Dynamic Contested Argument

(continued)

IS B

N 1

-2 56

-9 77

08 -X

30 CHAPTER 1 The Process of Policy Analysis

FIGURE C1.4.2 Complex argument maps are dynamic and contested

Congress should reinstate the 55 MPH speed limit.

Since it is obvious that the decline is due to the speed limit. No other factors can explain the decline, which is the largest in U.S. history.

B But this is not certain at all.

Given that there was a decline of 8,300 traffic fatalities in the year following the implementation of the 55 MPH speed limit.

I Because the speed limit was responsible for the decline in traffic fatalities.

W Certainly Q

support

support opposes

But most of the decline was due to the 1974 recession, the rise in unemployment, the doubling of gasoline prices, and the consequent sharp decline in miles driven, which drastically reduced the exposure to accidents.

O opposes

(b) The 55 mph Speed Limit—Simple Uncontested Argument

IS B

N 1-256-97708-X