EDU 571 Paper 2

CHAPTER 6

Today many approaches to evaluation begin their focus with learning more about some key features of the program to be evaluated. These features then serve to help the evaluator decide which questions should be addressed. The most prominent program-oriented approaches are the objectives-oriented approach and approaches that make use of logic models or program theory. In fact, theory-based evaluation is one of the most rapidly growing areas of evaluation (Weiss, 1995; Donaldson, 2007). Many government funding agencies and foundations require logic models, a variant of program theory, for program planning, evaluation, and research. Both logic models and program theory have evolved to help evaluators gain a better understanding of the rationale or reasoning behind the program’s intended effects; this represents a great improvement over the more traditional objectives-oriented evaluation, which focused only on stated program outcomes.

Fitzpatrick, Jody L.. Program Evaluation (p. 153). Pearson Education. Kindle Edition.

In this chapter, we will cover the original program-oriented evaluation approach—objectives-oriented evaluation—which continues to be used frequently today. We will then describe theory-oriented approaches and their cousin, logic models, and their applications today in helping evaluators make the critical choices of what to evaluate. The Objectives-Oriented Evaluation Approach The distinguishing feature of an objectives-oriented evaluation approach is that the purposes of some activity are specified, and then the evaluation focuses on the extent to which those purposes, or objectives, are achieved. In many cases, programs already have specified objectives. In other cases, the evaluator may work with stakeholders to articulate the program objectives, sometimes called goals or standards. The key role for the evaluator in an objectives-oriented evaluation is to determine whether some or all of the program objectives are achieved and, if so, how well they are achieved. In education, the objectives may be concerned with the purposes of a single lesson or training program or the knowledge students should attain during an entire year. In public health programs, the objectives may concern the effects of prevention efforts, community health interventions, or patient education. Objectives in environmental programs might include such quantitative outcomes as reduction in air pollutants or outcomes that are more difficult to measure such as citizens’ beliefs and behaviors about energy use. The information gained from an objectivesoriented evaluation could be used to determine whether to continue funding the program, change significant portions of it, or throw it out and consider other approaches. Many people have contributed to the evolution and refinement of the objectives-oriented approach to evaluation since its inception in the 1930s, but the individual most credited with conceptualizing and popularizing the focus on objectives in education is Ralph W. Tyler (1942, 1950). The Tylerian Evaluation Approach Tyler had a tremendous influence on both evaluation and education., His work influenced the Elementary and Secondary Education Act (ESEA) of 1965, the first federal act to require evaluation of educational programs. At the end of his career, he chaired the committee that started the National Assessment of Educational Progress (NAEP), which in the United States today remains the only way to examine educational achievement across all 50 states because of the different standards in each state. In the 1920s and 1930s, while working closely with teachers and schools, Tyler began to formulate his views on education and evaluation. His writings and work foreshadowed today’s concepts of continuous improvement and multiple means of assessment. He saw objectives as a way for teachers to

Fitzpatrick, Jody L.. Program Evaluation (p. 154). Pearson Education. Kindle Edition.

define what they wanted students to learn. By stating objectives in terms of what students should be able to do, Tyler believed that teachers could more effectively plan their curricula and lessons to achieve those objectives. Unlike later versions of behavioral objectives, however, Tyler believed that objectives should concern principles, not minute behaviors. He worked closely and cooperatively as an evaluator with teachers to make evaluation and education cooperative endeavors (Goodlad, 1979; Madaus, 2004; Madaus & Stufflebeam, 1989). Tyler considered evaluation to be the process of determining the extent to which the objectives of a program are actually being achieved. His approach to evaluation followed these steps: 1. Establish broad goals or objectives. 2. Classify the goals or objectives. 3. Define objectives in behavioral terms. 4. Find situations in which achievement of objectives can be shown. 5. Develop or select measurement techniques. 6. Collect performance data. 7. Compare performance data with behaviorally stated objectives. Discrepancies between performance and objectives would lead to modifications intended to correct the deficiency, and the evaluation cycle would be repeated. Tyler’s rationale was logical, scientifically acceptable, readily adoptable by evaluators (most of whose methodological training was very compatible with the pretest-posttest measurement of behaviors stressed by Tyler), and had great influence on subsequent evaluation theorists. Tyler advocated multiple measures of different types and considered many elements of a program during an evaluation. However, the objectives-oriented approaches that evolved from Tyler’s work in the 1960s and 1970s and that continue to be used in some settings today focused on a basic formula: articulate program objectives; identify the means, typically tests, to measure them; administer the tests; analyze the data in reference to previously stated objectives; and determine program success. This basic, objectives-oriented approach is largely discredited by professional evaluators today. However, many funding sources have not caught up with present-day evaluation approaches and require evaluations to make use of this traditional approach. Its strengths and limitations are discussed in the conclusion of the chapter. Provus’s Discrepancy Evaluation Model Another approach to evaluation in the Tylerian tradition was developed by Malcolm Provus, who based his approach on his evaluation assignments in the Pittsburgh public schools (Provus, 1971, 1973). Provus viewed evaluation as a continuous information-management process designed to serve as “the watchdog of program management” and the “handmaiden of administration in the management of program development through sound decision making” (Provus, 1973, p. 186). Although his was, in some ways, a management-oriented

Fitzpatrick, Jody L.. Program Evaluation (p. 155). Pearson Education. Kindle Edition.

evaluation approach, the key characteristic of his proposals stemmed from the Tylerian tradition. Provus viewed evaluation as a process of (1) agreeing on standards (another term used in place of objectives),1 (2) determining whether a discrepancy exists between the performance of some aspect of a program and the standards set for performance, and (3) using information about discrepancies to decide whether to improve, maintain, or terminate the program or some aspect of it. He called his approach, not surprisingly, the Discrepancy Evaluation Model (DEM). Provus determined that, as a program is being developed, it goes through four developmental stages, to which he added a fifth, optional stage: 1. Definition 2. Installation 3. Process (interim products) 4. Product 5. Cost-benefit analysis (optional) During the definition, or design, stage, the focus of work is on defining goals and processes or activities and delineating necessary resources and participants to carry out the activities and accomplish the goals. Provus considered programs to be dynamic systems involving inputs (antecedents), processes, and outputs (outcomes). Standards or expectations were established for each stage. These standards were the objectives on which all further evaluation work was based. The evaluator’s job at the design stage is to see that a complete set of design specifications is produced and that they meet certain criteria: theoretical and structural soundness. At the installation stage, the program design or definition is used as the standard against which to judge program operation. The evaluator performs a series of congruency tests to identify any discrepancies between expected and actual implementation of the program or activity. The intent is to make certain that the program has been installed as it has been designed. This is important because studies have found that staff vary as much in implementing a single program as they do in implementing several different ones. The degree to which program specifications are followed is best determined through firsthand observation. If discrepancies are found at this stage, Provus proposed several solutions to be considered: (a) changing the program definition to conform to the way in which the program is actually being delivered if the actual delivery seems more appropriate, (b) making adjustments in the delivery of the program to better conform to the program definition (through providing more resources or training), 1 Although standards and objectives are not synonymous, they were used by Provus interchangeably. Stake (1970) also stated that “standards are another form of objective: those seen by outside authority figures who know little or nothing about the specific program being evaluated but whose advice is relevant to

Fitzpatrick, Jody L.. Program Evaluation (p. 156). Pearson Education. Kindle Edition.

or (c) terminating the activity if it appears that further development would be futile in achieving program goals. During the process stage, evaluation focuses on gathering data on the progress of participants to determine whether their behaviors changed as expected. Provus used the term “enabling objective” to refer to those gains that participants should be making if longer-term program goals are to be reached. If certain enabling objectives are not achieved, the activities leading to those objectives are revised or redefined. The validity of the evaluation data would also be questioned. If the evaluator finds that enabling objectives are not being achieved, another option is to terminate the program if it appears that the discrepancy cannot be eliminated. At the product stage, the purpose of evaluation is to determine whether the terminal objectives for the program have been achieved. Provus distinguished between immediate outcomes, or terminal objectives, and long-term outcomes, or ultimate objectives. He encouraged the evaluator to go beyond the traditional emphasis on end-of-program performance and to make follow-up studies, based on ultimate objectives, a part of all program evaluations. Provus also suggested an optional fifth stage that called for a cost-benefit analysis and a comparison of the results with similar cost analyses of comparable programs. In recent times, with funds for human services becoming scarcer, cost-benefit analyses have become a part of many program evaluations. The Discrepancy Evaluation Model was designed to facilitate the development of programs in large public school systems and was later applied to statewide evaluations by a federal bureau. A complex approach that works best in larger systems with adequate staff resources, its central focus is on identifying discrepancies to help managers determine the extent to which program development is proceeding toward attainment of stated objectives. It attempts to assure effective program development by preventing the activity from proceeding to the next stage until all identified discrepancies have been removed. Whenever a discrepancy is found, Provus suggested a cooperative problem-solving process for program staff and evaluators. The process called for asking the following questions: (1) Why is there a discrepancy? (2) What corrective actions are possible? (3) Which corrective action is best? This process usually required that additional information be gathered and criteria developed to allow rational, justifiable decisions about corrective actions (or terminations). This particular problem-solving activity was a new addition to the traditional objectivesoriented evaluation approach. Though the Discrepancy Evaluation Model was one of the earliest approaches to evaluation, elements of it can still be found in many evaluations. For example, in Fitzpatrick’s interview with David Fetterman, a developer of empowerment evaluation, on his evaluation of the Stanford Teacher Education Program (STEP), Fetterman uses the discrepancy model to identify program areas (Fitzpatrick & Fetterman, 2000). The fact that the model continues to influence evaluation studies 30 years later is evidence of how these seminal

Fitzpatrick, Jody L.. Program Evaluation (p. 157). Pearson Education. Kindle Edition.

The three dimensions of the cube are as follows: 1. Needs of youth (the client): categories developed by Stufflebeam (1977) and expanded by Nowakowski et al. (1985) are • intellectual • physical recreation • vocational • social • moral • aesthetic/cultural • emotional 2. Age of youth (this dimension could be any relevant characteristic of the client): prenatal through young adult 3. Source of service to youth, such as • housing • social services • health services • economic/business • public works • justice • education • religious organizations In any category along any of the three dimensions, those planning a communitybased youth program may choose to establish relevant objectives. Few, if any, stakeholders in community-based programs will be interested in every cell of the cube, but the categories contained in each of the three dimensions will provide a good checklist for making certain that important areas or categories of objectives are not overlooked. Obviously, use of the cube is not limited to community-based programs but could extend to other types of programs as well. Logic Models and Theory-Based Evaluation Approaches Logic Models One of the criticisms of objectives-oriented evaluation is that it tells us little about how the program achieves its objectives. This can be a particular problem when programs fail to achieve their objectives, because the evaluation can provide little advice on how to do so. Logic models have developed as an extension of objectives-oriented evaluation and are designed to fill in those steps between the program and its objectives. Typically, logic models require program planners or evaluators to identify program inputs, activities, outputs, and outcomes, with outcomes reflecting longer-term objectives or goals of the program and outputs

Fitzpatrick, Jody L.. Program Evaluation (p. 159). Pearson Education. Kindle Edition. representing immediate program impacts. The model, typically presented in a diagram form, illustrates the logic of the program. A typical logic model may include the following: Inputs—annual budgets, staffing facilities, equipment, and materials needed to run the program Activities—weekly sessions, curriculum, workshops, conferences, recruitment, clinical services, newsletters, staff training, all the key components of the program Outputs—numbers of participants or clients served each week, number of class meetings, hours of direct service to each participant, number of newsletters and other immediate program products Immediate, intermediate, long-term, and ultimate outcomes—the longitudinal goals for participant change (development) Logic models are widely used in program planning and evaluation today. They have influenced evaluation by filling in the “black box” between the program and its objectives. Evaluators can use logic models to help program staff articulate and discuss their assumptions about how their program might achieve its goals and what elements are important to evaluate at any given time and generally to build internal evaluation capacity or the ability to think in an evaluative way. (See Taylor-Powell & Boyd [2008] for an example of the use of logic models in cooperative extension to build organizational capacity. Knowlton and Phillips [2009] also provide guidance for building logic models.) The United Way of America was one of the major organizations to bring logic models to evaluation through the logicmodel-based approach it requires for the organizations it funds (United Way, 1996). Other foundations, such as the W.K. Kellogg Foundation and the Annie E. Casey Foundation, have also been instrumental in training organizations in the use of logic models to improve program planning and evaluation. Theory-Based or Theory-Driven Evaluation Carol Weiss first discussed basing evaluation on a program’s theory in her 1972 classic book building on earlier writings by Suchman (1967) on the reasons that programs fail (Weiss, 1997; Worthen, 1996a). She has remained an effective and long-term advocate for theory-based evaluations (Weiss, 1995, 1997; Weiss & Mark, 2006). In the 1980s and 1990s, Huey Chen, Peter Rossi, and Leonard Bickman began writing about theory-based approaches to evaluation (Bickman, 1987, 1990; Chen & Rossi, 1980; 1983; Chen, 1990). Stewart Donaldson (2007) is one of the principal evaluators practicing and writing about the theory-driven evaluation approach today.2 Edward Suchman (1967) had first made the point that programs can fail to

Fitzpatrick, Jody L.. Program Evaluation (p. 160). Pearson Education. Kindle Edition.

achieve their goals for two distinctly different reasons: (a) the program is not delivered as planned and, therefore, is not really tested (implementation failure); and (b) the program is delivered as planned and the results, then, clearly indicate that the program theory is incorrect (theory failure). He and Weiss recognized that, if an evaluation were examining whether a program achieved its goals and that program failed, it was important to know whether the failure was an implementation failure or a theory failure. With this information, the evaluator could then reach appropriate conclusions about the program and make useful recommendations for the decision maker. To distinguish between implementation failure and theory failure, the evaluator had to know two things in addition to simply measuring outcomes: (a) the essentials of the program theory. and (b) how the program was implemented. With this information, the evaluator could then determine whether the program implementation matched the theory. This was the beginning of program theory and the recognition of its importance to evaluation practice. Chen’s and Bickman’s approaches to theory-based evaluation arose for these reasons, but also from their desire for evaluations to contribute more directly to social science research knowledge. Chen, for example, argued that evaluators of the time erred in focusing solely on methodology and failing to consider the theory or tenets of the program. For many of those writing about theory-based evaluation as it first emerged in the late 1980s and 1990s, theory meant connecting evaluation to social science research theories. Chen (1990), for example, encouraged evaluators to search the scientific research literature to identify social science theories that were relevant to the program and to use those theories in planning the evaluation. Evaluation results could then contribute to social science knowledge and theory as well as to program decisions (Bickman, 1987). Thus, theory-based evaluation arose from a science-based perspective and was often considered to be a strictly quantitative approach by others during the debates on qualitative and quantitative methods in the 1990s. However, today, theory-based evaluation is used by evaluators in many settings to gain a better understanding of the program. (See Rogers, 2000, 2001.) They can then use that understanding, the program theory, to better define the evaluation questions the study should address, to aid their choices of what concepts to measure and when to measure them, and to improve their interpretation of results and their feedback to stakeholders to enhance use. But what is program theory? And what do evaluators using theory-based evaluation approaches do? Bickman defines program theory as “the construction of a plausible and sensible model of how a program is supposed to work” (Bickman, 1987, p. 5). More recently, Donaldson defines program theory as “the process through which program components are presumed to affect outcomes and the conditions under which these processes are believed to operate” (2007, p. 22). In both cases, and in other definitions, program theory explains the logic of the program. How does it differ from a logic model? In fact, they are quite similar. A logic model may depict the program theory if its articulation of program inputs, activities, outputs, and outcomes is sufficient to describe why the program is intended to achieve its outcomes. Logic models are sometimes used as tools to develop program theory. In other words, a program theory may look like a logic model. In our experience, because the emphasis in logic models is on the stages of

Fitzpatrick, Jody L.. Program Evaluation (p. 161). Pearson Education. Kindle Edition.

input, activities, outputs, and outcomes, the person developing the logic model focuses on listing all of the components in each of those categories and may, though certainly does not always, fail to describe the rationale or reasoning behind program success. In contrast, a program theory, while not containing prespecified categories such as inputs, activities, outputs, or outcomes, is intended to present the details of that reasoning. Bickman (1987) notes that program theory should clarify the relationship between the problem that the program assumes the student or client has and the program actions. Therefore, a program theory should begin with describing the client, or the assumptions the program has about the client, before the client begins the program. Logic models typically start a step after that, with program input. Another difference is simply that the organizations and people who write about logic models are different from those who write about program theory and theory-based evaluation. Using Program Theory in Evaluation. The central element of theory-based evaluation approaches involves developing the theory for why the program should achieve its desired outcomes. It is this stage that distinguishes theory-based evaluations from other approaches. Chen (1990) differentiates between two models for developing program theory: (a) the stakeholder approach, in which the evaluator works with stakeholders, typically key program people, to discover their reasoning or underlying assumptions for program success; and (b) the social science approach, in which evaluators make use of their own knowledge both of the program and of the social science theory and research to develop a model. Both Bickman and Chen note that developing program theory with stakeholders alone can often be problematic. Stakeholders may not know the program theory or, as Bickman writes, their theory may be “a vague notion or hunch” or “may be nothing more than a few simple assumptions about why the program should work” (1987, p. 6) because they are not well trained in social science theory or research. Or they may be purposefully obtuse or vague about program theory in order to obtain political support or funding or to avoid alienating an important group. Weiss, more recently, argues that improving the quality of program theory is one of the key challenges to evaluators. She notes that program managers or policymakers may or may not be able to develop a good-quality program theory. Program theories, she notes, must articulate the causal linkages between program actions and goals; if they do not, the theory is simply a model for implementation, a description of program processes. And, like Bickman, she expresses concern with relying too much on program stakeholders, planners, and practitioners to articulate that theory. She observes, “Many of these theories are elementary, simplistic, partial, or even outright wrong” (Weiss, 1997, p. 78), and emphasizes the need for evaluators to combine stakeholder input with social science research to build sound program theory. The process of developing program theory should, therefore, rely on a combination of input from stakeholders, theories and research from relevant social science studies, and the evaluators’ knowledge and expertise. Donaldson (2007), in his book on theory-based evaluation, decribes the steps for theory development

Fitzpatrick, Jody L.. Program Evaluation (p. 162). Pearson Education. Kindle Edition.

more precisely than early writers, based on his experience in using theory-based evaluation in several large projects. His steps reflect this balance: 1. Engage relevant stakeholders. The evaluator talks with as many representatives as possible from different constituencies to get their views on the program, its intended long-term outcomes, and the process the program uses to achieve those outcomes. 2. Develop a first draft of program theory. This step is undertaken by the evaluator or the evaluation team. 3. Present the draft to stakeholders for further discussion, reaction, and input. 4. Conduct a plausibility check. Now, the evaluators consult existing research and evaluations that are relevant to the program theory to assess the plausibility of each link. Does research suggest that the link could occur as planned? That the program action could lead to the intended outcome? 5. Communicate these findings to key stakeholders and revise the program theory as needed. Donaldson observes that the plausibility check may suggest that serious program changes are needed or that stakeholders have been overly optimistic about the outcomes that might be achieved. Evaluators present research findings to stakeholders and work with them to revise the program theory and/or the program itself so that the model accurately represents what will be done and what can be achieved. 6. Probe arrows for model specificity. As in step 4, the evaluators take the lead in examining the program theory “at a deeper level of detail.” Donaldson indicates that at this stage the evaluators’ focus is typically on critical links and discussion of details such as the length of time required for the outcome to occur and the nature of the process. The purpose, here, is for the evaluation team to ensure that they have an accurate, in-depth understanding of the program as it is intended to proceed. Such understandings can influence, for example, when evaluators collect data on outcomes and the type of data they collect. 7. Finalize program impact theory. The stakeholders have the final say in approving the model that will serve as the foundation for studying the program. Donaldson notes that he prefers relatively parsimonious models, as do some stakeholders, but others prefer models with more detail (Donaldson, 2007, pp. 33–39). This theory-development process precedes any decisions about the evaluation. In fact, Donaldson indicates that stakeholders often want to move into thinking about the evaluation and the implications of the program theory during the process of discussing the evaluation. However, it is important for the program theory to be fully developed before moving into identifying the evaluation questions to be answered or the methods to be used to address such questions. The program theory, its key principles, should not be influenced by how the evaluation will be conducted—for example, by worrying about how certain linkages would be tested—but should instead reflect a true picture of what the program is intended to do and how it will do it.

Fitzpatrick, Jody L.. Program Evaluation (p. 163). Pearson Education. Kindle Edition.

The second phase of theory-driven evaluation is similar to that in most other evaluations. Working with stakeholders, the evaluators identify the key questions to be answered in the evaluation and the appropriate designs and methods for answering those questions. Although the theory-based evaluation approach has been considered a more quantitative approach because its proponents often come from quantitative areas, the approach does not prescribe or proscribe specific methods or designs. Often the emphasis is on testing the program model, that is, on questions of causality. The constructs identified in the program theory provide guidance as to what to measure and when to measure it (Lipsey, 1993). As Donaldson indicates, the selection of the evaluation questions to be addressed depend very much on the stage of the program (young versus mature enough to examine long-term outcomes) and what the stakeholders hope to learn. The development of program theory helps the evaluator to learn more about the program and its assumptions and, therefore, provides the evaluator with critical information that can be used throughout the evaluation. This includes determining what to study at different stages, identifying the constructs of interest, interpreting results, and making recommendations. A typical model for theory-based evaluation would be to first study program implementation, focusing on whether key elements of the program theory are, in fact, delivered as planned. If so, the evaluator can then go on to study program outcomes knowing that this evaluation will be a test of program theory. If the program fails, it will mean that the theory does not work, at least with this client group in this context. But if the program is not implemented as planned, the evaluator may recommend changing the implementation to match the model, discarding the model as not feasible in this context, or trying some other model. In any case, the evaluation would not move on to measure outcomes because the program theory had not, in fact, been implemented. If outcomes were studied and success was achieved, the implementation study would demonstrate how the program that was delivered differed from the program theory. This modified program delivery might then become the standard model or program theory for the future. Thus, the theory-based or theory-driven approach overcomes some of the failures of the objectives-oriented approach. It provides the evaluator with a way to look inside the black box and better understand what is happening between the time a student or client begins a program and when he or she concludes the program. With this information, theory-based evaluators argue, they can better test and determine the reasons for a program’s success or failure. How Program-Oriented Evaluation Approaches Have Been Used The objectives-oriented approach has dominated the thinking and development of evaluation since the 1930s, both in the United States and elsewhere (Madaus & Stufflebeam, 1989). Its straightforward procedure of using objectives to determine a program’s success or failure and to serve as a foundation for program

Fitzpatrick, Jody L.. Program Evaluation (p. 164). Pearson Education. Kindle Edition.

improvements, maintenance, or termination of program activities has proved an attractive prototype. In education, the approach influenced the development of taxonomies of educational objectives (Bloom, Hastings, & Masia, 1971), the criterion-referenced testing movement of the 1960s and 1970s, and today’s standards-based movement. As we noted in Chapter 2 in describing current trends in evaluation, the focus of evaluation today is on measuring outcomes; in schools, that takes the form of educational standards. No Child Left Behind (NCLB), the legislation passed by Congress in 2001, required all states that had not already done so to develop rigorous content standards for learning and tests to be given annually to measure accomplishment of those standards. This objectives-oriented means of evaluation now dominates K–12 education. Annual measurable objectives (AMOs) are used as a means of measuring progress toward the standards. The objectives-oriented tradition has also influenced evaluation and management practices from the 1960s when Robert McNamara and the Rand Corporation brought Planning, Programming, and Budgeting Systems (PPBS) to the U.S. Defense Department, Management by Objectives (MBO), outcome monitoring (Affholter, 1994), and the Government Performance and Results Act (GPRA) (National Performance Review, 1993). Today, the Program Assessment Rating Tool (PART) has replaced GPRA (Office of Management and Budget, 2004), and performance monitoring has become a mainstay of many government management systems. Dahler-Larson (2006), in commenting on trends in evaluation today, sees performance monitoring as today’s version of objectives-oriented or goal-oriented evaluation approaches. He observes that in-depth, goal-oriented evaluations have been replaced by monitoring of performance indicators. Performance monitoring systems are used by managers to monitor progress toward results. The systems may look at outputs, productivity, efficiency, service quality, customer satisfaction, or outcomes, but the focus is on the program and on the results (Positer, 2004). Logic models are sometimes used to identify the critical elements in the system that should be monitored, but to make it feasible to monitor these elements in an ongoing way requires compromises. So, data for performance monitoring systems tend to be solely quantitative and cost-effective to collect. Rarely are true long-term outcomes measured in an ongoing performance monitoring system if those outcomes are at all complex. Although versions of objectives-oriented approaches continue to be popular with many government agencies and foundations for intensive evaluations, theory-based evaluation approaches are often the approach of choice by professional evaluators, particularly those with a more scientific bent. Many government funding agencies, particularly at the federal level in the United States, require programs to articulate their program theory or logic model. In addition, foundations such as the Aspen Institute, with their work on comprehensive community initiatives designed to have an impact at the community level, have pursued theory-based evaluations as a way to help them articulate the theory of complex programs and, then, to evaluate that theory as implemented (Weiss, 1995).

Fitzpatrick, Jody L.. Program Evaluation (p. 165). Pearson Education. Kindle Edition.

Strengths and Limitations of Program-Oriented Evaluation Approaches Probably the greatest strength and appeal of the objectives-oriented approach lies in its simplicity. It is easily understood, is easy to follow and implement, and produces information that program directors generally agree is relevant to their mission. This approach has caused program directors to reflect about their intentions and to clarify formerly ambiguous generalities about intended outcomes. Discussions of appropriate objectives with the community being served have given objectives-oriented evaluation the appeal of face validity—the program is, after all, merely being held accountable for what its designers said it was going to accomplish, and that is obviously legitimate. Useful as this approach to evaluation seems to its many adherents, it has many drawbacks. Principal ones include the single-minded focus on objectives and their measurement. The focus on objectives can cause evaluators to ignore other important outcomes of the program, both beneficial and detrimental, and if the evaluation draws final conclusions, the judgment of the program may be seriously incomplete. The evaluator may focus on the objectives like a horse wearing blinders. He or she may ignore the road to the right that drops off precipitously or the absolutely breathtaking view to the left in its efforts to look at (and measure) only the goal to be reached—the objectives that have been articulated. The objectives-oriented approach also neglects program description, the need to gain an understanding of the context in which the program operates and the effects of that context on program success or failure. Finally, evaluators using this approach may neglect their role in considering the value of the objectives themselves. Are these objectives, in fact, important ones for the program and its clientele? The ethical principles of evaluation, in Guiding Principle E, require the evaluator to consider “not only the immediate operations and outcomes of whatever is being evaluated but also its broad assumptions, implications, and potential side effects” (American Evaluation Association, 1995, p. 25). The objectives-oriented approach can appear seductively simple to novice evaluators who are only partially familiar with its philosophical and practical difficulties. Choices are involved in deciding which objectives to evaluate and how to interpret success or failure in each. In today’s standards-based environment, evaluators have little authority over state-required tests to measure standards. However, evaluations concerning standards should help stakeholders consider which standards are appropriate for their students and what levels, when reached, will be considered a success. If test items do not fully reflect the objectives of a particular school or district, results from alternative measures can be provided as support for achievement to parents and community leaders. Such evaluations can open up discussions about standards and goals in education for different communities. Evaluations that make use of logic models or program theory to learn more about the program and to shed light on what to evaluate and the appropriate means for doing so obviously overcome some of the criticisms to the objectives-oriented

Fitzpatrick, Jody L.. Program Evaluation (p. 166). Pearson Education. Kindle Edition.

approach. Evaluators engage the stakeholders in dialogue so that they can learn more about the program, begin to develop a relationship with the stakeholders, and thereby gain a better understanding of what the evaluation might do. By the time a good theory-based evaluator moves to planning the evaluation, she should have a much better understanding of the values and concerns of the stakeholders regarding the program and the evaluation. The requirement for dialogue, as well as achieving an understanding and a clear articulation of the reasoning behind the program, is an obvious advantage of logic models and theory-based evaluations. These evaluators have focused on the program to identify and formulate the questions the evaluation will address, the timing of data collection, and the appropriate methods to be used. Some theory-based evaluators may feel obligated to evaluate the entire program theory, but that comprehensive focus is not the point of the process. The point is to understand the program from beginning to end and, then, to choose the appropriate links or components to evaluate given the stage of the program and the information needs of the stakeholders. Nevertheless, theory-based evaluators, like objectives-oriented evaluators, may struggle to see beyond their self-imposed blinders. Of course, theory-based evaluators are likely to focus on the theory and may ignore unintended program actions, links, outputs, or outcomes that merit attention. Further, their desire to test the theory as a whole may prompt them to neglect values or information needs of stakeholders. (See Fitzpatrick & Donaldson [2002] as Donaldson describes the pressures to evaluate the program theory for a training program for the unemployed that had been validated in Detroit, but now moved to a different population in California.) Theory-based approaches are also criticized for oversimplifying the complexity of program delivery and context (Pawson, 2003). The reality of delivering programs is complex and, certainly, program theories simplify that complexity. But that is the purpose of theories or models—to reduce the messiness and complexity of actual program delivery to a parsimonious model that identifies the key assumptions or critical elements necessary for program success. In that way, the model helps the evaluator identify the most important elements or linkages to evaluate. Nevertheless, such reductionism does fail to convey the complexity of the real program and describing that complexity can also be an important role of evaluation. Oversimplification often leads citizens and policymakers alike to fail to understand how difficult, and costly, it is for programs or schools to achieve stated goals. Dahler-Larson calls for the need “to bridge the gap between conventions for relatively simple representations of causal models, on the one hand, and complex reality on the other” (2006, p. 152). He argues that theory-based evaluators should attempt to develop “different representations of program theories which will be fruitful in various ways depending on the purpose of the evaluation” (2006, p. 152). We close our discussion of limitations of these types of evaluation approaches by briefly describing Scriven’s goal-free evaluation. His concerns with the limitations of objectives-oriented approaches led him to develop his now widely known proposals for goal-free evaluation (1972), still discussed today to make evaluators aware of the bias that a focus on particular program elements can impose. Although intentionally the opposite of objectives-oriented approaches, it seems logical to discuss this proposal here.

Fitzpatrick, Jody L.. Program Evaluation (p. 167). Pearson Education. Kindle Edition.

Goal-Free Evaluation The rationale for goal-free evaluation can be summarized as follows: First, goals should not be taken as given. Goals, he argues, are generally little more than rhetoric and seldom reveal the real objectives of the project or changes in intent. In addition, many important program outcomes are not included in the list of original program goals or objectives. Scriven (1972) believes that the most important function of goal-free evaluation is to reduce the bias that occurs from knowing program goals and, thus, to increase objectivity in judging the program as a whole. In objectives-oriented evaluation, an evaluator is told the goals of the program and is, therefore, immediately limited in her perceptions—the goals act like blinders, causing her to miss important outcomes not directly related to those goals. For example, suppose an evaluator is told that the goals of a dropout rehabilitation program are to (1) bring school dropouts into a vocational training program, (2) train them in productive vocations, and (3) place them in stable jobs. She may spend all her time designing and applying measures to look at such things as how many dropouts have been recruited into the program and how many have been placed and have remained in paying jobs. These are worthwhile goals, and the program may be successful on all these counts. But what about the fact that the crime rate of others (non-dropouts) who are receiving employment training has tripled since the dropouts were brought into the vocational training program? Indeed, a hidden curriculum seems to have sprung up: stripping cars. This negative side effect is much more likely to be picked up by the goal-free evaluator than by the objectives-oriented evaluator working behind her built-in blinders. The following are major characteristics of goal-free evaluation: • The evaluator purposefully avoids becoming aware of the program goals. • Predetermined goals are not permitted to narrow the focus of the evaluation study. • Goal-free evaluation focuses on actual outcomes rather than intended program outcomes. • The goal-free evaluator has minimal contact with the program manager and staff. • Goal-free evaluation increases the likelihood that unanticipated side effects will be noted. It might be helpful to point out that objectives-oriented and goal-free evaluations are not mutually exclusive. Indeed, they supplement one another. The internal staff evaluator, of necessity, conducts a goal-directed evaluation. She can hardly hope to avoid knowing the goals of the program, and it would be unwise to ignore them even if she could. Program managers obviously need to know how well the program is meeting its goals, and the internal evaluator uses goal-directed evaluation to provide administrators with that information. At the same time, it is important to know how others judge the program, not only on the basis of how well it does what it is supposed to do, but also on the basis of what it does in all

Fitzpatrick, Jody L.. Program Evaluation (p. 168). Pearson Education. Kindle Edition.

areas on all its outcomes, intended or not. This is a task for the external goal-free evaluator who knows nothing of the program goals. Thus, goal-directed evaluation and goal-free evaluation can work well together. And, while the major share of a program’s evaluation resources should not go to goal-free evaluation, it is unfortunate when all resources go to goal-directed evaluation on a program when the stated goals do not even begin to include all of the important outcomes. Major Concepts and Theories 1. The objectives-oriented evaluation approach was one of the first approaches to evaluation and is still commonly used today. Some of its present-day forms are standardsbased testing and accountability in education and performance monitoring systems used in many government programs. 2. The objectives-oriented approach focuses on articulating the objectives of a program and collecting data to determine the extent to which they are achieved. Ralph Tyler and Malcolm Provus were early advocates of different facets of objectives-oriented evaluation. 3. Logic models are often used by program managers and evaluators today to link program inputs, processes, outputs, and outcomes and can serve as the foundation for making decisions about program or evaluation activities. 4. Theory-based or theory-driven evaluation approaches make use of social science theories and research relevant to the program and stakeholders’ assumptions about why the program should work, to develop program theories. These program theories then serve as a foundation for selecting evaluation questions and making decisions about what to study and when to collect data on it. Theory-based evaluations are a frequently used approach in evaluation today. 5. The goal-free evaluation approach was proposed primarily to identify any unanticipated side effects of a program that a goal-directed or objectives-oriented evaluation might miss because of the focus on the intended program outcomes rather than on the actual program outcomes.

Fitzpatrick, Jody L.. Program Evaluation (p. 169). Pearson Education. Kindle Edition.