Law This is a 2 part assignment due on 08/03 and 08/04

1122924 - SAGE Publications, Inc. (US) ©

What is an evaluator’s responsibility with regard to evaluation focus? To what extent is an evaluator only obligated to evaluate the objectives of a program as stated by program leadership? To what extent does an evaluator have a responsibility to study other potential impacts of a program beyond the intended goals?

In the Real World … Revisiting the Cambridge-Somerville Youth Study (CSYS): Ideology. The CSYS was introduced in Chapter 2. The purpose of CSYS (Cabot, 1940) was both to prevent juvenile delinquency among boys as well as to study the effectiveness of juvenile delinquency interventions. It is revisited here to illustrate how ideology relates to design, role, and methods.

While it is impossible to truly know what Cabot’s ideology was with regard to evaluation, we can surmise from descriptions of the study where his beliefs might fall along each calibrator continuum. With regard to the design calibrators, the design indicates he favored the medical model of research design and it does not appear that findings were used for program improvement. With regard to role calibrators, the evaluators were external and do not appear to have had much involvement with the program beyond data collection. There is no evidence that CSYS was a capacity-building evaluation. With regard to methods calibrators, while some qualitative methods may have been used, the predominant methods appear to have been quantitative and focused on the objectives of the CSYS.

What if the CSYS evaluators had had a different ideology? Consider each scenario and identify ways that the change may have affected the findings from the study.

SCENARIO 1: Suppose the evaluators chose to forgo a control group and included all youth in the program.

SCENARIO 2: Suppose findings were used throughout the program to improve services for the children involved.

SCENARIO 3: Suppose the evaluator was someone internal to the program.

SCENARIO 4: Suppose the evaluators worked closely with stakeholders throughout the program, building processes for them to collect and analyze their own data.

SCENARIO 5: Suppose the youth were observed and data were collected from these observations, instead of from instruments designed to measure behavior.

SCENARIO 6: Suppose the evaluators did not use the stated objectives of the program to drive the study, but instead examined any potential outcome of the program.

Ideally, the relationship between evaluators and stakeholders would be a partnership, such that decisions regarding the focus of an evaluation can be jointly determined. See “In The Real World” for a discussion of how ideology may have influenced the Cambridge-Somerville Youth Study.

4.2.4 Calibrators and Ideology

The six calibrators discussed above are provided for you as areas to reflect upon as you develop your own ideology around evaluation. Figure 4.2 includes a graphical representation of how these calibrators shape ideology—and how ideology, in turn, guides our choices regarding evaluation designs and approaches. On the right side of the diagram are additional influencers that affect our use of evaluation designs and approaches. For instance, resources and context can constrain the types of research designs that might be employed. Evaluator skills and experiences, as well as the degree of access we have to stakeholders, influence the approaches that we are able to take with regard to a particular evaluation. The following two sections will address evaluation design and approaches. The section on designs focuses on how they were shaped by early evaluators and only includes a brief explanation of their purpose (Chapter 8 provides detailed information on evaluation design). Evaluation approaches describe some common evaluation approaches in the field and who contributed to the development of each approach.

1122924 - SAGE Publications, Inc. (US) ©

Description

Figure 4.2 Evaluation Ideology Calibrators and Influencers

Quick Check 1. How does an evaluator’s ideology affect their choice of evaluation designs and approaches? 2. Do you think an evaluator should build evaluation capacity among the staff of the program they are evaluating? Why or why not? 3. If you had limited resources for an evaluation, would you use qualitative or quantitative methods? Explain your reasoning. 4. Explain the six calibrators that affect an evaluator’s ideology. What are your thoughts on each calibrator? Do you have strong preferences regarding

any of the calibrators?

4.3 EVALUATION DESIGN Ideology influences the evaluation designs we choose to use. In particular, our philosophy regarding the design and methods calibrators drives the overall structure of our evaluation. While there are additional factors that affect and constrain choices regarding evaluation design and methods, our underlying ideology shapes the extent to which we view different research designs as strong or weak. In this section, major contributors to evaluation design will be discussed, within the framework of the evaluation designs themselves. Chapter 8 will explore evaluation design in more detail.

4.3.1 Experimental Designs

Donald Campbell.

Donald Campbell was one of the most critical pioneers in the call for social experimentation in the field of evaluation (Rossi, Lipsey, & Henry, 2018). His groundbreaking work pioneered the application of the experimental model used in psychological research to the evaluation field (Christie & Alkin, 2013). The experimental model of research includes random assignment of subjects to a program/intervention or to a control condition. Campbell’s perspective was that decisions about policy and programs should be made on the basis of experimental research. His perspectives are detailed in his 1969 article “Reforms as Experiments.” For over half a century, his work has guided social science researchers and evaluators on how to conduct rigorous research aimed at establishing causal inference. Causal inference is the ability of evaluators to claim that the program they are evaluating is responsible for the outcomes they measured. One of the most influential books in the field was written by Campbell and his coauthor Julian Stanley. Campbell and Stanley’s (1963) seminal work Experimental and Quasi-Experimental Designs for Research is one that every researcher and evaluator should have in their library. They detail design considerations for randomized controlled experiments, quasi-experiments, and nonexperimental studies. Their work has had a lasting impact on the field of evaluation and has facilitated the use of both experimental and quasi-experimental designs (Shadish & Luellen, 2013). It began a shift in the field, which led to randomized experiments being considered the “gold standard” design for establishing causal inference (Christie & Alkin, 2013).

Causal inference: the ability of an evaluator to claim that the program they are evaluating is responsible for the outcomes they measured; causality can be claimed with experimental designs.

Robert Boruch.

Similar to Campbell’s legacy, Robert Boruch has been instrumental in furthering the use of randomized experiments in the evaluation field. One of Boruch’s (1997) most influential works, Randomized Experiments for Planning and Evaluation, provides a practical guide to randomized experiments for evaluators. Boruch is a

1122924 - SAGE Publications, Inc. (US) ©

strong proponent of using randomized experiments, promotes them as the most effective method for evaluating a program’s effects, and argues that any program can employ randomized experiments to determine their effectiveness. As stated by Christie and Alkin (2013),