Program Evaluation on Early Head Start
What is Evaluation Research?
Application of social research techniques to assess the conceptualization, design, implementation, and utility of social intervention programs
Allows us to see if what we want a program to accomplish is being accomplished
In order to know whether social programs work, or how they work, we have to evaluate them systematically and fairly, regardless of whether we personally like the program
Systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming (Patton, 1997:23).
1
Why do We Need Evaluation Research?
Evaluation is not a data collection method, but uses multiple methods learned in previous chapters
Surveys may be used to assess need for services program provides
Experimental and quasi-experimental designs may determine effects of the intervention (“treatment”) on the desired outcome
Qualitative research techniques, participant observation to understand how clients experience the programs and whether/how services are carried out
Evaluations can address accountability
Most programs affiliated with the criminal justice system or that serve victims or offenders are publicly funded. As such, they are often held accountable for spending funds appropriately and providing services for which funds are appropriated
Evaluators usually look at the program as a “system” and conduct an evaluation using a systems approach
2
Standards for Evaluation Research
Utility: intended to ensure that an evaluation will serve the practical information needs of intended users
Feasibility: intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal
Propriety: intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results
Accuracy: intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated
A Model of Program Evaluation
Resources, managers, staff, funders, clients
Services/
activities that the program includes
Services actually delivered
Impact of program on cases (clients)
Individuals/groups who have a basis of interest in program
Information about program process, outputs, outcomes that is available to any program inputs
4
Adapted from Martin & Kettner, 1996. Used with permission.
From text:
Feedback also broadens this loop to include connections to parties outside of the program itself. A funding agency or political authority may mandate the research, outside experts may be brought in to conduct the research, and the evaluation research findings may be released to the public, or at least funders, in a formal report.
Example of Evaluation Model: D.A.R.E.
Inputs
Money, D.A.R.E. officers, students
Program process
Activities and instruction that D.A.R.E. officers provide to students
Outputs
Number of hours of instruction received by students, educational materials delivered to students
Outcomes
Substance abuse by students after receiving D.A.R.E. services
Feedback
Reports produced for funding agencies regarding services provided or outcomes
Stakeholders
Police departments, D.A.R.E. officers, students, parents, school officials, elected officials, community
5
Example of Program Theory
Alcohol Treatment Program
6
Researcher or Stakeholder Orientation
How much influence should clients/stakeholders have on a researcher’s methods?
Stakeholder approaches – encourages researchers to be responsive to stakeholders
Utilization-focused evaluation – evaluator forms task force for stakeholders who help shape the project
Action or participatory research – program participants are engaged as co-researchers in helping design and conduct the project
Appreciate inquiry – eliminates professional researchers in favor of structured dialog with program participants
7
Researcher or Stakeholder Orientation
Social Science approaches – emphasize the importance of researcher expertise in the development of unbiased evaluation
Integrative approaches – attempt to be a compromise between stakeholder wishes and researcher concerns
Experimental Design in Evaluation
Random assignment to treatment and control conditions helps to control for many extraneous influences that could create spurious relationships and outcomes
However, experiments are not always feasible
Program design/structure is generally out of evaluator’s control
Quasi-experimental designs are most powerful alternative
Quantitative & Qualitative Methods
Evaluation research is largely quantitative in nature
However, qualitative methods add depth, detail, nuance to quantitative approaches
allows researchers to see how individuals react with treatment
Ethical Issues in Program Evaluation
Most issues concern outcome and impact studies
Issues are somewhat different from other research
Issues related to clients
Distribution of benefits
Preserving confidentiality is difficult
Participant burden may be considerable
Issues related to evaluation design and conduct
Research designs may be shaped by politics
Intent of evaluation may be political
11
Ethical Issues, cont’d
Issues related to use of evaluation results
Sharing results
only with sponsors?
with clients?
with general public?
Scientific credibility
If design is poor, the results should not be used, but if client controls use, the researcher may not be able to avoid their use