human resource management

Guide to Program Evaluation Getting Started

What is Evaluation; Types of Evaluation Activities; Benefits of Evaluation; Evaluation Concerns; Evaluation Constraints

Planning the Evaluation

Are You Ready for Evaluation; Working With an Outside Evaluator; Developing an Evaluation Plan; Developing and Working With Program Logic Models

Assessing Program Performance

Identifying Goals and Objectives; Measuring Activities and Outputs (Process Evaluation); Measuring Outcomes (Impact Evaluation); Establishing the "Activities-Outcomes" Connection (Evaluation Experiments)

Data Collection

New or Existing Data; Using Existing Data; Using New Data; Other Considerations

Reporting and Using Evaluation Results

Reviewing Evaluation Findings With Stakeholders; Writing a Final Report; Using Evaluation Results

Getting Started What Is Evaluation? Evaluation is a systematic, objective process for determining the success of a policy or program. It addresses questions about whether and to what extent the program is achieving its goals and objectives.

Learn More... A Typology of Evaluation Levels (Office of Juvenile Justice and Delinquency Prevention) An Overview of Education Evaluation (Department of Education) Developing a Strategy for Evaluation (National Institute of Justice) Identifying Effective Criminal Justice Programs: Guidelines and Criteria for the Nomination of Effective Programs (Bureau of Justice Assistance) Underlying Premise of Assessment and Evaluation (Bureau of Justice Assistance)

Types of Evaluation Activities Program Monitoring Program monitoring involves the ongoing collection of information to determine if programs are operating according to plan. Monitoring provides ongoing information on program implementation and functioning.

Learn More... Basic Monitoring and Comparative Monitoring (Office of Juvenile Justice and Delinquency Prevention) Install a Monitoring System to Provide Continuous Feedback (National Institute of Justice) Selecting an Evaluation Design (National Institute of Justice)

Performance Measurement/Assessment Program measurement or assessment involves the ongoing collection of information on whether a program is meeting its goals and objectives. Performance measures can address project activities, services delivered, and the products of those services.

Learn More... Introduction (Fairfax County Department of Management and Budget, pp. 4-7) Types of Program Performance Assessment (Government Accounting Office) Using Indicators Effectively (Vera Institute of Justice, pp. 2-15)

Process or Implementation Evaluation Process evaluation focuses on program implementation and operation. A process evaluation can answer questions regarding program effort; identify processes or procedures used to carry out the functions of the program; and address program operation and performance.

Learn More... Documenting and Analyzing Program Installation and Operations (Department of Education) Implement a Process Evaluation to Document What is Done, When, By Whom, To Whom (National Institute of Justice) Process Evaluation (Bureau of Justice Assistance)

Outcome or Impact Evaluation This type of evaluation focuses on program success and accomplishments. These evaluations answer questions regarding program effectiveness; address whether a program is achieving its goals and objectives; and examine unintended consequences, both positive and negative.

Learn More... Basic Outcome Evaluation and Comparative Outcome Evaluation (Office of Juvenile Justice and Delinquency Prevention) Impact Evaluation (Bureau of Justice Assistance) Observing Behavioral Outcomes and Attributing Changes to the Program (Department of Education)

Cost-Effectiveness and Cost-Benefit Assessment These assessments focus on using the results from a sound program evaluation to assess how effective the program is relative to other program alternatives in terms of cost. Cost-benefit analysis does not answer the question of whether the program works; instead, it uses the results of evaluations to compare the economic value of the outcomes and costs of one program with another.

Learn More... Comparative Costs and Benefits of Programs to Reduce Crime, Version 4.0 (Washington State Institute for Public Policy) Distinguishing Cost-Benefit Analysis from Program Evaluation (Justice Research and Statistics Association, p. 6)

Benefits of Evaluation Programs that participate in evaluations will obtain objective information about their performance and how it can be improved. Evaluation can provide objective evidence that a program is effective, demonstrating positive outcomes to funding sources and the community. It can help improve program effectiveness and can create opportunities for programs to share information with other similar programs and agencies. Programs can use evaluation findings in a number of ways. For example, the program, to make a case for continued funding and to attract new funding sources, can use evidence of program success. A well-executed evaluation will point out areas in which the program can improve its operations. Also, sharing the results of evaluation has benefits to others outside of the program seeking to replicate justice interventions that work.

Learn More... Benefits of Evaluation (Department of Housing and Urban Development) Introduction (National Institute of Justice)

Evaluation Concerns Program managers and staff can sometimes be reluctant participants in the evaluation process. Below are some frequently-expressed concerns about program evaluation and responses to those concerns. Concern: Evaluation draws resources away from program services.

Response: Without evaluation, how do you know that the services being provided are effective? Program managers can explore options for obtaining evaluation services inexpensively.

Concern: Evaluation increases the burden on program staff.

Response: Evaluators can often implement changes to current client data collection procedures, resulting in little additional effort on the part of program staff. To reduce the burden and increase "buy-in," program staff should be involved in designing evaluation instruments and interpreting evaluation findings.

Concern: Evaluation is too complicated for program managers and staff to

understand.

Response: An evaluation does not need to have the most rigorous scientific method, design, and analysis to be considered useful and valuable. Evaluation findings should be expressed in a manner that can be readily understood and used by program managers, staff, and other stakeholders.

Concern: Evaluation may produce negative results that will harm the program.

Response: A good evaluation will point out both program strengths and weaknesses. No reputable evaluator will willingly participate in an evaluation designed to harm a program.

Learn More... Common Concerns about Evaluation (Department of Housing and Urban Development) Guide to Frugal Evaluation for Criminal Justice (National Institute of Justice, Chapter 6)

Evaluation Constraints Every evaluation is carried out under certain constraints or limitations. These constraints should be identified as part of the planning process for the evaluation. Two major evaluation constraints are time and cost. Evaluation results that are not timely are not useful to program managers and funding agencies. When evaluation information is needed quickly, the evaluation must address fewer questions. Similarly, the financial resources available for the evaluation help to determine its scope. The strengths and weaknesses of various evaluation approaches should be considered while keeping in mind the level of resources available.

Learn More... Considering the Evaluation's Constraints (General Accounting Office)

Planning the Evaluation

Are You Ready for Evaluation? Not all programs are ready to be evaluated; that is, they are not able to provide information or otherwise fully participate in the evaluation. To determine whether a program is ready for evaluation, evaluators have developed the process of "evaluability assessment." An evaluability assessment, undertaken prior to an evaluation, is designed to address the question of whether the program can participate fully in an evaluation. Some examples of questions that can be addressed in an evaluability assessment are listed below.

Is there a formal program design or model in place?

Programs must be able to document their goals and objectives, and the strategies they employ to achieve those goals and objectives.

Is the program design or model a sound one?

If program goals are unrealistic or strategies are not based in theory or prior evidence, or if program managers cannot explain how the activities and services they provide are expected to lead to the program’s desired outcomes, then evaluation is not a good investment.

Can the program participate in the evaluation?

Evaluations require data and information. If the program does not collect data, and has no capacity to generate data, then the evaluation will not be successful.

Example of an Evaluability Assessment The Youth Monitoring Program

Learn More... Assessing Readiness for Evaluation (National Institute of Justice)

Determining Whether to Evaluate at All (National Institute of Justice)

Evaluability Assessment: Examining the Readiness of a Program for Evaluation (Justice Research and Statistics Association)

Selecting Critical Programs (Department of Housing and Urban Development)

Time Frame for Evaluation (Office of Juvenile Justice and Delinquency Prevention)

Working With an Outside Evaluator One of the first issues that programs need to address when considering an evaluation is whether to use an evaluation expert, and whether that person can be in-house (if such expertise exists) or outside of the agency or program being evaluated. If funds are available, a trained and experienced evaluator can be of great assistance to a program throughout the evaluation process. If in-house expertise is available, the advantages and disadvantages of using this person or an external evaluator must be weighed.

Regardless of whether the evaluator is internal or external to the agency being evaluated, finding a qualified evaluator is essential. A qualified evaluator should be experienced in evaluating similar programs; should try to balance the needs and concerns of a variety of decision-makers, including the program managers, with issues related to the objectivity of the evaluation; and should be able to communicate with a wide variety of individuals who have an interest in the results of their work.

Learn More... Building Evaluation into a Program RFP and Preparing an Evaluation RFP (Office of Juvenile Justice and Delinquency Prevention)

Choosing an Evaluator (Office of Juvenile Justice and Delinquency Prevention)

Conducting Evaluations In-House or Under Contract (National Institute of Justice)

Hiring and Working with an Evaluator (Justice Research and Statistics Association)

Who Should Conduct Your Evaluation? (Department of Housing and Urban Development)

Developing an Evaluation Plan Once you have determined that you are ready for evaluation and have decided who will conduct the evaluation, the next step is to develop an evaluation plan. An evaluation plan is a description of the evaluation process. Some of the key elements that should be addressed in the evaluation plan include: who is the target audience for the evaluation; what evaluation questions will be asked; how the evaluation will be designed; what data will be collected, how and by whom; and what final products will be produced.

The evaluation plan should detail the roles that various individuals will play in the evaluation process; these individuals include the evaluator, the program manager, staff, clients, and any other stakeholders. Opportunities for preliminary review of findings and conclusions should be built into the plan.

Learn More... Developing an Evaluation Plan (Department of Housing and Urban Development).

Developing an Evaluation Plan (Justice Research and Statistics Association, p. 7)

Steps in Planning Evaluations (U.S. Department of Education)

Developing and Working with Program Logic Models While there are many forms, logic models specify relationships among program goals, objectives, activities, outputs, and outcomes. Logic models are often developed using graphics or schematics and allow the program manager or evaluator to clearly indicate the theoretical connections among program components: that is, how program activities will lead to the accomplishment of objectives, and how accomplishing objectives will lead to the fulfillment of goals. In addition, logic models used for evaluation include the measures that will be used to determine if activities were carried out as planned (output measures) and if the program's objectives have been met (outcome measures).

Why Use a Logic Model? Logic models are a useful tool for program development and evaluation planning for several reasons:

• They serve as a format for clarifying what the program hopes to achieve;

• They are an effective way to monitor program activities; • They can be used for either performance measurement or evaluation; • They help programs stay on track as well as plan for the future; and • They are an excellent way to document what a program intends to do and what it is

actually doing.

Learn More About What a Logic Model Is and Why To Use It Developing a Logic Model (The Urban Institute)

Developing and Using a Logic Model (The Urban Institute)

A Guide on Logic Model Development for CDC’s Prevention Research Centers (Sundra, Scherer, and Anderson)

Logic Model for Program Planning and Evaluation (University of Idaho-Extension)

How to Develop a Logic Model Developing a logic model requires a program planner to think systematically about what they want their program to accomplish and how it will be done. The logic model should illustrate the linkages of among the elements of the program including the goal, objectives, resources, activities, process measures, outcomes, outcome measures, and external factors.

Logic Model Schematic The following logic model format and discussion was developed by the Juvenile Justice Evaluation Center (JJEC) and maintained online by the Justice Research and Statistics Association (www.jrsa.org) from 1998 to 2007.

The following discussion explains the interconnectedness among the elements of the logic model.

At the top of the logic model example is a goal which represents a broad, measurable statement that describing the desired long-term impact of the program. Knowing the expected long-term achievements a program is expected to make will help in determining what the overall program goal should be. Sometimes goals are not always achieved during the operation of a program. However, evaluators or program planners should continually re-visit the program's goals during program planning.

An objective is a more specific, measurable concept focused on the immediate or direct outcomes of the program that support accomplishment of your goal. Unlike goals, objectives should be achieved during the program. A clear objective will provide information concerning the direction, target, and timeframe for the program. Knowing the difference your program will make, who will be impacted, and when will be helpful when developing focused objectives for your program.

Resources or inputs can include staff, facilities, materials, or funds, etc--anything invested in the program to accomplish the work that must be done. The resources needed to conduct a program should be articulated during the early stages of program development to insure that a program is realistically implemented and capable of meeting its stated goal(s).

Activities represent efforts conducted to achieve the program objectives. After considering the resources a program will need, the specific activities that will be used to bring about the intended changes or results must be determined.

Process Measures are data used to demonstrate the implementation of activities. These include products of activities and indicators of services provided. Process measures provide documentation of whether a program is being implemented as originally intended. For example, process measures for a mental health court program might include the number of treatment contacts or the type of treatment received.

Outcome measures represent the actual change(s) or lack thereof in the target (e.g., clients or system) of the program that are directly related to the goal(s) and objectives. Outcomes may include intended or unintended consequences. Three levels of outcomes to consider include:

Initial outcomes: Immediate results of a program. Intermediate outcomes: The results following initial outcomes. Long Term: The ultimate impact of a program.

External Factors, located at the bottom of the logic model example, are factors within the system that may affect program operation. External factors vary according to program setting and may include influences such as development of or revisions to state/federal laws, unexpected changes in data sharing procedures, or other similar simultaneously running programs. It is important to think about external factors that might change how your program operates or affect program outcomes. External factors should be included during the development of the logic model so that they can be taken into account when assessing program operations or when interpreting the absence or presence of program changes.

If-Then Logic Model Another way to develop a logic model is by using an "if-then" sequence that indicates how each component relates to each other. Conceptually, the if-then logic model works like this: IF [program activity] THEN [program objective] and IF [program objective] THEN [program goal].

In reality, the if-then logic model looks like this: IF a truancy reduction program is offered to youth who have been truant from school THEN their school attendance will increase and IF their school attendance is increased THEN their graduation rates will increase.

Another way to conceptualize the "if-then" format:

• If the required resources are invested, then those resources can be used to conduct the program activities.

• If the activities are completed, then the desired outputs for the target population will be produced.

• If the outputs are produced, then the outcomes will indicate that the objectives of the program have been accomplished.

Developing program logic using an "if-then" sequence can help a program manager or evaluator maintain focus and direction for the project and help specify what will be measured through the evaluation.

Common Problems When Developing Logic Models

• Links among elements (e.g., objectives, activities, outcome measures) of the logic model are unclear or missing. It should be obvious which objective is tied to which activity, process measure, etc. Oftentimes logic models contain lists of each of the elements of a logic model without specifying which item on one list is related to which item on another list. This can easily lead to confusion regarding the relationship among elements or result in accidental omission of an item on a list of elements. • Too much (or too little) information is provided on the logic model. The logic model should include only the primary elements related to program/project design and operation. As a general rule, it should provide the "big picture" of the program/project and avoid providing very specific details related to how, for example, interventions will occur, or a list of all the agencies that will serve to improve collaboration efforts. If you feel that a model with all those details is necessary, consider developing two models; a model with the fundamental elements and a model with the details. • Objectives are confused with activities. Make sure that items listed as objectives are in fact objectives rather than activities. Anything related to program implementation or a task that is being carried out in order to accomplish something is an activity rather than an objective. For example, 'hire 10 staff members' is an activity that is being carried out in order to accomplish an objective such as 'improve response time for incoming phone calls.' • Objectives are not measurable. Unlike goals which are not considered measurable because they are broad, mission-like statements, objectives should be measurable and directly related to the accomplishment of the goal. An objective is measurable when it specifically identifies the target (who or what will be affected), is time-oriented (when it will be accomplished), and indicates direction of desired change. In many cases, measurable objectives also include the amount of change desired.

Other Logic Model Examples Phoenix Gang Logic Model

OJJDP Generic Logic Model

United Way Program Outcome Model

University of Missouri Extension Program Planning and Development Logic Model

Learn More About How to Develop a Logic Model Developing a Basic Logic Model for Your Program (The University of Arizona School of Public Health)

Enhancing Performance with Logic Models (University of Wisconsin-Extension, Division of Cooperative Extension)

Establishing Goals, Objectives and Evaluation Criteria (U.S. Department of Housing and Urban Development)

Using the Logic Model for Program Planning (Legal Service Corporation Resource Information)

Assessing Program Performance

Identifying Goals and Objectives Programs must have clearly specified goals and objectives before an evaluation can take place. A program goal is a broad statement of what the program hopes to accomplish or what changes it expects to produce. Examples of program goal statements include:

• Reduce reoffending among substance abusing offenders served by the program • Reduce the crime rate in the neighborhood targeted by the program • Restore a sense of well-being to victims of crime

An objective is a specific and measurable condition that must be attained in order to accomplish a particular program goal. There are many different ways to specify objectives; the program and

evaluator should choose the method that works best for each situation. Examples of program objectives include:

• Assist substance abusing offenders in abstaining from drug use • Ensure that victims of crime feel compensated for their losses • Improve by one grade level reading scores for 80% of the juveniles who complete the

program

Learn More... Establishing Evaluation Criteria (U.S. Department of Housing and Urban Development)

The Logic of Evaluation (Office of Juvenile Justice and Delinquency Prevention)

Measuring Performance When There is No Bottom Line (Bureau of Justice Assistance)

The Problem of Defining Agency Success (Bureau of Justice Assistance)

State your Program Objectives in Measurable Terms (U.S. Department of Housing and Urban Development)

What You Expect: Building A Theory of Action (National Institute of Justice, Chapter 2)

Measuring Activities and Outputs: Process Evaluation Once a program has identified its goals and objectives, it needs to specify the major activities or processes that it will undertake that will lead to accomplishing these goals and objectives. One component of measuring a program's performance is to determine whether activities were actually implemented as planned. The reason that this is important is that if activities are not implemented as planned, then there is no reason to believe that the activities as they were implemented will produce the desired objectives.

The immediate results of activities are referred to as outputs. Output measures are indicators of the degree to which activities were implemented as planned. Examples of output measures include:

• Number of offenders receiving counseling services • Number of community service projects completed • Proportion of parolees who receive drug tests

Process evaluation focuses on program implementation. Process evaluations generally involve: reviewing program documents, interviewing program staff, observing program operations, and collecting data from program files. In addition to collecting data on output measures, process evaluations examine a number of additional questions; for example:

• How well were key program elements, such as multiagency collaboration, implemented? • Did the program serve its target group (for example, high risk probationers)? • What was the dropout rate for the program, and how can this rate be reduced?

Learn More... Implement a Process Evaluation to Document What is Done, When, by Whom, To Whom (National Institute of Justice)

Measurement Issues (Office of Juvenile Justice and Delinquency Prevention)

Process Analysis (The Urban Institute)

Program Implementation (General Accounting Office)

Measuring Outcomes: Impact Evaluation Another component of measuring a program's performance is determining whether the activities produced the desired effects or outcomes or, put another way, whether the program achieved its objectives. Measuring outcomes tells the program and the evaluator what impacts the program has had or what results it has achieved. Such impacts are usually expressed in terms of behavior change in those served by the program: reducing reoffending or increasing knowledge about the negative consequences of substance abuse. Outcomes may be divided into short-term, intermediate, and long-term outcomes, with the last usually being the program goal.

There are a number of different ways to define and measure any particular outcome. The choice of a measurement method is critical to the program assessment process. A professional evaluator can be useful in helping to develop and identify valid and reliable outcome measures.

Learn More... Basic Outcome Evaluation and Comparative Outcome Evaluation (Office of Juvenile Justice and Delinquency Prevention)

Measuring Program Outcomes (Office of Juvenile Justice and Delinquency Prevention)

Varieties of Outcome Measures (Office of Juvenile Justice and Delinquency Prevention)

Establishing the "Activities-Outcomes" Connection: Evaluation Experiments Performance measurement can and should assess program outcomes. However, in order to establish the connection between a program's activities and observed outcomes, an impact evaluation, in the form of an experiment or randomized controlled trial (RCT), is necessary. The RCT involves assigning individuals randomly to participate in the program, then comparing outcomes for program participants and non-participants. While in theory all programs should be evaluated using RCTs, practical considerations limit their use in many situations. In order to illustrate the advantages and disadvantages of evaluation experiments, three common evaluation designs are reviewed:

• Pre-experimental (pre-post) design • Quasi-experimental (comparison group) design • Experimental (control group) design (randomized controlled trial)

Learn More... Allocate Sufficient Funds for an Impact Evaluation: If Controlled Experimentation is Infeasible, Approach Less Rigorous Designs with Caution and Imagination (National Institute of Justice)

Impact Evaluation Designs and The Impact Evaluation Design 'Decision Tree' (The Urban Institute)

Methods of Analyzing Data (National Institute of Justice)

Observing Behavioral Outcomes and Attributing Changes to the Program (U.S. Department of Education) Establishing the "Activities-Outcomes" Connection: Evaluation Experiments Quasi-Experimental (Comparison Group) Design In this design, change is assessed by comparing perceptions or behaviors of program participants with those of non-participants (comparison group). If outcomes for the two groups differ in the expected way (e.g., program participants have lower recidivism rates than non-participants), then the evaluator assumes that the difference was caused by the program. The assumption here is that the program participants are exactly like the non-participants in every way except that they received the program services, so any differences between the two

must be due to the program. In such designs, evaluators often select non-participants who match participants on key factors, such as age, gender, and criminal history. The trouble with this design, however, is that the evaluator can never be certain that the groups are exactly the same on every factor that might lead to differences in observed outcomes. The evaluator can have more confidence in the results of a quasi-experiment than he or she can in the results of the pre-post design, but still cannot be certain that the program activities caused the observed outcomes. Learn More... The Nonequivalent Comparison Group Design (Government Accounting Office) Non-Random Comparison Group (National Institute of Justice, pp. 4.5-4.6)

Establishing the "Activities-Outcomes" Connection: Evaluation Experiments

Pre-Experimental (Pre-Post) Design The pre-post design measures program outcomes by comparing perceptions or behaviors at the end of the program (post) to some baseline, usually the same elements measured at prior to the start of the program (pre). If program participants change in the expected direction, then the outcomes are said to have been achieved.

The difficulty with this design is that it is not possible to attribute any observed changes to the program itself, as opposed to other factors that might have produced the changes. In other words, it is impossible to conclude that the program activities caused the observed outcomes.

Learn More... The Before-and-After Design (General Accounting Office)

Pre- and Post-Test Scores (National Institute of Justice, p. 4.8)

Threats to Validity (Office of Juvenile Justice and Delinquency Prevention)

Establishing the "Activities-Outcomes" Connection: Evaluation Experiments

Quasi-Experimental (Comparison Group) Design In this design, change is assessed by comparing perceptions or behaviors of program participants with those of non-participants (comparison group). If outcomes for the two groups differ in the expected way (e.g., program participants have lower recidivism rates than non-participants), then the evaluator assumes that the difference was caused by the program.

The assumption here is that the program participants are exactly like the non-participants in every way except that they received the program services, so any differences between the two must be due to the program. In such designs, evaluators often select non-participants who match participants on key factors, such as age, gender, and criminal history.

The trouble with this design, however, is that the evaluator can never be certain that the groups are exactly the same on every factor that might lead to differences in observed outcomes. The evaluator can have more confidence in the results of a quasi-experiment than he or she can in the results of the pre-post design, but still cannot be certain that the program activities caused the observed outcomes.

Learn More... The Nonequivalent Comparison Group Design (Government Accounting Office)

Non-Random Comparison Group (National Institute of Justice, pp. 4.5-4.6)

Establishing the "Activities-Outcomes" Connection: Evaluation Experiments

Experimental (Control Group) Design (Randomized Controlled Trial) As in the quasi-experiment, a randomized controlled trial (RCT) involves comparing program participants and non-participants. In order to ensure equivalence, the RCT involves randomly assigning participants to groups. This means that which offenders receive program services and which do not is decided not by a judge or other criminal justice administrator, but by the evaluator. This random assignment procedure is the best way of ensuring that there are no differences between program participants and non-participants except for the program services provided to the former group.

This design, however, cannot always be employed to assess criminal justice initiatives. For some initiatives, like community-wide efforts and multijurisdictional law enforcement drug task forces, assigning cases randomly is not feasible. In other cases, judges and other criminal justice administrators may refuse to surrender their discretion in the interests of sound evaluation practice.

Learn More... Random Assignment (National Institute of Justice, pp. 4.3-4.4)

The True Experiment (General Accounting Office)

Use of Random Assignment (Office of Juvenile Justice and Delinquency Prevention)

Data Collection

New or Existing Data? Most programs collect some information that is potentially useful for evaluation. At the outset, the evaluation needs to assess what data already exist, what the quality of the data are, and whether they are readily available in a useable form. The answers to these questions will help to determine whether existing data can be used, or whether new data must be collected.

When planning an evaluation, the evaluator must determine whether existing or new data will be used in data analysis. The advantage of using new data is the greater control an evaluator has over the measures, procedures, and data collection staff, which can contribute to greater reliability and validity of the data. Using existing data has the advantage of cost savings, because time, effort, and other resources are not spent on collecting new data.

Learn More... Data Collection (U.S. Department of Housing and Urban Development)

How Do You Get the Information You Need for Your Evaluation? (U.S. Department of Housing and Urban Development)

Obtaining Information for Evaluations - Use Existing Data or Collect New Information? (National Institute of Justice)

Using Existing Data Sometimes evaluators are able to use information that already exists without going through the expensive and time-consuming process of collecting new data. Information collected by the program for a variety of purposes may have value for performance measurement and evaluation. Evaluators can often make relatively small changes in the program's practices and procedures that will result in data that can be more readily used for evaluation. Examples of existing data on program participants that might be able to be used for evaluation include:

• Attendance records • Counseling forms and progress notes • Discharge summaries • Presentence investigation reports • Psychological testing and other classification information

Learn More... Ensuring That Evaluations Yield Valid and Reliable Findings (U.S. Department of Education)

Verifying the Accuracy of the Data (U.S. Department of Housing and Urban Development)

Using New Data Even if some evaluation data are currently collected, they will often need to be supplemented by the collection of additional data. These new data can be collected through various strategies:

Direct Observation Obtaining data by on-site observation has the advantage of providing an opportunity to learn in detail how the project works, the context in which it exists, and what its various consequences are. However, this type of data collection can be expensive and time- consuming. Observations conducted by program staff, as opposed to an outside evaluator, may also suffer from subjectivity.

Interviews Interviews are an effective way of obtaining information about the perceptions of program staff and clients. An external evaluator will often conduct interviews with program managers, staff members, and clients to obtain their perceptions of how well the program functions. A disadvantage to conducting interviews is that they can be time- consuming and costly, and produce subjective information.

Surveys and Questionnaires Surveys and questionnaires can provide information on program staff members' perceptions of program operations and their own functions. Surveys of clients can provide information on attitudes, beliefs, and self-reported behaviors. An important benefit of surveys is that they provide anonymity to respondents, which can reduce the likelihood of biased reporting and increase data validity. A variety of issues are associated with the use of surveys and questionnaires, including reading level, cultural bias, and sensitivity to particular wording.

Official Records Official records and files are one of the most common sources of data for criminal justice evaluations. Arrest reports, court files, and prison records all contain much useful information for assessing program outcomes. Often these files are automated, making accessing these data easier and less expensive.

Learn More... Basic Guidelines for the Development of Survey Items (Office of Juvenile Justice and Delinquency Prevention)

Data Collection Strategies (The Urban Institute)

Developing and Using Questionnaires (General Accounting Office)

Consortium Survey Questions on Drug Use and Drug Control Strategies (Bureau of Justice Assistance)

Special Topics in Program Evaluation: Sources of Data (Office of Juvenile Justice and Delinquency Prevention)

Use of Observation in Program Evaluation (Office of Juvenile Justice and Delinquency Prevention)

Using Structured Interviewing Techniques (General Accounting Office)

Other Considerations A number of other important issues should be considered in developing a data collection strategy for evaluation. These issues include:

Sampling For very large initiatives, including all participants may not be desirable or feasible. In such cases, a sample, or subgroup, should be included in the evaluation. Ideally, the sample should be selected randomly; that is, program participants should be selected entirely by chance. Program participants selected nonrandomly, based on particular characteristics, are not representative of the entire group, and evaluation findings based on the sample may therefore not apply to the entire group.

Human Subjects All evaluations that involve program participants or their records are subject to rules governing the treatment of human subjects in research. A certified human subjects review ommittee must review evaluations. This process helps to ensure participants' safety and the confidentiality of the information collected about them.

Informed Consent Evaluations must obtain the consent of participants to be included in the assessment. Participants must voluntarily sign consent forms that inform them of the purpose of the evaluation, what data will be collected and reported, and to what degree these data will be confidential.

Learn More... Common Errors in Evaluation (Office of Juvenile Justice and Delinquency Prevention)

Designing the Sample or Population for Data Collection (U.S. General Accounting Office)

Informed Consent, Follow-up Arrangements, and Confidentiality/Data Security (The Urban Institute)

Protection of Human Subjects Regulation (National Institute of Justice)

Who Should Be Surveyed? (U.S. Department of Education)

Reporting and Using Evaluation Results

Reviewing Evaluation Findings With Stakeholders Communicating and disseminating the evaluation findings is a critical step in building support for a program. Evaluators should plan the reporting process as carefully as the evaluation itself, and build in opportunities to share findings with key stakeholders, such as program managers and staff, prior to the final report.

While evaluation findings must be reported objectively, interpreting those findings and reaching conclusions can be a challenging process. The evaluator should include key stakeholders in this process by reviewing findings and preliminary conclusions with them prior to writing a formal report. Circulating an interim or draft report and meeting to discuss it provides a means of obtaining feedback. Discussions with staff can provide new perspectives on the meaning and interpretation of the findings. These perspectives can then be included in the final report.

Briefings on the findings of the evaluation may be more useful for stakeholders outside the program that do not have time to read an interim report. The briefing can be used to generate ideas and feedback in much the same way as an interim or draft report. It can also be used to obtain feedback on how findings should be presented.

Learn More... Using Evaluations: Audiences and Products (Office of Juvenile Justice and Delinquency Prevention)

Using Evaluation Findings for Decision Making (National Institute of Justice)

Writing a Final Report The product of most evaluations is a written final report. Final reports should be concise and nontechnical in nature. An executive summary, which is a condensed summary of the main points of the report, should be included with the final report. Technical material not of interest to a general audience can be included in appendices or in separate volumes. A final evaluation report should include:

• Review of the findings of previous evaluations of similar programs • Discussion of why the evaluation was conducted and questions the evaluation sought to

address • Program description, including goals, objectives, and activities • Explanation of the methods and the procedures undertaken to collect and analyze data,

including a description of output and outcome measures and the evaluation design • Presentation of the results • Interpretation of the results and conclusions • Limitations of the evaluation methodology • Recommendations for future steps, including short- and long-term suggestions for

program improvement

Learn More... Report Formatting: Issues of Content and Graphic Design (Bureau of Justice Assistance)

Interpreting and Reporting Evaluation Findings (U.S. Department of Education)

Reporting Your Findings and Sample Outline - Final Evaluation Report (Department of Housing and Urban Development)

Using Evaluation Results Evaluation information can be a powerful tool for a variety of stakeholders. Program managers can use the information to make changes in their programs that will enhance their effectiveness. Decisionmakers can ensure that they are funding effective programs. Grant monitors can ensure that programs are developed as intended and have sufficient resources to implement activities and meet their objectives.

Ideally, evaluation is an ongoing process that is embedded in the process of program planning, implementation and improvement. Evaluation findings can be used to revise policies, procedures, activities, and objectives to allow programs to provide the best possible service to their clients.

Learn More... Identifying Effective Criminal Justice Programs: Guidelines and Criteria for the Nomination of Effective Programs (Bureau of Justice Assistance)

Using Evaluation Information (The Urban Institute)