Summer End 2

Chapter 8 Notes Evaluation Purpose, Types and Questions Most evaluations have multiple purposes. To focus the evaluation it is important to identify people with whom to work with in the evaluation process—beginning to end—in order to discern why the evaluation is needed, who may need to be involved, the type of evaluation that may be appropriate and the types of questions that may be pursued. This is an important and delicate task as it can serve to undermine or compromise the evaluation process. Evaluations can be characterized as legitimate and illegitimate. If evaluations are conducted to support forgone conclusions or as a public relations pieces, then these are illegitimate purposes for evaluation and should be avoided. Legitimate purposes of evaluation are undertaken with an honest desire to gather information that is well balanced and adheres to the standards and guidelines for program evaluation. The majority of evaluations are conducted for multiple purposes. Therefore, the principle task for evaluator is to discern the purposes and identify the most appropriate approach to achieve these purposes. Evaluations can serve one of four general purposes:

 To gain insights or to determine necessary inputs

 To find areas in need of improvement or to change practices

 To assess program effectiveness

 To address human rights and social justice We will consider these general purposes and identify types of evaluation that are subsumed under each and identify questions that illustrate them.

TO GAIN INSIGHTS OR TO DETERMINE NECESSARY INPUTS Capacity Building—Evaluators can benefit from knowing about an organizations past experiences with evaluation, as well as its expertise in and willingness to use evaluation as a way of ongoing improvement. If the expertise is absent or insufficient, then evaluators or need to undertake capacity building with the organization by instituting training programs, workshops, or community meetings to this end. Need/Asset—needs and asset assessment can provide a picture of the community (context), identify demographic groups and geographic areas in need; provide guidance in the prioritization and use of resources such as funding to address important needs; and convince policy makers that they should support initiatives.

Examples Questions

Context Organizational

To identify needed inputs, barriers, and facilitators to program development or implementation

What are the values that underlie this project and how do those map onto the values of the parent organization? What is the nature of the relationship between the project and the parent organization in terms of finances, physical space and administrative structures? How does the leadership and organizational structure support or impede the success of the project? What the characteristics of the staff and the leadership? What is the organizational culture with regard to the project and the evaluation? What resources are available in terms of funding, staffing, organizational support, expertise and educational opportunities?

Capacity Building

To assess and build capacity in the community

What is the organization past experience with evaluation? What is their expertise or willingness to use evaluations? What training, workshops, or other activities have been done or are needed?

Need/Asset To assess needs, desires, assets of community members

What issue or problem is concerning to you? What knowledge do you have about it right now? What resources are available to you to understand this issue? Which groups of people are most affected by the discrepancy between what is and what should be? Are there differences in opinion on this? What has the organization done in the past to address this discrepancy? What are the challenges that still remain? How should we adapt the program we are considering? Should we being the program? Is there sufficient need to warrant the program?

Relevance To determine feasibility of methods to describe program evaluation activities

To what extent are objectives of the program still valid? To what extent are the activities and outputs of the current program consistent with the overall aims of the program and the intended outcomes? How do these activities contribute to the attainment of objectives?

TO FIND AREAS IN NEED OF IMPROVEMENT OR TO CHANGE PRACTICES Implementation—is useful when a new program is being implemented or if data indicate that goals of an existing program are not being satisfactorily achieved. It can be focused on identifying strengths and challenges in the implementation of a program, reassessing the appropriateness of the program under changing conditions, assessing the extent to which the appropriate resources are available; measuring perceptions of the program by the community, staff and participants; determining the quality of the services provided; and monitoring stakeholder’s experiences.

Examples Questions

Implementation To characterize the extent to which intervention or plans were implemented

What are the critical components/activities of this project (implicit and explicit)? How do these components connect to the goals and intended outcomes for this project? What aspects of the implementation process are facilitating success or acting as stumbling blocks for the project? How is the program being implemented and how does that compare to the initial plan for implementation? What changes might be necessary in organizational structure, recruitment materials, support for participants, resources, facilities, scheduling location, transportation, strategies, or activities? Were the required resources available? To what extent was the program implemented according to the core components described in the plan? How competent were the service providers with specific reference to the program’s core components?

Responsive Participatory

To enhance programs cultural competence To very participants rights are protected To mobilize community support for the program

What is the match between what was planned and what was delivered? Is feedback being provided to stakeholders? And what impact is the feedback having on the program design and delivery?

Process Monitoring

To refine plans for introducing a new service To set priorities for staff training To determine whether customer satisfaction can be improved

Is the program achieving its objectives? Is the program measuring up against performance standards? Which aspects of operations have had an impact on the intended beneficiaries? Which factors in the environment have impeded or contributed to the program’s success? How is the relationship between the program’s inputs, activities, outputs and outcomes most accurately explained? What impacts has the program had beyond its intended objectives? What would have occurred if the program had not been implemented? How has the program performed in comparison to similar programs? Are sufficient numbers of the target audience participating in the program? Is more training of staff needed to deliver the program appropriately?

Formative Developmental

To improve content of educational materials To make midcourse adjustments to improve participant logistics

What is working? What needs to be improved? How can it be improved?

TO ASSESS PROGRAM EFFECTIVENESS

Example Questions

Outcome Impact

To document the level of success in accomplishing objectives To assess skill development, knowledge gain, and/or attitude and behavior changes by program participants

What are the critical outcomes you are trying to achieve? What impact is the project having on its clients, its staff, its umbrella organization, and its community? What unexpected impact has the project had? To what extent were the objectives achieved? What factors influence the achievement of objectives? What has happened as a result of the program? What real difference has the activity made to the beneficiaries? How many people were affected?

Summative To decide where to allocate new resources

What results occur? With whom? Under what conditions? With what training? At what costs? Is this program achieving its goals?

Policy To demonstrate accountability requirements are fulfilled

What types and levels of policy need to be changed? Which persons, agencies, and so on do we need to contact and influence? What do they want to hear?

Replicability Transferability

To aggregate information from several evaluations to estimate outcome effects for similar kinds of programs To find out which participants do well in the program?

What is unique about this project? Can the project be effectively replicated? What are the critical implementation elements? How might contextual factors affect replication? Do impacts vary for different groups of people and programs?

Sustainability To compare changes in providers behavior over time

To what extent did the benefits of a program continue beyond the program? What factors influenced the continuance of benefits? What social, political, cultural, economic conditions influence the growth and sustainability of program? What happens when funding runs out? What conditions need to be in place for the sustainability of the program

Cost To compare cost-benefits Were activities cost-effective? Where objectives achieved on time>