Hsm/270 Project-Wide or objective-Oriented Evaluations

hsm270_week4_reading1.pdf

Home >Psychology homework help >Hsm/270 Project-Wide or objective-Oriented Evaluations

DEVELOPING AN EVALUATION PLAN

As program staff start planning for program development and begin addressing the components of the logic model, they should keep in mind that as the logic model takes shape, the develop- ment of evaluation plans will follow soon after. The logic model can act as a guide to assist you in determining when your evaluation plan should be developed. As you identify the accom- plishments and outcomes in your logic model, the next question to ask yourself is, “How will I determine if I met my accomplishments and reached the outcomes I identified?” The evalua- tion plan will provide a framework for why the evaluation is to be conducted, what is to be eval- uated, and how the evaluation will be conducted. After you have assessed the community needs and determined what services/activities will be implemented to address those needs, the evalu- ation plan is the next step in your planning process.

Empowerment Evaluation

Before learning how to develop an evaluation plan, it is important to look at the purpose of con- ducting a program evaluation and understand the type of evaluation models available. This sec- tion of the chapter introduces the empowerment evaluation model and the benefits of using this model for program development and improvement.

Gutierrez (1994) defines empowerment as the “process of increasing personal, interpersonal, or political power so that individuals, families, and communities can take action to improve their situations” (p. 202). It is a state of mind of worthiness and control that results in the change of the social and power structure (Swift & Levin, 1987). DuBois and Miley (1996) assert that “empow- erment presumes that people themselves should be integrally involved in the change process— from defining their situation to determining goals, selecting their course of action, and evaluating the results” (p. 27). Empowerment evaluation involves program staff as evaluators, fosters their professional development, and promotes self-sufficiency and competency. Fetterman, Kaftarian, and Wandersman (1996), in their definition of the empowerment approach, encourage program staff to be involved in the evaluation development and implementation: “Empowerment evalua- tion has an unambiguous value orientation—it is designed to help people help themselves and improve their programs using a form of self-evaluation and reflection. Program participants con- duct their own evaluations and typically act as facilitators; an outside evaluator often serves as a coach or additional facilitator depending on internal program capabilities” (p. 5). Ginsberg (2001) summarizes the sentiments of the proponents of empowerment evaluation that they see it as a democratic process. It “involves receipts of service in defining the design and implementation as well as the analysis of a program evaluation” (p. 38). Empowerment evaluation promotes partici- pation by all concerned parties and it is considered “comparable to community development or community organization activities in social work” (Ginsberg, 2001, p. 39).

C H A P T E R

Purpose of Evaluation

Some program administrators have little knowledge about program evaluation and are learn- ing on the job. When asked why one needed to conduct an evaluation, their response was a general statement describing the need to meet compliance by reporting to the funding source on what occurred in the program. They prefer not to conduct an evaluation if given the choice, although they are aware that evaluation could assist in improving program services.

A clear understanding of how evaluation can be beneficial for program improvement, prior to implementing a project, can provide program staff with the incentive to take on the task of evaluating their program. To determine the purpose of evaluating their program, program administrators need to identify what questions they want the evaluation process to answer as well as who is asking these questions. Two other criteria should be considered; first, what resources are available to conduct the evaluation, both in the context of professional expertise as well as funding; and second, how the information will be used. The answers to these ques- tions will assist program staff in planning for their evaluation.

Choosing the Empowerment Evaluation Model

There are a number of types of evaluation models that serve a variety of purposes. Each model has a purpose, and each model plays an important role in the field of evaluation. Therefore, it is important to determine who wants to know the answers to what questions, and for what pur- pose. The level of resources available also needs to be considered when making these decisions.

When is it appropriate to use empowerment evaluation? The purpose of empowerment evaluation is to assist program staff in conducting a self-assessment of their program services and to use the evaluation results for program improvement. Therefore, the evaluation questions that program staff might ask include:

1. Do the services provided to the community meet the needs identified prior to providing these services?

2. Can the program services be improved so that the quality of services offered to the com- munity increases?

There are three advantages in using the empowerment evaluation. These advantages sup- port the premise of this book that evaluation and program development go hand in hand.

First, because program staff will be involved in conducting the evaluation, the cost of con- ducting empowerment evaluation will be minimal as compared to contracting with an inde- pendent (outside) evaluator.

Second, because program staff are involved in the decisions of what to evaluate and how the evaluation should be conducted, the results will be more meaningful to staff for use in program improvement. Program staff will also be more invested in participating in the evaluation.

Third, the process of developing the evaluation plans forces program staff to take a close look at the program services they intend to provide. As program staff design the plan, they must strategically lay out the community needs, activities to address the needs, target population to be served, and the desired outcome of the services provided. At this point of the evaluation development process, program staff can begin to see if the steps “align,” or support, one anoth- er. For example, will the activity address the need? Is the population to be served a part of the need? Will the desired outcome occur as a result of the activity?

DEVELOPING AN EVALUATION PLAN 47

However, one must not overlook the challenge program staff face when choosing to use the empowerment evaluation approach. Because program staffs are taking a large role in conduct- ing the evaluation, two hurdles stand before them:

1. Program staff may have little knowledge and/or experience in designing and conducting an evaluation. They may need assistance from a professional evaluator and the willingness to learn how to conduct their own self-assessment of program services. Attitude toward learning how to conduct an evaluation using the empowerment approach is critical to the success of implementation.

2. Program staff are given an added responsibility, in addition to program implementation. They must also be responsible for conducting the evaluation of program services. This places an added burden on a staff’s already busy schedule. There are two approaches that can ease this burden. First, to the extent possible, program staff can infuse the evaluation tasks as part of the program delivery. Second, the more program staff accept the evalua- tion tasks as being a necessary part of the overall program service effort, the less burden- some the tasks will seem. Again, attitude toward evaluation plays a critical role in the quality of evaluation conducted.

As described above, program staff are conducting their own evaluation, including collecting their own data and, in some cases, making subjective decisions in determining whether or not gain has occurred. Because this is a self-assessment, program staff must maintain a high standard of integrity. If staff want to, they can manipulate the data to make their program look successful, regardless of the actual outcomes. However, this would defeat the purpose of conducting an empowerment evaluation. Therefore, the need to revisit the purpose of this evaluation approach is important. The evaluation results are to be used for program improvement. If staff choose to manipulate the data (e.g., information being collected), the evaluation results may be meaning- less and not helpful for program improvement.

The Evaluation Plan

What is an evaluation plan, when should this plan be fully developed, and who should develop the plan? Using the empowerment evaluation approach, program staff enagage in developing an evaluation plan for their programs.

Program staff understand how to develop and implement services to meet the needs of the community. Some of them may not have the knowledge or experience to determine how the program will be evaluated. Certain program staff rely on “others” to design evaluation plans and conduct an evaluation while they attend to program issues. However, the empowerment approach does not follow this model.

In a perfect world, program staff will begin to view empowerment evaluation as a necessary part of the program delivery. For example, in an after-school tutoring program, just as the pro- gram would not be functional if tutors did not tutor students, so would the program not be func- tional if an evaluation was not conducted to measure desired outcomes for program improvement.

This chapter will describe incidents of how community organizations, local governmental agencies, educational systems, and faith-based organizations implemented the empowerment evaluation approach. The benefits of this approach, the burden placed on the programs, and how the evaluation approach can be infused with program development or improvement will also be discussed.

48 CHAPTER FOUR

DEVELOPING AN EVALUATION PLAN 49

BOX

4.1 FIVE STEPS IN DEVELOPING AND IMPLEMENTING EMPOWERMENT EVALUATION

1. Develop an evaluation plan. 2. Identify or develop measurement tools. 3. Collect the data. 4. Analyze the data. 5. Report evaluation results.

A five-step approach is recommended to use for the development and implementation of program evaluation (see Box 4.1). The first of these five steps is the development of an evalua- tion plan.

Why Develop an Evaluation Plan?

When implementing program services, it is good practice to have a plan before proceeding. Usually the grant proposal serves this purpose. In the case of implementing process and out- come evaluations, evaluation plans also serve this purpose. The evaluation plan describes what the evaluation will look like and what information will be collected. Plans can be developed in a number of different formats, from an outline framework to detailed text. The outline format is particularly useful when using the empowerment approach in developing an evaluation plan and having the expectation of program staff to participate in its design.

Evaluation plans serve both as a guide for program staff to conduct the evaluation, as well as a means to show stakeholders what is to be evaluated and how the evaluation will be conducted. There are two basic types of evaluation plans; one plan will serve those programs that want to con- duct a project-wide evaluation, and the second plan will serve those programs that want to utilize an objective-oriented evaluation. Examples of evaluation plans appear in Chapter 7.

When Should the Evaluation Plan Be Developed?

Typically, as program staff begin to conduct their program planning and implementation, the thought of evaluating program services does not occur until the end of a program delivery cycle, or when a program report must be developed. Subsequently, the write-ups about the evaluation are vague. In some cases, they reflect the staff’s lack of knowledge and understanding of the util- ities and feasibility of evaluation. Ideally, the evaluation plan should be developed as the pro- posal is being written, or a detailed evaluation plan should be developed after the program design has been established but before the implementation of services begins.

The process of developing the evaluation plan will help form the direction and design of the program as well as clarify the purpose of conducting the activity or service. The authors have assisted many community-based programs in developing evaluation plans using the objective-oriented evaluation approach. When developing an evaluation plan, the one burning question that forces program staff to take a second look at how they are conducting their pro- gram services is:

What are you hoping the participants will gain or change after having participated in your services?

Having a clear vision of what the desired result will be after services are provided can help program staff carefully decide how they plan to implement their services (see Box 4.2). This process provides program staff with an opportunity to again question whether or not the pro- gram services planned will, in fact, provide the information, experience, or skills to participants, so that the desired result is reached.

Who Should Be Involved in the Development of the Evaluation Plan?

The empowerment approach stresses the need for program staff and stakeholders to become involved in the design and implementation of the evaluation plan. Both program staff and key stakeholders are encouraged to be involved in the development of the evaluation plan. This is one of the places where they have the opportunity to have a say in what they want to evaluate and how best to conduct the evaluation. They will more likely be invested in supporting the evaluation, and in some cases conducting the evaluation, because they will be obtaining infor- mation they can use to help improve the services they offer, determine if the type of services provided are appropriate to meet the community needs, and justify the good work they are providing. As demonstrated in Box 4.2, when program staff know what desired change is want- ed, they will be more inclined to design and implement services that will focus on those desired changes.

The ability to demonstrate a change in a situation when services are to prevent negative results from occurring can be challenging. Some program staff that have implemented commu- nity services have demonstrated the ability to conduct process evaluation, documenting the accomplishments of services provided. However, they may not have the knowledge or experi- ence to design a detailed evaluation plan without assistance, especially when developing an out- come evaluation plan for prevention services. Over time, program staff will develop skills in preparing and implementing an evaluation plan.

How Should the Evaluation Plan Be Designed?

Evaluation plans can be multifaceted, consisting of a variety of detailed information. On the other hand, evaluation plans can provide the essential information needed to conduct a basic evaluation. We have found that developing an evaluation plan using a framework design for why, when and how to conduct an evaluation works well when facilitating program staff to conduct their own evaluation.

In developing an evaluation plan, there is no “one way” to proceed. There are many options that can be used. However, the framework approach is very simple yet provides the detailed information needed to guide program staff through the implementation. It makes sense to the program staff and the results can be effectively used for program improvement. This model will also provide information that can be used for other purposes, such as quality assurance or writ- ing grant proposals.

As demonstrated in Chapter 3, the grant proposal defines the need for the service, the type of service/activity proposed that will respond to the need, the resources (input) needed to imple- ment the program, the accomplishments (output) expected to be achieved, and the benefits (outcomes) anticipated as a result of the services provided by program staff. The evaluation plan may include a synopsis of this information as well as additional information on how accomplish- ment data will be collected and what outcome data will be collected.

50 CHAPTER FOUR

Types of Evaluation Plans

The Project-Wide Evaluation Plan and the Objective-Oriented Evaluation Plan are the two types of evaluation plans that have many utilities for human service program.

Project-Wide Evaluation Plan

When developing an evaluation plan for a project that looks at the broad goals of the project, an evaluation plan format that provides a “big picture” outline is suggested. For the sake of identi- fying this approach, it will be called the Project-Wide Evaluation Plan.

One of the first steps in developing this type of evaluation plan is to identify the “big” ques- tions that the evaluation is to answer. After these Evaluation Questions have been developed, one must determine the sources that will be used to provide the information that will answer these questions. For example, in a tutoring program, if the big question is whether or not students increase their literacy skills, the source of information can be the students, teachers, parents, or tutors. After identifying where the source of information will be retrieved, the next step to address is how this information will be collected, or what evaluation method will be used to collect this information. For example, will students be tested, will teachers complete a survey, will parents be interviewed, or will the tutors document progress throughout the school year as they provide tutoring services? Box 4.3 demonstrates what a typical project-wide evaluation plan might look like. The type of descriptions in each of the columns can vary, and the number of columns can expand or be reduced, depending on how detailed the evaluation plan is to be developed.

Objective-Oriented Evaluation Plan

The sections that make up an objective-oriented evaluation plan are stated below. Keep in mind that the plan should address two focus areas: accomplishments or process evaluation, and ben- efits to the recipients of service or outcome evaluation. For a clear picture of the components of an objective-oriented evaluation plan, see Box 4.4.

DEVELOPING AN EVALUATION PLAN 51

BOX

4.2 AFTER-SCHOOL PROGRAM FOR ELEMENTARY STUDENTS

Many of the after-school programs offered academic support or recreational activities and had a general “purpose statement” regarding why they implemented those services (e.g., have a supervised place where youth can go). Although the services they offered served a purpose and, in many cases, their programs provided youth with specific cognitive, intellectual, social, or emotional benefits, there was a lack of focus on specific outcome gains. Developing an evaluation plan helped program staff improve their program by focusing their recreational services so that the youth had opportunities to develop knowledge or skills in a social/emotional area or cognitive/intellectual area.

For example, program staff decided to focus their recreational activities to teach youth how to handle con- flict in a constructive manner as youth participated in these recreational activities. Therefore, program staff planned conflict resolution training for themselves so that they could model appropriate methods of resolving conflict when conflicts arose between youths during recreational play. Program staff also dis- cussed structuring the recreational activities so that youth would have the opportunity to use conflict reso- lution skills in organizing recreational activities, developing rules, or becoming team leaders.

This framework can be used to develop both the process evaluation as well as the outcome evaluation. However, if desired, an evaluation plan can be developed for each of these areas. Let us first focus on developing an evaluation plan that includes both the process evaluation and out- come evaluation. Again, process evaluation will answer the question, “Did you do what you said you were going to do?” The outcome evaluation will answer questions such as, “What benefits did the recipients of service obtain?”, and “How well did you do?”

Developing an Evaluation Plan that Includes Process Evaluation and Outcome Evaluation

Process and outcome evaluation are the first two common areas of focus for the empowerment evaluation approach. A third focus area is impact evaluation, the long-term effects that have occurred as a result of the services provided. This latter focus area tends to be more complicat- ed and, many times, requires professional assistance.

52 CHAPTER FOUR

BOX

4.4 SAMPLE EVALUATION PLAN COMPONENTS FOR AN OBJECTIVE-ORIENTED EVALUATION

1. Community need 2. Activity or service to be implemented 3. Desired result 4. Indicators (elements to be measured/collected) 5. Method of measurement 6. Type of instrument to be used 7. Minimum level of success the program hopes to achieve 8. The implementation steps

A. Who is going to collect the data? B. When will the data be collected (dates/timeline)? C. Who will be responsible for aggregating and analyzing the data? D. How will the results be used, and who will be responsible for determining the format for use of the

results? ● Reports ● Reflection ● Public relations ● Fundraising ● Other uses of the results

BOX

4.3 TYPICAL PROJECT-WIDE EVALUATION PLAN

Evaluation Question Source of Information Method Timeline Responsible Person

DEVELOPING AN EVALUATION PLAN 53

Project-Wide Evaluation Plan

For each “big” evaluation question, determine whether you are going to conduct a process eval- uation, outcome evaluation, or both. Identify sources of whom and from where the information will be provided, and methods of how the information will be collected. Remember that process evaluation requires the documentation of accomplishments made, while outcome evaluation requires the gathering of information from a variety of possible sources that indicate benefits were achieved as a result of the services provided. Next, identify a timeline when the informa- tion will be collected and the person responsible for collecting the data. For a detailed example, see Box 4.5.

There is no limit to the number of “big” questions to ask. However, the more questions asked, the more effort program staff will need to make to answer the questions. Therefore, keep the number of “big” questions to a minimum, those that will provide information for program improvement, quality assurance, or other necessary qualifications.

Objective-Oriented Evaluation Plan

For those developing objective-oriented evaluation plans from a grant proposal, the proposal should already have information describing the needs, the services that are being proposed to address the needs, and the accomplishments and desired results anticipated as a result of the services provided. In the spirit of program improvement, here is an opportunity to revisit the services being offered and discuss whether or not these services need to be modified, expanded, or eliminated. This information will be used as part of the evaluation plan. The next steps will be to determine the data needed and how these data will be collected so that the evaluation will demonstrate whether or not the accomplishments and outcomes have been met. The Objective and Evaluation Plan Development Form (see Box 3.11 on page 35) lists the major components and steps in designing and implementing evaluation strategies for an objective. The following sections of this chapter provide descriptions and examples of each of the key components.

BOX

4.5 SAMPLE PROJECT-WIDE EVALUATION PLAN

Evaluation Question Source of Information Method Timeline Responsible Person

To what extent did the students increase their literacy skills after hav- ing participated in the after-school literacy tutoring program?

Do students have a more positive attitide towards school after they participate in the tutoring program?

Number of students served at least 10 weeks, 3 ses- sions per week.

Students’ achievement

Teachers’ attitudes

Parents attitudes

Principal

Teachers

Student Participant Log (process)

Assessment tests

Survey

Interviews

Survey

Ongoing

August (obtain results)

May

Tutors

Project Director

Site Coordinator

Project Director

Need A description of the service need should already be described in the grant proposal or mission statement of existing services. A brief statement of the need would suffice here (see Box 4.6). If this particular objective focuses on a specific area of need of the grant proposal, state only the specific area that will be addressed as a result of implementing the activity or service. For example, if an after-school enrichment program’s particular evaluation plan focus- es only on youth leadership services, only these services need to be described, even if the pro- gram will also offer arts and crafts to increase cultural identity, and recreational activities to increase social skills.

Activity or Service This section should state who is providing the service, what service is pro- vided, where the service is offered, when the service is offered, and how often the service is offered. Using the example previously described, if this objective is focused on the mentoring program, provide information only on the mentoring efforts (see Box 4.7). Do not discuss the arts and crafts activities or the recreational activities.

Beneficiaries The number of beneficiaries and the characteristics of the beneficiaries should be discussed here. The type of characteristics can determine the challenges program staff will face when providing services. For example, the number of students participating in the mentoring pro- gram should be discussed here, as well as any characteristics, such as students who have behav- ioral problems or introverted students, which may affect the outcomes of services provided (see Box 4.8). Again, this information should already be described in the grant proposal.

54 CHAPTER FOUR

BOX

4.6 STATEMENT OF NEED

Need Example

Over 50% of the students in Wayne County School District come from single-parent family homes whose parents hold full-time jobs. Negative attitudes of students in grades 3–8 toward educational achievement have become a major con- cern over the last three years as indicated by increased truan- cy and disrespectful behavior.

Describe the need in the community that will be addressed by proposed services.

BOX

4.7 DESCRIPTION OF ACTIVITY OR SERVICE

Activity or Service Example

Fifty senior citizen volunteers, trained to become mentors, will be placed at five Wayne County elementary schools, ten per school, to provide one-on-one mentoring to third and fourth grade students; each mentor will be matched with two stu- dents for the entire school year, meeting with each student three times per week, 30 minutes per session, during after- school hours on school campus.

Describe the service activity you will be evaluating (who will do what, when, and where).

DEVELOPING AN EVALUATION PLAN 55

BOX

4.8 DISCUSSION OF BENEFICIARIES

Beneficiaries Example

100 third and fourth grade students in Wayne County School District who demonstrate poor attitudes toward school and disrespect toward peers and school staff, identified by school administration.

Briefly describe the people (and esti- mated number) your activity will serve.

Desired Accomplishments Accomplishments, as process data, are the actions and numbers that occurred during the implementation of the program. These are the elements that can be documented to answer whether the program actually did what was stated in the proposal. For example, in a mentoring program, the accomplishments to be collected may include:

● Number of youth matched with an adult mentor. ● Number of mentors who completed the training and were matched with a youth. ● Number of mentors who completed their mentor commitment. ● Number of youth who followed through with participating in the mentor program. ● Frequency and amount of time (i.e., minutes, hours) mentors and youth met.

The accomplishments described above can answer the question of whether or not the program provided the services to the degree it stated it would in the proposal (see Box 4.9).

Desired Result Desired result, as outcome data, is the big picture of what is hoped for as a result of providing services to the beneficiaries (see Box 4.10). The desired result may or may not be reachable within the timeline of the evaluation cycle (e.g., annual evaluation); however, if the services continue for an extended period of time, the result could be realized.

Indicators Indicators are the elements that are documented and counted to determine if the services provided were actually fulfilling what was proposed in the proposal (see Box 4.11). These elements are the changes that can be reached within the timeline of the evaluation cycle and, if continued over an extended period of time, will lead to the desired result. For example, if the proposal stated that the mentoring program would increase youth leadership skills, what indicators will be documented to determine if the youth did in fact increase their skills? Examples of indicators include:

● Becoming more involved in school and/or community projects. ● Becoming a role model for other youth. ● Willingness to do tasks when not having to do those tasks.

Method of Measurement This section should describe the method to be used to document the desired accomplishments and desired results (see Box 4.12). For process evaluation, deter- mining a method to document accomplishments usually involves tallying or keeping a record of what occurs. For example, the method of process measurement for a mentoring program may include:

56 CHAPTER FOUR

BOX

4.10 RECORD OF DESIRED RESULTS

Desired Results Example

Increased positive academic performance in school.Explain what outcome changes will occur because of the described activity.

BOX

4.11 LIST OF INDICATORS

Indicators Example

Increased participation in school activities and academic per- formance; as well as increased respect towards school per- sonnel and peers.

Describe the concrete, observable indi- cators of progress toward your out- come desired result.

BOX

4.12 DOCUMENTING METHODS OF MEASUREMENT

Method of Measurement Example

Process: Volunteer mentors will document sessions held with assigned students.

Outcome: Project director will interview the school administra- tion regarding student performance in academic achievement, participation in school activities, and respect towards school personnel/peers.

Describe the method you will use to determine if the described change occurs.

BOX

4.9 DESIRED ACCOMPLISHMENTS

Desired Accomplishments Example

50 senior citizen volunteers will complete the training, and each volunteer will mentor two students for the entire school year, meeting with each student three times per week.

Describe what type and quantity of services you hope to provide (e.g., num- ber of times met, time providing service, length of the program services).

DEVELOPING AN EVALUATION PLAN 57

● Tallying the number of mentors recruited, by documenting how many completed the train- ing and how many were matched with a youth, the number of times participated in support group meetings or special mentor/mentee events and completed their commitment to be a mentor, and determining the number willing to continue as a mentor after their initial commitment.

● Tallying the number of youth agreeing to participate in the mentor program, times keep- ing commitments with their mentor, number of times/hours meeting with the mentor, completed the commitment as a mentee.

● Number of support groups held and number of mentors attending each meeting, number of special events held and number of mentors/mentee teams participating in each event.

For outcome evaluation, determining a method to document indicators can consist of con- ducting observations, administering surveys, holding focus groups or interviews, or collecting existing data. For example, the method of an outcome measurement for a mentoring program may include surveying youth to determine their attitudes toward becoming involved in school and community activities, or conducting focus groups with mentors to determine how they observed changes in the leadership skills of the youth they mentored.

Type of Instruments The forms needed to document the elements for process evaluation generally include tally forms and attendance sheets (see Box 4.13). These forms are very basic and do not require much effort in developing. However, all too often program staff do not use forms; they just use papers that happen to be available to document the needed information. While this is acceptable because the information is indeed being collected, it is not an organized and concerted effort to collect useful data. As basic as a tally form or attendance sheet may be to develop, having “official forms” to collect these data is important for the following reasons:

● As basic as the information needed for each form (e.g., date, name of participant, location, type of event), a form acts as a “standard” for requiring staff to complete the required data.

● A form looks official; this places more emphasis on the importance of this information and the need to collect it on a regular basis.

● If different people are completing these forms, having the same forms keeps the data col- lection in an organized fashion.

When identifying instruments to collect “indicator” information for the outcome evaluation, program staff may not yet have identified an instrument, or may need to develop an instrument to meet their needs. However, program staff should still state, as detailed as possible, the type of instrument they plan to use. Examples of instrument descriptions can include a Pre-Post Student Leadership Attitude Survey or a Mentor Focus Group Protocol.

Standard of Success When writing a proposal and identifying the number of beneficiaries to serve, the efforts to accomplish, or the level of change wanted, the ability to meet these proposed accomplishments and desired results may or may not be reachable. The grant proposal is just that, a proposal; the information is what you hope to accomplish. There may be many situations that prevent program staff from implementing the service as intended. Therefore, when developing your evaluation plan, include what you hope to accomplish minimally (see Box 4.14). Using the information stated in their proposal, program staff have the opportunity to set minimum stan- dards they hope the program to reach. For example, a mentoring program may want to have a minimum of 90% of the mentors complete the training, and 80% of those trained complete their

mentor commitment. The standard for its outcome evaluation may be to have minimally 70% of the youth participating in the mentoring program to increase their leadership skills.

Respondents and Measurements Schedule The respondents to be identified here are the persons completing the instrument (see Box 4.15). This does not necessarily mean that the recipients of service will always be the respondents. For example, if youth participating in a mentoring program complete a survey, they would be the respondents; if the mentors complete a survey, they would be the respondents; if the supervisory staff completed an observational checklist on confidence skills, they would be the respondents. Also include the dates when the instruments are to be completed.

Data Collection, Data Aggregation, and Data Analysis The next three elements should list the persons who will be responsible for data collection, data aggregation, and data analysis (see Box 4.16). These people may not necessarily be the people who conduct the tasks; how- ever, they are the persons that need to make sure the tasks are conducted in a timely manner, the tasks are completed, and the instruments are secured and in a confidential location. Ideally, a person’s name is listed for each of these tasks; however, at the very least, the title or specif- ic position is identified and stated for these tasks. To list program staff as the responsible per- sons for any of these tasks is too vague. One person needs to be identified and held accountable.

58 CHAPTER FOUR

BOX

4.14 MINIMAL STANDARDS

Standard of Success Example

Process: 90% of the senior citizen mentor volunteers will complete their training.

Outcome: a. 70% of the students will increase their participa-

tion in school academics and activities. b. 80% will become more respectful to school per-

sonnel and peers.

Define the minimum level of success you hope to achieve.

BOX

4.13 LISTING TYPES OF INSTRUMENTS

Types of Instruments Example

Process: Volunteer mentor session documentation form

Outcome: Teacher Interview Protocol on student mentee participation and respect for others

Describe the instrument(s) that you will use.

DEVELOPING AN EVALUATION PLAN 59

BOX

4.15 RESPONDENTS AND MEASUREMENTS SCHEDULE

Respondents and Measurements Schedule Example

Process: 50 volunteer mentors will document the dates and length of time of each mentoring session.

Outcome: Teachers at each of the five schools will participate in quarterly interviews regarding student participation in academics, activities, and respect.

Describe who will be completing each instru- ment and the schedule for completing the instrument(s).

BOX

4.16 RESPONSIBLE PERSONS FOR DATA COLLECTION, DATA AGGREGATION, AND DATA ANAYSIS

Data Collection Example

Process: Site coordinators will be responsible for col- lecting the process evaluation data weekly.

Outcome: The project director will be responsible for collecting the outcome evaluation data quarterly.

Data Aggregation Example

Process: Each site coordinator will aggregate the process evaluation data for their school site at the end of each semester (January and May) and submit the data to the project director.

Outcome: Project director will compile the interview data quarterly.

Data Analysis Example

The project director will analyze the process evalua- tion data received from the five site coordinators; project director will analyze the outcome data at the end of each semester.

Describe who will collect the data and how often it will be collected.

Describe who will be responsible for aggregat- ing the data and how often will it be aggregated.

Describe who will be responsible for analyzing the data.

Reporting Results The last element of the evaluation plan is to identify the responsible person that will be writing the evaluation report (see Box 4.17). Many times, the person responsible for the aggregation and analysis is the same person responsible for reporting the results. More than one report may be planned or required; therefore, due dates should also be listed here so that the person responsible will know when the data collection, aggre- gation, and analysis need to be completed, allowing time for the report to be completed by the deadline dates.

This description on developing an evaluation is very basic, yet provides the necessary infor- mation to plan and conduct the evaluation. Examples of evaluation plans can be found at the end of Chapter 5, “The Strategies and Tools for Data Collection.”

Attached to each evaluation plan is the instrument identified for that particular evaluation plan. When reviewing the evaluation plan, observe how the instrument identified supports the plan in the areas of desired result, indicators, method of measure, and respondents. Also, look at how the standard of success is stated and the type of data collected. The type of data collect- ed should answer whether or not the standard of success has been met.

Summary

Developing an evaluation plan is the first of five steps (see Box 4.1 on page 49) in setting up and conducting an evaluation of community services. A number of evaluation models exist; each has an important role in the research and evaluation field. Choosing the appropriate evaluation model to meet the stakeholders’ needs determines how the evaluation will be developed and who will develop the plan.

This chapter describes the empowerment evaluation approach in conducting both a process evaluation and outcome evaluation of program services. The primary purpose of this approach is to use the evaluation results for continuous program improvement. Program staffs become involved in the planning, decision making, and implementation of the empowerment evaluation. Both the process of developing the evaluation plan, as well as implementation of the evaluation plan, contributes to the knowledge base and experience for improving program services.

References DuBois, B. & Miley, K. K. (1996). Social work: An empowering profession (2nd ed.).

Needham Heights, MA: Allyn & Bacon. Fetterman, D. M., Kaftarian, S. J., & Wandersman, A. (1996). Empowerment evaluation:

Knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage. Ginsberg, L. H. (2001). Social work evaluation: Principles and methods. Boston: Allyn and

Bacon. Gutierrez, L. M. (1994). Beyond coping: An empowerment perspective on stressful life events.

Journal of Sociology and Social Welfare, 21, 201–219. Swift, C., & Levin, G. (1987). Empowerment: An emerging mental health technology. Journal

of Primary Prevention, 8, 71–94. Witkin, B. R., Altschuld, J. W. (1995). Planning and conducting needs assessments: A practical

guide. Thousand Oaks, CA: Sage. Wolcott, H. F. (1991). Writing up qualitative research. Thousand Oaks, California: Sage.

60 CHAPTER FOUR

BOX

4.17 RESPONSIBLE PERSON FOR REPORTING RESULTS

Reporting Results Example

The project director will write the evaluation report in October and conduct an oral presentation before the Board of Education during the first quarter of the next school year.

Describe who will be responsible for writing the report and how often will reports be due.