Program Theory and Logic Model

patrifl_14i6

chapter9.pdf

Home >Social Science homework help >Program Theory and Logic Model

191

C h a p t e r

PREPARING FOR AN EVALUATION

CENTERS FOR DISEASE CONTROL AND PREVENTION

The six chapters in Part I provided you with the essential ingredients you need to digest before you can undertake any kind of evaluation: the

evaluation process (Chapter 3) and standards (Chap- ter 4), in addition to ethical (Chapter 5) and cultural (Chapter 6) considerations. The two chapters in Part II provided information on how to design social work programs (Chapter 7) with the use of logic models (Chapter 8).

Simply put, you need to know what goes into a program before you can evaluate it. Otherwise, how would you know what you’re evaluating? With Parts I and II under your belt you’re now well on your way to getting your “feet wet” by actually doing one or more of the four different types of evaluations that are cov- ered in the following four chapters: need (Chapter 10), process (Chapter 11), outcome (Chapter 12), and effi- ciency (Chapter 13).

No matter what type of evaluation your evalu- ation team decides to do, you have to know what to realistically expect before you start one—the topic of this chapter. Thus you need to begin thinking about how you are going to implement your evaluation before you actually carry it out. The expression “look before you leap” readily comes to mind here.

We have distilled the combined experience of a number of evaluation practitioners into nine evalua- tion implementation strategies contained in Box 9.1 that we believe will help support your evaluation’s success.

PLANNING AHEAD

Although this chapter discusses evaluation implemen- tation strategies, we still talk about planning. By doing so, we are asking that you “plan for the implementa- tion of your evaluation” by incorporating the nine strategies in Box 9.1 to guide your evaluation team in conducting a particular evaluation. In a nutshell, they represent important steps you need to plan for that will help you to implement your evaluation more smoothly.

Reading through this chapter during the evalua- tion planning process will remind you of things you will want to incorporate into your actual evaluation plans as you think ahead toward implementing it. In addition to discussing these helpful implementation strategies, we also provide a checklist (see Table 9.3) that you can use to keep track of your own progress in preparing for the eventual implementation of your evaluation.

Each of the four types of evaluations can, at times, be a complex undertaking that requires the cooperation and coordination of multiple people and other resources. By managing your evaluation care- fully, paying attention to the evaluation standards (i.e., utility, feasibility, propriety, accuracy), and closely following the steps in the evaluation process as presented in Chapter 3, you can facilitate a more smoothly run evaluation. Once again, key strategies developed by practitioners to minimize potential challenges and promote effective evaluation imple- mentation are listed in Box 9.1.

9When you translate a dream into reality, it's never a full implementation. It’s always easier to dream than to do.

~ Shai Agassi

Grinnell, R. M., Gabor, P. A., & Unrau, Y. A. (2015). Program evaluation for social workers : Foundations of evidence-based programs. Oxford University Press, Incorporated. Created from capella on 2023-01-30 18:42:52.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

192 Par t III: Implementing Evaluations

On planning …

Planning is bringing the future into the present so that you can do something about it now.

~ Alan Lakein

In the pages that follow, we highlight what’s involved in each of these general strategies, which aspects of your evaluation they can help you address, and what benefits you can expect from each strategy. Luckily, the majority of these strategies are simply a part of good project management, something most social workers do on a daily basis.

STRATEGY 1: WORKING WITH STAKEHOLDERS

Many of the causes of misunderstandings about evaluations—and of barriers to productive use of their findings—can be avoided or minimized when your stakeholders are included in key discussions at various points throughout your evaluation’s life cycle. Including those who are important to your program in conversations about the program, the evaluation itself, and what you hope to learn from it can make them feel included and less anxious about the results (see Tool C on how to reduce evaluation anxiety). Their involvement can also offer you fresh perspectives on what your evaluation can potentially accomplish and

ways to make the entire evaluation process run more smoothly.

Some stakeholders you may want to consider involving in your evaluation (or with whom you will want to communicate about it in other ways) include all those folks we mentioned in Chapters 1 through 3. Table 9.1 presents a variety of ways to work with them throughout your evaluation. Note that to engage stakeholders effectively, you will first need to gauge their level of knowledge and experi- ence regarding evaluation. It may also be necessary to provide them with an overview of program evalu- ation basics.

Perhaps you are wondering how you will manage the involvement of so many people in your evaluation, such as program directors, program staff, partners, evaluator(s), evaluation team members, and other program stakeholders. Questions you need to ask and answer are:

• Who will play what role(s)? • Who is in charge of which aspects of the

evaluation? • Who has decision-making authority over

which aspects of the evaluation?

As you explore working with your stakeholders, it’s important to recognize that you have a range of options for how you can structure these relation- ships and that there’s no “correct” or “incorrect” structure.

BOX 9.1 IMPLEMENTATION STRATEGIES TO MAKE YOUR EVALUATION RUN SMOOTHLY

Strategy 1 Work with all stakeholder groups throughout the evaluation life cycle—from initial design through action planning and implementation—in order to help focus on questions of interest to them and to incorporate their perspectives

Strategy 2 Develop a concrete process for managing the tasks, resources, and activities necessary for your evaluation

Strategy 3 Pilot-test data collection instruments and procedures

Strategy 4 Train data collection staff Strategy 5 Monitor the evaluation’s progress, budget,

timeline, and scope. Communicate frequently and

effectively with the evaluation implementation team and key stakeholders

Strategy 6 Disseminate results to all stakeholders in an accessible manner. Consider interim reporting where appropriate

Strategy 7 Develop an action plan to implement evaluation recommendations that includes clear roles, responsibilities, timeline, and budget

Strategy 8 Document lessons learned throughout the evaluation for use in future evaluations

Strategy 9 Link findings from the evaluation back to the strategic evaluation plan in case there are implications for the revision of the plan

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 9: Preparing for an Evaluation 193

Benefits of working with stakeholders:

• Encourages positive community response to your evaluation

• Builds “political will” to support your evaluation

• Develops support among program leadership for the program and/or for your evaluation in general

• Facilitates appropriate timing of your evaluation in relation to information needs

• Leads to development of relevant evaluation questions, which in turn supports use

• Promotes findings that are credible, used, understood, and accepted by all your stakeholder groups.

Table 9.1: Ways to Work with Stakeholders.

Category Detail (if appropriate to your situation)

Upfront discussions with

stakeholders about … • Plans for the evaluation (yours and theirs)

• Program priorities (yours and theirs)

• Information needs and evaluation questions to explore (yours and theirs)

• When information is needed

• What evidence would be considered credible

• How the data to be collected will answer the evaluation questions

• How findings can be used

• Community member perspectives to consider

• Privacy, confidentiality, and cultural sensitivity

• Limitations of evaluation

• What to do if findings suggest immediate need for program modifications

• A proactive approach to public relations, referred to as issues management, if the

evaluation may reflect negatively on program or community

Frequent communication

throughout the evaluation

with stakeholders about …

• Results from pilot tests

• Implementation progress

• Early findings

• Successes achieved

• Challenges encountered

• Other topics

Postevaluation

discussions with

stakeholders about …

• Turning findings into conclusions

• Celebrating strengths

• Developing recommendations grounded in findings

• Developing strategies for disseminating results

• Lessons learned

• Limitations of the evaluation

• Implications of the current evaluation for changes needed in the strategic evaluation plan

• Designing an action plan with clear information on recommended strategies, roles and

responsibilities, timeline, and budget

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

194 Par t III: Implementing Evaluations

The first step is to consider upfront what you want the roles, responsibilities, and lines of author- ity for those involved in your evaluation to look like. Here the evaluation literature can help you. For exam- ple, King and Stevahn (2002) have put considerable thought into the various roles an evaluator can play in relation to other evaluation stakeholders, within the organization sponsoring the evaluation, and in terms of managing interpersonal conflict. Tools A and B offer more information about these evaluator roles.

The second step is to clarify roles and responsibil- ities for everyone involved in order to avoid any mis- understandings. A “Roles and Responsibilities Table” lays out in detail who is responsible for what (shown as Table G.2 in Tool G),

As discussed further under Strategy 5, open and ongoing communication among evaluation stake- holders is paramount in conducting a successful eval- uation. Tool G provides suggestions on ways to keep team members and other stakeholders informed as to the progress of your evaluation. Devising fair and minimally burdensome ways to obtain feedback is another important aspect of communication.

For example, depending on the size of your pro- gram’s client catchment area and the dispersion of your stakeholders throughout your state, you may need to come up with creative ways for them to pro- vide their input remotely, whether they are formally serving on your evaluation team or their expertise is being sought for other reasons.

Meeting by teleconference—rather than in person—or allowing stakeholders to provide input electronically are some ways to ease the burden of their potential participation. Webinar software, should you or one of your partners have access to it, allows remote stakeholders to view graphics and other docu- ments online during tele-discussions. Some computer software packages of this type permit collaborative editing of documents, whereby all stakeholders can view edits on screen as they are being made.

Once you have drafted the final version of your evaluation plan, you will want to revisit the compo- sition of your evaluation team to see if you wish to constitute it differently as you move toward the actual implementation of your evaluation. The design may

have evolved in unexpected directions during plan- ning, or new individuals or organizations may have joined your partnership with a stake in the proposed evaluation.

Should additional stakeholders review your draft plan? Should some of them join the evaluation team that will carry the evaluation forward—those able to facilitate as well as those able to obstruct its prog- ress? Addressing concerns these individuals raise will help ensure your evaluation plan is feasible and that it receives the support it needs.

STRATEGY 2: MANAGING THE EVALUATION

Running a program evaluation is much like running any other project. The things you “worry about” may be a little different for an evaluation than for other kinds of projects, but the good management prac- tices that help you elsewhere in your professional life will also work well for you with an evaluation. Good management includes thinking ahead about what is most important, which activities precede which other activities, who will do what, what agreements and clearances are needed, when important products are due, how far your budget will stretch, and how to make the budget stretch further.

You will also want to monitor progress and com- municate frequently and efficiently with others on your evaluation team throughout the entire evalua- tion (see Strategy 5).

As part of your evaluation planning process, you must think ahead to the eventual implementation of your evaluation. We cannot stress this enough—think ahead. This is the purpose of this chapter: to encourage you to think ahead of what’s to come. For example, if your own staff resources are lacking, either in terms of skill level or time available, you may want to reach out to partners and contractors to fill that gap. You may also need to develop memoranda of agreement or contracts to engage this external support in a timely fashion.

If required by your agency or one of the partners engaged in your program, you may need clearances for the protection of human subjects such as those that may

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 9: Preparing for an Evaluation 195

be needed for an institutional review board (IRB) and Health Insurance Portability and Accountability Act (HIPPA). This can be requested as soon as your meth- odology has been finalized and your measuring instru- ments and consent (e.g., Box 5.2) and assent (Box 5.3) forms required by these entities have been developed.

Finally, you need to anticipate things that could cause problems down the road—such as the potential evaluation challenges presented in Tool D. Having identified potential challenges, you then need to put in place as many safeguards as possible to prevent them from happening, with contingency plans in

Table 9.2: Management Evaluation Strategies.

Category What to Look For

Logistics • Staff have skills required for evaluation tasks and are aware of their roles and responsibilities

• Staff are available to work on evaluation activities or alternatives have been considered

• Estimates of likely cost of evaluation in the individual evaluation plans are complete and feasible

• Efficiencies possible across evaluations have been identified

• Other sources of financial or staff support for evaluation (e.g., partner organizations, local

universities, grant funding) have been identified

• Actions to expand staff resources—such as contracting externally, training existing staff in

needed skills, “borrowing” partner staff, interns from local colleges and universities—have been

established

• Agreements are developed and executed that may be needed to contract out a portion of

the work (e.g., specific data collection activities, data analysis, development/distribution of

reports), to access data sources, to facilitate meetings with partners (schools, workplaces, etc.)

• Clearances/permissions that may be needed (such as IRB clearance, data-sharing agreements,

permission to access schools or medical facilities) are in place

Data

collection

• Appropriate data storage, data system capacity, data cleaning, and data preparation procedures

are established and communicated

• Procedures for protection of data are in place (considering such safeguards as frequent data

backups, use of more than one audio recorder for interviews and focus groups)

• Safeguards for respondent confidentiality and privacy have been developed

• Those collecting or compiling data have been trained in the procedures

• Monitoring systems are in place to assess progress and increase adherence to procedures for

data protection, assurance of privacy and confidentiality

• Cultural sensitivity of instruments has been tested

• Respondent burden has been minimized (e.g., length of instrument considered, data collection

strategies designed to be optimally appealing and minimally burdensome)

• Ways to maximize respondent participation are in place

• Existing data useful for the evaluation have been identified and permission to access those data

has been obtained

Data analysis • Procedures for how incoming data will be analyzed to answer the evaluation questions are

in place

• Table shells showing analyses to be conducted are developed

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

196 Par t III: Implementing Evaluations

mind should things not go as planned. And yes, some- times things do go south in an evaluation—way south.

Benefits of good evaluation management practice:

• Maintains clarity among team members about everyone’s roles and responsibilities

• Identifies and secures resources to complete the evaluation

• Keeps your evaluation on track in terms of timeline, budget, or scope

• Provides a sound plan for managing incoming data

• Enables all evaluation team members to follow clear procedures for working with contractors, consultants, and evaluation partners

This type of planning should be undertaken with your evaluation team members, program stakehold- ers, and individuals experienced in evaluation in the areas outlined in Table 9.2. Depending on your own level of familiarity with evaluation logistics, you may or may not feel the need for outside help in working through this process.

In either case, it’s important to consider how you will document the decisions made as part of this pro- cess so that you or others can refer back to them at a later date. How you do this is up to you and your evaluation team members.

You may find it helpful to integrate information on managing evaluation logistics into the individual evaluation plan, perhaps as an appendix. Or you may want to produce a separate document containing this information. The tips in Tool D will help you with this process, though you are not required to use them; they are there to use or not as you see fit.

STRATEGY 3: PILOT-TESTING

You should plan to pilot-test your data collection instruments and procedures. This is one good way to preempt some of the implementation challenges you

might otherwise face. This is important whether you are conducting mail and/or telephone surveys; car- rying out individual interviews, group interviews, or focus groups; or abstracting data from archival sources.

Benefits of pilot-testing measuring instruments and data collection procedures:

• Generates effective data collection instruments that collect required data that work with the designed data analysis plan

• Clarifies procedures for all data collection, whether carried out by your staff, contractors, consultants, or other data collection partners

• Improves the validity and reliability of the data collected

During the pilot test you will be looking at such issues as clarity of instructions, appropriateness and feasibility of the questions, sequence and flow of ques- tions, and feasibility of the data collection procedures. Use lessons learned during the pilot test to modify your data collection instruments and/or your train- ing materials for your data collectors. See Tool I for additional information on training data collectors.

STRATEGY 4: TRAINING DATA COLLECTION STAFF

Even if you are working with experienced individu- als who are evaluation savvy, training those who will be involved in data collection on the specific mea- suring instruments and data collection procedures you will use in this evaluation is another good way to avoid difficulties during the data collection phase. Training helps to ensure that all staff with data collec- tion responsibilities are familiar with the instruments and other forms that are part of your evaluation plan, as well as the procedures that will be followed and the safeguards that will be employed in implement- ing the plan. It will also promote consistency in data collection procedures across data collectors, thereby increasing the reliability of the data gathered.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 9: Preparing for an Evaluation 197

Benefits of training data collection staff:

• Promotes a consistent message about your evaluation to outside audiences

• Maintains consistency in data collection procedures

• Prevents loss of data and corruption of data integrity

• Guards against ethical breaches • Improves the validity and reliability of the

data collected

Training should be required whether data col- lection is being done by your own staff, by partner staff, or by contractors/consultants. Training ses- sions should cover not only the logistics of the work but also the ethical aspects, such as issues in human subjects protection, maintenance of confidentiality (Chapter 5), and observance of cultural sensitivity (Chapter 6). Tool I presents guidelines to help you develop and deliver training to data collection staff.

STRATEGY 5: MONITORING PROGRESS

As mentioned earlier, an evaluation like any other project, needs to be carefully managed. This includes not only thinking ahead during planning about what needs to be accomplished, who will do what, and what time and budget constraints exist (per Strategy 2); it also includes monitoring progress and maintaining open lines of communication among members of your evaluation team as your evaluation proceeds.

Benefits of tracking and ongoing communication:

• Maintains clarity among all your team members over their roles and responsibilities

• Keeps your evaluation on track in terms of timeline, budget, and scope

• Promotes effective communication with your stakeholders and maintains their engagement

Strategies such as those found in the Evaluation Management Tool (Tool G) are useful for project tracking and ongoing communication. These tools are equally helpful in managing an evaluation with lots of “moving parts.” You are not required to use these tools. However, you may find them helpful in identi- fying emerging issues that require your attention and in making sure you stay on track in terms of timeline and budget.

The tools are designed to help you track prog- ress overall and against your established budget and timeline, identify performance issues by your staff or your contractor, identify implementation issues such as data access and data collection, and moni- tor the quality of your evaluation. Information to help you budget for your evaluation is included in Tool F.

STRATEGY 6: REPORTING RESULTS

Interim Reporting

Where appropriate, sharing interim findings derived from your evaluation not only helps maintain stake- holder interest in the evaluation process but also increases the likelihood that your stakeholders have the information they need in a timely manner. If you decide to share findings midway through the evalu- ation, be sure to couch the interim findings in terms of caveats that the data are only preliminary at this point. Furthermore:

• Share only what information you are comfortable sharing at any given point in time

• Focus on information you feel is important for stakeholders to begin thinking about

• Consider presenting the information as “food for thought” based on what you are seeing thus far

Disseminating Final Results

Dissemination of an evaluation’s final results to stake- holders should be a process tailored to the informa- tion needs of your different stakeholder groups. While final reports are a common way to share findings, it’s

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

198 Par t III: Implementing Evaluations

important to consider whether a large, formal final report is the most appropriate way to disseminate findings to the specific stakeholders with whom you are working.

By “appropriate way” we mean a tailoring of both message and format to the information needs of a given audience; that is, you need to consider the best way(s) to make the information you plan to share accessible to that particular audience. For exam- ple, some stakeholders may strongly desire a final report—they may even need it for documentation or accountability purposes. However, keep in mind that for other stakeholders a final report may include more information than they need or want.

Benefits of interim and final reporting:

• Facilitates appropriate timing of your evaluation in relation to information needs

• Facilitates the comprehension and use of the findings that were derived from your evaluation

• Helps ensure, through interim reporting, that there are few or no “surprises” in the final evaluation report

Figure 9.1 presents a list of some alternative means to disseminate evaluation findings. Depending on the composition of your stakeholder groups, you may want to experiment with one or more of these alternative

approaches. Additional guidance for presenting the results of an evaluation is provided in Tool J.

Remember to set aside resources in your budget to support communication activities—something that is easy to forget to do. The communications por- tion of your budget can be based on the communica- tion ideas put forward in your evaluation plans.

Depending on the communication venue(s) you choose, costs for communication activities might include such things as staff time for materials devel- opment and attendance at stakeholders’ meetings, meeting space, refreshments, printing costs, or web- site maintenance.

Also remember to check with your funders about which of these costs are allowable under your grant(s). Communication may be something your partners can help with in various ways, but if tight resources limit you, then focus on your primary stakeholders.

STRATEGY 7: DEVELOPING A PLAN

Another important step in linking evaluation to action involves developing an action plan containing strategies for implementing evaluation recommenda- tions. The action plan should, at a minimum, contain the following items:

• Rationale for recommended strategies • Clear roles and responsibilities for

implementing the elements of the action plan

Most Interactive with Audience

� Memos and postcards � Comprehensive written

reports � Executive summaries � Newsletters, bulletins,

brochures � News media communications

� Verbal presentations � Videotape or computer-

generated presentations � Posters � Internet communications

� Working sessions � Impromptu or planned

meetings with individuals

Least Interactive with Audience

Figure 9.1: Alternative communication formats.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 9: Preparing for an Evaluation 199

• Timeline • Sources of funding for program or

intervention modifications, if needed

Define roles for stakeholders and community members in the action planning and the action imple- mentation processes. For example, you can convene a “working session” that combines a briefing on find- ings for stakeholders with joint planning on next steps and development of an action plan.

Benefits of action planning:

• Facilitates the comprehension and use of the evaluation’s findings

• Engages stakeholders in the improvement of your program

• Promotes accountability for use of your evaluation’s findings

Involving a variety of stakeholders in the action planning process will help facilitate stakeholder and decision-maker buy-in and thereby facilitate imple- mentation of any recommendations that make sense for your program. Tool K contains an Action Plan template you can adapt to the needs of your own program.

STRATEGY 8: DOCUMENTING LESSONS LEARNED

History repeats itself—because we weren’t listening the first time. That’s as true for evaluation as it is any- where else. Yet by documenting lessons learned from one evaluation for use in future evaluations you can begin building a historical record of knowledge about evaluation to pass on to future “generations” in your program. Consider adopting the habit of closing your evaluation team meetings by asking attendees:

• What have we learned? • What can we do better next time?

Document these discussions in your meet- ing minutes for later reference. In this way you are

encouraging your team members to reflect on their evaluation practice, and this will lead to evaluation capacity building.

Benefits of documenting lessons learned:

• Avoids repeating past mistakes • Builds evaluation capacity among you

and your stakeholders • Transfers knowledge to those who come

after you • Creates an archive of good evaluation

practices over time

As your various evaluations proceed and as you “learn by doing,” make sure you and your team members pause occasionally to reflect upon what you have learned and document those things you want to remember to make your next evaluation go more smoothly. In some cases, you may learn things you would like to share more broadly, for example, through presentations at a grantee meeting, a profes- sional conference, or even in a peer-reviewed journal article.

STRATEGY 9: LINKING BACK TO YOUR EVALUATION PLAN

Linking your evaluation findings back to your evalu- ation plan is a critical final strategy in ensuring an evaluation’s use and promoting ongoing program improvement. It’s not uncommon that an evalua- tion report raises more questions than it answers. This is actually a good thing. In a sense, each evalua- tion you conduct helps you set the agenda for future evaluations.

On planning . . .

Failing to plan is planning to fail. ~Alan Lakein

Findings from your evaluation may suggest, for example, that a particular component of the program

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

200 Par t III: Implementing Evaluations

was functioning well (e.g., a parent training compo- nent) but that another component you touched on only tangentially is functioning less well and should be looked into more closely (e.g., community aware- ness of available parent training classes). Or findings may demonstrate that another component of your program is not working well yet not really explain why that is so or how the problem could be remedied.

The why and how of what isn’t working may then become grist for the mill of a future evaluation. Further, findings regarding issues encountered with the logis- tics of the evaluation itself may suggest that alternative approaches need to be tried in upcoming evaluations.

This is not to say that you need to completely revamp your evaluation plan every time you complete another individual evaluation. Rather, we propose that new information gleaned from each succes- sive evaluation be viewed within the context of your long-range evaluation plans to see if any midcourse corrections are warranted.

While it’s possible that recently compiled find- ings may occasionally imply that a planned evaluation should be scrapped and replaced with one of greater urgency, it’s far more likely that your revised approach will involve only minor modifications to one or more proposed evaluations.

Findings may also help you generate ideas for an evaluation “wish list” pending the next evaluation cycle—or the sudden availability of additional evalu- ation funds. What you want is for your evaluation to continually inform not only your immediate program improvement efforts but also your longer range strat- egies for evaluations. That’s why linking evaluation findings back to your strategic evaluation plan is so critical.

As a last check, before you call an evaluation plan “final” and begin to implement your evaluation, use the checklist in Table 9.3 to see if you have covered all the steps that will help lead to a successful implemen- tation of your evaluation.

Table 9.3. Preevaluation Checklist for the Successful Implementation of an Evaluation Plan.

Category Yes No

Do we have an evaluation planning team composed of individuals with the

knowledge, skills, and experience relevant to planning this evaluation?

Do we have an evaluation implementation team of individuals who will take

responsibility for implementing the evaluation, providing access to data, overseeing

data collection, analyzing the data, and preparing the evaluation report?

Have we identified our key stakeholders for this evaluation? See Chapters 1–3.

Have we thought about how to work with our stakeholders? (Table 9.1)

• Preevaluation?

• During the evaluation?

• Postevaluation?

• To develop the Action Plan (Tool K)?

• To manage public relations?

• To minimize evaluation anxiety (Tool C)?

Will the evaluation design (Tool E) and data collection methods (Tool H) result in …

• Methodology that is feasible given resource and practical constraints?

• Data that are credible and useful to stakeholders?

• Data that are accurate?

• Data that will help answer the evaluation questions in a timely manner?

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 9: Preparing for an Evaluation 201

Spending some “quality time” over a glass of wine—or two—with your evaluation plan will pay off in the long run as you move forward to its imple- mentation. With a solid individual evaluation plan

in hand, you will be in the best possible position to implement an evaluation that meets the standards of utility, feasibility, propriety, and accuracy that were covered in Chapter 4.

Category Yes No

Are we prepared logistically? (Table 9.2)

Do we have plans for …

• Staffing?

• Budget (Tool F)?

• Funding?

• Data sharing and other types of contracts/agreements?

• Human subjects (IRB), HIPAA, and organizational clearances/permissions?

Are we prepared for data collection? (Table 9.2)

Have we addressed …

• Finalization and approval of data collection instruments?

• Propriety of the evaluation, including protection of human subjects?

• Cultural sensitivity, clarity, and user-friendliness of instruments?

• Respondent burden?

• Methods to obtain high response rates or complete data?

• Data handling, processing, storage?

• Data confidentiality, security?

Did we pilot-test our measuring instruments and data collection procedures?

Did we train the data collection staff? (Tool I)

Will the data analyses answer our evaluation questions?

Have we specified the …

• Analyses to answer each evaluation question?

• Table shells that show how the results will be presented?

Do we have methods in place (Tool G) to track evaluation implementation and to

promote communication within the evaluation implementation team?

For example, do we have a …

• Timeline?

• Budget?

• Roles and responsibilities table?

• Project description?

• Project status form?

Have we planned for sharing interim results (if appropriate) and for disseminating

the final results? (See Tool J.)

Table 9.3: Continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

202 Par t III: Implementing Evaluations

Also, by following the strategies described here that relate to stakeholder engagement and sharing results—“Working with Stakeholders,” “Monitor- ing Progress and Promoting Ongoing Communi- cation,” “Interim Reporting and Dissemination of Final Results,” “Developing an Action Plan,” “and “Linking Back to the Evaluation Plan”—you will be better able to translate your evaluation findings into shared action by you and your stakeholders alike.

SUMMARY

This chapter briefly provided the nine basic strategies that need to be followed when you are going to do any type of evaluation. Therefore, when reading the fol- lowing four chapters, keep in mind that the strategies outlined in this chapter must be applied to each one. In a nutshell, they represent important steps you can take during the planning stages of your evaluation that will help you to implement your plans more smoothly.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Study Questions Chapter 9 The goal of this chapter is to provide you with a beginning knowledge base for you to feel comfortable in answering the

following questions. AFTER you have read the chapter, indicate how comfortable you feel you are in answering each

question on a 5-point scale where

1 Very

uncomfortable

2 Somewhat

uncomfortable

3 Neutral

4 Somewhat

comfortable

5 Very

comfortable

If you rated any question between 1–3, reread the section of the chapter where the information for the question is found. If

you still feel that you’re uncomfortable in answering the question, then talk with your instructor and/or your classmates for

more clarification.

Questions Degree of comfort? (Circle one number)

1 List the nine strategies that you need to consider before doing any type of program

evaluation. 1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to your program’s stakeholders before you actually carry out your

evaluation (Strategy 1).

List, and then discuss, the issues you feel need to be addressed in relation to

your stakeholder groups. Provide as many examples as you can throughout your

discussion. And, more important, don’t forget to utilize the tools contained in the

Toolkit when appropriate.

1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work program.

However, you fully realize that you need to address several issues in reference to

developing a good process for managing your evaluation before you actually carry out

your evaluation (Strategy 2).

List, and then discuss, the issues you feel need to be addressed in relation to

developing a process to managing your evaluation. Provide as many examples as you

can throughout your discussion. And, more important, don’t forget to utilize the tools

contained in the Toolkit when appropriate.

1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work program.

However, you fully realize that you need to address several issues in reference to

pilot-testing your data collection instruments before they are used to collect data for

your evaluation (Strategy 3).

List, and then discuss, the issues you feel need to be addressed in relation to

pilot-testing your measuring instruments and data collection procedures. Provide as

many examples as you can throughout your discussion. And, more important, don’t

forget to utilize the tools contained in the Toolkit when appropriate.

1 2 3 4 5

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to training the folks who will be collecting data before they actually collect

them (Strategy 4).

List, and then discuss, the issues you feel need to be addressed in relation to training

your data collectors when it comes to training data collection staff. Provide as many

examples as you can throughout your discussion. And, more important, don’t forget to

utilize the tools contained in the Toolkit when appropriate.

1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to how you are going to monitor the progress of your evaluation in addition

to how you are going to promote ongoing communication within your stakeholder

groups before you actually carry out your evaluation (Strategy 5).

List, and then discuss, the issues you feel need to be addressed in relation to

monitoring your evaluation’s progress in addition to promoting ongoing communication

with your stakeholder groups. Provide as many examples as you can throughout your

discussion. And, more important, don’t forget to utilize the tools contained in the Toolkit

when appropriate.

1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to how you are going to handle interim reporting procedures and the

dissemination of your findings before you actually carry out your evaluation (Strategy

6).

List, and then discuss, the issues you feel need to be addressed. Provide as many

examples as you can throughout your discussion. And, more important, don’t forget to

utilize the tools contained in the Toolkit when appropriate.

1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to how you are going to develop an action plan before you even begin your

evaluation (Strategy 7).

List, and then discuss, the issues you feel need to be addressed in relation to

developing an action plan for your evaluation. Provide as many examples as you can

throughout your discussion. And, more important, don’t forget to utilize the tools

contained in the Toolkit when appropriate.

1 2 3 4 5

Study Questions for Chapter 9 Continued

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to how you are going to document what you have learned from your

evaluation (Strategy 8).

List, and then discuss, the issues you feel need to be addressed in relation to

documenting what you have learned from your evaluation. Provide as many examples

as you can throughout your discussion. And, more important, don’t forget to utilize the

tools contained in the Toolkit when appropriate.

1 2 3 4 5

Pretend, for the moment, that you have been hired to evaluate your social work

program. However, you fully realize that you need to address several issues in

reference to how you are going to link your findings back to your original evaluation

plan even before you begin the evaluation (Strategy 9).

List, and then discuss, the issues you feel need to be addressed in relation to linking

your evaluation findings back to your original evaluation plan. Provide as many

examples as you can throughout your discussion. And, more important, don’t forget to

utilize the tools contained in the Toolkit when appropriate.

1 2 3 4 5

Study Questions for Chapter 9 Continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 9 Assessing Your Self-Efficacy

AFTER you have read this chapter AND have completed all of the study questions, indicate how knowledgeable you feel you

are for each of the following concepts on a 5-point scale where

1 Not knowledgeable

at all

2 Somewhat

unknowledgeable

3 Neutral

4 Somewhat

knowledgeable

5 Very

knowledgeable

Concepts Knowledge Level? (Circle one number)

1 Overall, the nine strategies that can be implemented to increase the overall

success of an evaluation 1 2 3 4 5

2 Working with stakeholders in an effort to increase the overall success of an

evaluation 1 2 3 4 5

3 Developing a process for managing an evaluation in an effort to increase its overall

success 1 2 3 4 5

4 Pilot-testing data collection instruments in an effort to increase the overall

success of an evaluation 1 2 3 4 5

5 Training data collection staff in an effort to increase the overall success of an

evaluation 1 2 3 4 5

6 Monitoring the progress of an evaluation in an effort to increase its overall

success 1 2 3 4 5

7 Writing interim and final evaluation reports in an effort to increase the overall

success of an evaluation 1 2 3 4 5

8 Developing an action plan in an effort to increase the overall success of an

evaluation 1 2 3 4 5

9 Documenting the lessons learned from an evaluation in an effort to increase its

overall success 1 2 3 4 5

10 Linking evaluation findings back to the evaluation’s strategic plan 1 2 3 4 5

Add up your scores (minimum = 10, maximum = 50) Your total score =

A 47–50 = Professional evaluator in the making

A– 45–46 = Senior evaluator

B+ 43–44 = Junior evaluator

B 41–42 = Assistant evaluator

B– 10–40 = Reread the chapter and redo the study questions

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

CHAPTER OUTLINE

WHAT ARE NEEDS ASSESSMENTS?

DEFINING SOCIAL PROBLEMS Social Problems Must Be Visible

DEFINING SOCIAL NEEDS The Hierarchy of Social Needs

FOUR TYPES OF SOCIAL NEEDS Perceived Needs

Normative Needs

Relative Needs

Expressed Needs

SOLUTIONS TO ALLEVIATE SOCIAL NEEDS

STEPS IN DOING A NEEDS ASSESSMENT

STEP 3A: FOCUSING THE PROBLEM Example

STEP 4A: DEVELOPING NEEDS ASSESSMENT QUESTIONS

STEP 4B: IDENTIFYING TARGETS FOR INTERVENTION Establishing Target Parameters

Selecting Data Sources (Sampling)

STEP 4C: DEVELOPING A DATA COLLECTION PLAN Existing Reports

Secondary Data

Individual Interviews

Group Interviews

Telephone and Mail Surveys

STEP 4D: ANALYZING AND DISPLAYING DATA Quantitative Data

Qualitative Data

STEP 6A: DISSEMINATING AND COMMUNICATING EVALUATION RESULTS

SUMMARY

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .