Discussion 3

The goal of program evaluation is to locate and examine evidence that will indicate the level of program success. However, the success of the evaluation is predicated upon utilizing a procedural framework that is useful, feasible, proper, and accurate. The procedural framework discussed below is a practical, adaptable approach that identifies required elements of program evaluation.

Step 1: Describe the institutional context of the program.

Useful program evaluation . . . must begin with an understanding of why the program is thought to be necessary and what needs are being addressed. Both the why and what are intrinsic to institutional values and are [typically] articulated in its mission and goal statements. Thus, in higher education, systematic program evaluation must begin with understanding the nature and purpose of the institution. (Grayson, 2012, p. 463)

Awareness and "sensitivity to each program's unique context (for example, the program's history and organizational climate) is essential for [effective program] evaluation (Milstein & Wetterhall, 2014). Being cognizant of the overarching organizational structure, culture, and political environment will serve as a starting point around which the program evaluator(s) can tailor the evaluation to best serve and meet the needs of the academic community (Milstein & Wetterhall, 2014).

Step 2: Clarify the program's purpose.

Programs are results of deliberate allocations of resources in support of specific strategies or activities to produce defined services . . . These services . . . are intended to address strategic problems or issues and may be considered necessary to fulfilling the institution's mission. In other words, programs are intended to produce desired outcomes. Describing a program [with regard to its] resources . . . activities, [components, outputs, participants,] outcomes, and impact [defines] a program's logic [or purpose]. (Grayson, 2012, p. 465)

When these elements are placed together in such a way as to demonstrate relationships (for example, between activities and expected benefits, or between changes in participants and addressing an issue or problem), a clear understanding of the program's purpose can be gained.

Step 3: Identify and engage stakeholders.

Stakeholders are:

supporters, implementers, recipients, and decision-makers related to your program. Getting them involved early on will help you get different perspectives on the program and establish common expectations. This helps to clarify goals and objectives of the program you'll evaluate, so everyone understands its purpose. (NWCPHP, 2014)

In an effort to foster input, participation, and power-sharing among those who have an investment in the conduct of the evaluation and the findings, it is especially important to engage primary users of the evaluation. Doing so will help increase the likelihood that the evaluation will be useful, and can improve the evaluation's credibility, clarify roles and responsibilities, enhance cultural competence, help protect human subjects, and avoid real or perceived conflicts of interest. Activities used to engage stakeholders might include (CDC, 2014):

· Consulting insiders (e.g., leaders, staff, clients, and program funding sources) and outsiders (e.g., skeptics);

· Taking special effort to promote the inclusion of less powerful groups or individuals;

· Coordinating stakeholder input throughout the process of evaluation design, operation, and use; and

· Avoiding excessive stakeholder identification, which might prevent progress of the evaluation.

Step 4: Define the purpose and design of the evaluation.

"Identifying a clear purpose helps determine how to conduct the evaluation" (Instructional Assessment Resources, 2014). Generally, there are three purposes for program evaluations. First and foremost, identifying the purpose affords the opportunity to gain insight and to clarify how program activities should be designed to bring about expected changes. Second, identifying the purpose of the evaluation promotes improvement in the quality, effectiveness, or efficiency of program activities. And third, identifying the purpose of the evaluation helps to examine the relationship between program activities and observed consequences (Adapted from Instructional Assessment Resources, 2014).

Understanding the purpose of the program evaluation brings into focus the:

intended uses [or] specific ways evaluation results will be applied. They are the underlying goals of the evaluation, and are linked to the central questions of the study that identify the specific aspects of the program to be examined. The purpose, uses, and central questions of [a program] evaluation are all closely related. (Instructional Assessment Resources, 2014)

The evaluation [design] outlines how to implement the evaluation [and may include components such as] identification of the sponsor and resources available for implementing the design, what information is to be gathered, the research method(s) to be used, a description of the roles and responsibilities of sponsors and evaluators, and a timeline for accomplishing evaluation tasks. (Instructional Assessment Resources, 2014)

Step 5: Identify evaluation questions and criteria.

"It is important to craft [program] evaluation questions clearly and completely. Evaluation questions take many forms. . . .  Questions are often broader and are focused on the larger picture or goals of the program rather than a small component of it" (Wall, n.d., p. 3), for example, "How well has the career services program prepared graduates to be successful in their field of study?"

Program evaluation questions can be inspired by several sources, which may include program objectives, program goals, strategic plans, needs assessments that have been conducted, inquiries and priorities from an advisory council, or the mission statement of the college or university (Adapted from Wall, n.d., p. 4). "After you have written the evaluation questions, define the criteria for the evidence you needed to answer the questions. Consider the following examples" (Adapted from Wall, n.d., p. 5):

Step 6: Locate, collect, and analyze credible, relevant data.

This step is about the practical details of gathering evidence to improve [the program] and prove its value to the stakeholders. One aspect . . . to consider prior to reviewing data collection methods, is the difference between quantitative and qualitative approaches. Most [program] evaluations deal to some extent with quantitative information [numerical; things that can be counted,] but the assumption is often made that qualitative information [cannot] be measured. In reality numerical techniques can be essential to the interpretation of qualitative data. Some stakeholders such as senior managers have a preference for summarized, quantified results.

Before deciding on methods for data collection consideration should be given to where data may be found. Make as much use of existing sources (secondary data) as is realistically and credibly possible. It will save time and effort, and may also extend the possibilities for comparative analysis. Primary data (new [data]) will be needed to ascertain the effect of particular program activities such as the implementation of new processes. (Glenaffric Ltd., 2007, p. 13)

Much has been written and is available regarding data collection methods. The final choice should be influenced by a number of factors, including the availability and experience of evaluation participants to gather data and analyze results; time; availability of and access to data; and the type of program and its context (Adapted from Glenaffric Ltd., 2007, p. 13). Further, the type of data collected will largely dictate the type of analysis that is done. For example, if there are lots of numbers, utilize a spreadsheet or heavyweight data analysis tool (Glenaffric Ltd., 2007).

The analysis of evaluation data is not simply about reporting findings and letting the results speak for themselves. The key difference from conventional research is the value element - the results need to be interpreted in terms of context and the stakeholders' views. This does not mean compromising findings by skewing results to take account of vested interests but it does require an integrated and balanced approach. (Glenaffric Ltd., 2007, p. 16)

Step 7: Justify and report evaluation conclusions.

When stakeholders agree that the conclusions drawn are justified they will be more inclined to make use of the evaluation results. This may be complicated by differing views but the early involvement of the stakeholders in the evaluation and the inclusion of their views should help to facilitate a consensus being reached. (Glenaffric Ltd., 2007, p. 16)

An essential purpose of program evaluation is to use the information to continuously improve programs while they are running, to draw out the value of what has been learned, and to provide guidance for future programs. The results of the evaluation can be used to demonstrate the effectiveness of the program, identify ways to improve future programs, modify program planning, demonstrate accountability, and justify funding. Anchor all interpretation to the original program evaluation questions. Keep audience(s) in mind when preparing the report. Consider any specific interest in the program evaluation. Include the limitations of the evaluation in the final report. Compare and contrast evaluation results of similar programs and, if applicable, the results of different parts of the evaluation. To illustrate results, use simple visuals whenever possible. Create shorter and/or otherwise tailored versions of the final report for different stakeholders, depending on how they plan to use the information (Adapted from Ontario Agency for Health Protection and Promotion, 2011, p. 3).

Step 8: Ensure use and share lessons learned.

At this stage . . . share the [program] evaluation report and recommendations with relevant stakeholders. These decision makers will play an integral role in whether program improvements or alterations are implemented. The recommendations, along with stakeholder input, will drive which actions are implemented [and when]. (eXplorance, 2013)

Make changes to improve the program based on the evaluation results. Involve stakeholders to determine what program adjustments are warranted. Create a list of recommended actions that address the outcomes of your evaluation. Prioritize those changes that are most important and feasible to implement. Set up an action plan to implement the recommended changes. Program changes can range from small improvements to the curriculum to strategic decisions such as ending the program. Regardless of the actions, "ensure all aspects are considered such as impact, policy implications and budget"(eXplorance, 2013).

References

Centers for Disease Control and Prevention (CDC). (2014). Engaging stakeholders. Retrieved from http://www.cdc.gov/EVAL/steps/engagingstakeholders.PDF

eXplorance. (2013, October 10). The 8 step program evaluation process: Make improvements. Retrieved from http://www.explorance.com/blog/2013/10/8-step-program-evaluation-process-part-2/

Glenaffric Ltd. (2007). Six steps to effective evaluation: Analyze results. Retrieved from http://www.jisc.ac.uk/media/documents/programmes/digitisation/SixStepsHandbook.pdf

Grayson, T. E. (2012). Program evaluation in higher education. In D. B. Denison & C. Secolsky (Eds.), Handbook on measurement, assessment, and evaluation in higher education

Instructional Assessment Resources. (2014). Evaluate programs: Program evaluation process.The University of Texas at Austin. Retrieved from https://www.utexas.edu/academic/ctl/assessment/iar/programs/plan/why-process.php

Milstein, B., & Wetterhall, S. (2014). A framework for program evaluation: A gateway to tools.Community Tool Box. Retrieved from http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/framework-for-evaluation/main

Northwest Center for Public Health Practice (NWCPHP). (2014). Six steps of program evaluation. Retrieved from http://www.nwcphp.org/evaluation/tools-resources/program-evaluation-tips

Ontario Agency for Health Protection and Promotion. (2011). The ten steps to evaluating a health promotion program: Interpret and disseminate results. Retrieved from http://www.thcu.ca/infoandresources/publications/evalaagtablev0.4.pdf

Wall, J. E. (n.d.). Program evaluation models: 9-step process. Retrieved from http://region11s4.lacoe.edu/attachments/article/34/(7)%209%20Step%20Evaluation%20Model%20Paper.pdf