615 forum
388 Copyright © SLACK Incorporated
administrative angles Associate Editors: Pamela S. Dickerson, PhD, RN-BC, FAAN
Barbara G. Lubejko, MS, RN Author:
Associate Editors: Pamela S. Dickerson, PhD, RN-BC, FAAN Barbara G. Lubejko, MS, RN
Author: Barbara G. Lubejko, MS, RN
Developing a Program Evaluation Plan: Options and Opportunities
Figuring out the best way to evaluate educational programs is one chal- lenge that has arisen with implementa- tion of the new requirements for orga- nizations accredited by the American Nurses Credentialing Center (ANCC, 2016). The usual reaction-based, learner-completed evaluation forms are not going to provide data needed to evaluate a majority of outcomes, so it is time to think about program evaluation in a new way.
DEVELOPING AN EVALUATION PLAN
Program evaluation needs to be a thoughtful process begun early in
planning and refined as decisions are made about the desired outcome, content, format, and teaching meth- ods. As opposed to an afterthought, the evaluation method and data to be collected should be determined based on a carefully thought-out plan. To determine what data to collect and the most appropriate way to collect it, consider six questions (Table).
To begin, think about the type of gap that was identified and the de- sired outcome for the program. Was the gap identified one of knowledge, skill, or practice, or a combination of these? This will help you determine the best learning approach, as well as your options for evaluation.
As you are planning, logistical issues will also play a role in determining the best evaluation approach to take. You will need to determine the types of data you can realistically collect from your audience. This is dictated not only by the data you need to determine wheth- er your learners have met the program outcome, but also by the kind of ac- cess you have to your audience. Will you be interacting with your audience in person, or will your audience be at a distance? If you are planning a distance program, will there be opportunities
for live interaction, or is all learner en- gagement through a learning manage- ment system or computer interface? Also, how large is your learner group? Will there be opportunities for individ- ual interactions?
After you have determined the data you will collect, you will need to de- termine the best methods by which to collect the data and when. Should you collect data during or immediately af- ter the completion of your program, or should you wait and collect informa- tion about the impact of your program at a later point in time? Should you col- lect data directly from the learners, or are there better sources to demonstrate whether your learners have been able to achieve your desired outcome?
Finally, you should consider how the data you collect will be used. There can be a tendency to collect as much data as possible while you have the learner’s attention. However, col- lecting data that will not be used for any specific purpose is likely not the best use of time. Instead, focus on data that will tell you whether the gap has been narrowed and what more you can do to help the learners achieve the desired outcome. In addition, think about what data will be important to share with your stakeholders to dem- onstrate the efficacy of the program and value of your efforts.
METHODS OF EVALUATING PROGRAM OUTCOMES
Evaluation data can be collected in many ways. Some of these methods are considered learner-engagement techniques, as opposed to evalua- tion methods. However, any inter-
Ms. Lubejko is Oncology Clinical Specialist, Oncology Nursing Society, Pittsburgh, Pennsyl- vania.
The author reports personal fees from American Nurses Credentialing Center outside the submitted work. The author reports no other conflicts of interest, financial or otherwise.
Dr. Dickerson is a commissioner and Ms. Lubejko is an appraiser with the ANCC’s Commis- sion on Accreditation. Views expressed by them or their guest authors are their own and are not representative of the Commission, except as specifically noted.
Address correspondence to Barbara G. Lubejko, MS, RN, Oncology Clinical Specialist, On- cology Nursing Society, 125 Enterprise Drive, Pittsburgh, PA 15275; e-mail: blubejko@ons.org.
doi:10.3928/00220124-20160817-02
abstract Planning how to evaluate edu-
cational programs needs to be a thoughtful process that begins early in program planning. By considering a series of questions and choosing between direct and indirect out- come measures, the evaluation pro- cess can provide useful data about successes and opportunities. J Contin Educ Nurs. 2016;47(9):388- 389.
389The Journal of Continuing Education in Nursing · Vol 47, No 9, 2016
action that allows you to get a sense of changes in knowledge or skill can provide valuable outcomes data.
Traditional evaluation methods have tended to rely on indirect eval- uation of program outcomes, such as through learner self-report. For some types of educational programs, asking learners whether they have gained knowledge, improved their skills, or intend to change their prac- tice may be the best data available. This, in addition to questions about what else they need to further close the identified gap, can provide valu- able information when more direct methods of evaluation are not realis- tic. Other methods that provide less direct measures of outcomes include debriefing at the end of a program and follow-up surveys or interviews.
Whenever possible, it is best to use more direct methods for evalu- ating outcomes. When a program is designed to close a knowledge gap, some options include pre-, post-, and follow-up tests done after the learner has returned to practice. Other op- portunities to evaluate for increase in knowledge include case studies with learner participation, polling the au- dience (e.g., a show of hands or an audience response system), or role- play. Role-play can also be beneficial in evaluating for the improvement in skills along with demonstration or return demonstration and simulation with learner involvement. To evaluate a change in practice, you will most likely need to use a more long-term evaluation method, such as obser- vation of practice, chart reviews, or checking for a change in results on quality measures (Ford, 2010; Lube- jko, 2015).
EXAMPLE An example of this could arise
when planning a legislative action ex- perience for your organization. You have found that many nurses are not currently interacting with their leg-
islators and, when queried, identify a lack of knowledge of the political pro- cess and methods by which to effectively communicate with their elected offi- cials. They also lack the necessary skills to express their con- cerns quickly and concisely, in the time allotted for a legislative visit. To help close this gap, you plan a blended learning activity with the following desired outcome. By the end of the program, the learner will develop and present a 1- to 2-minute pitch (i.e., el- evator speech) to one of their elected officials.
You have your participants pre- pare for the in-person part of the program by doing some required reading and by identifying a key issue they would like to address with their elected official. On the day of the in- person portion of the program, you plan an interactive session to provide them with the content indicated by your gap analysis, followed by hav- ing them draft their elevator speech. They then meet in small groups to practice and receive feedback about their elevator speech. In the after- noon, participants meet with one of their elected officials and present their elevator speeches. At the end of the day, you hold a structured debriefing to gauge the success of their legisla- tive visits and have participants draft letters thanking the officials they met.
You decide on a two-part evaluation plan. For the short term, you monitor the small groups as they practice and provide feedback about their draft el- evator speeches. You also use the struc- tured debriefing to validate that they presented their elevator speech to their elected official and elicit information
about whether they felt that they were successful. For more long-term evalua- tion, you send a short survey to partici- pants to see whether they received any response to the letter of thanks they sent to their elected official.
CONCLUSION Program evaluation is an essential
part of any educational activity. It pro- vides the opportunity to collect data about the effectiveness of the activity in closing a practice gap and identify areas of additional need. However, with the new requirements for organi- zations accredited by ANCC (2016), providers need to think about evalu- ation in a new way. Although this is challenging, developing an evaluation plan that integrates the desired out- come, content, format, and teaching methods provides opportunities for the collection of more useful data that can guide future educational efforts.
REFERENCES American Nurses Credentialing Center.
(2016). 2016 ANCC Primary accreditation provider application manual. Silver Spring, MD: Author.
Ford, D.J. (2010). Planning your evaluation project. In P.P. Phillips (Ed.), ASTD hand- book of measuring and evaluating training (pp. 29-51). Alexandria, VA: ASTD Press.
Lubejko, B.G. (2015). Using gaps to design educational programs. The Journal of Con- tinuing Education in Nursing, 46, 241-243.
TABLE
CONSIDERATIONS FOR PUTTING TOGETHER AN EFFECTIVE EVALUATION PLAN
1. What type of gap was identified (e.g., knowledge, skill, practice)?
2. What is the desired program outcome?
3. What type of learning approach can I take to overcome this gap?
4. What types of data can I reasonably collect from my audience?
5. How can I best collect that data? At what points in time should I access this data?
6. How will I use the data I collect?
Adapted from (Ford, 2010) and (Lubejko, 2015).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.