Evaluation models in HRD
HEALTH PROMOTION PRACTICE / October 2000
Evaluation in Practice
Choosing Effective Evaluation Methods
Frances D. Butterfoss, PhD Vincent Francisco, PhD
Ellen M. Capwell, PhD, CHES
A s a practitioner, you have been convinced that you should evaluate your program. You have talked with stakeholders about what purposes they have for the evaluation. Stake-
holders and evaluators have agreed on the elements and purpose of the program to be evaluated as well as the priority questions to be answered. You have focused the evaluation and determined an evaluation design. Now it is time to select appropriate methods for obtain- ing evaluation information.
The many possible methodologies make it diffi- cult for the practitioner to choose. The focus of this is- sue’s column is on identification of basic types of data- gathering strategies, strengths and weaknesses of each, and variables to consider in making choices.
TWO ISSUES IN DATA COLLECTION There are at least two issues that form recurring
themes in evaluation and should be discussed here. The first is the use of quantitative and qualitative data col- lection methods (Bernard, 1998; Green & Lewis, 1986; Patton, 1990; Shadish, Cook, & Leviton, 1991). All de- bates about these issues aside for the moment, each ap- proach has advantages for answering certain kinds of questions. Table 1 has some detailed information about the advantages and disadvantages of qualitative and quantitative measures. An upcoming article in this Evaluation in Practice column will discuss that at greater length. Most of the suggested types of measures discussed here will be a mix of qualitative (e.g., focus group interviews or field observations) and quantitative (e.g., physiological measures and questionnaires) measures.
The second is the use of obtrusive and nonobtrusive measures (Bernard, 1998; Shadish et al., 1991). Obtru-
sive measures are those that are completed in the open, or in plain view, and with the direct knowledge of the participants. Unobtrusive measures are those that are done in the absence of direct awareness by participants, although they do not preclude prior knowledge and per- mission by participants. Obtrusive measures might in- clude surveys and diaries. Unobtrusive measures might include key informant interviews, as well as direct ob- servation of public behaviors.
STRATEGIES FOR GATHERING EVALUATION DATA Although a myriad of data-gathering methods and
variations exists, it may be helpful to begin by grouping them according to obtrusive (interactive) and unobtru- sive (noninteractive) methods. A data-gathering method may involve actual measurement or asking questions of people in an interactive way, or obtaining information without direct participant input. Consideration should also be given to whether qualitative, quantitative, or mixed-method approaches are best for the situation.
Physiological Measures Perhaps the most objective means of gathering eval-
uation information is to take direct physiological mea- surements of individuals (Cooper, Heron, & Heward, 1987; Green & Lewis, 1986). For example, measures of the effectiveness of a 6-month fitness walking program might include blood pressure, serum cholesterol, and recovery heart rate. A program designed to improve the health status of newborns may include measuring the birth weight of all infants born to mothers involved in an experimental program or control group. Although direct physiological measures can provide clear quanti-
307
Health Promotion Practice / October 2000 / Vol. 1, No. 4, 307-313 ©2000 Sage Publications, Inc.
tative data for analysis, be aware of disadvantages as well. These relate to accuracy of measurement instru- ments, skill of those taking the measurements, and con- founding variables related to conditions of the environ- ment or the individual on whom measurements are taken.
Self-Completed Questionnaires A variety of means can be used to ask questions of
program participants or members of the priority popu- lation (Berg, 1998; Neuber et al., 1980; Rutman, 1984). You can use self-completed questionnaires on-site with captive audiences. For example, information about knowledge gained from a group presentation or satis- faction with that presentation can be obtained from written questionnaires completed by program partici- pants. The priority population may not be directly ac- cessible on-site, however. If, for example, you need in- formation about health practices, purchasing patterns, or preferences of individuals located in different com- munities or states, a mailed questionnaire may be ap- propriate. Generally, written questionnaires to be com-
pleted by respondents need to be structured with more closed-ended rather than open-ended questions because many respondents find it challenging and time-con- suming to express their thoughts in writing. Self-com- pleted questionnaires often provide quantitative data. However, qualitative data may be obtained from a se- lect, yet dispersed, group of people with common inter- ests or expertise through means of a Delphi process. This is often used to generate ideas about priorities or future issues. Table 2 summarizes the advantages and limitations of written questionnaires as a means of gathering evaluation data.
Interviews While written responses provide one form of data,
oral questioning and recording by interviewers pro- vides an alternative (Bailey, 1994; Fawcett et al., 1994). Phone interviews allow us to orally question in- dividuals who may be dispersed geographically. The Behavioral Risk Factor Surveillance System is an ex- ample of a method that gathers data by means of phone interviewing. Face-to-face interviews allow more ver-
308 HEALTH PROMOTION PRACTICE / October 2000
TABLE 1 Differences Between Quantitative and Qualitative Approaches
Quantitative Evaluation Qualitative Evaluation
Larger numbers—generalizable to broader population Smaller number of people / cases
Deductive generalizations—objectivity; strength of the Inductive process—phenomenological inquiring; scientific method; experimental/quasi-experimental designs; naturalistic, holistic; understanding the experience in statistical analysis context; content or case analysis
Valid, reliable instrument used to gather data; specific Researcher is the instrument; less rigid protocol administration protocol
Use of standardized measures; predetermined response Able to study selected issues in depth and detail categories
Rigor Flexibility, insight
Results easily aggregated for analysis and easily presented Understanding of what individual variation means; deepening understanding, insights
Can be perceived as biased, predictable, or rigged to obtain Offers credibility of an outsider making assessment certain result
Results easily aggregated for analysis and easily presented Results are longer, detailed, variable in content; difficult to analyze
Data include actual numbers; frequencies/counts of people, Data include group or individual opinions or perceptions; events, systems changes, passage of policy/legislation, relationships, anecdotal comments, assessment of quality; trends descriptions; case studies; unanticipated outcomes
Experimental conditions and designs to control or reduce Openness to variation and multiple directions variation in extraneous variables; focus on limited number of predetermined measures
satility. You can obtain information from individuals while in their homes or workplaces. The census is an example of a face-to-face interview. Table 3 lists ad- vantages and limitations that you might consider to de- termine whether interviews are appropriate for your purpose. The interview approach can also be used to obtain qualitative information about opinions, atti- tudes, or judgements of individuals. Key informant in- terviews or community opinion leader surveys may provide information about perceptions of community
needs that can guide program planning. These inter- views may range from semi-structured (with questions and leading probes for further questioning) to unstruc- tured (with a few very general questions that partici- pants are invited to expound on) depending on the needs of stakeholders.
Self-Completed Diaries or Logs Some evaluation indicators of interest may be best
measured or monitored by means of diaries. Individuals may use diaries to record progress on a behavioral man- agement program or subjective responses to given phe- nomena. For example, participants might be asked to record occasions of stress, their symptoms, and how they handled it as a means of qualitatively evaluating effectiveness of a stress management program. Simi- larly, participants might record their daily intake of fruits and vegetables for a 5-a-day program or incidents of injury in toddlers for a Safe Kids program. You can also establish record-keeping systems, more aligned
Butterfoss et al. / EFFECTIVE EVALUATION METHODS 309
TABLE 2 Written Questionnaires
Advantages • Can be answered anonymously • Can be administered to many people simultaneously;
can be mailed • Allow respondent time to think before responding • Impose uniformity by asking all respondents the same
questions • Risk the least social desirability bias • Are considered highly credible • Are relatively easy to administer • Generate large amounts of data for aggregation and
analysis by computer • Are relatively easy to complete
Limitations • Potential self-report bias • Difficult to obtain information from open-ended ques-
tions; respondents not always able to express them- selves on paper
• Not all questions may be answered • Not appropriate for low literacy populations • Limited by simplicity of questions needed for the gen-
eral public • Difficult to deal with skip patterns without confusing
respondents • If survey is too short, information gathered may be
too general • If survey is long enough to gather needed detail,
respondents may be discouraged • Contamination is possible with mailed surveys
because intended respondent may not be the one who completes the survey
• Difficult to persuade people to complete and return questionnaires
• Mailed surveys are often discarded without responding
• Mailed surveys take time to administer properly • Time and expertise needed to develop valid and reli-
able instruments
TABLE 3 Interviews
Advantages • Flexibility; can probe and pursue unanticipated lines
of inquiry • Can control sequence of questions • Can handle answers to open-ended questions • Can easily use screen questions and follow skip
patterns • Can obtain information from those who have diffi-
culty reading, seeing, or understanding English • Respondent has opportunity for free expression to
reveal attitudes and feelings • Higher response rate than mailed survey • Phone surveys are more economical and efficient than
face to face • Most people have phones • Nonverbal behaviors can be observed and recorded in
face-to-face interviews Limitations
• Difficult to reach intended respondent; new phone features allow respondents to screen calls
• Time-consuming • Considerable training and skill required of
interviewers • Interviewer can intentionally or unintentionally influ-
ence responses as a result of their personal character- istics (age, gender, race), dissimilarity with respon- dent, or skill in interviewing
with quantitative approaches, to record events that meet pre-established criteria. Francisco, Paine, and Fawcett (1994) describe a mechanism for tracking products, services, resources, or media coverage of a community coalition and community actions or changes that result from its efforts. This involves a carefully constructed system involving coalition members logging events as they occur in the coalition and/or community.
Community Forums Community forums provide an excellent means for
gathering qualitative data in the needs assessment phase of an evaluation. Through this method, members of the general public provide input regarding concerns of community residents in a group setting. This may in- volve an open discussion, or a more structured interac- tion. A nominal group process is helpful to use with smaller groups to help prioritize major issues or needs of a group. Any time the priority population is involved in determining program direction, the potential level of support for the resulting program increases. Among the disadvantages, however, is that those who participate in community forums or nominal group processes may not represent the entire community.
Focus Groups Focus groups have become a popular means for ob-
taining qualitative information (Berkowitz, 1982; Delbecq, 1975; Chekki & Sanders, 1980). They are of- ten used in the early stages of program planning or im- plementation to determine the best way to approach an intervention. For example, they can provide guidance in determining the best message to use in a public ser- vice announcement (PSA) that encourages the general public to drink milk. A focus group strategy can also provide insight into barriers that might prevent African American women from obtaining mammograms. Table 4 summarizes advantages and limitations of focus groups (Krueger, 1994).
Direct Observations Asking questions of individuals or groups provides a
record of their words or explanations. Sometimes direct measurement or questioning of those exposed to a pre- vention or intervention program is not feasible. Unob- trusive evaluation measures are useful when interaction with the priority population could alter the accuracy of the evaluation information obtained. Observations al- low us to obtain information from different perspec-
tives about what actually occurs (Cooper et al., 1987; Martin & Pear, 1992). They provide information about how a program operates and its actual effects without directly involving individuals. Observations can range from quantitative tabulation of actions to qualitative field observation of complex social interactions. Exam- ples of quantifiable observations include counting peo- ple in cars who wear seatbelts, students who enter a school building with snuff can rings on their jeans pockets, people who select whole wheat versus white bread in the cafeteria line, number and type of people who use a community walking path, or length of time patients wait in a clinical setting. You can capture a considerable amount of information about what actu- ally happens, not just what is reported, by training inter- viewers to use a carefully developed instrument. Photo- graphs and video recordings offer variations that allow others to observe besides those who actually gather the information. Although there are several advantages to observations, there are also limitations, as outlined in Table 5.
The detailed description gathered from qualitative field observations can enrich understanding of social interactions and dynamics within a group of people gathered for a common purpose. Field observations (Bernard, 1998) can be used to evaluate flow patterns in a clinic, or leadership qualities in a group of children at play, for example. Case studies use a combination of
310 HEALTH PROMOTION PRACTICE / October 2000
TABLE 4 Focus Groups
Advantages • Can probe; explore issues in-depth • Can obtain perceptions and views • Interview framework is easy to design • Require fewer respondents than surveys • High face validity • Socially oriented; more natural environment • Can provide fast results • Can be relatively low cost
Limitations • Require expert facilitation • More than one group is needed for reliability because
groups may vary • May be difficult to access and assemble desired prior-
ity groups • Nonquantitative; may not represent the population • Cannot achieve consensus • Less control compared to individual interview • Data more difficult to analyze
methodologies, including observations and interviews, to gain insights into selected situations or populations.
Extant Data Extant data are sources of information that are al-
ready available. Records kept for purposes that are di- rectly or indirectly related to the program being evalu- ated can provide valuable information. Types of extant data include demographic or geographic characteristics of people, trends in their activity, insurance claims data, enrollment or attendance records, transportation routes and scheduling, media PSA airing, or financial records necessary to conduct cost analyses (Tyler Norris Asso- ciates, 1997). More complex documents may be in the form of portfolios, proposals for funding, program re- ports, minutes or proceedings, policies or legislation, and curricula. As with other evaluation methods, con- sider advantages and limitations of using records to obtain answers to evaluation questions, as outlined in Table 6.
HOW TO DECIDE ON THE BEST EVALUATION METHODS FOR YOUR PROGRAM
The many options referenced above, plus the accom- panying advantages and limitations of each, may leave you, the practitioner, in just as much of a quandary as if no methods had been suggested. Although some would like to have a simple formula for evaluating programs, the diversity in programs and people make it impossi-
ble to prescribe how all evaluations should be con- ducted. Some guidelines for making evaluation deci- sions may help. As you will recall from the second is- sue of Health Promotion Practice, an important element of getting started is to identify your stakeholders—those individuals who determine what questions they want to have answered and what evi- dence will convince them that the program is working. When selecting methods for gathering evaluation data, be clear about what type of information is desired by and acceptable to the stakeholders.
Evaluators can help stakeholders understand what evaluation options are feasible to answer their evalua- tion questions. They need to determine, for example, whether data should document effectiveness, answer specific questions, be generalizable to a larger popula- tion, or provide deeper understanding of an event, atti- tude, or relationship. As Figure 1 shows, consider whether it is reasonable to gather information by inter- acting with those affected by the program. Can partici-
Butterfoss et al. / EFFECTIVE EVALUATION METHODS 311
TABLE 5 Observations
Advantages • Can collect first-hand information about what actually
happens • Permit understanding beyond what can be obtained
through interviews • May provide unanticipated information • Considered highly credible • Provides outside perspective
Limitations • Presence of observers may alter what takes place • Need credible, well-trained observers • Time-consuming to develop instruments, guidelines,
and to train observers • May be time-consuming to conduct multiple
observations • Scheduling problems can occur
TABLE 6 Extant Data
Advantages • Records kept for other purposes may provide valu-
able information • Using existing data saves time and effort of develop-
ing data-collection instruments • Data can be collected in a relatively unobtrusive
manner • Efficiency of time and resources • Considered credible because information was gath-
ered as events occurred Limitations
• Existing records not designed for your specific needs and may not meet all of your requirements
• May require substantial data “cleaning” to be useful • Sensitive information may be missing • Information may be dated • Secondary data may not be reliable or complete • Accessing existing documents may require substan-
tial effort • Examining documents and extracting information can
be time-consuming and expensive • May be ethical or legal constraints in examining
some existing records • Need to train reviewers and validate reviews to
assure reliability and accuracy • Cannot obtain information about relationships or
interactions
pants be asked about it? Are the priority individuals ca- pable of reading or responding to questions? Will interacting with individuals bias the information ob- tained? Both quantitative and qualitative approaches involve asking questions or involving people directly. Also, consider whether the element of interest can be observed. Are there documents already available that may contain information about the element of interest? How concerned are you about bias that may be intro- duced with a particular method? In addition to feasibil- ity, also consider cost and time efficiencies within the parameters provided for the evaluation. Although rec- ommendations can be made based on these consider- ations, consult stakeholders with regard to the accept- ability of proposed methods before a final decision is made.
Figure 1 illustrates that many methods are available for gathering data, either obtrusively or unobtrusively. It also emphasizes the variability possible in ap- proaches, ranging from quantitative to qualitative or a combination of both. Using more than one method and
including both qualitative and quantitative approaches to address stakeholders’ questions can enrich and strengthen an evaluation.
The overview and concepts presented here are in- tended to provide a simplified look at possible evalua- tion methods and one model for conceptualizing the re- lationship of each to a qualitative or quantitative approach and to the stakeholders who ultimately direct the evaluation. For each method mentioned, important guidelines have been developed for application. These will be addressed separately in future Evaluation in Practice columns.
REFERENCES Bailey, K. D. (1994). Methods of social research. New York: Free Press.
Berg, B. L. (1998). Qualitative research methods for the social sciences. Boston: Allyn and Bacon.
Berkowitz, W. R. (1982). Community impact. Cambridge, MA: Schenkman.
Bernard, H. R. (Ed.). (1998). Handbook of methods in cultural anthropol- ogy. Walnut Creek, CA: AltaMira.
312 HEALTH PROMOTION PRACTICE / October 2000
FIGURE 1 Options for Selecting Evaluation Methods
Chekki, D. A., & Sanders, J. T. (1980). Community development: Theory and method of planned change. New Delhi: Vikas Publishing.
Cooper, J. O., Heron, T. E., & Heward, W. L. (1987). Applied behavior anal- ysis. New York: Macmillan.
Delbecq, A. L. (1975). Group techniques for program planning. Glenview, IL: Scott, Foresman.
Fawcett, S. B., Paine-Andrews, A., Francisco, V. T., Schultz, J. A., Richter, K. P., Lewis, R. K., Williams, E. L., Harris, K. J., Berkley, J. Y., Fisher, J. L., & Lopez, C. M. (1994). Work group evaluation handbook: Evaluating and supporting community initiatives for health and development. Lawrence: Work Group on Health Promotion and Community Development, Univer- sity of Kansas.
Francisco, V. T., Paine, A. L., & Fawcett, S. B. (1994). A methodology for monitoring health action coalitions. Journal of Health Education Research: Theory and Practice, 8(3), 403-416.
Green, L. W., & Lewis, F. M. (1986). Measurement and evaluation in health education and health promotion. Palo Alto, CA: Mayfield.
Krueger, R. A. (1994). Focus groups: A practical guide for applied research (2nd ed.). Thousand Oaks, CA: Sage.
Martin, G., & Pear, J. (1992). Behavior modification: What it is and how to do it. Eaglewood Cliffs, NJ: Prentice-Hall.
Neuber, K. A., Jacobson, J. A., & Reteurman, N. A. (1980). Needs assess- ment: A model for community planning. Beverly Hills. CA: Sage.
Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.
Rutman, L. (Ed.). (1984). Evaluation research methods: A basic guide. Newbury Park, CA: Sage.
Shadish, W. R., Jr., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.
Tyler Norris Associates, Redefining Progress, and Sustainable Seattle. (1997). The community indicators handbook: Measuring progress toward healthy and sustainable communities. Boulder, CO: Tyler Norris Associates.
Frances D. Butterfoss, PhD, is at the Center for Pediatric Research, Eastern Virginia Medical School, Norfolk, VA.
Vincent Francisco, PhD, is in the Work Group on Community Devel- opment and Health Promotion, University of Kansas, Lawrence, KS.
Ellen M. Capwell, PhD, CHES, is at Otterbein College, Westerville, OH.
Butterfoss et al. / EFFECTIVE EVALUATION METHODS 313