Nutrition

standard is used as a basis for the level of profici. ency

required. Instead of comparing learnerS with each other. the instructor compares each individual . with

a predefined, objective standard of performance.

what the learner is expected to know or to be able to do after instruction is complete. A criterion-ref. erenced measurement ascertains the person's statius in respect to a defined objective or standard, and test items, if tests are used, correspond to the objec. tives. If the learner can perform what is called forin the objective, he or she has been successful. If not. criterion-referenced testing which tends to be more diagnostic, indicates what the learner can and cannot

of

do, and more learning can be planned. Some instructors may believe that a test should

Educational outcomes should be not be too easy, but the degree of difficulty of a test may not be as important as whether a person can

perform. The instructor may believe that some of the questions have to be difficult so that a spread of scores is produced to separate the brightest from the

rest, the As from the Bs and Cs. Some tests are developed with the intent that not everyone will be successful and variation in individual scores is expected. Students are graded in a norm-referenced manner by comparison with other individuals on the same measuring device or with the nom of the group. A norm-referenced instrument indicates, for example, whether the individual's performance falls into the 50th percentile or the 9Oth per-centile in relation to the group norm. This method is not as appropnate

assessed.

Source: CDC Amanda Mills.

for affective and psychomotor objectives. With criterion-referenced evaluation, everyone can do well by attainins a minimum standard. Instruction has been successful when learners reach a defined level of expertise. The registration examinations for dietitians and for dietetic technicians are examples of criterion-referenced tests. Formative evaluation is almost always criterion-referenced. The instru tor wants to know who is having trouble learning, not where they rank compared with ohers. Summative evaluation may be either norm- Ot criterion-referenced.27

Types of Evaluation and Outcomes After considering the purpose (why) and timing (when) of evaluation the educator should resolve the question of what to evaluate. se types of evaluation can be used in measuring effectiveness. These are measurement of participant (client, employee) reactions to progra ms;

318

(2) measuremcnt of behavioral change: (3) measurement of results in

Partica cipant Reaction to Programs

what should be evaluated. Were participants pleased and satisfied with the

NLUAING LEARNING

319 ation; (4) evaluation of learning in the cognitive, affective, and ps

an sychomotor domains; and (5) evaluation of other outcomes. The

in

know aledee, attitudes or beliels, change in behavior, and other measures.

tion of health education is usually focused on one or more types:

The first type of evaluation deals with participant (employee, client) re-actions to educational programs and whether or not they are favorable oferences may vary by age ot the partiCipants, cultural or ethnic group, nder, socioeconomiC status, and other variables. You need to decide ram., subject nmatter, content, materials, speakers, room arrangements, physical facilities, and learning activities? When a program, meetingg or class is evaluated, the purpose is to improve decisions concerning its various

SDects, to see how the parts fit the whole, or to make program changes. asp The quality of learning elements, such as objectives, techniques, ma-

terials, and learning outcomes, may also be included. Hedonistic scales or happiness indexes, such as smiley faces or numerical scales, have been used to determine the degree to which participants "liked" various aspects. Although these judgments are subjective, they are not useless, since learners who dislike elements of a program may not be learning.

Behavioral Change

A second type of evaluation is the measurement of change in behavior. Did

employee or client behavior or habits change based on the learning? In mea- suring behavior, the focus is on what the person does. In employee training for example, you may assess changes in job behaviors to see whether transfer of training to the job has occurred. Continual quality improvement has

influenced the need for this type of evaluation. It is necessary to know what

the job performance was before training and to decide who will observe or

dSsess changed performance-the supervisor, peers, or the individual. This

ype or assessment is more difficult to measure and can be done selectively. he ultimate criterion for effectiveness of nutrition education is not

erely the improvement in knowledge of what to eat, but also changes

ndietary behaviors and practices as the individual develops better tood

nabits. Is the person consuming more fruits and vegetables, tor examples

ce changes are difficult to confirm and often depend on direct obser-

,Which is time-consuming; on self-reports; and on indirect outcome

dsures, such as weight gained or lost in a person on a weight reduction

t,reduction in blood pressure in hypertensive persons, or better controt

of blood sugars in diabetes mellitus.

320 /EDU

zation. Professionals involved with employee training gather a third

ositively ben

Organizational Results

type of the organ evaluative data to justify the time and expense to the OrpaPe

fit the

of

Management may want to know how training will positivelv alion

ms of the follov organization in relation to the cost. Results in terms of the ft

mora

the

improved efficiency or productivity, improved quality of worle,

work

aspects may be attributed, at least in part, to training: improved mang

better customer satisfaction, less employee turnover, tewer accidents ar.

ber of er's compensation claims, better attendance, dollar savings, numho. employee errors, number of grievances, amount of overtime, and the l. ke. Did changing employees' behavior on the job improve businesS resulhe

If not, it is not useful.

Learning Whether learning has taken place is a separate question, even if the pro-

gram rated highly on entertainment value. The learning of principles, facts, attitudes, values, and skills should be evaluated on an objective basis, and

this task is more complex. If the learning objectives are written in terms of measurable performance, they serve as the source of the evaluation. To what degree were the objectives achieved by the learner

Whether a person has succeeded in learning can be determined by de- veloping situations, or test items, based on the objectives of instruction. A program is ineffective if it has not achieved its objectives. It is important for the test items to match the objectives in performance and conditions discussed in Chapter 11. If they do not match the objectives, it is not possible to assess whether instruction was successful, that is, whether the learner learned what was intended.

Mager pointed out that several obstacles must be overcome to assess the results of instruction successfully. Some obstacles are caused by poony written objectives, whereas others result from attitudes and beliets on tne part of instructors who use inappropriate test items. One of the problems in evaluation results from inadequately write objectives. If the performance is not stated, if conditions are omitted, an if the criterion is missing, it will be difficult to create a test situation." these deficiencies are discovered, the first step is to rewrite the objecu Mager suggested a series of steps to select appropriate test items 1. Note the performance (what the person will be able to say or do) sa

in the objective. Match the performance and conditions of the test hose of the obiectivpeormance and c say or do) stated to those of the objective.

2. Check whether the performance is a main intent or an indicator. It performance is the main intent, determine whether it is covert (invi> or overt (visible, audible). ible)

12/IMPLEMENTING AND EVALUATING LEARNING 321

nerformance is covert, such as solving a problem, check for an

3. If the perfor

indicator behavior, a visible or audible activity by which the performance

can be inferred.

Test for the overt indicator in objectives containing one rather than the 4.

main performance.

aso Arst step is to see whether the performance specified in the test item

t the same as that specified in the objective. If they do not match, the test

been accomplished. If the objective states that the performance is "to plan item must be revised, since it will not indicate whether the objective has

low-fat menus" or "t operate the dish machine," for example, the test

hald involve planning menus or operating the dish machine. It would be

or to label the parts of the dish machine on a diagram.

In addition to matching performance, the test should use the same

specific circumstances or conditions that are specified in the objective.

inappropriate to ask the learner to discuss the principles of writing menus

EXAMPLE (Given the disassembled parts of the meat slicer) is able

to reassemble the parts in correct sequence.

The conditions are "given the disassembled parts of the meat slicer."

The practitioner should provide a disassembled machine and ask the em-

ployee to reassemble it. An inappropriate test would be to ask the learner

to list the steps in reassembling the meat slicer or to discuss the safety

precautions to be taken.

If the learner must perform under a range of conditions, you may need

to test performance using the entire range. Ifa client eats at home and in

restaurants, the dietetics professional must determine whether the person

1s capable of following the dietary changes in both environments. If stu-

dents are learning to take a diet history, they should be taught to handle

the range of conditions, including people of different ages, socioeconomic

evels, and cultural groups. Not every condition will be taught and testea,

ne common conditions that the individual will encounter should be

included in the objectives and in testing.

he main intent of an obijective may be stated clearly or it may be

mplied. The main intent is the performance, whereas an indicator 1s an

acuvity (visible, audible) through which the main intent is inferred:

EXAMPLE (Given a copy of a sodium-restricted diet) is able to pilan a

menu for a complete day.

822 3/EDUCATION SKILLS

n foods per mitted and omitted on the diet, and the indicator is the ability

what

per

is not if accurate sodium-restricted menus are planned Test for

prove that the

In this example, the main intent is to discriminate between.

to plan

. Test for thei menus. You can infer that the client knows what i is permitted and plan

ndicator in objectives that contain one. This, of course, does not

ities, Covert actions are not visible, but are internal or mental activis:

Chapter 11, and

person will change eating behaviors.

Such as solving problems or identifying. Ifthe pertormance is covert, an Such

ndicator should have been added to the objective, as explainedi in Chapter 1dlor

the indicator should be tested.

EXAMPLE Is able to identify the parts of the slicer (on a diagram or

verbally).

For this example, the employee should be provided with the indicator

a diagram of a meat slicer, and asked to identify he parts. Although some performances are covert, others are overt. Overt actions

are visible or audible, such as writing verbally describing, and assembling If the performance is overt, determine whether the test item matches the

objective.

EXAMPLE Is able to reassemble the parts of the meat slicer.

The employee should be provided with the parts of the meat slicer and asked to reassemble them. Performance tests are appropriate when skills are taught. If the employee is being taught to use equipment, the evaluation should be to have him or her demonstrate its operation. Ir a student is learning interviewing skills, an interview session is indicated as the evaluation.

The discussion so far has used examples of objectives in the cognitve and psychomotor domains. Affective objectives describe values, interesis, and attitudes that are thought to predispose dietary changes. Wnie cognitive and psychomotor domains are concerned with what indiviauai can do, the affective domain deals with what they are willing to do. In changes are covert or internal and develop more slowly over a periou time. Evaluation of their achievement is more difficult and needs to a different forms.

r do. Attitudes are inferred based on the evidence of what people say To assess whether the individual has been influenced by education, th

says or observe what he or she does, since both saying and dob

dividua. professional may conduct a discussion and listen to what the in r

12/IMPLEMENTING AND EVALUATING LEARNING 3233 h aviors. In measuring atlitudes and values, the person needs the

overt belh

Dportunity to express agreemen rather than deciding on right or wrong A self-reported atitude survey may be used, for example. State-

opp

answers. As

ngly agree" to

hchavior, the practituoner attempts to secure data that permit an inference

"strongly agree" to "strongly disagree." To evaluate change pont

in the scale,

learner's from

ments an be given to which he person responds on a 5-point scale, from

to he made regarding the person's future disposition in similar situations. In the affective domain, this is a more difficult task.

lt is conceivable that the individual may display a desirable overt behav.

iar only in the presence of the practitioner. The attitude toward following

diabetic diet or an employee work procedure may differ depending on

the dietetics professional s presence or absence. Since time is required

for change in attitudes and values, evaluation may have to be repeated at

designated intervals. To determine realistically how the person is dispose

to act, the measurement approach needs to evaluate volitional rather than

coerced responses.

Other Outcomes

An outcome is a result and can be defined as what does or does not hap-

pen after an intervention. The criterion of nutrition education program

effectiveness has generally been improvement in knowledge, in awareness,

and in dietary behaviors or physiologic parameters, or both. This criterion

can be measured in many ways depending on the application and outcome

data available. For professional education programs, the use of hard copy

and electronic portfolios representing evidence of skills and competency

28-30

Is one way to assess outcome measures.

SELF-ASSESSMENT 1

TOU have just discussed sodium restriction with a man diag-

Ses with hypertension. How can you assess what he has

learned?

Outcomes should have clear interpretations related to the dietitian's in-

vEntion in improving nutrition and health status. They may be of several

ypes:(1) physiologic or biologic measures, (2)

behavioral change based on

E-report, (3) diet-related psychosocial

measures, and (4)

environmental

or other measures of dietary behavior. Biologic

indicators are changes in

cal or biochemical indices, such as serum lipid levels in cardiovascuid

disease, hemoglobin or serum albumen level in pregnancy,

and glycosylated

fat intake or increasing fiber intake are

based on self-reports,

which can be

hemog oglobin level in diabetes. Eating behavior changes

such as decreasing

324 3/EDUCATION SKILLS

knowledge, change in on

attitude change, or self-efficacy for behavior but do not prove the

food choices. Other changes are in body mass index or weight, inc

the level of physical activity, decreased blood pressure, or reduction in

factors for disease and improved health (both long-term goale in r

subject to bias. Psychosocial outcomes include itncreased nutrition b

creases in in risk

ince they Care must be taken in interpreting some of these results e

can affect may reflect variables other than education. Stress, for example

diet is followed. nutri ferent

n a person's blood sugar even when the diabetic

ways ranging from observable food choices to dietary intakes. Thes ese may include reports by teachers or parents of children's food preferences, such

tion education interventions, behavior has been measured in

as refusing a food, willingness to taste a new food, and selectinouh nutritious food when other choices were available. Actual food choices more

onsumption, plate waste, and self-reported intake can be used to e

dietary intake. Other measures include 24-hour dietary recall, food rec and food frequency questionnaires, changing food preparation practicee

and recipes, or percentage of participation after an intervention,

Physical measures can include laboratory values, blood pressure, weicht

indices, urinary output, and physical activity status. Mean maternal weight gain, infant's birth weight, and Apgar scores at birth can be used to evaluate

pregnancy outcomes and the health of newborn infants.24 Organizational changes include changes in school lunch menus, such

as to lower fat and sodium, or food choices and nutrition information of.

aluate

fered at the worksite. Data can be collected on the number of work-related injuries or food sanitation incidents after safety training

Data Collection Techniques There are many techniques for collecting evaluation data: paper-and-pen tests, questionnaires, interviews, visual observation, job sample or pertor mance tests, simulation, rating forms or checklists, individual and grou

performance measures, individual and group behavior measures, and self-reports. As measurement devices that will be analyzed statisticaly, tne require the use of specific experimental designs. Regardless of the part instrument or technique used, it should be pretested with a smaller grou before actual use. Since comparisons are desired, it is usually necessaly

collect preliminary data on current performance or behaviors.

Tests

Tests, especially written tests, are probably the most common levices for measuring learning. Tests sample what one knows, and schools depei nd

heavily

on them. Multiple-choice, true-false, short answer, completion, matc

r, y

cSsay questions

are used

o measure

learning in

the cognitive domain.

These tests are appro-

priate when several

people are expected

to learn the same

content or material.

Sometimes, botha

pretest and a posttest

are used to measure Visual aids assist learning.

leaming. This methood

assists in controlling

variables, but be careful not to attribute all the changes noted on the posttest

to the learning experiences since other factors may have been involved.

Source: CDC/Amanda Mills.

CASE ANALYSIS 3

What evaluation or outcome measures do you suggest for the prenatal class?

Although tests are appropriate with school-aged children, adults may

respond less favorably. The practitioner should avoid evoking childhood

memories associated with the authoritarian teacher, the dependent child,

or the assigned degree of success and failure based on right or Wrong an-

swers. In one-on-one situations, the practitioner could ask the individual

to state verbally what he or she learned as though telling it to a spouse or

friend. Alternatively, a self-assessment instrument may be used.

Questionnaires Questionnaires may be preplanned and are often used to assess attitudes and

values that do not involve correct answers. Questions may be open-ended, multiple-choice, ranking, checklist, or alternate response, such as yes/no Or agree/disagree. In evaluating behavioral change on the job, trainees and

supervisors can both complete a questionnaire.

InterviewNs terviews conducted on a one-to-one basis are another form of evaluation.

ney are the oral equivalent to written questionnaires used to measure

COgnitive and affective objectives. Before the interview, the instructor

uld preplan and draw up a list of questions that will indicate whether

325

326 3/EDUCATION SKILLS

nsist of ask learning has taken place. After instruction, evaluation mau

niew is corred any that the evaluator can put the person at ease and immediately eVewi

is appropri

ing the learner to repeat important facts. An advantage of an in at

ation.

for people with low literacy or those less educated. Focus group inte

errors. Another advantage is that the interviewer can probe for additional information. Although this method is time-consuming, it is

views, tive, formative evaluat mentioned earlier, are an example of a qualitativ

Observation

In many cases, visual observation is an appropriate method of eunl.

and ing learning. The behaviors to be observed should be defined. and

an observation checklist may be helpful. When employees are under direc supervision, systematic ongoing observation over a period of time is a has. her

for evaluating learning. The supervisor can observe and report wherha basis

the employee is operating equipment correctly or following established procedures properly. If the employee has been taught sanitary procedure for example, the professional can see whether or not they are incorporated into the employee's work. Evaluate the pertormance, using what was taught as a standard. If discrepancies are found, further learning may be indicated Performance Tests

When direct observation is not possible or would be too time-consuming and costly, a simulated situation or performance test can be observed. Per-formance tests are appropriate in the cognitive and psychomotor domains You can ask a wait staff member to set a table, a cook to demonstrate the meat slicer, or a client to indicate what to select from a restaurant menu The client could be given a list of foods and asked to differentiate those appropriate for his or her diet. With permission from the learner, audiotape or videotape may be used to record the teaching session. The instructo and learner may discuss the results together and plan further learning correct any deficiencies. The observer needs to delineate which behavios are being observed and what is to be acceptable behavior. Rating Scales and Checklists

Rating scales and checklists have been used to evaluate learne rner performance evel

and teacher effectiveness. Categories or attributes such as knowledg or dependability are listed and should be defined in detail to avo biguity. Emphasis should be placed on attributes that can be co d

biguity. Emphae. sted and shoula utes such as knowledge level

objectively rather than judged subjectively. A 5- or 7-poin from allowing a midpoint, and the ratings should be defined, for exaintable used "excellent" The list should to "poor" include ae from "extremely acceptable" to "very unacceptao from int scale is

ed, for example The list should include as a possible response "No opportuniy eptable

to o

12/IMPLEMENTING AND EVALUATING LEARNING Rating scales are subject tO several errors. Two evaluators may judge the

327

sA of evaluators are essential. The ratings may suffer from personal

same indivi vidual fferently.To avoid error, definition of the terms and

training of

raIn addition, some raters have the tendency to be too lenient. Er-

biases. In additi

may result mav result if the rater is a perfectionist. Some evaluators tend to rate

ror

st people as average, believing that few people rank at the highest most people:

so Dositively or negativ impressed with one aspect of a person that he

evels. Another possible error is the "halo" error, in which an evaluator is he iudges all other qualities according to this one impressive aspect.

Performance Measures

ln employee training programs, individual and group performance measures may be assessed. These may include work quality and quantity, number af errors, days of absenteeism, number of grievances, and other types of problems that affect work performance.7

Self-Reports

Self-reports, self-evaluation, and self-monitoring are another approach to evaluation. In the affective domain, written questions or statements are presented and the individual supplies responses. "What changes, if any, have you made in your food choices?" "What are you doing differently?" Self-reports, such as a 3-day food record, have been used to measure be havioral change. Responses may be distorted or biased if the individual can ascertain the acceptable answer.

All methods of evaluation have advantages and limitations, which need to be considered. Although evaluation may not provide proof that an in-

tervention and education worked, it does produce a great deal of evidence

CASE ANALYSIS 4

wnat type of evaluation(s) would you use for the prenatal nutrition class?

Reliability and Validity ne concepts of reliability and validity are essential to the measurement ot

ne effectiveness of nutrition education outcomes and employee learning

y indicates whether we are measuring what we intend to measure

ere are different types of validity, such as content-related, construct-related,

riterion-related (concurrent and predictive) validity, all of which

p to "defend" the validity of the instrument. Content-related validiy,