NEED IN 20 hours: 6 page Article Critique Assignment – Human Resources

10.1177/1523422304272080 Advances in Developing Human Resources February 2005 Holton / HOLTON’S EVALUATION MODEL

Holton’s Evaluation Model: New Evidence and Construct Elaborations

Elwood F. Holton, III

The problem and the solution. Holton proposed the HRD Evaluation and Research Model as a comprehensive framework for diagnosing and understanding the causal influences of HRD intervention outcomes. Unfortunately, a full test of Holton’s model has not been possible because tools to measure the con- structs in the model did not exist. This article reviews recent studies relevant to the constructs in Holton’s model and updates it by delineating specific constructs that should be mea- sured in each of the conceptual categories proposed.

Keywords: HRD evaluation; evaluation models; HRD outcomes; HRD theory; Learning Transfer System Inventory; LTSI

Holton (1996) sharply criticized Kirkpatrick’s (1959) four-level evaluation model and proposed the HRD Evaluation and Research Model as a more comprehensive framework for diagnosing and understanding the causal influences of HRD intervention outcomes. The original model (see Figure 1) was theoretically derived and more conceptually comprehensive than Kirkpatrick’s simple four-level taxonomy. Three outcome levels are hypothesized in the model: learning, individual performance, and organiza- tional performance. Following Noe and Schmitt (1986), the macro- structure of that model hypothesizes that HRD outcomes are a function of factors in three construct domains: ability, motivation, and environmental influences. The model further specified conceptual constructs in each domain that are hypothesized to influence each of the three outcome levels. Secondary influences are also included, particularly those that affect moti- vation to learn.

The model addressed one of the biggest risks of the four-level model, specifically, that any failure to achieve outcomes from an intervention would be attributed to the intervention itself when it could well be due to moderating variables. Perhaps the best example of this is the situation that

Advances in Developing Human Resources Vol. 7, No. 1 Februar y 2005 37-54 DOI: 10.1177/1523422304272080 Copyright 2005 Sage Publications

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

arises when learning outcomes (level 2) from a training intervention are positive but no on-the-job behavior change occurs (level 3) because the transfer climate is poor. Unless the transfer climate is evaluated, the deci- sion derived from the four-level model would be that the training interven- tion had failed and needed to be changed. The correct evaluation decision derived from the Holton model would be that the training intervention did not need to be changed but the organization did not have the transfer climate to support it, so an organization development intervention would be needed.

Unfortunately, a full test of my model has not been possible because many of the tools to measure the constructs in the model did not exist. How- ever, since I first proposed my model in 1996, research evidence has contin- ued to accumulate and generally supports it, although some research sug- gests that modifications are needed. This article reviews selected recent studies relevant to the constructs in my model and updates the model by delineating specific constructs that should be measured in each of the con- ceptual categories proposed. In certain instances, the model is modified based on new research or theory. The result is an updated version of the model that is more appropriate for empirical testing.

38 Advances in Developing Human Resources Febr uar y 2005

Learning Individual

Performance Organizational Performance

Outcomes

Environment Elements

Ability/Enabling Elements

Motivation Elements

Secondary Influences

Motivation to Learn

Perceptions of Training

Transfer Climate

Motivation to Transfer

Learning Design

Ability

Transfer Design

Linkage to Organization Goals

Individual Characteristics

Intervention Readiness

Job Attitudes

Intervention Fulfillment

External Events

Expected Utility/Return on Investment

FIGURE 1: HRD Evaluation Research and Measurement Model From Holton (1996)

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

Influences on Learning This section reviews recent research on factors influencing individual

learning outcomes. These are the first domain of outcomes in the model moving sequentially from left to right.

Secondary Influences—Individual Learner Characteristics

Some of the most intriguing research in recent years has been focused on learner dispositional influences on learning, primarily through motivation to learn. Dispositional influences are a general category of individual traits that are relatively stable and enduring and that predispose a person toward certain tendencies or patterns. The best-known example of dispositional characteristics is personality, although there are many others.

Much of the work on personality traits has used the Big Five framework. The Five Factor Model dominates the current view of personality and pro- vides a unifying structure to its study. This model has, in fact, garnered so much support that the FFM “has now become an almost universal template for understanding the structure of personality” (Ferguson & Patterson, 1998, p. 789). As the name implies, this model suggests that there are five broad categories of traits at the top of the personality trait hierarchy. The rel- atively orthogonal five-factor taxonomy resulted from decades of research on the structure of human personality (Costa & McCrae, 1992) and has gained the support of numerous researchers (e.g., Costa & McCrae, 1995; Digman, 1990; Goldberg, 1990). As highlighted by Costa and McCrae (1992), the five dimensions of the Five Factor Model of personality are neuroticism (or emotional stability), extraversion, openness to experience, agreeableness, and conscientiousness.

Colquitt, LePine, and Noe’s (2000) seminal meta-analysis on training motivation research examined research on a variety of influences on learn- ing and transfer outcomes of training. There were a number of important findings in their work relevant to this model. First, two dimensions of the Big Five personality measures were found to influence motivation to learn through pretraining self-efficacy. They were conscientiousness and anxiety (a component of neuroticism). Conscientiousness refers to the extent to which someone is dependable, persevering, hardworking, disciplined, deliberate, and achievement oriented (Herold, Davis, Fedor, & Parsons, 2002, p. 854). It has consistently been linked to motivation to learn and training outcomes (Barrick & Mount, 1991; Colquitt & Simmering, 1998; Naquin & Holton, 2002).

Emotional stability, also known by its opposite trait of neuroticism, reflects the absence of feelings of anxiety, insecurity, and nervousness. It reflects to some degree positive psychological adjustment. People scoring

Holton / HOLTON’S EVALUATION MODEL 39

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

high on this dimension are generally less anxious and more upbeat in their outlook. Emotional stability has been found to be related to training out- comes, including transfer outcomes (Herold et al., 2002).

Several studies also found that openness to experience, another of the Big Five personality dimensions, is related to both learning and transfer out- comes (Barrick & Mount, 1991; Herold et al., 2002; LePine, Colquitt, & Erez, 2000), although Naquin and Holton (2002) did not find a significant effect for this construct. Openness to experience is associated with intellect, curiosity about one’s environment, and a willingness to explore new things. Because training requires persons to embrace new things, this trait should be helpful to trainees (Herold et al., 2002, p. 855).

Naquin and Holton (2002) found support for agreeableness as a predic- tor, whereas most studies have not. Agreeableness is an interpersonal trait and reflects the degree to which a person is generally a cooperative, compas- sionate, and trusting person in interpersonal situations. Only limited sup- port has been found, perhaps because its effect is limited to training related to interpersonal skills training.

In sum, three of the Big Five factors—conscientiousness, neuroticism, and openness to experience—have received strong support in the literature, whereas agreeableness has received weak support. Extraversion appears to have no relationship with training outcomes. Thus, the three personality traits with strong support are recommended for inclusions in the model as personality trait measures.

Moving beyond the Big Five, the latest trait to emerge in literature has been goal orientation. Goal orientation posits that individuals are of two types—learning oriented versus performance oriented.

A learning orientation is characterized by a desire to increase one’s competence by developing new skills and mastering new situations. In contrast, performance orientation reflects a desire to demonstrate one’s competence to others and to be positively evaluated by others. (p. 498)

Research has shown that individuals with a learning orientation tend to pursue difficult learning challenges and persist in the face of failure of learning difficul- ties. Persons with a performance orientation tend to see the same situations as threatening and to withdraw from them. Thus, a learning orientation is associ- ated with more positive learning outcomes, whereas a performance orientation is associated with negative or neutral learning outcomes (Bell & Kozlowski, 2002; Chen, Gully, Whiteman, & Kilcullen, 2000; Colquitt & Simmering, 1998; Ford, Smith, Weisbein, & Gully, 1998). Most recently, goal orientation has been shown to have an interactive effect with cognitive ability (Bell & Kozlowski, 2002). Because of the compelling evidence in the literature, goal orientation is recommended as another individual difference variable for the model.

Another dispositional variable that has received recent support in the lit- erature is locus of control (Colquitt et al., 2000). Persons with an internal locus of control tend to have more positive attitudes and motivation toward

40 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

training because they are more likely to believe that they can change their abilities and motivation through their own actions. Persons with an external locus of control are more likely to believe that changes in performance are only possible through changes in factors external to themselves. Logically, then, persons with an internal locus of control are more likely to believe that they can improve their skills and performance by exerting effort in training. Locus of control has been shown to be a significant predictor of motivation to learn in Colquitt et al.’s (2000) meta-analysis and thus is recommended for inclusion in the model.

In summary, recent research points to five variables as measures of the individual characteristics category in the Holton Evaluation Model. They are (a) conscientiousness, (b) neuroticism (emotional stability), (c) open- ness to experience, (d) goal orientation, and (e) locus of control. The origi- nal theory posited that the effect of these variables would be fully mediated by motivation to learn. However, Colquitt et al.’s (2000) meta-analysis showed convincingly that these learner characteristics had both an indirect effect through motivation to learn as well as a direct effect on learning. Thus, it seems appropriate to add a path directly from these individual characteristics to learning.

Secondary Influences—Job Attitudes

The second category of secondary influences posited in the Holton model was job attitudes. The theory posits that job attitudes should affect both motivation to learn and motivation to transfer learning. Logically, it presumes that individuals who have more positive attitudes toward their organization would be more likely to engage in learning that will benefit the organization through improved performance. Unfortunately, as Colquitt et al. (2000) note in their meta-analysis, job attitude variables have not received sufficient research attention in the training literature. In fact, most of the job attitude variables could not even be included in their meta-analytic path analysis because of the paucity of studies.

The limited research on job attitudes has focused on several variables. First is job involvement, which has been the most frequently researched job attitude variable and has consistently been shown to be a significant predic- tor of motivation. Second is organizational commitment. Naquin and Holton (2002) provide compelling evidence of the importance of these vari- ables in their study of the effects of dispositional variables and work atti- tudes on a construct they identified as Motivation to Improve Work Through Learning (MTIWL). MTIWL is a higher order construct combining motiva- tion to learn and motivation to transfer. In their study, work commitment, which included both job involvement and two dimensions of organizational commitment, was the second strongest predictor of motivation after positive

Holton / HOLTON’S EVALUATION MODEL 41

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

affectivity. Work commitment also mediated the effects of conscientious- ness on motivation. Thus, work commitment was shown to have a strong and direct effect on both dimensions of motivation. However, Colquitt et al.’s (2000) meta-analysis did not find job involvement to be a significant predic- tor and they were unable to test organizational commitment due to an insufficient number of studies.

Given the limited evidence, it is hard to know definitively which job atti- tudes to measure. It is clear that more research is needed in this area. Given that there is some evidence supporting job involvement and organizational commitment as predictors of motivation, these two variables are included in the model primarily because they need to be more carefully studied to understand their effect on learning and transfer outcomes.

Perceptions of Training

Perceptions of training, also known as reactions, have been the most con- troversial domain of evaluation. Kirkpatrick (1998) sanctioned reactions as legitimate outcomes, calling them level 1 outcomes. Holton (1996) demoted them to a moderator variable in the HRD Evaluation and Research Model, based largely on meta-analysis research that has shown low or no relation- ship between reaction outcomes and higher level outcomes of learning and performance (Alliger & Janak, 1989; Alliger, Tannenbaum, Bennett, Trave, & Shotland, 1997). Swanson and Holton (1999) included them as an out- come but advocated not trying to achieve high levels of reaction results in order not to overemphasize reaction outcomes because doing so often underemphasizes learning and performance outcomes.

Recent research has shed new light on the multidimensionality of reac- tions and their relationship to learning and performance outcomes. Morgan and Casper (2000) conducted a factor analytic study of a large database of responses to a multidimensional reaction instrument. They demonstrated that reaction measures were, in fact, multidimensional, including a factor they labeled as “Utility Reactions” along with other more typical affective reaction measures. These results were cross-validated with confirmatory factor analysis in a split-sample design. They suggested that models of training effectiveness such as the Holton model might benefit from incorpo- rating a multidimensional treatment of participant reactions.

Two studies have suggested that utility reactions may have some incre- mental validity in predicting learning or performance outcomes. Tan, Hall, and Boyce (2003) created a utility reaction scale that also incorporated behavioral intentions and tested the effect of the affective and utility reac- tion measures on learning and performance. Their results showed that one utility reaction scale was a significant predictor over and beyond a pretest in

42 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

predicting learning. However, no reaction measures were significant in predicting performance.

Ruona, Leimbach, Holton, and Bates (2002) added support to a role for utility reactions in predicting motivation to transfer learning. In their analy- sis, they first entered transfer system perceptions into a hierarchical regres- sion model and then forced the utility reaction measure to enter last. Utility reactions were a significant predictor and increased the model R2 from .41 to .46, suggesting that utility reactions added predictive power over and beyond transfer system perceptions.

Together, this research suggests that perceptual measures of training completed by trainees may have a role in predicting learning and perfor- mance outcomes, albeit a small one. It is clear that the research is inconclu- sive, but it suggests several tentative conclusions. First, there is no support for including affective reactions in the model. No research is emerging that shows that affective reactions have any role in predicting important out- comes from training. Second, research suggests that utility reactions offer small but significant predictive power to the model. Finally, an intriguing new dimension, behavioral intentions, has emerged with a strong theoretical base that warrants inclusion in the model and further testing. Based on these findings, two constructs are recommended for the Holton model for the per- ceptions of training domain: utility reactions and behavioral intentions.

Influences on Individual Performance The next domain of outcomes to consider is individual performance. In

this section, research on factors influencing individual performance is examined and enhancements to the Holton model are proposed.

Learning Transfer Research

Immediately after publishing the model shown in Figure 1, I began work to move toward a comprehensive test of the model. Unfortunately, there were glaring gaps in both theory and measurement tools that needed to be addressed before any full test was possible. Perhaps the most glaring gap was in the area of transfer climate. At that time, there was no validated framework to define transfer climate nor any generally accepted and vali- dated instrument to measure what was identified in my model as transfer design, transfer climate, and motivation to transfer. My associates and I quickly began to work on a conceptual framework and measurement instru- ment. Few people realized that it was not learning transfer per se that drove our interest but rather our desire to work toward a comprehensive test of the Holton model.

Holton / HOLTON’S EVALUATION MODEL 43

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

This program of research now focuses on what we call the learning trans- fer system, instead of transfer climate (Holton, Bates, & Ruona, 2000). The learning transfer system is defined as all factors in the person, training, and organization that influence transfer of learning to job performance. Transfer climate, the more common term, is actually but one subset of factors that influences transfer, although the term is sometimes incorrectly used to refer to the full set of influences. Other influences on transfer include training design, personal characteristics, opportunity to use training, and motiva- tional influences. Thus, the transfer system is a broader construct than trans- fer climate but includes all factors traditionally referred to as transfer cli- mate. Transfer can only be completely understood and influenced by examining the entire system of influences.

The instrument that resulted from this program of research is called the Learning Transfer System Inventory (LTSI). It has been well tested with strong evidence of construct validity (Bookter, 1999; Holton et al., 2000), initial evidence of criterion validity (Bates, Holton, Seyler, & Carvalho, 2000; Ruona et al., 2002; Seyler, Holton, Bates, Burnett, & Carvalho, 1998), and good cross-cultural validity (Chen, 2003; Khasawneh, 2004; Yamnill, 2001). The LTSI program of research has resulted in a framework that defines 16 constructs that make up the learning transfer system. Table 1 shows the constructs measured by the LTSI, their definition, and a sample item along with scale reliabilities. These 16 factors are believed to measure the full system of factors that influence learning transfer, although further criterion validation studies are still under way.

The LTSI defines and measures all the variables that should be measured in the transfer design, transfer climate, and motivation to transfer boxes of the Holton model. Figure 2 shows how the constructs of the LTSI map to these three conceptual categories.

Motivation to Improve Work Through Learning One of the most important developments in the Holton model has been

the reconceptualization of the motivation constructs for both learning and transfer. In the original model, motivation was conceptualized in the tradi- tional way, namely, as the two separate construct domains of Motivation to Learn (MTL) and Motivation to Transfer (MTT). As shown in Figure 1, a relationship between the two was posited in that MTT was hypothesized to influence MTL, but they were still conceptualized as separate constructs.

Naquin and Holton (2003) completely reconceptualized motivation by creating the construct MTIWL. The work improvement process in HRD, they argue, is not just a function of learning or training. Rather, it requires learners to acquire knowledge and transfer that knowledge into improved work outcomes or productivity. They further argue that using MTL is too

44 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

45

TA B LE

1 :

L e a rn

in g T

ra n

sf e r

S ys

te m

I n ve

n to

ry (

LT S

I) S

ca le

D e fi

n it

io n

s a n

d S

a m

p le

I te

m s

# o

f F

a ct

o r

D e fi

n it

io n

S a m

p le

I te

m It

e m

s α

T ra

in in

g- sp

ec ifi

c sc

al es

Le ar

n er

re ad

in es

s ex

te n t

to w

h ic

h in

d iv

id u al

s ar

e p re

p ar

ed to

en te

r an

d p ar

ti ci

p at

e in

tr ai

n in

g B

ef o

re th

e tr

ai n in

g, I

h ad

a go

o d

u n d er

- st

an d in

g o

f h o

w it

w o

u ld

fit m

y jo

b -r

el at

ed d ev

el o

p m

en t.

4 .7

3

M o

ti va

ti o

n to

tr an

sf er

d ir

ec ti

o n , in

te n si

ty , an

d p er

si st

en ce

o f

ef fo

rt to

w ar

d u si

n g

in a

w o

rk se

tt in

g sk

ill s

an d

kn o

w le

d ge

le ar

n ed

I ge

t ex

ci te

d w

h en

I th

in k

ab o

u t

tr yi

n g

to u se

m y

n ew

le ar

n in

g o

n m

y jo

b .

4 .8

3

P o

si ti

ve p er

so n al

o u tc

o m

es d eg

re e

to w

h ic

h ap

p ly

in g

tr ai

n in

g o

n th

e jo

b le

ad s

to o

u tc

o m

es th

at ar

e p o

si ti

ve fo

r th

e in

d iv

id u al

E m

p lo

ye es

in th

is o

rg an

iz at

io n

re ce

iv e

va ri

o u s

“p er

ks ”

w h en

th ey

u se

n ew

ly le

ar n ed

sk ill

s o

n th

e jo

b .

3 .6

9

N eg

at iv

e p er

so n al

o u tc

o m

es ex

te n t

to w

h ic

h in

d iv

id u al

s b el

ie ve

th at

n o

t ap

p ly

in g

sk ill

s an

d kn

o w

le d ge

le ar

n ed

in tr

ai n in

g w

ill le

ad to

n eg

at iv

e p er

so n al

o u tc

o m

es

If I

d o

n o

t u se

m y

tr ai

n in

g, I

w ill

b e

ca u -

ti o

n ed

ab o

u t

it .

4 .7

6

P er

so n al

ca p ac

it y

fo r

tr an

sf er

ex te

n t

to w

h ic

h in

d iv

id u al

s h av

e th

e ti

m e,

en er

gy , an

d m

en ta

l sp

ac e

in th

ei r

w o

rk liv

es to

m ak

e ch

an ge

s re

q u ir

ed to

tr an

sf er

le ar

n in

g to

th e

jo b

M y

w o

rk lo

ad al

lo w

s m

e ti

m e

to tr

y th

e n ew

th in

gs I

h av

e le

ar n ed

. 4

.6 8

P ee

r su

p p o

rt ex

te n t

to w

h ic

h p ee

rs re

in fo

rc e

an d

su p p o

rt u se

o f

le ar

n in

g o

n th

e jo

b M

y co

lle ag

u es

en co

u ra

ge m

e to

u se

th e

sk ill

s I

h av

e le

ar n ed

in tr

ai n in

g. 4

.8 3

(c o n ti

n u ed

)

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

46

Su p er

vi so

r su

p p o

rt ex

te n t

to w

h ic

h su

p er

vi so

rs /m

an ag

er s

su p -

p o

rt an

d re

in fo

rc e

u se

o f

tr ai

n in

g o

n th

e jo

b

M y

su p er

vi so

r se

ts go

al s

fo r

m e

th at

en co

u ra

ge m

e to

ap p ly

m y

tr ai

n in

g o

n th

e jo

b .

6 .9

1

Su p er

vi so

r sa

n ct

io n s

ex te

n t

to w

h ic

h in

d iv

id u al

s p er

ce iv

e n eg

at iv

e re

sp o

n se

s fr

o m

su p er

vi so

rs /m

an ag

er s

w h en

ap p ly

in g

sk ill

s le

ar n ed

in tr

ai n in

g

M y

su p er

vi so

r o

p p o

se s

th e

u se

o f

th e

te ch

n iq

u es

I le

ar n ed

in tr

ai n in

g. 3

.6 3

P er

ce iv

ed co

n te

n t

va lid

it y

ex te

n t

to w

h ic

h tr

ai n ee

s ju

d ge

tr ai

n in

g co

n -

te n t

to ac

cu ra

te ly

re fl ec

t jo

b re

q u ir

em en

ts W

h at

is ta

u gh

t in

tr ai

n in

g cl

o se

ly m

at ch

es m

y jo

b re

q u ir

em en

ts .

5 .8

4

T ra

n sf

er d es

ig n

d eg

re e

to w

h ic

h (a

) tr

ai n in

g h as

b ee

n d es

ig n ed

an d

d el

iv er

ed to

gi ve

tr ai

n ee

s th

e ab

ili ty

to tr

an sf

er le

ar n in

g to

th e

jo b , an

d (b

) tr

ai n in

g in

st ru

ct io

n s

m at

ch jo

b re

q u ir

em en

ts

T h e

ac ti

vi ti

es an

d ex

er ci

se s

th e

tr ai

n er

s u se

d h el

p ed

m e

kn o

w h o

w to

ap p ly

m y

le ar

n in

g o

n th

e jo

b .

4 .8

5

O p p o

rt u n it

y to

u se

ex te

n t

to w

h ic

h tr

ai n ee

s ar

e p ro

vi d ed

w it

h o

r o

b ta

in re

so u rc

es an

d ta

sk s

o n

th e

jo b

en ab

lin g

th em

to u se

tr ai

n in

g o

n th

e jo

b

T h e

re so

u rc

es I

n ee

d to

u se

w h at

I le

ar n ed

w ill

b e

av ai

la b le

to m

e af

te r

tr ai

n in

g.

4 .7

0

G en

er al

sc al

es T

ra n sf

er ef

fo rt

- p er

fo rm

an ce

ex p ec

ta ti

o n s

ex p ec

ta ti

o n

th at

ef fo

rt d ev

o te

d to

tr an

sf er

- ri

n g

le ar

n in

g w

ill le

ad to

ch an

ge s

in jo

b p er

fo rm

an ce

M y

jo b

p er

fo rm

an ce

im p ro

ve s

w h en

I u se

n ew

th in

gs th

at I

h av

e le

ar n ed

. 4

.8 1

TA B LE

1 (c

o n

ti n

u e d

)

# o

f F

a ct

o r

D e fi

n it

io n

S a m

p le

I te

m It

e m

s α

(c o n ti

n u ed

)

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

47

P er

fo rm

an ce

-o u tc

o m

es ex

p ec

ta ti

o n s

ex p ec

ta ti

o n

th at

ch an

ge s

in jo

b p er

fo rm

an ce

w ill

le ad

to va

lu ed

o u tc

o m

es W

h en

I d o

th in

gs to

im p ro

ve m

y p er

fo r -

m an

ce , go

o d

th in

gs h ap

p en

to m

e. 5

.8 3

R es

is ta

n ce

/o p en

n es

s to

ch an

ge ex

te n t

to w

h ic

h p re

va ili

n g

gr o

u p

n o

rm s

ar e

p er

ce iv

ed b y

in d iv

id u al

s to

re si

st o

r d is

- co

u ra

ge th

e u se

o f

sk ill

s an

d kn

o w

le d ge

ac q u ir

ed in

tr ai

n in

g

P eo

p le

in m

y gr

o u p

ar e

o p en

to ch

an gi

n g

th e

w ay

th ey

d o

th in

gs .

6 .8

5

P er

fo rm

an ce

se lf-

ef fic

ac y

an in

d iv

id u al

’s ge

n er

al b el

ie f

th at

th ey

ar e

ab le

to ch

an ge

th ei

r p er

fo rm

an ce

w h en

th ey

w an

t to

I am

co n fid

en t

in m

y ab

ili ty

to u se

n ew

ly le

ar n ed

sk ill

s o

n th

e jo

b .

4 .7

6

P er

fo rm

an ce

co ac

h in

g fo

rm al

an d

in fo

rm al

in d ic

at o

rs fr

o m

an o

rg a-

n iz

at io

n ab

o u t

an in

d iv

id u al

’s jo

b p er

fo rm

an ce

A ft

er tr

ai n in

g, I

ge t

fe ed

b ac

k fr

o m

p eo

p le

ab o

u t

h o

w w

el l I

am ap

p ly

in g

w h at

I le

ar n ed

.

4 .7

0

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

limiting for organizational learning environments. What employees are really engaged in is the process of improving work through the learning pro- cess that necessarily entails transferring learning into job application.

MTIWL was defined as

Motivation to Improve Work Through Learning (MTIWL) = ƒ(Motivation to Train, Motivation to Transfer)

Naquin and Holton (2003) demonstrated the initial construct validity of this new construct using confirmatory factor analysis.

Although on the surface it might appear that this construct only combines two existing constructs in an additive fashion, the effects are likely to be more significant on outcomes in the Holton model. That is, persons entering a learning situation with high levels of MTIWL are likely to have greater motivation to engage in work-relevant learning experiences offered with strong transfer designs that stress practice and job application than persons with high levels of simple MTL. Thus, the nature of the MTIWL motivation in the learning environment is expected to be substantively different from simple MTL and will demand different types of learning experiences. Fur- ther, they can be expected to exhibit higher rates of transfer to individual performance.

48 Advances in Developing Human Resources Febr uar y 2005

Outcomes

Environment

Motivation

Secondary Influences

Ability

Learning Individual

Performance

Content Validity Transfer Design Personal Capacity for Transfer Opportunity to Use

Motivation to Transfer Transfer Effort --> Performance Performance--> Outcomes

Feedback Peer Support Supervisor Support Openness to Change Personal Outcomes-Positive Personal Outcomes-Negative Supervisor Sanctions

Performance Self-Efficacy learner Readiness

Organizational Performance

FIGURE 2: Learning Transfer System Inventory (LTSI) Conceptual Map of Constructs

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

Obviously, this new construct remains untested and has unknown crite- rion validity. However, it is promising enough that I am changing the Holton model to incorporate it and spur research on its validity. The constructs pre- viously included in the MTL and MTT boxes in the model are now inte- grated into one motivation domain, MTIWL.

Also unknown is whether the relationships found in the literature as dis- cussed above between secondary influences such as dispositional factors, job attitudes, and MTL will also hold true for MTIWL. However, I suggest that it is a short theoretical leap to hypothesize that they will, although only further research will definitively validate them.

Influences on Organizational Performance Unfortunately, there has been almost no research on factors influencing

the transfer of individual performance into organizational performance results. This reflects the almost exclusive focus in training research on the individual level of analysis. It also highlights a serious shortcoming of train- ing research in that the greatest effect on organizations obviously occurs through attaining organizational performance, yet this domain receives the least attention in the literature.

One notable exception is a study by Montesino (2002) in which he exam- ined the linkage between training, the strategic direction of the organiza- tion, transfer enhancing behaviors, and usage of training on the job. Several important findings emerged. First, a low to moderate correlation was found between managers’ and trainees’ engagement in transfer enhancing behav- iors and their perceptions that training was congruent with the strategic direction of the firm. This directly supports the relationship in the HRD Evaluation and Research Model between Expected Return/Utility of Train- ing and MTT in that training perceived to be congruent with the firm’s strat- egy is expected to have higher utility and therefore to increase trainee MTT.

Second, trainees who reported very high usage of training perceived a signifi- cantly higher alignment of training with the strategic direction of the firm (Montesino, 2002, p. 102). Montesino concludes that

apparently those trainees who saw more clearly the connection of the training program with the strategic direction of the organization were able to apply on the job the skills they learned in the training program in greater proportion than were the trainees who did not see that connection clearly. (p. 103)

This clearly supports the relationship between Linkage to Organizational Goals and individual performance in the ability domain of the Holton model.

Although this is only one study, it offers clear evidence that the relation- ships and constructs in my model are promising. Equally evident is that there is a dearth of research in this area and a rich opportunity exists for

Holton / HOLTON’S EVALUATION MODEL 49

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

additional research into factors affecting the strategic effect of training on organizational performance.

Bates and Khasawneh (2004) provided some additional insight into how this linkage might operate. In their study, they examined the effects of a learning organization culture and selected dimensions of transfer climate on organizational innovation, an important organizational outcome from learn- ing organization interventions. Their results suggested that a learning orga- nization culture had both direct and indirect effects on organizational inno- vation, with transfer climate constructs partially mediating the effects of the learning organization culture. Thus, this study provides additional evidence that Expected Utility/Return on Investment and Linkage to Organizational Goals are important organizational factors as hypothesized in this model. Learning organizations typically include a strong linkage between organi- zational goals and learning, presumably resulting in higher perceived return on investment, which should lead to greater motivation to learn. In addition, they have indirect influences through transfer climate as shown in this model by the relationships hypothesized between organizational influences and transfer climate factors at the individual performance level.

Conclusion This article accomplished three goals. First, it reviewed recent research

that supports the Holton Evaluation and Research Model. Second, it pro- vided an update to the model by modifying it to reflect new theory, particu- larly in the area of motivation. Third, it presented an elaboration of the model by identifying the specific variables that should be measured within each of the conceptual construct domains specified in Holton (1996). By doing so, it moved the model one step closer to empirical testing and validation.

Figure 3 presents the revised model with complete construct definitions, where possible. The obvious challenge that remains is to validate the model. Although complex, I believe that the model can one day be validated, at least in part if not in its entirety. This article and the work that has transpired in the past 8 years have made it now feasible to consider a validation study where it was impossible to even consider 8 years ago. Most likely, the initial valida- tion study will include just the learning and individual performance out- come domains, as much work remains to be done in the organizational per- formance domain. Nonetheless, tremendous progress has been made and validation looks more feasible than ever before.

A validation effort would likely need to be conducted in steps. First, vali- dation studies could be conducted on a single level of outcomes. For exam- ple, researchers would measure all the intervening variables affecting learn- ing as an outcome and test their effect on learning. Or, the intervening

50 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

variables affecting transfer to job performance would be measured and tested for their effect on transfer.

Second, a multilevel analysis that combines the two approaches dis- cussed above could be conducted. Specifically, the intervening variables affecting learning and those affecting performance would be measured and tested for their ability to predict transfer to job performance.

Ultimately, the goal would be to test the entire model. That will require considerable work to define and operationalize constructs influencing orga- nizational results. Montesino’s (2002) study is a promising example of the kind of research that can be conducted to further define intervening vari- ables at the organizational level and how they influence both individual transfer and organizational results.

It is clear that these analyses are complex and will require the use of advanced statistical analysis techniques and partners willing to engage in extensive data collection efforts. The validation studies will require struc- tural equation modeling analysis to study the causal relationships hypothe- sized among the construct. Because the model is a multilevel one, hierarchi- cal linear modeling would likely be employed to analyze the cross-level relationships.

Holton / HOLTON’S EVALUATION MODEL 51

Outcomes

Environment

Motivation

Secondary Influences

Ability

Learning Individual Performance

Content Validity Transfer Design Personal Capacity for Transfer Opportunity to Use

Motivation to Improve Work Through Learning (MTIWL) Motivation to Learn Motivation to Transfer Transfer Effort --> Performance Performance --> Outcomes

Feedback Peer Support Supervisor Support Openness to Change Personal Outcomes -Positive Personal Outcomes -Negative Supervisor Sanctions

Performance Self -Efficacy Learner Readiness

Organizational Performance

Personality Traits Conscientiousness Neuroticism Openness to Experience Goal Orientation Locus of Control

Job Attitudes Organizational Commitment Job Involvement

Perceptions Utility Perceptions Behavioral Intentions

Linkage to Organization Goals

External Events

Expected Utility/Return on

Investment

Learning Design

Ability

FIGURE 3: Revised HRD Evaluation and Research Model

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

In conclusion, the validation of this model will clearly be ambitious and demanding research. Nonetheless, if the network of causal influences on important HRD intervention outcomes is to be understood, then this type of research is necessary and should be undertaken.

References

Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick’s levels of training criteria: Thirty years later. Personnel Psychology, 42, 331-342.

Alliger, G. M., Tannenbaum, S. I., Bennett, W., Trave, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341-358.

Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job per- formance: A meta-analysis. Personnel Psychology, 44, 1-26.

Bates, R. A., Holton, E. F., III, Seyler, D. L., & Carvalho, M. B. (2000). The role of inter- personal factors in the application of computer-based training in an industrial setting. Human Resource Development International, 3, 19-42.

Bates, R. A., & Khasawneh, S. (2004). Organizational learning culture, transfer climate, and perceived innovation in Jordan. In T. M. Egan & M. L. Morris (Eds.), Proceedings of the 2004 Academy of Human Resource Development Conference (514-521). Bowl- ing Green, OH: Academy of Human Resource Development.

Bell, B. S., & Kozlowski, S.W.J. (2002). Goal orientation and ability: Interactive effects on self-efficacy, performance and knowledge. Journal of Applied Psychology, 87, 497-505.

Bookter, A. I. (1999). Convergent and divergent validity of the learning transfer ques- tionnaire. Unpublished doctoral dissertation, Louisiana State University, Baton Rouge.

Chen, G., Gully, S. M., Whiteman, J. A., & Kilcullen, R. N. (2000). Examination of rela- tionships among trait-like individual differences, state-like individual differences, and learning performance. Journal of Applied Psychology, 85, 835-847.

Chen, H.-C. (2003). Cross-cultural construct validation of learning transfer system inventory in Taiwan. Unpublished doctoral dissertation, Louisiana State University, Baton Rouge.

Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of train- ing motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678-707.

Colquitt, J. A., & Simmering, M. J. (1998). Conscientiousness, goal orientation and moti- vation to learn during the learning process: A longitudinal study. Journal of Applied Psychology, 83, 654-665.

Costa, P. T., & McCrae, R. R. (1992). Revised NEO personality inventory and the five-factor inventory professional manual. Odessa, FL: Psychological Assessment Resources.

52 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

Costa, P. T., & McCrae, R. R. (1995). Solid ground in the wetlands of personality: A reply to Block. Psychological Bulletin, 117, 216-220.

Digman, J. M. (1990). Personality structure: Emergence of the five factor model. Annual Review of Psychology, 41, 417-440.

Ferguson, E., & Patterson, F. (1998). The five factor model of personality: Openness a distinct but related construct. Personality and Individual Differences, 24, 789-796.

Ford, J. K., Smith, E. M., Weisbein, D. A., & Gully, S. M. (1998). Relationship of goal ori- entation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83, 218-233.

Goldberg, L. R. (1990). An alternative “description of personality”: The big-five factor structure. Journal of Personality and Social Psychology, 59, 1216-1229.

Herold, D. M., Davis, W., Fedor, D. B., & Parsons, C. K. (2002). Dispositional influences on transfer of learning in multistage training programs. Personnel Psychology, 55, 851-869.

Holton, E. F., III. (1996). The flawed four-level evaluation model. Human Resource Development Quarterly, 7(1), 5-21.

Holton, E. F., III, Bates, R. A., & Ruona, W.E.A. (2000). Development of a generalized learning transfer system inventory. Human Resource Development Quarterly, 11(4), 333-360.

Khasawneh, S. A. (2004). Construct validation of an Arabic version of the learning transfer system inventory for use in Jordan. Unpublished doctoral dissertation, Loui- siana State University, Baton Rouge.

Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco: Berrett-Koehler.

Kirkpatrick, D. (1995). Techniques for evaluating training programs. Journal of the American Society for Training and Development, 13, 3-9.

LePine, J. A., Colquitt, J. A., & Erez, M. (2000). Adaptability to changing task contexts: Effects of general cognitive ability, conscientiousness, and openness to experience. Personnel Psychology, 53, 563-593.

Montesino, M. U. (2002). Strategic alignment of training, transfer-enhancing behaviors, and training usage: A post training study. Human Resource Development Quarterly, 13, 89-108.

Morgan, R. B., & Casper, W. J. (2000). Examining the factor structure of participant reac- tions to training: A multidimensional approach. Human Resource Development Quarterly, 11(3), 301-317.

Naquin, S. S., & Holton, E. F., III. (2002). The effects of personality, affectivity, and work commitment on motivation to improve work through learning. Human Resource Development Quarterly, 13, 357-376.

Naquin, S. S., & Holton, E. F., III. (2003). Motivation to improve work through learning in human resource development. Human Resource Development International, 6, 355-370.

Noe, R. A., & Schmitt, N. (1986). The influence of trainee attitudes on training effective- ness: Test of a model. Personnel Psychology, 39, 497-523.

Holton / HOLTON’S EVALUATION MODEL 53

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

Ruona, W.E.A., Leimbach, M., Holton, E. F., III, & Bates, R. A. (2002). The relationship between learner utility reactions and predicted learning transfer among trainees. International Journal of Training and Development, 6(4), 218-228.

Seyler, D. L., Holton, E. F., III, Bates, R. A., Burnett, M. F., & Carvalho, M. B. (1998). Factors affecting motivation to transfer training. International Journal of Training and Development, 2, 2-16.

Swanson, R. A., & Holton, E. F., III. (1999). Results: How to assess performance, learn- ing and perceptions in organizations. San Francisco: Berrett-Koehler.

Tan, J. A., Hall, R. J., & Boyce, C. (2003). The role of employee reactions in predicting training effectiveness. Human Resource Development Quarterly, 14, 397-411.

Yamnill, S. (2001). Factors affecting transfer of training in Thailand. Unpublished doc- toral dissertation, University of Minnesota, Twin Cities.

Elwood F. Holton, III, is the Jones S. Davis Distinguished Professor of Human Resource, Leadership and Organization Development at Louisiana State University. He has authored 19 books and more than 200 articles in HRD. He was the founding editor of Human Resource Development Review as well as the president of the Academy of Human Resource Development. He received the AHRD Outstanding Scholar Award in 2001.

Holton, E. F., III. (2005). Holton’s evaluation model: New evidence and construct elaborations. Advances in Developing Human Resources, 7(1), 37-54.

54 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from