Research Paper

Human resource development, evaluation, and sustainability: what are the relationships?

Darlene F. Russ-Eft*

College of Education, Oregon State University, Corvallis, USA

(Received 1 December 2013; accepted 14 July 2014)

This article discusses the relationships among the concepts of human resource devel- opment (HRD), evaluation, and sustainability. It defines each of these concepts and argues for a view of sustainability that focuses on the long-term existence of a programme. It then presents and elaborates a development of a theoretical model connecting HRD with programme evaluation leading to sustainable HRD programmes. Suggestions for future research are discussed, along with the implications for practice.

Keywords: human resource development; programme evaluation; programme sustainability

This article examines the relationships among human resource development (HRD), evaluation, and sustainability. It begins and ends with vignettes of actual evaluation experiences that reveal some of the difficulties that can arise in undertaking an evaluation of an HRD initiative. These vignettes suggest the importance of considering the multiple factors affecting programme implementation when undertaking such evaluations.

Imagine that you have been asked to evaluate a teacher development programme on the edge of one of the larger cities in Oregon. But, of course, the population is only about 147 thousand residents (in a state of about 3.8 million residents). Picture that this part of the community is filled with old and abandoned manufacturing plants and various storage facilities. Picture an old secondary school in which most of the children participate in the free school lunch programme. This is the school for which you are being asked to undertake an evaluation.

The programme is focused on working with teachers to integrate algebraic concepts into the entire curriculum. The programme has been in existence for about 2.5 years, and you (or your graduate assistants) have served as observers and evaluators. In that capacity, you have provided expert advice and technical support to the programme development team.

The observations show that the principal is very supportive of the project as are some but not all of the teachers. For example, the health and sciences teachers are actively engaged in showing connections between science activities and science projects and algebra. In contrast, the algebra teachers seem to focus primarily on having students complete one worksheet after another. More recently, however, a particularly gifted instructor and facilitator has been working with teachers to demonstrate ways of incor- porating needed activities into the curriculum, and teachers appear to be adopting or adapting some of those ideas. So far, so good.

*Email: [email protected]

Human Resource Development International, 2014 Vol. 17, No. 5, 545–559, http://dx.doi.org/10.1080/13678868.2014.954190

© 2014 Taylor & Francis

But, there is another part of the story, and that involves the public. As you may be aware, most US states are experiencing severe budget constraints, and those budget constraints are affecting school districts. Rumours are flying that there may be a con- solidation and that the principal and some of the teachers may be moved to other schools.

So, now what? To what extent should the evaluation reflect any of these anxieties? Indeed, what does this context mean in terms of sustainability of the development effort? Furthermore, the concerns with sustainability and evaluation also intersect with that of ethics. Indeed, Robertson (2008) completed a content analysis of 10 years (from 1996 to 2006) of research on business ethics appearing in the Strategic Management Journal, one of the top-ranked journals in the field of strategic management. His analysis indicated that the largest percentage (about 30%) of the articles focused on issues of sustainability and that the second largest percentage (about 22%) focused on performance, assessment and evaluation, and planning. Thus, the concepts of sustainability and evaluation were con- sidered critical.

The purpose of this article is to address the intersection of HRD, sustainability and evaluation. I begin by examining the definition of the terms ‘human resource develop- ment’, ‘evaluation’, and ‘sustainability’. Through these definitions, I hope to extract areas of similarity, overlap, influence and intersection. This exploration then leads to the development of a theoretical model connecting HRD with programme evaluation leading to sustainable HRD programmes. Then, based on that model, I suggest some areas for future HRD research and HRD practice. All of these, hopefully, lead to research that confirms factors that can be embedded within the development and implementation of HRD interventions and evaluations of those interventions that lead to what Scheier and Dearing (2011) term ‘sustainability outcomes’. Both HRD researchers and practitioners should be concerned about both the short-term outputs and the long-term outcomes and thus interested in factors leading to HRD programme sustainability.

Definition of terms

So, let us then consider the definitions of each of the terms – HRD, evaluation, and sustainability. Following this section on definitions is the development of a model connecting the three concepts.

Human resource development

One approach to defining terms is to examine the dictionary. The Oxford English Dictionary (www.oed.com) includes this definition: ‘Of the nature of the human race; that is a human, or consists of human beings; belonging to the species Homo sapiens or other (extinct) species of the genus Homo’. Dictionary.com (http://www.dictionary.com) and Merriam-Webster.com (http://www.merriam-webster.com) have similar definitions. Furthermore, the origins stem from 1350 to 1450 from Middle English. It appeared in The book of the Knight of La Tour-Landry compiled for the instruction of his daughters by Geoffroy de La Tour Landry.

The word ‘resource’ is defined by both Dictionary.com and Merriam-Webster.com as ‘a source of supply or support’. From the Oxford English Dictionary, we find ‘A means of supplying a deficiency or need; something that is a source of help, information, strength, etc., a natural source of wealth or revenue’. In this case the word appears in a French and English dictionary dated 1611.

546 D.F. Russ-Eft

Finally, the word development is defined by both Dictionary.com and Merriam- Webster.com as ‘The act or process of developing; growth; progress’. And from the Oxford English Dictionary, we find ‘A gradual unfolding, a bringing into fuller view’. This has an even more recent origin – from 1745 to 1755, and again it stems from French.

From these dictionary definitions, we might construct HRD as the ‘act or process of unfolding and growth of human beings so as to supply a deficiency or need or to serve as a source of help, information, and strength’. Although this may serve as an interesting introduction, we need to recognise that the definition of ‘HRD’ has occupied a central role and source of controversy in the HRD literature.

Many trace the definitions of the field to Nadler (1970) from his book titled, Developing Human Resources. However, two economists (Frederick Harbison and Charles A. Myers) coined the term in 1964 in their book, Education, Manpower, and Economic Growth: Strategies of Human Resource Development. More recently, Weinberger (1998) compiled an extensive review of definitions, and one of the early issues of Advances in Developing Human Resources, specifically issue 7 (2000) was devoted to defining the field. Some examples of these definitions include the following: (a) HRD is the process of increasing the knowledge, the skills, and the capacities of all the people in a society (Harbison and Myers 1964, 2); (b) HRD as a combination of training, career development, and organisational development … (Marsick and Watkins 1994, 355); and (c) HRD is a process of developing and/or unleashing human expertise through organisation development and personnel training and development for the purpose of improving performance (Swanson 1995, 208). Other definitions from Nadler (1970), Chalofsky and Lincoln (1983), and McLagan (1989) contribute similar ideas.

Some criticism has arisen concerning HRD as being a tool of management to extract maximum short-term financial benefits (e.g., Garavan and McGuire 2010; Sambrook 2009). In contrast, Fenwick and Bierema (2008) argued for HRD being concerned about not only organisational development but also individual development.

Another source of a definition of HRD comes from the AHRD Standards on Ethics and Integrity (Russ-Eft et al. 1999, 2), specifically ‘Human Resource Development (HRD) as a profession is an interdisciplinary field. It is focused on systematic training and development, career development, and organisation development to improve pro- cesses and enhance the learning and performance of individuals, organisations, commu- nities, and society’. More recently, Kuchinke (2010) argued that (1) human flourishing can and should be the guiding value of HRD and (2) HRD can and should be viewed as a special case of the broader concept of human development.

In summary then, HRD is an interdisciplinary field that is focused on growth or flourishing of human beings so that they can serve as a source of help, information, and strength to organisations and to society.

Evaluation

The term ‘evaluation’ means many different things. We might picture certain objects – a magnifying glass, a microscope or a light bulb – all of which help us to see an object better. Or, we might picture examinations and report cards, suggesting the testing and assessment aspects of evaluation. Other images might be a ruler or thermometer, indicat- ing that measurement is involved. And still others think of audits and statistics. Finally, we might think of performance reviews, performance appraisals, and performance

Human Resource Development International 547

management. Common among all of these images is the fact that there is some judgement and possibly some value placed on that judgement.

Evaluation is certainly a value-laden field. Indeed the word ‘value’ is embedded within the word ‘evaluation’. So, what do our dictionaries suggest? According to Merriam-Webster (n.d.), evaluation ‘means to determine or fix the value’ or ‘to determine the significance, worth, or condition’. Dictionary.com suggests that evaluation means ‘an act of appraising’. The Oxford English Dictionary (www.oed.com) combines these defini- tions thusly: ‘The action of appraising or valuing (goods, etc.); a calculation or statement of value’.

Scriven (1991, 39) used similar words to define evaluation by stating that ‘Evaluation refers to the process of determining the merit, worth, or value of something, or the product of that process’. He proceeded to define each of these main concepts. Merit involves intrinsic value. Worth is defined as the value to an institution or a group. Or, as Mathison (1994) stated ‘…when one judges the worth of something, one is judging its value within a particular context’ (469). Alternatively, Patton (2008) suggested that the systematic collection of information helps programmes and decision makers ‘make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding’ (38). Thus, when finally coming to the value of the programme, a judgement must be made that extends to all claims of programme merit and worth or to all claims of intrinsic or extrinsic value.

Thus, the merits of an HRD programme within an organisation might be found in the benefits that it provides to employees. These benefits, such as increased knowledge and skill, may lead to more tangible outcomes, such as increased wages or financial prosperity and potentially increased social equity. Whether such an HRD programme yields ecolo- gical benefits may depend upon the specific focus of the programme. In a similar fashion, the merit of a community programme focused on sustainable development, possibly the development of an improved public sewer system, might be found in the benefits that it provides to the community and the larger society. These benefits may lead to ecological integrity, but they may also yield social equity and economic prosperity. One way to determine whether any of the merits and worth of an intervention – be it an HRD intervention within the workplace or a community public works project – would be to undertake an evaluation in order to determine the intrinsic and extrinsic value.

Given that our focus here is on evaluation and that evaluation is taking place within a complex system – that of the workplace or the community – a systems model for that evaluation seems appropriate. The systems model, shown in Figure 1, was introduced by Preskill and Russ-Eft (2001, 2003) and has been explicated in subsequent work (Russ- Eft and Preskill 2001, 2005, 2009; Preskill and Russ-Eft 2005). The model suggests that the multiple factors throughout the system can influence an evaluation. These factors can be internal or external to the evaluation process and the organisation or community. The model begins with processes internal to the evaluation itself. These include focusing the evaluation, determining the evaluation’s design and data collection methods, collect- ing data, analysing the data and communicating and reporting evaluation processes and findings (Russ-Eft and Preskill 2001, 2009). Planning, managing, and budgeting take place throughout the evaluation and are central to all of the processes.

Three additional factors are related to the evaluation but involve outside influences. These include the political context in which the evaluation is being conducted, stake- holders’ intended use of the findings, and the evaluator’s characteristics. The political context involves those in positions of power, such as the executives within an organisation sponsoring the HRD intervention or the national government funding the community

548 D.F. Russ-Eft

sewer system. The intended uses of the evaluation tend to come from the previously mentioned groups – the executives or the national government. The last factor that of the evaluator characteristics includes such aspects as competence, expertise, and previous experience

The next layer in the systems model recognises the influence of the organisation or the community in which the evaluation is taking place. For an evaluation (or in fact any intervention) to succeed, it must be aligned with the organisation’s mission, vision, and strategic plan. In addition, the organisation’s infrastructure influences the ways in which the evaluation can be implemented and the findings are used (Preskill and Torres 1999; Preskill and Russ-Eft 2003). As with most organisational or community interventions, supportive leadership lends credibility to the evaluation effort and suggests that evaluation findings will be used. The organisation’s systems and structures can facilitate data collection and help to ensure participation. Finally, the communication system can provide effective methods for disseminating the findings. Such communication and reporting has

RD

Focusing the

Evaluation

Analyzing Evaluation

Data

Managing the

Evaluation

Designing the Evaluation &

Data Collection Instruments,

Collecting Data

Communicating & Reporting Evaluation

Processes and Findings

Customer Expectations

Competition Workforce Diversity

Legal Requirements

Technology

Global Context

Figure 1. Systems view of evaluation. Adapted from Preskill and Russ-Eft (2001).

Human Resource Development International 549

been shown to be a critical factor leading to decision making and action (Torres, Preskill, and Piontek 2005).

Finally, the model recognises that any organisational evaluation is influenced by the external context, including such factors as competition, customer expectations, workforce diversity, legal requirements, technology and the global context. These variables can often affect the extrinsic and intrinsic needs and values of the organisational stakeholders.

So, in summary, we can define evaluation as the systematic and ongoing processes for gathering data about programmes, organisations, and whole societies to enhance knowl- edge and decision making.

Sustainability

Let’s begin with the dictionary definition of sustainability. According to the Merriam- Webster Online Dictionary and Dictionary.com, the word ‘sustain’ means ‘to give support or relief to; or to keep up or prolong; to support’. The Oxford English Dictionary (www. oed.com) actually does have a definition of sustainability: ‘Capable of being maintained at a certain rate or level’.

The more recent focus on environmental issues as a focus of sustainability has arisen in response to the failure of decision-making regarding development efforts. These failures resulted because of a lack of recognition that various factors, such as local cultures and capabilities (or competencies), were interconnected. More specifi- cally, the United Nation’s World Commission on Environment and Development prepared a report in 1987 titled Our Common Future. This Commission, also called the Brundtland Commission, being named for the chair and former Norwegian prime minister, Gro Harlem Brundtland, defined sustainable development as ‘development that meets the needs of the present without compromising the ability of future generations to meet their own needs’ (World Commission on Environment and Development 1987, 43). It was the first to suggest the need for balancing social, economic, and ecological interests by stating that ‘inability to promote the common interest in sustainable development is often a product of the relative neglect of economic and social justice within and amongst nations’ (49).

Although these ideas may sound rather simple and straightforward, we need to examine the concept of sustainability more deeply. Krizek and Power (1996) stated that such sustainability ‘must balance social equity, economic prosperity, and environmental integrity’ (7). More recently, Goodland (2002) identified three, overlapping dimensions of sustainability – environmental sustainability, economic sustainability, and social sustain- ability. Rather than label this as a three-legged stool, however, I use three of the five

Economic prosperity

Ecological / environmental integrity

Figure 2. Venn diagram depicting concept of sustainability.

550 D.F. Russ-Eft

Olympic rings; or we might call it a Venn diagram (see Figure 2). We should also recognise that the concept of resilience has become popular in recent years within the development community (Tanner 2013; USAID 2012).

Whether we call the concept ‘sustainability’ or ‘resilience’, we must recognise that various criticisms of this definition have appeared. Jickling (2000), for example, sug- gested that the concept is too generic. Thus, he pointed to a situation in which both environmentalists and those supporting the development of local mining claimed sus- tainability as part of their agenda. Newton and Freyfogle (2005) mounted a similar argument. Although the term implies ‘considering the long term’ (24), it does not clarify what is being sustained and by whom. A final more recent concern was raised by Gibson (2006). The focus on the three categories of social, economics, and ecology leads to separation and concerns for balancing the three issues. Instead, there needs to be more focus on the interdependence of these factors and the methods for mutually supporting all three.

At this point, it might be appropriate to mention another view of sustainability. Shediac-Rizkallah and Bone (1998) defined sustainability as the continuation of a pro- gramme. In later work, Scheirer (2005) has examined the issue of programme sustain- ability. Specifically, her work has focused on aspects of programmes that lead to their being prolonged. Her analysis of 19 US and Canadian studies of health programmes revealed four separate definitions of sustainability: (a) continued programme activities, (b) continued client benefits or outcomes, (c) continued community support and collabora- tion, and (d) dissemination of programme work to others. Later, Scheirer, Hartling, and Hagerman (2008) undertook a follow-up of foundation-funded health programmes to determine the ways in which their work was sustained, and they confirmed the four definitions.

More recently, Backus and Russ-Eft (2011) used the work of Scheier and others to examine the sustainability of chemical technology programmes nationwide. A factor analysis revealed a six-factor solution that included community perceptions and market- ing, external engagement by business and industry, programme operations and adminis- tration, foundations for programme support, institutionally funded support, and employer student support. Further, Levine et al. (2013) undertook an examination of the sustain- ability of federally funded health-care research projects. This latter study showed that certain factors emerged as project facilitators included partnership development, suppor- tive administration, and external funding. Adding to these empirical studies was the work of Schell et al. (2013). In trying to determine factors related to the sustainability of public health programmes, they began with an extensive literature review on factors affecting programme sustainability. This was followed by a concept mapping process involving 38 experts, including researchers/scientists, funders/advisors, and state and local practitioners from the fields of tobacco control, heart disease and stroke, physical activity and nutrition, and injury prevention. Reconciling these two data sources, they identified nine domains: political support, funding stability, partnerships, organisational capacity, programme eva- luation, programme adaptation, communications, public health impacts, and strategic planning.

So, to summarise, we define sustainability as the continuation of a programmatic effort through continued programme activities, continued programme benefits or out- comes, continued community support, and continued dissemination of programme work. Furthermore, there are a number of factors that seem to lead to sustainability or to what Scheier and Dearing (2011) have called ‘sustainability outcomes’.

Human Resource Development International 551

Intersection and model development

Now that I have defined the terms ‘human resource development’, ‘programme evalua- tion’, and ‘sustainability’, I turn to examine the intersection of these concepts through the development of a model of sustainability for HRD. According to Woodall and Douglass (1999), development efforts, and we might say sustainable development efforts, have positive outcomes. In order to achieve sustainable development, then, certain factors must be present.

These factors have been identified in the research of Scheirer (2005) and Scheirer, Hartling, and Hagerman (2008) mentioned previously. They have also been identified in a planning model for sustaining innovations by Johnson et al. (2004). These factors have been confirmed in the work by Backus and Russ-Eft (2011) and Levine et al. (2013). Finally, the work of Schell et al. (2013) has provided further confirmation.

Cassidy, Leviton, and Hunter (2006) labelled three categories of factors related to programme sustainability as (a) upstream, (b) midstream, and (c) downstream. It should be noted that all three authors are affiliated with foundations and are concerned with the sustainability of programmes that their foundations fund. By upstream, the authors were referring to external factors. Midstream involved organisational factors, and downstream concerned programme-specific factors. Basically, what these authors have identified can fit into the evaluation model previously presented in Figure 1 and can connect with certain dimensions of the model of sustainability presented in Figure 2.

The following paragraphs provide further description of the model of sustainable HRD interventions, as shown in Figure 3. It begins by discussing the upstream or external factors. Then it turns of the midstream or organisational factors. The discussion concludes with the downstream or programme-specific factors. With each of these factors, the discussion also links to the dimensions provided in the model of sustainability from Figure 2.

External factors

One external factor appears to be most critical, and it involves the formal linkages and partnerships with other organisations, along with the associated funding and support from those organisations. Johnson et al.’s (2004) review of the literature identified resources as a major factor leading to sustainability. Scheirer’s (2005) review of 19 empirical studies of the sustainability US and Canadian health-related programmes identified the factor of ‘support from other organisations’ as one of the five most frequently cited. Scheirer, Hartling, and Hagerman (2008) confirmed that finding in their survey of health-related programmes; they found that 25 (or 83%) of the respondents mentioned ‘obtaining funds from external sources’ as a major barrier to sustainability. Schell et al. (2013) identified partnerships and funding as separate factors, but empirical work has suggested that the primary benefit of partnerships revolves around funding issues.

In the field of health-related research infrastructure development, Levine et al. (2013) identified that external partners and external funding were reported to be major factors affecting successful development efforts. Similarly, Backus and Russ-Eft (2011) reviewed the literature on programme sustainability and found that partnerships represented a critical factor. Two common elements of such partnerships include economic benefits to the partners and the involvement and commitment of the organisations’ leadership. A factor analysis of the data from a later national survey of chemistry-based technical programmes confirmed the importance of partnerships to sustainability.

552 D.F. Russ-Eft

Figure 3 depicts such external partnering and funding as an important factor related to HRD evaluation sustainability. Furthermore, this factor is also related to one of the dimensions of sustainability – that of economic prosperity.

Organisation factors

We now turn to the midstream or organisational factors supporting sustainable HRD development. It is important to note that some of these organisational factors may overlap with those included within the programme-specific realm. Certainly, they will directly affect the programme.

Focusing the

Evaluation

Analyzing Evaluation

Data

Administrative Support /

Programme Operation/

Designing the Evaluation &

Data Collection Instruments,

Collecting Data

Communicating & Reporting Evaluation

Processes and Findings

External Funding /

Partnerships

Competition Workforce Diversity

Legal Requirements

Technology

Global Context

Figure 3. Systems model applied to HRD, evaluation, and programme sustainability. Adapted from Preskill and Russ-Eft (2001).

Human Resource Development International 553

One critical factor involves the level of internal partnering and support. Johnson et al. (2004) included internal partnering and support as a component of the resources and funding that were needed for sustainability. Scheirer’s (2005) review of empirical studies identified the perceived benefits to staff and involvement of such staff as critical to sustainability. The later survey of health-related programmes revealed that obtaining such internal funding was reported by 15 (or 53%) respondents as a major barrier to sustainability. Levine et al. (2013) found that internal partnerships were a critical compo- nent related to the success and sustainability of research infrastructure development programmes. Clinton (2014) in examining two public health programmes over a 4-year period showed that partnering (in the form of stakeholder involvement) related to pro- gramme sustainability.

Just as with external partnering and funding, internal partnering appears to provide the financial support to help ensure programme sustainability. Again, such partnering and funding appears to be related to financial prosperity as shown in Figure 3.

A second critical factor involves leadership or a programme champion. This represents a factor that can reside within the organisation or can be based in the programme itself. Again, Scheirer’s (2005) empirical review and the later survey (Scheirer, Hartling, and Hagerman 2008) identified the importance of leadership and a programme champion. Similarly, Johnson (2004), Levine et al. (2013), and Backus and Russ-Eft (2010, 2011) recognised the criticality of this factor.

Leadership can be considered a factor related to the structure of the organisation and to the individuals leading the organisation, as shown in Figure 3. The model suggests a needed level of support provided to the programme for that programme to be sustained. Furthermore, such leadership can also be connected to the sustainability dimension of social equity. That latter connection suggests that the support provided by the organisa- tional leaders enables the HRD programmes to be sustained over time and, furthermore, that such programmes can lead to social equity.

Another organisational and programme factor revolves around the programme’s fit with the organisational mission. A close fit between the goals of the programme and the organisation’s mission, as well as perceived benefits to clients and staff, appeared as clear and somewhat related factors in Scheirer’s (2005) review. Levine et al. (2013, 6) stated that the programme ‘needed to build on or leverage larger organisational changes’. Schell’s theory development work also suggested the importance of the programme’s mission and the organisation’s mission as a critical factor leading to sustainability.

The model shown in Figure 3 suggests the importance of both the organisation’s mission as well as the programme’s mission. The model further indicates that the fit of those two missions should lead to programme sustainability. It is through the organisa- tional and programme missions that we can, perhaps, make a connection to the sustain- ability dimension of ecological or environmental integrity, suggested by Krizek and Power (1996) and Goodland (2002). What may be of concern, however, is whether or not all such mission statements must include an ecological or environmental element in order to lead to the sustainability of the programme.

Programme-specific factors

This section considers the downstream of programmatic factors supporting sustainable HRD development. Two of the factors mentioned within the organisational dimension may actually be represented as programme-specific factors – that of programme champion and that of the programme mission. The programme champion may actually be the leader

554 D.F. Russ-Eft

of the programme, and that person is recognised throughout the organisation. This is depicted within Figure 3, and such a champion would be connected to the sustainability dimension of social equity, given that this individual must function effectively within the social milieu of the organisation. The second factor, that of the programme’s mission, has already been introduced earlier because of the importance of the fit with the organisation’s mission. Thus, it should be recognised that it is the responsibility of the programme champion and the staff to adjust the focus of the programme to be in alignment with the organisation’s mission. Indeed, Scheirer (2005) found that programme modifiability was a critical component related to programme sustainability.

Another programme-specific factor identified by some but not all authors was that of administrative support. Johnson et al. (2004) discussed the importance of administrative policies and procedures. Levine et al. (2013) found that administrative support was also important to grantees. It should be recognised that Scheirer, Hartling, and Hagerman (2008) results suggested the negative effect of staff turnover. Respondents viewed such turnover as a barrier to sustainability. Thus, model in Figure 3 depicts that administrative support/programme operations represent an important factor related to HRD-evaluation sustainability. The linkage to the model of sustainability would be through social equity.

Another programme-specific factor appears to be key: that there are some sorts of evaluations of the project being undertaken. In fact, such projects or programmes include a cycle of continuous, ongoing, or what Michael Quinn Patton (2010) terms ‘develop- mental evaluation’. Johnson, Hays, Center, and Daley (2004, 139), for example, stated ‘implement, evaluate, and reassess and reassess and modify’. Such evaluation, reassess- ment, and modification may be related to Scheirer’s finding of the importance of pro- gramme modifiability. These evaluative processes presumably also have a positive effect on implementation effectiveness and quality, as shown in Figure 3.

Summary

So, we have come full circle. We began by examining the sustainability of HRD initiatives or programmes, and we have arrived at the importance of the evaluation efforts. Indeed, we must think of these evaluation efforts, not as some sort of final grade or measuring the return on investment, but rather as a learning opportunity. If such evaluation is a learning opportunity, then it must be built into the fabric of the project in order to lead to sustainable development. Figure 3, then, provides a picture of this view of sustainable development, with an embedding of evaluation.

There are, nevertheless, some issues that emerge as we consider this model. One issue involves that of the factor of partnerships and funding. Are partnerships really a separate factor from that of funding? Or is the main benefit and connection of partnerships, whether external or internal, related to the possible funding and support that they contribute? The works of Scheier (2005), Scheirer, Hartling, and Hagerman (2008), and Levine et al. (2013) suggested the linkage of partnerships with funding. In contrast, Schell et al. (2013) indicated that these are two separate factors. It will be important to determine whether funding alone or some other factors are important in the contribution of external and internal partnerships to programme sustainability.

Another issue involves the relationship of mission, whether organisational or pro- grammatic, to that of ecological or environmental integrity, identified by Krizek and Power (1996) and Goodland (2002). What are the repercussions of an organisational and/or a programmatic mission that fails to include ecological or environmental integrity? Will such an omission lead to the lack of sustainability of the HRD programme? Certainly,

Human Resource Development International 555

the organisational literature has shown that attention to ecological and environmental integrity is of major concern today (e.g., Gibson 2006; Robertson 2008). What will need further research is to determine whether ecological and environmental concerns (in their broadest sense) are critical to the sustainability of HRD development efforts?

A third issue involves the tension that seems to exists between more effort focused on programme development and more effort focused on programme evaluation. Certainly, this tension can be overcome if one understands that evaluation not only exists to measure ultimate outcomes but can also aid in identifying important and needed alterations to the programme. Nevertheless, tensions over what to fund and how much to fund will always exist.

Another issue concerns the concept of sustainability itself. Sustainability should not mean immutability. As shown in Scheirer (2005), programme sustainability does not mean that no changes take place; rather, the sustainable programmes were those that were able to adapt to changes in the global and organisational context. A question or tension, then, may exist over the issue of how much change can be made and still result in the sustainability of the programme.

Future directions for research

What has been outlined represents an initial model based on limited theoretical and empirical work undertaken in some selected fields, most specifically that of health care. The next step in this work would be to undertake some confirmation of the model within HRD. One approach to such research would follow the methods taken by Schell et al. (2013). This could involve the surveying of organisational leaders, managers, employees, and HRD professionals using concept mapping or the Delphi method. Another approach would be to conduct a critical incident study to identify the factors needed for sustain- ability. Such studies would both validate the factors identified in the model and potentially contribute some additional factors.

A second or additional approach to such research would involve the development of an instrument or assessment tool measuring each of the factors. Such a tool could then be administered to HRD practitioners in various organisations. As part of that work, some determination as to the importance or criticality of each of the factors would be useful. As part of this work, it would be important to examine any differences that might exist among the various industry sectors and among different sizes of organisations.

As mentioned in the previous section, another area for research would involve an examination of the longevity of HRD programmes. One might undertake some case studies to examine programmes that have existed within selected organisations for a number of years to confirm the factors included in the model and to determine the degree of adaptability and change that has taken place in such long-term programmes. In addition to the case study approach one might administer the previously described assessment tool to organisations with HRD programmes that have been sustained for varying amounts of time. Then, using discriminate analysis, one might be better able to identify the critical factors that yielded long-term programme sustainability.

Another area of concern should focus on the cultural and international applicability of such a model. Since much of the literature comes from a North American and European perspective, additional research will be needed to determine whether these factors remain valid in other parts of the world, including not only the developed economies but also the emerging economies.

556 D.F. Russ-Eft

Future directions for practice

While concerned about some of the research implications, I do not think that we should ignore the practical implications. Future HRD initiatives should begin to focus on sustainability, particularly those related to programme and process sustainability. To do so, however, means being attuned to some of the factors identified in the proposed model, specifically (a) external and internal partnerships and support, (b) a programme champion who remains with the programme and the organisation, (c) administrative support, (d) committed and trained staff, and (e) programmes that fit with the organisation’s mission.

From a focus on practice, though, there are some remaining questions. How can we convince programme planners and funders to focus on sustainability as an outcome rather than to the immediate outcomes experienced by clients/organisation? How can we con- vince programme planners to embed evaluation as a means to learn from the implementa- tion and to make needed modifications? These are issues that challenge all of us and hopefully stimulate us to continue to work in the field of HRD.

Conclusion

Let me end with another and perhaps more positive vignette, along with some questions to ponder. You are being asked to serve as the evaluator of a development programme for chief executive officers (CEOs) of arts and cultural organisations. This will involve CEOs primarily from the USA but also including selected CEOs from other countries as well. These CEOs will come together for week-long sessions being held in three different locations in the USA over the course of a year and a half. The purpose of the initiative is to transform the arts and cultural world. As with the previous case involving the school, you are working with the programme developers from the initial conception and design.

What issues should be of concern? How can you embed the concept of programme sustainability from the beginning through the implementation of the programme? What must be done to obtain the commitment and support of top leadership? How can the mission of the programme be best aligned to the mission of the organisation? Who should be encouraged to become the programme champion? What external partners might be most appropriate to involve in the programme? Which internal partners can be developed to support the programme? And, from my perspective as an HRD evaluator, how can evaluation be embedded in the programme to ensure continued learning, continued improvement, and sustainability of the programme? Presumably by answering these questions you can help to ensure the sustainability of the HRD programme.

References Backus, B., and D. Russ-Eft. 2010. “Sustainability of Two Year Chemistry-Based Technology

Training Programs: A Literature Review.” In Proceedings of the Workplace Learning and Sustainable Development for Individual, Organization and Society: The 9th International Conference of the Academy of HRD (Asia Chapter). Shanghai: East China Normal University.

Backus, B., and D. Russ-Eft. 2011. “Program Sustainability: Critical Factors for Chemistry-Based Technology Programs. In Proceedings of the 12th International Conference of HRD Across Europe. Cheltenham: University of Gloucestershire.

Cassidy, E. F., L. C. Leviton, and D. E. K. Hunter. 2006. “The Relationships of Program and Organizational Capacity to Program Sustainability: What Helps Programs Survive?” Evaluation and Program Planning 29: 149–152. doi:10.1016/j.evalprogplan.2005.12.002.

Chalofsky, N., and C. Lincoln. 1983. Up the HRD Ladder. Reading, MA: Addison-Wesley.

Human Resource Development International 557

Clinton, J. 2014. “The True Impact of Evaluation: Motivation for ECB.” American Journal of Evaluation 35 (1): 120–127. doi:10.1177/1098214013499602.

Fenwick, T., and L. Bierema. 2008. “Corporate Social Responsibility: Issues for Human Resource Development Professionals.” International Journal of Training and Development 12: 24–35. doi:10.1111/j.1468-2419.2007.00293.x.

Garavan, T., and D. McGuire. 2010. “Human Resource Development and Society: Human Resource Development’s Role in Embedding Corporate Social Responsibility, Sustainability, and Ethics in Organizations.” Advances in Developing Human Resources 12: 487–507. doi:10.1177/ 1523422310394757.

Gibson, R. B. 2006. “Beyond the Pillars: Sustainability Assessment as a Framework for Effective Integration of Social, Economic and Ecological Considerations in Significant Decision- Making.” Journal of Environmental Assessment Policy and Management 8: 259–280. doi:10.1142/S1464333206002517.

Goodland, R. 2002. Sustainability: Human, Social, Economic and Environmental. In Social and Economic Dimensions of Global Change: Vol. 5, Encyclopedia of Global Environmental Change, edited by P. Timmerman, the faculty of environmental studies, York University Hoboken, NJ: Wiley.

Harbison, F., and C. A. Myers, eds. 1964. “Economic Growth: Strategies of Human Resource Development.” In Education, Manpower, and Economic Growth: Strategies of Human Resource Development. New York: McGraw-Hill.

Jickling, B. 2000. “A Future for Sustainability?” Water, Air, and Soil Pollution 123: 467–476. doi:10.1023/A:1005211410123.

Johnson, K., C. Hays, H. Center, and C. Daley. 2004. “Building Capacity and Sustainable Prevention Innovations: A Sustainability Planning Model.” Evaluation and Program Planning 27: 135–149. doi:10.1016/j.evalprogplan.2004.01.002.

Krizek, K., and J. Power. 1996. A Planner’s Guide to Sustainable Development (Planning Advisory Service Report No. 467). Chicago, IL: American Planning Association.

Kuchinke, K. P. 2010. “Human Development as a Central Goal for Human Resource Development.” Human Resource Development International 13: 575–585. doi:10.1080/ 13678868.2010.520482.

Levine, R., D. Russ-Eft, D. A. Burling, J. Stephens, and J. Downey. 2013. “Evaluating Health Services Research Capacity Building Programs: Implications for Health Services and Human Resource Development.” Evaluation and Program Planning 37: 1–11. doi:10.1016/j. evalprogplan.2012.12.002.

Marsick, V., and K. Watkins. 1994. “The Learning Organization: An Integrative Vision for HRD.” Human Resource Development Quarterly 5: 353–360. doi:10.1002/hrdq.3920050406.

Mathison, S. 1994. “Evaluation.” In Encyclopedia of English Studies Language Arts, edited by A. C. Purves. Champaign, IL: NCTE and Scholastic.

McLagan, P. 1989. Models for HRD Practice. Alexandria, VA: ASTD Press. Nadler, L. 1970. Developing Human Resources. Houston, TX: Gulf. Newman, D. L., and R. D. Brown. 1996. Applied Ethics for Program Evaluation. Thousand Oaks,

CA: Sage. Newton, J. L., and E. T. Freyfogle. 2005. “Sustainability: A Dissent.” Conservation Biology 19 (1):

23–32. doi:10.1111/j.1523-1739.2005.00538.x. Patton, M. Q. 2008. Utilization-Focused Evaluation. 4th ed. Thousand Oaks, CA: Sage. Patton, M. Q. 2010. Developmental Evaluation: Applying Complexity Concepts to Enhance

Innovation and Use. New York: Guilford Press. Preskill, H., and D. Russ-Eft. 2001. “A Systems Model for Evaluating Learning Performance.” In

Academy of Human Resource Development: Defining the Cutting Edge, edited by D. H. Redmann, 57–63. Baton Rouge, LA: Academy of Human Resource Development.

Preskill, H., and D. Russ-Eft. 2003. “A Framework for Reframing HRD Evaluation Practice and Research.” In Critical Issues in HRD, edited by A. M. Gilley, L. Bierema, and J. Callahan, 199– 257. Cambridge, MA: Perseus.

Preskill, H., and D. Russ-Eft. 2005. Building Evaluation Capacity: 72 Activities for Teaching and Training. Thousand Oaks, CA: Sage.

Preskill, H., and R. T. Torres. 1999. Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.

558 D.F. Russ-Eft

Robertson, C. J. 2008. “An Analysis of 10 Years of Business Ethics Research in Strategic Management Journal: 1996–2005.” Journal of Business Ethics 80: 745–753. doi:10.1007/ s10551-007-9466-5.

Russ-Eft, D., J. Z. Burns, P. J. Dean, T. G. Hatcher, F. L. Otte, and H. S. Preskill. 1999. Academy of Human Resource Development Standards on Ethics and Integrity. St. Paul, MN: AHRD.

Russ-Eft, D., and H. Preskill. 2001. Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. New York: Basic Books.

Russ-Eft, D., and H. Preskill. 2005. “In Search of the Holy Grail: Return on Investment Evaluation in Human Resource Development.” Advances in Developing Human Resources 7: 71–85. doi:10.1177/1523422304272169.

Russ-Eft, D., and H. Preskill. 2009. Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. 2nd ed. New York: Basic Books.

Sambrook, S. 2009. “Critical HRD: A Concept Analysis.” Personnel Review 38: 61–73. doi:10.1108/00483480910920714.

Scheirer, M. A. 2005. “Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability.” American Journal of Evaluation 26: 320–347. doi:10.1177/ 1098214005278752.

Scheirer, M. A., and J. W. Dearing. 2011. “An Agenda for Research on the Sustainability of Public Health Programs.” American Journal of Public Health 101: 2059–2067. doi:10.2105/ AJPH.2011.300193.

Scheirer, M. A., G. Hartling, and D. Hagerman. 2008. “Defining Sustainability Outcomes of Health Programs: Illustrations from an On-Line Survey.” Evaluation and Program Planning 31: 335– 346. doi:10.1016/j.evalprogplan.2008.08.004.

Schell, S. F., D. A. Luke, M. W. Schooley, M. B. Elliott, F. S. H. Herbers, N. B. Mueller, and A. C. Bunger. 2013. “Public Health Program Capacity for Sustainability: A New Framework.” Implementation Science 8: 15. doi:10.1186/1748-5908-8-15.

Scriven, M. 1991. Evaluation Thesaurus. 4th ed. Thousand Oaks, CA: Sage. Shediac-Rizkallah, M. C., and L. R. Bone. 1998. “Planning for the Sustainability of Community-

Based Health Programs: Conceptual Frameworks and Future Directions for Research, Practice, and Policy.” Health Education Research 13: 87–108. doi:10.1093/her/13.1.87.

Swanson, R. A. 1995. “Human Resource Development: Performance Is the Key.” Human Resource Development Quarterly 6: 207–213. doi:10.1002/hrdq.3920060208.

Tanner, V. (2013). “Resilience: Development Fad or Concept with Staying Power?” Developing Alternatives. Accessed August 21, 2014. http://dai-global-developments.com/articles/resilience- development-fad-concept-with-staying-power.html

Torres, R. T., H. Preskill, and M. Piontek. 2005. Evaluation Strategies for Communicating and Reporting: Enhancing Learning in Organizations. 2nd ed. Thousand Oaks, CA: Sage.

USAID. 2012. The Resilience Agenda: Helping Vulnerable Communities Emerge from Cycles of Crisis onto a Pathway Toward Development. Washington, DC: USAID.

Weinberger, L. A. 1998. “Commonly Held Theories of Human Resource Development.” Human Resource Development International 1: 75–93. doi:10.1080/13678869800000009.

Woodall, J., and D. Douglass. 1999. “Ethical Issues in Contemporary Human Resource Development.” Business Ethics: A European Review 8 (4): 249–261. doi:10.1111/1467- 8608.00159.

World Commission on Environment and Development. 1987. Our Common Future. New York: Oxford University Press.

Human Resource Development International 559

Copyright of Human Resource Development International is the property of Routledge and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.