Post a brief description of the program being evaluated in the article you selected. Explain which evaluation approach was used in the evaluation and why it was appropriate for the task. Finally, explain benefits and challenges to applying this type of ev

Evaluation and Program Planning 45 (2014) 110–118

An evidence-based approach to organization evaluation and change in human service organizations evaluation and program planning

Robert L. Schalock a,b,*, Tim Lee c, Miguel Verdugo d, Kees Swart e, Claudia Claes f, Jos van Loon g, Chun-Shin Lee c

a Hastings College, USA b University of Salamanca, Spain and Gent University, Belgium c Qi Zhi Vocational Training Center, Taipei, Taiwan d Institute on Community Inclusion (INICO), University of Salamanca, Spain e Arduin Foundation, Middleburgh, The Netherlands f Faculty of Social Work and Welfare Studies, University College, Gent University, Belgium g Department of Special Education, University of Gent, Belgium

A R T I C L E I N F O

Article history:

Received 14 October 2013

Received in revised form 14 March 2014

Accepted 20 March 2014

Available online 31 March 2014

Keywords:

Collaborative evaluation

Continuous quality improvement

Evidence-based indicators

Organization change

Organization evaluation

Organization transformation

Performance-based perspectives

A B S T R A C T

The work described in this article focuses primarily on how human service organizations can use an

evidence-based, self-assessment approach to organization evaluation to facilitate continuous quality

improvement and organization change. Real-life examples are presented, strengths and challenges

discussed, and future conceptual and measurement issues identified.

� 2014 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Evaluation and Program Planning

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / e v a l p r o g p l a n

1. Introduction

Nonprofit organizations, such as those providing services and supports to persons with intellectual and closely related develop- mental disabilities (IDD) are currently facing a number of challenges both nationally and internationally. Chief among these challenges are calls for increased effectiveness and efficiency based on outcomes evaluation, increased demands for services and supports commensurate with diminishing financial resources, and the need to focus on continuous quality improvement to increase an organization’s effectiveness and efficiency.

* Corresponding author at: PO Box 285, Chewelah, WA 99109, USA.

Tel.: +1 509 935 8176; fax: +1 509 935 6101.

E-mail addresses: [email protected] (R.L. Schalock), [email protected]

(T. Lee), [email protected] (M. Verdugo), [email protected] (K. Swart),

[email protected] (C. Claes), [email protected] (J. van Loon),

[email protected] (C.-S. Lee).

http://dx.doi.org/10.1016/j.evalprogplan.2014.03.012

0149-7189/� 2014 Elsevier Ltd. All rights reserved.

These challenges need to be approached within the context of four significant trends in the fields of program evaluation and organization change (Claes, van Loon, Vandevelde, & Schalock, in press; Schalock & Verdugo, 2012, 2013; van Loon et al., 2013). The first is the increased focus on performance evaluation based on the assessment of objective, best practice indicators around which evidence can be garnered. The second is the emergence of indicators and practices related to multiple performance-based perspectives that involve the customer, and the organization’s growth, financial analyses, and internal processes. The third is a collaborative approach to evaluation that involves organization personnel who are familiar with the cultural milieu of the organization and the organization’s policies, practices, and data systems. The fourth trend is an integrative approach to continuous quality improvement, which begins with an organization-based self-assessment and continues through quality improvement activities. These four trends provide the framework for the conceptual and measurement model discussed next, and the backdrop for the work described later in the article on how nonprofit organizations can use an organization

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118 111

assessment tool based on best practice indicators to facilitate continuous quality improvement and organization change.

2. Evidence-based evaluation conceptual and measurement model

2.1. Best practice indicators

Best practice indicators are objective measures of organization processes and performance. Such indicators: (a) are based on current evidence that is obtained from credible sources that used reliable and valid methods; (b) are based on a clearly articulated, empirically supported theory or rationale; and (c) can be used for multiple purposes including the evidence in evidence-based practices, the items of an organization self-assessment tool, and the strategies employed in continuous quality improvement activities (Schalock & Verdugo, 2012, 2013; Schalock, Verdugo, & Gomez, 2011). As summarized in Table 1, and discussed more fully in reference to the organization self-assessment tool described in Section 3, these indicators can be aggregated into the four performance-based perspectives one commonly finds in the management and program evaluation literature. The indicators listed in Table 1 were identified based on a thorough literature review of the areas of performance evaluation, performance management, and program evaluation. This literature review drew heavily on the work of Bishop (2007), Bourgeois, Hart, Townsend, and Gagne (2011), Cooksy, Gill, and Kelly (2001), Cousins and Chouinard (2012), Donaldson (2007), Fuller (1997), Grol, Baker, and Moss (2004), Hunter (2006), Kong (2003), Lencioni (2012), Letts, Ryan, and Grossman (1999), Lick (2006), Pawson (2006), Pluye, Potvin, Denis, Pelletier, and Mannoni (2005), Scheirer (2005), Scheier, Hartling, and Hagerman (2008), Selden and Sowa (2011), Veerman and van Yperen (2007), and Wasserman (2010).

Table 1 The performance-based perspectives and best practice indicators used in this

article.

Customer perspective

1. Aligns services/supports to identified support needs

2. Reports the number of clients living or working in more independent,

productive, and community-integrated environments

3. Measures personal outcomes

4. Reports and analyzes aggregated personal outcomes

5. Uses technology to enhance personal outcomes

Growth perspective

6. Articulates the organization’s mission and intended results

7. Enters into partnerships

8. Develops program options

9. Utilizes and evaluates high performance teams

10. Monitors job satisfaction and develops job

enrichment programs

Financial perspective

11. Compares unit costs across different locations and service delivery

platforms

12. Reports percentage of budget allocated to

client-referenced supports

13. Monitors the relationship between social capital and agency-based fiscal

capital

14. Uses fixed and variable cost data to establish a baseline cost rate

15. Analyzes overhead rate to increase efficiency

Internal processes perspective

16. Horizontally aligns input, throughput, and

output components

17. Vertically aligns an organization’s input, throughput, and output components

to the corresponding individual-level input, throughput, and output components

18. Demonstrates relationship between units of service/support provided and

the clienteles’ assessed support needs

19. Uses data related to personal and organization outcomes for multiple

purposes

20. Uses evidence-based indicators for continuous quality improvement

2.2. Multiple performance-based perspectives

A multidimensional approach to organization evaluation and change is an emerging characteristic among IDD organizations (Schalock & Verdugo, 2012, 2013). This multidimensional ap- proach is consistent with the balanced scorecard concept that was first introduced by Kaplan and Norton (1996) to replace the traditional performance system that typically focuses on assessing only financial performance. Incorporating multiple perspectives into performance evaluation allows for a more balanced perspec- tive of an organization’s performance, thus providing more useful information to leaders and managers (Niven, 2008; Tsai, Chou, & Hsu, 2009; Wu, Lin, & Chang, 2011). As reflected in Table 1:

� The customer perspective focuses on personal goals, assessed support needs, individualized supports, and personal outcomes. � The growth perspective focuses on program options, high

performance teams, direct support staff involvement, and networks, consortia, and partnerships. � The financial perspective focuses on a standardized approach to

calculating unit costs, cost accounting, cost allocation, social capital, fixed and variable costs, overhead rate, and resource allocation models. � The internal processes perspective focuses on horizontal and

vertical alignment of program components, mapping system(s), research and evaluation capacity, data sets, data collection systems, and quality improvement activities.

2.3. Collaborative approach to evaluation

A collaborative approach to evaluation is consistent with approaches such as participatory evaluation, utilization-focused evaluation, and empowerment evaluation (Fitzpatrick, Sanders, & Worthen, 2011; O’Sullivan, 2012; Patton, 2008). Collaborative evaluation involves organization participants, such as adminis- trators, managers, and knowledgeable support personnel, who are involved jointly in assessing or evaluating organization processes and functions. The ultimate goals of collaborative evaluation are to increase: (a) the knowledge and understanding of the evaluation/ assessment process; (b) the capacity for self-critique, self- determination, and systematic inquiry at the level of the individual and the organization; (c) organization learning that fosters shared values and understanding among organization members; and (d) the likelihood that the assessment’s findings will be incorporated into subsequent quality improvement efforts (Cousins & Choui- nard, 2012; Fitzpatrick, 2012; Luskin & Ho, 2013; Nichols, 2002; O’Sullivan, 2012; Rodriguez-Campos, 2012).

Collaborative evaluation has a number of benefits resulting from the use of the organization assessment tool described in Section 3. Among these benefits are increased knowledge and understanding of the evaluation/assessment process; enhanced capacity for systematic inquiry at the level of the individual and the organization; increased sensitivity to key concepts that include quality of life, personal outcomes, individualized supports, systems thinking, balanced scorecard, outcomes evaluation, alignment, continuous quality improvement, program logic models, and best practices; and an increased likelihood that the assessment’s findings will be incorporated into subsequent decision making to improve organization performance.

2.4. Continuous quality improvement

There is wide consensus that continuous quality improvement needs to be approached from a holistic and collaborative perspective and one based on best practices and actionable

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118112

information (Friedman, 2013; Kapucu, Healy, & Arslan, 2011; Krumdieck, 2013; McLaughlin & Kaluzny, 2004; Munk & Dempsey, 2010; Schalock & Verdugo, 2012). There is also agreement that these actions involve a sequential process that requires participa- tive leadership and a new leadership role: that of a transformation engineer (Krumdieck, 2013; Krumdieck & Page, 2012).

We define continuous quality improvement (CQI) as an integrative, sequential, and participatory process that is based on best practices and whose primary purpose is to enhance an organization’s effectiveness and efficiency from a multiple, performance-based perspective (see Table 1). We define effective- ness as the degree to which an organization’s intended results are achieved in reference to the customer and the organization’s growth. We define efficiency as the degree to which the organization produces its planned results in reference to financial analyses and internal processes.

The approach to CQI discussed in the article is integrative in that it incorporates the four elements or steps of the continuous improvement cycle of plan-do-evaluate-act that are discussed more fully in Section 4 and in the published work of Deming (2000), Six Sigma Cycle (2013), Richards (2013), Ries (2011), and Sokovic, Pavietic, and Kern Pipan (2010). The process is sequential in that self-assessment leads to planning which in turn leads to doing, evaluation (or checking), and action. It is participatory since organization personnel are involved in self-assessment and CQI activities. CQI cycle processes are described more fully in Section 4.

3. An evidence-based assessment tool

3.1. Overview

The Organization Effectiveness and Efficiency Scale (OEES) was developed by the International Consortium on Evidence-Based Practices (2013a, 2013b) based on the four component evaluation conceptual and measurement model just described (see Sections 2.1–2.4). The major purpose of the OEES is to assist nonprofit organizations meet the increasing need to be more effective in terms of achieving their intended results, more efficient in terms of their resources utilization, and more sustainable in terms of adapting to change and providing a range of sound service delivery opportunities and practices. The OEES is based on the 20 best practice indicators listed in Table 1 and can be used for multiple purposes including self-assessment and continuous quality improvement. Full details regarding its development, standardi- zation, multiple language versions, and on-line administration and scoring are available at: http://www.oeesonline.org.

The OEES reflects a new generation of organization-referenced assessment instruments that are based on a program logic model that incorporates input, throughput, and output components. As discussed by Hansen, Aiken, and Wallace (2013), the context or input component of such an assessment instrument involves evaluators or interviewers who have knowledge and technical skills regarding evaluation logic and methods, and participants/ respondents who are organization stakeholders who understand the concepts and processes being assessed, and who are decision makers committed to organization learning and CQI. The throughput or assessment component focuses on whether the obtained information is perceived as valid and credible, meets the information needs of the stakeholders, and can be integrated into the organization’s culture. The output component focuses on stakeholders increasing their understanding of both the organiza- tion and key quality improvement processes and strategies that involve best practices, evidence criteria, multiple perspectives, collaborative evaluation, individualized services and supports, personal outcomes, and organization outputs.

3.2. Development and standardization

As discussed more fully in the OEES Manual (International Consortium on Evidence-Based Practices, 2013b) the OEES was developed after a thorough literature review of program manage- ment and program evaluation literature that identified the most common best practice indicators used to evaluate organization performance and the four most common performance-based perspectives used in performance evaluation and management (see references associated with Table 1). Indicators were selected based on criteria related to the indicator’s validity, relevance, applicability across organizations, reliability, sensitivity, robust- ness, clarity and ease of understanding, and availability or ease of collection (Brown, Hatton, & Emerson, 2013; International Consortium on Evidence-Based Practices, 2013b). Once these two elements (i.e. indicators and perspectives) of the measure- ment framework were identified concept mapping (Kane & Trochim, 2007; Rosas & Camphausen, 2007; Rosas & Kane, 2012; Sutherland & Katz, 2005) was used by a panel of experts to aggregate five best practice indicators to each of the four performance-based perspectives (see Table 1).

The initial version of the OEES was developed using the following parameters. First, three evidence criteria were developed by the expert panel for each best practice indicator. These criteria were sequenced into logical, sequential steps through which an organization would proceed in planning, doing, and evaluating the indicator. The evidence criteria also provided the metric by which the level of the indicator could be assessed. Second, examples of evidence for each of the 20 indicators were developed based on discussions with service providers and consensus within the expert panel. These examples anchor the respective indicator to typical organization-based and generally available documents, reports, and processes. Third, a 3-point Likert scale was developed to assess the status of each indicator. This commonly used metric/ assessment technique was field tested to determine its sensitivity and utility in evaluating the level of each indicator. Scoring is based on the number of evidence criteria met. On this rating: 2 = 3 evidence criteria met; 1 = 1 or 2 evidence criteria met; and 0 = no evidence criteria available. Fourth, a description was written, along with administration and scoring instructions. This material was edited jointly by the instrument’s authors. Once completed, and the English version of the Initial Scale finalized, the OEES underwent three field tests. Slight revisions were made after each field test.

The third (and final) field test used the electronic administra- tion and scoring version of the OEES. The standardization sample involved 44 organizations in 8 countries. The majority (88.4%) of the organizations, which varied in size from 8 to 3700 clients, provided services and supports to person with IDD, with more than half (52.6%) providing services and supports to people of age, persons with emotional/behavioral problems, children and youth within special education, and persons with complex medical conditions. Based on scores from these 44 organizations, OEES item and perspective mean scores (and standard deviations) were obtained, along with internal consistency/reliability determina- tions. These data are presented in Table 2.

3.3. Evaluation procedure

The OEES is administered by an individual (internal or external to the organization) who is competent in assessment strategies and the collaborative approach to evaluation, and who is familiar with organization management and the evaluation conceptual and measurement model described in Section 2. At least two respondents are interviewed. These individuals are managerial level or above in the organization and need to be familiar with data

Table 2 OEES item mean scores and Cronbach’s alpha reliability coefficients.

Perspective Item mean

and standard

deviation

Perspective mean

and standard

deviation

Cronbach’s

alpha coefficients

Client 1. 1.47 (.63) 5.56 (2.42) .73

2.0.98 (.71)

3. 1.3 (.64)

4. 0.88 (.76)

5. 0.93 (.67)

Growth 6. 1.42 (.55) 6.49 (2.13) .70

7. 1.47 (.55)

8. 1.26 (.69)

9. 1.16 (.72)

10. 1.19 (.63)

Financial 11. 1.12 (.73) 4.60 (2.53) .75

12. 1.12 (.70)

13. 0.56 (.59)

14. 0.84 (.75)

15. 0.98 (.80)

Internal

processes

16. 0.98 (.59) 4.49 (2.53) .81

17. 0.77 (.65)

18. 0.91 (.68)

19. 0.98 (.67)

20. 0.86 (.77)

Fig. 1. Graphic summaries of an organization’s perspectives profile and OEES indices.

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118 113

sets in the organization’s management information system, and knowledgeable in how to access and interpret information.

Consistent with the collaborative approach to evaluation, the interviewer uses a conversation format to obtain from the respondents a consensus score for each of the 20 indicators. A template is provided to the Interviewer to facilitate this process. The essential components of this template are presented in Table 3 in reference to the customer perspective. An analogous format/ template is used for the other three perspectives.

3.4. Summary information

Indicator scores are aggregated into four performance-based perspective profiles that reflect the perspective of the customer, and the organization’s growth, financial analyses, and internal processes. These profiles are depicted graphically in a Radar Chart such as that shown in Fig. 1. Three evidence-based indices are also computed and depicted graphically, as shown in the Dash Board presented in the bottom section of Fig. 1: An Effectiveness Index (the total of the Customer and Growth Perspectives), An Efficiency Index (total of the Financial and Internal Processes Perspectives), and a Sustainability Index (total of the two indices). These profiles and indices are computed in real time and are available to the interviewer and respondents immediately following the on-line assessment. This summary information along with item raw scores can be used as a basis for quality improvement, which we describe next.

Table 3 OEES assessment template.

Perspective Evidence-based indicator Evidence criteriaa

Customer Aligns services/supports to

identified support needs

- Develops a standardized approach to

assessment of individual support need

are important to and for the individua

- Uses that information to develop an

Individual Supports Plan

- Determines that individualized suppo

provided are aligned with support nee

that are important to and for the indiv

a Sequenced according to plan-do-evaluate quality improvement cycle.

4. Quality improvement process steps

4.1. Key implementation factors

Continuous quality improvement (CQI) is probably one of the greatest challenges faced by any organization. It is like fighting gravity. The natural tendency is disorder and quality decay, the second law of thermal dynamics. CQI needs to be part of any organization’s deep culture in order to be a reality. We have found that five key factors make it possible for CQI to be deeply implanted in an organization’s culture. First, CQI framework, CQI is a transformative process that requires a conceptual, measurement, and implementation framework such as the quality improvement loop presented in Fig. 2 that incorporates the continuous improvement approaches/cycles discussed by Deming (2000) and others (e.g. Richards, 2013; Ries, 2011), and the quality implementation framework developed by Meyers, Durlak, and Wandersman (2012). Such a CQI framework needs to be sensitive to the organization’s receptivity, furthers the organization’s unique competitive position, provides a mix of values to stakeholders, and be easily understood and taught via consultation and learning teams but within the constraints of organization resources. Second, evidence-based, concrete and objective data are absolutely necessary to make quality issues evident. Although one should manage on the basis of vision and not numbers, numbers generally focus people’s attention. Third, performance-based perspectives,

Examples of evidence Scoring of evidence-based indicator

the

s that

l

Assessed support needs data;

incorporation of support needs

data into the supports plan

2 1 0

2 = 3 evidence criteria met

1 = 1 or 2 evidence criteria met

rts

ds

idual

0 = no evidence criteria met

Do

Evalua te

Act

Plan based o n self -ass ess men t

Fig. 2. Continuous improvement framework.

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118114

quality is an issue in all areas of an organization. It needs to be holistic and comprehensive, not just in products, services, or programs, but also in teams, processes, and procedures. Fourth, an evidence-based self-assessment tool, ongoing use of a quality- improvement focused, self-assessment tool such as the OEES facilitates CQI by increasing knowledge and understanding of the evaluation/assessment process and the likelihood that the self- assessment findings will be incorporated into subsequent decision making. Fifth, collaborative implementation, collaboration creates a deep understanding that is necessary before any quality improve- ment is possible. There are levels of mastery involved here: to thoroughly know and practice the QI strategy (ies), thoroughly understand the principles behind the strategy (ies), and creatively developing improvements. An honest, transparent collaborative process is necessary for understanding across stakeholders for genuine improvement to occur. Otherwise it is just all talk and no practice.

4.2. Quality improvement framework

The quality improvement framework presented in Fig. 2 shows the integrative, sequential, and participatory nature of CQI. Fig. 2 also depicts the four quality improvement process steps encom- passing CQI: planning based on self-assessment, doing, evaluating, and acting.

4.2.1. Planning

Planning is a disciplined effort to produce fundamental decisions and actions that shape what an organization is, what it does with its resources, and why it does it, with a focus on best practices. Planning is based on self-assessment, builds on a shared vision that is values-based and action oriented, and results in the alignment of the organization’s resources to personal outcomes and organizational outputs.

Planning needs to be based on an organization’s quality improvement (QI) needs as reflected in what is important to and for the organization. Organization goals represent what is important to the organization. Assessed QI needs reflect what is important for the organization. The OEES represents a balanced approach to determining an organization’s QI needs. For example, from the customer’s perspective, the QI needs might be to focus on

personal goals, assessed support needs, individualized supports, and personal outcomes. From the organization’s growth perspec- tive, the QI needs might be to focus on developing program options, implementing high performance teams, increasing direct staff involvement, and increasing networking, consortia membership, and/or partnerships. From the organization’s financial perspective, the QI needs might be to focus on a standardized approach to calculating unit costs, developing a cost allocation formula, increasing social capital, evaluating overhead rate, and/or imple- menting a resource allocation model. From an organization’s internal processes perspective, the QI needs might be to focus on horizontal and vertical alignment, research and evaluation capacity, data sets and data collection systems, and/or quality improvement activities.

4.2.2. Doing

Human service organizations are changing what they do in reference to developing and implementing quality improvement plans (QIP). Generally speaking, these ‘doing activities’ employ a participative approach in which a Quality Improvement Team uses an outcome or output-focused program planning format, employs user-friendly formats, synthesizes or integrates goals and QI needs with specific strategies, develops quality improvement objectives that have an action verb, a specific strategy, and an intended result, and develops, implements, monitors, and evaluates the respective plan.

A Quality Improvement Team provides the vehicle for ensuring that quality improvement is a participative process. The team: (a) is characterized by their being involved, informed, organized, empowered, and accountable; and (b) is a horizontally structured work group that exhibits a sense of ownership and task completion. Furthermore, the Team is composed of stakeholders who are knowledgeable about the organization’s policies, prac- tices, and information systems (e.g. managers), and who are involved in implementing organization practices (e.g. support coordinators and direct support staff). The Team’s major responsi- bilities are to determine organization goals and assessed QI needs, identify quality improvement strategies, develop quality improve- ment objectives, and develop, monitor, and evaluate the QIP.

Doing involves implementing quality improvement strategies. A system of quality improvement strategies has emerged from the transformation era and the organizational changes that are occurring commensurate with this era (Schalock & Verdugo, 2012, 2013). Exemplary components of such a system are provided in Table 4. It should be noted that the strategies within each performance-based perspective reflect the actual best practice indicators assessed on the OEES.

One of the challenges faced by any organization embarking on CQI efforts is to use a process that is realistically feasible, efficient in terms of resource utilization, and effective in terms of results. As stated by one of the reviewers of this article, ‘‘I have been privileged to work with literally hundreds of nonprofit, small organizations and a surprising number have been led through the strategic planning process in the nature of technical assistance, training, or capacity building. The vast majority of those plans are never implemented after the consultant has completed the training and departed.’’

To overcome this common situation and to implement an organization-based participative approach to CQI that is function- al, relevant, and outcome oriented, ‘user-friendly’ quality improve- ment formats have emerged in a number of jurisdictions (Schalock & Verdugo, 2013). An example is shown in Fig. 3. As depicted in the first column, the QI area/perspective is one or more of the four performance-based perspectives assessed on the OEES. The second column focuses on the most important organization goals and assessed QI needs. Operationally, what is important to the

Quali ty I mpr ovement Area

Most Import ant Organization Goals and

Ass essed Quality Impr ovement Needs

Quali ty I mpr ovement Strategies B ased on a

Syste m of Quali ty Impr ovement Needs

Quali ty I mpr ovement Objective for Each

Quali ty I mpr ovement Strategy

Customer Perspective

Growth Perspective

Financial Analysis Perspective

Inter nal P rocesses Perspective

Fig. 3. Quality improvement plan format.

Table 4 A system of exemplary quality improvement strategies.

Perspective Exemplary quality improvement strategies

Customer - Aligns services/supports to identified support needs

- Measures personal outcomes

- Uses technology to enhance personal outcomes

- Implements a system of supports

Growth - Enters into partnerships (e.g. networks, consortia)

- Develops program options

- Utilizes high performance teams (e.g. Support Team;

Quality Improvement Team)

- Monitors job satisfaction and develops job enrichment

programs

Financial - Reports percentage of budget allocated to

client-referenced supports

- Monitors the relationship between social capital and

agency-based fiscal capital

- Analyzes overhead rate to increase efficiency

- Bases resource allocation on major cost drivers

Internal processes - Horizontally aligns input, throughput, and output

components

- Vertically aligns the organization’s input, throughput,

and output components to the corresponding

individual-level input, throughput, and outcome

components

- Aligns information systems to performance-based

perspectives

- Increases knowledge transfer through real-time

information technology

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118 115

organization is based on maintaining what explicitly needs to stay as is, and changing those things for which a desire or need to change exists; what is important for the organization is based on a standardized assessment of the organization’s QI needs such as that accomplished via the OEES.

Specific quality improvement strategies are based on those exemplary strategies listed in Table 4, or others taken from the OEES, previous strategies used successfully by the organization, or published literature. For each QI strategy, there is a QI objective that includes an action verb (e.g. implements), specific support strategy (as exemplified in Table 4), and an intended result. These two columns (QI strategies and QI objectives) included in Fig. 3 become the focus for evaluation, which is discussed next.

4.2.3. Evaluating

The term ‘evaluation’ is used differently depending on the quality improvement approach. For some, it refers to checking or studying (Deming, 2000); to others it refers to analyzing (Six Sigma Cycle, 2013), deciding (Richards, 2013), learning (Ries, 2011), or

assessing/refining (Sokovic et al., 2010). In reference to the quality improvement framework presented in Fig. 2, and the quality improvement plan format presented in Fig. 3, evaluation activities can be based the status of the quality improvement objectives and/ or reassessment on the OEES.

Evaluating the status of quality improvement objectives begins with the QI Team determining the status of the respective QI strategy. Since the focus of CQI is on implementing specific quality improvement strategies, the monitoring process involves these four steps: (1) listing each QI strategy; (2) listing the person(s) responsible for implementing the strategy; (3) evaluating the status of each strategy using a 3-point Likert scale: 2 = fully implemented, 1 = partially implemented, and 0 = not implemen- ted; and (4) analyzing the results of the monitoring. In reference to the respective QI strategy: (a) if the strategy has been fully implemented, does it need to remain and if so, in the same way; (b) if the strategy has been partially implemented, what is the reason for the partial implementation: is the strategy not clear, is it not in place, or do the persons involved in its implementation need further clarification, training, or implementation techniques; and (c) if the strategy has not been implemented, why? The actual evaluation of a QIP involves assessing the status of the intended result(s) of the specific quality improvement objective. Did the objective produce its intended result? If yes, the QIP should be modified to include another priority QI need; if not, those questions referenced in (4) above need to be addressed.

Self-evaluation using the OEES can occur at both the beginning of the CQI process and after planning and doing have occurred. Each time frame represents different purposes. When the OEES is used for initial self-assessment, the information obtained from the assessment is used primarily for planning and doing. Subsequent self-assessment information is typically used for reporting, benchmarking, and as a basis for additional quality improvement activities. However, this evaluation information is used for organiza- tion-referenced activities and not to compare organizations.

4.2.4. Acting

As part of the QI framework depicted in Fig. 2, action refers to ‘what happens after.’ That is, what happens after planning based on self-assessment has occurred, and after a quality improvement plan has been developed, implemented, monitored, and evaluat- ed? As part of the integrative, sequential, and participative quality improvement process, action can focus on capacity building in reference to services and supports, resource development, and research and evaluation, or on strategic anchors. The following two examples of these QI actions are based on the authors’ experiences to date with the OEES. Undoubtedly others will emerge.

Capacity building improves an organization’s ability to achieve its mission in an effective and efficient manner (Crisp, Swerissen, & Duckett, 2000; Johnson, Hays, Center, & Daley, 2004; Letts et al., 1999; Levine, Russ-Eft, Burling, Stephens, & Downey, 2013; Loza, 2004; Millesen & Bies, 2007; Schuh & Leviton, 2006; Sobeck & Agius, 2007; Stevenson, Florin, Mills, & Andrade, 2002). The best practice indicators assessed on the OEES can be used as capacity building strategies in three areas: services and supports, resource development, and research and evaluation. Table 5 presents a number of exemplary quality improvement strategies associated with each of these three capacity building areas. These strategies are based on either the 20 indicators assessed on the OEES, and/or the evidence criteria used to evaluate the respective indicator.

As a capacity building example, the KARE nonprofit organiza- tion servicing persons with IDD in South Kildare, Ireland has used the capacity building focus and exemplary quality improvement strategies matrix presented in Table 5 to develop a Capacity Building Profile based on scores from the OEES (Kelly & Lynch, 2013). Based on this profile, they identify and prioritize areas for

Table 5 Exemplary quality improvement strategies used for capacity building.

Capacity

building focus

Exemplary quality improvement strategy

Services

and supports

- Aligns services/supports to identified support needs

- Develops program options

- Horizontally aligns input, throughput, and output

program components

- Vertically aligns an organization’s input, throughput,

and output components to the corresponding

individual-level input, throughput, and output

components

Resource

development

- Uses technology to enhance personal outcomes

- Enters into partnerships

- Utilizes high performance teams

- Monitors job satisfaction and develops job enrichment

programs

- Analyzes overhead rate to increase efficiency

Research and

evaluation

- Measures personal and organizational outcomes

- Reports and analyses aggregated personal outcomes

- Compares unit costs across different locations and

service delivery platforms

- Demonstrates the relationship between units of

service/supports provided and the clienteles’

assessed support needs

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118116

capacity building based on those strategies that will most impact their organization’s effectiveness and efficiency. In their initial assessment, research and evaluation was the lowest scoring capacity building area. As a result, they have introduced a standardized approach to the assessment of personal outcomes, and established a standardized approach to calculating unit costs across locations and programs.

As a second example, strategic anchors, which are believed by organization personnel to move the organization toward its vision, can be based on OEES items that are deemed most critical to the organization’s current state of development. A strategic action map depicts the organization’s main actions/activities in relation to the strategic anchors so that everyone within the organization is clear on their roles in achieving the organization’s purpose and vision. The map might include strategic anchors such as ‘enhance employee professional life development, develop and implement customer-centered service system, and building partnerships to evangelize core service values and concepts’. These anchors are then associated with specific items on the OEES, which can be used as the framework for quality improvement strategies (Lee, 2013).

5. Strengths and challenges of an evidence-based approach to organization evaluation and change

5.1. Strengths

One of the major strengths of an evidence-based assessment instrument such as the OEES is that it is an organization assessment tool that represents a new approach to organization evaluation based on self-assessment. This approach integrates into continu- ous quality improvement best practice indicators, multiple performance-based perspectives, a collaborative approach to self-assessment, and a participative and integrative approach to CQI. A second strength is that leaders and managers are provided with an approach to performance evaluation that is clear, simple, and actionable. The conceptual and measurement model presented in Section 2 indicates clearly how an evidence-based approach to organization evaluation and change can be based on clearly stated and operationalized indicators, the multiple perspectives on performance-based evaluation and management, a collaborative approach to evaluation, and an integrative approach to CQI. Third, the approach involves: (a) evaluators or interviewers who have

knowledge and technical skills regarding evaluation logic and methods; (b) respondents who understand the concepts and processes being assessed and are decision makers who are committed to organization learning and organization transforma- tion; (c) stakeholders who have a shared understanding of an organization’s policies and practices; (d) information that meets the needs of stakeholders, is responsive, relevant, and timely, and can be integrated into the organization’s culture; and (e) stakeholders who use the assessment information for decision making and CQI.

5.2. Challenges

One of the challenges to an evidence-based approach to organization evaluation and change involves self-assessment. For example, throughout the three field tests of the OEES and its subsequent uses, we have found that respondents are sometimes hesitant to be honest and open regarding the level of scoring on the self-assessment, focusing typically on what is desired as opposed to what is. It is common for IDD organizations to have been assessed or evaluated for purposes related to licensure, certifica- tion, accreditation, or funding. Additionally, since they have been compared to other organizations for various purposes, respondents want their organization to ‘look good.’ As a result, valid self- assessment requires a new mind-set that involves four insights: (a) interviewers and respondents need to understand that self- assessment is an internal organization process that involves a set of best practices that frame both the collaborative evaluation process and quality improvement; (b) organization personnel must be honest in their assessment of the status of indicators and formulate their evaluation on the basis of ‘what is’ rather than ‘what someone might want to see’; (c) the evaluation process needs to be viewed as a collaborative effort that increases knowledge and understanding of the evaluation/assessment process, that encourages self-critique and systematic inquiry at the level of the individual and the organization, that enhances organization learning, and that allows organization personnel to incorporate assessment findings into subsequent decision making; and (d) all stakeholders need to realize that collaborative evaluation is consistent with the emerging participative scientific research method (Nielsen, 2011; Toerpe, 2013).

A second challenge relates to how a self-assessment instrument using organization-based participants can be integrated with other performance evaluation and management systems that are frequently mandated or highly recommended by specific jurisdic- tions. Examples are CARF standards in the U.S. and Canada, and the EFQM Business Excellence Model used widely in Europe (Heras- Saizarbitoria, Casadesus, and Marimon, 2011; Vallejo et al., 2006). The OEES was not developed to replace these systems, but to augment them in reference to organization evaluation and change.

A third challenge is to determine what constitutes best practices across service delivery fields. Although there is some suggestion that the 20 indicators listed in Table 1 apply to special education and human service programs dealing with people of age and those with mental and behavioral impairments (see e.g. Schalock & Verdugo, 2012), we have yet to use the OEES widely in these fields. Thus, generalizations about the approach described in this article should be limited at this point to those organizations providing services and supports to persons with IDD.

6. Future issues

Reviewers of earlier versions of this article offered valuable content and format suggestions, and also raised important conceptual and measurement issues that need to be addressed in subsequent work. Although we have integrated their suggestions

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118 117

into the present article, future work will be impacted by a number of issues not fully addressed in this article. Chief among these are, first, what are the short term and long-term benefits to organizations of using an evidence-based approach to organiza- tion evaluation and change? Work presented in the article relates only to the initial implementation and in-house evaluation of the processes described. Second, does a management approach to quality control such as reflected in the quality improvement framework depicted in Fig. 2 lends itself to more general program evaluation? We think it does, and the indicators incorporated into the OEES were gleaned from both management and program evaluation literature. Third, what does standardization mean in reference to an organization self-assessment tool whose results are to be used primarily as a basis for CQI? We stress that any comparisons are within the organization over time and not between organizations. Fourth, what is the nature of continuous quality improvement? Is it a process-level function that targets incremental process effectiveness, or is it efficiency manage- ment at the micro-level, or organization change and transfor- mation at the macro-level? We know it is incremental, but full implementation and impact studies need to be done.

The fifth issue identified by the reviewers was, is an evidence- based approach to organization evaluation/self-assessment ap- propriate to both small and large organizations? The two are potentially different regarding the market from which they derive their resources, the internal regulatory functions, and the more informal processes of smaller organizations. Our three field tests were done across a wide range of organization types, sizes and target populations, but we have yet to demonstrate increased utilization, effectiveness, or efficiency as a function of size. Sixth, what organization outputs should be used to evaluate the results or impacts of quality improvement activities? Although Effective- ness, Efficiency, and Sustainability Indices are provided as summary information, there are other ways to operationalize organization outputs. For example, as one reviewer suggested, ‘‘Effectiveness from a measurement perspective is a function of the number of successful interventions divided by the total number of interventions (i.e. E = S/A), whereas efficiency is the ratio of outputs to process (O/P)’’. Part of this issue may be the focus of the discussion: personal outcomes vs. organization outputs. Seventh, what does sustainability mean? Is it adapting to change and providing a range of sound service delivery opportunities and practices as defined in this article, or does sustainability relate more to stability of funding and diversity of funding sources? This issue points out the importance of embracing the four perfor- mance-based perspectives. Finally, what is the role of context? Although the challenges faced by IDD organizations are quite similar across the eight countries involved in the field tests, the terms and concepts used in the evaluation and conceptual model (Section 2) and the OEES (Section 3) do vary across language/ cultural groupings. We approached this challenge by: (a) using the back translation method of Brislin (1986) in finalizing the Chinese, Dutch, and Spanish versions of the Scale; (b) developing a Glossary that is an integral part of the OEES Manual and training activities; and (c) developing an OEES Interviewer Template (part of which is shown in Table 3). Both (b) and (c) are available online in the OEES Manual.

In conclusion, human service organizations are undergoing change and transformation. Chief among these are first, the person is central. Accompanying this change is the shift from general services to individualized supports, and the alignment of person- centered values with service delivery practices. These person- centered values relate to quality of life, self-determination, inclusion, empowerment, and equity; the service delivery prac- tices relate to the assessment of personal goals and person-refer- enced support needs, the provision of an individualized system of

supports, and the evaluation of personal outcomes. Second, organizations are becoming more streamlined with a correspond- ing movement from vertical to horizontal structure that is accompanied by the increasing use of collaborative approaches to organization leadership and management, and the implemen- tation of high performance teams. Third, data systems are becoming information based and organized around perfor- mance-based perspectives that provide a balanced scorecard that can be used for self-assessment, reporting, benchmarking, and quality improvement. Fourth, quality improvement is a continuous process that integrates self-assessment with specific quality improvement strategies. Across these four changes, both organi- zation self-assessment and participative quality improvement activities require an honest and transparent collaborative effort for organization change to occur.

References

Bishop, S. (2007). Linking nonprofit capacity effectiveness in the new public manage- ment era: The case study of community action agencies. State and Local Government Review, 39, 144–152.

Bourgeois, I., Hart, R. E., Townsend, S. H., & Gagne, M. (2011). Using hybrid models to support the development of organization evaluation capacity: A case narrative. Evaluation and Program Planning, 34, 228–235.

Brislin, R. W. (1986). The wording and translation of research instruments. In W. Kinner & I. Berry (Eds.), Field methods in cross-cultural research (pp. 137–164). Beverly Hills, CA: SAGE.

Brown, I., Hatton, C., & Emerson, E. (2013). Quality of life indicators for individuals with intellectual disabilities: Extending current practices. Intellectual and Developmental Disabilities, 51, 316–332.

Claes, C., van Loon, J., Vandevelde, S., & Schalock, R. L. (in press). Standards and guidelines for evaluating and implementing evidence-based practices in the field of intellectual and closely related developmental disabilities. Evaluation and Program Planning (in press).

Cooksy, L. J., Gill, P., & Kelly, P. A. (2001). The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning, 24, 119–128.

Cousins, J. B., & Chouinard, J. (2012). Participative evaluation up close: A review and interpretation of research-based knowledge. Charlotte, NC: Information Age Press.

Crisp, B. R., Swerissen, H., & Duckett, S. J. (2000). Four approaches to capacity building in health: Consequences for measurement and accountability. Health Promotion International, 15, 99–107.

Deming, W. E. (2000). Out of crisis. Cambridge, MA: First MIT Press. Donaldson, S. E. (2007). Program theory-driven evaluation science. Mahwah, NJ: Lawr-

ence Erlbaum Associates. Fitzpatrick, J. L. (2012). Commentary: Collaborative evaluation within the larger

evaluation context. Evaluation and Program Planning, 35, 558–563. Fitzpatrick, J. L., Sanders, J. P., & Worthen, B. R. (2011). Program evaluation: Alternative

approaches and practical guidelines (4th ed.). Boston: Allyn and Bacon. Friedman, T. L. (2013). It’s a 401(k) world. New York Times, A. 25. Fuller, G. W. (1997). Key performance indicators for benchmarking health and safety

management in intra-and inter-company comparisons. Benchmarking for Quality Management and Technology, 4, 165–180.

Grol, R., Baker, R., & Moss, E. (Eds.). (2004). Quality improvement research. London: BMJ Books.

Hansen, M., Aiken, M. C., & Wallace, T. C. (2013). Depicting the logic of three evaluation theories. Evaluation and Program Planning, 38, 34–43.

Heras-Saizarbitoria, I., Casadesus, M., & Marimon, F. (2011). The impact of the ISO 9001 standard and the EFQM model: The view of assessors. Total Quality Management, 22, 197–218.

Hunter, D. E. K. (2006). Using a theory of change approach to build organization strength, capacity, and sustainability with not-for-profit organizations in the human services sector. Evaluation and Program Planning, 29, 193–200.

International Research Consortium on Evidence-Based Practices. (2013a). Organization Effectiveness and Efficiency Scale Retrieved from: http://www.oeesonline.org.

International Research Consortium on Evidence-Based Practices. (2013b). Organization Effectiveness and Efficiency Scale. Manual Retrieved from: http://www.oeesonli- ne.org.

Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building capacity and sustainable prevention innovations: A sustainability planning model. Evaluation and Program Planning, 27, 135–149.

Kane, M., & Trochim, W. (2007). A review of concept mapping for planning and evaluation. Thousand Oaks, CA: SAGE.

Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action. Boston: Harvard Business Press.

Kapucu, N., Healy, B. F., & Arslan, T. (2011). Survival of the fittest: Capacity building for small nonprofit organizations. Evaluation and Program Planning, 34, 236–245.

Kelly, S., & Lynch, C. (2013). The Organization Effectiveness and Efficiency Scale and capacity building. In International research consortium on evidence-based practices manual (pp. 59–61) Retrieved from: http://www.oeesonline.org.

R.L. Schalock et al. / Evaluation and Program Planning 45 (2014) 110–118118

Kong, E. (2003). Using intellectual capital as a strategic tool for non-profit organiza- tions. International Journal of Knowledge, Culture, and Change Management, 3, 467– 474.

Krumdieck, S. (2013). Transition engineering: Planning and building the sustainable world. The Futurist, July–August, 35–41.

Krumdieck, S. M., & Page, D. S. (2012). Design and implementation of a community based sustainable research method. Social Business, 2, 291–337.

Lee, T. (2013). OEES and the formulation of organizational strategy. In International research consortium on evidence-based practices manual (pp. 56–58) Retrieved from: http://www.oeesonline.org.

Lencioni, P. M. (2012). The advantage: Why organizational health trumps everything else in business. San Francisco: Jossey-Bass.

Letts, C. W., Ryan, W. P., & Grossman, A. (1999). High performance non-profit organiza- tions. Managing upstream for greater impact. New York: John Wiley & Sons.

Levine, R., Russ-Eft, D., Burling, A., Stephens, J., & Downey, J. (2013). Evaluating health services research capacity building programs: Implications for health service and human resource development. Evaluation and Program Planning, 37, 1–11.

Lick, D. W. (2006). A new perspective on organizational learning: Creating learning teams. Evaluation and Program Planning, 29, 88–96.

Loza, J. (2004). Business-community partnerships: The case for community organiza- tion capacity building. Journal of Business Ethics, 53, 297–311.

Luskin, R. J. C., & Ho, T. (2013). Comparing the consequences of three theories of evaluation. Evaluation and Program Planning, 38, 61–66.

McLaughlin, C. P., & Kaluzny, A. D. (2004). Continuous quality improvement in health care. Sudbury, MA: Jones and Bartlett.

Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The quality implementation framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50, 462–480.

Millesen, J., & Bies, L. (2007). Nonprofit capacity building: Who is doing what for whom and to what end? Journal for Nonprofit Management, 11, 18–27.

Munk, D. D., & Dempsey, T. L. (2010). Leadership strategies for successful school-wide inclusion: The STAR approach. Baltimore: Paul H Brookes.

Nichols, L. (2002). Participatory program planning: Including program participants and evaluators. Evaluation and Program Planning, 25, 1–14.

Nielsen, M. (2011). Reinventing discovery: The new era of networked services. Princeton, NJ: Princeton University Press.

Niven, P. R. (2008). Balanced scorecard step-by-step for government and non-profit agencies (2nd ed.). Hoboken, NJ: John Wiley & Sons.

O’Sullivan, R. G. (2012). Collaborative evaluation within a framework of stakeholder- oriented evaluation approaches. Evaluation and Program Planning, 35, 518–522.

Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks, CA: SAGE. Pawson, R. (2006). Evidence-based policy: A realistic perspective. London: SAGE. Pluye, P., Potvin, L., Denis, J.-L., Pelletier, J., & Mannoni, C. (2005). Program sustainability

begins with the first events. Evaluation and Program Planning, 28, 123–137. Richards, C. W. (2013). Certain to win: The strategy of John Boyd applied to business.

Bloomington, IN: Xlibris Corporation. Ries, E. (2011). The lean setup: How today’s entrepreneurs use continuous innovation to

create radically successful businesses. New York: Crown Publishing Group. Rodriguez-Campos, L. (2012). Advances in collaborative evaluation. Evaluation and

Program Planning, 35, 523–528. Rosas, S. R., & Camphausen, L. C. (2007). The use of concept mapping for scale

development and validation in evaluation. Evaluation and Program Planning, 30, 125–135.

Rosas, S. R., & Kane, M. (2012). Quality and rigor of the concept mapping methodology: A pooled analysis. Evaluation and Program Planning, 35, 236–245.

Schalock, R. L., & Verdugo, M. A. (2012). A leadership guide for today’s disabilities organizations: Overcoming challenges and making change happen. Baltimore: Brookes Publishing Company.

Schalock, R. L., & Verdugo, M. A. (2013). The transformation of disabilities organiza- tions. Intellectual and Developmental Disabilities, 51, 273–286.

Schalock, R. L., Verdugo, M. A., & Gomez, L. E. (2011). Evidence-based practices in the field of intellectual and developmental disabilities. Evaluation and Program Plan- ning, 34, 273–282.

Scheirer, M. A. (2005). Is sustainability possible? A review and commentary on empirical studies of program sustainability. American Journal of Evaluation, 26, 320–347.

Scheier, M. A., Hartling, G., & Hagerman, D. (2008). Defining sustainability: Outcomes of health programs: Illustrations from an on-line survey. Evaluation and Program Planning, 31, 335–346.

Schuh, R. G., & Leviton, L. C. (2006). A framework to assess the development and capacity of non-profit agencies. Evaluation and Program Planning, 29, 171–179.

Selden, S., & Sowa, J. E. (2011). Performance management appraisal in human service organizations: Management and staff perspectives. Public Personnel Management, 40, 214–251.

Six Sigma Cycle (2013). Retrieved from: http://www.sixsigmaonline.org. Sobeck, J., & Agius, E. (2007). Organizational capacity building: Addressing a research

and practice gap. Evaluation and Program Planning, 30, 237–246. Sokovic, M., Pavietic, D., & Kern Pipan, K. (2010). Quality improvement methodolo-

gies—PDCA cycle, RADAR matrix, DMAIC, and DFSS. Journal of Achievements in Materials and Manufacturing Engineering, 43, 476–483.

Stevenson, J. F., Florin, P., Mills, D. S., & Andrade, M. (2002). Building evaluation capacity in human service organizations: A case study. Evaluation and Program Planning, 25, 233–238.

Sutherland, S., & Katz, S. (2005). Concept mapping methodology: A catalyst for organizational learning. Evaluation and Program Planning, 28, 257–269.

Toerpe, K. (2013, July–August). The rise of citizen science. The Futurist, 25–30. Tsai, W. H., Chou, W. C., & Hsu, W. (2009). The sustainability balanced scorecard as a

framework for selecting socially responsible investment. Journal of Operational Research Society, 60, 1396–1410.

Vallejo, P., Saura, R. M., Sunol, R., Kazandjian, V., Urena, V., & Mauri, J. (2006). A proposed adaptation of the EFQM fundamental concepts of excellence to health care based on the PATH framework. International Journal for Quality Health Care, 8, 327–335.

van Loon, J. H. M., Bonham, G. S., Peterson, D. D., Schalock, R. L., Claes, C., & Decramer, A. E. (2013). The use of evidence-based outcomes in systems and organizations providing services and supports to persons with intellectual disability. Evaluation and Program Planning, 36, 80–87.

Veerman, J. W., & van Yperen, T. A. (2007). Degrees of freedom and degrees of certainty: A developmental model for the establishment of evidence-based youth care. Evaluation and Program Planning, 30, 212–221.

Wasserman, D. L. (2010). Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations. Evaluation and Pro- gram Planning, 33, 67–80.

Wu, H.-Y., Lin, Y.-K., & Chang, C.-H. (2011). Performance evaluation of extension education centers in universities based on the balanced scorecard. Evaluation and Program Planning, 34, 37–50.

Robert L. Schalock Professor Emeritus, Hastings College (Nebraska), Adjunct Professor University of Salamanca, Spain and Gent University, Belgium.

Tim Lee Executive Director, Qi Zhi Vocational Training Center, Taipei, Taiwan.

Miguel Verdugo Professor of Psychology and Director of the Institute on Community Inclusion (INICO), University of Salamanca, Spain.

Kees Swart Microsoft Certified Professional and Web-developer, Arduin Foundation, Middleburgh, The Netherlands.

Claudia Claes Lecturer, Faculty of Social Work and Welfare Studies, University College, Gent University, Belgium.

Jos van Loon Psychologist, Manager, and Researcher, Arduin Foundation (Middle- burgh, The Netherlands) and Visiting Professor, Department of Special Education, University of Gent, Belgium.

Chun-Chin Lee Chairman of the Board, Qi Zhi Vocational Training Center, Taipei, Taiwan.