Levels of Evidence Table

Measuring outcomes and managing for results

Robert L. Schalocka,b,c,*, Gordon S. Bonhamd

aHastings College, USA bBeach Center on Disabilities, USA

cP.O. Box 285, Chewelah, WA 99109, USA dBonham Research, USA

Abstract

Public and private health and human service organizations currently face two challenges regarding measuring outcomes and managing for

results: demonstrating fiscal and programmatic accountability, and using person and organization outcomes for continuous program

improvement. Both managers and program evaluators have attempted to respond to these two challenges through techniques and strategies

such as total quality management and improved evaluation utilization techniques. Despite these efforts, three difficulties in measuring

outcomes and managing for results are typically reported and involve the lack of: (a) a program evaluation model and measurement methods

that clearly delineate organization and individual-referenced outcomes and that meet the dual requirements of increased accountability and

continuous program improvement; (b) a program logic model that helps program managers see the relationship among inputs, processes,

and outputs and the key roles played by formative feedback and contextual variables in managing for results; and (c) a mechanism to manage

for results that includes feedback to service providers, a quality improvement process, and performance standards. The present article

discusses strategies to overcome these three difficulties based on the authors’ work over the last 6 years with a participatory action research

and evaluation project for persons with developmental disabilities.

q 2003 Published by Elsevier Science Ltd.

Keywords: Feedback; Management; Outcomes; Outputs; Program evaluation; Results

1. Introduction

Public and private health and human service

organizations currently face two challenges: demonstrating

fiscal and programmatic accountability, and using person

and organization outcomes for continuous program

improvement. Both of these challenges stem from the

quality revolution of the 1980s and the reform movement of

the 1990s that have brought about significant changes in

how people view the purposes, characteristics,

responsibilities, and desired outcomes from health and

human service programs. Major characteristics of these

changes involve focusing on outcomes rather than inputs,

being guided by goals related to person-referenced and

valued outcomes, redefining clients as customers, and

decentralizing authority (Gardner & Nudler, 1999; Hakes,

2001; Mawhood, 1997; Newcomer, 1997; Schalock, 2001;

Wholey, 1997).

Both managers and program evaluators have attempted

to respond to these challenges through strategies such as

total quality management (TQM) and improved evaluation

utilization techniques. TQM involves management

techniques related to strong quality leadership, consumer

orientation, continuous improvement, data-driven decision

making, teamwork, and a focus on organization process

(Albin-Dean & Mank, 1997; Drucker, 1998; Hodges &

Hernandez, 1999; Hoffman, Lechman, Russo, & Knauf,

1999). In reference to evaluation utilization, numerous

models have been developed to improve evaluation

utilization, including those of Johnson (1998) and Patton

(1997). Common themes among these models include:

the importance of stakeholder involvement in planning and

implementation; the use of evaluation information for

programmatic change and improvement; the key roles that

both the program’s internal and external environments play

in the utilization of results and programmatic change;

the need to impact decision maker’s understanding of—and

commitment to—change based on the evaluation data;

the necessity of changing managers’ performance based on

the evaluation’s results; and the realistic potential for

organization learning to occur.

Despite these efforts, three difficulties in measuring

outcomes and managing for results are typically reported

0149-7189/03/$ - see front matter q 2003 Published by Elsevier Science Ltd.

doi:10.1016/S0149-7189(03)00027-2

Evaluation and Program Planning 26 (2003) 229–235

www.elsevier.com/locate/evalprogplan

* Corresponding author. Fax: þ509-935-6101.

E-mail address: [email protected] (R. L. Schalock).

and are due to a lack of: (1) a program evaluation model and

measurement methods that clearly delineate organization

and individual-referenced outcomes and that meet the dual

requirements of increased accountability and continuous

program improvement; (2) a program logic model that helps

program managers see the relationship among inputs,

processes, and outputs and the key roles played by formative

feedback and contextual variables in managing for results;

and (3) a mechanism to manage for results that includes

feedback to service providers, a quality improvement

process, and performance standards (Dewa, Horgan,

Russell, & Keats, 2001; McLaughlin & Jordan, 1999;

Rogers, 1995; Schalock, 2001). The present article is

written in light of these three challenges and contains

three sections. The first introduces the reader to a program

evaluation model based on the two dimensions of the reform

movement: accountability and quality; the second presents a

program logic model that is helpful in managing for results;

and the third presents an example of how the use of these

two models can lead to increased data utilization and

programmatic changes by providing feedback to program

managers that enhances their ability to manage for results.

Data presented in the article are based on the authors’ work

with the Ask Me!sm Project that is a consumer-based,

quality of life measurement and program change effort in

the developmental disabilities system of the State of

Maryland (Schalock, Bonham, & Marchand, 2000; Bonham

et al., 2003).

2. Measuring outcomes: a program evaluation model

Most evaluations of health and human service

programs use a combination of performance measurement

and value assessment. The increasing emphasis on value

assessment reflects heightened interest during the past

decade in consumer/client center health care and

rehabilitation services that try to individualize service

delivery and engage the consumer as an active decision

maker and participant in his/her treatment and rehabilita-

tion. This emphasis is consistent with a significant

change that has occurred during the past decade in

program evaluation: the use of both quantitative and

qualitative research methods. The Program Evaluation

Model shown in Fig. 1 reflects this change and has the

following three core dimensions: (1) an evaluation

standard of performance or value; (2) an evaluation

focus on the organization or the individual; and

(3) evaluation specificity related to outcomes and

measures. As shown in Fig. 1, the model also has: four

‘outcome indicator’ cells that identify the focus of

outcome based evaluation: program process, program

outputs, or organization/person-referenced outcomes; and

suggested measurement methods that are dependent on

the evaluation focus. Table 1 elaborates on the contents

of the model’s eight cells.

The evaluation model presented in Fig. 1 has five

important implications for measuring outcomes and

managing for results. First, it guides and clarifies the

evaluation process, consistent with the postmodernists’

emphasis on responsive, constructive evaluation (Guba &

Lincoln, 1989) and the development of participatory action

research strategies (Schalock, 2001). Second, it balances the

competing requirements of measuring valued, individual-

referenced outcomes with organization-referenced

outcomes. Third, all measurements and assessments are

Fig. 1. Program evaluation model.

Table 1

Key aspects of the program evaluation model (Fig. 1)

Organization performance (‘program process’)

Outcomes: health and safety, financial stability, staff development, and

organization efficiency

Preferred measurement methods: performance assessment measures that

include performance planning and reporting, licensure requirements, staff

certifications, performance indicators (such as critical performance

indicators and report cards), and financial accountability measures (such

as financial audit)

Organization value (‘organization outcomes’)

Outcomes: access to services, consumer satisfaction, staff competencies,

family/consumer supports, wrap-around services, and community support

Preferred measurement methods: consumer appraisal measures that include

consumer satisfaction surveys, measures reflecting fidelity to service

delivery model, benchmarks, and standards of excellence

Individual performance (‘program outputs’)

Outcomes: physical and mental health, functional status (activities of daily

living and instrumental activities of daily living), financial well-being,

residential status, and educational development

Preferred measurement methods: functional assessment measures that

include rating scales, observation, objective behavioral measures, and

status indicators (such as education, living, and employment)

Individual value (‘individual outcomes’)

Outcomes: self-determination, social inclusion, social relationships and

friendships, rights and personal dignity, and personal development

Preferred measurement methods: personal appraisal methods that include

quality of life evaluations, personal interviews, surveys, or focus groups

R.L. Schalock, G. S. Bonham / Evaluation and Program Planning 26 (2003) 229–235230

focused on agree upon ‘outcome indicators’ related to the

person or the organization, which facilitates the

development of organization and individual-referenced

benchmarks. Fourth, the model allows program evaluators

and program managers to meet the following objectives of

mixed-method evaluation strategies: (a) triangulation, or the

determination of correspondence of results among

the ‘outcome indicator’ cells (Cook, 1985); (b) comple-

mentarity, or the use of qualitative and quantitative methods

to measure the overlapping, but distinct, facets of the

outcomes (Greene, Caracelli, & Graham, 1989); and

(c) initiation, or the recasting of questions or results

(that is, if…then) from one strategy with questions or

results to a contrasting strategy (Caracelli & Greene, 1993).

Fifth, it reflects the value of measuring results and suggests

desired characteristics of good measures that include

clarity, credibility, balance, flexibility, and relevance

(Hakes, 2001).

The Program Evaluation Model summarized in Fig. 1

and Table 1 outlines clearly the types of outcomes on which

program managers should focus. However, it does not relate

to how these outcomes might be used to better manage for

results. Overcoming this challenge becomes clearer by

focusing on the program logic model discussed next.

The model shows clearly how program processes

and environmental context can affect individual and

organization-referenced outcomes.

3. Managing for results I: a program logic model

The Program Logic Model shown in Fig. 2 emphases

three aspects of managing for results: (1) it shows the

relationship among program inputs, processes, outputs, and

short and long-term outcomes; (2) it indicates that each of

these components is affected by the program’s

environmental context; and (3) it demonstrates the

important role that feedback regarding outputs and

outcomes play in programmatic processes. Thus, it views

a health or human service program as a ‘system’ that is

characterized by inputs, processes, and outputs (short and

long term-outcomes) that occur within an environmental

context and that is influenced by feedback. There are at least

five advantages of this model as an integral part of

measuring outcomes and managing for results.

1. The ‘input’ component allows program managers and

evaluators to focus on the predictors of desired outcomes,

rather than focusing exclusively on the outcomes per se.

This advantage reflects the significant shift in program

evaluation to the use of multi-variate research designs

that allow one to determine the factors that potentially

influence or cause obtained results. Once the significant

predictors are identified, program resources can be

directed at influencing these predictors with the clear

anticipation of improving desired outcomes.

2. The ‘process’ component allows program managers to

better align services and supports to the predictors of

desired outcomes. Alignment can occur along two

dimensions of a health or human service program:

vertical and horizontal (Labovitz & Rosansky, 1997).

Vertical alignment involves aligning the organization’s

strategy and its staff. It requires the rapid deployment of

the organization’s strategy, and transforming that

strategy into meaningful work and results-oriented

program services and supports. When vertical alignment

is reached, staff understand organization-wide goals and

their role in achieving them. In distinction, horizontal

alignment involves aligning agency processes, consumer

needs, and customer outcomes. It requires business

practices that cut across the different functions of the

organization to create what the consumers most value.

Thus, what staff focus on is determined by consumers,

including outcomes related to the four ‘outcome

indicator’ cells described in Table 1.

3. The distinction between short and long-term outcomes is

stressed. Short-term outcomes are the program effects

that occur shortly after the program outputs, such as

within 6 months or a year; whereas long-term outcomes

are expected to occur several years after the short-term

outcomes. Most health and human service programs

typically effect short-term outcomes, whereas program

managers are frequently held accountable for long-term

outcomes. The distinction between the two is essential in

the accountability dialog.

4. The feedback loop allows program managers to

understand better the use of outcome-oriented data and

the predictors of desired outcomes, rather than viewing

the outcomes only as a judgment on the success or failure

of a program and used to reward or punish.

5. The ‘environmental context’ of the model allows

program managers to understand and influence many of

the external (that is contextual) influences on outcomes.

Key contextual variables include respondent

characteristics, organization philosophy and goals,

organization services, phase of program development,

resources, formal linkages, community factors, and

family variables (Schalock, 2001).Fig. 2. Program logic model.

R.L. Schalock, G. S. Bonham / Evaluation and Program Planning 26 (2003) 229–235 231

4. Managing for results II: an example

The exemplary data sets and strategies discussed in this

section of the article are related to the Ask Me! Project

discussed next and the following aspects of the

Program Evaluation and Program Logic Models shown in

Figs. 1 and 2, respectively. The outcomes (Fig. 1) are

short-term individual outcomes (bottom right cell of Fig. 1)

related to emotional well-being, interpersonal relations,

physical well-being, rights, material well-being, and

personal development. These categories represent six of

the eight core quality of life domains discussed more fully in

Schalock and Verdugo (2002). The feedback (Fig. 2)

includes descriptive and comparative data provided to

service managers based on the Maryland Ask Me! Project.

Contextual variables that are discussed later in reference to

Fig. 3 are the significant predictors of personal development

from the most recent Project data analysis: proxy

respondent, agency transportation, hearing and visual

difficulty, and IQ grouping. A complete description of the

predictors and individual-referenced outcomes can be found

in Bonham et al. (2003).

4.1. Overview of the ask me! project

The Ask Me! Project sponsored by the Maryland

Developmental Disabilities Administration has over the

last 6 years developed and standardized an assessment tool

to measure quality of life that is responsive to the desires of

people with developmental disabilities to be heard and

understood. People with developmental disabilities

participate in all aspects of the Project, including being

the primary surveyors during the data collection phase.

People with disabilities are also key panelists when the

Project is presented and results discussed, including

discussions as to how the results can be used for

programmatic change and improvement. A more detailed

discussion of training and survey procedures can be found in

Schalock et al. (2000) and Bonham et al. (2003).

The Maryland Ask Me! Project includes a central quality

assurance training session at the beginning of each year for

all participating providers, and regional workshops during

the year. The training and workshops communicate five

topics: importance of the state places on the quality of life of

people it supports, background on quality of life concepts

and measurement, findings on the quality of life of

Marylanders with developmental disabilities, how to read

and understand the data agencies receive, and strategies to

use information in program planning and service

enhancement.

The Maryland Developmental Disabilities

Administration (DDA) sponsors the Maryland Ask Me!

Project which interviews 1000 randomly selected people

each year. DDA uses the data to develop its goals and

monitor their achievements as it manages for results,

a requirement of the governor and legislature for all state

agencies as part of the budgetary process. The Ask Me!

results allow DDA to move beyond the traditional licensing

approach to an approach of enhancing quality for all people

while maintaining a minimum threshold. Basic to

this approach is providing feedback to managers on:

(a) the predictors of personal development (which is the

initial quality of life domain that DDA is focusing on, given

the domain’s significant relationship to the other quality of

life domains and its consistency with DDA’s mission

statement); and (b) individual assessed quality of life scores

from the Ask Me! Quality of Life Survey.

4.2. Feedback to managers

Predictors of personal development. The Ask Me!

Project provides more than data to set DDA goals and

measure their achievement. The Ask Me! findings provide a

potential causal model on how quality of life can be

enhanced. The Ask Me! data presented to program

managers summarize the factors that statistically relate to

personal development and can be modeled in a manner such

as shown in Fig. 3 (although it is important to point out that

these are cross-sectional data collected at the same point in

time). The numbers on the arrows are path coefficients

(standardized multiple regression coefficients) and indicate

the statistically significant ðp , 0:01;N ¼ 923Þ strengths of

the relationship. As shown, the largest contributing factor to

personal development (largest number shown on an arrow)

is the interpersonal relationships people receiving services

have with other people. These other people include staff of

providers. Next in importance are rights and physical

well-being. Efforts to help provider staff develop inter-

personal relations that respect the rights of the people served

and enhance their physical well-being will also contribute to

personal development. Less important, but still statistically

significant, is the amount of transportation provided by

service agencies. The more frequent the transportation,

the greater is the person’s perception of their personal

development. People with higher cognitive functioning

express slightly higher levels of personal development than

those with lower cognitive functioning. However, these

personal characteristics are no more important thanFig. 3. Influences on personal development.

R.L. Schalock, G. S. Bonham / Evaluation and Program Planning 26 (2003) 229–235232

transportation, and much less important than interpersonal

relationships. The model also provides a cautionary note on

judging personal development. Proxies tend to report lower

levels of personal development than do people with similar

characteristics who respond for themselves. Since over half

of the proxies are staff, this suggests that people themselves

should be involved in every way possible in setting their

own goals for personal development and in measuring

their progress toward them.

Individual assessed quality of life scores. The 1000

people interviewed in the Ask Me! Project are clustered

within 35 providers, with about 30 people served by each of

the agencies included in the survey. All agencies serving 55

or more people are included within a 4 year cycle, along with

a sample of smaller agencies. The 30 people randomly

selected from provider agencies provide estimates of the

quality of life of people served by the agencies. The number

of state-funded interviews is too small to differentiate

between groups within an agency, but providers may fund

additional interviews themselves. The project provides each

participating provider with a chart showing how the average

quality of life of the people it serves compares to all people

supported by DDA, a printout of responses to the individual

question by both the people they serve and all those in

Maryland, and an electronic spreadsheet with survey

responses for each person, but unidentified to protect

confidentiality. The workshops (which are the first step in

the quality improvement process) help providers learn how

to read the data they receive and how other providers have

used their information.

4.3. Quality improvement process

The workshops suggest that providers first compare the

average quality of life reported by their consumers to that of

all consumers in the state, and hypothesize reasons why the

people they support have higher, lower or the same scores as

the state. For example, average quality of life scores for a

provider with a significant share of the state’s consumers is

expected. This may not satisfy such a provider that believes

it is a leader in quality services, so the second step is to

compare the Ask Me! findings with the provider’s goals.

If the quality of life data do not reflect the goals of the

provider, the provider should then ask how it might change

its services to best enhance its consumers’ quality of life.

Fig. 4 provides an illustration of how an agency might work

back from a desired long-term outcome to specific program

processes based upon the Program Logic Model shown in

Fig. 2.

As an example, the agency determines that it wants to

enhance the quality of life of the people they serve in the

domain of personal development. This is a long-term

outcome goal and the agency should not expect to see

immediate and dramatic changes, since outcomes regarding

this goal are influenced by factors such as past experiences,

the individual’s personal value system, and services

currently provided by other agencies. A more immediate

outcome for the agency is the satisfaction that individuals

have with their own personal development. Since consumer

satisfaction is primarily a value of the organization rather

than the people served (‘organization value’ outcome,

see Table 1), it might be measured along with or soon

after output goals have been measured. A key output

objective for enhancing personal development would be that

the consumers accomplish the goals that they have set for

themselves. Thus, they help consumers set goals during

their individual planning meetings, assign staff and other

resources in ways that will help the people progress towards

those goals, and monitor progress. Strategic planning works

from the desired long-term outcomes back to detail program

process. In distinction, effective goal setting during

individual planning, accompanied by the allocation of

resources (responsibility, training, authority, and support)

and careful monitoring of achievement, should lead to

goal-accomplishing outputs, which in turn should lead

to consumer satisfaction with their accomplishment of

specific goals, which should also lead to increased overall

satisfaction with their quality of life in the domain of

personal development.

4.4. Performance standards

Hakes (2001) suggests the following five strategies for

implementing measurement systems and change: coordinate

well with users; develop successful implementation

strategies within agencies; base systems for measuring

results on a broad commitment to customer service and civic

mission; commit to improved performance rather than a set

of rules that employees might fear will be used to evaluate

them and assign punishment; and establish reasonable

expectations about what results should be measured through

consultation with outside overseers. These suggestions are

consistent with three new initiatives in Maryland to

encourage performance-based assessments and desired

programmatic changes: (a) The Standards for Excellence

(Maryland Association of Nonprofit Organizations, 1998)

that outlines performance standards against which all

Fig. 4. Expected sequence of effects.

R.L. Schalock, G. S. Bonham / Evaluation and Program Planning 26 (2003) 229–235 233

nonprofit organizations should measure their progress;

(b) the new quality assurance plan developed by the

Developmental Disabilities Administration (1998) requires

service providers to develop quality assurance plans

including goals, submit annual reports, identify measures

for assessment, and evaluate if the goals have been

achieved; and (c) the Maryland State Government

management process called Managing for Results (Devel-

opmental Disabilities Administration, 2001).

DDA’s Managing for Results plan includes 16 specific

targets related to threshold standards and average score

standards for the eight quality of life domains measured.

The threshold standard is a positive quality of life score

(more positive than negative responses to the component

questions) and the targets to be maintained or exceeded are

the percent of people with positive scores for each domain at

the baseline survey. This threshold ensures that the people

whose quality of life is most problematic are not forgotten in

the pursuit of enhancing quality of life for all people served.

The focal target, however, is enhancing the quality of life

for all people, measured by an increase in the average

quality of life score. For its first year, DDA set its goal as

increasing the average score in the personal development

domain by 4%. Since personal development has strong

statistical relationship to the other seven quality of life

domains measured (that is, emotional well-being,

interpersonal relations, material well-being, physical well-

being, self determination, social inclusion, and rights),

an improvement in this focal domain will require, or result

in, enhancements in all the other domains. The goals for the

other seven domains are to maintain or increase the average

score. DDA does not require individual service providers to

use Ask Me! data as they develop their quality

assurance plan, but encourages its use to measure

achievement of goals.

5. Conclusion

In addition to overcoming those three difficulties in

measuring outcomes and managing for results discussed in

Section 1, the processes and strategies described in this

article have also allowed developmental disabilities services

in at least one state (Maryland) to overcome the four

significant challenges presented by the Government

Performance and Results Act of 1993 (GPRA; US General

Accounting Office (GAO), 1998): identifying goals,

developing performance measures, collecting data, and

analyzing data and reporting results. Additionally service

providers are beginning to see and report several advantages

to measuring quality of life outcomes and using those

outcomes to manage for results. Among the more important

advantages shared thus far are: enhancing public

accountability, improving internal accountability, focusing

on long term goals and strategic objectives, providing

performance information to key stakeholders, enhancing

decision making, allowing the organization to determine

effective resource use, and meeting better customers’

desires for an enhanced quality of life.

Measuring outcomes and managing for results is neither

easy nor can it be separated from other issues involving the

utilization of evaluation information and the implemen-

tation of organization change. The strategies described in

this article reflect a potentially effective approach to not

only selecting and measuring outcomes and providing

information to mangers to help them manage for results, but

the strategies also have the potential to impact evaluation

utilization and programmatic processes. For example,

a number of key elements of evaluation utilization are

evident in the Maryland Project including (Johnson, 1998;

Patton, 1997): stakeholder involvement, the use of

evaluation information as a marketplace of ideas and

information, a focus on the predictors of desired outcomes,

and the use of data to impact decision makers’ commitment

to change and managers’ performance. Similarly, a number

of key elements of organization change and continuous

improvement are also evident in the Project including

(Green & Newman, 1999; Hodges & Hernandez, 1999):

adoption of quality outcomes, responsiveness to consumer

needs, teamwork, and a focus on organization process.

Although the short term effects of the Project in regard to

measuring outcomes and managing for change are

promising, long term, systemic changes in information

utilization and programmatic processes are currently being

evaluated.

References

Albin-Dean, J. E., & Mank, D. M. (1997). Continuous improvement and

quality of life: lessons from organizational management. In R. L.

Schalock (Ed.), Quality of life: Volume II: Application to persons with

disabilities (pp. 165–180). Washington, DC: American Association on

Mental Retardation.

Bonham, G. S., Basehart, S., Schalock, R. L., Marchand, C. B., Kirchner,

N., & Rumenap, J. M. (2003). Consumer based quality of life

assessment: the maryland ask me! project. Mental Retardation, in press.

Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-

method evaluation designs. Educational Evaluation and Policy

Analysis, 15(2), 195–207.

Cook, T. D. (1985). Postpositivist critical mulipluralism. In R. L. Shortland,

& M. M. Marks (Eds.), Social science and social policy (pp. 21–62).

Beverly Hills, CA: Sage Publications.

Developmental Disabilities Administration (1998, August). COMAR,

10.22.02.14

Developmental Disabilities Administration (2001). Maryland DHMH/DDA

Budgets Fiscal Year, 2003

Dewa, C. S., Horgan, S., Russell, M., & Keates, J. (2001). What? Another

form? The process of measuring and comparing service utilization in a

community mental health program model. Evaluation and Program

Planning, 24, 239–247.

Drucker, P. F. (1998). Management’s new paradigms. Forbes, 152–176.

Gardner, J. F., & Nudler, S. (Eds.), (1999). Quality performance in human

services: Leadership, values and vision. Townson, MD: Brookes.

Green, R. S., & Newman, F. L. (1999). Total quality management

principles promote increased utilization of client outcome data in

R.L. Schalock, G. S. Bonham / Evaluation and Program Planning 26 (2003) 229–235234

behavioral health care. Evaluation and Program Planning, 22,

179–182.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a

conceptual framework for mixed-method evaluation designs. Edu-

cational Evaluation and Policy Analysis, 11, 255–274.

Guba, E. S., & Lincoln, Y. S. (1989). Fourth generation evaluation.

Newbury Park, CA: Sage Publications.

Hakes, J. E. (2001). Can measuring results produce results: one manager’s

views. Evaluation and Program Planning, 24, 319–327.

Hodges, S. P., & Hernandez, M. (1999). How organizational culture

influences outcome information utilization. Evaluation and Program

Planning, 22, 183–197.

Hoffman, F. L., Lechman, E., Russo, N., & Knauf, L. (1999). In it for the

long haul: the integration of outcomes assessment, clinical services, and

management decision-making. Evaluation and Program Planning, 22,

211–219.

Johnson, R. B. (1998). Toward a theoretical model of evaluation utilization.

Evaluation and Program Planning, 21, 93–110.

Labovitz, G., & Rosansky, V. (1997). The power of alignment: How great

companies stay centered and accomplish extraordinary things. New

York: Wiley.

Maryland Association of Nonprofit Organizations (1998). Standards for

excellence: An ethics and accountability code for the nonprofit sector.

Baltimore, MD: Maryland Association of Nonprofit Organizations.

Mawhood, C. (1997). Performance measurement in the United Kingdom

(1985–1995). In E. Chelimsky, & W. R. Shadish (Eds.), Evaluation for

the 21st century: A handbook (pp. 134–144). Thousand Oaks, CA: Sage

Publications.

McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: a tool for telling

your program’s performance story. Evaluation and Program Planning,

22, 65–72.

Newcomer, K. E. (1997). Using performance measurement to improve

public and nonprofit programs. San Francisco: Jossey-Bass Publi-

cations.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed). Thousand

Oaks, CA: Sage Publications.

Rogers, E. M. (1995). Diffusion of innovations (4th ed). New York: The

Free Press.

Schalock, R. L. (2001). Outcome-based evaluation (2nd ed). Dordecht/New

York: Kluwer/Academic/Plenum Publications.

Schalock, R. L., Bonham, G. S., & Marchand, C. B. (2000). Consumer

based quality of life assessment: a path model of perceived satisfaction.

Evaluation and Program Planning, 23(1), 77–88.

Schalock, R. L., & Verdugo, M. A. (2002). Handbook on quality of life

handbook for human service practitioners. Washington, DC: American

Association on Mental Retardation.

US General Accounting Office (GAO) (1998). Managing for results:

Measuring program results that are under limited control, (Report

GAO/GGD-99-16). Washington, DC: US General Accounting Office.

Wholey, J. S. (1997). Trends in performance measurement. In E.

Chelimsky, & W. R. Shaddish (Eds.), Evaluation for the 21st century:

A handbook. Thousand Oaks, CA: Sage Publications.

R.L. Schalock, G. S. Bonham / Evaluation and Program Planning 26 (2003) 229–235 235