U8D1-28 - Needs-Based Action Learning and Change Management Planning - Please follow all instructions as written to complete this homework. See Details Below.
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 1/10
Chapter 10 Performance Measurement, Monitoring, And Program Evaluation
Chapter Overview The purpose of this chapter is to explain:
Performance measurement, monitoring, and program evaluation The different types of data required to perform these functions The ways in which data are used
The following topics are covered in this chapter:
The Link Between Performance Measurement, Monitoring, and Evaluation and Information Systems Feedback and Self-Evaluating Systems Performance Measurement Monitoring Program Evaluation Program Data Requirements
Coverage Data Equity Data Process Data Effort (Output) Data Cost-Efficiency Data Results (Outcome) Data Cost-Effectiveness Data Impact Data
Performance Measurement, Monitoring, Program Evaluation, and Program Data The Utility of Performance Measurement, Monitoring, and Program Evaluation Summary Case Example Review Questions
The Link Between Performance Measurement, Monitoring, and Evaluation and Information Systems Although the topics of performance measurement, monitoring, and program evaluation appear toward the end of this book, they are an integral part of the entire process of effectiveness-based program planning. The material presented in the preceding chapters (including the discussion of needs assessment, planning, program design, and the establishment of goals and objectives) can all be seen as laying the groundwork for a discussion of performance measurement, monitoring, and program evaluation. Through the use of these assessment tools, human service agencies can self-monitor their programs as well as address the performance measurement and evaluation requirements mandated by funding sources today.
A strong link needs to exist between a human service agency’s performance measurement, monitoring, and program evaluation systems and its information system. The nexus here involves specifying up front the types of data to be collected in order to have available the necessary information to conduct performance measurement, monitoring, and program evaluation (Hatry, 2014).
In the case of monitoring in particular, it is necessary for a human service agency to have current, even real-time, data in order to adequately carry out this critical function. Historically, many human service agencies waited
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 2/10
until a program was in existence for some time before thinking about performance measurement, monitoring, and program evaluation. This approach virtually ensured that the necessary data and information would not be available. Attempting to reconstruct performance measurement, monitoring, and program evaluation data after the fact is not an optimum strategy.
Feedback and Self-Evaluating Systems Throughout this book, we have used the systems framework as a model for thinking about effectiveness-based program planning. We return to it once more to assist us in thinking about performance measurement, monitoring, and program evaluation. Self-learning systems, sometimes referred to as heuristic systems, learn by doing. Self-learning systems take data and information (feedback) about their operations, analyze it, and then use it to make adjustments in the way the program operates. For example, a thermostat is a self-learning system. A thermostat monitors the temperature in a room and adjusts the air conditioning as needed to maintain a comfortable environment. If we think of a human service program as a self-learning system (see Figure 10.1), performance measurement, monitoring, and program evaluation data provide feedback about its operations and functioning. This feedback becomes new input that can lead to changes, refinements, and improvements in how the program is designed and implemented. As Figure 10.1 illustrates, feedback is provided on the outputs (including quality outputs) and outcomes of a program. It is this feedback that is used for performance measurement, monitoring, and program evaluation purposes.
Figure 10.1 The Systems Framework, Feedback, and Self-Learning Systems
Performance Measurement Performance measurement can be defined as the ongoing reporting of program accomplishments, specifically the outputs (efficiency), quality, and outcomes (effectiveness) of programs (Martin & Kettner, 2010; McDavid, Huse, & Hawthorn, 2013). As Table 10.1 indicates, the primary purpose of performance measurement is external reporting. In terms of perspective, performance measurement has both a financial and a managerial orientation, with an equal emphasis placed on both, owing to its roots in accounting and performance auditing. The central idea behind performance measurement is to have government agencies and programs, including human service agencies and programs, collect and report to stakeholders (e.g., funding sources, clients, elected officials, advocacy groups, citizens) data and information on their performance.
Performance measurement seeks to make government-funded programs more transparent to stakeholders. Through transparency, stakeholders become more knowledgeable about how government-funded programs operate and how they help to improve the lives of citizens and clients. This type of information is helpful in maintaining and increasing stakeholder support for human service programs.
Performance measurement is concerned with providing answers to such frequent stakeholder questions as the following: How much product or service (outputs) does the program provide? What is the quality (quality outputs) of the product or service provided by the program? How many clients completed treatment (service completions)? What outcomes (intermediate and final) does the program achieve? How cost-efficient and cost- effective is the program?
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 3/10
Performance measurement has become institutionalized at the federal government level through the provisions of the Government Performance and Results Act (GPRA) of 1993 (Public Law 103-62). GPRA requires all federal departments and agencies to annually collect and report performance measurement data and information about their various programs (Kamensky, 2011). The federal government’s commitment to performance measurement was reaffirmed with passage of the GPRA Modernization Act of 2010 (Public Law 111-352).
The Governmental Accounting Standards Board (GASB), the organization that establishes what are called “generally accepted accounting principles” for state and local governments, encourages these same governments to also report performance measurement data. GASB (n.d.) believes that state and local governments should report to stakeholders the same types of performance data and information that businesses report to stockholders. In order for governments (federal, state, and local) to collect and report performance measurement data and information, they necessarily have to involve their grantees and contractors.
Not to be overlooked are the efforts of the United Way of America and some private foundations (e.g., Bill & Melinda Gates Foundation, Robert Wood Johnson Foundation) that for some time now have advocated, and in some cases required, that their grantees collect and report performance measurement data (Hatry, 2011). Given the influence of GPRA, GASB’s service efforts and accomplishments (SEA) reporting, the United Way and foundations, most nonprofit human service agencies today are involved to some extent with performance measurement.
It should be noted that performance measurement is concerned only with the collection and reporting of performance data and information. Performance measurement does not concern itself with questions such as these: Was the program implemented as planned? Did the program achieve its intended results? What was the program’s impact? For feedback that addresses these types of questions, human service agencies must turn to monitoring and program evaluation.
Monitoring Monitoring can be defined as an assessment of the extent to which a program is implemented as designed and serves its intended target group (Rossi, Lipsey, & Freeman, 2004). Some program evaluation texts refer to monitoring as formative evaluation because it takes place during program implementation (McDavid et al., 2013). The word monitoring comes from the Latin word monere, meaning “to warn.” Monitoring provides feedback to warn a human service administrator when the implementation of a program starts to deviate from its original design. The administrator can then take corrective action to bring the program back into line with its program design. The primary purpose of monitoring (see Table 10.1) is program management. Consequently, monitoring has a managerial perspective. Monitoring provides a human service administrator with feedback on the current status of a program in terms of several questions: Is the program being implemented as designed? What proportion of the community need is the program meeting? Are only eligible target-group clients being served? Are subgeographical areas and subgroups (e.g., ethnic minorities, women, persons with disabilities) being served in appropriate numbers? What products and services are being provided and in what amounts? What results are being achieved in terms of outputs and outcomes? In the assessment of program outputs (including quality outputs) and outcomes, monitoring and performance measurement overlap somewhat. Performance measurement and monitoring are both concerned with outputs, quality, and outcomes as well as issues of cost-efficiency and cost-effectiveness. The difference between the two is that performance measurement is concerned with the reporting of feedback data and information to external stakeholders, whereas monitoring is concerned with using the feedback data and information to internally track the implementation of programs and to make changes and refinements as needed.
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 4/10
Monitoring also lays the groundwork for program evaluation by helping to ensure that a program is implemented as intended. No useful purpose is served in conducting an evaluation if a program is not implemented as intended.
Program Evaluation Program evaluation can be defined as social science techniques applied to determining the workings and effectiveness of human programs (Rossi et al., 2004). Program evaluation is frequently concerned with determining the impact of a program. Impact assessment necessitates establishing cause-and-effect relationships. The program (the cause), and no other external factors, accounts for the achievement of the program outcomes (the effect). Because cause-and-effect relationships are difficult to establish, the assessment of program impact usually involves the use of social science research techniques and the application of statistics (Rossi et al., 2004).
The primary purpose of program evaluation (see Table 10.1) is to provide feedback on results, accomplishments, or impacts (outcomes) and to inform policymakers and planners about the effectiveness of programs and the appropriateness of the social intervention hypotheses that underlie them. Here are some questions addressed by program evaluation: Did the program work as intended? What results or accomplishments (outcomes) did the program accomplish? What measurable impacts (outcomes) did the program achieve? Is the program cost- effective? Program evaluation can also lead to the discovery of positive as well as negative unintended consequences, thereby suggesting needed changes and refinements in social policy or program design.
In terms of the assessment of results, accomplishments, or impacts (outcomes), program evaluation overlaps somewhat with both performance measurement and monitoring. The difference is that program evaluation is not concerned with the reporting of outcome data and information to external stakeholders (performance measurement) or with tracking a program’s implementation (monitoring) but, rather, is concerned with using program outcomes feedback to improve policy and planning. Program evaluation also overlaps with monitoring. The difference is that monitoring assesses a program during implementation, whereas program evaluation can take place either during implementation (formative evaluation) or after the fact (summative evaluation).
Program Data Requirements At this point, the reader has probably already concluded that performance measurement, monitoring, and program evaluation are all important and necessary components of what might be called program assessment. Through the use of information systems and dashboards, human service agencies need to ensure only that they are collecting the necessary data and they should be able to report performance measurement data and information to external stakeholders, perform ongoing program monitoring, and conduct program evaluations.
The types of program data that human service agencies need to collect to satisfy the requirements of performance measurement, monitoring, and program evaluation are coverage, equity, process, effort, cost- efficiency, results, cost-effectiveness, and impact. These eight types do not constitute the entire universe of program data, but they are the ones that performance measurement, monitoring, and program evaluation are
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 5/10
generally most concerned with. Each of these eight data types is discussed below, including the types of assessment questions the data are used to address (see Table 10.2).
Coverage Data
Coverage data provide feedback on the extent to which a program is (a) meeting the community need and (b) reaching its target population. Monitored during program implementation, coverage data can be used not only to determine the extent to which the target group is being reached but also to ensure that individuals ineligible for the program are not served. Conducted at the end of a program, or at some defined end point (e.g., the end of the fiscal year), coverage data can be used in a program evaluation mode to document that only eligible clients were served and to assess the adequacy, or inadequacy, of current program funding and service levels to meet the community need.
Equity Data
Equity data provide feedback on the extent to which specific subgeographical areas of a community as well as specific subgroups (such as ethnic minorities, women, persons with disabilities, children, or the elderly) are being served by a program. Unless a program is targeted at a specific subgeographical area or specific subgroup of a community, all other things being equal, geographical subareas and subgroups should be served by a program in roughly the same proportion as their composition in the community. Equity data can be used to ensure adequate coverage of subgeographical areas and subgroups during implementation (monitoring) or at the end of a program (program evaluation) to document that a program is or is not reaching some geographical subarea or subgroup. Utilized in a performance measurement approach, coverage data provide stakeholders with information about the distribution of outputs, quality outputs, and outcomes across subgeographical areas and subgroups.
Process Data
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 6/10
Process data provide feedback (monitoring) on the extent to which a program is implemented as designed. During implementation, process data and information can be used to compare actual service delivery to planned service design to ensure conformance with such items as service definitions, service tasks, service standards, work statements, service specifications, and other service requirements. At the end of a program, process data can be used to determine and document (program evaluation) that a program was implemented as intended.
Process data can also be used to monitor the extent to which human service programs are implemented in a culturally competent fashion and that the customs and needs of the persons being served are taken into consideration in the service definitions, service tasks, service standards, work statements service specifications, and other service requirements.
Effort (Output) Data
Effort (output) data provide feedback on the amount of products and services (intermediate outputs) provided, the amount of quality products and services (quality outputs) provided, and the number of service completions (final outputs) achieved. Effort (output) data can be monitored during implementation to compare actual effort to planned effort and to take corrective action when actual effort fails to coincide with planned effort. Effort (output) data can be used at the end of a program to document service delivery levels (program evaluation) and for purposes of performance measurement reporting. Effort (output) data are also used in determining a program’s cost-efficiency. Effort (output) data are one of the three major types of performance measurement information that would routinely be reported to stakeholders.
Cost-Efficiency Data
Cost-efficiency data provide feedback on the costs of providing program products and services, including intermediate outputs, quality outputs, and final outputs (service completions). Cost-efficiency data are developed by computing costs per output: intermediate, quality, and final. During implementation, actual cost data can be compared with planned costs (monitoring). At the end of a program year, cost-efficiency data can be used to assess a program’s productivity and are also required for purposes of performance measurement reporting.
Results (Outcome) Data
Results (outcome) data provide feedback on the extent to which a program achieves its intended results (outcomes), both intermediate and final. Results (outcome) data can be monitored during implementation to compare actual results achieved with planned results. Used at the end of a program year, results (outcome) data document for policy and planning purposes the results achieved by a program. Results (outcome) data are also required for purposes of performance measurement reporting. Finally, results (outcome) data are used in determining a program’s cost-effectiveness.
Cost-Effectiveness Data
Cost-effectiveness data provide feedback on the costs of achieving program results (outcomes), both intermediate and final. Cost-effectiveness data are developed by computing cost per intermediate outcome and cost per final outcome. Cost-effectiveness data are usually available only at the end of the program year (program evaluation) and are used to document the costs of achieving results (outcomes) for policy and planning purposes and for purposes of performance measurement reporting.
Impact Data
Impact data provide feedback on the most difficult assessment question of all: What happened to clients as a result of participation in a program that would not have happened in the program’s absence? To address this question, impact data are usually generated using social science research techniques, including the creation of a
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 7/10
control group for comparison purposes and the use of statistics to measure the magnitude of the impact. The development of impact data is a difficult undertaking. Nevertheless, limitations on program and agency resources make it imperative that available resources be put to the best use possible. Impact data can provide human service administrators with hard social science-based information that demonstrates the extent to which a program achieves measurable impacts with its clients.
Performance Measurement, Monitoring, Program Evaluation, and Program Data As illustrated in Table 10.3, performance measurement, monitoring, and program evaluation overlap in terms of their use of program data. In particular, considerable program data overlap exists between monitoring and program evaluation, and performance measurement overlaps somewhat with both monitoring and program evaluation.
The overlap in terms of types of program data leads to an important observation: To a great extent, the differences between performance measurement, monitoring, and program evaluation are not so much differences in the types of program data used as differences in when and how program data are used. Performance measurement uses program data for external reporting purposes. Monitoring uses program data during implementation for management purposes. Program evaluation uses program data after implementation for policy and planning purposes.
The Utility of Performance Measurement, Monitoring, and Program Evaluation The test of any feedback system is the usefulness of the resulting data and information. As we repeatedly stressed in earlier chapters, the critical questions to effectiveness-based program planning are these:
What types of clients? With what types of problems? Receive what types of services? With what results? At what cost?
The program feedback data and information produced by the answers to the above questions are useful for a number of purposes.
First, and most important, program feedback data provide information needed by human service administrators and front-line managers. These are the individuals most immediately responsible for implementing programs, and it is critical that they have current data and information on which to base decisions.
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 8/10
Second, program feedback data satisfy the reporting requirements of human service funding agencies, including federal, state, and local governments; the United Way; and foundations.
Third, program feedback data meet the needs of external stakeholders. The political environment of the human services today is often turbulent, irrational, demanding, and frustrating. Every human service administrator has experienced the helpless feeling of not being able to respond to questions about why his or her agency or program is, or is not, doing something that a particular politician, reporter, or interest group favors or opposes. Program assessment data and information represent tools that can be used by human service administrators to respond to inquiries about their agencies and programs. Data and information, of course, do not guarantee that all community and political interests will be satisfied, but at least they provide a sound, data-based response. Assessment data and information can also be used to sell a program and to enlarge its support base in a community.
Summary Performance measurement, monitoring, and program evaluation are similar in that they all use data and information to assess human service programs. The data and information generated by performance measurement, monitoring, and program evaluation constitute feedback. How this feedback is utilized differentiates principal purposes of performance measurement, monitoring, and program evaluation. Performance measurement is principally concerned with using feedback on program efficiency (output), quality, and effectiveness (outcome) for external reporting purposes. Monitoring is principally concerned with using feedback during program implementation for managerial purposes, including ensuring that programs are delivered as intended and in assessing program efficiency (output), quality, effectiveness (outcome), cost- efficiency, and cost-effectiveness. Program evaluation is principally concerned with using feedback for policy and planning purposes, including the assessment of program results, accomplishment, and impacts (outcomes).
Case Example Back in the late 1970s, the then–Title VII of the Older Americans Act (OAA) provided funding for meals programs for the elderly. The theory behind the meals programs was that if you improved the nutrition of older Americans by providing them with at least one hot meal 5 days per week, then you would improve their overall health. Program monitoring data collected during the first few years of the program identified some major implementation problems that were having adverse effects for one particular target group: older Native Americans living on reservations. The Title VII meals programs were not achieving their hoped-for performance because the OAA regulations were not culturally sensitive to the needs of older Native Americans living on reservations.
First, the OAA regulations stated that only persons over the age of 60 could participate. In those days, the average age at mortality for Native Americans living on reservations was about 45.
Second, the OAA regulations stated that home-delivered meals could be provided only as a subcomponent of a congregate meals program. On many reservations, particularly those in western states, the vast geographical distances between the home sites of elderly Native Americans made congregate meals programs difficult, if not impossible, to establish.
The combined effect of these two OAA regulations was that meals programs on Native American reservations were not achieving their hoped-for performance (output, quality, and outcome). Through the monitoring and analysis of performance data and information, the federal government eventually realized that changes had to be made to make meals programs more culturally relevant for Native Americans living on reservations. The OAA regulations were eventually changed to allow tribes and nations to determine for themselves who constituted an older Native American and the best approach to operating meals programs (congregate, home delivered, or a combination) on Native American reservations.
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 9/10
Review Questions 1. How might the OAA meals programs operating on Native American reservations have used the principles
of performance measurement, monitoring, and program evaluation to avoid the lack of fit between the needs of older Native Americans and program provisions?
2. Which of the three approaches (performance measurement, monitoring, or program evaluation) would likely be most useful in identifying the problems with the ages of Native Americans and the requirements of the OAA meals programs?
3. How might the principles of self-learning systems have helped to identify the problems with implementing the OAA regulations?
4. What types of data from Table 10.2 would have been most helpful in addressing the problems experienced by the Native American reservation meals programs?
5. Using the Native American meals program as an example, see if you can answer the critical questions about effectiveness-based program planning.
What types of clients?
With what types of problems?
Receive what types of services?
With what results?
At what costs?
References Governmental Accounting Standards Board. (n.d.). Service efforts and accomplishments reporting. Retrieved
from http://gasb.org/cs/ContentServer? c=GASBContent_C&pagename=GASB%2FGASBContent_C%2FProjectPage&cid=1176156646053
Hatry, H. (2011). Outcome measurement in the human services: Lessons learned from public and private sectors. In J. Magnabosco & R. Mandercheid (Eds.), Outcomes measurement in the human services (pp. 17–29). Washington, DC: NASW Press.
Hatry, H. (2014). Transforming performance measurement for the 21st century. Washington, DC: Urban Institute.
Kamensky, J. M. (2011). GPRA Modernization Act of 2010 explained: Part 1. Retrieved from http://www.businessofgovernment.org/blog/business-government/gpra-modernization-act-2010-explained- part-1
Martin, L., & Kettner, P. (2010). Measuring the performance of human service programs (2nd ed.). Thousand Oaks, CA: Sage.
McDavid, J., Huse, I., & Hawthorn, L. (2013). Program evaluation and performance measurement: An introduction to practice (2nd ed.). Thousand Oaks, CA: Sage.
6/2/2018 Bookshelf Online: Designing and Managing Programs
https://online.vitalsource.com/#/books/9781483388328/cfi/6/52!/4/2/4/2@0:0 10/10
Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.