4D1-9 - Stakeholders and Creating Buy-In for Implementation. see details below. Please follow all instructions given and answer the questions as given.

determining whether the neighborhood watch program was the likely cause of the observed changes in the reported burglary rate.

But continuity can also make a system less relevant over time. Suppose, for example, that a performance measurement system was designed to pull data from several different databases, and the original information system programming to make this work was expensive. Even if the data needs change, there may well be a desire not to go back and repeat this work, simply because of the resources involved. Likewise, if a performance measurement system is based on a logic model that becomes outdated, then the measures will no longer fully reflect what the program(s) or the organization is trying to accomplish. But going back to redo the logic model (which can be a time-consuming, iterative process) may not be feasible in the short term, given the resources available. The price of such a decision might be a gradual reduction in the relevance of the system, which may not be readily detected.

With all the activity to design and implement performance measurement and reporting systems, there has been surprisingly little effort to date to evaluate their effectiveness (McDavid & Huse, 2012). In Chapter 10, we will discuss what is known now about the ways in which performance information is used, but it is appropriate here to suggest some practical steps to generate feedback that can be used to modify and better sustain performance measurement systems:

• Develop channels for user feedback. This step is intended to create a process that will allow the users to provide feedback and suggest ways to revise, review, and update the performance measures. Furthermore, this step is intended to help identify when corrections are required and how to address errors and misinterpretations of the data.

• Create an expert review panel of persons who are both knowledgeable about performance measurement and do not have a stake in the system that is being reviewed. Performance measurement should be conducted on an ongoing basis, and this expert panel review can provide feedback and address issues and problems over a long-term time frame. A review panel can also provide an independent assessment of buy-in and use of performance information by managers and staff, and track the (intended and unintended) effects of the system on the organization.

The credibility of performance information is an enduring concern. Davies and Warman (1998) point to the importance of auditing in the context of the performance reports of the (British) National Meteorological Office:

An independent audit, then, is not a luxury, it is a necessity. The credibility of the whole system of agencies is put at risk if the data from one is found to be unverified and open to dispute. Where performance-related bonuses are linked with outcomes, it is unreasonable to expect staff concerned to be responsible for the measurement and reporting of results in an objective manner when the very same results will determine their own pay. (p. 47)

Legislative auditors, in addition to recommending principles to guide public performance reporting, have been active in promoting audits of performance reports (CCAF-FCVI, 2002; Klay et al., 2004). Externally auditing the performance reporting process is suggested as an important part of ensuring the longer-term credibility of the system. With varying degrees of regularity and intensity, external audits of performance reports are occurring in some jurisdictions at the national, state, province, and/or local levels (Gill, 2011; Schwartz & Mayne, 2005). In Britain, for example, between 2003 and 2010, the National Audit Office (NAO)

conducted assessments of the performance measures that were integral to the Public Service Agreements between departments and the government. The NAO audits focused on performance data systems “to assess whether they are robust, and capable of providing reliable, valid information” (NAO, 2009).

PERFORMANCE MEASUREMENT FOR PUBLIC ACCOUNTABILITY

Performance results can be used for two general purposes: (1) to meet public accountability expectations and (2) to improve performance. Together, these purposes are often referred to as performance management.

Many jurisdictions have embraced results-focused performance measurement systems with the goal of improving public accountability (Dubnick, 2005). Performance measurement systems can be developed so that the primary emphasis, as they are implemented, is on setting public performance targets for each organization, measuring performance, and, in public reports, comparing actual results with targeted outcomes. Usually, performance reports are prepared at least annually and delivered to external stakeholders. In most jurisdictions, elected officials and the public are the primary recipients.

The logic that has tended to underlie these systems assumes that public performance reporting can also drive performance improvement, in that an approach that makes public accountability the principal goal gives organizations the incentive to become more efficient and effective (Auditor General of British Columbia, 1996). Performance improvements are expected to come about because elected officials and other stakeholders can put pressure, via public performance reports, on organizations to “deliver” results. Fully realized performance management systems are expected to include internal organizational performance incentives that are geared toward improving performance (Moynihan, 2008).

Figure 9.3 is a normative model of key intended relationships between performance measurement, public reporting, public accountability, and performance improvement. In the figure, public performance reporting is expected to contribute to both public accountability and performance improvement. Furthermore, performance improvement and public accountability are expected to reinforce each other.

The model in Figure 9.3 suggests expected relationships among performance measurement, public reporting, public accountability, and performance improvement that are implied in governmental reforms in many jurisdictions. As we have noted earlier, however, public performance reporting introduces a higher-stakes side of developing and implementing performance measurement systems. Once performance information is rendered in public reports, it can be used in ways that have consequences (intended and unintended) for both managers and elected officials. The literature on using performance information is rich with findings that suggest that the characteristics of the political culture in which government organizations are embedded can substantially influence both the quality and the uses of performance information (de Lancer Julnes, 2006; de Lancer Julnes & Holzer, 2001; Thomas, 2006).

Figure 9.3 A Normative Model of the Intended Relationship Between Public Accountability and Performance Improvement

We have suggested in this chapter that performance measurement systems, to be sustainable, need to be designed and implemented so that managerial use of performance information is the central purpose. In contrast to the relationships suggested in Figure 9.3, we believe that in many settings, particularly where the political culture is adversarial, public performance reporting may undermine the use of the performance information for performance improvement (McDavid & Huse, 2012). We will explore this problem in Chapter 10.

SUMMARY

The 12 criteria for designing and implementing performance measurement systems discussed in this chapter reflect both a technical/rational and a political/cultural view of organizations. Both perspectives are important in designing and implementing performance measurement systems that are sustainable. Collectively, the criteria impose some demanding requirements on the process. It is quite likely that in any given situation, one or more of these criteria will be difficult to address. Does that mean that, unless performance measurement systems are designed and implemented with these 12 criteria in view, the system will fail? No, but it is reasonable to assert that each criterion is important and does enhance the likelihood of success.

Our view is that for performance measurement systems to be sustainable, managerial involvement is key. In this chapter, we have developed an approach to performance measurement that emphasizes utilization of the information obtained for improving performance. Performance measurement for public accountability is one purpose of such systems, but making that the main purpose will tend to weaken managerial commitment to the system over time, and thus undermine the usefulness of the measures for improving efficiency and effectiveness.

Among the 12 criteria, six are more critical. Each contributes something necessary for successful design and implementation, and again these reflect a mix of technical/rational and organizational-political/cultural perspectives.

1. Sustained leadership: Without this, the process will drift and eventually halt. 2. Good communications: They are essential to developing a common understanding of the

process and increasing the likelihood of buy-in. 3. Clear expectations from the system: Be open and honest about the purposes behind the

process so that key stakeholders (managers and others) are not excluded or blindsided.

4. Resources sufficient to free up the time and expertise needed: When resources are taken away from other programs, to measure and report on performance, the process is viewed as a competitor to important organizational work and is often given short shrift.

5. Logic models that identify the key program and organizational constructs: The process of logic modeling is very important to informing the selection of constructs and the development of performance measures.

6. A measurement process that succeeds in producing valid measures in which stakeholders have confidence: Too few performance measurement systems pay attention to measurement validity and reliability criteria that ultimately determine the perceived usefulness of the system.

These six criteria can be thought of as individually necessary, but they will vary in importance in each situation. Performance measurement is a craft. In that respect, it is similar to program evaluation. There is considerable room for creativity and professional judgment as organizations address the challenges of measuring results.

DISCUSSION QUESTIONS

1. Assume that you are a consultant to the head of a government agency (1,000 employees) that delivers social service programs to families. The families have incomes below the poverty line, and most of them have one parent (often the mother) who is either working for relatively low wages or is on social assistance. The agency is under some pressure to develop performance measures as part of a broad government initiative to make service organizations more efficient and effective. In your role, you are expected to give advice to the department head that will guide the organization into the process of developing and implementing a performance measurement system. What advice would you give about getting the process started? What things should the department head do to increase the likelihood of success in implementing performance measures? How should he or she work with managers and staff to get them onside with this process? Try to be realistic in your advice—assume that there will not be significant new resources to develop and implement the performance measurement system.

2. Performance measurement systems are usually intended to improve the efficiency and effectiveness of programs or organizations (improve performance). But, very few organizations have taken the time to assess whether their performance measurement systems are actually making a difference. Suppose that the same organization that was referred to in Question 1 has implemented its performance measurement system. Assume it is three years later. The department head now wants to find out whether the system has actually improved the efficiency and effectiveness of the agency’s programs. Suppose that you are giving this person advice about how to design an evaluation project to assess whether the performance measurement system has “delivered.” Think of this as an opportunity to apply your program evaluation skills to finding out whether this performance measurement system was successfully implemented. What would be possible criteria for the success of the system? How would you set up research designs that would allow you to see whether the system had the intended incremental effects? What would you measure to see if the system has been effective? What comparisons would you build into the evaluation design?

APPENDIX A: Organizational Logic Models

Table 9A.1 Logic Model for Ministry of Human Resources, British Columbia, Canada

Note: FTE = Full-Time Equivalent; HRDC = Human Resources Development Canada.

a. Employee Performance Development Plans.

Figure 9A.1 Organizational Logic Model for Human Resources and Skills Development Canada

Source: Integrated Business Plan 2010–2013, Human Resources and Skills Development Canada, 2010. page 16, http://publications.gc.ca/collections/collection_2010 /rhdcc-hrsdc/HS1-11-2010-1-eng.pdf. Reproduced with the permission of the Minister of Public Works and Government Services Canada, 2012.

REFERENCES

Auditor General of British Columbia. (1996). 1996 Annual report: A review of the activities of the office. Victoria, British Columbia, Canada: Queen’s Printer.

Bakvis, H., & Juillet, L. (2004). The horizontal challenge: Line departments, central agencies and leadership. Ottawa, Ontario, Canada: Canada School of Public Service.

Bevan, G., & Hamblin, R. (2009). Hitting and missing targets by ambulance services for emergency calls: Effects of different systems of performance measurement within the UK. Journal of the Royal Statistical Society. Series A (Statistics in Society), 172(1), 161–190.

Bish, R., & McDavid, J. C. (1988). Program evaluation and contracting out government services. Canadian Journal of Program Evaluation, 3(1), 9–23.

Brimson, J. (1991). Activity accounting: An activity-based costing approach. New York: Wiley.

Campbell, D. (2002). Outcomes assessment and the paradox of nonprofit accountability. Nonprofit Management & Leadership, 12(3), 243–259.

CCAF-FCVI. (2002). Reporting principles: Taking public performance reporting to a new level. Ottawa, Ontario, Canada: Author.

Davies, M., & Warman, A. (1998). Auditing performance indicators: The meteorological office case study. Journal of Cost Management (January/February), 43–48.

Davies, R., & Dart, J. (2005). The “Most Significant Change” (MSC) technique: A guide to its use. Retrieved from http://www.mande.co.uk/docs/MSCGuide.pdf

de Lancer Julnes, P. (1999). Lessons learned about performance measurement. International Review of Public Administration, 4(2), 45–55.

de Lancer Julnes, P. (2006). Performance measurement: An effective tool for government accountability? The debate goes on. Evaluation, 12(2), 219–235.

de Lancer Julnes, P., & Holzer, M. (2001). Promoting the utilization of performance measures in public organizations: An empirical study of factors affecting adoption and implementation. Public Administration Review, 61(6), 693–708.

de Waal, A. A. (2003). Behavioral factors important for the successful implementation and use of performance management systems. Management Decision, 41(8), 688–697.

Dubnick, M. (2005). Accountability and the promise of performance: In search of the mechanisms. Public Performance & Management Review, 28(3), 376–417.

Gill, D. (Ed.). (2011). The iron cage recreated: The performance management of state organisations in New Zealand. Wellington, New Zealand: Institute of Policy Studies.

Goodwin, L. D. (1997). Changing conceptions of measurement validity. Journal of Nursing Education, 36(3), 102–107.

Government of Alberta. (1995). Government Accountability Act. Revised Statutes of Alberta 2000, Chapter G-7 (2009 compilation). Edmonton, Alberta: Alberta Queen’s Printer.

Government of Alberta. (2011). Government of Alberta 2010–11 annual report. Retrieved from http://www.finance.alberta.ca/publications/measuring/ ministry-annual-reports.html

Government of British Columbia. (2000). Budget Transparency and Accountability Act: [SBC 2000 Chapter 23]. Victoria, British Columbia: Queen’s Printer.

Government of British Columbia. (2001). Budget Transparency and Accountability Act [SBC 2000 Chapter 23] (amended). Victoria, British Columbia: Queen’s Printer.

Hildebrand, R., & McDavid, J. (2011). Joining public accountability and performance management: A case study of Lethbridge, Alberta. Canadian Public Administration, 54(1), 41–72.

Hood, C. (1991). A public management for all seasons? Public Administration, 69(1), 3–19. Hood, C. (2006). Gaming in targetworld: The targets approach to managing British public

services. Public Administration Review, 66(4), 515–521.

Human Resources and Skills Development Canada. (2010). Integrated business plan 2010 –2013, p. 16. Retrieved from http://publications.gc.ca/collections/collection_ 2010/rhdcc-hrsdc/HS1-11-2010-1-eng.pdf

Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action. Boston, MA: Harvard Business School Press.

Kates, J., Marconi, K., & Mannle, T. E., Jr. (2001). Developing a performance management system for a federal public health program: The Ryan White CARE ACT Titles I and II. Evaluation and Program Planning, 24(2), 145–155.

Klay, W. E., McCall, S. M., & Baybes, C. E. (2004). Should financial reporting by government encompass performance reporting? Origins and implications of the GFOA-GASB conflict. In A. Khan & W. B. Hildreth (Eds.), Financial management theory in the public sector (pp. 115–140). Westport, CT: Praeger.

Kravchuk, R. S., & Schack, R. W. (1996). Designing effective performance-measurement systems under the Government Performance and Results Act of 1993. Public Administration Review, 56(4), 348–358.

Levine, C. H., Rubin, I., & Wolohojian, G. G. (1981). The politics of retrenchment: How local governments manage fiscal stress (Vol. 130). Beverly Hills, CA: Sage.

Martin, L. L., & Kettner, P. M. (1996). Measuring the performance of human service programs. Thousand Oaks, CA: Sage.

Mayne, J. (2001). Addressing attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16(1), 1–24.

Mayne, J. (2008). Building an evaluative culture for effective evaluation and results management. Retrieved from http://www.cgiar-ilac.org/files/publications/ briefs/ILAC_Brief20_Evaluative_Culture.pdf

Mayne, J., & Rist, R. C. (2006). Studies are not enough: The necessary transformation of evaluation. Canadian Journal of Program Evaluation, 21(3), 93–120.

McDavid, J. C. (2001a). Program evaluation in British Columbia in a time of transition: 1995 –2000. Canadian Journal of Program Evaluation, 16(Special Issue), 3–28.

McDavid, J. C. (2001b). Solid-waste contracting-out, competition, and bidding practices among Canadian local governments. Canadian Public Administration, 44(1), 1–25.

McDavid, J. C., & Huse, I. (2006). Will evaluation prosper in the future? Canadian Journal of Program Evaluation, 21(3), 47–72.

McDavid, J. C., & Huse, I. (2012). Legislator uses of public performance reports: Findings from a five-year study. American Journal of Evaluation, 33(1), 7–25.

Morgan, G. (2006). Images of organization (Updated ed.). Thousand Oaks, CA: Sage. Moynihan, D. P. (2008). The dynamics of performance management: Constructing information

and reform. Washington, DC: Georgetown University Press. Moynihan, D. P., Pandey, S. K., & Wright, B. E. (2012). Setting the table: How

transformational leadership fosters performance information use. Journal of Public Administration Research and Theory, 22(1), 143–164.

National Audit Office. (2009). Performance frameworks and board reporting: A review by the performance measurement practice. Retrieved from http://www.nao.org.uk/guidance__good_practice/ performance_measurement1.aspx

Newcomer, K. E. (Ed.). (1997). Using performance measurement to improve public and nonprofit programs (New Directions for Evaluation, No. 75). San Francisco, CA: Jossey- Bass.

Norman, R. (2001). Letting and making managers manage: The effect of control systems on management action in New Zealand’s central government. International Public Management Journal, 4(1), 65–89.

Norman, R., & Gregory, R. (2003). Paradoxes and pendulum swings: Performance management in New Zealand’s public sector. Australian Journal of Public Administration, 62(4), 35–49.

Oregon Progress Board. (2003). Is Oregon making progress? The 2003 benchmark performance report. Salem, OR: Author.

Otley, D. (2003). Management control and performance management: Whence and whither? British Accounting Review, 35(4), 309–326.

Pollitt, C. (2007). Who are we, what are we doing, where are we going? Retrieved from http://www.koz-gazdasag.hu/images/stories/ 2per1/8-pollitt.pdf

Pollitt, C., Bal, R., Jerak-Zuiderent, S., Dowswell, G., & Harrison, S. (2010). Performance regimes in health care: Institutions, critical junctures and the logic of escalation in England and the Netherlands. Evaluation, 16(1), 13–29.

Prebble, R. (2010). With respect: Parliamentarians, officials, and judges too. Wellington, New Zealand: Victoria University of Wellington, Institute of Policy Studies.

Propper, C., & Wilson, D. (2003). The use and usefulness of performance measures in the public sector. Oxford Review of Economic Policy, 19(2), 250–267.

Schwartz, R., & Mayne, J. (Eds.). (2005). Quality matters: Seeking confidence in evaluating, auditing, and performance reporting. New Brunswick, NJ: Transaction Publishers.

Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization (1st ed.). New York: Doubleday/Currency.

Sigsgaard, P. (2002). MCS approach: Monitoring without indicators. Evaluation Journal of Australasia, 2(1), 8–15.

Staudohar, P. D. (1975). An experiment in increasing productivity of police service employees. Public Administration Review, 35(5), 518.

Texas State Auditor’s Office. (2012). SAO reports: Audits of performance measures. Retrieved from http://www.sao.state.tx.us/Reports/reportpost. cfm/perfmeas/yes

Thomas, P. G. (2004). Performance measurement, reporting and accountability: Recent trends and future directions (SIPP Public Policy Paper Series No. 23). Retrieved from http://www.uregina.ca/sipp/documents/pdf/ PPP23_P%20Thomas.pdf

Thomas, P. G. (2006). Performance measurement, reporting, obstacles and accountability: Recent trends and future directions. Canberra, ACT, Australia: ANU E Press. Retrieved from http://epress.anu.edu.au/anzsog/performance/pdf/ performance-whole.pdf

Thor, C. G. (2000). The evolution of performance measurement in government. Journal of Cost Management, May/June, 18–26.

Treasury Board of Canada Secretariat. (2012a). Policy on evaluation. Retrieved from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024

Treasury Board of Canada Secretariat. (2012b). Policy on management, resources and results structures. Retrieved from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?evttoo= X&id=18218&section=text

Wildavsky, A. B. (1979). Speaking truth to power: The art and craft of policy analysis. Boston, MA: Little Brown.

Williams, D. W. (2003). Measuring government in the early twentieth century. Public Administration Review, 63(6), 643–659.

Wilson, J. Q. (1989). Bureaucracy: What government agencies do and why they do it. New York: Basic Books.

WorkSafeBC. (2011). 2010 annual report and 2011–2013 service plan. Retrieved from http://www.worksafebc.com/publications/reports/ annual_reports/assets/pdf/2010/AnnualReport.pdf