4D2-9 - Performance Reports. see details below. Please follow all instructions given and answer the questions as given.

If program managers are expected to be involved in developing, implementing, and using performance measures (particularly, measures that focus on outcomes), the intended uses become important. There will inherently be more incentives for managers to take “ownership” of measures that are used formatively, as opposed to performance measures that will be seen to be indicators reflecting the program outcomes attributable to the organization to be used for summative purposes.

Summative uses of outcome results create situations where managers, knowing that they often do not control the environmental factors that can affect their program outcomes, perceive a dilemma. If they are expected to develop and use performance measures, they can be called to account if they fail to do so. But if they do develop and use performance measures, they could be held responsible for poor performance results that reflect factors that they cannot control. In many public organizations—NPM initiatives notwithstanding—procedural rules and restrictions, and political risk aversion, still mean that even if managers do see innovative ways of changing programs or program environments to increase the probability of success, they may have limited freedom to do so.

There is a related problem for organizations that deliver human service programs. In Chapter 2, we introduce the idea that programs vary in the a priori robustness of their intended cause-and-effect linkages. Programs that are based on engineering knowledge, such as a highway maintenance program, typically incorporate technologies that have a high probability of success. If the program is implemented as planned, it is highly likely that the intended outcomes will occur. At the other end of the continuum, there are many programs that operate with low-probability technologies. In general, these programs focus on human service issues and involve efforts to ameliorate or change human conditions, knowledge, attitudes, or behaviors. Even if these programs are fully implemented, we often observe mixes of program successes and failures. Our knowledge of what works tends to be far less certain than is true of high-probability technologies. There are usually important environmental variables that affect both program activities and outcome variables. In many situations, it is very difficult to mitigate the influences of these variables—low-probability technologies tend to be much more vulnerable to external influences.

For program managers who are involved in delivering programs that incorporate low- probability technologies, the best efforts of their organization may not succeed. For these kinds of programs, being accountable for outcome-based performance measures is daunting. Aside from the challenges of measuring intended outcomes, program managers will point out that outcomes are not really under their control.

An example might be a social welfare agency that has a program focused on single mothers. The objective of the program is to support these women in their efforts to obtain permanent jobs. The logic model for the program emphasizes training and child care support during the training. But the single mothers may have more needs than this program can meet. Some may need to deal with substance abuse problems or deal with former partners who continue to harass the family. Some may need to deal with psychological issues from their own family of origin. Single programs typically will not be able to address this range of issues, so to hold any one of them accountable for the program objective is not appropriate. Even if a comprehensive program could be designed and implemented, the state of the art of our knowledge of how to achieve the objective is likely to mean that there will be a lot of partial successes and failures in program outcomes. Holding managers to account for their efforts is appropriate, but holding them to account for the outcome may easily result in gaming behaviors.

The Levels of Analysis Problem: Conflating Organizational, Program, and Individual Performance

In our discussions of performance measurement, we have generally referred to situations where program performance is being measured. The logic modeling approaches discussed in Chapter 2, the research design considerations included in Chapter 3, and the measurement issues in Chapter 4 are focused on evaluating programs. However, performance measurement can occur at different levels within and between public and nonprofit organizations.

In principle, we can measure the performance of individuals, the performance of programs or administrative units, the performance of organizations, the performance of sectors (e.g., the health and social services sector of a government), and the performance of whole governments. In some performance measurement systems where public reporting has been mandated, there is a problem with the conceptual framework for performance measurement. Even in the scholarly literature, program and organizational performance are sometimes used interchangeably. Hatry (2006), for example, refers to program performance in his book but does not always distinguish between program and organizational performance.

Where government departments are expected to report on their performance, there appears to be an assumption that if they measure the performance of the departments, it can be assumed that program performance within the overall organization is also known. Similarly, if a department is meeting its performance targets, there can be a tendency to conclude that its programs must also be performing well. Although this assumption simplifies the task of measuring performance, it is not the case that one level of analysis is equivalent to the other. Knowing how ministries or departments are performing is not equivalent to knowing how their programs are performing, or vice versa.

This levels of analysis problem can be extended. Performance measurement can also focus on how well individuals are doing in an organization or a program. Again, it might seem reasonable to assume that if the people are performing well, then the organization or program is performing well. But as managers know, there is more to effective organizations than the programs and the people who deliver them. It is fallacious to assume that knowledge of performance at one level implies knowledge of performance at other levels. To fully measure performance, it will be necessary to measure it at each level in the organization (individual, program, department, organization as a whole) in order to obtain a credible picture of how well the organization is doing.

SUMMARY

Performance measurement and public reporting is now a central part of governmental efforts to demonstrate public accountability. The normative performance management cycle we introduced in Chapter 1 suggests that public performance reporting should have real consequences for the reporting organizations and that the prospect of these consequences will serve as a driver for performance improvements.

In Chapter 10, we have looked at the ways that performance information is used and whether public performance reporting does improve performance. The experience with public reporting is that where performance reporting is high stakes and is accompanied by independent organizations rating organizations—in England between 2000 and 2005 a three- star rating system was used—challenging the reputations of public organizations will improve performance.

But high-stakes performance measurement settings usually produce unintended side effects, and the key one is gaming of performance results. In the English ambulance service, gaming included modifying the reported response times for ambulances to meet the 8-minute target for emergency responses to calls. Managing gaming responses to performance targets requires investing in strategies such as audit functions, capable of checking data systems and performance results. Gaming is dynamic, that is, it evolves over time. Responses to gaming will reduce it but probably never eliminate it.

In most settings, public performance reporting is high stakes in the sense that negative political consequences can develop. The risk of that happening is situational. In adversarial political cultures, for example, where risk aversion is a factor in administrative and even policy decisions, public performance reports can become part of efforts to minimize risks, including making sure that they contain “good news” or at least performance results that are not negative.

In high-risk settings where public organizations collect their own performance data and prepare their own performance reports, performance information that is included in public reports may be decoupled from the performance information that is used for internal management purposes. Decoupling is a strategy for reducing risk; it offers performance results to meet nominal accountability expectations, and it protects managers from possible blowback from political controversy over a public performance result.

In low-risk settings (e.g., most local governments) it is easier to use performance results for both internal performance management and external accountability. Although there is limited research on this relationship, in one local government in Western Canada, managers and council members agreed that the performance information that was produced by managers was both useful and credible. At the same time, neither the managers nor the council members were substantially concerned about reporting negative performance results publicly.

There is little empirical evidence that politicians use ex post performance information in their roles and deliberations. A study that examined the ways that legislators use performance reports over time shows that expectations were high before the first reports were received, but actual uses were very modest and were mainly focused around general (and perhaps symbolic) accountability uses as well as information dissemination uses.

Although there has been widespread adoption of performance measurement and public reporting in many countries, there is growing evidence of a pulling back from high-stakes performance measurement systems. In 2005, the British government stopped the “naming and shaming” approach that was being used in England. In 2010, the U.S. government stopped using the high-stakes PART process for rating programs on their effectiveness.

Although there has been a pulling back from high-stakes performance measurement and public reporting, performance measurement is here to stay. It has become an expected part of public accountability and will continue to be used to produce public performance reports. Performance measurement has survived and evolved with different governmental reform initiatives, the latest one being NPM. Even if NPM fades or is replaced by another wave of public sector reforms, it is unlikely that performance measurement will fade.

For program evaluators, performance data can be a useful part of evaluations, but the gaming-related problems that can develop in high-stakes performance measurement and reporting settings suggest that evaluators have to be cautious about which data they rely on. Performance information that is developed for internal use is likely to be more valid and reliable than information produced for external, summative purposes.

DISCUSSION QUESTIONS

1. What are the key differences between the technical/rational view of implementing performance management in organizations and the political/cultural view?

2. Some commentators have suggested that failures of performance measurement systems to live up to their promises are due to poor or inadequate implementation. This view suggests that if organizations properly implement performance measurement, paying attention to what is really needed to get it right, performance measurement will be successful. Another view is that performance measurement itself is a flawed idea and that no amount of attention to implementation will solve its problems. What are your views on this issue?

3. Will auditing performance reports increase their usefulness? Why? 4. If you were making recommendations to the governor of your state about ways of

improving the performance measurement system, what recommendations would you make?

5. The record on uses of performance reports suggests that they are used by decision makers to some extent, but there is much room for improvement. What three things would you suggest to make performance reports more useful for elected decision makers?

6. What is the levels of analysis problem in performance measurement systems? 7. What does it mean for organizational managers to “game” performance measures? What

are some ways of reducing the occurrence of this problem? 8. What is the “ratchet effect” in setting targets for performance measures?

REFERENCES

Askim, J. (2007). How do politicians use performance information? An analysis of the Norwegian local government experience. International Review of Administrative Sciences, 73(3), 453–472.

Auditor General of British Columbia. (2008). Strengthening accountability in British Columbia: Trends and opportunities in performance reporting. Victoria, British Columbia, Canada: Queen’s Printer.

Auditor General of British Columbia & Deputy Ministers’ Council. (1996). Enhancing accountability for performance: A framework and an implementation plan—Second joint report. Victoria, British Columbia, Canada: Queen’s Printer.

Barrett, K., & Greene, R. (2008). Grading the states ’08: The mandate to measure. Governing, 21(6), 24–95.

Bevan, G., & Hamblin, R. (2009). Hitting and missing targets by ambulance services for emergency calls: Effects of different systems of performance measurement within the UK. Journal of the Royal Statistical Society. Series A (Statistics in Society), 172(1), 161–190.

Bevan, G., & Hood, C. (2006). Health policy: Have targets improved performance in the English NHS? BMJ, 332(7538), 419–422.

Bish, R. (1971). The public economy of metropolitan areas. Chicago, IL: Markham. Bouckaert, G., & Halligan, J. (2008). Managing performance: International comparisons. New

York: Routledge. CCAF-FCVI. (2002). Reporting principles: Taking public performance reporting to a new

level. Ottawa, Ontario, Canada: Author.

CCAF-FCVI. (2006). Users and uses: Towards producing and using better public performance reporting: Perspectives and solutions. Ottawa, Ontario, Canada: Author.

CCAF-FCVI. (2007a). CCAF’S improved public performance reporting program gets guidance from task force. Retrieved from http://www.ccaf-fcvi.com/english/updates/ PPRPTaskForce03–15–07.html

CCAF-FCVI. (2007b). How to improve public performance reports: Major topic at conference chaired by CCAF. Retrieved from http://www.ccaf-fcvi.com/english/updates/Victoria Conference02–15–07.html

Downs, A. (1965). An economic theory of democracy. New York: Harper & Row. Dunleavy, P., Margetts, H., Bastow, S., & Tinkler, J. (2006). New public management is dead:

Long live digital-era governance. Journal of Public Administration Research and Theory, 16(3), 467–494.

Epstein, P. D., Grifel, S. S., & Morgan, S. S. (2004). Auditor roles in government performance measurement: A guide to exemplary practices at the state, local and provincial levels. Apharetta, GA: Institute of Internal Auditors Research Foundation.

Feller, I. (2002). Performance measurement redux. American Journal of Evaluation, 23(4), 435 –452.

Frisco, V., & Stalebrink, O. (2008). Congressional use of the Program Assessment Rating Tool. Public Budgeting & Finance, 28(2), 1–19.

Gill, D. (Ed.). (2011). The iron cage recreated: The performance management of state organisations in New Zealand. Wellington, NZ: Institute of Policy Studies.

Gilmour, J., & Lewis, D. (2006). Does performance budgeting work? An examination of the Office of Management and Budget’s PART scores. Public Administration Review, 66(5), 742–752.

Government of BC and the Auditor General of BC. (2003). Reporting principles and an assurance program for BC: Progress report on the February 2002 recommendations of the Public Accounts Committee of British Columbia related to building better reports. Victoria, British Columbia, Canada: Author.

Government of British Columbia. (2000). Budget Transparency and Accountability Act: [SBC 2000 Chapter 23]. Victoria, British Columbia, Canada: Queen’s Printer.

Government of British Columbia. (2001). Budget Transparency and Accountability Act [SBC 2000 Chapter 23] (amended). Victoria, British Columbia, Canada: Queen’s Printer.

Government Performance and Results Act of 1993, Pub. L. No. 103-62. Government Performance and Results Act Modernization Act of 2010, Pub. L. No. 111-352. Hatry, H. P. (1974). Measuring the effectiveness of basic municipal services. Washington, DC:

Urban Institute and International City Management Association. Hatry, H. P. (1980). Performance measurement principles and techniques: An overview for

local government. Public Productivity Review, 4(4), 312–339. Hatry, H. P. (2006). Performance measurement: Getting results (2nd ed.). Washington, DC:

Urban Institute Press. Hibbard, J. (2008). What can we say about the impact of public reporting? Inconsistent

execution yields variable results Annals of Internal Medicine, 148, 160–161. Hibbard, J., Stockard, J., & Tusler, M. (2003). Does publicizing hospital performance stimulate

quality improvement efforts? Health Affairs, 22(2), 84–94.

Hildebrand, R., & McDavid, J. (2011). Joining public accountability and performance management: A case study of Lethbridge, Alberta. Canadian Public Administration, 54(1), 41–72.

Hood, C. (1991). A public management for all seasons? Public Administration, 69(1), 3–19. Hood, C. (2006). Gaming in targetworld: The targets approach to managing British public

services. Public Administration Review, 66(4), 515–521. Hood, C., Dixon, R., & Wilson, D. (2009). “Managing by numbers”: The way to make public

services better? Retrieved from http://www.christopherhood.net/pdfs/Managing_by_numbers.pdf

Kettl, D., & Kelman, S. (2007). Reflections on 21st century government management. Washington, DC: IBM Center for the Business of Government.

Klay, W. E., McCall, S. M., & Baybes, C. E. (2004). Should financial reporting by government encompass performance reporting? Origins and implications of the GFOA-GASB conflict. In A. Khan & W. B. Hildreth (Eds.), Financial management theory in the public sector (pp. 115–140). Westport, CT: Praeger.

Le Grand, J. (2010). Knights and knaves return: Public service motivation and the delivery of public services. International Public Management Journal, 13(1), 56–71.

Manitoba Office of the Auditor General. (2000). Business and performance measurement: Study of trends and leading practices. Winnipeg, Manitoba, Canada: Author.

Mayne, J., & Wilkins, P. (2005). “Believe it or not?” The emergence of performance information auditing. In R. Schwartz & J. Mayne (Eds.), Quality matters: Seeking confidence in evaluating, auditing, and performance reporting (pp. 237–259). New Brunswick, NJ: Transaction.

McDavid, J. C. (2001). Solid-waste contracting-out, competition, and bidding practices among Canadian local governments. Canadian Public Administration, 44(1), 1–25.

McDavid, J. C., & Huse, I. (2012). Legislator uses of public performance reports: Findings from a five-year study. American Journal of Evaluation, 33(1), 7–25.

McLean, I., Haubrich, D., & Gutierrez-Romer, R. (2007). The perils and pitfalls of performance measurement: The CPA regime for local authorities in England. Public Money and Management, 27(2), 111–118.

Melkers, J. (2006). On the road to improved performance: Changing organizational communication through performance management. Public Performance & Management Review, 30(1), 73–95.

Melkers, J., & Willoughby, K. (1998). The state of the states: Performance-based budgeting requirements in 47 out of 50 states. Public Administration Review, 58(1), 66–73.

Moynihan, D. P. (2008). The dynamics of performance management: Constructing information and reform. Washington, DC: Georgetown University Press.

Niskanen, W. A. (1971). Bureaucracy and representative government. New York: Aldine- Atherton.

Osborne, D., & Gaebler, T. (1992). Reinventing government: How the entrepreneurial spirit is transforming the public sector. Reading, MA: Addison-Wesley.

Otley, D. (2003). Management control and performance management: Whence and whither? British Accounting Review, 35(4), 309–326.

Pentland, B. T. (2000). Will auditors take over the world? Program, technique and the verification of everything. Accounting, Organizations and Society, 25(3), 307–312.

Perrin, B. (1998). Effective use and misuse of performance measurement. American Journal of Evaluation, 19(3), 367–379.

Poister, T. H. (2010). The future of strategic planning in the public sector: Linking strategic management and performance. Public Administration Review, 70(Suppl. S1), S246–S254.

Pollanen, R. M. (2005). Performance measurement in municipalities: Empirical evidence in Canadian context. International Journal of Public Sector Management, 18(1), 4–24.

Pollitt, C. (2011). 30 years of public management reforms: Has there been a pattern? (Background paper for the World Bank consultation exercise). Retrieved from http://siteresources.worldbank.org/EXTGOVANTICORR/ Resources/Politt.doc

Pollitt, C., Bal, R., Jerak-Zuiderent, S., Dowswell, G., & Harrison, S. (2010). Performance regimes in health care: Institutions, critical junctures and the logic of escalation in England and the Netherlands. Evaluation, 16(1), 13–29.

Power, M. K. (1997). The audit society: Rituals of verification. Oxford, UK: Oxford University Press.

Propper, C., & Wilson, D. (2003). The use and usefulness of performance measures in the public sector. Oxford Review of Economic Policy, 19(2), 250–267.

Savas, E. S. (1982). Privatizing the public sector: How to shrink government. Chatham, NJ: Chatham House.

Savas, E. S. (1987). Privatization: The key to better government. Chatham, NJ: Chatham House.

Schaffner, B. F., Streb, M., & Wright, G. (2001). Teams without uniforms: The nonpartisan ballot in state and local elections. Political Research Quarterly, 54(1), 7–30.

Schwartz, R., & Mayne, J. (Eds.). (2005). Quality matters: Seeking confidence in evaluating, auditing, and performance reporting. New Brunswick, NJ: Transaction.

Steele, G. (2005, April). Re-aligning resources and expectations: Getting legislators to do what they “should.” Paper presented at the 25th Anniversary Conference of CCAF-FCVI, Ottawa, Ontario, Canada.

Sterck, M. (2007). The impact of performance budgeting on the role of the legislature: A four- country study. International Review of Administrative Sciences, 73(2), 189–203.

Streib, G. D., & Poister, T. H. (1999). Assessing the validity, legitimacy, and functionality of performance measurement systems in municipal governments. American Review of Public Administration, 29(2), 107–123.

Thomas, P. G. (2006). Performance measurement, reporting, obstacles and accountability: Recent trends and future directions. Canberra, ACT, Australia: ANU E Press. Retrieved from http://epress.anu.edu.au/anzsog/performance/pdf/ performance-whole.pdf

Thomas, P. G. (2008). Why is performance-based accountability so popular in theory and so difficult in practice? In KPMG International (Ed.), Holy grail or achievable quest? International perspectives on public sector performance management (pp. 169–191). Toronto, Canada: KPMG International.

Treasury Board of Canada Secretariat. (2009). Policy on evaluation. Retrieved from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024

Wankhade, P. (2011). Performance measurement and the UK emergency ambulance service. International Journal of Public Sector Management, 24(5), 384–402.

Weber, M. (1930). The Protestant ethic and the spirit of capitalism. London, England: George Allen.

Wilkins, P., & Mayne, J. (2002). Providing assurance on performance reports: Two jurisdictions compared. Perth, Western Australia, Australia: Office of the Auditor General.

Williams, D. W. (2003). Measuring government in the early twentieth century. Public Administration Review, 63(6), 643–659.

Willoughby, K., & Benson, P. (2011). Program evaluation, performance budgeting and PART: The U.S. Federal Government experience. Atlanta: Georgia State University.