DB.

Network governance and performance measures: Challenges in collaborative design of hybridized environments

Henrik P. Minassians*

Urban Studies and Planning, California State University Northridge, 18111 Nordhoff Street, Northridge, CA, USA

(Received 30 November 2014; revised 8 April 2015; accepted 4 May 2015)

Measuring performance of public goals can be highly complicated considering that, in the achievement of those goals, governments are no longer directly involved in the delivery of goods and services. Hybridization of public functions, purposes and processes has created new challenges by fragmenting public bureaucratic functions into the hands of multiple players, thus complicating measuring and reporting per- formance outcomes. This article presents the results of a study that examined perfor- mance indicators across multiple public agencies in a large municipal county. The purpose was to assess whether performance measures link organizational collective goals. It specifically addresses the following question: How does the governance of multi-actor networks influence the design of performance measures? The study emphasizes a need for the design of interlinked performance measures in a large net- worked environment. The findings indicate that design of performance measures remains insular to the organization and does not reflect the hybridized collaboration between various entities.

Keywords: performance measures; complex hybrid networks; collaboration

Introduction

Democratic governments are designed to implement the collective preferences of their citi- zens. In so doing, citizens expect that government officials utilize taxes effectively to achieve common public goals. The promise of the Reinvention Movement in the US and New Public Management in Europe has included achieving savings in public expenditures, creating more efficient governmental operations, and improving the quality of services to ultimately achieve effectiveness in delivery of these services. In order to meet the expecta- tions of their constituents, public agencies in the United States have focused on measuring government results through the design of performance-measurement initiatives.

Utilizing performance measures to evaluate the results of government action has become ubiquitous in public and non-profit organizations. There is a wide range of pos- sible reasons for the increase of performance measures in the public sector such as lim- ited fiscal resources, citizen demands, and, ultimately, limitations that fragmented democratic systems create with the oversight of public bureaucracies. Clear justification of programs enabled by valid assessment through effective performance measurement can become a public sector management tool that ensures continued funding of particu- lar programs.

*Email: [email protected]

© 2015 The Korean Association for Public Administration

International Review of Public Administration, 2015 Vol. 20, No. 4, 335–352, http://dx.doi.org/10.1080/12294659.2015.1088689

However, measuring performance of public goals can be highly complicated consid- ering that, in the achievement of those goals, governments are no longer directly involved in the delivery of goods and services. Instead, public agencies rely heavily on other public agencies and/or private and non-profit actors to achieve public goals. Hybridization of public functions, purposes and processes has created new challenges by fragmenting public bureaucratic functions into the hands of multiple players. In fact, many public managers work in multisectoral networks with unique relationships across different organizations (Herranz, 2010). Moreover, measuring and reporting perfor- mance becomes complicated when a number of mandates are imposed by external stakeholders–in this case, federal and state governments. Local public agencies work in a continually turbulent environment in which they are regularly subject to state and fed- eral governments shifting priorities, funding, and goals. This study is designed to increase understanding of complexities with the design of multisectoral, multiorganiza- tional and multipurpose performance indicators and their role in a networked setting. Through this strategy, the study provides better strategic guidance to public managers, the literature on performance measures, and the role of management when it comes to multilevel performance of multisectoral networks.

In this context, this article reports the findings of a case study of Los Angeles County Department of Children and Family Services that addressed the following ques- tions: How does the governance of multi-actor public networks influence the design of performance measures? And, how do hybrid settings affect the development of metrics within large and complex municipal government settings such as Los Angeles County? The goal is to understand the dynamics and make recommendations on the most fruitful approaches for designing performance metrics in order to ensure accountability and out- put/outcomes measures in hybrid settings.

The article proceeds as follows. First, it outlines theoretical issues by incorporating the relatively scant literature on performance measures and networks. In this section, the article provides the evolution of performance movement and the challenges associ- ated with it. The second part outlines the research subjects and methodology utilized. The third section looks at the governance structure for the county and examines multi- ple departmental performance reports/indicators and analyses of these indicators to examine whether they represent intersectoral and interorganizational networked linkages or not. This latter examination would show the complexity of working relations between multiple entities with a paucity of networked indicator designs.

Theoretical Rationale

Although the mid to late 1980s witnessed increased movement towards performance measures in the public sector where citizens demanded greater accountability and better management of taxpayers’ dollars, the idea of measuring performance predates that per- iod. Performance measurement has its roots in the early accounting systems of the Medicis (Johnson, 1981). Progression to modern times finds that for much of the 19th

and 20th centuries, accountability and performance in the public sector focused on fiscal accountability by assessing how much money was spent (de Lancer Julnes, 2006). With increasing frequency in the late 1980s and early 1990s, local governments–particularly those that relied on state sales tax as the sole mechanism for raising public funds–de- veloped interest in more balanced performance measurement to better manage public funds and increase the focus on results. During this period, measuring performance of organizations mainly was used for budgetary purposes.

336 H.P. Minassians

By the mid 1990s, the United States shifted to the reinvention movement that refo- cused attention from budgetary regulations to calibrating results. Newcomer (2007) found that 24 percent of US government agencies used performance information for funding purposes. Performance-reporting measures were developed with the goal of uti- lizing publicity as its lever on performance. Even though scholars of public manage- ment at that time (e.g., Bardach, 1998b) argued that finding ways to measure results and pay attention to what the measures reveal should improve public management and organizational performance, we still saw that politics dominated decisions related to performance measures and assessment of organizational outcomes.

By mid 2000, reporting performance measures had become the presentational strat- egy of pundits who believe that “what gets measured is what gets valued.” Proponents of this strategy argue that measuring results can create public value strictly because of its informational value; in other words, informational values could result in efficiency and a greater level of services. This requires organizations to use performance measures to become learning organizations, a potential challenge in that Weiss (1998) suggests that rigid limits imposed by law, tradition, procedures, and habits, restrain organizations from collaborating.

However, accountability through performance measures is no longer dependent on internal factors such as how finances are managed. Rather, accountability depends on what external partners request from public agencies (Melkers, 2005). These expecta- tions can be extended to have greater citizen inputs. “Citizens are viewed as important players in shaping the quality and responsiveness of government programs in their community,” (Epstein et al., 2000).

Greene (1999), on the other hand, expresses concerns with the advocates of perfor- mance measures by arguing that availability and meaning of performance measures information to different audiences could be misleading. Perrin (1998) supports this claim and suggests that the use of performance measures within a networked environ- ment might appear as rational decision-making when, in reality, many decisions based on performance measures support political goals. The question is whether using perfor- mance measures across multiple departments can enhance collaboration among the units and, thereby, lead to better public management outcomes. One of the main conceptual challenges is design of performance indicators that reflect multisectoral priorities for management (Provan et al., 2007).

Provan, Fish, and Sydow (2007) suggest one key challenge in managing networks is substantial operating autonomy by organizations. This is a crucial component of effective public management, and use of performance measures should be an extension of creating better collaboration between various public and non-public entities. O’Toole (1997) maintains that, in public administration, networks should be given serious atten- tion mainly due to the challenges that current managers face when it comes to crafting decisions. Bardach (1998a) suggests that interagency collaboration should be the main focus for building theory for leadership in collaborative settings. “Despite increased attention by researchers, theories about how to manage public networks remain rela- tively underdeveloped” (Herranz, pg. 446). These theoretical arguments should be fully developed and supported through empirical evidence where it can propose ways for public managers to design collaborative networks. This article’s primary contribution is to offer a better understanding on the complexity of designing a multisectoral and mul- tiorganizational network of indicators. Provan and Milward (1995) theorize three levels of network effectiveness: community, network and organizational. Their concept was primarily derived from their prior research and did not present empirical evidence.

International Review of Public Administration 337

While suggesting several possible categories for qualitative and quantitative indicators, they did not identify specific measurements. What is lacking from their study and anal- ysis is network participation, and the structure-based framework does not fully account for interacting coordinating processes across levels or differences in network member- ship and participation. More specifically, what is missing is the role that political fac- tors play in determining the type and purpose of performance measures when it comes to a networked environment. This study addresses the complexity of designing a multi- actor network and how governance structure hinders the design of interconnected per- formance indicators.

Furthermore, some argue that the new multisector governance is crucial to create public value (Frederickson, 1997). However, in such a “disarticulated state” or frag- mented structure, the sole responsibilities of outcomes are no longer in the hands of a single agency, and the existence of collaboration or shared decision-making has become the core element of public decisions. The use of performance measures as currently practiced should not focus solely on interdepartmental measures, but it is important to build intraorganizational indicators for the purpose of linking outcomes. Others such as Alter and Hage (1993) argue that, in an informational era, the main challenge remains how to manage complexity, scope, speed and adaptability in a network context. These challenges are heightened in managing these governance networks through use of per- formance metrics.

Agranoff (2001) argues that in a context of network governance, public managers encounter horizontal overcrowding where collaboration between and among various public entities remains paramount. One could argue that this overcrowding places greater importance on the design of interlinked performance measures and indicators, challenges which will be examined in this study. Using performance measures and reporting outcomes has become the preferred mode of building multisectoral and multi- organizational performance ties. Performance reporting is seen as a tool to link organi- zational networks.

Using the measuring tools discussed below can strengthen delivery of public ser- vices to various stakeholders. Furthermore, reporting to intergovernmental and non- governmental entities (citizens) can enhance performance management as a phenomenon of networks. Performance reporting, on the other hand, searches for some common indicators within similar types of institutions and public organizations to examine and compare. Parmenter, in particular, suggests that organizations should rely not only on performance measures but must draw “key performance indicators” (2007). However, the missing link within the performance chain requires interlinking perfor- mance indicators across various intra-organizational units so that outputs and outcomes are measurable.

To summarize, the demand for accountability has to do with the satisfaction level of constituents with the ways in which tasks have been performed. “Managing for Results” attempts to link organizational outcomes to the needs of intergovernmental constituents or citizens at large. Also, the use of performance measures allows stake- holders to create multilevel reporting strategies by building a pyramid of indicators with some focusing mainly on organizational efficiency and others reporting results to intergovernmental and community-based stakeholders. This creates what O’Leary and Bingham (2009) call the interdependence of stakeholders.

338 H.P. Minassians

Research Subjects: Los Angeles County

The use of performance indicators for management of public organizations and report- ing purposes to external governmental entities (intergovernmental ties) can be divided into four types: inputs, processes, outputs, and outcomes (Carter, Klein, & Day, 1992; Cave, Hanney, & Kogan, 1991; Richardson, 1994; Hatry, 1999).

The three affirmed goals of performance reporting are to: 1) demonstrate account- ability; 2) improve performance of the agency; and 3) meet the needs of both the gov- ernmental entity and its constituents (Burke & Minassians, 2003). These three goals are critical in linking inter- and intra-departmental ties since many of the performance indi- cators used for managing public programs are devised and at times dictated by manage- ment of public organizations.

Los Angeles County is the second largest municipal government in the United States with 10 million residents and nearly 100,000 public sector employees. In 2006, the county shifted its governance structure from a Chief Administrative model to a Chief Executive model. Under the new CEO model, county departments are divided into five clusters, each having unique goals that tie its organizational structure to the county’s strategic goals and plans. The idea behind shifting the county governance structure is to increase collaboration between departments and design performance mea- sures across each department that will result in better outcomes and better connectivity with the strategic plan of the county. Reliance on different clusters and the design of department-level performance measures is a new attempt to create better intra-organiza- tional metrics and measurement in ensuring accountability (see Appendix A).

Nevertheless, the clustering of the departments is based on general purpose and not functionality across programmatic purposes and working relations. Thus, the design of indicators by type does not reflect any linkages in this environment where a single agency does not have complete autonomy over outputs and outcomes of a program. These clusters consist of Operations; Children and Families’ Well-Being, Health and Mental Health Services; Community Services and Capital Programs; and Public Safety. The Department of Children and Family Services and Sheriffs Department are not part of any of these clusters and, instead of reporting to the deputy chief executive officer, they directly report to the elected body, the Board of Supervisors. These changes have occurred due to various political challenges and desire for autonomous functionality by the DCFS and other challenges that these departments face. Los Angeles County has instituted a program called “Performance Counts,” which is part of the annual budget document and provides program summaries and performance measures for each depart- ment. The reporting of indicators by departmental operations is used to conduct longi- tudinal analyses as well as being a vital budgetary tool. Missing from the content of these reports, however, is a clear statement of goals that specifically identify how the measurement of results provides value to taxpayers. A primary reason for limiting dis- tribution of the reports is that they may be misinterpreted by laypeople because of the lack of additional information with which other counties’ and agencies’ performance may be compared.

Here, we examine the design of performance measures across multiple clusters that are directly engaged in addressing “Children and Families’ Well-being.” These clusters consist of multiple departments such as Child Support Services in the Department of Children and Family Services (DCFS), the Department of Public Social Services (DPSS), Department Health and Mental Health (DMH), Department of Probation and the Sheriff’s Department. This study will examine the linkages, if any, that exist across

International Review of Public Administration 339

four departments–DCFS, DPSS, DMH and the Sheriff’s Department–and regions and analyze whether performance measures actually link intra-organizational functions or if they rely solely on single-department reporting.

The continuous issues that the DCFS has encountered over the years concerning the protection of children within the Foster Care system and/or under the direct supervision of families under the DCFS have served as the catalyst for this study. During the past decade, DCFS has witnessed the death of a number of children under their direct super- vision. Appendix B presents a hypothetical family in Los Angeles County that depends on various public services. There are also numerous private and not-for-profit providers that work directly as contractors or sub-contractors with county departments to provide services to needy children and families. The issue under consideration is the effectiveness of performance measures utilized by various departments in addressing intra-departmental activities.

Methodology

To answer the research questions posed for this study—Does the governance of multi- actor public networks influence the design of performance measures? and How do hybrid settings affect the development of metrics within large and complex municipal government settings such as Los Angeles County?–this study examined performance indicators to determine whether or not inter-organizational and intersectoral measure- ments of performance exist. Organizational level indicators were examined to see if col- laborative measurement indicators exist.

Data collected from the latest performance reports published by the county and by means of interviews with five elite directors or former directors of county departments were organized into various categories outlined in Appendix C for analytical purposes. Finally, a closer examination of the indicators was reflective of whether they depict multisectoral and multiorganizational characteristics or if the indicators were insular from their inceptions.

First, by acquiring the most recent documents and information available on Pro- gram Summary and Performance Measures from Los Angeles County, this study was able to analyze the format and content to determine readability. Of special note were all comments relative to the intended audiences, purposes, and priorities of the reports, and the type and quantity of the performance indicators. In all, there are 309 indicators used for measuring performance outcomes across the four departments examined for this study: Department of Public Social Services, Department of Mental Health, Depart- ment of Children and Family Services, and the Sheriff’s Department. Each indicator was categorized by the types, values, and collaborative nature of each indicator (see Appendix C). Second, by categorizing each indicator type, one could discern the public values projected by each, whether those values represent the internal concerns of the organization or address external concerns, and finally, what collaborative nature was applied to these indicators.

Upon completion of these steps, a descriptive analysis was utilized in order to identify the percentage of indicators that reflect the values, types, and collaboration of perfor- mance reporting priorities. The chi-square method was utilized to identify the relationship between various departments and their indicators. Closer assessment of performance mea- sures indicates that they reflect multi-organizational priorities and intergovernmental ties. What is lacking, however, are indicators crossing multiple organizations. Categorizing these indicators into different values, types, and interconnectedness remains a subjective

340 H.P. Minassians

call by the reader. Scrupulous examination of each indicator and assigning specific categories are challenging tasks and could represent the weakest link in the methodology utilized for this study. Although assignment of different categories to each indicator is a subjective call, the value of assigning categories to each indicator sheds new light and understanding on how indicators are designed.

Analysis and Findings: What Do They Denote and How Are They Used?

As discussed earlier, performance-reporting indicators may be categorized into four types: inputs, processes, outputs, and outcomes. Inputs involve the human, financial, and physical resources employed by public agencies to support programs, activities, and services. Processes are the means used to deliver services—in this case, completion of client eligibility in a timely manner. Outputs reflect the quantity of products or ser- vices actually provided or the number of services completed, e.g., the percent of wel- fare recipients placed in a job. Outcomes cover the quality of the benefits, activities, and overall impact of the services to recipients or the community at large, an example of which would be customer service satisfaction levels.

Considering 309 indicators designated by the county out of the four agencies pre- sented (see Table 1), there exist ranges of 30 to 75 percent reflecting inputs, 8 to 35 percent reflecting processes, and 5 to 16.4 percent reflecting outputs. Surprisingly, only 11 percent of the indicators for the DCFS department deal with outcomes. The other departments either do not have any outcome measures or those measures represent only 1.5 percent of the indicators. The results of a chi-square statistical analysis suggest that the four departments are not consistent in weighing the indicators. This suggests that each department has designed their own indicators without placing emphasis or thought into how input indicators impact the outputs or processes of another department.

Examining outcome indicators within the DCFS more closely, one can see that out- comes tend to be more programmatic in design, such as “children who do not have a recurrence of substantiated abuse or neglect within 6 or 12 months after receiving Vol- untary Family Reunification or Family Preservation services.” Missing from the out- come indicators is the working relationships that DCFS has with the departments and units listed below and how inputs, tasks, processes, and output of activities from these entities impact the well-being of a child. The various departments and units with which the DCFS has relationships are, the Los Angeles Sheriff’s Department (LASD), the Department of Mental Health (DMH), the District Attorney’s Office (DA), the Depart- ment of Health Services (DHS), the Department of Public Health (DPH), the Depart- ment of Coroner, and the Department of Public Social Services (DPSS). At the same time, through contractual agreements, DCFS works with at least 30 other public, non- profit, for-profit and community-based organizations such as the Inter-Agency Council on Child Abuse and Neglect (ICAN), First 5 LA, Domestic Violence Council, and Los Angeles County and University of Southern California Medical Centers. Some of the community-based organizations are Grandparents as Parents, and ROCK as well as rep- resentatives from the Countywide Community Child Welfare Coalition, including SHIELDS for Families, Project IMPACT, Bienvenidos, Para Los Niños, and Children’s Institute, Inc.

What the list of these public, non-profit and community-based organizations repre- sents is a paucity of indicators across multiple players in a hybridized environment. One might raise the question of why design or performance indicators in a networked hybridized environment remain insular. During interviews with the county department

International Review of Public Administration 341

T ab le

1 .

Im p o rt an ce

o f In d ic at o rs

b y D ep ar tm

en ts .

R es u lt s

D C F S

D M H

S h er if f

D P S S

R o w T o ta ls

In p u ts

1 4 † † (3 6 .0 7 )† † [1 3 .5 0 ]

2 5 † † (2 2 .0 4 )† † [0 .4 0 ]

1 0 8 † † (8 8 .1 6 )† † [4 .4 6 ]

3 2 † † (3 2 .7 3 )† † [0 .0 2 ]

1 7 9

P ro ce ss es

2 4 † † (9 .0 7 )† † [2 4 .5 9 ]

4 † † (5 .5 4 )† † [0 .4 3 ]

1 2 † † (2 2 .1 6 )† † [4 .6 6 ]

5 † † (8 .2 3 )† † [1 .2 7 ]

4 5

O u tp u ts

1 6 † † (8 .8 7 )† † [5 .7 4 ]

4 † † (5 .4 2 )† † [0 .3 7 ]

1 2 † † (2 1 .6 7 )† † [4 .3 2 ]

1 2 † † (8 .0 4 )† † [1 .9 4 ]

4 4

C o lu m n T o ta ls

5 4

3 3

1 3 2

4 9

2 6 8 † † (G

ra n d T o ta l)

N o te s: T h e ch i- sq u ar ed

st at is ti c is 6 1 .6 9 9 7 . T h e p -v al u e is < .0 0 0 0 1 . T h e re su lt is si g n ifi ca n t at

p < .0 5 .

342 H.P. Minassians

heads and former directors of various departments, one explanation was that collabora- tion across county departments is sparse during a crisis since no one wants to own the blame (03/12/2014, 07/10/2014, 01/10/2015 Anonymous). Moreover, in a recent inter- view with the former county top officials it has become apparent that the biggest chal- lenge to progress, was the lack of political support from the Board of Supervisors. Some of the Board members felt such a governance system was circumventing their ultimate authority over county government. In fact, the Board is currently trying to undo the very governance structure that was designed to provide a more holistic approach to integrated services. This latest observation and finding suggests that at times politics trumps the logical and rational processes that pundits of performance measures recommend.

The trend in performance data collection and reporting is to examine and evaluate a diverse array of policy values such as choice, equity, efficiency, and quality (Richardson, 1994). Choice as a policy value allows policymakers to choose from a range of options. An example of this is the percent of clients referred for clinical assessment. This indicator reflects the options available to the recipients of agency ser- vice as well as suggesting that taxpayers’ money is being spent efficiently to improve the lives of clients and reduce their dependency on other public programs. Choice as a policy value provides to the external clientele and stakeholders access to a variety of services. It is reflective of the concerns of stakeholders and simultaneously provides feedback to internal agency management as to where resources should be invested.

Efficiency is measured by calculating the resources received in relation to the results achieved, indicative of a cost/benefit analysis. These sets of indicators explain how public resources are being spent and the ratio of efficiency. An example of this indicator is the percent of performance evaluations completed by a stated due date with the Department of Mental Health. Efficiency has an important value since many exter- nal stakeholders focus on efficient service delivery and how taxpayers’ dollars are spent. Equity represents how the agency responds to disparities in needs and the response towards diversity among participants. Quality consists of achieving an intangi- ble set of values that goes beyond efficiency. Measuring quality is, of course, the most elusive value because it is difficult to measure and even more difficult to agree upon consistent definitions of this factor.

The focus on efficiency is aligned with what Cole and Parston (2006) call public service organizational values that aim at “delivering a set of outcomes that are aligned to citizen priorities in a cost-effective manner” (Cole and Parston, page 6). However, indicators utilized by the agency have fewer outcome indicators designed to furnish information of value specifically to clients and citizens. One explanation for the paucity of quality and choice indicators may be the requirement of external accountability to state and federal agencies. Programs funded by federal, state, and local government tend to focus on efficiency due to their accountability to taxpayers. Use of indicators by external stakeholders is a mechanism to assess the effectiveness of programs.

Closely analyzing all indicators, one will see that efficiency remains the dominant value. This is probably due to the fact that many departments are focused on respond- ing to the elective bodies and the public to explain how tax dollars are being spent. The Sheriff’s Department reports 82 percent of indicators in efficiency as the highest priority, and DPSS reports 45 percent of indicators, representing the lowest priority of the four departments. Meanwhile, values of quality remain low, hovering around 15 percent. The exception is the Department of Mental Health with 29 percent of the indi- cators, reflecting DMH’s reliance on 10 indicators that examine the re-hospitalization

International Review of Public Administration 343

rate for various age groups within various timeframes. The idea behind these indicators is that they represent the quality of care that each patient received as measured by suc- cessful medical intervention that does not require additional care. However, as far as mental health and quality of care, the indicators representing this particular value remain elusive. At the same time, political makeup of the legislative body and their political muddling, Los Angeles County Board of Supervisors, undermines the pro- grammatic realities. Performance-reporting indicators signify specific characteristics that public sector management teams and various stakeholders within a networked environ- ment use to determine how many and which kinds of resources will be utilized to deli- ver services. Clearly, more measures are no defense of quality in indicator systems for public management. The lack of emphasis on a few quality measures may well reflect the reluctance of departments to more clearly define the critical elements of service quality. The challenge is that the public organizations in the social services area have failed to design measurable indicators that would enable internal and external support. The debate surrounding the shift from New Public Management to New Public Gover- nance where value created by the whole of government activities requires effective management of a hybridized environment suggests that the objective of performance measures should be broadened beyond efficiencies and effectiveness of governmental programs (Morgan et al., 2014). Instead, measuring the performance of organizations should address a wide array of substantive political values and the creation of trust upon which citizens can rely.

Finally, an important policy question is whether the selected indicators express the needed strategic linkages between departments or solely the internal concerns of the four departments. A close analysis of the indicators suggests that internal purposes for indicators for DMH, DCFS and Sheriff’s Department dominate their presence (see Figure 1).

A closer inspection reveals that the Sheriff’s Department’s focus on internal depart- mental concerns was at 87 percent of indicators, followed by DCFS at 78 percent and DMH at 68 percent. Meanwhile, DPSS uses only 9 percent of their indicators as an internal purpose to the organization. This suggests that these four departments do not place the same weight on indicators representing internal or intra-organizational con- cerns. One interpretation could be that while DPSS functions as an organization in the county verifying eligibility of clientele who receive services directly from DCFS, the DMH or Housing Authority do so indirectly by issuing Medicaid cards or identifying

Figure 1. Indicators by Inter/Intra-Organizational Purpose.

344 H.P. Minassians

the eligibility of welfare recipients, which are services aimed at building individual capacity for each recipient to become better integrated into the community.

The heavy internal emphasis on performance reporting is especially surprising, likely the result of indicator selection remaining purely in the realm of the departments, unlike the use of other collaborative strategies.

Discussion of Finings: Indicators as a Tool in Evaluation of Networked Governance

The preceding sections presented a grounded analysis of performance measures utilized in a large municipal setting through analysis of indicators used by four county depart- ments responsible for child welfare and the county structure that was designed to enhance intersectoral collaboration. This article specified and analyzed sets of perfor- mance indicators across multiple public organizations, then examined whether the design of performance measures by each public agency responsible for the welfare of families and children has tied these measures across the network of public-public and public-not-for-profits and community-based organizations. This study compared the multi-bureaucratic performance indicators to assess whether performance designs reflect network ties across public-public, public-private and public-not-for-profit partnerships that represent the network of participants. The analysis, reflecting the previous studies of Provan and Milward (2001), Herranz (2010), and Burke and Minassians (2002) found that a bureaucratically-oriented network of coordination approach and perfor- mance indicator designs were associated with a paucity of indicators reflecting network connections and practices. In other words, even though there are existing networks of participants with various sizes of nodes and multiple connective pivots, the perfor- mance indicators were insular to the organizations without reflecting the network itself.

This article submits that even though a theory of network performance may be bet- ter organized in a centralized network of administrative organizations, multisectoral net- works lack coordination when it comes to design of performance indicators. A prime example is the lack of performance indicators across county-created clusters and vari- ous departments. Future studies should examine multisectoral network membership and coordination mechanisms for assessing interconnectedness of performance measures. Clustering governmental departments based on similar functions does not automatically constitute better collaboration. Moreover, in lieu of clustering, intradepartmental and intersectoral performance indicators could enhance greater collaboration. Fundamen- tally, the use of performance measures is to assess outcomes of public programs within an integrated network of players.

In the case of Los Angeles County and the participation of all departments in the design and dissemination of performance indicators, one might argue that accountability is being achieved. However, the real missing link in the performance chain is the lack of integrated indicators that would allow measurement of overlapping functions, out- puts, and, ultimately, the outcomes of multiple departments. Significant in its absence are the linkages between county departments as well as between community and exter- nal groups, and contractors and providers to the overall performance of a network of participants. Performance indicators should link the performance of community-based organizations (acting, at times, as contractors) to the performance of various county departments. This could be achieved by creating vertical and horizontal indicators that, when linked, would encourage more and better collaboration.

International Review of Public Administration 345

Even though this article attempted to investigate whether there exists an array of performance indicators for measuring the outcomes of a network, multisectoral net- works might well consider the potential for network performance coordination when considering the design of accountability measures. This recommendation would result in a pyramid of indicators. At the top would be the core common measures of perfor- mance on the most critical contributions while allowing mid- to higher-level public managers to make needed changes within performance indicators at the departmental level.

A well-developed plan of action is critical to the success of any program, and per- formance measures can bring some form of alignment and focus to actual activities (Cole & Parston, 2006; Monahan, 2001; Moore, 1997). Within the context of hybridization of intergovernmental and non-governmental ties, performance measures and the use of indicators remain critical in assessing and evaluating the outcome of public programs. Using performance measures allows various external actors and stake- holders to link the policy continuum, from upstream policy development at the federal and/or state level to the midstream of policy implementation and, specifically, to the downstream—policy/decision-enforcement—level.

One aspect of this article is that performance measures have certain normative assumptions embedded within them (Anglin, 2004). Thus, if indicators used for perfor- mance measures aim to assess collaborative characteristics of different organizations, one could argue that the public value generated by these organizations are collabora- tive. When the collaborative practices of different organizations are being managed, therefore, it is common for performance measures to be mandated by external legisla- tive bodies.

Also, close examination of performance measures for these agencies suggest that measuring outcomes is more challenging when compared to public services with more quantifiable actions. The use of performance measures might assess organizational prac- tices, but whether these practices can shape the behavior of the target population’s behavior remains unclear. For example, one might ask what is missing from the perfor- mance chain between various departments in relation to the number of child-abuse cases investigated that lead to referrals to DCFS; or the number of abuse cases that lead to referral to the District Attorney’s Office, and how many of these then lead to suc- cessful prosecution. For DCFS, indicators are successful placement of foster children and the length of time (measured in months and years) that foster children stay with those families. Access to mental health services has also been identified as a need by foster parents, especially for the adolescents in their care. One might also examine the level of support provided to foster parents by the child welfare agency and, as part of that process, identify some of the difficulties that foster parents experience in communi- cating with the welfare agency.

Overall, this article argues that designing different performance measures across net- works could create differential multilevel network performance. This study suggests that researchers and public managers consider the performance implications of strategic orientations in the design of performance measures in a network environment to pro- mote improved managerial coordination. Finally, this study suggests that creating clus- ters for increased collaboration without the actual functional relationship between various county departments would not automatically lead to better collaboration and/or the creation of multi-departmental or intersectoral performance indicators.

346 H.P. Minassians

Lessons Learned and Conclusion

Focusing accountability on improvement begins at the institutional level by establishing clear goals, realistic objectives, and relevant indicators that reflect the county mission and state needs. Data generated by public agencies do not identify meanings and signif- icance, but data translated into relevant information and, more specifically, utilized to generate additional knowledge can assist organizations to move from their current sta- tus quo into learning organizations. Decision makers may receive information, but orga- nizational change through generations of knowledge gathering requires profound understanding and reflection. This trend requires more intense involvement of local departments and districts in the design, implementation, and interpretation of the pro- cess of performance measurement. One strategy for organizations to utilize to accom- plish their desired goals is explicit identification of the social benefits that each activity generates. Ultimately, the design of performance measures in a hybridized and net- worked environment requires greater understanding of procedural logic associated with these types of programs and intersectoral relations.

Findings from the study presented here suggest a number of recommendations for improving the development of indicators in hybrid environments similar to Los Angeles County. There are number of lessons that this study reveals. First, design of governance structure is crucial in establishing interdepartmental and intersectoral collaborations. Even though the county has clustered various departments based on their objectives and functions, the missing link is addressing functions and activities that cross multiple clusters and departments (e.g., Sheriff’s Department vs. DCFS vs. District Attorney’s Office vs. DMH, etc.) Also, this elevated emphasis on efficiency rather than on other program and organizational values is expected, since public organizations are now under a greater mandate to deliver programs that are perceived as efficient. For practi- cal purposes, however, efficiency cannot be used as the only measurement of output and outcome. Closing the accountability gap requires transferring the reporting respon- sibility on some common indicators down to the departments and, if needed, even down to the regional offices. Further research is required to identify what kind of com- mon core performance measures and indicators can be designed to assess overall output and outcome of clustered programs. An additional question is how one can assess and manage the political variables that hinder design of such a system. Also, it is essential for various departments to work with advocacy and community non-profit groups to establish measures to link external functions to the internal processes and the outputs and outcomes of these functions. For example, kinship groups such as “Grandparents as Parents” (GAP) and the level of support and services that DCFS provides them should be linked into the performance measures indicators. This is the idea behind the creation of a pyramid of indicators that reflect internal, external, inter- and intra-organi- zational functions. Further research is needed in order to measure how activities of non-governmental actors contribute to the overall success of programs. More research is needed to identify if there is any best model for enhancement of intergovernmental and intersectoral collaboration and better design of performance measures linking out- puts and outcomes.

From this study, one can draw two recommendations for practitioners to improve delivery of public services in an urban setting and more accurately measure those ser- vices to translate the data in a meaningful way. These recommendations will also strengthen intergovernmental ties. It is recommended that federal, state, and county planners have coordinated, common reporting indicators with an emphasis on selected

International Review of Public Administration 347

critical areas. Each agency should then be encouraged to develop specific indicators representing the unique needs of its own community. The ideal model for designing performance indicators to strengthen collaborative and intergovernmental ties is through the creation of indicator pyramids where some indicators report outcomes to the federal government, some to the state, and some to the local Board of Supervisors and the community. Finally, measurement indicators must focus more on the value of services as they are perceived by both clientele and taxpayers.

Notes on contributor Henrik P. Minassians, PhD, is associate professor of Urban Studies and Planning, California State University, Northridge. Prior to his appointment at Northridge, Dr. Minassians served as a senior research associate for the higher education program at the Nelson A. Rockefeller Institute of Government at the University of Albany. Email: [email protected]

References Agranoff, Robert. (2001). Managing within the matrix: Does collaborative federalism exist? Pub-

lius: The Journal of Federalism, 31(Summer), 31–56. Alter, Catherine & Hage, Jerald. (1993). Organizations working together. Newbury Park, CA:

Sage. Anglin, R. V. (2004). Building the organizations that build communities. Washington, DC: US

Department of Housing and Urban Development. Bardach, E. (1998a). Getting agencies to work together. Washington, DC: Brookings Institution

Press. Bardach, Eugene. (1998b). Managerial craftsmanship: Getting agencies to work together. Wash-

ington, DC: Brookings. Burke, J. C., & Minassians, H. P. (2003). Reporting high education results: Missing links in the

performance chain. San Francisco, CA: Jossey-Bass. Carter, N., Klein, R., & Day, P. (1992). How organizations measure success: The use of perfor-

mance indicators in government. London: Routledge. Cave, M., Hanney, S., & Kogan, M. (1991). The use of performance indicators in higher educa-

tion: A critique and analysis of developing practice (2nd ed.). London: Kingsley. Cole, M., & Parston, G. (2006). Unlocking public value. New Jersey, NJ: John Wiley & Sons. Julnes, De Lancer (2006). Patria. “performance measurement: An effective tool for government

accountability”. Evaluation, 12(2), 219–235. Epstein, P., Wray, L., Marshall, M., & Grifel, S. (2000). “Engaging citizens in achieving results

that matter: A model for effective 21st century governance”, Paper presented at the Center for Accountability and Performance’s Symposium on Leadership of Results-Oriented Manage- ment in Government, Washington, DC, 11 February.

Frederickson, H. George. (1997). The spirit of public administration. San Francisco, CA: Jossey- Bass.

Greene, J. (1999). The Inequality of Performance Measurements. Evaluation, 5(2), 160–172. Hatry, H. (1999). Performance measurement: Getting results. Washington, DC: Urban Institute. Herranz, Jr., Joaquin. (2010). “Multilevel performance indicators for multisectoral networks and

management”. The American Review of Public Administration, 40(4), 445–460. Johnson, H. T. (1981). “Towards a new understanding of nineteenth century cost accounting”,

The Accounting Review, 56, 510–518. Melkers, J., & Willoughby, K. (2005). Models of performance-measurement use in local govern-

ment: Understanding budgeting, communication, and lasting effects. Public Administration Review., 65(2), 180–190.

Monahan, K. E. (2001). Balanced measures: For strategic planning. Vienna, VA: Management Concepts.

Moore, M. H. (1997). Creating public value. Cambridge, MA: Harvard University Press. Morgan, Douglas F., & Cook, Brin. J. (2014). New public governance: A regime-centered

perspective. NY: M.E. Sharpe.

348 H.P. Minassians

Newcomer, K. (2007). How does program performance affect program management in the federal government? Public Performance and Management Review, 30(3), 332–350.

O’Leary, R., & Bingham, L. B. (2009). The collaborative public manager: New ideas for the twenty-first century. Washington, DC: Georgetown University Press.

O’Toole, Laurence J. (1997). Treating networks seriously: Practical and research-based agendas in public administration. Public Administration Review, 57, 45–52.

Parmenter, D. (2007). Key performance indicators. Hoboken, NJ: John Wiley and Sons. Perrin, B. (1998). Effective use and misuse of performance measurement. American Journal of

Evaluation, 19(3), 367–380. Provan, K., Fish, A., & Sydow, J. (2007, June). Interorganizational networks at the network level:

A review of the empirical literature on whole networks. Journal of Management, 33(3), 479– 516.

Provan, K. G. & Milward, H. B. (2001). Do networks really work? a framework for evaluating public sector organizational networks. Public Administration Review, 61, 414–423.

Provan, K. G. & Milward, H. B. (1995). A preliminary theory of interorganizational network effectiveness: A comparative study of four community mental health systems. Administrative Science Quarterly, 40, 1–33.

Richardson, R. C. (1994). Effectiveness in undergraduate education: An analysis of state quality indicators. In S. S. Ruppert (Ed.), Charting higher education accountability: A sourcebook on state-level performance indicators (pp. 40–51). Denver, CO: Education Commission of the States.

Weiss, C. H. (1998). Have we learned anything about the use of evaluation? American Journal of Evaluation, 19(1), 21–33.

International Review of Public Administration 349

Appendix A.

350 H.P. Minassians

Appendix B.

Source: Margaret Dunkle, The George Washington University & The LA County Children’s Planning Council, 2002.

International Review of Public Administration 351

Appendix C.

DPSS – Few Sample of Indicators

Indicators - Social Services IHSS † † † Customer service & satisfaction rating OUT QUL Inter Percentage determining eligibility in timely manner P EFF Inter/Intra IHSS reassessment, percent completed P/OP EFF Inter/Intra IHSS consumers qualify and receive care, percent

completed P/OP EFF Inter/Intra

Number of IHSS consumers served OP EFF Inter/Intra

Sheriff – Few Sample of Indicators Performance Measures Type Policy

Value Collaborative

Court Services Budget Unit † † † Trial Court funding contract Input Quality Inter/Intra LAPD release contract input Efficiency Inter/Intra Courthouse visitors input Efficiency Inter/Intra Annual inmate population (per day court appearance

annually) input Efficiency Inter/Intra

Courthouse incidents input Quality inter Arrests input Quality inter Weapons seized input Quality inter

DCFS - Few Sample of Indicators Performance Measures Type Policy

Value Collaborative

Early Intervention † † † Percent of children receiving Family Support services who

do not have a recurrence of substantiated abuse and/or neglect while receiving Family Support Services

Output Eff Inter

Percent of children receiving services who do not have a recurrence of substantiated abuse and/or neglect w/in 12 months after receiving Family Support Services

Output Eff Inter

Percent of children receiving Alternative Response services who do not have a recurrence of substantiated abuse and/or neglect while receiving Alternative Response

Output Eff Inter

Percent of children who do not have a recurrence of substantiated abuse and/or neglect within 12 months after receiving Alternative Response services

Outcome Eff Inter

DMH - Few Sample of Indicators Outpatient Mental Health Services † † † Percent of clients who are satisfied or very satisfied with

services and the quality of life Output Quality Inter

Percent of all clients receiving outpatient services who are identified with substance related disorder

Input Eff Inter/intra

Percent of clients with open outpatient cases have received outpatient services w/in 90 days previously

Input Eff/ Quality

Inter/intra

Percent of discharges from an acute hospital, seen within 7 days, at an outpatient facility:

Input Eff Inter/intra

Children (15 and under) Input Eff Inter/intra Transition age youth (16-25 years) Input Eff Inter/intra Adults (26-59 years) Input Eff Inter/intra Older adults (60 years and over) Input Eff Inter/intra

352 H.P. Minassians

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.