Module 6: Group Discussion

472 Public Administration Review • May | June 2018

Public Administration Review,

Vol. 78, Iss. 3, pp. 472–478. © 2017 by

The American Society for Public Administration.

DOI: 10.1111/puar.12888.

This manuscript was originally submitted

and accepted as an Evidence in Public Administration article. The feature

editors, Kimberley R. Isett, Brian W.

Head, and Gary VanLandingham, are

gratefully acknowledged for their work

in soliciting and developing this content.

Effective with Volume 78, the Evidence in Public Administration feature has been

discontinued.

Chris Silvia is assistant professor in the

Romney Institute of Public Management

at Brigham Young University. His research

focuses on leadership and management

in intersectoral networks. His work has

been published in Journal of Public Administration Research and Theory, Public Administration, Public Administration Review, and the Leadership Quarterly .

E-mail: [email protected]

Viewpoint

Abstract: Collaboration has become the predominant approach to solving complex public problems. This choice, however, often is not driven by demonstrated effectiveness. Collaboration is instead chosen in the hope that a networked arrangement will be more effective than individual organizations working on the issue alone. Questions regarding collaborative effectiveness persist and constitute a significant challenge facing both public management practitioners and public administration scholars. In light of the case study in this issue of Public Administration Review by Maurits Waardenburg and colleagues, this article reviews the current thinking on the measurement of collaborative performance and discusses steps that professionals can take to evaluate the effectiveness of their collaborative endeavors .

Chris Silvia Brigham Young University

Evaluating Collaboration: The Solution to One Problem Often Causes Another

Public organizations at all levels of government routinely engage in collaborative arrangements to deliver services and develop policy. The

public management literature offers many rationales for these organizations ’ decision to collaborate. Some believe that an increase in collaboration has been spurred by the realization that the issues with which public organizations are grappling are often too complex, costly, or extensive for a single entity to adequately address (O ’ Leary, Choi, and Gerard 2012 ). Organizations must therefore work together to effectively confront these complex problems.

The promise of the effectiveness of collaboration has resulted in many organizations turning to or even requiring collaborative approaches. Drawing on institutional theory (DiMaggio and Powell 1983 ), Skelcher and Sullivan ( 2008 ) discuss three isomorphic explanations for the rise of collaborative arrangements. The first is mimetic isomorphism. Having been influenced by the prevalence of collaborative networks, many government agencies have attempted to mimic their peers and replicate their success, or purported success, by collaborating. Second, many professional organizations, such as the International City/County Management Association, and government organizations, such as the Canadian Privy Council Office, the Australian Public Service Commission, the New Zealand State Services Commission, and the South African Department of Public Service and Administration, have promoted the practice of collaboration (normative isomorphism). Believing that collaboration is the way of the future, many professional organizations have established collaboration as a norm of professional practice.

Finally, government agencies have been required to pursue networked approaches through mandates from higher levels of government (coercive isomorphism) because of their presumed effectiveness (Bryson, Ackermann, and Eden 2016 ). This belief has resulted in higher levels of government making their funding and support conditional upon a collaborative approach as a hedge against risk (Bryson, Crosby and Stone 2006 ; Ostrower 2005 ).

As Dickinson and Glasby ( 2010 ) point out, however, collaborative approaches often are not adopted because they are effective. Rather, collaborative approaches are frequently pursued because of internal and external pressures based on the assumption that collaboration will be effective. In essence, for many, collaboration has become a hammer and nearly all problems have become nails. In absence of empirical evidence of effectiveness, public decision makers have to make the best of what they have available by relying on “anecdote or efforts of comparators or [taking] cues from various innovation awards” (Isett, Head, and VanLandingham 2016 , 22). As a result, many governments invest substantial time and money in collaborative networks without knowing whether what they are doing is effectively tackling the problem or how to measure the collaborative ’ s effectiveness (Koontz and Thomas 2006 ). Evidence of effectiveness is critically important because it allows the network to “focus [its] efforts and account for [its] behavior and the way [it spends its] resources” (Koppenjan 2008 , 700).

Turning to collaboration to address a complex problem has spawned a new problem: how to measure collaborative effectiveness. “The difficulty

Stephen E. Condrey,

Associate Editor

Evaluating Collaboration: The Solution to One Problem Often Causes Another 473

of determining the effectiveness of network collaboration is due to the fact that traditional measures used for assessing public policy are inadequate” (Koppenjan 2008 , 701). Thus, the critical question facing nearly every collaborative endeavor is how to measure its effectiveness. The collaborative partners must determine why the effectiveness of the collaboration is being assessed, from whose perspective to evaluate the effort, what to measure, and when and how it will be measured. As Isett, Head, and VanLandingham ( 2016 ) argue, this requires the public manager to collect quality data and use appropriate analytical approaches in order to determine “what works.”

These very issues are discussed in the case study by Maurits Waardenburg and colleagues, “Evidence-Based Prevention of Organized Crime: Assessing a New Collaborative Approach,” in this issue of Public Administration Review . The authors describe the genesis for the formation of a collaborative approach to combatting organized crime and human trafficking in the Netherlands and the techniques that have been used to evaluate its effectiveness. In response, this article discusses the current thinking on collaboration and how professionals can evaluate its effectiveness.

Why Measure? The purpose for assessing the collaborative effort should drive the measurement strategy. As Waardenburg et al. note, metrics are often required for accountability reasons. In fact, Linden quotes David Osborne as saying that “the most important thing that you can do to drive interagency collaboration is to make all agencies accountable for improving outcomes” (Linden 2010 , 219). The public administration literature contains a number of perspectives on collaborative accountability. Lewis, O ’ Flynn, and Sullivan ( 2014 ) identify different purposes that accountability serves by applying Fitzpatrick ’ s ( 2008 ) discussion of three moral philosophies. An accountability system can focus on trying to ensure that the party being held accountable is doing good (consequentialism), doing what is right (Kantianism), and/or being good (virtuism). The accountability metrics described by Waardenburg et al. take a consequentialistic approach in that they are focused on assessing how well the network ’ s actions address the problem of human trafficking in the Netherlands. While this is perfectly justifiable, other perspectives are equally appropriate. The network could have taken a Kantian approach, which “situates accountability as the means by which universal principles can be protected and promoted,” or virtuism, “with its emphasis on individual experience as well as action, affords accountability the means to support individual development” (Lewis, O ’ Flynn, and Sullivan 2014 , 402). Regardless of the philosophic rationale for the accountability strategy, accountability is very complicated in a collaborative environment. “When public policy is produced in complex networks featuring multiple, overlapping coordination mechanisms ... accountability easily gets lost in the cracks of horizontal and hybrid governance” (Bovens, Schillemans, and ’t Hart 2008 , 240).

Effectiveness for Whom? There is considerable debate swirling around the topic of how to measure effectiveness in a collaborative environment. There are differences of opinion regarding for whom the collaborative should be effective, how effectiveness should be measured, what should be measured, when it should be measured, and why it should

be measured. Before addressing these questions, it is important to consider the different perspectives at play in a collaborative environment. The stakeholders will likely have varying perspectives, disciplinary approaches, and organizational biases (Head 2008 ) and therefore different, albeit related, understandings of effectiveness (Koppenjan 2008 ). Thus, it has been argued that network effectiveness might best be determined by assessing the “extent to which a network fulfills the collective needs of the participants, whatever the needs are and however they have been formulated” (McGuire and Silvia 2009 , 37).

The network itself is a critical determiner of what constitutes the effectiveness of the collaborative. One of the most critical steps a network must take is to establish a common understanding of its goals and mission (Agranoff and McGuire 2001 ; McGuire 2002 ; McGuire and Silvia 2009 ). Coalescing around these common objectives sets the stage for defining effectiveness, at least from the network ’ s perspective. It is important to note that this shared perspective is the amalgamation of the views of the individuals who participate in the network as well as those of the home organizations that send these participants to the collaborative as their representatives. Therefore, the individual collaborators and their home organizations are important stakeholders of the network.

Each network participant faces the potential of three different influences as they conceptualize what effectiveness means to them. They have their own personal opinion, the perspective of the home organization that they represent, and the collective view of the network (Head 2008 ). The challenge for the network participant is to reconcile them. The home organizations often have a similarly daunting task. They must square the professional opinions of the bureaucratic and professional staff with those of the legislative body and their constituency.

The home organizations ’ constituencies are also interesting because of the multiplicity of their perspectives. Some of these stakeholders are direct recipients of the collaboratively delivered service or are directly impacted by the policy derived from the collaborative effort; others are not. Those who are not are still indirectly impacted through the opportunity cost associated with their government ’ s expenditure of resources as part of the network. Even within these two groups, there are certainly differences of opinion regarding what effectiveness means and how it should be measured. All in all, this makes for a very complicated landscape within which the collaborative public manager must select appropriate effectiveness measures to evaluate the network ’ s efforts.

Finally, not only do the individual collaborators and their home organizations need to be concerned with their own constituents, but also they must be cognizant of the interests of the constituencies of the other partners in the collaborative. Whether participation in the collaborative is voluntary or not, individual and home organization effort is. If the needs of the individuals, home organizations, and their constituencies are not met, it is very likely that they will loaf, thereby decreasing the effectiveness of the collaborative as a whole. Thus, it is in the best interest of all collaborators to serve not only their own interests but also those of the others with whom they are collaborating (Agranoff 2007 ).

474 Public Administration Review • May | June 2018

Stakeholder issues arise in the Waardenburg et al. case study. In it, they describe a number of these stakeholders, including the police and prosecutors. The authors note that while many of the performance metrics changed as a result of the government ’ s collaborative effort, the indicators for police and prosecutors did not. Further, they comment that it was unclear whether the number of prosecutions was because of or in spite of the network. While this is certainly true, prosecution numbers are an important metric for law enforcement and attorneys. That is how they are evaluated by their home organization and within their profession. The network must take this into account. Because these metrics are important to the legal and law enforcement communities, they also need to be important to the other members of the network. While prosecutions and prosecution rates may not constitute an ideal measure of network effectiveness, the network and the other network participants must realize that these metrics are critically important for their counterparts. Thus, metrics like these are necessary but not sufficient. They are an essential measure for the police and prosecutors for professional reasons but not a solely adequate measure for assessing the collaborative effort.

What to Measure? This raises the question of what to measure. In essence, this is the very heart of evaluating effectiveness. Koppenjan argues that “the effectiveness of collaboration should be based on the outcomes of processes in the relation to costs and benefits that are created for the parties involved” (2008, 701). The costs of collaboration, which include the expenditure of resources, such as time, manpower, money, etc., are often significant—so much so that Huxham and Vangen ( 2005 ) caution both practitioners and policy makers not to collaborate unless they have to because it is so resource intensive. However, the benefits may be just as noteworthy. These benefits are specific to and realized by entities at three levels of network operations: operating, organizational, and environmental (Mandell and Keast 2008 ).

The operating level refers to the interactions between network participants. “Effectiveness at this level is therefore determined by the extent to which participants have developed not only a better understanding of each other, but whether they have developed a shared language and culture, new ways of communication and the ability to find common ground” (Mandell and Keast 2008 , 722). The network itself makes up the organizational level. Here, effectiveness is seen in terms of activating, framing, mobilizing, and synthesizing activities (Agranoff and McGuire 2001 ; McGuire 2002 ; McGuire and Silvia 2009 ), such as the creation of a shared vision, the establishment of member commitment to the network ’ s mission, and the inclusion of all network members in the collaborative process (Mandell and Keast 2008 ). Both effectiveness at the operating and organizational levels focus on process and relationships. “However, such a focus on process is often dismissed by politicians as ‘bureaucratic’ and time-consuming, since politicians are inherently impatient for results” (Head 2008 , 739).

Politicians are much more concerned with the environmental level, which includes all of the network ’ s external stakeholders. Effectiveness at this level involves two distinct but related facets. The most obvious is the network ’ s ability to successfully meet the needs of these stakeholders and constituencies. The more this

happens, the more these external individuals and entities will support the ongoing work of the network. Thus, external support is the second measure of network effectiveness at the environmental level.

The effectiveness metrics such as citizen awareness, prosecutions, confiscated assets, and the outcome proxies discussed in Waardenburg et al.’s case study focus strictly on environmental level success. There is no mention in their essay of measures of effectiveness at the other levels of analysis. This is understandable since public managers are under pressure to show results to their stakeholders in order to garner and/or maintain support for the network. But by concentrating on only one level of analysis, this collaborative effort is potentially missing out on some of the important information that can be ascertained by analyzing all three levels.

Importantly, Mandell and Keast ( 2008 ) note that although the three levels of analysis are distinct, they are interrelated. If the effectiveness at only one level of analysis is considered, all deficiencies in effectiveness will be attributed to problems at that level. In the case of the Dutch anti–human trafficking network, the authors note that the number of prosecutions, an environmental effectiveness metric, remained constant despite the collaborative efforts. This led them to question whether the collaboration was effective.

Solely assessing effectiveness at one level risks problem misattribution. What if the plans and actions of the network were not the cause of the perceived problem? What if the root of the problem resided not at the environmental level but at the organizational level? If consensus regarding the mission of the network is not achieved and a shared vision of the network is not established, then the network may not be able to function properly. “Practically speaking, spending time on framing behaviors in which the manager is focused on gaining consensus on formulating the goals of the network diverts time and effort from behaviors intended to achieve the network ’ s goals effectively. If the network is concentrating on determining what needs to be done to be effective, it is more difficult to be effective at doing it” (McGuire and Silvia 2009 , 57). Thus, the root cause of the problem may not be with the steps the network is trying to take to solve the problem. Rather, the problem could be that the network cannot function effectively enough to work together successfully.

In addition, failing to assess effectiveness at all three levels of network operation ignores the full gamut of benefits of collaborative endeavors. Benefits other than environmental-level outcomes should also be considered. If effectiveness is assessed using cost–benefit analysis, overlooking the benefits ensuing from collaboration at the operational level may result in the network being perceived as ineffective because it was determined that the benefits do not outweigh the costs. Unfortunately, these benefits are commonly ignored in both research and practice. As Agranoff posits, “perhaps the most overlooked dimension of the literature relates to those benefits that accrue to the boundary-spanning individuals who represent organizations in networks” (2007, 157). In the course of his research, Agranoff identified a number of benefits for the collaborator, including increased technical knowledge, learning

Evaluating Collaboration: The Solution to One Problem Often Causes Another 475

how to work with different disciplines and in different organization cultures, expanding their managerial skills, professional networking, and broadening their awareness of different perspectives on the issue (Agranoff 2007 ). Benefits are not only for the individual collaborators while working as part of the network but also while working in their full-time position in their home agency.

Further, the home agency may benefit from their employee ’ s collaborative efforts through the relationships forged in the network. The establishment of working relationships as a result of one collaborative endeavor may pay dividends in the future when a collaborative approach is needed for another problem or policy area by decreasing transaction costs. Scholars have found that working with familiar individuals increases productivity and decision-making effectiveness (Sawyer 2007 ). For one, familiarity decreases the search costs for future collaborative partners. Familiarity also decreases the transaction costs associated with framing. “Members who are familiar with one another and with their work context are able to settle in and focus on the work rather than waste time and energy getting oriented to new coworkers or circumstances” (Hackman 2011 , 62). Finally, past experience is the best predictor of future performance. Therefore, familiarity has been found to engender trust between partners because they are better able to predict each other ’ s actions due to their past experiences working together (Gulati and Sytch 2008 ).

Page et al. ( 2015 ) offer a different perspective on collaborative assessment. Their framework, which focuses on the creation of public value, involves three main dimensions: democratic accountability, procedural legitimacy, and substantive outcomes. Their conceptualization of democratic accountability includes two facets: responsiveness and accountability to authorizers and mandates (vertical democratic accountability) and to collaborative partners and external stakeholders (horizontal democratic accountability). Collaborative decision-making and management procedures can also produce public value.

Page et al.’s ( 2015 ) notion of procedural legitimacy is composed of three components: procedural rationality, procedural justice, and operational control. Procedural rationality is the “extent to which decisions are based on technically and administratively sound data, analysis, and planning” (8). Public value is also created by a collaborative though procedural justice, which is when internal and external stakeholders view the decision-making process as being fair and transparent, and through operational control, which is when “requirements, budgets, and schedules [are used] to oversee projects and activities” (8).

Page et al.’s ( 2015 ) final dimension, substantive outcomes, includes goal achievement (effective performance), goal achievement relative to cost (efficiency performance), equitable distribution of costs and benefits (equity of benefits and equity of payment), and the collaborative partners ’ increase in problem-solving potential (problem- solving capacity). Importantly, the authors note that there are numerous trade-offs between and within attributes that complicate judgments regarding whether public value has been created.

When discussing the Dutch collaborative ’ s metrics, Waardenburg et al. state that the scope of the problem, and therefore the metrics

used to track progress, had to be appropriately limited. As a result, the authors focus on what Page et al. ( 2015 ) would term vertical accountability and effective performance in that they describe the Dutch collaborative as using goal achievement oriented results to justify the resources being spent on the collaborative. It is understandable that limits have to be placed on the assessment strategy. Page et al.’s ( 2015 ) framework is admittedly complex, and tough decisions have to be made by evaluators of the collaborative. However, its power is in its comprehensiveness. Again, focusing on some aspects at the expense of ignoring others may lead to faulty conclusions regarding the effectiveness of the collaborative. In the case of the Dutch network, performance effectiveness was a focus, but equity was ignored. Thus, the network could be seen as being effective, but the cost, benefits, or both may not have been equitably distributed across the stakeholders, thereby decreasing the creation of public value for all parties.

When to Defi ne the Metrics and When to Measure Them? Performance metrics are formulated prior to engaging in the collaborative (ex ante) or after collaborative activities have begun (ex post). Many scholars contend that objectives established ex ante are problematic because the act of collaborating with others often results in the interactive adaptation of perceptions and performance goals (Agranoff 2007 ; Klijn and Koppenjan 2000 ; Koppenjan 2008 ). “It is suggested that determining the effectiveness of collaboration should be based on the outcomes of processes in relation to costs and benefits that are created for the parties involved” (Koppenjan 2008 , 701). Therefore, ex ante standards may not reflect what the collaborative ultimately intends to or actually does achieve. Conversely, an ex post evaluation approach allows the goals to change as the network evolves (Vento 2017 ). In other words, ex post criteria are preferred because they result from the collaborative dynamics that occur within the network. However, waiting until after the collaboration has started to identify objectives is often difficult for home agency administrators, who like to know what the network intends to do before committing resources.

In the Dutch case study, it seems that the network successfully wended its way through this issue. Initially, it set the ex ante goals of erecting barriers for the facilitators of human trafficking. However, as a result of its collaborative efforts, jointly conceived ex post objectives, such as increased stakeholder awareness, were successfully instituted.

Another evaluation challenge that scholars and practitioners face is the question regarding when to assess the effectiveness of a collaborative. Networks have life cycles, and effectiveness indicators will likely be different at each phase of the cycle. Early in the collaborative life cycle, network members are engaging in both activation and framing activities. These activities lay an important foundation for the later stages of collaborative development, when the tasks for which the collaborative convened occur (Mandell and Keast 2008 ; McGuire and Silvia 2009 ). Therefore, predominantly organizational- and operational-level effectiveness measures are appropriate at the early stages of the collaboration. McGuire and Silvia ( 2009 ) point out that activation and framing activities are not supplanted by mobilization and synthesizing activities as the network matures. Rather, all four activities occur simultaneously in a mature network. Thus,

476 Public Administration Review • May | June 2018

environmental-level effectiveness measures, along with those at the operational and organizational levels, should be used.

How to Measure Effectiveness, Collect Data, and Conduct the Analysis The issue of effectiveness measurement is absolutely critical to the evaluation process. As Isett, Head, and VanLandingham ( 2016 ) state, the “gold standard” of randomized controlled trials (RCT) is generally not feasible for many social science research contexts. This is certainly the case with many networks because of their idiosyncrasies, small sample sizes, and the sensitivity of their policy contexts. Even if it were possible to study a network using an RCT, it is hard to imagine a design that could encompass all of the different concepts and facets of collaborative performance described here. Therefore, “best available” techniques must be employed.

In both research and practice, data are most often collected from data repositories and/or through surveys, interviews, or action research observations. It will likely take a combination of more than one of these methods or data sources to achieve a comprehensive evaluation. A great example of this comes from the case of the Dutch collaborative. One of the most interesting aspects of Waardenburg et al.’s case study is their discussion of the data that the Dutch network used. Instead of relying on the obvious measures, the collaborators used innovative proxy measures of effectiveness coupled with the compilation of data from diverse sources. This demonstrates not only the collaborators ’ ingenuity but also the power of collaborative approaches to public problem solving. Without the network, these data and measures would not be possible. Thus, although not used as such, the combination of this data is evidence of what Page et al. ( 2015 ) would call problem-solving capacity because it helps the Dutch network identify other types of criminal activity as well.

Once collected, the data are generally analyzed using traditional statistical techniques. Sometimes these familiar approaches constitute best available techniques. Sometimes they do not. Often, the decision regarding the analytic technique to use is based upon the methods with which the practitioner or researcher is most comfortable. But comfort is not a rationale for an appropriate analytic approach. Instead, the research question should determine the method. The field of public administration, led by the academic community, needs to expand the analytic toolkit by either exploring the use of techniques commonly used in other disciplines or, as Gill and Meier ( 2000 ) suggest, create specific analytic techniques to answer public administration-specific questions like collaborative effectiveness.

Summary The myriad stakeholders and the diversity of their perspectives on the network make evaluating collaborative effectiveness complex, just like the problems that collaboration is intended to address. The complexity common to collaboration and its evaluation is therefore not surprising, and probably entirely appropriate. The home organization wants evidence to justify their decision to expend resources in support of the network. The constituencies of the collaborating organizations want to see that their funds are being used wisely. And those who are directly being served by the network want their situation to be improved as a result of the collaborative endeavor. Collaborative public managers operate within this environment. They are responsible for providing evidence that the

network within which they work is effective for their numerous stakeholders. Thus, a holistic approach for demonstrating effectiveness must be used, including the following:

• Clearly establish the network ’ s goals and what it will be accountable for achieving. Also, identify each collaborator ’ s part in achieving the goal, thereby providing a mechanism for holding individual network members accountable (Thomson and Perry 2006 ). An example of this can be seen in Linden ’ s ( 2010 ) description of the Jamestown, New York, Strategic Planning and Partnership Commission ’ s (SPPC) quarterly report card. The SPPC published the report card in the newspaper, posted it on the city ’ s website, and included it with residents ’ utility bills. It showed the SPPC ’ s progress toward each of their priority items and briefly described what the collaborative had done to achieve that status. This approach not only helped keep the SPPC ’ s membership accountable for achieving their goals but also served to keep the SPPC ’ s constituency apprised of the successes the collaborative was having. • Construct an evaluation strategy that includes metrics that are particularly important to each of the collaborative partners and their home organizations. This will help keep each collaborator actively involved and committed to the network because the network is assessing effectiveness by measuring things that are important to them. “In a network setting the objective of only one of the parties in the network will not do as a measure to determine effectiveness” (Koppenjan 2008 , 11). Rather, Koppenjan argues that the opinions of other participants and stakeholders must be considered. This is highlighted in his discussion of Park Woods Ghent. Initially, the goal was to create a public woods. If that remained the sole objective, the farmers in the area would have been disenfranchised. Therefore, a compromise was reached. “By broadening the scope of the project, it became possible to entwine different objectives” (11). • Establish an evaluation strategy including concepts based on the framework developed by Mandell and Keast ( 2008 ). Whereas external pressures generally focus solely on assessment at the environmental level, using metrics that assess effectiveness at the various levels of network operations will assist in ensuring that the varied interests of the network, network members, and external stakeholders are accounted for and will safeguard against the misattribution of collaborative success or failure. In their study of a disaster response network in Léogâne, Haiti, following the 2010 earthquake, Nolte and Boenigk ( 2011 ) interview network participants in order to identify indicators of collaborative effectiveness. This list includes operating-level outcome measures, such as the increase in the strength of the partnership relationships; organizational-level outcome measures, such as commitment to network goals; and environmental- level outcome measures, such as strengthening the affected community. The respondents ’ inclusion and discussion of outcome measures at all three levels of analysis demonstrate that positive environmental level outcomes are not the only significant benefits of collaboration. • Expand the definition of effectiveness to include the generation of the different forms of public value described by Page et al. ( 2015 ). By doing this, the network can demonstrate the good that it is achieving. This idea is illustrated in the discussion of the Goodna Service Integration Project (SIP) in Mandell and Keast

Evaluating Collaboration: The Solution to One Problem Often Causes Another 477

( 2007 ). In that case, the performance of the collaborative focused not only on improving the provision of government services but also on building better relationships among the partnering agencies and the community. Although the SIP successfully improved service delivery in the community, “for many, the dense and embedded relationships that have formed and the new forms of engagement that are now built into the culture and psyche of the region that are based on this collaborative effort represent the most important outcomes of the project because they provide a basis of sustained commitment to enable this community to mobilize and act when necessary” (Mandell and Keast 2007 , 584–85, citing Woolcock and Boorman 2003). • Resist the temptation to define effectiveness ex ante. Such measures often fail to account for the complexities, interdependencies, and dynamics of the collaborative environment (Koppenjan 2008 ). Prematurely determining the measures of collaborative effectiveness ignores the negotiated agreements made by the partners as they collaborate. They can also artificially constrain the collaborative’s realm of possibilities. Since networks will seek to achieve what is measured, ex ante measures constrain the actions and direction of the collaborative. Conversely, an ex post approach for determining the effectiveness measures allows time for the partners to learn from each other and determine what the performance indicators ought to be (Bryson, Crosby, and Stone 2015 ; Koppenjan 2008 ). In his discussion of the Sacramento Water Forum, Koppenjan ( 2008 ) comments that the collaborators were unable to agree upon an ex ante collaborative assessment strategy because they could not foresee the eventual outcome of their joint efforts. Conversely, they were able to agree ex post because they were able to more clearly see the direction of their joint action. • Follow the example of the Dutch network described in Waardenburg et al. and find innovative metrics to indicate effectiveness. Direct measures often do not exist, so proxies must be used. Brainstorming among network partners who have different perspectives and access to different information can facilitate this. This is one of the strengths of collaboration.

Public managers need not tackle the thorny issue of evaluating collaboratives alone. Rather, the academic and practitioner communities must work together to learn how to better assess collaborative effectiveness. Practitioners need to open up their networks and data sets to scholars for study. Scholars, on the other hand, need to study questions that are both theoretically interesting and practically important. These questions could include the following:

• What are the most effective ways to keep collaborative partners and the network as a whole accountable? • What measures best capture effectiveness at the various stages of the network life cycle? • How do collaborative participants negotiate differences between their perspective and those of their fellow collaborators and those of their home agency? • Given time and manpower constraints, what facets of collaborative effectiveness and value creation should public managers focus on? • What analytic techniques and strategies are most appropriate for determining network effectiveness?

The complexity of the network effectiveness question is one that requires a collaborative approach. It is an issue that both practitioners and scholars need to work on together because it cannot be solved, or easily solved, by either group alone.

References Agranoff , Robert . 2007 . Managing within Networks: Adding Value to Public

Organizations . Washington, DC : Georgetown University Press . Agranoff , Robert , and Michael McGuire . 2001 . Big Questions in Public Network

Management Research . Journal of Public Administration Research and Theory 11 ( 3 ): 295 − 326 .

Bovens , Mark , Thomas Schillemans , and Paul ’t Hart . 2008 . Does Public Accountability Work? An Assessment Tool . Public Administration 86 ( 1 ): 225 −42 .

Bryson , John M. , Fran Ackermann , and Colin Eden . 2016 . Discovering Collaborative Advantage: The Contributions of Goal Categories and Visual Strategy Mapping . Public Administration Review 76 ( 6 ): 912 − 25 .

Bryson , John M. , Barbara C. Crosby , and Melissa Middleton Stone . 2006 . The Design and Implementation of Cross-Sector Collaborations: Propositions from the Literature . Special issue, Public Administration Review 66 : 44 – 55 .

_____. 2015 . Designing and Implementing Cross-Sector Collaborations: Needed and Challenging . Public Administration Review 75 ( 5 ): 647 – 63 .

Dickinson , Helen , and Jon Glasby . 2010 . “Why Partnership Working Doesn ’ t Work”: Pitfalls, Problems and Possibilities in English Health and Social Care . Public Management Review 12 ( 6 ): 811 − 28 .

DiMaggio , Paul J. , and Walter W. Powell . 1983 . The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields . American Sociological Review 48 ( 2 ): 147 − 60 .

Fitzpatrick , Tony . 2008 . Applied Ethics and Social Problems: Moral Questions of Birth, Society and Death . Chicago : University of Chicago Press .

Gill , Jeff , and Kenneth J. Meier . 2000 . Public Administration Research and Practice: A Methodological Manifesto . Journal of Public Administration Research and Theory 10 ( 1 ): 157 − 99 .

Gulati , Ranjay , and Maxim Sytch . 2008 . Does Familiarity Breed Trust? Revisiting the Antecedents of Trust . Managerial and Decision Economics 29 ( 2−3 ): 165 − 90 .

Hackman , J. Richard . 2011 . Collaborative Intelligence: Using Teams to Solve Hard Problems . San Francisco : Berrett-Keohler .

Head , Brian W. 2008 . Assessing Network-Based Collaborations . Public Management Review 10 ( 6 ): 733 – 49 .

Huxham , Chris , and Siv Vangen . 2005 . Managing to Collaborate: The Theory and Practice of Collaborative Advantage . New York : Routledge .

Isett , Kimberley R. , Brian W. Head , and Gary VanLandingham . 2016 . Caveat Emptor: What Do We Know about Public Administration Evidence and How Do We Know It? Public Administration Review 76 ( 1 ): 20 − 23 .

Klijn , ErikHans , and Joop F. M. Koppenjan . 2000 . Public Management and Policy Networks: Foundations of a Network Approach to Governance . Public Management 2 ( 2 ): 135 −58 .

Koontz , Tomas M. , and Craig W. Thomas . 2006 . What Do We Know and Need to Know about the Environmental Outcomes of Collaborative Management? Special issue, Public Administration Review 66 : 111 – 21 .

Koppenjan , Joop . 2008 . Creating a Playing Field for Assessing the Effectiveness of Network Collaboration by Performance Measures . Public Management Review 10 ( 6 ): 699 − 714 .

Lewis , Jenny M. , Janine O ’ Flynn , and Helen Sullivan . 2014 . Accountability: To Whom, in Relation to What, and Why? Australian Journal of Public Administration 73 ( 4 ): 401 − 7 .

Linden , Russell M. 2010 . Leading across Boundaries: Creating Collaborative Agencies in a Networked World . San Fransisco : Jossey-Bass .

478 Public Administration Review • May | June 2018

Mandell , Myrna P. , and Robyn Keast . 2007 . Evaluating Network Arrangements: Toward Revised Performance Measures . Public Performance and Management Review 30 ( 4 ): 574 – 97 .

_____. 2008 . Evaluating the Effectiveness of Interorganizational Relations through Networks . Public Management Review 10 ( 6 ): 715 − 31 .

McGuire , Michael . 2002 . Managing Networks: Propositions on What Managers Do and Why They Do It . Public Administration Review 62 ( 5 ): 599 − 609 .

McGuire , Michael , and Chris Silvia . 2009 . Does Leadership in Networks Matter? Examining the Effect of Leadership Behaviors on Managers ’ Perceptions of Network Effectiveness . Public Performance & Management Review 33 ( 1 ): 34 − 62 .

Nolte , Isabella M. , and Silke Boenigk . 2011 . Public-Nonprofit Partnership Performance in a Disaster Context: The Case of Haiti . Public Administration 89 ( 4 ): 1385 – 1402 .

O ’ Leary , Rosemary , Yujin Choi , and Catherine M. Gerard . 2012 . The Skill Set of the Successful Collaborator . Special issue, Public Administration Review 72 : 70 − 83 .

Ostrower , Francie . 2005 . The Reality Underneath the Buss of Partnerships: The Potentials and Pitfalls of Partnering. Stanford Social Innovation Review, Spring. https://ssir.org/articles/entry/the_reality_underneath_the_buzz_of_partnerships [accessed September 20, 2017].

Page , Stephen B. , Melissa M. Stone , John M. Bryson , and Barbara C. Crosby . 2015 . Public Value Creation by Cross-Sector Collaborations: A Framework and Challenges of Assessment . Public Administration 93 ( 3 ): 715 − 32 .

Sawyer , Keith . 2007 . Group Genius: The Creative Power of Collaboration . New York : Basic Books .

Skelcher , Chris , and Helen Sullivan . 2008 . Theory-Driven Approaches to Analysing Collaborative Performance . Public Management Review 10 ( 6 ): 751 − 71 .

Thomson , Ann Marie , and James L. Perry . 2006 . Collaboration Processes: Inside the Black Box . Special issue, Public Administration Review 66 : 20 – 32 .

Vento , Isak . 2017 . On Evaluation of Innovations: How to Evaluate the Transformative Value of Cohesion Policy for Managing Purposes? Scandinavian Journal of Public Administration 21 ( 2 ): 3 − 22 .