Week2part3

By Elliott S. Fisher, Stephen M. Shortell, Sara A. Kreindler, Aricca D. Van Citters, and Bridget K. Larson

A Framework For Evaluating The Formation, Implementation, And Performance Of Accountable Care Organizations

ABSTRACT The implementation of accountable care organizations (ACOs), a new health care payment and delivery model designed to improve care and lower costs, is proceeding rapidly. We build on our experience tracking early ACOs to identify the major factors—such as contract characteristics; structure, capabilities, and activities; and local context— that would be likely to influence ACO formation, implementation, and performance. We then propose how an ACO evaluation program could be structured to guide policy makers and payers in improving the design of ACO contracts, while providing insights for providers on approaches to care transformation that are most likely to be successful in different contexts. We also propose key activities to support evaluation of ACOs in the near term, including tracking their formation, developing a set of performance measures across all ACOs and payers, aggregating those performance data, conducting qualitative and quantitative research, and coordinating different evaluation activities.

T he implementation of accountable care organizations (ACOs), a new payment and delivery model de- signed to improve health care and lower costs, is proceeding rapidly

in both the public and private sectors. As of August 2012 we had identified 227 provider or- ganizations that have established ACO contracts with Medicare, Medicaid, private payers, or some combination thereof. The ACO concept originated in response to a

growing recognition that fee-for-service pay- ment was a major contributor to the rapidly ris- ing costs and poorly coordinated care that char- acterize the US health care system.1 Under this new payment model, provider groups willing to be accountable for the overall costs and quality of care for their patients are eligible for a share of the savings achieved by improving care. Proponents believe that ACOs will encourage

providers across the full range of practice set- tings—from individual office-based practices to

integrated delivery systems—to improve quality and slow spending growth. Under this model, payers establish quality benchmarks and risk- adjusted spending targets for the patients cared for by the physicians in the ACO. If the organi- zation meets the quality benchmarks, it is then eligible for a share of the savings achieved below the set spending target. In some models, the organization is also at risk for a portion of any spending that exceeds the target. Early evidence on ACO performance is promising.2–4

Challenges to the success of the model remain, however. Little is known about what capabilities and activities are most important to the long- term success of these new organizations. Also, the optimal design of accountable care contracts between providers and payers is uncertain. In addition, many stakeholders are concerned

about the complex interactions among public and private reform initiatives based on ACOs. For example, some economists wonder whether implementation of ACOs in the Medicare popu-

doi: 10.1377/hlthaff.2012.0544 HEALTH AFFAIRS 31, NO. 11 (2012): 2368–2378 ©2012 Project HOPE— The People-to-People Health Foundation, Inc.

Elliott S. Fisher (Elliott.S.Fisher@ dartmouth.edu) is director of the Center for Population Health at the Dartmouth Institute for Health Policy and Clinical Practice, in Hanover, New Hampshire.

Stephen M. Shortell is dean of the University of California, Berkeley, School of Public Health.

Sara A. Kreindler is a researcher with the Winnipeg Regional Health Authority, in Manitoba, Canada.

Aricca D. Van Citters is a researcher with the Dartmouth Institute.

Bridget K. Larson is the former director of health policy implementation at the Dartmouth Institute.

2368 Health Affairs November 2012 31:11

Accountable Care Organizations

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

lation will lead to provider consolidation and thus higher prices for private payers. Some pol- icy makers and providers are concerned that the financial incentives might not be large enough to motivate the substantial changes required to im- prove care and worry that the barriers to change will be too great to achieve hoped-for quality improvement and cost savings.5

Ongoing and systematic evaluation of the im- plementation of these new payment models is essential to providing useful guidance to these organizations about how best to invest to trans- form care.6 Additionally, systematic evaluation of the impact of ACOs on both quality and costs— under the different types of contracts that are already being implemented—would allow policy makers and private payers to refine the design of these contracts. In this article we give a brief overview of the

current status of ACO implementation and evalu- ation initiatives; provide a framework for think- ing about how ACOs may or may not achieve their impact, in the form of a “logic model”; draw on insights from the logic model to identify bar- riers to successful evaluation; and suggest spe- cific approaches to overcoming these barriers.

Early ACO Implementation And Evaluation Activities Several early case studies have described the paths that organizations have taken to establish ACOs.7–9 The Robert Wood Johnson Foundation recently announced that it will fund additional case studies of nascent ACOs, with a particular interest in those involving disadvantaged popu- lations.10

The Commonwealth Fund has supported a variety of activities, including a policy analysis of the challenges for vulnerable populations under the accountable care model,11 a survey of hospitals’ ACO activities,12 an ongoing evalu- ation of the Alternative Quality Contract in Massachusetts,3,4 and work by Dartmouth and the University of California, Berkeley,13 to iden- tify the challenges to implementing a compre- hensive approach to evaluation and find ways to address them. There is some evidence that the ACO payment

model will be successful in lowering costs and improving quality. For example, under the Blue Cross Blue Shield of Massachusetts Alternative Quality Contract, a payment model that allows provider groups in Massachusetts to retain the savings they achieve while receiving additional bonuses for documented quality improvements, providers have been able to make substantial quality improvements and achieve some savings.3,4

Similarly, an analysis of the Medicare Physi- cian Group Practice Demonstration, a test of the ACO model carried out from 2005 to 2010, re- vealedmodestsavings forMedicarebeneficiaries overall, but substantial savings for the highly vulnerable and high-cost “dual eligible” popula- tion who are beneficiaries of both Medicare and Medicaid.2

Public-Sector Programs In trying to learn from ACO implementation, it is important to understand current federal programs, because these are the largest in scope and will have a major influence on the future design of private- payer models. The Centers for Medicare and Medicaid Services (CMS) has three distinct ACO programs under way: the Medicare Shared Savings Program, the PioneerACO program, and the Advance Payment ACO program. The Medicare Shared Savings Program is a

national ACO program, not a demonstration or pilot project, established under the Affordable Care Act. The program—the largest of the three, with 115 provider organizations currently en- rolled—offers two incentive options: one in which ACOs obtain bonus payments if their costs are below their spending target, with no penal- ties if costs exceed the target (no risk); and a second option that offers greater bonuses but requires ACOs to pay a portion of costs that exceed spending targets (risk bearing). The Pioneer ACO program is a demonstration

project involving thirty-two organizations that are required to bear at least some degree of risk for costs that exceed their spending targets; the program has six slightly different financial in- centive designs. Finally, the Advance Payment ACO program is

a demonstration project that provides some up- front federal funding to help 20 smaller and poorly capitalized organizations—a subset of the 115 Medicare Shared Savings Program sites— launch an ACO. The Center for Medicare and Medicaid Innovation is required to evaluate only the two demonstration programs because, under the Affordable Care Act, the secretary of health and human services may extend these currently experimental payment models nationally if a rig- orous evaluation, which meets the standards of the CMS Office of the Actuary, shows that they have been effective. Private-Sector Efforts In addition to these

public-sector efforts, private-sector ACO imple- mentation is proceeding apace. Most of the ma- jor payers have ACO or ACO-like contracting ini- tiatives under way. Of the 227 organizations that we have identified with an ACO contract, 51 have only private-payer contracts (17 of which have contracts with more than one private payer), and 29 have both a private- and a public-payer ACO

November 2012 31:11 Health Affairs 2369

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

contract. Other than the Commonwealth Fund and Robert Wood Johnson Foundation initia- tives described above, however, we know of no formal efforts to evaluate the impact of these private-payer ACO programs.

A Framework For Evaluation: An ACO Logic Model During the past two years we have collaborated with researchers and stakeholders to explore how best to advance learning in this rapidly changing policy and practice environment. Our work was rooted in the principles of real-

istic evaluation. This approach explores average effects (for example, what the CMS actuary would need to know to conclude that, on aver- age, the Pioneer model led to reduced costs and improved care). It also evaluates what strategies and implementation activities pursued by payers and ACOs are most likely to lead to better per- formance, and how this varies across ACOs op- erating under different conditions and local contexts. A key step in this process of evaluating ACOs

was the development of a “logic model.” Such a model provides a graphical representation of how we believe that this approach to payment and delivery reform might achieve its intended effects, what those anticipated effects might be, and what local and national factors might influ- ence success. We developed and progressively revised the

logic model based on findings from the Brookings-Dartmouth ACO pilot sites,7,13 feed- back from national experts and ACO partici- pants, and insights from observing other efforts to evaluate delivery system reforms.3,4,14 The cur- rent version of the logic model is provided in the online Appendix.15 Exhibit 1 presents an abridged, schematic overview of the logic model with some specific examples of implementation activities, measurable outcomes, and associated potential impacts. Our first step in developing the model was to

be explicit about the impact those who proposed the ACO model hoped to achieve: better care and lower costs, not only for the patients served by the organization, but also for theircommunities, to ensure that ACOs do not achieve their benefi- cial impact on some patients at the expense of others.16

The model then distinguishes four major cat- egories of influences: the national, state, and local context within which ACOs are launched; the readiness of the ACOs and their payer part- ners to adopt the model, as well as the structures of the contracts themselves; the specific imple- mentation activities that ACOs and their payer

partners pursue; and the intermediate outcomes of those activities. As an example of the first type of influence,

national policy on health information technol- ogy and local health information exchanges could make it more likely that a given organiza- tion has a solid health information technology foundation at baseline. An example of the third set of influences would

include the strategic priorities that the ACO es- tablishes, such as setting quantitative targets for the adoption of diabetes registries across their primary care practices, and the specific clinical activities that they pursue as a result, such as aggressively treating people who have both dia- betes and poorly controlled blood pressure. Both the strategic priorities and specific clinical activ- ities undertaken would influence the degree to which organizations achieve changes in the pa- tient care processes required to measurably im- prove quality and lower costs.

Insights And Challenges Several key insights emerged from the develop- ment of the logic model and our review of the current status of ACO implementation and evaluation. Key Insights ▸DISTINGUISH FORMATION FROM OPERAT-

ING PERFORMANCE: First, it has become clear that distinguishing formation and implementa- tion activities from performance will be impor- tant. Successfully implementing an ACO—that is, signing a contract—and implementing a suc- cessful ACO are different things. The model clari- fies that these are distinct but overlapping processes. Capability development and implementation

For many organizations, performance could begin to improve simply through anticipation of a possible future ACO contract.

Accountable Care Organizations

2370 Health Affairs November 2012 31:11

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

activities in many of the organizations partici- pating in early accountable care initiatives began well before effective working relationships or contracts were established with payers. For many organizations, performance could begin to im- prove simply through anticipation of and prepa- ration for a possible future ACO contract—inde- pendent of whether they are yet willing to sign such a contract.

▸TRACK PROGRESS OVER TIME: Another key insight has to do with the rapid pace of change. Memories fade rapidly, so it is critical to track over time not only the characteristics of ACOs and their environments, but also the perceptions of their leaders about what goals were consid- ered important, what the leaders did to achieve these goals, what implementation activities lead- ers engaged in, and their motivation for doing so.

Populations Affected The logic model also clarifies the three distinct populations that could be affected by this new payment model: patients under the ACO contract; patients cared for by the organization but not covered by the contract; and, importantly, the community as a whole. The importance of understanding the impact on all three populations has implications for how one might track the impact of ACOs more broadly.

Evaluation Challenges This work also high-

lights four major challenges to ACO evaluation, including lack of clarity about what evaluators would need to know about each organization; the importance of measuring performance for those who are under an ACO contract and those who are not; the difficulty of coordinating evalu- ation activities and aligning measures across payers; and the need for efficient and standard- ized data collection approaches that can support empirical, quantitative analyses linking poten- tial influences to outcomes.We address each ma- jor challenge in the next section.

Moving Forward Key Domains: What Do We Need To Know About Each ACO? The logic model highlights the complex and dynamic nature of ACO imple- mentation. It also provides an aspirational view of the scope of information that would be useful. However, both political challenges and re-

source constraints will limit the scope of evalu- ative activity put in place.We therefore attempted to prioritize what information should be col- lected to support a sufficiently robust evaluation of the implementation of ACOs. We based our judgment on what would be important to collect now, what could not be collected by other means, or what data would require substantial lead time to organize.

Exhibit 1

Schematic Overview Of The Accountable Care Organization (ACO) Evaluation Logic Model

SOURCE Authors’ analysis. NOTES This is a schematic overview of the more complete model, provided in the online Appendix (Note 15 in text). IT is information technology.

November 2012 31:11 Health Affairs 2371

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

Based on our logic model, as well as a compre- hensive review of current ACO readiness assess- ment tools,17 the National Study of Physician Organizations survey instruments,18 and a hos- pital-focused ACO survey,19 we developed a list of potentially relevant domains and measures for ACO evaluation. Our survey instrument includes measures within each of these domains and is being implemented during the fall of 2012. Our planned measures fall into three broad

categories: ACO contract characteristics; ACO structure, capabilities, and activities; and local context. Exhibit 2 provides additional detail about the categories of information that we thought should be collected from each organiza- tion within each of these broad areas.20

Contract Characteristics At the heart of this new payment model is the contract itself. It will be important to assess the scope of local payers’ commitment to the accountable care model. We expect that providers’ motivation to change current practices will increase with the proportion of their patient populations being cared for under ACO contracts. In addition, the contracts themselves currently

differ along a number of important dimensions, including the thresholds established for quality and cost targets, arrangements for sharing sav- ings and allocating risk, mechanisms for linking quality to shared savings, definitions of the total cost of care, and allowances for modifications in contract terms. For example, many private-payer contracts allow annual modifications of the terms, while the Medicare contracts are gener- ally three years in duration. Subtle differences in these elements could have important implica-

tions for the ACOs and the ways in which they allocate resources toward caring for patients and improving care. The approach that payers and ACOs take to

patient engagement will also vary across payers and markets, as reflected in the structure of ACO contracts. Some benefit plans are likely to be restructured to encourage patients to choose a primary care physician within the ACO. Others may determine whether patients are cared for by a specific organization simply by tracking where they get their care (known as passive attri- bution). Some payers may offer financial incentives for

patients to participate in health risk appraisals or chronic disease management programs, while others may not. Interpreting apparent differences in ACO performance without under- standing such potential confounding influences will be difficult. Structure, Capabilities, And Activities Of

The ACO Provider organizations that hope to establish and succeed as an ACO are likely to be most interested in insights provided by an evaluation about the structures, capabilities, and activities of successful ACOs and how these compare to those of less successful ventures.We discuss each in turn. Provider organizations considering the

accountable care model begin their work at very different stages of development, ranging from those with no formal organizational structure that would make them eligible to participate in either a public- or a private-payer ACO program (that is, not yet having a defined legal entity with which a payer could contract) to fully integrated

Exhibit 2

What Should Be Known About Each Accountable Care Organization (ACO)?

ACO contract characteristics ACO structure, capabilities, and activities

Context within which ACO is forming and operating

Scope: which payers are involved; proportion of participating providers’ patients that are covered by contracts

Requirements: structures and processes required to be eligible to participate in the contract

Incentive design: how financial incentives are structured (for example, risk bearing) and how they are linked to performance

Patient engagement: how patients are assigned to, informed about (or not), and engaged as participants in the ACO

Structure, governance, leadership, and strategy of the ACO and its component organizations

Clarity of aims and motivation to develop an ACO, and quality of internal relationships

Physician and other clinician engagement strategies and processes

Health information technology capabilities, effectiveness, and implementation activities

Care management processes in primary care and across the care continuum

Quality improvement methods used, and their scope and extent of deployment

Local market structure and dynamics (for example, payer and provider concentration and market power)

Presence and focus of local multistakeholder initiatives in health care

State and local experience with public reporting and pay-for-performance

Current per capita spending and utilization (magnitude of opportunities for savings)

State policy environment (for example, percent uninsured, Medicaid payment levels)

SOURCE Authors’ analysis. NOTE The domains identified above were developed from a review of currently available ACO readiness assessment tools and provider surveys and the authors’ analyses of the local market factors likely to influence both formation and performance of ACOs.

Accountable Care Organizations

2372 Health Affairs November 2012 31:11

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

delivery systems that are already operating with risk-based contracts that hold them accountable for the total cost of care with one or more private payers. Provider groups also differ in the degree of

influence over the operation of the ACO exerted by physicians, versus hospitals and insurers. Almost every ACO leader with whom we have spoken highlights the importance of engaging physicians and other clinicians in the transition to this new model of care and payment. As another example, the scope, capabilities,

and effectiveness of health information technol- ogy, from registries to electronic health records, will influence both care provision and the inter- nal use of performance measurement. The extent and effectiveness of care management processes in primary care, in specialty care, and across the care continuum may strongly affect an organiza- tion’s ability to coordinate and improve care. Whether an ACO adopts a specific approach to quality improvement, such as the “Lean” ap- proach, could also make a difference. It will be important to assess such organiza-

tional capabilities both at baseline and over time to be able to analyze how they influence changes in performance. If possible, comparisons should be made to similar populations of patients re- ceiving care from providers who are not involved in accountable care arrangements. This type of comparison across different groups of patients would allow policy makers and payers to begin to assess the impact of this new approach to pay- ment compared to traditional fee-for-service payment. If ACOs are successful at improving care and

lowering costs, the challenge would then be to spread these models—and insights about how they achieved their impact—across the country,

taking into account local context. Local Context The local context, or environ-

ment, within which an ACO is forming and op- erating will be important to understand as a component of the evaluation. Examples of local context include the structure of the local market, the degree of market concentration, the extent of collaboration and competitionamongproviders, and the presence and focus of local multistake- holder initiatives. Local context also could include state and local

experience with delivery system payment re- forms such as pay-for-performance, bundled payments, and patient-centered medical homes; public reporting; current level of per capita spending and utilization, which could affect the likelihood of achieving shared savings; and state policy environment, such as the percentage of the population uninsured, generosity of the Medicaid program, and relevant regulations. Tracking Performance: What Do We Need

To Know? Without information on the actual performance of ACOs, we will not be able to judge whether this approach to payment reform is having its intended impact on the costs and quality of care. Exhibit 3 underscores the two critical dimensions of performance measure- ment: populations of interest, and processes and outcomes of care. In other words, what should be measured and for whom? Both payers and providers have recognized

that the provisions of an ACO contract with one payer could affect patients covered by other payers, in terms of both care patterns and prices. For instance, many observers are concerned that ACOs that contract with Medicare could take advantage of any market power they gain from consolidation, of either hospitals or physician groups, to shift costs to private payers by raising prices for the privately insured.21

In addition, any evaluation effort should track the impact of the accountable care model on subgroups of the population that may face greater risks, such as socioeconomically dis- advantaged populations or people who are cared for by safety-net providers. For example, ACOs taking care of a predominantly disadvantaged population would have a harder time meeting performance targets that reward a specific level of achievement, such as 60 percent of people with diabetes having excellent control of blood pressure, rather than the degree of improve- ment, such as improving the proportion of patients with well-controlled blood pressure by 10 percent. The former would be a harder target to achieve for an organization with a high pro- portion of patients with poorly controlled diabetes.11

Exhibit 3 also points to the importance of striv-

Collaboration between evaluators, policy leaders, and other key stakeholders will be essential to a timely, robust, and efficient approach to ACO evaluation.

November 2012 31:11 Health Affairs 2373

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

ing to measure both the processes of care—the major focus of current performance measures— and outcomes that may be more meaningful to the public, such as patients’ experiences, health outcomes, and the total cost of care. Medicare claims data are readily available and have already provided the foundation for some early ACO evaluation work. CMS will also have access to surveys of beneficiaries in ACOs and, for com- parison, non-ACO populations. Whether data from private payers will be made available for purposes of evaluation is less clear. Getting The Information We Need: Coordi-

nation And Data Collection We have identi- fied broad areas of activity needed to support timely evaluation of ACO implementation and performance (Exhibit 4). First, it will be critical to coordinate and align

evaluation initiatives. To accomplish this goal, it willbe necessary to establish and support a work- ing group of funders and evaluators to maximize the opportunity to learn, minimize avoidable redundancy, and reduce the burden imposed on payers and providers, who are already busy enough trying to improve care. The Center for Medicare and Medicaid Innovation or a founda- tion with a strong interest in accountable care could convene and organize this type of collabo- rative working group. Another necessary task will be to develop a

parsimonious and aligned set of performance measures across all ACOs and payers. Work is under way to align measures across federal per- formance measurement initiatives, but a parsi- monious set of measures for ACO contracts— public and private—should be a high priority. To collect the information needed, the follow-

ing distinct activities should be pursued: tracking ACO formation; conducting qualitative

research, such as case studies; conducting quan- titative research, such as surveys; and aggregat- ing performance data. Maintaining a census of payers’ current

accountable care initiatives, the structure of their contracts, and a list of participating pro- vider organizations would provide the informa- tion needed to identify organizations to partici- pate in case studies and to determine which organizations should complete needed surveys. Information on contract characteristics should be obtained directly from the payers. Case studies of early ACO implementation ex-

periences have helped—and should continue to help—to advance our understanding of what fac- tors are associated with success. But more com- prehensive and standardized surveys of emerg- ing ACOs will be necessary to ensure that the organizational attributes and local environmen- tal context influencing ACO formation and performance are collected in a sufficiently stand- ardized and structured format to support quan- titative analyses. Quantitative and qualitative research ap-

proaches are synergistic, and a standardized survey—such as the one we are currently imple- menting—would preclude neither the collection of more detailed data through site visits nor the targeted efforts of current learning networks. Finally, data on the actual performance of

ACOs will require comprehensive information that could be aggregated to the level of the ACO, stratified at least by contract type or payer, and to the level of the local health care market. Examining the impact of ACOs on the entire population of a local market will shed light on whether their overall effect benefits not just those covered by the contract, but also the com- munity as a whole. This examination should in- clude not only cost and basic performance data, which could be assessed from claims data, but it would ideally allow collection and integration of other quality measures. In the short term, the most practical approach

to obtaining these data would be to adopt a dis- tributed data model.22 Under such a model, payers would calculate basic measures of perfor- mance for their members—those covered by the ACO and those not covered—which could then be combined across payers for each provider or local market. As part of the Dartmouth Atlas Project, a pilot

project is under way to create community-based total-cost-of-care reports based on summary data prepared by the major payers.23 This project ap- pears to be technically possible, requiring rela- tively simple tabulations of spending claims data similar to what payers are already doing for the National Committee for Quality Assurance’s

Exhibit 3

Two Critical Dimensions Of Accountable Care Organization (ACO) Performance And Impact

Populations of interest Processes and outcomes of care Those directly and indirectly affected Patients covered by the ACO contract Other patients cared for by the ACO All residents of the community Communities not served by ACOs Subgroups most likely to benefit or be harmed High-risk or high-cost clinical populations

Socioeconomically vulnerable populations

Safety-net providers and their patients

Key process measures (accessibility, coordination, provision of evidence- based care)

Patients’ experience of care, including degree of activation, engagement in shared decision making, decision quality

Health outcomes, including health risk, disease status, functional outcomes

Per person costs (for patients) and per capita costs (for community)

SOURCE Authors’ analysis.

Accountable Care Organizations

2374 Health Affairs November 2012 31:11

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

Healthcare Effectiveness Data and Information Set reports. A distributed data model is also politically and

legally feasible because the aggregated data at the geographic level would protect individual payers’ pricing information. This distributed data model, long advocated by Mark McClellan of the Engelberg Center for Health Care Reform, could be extended to include other measures, such as quality. Eventually, the model could al- low aggregation of claims-based cost and quality measures at the ACO level.

Discussion And Policy Implications This analysis has highlighted the challenges of evaluating the emergence of accountable care organizations—a new delivery and payment model. ACO implementation is proceeding rap- idly in both the private and public sectors. Understanding interactions among private and public payment models will be essential to any serious evaluation. Stakeholders have different existing commit-

ments, statutory or fiduciary obligations, and core interests. Federal funding for research, al- ready limited, is likely to become more so. Congress has made some provisions to protect Medicare from the across-the-board cuts in- cluded in the sequestration legislation, but Na- tional Institutes of Health and Agency for Healthcare Research and Quality funding is at risk from the cuts to discretionary spending that

appear likely. And there is no guarantee that Medicare will be protected from cuts in the longer run. The unavoidable implication is that collabora-

tion between evaluators, policy leaders, and other key stakeholders will be essential to the successful implementation of a timely, robust, and efficient approach to ACO evaluation. Developing a strong evidence base to address

the questions that policy makers, payers, and providers have about ACOs will require a struc- tured evaluation effort that extends over many years.We propose that evaluation efforts address several key domains related to ACO formation and implementation: ACO contract characteris- tics; ACO structure, capabilities, and activities; and local context. Evaluation efforts should also track ACO performance. We also propose several important data collec-

tion strategies, including coordination and alignment of evaluation efforts; maintaining an ACO census; conducting qualitative evalua- tions, such as case studies, and quantitative eval- uations, such as surveys; and obtaining aggre- gate performance data at the ACO and local health market levels. Such data could support quasi-experimental studies such as the recent evaluation of the Physician Group Practice Demonstration.2

More data about the attributes of the ACOs and which programs they are participating in would support diverse and valuable studies, such as comparisons between the Medicare Shared Sav-

Exhibit 4

Suggestions For Collecting Information Needed For Accountable Care Organization (ACO) Evaluation

Aim Activity Product

Coordinate and align formative, process, and summative evaluation initiatives to maximize learning and minimize redundancy and respondent burden

Establish and support working group to track current evaluation activities and suggest opportunities for alignment

Define common set of core ACO performance measures that all payers would be encouraged to adopt

Inventory of existing ACO evaluation tools and projects and recommendations on how gaps might be filled

Technical specifications for core set of measures that could be adopted by private payers in their ACO contracts

Collect data needed to refine the logic model; to characterize ACOs in terms of contract characteristics, structure, capabilities, activities; and local context; and to evaluate the actual performance of ACOs

Track payers implementing ACOs, the structure of their contracts, and a list of participating ACOs

Conduct qualitative research needed to advance understanding of ACO formation and refine the logic model

Survey emerging ACOs on a regular basis to ascertain current ACO structures, capabilities, and local contextual factors

Support aggregation of payer data to track impact of ACOs on their covered populations and communities

Structure of ACO contracts and list of ACOs that would provide sample frame for surveys and case studies

Case studies and cross-case analyses critical to early understanding (before performance data are available)

Data required to link ACO structures, capabilities, and local contextual factors to performance

Data required to understand actual impact of ACOs on quality and costs of care

SOURCE Authors’ analysis. NOTES Collecting the above data would provide a foundation for quantitative analyses of the impact of ACO formation and the key factors that predict better or worse performance. The information would also help identify ACOs in which more detailed qualitative case studies could be carried out.

November 2012 31:11 Health Affairs 2375

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

ings Program and the Pioneer ACO program, between ACOs with registries and those without, and between organizations that are led by physi- cians and those led by hospitals. We could then learn not only whether ACOs, on average, do better than other payment models, but also when, where, and how ACOs form and achieve their impact. A balanced scorecard of comparative perfor-

mance indicators could be developed that would allow CMS and other payers to track ACO per- formance and begin to learn what changes in incentives or payment arrangements would be most likely to promote more cost-effective, co- ordinatedcare. ACOs themselvescould alsolearn

what activities are likely to be most important in their pursuit of the same goal. The barriers to organizing and implementing

such an effort are real. The alternative, however, would be to allow this new payment model to proceed without the kind of timely feedback re- quired to assist providers, payers, and policy makers in implementing the needed reforms. A comprehensive measurement infrastructure would also ensure that ACOs were held account- able not only for the care of the patients under the ACO contract, but also for their impact on the patients of nonparticipating payers and on their communities as a whole. ▪

The authors acknowledge the Commonwealth Fund for its support of this work. This work was informed by members of an accountable care organization (ACO) evaluation working group, which included the following organizations: America’s Health Insurance Plans; American Medical Group Association; Bailit Health Purchasing; Catalyst for Payment

Reform; Centers for Medicare and Medicaid Services; Commonwealth Fund; Dartmouth Institute for Health Policy and Clinical Practice; Engelberg Center for Health Care Reform at the Brookings Institution; Health Research and Educational Trust; Institute for Healthcare Improvement; Medical Group Management Association; Premier Inc.; Robert Wood Johnson Foundation;

University of California, Berkeley, School of Public Health; and Winnipeg Regional Health Authority. The authors also acknowledge the support of Kathleen Carluzzo, Josette Gbemudu, Asha McClurg, Valerie Lewis, Eugene Nelson, Frances Wu, Patty Ramsay, and Thomas Rundall in conceptualizing this evaluation framework.

NOTES

1 Fisher ES, Staiger DO, Bynum JPW, Gottlieb DJ. Creating accountable care organizations: the extended hospital medical staff. Health Aff (Millwood). 2007;26(1):w44–57. DOI: 10.1377/hlthaff.26.1.w44.

2 Colla CH, Wennberg DE, Meara E, Skinner JS, Gottlieb D, Lewis VA, et al. Spending differences associ- ated with the Medicare Physician Group Practice Demonstration. JAMA. 2012;308(10):1015–23.

3 Song Z, Safran DG, Landon BE, He Y, Ellis RP, Mechanic RE, et al. Health care spending and quality in year 1 of the alternative quality contract. N Engl J Med. 2011; 365(10):909–18.

4 Chernew ME, Mechanic RE, Landon BE, Safran DG. Private-payer inno- vation in Massachusetts: the “Alter- native Quality Contract.” Health Aff (Millwood). 2011;30(1):51–61.

5 Fisher E, McClellan M. Accountable care organizations: a framework for evaluating proposed rules. Health Affairs Blog [blog on the Internet]. 2011 Mar 31 [cited 2012 Oct 19]. Available from: http://healthaffairs .org/blog/2011/03/31/accountable- care-organizations-a-framework-for- evaluating-proposed-rules/

6 Walker J, McKethan A. Achieving accountable care—“it’s not about the bike.” N Engl J Med. 2012;366(2): e4.

7 Van Citters AD, Larson BK, Carluzzo KL, Gbemudu JN, Kreindler SA, Wu FM, et al. Toward accountable care: four health care organizations’ ef-

forts to improve patient care and reduce costs [Internet]. New York (NY): Commonwealth Fund; 2012 13 Jan [cited 2012 Oct 10]. Available from: http://www.commonwealth fund.org/Publications/Case- Studies/2012/Jan/Four-Health- Care-Organizations.aspx

8 Bailit M, Hughes C. Key design ele- ments of shared-savings payment arrangements [Internet]. New York (NY): Commonwealth Fund; 2011 Aug 16 [cited 2012 Oct 10]. Available from: http://www.commonwealth fund.org/Publications/Issue-Briefs/ 2011/Aug/Shared-Savings-Payment- Arrangements.aspx

9 Delbanco SF, Anderson KM, Major CE, Kiser MB, Toner BW. Promising payment reform: risk-sharing with accountable care organizations [In- ternet]. New York (NY): Common- wealth Fund; 2011 Jul [cited 2012 Oct 10]. Available from: http:// www.commonwealthfund.org/ Publications/Fund-Reports/2011/ Jul/Promising-Payment-Reform .aspx

10 The call for proposals in this pro- gram is now closed. Robert Wood Johnson Foundation. Accountable care organizations: testing their impact—2012 call for proposals [In- ternet]. Princeton (NJ): RWJF; [cited 2012 Oct 18]. Available from: http://pweb1.rwjf.org/applications/ solicited/cfp.jsp?ID=21401

11 Lewis VA, Larson BK, McClurg AB, Boswell RG, Fisher ES. The promise and peril of accountable care for

vulnerable populations: a framework for overcoming obstacles. Health Aff (Millwood). 2012;31(8):1777–85.

12 Audet AMJ, Kenward K, Patel S, Joshi MS. Hospitals on the path to accountable care: highlights from a 2011 national survey of hospital readiness to participate in an accountable care organization [In- ternet]. New York (NY): Common- wealth Fund; 2012 Aug 17 [cited 2012 Oct 10]. Available from: http:// www.commonwealthfund.org/ Publications/Issue-Briefs/2012/ Aug/Hospitals-on-the-Path-to- Accountable-Care.aspx

13 Larson BK, Van Citters AD, Kreindler SA, Carluzzo KL, Gbemudu JN, Wu FM, et al. Insights from transfor- mations under way at four Brook- ings-Dartmouth accountable care organization pilot sites. Health Aff (Millwood). 2012;31 (11):2395–2406.

14 Shields MC, Patel PH, Manning M, Sacks L. A model for integrating in- dependent physicians into account- able care organizations. Health Aff (Millwood). 2011;30(1):161–72.

15 To access the Appendix, click on the Appendix link in the box to the right of the article online.

16 Fisher ES, Shortell SM. Accountable care organizations: accountable for what, to whom, and how. JAMA. 2010;304(15):1715–6.

17 Shortell SM, Weinberger M. Safety net accountable care organization (ACO) readiness assessment tool [Internet]. Berkeley (CA): University

Accountable Care Organizations

2376 Health Affairs November 2012 31:11

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

of California; 2012 Feb [cited 2012 Sep 11]. Available from: http:// www.law.berkeley.edu/files/bclbe/ Mar6_FINAL_combined.pdf

18 University of California, Berkeley. National Study of Physician Organi- zations [home page on the Internet]. Berkeley (CA): University of California; [cited 2012 Sep 11]. Available from: http://nspo.berkeley .edu

19 American Hospital Association, Health Research and Educational Trust. Surveying hospitals and health systems about their readiness to be accountable for the continuum of patient care [Internet]. Chicago (IL): HRET; [cited 2012 Oct 10]. Available from: http://www.hret

.org/reform/projects/surveying- hospitals-health-systems- accountable-continum-patient- care.shtml

20 A more detailed list of measures is in the pilot ACO data collection tools, available on request from the authors.

21 Scheffler R, Shortell S, Wilensky G. Accountable care organizations and antitrust: restructuring the health care market. JAMA. 2012;307(14): 1493–4.

22 For example, the High Value Health Care Project was an initiative of the Quality Alliance Steering Commit- tee, supported by Robert Wood Johnson Foundation, America’s Health Insurance Plans, and the

Brookings Institution, that devel- oped a standardized approach to aggregating administrative data across multiple health plans.

23 Participating payers implement the HealthPartners measure of total medical expense (actual payments) and price-standardized payments for each member of their commercial populations. These are summarized at the local-market (hospital service area) level, stratified by age and sex, and sent to Dartmouth for aggrega- tion across payers. The same kind of data, if made available on a timely basis, could support ACOs in their improvement work.

ABOUT THE AUTHORS: ELLIOTT S. FISHER, STEPHEN M. SHORTELL, SARA A. KREINDLER, ARICCA D. VAN CITTERS & BRIDGET K. LARSON

Elliott S. Fisher is director of the Center for Population Health at the Dartmouth Institute for Health Policy and Clinical Practice.

In this month’s Health Affairs, Elliott Fisher and coauthors propose a program to evaluate accountable care organizations (ACOs), to guide policy makers and payers in improving the design of proliferating ACO contracts. They recommend tracking ACO formation; developing a set of performance measures across all ACOs and payers; aggregating the performance data; conducting qualitative and quantitative research; and coordinating different evaluation activities. Fisher is director of the Center

for Population Health at the Dartmouth Institute for Health Policy and Clinical Practice. He is a professor of medicine and of community and family medicine at the Geisel School of Medicine at Dartmouth. He is the co–principal

investigator for the Dartmouth Atlas of Health Care. His research has focused on exploring the causes of the twofold differences in spending observed across US regions and health care systems; understanding the consequences of these variations on health; and developing and testing approaches to performance measurement and payment reform that can support improvement. Fisher’s current policy work

focuses on advancing the concept of ACOs, including codirecting a Brookings-Dartmouth program to advance ACOs through research, coordinating public and private initiatives, and creating a learning collaborative with several pilot ACO sites across the United States. He is also a member of the Institute of Medicine. Fisher received a master’s degree in public health from the University of Washington and a medical degree from Harvard Medical School.

Stephen M. Shortell is dean of the University of California, Berkeley, School of Public Health.

Stephen Shortell is dean of the School of Public Health as well as the Blue Cross of California Distinguished Professor of Health Policy and Management and a professor of organization behavior at the School of Public Health and the Haas School of Business, all at the University of California, Berkeley. He received the distinguished Baxter-Allegiance Prize for his contributions to health services research and is an elected member of the Institute of Medicine. Shortell is the lead investigator

of the National Survey of Physician Organizations, and he is also conducting research on the evaluation of quality improvement initiatives and on the implementation of evidence-based medicine practices in physician organizations. He has a master’s

November 2012 31:11 Health Affairs 2377

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

degree in public health from the University of California, Los Angeles; an MBA from the University of Chicago; and a doctorate in behavioral science from the University of Chicago.

Sara A. Kreindler is a researcher with the Winnipeg Regional Health Authority.

Sara Kreindler is a researcher with the Winnipeg Regional Health Authority, in Canada, and an assistant professor in the University of Manitoba’s Department of Community Health Sciences. Her work involves producing knowledge syntheses to help inform regional decision making. She also researches topics such as health care integration, service delivery models, chronic disease, and patient engagement.

Kreindler joined the Brookings- Dartmouth ACO evaluation team in 2011 as a Harkness Fellow through the Commonwealth Fund and led a substudy on social identity

dynamics in nascent ACOs. She holds a doctorate in social psychology from Oxford University.

Aricca D. Van Citters is a researcher with the Dartmouth Institute.

Aricca Van Citters is a researcher with the Dartmouth Institute for Health Policy and Clinical Practice. Her work focuses on health care evaluation and improvement. She has more than a decade of experience conducting qualitative and quantitative process and outcomes evaluations in a variety of health care settings. Van Citters’s recent research

projects focus on understanding the formation and performance of ACOs and the factors that contribute to rapid improvement in hospital quality, costs, and mortality. She has a master’s degree in evaluative clinical sciences from Dartmouth College.

Bridget K. Larson is the former director of health policy implementation at the Dartmouth Institute.

Bridget Larson is the former director of health policy implementation at the Dartmouth Institute for Health Policy and Clinical Practice. Her work focuses on advancing payment and delivery system reform to improve population health. While at the Dartmouth Institute, she led the implementation and evaluation of the ACO model through close collaboration with five national ACO pilot sites and the Brookings Institution. Previously, Larson worked at the

Dana-Farber Cancer Institute on developing best-practice models for a new ambulatory cancer center. She has also held a variety of roles in the private sector in policy, regulatory affairs, and process development. She holds a master’s degree in health policy and management from Harvard University.

Accountable Care Organizations

2378 Health Affairs November 2012 31:11

by Rachel McCartney on November 9, 2012Health Affairs by content.healthaffairs.orgDownloaded from

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.