HE380.0.1 Managed Healthcare Assignment 8

By William Shrank

The Center For Medicare And Medicaid Innovation’s Blueprint For Rapid-Cycle Evaluation Of New Care And Payment Models

ABSTRACT The Affordable Care Act established the Center for Medicare and Medicaid Innovation to test innovative payment and service delivery models. The goal is to reduce program expenditures while preserving or improving the quality of care provided to beneficiaries of Medicare, Medicaid, and the Children’s Health Insurance Program. Central to the success of the Innovation Center is a new, rapid-cycle approach to evaluation. This article describes that approach—setting forth how the Rapid Cycle Evaluation Group aims to deliver frequent feedback to providers in support of continuous quality improvement, while rigorously evaluating the outcomes of each model tested. This article also describes the relationship between the group’s work and that of the Office of the Actuary at the Centers for Medicare and Medicaid Services, which plays a central role in the assessment of new models.

I n 2010 section 3021 of the Affordable Care Act established the Center for Medicare and Medicaid Innovation to test innovative payment and service de- livery models. The goal is to reduce pro-

gram expenditures in Medicare, Medicaid, and the Children’s Health Insurance Program while preserving or improving the quality of care provided. Congress also granted a new, unique authority to the secretary of the Department of Health and Human Services to expand the dura- tion and scope of the testing of such models. The secretary may expand the scope and dura-

tion of testing through rule making, including nationwide testing, if she finds either that a model reduces spending without reducing the quality of care or that it improves the quality of care without increasing spending. To expand a model, the Centers for Medicare and Medicaid Services (CMS) chief actuary must certify that such expansion would reduce, or not result in any increase in, net program spending, and the secretary must determine that the expansion would not deny or limit the coverageor provision

of benefits. The decision to expand the duration and scope of a model being tested will be in- formed by the evaluations performed by CMS. The Innovation Center has announced a broad

agenda of tests of new payment and service de- livery models. These models aim to realign in- centives for providers to reward quality and the coordination of care instead of volume of ser- vices provided. Central to the success of the center is the ability to assess the effectiveness of models being tested. The center must conduct evaluations with urgency, routinely and rapidly assessing the effectiveness of interventions in the field, to promote continuous improvement of the implementation of those interventions and identify effective policies without delay. To assess the success of initiatives, the Inno-

vation Center has assembled the Rapid Cycle Evaluation Group, responsible for evaluating the impact of each payment and service delivery model on the cost and quality of care and on health outcomes. As important as it is to have the models assessed rapidly, rigor cannot be sac- rificed for speed. The group aims to transform

doi: 10.1377/hlthaff.2013.0216 HEALTH AFFAIRS 32, NO. 4 (2013): 807–812 ©2013 Project HOPE— The People-to-People Health Foundation, Inc.

William Shrank ([email protected]) is the director of the Rapid Cycle Evaluation Group at the Center for Medicare and Medicaid Innovation, Centers for Medicare and Medicaid Services, in Baltimore, Maryland.

April 2013 32:4 Health Affairs 807

Web First

by Rachel McCartney on April 17, 2013Health Affairs by content.healthaffairs.orgDownloaded from

the evaluation process at CMS in critical ways to meet the standards imposed by the statute. “Rapid cycle” refers both to the approach to assessing the effectiveness of interventions more rapidly and to a philosophy of providing ongoing feedback to participating providers to support continuous quality improvement. This article presents both the current evalu-

ation philosophy and the specific approaches that the evaluation group will apply to transform the evaluation process at CMS.

New Opportunity For New Ideas The priority of the CMS Innovation Center is to test models that reduce costs through care improvement, supported by the statute’s call for preferenceto be given to models that improve the coordination, quality, and efficiency of ser- vices. The statute requires an evaluation of each model. The evaluation is to include an analysis of the quality of care furnished under the model, including the measurement of patient-level out- comes and patient-centeredness criteria as well as changes in spending. Since its inception, the center has announced

a broad agenda of new models attempting to realign incentives for providers, hospitals, and health systems to promote improved quality, ef- ficiency, and outcomes under the authority of section 3021 of the Affordable Care Act. (A list of models is in Appendix Exhibit 1.)1

Some examples include testing of the Pioneer and Advance Payment Accountable Care Orga- nization models, which aim to align incentives for health systems to promote higher-quality care for the population served and greater accountability for the total cost of care; the Bundled Payments for Care Improvement initia- tive, which is a series of models to realign incen- tives for hospitals and postacute care providers; and the Comprehensive Primary Care initiative, which provides up-front support to transform primary care practice while focusing incentives to reward care coordination, quality, and effi- ciency. Each model has been developed to create a business case for quality improvement, relying on marketplace innovation to reduce variability and waste while improving patients’ experiences and health outcomes.

Promoting Continuous Quality Improvement The models that the Innovation Center is testing require structural changes in care delivery. Many of these changes challenge traditional assump- tions about the way in which care is provided. Primary care physicians participating in patient-

centered medical homes or accountable care or- ganizations must adapt their practices to deliver on the promise of the new model. It is likely that they will need to invest in new information sys- tems and hire new staff to support more coordi- nated care. Similarly, hospitals participating in the Bundled Payments for Care Improvement initiative; in accountable care organizations; or in the Partnership for Patients initiative, which aims to save lives by dramatically increas- ing hospital patient safety and to cut avoidable readmissions, must invest in and redesign their infrastructures to support improved care for their populations and to reduce costs. Substantial learning and adaptation will be

essential for providers or health systems to achieve the greatest efficiencies and improve- ments. There are no simple “turnkey” solutions. The CMS Innovation Center does not plan to

sit on the sidelines and assess performance at the conclusion of each test. Rather, evaluators will be part of the solution by gathering real-time information and making use of CMS claims data to promote and support continuous quality improvement. For example, in the Pioneer Accountable Care Organization program, the Innovation Center is providing regular raw data feeds to participants, interpretable performance data allowing participants to compare them- selves with other Pioneers, and a learning col- laborative that enables Pioneers to share their experiences and that encourages the adoption of best practices. CMS is also encouraging participating pro-

viders to collect their own performance data and use that data to better manage outcomes. The agency wants to make sure participating providers have information both about their own performance and about the successes and failures of other model participants so that they can successfully implement the model. The hope is that providing data and rapid-cycle feedback to providers will enhance their ability to improve and test the merits of the model. At the core of this approach is the recognition

that evaluators must not only assess results but must also understand the context of those re- sults. In each of our models, evaluators will collect qualitative information about providers’ practices, their organizations, and the systems in which they practice. Evaluators must understand how participants

implement interventions, their perceptions about the opportunity, and the barriers and en- ablers to change. Without this thorough under- standing of how each participant has acted to improve care, the Innovation Center cannot pro- vide a truly accurate portrait of how favorable outcomes were obtained.

Web First

808 Health Affairs April 2013 32:4

by Rachel McCartney on April 17, 2013Health Affairs by content.healthaffairs.orgDownloaded from

These data will be merged with performance metrics so that CMS can provide actionable feed- back to participants about howthey canimprove. Doing so will allow evaluators to more thought- fully “connect the dots” and assess what features of interventions are associated with successful outcomes. Without dissemination of the findings and

frequent course corrections, understanding the effectiveness of interventions will not lead to meaningful practice improvement. Evaluators will provide data to a dedicated Learning and Diffusion team at the CMS Innovation Center that will organize learning collaboratives among model participants to spread effective ap- proaches and to disseminate best practices. This close collaboration will help ensure that the best practices are harvested and disseminated rap- idly, and it is intended to generate a more col- laborative community of providers working to- gether to improve the quality of care. By separating dissemination activities from evalu- ation activities, CMS preserves the objectivity of this evaluation team.

Evaluating With Speed And Without Sacrificing Rigor Beyond providing feedback to providers to sup- port their improvement, CMS will strive to assess the overall impact of its models more rapidly than in the past. A key component of this approach is an effort to evaluate each model regularly and frequently after implementation, allowing both the rapid identification of oppor- tunities for course correction and improvement and timely action on that information. The length of funding for each intervention is

assigned to be sure that the program is in the field long enough to have a reasonable chance of success.Yet waiting until the predetermined end date of a program to assess effectiveness is not desirable; better would be to draw conclusions about each model far sooner, if sufficient evi- dence is available. For example, it may not take five years for savings to be measured in the Comprehensive Primary Care initiative, and measuring key outcomes will begin shortly after implementation. The Rapid Cycle Evaluation Group will seek evidence of meaningful improve- ment shortly after implementation of each new model. Again, it is important to note that the rigor of

the evaluationcannotbe sacrificedfor thesake of speed. Failing to characterize accurately the re- sults of innovative models can carry large con- sequences. The group must be certain that we obtain dependable evidence for the impact of the models on spending and quality of care to

determine which models will achieve the goal of providing higher-quality care at lower costs. To do so, we must use advanced statistical

methods to measure effectiveness. Our methods must provide results that meet a high standard of evidence, even though strict experimental conditions cannot always be met. For example, randomizing the assignment of providers or beneficiaries to payment and service delivery models may often be infeasible because of logis- tical constraints. As a result, our evaluations will have to account for potential confounding re- lated to providers’ characteristics that might influence outcomes independent of the interven- tion being studied.We must account for the fact that providers who choose to participate in our models may differ from nonparticipants in im- portant ways, and the populations they serve may differ as well. The group will apply the most advanced methods to carefully adjust for these sources of confounding. To achieve our goals, appropriate selection of

comparison groups will be an essential compo- nent of every evaluation. We will be sure to con- sider environmental and policy characteristics when selecting controls. When appropriate, we will use various methods now common in the social sciences—such as propensity score ap- proaches and instrumental variables—to help control for sources of bias and to clarify specific causal mechanisms. We will make a practice, whenever possible, of identifying multiple com- parison groups for each intervention group, to determine the robustness of our findings. We will increasingly use repeated measures—

time-series analyses—which use the same sub- jects under different conditions over time and allow us to better understand the relationship between implementation of new models and both immediate changes in outcomes and the rate of change of those outcomes. These designs will allow us to better account for trends or pat- terns of improvement prior to the inception of a model and to isolate the effect of the intervention more precisely on the outcomes we are measur- ing. These more nuanced statistical methods will be essential to assessing the impact of new mod- els more precisely and rapidly.

Blurring The Lines Between Feedback And Evaluation The evaluation group is committed to the rapid provision of timely data to providers and inter- pretation of those data to synthesize both the contextual and operational features of the par- ticipating providers and their outcomes. These approaches constitute what is known as forma-

April 2013 32:4 Health Affairs 809

by Rachel McCartney on April 17, 2013Health Affairs by content.healthaffairs.orgDownloaded from

tive evaluation, which generally focuses on the process aspects of an innovation while the innovation is still in its formative stages. Our feedback will allow providers to track their per- formance compared with their own historical outcomes and the performance of other partici- pating providers and comparison physicians who are not participating in the model being tested. For example, in the Comprehensive Primary

Care initiative, as in all of our initiatives, we will provide at least quarterly feedback on dozens of performance metrics, which include process, outcome, and cost measures.We will also capture contextual information about the organizational structure and specific features of each partici- pant, to identify characteristics associated with superior performance. We will explore whether practices with certain

types of organizational structures, or in which care is delivered not only by physicians but also by nurse practitioners, nurses, allied health professionals, and others, tend to experience improved outcomes. We will feed this informa- tion back to model participants in quarterly reporting to help them learn approaches that may hasten improvement and to allow their lead- ers to manage to specific, measurable outcomes. Similarly, when we have the necessary data, we

plan to conduct regular impact analyses to assess the success of the models overall, beginning shortly after implementation of a new model. In the past, researchers often waited until the end of a demonstration to measure its success, delaying the opportunity to identify meaningful changes. We will now be evaluating critical out- comes quarterly, using rigorous evaluation tech- niques, to assess the overall impact of the pay- ment model. In this way, we hope to identify, more rapidly than before, successes and areas where improvement is needed. Impact analyses are considered to be “summa-

tive evaluation,” which, in contrast to formative evaluation, is retrospective in nature and re- quires sufficient rigor to make programmatic decisions. Historically, formative evaluation has used less rigorous methods than summative evaluation. However, with conscientious use of our data, we plan to use methods for perfor- mance feedback (formative evaluation) that are similar to our methods of conducting impact (summative) evaluations. As a result, some of the classic differentiation

between the provision of formative feedback and the more rigorous summative evaluation will be blurred. Better collection and use of data will allow us to conduct rigorous, objective, and com- prehensive evaluations that will meet our statu- tory criteria while simultaneously helping pro-

viders improve. When these approaches are ap- plied in a timely manner, we will be able to use the same calculations for both purposes.

Implementing A New Model And Metrics For Success When a model is considered for testing by the CMS Innovation Center, staff from the Rapid Cycle Evaluation Group, as well as staff from the CMS Office of the Actuary, are immediately assigned to participate in the creation of the model. There is a shared understanding that every model we implement is a test, and we can determine whether a test is successful only if the model can be rigorously assessed. Including the evaluationand actuarial staffin themodel design embeds the evaluation concept in the model from the start, laying the groundwork for a model that can be thoroughly evaluated. From a practical standpoint, evaluation staff

are engaged at the outset of every model in de- termining the generalizability of the interven- tion to groups other than the targeted popula- tion. We also determine the necessary sample size of the intervention group, after considering possible attrition from the model and patient clustering within providers, practices, or health systems.We must also address the availability of necessary data to measure outcomes and the ap- propriate comparison group or groups for each intervention. We consider our ability to measure the effec-

tiveness of the model as a whole as well as its effectiveness with subgroups that may differ in important ways.We clarify the challenges of each evaluation in the setting of other policy or prac- tice changes and the varying accessibility of data from different sources. By being engaged at the beginning of the development of each model, we will have the best chance of achieving a robust, statistically sound evaluation. Establishing effective metrics at the outset

of each model is critical to defining success. Innovation Center evaluators have collaborated with the CMS Center for Clinical Standards and Quality to ensure that our metrics are consistent across programs as appropriate, and that we can thoughtfully compare the results of different models. We must be sure to align not only mea- sures of cost and quality but also other important metrics, such as patients’ functional status, pop- ulation health, patient and provider experience, and key social determinants of health. We plan to identify and promote population

health metrics—measures of the functional sta- tus, healthy behavior, and health outcomes of a population—so that we may better emphasize the importance of greater disease prevention in-

Web First

810 Health Affairs April 2013 32:4

by Rachel McCartney on April 17, 2013Health Affairs by content.healthaffairs.orgDownloaded from

stead of treating conditions that have already arisen. Our measures of cost will generally focus on the total cost of care of a beneficiary, instead of the cost of care delivered by specific providers, to encourage a more accountable and coordi- nated health care system that promotes effi- ciency and comprehensive management of each beneficiary. We will also measure disparities in care in every model, to enable better understand- ing of which models best address health and health care inequities.

Partnering With The Actuaries The relationship between the CMS Innovation Center and the Office of the Actuary is a particu- larly critical one. The Office of the Actuary provides timely, impartial, and authoritative ac- tuarial, economic, and statistical estimates and analysis of health care financing and expendi- tures. Similar to the engagement of evaluation staff

when new payment and service delivery ap- proaches are conceived, the actuaries are in- cluded in model development teams. They pro- vide insight and guidance into the process that will be used to assess the impact of Innovation Center programs on Medicare, Medicaid, and Children’s Health Insurance Program spending. The estimate of each model’s impact on quality, health outcomes, and costs will need to account for any material differences in model design, provider, marketplace, population, or other fac- tors between the testing phase and the projec- tion period. The Office of the Actuary is in close communication with the model implementation and evaluation teams throughout model devel- opment, deployment, and evaluation, to offer consultation at each step of the process. As described above, after a payment or service

delivery model has been tested, the secretary can only expand the duration and scope of the model test through rule making if the model is expected either to reduce spending without reducing the quality of care or to improve the quality of care without increasing spending. In this context, the chief actuary must certify that the expansion would reduce, or not result in any increase in, net program spending. Models also must be modified or terminated after testing has begun unless the secretary determines, and the chief actuary certifies, that the model is expected to improve the quality of care without increasing spending, reduce spending without reducing the quality of care, or improve the quality of care and reduce spending. To determine the cost impact of the model, the

Office of the Actuary will monitor Innovation Center initiatives once testing has begun and will

use data from the evaluation as well as other available sources to certify results.

Developing Methods The quasi-experimental design of our experi- ments and the complex and rapidly changing setting of our “laboratory”—the real world—will require the consideration and development of a number of new methods. The simultaneous im- plementation of numerous models and policy changes presents a key challenge, requiring con- sideration of when and how to disentangle the effects of multiple co-occurring events. To address these needs, the Rapid Cycle

Evaluation Group is relying on improved data systems that will allow CMS to track the imple- mentation of each model at the beneficiary level. Our evaluations must then consider overlap with Innovation Center and other programs in the selection of similar comparison groups and must account for overlap in the conclusions that can be drawn. We will aim to recruit sufficient sam- ples of patients enrolled in a single model to evaluate the model’s effect on outcomes. However, in addition to disentangling model

effects, we aim to study the interactions of differ- ent models, to better understand how they may optimally be applied to improve quality and re- duce costs. Although we recognize that we may not have sufficient sample size to assess inter- actions in some subgroups definitively, studying the interactions among interventions will pro- vide rich opportunities to study potential syner- gies or challenges when different programs in- teract. Summative evaluations will include numerous sensitivity analyses comparing inter- vention and comparison groups where single interventions were implemented, as well as sub- groups where overlap may exist.

Contributing To The Evidence: Our Path Forward The key research questions we intend to answer at theCMS InnovationCenterarecentralto deliv- ering on the goal of providing higher-quality care to beneficiaries while reducing costs.We will work with external research contractors on our evaluations of Innovation Center model tests, to provide objectivity in our results. Effective and proactive intramural research is

also under way to support changes to CMS pro- grams and payment policy. Researchers from the evaluation group have organized into affinity groups and are using CMS data to answer critical policy questions that may inform the develop- ment of future payment and service delivery models. Our research agenda is developed

April 2013 32:4 Health Affairs 811

by Rachel McCartney on April 17, 2013Health Affairs by content.healthaffairs.orgDownloaded from

collaboratively with CMS leadership to identify research questions whose answers can best sup- port evidence-based policy making. By using our data to better understand variability, waste, and disparities in care, we aim to better inform and generate new policies that will get us closer to our ultimate goal of better care, better health, and lower costs. Of course, all payers—and all Americans—

share the goals of improved care and lower costs. Results from the evaluations of our models will be useful not only to CMS and its beneficiaries but also to commercial and other government payers. As a result, we appreciate the importance of rapid dissemination of our results to other payers, to expand the evidence base for payment reform and to ensure the diffusion of successful interventionsthrough the health care system as a whole. To do so, our goal is both to release our reports rapidly on our website and to publish key results subsequently in the peer-reviewed liter-

ature and disseminate our findings broadly so that taxpayer-funded research is sure to benefit all Americans. The CMS Innovation Center was born out of

the recognition that the health system requires payment and delivery reform. The need to im- prove the coordination and quality of the care available to the beneficiaries that CMS serves, together with the need to slow spending growth, underscores the urgency of the situation. A commitment to change must be coupled

with an acknowledgment that the solutions that offer promise must be assessed objectively. The members of the evaluation group at the CMS Innovation Center are eager to collaborate with and learn from other federal, state, and commer- cial partners as we strive to evaluate—rapidly and rigorously—an ambitious agenda to improve the quality of health care received by Medicare, Medicaid, and Children’s Health Insurance Programbeneficiaries and allother Americans.▪

The author thanks Renee Mentnech, the deputy director of the Rapid Cycle Evaluation Group, for her contribution to the strategies outlined here. He also

thanks John Shatto, the deputy director of the Office of the Actuary, Centers for Medicare and Medicaid Services, for his constructive feedback and careful

comments on this article, and Daniel Farmer for his support. [Published online March 27, 2013.]

NOTE

1 To access the Appendix, click on the Appendix link in the box to the right of the article online.

ABOUT THE AUTHOR: WILLIAM SHRANK

William Shrank is the director of the Rapid Cycle Evaluation Group at the Center for Medicare and Medicaid Innovation.

In this month’s Health Affairs, William Shrank, director of the Rapid Cycle Evaluation Group at the Center for Medicare and Medicaid Innovation, Centers for Medicare and Medicaid Services (CMS), describes the approaches his group will take as it evaluates new delivery and payment models launched by the Innovation Center

under the Affordable Care Act. Shrank describes how the Rapid Cycle Evaluation Group will deliver frequent feedback to providers in support of continuous quality improvement, while rigorously evaluating the models’ outcomes. Shrank also describes how his

group’s work will intersect with that of the CMS Office of the Actuary, which must certify the results of the tested models if the secretary of health and human services is to extend them under the Affordable Care Act. Before joining CMS, Shrank was

an assistant professor of medicine at Harvard Medical School and an associate physician in the Division of Pharmacoepidemiology and

Pharmacoeconomics at Brigham and Women’s Hospital in Boston, where he practiced general internal medicine and published more than a hundred articles in the peer- reviewed literature focused on improving the safe, appropriate, and cost-effective use of prescription medications. Shrank previously served on

national advisory committees for the Food and Drug Administration and the Agency for Healthcare Research and Quality, among other government agencies. He earned a medical degree from Cornell University and a master’s degree in health services from the University of California, Los Angeles.

Web First

812 Health Affairs April 2013 32:4

by Rachel McCartney on April 17, 2013Health Affairs by content.healthaffairs.orgDownloaded from

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.