Healthcare Analytics; Regulations, Clinical Quality, and Patient Safety

By David W. Bates, Suchi Saria, Lucila Ohno-Machado, Anand Shah, and Gabriel Escobar

Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients

ABSTRACT The US health care system is rapidly adopting electronic health records, which will dramatically increase the quantity of clinical data that are available electronically. Simultaneously, rapid progress has been made in clinical analytics—techniques for analyzing large quantities of data and gleaning new insights from that analysis—which is part of what is known as big data. As a result, there are unprecedented opportunities to use big data to reduce the costs of health care in the United States. We present six use cases—that is, key examples—where some of the clearest opportunities exist to reduce costs through the use of big data: high-cost patients, readmissions, triage, decompensation (when a patient’s condition worsens), adverse events, and treatment optimization for diseases affecting multiple organ systems. We discuss the types of insights that are likely to emerge from clinical analytics, the types of data needed to obtain such insights, and the infrastructure—analytics, algorithms, registries, assessment scores, monitoring devices, and so forth—that organizations will need to perform the necessary analyses and to implement changes that will improve care while reducing costs. Our findings have policy implications for regulatory oversight, ways to address privacy concerns, and the support of research on analytics.

T he cost of health care in the United States is high, nearly twice that in most other developed countries,1

and it continues to grow rapidly. The unsustainable projected trajec-

tory of US health care costs has led to calls for improving the value of health care.2 However, the Affordable Care Act—the most substantial policy reform in US health care in decades— has been criticized for not doing enough to con- tain costs.3

As health reform progresses, one key dynamic of theUShealth care system is the rapid adoption of electronic health records (EHRs). The growth of EHRs will make it possible to access unprece- dented amounts of clinical data and offers the potential for cost savings.4 The extent of those cost savings is still to be determined,5 but EHRs’

value in increasing health care providers’ access to patients’ records is not in question. In other industries, companies have been very

successful at using big data to improve their effi- ciency.6 By big data, we refer to the high volume, variety, and potential for the rapid accumulation of data and to analytics, which is the discovery and communication of patterns in data. Examples include Amazon’s product recom-

mendation system for online shopping, creating efficient pricing in the stockmarket, andpredict- ing players’ statistics in baseball. “Watson”—an application developed by IBM—had a recent suc- cess on the television quiz show Jeopardy, using some of these big-data approaches.7 However, the extent to which these tactics will be applica- ble to clinical questions is as yet uncertain.8

The underlying techniques used in big data

doi: 10.1377/hlthaff.2014.0041 HEALTH AFFAIRS 33, NO. 7 (2014): 1123–1131 ©2014 Project HOPE— The People-to-People Health Foundation, Inc.

David W. Bates (dbates@ partners.org) is chief of the Division of General Medicine, Brigham and Women’s Hospital, in Boston, Massachusetts.

Suchi Saria is an assistant professor of computer science and health policy management at the Center for Population Health and IT, Johns Hopkins University, in Baltimore, Maryland.

Lucila Ohno-Machado is associate dean for informatics and technology in the Division of Biomedical Informatics, University of California, San Diego, in La Jolla.

Anand Shah is vice president of clinical services at PCCI, in Dallas, Texas.

Gabriel Escobar is regional director of hospital operations research and director of the Systems Research Initiative, Division of Research, Kaiser Permanente, in Oakland, California.

July 2014 33:7 Health Affairs 1123

Predictive Analytics

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

have improved substantially in the past decade, and they often involve hypothesis-free ap- proaches such as data mining. Many experts have called for health care to adopt big-data ap- proaches,9 but uptake has been relatively limited so far. That may be about to change. Payment reform

strategies that incentivize value such as account- able care (a key strategy of the Affordable Care Act, in which entities are asked to be “account- able” for the care they provide) and bundling (a payment approach in which providers are asked to deliver a set of services for a predefined price) are intended to motivate organizations to im- prove the efficiency of their care. One tactic that health care organizationswill likely deploy is the more effective use of predictive analytics. Ideally, predictive analytics will involve link-

ing data from multiple sources, including clini- cal, genetic and genomic, outcomes, claims, and social data.Many new sources of data are becom- ing available, such as data from cell phones and social media applications. Aggregating these data for the purpose of achieving clinical predic- tive analytics will require the adoption of stand- ards,10 raise privacy and ethical concerns,11 and require new ways to preserve privacy.12

Big data sets can be subjected to many other types of analytic approaches, including pattern recognition and natural history—that is, the course of a disease process. However, we believe that even in the short term, it will be possible for health care organizations to realize substantial benefits from deploying predictive systems. Pre- dictive systems are software tools that allow the stratification of risk to predict an outcome. Such tools are important because many potential out- comes are associated with harm to patients, are expensive, or both. In health care, we suggest that one way to use

predictive systemswould be to identify andman- age six verypractical use cases—that is, examples of instances in which value is likely to be achieved. They are high-cost patients, readmis- sions, triage, decompensation (when a patient’s condition worsens), adverse events, and treat- ment optimization for diseases affecting multi- ple organ systems (such as autoimmune dis- eases, including lupus). Below we address the types of data and infrastructure that health care organizationswill need for eachuse case.We also discuss what organizations will need to do to actually improve care.

High-Cost Patients Approximately 5 percent of patients account for about 50percent of allUShealth care spending.13

One approach to reducing costs is to identify

suchpatients andmanage themmore effectively, oftenbyhaving casemanagersworkwith themto improve their care. Suchanapproachhas already resulted in cost reductions.14 However, the iden- tification of potentially high-cost patients has not always produced the desired results. For ex- ample, a number of Medicare demonstration projects did not lower costs even though the projects were able to identify high-risk pa- tients.14,15

To effectively implement analytic methods for identifying potentially high-cost patients, a number of issuesmust be considered. First, what approach should be used to predict which pa- tients who are likely to be high risk or high cost? Second, what new measurement sources can be incorporated to improve the predictions? Attri- butes associated with high-cost patients may include behavioral health problems or socio- economic factors such as poverty or racial mi- nority status. Thus, integrating data about men- tal health, socioeconomic status, or other issues such as marital and living status from various sources16 may significantly change the quality of the predictions that can be made. A third issue is how to make predictions ac-

tionable, by identifying which patients are most likely to benefit from an intervention and what specific interventions can most improve care. The effective implementation of new analytic systems to identify potentially high-cost patients will require making predictions easily available with minimal changes to clinical work flows, to increase the chances that health care providers will act on the predictions. Many organizations and companies that cur-

rently use analytic systems have focused on iden- tifying thealgorithmthat canbest stratify data by risk of future costs while not addressing other issues. The variation among algorithms may not be large, and a more practical algorithm may be better than a slightly more accurate one. Algo- rithms aremost effective and perform best when they are derived from and then used in similar populations.17–19

A fourth issue is how to account for the fact that many cases of outcomes in predictive mod- els often come from low-risk groups. This sug- gests the need for more accurate modeling, par- ticularly for population management. We suggest that it is important in using analyt-

ic systems to identify potentially high-cost pa- tients to determine the patients’ specific needs and gaps in care. It is especially important to identify andaddressbehavioralhealthproblems, because a large portion of the patients at high risk for hospital admission have some sort of behavioral health issue, with depression being especially frequent.20

Predictive Analytics

1124 Health Affairs July 2014 33:7

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

Programs to manage high-cost patients are expensive. They will bemuchmore cost-effective if interventions can be precisely tailored to a patient’s specific problems, which might be re- lated to transportation, medication nonadher- ence, or family conflict. Resources inhealth care arebecoming increas-

ingly limited, which requires greater emphasis on value. Thus, it will be important to investigate analytic techniques that identify not only high- risk people, but also thosewho areat particularly low risk. For instance, the standard approach may be to give all patients who are discharged from the hospital a follow-up appointment in two weeks. But it might make more sense to ensure that the highest-risk patients are seen within twodays,while patientswith very low risk might require follow-up care only as needed. Algorithms can help reallocate resources more effectively at both the high-risk and low-risk ends of the spectrum.

Readmissions Much has been made of the frequency and high cost of hospital readmissions.21 The Centers for Medicare and Medicaid Services (CMS) has strongly incentivized organizations to reduce their frequency.22 As many as one-third of re- admissions have been posited to be preventable and, therefore, to present a significant opportu- nity for improving care delivery.23

Health care organizations should all use an algorithm to predict who is likely to be readmit- ted to the hospital. However, the predictive value of the algorithms tends to be similar. Four areas of a predictive algorithm may be important dif- ferentiators: tailoring the intervention to the in- dividual patient, ensuring that patients actually get the precise interventions intended for them, monitoring specific patients after discharge to find out if they are having problems before they decompensate, and ensuring a low ratio of pa- tients flagged for an intervention topatientswho experience a readmission (that is, a low false

positive rate). Some work has already been done in predict-

ing readmissions,24 and analytics will play a key role in further work. For example, it may make sense soon to ask patients with a smartphone to allow health care organizations to access data from their phones that will help identify patients who are not managing a chronic condition well or that will monitor people recently discharged from the hospital, since it appears that patients who are not making calls or sending e-mail with their usual frequency may be depressed or suf- fering from other issues.25 Patients may also be asked to wear some type of device that monitors physiological parameters, such as heart rate or rhythm. These data will be most effective in in- forming health care decisions if they are proc- essed with analytics.

Triage Estimating the risk of complications when a pa- tient first presents to a hospital can be useful for a number of reasons, such as managing staffing and bed resources, anticipating the need for a transfer to the appropriate unit, and informing overall strategy for managing the patient. In the neonatal setting, for example, the invention of the Apgar score revolutionized the management of newborn resuscitation.26,27 However, comput- ing the score required training caregivers to as- sess subjective parameters such as irritability and “color” (a proxy for tissue perfusion, or how well blood is flowing to tissues). In new- borns and many other populations, using mod- ern big-data techniques28 that combine routinely collected physiological measurements makes much more accurate assessments possible with a minimal burden of training and implemen- tation.29

In integrating a triage algorithm into clinical work flow, it is vital to have a detailed guideline that clarifies how the algorithmwill inform care. Twopilot programs inKaiserPermanenteNorth- ern California (KPNC), an integrated health care delivery system with comprehensive informa- tion systems, are using this approach. The first pilot involves evaluating newborns

for early onset sepsis. The goal is to reduce the number of newborns who receive antibiotics un- necessarily.Hundreds of thousands of newborns are evaluated for early-onset sepsis each year.30–32

Recently, a team of scientists from KPNC, Harvard University, and the University of Cali- fornia, San Francisco and Santa Cruz, developed a two-step protocol that can be expected to de- crease the number of these evaluations and re- duce theprescriptionof antibiotics fornewborns dramatically in the United States. In the first

One tactic that health care organizations will likely deploy is the more effective use of predictive analytics.

July 2014 33:7 Health Affairs 1125

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

step, which can be embedded in an EHR, objec- tive maternal data are used to assign a prelimi- nary (prior to birth) probability of early-onset sepsis.33 In the second step, a simplified set of clinical findings are combined with the estimate based on maternal data to yield a new posterior probability for risk of sepsis following birth.34

The combination of these two steps could lead to asmany as 240,000 fewerUSnewborns’being treated with systemic antibiotics each year. The second KPNC pilot addresses adult pa-

tients in the emergency department. Severity- of-illness scores for adult intensive care patients have been available for some time.35,36 However, the scores’ impact on triage has been limited. This is in part because the most important of these—the Acute Physiology and Chronic Health Evaluation (APACHE)37 and the Simplified Acute Physiology Score (SAPS)38—involve data that are captured after a patient has entered intensive care. In the second pilot, clinicians in the emergen-

cy departmentwill beprovidedwith two compos- ite scores that have been calibrated using mil- lions of patient records and that are applicable to all hospitalized patients, not just those in inten- sive care. The first of these scores summarizes a patient’s global comorbidity burden during the preceding twelve months; the second captures a patient’s physiological instability in the preced- ing seventy-two hours.39 In addition, these two scores, available in real time, are combined with vital signs, trends in vital signs, and other infor- mation, such as how long a patient has been in the hospital. If the information collectively indi- cates that a patient has ≥8 percent risk of deteri- orating in the next twelve hours, an alert is sent to the responsible providers. Importantly, the KPNC early-onset sepsis and

emergency department composite score pilots are both designed for patients who are not being monitored continuously, yet they take advantage of big-data methodologies. In both cases, teams of clinicians are developing work flows that in- tegrate big-data components (real-time risk es- timates) with traditional components (such as clinical examinations and care pathways).

Decompensation Oftenbefore decompensation—theworseningof a patient’s condition—there is a period in which physiological data can be used to determine whether the patient is at risk for decompensat- ing. Much of the initial rationale for intensive care units (ICUs) was to allow patients who were critically ill to be closely monitored. A host of technologies40 are now available that can be used to monitor patients who are in general care

units, in nursing homes, or even at home but at risk of some sort of decompensation. Real- time indices such as the Rothman Index are also available.41–43

Some of these technologies have been avail- able for many years, such as electrocardiograph- ic monitoring and oxygen monitoring. Others are newer, such as end-tidal CO2 monitoring and monitors that allow detection of whether or not a patient is moving.44,45 A problem with all of these technologies has been the signal-to- noise ratio: Alarms are often false positives. Monitors are becoming available in which mul- tiple data streams can be compared simulta- neously, and analytics can be used in the back- ground to determine whether or not the signal is valid. One example of these newmonitors is a device

that sits under themattress and that collects data about thepatient’s respiratory rate andpulse and whether or not the patient is moving.45 The data are transmitted to a server, where analytics are used in real time to determine if the patient appears likely to be decompensating. When the system detects a likely decompensation, an e-mail message is sent to an on-duty nurse’s smartphone. With this system, the likelihood that a true

decompensation is present has been increased to approximately 50 percent—far better than for cardiac telemetry, for which it is typically 5– 10 percent. In one small trial, the system reduced the number of subsequent ICU days for patients in general care units by 47 percent, compared to controls.45

Analytics that use multiple data streams to effectively detect decompensation are already at work in some ICUs, and such use is expected to grow. Analytic tools are likely to make their way into other clinical settings as well to predict decompensation.

Some work has already been done in predicting readmissions, and analytics will play a key role in further work.

Predictive Analytics

1126 Health Affairs July 2014 33:7

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

Adverse Events Another use case for analytics will be to predict which patients are at risk of adverse events of several types. Adverse events are expensive46 and cause substantial morbidity and mortality, yet many are preventable.

Renal Failure Renal failure is extremely ex- pensive and carries a high risk of mortality.47

However, renal function is readily measured, and early changes in it are often apparent well before major decompensation occurs. It seems likely that analytics could be combinedwith data about exposures to specific medications and with measures of kidney function, blood pres- sure, urine output, and other processes to iden- tify patients at risk of decompensation.

Infection Analytics can be effective in man- aging infection. One example involves monitor- ing and interpreting changes in heart rate vari- ability for detection of major decompensation in infants with very low birthweights before the emergence of an infection.48 Monitoring the heart-rate characteristics of newborns alone has already resulted in reductions in mortality and increases in the number of ventilator-free days. However, there is room for improvement using increasingly sophisticated analytics that account for subtle signals28 but also filter out extraneous patterns,49 such as those that occur when the baby moves.

Adverse Drug Events Adverse drug events, which occur frequently50 and are costly,51 are an- other area where analytics can be effective. Most efforts so far to predict which patients will suffer an adverse drug event have not been very effec- tive.52,53 However, analytics have the potential to predict with substantial accuracy which patient may suffer an adverse drug event and to detect patients who are in the early stages of such an event, by assessing genetic and genomic infor- mation, laboratory data, information on vital signs, and other data.

Diseases Affecting Multiple Organ Systems Chronic conditions that span more than one or- gan system or are systemic in nature are some of the costliest conditions tomanage.54,55 Anysingle disease may include cutaneous (skin), mucosal, renal, musculoskeletal, pulmonary, hematologi- cal, immunological, and neurologic manifesta- tions.56 Autoimmune disorders such as sclero- derma, rheumatoid arthritis, and systemic lupus erythematosus are examples of such con- ditions. The ability to accurately predict the trajectory

of a patient’s disease could allow the caregiver to better target complicated and expensive thera- pies to patients who stand to benefit the most from them, thus reducing the burden of disease on those patients and on the health care system. Currently, the caregiver’s ability to optimize treatment is limited by the complexities result- ing from the heterogeneity in clinical pheno- types, the diversity of available measurements, and lack of high-precision biomarkers.57

This area is ripe for computing approaches that can combine the multitude of measure- ments taken as part of routine care to infer the progression of a patient’s disease and tailor treatments to that patient. There are already some successful examples of these ap- proaches.58,59

Multisite longitudinal registries that allow the aggregation of populations of patients with a disease or condition60 have been initiated. In the near future, clinical data networks are likely to play the role that registries now do. One example of such a network is the National Patient-Centered Clinical Research Network (PCORnet),61 which itself comprises multiple clinical data research networks. Access to longi- tudinal records has been the biggest limitation for making progress in the area of chronic dis- eases in multiple organ systems. As EHRs and clinical data networks based on EHRs become widespread,weexpect to see thebenefits of these technologies in improving care for patients with such diseases.

Discussion We have discussed six use cases for high-risk patients in which clinical analytics are likely to be highly beneficial. This is by no means an ex- haustive list. The evidence of benefit varies wide- ly across the six use cases, but the current costs for the patients in each case are very great. We focused in particular on use cases that in-

clude the hospital inpatient setting, in part be- cause that is where the most data are available. However, analytics will almost certainly be use-

Analytics will almost certainly be useful across the health care continuum.

July 2014 33:7 Health Affairs 1127

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

ful across the health care continuum—for exam- ple, in evaluating the overall drivers of costs and using tools like geocoding (coding data by geog- raphy) to detect epidemics or to identify “hot spots” (of diseases, high costs, and so on). Both predicting outcomes of patients—such as who will be a high-cost patient, be readmitted, or suffer an adverse event—and tailoring the man- agement of patients should result in substantial savings for the health care system. One question is to what extent to use disease-

specific models versus more general ones in big- data analytics in health care. Much of US health care organizations’ focus in their use of analytics has been on patients with one condition, such as congestive heart failure. This approach can often be effective. However, we believe that ap- proaches that address multiple conditions are likely to have a bigger impact on care outcomes and cost savings in the long run. Another question is how to incorporate the

narrative text from EHRs into big-data analytics in health care. Extracting clinically relevant con- cepts using natural language processing is diffi- cult.62,63 Essential elements of the narrative, such as temporal relationships and co-references (that is, narrative that refers to more than one thing), may be lost or incorrectly assigned.64,65

Nonetheless, clinical natural language process- ing is already quite usable, and even simple ap- proaches can find 90 percent of factual informa- tion of many types.66,67 A related problem is that longitudinal follow-up is hindered by the paucity of information exchange among health systems and registries of vital statistics. Modern analytic approaches have shown de-

monstrable performance gains in other indus- tries and are markedly different from the typical data analytic approaches used in health care. The health care system has generally used simple decision tree or logistic regression models, in part because these often have to be implemented under time constraints at the point of care. EHRs make it possible to use models of diag-

nosis and care that combine thousands of dis- parate measurements to generate evidence in real time. Thesemodels can be farmore complex than their predecessors: For example, instead of identifying one or two key markers, such as smoking and high blood pressure, complex ana- lyticmodels cancombine subtle cues froma large number of markers. This increased complexity makes the newmodelsmore difficult to interpret and their reliability less easy to assess, compared to previous models. Other industries have grown accustomed to

running mission-critical systems using such complex and advanced approaches while also establishing reliability—typically through exten-

sive test implementations before deployment in production. Attention must be paid to the gen- eralizability of existing results inmodels’ perfor- mance to evaluate the size and scope of appro- priate test implementations in health care.16

Another limiting factor in the use of analytics in the health care setting has been delivering predictions to providers—especially in real time—to enable action. That is becoming pro- gressively easierwithEHRsandmodern commu- nication tools. However, many EHRs do not in- clude robust event engines—tools that sift through data and use rules to notify providers when appropriate—or robust approaches for de- termining which provider is responsible for a specific patient at a given time.

Policy Implications Our observations have a number of implications with respect to research, regulation, payment, and privacy, among other areas. Research Regarding research, more system-

atic evaluation is needed to move from potential to realization in many areas. Specifically, we be- lieve that federal support for research that eval- uates the use of analytics and big data to address the six use cases discussed above is warranted. Especially useful would be studies of the tailor- ingof solutions forhigh-riskpatients and theuse of multiple streams of data—in particular, from sensor technologies—for the prediction of ad- verse events and for therapy selection for pa- tients with diseases that affect multiple organ systems. Yet to be determined is the extent to which

hypothesis-driven (the traditional approach) or hypothesis-free approaches (such as those used in data mining) are appropriate. Also still unclear is the relative importance of developing specific approaches and of implementing and disseminating them. We believe that there is more need to develop approaches, because pay-

Federal support for research that evaluates the use of analytics and big data to address the six use cases is warranted.

Predictive Analytics

1128 Health Affairs July 2014 33:7

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

ment reform is likely to offer strong incentives for their implementation and dissemination.

Regulation From the regulation perspective, a key question will be to what extent these pre- dictive analytic approaches will be regulated by the Food and Drug Administration (FDA). In August 2013 the Food and Drug Administration Safety and Innovation Act working group tasked with evaluating emerging health information technology (IT) published a draft report con- cluding that FDA premarket review of health IT applications, such as analytics, would not be beneficial.68 The report also concluded that if health IT applications used analytics to deliver strong clinical decision support or were embed- ded in devices, they might require FDA review. Thus, there is clearly a tension between the need for regulatory oversight and for protection of the public. The FDA has already released another report on this topic in 2014.69

Payment With respect to payment, strategies such as the accountable care organizationmodel that encourage organizations to invest in cost reduction will likely accelerate the adoption of

analytics. However, as many experts have com- mented, the current provisions of the Affordable CareActmay not be sufficient on their own to get providers to focus on costs.70

Privacy Regarding privacy, there are many thorny issues, as the growing controversy over the National Security Agency’s collection of data about private phone calls has illustrated. Many people will not wish to have some types of data about them linked with other types of data, and this issue may be even more sensitive in health care than in other domains. However, Ruth Faden and coauthors have argued that in a just health care system, patients have amoral obliga- tion to contribute to the common purpose of improving the quality and value of clinical care.71

Policy makers have been reluctant to alter the provisions of the Health Information Portability andAccountabilityAct (HIPAA)of 1996,which is themajor legislation related to privacy and secu- rity issues in health care. However, the act does not address many issues that will become rele- vant as more disparate data sources become linked.

Conclusion Big data, including analytics, is a powerful tool that will be as useful in health care as it has been in other industries. The choice of these specific use cases that we have discussed in this article canbedebated.Nonetheless,webelieve that they will be among those that deliver the greatest value for health care organizations in the near term. This general approach has great potential for improving value in health care. We believe that organizations that employ it in many do- mains will benefit, especially under payment reform. ▪

David Bates is on the clinical advisory board for and has received research funding from EarlySense, a company that uses analytics and sensor

technology to improve care. The authors thank Stephanie Klinkenberg-Ramirez for her assistance with the preparation of this article. Funding was provided by

Framework and Action Plan for Predictive Analytics Grant No. 3861 from the Gordon and Betty Moore Foundation.

Current provisions of the Affordable Care Act may not be sufficient on their own to get providers to focus on costs.

July 2014 33:7 Health Affairs 1129

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

NOTES

1 Davis K. 2012 annual report: presi- dent’s message—health care reform: a journey [Internet]. New York (NY): Commonwealth Fund; 2012 Dec 26 [cited 2014 Apr 24]. Available from: http://www.commonwealthfund .org/Publications/Annual-Report- Essays/2012/Dec/Health-Care- Reform-A-Journey.aspx

2 Porter ME. What is value in health care? N Engl J Med. 2010;363(26): 2477–81.

3 Mechanic RE, Altman SH, McDonough JE. The new era of payment reform, spending targets, and cost containment in Massachu- setts: early lessons for the nation. Health Aff (Millwood). 2012;31(10): 2334–42.

4 Xierali IM, Hsiao CJ, Puffer JC, Green LA, Rinaldo JC, Bazemore AW, et al. The rise of electronic health record adoption among family physicians. Ann Fam Med. 2013; 11(1):14–9.

5 Bassi J, Lau F. Measuring value for money: a scoping review on eco- nomic evaluation of health infor- mation systems. J Am Med Inform Assoc. 2013;20(4):792–801.

6 Davenport TH, Harris JG. Compet- ing on analytics: the new science of winning. Boston (MA): Harvard Business School Press; 2007.

7 Ferrucci D, Brown E, Chu-Carroll J, Fan J, Gondek D, Kalyanpur AA, et al. Building Watson: an overview of the DeepQA project. AI Magazine. 2010;31(3):59–79.

8 Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: an introduction. J Am Med Inform Assoc. 2011;18(5): 544–51.

9 Murdoch TB, Detsky AS. The inevi- table application of big data to health care. JAMA. 2013;309(13):1351–2.

10 Tenenbaum JD, Sansone SA, Haendel M. A sea of standards for omics data: sink or swim? J Am Med Inform Assoc. 2013;21(2):200–3.

11 Agaku IT, Adisa AO, Ayo-Yusuf OA, Connolly GN. Concern about secu- rity and privacy, and perceived con- trol over collection and use of health information are related to with- holding of health information from healthcare providers. J Am Med In- form Assoc. 2013;21(2):374–8.

12 Ohno-Machado L. To share or not to share: that is not the question. Sci Transl Med. 2012;4(165):165cm15.

13 Schoenman JA, Chockley N. Under- standing U.S. health care spending [Internet]. Washington (DC): Na- tional institute for Health Care Management Research and Educa- tional Foundation; 2011 Jul [cited 2014 Apr 24]. (Data Brief). Available from: http://www.nihcm.org/ images/stories/NIHCM-CostBrief- Email.pdf

14 Nelson L. Lessons from Medicare’s

demonstration projects on disease management and care coordination [Internet]. Washington (DC): Con- gressional Budget Office; 2012 Jan [cited 2014 Apr 24]. (Working Paper No. 2012-01). Available from: http:// www.cbo.gov/sites/default/files/ cbofiles/attachments/WP2012- 01_Nelson_Medicare_DMCC_ Demonstrations.pdf

15 Weil E, Ferris T, Meyer G. Fact sheet—phase one: MGH Medicare demonstration project for high-cost beneficiaries [Internet]. Boston (MA): Massachusetts General Hos- pital Physician Group; [cited 2014 Apr 24]. Available from: http:// www.massgeneral.org/News/assets/ pdf/CMS_project_phase1Fact Sheet.pdf

16 Paxton C, Niculescu-Mizil A, Saria S. Developing predictive models using electronic medical records: chal- lenges and pitfalls. AMIA Annu Symp Proc. 2013;2013:1109–15.

17 Turner-McGrievy GM, Beets MW, Moore JB, Kaczynski AT, Barr- Anderson DJ, Tate DF. Comparison of traditional versus mobile app self- monitoring of physical activity and dietary intake among overweight adults participating in an mHealth weight loss program. J Am Med In- form Assoc. 2013;20(3):513–8.

18 Jiang X, Menon A, Wang S, Kim J, Ohno-Machado L. Doubly Optimized Calibrated Support Vector Machine (DOC-SVM): an algorithm for joint optimization of discrimination and calibration. PLoS One. 2012;7(11): e48823.

19 Jiang X, Boxwala AA, El-Kareh R, Kim J, Ohno-Machado L. A patient- driven adaptive prediction technique to improve personalized risk esti- mation for clinical decision support. J Am Med Inform Assoc. 2012; 19(e1):e36–42.

20 Freund T, Kunz CU, Ose D, Szecsenyi J, Peters-Klimm F. Pat- terns of multimorbidity in primary care patients at high risk of future hospitalization. Popul Health Manag. 2012;15(2):119–24.

21 Jencks SF, Williams MV, Coleman EA. Rehospitalizations among pa- tients in the Medicare fee-for-service program. N Engl J Med. 2009; 360(14):1418–28.

22 Clancy CM. Commentary: reducing hospital readmissions: aligning fi- nancial and quality incentives. Am J Med Qual. 2012;27(5):441–3.

23 Kocher RP, Adashi EY. Hospital re- admissions and the Affordable Care Act: paying for coordinated quality care. JAMA. 2011;306(16):1794–5.

24 Bayati M. Data-driven decision making in healthcare systems [In- ternet]. Redmond (WA): Microsoft Corporation; 2011 Sep 27 [cited 2014 Apr 24]. (20th Anniversary Lecture Series). Available from: http://

research.microsoft.com/apps/ video/default.aspx?id=159290

25 Madan A, Cebrian M, Lazer D, Pentland A. Social sensing for epi- demiological behavior change. In: Proceedings of the 12th ACM Inter- national Conference on Ubiquitous Computing. New York (NY): ACM Press; 2010. p. 291–300.

26 Apgar V. The newborn (Apgar) scoring system. Reflections and ad- vice. Pediatr Clin North Am. 1966; 13(3):645–50.

27 Finster M,Wood M. The Apgar score has survived the test of time. Anes- thesiology. 2005;102(4):855–7.

28 Saria S, Koller D, Penn A. Learning individual and population level traits from clinical temporal data [Inter- net]. Submitted to: Neural Informa- tion Processing Systems (NIPS) Foundation. Predictive Models in Personalized Medicine workshop; Whistler, BC; 2010 Dec 11 [cited 2014 May 22]. Available from: https:// sites.google.com/site/personalmed models/proceedings/Saria.pdf

29 Saria S, Rajani AK, Gould J, Koller D, Penn AA. Integration of early phys- iological responses predicts later illness severity in preterm infants. Sci Transl Med. 2010;2(48):48ra65.

30 Escobar GJ. The neonatal “sepsis work-up”: personal reflections on the development of an evidence- based approach toward newborn infections in a managed care orga- nization. Pediatrics. 1999;103(Suppl E1):360–73.

31 Escobar GJ, Li DK, Armstrong MA, Gardner MN, Folck BF, Verdi JE, et al. Neonatal sepsis workups in infants >=¼ 2000 grams at birth: a population-based study. Pediatrics. 2000;106(2 Pt 1):256–63.

32 Mukhopadhyay S, Eichenwald EC, Puopolo KM. Neonatal early-onset sepsis evaluations among well- appearing infants: projected impact of changes in CDC GBS guidelines. J Perinatol. 2013;33(3):198–205.

33 Puopolo KM, Draper D, Wi S, Newman TB, Zupancic J, Lieberman E, et al. Estimating the probability of neonatal early-onset infection on the basis of maternal risk factors. Pedi- atrics. 2011;128(5):e1155–63.

34 Escobar GJ, Puopolo KM,Wi S, Turk BJ, KuzniewiczMW,Walsh EM, et al. Stratification of risk of early-onset sepsis in newborns ≥34 weeks’ ges- tation. Pediatrics. 2014;133(1):30–6.

35 Ohno-Machado L, Resnic FS, Matheny ME. Prognosis in critical care. Annu Rev Biomed Eng. 2006; 8:567–99.

36 Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a se- verity of disease classification sys- tem. Crit Care Med. 1985;13(10): 818–29.

37 Zimmerman JE, Kramer AA. Out- come prediction in critical care: the

Predictive Analytics

1130 Health Affairs July 2014 33:7

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

Acute Physiology and Chronic Health Evaluation models. Curr Opin Crit Care. 2008;14(5):491–7.

38 Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3—from evaluation of the pa- tient to evaluation of the intensive care unit. Part 1: objectives, methods and cohort description. Intensive Care Med. 2005;31(10):1336–44.

39 Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk- adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5): 446–53.

40 Kodali BS. Capnography outside the operating rooms. Anesthesiology. 2013;118(1):192–201.

41 Rothman MJ, Rothman SI, Beals J 4th. Development and validation of a continuous measure of patient con- dition using the electronic medical record. J Biomed Inform. 2013; 46(5):837–48.

42 Finlay GD, Rothman MJ, Smith RA. Measuring the Modified Early Warning Score and the Rothman Index: advantages of utilizing the electronic medical record in an early warning system. J Hosp Med. 2014; 9(2):116–9.

43 Rothman SI, Rothman MJ, Solinger AB. Placing clinical variables on a common linear scale of empirically based risk as a step towards con- struction of a general patient acuity score from the electronic health record: a modelling study. BMJ Open. 2013;3(5).

44 Donald MJ, Paterson B. End tidal carbon dioxide monitoring in pre- hospital and retrieval medicine: a review. Emerg Med J. 2006;23(9): 728–30.

45 Brown H, Terrence J, Vasquez P, Bates DW, Zimlichman E. Continu- ous monitoring in an inpatient medical-surgical unit: a controlled clinical trial. Am J Med. 2014; 127(3):226–32.

46 Jha AK, ChanDC, Ridgway AB, Franz C, Bates DW. Improving safety and eliminating redundant tests: cutting costs in U.S. hospitals. Health Aff (Millwood). 2009;28(5):1475–84.

47 Bates DW, Su L,Yu DT, Chertow GM, Seger DL, Gomes DR, et al. Mortality and costs of acute renal failure as- sociated with amphotericin B thera- py. Clin Infect Dis. 2001;32(5): 686–93.

48 Moorman JR, Carlo WA, Kattwinkel J, Schelonka RL, Porcelli PJ, Navarrete CT, et al. Mortality re- duction by heart rate characteristic

monitoring in very low birth weight neonates: a randomized trial. J Pe- diatr. 2011;159(6):900–6.

49 Quinn JA,Williams CK, McIntosh N. Factorial switching linear dynamical systems applied to physiological condition monitoring. IEEE Trans Pattern Anal Mach Intell. 2009; 31(9):1537–51.

50 Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, et al. Incidence of adverse drug events and potential adverse drug events. Im- plications for prevention. ADE Pre- vention Study Group. JAMA. 1995; 274(1):29–34.

51 Bates DW. Drugs and adverse drug reactions: how worried should we be? JAMA. 1998;279(15):1216–7.

52 Sakuma M, Bates DW, Morimoto T. Clinical prediction rule to identify high-risk inpatients for adverse drug events: the JADE Study. Pharma- coepidemiol Drug Saf. 2012;21(11): 1221–6.

53 Field TS, Gurwitz JH, Harrold LR, Rothschild J, DeBellis KR, Seger AC, et al. Risk factors for adverse drug events among older adults in the ambulatory setting. J Am Geriatr Soc. 2004;52(8):1349–54.

54 Fortin M, Bravo G, Hudon C, Vanasse A, Lapointe L. Prevalence of multimorbidity among adults seen in family practice. Ann Fam Med. 2005;3(3):223–8.

55 Wolff JL, Starfield B, Anderson G. Prevalence, expenditures, and com- plications of multiple chronic con- ditions in the elderly. Arch Intern Med. 2002;162(20):2269–76.

56 Petri M. Systemic lupus erythema- tosus: 2006 update. J Clin Rheuma- tol. 2006;12(1):37–40.

57 Hummers LK, Wigley FM. Sclero- derma. New York (NY): Lange Medical Books/McGraw Hill; 2013.

58 Leeper NJ, Bauer-Mehren A, Iyer SV, Lependu P, Olson C, Shah NH. Practice-based evidence: profiling the safety of cilostazol by text- mining of clinical notes. PLoS One. 2013;8(5):e63499.

59 Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med. 2011;365(19):1758–9.

60 Natter MD, Quan J, Ortiz DM, Bousvaros A, Ilowite NT, Inman CJ, et al. An i2b2-based, generalizable, open source, self-scaling chronic disease registry. J Am Med Inform Assoc. 2013;20(1):172–9.

61 National Patient-Centered Clinical Research Network [home page on the Internet]. Boston (MA): PCORnet; [cited 2014 Apr 24].

Available from: http://pcornet.org/ 62 Meystre SM, Savova GK, Kipper-

Schuler KC, Hurdle JF. Extracting information from textual documents in the electronic health record: a review of recent research.Yearb Med Inform. 2008:128–44.

63 Saria S, McElvain G, Rajani AK, Penn AA, Koller DL. Combining structured and free-text data for au- tomatic coding of patient outcomes. AMIA Annu Symp Proc. 2010; 2010:712–6.

64 Sun W, Rumshisky A, Uzuner O. Temporal reasoning over clinical text: the state of the art. J Am Med Inform Assoc. 2013;20(5):814–9.

65 Uzuner O, Bodnari A, Shen S, Forbush T, Pestian J, South BR. Evaluating the state of the art in coreference resolution for electronic medical records. J Am Med Inform Assoc. 2012;19(5):786–91.

66 D’Avolio LW, Nguyen TM, Goryachev S, Fiore LD. Automated concept-level information extraction to reduce the need for custom software and rules development. J Am Med Inform As- soc. 2011;18(5):607–13.

67 LePendu P, Iyer SV, Bauer-Mehren A, Harpaz R, Mortensen JM, Podchiyska T, et al. Pharmacovigi- lance using clinical notes. Clin Pharmacol Ther. 2013;93(6):547–55.

68 Bates DW. Draft FDASIA Committee report [Internet]. Silver Spring (MD): Food and Drug Administra- tion; [cited 2014 Apr 25]. Available from: http://www.healthit.gov/ facas/sites/faca/files/FDASIA RecommendationsDraft030913_v2 .pdf

69 Food and Drug Administration. FDASIA health IT report: proposed strategy and recommendations for a risk-based framework [Internet]. Silver Spring (MD): FDA; 2014 Apr [cited 2014 May 5]. Available from: http://www.fda.gov/downloads/ AboutFDA/CentersOffices/Officeof MedicalProductsandTobacco/ CDRH/CDRHReports/UCM391521 .pdf

70 Noble DJ, Casalino LP. Can ac- countable care organizations im- prove population health?: should they try? JAMA. 2013;309(11): 1119–20.

71 Faden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a de- parture from traditional research ethics and clinical ethics. Hastings Cent Rep. 2013;(S1):S6–27.

July 2014 33:7 Health Affairs 1131

by Rachel McCartney on July 10, 2014Health Affairs by content.healthaffairs.orgDownloaded from

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.