Evaluation Plan Project: Literature Review

Contents lists available at ScienceDirect

International Journal of Medical Informatics

journal homepage: www.elsevier.com/locate/ijmedinf

Development and evaluation of an osteoarthritis risk model for integration into primary care health information technology Jason E. Blacka,*, Amanda L. Terryb, Daniel J. Lizottec a Graduate Program in Epidemiology & Biostatistics, Western University, 1151 Richmond Street, London, Ontario, N6A 5C1, Canada b Department of Family Medicine, Department of Epidemiology & Biostatistics, Schulich Interfaculty Program in Public Health, Western University, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada c Department of Computer Science, Department of Epidemiology & Biostatistics, Schulich Interfaculty Program in Public Health, Department of Statistical and Actuarial Sciences, 1151 Richmond Street, Western University, London, Ontario, N6A 3K7, Canada

A R T I C L E I N F O

Keywords: Prognostic prediction model Risk Electronic medical record Osteoarthritis CPCSSN

A B S T R A C T

Background: We developed and evaluated a prognostic prediction model that estimates osteoarthritis risk for use by patients and practitioners that is designed to be appropriate for integration into primary care health in- formation technology systems. Osteoarthritis, a joint disorder characterized by pain and stiffness, causes sig- nificant morbidity among older Canadians. Because our prognostic prediction model for osteoarthritis risk uses data that are readily available in primary care settings, it supports targeting of interventions delivered as part of clinical practice that are aimed at risk reduction. Methods: We used the CPCSSN (Canadian Primary Sentinel Surveillance Network) database, which contains aggregated electronic health information from a cohort of primary care practices, to develop and evaluate a prognostic prediction model to estimate 5-year osteoarthritis risk, addressing contextual challenges of data availability and missingness. We constructed a retrospective cohort of 383,117 eligible primary care patients who were included in the cohort if they had an encounter with their primary care practitioner between 1 January 2009 and 31 December 2010. Patients were excluded if they had a diagnosis of osteoarthritis prior to their first visit in this time period. Incident cases of osteoarthritis were observed. The model was constructed to predict incident osteoarthritis based on age, sex, BMI, previous leg injury, and osteoporosis. Evaluation of the model used internal 10-fold cross-validation; we argue that internal validation is particularly appropriate for a model that is to be integrated into the same context from which the data were derived. Results: The resulting prediction model for 5-year risk of osteoarthritis diagnosis demonstrated state-of-the-art discrimination (estimated AUROC 0.84) and good calibration (assessed visually.) The model relies only on in- formation that is readily available in Canadian primary care settings, and hence is appropriate for integration into Canadian primary care health information technology. Conclusions: If the contextual challenges arising when using primary care electronic medical record data are appropriately addressed, highly discriminative models for osteoarthritis risk may be constructed using only data commonly available in primary care. Because the models are constructed from data in the same setting where the model is to be applied, internal validation provides strong evidence that the resulting model will perform well in its intended application.

1. Introduction

Prognostic prediction models (PPMs) estimate a patient’s risk of disease development [1,2] based on various predictors [3,4]. Predictors may include patient demographics (such as age and sex), family history,

lifestyle factors (such as smoking status or physical activity level), prior medical conditions, laboratory test results, radiographic imaging, or genetic markers [5]. In turn, health care practitioners and patients can make decisions informed by disease risk [6,7]. For example, a patient found to be at high risk of lung cancer may be advised by their health

https://doi.org/10.1016/j.ijmedinf.2020.104160 Received 30 December 2019; Received in revised form 28 February 2020; Accepted 24 April 2020

Abbreviations: PPM, prognostic prediction model; EMR, electronic medical record; CPCSSN, Canadian Primary Care Sentinel Surveillance Network; BMI, body mass index; ICD-9, International Classification of Disease, Ninth Revision; AUROC, Area Under the receiver operating characteristic Curve; TOARP, Tool for Osteoarthritis Risk Prediction; MOST, Multicenter Osteoarthritis Study

⁎ Corresponding author at: Graduate Program in Epidemiology & Biostatistics, Western University, 1151 Richmond Street, London, Ontario, N6A 5C1, Canada. E-mail addresses: [email protected] (J.E. Black), [email protected] (A.L. Terry), [email protected] (D.J. Lizotte).

International Journal of Medical Informatics 141 (2020) 104160

Available online 01 May 2020 1386-5056/ © 2020 Elsevier B.V. All rights reserved.

T

care provider to quit smoking. Several PPMs estimate a patient’s risk of developing osteoarthritis [8–11]; however, all existing PPMs for os- teoarthritis that we identified require information on predictors that are not routinely collected in primary care, such as the Kellgren and Lawrence grade (which requires radiographic imaging), and hence are not suitable for integration into primary care health information sys- tems. Using existing data within primary care electronic medical re- cords (EMRs) instead would eliminate the need for collection of addi- tional, oftentimes burdensome, measures and would enable real-time risk estimation at the point of care.

There is the potential for significant benefit to be derived by de- ploying such a risk engine in primary care. Affecting an estimated 13 % of adults over the age of 20, osteoarthritis causes significant morbidity in Canada [12]. This estimate increases to 29 % in adults 70 years of age and older who receive primary care [13]. Symptoms of osteoar- thritis include joint pain and stiffness [14], commonly affecting the joints of the hands, neck, lower back, hips, and knees. Osteoarthritis treatment largely consists of symptom management (e.g., non-steroidal anti-inflammatory medications for pain management), rather than treatment of underlying disease mechanisms [15]. Total joint replace- ment is often required after significant degradation of the affected joint. To mitigate this burden, prevention strategies have shown potential in reducing the incidence of osteoarthritis. For example, a diet and ex- ercise program aimed at weight loss reduced the incidence of osteoar- thritis, though not statistically significantly [16]. Injury prevention programs have been suggested as a potential strategy to prevent os- teoarthritis [17]. Interventions such as these may be improved by se- lectively targeting those at the greatest risk of osteoarthritis in order to reduce their risk. To perform this selective targeting, individualized risk estimates for osteoarthritis are required.

Designing and evaluating a PPM specifically in the primary care context leads to two important design decisions: 1) the PPM should use only data that are easily available in the primary care context where it is to be deployed, for example those that exist in EMRs already, and 2) evaluation of the PPM should reflect the population where it is to be deployed, that is, in primary care encounters. Researchers have begun to recognize the value of EMR data for research purposes more gen- erally [18]; however, data quality within these databases remains un- certain [19]. Issues such as implausible data and missing data are common in EMR data. When working with EMR data, researchers must address these contextual challenges [20].

In this work, we developed and validated a PPM to estimate a pa- tient’s five-year risk of osteoarthritis development using primary care EMR data. Ultimately, we see this model being developed into a pur- pose-built tool to be used routinely by primary care practitioners during patient encounters to: 1) deliver a quantitative assessment of osteoar- thritis risk in patients where the patient and/or primary care practi- tioner is concerned about osteoarthritis risk; and 2) act as a passive risk screening tool to identify high-risk patients who may have gone un- detected otherwise. We are confident that our learnings from this work will be of use to others who are designing and evaluating PPMs using EMR from and for primary care.

2. Methods

We developed and validated a prognostic prediction model to esti- mate the risk of osteoarthritis development within five years among Canadian adults receiving primary care. Model development was in- formed by strategies of PPM development suggested by the TRIPOD statement [21], Steyerberg [22], Lee et al. [5], and Hendriksen et al. [3]. First, we compiled a list of risk indicators for osteoarthritis devel- opment based on the existing literature. Next, we identified a cohort of patients whose risk indicator status was known at baseline and assessed whether they subsequently developed osteoarthritis within five years.

Based on this cohort, a multivariable model was constructed to enable the estimation of individual patient risk. This model’s performance was assessed in terms of its discrimination and calibration.

We began by examining existing literature to identify established predictors of osteoarthritis development. We identified: BMI (body mass index) [23–29], previous leg injury [23,25–27], leg length in- equality [30], older age [24–26,28,29], female sex [24–26,29], osteo- porosis [24], family history [29], occupation [29], and physical workload [28].

We used the CPCSSN (Canadian Primary Care Sentinel Surveillance Network) database to develop our PPM. The CPCSSN database contains de-identified patient records from 12 regional networks across Canada and includes more than 1.5 million patients [31]. Within this database are all structured patient records stretching back to 2008, including patient encounters, patient demographics, billing codes, laboratory results, prescriptions, referrals, risk indicator information, and medical procedures. The CPCSSN database contains only structured data; un- structured data (e.g., free-text clinician notes) are not available. We found CPCSSN contained data describing five of the nine risk indicators for osteoarthritis: BMI, previous leg injury, older age, female sex, and osteoporosis.

2.1. Measures of risk indicators and outcomes

As is common when working with secondary data sources such as EMRs or health administrative data, risk indicators and outcomes may not be found directly in EMR databases; we therefore developed a strategy to mitigate this contextual challenge. In consultation with ex- pert EMR users, we searched the EMR for data elements that we thought were strongly associated with identified risk indicators. For example, we considered a patient with any of the following diagnostic codes to have had a lower leg injury: ICD-9 (International Classification of Diseases – Ninth Revision) 820-29 (fracture of lower limb), ICD-9 843 (sprain or strain of hip and thigh), ICD-9 844 (sprain or strain of knee and leg), or ICD-9 928 (crushing injury to lower limb). We compiled these data elements into risk indicator definitions (Table 1).

We defined the risk indicator osteoporosis using evidence in the EMR for any suspected bone disorder, despite this not referring specifically to osteoporosis. Our definition includes use of the ICD-9 code 733 (used to note osteoporosis and other bone disorders) in the problem list, billing data, or encounter diagnosis fields; the term “osteoporosis”; or pre- scription of a medication commonly used for the treatment or preven- tion of osteoporosis. Specific diagnoses of osteoarthritis (ICD-9 code: 733.0) were unavailable.

Table 1 Risk indicator definitions for CPCSSN database.

Risk Indicator Data Source Value

Age Patient Demographics Numeric Sex Patient Demographics Male or female BMI Patient Encounter Numeric Leg Injury Billing ICD-9 Codes:

Health Condition Encounter Diagnosis

• 820−29: fracture of lower limb• 843: sprain or strain of hip and thigh• 844: sprain or strain of knee and leg• 928: crushing injury to lower limb Osteoporosis Billing ICD-9 Code:

Health Condition Encounter Diagnosis

• 733: Osteoporosis and other bone disorders

Health Condition Encounter Diagnosis Risk Factor

“osteoporosis”

Medications Alendronic acid Risedronic acid Ibandronic acid

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

2

In an effort to increase the usability of CPCSSN data, Williamson et al. previously constructed a case definition for osteoarthritis by consulting with several expert EMR users [32]; this case definition was validated by estimating the sensitivity and specificity of this definition compared to chart review by an expert EMR user as a “gold standard.” The case definition for osteoarthritis consisted of a combination of billing codes and problem list diagnoses (Table 2).

2.2. Cohort construction

We included any adult patient (known to be age 18 or older) who had at least one interaction with a primary care practitioner between 1 January 2009 and 31 December 2010 and had not been previously diagnosed with osteoarthritis. The data used for this work spanned 2008–2016. The selection of interactions within this window allowed for a wide window with sufficient time for follow-up. Risk indicators were assessed on the interaction within this window by examining re- cords up to and including the date of this interaction. For example, if a patient’s records indicated any diagnosis of osteoporosis prior to this interaction, they were considered to have the risk indicator osteo- porosis. Patients diagnosed with osteoarthritis in the five years fol- lowing this interaction date were considered cases of osteoarthritis. Thus, each patient had their own start date and follow-up period. Where there were multiple interactions between 1 January 2009 and 31 December 2010, the earliest one was used.

2.3. Model building and evaluation

All available data were used in the development and validation of the model to maximize the predictive ability of the resulting model. Ten-fold cross validation was used to evaluate the model: all data were used to create ten partitions; one partition was reserved for validation, the remaining nine were used to estimate the model parameters. This step was repeated ten times, such that each partition was used for va- lidation once. Final estimates of validation measures were obtained by averaging across each of the cross-validation models. For comparison purposes and to further investigate the question of evaluation in this context, we also present abbreviated results of a single-split validation.

Missing data are a common contextual challenge of primary care EMR data analysis; we used a combination of complete case analysis and imputation to deal with missing data issues as follows. Some pa- tients’ birth years were zero; we treated these data as missing. We eliminated any patient whose age was missing (n = 259), as we in- tended to estimate risk in confirmed adults. We used multiple im- putation to address missing data for BMI and sex [33,34]. For each cross-validation step, separate imputation models were constructed for the development and validation sets, as recommended by Wahl et al. [35]. We imputed five datasets for each partition using the MICE package in R [34]. In addition to the identified risk factors for os- teoarthritis, we included several covariates in the imputation model, including diagnoses of several chronic conditions and rurality to max- imize the accuracy of the imputations (Appendix Table A1). Diagnosis of osteoarthritis was also included in the imputation models, as

suggested by Moons et al. [36]. We then assessed the plausibility of the imputations by plotting the distribution of each variable with missing data to ensure that the imputed data followed a similar distribution as the original data. While the imputation process does smooth the cov- ariate distribution somewhat, the general location and scale of the imputed and observed data are similar.

Logistic regression was used to construct a prediction model for the development of osteoarthritis using the entire cohort; this model was produced by combining models from the five imputed datasets using Rubin’s rules. All risk indicators were included in the model in their original form (e.g., all continuous risk indicators were included as continuous variables; no binning was performed). Investigations per- formed on a similar dataset did not reveal non-linear associations be- tween risk indicators and the outcome: log-transformation of con- tinuous variables; the addition of polynomial transformations of continuous variables; and the use of generalized additive models all resulted in models that performed similarly to logistic regression. The simpler logistic regression model was preferred.

Model discrimination and calibration were evaluated using cross- validation. As discussed, AUROC was used to assess discrimination; a precision-recall curve was also used to investigate the precision and recall of the model across various thresholds. Precision (or positive predictive value) is the proportion of true positives amongst all pre- dicted positives. Recall (or sensitivity) is the proportion of predicted positives amongst all true positives. The precision recall curve allows us to examine both precision and recall without setting an arbitrary risk threshold [37]. To evaluate calibration, a calibration plot was used because measures such as the Hosmer-Lemeshow goodness-of-fit test have been shown to be oversensitive in large sample sizes [5]. A cali- bration plot displays the average predicted risk within each risk decile against the observed proportion of patients who develop osteoarthritis within that decile (i.e., observed risk). The calibration plot of a model that displays good calibration should closely follow a line with an in- tercept of zero and a slope of one, demonstrating strong agreement between the estimated and observed risks. Additionally, we present calibration in the large and calibration slope to further assess calibra- tion.

Other modelling techniques were considered, including Cox pro- portional hazards; however, our goal is to predict and convey risk ra- ther than to establish the relationship between outcomes and covari- ates, and using a Cox model for this purpose would require the additional step of estimating baseline risk. Furthermore, although a survival analysis method could take into account censoring, the EMR data lack censoring information in the way it would be conceived in a traditional longitudinal cohort study because it is not clear when a primary care patient is “lost to follow-up”; we illustrate this issue using a sensitivity analysis that demonstrates that any such censoring would have limited impact on our model. Therefore, we chose to forego sur- vival analysis for this work and opt for the simpler logistic regression model.

3. Results

The final cohort was composed of 383,117 patients (Fig. 1). Patient characteristics were typical of a primary care population, as they were slightly older and more likely to be female [38–40] (Table 3). After five years of follow-up, 12,803 (3.3 %) patients developed osteoarthritis.

Data were commonly missing for BMI, while sex was almost never missing (Table 4). Multiple imputation was used to address missing data for BMI and sex.

Kernel density estimates of the distribution over the five imputed datasets (Fig. 2) demonstrate that the distributions of the imputed va- lues are similar to the distributions of the original (unimputed) values.

Table 2 Validated case definition for osteoarthritis.

Billing Problem List

Any occurrence of the following codes: Any occurrence of the following codes:

• 715, Osteoarthritis and allied disorders

• 721, Spondylosis and allied disorders

• 715, Osteoarthritis and allied disorders

• 721, Spondylosis and allied disorders

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

3

The final model is given in Table 5. All risk indicators were sig- nificantly associated with incident osteoarthritis diagnosis.

Internal validation of the model based on cross-validation yielded a high AUROC (0.84, 95 % bootstrapped CI 0.83 to 0.85). Calibration in the large (−0.00002) and the calibration slope (0.999) indicated ex- cellent calibration, as would be expected due to our use of validation data derived from the same population as for model development. Visual inspection of the calibration plot shows that predicted risk clo- sely followed the observed risk (Fig. 3). Risk was slightly under- estimated among lower-risk patients, while the risk of moderate to high-risk patients was slightly overestimated. We also performed a single-split evaluation of our model for comparison purposes using 238,875 for training and 144,242 for validation to ensure a precise estimate of AUROC. This resulted in an identical estimated AUROC (0.84) with a 95 % bootstrapped CI of 0.83 to 0.84.

As seen in the precision-recall curve shown in Fig. 4, the model has moderate precision-recall characteristics but is able to identify those with higher risk.

4. Discussion

We produced a prognostic prediction model for the diagnosis of osteoarthritis using EMR data that are commonly available in primary care. We see this model ultimately being used in two ways in Canadian primary care settings. First, the model can provide estimates of os- teoarthritis risk on demand when requested by a provider or patient who is interested in osteoarthritis risk. Second, the model can operate in the background of the provider’s EMR system during all patient visits and automatically flag any patients whose osteoarthritis risk surpasses some prespecified threshold, or to order patients from highest-risk to lowest-risk.

Our model compares extremely favourably with existing work. Existing models for osteoarthritis risk estimation include the Tool for Osteoarthritis Risk Prediction (TOARP) [8]; the Nottingham knee os- teoarthritis risk prediction models [9]; and models derived from data from the Rotterdam Study-1 [10] and the Multicenter Osteoarthritis Study (MOST) [11]. These models estimate risk of knee osteoarthritis, whereas our model estimates risk of osteoarthritis in any joint. All ex- isting models were constructed using population-based cohorts ranging in size from 400 to 3000 people who were assessed using interviews, physical examinations, and laboratory tests, including radiographic imaging. In contrast, our model was constructed using existing EMR data from a primary care population of over 380,000 patients. All models, including ours, used multivariable logistic regression to con- struct the prediction model. Internal validation of the existing models reported AUROC ranging from 0.70 to 0.79. Our model had the highest discriminative ability (0.84) and hence we consider it state-of-the-art for its purpose of a primary care tool.

Our model has several limitations. First, not all risk indicators for osteoarthritis were available within the CPCSSN database: data de- scribing family history, occupation, leg length inequality, and physical workload were not available. We were unable to link to additional databases to obtain these data. Thus, estimated risk for those who possess these uncaptured risk indicators will be an underestimation of their true risk. Second, there are elements of a patient’s treatment that we may not be able to observe. As such, we cannot adjust for the impact this treatment potentially has on patient risk. Third, the model we produced estimates a patient’s risk of osteoarthritis without specifying the joint affected. However, interventions to reduce osteoarthritis risk are typically not joint specific [16], thus knowing which joint is likely to develop osteoarthritis does not inform prevention.

The precision of our model was somewhat limited for larger recall thresholds (Fig. 4). However, this is not of great concern for this par- ticular application because the cost associated with the treatment of false positives in this context is minimal; most treatments for osteoar- thritis consist of lifestyle modifications with little to no risk of harm [16]. Thus, the lower precision of our model is not alarming. Ad- ditionally, the burden on primary care practitioners and patients using our model is low; no additional measures beyond those routinely col- lected in the EMR are required. The risk information gained from our model comes at minimal cost to the practitioner and may be used to inform further targeted screening that gathers additional tailored in- formation.

Our model did not account for censoring of patients. This followed from the assumption that if a primary care patient did not seek care from their primary care practitioner and receive an osteoarthritis di- agnosis, they did not develop osteoarthritis. Given the population (pa- tients who visit their doctors) and given that osteoarthritis diagnosis is

Fig. 1. Patient flowchart.

Table 3 Descriptive statistics for the dataset.

CPCSSN (n = 383,117)

Age, median (IQR) 44 (30–59) BMI, median (IQR) 26.6 (23.3–30.7) Female, n (%) 221,021 (57.6 %) Prior leg injury, n (%) 10,893 (2.8 %) Prior diagnosis of osteoporosis, n (%) 11,647 (3.0 %)

Table 4 Predictors with missing data.

Predictor CPCSSN (n = 383,117)

BMI, n (%) 256,413 (66.9) Sex, n (%) 59 (0.02)

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

4

often instigated by patients reporting their symptoms, we argue that this is a reasonable assumption. We nonetheless recognize that patients may have been censored through “loss to follow-up” (perhaps because they changed providers) or died during the study; however, unlike in some prospective studies, we are not able to directly observe this cen- soring. To investigate the impact of censoring, we conducted a sensi- tivity analysis whereby we excluded patients who did not have at least one interaction with their primary care practitioner after the end of their follow-up window. This ensured that patients in the remaining cohort were unlikely to have been lost to follow-up. This sensitivity analysis revealed that the potential impact of censoring on the model was minimal: model estimates based on this restricted cohort were si- milar to those of the original cohort (Appendix Table A2); model per- formance remained strong as well (AUROC: 0.83, 95 % CI 0.82 to 0.84; see Appendix Fig. A1 for calibration plot).

One consideration that may or may not be a limitation depending on one’s goal is the potential lack of generalizability of our model to the general population, since it was derived from primary care data. If the goal of the model was to be deployed in a population-level public health campaign aimed at reducing osteoarthritis incidence, for example, it may not be an appropriate tool. However, the use of primary care EMR data positions our model strongly for deployment in the primary care setting. The data used for risk estimation are those already currently collected within EMR systems; no additional measures need to be col- lected in order to use our prediction model. This aspect is unique to our model: all other existing models require some radiographic imaging that is not routinely collected. This consideration also supports the use of internal validation as an approach for evaluation; given that the

model is anticipated to be applied to the same population from which it was derived, internal validation is an appropriate way to evaluate performance. Specifically, our model is most appropriate for use in the

Fig. 2. Kernel density estimates for the marginal distribution of the five imputed datasets (red, shorter peaks) and the original data (blue, taller peaks) (left: development sets; right: validation sets) (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.).

Table 5 Model estimates.

Reference category/units β coefficient (95% CI) Odds ratio (95 % CI) P value

(Intercept) −8.13 (−8.23 to −8.02) — <0.001 Age Years 0.059 (0.058 to 0.060) 1.061 (1.059–1.062) <0.001 BMI kg/m2 0.042 (0.039 to 0.044) 1.043 (1.040–1.045) <0.001 Sex Male (Reference) (Reference)

Female 0.21 (0.17 to 0.25) 1.24 (1.19–1.29) <0.001 Prior leg injury No (Reference) (Reference)

Yes 1.61 (1.54–1.67) 5.00 (4.68–5.34) <0.001 Prior diagnosis of osteoporosis No (Reference) (Reference)

Yes 0.92 (0.86 to 0.99) 2.52 (2.37–2.68) <0.001

CI: confidence interval; BMI: body mass index.

Fig. 3. Calibration plot based on validation sets.

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

5

Canadian primary care setting, as data were derived from Canadian primary care EMRs. Use in new regions would ideally be preceded by model ‘updating’ using data from the new region [41].

A related issue arises given that we have used what might be termed “surrogate” measures of known risk factors. For example, consider the collection of diagnosis codes we have used to indicate a leg injury. Because we do not have gold standard information for leg injuries, we do not know the accuracy of this surrogate; however, we have shown that the surrogate is predictive of osteoarthritis in its own right, and so long as it is available at the time of prediction (which it would be if the model were deployed in a similar primary care context as that which generated the data) then it is a valid and useful risk indicator.

It is important to note that our osteoarthritis prediction model quantifies patient risk to guide decision making and identify high risk patients. As with all prediction models, it would be inappropriate to infer causality between risk indicators and osteoarthritis based on our model. The recommendation of any intervention to reduce risk of os- teoarthritis should be based on evidence demonstrating the effective- ness of that intervention in reducing osteoarthritis risk.

5. Conclusions

Primary care EMRs are a rich, yet underutilized, source of long- itudinal health data that can support the development of novel tools for integration into primary care health information systems. Our work demonstrates the utility of these data for constructing PPMs despite contextual challenges such as missing data, using an osteoarthritis risk model as a success story, and provides a strategy and rationale for in- ternal validation. Two key future directions for this work will be to 1) design and evaluate strategies for incorporating the model into clinical

workflows as appropriate and 2) to consider and evaluate more com- plex PPMs both for osteoarthritis and other diseases, for example those derived using machine learning techniques; this second direction will require a careful trade-off between potential performance improve- ments and interpretability, and will thus go hand-in-hand with activ- ities that evaluate how best to enhance primary care health information technology.

Summary Points

- Developing prognostic prediction models (PPMs) presents contextual challenges that depend simultaneously on data provenance and on the target population.

- We present a new, state-of-the-art PPM for osteoarthritis de- rived from primary care data that can integrate into primary care health information technology.

- We describe how using primary care EMR data and deploying to primary care influences model design and evaluation choices.

Author statement

Lead author was JB. All authors initiated the research idea. JB drafted the research idea. JB wrote the article that is being submitted. AT and DL revised the article. JB extracted and analyzed the CPCSSN data. DL and AT supported the methodology development and con- tributed to the writing of the article. Each author has read and approved the final version of this article.

Ethics

Ethics approval was obtained from the Western University Research Ethics Board #107572.

Consent for publication

Not applicable as no individual person’s data were presented.

Availability of data and materials

The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) database is not publicly accessible, in keeping with the intent of the agreement made with primary health care practitioners con- tributing to the CPCSSN database. Researchers can request data from CPCSSN directly.

Funding

Funding for this research was provided by the Natural Sciences and Engineering Research Council of Canada. The funding source played no role in this research.

Declaration of Competing Interest

The authors have no conflicts of interest to declare.

Fig. 4. Precision-recall curve based on validation sets.

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

6

Appendix A

Table A1 Covariates included in imputation model.

Covariates for imputation model

• Previous diagnosis of: o Diabetes o Hypertension o Depression o Alcohol use disorder o Epilepsy o Schizophrenia o Anxiety disorder o Cancer o Cardiovascular disease o Chronic obstructive pulmonary disease o Rheumatoid arthritis o Chronic kidney disease

• Rurality• Income

Table A2 Model estimates based on sensitivity analysis.

Reference category/units β coefficient (95% CI) Odds ratio (95 % CI) P value

(Intercept) −8.22 (−8.35 to −8.10) — <0.001 Age Years 0.064 (0.062 to 0.065) 1.066 (1.064–1.067) <0.001 BMI kg/m2 0.040 (0.037 to 0.043) 1.041 (1.038–1.043) <0.001 Sex Male (Reference) (Reference)

Female 0.25 (0.20 to 0.29) 1.28 (1.23–1.34) <0.001 Prior leg injury No (Reference) (Reference)

Yes 1.59 (1.52–1.66) 4.90 (4.56–5.28) <0.001 Prior diagnosis of osteoporosis No (Reference) (Reference)

Yes 0.82 (0.75 to 0.89) 2.27 (2.12–2.44) <0.001

CI: confidence interval; BMI: body mass index.

Fig. A1. Calibration plot based on sensitivity analysis.

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

7

Appendix B. Supplementary data

Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.ijmedinf.2020.104160.

References

[1] Public Health Agency of Canada, Canadian Chronic Disease Indicators, Quick Stats, (2017) Edition. Ottawa, ON; 2017.

[2] R. Birtwhistle, R. Morkem, G. Peat, T. Williamson, M.E. Green, S. Khan, et al., Prevalence and management of osteoarthritis in primary care: an epidemiologic cohort study from the Canadian Primary Care Sentinel Surveillance Network, C Open 3 (3) (2015) E270–5.

[3] M. Doherty, J.W.J. Bijlsma, N. Arden, D. Hunter, N. Dalbeth, Osteoarthritis and Crystal Arthropathy, 3rd ed., Oxford University Press, 2016 507 p..

[4] M.C. Hochberg, R.D. Altman, K.T. April, M. Benkhalti, G. Guyatt, J. Mcgowan, et al., American College of Rheumatology 2012 recommendations for the use of nonpharmacologic and pharmacologic therapies in osteoarthritis of the hand, hip, and knee, Arthritis Care Res. (Hoboken) (2012).

[5] J. Runhaar, M. van Middelkoop, M. Reijman, S. Willemsen, E.H. Oei, D. Vroegindeweij, et al., Prevention of knee osteoarthritis in overweight females: the first preventive randomized controlled trial in osteoarthritis, Am. J. Med. 128 (8) (2015) 888–895.e4.

[6] C.A. Emery, E.M. Roos, E. Verhagen, C.F. Finch, K.L. Bennell, B. Story, et al., OARSI Clinical Trials Recommendations: design and conduct of clinical trials for primary prevention of osteoarthritis by joint injury prevention in sport and recreation, Osteoarthr. Cartil. 23 (5) (2015) 815–825.

[7] D.M. Lloyd-Jones, P.W.F. Wilson, M.G. Larson, A. Beiser, E.P. Leip, R.B. D’Agostino, et al., Framingham risk score and prediction of lifetime risk for coronary heart disease, Am. J. Cardiol. 94 (1) (2004) 20–24.

[8] S.A.M. Nashef, F. Roques, P. Michel, E. Gauducheau, S. Lemeshow, R. Salamon, European system for cardiac operative risk evaluation (EuroSCORE), Eur. J. Cardio- Thorac. Surg. 16 (1) (1999) 9–13.

[9] J.M.T. Hendriksen, G.J. Geersing, K.G.M. Moons, J.A.H. de Groot, Diagnostic and prognostic prediction models, J. Thromb. Haemost. 11 (Suppl 1) (2013) 129–141.

[10] E.W. Steyerberg, Y. Vergouwe, Towards better clinical prediction models: seven steps for development and an ABCD for validation, Eur. Heart J. 35 (29) (2014) 1925–1931.

[11] Y.H. Lee, H. Bang, D.J. Kim, How to establish clinical prediction models, Endocrinol. Metab. (Seoul, Korea). 31 (1) (2016) 38–44.

[12] D.T. Felson, Y. Zhang, J.M. Anthony, A. Naimark, J.J. Anderson, Weight loss re- duces the risk for symptomatic knee osteoarthritis in women. The Framingham study, Ann. Intern. Med. 116 (7) (1992) 535–539.

[13] D.T. Felson, Weight and osteoarthritis, Am. J. Clin. Nutr. 63 (3 Suppl) (1996) 430S–432S.

[14] G.B. Joseph, C.E. McCulloch, M.C. Nevitt, J. Neumann, A.S. Gersing, M. Kretzschmar, et al., Tool for osteoarthritis risk prediction (TOARP) over 8 years using baseline clinical data, X-ray, and MRI: data from the osteoarthritis initiative, J. Magn. Reson. Imaging 47 (6) (2018) 1517–1526.

[15] W. Zhang, D.F. McWilliams, S.L. Ingham, S.A. Doherty, S. Muthuri, K.R. Muir, et al., Nottingham knee osteoarthritis risk prediction models, Ann. Rheum. Dis. 70 (9) (2011) 1599–1604.

[16] H.J.M. Kerkhof, S.M.A. Bierma-Zeinstra, N.K. Arden, S. Metrustry, M. Castano- Betancourt, D.J. Hart, et al., Prediction model for knee osteoarthritis incidence, including clinical, genetic and biochemical risk factors, Ann. Rheum. Dis. 73 (12) (2014) 2116–2121.

[17] D.L. Riddle, P.W. Stratford, R.A. Perera, The incident tibiofemoral osteoarthritis with rapid progression phenotype: development and validation of a prognostic prediction rule, Osteoarthr. Cartil. 24 (12) (2016) 2100–2107.

[18] H. Carr, S. de Lusignan, H. Liyanage, S.-T. Liaw, A. Terry, I. Rafi, Defining di- mensions of research readiness: a conceptual model for primary care research networks, BMC Fam. Pract. 15 (2014) 169.

[19] S. de Lusignan, S.-T. Liaw, P. Krause, V. Curcin, M.T. Vicente, G. Michalakidis, et al., Key concepts to assess the readiness of data for international research: data quality, lineage and provenance, extraction and processing errors, traceability, and cura- tion. Contribution of the IMIA Primary Health Care Informatics Working Group, Yearb. Med. Inform. 6 (2011) 112–120.

[20] A.L. Terry, M. Stewart, S. Cejic, J.N. Marshall, S. de Lusignan, B.M. Chesworth,

et al., A basic model for assessing primary health care electronic medical record data quality, BMC Med. Inform. Decis. Mak. 19 (1) (2019) 30.

[21] K.G.M. Moons, D.G. Altman, J.B. Reitsma, J.P.A. Ioannidis, P. Macaskill, E.W. Steyerberg, et al., Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration, Ann. Intern. Med. 162 (1) (2015) W1.

[22] E.W. Steyerberg, Clinical Prediction Models, Springer New York, New York, NY, 2009 (Statistics for Biology and Health).

[23] C. Cooper, H. Inskip, P. Croft, L. Campbell, G. Smith, M. McLaren, et al., Individual risk factors for hip osteoarthritis: obesity, hip injury and physical activity, Am. J. Epidemiol. 147 (6) (1998) 516–522.

[24] K.M. Lee, C.Y. Chung, K.H. Sung, S.Y. Lee, S.H. Won, T.G. Kim, et al., Risk factors for osteoarthritis and contributing factors to current arthritic pain in South Korean older adults, Yonsei Med. J. 56 (1) (2015) 124.

[25] T. Neogi, Y. Zhang, Osteoarthritis prevention, Curr. Opin. Rheumatol. 23 (2) (2011) 185–191.

[26] V. Silverwood, M. Blagojevic-Bucknall, C. Jinks, J.L. Jordan, J. Protheroe, K.P. Jordan, Current evidence on risk factors for knee osteoarthritis in older adults: a systematic review and meta-analysis, Osteoarthr. Cartil. 23 (4) (2014) 507–515.

[27] E. Vignon, J.-P. Valat, M. Rossignol, B. Avouac, S. Rozenberg, P. Thoumie, et al., Osteoarthritis of the knee and hip and activity: a systematic international review and synthesis (OASIS), Joint Bone Spine 73 (4) (2006) 442–455.

[28] I. Vrezas, G. Elsner, U. Bolm-Audorff, N. Abolmaali, A. Seidler, Case-control study of knee osteoarthritis and lifestyle factors considering their interaction with physical workload, Int. Arch. Occup. Environ. Health 83 (3) (2010) 291–300.

[29] G.J. Leung, K.D. Rainsford, W.F. Kean, Osteoarthritis of the hand I: aetiology and pathogenesis, risk factors, investigation and diagnosis, J. Pharm. Pharmacol. 66 (3) (2014) 339–346.

[30] W.F. Harvey, M. Yang, T.D.V. Cooke, N.A. Segal, N. Lane, C.E. Lewis, et al., Association of leg-length inequality with knee osteoarthritis a cohort study, Ann. Intern. Med. 152 (5) (2010) 287–W92.

[31] R. Birtwhistle, K. Keshavjee, A. Lambert-Lanning, M. Godwin, M. Greiver, D. Manca, et al., Building a pan-Canadian primary care sentinel surveillance net- work: initial development and moving forward, J. Am. Board Fam. Med. 22 (4) (2009) 412–422.

[32] T. Williamson, M.E. Green, R. Birtwhistle, S. Khan, S. Garies, S.T. Wong, et al., Validating the 8 CPCSSN case definitions for chronic disease surveillance in a pri- mary care database of electronic health records, Ann. Fam. Med. 12 (4) (2014) 367–372.

[33] M.J. Azur, E.A. Stuart, C. Frangakis, P.J. Leaf, Multiple imputation by chained equations: what is it and how does it work? Int. J. Methods Psychiatr. Res. 20 (1) (2011) 40–49.

[34] S. van Buuren, K. Groothuis-Oudshoorn, Mice: multivariate imputation by chained equations in R, J. Stat. Softw. 45 (3) (2011) 1–67.

[35] S. Wahl, A.-L. Boulesteix, A. Zierer, B. Thorand, M.A. van de Wiel, Assessment of predictive performance in incomplete data by combining internal validation and multiple imputation, BMC Med. Res. Methodol. 16 (1) (2016) 144.

[36] K. Moons, R. Donders, T. Stijnen, F. Harrell, Using the outcome for imputation of missing predictor values was preferred, J. Clin. Epidemiol. 59 (2006) 1092–1101.

[37] T. Saito, M. Rehmsmeier, The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, PLoS One 10 (3) (2015) e0118432.

[38] C.A. Mustard, P. Kaufert, A. Kozyrskyj, T. Mayer, Sex differences in the use of health care services, N. Engl. J. Med. 338 (23) (1998) 1678–1683.

[39] K.D. Bertakis, R. Azari, L.J. Helms, E.J. Callahan, J.A. Robbins, Gender differences in the utilization of health care services, J. Fam. Pract. 49 (2) (2000) 147–152.

[40] J.X. Nie, L. Wang, C.S. Tracy, R. Moineddin, R.E. Upshur, Health care service uti- lization among the elderly: findings from the Study to Understand the Chronic Condition Experience of the Elderly and the Disabled (SUCCEED project), J. Eval. Clin. Pract. 14 (6) (2008) 1044–1049.

[41] K.J.M. Janssen, Y. Vergouwe, C.J. Kalkman, D.E. Grobbee, K.G.M. Moons, A simple method to adjust clinical prediction models to local circumstances, Can. J. Anesth. 56 (3) (2009) 194–201.

J.E. Black, et al. International Journal of Medical Informatics 141 (2020) 104160

8