helpfn

2922 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 24, NO. 10, OCTOBER 2020

EMR-Based Phenotyping of Ischemic Stroke Using Supervised Machine Learning and

Text Mining Techniques Sheng-Feng Sung , Chia-Yi Lin , and Ya-Han Hu

Abstract—Ischemic stroke is a major cause of death and disability in adulthood worldwide. Because it has highly heterogeneous phenotypes, phenotyping of is- chemic stroke is an essential task for medical research and clinical prognostication. However, this task is not a trivial one when the study population is large. Phenotyping of ischemic stroke depends primarily on manual annotation of medical records in previous studies. This article evaluated various strategies for automated phenotyping of ischemic stroke into the four subtypes of the Oxfordshire Com- munity Stroke Project classification based on structured and unstructured data from electronical medical records (EMRs). A total of 4640 adult patients who were hospital- ized for acute ischemic stroke in a teaching hospital were included. In addition to the structured items in the Na- tional Institutes of Health Stroke Scale, unstructured clini- cal narratives were preprocessed using MetaMap to identify medical concepts, which were then encoded into feature vectors. Various supervised machine learning algorithms were used to build classifiers. The study results indicate that textual information from EMRs could facilitate phe- notyping of ischemic stroke when this information was combined with structured information. Furthermore, de- composition of this multi-class problem into binary clas- sification tasks followed by aggregation of classification results could improve the performance.

Manuscript received August 6, 2019; revised December 7, 2019 and February 5, 2020; accepted February 23, 2020. Date of publication February 28, 2020; date of current version October 5, 2020. This work was supported in part by the Ministry of Science and Technology under Grant MOST 107-2314-B-705-001 and in part by the Center for Inno- vative Research on Aging Society from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan. (Corresponding author: Ya-Han Hu.)

Sheng-Feng Sung is with the Division of Neurology, Department of Internal Medicine, Ditmanson Medical Foundation Chiayi Christian Hospital, Chiayi City 600, Taiwan, with the Department of Information Management, Institute of Healthcare Information Management, National Chung Cheng University, Chiayi 62102, Taiwan, and also with the De- partment of Nursing, Min-Hwei Junior College of Health Care Manage- ment, Tainan 736, Taiwan (e-mail: [email protected]).

Chia-Yi Lin is with the Department of Information Management, In- stitute of Healthcare Information Management, National Chung Cheng University, Chiayi 62102, Taiwan (e-mail: [email protected]).

Ya-Han Hu is with the Department of Information Management, Na- tional Central University, Taoyuan 320, Taiwan, with the Center for In- novative Research on Aging Society (CIRAS), National Chung Cheng University, Chiayi 62102, Taiwan, and also with the MOST AI Biomedical Research Center, National Cheng Kung University, Tainan 701, Taiwan (e-mail: [email protected]).

This article has supplementary downloadable material available at http://ieeexplore.ieee.org, provided by the authors.

Digital Object Identifier 10.1109/JBHI.2020.2976931

Index Terms—Classification algorithm, clinical diagnosis, electronic medical records, machine learning, natural language processing, text mining.

I. INTRODUCTION

S TROKE is one of the leading causes of death and disabilityand poses a huge burden on the healthcare system. Despite the decline in stroke incidence over the past decade in some countries [1]–[3], the world overall stroke burden has increased in both men and women of all ages, ranking from fifth in 1990 to third in 2013 among all diseases [4]. Strokes are mainly classified into ischemic and hemorrhagic types and ischemic stroke constitutes approximately 80% of all strokes [1], [5]. Because ischemic stroke has highly heterogeneous phenotypes (subtypes), outcomes after ischemic stroke, including mortality, functional recovery, and hospital readmission, are partly deter- mined by its subtypes [6]–[9].

Previous genetic studies of sporadic stroke have used various classification systems for phenotyping ischemic stroke [10]. The ability to derive patient phenotypes from medical records helps define genotype-phenotype correlations and could advance medical research and clinical care [11]. Among the existing classification systems of ischemic stroke subtypes, some are principally targeted at mechanistic or etiologic stroke subtyping based on clinical findings as well as results of laboratory inves- tigations [12]. In contrast, the Oxfordshire Community Stroke Project (OCSP) classification system focuses on the anatomical location of stroke, classifying ischemic stroke into four distinct subtypes based primarily on the initial clinical findings [13].

According to the presenting symptoms and signs of a stroke patient, the OCSP classification system classify ischemic stroke into four clinical syndromes (phenotypes), that is, total anterior circulation infarcts (TACI), lacunar infarcts (LACI), partial ante- rior circulation infarcts (PACI), and posterior circulation infarcts (POCI) [13]. Table I lists the OCSP phenotypes of ischemic stroke. The OCSP classification system has been effective in pre- dicting various stroke outcomes including mortality, functional recovery, length of hospital stay, and post-stroke complications [13]–[16]. In addition, the OCSP classification can be used to prognosticate ischemic stroke patients undergoing intravenous thrombolysis [7], [17], [18]. Consequently, it has been incorpo- rated into risk score models for predicting clinical outcomes in patients with acute stroke [19]–[21].

Electronic medical records (EMRs) have progressively re- placed traditional paper records in the past decades. EMRs are

2168-2194 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

SUNG et al.: EMR-BASED PHENOTYPING OF ISCHEMIC STROKE USING SUPERVISED MACHINE LEARNING 2923

TABLE I PHENOTYPES OF ISCHEMIC STROKE ACCORDING TO THE OCSP

CLASSIFICATION SYSTEM

OCSP = Oxfordshire Community Stroke Project.

used to collect textual and numeric information on all aspects of health care, typically comprising narrative clinical notes, ra- diology and pathology reports, laboratory results, and prescrip- tions. Although narrative clinical notes contain rich and useful information, they are less amenable to utilization as compared to structured information. Therefore, various natural language processing (NLP) tools have been developed to unlock useful clinical information from narrative notes [22] in order to support automated clinical information systems. NLP techniques have been successfully applied to a number of applications ranging from assessment of care quality [23], disease identification [24], [25], to clinical decision support [26], [27]. The ability of NLP tools to extract information from medical words and expressions in EMRs forms the basis of artificial intelligence applications in medicine, which has the potential to optimize patient care, reduce medical errors, and facilitate precision medicine [28].

This study aimed to develop multi-class classification mod- els to determine the OCSP phenotype of ischemic stroke and to examine whether the inclusion of textual information from EMRs is beneficial to the classification task. Because multi-class classification problems have higher complexity than binary clas- sification problems, binarization strategies are commonly used in such situations. These strategies divide the original multi- class classification problem into multiple binary classification problems using, for example, one-vs-one (OVO) or one-vs-all (OVA) decomposition, followed by combining the outputs from the binary classifiers to obtain the final class label.

A previous study indicated that the prediction performance of multi-class classification models can be significantly affected by their aggregation algorithms [29]. That is, it is feasible to further improve the prediction performance of existing combination strategies by developing an aggregation algorithm to combine the outputs of the OVO or OVA binary classifiers. In this manner, this study addressed the following questions:

� Whether the classification performance of phenotyping of ischemic stroke can be improved by including textual information mined from EMRs?

� How do the proposed aggregation algorithms optimize the performance of multi-class classification from the sets of OVO and OVA binary classifiers?

TABLE II NATIONAL INSTITUTES OF HEALTH STROKE SCALE

LOC = level of consciousness.

� Can the proposed aggregation algorithms further improve the classification performance of phenotyping of ischemic stroke?

In experimental evaluation, we first extracted information from unstructured clinical notes using NLP tools. Next, by employing supervised machine learning techniques, we used textual information, structured information, or a combination of both to build the sets of OVO and OVA binary classifiers to assign stroke phenotypes according to the OCSP classification. Finally, we compared the performance of different classifiers and ensemble methods.

II. RELATED WORK

A. Algorithms to Assign OCSP Phenotypes

Although an OCSP syndrome diagnosis can be readily made by experienced neurologists after synthesizing clinical infor- mation at the bedside, it may be difficult for general internists or training residents to make a correct diagnosis. Furthermore, such a syndrome diagnosis is generally not recorded in medical records. Therefore, rule-based algorithms have been developed to assign the OCSP phenotypes according to a standard list of neurological symptoms and signs in previous stroke trials [30], [31]. In one study, the correlation between the list-derived OCSP diagnosis and the expert diagnosis was fair [31], with sensitivities varying from 0.50 for POCI to 0.73 for LACI and specificities from 0.75 for PACI to 0.98 for POCI. The agreement, assessed as kappa, was 0.50, 0.63, and 0.37 for TACI, LACI, and PACI, respectively. The accuracy and kappa for the overall OCSP diagnosis were 0.655 and 0.498, respectively. In the other study, diagnosis based on computed tomography (CT) of the brain was used as the gold standard. When patients without visible infarcts on CT were excluded, their algorithm achieved an accuracy rate of 0.646 and a kappa value of 0.488 [30].

In order to standardize the OCSP syndrome diagnosis, Lerner et al. devised a rule-based algorithm based on a modified ver- sion of the National Institutes of Health Stroke Scale (NIHSS) [32]. The NIHSS is a 15-item neurological examination scale (Table II), which measures the level of consciousness, eye move- ment, visual field, facial movement, arm and leg muscle strength,

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

2924 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 24, NO. 10, OCTOBER 2020

coordination, sensation, language and speech, and inattention [33]. It is a standard and highly reliable tool for assessing the neurological function of stroke patients. The modified version of the NIHSS includes information on the laterality of neurological deficits. Based on the results from a small sample of 20 stroke patients, the algorithm achieved a moderate agreement (kappa 0.59) with the OCSP syndrome diagnosis according to neu- roimaging results [32]. Even though these rule-based algorithms achieved some level of success, extra manual effort is needed to extract the features in order to apply these algorithms.

B. Natural Language Processing in Stroke Medicine

Narrative clinical notes, as a major part of EMRs, are used by clinicians to document patient history, clinical observations, impressions, and treatment plans. It is also particularly used to describe the medical reasoning or thought processes behind clinical decisions [34]. They are thus written in free text instead of structured formats because free text allows the flexibility to express complex healthcare processes and exceptional circum- stances. In order to distill valuable information from clinical notes, NLP tools are generally used to recognize named entities or medical concepts in the text and to map them to codes in a controlled vocabulary [11]. Examples of NLP tools for extract- ing information from clinical text include cTAKES, MetaMap, and MedLee [22].

Nevertheless, applications of clinical information extraction have only recently appeared in the field of stroke. Literature review yielded very scarce information regarding natural lan- guage processing of clinical notes for patients with stroke. For example, Mowery et al. designed a specific-purpose NLP tool called pyContext, which can be used to identify significant carotid stenosis, a risk factor for stroke, by automatically filtering radiology reports [35]. Sung et al. applied MetaMap to extract relevant information regarding intravenous thrombolysis from clinical notes, which can be presented on a task-specific EMR interface to support clinical decision making in the management of stroke patients [27]. Govindarajan et al. proposed a prototype to classify stroke into ischemic and hemorrhagic types by com- bining text mining tools and machine learning algorithms [36]. Sedghi et al. build an application that uses NLP and machine learning algorithms to predict whether a patient has a stroke based on clinical history obtained during stroke triage [37]. Kim et al. used NLP and machine learning algorithms to classify brain magnetic resonance imaging reports into ischemic stroke and non-ischemic stroke phenotypes [38]. Garg et al. used NLP of clinical notes combined with machine learning techniques to subtype ischemic stroke according to an etiologic classification system [39].

III. METHODS

A. Data Source and Study Cohort

The data source for this study was the stroke registry main- tained at the Ditmanson Medical Foundation Chia-Yi Chris- tian Hospital, which is a 1000-bed teaching hospital serving a city and its adjoining rural area consisting of around 500,000

inhabitants. The stroke registry was set up conforming to the design of the nationwide Taiwan Stroke Registry [40] and has operated continuously since 2007. It prospectively registers all stroke patients admitted within 10 days of symptom onset. Stroke severity of each patient was determined using the NIHSS at admission. Ischemic stroke was defined as “acute onset of neurological deficit with signs or symptoms persisting longer than 24 hours with or without acute ischemic lesion(s) on brain CT or with acute ischemic diffusion-weighted imaging lesion(s) on MRI that corresponded to the clinical presentations” [40].

All consecutive adult patients hospitalized for acute ischemic stroke between October 2007 and November 2017 were iden- tified from the stroke registry. Patients who had a prior stroke history or who suffered an in-hospital stroke were excluded. Patients whose admission NIHSS score was missing or whose EMRs could not be accessed were also eliminated. The study protocol was approved by the Ditmanson Medical Foundation Chia-Yi Christian Hospital Institutional Review Board (CYCH- IRB No.106100). Patient identifiers were replaced by a unique study identification number to ensure confidentiality. An in- formed consent was thus exempted.

B. Diagnosis of OCSP Syndromes

For each patient with ischemic stroke, the attending stroke neurologist made a syndrome diagnosis according to the OCSP criteria based on clinical findings, such as motor weakness, sensory deficit, speech dysfunction, visual field defect, disorder of eye movement, and impaired coordination [13]. The clinical syndrome diagnosis was used as the “gold standard” of the OCSP stroke subtypes.

C. Text Preprocessing and Natural Language Processing

Because misspellings, non-word symbols, abbreviations and acronyms are common in clinical notes, the narrative clinical notes retrieved from the EMR database need to undergo a series of text preprocessing before NLP techniques can be applied (Fig. 1). In the first step, misspelled words were corrected using the Google spell checker, a Google search engine-based spell check API [41]. The Google spell checker receives an input string, then it detects misspelled words, generates candidate words, and ranks candidate words. If multiple candidate words were generated, the first word on the list was unconditionally accepted in this study. For example, “she recovered her mus- cle strangth” will be replaced by “she recovered her muscle strength” by the Google spell checker. Second, abbreviations and acronyms were expanded to their forms by looking up a list of common clinical abbreviations and acronyms used locally. Third, non-ASCII characters and non-word symbols were removed.

Next, we used MetaMap to identify medical entities from clinical text. MetaMap, developed by the National Library of Medicine, is an NLP tool capable of extracting information from biomedical text [42]. It breaks down input text through tokenization and sentence boundary determination, followed by part-of-speech tagging and parsing, and produces variants

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

SUNG et al.: EMR-BASED PHENOTYPING OF ISCHEMIC STROKE USING SUPERVISED MACHINE LEARNING 2925

Fig. 1. Flowchart of data preparation and analysis.

of resulting words or phrases. MetaMap locates each matched medical entity in the Unified Medical Language System (UMLS) Metathesaurus, assigns it a semantic type, and returns a concept unique identifier (CUI) with a score between 0 and 1000, with a higher value representing a closer match [42]. The quality of the match between a phrase and a Metathesaurus medical entity is evaluated by assessing the measures of centrality, variation, coverage, and cohesiveness. A weighted average is computed from the four measures and is then normalized to obtain the concept score. The details of the evaluation process can be found elsewhere [43]. In this study, MetaMap was configured with the word sense disambiguation option and UMLS Metathesaurus version 2016AA was used. The NegEx algorithm was used to detect negated concepts. Because MetaMap may produce multiple CUIs from the same phrase [44], we selected only the first CUI (i.e., highest matching score) from the ordered list output by MetaMap.

D. Datasets

A total of 4984 patients were hospitalized for ischemic stroke during the study period. After excluding 344 patients with missing data, the remaining sample for this analysis consisted of 4640 patients (41% female, mean age 69 years), among whom 585, 1880, 1392, and 783 were assigned to TACI, LACI, PACI, and POCI by the stroke neurologists, respectively.

Narrative clinical notes were extracted from the EMR database. This study used only the admission note for each patient because, in general, patients’ clinical symptoms are recorded comprehensively in the admission note. A total of 4640 documents were retrieved and each document contained 156 to- kens in average. In total, there were 16215 unique unigrams and 93467 unique bigrams in the text corpus. The clinical text was

Fig. 2. An example of feature extraction from the clinical note.

processed through the text preprocessing and NLP procedures mentioned in section III-C. In the phase of feature extraction, all the identified CUIs were used as features to represent each patient’s admission note. Fig. 2 gives an example of feature extraction from the clinical note. For a given admission note, a feature (CUI) was coded respectively as 0 (not mentioned), 1 (mentioned), and 2 (mentioned as negated), thus forming the text dataset (left half of Fig. 1). These codes were treated as categorical variables in the machine learning algorithms. Because many features (CUIs) in the text dataset occurred very few times, a feature was filtered out if it occurred only once.

Basic demographic data, the admission NIHSS score, and the OCSP subtype for each patient were obtained from the stroke registry. Because certain items of the NIHSS, such as visual field defect, motor deficit, and disorder of eye movement, are

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

2926 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 24, NO. 10, OCTOBER 2020

Fig. 3. Examples of the datasets.

useful for assigning OCSP subtype, each NIHSS item score and the total NIHSS score were included as features and comprised the NIHSS dataset (right half of Fig. 1). In addition, two new predictors were created by calculating the absolute value of the difference between NIHSS left arm and right arm scores and that between NIHSS left leg and right leg scores as fol- lows: NIHSS_arm_diff = ABS(NIHSS_5a – NIHSS_5b) and NIHSS_leg_diff = ABS(NIHSS_6a – NIHSS_6b). These two scores were devised to capture the difference in motor deficit scores between bilateral limbs.

MetaMap identified 8584 unique CUIs, including 173 negated concepts, from the narrative clinical notes. In total, 3467 CUIs occurred in only one instance and were thus eliminated from the feature set. Consequently, the NIHSS-only dataset, text- only dataset, and NIHSS-plus-text dataset contained 18, 5117, and 5135 features, respectively. Fig. 3 shows examples of the datasets. The top-10 CUIs based on the feature selection algo- rithm are listed in Appendix A (Supplementary material). The correlation between the OCSP subtypes and the selected UMLS concepts is not always readily interpretable at first glance. How- ever, the relationship becomes apparent when the context of the UMLS concept is considered. For example, the word “territory” in “brain computed tomography revealed right anterior cerebral artery territory infarction” was mapped to the UMLS concept of “C1301808: Geographic state”. As a result, this UMLS concept was highly correlated with OCSP subtypes that involve large artery territory infarction, such as TACI or PACI.

To examine the accuracy of annotation by MetaMap, the first 100 documents were reviewed by one of the authors. In these documents, MetaMap identified a total of 4430 concepts, including 3948 affirmative CUIs and 485 negated CUIs. The mistakes made by MetaMap were as follows. Two affirmative CUIs were erroneously annotated as negated CUIs whereas four negated CUIs were annotated as affirmative ones. MetaMap failed to identify four CUIs that were judged to be medical concepts by manual annotation (Appendix B, Supplementary material).

Fig. 4. Experimental design.

E. Experimental Design

Fig. 4 illustrates the experimental design. Because we aimed to explore whether textual information could improve the perfor- mance of the classification task, three datasets, i.e., NIHSS-only dataset, text-only dataset, and NIHSS-plus-text dataset, were used to build classifiers independently. We randomly split each dataset into training and test sets (2:1 ratio), with 3093 patients in the training set and 1547 patients in the holdout test set. The training set was used to build classifiers. A correlation-based feature selection technique, i.e., the CfsSubsetEval module of Weka, was applied to select the optimal feature subsets that contain features highly correlated with the class but uncorrelated with each other [45]. Next, the training sets were filtered through the ClassBalancer filter of Weka to reweight the instances so that each class has the same total weight in order to avoid the problem of imbalanced class distribution.

To build classification models, this study carried out experi- ments using six commonly used machine learning algorithms, including C4.5, classification and regression tree (CART), k- nearest neighbor (kNN), random forest (RF), support vector machine (SVM), and logistic regression (LGR). Specifically, the J48, SimpleCart, IBK, RandomForest, SMO, and Logistic mod- ules of Weka 3.8.3 open-source data mining software (Hamilton, New Zealand, www.cs.waikato.ac.nz/ml/weka) were employed. During the training process, the CVParameterSelection met- alearner module of Weka was used to optimize the hyperparam- eters for each classifier via 10-fold cross-validation (Table III). In this experiment, we used Python 3.7 with python-weka- wrapper3 package version 0.1.7 running on MacOS 10.14.5 operating system.

Four aggregation algorithms (Appendix C, Supplementary material), named OVA-Label, OVO-Label, OVA-Prob, and OVO-Prob, were proposed for making final predictions based on two sets of binary OVA and OVO classifiers [47]. Specifically, OVO algorithms consider the set of binary classifiers that dis- criminate between each possible pair of class labels (e.g., TACI vs LACI) while OVA algorithms consider those that discriminate the target class from all other classes (e.g., the OCSP subtype TACI vs non-TACI).

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

SUNG et al.: EMR-BASED PHENOTYPING OF ISCHEMIC STROKE USING SUPERVISED MACHINE LEARNING 2927

TABLE III HYPERPARAMETER TUNING IN WEKA

CART = classification and regress tree, kNN = k-nearest neighbor, LGR = logistic regression, RF = random forest, SVM = support vector machine.

To experiment with binary strategies with the four proposed aggregation algorithms, the six Weka classification modules were used to build binary classifiers from the training set, result- ing in a total of 36 OVO classifiers and 24 OVA classifiers. Sim- ilarly, the CVParameterSelection metalearner module of Weka was used to optimize the hyperparameters for each classifier via 10-fold cross-validation (Table III). Then the results of the binary classifiers were combined using the four proposed algorithms, i.e., OVA-Label, OVO-Label, OVA-Prob, and OVO-Prob. Dur- ing the process of ensemble, we further tested two varieties of aggregation methods. The first is to use the results of all six Weka classifiers for each binary classification problem to generate multi-class classification models, where were termed OVA-Label-1, OVO-Label-1, OVA-Prob-1, and OVO-Prob-1. The second is to use the results of five Weka classifiers by removing the one with the lowest discrimination ability. These models were termed OVA-Label-2, OVO-Label-2, OVA-Prob-2, and OVO-Prob-2.

For comparison, the rule-based algorithm devised by Lerner et al. [32] was used as a baseline model. Furthermore, a model with bag-of-words text representation and LGR classifier was trained as an additional baseline comparison.

F. Statistical Analysis

In the training phase, the performance of the binary classifi- cation models was assessed with sensitivity, specificity, and the area under the receiver operating characteristic curve (AUC), which measures how well a model discriminates between in- stances of different classes. These performance measures were estimated by 10-fold cross-validation.

To assess the performance of the multi-class classifiers in Weka and the proposed ensemble classifiers on the holdout test sets, we used the accuracy rate and Cohen’s kappa [29]. The accuracy rate is the proportion of correctly classified instances among all the test set samples. However, the accuracy rate does not take into account the possibility of the agreement arising by chance. In contrast, Cohen’s kappa compensates for the agreement that is attributed to mere chance. It is a statistic for assessing the agreement between two classifiers that classify the

same samples into mutually exclusive categories. The level of agreement is often judged as follows: κ < 0 poor, 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81-1 almost perfect agreement [48]. Nevertheless, the inter- pretation of kappa should be made cautiously because kappa is dependent on the number of categories in that its value is usually lower if there are more categories. In addition, kappa is dependent on the prevalence of the condition.

All statistical analyses were performed using R version 3.5.0 (R Foundation for Statistical Computing, Vienna, Austria). Be- cause the accuracy rates and the values of Cohen’s kappa from various classifiers were paired data, McNemar’s tests were used for pairwise comparisons of accuracy rates between classifiers. Hotelling’s T2 tests implemented in R package “multiagree” were used to test the equality of dependent Kappa values [49]. Two-tailed p values were considered statistically significant at < 0.05.

IV. RESULTS

A. Performance of 10-Fold Cross-Validation on the Training Sets

The performance measures of the binary classifiers according to 10-fold cross-validation are listed in Appendices D and E (Supplementary material). Although the performance across different machine learning algorithms and OCSP syndromes varied a lot, the best classifiers among the six classifiers using both NIHSS scores and textual information outperformed those using either source of information alone. However, classifiers based solely on NIHSS scores and those based on textual infor- mation alone were neck and neck. For example, among the best OVO binary classifiers, those based on NIHSS scores achieved higher AUCs for TACI vs LACI (0.965 vs 0.930), TACI vs PACI (0.887 vs 0.848), and TACI vs POCI (0.921 vs 0.920) than those based on textual information. In contrast, classifiers based on textual information performed better for LACI vs PACI (0.763 vs 0.696), LACI vs POCI (0.855 vs 0.748), and PACI vs POCI (0.837 vs 0.723) as compared to those based on NIHSS scores. Among the OCSP syndromes, TACI was generally well discriminated from other non-TACI syndromes, with the highest AUCs greater than 0.9 whether using OVO or OVA strategies. PACI was less well differentiated from other syndromes, in particular between PACI and LACI (AUC 0.806). Appendix F (Supplementary material) lists the performance measures of the multi-class classifiers according to 10-fold cross-validation. Among them, classifiers using both NIHSS scores and textual information performed the best, followed by those using NIHSS scores, and classifiers using textual information alone achieved the lowest AUCs.

B. Performance of the Multi-Class Classifiers and the Ensemble Classifiers on the Holdout Test Sets

Fig. 5 shows the performance of the classifiers on the holdout test sets. The p values for accuracy and kappa between pairs of classifiers are listed in Supplementary File 1 and 2, respectively. The rule-based Lerner’s algorithm achieved an accuracy rate of

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

2928 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 24, NO. 10, OCTOBER 2020

Fig. 5. Performance of the multi-class classifiers and the ensemble classifiers. Classifiers are ordered according to their accuracy rates (A) and kappa values (B).

0.332, which was significantly lower than all the other classifiers (all p < 0.001). Its kappa value (0.149) was significantly lower than almost all of the other classifiers (p < 0.001 to 0.005) except for CART (p = 0.313) and kNN (p = 0.120) classifiers based solely on textual information. The model with bag-of-words text representation and LGR classifier, LGR (BOW), attained an accuracy rate of 0.518 and a kappa value of 0.318, which were comparable with the ensemble classifiers based on textual information alone (p for accuracy, 0.106 to 1.000; p for kappa, 0.592 to 0.876) but were inferior to those based on both NIHSS scores and textual information (p for accuracy, < 0.001 to 0.005; p for kappa, < 0.001 to 0.030).

Most of the classifiers based on both NIHSS scores and textual features have higher accuracy rates (0.489 to 0.583) and kappa values (0.272 to 0.399) than classifiers solely based on NIHSS scores (accuracy 0.465 to 0.533; kappa 0.254 to 0.344) and those based on textual information alone (accuracy 0.401 to 0.540; kappa 0.170 to 0.328). For example, the best classifier (OVO-Prob-2) based on both NIHSS scores and textual features performed better than those (OVA-Label-2 in terms of accuracy

and LGR in terms of kappa) based on NIHSS score (accuracy 0.583 vs 0.533, p < 0.001; kappa 0.399 vs 0.344, p < 0.001) and that (OVA-Label-2) based on textual information (accuracy 0.583 vs 0.540, p < 0.001; kappa 0.399 vs 0.328, p < 0.001).

Notably, ensemble classifiers using binarization techniques generally performed better than the multi-class classifiers ex- cept those using Weka’s SMO and Logistic modules on the NIHSS-only dataset. The results are not surprising in that these two modules are intrinsically based on binarization strategies. Among the classifiers based on both sources of information, the classifier using OVO decomposition strategy and OVO-Prob-2 ensemble method had the highest accuracy rate, which was not statistically different from that achieved by the second- best multi-class classifier using OVO-Label-2 ensemble method (0.583 vs 0.575, p = 0.067). The kappa value of the OVO-Prob-2 ensemble method was marginally significantly higher (0.399 vs 0.386, p = 0.048) than the second-best classifier (OVO-Prob-1).

Table IV compares the approaches and results of previous studies on the determination of OCSP phenotypes with the current one. All the previous three studies used a rule-based

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

SUNG et al.: EMR-BASED PHENOTYPING OF ISCHEMIC STROKE USING SUPERVISED MACHINE LEARNING 2929

TABLE IV COMPARISON OF STUDIES ON THE DETERMINATION OF OCSP PHENOTYPES

algorithm to assign OCSP phenotypes based on structured data. Of note, the Lerner’s algorithm was only developed and vali- dated in a very small sample.

V. DISCUSSION

Although automated phenotyping of ischemic stroke was not perfect and needs refinement and improvement, this study demonstrated that textual information mined from EMRs could help subtype ischemic stroke when this information was com- bined with structured information. In addition, decomposition of this multi-class problem into binary classification tasks followed by aggregation of the outputs of the binary classifiers could improve the performance of classification. Among the OCSP syndromes, PACI was particularly hard to be differentiated from other syndromes. Distinguishing PACI from LACI is the most difficult challenge.

Even though the existing rule-based algorithms seemed to perform better in terms of accuracy and agreement than the machine-learning approach used in this study, they all need extra manual effort to extract necessary information from clinical notes because not all the required structured data is available from EMRs. Furthermore, despite its success in a small valida- tion cohort, the Lerner’s algorithm did not perform as well in our test set.

A. Integration of Structured and Unstructured Data for Clinical Utility

The results of this study support the idea that information extracted from unstructured textual data could supplement struc- tured data in the phenotyping of ischemic stroke. Tradition- ally, healthcare processes are documented in narrative notes by healthcare providers. Clinical notes are generated to record clinical observations, actions related to patient care, and the underlying medical reasoning processes. Presently, with the advance of information technology, large amounts of structured data, including diagnosis codes, medications, and laboratory results, were captured and stored in EMRs. Even structured entry systems have been developed for clinical documentation [50], enabling capturing of standardized clinical data for reuse in downstream applications.

Despite this, busy clinicians generally stick with writing nar- rative notes because structured documentation systems may lack

flexibility and expressivity in generating comprehensive and accurate clinical notes and can be so cumbersome to use as they slow down clinical workflow [51]. Therefore, unstructured clin- ical narratives still comprise a large proportion of EMRs. Both structured and unstructured data sources have their strengths and weakness in the aspects of information completeness, informa- tion expressiveness, concept encoding granularity, and accuracy of temporal information [52]. The integration of structured and unstructured data sources has yielded more reliable results in the automation of clinical processes, such as case-finding for disease surveillance or clinical trial pre-screening [52]–[55], generation of problem lists [56], and identification of patient need for services [57].

B. Clinical Correlates and Future Directions

Ischemic stroke is a disease entity with highly heterogeneous pathogenesis and manifestations, which can be subtyped accord- ing to either mechanistic or syndromic classification systems [58]. No matter what kind of phenotyping is used, ischemic stroke phenotypes correlate closely with clinical outcomes of stroke patients [12], [14]. Conventionally, phenotyping ischemic stroke in clinical studies largely depends on manual review by human experts with or without the assistance of rule-based algorithms [30], [31], [59]. Nevertheless, automated phenotyp- ing is preferred when the sample size is large, particularly in studies using large databases of EMRs. So far in the literature available, there have been limited studies that investigated au- tomated methods for phenotyping stroke. For example, Ni et al. developed a machine learning approach to detect patients with stroke from EMRs and to categorize them as having is- chemic stroke, hemorrhagic stroke, transient ischemic attack, and non-stroke conditions [60]. Govindarajan et al. proposed a machine learning application for classifying stroke into ischemic stroke and hemorrhagic stroke according to stroke symptoms text-mined from medical records [36]. Further OCSP subtyping of ischemic stroke was not explored in both studies. To the best of our knowledge, the present study was the first to apply NLP tools and machine learning techniques to this clinical problem.

Among the four distinct OCSP syndromes, PACI was the most difficult to be separated from other syndromes and was particularly poorly differentiated from LACI. A previous study also found that even human experts inadequately distinguished between PACI and LACI [61]. It is partly because patients with PACI had comparable stroke severity to those with LACI [61]. Furthermore, since patients with PACI and those with LACI may have similar outcomes [14], [62], it is not that meaningful to differentiate PACI from LACI. In contrast, patients with TACI have a totally different post stroke outcome from those with other OCSP syndromes [14], [62]. POCI may be associated with worse outcome but less risk of symptomatic intracerebral hemorrhage following intravenous thrombolysis as compared to other syndromes [9], [17]. It may be reasonable and feasible to treat PACI and LACI as a single phenotype. This notion is supported by a study that developed a prognostic scoring system for stroke [19], in which PACI and LACI were assigned 0 point whereas TACI and POCI were assigned 2 and 1 point, respectively.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

2930 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 24, NO. 10, OCTOBER 2020

Currently, the NIHSS has become a standard method for mea- suring neurological impairment following stroke and is routinely recorded in the medical records of stroke patients. In this study, the NIHSS score was shown promising for assigning OCSP subtypes. Although Lerner et al. proposed a modified version of the NIHSS with a rule-based algorithm, achieving a higher performance in the classification of OCSP subtypes [32], their results were based on a small sample of only 20 patients and could not be reproduced in our test set. Besides, the use of the modified version of the NIHSS still relies on human judgment and is not the current standard of practice.

Overall, the results of this study were unsatisfactory. We are aware of some potential problems of our approach. First, only admission notes were used as the source of textual information. Second, text features were coded using the UMLS medical concepts extracted from the clinical notes. Although the UMLS- based approach could improve predictive performance more than other feature extraction techniques [63], sometimes it may perform worse than n-gram models [39]. Therefore, in addition to the textual information from narrative clinical notes, other data elements in the EMRs, such as laboratory results and imaging reports, may be integrated to refine the automated phenotyp- ing of ischemic stroke, hopefully improving the classification accuracy. Furthermore, various feature extraction approaches, such as bag-of-words models, term frequency-inverse document frequency models, topic modeling [63], and word embeddings [64], can be applied on text data. Finally, a combined approach using both rule-based and machine learning algorithms may be considered in future studies.

VI. CONCLUSION

With the advancement of precision medicine, disease pheno- typing is becoming increasingly important not only for genomic discovery but also for determination of potential interactions between drugs and specific genetic traits [65]. Although EMRs provide a rich source of phenotypic data, manual annotation and curation are resource-intensive and extremely time-consuming, making them difficult to scale. In this regard, automated pheno- typing may serve as a feasible alternative solution.

This study attempted to explore an automated approach to EMR-based phenotyping of ischemic stroke. Through the use of NLP tools and machine learning techniques, structured data and unstructured textual data were used to build classifiers to distinguish different subtypes of ischemic stroke. The study results showed that classifiers based solely on unstructured textual data had a comparable performance to those based on structured data alone, whereas classifiers using both data sources led to the best performances. As for the multi-class classification problem, binarization strategies achieved better results than native multi-class classifiers, such as decision tree or random forest.

ACKNOWLEDGMENT

The authors would like to thank Ms. L.-Ying Sung for English language editing

REFERENCES

[1] C. S. Tan et al., “Trends in stroke incidence and 28-day case fatality in a nationwide stroke registry of a multiethnic asian population,” Stroke, vol. 46, no. 10, pp. 2728–2734, Oct. 2015.

[2] L. Ramirez et al., “Trends in acute ischemic stroke hospitalizations in the United States,” J. Amer. Heart Assoc., vol. 5, no. 5, May 2016, Art. no. e003233.

[3] M. Lee, Y.-L. Wu, and B. Ovbiagele, “Trends in incident and recurrent rates of first-ever ischemic stroke in Taiwan between 2000 and 2011,” J. Stroke, vol. 18, no. 1, pp. 60–65, Jan. 2016.

[4] V. L. Feigin, B. Norrving, and G. A. Mensah, “Global burden of stroke,” Circulation Res., vol. 120, no. 3, pp. 439–448, Feb. 2017.

[5] G. C. Fonarow et al., “Characteristics, performance measures, and in- hospital outcomes of the first one million stroke and transient ischemic attack admissions in get with the guidelines-stroke,” Circulation Cardio- vascular Qual. Outcomes, vol. 3, no. 3, pp. 291–302, May 2010.

[6] S. B. Jones, S. Sen, K. Lakshminarayan, and W. D. Rosamond, “Post- stroke outcomes vary by pathogenic stroke subtype in the Atheroscle- rosis risk in communities study,” Stroke, vol. 44, no. 8, pp. 2307–2310, Aug. 2013.

[7] Y. Yang et al., “The Oxfordshire Community Stroke Project classification system predicts clinical outcomes following intravenous thrombolysis: A prospective cohort study,” Therapeutics Clin. Risk Manage, vol. 12, pp. 1049–1056, 2016.

[8] M. L. Schmitz et al., “Ischemic stroke subtype is associated with outcome in thrombolyzed patients,” Acta Neurologica Scandinavica, vol. 135, no. 2, pp. 176–182, Feb. 2017.

[9] P. Sommer et al., “Is functional outcome different in posterior and anterior circulation stroke?” Stroke, vol. 49, no. 11, pp. 2728–2732, Nov. 2018.

[10] J. F. Meschia, “Addressing the heterogeneity of the ischemic stroke pheno- type in human genetics research,” Stroke, vol. 33, no. 12, pp. 2770–2774, Dec. 2002.

[11] P. B. Jensen, L. J. Jensen, and S. Brunak, “Mining electronic health records: Towards better research applications and clinical care,” Nature Rev. Genetics, vol. 13, no. 6, pp. 395–405, Jun. 2012.

[12] E. M. Arsava et al., “Assessment of the predictive validity of etiologic stroke classification,” JAMA Neurology, vol. 74, no. 4, pp. 419–426, Apr. 2017.

[13] J. Bamford, P. Sandercock, M. Dennis, J. Burn, and C. Warlow, “Classi- fication and natural history of clinically identifiable subtypes of cerebral infarction,” Lancet, vol. 337, no. 8756, pp. 1521–1526, Jun. 1991.

[14] S. J. Pittock, D. Meldrum, O. Hardiman, J. Thornton, P. Brennan, and J. T. Moroney, “The Oxfordshire Community Stroke Project classification: Correlation with imaging, associated complications, and prediction of outcome in acute ischemic stroke,” J. Stroke Cerebrovascular Diseases, vol. 12, no. 1, pp. 1–7, Jan. 2003.

[15] M. Paci, L. Nannetti, D. Casavola, and B. Lombardi, “Differences in motor recovery between upper and lower limbs: Does stroke subtype make the difference?” Int. J. Rehabil Res, vol. 39, no. 2, pp. 185–187, Jun. 2016.

[16] M. He et al., “Effects of blood pressure in the early phase of ischemic stroke and stroke subtype on poststroke cognitive impairment,” Stroke, vol. 49, no. 7, pp. 1610–1617, Jul. 2018.

[17] S.-F. Sung, C.-S. Wu, Y.-C. Hsu, M.-C. Tseng, and Y.-W. Chen, “Ox- fordshire community stroke project classification but not NIHSS predicts symptomatic intracerebral hemorrhage following thrombolysis,” J. Neu- rological Sci., vol. 324, no. 1, pp. 65–69, Jan. 2013.

[18] S.-F. Sung et al., “Oxfordshire community stroke project classification improves prediction of post-thrombolysis symptomatic intracerebral hem- orrhage,” BMC Neurology, vol. 14, no. 1, Mar. 2014, Art. no. 39.

[19] P. K. Myint et al., “The SOAR (stroke subtype, Oxford community stroke project classification, age, prestroke modified Rankin) score strongly predicts early outcomes in acute stroke,” Int. J. Stroke, vol. 9, no. 3, pp. 278–283, Apr. 2014.

[20] S.-F. Sung, Y.-W. Chen, L.-C. Hung, and H.-J. Lin, “Revised iScore to predict outcomes after acute ischemic stroke,” J. Stroke Cerebrovascularc Diseases, vol. 23, no. 6, pp. 1634–1639, Jul. 2014.

[21] A. H. Abdul-Rahim et al., “Derivation and validation of a novel prognostic scale (modified-stroke subtype, Oxfordshire community stroke project classification, age, and prestroke modified rankin) to predict early mortality in acute stroke,” Stroke, vol. 47, no. 1, pp. 74–79, Jan. 2016.

[22] Y. Wang, S. Moon, Y. Zeng, and S. Sohn, “Clinical information extraction applications: A literature review,” J. Biomed. Informat., vol. 77, pp. 34–49, Jan. 2018.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.

SUNG et al.: EMR-BASED PHENOTYPING OF ISCHEMIC STROKE USING SUPERVISED MACHINE LEARNING 2931

[23] J.-H. Chiang, J.-W. Lin, and C.-W. Yang, “Automated evaluation of elec- tronic discharge notes to assess quality of care for cardiovascular diseases using medical language extraction and encoding system (MedLEE),” J. Amer. Med. Informat Assoc., vol. 17, no. 3, pp. 245–252, May 2010.

[24] R. J. Byrd, S. R. Steinhubl, J. Sun, S. Ebadollahi, and W. F. Stewart, “Automatic identification of heart failure diagnostic criteria, using text analysis of clinical notes from electronic health records,” Int. J. Med. Informat., vol. 83, no. 12, pp. 983–992, Dec. 2014.

[25] N. Afzal et al., “Natural language processing of clinical notes for identifi- cation of critical limb ischemia,” Int. J. Med. Informat, vol. 111, pp. 83–89, Mar. 2018.

[26] K. B. Wagholikar et al., “Clinical decision support with automated text processing for cervical cancer screening,” J. Amer. Med. Informat. Assoc., vol. 19, no. 5, pp. 833–839, Sep. 2012.

[27] S.-F. Sung, K. Chen, D. P. Wu, L.-C. Hung, Y.-H. Su, and Y.-H. Hu, “Ap- plying natural language processing techniques to develop a task-specific EMR interface for timely stroke thrombolysis: A feasibility study,” Int. J. Med. Informat., vol. 112, pp. 149–157, Apr. 2018.

[28] D. D. Miller and E. W. Brown, “Artificial intelligence in medical practice: The question to the answer?” Amer. J. Med., vol. 131, no. 2, pp. 129–133, Feb. 2018.

[29] M. Galar, A. Fernández, E. Barrenechea, H. Bustince, and F. Herrera, “An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes,” Pattern Recognit., vol. 44, no. 8, pp. 1761–1776, Aug. 2011.

[30] A. Włodek, I. Sarzyńska-Długosz, P. A. G. Sandercock, and A. Członkowska, “Agreement between the Clinical Oxfordshire Com- munity Stroke Project classification and CT findings in Poland,” Eur. J. Neurology, vol. 11, no. 2, pp. 91–96, Feb. 2004.

[31] L. Aerden, G. J. Luijckx, S. Ricci, A. Hilton, F. Kessels, and J. Lodder, “Validation of the Oxfordshire Community Stroke Project syndrome diag- nosis derived from a standard symptom list in acute stroke,” J. Neurological Sci., vol. 220, no. 1, pp. 55–58, May 2004.

[32] D. P. Lerner, B. P. Tseng, and L. B. Goldstein, “Development and as- sessment of a computer algorithm for stroke vascular localization using components of the national institutes of health stroke scale,” J. Stroke Cerebrovascularc Diseases, vol. 25, no. 2, pp. 281–287, Feb. 2016.

[33] T. Brott et al., “Measurements of acute cerebral infarction: A clinical examination scale,” Stroke, vol. 20, no. 7, pp. 864–870, Jul. 1989.

[34] Y.-X. Ho, C. S. Gadd, K. L. Kohorst, and S. T. Rosenbloom, “A qualitative analysis evaluating the purposes and practices of clinical documentation,” Appl. Clin. Informat., vol. 5, no. 1, pp. 153–168, 2014.

[35] D. L. Mowery et al., “Extracting a stroke phenotype risk factor from Veteran Health Administration clinical reports: An information content analysis,” J. Biomed. Semantics, vol. 7, no. 1, 2016, Art. no. 26.

[36] P. Govindarajan, R. K. Soundarapandian, A. H. Gandomi, R. Patan, P. Jayaraman, and R. Manikandan, “Classification of stroke disease using machine learning algorithms,” Neural Comput. Appl., vol. 32, no. 3, pp. 817–828, 2020.

[37] E. Sedghi, J. H. Weber, A. Thomo, M. Bibok, and A. M. W. Penn, “Mining clinical text for stroke prediction,” Netw. Model. Anal. Health Informat. Bioinf., vol. 4, no. 1, Jul. 2015, Art. no. 16.

[38] C. Kim, V. Zhu, J. Obeid, and L. Lenert, “Natural language processing and machine learning algorithm to identify brain MRI reports with acute ischemic stroke.,” PLOS One, vol. 14, no. 2, 2019, Art. no. e0212778.

[39] R. Garg, E. Oh, A. Naidech, K. Kording, and S. Prabhakaran, “Automating ischemic stroke subtype classification using machine learning and natural language processing.,” J. Stroke Cerebrovascularc Diseases, vol. 28, no. 7, pp. 2045–2051, Jul. 2019.

[40] F.-I. Hsieh et al., “Get with the guidelines-stroke performance indicators: Surveillance of stroke care in the Taiwan stroke registry: Get with the guidelines-stroke in Taiwan,” Circulation, vol. 122, no. 11, pp. 1116–1123, Sep. 2010.

[41] N. Coad, “Google spell check,” 2013. [Online]. Available: https://github. com/noahcoad/google-spell-check. Accessed: on Jan. 9, 2019.

[42] A. R. Aronson and F.-M. Lang, “An overview of MetaMap: Historical perspective and recent advances,” J. Amer. Med. Informat. Assoc., vol. 17, no. 3, pp. 229–236, May 2010.

[43] A. Aronson, “MetaMap: Mapping text to the UMLS metathesaurus,” Bethesda, MD, USA: NLM, NIH, DHHS, 2006.

[44] A. B. Abacha and P. Zweigenbaum, “Medical entity recognition: A com- parison of semantic and statistical methods,” in Proc. BioNLP Workshop. Assoc. Computat. Linguist., 2011, pp. 56–64.

[45] M. A. Hall, “Correlation-based feature selection for machine learning,” Ph.D. dissertation, Univ. Waikato, Hamilton, New Zealand, 1999.

[46] T. Hastie and R. Tibshirani, “Classification by pairwise coupling,” Annals Statist., vol. 26, no. 2, pp. 451–471, Apr. 1998.

[47] A. C. Lorena, A. C. P. L. F. de Carvalho, and J. M. P. Gama, “A review on the combination of binary classifiers in multiclass problems,” Artif. Intell. Rev., vol. 30, no. 1, pp. 19–37, Aug. 2009.

[48] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, Mar. 1977.

[49] S. Vanbelle, “Comparing dependent kappa coefficients obtained on mul- tilevel data,” Biometrical J., vol. 59, no. 5, pp. 1016–1034, Sep. 2017.

[50] S. T. Rosenbloom et al., “Generating complex clinical documents using structured entry and reporting,” Stud. Health Technol. Informat., vol. 107, no. 1, pp. 683–687, 2004.

[51] S. T. Rosenbloom, J. C. Denny, H. Xu, N. Lorenzi, W. W. Stead, and K. B. Johnson, “Data from clinical notes: A perspective on the tension between structure and flexible documentation,” J. Amer. Med. Informat. Assoc., vol. 18, no. 2, pp. 181–186, Mar. 2011.

[52] L. Li, H. S. Chase, C. O. Patel, C. Friedman, and C. Weng, “Comparing ICD9-encoded diagnoses and NLP-processed discharge summaries for clinical trials pre-screening: A case study,” in Proc. AMIA Annu. Symp., Nov. 2008, vol. 2008, pp. 404–408.

[53] S. Abhyankar, D. D.-Fushman, F. M. Callaghan, and C. J. McDonald, “Combining structured and unstructured data to identify a cohort of ICU patients who received dialysis,” J. Amer. Med. Informat. Assoc., vol. 21, no. 5, pp. 801–807, Sep. 2014.

[54] L. B. Moreira and A. A. Namen, “A hybrid data mining model for diag- nosis of patients with clinical suspicion of dementia,” Comput. Methods Programs Biomed., vol. 165, pp. 139–149, Oct. 2018.

[55] S. Cohen, A.-S. Jannot, L. Iserin, D. Bonnet, A. Burgun, and J.-B. Escudié, “Accuracy of claim data in the identification and classification of adults with congenital heart diseases in electronic medical records,” Archives Cardiovascular Diseases, vol. 112, no. 1, pp. 31–43, Jan. 2019.

[56] M. V. Devarakonda, N. Mehta, C.-H. Tsou, J. J. Liang, A. S. Nowacki, and J. E. Jelovsek, “Automated problem list generation and physicians perspective from a pilot study,” Int. J. Med. Informat, vol. 105, pp. 121– 129, Sep. 2017.

[57] J. R. Vest, S. J. Grannis, D. P. Haut, P. K. Halverson, and N. Menachemi, “Using structured and unstructured data to identify patients’ need for services that address the social determinants of health,” Int. J. Med. Informat., vol. 107, pp. 101–106, Nov. 2017.

[58] J. F. Meschia, “Subtyping in ischemic stroke genetic research,” J. Stroke Cardiovascular Diseases, vol. 11, no. 5, pp. 208–219, Sep. 2002.

[59] H. Ay et al., “A computerized algorithm for etiologic classification of ischemic stroke: The causative classification of stroke system,” Stroke, vol. 38, no. 11, pp. 2979–2984, Nov. 2007.

[60] Y. Ni et al., “Towards phenotyping stroke: Leveraging data from a large- scale epidemiological study to detect stroke diagnosis,” PLOS One, vol. 13, no. 2, 2018, Art. no. e0192586.

[61] N. Asdaghi et al., “Oxfordshire community stroke project classifica- tion poorly differentiates small cortical and subcortical infarcts,” Stroke, vol. 42, no. 8, pp. 2143–2148, Aug. 2011.

[62] M. Paci, L. Nannetti, P. D’Ippolito, and B. Lombardi, “Outcomes from ischemic stroke subtypes classified by the Oxfordshire Community Stroke Project: A systematic review,” Eur. J. Physical Rehabil. Med., vol. 47, no. 1, pp. 19–23, Mar. 2011.

[63] M. Hoogendoorn, P. Szolovits, L. M. G. Moons, and M. E. Numans, “Utilizing uncoded consultation notes from electronic medical records for predictive modeling of colorectal cancer,” Artif. Intell. Med., vol. 69, pp. 53–61, May 2016.

[64] J. S. Obeid et al., “Automated detection of altered mental status in emer- gency department clinical notes: A deep learning approach,” BMC Med. Informat. Decis. Making, vol. 19, no. 1, pp. 164–169, Aug. 2019.

[65] W.-Q. Wei and J. C. Denny, “Extracting research-quality phenotypes from electronic health records to support precision medicine,” Genome Med., vol. 7, no. 1, 2015, Art. no. 41.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:05:37 UTC from IEEE Xplore. Restrictions apply.