W#15 Health Promotion

Article2.pdf

Home >Nursing homework help >W#15 Health Promotion

Development of the SaFETy Score: A Clinical Screening Tool for Predicting Future Firearm Violence Risk Jason E. Goldstick, PhD; Patrick M. Carter, MD; Maureen A. Walton, MPH, PhD; Linda L. Dahlberg, PhD; Steven A. Sumner, MD, MSc; Marc A. Zimmerman, PhD; and Rebecca M. Cunningham, MD

Background: Interpersonal firearm violence among youth is a substantial public health problem, and emergency department (ED) physicians require a clinical screening tool to identify high- risk youth.

Objective: To derive a clinically feasible risk index for firearm violence.

Design: 24-month prospective cohort study.

Setting: Urban, level 1 ED.

Participants: Substance-using youths, aged 14 to 24 years, seeking ED care for an assault-related injury and a proportion- ately sampled group of non–assault-injured youth enrolled from September 2009 through December 2011.

Measurements: Firearm violence (victimization/perpetration) and validated questionnaire items.

Results: A total of 599 youths were enrolled, and presence/ absence of future firearm violence during follow-up could be ascertained in 483 (52.2% were positive). The sample was ran- domly split into training (75%) and post–score-construction vali- dation (25%) sets. Using elastic-net penalized logistic regression, 118 baseline predictors were jointly analyzed; the most predic- tive variables fell predominantly into 4 domains: violence victim-

ization, community exposure, peer influences, and fighting. By selection of 1 item from each domain, the 10-point SaFETy (Se- rious fighting, Friend weapon carrying, community Environment, and firearm Threats) score was derived. SaFETy was associated with firearm violence in the validation set (odds ratio [OR], 1.47 [95% CI, 1.23 to 1.79]); this association remained (OR, 1.44 [CI, 1.20 to 1.76]) after adjustment for reason for ED visit. In 5 risk strata observed in the training data, firearm violence rates in the validation set were 18.2% (2 of 11), 40.0% (18 of 45), 55.8% (24 of 43), 81.3% (13 of 16), and 100.0% (6 of 6), respectively.

Limitations: The study was conducted in a single ED and involved substance-using youths. SaFETy was not externally validated.

Conclusion: The SaFETy score is a 4-item score based on clini- cally feasible questionnaire items and is associated with firearm violence. Although broader validation is required, SaFETy shows potential to guide resource allocation for prevention of firearm violence.

Primary Funding Source: National Institute on Drug Abuse R01024646.

Ann Intern Med. 2017;166:707-714. doi:10.7326/M16-1927 Annals.org For author affiliations, see end of text. This article was published at Annals.org on 11 April 2017.

Firearm violence has been identified by health andlegal professionals as a critical public health prob- lem (1). Homicide is the third leading cause of death in the United States among youth aged 15 to 24 years, with more than 86% of these deaths due to firearms (2). Furthermore, firearm violence results in substantial monetary cost; for example, medical and work-loss costs of nonfatal firearm injuries treated in U.S. emer- gency departments were estimated to exceed $2.9 bil- lion in 2010 (3). Mitigating this public health issue requires novel hospital and community-based interven- tions that are focused on at-risk youth, especially those in urban communities. Urban emergency departments (EDs) have been identified as a critical access point for identifying and intervening with such youth (4). Firearm violence encompasses interpersonal, self-directed, and unintentional firearm-related incidents, but in this study we focus on interpersonal firearm violence, which we refer to simply as “firearm violence” throughout.

Although previous ED-based research (5) has iden- tified risk factors associated with firearm violence in- volvement among high-risk youth, the field of hospital and ED-based youth violence prevention programs lacks a short, clinically relevant screening tool that can be applied as part of routine clinical care in urban set- tings. Such a tool could play a key role in determining

where to focus prevention or intervention efforts. Youth identified during an ED visit, particularly violently in- jured youth, are at elevated risk for future firearm vio- lence (5) and thus would benefit most from early inter- vention, including case management and therapeutic services. Previous screening tools for youth violence (6 – 8) primarily focused on primary care settings, lack a specific focus on firearm violence, or are too lengthy for practical use in a busy ED setting. Furthermore, re- search on the construction of violence screening tools (6, 8) has been limited by small sample sizes and has not considered out-of-sample predictive power in de- vising the screen. Developing an ED/hospital-based clinical screening tool that is focused on assessing risk for future firearm violence will enable ED and hospital health systems to better focus prevention resources on patients at the highest risk.

In the current study, we seek to develop a clinical screening tool for future risk for firearm violence by ex-

See also:

Web-Only Supplement

Annals of Internal Medicine ORIGINAL RESEARCH

amining data collected as part of a 2-year prospective study of youth aged 14 to 24 years seeking ED care. First, we used machine learning methods to determine which variables measured at the baseline of a 2-year longitudinal study were most predictive of future fire- arm violence. Second, on the basis of breadth and clin- ical feasibility, we selected 4 items from among the most predictive variables. Third, we developed cut- points and assigned point values to each level based on their relative effects, resulting in the SaFETy (Serious fighting, Friend weapon carrying, community Environ- ment, and firearm Threats) score. Finally, we examined the relationship between the SaFETy score and rates of future firearm violence within training and internal vali- dation data sets.

METHODS Study Design and Setting

Data were collected during the Flint Youth Injury study (9 –11), a 2-year prospective cohort study of assault-injured youth (age 14 to 24 years) with any drug use in the past 6 months and a comparison group of non–assault-injured, drug-using youth seeking ED care at a level 1 trauma center in Flint, Michigan. The parent study focused on service needs and utilization among substance (predominantly marijuana) users. Although this potentially limits generalizability, we note that most youth who seek care for assault injuries in this setting are substance users (9). Patients were recruited from December 2009 through September 2011, 24 hours per day on Thursday through Monday and from 5 a.m. to 2 a.m. on Tuesday and Wednesday. Youth who sought care for sexual assault, child abuse, suicidal ide- ation or attempt, or any conditions that preclude con- sent (such as altered mental status) were excluded. In- stitutional review boards at the University of Michigan and Hurley Medical Center approved the study. A Na- tional Institutes of Health (NIH) Certificate of Confiden- tiality (COC) was obtained.

Potential participants were ascertained through electronic patient logs and approached by research as- sistants in waiting or treatment areas. All assault-injured youth, including those who were initially unstable but stabilized with 72 hours of presentation, were ap- proached and screened for study eligibility. In se- quence, the next available age group (14 to 17, 18 to 20, and 21 to 24 years) and sex-matched, non–assault- injured ED entrant was screened for the comparison group. Those providing consent (or assent with paren- tal consent for those younger than age 18 years) pri- vately self-administered the screening survey using a tablet device and received a $1.00 gift for participation. Individuals who self-reported drug use in the past 6 months (98% used marijuana) were considered eligible and consented to the subsequent 2-year longitudinal study. Appendix Figure 1 (available at Annals.org) shows a flow chart of the original study. Remunera- tion was $20 for completion of a subsequent self- administered baseline survey. Follow-up assessments were conducted at 6, 12, 18, and 24 months, and par-

ticipants were compensated $35, $40, $40, and $50 for each sequential follow-up. Baseline characteristics (9) and 2-year outcomes (5, 10) are reported elsewhere.

Measures The following measures were assessed: The outcome variable was a binary indicator of fire-

arm violence (victimization, perpetration, firearm injury requiring medical care, or firearm death) during the 24- month follow-up period, ascertained through a com- posite of self-report, medical chart review, and vital re- cords databases (see Carter and colleagues [5] for greater detail). Both peer and partner firearm violence was included. Although the dynamics of peer and part- ner violence differ, we justify combining them by noting the large overlap between victims (12, 13) and perpe- trators (14 –16) of peer and partner violence.

Candidate predictor variables were taken from baseline self-report surveys; in addition to age, sex, and reason for ED visit (assault-injured/non–assault-injured), we included 115 survey items. Other variables that were measured but judged unlikely to be assessed ac- curately and truthfully (for example, serious violence perpetration) without an NIH Certificate of Confidenti- ality were not considered. See the Supplement (avail- able at Annals.org) for question wording and response options for all items described below.

1. Violence items (13 items) from the National Lon- gitudinal Study on Adolescent Health (17) captured the frequency of received threats/violence, perpetrated threats, fighting, and carrying a weapon while intoxi- cated in the past 6 months.

2. Partner aggression (13 items) was assessed with Conflict Tactics Scale items (18), which measured the frequency of partner violence victimization in the past 6 months.

3. Nonpartner aggression (13 items) was assessed with questions modified from the Conflict Tactics Scale (18), measuring the frequency of nonpartner violence victimization in the past 6 months.

4. Community violence exposure (5 items) included assessment of the frequency of exposure to violence and neighborhood crime in the past 6 months (19).

5. Mental health (12 items) was measured with the Brief Symptom Inventory checklist (20), which assessed severity of depression and anxiety in the past week.

6. Drug and alcohol efficacy (16 items) assessed confidence in avoiding drug (8 items) or alcohol (8 items) use in various situations (21, 22).

7. Alcohol use (10 items) was assessed with the Al- cohol Use Disorders Identification Test (AUDIT), which measures the frequency of alcohol consumption and alcohol-related consequences in the past 6 months (23, 24).

8. Peer influences (11 items) included items from the Flint Adolescent Study (25) regarding the number of friends providing positive (4 items) and negative (7 items) influences; positive items were reverse coded.

9. Parental behavior (10 items) included items from the Flint Adolescent Study (25) assessing parental sup-

ORIGINAL RESEARCH Development of the SaFETy Score

708 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

port (6 items) and level of parental drug/alcohol use (4 items); parental support was reverse coded.

10. Retaliatory attitudes (7 items) included items as- sessing willingness to engage in violent retaliation; higher scores indicate greater willingness (26, 27).

11. Fight self-efficacy (5 items) assessed perceived ability to avoid conflicts (28).

Statistical Analysis We first randomly split the data into a training set

(75%) and a validation set (25%), ensuring that the prevalence of firearm violence was equivalent in each. In determining variable importance, developing cut- points, and assigning point values, we used only the training data. Multiple imputation by chained equations (29), implemented in the R statistical software package mice (29), was used to impute missing outcomes (there were no missing values in the predictors measured at baseline) for inclusion in the training data. Variable im- portance analyses and determination of risk score con- tributions were based on estimates pooled across 50 multiply imputed data sets.

To determine variable importance, we used elastic- net penalized logistic regression, a common machine learning approach to binary classification, which shrinks the regression coefficient estimates to improve out-of- sample prediction. The elastic net (30) penalizes both the absolute value of the coefficients, which performs automatic variable selection by shrinking irrelevant co- efficients to zero (31), and the squared size of the coef- ficient, which limits the effect of collinearity (32). The elastic net is particularly applicable when the number of predictors is large relative to the sample size and when interest lies in variable selection wherein the ef- fect of collinearity is reduced (30). To evaluate out-of- sample prediction accuracy, we used leave-one-out cross-validation (LOOCV). In LOOCV, the model is fit to the entire sample, minus 1 point, and is used to predict the excluded case; this is repeated so each point is left out once. The level of coefficient shrinkage that mini- mizes the LOOCV error rate was chosen. Model fitting and cross-validation were performed by using the R statistical software package glmnet (33).

After determining the optimally penalized model, we ranked predictor importance by using the size of the standardized regression coefficient estimates. Us- ing the variable importance rankings and the variable domains, we selected 4 variables that were predictive and covered distinct content. We justify the use of 4 items by noting that exploration with 3 item scores (not shown) indicated that the 4-item score yielded a more thorough risk gradient (in the training data), and this was not notably improved by adding a fifth item; be- cause clinical feasibility is paramount, item scores of 6 or more were not explored. Cut-points for each item were chosen by cycling through all possible categori- zations of each variable, fitting a logistic regression model of future firearm violence (in the training set) using the categorized variables, and choosing the cat- egorization that minimized the finite-sample– corrected Akaike information criteria (34). To avoid overfitting the

training data, a maximum of 3 categories were consid- ered for each variable and overly small categories (<20 people) were not considered. Finally, we determined score contributions by 1) entering the categorized pre- dictors into a single logistic regression model and 2) scaling by the minimum regression coefficient and rounding to an integer (as in reference 35). Properties of the risk score in the validation set (sensitivity, speci- ficity, and odds ratio [OR] with firearm violence) were examined and stratified by assault-injured/non–assault- injured group.

Role of the Funding Source Our funding sources had no role in the design,

conduct, or analysis of our study or the decision to sub- mit the manuscript for publication.

RESULTS Sample Characteristics

In total, 599 youth (349 assault-injured and 250 non–assault-injured) participated in the study. Follow-up rates were 85.3%, 83.7%, 84.2%, and 85.3% at 6, 12, 18, and 24 months. Among participants, 483 (80.6%) could be definitively classified as having been involved with firearm violence (n = 252 [52.2%]) or not (n = 231 [47.8%]) during the follow-up period. Of the participants, 57.3% were male and 62.5% were African American; the average age at baseline was 19.9 years (SD, 2.4 years); greater detail is published elsewhere (36). One fourth of those with (n = 63) and those with- out (n = 58) firearm violence were randomly removed for post–score-construction validation; there were no significant demographic differences between the train- ing and validation data.

Variable Importance Analysis Appendix Figure 2 (available at Annals.org) shows

the relative size of the top 20 largest standardized co- efficients. Table 1 lists the selected items and their uni- variate associations with firearm violence. Selected items largely fell into 4 domains that were observed post hoc: 1) violence victimization (peer and partner), 2) community violence exposure, 3) peer/family influ- ences, and 4) fighting.

Risk Score Construction We narrowed the 20 variables to 4 items from dif-

ferent domains to construct the SaFETy score. From vi- olence victimization, we chose the highest-ranking item: being threatened with a firearm. For practical use this could be combined with the number 2–ranked and number 9 –ranked items into “threatened or shot you” because of similar content. “Friend weapon carrying” was chosen from the peer influence domain. Among the community violence items, we chose the lower- ranking item—frequency of hearing gunshots— because of the greater likelihood of a truthful response in a clin- ical setting (compared with the frequency of seeing someone shot). Similarly, we chose “frequency of being in a serious fight” over “frequency of putting someone in the hospital” from the fighting domain.

Development of the SaFETy Score ORIGINAL RESEARCH

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 709

Table 2 shows the derived cut-points and weights for each category, and how the selected items corre- spond to the SaFETy mnemonic. “Received firearm threats” was divided into “never,” “once,” and “2+ times”; “hearing gunshots” was divided into “less than many times” and “many times”; “fight frequency” was stratified into “never,” “1–5 times,” and “6+ times”; and “friend weapon carrying” was collapsed into whether or not “many, most, or all” friends carry weapons. The largest weights corresponded to received firearm vio- lence threats and high-frequency serious fighting.

Risk Score Performance In the validation set, a 1-point increase in SaFETy

score corresponded to higher risk for firearm violence (OR, 1.47 [95% CI, 1.23 to 1.79]). The area under the receiver-operating characteristic curve was 0.73, indi- cating reasonable out-of-sample discriminatory power. The Figure shows the distribution of SaFETy scores among those with and without firearm violence in the validation set. Table 3 shows the sensitivity and speci- ficity of SaFETy score in the validation set for each cut- point between 0 and 10 (Appendix Table 1, available at Annals.org, shows training sample results). Informal ex- amination of the training data indicated 5 risk strata:

SaFETy scores of 0, 1 to 2, 3 to 5, 6 to 8, and 9 to 10; the same risk gradient is apparent in the validation set (Appendix Figure 3, available at Annals.org), with each level corresponding to future firearm violence rates of 18.2% (2 of 11), 40.0% (18 of 45), 55.8% (24 of 43), 81.3% (13 of 16), and 100.0% (6 of 6), respectively.

Sensitivity Analysis Because membership in the assault-injured group

is itself associated with future firearm violence (Appen- dix Tables 2 and 3, available at Annals.org), we present several stratified analyses. First, we conducted an om- nibus test comparing the model with all 20 of the top variables that do versus do not allow the effects to vary by assault-injured/non–assault-injured group, which yielded a nonsignificant result (P = 0.09). Appendix Ta- bles 4 and 5 (available at Annals.org) show univariate associations between each variable and firearm vio- lence, stratified by assault-injured/non–assault-injured group. Second, we estimated the joint effects of SaFETy score and assault-injured group membership (Appen- dix Table 3) and found that 1) with SaFETy score in- cluded in the model, assault-injured group member- ship was not significant and 2) the effect of SaFETy score was similar after the inclusion of the assault-

Table 1. Highest-Ranked Prognostic Factors for Future Firearm Violence

Prognostic Factor Response Type*

Importance Rank

Timeframe Odds Ratio (95% CI)†

Standardized Odds Ratio‡

Received threats Someone pulled a gun on you Frequency (0–6) 1 6 mo 2.44 (1.89–3.20) 2.90 Someone used a gun on you Frequency (0–6) 2 6 mo 3.31 (1.98–5.52) 2.22 Someone pulled a knife on you Frequency (0–6) 6 6 mo 1.89 (1.40–2.54) 1.81 Someone shot you Frequency (0–6) 9 6 mo 2.92 (1.69–5.06) 1.73 Someone threw something at you Frequency (0–6) 12 6 mo 1.58 (1.21–2.07) 1.58 Someone cut/stabbed you Frequency (0–6) 14 6 mo 2.22 (1.47–3.34) 1.73

Community I have seen someone shot Frequency (0–3) 3 6 mo 1.87 (1.44–2.42) 1.78 I have heard guns shot Frequency (0–3) 5 6 mo 1.55 (1.27–1.90) 1.61 Seen gangs in neighborhood Frequency (0–3) 18 6 mo 1.31 (1.11–1.55) 1.40 My house was broken into Frequency (0–3) 20 6 mo 1.70 (1.23–2.34) 1.45

Friends My friends carry weapons Number (1–5) 10 Current 1.56 (1.26–1.92) 1.62 My friends smoke marijuana Number (1–5) 15 Current 1.27 (1.07–1.50) 1.34 Friend legal trouble (drug-related) Number (1–5) 17 Current 1.63 (1.25–2.12) 1.53

Partner violence Partner used a knife on you Frequency (0–6) 13 6 mo 3.46 (1.46–8.24) 2.37

Fighting Been in a serious fight Frequency (0–6) 4 6 mo 1.46 (1.25–1.71) 1.76 Put someone in the hospital Frequency (0–6) 7 6 mo 1.71 (1.35–2.16) 1.85 Drank before fighting Frequency (0–6) 8 6 mo 1.78 (1.33–2.38) 1.72

Other Understand another's point of view Agree (1–5) 11 6 mo 1.38 (1.16–1.64) 1.49 Today's ED visit for violent injury Yes/no 16 Current 1.89 (1.23–2.89) NA Unable to stop drinking Frequency (0–4) 19 6 mo 1.56 (1.10–2.20) 1.39

ED = emergency department; NA = not available. * Frequency (0 – 6) measures frequency on a 7-point scale from 0 (never) to 6 (20+ times). Frequency (0 –3) measures frequency on a 7-point scale from 0 (never) to 3 (many times). Frequency (0 – 4) measures frequency on a 5-point scale from 0 (never) to 4 (daily). Number (1–5) measures frequency on a 5-point scale from 1 (none) to 5 (all). Agree (1–5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true). Yes/no denotes a binary (1/0) indicator. † CIs with lower bounds of 1.00 are entirely above 1.00. ‡ Standardized odds ratios were those obtained by using the standardized predictors.

ORIGINAL RESEARCH Development of the SaFETy Score

710 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

injured group. Appendix Tables 6 and 7 (available at Annals.org) show frequency distributions of SaFETy scores and sensitivity/specificity estimates, respectively, stratified by assault-injured group (corresponding train- ing sample results are shown in Appendix Tables 8 and 9, available at Annals.org).

DISCUSSION In the current study, we used machine learning

methods and data from a prospective cohort of ED pa- tients to identify factors predictive of future firearm vio- lence that could be incorporated into a brief clinical screening tool for ED use. This data set is unique in that high-risk urban youth were followed successfully over 2 years and that the interviews capture not only firearm violence that resulted in injury but incidents that re- sulted in near-injury (for example, being threatened with a gun). SaFETy clearly defines a gradient for future firearm violence risk in this population; this steady in- crease (rather than a sharp increase at 1 point) inhibits determination of a threshold with strong combined sensitivity/specificity but creates a strong basis for allo- cation of prevention resources.

Coupling risk stratification with effective prevention tools is an important potential use of the SaFETy score. Very-high-risk individuals (e.g., those with a SaFETy score !6) may represent sensible candidates for entry into resource-intensive programs (for example, 1-year wraparound programs), whereas individuals in the mid- dle range (a SaFETy score !1 but "5) may benefit from graduated levels of targeted interventions designed to interrupt a negative trajectory. Programs focusing on primary prevention may be appropriate for lower- scoring (a SaFETy score of 0) individuals. Although

SaFETy has predictive power among both assault- injured and non–assault-injured youth, it is most appli- cable among non–assault-injured youth because there is no other current means of stratifying risk in that group. Furthermore, given the excess risk for future vic- timization among those presenting with assault injury, particularly firearm injuries (37), prevention resources should also be considered for this group, even among those with low SaFETy scores. Previous violence pre- vention programs have been shown to be cost-effective (38 – 42), specifically with regard to the costs of treating repeated violent injuries (38, 39) and preventing incar- ceration due to violence-related offenses (38, 40). Given that the average ED visit for a firearm assault costs $1200 and average inpatient costs approach $24 000 (43), even a moderately effective prevention program directed at individuals in higher-risk strata would be cost-effective.

The items selected for the screening tool confer strong face validity to the data-derived prediction tool. Two items— history of receiving gun threats and hearing gun violence in one's community— confirm the impor- tance of previous violence exposure in the risk for fu- ture firearm violence. In addition, peer influences, whose importance is most pronounced during the pe- riod of adolescence and emerging adulthood (44), is an important prognostic factor, in the form of friend weapon carrying. The emergence of serious fighting as a strong predictor agrees with prior violence screening tools (7) and underscores the role of impulse control and aggression in firearm violence. These results high- light the broader importance of incorporating commu- nity and peer factors into prevention programs, in ad- dition to addressing psychological distress stemming

Table 2. Rules for Calculation of the SaFETy Score

Mnemonic Category Question/Scale Levels SaFETy Contribution

S Serious Fighting In the past 6 mo, including today, how often did you get into a serious physical fight?

0 (never) 0 1 (once) 1 2 (twice) 1 3 (3–5 times) 1 4+ (6 or more times) 4

F Friend Weapon Carrying How many of your friends have carried a knife, razor, or gun? 1 (none) 0 2 (some) 0 3+ (many, most, or all) 1

E Community Environment In the past 6 mo, how often have you heard guns being shot? 0 (never) 0 1 (once or twice) 0 2 (a few times) 0 3 (many times) 1

T Firearm Threats How often, in the past 6 mo, including today, has someone pulled a gun on you?

0 (never) 0 1 (once) 3 2+ (twice or more) 4

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Development of the SaFETy Score ORIGINAL RESEARCH

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 711

from victimization history, consistent with trauma- informed care practices (45).

We also note that male sex, which is consistently identified as a risk factor for gun violence (5, 46, 47), was not predictive of future gun violence. This suggests that sex differences in firearm violence risk may be sub- sumed by other risk exposure (such as affiliation with high-risk peers). Similarly, depression and anxiety were not identified as important prognostic factors, suggest- ing that their association with firearm violence may also be largely subsumed by proximal exposures to high- risk community violence. We note that self-reported ability to understand another's point of view was highly associated with future gun violence but was not se- lected for the final screen because of concerns about its accurate assessment and wording that may be mis- understood in a brief clinical screen.

The results here are encouraging because they rely only on items considered feasible to ascertain in a clin- ical setting. The feasibility of screening on the basis of such measures as “Have you shot someone in the last 6 months?” is limited when the respondents may not per- ceive themselves as having the same degree of confi- dentiality as in a clinical study with an NIH Certificate of Confidentiality. In addition, that question is not likely to be asked or answered in a way that is perceived as nonjudgmental or nonincriminating by staff or patients. Asking youth about peer behavior that they are more likely to report—and is not incriminating— has been done for other risk behavior tools, such as the National Institute on Alcohol Abuse and Alcoholism alcohol screening tool for adolescents (48). None of the mea- sures selected specifically ask respondents to 1) incrim-

inate themselves, 2) incriminate any specific person, or 3) embarrass themselves. Although a superior predic- tive tool may be derived by including a broader class of measures, this gain may be offset by reduced willing- ness to answer or response accuracy.

Our study had several limitations. First, our sample is limited to a single urban ED; validation in other high- risk populations is required. Second, the analytic sam- ple is limited to drug-using youth. Although we cannot ascertain the efficacy of this screen in non– drug-using youth, we note that substance use has been linked to both gun carrying (36) and violence (49) and that a large majority of individuals screened into the study be- cause of marijuana use. To mitigate this limitation, cli- nicians could first inquire about marijuana use in the past 6 months, a standard history and physical exami- nation question. Alternatively, given the lack of valid screening instruments for firearm violence and the knowledge that this is the leading cause of death for youth in urban communities, it would be clinically rea- sonable to suggest that high-scoring urban youth, even those who have not used marijuana in the past 6 months, warrant preventive services. However, future research is needed to validate this tool among non– substance-using youth.

Third, self-reported data were a large component of identifying those with versus those without firearm violence during the follow-up period, which is a con- cern for underreporting of firearm violence. This limita- tion is partly mitigated by the use of full validated scales, such as the Conflict Tactics Scale (18), which were privately administered on a tablet. Fourth, our mental health assessments focused only on depression and anxiety. Because such symptoms as suspicious- ness, delusions, and extreme anger have been linked to violence and gun carrying (50, 51), future work is needed to assess their power to predict future gun vi- olence. Finally, our missing-data imputation relies on the untestable missing-at-random assumption. Noting the high follow-up rate and that no covariates that made up the SaFETy score differed significantly in terms of missing versus nonmissing cases, the role of nonrandom attrition was likely minimal.

In conclusion, we used machine learning methods to determine the most important predictors of future

Figure. Distribution of SaFETy scores among youth with and without firearm violence during the follow-up period in the validation data.

D is

tr ib

u ti

o n o

f Sa

FE Ty

S co

re s,

SaFETy Score

No firearm violence (n = 63) Firearm violence (n = 58)

1 2 3 4 6 7 9 105

SaFETy = Serious fighting, Friend weapon carrying, community Envi- ronment, and firearm Threats.

Table 3. Sensitivity and Specificity for SaFETy Score Thresholds Between 1 and 10 in the Validation Set

Threshold Sensitivity, n/N (%) (95% CI)

Specificity, n/N (%) (95% CI)

1 61/63 (96.8 [88.0–99.4]) 9/58 (15.5 [7.8–27.9]) 2 53/63 (84.1 [72.3–91.7]) 27/58 (46.6 [33.5–60.0]) 3 43/63 (68.3 [55.2–79.1]) 36/58 (62.1 [48.3–74.2]) 4 36/63 (57.1 [44.1–69.3]) 40/58 (69.0 [55.3–80.1]) 5 32/63 (50.8 [38.0–63.5]) 48/58 (82.8 [70.1–91.0]) 6 19/63 (30.2 [19.6–43.2]) 55/58 (94.8 [84.7–98.7]) 7 7/63 (11.1 [5.0–22.2]) 57/58 (98.3 [89.5–100.0]) 8 6/63 (9.5 [3.9–20.2]) 58/58 (100.0 [92.3–100.0]) 9 6/63 (9.5 [3.9–20.2]) 58/58 (100.0 [92.3–100.0]) 10 5/63 (6.3 [2.1–16.3]) 58/58 (100.0 [92.3–100.0])

SaFETy = Serious fighting, Friend weapon carrying, community Envi- ronment, and firearm Threats.

ORIGINAL RESEARCH Development of the SaFETy Score

712 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

firearm violence in a high-risk ED sample. This is the first scale to provide risk stratification for firearm vio- lence and the first developed in and specifically for an ED setting (rather than primary care). Previous risk scores were developed to predict related but distinct behaviors, such as nonspecific violent injury (6, 8) and firearm carrying (7). The common thread between SaFETy and previous scales is the importance of fight- ing (6 – 8) and received threats (7) as prognostic factors. The SaFETy instrument, which can be administered in 1 to 2 minutes, defines a gradient of future firearm vio- lence risk that can be adapted to a variety of settings. Emergency departments have been previously used as opportunities for identifying high-risk individuals for other types of violence (52, 53), but the current lack of an easily administered screening tool for firearm vio- lence has limited our ability to harness the same oppor- tunity for firearm violence. Our results suggest that SaFETy fills this gap.

From University of Michigan School of Medicine and Univer- sity of Michigan School of Public Health Ann Arbor, Michigan; Centers for Disease Control and Prevention, Atlanta, Georgia; and Hurley Medical Center, Flint, Michigan.

Disclaimer: The findings and conclusions in this manuscript are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Acknowledgment: The authors thank the staff and patients of Hurley Medical Center for their support of this project and Wendi Mohl, BS, and Sonia Kamat, MS, for their assistance with manuscript preparation.

Grant Support: National Institute on Drug Abuse grant R01 024646 (principal investigator, Rebecca M. Cunningham), 1 June 2009 to 30 April 2014; National Institutes of Health/Na- tional Institute on Drug Abuse (NIDA) grants K23DA039341 (principal investigator: Patrick M. Carter) and 16IPA605200 (principal investigator: Jason Goldstick), 1 July 2016 to 30 June 2017.

Disclosures: Dr. Walton reports grants from NIDA during the conduct of the study. Authors not named here have dis- closed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms .do?msNum=M16-1927

Reproducible Research Statement: Study protocol and data set: Not available. Statistical code: Available from Dr. Gold- stick (e-mail, [email protected]).

Requests for Single Reprints: Jason Goldstick, PhD, University of Michigan Injury Center, University of Michigan School of Medicine, 2800 Plymouth Road, NCRC G10-080, Ann Arbor, MI 48109; e-mail, [email protected].

Current author addresses and author contributions are avail- able at Annals.org.

References 1. Weinberger SE, Hoyt DB, Lawrence HC 3rd, Levin S, Henley DE, Alden ER, et al. Firearm-related injury and death in the United States: a call to action from 8 health professional organizations and the American Bar Association. Ann Intern Med. 2015;162:513-6. [PMID: 25706470] doi:10.7326/M15-0337 2. Centers for Disease Control and Prevention. Injury Prevention & Control: Data & Statistiscs (WISQARS). Leading causes of death. 2015. Accessed at www.cdc.gov/injury/wisqars/leading_causes _death.html on 6 June 2016. 3. Centers for Disease Control and Prevention. Data & Statistiscs (WISQARS): Cost of Injury Reports. 2014. Accessed at https://wisqars .cdc.gov:8443/costT on 23 June 2016. 4. Cunningham R, Knox L, Fein J, Harrison S, Frisch K, Walton M, et al. Before and after the trauma bay: the prevention of violent injury among youth. Ann Emerg Med. 2009;53:490-500. [PMID: 19162376] doi:10.1016/j.annemergmed.2008.11.014 5. Carter PM, Walton MA, Roehler DR, Goldstick J, Zimmerman MA, Blow FC, et al. Firearm violence among high-risk emergency depart- ment youth after an assault injury. Pediatrics. 2015;135:805-15. [PMID: 25847808] doi:10.1542/peds.2014-3572 6. Sege R, Stringham P, Short S, Griffith J. Ten years after: examina- tion of adolescent screening questions that predict future violence- related injury. J Adolesc Health. 1999;24:395-402. [PMID: 10401967] 7. Hayes DN, Sege R. FiGHTS: a preliminary screening tool for ado- lescent firearms-carrying. Ann Emerg Med. 2003;42:798-807. [PMID: 14634605] 8. Sigel E, Hart J, Hoffenberg A, Dodge M. Development and psy- chometric properties of a violence screening tool for primary care. J Adolesc Health. 2011;48:358-65. [PMID: 21402264] doi:10.1016 /j.jadohealth.2010.07.024 9. Cunningham R, Ranney M, Newton M, Woodhull W, Zimmerman M, Walton M. Cunningham RM, Ranney M, Newton M, Woodhull W, Zimmerman M, Walton MA. Characteristics of youth seeking emer- gency care for assault injuries. Pediatrics. 2014;133:e96-105. [PMID: 24323994] doi:10.1542/peds.2013-1864;133(1):96-105. 10. Cunningham RM, Carter PM, Ranney M, Zimmerman MA, Blow FC, Booth BM, et al. Violent reinjury and mortality among youth seeking emergency department care for assault-related injury: a 2-year prospective cohort study. JAMA Pediatr. 2015;169:63-70. [PMID: 25365147] doi:10.1001/jamapediatrics.2014.1900 11. Bohnert KM, Walton MA, Ranney M, Bonar EE, Blow FC, Zimmer- man MA, et al. Understanding the service needs of assault-injured, drug-using youth presenting for care in an urban Emergency Depart- ment. Addict Behav. 2015;41:97-105. [PMID: 25452051] doi:10.1016/j .addbeh.2014.09.019 12. Finkelhor D, Turner H, Hamby S, Ormrod R. Polyvictimization: children's exposure to multiple types of violence, crime, and abuse. Office of Juvenile Justice and Delinquency Prevention and Centers for Disease Control and Prevention. Juvenile Justice Bulletin. Na- tional Survey of Children's Exposure to Violence. 2011. Accessed at www.ncjrs.gov/pdffiles1/ojjdp/235504.pdf on 13 March 2017. 13. Hamby S, Grych J. The Web of Violence: Exploring Connections among Different Forms of Interpersonal Violence and Abuse. New York: Springer; 2013. 14. Klevens J, Simon TR, Chen J. Are the perpetrators of violence one and the same? Exploring the co-occurrence of perpetration of physical aggression in the United States. J Interpers Violence. 2012; 27:1987-2002. [PMID: 22328658] doi:10.1177/0886260511431441 15. Bossarte RM, Simon TR, Swahn MH. Clustering of adolescent dating violence, peer violence, and suicidal behavior. J Interpers Violence. 2008;23:815-33. [PMID: 18252941] doi:10.1177 /0886260507313950 16. Foshee VA, McNaughton Reyes L, Tharp AT, Chang LY, Ennett ST, Simon TR, et al. Shared longitudinal predictors of physical peer and dating violence. J Adolesc Health. 2015;56:106-12. [PMID: 25287983] doi:10.1016/j.jadohealth.2014.08.003 17. Bearman P, Jones J, Udry J. Add Health. 1997. Accessed at www .cpc.unc.edu/projects/addhealth on 22 December 2016.

Development of the SaFETy Score ORIGINAL RESEARCH

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 713

18. Straus MA, Hamby SL, Boney-McCoy S, Sugarman DB. The re- vised Conflict Tactics Scale (CTS2): development and preliminary psychometric data. J Fam Issues. 2016;17:283-316. 19. Richters JE, Saltzman W. Survey of Exposure to Community Vio- lence: Self-Report Version. Rockville, MD: JE Richters; 1990. 20. Piersma HL, Boes JL, Reaume WM. The Brief Symptom Inventory as an Outcome Measure for Adolescent Psychiatric Inpatients. As- sessment. 1994;1:151-8. [PMID: 9465145] 21. Belgrave FZ, Reed MC, Plybon LE, Corneille M. The impact of a culturally enhanced drug prevention program on drug and alcohol refusal efficacy among urban African American girls. J Drug Educ. 2004;34:267-79. [PMID: 15648887] 22. Conners NA, Plybon LE, Pegg P. Specific Events Drug and Alco- hol Refusal Self-Efficacy Scale. Richmond, VA: Virginia Common- wealth Univ; 2003. 23. Babor TF, Higgins-Biddle JC, Saunders JB, Monteiro MG. The Alcohol Use Disorders Identification Test: Guidelines for Use in Pri- mary Care. Geneva: World Health Organization; 2001. 24. Chung T, Colby SM, Barnett NP, Rohsenow DJ, Spirito A, Monti PM. Screening adolescents for problem drinking: performance of brief screens against DSM-IV alcohol diagnoses. J Stud Alcohol. 2000;61:579-87. [PMID: 10928728] 25. Zimmerman M. Flint Adolescent Study (FAS): a longitudinal study of school dropout and substance use, 1994-1997. Ann Arbor, MI: University of Michigan; 2014. 26. Hill HM, Noblin V. Children's Perceptions of Environmental Vio- lence. Washington, DC: Howard Univ; 1991. 27. Copeland-Linder N, Jones VC, Haynie DL, Simons-Morton BG, Wright JL, Cheng TL. Factors associated with retaliatory attitudes among African American adolescents who have been assaulted. J Pediatr Psychol. 2007;32:760-70. [PMID: 17403911] 28. Bosworth K, Espelage D. Teen Conflict Survey. Bloomington, IN: Center for Adolescent Studies, Indiana Univ; 1995. 29. van Burren S, Groothuis-Oudshoorn K. mice: Multivariate impu- tation by chained equations in R. J of Stat Softw. 2011;45:1-67. doi: 10.18637/jss.v045.i03 30. Zou H, Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B. 2005;67: 301-320. 31. Tibshirani R. Regression shrinkage and selection via the lasso. J Royal Stat Soc: Series B. 1996;58:267-288. 32. Hoerl AE, Kennard RW. Ridge regression: applications to nonor- thogonal problems. Technometrics. 1970;12:55-67. 33. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Gen- eralized Linear Models via Coordinate Descent. J Stat Softw. 2010; 33:1-22. [PMID: 20808728] 34. Burnham KP, Anderson DR. Multimodel inference: understand- ing AIC and BIC in Model Selection. Sociol Methods Res. 2004;33: 261-304. 35. Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of mul- tivariate data for clinical use: The Framingham Study risk score func- tions. Stat Med. 2004;23:1631-60. [PMID: 15122742] 36. Carter PM, Walton MA, Newton MF, Clery M, Whiteside LK, Zim- merman MA, et al. Firearm possession among adolescents present- ing to an urban emergency department for assault. Pediatrics. 2013; 132:213-21. [PMID: 23837181] doi:10.1542/peds.2013-0163 37. Rowhani-Rahbar A, Zatzick D, Wang J, Mills BM, Simonetti JA, Fan MD, et al. Firearm-related hospitalization and risk for subse- quent violent injury, death, or crime perpetration: a cohort study. Ann Intern Med. 2015;162:492-500. [PMID: 25706337] doi:10.7326 /M14-2362 38. Cooper C, Eslinger DM, Stolley PD. Hospital-based violence in- tervention programs work. J Trauma. 2006;61:534-7. [PMID: 16966983]

39. Chong VE, Smith R, Garcia A, Lee WS, Ashley L, Marks A, et al. Hospital-centered violence intervention programs: a cost- effectiveness analysis. Am J Surg. 2015;209:597-603. [PMID: 25728889] doi:10.1016/j.amjsurg.2014.11.003 40. Shibru D, Zahnd E, Becker M, Bekaert N, Calhoun D, Victorino GP. Benefits of a hospital-based peer intervention program for vio- lently injured youth. J Am Coll Surg. 2007;205:684-9. [PMID: 17964444] 41. Sharp AL, Prosser LA, Walton M, Blow FC, Chermack ST, Zim- merman MA, et al. Cost analysis of youth violence prevention. Pedi- atrics. 2014;133:448-53. [PMID: 24515518] doi:10.1542/peds.2013- 1615 42. Purtle J, Rich JA, Fein JA, James T, Corbin TJ. Hospital-Based Violence Prevention: Progress and Opportunities. Ann Intern Med. 2015;163:715-7. [PMID: 26301734] doi:10.7326/M15-0586 43. Howell EM, Abraham P. The Hospital Costs of Firearm Assaults. Washington, DC: The Urban Inst; 2013. 44. Steinberg L, Monahan KC. Age differences in resistance to peer influence. Dev Psychol. 2007;43:1531-43. [PMID: 18020830] 45. Ursano RJ, Benedek DM, Engel CC. Trauma-informed care for primary care: the lessons of war. Ann Intern Med. 2012;157:905- 6. [PMID: 22964876] doi:10.7326/0003-4819-157-11-201212040 -00542 46. Walton MA, Cunningham RM, Goldstein AL, Chermack ST, Zim- merman MA, Bingham CR, et al. Rates and correlates of violent be- haviors among adolescents treated in an urban emergency depart- ment. J Adolesc Health. 2009;45:77-83. [PMID: 19541253] doi:10 .1016/j.jadohealth.2008.12.005 47. Pickett W, Craig W, Harel Y, Cunningham J, Simpson K, Molcho M, et al; HBSC Violence and Injuries Writing Group. Cross-national study of fighting and weapon carrying as determinants of adolescent injury. Pediatrics. 2005;116:e855-63. [PMID: 16322143] Pickett W, Craig W, Harel Y, et al. Cross-national study of fighting and weapon carrying as determinants of adolescent injury. Pediatrics. 2005; 116(6):855-853. 48. National Institute on Alcohol Abuse and Alcoholism. Alcohol Screening and Brief Intervention for Youth: A Practitioner's Guide. Accessed at www.niaaa.nih.gov/Publications/EducationTraining Materials/Pages/YouthGuide.aspx on 22 December 2016. 49. McGinty EE, Choksy S, Wintemute GJ. The Relationship Between Controlled Substances and Violence. Epidemiol Rev. 2016;38:5-31. [PMID: 26905893] doi:10.1093/epirev/mxv008 50. Swanson JW, Sampson NA, Petukhova MV, Zaslavsky AM, Ap- pelbaum PS, Swartz MS, et al. Guns, Impulsive Angry Behavior, and Mental Disorders: Results from the National Comorbidity Survey Replication (NCS-R). Behav Sci Law. 2015;33:199-212. [PMID: 25850688] doi:10.1002/bsl.2172 51. Swanson JW, McGinty EE, Fazel S, Mays VM. Mental illness and reduction of gun violence and suicide: bringing epidemiologic re- search to policy. Ann Epidemiol. 2015;25:366-76. [PMID: 24861430] doi:10.1016/j.annepidem.2014.03.004 52. Walton MA, Chermack ST, Shope JT, Bingham CR, Zimmerman MA, Blow FC, et al. Effects of a brief intervention for reducing vio- lence and alcohol misuse among adolescents: a randomized con- trolled trial. JAMA. 2010;304:527-35. [PMID: 20682932] doi:10 .1001/jama.2010.1066 53. Cunningham RM, Chermack ST, Zimmerman MA, Shope JT, Bingham CR, Blow FC, et al. Brief motivational interviewing interven- tion for peer violence and alcohol use in teens: one-year follow-up. Pediatrics. 2012;129:1083-90. [PMID: 22614776] doi:10.1542/peds .2011-3419

ORIGINAL RESEARCH Development of the SaFETy Score

714 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Current Author Addresses: Drs. Goldstick, Carter, and Cun- ningham: University of Michigan Injury Center, University of Michigan School of Medicine, 2800 Plymouth Road, NCRC G10-080, Ann Arbor, MI 48109. Dr. Walton: Department of Psychiatry, University of Michigan Addiction Research Center, University of Michigan School of Medicine, 4250 Plymouth Road, Ann Arbor, MI 48109. Dr. Zimmerman: Michigan Youth Violence Prevention Center, University of Michigan School of Public Health, 1415 Washing- ton Heights, Ann Arbor, MI 48109. Drs. Dahlberg and Sumner: Division of Violence Prevention, National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, 4770 Buford Highway, NE, MS-F64, Atlanta, GA 30341.

Author Contributions: Conception and design: P.M. Carter, R. Cunningham, L.L. Dahlberg, J. Goldstick, M. Walton, M.A. Zimmerman. Analysis and interpretation of the data: P.M. Carter, R. Cun- ningham, J. Goldstick, S.A. Sumner. Drafting of the article: P.M. Carter, L.L. Dahlberg, J. Goldstick, M.A. Zimmerman. Critical revision for important intellectual content: J. Goldstick, S.A. Sumner, M.A. Zimmerman. Final approval of the article: P.M. Carter, R. Cunningham, L.L. Dahlberg, J. Goldstick, S.A. Sumner, M. Walton, M.A. Zimmerman. Provision of study materials or patients: M. Walton. Statistical expertise: J. Goldstick, S.A. Sumner. Obtaining of funding: M. Walton, M.A. Zimmerman. Administrative, technical, or logistic support: P.M. Carter, R. Cunningham, S.A. Sumner. Collection and assembly of data: M. Walton.

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Appendix Figure 1. Flint Youth Injury study flow chart.

Ineligible for screening (n = 474) Sexual assault or child abuse: 128 Mental health issue precluding consent (e.g., schizophrenia): 88 Parental consent not granted: 65 Screened previously: 58 Injured >72 h ago: 41 Currently incarcerated: 30 Other: 64

Missed (n = 76) Study staff screening another participant: 45 Discharged before being located by study staff: 15 Youth could not be located: 5 Other: 11

Declined (n = 131) Did not feel well enough: 62 Unwilling to participate: 29 Family did not allow access: 16 Other: 24

Declined (n = 33) Did not complete baseline after agreeing to do so: 22 Decided not to participate: 6 Left hospital/discharged before completing baseline: 3 Other: 2

Assault-injured youth seeking ED

care (n = 1718)

The non–assault-injured group was sampled so that the next available age- and

sex-matched non–assault-injured youth was approached

Eligible for screen (n = 925)

Approached (n = 849)

Approached (n = 846)

Declined (n = 116) Did not feel well enough: 51 Unwilling to participate: 31 Family did not allow access: 17 Discharged/didn’t want to stay: 10 Other: 7

Declined (n = 27) Did not complete baseline after agreeing to do so: 15 Decided not to participate: 7 Left hospital/discharged before completing baseline: 5

Screened (n = 718)

Screened (n = 730)

Eligible (n = 388)

Baseline complete (n = 349)

Baseline complete (n = 250)

Note: n = 70 arrived for firearm injury

Eligible (n = 278)

Assault-injured youth arriving during

recruitment shifts (n = 1399)

Ineligible for screening (n = 599) Parental consent not granted: 60 Mental health issue precluding consent (e.g., schizophrenia): 81 Screened previously: 66 Other: 392

Excluded (n = 6) Excluded (n = 1)

ED = emergency department.

Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Appendix Figure 2. Variable importance determined by the predictive model, expressed as standardized regression coefficients (divided by the minimum so that the smallest score is 1).

Standardized Regression Coefficient (Scaled by the Minimum) 0 5 15 20 25 30 35 40 45 5010

My house was broken into

Unable to stop drinking

Seen gangs in neighborhood

Friends in legal trouble (drugs)

Violent injury

My friends smoke marijuana

Someone cut/stabbed you

Partner used knife on you

Someone threw something at you

Understand other's POV

My friends carry weapons

Someone shot you

Drank before a serious fight

Put someone in the hospital

Someone pulled a knife on you

Heard shots in neighborhood

Got into serious fight

Seen someone shot

Someone used a gun on you

Someone pulled a gun on you

POV = point of view.

Appendix Table 1. Sensitivity and Specificity for SaFETy Score Thresholds Between 1 and 10 in the Training Set

Threshold Sensitivity (95% CI) Specificity (95% CI)

1 186/189 = 98.4% (95.1%–99.6%) 29/173 = 16.8% (11.7%–23.4%) 2 160/189 = 84.7% (78.5%–89.3%) 77/173 = 44.5% (37.0%–52.2%) 3 132/189 = 69.8% (62.7%–76.2%) 125/173 = 72.3% (64.9%–78.6%) 4 122/189 = 64.6% (57.2%–71.3%) 134/173 = 77.5% (70.4%–83.3%) 5 109/189 = 57.7% (50.3%–64.7%) 140/173 = 80.9% (74.1%–86.3%) 6 74/189 = 39.2% (32.2%–46.5%) 162/173 = 93.6% (88.6%–96.6%) 7 35/189 = 18.5% (13.4%–25.0%) 167/173 = 96.5% (92.3%–98.6%) 8 19/189 = 10.1% (6.3%–15.5%) 171/173 = 98.8% (95.4%–99.8%) 9 15/189 = 7.9% (4.7%–13.0%) 173/173 = 100.0% (97.3%–100.0%) 10 10/189 = 5.3% (2.7%–9.8%) 173/173 = 100.0% (97.3%–100.0%)

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Appendix Table 2. Joint Distribution of Violent Injury and Future Firearm Violence Status

Status AI Non-AI Total

Future firearm violence 167 85 252 No future firearm violence 116 115 231 Unknown 66 50 116 Total 349 250 599

AI = assault-injured.

Appendix Table 3. Relationship Between Future Firearm Violence and SaFETy Score and Assault Injury Presentation

Model Validation (95% CI) Training (95% CI)

1: Assault injury 2.14 (1.03–4.45) 1.89 (1.24–2.89)

2: SaFETY score 1.47 (1.23–1.79) 1.56 (1.41–1.75)

3 Assault injury 1.49 (0.67–3.32) 1.23 (0.75–2.00) SaFETY score 1.44 (1.20–1.76) 1.54 (1.39–1.73)

SaFETy = Serious fighting, Friend weapon carrying, community Envi- ronment, and firearm Threats.

Appendix Figure 3. Future firearm violence rates in the validation data set (dashed line) in 5 risk strata identified by using the training data set (solid line).

Training sample (n = 362) Validation sample (n = 121)

0 1–2 3–5 9–106–8

100

SaFETy score

P ar

ti ci

p an

ts W

it h F

u tu

re F

ir ea

rm V

io le

n ce

, %

SaFETy = Serious fighting, Friend weapon carrying, community Envi- ronment, and firearm Threats.

Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Appendix Table 4. Description of Highest-Ranked Factors for Future Firearm Violence Among Those Presenting for Violent Injury

Factor Response Type†

Importance Rank

Timeframe Odds Ratio (95% CI)*

Standardized Odds Ratio

Received threats Someone pulled a gun on you Freq (0–6) 1 6 mo 1.91 (1.41–2.61) 2.17 Someone used a gun on you Freq (0–6) 2 6 mo 2.44 (1.40–4.26) 1.81 Someone pulled a knife on you Freq (0–6) 6 6 mo 1.37 (0.99–1.67) 1.34 Someone shot you Freq (0–6) 9 6 mo 2.37 (1.27–4.43) 1.56 Someone threw something at you Freq (0–6) 12 6 mo 1.41 (1.03–1.92) 1.41 Someone cut/stabbed you Freq (0–6) 14 6 mo 1.80 (1.15–2.83) 1.50

Community I have seen someone shot Freq (0–3) 3 6 mo 1.92 (1.34–2.73) 1.82 I have heard guns shot Freq (0–3) 5 6 mo 1.28 (0.99–1.67) 1.31 Seen gangs in neighborhood Fred (0-3) 18 6 mo 1.19 (0.95–1.49) 1.24 My house was broken into Freq (0–3) 20 6 mo 1.31 (0.86–1.99) 1.23

Friends My friends carry weapons Number (1–5) 10 Current 1.44 (1.09–1.91) 1.50 My friends smoke marijuana Number (1–5) 15 Current 1.24 (0.99–1.54) 1.30 Friend legal trouble (drug-related) Number (1–5) 17 Current 1.43 (1.00–2.05) 1.37

Partner violence Partner used a knife on you Freq (0–6) 13 6 mo 3.20 (1.07–9.61) 2.24

Fighting Been in a serious fight Freq (0–6) 4 6 mo 1.21 (0.99–1.47) 1.32 Put someone in the hospital Freq (0–6) 7 6 mo 1.33 (1.00–1.77) 1.39 Drank before fighting Freq (0–6) 8 6 mo 1.37 (1.00–1.88) 1.35

Other Understand another's point of view Agree (1–5) 11 6 mo 1.37 (1.10–1.72) 1.48 Today's ED visit for violent injury Yes/No 16 Current NA NA Unable to stop drinking Freq (0–4) 19 6 mo 1.53 (0.94–2.50) 1.20

ED = emergency department; Freq = frequency; NA = not available. * CIs with lower bounds of 1.00 are entirely above 1.00. † Freq (0 – 6) measures frequency on a 7-point scale from 0 (never) to 6 (!20 times); Freq (0 –3) measures frequency on a 7-point scale from 0 (never) to 3 (many times); Freq (0 – 4) measures frequency on a 5-point scale from 0 (never) to 4 (daily); Number (1–5) measures frequency on a 5-point scale from 1 (none) to 5 (All); Agree (1-5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true); Yes/No denotes a binary (1/0) indicator.

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Appendix Table 5. Description of Highest-Ranked Factors for Future Firearm Violence Among Those Not Presenting for Violent Injury

Factor Response Type†

Importance Rank

Timeframe Odds Ratio (95% CI)*

Standardized Odds Ratio

Received threats Someone pulled a gun on you Freq (0–6) 1 6 mo 3.82 (2.14–6.85) 4.96 Someone used a gun on you Freq (0–6) 2 6 mo 7.03 (1.69–29.16) 3.66 Someone pulled a knife on you Freq (0–6) 6 6 mo 4.25 (2.09–8.63) 3.87 Someone shot you Freq (0–6) 9 6 mo 3.54 (1.06–11.79) 1.91 Someone threw something at you Freq (0–6) 12 6 mo 1.85 (1.13–3.04) 1.85 Someone cut/stabbed you Freq (0–6) 14 6 mo 3.44 (1.29–9.17) 2.33

Community I have seen someone shot Freq (0–3) 3 6 mo 1.75 (1.20–2.55) 1.67 I have heard guns shot Freq (0–3) 5 6 mo 2.01 (1.41–2.85) 2.12 Seen gangs in neighborhood Fred (0-3) 18 6 mo 1.53 (1.17–1.99) 1.70 My house was broken into Freq (0–3) 20 6 mo 2.40 (1.43–4.01) 1.84

Friends My friends carry weapons Number (1–5) 10 Current 1.83 (1.30–2.56) 1.94 My friends smoke marijuana Number (1–5) 15 Current 1.38 (1.05–1.81) 1.49 Friend legal trouble (drug-related) Number (1–5) 17 Current 2.00 (1.32–3.03) 1.84

Partner Violence Partner used a knife on you Freq (0–6) 13 6 mo 3.73 (0.94–14.84) 2.49

Fighting Been in a serious fight Freq (0–6) 4 6 mo 1.80 (1.36–2.37) 2.41 Put someone in the hospital Freq (0–6) 7 6 mo 2.62 (1.65–4.16) 3.02 Drank before fighting Freq (0–6) 8 6 mo 3.37 (1.71–6.67) 3.13

Other Understand another's point of view Agree (1–5) 11 6 mo 1.29 (0.97–1.73) 1.38 Today's ED visit for violent injury Yes/No 16 Current NA NA Unable to stop drinking Freq (0–4) 19 6 mo 1.68 (1.02–2.75) 1.46

ED = emergency department; Freq = frequency; NA = not available; OR = odds ratio. * CIs with lower bounds of 1.00 are entirely above 1.00. † Freq (0 – 6) measures frequency on a 7-point scale from 0 (never) to 6 (!20 times); Freq (0 –3) measures frequency on a 7-point scale from 0 (never) to 3 (many times); Freq (0 – 4) measures frequency on a 5-point scale from 0 (never) to 4 (daily); Number (1–5) measures frequency on a 5-point scale from 1 (none) to 5 (All); Agree (1-5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true); Yes/No denotes a binary (1/0) indicator.

Appendix Table 6. Sensitivity and Specificity in the Validation Set, Stratified by AI or Non-AI Group (95% CIs)

Threshold Sensitivity (AI) Specificity (AI) Sensitivity (Non-AI) Specificity (Non-AI)

1 39/41 = 95.1% (82.2–99.2%) 0/27 = 0.0% (0.0%–15.5%) 22/22 = 100.0% (81.5%–100.0%) 9/31 = 29.0% (14.9%–48.2%) 2 36/41 = 87.8% (73.0–95.4) 7/27 = 25.9% (11.9%–46.6%) 17/22 = 77.3% (54.2%–91.3%) 20/31 = 64.5% (45.4%–80.2%) 3 31/41 = 75.6% (59.4–87.1%) 13/27 = 48.1% (29.2%–67.6%) 12/22 = 54.5% (32.7%–74.9%) 23/31 = 74.2% (55.1%–87.5%) 4 26/41 = 63.4% (46.9–77.4) 16/27 = 59.3% (39.0%–77.0%) 10/22 = 45.5% (25.1%–67.3%) 24/31 = 77.4% (58.5%–89.7%) 5 23/41 = 56.1% (39.9%–71.2%) 21/27 = 77.8% (57.3%–90.6%) 9/22 = 40.9% (21.5%–63.3%) 27/31 = 87.1% (69.2%–95.8%) 6 12/41 = 29.3% (16.6%–45.7%) 26/27 = 96.3% (79.1%–99.8%) 7/22 = 31.8% (14.7%–54.9%) 29/31 = 93.5% (77.2%–98.9%) 7 5/41 = 12.2% (4.6%–27.0%) 27/27 = 100.0% (84.5%–100.0%) 2/22 = 9.1% (1.6%–30.6%) 30/31 = 96.8% (81.5%–99.8%) 8 5/41 = 12.2% (4.6%–27.0%) 27/27 = 100.0% (84.5%–100.0%) 1/22 = 4.5% (0.2%–24.9%) 31/31 = 100.0% (86.3%–100.0%) 9 5/41 = 12.2% (4.6%–27.0%) 27/27 = 100.0% (84.5%–100.0%) 1/22 = 4.5% (0.2%–24.9%) 31/31 = 100.0% (86.3%–100.0%) 10 3/41 = 7.3% (1.9%–21.0%) 27/27 = 100.0% (84.5%–100.0%) 1/22 = 4.5% (0.2%–24.9%) 31/31 = 100.0% (86.3%–100.0%)

AI = assault-injured.

Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Appendix Table 8. Sensitivity and Specificity in the Training Set, Stratified by AI or Non-AI Group (95% CIs)

Threshold Sensitivity (AI) Specificity (AI) Sensitivity (Non-AI) Specificity (Non-AI)

1 126/126 = 100.0% (96.3%–100.0%) 5/89 = 5.6% (2.1%–13.2%) 60/63 = 95.2% (85.8%–98.8%) 24/84 = 28.6% (19.5%–39.6%) 2 113/126 = 89.7% (82.7%–94.2%) 24/89 = 27.0% (18.4%–37.6%) 47/63 = 74.6% (61.8%–84.4%) 53/84 = 63.1% (51.8%–73.2%) 3 93/126 = 73.8% (65.1%–81.0%) 54/89 = 60.7% (49.7%–70.7%) 39/63 = 61.9% (48.8%–73.6%) 71/84 = 84.5% (74.6%–91.2%) 4 86/126 = 68.3% (59.3%–76.1%) 60/89 = 67.4% (56.6%–76.8%) 36/63 = 57.1% (44.1%–69.3%) 74/84 = 88.1% (78.8%–93.8%) 5 75/126 = 59.5% (50.4%–68.1%) 61/89 = 68.5% (57.7%–77.7%) 34/63 = 54.0% (41.0%–66.4%) 79/84 = 94.0% (86.0%–97.8%) 6 46/126 = 36.5% (28.3%–45.6%) 80/89 = 89.9% (81.2%–95.0%) 28/63 = 44.4% (32.1%–57.4%) 82/84 = 97.6% (90.9%–99.6%) 7 22/126 = 17.5% (11.5%–25.5%) 84/89 = 94.4% (86.8%–97.9%) 13/63 = 20.6% (11.9%–33.0%) 83/84 = 98.8% (92.6%–99.9%) 8 10/126 = 7.9% (4.1%–14.5%) 87/89 = 97.8% (91.4%–99.6%) 9/63 = 14.3% (7.1%–25.9%) 84/84 = 100.0% (94.6%–100.0%) 9 8/126 = 6.3% (3.0%–12.5%) 89/89 = 100.0% (94.8%–100.0%) 7/63 = 11.1% (5.0%–22.2%) 84/84 = 100.0% (94.6%–100.0%) 10 5/126 = 4.0% (1.5%–9.5%) 89/89 = 100.0% (94.8%–100.0%) 5/63 = 7.9% (3.0%–18.3%) 84/84 = 100.0% (94.6%–100.0%)

AI = assault-injured.

Appendix Table 9. Frequency Tables of the SaFETy Score in the Training Data Set, Stratified by Group Membership

Score Non-AI Group AI Group

0 27 (18.4%) 5 (2.3%) 1 42 (28.6%) 32 (14.9%) 2 26 (17.7%) 50 (23.3%) 3 6 (4.1%) 13 (6.0%) 4 7 (4.8%) 12 (5.6%) 5 9 (6.1%) 48 (22.3%) 6 16 (10.9%) 28 (13.0%) 7 5 (3.4%) 15 (7.0%) 8 2 (1.4%) 4 (1.9%) 9 2 (1.4%) 3 (1.4%) 10 5 (3.4%) 5 (2.3%) Total 147 (100.0%) 215 (100.0%)

AI = assault-injured; SaFETy = Serious fighting, Friend weapon carry- ing, community Environment, and firearm Threats.

Appendix Table 7. Frequency Tables of the SaFETy Score in the Validation Data Set, Stratified by Group Membership

Score Non-AI Group AI Group

0 9 (17.0%) 2 (2.9%) 1 16 (30.2%) 10 (14.7%) 2 8 (15.1%) 11 (16.2%) 3 3 (5.7%) 8 (11.8%) 4 4 (7.5%) 8 (11.8%) 5 4 (7.5%) 16 (23.5%) 6 6 (11.3%) 8 (11.8%) 7 2 (3.8%) 0 (0.0%) 8 0 (0.0%) 0 (0.0%) 9 0 (0.0%) 2 (2.9%) 10 1 (1.9%) 3 (4.4%) Total 53 (100.0%) 68 (100.0%)

AI = assault-injured; SaFETy = Serious fighting, Friend weapon carry- ing, community Environment, and firearm Threats.

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017