Assignment: DoDEA, LEAs Education, and Support

ENS Kendall D. Moore, MSC USNR*; Amanda J. Fairchild, PhD*

; LTC Nikki R. Wooten, LG USAR†; Zi Jia Ng, MA*

ABSTRACT Introduction: Military-connected youth are at increased risk for experiencing distinct psychosocial and behavioral health vulnerabilities. Although behavioral health interventions have been developed to treat vulnerabilities in military-connected youth, little is known about the methodological quality of studies evaluating these interventions. In this study, a systematic review of behavioral health interventions for military-connected youth was conducted to examine methodological quality and treatment outcomes. Materials and Methods: Electronic databases were systematically searched for studies evaluating behavioral health interventions for military-connected youth which yielded 3,324 citations. Methodological quality was evaluated by 2 researchers with 3 measures that assessed scientific rigor, transparency, external and internal validity, and power for quantitative, qualitative, and mixed-method trials. Interrater reliability was strong (κ = 0.81). Sample characteristics and treatment outcomes were also assessed. Results: Fourteen studies meeting full inclusion criteria evaluated 10 behavioral health interventions. Methodological quality scores for all studies were poor to fair, with limitations in reporting, external and internal validity, and power. Research designs were predominantly nonexperimental. Treatment effects for both psychosocial and behavioral health outcomes were consistently positive for all studies. In studies reporting effect sizes, treatment effects were small to moderate (d = 0.01 –0.42, odds ratio = 0.04–0.47, b = −0.02–0.56). Demographic and military characteristics of samples were inconsistently reported. Conclusion: Behavioral health interventions for military-connected youth have noteworthy methodological limitations, indicating a need to employ more rigorous research strategies. Positive treatment outcomes, however, suggest promising interventions for improving psychosocial and behavioral health problems in military-connected youth. Future research directions and implications for clinical-community practice are also discussed.

INTRODUCTION Historically, both epidemiological and intervention research focused on behavioral health outcomes exclusively among military service members, rather than also considering military spouses and children.1,2 Recent investigations show that military-connected youth experience distinct psychosocial and behavioral health vulnerabilities as a result of cultural aspects of the military lifestyle.3 The importance of behavioral health professionals to intervene and mitigate these vulnerabilities and to maintain military family readiness has been realized by both Department of Defense and community leaders, leading to the development and implementation of numerous behavioral health interventions to address these concerns in military-connected youth.4–6 However, limited evidence exists on the methodological quality (MQ) of these studies despite calls for additional evaluation.7 To date, no systematic review has been conducted to assess the MQ of behavioral health interventions for military-connected youth.7 Given the family focus of military health promotion and overall lack of evaluation evidence,8,9 examining behavioral health interventions for military-connected youth can inform future military behavioral health research and practice. To expand this knowledge base, this systematic review

evaluated MQ10,11 and intervention outcomes to determine the state of the field of behavioral health interventions for military-connected youth.

Military Culture and Unique Stressors of Military-Connected Youth

Given the values and demands of the military lifestyle, military connected youth experience unique stressors compared to their civilian counterparts. A salient stressor is the deployment cycle which consists of 4 phases: pre-deployment, deployment, reunion, and postdeployment.4 Each phase of the deployment cycle has been associated with a variety of family stressors12 that impacts psychosocial adjustment and results in anxiety, depression, behavioral problems, substance use, and suicidal ideation among military-connected youth across all ages.3,13 Academic performance and socioemotional development is also affected. On average, military-connected youth experience 9 school transitions throughout their academic careers.14 Relocating to new school districts impacts academic performance and achievement, and disrupts social connections and friendships.15,16 Numerous relocations and military deployments may present structural barriers to working with military-connected youth and their families. To improve intervention efficacy, behavioral health professionals should be aware of military lifestyle factors that can influence both intervention dissemination and outcomes.

For example, the military chain of command distinguishes roles and responsibilities between enlisted service members and officers, and ensures order, discipline accountability, and efficiency. Behavioral health providers may need permission from senior military leadership to implement and evaluate intervention programs for military service members and families.17 Awareness of this structure and how to effectively work within military organizations is critical for effective treatment services. Doctrine and core values are also fundamental aspects of military culture relevant to providing behavioral health interventions. Basic military training experiences and subsequent ideological transformations among service members may have an unintended impact on family functioning.18 Flittner O’Grady et al19 found that intervention programs for military-connected youth may have low participation because of perceived stigma and potentially a negative impact on the service member’s military career and deployment readiness.20

Without a scientifically rigorous review, resources allocated to develop and disseminate military interventions are inefficiently used and their efficacy, transportability, and cost-effectiveness21 remain relatively unknown. Therefore, assessing the MQ of both randomized and nonrandomized studies evaluating behavioral health interventions for military connected youth provides a crucial next step in military behavioral health research and practice. Downs and Black22 noted that randomized and nonrandomized quantitative trials should measure 3 factors—the intervention, potential confounders, and treatment outcomes—to determine whether there is an association between the intervention and outcome, and to minimize research design weaknesses that introduce measurement bias in casual inference. However, limited measurement tools exist to identify strengths and weaknesses of each dimension of MQ, which include reporting and transparency, internal and external validity, confounding, selection bias, and power.22,23 For qualitative investigations, evaluating a study ’s capacity to generate knowledge, facilitate interpretation of quantitative studies, and highlight factors relevant to an intervention’s effectiveness are suggested.10

In this systematic review, 2 research questions are addressed: (1) What sample characteristics and treatment outcomes are examined in studies evaluating behavioral health interventions for military-connected youth? and (2) What is the MQ of studies evaluating behavioral health interventions for military connected youth? Findings from reviewed studies are summarized by MQ dimensions, studies demonstrating strong MQ, and behavioral health interventions requiring additional evidentiary support. Assessing the MQ of studies evaluating treatment outcomes targeted by military behavioral health interventions will assist in better identification and dissemination of the most efficacious interventions for military-connected youth.

METHOD

Study Eligibility Criteria

Cochrane systematic review methods24 were used with the following inclusion criteria: (1) peer-reviewed research articles, (2) published in English from 1995 to 2016, and (3) implementation/evaluation of behavioral health interventions involving U.S. military-connected youth (0–17 years of age). Exclusion criteria were (1) theses or dissertations, (2) articles not published in English, (3) nonempirical articles, (4) articles evaluating pharmacological interventions, (5) review articles, and (6) evaluations which did not report military-connected youth outcomes.

Search Strategy and Study Selection

Databases searched included PsycINFO, PubMed, PsycARTICLES, PILOTS, and the Military Family Research Institute website. Key words included military AND (child OR youth OR adolescent OR family) AND (mental health OR behavioral health) AND (intervention OR prevention OR treatment OR therapy). Reference lists from theoretical or review articles not meeting inclusion criteria were also searched. Search results yielded 3,324 citations. After removing duplicate articles and reviewing articles for relevance, 302 articles were screened. Of articles screened, 14 studies met full inclusion criteria for synthesis. Figure 1 displays study selection methodology. Coding Procedures and MQ Measures Three MQ measures were used to evaluate quantitative, qualitative, and mixed-method trials. Two researchers coded the MQ of studies. Interrater reliability across coders, as defined by Cohen’s kappa coefficient (κ),25 demonstrated strong agreement between raters (κ = 0.81).26

Quantitative Trials

The Quality Checklist for RCTs and Observational Studies is a 33-item checklist that assessed the MQ of both randomized and nonrandomized trials.22 The Reporting, External Validity, Bias, and Confounding subscales were scored 0 (no or unable to determine), or 1 (yes); a single item for “Power” indicated whether a study conducted a power analysis (0 = no, 1 = yes, one measure, 2 = yes, two or more measures; Kennelly10). The total MQ score ranged from 0 to 32 for both randomized and nonrandomized trials. The total MQ score for randomized and nonrandomized trials were categorized as poor (<14), fair (15–19), and good (>20).10 The suitability of a study to assess intervention effectiveness was given a classification rating of greatest, moderate, or least.10 The Quality Checklist has demonstrated good internal consistency reliability,22 test-retest reliability,27,28 and convergent validity.29

Qualitative Trials

Adapted from Kennelly,10 this 26-item measure has six subscales, including Research Design, Sampling, Data Collection, Data Analysis, Findings/Results, and Research Value. All items are scored (0 no or unable to determine) or 1 (yes) and scores range from 0 to 26. Studies were classified as high, fair, or low value contingent on MQ scores. See Kennelly5 for classifier definitions.

Mixed-Methods Trials The Mixed-Method Appraisal Tool (MMAT)11 is a 19-item measure designed for integrated systematic reviews which include mixed-method studies. The instrument has five subscales, including Qualitative, Quantitative Randomized Controlled (trials), Quantitative Nonrandomized, Quantitative Descriptive, and Mixed Methods. All items are scored 0 (no) or 1 (yes). For interpretive purposes, the overall quality score is the lowest score of the study components. See Pluye et al11 for scoring details. The MMAT has demonstrated evidence for validity and reliability.30

RESULTS Fourteen studies meeting inclusion criteria evaluated 10 behavioral health interventions (Tables I and II). The majority (64.3%) of interventions included family components (i.e., for both military-connected youth and their parents) and group level (57.1%) components (i.e., parents-only or youth-only); 78.6% of studies were nonexperimental. The majority (60%) of the 10 quantitative studies used pre/post-test designs and three of the studies used longitudinal designs.31–33 Two studies34,35 used qualitative designs and two studies used mixed methods designs.36,37

Sample Characteristics Three primary types of samples were reported: (1) military connected youth, (2) military/nonmilitary families, and (3) military/nonmilitary caregivers or parents. For studies including military-connected youth, sample sizes ranged from N = 15 to 3,810 (M = 645.89, SD = 1,202.16). Family sample sizes ranged from N = 19 to 2,615 (M = 716.2, SD = 1,069.43). Caregiver/parent sample sizes ranged from N = 28 to 607 (M = 267.5, SD = 248.96). For military-connected youth samples (n = 10), six studies reported mean age, nine reported sex characteristics, and four reported race/ethnicity. Among all studies (n = 14), seven reported military characteristics, which included branch of service, service component, “or” the number of deployments. Distribution of research participants among branches of service varied: Army (41.7%), Marine Corps (28%), Navy (17.9%), Air Force (9.2%), and Coast Guard (<1%). Of the six studies reporting the service component of the caregiver(s), 67.8% were active duty families, 18.9% were Reserve families, and 11.3% were National Guard families.

Intervention Outcomes and Key Findings Table III presents all behavioral health and psychosocial outcomes evaluated in the reviewed studies. Six of 10 quantitative studies reported effect sizes of their primary outcomes (Table IV). The majority of these studies reported positive intervention outcomes, although their effect sizes had modest clinical significance on the basis of recommended thresholds.27,38 Four quantitative studies did not report effect sizes, but three indicated that behavioral health interventions could potentially improve global self-worth for military-connected youth39 and mental health for youth and caregivers.33,40 Missing data were reported in five studies.33,34,37,41,42.

Of these, only two implemented currently recommended strategies to handle missing observations.33,34 In two qualitative trials, overarching themes were that both caregiver34 and teacher35 participants generally reported interventions to be helpful and that programs strengthened targeted outcomes. Improving the role of family communication, quality of family relationships, and parenting were identified as positive outcomes.34 Interventions also provided a critical opportunity for military-connected youth to interact with other youth and families, which offered a safe environment to discuss deployment stressors. In one mixed-methods study, Perkins and Borden37 indicated that participants in their parent-only focus group unanimously reported positive social and academic outcomes from the “Youth Action Program” as a result of participation in team-building and “big connection” activities, which provided social support for military-connected youth during parental deployments.

Methodological Quality

Table I provides aggregate MQ point distributions by dimension for each quantitative study, with overall MQ scores ranging from 6 to 25 of a possible 32 points for randomized and nonrandomized intervention trials (M = 17.5, SD = 5.06). Table II provides point distributions for qualitative and mixed-methods studies.

Randomized Quantitative Trials

Three RCTs19,33,42 demonstrated relative strengths on the “Reporting” subscale with scores ranging from 10 to 13 of a maximum of 13 points (M = 11.3, SD = 1.53). External validity was a relative weakness, with all studies scoring MQ = 2 of a maximum of 4 points. These studies demonstrated fair “internal validity bias” and “confounding bias” scores ranged from 9 to 10 of a maximum of 13 points (M = 9.33, SD = 0.58). A primary concern regarding “internal validity” included ambiguous recruitment procedures.

Nonrandomized Quantitative Trials

These studies demonstrated poor to good scores of MQ on the “Reporting” subscale with scores ranging from 4 to 10 of a maximum of 13 points (M = 7.9, SD = 2.04). Several studies reported limited psychosocial or military characteristics, and did not identify potential confounders for primary outcomes.32,39,41,43 External validity ranged from poor to good for all studies (0–3 of a maximum of 4 points). Internal validity scores were poor to fair and ranged from 2 to 8 of a maximum of 13 points. The majority of studies31,32,40,41,44 had moderate scores, but lacked randomization and a control or comparison group. Only one quantitative study41 reported power or mentioned a priori effect sizes to inform primary hypotheses, and scores ranged from 0 to 2 (M = 0.2, SD = 0.63).

Qualitative and Mixed-Method Trials

Qualitative studies reviewed34,35 demonstrated similar scores across subscales of methodological rigor (M = 17, SD = 4.24). Similar to all studies reviewed, qualitative studies demonstrated a relative weakness in sampling procedures and treatment fidelity. Overall MQ scores for both mixed methods studies were 5 of a possible 11 points which classified as scores of 0% and 25%, respectively, consistent with recommendations by MMAT developers.11 Similar to quantitative and qualitative trials, areas of concern included sampling procedures and low response rates (i.e., below 60%) in follow-up analyses.36

DISCUSSION

This systematic review assessed MQ, sample characteristics, and treatment outcomes of studies evaluating behavioral health interventions for military-connected youth. The majority of studies received poor MQ scores on external validity, suggesting limited generalizability to the broader military-connected youth population. Other noteworthy methodological weaknesses included treatment fidelity and attrition. Variability in sample characteristics and treatment outcomes was observed, which may be an artifact of this emerging area of inquiry and military factors that pose barriers to recruiting military connected youth and their families for research.17,45 Positive treatment outcomes were consistently found for various psychosocial and behavioral indicators for both military-connected youth and caregivers. Studies which provided effect sizes, however, reported small to moderate effects. Our findings suggest the need for more scientific rigor in future behavioral health intervention research involving military-connected youth.

Components Impacting MQ

Study samples were primarily affiliated with the Marine Corps (n = 4) or Army ( n = 6). Few military-connected youth affiliated with the Air Force or Coast Guard participated in these studies. Although Marines and Soldiers experience longer deployments than service members in the Air Force, Navy, and Coast Guard,46 this finding suggests limited development and implementation of behavioral health interventions for youth from these service branches. Sociodemographics (e.g., age and race/ethnicity) and military characteristics (e.g., military rank, deployment frequency, and length) were inconsistently reported; and sample sizes varied widely. Limited research funding or ability to obtain permission to access and conduct research with military populations may also account for variability in sample characteristics.47 Wooten et al45 describe innovative recruitment strategies, including social media and remote communications, and also highlight the need for permission from senior leadership and multiple institutional review board approvals to conduct research with military populations. Future research may benefit from an examination of how technology and other nontraditional recruitment strategies can be utilized to recruit military-connected youth and their families for research. Both quantitative and qualitative studies reported improved psychosocial outcomes (e.g., academic performance, family relationships). Positive psychology domains such as resilience and global sense of self-worth were assessed in these intervention studies which is consistent with recent approaches to behavioral health48 and military health promotion programs such as Comprehensive Solider and Family Fitness.49 Military-connected youth may benefit from strengths-based and resiliency-enhancing strategies which are less stigmatizing and capitalize on skills military families acquire throughout the service member’s military career.50

Methodological Considerations for Quantitative Trials

In addition to limitations concerning generalizability of study samples, treatment fidelity was another methodological concern.19,42,44 Fidelity monitoring was inconsistently reported or monitoring protocols lacked sufficient detail to determine monitoring strategies and remedial training implemented. Future research may benefit from continuous monitoring of program implementation to assess treatment fidelity and improve MQ.51 Limitations with statistical analyses included not using sample weighting or missing data techniques when necessary and not conducting power analyses. In addition, only one study evaluated mediation effects.33 Modern approaches to prevention and intervention science include conducting mediation analysis to better understand underlying intervention processes and treatment components that result in targeted behavioral health outcomes.52 The inclusion of mediation analyses is a natural next step for future research.

Methodological Considerations for Qualitative and Mixed-Method Trials

Methodological rigor for qualitative studies varied. Ashurst et al34 and Fees et al35 both used focus groups which are designed to collect data on a specific topic, but lack generalizability and are sensitive to confirmation bias.53 Neither qualitative study used triangulation which improves methodological rigor in qualitative research.54 MQ for the two mixed-method studies36,37 was poor because of discrepancies between the rigor of quantitative compared to qualitative analyses. Methodological weaknesses included low response rates and sampling strategies. Although mixed method evaluations are integrative, significant limitations were observed in the “mixing” of qualitative and quantitative results to assess treatment outcomes. Future qualitative and mixed-methods studies evaluating behavioral health interventions for military-connected youth should employ research strategies that increase scientific rigor and generalizability.

Future Research Recommendations

Given methodological weaknesses identified in this review, we propose several suggestions for future research. Replication and expansion on current evidence is needed that incorporates the following: (1) consistent reporting of sociodemographic and military characteristics, and diversifying samples by branch of service, service component, and military rank to increase external validity, (2) modifying existing behavioral health interventions for specific military populations to increase scale and transportability, (3) implementing statistical techniques that account for confounders and missing data to improve scientific rigor, and (4) training interventionists on the intervention protocol and closely monitor treatment fidelity to improve internal validity.

Clinical-Community Practice Implications

Improving military family relationships can have positive effects on military-connected youth behavioral health functioning.55 At the individual level, academic performance, social, and emotional well-being can be improved. School and community-based interventions can increase social support networks for military-connected youth especially after relocating to a new military installation and during deployments. Although school-based interventions for military connected youth are limited, existing models56 and programs such as the “Student-2-Student” program,57 have shown promising outcomes. Additional advantages to integrating school-based interventions include available peer and social support networks, better access to services, and decreases in social sigma which may be protective during developmental milestones and military-related transitions during childhood.14,58 Development and “maintenance” of community capacity and partnerships to support military-connected youth and their families is also important for both researchers and community organizations. Community-based approaches were integrated into Lester’s et al31,32,40 evaluations of FOCUS, and maintaining leadership and organizational partnerships were instrumental to implementation quality, treatment fidelity, outcomes, resources, research and staff support.59 Community partnerships can also be critical to the transportability of interventions and sustainability of resiliency-enhancing programs.60

CONCLUSION

To our knowledge, this is the first comprehensive review of behavioral health interventions for military-connected youth. A key strength of the study is that we analyzed the MQ of existing interventions using empirically validated measures for randomized and nonrandomized quantitative, qualitative, and mixed-methods trials10,11,22 consistent with current recommendations.24 Although using MQ measures increased the rigor of our assessment, it limited our ability to compare MQ scores “between” quantitative, qualitative, and mixed method trials. As additional research is conducted to both improve the assessment of MQ and progress towards gold standard methodology of integrative systematic reviews,61 perhaps replication of this study using a single instrument for measuring MQ may facilitate a comparative assessment of MQ. The state of the science on behavioral health interventions for military-connected youth has noteworthy strengths and limitations. Strengths include the development, implementation, and evaluation of interventions that provide empirical evidence about strategies that improve individual and family psychosocial and behavioral health outcomes in military connected youth.

Limitations include the predominant use of nonexperimental research designs and less rigorous statistical methods to evaluate these interventions. Our findings build on prior evidence8 and suggest more rigorous research designs examining intervention effectiveness, efficacy, and transportability of behavioral health interventions for military connected youth. Collaborative efforts between the Department of Defense, school, and community stakeholders to improve MQ, treatment outcomes, and generalizability of research evaluating behavioral health interventions for military-connected youth across branches of military service and service components is vital to both military health and readiness of the U.S. Armed Forces.

ACKNOWLEDGMENTS

Mr. Moore is an ensign in the U. S. Naval Reserves and Dr. Wooten is a lieutenant colonel in the U. S. Army Reserves, but neither conducted this study as a part of their official military duties. Other listed authors reported no conflicts of interest. All listed authors approved the manuscript and its submission to Military Medicine. No other individuals contributed to the work reported in this manuscript. Dr. Wooten’s effort was funded by the National Institute on Drug Abuse (K01DA037412; PI: Dr. Wooten). No other authors were funded for their contributions.

REFERENCES

Adams R, Nikitin RV, Wooten NR, Williams TV, Larson MJ: The association of combat exposure and injuries with behavioral health problems among enlisted female army members returning from Iraq or Afghanistan. J Traumatic Stress 2016; 10(1): 1. 2.

Resick PA, Wachen JS, Mintz J, et al: A randomized clinical trial of group cognitive processing therapy compared with group presentcentered therapy for PTSD among active duty military personnel.

J Con and Clin Psych 2015; 83(6): 1058–68. 3.

Cederbaum JA, Gilreath TD, Benbenishty R, et al: Well-being and suicidal ideation of secondary school students from military families.

J Adol Health 2014; 54(6): 672–77.

Johnson SJ, Sherman MD, Hoffman JS, et al: The psychological needs of US military service members and their families: a preliminary report. American Psychological Association Presidential Task Force on Military Deployment Services. 2007; Feb: 1–67. Available at http://www .ptsd.ne.gov/publications/military-deployment-task-force-report.pdf; accessed February 2, 2017.

Department of Veterans Affairs/Department of Defense: VA/DoD clinical practice guidelines. Washington, DC: Veteran Health Administration, Department of Defense, 2013. Available at http://www.healthquality.va .gov/guidelines/MH/srb/; accessed February 14, 2017. 6.

Department of Defense. Plans for the Department of Defense for the support of military family readiness. 2008; 581. Available at http:// download.militaryonesource.mil/12038/MOS/Reports/FY2009-ReportMilitaryFamilyReadinessPrograms.pdf; accessed February 14, 2017. 7.

Esposito-Smythers C, Wolff J, Lemmon KM, Bodzy M, Swenson RR, Spirito A: Military youth and the deployment cycle: emotional health consequences and recommendations for intervention. J Fam Psych 2011; 25(4): 497–507.

Slomski A: IOM: Military psychological interventions lack evidence. JAMA 2014; 311(15): 1487–88. Available at http://jamanetwork.com/ journals/jama/fullarticle/1860458; accessed November 3, 2016. 9.

Ohye B, Bui E: The need for an evidence base for interventions to support resilience among military-connected children. JAACAP 2016; 55(1): 10–1. 10.

Kennelly J: Methodological approach to assessing the evidence. In: Reducing Racial/Ethnic Disparities in Reproductive and Perinatal Outcomes, pp 7–19. Edited by Joan IA. New York, NY, Springer-Verlag New York, 2011.