Psychological Trauma

Journal of Traumatic Stress October 2015, 28, 480–483

B R I E F R E P O R T

Comparison of the PTSD Checklist (PCL) Administered via a Mobile Device Relative to a Paper Form

Matthew Price,1 Eric Kuhn,2 Julia E. Hoffman,2,3 Josef Ruzek,2 and Ron Acierno4,5 1Department of Psychological Science, University of Vermont, Burlington, Vermont, USA

2National Center for PTSD, Dissemination and Training Division, Department of Veterans Affairs Palo Alto Health Care System, Palo Alto, California, USA

3Center for Healthcare Evaluation, Department of Veterans Affairs Palo Alto Healthcare System, Palo Alto, California, USA 4Ralph H. Johnson Veterans Affairs Medical Center, Charleston, South Carolina

5Medical University of South Carolina, Charleston, South Carolina, USA

Mobile devices are increasingly used to administer self-report measures of mental health symptoms. There are significant differences, however, in the way that information is presented on mobile devices compared to the traditional paper forms that were used to administer such measures. Such differences may systematically alter responses. The present study evaluated if and how responses differed for a self-report measure, the PTSD Checklist (PCL), administered via mobile device relative to paper and pencil. Participants were 153 trauma- exposed individuals who completed counterbalanced administrations of the PCL on a mobile device and on paper. PCL total scores (d = 0.07) and item responses did not meaningfully or significantly differ across administrations. Power was sufficient to detect a difference in total score between administrations determined by prior work of 3.46 with a d = 0.23. The magnitude of differences between administration formats was unrelated to prior use of mobile devices or participant age. These findings suggest that responses to self-report measures administered via mobile device are equivalent to those obtained via paper and they can be used with experienced as well as naı̈ve users of mobile devices.

Mobile devices can advance traumatic stress research and treatment (Luxton et al., 2011; Price et al., 2014) through the collection of ecologically valid data (Shiffman, Stone, & Huf- ford, 2008). Use of mobile devices requires that responses to mobile-administered measures are equivalent to responses from paper measures. This assumption is open to empirical investiga- tion and should be evaluated to ensure mobile devices provide valid and reliable measurements.

Mobile devices systematically change the administration of self-report measures. When delivered via paper, items are dis- played in an array that allows all responses to be viewed simul- taneously such that initial responses may influence subsequent answers (Richman, Kiesler, Weisband, & Drasgow, 1999). Al- ternatively, mobile devices typically display a single item per screen. Administration of individual items may focus atten- tion towards item content resulting in systemically different responses.

The present study examined if responses to a self-report mea- sure, the PTSD Checklist (PCL; Weathers et al., 2013), admin-

Copyright C© 2015 Wiley Periodicals, Inc., A Wiley Company. View this article online at wileyonlinelibrary.com DOI: 10.1002/jts.22037

istrated via mobile device differed from paper administration. The PCL has been extensively validated as a measure of PTSD symptoms across diverse samples (Ruggiero, Ben, Scotti, & Ra- balais, 2003). A standardized paper version of the PCL is avail- able via request from the National Center for PTSD (NCPTSD). The PCL is available in a standardized format for mobile de- vices as part of the PE Coach mobile application (Reger et al., 2013). It was hypothesized that responses between PCL total score and item responses across mobile and paper administra- tions would be comparable due to prior evidence that suggested minimal differences between standardized tests administered via paper and computer (Bush et al., 2013; Campbell et al., 1999; Finger & Ones, 1999).

Method

Participants

Participants, aged M = 32.34 years (SD = 14.42), were 153 individuals recruited from a Level 1 trauma center (n = 22, 14.3%), a Veteran’s Affairs medical center outpatient mental health service (VAMC; n = 38, 24.7%), an outpatient clinic

480

Mobile Comparison of PCL 481

Table 1 Descriptive Information Unadjusted for Time Between PCL Administrations

Variable n %

Location Veteran Affairs Medical Center 38 24.7 Female 8 21.1 Community 87 57.1 Female 65 73.9 Outpatient clinic 6 3.9 Female 6 100.0 Trauma center 22 14.3 Female 6 27.3

PTSD diagnosis 62 40.3 Own smartphone 118 76.6

Use e-mail on phone 117 76.0 Use apps on phone 113 73.4 Use games on phone 97 63.0 Use Internet on phone 122 79.2

Note. N = 153. PCL = Posttraumatic Stress Checklist.

for trauma victims (n = 6, 3.9%), and the community (n = 87, 57.1%). Descriptive information is presented in Table 1.

Measures and Procedure

The Posttraumatic Checklist-Civilian Version (PCL-C: Weath- ers, Litz, Huska, & Keane, 1994) is a 17-item self-report mea- sure that assesses PTSD symptom severity. Symptoms are rated on a 5-point Likert-type scale, ranging from 1 = not at all to 5 = extremely, for the past month. Internal consistency for the current study was excellent with α = .95 for both ad- ministrations. The measure was administered twice, once via the paper form available from the NCPTSD (Weathers, Litz, Huska, & Keane, 2003) and once via PE Coach. The Life Events Checklist (LEC; Weathers et al., 2013) is a 17-item self-report measure assessing trauma exposure. Use of Inter- net and mobile devices was assessed with questions adapted from a survey from the Pew Internet and American Life Project (2012). Questions assessed if various tasks were completed reg- ularly completed on smartphones and mobile devices using a yes/no format (e.g., “Do you regularly check e-mail on your smartphone?”).

Medical records were used to confirm trauma exposure for Level-1 trauma, VAMC, and outpatient clinic participants. A diagnosis of PTSD was the indicator of trauma exposure for VAMC and outpatient clinic participants whereas the presenting trauma was used for Level-1 trauma center participants. Com- munity participants were screened with the LEC to determine if they experienced or witnessed a traumatic event. Follow- up questions confirmed the validity of the Criterion A event. The community sample was administered the PTSD module of the Structured Clinical Interview for the DSM-IV by trained

research staff for the most stressful event identified by the LEC (SCID; First, Spitzer, Gibbon, & Williams, 2002). No other modules of the SCID were administered.

Participants completed the PCL on an iPod Touch (4th gen- eration, 3.5′′ screen) and on paper with a 35-minute (Med = 35, interquartile range: 25) interval between administrations. After the second administration, participants completed the use of Internet and mobile devices survey, and demographics questionnaire. Participants from the community were also given the PTSD module from the SCID and 27% met criteria for PTSD. Interviews were administered by trained research assis- tants and audio recorded. Interviews were double coded from the recording by a clinical psychologist with 100% diagnos- tic agreement. The order in which mobile and paper versions were administered was counterbalanced using a randomization sequence. Randomization occurred in blocks of 10 and each data collection site was allocated 10 blocks. Institutional re- view boards of the agencies where this research was conducted approved all procedures and all participants consented to the study.

Data Analysis

Using the guidelines of Bland and Altman (1986), a clini- cally meaningful margin of error between the two methods of measurement of 3.46 was established (see Supplemental Table 1) from nine prior studies where the PCL was adminis- tered repeatedly. A difference score between the total scores for both administrations was obtained by subtracting mobile device scores from paper scores. Comparisons were made with repeated-measures analysis of covariance in which length be- tween administrations was used as a covariate. The mean of the distribution of difference scores was calculated with the 95% confidence interval (CI). If the 95% CI of the difference scores was within the clinically meaningful margin of error then the two methods were considered interchangeable. A mar- gin of error of 1.00 was used for differences between indi- vidual items. Bivariate correlations between both measure ad- ministrations and intraclass correlation coefficients (ICC) were also computed. One participant declined to answer questions about use of a mobile devices after reporting they did not own a smartphone. There were no missing data on the PCL administrations.

Results

Adjusted for time between administrations, the mean difference between paper (M = 40.24, SD = 16.69) and mobile device (M = 39.08, SD = 15.97) administration was 1.17 points with 95% CI [1.13, 1.21] (Table 2). The upper limit of the 95% CI for the mean difference was within the margin of error. The effect size for the difference was d = 0.07. Test-retest reliability was r = .93. The ICC was .96, 95% CI [.95, .97]. Mean differences at the item level ranged from 0.001 to 0.22. The highest upper limit for the 95% CI at the item level was 0.37 for Item 8. Therefore,

Journal of Traumatic Stress DOI 10.1002/jts. Published on behalf of the International Society for Traumatic Stress Studies.

482 Price et al.

Table 2 Mean Difference and 95% CI for PCL Items and Total Score

PCL M Diff 95% CI

Item 1. Intrusive thoughts 0.02 [−0.11, 0.15] 2. Nightmares 0.05 [−0.07, 0.18] 3. Reliving 0.13 [−0.01, 0.27] 4. Emotional cue reactivity 0.12 [−0.04, 0.28] 5. Physiological cue reactivity 0.14 [0.00, 0.27] 6. Avoidance of thoughts 0.04 [−0.14, 0.22] 7. Avoidance of reminders 0.13 [−0.04, 0.29] 8. Trauma-related amnesia 0.22 [0.08, 0.37] 9. Loss of interest 0.05 [−0.08, 0.17] 10. Feeling detached −0.09 [−0.22, 0.05] 11. Lack of positive emotion 0.09 [−0.03, 0.22] 12. Foreshortened future 0.03 [−0.11, 0.17] 13. Sleep problems −0.01 [−0.13, 0.12] 14. Irritability or anger 0.07 [−0.05, 0.20] 15. Difficulty of concentrating −0.04 [−0.18, 0.10] 16. Overly alert 0.04 [−0.08, 0.16] 17. Easily startled 0.14 [0.02, 0.26] Total 1.17 [1.13, 1.21]

Note. Sample size = 153. Margin of error for Total scale = 3.46. Margin of error for items = 1.00. Difference score calculated as paper minus mobile. PCL = Posttraumatic Stress Checklist.

all of the items were within the margin of error (1.00). Test- retest reliability at the item level ranged from r = .66 to .88 and ICC = .75 to .93.

There were no differences in administrations across the dif- ferent locations, F(3, 149) = 1.05, p = .373. Results were consistent across the combined sample in that the upper limit of the 95% CI for the sample obtained from the trauma cen- ter, M = 0.45, 95% CI [0.45, 0.45]; VAMC, M = 2.72, 95% CI [2.60, 2.85]; and community sample, M = 0.65, 95% CI [0.58, 0.72] were within the margin of error for the total scale. Test-retest reliability within each group was consistent with the total sample: trauma center, r = .89, ICC = .94, 95% CI [.86, .98]; VAMC, r = .89, ICC = .94, 95% CI [.89, .97]; com- munity sample, r = .91, ICC = .95, 95% CI [.93, .97]. Mean differences at the item level ranged from 0.00 to 0.36 for the trauma center, from 0.00 to 0.37 for the VAMC, and from 0.01 to 0.20 for the community sample. The highest upper limit for the 95% CI for each item was within the margin of error for the trauma center (0.65), VAMC (0.65), and the community sample (0.40).

The relation between use of smartphone functions and dif- ference in total PCL scores across the administrations was as- sessed with one-way analyses of variance. Differences in total scores were not related to smartphone ownership, F(1, 149) = 1.51, p = .221; use of e-mail via smartphone, F(1, 148) = 0.60, p = .439); use of apps, F(1, 147) = 0.78, p = .378); use of games, F(1, 148) = 0.78, p = .379; and use of the In-

ternet on a smartphone, F(1, 148) = 0.78, p = .379. Finally, differences in total PCL scores were unrelated to age (r = .04, p = .598).

Discussion

The present study suggested that there were minimal dif- ferences between a self-report measure of PTSD symptoms administered via mobile device or paper in a heterogeneous sample of trauma-exposed adults. The lack of a relation be- tween prior experiences using a mobile device, age, and differ- ences in total score indicates that mobile devices are a viable strategy for those who have minimal training or experience with this technology. Prior work demonstrated that among patients, demographic characteristics and prior experience is largely un- related to willingness to use technology for healthcare (Price et al., 2013). There is evidence, however, to suggest that prior use is relevant for clinicians (Kuhn et al., 2014). Clinicians with experience using mobile devices or who own a personal mobile device were more receptive to use such technologies in treatment. Ensuring that clinicians are capable and comfort- able with such devices will be necessary for proper measure administration as patients are likely to turn to their therapist for technical assistance or tutorials with these technologies (Price & Gros, 2014).

The present study had several limitations. The mobile ad- ministration was not conducted in a naturalistic environment where such measures administered via mobile device are most likely to be completed insofar as this was a research study with informed consent processes. The effect of environmental influences on responses is unknown. Although it is unlikely that the environment would systematically influence mobile re- sponses relative to paper response, measures completed on a mobile device are more likely to be completed in a variety of contexts in which other factors could influence responses. Re- searchers are advised to collect data on the context in which measures are completed to assess potential sources of bias. The study evaluated a single self-report measure of PTSD without a lengthy assessment battery. Thus, the current study was unable to examine effects related to fatigue across the administration of multiple measures via a mobile device. The current study supported the null hypothesis that there were no differences between scores across paper and mobile versions of the PCL, which is conceptually and pragmatically challenging (Piaggo, Elbourne, Pocock, & Evans, 2006). Although the current study had sufficient power to detect an effect as small as 0.23, con- siderably more power would be needed to detect an effect at the obtained effect size of 0.07 (n = 1,604). Continued studies that demonstrate the clinical equivalence of measurements ob- tained via mobile device relative to paper should be conducted to further validate these findings. Finally, PTSD diagnoses were obtained with different methods across the subsamples, and the accuracy of diagnoses in medical records has been questioned (Holowka et al., 2014).

Journal of Traumatic Stress DOI 10.1002/jts. Published on behalf of the International Society for Traumatic Stress Studies.

Mobile Comparison of PCL 483

The current study provides empirical support regarding the lack of differences for measures administered via mobile de- vice. Given the high rates of smartphone ownership, the results from the present study suggest that mobile devices are an appro- priate method for population screens of PTSD. Such a method would assist in the efficient allocation of resources in events of mass trauma such as a natural disaster.

References Bland, M. J., & Altman, D. G. (1986). Statistical methods for assessing agree-

ment between two methods of clinical assessment. The Lancet, 327, 307– 310. doi:10.1016/S0140-6736(86)90837-8

Bush, N. E., Skopp, N., Smolenski, D., Crumpton, R., & Fairall, J. (2013). Behavioral screening measures delivered with a smartphone app: psycho- metric properties and user preference. The Journal of Nervous and Mental Disease, 201, 991–995. doi:10.1097/NMD.0000000000000039

Campbell, K. A., Rohlman, D. S., Storzbach, D., Binder, L. M., Anger, W. K., Kovera, C. A., . . . Grossmann, S. J. (1999). Test-retest reliability of psychological and neurobehavioral tests self-administered by computer. As- sessment, 6, 21–32. doi:10.1177/107319119900600103

Finger, M. S., & Ones, D. S. (1999). Psychometric equivalence of the computer and booklet forms of the MMPI: A meta-analysis. Psychological Assessment, 11, 58–66. doi:10.1037/1040-3590.11.1.58

First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (2002). Struc- tured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition. N ew York, NY: Biometrics Research, New York State Psychiatric Institute.

Holowka, D. W., Marx, B. P., Gates, M. A., Litman, H. J., Ranganathan, G., Rosen, R. C., & Keane, T. M. (2014). PTSD diagnostic validity in Vet- erans Affairs electronic records of Iraq and Afghanistan veterans. Jour- nal of Consulting and Clinical Psychology, 82, 569–579. doi:10.1037/ a0036347

Kuhn, E., Eftekhari, A., Hoffman, J. E., Crowley, J. J., Ramsey, K. M., Reger, G. M., & Ruzek, J. I. (2014). Clinician perceptions of using a smartphone app with prolonged exposure therapy. Administration and Policy in Mental Health and Mental Health Services Research, 1–8. doi:10.1007/s10488-013- 0532-2

Luxton, D. D., McCann, R. A., Bush, N. E., Mishkind, M. C., & Reger, G. M. (2011). mHealth for mental health: Integrating smartphone technology in behavioral healthcare. Professional Psychology: Research and Practice, 42, 505–512. doi:10.1037/a0024485

Pew Internet and American Life Project. (2012, September). Explore Survey Questions. Retrieved from http://www.pewinternet.org/Static-Pages/Data-

Tools/Explore-Survey-Questions/Roper-Center.aspx?item={0368CEFB- 1706-4995-B395-925639C0B22F}

Piaggo, G., Elbourne, D. R., Pocock, S. J., & Evans, S. J. W. (2006). Reporting of noninferority and equivalence randomized trials: An extension of the CONSORT statement. Journal of the American Medical Association, 295, 1152–1161. doi:10.1001/jama.295.10.1152

Price, M., & Gros, D. F. (2014). Examination of prior experience with telehealth and comfort with telehealth technology as a moderator of treatment response for PTSD and depression with veterans. International Journal of Psychiatry in Medicine, 48, 57–67. doi:10.2190/PM.48.1.e

Price, M., Williamson, D., McCandless, R., Mueller, M., Gregoski, M., Brunner-Jackson, B., . . . Treiber, F. (2013). Hispanic migrant farm work- ers’ attitudes toward mobile phone-based telehealth for management of chronic health conditions. Journal of Medical Internet Research, 15, e76. doi:10.2196/jmir.2500

Price, M., Yuen, E. K., Goetter, E. M., Herbert, J. D., Forman, E. M., Acierno, R., & Ruggiero, K. J. (2014). mHealth: A mechanism to deliver more acces- sible, more effective mental health care. Clinical Psychology & Psychother- apy, 21, 427–436. doi:10.1002/cpp.1855

Reger, G. M., Hoffman, J., Riggs, D., Rothbaum, B. O., Ruzek, J., Holloway, K. M., & Kuhn, E. (2013). The “PE coach” smartphone application: An innovative approach to improving implementation, fidelity, and homework adherence during prolonged exposure. Psychological Services, 10, 342–349. doi:10.1037/a0032774

Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta- analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Ap- plied Psychology, 84, 754–775. doi:10.1037/0021-9010.84.5.754

Ruggiero, K. J., Ben, K. D., Scotti, J. R., & Rabalais, A. E. (2003). Psy- chometric properties of the PTSD Checklist—Civilian version. Journal of Traumatic Stress, 16, 495–502. doi:10.1023/A:1025714729117

Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological mo- mentary assessment. Annual Review of Clinical Psychology, 4, 1–32. doi:10.1146/annurev.clinpsy.3.022806.091415

Weathers, F., Litz, B., Huska, J., & Keane, T. (1994). Post-Traumatic Stress Disorder Checklist (PCL-C) for DSM-IV. Boston, MA: National Center for PTSD.

Weathers, F., Litz, B., Huska, J., & Keane, T. (2003). PTSD Check- list - Civilian Version. Retrieved from http://www.mirecc.va.gov/docs/ visn6/3_PTSD_CheckList_and_Scoring.pdf

Weathers, F. W., Blake, D. D., Schnurr, P. P., Kaloupek, D. G., Marx, B. P., & Keane, T. M. (2013). The Life Events Checklist for DSM-5 (LEC-5). Unpublished instrument. Retrieved from http://www.ptsd.va.gov

Journal of Traumatic Stress DOI 10.1002/jts. Published on behalf of the International Society for Traumatic Stress Studies.