Systematic Review

Assessing Children’s Language Skills at a Distance: Does it Work? Brian Manzanares

Pui Fong Kan

Department of Speech, Language, and Hearing Sciences, University of Colorado at Boulder Boulder, CO Disclosures: Financial: Brian Manzanares and Pui Fong Kan have no financial interests to disclose. Nonfinancial: Brian Manzanares and Pui Fong Kan have no nonfinancial interests to disclose.

Abstract

Down

The purpose of this study was to explore the effects of using videoconferencing to assess children’s language skills. Participants were 6 typically developing monolingual English- speaking children, ages 3;0–5;11, from middle class families. Using a within-subjects design, the participants completed a story-retell task in both videoconferencing (VC) and face-to-face (F2F) conditions. During the task for each condition, children were presented with a story, along with a wordless book. In addition, 4 unfamiliar words were embedded within the story. Results showed that there were no significant differences in the microstructures of their narratives between F2F and VC conditions. Results also showed that children learned the target words in both conditions equally well. The findings in this study provide evidence that the VC and F2F conditions are comparable when administering the story-retell task to typically developing young children. Despite these preliminary findings, more research is needed to verify whether or not similar results would be found with young children with communication challenges.

This study explores whether or not videoconferencing is a valid tool for measuring children’s language skills at a distance. Specifically, we examined preschool children’s story-retell performance and fast-mapping performance in two different conditions—videoconferencing (VC) and face-to face (F2F) conditions. Many clinicians have been using multimedia technologies to offer speech-language services (Theodoros, 2008; Waite, Theodoros, Russell, & Cahill, 2010). The American Speech-Language-Hearing Association (ASHA) commonly refers to this method as telepractice. Although many have considered telepractice to be useful for reaching out to clients with limited access to clinical services (e.g., those who live in remote areas, those who speak a minority language as a home language) and for addressing the shortages of SLPs (Juenger, 2009; Mashima & Doarn, 2008; Rose et al., 2000; Waite et al., 2010), however, very little is known about whether or not common assessment procedures (e.g., eliciting language samples) through telepractice are as effective as those through traditional F2F interactions. Thus, validating these procedures using telepractice is critically needed in order to ensure the quality of the services (ASHA, 2005a; Antonacci, Bloch, Saeed, Yildirim, & Talley, 2008; Brown, Brannon, & Romanow, 2010).

Measuring Children’s Performance at a Distance

According to ASHA, “Telepractice is the application of telecommunication technology to deliver professional services at a distance” (ASHA, 2005a, p. 1). There are several considerations concerning the technology required for telepractice. These include the hardware or devices used, software or programs used, and Internet connections with adequate bandwidth for transmitting large amounts of digital data in real-time. Examples of these types of devices include videophones, closed circuit televisions, Internet-based software with webcams, personal computing devices,

34

loaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

image capture devices, dedicated VC systems, and interfacing instrumentation (such as video laryngoscopy) for telepractice use (ASHA, 2005b). See Appendix A for lists of teleconferencing software and services that are designed for health professionals and other commercial teleconferencing software and services. In this study, we used a web-based software program called Adobe ConnectPro (2012). Regardless of the types of devices used, it is important to ensure privacy and security when using telepractice. That is, SLPs must have a general understanding of the Internet, networking, and security. For example, although the audio-video signals transmitted by Skype’s online software are encrypted and not stored on Skype’s system, text that is sent on the Skype chat screen is stored on the Skype servers.

Several studies found that videoconferencing appears to be an effective modality for measuring children’s language and learning skills. For example, through two experiments, Troseth, Saylor, and Archer (2006) presented instructions on finding toys in rooms to toddlers in three conditions—F2F, VC with adult-child interaction, and noninteractional prerecorded videos. They found that young children (23–25 months old) were able to learn instructions and retrieve toys at above chance levels in only the F2F and VC with interaction conditions. Moreover, they found that the children’s performance in these two conditions were statistically equivalent. More recently, Waite and colleagues (2010) administered a standardized test, Clinical Evaluation of Language Fundamentals –Fourth Edition (CELF-4), to 25 children ages 5–9 years old in both F2F-led and VC-led conditions. Additionally, they simultaneously assessed children in both conditions with offsite and F2F clinicians. They found no significant differences between the F2F and VC conditions and concluded that these two CELF-4 testing environments were not significantly different. These studies provide some evidence that videoconferencing can potentially be used for measuring children’s language and learning skills at a distance.

Despite this promising research, some questions remain. Specifically, does the VC modality work with procedures that require a child to pay close attention to the audio-visual contents (e.g., when implementing a story-retell task or measuring children’s word-learning skills)? The story retelling task (SRT), typically involving the presentation and elicitation of a story, has been widely used to elicit spontaneous, connected speech from children and adults for research and clinical purposes (e.g., Gazella & Stockman, 2003; Hayward, Gillam, & Lien, 2007; Wong, Au, & Stokes, 2004). Story samples have provided information about children’s productive language skills at all levels of performance, including syntax, semantics, and pragmatics (e.g., Gazella & Stockman, 2003; Gillam & Carlile, 1997; Wong et al., 2004). The word-learning task (WLT), which involves a presentation phase and a probing phase, has been used to examine children’s ability to learn new words (e.g., Kan & Windsor, 2010). The target/new words of a WLT can be embedded and presented in a story in the presentation phase of an SRT and children’s knowledge about the new words can be measured after the elicitation phase. Consistent with dynamic assessment, the WLT has been considered to be a potential measure to identify culturally-linguistically diverse children with language impairment (e.g., Burton & Watkins, 2007; Hwa-Froelich & Matsuo, 2005; Kapantzoglou, Restrepo, & Thompson, 2012). Given these procedures, audio-visual input from the VC modality could present challenges for young children engaging in the SRT and the WLT.

The current study attempted to get some preliminary data about the effect of teleconferencing modality on children’s performance on the SRT and the WLT. Specific research questions were:

Down

1. Are there differences in children’s story-retelling performances between VC and F2F conditions?

2. Can children learn new words in VC condition?

3. Are there differences between F2F and VC conditions in the number of novel words children are able to learn while participating in the story retell procedure?

35

loaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

Methods

Participants Participants were 6 typically developing English-speaking children (4 boys and 2 girls;

mean age=4;0; SD=1;0). All of the children were from middle class families without diagnosis of language disorder, hearing loss, or emotional or behavioral disorders as reported by parents in a parent survey. On average, they spent about 9 to 25 hours per week watching television (Mean=14.67; SD=5.57). Three of them had communicated through VC before, two of them had seen or used VC once, and one of them had never experienced VC.

Stimuli Two wordless storybooks by Mercer Mayer—A Boy, a Dog and a Frog (story 1) and Frog

Where Are You? (story 2)—were used. The stories were presented in two conditions: face-to-face (F2F) and videoconferencing (VC) conditions. For the F2F condition, the original storybooks were used. To accommodate for VC, portable document format (PDF) copies of the stories were made to display on the VC website (discussed below). For both conditions, test administrators read the narrative scripts corresponding with the storybooks to the children before another naïve listener elicited retellings of the story. Narrative story scripts were written for each wordless storybook. These scripts mostly contain descriptive sentences about the story.

Four unfamiliar words were embedded in each of the two wordless book stories (i.e., A Boy, a Dog and a Frog and Frog Where Are You?). For example, we used “hornet” to replace a common word, “bee,” and “antelope” to replace “deer”. A total of eight words were included in the two storybooks for the evaluation of word learning. A word identification task was administered F2F before and after each story-retell task to assess whether the children learned the unfamiliar words.

Procedures: Videoconferencing (VC) and Face-to-Face (F2F) Conditions All participants were examined in the story-retelling task in two story presentation

conditions: (1) VC condition and (2) F2F condition. In order to control for the learning effect, the participants were counterbalanced to have either story 1 or story 2 for the VC condition and have the other story for the F2F condition. In order to control for the order effect, the order of the conditions (VC or F2F) was also counterbalanced. Half of the participants were examined with VC first and half with F2F first.

Prior to the story-retell task, children’s knowledge of the 8 target words was measured using a word identification task. For the traditional F2F condition, the examiner showed the participant the sequence of story illustrations while reading the script that corresponded to each page. Then, the participant was introduced to another examiner who served as a naïve listener (through the same condition as the story was told). The listeners were trained to facilitate the story-retelling using a narrow script of phrases that minimized the examiner’s influence on the participants’ language samples. The participant then retold the story to the naïve listener using the same book. The entirety of the story-retelling task was video-recorded to allow for language samples to be transcribed. Finally, the word identification task was administered after the story- retell task to measure children’s knowledge of the targeted unfamiliar words.

Similar procedures were done for the VC condition. The only difference was that the presentation and the elicitation phases were done on two computers using Adobe ConnectPro (mentioned above) in two separate rooms of the Speech, Language, and Hearing Sciences building at the University of Colorado, Boulder. The Adobe ConnectPro website enabled video-conferencing and facilitated the presentation of the PDF slideshows of the wordless storybooks. During the presentation phase, the examiner presented this electronic version of the story to the participant. During the elicitation phase, a naïve listener facilitated the story retelling through Adobe ConnectPro using the same narrow script of phrases. Then, the word identification task was implemented after the story-retell task. Inter-rater reliability for the language samples was 94%.

36

Downloaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

Analysis The language samples of the participants were transcribed and were analyzed using

Systematic Analysis of Language Transcripts (SALT). For the story retelling, we focused on microstructures of the narratives produced by the participants (Heilmann, Miller, Nockerts, & Dunaway, 2010). Specifically, we examined children’s syntactic complexity and lexical diversity using two measures: mean length of utterance (MLU) and number of different words (NDW). For the word learning performance, we examined the total number of target words that children answered correctly on the word identification task before and after the story-retell task.

Results

Table 1 summarizes participants’ performance in the F2F and VC conditions. A multivariate repeated measures analysis of variance (MANOVA) was used to examine the data. The dependent variables were MLU and NDW and the within-subject independent variable was presentation condition (i.e., F2F vs. VC conditions). There were no main effects of presentation condition for MLU, F (1, 4)=.24, p=.67, η2=.01, or for NDW, F (1, 4)=.04, p=.85, η2=.02. The findings indicate that there were no differences between children’s performance in F2F and VC conditions in terms of the microstructures of their narratives (MLU and NDW).

Table 1. Participants’ Performance Across the Two Learning Conditions: F2F and VC

Dependent Variable F2F VC

Number of Different Words 70.8 (20.1) 73.5 (14.1)

Mean Length of Utterances 5.86 (1.30) 6.17 (1.26)

Target Words Learned 1.17 (1.6) 1.00 (0.63)

Note. The values indicate the sample means and standard deviations across the two conditions. Analysis showed that children’s word learning performance on the word identification task was above chance, t(5) = 3.2, p < 0.05.

A repeated measures ANOVA was used to examine the effects of videoconferencing on children’s fast mapping skills. Results showed that, compared to the pretest scores, significant gain was found in the post-test, F (1, 4)=32, p<.01, η2=.89. Importantly, there were no significant differences in the number of words learned between the VC and F2F conditions, F (1, 5)=0.04, p=.84, η2=0.01. Due to the small sample size, power analysis has been done and found that more than 300 children are needed to reach a moderate effect. These findings indicate that children had similar word learning performances in the VC and F2F conditions.

Discussion

The purpose of the current study is to examine the effects of the videoconferencing presentation modality on typically developing children’s story-retell performance and their word learning skills. Results showed that there were no differences between conditions in the microstructures of the narratives (i.e., MLU and NDW) produced by the participants. In addition, children were able to learn new words in both conditions and there were no differences in their word learning performance between the VC and F2F conditions. These findings, along with the power analysis, showed that videoconferencing modality does not affect the typically developing children’s performance of two conditions that require children to pay close attention on the audio-visual input. These findings are consistent with previous telepractice and multimedia technology research on both children and adults (e.g., Georgeadis, Brennan, Barker, & Baron,

37

Downloaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

2004; Krcmar, Grela, & Lin, 2007; Troseth et al., 2006) and add to the evidence base supporting the use of VC in providing speech and language services.

Based on the established need for research on the use of telepractice in speech-language pathology, the current study has several contributions to the current research base. First, this study provides information about the use of VC to administer interactional language tasks with young children. The findings—the lack of VC-F2F effects on different language measures—are consistent with previous studies that examined consultation via VC (e.g., McCullough et al., 2001) and studies that tested adult populations via VC (e.g., Georgeadis et al., 2004). Second, apart from previous studies that focused on standardized tests (e.g., Waite et al., 2010), the current study examined common nonstandardized procedures—tasks that are appropriate for children from culturally and/or linguistically diverse backgrounds. Finally, the story-retell materials, along with the hardware and software, that were used in this study are readily available to SLPs. Clinicians can easily adapt materials for using these VC procedures.

Before recommending the use of VC to administer the SRT in clinical practice, several considerations should be noted. First, the personnel involved in the current study were all graduate and undergraduate students who have some knowledge about speech-language development and disorders. In practice, parent/caregiver training is critical for the success of implementing assessment via VC. Furthermore, all of the participants were typically developing children whose parents reported no speech, language, or hearing concerns. It is possible that the results may not apply to clinical populations with speech-language challenges (Krcmar et al., 2007; Schols et al., 2009). Although this study has provided some evidence to suggest that the application of telepractice is potentially feasible, it has also left some questions unanswered. Namely, is telepractice a comparable method of providing interactional services to young children with speech and language concerns/challenges? Further, what are the effects of lower verbal and spatial abilities and other developmental challenges on young children’s performance on interactional tasks administered via telepractice? Finally, what are the best indicators for determining young children’s candidacy for telepractice? Future studies are critically needed on the use of VC to implement interactional services to young children with language disorders.

Acknowledgements

This study is Brian Manzanares’ Master thesis. We would like to thank research assistants, Meryl Meisner, Jessica Morgan, and Lindsey Miller, for their help with stimuli development, data collection, and data analysis. We would also like to thank Arlene Stredler- Brown MA-SLP CCC and Eliana Colunga for their comments. I would like to extend a special thanks to all the participants and their parents for their willingness to contribute their time and energy to this project.

References Adobe ConnectPro [computer software]. (2012). San Jose, CA: Adobe Systems Incorporated.

American Speech-Language-Hearing Association. (2005a). Speech-language pathologists providing clinical services via telepractice: Technical report [Technical Report]. Available from www.asha.org/policy

American Speech-Language-Hearing Association. (2005b). Evidence-based practice in communication disorders [Position Statement]. Available from www.asha.org/policy

Antonacci, D. J., Bloch, R. M., Saeed, S. A., Yildirim, Y., & Talley, J. (2008). Empirical evidence on the use and effectiveness of telepsychiatry via videoconferencing: Implications for forensic and correctional psychiatry. Behavioral Sciences & the Law, 26, 253–269.

Brown, J., Brannon, J., & Romanow, K. (2010). Reimbursement for telespeech. Perspectives on Voice and Voice Disorders, 20, 16–21.

Burton, V. J., & Watkins, R. V. (2007). Measuring word learning: Dynamic versus static assessment of kindergarten vocabulary. Journal of Communication Disorders, 40, 335–356.

38

Downloaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

Gazella, J., & Stockman, I. J. (2003). Children’s story retelling under different modality and task conditions: implications for standardizing language sampling procedures. American Journal of Speech-Language Pathology, 12(1), 61–72.

Georgeadis, A., Brennan, D., Barker, L., & Baron, C. (2004). Telerehabilitation and its effect on story retelling by adults with neurogenic communication disorders. Aphasiology, 18(5–7), 639–652.

Gillam, R. B., & Carlile, R. M. (1997). Oral reading and story retelling of students with specific language impairment. Language, Speech, and Hearing Services in Schools, 28(1), 30–42.

Hayward, D. V., Gillam, R. B., & Lien, P. (2007). Retelling a script-based story: do children with and without language impairments focus on script and story elements? American Journal of Speech-Language Pathology, 16, 235–245.

Heilmann, J., Miller, J., Nockerts, A., & Dunaway, C. (2010). Properties of the narrative scoring scheme using narrative retells in young school-age children. American Journal of Speech-Language Pathology, 19, 154–166.

Hwa-Froelich, D. A., & Matsuo, H. (2005). Vietnamese children and language-based processing tasks. Language Speech and Hearing Services in Schools, 36, 230–243.

Juenger, J. M. (2009, September 22). Telepractice in the schools. Retrieved from http://www.asha.org/ publications/leader/2009/090922/090922f/

Kan, P. F., & Windsor, J. (2010). Word learning in children with primary language impairment: A meta- analysis. Journal of Speech Language and Hearing Research, 53, 739–756. doi: 10.1044/1092-4388(2009/ 08-0248)

Kapantzoglou, M., Restrepo, M. A., & Thompson, M. S. (2012). Dynamic assessment of word learning skills: Identifying language impairment in bilingual children. Language Speech and Hearing Services in Schools, 43(1), 81–96.

Krcmar, M., Grela, B., & Lin, K. (2007). Can toddlers learn vocabulary from television? An experimental approach. Media Psychology, 10(1), 41–63.

Mashima, P. A., & Doarn, C. R. (2008). Overview of telehealth activities in speech-language pathology. Telemedicine and e-Health, 14, 1101–1117. doi:10.1089/tmj.2008.0080

McCullough, A. (2001). Viability and effectiveness of teletherapy for pre-school children with special needs. International journal of language & communication disorders / Royal College of Speech & Language Therapists, 36 Suppl, 321–326.

Rose, D. A. D., Furner, S., Hall, A., Montgomery, K., Katsavras, E., & Clarke, P. (2000). Videoconferencing for speech and language therapy in schools. BT Technology Journal, 18(1), 101–104.

Schols, M. (2009). Potential pedagogical benefits and limitations of multimedia integrated desktop video conferencing technology for synchronous learning. International Journal of Advanced Corporate Learning, 2(4), 30–34.

Theodoros, D. G. (2008). Telerehabilitation for service delivery in speech-language pathology. Journal of Telemedicine and Telecare, 14, 221–224.

Troseth, G. L., Saylor, M. M., & Archer, A. H. (2006). Young children’s use of video as a source of socially relevant information. Child Development, 77, 786–799.

Waite, M. C., Theodoros, D. G., Russell, T. G., & Cahill, L. M. (2010). Internet-based telehealth assessment of language using the celf-4. Language, Speech, and Hearing Services in Schools, 41, 445–458. doi:10.1044/ 0161-1461(2009/08-0131)

Wong, A. M.-Y., Au, C. W.-S., & Stokes, S. F. (2004). Three measures of language production for cantonese- speaking school-age children in a story-retelling task. Journal of Speech, Language, and Hearing Research, 47, 1164–1178.

Appendix A

Videoconferencing software and services designed for telemedicine

Down

� American Well Online http://www.americanwell.com � Avaya http://www.avaya.com/usa/industry/healthcare/

39

loaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

Down

� CloudVisit Telemedicine: http://www.cloudvisittm.com � Teladoc connect http://www.teladocconnect.com � Vidyo http://www.vidyo.com/solutions/healthcare/ � Vsee http://vsee.com/telemedicine

Commercial videoconferencing software and services

� AccuConference: http://www.accuconference.com/web-conferencing/video- conferencing.html

� Adobe Systems: Adobe Connect http://www.adobe.com/products/adobeconnect.html � Airtime: https://www.airtime.com � Avaya http://www.avaya.com/usa/product/avaya-aura-conferencing � Bistri https://bistri.com � Blue Jeans Network http://bluejeans.com � BrightCom: http://www.brightcom.com � Cisco WebEx Meeting Center http://www.cisco.com/en/US/products/ps10409/ � Citrix GoTo Mmeeting http://www.citrix.com/products/gotomeeting/overview.html � Facetime http://www.apple.com/ios/facetime/ � Fuze Meeting Pro https://www.fuzebox.com � Google hangout http://www.google.com/+/learnmore/hangouts/ � Implix ClickMeeting http://www.implix.com/products/clickmeeting/ � InterCall http://www.intercall.com � IOCOM: http://www.iocom.com/features/ � IVèS: Live Video Plugin (for deaf and hard of hearing) http://www.ives.fr/index.php/

en/ives-solutions/ives-deaf-hard-of-hearing

� Librestream: http://www.librestream.com/products/onsight-expert.html � LifeSize: http://www.lifesize.com/en/products/video-conferencing-software/

softphone

� MegaMeeting http://www.megameeting.com � Microsoft: Office Live Meeting http://office.microsoft.com/en-us/live-meeting-help/

demo-microsoft-office-live-meeting-HA010240323.aspx

� Mirial: http://www.mirial.com � Nefsis: http://www.nefsis.com � ooVoo Pro http://www.oovoo.com/Business.aspx � PGi: GlobalMeet http://www.pgi.com/globalmeet/ � PGi: iMeet http://www.pgi.com/imeet/ � Polycom: http://www.polycom.com � Radvision: http://www.radvision.com

40

loaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions

Down

� Skype www.skype.com � TeamViewer: http://www.teamviewer.com/en/products/online-meeting.aspx � Techviable: https://techviable.com/it-services/videoconferencing/ � Tokbox: http://tokbox.com � TrueConf: http://trueconf.com � VeaMea: http://www.veamea.com � VideoMost: http://www.videomost.com/en/videoconferencing/ � vzRoom: http://www.vzroom.com/eng/video_conferencing.html

41

loaded from: https://pubs.asha.org Alliete Alfano on 12/26/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions