Systematic Review

Table 3. Quality Assessment Tool and Scoring Guidance Notes

Criteria 0 = Not at all 1 = Very slightly 2 = Moderately 3 = Complete

Explicit theoretical

framework

No mention at all. Reference to broad theoretical

basis.

Reference to a specific

theoretical basis.

Explicit statement of theoretical

framework and/or constructs

applied to the research.

Statement of aims/objectives

in main body of report

No mention at all. General reference to

aim/objective at some point in

the report including abstract.

Reference to broad

aims/objectives in the main body

of report.

Explicit statement of

aims/objectives in the main body of

report.

Clear description of research

setting

No mention at all. General description of research

area and background, e.g. ‘in

primary care’.

General description of research

problems in the target

population, e.g. ‘among GPs in

primary care’.

Specific description of the research

problem and target population in

the context of the study, e.g. nurses

and doctors from GP practices in

the east midlands.

Evidence of sample size

considered in terms of

analysis

No mention at all. Basic explanation for choice of

sample size. Evidence that size

of the sample has been

considered in study

design.

Evidence of consideration of

sample size in terms of

saturation/information

redundancy or to fit generic

analytical requirements.

Explicit statement of data being

gathered until information

redundancy/saturation was reached

or to fit exact calculations for

analytical requirements.

Representative sample of

target group of a reasonable

size

No statement of

target group.

Sample is limited but represents

some of the target group or

representative but very small.

Sample is somewhat diverse but

not entirely representative, e.g.

inclusive of all age groups,

experience but only one

workplace. Requires discussion

of target population to determine

what sample is required to be

representative.

Sample includes individuals to

represent a cross section of the

target population, considering

factors such as experience, age and

workplace.

Description of procedure for

data collection

No mention at all. Very basic and brief outline of

data collection procedure,

States each stage of data

collection procedure but with

Detailed description of each stage

of the data collection procedure,

e.g. ‘using a questionnaire

distributed to staff’.

limited detail, or states some

stages in details but omits others.

including when, where and how

data were

gathered.

Rationale for choice of data

collection tool(s)

No mention at all. Very limited explanation for

choice of data collection tool(s).

Basic explanation of rationale

for choice of data collection

tool(s), e.g. based on use in a

prior similar study.

Detailed explanation of rationale for

choice of data collection tool(s),

e.g. relevance to the study aims and

assessments of tool quality either

statistically, e.g. for reliability &

validity, or relevant qualitative

assessment.

Detailed recruitment data No mention at all. Minimal recruitment data, e.g.

no. of questionnaire sent and no.

returned.

Some recruitment information

but not complete account of the

recruitment process, e.g.

recruitment figures but no

information on strategy used.

Complete data regarding no.

approached, no. recruited, attrition

data where relevant, method of

recruitment.

Statistical assessment of

reliability and validity of

measurement

tool(s)

No mention at all. Reliability and validity of

measurement tool(s) discussed,

but not statistically assessed.

Some attempt to assess

reliability and validity of

measurement tool(s) but

insufficient, e.g. attempt to

establish test–retest reliability is

unsuccessful but no action is

taken.

Suitable and thorough statistical

assessment of reliability and

validity of measurement tool(s)

with reference to the quality of

evidence as a result of the measures

used.

Fit between stated research

question and method of data

collection

No research

question stated.

Method of data collection can

only address some aspects

of the research question.

Method of data collection can

address the research question but

there is a more suitable

alternative that

could have been used or used in

addition.

Method of data collection selected

is the most suitable

approach to attempt to answer the

research question.

Fit between research

question and method of

analysis

No research

question stated.

Method of analysis can only

address the research question

basically or broadly.

Method of analysis can address

the research question but there is

a more suitable alternative that

could have been used or used in

addition to offer greater detail.

Method of analysis selected is the

most suitable approach to attempt to

answer the research question in

detail, e.g. for qualitative IPA

preferable for experiences vs.

content analysis to elicit frequency

of occurrence of events, etc.

Good justification for

analytical method selected

No mention at all. Basic explanation for choice of

analytical method.

Fairly detailed explanation of

choice of analytical method.

Detailed explanation for choice of

analytical method based on nature

of research question(s).

Evidence of user

involvement in design

No mention at all. Use of pilot study but no

involvement in planning stages

of study design.

Pilot study with feedback from

users informing changes to the

design.

Explicit consultation with steering

group or statement or formal

consultation with users in planning

of study design.

Strengths and limitations

critically discussed

No mention at all. Very limited mention of

strengths and limitations with

omissions of many key issues.

Discussion of some of the key

strengths and

weaknesses of the study but not

complete.

Discussion of strengths and

limitations of all aspects of study

including design, measures,

procedure, sample & analysis.

Table 4. Characteristics of included studies

Authors, Year, & Country Study Aim

Study Design

Sample &

Setting

Intervention

Findings/ Outcomes

Article 1

Alt, M. and Humphrey, M. (2012). US

To determine if there is alternate form’s reliability for paper- and computer- administered standardized vocabulary tests and to determine whether the behavioral ratings of children with autism spectrum disorders (ASDs) would improve during the computer- administered testing sessions secondary to a decreased need for social interaction.

Non-

randomized

18 children with a diagnosis of ASD (5 female/ 13 male) and 18 NT children as a control group. Children were between 5 and 13 years of age with English as their first language. Setting not discussed.

Two versions (i.e., paper vs. computer) of the Expressive One-Word Picture Vocabulary Test (EOWPVT–2000; Brownell, 2000a) and the Receptive One-Word Picture Vocabulary Test (ROWPVT–2000; Brownell, 2000b). Also used a behavioral rating instrument created for this study that was used to record observable negative behaviors and to record overall impressions of the testing session.

No significant difference between Standard scores for both versions of the tests for both groups of participants. No significant difference in behavioral ratings between the two methods of test presentation.

Article 2 Article 3

Sutherland et al. (2017). Australia Sutherland et al. (2019). Australia

To determine whether, within an existing service, a web-based telehealth application using consumer grade, commercially available computer equipment could be used to provide a formal language assessment that is feasible, reliable, and well-tolerated by participants and their families. To investigate the reliability of language assessments for children on the autism spectrum,

Method comparison

Method Compariso

n

23 children (5 female/ 18 male) aged 8-12 years with a history of reading difficulties and known/ suspected language impairment. All participants were attending mainstream schools. Participants had not had a language assessment using the CELF-4 within the previous six months. The assessments were conducted from the metropolitan telehealth site in Westmead, NSW, to the three hub sites in rural NSW (Hub 1 and Hub 3) and suburban Sydney, NSW (Hub 2). 13 children (3 female/ 10 male) between 9;5 to 12;3 years of age with autism and attend mainstream schools or

Two versions (i.e., paper vs. computer) of the CELF-4. Children in the 5–8-year-old range were administered the Concepts and Following Directions, Word Structure, Recalling Sentences and Formulated Sentences subtests. Children in the 9– 12-year-old range were administered the same subtests with the addition of word classes.

Subtests (Concepts and Following Directions, Recalling Sentences, Formulated Sentences and Word Classes) of the

Determined system for telehealth delivery was feasible and presented adequate reliability with high levels of agreement between telehealth and face-to- face delivery. Parent and child reactions to the use of telehealth were largely positive and supportive of using telehealth to assess rural children. High agreement between assessment scores obtained via telehealth and face-to- face SLPs suggested

Article 4

Waite et al. (2010). Australia

delivered via telehealth and to explore the feasibility of the use of telehealth with children with autism, exploring their behavioral responses to tele-health and face-to-face assessment conditions.

To examine the validity and reliability of an Internet-based telehealth system for assessing childhood language disorders on the core components

Non- randomized

support classes. Face-to- face administration took place in a remote location in Westmead, NSW and telehealth assessment was delivered remotely from Melbourne, VIC.

25 children (8 female/ 17 male) between 5 to 9 years of age with a previous diagnosis of language impairment or identified as having difficulties in language by a parent or teacher but has not been formally assessed. Primary

Clinical Evaluation of Language Fundamentals 4th Edition, Australia & New Zealand (CELF-4; Semel et al. 2003) were used in both the telehealth and face-to- face assessment conditions. Also used a behavior observation rating scale that was adapted from the Clinical Evaluation of Language Fundamentals— Preschool, 2nd edition, Australian and New Zealand (CELF-P2; Wiig et al. 2006) behavior checklist.

Clinical Evaluation of Language Fundamentals- Fourth Edition (CELF–4), Australian adaptation (Semel, Wiig, & Secord, 2003). The four- core language subtests for children ages 5 to 8 years (Concepts and Following Directions, Word Structure, Recalling Sentences, and

that scores obtained in each setting were similar. Regarding behavioral measures, this study suggests that the responses of children with ASD to telehealth are likely to be highly individual. However, there was no clear difference between the conditions at the group level.

No significant difference was found between the online and face-to-face total raw and scaled scores in each subtest.

Article 5

Hodge et al. (2019). Australia

of a standardized assessment tool.

To determine the feasibility and reliability of telepractice assessments, using consumer-grade technology, in children with reading difficulties.

Non-

randomized

language of participants was English. Assessments were conducted between two rooms within the same building at the University of Queensland. 37 children between 8 to 12 years of age with a diagnosis of Specific Learning Disorder with impairment in reading. Children were located remotely (in Dubbo, Wagga Wagga, Westmead, or Manly) and accompanied by local staff (face-to-face teacher).

Formulated Sentences) were administered. Assessments delivered via a web-based application by a remotely located research assistant. Subtests from the Woodcock Reading Mastery Tests-Third Edition (WRMT-III), including Word Identification, Word Attack, and Passage Comprehension. The Phonemic Decoding Efficiency subtest from the Test of Word Reading Efficiency-Second Edition (TOWRE-2). The MultiLit Sight Words Test and the Dalwood Spelling Test. Also had parents complete a brief survey indicating the perceived comfort level of their child during the

Found strong agreement between telepractice and face-to-face rated scores. Parents reported high degree of comfort with the telepractice assessments.

Article 6

Manzanares, B. and Pui, F. K. (2014). US Raman et al. (2019). India

To explore the effects of using videoconferencing to assess children's language skills. To (1) compare receptive and expressive scores

Non-

randomized

Non-

randomized

6 typically developing children (2 female/ 4 male; mean age = 4;0) whose primary language was English and came from middle class families. Both face-to- face and videoconferencing conditions were conducted in dedicated rooms of the Speech, Language, and Hearing Sciences building at the University of Colorado, Boulder.

32 children (mean age = 6;3) in first grade were selected from a primary school in Tirupur town. 15 children were identified by teachers as

assessment as well as their own comfort level. Story-retell task in both videoconferencing (VC) and face-to-face (F2F) conditions using a story, along with a wordless book. In addition, 4 unfamiliar words were embedded within the story. The Assessment of Language Development (ALD; Lakkanna, Venkatesh, & Bhat, 2008) using digitized picture stimuli presented through videoconferencing and

No significant difference in narratives between videoconferencing and face-to-face conditions. Also found that children learned target words in both conditions equally well. No significant difference in receptive and expressive domains between videoconferencing and face-to-face conditions.

Article 7

obtained on a language screening tool through in- person testing and telemethod among children and (2) to assess technology- and child-related factors influencing screening via telemethod.

having concerns in one or more areas of hearing, speech, language, or academic performance. The remaining 17 children had no specific concerns and were randomly selected from the same classrooms. The in-person and telemethod language screenings were carried out in a single dedicated space allocated within the school premises.

remote computing with assistance of a facilitator at school site. Technology and child related factors influencing screening were documented using an inventory.

Found that using multiple internet options at both sites helped overcome technical challenges related to connectivity during screening through telemethod. Additionally, trained facilitators were essential in overcoming child-related factors (e.g., poor speech intelligibility, poor audibility of voice, motivation, interaction with SLP, and need for frequent breaks).