Catherine Owens only

sytematic-ASD.pdf

Home >Education homework help >Catherine Owens only

Undertaking a Systematic Literature Review of Educational

Interventions for Children and Young People on the Autism

Spectrum

Contributors: Caroline Bond & Wendy Symes Pub. Date: 2018 Access Date: March 30, 2019 Academic Level: Postgraduate Publishing Company: SAGE Publications Ltd City: London Online ISBN: 9781526427885 DOI: https://dx.doi.org/10.4135/9781526427885

javascript:void(0);

https://dx.doi.org/10.4135/9781526427885

Abstract This case study focuses on the process of undertaking a commissioned systematic literature review of educational interventions for children and young people on the autism spectrum. The research was commissioned by the National Council for Special Education, Ireland, to contribute to its review of educational provision for persons on the autism spectrum. A rigorous systematic literature search process identified 1,021 possible articles for inclusion in the review, and following the application of inclusion/exclusion criteria and appraisal of methodological quality, methodological appropriateness, and outcome, 85 best-evidence studies were included in the final review. The review highlighted that research to date has focused on pre-school children and those aged 5 to 8 years, and the majority of interventions seek to address core difficulties relating to autism—social interaction, communication, and challenging behavior. Limitations particularly relate to a lack of interventions for older young people and the application of interventions in school settings. The case study will describe the commissioning process and how the review framework was developed, refined, and applied. The review’s distinctive contribution in evaluating educational utility will be highlighted and the challenges of applying the framework and reporting the findings will be discussed.

Learning Outcomes

By the end of this case, students should be able to

• Develop a framework for undertaking a systematic review • Understand why different tools are used for evaluating research • Select appropriate tools for evaluating different types of research studies • Tailor a systematic review to address specific research question(s)

Project Overview and Context

This systematic review was commissioned by the National Council for Special Education (NCSE), Ireland, to inform government policy in relation to educational provision for children and young people on the autism spectrum in Ireland. Our systematic review forms one strand of the evidence which also included the following:

• An international review of policy documents and country case studies of educational provision for children with autism, which was undertaken as part of our project;

• A separate research study evaluating education provision for students with autism spectrum disorder (ASD) in Ireland.

The rate of autism spectrum diagnosis has significantly increased since the 1990s, and there has been a

SAGE Research Methods Cases Part 2

Page 2 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

corresponding increase in research into interventions for children and young people on the autism spectrum. A review of this evidence was important to assist the Irish government in recommending evidence-based interventions to education settings and ensuring efficient, equitable and effective use of limited financial resources by education settings.

Research Practicalities

As this was a funded research project, we were able to convene a research team to undertake the work. The team was led by the principal investigator who is an academic with knowledge of autism, education, and systematic reviews. A second academic provided consultation particularly in relation to autism and the systematic review. A lead research assistant who also has experience of systematic reviews and autism oversaw the day-to-day running of the project and contributed to the write-up of the research. Two other research assistants and an academic undertook specific tasks, such as screening studies for inclusion and evaluating selected studies.

Our review was commissioned to update a previous review of educational interventions for children and young people on the autism spectrum which was also undertaken for NCSE (Parsons et al., 2009). This meant that our review of the evidence was limited to 2008-2013. The research question for the review was, “What does the international research evidence tell us about what works best in the provision of education for persons with ASD?”

Throughout the planning, delivery, and dissemination of the review, the research team worked closely with a stakeholder group appointed by NCSE. This group included academics, policy-makers, and practitioners. Lead members of the research team met with the stakeholder group on several occasions. This was helpful for understanding the wider context of the research, for example, why this systematic review was needed and for agreeing the parameters of the research. The final report reflected ongoing discussion with stakeholders through the inclusion of a chapter on local context and need. Meetings with stakeholders also directly informed the review process, for instance, by agreeing the search terms for identifying studies and screening criteria. Wider consultation also took place with other people interested in the review, such as academics in Irish universities. As part of a rigorous NCSE review process, the final report was evaluated by two independent experts in the field prior to publication. These processes enabled the review team to understand the local context and ensured the systematic review was fit for purpose.

The review team comprised the lead researcher and lead research assistant and five other team members. The design of the research was developed by the lead researcher, lead research assistant, two academics, and an inclusion manager in a secondary school. One academic assessed studies for inclusion in the review, and the lead research assistant with two other research assistants assessed studies according to the review framework with regular checking between coders and with the lead researcher. The lead researcher and lead research assistant compiled the research report. Subgroups from the team met regularly to discuss aspects such as design of the project and coding of studies. Differences of opinion such as how to interpret evaluation

SAGE Research Methods Cases Part 2

Page 3 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

criteria were addressed through team meeting discussions and/or consultation between the lead research assistant and the principal investigator. A concern highlighted by the commissioners was addressed by re- evaluating the data and undertaking some further data collection as a result.

Research Design

The research design agreed with the stakeholder group was a systematic review rather than a meta-analysis. This meant that we were reviewing the studies using their own data and we compared effect sizes between studies if these were available from the authors. However, given the variety of research methods used in the field, it was not possible to undertake further analysis to standardize the data and compare effect sizes across all studies, as would be done in a meta-analysis.

There are standard frameworks and methods available for conducting a systematic review. We agreed with stakeholders that our review process would build upon Parsons et al. (2009) by including an increased number of databases and expanding their inclusion and exclusion criteria. We also strengthened the research process by adopting the PRISMA framework (Moher, Literati, Tetzlaff, Altman, & the PRISMA Group, 2009) which is a widely used process for demonstrating how studies have been identified, screened, and assessed. In addition to using the PRISMA framework, we also strengthened our review by evaluating included research studies in relation to their Weight of Evidence in three key areas: methodological quality, methodological appropriateness, and outcome (Gough, 2007). This involved using frameworks developed in previous research to assess the following:

• The methodological quality of (a) quantitative and (b) qualitative studies (for a full description of these criteria, see Bond, Woods, Humphrey, Symes, & Green, 2013).

In consultation with stakeholders, we decided not to use a separate framework for assessing single-case experimental design studies (Reichow, Volkmar, & Cicchetti, 2008) and instead merged aspects from the Reichow et al. (2008) framework within the quantitative evaluation criteria described by Bond et al. (2013). This enabled us to use just one framework to screen all quantitative studies.

We also developed criteria to evaluate the following:

• The methodological appropriateness of included research in relation to the research question posed for the review—particularly educational utility;

• The overall effectiveness of the research—This related to the overall success of the intervention evaluated and the relevance of the outcomes identified in relation to the questions posed for the review.

As evaluations of methodological appropriateness and effectiveness were review specific, we developed our own criteria for these (see Bond, Symes, Hebron, Humphrey, & Morewood, 2016a). As effectiveness in an educational context was a key part of the overall research question, we agreed with the stakeholders

SAGE Research Methods Cases Part 2

Page 4 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

that assessing studies for their educational utility would be an important part of assessing methodological appropriateness. This is particularly important in the field of autism spectrum research as many interventions have been developed in research rather than educational settings without any evidence of how they might work in a real-life education context. Evaluating educational utility has not been attempted previously in reviews of autism interventions, and we agreed with stakeholders that this would be an important contribution for linking research to policy and practice in schools. Assessing educational utility was one of the most challenging aspects of the review as this needed to be clearly defined and operationalized for our team of three reviewers. We agreed that educational utility would be evident if there was demonstration of either utility or effectiveness in an educational setting:

a. Utility was demonstrated through either data collection from staff within the child’s education setting about the utility of the intervention (which might include social validity measures) or clear involvement of school staff or peers in the delivery of the intervention.

b. Effectiveness in the educational context was demonstrated through at least one outcome measure focusing on the child and undertaken in the primary education setting, for example, observations, questionnaires, or interviews with staff and/or peers undertaken in the young person’s school.

In addition to assessing whether effectiveness was measured appropriately, a separate assessment for evaluating the overall effectiveness of the research was also developed. Based on the researcher’s own reports, studies were identified as being of low effectiveness if they had no or a negative effect and medium if they had a positive effect but no control intervention was used or if the intervention was predicted to perform better than the control but both scored equally. Studies scored high if the intervention scored better than control or comparison if this was predicted.

To be included in the final review, studies needed to score at least medium in relation to all three criteria (methodological quality, methodological appropriateness, and effectiveness of the research). Following on from a previous review of autism interventions (Wong et al., 2013) the final 85 studies included in the review were presented in groups according to the focus of intervention, for instance, all joint attention interventions were presented together. Finally, the amount of evidence provided for each intervention included in the review was also assessed and the evidence for each intervention was categorized as most, moderate, some, or a small amount from this review. For instance, to achieve a rating of most evidence, an intervention needed to have at least four studies providing positive evidence, which included a positive randomized controlled study (RCT), or quasi-experimental study, or six or more single-case experimental studies within the 2008-2013 time frame.

Method in Action

The process of conducting the systematic review involved three main stages, namely, (a) identifying potentially relevant articles to include in the review, (b) screening those articles in accordance with inclusion/

SAGE Research Methods Cases Part 2

Page 5 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

exclusion criteria, and (c) coding the included articles to identify the studies to be included in the final review. This section will outline the steps that were taken at each of these stages and the challenges that arose throughout the review process.

Identification

One researcher was responsible for identifying studies that could potentially be included in the review. To do so, 15 social sciences databases were searched (including PsycINFO and Australian Education Index) and web searches were undertaken (including Google Scholar and NCSE’s research database) using search terms that had been agreed in collaboration with the research steering committee. The terms were divided into seven categories:

• Terms relating to ASD; • Terms relating to the population under study; • Terms relating to outcomes; • Terms relating to educational provision; • Terms relating to age/stage of schooling; • Terms relating to type of study; • Terms relating to assessment.

A list of words was generated for each of the categories. For example, terms relating to ASD included autistic spectrum disorder, autistic spectrum condition, autism, Asperger syndrome, high functioning autism, pervasive developmental disorders, pervasive developmental disorder not otherwise specified, Rett’s, Rett’s syndrome, childhood disintegrative syndrome, and Kanner’s autism. These words were then entered into the databases and a record was kept of the hits returned from each search. Hits were limited to published, peer-reviewed studies only. To ensure that the search terms were returning relevant articles, the researcher checked that a number of studies they were already aware of (that were suitable for inclusion in the review) were being returned. Although the search terms did seem to adequately identify relevant studies, the large number of search terms and search terms categories meant that the process of searching the databases was time-consuming, and huge numbers of studies were returned. In total, 6,232 potential studies were identified during the search phase. Some additional articles were found via reference harvesting and through handsearching relevant journals. Once duplicates had been removed, only 1,141 unique hits remained. The inclusion/exclusion criteria were then applied to 1,021 of these studies (120 studies could not be retrieved).

Screening

Three researchers were involved in screening the returned articles to see whether they should be included in the review. To be included in the review, the study had to satisfy the following criteria:

• Include more than one child or young person with ASD;

SAGE Research Methods Cases Part 2

Page 6 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

• Include a researcher-manipulated intervention; • Have educational utility; • The intervention had to take place in the home, community, clinic, or school-based setting; • Include the collection of quantitative or qualitative data; • Report on at least one outcome measure about the children or young people with ASD; • Be written in English; • Be published after 2007.

To apply these inclusion criteria, each article was read until it was clear that it either fulfilled the inclusion criteria or not. If it fulfilled the inclusion criteria, it was retained for further coding in Stage 3. If it did not fulfill the inclusion criteria, it was excluded from the review, and the reason for its exclusion was recorded. As soon as a study failed to meet one of the inclusion criteria, it was not read any further, so it was not clear how many of the inclusion criteria an excluded study did not meet. To ensure that inclusion criteria were being applied reliably, the three researchers met at the start of the process to discuss how to apply it. Fifty studies were jointly coded to pilot the inclusion criteria. As a result of this process, the inclusion criteria relating to educational utility were refined (as discussed in the “Research Design” section, above). The new inclusion criteria were then applied to 50 more studies until 100% agreement on how the inclusion criteria should be applied was reached. Then, the remaining studies were screened by the three research assistants. At the end of the screening process, the lead researcher checked 10% of both the included and excluded studies and agreed with the original screener’s assessment 94% of the time. Thus, it appeared that the inclusion criteria had been applied consistently by the three research assistants. Although reliability could have been improved by having at least two researchers screen each study, this was not possible within the resources and time available. At the end of the screening stage, 176 studies were retained for inclusion in the review.

Coding

In the third stage of the review, relevant information from each of the studies was identified to help address the questions guiding the review. Relevant information included descriptive information about the study, such as the number and age of participants, the type of intervention used, and the reported findings. Information was also coded to allow judgments to be made about the quality of the evidence presented, the methodological appropriateness of the evidence to the review aims, and the effectiveness of the intervention being evaluated. For further information on how these aspects were judged, please see the “Research Design” section. To ensure that information was identified in a systematic and reliable way, a coding framework was devised. Categories were included in the framework based on discussions between the project steering group and research team and to ensure that information relevant to the review was captured. As a result, the coding framework was trialled four times, with amendments made after each trial, until the researchers and steering group were happy that the framework was suitable. Each of the 176 included articles was then coded using the fifth version of the framework. The same three research assistants who had selected which studies to include in the review coded the studies. Initially, three studies were coded by each researcher independently and then compared to ensure that reviewers were coding information in the same way. Once agreement

SAGE Research Methods Cases Part 2

Page 7 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

was reached, the researchers coded the rest of the studies, with only one researcher coding each study. Any uncertainties throughout this process were raised, and the research assistants discussed how to best deal with them. The lead research assistant checked a sample of coded studies each week to ensure that information was being coded accurately, and any discrepancies were discussed. The major challenge at this stage was ensuring enough information could be extrapolated from a study to allow judgments regarding its quality and appropriateness to be made. The extent to which studies provided information about the intervention that was delivered, for example, varied widely from study to study. Thus, it is possible that studies that were of high quality were given lower ratings, purely because sufficient information was not provided. At the end of this stage, only the 85 studies that were coded as having medium or high quality and appropriateness were included in the final review.

Practical Lessons Learned

A number of important practical lessons were learned from this review process:

• The need for rigor and accountability means that a review process is likely to take significantly more time than anticipated. This review highlighted the benefits of working as a team in being able to work through the process quickly. However, further discussion and joint coding of studies early on would have ensured greater consistency among the coding team from the outset and helped to reduce the need to rescreen some papers later on.

• Scoping the field early on may also be useful. It may be, as in this case, that significantly more records are identified during initial searches, which means the process will take longer. Building in contingency time is a helpful strategy.

• It is important to have a strong rationale for your time frame. Although the limited time frame for the current review helped to limit the number of included studies, the lack of comparability with Parsons et al. (2009) meant that conclusions were limited to 2008-2013. It was therefore only possible to comment on trends between 2008 and 2013 rather than on the overall strength of evidence for each intervention over time (for instance, some interventions such as self-management may not have been researched during this time frame but might have evidence over a longer period of time).

• It is important to have one or a small number of clear and specific systematic literature review (SLR) questions which can be operationalized. This will provide focus and help to ensure that the number of included studies is manageable.

• It is helpful to have stakeholder input during the process of operationalizing a systematic review as this can bring additional expertise to the process. However, this also needs to be managed so that expectations are realistic, for example, in the current review, a large number of search terms relevant to the research question were generated by the stakeholder group which needed further revision in consultation with NCSE to reduce duplication of similar terms.

• It is helpful to have some familiarity with the field beforehand. Although we had some knowledge of the field, we were surprised that there were not any qualitative studies and only one mixed-

SAGE Research Methods Cases Part 2

Page 8 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum

methods study included in our final 85 studies. Given that the majority of studies included in the review were single-case experimental designs, a more fine-grained assessment might have been achieved if a single-case experimental evaluation framework such as Reichow et al. (2008) had been incorporated.

• SLRs are an evolving field and new frameworks are regularly being developed. Although the assessment of educational utility represents a unique contribution to the field, its development did present some challenges. Due to the review timescales, these criteria were developed while undertaking the review, but in an ideal scenario it would be most efficient to trial and operationalize these prior to undertaking the review.

• Given that undertaking an SLR involves making many decisions and operationalizing criteria, inter- rater reliability checking is an important part of the process. If you do not have access to a research team, having peers or supervisors who can code a sample of studies is crucial for ensuring a rigorous review process.

Conclusion

This account of undertaking a systematic review has illustrated some of the challenges of undertaking what might initially appear to be a relatively straightforward task of synthesizing literature. This example has illustrated the need for feasible research questions, clear operationalization of inclusion criteria, transparent review frameworks, and rigor and consistency. When working as a team, regular communication is essential for shared understanding of review criteria and processes, and when working alone, access to colleagues who can support the development of new processes and participate in inter-rater checking is essential. Whether you are undertaking a review as part of a team or as a lone researcher, it is crucial to allow sufficient time for familiarizing yourself with the field, undertaking, documenting, and reporting the review process. Although undertaking an SLR can be challenging, it is a good opportunity to reflect on and strengthen your own skills in effective research design, evaluation, and reporting.

Exercises and Discussion Questions

1. In the context of this case study, what do you think the benefits of a systematic review were over a meta-analysis?

2. The researchers rated studies on the basis of their methodological quality, appropriateness, effectiveness, and relevance of their findings to their review questions. Do you think any other factors could have been taken into account when determining which studies constituted the best evidence?

3. How did the systematic literature review described in this case study add to, or improve on, earlier reviews of educational interventions for children and young people on the autism spectrum?

SAGE Research Methods Cases Part 2

Page 9 of 10 Undertaking a Systematic Literature Review of Educational Interventions for Children and Young People on the Autism Spectrum