week 5

U.S. Department of Education

September 2019

Evaluation of Support for Using Student Data to Inform Teachers’

Instruction

EXECUTIVE SUMMARY

Philip Gleason Sarah Crissey

Greg Chojnacki Marykate Zukiewicz

Tim Silva Mathematica

Sarah Costelloe Abt Associates

Fran O’Reilly

Evidence-Based Education Research & Evaluation, LLC

Erica Johnson Project Officer

Institute of Education Sciences

NCEE 2019-4009 Evaluation of Support for Using Student Data to Inform Teachers’ Instruction i

EXECUTIVE SUMMARY

As part of their improvement efforts, schools have increasingly turned to the use of data to improve instruction. This is due in part to the increasing availability of student assessment data throughout the school year. The strategy of using assessment and other data to inform teachers’ instruction is often called data-driven instruction (DDI). Under DDI, teachers analyze student data to better understand students’ learning needs and to identify and improve instructional practices to address those needs.

Research on the effectiveness of DDI is limited. Overall, the research has found that DDI does not consistently change what teachers do in the classroom or improve student achievement. Different studies of DDI examined different kinds of interventions. The earliest studies examined interventions that largely focused on getting data to teachers in a usable form, with less emphasis on providing supports for teachers’ data use. This study contributes to a body of more recent research examining interventions that placed more emphasis on supporting data use. These interventions provided supports to school leaders or teachers on how to analyze data and select appropriate instructional strategies through training, coaching, or facilitated collaboration with others.

Specifically, this study examined a DDI intervention that provided substantial training and support to school leaders and teachers to help teachers use data effectively to improve their instruction and increase student achievement. The study examined the following questions:

1. How did support for data-driven instruction affect teachers’ use of data and instructional strategies?

2. How did support for data-driven instruction affect students’ achievement?

Box ES.1. How Was the Study Conducted?

Study Method: The study used a random assignment design. Schools were selected to participate in the study based on their interest in DDI and willingness to be in the study. Schools were randomly assigned to the treatment group or the control group. The treatment schools implemented DDI in grades 4 and 5 from December 2014 through June 2016. DDI effects were measured as differences in outcomes between treatment and control schools after 1.5 years.

Study participants: A total of 102 schools in 12 districts and 8 states participated in the study. The study districts were medium to large in size, located throughout the United States, economically disadvantaged, and demographically diverse.

Data Sources: Three types of data were used in the study: (1) information collected from data coaches in the treatment group on the frequency and nature of coaching activities through interviews and logs; (2) spring 2016 surveys of principals and teachers in treatment and control schools on the supports provided to teachers from coaches and school leaders, the frequency and nature of data analysis and use, and teachers’ instructional practices; and (3) student-level administrative data on student characteristics and achievement in the treatment and control groups.

Outcomes: The primary outcomes of interest are student achievement in math and English/language arts. Intermediate outcomes include: (1) use of data by teachers and school leaders to set and monitor student progress; and (2) teacher reports of adjustments to instructional practice to address student needs and improve achievement.

NCEE 2019-4009 Evaluation of Support for Using Student Data to Inform Teachers’ Instruction ii

The Professional Development Program to Support Data-Driven Instruction Intervention

The DDI intervention included two key supports, a half-time data coach for each school and consultants from a DDI provider. Consultants provided school leaders and data coaches with professional development and ongoing technical assistance. The professional development included a two-day introductory session and six subsequent one-day sessions, which mostly occurred during spring 2015 so that treatment schools would be ready to fully implement DDI during the 2015-16 school year. The ongoing technical assistance began in spring 2015 and continued throughout the 2015-16 school year. The intervention also encouraged school-level structures and activities to promote and support school leaders and teachers in increasing data use throughout the school year. As shown in Figure ES.1, the structures included a school leadership team as well as grade-level teacher teams in 4th and 5th grades.

Figure ES.1. Activities Expected of School Leadership Team and Grade-Level Teacher Teams Under Data-Driven Instruction

Key Findings

Most aspects of the intervention were implemented as planned. Hiring data coaches, providing them professional development on DDI, and holding regular data-focused meetings among school leadership and grade-level teacher teams were all key to implementing the intervention as intended. All treatment schools hired experienced educators as half-time data coaches, but only 36 percent of data coaches had previous coaching experience and it was rare for them to have previously worked as a data coach. Data coaches and school leaders largely participated in the expected professional development; depending on the session, participation ranged from 87 to 97 percent for data coaches and 75 to 89 percent for principals. Even so, about one-fifth (22 percent) reported that their training either did not prepare them or prepared them to carry out some but not all data coach tasks by the end of the study period.

Principals were expected to meet weekly with data coaches to strategize for DDI implementation, allocate time and resources to support DDI, and set expectations and provide guidance on data use by teachers. Most principals (59 percent) met with data coaches at least three times per month to monitor and support DDI, and most (at least 66 percent, depending on the type of guidance) provided overall guidance on DDI. Within this context, more teachers in treatment than control schools worked one-on-one with a data coach or school leader on data- related activities; for example, 69 percent of treatment teachers compared with 56 percent of

NCEE 2019-4009 Evaluation of Support for Using Student Data to Inform Teachers’ Instruction iii

control teachers reported at least monthly one-on-one interactions on analyzing or interpreting student data. Most treatment schools also established grade-level teams of teachers that met regularly with the data coach (77 percent of treatment teachers reported at least monthly guidance on using and analyzing data compared with 61 percent of control teachers). More teachers in treatment than control schools reported receiving coaching on data-related activities in collaboration with other teachers during common planning periods; for example, 52 percent of treatment teachers compared to 34 percent of control teachers reported at least monthly training or coaching on how to analyze and interpret student data.

Despite additional resources and emphasis placed on data-related activities, the support for data-driven instruction intervention did not increase key data-related activities. Teachers were expected to work together in teams to examine data in order to understand individual student needs at least twice per month. However, similar percentages of teachers in treatment and control schools reported data-related activities during common planning time; for example, 79 percent of treatment teachers and 73 percent of control teachers reported jointly analyzing data to understand student needs at least monthly. These activities were fairly common among teachers in control schools even without the additional resources provided as part of the DDI intervention.

The support for data-driven instruction intervention did not increase teachers’ data use or change their instructional practices. The intervention was intended to increase data analysis and collaboration with other teachers, which would then lead teachers to change their instructional practices in order to appropriately address student needs. Because the intervention did not increase data-related activities, it is consistent that similar percentages of teachers in treatment and control schools reported using each of nine data practices. For example, 38 percent of treatment teachers and 35 percent of control teachers reported monitoring student progress, and 43 percent of treatment teachers and 44 percent of control teachers reported planning individualized instruction, daily or several times per week, in math. There were also no treatment-control differences in English/language arts. Nor did the intervention lead teachers to report more frequent use of any of five instructional practices potentially associated with DDI that were examined (figure ES.2).

Figure ES.2. Percentage of 4th and 5th grade teachers who used instructional practices daily or several times per week during 2015-2016

Source: Teacher survey (n = 397-411). ELA = English/language arts. *Difference is statistically significant at the .05 level, two-tailed test.

NCEE 2019-4009 Evaluation of Support for Using Student Data to Inform Teachers’ Instruction iv

The support for data-driven instruction intervention did not affect students’ achievement. On average, 4th and 5th grade students in treatment and control schools had similar achievement in math and English/language arts (figure ES.3). Students in each group scored at about the 40th percentile on state assessments in each subject, on average. The study also found that support for DDI did not improve achievement for any subsets of students examined, such as students with different prior achievement levels. Nor did the study find that DDI consistently improved achievement for any of the subsets of schools examined, such as schools with greater readiness to implement DDI.

Figure ES.3. Mean student achievement on 2016 state assessments in math and English/language arts

Source: District student records (n = 12,018-12,036).

Neither difference is statistically significant at the .05 level, two-tailed test.

Concluding Thoughts

This study is part of a growing body of research on how DDI affects teacher practices and student achievement. While early studies focused on the effects of giving teachers greater access to data, this study contributes to more recent evidence focused on the effects of giving teachers more support in their efforts to use data to improve instruction. The DDI intervention being examined in this study was designed to give teachers support from a half-time data coach, actively engaged school leaders, and fellow teachers in a group setting.

The treatment schools implemented most aspects of DDI as planned, and teachers in these schools reported receiving more support from coaches and school leaders than those in control schools. But support for DDI did not increase teachers’ data use or change how often they used a set of instructional practices related to DDI. Most importantly, support for DDI did not affect student achievement. These findings are similar to those of other recent studies. As a strategy to improve student achievement, DDI relies on: (1) using a data-focused approach to identify areas for instructional focus, (2) finding appropriate strategies to improve instruction, and (3) implementing teacher practices that improve student performance in the focused area. This study’s findings in conjunction with other recent studies suggest that simply giving teachers more support in their data use is not sufficient, on its own, to improve student achievement. Future research might turn to other possible ways of improving DDI interventions, such as improving the quality of data-use support provided to teachers or changing the nature of this support by placing greater emphasis on how teachers use data to identify and implement effective instructional practices.

U.S. Department of Education Betsy DeVos Secretary

Institute of Education Sciences Mark Schneider Director

National Center for Education Evaluation and Regional Assistance Matthew Soldner Commissioner

September 2019

The report was prepared for the Institute of Education Sciences under Contract No. ED-IES-12- C-0086. The project officer is Erica Johnson in the National Center for Education Evaluation and Regional Assistance.

IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the reports.

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should read:

Gleason, Philip, Sarah Crissey, Greg Chojnacki, Marykate Zukiewicz, Tim Silva, Sarah Costelloe, and Fran O’Reilly. (2019). Evaluation of Support for Using Student Data to Inform Teachers’ Instruction (NCEE 2019-4009). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

This report is available on the IES website at http://ies.ed.gov/ncee.

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113.