assessment article reflection

764097TCXXXX10.1177/0040059918764097Council for Exceptional ChildrenTEACHING Exceptional Children research-article2018

Strategies for Minimizing Variability in

Progress Monitoring of Oral Reading Fluency

Kaitlin Bundock, Breda V. O’Keeffe, Kristen Stokes, and Kristin Kladis

Progress Monitoring T

E A

C H

IN G

E xc

ep ti

on a l C h il d re

n ,

V o l. 5

0, N

o .

5, p

p .

27 3 –

28 1.

C o p yr

ig h t

20 18

T h e

A u th

o r(

s) . D

O I:

1 0.

11 77

/0 04

00 59

91 87

64 09

7

274 CounCil for ExCEptional ChildrEn

Mr. Long is a special education teacher in an urban school district. Three times per year, he uses Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Next to assess his students’ oral reading fluency (ORF) skills at their chronological grade level. Mr. Long conducts weekly progress monitoring of all students who score below the expected benchmark score for words read correctly per minute (WCPM). Students are assessed at either their grade level, if they are reading at or above 50 WCPM according to the DIBELS Next progress- monitoring guidelines (Dynamic Measurement Group, 2012), or at their instructional level based on results from a survey level assessment. To conduct the assessments, Mr. Long takes students out of the classroom during various times of the day. Depending on the time of day, Mr. Long uses different setting locations, including the hallway, a conference room, and an unused classroom. Students are taken individually or in small groups, depending on how far away he must take them for the assessment.

After a few weeks, Mr. Long notices one of the students, Laine, has inconsistent scores in her data set (see Figure 1). Laine, a third-grade student with a specific learning disability, had scores of 61, 43, 75, and 57 WCPM over 4 weeks. Mr. Long compares Laine’s scores with those of other students in the group and notices the other students’ scores are more consistent. For example, Mason’s scores are 66, 71, 71, and 72 WCPM during the same time period (see Figure 2). Mr. Long consults with the school’s reading specialist and finds out that “high variability” includes a range of 10 or more words read correctly above or below the trend line. Because Mr. Long has graphed Laine’s data with a trend line characterizing the data, he can quickly determine that her data are highly variable. Mr. Long realizes that highly variable data can obscure what Laine’s true progress might be. He sees the need to collect more data to determine if the variability can be reduced before a good decision about changing her intervention can be made.

CBM is useful and effective for monitoring student progress in important skills, such as reading, mathematics, and writing. Research has shown that (a) CBM can be easily implemented and interpreted by teachers (e.g., Fuchs, Deno, & Mirkin, 1984), (b) student outcomes have improved when teachers use CBM to inform instructional decision making (e.g., Fuchs, Fuchs, Hamlett, & Stecker, 1991), (c) reliable and valid measures have been developed that predict important student outcomes (e.g., Fuchs, Fuchs, & Maxwell, 1988; Kim, Petscher, Schatschneider, & Foorman, 2010; Wayman, Wallace, Wiley, Tichá, & Espin, 2007), and (d) CBM can be an integral component of multi-tiered systems for identifying and monitoring students’ academic needs (e.g., Kovaleski, VanDerHeyden, & Shapiro, 2013; M. R. Shinn, 2007). CBM for reading (CBM-R) is an efficient and effective research-based progress- monitoring tool to monitor student growth in reading and to evaluate the effectiveness of targeted instruction (Good et al., 2011; Hosp, Hosp, & Howell, 2016). CBM-R is easy to administer and requires minimal resources, such as time and materials. Furthermore, the feedback teachers receive from administering CBM-R can inform instructional decision making and provide critical data about

individual student progress toward reading goals. Given the utility of CBM-R, it is widely used as a key data source for instructional and eligibility decision making (Ardoin, Christ, Morena, Cormier, & Klingbeil, 2013).

The most commonly used CBM-R is ORF (CBM ORF). CBM ORF is a research-based, standardized assessment of connected text that is administered to individual students. CBM ORF is a good indicator of a student’s current skill level and predictor of future reading performance (Deno, Fuchs, & Marston, 2001; Fuchs, Fuchs, Hosp, & Jenkins, 2001; Kim et al., 2010). CBM ORF requires the student to use a variety of different literacy skills, such as decoding, vocabulary, and comprehension (Hosp et al., 2016). CBM ORF originated in the 1970s, when practitioners randomly selected passages from the curriculum materials used in the classroom (e.g., Deno, 1985; Deno, Marston, Shinn, & Tindal, 1983). This practice increased the utility and validity of the measure for making instructional decisions; however, researchers found that student performance on passages within a grade level varied substantially, decreasing the reliability of these measures (see Hintze & Christ, 2004). Later iterations of CBM ORF included development of passages equated based on readability formulae (e.g., Aimsweb;

Figure 1. Laine, third-grade student curriculum-based measurement oral reading fluency, high variability to moderate variability

TEACHING ExCEptional ChildrEn | May/JunE 2018 275

M. M. Shinn & Shinn, 2002; DIBELS, 6th ed.; Good & Kaminski, 2002). Unfortunately, student performance on these passages continued to be excessively variable within grade levels (e.g., Poncy, Skinner, & Axtell, 2005). Excessive variability makes the data difficult to interpret, and therefore, recommendations for instructional modifications become unclear.

Currently, CBM ORF passages have been written using readability formulae for initial equating, then field-tested with students to choose the most equivalent passages to include in published sets (e.g., DIBELS Next; Good et al., 2011; easyCBM; Alonzo, Tindal, Ulmer, & Glasgow, 2006; FastBridge; Christ & Colleagues, 2015). Although some researchers have found persistent variability among these more modern passages (Cummings, Park, & Schaper, 2013), those studies were conducted with higher-performing students than those who are typically included in progress monitoring (e.g., students scoring at or above benchmark at screening). Other researchers found that when passages are implemented as intended, such as to progress monitor students below or well below benchmark, acceptably low levels of variability are seen (O’Keeffe, Bundock, Kladis, Yan, & Nelson, 2017;

Tindal, Nese, Stevens, & Alonso, 2016). Given the challenges presented by excessive variability, educators should be aware of possible sources of

variability and have strategies to prevent and address variability in CBM ORF progress monitoring. These strategies should be followed in addition to the recommendations from the specific publisher of the CBM ORF in use and from general recommendations for implementing and interpreting CBM (e.g., Hosp et al., 2016).

Indicators of Excessive Variability in CBM ORF Progress Monitoring

Educators need to determine how much variability is too much when evaluating student progress-monitoring data. Typically, educators evaluate progress-monitoring data using time

series graphs, with words read correctly on each measurement occasion graphed over time. When educators use visual analysis to determine if a student is making adequate progress or not, multiple graphical components can affect this decision. For example, the amount of variability and the degree of slope in the data can make evaluation decisions more or less accurate, with higher variability and lower slope making decisions substantially less accurate (Nelson, Van Norman, & Christ, 2017; Ottenbacher, 1990; Van Norman & Christ, 2016). If inaccurate decisions are made based on variable data, students who need a change in intervention may not receive it, whereas students who do not need a change may experience an unneeded change in intervention. For CBM ORF, researchers have suggested that very low variability exists when most (i.e., 2/3) of the data points fall within five correctly read words per minute (five above and five below) of a trend line,

and acceptable variability exists when most of the data points fall within 10 correctly read words per minute (10 above and 10 below) of a trend line (Christ, Zopluoglu, Monaghen, & Van Norman, 2013). These values are based on ranges across grade levels (e.g., Christ & Silberglitt, 2007); therefore, students who read more slowly would have lower limits of variability that are acceptable. If available through an electronic database (e.g., AimswebPlus; Pearson, 2017), researchers recommend making these determinations based on confidence intervals, which are generated statistically with the student data (Christ & Silberglitt, 2007). Values that fall outside these ranges can be considered extreme values, which can

Figure 2. Mason, third-grade student curriculum-based measurement oral reading fluency, very low variability

If inaccurate decisions are made based on variable data, students who need a change in intervention may not receive it, whereas students who do not need a change may experience an unneeded change in intervention.

276 CounCil for ExCEptional ChildrEn

affect the interpretation of the data. Although variability will always be a part of assessments using multiple forms—as are used for CBM ORF— educators can improve their decision making by preventing variability as much as possible, identifying excessive variability, understanding how it affects data interpretation, and taking steps to minimize the impact of the variability on decision making.

Sources of Variability

Three primary sources account for the majority of the variability in CBM ORF progress monitoring. These sources include passage-level, student-level, and setting factors.

Passage Factors Contributing to Variability

Variability that is attributable to passage-level factors has decreased over time. Publishers of CBM ORF have taken steps to reduce passage variability, starting with the inclusion of carefully written passages to counter variability that resulted from teacher- selected passages (Good & Kaminski, 2002; Pearson, 2017; M. M. Shinn & Shinn, 2002) and, more recently, including the use of readability formulae, field-testing, and statistical equating to decrease passage variability (Ardoin & Christ, 2009; Christ & Ardoin, 2009; Poncy et al., 2005; Powell-Smith, Good, & Atkins, 2010). In spite of these actions, some passage- level variability remains (Briggs, 2011; O’Keeffe et al., 2017). In particular, research has indicated that there are differences in difficulty level between narrative and expository passages at the same reading level, but there is not a consensus regarding which type of passage tends to be more difficult (Briggs, 2011; O’Keeffe et al., 2017). Publishers continue to include both

narrative and expository passages to increase the validity of their passage sets for monitoring progress toward important goals, which would presumably include the ability to read and comprehend narrative and expository text. The variability between passage types may be explained by passage features that contribute to variability that currently are not captured by readability formulae, such as the repetition in phrasing found in the DIBELS Next progress-monitoring expository passage “Amazing Dolphins,” in which the first three sentences are all formatted as questions starting with “Can you . . .” or “Could you . . .” (O’Keeffe et al., 2017). These features may contribute to

students’ reading this expository passage faster than other passages at this grade level, even though previous research indicated that expository passages tended to be more difficult (Briggs, 2011). Given the evidence of continued variability found among passages types, practitioners should take steps to minimize variability in other ways.

Student Factors Contributing to Variability

Student-level factors also contribute to variability of CBM ORF scores. Many studies evaluating variability of CBM ORF have included participants who score at or above benchmark proficiency (Ardoin & Christ, 2009; Betts, Pickart, & Heistad, 2009; Briggs, 2011; Christ & Ardoin, 2009; Francis et al., 2008; Hintze & Christ, 2004; Poncy et al., 2005). These studies, including higher-performing students, found higher rates of variability than studies conducted only with students who scored below benchmark proficiency (O’Keeffe et al., 2017; Powell-Smith et al., 2010; Tindal et al.,

2016). Students who score below benchmark are the typical population of students who receive progress monitoring (unless there are other concerns), whereas all students across the range of high to lower skills should receive benchmark assessments (Good et al., 2011). Therefore, decisions about variability in progress-monitoring passages should be made based on research with the target population (i.e., students who scored below benchmark). In addition, although it may be common for practitioners to attribute variability in CBM ORF scores to student factors, such as interest in the passage, excitement due to an upcoming holiday, or student mood, these factors have not been found to influence variability beyond passage-level variability (Briggs, 2011). Practitioners need to have an accurate sense of the factors that contribute to variability at the student level so they can take steps to best control this variability.

Setting Factors Contributing to Variability

Studies have found that setting factors, including where assessments are administered, who administers assessments, and the procedures used to administer assessments, contribute to variability in CBM ORF scores. A study that compared two assessment administrators and three assessment settings found significant differences among students’ correct words per minute based on who administered the assessments as well as where the assessments were administered (Derr- Minneci & Shapiro, 1992). Additionally, variability in students’ scores has been attributed to the degree to which assessment administrators follow standardized procedures (Reed & Sturges, 2012). Among a group of trained assessment administrators, 8% of assessments were found to have uncorrectable abnormalities, including forgetting to set a timer, allowing students to continue after a timer went off, not adhering to scripted procedures, forgetting to administer a passage, and providing unscripted encouragement to students (Reed &

Three primary sources account for the majority of the variability in CBM ORF progress monitoring. These sources include passage-level, student- level, and setting factors.

TEACHING ExCEptional ChildrEn | May/JunE 2018 277

Sturges, 2012). Additionally, 91% of assessments had correctable mistakes, including miscounting the number of errors, counting inserted words as errors, and miscalculating the words correct per minute (Reed & Sturges, 2012). Even among trained assessors, it is common for adherence to procedures to diminish over time if periodic refresher trainings are not provided (Reed & Sturges, 2012). Due to the sensitivity of CBM ORF to setting factors, such as administrator characteristics, environment, and procedures, practitioners should ensure fidelity of assessment administration (Christ & Silberglitt, 2007) and provide a consistent setting for assessments.

Recommendations for Minimizing Variability

Educators can reduce variability related to passage-, student-, and setting- related factors.

Passages

Because CBM ORF typically involves the use of connected text to present a coherent story or describe a specific topic, some variability is to be expected. To minimize the variability due to passage differences, educators should use the most recently updated, published passage sets (e.g., DIBELS Next; Good et al., 2011; easyCBM; Alonzo et al., 2006; FastBridge; Christ and Colleagues, 2015). Choose passage sets that have been written specifically for the purpose of assessment using grade-level guidelines (i.e., not chosen randomly from

instructional materials) and have been field-tested with students and chosen based on minimal variability in actual student performance (i.e., not just leveled with readability scores). To date,

there is no published research indicating which current, published probe sets have more or less error. In addition, educators should implement the passages in the order that they were published within each grade level. For example, if progress-monitoring passages are numbered 1 to 20, educators should administer the passages in that order. Authors have typically ordered the passages to distribute any remaining variability evenly across the passages (e.g., Powell-Smith et al., 2010). To increase the external validity of the assessment, passage sets often include both narrative (i.e., story-based) and expository passages (i.e., content area texts, such as science, social studies, history). Research has shown that there are often differences in reading performance across these types of passages, but some research has shown that narrative passages are easier than expository (e.g., Briggs, 2011), whereas other research has found some expository passages to be easier than narrative passages (e.g., O’Keeffe et al., 2017). No clear recommendations exist

about how to minimize variability for these types of passage differences, but educators should be aware that these differences may occur in progress monitoring so they can track trends in

students’ scores related to type of passage if high variability is observed. Tables 1 and 2 are checklists of recommendations to reduce variability in CBM ORF progress monitoring before and during assessment administration.

Student

Although the selection of students to progress monitor and the level of materials to use should be based on the types of decisions to be made (i.e., evaluating effects of interventions vs. evaluating grade-level proficiency), we recommend that schools select students for progress monitoring who score below or well below established benchmarks at screening. For example, DIBELS’ authors suggest that students who score below the benchmark goal on one or more benchmark assessment measures should receive progress monitoring (Good et al., 2011). Typically, these students require more intensive and individualized instructional interventions in one or more skill areas. Frequent progress

Table 1. Before Assessment Checklist of Recommendations for Reducing Variability in CBM ORF Progress Monitoring

•• Use most recently updated, published progress-monitoring passage sets.

•• Choose passage sets written for assessment, field-tested with students to establish equivalence, with minimal variability in student performance.

•• Choose students for progress monitoring who score below or well below publisher benchmarks at screening.

•• Use a survey-level assessment or “testing back” to determine correct instructional grade level for progress monitoring, as recommended by publisher of progress-monitoring system.

•• Train all individuals who will administer CBM ORF assessments.  Include training on administration and scoring procedures.  Include opportunities to practice with feedback.

Note. CBM = curriculum-based measurement; ORF = oral reading fluency.

Practitioners need to have an accurate sense of the factors that contribute to variability at the student level so they can take steps to best control this variability.

278 CounCil for ExCEptional ChildrEn

monitoring is recommended to ensure these students are making adequate progress toward reading goals. Student data from progress monitoring will allow educators to make decisions on whether to increase, decrease, or modify reading interventions.

After identifying students to monitor, the next step is to determine the instructional level at which to progress monitor each student. DIBELS’ and Aimsweb’s authors recommend using a survey-level assessment or “testing back” until the correct instructional grade level is determined (Good et al., 2011; Pearson, 2017). A survey-level assessment is a tool that can be used to determine a student’s instructional level and identify an appropriate grade level to progress monitor a student and an applicable fluency goal. For example, if a student is unable to meet the grade-level benchmark, educators can administer a survey-level assessment to identify an appropriate instructional level to progress monitor the student, which may be lower than the student’s chronological grade level. If educators progress monitor a student on a different grade level, they should still administer on-grade-level assessments at least three times per year during benchmark screening assessments. Assessing a student on-grade level three times per year will help determine if generalization from

intervention to grade-level standards is occurring. In addition, for students with disabilities, benchmark assessments allow the educator to determine if the student is making progress in grade-level standards.

Setting

Strategies for reducing setting-related variability include ensuring consistent and accurate administration and scoring within appropriate settings.

Assessors. Ensuring fidelity of assessment procedures and scoring practices is an essential element of progress monitoring. Consistent implementation of progress-monitoring methods and scoring procedures across the school is necessary because the data are used in making both low- stakes and high-stakes decisions. All individuals who will administer CBM ORF assessments need to be trained in the procedures, regardless of the CBM ORF program used. In addition, review of training should occur at regular intervals (e.g., annually) to prevent drift from accepted procedures. Training should include administration and scoring procedures and include

opportunities to practice administering and scoring assessment measures with feedback. In addition to providing training, we recommend self-checking assessment accuracy or having a schoolwide system for regularly

monitoring and giving feedback on assessment accuracy. School personnel can use assessment accuracy checklists provided by the CBM ORF publisher (e.g., for DIBELS Next, Good et al., 2011, pp. 113–120) or created by school personnel. The principal or reading specialist in the school could administer fidelity checks at each grade level throughout the school year. The fidelity checks should be used as learning tools for educators and contribute to their professional development.

Location of assessment. Assessors should administer progress-monitoring measures consistently. Most research recommends that CBM ORF measures be administered multiple times per week (Shapiro, 2012). Studies have demonstrated that inconsistencies in CBM ORF administration, including where passages are administered and

Table 2. During Assessment Checklist of Recommendations for Reducing Variability in CBM ORF Progress Monitoring

•• Implement passages in their published order within each grade level.

•• Administer CBM ORF measures in a consistent, quiet location with few distractions or disruptions.  Avoid administering in high-traffic or noisy locations.

•• Be aware of differences in difficulty between narrative and expository passages.  Indicate which passages are narrative or expository on individual students’ graphs if scores are highly variable.

•• If monitoring off grade level, administer on-grade-level assessments at least three times per year during benchmark screening assessments.

•• Use a strong set of data to determine aimlines.  Administer three passages and use the median score to establish the baseline data point for the aimline.

•• Monitor assessment accuracy at each grade level throughout the year.  Use self-checks or establish a schoolwide system for regular monitoring and feedback on assessment accuracy.  Use assessment accuracy checklists provided by CBM ORF publisher (if available), or create accuracy checklists for the

school.

Ensuring fidelity of assessment procedures and scoring practices is an essential element of progress monitoring.

TEACHING ExCEptional ChildrEn | May/JunE 2018 279

the extent to which the standardized directions are followed, can influence variability in students’ scores (Derr- Minneci & Shapiro, 1992). Administer CBM ORF measures in a consistent, quiet location with few distractions or disruptions. Examples of appropriate places to administer ORF measures include quiet, low-traffic areas of the library or classroom; private offices (e.g., counselor or principal’s office); or resource rooms. Avoid administering CBM ORF passages in high-traffic or noisy locations, such as hallways or busy classrooms.

Recommendations for Data Displays and Interpretation

Once data have been collected using the highest-quality assessment passages according to standardized instructions and procedures and in a consistent, quiet location, educators can graph and interpret the CBM ORF data in ways that minimize the impact of some variability. Recent research studies suggest that better decisions for individual students were made when educators (a) used graphical supports, such as a trend line and a goal line for comparison of progress (e.g., Van Norman & Christ, 2016; Van Norman, Nelson, Shin, & Christ, 2013); (b) collected data for longer periods of time in the presence of variability and

low slope (e.g., 12–14 weeks of once- weekly measures, as opposed to 6 weeks of once-weekly measures in the presence of low variability and higher slope, such as growth of at least 1.5 WCPM per week; Christ et al., 2013; Van Norman & Christ, 2016); (c) received training on graph and data interpretation, such as identifying and removing extreme values that can skew a trend line (Nelson et al., 2017); and (d) used visual analysis procedures (e.g., comparing trend line with goal line in the context of the data) rather than decision rules (e.g., 3 data points above or below the line; Van Norman & Christ 2016). It is important to note that educators may need more advanced training to conduct some of these procedures (i.e., treatment of extreme values) to ensure that accuracy of measurement is preserved. In addition, some researchers have suggested that collecting more probes each week (e.g., three passages vs. one passage) can reduce variability (Christ et al., 2013). This may not be a practical solution for all settings. However, when data are used for high- stakes decisions, such as eligibility for special education services, it is imperative that the data used for these decisions be accurate to ensure students receive appropriate services and supports. These procedures improve educator decision making with

CBM ORF in general but are particularly important in the presence of increased variability and low-to- moderate slope (see Table 3 for a checklist of recommendations for data display and interpretation).

Mr. Long considers three possible sources of variability: passage-level factors, student-level factors, and setting factors. To control for passage-level factors, he reviews the DIBELS Next technical manual and verifies that the passages were field-tested with students to reduce variability, and he is using the correct instructional-level passages. In consideration of student-level factors, Mr. Long verifies that Laine scored below benchmark on the recent screening assessment. He also talks with other applicable school staff about Laine. Her behavior in school has remained consistent—her motivation to learn is high, and she enjoys spending time learning to read. Mr. Long has also checked in with her parents and has confirmed that nothing unusual is happening at home. Mr. Long then addresses variability due to setting-level factors. After a discussion with the reading specialist and some fidelity checks, Mr. Long confirms that he is adhering to the standardized administration protocols: He reads the scripted directions, starts and stops the timer when required, and checks his

Table 3. Recommendations for Data Display and Interpretation for CBM ORF Progress Monitoring

•• Include trend line and goal line on individual graphs to aid in visual analysis.  Evaluate progress by comparing trend line and goal line.

•• Use visual analysis instead of decision rules (e.g., 3 points above or below the goal line for making changes).  Visual analysis of data includes evaluation of slope, trend, stability, level, and immediacy of effects.

•• Pursue training on graph and data interpretation.  Training may be needed to aid in identifying and addressing extreme values that skew a trend line.  If available, generate and report confidence intervals based on student data to note high and low variability more

accurately.

•• Collect more data (e.g., 12–14 weeks) when the data are highly variable or the slope is low (e.g., per-week growth of 0.50–1.0 correctly read words per minute).

•• Consider collecting more probes each week (e.g., three passages per week) to decrease variability and make a decision sooner (e.g., 8–10 weeks).

•• If variability remains despite actions taken to reduce it, consider administering additional academic and behavioral assessments, and evaluate contextual factors (e.g., auditory or visual supports, motivation, etc.).

280 CounCil for ExCEptional ChildrEn

scoring prior to graphing the data. Mr. Long uses the classroom next door when giving most students the progress- monitoring assessments. However, he often must take Laine to the hallway to complete the probe(s) because there is no other available space close to the classroom during the time Laine is able to take the assessment. Sometimes other groups of students are walking to art or physical education classes during Laine’s assessment. Although Laine is used to reading in the hallway on the floor, the added distractions on some occasions may be contributing to variability in her scores. Mr. Long works with other teachers to find a less distracting location (i.e., an unused office in the library) to administer the assessment. After this change in setting, Mr. Long monitors Laine’s data (see Figure 1). Laine’s data have moderate levels of variability. On the basis of more consistent data, Mr. Long determines that an instructional change needs to be made for Laine because she is not meeting the aimline he set for her based on grade-level benchmarks.

Conclusion

Using CBM ORF progress-monitoring measures is an effective way to consistently evaluate students’ reading performance and make data-based instructional decisions. However, educators should understand and properly control for factors that may influence variability. Working with other professionals at their schools, educators should ensure that data are collected and analyzed following the recommendations presented in this article as well as the recommendations from the publisher of the CBM ORF measure to maximize the use of CBM ORF. It is also important to recognize that variable data are not necessarily inaccurate. If these recommended strategies do not reduce variability adequately, we recommend that educators administer additional assessments and assess contextual variables (e.g., student’s use of visual and auditory supports, level of focus, motivation) to obtain a more complete

picture of the student’s academic and behavioral performance.

References

Alonzo, J., Tindal, G., Ulmer, K., & Glasgow, A. (2006). easyCBM® online progress monitoring assessment system. Eugene: University of Oregon, Behavioral Research and Teaching.

Ardoin, S. P., & Christ, T. J. (2009). Curriculum-based measurement of oral reading: Standard errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an experimental passage set. School Psychology Review, 38, 266–283.

Ardoin, S. P., Christ, T. J., Morena, L. S., Cormier, D. C., & Klingbeil, D. A. (2013). A systemic review and summarization of the recommendations and research surrounding curriculum- based measurement of oral reading fluency (CBM-R) decision rules. Journal of School Psychology, 51, 1–18. doi:10.1016/j.jsp.2012.09.004

Betts, J., Pickart, M., & Heistad, D. (2009). An investigation of the psychometric evidence of CBM-R passage equivalence: Utility of readability statistics and equating for alternate forms. Journal of School Psychology, 47, 1–17. doi:10.1016/j.jsp.2008.09.001

Briggs, R. N. (2011). Investigating variability in student performance on DIBELS oral reading fluency third grade progress monitoring probes: Possible contributing factors (Doctoral dissertation). Retrieved from ProQuest database. (UMI No. 3466319)

Christ, T. J., & Ardoin, S. P. (2009). Curriculum-based measurement of oral reading: Passage equivalence and probe-set development. Journal of School Psychology, 47, 55–75. doi:10.1016/j. jsp.2008.09.004

Christ, T. J., & Colleagues. (2015). Formative Assessment System for Teachers: Abbreviated technical manual, Version 2.0. Minneapolis, MN: Author and FastBridge Learning.

Christ, T.J., & Silberglitt, B. (2007). Estimates of the standard error of measurement for curriculum-based measures of oral reading fluency. School Psychology Review, 36(1), 130–146.

Christ, T. J., Zopluoglu, C., Monaghen, B. D., & Van Norman, E. R. (2013). Curriculum-based measurement of oral reading: Multi-study evaluation of schedule, duration and dataset quality on progress monitoring outcomes.

Journal of School Psychology, 51, 19–57. doi:10.1016/j.jsp.2012.11.001

Cummings, K. D., Park, Y., & Schaper, H. A. B. (2013). Form effects on DIBELS Next oral reading fluency progress- monitoring passages. Assessment for Effective Intervention, 38, 91–104. doi:10.1177/1534508412447010

Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219–232. doi:10.1177/001440298505200303

Deno, S. L., Fuchs, L. S., & Marston, D. (2001). Using curriculum-based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507–524.

Deno, S. L., Marston, D., Shinn, M., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling reading disability. Topics in Learning and Learning Disabilities, 2(4), 53–59.

Derr-Minneci, T. F., & Shapiro, E. S. (1992). Validating curriculum- based measurement in reading from a behavioral perspective. School Psychology Quarterly, 7, 2–16.

Dynamic Measurement Group. (2012). Progress monitoring with DIBELS Next®. Eugene, OR: Author.

Francis, D. J., Santi, K. L., Barr, C., Fletcher, J. M., Varisco, A., & Foorman, B. R. (2008). Form effects on the estimation of students’ oral reading fluency using DIBELS. Journal of School Psychology, 46, 315–342. doi:10.1016/j. jsp.2007.06.003

Fuchs, L. S., Deno, S. L., & Mirkin, P. K. (1984). The effects of frequent curriculum-based measurement and evaluation on pedagogy, student achievement and student awareness of learning. American Education Research Journal, 21, 449–460. doi:10.2307/1162454

Fuchs, L. S., Fuchs, D., Hamlett, C. L., & Stecker, P. M. (1991). Effects of curriculum based measurement and consultation on teacher planning and student achievement in mathematics operations. American Education Research Journal, 28, 617–641. doi:10.3102/00028312028003617

Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239–256. doi:10.1207/ S1532799XSSR0503_3

TEACHING ExCEptional ChildrEn | May/JunE 2018 281

Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal measures of reading comprehension. Remedial and Special Education, 9, 20– 28. doi:10.1177/074193258800900206

Good, R. H., III, & Kaminski, R. A. (2002). Dynamic Indicators of Basic Early Literacy Skills (6th ed.). Eugene, OR: Institute for the Development of Educational Achievement.

Good, R. H., III, Kaminski, R. A., Cummings, K., Dufour-Martel, C., Petersen, K., . . . Wallin, J. (2011). DIBELS Next assessment manual. Eugene, OR: Dynamic Measurement Group.

Hintze, J. M., & Christ, T. J. (2004). An examination of variability as a function of passage variance in CBM progress monitoring. School Psychology Review, 33, 204–217.

Hosp, M. K., Hosp, J. L., & Howell, K. W. (2016). The ABCs of CBM: A practical guide to curriculum-based measurement (2nd ed.). New York, NY: Guilford Press.

Kim, Y., Petscher, Y., Schatschneider, C., & Foorman, B. (2010). Does growth rate in oral reading fluency matter in predicting reading comprehension achievement? Journal of Educational Psychology, 102, 652–667.

Kovaleski, J. F., VanDerHeyden, A. M., & Shapiro, E. S. (2013). The RTI approach to evaluating learning disabilities. New York, NY: Guilford Press.

Nelson, P. M., Van Norman, E. R., & Christ, T. J. (2017). Visual analysis among novices: Training and trend lines as graphic aids. Contemporary School Psychology, 21, 93–102. doi:10.1007/ s40688-016-0107-9

O’Keeffe, B. V., Bundock, K., Kladis, K. L., Yan, R., & Nelson, K. (2017). Variability in DIBELS Next progress monitoring measures for students at risk for reading difficulties. Remedial and Special Education, 38, 272–283. doi:10.1177/0741932517713310

Ottenbacher, K. J. (1990). Visual inspection of single-subject data: An empirical

analysis. Mental Retardation, 28, 283–290.

Pearson. (2017). AimswebPlus progress monitoring guide. Bloomington, MN: NCS Pearson.

Poncy, B. C., Skinner, C. H., & Axtell, P. K. (2005). An investigation of the reliability and standard error of measurement of words read correctly per minute using curriculum based measurement. Journal of Psychoeducational Assessment, 23, 226–238. doi:10.1177/073428290502300403

Powell-Smith, K. A., Good, R. H., & Atkins, T. (2010). DIBELS Next oral reading fluency readability study (Tech. Rep. No. 7). Eugene, OR: Dynamic Measurement Group.

Reed, D. K., & Sturges, K. M. (2012). An examination of assessment fidelity in the administration and interpretation of reading tests. Remedial and Special Education, 34, 259–268. doi:10.1177/0741932512464580

Shapiro, E. S. (2012). Commentary on progress monitoring with CBM-R and decision making: Problems found and looking for solutions. Journal of School Psychology, 51, 59–66. doi:10.1016/j. jsp.2012.11.003

Shinn, M. M., & Shinn, M. R. (2002). AIMSweb training workbook: Administration and scoring of reading curriculum-based measurement (R-CBM) for use in general outcome measurement. New York, NY: Pearson.

Shinn, M. R. (2007). Identifying students at risk, monitoring performance, and determining eligibility within response to intervention: Research on educational need and benefit from academic intervention. School Psychology Review, 36, 601–617.

Tindal, G., Nese, J. F. T., Stevens, J. J., & Alonso, J. (2016). Growth on oral reading fluency measures as a function of special education and measurement sufficiency. Remedial and Special Education, 37, 28–40. doi:10.1177/0741932515590234

Van Norman, E. R., & Christ, T. J. (2016). How accurate are interpretations of curriculum-based measurement progress monitoring data? Visual analysis versus decision rules. Journal of School Psychology, 58, 41–55. doi:10.1016/j. jsp.2016.07.003

Van Norman, E. R., Nelson, P. M., Shin, J., & Christ, T. J. (2013). An evaluation of the effects of graphic aids in improving decision accuracy in a continuous treatment design. Journal of Behavioral Education, 22, 283–301. doi:10.1007/ s10864-013-9176-2

Wayman, M. M., Wallace, T., Wiley, H. I., Tichá, R., & Espin, C. A. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41, 85–120. doi:10.11 77/00224669070410020401

Kaitlin Bundock, Assistant Professor, Department of Special Education and Rehabilitation, Utah State University, Logan; Breda V. O’Keeffe, Assistant Professor, Department of Special Education, Kristen Stokes, Doctoral Student, Department of Special Education, and Kristin Kladis, Doctoral Candidate, Department of Special Education, University of Utah, Salt Lake City.

Address correspondence concerning this article to Kaitlin Bundock, PhD, Utah State University, 2865 Old Main Hill, Logan, UT 84322-2865 (e-mail: kaitlin.bundock@usu. edu).

The development of this article was supported in part by a grant from the College of Education at the University of Utah. Opinions expressed herein are the authors’ and do not necessarily reflect the position of the College of Education at the University of Utah, and such endorsements should not be inferred.

TEACHING Exceptional Children, Vol. 50, No. 5, pp. 273–281. Copyright 2018 The Author(s).

Copyright of Teaching Exceptional Children is the property of Sage Publications Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.