Education Week 4 assignment

323

Chapter 13

Making Sense Out of Standardized Test Scores

Chief Chapter Outcome

A sufficient knowledge of standardized educational testing’s fundamentals so that, when an issue related to such testing is encountered, or when a basic question about standardized testing is asked, appropriate responses can be supplied

Learning Objectives

13.1 Describe and apply the fundamental characteristics of standardized testing.

13.2 Describe the contributions and characteristics of instructionally diagnostic tests.

Classroom teachers need to be able to interpret the results not only of their own assessment procedures, but also of the various kinds of standardized tests that are frequently administered to students. Teachers should be able to interpret such test results so they can base at least some of their classroom instructional deci- sions on those results. Teachers also should be able to respond accurately when students’ parents raise such questions as, “What does my child’s grade-equivalent score of 7.4 really mean?” or “When my child’s achievement test results are at the 97th percentile, is that three percentiles from the top or from the bottom?”

This chapter focuses on the task of making sense out of students’ perfor- mances on standardized achievement and aptitude tests. One of the kinds of tests under consideration will be the achievement tests (for instance, in mathematics or

M13_POPH0936_10_SE_C13.indd 323M13_POPH0936_10_SE_C13.indd 323 09/11/23 6:09 PM09/11/23 6:09 PM

324 Chapter 13 Making Sense Out of Standardized test Scores

reading) developed and distributed by commercial testing companies. Achieve- ment tests are also developed and administered by state departments of educa- tion as a part of annual statewide accountability assessments in such subjects as social studies, sciences, mathematics, reading, and writing. These state tests often employ reporting procedures akin to those used with commercially distributed standardized achievement tests.

And, in spite of an earnest attempt to avoid numbers much larger than 9 in this chapter, you’ll encounter a few that are, indeed, larger. Don’t be dismayed. Simply go through the examples, one step at a time, and you’ll emerge a finer human being. The numbers in this chapter will be easier to deal with than they appear to be at first glance.

These days, standardized tests (especially standardized achievement tests) are often misused. In Chapter 15 you’ll learn why it is unwise to try to evaluate edu- cators’ effectiveness based on students’ scores on many such tests. Nonetheless, standardized tests do have an important, educationally useful role to play. This chapter focuses on the appropriate uses of standardized tests.

The chapter will conclude with a consideration of two aptitude tests that have—until the last few years—had a whopping impact on students’ lives. You’ll be looking at the two examinations that have often functioned as determiners of a student’s continued education. That’s right; we’ll be dealing with the two most widely used college entrance exams: the SAT and the ACT.

If you’re a current or would-be secondary school teacher, you’ll immediately recognize the need for you to know about the SAT and the ACT. After all, some of your students may soon be taking one or both of these tests. But if you’re an elementary school teacher, or are preparing to be one, you might be think- ing, “What do college entrance exams have to do with me?” Well, all teachers— elementary and secondary—ought to understand at least the most significant facts about these two examinations. Indeed, even teachers of primary-grade kids will find parents asking questions such as, “How can we get our child really ready for those important college entrance tests when the time comes?” Every teacher should be familiar with any truly significant information associated with the teaching profession. Thus, because most students and their parents will, at some point, want to know more about college entrance exams, it seems that every single teacher—and even married ones—should know most of the basic SAT and ACT facts you’ll learn about in this chapter.

Standardized Tests A standardized test is a test that is designed to yield either norm-referenced or criterion-referenced inferences and that is administered, scored, and interpreted in a standard, predetermined manner. Almost all nationally standardized tests are distributed by commercial testing firms. Most such firms are for-profit cor- porations, although there are a few not-for-profit measurement organizations,

M13_POPH0936_10_SE_C13.indd 324M13_POPH0936_10_SE_C13.indd 324 09/11/23 6:09 PM09/11/23 6:09 PM

Standardized tests 325

such as the Educational Testing Service (ETS), that distribute nationally standard- ized tests. Almost all nationally standardized tests, whether they’re focused on the measurement of students’ aptitude or of students’ achievement, are chiefly intended to provide norm-referenced interpretations.

Standardized achievement tests have also been developed in a number of states under the auspices of state departments of education. In some instances, these state-chosen standardized tests may have been acquired from a confed- eration of states, for instance, the states affiliated with the Smarter Balanced Assessment Consortium (SBAC)—or even from smaller consortia of only a few in-collaboration states. These statewide tests (clearly intended to be administered, scored, and interpreted in a standard, predetermined fashion) have usually been installed to satisfy some sort of legislative mandate that establishes an educational accountability program for that state. In certain instances, important decisions about individual students are made on the basis of a student’s test performance. In some states, for example, if a student does not pass a prescribed statewide basic-skills examination by the end of high school, the student is not awarded a state-sanctioned diploma—even though the student has satisfied all other curricu- lar requirements. In other instances, although no contingencies for individual stu- dents depend on how a student performed on a test, the results of student tests are publicized by the media on a district-by-district or school-by-school basis. The test results are thus regarded as an indicator of local educators’ effectiveness, at least in the perception of many citizens. These state-sired standardized achievement tests are generally intended to yield criterion-referenced interpretations. Educational aptitude tests are rarely, if ever, developed by state departments of education.

Although standardized tests have traditionally consisted almost exclusively of selected-response items, in recent years the developers of standardized tests have attempted to incorporate a certain number of constructed-response items in their tests. Standardized tests, because they are intended for widespread use, are devel- oped with far more care (and cost) than is possible in an individual teacher’s class- room. Even so, the fundamentals of test development that you’ve learned about in earlier chapters are routinely employed when standardized tests are developed. In other words, the people who create the items for such tests attempt to adhere to the same kinds of item-writing and item-improvement precepts that you’ve learned about. The writers of multiple-choice items for standardized tests worry, just as you should, about inadvertently supplying students with clues that give away the cor- rect answer. The writers of short-answer items for standardized tests try to avoid, just as you should, the inclusion of ambiguous language anywhere in their items.

Because of the pervasive national attention received by the COVID pandemic, the nation’s news media have been following with interest what has been hap- pening recently to students’ performances on the National Assessment of Stu- dent Progress (NAEP) described earlier in this book’s Chapter 2. For example, the New York Times indicated that the NAEP scores released nationally near the close of October 2022 offer “the most definitive indictment yet of the pandemic’s impact on millions of schoolchildren.” Those NAEP scores, which, prior to the

M13_POPH0936_10_SE_C13.indd 325M13_POPH0936_10_SE_C13.indd 325 09/11/23 6:09 PM09/11/23 6:09 PM

326 Chapter 13 Making Sense Out of Standardized test Scores

pandemic’s arrival more than 2 years earlier, routinely indicated no change or modest improvements for U.S. students in mathematics and reading at grades 4 and 8, were described as “plummeting” by Eugene Robinson, esteemed columnist of the Washington Post. He claimed that the lowering of students’ NAEP scores during the pandemic “ushered in a falling tide that lowered all boats.”

Clearly credible because of NAEP’s federally overseen rigor in its collection and interpretation of students’ performances, the substantial drops in students’ test scores will be watched keenly by commentators on American schooling. Although other standardized tests will be administered and reported in years to come, the traditionally careful administrative procedures employed by NAEP oblige educa- tors and policymakers to heed its scores and the implication of those scores.

During these tumultuous days of U.S. schooling, an era when many less than rigorous major examinations will probably be touted by commercial assessment firms as providing an accurate and believable picture of students’ learning status, this is a time for educators and parents to gauge with care the credibility of stu- dents’ test results. Just because a standardized test’s performances are reported numerically, and just because many students have been involved in its adminis- tration, it should not be automatically assumed that a major standardized test’s reporting and interpretations warrant our acceptance.

There are, of course, substantial differences in the level of effort associated with the construction of standardized tests and the construction of classroom tests. A commercial testing agency may assign a flotilla of item-writers and a fleet of item-editors to a new test-development project, whereas teachers are fortunate if they have a part-time aide or, possibly, a malleable spouse who can proofread teacher-made tests in search of typographical errors.

Decision time Which test to Believe

Each spring in the Big Valley Unified School District, students in grades 5, 8, 10, and 12 complete nationally standardized achievement tests in reading and mathematics, as well as a nationally standardized test described by its publishers as “a test of the student’s cognitive aptitude.” Because William Avory teaches eighth-grade students in his English classes, he is given the task of answering any questions raised by his eighth-graders’ parents about the test results.

He is faced with one fairly persistent question from most parents, particularly those parents whose children scored higher on the aptitude test than on

the achievement test. For example, Mr. and Mrs. Wilkins (Wanda’s parents) put the question like this: “If Wanda scored at the 90th percentile on the aptitude test and only at the 65th percentile on the achievement test, does that mean she’s not logging enough study time? Putting it another way,” they continued, “should we really believe the aptitude test’s results or the achievement test’s results? Which is Wanda’s ‘true’ test performance?”

If you were William and had to decide how to answer the questions posed by Wanda’s parents, what would your answer be?

M13_POPH0936_10_SE_C13.indd 326M13_POPH0936_10_SE_C13.indd 326 09/11/23 6:09 PM09/11/23 6:09 PM

Group-Focused test Interpretation 327

Group-Focused Test Interpretation Although the bulk of this chapter will be devoted to a consideration of score-reporting mechanisms used to describe an individual student’s performance, you’ll sometimes find you need to describe the performance of your students as a group. To do so, you’ll typically compute some index of the group of scores’ central tendency, such as when you determine the group’s mean or median perfor- mance. For example, you might calculate the mean raw score or the median raw score for your students. A raw score is simply the number of items that a student has answered correctly. The mean, as you probably know, is the arithmetic average of a set of scores. For example, the mean of the scores 10, 10, 9, 8, 7, 6, 3, 3, and 2 is 6.4 (found by summing the nine scores and then dividing that sum by 9). The median is the midpoint of a set of scores. For the nine scores in the previous example, the median is 7 because this score divides the group into two equal parts. Means and medians are useful ways to describe the point at which the scores in a set of scores are centered.

In addition to describing the central tendency of a set of scores (via the mean and/or median), it is also helpful to describe the variability of the scores—that is, how spread out the scores are. One simple measure of the variability of a set of students’ scores is the range. The range is calculated by simply subtracting the lowest-scoring student’s score from the highest-scoring student’s score. To illustrate, suppose the highest test score earned by students in your class was 49 correct out of 50, earned by Hortense (she always tops your tests; it is surprising that she missed one item). Suppose, further, that the lowest score of 14 correct was, as usual, earned by Ed. The range of this set of scores would be 35—that is, Hortense’s 49 minus Ed’s 14.

Because only two scores influence the range, it is less frequently used as an index of test-score variability than is the standard deviation. A standard devia- tion is a kind of average. More accurately, it’s the average difference between the individual scores in a group of scores and the mean of that set of scores. The larger the size of the standard deviation, the more spread out are the scores in the distribution. (That’s a posh term to describe a set of scores.) Here is the formula for computing a standard deviation for a set of scores:

S D X M N

Standard Deviation ({ . .}) ( )2 = Σ −

where X M( )2Σ − = the sum of the squared raw scores −X( ) the mean (M)

=N the number of scores in the distribution

Here’s a step-by-step description of how you compute a standard deviation using this formula. First, compute the mean of the set of scores. Second, subtract the mean from each score in the distribution. (Roughly half of the resulting val- ues, called deviation scores, will have positive values, and roughly half will have negative values.) Third, square each of these deviation scores, that is, multiply

M13_POPH0936_10_SE_C13.indd 327M13_POPH0936_10_SE_C13.indd 327 09/11/23 6:09 PM09/11/23 6:09 PM

328 Chapter 13 Making Sense Out of Standardized test Scores

each number by itself. This will make them all positive. Fourth, add the squared deviation scores together. Fifth, divide the resulting sum by the number of scores in the distribution. Sixth, and last, take the square root of the results of the division you did in the fifth step. The square root that you get is the standard deviation. Not too terrifyingly tough, right?

To illustrate the point that larger standard deviations represent more spread in a distribution of scores than smaller standard deviations, compare the two ficti- tious sets of scores on a 10-item short-answer test presented in Figure 13.1. Both sets of scores have a mean of 5.0. The distribution of scores at the left is much more homogeneous (less spread out) than the distribution of scores at the right. Note that the standard deviation for the more homogeneous scores is only 1.1, whereas the standard deviation for the more heterogeneous scores is 3.2. The larger the standard deviation, therefore, the more distant, on average, will be the distribu- tion’s scores from the distribution’s mean.

You may have occasions to describe the scores of an entire group of the stu- dents you teach. Those descriptions might portray your students’ performances on standardized tests or on teacher-made tests. If you get at all comfortable and cuddly with means and standard deviations, those two indices usually provide a better picture of a score distribution than do the median and range. But, if you think means and standard deviations are statistical gibberish, then go for the median (midpoints are easy to identify) and range (ranges require only your com- petence in subtraction). With group-based interpretations out of the way, let’s turn now to interpreting individual students’ scores from the kinds of standardized tests commonly used in education.

Individual Student Test Interpretation Two overriding frameworks are generally used to interpret students’ test scores. Test scores are interpreted in absolute or relative terms. When we interpret a stu- dent’s test score absolutely, we infer from the score what it is that the student can or cannot do. For example, based on a student’s performance on test items dealing with mathematics computation skills, we make an inference about the

More Homogeneous Scores: More Heterogeneous Scores:

3, 4, 4, 5, 5, 5, 5, 6, 6, 7 0, 1, 2, 4, 5, 5, 6, 8, 9, 10

Mean = 5.0 Mean = 5.0

S.D. = A 12 10

= 1.1 S.D. = A 102 10

= 3.2

Figure 13.1 two Fictitious Sets of tests Scored with equal Means but Different Standard Deviations

M13_POPH0936_10_SE_C13.indd 328M13_POPH0936_10_SE_C13.indd 328 09/11/23 6:10 PM09/11/23 6:10 PM

Individual Student test Interpretation 329

degree to which the student has mastered the array of computation skills rep- resented by the test’s items. The teacher may even boil the interpretation down to a dichotomy—namely, whether the student should be classified as having mastered or as not having mastered the skill or knowledge being assessed. A mastery-versus-nonmastery interpretation represents an absolute interpretation of a student’s test score. Classroom teachers sometimes use this absolute interpre- tive approach when creating tests to assess a student’s knowledge or skills based on a particular unit of study.

When we interpret a student’s test score relatively, we infer from the score how the student stacks up against other students who are currently taking the test or who have already taken the test. For example, when we say that Jamal’s test score is “above average” or “below average,” we are making a relative test interpretation because we use the average performance of other students to make sense of Jamal’s test score.

As pointed out earlier, this chapter focuses on how teachers and parents can interpret scores on standardized tests. Because almost all standardized test scores require relative interpretations, the three interpretive schemes to be considered in the chapter are all relative score-interpretation schemes. The vast majority of standardized tests, whether achievement tests or aptitude tests, provide relative interpretations. Accordingly, teachers need to be especially knowledgeable about relative score-interpretation schemes.

Percentiles The first interpretive scheme we’ll consider, and by all odds the most commonly used one, is based on percentiles or, as they are sometimes called, percentile ranks. Percentiles are used most frequently in describing standardized test scores, because percentiles are readily understandable to most people.

A percentile compares a student’s score with those of other students in a norm group. A student’s percentile indicates the percent of students in the norm group that the student outperformed. A percentile of 60, for example, indicates that the student performed better than 60 percent of the students in the norm group.

Let’s spend a moment describing what a norm group is. As indicated, a per- centile compares a student’s score with scores earned by those in a norm group. This comparison with the norm group is based on the performances of a group of individuals who have already been administered a given examination. For instance, before developers of a new standardized test publish their test, they usu- ally administer the test to a large number of students who then become the norm group for the test. Typically, different norm groups of students are assembled for all the grade levels for which percentile interpretations are made.

Figure 13.2 shows a graphical depiction of a set of 3,000 students’ scores such as might have been gathered during the norming of a nationally standard- ized achievement test. Remember, we refer to such students as the norm group. The area under the curved line represents the number of students who earned

M13_POPH0936_10_SE_C13.indd 329M13_POPH0936_10_SE_C13.indd 329 09/11/23 6:10 PM09/11/23 6:10 PM

330 Chapter 13 Making Sense Out of Standardized test Scores

scores at that point on the baseline. You’ll notice that for a typical norm group’s performance, most students score in the middle, and only a few students earn very high or very low scores.

In fact, if the distribution of test scores in the norm group is perfectly normal, then, as you see in Figure 13.3, over two thirds of the scores (represented by the area under the curved line) will be located relatively close to the center of the distribution—that is, plus or minus 1 standard deviation (S.D.) from the mean.

Not all norm groups are national norm groups. Sometimes test publishers, at the request of local school officials, develop local norms. These local norms can be either state norms or school-district norms. Comparisons of students on the basis of local norms are sometimes seen as being more meaningful than comparisons based on national norms. Indeed, if the educators in a relatively large school

Low Scores High Scores

3,000 Students’ Test Scores

Norm Group

Figure 13.2 a typical Norm Group

Mean

2% 2% 14%

-2 S.D. +2 S.D.-1 S.D. +1 S.D.

14%

34% 34%

Norm Group

Figure 13.3 a Normally Distributed Set of Norm-Group Scores

M13_POPH0936_10_SE_C13.indd 330M13_POPH0936_10_SE_C13.indd 330 09/11/23 6:10 PM09/11/23 6:10 PM

Individual Student test Interpretation 331

district wished to compare students’ test scores within their district, then it would not take a ton of energy to compute tables of district-level norms and distribute them to district educators. In other words, when reasonably large numbers of students are involved, norm groups can be constructed in almost any setting.

In many instances, local norms are different from national norms because the students in a particular locality are often not representative of the nation’s children as a whole. If there is a difference between local and national norms, then there will be a difference in a student’s percentile scores. A student’s raw score—that is, the number of test items answered correctly—might be equal to the 50th percentile based on national norms, but be equal to the 75th percentile based on local norms. This kind of situation would occur if the students in the local group hadn’t performed as well as students in the nation at large. National and local norms provide decisively different frameworks for interpreting standardized test results. When reporting test scores to parents, make sure you communicate clearly whether a child’s percentiles are based on national or local norms.

It’s also true that some norm groups have been more carefully constituted than others. For example, certain national norm groups are more representative of the nation’s population than are other national norm groups. There are often large differences in the representativeness of norm groups based on such variables as gender, ethnicity, geographic region, and socioeconomic status of the students in the groups. In addition, many standardized tests are re-normed only every 5 to 10 years. It is important to make sure that the normative information on which percentiles are based is representative, current, and clearly communicated to those using a test’s results.

Grade-Equivalent Scores Let’s turn from percentiles to look at grade equivalents, or as they’re often called, grade-equivalent scores. Grade-equivalent scores constitute another effort to pro- vide a relative interpretation of standardized test scores. A grade equivalent is an indicator of a student’s test performance based on grade levels and months of the school year. The purpose of grade equivalents is to transform scores on stan- dardized tests into an index reflecting a student’s grade-level progress in school. A grade-equivalent score is a developmental score in the sense that it represents a continuous range of grade levels.

Let’s look at a grade-equivalent score of 4.5:

A Grade-Equivalent Score Grade 4.5 Month of School Year→ ←

The score consists of the grade, a decimal point, and a number representing months. The number to the left of the decimal point represents the grade level— in this example, the fourth grade. The number to the right of the decimal point represents the month of the school year—in this example, the fifth month of the school year.

M13_POPH0936_10_SE_C13.indd 331M13_POPH0936_10_SE_C13.indd 331 09/11/23 6:10 PM09/11/23 6:10 PM

332 Chapter 13 Making Sense Out of Standardized test Scores

Some test publishers, by using statistical schemes, convert raw scores on stan- dardized tests to grade-equivalent scores. These grade-equivalent scores often appear on students’ individual score reports. Grade-equivalent scores are most appropriate for basic skills areas, such as reading and mathematics, where it can be assumed that the degree of instructional emphasis given to the subject is fairly uniform from grade to grade.

The appeal of grade-equivalent scores is that they appear to be readily inter- pretable to both teachers and parents. However, many teachers and parents actu- ally have an incorrect understanding of what grade-equivalent scores signify. To see why these scores are misunderstood, it’s necessary to understand a bit about where they come from in the first place.

To determine the grade-equivalent scores that will be hooked up with par- ticular raw scores, test developers typically administer the same test to students in several grade levels and then establish a trend line reflecting the raw-score increases at each grade level. Typically, the test developers then estimate—at other points along this trend line—what the grade equivalent for any raw score would be.

Let’s illustrate this point. In Figure 13.4, you will see the respective perfor- mances of students at three grade levels. The same 80-item test has been given to students at all three grade levels: grades 4, 5, and 6. A trend line is then established from the three grades where the test was actually administered. The result of that estimation procedure is seen in Figure 13.5.

In order for these estimated grade-equivalent scores to be accurate, several assumptions must be made. First, it must be assumed that the subject area tested is emphasized equally at each grade level. It must also be assumed that students’ mastery of the tested content increases at a reasonably constant rate at each grade level over an extended period of time. The assumption that the mastery

Average Performance per Grade

R aw

S co

re

2 3 4 5 6 7 8 20

30

40

50

60

70

80

Figure 13.4 Student performances at three Grade Levels

M13_POPH0936_10_SE_C13.indd 332M13_POPH0936_10_SE_C13.indd 332 09/11/23 6:10 PM09/11/23 6:10 PM

Individual Student test Interpretation 333

of the test’s content increases consistently over time is particularly difficult to support in subject areas other than reading and mathematics. And even in those two bread-and-butter subjects, students’ grade-to-grade consistency of increased content mastery is definitely debatable.

The implied precision associated with a grade-equivalent score of 6.2, there- fore, is difficult to defend. A 6.2 grade-equivalent score suggests a degree of accuracy that’s simply not warranted. An unskilled interpreter of the results of a standardized achievement test may think a 6.2 grade-equivalent score indicates the student’s raw score represents a performance equal to that of a sixth-grade stu- dent during the second month of the student’s sixth-grade year. Remember, most grade-equivalent scores are created on the basis of estimation, not real test-score data. Because substantial sampling and estimation errors are apt to be present, grade-equivalent scores should always be taken with several grains of salt.

Now that you understand grade-equivalent scores to be, at best, rough esti- mates, let’s return to the misinterpretations that teachers and parents often make regarding grade-equivalent scores. For example, let’s assume a third-grade student makes a grade-equivalent score of 5.5 in reading. What does this grade-equivalent score mean? Here’s a wrong answer: “The student can do fifth-grade work.” Here’s a really wrong answer: “The student should be promoted to the fifth grade.” The right answer, of course, is that the third-grader understands those reading skills that the test covers about as well as an average fifth-grader does at midyear. A grade-equivalent score should be viewed as the point where a student currently is along a developmental continuum, and definitely not as the grade level in which the student should be placed.

If standardized tests are used in your school district, and if those tests yield grade-equivalent scores, it is important to provide parents with an accurate

Average Performance per Grade

R aw

S co

re

2 3 4 5 6 7 8 20

30

40

50

60

70

80

Figure 13.5 a trend Line Used to estimate average performance of Students at Nontested Grades

M13_POPH0936_10_SE_C13.indd 333M13_POPH0936_10_SE_C13.indd 333 09/11/23 6:10 PM09/11/23 6:10 PM

334 Chapter 13 Making Sense Out of Standardized test Scores

picture of what a grade-equivalent score means. Parents who have not been given an accurate description of such scores frequently think a high grade-equivalent score means their child is capable of doing work at the grade level specified. Some parents even use high grade-equivalent scores as a basis for arguing that their child should be advanced to a higher grade. This is because many parents mis- understand what a grade-equivalent score means, and more than a few parents may already have an inflated estimate of their child’s level of achievement. The high frequency of parental (and teacher) misinterpretations is the chief reason that some districts have, as a matter of policy, eliminated grade-equivalent scores when reporting standardized test results.

Remember, if a fourth-grade student gets a grade-equivalent score of 7.5 in mathematics, it is not accurate to say the fourth-grader is doing well in seventh-grade mathematics. It is more appropriate to say that a grade-equivalent score of 7.5 is an estimate of how an average seventh-grader might have per- formed on the fourth-grader’s mathematics test. Obtaining a 7.5 grade-equivalent score doesn’t mean the fourth-grader has any of the mathematics skills taught in the fifth, sixth, or seventh grade. Typically, those mathematics skills were probably not even measured on the fourth-grade test.

But what happens when grade-equivalent scores are below the actual grade level tested? Let’s say a fifth-grader earns a mathematics grade-equivalent score of 2.5. It doesn’t make much sense to say the fifth-grader is doing fifth-grade mathematics work as well as a second-grader, because second-graders obviously aren’t usually given fifth-grade mathematics assignments. About the best you can

M13_POPH0936_10_SE_C13.indd 334M13_POPH0936_10_SE_C13.indd 334 09/11/23 6:10 PM09/11/23 6:10 PM

Individual Student test Interpretation 335

say is that in mathematics achievement, the fifth-grader appears to be lagging several years behind grade level.

Given the shortcomings of grade-equivalent scores, most measurement spe- cialists are delighted that the use of these scores is diminishing.

Scale Scores Let’s move, now, to the last of our three score-interpretation schemes: scale scores. A scale score constitutes yet another way to give relative meaning to a student’s performances on standardized tests. Scale scores are being employed with increas- ing frequency these days to report the results of national and state-level standard- ized programs.

Although scale scores are sometimes not used in reporting standardized test results to parents, scale-score reporting systems are often employed in describ- ing group test performances at the state, district, and school levels. Because of the statistical properties of scale scores, they can be used to permit longitudinal tracking of students’ progress. Scale scores can also be used to make direct comparisons among classes, schools, or school districts. The statistical advan- tages of scale scores are considerable. Thus, we see scale-score reporting sys- tems used with more frequency in recent years. As a consequence, you need to become familiar with the main features of scale scores, because such scores are likely to be employed when you receive reports of your students’ performances on standardized tests—particularly those performances earned on important, high-stakes tests.

A scale used for reporting test scores typically consists of numbers assigned to students on the basis of their test performances. Higher numbers (higher scale scores) reflect superior levels of achievement or ability. Thus, such a scale might be composed of a set of raw scores where each additional test item that is correctly answered yields 1 more point on the raw-score scale. Raw scores, all by them- selves, however, are difficult to interpret. A student’s score on a raw-score scale provides no idea of the student’s relative performance. Therefore, measurement specialists have devised different sorts of scales for test-interpretation purposes.

Scale scores are converted raw scores that use a new, often arbitrarily chosen scale to represent levels of achievement or ability. Shortly, you’ll be given some examples to help you understand what is meant by converting scores from one scale to another. In essence, a scale-score system is created by devising a brand- new numerical scale that’s often very unlike the original raw-score scale. Stu- dents’ raw scores are then converted to this brand-new scale so that, when score interpretations are to be made, those interpretations rely on the converted scores based on the new scale. Such converted scores are called scale scores.

For example, in Figure 13.6, you see a range of raw-score points from 0 to 40 for a 40-item test. Below the raw-score scale, you see a new, converted scale rang- ing from 500 to 900. For a number of reasons to be described shortly, it is some- times preferable to use a scale-score reporting scheme rather than a raw-score

M13_POPH0936_10_SE_C13.indd 335M13_POPH0936_10_SE_C13.indd 335 09/11/23 6:10 PM09/11/23 6:10 PM

336 Chapter 13 Making Sense Out of Standardized test Scores

reporting scheme. Thus, a student who achieved a raw score of 30 items correct might be assigned a scale score of 800, as shown in Figure 13.7.

One of the reasons why scale scores have become popular in recent years is the necessity to develop several equidifficult forms of the same test. For example, a basic-skills test must sometimes be passed before high school diplomas are awarded to students. Those students who initially fail the test are typically given other opportunities to pass it. The different forms of the test used for such retake purposes should, for the sake of fairness, represent assessment challenges for students that are equivalent to those represented by the initial form of the test. However, because it is next to impossible to create test forms that are absolutely identical in difficulties, scale scores can be used to help solve the problem. Scores on two test forms of differing difficulty levels can be statistically adjusted so that, when placed on a converted-score scale, the new scale scores represent students’ performances as if the two test forms had been completely equidifficult.

Most of the more popular types of scale-score systems are based on what statisticians refer to as item response theory, or IRT. These types of scale-score reporting systems are distinctively different from raw-score reporting systems, because IRT scale-score schemes take into consideration the difficulty and other technical properties of every single item on the test. Thus, some test publishers have produced IRT-based scale scores for their tests ranging from 0 to 1000 across an entire K–12 grade range. For each grade level, there is a different average scale score. For example, the average scale score for third-graders might be 585, and the average scale score for the 10th-graders might be 714.

These IRT-based scale scores can, when constructed with care, yield useful interpretations if one can link them to some notion of relative performance such as

Raw-Score Scale 0 10 20 30 40

Converted-Score Scale 500 600 700 800 900

Figure 13.6 a raw-Score Scale and a Converted-Score Scale

Raw-Score Scale 0 10 20 30 40

Converted-Score Scale 500 600 700 800 900

×

×

Figure 13.7 an Illustration of a raw-Score Conversion to a Scale Score

M13_POPH0936_10_SE_C13.indd 336M13_POPH0936_10_SE_C13.indd 336 09/11/23 6:10 PM09/11/23 6:10 PM

Individual Student test Interpretation 337

percentiles. If average scale scores are provided at different grade levels, this also can aid in the interpretation of scale scores. Without such interpretative assists regarding a scale score’s relative meaning, however, scale scores cannot be mean- ingfully interpreted by either educators or parents.

Item response theory scales are often employed by psychometricians to devise a vertical scale that will allow students’ test performances to be tracked across a number of grade levels. Clearly, especially in an accountability context where educators are often eager to show students’ growth from year to year, such vertical scales have a special appeal. But they also have a serious drawback. In order for a vertical scale to do its job across many grade levels, the test items being used must correlate well with a single trait—for instance, a unitary trait such as “students’ mathematical comprehension.” It is this global unitary trait that permits a vertical scale to do its job properly. However, because the test’s items must be linked to the unitary trait on which the vertical scale is based, those items are unable to yield meaningful diagnostic information to teachers about students’ differing mathematics skills and knowledge. Legitimate diag- nosticity must often be jettisoned if a test’s developers are preoccupied with making all their items link accurately to the unitary trait chosen for their test’s vertical scale.

So, if officials in your school or district are considering adoption of a stan- dardized test whose chief virtue is that it supplies a meaningful vertical scale for cross-grade comparisons of students’ scores, then don’t be fooled, even if the test is accompanied by rapturous marketing assertions about the test’s diagnostic dividends. The vertical scale has been achieved by undermining the precise sort of diagnostic evidence that teachers typically need for sensible instructional deci- sion making.

Errors are often made in interpreting scale scores because teachers assume all scale scores are somehow similar. For example, when the SAT was initially administered almost a century ago, the mean scale score on the verbal section of the Scholastic Assessment Test was 500. This does not signify that the mean score on that test today is 500 or that other tests using scale scores will always have mean scores of 500. Scale-score systems can be constructed so that the mean score is 50, 75, 600, 700, 1000, or any number the scale’s constructor has in mind.

One kind of scale score you may encounter when you work with your stu- dents’ scores on standardized tests is the normal curve equivalent. A normal curve equivalent, usually referred to as an NCE, represents an attempt to use a student’s raw score to arrive at a percentile for a raw score as if the distribution of scores on the test had been a perfectly symmetrical, bell-shaped normal curve. So, one of your students might get an NCE score indicating that the student was perform- ing at or near the top of a norm group’s performance. Sometimes, unfortunately, neither the norm group’s performance nor that of subsequent test-takers is dis- tributed in a normal fashion, so the meaningfulness of the student’s NCE simply evaporates. Normal curve equivalents were originally created in an effort to amal- gamate students’ performances from different standardized tests, perhaps none of

M13_POPH0936_10_SE_C13.indd 337M13_POPH0936_10_SE_C13.indd 337 09/11/23 6:10 PM09/11/23 6:10 PM

338 Chapter 13 Making Sense Out of Standardized test Scores

which actually yielded normal distributions. As might be suspected, NCEs failed to solve this problem satisfactorily.

A final kind of scale score you should know about is called a stanine. Stan- ines are somewhat like normal curve equivalents in the sense that they assume a normal distribution for a set of scores. And this assumption, as you just read, runs counter to reality. But, as you’ll soon see, a stanine score is a more gross score, so it can more easily tolerate a score distribution’s departure from a normal shape.

A stanine scale divides a score distribution into nine segments that, although they are equal along the baseline of a set of scores (actually one-half standard deviation in distance), contain differing proportions of the distribution’s scores. As you can see in Figure 13.8, the fifth stanine is at the center of a score distribu- tion and contains about 20 percent of the scores. A ninth-stanine score, however, will be found among only about 4 percent of the scores.

One advantage of stanines, as noted earlier, is that they are approximate scale scores, and their very inexact nature conveys more clearly to everyone that edu- cational measurement is not a superprecise assessment enterprise. As with most hierarchical classification systems, of course, a student may be almost at the next higher (or lower) stanine, and yet may miss out by only a point or two. Such is the real-world imprecision of educational measurement, even if you’re using a deliberately approximation-engendering scoring system.

Contrasting Common Interpretive Options We’ve now considered several types of score-interpretation schemes, the most frequently used of these being percentiles, grade-equivalent scores, and scale scores. (Stanines and NCEs tend to be used less frequently.) It’s time to review and summarize what’s really important about the three most popular ways of making sense out of standardized test scores.

1 2 3 4 5 6 7 8 9 Low High

4% 12%

20%

17% 17%

12%

7% 7%

4%

Figure 13.8 Stanine Units represented in approximate percentages of the Normal Curve

M13_POPH0936_10_SE_C13.indd 338M13_POPH0936_10_SE_C13.indd 338 09/11/23 6:10 PM09/11/23 6:10 PM

Individual Student test Interpretation 339

We first considered percentiles. A percentile indicates a student’s standing in relationship to that of a norm group. If a student’s test score is equal to the 25th percentile, the student’s performance exceeds the performances of 25 percent of the individuals in the norm group. One advantage of percentiles is that they’re easy to interpret. And, for the most part, people’s interpretations of percentiles are accurate. A disadvantage of percentiles is that the defensibility of the inter- pretation is totally dependent on the nature of the normative data on which the percentiles are based. Unrepresentative or out-of-date norm data yield inaccurate percentile interpretations. As pointed out earlier, because percentile interpreta- tions are so widely used, it’s imperative for teachers to be knowledgeable about such interpretations.

percentiles Advantage: readily interpretable Disadvantage: dependent on quality of norm group

Next, we considered grade-equivalent scores. A grade equivalent indicates the nature of a student’s test performance in terms of grade levels and months. Thus, a grade equivalent of 3.7 indicates that the student’s test score was esti- mated to be the same as the average performance of a third-grader during the seventh month of the school year. One advantage of grade-equivalent scores is that, because they are based on grade levels and months of the school year, they can be readily communicated to parents. A significant disadvantage associated with grade-equivalent scores, however, is that they’re frequently misinterpreted.

Grade equivalents Advantage: readily communicable Disadvantage: often misinterpreted

Finally, scale scores were described. Scale scores are interpreted according to a converted numerical scale that allows us to transform raw scores into more statistically useful scale-score units. A student who gets a raw score of 35 correct out of 50 items, for example, might end up with a converted scale score of 620. An advantage of scale scores is they can be used to create statistically adjusted equidifficult test forms. Scale-score schemes based on item response theory (IRT) do

M13_POPH0936_10_SE_C13.indd 339M13_POPH0936_10_SE_C13.indd 339 09/11/23 6:10 PM09/11/23 6:10 PM

340 Chapter 13 Making Sense Out of Standardized test Scores

this by weighting individual test items differently based on an item’s difficulty and other technical properties in relation to those same properties for all of a test’s items. A disadvantage of scale-score reporting schemes is that, with the possible exception of stanines, they’re almost impossible to interpret all by themselves. Unless we mentally link scale scores to percentiles or average grade-level scale scores, such scores are essentially uninterpretable.

Scale Scores Advantage: useful in equalizing difficulties of different test forms Disadvantage: not easily interpretable

It should be clear to you that each of these three score-interpretation schemes has some potential virtues as well as some potential vices. You probably realize that, even if you understand the general nature of each of these three interpreta- tion schemes, it may be necessary to secure additional technical information to interpret each reporting scheme with confidence. Such information is usually found in the technical manuals that accompany standardized tests. For example, what was the nature of the norm group on which a test’s percentiles were based? What sort of mathematical operations were used in generating a scale-score reporting scheme? Is a score distribution sufficiently normal to warrant the use of NCEs? To interpret a particular standardized test’s results sensibly, teachers sometimes need to do a bit of homework themselves regarding the innards of the reporting scheme being used.

The Instructional Yield from Standardized Achievement Tests Nationally standardized achievement tests, as noted previously, are developed and sold by commercial testing companies. The more tests those companies sell, the more money those companies make. Accordingly, the representatives of com- mercial testing companies usually suggest that their standardized achievement tests will not only yield valid norm-referenced interpretations about the students who take the tests, but will also provide classroom teachers with a galaxy of hopefully useful information for instructional decision making. In many educa- tors’ experience, however, the instructional payoffs of standardized achievement tests are more illusory than real. Frequently, claims for the instructional dividends of such achievement tests reflect the zeal of a testing firm’s sales force, not how well teachers can actually use the results of standardized tests for instructional purposes.

M13_POPH0936_10_SE_C13.indd 340M13_POPH0936_10_SE_C13.indd 340 09/11/23 6:10 PM09/11/23 6:10 PM

Instructionally Diagnostic tests 341

Instructionally Diagnostic Tests Diagnostic test is a label more often uttered than understood. Yes, although the phrase pops up in educators’ conversations every few days, it is a safe bet that most of those who use this phrase have only a murky understanding of what constitutes a genuine diagnostic test. Yet, because today’s teachers are increasingly buffeted by commercial vendors trying to peddle their “instructionally powerful” diagnostic tests, teachers ought to understand whether vendors’ claims are legitimate.

Generally, of course, diagnostic tests are thought to be good things. Most people regard such tests as useful measurement tools—often associated with medical applications, for example, when physicians employ diagnostic tests to pinpoint a patient’s illness. A diagnostic test, according to most dictionaries, is “concerned with identification of the nature of illness or other problems” (Frank R. Abate, Elizabeth Jewel, The New Oxford American Dictionary, Oxford University Press, 2001). When diagnostic testing occurs in the field of education, we usually see such tests used for purposes of either classification or instruction.

Classification-focused diagnostic tests are often employed by educators who are working with atypical students—that is, with students who are particularly gifted or students who have pronounced disabilities. Such tests allow educators to identify with greater accuracy the exact nature of a student’s exceptionality—so that the student can then be classified in a specific category.

Instruction-oriented diagnostic tests are used when teachers attempt to provide particularized instruction for individual students so the teacher’s upcoming instruc- tional activities will better mesh with the precise learning needs of different students. The following analysis will be devoted exclusively to instructionally diagnostic tests.

What Is an Instructionally Diagnostic Test? Because the mission of an instructionally diagnostic test is to help teachers do an effective instructional job with their students, we need to identify how using such a test would enlighten a teacher’s instructional decisions—for example, such deci- sions as when to give or withhold additional or different instruction from which students. Appropriately matching instruction, both its quantity and its type, with students’ current needs constitutes a particularly important element of effective teaching. An instructionally diagnostic test, if properly fashioned, permits teach- ers to identify those students who need more or less instruction, and, if the diag- nostic test is shrewdly constructed, such a test might even help teachers determine what kinds of instruction are likely to succeed with those students.

Here, then, is a definition of an instructionally diagnostic test:

An instructionally diagnostic test is an assessment instrument whose use permits teachers to draw accurate inferences about individual test-takers’ strengths and/or weaknesses with respect to two or more skills or bodies of knowledge—thereby permitting teachers to take more effective next-step instructional actions.

M13_POPH0936_10_SE_C13.indd 341M13_POPH0936_10_SE_C13.indd 341 09/11/23 6:10 PM09/11/23 6:10 PM

342 Chapter 13 Making Sense Out of Standardized test Scores

Several components of this definition can influence the development of instruc- tionally diagnostic tests and, thus, warrant elaboration.

SOLO-STUDENT INFERENCES Let’s begin with the idea that, based on students’ performances on an instruc- tionally diagnostic test, a teacher will be able to draw more valid (or, if you prefer, more accurate) inferences about an individual student’s status. Putting it differently, a good instructionally diagnostic test will help a teacher get a more exact fix on what each of the teacher’s students currently knows or is able to do. Clearly, if a teacher chooses to aggregate the individual test results of multiple students in order to make subgroup-focused inferences, or even inferences focused on the whole class, this is altogether appropriate. However, the preceding definition specifies that an honest-to-goodness instructionally diagnostic test must be capable of yielding valid inferences about the status of individual test-takers.

STRENGTHS AND/OR WEAKNESSES Some words (e.g., however) or phrases (e.g., of course) and some abbreviations (e.g., e.g.) are seen so often in our everyday reading that we tend to pay scant attention to them. One important example of this sort of familiarity occurs with the oft-encountered, paired conjunctions and/or. In the definition given for an instructionally diagnostic test, however, the and/or has an important role to play. It signifies that a student’s performance on an instructionally diagnostic test can provide results permitting a teacher to draw inferences exclusively about the par- ticular student’s strengths, exclusively about the particular student’s weaknesses, or about both the strengths and weaknesses of the particular student. Although, in most settings, teachers are likely to be more interested either in students’ strengths or in their weaknesses, the proposed definition of instructionally diagnostic tests makes it clear that this and/or should be taken seriously.

TWO OR MORE SKILLS OR BODIES OF KNOWLEDGE The strengths and/or weaknesses addressed in an instructionally diagnostic test must be at least two. This is where the diagnosticity of such tests trots onstage. If a test only helps teachers establish a student’s status with respect to one cognitive skill that’s been tested, or if it identifies the student’s mastery level regarding only one body of knowledge, such information might obviously be useful to a teacher. For example, if a fifth-grade teacher uses a teacher-made or commercial test to measure students’ status regarding a single high-level skill in mathematics, results from the test could reveal which students have, and which have not, mastered the specific math skill being assessed. This is definitely useful information. The teacher can rely on it to make better current or future instructional decisions. But a single-focus test is not instructionally diagnostic. It’s not a bad test; it’s just not instructionally diagnostic.

M13_POPH0936_10_SE_C13.indd 342M13_POPH0936_10_SE_C13.indd 342 09/11/23 6:10 PM09/11/23 6:10 PM

Instructionally Diagnostic tests 343

NEXT-STEP INSTRUCTIONAL ACTIONS The final requisite feature of the much-discussed definition stems from the fun- damental orientation of an instructionally diagnostic test—namely, instruction. The definition indicates that, based on the results of such tests, teachers can make next-step instructional decisions, such as what to teach tomorrow or how to teach something next week. Moreover, those next-step instructional decisions are apt to be more effective than would have been the case had a teacher’s decisions not been abetted by information about students’ current status.

What this definitional requirement calls for, in plain language, are test results that can be readily translated into sound next-step pedagogical moves by a teacher. To supply such data, an instructionally diagnostic test must provide its results in an actionable grain size—that is, at a level of generality addressable by routine teacher-chosen instructional actions. Accordingly, if the reports from a supposedly diagnostic test are provided at such a broad grain size that the only reasonable action implication is for the teacher to “make students smarter,” this test would be a diagnostic dud.

I recently authored an essay appearing in an academic journal whose articles are published in both English and Turkish. It was an invited article about the importance of providing actionable reports of students’ performances if any test is to be employed instructionally. Unable to speak or write in Turkish, I relied on collegial translations of my English-language blather. My analysis pleaded with our Turkish colleagues to report formatively oriented test performances of students so that they can be reported as readily usable “right size” information. I hope that this is the main idea of the Turkish version.

How Good Is an Instructionally Diagnostic Test? Once tests have been identified as instructionally diagnostic, they are still apt to differ in their quality, sometimes substantially. The following evaluative criteria can be employed to help determine an instructionally diagnostic test’s merits. A brief description will now be presented of each of the following attributes of an instructionally diagnostic test’s quality: (1) curricular alignment, (2) suitability of item numbers, (3) item quality, and (4) ease of usage.

Many commercial vendors currently realize that “There’s gold in them thar schools.” Some of this “gold” can be seen when districts, and even schools within those districts, plunk down big hunks of their school budgets to purchase instruc- tionally useful tests, particularly instructionally diagnostic tests touted to boost students’ achievement. For this reason, teachers really ought to be a position to offer educational officials advice regarding the wisdom of spending big bucks on those assessment devices being marketed as “instructionally diagnostic tests.” Some of these diagnostic tests will be terrific; some will be terrible. You need to know what to look for when appraising such assessments. The following evalu- ative factors should, it is hoped, be of assistance when you review the virtues of an instructionally diagnostic test.

M13_POPH0936_10_SE_C13.indd 343M13_POPH0936_10_SE_C13.indd 343 09/11/23 6:10 PM09/11/23 6:10 PM

344 Chapter 13 Making Sense Out of Standardized test Scores

CURRICULAR ALIGNMENT The overriding purpose of educational tests is to collect students’ responses in such a way that we can use students’ overt responses to test questions to arrive at valid inferences about those students’ covert status with respect to such vari- ables as students’ knowledge or their cognitive skills. Ideally, we would like instructionally diagnostic tests to measure lofty curricular aims such as students’ attainment of truly high-level cognitive skills or their possession of genuinely significant bodies of knowledge. But the grandeur of the curricular aims that a test sets out to measure is often determined not by the test-makers but, rather, by higher-level educational authorities.

Thus, if the developers of an instructionally diagnostic test have been directed by their superiors to create a test assessing students’ mastery of rather trifling outcomes, and the resultant test performs this measurement job quite successfully, it seems unfair to castigate the test itself because of the paltry curricular dictates over which the test’s developers had no control.

The most important judgment to be made when evaluating a test on the basis of this first evaluative criterion (curricular alignment) hinges on the answer to a question such as: “Will students’ responses to items in this test provide evidence permitting valid inferences to be drawn about a test-taker’s status with respect to each body of knowledge and/or skill being assessed?”

Ideally, because the match between the curricular aims being assessed and the test’s items being used to measure students’ mastery of those aims is so impor- tant, evaluators of diagnostic tests should look for compiled judgmental evidence, perhaps supplied by a specially convened content-judgment committee, regard- ing the degree to which the items employed do, indeed, elicit scores that reflect students’ worthwhile mastery levels.

SUITABILITY OF ITEM NUMBERS This second evaluative criterion for appraising an instructionally diagnostic test is easy to understand, but difficult to formally operationalize. Put simply, a decent instructionally diagnostic test needs to contain enough items dealing with each of the skills and/or bodies of knowledge being measured so that, when we see a stu- dent’s responses to those items, we can identify—with reasonable confidence— how well a student has achieved each of the assessment’s targets.

This is an easily understood feature of diagnostic testing because most of us realize intuitively that we will usually get a more accurate fix on students’ mas- tery levels if we ask those students to respond to more items rather than fewer items. And here, of course, is where sensible, experienced educators can differ. Depending on the nature of the curricular targets being measured by an instruc- tionally diagnostic test, seasoned educators might understandably have different opinions about the number of items needed. For larger grain-size curricular tar- gets, more items would typically be needed than for smaller grain-size curricu- lar targets. What’s being sought from students is not certainty about mastery of what’s being measured in a test. No, to reach absolute certainty, then we might well

M13_POPH0936_10_SE_C13.indd 344M13_POPH0936_10_SE_C13.indd 344 09/11/23 6:10 PM09/11/23 6:10 PM

Instructionally Diagnostic tests 345

be required to measure students’ mastery of the complete universe of items that could conceivably be generated to measure students’ mastery. Such exhaustive testing would constitute an inefficient use of both students’ and teachers’ time.

This second sought-for attribute of an instructionally diagnostic test is far more significant than it might initially appear to be. This is because the numbers of items devoted to measuring a test-taker’s distinctive abilities can sometimes, all by them- selves, allow instructionally diagnostic tests to succeed or not. If there are too few items per assessed attribute, then teachers may have difficulty inferring the test- taker’s status with respect to what’s being measured. (Can a student’s performance on a single long-division item reveal how skilled at long-division the student truly is?) However, if a test becomes too long, because unnecessarily large numbers of items per measured attribute have been employed, then test-takers may become exhausted, or downright annoyed, during the too-long test-taking experience. If test-constructors force students to complete large numbers of items signifying what’s being assessed, teachers will soon chafe against the need to tackle a seemingly endless parade of items. Rarely are teachers happily willing to endure the demand that they complete interminable sets of items. It is all too understandable.

What’s needed, then, as indicated earlier, is a serious effort to provide the users of test results with right-sized results, that is, students’ scores on a sufficient number of items that will help teachers make a defensible decision for the issue at hand without overwhelming those teachers because of an excessive number of items. Right-sizing students’ test results is easier to describe than do.

Suppose, for example, that a teacher is setting out to create an instructionally diagnostic test that will help the teacher identify those subskills that a teacher needs to re-teach again, but differently. The teacher surely knows that one or two items per subskill won’t be sufficient to get this diagnostic task done, but how many items will yield a sub-score that warrants the teacher’s confident accep- tance? Moreover, how many items will be too many to get the job done? It’s tricky.

Fortunately, a teacher doesn’t have to “go it alone” when attempting to arrive at a suitable number of items to which students must respond while the teacher tries to identify the requisite right-sized numbers of items in a test. If possible, simply contact a few trusted colleagues who are familiar with the subject matter under consideration. Just put together a rudimentary little survey that presents a description of the content being assessed along with representative examples of any subcategories of content. The survey’s directions seek your colleagues’ best judgments regarding the lowest number of test items needed and the high- est number of those items to accurately indicate a students’ status with regard to each content-specific set of test items. After reviewing your colleagues’ estimates, it is almost certain that you will then be better positioned to arrive, finally, at a reasonable number of items needed for an accurate interpretation of the content subcategories with which you are dealing.

If you realize how crucial right-sizing is to teachers’ continued use of instruc- tionally diagnostic tests, you will recognize how important it is, over a time span of successive academic years, to avoid getting the right-sizing wrong.

M13_POPH0936_10_SE_C13.indd 345M13_POPH0936_10_SE_C13.indd 345 09/11/23 6:10 PM09/11/23 6:10 PM

346 Chapter 13 Making Sense Out of Standardized test Scores

ITEM QUALITY With few exceptions, educational tests are made up of individual items. Occa- sionally, of course, we encounter one-item tests such as when we ask students to provide a writing sample by composing a single, original essay. But most tests, and especially those tests intended to supply instructionally diagnostic informa- tion, contain multiple items. If a test’s items are good ones, the test is obviously apt to be better than a test composed of shoddy items. This is true whether the test is supposed to serve a classification function, an instructional function, or some other function completely. Better items make better tests. You have already seen in earlier chapters a flock of guidelines regarding how to generate high-quality items for educational tests. Those guidelines surely apply to the items used in instructionally diagnostic tests.

EASE OF USAGE Testing takes time. Teaching takes time. Most teachers, therefore, have very little discretionary time to spend on anything else—including diagnostic test- ing. Accordingly, an effective instructionally diagnostic test, if it is going to be employed by many teachers, must be genuinely easy to use. If an instruction- ally diagnostic test is difficult to use—or time-consuming to use—its easiness evaporates.

If, for example, hand-scoring of students’ responses must be carried out by the teacher personally, then such scoring should require minimal time and very little effort. These days, of course, because of advances in technology, much scor- ing is done electronically. But if hassles arise when using a test, and most certainly when scoring a test, these hassles will diminish the test’s use.

Then there are the results of the test—that is, the numbers or the words that help teachers (and, if the teacher wishes, can help students as well) determine what a student’s strengths and weaknesses actually are. Some time ago, one of our nation’s leading test-development agencies published a test that it argued was instructionally helpful to teachers. The test’s results, however, were reported exclusively on an item-by-item basis, thus requiring the teacher to consider the instructional implications of the test’s results only one item at a time. Absurd!

What this ease-of-usage evaluative criterion addresses, of course, is the inher- ent practicality of the test, meaning its administration, its scoring, its interpretation, and the instructional actionability of its results. And this is when a “satisfactory” diagnostic test can, if carefully conceptualized, become truly “stellar.” A diag- nostic test that’s super will have carved up the content and subskills it measures in right-sized reports so that teachers and students can tackle greater mastery of diagnosed shortcomings directly. The more instructionally addressable are the “chunks” in which the test’s results are reported. the easier it will be for teachers or students to directly ameliorate diagnosed weaknesses.

By employing these four evaluative criteria to judge tests that have been identified as instructionally diagnostic assessments, it is possible to make at least rudimentary qualitative distinctions among such tests.

M13_POPH0936_10_SE_C13.indd 346M13_POPH0936_10_SE_C13.indd 346 09/11/23 6:10 PM09/11/23 6:10 PM

Instructionally Diagnostic tests 347

Looking back, then, at instructionally diagnostic tests, today’s teachers will be urged by commercial testing firms to “buy our nifty diagnostic tests.” Well, as you will see, some of those allegedly diagnostic tests really are nifty-free. Because bona fide diagnostic tests can be remarkably useful to teachers, if you ever have the opportunity to influence a decision regarding the potential purchase of com- mercially created diagnostic tests, be sure you review the potential acquisitions with considerable care. Not all so-called instructionally diagnostic tests really are.

Yet, teachers would be foolish to believe that most standardized tests will provide the most appropriate results from which both teachers and students can arrive at interpretations regarding their next-step instructional moves. Remember, in almost all instances, the major standardized achievement tests that are used in our schools, whether those tests be state or national, were created by psychometric specialists who know oodles about test building but very little about using test results instructionally. Are standardized achievement tests better than no tests at all? Of course, they are. But teachers need to be significantly wary regarding the instructionally useful insights provided by many of the nation’s standardized achievement tests.

But What Does this have to Do with teaching? Depending on the grade levels at which elementary school teachers teach, or the different subjects that secondary school teachers teach, many teachers these days find their attention focused on their students’ scores on standardized achievement tests. That’s because students’ scores on such tests are increasingly used as an indicator of educators’ instructional success. So, if you’re a teacher whose state (or school district) requires that standardized achievement tests be administered at the end of school for a grade level you teach, you can be assured that plenty of attention will be paid to your students’ standardized test performances. In Chapter 15 you’ll learn why this practice is flat-out wrong whenever inappropriate standardized tests are employed.

But even if you’re a teacher whose students are at grade levels or in classes where standardized tests don’t play a major role, you still need to become familiar with the way teachers and parents should interpret such test performances. And that’s because, as a professional, a teacher is supposed to know such things. Parents will sometimes ask

you what is meant by a percentile or an NCE. You definitely don’t want to blubber forth, “Beats me!”

And, of course, if your students are assessed with a standardized achievement test, such as statewide accountability exams, you’ll want to be able to interpret what the students’ scores signify. One reason is that such scores may help you instructionally. Suppose, for example, it is the first week of school and you’re reviewing your new fifth-grade students’ end-of-year fourth-grade test performances. You discover that Martha Childs earned a 94th percentile in mathematics, but only a 43rd percentile in reading. This information should give you some insight into how you’ll approach Martha instructionally.

And if Roy Romera’s fourth-grade test scores gave him a seventh stanine in language arts, but a first stanine in math, perhaps you should not ask Roy to keep track of your students’ lunch money.

The better fix you get on your students’ relative achievement levels, the more appropriately you can teach them. Proper interpretations of standardized test scores will often help you do so.

M13_POPH0936_10_SE_C13.indd 347M13_POPH0936_10_SE_C13.indd 347 09/11/23 6:10 PM09/11/23 6:10 PM

348 Chapter 13 Making Sense Out of Standardized test Scores

If you wish to get a quick fix on the instructional virtues of a standardized achievement test, here’s the most direct way to do it: Simply find out how the test’s results are reported. This lets you know immediately whether the test’s reported results are being doled out at a grain size that has a decent chance of being accu- rate, yet it can also help teachers know what next-step instructional actions to take. If the results are reported at too large a grain size, such as the test-taker “can read” or “can’t read,” then what’s a teacher to do with such an undifferentiated lump of assessment truth? At the other extreme, if the grain size is so small that the teacher becomes overwhelmed with a plethora of tiny status reports about each student, then this too constitutes a dysfunctional reporting structure. To spot a standard- ized test’s instructional utility, simply scurry to see how its results are reported.

The SAT and the ACT: Three-Letter, High-Import Exams Before the arrival of the COVID pandemic, every year, millions of American stu- dents put their futures on the line when they sat down to take the SAT and ACT exams, the nation’s college entrance tests. Because not only students, but also students’ parents, are understandably apprehensive about one or both of these college admission tests, teachers at all grade levels are likely to be questioned about the SAT or the ACT. Therefore, in the next few paragraphs you’ll be receiv- ing brief general descriptions of the nature of both exams, how they came into existence, and what their score reports look like. This descriptive treatment of the two tests will conclude with a candid look at the accuracy with which those tests predict a high school student’s academic success in college.

It is important for you to realize that, as the manuscript of this 10th edition of Classroom Assessment was heading heroically from yours truly to the book’s pub- lisher (Pearson), a host of last-minute potential changes in the ways that the SAT and ACT might operate were being discussed. Thus, if I were once more a high school teacher, I would surely try to bone up on the most recent nature of how these two influential college admission tests currently worked. Your students’ parents, of course, would be expecting you to know at least the chief feature of how the SAT and ACT hum. (Yet, if I were once more a teacher in Oregon’s Heppner High School, there would be no television teaching, no electronic test scoring, nor any Zoom faculty meetings. After all, only a year or two earlier had fire been discovered.)

Accordingly, if you think it likely that you might be receiving questions from your students or, more likely, their parents, please set aside an hour or two so you can have at least a nodding acquaintance with how the purveyors of the SAT and ACT are currently operating. It will be time wisely spent. What you’ll be getting in the next few pages are whittled-down descriptions of these two famous college admissions examinations.

M13_POPH0936_10_SE_C13.indd 348M13_POPH0936_10_SE_C13.indd 348 09/11/23 6:10 PM09/11/23 6:10 PM

the Sat and the aCtT: three-Letterr, high-Import exams 349

The significance of the SAT and ACT was decisively ratcheted up by the enactment of ESSA in December 2015. In this law, the successor to NCLB, it was stipulated that if states wished to employ nationally recognized tests (such as the ACT or SAT) as the assessments they employ to carry out annual accountability determinations, they could do so. Even though neither of these quasi-aptitude tests had been designed for school evaluation, and they were not accompanied by evidence supporting their suitability for this measurement mission, a handful of states have opted to have their high school students complete one of the two tests for accountability purposes. Surely, then, if you happen to be teaching in a state whose officials have opted to use the SAT or ACT to satisfy ESSA, you will want to learn more about your state-chosen high school assessment. Tests that, prior to ESSA, were merely high-stakes tests, had become “high-as-the-sky”-stakes tests.

What we saw occurring in the college admissions test game during the early COVID onslaught was an increasing adherence to a new test-optional college admissions policy. Yet, an ever-expanding fleet of colleges and universities— even when the most virulent phases of the pandemic had subsided—chose to adhere to a test-optional policy in their admissions process. A major claim dur- ing this early pandemic period was that test-optional admissions stimulated a more equitable array of successful college applicants than did the predecessor policy of test-mandatory admissions applications. Currently, a number of major universities have signified their intent to study relevant results of test-optional approaches, then announce admission policies in accord with these new data.

Recent developments regarding admissions exams in a host of vocationally focused fields have indicated that the nationwide dispute about using profession- ally overseen standardized admissions examinations is far from settled. For example, in a November 18, 2022, Reuters report, Karen Sloan described an important deci- sion reached by an arm of the American Bar Association (ABA) responsible for the Law School Aptitude Test. A last-minute revision, however, indicated that the rule change will not go into effect until the fall of 2025—allowing law schools a period to plan for new ways to admit students. The ABA’s Council of the Section of Legal Education and Admissions to the Bar overwhelmingly voted to do away with its testing mandate over the objections of almost 60 law school deans who warned that such a measurement move could harm the goal of diversifying the legal profession.

Given the substantial time and effort expended by the groups responsible for these admissions exams, not just to be admitted to college, but also to a galaxy of professional specializations, it is likely that not only the SAT and ACT will be under continuing scrutiny regarding their impact on the diversity of student bod- ies, but also the degree to which the absence of traditional entrance examinations will impact the quality of those student bodies.

The SAT First administered in 1926, the SAT was initially called the Scholastic Aptitude Test, and its function was to assist admissions officials in a group of elite Northeastern

M13_POPH0936_10_SE_C13.indd 349M13_POPH0936_10_SE_C13.indd 349 09/11/23 6:10 PM09/11/23 6:10 PM

350 Chapter 13 Making Sense Out of Standardized test Scores

universities to determine which applicants should be admitted. The original name of the test really reveals what the test’s developers were up to—namely, they wanted their tests to measure the largely inborn academic aptitudes of high school students. Because, much later, the idea of a test’s measuring innate, genetically determined aptitudes became unappetizing to many educators, the Scholastic Aptitude Test became known only as the SAT. (One wonders whether the devel- opers of the SAT took their cues from the purveyors of Kentucky Fried Chicken who, recognizing the public’s concern about the health risks of fried foods, trun- cated their famous spelled-out label into KFC. It is, of course, difficult to become genuinely negative toward any three-letter label—with the possible exception of the Internal Revenue Service, that is, the IRS.)

The SAT was originally conceived of as a group-administrable intelligence test designed to compare test-takers according to their inherited verbal, quantita- tive, and spatial aptitudes. For many years, the SAT functioned in precisely that

parent talk Suppose that one of your students’ parents, Mr. Lopez, visits your classroom during a back- to-school evening and raises a series of questions about his daughter, Concepción. Most prominently, Mr. Lopez is concerned with Concepción’s scores on the nationally standardized achievement tests administered each spring in your district’s schools. Mr. Lopez is troubled by spring-to-spring changes in Concepción’s percentiles. As he says, “For the last three springs, her percentile scores in language arts have been the 74th, the 83rd, and the 71st. What’s going on here? Is she learning, and then unlearning? What accounts for the differences?”

If I were you, here’s how I’d respond to Mr. Lopez:

“You are certainly wise to look into what appears to be inconsistent measurement on Concepción’s standardized achievement tests. But what you need to understand, Mr. Lopez, is that these sorts of tests, even though they are nationally standardized, developed by reputable testing firms, and widely used, are not all that precise in how they measure students. Many

parents, of course, recognize that their children’s test scores are often reported numerically in the form of national percentiles. But this does not mean that a given child’s national percentile is unerringly accurate.

“The kinds of year-to-year shifting you’ve seen in Concepción’s language arts percentiles is quite normal because a given year’s test result is not unerringly accurate. Assessment specialists even admit the anticipated amount of flip-flop in students’ scores when they talk about each test’s ‘standard error of measurement.’ It’s something like the sampling errors we hear about for national surveys—you know, ‘plus or minus so many percentage points.’ Well, measurement error is akin to sampling error. And let me assure you that Concepción’s future test scores will, quite predictably, vary from spring to spring.

“What we do see in her language arts performances over the last three years is a consistently high (indeed, it’s well above average) achievement level. When her performance is compared to that of a representative, nationally normed sample, Concepción is doing very well, indeed.”

Now, how would you respond to Mr. Lopez?

M13_POPH0936_10_SE_C13.indd 350M13_POPH0936_10_SE_C13.indd 350 09/11/23 6:10 PM09/11/23 6:10 PM

the Sat and the aCtT: three-Letterr, high-Import exams 351

way, although during recent years we have seen substantially less attention given to the inherited nature of what’s being assessed.

The current version of the SAT, introduced in 2016, takes 3 to 4 hours to complete—depending on whether the student completes an optional essay exam. Scores on the SAT range from 400 to 1600, combining results from two 800-point sections, (1) Mathematics and (2) Critical Reading and Writing. Scores are reported in 10-point increments for each of the two sections. Over the years, the specific makeup of the SAT has been periodically modified, so if teachers and students wish to remain abreast of the latest configurations of this exam, they should defi- nitely consult the SAT website.

Although a 2005 addition of an essay-writing task to the SAT attracted sub- stantial media attention, students’ scores on the optional essay section do not contribute heavily to their overall SAT scores—or even to their scores on the Writ- ing section.

The test is still owned by the College Board, the group that originally created the test, but since 1947 the SAT has been developed and administered for the College Board by the Educational Testing Service (ETS). Both organizations are not-for-profit entities. A less demanding version of the test, referred to most often as the PSAT, is administered once each year—typically to high school students during their sophomore or junior year.

Historically, the SAT was originally designed not to be aligned with high school curricula, but in 2016 David Coleman, College Board president since 2012, announced his intention to make the test’s content more closely aligned with what students learned in high school. Commencing with the 2015–16 school year, the College Board announced that it would be teaming with the Khan Academy, a free online education site, to provide free-of-charge SAT preparation for prospec- tive test-takers.

Because of the personal college admission significance to so many young boys and girls, as well as their parents, teachers should be aware of the lengths to which some individuals will go to come up with glistening scores. Although, in years past, most educators presumed that super-significant examinations such as college admission tests were essentially tamper-proof, this is simply not the case. An individual teacher can do little to forestall such security violations, of course, because test-administration operations are typically in the hands of test-company personnel. Yet, to regard high-stakes college admission tests as completely invul- nerable to the threat of sophisticated cheating would be naïve.

The ACT For the past several decades, the SAT was typically used more often by students living in coastal states, whereas the ACT was completed more frequently in the Midwest and South. However, in recent years we have seen increasing num- bers of high school graduates on the East and West coasts completing the ACT. (As noted earlier, the arrival of the multi-year COVID pandemic substantially

M13_POPH0936_10_SE_C13.indd 351M13_POPH0936_10_SE_C13.indd 351 09/11/23 6:10 PM09/11/23 6:10 PM

352 Chapter 13 Making Sense Out of Standardized test Scores

upset traditional patterns in students’ completions of both exams.) Often, of course—and depending on the affluence of the test-taker’s family—students complete both the SAT and the ACT. We now turn to this somewhat more recent addition to the college admissions test options, the ACT.

Although both the SAT and the ACT are typically characterized as aptitude exams because they are employed in an attempt to predict a high school student’s subsequent academic performance in college, the ACT’s lineage makes it a deci- sively different kind of “aptitude” test than the SAT. Following World War II, and the subsequent enactment of the G.I. Bill of Rights (a federal law providing college tuition for returning military personnel), many more American men and women wanted to secure a college education. There was therefore a substantially increased need for college entrance exams.

At the University of Iowa, a gifted professor named E. F. Lindquist believed that the SAT, given its ancestry as an aptitude test designed for students attend- ing elite colleges, was not a suitable exam for many of these post-WWII college applicants. Thus, in 1959, Lindquist and his colleagues established the American College Testing Program as a not-for-profit testing company; later, in 1996, its official name was changed to ACT.

From the very outset, the ACT was intended to be an unabashed achievement test. Architects of the ACT certainly wanted their new test to be predictive of a high school student’s academic success in college, but they conceived of their new test chiefly as a measure of a student’s “educational development.” Developers of the ACT were convinced there would be sufficient variability in test-takers’ performances on this sort of achievement test that students’ ACT performances would do a great job of predicting students’ overall college achievement.

The ACT is made up of separate tests in essentially the same four content areas it initially assessed: English, mathematics, reading, and science. An optional writing test is now available, in which the student has 30 minutes to write an essay supporting one of two positions presented in a writing “prompt,” or a task. Like the SAT, the ACT takes between 3 and 4 hours to complete.

Developers of the ACT routinely solicit content suggestions, in each of the content areas tested, from large samples of secondary teachers and curriculum coordinators, as well as from college professors in the four core subject areas. Sec- ondary school educators are asked to identify the skills and content that students in their school or district have an option to learn by the end of their junior year in high school. College professors are asked what skills or knowledge acquired at the secondary school level are essential for students to be academically success- ful during their first year of college. Based on a combination of what’s taught in secondary schools and what’s considered requisite for first-year success in college, items are then developed for the ACT.

Every ACT item is multiple-choice in nature, and a student gets one point for every correct answer, with no subtraction of points (or partial points) for a wrong answer. Number-correct scores are then converted to a new 1- to 36-point scale (all scores being reported as integers). An average of the four separate-tests’

M13_POPH0936_10_SE_C13.indd 352M13_POPH0936_10_SE_C13.indd 352 09/11/23 6:10 PM09/11/23 6:10 PM

the Sat and the aCtT: three-Letterr, high-Import exams 353

1 to 36 points is computed to arrive at an overall (composite) score (unlike the SAT, where scores from its three sections are added together). The optional writ- ing test, equivalent to the SAT’s essay test, is scored on the basis of a 2- to 12-point scale using the judgments of two scorers, who can each award from 1 to 6 points to a student’s essay.

As with the SAT, there are variations of the ACT available to be used with younger students. EXPLORE is aimed at eighth- and ninth-grade students, and PLAN is available for 10th-graders. More information about these three tests is available on the ACT website. More generally, the websites of both the ACT and the SAT provide up-to-date depictions of how these two exams are currently functioning in test-optional college admissions.

Predictive Accuracy—And Its Implications Because of the important decisions linked to secondary students’ scores on the SAT and the ACT, test-preparation options abound for each test. These test-prep possibilities range from low-cost printed and online Internet materials all the way up to very pricy in-person preparation classes and tutorials. Although there are exceptions, the more money that’s spent on a preparation activity for either of these two tests, the more effective such preparation is apt to be. This is a troubling real- ity, of course, because children from more affluent families have greater access to the more costly preparation options, so affluent students tend to score better than their less affluent classmates who can’t afford high-cost preparation alternatives.

But, as important as it is, let’s put test-prep bias aside for a moment. There is one overridingly important realization all teachers must arrive at regarding the predictive accuracy of the SAT and the ACT. Here it is, all gussied up in italics:

Only about 25 percent of academic success in college is associated with a high school student’s performance on the SAT or ACT.

To calculate how well a high school student’s scores on a college entrance test accurately predict his or her academic success in college, a correlation is computed between high school students’ scores on the SAT (or the ACT) and their subse- quent college grades—for instance, their first-year college grade-point averages. With few exceptions, the resulting correlation coefficients turn out to be approxi- mately .50. A predictive coefficient of .50 is statistically significant and certainly reflects a bona fide relationship between students’ scores on college entrance tests and their subsequent college grades.

However, to determine the practical meaningfulness of that relationship, what we must do is square the .50 coefficient so that it becomes × =.25 (.50 50 .25). This tells us that 25 percent of students’ college grades can be linked to their performances on college admission tests. In other words, fully 75 percent of a student’s college grades are due to factors other than the student’s scores on the SAT or the ACT. Such factors would include a student’s motivation, study habits, and other variables over which a student has substantial control.

M13_POPH0936_10_SE_C13.indd 353M13_POPH0936_10_SE_C13.indd 353 09/11/23 6:10 PM09/11/23 6:10 PM

354 Chapter 13 Making Sense Out of Standardized test Scores

Even if you are a numerical ninny, you’ll realize that a student’s college grades are three times more dependent on nontest factors than they are on test factors— that is, 75 percent (test-unrelated) versus 25 percent (test-related). And, yet, we find hoards of students who have earned only “so-so scores” on the SAT or the ACT simply writing themselves off as “not smart enough” to succeed in college. College success, and success in life, is unarguably dependent on much more than a test score. Unfortunately, because the nation’s educators are often unfamiliar with the predictive imprecision of college entrance tests, they don’t do enough to dissuade students from prematurely foreclosing their college aspirations.

Over the years, a student’s SAT or ACT scores have become, to many and to students themselves, an unalterable definition of one’s ability. That’s simply not true. And teachers need not only to know it, but also to make sure their students (and their students’ parents) know it too.

The SAT and the ACT help college officials distinguish among applicants in the common fixed-quota setting where there are more applicants for admission to a college than there are openings at that college. But the general predictive power of the SAT and the ACT doesn’t transform those exams into definitively accurate assessment tools for any particular student. Teachers at all levels need to help students recognize what the SAT and the ACT exams can do—and what they can’t.

Of Belated Realizations At this point in your reading of this book, you’ve already bumped into the most important factors we use when distinguishing between educational tests that are woeful and those that are wondrous. You’ve seen that assessment guidelines called Standards for Educational and Psychological Testing are issued every decade or so, and that the most recent of those guidelines, the 2014 Standards, stress the need for significant educational tests to be accompanied by ample evidence related to assessment validity, assessment reliability, and assessment fairness. You’ve also learned that educational tests, particularly the significant standardized tests we employ to make high-stakes decisions about students and the educators who teach them, are typically constructed by nonprofit or profit-making groups, and that such tests often play a prominent role in reaching employment decisions about or evaluation judgments about instructional programs.

In addition, you’ve discovered that collections of test-construction, test-improvement, and test-reporting guidelines have been assembled over the years by experienced item-writers. In short, you’ve seen that those of us who are in any way involved in the building or bettering of high-stakes educational tests—whether for use in federal, state, or local settings—now have on hand the procedures to build improved educational tests and to collect compelling evi- dence regarding the caliber of those tests. The more significant the consequences of using a specific test currently under consideration, the more attention that should be given to documenting the construction and evaluation of the test.

M13_POPH0936_10_SE_C13.indd 354M13_POPH0936_10_SE_C13.indd 354 09/11/23 6:10 PM09/11/23 6:10 PM

the Sat and the aCtT: three-Letterr, high-Import exams 355

Often, for high-visibility standardized tests, a technical manual is made avail- able by the test’s developers, describing results of the test’s initial and subse- quent uses. Because most teachers don’t have the discretionary hours to prepare full-blown technical reports for their own teacher-made classroom tests, less detailed summaries of such tests’ performances ordinarily suffice.

All of this book’s aforementioned factors for building and using an educa- tional test should be considered, of course, by teachers who are creating their own classroom tests or are employing tests built by others. Builders or users of high-quality educational tests can, clearly, collect the evidence needed to make determinations regarding a test’s quality. Happily, excellent tests used properly usually lead to defensible educational decisions. Shoddy tests, just as unhappily, frequently lead to shoddy educational decisions.

Accordingly, now is the time for you to arrive at an important realization about the use of today’s educational tests. This realization will be triggered by your thoughtful ruminations regarding the following crucial question: Who makes sure that an educational test’s quality is adequate for its intended use?

It is with regret that the answer to the previous paragraph’s crucial query turns out to be a big, fat No One! Although we might wish that there was some agency—governmental or nongovernmental—that could routinely swoop in to identify deficient high-stakes educational tests, then require that those deficits be remedied, such organizations simply don’t exist. Although we all hope that those who take part in the building or improving of educational tests adhere scrupulously to the recommendations contained in the latest version of the Stan- dards, if a test is created that runs directly counter to those recommendations, you truly need to know that no individual or group has been authorized to “fix things.”

Those of us who live in the United States, over the years, have become accus- tomed to a host of federal, state, and local laws or regulations that, if not adhered to by citizens, can lead to meaningful financial penalties. Most American drivers, for example, don’t want to be given a speeding ticket that signifies a violation of an existing legal regulation and, therefore, requires an offender to pay a monetary fine—sometimes a whopper one. The prospect of being obliged to pay such pen- alties can, in most settings, meaningfully deter a would-be lawbreaker who gets behind an auto’s steering wheel.

But with respect to violations of test-building recommendations—even if such guidelines are provided by esteemed assessment experts—there are no mea- surement sheriffs who can issue significant penalties to the generators of truly ter- rible tests. No, we rely on the integrity of test developers and test users to provide honest and sufficient information so that others can thereafter accurately judge the suitability of educational tests for their intended uses. And it is for this reason that we want those who participate in the construction, evaluation, or usage of educa- tional tests to provide the world with clear, readily comprehensible information about the quality of any significant educational test. Because a substantial number

M13_POPH0936_10_SE_C13.indd 355M13_POPH0936_10_SE_C13.indd 355 09/11/23 6:10 PM09/11/23 6:10 PM

356 Chapter 13 Making Sense Out of Standardized test Scores

of today’s standardized educational tests are now administered each year to so many test-takers, considerable dollars are definitely in play when such tests are adopted. The lure of profits from such test adoptions are, regrettably, a temptation to many of those involved in high-stakes testing.

Indeed, one prominent reason that today’s educators should acquire at least a basic dose of assessment literacy is so that—on behalf of their students—they can identify deficits in commercially available or locally constructed tests. Such shortcomings, then, can be brought to the attention of those who have been autho- rized to select high-quality tests.

For teachers, or those preparing to become teachers, a particularly illuminat- ing example can be found regarding a recently contested dispute regarding the quality of a portfolio-assessment standardized test, the edTPA, an examination widely used nationally for teacher licensure. Reported by Gitomer, Martinez, and Battey (2021) in the widely read Phi Delta Kappan, the edTPA was devel- oped at a major American university and introduced nationally in 2018. This licensure-related examination is currently required of licensed teachers in about 20 states and approved in a similar number of additional states as part of those state’s preservice teachers’ initial certification. Given that tens of thousands of candidates for this important examination typically expend substantial hours pre- paring themselves for the examination, and that in 2021 a test-required examinee was required to pay $300 to take the exam, it is certain that this teacher-licensure test should be characterized as a “high-stakes” test.

During the spring 2018 national meeting of the National Council on Edu- cational Measurement, Gitomer and Martinez registered concerns regarding what they described, based on materials supplied by the test’s publishers, as reliability rates that seemed “implausibly high” and error rates that seemed “implausibly low.” In a subsequent December 2019 article in the American Educational Research Journal, Gitomer et al. (2019) argued that the reliability- determination procedures being used “were, at best, woefully inappropriate and, at worst, fabricated to convey the misleading impression that its scores are more reliable and precise than they truly are.” Thereupon, these critics called for a moratorium on using the test’s scores “for high-stakes decisions about teacher licensure.”

Although their 2019 castigation of a widely used teacher licensure test had been published in one of the nation’s more prestigious research journals, no subsequent criticisms of that test’s technical merits or demerits appeared in response to the Gitomer et al. critiques. The developers of the licensure exam under attack, however, described the naysaying of Gitomer et al. (2021) as “sim- ple matters of reasonable professional disagreement,” rather than as the more fundamental issues of potential malpractice those critics had claimed.

Although these critics of the edTPA had foreseen the emergence of collegial support from other researchers, heightened interest from colleagues, or collab- orative concerns from teacher education organizations, none were seen. Upon looking back at their dispute regarding the merits of the edTAP, Gitomer and

M13_POPH0936_10_SE_C13.indd 356M13_POPH0936_10_SE_C13.indd 356 09/11/23 6:10 PM09/11/23 6:10 PM

What Do Classroom teachers really Need to Know about Interpreting Standardized test Scores? 357

his collaborators conclude their Phi Delta Kappan analysis (2012) with the fol- lowing contention: “But no matter where one comes down on this issue, there is no place for high-stakes assessments that do not satisfy minimum standards of measurement.”

Against the backdrop of this abbreviated illustration of the conflict between creators of a national teacher-licensure test and critics of that test’s technical ade- quacy, we see again why assessment-literate educators can have such an impor- tant role to play with respect to the use of high-stakes educational tests. One or two isolated complaints may, as we saw in the case of the edTAP, have scant impact on real-world assessment practices. Yet, one or two dozen coordinated complaints, on the other hand, can. In view of your own increasing level of assessment literacy, you can become one of those more numerous and more influential critics.

What’s most salient for you to consider at this point in your ever-increasing conversance with the fundamentals of educational assessment is the reality that was all too apparent in the case of the edTAP’s adequacy, namely, is the test under consideration of sufficient quality to be employed? The overly general answer to this overly general question is straightforward. Only when scrutiny of a test’s quality is provided in terms of the pivotal evidence of the most content treated in this book should an under-consideration test be regarded as having passed muster. Quite possibly, you could play an important role in reviewing the worth of any test you contemplate using. The more important are the stakes dependent on the test’s results, the greater the level of scrutiny that should be given to the test’s quality.

What Do Classroom Teachers Really Need to Know About Interpreting Standardized Test Scores? Because the results of standardized tests are almost certain to come into your professional life, you need to have some understanding of how to interpret those results. The most common way of describing a set of scores is to provide an index of the score distribution’s central tendency (usually the mean or median) and an index of the score distribution’s variability (usually the standard deviation or the range). There will probably be instances in which you must either describe the performances of your students in this manner or interpret the meaning of these performance descriptions for other groups of students (such as all students in your school or your district).

To make accurate interpretations of standardized test scores for your own students, and to help parents understand how to make sense of their child’s stan- dardized test scores, you really need to comprehend the basic meaning of percen- tiles, grade-equivalent scores, and scale scores. You also ought to understand the

M13_POPH0936_10_SE_C13.indd 357M13_POPH0936_10_SE_C13.indd 357 09/11/23 6:10 PM09/11/23 6:10 PM

358 Chapter 13 Making Sense Out of Standardized test Scores

major advantage and disadvantage of each of those three score-interpretation schemes. You’ll then be in a position to help parents make sense of their children’s performances on standardized tests.

Two of the more influential tests many students will ever take are the SAT and the ACT, the most widely used college entrance tests. Classroom teachers need to know that, because of their differing lineages, the SAT and the ACT rep- resent somewhat overlapping but fundamentally different assessment strategies. Whereas the SAT’s focus is more heavily on assessing students’ academic apti- tudes, the ACT’s focus is on measuring the kinds of skills and knowledge that, having been previously learned, are necessary in college. Although both tests are predictive of a student’s college grades, only about 25 percent of students’ college grades are associated with students’ SAT or ACT scores.

Chapter Summary Standardized tests were defined as assessment instruments administered, scored, and inter- preted in a standard, predetermined manner. Standardized tests are used to assess students’ achievement and aptitude. Tests of student achievement are developed by large testing companies, as well as by state departments of education (usually to satisfy a legislatively imposed educational accountability require- ment). Although most standardized tests feature the use of selected-response items, many devel- opers of such tests attempt to incorporate modest numbers of constructed-response items in their assessment devices.

We saw that a distribution of test scores can be described using two indices of central tendency— the mean and the median—as well as two indices of variability—the range and the standard deviation.

Several ways of interpreting the results of standardized tests were discussed: percentiles, grade-equivalent scores, scale scores, stanines, and normal curve equivalents (NCEs). The nature of each of these interpretation procedures, as well as the strengths and weaknesses of the

three most popular interpretation procedures, was recounted.

The contribution of instructionally diag- nostic tests was described. Factors to consider were presented to help teachers determine whether a test so marketed is indeed instruc- tionally diagnostic, and if so, how to gauge its quality.

The SAT and the ACT are, by far, America’s most widely used college entrance tests. Because of its history, the SAT (originally known as the Scholastic Aptitude Test) aims to assess students’ academic aptitudes—for example, a student’s innate verbal, quantitative, and spatial aptitudes. The ACT attempts to function as an achievement test that, although it is predictive of students’ college success, measures high school students’ mastery of the skills and knowledge needed in college. Although predictive criterion-related validity coefficients of approximately .50 indicate that students’ SAT and ACT scores are certainly correlated with students’ college grades, fully 75 percent of a student’s academic success in col- lege is linked to factors other than scores on either of these two tests.

M13_POPH0936_10_SE_C13.indd 358M13_POPH0936_10_SE_C13.indd 358 09/11/23 6:10 PM09/11/23 6:10 PM

references 359

References Abate, F. R., & Jewell, E. (2001). The new Oxford

American dictionary. Oxford University Press. Barnum, M. (2021, February 24). This year’s

state test results will be tough to make sense of, experts warn. Chalkbeat. Retrieved September 20, 2022, from https://www. chalkbeat.org/2021/2/24/22299804/ schools-testing-covid-results-accuracy

Brookhart, S. M., & Nitko, A. J. (2018). Educational assessment of students (8th ed.). Pearson.

Gewertz, C. (2021, October 21). State test results are in. Are they useless? Education Week, 6–7. Education Week.

Gitomer, D. H., Martínez, J. F., Battey, D., & Hyland, N. E. (2019, December). Assessing the assessment: Evidence of reliability and validity in the edTPA, American Educational Research Journal, 58(1): 3–31.

Gitomer, D. H., Martínez, J. F., & Battey, D. (2021, March). Who’s assessing the assessment? The cautionary tale of the edTPA, Phi Delta Kappan, 102(6): 38–43.

Goldhaber, D., & Özek, U. (2019). How much should we rely on student test achievement as a measure of success? Educational Researcher, 48(7): 479–483. https://doi. org/10.3102/0013189X19874061

Mervosh, S., & Wu, A. (2022, October 25). Math scores fell in nearly every state, and reading

dipped on national exam, New York Times. https://www.nytimes.com/2022/10/24/ us/ math-reading-scores-pandemic.html

Miller, M. D., & Linn, R. (2013). Measurement and assessment in teaching (11th ed.). Pearson.

Popham, W. J. (2022). Formative assessment: Stalled by too few right-size reports, International Journal of Assessment Tools in Education, 9 (Special Issue): 1–5. https://doi. org/10.21449/ijate.1196705

Robinson, E. (2022, October 24). Plummeting U.S. test scores aren’t a red state vs. blue state thing, Washington Post. https://www.washingtonpost.com/ opinions/2022/10/24/ test-scores-plummet- red-and-blue-states/

Sloan, K. (2022, November 18). ABA votes to end law schools’ LSAT requirement, but not until 2025. Reuters. https://www.reuters. com/legal/legalindustry/aba-votes- end-law-schools-lsat-requirement-not- until-2025-2022-11-18/

Starr, J. P. (2020, February 24). Standardized testing: Making the best of it. Kappan. https:// kappanonline.org/ standardized-testing- making-the-best-of-it/

Waugh, C. K., & Gronlund, N. (2012). Assessment of student achievement (11th ed.). Pearson.

M13_POPH0936_10_SE_C13.indd 359M13_POPH0936_10_SE_C13.indd 359 09/11/23 6:10 PM09/11/23 6:10 PM

360 Chapter 13 Making Sense Out of Standardized test Scores

A Testing Takeaway

Score Reporting: Where Recipients Must Rule* W. James Popham, University of California, Los Angeles

If anyone wants to get a good idea of what a standardized educational test is measuring, one of the best ways to do so is to consult the score report describing students’ test performances. In many instances, different versions of these reports are created for different recipients. Such audience-tailored score reports might be designed for teachers, for students’ parents, and for educational policymakers such as school boards and state legislators.

But no matter who the recipients for a standardized test’s score report are, one essential attribute of such reports is that they should be easy to comprehend. If the users of a score report can’t readily make sense of the report’s information, then why test students at all?

A second essential feature of a standardized test’s score report is actionability. We shouldn’t be testing students merely because the results are “interesting.” Instead, a standardized test’s score report should help inform decisions about what should come next educationally.

Accordingly, even though standardized educational tests can be employed for several important uses—such as evaluation of instructional quality, improvement of instruction, and prediction of test-takers’ future success—those tests’ score reports should all embody the following two attributes:

• Clarity. Descriptions of students’ results should be easily understood. Whether a standard- ized test’s score report is provided for a single student, a schoolful of students, or an entire state’s students, recipients of that report must be able, without difficulty, to easily grasp what’s being reported.

• Actionability. Having become aware of what’s being reported, the recipients of a standard- ized test’s score report must then be able to use those reported results to select the actions for students to take next.

These two requirements will vary depending on the intended purpose of a standardized test. For example, if a standardized test’s results are intended to help teachers improve their instruction, then results should be given as right-sized reports judged to be optimal for teachers’ instructional planning. Test results aimed at modifying a state’s major curricular emphases should be reported at much broader grain sizes.

The most important realization in judging the quality of a standardized test’s score report is that it’s less important to satisfy the preferences of the measurement experts who design the report than to satisfy the needs of users of the score reports. If a standardized test’s score report fails to satisfy report users, recipients of the report should band together and raise such a ruckus that the report will definitely be improved.

*From Chapter 13 of Classroom Assessment: What Teachers Need to Knowr, 10th ed.r, by W. James popham. Copyright 2022 by pearsonr, which hereby grants permission for the reproduction and distribution of this Testing Takeawayr, with proper attributionr, for the intent of increasing assessment literacy. a digitally shareable version is available from httpsT://www.pearson.com/store/en-us/pearsonplus/login.

M13_POPH0936_10_SE_C13.indd 360M13_POPH0936_10_SE_C13.indd 360 09/11/23 6:10 PM09/11/23 6:10 PM