EDU 530 Week 1 Discussion

162

Chapter 6

Selected-Response Tests

Chief Chapter Outcome

The ability to accurately employ professionally accepted item-writing guidelines, both general and item-type specific, when constructing selected-response items or evaluating those items constructed by others

Learning Objectives

6.1 Using the “Five General Item-Writing Precepts” found in this chapter, identify and explain common characteristics of poorly constructed selected-response test items.

6.2 Differentiate between the four different varieties of selected-response test items (binary-choice items, multiple binary-choice items, multiple-choice items, matching items) and be able to create an example of each.

In this and the following four chapters, you will learn how to construct almost a dozen different kinds of test items you might wish to use for your own class- room assessments. As suggested in the preceding chapters, you really need to choose item types that mesh properly with the inferences you want to make about students—and to be sure those inferences are directly linked to the educational decisions you need to make. Just as the child who’s convinced that vanilla is ice cream’s only flavor won’t benefit from a 36-flavor ice cream emporium, the more item types you know about, the more appropriate your selection of item types will be. So, in this chapter and the next four, you’ll be learning about blackberry-ripple exams and mocha-mango assessment devices.

Realistically, what should you expect after wading through the exposition about item construction contained in the upcoming text? Unless you’re a remarkably quick study, you’ll probably finish this material and not be instantly transformed

M06_POPH0936_10_SE_C06.indd 162M06_POPH0936_10_SE_C06.indd 162 09/11/23 6:23 PM09/11/23 6:23 PM

Expanding Electronic Options 163

into a consummately skilled test constructor. It takes more than reading a brief explanation to turn someone into a capable item developer. But, just as the journey of a thousand miles begins with a single step, you’ll have initiated a tentative trot toward the Test Construction Hall of Fame. You’ll have learned the essentials of how to construct the most common kinds of classroom assessments.

What you’ll need, after having completed your upcoming, fun-filled study of test construction is tons of practice in churning out classroom assessment devices. And, if you remain in teaching for a while, such practice opportunities will surely come your way. Ideally, you’ll be able to get some feedback about the quality of your classroom assessment procedures from a supervisor or colleague who is experienced and conversant with educational measurement. If a competent cohort critiques your recent test construction efforts, you’ll profit by being able to make needed modifications in how you create your classroom assessment instruments.

Expanding Electronic Options Once upon a time, when teachers churned out all their own classroom tests, about the only approach available was reliance on paper in order to present items that, back then, students responded to using pencils or pens. Oh, if you head back far enough in history, you might find Egyptian teachers relying on papyrus or, perhaps, pre-history teachers dishing up pop quizzes on tree bark.

But we have evolved (some more comfortably than others) into a technological age in which whole classrooms full of students possess laptop computers, elec- tronic tablets, or super-smart cell phones that can be employed during instruction and assessment. Accordingly, because the availability of such electronically provided assessment options depends almost totally on what’s available for use in a particu- lar district or school, some of the test construction guidelines you will encounter here may need to be massaged because of electronic limitations in the way a test’s items can be written. To illustrate, you’ll learn how to create “matching” items for a classroom assessment. Well, one of the recommendations to teachers who use such items is that they put everything for a given item on a single page (of paper)—so that the students need not flip back and forth between pages when selecting their answers. But what if the electronic devices that a teacher’s students have been given do not provide sufficient room to follow this all-on-one-page guideline?

Well, in that situation it makes sense for a teacher to arrive at the most reason- able/sensible solution possible. Thus, the test construction guidelines you’ll encounter from here on in this book will be couched almost always in terms suitable for paper- presented tests. If you must create classroom assessments using electronic options that fail to permit implementation of the guidelines presented, just do the best job you can in adapting a guideline to the electronic possibilities at hand. Happily, the use of electronic hardware will typically expand, not truncate, your assessment options.

Before departing from our consideration of emerging digital issues concern- ing educators, a traditional warning is surely warranted. Although the mission and the mechanics of educational testing are generally understood by many

M06_POPH0936_10_SE_C06.indd 163M06_POPH0936_10_SE_C06.indd 163 09/11/23 6:23 PM09/11/23 6:23 PM

164 ChaptEr 6 Selected-response tests

educators, the arrival of brand-new digitally based assessment procedures is apt to baffle today’s educators who fail to keep up with evolving versions of modern educational measurement. New ways of testing students and more efficient ways of doing so suggest that today’s teachers simply must keep up with innovative test- ing procedures heretofore unimagined.

This advice was confirmed in an April 12, 2022, Education Week interview with Sal Khan, founder of the nonprofit Khan Academy, which now counts 137 million users in 190 nations. Kahn was asked how best to employ new technology and soft- ware tools to close the learning gaps that emerged during the COVID-19 pandemic. He remarked, “I think that’s going to be especially important because traditional testing regimes have been broken. And it’s unclear what they’re going back to.”

Because of pandemic-induced limitations on large crowds, such as those rou- tinely seen over the years when students were obliged to complete high-stakes examinations, several firms have been exploring the virtues of providing custom- ers with digitalization software that can track a test-taker’s eye movements and even sobbing during difficult exams. Although the bulk of these development efforts have been aimed at college-level students, it is almost certain that experi- mental electronic proctoring systems will soon be aimed at lower grade levels.

A May 27, 2022, New York Times report by Kashmir Hill makes clear that the COVID-19 pandemic, because of its contagion perils, created “a boom time for companies that remotely monitor test-takers.” Suddenly, “millions of people were forced to take bar exams, tests, and quizzes alone at home on their laptops.” Given the huge number of potential customers in the nation’s K–12 schools, is not a shift of digital proctoring companies into this enormous market a flat-out certainty?

Ten (Divided by Two) Overall Item-Writing Precepts As you can discern from its title, this text is going to describe how to construct selected-response sorts of test items. You’ll learn how to create four different varieties of selected-response test items—namely, binary-choice items, multiple binary-choice items, multiple-choice items, and matching items. All four of these selected-response kinds of items can be used effectively by teachers to derive defensible inferences about students’ cognitive status—that is, the knowledge and skills that teachers typically try to promote in their students.

But no matter whether you’re developing selected-response or constructed-response test items, there are several general guidelines that, if adhered to, will lead to better assessment procedures. Because many ancient sets of precepts have been articulated in a fairly stern “Thou shall not” fashion, and have proved successful in shaping many folks’ behavior through the decades, we will now dish out five general item-writing commandments structured along the same lines. Following these precepts might not get you into heaven, but it will make your assessment schemes slightly more divine. All five item-writing

M06_POPH0936_10_SE_C06.indd 164M06_POPH0936_10_SE_C06.indd 164 09/11/23 6:23 PM09/11/23 6:23 PM

ten (Divided by two) Overall Item-Writing precepts 165

precepts are presented in a box below. A subsequent discussion of each precept will help you understand how to adhere to the five item-writing mandates being discussed. It will probably help if you refer to each of the following item-writing precepts (guidelines) before reading the discussion of that particular precept. Surely, no one would be opposed to your doing just a bit of cribbing!

Opaque Directions Our first item-writing precept deals with a topic most teachers haven’t thought seriously about—the directions for their classroom tests. Teachers who have been laboring to create a collection of test items typically know the innards of those items very well. Thus, because of the teacher’s intimate knowledge not only of the items, but also of how students are supposed to deal with those items, it is often the case that only sketchy directions are provided to students regarding how to respond to a test’s items. Yet, of course, unclear test-taking directions can result in confused test-takers. And the responses of confused test-takers don’t lead to very accurate inferences about those test-takers.

Flawed test directions are particularly problematic when students are being introduced to assessment formats with which they’re not very familiar, such as the performance tests to be described in Chapter 8 or the multiple binary-choice tests to be discussed later in this chapter. It is useful to create directions for students early in the game when you’re developing an assessment instrument. When gener- ated as a last-minute afterthought, test directions typically turn out to be tawdry.

Ambiguous Statements The second item-writing precept deals with ambiguity. In all kinds of classroom assessments, ambiguous writing is to be avoided. If your students aren’t really sure about what you mean in the tasks you present to them, the students are apt to misinterpret what you’re saying and, as a consequence, come up with incor- rect responses, even though they might really know how to respond correctly. For example, sentences in which pronouns are used can fail to make it clear to which individual or individuals a pronoun refers. Suppose that, in a true–false test item, you asked your students to indicate whether the following statement

Five General Item-Writing precepts 1. Thou shalt not provide opaque directions to students regarding how to respond. 2. Thou shalt not employ ambiguous statements in your assessment items. 3. Thou shalt not provide students with unintentional clues regarding appropriate

responses. 4. Thou shalt not employ complex syntax in your assessment items. 5. Thou shalt not use vocabulary that is more advanced than required.

M06_POPH0936_10_SE_C06.indd 165M06_POPH0936_10_SE_C06.indd 165 09/11/23 6:23 PM09/11/23 6:23 PM

166 ChaptEr 6 Selected-response tests

was true or false: “Leaders of developing nations have tended to distrust leaders of developed nations due to their imperialistic tendencies.” Because it is unclear whether the pronoun their refers to the “leaders of developing nations” or to the “leaders of developed nations,” and because the truth or falsity of the statement depends on the pronoun’s referent, students are likely to be confused.

Because you will typically be writing your own assessment items, you will know what you mean. At least you ought to. However, try to slide yourself, at least figuratively, into the shoes of your students. Reread your assessment items from the perspective of the students, and then modify any statements apt to be even a mite ambiguous for those less well-informed students.

Unintended Clues The third of our item-writing precepts calls for you to intentionally avoid some- thing unintentional. (Well, nobody said that following these assessment precepts was going to be easy!) What this precept is trying to sensitize you to is the tendency of test-development novices to inadvertently provide clues to students about appropriate responses. As a consequence, students come up with correct responses even if they don’t possess the knowledge or skill being assessed.

For example, inexperienced item-writers often tend to make the correct answer to multiple-choice items twice as long as the incorrect answers. Even the most confused students will often opt for the lengthy response; they get so many more words for their choice. As another example of how inexperienced item-writers unintentionally dispense clues, absolute qualifiers such as never and always are sometimes used for the

M06_POPH0936_10_SE_C06.indd 166M06_POPH0936_10_SE_C06.indd 166 09/11/23 6:23 PM09/11/23 6:23 PM

ten (Divided by two) Overall Item-Writing precepts 167

Computer-adaptive assessment: pros and Cons Large-scale assessments, such as statewide accountability tests or nationally standardized achievement tests, are definitely different from the classroom tests that teachers might, during a dreary weekend, whip up for their students. Despite those differences, the assessment tactics used in large-scale tests should not be totally unknown to teachers. After all, parents of a teacher’s students might occasionally toss out questions at a teacher about such tests, and what teacher wants to be seen, when it comes to educational testing, as a no-knowledge ninny?

One of the increasingly prevalent variations of standardized achievement testing encountered in our schools is known as computer-adaptive assessment. In some instances, computer- adaptive testing is employed as a state’s annual, large-scale accountability assessment. In other instances, commercial vendors offer computer- adaptive tests to cover shorter segments of instruction, such as two or three months. School districts typically purchase such shorter-duration tests in an attempt to assist classroom teachers in adjusting their instructional activities to the progress of their students. In general, these more instructionally oriented tests are known as interim assessments, and we will consider such assessments more deeply later (in Chapter 12).

Given the near certainty that students in many states will be tested via computer-adaptive assessments, a brief description of this distinctive assessment approach is in order.

Not all assessments involving computers, however, are computer-adaptive. Computer-based assessments rely on computers to deliver test items to students. Moreover, students respond to these computer-transmitted items by using a computer. In many instances, immediate scoring of students’ responses is possible. This form of computer-abetted assessment is becoming more and more popular as (1) schools acquire enough computers to make the approach practicable and (2) states and school districts secure sufficient “bandwidth” (whatever that is!) to transmit tests and receive students’ responses electronically. Computer-based assessments, as you can see, rely on computers only as delivery and retrieval mechanisms. Computer-adaptive assessment is something quite different.

Here’s a shorthand version of how computer- adaptive assessment is usually described. Notice, incidentally, the key term adaptive in its name. That word is your key to understanding how this approach to educational assessment is supposed to function. As a student takes this kind of test, the student is given items of known difficulty levels. Then, based

false items in a true–false test. Because even uninformed students know there are few absolutes in this world, they gleefully (and often unthinkingly) indicate such items are false. One of the most blatant examples of giving unintended clues occurs when writers of multiple-choice test items initiate those items with incomplete statements such as “The bird in the story was an . . .” and then offer answer options in which only the correct answer begins with a vowel. For instance, even though you had never read the story referred to in the previous incomplete statement, if you encountered the following four response options, it’s almost certain that you’d know the correct answer: A. Falcon, B. Hawk, C. Robin, D. Owl. The article an gives the game away.

Unintended clues are seen more frequently with selected-response items than with constructed-response items, but even in supplying background information to students for complicated constructed-response items, the teacher must be wary of unintentionally pointing truly unknowledgeable students deftly down a trail to the correct response.

(Continued)

M06_POPH0936_10_SE_C06.indd 167M06_POPH0936_10_SE_C06.indd 167 09/11/23 6:23 PM09/11/23 6:23 PM

168 ChaptEr 6 Selected-response tests

on the student’s responses to those initial items, an all-knowing computer supplies new items that are tailored in difficulty level on the basis of the student’s previous answers. For instance, if a student is correctly answering the early items doled out by the computer, then the next items popping up on the screen will be more difficult ones. Conversely, if the student stumbles on the initially presented items, the computer will then cheerfully provide easier items to the student, and so on. In short, the computer’s program constantly adapts to the student’s responses by providing items better matched to the student’s assessment-determined level of achievement.

Using this adaptive approach, a student’s overall achievement regarding whatever the test is measuring can be determined with fewer items than would typically be required. This is because, in a typical test, many of the test’s items are likely to be too difficult or too easy for a given student. Accordingly, one of the advertised payoffs of computer-adaptive testing is that it saves testing time—that is, it saves those precious instructional minutes often snatched away from teachers because of externally imposed assessment obligations. And there it is, the promotional slogan for computer- adaptive testing: More Accurate Measurement in Less Time! What clear-thinking teacher does not get just a little misty eyed when contemplating the powerful payoffs of computer-massaged testing?

But there are also limitations of computer- adaptive testing that you need to recognize. The first of these limitations stems from the necessity for all the items in such assessments to be measuring a single variable, such as students’ “mathematical mastery.” Because many items are needed to make computer-adaptive assessment purr properly, and because the diverse difficulties of these items must all be linked to what’s sometimes referred to as “a unidimensional trait” (for instance, a child’s overall reading prowess), com puter- adaptive assessment precludes the possibility of providing student-specific diagnostic data. Too few items dealing with a particular subskill or a body of enabling knowledge can be administered during a student’s abbreviated testing time. In other words, whereas computer-adaptive assessment can supply teachers with an efficiently garnered general fix on a student’s achievement of what’s

often a broadly conceptualized curricular aim, it often won’t reveal a student’s specific strengths and weaknesses. Thus, from an instructional perspectiv e, computer-adaptive assessment usually falls short of supplying the instructionally meaningful results most teachers need.

Second, as students wend their way merrily through a computer-adaptive test, depending on how they respond to certain items, different students frequently receive different items. The adjustments in the items that are dished up to a student depend on the student’s computer-determined status regarding whatever unidimensional trait is being measured. Naturally, because of differences in students’ mastery of this “big bopper” variable, different students receive different items thereafter. Consequently, a teacher’s students no longer end up taking the same exam. The only meaningful way of comparing students’ overall test performances, therefore, is by employing a scale that represents the unidimensional trait (such as a student’s reading capabilities) being measured. We will consider such scale scores later, in Chapter 13. Yet, even before we do so, it should be apparent that when a teacher tries to explain to a parent why a parent’s child tackled a test with a unique collection of items, the necessary explanation is a challenging one—and sometimes an almost impossible one, when the teacher’s explanations hinge on items unseen by the parent’s child.

Finally, when most educators hear about computer-adaptive assessment and get a general sense of how it operates, they often assume that it does its digital magic in much the same way—from setting to setting. In other words, educators think the program governing the operation of computer- adaptive testing in State X is essentially identical to the way the computer-adaptive testing program operates in State Y. Not so! Teachers need to be aware that the oft-touted virtues of computer-adaptive testing are dependent on the degree of adaptivity embodied in the program that’s being employed to analyze the results. Many educators, once they grasp the central thrust of computer-adaptive testing, believe that after a student’s response to each of a test’s items, an adjustment is made in the upcoming test items. This, of course, would represent an optimal degree of adaptivity. But item-adaptive adjustments are not often employed in the real world because of such practical considerations as the costs involved.

M06_POPH0936_10_SE_C06.indd 168M06_POPH0936_10_SE_C06.indd 168 09/11/23 6:23 PM09/11/23 6:23 PM

ten (Divided by two) Overall Item-Writing precepts 169

Complex Syntax Complex syntax, although it sounds something like an exotic surcharge on cigarettes and alcohol, is often encountered in the assessment items of neo- phyte item-writers. Even though some teachers may regard themselves as Steinbecks-in-hiding, an assessment instrument is no setting in which an item-writer should wax eloquent. This fourth item-writing precept directs you to avoid complicated sentence constructions and, instead, to use very simple sentences. Although esteemed writers such as Thomas Hardy and James Joyce are known for their convoluted and clause-laden writing styles, they might have turned out to be mediocre item-writers. Too many clauses, except at Christmas- time, mess up test items. (For readers needing a clue regarding the previous sen- tence’s cryptic meaning, think of a plump, red-garbed guy who brings presents.)

Difficult Vocabulary Our fifth and final item-writing precept is straightforward. It indicates that when writing educational assessment items, you should eschew obfuscative verbiage. In other words—and almost any other words would be preferable—use vocabulary

Accordingly, most of today’s computer-adaptive tests employ what is called a cluster-adaptive approach. One or more clusters of items, perhaps a half-dozen or so items per cluster, are used to make an adjustment in the upcoming items for a student. Such adjustments are based on the student’s performance on the cluster of items. Clearly, the more clusters of items that are employed, the more tailored to a student’s status will be the subsequent items. Thus, you should not assume that the computer-adaptive test being employed in your locale is based on an item-adaptive approach when, in fact, it might be doing its adaptive magic based on only a single set of cluster-adaptive items. Computer adaptivity may be present, but it is a far cry from the mistaken conception of sustained adaptivity that many educators currently possess.

To sum up, then, computer-adaptive assessments can, indeed, provide educators with a pair of potent advantages—they deliver more accurate assessment and take less time to do so. However, of necessity, any meaningful diagnostic dividends usually scurry out the window with such computer-adaptive testing, and even the advertised dividends of computer-adaptive testing tumble if cost-conscious versions of computer adaptivity are being employed. If computer-adaptive testing is going on in your locale, find out how it does

its digital dance so that you can determine what instructional help, if any, this avant-garde version of educational assessment supplies to you. Few classroom teachers will attempt, on their own, to build computer-adaptive tests just for their own students. Classroom teachers are too smart— and have other things going on in their lives. Accordingly, teachers’ knowledge about the general way in which computer-adaptive testing functions should position such teachers to better evaluate any district-selected or state-selected computer- adaptive assessments. Does a district-chosen computer-adaptive test that purports to be a boon to teachers’ instructional decision making show evidence that it truly does so? Is a state-imposed computer-adaptive test that’s intended to evaluate the state’s schools accompanied by evidence that it does so accurately? In short, teachers should not assume that merely because an educational test is computer-adaptive, this guarantees that the test is reliable, valid, or fair for its intended uses. If you have any required test being used in your setting without enough evidence of its suitability for its intended purpose, you should consider registering your dismay over this educationally unsound practice. And this is true even for assessments jauntily cloaked in computer-adaptive attire.

M06_POPH0936_10_SE_C06.indd 169M06_POPH0936_10_SE_C06.indd 169 09/11/23 6:23 PM09/11/23 6:23 PM

170 ChaptEr 6 Selected-response tests

suitable for the students who’ll be taking your tests. Assessment time is not the occasion for you to trot out your best collection of polysyllabic terms or to secure a series of thesaurus-induced thrills. The more advanced that the vocabulary level is in your assessment devices, the more likely you’ll fail to get a good fix on your students’ true status. They will have been laid low by your overblown vocabulary. In the case of the terminology to be used in classroom assessment instruments, simple wins.

In review, you’ve now seen five item-writing precepts that apply to any kind of classroom assessment device you develop. The negative admonitions in those precepts certainly apply to tests containing selected-response items. And that’s what we’ll be looking at in the rest of the chapter. More specifically, you’ll be encountering a series of item-writing guidelines to follow when construct- ing particular kinds of selected-response items. For convenience, we can refer to these guidelines linked to particular categories of items as “item-category guidelines” or “item-specific guidelines.” These item-category guidelines are based either on empirical research evidence or on decades of teachers’ experi- ence in using such items. If you opt to use any of the item types to be described here in your own classroom tests, try to follow the guidelines for that kind of item. Your tests will typically turn out to be better than if you hadn’t.

Here, and in several subsequent chapters, you’ll be encountering sets of item-writing, item-revision, and response-scoring guidelines that you’ll be encouraged to follow. Are the accompanying guidelines identical to the guide- lines you’re likely to find in other texts written about classroom assessment? No, they are not identical, but if you were to line up all of the texts ever written about classroom testing, you’d find that their recommendations regarding the care and feeding of items for classroom assessment are fundamentally similar.

In your perusal of selected-response options, when deciding what sorts of items to employ, this is a particularly good time to make sure that your items match the level of cognitive behavior that you want your students to display. As always, a student’s performance on a batch of items can supply you with the evidence that’s necessary to arrive at an inference regarding the student’s status. If you are attempting to promote student’s higher-order cognitive skills, then don’t be satisfied with the sorts of selected-response items that do little more than tap into a student’s memorized information.

Binary-Choice Items A binary-choice item gives students only two options from which to select. The most common form of binary-choice item is the true–false item. Educators have been using true–false tests probably as far back as Socrates. (True or False: Plato was a type of serving-dish used by Greeks for special meals.) Other variations of binary-choice items are those in which students must choose between the pairs yes–no, right–wrong, correct–incorrect, fact–opinion, and so on.

M06_POPH0936_10_SE_C06.indd 170M06_POPH0936_10_SE_C06.indd 170 09/11/23 6:23 PM09/11/23 6:23 PM

­inary-CCoice Items 171

The virtue of binary-choice items is they are typically so terse that students can respond to many items in a short time. Therefore, it is possible to cover a large amount of content in a brief assessment session. The greatest weakness of binary-choice items is that, because there are only two options, students have a 50–50 chance of guessing the correct answer even if they don’t have the fog- giest idea of what’s correct. If a large number of binary-choice items are used, however, this weakness tends to evaporate. After all, although students might guess their way correctly through a few binary-choice items, they would need to be extraordinarily lucky to guess their way correctly through 30 such items.

Here are five item-category guidelines for writing binary-choice items. A brief discussion of each guideline is provided in the following paragraphs.

Item-Writing precepts for ­inary-CCoice Items

1. Phrase items so that a superficial analysis by the student suggests a wrong answer.

2. Rarely use negative statements, and never use double negatives. 3. Include only one concept in each statement. 4. Employ an approximately equal number of items representing the two

categories being tested. 5. Keep item length similar for both categories being tested.

Phrasing Items to Elicit Thoughtfulness Typically, binary items are quite brief, but brevity need not reflect simplistic choices for students. In order to get the most payoff from binary-choice items, you’ll want to phrase items so that students who approach the items superficially will answer them incorrectly. Thus, if you were creating the items for a true–false test, you would construct statements for your items that were not blatantly true or blatantly false. Blatancy in items rarely leads to accurate inferences about students. Beyond blatancy avoidance, however, you should phrase at least some of the items so that if students approach them unthinkingly, they’ll choose false for a true statement, and vice versa. What you’re trying to do is to get students to think about your test items and, thereby, give you a better idea about how much good thinking the students can do.

Minimizing Negatives With binary-choice items, many students have a really difficult time responding to negatively phrased items. For instance, suppose that in a true–false test you were asked to decide about the truth or falsity of the following statement: “The League of Nations was not formed immediately after the conclusion of World War II.” What the item is looking for as a correct answer to this statement is true, because the League

M06_POPH0936_10_SE_C06.indd 171M06_POPH0936_10_SE_C06.indd 171 09/11/23 6:23 PM09/11/23 6:23 PM

172 ChaptEr 6 Selected-response tests

of Nations was in existence prior to World War II. Yet, the existence of the word not in the item really will confuse some students. They’ll be apt to answer false even if they know the League of Nations was functioning before World War II commenced.

Because seasoned teachers have churned out their share of true–false items over the years, most such teachers know all too well how tempting it is to simply insert a not into an otherwise true statement. But don’t yield to the temptation. Only rarely succumb to the lure of the nagging negative in binary-choice items. Items containing double negatives or triple negatives (if you could contrive one) are obviously to be avoided.

Avoiding Double-Concept Items The third guideline for binary-choice items directs you to focus on only a single concept in each item. If you are creating a statement for a right–wrong test, and you have created an item in which half of the statement is clearly right and the other half is clearly wrong, you make it mighty difficult for students to respond correctly. The presence of two concepts in a single item, even if both are right or both are wrong, tends to confuse students and, as a consequence, yields test results that are apt to produce inaccurate inferences about those students.

Balancing Response Categories If you’re devising a binary-choice test, try to keep somewhere near an equal num- ber of items representing the two response categories. For example, if it’s a true– false test, make sure you have similar proportions of true and false statements. It’s not necessary to have exactly the same number of true and false items. The numbers of true and false items, however, should be roughly comparable. This fourth guideline is quite easy to follow if you simply keep it in mind when creat- ing your binary-choice items.

Maintaining Item-Length Similarity The fifth guideline is similar to the fourth because it encourages you to structure your items so there are no give-away clues associated with item length. If your two response categories are accurate and inaccurate, make sure the length of the accurate statements is approximately the same as the length of the inaccurate statements. When creating true–false tests, there is a tendency to toss in qualifying clauses for the true statements so that those statements, properly qualified, are truly true—but also long! As a result, there’s a systematic pattern wherein long statements tend to be true and short statements tend to be false. As soon as students catch on to this pattern, they can answer items correctly without even referring to an item’s contents.

In review, we’ve considered five item-writing guidelines for binary-choice items. If you follow those guidelines and keep your wits about you when creat- ing binary-choice items, you’ll often find this type of test will prove useful in the classroom. And that’s true, not false.

M06_POPH0936_10_SE_C06.indd 172M06_POPH0936_10_SE_C06.indd 172 09/11/23 6:23 PM09/11/23 6:23 PM

Multiple ­inary-CCoice Items 173

Multiple Binary-Choice Items A multiple binary-choice item is one in which a cluster of items is presented to stu- dents, requiring a binary response to each of the items in the cluster. Typically, but not always, the items are related to an initial statement or set of statements. Multiple binary-choice items are formatted so they look like traditional multiple-choice tests. In a multiple-choice test, the student must choose one answer from several options, but in the multiple binary-choice test, the student must make a response for each statement in the cluster. Figure 6.1 is an example of a multiple binary-choice item.

David Frisbie (1992) reviewed research on such items and concluded that mul- tiple binary-choice items are (1) highly efficient for gathering student achievement data, (2) more reliable than other selected-response items, (3) able to measure the same skills and abilities as multiple-choice items dealing with comparable content, (4) a bit more difficult for students than multiple-choice tests, and (5) perceived by students as more difficult but more efficient than multiple-choice items. Frisbie believes that when teachers construct multiple binary-choice items, they must be attentive to all of the usual considerations in writing regular binary-choice items. However, he suggests the following two additional guidelines.

• • • Suppose that a dozen of your students completed a 10-item multiple-choice test and earned the following numbers of correct scores:

5, 6, 7, 7, 7, 7, 8, 8, 8, 8, 9, 10

9. The median for your students’ scores is 7.5. (True)

10. The mode for the set of scores is 8.0. (False)

11. The range of the students’ scores is 5.0. (True)

12. The median is different than the mean. (False)

Figure 6.1 an Illustrative Multiple true–False Item

Item-Writing precepts for Multiple ­inary-CCoice Items

1. Separate item clusters vividly from one another. 2. Make certain that each item meshes well with the cluster’s stimulus material.

Separating Clusters Because many students are familiar with traditional multiple-choice items, and because each of those items is numbered, there is some danger that students may become con- fused by the absence of numbers where such numbers are ordinarily found. Thus, be

M06_POPH0936_10_SE_C06.indd 173M06_POPH0936_10_SE_C06.indd 173 09/11/23 6:23 PM09/11/23 6:23 PM

174 ChaptEr 6 Selected-response tests

sure to use some kind of formatting system to make it clear that a new cluster is com- mencing. In the illustrative item seen in Figure 6.1, notice that three dots (• • •) have been used to signify the beginning of a new cluster of items. You can use asterisks, lines, boxes, or some similar way of alerting students to the beginning of a new cluster.

Coordinating Items with Their Stem In multiple-choice items, the first part of the item—the part preceding the response options—is usually called the stem. For multiple binary-choice items, we refer to the first part of the item cluster as the cluster’s stem or stimulus material. The second item-writing guideline for this item type suggests that you make sure all items in a cluster are, in fact, linked in a meaningful way to the cluster’s stem. If they’re not, then you might as well use individual binary-choice items rather than multiple binary-choice items.

There’s another compelling reason why you should consider adding multiple binary-choice items to your classroom assessment repertoire. Unlike traditional binary-choice items, for which it’s likely students will need to rely on memorized infor- mation, that’s rarely the case with multiple binary-choice items. It is rare, that is, if the stimulus materials contain content that’s not been encountered previously by stu- dents. In other words, if the stem for a subset of multiple binary-choice items contains material that’s new to the student, and if each binary-choice item depends directly on the previously unencountered content, it’s dead certain the student will need to func- tion above the mere recollection of knowledge, the lowest level of Bloom’s cognitive taxonomy (Bloom et al., 1956). So, if you make certain that your stimulus material for multiple binary-choice items contains new content, those items will surely be more intellectually demanding than the run-of-the-memory true–false item.

In review, we’ve considered an item type that’s not widely used but has some special virtues. The major advantage of multiple binary-choice items is that students can respond to two or three such items in the time it takes them to respond to a single multiple-choice item. Other things being equal, the more items students respond to, the more reliably we can gauge their abilities.

I must confess to a mild bias toward this oft-overlooked item type because for over 20 years I used such tests as the final examinations in an introductory educa- tional measurement course I taught in the UCLA Graduate School of Education. I’d give my students 20 one- or two-paragraph descriptions of previously unen- countered educational measurement situations and then follow up each description with five binary-choice items. In all, then, I ended up with a 100-item final exam that really seemed to sort out those students who knew their stuff from those who didn’t. The five items in each cluster were simply statements to which students were to respond accurate or inaccurate. I could just as appropriately have asked for true or false responses, but I tried to add a touch of suave to my exams. After all, I was teaching in a Graduate School of Education. (In retrospect, my decision seems some- what silly.) Nonetheless, I had pretty good luck with my 100-item final exams. I used 50-item versions for my midterm exams, and they worked well too. For certain kinds of purposes, I think you’ll find multiple binary-choice items will prove useful.

M06_POPH0936_10_SE_C06.indd 174M06_POPH0936_10_SE_C06.indd 174 09/11/23 6:23 PM09/11/23 6:23 PM

Multiple-CCoice Items 175

Multiple-Choice Items For a number of decades, the multiple-choice test item has dominated achieve- ment testing in the United States and many other nations. Multiple-choice items can be used to measure a student’s possession of knowledge or a student’s abil- ity to engage in higher levels of thinking. A strength of multiple-choice items is that they can contain several answers differing in their relative correctness. Thus, the student can be called on to make subtle distinctions among answer options, several of which may be somewhat correct. A weakness of multiple- choice items, as is the case with all selected-response items, is that students need only recognize a correct answer. Students need not generate a correct answer. Although a fair amount of criticism has been heaped on multiple-choice items, particularly in recent years, properly constructed multiple-choice items can tap a rich variety of student skills and knowledge, and thus they can be useful tools for classroom assessment.

The first part of a multiple-choice item, as noted earlier, is referred to as the item’s stem. The potential answer options are described as item alternatives. Incorrect alternatives are typically referred to as item distractors. Two common ways of creating multiple-choice items are to use an item stem that is either (1) a direct question or (2) an incomplete statement. With younger students, the direct-question approach is preferable. Using either direct-question stems or incomplete-statement stems, a multiple-choice item can ask students to select either a correct answer or a best answer.

In Figure 6.2 there are examples of a direct-question item requesting a best-answer response (indicated by an asterisk) and an incomplete-statement item requesting a correct-answer response (also indicated by an asterisk). One frequently cited advantage of using a best-answer rather than a correct-answer

Direct-Question Form (best-answer version) Which of the following modes of composition would be most effective in explaining to someone how a bill becomes a law in this nation?

A. Narrative * B. Expository

C. Persuasive D. Descriptive

Incomplete-Statement Form (correct-answer version) Mickey Mouse’s nephews are named

A. Huey, Dewey, and Louie. B. Mutt and Jeff. C. Larry, Moe, and Curly.

* D. Morty and Ferdie.

Figure 6.2 Illustrative Multiple-CCoice Items

M06_POPH0936_10_SE_C06.indd 175M06_POPH0936_10_SE_C06.indd 175 09/11/23 6:23 PM09/11/23 6:23 PM

176 ChaptEr 6 Selected-response tests

to these sorts of items is that when students are faced with a set of best-answer choices, we can see whether they are able to make distinctions among subtle differ- ences in a set of answers—all of which are technically “correct.” Moreover, because best-answer items can require the student to do some serious thinking when choos- ing among a set of options, these sorts of items can clearly elicit higher-order cog- nitive thinking from students. And, of course, that’s usually a good thing to do.

Let’s turn now to a consideration of item-category guidelines for multiple-choice items. Because of the widespread use of multiple-choice items over the past half-century, experience has generated quite a few suggestions regarding how to create such items. Here, you’ll find five of the more frequently cited item-specific recommendations for constructing multiple-choice items.

Stem Stuffing A properly constructed stem for a multiple-choice item will present a clearly described task to the student so the student can then get to work on figuring out which of the item’s options is best (if it’s a best-answer item) or is correct (if it’s a correct-answer item). A poorly constructed stem for a multiple-choice item will force the student to read one or more of the alternatives in order to figure out what the item is getting at. In general, therefore, it’s preferable to load as much of the item’s content as possible into the stem. Lengthy stems and terse alternatives are, as a rule, much better than skimpy stems and long alternatives. You might try reading the stems of your multiple-choice items without any of the alternatives to see whether the stems (either direct questions or incomplete statements) make sense all by themselves.

Knocking Negatively Stated Stems It has been alleged, particularly by overly cautious individuals, that “one robin does not spring make.” Without debating the causal relationship between sea- sonal shifts and feathered flyers, in multiple-choice item writing we could say

Item-Writing precepts for Multiple-CCoice Items

1. The stem should consist of a self-contained question or problem. 2. Avoid negatively stated stems. 3. Do not let the length of alternatives supply unintended clues. 4. Randomly assign correct answers to alternative positions. 5. Never use “all-of-the-above” alternatives, but do use “none-of-the-above”

alternatives to increase item difficulty.

M06_POPH0936_10_SE_C06.indd 176M06_POPH0936_10_SE_C06.indd 176 09/11/23 6:23 PM09/11/23 6:23 PM

Multiple-CCoice Items 177

with confidence that “one negative in an item stem does not confusion unmake.” Negatives are strange commodities. A single not, tossed casually into a test item, can make students crazy. Besides, because not is such a tiny word, and might be overlooked by students, a number of students (who didn’t see the not) may be trying to ferret out the best alternative for a positively stated stem that, in reality, is negative.

For example, let’s say you wanted to do a bit of probing of your students’ knowledge of U.S. geography and phrased the stem of a multiple-choice item like this: “Which one of the following cities is located in a state west of the Missis- sippi River?” If your alternatives were: A. San Diego, B. Pittsburgh, C. Boston, and D. Atlanta, students would have little difficulty in knowing how to respond. Let’s say, however, that you decided to add a dollop of difficulty by using the same alter- natives but tossing in a not. Now your item’s stem might read something like this: “Which one of the following cities is not located in a state east of the Mississippi River?” For this version of the item, the student who failed to spot the not (this, in psychometric circles, might be known as not-spotting) would be in big trouble.

By the way, note that in both stems, the student was asked to identify which one of the following answers was correct. If you leave the one out, a student might have interpreted the question to mean that two or more cities were being sought.

If there is a compelling reason for using a negative in the stem of a multiple-choice item, be sure to highlight the negative with italics, boldface type, or underscoring so that students who are not natural not-spotters will have a fair chance to answer the item correctly.

Attending to Alternative Length Novice item-writers often fail to realize that the length of a multiple-choice item’s alternatives can give away what the correct answer is. Let’s say choices A, B, and C say blah, blah, blah, but choice D says blah, blah, blah, blah, blah, and blah. The crafty student will be inclined to opt for choice D not simply because one gets many more blahs for one’s selection, but because the student will figure out the teacher has given so much attention to choice D that there must be something special about it.

Thus, when you’re whipping up your alternatives for multiple-choice items, try either to keep all the alternatives about the same length or, if this isn’t possible, to have at least two alternatives be of approximately equal length. For instance, if you were using four-alternative items, you might have two fairly short alternatives and two fairly long alternatives. What you want to avoid is having the correct alternative be one length (either short or long) while the distractors are all another length.

Incidentally, the number of alternatives is really up to you. Most fre- quently, we see multiple-choice items with four or five alternatives. Because students can guess correct answers more readily with fewer alternatives, three

M06_POPH0936_10_SE_C06.indd 177M06_POPH0936_10_SE_C06.indd 177 09/11/23 6:23 PM09/11/23 6:23 PM

178 ChaptEr 6 Selected-response tests

alternatives are not seen all that often, except with younger students. Having more than five alternatives puts a pretty heavy reading load on the student. Many teachers usually employ four alternatives in their own multiple-choice tests, but in a few instances, the nature of the test’s content leads them to use three or five alternatives.

Assigning Correct Answer-Positions A fourth guideline for writing multiple-choice items is to make sure you scatter your correct answers among your alternatives, so students don’t “guess their way to high scores” simply by figuring out your favorite correct answer-spot is, for instance, choice D or perhaps choice C. Many novice item-writers are reluctant to put the correct answer in the choice-A position because they believe that gives away the correct answer too early. Yet the choice-A position deserves its share of correct answers too. Absence-of-bias should also apply to answer-choice options.

As a rule of thumb, if you have four-alternative items, try to assign approximately 25 percent of your correct answers to each of the four positions. But try to avoid always assigning exactly 25 percent of the correct answers to each position. Students can be remarkably cunning in detecting your own cunning- ness. It may be necessary to do some last-minute shifting of answer positions in order to achieve what is closer to a random assignment of correct answers to the available positions. But always do a last-minute check on your multiple-choice tests to make sure you haven’t accidentally allowed your correct answers to appear too often in a particular answer-choice position.

Dealing with “Of-the-Above” Alternatives Sometimes a beginning item-writer who’s trying to come up with four (or five) reasonable alternatives will toss in a none-of-the-above or an all-of-the-above alternative simply as a “filler” alternative. But both of these options must be con- sidered carefully.

The fifth guideline for this item type says quite clearly that you should never use the all-of-the-above alternative. Here’s why. Let’s say you’re using a five-option type of multiple-choice item, and you want to set up the item so that the fifth option (choice E), “all of the above,” would be correct. This indicates that the first four answers, choices A through D, must all be correct. The prob- lem with such an item is that even a student who knows only that two of the first four alternatives are correct will be able to easily select the all-of-the-above response because, if any two responses are correct, choice E is the only possible best answer. Even worse, some students will read only the first alternative, see that it is correct, mark choice A on their response sheet, and move on to the next test item without going over the full array of enticing alternatives for the item. For either of these reasons, you should never use an all-of-the-above alternative in your multiple-choice items.

M06_POPH0936_10_SE_C06.indd 178M06_POPH0936_10_SE_C06.indd 178 09/11/23 6:23 PM09/11/23 6:23 PM

Multiple-CCoice Items 179

What about the none-of-the-above alternative? The guideline indicates that it may be used when you wish to increase an item’s difficulty. You should do so only when the presence of the none-of-the-above alternative will help you make the kind of test-based inference you want to make. To illustrate, let’s say you want to find out how well your students can perform basic math- ematical operations such as multiplying and dividing. Moreover, you want to be confident that your students can really perform the computations “in their heads,” by using scratch paper, by employing an appropriate app with their cell phones or digital tablets, or—if permitted—by renting a mathematics consultant from a local rental agency. Now, if you use only four-alternative multiple-choice items, there’s a real likelihood that certain students won’t be able to perform the actual mathematical operations, but may be able to select, by estimation, the answer that is most reasonable. After all, one of the four options must be correct. Yet, when you simply add the none-of-the-above option (as a fourth or fifth alternative), students can’t be sure that the correct answer is silently sitting there among the item’s alternatives. To determine whether the correct answer is really one of the alternatives provided for the item, students will be obliged to perform the required mathematical operation and come up with the actual answer. In essence, when the none-of-the-above option is added, the task presented to the student more closely approximates the task in which the teacher is interested. A student’s chance of guessing the correct answer to a multiple-choice item is markedly less likely when a none- of-the-above option shows up in the item.

Here’s a little wrinkle on this guideline you might find useful. Be a bit careful about using a none-of-the-above option for a best-answer multiple-choice item. Care is warranted because there are wily students waiting out there who’ll parade a series of answers for you that they regard as better than the one option you think is a winner. If you can cope with such carping, there’s no problem. If you have a low carping-coping threshold, however, avoid none-of-the-above options for most—if not all—of your best-answer items.

If you opt for the use of multiple-choice items as part of your classroom-assessment repertoire, you might be prepared to respond to students’ questions about correction-for-guessing scoring. Indeed, even if the teacher does not use any sort of correction-for-guessing when scoring students’ choices, it is often the case that—because such scoring approaches are used with certain stan- dardized tests—some students (or some parents) will raise questions about such scoring. The formula used on certain standardized tests to minimize guessing typically subtracts, from the number of correctly answered items, a fraction based on the number of items answered incorrectly, but not those items for which no response was given by the student.

For example, suppose that on a 50-item test composed of 4-option multiple-choice items, a student answered 42 items correctly, but also incorrectly answered 8 items. That student, instead of receiving a raw score of 42 items correct, would have 2 points subtracted—resulting in a score of 40 correct. It is assumed

M06_POPH0936_10_SE_C06.indd 179M06_POPH0936_10_SE_C06.indd 179 09/11/23 6:23 PM09/11/23 6:23 PM

180 ChaptEr 6 Selected-response tests

that, by chance alone on each of the 8 incorrectly answered items, taking one chance out of four would yield 2 correct answers (one out of the four options for the 8 items). If another student also answered 42 items correctly, but did not guess on any items—preferring instead to leave them blank—then that more cautious stu- dent would retain a score of 42 correct. Few teachers employ this sort of correction formula for their own classroom tests, but teachers should have at least a ball-park notion of what makes such formulae tick. (Please note the ritzy Latin plural of “formula” just employed. Clearly, this is a classy book that you’re reading.)

In review, we’ve just sauntered through a cursory look at five guidelines for the creation of multiple-choice items. There are other, more subtle suggestions for creating such items, but if you combine these guidelines with the five general item-writing precepts presented earlier in the chapter, you’ll have a good set of ground rules for devising decent multiple-choice items. As with all the item types being described, there’s no substitute for oodles of practice in item writing that’s followed by collegial or supervisorial reviews of your item-writing efforts. It is said that Rome wasn’t built in a day. Similarly, in order to become a capable con- structor of multiple-choice items, you’ll probably need to develop more than one such test—or even more than two.

Decision time Multiple-Guess test Items?

For all six years that he has taught, Viraj Patel has worked with fifth-graders. Although Viraj enjoys all content areas, he takes special pride in his reading instruction, because he believes his students enter the sixth grade with dramatically improved comprehension capabilities. Viraj spends little time trying to promote his students’ isolated reading skills but, instead, places great emphasis on having students “construct their own meanings” from what they have read.

All of Viraj’s reading tests consist exclusively of four-option multiple-choice items. As Viraj puts it, “I can put together a heavy-duty multiple-choice item when I put my mind to it.” Because he has put his mind to it many times during the past six years, Viraj is quite satisfied that his reading tests are as good as his reading instruction.

At the most recent open house night at his school, however, a group of five parents registered genuine unhappiness with the exclusive multiple- choice makeup of Viraj’s reading tests. The parents

had obviously been comparing notes prior to the open house, and Mrs. Davies (the mother of one of Viraj’s fifth-graders) acted as their spokesperson. In brief, Mrs. Davies argued that multiple-choice tests permitted even weak students to “guess their way to good scores.” “After all,” Mrs. Davies pointed out, “do you want to produce good readers or good guessers?”

Surprised and somewhat shaken by this incident, Viraj has been rethinking his approach to reading assessment. He concludes that he can (1) stick with his tests as they are, (2) add some short-answer or essay items to his tests, or (3) replace all his multiple-choice items with open-ended items. As he considers these options, he realizes that he has given himself a multiple-choice decision!

If you were Viraj, what would your decision be?

M06_POPH0936_10_SE_C06.indd 180M06_POPH0936_10_SE_C06.indd 180 09/11/23 6:23 PM09/11/23 6:23 PM

MatcCing Items 181

Column A Column B

____ 1. World War I A. Bush (the father)

____ 2. World War II B. Clinton

____ 3. Korea C. Eisenhower

____ 4. Vietnam D. Johnson

____ 5. First Persian Gulf E. Nixon

F. Roosevelt

G. Truman

H. Wilson

Directions: On the line to the left of each military conflict listed in Column A, write the letter of the U.S. president in Column B who was in office when that military conflict was concluded. Each name in Column B may be used no more than once.

Figure 6.3 an Illustrative MatcCing Item

Matching Items A matching item consists of two parallel lists of words or phrases requiring the stu- dent to match entries on one list with appropriate entries on the second list. Entries in the list for which a match is sought are widely referred to as premises. Entries in the list from which selections are made are referred to as responses. Usually, students are directed to match entries from the two lists according to a specific kind of asso- ciation described in the test directions. Figure 6.3 is an example of a matching item.

Notice that in Figure 6.3’s illustrative matching item, both lists are homogeneous. All of the entries in the column at the left (the premises) are U.S. military conflicts, and all of the entries in the column at the right (the responses) are names of U.S. presi- dents. Homogeneity is an important attribute of properly constructed matching items.

An advantage of matching items is that their compact form takes up little space on a printed page or a computer screen, thus making it easy to consider a good deal of information efficiently. Matching items (presented on paper tests) can be easily scored by simply holding a correct-answer template next to the list of premises where students are to supply their selections from the list of responses. A disadvantage of matching items is that, like binary-choice items, they sometimes encourage students’ memorization of low-level factual information that, in at least some instances, is of debatable utility. The illus- trative matching item is a case in point. Although it’s relatively easy to create matching items such as this, is it really important to know which U.S. chief executive was in office when a military conflict was concluded? That’s the kind of issue you’ll be facing when you decide what sorts of items to include in your classroom assessments.

M06_POPH0936_10_SE_C06.indd 181M06_POPH0936_10_SE_C06.indd 181 09/11/23 6:24 PM09/11/23 6:24 PM

182 ChaptEr 6 Selected-response tests

Typically, matching items are used as part of a teacher’s assessment arsenal. It’s pretty difficult to imagine a major classroom examination consisting exclu- sively of matching items. Matching items don’t work well when teachers are try- ing to assess relatively distinctive ideas, because matching items require pools of related entries to insert into the matching format.

Let’s consider a half-dozen guidelines you should think about when creating matching items for your classroom assessment instruments. The guidelines are presented here.

Employing Homogeneous Entries As noted earlier, each list in a matching item should consist of homogeneous entries. If you really can’t create a homogeneous set of premises and a homo- geneous set of responses, you shouldn’t be mucking about with matching items.

Going for Relative Brevity From the student’s perspective, it’s much easier to respond to matching items if the entries in both lists are relatively few in number. About 10 or so premises should be the upper limit for most matching items. The problem with longer lists is that students spend so much time trying to isolate the appropriate response for a given premise that they may forget what they’re attempting to find. Very lengthy sets of premises or responses are almost certain to cause at least some students difficulty in responding because they lose track of what’s being sought. It would be far better to take a lengthy matching item with 24 premises, then split it into three 8-premise matching items.

In addition, to cut down on the reading requirements of matching items, be sure to place the list of shorter words or phrases at the right. In other words, make the briefer entries the responses. In this way, when students are scanning the response lists for a matching entry, they won’t be obliged to read too many lengthy phrases or sentences.

Item-Writing precepts for MatcCing Items

1. Employ homogeneous lists. 2. Use relatively brief lists, placing the shorter words or phrases at the right. 3. Employ more responses than premises. 4. Order the responses logically. 5. Describe the basis for matching and the number of times responses may be used. 6. Place all premises and responses for an item on a single page (or screen).

M06_POPH0936_10_SE_C06.indd 182M06_POPH0936_10_SE_C06.indd 182 09/11/23 6:24 PM09/11/23 6:24 PM

MatcCing Items 183

Loading Up on Responses A third guideline for the construction of matching items is to make sure there are at least a few extra responses. Otherwise, if the numbers of premises and responses are identical, the student who knows, say, 80 percent of the matches to the premises may be able to figure out the remaining matches by a process of elimination. A few extra responses reduce this likelihood substantially. Besides, these sorts of responses are inexpensive. Be a big spender!

Ordering Responses So that you’ll not provide unintended clues to students regarding which responses go with which premises, it’s a good idea in matching items to order the responses

parent talk Assume that you’ve been using a fair number of multiple-choice items in your classroom examinations. Benito’s parents, Mr. and Mrs. Olmedo, have set up a 15-minute conference with you during a Back-to-Classroom Night to talk about Benito’s progress.

When they arrive, they soon get around to the topic in which they are most interested—namely, your multiple-choice test items. As Mr. Olmedo puts it, “We want our son to learn to use his mind, not his memory. Although my wife and I have little experience with multiple-choice tests because almost all of our school exams were of an essay nature, we believe multiple-choice tests measure only what Benito has memorized. He has a good memory, as you’ve probably found out, but we want more for him. Why are you using such low-level test items?”

If I were you, here’s how I’d respond to Mr. Olmedo’s question:

“I appreciate your coming in to talk about Benito’s education and, in particular, the way he is being assessed. And I also realize that recently there has been a good deal of criticism of multiple-choice test items in the press. More often than not, such criticism is altogether appropriate. In far too many cases, because multiple-choice items are easy to score, they’re used to assess just about everything. And, all

too often, those kinds of items do indeed ask students to do little more than display their memorization skills.

“But this doesn’t need to be the case. Multiple-choice items, if they are carefully developed, can assess a wide variety of truly higher-order thinking skills. In our school, all teachers have taken part in a series of staff- development workshop sessions in which every teacher learned how to create challenging multiple-choice items that require students to display much more than memory.” (At this point, you might whip out a few examples of demanding multiple-choice items used in your own tests and then go through them, from stem to alternatives, showing Mr. and Mrs. Olmedo what you meant. If you don’t have any examples of such items from your tests, you might think seriously about the possible legitimacy of Benito’s parents’ criticism.)

“So, although there’s nothing wrong with Benito’s acquiring more memorized information, and a small number of my multiple-choice items actually do test for such knowledge, the vast majority of the multiple-choice items that Benito will take in my class call for him to employ that fine mind of his.

“That’s what you two want. That’s what I want.”

Now, how would you respond to Mr. and Mrs. Olmedo’s concerns?

M06_POPH0936_10_SE_C06.indd 183M06_POPH0936_10_SE_C06.indd 183 09/11/23 6:24 PM09/11/23 6:24 PM

184 ChaptEr 6 Selected-response tests

in some sort of logical fashion—for example, alphabetical or chronological sequence. Notice that the names of the U.S. presidents are listed alphabetically in the illustrative matching item in Figure 6.3.

Describing the Task for Students The fifth guideline for the construction of matching items suggests that the direc- tions for an item should always make explicit the basis on which the matches are to be made and, at the same time, the number of times a response can be used. The more clearly students understand how they’re supposed to respond, the more accurately they’ll respond, and the more validly you’ll be able to make score-based inferences about your students.

Same-Page Formatting The final guideline for this item type suggests that you make sure all premises and responses for a matching item are on a single page or a solo electronic screen. Not only does this eliminate the necessity for massive (and potentially disrup- tive) page turning and encourages (quieter) screen trading by students, but it also decreases the likelihood that students will overlook correct answers merely because these answer choices were on the “other” page.

Matching items, if employed judiciously, can efficiently assess your stu- dents’ knowledge. The need to employ homogeneous lists of related content tends to diminish the applicability of this type of selected-response item. Nonetheless, if you are dealing with content that can be addressed satisfacto- rily by such an approach, you’ll find matching items a useful member of your repertoire of item types.

What Do Classroom Teachers Really Need to Know About Selected-Response Tests? If you engage in any meaningful amount of assessment in your own classroom, it’s quite likely you’ll find that selected-response items will be useful. Selected-response items can typically be used to ascertain students’ mastery of larger domains of con- tent than is the case with constructed-response kinds of test items. Although it’s often thought that selected-response items must measure only lower-order kinds of cognitive capabilities, inventive teachers can create selected-response options to elicit very high levels of cognitive skills from their students.

As for the four types of items treated in this chapter, you really need to under- stand enough about each kind to help you decide whether one or more of those item types would be useful for a classroom assessment task you have in mind. If you do decide to use binary-choice, multiple binary-choice, multiple-choice, or

M06_POPH0936_10_SE_C06.indd 184M06_POPH0936_10_SE_C06.indd 184 09/11/23 6:24 PM09/11/23 6:24 PM

WCat Do Classroom teacCers really eed to now about Selected-response tests? 185

matching items in your own tests, then you’ll find that the sets of item-writing guidelines for each item type will come in handy.

In Chapter 12, you will learn how to employ selected-response items as part of the formative-assessment process. Rather than relying completely on paper-and-pencil versions of these items, you will see how to employ such items in a variety of less formal ways. Yet, in honesty, even if you adhere to the five general item-writing precepts discussed here as well as to the set of guidelines for particu- lar types of items, you’ll still need practice in constructing selected-response items such as those considered here. As was suggested several times in the chapter, it is exceedingly helpful if you can find someone with measurement moxie, or at least an analytic mind, to review your test-development efforts. It’s difficult in any realm to improve if you don’t get feedback about the adequacy of your efforts. That’s surely true with classroom assessment. Try to entice a colleague or supervisor to look over your selected-response items to see what needs to be strengthened. How- ever, even without piles of practice, if you adhere to the item-writing guidelines provided here, your selected-response tests won’t be all that shabby.

The most important thing to learn from this chapter is that there are four useful selected-response procedures for drawing valid inferences about your stu- dents’ status. The more assessment options you have at your disposal, the more appropriately you’ll be able to assess those students’ status with respect to the variables in which you’re interested.

­ut WCat Does tCis have to Do witC teacCing? In this chapter, not only have you become acquainted with varied types of test items, but you have also learned how to construct them properly. Why, you might ask, does a classroom teacher need to possess such a resplendent repertoire of item types? After all, if a student knows that Abraham Lincoln was assassinated while attending a play, the student could satisfactorily display such knowledge in a variety of ways. A good old true–false item or a nifty multiple- choice item would surely do the necessary assessing.

Well, that’s quite true. But what you need to recall, from a teaching perspective, is that students ought to master the skills and knowledge they’re being taught so thoroughly that they can display such mastery in a good many ways, not just one. A cognitive skill that’s been learned well by a student will be a cognitive skill that can be displayed in all sorts of ways, whether via selected-response items or constructed-response items.

And this is why, when you teach children, you really need to be promoting their generalizable mastery of whatever’s being taught. Consistent with such a push for generalizable mastery, you will typically want to employ varied assessment approaches. If you gravitate toward only one or two item types (that is, if you become “Multiple-Choice Maria” or “True–False Fredrico”), then your students will tend to learn things only in a way that meshes with your favored item type.

Eons ago, when I was preparing to be a teacher, my teacher education professors advised me to employ different kinds of items “for variety’s sake.” Well, classroom teachers aren’t putting on a fashion show. Variety for its own sake is psychometrically stupid. But using a variety of assessment approaches as a deliberate way of promoting and measuring students’ generalizable mastery of what’s been taught—this is psychometrically suave.

M06_POPH0936_10_SE_C06.indd 185M06_POPH0936_10_SE_C06.indd 185 09/11/23 6:24 PM09/11/23 6:24 PM

186 ChaptEr 6 Selected-response tests

Chapter Summary The chapter began with a presentation of five item- writing precepts that pertain to both constructed- response and selected-response items. The five admonitions directed teachers to avoid unclear directions, ambiguous statements, unintentional clues, complex syntax, and hypersophisticated vocabulary.

Consideration was then given to the four most common kinds of selected-response test items: binary-choice items, multiple

binary-choice items, multiple-choice items, and matching items. For each of these four item types, after a brief description of the item type and its strengths and weaknesses, a set of item-writing guidelines was presented. These four sets of guidelines were presented on pages 170, 173, 176, and 181. Each guideline was briefly discussed. Readers were encouraged to consider the four item types when deciding on an answer to the how-to-assess-it question.

References Bloom, B. S., et al. (1956). Taxonomy of educational

objectives: Handbook I: Cognitive domain. New York: David McKay.

Brookhart, S. M., & Nitko, A. J. (2018). Educational assessment of students (8th ed.). Pearson.

Frisbie, D. A. (1992). The multiple true– false format: A status review, Educational Measurement: Issues and Practice, 11, no. 4 (Winter): 21–26.

Herold, B. (2022, April 14). Khan Academy founder on how to boost math performance and make free college a reality, Education Week. https://www.edweek.org/

technology/khan-academy-founder-on-how- to-boost-math-performance-and-make-free- college-a-reality/2022/04

Hill, K. (2022, May 27). Accused of cheating by an algorithm, and a professor she had never met, New York Times. https://www.nytimes. com/2022/05/27/technology/ college- students-cheating-software-honorlock.html

Rodriguez, M. C., & Haladyna, T. M. (2013). Writing selected-response items for classroom assessment. In J. H. McMillan (Ed.), SAGE Handbook of research on classroom assessment (pp. 293–312). SAGE Publications.

M06_POPH0936_10_SE_C06.indd 186M06_POPH0936_10_SE_C06.indd 186 09/11/23 6:24 PM09/11/23 6:24 PM