PLAGIARISM FREE "A" WORK 18 HOURS or LESS

NeNe1994

Chapter6SelectedResponseItems.pdf

Home >Applied Sciences homework help >PLAGIARISM FREE "A" WORK 18 HOURS or LESS

In this chapter, I’ll be describing selected-response items, focusing on

the three main types: (1) binary-choice items, (2) matching items, and

(3) multiple-choice items. I’ll first take a quick peek at the advantages

and disadvantages of each item-type and then list a set of rules for

constructing that kind of item.

Binary-Choice Items A binary-choice item is one in which the test-taker must choose

between only two options. The most common kind of binary-choice

item is the True-False item, in which students are given a set of state-

ments and then asked to indicate whether each statement is true or

false.

Perhaps you’ve never encountered the phrase “binary-choice”

before, and you suspect I’m trying to impress you by tossing out an

esoteric assessment term. What I want you to realize is that you can

build all sorts of decent test items by presenting students with a two-

choice challenge. The student’s choices, for instance, could be

between right and wrong, correct and incorrect, accurate or inaccurate,

and so on. If you think of binary-choice items only as True-False, you

may overlook some useful dichotomous-item possibilities.

6 Selected-Response Items

7 2

ch6.qxd 7/10/2003 9:57 AM Page 72

Popham, W. James. Test Better, Teach Better : The Instructional Role of Assessment, Association for Supervision & Curriculum Development, 2003. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/amridge/detail.action?docID=5704436. Created from amridge on 2022-01-20 14:33:07.

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 7 3

Incidentally, there’s a lesser known variation on this selected-

response item-type that I’ve found very useful over the years. It’s

called a multiple binary-choice item. Here’s how it works. First, the stu-

dent is given a somewhat lengthy stimulus passage of perhaps two or

three paragraphs. Then, after reading the paragraphs, the student

must respond to each item in a set of four to six binary-choice items

based directly on those paragraphs. When I taught educational meas-

urement classes at UCLA, my midterm exams consisted of 50 items. I

used 10 multiparagraph descriptions of measurement-related events,

each of which was followed by 5 binary-choice items based on the

description. My final exam was twice that long: 20 sets of stimulus

material and 100 binary-choice items. I always thought my tests were

terrific. That benign view, of course, may merely represent another

form of assessment bias!

Seriously, though, if you asked any of my former students about

my midterm or final exams, they’d tell you that the multiple binary-

choice items I used were tough. Binary-choice items are not necessar-

ily “easy,” a lazy way out, or focused on low-level cognitive chal-

lenges. My tests included all sorts of challenges. My students had to

consider previously unencountered stimulus information (in two or

three paragraphs) and then make subtle decisions for each of the

related binary-choice items. Remember, even with two options, it is

possible to present students with a pair of choices that require a

sophisticated understanding to sort out.

Advantages and Disadvantages The chief advantage of binary-choice items is their terseness, which

allows a teacher to cover a wide range of content by using a swarm of

such items. For instance, if a history teacher wanted to discover if stu-

dents knew a ton of facts about U.S. history, then a collection of

Right-Wrong or True-False items would be an efficient way to uncover

students’ knowledge of those facts.

ch6.qxd 7/10/2003 9:57 AM Page 73

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

T E S T B E T T E R , T E A C H B E T T E R7 4

The major disadvantage of binary-choice items is that when they

are used exclusively, they tend to promote memorization of factual

information: names, dates, definitions, and so on. While factual

information is often a necessary component of instruction, few

would hold it up as the sole objective. Yes, it is possible for teachers

to build subtleties into their binary-choice items, but teachers are

often tempted to write out a statement that is quite decisively either

true or false. Some teachers simply lift statements from textbooks,

then toss a “not” (or, if in a foul mood, two “nots”) into some of

those statements. From a student’s perspective, of course, the best

way to prepare for those sorts of fact-based items is to memorize,

memorize, memorize.

Some argue that another weakness of binary-choice items is that

students can “guess their way to success.” On probabilistic grounds,

for each binary-choice item, an unadulterated guess is apt to be cor-

rect 50 percent of the time. Well, that’s true for a single binary-choice

item, but when students must cope with 10 or 20 such items, raw

guesswork just won’t work. A student would need to be extremely

lucky to guess to success on a 20-item binary-choice test. Odds are

that a student would stumble onto the correct answers for only about

10 of the 20 items . . . and 50 percent correct on a binary-choice test

should not evoke applause or lead a teacher to infer that the student

had mastered the material.

Summing up, binary-choice items can cover considerable con-

tent, but their use may encourage students to engage in massive

memorization rather than try to acquire any sort of higher-level intel-

lectual mastery of a body of content.

Item-Writing Rules To keep your reading requirements from becoming onerous, I’m

going to dispense all of this book’s item-writing rules in a short-order

manner. Each rule I present will be followed by few sentences to

clarify or qualify the rule. And often, but not always, I’ll toss in an

ch6.qxd 7/10/2003 9:57 AM Page 74

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 7 5

example of a test item that is either consistent with the rule or vio-

lates that rule. I hope these illustrative test items will help you see

how the item-writing rules should be followed. As is always the case,

the rules are intended to help you, the teacher, arrive at more accu-

rate inferences about what’s going on inside your students’ heads.

▲ Generate binary-choice items in pairs, not singly.

Because binary-choice items work best when they are sampling two

discrete categories (e.g., true or false), if there is a statement repre-

senting one of those categories, there ought to be an opposite state-

ment representing the other category. It’s a time-saving tactic to cre-

ate both items at once, and if you can’t, then you probably don’t have

even one good binary-choice item. Putting it another way, because

your binary choices need to be mutually exclusive, there really ought

to be readily reversible statements lurking out there. You might use

only one item in a given test, but then you will have a nifty unused

item for next year!

▲ Phrase each item so that a cursory reading leads to an

incorrect answer. Knowing that binary-choice items may encour-

age students to memorize, it’s the teacher’s responsibility to do every-

thing possible to ensure that memorization is accurate. If you rou-

tinely word binary-choice items so that a superficial reading will lead

students toward wrong answers, then students will soon learn to read

those items with care. Take a look at the following example:

PHRASING THAT FOILS A SUPERFICIAL RESPONSE

T F Given that almost all teachers care deeply about

the progress of their students, and recognizing the

inherent diversity in the cognitive capabilities of

those students, most classroom teachers design

truly individualized instructional programs for their

students. (Answer: False)

ch6.qxd 7/10/2003 9:57 AM Page 75

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

T E S T B E T T E R , T E A C H B E T T E R7 6

Notice that although the item-writer uses appealing phrases (“care

deeply” and “inherent diversity in the cognitive capabilities”) that

might incline a casual or lazy reader toward a validating “True” re-

sponse, closer analysis would reveal that the practical realities of class-

rooms preclude “most” teachers from designing “truly individualized”

programs. This rule is not about trying to trick students into making

mistakes; rather, it’s a reminder to develop your students’ disposition

to read all binary-choice items carefully and respond thoughtfully.

▲ Avoid negative statements, and never use double neg-

atives. In Right-Wrong or True-False items, negatively phrased state-

ments make it needlessly difficult for students to decide whether that

statement is accurate or inaccurate. Consider the following perplex-

ingly stated item:

TOO-NUMEROUS NEGATIVES

T F It is hardly ever the case that students’ lack of

interest in a subject such as science will disincline

teachers to give those students very low grades.

(Answer: Who knows?)

When you create binary-choice items, be sure to steer clear of nega-

tivity, especially doubly confusing, doubled-up negativity.

▲ Restrict single-item statements to single concepts. If

you double-up two concepts in a single item statement, how does a

student respond if one concept is accurate and the other isn’t? Take a

look at this confusing item:

A DOUBLE-CONCEPT TRUE-FALSE ITEM

T F Today’s professional tennis players serve the ball at

much faster speeds than yesteryear’s players be-

cause of high-tech racquets. (Answer: Sort of True)

ch6.qxd 7/10/2003 9:57 AM Page 76

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 7 7

Yes, there are two separate concepts in this single statement: the inci-

dence of faster tennis serves and the reason for those faster serves.

One or both may be true.

▲ Use an approximately equal number of items, reflect-

ing the two categories tested. There is no need for a 20-item

Right-Wrong test to have precisely 10 right items and 10 wrong items.

Still, avoid having one of your two categories represented by 80 or 90

percent of your statements. If you typically overbook on false items

in your True-False tests, students who are totally at sea about an item

will be apt to opt for a false answer and will probably be correct.

▲ Make statements representing both binary categories

equal in length. Again, to avoid giving away the correct answers,

don’t make all your false statements brief and (in an effort to include

necessary qualifiers) make all your true statements long. Students

catch on quickly to this kind of test-making tendency.

Matching Items A matching item consists of two lists of words or phrases. The test-

taker must match components in one list (the premises, typically pre-

sented on the left) with components in the other list (the responses,

typically presented on the right), according to a particular kind of

association indicated in the item’s directions. Here’s an illustrative

matching item based on my favorite films.

A MATCHING ITEM

Directions. On the line to the left of each film category in

Column A, write the letter of the film from Jim’s Favorite Films

in Column B that represents that type of film. Each film in

Column B may be used no more than once.

ch6.qxd 7/10/2003 9:57 AM Page 77

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

T E S T B E T T E R , T E A C H B E T T E R7 8

Column A: Film Categories Column B: Jim’s Favorite Films

1. Action Adventure a. The American President

2. Western b. Gunga Din

3. Musical c. Mrs. Miniver

4. Romance d. Red River

e. The Sound of Music

(Answers: 1: b, 2: d, 3: e, 4: a)

Advantages and Disadvantages Like binary-choice items, matching items can cover a good deal of

territory in an efficient fashion, and they are a great choice if you’re

interesting in finding out if your students have memorized factual

information. (Of course, memorization is a lower-level goal.)

Matching items sometimes can work well if you want your students

to cross-reference and integrate their knowledge regarding the listed

premises and responses. However, because matching items require

pools of related ideas, such items don’t work well if you’re trying to

assess your students’ mastery of distinctive ideas.

Item-Writing Rules Most of the rules for constructing matching items are fairly self-

explanatory.

▲ Use fairly brief lists, placing the shorter entries on the

right. If the premises and responses in a matching item are too long,

students tend to lose track of what they originally set out to look for.

The words and phrases that make up the premises should be short,

and those that make up the responses should be shorter still.

▲ Employ homogeneous lists. Both the list of premises and

responses must be composed of similar sorts of things. If not, an alert

student will be able to come up with the correct associations simply

by “elimination” because some entries in the premises or responses

stand out too much from their siblings. For instances, if I had

included Kung Fu Halloween III as a possible response for my favorite-

ch6.qxd 7/10/2003 9:57 AM Page 78

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 7 9

film matching item, I have little doubt that you would have discarded

it immediately.

▲ Include more responses than premises. If you use the

exact same number of responses as premises in a matching item, then

a student who knows half or more of the correct associations is in posi-

tion to guess the rest of the associations with pretty decent odds. Make

it tougher to get lucky by tossing in at least a few extra responses.

▲ List responses in a logical order. This rule is designed to

make sure you don’t accidentally give away hints about which re-

sponses hook up with which premises. Choose a logical ordering

scheme for your responses (say, alphabetical or chronological) and

stick with it. In my Favorite Films example, I listed the responses

alphabetically.

▲ Describe the basis for matching and the number of

times a response can be used. To satisfy this rule, you need to

make sure your test’s directions clarify the nature of the associations

you want students to use when they identify matches. Regarding the

student’s use of responses, a phrase such as the following is often

employed: “Each response in the list at the right may be used once,

more than once, or not at all.”

▲ Try to place all premises and responses for any match-

ing item on a single page. This rule’s intent is to spare your stu-

dents lots of potentially confusing flipping back and forth in order to

accurately link responses to premises. During exams, it also makes for

a quieter classroom.

Multiple-Choice Items The most popular type of selected-response item has got to be the

multiple-choice item. A student is first given either a question or a par-

tially complete statement. This part of the item is referred to as the

item’s stem. Then three or more potential answer-options are pre-

sented. These are usually called alternatives or options.

ch6.qxd 7/10/2003 9:57 AM Page 79

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

T E S T B E T T E R , T E A C H B E T T E R8 0

Multiple-choice items are widely employed, especially for large-

scale assessments, because the format allows students’ responses to be

electronically scanned. More recently, modestly priced scanners are

increasingly available to teachers for use in scoring their students’

responses to teacher-made, or commercially distributed, multiple-

choice tests.

There are two important variants in a multiple-choice item:

(1) whether the stem consists of a direct question or an incomplete state-

ment and (2) whether the student’s choice of alternatives is supposed

to be a correct answer or a best answer. I’ve worked the four variations

into the following two sample items.

A DIRECT-QUESTION (BEST-ANSWER) MULTIPLE-CHOICE ITEM

During the early years of this century, which of the following

states has experienced the most difficulties in supplying its

citizens with electrical power?

a. California

b. Kansas

c. Louisiana

d. Washington

AN INCOMPLETE-STATEMENT (CORRECT-ANSWER) MULTIPLE-CHOICE ITEM

The capital of Oregon is

a. Columbia.

b. Portland.

c. Salem.

d. Sacramento.

Although there are some exceptions, direct questions usually make

for better multiple-choice questions than incomplete statements do,

simply because there’s less chance for confusion. This is especially

ch6.qxd 7/10/2003 9:57 AM Page 80

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 8 1

true for children in the early grades, who don’t have tons of experi-

ence making sense out of incomplete statements. And, generally

speaking, best-answer items work better than correct-answer items

because the format allows item-writers to build gradients of correct-

ness into the alternatives and, thereby, increase the challenge. To

illustrate, the response to the Best Answer example question ought to

be choice a (California). A test-taker might be inclined to choose one

of the other three states listed (Washington, because of its hydroelec-

tric power; Louisiana, because of its petroleum reserves; or Kansas,

because it lacks either of these resources). However, the best answer at

this particular time in the 21st century is demonstrably “California.”

Advantages and Disadvantages A key plus of the multiple-choice item is its widespread applicability

to the assessment of cognitive skills and knowledge, as well as to the

measurement of students’ affect. (See Chapter 8.) It really is a remark-

ably versatile format. Another advantage of multiple-choice items is

that it’s possible to make them quite varied in the levels of difficulty

they possess. Cleverly constructed multiple-choice items can present

very high-level cognitive challenges to students. (In graduate school,

the most difficult test items I encountered were the carefully crafted

multiple-choice items in a philosophy of education course. The dis-

tinctions among answer options were incredibly subtle.) And, of

course, as with all selected-response items, multiple-choice items are

fairly easy to score. Finally, and this is a big plus from an instructional

perspective, the multiple-choice format allows teachers to build

in certain wrong-answer alternatives that, if chosen by many stu-

dents, will signal the need to deal with particular types of students’

misunderstandings.

The key weakness of multiple-choice items is that when students

review a set of alternatives for an item, they may be able to recognize

a correct answer that they would never have been able to generate

on their own. In that sense, multiple-choice items can present an

ch6.qxd 7/10/2003 9:57 AM Page 81

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

T E S T B E T T E R , T E A C H B E T T E R8 2

exaggerated picture of a student’s understanding or competence,

which might lead teachers to invalid inferences.

Another serious weakness, one shared by all selected-response

items, is that multiple-choice items can never measure a student’s abil-

ity to creatively synthesize content of any sort. Finally, in an effort to

come up with the necessary number of plausible alternatives, novice

item-writers sometimes toss in some alternatives that are blatantly

incorrect. This shortcoming, of course, is more a weakness of the item-

writer than the item-type. Like all test items, a multiple-choice item’s

value as a tool for ferreting out educational variables is predicated on

the item-writer’s skill; however, multiple-items do seem to be less for-

giving. If you’re trying to create a four-alternative multiple-choice

item and two or more of your alternatives are giveaways, then you’re

numerically back to a binary-choice item . . . or a dead giveaway. Note

the patent absurdity of three of the alternatives in the following

mouse-related item:

DISMAL DISTRACTERS

What are the names of Mickey Mouse’s two nephews?

a. Huey, Dewey, and Louie

b. Louie Dewey

c. Morty and Ferdy

d. Minnie

Item-Writing Rules Well-constructed multiple-choice items, when deployed along with

other types of items, can make a genuine contribution to a teacher’s

assessment arsenal. Here are some useful rules for you to follow.

▲ The question or problem in the stem must be self-

contained. Sometimes, beginning item-writers fail to make a multi-

ple-choice item’s stem sufficiently complete so that a student can

read the stem and then set out to select the best (or correct) answer

ch6.qxd 7/10/2003 9:57 AM Page 82

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 8 3

from the alternatives. The stem should contain as much of the item’s

content as possible, thereby rendering the alternatives much shorter

than would otherwise be the case.

▲ Avoid negatively stated stems. Just as with binary-choice

items, negatively stated stems can engender genuine confusion in

students.

▲ Each alternative must be grammatically consistent

with the item’s stem. Think back to the item-writing roadblocks

presented in Chapter 5, one of which was to give students no unin-

tended clues. Well, as you can see from the next sample item, gram-

matical inconsistency for three of these answer-options supplies stu-

dents with a whopping unintended clue.

A NON-SHERLOCKIAN CLUE

In our federal government, the individuals who exercise ulti-

mate authority over legislation and executive actions are the

a. president.

b. vice president.

c. Speaker of the House of Representatives.

d. justices of the Supreme Court.

▲ Make all alternatives plausible, but be sure that one

of them is indisputably the correct or best answer. As I indi-

cated when describing the weaknesses of multiple-choice items,

teachers sometimes toss in one or more implausible alternatives,

thereby diminishing the item substantially. Although avoiding that

problem is important, it’s even more important to make certain that

you really do have one bona fide correct answer in any item’s list of

alternatives, rather than two similar answers, either of which could

be arguably correct.

▲ Randomly use all answer positions in approximately

equal numbers. Too many teachers, reluctant to “give away” the

ch6.qxd 7/10/2003 9:57 AM Page 83

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

T E S T B E T T E R , T E A C H B E T T E R8 4

correct answer too early, tend to concentrate correct answers in the C,

D, or E range of alternatives. It doesn’t take students long to figure

out this kind of pattern. If you use four-option items, make sure that

roughly one-fourth of the correct answers turn out to be A, one-

fourth B, and so on.

▲ Never use “all of the above” as an answer choice, but

use “none of the above” to make items more demanding.

Students often become confused when confronted with items that

have more than one correct answer. Usually, what happens is they’ll

see one correct alternative and instantly opt for it without recogniz-

ing that there are other correct options later in the list. Although

using “all of the above” is always improper, tests involving content

such as mathematics can be made more difficult by presenting three

or four answer options, none of which is correct, followed by a cor-

rect “none-of-the-above” option.

Selected-Response Items and Instruction By this point in the book, I know you know that the reason teachers

test their students is to get a fix on students’ status with respect to

covert educational variables. Rarely will one kind of selected-response

test item do such a thorough job that a teacher can come up with a

valid inference about the student’s status based on that one item-type

alone. Accordingly, I recommend that you operationalize your cur-

ricular targets by using different types of items. Remember, to get

your students ready to display their mastery in myriad ways, you

must use your instruction to promote a level of generalizable mastery,

the kind that can display itself via students’ responses to various

types of test items, both selected-response and constructed-response.

More importantly, when your students leave you, they will be able to

use their generalizably mastered skills and knowledge throughout the

remainder of their schooling and throughout the rest of their lives.

ch6.qxd 7/10/2003 9:57 AM Page 84

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .

S e l e c t e d - R e s p o n s e I t e m s 8 5

Recommended Resources

Haladyna, T. M. (1999). Developing and validating multiple-choice test items (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Linn, R. L., & Gronlund, N. E. (2000). Measurement and assessment in teaching (8th ed.). Upper Saddle River, NJ: Merrill.

McMillan, J. H. (2001). Classroom assessment: Principles and practice for effective instruction (2nd ed.). Boston: Allyn & Bacon.

Northwest Regional Educational Laboratory. (1991). Paper-and-pencil test devel- opment [Videotape]. Los Angeles: IOX Assessment Associates.

Popham, W. J. (Program Consultant). (1996). Creating challenging classroom tests: When students SELECT their answers [Videotape]. Los Angeles: IOX Assessment Associates.

Popham, W. J. (2002). Classroom assessment: What teachers need to know (3rd ed.). Boston: Allyn & Bacon.

Stiggins, R. J. (Program Consultant). (1996). Common sense paper and pencil assessments: A professional development video [Videotape]. Portland, OR: Assessment Training Institute.

INSTRUCTIONALLY FOCUSED TESTING TIPS

• Become familiar with the advantages and disadvantages of

binary-choice, matching, and multiple-choice items.

• If you use these types of selected-response items in your class-

room tests, make certain to follow experience-based rules for cre-

ating each item type.

ch6.qxd 7/10/2003 9:57 AM Page 85

C o p yr

ig h t ©

2 0 0 3 . A

ss o ci

a tio

n f o r

S u p e rv

is io

n &

C u rr

ic u lu

m D

e ve

lo p m

e n t. A

ll ri g h ts

r e se

rv e d .