EDU 530 Week 1 Discussion

188

Chapter 7

Constructed- Response Tests

Chief Chapter Outcome

A sufficient understanding of generally approved guidelines for creating constructed-response items, and scoring students’ responses to them, so that errors in item-construction and response-scoring can be identified and remedied

Learning Objectives

7.1 Define and distinguish guidelines for constructed-response items that are task-understandable to test-takers.

7.2 Identify and employ evaluative criteria—holistically or analytically—when judging the quality of students’ responses.

You’re going to learn about constructed-response tests in this chapter. To be truth- ful, you’re going to learn about only two kinds of paper-and-penci l constructed- response items—namely, short-answer items and essay items (including students’ written compositions). Although you probably know that “you can’t tell a book by its cover,” now you’ve discovered that a chapter’s title doesn’t always describe its contents accurately either. Fortunately, there are no state or federal “ truth-in-titling” laws.

You might be wondering why your ordinarily honest, never-fib author has descended to this act of blatant mislabeling. Actually, it’s just to keep your reading chores more manageable. Earlier, you learned that student-constructed responses can be obtained from a wide variety of item types. In this chapter, we’ll be look- ing at two rather traditional forms of constructed-response items, both of them paper-and-pencil in nature. In the next chapter, the focus will be on performance tests, such as those that arise when we ask students to make oral presentations

M07_POPH0936_10_SE_C07.indd 188M07_POPH0936_10_SE_C07.indd 188 09/11/23 12:31 PM09/11/23 12:31 PM

Constructed-Response Tests 189

or supply comprehensive demonstrations of complex skills in class. After that, in Chapter 9, we’ll be dealing with portfolio assessment and how portfolios are used for assessment purposes.

Actually, all three chapters could be lumped under the single description of performance assessment or constructed-response measurement. This is because anytime you assess your students by asking them to respond in other than a make-a-choice manner, the students are constructing; that is, they are performing. It’s just that if all of this performance assessment stuff had been crammed into a single chapter, you’d have thought you were experiencing a season-long TV mini-series. Yes, it was an inherent concern for your well-being that led to a flagrant disregard for accuracy when titling this chapter.

The major payoff of all constructed-response items is they elicit student responses more closely approximating the kinds of behavior that students must display in real life. After students leave school, for example, the demands of daily living almost never require them to choose responses from four nicely arranged alternatives. And when was the last time, in normal conversation, you were obliged to render a flock of true–false judgments about a set of statements that were presented to you? Yet, you may well be asked to make a brief oral presenta- tion to your fellow teachers or to a parent group, or you may be asked to write a brief report for the school newspaper about your students’ field trip to City Hall. Constructed-response tasks unquestionably coincide more closely with custom- ary nonacademic tasks than do selected-response tasks.

As a practical matter, if the nature of a selected-response task is sufficiently close to what might be garnered from a constructed-response item, then you may wish to consider a selected-response assessment tactic to be a reasonable sur- rogate for a constructed-response assessment tactic. Selected-response tests are clearly much more efficient to score. And, because almost all the teachers are busy folks, time-saving procedures are not to be scoffed at. Yet there will be situ- ations wherein you’ll want to make inferences about your students’ status when selected-response tests just won’t fill the bill. For instance, if you wish to know what kind of a cursive writer Florio is, then you’ll have to let Florio write cur- sively. A true–false test about i dotting and t crossing just doesn’t cut it.

Given the astonishing technological advances we see every other week, what today is typically a paper-and-pencil test is apt, perhaps by tomorrow or the follow- ing day, to be some sort of digitized assessment linked to an exotic outer-space satel- lite. It may become quite commonplace for teachers to develop computer-generated exams themselves. However, because such a digital-assessment derby does not yet surround us—and some classroom teachers have no idea about how to construct a flock of electronically dispensed items—in this chapter the guidelines and illustra- tions will tend to reflect paper-and-pencil assessment. Happily, most of the guide- lines you encounter here will apply with equal force to either paper-and-pencil or electronically dispensed assessments. While digital assessment is becoming increasingly commonplace, the same test-design principles that one would use when designing a pen-and-paper test still hold true.

M07_POPH0936_10_SE_C07.indd 189M07_POPH0936_10_SE_C07.indd 189 09/11/23 12:31 PM09/11/23 12:31 PM

190 ChapTeR 7 Constructed-Response Tests

Short-Answer Items The first kind of constructed-response item we’ll look at is the short-answer item. These types of items call for students to supply a word, a phrase, or a sentence in response to either a direct question or an incomplete statement. If an item asks students to come up with a fairly lengthy response, it is considered an essay item, not a short-answer item. If the item asks students to supply only a single word, then it’s a really short-answer item.

Short-answer items are suitable for assessing relatively simple kinds of learn- ing outcomes such as those focused on students’ acquisition of knowledge. If crafted carefully, however, short-answer items can measure substantially more challenging kinds of learning outcomes. The major advantage of short-answer items is that students need to produce a correct answer, not merely recognize it from a set of selected-response options. The level of partial knowledge that might allow a student to respond correctly to a multiple-choice item won’t be sufficient if the student is required to generate a correct answer to a short-answer item.

The major drawback with short-answer items, as is true with all constructed-response items, is that students’ responses are difficult to score. The longer the responses sought, the tougher it is to score them accurately. And inaccurate scoring, as we saw in Chapter 3, leads to reduced reliability—which, in turn, reduces the validity of the test-based interpretations we make about students—which, in turn, reduces the quality of the decisions we base on those interpretations. Educational measurement is much like the rest of life—it’s simply loaded with trade-offs. When classroom teachers choose constructed-response tests, they must be willing to trade some scoring accuracy (the kind of accu- racy that comes with selected-response tests) for greater congruence between constructed-response assessment strategies and the kinds of student behaviors about which inferences are to be made.

Here, you will find five straightforward item-writing guidelines for short-answer items. Please look them over briefly. Thereafter, each guideline will be amplified by describing the essence of how it works.

Item-Writing Guidelines for Short-answer Items

1. Usually employ direct questions rather than incomplete statements, particularly for young students.

2. Structure the item so that a response should be concise. 3. Place blanks in the margin for direct questions or near the end of incomplete

statements. 4. For incomplete statements, use only one or, at most, two blanks. 5. Make sure blanks for all items are equal in length.

M07_POPH0936_10_SE_C07.indd 190M07_POPH0936_10_SE_C07.indd 190 09/11/23 12:31 PM09/11/23 12:31 PM

Short-answer Items 191

Using Direct Questions Rather Than Incomplete Statements For young children, the direct question is a far more familiar format than the incomplete statement. Accordingly, such students will be less confused if direct questions are employed. Another reason why short-answer items should employ a direct-question format is that the use of direct questions typically forces the item-writer to phrase the item so that less ambiguity is present. With incomplete-statement formats, there’s often too much temptation simply to delete words or phrases from statements the teacher finds in texts. To make sure there isn’t more than one correct answer to a short-answer item, it is often helpful if the item-writer first decides on the correct answer and then builds a question or incomplete statement designed to elicit a unique correct response from knowl- edgeable students.

Nurturing Concise Responses Responses to short-answer items, as might be inferred from what they’re offi- cially called, should be short. Thus, no matter whether you’re eliciting responses that are words, symbols, phrases, or numbers, try to structure the item so a brief response is clearly sought. Suppose you conjured up an incomplete statement item such as this: “An animal that walks on two feet is a __________.” There are all sorts of answers a student might legitimately make to such a too-general item. Moreover, some of those responses could be somewhat lengthy. Now note how a slight restructuring of the item constrains the student: “An animal that walks on two feet is technically classified as a __________.” By the addition of the phrase “technically classified as,” the item-writer has restricted the appropriate responses to only one—namely, “biped.” If your short-answer items are trying to elicit stu- dents’ phrases or sentences, you may wish to place word limits on each, or at least to indicate, in the test’s directions, that only a short one-sentence response is allowable for each item.

Always try to put yourself, mentally, inside the heads of your students and try to anticipate how they are apt to interpret what sort of response is needed by an item. What this second guideline suggests is that you massage an item until it truly lives up to its name—that is, until it becomes a bona fide short-answer item.

Positioning Blanks If you’re using direct questions in your short-answer items, place the students’ response areas for all items near the right-hand margin of the page, immediately after the item’s questions. By doing so, you’ll have all of a student’s responses nicely lined up for scoring. If you’re using incomplete statements, try to place the blank near the end of the statement, not near its beginning. A blank positioned too early in a sentence tends to perplex the students. For instance, notice how this too-early blank can lead to confusion: “The __________ is the governmental body

M07_POPH0936_10_SE_C07.indd 191M07_POPH0936_10_SE_C07.indd 191 09/11/23 12:31 PM09/11/23 12:31 PM

192 ChapTeR 7 Constructed-Response Tests

that, based on the United States Constitution, must ratify all U.S. treaties with foreign nations.” It would be better to use a direct question or to phrase the item as follows: “The governmental body that, based on the United States Constitution, must ratify all U.S. treaties with foreign nations is the __________.”

Limiting Blanks For incomplete-statement types of short-answer items, you should use only one or two blanks. Any more blanks and the item can be labeled a “Swiss-cheese item,” or an item with holes galore. Here’s a Swiss-cheese item to illustrate the confusion that a profusion of blanks can inflict on what is otherwise a decent short-answer item: “After a series of major conflicts with natural disasters, in the year __________, the explorers __________ and __________, accompanied by their __________, discovered __________.” The student who could supply correct answers to such a flawed short-answer item could also be regarded as a truly successful explorer!

Inducing Linear Equality Too often in short-answer items, a beginning itemwriter will give away the answer by varying the length of the answer blanks so that short lines are used when short answers are correct, and long lines are used when lengthier answers are correct. This practice tosses unintended clues to students, so it should be avoided. In the interest of linear egalitarianism, not to mention decent item writing, try to keep all blanks for short-answer items equal in length. Be sure, however, that the length of the answer spaces provided is sufficient for students’ responses—in other words, not so skimpy that students must cram their answers in an illegible fashion.

Okay, let’s review. Short-answer items are the most simple form of constructed-response items, but they can help teachers measure important skills and knowledge. Because such items seek students’ constructed rather than selected responses, those items can be employed to tap some genuinely higher- order skills. Although students’ responses to short-answer items are more difficult to score than their answers to selected-response items, the scoring of such items isn’t impossible. That’s because short-answer items, by definition, should elicit only short answers.

Essay Items: Development The essay item is surely the most commonly used form of constructed-response assessment item. Anytime teachers ask their students to churn out a paragraph or two on what the students know about Topic X or to compose an original composi- tion describing their “Favorite Day,” an essay item is being used. Essay items are particularly useful in gauging a student’s ability to synthesize, evaluate, and com- pose. Such items have a wide variety of applications in most teachers’ classrooms.

M07_POPH0936_10_SE_C07.indd 192M07_POPH0936_10_SE_C07.indd 192 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: eeelopment 193

A special form of the essay item is the writing sample—when teachers ask students to generate a written composition in an attempt to measure students’ composition skills. Because the procedures employed to construct items for such writing samples and, thereafter, for scoring students’ compositions, are so similar to the procedures employed to create and score responses to any kind of essay item, we’ll treat writing samples and other kinds of essay items all at one time at this point. You’ll find it helpful, however, to remember that requiring students to generate a writing sample is, in reality, a widely used type of performance test. We’ll dig more deeply into performance tests in the following chapter.

For assessing certain kinds of complex learning outcomes, the essay item is our hands-down winner. It clearly triumphs when you’re trying to see how well students can create original compositions. Yet, there are a fair number of draw- backs associated with essay items, and if you’re going to consider using such items in your own classroom, you ought to know the weaknesses as well as the strengths of this item type.

One difficulty with essay items is that they’re harder to write—at least to write properly—than is generally thought. I must confess that as a first-year high school teacher, I sometimes conjured up essay items while walking to school and then slapped them up on the chalkboard so that I created almost instant essay exams. At the time, I thought my essay items were pretty good. Such is the pride of youth—and the consequence of naivete. I’m glad I have no record of those items. In retrospect, I must assume that they were pretty putrid. I now know that generating a really good essay item is a tough task—a task not accomplishable while strolling to school. You’ll see this is true from the item- writing rules to be presented shortly. It takes time to create a nifty essay item. You’ll need to find time to construct suitable essay items for your own classroom assessments.

The most serious problem with essay items, however, is the difficulty that teachers have in reliably scoring students’ responses. Let’s say you use a six-item essay test to measure your students’ ability to solve certain kinds of problems in social studies. Suppose that, by some stroke of measurement magic, all your students’ responses could be transformed into typed manuscript form so you could not tell which response came from which student. Let’s say you were asked to score the complete set of responses twice. What do you think is the likelihood your two sets of scores would be consistent? Well, experience suggests that most teachers often aren’t able to produce very consistent results when they score stu- dents’ essay responses. The challenge in this instance, of course, is to increase the reliability of your scoring efforts so that you’re not distorting the validity of the score-based inferences you want to make on the basis of your students’ responses.

Although the potential of having computers gleefully and accurately score students’ essays is often trotted out during discussions of essay scoring, for the present it seems that really particularized scoring of students’ essay responses is somewhat in the future for many teachers. Even though computer-based scoring

M07_POPH0936_10_SE_C07.indd 193M07_POPH0936_10_SE_C07.indd 193 09/11/23 12:31 PM09/11/23 12:31 PM

194 ChapTeR 7 Constructed-Response Tests

ecision Time Forests or Trees?

Anh Nguyen is a brand-new English teacher assigned to work with seventh-grade and eighth- grade students at Dubois Junior High School. Anh has taken part in a state-sponsored summer workshop that emphasizes “writing as a process.” Coupled with what she learned while completing her teacher education program, Anh is confident that she can effectively employ techniques such as brainstorming, outlining, early drafts, peer critiquing, and multiple revisions. She assumes that her students not only will acquire competence in their composition capabilities, but also will increase their confidence about possessing those capabilities. What Anh’s preparation failed to address, however, was how to grade her students’ compositions.

Two experienced English teachers at Dubois Junior High have gone out of their way to help Anh get through her first year as a teacher. Mrs. Miller and Mr. Quy had both been quite helpful during the early weeks of the school year. However, when Anh asked them, one day during lunch, how she should judge the quality of her students’ compositions, two decisively different messages were given.

Mr. Quy strongly endorsed holistic grading of compositions—that is, a general appraisal of each composition as a whole. Although Mr. Quy bases his holistic grading scheme on a set of explicit criteria, he believes a single “gestalt” grade should be given so that “one’s vision of the forest is not obscured by tree-counting.”

Arguing with equal vigor, Mrs. Miller urged Anh to adopt analytic appraisals of her students’ compositions. “If you supply your students with a criterion-by-criterion judgment of their work,” she contended, “each student will be able to know precisely what’s good and what isn’t.” (It was evident, during their fairly heated interchanges, that Mrs. Miller and Mr. Quy had disagreed about this topic in the past.) Mrs. Miller concluded her remarks by saying, “Forget about that forest-and- trees metaphor, Anh. What we’re talking about here is clarity!”

If you were Anh, how would you decide to judge the quality of your students’ compositions?

programs can be taught to do some terrific scoring, the wide range of students’ potential content typically limits computer-based scoring to judging only the most rudimentary sorts of good and bad moves by an essay writer. If the formidable obstacles to tailored computer scoring ever get wrestled to the mat, the classroom teachers will have access to a sanity-saving way of scoring students’ essays. This is a scoring advance greatly to be sought.

Creating Essay Items Because the scoring of essay responses (and students’ compositions) is such an important topic, you’ll soon be getting a set of guidelines on how to score responses to such items. The more complex that the nature of students’ con- structed responses becomes, as you’ll see in the next two chapters, the more attention you’ll need to lavish on scoring. You can’t score responses to items that you haven’t yet written, however, so let’s look now at five guidelines for the construction of essay items.

M07_POPH0936_10_SE_C07.indd 194M07_POPH0936_10_SE_C07.indd 194 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: eeelopment 195

Item-Writing Guidelines for esssay Items

1. Convey to students a clear idea regarding the extensiveness of the response desired.

2. Construct items so the student’s task is explicitly described. 3. Provide students with the approximate time to be expended on each item, as

well as each item’s value. 4. Do not employ optional items. 5. Precursively judge an item’s quality by composing, mentally or in writing, a

possible response.

Communicating the Extensiveness of Students’ Responses Sought It is sometimes thought that when teachers decide to use essay items, students have total freedom of response. On the contrary, teachers can structure essay items so that students produce (1) barely more than they would for a short-answer item or, in contrast, (2) extremely lengthy responses. The two types of essay items that reflect this distinction in the desired extensiveness of students’ responses are described as restricted-response items and extended-response items.

A restricted-response item decisively limits the form and content of students’ responses. For example, a restricted-response item in a health education class might ask students the following: “Describe the three most common ways in which HIV is transmitted. Take no more than 25 words to describe each method of transmission.” In this example, the number of HIV transmission methods was specified, as was the maximum length for each transmission method’s description.

In contrast, an extended-response item provides students with far more lati- tude in responding. Here’s an example of an extended-response item from a social studies class: “Identify the chief factors contributing to the U.S. government’s financial deficit during the past two decades. Having identified those factors, decide which factors, if any, have been explicitly addressed by the U.S. legislative and/or executive branches of government in the last five years. Finally, critically evaluate the likelihood that any currently proposed remedies will bring about significant reductions in the U.S. national debt.” A decent response to such an extended-response item not only should get high marks from the teacher but might also be the springboard for a student’s successful career in politics.

One technique that teachers commonly employ to limit students’ responses is to provide a certain amount of space on the test paper, in their students’ response booklets, or in the available response area on a computer screen. For instance, the teacher might direct students to “Use no more than two sheets (both sides) in your blank-lined blue books to respond to each test item” or “Give your answer in

M07_POPH0936_10_SE_C07.indd 195M07_POPH0936_10_SE_C07.indd 195 09/11/23 12:31 PM09/11/23 12:31 PM

196 ChapTeR 7 Constructed-Response Tests

the designated screen-space below.” Although the space-limiting ploy is an easy one to implement, it really puts at a disadvantage those students who write in a large-letter, scrawling fashion. Whereas such large-letter students may only be able to cram a few paragraphs onto a page, those students who write in a small, scrunched-up style may be able to produce a short novella in the same space. Happily, when students respond to a computer-administered test “in the space provided,” we can prohibit their shifting down to a tiny, tiny font that might allow them to yammer endlessly.

This first guideline asks you to think carefully about whether the test-based inference that you wish to make about your students is best served by students’ responses to (1) more essay items requiring shorter responses or (2) fewer essay items requiring extensive responses. Having made that key decision, be sure to make it clear to your students what degree of extensiveness you’re looking for in their responses.

Describing Students’ Tasks Students will find it difficult to construct responses to tasks if they don’t under- stand what the tasks are. Moreover, students’ responses to badly understood tasks are almost certain to yield flawed inferences by teachers. The most important part of an essay item is, without question, the description of the assessment task. It is the task students respond to when they generate essays. Clearly, then, poorly described assessment tasks will yield many off-target responses that, had the stu- dent truly understood what was being sought, might have been more appropriate.

There are numerous labels used to represent the assessment task in an essay item. Sometimes it’s simply called the task, the charge, or the assignment. In essay items that are aimed at eliciting student compositions, the assessment task is often referred to as a prompt. No matter how the assessment task is labeled, if you’re a teacher who is using essay items, you must make sure the nature of the task is really set forth clearly for your students. Put yourself in the student’s position and see whether, with the level of knowledge possessed by most of your students, the nature of the assessment task is apt to be understood.

To illustrate, if you wrote the following essay item, there’s little doubt that your students’ assessment task would have been badly described: “In 500 words or less, discuss democracy in Latin America.” In contrast, notice in the following item how much more clearly the assessment task is set forth: “Describe how the checks and balances provisions in the U.S. Constitution were believed by the Constitu- tion’s framers to be a powerful means to preserve democracy (300–500 words).”

Providing Time-Limit and Item-Value Guidance When teachers create an examination consisting of essay items, they often have an idea regarding which items will take more of the students’ time. But students don’t know what’s in the teacher’s head. As a consequence, some students will lavish loads of attention on items that the teacher thought warranted only modest effort,

M07_POPH0936_10_SE_C07.indd 196M07_POPH0936_10_SE_C07.indd 196 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: eeelopment 197

and yet will devote little time to items that the teacher thought deserved substan- tial attention. Similarly, sometimes teachers will want to weight certain items more heavily than others. Again, if students are unaware of which items count most, they may toss reams of rhetoric at the low-value items and thus end up without enough time to supply more than a trifling response to some high-value items.

To avoid these problems, there’s quite a straightforward solution—namely, letting students in on the secret. If there are any differences among items in point value or in the time students should spend on them, simply provide this informa- tion in the directions or, perhaps parenthetically, at the beginning or end of each item. Students will appreciate such clarifications of your expectations.

Avoiding Optionality It’s fairly common practice among teachers who use essay examinations to pro- vide students with a certain number of items and then let each student choose to answer fewer than the number of items presented. For example, the teacher might allow students to “choose any three of the five essay items presented.” Students, of course, really enjoy such an assessment procedure because they can respond to items for which they’re well prepared and avoid those items for which they’re inadequately prepared. Yet, other than inducing student jubilation, this optional-items assessment scheme has little going for it.

When students select different items from a menu of possible items, they are actually responding to different examinations. As a consequence, it is impos- sible to judge their performances on some kind of common scale. Remember, as a classroom teacher you’ll be trying to make better educational decisions about your students by relying on test-based interpretations regarding those students. It’s tough enough to make a decent test-based interpretation when you have only one test to consider. It’s far more difficult to make such interpretations when you are faced with a medley of potentially dissimilar tests because you allowed your students to engage in a personal mix-and-match measurement procedure.

In most cases, teachers rely on an optional-items procedure with essay items when they’re uncertain about the importance of the content measured by the items on their examinations. Such uncertainty gives rise to the use of optional items because the teacher is not clearheaded about the inferences (and resulting decisions) for which the examination’s results will be used. If you spell out those inferences (and decisions) crisply, prior to the examination, you will generally find you’ll have no need for optional-item selection in your essay examinations.

Previewing Students’ Responses After you’ve constructed an essay item for one of your classroom assessments, there’s a quick way to get a preliminary fix on whether the item is a winner or a loser. Simply squeeze yourself, psychologically, into the head of one of your typi- cal students, and then anticipate how such a student would respond to the item. If you have time and are inclined to do so, you could even try writing a response

M07_POPH0936_10_SE_C07.indd 197M07_POPH0936_10_SE_C07.indd 197 09/11/23 12:31 PM09/11/23 12:31 PM

198 ChapTeR 7 Constructed-Response Tests

that the student might produce. More often than not, because you’ll be too busy to conjure up such fictitious responses in written form, you might try to compose a mental response to the item on behalf of the typical student you’ve selected. An early mental run-through of how a student might respond to an item can often help you identify deficits in the item, because when you put yourself, even hypothetically, on the other side of the teacher’s desk, you’ll sometimes discover shortcomings in items that you otherwise wouldn’t have identified. Too many times teachers assemble a platoon of essay questions, send those soldiers into battle on examination day, and only then discover that one or more of them are not adequately prepared for the task at hand. Mental previewing of likely student responses can help you detect such flaws while there’s still time for repairs.

In review, we’ve looked at five guidelines for creating essay items. If you remember that all of these charming little collections of item-specific recommen- dations should be adhered to, in addition to the five general item-writing precepts set forth in Chapter 6 (see page 164), you’ll probably be able to come up with a pretty fair set of essay items. Then, perish the thought, you’ll have to score your students’ responses to those items. That’s what we’ll be looking at next.

Essay Items: Scoring Students’ Responses Later in this text, you’ll be looking at how to evaluate students’ responses to perfor- mance assessments in Chapter 8 and how to judge students’ portfolios in Chapter 9. In short, you’ll be learning much more about how to evaluate your students’ perfor- mances on constructed-response assessments. Thus here, to spread out the load a bit, we’ll be looking only at how to score students’ responses to essay items (including tests of students’ composition skills). You’ll find that many of the suggestions for scoring students’ constructed responses you will encounter will also be applicable when you’re trying to judge your students’ essay responses. But just to keep matters simple, let’s look now at recommendations for scoring responses to essay items.

Guidelines for Scoring Responses to esssay Items

1. Score responses holistically and/or analytically. 2. Prepare a tentative scoring key in advance of judging students’ responses. 3. Make decisions regarding the importance of the mechanics of writing prior to

scoring. 4. Score all responses to one item before scoring responses to the next item. 5. Insofar as possible, evaluate responses anonymously.

M07_POPH0936_10_SE_C07.indd 198M07_POPH0936_10_SE_C07.indd 198 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: Scoring Students’ Responses 199

Choosing an Analytic and/or Holistic Scoring Approach During the past several decades, the measurement of students’ composition skills by having students generate actual writing samples has become widespread. As a consequence of all this attention to students’ compositions, educators have become far more skilled in evaluating students’ written compositions. Fortu- nately, classroom teachers can use many of the procedures that were identified and refined as educators scored thousands of students’ compositions during state- wide assessment extravaganzas.

A fair number of the lessons learned about scoring students’ writing samples apply quite nicely to the scoring of responses to any kind of essay item. One of the most important of the scoring insights picked up from this large-scale scoring of students’ compositions is that almost any type of student-constructed response can be scored either holistically or analytically. This is why the first of the five guidelines suggests that you make an early decision about whether you’re going to score your students’ responses to essay items using a holistic approach or an analytic approach or, perhaps, score them using a combination of the two scoring approaches. Let’s look at how each of these two scoring strategies works.

A holistic scoring strategy, as its name suggests, focuses on the essay response (or written composition) as a whole. At one extreme of scoring rigor, the teacher can, in a somewhat unsystematic manner, supply a “general impression” overall grade to each student’s response. Or, in a more systematic fashion, the teacher can isolate, in advance of scoring, those evaluative criteria that should be attended to in order to arrive at a single, overall score per essay. Generally, a score range of 4 to 6 points is used to evaluate each student’s response. (Some scoring schemes have a few more points, some a few less.) A teacher, then, after considering whatever factors should be attended to in a given item, will give a score to each student’s response. Here is a set of evaluative criteria that teachers might use in holistically scoring a student’s written composition.

Illustrstiee eeslustiee Criteris to Be Considered When Scoring Students’ esssay Responses holisticsllay For scoring a composition intended to reflect students’ composition prowess:

• Organization • Communicative Clarity • Adaptation to Audience • Word Choice • Mechanics (spelling, capitalization, punctuation)

M07_POPH0936_10_SE_C07.indd 199M07_POPH0936_10_SE_C07.indd 199 09/11/23 12:31 PM09/11/23 12:31 PM

200 ChapTeR 7 Constructed-Response Tests

And now, here are four evaluative factors that a speech teacher might employ in holistically scoring a response to an essay item used in a debate class.

potentisl eeslustiee Criteris to Be Used When Scoring Students’ esssay Responses in s ebste Clsss For scoring a response to an essay item dealing with rebuttal preparation:

• Anticipation of Opponent’s Positive Points • Support for One’s Own Points Attacked by Opponents • Isolation of Suitably Compelling Examples • Preparation of a “Spontaneous” Conclusion

When teachers score students’ responses holistically, they do not dole out points-per-criterion for a student’s response. Rather, the teacher keeps in mind evaluative criteria such as those set forth in the previous two boxes. The speech teacher, for instance, while looking at the student’s essay response to a question about how someone should engage in effective rebuttal preparation, will not nec- essarily penalize a student who overlooks one of the four evaluative criteria. The response as a whole may lack one factor, yet otherwise represent a really terrific response. Evaluative criteria such as those illustrated here simply dance merrily in the teacher’s head when the teacher scores students’ essay responses holistically.

In contrast, an analytic scoring scheme strives to be a fine-grained, specific point-allocation approach. Suppose, for example, that instead of using a holistic method of scoring students’ compositions, a teacher chose to employ an ana- lytic method. Under those circumstances, a scoring guide such as the example in Figure 7.1 might be used by the teacher. Note that for each evaluative criterion in

Factor Unacceptable

(0 points ) Satisfactory

(1 point) Outstanding

(2 points)

1. Organization ✓

2. Communicative Clarity ✓

3. Adaptation to Audience ✓

4. Word Choice ✓

5. Mechanics ✓

Total Score = 7

Figure 7.1 an Illustrstiee Guide for anslayticsllay Scoring s Student’s Written Composition

M07_POPH0936_10_SE_C07.indd 200M07_POPH0936_10_SE_C07.indd 200 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: Scoring Students’ Responses 201

the guide, the teacher must award 0, 1, or 2 points. The lowest overall score for a student’s composition, therefore, would be 0, whereas the highest overall score for a student’s composition would be 10 (that is, ×2 points 5 criteria ).

The advantage of an analytic scoring system is that it can help you identify the specific strengths and weaknesses of your students’ performances and, there- fore, allows you to communicate such diagnoses to your students in a pinpointed fashion. The downside of analytic scoring is that a teacher sometimes becomes so attentive to the subpoints in a scoring system that the forest (overall quality) almost literally can’t be seen because of a focus on individual trees (the separate scoring criteria). In less metaphoric language, the teacher will miss the communi- cation of the student’s response “as a whole” because of paying excessive atten- tion to a host of individual evaluative criteria.

A middle-of-the-road scoring approach can be seen when teachers ini- tially grade all students’ responses holistically and then return for an analytic scoring of only those responses that were judged, overall, to be unsatisfac- tory. After the analytic scoring of the unsatisfactory responses, the teacher then relays more fine-grained diagnostic information to those students whose unsatisfactory responses were analytically scored. The idea underlying this sort of hybrid approach is that the students who are most in need of fine- grained feedback are those who, on the basis of the holistic evaluation, are performing less well.

This initial guideline for scoring students’ essay responses applies to the scoring of responses to all kinds of essay items. As always, your decision about whether to opt for holistic or analytic scoring should flow directly from your intended use of the test results. Putting it another way, your choice of scoring approach will depend on the educational decisions that are linked to the test’s interpreted results. (Is this beginning to sound somewhat familiar?)

Devising a Tentative Scoring Key No matter what sort of approach you opt for in scoring your students’ essay responses, you’ll find that it will be useful to develop a tentative scoring key for responses to each item in advance of actually scoring students’ responses. Such tentative scoring schemes are almost certain to be revised based on your scor- ing of actual student papers, but that’s to be anticipated. If you wait until you commence scoring your students’ essay responses, there’s too much likelihood you’ll be unduly influenced by the responses of the first few students whose papers you grade. If those papers are atypical, the resultant scoring scheme is apt to be unsound. It is far better to think through, at least tentatively, what you really hope students will supply in their responses, and then modify the scoring key if unanticipated responses from students suggest that alterations are needed.

If you don’t have a tentative scoring key in place, you are very likely to be influenced by such factors as a student’s vocabulary or writing style, even though, in truth, such variables may be of little importance to you. Advance exploration

M07_POPH0936_10_SE_C07.indd 201M07_POPH0936_10_SE_C07.indd 201 09/11/23 12:31 PM09/11/23 12:31 PM

202 ChapTeR 7 Constructed-Response Tests

of the evaluative criteria you intend to employ, either holistically or analytically, is a winning idea when scoring responses to essay items.

Let’s take a look at what a tentative scoring key might look like if it were employed by a teacher in a U.S. history course. The skill being promoted by the teacher is a genuinely high-level one that’s reflected in the following curricular aim:

When presented with a description of a current real-world problem in the United States, the student will be able to (1) cite one or more signifi- cant events in American history that are particularly relevant to the pre- sented problem’s solution, (2) defend the relevance of the cited events, (3) propose a solution to the presented problem, and (4) use historical parallels from the cited events to support the proposed solution.

As indicated, this is no rinky-dink, low-level cognitive skill. It’s way, way up there in its cognitive demands on students. Now, let’s suppose our hypothetical history teacher routinely monitors students’ progress related to this skill by hav- ing students respond to similar problems either orally or in writing. The teacher presents a real-world problem. The students, aloud or in an essay, try to come up with a suitable four-part response.

Figure 7.2 illustrates a tentative scoring key for evaluating students’ oral or written responses to constructed-response items measuring their mastery of this curricular aim. As you can discern, in this tentative scoring key, greater

M07_POPH0936_10_SE_C07.indd 202M07_POPH0936_10_SE_C07.indd 202 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: Scoring Students’ Responses 203

weight has been given to the fourth subtask—namely, the student’s provision of historical parallels to support the student’s proposed solution to the real-world problem the teacher proposed. Much less significance was given to the first sub- task of citing pertinent historical events. This, remember, is a tentative scoring key. Perhaps, when the history teacher begins scoring students’ responses, it will turn out that the first subtask is far more formidable than the teacher originally thought and the fourth subtask appears to be handled without much difficulty by students. At that point, based on the way students are actually responding, this hypothetical history teacher surely ought to do a bit of score-point juggling in the tentative scoring key before applying it in final form to the appraisal of students’ responses.

Deciding Early About the Importance of Mechanics Few things influence scorers of students’ essay responses as much as the mechanics of writing employed in the response. If the student displays subpar spelling, chaotic capitalization, and pathetic punctuation, it’s pretty tough for a scorer of the student’s response to avoid being influenced adversely. In some instances, of course, mechanics of writing play a meaningful role in scoring students’ performances. For instance, suppose you’re scoring students’ written responses to the task of writing an application letter for a position as a reporter for a local newspaper. In such an instance, it is clear that mechanics of writ- ing would be pretty important when judging the students’ responses. But in a chemistry class, perhaps the teacher cares less about such factors when scoring students’ essay responses to a problem-solving task. The third guideline simply suggests that you make up your mind about this issue early in the process so

Tentative Point Allocation

Students’ Subtasks Weak

Response Acceptable Response

Strong Response

1. Citation of Pertinent Historical Events

2. Defense of the Cited Historical Events

3. Proposed Solution to the Presented Problem

4. Historical Support of Proposed Solution

0 pts. 5 pts. 10 pts.

0 pts. 10 pts. 20 pts.

0 pts. 10 pts. 20 pts.

0 pts. 20 pts. 40 pts.

Total Points Possible = 90 pts.

Figure 7.2 an Illustrstiee Tentstiee Scoring Keay for s high-Leeel Cognitiee Skill in historay

M07_POPH0936_10_SE_C07.indd 203M07_POPH0936_10_SE_C07.indd 203 09/11/23 12:31 PM09/11/23 12:31 PM

204 ChapTeR 7 Constructed-Response Tests

that, if mechanics aren’t very important to you, you don’t let your students’ writing mechanics subconsciously influence the way you score their responses.

Scoring One Item at a Time If you’re using an essay examination with more than one item, be sure to score all your students’ responses to one item, then score all their responses to the next item, and so on. Do not score all responses of a given student and then go on to

psrent Tslk The mother of one of your students, Jill Jenkins, was elected to your district’s school board 2 years ago. Accordingly, anytime Mrs. Jenkins wants to discuss Jill’s education, you are understandably “all ears.”

Mrs. Jenkins recently stopped by your classroom to say that three of her fellow board members have been complaining there are too many essay tests being given in the district. The three board members contend that such tests, because they must be scored “subjectively,” are neither reliable nor valid.

Because many of the tests you give Jill and her classmates are essay tests, Mrs. Jenkins asks you whether the three board members are correct.

If I were you, here’s how I’d respond to Jill’s mother. (Oh yes, because Mrs. Jenkins is a board member, I’d simply ooze politeness and professionalism while I responded.)

“Thanks for giving me an opportunity to comment on this issue, Mrs. Jenkins. As you’ve already guessed, I believe in the importance of constructed-response examinations such as essay tests. The real virtue of essay tests is that they call for students to create their responses, not merely recognize correct answers, as students must do with other types of tests, such as those featuring multiple-choice items.

“Your three school board colleagues are correct when they say it is more difficult to score constructed-response items consistently than it is to score selected-response items consistently.

And that’s a shortcoming of essay tests. But it’s a shortcoming that’s more than compensated for by the far greater authenticity of the tasks presented to students in almost all constructed-response tests. When Jill writes essays during my major exams, this is much closer to what she’ll be doing in later life than when she’s asked to choose the best answer from four options. In real life, people aren’t given four- choice options. Rather, they’re required to generate a response reflecting their views. Essay tests give Jill and her classmates a chance to do just that.

“Now, let’s talk briefly about consistency and validity. Actually, Mrs. Jenkins, it is important for tests to be scored consistently. We refer to the consistency with which a test is scored as its reliability. And if a test isn’t reliable, then the interpretations about students we make based on test scores aren’t likely to be valid. It’s a technical point, Mrs. Jenkins, but it isn’t a test that’s valid or invalid; it’s a score-based interpretation about a student’s ability that may or may not be valid. As your board colleagues pointed out, some essay tests are not scored very reliably. But essay tests can be scored reliably and they can yield valid inferences about students.

“I hope my reactions have been helpful. I’ll be happy to show you some of the actual essay exams I’ve used with Jill’s class. And I’m sure the district superintendent, Dr. Stanley, can supply you with additional information.”

Now, how would you respond to Mrs. Jenkins?

M07_POPH0936_10_SE_C07.indd 204M07_POPH0936_10_SE_C07.indd 204 09/11/23 12:31 PM09/11/23 12:31 PM

esssay Itemss: Scoring Students’ Responses 205

the next student’s paper. There is way too much danger that a student’s responses to early items will unduly influence your scoring of the student’s responses to later items. If you score all responses to item number 1 and then move on to the responses to item number 2, you can eliminate this tendency. In addition, the scor- ing will often go a bit quicker because you won’t need to shift evaluative criteria between items. Adhering to this fourth guideline will invariably lead to more con- sistent scoring and, therefore, to more accurate response-based inferences about your students. There’ll be more paper handling than you might prefer, but the increased accuracy of your scoring will be worth it. (Besides, you’ll be getting a smidge of psychomotor exercise with all the paper shuffling.)

Striving for Anonymity Because I’ve been a teacher, I know all too well how quickly teachers can identify their students’ writing styles, particularly those students who have especially distinctive styles, such as the “scrawlers,” the “petite letter-size crew,” and those who dot their i’s with half-moons or cross their t’s with lightning bolts. Yet, insofar as you can, try not to know whose responses you’re scoring. One simple way to help in that effort is to ask students to write their names on the reverse side of the last sheet of the examination in the response booklet. Try not to peek at the students’ names until you’ve scored all of the exams.

I used such an approach for three decades of scoring graduate students’ essay examinations at UCLA. It worked fairly well. Occasionally I was really surprised because students who had appeared to be knowledgeable during class discussions sometimes displayed just the opposite on my exams, while several in-class Silent Sarahs and Quiet Quentins came up with really solid exam performances. I’m sure that had I known whose papers I was grading, I would have been improperly influenced by my classroom-based percep- tions of different students’ abilities. I am not suggesting that you shouldn’t use students’ classroom discussions as part of your evaluation system. Rather, I’m advising you that classroom-based perceptions of students can sometimes cloud your scoring of essay responses. This is one strong reason for you to favor anonymous scoring.

In review, we’ve considered five guidelines for scoring students’ responses to essay examinations. If your classroom assessment procedures involve any essay items, you’ll find these five practical guidelines will go a long way in helping you come up with consistent scores for your students’ responses. And consistency, as you learned in Chapter 3, is something that makes psychometricians mildly euphoric.

In the next two chapters, you’ll learn about two less common forms of constructed-response items. You’ll learn how to create and score performance assessments and how to use students’ portfolios in classroom assessments. You’ll find that what we’ve been dealing with in this chapter will serve as a useful springboard to the content of the next two chapters.

M07_POPH0936_10_SE_C07.indd 205M07_POPH0936_10_SE_C07.indd 205 09/11/23 12:31 PM09/11/23 12:31 PM

206 ChapTeR 7 Constructed-Response Tests

What Do Classroom Teachers Really Need to Know About Constructed- Response Tests? At the close of a chapter dealing largely with the nuts and bolts of creating and scoring written constructed-response tests, you probably expect to be told you really need to internalize all those nifty little guidelines so that when you spin out your own short-answer and essay items, you’ll elicit student responses you can score accurately. Well, that’s not a terrible aspiration, but there’s really a more important insight you need to walk away with after reading the chapter. That insight, not surprisingly, derives from the central purpose of classroom assessment—namely, to draw accurate inferences about students’ status so you

But Whst oes This hsee to o with Tesching? A teacher wants students to learn the stuff that’s being taught—to learn it well. And one of the best ways to see whether students have really acquired a cognitive skill, or have truly soaked up scads of knowledge, is to have them display what they’ve learned by generating answers to constructed- response tests. A student who has learned something well enough to toss it out from scratch, rather than choosing from presented options, clearly has learned it pretty darn well.

But, as this chapter has often pointed out, the creation and scoring of first-rate constructed- response items, especially anything beyond short- answer items, require serious effort from a teacher. And if constructed-response items aren’t first-rate, then they are not likely to help a teacher arrive at valid interpretations about students’ knowledge and/or skills. Decent constructed-response tests take time to create and to score. And that’s where this kind of classroom assessment runs smack into a teacher’s instructional-planning requirements.

A classroom teacher should typically reserve constructed-response assessments for a modest number of really significant instructional outcomes. Moreover, remember, a teacher’s assessments should typically exemplify the outcomes that the teacher wants most students to master. It’s better

to get students to master three to five really high- level cognitive skills, adroitly assessed by excellent constructed-response tests, than it is to have students master a litany of low-level—easy-come, easy-go— outcomes. In short, choose your constructed- response targets with care. Those choices will have curricular and instructional implications.

Putting it another way, if you, as a classroom teacher, want to determine whether your students have the skills and/or knowledge that can be best measured by short-answer or essay items, then you need to refresh your memory regarding how to avoid serious item-construction or response- scoring errors. A review of the guidelines presented on pages 190, 195, and 198 should give you the brushup you need. Don’t assume you are obligated to use short-answer or essay items simply because you now know a bit more about how to crank them out. If you’re interested in the extensiveness of your students’ knowledge regarding Topic Z, it may be far more efficient to employ fairly low-level selected- response kinds of items. If, however, you really want to make inferences about your students’ skills in being able to perform the kinds of tasks represented by excellent short-answer and essay items, then the guidelines provided in the chapter should be consulted.

M07_POPH0936_10_SE_C07.indd 206M07_POPH0936_10_SE_C07.indd 206 09/11/23 12:31 PM09/11/23 12:31 PM

References 207

can make more appropriate educational decisions. What you really need to know about short-answer items and essay items is that you should use them as part of your classroom assessment procedures if you want to make the sorts of inferences about your students that those students’ responses to such items would support.

Chapter Summary After a guilt-induced apology for mislabeling this chapter, we started off with a description of short-answer items accompanied by a set of guidelines (page 190) regarding how to write short-answer items. Next, we took up essay items and indicated that, although students’ written compositions constitute a particular kind of essay response, most of the recommendations for con- structing essay items and for scoring students’ responses were the same, whether measuring

students’ composition skills or skills in subject areas other than language arts. Guidelines were provided for writing essay items (page 195) and for scoring students’ responses to essay items (page 200). The chapter concluded with the sug- gestion that much of the content to be treated in the following two chapters, because those chap- ters also focus on constructed-response assess- ment schemes, will also be related to the creation and scoring of short-answer and essay items.

References Brookhart, S. M., & Nitko, A. J. (2018).

Educational assessment of students (8th ed.). Pearson.

Hawe, E., Dixon, H., Murray, J., & Chandler, S. (2021). Using rubrics and exemplars to develop students’ evaluative and productive knowledge and skill, Journal of Further and Higher Education, 4(8): 1033–1047. https:// doi.org/10.1080/0309877X.2020.1851358

Hogan, T. P. (2013). Constructed-response approaches for classroom assessment. In J. H. McMillan (Ed.), SAGE Handbook of research on classroom assessment (pp. 275–292). SAGE Publications.

Lane, S., Raymond, M. R., & Haladyna, T. M. (Eds.). (2016). Handbook of test development (2nd ed.). Routledge.

Miller, M. D., & Linn, R. (2013). Measurement and assessment in teaching (11th ed.). Pearson.

Stiggins, R. J., & Chappuis, J. (2017). An introduction to student-involved assessment FOR learning (7th ed.). Prentice Hall.

To, J., Panadero, E., & Carless, D. (2021). A systematic review of the educational uses and effects of exemplars, Assessment & Evaluation in Higher Education. https://doi. org/10.1080/02602938.2021.2011134

M07_POPH0936_10_SE_C07.indd 207M07_POPH0936_10_SE_C07.indd 207 09/11/23 12:31 PM09/11/23 12:31 PM

208 ChapTeR 7 Constructed-Response Tests

A Testing Takeaway

Evaluating Student-Constructed Responses Using Rubrics* W. James Popham, University of California, Los Angeles

Two prominent categories of students’ responses account for practically all items found in educational tests, namely, selected-response items and con structed-response items. Whereas selected- response items are particularly efficient to score, they sometimes constrain the originality of a test-taker’s response. In contrast, although constructed-response approaches such as essay and short-answer items provide test-takers with ample opportunities to invoke their originality, the scoring of students’ responses frequently represents a formidable challenge.

One strong argument in favor of constructed-response items is that the challenges they embody often coincide directly with what’s required of people in the real world. Although a friend may often ask you to “explain your reasons” for supporting a politician’s proposed policy changes, rarely does that friend solicit your opinion by providing you with four possible options from which you must choose only one.

Given the potentially wide range of students’ constructed responses, scoring those responses is a substantial task. If a satisfactory job of scoring students’ responses to essay or short-answer items cannot be done, of course, it would be better never to employ such items in the first place.

For the last several decades, educators have evaluated the quality of students’ constructed responses by employing a rubric. Such rubrics—that is, scoring guides—help identify whether a student’s response is dazzling or dismal. A rubric that’s employed to score students’ responses has, at minimum, three features:

• Evaluative criteria. The factors used to judge the quality of a student’s response

• Descriptions of qualitative differences for all evaluative criteria. Clearly spelled-out descriptions of qualitative distinctions for each criterion

• Scoring approach to be used. Whether the evaluative criteria are to be applied collectively (holistically) or on a criterion-by-criterion basis (analytically)

The virtue of a rubric is that it brings clarity to those who score a student’s responses. But also, if provided to students early in the instructional process, rubrics can clarify what’s expected. Yet not all rubrics are created equal. Here are two losers and one winner:

• Task-specific rubric. This loser focuses on eliciting responses to a single task, not demonstrat- ing mastery levels of a skill.

• Hypergeneral rubric. Another loser, its meanings are murky because it lacks clarity.

• Skill-focused rubric. This winner addresses mastery of the skill sought, not of an item.

Thus, when working with rubrics, either to teach students or to score their work, always remember that even though many rubrics represent real contributors to the educational enterprise, some simply don’t.

*From Chspter 7 of Classroom Assessment: What Teachers Need to Know, 10th ed., bay W. Jsmes pophsm. Copayright 2022 bay pesrson, which herebay grsnts permission for the reproduction snd distribution of this Testing Takeaway, with proper sttribution, for the intent of incressing sssessment literscay. a digitsllay shsresble eersion is sesilsble from httpss://www.pesrson.com/store/en-us/pesrsonplus/login.

M07_POPH0936_10_SE_C07.indd 208M07_POPH0936_10_SE_C07.indd 208 09/11/23 12:31 PM09/11/23 12:31 PM