Cultural Validity in Assessment

Cultural_Validity_in_Assessment_Addressing_Linguis..._----_Part_II_Assessing_Students_of_Linguistically_and_Culturally_Diverse_Ba....pdf

Home >Education homework help >Cultural Validity in Assessment

Chapter 5

Using Appropriate Assessment Processes How to Get Accurate Information about the Academic Knowledge and Skills of English Language Learners

Rebecca Kopriva and Ursula Sexton

Chapter Overview

This chapter focuses on approaches teachers can use to collect information about how students who are English language learners (ELLs) are thinking, understand- ing, and developing skills in mathematics, science, or other academic content areas. The chapter does not address how English language proficiency should be assessed. Rather, it addresses how English proficiency and emergent language skills should be taken into account when assessing academic content among ELLs, and the strat- egies and communication supports they need to have in order to demonstrate their understandings. Therefore, this chapter will discuss some key aspects of classroom assessment that may be particularly effective for this population, as well as provide an overview of best practice. Since the chapter is focused entirely on what teachers can do, there is no separate section on that topic, as is the case for other chapters in this volume.

The premise of this chapter is that classroom assessment is primarily about informing instruction and that it actively involves and benefits both teachers and students through the use of ongoing assessment activities that include:

• Developing promising instructional situations where inquiry into and explo- ration of the students’ grasp of the content and concepts is embedded in learning;

• Developing effective ways of asking questions or otherwise obtaining infor- mation about the students’ knowledge and skills at particular time points and, if possible, the specific reasons that hamper their learning;

• Implementing the situations and assessment opportunities in a manner that is likely to render reasonably accurate information; and

• Analyzing student responses to inform instruction and inform students about where they need to focus their efforts, what they need to think about differently, or otherwise take next steps in interacting with future learning opportunities.

Basterra, M. D. R., Trumbull, E., & Solano-Flores, G. (Eds.). (2010). Cultural validity in assessment : Addressing linguistic and cultural diversity. Retrieved from http://ebookcentral.proquest.com Created from wayne on 2019-11-20 16:37:28.

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 97

Assumptions in this chapter are that students will be taught content that is at grade level (knowing that remediation may be part of the challenge) and that the state’s (and perhaps also the district’s) content standards will be specifying what is being taught in mathematics, science, and the other academic subject areas. These assumptions, then, put teachers of ELLs on the same page as what most other teachers are expected to teach to most of the students in their district. This parallel status can and should be used to keep communication lines open between teachers and between students from different classrooms.

What Teachers Can Do

What You Need to Consider Before You Start

Planning is Key

The process of designing an integrated and potentially successful classroom assessment system throughout the year requires a teacher’s thorough analysis of the planning process and an integration of learning and assessment activities. This includes reflecting upon and planning from the beginning of the instruction unit about how to effectively develop assessment activities that produce informa- tion on how students are benefitting from instruction and how the teacher might adapt further instruction to respond to assessment results. It also includes how to evaluate the value of any existing, commercially available assessments and those included in textbooks being used in the program. All in all, effective plan- ning is about how to bridge gaps between teaching practices, assessment prac- tices, and teacher’s adaptations based on student responses to the assessment queries. As part of planning, through the use of grade level articulation and the guide- lines in the state or district content frameworks, a teacher may take into consid- eration questions like the ones below, to guide the conceptual flow of a lesson or series of lessons, and to bring into focus the type of assessments and criteria that will be used. For instance:

• What was taught in the previous year, month, or week? What will be taught in the next year, month, or week?

• What concepts come before or after this concept is introduced? • How can it be contextualized? How will I teach and assess meaningful

content?

Having a clear picture of how a specific lesson or activity fits within the con- ceptual flow of the year-long academic curriculum entails looking more closely within each instructional unit and facilitating the effective placement of assess- ment activities within specific concepts. Teachers should address issues such as:

• How does a lesson or activity fit into the conceptual flow of ideas in the unit?

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

98 R. Kopriva and U. Sexton

• What precedes it? What follows it? What knowledge do students have prior to beginning instruction?

• What should I expect to hear and see while a task is performed and a con- cept is addressed? Answers to this question become the criteria for scoring student responses.

• In order to verify if the task and its response format are effective as assess- ment tools, how can I plan to make use of student work to inform both instruction and students on their progress?

Another important part of planning is identifying the types of steps to take after the implementation of the lessons and assessment queries have been com- pleted. Issues to address at this stage include:

• What should students know after a topic is taught? • What have we (teacher and students) gained from the analysis of student

work? • Were the student assessment tasks and the scoring criteria adequate? (This

topic is also addressed in Durán, Chapter 6, this volume.) • Were assessment formats and approaches relevant and meaningful? • Are my students involved in the development of assessment criteria and the

feedback processes? How? When? • What other student needs must be identified and reflected in adaptations for

future instruction and assessment activities?

A discussion of some kinds of assessment tasks and queries, scoring guide- lines, and what types of tasks to use to collect what types of information are out- lined below. It is recommended that teachers use tips offered in this section when making planning decisions about how to integrate teaching and assessment activities within particular types of lessons.

Plan to Teach and Assess the Full Range of Content Complexity

In many professional discussions surrounding ELLs, we hear: “Just because a child doesn’t speak much English doesn’t mean she is not thinking in complex ways.” Nonetheless, as educators we seem to fall short in following this conver- sation with the necessary instructional preparation that provides opportunities in the classrooms for students to engage in and communicate complex think- ing. This continues to be true when we consider the kinds of assessment oppor- tunities we use in the classroom in order to assess actual student understanding and learning. Consequently, teachers need to seek and have tools that allow them to be the most effective facilitators of knowledge in their classroom instruction and ongoing assessment of student progress. The type and level of instructional opportunities that invite more cognitively complex learning and the types of questioning associated with the range of cognitive inquiry should be part of all lessons in all content areas. Bloom and Krathwohl’s (1956) taxon- omy, new Bloom’s taxonomy (Anderson & Krathwohl, 2001), and the taxon-

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 99

omy of types of knowledge developed by Li, Ruiz-Primo, and Shavelson (2006), and shown in Table 5.1 on page 106 (with adaptations), may be helpful for that purpose. Practices that encourage these kinds of learning occasions need to utilize a variety of participatory structures that promote delving into and exploring the complexity of problems during instruction and allow for frequent opportuni- ties for students to engage in dialogue and self-analysis and to listen to the teacher.

Plan for Diversity

One part of planning that is often overlooked and undervalued is a careful assess- ment of the strengths and challenges of individual students. The effectiveness of instruction and meaningfulness of assessment results are directly related to how well this step is carried out. Lesson and assessment planning that considers how the students will benefit from the material and the questions and assessment response criteria that guide thinking about what students mean when they respond in particular ways should prove to be more useful in sustaining learning. Such considerations include:

• How students’ different cultural backgrounds affect how they interpret what is being taught (this topic is also addressed in Basterra, Chapter 4, this volume);

• The experiential knowledge associated with the lessons that the students bring into the classroom;

• The tools for delivery; • The process and participatory structures for classroom interaction that best

match students’ socialization; • Learning styles; and • Students’ input.

Educators encourage teachers to understand what prior knowledge and skills students bring to the instructional tasks in order to gain insight into the learning process. These include both knowledge and skills directly related to what is being taught and other types of information and expertise that could facilitate or hamper learning. Additional questions to consider when planning instruction and assessment alignment are:

• What do the students know about the targeted content? • What level of language development do they have? What level of academic

language do they use? • Are there considerations that will act as barriers to their learning? • How else might I be able to communicate content with them, and how else

might they be able to communicate with me about what they know, without changing the rigor and cognitive complexity of what I want to teach and assess?

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

100 R. Kopriva and U. Sexton

A Note About Teacher Knowledge

A body of research emphasizes that their own expertise in a content area and a deep understanding from different perspectives of the information being taught have a substantial impact on teachers’ ability to effectively and flexibly commu- nicate with students. When teachers know the content so as to be able to estab- lish cross-disciplinary connections, when they can map their expectations backwards and break down the cognitive demand into smaller “chunks” of infor- mation, their instruction often reflects their expertise (Ball & Bass, 2000). In con- trast, when teachers are not as confident about their knowledge of the subject matter, they tend to depend solely on textbooks or materials not critically reviewed. Instruction is, then, watered down or weakened. This situation can serve as a self-assessment indicator that points to the need for additional support from mentors and professional development. The same is true for assessment. When teachers lack expertise in assessment, they seem to more often rely on textbook-based tests and pre-packaged test items. They tend to not know what to do with their students’ responses to these items, and further instruction seems detached from assessment results. On the other hand, teachers who know the content deeply and have interest in communicating effectively with their students integrate many “mini-feedback loops” into their instructional tasks and keep on top of how students are hearing what they are trying to convey. Further, their teaching reflects their responsive- ness based on what students are saying. Knowing that students communicate in varied ways, and that various levels of cognitive complexity in the targeted subject matter are often communicated differently as well, these teachers have and are able to use a large repertoire of practices when they assess their students (DiRanna et al., 2008; Basterra, Chapter 4, this volume; Lee et al., Chapter 11, this volume). Finally, teachers who know the content deeply and have an interest in com- municating effectively with their students are familiar with common misconcep- tions and naive and unsophisticated skills; and they have thought about optimal ways to interact with students in order to effectively facilitate future learning. As such, in addition to finding out whether a student meets certain target learning goals, these teachers have ways of finding out how their students are processing what is being taught (Darling-Hammond & Bransford, 2005) and integrate opportunities into instruction that allow students to learn from their mistakes.

The Nuts and Bolts of an Effective Academic Classroom Assessment Program

What Does an Effective Program Look Like?

The development of best practices in teaching most academic subjects (e.g., mathematics, science, and social studies) has been debated and shaped over the last 20 years. Most educators today believe that students learn best through first- hand experience and by being provided with ample opportunities to engage in

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 101

discourse and connect new information to what they already know through social interaction. Solely listening to a teacher lecture or reading a textbook are not rich enough experiences that enable them to learn. Most educators today understand the importance of providing ample opportunities to engage in dis- course and in iterative and interactive skill development to connect new infor- mation to what students already know. This applies not only to learning rote and factual knowledge, but also to the development of higher-order thinking and rea- soning skills associated with various types of content (for example, see Brans- ford, Brown, & Cocking, 2000; Duschl, Schweingruber, & Shouse, 2007; Yager, 2005). Recently, interest has surged in the use of learning progressions in assessment (see Shavelson, 2009). Serious efforts have been made to design test items in ways in which their features are linked to models of cognitive development for the constructs assessments are intended to measure. For example, each of the options of multiple-choice items can be designed to reflect different levels of development of knowledge and understanding of the construct being measured, which allows for proper interpretation of the cognitive correlates of student per- formance (see Briggs, Alonzo, Schwab, & Wilson, 2006). This notion can be used by teachers in building lessons and assessment tools, and aligning them throughout the year. An example of the use of the notion of learning progressions in classroom assessment can be found in the FOSS science curriculum products developed by the University of California, Berkeley’s Law- rence Hall of Science and in the work completed recently by the K-12 education arm of the American Association for the Advancement of Science (AAAS). In alignment with these pedagogical advances, assessment experts encourage educators to design assessments that provide information about their students’ thinking and problem-solving strategies and examine how teachers might track factual, reasoning, and processing skills in their students (e.g., Pellegrino, Chu- dowsky, & Glaser, 2001; Basterra, Chapter 4, this volume). Instructionally embedded items developed by teachers tend to focus on recall, rather than the application or development of ideas. Few of these items can elicit students’ higher-order thinking. Figure 5.1 provides an example of how some teachers approached learning about their students’ knowledge and skills at various levels of inquiry. Sometimes, assessment tasks are developed to capture information about student learning from the students’ errors as well as their successes. This diag- nostic information can then be used to confirm that students appear to have learned particular pieces of knowledge and mastered certain concepts or skills. This information can also be used to inform future instruction in order to correct misconceptions that may be responsible for incorrect responses. Black, Wiliam, Stiggins, and Popham, among others, have written extensively about these kinds of classroom assessment tasks and provide many examples (e.g., see Popham, 2006; Stiggins, 2003). Ongoing classroom assessment opportunities that gauge the accomplishments and errors of individual students explicitly link feedback to student performance and provide students with strategies for improvement are called “formative assessment” (see Shepard, 2006).

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

102 R. Kopriva and U. Sexton

The illustration below shows several organisms within an ecosystem.

Squirrel

Mushrooms

Bear

Hawk Snake

Grass

Coyote

FlowerRabbit

Butterfly

Use the illustration above and what you know about the food chain to 1. complete the blanks below.

Select one organism, and connect it to the food chain category. Briefly explain each category’s function:

The mushroom is a decomposer that feeds off dead organisms (organism) (category) (function)

The is a omnivore that eats plants and meat (organism) (category) (function)

The is a carnivore that (organism) (category) (function)

The is a producer that (organism) (category) (function)

The is a herbivore that (organism) (category) (function)

Put each category of the food chain in sequential order.2.

Food chain

Categories: Carnivore Decomposer Herbivore Omnivore Producer

5 Producer

Using all of the previous information, what effect would it have in the ecosys-3. tem if one of the categories of the food chain was eliminated or destroyed? Explain in the space below using words or pictures.

Figure 5.1 An Example of a Multi-Step Assessment Task (source: Dual Language Concurrent Assessment Development for English Learners—Science; NSF: REC 0352148, SGER Grant).

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 103

Informal and formal formative assessment opportunities aimed at evaluating learning of the full range of concepts and skills should be integrated frequently into ongoing instruction (see Ruiz-Primo & Furtak, 2007). These assessments should be explicitly and purposefully considered and developed as part of the planning process. Teachers should plan these assessment activities based on:

• Deeply knowing the content (including pitfalls and students’ common mis- takes);

• Learning to anticipate what is expected of students through assessing stu- dents’ prior knowledge and their communication strengths and challenges;

• Performing regular observations during and between lessons and making proper adjustments as they go; and

• Critically analyzing student work from current opportunities in order to adapt future instruction and assessment plans.

Well-designed and varied assessments of student learning can serve to rein- force effective instructional practices, as teachers begin to better understand how and when to use assessments as an integral part of the curriculum. Figuring out what to collect will be discussed briefly below. How to collect the proper data about students’ knowledge and strategies is multifaceted and will be discussed in more detail. Different forms of assessment provide different kinds of information on student content knowledge, skills, and understanding. The same way as no single form of instruction provides the best way to teach all content and skills, no single form of assessment can provide all the evidence of student learning (Basterra, Chapter 4, this volume). As is widely acknowledged, pre-packaged tests from textbooks, well-meaning curriculum companies, the Internet, and other teachers abound. In critiquing these tests or in developing your own, the focus should be on building tasks that fit your instruction and properly capture information about whether the student knows the content and the processes underlying incorrect responses. The issue is what type of task (e.g., multiple-choice items or embedded performance tasks) measures the targeted content or skills effectively. There is great value in examining students’ responses to assessment tasks with the purpose of determining how they can be improved to increase their effective- ness as formative assessment tools. In addition, asking a wide range of students about their interpretation of the tasks and the reasonings they use when they respond to them helps students become aware of their own learning, and helps teachers to improve their instruction (Bell & Cowie, 1999). In schools where teachers are encouraged and supported to work collaboratively in these activities, analyzing student work within grade levels becomes an excellent embedded pro- fessional development opportunity.

Know What You Want to Assess

Assessment should not be an afterthought or take place as an experience isolated from the rest of the instruction (Stiggins, 2003). Thinking about how we want to

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

104 R. Kopriva and U. Sexton

evaluate students’ needs should occur as a part of our instructional planning and should be integrated and aligned in purpose with what and how the students are supposed to learn. As educators, before we choose or develop any assessment task, we need to have a clear idea of the forms of knowledge we want to assess. Knowing what we want to assess comes directly from the desired student learn- ing outcomes in the lesson plans which state what students are supposed to learn from instruction. Because of the pervasive use of multiple-choice items in today’s standardized tests, textbooks, and other materials, many of us may have been conditioned to believe that these kinds of items suffice to measure all the forms of knowledge or skills, or we may assume that published items are better than anything we could design. However, rarely will commercially available, off-the-shelf multiple-choice items fit our specific classroom assessment needs. In deciding what we want to assess, we must focus on the knowledge and skills instruction is supposed to support. For instance, when students are engaged in science investigations, we may be interested in promoting among them the use of inquiry through social interaction. To measure these kinds of skills, we may want to learn how well students are able to organize, categorize, predict, or communicate information, how materials are handled, or how a problem is being approached and solved. Additionally, we may want students to learn spe- cific scientific concepts used in their investigations. We may want to know about students’ learning of specific content at a recall or declarative level; how students are interpreting systems of conceptual information; how well they can manipu- late and interpret data within the context of the concepts; and how well they are making substantive connections within or across disciplines as they discuss, summarize, and record their findings. It is important to remember that all students—not just older ones or native English speakers—should be challenged routinely with instructional and assess- ment activities that promote higher-order thinking skills. The examples in Figure 5.1 include measuring more complex or higher-order reasoning. In this instance, students are asked to predict what might happen to an ecosystem from their understanding of a given set of environmental conditions.

What to Use When You Want to Assess Different Kinds of Knowledge and Skills

Assessment tasks range from traditional multiple-choice and short-answer items to essay and other constructed-response tasks that engage students in designing, conducting, and reporting the results of investigations, projects, or experiments. Formal and informal assessment may include protocols by which teachers observe students as they engage in solving problems, using arguments to build their cases or participating in discussions that involve student–student or teacher–student interactions. Performance is assessed according to certain criteria, according to scoring rubrics, which are familiar to many teachers. Assessment tasks that go beyond short answers require students to demon- strate what they know and are able to do—and how they can apply their

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 105

content knowledge to new problems. Teachers may choose to use and adapt pre-packaged assessment tasks. Or, they may opt to develop their own tasks individually or collaboratively, which constitutes a valuable professional devel- opment activity. Below is an example of a knowledge-based approach for measuring science learning (see Ruiz-Primo, 2007). The framework proposes the existence of four types of knowledge:

1. Declarative knowledge (D) includes facts, terms, and definitions that can be learned and held in memory as words, images, or sensory representations. It can also include statements such as “CO2 is carbon dioxide.” Declarative knowledge can be described as “knowing that.”

2. Procedural knowledge (P) entails if–then rules or sequences of steps such as in algorithms. Examples include reading data tables or designing experi- ments. Importantly, a central ingredient of procedural knowledge is that, with practice, individuals can convert it into automatic responses. Pro- cedural knowledge can be described as “knowing how.”

3. Schematic knowledge (Sc) typically entails the application of scientific prin- ciples or explanatory models and knowledge that guides actions, trouble- shoots systems, or predicts the effect of changes. As examples of this form of knowledge, Li et al. (2006) mention “explaining why the moon generally rises about 50 minutes later each day or how a particular virus functions” (p. 292). Schematic knowledge can be described as “knowing why.”

4. Strategic knowledge (St) entails the proper application of knowledge. Exam- ples of this form of knowledge are strategies such as using skills to represent problems or deal with certain tasks unknown to the individual, general monitoring abilities or planning strategies—for example, dividing a task into sub-tasks or integrating different pieces of information. Strategic knowledge can be described as “knowing when, where, and how.”

The framework, presented in Table 5.1, shows the link between type of knowl- edge and type of assessment task. The first and second columns show four basic types of tasks and their characteristics. The third column shows examples of spe- cific activities corresponding to those tasks. The fourth column describes the types of knowledge to which those types of tasks are sensitive. As an example, the first sub-category of task demands is “defining/using terms.” This sub-category is best associated with measuring declarative knowl- edge. Declarative knowledge is also associated with the cognitive demand sub- category “recalling facts.” As the table recommends, “selecting responses” under item openness is good for declarative knowledge as well. The table does not tell us directly what question or task type (multiple-choice, true/false, extended constructed response, performing an experiment, etc.) is best for assessing which kind of knowledge. Rather, it allows the reader to think in terms of knowledge purpose, and then find or build an item that fits those task characteristics. For instance, if we are interested in assessing how well students can reason using models, we could build or find a multiple-choice item that

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Ta bl

e 5

.1 K

n o

w le

d ge

-B as

e d

F ra

m ew

o rk

f o

r Id

e n ti

fy in

g P

ro p e r

Q u e st

io n s

o r

T as

k T

yp e s

A sp

ec ts

o f

ta sk

s D

es cr

ip ti

on Se

le ct

ed e

xa m

p le

s of

c od

in g

su b-

ca te

go ri

es Po

ss ib

le l

in ks

w it

h kn

ow le

dg e

ty p e

T as

k d

e m

an d

s T

as k (s

) th

at s

tu d

e n ts

a re

a sk

e d

t o

p e rf

o rm

• D

e fi

n in

g/ u si

n g

te rm

s •

P ro

vi d

in g

e x p la

n at

io n s

• D

e si

gn in

g an

e x p e ri

m e n t

• M

at h e m

at ic

al c

al cu

la ti

o n

• D

/S c

• P

C o

gn it

iv e d

e m

an d

s In

fe rr

e d

c o

gn it

iv e p

ro ce

ss e s

st u d

e n ts

l ik

e ly

a ct

u p o

n t

o p

ro vi

d e

re sp

o n se

• R

e ca

ll in

g fa

ct s

• R

e ad

in g

fi gu

re s

• R

e as

o n in

g w

it h m

o d

e ls

• C

o m

p ar

in g

an d

s e le

ct in

g st

ra te

gi e s

• D

• P

• Sc

• St

T as

k o

p e n n e ss

D e gr

e e o

f fr

e e d

o m

s tu

d e n ts

a re

lo w

e d

t o

h av

e i

n s

h ap

in g

re sp

o n se

s to

a n i

te m

o r

ta sk

• Se

le ct

in g

vs .

ge n e ra

ti n g

re sp

o n se

s •

St e m

p re

se n te

d a

s a

co m

p le

q u e st

io n v

s. a

n i

n co

m p le

te s

te m

• O

n e v

s. m

u lt

ip le

c o

rr e ct

a n sw

e rs

• R

e ci

p e d

ir e ct

io n s

• Se

le ct

in g

go o

d f

o r

D •

In co

m p le

te s

te m

g o

o d

f o

r D

, b u t

n o

t go

o d

f o

r Sc

/P /S

t •

O n e g

o o

d f

o r

D v

s. m

u lt

ip le

o d

f o

r Sc

/S t

• N

o t

go o

d f

o r

A d

d it

io n al

f ac

to rs

D o

m ai

n -g

e n e ra

l fa

ct o

rs t

h at

fl u e n ce

t h e i

te m

d if

fi cu

lt y

• L in

gu is

ti c

ch al

le n ge

s •

T e x tb

o o

k v

s. n

o ve

l ta

sk s

• In

cl u si

o n o

f ir

re le

va n t

in fo

rm at

io n

• N

o t

go o

d f

o r

an y

ty p e

• T

e x tb

o o

k g

o o

d f

o r

D v

s. n

o ve

l go

o d

f o

r St

• G

o o

d f

o r

So u rc

e :

A d

ap te

d f

ro m

: L i

R u iz

-P ri

m o

, an

d S

h av

e ls

o n (

2 0 0 6 ).

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 107

allows students to “select all that apply” of several options, as long as the options have been carefully designed to provide various kinds of partial information. Only certain combinations of selected responses would receive full credit, but partial credit could be given if the student selected certain other types of combinations. The American Association of the Advancement of Science has a large pool of available items like these. You can build an open-ended question that requires students to show their reasoning by writing about it or drawing diagrams. If lan- guage is still needed, teachers of ELLs would need to think about how they could support their students in explaining themselves. For instance, they may be able to read the students’ code-switching explanations, they may need these students to orally explain their diagrams until the teacher is satisfied, or the teacher can ask them to respond to other items to see if the student has a firm grasp on the targeted reasoning. This topic is also addressed in Trumbull and Koelsch, chapter 9, this volume. In each case, what would not be acceptable is a fact-finding item, as students could “differentiate” or “guess” their way through without necessarily having to demonstrate their grasp of the reasoning skills. As borne out by research findings, the authors of this table argue that a given set of skills and competencies is best measured with a given, optimal combina- tion of tasks. Also, improper use of certain tasks could restrict the students’ ability to demonstrate knowledge. For ELLs, developing appropriate tasks would certainly entail consideration of the kinds of information found in Table 5.1. In addition, other factors need to be considered. One of them is the set of charac- teristics, such as language load or cultural references, that may hamper students’ understanding of what the task is asking or the opportunity for them to respond in ways that accurately reflect their competencies (see Trumbull & Solano-Flores, Chapter 2; Abedi, Chapter 3, this volume). So, what does all this look like in an ongoing classroom setting? Teachers are responsible for planning and implementing the instructional program, managing the dynamics of a classroom filled with students, and tracking and reporting the educational progress of both individual students and the class as a whole. Forma- tive assessment of student learning in the classroom is an integral part of these complex responsibilities, but teachers have to understand how to make the most of each instructional activity and where the assessment component could often be integrated into the teaching. Each type of task, whether it is a multiple-choice question, an open-ended task or a hands-on experiment, can only give partial information about the stu- dents’ skills or knowledge. A vast array of approaches—including opportuni- ties for students to “think aloud” during some activities, integrated sessions where students analyze their own work or other students’ work, as well as informal and formal teacher-led questioning (both orally or on paper)—are all opportunities for teachers to systematically collect and subsequently evaluate information about pre-planned targeted knowledge and skills (Basterra, Chapter 4, this volume). Additionally, many teachers have found it useful to work aloud with students when they are developing their assessment criteria and scoring rubrics.

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

108 R. Kopriva and U. Sexton

Several states, such as California, have put in place periodic Program Quality Review systems, which allow for reflective practices on behalf of teachers and administrators to come together in order to make meaningful and timely modifications in the school’s curricular area of focus. School edu- cators can use opportunities such as this to collectively think about assessment strategies associated with specific curricular goals or to analyze student work together. While time seems in very short supply, some experts suggest that the primary hurdle is learning how to evaluate individual students creatively, effectively, and frequently, and to do so in a way in which teachers learn not only what students know, but also identify misunderstandings or missing important knowledge or skills. As discussed above, this process, and how teachers adapt future instruction to reflect and to correct misunderstandings, are the ultimate goals of formative assessment.

Tips for Measuring Content Knowledge in ELLs

BefORe yOu STaRT

Given that each content area and discipline has its own register (academic lan- guage, including vocabulary and discourse style), the teaching of content lan- guage and academic concepts and processes require attention to detail for all students, but especially for ELLs (Trumbull & Solano-Flores, Chapter 2, this volume). Teaching both language and content is key for understanding and using the discourse of the subject being learned. Capitalizing on teachers’ passion for their subject matters, teachers who are experts in teaching ELLs could work with these teachers and educate them about the best ways to support their ELL students as they are learning academic language along with content. Thus, a first step in promoting effective assessment for ELLs consists of doing collaborative work that ensures that these students are included in each classroom discussion and take part in the wide range of formative assessment activities discussed above. That said, it is vitally important to remember that teaching and assessing the full range of knowledge and skills must be available for ELLs as well as for non- ELLs. It is often easier to restrict the range of instruction and testing of ELLs to lower levels of knowledge because the language tends to be more concrete, simpler in structure, and more reliant on high-frequency words. However, doing this keeps these students from developing more complex thinking and reasoning skills, in general, and in each specific content area. Each school year that this occurs, ELLs fall further behind, making it harder, if not impossible, for them to catch up. Further, accountability systems (including classroom grades and school benchmark tests) that allow this to happen provide distorted and misleading information about these students’ learning and ability to learn. A false sense of mastery will become evident later when students cannot compete in more chal- lenging classes and end up being shunted into lower academic tracks or even special education (Garcia & Ortiz, 2006). The second objective, then, of assessing the knowledge and skills of ELLs is not to let these students “off the hook” but,

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 109

instead, make sure they receive the full range of instruction and are held account- able for demonstrating their learning relative to that range.

NOw whaT?

How can formative assessment tasks be designed to retain the rigor and variation of the relevant content while still being accessible to ELLs? To date, most of the published assessment tasks being designed to collect these types of formative data on student learning tend to depend largely on the use of text to ask ques- tions and communicate meaning and context. While some tasks let students demonstrate their responses in a limited number of alternative ways, for the most part they assume students can comprehend the nuances and subtle seman- tic distinctions in the principled but wordy multiple-choice options, or they require them to use expressive language skills to write textual responses that explain their thinking. This problem pertains to not only assessment tasks pre- sented in static, paper-and-pencil tests but also most of the computer-based tasks that are presented in engaging and dynamic interactive environments. So, how can formative assessment tasks be properly designed for ELLs? This question has many answers, depending on the students’ challenges and strengths. Instead of avoiding the kinds of tasks being suggested in a good, for- ward-looking classroom where formative feedback is part of learning, the chal- lenge is to figure out how to adapt these materials. First, an assessment of students’ individual strengths (for instance, their primary language [L1] profi- ciency in general, literacy in L1 if available, compensatory learning styles, pre- vious learning experiences, and even perhaps “street smarts”) is essential and can be implemented across subject areas. An assessment of ELL students’ chal- lenges is also vital, for example, beyond their English proficiency in the four domains (speaking, listening, reading, and writing). For example, a full assess- ment would include inquiry about their home communities’ assumptions about testing that may (or may not) be inconsistent with ongoing assessments used in the U.S., a restriction of assessment experiences in previous grades in the U.S. or in their home country, or a lack of previous instructional experi- ences related to particular curricula (Abedi, Chapter 3, this volume). Second, once these assessments are completed for each student (if possible by the ELL expert in the school) and explained to other teachers as relevant, the ELL expert can develop and use various adaptations that do not lessen the complex- ity of the content. If possible, this professional can work with the content-area teachers to share these adaptations and identify other suitable solutions. For instance, teachers can use:

• “Sheltered English”; • L1 resources from the community for oral help (for benchmarks, scripts

should be reviewed by others, and then recorded and used that way, so that cueing is not an issue);

• Bilingual glossaries (if students are literate enough in either L1 or in English to benefit);

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

110 R. Kopriva and U. Sexton

• Examples of ways students can demonstrate knowledge, so that they do not always have to write about it;

• Examples of ways to score responses that allow for pictures, code-switching, etc.

Often adaptations can be used in tandem. A few guidelines continue below:

1. The simple act of providing an “opening” instructional activity that is engag- ing and that involves students in dialogue, discovery, and communication allows the teacher to monitor emergent language; to obtain a basal assess- ment of students’ understanding of concepts; and to assess prior knowledge in context with the concepts at hand. By using a task-based engaging intro- duction, the teacher is able to tap into the students’ linguistic and conceptual reservoirs, while listening attentively to their dialogue during the engaging task (i.e., monitoring, roving, probing, and recording key student ideas and language). Then, the teacher is able to bridge and provide tools (i.e., word/ concept organizers) for learning, to clarify ideas, to model and apply the les- son’s intended academic language, while providing contextualized cues, visuals, and ensuing multiple opportunities for their use.

2. Contrary to popular opinion, ELLs do not necessarily like or do well with “language-less” assessment tasks. Rather, we recommend that teachers use context wisely, so that students can understand the meaning of what is asked. Word problems use context to measure if and how student apply concepts or skills; taking language out changes what the item is measuring. Some computer-based items for ELLs are using animation and simulation to provide context; think about how context can be put in items clearly, and think about how the language can be supported with student strengths. Additionally, more cognitively complex tasks in most subjects use language to communicate a level of precision, and this language is part of what needs to be conveyed to and from the student. It is important to know when this is so, or when the complexity of language can be reduced without jeopardizing conceptual and cognitive target goals. This may mean simplifying the struc- ture of a sentence without substituting key academic language being mea- sured. For example, in testing the “role” of consumers in an ecosystem, the term “role” implies a relational interaction that the word “job” does not have. Therefore, while during instruction, the targeted language can be amplified, with synonyms, clarifications, examples, and definitions, the key academic language needs to be applied frequently. Then, in the assessment, the academic terms can be used with the same rigorous academic meaning ELLs will need in order to develop effective academic discourse. Constructed responses may be easier for these students than trying to navigate the short- hand, nuanced language of multiple-choice or other forced-choice tasks. What is needed here is not so much changing how the student demonstrates what he or she knows, but improving how these kinds of responses are scored. Kopriva and Saez (1997) and Kopriva and Sexton (1998) provide good manuals for educating monolingual, English-speaking teachers about

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 111

how to score the constructed responses of ELLs, respectively in mathematics and science assessment. (These materials are available at www.ccsso.org.)

3. Scaffolding questions is something often discussed as a way to adapt assess- ment activities for ELLs. Scaffolding can be supportive as long as it does not change what is being measured. For instance, cueing when the cueing in any way narrows down the possible responses or points towards one answer is an inappropriate use of scaffolding for assessment purposes. In addition, an essential component of more complex tasks is that they are measuring the students’ skills in multi-step problem-solving. Leading the students through the steps destroys a fair amount of the complexity. But see Trumbull & Koelsch, Chapter 9, this volume, for a counter-example.) A serious critique of the targeted intent of tasks is needed before scaffolding is done.

4. As noted above, sometimes teachers think that assessment information must be collected using trite multiple-choice, true/false, or fill-in-the-blank items. These question types may elicit informative student knowledge under cer- tain conditions. However, there are multiple ways in which knowledge and skills can be evaluated that go beyond these methods, especially when more cognitively complex concepts are being measured. One of the best ways of assessing student progress is to utilize some of the tools used as part of instruction and learning. Properly implemented, the practice of using famil- iar materials reduces the gap between instructional and assessment prac- tices, providing a basic scaffold of support without reducing the complexity of the intended questions. It also lowers the threshold of anxiety among ELLs and offers opportunities for them to demonstrate their understanding. For example, when recording tools are used during instruction, such as graphic organizers, charts, and tables, the same formats can be used effec- tively during formative assessment situations. Strategies such as Sentence Starters, and the use of multiple modes of representation (i.e., illustrations, matching cards, oral presentations) that are used routinely in content teach- ing for ELLs may be productive when certain kinds of knowledge and skills are being evaluated. (cf. Trumbull & Koelsch, Chapter 9, this volume.)

5. There is an inherent value in performing a qualitative analysis of student work with students. This includes having students discuss their interpreta- tions of content and their reasoning during and after assessment tasks. This feedback forms part of student self-assessment with guidance from the teacher. It also helps to address cultural diversity, as students from various other backgrounds are socialized in ways that encourages their active partic- ipation in self-assessment, observation, and reflective practices.

6. Practice, practice, practice. Assessment methods most validly reflect student learning when they are asking the questions you want asked, and when they are used by students frequently throughout the year. This way, the methods, appropriately adapted to be accessible, facilitate the communication of desired knowledge and skills to and from the student, instead of being some- thing that, in addition to the targeted content, acts as a hurdle or barrier for students to struggle with as they try to demonstrate what they know and can do.

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

112 R. Kopriva and U. Sexton

Figure 5.1 (page 102) puts some of these ideas together to measure students’ multi-faceted understandings of ecosystems. First, teachers who developed the example worked backwards, thinking about what is expected of students in fourth grade to conceptualize and understand about how food webs function. Second, the selection of images was contextualized according to the type of local habitat the students have experienced or have visited in field trips. Third, the format of the task allows for flexibility in levels of language. In other words, using “cloze” as a format with examples, prompt sentence frames, “word walls,” and providing visual input with an illustration allows students at lower levels of English language development to demonstrate whether they know the content by using scaffolds embedded throughout without improperly cuing certain responses. Fourth, to ensure that the students had a thorough understanding of the content, the teachers decided to use three items within the task to measure conceptual knowledge from different perspectives. The first section of the item tells the teacher if the student has a general understanding of the content or not. The second section includes a cyclical image, which embeds a slightly more com- plex level of reasoning, while the use of a “word wall” provides support for stu- dents. The development team felt that giving them the first piece (Producers) as an entry would allow students to focus their attention on the interactions within the depicted ecosystem, while providing a degree of success for all students. Finally, the last section of the item is more open-ended and asks students to pre- dict what might happen under certain future conditions. This section measures more cognitive complex reasoning than the earlier items. To support their cur- rent level of language development, students are encouraged to use whatever methods they need to successfully get their points across. Setting up a good formative assessment program that uses a variety of methods repeatedly throughout the year is the goal. Building up the repertoire is the hard part. But, with the aid of suggestions such as those discussed here, ELLs can benefit from a rich learning environment and at the same time thrive with a range of assessment tools that they recognize and are comfortable with.

Conclusion

Readers might think this chapter should begin with the last section, that is, spe- cific tips and ideas for measuring content knowledge among ELLs. Actually, this is the end product. The first step is careful planning that identifies and intercon- nects the overarching constructs that embody various instructional units and also identifies the core ideas within each unit that teachers want their students to learn. These core ideas, as well as the overarching constructs that include the development of reasoning and strategic skills, become the focus for the ongoing assessments of students. Each assessment task should have a purpose that is clearly and explicitly spelled out by the teacher before he or she makes a decision about how exactly to assess for that purpose. Some task methods work better for some kinds of knowledge and skills, and building a repertoire of diverse approaches is needed if certain techniques are to be considered for a specific purpose. Just as it is

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

Using Appropriate Assessment Processes 113

important to teach and assess the nuts and bolts of content, it is also important to evaluate more complex content and skills. Finally, of course, depth of teacher knowledge is key as well. All in all, the purpose of this chapter is to provide guid- ance on how to develop a thoughtful formative assessment program. Success in this task will certainly provide a strong foundation for ELLs, as they continue to learn and grow within each of the content domains.

References

Anderson, L., & Krathwohl, D.A. (2001). Taxonomy for learning, teaching and assessing: A revision of Bloom’s Taxonomy of Educational Objectives. New York, NY: Longman.

Ball, D.L., & Bass, H. (2000). Interweaving content and pedagogy in teaching and learning to teach: Knowing and using mathematics. In J. Boaler (Ed.), Multiple perspectives on teaching and learning (pp. 83–104). Westport, CT: Ablex Publishing.

Bell, B., & Cowie, B. (1999). Formative assessment and science education, research. Report of the Learning in Science Project (assessment), August.

Bloom, Benjamin S., & David R. Krathwohl. (1956). Taxonomy of educational objectives: The classification of educational goals, by a committee of college and university examin- ers. Handbook 1: Cognitive domain. New York, NY: Longmans.

Bransford, J.D., Brown, A.L., & Cocking, R.R. (Eds.) (2000). How people learn: Brain, mind, experience, and school (Expanded Edition). Washington, DC: National Academy Press.

Briggs, D.C., Alonzo, A.C., Schwab, S., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11, 33–63.

Darling-Hammond, L., & Bransford, J. (Eds.) (2005). Preparing teachers for a changing world: What teachers should learn and be able to do. San Francisco, CA: Jossey-Bass.

DiRanna, K., Osmundson, E., Topps, J., Barakos, L., Gearhart, M., Cerwin, K., et al. (2008). Assessment-centered teaching: A reflective practice. Thousand Oaks, CA: Corwin.

Duschl, R.A., Schweingruber, H.A., & Shouse, A.W. (Eds.) (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, DC: National Academy of Sciences. Retrieved December 30, 2007, from: www.nap.edu/catalog. php?record_id=11625.

Garcia, S.B., & Ortiz, A.A. (2006). Preventing disproportionate representation: Culturally and linguistically responsive prereferral interventions. National Center for Culturally Responsive Education Systems. Retrieved from: www.nccrest.org/Briefs/Pre-referral_ Brief.pdf?v_document_name=Pre-Referral%20Brief.

Kopriva, R.J., & Saez, S. (1997). Guide to scoring LEP student responses to open-ended mathematics items. Washington, DC: Council of Chief State School Officers.

Kopriva, R.J., & Sexton U. (1998). Guide to scoring LEP student responses to open-ended science items. Washington, DC: Council of Chief State School Officers.

Li, M., Ruiz-Primo, M.A., & Shavelson, R.J. (2006). Towards a science achievement frame- work: The case of TIMSS 1999. In S.J. Howie and T. Plomp (Eds.), Contexts of learning mathematics and science: Lessons learned from TIMSS. New York, NY: Routledge.

Pellegrino, J.W., Chudowsky, N., & Glaser, R. (Eds.) (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.

Popham, W.J. (2006). Assessment for learning: An endangered species? Educational Lead- ership, 63(5), 82–83.

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .

114 R. Kopriva and U. Sexton

Ruiz-Primo, M.A. (2007). Assessment in science and mathematics: Lessons learned. In M. Hoehpl and M.R. Lindstrom (Eds.), Assessment of technology education. New York, NY: McGraw Hill.

Ruiz-Primo, M.A., & Furtak, E.M. (2007). Exploring teachers’ informal formative assess- ment practices and students’ understanding in the context of scientific inquiry. Journal of Research in Science Teaching, 44(1), 57–84.

Shavelson, R.J. (2009). Reflections on learning progressions. Paper presented at the Learn- ing Progressions in Science (LeaPS) Conference, June, Iowa City, IA.

Shepard, L. (2006). Classroom assessment. In R. Brennan (Ed.), Educational measurement (4th edition, pp. 624–646). Westport, CT: Praeger.

Stiggins, R. (2003). Student involved classroom assessment (3rd edition). Upper Saddle River, NJ: Merrill Prentice Hall.

Yager, Peter. (2005). The effects of varied inquiry experiences on teacher and student questions and actions in STS classrooms. Bulletin of Science Technology and Society, 25(5), 426–434.

C op

yr ig

ht ©

2 01

0. R

ou tle

dg e.

A ll

rig ht

s re

se rv

ed .