Assessment Course

OccasionalPaper29.pdf

Home >Education homework help >Assessment Course

National Institute for Learning Outcomes Assessment | 1

k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d i n t e l l e c t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e e d u c a t e i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n a c c e s s q u a l i t y s e l f - r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d i n t e l l e c t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c o n n e c t i o n k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o n n e c t i o n s e l f - r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s e d u c a t e a c t i o n c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e e d u c a t e i n n o v a t i o n s u c c e s s s e l f - r e f l e c t i o n k n o w l e d g e a c c o u n t a b i l i t y c o m m u n i c a t e l i s t e n l e a r n a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s s e l f - r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n u n d e r s t a n d e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f - r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n

Equity and Assessment: Moving Towards Culturally Responsive Assessment

Erick Montenegro and Natasha A. Jankowski

National Institute for Learning Outcomes Assessment January 2017

O c c a s i o n a l P a p e r # 2 9 www.learningoutcomesassessment.org

National Institute for Learning Outcomes Assessment | 2

NILOA Mission

The National Institute for Learning Outcomes Assessment’s (NILOA) primary objective is to discover and disseminate the ways that academic programs and institutions can productively use assessment data internally to inform and strengthen undergraduate education, and externally to communicate with policy makers, families, and other stakeholders.

Abstract....3

Equity and Assessment: Moving Towards Culturally Responsive Assessment...4

Limiting Learning Demonstration...6

Culturally Responsive Assessment...8

Student Learning Outcomes Statements...11

Assessment Approaches...12

Use of Assessment Results...13

Final Thoughts...14

References...17

NILOA National Advisory Panel...21

About NILOA...22

Table of Contents

Please cite as: Montenegro, E., & Jankowski, N. A. (2017, January). Equity and assessment: Moving towards culturally responsive assessment. (Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Erick Montenegro is a doctoral candidate in the Education Policy, Organization and Leadership program at the University of Illinois. He is responsible for NILOA’s integrated communications effort including developing media, maintaining the website, promoting activities that benefit NILOA and its partners, and providing access to resources for NILOA’s various audiences and stakeholder groups. Erick received a dual B.S. in Marketing and Business Administration with a concentration in International Business, and an Ed.M. in Education Policy, Organization and Leadership with a concentration in Higher Education both from the University of Illinois. His research interests include issues of equity in assessment, culturally responsive assessment, outcomes assessment practices at Minority- Serving Institutions, and issues affecting Latinx students in higher education.

Dr. Natasha Jankowski is Director and Research Assistant Professor with the Department of Education Policy, Organization and Leadership at the University of Illinois Urbana-Champaign, has presented at numerous national conferences and institutional events, and written various reports for NILOA. Her main research interests include assessment and evaluation, organizational evidence use, and evidence-based storytelling. She holds a PhD in Higher Education from the University of Illinois, an M.A. in Higher Education Administration from Kent State University, and a B.A. in philosophy from Illinois State University. She previously worked for GEAR UP Learning Centers at Western Michigan University and worked with the Office of Community Research and Leadership studying community colleges and public policy.

About the Authors

National Institute for Learning Outcomes Assessment | 3

Abstract

As colleges educate a more diverse and global student population, there is increased need to ensure every student succeeds regardless of their differences. This paper explores the relationship between equity and assessment, addressing the question: how consequential can assessment be to learning when assessment approaches may not be inclusive of diverse learners? The paper argues that for assessment to meet the goal of improving student learning and authentically document what students know and can do, a culturally responsive approach to assessment is needed. In describing what culturally responsive assessment entails, this paper offers a rationale as to why change is necessary, proposes a way to conceptualize the place of students and culture in assessment, and introduces three ways to help make assessment culturally responsive.

National Institute for Learning Outcomes Assessment | 4

Equity and A ssessment:

Moving Towards Culturally Responsive A ssessment

Erick Montenegro and Natasha A. Jankowski

Introduction

College enrollment has become increasingly diverse in terms of students’ race, ethnicity, gender identity, socioeconomic status, sexual orientation, age, ability, etc. This trend is only expected to continue as the United States moves into a majority-minority nation by the year 2050, and college enrollments continue to increase. Conducting assessment in a manner that takes into consideration the various needs of different student populations is a responsibility of higher education. For one, underrepresented students are more likely to be low-income and first-generation (Del Rios & Leegwater, 2008; Li & Carroll, 2007; Benitez, 1998), and there are vast differences between the needs of lesbian, gay, bisexual, transgender, queer, intersex, asexual (LGBTQIA) (Check & Ballard, 2014; Mallory, 2009), undocumented (Kim & Diaz, 2013; Perez, 2010), nontraditional (Macqueen, 2012), and special- needs students attending higher education institutions (Froese-Germain & McGahey, 2012). Further, students are increasingly mobile, with transfer students coming from mostly traditionally underrepresented backgrounds, attending multiple institutions (Backes & Velez, 2015; Shapiro et al, 2012) and facing their own challenges in higher education (Tobolowski & Cox, 2012).

Various areas of higher education are aware of the need to accommodate different student populations because “individual differences are clearly important to student success” (Strange & Banning, 2015, p. 61). For example, approaches to teaching, student development, student services, and campus programs have been analyzed and altered to improve outcomes for specific student groups (Ladson-Billings, 1995a; Ladson-Billings, 1995b; Schuh, Jones, Harper, & Associates, 2011; Kezar, 2011; Lara & Wood, 2015; Bowen, Chingos, & McPherson, 2009, Aronson & Laughter, 2016). Within the field of campus advising, the issue of microaggressions through lack of cultural awareness has been raised (Chu, 2016) and the work of the Association of American Colleges and Universities (AAC&U) in their Equity Imperative outlines the need to understand who students are, disaggregate data to look for inequities, and explore policy changes for unintended impacts on student groups. Conversations in K-12 have addressed the notion of equity from the standpoint of equity traps (McKenzie & Scheurich, 2004) within schools and the need to prepare school leaders to not only expose but address them through courageous conversations about inequities (Singleton, 2012). In a literature review of culturally responsive school leadership, Khalifa, Gooden, and Davis (2016) argue that culturally responsive leaders need to continuously support minoritized students through examination of assumptions about race and culture. Further, they argue that as demographics continue to shift, so should practice that responds to student needs, finding that it is “deleterious for students to have their cultural identities rejected in school and unacknowledged as integral to student learning” (p. 1285).

Conducting assessment in a manner that takes into consideration the various needs of different student populations is a responsibility of higher education.

National Institute for Learning Outcomes Assessment | 5

Culturally relevant and culturally responsive pedagogies sought to outline ways in which teachers could address unique learning needs of diverse student populations. Gloria Ladson-Billings (1995a, 1995b) recommends using culturally relevant pedagogy as a way to allow students in populations outside of the majority to maintain their cultural integrity all-the-while succeeding academically. Culturally relevant pedagogy aims to “produce students who can achieve academically…demonstrate cultural competence, and develop students who can both understand and critique the existing social order” (Ladson-Billings, 1995a, p. 474). In culturally responsive pedagogy, teachers use aspects of students’ cultures in an asset-based approach as opposed to deficit-based to make the course material relevant to them, and increase their skill acquisition, engagement, and learning outcomes (Ladson-Billings, 1995a). Yet, Geneva Gay (2010) has argued that solely modifying teaching practices cannot solve the challenges faced by ‘minoritized’ students.

In terms of assessing student learning, the field has been largely quiet when it comes to issues of equity. Assessment, if not done with equity in mind, privileges and validates certain types of learning and evidence of learning over others, can hinder the validation of multiple means of demonstration, and can reinforce within students the false notion that they do not belong in higher education. For equity gaps to be addressed, an entire institution needs to explore the combination of solutions and supports needed for students to be successful (Jones, 2015; Methvin & Markham, 2015), of which assessment is one. However, little of the conversation thus far has focused on the connection points between demonstration of student learning and issues of equity. Instead, assessment has remained largely unchanged in regards to inclusivity, and little urgency has been given to ensuring that students are provided with just and equitable means to demonstrate their learning. There is a difference between assessing all students in the same way in relation to a specific outcome of interest and making sure assessments are appropriate and inclusive of all students. Being attentive to how students may understand questions, tasks, and assignments differently, as well as feedback regarding their learning, is not only beneficial to students but to internal improvement efforts as well. Intentionally choosing appropriate assessment tools or approaches that offer the greatest chance for various types of students to demonstrate their learning so that assessment results may benefit students from all backgrounds advances our collective interest in student success.

Without examining issues of equity the students who may stand the most to gain from assessment efforts may have the least benefit since their learning is not accurately assessed and feedback may not be relevant to impact learning. If assessments are to be holistic in their goal of improving student learning, then incorporating a culturally responsive approach to assessment is a priority. As C. Carney Strange and James Banning (2015) state, student cultures “can play an important role, for good or otherwise, in introducing students to and maintaining their engagement in the learning process” (p. 53). It also creates opportunities for students to experience deep learning (Entwistle, 2001) by honoring students’ prior knowledge and experience. However, before we present the concept of culturally responsive assessment, it is useful to unpack an assumption that hinders consideration of diverse learner needs within assessment—that while learners may take multiple paths to and through learning, they must demonstrate their knowledge and skills in the same way.

Assessment, if not done with equity in mind, privileges and validates certain types of learning and evidence of learning over others, can hinder the validation of multiple means of demonstration, and can reinforce within students the false notion that they do not belong in higher education.

National Institute for Learning Outcomes Assessment | 6

Limiting Learning Demonstration

There is an assumption at play within the field of assessment that while there are multiple ways for students to learn, students need to demonstrate learning in specific ways for it to count. For instance, in a specific course different approaches may be used to engage students in the material, but demonstration of a students’ knowledge, skills, and abilities are done uniformly in the same assignment or approach—so while there may be multiple approaches and methods used across a program or institution for assessing student learning, at each instance of demonstration a single approach is employed. Regardless of the literature on the multiple ways students acquire knowledge, assessment asks students, at each instance of demonstration, to show they have the knowledge and skills of interest through the same means. William Sedlacek (1994) discusses the need for the development of multicultural assessment standards within the Association for Assessment in Counseling (AAC). While the focus is upon assessment within the context of counseling support and services, the interest of addressing the needs of those with “cultural experiences different from…White middle-class men of European descent, those with less power to control their lives, and those who experience discrimination in the United States” (p. 550), remains the same for assessment of or for learning in higher education. Sedlacek (1994) identifies five fallacies related to culture and assessment, stressing that most measures were not designed with nontraditional or underserved populations in mind, that few assessment specialists are trained in developing measures for use with nontraditional populations, and that larger issues exist that need to be explored and addressed when promoting diversity, equity, and inclusivity through assessment. Of note is the fallacy referred to as the three musketeers, which is the idea that in order to make a measure equally valid for everyone, everyone completes the same measure—all for one and one for all—as a means to ensure fairness instead of using different measures for different groups. Yet, Sedlacek (1994) argues, “if different groups have different experiences and different ways of presenting their attributes and abilities…it is unlikely that we could develop a single measure or test item that would be equally valid for all” (p. 550); further arguing that there is no need to employ the same measure when what is desired is equity of results, not process.

There are institutions providing students with support and opportunity to choose from a variety of approaches or even design how they will be assessed in cooperation with faculty members, presenting students with agency and choice in the assessment process (Singer-Freeman & Bastone, 2016) and most institutions use a combination of assessment methods to gauge learning (Kuh et al., 2014). In a study at the University of East London, students were allowed to choose how they were assessed, significantly improving attainment among learners without an academic background (Grove, 2016). Instead of completing exams based on coursework, students were given the option to do a presentation, poster, or debate. Using the alternative assessment techniques “helped mitigate the fact that many first-year students had not been in formal education for some time” allowing them space to demonstrate their learning, not their exam-taking abilities (Grove, 2016). Further, a similar approach was used at the University of Dublin where students were able to make a poster instead of taking an exam. In both instances, students had to

There is an assumption at play within the field of assessment that while there are multiple ways for students to learn, students need to demonstrate learning in specific ways for it to count.

National Institute for Learning Outcomes Assessment | 7

The manner in which students demonstrate learning is irrelevent when student demonstration is held to the same learning outcomes and evaluative criteria.

demonstrate their learning on the same learning outcomes and evaluative criteria, but the manner in which they did so was irrelevant. Rubrics were used such that the evaluation of the work was the same, thus quality ensured, but the demonstration could be different. In Canada, a study was undertaken within a large, third-year psychology class regarding differentiated evaluation to examine student engagement, quality of learning experience, and address challenges associated with increased student diversity (Gosselin & Gagné, 2014). Differentiated evaluation allowed students to choose how they would be evaluated though all students were still required to take mid-term and final exams. Students had the option of adding a term project through preparing a mini-class or participating in a community service learning program. The study found positive impact on student achievement and on the learning experience, with students performing below class average seeing grade improvement when completing a term project. Further, students who completed the project performed better on the final exam in comparison to those that did not, and the option helped to alleviate stress of sitting for an exam. Qualitative responses from students that selected the project option indicated that they saw the alternative as an opportunity to demonstrate their learning through a format over which they felt more control. Gosselin and Gagné (2014) argued that there are “methods of assessment that can foster inclusiveness and academic success whilst upholding high standards for the quality of student learning” yet interestingly “most innovations in this context have focused on teaching rather than on student learning” (p. 6). The differentiated evaluation approach complemented the existing structure and allowed the relationship between faculty and student to shift to one of collaboration instead of power, regarding decisions about how students demonstrate their learning.

The need to fold in culture and student experience into assessment is stressed in the everyday expertise framework—a perspective of learning that takes into account how students demonstrate knowledge and skills in their daily life with the other people around them (Toomey Zimmerman & Bell, 2012). The framework allows for learning to have multiple dimensions including individual, social, and cultural, requiring a broad consideration of how people learn within and across learning environments, noting that learners do not act with equal competency in all settings, even if the content is the same. Toomey Zimmerman and Bell (2012) argue that the difference in performance indicates that learners competent in informal and everyday settings may falter in more formalized learning settings, requiring alternative means to demonstrate their knowledge outside of the traditional classroom.

Beyond the many benefits from engaging students in co-curricular experiences (Meents‐DeCaigny & Sanders, 2015; Schuh, Jones, Harper, & Associates, 2011; Schuh, 2009; Pascarella & Terenzini, 2005; Del Rios & Leegwater, 2008), co-curricular learning provides a means to address the issues raised by the everyday expertise framework by widening our lens of where learning happens to include experiences beyond the classroom. In addition to conceptions shifting where learning happens, there has been a rise in competency-based education (CBE) which releases the time structure in which learning occurs in terms of credit hours. CBE programs stress that authentic artifacts, or demonstrations of student learning, need to come from a variety of sources to engage learners with curricula and assessment

National Institute for Learning Outcomes Assessment | 8

that reflect not just multiple ways to learn but multiple ways to demonstrate mastery of a competency (Jobs for the Future, 2016). However, there are calls for significant research to determine how best to design assessments for underprepared learners that also elevate and validate their skills through alternative measures (Person, Goble, & Bruch, 2014). While learning may happen anywhere and learners may need different lengths of time in their learning process, there is still the issue of who gets to validate that learning has occurred, or that demonstrations of learning are of the ‘right type.’

While there is movement to more inclusive means of assessment and active engagement with students as partners in learning, it is clear that the challenges of various minority groups on campus differ from those of the majority (Ellis & Chen, 2013; Rankin, Weber, Blumenfeld, & Frazer, 2010; Miller, Bradbury, & Pedley, 1998), yet higher education still privileges certain types of learners, certain ways of demonstrating knowledge, and certain learning spaces by not consistently offering transparency, differentiated assessments, or empowering students in their own learning. Students need to develop and apply their knowledge and skills across multiple contexts in different courses through a range of methods (Newman, Carpenter, Grawe, & Jaret- McKinstry, 2014, p. 14) with integrative liberal learning requiring students to engage in “ongoing demonstration to themselves and to others, of the gains made through curricula, programs, and the educational experience as a whole” (Ferren & Paris, 2015, p. 5). Yet, the signals education sends to students about what is validated or counts as demonstration of learning can be detrimental and reinforce for marginalized students that they do not belong because their learning ‘doesn’t count.’ What is needed is collaboration, where students, faculty and staff “draw together their life experiences and aspirations with classroom, co-curricular, and community opportunities” (Ferren & Paris, 2015, p. 20).

Culturally Responsive Assessment

Defining “culture” and explaining what is meant by culturally responsive assessment is complicated. The issue is that culture, whether speaking about it in terms of an organization, a campus, or an individual, has been historically difficult to define. Higher education has a tendency to group student differences and issues around race under the term ‘diversity,’ which is often discussed in relation to benefits to White students as opposed to African Americans, Latinx, Asian Americans, and Native Americans who continue to be underrepresented in higher education (Dowd & Bensimon, 2015). While diversity efforts on college campuses have brought attention to the vast differences among students—including gender, religion, sexual orientation, etc.—the term diversity fails to address issues surrounding race/ ethnicity and does not account for the different histories, needs, interests, and issues affecting distinct groups of students on campus. With this in mind, one can see why it would be beneficial to use culture instead of diversity as the imperative to refocus assessment into a more inclusive endeavor.

This paper draws from and expands on past definitions of culture to develop an understanding that culture should be thought of as: (1) the explicit elements that makes people identifiable to a specific group(s) including behaviors, practices, customs, roles, attitudes, appearance, expressions of

While learning may happen anywhere and learners may need different lenghts of time in their learning process, there is still the issue of who gets to validate that learning has occurred, or that demonstrations of learning are of the ‘right type.’

National Institute for Learning Outcomes Assessment | 9

identity, language, housing region, heritage, race/ethnicity, rituals, religion; (2) the implicit elements that combine a group of people which include their beliefs, values, ethics, gender identity, sexual orientation, common experiences (e.g. military veterans and foster children), social identity; and (3) cognitive elements or the ways that the lived experiences of a group of people affect their acquisition of knowledge, behavior, cognition, communication, expression of knowledge, perceptions of self and others, work ethic, collaboration, and so on. The culturally relevant component involves assuring that the assessment process—beginning with student learning outcome statements and ending with improvements in student learning—is mindful of student differences and employs assessment methods appropriate for different student groups. Underlying the culturally relevant component is the focus on students—the importance of keeping students at the center, which requires their involvement at every step in the assessment process and builds upon their lived experience.

In addition, it is important to understand the concept of intersectionality and its effect on culture. Traditionally, intersectionality is thought of in racial/ ethnic identity intersecting with class, gender, and sexual orientation to shape how people of color experience oppression (Solorzano & Yosso, 2002; Huber, 2010; Cho, 1997). However, for purposes of this paper, intersectionality is the way that aspects of a person’s identity cannot be fully separated from one another, play a central role in peoples’ experiences and making meaning of those experiences. This is related to Susan Jones and Marylu McEwen’s (2000) multiple dimensions of identity which treats a student’s identity as dynamic and changing depending on the relative contextual salience of other elements of one’s identity (e.g. race/ethnicity, sexual orientation, gender, religion); with no single aspect of one’s identity understood singularly, but only in relation to the other dimensions. For example, a White male that identifies as a member of the lesbian, gay, bisexual, transgender, queer, intersex, and asexual (LGBTQIA) community and practices Judaism is shaped by the intersectionality of these four elements. A Latina that is a single-mother from a low-socioeconomic background is shaped by the intersectionality of these elements. An undocumented English as a second language, first-generation student will experience college, acquire knowledge, and demonstrate knowledge differently than an international English as a second language first-generation student. The culture—the explicit, implicit, and cognitive elements—of the people in these examples shape their college experiences, and while one aspect of their culture may manifest itself more than another in specific contexts, they all affect the outcomes being assessed.

Thinking of culture in the way that it is defined here can serve as a reference point for what to consider when engaging in assessment and developing/ choosing/implementing assessment tools and methods. Culture is by no means simple, and it is by no means easily definable. It is dependent on the context in which culture is discussed. Culture permeates the individual, group, entire institutions, countries, and continents; and at the same time the individuals that comprise cultural groups are multicultural through intersectionality. Perhaps Lang (1997) stated it best when he said “attempts at defining culture in a definite way are futile” (p. 389). However, developing an inclusive understanding of culture, and making it explicit that culture is much more than race/ethnicity and affects students’ lives on multiple levels,

The culturally relevant component involves assuring that the assessment process— beginning with student learning outcome statements and ending with improvements in student learning—is mindful of student differences and employs assessment methods appropriate for different student groups.

National Institute for Learning Outcomes Assessment | 10

including learning and how they demonstrate learning, will help ensure culturally responsive assessment and increase the effectiveness and impact of learning outcomes assessment efforts.

In addition to the term of culture, it is important to note the use of responsive to indicate “an action-based, urgent need to create contexts and curriculum that responds to the social, political, cultural, and educational needs of students; it is affirmative and seeks to identify and institutionalize practices that affirm indigenous and authentic cultural practices of students” (Khalifa, Gooden, & Davis, 2016, p. 1278). Students who experience validation from faculty and integrate academically and socially are more likely to persist and be successful (Karp, Hughes, & O’Gara, 2011). Assessment approaches and processes can help reinforce a sense of belonging or add to students’ belief that they do not belong because their learning or experiences are not deemed as valid or important. Susan Headden and Sarah McKay (2015) stress this point, arguing that student motivation is connected to student’s beliefs that they are able to do the work and have a sense of control over the work. For first-generation college goers and African American students “stereotypes about academic performance can turn into self-fulfilling prophecies…even feedback on papers can reinforce or foster learning…that students are cared about and respected as learners” (Headden & McKay, 2015, p. 15). An environment focused on students’ unique learning interests and needs enables students to incorporate prior and everyday experiences in meaning construction (Land, Hannafin, & Oliver, 2012). Involvement with culture is also important as Cathleen Spinelli (2008) argues that there are a disproportionate number of students with cultural and linguistic differences that are misidentified as learning disabled. As a result, students are classified incorrectly, not academically challenged, and do not receive appropriate services. Spinelli (2008) further argues that when looking specifically at the case of English language learners, informal assessment provided a solution to the need of assessment of learning, but in a manner adaptable to language and cultural diversity, individual learning styles, and personal challenge while also informing instruction.

Culturally responsive assessment is thus thought of as assessment that is mindful of the student populations the institution serves, using language that is appropriate for all students when developing learning outcomes, acknowledging students’ differences in the planning phases of an assessment effort, developing and/or using assessment tools that are appropriate for different students, and being intentional in using assessment results to improve learning for all students. Culturally responsive assessment involves being student-focused, which does not simply mean being mindful of students. Instead, being student-focused calls for student involvement throughout the entire assessment process including the development of learning outcome statements, assessment tool selection/development process, data collection and interpretation, and use of results. An essential aspect of maintaining focus on students is truly understanding the student population at the institution and/or level at which the assessment is being conducted. Once we understand who our students are we can begin to tailor assessment processes and materials to have the greatest impact for their learning. Institutions with high enrollment of traditionally underrepresented students have already begun tailoring their learning outcomes assessment approaches based on

Assessment approaches and processes can help reinforce a sense of belonging or add to students’ belief that they do not belong because their learning or experiences are not deemed as valid or important.

National Institute for Learning Outcomes Assessment | 11

the student populations that they serve (Montenegro & Jankowski, 2015; Nunley, Bers, & Manning, 2011; Baker, Jankowski, Provezis, & Kinzie, 2012). Minority-Serving Institutions (MSIs) have been found to increase self-esteem, engagement, critical thinking skills, leadership skills and opportunities, and help the identity formation processes for traditionally underrepresented students; which helps increase students’ persistence through college (Conrad & Gasman, 2015; Del Rios & Leegwater, 2008; Conrad et al, 2013). The work at these institutions can serve as guideposts for the development of culturally responsive assessment practices.

Student Learning Outcomes Statements

Learning outcomes assessment as a process begins with developing learning outcome statements that clearly state what students should know and be able to demonstrate upon completion of a course, academic program, college, making use of student services, etc. To develop student learning outcomes statements using a cultural lens necessarily involves students in the development process. Poorly constructed learning outcomes make it difficult for students to demonstrate their learning for a myriad of reasons (e.g. not understanding what is expected of them, not understanding how the course/program is expected to contribute to their learning). In addition, it is students that will directly benefit from the feedback they receive as a result of assessment. Clarity of outcomes and curricular structure matters in general education (Gaston, 2015), assignment design (Winkelmes et al, 2016), co-construction of knowledge for deep learning (Juvova et al, 2015; von Glasersfeld, 2005), and new course design models like competency-based education (Jobs for the Future, 2016). Further, in the National Research Council report, How People Learn, (2000) principles for designing learner- centered environments emphasized the importance of individual social and cultural contexts in learning. Such perspectives require different approaches to curricular design, teaching, and assessment, and squarely place learner preconceptions and experiences as an integral part of the learning process.

Assessment is a field of alignment, and this also originates from learning outcomes statements. Hutchings (2016) defines alignment as “the linking of intended student learning outcomes with the processes and practices needed to foster those outcomes” (p. 5). Similarly to how academic programs, student services, and other institutional programs aim to align with and promote the mission of the college or university, learning outcomes statements of departments, programs, and courses should align with those of the institution. Outcome statements need to be culturally responsive because they align with assignments, evaluative criteria, and institutional and departmental goals. If outcome statements are not culturally responsive, then there are implications for various levels of the institution; not just for students. Learning outcome statements which are written to inform educational policy and practice, and are clear about expected proficiencies make it possible for programs, departments, institutions, and students to meet their goals (National Institute for Learning Outcomes Assessment, 2016). The language and operative verbs in learning outcome statements serve as a guide for students to understand departmental/program expectations, as well as understand how their educational experiences prepare them for their careers and lives after college (National Institute for Learning Outcomes Assessment, 2016). However, if learning outcomes statements are not written with attention to

To develop student learning outcomes statements using a cultural lens necessarily involves students in the development process.

National Institute for Learning Outcomes Assessment | 12

cultural relevancy, then it becomes difficult to accurately infer the learning gains of different groups of students.

Cliff Adelman (2015) speaks about the importance of being intentional and mindful of language when writing learning outcomes statements as this can lead to creating assignments that allow for genuine judgement of student achievement. However, taking this a step further and being mindful of how the language of learning outcomes statements might be appropriate for/ inclusive of certain student groups but not others can lead to more holistic assessments. Flawed assessment designs may unintentionally skew scores for certain student populations and ensuring this does not happen begins with the writing of culturally responsive learning outcomes statements that consider students, their different ways of learning, and the diverse ways they demonstrate learning. One way to make statements more culturally responsive is to explicitly define terms and use scenarios or examples that are relatable to various student groups. A sample tool that incorporates these elements of being intentional and explicit in writing learning outcomes and clearly defining learning is the Degree Qualifications Profile (DQP) and Tuning process. The DQP was developed by Lumina Foundation (2014), and by coupling Tuning processes to it serves as a way to clearly outline what students know and should be able to do after attaining a degree (Ewell, 2013).

Traditionally, learning outcomes statements are written by and for faculty and administrators. As a result, faculty and administrators define the intended learning outcomes and what it looks like to demonstrate those outcomes. If, instead, we write learning outcome statements for and with students, then we increase the chances of students understanding what is expected of them. In addition, instead of students’ knowledge conforming to how we traditionally measure it, students would now have agency in how to demonstrate learning. This would result in learning outcomes, as well as the assessment process, becoming a more inclusive endeavor.

Assessment Approaches

There is a need for assessments that allow students to demonstrate their learning in various ways while also being transparent about the learning that is taking place, help students reflect on their learning experiences, and allow students to actively participate in the learning and assessment process. Course-level assessments such as culturally responsive rubrics, portfolios, and capstone projects can lead to more valid, appropriate, holistic, and formative assessment where results are more indicative of what all students can do or lead to more targeted improvements in teaching and learning. Rubrics, which help instructors gauge student learning, skills development, and acquisition of learning outcomes, provide criteria by which to assess whether or not the learning outcome was demonstrated. Rubrics, when they undergo a culturally conscious development process and are shared with students, can be a way to accurately assess learning for all students while allowing variation in how the learning is demonstrated. While rubrics are at times created by individual faculty members to fit the context of specific courses or programs, the Association of American Colleges and Universities (AAC&U) Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics serve as examples of rubrics for institutions to employ. In addition,

National Institute for Learning Outcomes Assessment | 13

capstones—which can be entire courses or student projects—can also be an avenue through which the learning of students can be better gauged by allowing students to design their own projects in partnership with faculty. They provide “tangible, visible, self-explanatory evidence of exactly what students have and haven’t learned” (Suskie, 2004, p. 95). As comprehensive, culminating experiences, capstones allow students to demonstrate a wide- range of skills and knowledge that oftentimes draws from previous work, experiences, and learning that occurred throughout their coursework.

Finally, portfolios offer a similar freedom for students to demonstrate their learning and provide a more holistic representation of what students know and can do. The use of portfolios provides students the option to select demonstrations and add commentary and reflection, furthering their agency in the process and selection of assessment evidence. Portfolios represent student work over time and demonstrate various forms of learning (Kuh et al, 2015; Banta, Griffin, Flateby, & Kahn, 2009) which may not be easily captured by other forms of assessment. Portfolios are “authentic assessment that draws on the work students do in regular course activities and assignments” and “reconnect assessment to the ongoing work of teaching and learning and to the work of faculty, raising the prospects for productive use” (Kuh et al, 2015, p. 36). Portfolios provide the opportunity to get students invested into the course beyond grade attainment, and help to deepen students’ educational experiences through allowing them to make connections between conceptual issues, theoretical knowledge, and real world experiences (Singer-Freeman & Bastone, 2016). Additionally, portfolios can be made available online. Eportfolios can be easily accessed by potential employers, as well as other institutions, which provides students in the job market or looking to transfer a means to easily demonstrate their knowledge and skills. Kuh et al (2015) mention a few of the advantages that portfolios have for assessment, including advancing student success, catalyzing change, and making learning more visible for students. These impacts can be furthered by applying a cultural lens when assessing student portfolios. By being mindful of how culture affects students’ meaning- making processes, cognition, and demonstrations of learning, we can better understand and appreciate the learning gains that students make. In fact, at the program-level, assessment approaches such as rubrics and portfolios are used more often than surveys and other approaches (Ewell, Paulson, & Kinzie, 2011).

Use of Assessment Results

Implementing formative assessment methods means very little if assessment data are not used to inform learning at various levels of the institution or if it has no meaning to students to improve their own learning. The first step in creating change is analyzing the data by student populations. Disaggregating the data is instrumental in informing changes to higher education. While the data may tell a positive story about overall learning, disaggregation may yield the observation that first-generation students are struggling in a course, female students are making use of resources aimed at supporting their education at disproportionate rates, or Latinx students are not reaching the same institutional learning outcomes as other racial/ethnic groups. In either hypothetical case, disaggregating the data allows researchers, administrators, and practitioners to see themes that they otherwise would have missed and

By being mindful of how culture affects students’ meaning- making processes, cognition, and demonstrations of learning, we can better understand and appreciate the learning gains that students make.

National Institute for Learning Outcomes Assessment | 14

could inform changes that would positively impact students’ education. In addition, disaggregation of assessment data should not only be used to uncover surface-level findings such as Latinx students excel at “ABC” while first-generation students struggle with “XYZ.” As Aydin Bal & Audrey Trainor (2016) state, “researchers must also include an examination of processes (e.g. the racialization of disability… and the institutional acts of exclusion based on ability differences) and institutions…that reproduce, regardless of intentionality, disparities” (p. 330-331). This means that disaggregating data should explore why the condition exists in the first place, and then be used to inform/develop possible solutions.

In using assessment results, it is also useful to be mindful of our own assumptions. Similar to how a researcher’s bias cannot be fully removed from his/her/zer’s study and can either harm or enhance his/her/zer’s research, so can the biases of faculty and staff affect assessment efforts and use of results. It is unrealistic and counterproductive for assessment professionals to think they are approaching their work from an impartial stance or to assume that the students being assessed also operate from an impartial stance. Failing to recognize how culture and our own experiences affect the assessment process can limit the impact of assessment. In discussing the need for faculty to be attentive to the changes in the institution’s student population, Goldrick- Rab and Cook (2011) warn against comparing all students against the researcher’s subconscious idea of what students do/should do. Failing to be aware of our own biases or subconscious ideas and failing to disaggregate assessment data in a culturally responsive manner may cause the assessment endeavor to implement outdated norms as a means of comparison, which can misclassify certain students as underachievers, confusing, or outliers; and can also lead to the mistake of failing to connect the data to the actual lived experiences and realities of the students the institution serves (Goldrick-Rab & Cook, 2011). This can also lead to unintentionally reinforcing negative assumptions about certain student groups. Treating different racial/ethnic groups under an aggregate umbrella, as has been the recent case with the term “underrepresented minorities,” minimizes the voice of various groups and ignores their salient differences (Dowd & Bensimon, 2015) which impact their needs, experiences, learning, and demonstration of that learning. Finally, it would be worthwhile to connect assessment results to other campus assessment strategies. While certain data collection efforts on campus may seem unrelated, occurrences on campus seldom happen in isolation. Connecting different assessment efforts and resulting data sets can better inform issues related to student attrition, success, campus climate, pedagogy, and others (Hurtado & Halualani, 2014).

Final Thoughts

Students’ college experiences are inseparable from other daily experiences such as those encountered at work, microaggressions endured on campus, family life, and employment. More often than not, students’ college experiences are affected by students’ own culture and cultural differences with faculty, staff, and peers. It has long been known that students of different backgrounds experience college differently and respond differently to similar situations, stimuli, experiences, requests, questions, etc. So, if we also know that students from different cultures who have similar education backgrounds respond

Students’ college experiences are inseparable from other daily experiences such as those encountered at work, microaggressions endured on campus, family life, and employment.

National Institute for Learning Outcomes Assessment | 15

and perform significantly different, why would we design assessments, execute them, and then make changes based on assessment results without considering the cultural relevance of the assessment effort and analyze how the assessment might affect all students/benefit certain population(s) and hinder others? Why would we not include students in the assessment process to improve our approaches?

The focus of assessment as a means to improve student learning is an agreed upon purpose of the work. As Kuh, Ikenberry, Jankowski, Cain, Ewell, Hutchings, and Kinzie (2015) state, “gathering information about collegiate outcomes has a practical goal: using it to improve both student learning and institutional performance” (p. 51); and “harnessing evidence of student learning, making it consequential in the improvement of student success and strengthened institutional performance is what matters” (p. 4). Yet, how consequential can assessment truly be when assessment approaches are minimally inclusive of our current student populations? Using assessment tools and approaches that work for the majority of students but are less mindful of students identifying with groups outside of the majority population places a significant portion of students at a disadvantage, leads to a decrease in the quality of education, creates a disconnect between students and the institution, and contributes to achievement gaps (Slee, 2010; Sullivan, 2010; Qualls, 1998). Assessment that overlooks issues of diversity and equity contributes to inequalities in outcomes (Bal & Trainor, 2016). The same can be said for assessment approaches that do not take into account students’ culture.

Students have different ways to demonstrate their knowledge and we need to use assessment metrics that appropriately elicit demonstrations of what students know. One example of the diverse ways students can demonstrate learning comes from Nick Sousanis’ (2015) published dissertation exploring how people construct knowledge. Instead of writing a typical manuscript, Sousanis demonstrated his knowledge in a graphic novel format. At times, the illustrations said more than the words on the page, and both pictures and words united to tell a powerful academic story. This way of presenting scholarly work, while unconventional in academia, is still a powerful demonstration of learning. Sousanis’ chosen method of demonstrating his knowledge on a specific topic is not wrong, it is just different. We undo boundaries through the awareness that “it is our [own] vision, and not what we are viewing, that is limited” (Sousanis, 2015, p. 42). How assessment is often operationalized or experienced by students has not moved to a position where it continuously regards students’ diverse methods of demonstrating knowledge as appropriate. Instead, different can often be marked as wrong.

If assessment is about demonstrating learning, then we need to allow students the space to show their knowledge. Students are highly varied in customs, identity, and understanding, and it is all shaped by culture which affects learning; and thus, should affect how we measure learning. If assessment is done for improvement and with the goal of using the results to benefit student learning, then having outcome assessments that appropriately tell the stories of what students know and can do is of imperative importance. Our assessments approaches—how we assess and the process of assessment itself—should align with the students we have, empowering them with narratives to share and document their learning journey.

Assessment that overlooks issues of diversity and equity contributes to inequalities in outcomes. The same can be said for assessment approaches that do not take into account students’ culture.

National Institute for Learning Outcomes Assessment | 16

What is needed is not to help learners conform to the ways of higher education, thus reinforcing inequities and expectations based on ideologies the students may ascribe to, but to empower students for success through intentional efforts to address inequality within our structures, create clear transparent pathways, and ensure that credits and credentials are awarded by demonstration of learning, in whatever form that may take.

In summary, assessing students in the same way without paying attention to their differences works if students are all privy to the same educational opportunities, are all at the same academic standing, have similar experiences on campus, work through knowledge in similar fashion, understand questions in similar ways, and benefit from the same programs, pedagogical styles, support services, and interactions. However, we know this is not the case. “While absolute growth in the college-going population helped shape today’s college milieu, compositional changes also impacted the college experience, turning it into a set of highly diverse experiences that led to very different outcomes” (Goldrick-Rab & Cook, 2011, p. 257). Sara Goldrick-Rab & Marjorie Cook (2011) continue to say that “as the student body grew more diverse, so did the kinds of colleges and universities serving them; at the same time, opportunities both expanded in number and became more distinct and disparate, reflecting and preserving key aspects of the inequality of opportunity and outcomes” (p. 255). Continuing to assess students as if there are no differences will only work to preserve key aspects of inequality and widen the achievement gap. It is no secret that there is a disparity between the academic attainment of students based on race/ethnicity (Bowen, Chingos, & McPherson, 2009; Condron, Tope, Steidl, & Freeman, 2013; Santiago, Galdeano, & Taylor, 2015; Carnevale & Strohl, 2013) and social class (Kezar, 2011; Bowen, Chingos, & McPherson, 2009). We need to ask ourselves, is it that we want students to demonstrate their knowledge and skills or attainment of learning outcomes in a particular way, or that they demonstrate their learning? What is needed is not to help learners conform to the ways of higher education, thus reinforcing inequities and expectations based on ideologies the students may not ascribe to, but to empower students for success through intentional efforts to address inequality within our structures, create clear transparent pathways, and ensure that credits and credentials are awarded by demonstration of learning, in whatever form that may take.

National Institute for Learning Outcomes Assessment | 17

References

Adelman, C. (2015). To imagine a verb: The language and syntax of learning outcomes statements. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Aronson, B., & Laughter, J. (2016). The theory and practice of culturally relevant education: A synthesis of research across content areas. Review of Education Research, 86(1), 163-206.

Backes, B., & Velez, E. D. (2015). Who transfers and where do they go? Community college students in Florida. Washington, DC: National Center for Analysis of Longitudinal Data in Education Research (CALDER).

Baker, G. R., Jankowski, N., Provezis, S. & Kinzie, J. (2012). Using assessment results: Promising practices of institutions that do it well. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Bal, A., & Trainor, A. A. (2016). Culturally responsive experimental intervention studies: The development of a rubric for paradigm expansion. Review of Education Research, 86(2), 319-359.

Banta, T. W., Griffin, M., Flateby, T. L., & Kahn, S. (2009, December). Three promising alternatives for assessing college students’ knowledge and skills. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.

Benitez, M. (1998). Hispanic-serving institutions: Challenges and opportunities. New Directions for Higher Education, 102(2), 57-68.

Bowen, W. G., Chingos, M. M. & McPherson, M. S. (2009). Crossing the finish line: Completing college at America’s public universities. Princeton, NJ: Princeton University Press.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.

Carnevale, A. P., & Strohl, J. (2013). Separate and unequal: How higher education reinforces the intergenerational reproduction of White racial privilege. Washington, DC; Georgetown University, Georgetown Public Policy Institute, Center on Education and the Workforce.

Check, E., & Ballard, K. (2014). Navigating emotional, intellectual, and physical violence directed toward LGBTQQ students and educators. Art Education, 67(3), 6-11.

Cho, S. K. (1997). Converging stereotypes in racialized sexual harassment: Where the model minority meets Suzie Wong. In A. K. Wing, (Ed). Critical Race Feminism: A Reader (2nd. Ed). New York, NY: New York University Press.

Chu, J. Y. (2016, October 21). The need to be culturally aware when advising students (essay). Inside Higher Ed. Retrieved from https://www.insidehighered.com/views/2016/10/21/need-be-culturally-aware-when-advising-students-essay.

Condron, D. J., Tope, D., Steidl, C. R., & Freeman, K. J. (2013). Racial segregation and the Black/White achievement gap, 1992 to 2009. The Sociological Quarterly, 54, 130-157.

Conrad, C., Gasman, M. Lundberg, T., Nguyen, T., Commodore, F., & A. C. Samayoa (2013). Using educational data to increase learning, retention, and degree attainment at Minority Serving Institutions. Philadelphia, PA: Penn Center for Minority Serving Institutions. Retrieved from http://www.gse.upenn.edu/pdf/cmsi/using_educational_data.pdf.

Conrad, C., & Gasman, M. (2015). Educating a diverse nation: Lessons from Minority Serving Institutions. Cambridge, MA: Harvard University Press.

Del Rios, M., & Leegwater, L. (2008). Increasing student success at minority serving institutions: Findings from the BEAMS project. Washington, DC: Institute for Higher Education Policy (IHEP).

Dowd, A. G., & Bensimon, E. M. (2015). Engaging the “race question”: Accountability and equity in U.S. higher education. New York, NY: Teachers College at Columbia University.

Ellis, L. M., & Chen, E. C. (2013). Negotiating identity development among undocumented immigrant college students: A grounded theory study. Journal of Counseling Psychology, 60(2), 251-264.

Entwistle, N. (2001). Styles of learning and approaches to studying in higher education. Kybernetes, 30(5/6), 593-603. Ewell, P. T. (2013). The Lumina Degree Qualifications Profile: Implications for assessment. Urbana, IL: University of Illinois

and Indiana University, National Institute for Learning Outcomes Assessment. Ewell, P. T., Paulson, K., & Kinzie, J. (2011). Down and in: Assessment practices at the program level. Urbana, IL: University

of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

National Institute for Learning Outcomes Assessment | 18

Ferren, A. S., & Paris, D. C. (2015). Faculty leadership for integrative liberal learning. Washington, DC: Association of American Colleges and Universities.

Froese-Germain, B., & McGahey, B. (2012). Optimizing conditions of teacher’s professional practice to support students with special educational needs. Ottawa, ON, Canada: Canadian Teacher’s Federation. Retrieved from http://files.eric.ed.gov/ fulltext/ED532565.pdf

Gaston, P. L. (2015). General education transformed: How we can, why we must. Washington, DC: Association of American Colleges and Universities.

Gay, G. (2011). Culturally responsive teaching: Theory, research, and practice (2nd ed.). New York, NY: Teacher College Press.

Goldrick-Rab, S., & Cook, M. A. E. (2011). College students in changing contexts. In P. G. Altbach, P. J. Gumport, & R. O. Berdahl, (Eds) American higher education in the twenty-first century: Social, political, and economic challenges (3rd Ed.) pp. 254-278. Baltimore, MD: The John Hopkins University Press.

Gosselin, J., & Gagné, A. (2014). Differentiated evaluation: An inclusive evaluation strategy aimed at promoting student engagement and student learning in undergraduate classrooms. Toronto, Canada: Higher Education Quality Council of Ontario.

Grove, J. (2016, September, 29). Mature students ‘do better with non-written assessment’. Times Higher Education. Retrieved from https://www.timeshighereducation.com/news/mature-students-do-better-non-written-assessment.

Headden, S., & McKay, S. (2015, July). Motivation matters: How new research can help teachers boost student engagement. Stanford, CA: Carnegie Foundation for the Advancement of Teaching.

Huber, L. P. (2010). Using Latina/o critical race theory (LatCrit) and racist nativism to explore intersectionality in the educational experiences of undocumented Chicana college students. Educational Foundations, 24, 77-96.

Hurtado, S., & Halualani, R. (2014). Diversity assessment, accountability, and action: Going beyond the numbers. Diversity & Democracy, 17(4), 8-11.

Hutchings, P. (2016). Aligning educational outcomes and practices. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Jobs for the Future. (2016, February). Expanding competency-based education for all learners. Washington, DC: Author. Jones, S. (2015). The game changers: Strategies to boost college completion and close attainment gaps. Change: The

Magazine of Higher Learning, 47(2), 24-29. Jones, S. R., & McEwen, M. K. (2000). A conceptual model of multiple dimensions of identity. Journal of College Student

Development, 41, 405-414. Juvova, A., Chudy, S., Neumeister, P., Plischke, J., & Kvinotova, J. (2015). Reflection of constructivist theories in current

educational practice. Universal Journal of Educational Research, 3(5), 345-349. Karp, M. M., Hughes, K. L., & O’Gara, L. (2010). An exploration of Tinto’s integration framework for community

college students. Journal of College Student Retention: Research, Theory & Practice, 12(1), 69-86. Kezar, A. (2011). Recognizing and serving low-income students in higher education: An examination of institutional policies,

practices, and culture. New York, NY: Routledge. Khalifa, M. A., Gooden, M. A., & Davis, J. E. (2016). Culturally responsive school leadership: A synthesis of the literature.

Review of Educational Research, 86(4), 1272-1311. Kim, E., & Diaz, J. (2013). Undocumented students and higher education. ASHE Higher Education Report, 38(6), 77-90. Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. T., Hutchings, P., & Kinzie, J. (2015). Using evidence

of student learning to improve higher education. San Francisco, CA: Jossey-Bass. Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing what students know and can do: The current

state of student learning outcomes assessment in US colleges and universities. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Ladson-Billings, G. (1995a). Toward a theory of culturally relevant pedagogy. American Educational Research Journal, 32(3), 465-491.

Ladson-Billings, G. (1995b). But that’s just good teaching! The case for culturally relevant pedagogy. Theory Into Practice, 34(3), 159-165.

National Institute for Learning Outcomes Assessment | 19

Land, S. M., Hannafin, M. J., & Oliver, K. (2012). Student-centered learning environments: Foundations, assumptions and design. In D. Jonassen, & S. Land (Eds.). Theoretical foundations of learning environments (2nd ed.) pp. 3-26. New York, NY: Routledge.

Lang, A. (1997). Thinking rich as well as simple: Boesch’s cultural psychology in semiotic perspective. Culture & Psychology, 3, 383–394.

Lara, R., & Wood, J. L. (2015). Impact of non-cognitive behaviors on student engagement for Latino male students attending a designated two-year Hispanic-serving institution. In J. P. Mendez, F. A. Bonner II, J. Mendez-Negrete, & R. T. Palmer (Eds) Hispanic-serving institutions in American higher education: Their origin, and present and future challenges. Sterling, VA: Stylus Publishing, LLC.

Li, X., & Carroll, C. D. (2007). Characteristics of minority-serving institutions and minority undergraduates enrolled in these institutions. Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

Lumina Foundation (2014). The Degree Qualifications Profile. Indianapolis, IN: Author. Macqueen, E. (2012). Studying the experiences of non-traditional students in teacher education. Sydney, Australia: Joint AARE

APERA International Conference. Retrieved from http://files.eric.ed.gov/fulltext/ED544492.pdf. Mallory, A. (2009). Access and equity for all students: Meeting the needs of LGBT students. California Postsecondary Education

Commission. Retrieved from http://files.eric.ed.gov/fulltext/ED507508.pdf. McKenzie, K. B., & Scheurich, J. J. (2004). Equity traps: A useful construct for preparing principals to lead schools that are

successful with racially diverse students. Educational Administration Quarterly, 40, 601-632. Meents-DeCaigny, E., & Sanders, L. (2015). Framing research and assessment of co-curricular learning. New Directions for

Institutional Research, 164, 15-25. Methvin, P., & Markham, P. N. (2015). Turning the page: Addressing the challenge of remediation. Change: The Magazine

of Higher Learning, 47(4), 50-56. Miller, R., Bradbury, J., & Pedley, J. (1998). Academic performance of first and second language students: Disadvantages and

underpreparedness. South African Journal of Science, 95, 30-34. Montenegro, E., & Jankowski, N. A. (2015). Focused on what matters: Assessment of student learning outcomes at minority-

serving institutions. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

National Institute for Learning Outcomes Assessment. (2016, May). Higher education quality: Why documenting learning matters. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.

Newman, L. E., Carpenter, S., Grawe, N., & Jaret-McKinstry, S. (2014). Creating a culture conducive to integrative learning. Peer Review, 16(4), 14-15.

Nunley, C., Bers, T, & Manning, T. (2011, July). Learning outcomes assessment in community colleges. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students. San Francisco, CA: Jossey-Bass. Perez, W. (2010). Higher education access for undocumented students: Recommendations for counseling professionals.

Journal of College Admission, 206, 32-35. Person, A. E., Goble, L., & Bruch, J. (2014, April). Developing competency-based program models in three community colleges.

Oakland, CA: Mathematica Policy Research. Qualls, A. L. (1998). Culturally responsive assessment: Development strategies and validity issues. The Journal of Negro

Education, 67(3), 296-301. Rankin, S., Weber, G., Blumenfeld, W., & Frazer, S. (2010). 2010 state of higher education for lesbian, gay, bisexual, and

transgender people. Charlotte, NC; Campus Pride. Santiago, D. A., Galdeano, E. C., & Taylor, M. (2015). The condition of Latinos in education: 2015 factbook. Washington,

DC; Excelencia in Education. Schuh, J. H. (2009). Assessment methods for student affairs. San Francisco, CA: Jossey-Bass.

National Institute for Learning Outcomes Assessment | 20

Schuh, J. H., Jones, S. R., Harper, S. R., & Associates. (2011). Student services: A handbook for the profession (5th Ed). San Francisco, CA: Jossey-Bass.

Sedlacek, W. E. (1994). Issues in advancing diversity through assessment. Journal of Counseling & Development, 72, 549-553. Shapiro, D., Dundar, A., Chen, J., Ziskin, M., Park, E., Torres, V., & Chiang, Y. (2012). Completing college: A national view

of student attainment rates. Herndon, VA: National Student Clearinghouse Research Center. Singer-Freeman, K. & Bastone, L. (2016, July). Pedagogical choices make large classes feel small. (Occasional Paper No. 27).

Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Singleton, G. E. (2012). More courageous conversations about race. Thousand Oaks, CA: Corwin. Slee, J. (2010). A systemic approach to culturally responsive assessment practices and evaluation. Higher Education Quarterly,

64(3), 246-260. Solorzano, D. G., & Yosso, T. J. (2002). Critical race methodology: Counter-storytelling as an analytical framework for

education research. Qualitative Inquiry, 8(1), 23-44. Sousanis, N. (2015). Unflattening. Cambridge, MA: Harvard University Press. Spinelli, C. G. (2008). Addressing the issue of cultural and linguistic diversity and assessment: Informal evaluation measures

for English language learners. Reading & Writing Quarterly, 24(1), 101-118. Strange, C. C., & Banning, J. H. (2015). Designing for learning: Creating campus environments for student success (2nd Ed.).

San Francisco, CA: Jossey-Bass. Sullivan, A. L. (2010). Preventing disproportionality: A framework for culturally responsive assessment. Communique, 39(3),

24-26. Suskie, L. (2004). Assessing student learning: A common sense guide. Boston, MA: Anker Publishing Company. Tobolowski, B. F., & Cox, B. E. (2012). Rationalizing neglect: An institutional response to transfer students. Journal of

Higher Education, 83(3), 389-410. Toomey Zimmerman, H., & Bell, P. (2012). Everyday expertise: Learning within and across formal and informal settings.

In D, Jonassen & S. Land (Eds.). Theoretical foundations of learning environments (2nd ed.) pp. 224-241. New York, NY: Routledge.

von Glasersfeld, E. (2005). Instroduction: Aspects of constructivism. In C. T. Fosnot (Ed.), Constructivism: Theory, Perspectives, and practice (2nd ed.), pp. 3-7. New York, NY: Teachers College Press.

Winkelmes, M., Bernacki, M., Butler, J., Zochowski, M., Golanics, J., & Harriss Weavil, K. (2016). A teaching intervention that increases underserved college students’ success. Peer Review, 18(1-2), 31-36.

National Institute for Learning Outcomes Assessment | 21

NILOA National Advisory Panel James Anderson Interim Dean Edward William and Jane Marr Gutsgell Professor University of Illinois Urbana-Champaign Wallace Boston CEO American Public University System Molly Corbett Broad President American Council on Education Judith Eaton President Council for Higher Education Accreditation Richard Ekman President Council of Independent Colleges Keston Fulcher Director of the Center for Assessment and Research Studies Associate Professor, Graduate Psychology James Madison University Paul Gaston, III Trustees Professor Kent State University Mildred Garcia President California State University, Fullerton Susan Johnston Executive Vice President Association of Governing Boards Norman Jones Professor Utah State University Peggy Maki Higher Education Consultant George Mehaffy Vice President for Academic Leadership and Change American Association of State Colleges and Universities Lynn Pasquerella President Association of American Colleges & Universities

George Pernsteiner President State Higher Education Executive Officers Association Mary Ellen Petrisko President WASC Senior College and University Commission Kent Phillippe Associate Vice President, Research and Student Success American Association of Community Colleges

Robert Sheets Research Professor George Washington Institute of Public Policy

Ralph Wolff Founder and President The Quality ASsurance Commons for Higher and Postsecondary Education

Ex-Officio Members Peter Ewell President Emeritus National Center for Higher Education Management Systems

Stanley Ikenberry President Emeritus and Regent Professor University of Illinois

Natasha Jankowski Director, NILOA Research Assistant Professor

George Kuh Founding Director, National Institute for Learning Outcomes Assessment Adjunct Research Professor, University of Illinois Urbana-Champaign Chancellor’s Professor of Higher Education Emeritus, Indiana University

Paul Lingenfelter President Emeritus State Higher Education Executive Officers

NILOA Mission

NILOA’s primary objective is to discover and disseminate ways that academic programs and institutions can productively use assessment data internally to inform and strengthen undergraduate education, and exter- nally to communicate with policy makers, families and other stake- holders.

NILOA Occasional Paper Series

NILOA Occasional Papers are commissioned to examine contemporary issues that will inform the academic community of the current state-of-the art of assessing learning outcomes in American higher education. The authors are asked to write for a general audience in order to provide comprehensive, accurate information about how institutions and other organizations can become more proficient at assessing and reporting student learning outcomes for the purposes of improving student learning and responsibly fulfilling expectations for transparency and accountability to policy makers and other external audiences.

Comments and questions about this paper should be sent to: niloa@education.illinois.edu

National Institute for Learning Outcomes Assessment | 22

About NILOA

• The National Institute for Learning Outcomes Assessment (NILOA) was estab- lished in December 2008.

• NILOA is co-located at the University of Illinois and Indiana University.

• The NILOA website contains free assessment resources and can be found at http:// www.learningoutcomesassessment.org/.

• The NILOA research team has scanned institutional websites, surveyed chief academic officers, and commissioned a series of occasional papers.

• NILOA’s Founding Director, George Kuh, founded the National Survey for Student Engagement (NSSE).

• The other co-principal investigator for NILOA, Stanley Ikenberry, was president of the University of Illinois from 1979 to 1995 and of the American Council of Education from 1996 to 2001.

NILOA Staff NATIONAL INSTITUTE FOR LEARNING OUTCOMES ASSESSMENT

Stanley Ikenberry, Co-Principal Investigator

George Kuh, Founding Director, Senior Scholar, and Co-Principal Investigator

Natasha Jankowski, Director

Gianina Baker, Assistant Director

Filip Przybysz, Communications Coordinator

Katie Schultz, Project Manager

Peter Ewell, Senior Scholar

Pat Hutchings, Senior Scholar

Jillian Kinzie, Senior Scholar

Paul Lingenfelter, Senior Scholar David Marshall, Senior Scholar

Erick Montenegro, Research Analyst

Verna F. Orr, Research Analyst

Anthony B. Sullers, Jr., Research Analyst

Emily Teitelbaum, Research Analyst

Terry Vaughan III, Research Analyst

Karie Brown-Tess, Research Analyst

NILOA Sponsors Lumina Foundation for Education

University of Illinois, College of Education

Produced by Creative Services | Public Affairs at the University of Illinois for NILOA. 10.032

National Institute for Learning Outcomes Assessment | 23

National Institute for Learning Outcomes Assessment

For more information, please contact:

National Institute for Learning Outcomes Assessment (NILOA) University of Illinois at Urbana-Champaign 360 Education Building Champaign, IL 61820

learningoutcomesassessment.org niloa@education.illinois.edu Phone: 217.244.2155

http://www.learningoutcomesassessment.org/