An Initial Examination of Institutional Practices

P olicymakers, accrediting bodies, and association leaders continue to focus on assessing student learning outcomes. But what is happening on the ground at colleges and universities? Where does student learning out-

comes assessment rank in importance on an institution’s action agenda? To what extent are faculty involved in assessment activities and using the results for improv- ing student learning?

The National Institute for Learning Outcomes Assessment (NILOA) is a mul- tiyear effort to further the student learning outcomes agenda nationally. NILOA staff conducted four focus groups with academic deans, provosts, presidents, and directors of institutional research from a variety of two- and four-year institutions during 2009–2010 to discuss the state of assessment of student learning outcomes on campus. Roundtable discussions were conducted at meetings of the Association of American Colleges & Universities (AAC&U), American Council on Education (ACE), and the Association for Institutional Research (AIR). All told, we talked with about forty-five academic leaders, representing a range of institutional types and regions, to gain first-hand accounts of the state of efforts under way on cam- puses. This article summarizes what these leaders had to say and considers how the perceptions of academic leaders comport with findings from the 2009 NILOA Survey report “More Than You Think, Less Than We Need: Learning Outcomes Assessment in American Higher Education” (www.learningoutcomeassessment. org/NILOAsurveyresults09.htm), which describes what colleges and universities are doing to measure student learning. Four prominent themes cut across the focus group discussions and organize the main ideas in this paper:

1. Assessment has taken root on campus. 2. Accreditation is the major catalyst for student learning outcomes assessment. 3. Faculty involvement is central to meaningful assessment. 4. Best practices in assessment weave assessment into organizing structures.

Assessment UPdate

Perspectives from Campus Leaders on the Current State of Student Learning Outcomes Assessment Jillian Kinzie

September–October 2010 Volume 22, Number 5

Progress, Trends, and Practices in Higher Education

CONTENTS ARTICLES Perspectives from Campus 1 Leaders on the Current State of Student Learning Outcomes Assessment Jillian Kinzie

Editor’s Notes 3 Trudy W. Banta

Peer Review of Program 5 Assessment Efforts: One Strategy, Multiple Gains Margaret Fong Bloom Assessing Doctoral Applicants’ 8 Readiness for Doctoral-Level Work Michelle A. Maher, Benita J. Barnes

COLUMN From the States 11 Peter T. Ewell FEATURE Recommended Reading 13 Susan Kahn

View this newsletter online at wileyonlinelibrary.com

DOI 10.1002/au.225 © 2010 Wiley Periodicals, Inc.

2� Assessment�Update� •� September–October�2010� •� Volume�22,�Number�5� •� ©�2010�Wiley�Periodicals,�Inc.� •� DOI�10.1002/au

Assessment Has Generally Taken Root, and on Many Campuses Student

Learning Outcomes Assessment Is Thriving In the last decade, assessment has realized some important developmental steps. Accord-

ing to the dean at Drew University, “We’re in a different place than ten years ago. There is a core of people who believe in assessment and work with their colleagues on outcomes assessment.” Most campus leaders credited regional and professional accreditation with helping assessment gain ground on campus. Several leaders indicated that visible efforts like the Voluntary System of Accountability (VSA) and the independent college-sponsored site, U-CAN, had helped expand discussions about assessment and accountability among campus constituents and external stakeholders. Teagle Foundation-funded initiatives have furthered progress in developing assessment systems in the private liberal arts college sector since 2004 (see grantmaking activity www.teaglefoundation.org). The Luther College aca- demic dean reported that learning outcomes assessment practices there had matured over the last several years, so that now at least half of the faculty are involved in assessment. Luther faculty are learning a lot about student learning outcomes through work with a consortium of colleges in the Midwest focused on the assessment of writing and critical thinking. At Or- egon State University and Towson University all undergraduate programs now have learn- ing outcomes. After a flawless accreditation review, Eastern Kentucky University embarked on an extensive analysis of course syllabi to examine the extent to which learning outcomes were common across sections of the same course. A president in the California State Univer- sity system declared that assessment has been aggressively approached at the system level, and it seems that “we’ve been assessing everything that moves.”

Assessment at Westminster College grew out of the strategic planning process, and discussions about what it meant to be a Westminster graduate culminated in the articula- tion of learning goals specific to a Westminster education. Westminster since has devel- oped rubrics in each academic program to assess learning goals.

Assessment Update Progress, Trends, and Practices in Higher Education September–October�2010� Volume�22,�Number�5

Editor Trudy�W.�Banta,�professor of higher education and senior advisor to the chancellor,�Indiana�University–� Purdue�University�Indianapolis�(IUPUI)

Managing Editor Karen�Elaine�Black,�director of program review,�IUPUI

Assistant Editor Frances�W.�Oblander,�director�of�institutional� research�&�effectiveness,�South�University

Book Review Editor Susan�Kahn,�director�of�institutional� effectiveness,�IUPUI

Consulting Editors Peter�T.�Ewell,�National�Center�for�Higher� Education�Management�Systems Thomas�Anthony�Angelo,�La�Trobe�University, Victoria,�Australia T.�Dary�Erwin,�James�Madison�University Cecilia�L.�Lopez,�Harold�Washington�College Marcia�Mentkowski,�Alverno�College Jeffrey�A.�Seybert,�Johnson�County� Community�College Peter�J.�Gray,�United�States�Naval�Academy Gary�R.�Pike,�IUPUI

Assessment Update: Progress, Trends, and Prac­ tices in Higher Education (Print�ISSN�1041-6099;� online� ISSN� 1536-0725� at� Wiley� Online� Library,� wileyonlinelibrary.com)� is� published� bimonthly� by� Wiley�Subscription�Services,�Inc.,�A�Wiley�Company,�at Jossey-Bass,�989�Market�St.,�San�Francisco,�CA�94103- 1741.�Periodicals�Postage�Paid�at�San�Francisco,�CA, and�additional�mailing�offices.�Individual�subscriptions� are�$127�per�year�(institu�tional�$199).�Back�issues�are available�in�limited�supply�at�$29�per�issue.�To�order,� phone�toll-free�(888)�378-2537�or�fax�(888)�481-2665. Visit�our�Web�site�at�www.josseybass.com.�Postmaster: Send�address�changes�to�Assessment�Update,�Jossey- Bass,�989�Market�St.,�San�Francisco,�CA�94103-1741.

Copyright�©�2010�Wiley�Periodicals,� Inc.,�A�Wiley� Company.� All� rights� reserved.� Reproduction� or� translation� of� any� part� of� this� work� beyond� that� permitted�by�Section�107�or�108�of�the�1976�United� States� Copyright� Act� without� permission� of� the� copyright�owner�is�unlawful.�Requests�for�permis- sion� or� further� information� should� be� addressed� to� the� Permissions� Department,� c/o� John�Wiley� &� Sons,�Inc.,�111�River�St.,�Hoboken,�NJ�07030;�(201)� 748-8789,�fax�(201)�748-6326,�www.wiley.com/go/� permissions.

Call for Contributions The editor welcomes short articles and news items for Assessment Update. Guidelines follow for those who would like to contribute articles on outcomes assessment in higher education.

• Content: Please send an account of your experience with assessment in higher education. Include concrete examples of practice and results.

• Audience: Assessment Update readers are academic administrators, campus assess- ment practitioners, institutional researchers, and faculty from a variety of fields. All types of institutions are represented in the readership.

• Style: A report, essay, news story, or letter to the editor is welcome. Limited references can be printed; however, extensive tables cannot be included.

• Format: In addition to standard manuscripts, news may be contributed via letter, telephone, or fax (317) 274-4651. The standard manuscript format is a 60-space line with 25 lines per page. Articles may be sent to <[email protected]> as a Microsoft Word attachment. Please include your complete postal mailing address.

• Length: Articles should be four to eight typed, double-spaced pages (1,000–2,000 words). Annotations of recent publications for the Recommended Reading feature should be 200–500 words in length. Short news items and content for the Memos section should be about 50–200 words long.

• Copyright: Articles shall not have been registered for copyright or published elsewhere prior to publication in Assessment Update.

• Deadlines: Each issue is typically planned four months before its publication. Please address mailed contributions and comments to Trudy W. Banta, Editor, Assessment Update, Suite 140 Administration Bldg., 355 N. Lansing St., Indianapolis, IN 46202–2896. ■

(continued on page 14)

14� Assessment�Update� •� September–October�2010� •� Volume�22,�Number�5� •� ©�2010�Wiley�Periodicals,�Inc.� •� DOI�10.1002/au

The few presidents and deans who did not view their efforts as very far along indicated that the lack of progress was tied to faculty asserting that assigning grades was sufficient evidence of student achievement. Most leaders commented that the key to advancing assessment is for it to flow more directly out of exist- ing processes for learning. For most cam- puses, the next big challenge is how to use effectively an astounding amount of data to improve student learning.

Accreditation as Catalyst for the

Assessment of Student Learning. The 2009 NILOA Survey revealed that

across all institutional types regional and specialized accreditation were the primary drivers for student learning outcome as- sessment activity. Our focus group dis- cussions were consistent with the survey results, providing specific examples of the strong role that accreditation plays in the use of assessment results. One administra- tor commented that although it is accept- able for accreditation to drive assessment, the problem is that student learning out- comes assessment results are rarely being used to influence institutional improve- ment. Another administrator lamented that because accreditation is motivated by a compliance mentality, little attention is paid to the assessment interests and ques- tions about student learning that are im- portant to educational effectiveness.

Although the general complaint from campus leaders was that the compliance mentality can make assessment less mean- ingful, several campus leaders described using accreditation as a lever for assess- ment. When Roosevelt University started planning for their Higher Learning Com- mission (HLC) reaffirmation process, university leaders and faculty were intent on achieving a ten-year approval and sug- gested that the best way to achieve this

would be to develop meaningful assess- ments while ensuring that activities and evidence satisfied the HLC requirements. The president, two associate deans, and several senior faculty leaders participated in the HLC Assessment Academy to pro- vide further education and structured sup- port, and devoted time to translating HLC standards into institutional purposes.

Several academic leaders pointed to a negative accreditation review as stimulat- ing activity. As the dean of Centre College put it, “there had been lots of talk [up to then]…now we have to do it.” A president whose institution is accredited by the Mid- dle States Association (MSA) reported that the MSA review was sobering, as it con- cluded that the institution had made prog- ress, but needed to do more. This critique stimulated immediate action on his campus.

While accreditation is clearly a driver for assessment, it is worrisome that the

focus of assessment is too often for re- sponding to accreditation demands, and less so for improving student learning, allocating resources, or guiding strategic planning. In this sense, accreditation may devalue assessment for improvement. As one dean noted, “We were advised by our accreditors not to post ‘directions for future study’ on our Web site, and to only post the glowing aspects of our self- study,” underscoring the long-standing tension that Peter Ewell (2009) described between assessment for improvement and assessment for accountability.

Faculty Involvement Is Key

to Meaningful Assessment. As with our focus group participants,

provosts responding to the 2009 NILOA

Survey indicated that engaging more faculty is the major challenge to advanc- ing assessment. A dean commented that faculty view assessment as a “distraction from the important job of teaching” and grades as sufficient information about how well students are learning. Com- pounding the general complaint is that several of the current criteria for educa- tional effectiveness, such as increased student retention and graduation rates, are considered outside the purview and interest of faculty.

On a positive note, faculty are inter- ested in assessment evidence from au- thentic student work that is directly linked to teaching and learning. According to a liberal arts college dean, “faculty want to evaluate student work, and want to talk about what it demonstrates in terms of student learning.” The dean at Hobart and William Smith Colleges indicated that

about a third of the faculty there have ac- tively generated learning goals for their courses and are seeking some outcome measures, but these efforts are highly par- ticularized to students in their courses. She reported that efforts by The Teagle Foun- dation to convene faculty from multiple institutions to discuss how to improve student learning has “softened faculty re- sistance to assessment.” She noted that “the minute talk turns to student work, and faculty have students or student artifacts in front of them, then the discussion is highly energized and constructive.”

The president of CUNY LaGuardia in- dicated that she has seen great improve- ments in pedagogy as a result of devel- oping a more meaningful approach to as- sessing student work. Collecting student

Most�leaders�commented�that�the�key�to�advancing�assessment�is�for�it�

to�flow�more�directly�out�of�existing�processes�for�learning.�

Perspectives from Campus Leaders on the Current State of Student Learning Outcomes Assessment (continued from page 2)

Assessment�Update� •� September–October�2010� •� Volume�22,�Number�5� •� ©�2010�Wiley�Periodicals,�Inc.� •� DOI�10.1002/au 15

work in e-portfolios and using rubrics captured the interest of faculty and stu- dents at LaGuardia. One approach to in- volving faculty in assessment advocated by some campus leaders is to simply avoid the term “assessment” wherever possible. As one dean put it, “there is lots of interest in the topic, but not the tech- nique or superstructure.” Helping faculty improve their evaluation of student work is one alternative approach. Another is to use every opportunity to ask faculty: “How do you make academic decisions about what to teach, and how do you know what your students are learning?”

Assessment Is Furthered

When It Is Woven into

Institutional Structures. Nearly all campus leaders reported

that their progress in assessment in- volved creating structures and mecha- nisms to support and sustain assessment activities and making assessment part of standard institutional policies and pro- cedures. At Ohio State University, for example, departments must include stu- dent learning outcomes to modify cur- ricular requirements and must describe methods to assess these goals. This policy helped to facilitate the gradual phasing in of required learning goals and plans for assessment across a variety of departments. Albany State University adopted a similar approach by streamlin- ing its required reports. Now, every re- quired report—program reviews, annual reports, and assessment reports—must include information about student learn- ing outcomes.

Developing assessment expertise is also important. Several presidents and deans reported encountering difficulty in making the right choices about assess- ment tools and approaches. One dean noted, “We are confronted with a bewil- dering array of techniques and instru- ments.” Albany State developed a cadre of assessment experts by creating a rotat- ing, two-year appointment with course

release time for faculty members. Assess- ment directors serve their two-year term and then, armed with this experience, return to their departments. According to the director of institutional research, “The faculty member’s colleagues are now going to him [the faculty member

who rotated out of the term] whenever they have assessment questions. . . . The director that we have now is from the College of Sciences and Health Profes- sions, so we’ll work with her and then she’ll rotate back to her full college duties and we’ll pick somebody else, hopefully from the College of Business, . . . until we have faculty experts in assessment in every college.”

Deans and directors of institutional research at several institutions empha- sized the importance of focusing on what faculty members are already doing in their classrooms in terms of learn- ing outcomes assessment as the natural place to begin to advance assessment. At the University of Missouri, the direc- tor of the office of assessment focused on working with faculty in programs who signaled they wanted to work with him. His first step was asking program faculty, “What do you want your under- graduates to be able to do?”

In some institutions, faculty needed help bringing their assessment practices into the spotlight. Some faculty members found welcome connections between as- sessment activities and the scholarship of teaching and learning, or within centers for teaching and learning, while others benefited from being involved in cross- campus and in some cases cross-institu- tion initiatives.

Final Thoughts Assessment efforts are growing and

deepening on campuses, primarily im-

pelled by accreditation and other national accountability initiatives. While support from campus leadership is essential, real progress requires that faculty members take ownership of assessment processes and outcomes, particularly at small col- leges and universities. In addition, infra-

structure and policies must be stream- lined to support assessment activities. Pat Hutchings (2010) suggests a way into faculty ownership by proposing six strat- egies to make engagement more likely and assessment more useful.

The potential of student learning out- comes assessment activities and the ac- creditation process are weakened when assessment is undertaken primarily for the purpose of satisfying accreditation re- quirements. Authentic faculty ownership of the assessment process and integra- tion of assessment into supportive insti- tutional structures have the greatest po- tential to advance assessment of student learning outcomes. ■

References: Ewell, P. (2009, November). Assessment,

accountability, and improvement: Re- visiting the tension. (NILOA Occasion- al Paper No. 1). Urbana, IL: University of Illinois and Indiana University, Na- tional Institute for Learning Outcomes Assessment.

Hutchings, P. (2010, April). Opening doors to faculty involvement in as- sessment. (NILOA Occasional Paper No. 4). Urbana, IL: University of Illi- nois and Indiana University, National Institute for Learning Outcomes As- sessment.

Jillian Kinzie is associate research scientist at the Indiana University Center for Postsecondary Research in Bloomington, Indiana.

Faculty�are�interested�in�assessment�evidence�from�authentic�student�

work�that�is�directly�linked�to�teaching�and�learning.

Copyright of Assessment Update is the property of John Wiley & Sons, Inc. and its content may not be copied

or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.

However, users may print, download, or email articles for individual use.