I want to edit paper

1

Assessing the Skills that Matter

The National Association of Colleges and Employers’ (NACE) Job Outlook 2020 survey report reveals the attributes that businesses are looking for in recent graduates. After degree-specific knowledge, the top “most wanted” qualities include problem-solving and analytical skills, ability to work in a team, strong work ethic, leadership, and clarity in written and oral communication.

While grade point average still matters, em- ployers recognize that grades tell only part of a student’s story. The ability to apply skills learned in classrooms to real-world settings calls for different metrics, and colleges need to be thoughtful, strategic, and persistent in their approaches to assessment to ensure that they are meeting their missions.

Recent assessment trends have shown that forward-thinking institutions are building a culture of continuous assessment to en- hance their curriculums and improve out- comes. In her 2017 paper Ten Years After the Spellings Commission: From Accountability to Internal Improvement, Ou Lydia Liu, senior research director at Educational Testing

Service, laid out a framework to help schools plan their approach to this vital work. Liu proposes a four-element, one-enabler assessment cycle. The elements are:

• Identify and define the learning outcomes

• Translate the outcomes to specific assessment goals

• Evaluate a range of assessment tools • Work with professionals for assessment

implementation and interpretation

The enabler is engaging faculty and cultivat- ing an assessment culture.

“As assessment in higher education shifts from a predominant focus on accreditation and accountability to a growing emphasis on internal improvement,” Liu writes, “it is critically important to view assessment as an iterative and progressive cycle as opposed to a one-shot activity. … The four elements represent a dynamic, iterative, and scientif- ic assessment process and the enabler of faculty involvement is indispensable for the implementation of the process.”

Assessing the Skills that Matter Colleges and universities need a culture of continuous assessment to ensure that they are preparing students for the real world

2

Assessing the Skills that Matter

What exactly is critical thinking? Is adher- ence to grammar norms the best way to measure the effectiveness of oral or written communication? What achievements signal competence in a field of study? Determining expected outcomes is essential to assessing whether the college or university is fulfilling its mission. Liu writes: “Only when insti- tutions carefully unpack and define what aspects of critical thinking and global compe- tency that they aim to target are they able to identify necessary mechanisms that can be used to assess such aspects.”

It’s also crucial that academic programs and outcomes are aligned. In her paper, Liu cites James Madison Uni- versity, one of the first higher education

institutions to require nearly all first-year students to participate in assessments of their general and course-specific knowledge — and to repeat those assessments later in their college careers. The goal is not just to gauge student growth as they prepare to leave, but to keep refining the courses for those who have yet to take them. “Assessing learning by itself will not lead to learning im- provement,” Liu explains. “Linking the testing results to courses and programs provides an opportunity to improve [them].”

Like NACE’s Job Outlook 2020 survey, the report Fulfilling the American Dream: Liberal Education and the Future of Work, from the Association of American Colleges and Uni- versities, is useful for its insights into busi- ness leaders’ preferences when hiring, and for their perspective on recent graduates’ preparedness. For example, the report sug- gests that “only 40 percent of executives rate recent college graduates as well prepared in oral communication, the quality that they prioritize most highly (80 percent) among the 15 [outcomes] tested.” A large majority of business leaders also indicated that they prize more industry-specific proficiency — reflected in “the ability to apply knowledge and skills in real-world settings” — but they struggle to find graduates who meet their expectations.

As such, it is left to colleges and universi- ties to determine how to measure their success in imparting such a wide range of

degree-centric and general competencies. Institutions across the country are putting initiatives in place to ensure students are gaining these critical skills.

Lebanon Valley College in Pennsylvania is using data from an ongoing longitudinal study to gauge progress from curriculum changes implemented to support their goal of im- proving critical thinking. This is similar to the above-mentioned approach undertaken by James Madison University. Moreover, Quinnip- iac University in Connecticut is administering assessments to almost 80 percent of incoming students, with a smaller portion of that cohort taking a follow-up assessment as part of their senior capstone projects.

An institution could also decide to compare first-year and fourth-year students by testing both groups at the same time and using a cross-sectional approach. In either case, the data can be used to evaluate student learning.

Identifying and defining the learning outcomes

Translating outcomes to specific assessment goals

The goal is not just to gauge student growth as they prepare to leave, but to keep refining the courses for those who have yet to take them.

3

Assessing the Skills that Matter

Institutions take multiple approaches to assessing outcomes. There is no one tool or method that is suitable for all assessment needs. Each has advantages and limitations and should be chosen according to the assessment goals. For example, national student surveys can provide data on student engagement and perceived learning, but typically won’t produce evidence of learning. Assessments developed by institutions for their own use can be useful for measuring course effectiveness, but the data may be of little or no use in comparing the institution to other similar institutions across the nation.

“It may not be the best practice in every case to abandon an assessment tool because it has a limitation or embrace one because

it has an advantage,” Liu writes in Ten Years Af- ter the Spellings Commission. “The assessment activities are more likely to be fruitful and effective if the decision can be made based on a holistic consideration of priorities and constraints.”

As an example, Liu cites Clemson University’s Clemson Thinks2 program for sophomores, which integrates teaching, learning, and assessment of critical thinking through a wide range of seminars and uses multiple nationally validated assessments to gauge results.

“All the potential pitfalls in assessment de- sign, implementation, and analysis require institutions to purposefully rely on profes- sionals to ensure that their assessment activity represents a scientific process,” Liu advises.

Institutions that build their own assess- ments need to rely on professionals with experience in assessment development and psychometrics to ensure that methods are theoretically valid and technically sound. They also must be careful not to inadvertent- ly disadvantage minority students or those from lower economic status, students with disabilities or English-language learners.

Institutions must also be aware of the effect of students’ test-taking motivation — or lack thereof. Students know that institution-wide assessments typically have little to no impact on their grades, and the assessment results often reflect that. Liu has extensively researched and written

about this phenomenon. In a 2012 paper, Measuring Learning Outcomes in Higher Education: Motivation Matters, she warned: “It is dangerous to make conclusions about the quality of U.S. higher education based on learning outcomes assessment data without considering the role of motivation.” In a study that she led (The Effects of Motivational Instruction on College Students’ Performance on Low-Stakes Assessment, 2015), 58 percent of students in a control group were identified as “unmotivated examinees.”

Reminders of the importance of the data to the institution’s rankings — and to the perceived value of its diplomas — can help. So can administering tests online, where re- sponse times can be tracked and utilized to flag students who may be rapidly guessing in order to finish quickly. In Ten Years After the Spellings Commission, Liu advises a two- pronged approach of boosting motivation before the test and filtering out unmotivated respondents afterward.

Evaluating a range of assessment tools

Working with professionals for assessment implementation and interpretation

There is no one tool or method that is suitable for all assessment needs.

4

Assessing the Skills that Matter

Liu stresses in Ten Years After the Spell- ings Commission that faculty involve- ment should be embedded in all of the above-described elements. “Faculty need to recognize the value of the identified learning outcomes and be made aware of the purposes and processes of assessment activities. It is extremely important to cul- tivate a culture of assessment that values evidence-based and data-driven decisions and also encourages faculty to see assess- ment as an opportunity to gather ongoing

data for instructional improvement as op- posed to a barrier or interruption to their daily work.”

To conclude Ten Years After the Spellings Com- mission, Liu chose a line from a 2006 Com- mission on the Future of Higher Education report that she considers timeless, “Better data about real performance and lifelong working and learning ability is absolutely es- sential if we are to meet national needs and improve institutional performance.”

Engaging faculty and cultivating an assessment culture