Discussion Question

Chapter5.pdf

Home >Education homework help >Discussion Question

105

looking at cHange management as design is one thing; measur- ing outcomes and impacts of these changes is another. This chapter deals with questions of assessment and research in a higher education design organization. We contend that design should be grounded in these practices, from the assessment of innovative designs that inform decision-making to the production of scholarly work that contributes to research domains. We describe approaches to assessment and re- search that are useful for conceptualizing problems and creating op- portunities for change within a design organization. Along the way, we use a few examples to illustrate the role of assessment or research in our design work. The key takeaway is that assessment and research are vital to a design agenda that seeks to be transformative.1

Sometimes assessment and research are just that: ways to observe and understand the environment with the goal of developing theory and accumulating knowledge. In the Hub for Innovation in Learn- ing and Technology, which is focused on transformation in an institu- tion that values and rewards assessment and research, these activities have additional symbolic value. When a “third space,” such as the Hub, uses familiar terms, it becomes clearer to partners that a relationship

5 Assessment and Research in a

Higher Education Design Organization

WILLIAM F. HEINRICH and REBECCA L. MATZ

Grabill, Jeffrey T., et al. Design for Change in Higher Education, Johns Hopkins University Press, 2022. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/franklin-ebooks/detail.action?docID=6845327. Created from franklin-ebooks on 2025-01-11 19:47:25.

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

106 design for chAnge in higher educAtion

of trust and accountability—necessary for achieving a design agenda— is possible.

The skills of an investigator include the ability to frame a problem theoretically, select and justify an assessment or research design, iden- tify and evaluate sources, place a problem in both a literature-based and local context, deploy methods to collect and analyze data, infer meaning, and communicate with peers. Design work in higher edu- cation is no different, requiring assessment and research in project scoping, implementation, and evaluation. Importantly, the research- ers, scholars, and artists who can do this work are located all across, and in some cases outside, the institution, and at multiple levels of the typical academic hierarchy as well (students, faculty, and staff).

Defining Assessment and Research in a Loosely Coupled System

Often, the terms assessment and research carry a specific set of mean- ings in higher education (the same can be said for evaluation, which we won’t take up here). Assessment and research with respect to stu- dent learning have different purposes in higher education yet often use similar methods, so it is useful to clarify differences between the terms as we use them here. Assessment of and research on learning are some of the more sacred ideas in the academy, so deploying these terms carefully and knowing how they work for different stakehold- ers is especially important when it comes to working on design across the institution. Specifically, it is important for design-driven investi- gators to clarify the nature of the question they’re asking: Are they primarily trying to investigate phenomena and apply findings to a gen- eralizable space (research)? Or are they evaluating the nature, quality, or effectiveness of a campus activity for internal purposes (assessment)? Reward structures aside, neither approach is more or less correct, but there are pros and cons to each.

Assessment in higher education is used at multiple scales and layers of the institution, from student learning to program assessment to as- sessment for accreditation. In all cases, however, assessment is focused

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 107

on improving the nature of the local educational context and ulti- mately, therefore, on student learning and the student experience.2 Linda Suskie’s guide to assessing student learning identifies four steps: establish goals, create learning opportunities, systematically collect data, and use results to improve.3 Assessment in higher education is an idea well used and understood at the student learning level yet often loathed at the institutional level by the same people. So, when we talk about assessment in our design practice, we are usually talking about a general form of assessment rather than a specific approach or instrument. We talk about the value of assessment and then about how to operationalize that value. For example, program assessment for accreditation is an important value proposition, but the value is much different from that of a well-designed assessment survey to inform and improve a student experience. If the survey is about an activity in that program, obviously the details are linked. But the ex- perience of the people we’re talking to is a key factor in how we talk about assessment, adapting to the needs of our partners.

Research in higher education that supports designing and redesign- ing experiences attempts to accomplish a similar goal as assessment— improving the nature of a local educational context—while also speak- ing to a broader audience, with a coherent theoretical perspective and study design. Scholarly outlets such as academic journals and confer- ence presentations require, implicitly or otherwise, alignment with disciplinary norms for social and applied science research. It is, there- fore, necessary that research in support of a design agenda draw on research traditions and expertise in education and the social sciences. It is additionally necessary to draw on disciplinary knowledge for the problem at hand and from communities that identify with discipline- based education research and the scholarship of teaching and learn- ing.4 In the midst of focusing on design, testing theory and creating new knowledge are often possible.

The Hub, and therefore its application of assessment and research, exists within a large educational institution, what Karl Weick describes as a loosely coupled system.5 A loosely coupled system has a decentral- ized organizational structure, which supports collaborative goal set-

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

108 design for chAnge in higher educAtion

ting and prioritization, adaptive responses, autonomy in work, and alignment. The system acts as an informational network, which, in turn, helps stakeholders coordinate a robust response to problems. Trust generated within the group creates an information portal for extended networks of loosely coupled campus partners, who collabo- rate to tackle a similarly loosely coupled set of problems. Weick de- scribes seven functions of loose coupling that can drive organizational behaviors.

1. Loose coupling lowers the probability that the organization will have to—or be able to—respond to each little change in the environment that occurs.

2. Loosely coupled systems preserve many independent sensing elements and therefore “know” their environments better than is true for more tightly coupled systems, which have fewer ex- ternally constrained, independent elements.

3. A loosely coupled system may be a good system for localized adaptation.

4. Loosely coupled systems preserve more diversity in responding than do tightly coupled systems and therefore can adapt to a considerably wider range of changes in the environment.

5. If there is a breakdown in one portion of a loosely coupled sys- tem, this breakdown is sealed off and does not affect other parts of the organization.

6. In a loosely coupled system there is more room available for self-determination by the actors, leading to greater efficacy.

7. Loosely coupled systems seem to hold the costs of coordination to a minimum. Despite this being an inexpensive system, loose coupling is also a nonrational system of fund allocation and is, therefore, unspecifiable, unmodifiable, and incapable of being used as means of change.6

The university’s behavior as a loosely coupled system yields both affordances and constraints for assessment and research. As an ex- ample of an affordance, the Hub is positioned in a boundary-spanning role. Faculty and staff work there, yet the Hub is not a typical academic

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 109

department or unit bounded by one discipline. Administrators work there too, but the Hub is not solely an administrative entity. As de- scribed elsewhere in this book, the Hub operates as a third space on campus, a design group within a research university. This positioning within the loosely coupled system allows the Hub to be seen as a safe and creative space for those who seek it, while allowing other parts of the organization to ignore the work of the Hub and carry on with their daily activities.

In contrast, the loosely coupled nature of the system also presents challenges. For example, university actors are not always aware of the affirmative functions of loosely coupled systems or how best to exploit them. Hub staff are therefore responsible for engaging approaches that recognize and work to leverage the useful parts of these systems. Doing so helps our partners, but the Hub staff also operates without author- ity to make changes themselves and so must rely on relationships, networks, and taking advantage of the space left by nonsystemic plan- ning in a loosely coupled system (the zones of ambiguity mentioned in the introduction).

The Hub uses assessment and research (among other tools) to con- nect loosely coupled elements within the institution. That is, assess- ment and research in the service of design help to provide frames that make sense of the loosely coupled system for a partner unit that is trying to effect change. Using the scaled and layered structures of as- sessment and research, we can better understand how the organiza- tion is linked together structurally, but also, politically, relationally, and symbolically.7 A key challenge for the Hub is to effectively func- tion as a boundary-spanning unit and leverage design work for good practices in the institution while subtly shifting the larger organiza- tion toward designing for the student experience in the presence of other considerable priorities (e.g., research).

Finally, it is important to understand a large university as a loosely coupled system because most of the partners we work with have a change mandate in hand, and we need to know how the larger system is operating around them. Working in the system allows us to best leverage its various elements and help co-design solutions for our part-

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

110 design for chAnge in higher educAtion

ners. For example, sometimes a project seeks to make something ob- solete that is not working in order to create space for a new, more sustainable solution. In a loosely coupled system, objectives need not always be focused on creating something new but could be thought of as removing existing barriers in order to accelerate changes. We as- sume that the assessment and research work is responding to a design question, giving the work a clear purpose that helps the university meet its goals to transform itself for student success. To be effective, we must first know what to do and then gain insights into how to do it.

Tactical Uses of Assessment and Research in Design Work

In any project, research and assessment are critical to success. The Hub’s focus on improving student experiences, academic programs, and in some cases, the professional learning of colleagues has yielded a pattern of research and assessment approaches and methods de- ployed at the beginning, during, and at the end of a project. The pur- poses of research and assessment vary for each project but remain aligned with the improvement goals set at the beginning.

In design approaches, we often lead with a key question, which amounts to “How might we make things better for people?” This question- driven approach is aimed at preventing ourselves and our partners from leading with a solution. To understand the context of the challenge, the Hub uses a variety of assessment and research methods to help everyone understand the landscape, including stake- holder needs assessments, such as interviews, focus groups, or obser- vations of an environment. For more contextual understanding, we might use landscape analyses, such as market research in the cases of a new or revised academic program design. These are just a few exam- ples of methods used to understand the starting point. Design work then proceeds past stakeholder engagement to problem definition with stakeholders, ideation, prototyping, testing, and implementation. For examples of activities to help with every stage of design, a graphic from Libby Hoffman (figure 5.1) helps visualize some available methods.8

As a project matures and becomes more complex, two tools that

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 111

help with process description and assessment are a logic model and a theory of change.9 Logic models typically follow a structure similar to those shown in table 5.1, describing resources, activities, outputs, out- comes, and impact. This simple approach helps users think through their work from start to finish or, working backward, from hoped-for outcomes to needed resources, processes, and actions that will sup- port accomplishing the stated goals. A theory of change builds on the same processes as a logic model and, to a first approximation, adds a lot of detail. Of note, the theory of change helps detail multiple layers of activity, assumptions, and barriers to action, all of which can be evaluated. In a course or across an academic or cocurricular program,

Figure 5.1. Libby Hoffman’s Map of Design Methods

UNDERSTAND THE

PRODUCE A

ASSESS

CHOOSE

BUILD & TEST OB

SE RV

IMAGINE CONNECT

STRATEGIZE

DATA COLLECTION

POLARITY MAP

FUNCTION

SCENARIO BUILDING

STOCK-FLOW DIAGRAM

SILENT FEEDBACK

SCENARIO TESTING5-MIN MOCKUPS

DIVIDE & CONQUER

COMPETE

YES AND 6 SKETCHES

INVIDIDUAL & GROUP

IDEATION WACKY

X Y

VALUE MATRIX EMOTIONAL

INTELLECTUAL

PRACTICAL INNOVATIVE

FLOWS STOCKS

5 WHYS/ LATTER OF INFERENCE

VICIOUS V. VIRTUOUS CYCLE

PRIORITIES

BELIEFS BEHAVIORS

INPUTS OUTPUTS

ACTIVITY LONG TERM

LATER NOW

ENDS V. MEANS

GRAPHING

PRIORIZATION

EFFECTIVENESS3X3

POLARITY MAPIMMED, SHRT, LNG TERM

TO P,

M ID

, B OT

U P

RESOURCES

VALUE MATRIX

5 WHATS

INTERVIEWS

SEE HEAR

THINK/FEEL

SAY/DO

5 SENSE OBSERVATIONS

UNPACK THE CAR

CONCEPT MAP

PERSONAS

HOW MIGHT WE

GROUPING & LABELING

USER BEHAVIOR LOCATION ICEBERG

EVENTS PATTERNS

STRUCTURES VALUES

5 HOWS

WHAT’S IN THE BOX? INFINITY

USES

DAY IN THE LIFE/ CUSTOMER JOURNEY MAP

STAKEHOLDER INSIGHT/ USER FEEDBACK

DOT VOTES

SYMPATHY MAP

EQUILLIBRIUM

IMPACT ASSESSMENT

HUMAN

MATERIAL

HORSESHOE/ BARRIERS

SMART GOALS

INTERTIA

DOUBLE FEEDBACK LOOP

GOAL STRATEGY PLAN

NEE D

PR OB

LE M ST

AT EM

EN T

RESPONSE

PROTOTYPE

COLLECT APPLY

BUSINESS MODEL CANVAS

INCENTIVES

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Table 5.1. Logic Model Program Implementation Template

Resources Activities Outputs Short- & Long-Term Outcomes Impact

In order to accomplish our set of activities, we will need the following:

In order to address our problem or asset, we will accomplish the following activities:

We expect that, once accomplished, these activities will produce the following evidence or service delivery:

We expect that, if accomplished, these activities will lead to the following changes in 1–3 then 4–6 years:

We expect that, if accomplished, these activities will lead to the following changes in 7–10 years:

Source: W. K. Kellogg Foundation, “Logic Model Development Guide,” https://www.aacu.org/node/5682, 2004.

G rabill, Jeffrey T

., et al. D esign for C

hange in H igher E

ducation, Johns H opkins U

niversity P ress, 2022. P

roQ uest E

book C entral,

http://ebookcentral.proquest.com /lib/franklin-ebooks/detail.action?docID

= 6845327.

C reated from

franklin-ebooks on 2025-01-11 19:47:25.

Assessment and research in a design organization 113

a theory of change can help identify multiple outcomes (competencies) and multiple locations for action (courses or sets of courses). A theory of change should be built on a more general theory or framework for change (those ideas that go beyond any single project) and be more operational in nature.10

The efforts to detail the layers of action in a theory of change or logic model typically emerge during a prototyping phase. The tools we have mentioned here can both act as a long-range planning tool for the project itself and serve as a basis for evaluation and peer review. The theory of change makes a project visible to external stakeholders, creating accountability for quality as well as early opportunities for feedback and iterations, both important aspects of design. A well- developed theory of change is often co-constructed by us with our partner, and the act of co-creating usually yields important insights, gaps, and intellectual outcomes that improve the overall project. A theory of change then serves as a reminder of the plan, a basis for evaluation, and a way to share with new participants. It also provides one way to talk about work publicly.

Across the Hub’s portfolio, staff members use a shared logic model in the form of our design process (see chapter 2 for more on that pro- cess and variations on the model). Because we integrate our process with Asana, our current project management software solution, our process functions to detail the layers of activity and effort in a project. We build in data-gathering tools to better identify pivots in the trajec- tory of projects, which facilitates our ability to see larger patterns and needs across projects. As a model of practice, we can evaluate prac- tices across projects for opportunities to improve and better serve the needs of the campus.

Measuring project outcomes is an important activity across the Hub’s portfolio, and it is critical when we come to the end of a project. To answer the question, “How do we know if our work has been suc- cessful, had any value, or made an impact?,” we have to measure against goals. One key challenge in this work is that all the projects in the Hub’s portfolio are, by design and definition, new work. Inno- vation presents a particular challenge because project leads and part-

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

114 design for chAnge in higher educAtion

ners have to anticipate the milestones on the path toward success and what the eventual goals might look like. This means that the key and challenging question is something like, “What gets measured in the face of a new problem?” Here, we use a set of process-based objectives (table 5.2), which, when implemented fully, link the success of the Hub, in aggregate, to the success of each project. Without an external standard, however, participation, iteration, and communication are essential in our project and partnership model.

Project-level objectives are typically co-developed in a project char- ter and can be assessed and researched at the end of the project, yield- ing information about the ways a project might move beyond outputs (e.g., a curriculum plan) to outcomes (the plan is adopted and imple- mented by faculty) to actual impact (students demonstrate more usable knowledge than previous comparable cohorts). Surveys, case studies, interviews, focus groups, program evaluations, and other more sum- mative approaches are the typical methodologies used to gather infor- mation on the success of a project. Measures of student learning and progress toward degree completion can also be employed to help eval- uate project goals. Assessing project objectives is a way to determine

Table 5.2. Hub Performance Objectives

Objective 1. Facilitate human-centered design (HCD) processes that elicit stake- holder perspectives to inform needs before seeking a solution.

Objective 4. Partners implement the designed learning experience, pro- gram, service, or intervention and assess measures of impact.

Objective 2. Contribute to partners’ organizational transformation process and help build capacity for partners to lead future transformation. (Transfor- mation is loosely defined as a personal or organizational identity shift that accompanies behavior or policy changes. Transformation would be identified after project delivery as an impact of the efforts.)

Objective 5. Partners prepare program- matic, political, financial, and edu cational conditions for sustaining valuable student experiences derived from the project.

Objective 3. Include student perspectives and insights from the HCD process to help partners create a more valuable student experience.

Objective 6. Partners have such an exemplary innovation that it warrants consideration for scaling up or scaling out (this may require a time delay to measure).

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 115

if and how the partner experienced actual changes as articulated in the project charter.

Organizational Uses of Assessment and Research

Once again, there is no design without assessment and research. While that has been true across our project portfolio, a statement like this means something different in higher education. We label forms of inquiry as assessment and research that might not be widely accepted inside higher education but that would be uncontroversial in other professional contexts. Our goal here is to provide case examples of assessment and research that have helped our design work succeed. Drawing from some of the longer-term projects in our portfolio, we intend to show how we have used more formal assessment and re- search in support of designing for change.

One of the more stable areas of work for the Hub has been in learn- ing analytics,11 which has taken various forms over the last several years. Initially, the desire for developing MSU’s learning analytics function came from a general recognition that we needed to make bet- ter use of the data on campus to improve the institution, to leverage analytics in academic business processes, and to enhance students’ ability to successfully complete their degrees. As is often the case at a large institution, small, decentralized (and loosely coupled) pockets of people all across campus were doing this work, but they lacked a coordinating body and the benefit of collaboration and knowledge- sharing. Moreover, for some functions we were relying on external vendors, who in the short term were useful to business practices but would eventually fail to meet the institution’s needs. These vendor tools are generally designed for narrow-use cases and include propri- etary models, which limits our ability to effectively use and adapt them.

In this milieu, MSU’s Learning Analytics Group (LAG) was com- posed of individuals drawn from relevant units across campus and charged with the mission to coordinate and develop analytics resources and activities on campus (it was a good example of an “adhocracy,” which we discuss in chapter 6). One goal of the group was to encourage

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

116 design for chAnge in higher educAtion

a culture of data-driven decision-making in support of increasing the diversity and success of our students.

Encompassing the expertise of IT, administrative, curricular, and research professionals across campus, LAG existed to build and link together databases, analytical tools, and expertise for analyzing MSU’s institutional data. The tool-building work enabled MSU to pur- sue policy- relevant research in order to effect specific improvements in undergraduate student learning and success at MSU, by means of course reform, curricular innovation, and advising practices. LAG was open to faculty and staff interested in using data to improve student learning and success, and most members worked with quantitative institutional data (data stored in university systems), classroom-level data, or both. The group intentionally drew participants from differ- ent units across campus, such as the Hub, IT, institutional research, undergraduate education, financial aid, and colleges and departments.

LAG became a group that encouraged data-informed leadership of the university with members collaborating across units, learning from other perspectives, discussing data issues and interpretation, and getting feedback for projects through friendly-yet-serious peer review. Within LAG, assessment work for the overall purpose of im- proving the student experience included projects such as investigat- ing the effects of a cohort-based experience for first-year students in general studies courses; identifying admittance patterns and six-year outcomes for students with limited-enrollment majors (e.g., business and engineering); and assessing the likelihood for a student to go on probationary status, based on the probationary status of the room- mates and floor mates within the university housing system, among many other factors.

One key example of LAG’s success in this assessment work was the analytical work done to support MSU’s credit momentum campaign. Administrators noticed an overall decrease in credit taking for the student body and a sharper decrease among students for whom we observed opportunity gaps (e.g., first-generation students). We also had indicators that those students who completed at least thirty cred- its in their first year maintained academic momentum and had better

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 117

graduation outcomes. The indicators suggested that this was true across demographic categories and not simply the result of better- prepared and better-supported students taking more credits.

What followed was LAG’s systematic analysis of our historical data to see, if it was true, what level of credit completion in the first year appeared to be related to good outcomes and to whom this applied. We learned enough to be comfortable with designing and pushing a credit- momentum campaign. The assessment of the campaign is ongoing, but it has been largely successful to date, increasing the number of stu- dents who are on track to complete their undergraduate degree in four years. Beyond the success of the program and the students themselves, the process itself was successful. As a coordinated system-wide effort to prioritize credit momentum, the process of changing rules, prac- tices, and norms helped many individuals think differently about the idea of pushing (some) students to take additional credits. This and other LAG projects have addressed the overall design question of how to improve student success and close opportunity gaps at MSU. In act- ing on the analyses and reports, these activities were able to facilitate improving the student experience.

When it comes to research, we have developed a core set of staff, including graduate students interested in pursuing research projects within the portfolio, in addition to their own independent projects. Such work has included supporting and conducting design-based and other “translational” research focused on the student experience and student learning. When projects generated lessons that were surpris- ing or generalizable enough to be valuable to a research literature and external colleagues, the project partners and Hub staff would report these impacts in conference papers and presentations as well as jour- nal articles.

An important relationship in the Hub’s portfolio that was fruit- ful for generating research products was a partnership between the Hub, the Department of Mathematics, the Program in Mathematics Education, and the Office of the Associate Provost for Undergraduate Education. In this work, staff in the Hub and the other partner units supported faculty in the Department of Mathematics as they made

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

118 design for chAnge in higher educAtion

significant structural and curricular changes to introductory mathe- matics courses.

Specifically, a developmental course that, on the whole, was not sup- porting student success was replaced with a series of credit-bearing courses with embedded support to best facilitate success for students who placed at this level. Similar developmental (sometimes called remedial) mathematics courses are common across the landscape of higher education.12 While the success rate for MSU students in this developmental course was reasonable in light of the large number of students nationally in developmental mathematics courses, faculty, staff, and students at MSU felt that students could be much better sup- ported and, overall, more successful.

A major goal of the revised curricular pathways is to better prepare students with the mathematics that would be useful to them in their degree programs and future careers. Toward this end, students not tracking for calculus (e.g., those majoring in history) were able to ful- fill the university mathematics graduation requirement by taking two newly developed quantitative literacy courses that explore the mathe- matics relevant to health and risk decisions, politics, finance, envi- ronmental issues, and other topics relevant to the students them- selves.13 These courses were designed with sufficient adaptive support that students who otherwise would have enrolled in the developmen- tal algebra course could enroll directly in the quantitative literacy courses and still be successful at MSU.

Students who placed at the developmental level and had an interest in degree programs that require calculus (e.g., engineering) also had a new curricular path. Previously, these students would have taken one semester of developmental algebra and one semester of college- level algebra. In the revised pathway, some students now directly enroll in the college-level course while others take a two-semester “stretched” version of the college-level algebra course. While this new sequence still takes two semesters to finish, there are several added benefits: both college-level courses are credit-bearing, whereas the developmental course was not; the college-level courses are offered in relatively small, in-person course sections, whereas the developmen-

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 119

tal course was generally offered as one large, asynchronous online course; and the curriculum for the entire two-semester sequence is better aligned internally as well as streamlined for the follow-on cal- culus courses.

In evaluating the impact of these major curricular and structural changes, we assessed some aspects of the changes that were locally important (such as if and how the “backlog” of students needing to finish their university mathematics requirement were able to do so with the quantitative literacy courses). We also investigated several questions that we knew would be of wider interest, given that, at MSU and elsewhere, supporting students in introductory undergraduate mathematics courses is of paramount importance to those concerned with student success in STEM degree programs.14

Asking and answering these design questions through a research lens allowed us to identify the reasoning patterns students were using in one of the new quantitative literacy courses, explore the alignment between learning objectives and course assessments in the same quan- titative literacy course, and quantitatively evaluate the effect of em- bedded remediation as a pathway for equitable access to introductory math courses.15 These findings spurred conversations among scholars both within and outside the institution, and importantly, informed the design process.

Concluding Thoughts: Ethical Assessment and Research in a Third Space

In the midst of planning and carrying out these projects, we have gleaned some useful ideas for doing ethical assessment and research specifically in an organization like the Hub. There is no need here to reproduce a list of typical ethical considerations in scholarly work— taking care not to plagiarize sources, being transparent with the so- licitation and inclusion of coauthors on reports and research prod- ucts, facilitating a culture of informed consent, storing identified data securely, and so on. Such considerations are available in fine form elsewhere.16 Here, instead, we share four considerations for doing eth-

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

120 design for chAnge in higher educAtion

ical research and assessment work specifically in design organizations that are, perhaps, less obvious. We contend that these considerations emphasize an organizational culture of character, not personality.17

First, the pervasiveness of a busy culture in both higher education and American life more generally can lead to a frenetic pace of work, which is actually a liability. In many ways, students are subjected to the social norm that stuffing their schedules full of extra courses, work experiences, extracurricular clubs, and incessant volunteering are the keys to success following graduation. Faculty similarly com- mit to ancillary involvement in too many research projects, mentor groups that are too large, and agree to serve on too many committees. Whether real or perceived, the sense of busyness is palpable and feels increasingly acute with the fallout from the COVID-19 pandemic.

In this context, it is tempting for the assessment and research work in a design organization to follow a sense of commitment to every- thing and attempt to keep up with the “sprint” schedule in innovation. Such fast movement is not always productive toward the change goal. While design sprints are hugely important to projects, it is not sus- tainable or necessary for all projects to sprint. The pace of teaching and learning on which much of the Hub’s research and assessment portfolio is built is slower than much of the other work we do in the Hub. Assessment projects, research projects, and indeed, change proj- ects themselves, must be carried out with a laser-like focus and only sometimes with sprints, to deliver the product or process that will improve the partners’ experience.

Second, the university is not nimble by default or design. In this slowly changing environment, it is important to think about moving fast and prototyping quickly, but we should not move fast simply for the sake of moving fast or because this helps us feel as if we are enact- ing change when we really are creating noise and burden. That is, some projects and indeed some changes require sustained attention, resources, and relationships that last for years. As Diane Sawyer has said, there is no substitute for paying attention, and some changes we seek might be measured—designed, even—in careers, not sprints.18

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 121

As an example, when we stewarded the design and creation of a new cocurricular record, scholar-practitioner staff conducted both research and assessment to create good designs for the record and also repeatable, theory-driven practices. In delivering the project, a new director of the project assumed ownership of the platform and began to develop practices to make it work in the long run. What was de- signed in the Hub will change the way students report on and value cocurricular experiences, but those designs were developed fairly quickly and then deliberately carried forward in a sustainable way within the context of the slowly changing university.

Third, it is important that a design organization intentionally make space for dissenting voices. Any organization on the edges of a tradi- tional institution will draw some fire for being different. In our higher education context, we must be flexible enough to fully consider criti- cism of our assessment and research practices and honestly sort the beneficial from the rest. We ask ourselves which assessment and re- search practices are sacrosanct and valuable, and which are just sac- rosanct. Organizations on the edges must be able to respect and learn from the systems from which they were born and indeed are trying to change. There is no room for an “us versus them” mentality, and myo- pic generalizations about the partners (generally faculty) are not help- ful. In this way, those doing assessment and research in organizations like the Hub—and, indeed, the organization itself—requires a dispo- sition of “generous orthodoxy,” to borrow from theologian Hans Frei.19 Change is hard for most people and organizations, and in our context, the Hub represents that change, quite noticeably. Holding open space to engage in dissent, as a means of creating understanding, is an im- portant strategy for building trust and coalitions across the campus.

Fourth, change is both a short- and a long-term goal. Particularly for long-term change goals, research and assessment can yield good outcomes by supporting follow-through in the design work. After the reports are written and the papers published, much is revealed in the maintenance activities, or lack thereof, of those who completed the assessment or research, those who perhaps designed the broader

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

122 design for chAnge in higher educAtion

project, and campus partners. That is, if assessment and research are foundational to a given design, we should look for and perhaps find some evidence of improved outcomes. A key component of designing for sustainability, then, is building in regular assessments and itera- tive improvements, even after the official “end” of the project; main- tenance for the change, in other words. Depending on the nature of the project, someone involved from the design organization might conduct these follow-up assessment and research activities, but more likely someone in the local context would continue to follow the out- comes and identify evidence of change for the local community. Push- ing for initial outcomes only at the surface level yields a performative, surface-level result in the long term.20

Finally, returning to the idea of the Hub as a “third space” organi- zation, it is important to consider how credit is ascribed. On the whole, partnering units should be recognized as primary drivers and receive primary credit even when the balance of work falls to Hub staff. While this may seem unnecessary at best or disingenuous at worst, this dis- position reflects that the Hub (unlike a traditional department) doesn’t exist without partners to serve. In tension with the Hub’s commit- ment to our partners’ success is the Hub’s willingness to be wrong. Given the possible constraints of busyness, pacing, and dissent, typi- cal partners come to us seeking help because they believe the Hub will help them change efficiently. The paradox of efficient design, though, is taking small bets, missing a few (being wrong), and quickly improv- ing. Communication of these expectations and paradigms is impor- tant for the Hub to be able to also absorb responsibility—to a point— for a change process not succeeding. Our successes only go as far as the successes of our partners, and in this way, the Hub exists as an orga- nization supporting the work of the institution.

These ethical considerations, loaded as they are with complex power relationships, cannot be dismissed. They are integral to what it means to work in a loosely coupled system like MSU. It is true that assess- ment and research are typical activities in any institution of higher education, but conducting these activities in a design organization like the Hub requires particular kinds of attention. We don’t practice

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .

Assessment and research in a design organization 123

within a department or enjoy the “protections” of disciplinarity. The campus community at large is generally unfamiliar with the role of research in design, and our efforts are, by definition, transdisciplinary and therefore require care, communication, and generosity.

At the same time, of course, the work can yield unforeseen collabo- rations and the opportunity for addressing new kinds of research questions. In addition to producing actual data and evidence that sup- port decision-making around improving the student experience, doing research and assessment work confers some legitimacy to the organi- zation since it mimics work that those in higher education are already familiar with. In this way, the assessment and research activities in the Hub have supported the overall success of the organization. More importantly, a commitment to research, assessment, and other forms of inquiry are required if we are designing.

C op

yr ig

ht ©

2 02

2. J

oh ns

H op

ki ns

U ni

ve rs

ity P

re ss

. A ll

rig ht

s re

se rv

ed .