650- MODULE 9 DISCUSSION

57Performance Funding for Higher Education

Obstacles to the Eff ectiveness of Performance Funding

A SENTIMENT THAT IS QUITE COMMON among institutional offi cials is that performance funding has had little real impact on insti- tutional performance and that it is largely a symbolic practice. Th ree studies on Tennessee and one on North Carolina cited several administrators and faculty members who argued that performance funding has simply been a rote activity, with actors only going through the motions of collecting data and submitting reports. For example, the faculty senate chairman at one North Carolina community college dismissed performance funding as “mere paper shuffl ing” (Harbour & Nagy, 2005, pp. 457–458). A department chair at the University of Tennessee at Knoxville argued: “Th e impact that this has had on us in the department has really been to simply add another administrative task. I don’t think . . . that it has changed the way a single faculty member teaches, the kind of materials that a single faculty member presents. It has had no impact on our curriculum” (as quoted in Hall, 2000, pp. 78–79).

Th is pervasive undercurrent of skepticism about performance funding refl ects the many obstacles that it encounters when implemented. Among the obstacles that crop up in the research literature on the impacts of performance funding are the inappropriateness of many performance measures employed; instability in funding levels, indicators, and measures; the brief duration of many performance funding programs; funding levels that are too low; short- falls in regular state funding for higher education; uneven knowledge and expertise about performance funding within institutions; inequalities in

58

institutional capacity; and resistance and “game-playing” by institutions. Th ese obstacles are discussed in this chapter.

Inappropriate Performance Funding Measures Many studies discuss the reservations held by administrators and faculty about how well diff erent indicators and their measures capture the real per- formance of their institutions. In fact, a faculty leader at one North Carolina community college dismissed performance indicators in North Carolina as “pretend” measures (Harbour & Nagy, 2005, p. 458).

Learning Gains

In eight studies of Tennessee, two of South Carolina, and one of Florida, institutional offi cials expressed skepticism about the validity of the learning assessments being used as measures of institutional performance. In Tennessee, respondents were particularly skeptical that the state-mandated assessment of general education and the major fi eld exit standards adequately captured what faculty aimed to teach (Banta & Fisher, 1984, p. 34; Freeman, 2000, p. 98; Hall, 2000, pp. 95–96; Shaw, 2000, pp. 88–89; Tanner, 2005, p. 85; Williams, 2005, p. 93). For example, arguing in terms echoed by sev- eral other interviewees, an administrator at the University of Tennessee asserted:

We know general education is important because we all do it. But we don’t know what it is and we don’t know how to fi gure out whether this school or that school is doing a good job with it. We don’t know that any better now than we did twenty years ago. And so, giving people money or withholding money on the basis of general education is a very slippery proposition. (Quoted in Hall, 2000, p. 96)

Retention and Graduation Rates

Three studies each on Florida and Tennessee and one each on Ohio and Washington raised concerns regarding how retention and graduation rates are

58

59Performance Funding for Higher Education

measured. Community college offi cials and faculty in Florida asked why col- leges should be penalized if vocational students leave college without a degree for well-paying jobs during times of economic growth. Th e community col- lege offi cials and faculty argued that it is enough if students have been able to reach their goals or at least have attained a useful level of education (Bell, 2005, p. 109; Gray et al., 2001, p. 32; Morris, 2002, p. 131; see also Dougherty & Hong, 2006a, pp. 70–71). Furthermore, an Ohio study noted that graduation rates usually do not take transfer into account; thus, com- munity colleges with students who successfully transfer to a 4-year college without having fi rst received an associate degree usually cannot count such students as graduates. In fact, transfer students are often mistakenly treated as if they were dropouts (O’Neal, 2007, pp. 135–136).22 Finally, some studies raised the issue that graduation rates do not take into account diff erences between institutions in the academic preparation or degree ambitions of stu- dents. For example, community colleges tend to have higher percentages of students with social and academic disadvantages that make it hard to get a degree, even if that is the students’ intent (Dougherty & Hong, 2006a, pp. 70–71).

A problem has also been noted with the use of numbers graduating rather than rates of graduation. A college could increase its numbers graduating, even if the graduation rate is declining, if it is experiencing sizable enrollment increases (Jenkins et al., 2009; Shulock & Jenkins, 2011, p. 10).

Job Placement Rates

Concerns over job placement indicators cropped up in two studies each on Florida, Tennessee, and Washington. A major criticism by college offi cials and faculty was that job placement rates are dependent on the state of the local economy, which varies over time and by region, in ways that are not under the control of the colleges (Banta et al., 1996, p. 32; Bell, 2005, p. 137; Dougherty & Hong, 2006a, p. 71; Nisson, 2003, p. 113). An offi cial at a Washington community college noted:

I think the measures they chose were so ridiculous it became obvious. I mean one of them was a wage-level, a certain dollar

60

amount per hour that every college had to average in their pro- grams [that graduates attained]. Well, [my college] is always at the top because we have a lot of high-wage programs but Yakima and Walla Walla [rural community colleges], those places, no, they’re never going to get there probably. And it doesn’t make any sense. (Quoted in Dougherty & Hong, 2006a, p. 71)

In Florida, an additional concern was that job placement indicators hurt institutions whose graduates obtain jobs out of state, because those jobs are not counted by state datasets (Gray et al., 2001, p. 40).

Institutional Diff erences

Th ree studies on Washington, two on South Carolina, and single studies of Florida, Missouri, Ohio, Pennsylvania, and Tennessee noted the concern of college offi cials about how state performance funding programs failed to take into account diff erences among institutions in their mission and in their capacity to meet performance demands. With regard to mission, tensions arose in Tennessee over perceptions that the performance funding program insuffi ciently acknowledged that institutions have diff erent missions. For example, a university administrator argued that the research mission of that institution was not refl ected in its performance funding results (Hall, 2000, p. 95). Meanwhile, Washington community colleges with a greater focus on academic transfer argued that they are at a disadvantage because the highest potential for amassing performance points under the new Student Achievement Initiative occurs in adult basic education and developmental education (Jenkins et al., 2009, p. 37).

With regard to capacity to meet performance demands, Washington com- munity colleges serving greater numbers of at-risk students perceived them- selves to be at a disadvantage in amassing performance points because such students tend to need “costly wrap-around services” in order to succeed (Jenkins et al., 2009, p. 37).23 Similarly, in South Carolina, the state Legislative Audit Council found that:

The standardization of measures for schools in each sector raises opposition by institutional representatives. Th e measures

61Performance Funding for Higher Education

do not fully take into account the diff erences that exist among institutions within a sector. For example, a majority of the same measures have been applied to MUSC [Medical Univer- sity of South Carolina] and Clemson when they have radically different student populations. (South Carolina Legislative Audit Council, 2001, p. 23)

Similar sentiments about performance funding programs’ disregard of diff erences among institutions in their student bodies and therefore per- formance capacities arose as well in Missouri and Ohio (Naughton, 2004, pp. 89–90; O’Neal, 2007, pp. 130, 137; also see Dougherty & Hong, 2006a, pp. 71, 73).

Instability in Performance Funding Levels, Indicators, and Measures When budgets and indicators are unstable, higher education leaders fi nd it hard to decide where to focus the eff orts of their institutions and they are afraid to take chances. A survey in the late 1990s of community college and 4-year college offi cials in fi ve states with performance funding found that 40.1% of respondents rated budget instability as an extensive or very extensive problem of performance funding in their state (Burke, 2002a, p. 77).

We found validation for this point in one study each of Florida, Ohio, South Carolina, and Washington. In Florida, a dean of vocational education at a Florida community college argued that the state should “allow us to know what the rules are so that we can plan appropriately. I think that the indeci- sion each year has really put us in a predicament that has strapped us for resources” (quoted in Dougherty & Hong, 2006b). As it happens, in Florida, funding for the state’s now suspended Performance-Based Budgeting (PBB) program fl uctuated over the years. It started at 2% of state appropriations for community college operations in fi scal year 1996–1997, dropped below 1% in 2001–2002, stayed at that level until 2005–2006, and then jumped to 1.8% (Dougherty & Natow, 2010). In addition to shifts in the amount of funds that are involved in performance funding programs, the particular

62

performance indicators used to allocate those funds can also change. In Florida, the PBB program experienced considerable changes in the perform- ance indicators used. Florida’s PBB added 10 performance indicators and dropped three in the 12 years between 1996–1997 and 2007–2008, an aver- age of one change per year (Dougherty & Natow, 2010).

Th e Brief Duration of Many PF Programs Many performance funding programs do not last for many years, thus under- cutting their capacity to produce eff ects. At least half of all the states that have enacted performance funding programs have later discontinued them, often after only a few years. For example, performance funding lasted only 2 years in Arkansas and Washington (in the 1990s) and 4 years in Illinois and Minnesota (Burke, 2002a; Dougherty et al., 2012; also see Table 1). Th is short duration makes it hard for performance funding programs to eff ectively stimulate the organizational changes in colleges and universities that will pro- duce improved student outcomes.

Several factors are involved in the early demise of many performance funding programs. A key factor is higher education opposition to perform- ance funding, stimulated by a perceived lack of adequate state consultation with higher education institutions, the use of performance indicators that higher education institutions did not fi nd valid, and a perception of erosion of campus autonomy and of high implementation costs to institutions. Th is opposition hardened during the recessions of the early and late 2000s. As state appropriations for higher education faced cuts or failed to keep pace with enrollments, higher education institutions moved to protect their core state funding and turned against performance funding. Another important cause of program demise was the decision in Florida (the Workforce Development Education Fund) and Washington (the 1997–1999 program) to fi nance performance funding by holding back a portion of the state appro- priation to higher education institutions and requiring the institutions to earn it back through improved performance. Th is program design feature aroused great animosity on the part of higher education institutions (Dougherty et al., 2012).24

63Performance Funding for Higher Education

Inadequate State Funding of Performance Funding One obstacle cited in four studies of performance funding in Florida and one in Washington was the simple fact that not enough money is involved (Dougherty & Hong, 2006a; Jenkins et al., 2012). In many states, the pro- portion of state funding of higher education tied to performance outcomes has been 1% or less (Dougherty et al., 2013a). Even if the amount is higher, as it was in Florida for a while, the impact of performance funding will be undercut if it does not keep pace with rising performance. Funding under the Workforce Development Education Fund in Florida did not rise as fast as improved performance. As a result, the bounty for each graduate dropped over time (Bell, 2005, pp. 156–157; Dougherty & Hong, 2006a, p. 72; Gray et al., 2001, p. 29; Poisel, 1998, p. 94; Wright et al., 2002, p. 164).

Shortfalls in Regular State Funding Th is issue was raised in seven studies on Tennessee, four on Washington, and two on South Carolina. In Tennessee, the funds allocated under the regular enrollment-based formula had not kept pace with enrollment growth (Freeman, 2000, pp. 88–89; Hall, 2000, pp. 93–94; Latimer, 2001, pp. 95–98; Lorber, 2001, p. 85; Shaw, 2000, p. 97). For example, according to its chief fi nancial offi cer, Walters State Community College received only 89% of the base funding called for by the state’s regular funding formula for the 1999–2000 academic year (Shaw, 2000, p. 97). As a result of these shortfalls, performance funding no longer functioned as bonus funding but instead was used to make up the defi cit in regular state funding. A dean at the University of Memphis stated that performance money “gets chewed up just trying to keep the ship afl oat on a day to day basis” (Latimer, 2001, p. 95). Th is practice eventually led a number of Tennessee higher education offi cials to argue that performance funding provided the state with an excuse to cut the formula funding. Meanwhile, in Washington, even as the Student Achievement Initiative took effect in recent years, state formula funding dropped. As a result, many community college presidents and senior administrators became

64

resentful, feeling that performance funding was no longer a bonus but rather only a partial redress of dropping state support (Jenkins et al., 2009, pp. 40–41; Shulock & Jenkins, 2011, p. 12).

Uneven Knowledge about Performance Funding Within Colleges Th e eff ective implementation of performance funding has also been ham- pered by the fact that awareness of performance funding and its requirements varies greatly within institutions, with those at the top of the hierarchy pos- sessing greater understanding of and responsibility for the performance fund- ing process than middle-level administrators and faculty who also must play an important role in implementing performance funding. For example, in a survey of 2-year and 4-year college administrators in fi ve states with perform- ance funding, Burke (2002a, pp. 63–64) found that while 88% of the top administrators were “very familiar” or “familiar” with their state’s performance funding program, only 58% of the academic deans and 40% of the depart- ment chairs were familiar with it. Similar fi ndings show up in nine studies on Tennessee, three each on Ohio and Washington, and one on North Carolina.

In Ohio, a survey of 224 administrators at 13 public universities revealed that knowledge of performance funding was stratifi ed within institutions in a manner similar to that described by Burke (2002a). Executive-level adminis- trators such as presidents and vice presidents were more knowledgeable than were department/unit-level administrators, with 38% of the former but only 22% of the latter reporting that they were aware of the state’s Success Challenge performance funding program (Schaller, 2004, p. 151).25

In Washington, interviews at 17 community colleges established that, while the state’s Student Achievement Initiative (SAI) performance funding program was known and understood “fairly well” to “very well” by presidents, senior administrators, and institutional research staff , the same was not true of faculty and student support services staff . In fact, the majority of faculty members interviewed had only a limited understanding of the SAI (Jenkins et al., 2009, pp. 19–22, 33). Th e following description by a vice president of instruction at a Washington community college was typical: “With our

65Performance Funding for Higher Education

faculty we’ve told them that this initiative is happening. . . . Faculty know that something is happening, but that is the extent of it. . . . Th e faculty have had it explained to them, but if you talked to them, they couldn’t explain it back” (as quoted in Jenkins et al., 2009, p. 20).

Similar inequality of knowledge about performance funding was also reported in Tennessee (Freeman, 2000, pp. 81–82; Hall, 2000, pp. 73–74; Latimer, 2001, pp. 72–76; Shaw, 2000, pp. 66–67, 72, 86–87). At the University of Tennessee, Knoxville, an administrator noted: “I don’t think most people inside the university understand [the state performance funding system]. I would say 95% of the faculty don’t know anything about it” (as quoted in Hall, 2000, p. 74).

Th is informational inequality has been attributed to a number of diff erent causes. First, many administrators view performance data collection and anal- ysis as an administrative task that faculty need not be concerned about (Freeman, 2000, pp. 81–84; Hall, 2000, p. 73; Harbour & Nagy, 2005, p. 453; Jenkins et al., 2009, p. 21). For example, at Volunteer State Community College in Tennessee, a senior administrator noted: “[Faculty] don’t need to know. To me our campuses now are large enough, and they’re diverse enough, and they’re so specialized that people . . . really don’t have the time, energy, or intellect . . . for everybody to become an expert on the aspects of performance funding” (as quoted in Freeman, 2000, p. 82).

In Washington, administrative reluctance to widely publicize performance funding within their institutions was tied to uncertainty about its longevity and implications. College administrators were leery about widely publicizing the performance funding program until they got a better idea of how it would work and whether it would last. Th ey reportedly did not want to involve faculty in an evanescent eff ort that they might well resist (Jenkins et al., 2009, p. 21).

Further, lack of faculty awareness also may be tied to a faculty perception that performance funding is not central to the faculty role (Jenkins et al., 2009, p. 21; Shaw, 2000, p. 87). A faculty member at Walters State Community College in Tennessee noted: “As a faculty member . . . most of your work is wrapped up in your discipline, preparing notes for class, and spending time with students. Only when you as a faculty member are forced

66

to address those issues regarding performance funding, do you participate and integrate them” (as quoted in Shaw, 2000, p. 87).

In Washington, a factor contributing to faculty perceptions that perform- ance funding is not very relevant to their jobs is the fact that most of the community colleges have focused their eff orts initially on student services and improving basic skills, which are emphases at the margins of eff ort and atten- tion of most college-level faculty (Jenkins et al., 2009, p. 21).

Another factor contributing to lack of knowledge and feeling of responsi- bility on the part of faculty and middle-level administrators is the fact that performance indicators are typically measured at the institutional level alone and not at the unit level as well. As a result, faculty and middle-level admin- istrators may not be aware of the performance of their particular academic or administrative units relative to other units at their college or comparable units at other colleges (El-Khawas, 1998, p. 325; Ewell, 1994a).26

Buttressing this lack of awareness and responsibility at the unit level is the fact that performance funding typically fl ows into the general operating funds of institutions. Allocating performance funds to the general operating fund makes it diffi cult for those not directly responsible for the overall institutional budget to perceive the connection between their actions and the receipt of performance funding (Freeman, 2000, p. 93; Hall, 2000, p. 92; Lorber, 2001, pp. 84–85). For example, a department chair at Tennessee Technological University argued: “[Money from performance funding] is for the general fund and to most faculty that’s a black hole. . . . What am I going to get out of this? Nothing. . . . Do I get travel? No. Do I get a new computer? No” (as quoted in Lorber, 2001, p. 85).

Lack of faculty knowledge about and perceived responsibility for perform- ance funding makes it hard to mobilize faculty eff orts to make it eff ective. College success in meeting performance demands cannot be done only through administrative action but must ultimately involve the concerted action of the faculty, which in turn requires their knowledge and acceptance of performance funding. Moreover, lack of in-depth involvement by faculty and midlevel administrators in the design and implementation of perform- ance funding programs raises the possibility of unintended impacts that administrators and others cannot anticipate.

67Performance Funding for Higher Education

Inequality of Institutional Capacity Two studies each on Florida and Washington discussed how diff erences in institutional capacity serve are an obstacle to the eff ective implementation of performance funding (Bell, 2005, p. 135; Dougherty & Hong, 2006a, p. 73; Jenkins et al., 2009, p. 28; see also Dowd & Tong, 2007; Witham & Bensimon, 2012). In Florida, a state community college official men- tioned: “I think some of the smaller colleges have certainly the potential for that problem. [Two colleges] come to mind, where there’s just so few people in some of these programs, I know this is causing them a lot of problems trying to keep up with things” (quoted in Dougherty & Hong, 2006b).

Meanwhile, in Washington, an evaluation of the Student Achievement Initiative performance funding program for community colleges (estab- lished in 2007) revealed wide disparities in institutional capacity to collect and analyze performance data. Th e data supplied by the Washington State Board for Community and Technical Colleges to the community colleges need to be supplemented by data collected by the institutions themselves, but the colleges diff er widely in their capacity for data analysis. At several colleges, there was a shortage of institutional research (IR) staff with the skills and time to rigorously analyze college performance data. And even colleges with larger IR departments still had to collect and analyze their own data, and they diff ered widely in their capacity to do so (Jenkins et al., 2009, p. 28). An early evaluation of the Washington Student Achievement Initiative concluded:

Even at colleges with larger IR departments, college person- nel suggested that the achievement point database does not provide enough information to pinpoint areas of weakness, let alone design improvement strategies or track the progress of ongoing student retention efforts. As a result, colleges have to use their own data to do such analyses, and there is wide variation in the capacity of colleges to do so. (Jenkins et al., 2009, p. 28)

68

Institutional Resistance to and Gaming of the System Th e obstacles to the eff ective implementation of performance funding are matters not just of capacity but also of will. Eight studies on Tennessee, two each on Florida, South Carolina, and Washington, and single studies of Missouri and Ohio documented ways in which institutions try to game the performance funding program to secure high performance scores without actually improving their performance. Th is gaming takes two main forms: setting low institutional goals that can be easily attained and taking actions that produce apparently desirable performance but in ways that require mini- mal eff ort and are not in keeping with the spirit of performance funding.

Setting Low Goals

One form of gaming occurs in systems that allow institutions to set their own goals or targets. Institutions can set goals that are easily achievable rather than goals that stretch the institution. Such behavior has been documented in South Carolina (South Carolina Legislative Audit Council, 2001, p. 19). Moreover, fi ve studies of Tennessee institutions reported the same (Banta et al., 1996, p. 35; Bogue, 2002, p. 98; Freeman, 2000, pp. 89–90; Latimer, 2001, pp. 90–91; Williams, 2005, p. 94). For example, a high-level adminis- trator at a Tennessee public university stated:

If your success, your monetary success, was not connected with that [your funding], you might sit down and make …goals that are a little more adventuresome. Because we know that achieve- ment is connected to the funding, we will set down and make, I’m not saying we do it, but the temptation to do it is there—we sit down and make goals that are a little more obtainable. . . . Let’s say we know we’ll increase our minority enrollment by a certain percent. We also know that corresponds to certain equiv- alency in funding. Is it the funding that is driving the goal or the realistic expectation that we can reach out for that goal? Th at’s just human nature to me. (Quoted in Latimer, 2001, p. 91)

69Performance Funding for Higher Education

Deceptive Compliance

In some instances, institutions have complied with the requirements associ- ated with performance funding but only minimally and deceptively. Given budget concerns and the potential for performance funds to be directed toward general operations, administrators have looked for ways in which per- formance points can be increased without substantial expenditures, eff ort, or even actual improvements. Th ree studies on Tennessee, two on Florida, and one on Ohio discussed ways in which participants tried to game the system. A vice president at a Tennessee university described how programs can manip- ulate their student assessment results by postponing fi eld exams that are likely to yield lower results because “you don’t want a low score to aff ect you for fi ve years” (Lorber, 2001, p. 72). Also, a faculty leader at one Tennessee university noted that departments could secure favorable external reviews of their departments by calling on friends to perform the external audits (Baxter, Brant, & Forster, 2008, p. 58). Finally, in Ohio, a number of university branch campuses were found to have relabeled as transfer students rising jun- iors who remained within the university system, in order to fulfi ll the transfer expectation of the Performance Challenge (Dunlop-Loach, 2000, p. 92).

In sum, performance funding programs encounter many diff erent obsta- cles, and these obstacles may play an important role in explaining why those programs have not had more striking impacts on student outcomes. But even if these obstacles were removed, there is another problem that needs to be considered: the sometimes sizable unintended impacts that performance funding programs can create, especially if they are not carefully designed. We now turn to reviewing research fi ndings on this subject.

Copyright of ASHE Higher Education Report is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.