Academic Analytics in Higher Education_ Barriers to Adoption

Chapter 1: Introduction to the Study

Introduction

Successful companies such as Amazon and Netflix collect and analyze customer

data to build their operations (Chen, Shiang, & Storey, 2012). The use of business

intelligence tools, such as analytics, has helped to increase the overall growth of business

operations including customer retention, return on investments, profit structure, and

business total value (Minkara, 2010). These successes are linked to the use of analytics in

retail, financial, manufacturing, and telecommunications industries (Seng & Chen, 2010).

Higher education, similar to the business sector, has collections data concerning

its customers and general operations. Student data regarding finances, grades, study

habits, education goals, and living arrangements are collected (Vialardi et al., 2011).

Operational data, including space allocation, police and safety activities, residential

accommodations, food services and maintenance issues, are also collected and stored

(Dziuban, Moskal, Cavanagh, & Watts, 2012). However, colleges and universities are

slow to analyze these data points to help make effective decisions and data-driven

forecasts (Baepler & Murdoch, 2010; Dawson, Heathcote, & Poole, 2010).

Researchers have used data to make decisions in higher education to increase

student retention; provide transparency of financial reporting; improve management of

space, safety, and security; provide visualization of operations in true time; and supply

decision support based on facts (Bichsel, 2012). Improved student retention can lead to

increased graduation rates (Bichsel, 2012). When colleges and universities use data to

manage key performance indicators, they save money, decrease time from enrollment to

graduation, and have more transparent ways to track successes and improve forecasts

(Dziuban et al., 2012; Smith, Lange, & Huston, 2011). I designed this study to explore

the reasons why an institution of higher education has not adopted analytics to increase

efficiencies.

In this chapter, I review the background of analytics in higher education

institutions in the United States. I also examine the problem and purpose of the study.

Research questions that guided the study are considered, coupled with the theoretical

framework, scope, and the limitations of the study.

Background

Higher education institutions have traditionally operated in the United States in a

nonprofit model, depending on state and federal funding to sustain their efforts (Metcalfe,

2010; Oblinger, 2012). Recent budgetary constraints have led colleges and universities to

reconsider their operational practices and focus more on meeting budgetary obligations

(Ravishanker, 2011). The use of academic analytics may significantly assist colleges and

universities in these efforts.

Academic analytics, as defined by Barneveld, has been adopted and used by

educational institutions to help retain students and increase funding resources; however,

there are few institutions of higher education that are adopting analytics (Barneveld,

2012). Barneveld (2012) defined academic analytics as data-driven decisions used “for

operational purposes at the university or college level, but it can also be applied to

student teaching and learning issues” (p. 4). Baylor University and Purdue University,

and a few other higher education institutions, implemented academic analytics to help

student retention, recruitment, fundraising, grant administration, and analysis (Baepler &

Murdoch, 2010). However, academic analytics is still in its infancy as a field, and higher

education institutions as a whole continue to challenge its use (Barneveld, 2012).

Baylor University has used analytic tools to help build predictive modeling to

increase efficiencies in student recruitment with measurable increases in admissions over

a 1-year time period (Willis, Campbell, & Pistilli, 2013). Purdue University is using

academic analytics to help predict student success through preemptive intervention

strategies within their learning management system (Willis et al., 2013). These examples

do not reflect the actions of the majority of higher education institutions and their

academic administrators’ use of analytics to manage key performance indicators (Dawson

et al., 2010; Ravishanker, 2011). A need exists for researchers to explore factors that

impede the adoption of analytic tools that increase efficiencies in the management and

operation of higher education institutions.

Problem Statement

Knowledge management is a broad term used to label activities such as the use of

business analytics and information technologies to bolster efforts in decision sciences,

decision making, and collaborative efforts to increase the competitive advantage of an

organization (Krogh et al., 2013). Knowledge management has gained popularity in

corporate businesses during the past decade (Davenport, Harris, & Morison, 2010).

Corporate businesses have begun to use knowledge management and knowledge workers

to enable employees to join other workers across global organizations, increase

communication lines, improve efficiency and effectiveness, and boost innovation and

competiveness (Davenport et al., 2010). The corporate world has embraced knowledge

management to the extent of hiring knowledge officers and knowledge managers;

however, knowledge management has not yet permeated institutions of higher learning

(Davenport et al., 2010; Dawson, 2010).

Higher education administration has not yet taken advantage of corporate business

strategies, such as the incorporation of knowledge management as a key partner to

efficiently manage business agendas (Dziuban et al., 2012). Knowledge, used effectively,

can help higher education administrators control their bottom line. Student success and

increased retention rates, heightened grant and alumni fundraising, increased full-time

and part-time faculty effectiveness, better space allocation, and fine-tuned recruitment

strategies are examples of how better use of knowledge through data analysis can help

colleges and universities increase efficiencies (Barneveld, 2012).

Purpose of Study

The purpose of this qualitative, phenomenological study was to explore the

barriers which academic administrators perceive as preventing an institution of higher

education from adopting analytic tools that would enable the analysis and use of data for

decisions, planning, and managing operations.

Research Questions

In this study, I sought to explore barriers related to the adoption of knowledge

management, specifically academic analytic tools, in higher education. The general

research question that guided this study was the following: What factors impede the

implementation of academic analytic tools in a higher education setting? Subsequent

questions included

1. Are academic administrators aware of how academic analytics could help

manage key performance indicators?

2. What types of discrete databases do academic administrators currently use to

help the management of their perspective departments?

3. How can knowledge management tools enhance the efficiency of a higher

education institution?

4. Does the climate of a secondary education institution hinder the adoption and

use of analytic tools, or are there funding/investment issues?

5. Would college administrators use academic analytics to help increase student

success and other managerial tasks?

Theoretical Framework

Institutions of higher education have become more like corporations due to

changes in their traditional sources of state or federal funding, declining grant and

research funding, and other decreases in investments or donations and financial gifts

(Metcalfe, 2010; Stocker, 2012). Colleges and universities must seek funding through

creative and nontraditional sources in the marketplace, thus bringing them closer to

operating like businesses in the private sector. The theory of academic capitalism is used

to address the ways in which institutions of higher education are becoming more like

business corporations. The concepts that provide the crucial underpinnings to this theory

include success, performance, competitiveness, and accountability (Park, 2011; Slaughter

& Cantwell, 2011). As higher education institutions become more like big business, such

institutions become more resource-conscious and market-focused and recognize a need

for transparency. Academic analytics are potential tools that can be used to measure the

concepts of success, performance, competitiveness, and accountability. Businesses use

knowledge management and tools such as business analytics and data mining to create a

competitive advantage. Role players within higher education setting could use these tools

for colleges and universities to control for tightening budgets and decreased funding

sources (Stocker, 2012). I discuss the theory of academic capitalism in detail in Chapter

Nature of Study

I designed this qualitative, phenomenological study to understand and explore the

experiences of individual academic managers in a higher education setting, their

experience in using or not using analytics, the meaning behind their perceptions of their

use or nonuse of analytic tools, and perceived barriers to the adoption of analytics. The

main manuscripts examined in determining the design for this study included Creswell

(2012, 2013), Merriam (2009), and Englander (2012). I designed the study to gather

personal data from the interview process to explore barriers that prevent colleges and

universities from adopting analytic tools to support management efficiencies. According

to Creswell, Merriam, and Englander, qualitative research methods allow for an interview

data collection process and the need for an intensive study. The mission of qualitative

research is to (a) explore how people understand their experiences, (b) discover how

people create their worlds, (c) understand how people make sense of their experiences,

and (d) describe how people understand their experience (Merriam, 2009).

I considered the case study and phenomenological research traditions for this

study (Creswell, 2012). Case study concerns an issue explored “through one or more

cases within a bounded system” (Creswell, 2012, p. 73). Simon (2011) reported that

researchers use case study research when the inquirer establishes a problem and uses

questions such as why and how. A case study was considered for this research because I

wished to explore a bounded system in which several individuals would be interviewed

and the research questions were focused on why and how. I deemed the choice of a case

study inappropriate due to the data collection sustained in such a design. Data collection

in a case study draws on multiple sources to include observations, documents, archival

records, physical objects, and audiovisual materials (Creswell, 2012). The primary data

collection for this study was rooted in in-depth, open-ended interviews.

The use of phenomenology was chosen because of the emphasis on open-ended

interviews as the primary data collection, the general inquiry into the meaning and

significance of the experiences from the participants, and the phenomenological approach

Cilesiz (2011) established in research of the use of technologies in educational settings.

Singleton and Straits (2009) and Cooper (2010) also posited that the social science

researcher’s purpose is to gain an understanding and to capture the essence about how

people think and feel and how they interact during phenomena. Additionally, Simon

(2011) stated that “phenomenological research is people’s experience in regard to a

phenomenon and how they interpret their experiences” (p. 105).

Definitions

Academic analytics: Academic analytics refers to “analytics used to help run the

business of the higher education institution” (Oblinger, 2012, p. 10). In this study,

academic analytics referred to the process by which education and academic personnel

use advanced applications and statistical techniques to analyze data sets (Baepler &

Murdoch, 2010).

Academic managers: Persons whose task it is to handle crises, complexities, and

to instill a unified culture within the organization (Din, Khan, & Murtaza, 2011). In this

study, academic managers were the managers at the college who had the task to increase

student engagement, align academic policy with curriculum, conduct faculty

observations, and increase student retention and student graduation rates.

Barriers to IT adoption: Barriers to IT adoption are those factors that inhibit

organizations or individuals in the implementation or strategic use of information

technology to increase competitive advantage and profitability (Davenport et al., 2010).

In this study, barriers to IT adoption included those factors that hinder academic

administrators in their adoption and use of academic analytic tools. Such barriers may

include cost, perceived usefulness, knowledge of available tools, training, and other

institutional issues.

Dashboards: A dashboard is the collection of disparate information systems and

huge data sets, gathered and displayed in an uncomplicated manner, which provide

graphic depictions of real-time insight in manager’s performance. Dashboards can often

give immediate snapshots of detailed information, which might have taken time-

consuming measures and inefficient time to produce manually (Stocker, 2012). In this

study, dashboards were used by educators to view key performance indicators (KPI)

visually and in real-time. The dashboards were customized dependent upon measured

indicators (KPIs) for each academic manager.

Key performance indicators: Key performance indicators are assessments and

indicators by which a University measures its efficiencies, performance, and success

(Sukboonyasatit, Thanapaisarn, & Manmar, 2011). In this study, the key performance

indicators that indicated measurement of academic management effectiveness included

student retention, faculty training and observation, the management full time equivalent

budgetary operations, curriculum reviews, and policy compliance.

Knowledge management: Knowledge management is the use of strategies to

manage corporate knowledge, insights, experiences, and the incorporation of those

experiences to add value to the corporation (Davenport et al., 2010). For this study,

knowledge management referred to the use of the results from data analysis of multiple

factors in the higher education including, but not limited to, admissions, retention,

financial services. Specific examples included in this study, is the use of data to manage

academic key performance indicators.

Shadow systems: Shadow systems are information technology programs,

applications, or systems that operate on the outside of an organization (Behrens, 2009). In

this study, shadow systems referred to information data collections not housed in the

official college database system. Examples included departmental and siloed

spreadsheets, FileMaker Pro databases, MS SQL, and other forms of information

technology that existed outside of the official college information system (Blanton,

2012).

Assumptions

For this study, the following assumptions were made:

1. Business analytic tools are valid and useful methods to increase the

efficiencies of businesses for success, increased financial viability, and

improvement.

2. Business analytic tools could also benefit institutions of higher education to

increase productivity measures, similar to the business corporate world.

3. Institutions of higher education have not yet adopted business analytic tools.

4. For this study, I assumed that higher education institutions have not yet

adopted business analytic tools because of existing barriers.

Scope and Delimitations

Researchers have indicated that the use of analytics to drive decisions improves

efficiencies in higher education institutions. I designed this study to explore the reasons

why colleges and universities do not adopt proven technologies, such as the use of

analyzing data, in order to improve performance. This study covered a large,

multicampus community college with a student population of approximately 85,000 full-

and part-time, campus-based, and on-line student body. The college employs

approximately 3,500 faculty and staff. The primary focus of the study was in

interviewing academic managers whose key performance indicators include student

retention, faculty training and observation, the management of full-time equivalent

budgetary operations, curriculum reviews, and policy compliance to ascertain why they

did not employ analytics to help them better control their key performance indicators.

These academic managers, who were responsible for the specified key performance

indicators, numbered 25 individuals. These academic managers worked in academic

divisions across all six campuses. The established period for the data collection occurred

in the Fall 2013 academic school year.

I excluded data collection from other departments outside the academic

departments within the college from this study. Primary examples of excluded

departments included the office of institutional reporting (this department collects and

cleans data for the college), the campus police department, student financial aid

department, the admissions department, business office operations, maintenance and

facilities departments, IT services and operations, human resources department, and

training departments. Most view these departments as “support” services for the main

academic mission of the college or university and, thus, do not directly affect academic

administrators’ goals of improving performance indicators. The excluded departments

would benefit from the use of analytics, but improving academic performance indicators

and the use of analytics, or barriers to the use, by academic administrators to achieve

those goals was the focus of this study.

Limitations

For this study, the following limitations were recognized:

1. I used a small sample and single setting for this study. Only 25 managers had

academic key performance indicators, as listed earlier.

2. There may have been additional administrators who were aware of, or were

using analytic tools, that I did not interview.

3. Due to the need for a criterion sample and the time available with academic

administrators, I used interviews as the primary method of gathering

information.

To control for these limitations, I conducted member checks of transcriptions and

peer review of results.

Significance

Knowledge management tools such as analytics have been used to successfully

help businesses use their intellectual capital more effectively, thus making a positive

impact on the bottom line (Davenport et al., 2010). Due to changing economies and

funding constraints, institutions of higher education need to develop strategies to meet

their fiscal responsibilities (Metcalfe, 2010). The adoption of academic analytics may be

a way in which colleges can become more efficient and increase the value of their

services. This study may help higher education academic administrators realize the

factors that impede adoption of analytics and ways in which these key tools can help

sustain their bottom line, increase efficiency, and promote graduation and placement

rates.

Summary

Institutions of higher education have large data collections that could assist these

organizations to operate more efficiently. Student data such as financial aid, grades, and

housing accommodations, and operational data including space allocation, food services,

and maintenance issues, are also collected. Colleges and universities are slow to analyze

these data points to help make effective decisions and data-driven forecasts to improve

their operations.

The use of data to make decisions in higher education increases student retention;

provides transparency of financial reporting; improves management of space, safety, and

security; provides visualization of operations in true-time; and supplies decision support

based on facts (Bichsel, 2012). When colleges and universities use data to manage key

performance indicators, they save money, decrease time from enrollment to graduation,

and they have more transparent ways to track successes and improve forecasts (Dziuban

et al., 2012; Smith et al., 2011). Despite the positive outcomes that analytic tools may

bring, academic administrators have not yet adopted these tools. I designed this study to

explore the barriers behind why academic administrators in a community college have

not adopted analytics in order to increase efficiencies.

In the next chapter, I provide an overview of business analytic tools, the use of

such tools in the corporate world, what is known about the current use in higher

educational settings, and the barriers to adoption that have been noted in other industries.

The following chapter, Chapter 3, describes how I conducted this study. In Chapter 4, I

present the data that were collected, and Chapter 5 contains a synopsis of the study,

interpretation of the findings, limitations of the study, recommendations, and implications

of the study.

Chapter 2: Literature Review

Introduction

The use of analytics to help drive decisions and meet key performance indicators

in higher education institutions has been proven to be effective (Barneveld, 2012).

However, colleges and universities continue to be slow to adopt academic analytics, even

though business industries have adopted and seen the benefit of its use (Dawson, 2010).

The purpose of this study was to gain an understanding of the barriers that impede the

implementation and use of knowledge management, as described in this literature review

as academic analytics, in a community college setting. The limited use of academic

analytics in selected colleges has had a positive effect on key working indicators, such as

reduction of student attrition, increased availability to track student registration and

course selection, and more effective use of space (Dziuban et al., 2012). However, the

use of analytics in the day-to-day operations of higher education institutions continues to

remain minimal (Bichsel, 2012).

This literature review begins with an overview of analytics and how corporations

use analytics in corporations to control for customer loyalty, customer fulfillment, and

approval and to track return on investments (Minkara, 2012). Technologies used in

analytics, and the value such technologies have in the business world, are discussed. I

then review the use of analytics in higher education institutions, with specific colleges

and their employment of analytic tools in operation. Further, I examine the value of

analytics in higher education, in addition to barriers that could cause universities and

colleges to not adopt analytics for wide-scale use in the management of operations

(Bichsel, 2012; Ravishanker, 2011). Finally, because institutions of higher education

have been slow to adopt analytics as an innovation that may improve performance and

there are limited studies in this area, I consider an examination of barriers to innovation

adoption that may provide areas that also impede adoption at institutions of higher

education.

Literature Search Strategy

A key word search using the following terms was conducted: academic analytics,

education analytics, student selection, academic data mining, student retention and data

mining, education and data mining, data mining and education management, business

intelligence and education, data mining and colleges, analytic tools definition, analytic

tools, analytic tools and business adoption, business analytic tools and adoption,

business analytic tools, data analytics, action analytics, barriers to IT adoption, barriers

to adoption and analytics, innovation adoption, and barriers to innovation adoption. The

search was done using Gartner, Business and Management Sage Database, Business

Source Complete, Google Scholar, Emerald, Science Direct, ProQuest, ERIC Education

Database, Education Research Complete, Education Full Text (H. W. Wilson),

Educational Administration Abstracts, Business Abstracts with Full Text (H. W. Wilson),

Business Source Complete, and Psychological and Behavioral Sciences Collection. The

search yielded 48,084 publications. The highest returning terms were from Google

Scholar data analytics (18, 400) and barriers to IT adoption and analytics (16,600).

Other high yielding terms included barriers to IT adoption (1,884) from ProQuest

academic analytics (337) from Sage, and data mining and education (863) from Emerald

Management.

Inclusion criteria for relevant articles were the following: (a) publications that

addressed analytic tools; (b) publications that addressed the use of analytic tools in

business; (c) publications examining the use of analytic tools in higher education

institutions; (d) publications addressing the new challenges higher education is facing; (e)

publications addressing how the use of analytics has helped higher education institutions;

(f) publications reporting barriers to IT adoption in businesses; and (g) articles discussing

barriers to IT adoption in higher education institutions, innovation adoption, and barriers

to innovation adoption.

Primarily, I rejected 47,853 articles by a review of the title because it did not meet

the inclusion criteria. I rejected an additional 156 after a review of the abstract. Of the 75

that met the inclusion criteria, 32 were excluded due to their focus on modeling and

structure functions, eight more were excluded due to their focus on singular database

role, and six were excluded because their use of analytics was concentrated solely on

research methodology.

Theoretical Foundation

The theory of academic capitalism was used to provide the theoretical foundation

for this study (Slaughter & Cantwell, 2011). Academic capitalism is the theory that

colleges and universities are changing and becoming more like corporate entities

(Walker, 2009). Slaughter and Cantwell (2011) described the links and resource

dependency that higher education institutions are sharing with industry and how these

links are allowing universities to compete in the globalization of a new economy. Park

(2011) described academic capitalism in market terms. Higher education institutions,

because of increasing scarcity of government funding, must obtain subsidies elsewhere.

Colleges must search and compete for external funding sources through endowment

monies, external grants, industry collaborations, contracts, and with the increase of

tuition and fees. Some universities have formed quasi-corporations through the creation

of university hospitals. A university does not technically own these university hospitals;

however, the affiliated university has the opportunity to garner resources, such as

laboratories, clinical space, and research, and has further access to additional external

grants and endowment funds (Park, 2011).

It has been shown that colleges and universities are increasingly interacting with

the business commercial sector. Park (2011) argued that institutions of higher education

interact in the economy through initiatives and continued development. Park claimed that

the Internet originated in a university, a tool that has changed the landscape of

economies, not only here in the United States, but globally. Colleges have also engaged

in the globalization of education using extensive online, distance, study abroad programs,

in some occasions, the opening of entire campuses in foreign countries (Park, 2011).

Universities show further examples of their movement towards the business sector in the

growth of university-owned patents. Patents held by universities more than tripled over

the past decade (Park, 2011). Additionally, universities have begun to acquire equity in

companies in which technologies, developed by the particular university, are licensed. As

a result, technology licensing offices, community outreach and economic development

offices, and fundraising departments have developed on campuses (Park, 2011).

Colleges and universities, in moving closer to the market place and competition,

are being required to become more transparent to measure outcomes and to demonstrate

success (Blanton, 2012; Grajeck, 2011; Ice et al., 2012; Metcalfe, 2010; Peterson, 2012;

Stocker, 2012). Metcalfe (2010) used the theory of academic capitalism as the foundation

for an analysis of the globalization of higher education and the use of information

technology to manage key performance indicators. Stiles (2012) entailed the key factors

affecting higher education, one of which was that colleges and universities need to

increase their economic competitiveness, accountability, and institutional business

decisions. Stiles stated, “Under the right circumstances, decision-making can be

enhanced by the tools and techniques of analytics. Large data sets, analytics engines, and

new data-visualization techniques have considerable potential to enhance both student

learning and institutional business intelligence” (p. 3). The use of analytics, as Stiles

indicated, can help college administrators make better decisions that may facilitate

decreased institutional costs and increase student performance.

Proponents of the theory of academic capitalism addressed the ways in which

institutions of higher education are becoming more like business corporations. The

concepts that provide the underpinnings to this theory include success, performance,

competitiveness, and accountability (Park, 2011; Slaughter & Cantwell, 2011).

Researchers have demonstrated that, with the use of analytic tools borrowed from

corporate business, colleges and universities may have success in meeting and exceeding

key performance indicators in areas such as student retention, student progress, budget

and planning, faculty training, and course scheduling (Anderson & Russell, 2012; Fritz,

2011; Macfadyen & Dawson, 2012; Obinger, 2012; Wishon & Rome, 2012). Businesses

use knowledge management and tools such as business analytics and data mining to

create a competitive advantage to achieve success, improve performance, and increase

economic competitiveness and accountability. Institutions of higher education are

becoming more like business corporations and must use all tools available to address key

performance indicators.

Key Concepts in Analytics

Businesses have collected unprecedented amounts of data regarding customers’

purchasing habits, decisions, values, and experiences (Fahey, 2009; Minkara, 2012).

Businesses have been able to store this mostly structured data in assorted databases and

various systems (Fahey, 2009). Recently, business organizations have begun to apply

these data to transform operations (Davenport et al., 2010). Data analysis entails the use

of data to enhance operations, and the tools used to perform this analysis include such

technologies as interactive visualization, dashboards, data mining, and predictive

modeling (Chen et al., 2012; Davenport et al., 2010).

Analysis of data, or business analytics, entails the use of tools such as statistical

and quantitative techniques, methodologies, applications and systems for industries to

make better decisions regarding market demands and customer expectations (Chen et al.,

2012; Davenport et al., 2010; Fahey, 2009). Researchers could also use these new

technologies, or analytic tools, to measure key performance indicators, return on

investments, and other business indicators that drive growth (Minkara, 2012). Business

leaders are engaging analytics to support strategic planning and progressive thinking to

transform the way their enterprise is operated (Davenport et al., 2010).

Business Use of Analytic Tools

A 2012 study conducted by the Aberdeen Group, found that businesses using

analytics achieved a greater growth rate (17.3%) than businesses not engaging in

analytics in their day-to-day operations (9.1%; Minkara, 2012). Minkara (2012) described

areas in which businesses excel in using analytics as (a) customer retention, (b) customer

value, (c) customer satisfaction, and (d) return on investments. Within these vital areas,

industries using analytics had positive year over year growth. Through analytics, it was

possible to provide customers valid customer-centric content, a single source of data for

key stakeholders, and the ability to track and make use of customer experience statistics.

Many businesses use analytics in e-commerce and marketing fields to collect and

analyze customer behavior patterns and opinions (Chen et al., 2012, Davenport et al.,

2010). Vendors such as Amazon use data analytics to create specific customer content

driven recommender systems based on customer preferences (Chen et al., 2012).

Business analysts analyze and collect data from social media outlets in order for

businesses to better understand the opinions and behaviors of customers, and target their

audience in a much more efficient way (Chen et al., 2012).

The United States Government, State Governments, and politicians are beginning

to use business analytics for blogs, research, and campaign advertising. The

aforementioned officials can use data mining to help support political discussions and to

help collect donations. Analytics support governmental accountability and transparency;

broader platforms including blogs, wikis, and other social media outlets track and

publicize programs (Chen et al., 2012).

Researchers within the fields of science and technology increasingly adopt big

data projects in order to help researchers and students push knowledge boundaries and

explore new developments through simulations and predictive modeling. Scientists in

astronomy and physics are amassing several hundred gigabytes of data each day that they

analyze using business analytics (Chen et al., 2012). This information will lead to

discoveries much faster and on a larger scale than the science community has previously

been able to deliver.

Business analytics contributes to health sciences and public health as well. As the

health services field moves to patient-centered, or customer-centered, medicine, business

analytics help in the area of decision sciences. Electronic health records play a large role

in preventative, evidenced-based practices, and analytics power these systems. New

modeling and process learning techniques are increasingly prevalent in the health

sciences (Chen et al., 2012).

Individuals within public security sectors use business analytics to bolster

counter-terrorism activities. The advancement of security informatics aids in cyberspace

intelligence, emergency preparedness, and international data exchanges. Intelligence

agencies worldwide are gathering statistics that cover the range from criminal threats,

terrorism activities, and organizational cyber security incidences. Business analytics uses

applications and platforms that enable security personnel to evaluate, analyze and in

many cases, prevent attacks (Chen et al., 2012).

Businesses use analytics applications for customer retention programs and

tracking, stock market prediction analysis, inventory and product analysis, and

advertising. Industries that have bought into business analytics include retail franchises,

financial enterprises, manufacturing, and telecommunications trades (Seng & Chen,

2010). Direct marketing, product to consumer analysis, product-rating predictions, yield

ratings and analysis, and fraud detection and collections are only a few of the widely used

applications that businesses employ analytic tools.

Analytics in Higher Education

Higher education institutions in America are among the casualties of

globalization, economic uncertainties, public funding shortfalls and drastic cutbacks, and

heightened accountability and transparency regulations (Picciano, 2012). Leadership in

these organizations needs to respond with financial plans that will control for these

challenges and set a path forward that will allow for stability and growth (Smith et al.,

2011). A solution that colleges have increased interest in is that of using technology to

drive change (Dziuban et al., 2012).

One technology that higher education institutions have adopted to control their

business is enterprise resource planning technologies. These systems collect transactions

in the areas of human resources, finances, and budgetary functions and deposit the

information in relational databases (Ravishanker, 2011). These systems have helped

colleges collect and store massive amounts of essential data.

Another technology that has permeated college existence is the expanding

platforms for course delivery (Picciano, 2012; Sinha, Arora, & Mishra, 2012). Blended

courses, a combination of both online and on ground instruction, is growing rapidly as

colleges make use of technology and as faculty become more comfortable with this

mixed design. Due to the growth and use of the Internet, millions of college students

enroll in online courses and fully online programs (Picciano, 2012). Colleges have

adopted learning management systems to control and distribute learning for students;

these systems have created a platform that enables students to access an education

environment virtually (Siemens & Long, 2011).

Both of these technologies, along with others that are outside of the scope of this

paper, collect massive amounts of data relating to the business operations of colleges.

The next step for colleges is to follow companies such as Netflix and Amazon, and make

use of their massive amounts of data to inform decisions. Business analysts have used

consumer data to help predict costumer purchasing habits, and, like Amazon, have built

recommender machines to recommend products to customers based on past purchases

and those of popular demand (Dziuban et al., 2012). The use of data is now common

practice in business; however, the use of data to drive decisions in higher education is

still in its early stages (Baepler & Murdoch, 2010; Dawson et al., 2010).

The analysis of large amounts of data for the use of decision-making in colleges

or universities for operational purposes is termed as academic analytics (Baepler &

Murdoch, 2010). Barneveld et al. (2012) suggested a conceptual framework that placed

academic analytics in an open infrastructure that allows for predictive and action

analytics to help inform managemen

visualization of academic analytics and its subcategories

analytics, action analytics, and decision sciences

Business Analytics

Actionable Intelligence (Action Analytics)

Figure 1.

Conceptual framework of analytics.

Fahey (2009

) recommended

and action analytics to guide decisions t

Figure 2.

That process includes the capturing of data, the reporting of the data,

predictions made from the data, an action taken, and then refi

Figure 2. Analytics p

rocess.

Colleges that have adopted enterprise resource planning systems, that use various

databases, and have implemented a learning management system for an online course

deliver platform, are all collecting and capturing

not connect with each other, nor do they have the flexibility for growth, or do they have

tools that can use prediction models to help fuel information that leaders can then act

upon (Ravishanker, 2011).

analytics to help inform managemen

t and faculty decision-making.

Figure

visualization of academic analytics and its subcategories

in learning analytics, predictive

analytics, action analytics, and decision sciences

(

decision making) analytics

Analytics

Business Analytics

Aca

demic Analytics

Learning Analytics

Predictive Analytics

Actionable Intelligence (Action Analytics)

Decision Making

Conceptual framework of analytics.

) recommended

, and Clow (2012) supported

the use of predicative

and action analytics to guide decisions t

hrough the manner of a

process a

That process includes the capturing of data, the reporting of the data,

predictions made from the data, an action taken, and then refi

nement.

rocess.

Colleges that have adopted enterprise resource planning systems, that use various

databases, and have implemented a learning management system for an online course

deliver platform, are all collecting and capturing

data.

The issue is that these systems do

not connect with each other, nor do they have the flexibility for growth, or do they have

tools that can use prediction models to help fuel information that leaders can then act

upon (Ravishanker, 2011).

Some colleges

have taken the next step and have adopted the

Figure

1 displays a

in learning analytics, predictive

decision making) analytics

demic Analytics

the use of predicative

process a

s described in

That process includes the capturing of data, the reporting of the data,

Colleges that have adopted enterprise resource planning systems, that use various

databases, and have implemented a learning management system for an online course

The issue is that these systems do

not connect with each other, nor do they have the flexibility for growth, or do they have

tools that can use prediction models to help fuel information that leaders can then act

have taken the next step and have adopted the

use of academic analytics; they apply technology to data to better manage their key

performance indicators (Goldstein, 2005).

Use of Historical References

This study references the 2005 survey conducted by Goldstein. This is a

benchmark survey in academic analytics. This survey, described later in this chapter,

established that of the colleges surveyed, most used academic analytics primarily for data

collection and retrieval. Colleges were not using analytics for strategic planning, decision

making, or in the management of key performance indicators.

Bichsel (2012) conducted a survey to indicate the status of analytics in higher

education institutions. Bischel surveyed 339 colleges and universities. Bichsel found that

from the 2005 Goldstein survey seven years prior, not much change had happened;

colleges and universities were collecting a rather large amount of institutional data, but

the data were not being analyzed to make decisions or being used by managers to better

control key performance indicators (Bichsel, 2012).

In this study, I used the Goldstein survey to establish a benchmark in academic

analytics. The 2005 survey provided a measure consistently referenced by other studies

and publications to establish a reference point; that in a period of seven years very little

has happened in the academic analytics field.

In this study, I used two interview protocol designs from studies conducted in

2008. These two studies were published in (a) the International Journal of Training and

Development (Ali & Magalhaes, 2008), and (b) the Journal of Decision Sciences

(Venkatesh & Bala, 2008). Venkatesh and Bala used the interview questions from the Ali

and Magalhaes study published in the International Journal of Training and Development

as a base of comparison to the interview questions. Studies of adoption in academic

settings heavily cite Ali and Magalhaes’ study. Al-alak and Alnawas (2011) cited Ali and

Magalhaes’ study. Fenio and Bright (2010) also cited the Ali and Magalhaes 2008 study

in a case study they conducted covering academics and adoption of technologies. Ali and

Magalhaes’ (2008) study proved invaluable in this current study covering academics and

the adoption of analytic technologies.

I modified questions from Venkatesh and Bala’s (2008) Technology Acceptance

Model 3 (TAM3) study to meet the needs of this study. Numerous researchers and in

excess of 800 studies cited this 2008 TAM3 study, and the model itself is used

consistently for studies in technology adoptions and user perceptions. Behrend, Wiebe,

London, and Johnson (2011) and Munguatosha, Muyinda, and Lubega (2011) used

Venkatesh and Bala’s (2008) TAM3 model in their study. The use of Venkatesh and

Bala’s TAM3 model was integral to the interview protocol in this study.

Use of Academic Analytics

Goldstein, in the employ of the Educause Center for Applied Research, described

five stages of the use of analytics to manage key operational areas in seven typical

college/university departments (Goldstein, 2005). The first and most-used stage of

analytics is that of transactional data and enterprise resource planning. Ravishanker

(2011) described this first stage as a system that collects data in one system for the use of

data retrieval. Goldstein (2005) explained stage two as that of analysis and monitoring of

operational performance. The following stages enact scenario building, predictive

modeling, and finally, a system that prompts warning signals and notifications

proactively. The outcome of Goldstein’s work concluded that most college departments

that were surveyed (n = 380) used academic analytics primarily in the Stage One area of

data collection and retrieval (Goldstein, 2005). Table 1 indicates the college departments

that were using academic analytics, the stages of development and usage, and the

percentage each department was in during the survey collection period.

Table 1

Survey Results of Academic Analytic Usage

Use AF BP BAP IR HR RA AA

Stage 1: Extraction

and reporting 56.9% 68.4% 49.6 48.8% 62.2% 45% 52.8%

Stage 2: Analysis

and monitoring of

operational

performance

11.0% 17.0% 19.6% 28.4% 7.8% 10.3% 18.2%

Stage 3: “What-if”

decision support 2.3% 1.9% 13.5% 4.1% 0.6% 0.9% 4.7%

Stage 4: Predictive

modeling 3.1% 3.0% 9.6% 11.6% 1.1% 1.7% 5.2%

Stage 5: Automatic

triggers of business

(alerts) 3.7% 2.5% 0.6% 7.1% 1.9% 1.1% 2.2%

Not active users 22.9% 7.1% 7.2% 0.0% 26.4% 41.0% 16.9%

Total 100.0% 100.0% 100.0% 99.9% 100.0% 100.0% 100.0%

Notes. Codes: AF = Advancement/Fundraising, BP = Business and Planning, BAP = Budget and

Planning, IR = Institutional Research, HR = Human Resources, RA = Research Administration,

AA = Academic Affairs.

Goldstein, P. (2005). Academic analytics: The uses of management information and technology

in higher education. EDUCAUSE Center for Applied Research, 1–12.

As demonstrated by Goldstein’s survey, there are a few colleges and universities

using analytic tools. One example of how a college is using academic analytics is Purdue

University (Pistilli & Arnold, 2010; Pistilli, Arnold, & Bethune, 2012). Purdue developed

an early warning alert system to help students in the coursework. This system is

behaviorally modeled; the system tracks how students use the on-line learning

management system, how much time they spend reading the required articles, viewing

the videos, reading the discussion boards, and engaging with other students and their

faculty (Pistilli & Arnold, 2010; Pistilli et al., 2012). The system tracks the effort the

student puts forth in the course. Whether a student takes the time to ask for help, contact

a tutor, or arrange an appointment with their instructor, is another indication of the

student’s effort. The first time a student’s quizzes fall below the prescribed threshold,

Purdue sends an e-mail to the student, automatically generated asking the student to

review resource materials. Purdue also alerts the student’s advisor and then calls the

student to encourage tutoring and discuss an improvement plan (Pistilli & Arnold, 2010).

Students at Purdue also have individual “dashboards” where they can track their own data

and compare their performance against other students in the same course. This allows

students to visualize and compare their efforts; they can see the resources used, time

spent in reviewing sessions, assignments submitted by their classmates. Pistilli and

Arnold tested two sets of students in the same course for two semesters. One set of

students used the analytic tools (Purdue has named the system “Signals”), and the other

set of students did not use the system. End of semester grades and help-seeking behaviors

increased in the students using the system. There were fewer Cs, Ds, and Fs from the

students using the system compared to those not using the system (Pistilli & Arnold,

2010; Pistilli et al., 2012).

Another use of analytics is the development of recommender systems. Vialardi et

al. (2011) studied the use of a recommender system for student use at the University of

Lima, Peru. The University found that students were taking courses based on inaccurate

information or a lack of knowledge about the courses. This method led students to take

too many courses, or courses that they were not prepared to take. The university created a

recommender system, with the use of data mining, to assist students in choosing courses.

The recommender system reviews students’ demographic information, prior grades

earned, the number of courses taken each semester, average grade, and the cumulative

grade the student has obtained (Vialardi et al., 2011). Additionally, the system allows for

the difficulty of the course, and reserves times and places within the courses. The

university then used this information to recommend courses in which the student has a

great potential for success.

Pace University is another university that has been experimenting with academic

analytics. Pace University had been collecting massive amounts of data on perspective

students, but was unable to utilize all of the information effectively. The leadership took

steps to allow for development of an analytics powered by Microsoft Business

Intelligence. The University found that a common language for data was lacking, many

different departments were using different definitions for similar data. Creating a data

dictionary was the first step in moving to a common analytics system. Pace purchased the

student module as the first module for implementation in order to help control for student

retention. Because of using this system, Pace started to see a more complete picture of

student data. They began to discover new data sources, which they could then combine

with other data and began to see new perspectives into student life and student

engagement (Ravishanker, 2011).

The University of Central Florida uses academic analytics to track faculty

development scheduling and teacher credentials, to follow productivity in student

registrations, course sections, student credit hours, and other operational projects

(Dziuban et al., 2012). Data that were stored in many different databases across several

various departments could be integrated and effectively used. Managers had the

flexibility to run reports concerning headcounts, student demographics, faculty grant

development progress, enrollment metrics, and teaching summaries. College

administrators had dashboards that visually tracked their key performance indicators; this

was in real time and allowed managers to see patterns, monitor growth, and efficiently

solve challenges before they leave a negative impact on the College (Ravishanker, 2011).

Academic analytics can be used to predict at-risk students. Smith et al. (2011)

studied the use of academic analytics in a community college to predict at-risk online

students. The college needed a way to predict at-risk students before they began showing

signs of failure, and a way in which to respond to the students through personalized

contacts. The data set was comprised of on-line students who interacted with the college

through a course management system; the students had no face-to-face interactions. The

sample size was n = 539 students. The researchers analyzed variables such as login

frequency, course management engagement, and points earned for assignments

submitted. Smith et al. used the Pearson r correlation coefficients to establish and

measure correlations. The results indicated a significant correlation (p < .05) between

final course outcome and the variables. The college was able to intervene prior to failure

with the use of analytics to predict at-risk students.

Another case evaluated by Forsythe, Chacon, Spicer, and Valbuena (2012)

established the use of analytics helped to address problems such as student recruitment

and retention. The University of Maryland Eastern Shore (UMES) began using an

analytic dashboard that provided real-time data and targeted for key performance

indicators specific to the admissions department and the retention specialists. UMES

created and tailored dashboards to match the key performance indicators of the roles of

end users such as administrators, faculty advisors, and support staff. UMES designed the

dashboards, created by analytic tools powered by the wealth of institutional data, in a

convenient format that allowed for alerts (Forsythe et al., 2012).

UMES, for example, created a dashboard to assist students and staff in the

financial and registration process used at the beginning of each semester. The dashboard

tracked students as they chose classes and then applied and used financial aid to pay for

their courses. Staff members, with the use of data pushed to their individual dashboard,

could monitor indicators daily to make sure students moved toward overall progress

(Forsythe et al., 2012).

Using academic analytics, UMES has seen growths in key missions of the

university. One of the important experiences that UMES has learned from the

implementation of analytics was that “analytic tool sets currently provide unprecedented

insight into data sets-allows users to disaggregate complex collections in real time”

(Forsythe et al., 2012, p. 6). The ability for academic and staff personnel to be able

manage, cut, slice, and drill down data at their desktops gave them huge opportunities to

proactively meet targets and key performance indicators, thus engaging in the total

mission of the college to help keep students retained and improve graduation rates

(Forsythe et al., 2012).

Successes measured during the first year UMES used analytics resulted in an

increase of student enrollment by 150%. The college was also able to recognize course

level structures and pinpoint areas of increased efficiencies in the management of courses

and adjunct faculty hires. Additionally, retention rates for students increased during the

third and fourth year terms. The college will eventually see a rise in graduation rates due

to the retention rates of the third and fourth year students (Forsythe et al., 2012).

In a further example of successful use of academic analytics, Philadelphia

University shared its challenges and goals when new leadership of the university set on a

path to explore the universities operations. The university wished to scrutinize its

operations by “examining trends, patterns and tendencies within the critical quality of

data” that had been gathered after 10 years of using a resource planning system (Cepuli,

Radhakrishanan, & Widder, 2012, p. 1). The university was certain that they had enough

data collected to provide historical support of past patterns and behaviors. However, there

was a lack of easy-to-use tools for leadership to access and an absence of an analytic

environment in which to analyze and predict trends (Cepuli et al., 2012).

The university took steps to collect the historical data. They asked the academic

deans to provide data regarding growth rates of programs and expansion of faculties. It

quickly became apparent that the data were scattered in different siloed departments, and,

that the data were mostly paper-driven, that information was not electronic. The

university also discovered that much of the data that they were seeking, enrollment,

registration, course scheduling and course frequencies, had not been made available to

the academic deans in any form (Cepuli et al., 2012).

Philadelphia University set a new and pressing goal. The university leadership

knew of the importance of newly established transparency objectives within the

university environment and that all parts of the university needed to operate from an

informed centralized data source. The first step in the process to align university data in

one central area, and to build usable dashboards for analytical trend spotting, was to

assess the Universities readiness for analytics, and to assess key performance indicators

in each area of operations (Cepuli et al., 2012).

The university took two years to develop and create dashboards for the use in

front-line departments. End users in these departments saw the ability to make better

decisions in course development and frequencies, resource utilization, consolidation of

enrollments, and space and time reallocations. The leadership of the university was able

to see a return on investment in the use of analytics, and a greater capacity to build a

culture of transparency throughout the University (Cepuli et al., 2012).

As noted, an increased need for college and university transparency is changing

the way higher education institutions handle their repository of data. At Portland State

University, a situation arose in which increasing costs and decreasing state and federal

funding was forcing the university to reevaluate how the university was using resources,

budget models, and its student success rates. They were unable to answer key questions

regarding these items because of the siloeing and inappropriate connection of legacy

reporting and data sets (Blanton, 2012).

Further investigation revealed that faculty and staff had created local “shadow

systems,” or different and numerous spreadsheets, databases, and word documents. The

primary use of these disconnected systems caused redundancy, errors, and misaligned

information (Blanton, 2012). Portland State University’s reporting environment “was a

disconnected collection of data and reports from multiple disparate sources that were

manipulated using a wide variety of tools” (Blanton, 2012, p. 2).

To move forward, the university had to plan to extract all the data from the

disconnected systems, devise a plan to organize the data, and begin to analyze the

coherent and grouped data. With this in mind, Portland State University assembled a

team that collaborated with all constituent parties, resolved differing term definitions, and

aligned the information with the key performance indicators of management and overall

university goals. In addition, the team ensured that each level of management had

appropriate access to the data, made certain new technologies were easy to use, and

educated staff, faculty, and management on the complexities of the new analytics

(Blanton, 2012).

The implementation of academic analytics resulted in evident positive outcomes

for Portland State University. End users of data began asking better questions about the

data and how the data could help in decisions making. There was increased collaboration

throughout the university, and, reports that once took weeks to assemble took a matter of

minutes to complete after implementation. Portland State University has begun to use

analytics to move toward performance-based budgeting, instead of relying on “gut

feelings.” As confidence in the new systems grows, the university made plans to abandon

the old legacy and shadow systems (Blanton, 2012).

In another case of adoption of analytic tools, Saint Michael’s College experienced

benefits in the use of a dashboard to control for management key performance indicators.

Typical problems faced by Saint Michael’s College included, “too many reports and

authors, inconsistent data definitions, a lack of systematic updates, poor coordination of

key measures, and haphazard sharing of reports and updates” (Anderson & Russell, 2012,

p. 1). The college admitted that many decisions were made by “gut feeling” due to the

lack of consistent data, dated, or inaccessible data (Anderson & Russell, 2012).

Leadership of the college understood that one specific goal for the college was to

attach benchmarking measures, or key performance indicators, to a dashboard, with the

use of analytics. To begin to use the dashboard to control key performance indicators, the

college needed to establish consistent data definitions, synchronize timing of data streams

and cycles, elucidate data interpretations, and create a culture of transparency. Anderson

and Russell (2012) hoped that with these objectives met, accountability for performance

of key measures could begin.

The development and college-wide usage of the dashboard experienced

challenges at Saint Michael’s College. There were pockets of stakeholders that were

unenthusiastic about sharing departmental data, and the college struggled to define,

clarify, and standardize the most basic, but complex, terms. The development team had to

explicitly focus on issues such as sharing of the data across departments and college-

wide, the interpretation of definitions, and the synchronization of data (Anderson &

Russell, 2012).

Because of the team’s collaboration, Saint Michael’s College saw growth in the

use of its dashboard to control for key performance indicators in the operational and

strategic applications of the college. The dashboard was highly exploited and its use had

increased to additional departments throughout the college. The college explained that

“the dashboard has filled a gap by providing more timely, tactical data and supplementing

our quarterly scorecard and annual fact book” (Anderson & Russell, 2012, p. 1).

A different success story of the use of academic analytics rests with Paul Smith’s

College. Paul Smith’s College served a high-risk student population in that over 50% of

the students are first-generation college students, and almost 50% of these students

graduated in the lower half of their high school graduating class (Taylor & McAleese,

2012). The college needed to increase the success of students through increased retention

and graduation rates. The challenge for the college was the early identification of its at-

risk students, and the automation of data gathering, reporting, and communication.

Paul Smith’s College implemented a predictive modeling analytics tool to predict

using data, students’ end-of term grade point average, and thus classify highly at-risk

students and present them with counseling and tutoring services prior to the first day of

the term. The college also implemented a system that would run routine reports and

analyses automatically and disseminate results to targeted support teams. This analytic

tool additionally sent communication to students regarding concerns of lower

examination scores and participation rates. Support teams were also notified so that staff

could intervene early, as opposed to before the tool when support staff only saw scores a

quarter or halfway through the term (Taylor & McAleese, 2012).

The college saw encouraging results with the use of the newly adopted analytic

tools. The percentage of students placed on academic probation decreased by 36%.

Additionally, the percentage of students who were academically suspended from the

college decreased 41%. Graduation rates of students saw an increase of 23%, and the

college experienced a rate of return on their investment of over $2 million dollars in net

student tuition (Taylor & McAleese, 2012).

A final instance of positive returns from the adoption of academic analytics was

that of Arizona State University (ASU). ASU is one of the largest higher education

institutions in the United States, reporting more than 72,000 students spread throughout

its four on-ground campuses. The growth of the institution and financial challenges

helped AUS become one of the early adopters of academic analytics (Wishon & Rome,

2012).

In 1993, ASU developed a formal institutional wide database where all data were

stored, and then used in various departments campus-wide. Users of this organized

integrated system could build reports, perform analysis, and integrate data where

necessary. The IT team used the integrated data to build dashboards to help recruitment

and admissions processes, research endeavors, financial and budgeting expenditures,

facilities management, human resources, and student affairs (Wishon & Rome, 2012).

To determine growth of analytics for ASU, the IT team began to think about

monitoring the dashboards to discover which dashboards were being utilized the most,

and which information was being accessed the most. The team created a dashboard that

observed and monitored the previously constructed dashboards; “they placed analytics on

top of analytics” (Wishon & Rome, 2012, p. 1). With this usage dashboard, the IT team

could see which departments were heavy users, what information they were using, and

which dashboards they did not access.

Given this information, ASU could pinpoint areas to focus funding and determine

growth patterns. The IT team could identify potential users and perform training when

necessary. The knowledge provided by the analysis of the dashboards, via the usage

dashboard, enabled AUS to become a data-driven decision making intuition (Crow, 2012;

Wishon & Rome, 2012).

Although there are definite cases whereby colleges and universities have adopted

academic analytics to great success, Bichsel’s survey conducted in 2012 concluded that

the majority of institutions surveyed had not yet begun the first steps to adopt an analytic

tool to help with the management of college enterprises, goals, and performance

measures (Bichsel, 2012). Wagner and Ice (2012) explained that although businesses

used pattern recognition and predictive analytics to make better decisions, analytics “are

not yet broadly used in educational settings, where they could assist with activities such

as selecting courses or predicting when students might be at a point of increased

academic risk” (p. 33).

Non-Adoption of Academic Analytics

Goldstein (2005) surveyed 380 higher education institutions to discover how

successful colleges and universities had been in adoption analytics to strategically drive

operations. His team asked questions about how prevalent the use of predictive modeling

and alerts was, and how universities used analytics to drive decisions. Goldstein (2005)

found that of the colleges and universities surveyed, only 15% used analytics in a

strategic way; and that 46% used data for static reporting solely.

Bichsel (2012) conducted the “2012 Analytics in Higher Education” study to

indicate the status of analytics in higher education institutions. Bichsel surveyed 339

colleges and universities, and found substantial amounts of institutional data collected in

the areas of enrollment, finance and budget, student progress, research, and learning

management were not integrated into one area whereby it could be analyzed to make

proactive decisions (Bichsel, 2012). Dawson et al. (2010) argued that despite pockets of

successful implementation of analytics in higher education institutions and a decade of

business use of analytics to drive decisions and strategically plan, adoption in the

education sector remained nominal.

Colleges and universities are under pressure to change the way they do business,

to become more efficient, provide higher quality of services, and to be able to measure

success (Siemens & Long, 2011). Colleges are faced with newer challenges of

competition and decreased governmental assistance (Dawson et al., 2010). Researchers

have shown that academic analytics increase student retention, provide answers to

questions such as the cost of a degree, improve resource management, provide

visualization of operations in true time, and supply decision support based on substantial

facts (Bichsel, 2012). However, higher education institutions are still slow to adopt

analytics due to either perceived or actual barriers (Bichsel, 2012; Dawson et al., 2010).

Barriers That Impede Adoption of Analytics

The biggest impediments for analytics adoption in business organizations lie in

managerial and cultural concepts such as managers not knowing how analytics could help

their business strategies, managerial priority competition, competing cultures within

departments not wanting to share data, and a lack of analytic skills in-house (Lavalle et

al., 2011). Unlike the barriers that impede businesses from adopting analytics, Bischel

(2012) argued that higher education institutions do not adopt due to cost. Bichsel also

indicated culture, infrastructure, and policy as being barriers. Other studies have indicated

resource competition may be a barrier, or a competition between adoption of analytic

tools and the option to hire additional instructors has placed colleges and universities at a

standstill (Ravishanker, 2011).

Because institutions of higher education have been slow to adopt analytic tools

which other business industries have found successful in helping to improve

performance, and because such tools represent an innovation in the way in which higher

education utilizes business processes, I also considered literature addressing barriers to

innovation adoption. These studies most often addressed the adoption of a recent

innovation in higher education, the adoption of eLearning technologies. They also

provided potential information as to the reasons why higher education institutions may be

reluctant to embrace innovative technologies, including analytics, even though they

demonstrated their success in other industries, including higher education.

Several issues can motivate IT adoptions. Reid (2014) found that five categories

influenced the adoption of an innovation such as instructional technologies. These issues

included (a) the technology itself, including access, reliability, and the complexity of the

system; (b) the process by which such technology was implemented and the support

provided to all levels of users; (c) administrative leadership and support; (d) the

environment such an innovative change is implemented into, including changes in roles,

control, and a shift in focus to a business model; and (e) the control and effectiveness of

the users of innovative technologies. Lane and Lyle (2011) found that expertise in

technology use, institutional support, and having strategies in place to facilitate adoption

of innovative technologies were key factors in encouraging adoption. Singh and Hardaker

(2014) also found institutional and managerial, or bureaucratic support necessary for the

adoption of innovations such as eLearning. Managers not only provide support in

resources, but also by providing role models for the use of such innovative initiatives and

absent this support, significant cultural barriers exist to innovation adoption. These

studies echoed earlier research by Johnson (2010), who found that the perception of risk,

knowledge of the value of innovation adoption, trust in the system, size of the

organizational system, and the readiness of the organization to utilize innovation may

result in barriers to adoption of innovative strategies, even if they improve performance.

Gap in the Literature

Following the literature review, I was able to recognize that there were limited

studies conducted as to why higher education institutions do not adopt analytics. The

literature review helped provide an overview as to why colleges and universities are slow

to adopt analytic tools that may be able to increase performance in key indicator areas.

There were few studies, if any, directly exploring the reasons behind non-adoption in

higher education institutions. Businesses have adopted analytic tools that have improved

key performance indicators (Chen et al., 2012; Davenport et al., 2010; Fahey, 2009;

Minkara, 2012). Several higher education institutional organizations have adopted such

tools with positive results (Dziuban et al., 2012; Pistilli & Arnold, 2010; Ravishanker,

2011; Smith et al., 2011; Vialardi et al., 2011). Relatively few studies have indicated the

reasons why few academic institutions have yet to adopt such analytics (Bichsel, 2012;

Ravishanker, 2011). A review of literature addressing barriers to adoption of other

innovative technology suggested that the technology itself, the users, and the bureaucratic

system may be major barriers to adoption (Johnson, 2010; Lane & Lyle, 2011; Reid,

2014; Singh & Hardaker, 2014).

This study extended the literature by exploring the reasons behind why a

community college has not adopted analytics to help its academic managers better control

their key performance indicators. These performance indicators included student

retention, student engagement, faculty training and observation, improved access,

curriculum updates, course scheduling, and student/faculty budget ratios.

By comparing the findings of this study to what is known through previous

literature on the use of academic analytics and potential barriers to such innovation

adoption, it was hoped that further research would be conducted. The goal of further

research would be to help design proactive strategies so that the adoption of such tools

could benefit both the users (administrators and students) and that success in key

performance indicators may be realized.

Summary of the Literature

Through the literature review, I discovered that businesses, such as Google and

Amazon, have been using analytics to increase productivity, strategically plan, and drive

profits (Chen et al., 2012). I examined case studies whereby colleges have also had

success using analytics to streamline admissions processes, increase student retention and

success rates, track and plan for growth, and evaluate challenges and solutions (Dziuban

et al., 2012; Smith et al., 2011). Through the study of the literature review, I was also

able to determine that there was not wide spread use of analytics in higher education

institutions, even after studies have indicated the positive results of usage (Bichsel, 2012;

Dawson et al., 2010). The following chapter, Chapter 3, describes how I conducted this

study.

Chapter 3: Research Method

Introduction

The purpose of this qualitative phenomenological study was to explore the

barriers that inhibit higher educational institutions in their adoption of proven analytic

tools to help improve management of key performance indicators. In this chapter, I

described how I conducted this study. This chapter includes the research design,

population, setting, instrumentation, data collection procedures, plan for data analysis,

and the ethical procedures undertaken.

Research Design and Rationale

I designed this study to explore concepts related to the nonadoption of knowledge

management, specifically academic analytic tools, in higher education. The general

research question that guided this study was the following: What factors impede the

adoption of academic analytic tools in a higher education setting? Subsequent questions

included

1. Are academic administrators aware of how academic analytics could help

manage key performance indicators?

2. What types of discrete databases do academic administrators currently use to

help the management of their perspective departments?

3. How can knowledge management tools enhance the efficiency of a higher

education institution?

4. Does the climate of a secondary education institution hinder the adoption and

use of analytic tools, or are there funding/investment issues?

5. Would college administrators use academic analytics to help increase student

success and other managerial tasks?

The main manuscripts examined in determining the design for this study included

Creswell (2012, 2013), Merriam (2009), and Englander (2012). I designed the study to

gather personal data from the interview process to explore barriers that prevented

colleges and universities from adopting analytic tools to support management

efficiencies. The mission of qualitative research is to (a) explore how people understand

their experiences, (b) discover how people create their worlds, (c) understand how people

make sense of their experiences, and (d) describe how people understand their experience

(Merriam, 2009).

I reviewed qualitative and quantitative methods to determine the best approach for

the study. Creswell (2013) noted key differences in qualitative and quantitative methods

by comparing the two research inquiry approaches. When the researcher needs in-depth

and detailed research, and when flexibility without categorization is desirable, qualitative

inquiry methods are best (Creswell, 2012, 2013). Researchers should consider

quantitative methods when they need to generalize large samples with limited responses

on a broad scale (Creswell, 2012).

I designed this study to explore why the participants at a community college do

not engage in the use of analytics to increase efficiencies. Creswell (2012) argued that the

search to establish meaning behind thoughts, experiences, or behaviors would necessitate

a qualitative research approach. I designed this study to explore, in detail, a complex

issue that needed understanding with the desire to allow participants to share their

experiences to help form a better understanding of the problem. Singleton and Straits

(2009) posited that the social science researcher’s purpose is to gain an understanding

about how people think, feel, and interact during a phenomena. To explore experiences

and actions of participants, the researcher should ask open-ended, succinct questions as

the principal strategy for qualitative social research (Creswell, 2012). These concepts

helped guide this research in the direction of collecting qualitative data that generated

straightforward quotes from people regarding their feelings, opinions, and experiences

with respect to their nonuse of analytical data in their daily management activities and

barriers that prevented them from usage (Singleton & Straits, 2009).

Phenomenological Study

I considered the case study and phenomenology research traditions for this study

(Creswell, 2012). A case study concerns an issue explored “through one or more cases

within a bounded system” (Creswell, 2012, p. 73). Simon (2011) reported that a

researcher uses case study research when the inquirer establishes a problem and uses

questions such as why and how. A case study was considered for this research because I

wished to explore a bounded system in which several individuals would be interviewed

and the research questions were why- and how-focused. I deemed the choice of a case

study inappropriate, however, due to the data collection sustained in a case study. Data

collection in a case study draws on multiple sources to include observations, documents,

archival records, physical objects, and audiovisual materials (Creswell, 2012). The

primary data collection for this study was rooted in in-depth, open-ended interviews.

I chose a phenomenological approach to qualitative research for this study. I

designed this study to understand and explore the experiences of individuals managing

departments in a higher education setting, their experience in using or not using analytics,

and the meaning behind their perceptions of analytic tools. Additionally, Simon (2011)

stated, “phenomenological research is people’s experience in regard to a phenomenon

and how they interpret their experiences” (p. 105). The use of phenomenology was also

chosen due to the emphasis of open-ended interviews as the primary data collection

(Creswell, 2012).

Role of the Researcher

I had professional relationships with the population; however, I did not supervise

any of the participants. This nonrelationship allowed me to remain as an outsider and an

objective interviewer.

I gained access to the institution by a structured meeting with the director of

institutional research for the college. The director of institutional research provided

verbal permission at the time of the meeting. I attributed this immediate response to my

employment within the college. The college’s institutional review board (IRB) conducted

further negotiations concerning the determination of actual participant lists and a formal

review prior to the data collection process.

My background in academics, specifically in managerial academic positions,

guided my interest in exploring higher education management uses of academic analytics.

However, I never worked in the capacity of an academic manager at the college under

investigation in this study. The resolve to engage in a study of this college rested on the

resources that were available to me, as well as my familiarity with the college’s

administrative structure (Simon, 2011). I classified the knowledge of the managerial

structure of the college as a strength for this study due to the need to interview key

managerial positions within the college.

Methodology

Participant Selection Logic

College Z employs approximately 3,500 staff and faculty members working in six

different locations and on-line. I took the population for the study from managers who

had accountable key performance indicators and not from other individuals who would

not have academic responsibilities that directly affect student retention, faculty

performance, and academic curriculum and academic policy outcomes. There were 25

persons in this category. Only persons who had key performance indicators, which were

measurable, would have the necessity to use analytic tools to assist them in meeting their

goals. An example of a key performance indicator for an academic dean is to retain a

certain amount of students in a program from one semester to the next.

I used a criterion sampling method to learn more about how, why, if and why not,

College Z used academic analytics. Merriam (2009) suggested, in a qualitative study, to

select participants from the sample in which the researcher can learn information. With

this in mind, I focused on participants who met certain criteria. Only employees who had

measurable key performance indicators were in a position to use analytic tools. Within

this population, I selected participants in an academic department. This selection was

necessary to control for a reasonable sample size. My purpose in this study was not to

generalize results to all U.S. colleges and universities, however, but to explore barriers to

analytical tool adoption specifically at College Z (Creswell, 2012). I selected the

participants in the sample size based on the following criteria: (a) had student- and

faculty-driven measurable key performance indicators and (b) worked in an academic

department.

I contacted and worked with the human resource department to obtain a list of

criterion-based participants. Personnel at the human resource department provided a list

of participant names, work phone numbers, and work e-mail addresses (see Appendix A).

Of this pool, the sampling size was random as it was self-selected and voluntary. I

conducted this purposeful random sampling from the criteria-established pool to add

credibility and reduce researcher bias (Creswell, 2012; Englander, 2012).

The next step I took was to review the guidelines for participant size in a

phenomenological study. Creswell (2012) suggested that for a phenomenology study,

collecting in-depth data involves participants ranging from three to 10 subjects. Twenty-

five individuals located in academic divisions throughout the college qualified for the

study. As my intention with the study was to serve as a representation of all United States

colleges and universities and their barriers to analytic adoption, Merriam (2009)

suggested a small information-rich sample size in which a deep understanding could be

achieved. Drawing from this logic a sample size of 40%, I chose a percentage that

obtained saturation, equaling 10 participants.

I sent a letter describing the research and the request for an interview to the

participant pool through the college e-mail system (see Appendix B). The letter gave an

overall summary of the research, why I needed an interview, the estimated time it would

take for the interview, and a strict notice of confidentiality. The letter asked the

participant to contact me if the possible participant was willing to grant an interview. I

then sent the first 10 responses closed the sample and a follow-up e-mail to all

participants in the pool, stating that I had reached the required research pool size. This e-

mail thanked the possible participants for any consideration they had given to take part in

the study (see Appendix C). I then e-mailed a letter to the 10 interview participants

thanking them for agreeing to participate, and describing the research, interview process,

and purpose in more detail. The letter asked availability of days and times in which to

schedule the interview (see Appendix D). I also attached the interview questions so that

participants could review the questions and form thoughts about the subject matter (see

Table 4).

Interview Process

All interviews were held in the office or a predetermined space identified by the

interviewee. I opened each interview by asking the interviewees whether they were

comfortable with questions regarding the use of analytic tools and information

technology in the management of the interviewee’s activities, and to remind them that I

would record the interview for transcription purposes. I reminded the interviewee that the

interview was voluntary and that the interviewee could stop the interview for any reason

at any time. I took minimal field notes and depended on the computer recording software

for later in-depth transcription.

I asked the interviewees whether there were any other comments or clarifications

needed before the closure of the interview sessions. I reminded the interviewees that I

would send a full transcription to the interviewees to review, clarify, and make any

comments as deemed necessary. I thanked each interviewee for their time and gave a date

at which I would send their transcription to them for follow-up review.

Instrumentation

The instrumentation used in the study included two previously published studies.

Creswell (2012) suggested the use of interview questions designed and validated in

previous studies to maximize credibility, to use as a foundation, background, and

strategy. I used two studies, namely Ali and Magalhaes (2008), and Venkatesh and Bala

(2008). I used the interview questions from Ali and Magalhaes’ study as a base of

comparison to the interview questions from Venkatesh and Bala’s study. I modified the

questions from Venkatesh and Bala’s study to meet the needs of this study. Modification

of the instrument involved the expansion of the concept of IT barriers from the original

instrument and the addition of newly designed and appropriate context detailed questions

to better describe the appropriate academic analytic tool term used for this study.

I provided comments to give participants a general background for each question

and to ensure participants had an accurate interpretation of each question, and that I

clearly understood the meaning of the participant’s response. The use of the comments

helped maximize credibility (Creswell, 2012).

The open-ended interview approach reduced interviewer bias, and permitted

evaluation of the collected data to be easily compared and analyzed (Creswell, 2012).

Face-to-face interviews allowed the interviewees to respond in-depth. I used a recorder

imbedded in a laptop computer so I would be able to concentrate on making the

participant(s) comfortable with adequate eye contact and to encourage the participant(s)

to speak and share ideas freely (Creswell, 2012). The interviews allowed for an

exhaustive exploration into barriers of academic analytic adoption at College Z.

Published Instrument

Ali and Magalhaes (2008) conducted a study in Kuwait with a sample of human

resource managers and IT development managers to determine barriers of an IT adoption

platform. In this context, Ali and Magalhaes’ case study was appropriate to use for

comparison purposes and as a foundational tool for this study.

Ali and Magalhaes’ (2008) interview tool was validated through the systematic

use of a previously published query list to guarantee internal validity, credibility, and

authenticity. Additionally, Ali and Magalhaes conducted a pilot study to further validate

the chosen interview instrument. The researchers established content and internal validity

through the pilot study (Ali & Magalhaes, 2008). See Appendix E for permission to

reprint the Ali and Magalhaes interview protocol.

1. To what extent is e-learning used in your company? Who are the users, who

are the providers and what is the range of courses covered through e-learning?

This question relates to the following research question in this current study:

Are academic administrators aware of how academic analytics could help

manage key performance indicators? The theme that resulted: There is an

awareness of analytics and analytic tools at the college. The finding that

resulted: Climate and Policy are barriers to adoption.

2. How closely does the organization’s training policy fit with e-learning? Did

the use of learning technologies raise the standards of employee’s

performance? How prepared is your organization to deal with the large and

increasingly complex e-learning marketplace? This question relates to the

following research questions in this current study: Are academic

administrators aware of how academic analytics could help manage key

performance indicators? What types of discrete databases are currently used

by academic administrators to help the management of their perspective

departments? The theme that resulted: Technologies currently used to manage

key performance indicators. The finding that resulted: Possible infrastructure

and policy are barriers to adoption.

3. What challenges does the organization face in the setting-up and/or

implementation of e-learning? From your organization’s experience, what are

the top 3 barriers of starting/implementing e-learning? This question relates to

the following research questions in this current study: Does the climate of a

secondary education institution hinder the adoption and use of analytic tools,

or are there funding/investment issues? And also, would college academic

administrators use academic analytics to help increase student success and

other managerial tasks? The theme that resulted: Investment of analytic tools.

The finding that resulted: Climate of the college may be a barrier to adoption.

4. Taking into consideration the challenges both employers and employees

encounter: (1) Is e-learning worth the investment? If yes, explain. This

question relates to the following research questions in this current study: Does

the climate of a secondary education institution hinder the adoption and use of

analytic tools, or are there funding/investment issues? And also, would college

academic administrators use academic analytics to help increase student

success and other managerial tasks? The theme that resulted: The theme that

resulted: Investment of analytic tools. The finding that resulted: Climate of the

college may be a barrier to adoption. (Ali & Magalhaes, 2008, pp. 38-39)

I reviewed and modified an additional interview instrument for this study.

Venkatesh and Bala (2008) explored barriers to IT implementation in companies and

institutions. Venkatesh and Bala designed the longitudinal field study to determine the

perceived usefulness and the perceived ease of use of an IT implementation from

employees working at four different organizations (Venkatesh & Bala, 2008).

Venkatesh and Bala (2008) used constructs validated from the Technology

Acceptance Model 2 study and operationalized it in a prior study. Questions regarding

barriers to IT implementation brought forth from the 2008 study were appropriate for this

study with appropriate modifications for specific content. See Appendix F for permission

to modify instrument.

1. What specific design characteristics will influence the determinants of

perceived usefulness and perceived ease of use? This question gives a broad

umbrella of usefulness and awareness related to this current research question:

Are academic administrators aware of how academic analytics could help

manage key performance indicators? The theme that resulted: There is an

awareness of analytics and analytic tools in higher education. Finding that

resulted: Climate and policy may be barriers to adoption.

2. What are the effects of the different ways of user participation on the key

determinants of perceived usefulness and perceived ease of use and

consequently, perceived usefulness and perceived ease of use? This question

relates to the following research question: How can knowledge management

tools enhance the efficiency of a higher education institution? The theme that

resulted: How analytics can help with the management of key performance

indicators. The finding that resulted: Training issues may be a barrier to

adoption. And also this question relates to the following research question:

What types of discrete databases are currently used by academic

administrators to help the management of their perspective departments? The

theme that resulted from this question: Technologies currently used to manage

key performance indicators. The finding that resulted: Policy and

infrastructure may be barriers to adoption.

3. What forms of management support are important in creating favorable

perceptions toward a new system? This question relates to the following

research question in the current study: Does the climate of a secondary

education institution hinder the adoption and use of analytic tools, or are there

funding/investment issues? The theme that resulted from this question was:

Investment of analytic tools. The finding that resulted: The climate of the

college may be a barrier to adoption. (Venkatesh & Bala, 2008, pp. 275-276)

Developed Instrument

I based the development of the interview protocol for this study (see Table 4) on

the prior studies by Ali and Magalhaes (2008), and Venkatesh and Bala (2008). Both

studies investigated barriers to IT implementation. Ali and Magalhaes (2008) developed

their study to discover IT implementation barriers in an academic setting. Venkatesh and

Bala (2008) focused on discerning IT implementation barriers and perceived usefulness.

I established content validity for the interview protocol for this study using a pilot

study. I chose three participants for the pilot study. I gave the interview to the participants

in the exact manner in which I conducted the main study. I asked the participants in the

pilot study questions regarding the content of the interview questions. I asked (a) did each

question made sense to them, (b) was each question clearly stated, and (c) was there a

better way to state the question? I recorded their responses and made improvements to the

interview questions.

Interview Protocol Used in Pilot Study

1. Can you think of how you use information technologies in the management of

your daily activities? Used to explore the research question: Are academic

administrators aware of how academic analytics could help manage key

performance indicators?

2. What are your primary key performance indicators/goals? Used to ensure the

participant met the criterion-based selection process.

3. Describe how you manage your primary key performance indicators/goals?

Used to explore the research question: Are academic administrators aware of

how academic analytics could help manage key performance indicators?

4. What is your position within the organization? How long have you worked for

the organization? Used for demographic information.

5. Describe the kinds of data you use in order to manage your performance

indicators/goals. Used to explore the research question: What types of discrete

databases do academic administrators currently use to help the management of

their perspective department?

6. What kinds of IT support do you believe would help you accomplish your

goals more effectively? Used to explore the research question: How can

knowledge management tools enhance the efficiency of a higher education

institution?

7. Describe your experience using technology to reach or exceed your

performance goals. Used to explore the research question: Are academic

administrators aware of how academic analytics could help manage key

performance indicators? Also used for background information.

8. Describe any training you have received in the usage of technology in your

workplace. Used for background information.

9. Do you believe the use of technology in academic management is worth the

investment? Please explain. Used to explore the following research questions:

Does the climate of a secondary education institution hind the adoption and

use of analytic tools, or are there funding/investment issues? And also, would

college administrators use academic analytics to help increase student success

and other managerial tasks?

10. If you do not use data and analytics to help manage your key performance

indicators, can you explain why not? Used to explore the following research

questions: Does the climate of a secondary education institution hinder the

adoption and use of analytic tools, or are there funding/investment issues?

Would college administrators use academic analytics to help increase student

success and other managerial tasks?

Pilot Study

I conducted the pilot study for several reasons. First, I used it to control for

validity. Secondly, I viewed it as valuable in that I asked the subjects of the pilot study

for feedback to identify vagueness in questions, and to identify difficult questions. Third,

I was able to record the time it took to complete the interviews. Fourth, I was able to re-

word ambiguous questions and discard unnecessary questions. I administered the

interviews in the same manner in which I conducted the main study.

I drew participants for the pilot study from academic managers, meeting the same

criteria as the main study, who worked for a different college: College X. I recruited

College X participants using a snowball purposeful sampling technique. This technique

allowed me to speak to information-rich criterion-met persons, while extending the pilot

to similar participants without the use of the ancillary resources garnered from College X

(Creswell, 2012; Merriam, 2009).

College X’s website identified Participant A as an academic manager. I sent an e-

mail inquiring whether participant A would be interested in participating in the pilot

study (see Appendix G). At the time of the interview, I asked Participant A for names of

persons who met the interviewee criterion and who would possibly be interested in

participating in the pilot study.

I was the sole data collector. I used a laptop-imbedded recorder, and took field

notes during the interview. I gave the participants information regarding the intent of the

pilot study, as well as the purpose of the main study. I gave the interviewees the interview

questions ahead of time, and asked whether they had questions about the interview prior

to the scheduled interview. I asked the participants about the structure of the questions,

their understanding of the questions, and to suggest any improvements. See Appendix H

for the IRB approval number.

Data Collection Procedures

I used interviews to explore barriers to the adoption of analytic tools in a higher

education organization. The interview questions were adapted and modified to meet the

needs of this study (Venkatesh & Bala, 2008). The individuals interviewed met criteria

based on their academic management roles. In the event that there were fewer

participants due to unexpected circumstances, I could have easily contacted members

from the original list of prospective participants.

The interviews took place in the office of the individual participants; this was

necessary, as the time an academic manager would lose leaving campus was valuable.

The use of the open-ended questions allowed for the participants to expand their answers

if they wished. It allowed me to elicit further information if there was an opportunity. I

used a built-in laptop recorder to record the interviews, and I took field notes during

interviews.

I reminded the participants at the time of the interview that their interview was

voluntary and that I would keep all confidentiality in place. I reminded the participants

that they could refuse without reason, to answer any question. I told the participants that

they would be able to review the transcript of their interview to make certain that I

recorded their answers appropriately.

I conducted the interviews within a period of four weeks. I scheduled each

individual participant for the interview at his or her convenience. I transcribed and

encoded the data collected during the interviews using the computer software MAXQDA,

see Appendices I, J, K, L for samples.

When the participants exited the interview session, I asked each interviewee again

to verify their contact information. I did this so that I could send the transcribed interview

to the interviewees for review. I sent the transcribed interviews to the participants, by e-

mail, so that they could make any adjustments they feel necessary.

Data Analysis

To explore barriers to adoption of analytic tools in College Z, I used data gained

from the in-depth interviews of academic managers. The goal of this data collection was

to obtain a deeper understanding of the factors that inhibited educational managers from

using analytical tools to help increase key performance outputs. I recorded and

transcribed each interview word for word to perform initial coding (Creswell, 2012).

I built the main categories of the study from the main research questions

(Schreier, 2012). I then derived the main categories from the coding frame. I

accomplished the coding frame by analyzing the content exhaustively (Schreier, 2012). I

chose conventional content analysis for this study based on the phenomenological

approach to the research question (Creswell, 2012).

To answer the research question described in the study, I developed categories

from significant statements in the interviews (Creswell, 2012). I then expanded the

categories into themes, or codes, which explored barriers in the adoption of analytical

tools (Creswell, 2012). The modified interview protocol safeguarded an equivalency

between the research questions and the interviews (Venkatesh & Bala, 2008).

Data analysis included the use of the MAXQDA qualitative software analysis

tool. I recorded the interviews using Apple’s MacBook Pro software and an imbedded

microphone. I will store all collected documents, and I will destroy said documents after

five years to ensure participant confidentiality.

Trustworthiness

I established credibility using member checks and peer review (Creswell, 2012). I

returned the transcribed interview sessions to each individual participant. In this manner,

I gave the participants the opportunity to adjust faults they found in the transcription.

Afterward, I gave them the themes that resulted from their session. This provided the

participants an occasion to challenge results, add information which they may have

omitted during the interview, or explain any misunderstanding (Creswell, 2012).

Peer review added credibility to the study. I met with a researcher outside of the

organization under study to debrief the interview notes. I also took notes and reviewed

these notes during the debriefing sessions (Creswell, 2012). I discussed methods,

procedures, understandings, and feelings to make sure that I had an outside review of the

research (Creswell, 2012).

A rich description was adapted to describe the setting and the participants’

interview session. This was done for readers of the research to “transfer information to

other settings and to determine whether the themes can be transferred” (Creswell, 2012,

p. 209). The use of thick description aided in external validity of the study (Creswell,

2012). This in-depth, rich description also established dependability of the research. The

exhaustive coverage allowed readers to repeat the procedures and methods used in this

study in another study with some understanding that they may find similar results (Lietz

& Zayas, 2010).

Because of the difficulty in ensuring real objectivity in cases where humans

interact with humans as in a qualitative study, I considered the participants’ experiences

and impressions brought forth from the interviews (Merriam, 2009). To control for

personal biases and personal experiences, I kept notes unembellished. Additionally, I kept

writing clear and concise with objectivity as an overall goal (Creswell, 2012).

Ethical Procedures

I obtained access to interview participants for this study from the appropriate

departmental manager at College Z (see Appendix M). This process involved e-mailing

the manager to obtain an informal meeting to discuss the study. During the meeting, I

explained in detail the manner and purpose of the study. The manager then e-mailed an

approval to use College Z for data collection.

I obtained approval from Walden University through the IRB. The IRB approval

number for this study is 01-28-14-0231112 and is valid through January 27, 2015 (see

Appendix H).

I e-mailed each participant a consent form with details regarding the treatment of

humans in a research study (see Appendix N and Appendix O). The consent form assured

the participants of confidentiality, the right to withdraw from participation at any time, up

to, during, or before the publication of the study. It stated that I would provide the

interviewees with the transcripts of their interview, and that I would ask them to review

for any errors. I further asked participants to read the consent form, and sign and return it

to me prior to scheduling the interview. Once I received the consent form(s), the

participant(s) were contacted in order to schedule the interview.

To address further issues of ethical concerns, I gave no therapy to the participants.

Questions from the interview did not ascertain humiliating or hostile information. The

interviews were private and confidential (Creswell, 2012). There were no incentives

given for participation in the study. I informed participants that I would store all

interview documents and recordings, and that I would destroy said documents after 5

years to safeguard confidentiality. Furthermore, although I conducted the study at my

place of employment, I have little to no contact with the interviewees or the content or

subject matter of the research within the College. My role at College Z does not intersect

with the issues brought forth in this study.

Summary

Researchers showed that the use of analytic tools improved key outcomes and

accountability measures for colleges (see Chapter 2). However, higher education

institutions are slow to adopt these proven tools (see Chapter 2). In this study, I explored

the barriers to the adoption of analytic tools in College Z. The exploratory nature of the

study led me to choose a qualitative method for the research. The intent of conducting

such a study was to ensure an in-depth examination of data collected at College Z.

I took measures to ensure privacy of the participants of the study. I safeguarded

credibility and validity through peer reviews and member checks. I provided external

validity by gathering detailed information from participants. I mitigated ethical concerns

using consent forms and approval of the IRB. In the next chapter, Chapter 4, I present the

data collected following this prescriptive chapter.

Chapter 4: Results

Introduction

The purpose of this phenomenological study was to explore the factors that

inhibited higher educational institutions in their adoption of proven analytic tools to help

improve management of key performance indicators. I interviewed academic managers at

a community college to explore their perspectives of this phenomenon. I used open-ended

interview questions to gain a greater understanding of the experiences and perceptions of

academic managers at College Z. The interviews allowed for an exploration into barriers

of academic analytic adoption at the institution. The general research question that guided

this study was the following: What factors impede the implementation of academic

analytic tools in a higher education setting? Subsequent guiding questions included

1. Are academic administrators aware of how academic analytics could help

manage key performance indicators?

2. What types of discrete databases do academic administrators currently use to

help the management of their perspective departments?

3. How can knowledge management tools enhance the efficiency of a higher

education institution?

4. Does the climate of a secondary education institution hinder the adoption and

use of analytic tools or is there an investment/monetary issue?

5. Would college administrators use academic analytics to help increase student

success and other managerial tasks?

In Chapter 4, I include a detailed description of the manner in which I conducted,

recorded, and transcribed the interviews. I also present the analysis and results of the

interviews. The final section contains a summary of the results representing and relating

to each participant. Open-ended interviews provided an opportunity to explore the

perspectives of academic managers at a higher education institution.

Pilot Study

I drew participants for the pilot study from academic managers who met the same

criteria as the main study and who worked for a different college, namely College X. I

recruited College X participants using a snowball, purposeful sampling technique. This

technique allowed me to speak to information-rich criterion-met persons, while extending

the pilot to similar participants without the use of the ancillary resources garnered from

College X (Creswell, 2012). I identified Participant A through College X’s website as an

academic manager. I sent an e-mail inquiring whether participant A would be interested

in participating in the pilot study (see Appendix G). Participant A agreed to be a

participant in the pilot study. At the time of the interview, I asked Participant A for the

names of persons who met the interviewee criteria and who would be interested in

participating in the pilot study. Participant A gave two other names of persons who met

the criteria.

I held the interviews at quiet, off-campus locations near the college that the

participants could easily access. I gave the participants information regarding the intent

of the pilot study, as well as the purpose of the main study. I also gave the interviewees

the interview questions ahead of time, and I asked if they had questions about the

interview prior to the scheduled interview.

I recorded interviews using a computer laptop, and I took sparse field notes. I

conducted the interview in the exact same manner in which I conducted the main study.

Directly after the interview, I asked the participants about the structure of the questions,

their understanding of the questions, and to suggest any improvements. I asked the

following interview questions.

Original Interview Protocol

1. Can you think of how you use information technologies in the management of

your daily activities? Used to explore the research question: Are academic

administrators aware of how academic analytics could help manage key

performance indicators?

2. What are your primary key performance indicators/goals? Used to ensure the

participant met the criterion-based selection process.

3. Describe how you manage your primary key performance indicators/goals?

Used to explore the research question: Are academic administrators aware of

how academic analytics could help manage key performance indicators?

4. What is your position within the organization? How long have you worked for

the organization? Used for demographic information.

5. Describe the kinds of data you use in order to manage your performance

indicators/goals. Used to explore the research question: What types of discrete

databases do academic administrators currently use to help the management of

their perspective department?

6. What kinds of IT support do you believe would help you accomplish your

goals more effectively? Used to explore the research question: How can

knowledge management tools enhance the efficiency of a higher education

institution?

7. Describe your experience using technology to reach or exceed your

performance goals. Used to explore the research question: Are academic

administrators aware of how academic analytics could help manage key

performance indicators? Also used for background information.

8. Describe any training you have received in the usage of technology in your

workplace. Used for background information.

9. Do you believe the use of technology in academic management is worth the

investment? Please explain. Used to explore the following research questions:

Does the climate of a secondary education institution hind the adoption and

use of analytic tools, or are there funding/investment issues? And also, would

college administrators use academic analytics to help increase student success

and other managerial tasks?

10. If you do not use data and analytics to help manage your key performance

indicators, can you explain why not? Used to explore the following research

questions: Does the climate of a secondary education institution hinder the

adoption and use of analytic tools, or are there funding/investment issues?

Would college administrators use academic analytics to help increase student

success and other managerial tasks?

The participants gave feedback regarding the term technology, and they suggested

that, because the meaning may have multiple interpretations, I should consider changing

the term to better reflect the description of academic analytics as defined in the study. I

completed the pilot study using the original interview protocol; however, I asked the

following two participants about the use of academic analytics instead of technology to

remain closer to the defined concept. The remaining two participants agreed that the use

of technology was overly broad. I made the slight wording change to the original

interview protocol to use in the regular study.

Revised Interview Protocol

1. Can you think of how you use academic analytics in the management of your

daily activities?

2. What are your primary key performance indicators/goals?

3. Describe how you manage your primary key performance indicators/goals.

4. What is your position within the organization? How long have you worked for

the organization?

5. Describe the kinds of data you use in order to manage your performance

indicators/goals.

6. What kinds of IT support do you believe would help you accomplish your

goals more effectively?

7. Describe your experience using technology to reach or exceed your

performance goals.

8. Describe any training you have received in the usage of analytics in your

workplace.

9. Do you believe the use of academic analytics in academic management is

worth the investment? Please explain.

10. If you do not use data and analytics to help manager your key performance

indicators, can you explain why not?

The pilot study allowed me to improve upon the interview protocol and discover

the length of the interviews to allow an average period for the main study interviews. The

pilot study also gave me the opportunity to make certain the laptop recording device

worked as believed. The recording laptop worked as planned.

I transcribed, verbatim, each interview and e-mailed it back to myself in Word

format. I listened again to each interview while reviewing the transcription. I only made

slight changes. I then e-mailed each transcription to the participants. I asked the

participants to read the transcription to ensure that the meaning of the interview was as

the participants wished. Each participant reviewed their transcript and added nothing else

to the transcription.

Settings

None of the participants disclosed any personal or organizational condition that

they felt might have influenced their responses. Some participants noted that a new

strategic planning cycle was occurring at the pilot study site; however, this was at the

macro level of discussion, and the use of academic analytic tools at the micro/unit level

was not at the level of discussion.

Demographics

I conducted 10 interviews for the main study. The human resources department of

College Z provided a list of personnel who met the academic management criteria

requirements. The participants represented all six campuses from academic divisions

such as Liberal Arts, Science, Business and Technology, Humanities, and Mathematics.

Data Collection

The participant size for the study was 10 academic managers based on criterion

sampling. I used criterion sampling to elicit responses from managers in an academic

higher education setting. Academic managers are persons whose key performance

indicators include student retention, faculty training and observation, managing full time

equivalent budgetary operations, curriculum reviews, and policy compliance. Criterion

sampling can be important when reviewing quality assurance endeavors and as in this

study, an extensive exploration into academic analytics (Creswell, 2012).

For the participants to remain confidential, I assigned each participant’s interview

with a code. I used a random code generator that included eight characters, upper and

lower-case, and numbers. I removed characters that look similar on screen, such as I, 1,

O, and 0. The codes were generated using randomcodegenerator.com.

I e-mailed a structured interview protocol to each participant. Included on the

protocol were the 10 open-ended questions. The questions allowed the participants to

share their insights, observations, and beliefs regarding the use of academic analytics and

technology in the management of their key performance indicators. I conducted each

interview in the office of the participant. The interviews lasted an average of 20-30

minutes.

I recorded each interview using a laptop. There was no external microphone in

use. The pilot study ensured the superior quality of the internal microphone in the laptop.

I e-mailed each interview to an online transcription service named Rev.com. The

transcription service returned the verbatim transcription in Word format within 24-48

hours. I listened to the interview while reviewing the transcription to ensure quality of the

transcribed interview. Listening to transcriptions revealed no unusual circumstances, as

the transcriptions were extremely accurate. I then e-mailed each transcribed Word

document to each respective participant. I asked the participants to review their

transcribed interviews and to identify any changes or additions they would like to append.

Participants identified no substantial changes.

Data Analysis

I reviewed the interviews the first time during the interview. I then listened to the

recorded interview again to ensure the quality of the recording and to ensure the accurate

length of the interview. I then uploaded the audio to the transcription service, Rev.com, to

have a complete verbatim transcription compiled on a Word document. This process took

an average of 24-48 hours.

Once I received the transcribed Word documented interview, I listened to the

audio interview again to compare the transcription to the interview in order to make sure

of accuracy of the transcription. I e-mailed each transcription to the respective participant

for a final examination and verification. I then imported the resulting verified

transcription in Word document format to the qualitative data analysis software

MAXQDA. I reviewed the transcriptions once again as they were imported into the

software. This review helped me recognize and triangulate the opinions and experiences

from each of the participants.

I analyzed the interviews with the organizational support of the MAXQDA

qualitative data analysis software. MAXQDA software allowed me to easily code, sort,

set up categories, and discover themes within a large amount of transcribed data. I was

able to extract phrases and key words, and was able to mark with symbols, color codes,

and emoticons, where appropriate.

The process I used to move from individual coded units to larger representative

themes was the application of the Moustakas method described by Creswell (2012).

The analysis included the following steps:

1. Listing and preliminary grouping.

2. Reduction and elimination.

3. Clustering and thematizing the invariant constituents.

4. Final identification of the invariant constituents and themes by application:

Validation.

5. Construct an individual textural description of the experience.

6. Construct an individual structural description of the experience.

7. Construct a textural-structural description of the experience

The first step, listing and preliminary grouping, was the process of listing each

expression relevant to the experience (Creswell, 2012).

Preliminary Grouping

I reviewed each transcript and denoted selective text as it was germane to the

research questions. Each selection of text was electronically marked using the qualitative

data software application MAXQDA. With MAXQDA, I was able to organize the data to

be efficient and logical.

The logical organization of the texts, as I marked them, resulted in an initial

coding of the text interviews. I collected and linked these codes to the research questions

in which they were relevant. This process allowed me to organize the textural data on an

equal basis, thus performing horizontalization of the data (Creswell, 2012)

Reduction and Elimination

I followed the initial coding of the data with an intensive review of each

individual invariant constituent to confirm validity. I reviewed the coded segments tested

to confirm the relativeness to the central question of factors that impede the

implementation of academic analytic tools in a higher education setting. This process

involved the use of two questions (Creswell, 2012):

1. What has the participant experienced in reference to the phenomenon?

2. Is it possible to abstract and label it? If so, it is a horizon of the experience.

The first question, “What has the participant experienced in reference to the

phenomenon?” involved the examination of the data to make sure the coded data linked

to the question of academic analytics in higher education. If I found the negative, I

eliminated the invariant constituent.

The next step in the process was to check if I could abstract and label the coded

data. I scrutinized the data once again to test whether the coded segments were

ambiguous, repetitive, or unclear. If the coded discrepant segments matched these

attributes, I removed them (Creswell, 2012). Because of the high organizational

capabilities of the MAXQDA software, the application aided in this step. I then reserved

the residual portions of this process and used these to build clusters.

Clustering and Theming the Codes

I grouped the residual data from the previous step into clusters or categories. I

reviewed the invariant constituents to consider similar experiences as expressed by the

participants. I examined the invariant constituents to determine whether I could unify

them into distinct significant units of experience (see list below). I further used these core

groups to group the experiences into major themes (Creswell, 2012). I identified the

categories as enumerated below.

1. How could analytics help? Is it worth the investment? (This sentiment

originated from the research questions “How can knowledge management

tools enhance the efficiency of a higher education institution? Does the

climate of a secondary education institution hinder the adoption and use of

analytic tools? Would college administrators use academic analytics to help

increase student success and other managerial tasks?”)

2. Currently using analytics (This sentiment originated from the research

question “Are academic administrators aware of how academic analytics

could help manage key performance indicators?”)

3. Why is participant not currently using analytics? (This sentiment originated

from the research question “Does the climate of a secondary education

institution hinder the adoption and use of analytic tools?”)

4. Training (This sentiment originated from the research question “Would

college administrators use academic analytics to help increase student success

and other managerial tasks?”)

5. Types of technologies used (This sentiment originated from the research

question “What types of discrete databases are currently used by academic

administrators to help the management of their perspective departments?”)

6. Types of data used (This sentiment originated from the research question

“What types of discrete databases are currently used by academic

administrators to help the management of their perspective departments?”)

7. Key performance indicators (This sentiment originated from the research

question “Are academic administrators aware of how academic analytics

could help manage key performance indicators?”)

8. Disappointments (This sentiment originated from the research questions

“What types of discrete databases are currently used by academic

administrators to help the management of their perspective departments? How

can knowledge management tools enhance the efficiency of a higher

education institution? Does the climate of a secondary education institution

hinder the adoption and use of analytic tools or is there an

investment/monetary issue? Would college administrators use academic

analytics to help increase student success and other managerial tasks?”)

Final Identification of Themes

According to Creswell (2012), the identification of the final themes of the study

requires validation of the invariant constituents to the actual transcript of the participant.

Comparing each coded invariant and the subsequent category to the transcript of each

participant helped with validation (see list below).

1. An awareness of analytics and analytic tools in higher education. The research

question that correlates to this theme is, “Are academic administrators aware

of how academic analytics could help manage key performance indicators?”

2. Technologies currently used to manage key performance indicators. The

research question that correlates to this theme is, “What types of discrete

databases do academic administrators currently use to help the management of

their perspective departments?”

3. Analytics and analytic tools to help with the management of key performance

indicators. The research question that correlates to this theme is, “How can

knowledge management tools, such as analytics and analytic tools, enhance

the efficiency of a higher education institution?”

4. Investment of analytic tools. The research question that correlates to this

theme is, “Does the climate of a secondary education institution hinder the

adoption and use of analytic tools or is there an investment/monetary issue?”

5. Current use of analytic tools. The research question that correlates to the

theme is, “Would college administrators use academic analytics to help

increase student success and other managerial tasks?”

Individual Textural Descriptions

Textural descriptions were used to describe how participants felt about and their

experience in the use of analytics in their particular management activities in higher

education. Creswell (2012) recommended the use of verbatim examples to develop

individual textural descriptions. I achieved this step by describing each participant’s

inclusive experience using analytics in their day-to-day activities of managing an

academic unit at a higher education organization.

Individual Structural Descriptions

I represented individual structural descriptions through the combination of

individual textural descriptions and imaginative variation (Creswell, 2012). I examined

the individual structural descriptions from reflections, analysis, and perspectives to arrive

at structural descriptions. I undertook this by portraying the comprehensive

understanding of each participant from the meaning of the individual coded text.

Textural-Structural Descriptions

I collected a textural-structural description using both the individual textural

descriptions and the individual structural descriptions. I developed this description, which

characterized a mixture of the combined analysis of textural and structural descriptions,

from the analyses of the meanings and elements of the individual participant’s

experiences of the use of analytics in the individual participant’s management activities.

Finally, I developed a group, or composite description combining the individual textural-

structural descriptions (Creswell, 2012).

Evidence of Trustworthiness

I verified credibility using member checks and peer review (Creswell, 2012). I

returned the transcribed interview sessions to each individual participant. I gave

participants the opportunity to read the interview session and make comments or

clarification as they saw necessary (Creswell, 2012).

I also allowed the conducting of peer review to add credibility to the study

(Creswell, 2012). I met with a peer researcher outside of College Z. I took notes during

the debriefing sessions (Creswell, 2012). We discussed methods, procedures,

understandings, and feelings to make sure that I gained an outside review of the research

(Creswell, 2012).

I used a rich description to describe the setting and the participants interview

session. This was done for readers of the research to “transfer information to other

settings and to determine whether the themes can be transferred” (Creswell, 2012, p.

209). The use of thick description aided in external validity of the study (Creswell, 2012).

This in-depth, rich description also helped me establish dependability of the research. The

exhaustive coverage will allow readers to repeat the procedures and methods used in this

study in another study, with some understanding that they may find similar results (Lietz

& Zayas, 2012).

Results

I constructed 10 open-ended questions to explore the experiences of higher

education academic managers at a community college and their thoughts and impressions

on the use and nonuse of analytics in their workplace. The participants were criterion

based drawn from a list provided from College Z (see Appendix A). Interviews took

place at the offices of the participants at a time convenient for each participant. I

transcribed each interview and analyzed the same using the qualitative software

MAXQDA.

The major themes addressed the relevant research questions of this study. These

questions are listed below:

Research Question 1. Are academic administrators aware of how academic

analytics could help manage key performance indicators? The related interview question

was, “ Can you think of how you use information technologies in the management of

your daily activities?” The theme that emerged was that there is an awareness of analytics

and analytic tools in higher education. The finding that developed was that climate and

policy may be barriers to the adoption of academic analytics at the college.

Research Question 2. What types of discrete databases do academic

administrators currently use to help the management of their perspective departments?

The related interview question was, “Describe how you manage your primary key

performance indicators/goals?” The theme that emerged was the technologies currently

used to manage key performance indicators. The finding that developed was that policy

and infrastructure may be barriers to the adoption of academic analytics at the college.

Research Question 3. How can knowledge management tools enhance the

efficiency of a higher education institution? The related interview question was, “What

kinds of IT support do you believe would help you accomplish your goals more

effectively?” The theme that emerged was determining how analytics and analytic tools

can help with the management of key performance indicators. The finding that developed

was that training issues may be a barrier to the adoption of academic analytics at the

college.

Research Question 4. Does the climate of a secondary education institution

hinder the adoption and use of analytic tools or is there an investment/monetary issue?

The related interview question was “Do you believe the use of technology in academic

management is worth the investment? Please explain.” The theme that emerged was the

investment of analytic tools. The finding that developed was that climate may be a barrier

to the adoption of academic analytics at the college.

Research Question 5. Would college administrators use academic analytics to

help increase student success and other managerial tasks? The related interview question

was, “If you do not use data and analytics to help manage your key performance

indicators, can you explain why not?” The theme that emerged was that there is no

current use of analytic tools. The finding that developed was that policy may be a barrier

to the adoption of academic analytics at the college.

I describe the major themes in the individual structural descriptions of the

participants.

Individual Structural Description for Participant mTXQRnmk

This participant had been an academic manager for two-and-a-half years.

Participant mTXQRnmk believed that he/she uses analytics on a day-to-day basis. The

participant experienced the use of running certain reports, if interested, through the

college database. This participant stated, “Most institutions of higher education these

days are very data driven, so getting the data from any kind of analytic tools, I think is

very important.” Participant mTXQRnmk used Excel spreadsheets and the college

website and databases when needed. When asked about the importance of data and

analytics, the participant responded that investment in analytic tools is worth the

investment “because it is such an evidence-based culture now, and everything’s data

driven... you have to show data.”

Individual Structural Description for Participant E6UcdPac

Participant E6UcdPac has been with the college for four years. This participant

says, “I can't think of any analytic tools that I use. I would love to have something that

could help me, to manage my daily activities.” The administrator used the college

database and website, along with e-mails, Excel, and Word documents to manage

workload. Participant E6UcdPac discussed the need for some kind of analytic tool that

would work together with each of the IT applications used on a daily basis to help

minimize errors. This participant also added, “If there’s technology out there that can do

that, that would be great.” This administrator stated, “I’ve got to be honest, this is the first

time I’ve heard more in terms of analytic tools…maybe no one has actually thought about

that.” Participant E6UcdPac believed, if available, analytic tools could help productivity.

Individual Structural Description for Participant hdt2odJ5

The participant hdt2odJ5 had been with the college for 12 years. This participant

did not use analytics to help manage daily activities, stating, “There is no software that I

use for that [management of key performance indicators], no software that I do know.”

Participant hdt2odJ5 believed that analytic tools could help with the management of

performance indicators, stating,

The concept is that then you would not have to run special reports. The reports

would be there. When you come to metrics, technology is a tool. If you don’t have

the metric systems in place, then the technology is useless. Now the college does

not really, I feel, have a system of metrics in making decisions based on metrics.

This participant relied on team members to query reports and pull them together

on Excel spreadsheets that the participant then e-mailed. The administrator believed that

“the college has to provide the leadership and the alignment” and that an adoption has to

come “from the VCCS [name of the 23 college system that the college belongs] down to

the college down to the campuses.”

Individual Structural Description for Participant pvofSD7u

The pvofSD7u participant had been with the college for seven years. This

administrator used the college database, college website, the student [administrative]

database, e-mails, and spreadsheets as the primary tools to manage key performance

indicators. The participant stated,

Most of what I do [to manage key performance indicators] is I pull reports and put

them into spreadsheets. I may privately use software purchased in simulation

models or in different, object driven models, to help me get a handle on

something, but it's not provided by the college.

The participant did speak of a system that delivered reports to users, but stated,

“The data in the system is not timely and not accurate. So if you have a dashboard that is

giving you data that was accurate as of 6 weeks ago, that could be a problem.” Participant

pvofSD7u, when asked if analytic tools would help the management of key performance

indicators, responded,

There oughta be a way I can either give you a picture or words or numbers to help

you make decisions and right now the only way to get there is to sit down and do

your own private, very labor intensive study.

Individual Structural Description for Participant Ti4eKAN8

This participant had been with the organization for 29 years, and in an academic

management position for seven years. Participant Ti4eKAN8 used primarily Word

documents, the college website, the college database which houses student and

curriculum data, and an extensive list of outside websites to manage key performance

indicators. When asked about the use of analytic tools to help manage daily activities, the

participant responded, “I don't have access to analytical tools. That would be very

helpful.” Participant Ti4eKAN8, when asked if analytic tools could help manage key

indicators, said, “If this [manually created Word document] would come up in front of

me every morning …and be updated and by order of last touch and maybe having yellow

or orange alerts,” that would be helpful. This participant added that there was not an

awareness of these tools and that, if given the opportunity, he/she would look for better

tools to help manage key goals.

Individual Structural Description for Participant 8d7RyjFS

Participant 8d7RyjFS had been with the organization for 39 years, and had been

an academic manager for eight years. This manager relied on teammates to query reports

and download them into an Excel Spreadsheet for dissemination. Tools regularly used

included Excel, Word, e-mail, college database, and the college website. This participant

expressed that analytics would not be useful to him/her because “I can trust my judgment

on things often without checking the data just because I know what's likely to happen.”

The administrator added, “I need to clarify, [I believe] technology changes rapidly that

when I did go through training, six months later I found out that my training was

obsolete.” When asked if an analytic tool could help with the management of key

performance indicators, Participant 8d7RyjFS did admit that, “There are technologies that

are useful and I can say that this printout from SIS [college database] which gives me

class by class statistics, it would take me hours, if not, weeks to do that by hand.”

Individual Structural Description for Participant cudkDAWQ

Participant cudkDAWQ had been with the organization for 17 years and in the

current administrative position for a year and a half. This participant relied on an

individual in a different department to e-mail information on an Excel spreadsheet to help

manage performance indicators. The participant mentioned, “Normally, it’s [the Excel

spreadsheet] coming to us almost too late to do much about it.” This administrator

admitted to not having analytic tools to help with management activities, stating,

We have very primitive tools to do it [manage key performance indicators]. The

fact that it’s one person sitting in an office, for the front-end part, determining

what our efficiency would be, that’s pretty primitive. Where also, that person is

very … I want to say … hard-working, very cooperative, but Excel is limited in

what it can do.

Participant cudkDAWQ discussed that this administrator received information

from a separate department, but the information is 6 months old, and the department

querying the information is “centralized” and “…a very closed part of our organization.

We have a very difficult time to get information from them [the centralized data

controlled department] as well…frequently the requests that we make are not honored.”

This participant believed that analytic tools would be of great use for the management of

key indicators, but stated that bureaucracy was a barrier in the adoption of any analysis or

analytic tools.

Individual Structural Description for Participant rn73xv8V

Participant rn73xv8V had been with the college for 16 years. This participant

depended on reports queried from a separate department. Using these reports, the

participant extracted information and built formulas in Excel spreadsheets and then e-

mailed them for dissemination. The participant used key college databases and websites

to gather information. Regarding the use of analytic tools to help manage performance

indicators, participant rn73xv8V believed,

Education is becoming so complex and so reliant upon databased decision-making

that anything that will screen the plethora of resources and the sea of information

that's out there has got to be helpful. We don't have time to spend on it by looking

at a lot of different databases or other resources.

The participant believed that the college has not adopted analytic tools for its

academic managers because there is very little knowledge of what is available. The

participant added that the use of analytics, a program that could gather important data that

is used to perform tasks, and push it visually (dashboard on computer), and update

constantly, would be useful, but only if it saved time.

Individual Structural Description for Participant stL64BGZ

Participant stL64BGZ had been with the organization for two years. This

participant looked at several different databases and websites, within and outside of the

college, to manage key performance indicators. Pertaining to the use of analytic tools in

the workplace, participant stL64BGZ stated “There are some new programs out that help

you visualize large amounts of data...they allow you to cut data vertically, horizontally,

diagonally, in three dimensions,” but that the organization did not currently have access

to any tools as such. The participant believed that the use of analytic tools for academic

managers is “…more than worth the investment. If you are not data driven, forget it. You

can't run a college with a large amount of public dollars on anecdotes.” This participant

believed that bureaucracy and size of the college prevented the adoption of analytic tools.

Individual Structural Description for Participant

JMLZXbUh

Participant JMLZXbUh had been with the college for 31 years. This participant

used, along with the college website and databases, a book of reports published through a

different department. The participant believed that the information in the book was a

“wonderful resource,” but that it was a static report and its information was usually a few

years old by time of publication. When asked if an analytic tool would be beneficial in

the management of daily goals and key performance indicators, participant JMLZXbUh

stated,

Yes, absolutely I do because I think that in Higher Ed we have a tendency to

make decisions based on our gut, and that's just wrong. A lot of times we aren't

aware that there are problems until they are so significant that we can no longer

ignore them. Had we been looking at things, had it been easy for us to study data

from day to day, or at least from month to month, we would have noticed there

was a problem ahead of time and maybe we could have avoided it.

Participant JMLZXbUh believed that the college did not adopt analytic tools

because, “There's a sense that because people are likely to misunderstand data it's better

for them not to have them at all.”

Textural-Structural Descriptions: “Themes”

I developed this description, which characterizes a mixture of the combined

analysis of textural and structural descriptions, from the analysis of the meanings and

elements of the individual participant’s experiences of the use of analytics in the

individual participant’s management activities. In the following sections, I describe the

major themes of the study through the perceptions of the participants. I also identify the

major themes that developed from the respective research questions, after which the

themes are described in detail.

There is an awareness of analytics and analytic tools in higher education. I

developed this theme from the research question: Are academic administrators aware of

how academic analytics could help manage key performance indicators? Only two

participants were not aware of analytics, or what analytic tools could do for higher

education organizations. Eight participants stated they knew of analytic tools, and had

seen analytic tools in other venues. They described how they had seen dashboards to

control for their cell phone usage, and how they knew companies like Amazon used

analytics to track purchases and give purchasing advice to customers. The participants

were aware that they could use analytics in education to help track student achievement,

student retention, and participation, and correlate information for better decision making.

Participant mTXQRnmk believed, “Most institutions of higher education these days are

very data driven, so getting the data from any kind of the analytics tools, I think is very

important.” Participants stL64BGZ and cudkDAWQ both had extensive awareness of

analytic tools. Participant stL64BGZ discussed analytic tools that can help “visualize

[data] and you can do ‘what-if’ scenarios.” Participant cudkDAWQ stated, regarding

analytic tools, “that these kinds of tools I know are available [and] could be available.” I

found that the academic managers were aware of how analytics could help them in their

day-to-day tasks, as well as their key performance indicators. The participants gave

instances whereby they believed an analytic tool such as a visualization dashboard could

greatly help them achieve success.

Technologies currently used to manage key performance indicators. I

developed this theme from the research question: What types of discrete databases do

academic administrators currently use to help the management of their perspective

departments? The top technology mentioned by most participants was the use of Excel

spreadsheets. Most academic managers interviewed used various databases and websites

to gather information. They would then transfer the information onto a spreadsheet, and

then e-mail it to team members for further dissemination. Participant pvofSD7u stated,

“Most of what I do is I pull reports and put them into spreadsheets.”

Various websites and databases were the next most widely used technology

employed by the administrators. Participant rn73xv8V stated using “Probably a half

dozen or so [websites], most are budget. Others are enrollment or student information

databases…a lot of the information that I need is found in the student database.”

Participant Ti4eKAN8 mentioned the use of several websites to collect information for

one situation, and Participant stL64BGZ mentioned the use of five websites to collect

information to follow trends. None of the participants said that they had a dashboard that

collected and correlated information for them in real-time and displayed it visually so that

the academic managers could have immediate, up to date information with alerts that aid

in decision making.

How analytics and analytic tools can help with the management of key

performance indicators. I developed this theme from the research question: How can

knowledge management tools enhance the efficiency of a higher education institution?

The participants felt that they could use analytic tools, something similar to a dashboard,

that when they came into work each morning, the dashboard would collect the

information needed, update and correlate the information, and place the needed

information in a visual graph that would enhance the understanding of the information.

Participants discussed an example of registration time for semesters. Currently, a dean

(academic manager) needs to manually check each section, each class, continually as

students register for a particular class. The dean (academic manager) must closely keep

watching each class to determine class population, and whether there is a possibility to

open another section when the class reaches its maximum. If the academic manager, in

this example, had the use of an analytic tool such as a visualized dashboard, this

information would be pulled continuously and placed in a graph of sorts, and, as updated

constantly, the graph could track course registration and give alerts, send e-mails, or

change colors as the course reached different levels. This would help the academic

manager better control the registration process, and would have the information needed to

make timely decisions.

Participant JMLZXbUh understood how the use of an analytic tool could help

with the management of key performance indicators, asserting,

It would be helpful to use those technologies to really make it so that it was very

specific to a program, and a program within a program like a specialization. Not

just lumping everything together. It would also be good to actually use it to

analyze. To pull two things together.

Participant rn73xv8V stated,

If there was something that could just let me put in keywords and I could see key

things so that I don't have to read the big three and I can focus on the ones that I

really am concerned about. That's where I would prefer to spend my time rather

than reading through a hundred page report and trying to figure out what's where.

Participants also saw the use of analytic tools to help with decision-making.

Participant cudkDAWQ stated,

Sometimes we make a decision not to do something because we’ve done it before,

and it didn’t seem to work. Whether if we actually looked at that … because as I

well know, what you think you know might not necessarily be the case. The data

might show something else, something that we weren’t aware of. Because we’ve

never had those tools, I can’t say that we’ve been there.

For participant pvofSD7u, receiving the data would also help decisions, “If you give me

the right information, I'll make better decisions.”

Other participants stated the use of an analytic tool would perhaps provide real-

time information. Participant JMLZXbUh discussed the use of an outdated report, stating,

“Of course one of the problems is, this is two years old already by the time the fact book

comes out.” Participant cudkDAWQ mentioned that information gathered is sometimes

not very timely, stating, “It’s six months old by the time we get it, if even that.”

Participants said they would use the tools if given an opportunity.

Investment of analytic tools. I developed this theme from the research question:

Does the climate of a secondary education institution hinder the adoption and use of

analytic tools or is there an investment/monetary issue? Participant E6UcdPac, when

asked if the investment of an analytic tool would be a good idea, stated,

I would say yes. If there's an analytic tool that can say, well take my job

description, those key performances, those key tasks that are always evolving, that

are always there. If there's something out there that can say…Well, this task here,

which is faculty evaluations is coming up so you need to get that going or moving

along or the schedule deadline is almost there. Something along those lines, I

think would be worthwhile, it would be a good investment.

The participant believed that analytics could help track student data points, as well as

faculty schedules and then correlate both to obtain a clearer understanding of faculty

performance. Participant JMLZXbUh stated, “Yes, absolutely I do because I think that in

Higher Ed we have a tendency to make decisions based on our gut, and that's just

wrong,” when asked if analytics would be worth the investment.

Participant mTXQRnmk also believed in the investment of analytic tools, stating,

I do, and that's particularly because it is such an evidence-based culture

now…when you want things, you have to show its data. You just can't say,

‘Because I feel like it. I just really feel it's important.’ You've got to show why.

The participants generally agreed that there was not a funding issue involved with

the adoption of analytics. They believed that analytics would certainly be worth the

college’s investment in both time in training and monetary costs. Cost of analytic tools

was not a barrier, and an analytic tool would be worth the investment.

No current use of analytic tools. I developed this theme from the research

question: Would college administrators use academic analytics to help increase student

success and other managerial tasks? Nine of the 10 participants stated that they did not

use analytic tools in the management of their key performance activities. Only participant

mTXQRnmk stated,

Yes. I think we do, on a day-to-day basis, we definitely use the analytic tools. We

don't have, probably, access to some of the more robust, but I think through SIS

system [this is the college data base system that holds student, course and

curriculum information], we can run certain reports ourselves and if we're

interested in certain trends, then we're able to get that.

Participants would use academic analytics to help with their daily goals and

activities if they had the tools at their disposal. However, no academic manager had

access to an analytic tool such as a dashboard, to help with their key performance goals.

Composite Description “Overall General Findings”

The composite description is a synthesis of the descriptions entirety. Creswell

(2010) explained this description as the essence of what the participants experienced. The

composite description addresses the overall general question, namely: What factors

impede the implementation of academic analytic tools in a higher education setting? I

used the five emergent themes to develop the findings that addressed this overall

question.

The findings that developed from the five themes were that climate, internal

policy, training, and possible infrastructure issues of the college hindered the adoption of

academic analytics. Participants generally believed that there was no use of analytic tools

because a discrete, centralized department within the college organization kept

information separate or controlled. Participant stL64BGZ mentioned that it would be

advantageous to have data accessible, “instead of having to use institutional research as

the only source of all-data.” This feeling was expressed by participant cudkDAWQ, who

stated, “I have to mention that we also have an office of institutional research here at the

college, who can provide a lot of analytic information, but it’s a centralized organization,

and it’s a very closed part of our organization.” Building on that idea, the participant

added, “I also think that, politically, we don’t have access to this information because of

that central organization.” Participant JMLZXbUh shared a reason for centralized data by

stating,

Partly because people who run institutional research want to be sure that data are

interpreted correctly, and to be sure that they really are cleaned up before people

start using them. If you make stuff available in real time, then there are chances

for error, and sometimes people don't understand that you're looking at a

snapshot. There's a sense that because people are likely to misunderstand data it's

better for them not to have them at all, and not every administrator likes numbers.

Participants also noted that there needed to be a more shared environment before

adopting an analytic tool to help manage key performances. Participant hdt2odJ5

expressed feelings that the college did not share information between departments and

campuses. The participant stated,

I say to have those tools without alignment is not going to do any good because

for example when we do the annual college planning, each campus develops their

annual plans. Each campus does it in a vacuum. There is no way to see how the

college is performing as a system.

Participant pvofSD7u voiced a similar concern stating, “We need people who agree on

the shared vision and the big picture. We're not there.”

The climate of the college emerged as another barrier to the adoption of an

analytic tool to assist academic managers at this college. Participant stL64BGZ stated,

The complexity of what we do is far beyond anything ... Tidewater Community

College approaches it, but not in the way we do. The amount of data we have to

deal with, the complexity of what we deal with, does not fit or is not needed at

Mount Empire Community College or Eastern Shore Community College with

four hundred students or whatever. They can get away with a lot of manual stuff.

Participant cudkDAWQ mimicked this idea, asserting,

I think it’s because we’re a very large bureaucracy, and we move slowly. I think

that, despite the fact that these kinds of tools I know are available, could be

available, getting around to using them takes a long time for someone to take

responsibility and get it in place and, therefore, get it to us.

I interpreted that the climate and internal policy of the college may be barriers in

the adoption of analytic tools for academic management use. The interviews revealed that

the academic managers of the college did not have access to many institutional data that

they felt would be beneficial in their daily activities. The academic managers stated that

they had to ask for data from an internal department, and that if they received the data

they asked for, it was usually not in a timely manner. Internal policy and the climate of

the college emerged as possible barriers to adoption.

I discovered that participants also thought training could be a barrier to the

adoption of analytic tools. Participants did not have time to receive training in another

technology or innovation, and that if they were to use analytics in their daily tasks, it

would have to be an easy system to learn. I interpreted that the academic managers did

not have the time to invest in a new technology that would be cumbersome or complex to

learn or to use.

I also interpreted possible infrastructure issues to be a barrier to analytic adoption.

Although not entirely within the scope of this study, I was lead to interpret that the many

discrete databases used by the varying academic managers, and the different websites

used to gather information, may have presented technology problems in adopting a tool

that would unite all the systems.

Summary

The purpose of this study was to explore the factors that inhibited higher

educational institutions in their adoption of proven analytic tools to help improve

management of key performance indicators. This inquiry provided an enriched

understanding of barriers to analytic adoption in College Z. I conducted open-ended

structured interviews with 10 academic managers in the data collection phase.

The data were audio recorded and transcribed. Each respective participant

reviewed their transcription to ensure validity and credibility. The transcriptions were

then, after verification from the respective participants, uploaded to the qualitative data

analysis software program MAXADQ for assistance with organization at the granular

levels.

I used the method described by Creswell (2012) to analyze the data. The analysis

included: (a) listing and preliminary grouping of the data; (b) the reduction and

elimination of the data; (c) clustering and thematizing, or listing significant statements;

(d) final identification of the invariant constituents, or the themes; (e) writing of

individual textural descriptions; (f) construction of individual structural descriptions; and

(g) composite synthesis, or overall findings (Creswell, 2012).

I accomplished this phenomenological research study using 10 interview

questions, and from the perceptions and experiences of the participants gathered during

the data collection phase and the subsequent analysis, from which I identified five major

themes. The five themes were (a) an awareness of analytics and analytic tools in higher

education, (b) technologies currently used to manage key performance indicators, (c)

analytics and analytic tools to help with the management of key performance indicators,

(d) investment of analytic tools and, (e) current use of analytic tools.

Through the five major themes, I was able to discover answers to the five guiding

research questions. In addition, my review of the composite synthesis provided answers

to the general research question: What factors impede the implementation of academic

analytic tools in a higher education setting? I discovered that participants at College Z

believed that restricted climate, a policy, training, and possible infrastructure issues were

all factors that hindered the adoption of academic analytics at their organization.

In Chapter 4, I provided a detailed description of the pilot study conducted, the

setting and demographics of the study, the data collection and analysis, and finally, a

granular description of the results of the study. Chapter 5 contains a synopsis of the

study, interpretation of the findings, limitations of the study, recommendations, and

implications of the study. In Chapter 5, I also provided the positive social change and the

key essence of the study.

100

Chapter 5: Conclusions and Recommendations

Introduction

In Chapter 5, I include a summary of the nature of the study and its purpose. This

chapter also includes interpretations of the themes from Chapter 4 and how those themes

relate to the literature review conducted in Chapter 2. I will discuss the limitations of the

study and describe recommendations for further research and the implication for positive

social change. The order of Chapter 5 is as follows: summary of key findings,

interpretation of the findings, limitations of the study, recommendations, implications,

and the conclusion of the study.

Summary of the Findings

In Chapter 1, I introduced the concept of analytics. The use of business

intelligence tools, such as analytics, has helped increase the overall growth of business

operations including customer retention, return on investments, profit structure, and

business total value (Minkara, 2010). These successes are linked to the use of analytics in

retail, financial, manufacturing, and telecommunications industries (Seng & Chen, 2010).

Researchers have shown that the use of analytic tools in higher education institutions has

helped increase student retention, provide transparency of financial reporting, improve

management of space, safety and security, and provide visualization of operations in true-

time (Baepler & Murdoch, 2010; Bichsel, 2012). However, colleges and universities have

not analyzed these data points to help make effective decisions and data-driven forecasts

(Baepler & Murdoch, 2010; Dawson et al., 2010).

101

The purpose of this study was to explore the factors that prevent institutions of

higher education from adopting analytic tools that would enable leadership and

management to analyze and use data for decisions, planning, and managing operations. I

designed this phenomenological study to understand and explore the experiences of

individuals managing departments in a community college setting, their experiences in

using or not using analytics, and the meaning behind their perceptions of analytic tools. I

collected data from interviews of academic personnel and academic managers in a

college setting. I analyzed the data using an analytic approach as described by Creswell

(2012). I used a qualitative data management software tool to help granularly organize

and compile the data.

Five themes emerged from the study, namely (a) an awareness of analytics and

analytic tools in higher education, (b) technologies currently used to manage key

performance indicators, (c) analytics and analytic tools to help with the management of

key performance indicators, (d) investment of analytic tools, and (e) current use of

analytic tools. Through the five major themes, I found answers to the five guiding

research questions.

An Awareness of Analytics and Analytic Tools in Higher Education

This theme related to the following research question: Are academic

administrators aware of how academic analytics could help manage key performance

indicators? I found that only two participants were not aware of analytics or what analytic

tools could do for higher education organizations. Eight participants stated that they knew

of analytic tools and had seen analytic tools in other venues. The participants were all

102

positive about the idea and concept of analytic tools. The key impressions from this

theme led me to the interpretation that the participants, if there was an analytic tool

available to them, would use it to help manage their key performance indicators.

However, climate and policy factors within the college did not allow academic managers

access to robust analytic tools.

Technologies Currently Used to Manage Key Performance Indicators

This theme related to the following research question: What types of discrete

databases do academic administrators currently use to help the management of their

perspective departments? I found that academic managers used up to five or more

databases, information systems, and websites to gather information needed to perform

their tasks. The participants mentioned having to collect various data points and then

transfer them into an Excel spreadsheet for easier use. This theme led me to interpret that

there could be infrastructure issues that presented a barrier to the adoption of analytics.

Analytics and Analytic Tools to Help with the Management of Key Performance

Indicators

This theme related to the following research question: How could knowledge

management tools, such as analytics, enhance the efficiency of a higher education

institution? The participants in the study felt that they could use an analytic tool to “pull

two things together” for a better analysis. Another participant saw that analytics could

help by sorting, combining, separating, and the research capabilities that a robust analytic

tool could provide. Generally, all participants stated that they would use analytic tools if

103

they were available. The participants mentioned that the college was complex and that

information was guarded and slow to its destination point. One participant stated,

I think it’s because we’re a very large bureaucracy, and we move slowly. I think

that, despite the fact that these kinds of tools I know are available, could be

available, getting around to using them takes a long time for someone to take

responsibility and get it in place and, therefore, get it to us.

These notions and feelings expressed by the participants led me to interpret that the

climate of the college presents a barrier to adoption.

Investment of Analytic Tools

This theme related to the following research question: Does the climate of a

secondary education institution hinder the adoption and use of analytic tools or is there an

investment/monetary issue? Participants agreed that they would use an analytic tool such

as a dashboard, and that the cost of an analytic program or service would be worth the

investment. Participant mTXQRnmk discussed whether the value of analytic tools was

worth the investment, stating,

I do, and that's particularly because it is such an evidence-based culture

now…when you want things, you have to show its data. You just can't say,

‘Because I feel like it. I just really feel it's important.’ You've got to show why.

None of the participants believed that the cost of an analytic tool was a barrier for

adoption. However, participants did mention that training, the complexity of a new

technology tool, and the time it took to learn the new tool would be of concern to their

already full daily agendas. Participant rn73xv8V mentioned,

104

If it is intuitive, it's wonderful. If the training is not extensive to the point that it

takes a half dozen steps to do something I can do somewhere else in two or three,

even if I've got to do it several times with several different databases. Time here is

more than anything else, the most precious commodity that's here and the one that

there's not enough of.

This theme led me to interpret that training in new advanced technologies, such as

analytics, may be an adoption barrier.

Current use of Analytic Tools

This theme related to the following research question: Would college

administrators use academic analytics to help increase student success and other

managerial tasks? Participants said that they would use analytics to help them with their

daily tasks, such as tracking student achievement rates and correlating those rates to

student participation and engagement, and faculty and curriculum changes. The

participants said that they did not have access to any sort of analytics or analytic tool that

they could use to manage their key performance indicators, such as student retention.

They mentioned that a central department houses data and current information, and that

they did not have direct access to the raw data. Participants mentioned, “Politically, we

don’t have access to this information because of that central organization.” Another

participant identified a possible reason why there was a lack of greater access to the data.

Participant JMLZXbUh stated,

Partly because people who run institutional research want to be sure that data are

interpreted correctly, and to be sure that they really are cleaned up before people

105

start using them. If you make stuff available in real time, then there are chances

for error, and also sometimes people don't understand that you're looking at a

snapshot. There's a sense that because people are likely to misunderstand data it's

better for them not to have them at all, and not every administrator likes numbers.

These opinions expressed by the participants led me to interpret that an unwritten

institutional policy impeded adoption of analytic tools.

I further interpreted the above themes to answer the main research question,

“What factors impede the implementation of academic analytic tools in a higher

education setting?” I discovered that participants at College Z believed that the climate

(organizational bureaucracy), policy (restricted organizational data), training, and the

possibility of infrastructural issues were all factors that hindered the adoption of

academic analytics at their organization.

Interpretation of the Findings

I based this phenomenological study on Metcalfe’s (2010) theory of academic

capitalism and the use of information technology to manage key performance indicators

in a higher educational setting.

Through the literature review performed in Chapter 2, I showed that there was not

widespread use of analytics in higher education institutions, even after studies indicated

the positive results of usage (Bichsel, 2012; Dawson et al., 2010). My review also

suggested that there were limited studies conducted as to why higher education

institutions did not adopt analytics. Bichsel (2012) suggested culture, policy, and

infrastructure as possible barriers to adoption of analytic tools in higher education.

106

Similarly, Lavalle et al. (2011) suggested the biggest impediments for analytics adoption

in corporations were to be found in the culture and the climate of a company. The

findings of this study confirmed Bischel’s suggestion, and extended this belief to include

impressions on training for the use of an analytic tool.

Climate

The themes that helped me interpret that the climate of the community college is a

barrier to adoption included (a) there is an awareness of analytics and analytic tools in

higher education, and (b) The investment of analytic tools. Participants discussed the

climate of the organization. Participants generally agreed that the size of the system

sometimes caused delays and administrative inflexibility regarding information sharing.

Another participant shared that the college was lacking alignment and repeatable

processes common throughout all the campuses of College Z. Bischel would agree that

bureaucracy and the culture of a college could prevent the necessary shared vision for the

adoption of an analytic tool (Bischel, 2012).

Policy

The themes that helped me interpret that the policy of the community college was

a barrier to adoption included (a) technologies currently used to manage key performance

indicators, and (b) there is an awareness of analytics and analytic tools in higher

education. The participants discussed policy when they mentioned access to data and the

current technologies they use. Participants shared that they did not have direct access to

institutional data, and that when they did receive reports generated for them, these reports

were outdated and static. The participants agreed that the old reports were extremely

107

useful for histories, but that they did not receive the data on a needs basis or in a timely

manner. Participant JMLZXbUh perhaps shed light on the closed access to institutional

data that the participants felt they had little access to. The participant stated that

institutions kept data for long periods and in a closed access manner so that the institution

could scrub data for inaccuracies and then redistribute data appropriately. This policy

mirrored Bischel’s (2012) impression that college policies could be a barrier to analytic

adoption.

Infrastructure

The theme that helped me interpret that the infrastructure of the community

college is a barrier to adoption was the technologies currently used to manage key

performance indicators. The participants often mentioned the different siloed sources of

information they used to manage their daily tasks. Most participants discussed how they

accessed the college website and at least two different databases to collect data. They, at

some point, transferred this data to a spreadsheet and then used meetings or e-mail to

disseminate the collected information. Participant rn73xv8V mentioned using six

different databases, including budgetary and student databases. This participant

mentioned receiving training on conversion software to convert information in websites

and databases into a spreadsheet format for easier dissemination and sorting capabilities.

Bischel (2012) suggested the infrastructure of a college could be a barrier to analytic

adoption. It was outside of the scope of this study to investigate the supporting

infrastructure of the flow and processing of data within the organization; however, the

108

processes in which academic managers acquire and use data at this college were

disconnected and time consuming.

Training

The theme that helped me interpret that the infrastructure of the community

college is a barrier to adoption was how analytics and analytic tools can help with the

management of key performance indicators. One concept Bischel’s (2012) study did not

explore was training issues in the use of analytic tools. The participants in the current

study felt that the use of analytic tools would be useful and helpful in the management of

their key performance indicators. However, the participants stated that training could be

an issue if an analytic tool was too complicated or took too much time to learn.

Participant cudkDAWQ suggested that improvement was needed, but that if the tool took

too long to learn, was difficult, or the training was deficient, that most academic

managers would not use it.

Limitations of the Study

There were several limitations to this study. I used a small sample and single

setting for this study. I only interviewed academic managers who had key performance

indicators to include student retention, faculty training and observation, the management

full time equivalent of budgetary operations, curriculum reviews, and policy compliance.

I did this to gather the perceptions of persons whose daily tasks involved the usage of

student data, curriculum, and budgetary data. There were other managers involved with

the operations of College Z who were not included in this study such as police officers,

109

facilities management, budgetary control, and financial aid representatives, as they fell

outside the scope of this study.

I did not interview the department that housed and controlled a significant portion

of the data and the reports that the participants in the study mentioned. In this study, I

focused on managers who used the data in the management of their tasks and

performance, not managers who gathered, cleaned, and packaged the data to give to the

front line academic managers. However, this could affect the results due to the

importance of the data controlled by the research department.

Due to the need for a criterion sample and the time available with academic

administrators, I used interviews as the primary method of gathering information. To

control for these limitations, I conducted member checks of transcriptions and a peer

review of results.

Another limitation was the role of the researcher. I was the sole data collector,

analyzer, and interpreter of the interview materials. Because of this, I needed to be aware

of biases, beliefs, and preconceptions. Due to this heightened awareness, I believe that I

did not affect the results of this phenomenological study.

Recommendations

Based on the literature review found in Chapter 2, the use of analytics to help

drive decisions and meet key performance indicators in higher education institutions has

been proven to be effective (Barneveld, 2012). However, colleges and universities

continue to be slow to adopt academic analytics (Dawson, 2010). The purpose of this

110

study was to gain an understanding of the factors that impeded the implementation and

use of an academic analytic tool in a community college.

I discovered that participants at College Z believed that the organizational

bureaucracy (climate), restricted organizational data (policy), training, and the possibility

of infrastructural issues were all factors that hindered the adoption of academic analytics

at their organization. These barriers mirrored Bischel’s (2012) study that suggested

culture, infrastructure, and policy may be barriers to adoption of an analytic tool in a

higher education organization.

Goldstein (2005) reported that educational institutions that adopted academic

analytics to improve institutional decision-making improved in the functional areas of

student retention and financial results. College Z had no analytic tools to aid academic

managers in their key performance indicators. College Z should consider addressing the

practice of departmental IT-generated reports that they then disseminate to the academic

managers (Ravishanker, 2011).

Participants of this study mentioned that they sometimes relied on reports

generated for them. Participant cudkDAWQ stated that in one instance, the participant

requested a certain report; however, the participant received no response or the data

requested. Participant stL64BGZ mentioned that all data derived from another source

(separate department), and that it was sometimes difficult to acquire reports and data in a

timely manner. This is a concern and the college leadership should address this

perception.

111

Future Research

The first research question that was asked was the following: Are academic

administrators aware of how academic analytics could help manage key performance

indicators? The participants stated they knew of analytic tools, and had seen analytic

tools in other venues such as companies like Amazon and Netflix. Participants stated that

they would use some sort of analytic tool, such as a dashboard, to help them with their

key tasks. Future research could explore how effective the use of a dashboard is in

helping to correlate student failure rates and faculty training.

The second research question that was asked was the following: What types of

discrete databases do academic administrators currently use to help the management of

their perspective departments? The participants all stated that they used the college

student database, departmental-created discrete databases, and the college website. They

stated that in most cases, they e-mailed Excel spreadsheets and Word documents

throughout their department for communication and to conduct procedural work. A future

study could determine the efficiency of using an analytic tool to manage work, as

opposed to the use of discrete databases and e-mailing spreadsheets for collecting and

storage of important data points.

The third research question that was asked was the following: How could

knowledge management tools, such as analytics, enhance the efficiency of a higher

education institution? Participants stated that they would use analytics to help with

student retention. Participants agreed that timely information was needed, especially

during critical periods such as student registration. They believed they would be more

112

effective managers if they had a program that could manage current data in a way that

was user friendly and required little training. Future researchers could conduct an

efficiency study to study whether academic managers are more efficient at achieving their

key performance indicators when they have analytic tools.

The fourth research question that was asked was the following: Does the climate

of a secondary education institution hinder the adoption and use of analytic tools or is

there an investment/monetary issue? The participants did not believe that there was a

monetary issue in the way of adoption, but they mentioned college policies, bureaucracy,

and infrastructure as possible barriers. A future study could explore barriers at a four-year

university and determine whether the themes are similar to the themes at a two-year

community college.

The fifth research question that was asked was the following: Would college

administrators use academic analytics to help increase student success and other

managerial tasks? All participants agreed that they would use academic analytics to help

increase student success. The participants felt that they, with the use of a tool such as a

visualization dashboard, would be able to better complete their goals. They agreed that

the college database is a helpful tool, but to have on-time current information, pulled

from the various pushed to them in a visual format, would be valuable. Further research

could explore IT adoptions, and whether managers used the new tools available to them

when adopting an analytic tool.

Additionally, a future study could include managers in other capacities of the

college. Future researchers could study the separate department that controlled

113

institutional data, and explore the policies and procedures regarding data accessibility for

the college. Future researchers could also compare colleges of different sizes, colleges

that are private and colleges that are profit based, and rural versus urban colleges. A

study could explore how these diverse colleges use analytics and how their unique

situation affects the use of their data. I identified a need for a future study of how students

use analytics to control and shape their college experience.

Implications

Researchers have shown that colleges and universities are collecting significant

amounts of institutional data in the areas of enrollment, finance and budget, student

progress, research, and learning management. These data are piecemeal and not widely or

easily available to all departments and administrators. Rather than having the data pushed

to them, administrators must decide what data points are most salient, know where to

access that data, pull it, potentially from multiple sources, and then analyze it. There is a

need for colleges and universities to integrate data into one accessible package where

researchers can analyzed and use the data to make proactive decisions with significant

impact (Bichsel, 2012). Researchers can customize a single digital interface to provide as

much or as little data as the user needs to more effectively manage departmental tasks

and outcomes, to supply this information on a real-time basis. Academic analytics

transforms colleges and universities in terms of increased student retention and

graduation rates, improved student access, more effective utilization of human and capital

resources, and provide answers and decision support based on data-driven evidence

(Bichsel, 2012).

114

College Z’s mission for social change drove the need for such analytics, as the

college strives to significantly increase the graduation rates of the students. By utilizing

analytical tools and real-time data pulled from multiple sources, academic managers at

the college may be able to more effectively analyze student data in a timely manner,

which will allow them to proactively assist students who are in academic distress or

students who are in danger or dropping out of school.

When colleges and universities use existing data to manage key performance

indicators more effectively, they save money, decrease time from enrollment to

graduation, and have more transparent ways to track successes and improve forecasts

(Dziuban et al., 2012; Smith et al., 2011).

I designed this phenomenological study to explore why College Z has not yet

adopted an academic analytic platform to manage key performance indicators to reach

and exceed its mission. From analysis of interviews, the major themes of (a) an

awareness of analytics and analytic tools in higher education, (b) technologies currently

used to manage key performance indicators, (c) analytics and analytic tools to help with

the management of key performance indicators, (d) investment of analytic tools and, (e)

current use of analytic tools were discovered. Further analysis resulted in the

interpretation that participants at College Z believed that the organizational climate,

policy, training, and the possibility of infrastructural issues were all factors that hindered

the adoption of academic analytics at their organization. The implications of the findings

of this study indicate that for College Z to realize its goal to positively affect the students

and potential employers in the region, it will need to be more efficient, provide a higher

115

quality of services, and be able to measure outcomes. The participants in this study

believed that the successful obtainment of College Z’s mission to increase retention rates

of students, to increase graduation and transfer rates, to increase career placement rates,

and to increase enrollment rates of underrepresented populations is achievable through

easier access and better use of their existing data.

Positive Social Change

Figure 3 illustrates the potential impact for positive social change at the societal

level, the organizational level, and individual level if College Z were to adopt analytic

tools. Researchers have proven that analytic tools help colleges sort interest level data of

perspective students, target under-served populations, and help with materials collection

for enrollment purposes. Analytic tools can better empower career and academic advisors

as they search for student employment opportunities, and help merge curriculum to

industry needs.

Organizationally, the adoption of analytic tools allows academic managers to

track student success and student needs in a timely manner. Academic counselors, faculty

members, and academic managers may instantly see when a student experiences a gap in

success, attendance, resource management, or other retention factors. Analytics can help

College Z capitalize on community partnerships and alumni contributions, making a

positive impact in serving the community and alumni using better data to strategize

where there is greater need for workforce development activities.

Individual students at the college would perhaps be the most impacted by analytic

tools. Students could analyze their own progress, and they could benchmark their

116

progress against other students’ progress and course goals and behaviors. Students could

use analytics tools to better plan their educational experience, search transfer locations,

seek financial aid prospects, and plan, research, and discover future employment

opportunities, including areas of which they would not otherwise be aware. Analytics

could be a liaison for current students and alumni to share like goals, employment

possibilities, and mutual interests. Analytics could provide a richer college experience

that keeps students engaged through their entire school life. Figure 3 demonstrates how

each sector could benefit from the use of academic analytics.

Figure 3. The organizational, individual, and societal impact of the use of academic

analytics in higher education.

Organizational:

increased student

access & success,

improved excellence

in teaching, increased

revenue and

community growth

Individual: Student

Success, Improved

Leadership, improved

decision making

abilities

Societal: Increased

College Access to

under-represented

populations, opened

doors to higher-wage

employment, career

advancement for low

income populations

117

Methodological Implications

I could have explored a different method of analysis for this research. In this

study, I used Creswell (2012) as a structure for the analysis of the data. The strengths of

using this framework included reaching an in-depth interpretation of the participants’

experiences. There are other frameworks for qualitative studies, and additional research

using a different framework may add to the richness of the data interpretation. In

addition, the availability of a larger subject pool would allow for the further testing and

refinement of the survey tool. This could lead to quantitative studies to establish which

factor(s) had the most impact as a barrier or barriers to implementation of academic

analytics. By conducting additional quantitative studies, future researchers could explore

a comparison of colleges and the use, nonuse, and barriers to adoption.

Academic Practice

A recommendation stemming from this study is that College Z could work to

build a more collaborative functioning environment between the separate data-driven

department and the college’s academic managers. College Z should evaluate how the

climate and data policy affects the management of the college as a whole. College Z

should also act to leverage the immense functional knowledge base of its software

developers, IT engineers, and data analysts by aligning them in teams to collaborate more

closely with the functional managers who need timely and dynamic data to perform their

key duties.

An additional recommendation stems from the possible infrastructure issues the

college may have. Participants of this study additionally mentioned the different

118

databases and websites they had to access to collect needed data. Participant rn73xv8V

said that much data used are not in a transferrable format. Participants also revealed that

collected data are and kept in different, non-centralized repositories. College Z needs to

evaluate local databases, sometimes labeled “shadow” systems (Ravishanker, 2011), as

well as separate discrete databases. Establishing one central repository that would bring

all sources together and that was easily accessible could have a tremendous institutional

impact. This central repository would help define institutional data across departments

and campuses and could afford academic managers innovative and fresh perspectives.

Participants identified training as a possible barrier to adoption. College Z needs

to assess the skills gap of trained academic administrators with experience in analytics.

Professional development in analytics, with an emphasis in the functional ranks

especially, and at all levels generally, would increase the awareness of analytics and the

use of analytics when and where available.

In closing, I considered Vidal’s (2014) discourse on a worldview and,

specifically, to where we are progressing as a society and as an intelligent life. In this

study, I provided an insight as to why academic managers at College Z did not use

proven analytic tools to help its student population prepare for and gain access to a better

life. Vidal noted that, based on values and provided with new information and new

answers, individuals have choices to make to move forward. The mission and the goals of

College Z prove that the college values its students. The next step for the college is to be

willing to work to diminish the barriers that hold them back from providing managers

with powerful analytic tools to ensure student success.

119

Conclusion

The results of this study support claims of prior researchers that cost was not a

significant barrier, but that factors internal to the organization acted as barriers to

adoption. The analysis of the data determined five themes that suggested academic

managers were aware of academic analytic tools, but these tools were not in wide use.

Further analysis of these themes revealed that policy, climate, infrastructure, and training

were barriers to the adoption and widespread usage of analytics at the college.

Researchers have shown that the use of academic analytics improved student

retention through early warning systems, alerts, and student engagement tracking. The

use of analytics could help students select the correct courses and levels based on their

past performance and prior courses taken. Academic managers could also use analytics in

academics to develop schedules, track teacher performance, and credentials, develop

strategies to increase grant and alumni funds, and increase student financial aid

opportunities.

As colleges and universities move to a more student centered learning

environment, the most important use of academic analytics may be in the hands of

students. Students will be able to plan their academic experience and track their progress

in each course, and be able to compare their efforts and results to those of their peers.

This powerful tool could aid students’ engagement in coursework and with their

engagement at the college. Students will also be able to uncover potential career and

employment prospects and design future educational and life goals. When students have

120

access to their own data and are able to relate to it in an applicable manner, they can

shape a more meaningful and real future for themselves.

121

References

Al-alak, B., & Alnawas, I. (2011). Measuring the acceptance and adoption of e-learning

by academic staff. Knowledge Management & E-Learning: An International

Journal, 3(2), 1-11. Retrieved from http:kmel-journal.org/ojs/index.php/online-

publication/article/viewArticle/33

Ali, G. E., & Magalhaes, R. (2008). Barriers to implementing e-learning: A Kuwaiti case

study. International Journal of Training and Development, 12(1), 36–53.

doi:10.1111/j.1468-2419.2007.00294.x

Anderson, W., & Russell, M. J. (2012). Evolutionary analytics: BI on a shoestring.

Retrieved from http://www.educause.edu/ero/article/evolutionary-analytics-bi-

shoestring

Baepler, P., & Murdoch, C. J. (2010). Academic analytics and data mining in higher

education. International Journal for the Scholarship of Teaching & Learning,

4(2), 1–9. Retrieved from

http://digitalcommons.georgiasouthern.edu/cgi/viewcontent.cgi?article=1237&co

ntent=ij-sotl

Barneveld, A., Arnold, K., & Campbell, J. (2012). Analytics in higher education:

Establishing a common language. Retrieved from

http://net.educause.edu/ir/library/pdf/ELI3026.pdf

Behrend, T., Wiebe, E., London, J., & Johnson, E. (2011). Cloud computing adoption and

usage in community colleges. Behaviour & Information Technology, 30(2), 231-

240. doi:10.1080/0144929X.2010.489118

122

Behrens, S. (2009). Shadow systems: The good, the bad, and the ugly. Communications

of the ACM, 52(2), 124-129. doi:10.1145/1461928.1461960

Bichsel, J. (2012). Analytics in higher education benefits, barriers, progress, and

recommendations. Retrieved from

https://net.educause.edu/ir/library/pdf/ERS1207/ers1207.pdf

Blanton, S. (2012). Datamaster: Success and failure on a journey to business

intelligence. Retrieved from http://www.educause.edu/ero/article/datamaster-

success-and-failure-journey-business-intelligence

Cepuli, J., Radhakrishanan, R., & Widder, R. (2012). Building a performance analytics

environment. Retrieved from http://www.educause.edu/ero/article/building-

performance-analytics-environment

Chen, H., Chiang, R. H. L., & Storey, V. C. (2012). Business intelligence and analytics:

From big data to big impact. MIS Quarterly, 36(4), 1165–1188. Retrieved from

http://www.misq.org/skin/frontend/default/misq/pdf/V36I4_ChenIntroduction.pdf

Cilesiz, S. (2011). A phenomenological approach to experiences with technology:

Current state, promise, and future directions for research. Educational

Technology, Research and Development. 59(4), 487-510. doi:10.1007/s11432-

010-9173-2

Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. Proceedings

from the LAK’12: 2nd International Conference on Learning Analytics and

Knowledge. New York, NY. doi:10.1145/2330601.2330636

123

Cooper, R. (2010). Creative combinations in qualitative inquiry. The Qualitative Report,

15(4), 998-1001. Retrieved from http://www.nova.edu/ssss/QR/QR15-4/butler.pdf

Creswell, J. W. (2012). Qualitative inquiry and research design: Choosing among five

approaches (3rd ed.). Thousand Oaks, CA: Sage.

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods

approaches (4th ed.). Thousand Oaks, CA: Sage.

Crow, M. (2012). “No more excuses”: Michael M. Crow on analytics. Educause Review,

47(4), 14-22. Retrieved from http://www.educause.edu/ecar

Dalkir, K. (2011). Knowledge management in theory and practice. Cambridge, MA: MIT

Press.

Davenport, T. H., Harris, J., & Morison, R. (2010). Analytics at work: Smarter decisions,

better results. Boston, MA: Harvard Business School Publishing.

Dawson, S., Heathcote, L., & Poole, G. (2010). Harnessing ICT potential: The adoption

and analysis of ICT systems for enhancing the student learning experience. The

International Journal of Educational Management, 24(2), 116–128.

doi:10.1108/09513541011020936

Din, M. N. U., Khan, F., & Murtaza, A. (2011). A study to explore task and role of

educational managers in educational organizations of NWFP. Interdisciplinary

Journal of Contemporary Research in Business, 2(9), 484-495. Retrieved from

http://search.proquest.com/docview/857738883?accountid=14872

124

Dziuban, C., Moskal, P., Cavanagh, T., & Watts, A. (2012). Analytics that inform the

university: Using data you already have. Journal of Asynchronous Learning

Networks, 16(3), 21–38. Retrieved from http://eric.ed.gov/?id=EJ982670

Englander, M. (2012). The interview: Data collection in descriptive phenomenological

human scientific research. Journal of Phenomenological Human Scientific

Research, 43, 13-35. doi:10.1163/156916212X632943

Fahey, L. (2009). Exploring “analytics” to make better decisions: The questions

executives need to ask. Strategy & Leadership, 37(5), 12–18.

doi:10.1108/10878570910986434

Fenio, A., & Bright, A. (2010). Exploring barriers to effective e-learning: Case study of

DNPA. Interactive Technology and Smart Education, 7(1), 55-65.

doi:10.1108/17415651011031653

Forsythe, R., Chacon, F., Spicer, D., & Valbuena, A. (2012). Two case studies of learner

analytics in the university system of Maryland. Retrieved from

http://www.educause.edu/ero/article/two-case-studies-learner-analytics-

university-system-maryland

Fritz, J. (2011). Classroom walls that talk: Using online course activity data of successful

students to raise self-awareness of underperforming peers. Internet and Higher

Education, 14(2), 89-97. doi:10.1016/j.iheduc.2010.07.007

Goldstein, P. (2005). Academic analytics: The uses of management information and

technology in higher education. Retrieved from

https://net.educause.edu/ir/library/pdf/ecar_so/ers/ERS0508/ekf0508.pdf

125

Grajek, S. (2011). Research and data services for higher education information

technology: Past, present, and future. Educause Review, 46(6), 46-60. Retrieved

from http://www.educause.edu/ecar

Ice, P., Diaz, S., Swan, K., Burgess, M., Sharkey, M., Sherrill, J., Huston, D., &

Okimoto, H. (2012). The par framework proof of concept: Initial findings from a

multi-institutional analysis of federated postsecondary data. Journal of

Asynchronous Learning Networks, 16(3), 63-86. Retrieved from

http://eric.ed.gov/?id=EJ982674

Johnson, M. (2010). Barriers to innovation adoption: A study of e-markets. Industrial

Management & Data Systems, 110(2), 157-174. doi:10.1108/02635571011020287

Kauppinen, I. (2012). Towards transnational academic capitalism. Higher Education,

64(4), 543-556. doi:10.1007/s10734-012-9511-x

Krogh, G., Takeuchi, H., Kase, K., & Canton, C. (2013). Towards organizational

knowledge: The pioneering work of Ikujiro Nonaka. New York, NY: Palgrave

Macmillan.

Lane, C., & Lyle, H. (2011). Obstacles and supports related to the use of educational

technologies: The role of technological expertise, gender, and age. Journal of

Computing in Higher Education, 23(1), 38-59. doi:10.1007/s12528-010-9034-3

Lavalle, S., Lesser, E., Shockley, R., Hopkins, M., & Kruschwitz, N. (2011). Big data,

analytics and the path from insights to value. MIT Sloan Management Review,

52(2), 1–13. Retrieved from

http://tuping.gsm.pku.edu.cn/Teaching/Mktrch/Readings/BigData

Students also viewed