article with 4 Question (Paper-Critique)

JOURNAL OF INFORMATION SYSTEMS American Accounting Association Vol. 31, No. 3 DOI: 10.2308/isys-51837 Fall 2017 pp. 81–99

When Should Audit Firms Introduce Analyses of Big Data Into the Audit Process?

Anna M. Rose Jacob M. Rose

Oregon State University

Kerri-Ann Sanderson Jay C. Thibodeau Bentley University

ABSTRACT: This study investigates how the timing of the consideration of Big Data visualizations affects an auditor’s evaluation of evidence and professional judgments. In addition, we examine whether the use of an intuitive

processing mode, as compared to a deliberative processing mode, influences an auditor’s use and evaluation of Big

Data visualizations. We conduct an experiment with 127 senior auditors from two Big 4 firms and find that auditors

have difficulty recognizing patterns in Big Data visualizations when viewed before more traditional audit evidence.

Our findings also indicate that auditors who view Big Data visualizations containing patterns that are contrary to

management assertions after they view traditional audit evidence have greater concerns about potential

misstatements and increase budgeted hours more. Overall, our results suggest that Big Data visualizations used

as evidential matter have fewer benefits when they are viewed before auditors examine more traditional audit

evidence.

Keywords: Big Data; visualizations; pattern recognition; intuitive processing; deliberative processing.

I. INTRODUCTION

T he financial statement audit process increasingly involves the use of greater amounts of data and more sophisticated

analytical tools. In order to leverage the value of new data sources and ultimately reduce the risk of material

misstatement, audit firms are now evaluating audit approaches that encompass multiple external and internal sources of

data (Yoon, Hoogduin, and Zhang 2015). Many of the new approaches involve harnessing the richness of information

contained in what is commonly referred to as Big Data. We examine auditors’ use of Big Data to identify relevant patterns that

can be used to inform their audit judgments and decisions.

Big Data consists of large, unstructured datasets that are beyond the processing capabilities of traditional querying tools

and that include data from financial and nonfinancial sources (Brown-Liburd, Issa, and Lombardi 2015). Big Data is generated

on a continuous basis from a wide variety of sources with varying degrees of veracity (Zhang, Yang, and Appelbaum 2015).

Audit firms wish to use this potentially vast source of evidence to enhance audit effectiveness, but research in this area remains

incomplete, in large part, due to rapid technological advances (KPMG 2012; Yoon et al. 2015).

While the prospects of using Big Data offer much promise to financial statement auditors, there is currently limited

understanding of the effects of Big Data on the judgment and decision-making processes of financial statement auditors

(Brown-Liburd et al. 2015). Indeed, there are a number of critical issues for audit firms to address before they can successfully

implement analyses of Big Data in practice, such as the types of data to analyze and the most appropriate presentation formats

to utilize (Cao, Chychyla, and Stewart 2015). In addition, there is a broad issue that has received little or no research attention

to date, but has the capacity to influence the costs and benefits of all forms of Big Data and all visualization formats. This issue

The authors thank participants of the 2016 Journal of Information Systems Conference and workshop participants at Bentley University for their insightful comments. We also thank the professional participants who took the time to complete the experimental materials. Finally, we are very grateful for the guidance of the special issue editors, A. Faye Borthick and Robin R. Pennington.

Editor’s note: Accepted by A. Faye Borthick.

Submitted: April 2016 Accepted: May 2017

Published Online: June 2017

81

involves when to provide audit teams with the results of analyses of Big Data. Failure to use appropriate data or presentation

formats (Alles 2015; Information Systems Audit and Control Association [ISACA] 2013) or providing audit teams with data

analyses at inappropriate times in the audit process has the potential to create audit inefficiencies, increase the possibility of

decision traps and biases, and even lead to audit failures.

In an effort to shed light on the effects of Big Data on auditor judgment and decision processes, the purpose of this study is

to examine how the timing of presenting Big Data visualizations influences an auditor’s evaluation of evidence and related

professional judgments. Audit firms are investing significant resources into research designed to explore potential applications

of Big Data visualizations in the audit process, and data-visualization groups are among the fastest growing practice areas at the

larger offices of the Big 4 firms. Our interviews of audit partners at several Big 4 firms indicate that visualizations are currently

being viewed at different points in the audit engagement, and auditors can examine these visualizations before or after

traditional audit evidence is examined. As an example of the use of visualizations prior to the examination of traditional

evidence, one firm distributes visualizations to the audit team for use in a fraud brainstorming session at the beginning of the

audit, and auditors need to recognize patterns before they evaluate traditional sources of audit evidence. In addition, our

interviews of Chief Audit Executives (CAEs) revealed that some Fortune 500 corporations are drastically changing their

internal audit processes by using Big Data analyses to identify patterns that are then used to derive audit plans before other

evidence is examined.

We asked audit partners and CAEs about the expected benefits of allowing auditors to examine visualizations of Big Data

early in the audit process and before examining traditional audit evidence, and the underlying assumption was that it is

beneficial for auditors to look for patterns in Big Data before they derive conclusions from other information. That is, there is an

assumption that Big Data visualizations can reveal more useful patterns and more valuable information if auditors have a ‘‘clean slate’’ and have not already formed hypotheses and drawn conclusions based on traditional audit evidence. We propose that such an approach may create threats to effective auditing and may limit the benefits of Big Data analytics.

In addition, we posit that intuitive versus deliberative processing of evidence influences auditors’ use of Big Data

visualizations. Prior research demonstrates that when decision makers employ intuitive processing, they are better able to

recognize evidence that does not conform to expectations than are decision makers who employ deliberative processing

(Wilson and Schooler 1991; Zhong 2011). Thus, intuitive processing should enhance auditors’ professional skepticism and

improve their ability to recognize and identify threats relative to deliberative processing. Our experiment, therefore, addresses

two key issues facing audit firms as they evaluate applications of Big Data in practice: (1) when to present the results of Big

Data analyses to audit teams in order to maximize judgment benefits and minimize undesirable judgment traps and biases, and

(2) to better understand factors (e.g., processing modes) that may affect auditors’ interpretation and incorporation of Big Data

visualizations into the financial statement audit process.

We employ a 2 3 2 between-participant research design that involves 127 experienced audit seniors from two Big 4 firms

to examine the implications of the timing of Big Data visualizations used during the audit. We manipulate when auditors

examine visualizations of Big Data (before or after examining more traditional client information that creates a decision anchor)

and investigate the effects of such timing on multiple auditor judgments. In addition, the experimental design includes a second

manipulation that is known to prime a deliberative versus an intuitive processing mode, which allows us to evaluate whether the

use of Big Data visualizations and their timing have different effects on auditor judgment depending upon the predominant

processing mode that auditors use to evaluate the data.

Our results indicate that auditors have difficulty recognizing relevant patterns in Big Data visualizations when viewed

before the evaluation of more traditional client information. In addition, and importantly, there are significant effects of the

timing of Big Data use on auditor judgments that affect both audit planning and overall audit effectiveness. Auditors who view

Big Data visualizations containing information that is contrary to managements’ disclosures after they review preliminary analytical procedures (i.e., traditional audit evidence) express more concerns about misstatements, relative to auditors who

receive these visualizations before reviewing preliminary analytical procedures. Further, auditors increase budgeted hours more when visualizations that are contrary to management assertions are presented after reviewing preliminary analytical procedures,

rather than before reviewing the analytical procedures. Mediation analyses reveal that these effects are driven, at least in part,

by the effects of presentation timing on beliefs about the truthfulness of management’s explanation for events and concerns

about the need to collect additional data when visualizations suggest conflict with the results of traditional audit procedures.

The results of this study are important because auditors may ignore Big Data visualizations entirely in favor of internal data

sources or fail to recognize relevant patterns in visualizations, such as trends that contradict information discovered during

traditional analytical procedures or relationships that are contrary to management assertions. Thus, presentation timing is

important to the recognition of meaningful patterns in Big Data and the integration of Big Data into decision processes. Overall,

our results indicate that Big Data offers fewer benefits and potentially undesirable consequences when the visualizations are

introduced into the decision processes of auditors before they have examined other audit evidence and formed initial

hypotheses and expectations that can drive their search for patterns in Big Data.

82 Rose, Rose, Sanderson, and Thibodeau

Journal of Information Systems Volume 31, Number 3, 2017

II. BACKGROUND AND HYPOTHESIS DEVELOPMENT

Use of Big Data in the Financial Statement Audit

Professional standards require external auditors to perform preliminary analytical procedures during the planning stages of

each financial statement audit. The objective of such procedures is to direct the auditor’s attention to those significant financial

statement accounts that might contain a material misstatement (Louwers, Blay, Sinason, Strawser, and Thibodeau 2018). To

complete the analytical procedures, auditors should consider all types of relevant data that would help to improve their

understanding of the client’s business and industry, including ‘‘data that is preliminary or data that is aggregated at a high level’’ in their analyses (Public Company Accounting Oversight Board [PCAOB] 2010, }48). As a result, and given the extent of data that are currently available from a wide variety of sources, the increased use of Big Data holds much appeal as a way to

improve the effectiveness of preliminary analytical review procedures.

This potential is acknowledged in a recent PricewaterhouseCoopers (PwC) publication that states that ‘‘data analytics are altering the way the audit process is done . . . auditors have new tools to extract and visualize data, allowing them to dig into larger, non-traditional data sets and perform more intricate analysis . . . the ability to analyze all of it leads to better insight’’ (PwC 2015, 6). However, in order to take advantage of Big Data when completing a preliminary analytical review, auditors

must be able to recognize patterns in the Big Data being analyzed.

For some time, researchers have known about the importance of an auditor’s ability to recognize data patterns while

assessing audit risks that may suggest the existence of errors or fraud in the financial statements (Libby 1985; Bedard and Biggs

1991; Coakley and Brown 1993; O’Donnell and Perkins 2011). New data sources offer exciting opportunities for the

identification of decision-relevant patterns that have previously been hidden from auditors. Indeed, auditors are taking a more

encompassing approach in their audit risk assessments by gathering and examining evidence from a variety of sources in an

effort to decrease the likelihood of material misstatement and audit failure (Yoon et al. 2015). Evidential matter from Big Data

has the potential to be instrumental in this process. Data analytic tools allow auditors to search for patterns in Big Data that

would likely be undetectable in typical audit samples or smaller datasets (Alles and Gray 2014; Alles 2015). However, the large

volumes of output from analyses of Big Data could be overwhelming for auditors to cognitively process and may exacerbate

auditors’ inability to effectively recognize patterns (Brown-Liburd et al. 2015).

In fact, prior research finds that auditors often fail to adequately recognize patterns in financial and nonfinancial data

(Libby 1985; Bedard and Biggs 1991; Bierstaker, Bedard, and Biggs 1999; Asare, Trompeter, and Wright 2000; O’Donnell and

Perkins 2011). Further, studies find that auditors can fail to correctly extrapolate findings from time-series data (Biggs and Wild

1985), and may not adequately use prior information when analyzing subsequently identified patterns to inform their ultimate

judgment (Bedard and Biggs 1991). Results from these prior studies suggest that there is a significant danger that auditors will

not recognize important patterns in new and more complicated data sources. In addition to the possibility that auditors will fail

to recognize patterns and use them as inputs to their judgments and decisions, visualizations of Big Data could also lead to the

identification of a vast number of meaningless patterns that distract and lead the decision maker to pursue irrelevant and/or

inefficient investigations (Brown-Liburd et al. 2015).

Although the prior literature suggests that auditors often fail to adequately identify relevant data patterns, there are factors

related to the auditor and the auditing environment that can improve the pattern recognition process. Hammersley (2006 ) finds

that industry-specialist auditors have better developed problem representations about their subject industry and are better able to

identify seeded misstatement patterns. Likewise, Selby (2011) finds that auditors with a procedural understanding of automated

controls better assimilate the meanings of risk patterns in automated controls. Studies also find that visualization strategies can

help auditors recognize patterns. O’Donnell and Perkins (2011) find that employing a systems-thinking tool approach can help

auditors better identify fluctuation patterns in accounts and appropriately assess misstatement risk. These studies provide

evidence that auditors’ pattern recognition deficiencies can be overcome by enhancing their knowledge structures and also by

altering the manner of information presentation. In this study, we focus on one critical aspect of information presentation (i.e.,

its timing). And, importantly, employing experienced auditors allows us to examine the effects of Big Data on a participant

pool that has knowledge and experience in the subject matter addressed in our decision task, thereby reducing the likelihood

that knowledge deficits or lack of training are causing the results.

Effects of Big Data Visualization Timing on Auditor Decisions

Auditing firms stress the importance of overcoming judgment biases during the audit process (KPMG 2011). The use of

Big Data in audit practice has the potential to create a variety of decision biases and exacerbate existing biases (Brown-Liburd

et al. 2015). For example, auditors may ignore relevant patterns in Big Data, fail to recognize patterns, or identify irrelevant

patterns. Failing to recognize patterns or focusing on irrelevant patterns would severely limit the value of Big Data to the audit

process and decrease audit efficiency and effectiveness. Focusing on patterns that are irrelevant to audit decisions is particularly

When Should Audit Firms Introduce Analyses of Big Data Into the Audit Process? 83

Journal of Information Systems Volume 31, Number 3, 2017

worrisome because decision makers tend to overgeneralize from interesting examples and anecdotes, while placing too little

emphasis on statistically supported conclusions (e.g., Borgida and Nisbett 1977; Hamill, Wilson, and Nisbett 1980; Fagerlin,

Wang, and Ubel 2005; Kida 2006 ). The novelty of Big Data visualizations could make these more interesting than traditional

audit procedures, and identification of irrelevant patterns would result in excessive auditing, focusing on irrelevant issues, and

possibly incorrect audit conclusions.

Big Data analysis has the potential to change the way auditors collect evidence and make audit decisions (Brown-Liburd et

al. 2015). Although there are a few prior studies that discuss the potential impacts of Big Data on auditors’ decisions (Brown-

Liburd et al. 2015) and factors that influence the use of Big Data on the audit (e.g., Alles 2015; Cao et al. 2015), there is a

dearth of research investigating how and in what ways the review of Big Data affects auditor judgment. This study aims to fill

this gap in the literature by examining how the timing of Big Data review influences auditors’ judgments and evidence

evaluation. As there is limited prior research in this area, we gather insights from practitioners about the potential effects of the

timing of Big Data review on auditors’ subsequent judgments. Interviews with partners at Big 4 firms suggest that there is no

defined standard for the timing of Big Data examination during the audit engagement. Big Data may be reviewed before or after

examining traditional audit evidence. In some large corporations, Internal Auditing departments require Big Data analysis and

review during the audit planning phase before examination of more traditional audit evidence. Practitioners argue that

examining Big Data visualizations before traditional evidence allows auditors to identify useful patterns that will enhance their

interpretation and analyses of more traditional audit evidence. Other practitioners argue that Big Data visualizations should

only be used to supplement other audit evidence and that Big Data should be reviewed after the examination of more traditional

audit evidence. While practitioners hold differing perspectives regarding when Big Data evidence should be reviewed during

the audit engagement, there is no research that we are aware of that investigates the potential effects of timing on auditor

decision making.

In the context of analyzing Big Data, we expect that developing an initial expectation or hypothesis about an audit matter

will provide auditors with a framework for understanding that will enhance an auditor’s pattern recognition. Consistent with

Hammersley (2006), our study contends that auditors need a framework within which to identify and interpret patterns revealed

by Big Data. Given the massive volumes of data and their widely varying veracity, Big Data is noisy and can easily contribute

to information overload for the auditor (Brown-Liburd et al. 2015). In this complex and messy Big Data environment, auditors

who understand the client’s financial issues and the results of traditional audit tests and who are able to form an initial

expectation will develop a decision framework within which they can be more professionally skeptical and better able to

identify relevant patterns presented in Big Data analyses.

Having at least one perspective or expectation about the evidence under consideration allows the evaluator to envision

factors that support or refute this initial expectation (Hammersley 2011). Auditors develop an expectation about the client’s

financial statements by gathering initial audit evidence through preliminary management inquiries, analytical procedures, and

other audit-related activities. Prior research contends that finding meaningful patterns in Big Data is challenging because the

datasets are so voluminous and varied that many different patterns can emerge (e.g., Brown-Liburd et al. 2015; Carraway

2013), and having an initial perspective of the patterns that would be consistent or inconsistent with prior expectations can

guide information search in a Big Data environment.

Given that having a decision framework and forming an initial expectation facilitate pattern recognition (Hammersley

2006), we expect that auditors who evaluate visualizations of Big Data after reviewing traditional audit evidence will better

recognize patterns of evidence. Pattern recognition will be improved because the traditional evidence provides the decision

framework and information needed to form expectations. This leads to our first hypothesis:

H1a: Auditors who examine Big Data visualizations after reviewing preliminary analytical procedures will be more likely to recognize important patterns of evidence than will auditors who examine Big Data visualizations before

reviewing analytical procedures.

In addition to facilitating pattern recognition, we also expect that presentation timing will influence auditor judgments and

decisions. Prior research in psychology indicates that the order in which different types of information are received changes the

processing mode that decision makers employ to analyze the data and ultimately form their judgments. An important prior

study in this area was conducted by De Martino, Kumaran, Seymour, and Dolan (2006). Their experiments employ functional

magnetic resonance imaging (fMRI) scans to determine what portions of the brain are activated by framing biases and different

decision processes. They find that individuals differ in their tendencies to use intuitive versus deliberative regions of the brain,

and that combining intuitive and deliberative judgment processes produces superior judgments relative to intuition or

deliberative processing alone. In addition, and important to our study, creating conflict between an intuitive judgment and a

deliberative process resulted in the activation of multiple regions of the brain and the most effective judgment processes. These

results suggest that Big Data analyses and visualizations offer more benefits when these data create conflicts with intuitions and

84 Rose, Rose, Sanderson, and Thibodeau

Journal of Information Systems Volume 31, Number 3, 2017

initial expectations. That is, some of the benefits of Big Data in the audit will come from the capacity of the data to create

conflict, activate skepticism, and produce judgments that combine intuition and deliberative reasoning.

To create this conflict, auditors need to form an initial hypothesis or expectation about audit evidence, which then allows

them to envision factors that could refute their expectation (Hammersley 2011). We posit that examining Big Data

visualizations after forming expectations based on traditional audit evidence will allow auditors to better integrate evidence that

is contrary to management assertions into their decision processes and employ professional skepticism. Auditors who receive

Big Data evidence that differs from management assertions will be more skeptical of the accuracy and reliability of accounting

disclosures when the visualizations are presented after auditors have viewed traditional audit evidence and have formed initial

expectations:

H1b: Auditors who examine Big Data visualizations (containing patterns of evidence that differ from management assertions) after reviewing preliminary analytical procedures will be more likely to believe that accounting figures

are misstated than will auditors who examine visualizations before reviewing analytical procedures.

Further, auditors’ heightened evidence pattern recognition (i.e., resulting from reviewing traditional audit evidence before Big

Data visualizations), coupled with their increased skepticism regarding management’s representations and accounting figures,

will lead auditors to request and examine more audit evidence. This leads to our next hypothesis:

H1c: Auditors who examine Big Data visualizations (containing patterns of evidence that differ from management assertions) after reviewing preliminary analytical procedures will increase their assessment of budgeted audit hours

more than will auditors who examine visualizations before reviewing analytical procedures.

Effects of Processing Mode on Auditor Judgments and Decisions

Psychology research indicates that individuals make decisions using two generic modes of information processing: an

intuitive mode (‘‘thinking fast’’), in which decisions are made using automatic and heuristic processes; and a deliberative or analytical mode (‘‘thinking slow’’), which engages in more controlled and systematic reasoning (Kahneman 2011). Intuitive processing is fast and encompasses an emotive approach to evaluation that is performed subconsciously (Dane and Pratt 2007).

Historically, researchers have suggested that intuitive and heuristic processing leads to poor decision making fraught with

systematic errors (e.g., Tversky and Kahneman 1974), while engaging in more effortful and deliberative processing improves

judgements and reduces individuals’ susceptibility to the processing pitfalls observed in heuristic processing (Kahneman 2011).

Recent psychology research has given cause to rethink the widely held belief that deliberative processing consistently

results in superior decision making relative to intuitive processing. Research suggests that intuitive processing accesses

subconscious cognitive structures that provide a multi-faceted evaluative approach to problem solving, whereas deliberative

processing crowds out intuition and leads people to only focus on the salient factors in a given context (e.g., Zhong 2011;

Wilson and Schooler 1991). For example, Ambady and Rosenthal (1993) find that students can make accurate evaluations of a

person’s teaching performance in a very short period of time, even though they may not be able to articulate the specific reasons

or factors leading to their conclusion. Similarly, Zhong (2011) finds that participants who exercise intuitive processing make

better moral decisions than those who exercise deliberative processing. Zhong (2011) suggests that these findings are a result of

the differential focus of deliberative versus intuitive processing. He reasons that compared to intuitive processing, deliberative

processing focuses attention on the saliency of factors in the given context with little consideration of whether these factors

could be legitimate causes of the effects observed.

Consequently, auditors are likely better able to identify deviations in patterns when they apply a more intuitive approach to

pattern evaluation. That is, when auditors engage in intuitive processing (versus deliberative processing) before evaluating audit

evidence, it is likely that they will be better able to recognize anomalies and instances where evidence factors do not conform to

expectations. In this way, auditors will be more skeptical during evidence evaluation when they employ an intuitive approach to

evidence evaluation. Research also finds that processing mode changes based on the decision context, and processing mode

can, therefore, be effectively primed (see, e.g., Hsee and Rottenstreich 2004; Small, Loewenstein, and Slovic 2007; Zhong

2011). Thus, there is an opportunity to influence auditors’ processing modes in order to enhance their ability to recognize

anomalies and important patterns in data.

Overall, research finds that intuitive processing engages evidence evaluation from a more comprehensive perspective

(Dane and Pratt 2007), rather than focusing on salient factors that may not be diagnostic of the environment. In our study,

auditors evaluate audit evidence related to a client’s gross margin derived from traditional analytical methods, as well as from

Big Data sources. Given that intuitive processing is an emotive response (e.g., Tversky and Kahneman 1974) that operates by

subconsciously matching patterns in evaluated evidence with a subconscious set of expectations (Lieberman 2000), we expect

that auditors who engage in intuitive processing will be more sensitive to patterns that challenge expectations (Wolfe,

When Should Audit Firms Introduce Analyses of Big Data Into the Audit Process? 85

Journal of Information Systems Volume 31, Number 3, 2017

Christensen, and Vandervelde 2016 ). As such, we expect that auditors engaged in intuitive processing will be more concerned

about the potential for errors in the client’s gross margin presentation, and will be more alert to multiple patterns of audit

evidence than will auditors who engage in deliberative processing. This leads to our next set of hypotheses:

H2a: Auditors exposed to an intervention that increases the level of intuitive processing will express more concerns about potential problems with the client’s reported gross margin than will auditors exposed to an intervention that

increases the level of deliberative processing.

H2b: Auditors exposed to an intervention that increases the level of intuitive processing will be more likely to recognize patterns of evidence in Big Data visualizations than will auditors exposed to an intervention that increases the level

of deliberative processing.

Given that prior research finds that developing an initial perspective about audit evidence allows auditors to conceptualize

factors that could support or counter their initial expectation (Hammersley 2011), we posit that developing an initial framework

enhances auditors’ pattern recognition. Therefore, we predict that auditors will be more likely to recognize patterns of evidence

in visualizations when they review Big Data visualizations after reviewing analytical procedures. We also expect that

employing an intuitive processing approach (versus a deliberative processing approach) will allow auditors to engage in more

comprehensive evidence evaluation (Dane and Pratt 2007) that will amplify pattern recognition and cause auditors to be more

sensitive to multiple patterns of audit evidence (Wolfe et al. 2016 ). Taken together, available theory suggests that auditors will

be most likely to recognize patterns of evidence in visualizations when they engage in intuitive processing and review Big Data

visualizations after reviewing analytical procedures. This leads to the following interaction hypothesis:

H3: Auditors exposed to an intervention that increases the level of intuitive processing (versus deliberative processing) and who examine Big Data visualizations after (versus before) reviewing preliminary analytical procedures will be

most likely to recognize patterns in Big Data visualizations.

III. TASK DESCRIPTION AND EXPERIMENT DESIGN

Analyses of Big Data can provide insights from operational, financial, and other types of electronic data using sources that

are internal or external to the company. The information garnered from these analyses may provide insights that are historical,

real-time, or predictive in nature (KPMG 2012). Our study examines the effects of timing of the use of Big Data from external

third-party sources to augment traditional auditing procedures. We examine best practices related to the process of analyzing

nonfinancial Big Data while completing preliminary analytical procedures, and investigate two significant issues facing audit

firms as they evaluate applications of Big Data in practice: (1) when to present the results of data analyses to audit teams in

order to maximize judgment benefits and minimize undesirable biases, and (2) cognitive processing factors that may affect

auditors’ interpretation and incorporation of Big Data analyses.

We address our research questions by conducting an experiment using professional auditors as participants (institutional

review board approval was received to conduct this experiment). Our experimental design manipulates auditors’ processing

modes (intuitive versus deliberative) and the order in which Big Data visualizations are presented (before or after reviewing

preliminary analytical procedures). We examine auditors’ evaluation of the client’s reporting of gross margin and their intention

to collect additional data.

Participants

One hundred twenty-seven auditors completed the study and were recruited from two Big 4 firms. The experimental task

was administered by multiple authors. Each participant had an average of 2.7 years auditing experience, 2.3 years of experience

conducting analytical reviews, and 55 percent were male.

Task Description

To complete the experimental task, participants first answered a set of five questions that required them to make

mathematical calculations or provide emotional responses to a list of statements. This task primes an intuitive or deliberative

processing mode. Participants then read a hypothetical case describing a technology company that develops, manufactures, and

markets gaming and wearable technology. Participants were then presented with one of two sources of information: (1) results

of traditional preliminary analytical procedures related to the company’s gross margin, or (2) Big Data visualizations collected

from corporate websites, product discussion forums, Twitter feeds, and social networking sites.

There were four Big Data visualizations. We employ visualization types with which our auditor participants are already

familiar in order to reduce familiarity effects, and the auditor participants have previously received training on the use of

86 Rose, Rose, Sanderson, and Thibodeau

Journal of Information Systems Volume 31, Number 3, 2017

graphical displays to identify patterns. Two of the visualizations did not contain informative patterns; one contained a pattern

that provided evidence that the large increase in gross margin would be unexpected; and the other suggested that management’s

explanation for the increase in gross margin could be questionable. 1

The visualizations are presented in Appendix A.

The first uninformative visualization presented a hashtag analysis that compared the number of social media messages

tagged with the audited firm’s name versus messages that were tagged with the hashtags of four major competitors in the

industry. There were no discernable patterns in this visualization that were relevant to the audit decisions in the case. A second

uninformative visualization displayed the volume and sentiment of online discussions related to the firm being audited during

the third quarter and, again, there were no relevant patterns. The first informative visualization displayed the relationships

between tweeting activity and sales revenue, and the pattern for the firm being audited was contrary to the pattern for all other

firms with increasing revenues (i.e., sales revenue increasing while tweeting activity decreasing). The second informative

visualization presented a word cloud of terms used in social media to describe the client’s products. The product responsible for

the increased gross margin (a wearable fitness band) was conspicuously absent from the word cloud, indicating that the client’s

high-growth product is not being mentioned on social media. Both of the informative visualizations contradict elements found

in other auditor evidence from preliminary analytical procedures related to gross margin.

Included in the information provided for the preliminary analytical procedures were results of company performance and

divisional performance for the current and the prior year. Metrics related to net sales, cost of goods sold, gross margin, and

gross margin percentage were provided. Participants also read a summary of an interview with the CFO, who explained the

underlying factors behind the increase in gross margin. After being exposed to either of the information sources, participants

answered dependent variable questions. 2

Participants next evaluated the alternate type of information set (i.e., the Big Data visualizations or traditional preliminary

analytical procedures data). After examining the second information set, participants answered the dependent variable questions

again. Finally, participants completed the post-experimental and demographic questionnaire.

Independent Variables

We employ a 2 3 2 between-subjects full-factorial design where we manipulate auditors’ processing mode and the order in

which Big Data visualizations are presented. The first independent variable, Processing Mode, is manipulated on two levels (Intuitive versus Deliberative). To operationalize auditors’ processing mode, we follow prior research in psychology (e.g., Zhong 2011) and ask participants to answer five questions in order to prime a processing mode. Previous studies have

demonstrated that asking individuals to calculate math problems versus reflect on their feelings effectively stimulates a

deliberative processing mode or an intuitive processing mode, respectively (e.g., Hsee and Rottenstreich 2004; Small et al.

2007).

Before starting the study, participants in the Deliberative condition answered five questions requiring mathematical calculations (see all five questions in Appendix B). For example:

If an object travels at five feet per minute, then by your calculations how many feet will it travel in 360 seconds?

Participants in the Intuitive condition answered five questions requiring them to examine their feelings (see all five questions in Appendix B). For example:

When you hear the name ‘‘Barack Obama,’’ what do you feel? Please use one word to describe your predominant feeling.

For the second independent variable, Big Data Order, we manipulate the order in which participants are presented with results of preliminary analytical procedures and visualizations of Big Data analyses on two levels (Big Data Before versus Big Data After). Big Data visualizations examine: (1) number of hashtag mentions in social media; (2) tweets; (3) text analysis of words used to describe the company’s fitness device on social media sites; and (4) text sentiment analysis. The analytical

procedures information consisted of company performance measures in current and prior periods (i.e., net sales, cost of goods

sold, gross margin, and gross margin percentage). Also included in the depiction of the preliminary analytical procedures

results is a narrative explanation from the company’s CFO for the current period’s gross margin results. Participants were also

presented with the company’s sales mix for each company division for both the current and prior years.

1 We intentionally designed the experimental instrument to contain two informative and two uninformative visualizations related to the company’s gross margin results. Discussion with practice professionals indicated that Big Data visualizations typically contain information that is relevant, as well as irrelevant, to audit tests. Consequently, we chose this design approach to maintain practice realism and reduce the likelihood of participants discovering the experiment’s purpose. This design choice also reduces the likelihood of obtaining our hypothesized results.

2 The final experimental instrument was developed following adjustments based on pilot testing we conducted with current and former auditing practitioners and the input from partners of one of the participating firms.

When Should Audit Firms Introduce Analyses of Big Data Into the Audit Process? 87

Journal of Information Systems Volume 31, Number 3, 2017

Dependent Variables

The dependent variable for H1a (Recognize Patterns) captures whether participants recognized the patterns in the two Big Data visualizations. To provide evidence that participants had recognized relevant patterns, we asked participants to indicate

any questions or concerns that they had about the gross margin percentage. One author and one research assistant

independently coded participant responses without reference to the treatment conditions. 3

A second author reviewed and

reconciled the coding without knowledge of the treatment condition for each coded response. Example comments related to the

pattern in the sales/tweets visualization included: ‘‘social media seemed to lean more towards a competitor. What did the competitor do in the period?’’ and ‘‘social media trends going down but margin way up?’’ Comments related to the pattern in the word cloud included: ‘‘social media buzz words related more to gaming, so why such an inverse in future products?’’ and ‘‘fitness did not really appear in social media; why?’’ The dependent variable, Recognize Patterns, is calculated as the number of patterns recognized (0, 1, or 2).

To test H1b, a question asked participants whether gross margin is properly recorded or misstated (Misstatement). We measure this variable after participants have reviewed both the analytical procedures results and Big Data visualizations, using

an 11-point anchored scale where�5 represents their response that gross margin is ‘‘very understated’’; 0 represents that gross margin is correct; and þ5 represents that gross margin is ‘‘very overstated.’’ For our next dependent variable, Audit Hours, we ask auditors to indicate how many hours they would budget for substantive testing of the sales account given that 100 hours

were budgeted in the prior year. The Audit Hours variable is used to test H1c. To measure Total Concerns for the test of H2a, we sum the number of unique concerns or questions that participants documented regarding the change in the client’s gross

margin percentage. 4

Finally, we use the dependent variable Recognize Patterns to test H2b and H3.

IV. RESULTS AND ANALYSES

Attention, Manipulation, and Completion Checks

Participants could not fail to attend to the order of presentation of evidence, and no attention check is needed to verify the

order manipulation. In order to determine whether participants attended to the processing mode manipulation, we examined the

solutions to math problems and statements of emotional response to verify that participants had completed the task

appropriately. All participants in the deliberative processing mode treatment condition completed the math questions. Five

participants did not complete the emotional response questions or used non-emotional response terms to complete the

questions; as such, these participants are not included in the analyses.

Consistent with the prior research using these math and emotional response items to prime processing mode, we do not ask

participants how the task affected their processing mode because the priming effects are primarily non-conscious, and

participants are not aware of the processing changes (e.g., Hsee and Rottenstreich 2004; Small et al. 2007; Zhong 2011).

Because participants are not aware of the processing mode changes, participants cannot be asked to indicate their processing

mode. Thus, we confirm that participants appropriately completed the processing mode task, which is the accepted and

validated approach for verifying this processing mode manipulation. In addition, we verify that priming influenced participants’

processing by examining beliefs about gross income made immediately after the presentation of the first evidence set (either the

traditional evidence or the Big Data visualization). Participants in the intuitive processing mode were more likely to question

the gross margin change ( p ¼ 0.02; untabled), which is consistent with the anticipated effects of a more intuitive processing mode. Finally, we excluded nine participants from the analyses because they failed to complete the experiment and two

participants who provided responses that were extreme outliers.

Hypothesis Testing

In H1a, we predict that auditors who examine Big Data visualizations after examining results of preliminary analytical

procedures are more likely to recognize relevant patterns in the visualizations than are auditors who examine Big Data

visualizations before reviewing results of preliminary analytical procedures. Consistent with this prediction, we find a main

effect for the examination order of audit evidence on auditors’ likelihood of recognizing patterns in data visualizations.

Specifically, we find that when Big Data visualizations are presented after results of analytical procedures (mean ¼ 0.27), relative to before (mean¼0.07), participants are better able to identify patterns in Big Data visualizations (F [1, 90]¼4.40; p¼ 0.02, one-tailed; Table 1, Panel B).

3 The Cohen’s Kappa for this set of codes was 0.75, which indicates a high level of inter-coder agreement.

4 We also asked participants to list specific audit procedures that they would recommend. We did not find any differences in these tests across treatment conditions.

88 Rose, Rose, Sanderson, and Thibodeau

Journal of Information Systems Volume 31, Number 3, 2017