Discussion

James00712212

out3.pdf

Home >Information Systems homework help >Discussion

SECURITY ANALYTICS TOOLS AND IMPLEMENTATION SUCCESS FACTORS:

INSTRUMENT DEVELOPMENT USING DELPHI APPROACH AND EXPLORATORY

FACTOR ANALYSIS

Sethuraman K Srinivas

BERNARD J. SHARUM, PhD, Faculty Mentor and Chair

JOHN HERR, PhD, Committee Member

JELENA VUCETIC, PhD, Committee Member

Rhonda Capron, EdD, Dean

School of Business and Technology

A Dissertation Presented in Partial Fulfillment

Of the Requirements for the Degree

Doctor of Philosophy

Capella University

March 2018

ProQuest Number:

INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed,

a note will indicate the deletion.

ProQuest

Published by ProQuest LLC ( ). Copyright of the Dissertation is held by the Author.

Microform Edition © ProQuest LLC.

ProQuest LLC. 789 East Eisenhower Parkway

P.O. Box 1346 Ann Arbor, MI 48106 - 1346

10807845

2018

Abstract

Over the past two decades, information security scientists have conducted in-depth research in

the area of security analytics to counter cyber-attacks that have challenged the security postures

of corporate networks and data. Learning from this research has immensely benefited security

analytic tools and contributed to their maturity, thereby enabling many organizations to

implement them. While adoption of these tools has increased, understanding factors that impact

the successful implementation of these tools has lagged and such understanding is critical to the

information security practice. The literature review revealed the lack of a validated survey

instrument dedicated to security analytic tools, which can help in extracting implementation

factors that security professionals would consider to be critical for success. The focus of this

research study was to develop a survey instrument and use the developed instrument to identify

factors that impact the successful implementation of any security analytic tool, including big data

based tools. Delphi method was used to develop the instrument in Phase 1 with the help of

security analytic tool experts located in the United States of America, and the same instrument

was used to collect responses during Phase 2 from practitioners located in North America. The

researcher used Delphi method for establishing content validity, exploratory factor analysis for

establishing construct validity and Cronbach’s alpha for testing reliability. The Delphi study

started with 16 experts in the first round, ended with 11 experts providing consensus in the fourth

round. An exploratory factor analysis study performed during Phase 2, involving a sample size of

206, identified seven factors that impacted the successful implementation of security analytic

tools. These factors are: large-scale security event analysis, functional utilization, incident

detection and correlation analysis, governance and chief information security officer metrics, log

source and use case management, threat and operational intelligence, real-time attack and

anomaly detection. By defining metrics for these factors, practitioners could use these factors as

key performance indicators to assess the success of security analytic tool implementation.

Exploring causal relationships among identified factors, such as threat intelligence and incident

detection, will help in tuning security analytic tools and products.

iii

Dedication

I primarily dedicate this dissertation to my spiritual master and one of the greatest

humanitarians to walk this earth, Shri Mata Amritanandamayi Devi. Her loving presence greatly

inspired me to pursue this mammoth academic effort. I also dedicate this dissertation to the Non-

Dual Brahman (Creator), the Essence in all living and nonliving beings. My late father Shri K.V.

Srinivasan was alive when I started this pursuit but passed on to ages during this doctoral

journey. An exemplar of a man he was, and he truly deserves this dedication. I am sure he will be

doubly joyed in heaven.

I also dedicate this dissertation to my vibrant mother, Smt Saraswathy Srinivasan, who

takes enormous pride in all my accomplishments. She is a truly inspiring mother, filled with

myriad talents, and her disciplining nature is the main reason that I stand as a successful man in

all walks of life. No effort of a family man is achievable without the strong support of a spouse.

My wife, Sow Subha Sethuraman, was a great and unfailing support to me throughout this

journey. She spared no efforts in providing a conducive environment for my doctoral journey, in

particular, during the intense preparation for my comprehensive exams. My son, Naveen

Sethuraman, sacrificed a lot of weekend getaways to support me and was able to be highly

independent in his homework and assignments, thereby giving me a lot of time to work on my

dissertation.

Acknowledgments

I profusely thank and acknowledge the guidance and mentorship provided by Dr. Bernard

Sharum, mentor and committee chair. His patience with all my questions and concerns that arose

during this journey needs a special mention. Without his guidance, I would not have made good

progress. I also wish to thank Dr. Jelena Vucetic and Dr. John Herr for their timely support and

guidance during this journey. My sincere thanks to both of them for their encouraging approach.

I am also very thankful to Dr. Shardul Pandya, core faculty member at Capella, for his positive

and motivational words. His help in orienting me to Delphi studies and readiness to answer my

calls are greatly appreciated. My sincere thanks to Dr. Steven Brown, core faculty member in the

information assurance department at Capella, for his guidance during the Delphi study and also

during the final stages of data analysis. I hereby express my heartfelt gratitude to Dr. Tsun

Chow, faculty chair of doctoral IT Programs, for his stellar guidance that greatly enhanced the

quality of this dissertation, during the school review.

My wife, Sow Subha Sethuraman, despite being an extremely busy information

technology start-up professional, dedicated a lot of her time to this dissertation to support me.

Her help in reviewing my documents and her thoroughness in her execution is a rarity in this fast

age. I lovingly acknowledge her unflinching support.

I sincerely thank Mr. Jayakumar Muthukumarsamy (JK), chief architect and fellow at

Shutterfly Inc., who was a source of great support during my dissertation. He provided a lot of

useful ideas during the data analysis stage. My sincere thanks are due to Mr. Srinivas

Tummalapenta, chief architect and distinguished engineer, IBM managed security services. He

provided immense guidance during the pre-Delphi field study and also during the data analysis

phase.

Finally, I also want to thank every expert who helped me during the Delphi study rounds.

Their incisive inputs made a big contribution to the Delphi study and its output.

Table of Contents

Acknowledgments.............................................................................................................. iv

List of Tables ..................................................................................................................... ix

List of Figures ..................................................................................................................... x

CHAPTER 1. INTRODUCTION ................................................................................................... 1

Background of the Problem ................................................................................................ 1

Statement of the Problem .................................................................................................... 2

Purpose of the Study ........................................................................................................... 4

Significance of the Study .................................................................................................... 5

Research Questions ............................................................................................................. 7

Definitions of Terms ........................................................................................................... 7

Research Design.................................................................................................................. 8

Assumptions and Limitations ........................................................................................... 11

Organization of the Remainder of the Study .................................................................... 13

CHAPTER 2. LITERATURE REVIEW ...................................................................................... 14

Overview ........................................................................................................................... 14

Methods of Searching ....................................................................................................... 14

Theoretical Orientation for the Study ............................................................................... 15

Review of the Literature ................................................................................................... 29

Critique of Previous Research Methods ........................................................................... 45

Findings............................................................................................................................. 47

Summary ........................................................................................................................... 51

vii

CHAPTER 3. METHODOLOGY ................................................................................................ 55

Purpose of the Study ......................................................................................................... 55

Research Questions ........................................................................................................... 57

Research Design................................................................................................................ 57

Procedures ......................................................................................................................... 59

Likert Scale Instrument ..................................................................................................... 71

Ethical Considerations ...................................................................................................... 72

Summary ........................................................................................................................... 73

CHAPTER 4. RESULTS .............................................................................................................. 74

Background ....................................................................................................................... 74

Research Questions ........................................................................................................... 74

Description of the Sample ................................................................................................. 75

Delphi Study ..................................................................................................................... 81

EFA ................................................................................................................................... 93

Summary ......................................................................................................................... 102

CHAPTER 5. DISCUSSION, IMPLICATIONS, AND RECOMMENDATIONS ................... 104

Introduction ..................................................................................................................... 104

Summary of the Results .................................................................................................. 104

Discussion of the Results ................................................................................................ 107

Findings........................................................................................................................... 114

Conclusions Based on the Results .................................................................................. 114

Limitations ...................................................................................................................... 118

viii

Implications for Practice ................................................................................................. 119

Recommendations for Further Research ......................................................................... 121

Conclusion ...................................................................................................................... 123

REFERENCES ........................................................................................................................... 124

STATEMENT OF ORIGINAL WORK ..................................................................................... 136

APPENDIX A. RESEARCHER-DESIGNED SECURITY ANALYTICS SURVEY .............. 138

List of Tables

Table 1 Assessment of Security Analytics Tools ......................................................................... 47

Table 2 Delphi Panel Member Experience ................................................................................... 77

Table 3 Participant Domain Experience ....................................................................................... 80

Table 4 Participant Industry .......................................................................................................... 81

Table 5 Results of the Delphi Rounds .......................................................................................... 82

Table 6 Delphi Round 1– Descriptive Statistics ........................................................................... 84

Table 7 Delphi Round 2 – Descriptive Statistics .......................................................................... 86

Table 8 Delphi Round 3 – Descriptive Statistics .......................................................................... 89

Table 9 Delphi Round 4 – Descriptive Statistics .......................................................................... 91

Table 10 Sample Adequacy Test ................................................................................................. 93

Table 11 Total Variance Explained (SPSS) .................................................................................. 95

Table 12 Rotated Factor Matrix – SPSS Output ........................................................................... 97

Table 13 Reliability – Overall Instrument Level ........................................................................ 100

Table 14 Case Processing Summary ........................................................................................... 100

Table 15 Cronbach’s Alpha at Factor Level ............................................................................... 101

Table 16 Factor Names ............................................................................................................... 102

Table 17 Factor Definitions ........................................................................................................ 108

Table 18 Factors and Subfactors ................................................................................................. 116

List of Figures

Figure 1. Security analytics – Theoretical foundation. ................................................................. 17

Figure 2. Security analytics: A functional architecture. ............................................................... 51

Figure 3. High-level research design. ........................................................................................... 58

Figure 4. Delphi study process flow. ............................................................................................ 64

Figure 5. Security analytics research – A waterfall approach to orient the Delphi study

participant. ................................................................................................................. 79

Figure 6. Phase 2 – EFA study...................................................................................................... 94

Figure 7. SPSS scree plot analysis. ............................................................................................... 96

CHAPTER 1. INTRODUCTION

This chapter introduces the topic of assessing the implementation of security analytic

tools. There is an increasing trend towards adoption of security analytic tools among North

American organizations. This trend necessitates the creation and validation of a survey

instrument focused on security analytics leading to the identification of implementation success

factors. This chapter states the research questions in the context of the problem’s background.

These questions form the basis for further research and identification of factors. This chapter

explains the purpose and significance of this research along with research design, assumptions,

and limitations. A description of the organization of the remainder of this study is provided at the

end of this chapter

Background of the Problem

Cyber-attacks have become a daily phenomenon over the past few years, with hacker

strategies becoming very sophisticated and vicious. Botnet-based attacks, advanced persistent

threat (APT) attacks, malware attacks, denial of service (DoS) attacks, and insider attacks are

flooding corporate networks all over the world. These attacks and threats form a major part of

daily security incidents in many enterprises. Cardenas, Manadhata, and Rajan (2013) observed

that enterprises collect terabytes of security log data every day, which leads to challenges even in

storage management. Analyzing large volumes of log data to identify security anomalies that

impact the safety of corporate businesses is a greater problem. The need for enterprises to

comply with national legislation such as Sarbanes-Oxley and payment card industry (PCI)

mandates, combined with normal security operations, are contributing to increases in log data.

Van de Moosdijk, Wagenaar, and Final (2015) elaborated on the contribution of log management

and security analytics in fulfilling compliance laws. These laws mandate retention of many years

of log data to satisfy auditing requirements.

As Crespo and Garwood (2014) epitomized in their botnet-related article, identifying

security incidents and correlating large and real-time security data segments to extract actionable

intelligence has warranted the need for deeper and faster analytics. The size, speed, and precision

of this analytics differ based on the need of the enterprise. Mahmood and Afzal (2013) presented

a very comprehensive survey of big data security analytic tools in their analysis of security

analytic trends. Their findings portrayed the capability of big data tools in managing security

incidents.

Most assessments in security analytics deal with self-assessment by product inventors or

tool comparisons by third-party critics. Enterprises that implement market-leading security

analytic tools or in-house tools are the real benefactors of these tools. The end users from these

organizations are in a better position to assess and rate these tools’ benefits and downsides.

Scholar-practitioners in the security analytic domain also benefit immensely from an unbiased

evaluation of these tools by experts and end users.

Statement of the Problem

The research literature on the security analytics domain indicates that there is an

explosive growth of tools and in-house packages in this domain. Such tools are starting to attain

maturity in terms of ensuring the safety of enterprises. However, there is no clear set of factors

and attributes to evaluate the success of these tools within an enterprise. Nicolett and Kavanagh

(2013) studied the critical capabilities of security analytics tools, and they found a relationship

between tool capabilities and effectiveness. However, they did not clearly state the areas within

these tools that needed an examination to conclude a successful implementation. Mateski et al.

(2012) defined cyber threat metrics as a part of their research work for Sandia National

Laboratories. While metrics measure the effectiveness of the governance aspects of security

analytic tools, we do not have exhaustive feedback from the user community of these security

analytic tools. The first part of the research problem is a lack of structured assessment of security

analytic tools from the user’s perspective in the currently available and surveyed literature. The

second part of the problem is a lack of identified factors that determine a successful

implementation of security analytic tools according to the tool users and experts.

Shackleford (2014) surveyed the existing commercial security analytic tools from the

perspective of product architecture. Shackleford did not assess whether these tools revealed a

successful implementation, leading to the safety of an enterprise. Howarth (2014) discussed in

detail the actionable intelligence that was generated by the security analytic tools. For example,

actionable intelligence refers to recommending a list of infected endpoint devices that need to be

quarantined. Howarth’s discussion did not focus on broader factors that influence a successful

implementation of security analytic tools. Mahmood and Afzal (2013) presented a

comprehensive survey of big data security analytic tools in their analysis of security analytic

trends. Even though they focused on big data security analytics as a domain, there was no

discussion on identification of factors to assess the success of an implementation. Hence, the

research problem (i.e., determining the factors that influence a successful implementation of

security analytics tools, leading to the safety of an enterprise) is a fresh problem and a gap in the

information security industry that needs to be addressed.

Use cases are similar to user scenarios and are driven by the risk scenarios that a security

analytic tool vendor is trying to address. Van de Moosdijk, Wagenaar, and Final (2015) asserted

that use cases are fundamental to security analytic tools. Nicolett and Kavanagh’s (2013) as a

part of their Gartner study identified three use cases for assessing a security analytic tool. These

use cases are threat management, compliance, and tool deployment. These use cases were

standardized based on the opinion of a few research analysts and addressed only a subsection of

security analytic tool assessment problem. Shackleford’s (2014) survey focused on popular

product features such as architecture and performance, in the domain of security analytics.

Above citations point to the fact that commercial surveys are built based on popular product

features and architecture. But the imminent need is a common and broad assessment framework

that will help to remove the bias in commercial product surveys. Cybenko and Landwehr (2012)

recognized that assessment produced by above-mentioned commercial surveys are likely to be

biased towards the sponsor of the survey. Another significant gap in commercial surveys that are

cited above is the lack of validated survey instrument produced by a panel of unbiased and

anonymous experts.

Purpose of the Study

The major goals of this study are to identify factors that determine the successful

implementation of security analytic tools that could be used to ensure the security of any

enterprise. The general problem that is addressed in this research is the lack of a structured

method or instrument to assess the implementation of security analytic tools in any given

organization. Development of such an instrument can solve the broader problem of lack of a

structured assessment method in the field of security analytics. However, literature analysis

revealed that the narrower research problem is the lack of factors that can be used to assess an

implementation. Shackleford (2014) developed a commercial survey in the domain of security

analytics which focused more on product features and did not use technology-neutral

implementation factors. Ferketich, Phillips, and Verran (1993) suggested that a researcher

modify an existing instrument or create a new instrument in the absence of a suitable instrument

to answer the research question. The creation of a new survey instrument with a broad focus on

all generations of security analytic tools is the first step that would lead to the extraction of

implementation factors. Since there was no instrument available to survey users of security

analytic tools, a Delphi study to build a questionnaire with the help of experts in the

cybersecurity industry was the immediate first step, and it formed Phase 1 of this research study.

Significance of the Study

The assessment of the implementation of security analytic tools is the broad goal of this

research study. This study is significant due to the recent proliferation of new security analytic

tools, both from popular vendors and independent scientists trying to analyze large volumes of

data to identify cyber-attacks. Security analytic tools, in the context of this research study,

include in-house solutions, intrusion detection and prevention solutions (IDPS), commercial

security information and event management (SIEM) tools, and solutions with big data capability.

The main beneficiaries of this study will be both professionals and researchers in the information

security industry.

The ultimate goal of this study is to identify factors that determine the successful

implementation of security analytic tools. The factors are identified after an initial survey

instrument is built. This study would provide many benefits to the information security industry

and its professionals. Some of the significant benefits are (a) identifying the implementation

areas of focus for security analytic tools. For example, certain types of tools may not have strong

focus on correlation processes to support correlation analysis of incidents; (b) identifying factors

that will help in ensuring the protection of valuable organizational assets and personally

identifiable information; and (c) identifying factors that will help in the fulfillment of business

stakeholders requirements like log management and compliance with industry compliance laws.

A survey instrument for which the contents and constructs are validated is a necessary pre-

requisite for identifying the implementation factors. This survey instrument will benefit the

researchers in the information security domain.

Commercial surveys do not always consider user inputs from hands-on experts to build

the survey. Nicolett and Kavanagh (2013) analyzed the critical capabilities of commercial SIEM

tools as a part of the Gartner study. This Gartner study is a based on commercial opinion surveys

built using product use-cases. Nicollet and Kavanagh did not examine user perceptions of tools

and factors that users consider as significant for the implementation of these tools. However, this

research study focusses on conducting a survey with end users and extracting factors by

performing factor analysis on the resulting survey data. During Phase 1 of this research study, a

survey instrument was created with a broad focus on all three generation of security analytic

tools. This researcher recruited security analytic industry experts, who had varied

implementation experience, to be part of a Delphi panel to build the initial instrument. The

experts in the Delphi panel and the four rounds of Delphi study contributed to the rigor of Phase

1 process. The participation of experts ensured that the survey instrument provided

implementation specific insights to (a) researchers in the area of security analytics, (b) product

vendors in the field of security analytics, and (c) security analytic tool implementers in any

organization.

Future researchers could apply the validated final survey instrument as a starting point for

more focused research. For example, a survey instrument with a specific focus on healthcare

industry or finance industry could be built based on this broader instrument. By determining

industry-specific implementation factors, this type of research attains more depth and maturity.

Vendors and implementers of security analytic products will benefit from the instrument and

factors that are identified by this research study. Some of the benefits are (a) these factors could

provide an initial assessment structure for product assessment, leading to improved product

design; (b) metrics that are defined based on these factors could help implementers to define

service level agreements both for internal use and external use with vendors; and (c) survey

instrument from this research study could be modified and enhanced by focusing only on the

area of tool adoption. For example, after a modification and confirmatory factor analysis, this

instrument could be used to test the conformance of security analytic family of tools to

technology acceptance model (TAM-3) that was defined by Venkatesh and Bala (2008).

Research Questions

Omnibus Research Question (ORQ): What are the factors that determine the successful

implementation of security analytics tools or packages?

Research Subquestion 1 (RSQ1): What are the factors that determine the successful

implementation of non-big data security analytics tools or packages?

Research Subquestion 2 (RSQ2): What are the factors that determine the successful

implementation of big data security analytics tools or packages?

Definitions of Terms

Big data. Big data describes large volumes of high-velocity data and variable data that

require advanced technologies for data management (TechAmerica, 2012).

Cyber-attack. A hostile act using a computer or related networks or systems, and

intended to disrupt and/or destroy an adversary’s critical cyber systems, assets, or functions

(Hathaway et al., 2012).

Delphi study. This will be the Phase 1 study. The Delphi method is an iterative process

for consensus-building among a panel of experts who are anonymous to each other (Garson,

2014).

Fraudulent behavior. In this research context, fraudulent behavior refers to fraudulent

hacker behavior in e-banking systems (Malekpour, Khademi, & Minae-Bidgoli, 2014).

Network analytics. A game-theoretic framework for modeling offender-defender

situations in computer networks (Roy et al., 2010).

Non-big data. Any dataset that is not a big dataset. Big data is a term for massive datasets

having large and complex structures with difficulties in storing and analyzing data (Sagiroglu &

Sinanc, 2013).

Probably approximately correct (PAC). The PAC model is the initial standard for

learning programs (Valiant, 1984).

Security analytics. Use of tools, methods, and algorithms that are useful in discovering

security breaches and attacks (Talabis, McPherson, Miyamoto, & Martin, 2014).

User acceptance. User acceptance in this research context refers to four TAM-3

constructs. They are PU, PEOU, behavioral intention, and usage behavior (Ahlan & Ahmad,

2014).

Research Design

The research design for this study was using a quantitative and nonexperimental Delphi

approach in Phase 1. In Phase 2, this study was using exploratory factor analysis (EFA) and

reliability analysis. This study consisted of two phases. Phase 1 was a Delphi study with the goal

of arriving at a consensus. The major focus in Phase 1 was the development of a survey

questionnaire. In Phase 2, the researcher conducted an EFA survey with the help of the survey

questionnaire and subjected the data to EFA and reliability analysis to extract the factors that

impact the successful implementation of security analytic tools.

Cardenas et al. (2013) argued that security analytics is a highly advanced technological

topic. However, academic research studies and investigations about this area of security analytics

still take place in nascent and developing conditions. A detailed search by this researcher did not

yield broad and expert-validated survey instruments in the field of security analytics, suitable for

the research questions of this study. Pinsonneault and Kraemer (1993) studied surveys during the

early days of IT growth. Accordingly, scholars invariably use exploratory models to develop

concepts and models of newly arrived fields of research. It is not possible to answer the research

question pertaining to this study by setting up an experiment for the following reasons: (a)

setting up a security analytics tool or SIEM for an experiment is an expensive proposition, (b)

even if the researcher set up an experiment, it would not be possible to simulate the typical

security events and load conditions that happen in a real-world setup, and (c) any results for this

type of research from an experimental approach would not be accurate. Pinsonneault and

Kraemer proposed that researchers conduct surveys to collect data to examine relationships

between variables. However, based on the literature review that follows, no pre-existing Likert

scale survey instrument, suitable for the research questions of this study was available. Hence the

researcher used the Delphi method to develop a survey questionnaire with the help of an expert

panel consisting of security analytic experts. The researcher subsequently performed EFA and

reliability analysis.

Data Collection (Delphi Study and EFA Study)

Security analytics, big data, and correlation analysis are areas of applied statistics that

researchers have increasingly used in data analytic tools in the past five to seven years. These

areas and related tools have not been probed with the goal of assessing their effectiveness post-

implementation. Based on the research design requirements of this study, a survey instrument in

the field of security analytics with a broad focus on all three generations of security analytic tools

is needed. Secondary security performance data on security analytic tools, available from many

private and nonprofit organizations, will not reveal the true picture, as these surveys are biased

towards the sponsors. Cybenko and Landwehr (2012) commented that such statistics may be

skewed because there is always a vested interest by major tool vendors sponsoring those surveys.

Delphi data collection usually takes place with a panel of experts, and there might be

three to four rounds of Delphi before panel consensus emerges. In this research study, the

researcher performed Phase 1 using the Delphi technique. In his evaluation of the Delphi

technique, Davidson (2013) asserted the importance of the principle of anonymity. Skulmoski et

al. (2007) mandated four major features of any Delphi study: (a) anonymity of Delphi

participants, (b) an iterative approach that allows the participant to refine his or her views on any

subject matter, (c) controlled feedback from the Delphi coordinator (researcher) to the

participants regarding other perspectives in a Delphi study, and (d) statistical analysis and

aggregation of the group response. Based on the above citations, during Phase 1 of this study, the

researcher circulated an initial questionnaire he had designed among a set of five experts and

then he refined it to establish content validity. This refined survey questionnaire was the first

document that went to the chosen panel of Delphi experts through e-mails. Until consensus

emerged, in each round the researcher collected input from the panel using e-mails. The

researcher used the refined survey in Phase 2 to conduct an EFA study through Qualtrics to

collect survey data. The researcher used data from this survey to perform EFA and reliability

analyses to extract the factors that determine the success of any security analytics

implementation.

Assumptions and Limitations

Assumptions

According to Orlikowski and Baroudi (1991), ontological assumptions have connections

to the nature of the phenomenon under investigation. The nature of phenomenon under

investigation may be objective, subjective, or a mix of the two. In the case of this security

analytics study, it is both subjective and objective, as the researcher solicited the opinions of

information security personnel. For this study, security analytic experts applied both their

perception of security analytic tools and their objective experience with the tools in answering

the survey questions.

Jupp (2006) clarified that epistemology deals with methods of achieving knowledge

about reality. The epistemological assumption of this study is positivist, which maintains that

human perception of information security-related data, in the form of security analytic tools,

observed over time, will reveal the true picture about these tools. Information security personnel

working with security analytic tools know from their experience the effect of these tools on the

overall safety of the organization and which factors influence the success of any implementation

most. Iivari (1991) asserted that methodological assumptions deal with choices of research

methods. The Delphi method, as explained by Okoli and Pawlowski (2004), uses consensus to

arrive at the final deliverable. It is the most suitable method for this research study. Pathirage,

Amaratunga, and Haigh (2005) mentioned that axiological assumptions relate to the value of any

given research study. In this study, the major stakeholders are information security personnel and

researchers. Apart from the above general assumptions, this researcher has identified a list of

specific assumptions for this study. These assumptions are explained in the following lines.

Experts in the field of security analytics are assumed to have good knowledge of the

implementation specifics and issues of security analytic tools. Participants for the Delphi study

were recruited using the LinkedIn professional network. It is assumed that Delphi participants

actually possess the expertise projected in their LinkedIn profile and can understand the Delphi

study orientation document. It was also important that Delphi participants would unbiasedly

participate in the study. This researcher also assumed that Phase 2 study participants were not the

same people who participated in the Phase 1 study and Phase 2 participants will not use proxies.

Limitations

Any restricting factor that limits the scope of this research is a limitation. The researcher

acknowledges a few limitations of this research study. The survey questionnaire validated in the

EFA study (Phase 2) may not be fully representative of the entire population of security analytic

tool users. Pinsonneault and Kraemer (1993) opined that cross-sectional surveys are not fully

representative of the target population. Lack of time prevented the researcher from performing

longitudinal surveys. Phase 2 survey was conducted only in the United States and Canada and no

other country was involved. Hence, this study does not fully reflect the security analytic usage in

other countries and is a geographic limitation. Industry related limitation is applicable to this

study because Phase 2 study focused only on retail, finance, healthcare and government sectors.

Other industries like manufacturing and energy sectors were not included in the study.

Organization of the Remainder of the Study

The remainder of the study focusses on the following major areas: (a) a review of extant

literature in security analytics and related areas, (b) an explanation of the research methodology,

CHAPTER 2. LITERATURE REVIEW

Overview

The increasing usage of security analytic tools and their impact on the safety of corporate

network and applications necessitated a closer examination of this phenomenon using a thorough

literature review leading to a survey instrument to support the research questions. A high-level

theoretical foundation suitable for this study included five theories. Those five theories are (a)

game theory, (b) MapReduce programming model, (c) TAM-3 theory, (d) computational

learning theory, and (e) Dempster-Shafer theory. The literature review provides an overview of

many independent research tools recently developed in the domain of security analytics.

Cardenas et al. (2013) defined three maturity levels for security analytic products, and this

research study has built a survey instrument that contains probing questions applicable to all the

three maturity levels. The literature review also evaluates the connection between the theories

mentioned above and the independent security analytic solutions. The literature review concludes

with a synthesis of the findings, gaps and a critique of the security analytic solutions including

that of commercial surveys done on security analytic products.

Methods of Searching

The search involved seeking out security analytics-related literature in many online

libraries. The researcher used six major sources to search for articles: (a) Capella University’s

online library and subscriptions to book and journal databases, (b) the Association for

Computing Machinery (ACM) online digital library, (c) the Institute of Electrical and Electronic

Engineers (IEEE) online digital library, (d) Google Scholar database, (e) ProQuest dissertation

database, and (f) SAGE journal database. The actual search strings that the researcher used were

(a) security analytics, (b) big data security analytics, (c) theories and security analytics, (d)

security analytics game theory, (e) security analytics computational learning theory, (f) security

analytics TAM-3 theory, (g) security analytics map reduce, (h) security analytics DS theory, and

(i) security analytics Dempster-Schafer theory. Capella University’s Summon search tool was

also extensively used in executing the above searches.

Theoretical Orientation for the Study

Foundational Theories

At the core of this research is the subject of security analytics. As this subject involves

connection to the entire landscape of IT within an organization, many foundational theories can

explain the functionality of security analytic tools. One of the significant components of security

analytics is network analytics. It is easy to model network behavior and to extract intelligence

from these models with the help of game theory and evidence theory. Liang and Xiao (2013)

applied game theoretical concepts to network security by exploiting game models inherent to

game theory. Dempster and Shafer developed DS theory and Shafer (1976) named it as the

theory of mathematical evidence. Big data based security analytic products invariably use

MapReduce theoretical concepts in reducing large datasets to smaller datasets, suitable for

extracting actionable insights. Dean and Ghemawat (2010) introduced MapReduce as a

programming model suitable for parallelizing and executing large programs using multiple

computing resources. Venkatesh and Bala (2008) presented technology acceptance model

(TAM-3) to explain the adoption of new technologies by the user community. Application of

TAM-3 constructs to the family of security analytic tools will explain the adoption maturity of

those tools. Machine learning concepts explain the learning component of security analytic tools.

Computational learning theory and its application are also useful to the domain of security

analytics. Goldman (1995) provided an explanation for the assessment of learning algorithms

using computational learning theory. A detailed treatment of the above theories is in the

following section. Figure 1 pictorially depicts the theoretical foundation for security analytics.

Game theory. A game in the simplest case is an interaction between two entities in any

given situation. For example, in a game of chess, there are always two players (entities) in play.

Barron (2013) mathematically described games as involving a number of players (N), a set of

strategies for each player, and a quantitative payoff for each player. Von Neumann and

Morgenstern (2007), for the first time, defined game theory in 1944 as a strategy of games to

solve problems in economics. Turocy and von Stengel (2001) discussed the application of game

theory to bidding in online auctions. Game theory is a mathematical tool that can describe and

solve games (Liang & Xiao, 2013). Liang and Xiao further elaborated about game theory in

their classic introduction on the following salient aspects: (a) Category 1 – based on number of

stages in a game, games are classifiable as static games, dynamic games, or stochastic games, (b)

Category 2 – based on information available on player’s actions, games are classifiable as games

of perfect information and games of imperfect information, (c) Category 3 – based on the

completeness of information available for players, games are classifiable as games of complete

information and games of incomplete information, and (d) equilibrium, which is a result or

combination of players’ strategies.

Figure 1. Security analytics – Theoretical foundation.

Nash (1950/1997) for the first time established the Nash equilibrium to indicate that finite

games have an equilibrium point. At this point of equilibrium, all players choose the best

possible action given the decisions their opponents make. Non-cooperative games are quite

popular in the world of information security, as such games model attacker and defender

situations.

Game theory and security analytics. Liang and Xiao (2013) opined that network

security is a typical game between attackers and defenders. Analyzing network traffic and related

data is a necessary prerequisite for bolstering network security, and it is an integral part of

security analytics. Theoretical inputs play a major role in modeling network situations. Game

theory has a wide variety of applications in network security. Roy et al. (2010) presented a

thorough survey of applications of game theory and argued that game theory-based solutions fill

the lack of a quantitative decision framework in traditional network analytics.

The Nash equilibrium is widely applicable in a multiple player situation. As per this

principle, it is not possible to predict the results of decisions emanating from multiple players in

a game situation by judging players in isolation. Mohammed, Fung, and Debbabi (2011) applied

this principle to solve a data integration problem for very large databases. Usually, an integrated

dataset obtained by joining multiple data sources reveals sensitive data. To solve this problem,

researchers developed a common algorithm based on Nash’s equilibrium principle in which

every data integrator participated. This algorithm isolated malicious participants. This technique

can detect insider attacks. One of the goals of insider attacks in an enterprise is to procure the

sensitive data of the employees. For example, merging of disparate sources of endpoint data can

reveal sensitive data relating to employees. The Nash equilibrium technique can be applied to

thwart such insider attacks. Security analytic tools involve extensive use of correlational

algorithms and data integration. Mohammed et al.’s research project is easily extendable and

applicable to any commercial security analytic product.

Honeypots are traps set for attackers by defenders (Spitzner, 2003). Deceiving attackers

is an evolving strategy based on game theory. Carroll and Grosu (2011) explained their

deception strategy using a comprehensive set of algorithms and dynamic games. Han (2012)

referred to dynamic games as situations in which one player has information about the other

player’s strategy or moves, and multiple moves take place over time. In this deception game, two

sets of players, the defending network and the attacking network, are involved. Camouflaging

normal computers as honeypots and vice versa is a strategy that defenders adopt. Attackers are

usually at a disadvantage in this deception game, as they are not the first movers of the game.

Since logging is a powerful feature of honeypots, security analytic tools will easily be able to

integrate these log files for analysis.

Games in which players do not know the payoffs or preferences of their opponents are

games with incomplete information (Han, 2012. Liu, Zang, and Yu (2005)) modeled a defense

system with the aim of capturing a DoS attacker’s intent as opposed to the attack pattern. The

attacker’s incentive lies in the core of his or her design. One of the incentives is to access

classified internal documents. This design used data from attacks to study and understand the

intent of the attacker. The game design of this model used a six-tuple game with two players

(defender and attacker), two strategy spaces (one each for attacker and defender), a Bayesian

game type, and a set of outcomes. A Bayesian game is a game in which there is incomplete

information about the other players in the game. In this scenario, there is less information about

attackers. While this approach was not a perfect design in terms of accuracy, this model inspired

many more research projects that applied game theory to security analytics.

Fielder, Panaousis, Malacaria, Hankin, and Smeraldi (2014) addressed the challenge of

supporting the decisions of security administrators and other personnel with respect to protecting

the information-related assets of any organization. They used game theoretic modeling

techniques to model the situation between the attacker and a team of administrators to establish

the Nash equilibrium. One of the aspects of security analytic tools is to model the attacker-

defender situation. This can lead to the determination of the number of security analysts

necessary to pursue the actionable intelligence the tool provides. Game theoretic techniques are

easily applicable here to solve the challenge of cybersecurity resource allocation.

Chung, Kamhoua, Kwiat, Kalbarczyk, and Iyer (2016) argued that game theory

techniques can combine with learning algorithms like Q-learning to react to adversarial behavior

in a network situation. Their novel technique does not need complete information from the

opponents. Security incidents belonging to the intrusion category are the main focus of this

approach. Q-learning is a model-free reinforcement learning technique, and it can model games

with incomplete information. Q-learning mainly learns from human behavior. For example, Q-

learning can learn from the type of decisions a security analyst takes. It also learns from the data

of earlier iterations.

Computational learning theory and machine learning. Computer programs that can

perform tasks with the help of human-like intelligence and learn and improve their performance

over time act as learning programs. Goldman (1995), in her elaborate introduction to

computational learning theory, explained the initial work in the early sixties in machine-based

learning in terms of learning theory offering a foundation for assessing learning algorithms.

Leeuwen (2004) introduced machine learning using simple and yet powerful examples. Some of

the typical examples he gave are (a) pattern recognition in an array of images, (b) identifying

words in a handwritten text, (c) discovering and extracting common information from distributed

data, and (d) speech recognition. Most security analytic tools incorporate one or more learning

algorithms that focus on learning about attacks. For example, learning algorithms easily spot

botnet attacks.

Computation learning theory is a branch of theoretical computer science that studies

machine learning with the help of a strong mathematical foundation. Gold (1967) provided the

first formal definition of learning. Gold’s theory of learning focused on the learner guessing a

rule behind a data sequence leading to a convergence of the sequence. However, Gold’s theory

of learning did not attempt to evaluate and assess the efficiency of a learning process, whereas

computational learning theory provided a framework to evaluate the learning process. Goldman

(1995) reiterated that computational learning theory provided a framework to compare and rate

different learning algorithms. From the perspective of assessment, computational learning theory

has a close connection to machine learning.

Probably approximately correct (PAC) learning model. The seminal work of Valiant

(1984) initiated computation learning theory in the form of the PAC model. The PAC model is

the initial standard for learning programs. Leeuwen (2004) provided a convincing mathematical

model for PAC. He explained that the goal of a PAC learning algorithm is to classify samples

generated from a sample space (X) into a concept space (C). Based on the samples, the researcher

forms a hypothesis, and with every sample, the researcher adjusts this hypothesis. A hypothesis

is good when the number of errors in classifying samples remains below a predefined bound. A

problem is PAC-learnable if there is an algorithm (A) which for any distribution D and any concept c

will when given some independently drawn samples and with high probability, produce a near error-

free hypothesis (h). Concept learning is a generic term that includes the PAC learning model.

Blum (1994) introduced the two popular phases involved in concept learning. Those two phases

are the training phase and the testing phase. During the training phase, the learning procedure

studies some examples and produces a hypothesis. During the testing phases, it evaluates the

hypothesis.

Application of PAC to fraud detection. A PAC learning algorithm can be either strong

or weak. Leeuwen (2004) described a weak learning algorithm as one that performs poorly in

terms of learning. A weak learning algorithm has less accuracy in terms of classification. This

phenomenon is likely due to not having a large training dataset. There are a few meta-learning

techniques that can boost the accuracy and performance of a weak learner. One such popular

technique is boosting. Boosting turns a weak learning algorithm into a strong learner. Malekpour

et al. (2014) elaborated a hybrid model approach in predicting fraudulent behavior in e-banking

systems. Security analytic tools ideally incorporate similar models, both supervised and

unsupervised, to identify financial frauds in banking institutions. Log sources feeding into

security analytic tools form the major inputs to these hybrid models. Hybrid models include both

classification and clustering techniques. The ensemble approach employs boosting algorithms to

increase the model accuracy. Malekpour et al. used this approach to increase their accuracy in

detecting frauds. Increased accuracy was the result of comparing results with earlier techniques.

Boosting the PAC algorithms was the key to improving accuracy.

Dempster-Shafer (DS) Theory and its applications. Science has succeeded in

modeling many aspects of uncertainty in daily life. Kohlas and Monney (1994) introduced the

theory of evidence with a strong computational flavor with the aim of modeling uncertainty.

Researchers also call this theory DS theory. Shafer (1976) released it as a theory of mathematical

evidence. As per Shafer, the main goals of this theory are to represent uncertainty with the help

of evidence and hints. Statistical modeling of uncertainty was a successful outcome of this

theory. Kohlas and Monney elaborated on the application of evidence theory in the area of

decision analysis. Decision analysis is an important application of evidence theory and is applied

in the area of intrusion detection. Detecting intrusion is one of the major functions of security

analytic tools. Database log files are a major input to security analytic tools. Intrusive activity in

databases is usually detected by a pre-defined set of rules. Panigrahi, Sural, and Majumdar

(2009) explored the application of evidence theory to intrusion detection in databases. Rules

provide evidence of intrusive behavior in database transactions. However, many pieces of

evidence are combined to form an initial belief in any given transaction. This combination of

evidence is accomplished using DS theoretical constructs as explained in the following section.

Database intrusion detection, a less popular concept, identified user behavioral anomalies

with a focus on preventing insider attacks. Panigrahi, Sural, and Majumdar (2009) elaborated on

a unique algorithm they developed to detect suspicious user activity in databases. Two rules

formed the initial belief function regarding whether a specific user activity is malicious or

normal. The first rule measured the sequence of activities in a transaction and any deviation from

the prebuilt user profile. The second rule detected spatiotemporal outliers in user behavior with

specific reference to location and time of user activity. The belief component of the detection

engine combined output from both the rules that act as evidence to determine malicious

behavior. Panigrahi et al. tested their algorithm using a transaction simulator. Rules in this

experiment are easily extendable to any security analytic tool. Weblogs and database logs are

suitable inputs to the security analytic tool in the application of these rules.

Detecting insider attacks is a high priority for many of the security analytic tools. Mobile

ad hoc network (MANET) is an evolving network model in communication networks, and it is

subject to insider attacks. Ehsan and Khan (2012) presented a detailed analysis of the security

attacks that normally happen in mobile ad hoc network (MANET). Many algorithms are

currently developed by researchers to combat MANET attacks. Wei, Tang, Yu, Wang, and

Mason (2014) developed an algorithm based on DS theory to fuse indirect information provided

by one-hop nodes located close to the mobile node under observation. The main purpose of this

fusion is to develop trust with the observed node. Indirect observations involved collecting and

fusing evidence of an observed node based on interactions between other nodes located close to

the observed node. Wei, Tang, Yu, Wang, and Mason applied the DS-theory concept of belief

function probabilities to develop a trust score for every node, and they stored the score in a

routing table to prevent insider attacks. The algorithm checked the routing table for a trust score

to understand whether there were any compromised nodes in the network. Identifying

compromised network nodes is an integral part of any security analytic tool, and the above

algorithm is suitable for next-generation security analytic tools.

Intrusion detection systems (IDSs) monitor network traffic with a view to identifying

intrusive or malicious activity. Most of the intrusion detection systems use signatures that are

embedded into the IDS sensor. Zomlot et al. (2011) considered the problem of false positives as

one of the major reasons for the failure of intrusion detection processes. For example, the sensor

might detect an attack scenario when it really does not exist. The belief functions provided by

DS theory solved this problem of false positives. Pursuing an attack alert by a human analyst

needed the enablement of evidence-based beliefs. Zomlot et al. applied DS theory’s evidence

fusion approach successfully to produce actionable inputs for a human analyst. One of the

achievements of this unique algorithm was its ability to stand the tests performed in a real

production system. Logs from intrusion detection and prevention devices regularly go to security

analytic tools to help in the analysis of attacks and false positives.

A major function of security analytic tools is the ability to fuse alerts generated from

heterogeneous inputs. Those inputs could be (a) intrusion detection devices (IDS), (b) web

application logs, or (c) firewall logs. Intelligent fusion of these alerts can greatly reduce the

number of alerts and false positives. Yu and Frincke (2005) proposed an algorithm to fuse alerts

based on extended DS theory. Original DS theory takes into consideration all the individual

alerts (observers) with equal weightings. However, in the algorithm, Yu and Frincke proposed,

weighting goes to the source of the alert. For example, different inputs may detect a distributed

denial of service (DDoS) attack with different levels of confidence. A web application log and an

IDS device may report a SQL-injection attack with different levels of confidence. Different

weights go to the different alert sources, and the final fused alert takes these weights into

consideration. Alserhani (2016) brought out the importance of alert data fusion and its unique

strengths in supporting cyber analytic tools and processes. The application of DS theory has

clearly resulted in a reduction in the time a security analyst dedicates to the correlation of alerts.

Algorithms that are based on DS theory perform most alert correlation tasks accurately.

MapReduce model and framework – History and basic principles. MapReduce

originated as a proprietary Google technology, but it is now genericized. Qin et al. (2012)

described MapReduce as a highly scalable, parallel computing framework. It is useful for

processing large data sets in distributed environments or clusters. The fault-tolerant framework

has two main components: map and reduce functions. The map function filters and sorts the

input into key-value pairs. The reduce function groups and aggregates the input key-value pairs

based on the maps. Dean and Ghemawat (2010) explained Google’s MapReduce model in one of

their introductory papers. Some of the salient features of the MapReduce model from that paper

are (a) MapReduce automatically parallelizes and executes programs on large machines; (b) the

run-time system takes care of splitting the data, scheduling the parallel jobs, handling machine

failures, and managing inter-machine communication; (c) MapReduce jobs use inverted indexes;

(d) MapReduce can handle data from heterogeneous systems; (e) MapReduce does not need to

perform full scans of entire input file; and (f) MapReduce produces a high level of inbuilt fault

tolerance by not needing to restart failed jobs from scratch.

MapReduce and its applications. Hu et al. (2014) presented Hadoop’s massive data

analytical capability, based on MapReduce framework’s group-aggregation model of computing.

Security analytics predominantly deal with large volumes of unstructured data that fulfilled the

six Vs of big data, namely velocity, volume, veracity, variety, value, and variability. Tripathi,

Gupta, Almomani, Mishra, and Veluru (2013) proposed the self-adaptive MapReduce scheduling

(SAMR) algorithm, a Hadoop-based solution, to counter DDoS attacks. This solution focuses on

the DoS type of malware attack.

MapReduce is a great solution for performing analytics on data at rest. Schales et al.

(2011) explained the application of MapReduce to performing analytics for data at rest. The

stream computing platform for analyzing threats involves combining historical data and real-time

streaming data to extract actionable intelligence. Analyzing the historical data of many years in

memory demanded a huge hardware infrastructure. Hence, Schales et al. decided to apply the

MapReduce framework to perform analytics on historical data. The unique batch processing

abilities of MapReduce solved the problem of scalability in a stream computing platform.

MapReduce has found extensive application in security analytics. Botnets are networks of

infected computers that spread malware. Singh, Guntuku, Thakur, and Hota (2014) designed a

scalable machine learning approach to detect botnet intrusion in near real time. Security analytic

tools have similar functionality to detect botnet attacks in near real time, and ideally, they should

incorporate similar machine learning approaches. After a network sniffer module has

successfully extracted network flow traces from network dataset, a MapReduce algorithm can

extract a feature set. An example of a feature is total_fpackets (total number of packets in the

forward direction). A machine learning module studies the features and applies machine learning

to detect malicious traffic. Buczak and Guven (2016) surveyed machine learning algorithms

suitable for intrusion and botnet detection and described the application of these algorithms.

Singh, Guntuku, Thakur, and Hota highlighted Apache Hive data warehouse and Hive query

language as useful tools in the feature identification and extraction process. Their big data based

approach has achieved peer-to-peer botnet detection.

TAM. The TAM, initially proposed by Davis (1986), has found its deserving place in IT

and information systems-related research. The TAM has its roots in the theory of reasoned

action. Davis applied the concepts of stimulus, organism, and response, taking into consideration

the information explosion that was to happen in the mid-eighties. Davis mapped the information

system’s user interactive features to stimulus, linking user motivation to an organism and actual

usage to response. User acceptance and perception of technology played a major role in

technology acceptance.

As per Lee et al. (2003), two major variables of the TAM that revolutionized the user

acceptance research were perceived usefulness (PU) and perceived ease of use (PEOU).

Behavioral intention and behavior were two more variables that definitely added value to TAM

research. While Lee et al. included many categories of IT research in their paper, there was no

direct mention of information assurance or security analytics in their research. Security analytics-

related research is a relatively new category, and user perception and acceptance is undergoing

research and is in a state of evolution. Categorizing security analytics as expert systems and a

contemporary of e-commerce systems, it is predictable that TAM theoretical constructs could

easily assess the user acceptance of security analytic technology tools and products.

TAM-3. User motivation to accept a new IT technology is not necessarily to be taken for

granted by product vendors and researchers. Venkatesh and Bala (2008) discussed the huge loss

experienced by Hewlett Packard and Levi Strauss in the early part of this century due to poor

user acceptance. Identifying interventions to aid the process of user acceptance and helping the

managerial community to make informed decisions were the major triggers leading to TAM-3.

Venkatesh and Bala argued that TAM-2 is deficient in PEOU and theorized that more

determinants would influence PEOU. In 2008, they introduced an integrated model (TAM-3),

comprised of TAM-2 features plus new determinants of PEOU and three new relationships.

TAM-3 is now an accepted standard in IT-related research.

Adoption of emerging technology in healthcare is an ideal topic to explain the application

of TAM-3. Ahlan and Ahmad (2014) explored the adoption and acceptance of healthcare IT in

developing countries. They successfully explored four TAM-3 constructs in their study: (a) PU,

(b) PEOU, (c) behavioral intention, and (d) usage behavior. The perceived threat had a negative

effect on the user acceptance of a new technology. Even though a similar in-depth study is not

the focus of this research on security analytics, these concepts were introduced in the field study

questionnaire. Those questions focused on the perception of security analytic systems by users in

terms of user interface and training impacting their perception and usage of the system.

Security analytics as a technology has been in use since 2005. Hinde (2005) underscored

the importance of security analytics in an introductory article that dealt with computer fraud and

security. Kent (2007) published the first set of standards for SIEM and log management as a part

of National Institute for Standards and Technology documentation. Grublješič and Jaklič (2015)

discussed the maturity of business intelligence systems and their acceptance in the IT domain.

Business intelligence systems is a very mature technology, but security analytics is growing in

terms of acceptance. A new technology grows from an adoptive stage to an embedded stage.

Security analytics technology is still in the acceptance and adoption stage. Some of the relevant

additional determinants highlighted by Grublješič and Jaklič for better technology acceptance

relate to: (a) individual, (b) technological, (c) organizational, (d) social, and (e) macro

environmental characteristics. Kavanagh, Rochford, and Bussa (2015) explained the process and

implementation details involved in the implementation of various security analytic tools. Based

on the discussion provided by Kavanagh, Rochford, and Bussa, the researcher introduced

implementation and process focused questions into the field study questionnaire to ascertain the

acceptance of security analytic tools by the user community.

Review of the Literature

Security Analytics – Recent Research

Security analytics deals with a large volume of data. Traditional database management

tools are not suitable to deal with large data. Oltsik (2013) defined the need for big data based

processing to deal with large datasets. As per Schales et al. (2011) most structured large data

analysis acts upon data at rest, while real-time analytics acts upon denormalized and raw data,

also known as data in motion. Freshly accumulated data in a web proxy log is an example of

real-time and denormalized data. For example, structured data analysis is roughly comparable to

biannual performance appraisals, whereas real-time analysis is a closer match to immediate, 360-

degree peer feedback. Real-time analysis and data correlation become necessities in analyzing

security data in motion. For example, if an APT attack needs identification and remediation, a

thorough scan of a few terabytes of stored data and real-time stream data is necessary. Cardenas

et al. (2013) compared the act of searching large datasets for cyber-attacks to searching for a

needle in a haystack and brought out the need for big-data based analytics. Analyzing threat logs

is a simpler exercise than correlation analysis, as Mahmood and Afzal (2013) explained.

Malware and worms can be easily identified using log analysis process whereas identifying

complex cyber-attacks involved correlation analysis process. Chari, Habeck, Molloy, Park, and

Teiken (2013) demonstrated an analytic approach combining both a traditional data-management

platform and a big data platform to extract access-related anomalies for enterprise users. Hence

security data analytics performed using large-scale data analysis in real-time is becoming a

necessity to manage cyber-attacks. However, such big data analysis demand a large

infrastructure, highly skilled data scientists, and above all, a strong business case for investment.

Many small enterprises are not ready for this type of investment. Nicholas and Kavanagh

(2013) discussed the importance of investing in security analytic tools. A survey-based study

with the participation of end users who have implemented security analytic tools definitely adds

value to CXO-level decision process in the domain of security analytics. Shackleford (2014)

conducted a survey with a focus on security information and event management (SIEM) tools.

Shackleford’s survey focus was on large enterprises. However, an end-user driven survey leading

to identifying critical implementation factors will help enterprises of all sizes. Many of the

available case studies in peer-reviewed journals point to both structured and big data analytic

solutions. Some of the best among such solutions have come from security leaders such as IBM

and RSA. For example, Kavanagh, Rochford, and Bussa (2015) discussed IBM’s QRadar and

HP’s ArcSight platforms. Even though these tools originated from technology leaders like IBM

and HP, empirical and critical evaluations of these tools and success factors performed with the

help of end-user driven surveys are scant in the literature. Evaluation of security analytic tools

using end-user driven surveys will also generate research ideas for more in-depth research in the

area of security analytics. A discussion of some of the recent advances in the security analytic

tool domain is provided in the following section.

Stream computing – Behavioral and threat analysis. Yen, Oprea, and Onarlioglu

(2013) challenged the security analytics industry with a unique design of big-data-based

analytics solution, called Beehive, which can perform threat, policy, and user behavioral

analysis. Beehive has three layers: normalization, feature generation, and clustering. Network

logs are the major input for this tool. Cardenas et al. (2013) discussed Beehive results and

claimed that Beehive could process one billion log messages in one hour. While this claim may

be empirically correct, accuracy in predicting user behavioral anomalies is the correct way to

judge these types of solutions. A critical read of the Beehive tool article indicated a few

deficiencies: (a) Cardenas et al. only tested beehive with two weeks’ worth of data, and (b) they

only tested it for the EMC corporate network. Stream computing is near real-time analytics, and

its main use is network data analysis. Schales et al. (2011) illustrated the power of stream

computing platforms that laid emphasis on enhanced middleware. They used a year of real data

to stress test this platform. Using the front-end tools of the platform, a security analyst can create

ad-hoc anomaly detection logic independent of the architectural components. The Beehive

solution by Yen et al. and the stream computing platform by Schales et al. are both comparable

from an architectural angle, but not from a performance perspective, due to the large data set in

use in stream computing. However, the Beehive tool can tackle simple user profiling and

suspicious host behavior well. Analyzing security analytic tools from a cyber-attack perspective

gives a deeper understanding of the tools’ capabilities. The next section discusses security

analytic solutions for different types of attacks.

Combating attacks using security analytics. Insider attacks are a common phenomenon

in today’s enterprises. Chari et al. (2013) proposed an analytic solution for detecting policy

violations by the user community. Users sometimes innocently share their credentials with their

colleagues, leading to misuse. Ineligible users access intellectual property documents due to

privilege elevations. Forming a base user profile and studying user login attempts based on user

geography and human resources-related data, this platform accurately pointed out anomalous

user behavior. Comparing this tool with the Beehive analytic tool developed by Yen et al.

(2013), it is easy to determine that Beehive had broader goals. But a major strength of the tool

developed by Chari et al. was the focus on avoiding false positives. For example, if a user has

concurrently logged on to the enterprise’s intranet, from both his or her iPhone and his or her

laptop, the system should not flag this as anomalous behavior, leading to false positives. This

tool eliminated the most common type of false positives. Many security analytic tools have

problems with false positives.

Botnet attacks are executed by a command and control center that is controlled by the

hacker community. Botnets consist of a series of infected devices connected to the Internet.

Crespo and Garwood (2014) described an ideal SIEM analytics engine as one with a built-in

scalable correlation module and the ability to handle new threats. Cardenas et al. (2013)

classified this type of solution as a third-generation, big data security analytic tool. This tool

performed net flow analytics using the MapReduce theorem to identify and quarantine botnet-

infected computers and used Hadoop clusters to reduce the processing time by a factor of seven.

The advanced cyber defense center (ACDC) project described by Crespo and Garwood scaled

new heights in Europe. This project served as a great attempt to conquer the botnet attacks.

MITRE Corporation developed the structured threat information expression language to

communicate cyber threats, and the ACDC project extensively employed this language. The

ACDC project conducted many pilots and handled many variations of botnet attacks. The size of

the ACDC project was far larger than the Net flow project described by Cardenas et al. (2013).

Judging by the tools it developed, the ACDC program was an ideal security analytics program,

since it made use of extensive collaboration, exploited all the available cybersecurity resources in

the European Union, and it justified the large resource usage.

An APT is a targeted attack against a high-value asset or a physical system (Cardenas et

al., 2013). Stealth mode is the master sign of this type of attack. Identifying APTs from the logs

is a labor-intensive project. The Beehive log analyzer developed by Yen et al. (2013) focuses on

APTs and tries to identify pre-attack probes as a part of the user behavior analysis, but it does not

mention anything about reducing false positives. A strategy that synthesizes the approach of the

Beehive analyzer and the access control analyzer developed by Chari et al. (2013) in a suitable

analytics platform could greatly reduce the possibility of an APT attack.

Big Data Security Analytics

Big data security analytics-based tools and processes have created a powerful impact on

the information security industry. Oltsik (2013) argued that big data security analytic tools are

natural extensions to log management and SIEM tools. SIEM technology-based tools brought

about a major shift over log management tools by providing better functionality in terms of

interfaces to security operations center (SOC) and user behavioral analytics. Chuvakin (2016)

highlighted the importance and benefits of user behavioral analytics and anomalous behavioral

analytics using big data tools. He stated that companies are even building custom-based big data

tools to independently analyze enterprise-wide security and log data. Oltsik divided big data

analytics into two major areas: (a) real-time big data analytics and (b) asymmetric big data

analytics. Oltsik stated that real-time big data analytics focusses on conventional database

repositories with limited querying capability and data log feeds. Asymmetric big data analytics

focusses on a typical big database platform like Hadoop with flexible querying capability and the

ability to consume heterogeneous data feeds.

Oltsik’s (2013) further elaboration equated real-time big data analytic tools to standard

SIEM platforms released by popular tool vendors. These tools do not include big data platforms

like Hadoop. However, asymmetric big data analytic tools or platforms include Hadoop-like

server clusters including the MapReduce component. These solutions are far more scalable than

real-time big data solutions. Assuming a continuum to define big data security analytic tools,

real-time tools are at the lower end of the continuum and asymmetric tools are at the higher end

of the continuum. Very few vendors have built separate tools for real-time and asymmetric

categories. Chuvakin (2016) summarized the key success factors for implementing big data

security analytic tools: (a) understanding of data science approaches, (b) ability to collect and

retain data, and (c) availability of people to use the tools and prepare data. A discussion of some

of the stellar research efforts in big data-driven security analytics follows with a focus on

extracting applicable concepts for the Delphi study. These discussions will also help in

enumerating the gaps in terms of assessment of these tools.

Wheelus, Bou-Harb, and Zhu (2016) designed and presented a big data-based

architecture that focused on threat intelligence analytics. They deviated from the conventional

approach by emphasizing on building intelligence use cases and tested this solution on large, real

network traffic. They exploited several big data tools and technologies as a part of this study.

They used machine learning algorithms to amalgamate heterogeneous network data sources to

create effective labels to create labeling artifacts. Attack classification and zero-day malware

inference were the major achievements of this tool. Streaming data and packet capture data

formed the major inputs to this tool. A commonly used Libpcap library file format was used to

interpret the input data. This tool included Apache Kafka that ingested streaming data into the

tool in the form of queues. Each queue had multiple feeders and multiple consumers. The

processing domain of this tool had two major focus areas, viz., validation of input data and

identification of network features. The tool adopted Hadoop distributed file system as the storage

medium and the HIVE component to query the data. The output (artifacts) this tool produced

supported predictive analytic models. Wheelus et al. conducted three case studies using the

above-mentioned tool. The first case study detected Sality malware, which is formidable, with

90% accuracy. They used artifacts from the output of this tool to train the system for specific

malware analysis. The second case study classified and distinguished different malware families

with only 1% false positives. The third case study found that the prediction model easily

predicted zero-day malware and its attributes. Shackleford (2016) stated that ability to unravel

new attack patterns is an ideal attribute of any security analytic tool. Wheelus et al. built a unique

architecture that had the ability to identify zero-day malware. However, one possible futuristic

direction for this tool would be to test the architecture with unlimited real-time data.

Most of the security analytic tools provide insights into network anomalies through

monitoring and correlation analytics. Chuvakin (2010) explored SIEM technology and

highlighted network anomaly analytics as a major use case in security analytics. Anomalous

network behavior analytics has since become a crucial force in the field of security analytics.

Bhuyan, Bhattacharyya, and Kalita (2014) surveyed network anomaly detection tools and

methods in their widely cited foundational work. Network anomaly detection functions are an

integral part of this research study, and they are inbuilt into any security analytic tool. It is

pertinent that this study discusses the basics of network anomaly detection tools, leading to

refined inputs for field study in this research. Anomalous network behavior detection refers to

the problem of finding exceptional patterns in network traffic that are deviations from normal

behavior. Anomaly and outlier are two common terms used to describe this problem.

Crespo and Garwood (2014) explained the role of anomaly detection in SIEM tools and

its ability to detect leakage of sensitive personal identifiers (SPI) and other security events. One

advantage of anomaly-based detection is that it can detect known as well as unknown attack

patterns. Security-related anomalies are of three types: (a) point anomaly, (b) contextual

anomaly, and (c) collective anomaly. Point anomaly refers to an isolated occurrence of a data

point from normal occurrences at any instant of time. This could be an outlier network flow

event. Contextual anomaly refers to abnormal behavior that happens within a given period. For

example, multiple transactions within a short time may mean a stolen credit card. Collective

anomaly refers to a collection of related data instances, but the individual instances are not

anomalies. For example, buying a lot of high-priced items by a group of card users is a collective

anomaly. Shackleford (2016) opined that machine learning techniques are now in widespread use

in providing guidance on anomalous behaviors.

Bhuyan et al. (2014) elaborated on the application of machine learning methods for

solving network anomaly problems. Matching of new instances of network data with an already

existing profile usually occurs using machine learning algorithms and a training dataset. Stored

network data profiles act as reference data and are regularly updated by batch jobs. Anomaly

detection is a problem of big data classification (supervised approach) or clustering

(unsupervised approach). Support vector machine classification scheme is a common method of

anomaly detection for security analytics. Hierarchical and ensemble-based clustering techniques

are common in the unsupervised category.

A recent trend in anomaly detection is the application of graph analytics and forensic

analysis to detect anomalous behavior. Puri and Dukatz (2015) designed a big data-based

architecture to detect APT attacks and network anomalies using graphs. They used data from a

live corporate network to pilot test this tool. Three major focus areas for this tool are (a) learning

of network behavior and transformation of network events to probabilistic event graphs, (b)

application of machine learning techniques in anomaly identification, and (c) real-time anomaly

detection. Network log files aggregated from SIEM tools were the main input to this tool. Some

major components of this tool are (a) a big data platform consisting of a big data core and

computing power, (b) a complex event processing (CEP) module, and (c) an in-memory database

system. Log content analysis (LCA) graph learning framework is the application module that

analyzes and presents the network events. Puri and Dukatz analyzed anomalous behavior within

the log content analysis module. The anomalous behaviors this tool identified provided insights

into the inner workings of the network and provided great inputs to the security analysts.

Anomalous big data-based behavioral analytics are used for generating evidence to

pursue investigations on compromised network machines. Probing attacks usually happen before

a botnet takes over a computer. Bou-Harb, Scanlon, and Fackha (2016) applied the concept of

behavioral service graphs to investigate infections. Perceived probing activities as inferred by

this tool can rapidly identify infected machines. Fusing big data analytics with formal graph

theoretical concepts generates the actionable forensic evidence necessary for investigations.

Security personnel has thwarted botnet campaigns by applying this actionable intelligence to real

darknet traffic. The forensic insights produced by this tool were highly accurate.

Distributed denial of service (DDoS) attacks involve the flooding of network data packets

to choke and exhaust the network bandwidth. This type of attack will result in denying service to

the user. Tripathi et al. (2013) discussed a unique solution called the self-adaptive MapReduce

scheduling (SAMR) algorithm to deal with the abnormal network traffic management problem

created by a DDoS attack. The SAMR scheduling concept distributed the traffic load using a task

parallel concept. Tripathi et al. did not elaborate on results. Judging from a performance

perspective, the advanced cyber defense center (ACDC) project, as described by Crespo and

Garwood (2014), could combat DDoS attacks using a data clearinghouse. The central

clearinghouse for analytical input data was a unique design of the ACDC project, and hence the

ACDC project scores better than SAMR algorithm.

The multicore approach refers to the parallelizing of tasks or data, or both, depending on

the correlation requirements. Cheng, Azodi, Jaeger, and Meinel (2013) projected the importance

of performing correlation analysis using in-memory processing techniques. The security

analytics lab architecture they proposed implemented all the popular correlation algorithms,

namely k-means, y-means, the robust clustering algorithm for categorical attributes, and a quick

version of the robust clustering algorithm for categorical attributes. From an analysis of results of

this experiment, it is easy to determine that Cheng et al. did not use real-time data to test the

multicore design. Cheng et al. contended that they used simulated data to test the performance.

Comparing the multicore solution with other analytic solutions, it is evident that performance

testing executed with simulated data will not prove the scalability of the solution. However, this

project is a great innovation in security analytics.

Strengthening the network defense of an enterprise is the main aim of security analytic

solutions. Colbaugh and Glass (2011), in their revolutionary research funded by Sandia National

Labs, delineated the possibility of using predictive analytics to predict attacks to empower

defenses. The two approaches they used in this proactive solution were (a) a transfer learning

algorithm that uses prior knowledge of attacks to predict zero-day attacks and (b) the generation

of synthetic data to train the cyber defenses of any organization based on the type of attack. They

tested both approaches against two large, publicly available datasets. Choosing a predictive

analytics approach over other approaches is a wise decision for an environment that is more

isolated, smaller, and better controlled.

Glass and Colbaugh (2011) forayed into web security analytics and developed a

framework using web crawling concepts. This predictive solution does not focus on the network

domain, but on text, relational, and temporal analytics, which can help users to understand web

events. Splunk is a powerful tool in security analytics. Zadrozny and Kodali (2013) explained the

functionality of Splunk forwarder in forwarding log data into a Splunk instance. Charishma and

Venkatesh (2015) studied application weblogs and used Splunk forwarder to identify and report

on cross-site scripting and brute force types of attacks. Even though web-analytic platforms do

not focus on the network side of analytics, assessing the reach of this algorithm indicates that it is

necessary to integrate web security analytic platforms into network analytic platforms. Web

analytics are a part of any security analytic commercial product and weblog sources feed into

security analytic tools for the purpose of correlation.

Bowers, Hart, Juels, and Triandopoulos (2013) referred to an emerging and dangerous

trend wherein an attacker destroys the entire analytic trail by removing all the security analytics

sources. Without the sources, it is not possible to perform a forensic analysis of the data. Bowers

et al. presented PillarBox, a tool built to securely relay security analytic source (SAS) messages.

PillarBox enforces integrity and prevents tampering with security analytics source messages.

One of the major drawbacks of Pillarbox is its inability to prevent full nelson attacks. Threat

prediction is yet another area of relevance to the domain of security analytics. Thompson (2013)

introduced techniques for threat prediction using traffic flow theory and traffic congestion

principles in his dissertation. The focus of his work is on text-based analysis of network data.

Text analytic techniques were customized based on traffic flow theoretical constructs and applied

on network data. Major outputs from the framework Thompson proposed are threat predictions

based on network traffic congestion, the location of threat, and nature of a threat. Thompson used

machine learning techniques such as classification and clustering to predict threat and attack

patterns. Experience-based cyber analytics is a new area of research. Chen (2013) devised a set

of algorithms to connect a security analyst’s experience to network intrusion analytic tools.

Interactions with analytic tools exert a heavy workload on security analysts. An experience base

builds dynamically based on the decisions security analysts take, and the experience-base also

acts as the repository for future decision.

The development of security analytics frameworks with specific goals has become

commonplace. Intrusion detection tools use two popular approaches: signature-based detection

and anomaly-based detection. Anomaly-based tools study deviations from normal behavior to

identify intrusions, unlike signature-based systems. Razaq, Tianfield, and Barrie (2016) designed

a cybersecurity analytic framework with a focus on intrusion detection. Some of the highlights of

this framework are (a) it derives inputs from a network log tool (NetL) and a process log (PrcL),

(b) data extraction uses a Hadoop big data process, (c) data normalization and correlation, and

(d) alert generation. Correlation is the key step in this algorithm. False alarms are a known

problem in anomaly-based detection, and this system is no exception. Razaq et al. showed that

security analytic tools and processes are undergoing many changes and improvements, and an

understanding of this study adds credence to this current research. The concept of correlation is

an ideal input for the Delphi study (Phase 1) in this research.

Insider threats are now a daily occurrence in many enterprises. Most of the security

analytic tools include use cases for insider attacks. Mayhew, Atighetchi, Adler, and Greenstadt

(2015) explored big data analytics combined with machine learning to provide behavior-based

access control (BBAC) to deal with insider attacks. They combined clustering, a popular

unsupervised learning technique, with classification (a supervised learning technique) in

developing this unique solution. BBAC uses support vector machines to classify normal versus

suspicious data. BBAC cybersecurity analytics solutions predominantly focused on handling

attacks on the Department of Defense.

One of the major components of any security analytic tool is to generate alerts based on

the use cases and rules that are available as inputs to the tool. Thousands of alerts arise every

day, overwhelming the security analysts. Analysts miss many of the alerts. Pierazzi, Casolari,

Colajanni, and Marchetti (2016) introduced a framework that automatically chooses an anomaly

detection technique for any set of alerts. It generates descriptive statistics for the different groups

of alerts and uses these statistics to decide the type of anomaly detection technique. This

framework is a unique, next-generation solution, which is a first of its kind. Many current

commercial security analytic tools might benefit from the approach of this framework. The

complexity of modern-day networks warrants this type of research, and this research by Pierazzi

et al. generated a few questions for the Delphi study.

Research projects on anomaly detection using big data forms a significant part of security

analytics domain. Researchers normally use anomaly detection when they are not sure about the

data patterns for which they are looking. Pinto (2014) opined that most of the recent-day

anomaly detection techniques employed in the big data domain have not proven their worth. He

further stated that very few anomaly detection solutions are effective, one of them being ISS

Network’s anomaly detection product (now part of IBM’s QRadar). Pinto further discussed three

big issues associated with anomaly detection approach, and they are (a) the curse of

dimensionality, (b) normality poisoning attacks, and (c) Hanlon’s razor. Chen (2009) defined the

curse of dimensionality as the exponential growth of the number of samples needed to estimate a

function in a given machine learning algorithm. The processing of large samples will result in

increased use of computational resources. As per Pinto, a normality poisoning attack refers to an

attack on training data involved in training a security analytic learning algorithm. This kind of

data poisoning can skew the accuracy of the algorithm. Pinto defined Hanlon’s razor as false

positive created due to a pseudo security issue like a developer experimenting with secure coding

in a production environment resulting in wasted investigation time.

More Security Analytic Solutions

Vehicular ad hoc networks (VANETs) are an evolving type of MANETs in very common

use in building smart cities. VANETs communicate between vehicles, vehicles to infrastructures,

and vehicles to the Internet and the cloud. Gantsou (2015) applied security analytics to detect

Sybil attacks. A Sybil attack is an attack in which a single physical computer or entity receives

multiple identities by an act of forgery. Gantsou uses a unique approach of correlation between

MAC address and IP address to solve the problem of Sybil attacks. It uses the OSSIM security

analytics platform, the only open-source platform, in its IDS layer, OSSEC. A unified security

analytic platform like OSSIM has a clear advantage in extending its power to MANETs like

VANET. Even though mobile ad hoc analytics are not part of the first research question, this

study by Gantsou provides an opportunity for future researchers to develop security analytic

platforms that include ad hoc networks, and it reveals the power of security analytics for

nonconventional network systems.

Security Analytics and Correlation

Correlating security events is a key function performed by any security analytic tool. The

process of correlation collects events from different log sources and normalizes them. For

example, normalizing the events based on timestamp on the events. Stroeh, Madeira, and

Goldenstein (2013) proposed machine learning based correlation in security analytic tools. The

solutions fuse normalized events into groups or meta-events. Meta-events present a complete

description of a possible attack scenario than single events. They constitute a more refined

expression of the underlying attack than isolated efforts. Grouping events into meta-events lead

to better situational awareness, improving the classification of real attacks and false alarms; it

also enhances the performance of the system, as the classification operates on meta-events rather

than events. Meta-events enable the contrast between new scenarios and previously learned

attack scenarios. Machine learning techniques can automate this process.

The event correlation process is the key to unraveling many types of cyber-attacks. Rosa,

Alves, Cruz, Simões, and Monteiro (2015) compared different event correlation engines. A

summary of rule-based correlation engines follows. Esper is an open-source event correlator that

specializes in detecting stealthy port scans and employs time-based correlation logic. Simple

event correlator (SEC) is an open-source tool predominantly used for detecting intrusions, and it

uses historical data to flag attacks. The Prelude tool has both open-source and commercial

versions. The main use of Prelude is in the intrusion detection domain. Simple event correlator is

the most common correlator for small sized environments. Correlation engines do not build

security analytic tools by themselves. They need full integration with a SIEM or an equivalent to

applying its power. The Delphi questionnaire included assessments of the correlational abilities

of security tools.

Security Analytics and Governance

Security analytics, in general, refers to three generations of technological maturity,

according to Cardenas et al. (2013). The three generations of security analytic tools are (a)

intrusion detection and prevention (IDPS) tools, (b) security information and event management

(SIEM) tools, and (c) big data driven security analytic tools. However, most security analytics

products focus on the SIEM domain. All the market-leading products use SIEM technology. In

recent years, SIEM products have dominated the security governance scenario in many

organizations. Van de Moosdijk, Wagenaar, and Final (2015) detailed all the major areas of

concern in the SIEM domain. Some of the governance-related highlights that are relevant to this

literature review are explained by them in the following lines. Organizational leaders understand

the need to implement SIEM and the benefits of implementing SIEM products. The requirements

for implementing SIEM are well defined. For example, risk appetite, regulatory or compliance

requirements, and reporting requirements are clear. This research study included questions in the

above areas, as a part of the Delphi study. Leaders properly assign roles and responsibilities

regarding SIEM technology. For example, SIEM architects and SIEM correlation engineers are

predominantly technical roles. Van de Moosdijk et al. further stated that SIEM program

managers are also implementation and reporting managers. This research study included such

experts as part of Phase 1. Managers identify SIEM tool requirements before making a product

choice. This research study distilled critical success factors for the implementation of any

security analytic products, including SIEM. Both business and IT stakeholders discuss use cases

requirements. Questions on security analytic use cases were a part of the Delphi study.

The establishment of metrics-driven governance is a necessary prerequisite for the

success of any security analytic program. Gordas (2014) discussed the importance of governance

through security metrics. Reports of key performance indicators (KPI) of any security analytic

program usually are routed to CISOs. Commonly reported categories of metrics are security

incidents, suspicious events, log sources, rule changes, use cases, and security investigations and

related escalations.

Security Analytics and security operation center (SOC)

Elliott (2016) dealt with SOC economics in detail. The cost of operating a SOC generally

includes the cost of SOC analysts, who are part of the SOC. Security analytic tools, in particular,

SIEM tools, enhance the productivity of the SOC. Schinagl, Schoon, and Paans (2015)

elaborated about the SOC and its interfaces. They dealt with cyber intelligence inputs to SOC in

detail. Monitoring and incident management are two important functions of SOC where security

analytic tools provide feeds. The triaging team informs SOC analysts about the incident tickets

that the security analytic team or tool creates on a regular basis. The effectiveness of security

analytic tools indirectly impacts the SOC efficiency and output.

Critique of Previous Research Methods

SIEM tools form the mainstream of security analytic products. Nicolett and Kavanagh

(2013) analyzed the critical capabilities of commercial SIEM tools as part of the Gartner study.

Three major use cases they identified were threat management, compliance, and general SIEM

deployment. They scored every commercial tool for these use cases by accounting for eight

critical capabilities: (a) real-time monitoring, (b) threat intelligence, (c) behavior profiling, (d)

data and user monitoring, (e) application monitoring, (f) analytics for root cause, (g) log

management and reporting, and (h) deployment and support simplicity.

A deeper evaluation of the Gartner report produced by Nicolett and Kavanagh (2013)

indicated that most security analytic products developed by popular vendors like IBM and HP

scored consistently in the compliance use case, but only a couple of them performed well in

threat and SIEM deployment. A comparison of Nicolett and Kavanagh’s (2013) study with

Shackleford’s (2014) SANS survey revealed several of knowledge gaps in security analytics with

respect to assessment. These gaps were present due to the lack of a validated survey instrument

that was developed with the help of implementation experts. Gartner and SANS surveys are

private surveys that were developed predominantly by research analysts. While commercial

surveys can assess security analytic products, they are likely to have biases towards the sponsors

of the survey. Many white papers released in security analytics have similarly biased views.

Construction of these surveys takes place without a strong practitioner focus and a theoretical

basis. A few experts from within one organization build these surveys, leading to bias in the

questions. Many security analytic tools and projects were discussed in this literature review

section. For most of these tools, self-assessment was made by the product owner. The popular

security analytic products like QRadar are assessed by research analysts from research

organizations like Gartner and SANS. Integrating self-assessments with the opinion of research

analysts revealed a lack of assessment standard for the security analytic tools. The lack of

standards for assessing security analytic tools results in organizations not fully benefiting from

the implementation of these tools. A discussion on the identified knowledge gaps will help in

understanding the research study area for broader and deeper research.

Findings

Table 1

Assessment of Security Analytics Tools

Security analytic tool name Function of tool Assessment Description

Beehive tool Threat, policy and behavior

analysis

High-speed network log

processing with less focus

on performance testing

Network predictive analytics,

Sandia Labs Attack prediction analysis

Transfer learning and

synthetic data approach

with limited testing

Gartner assessment of SIEM

tools

Security analytic tool with

focus on incident and event

management

Compliance focus and pre-

defined use case focus.

Assessment lacked

implementation and end-

user specific issues

Advanced cyber defense center

(ACDC) project Attack prevention

Prevention of Botnet

attacks. Lacked details on

User interface (UI)

SANS assessment of SIEM tools

Security analytic tool with

focus on incident and event

management

An assessment made using

product use cases.

Assessment lacked

implementation and end-

user specific issues

Multi-core security analytic tool Correlation event analysis

Employed popular

correlation algorithms but

lacked real-time testing.

It is evident from the discussion on security analytic tools in the previous section that

these tools are predominantly assessed from the perspective of performance data. Table 1

includes a description and assessment of security analytic tools that brought out the predominant

functions of those tools in the earlier part of the literature review section. Ideally, assessment of

such tools should take into consideration not only performance data but also other influencing

factors like user interface, ease of use, governance, and reporting capabilities. A recollection of

tool capabilities discussed in this literature review will throw light on the findings. The Beehive

tool presented by Yen et al. (2013) was not subject to a thorough performance testing, as it

underwent testing with only two weeks’ worth of real data. Real tests for any tool should involve

real-time customer data. The predictive analysis approach developed by Colbaugh and Glass

(2011), while unique in its approach, used a controlled and limited test environment of ten runs.

A predictive approach gains more accuracy with an increase in the size of datasets. Judging by

these examples, any research study should include scalability in its survey and interviews, and

the respective stakeholders should be happy with the results. For example, a security analyst

should be convinced about the scalability of an analytic correlation engine and a business

manager should be convinced about the implementation of compliance mandates. Based on the

findings in this section, the researcher introduced questions pertaining to performance and

scalability in the Delphi study.

Results are what matters in huge investments. Governmental compliance laws trigger

changes in the SIEM tools. For example, compliance with Sarbanes-Oxley legislation mandates

the archiving of all network log files for a predetermined number of days. Nicolett and Kavanagh

(2013) mentioned that investments in SIEM tools are predominantly due to the compliance

requirements of the business community. While supporting governmental laws are important for

SIEM tools, these tool owners should also consider other security aspects of the enterprise when

building new product features. The main focus of any research on security analytics is whether a

SIEM tool or an analytic tool made a difference in terms of security to the enterprise.

The user interface is yet another important aspect of any modern IT system. A good and

intuitive user interface can inspire an end user to explore an analytics tool. Crespo and Garwood

(2014) included a lot of information about user interfaces in their tool for the ACDC project.

Cheng et al. (2013) depicted a user interface in their architecture diagram. However, they neither

measured nor discussed its effectiveness. Shackleford (2014) did not directly include user

interface assessment in his analytics survey. TAM-3 variables introduced by Venkatesh and Bala

(2008) like the perceived ease of use (PEOU), has a direct application in the area of security tool

user interface. Research utilizing TAM-3 to study the effectiveness of tools should measure user

interface strengths and weaknesses. The Delphi study this researcher performed applied this

finding by incorporating user interface related questions into the survey. In his security analytics

survey, Shackleford (2014) portrayed the fact that 48% of end users were unsatisfied with the

analytics training they received. Without proper training, end users will not be able to utilize the

tool. For example, if a security analyst needs to develop his or her own algorithm to analyze

network traffic as explained in the stream computing platform discussion by Schales et al.

(2011), then he or she must receive proper training from the vendor and other stakeholders.

Shackleford (2016) explained that many organizations have started to collect more data for

analysis. However, a lack of skills at the SOC to deal with security analytic tools was prominent

in his survey.

Actionable intelligence is the ability of the analytic solution to provide inputs to decision

making at different levels of the security organization (Howarth, 2014). Many of the tools in this

paper concerned security intelligence, but the researchers did not share details about the actions

they took based on the intelligence the tool provided. It is possible these are confidential

decisions within an enterprise. However, any research study should include questions on the

effectiveness of actionable intelligence recommendations. For example, end-point device

quarantining will impact security threat metrics because a quarantined device is no longer active

in the network. Findings from a review of the literature indicate that extensive research is

progressing in security analytics. While research itself is a great boost to the domain of security

analytics, it is important that the industry focus on measuring the effectiveness of any given

security analytic product after its implementation, irrespective of the product benchmark claims.

Only an implementing organization and its team will be able to provide solid feedback on the

implementation of a security analytic tool.

The literature review of security analytic tools brought out many functional aspects of

these tools. These functional aspects were used in one or more security analytic tool projects in

determining the tool’s success. A functional architecture for a typical security analytic tool is

presented in Figure 2, by integrating all the tool aspects learned from the literature review

presented earlier in this chapter. Core concepts and flows presented in Figure 2 are derived from

the tools discussed earlier in this chapter. Some pertinent citations that comprehensively cover

the core functional concepts, included in Figure 2, are (a) Shackleford (2014) discussed user

training and use cases in his assessment surveys, (b) Elliott (2016) provided comprehensive

inputs on SOC, its operations and endpoints; (c) Nicolett and Kavanagh (2013) expanded on log

sources, use cases, compliance, attack detection and user interface; (d) Rosa, Alves, Cruz,

Simões, and Monteiro (2015) analyzed correlation engines and data analytics; (e) Gordas (2014)

introduced concepts on governance and reporting. Figure 2 includes all the underlying tool

concepts with the central circle explaining all the functions that were discussed in the above

literature review section. Arrows indicate the direction of information or data flow. For example,

a security analytic tool provides inputs to the governance and reporting function of an enterprise.

Figure 2. Security analytics: A functional architecture.

Summary

Concepts From Security Analytics Literature for Delphi Study

The detailed analysis of the literature in the previous sections yielded several concepts

that were used as the basis for developing survey questions in both the field study and Delphi

study. Yen et al. (2013) discussed the Beehive tool with a focus on threat intelligence and user

behavioral anomaly, thereby deserving a place in the initial Delphi questionnaire. Schales et al.’s

(2011) discussion on stream computing yielded the concept of network logging. All the network

log sources in a corporate network should go into the security analytic tool to help in the

correlation of logs and the extraction of threat intelligence. Chari et al. (2013) studied policy

violations using their tool. Detecting policy violation can lead to identifying insider attacks like

user privilege escalations. It is important that any security analytic tool has a native feature to

detect policy violations. Crespo and Garwood (2014) built a solution to address botnet attacks.

One of the important analytic concepts to emerge from this study is the correlation concept. A

correlation engine in any security analytics tool correlates logs from heterogeneous sources to

identify attacks. Questions relating to correlation were very applicable to the Delphi

questionnaire. Yet another concept Crespo and Garwood (2014) raised is the interface for the

user or administrator. Screen interface or user interface is the main gateway for any tool user to

understand the intricacies and capabilities of a tool. While command line interface is still

prevalent among administrators, sophisticated user interfaces and dashboards are part of many

SIEM tools. If a powerful security analytic tool has a poor user interface, users will not tap the

tool’s full potential because of their lack of motivation to navigate the user interface. Venkatesh

and Bala (2008) built the TAM-3 after the advent of e-commerce systems and introduced

variables of perception in their theory. Hence the researcher, introduced questions on user

interface into the Delphi study to gain the perceptions of administrators of the security analytic

tools on the friendliness of user interfaces. Malekpour et al. (2014) elaborated on big data based

learning and boosting algorithms that help in detecting fraudulent behavior in banking

transactions. The concept of learning based on either training data or anomaly identification is

popular in security analytics based on the literature reviews. Hence the researcher examined the

effectiveness of big data based learning in the field study. Bou-Harb et al. (2016) extracted

actionable forensic evidence with the help of big data based graphical analysis. Actionable

intelligence is a key term, and it is a great time saver for security analysts who work in the SOC.

Actionable intelligence inputs given by any security analytic tool to security analysts will

drastically reduce the lead time to mitigate an attack. Hence, the concept of actionable

intelligence was very relevant to the Delphi questionnaire.

Strategies to Fill Gaps in Literature

The goal of this research study is to directly address the gaps and findings discussed in

the previous two sections, by identifying factors that determine the success of implementing any

security analytic tool. To accomplish this goal, this researcher developed a security analytic

questionnaire through two stages: the first stage is the Delphi study and the second stage is the

exploratory factor analysis (EFA) study. The questionnaire focused on implementation-related

issues. The researcher performed statistical analysis on the final set of questions to extract factors

that influence a successful implementation. This set of factors is expected to help both the

technical and managerial professionals involved in the implementation of security analytic tools

and solutions.

Measuring security programs in an enterprise is a big challenge. Cybenko and Landwehr

(2012) compared security analytics and measurements to game theory concepts. In game theory,

researchers already know the rules and players, and hence measurement is easier in a game

situation. However, in security analytics, it is difficult to measure the effectiveness of a solution,

as the playing field is comparable to a moving target. Cardenas et al. (2013) discussed many

types of attacks, as well as the impact of security analytic in mitigating those attacks, but none of

the analytic tools, discussed in the literature review, convincingly assessed the tool against a

broad and common assessment framework.

An ideal approach to assess a security analytic tool is to measure security posture of an

organization using quantitative metrics, before the implementation of the tool and after the

implementation. Comparing the results obtained in the before and after scenario will reveal the

impact of the tool. Mateski et al. (2012) defined cyber threat metrics and also elaborated on the

challenge involved in measuring the security posture using metrics. Most organizations do not

even disclose data breaches due to the fear of customer backlash making measurements a

difficult task. Van de Moosdijk et al. (2015) discussed the ability of SIEM tools in producing

many security data governance reports including data breach related reports. Even though SIEM

tools can produce such reports, they are usually not available for researchers. Due to the above-

mentioned reasons, it is impossible to accurately measure the impact of these tools only by using

metrics data which are usually not made publicly available. However, it is possible to

understand the effectiveness of a security analytic tool indirectly by designing survey questions

to assess the different functional areas of the tool from a user’s perspective. A detailed literature

search and review did not yield expert-validated survey instrument in the field of security

analytics that was suitable to answer the research questions of this study. It is to be recollected

here that this research study broadly focused on all three generations of security analytic tools.

CHAPTER 3. METHODOLOGY

The security analytics domain has experienced rapid growth in the past ten years in terms

of maturity. Cardenas et al. (2013) discussed the three levels of maturity for the security analytic

domain, but the fourth level of maturity is now opening up in the information security industry,

that is, artificial intelligence (AI)-driven security analytics. Internet Protocol (IP) v6 has many

known vulnerabilities. Fuzzy rules are being used in machine learning to classify network

attacks. Salih, Ma, and Peytchev (2017) built a blended algorithm comprising fuzzy rules and

genetic algorithms to counter IPv6 vulnerabilities. Given the explosive growth of analytics in this

industry, it is important to produce expert-validated survey instruments in the security analytic

domain to help further research. Based on the discussion in earlier chapters, the researcher

adopted two phases for this research study. Phase 1 involved the development of a survey

instrument using a Delphi expert panel, and Phase 2 involved the refinement and validation of

the same instrument using construct validity and reliability techniques. The researcher tested for

construct validity using exploratory factor analysis (EFA) and for instrument reliability using

Cronbach’s alpha.

Purpose of the Study

The purpose of this research study was to identify factors that determine the

implementation success of security analytic tools that help in improving the security postures of

organizations. Security analytics is a relatively unexplored area from the perspective of academic

research. Cardenas et al. (2013) argued that security analytics is an evolving technology topic.

However, research studies and investigations in this area of security analytics are still in a

nascent and developing condition. A search by this researcher did not yield any expert-validated

survey instruments in the domain of security analytics. A casual Google search will provide

many results for assessments of commercial security information and events management

(SIEM) tools performed by Gartner group and other research groups. For example, Nicolett and

Kavanagh (2013) assessed SIEM tools using product use-cases but not using surveys built with

the help of hands-on implementers. Pinsonneault and Kraemer (1993) established that

researchers invariably use exploratory models to develop concepts and models in newly arrived

fields of research. Pinsonneault and Kraemer further asserted that surveys are an effective way to

collect data and quantitatively to analyze results. The literature review revealed that there are no

dedicated survey instruments validated by expert users in the domain of security analytics. A

survey instrument is a good input to identify important implementation factors for security

analytic tools as per Yong, and Pearce (2013). Hsu and Sandford (2007) opined that the Delphi

technique is a practical and suitable technique for working with experts and arriving at a

consensus on any emerging area of research. Newsom (2005) argued that exploratory factor

analysis (EFA) is an effective approach to validate a survey instrument when the researcher does

not have any hypothesis and the subject matter is a relatively unexplored area. The purpose of

this quantitative exploratory study was to identify the factors that determine a successful

implementation of a security analytic tool. Shackleford (2014) conducted a survey on security

analytic tool users from a commercial angle. His survey focused on popular vendor built security

analytic tools like QRadar and ArcSight. The survey questions were built by a few research

analysts with the help of product knowledge. In Shackleford’s survey, the user experts did not

pool their experience and build a survey questionnaire. Even though Shackleford’s survey was

not an expert-validated survey, it gave initial insights about security analytic tool usage, among

customers. However, in this research study executed by this researcher, security analytic tool

users and experts pooled their experience and validated the survey instrument, both in Phase 1

and Phase 2.

Research Questions

The research questions for this study were as follows:

Omnibus Research Question (ORQ): What are the factors that determine the successful

implementation of security analytics tools or packages?

Research Subquestion 1 (RSQ1): What are the factors that determine the successful

implementation of non-big data security analytics tools or packages?

Research Subquestion 2 (RSQ2): What are the factors that determine the successful

implementation of big data security analytics tools or packages?

Research Design

This study used a two-phase approach: Phase 1 was a quantitative Delphi study to arrive

at a survey instrument by working with experts using a consensus approach; Phase 2 was an EFA

study to refine the instrument by sending it to a large group of experts in the information

assurance industry (Figure 3). Skulmoski et al. (2007) argued that Delphi technique is the

predominant approach when there is a lack of complete knowledge about a particular research

area. This study fell under the category of non-experimental category because experiments

involve the creation of control and test groups, and this study does not involve any experiment

on security analytics tools. Spector and Meier (2014) established that non-experimental methods

involve observation without manipulation. Studying the factors involved in assessing a

successful implementation of security analytic tools is the real goal of this research. This

research involves the application of Delphi and exploratory data analysis to explore the research

questions given above. Garson (2014) opined that Delphi studies are applied mainly in the area

of exploratory research. Delphi models are more suitable for this study, as they are useful in

areas where there is incomplete information or incomplete maturity of the domain.

Figure 3. High-level research design.

Garson (2014) attested to the effectiveness of the quantitative Delphi approach in

reaching consensus on the composition of survey questionnaires. Garson also mentioned that

quantitative Delphi processes are suitable to achieve content and contrast validity. In this specific

Delphi study, building a consensus and distilling a survey instrument that will help in assessing

the success of security analytic tool implementation was the major focus. Gliddon (2006)

performed a classic Delphi study and created a list of competencies for innovative leadership

after three rounds of Delphi. Vonhof (2015) employed a similar design to build a survey

instrument and then used it to perform an exploratory factor analysis (EFA) study of telework

adoption.

Procedures

Methodology

This research employed a nonexperimental approach. Nonexperimental methods involve

observation without manipulation (Spector & Meier, 2014). There are no experiments or variable

manipulations involved in this research study. Phase 1 of this research study focused on building

an instrument with the help of experts. Phase 2 focused on validating the instrument and

extracting factors using EFA. Extracting factors from the survey instrument was the main goal of

Phase 2 of this study. In Phase 2, the output of Phase 1, that is, a survey instrument, was

administered to the industry experts with the aim of refining the items in the instrument.

The Delphi technique and its suitability to the current research question are of prime

importance in understanding the methodology. Hsu and Sandford (2007) expounded the Delphi

technique, stating that it is a very practical approach to achieve convergence of ideas and

opinions of experts in any domain of activity. A research question is a prime trigger to decide the

direction of an investigation, namely quantitative or qualitative. In the case of the current study,

the research questions involved the identification of factors that help in assessing the successful

implementation of a security analytic tool or package. This study did not focus on the impact of

security analytic tools on the job role of a security analyst, which would warrant a qualitative

study. Garson (2014) surveyed many Delphi studies and established that consulting domain

experts is an ideal approach for a less explored area of research. Hence the direction of the

research question clearly identified the need for a panel of experts, as only experts could furnish

elite information in security analytics.

Quantitative Delphi method. Gunaydin (2006) asserted that the Delphi technique is an

iterative process very suitable for integrating into the quantitative process if the research focuses

on construct validation. According to Garson (2014), the Delphi method has multiple rounds

until a consensus emerges between experts or the results from two successive rounds are the

same. Leclerc, Lefrancois, Dubé, Hébert, and Gaulin (1998) applied the quantitative Delphi

method to validate constructs. He built a self-actualization instrument using the Delphi method.

Vonhof (2015) used the quantitative Delphi approach to build and validate an instrument as a

part of his Capella dissertation on telework adoption. Chaturvedi, Singh, Gupta, and

Bhattacharya (2014) successfully implemented an information security Delphi survey with a

five-point Likert-type scale. This research study fell into the quantitative category of Delphi

study based on the close similarities to the above citations.

Delphi design explanation. Skulmoski et al. (2007) mandated four major features of any

Delphi study: (a) anonymity of Delphi participants, (b) an iterative approach that allows the

participants to refine their views on any subject matter, (c) controlled feedback from the Delphi

coordinator (researcher) to the participants regarding other perspectives in a Delphi study, and

(d) statistical analysis and aggregation of group responses. There is no concept of a population in

Delphi study. As it tested no hypothesis, a power analysis was not a requirement for this study.

The number of Delphi participants and the qualifications of the participants are of great

significance in a Delphi study. Taylor-Powell (2002) suggested that the qualifications and

experience of the Delphi panelists should be up to date and that participants should be unbiased.

Avella (2016) provided a range for the panel size for any given Delphi study. He suggested that

it ranges from a minimum of 10 to a maximum of 100. This researcher applied the anonymity

principle by ensuring that all Delphi participants sign the informed consent form. The consent

form contained all the qualification criteria required from the Delphi participant. Delphi

participants used a flow diagram developed by this researcher to understand all the steps

involved in the Delphi study. Based on the panel size recommended by Avella (2016) an initial

size of 20 was planned for the first round of Delphi. This researcher searched the LinkedIn

professional network for up to date profiles in the area of security analytics and found 43

members to be suitable for this research study. Invitations were sent to 43 members in the

LinkedIn network.

Consensus in Delphi studies. Arriving at a consensus is a collaborative effort usually

facilitated by the researcher. Garson (2014) advocated the iterative Delphi method to achieve

reliable expert consensus. In this specific research study, after the Delphi panel members

provided their inputs for every round, the researcher used a couple of consensus techniques to

assess consensus. The researcher also assessed the stability of the instrument after every round.

Heiko (2012) focused his research on consensus in Delphi studies. He examined many Delphi

studies in his research of consensus methods, and he concluded that while consensus was not the

sole aim of Delphi studies, it is a very important component in Delphi studies. Apart from

consensus, Heiko opined that stability of the instrument is also a necessary part of building

consensus. He added that consistency in responses between successive rounds of a Delphi study

is stability. In other words, if the instrument has remained consistent between two rounds, then

that instrument is stable. For example, there might be insignificant changes (in terms of

responses) between two rounds of Delphi. Descriptive statistics and inferential statistics are

essential in computing consensus in Delphi studies. Putnam, Spiegel, and Bruininks (1995)

recommended the use of the percent agreement method for Delphi surveys using a five-point

Likert-type scale, and this method was very suitable for the current research study on security

analytics. Cooper, Gallegos, and Granof (1995) recommended Kendall’s coefficient for

computing consensus in quantitative Delphi studies. Percent agreement method was used in

Phase 1 (Delphi study) to compute consensus after every Delphi round. Kendall’s coefficient was

also used at the end of each Delphi round in Phase 1. Stability, as defined by Heiko, was applied

in the Delphi study to understand the consistency of responses between any two rounds of

Delphi.

Participant Selection

One of the major applications of Delphi study is to validate constructs and build

instruments, but there is no hypothesis to be tested. Avella (2016) asserted that there is no

concept of a population in Delphi studies, but only expertise. Security analytic experts are

usually busy professionals, and they are usually not able to allocate time for research studies.

Since LinkedIn is now a highly sought-after professional networking tool and the profile

information of individuals is nearly accurate, this researcher chose LinkedIn to recruit security

analytic experts. Lops, De Gemmis, Semeraro, Narducci, and Musto (2011) analyzed LinkedIn

user profiles as a part of their content extraction. Their study revealed that LinkedIn is a very

useful professional networking tool and that users regularly update their profiles. Gliddon (2006)

surveyed 50 Delphi studies and found that in 95% of the cases, the researchers used purposive

sampling to select the experts. Gliddon recommended that researchers validate the experts by

going through their curriculum vitae. This Delphi study was a homogeneous study, that is, the

researcher selected experts only from within the information security domain, and all these

experts were in the United States. However, security, managerial, and operational personnel

share the knowledge necessary to assess the implementation of a security analytic tool.

Skulmoski et al. (2007) analyzed 41 Delphi dissertations in the information sciences and found

that the sample size ranged from eight to 345. The median value was 28. Gliddon (2006)

surveyed 46 Delphi studies and found that the average sample size was 24. In this specific study

on security analytics, the researcher identified the following categories of experts: (a) security

program managers, (b) IT managers working in the security group, (c) security analysts, (d)

security service managers, (e) security incident personnel, (f) product architects, (g) business

area managers, and (h) security analytic tool administrators. Hence the researcher chose a sample

size of 24 to start the study. According to Garson (2014), exploratory Delphi studies generally

have smaller panel sizes. Figure 4 explains the overall flow of the Delphi study. Figure 4 is

developed based on the inputs derived from Delphi method citations provided in the above

section.

The researcher used a purposive sampling approach by searching LinkedIn profiles and

creating connections with potential participants. Gliddon (2006) recommended purposive

sampling based on the survey he conducted on Delphi studies. As per Gliddon, 95% of Delphi

studies used purposive sampling. The search in LinkedIn for experts was based on the

qualification criteria that was outlined in the informed consent form. The qualification criteria

are explained later in this section. The search tool provided by the LinkedIn professional network

was used to search for suitable professionals. The search strings were (a) security analytics, (b)

big data security, and (c) security analyst. Invitations to participate in the Delphi study on

security analytics went to 43 connected LinkedIn users who were high-profile security analytic

experts. Gliddon further stated that experts should be recruited only after validation of their

profile or curriculum vitae. The researcher wrote a letter of invitation to all the shortlisted 43

members whose qualifications and experience fell into the criteria below. A consent letter was

also attached to the invitation.

Figure 4. Delphi study process flow.

The professional experience of these invitees ranged from network security experts to

CISOs. Of the 43 experts who received invitations, 18 experts agreed to sign the informed

consent form and to participate in the Delphi study. There were two possible qualification

criteria for the Phase 1 Delphi study. Option 1 consisted of all the following criteria: (a)

minimum seven years in the information security industry, (b) minimum 10 years of total

experience in the IT industry, (c) minimum three years of experience with security analytic tools,

in any capacity, within any enterprise, and (d) preferably a minimum professional certification or

degree in the area of IT security or IT. Option 2 consisted of only one criterion: 20+ years of

strong IT experience in managing large programs or projects or architecting major IT solutions.

Phase 2 of the study was the exploratory factor analysis (EFA) study. There were two

participant qualifying criteria for the Phase 2 EFA study: (a) minimum 10 years of experience in

the IT industry OR minimum six years of experience in the information security industry, and (b)

minimum one year of experience in the security analytics domain (any experience that relates to

security analytic tools like SIEM/big data/intrusion-prevention or equivalent).

Protection of Survey Participants

The two phases of this research study had different types of participant requirements.

Phase 1 was a Delphi study, and Phase 2 required a bigger audience to fulfill the requirements of

an EFA study. Tabachnick and Fidell (2013) argued that a minimum sample size of 100

participants are needed for an exploratory factor analysis (EFA) study. Phase 1 of the study

involved recruiting and interacting with the Delphi participants. Since recruiting took place

through LinkedIn, some participant data like e-mail addresses were available to the researcher.

However, the researcher collected no sensitive personal information about the participants.

Adams and Miles (2013) explained that Belmont principles for human subjects are the major

focus of any research. Any researcher needs to scrupulously apply justice, respect, and

beneficence principles. The respect principle leads to informed consent. The researcher informed

the panelists and survey participants about the pros and cons of this research using an informed

consent form. There were two versions of informed consent form: one for the Delphi study

(Phase 1) and other for the EFA study (Phase 2). Only after all the participants signed the

informed consent forms could they participate in the study.

The justice principle requires researchers to treat the participants in a fair manner. Adam

and Miles (2013) discussed the importance of justice principle for surveys. In this survey (both

Phase 1 and Phase 2), participants from different industries and with different levels of

experience received the same respect. Beneficence requires avoiding injury or any other form of

harm to human subjects. There was no coercion of participants in either phase. The researcher

implemented the Belmont principles through informed consent. Schrittwieser, Mulazzani, and

Weippl (2013) articulated a change in research direction for information security-related

research. Based on their opinion, the researcher did not force the survey participants to provide

definitive answers. The researcher did not ask the survey participants to provide confidential

information, due to the sensitive nature of the subject. In their study of the Menlo Report,

Dittrich and Kenneally (2012) explained the fourth principle of research: respect for law and the

public interest. The Menlo Report sets a new direction for information security research, as the

Department of Homeland Security published it. Survey questions respected the internal mandates

of the participating organizations. In both phases, there was a neutral option available to the

participants so that they had the freedom to avoid answering sensitive survey questions. The

Delphi panel also reviewed the survey questions through many iterations to remove any sensitive

questions before the start of the EFA phase.

Data Collection

Phase 1. Phase 1 of this study was a Delphi process. Goodman (1987) asserted that the

Delphi method is a highly suitable way to achieve consensus in an area of research where

maturity is low or there is no survey instrument. Prior to initiating the Delphi survey to build a

survey instrument that focused on security analytics, the researcher conducted a small field study

with the help of security analytics experts from IBM. Avella (2016) stressed the importance of

field study as a pre-requisite to Delphi rounds. A total of five experts helped this researcher to

build an initial starting point by providing feedback on an initial survey questionnaire that the

researcher built. The output of this field study (questionnaire) was the input for the first round of

the Delphi survey. Four rounds of Delphi iterations took place before the survey instrument

stabilized for the next phase (Phase 2). In the first round of Delphi, 16 security analytic experts

participated and provided feedback on the survey instrument. In the final round (fourth) of

Delphi, there were 11 participants. All the Delphi rounds took place via e-mail and every

participant anonymously participated. It took a total of 20 weeks to complete all four rounds of

the Delphi study. The researcher facilitated all the rounds of Delphi. Avella discussed the

importance of the researcher playing the role of a facilitator in Delphi rounds. Before the start of

the first round, the researcher informed the Delphi participants that there would be a maximum

of three rounds of Delphi to build the survey instrument. This constraint of three rounds was due

to the time limitation and the cost of the survey. However, due to the lack of consensus and

stability in the third round, one more round of Delphi was necessary, leading to a stable

instrument in the fourth and final round. The researcher notified potential Delphi participants

before the start of the Delphi rounds that there was no penalty for electing not to participate and

that they were free to drop out at any stage of the Delphi process. After the researcher collected

all the responses for any given Delphi round, the researcher consolidated the inputs and sent out

a document to the entire panel about the current Delphi round. This exercise helped the panel

members to learn about the collective perspective for every round.

Phase 2. The researcher administered Phase 2 of the study (EFA) through Qualtrics, a

survey and research organization. The researcher chose Qualtrics after considering top three

popular survey audience services. The choice of Qualtrics was made mainly because of

Qualtrics’s large database of information security panelists as revealed by the number of recent

dissertations in the field of information security. Burton and Mazerolle (2011) clarified that

construct validity is the assessment of whether the instrument is measuring what it should

measure. Normally EFA is one of the ways of determining construct validity. The factors that

emerge from the analysis process reflect the construct validity of the survey instrument

(Boudreau, Gefen, & Straub, 2001). EFA requires a minimum sample size.

Sample size for Phase 2. Mensch and Wilkie (2011) argued that 134 responses from a

survey that went out to 2,000 students were enough to perform an EFA. Janssens, de Pelsmacker,

and van Kenhove (2008) indicated that a minimum sample size of 100 is necessary to perform a

successful EFA. Štemberger, Manfreda, and Kovačič (2011), in their research on IT

management, found that a survey response of 152 was enough to extract factors using EFA.

Garrido-Moreno and Padilla-Meléndez (2011), in their discussion on the impact of knowledge

management on CRM success, managed to conduct an EFA using 153 completed responses.

Tabachnick and Fidell (2013) asserted that an EFA should have a minimum sample size of 100

and a maximum of 500, depending on the factor loadings. Lu and Ramamurthy (2011), in their

research on IT capability and organizational agility, successfully performed an EFA with 128

responses. Costello and Osborne (2005) provided best practices for EFA. Based on the above

citations and taking into consideration the cost of the survey and the quality of data, the

researcher chose a sample size of 200 completed responses for the Phase 2 survey. Subject to

item ratio is a common measure for determining sample size for any survey involving EFA.

Subject to item ratio means the number of participants divided by the number of survey

questions. Twenty-six percent (26%) of the EFA studies fall in the range of 2:1 to 5:1 ratio, that

is, two to five respondents per question. Since there are 50 items (questions) in this research

study, the chosen subject to item ratio for this study is 4:1 (200:50), which complies with the best

practices Costello and Osborne described. These 50 items were the output of four rounds of

Delphi study. But these 50 items included the hidden factors that form the foundation of any

security analytic implementation.

Data Analysis

Phase 1 – Delphi study. Kendall’s coefficient (W), the percent agreement method, and

stability are the primary techniques the researcher chose to assess consensus among participants.

Worrell, di Gangi, and Bush (2013) recommended Kendall’s W as an established technique to

determine consensus among participants. Vonhof (2015) stated that computation of Kendall’s W

is a common practice in arriving at a consensus. A value of W = 0 indicates complete

disagreement and a value of W = 1 indicates full agreement. Duffield (1993) explained that there

are many ways to arrive at a consensus among Delphi participants. Putnam et al. (1995) applied

the percent agreement method in their Delphi study to establish consensus. They used the

stability of responses between successful rounds as the sole criterion for defining consensus. In

this research, which used a quantitative Delphi technique, the researcher used Kendall’s W and

percent agreement method to assess consensus.

Phase 2 – EFA study. Yong and Pearce (2013) stated that factor analysis is very useful

in extracting underlying concepts from a large set of variables. Typically questions in a

questionnaire have hidden concepts that need extraction for interpretation and grouping. There

are two types of factor analysis: confirmatory factor analysis and EFA. Confirmatory factor

analysis generally confirms or disconfirms hypotheses, whereas EFA uncovers complex patterns

by exploring the dataset. In this research study, the output of the Phase 1 Delphi study was a 50-

item instrument. The researcher used this instrument to conduct an actual survey of industry

professionals in the security analytic domain. The researcher analyzed the resulting dataset for

hidden patterns or factors. Sadi and Noordin (2011) conducted an EFA study in mobile

commerce. They used factor analysis to refine their instrument. In this research, there was a

similar need to refine the instrument from Phase 1 (Delphi). Hence EFA was a logical choice for

this research study. Field (2013) asserted that factor analysis aims to reduce a set of variables

into smaller units called factors or dimensions. In this specific research study, factor analysis

took place on the final output of the Delphi study (Phase 1) with the main purpose of identifying

underlying factors that impact the successful implementation of any given security analytic tool.

In this process, elimination of items (questions) is the first step, and extraction of factors is the

second step. Many experts in EFA concurred that a multistep analysis process is necessary,

leading to the extraction of factors (Field, 2013; Williams, Onsman, & Brown, 2010).

Multistep analysis process. In any EFA exercise, it is necessary first to validate the

adequacy of the sample. Williams et al. (2010) asserted that researchers should perform tests to

assess the suitability of survey data for factor analysis. The researcher used the Kaiser-Meyer-

Olkin (KMO) measure of sampling adequacy in this research study to assess whether the sample

size was adequate to generate factors. Field (2013) opined that researchers must perform

correlations between variables/items to eliminate variables that do not correlate well. He

recommended ignoring variables with correlation values less than 0.3. Generation of a

correlation matrix is a necessary prerequisite for EFA. The researcher performed factor

extraction in this research study with the help of the principal axis factoring method. Williams et

al. (2010) recommended that principal axis factoring is most suitable to extract factors in factor

analysis studies. The researcher also used a scree plot to confirm the number of factors and to

decide on the number of factors for retention. A scree plot is a plot of eigenvalues on the y-axis

with factors on the x-axis. The point of inflection in the scree plot helps to decide the number of

factors. Kaiser (1960) recommended retaining factors with eigenvalues greater than 1 for further

analysis. Field (2013) suggested that using rotation techniques can improve the interpretability of

factors. Varimax rotation assumes that factors have no correlations and it connects variables to

factor loadings. Da Veiga, Martins, and Eloff (2007) established that researchers could include

variables with factor loadings of 0.3 and greater in any given factor. In this research study, the

researcher used a factor loading value of 0.3 and greater to include a variable in a factor.

Validity and Reliability

Malhotra (1981) explained that a field study usually establishes the face and content

validity of an instrument. In this research, the researcher completed a field study to achieve face

validity before conducting the Delphi study. Cronbach (1951) established that reliability is the

ability of the survey instrument to measure what it intends to measure. Combining the factor

analysis goals with that of reliability, the researcher used Cronbach’s alpha on the entire survey

instrument and on the individual factors.

Likert Scale Instrument

Questions in the newly created instrument were based on five-point Likert scale. A field

test conducted by this researcher produced the initial version of the survey instrument. Face and

content validity was established with the help of field test. Dawis (1987) recommended that

formatting is kept simple when designing a new instrument. Burton and Mazerolle (2011)

mandated that field test panel validate the instrument for appearance and appropriateness. Both

the above recommendations were applied during the field test in this research study. Field test

experts also reviewed the statements in the survey instrument to ensure that these constructs were

appropriate for the intended audience. The survey instrument underwent more enhancements

during the Delphi study (Phase 1) in terms of subject matter content. Costello and Osborne

(2005) opined that construct validity is usually established using exploratory factor analysis

(EFA). EFA technique established construct validity of the survey instrument during Phase 2 in

this study.

Ethical Considerations

The researcher fully followed the Belmont principle mandates, ensuring that survey

participants received respect, justice, and equity during the execution of the dissertation phase

(National Institutes of Health, Office of Human Subjects Research, 1979). This study did not

collect sensitive personal information or retain any of the contact e-mails of the survey

participants. All the responses from participants remained anonymous for both Phase 1 and

Phase 2 of the study. Even the initial field study that preceded the Delphi study took place in an

anonymous manner. There was no direct or indirect conflict of interest. Delphi participants had

the freedom not to participate in any round of the survey. In fact, over the course of the Delphi

study, a few participants dropped out of the survey. The researcher securely stored all the data he

obtained from the participants during the survey. All communications with the Delphi

participants were completely anonymous. Vonhof (2015) explained the need for anonymity in

his Delphi research study. This anonymity was necessary to avoid bandwagon types of

responses. For Phase 2 of the study, Qualtrics administered the survey under secure conditions,

so that only the qualified and approved participants fulfilling the survey requirements could take

the survey. All the data provided by Qualtrics are in secure storage and the researcher will safely

anchor the storage USB drive for the next seven years.

Summary

This chapter has discussed the multiple phases of this research study as applicable to the

research question. It has explained all the methods and process involved with both Delphi (Phase

1) and EFA (Phase 2) studies. Qualification criteria, participant selection, and research ethics

were some of the major highlights of this chapter. It has explained the data collection methods

for both Phase 1 and Phase 2 of the study. Similarly, it has delineated data analysis steps for both

the phases. Chapter 4 discusses the results of the data analysis. Chapter 5 provides the

interpretations of the research study.

CHAPTER 4. RESULTS

Background

The purpose of this multiphase, quantitative, nonexperimental, exploratory study was to

identify and extract factors that impact the successful implementation of any security analytic

tool. Vonhof (2015) extracted factors using exploratory factor analysis (EFA) in the area of

telework adoption. Factors in this context fall into two types. The first type is the intrinsic factors

that are part of the tool architecture (e.g., tool functionality). The second type is closely

connected with the landscape of the tool (e.g., security governance). This study also had another

significant purpose, that is, to build a comprehensive survey instrument to assess the

implementation of security analytic tool implementation for any corporation. Ferketich, Phillips,

and Verran (1993) suggested that development of a new survey instrument is the best approach

in the absence of existing instrument to probe research questions. While there were commercial

surveys available in security analytics, there were no expert-validated instruments. This survey

instrument should easily help the future research community to assess any security analytic

implementation.

Research Questions

The main research question (ORQ) was “What are the factors that determine the

successful implementation of security analytics tools?” There are two variations to this question.

The first variation (RSQ1) was to determine the factors that impact the successful

implementation of non-big data security analytics tools or packages. The second variation

(RSQ2) was to determine the factors that impact the successful implementation of big data

security analytics tools or packages. As Chapters 1 and 3 explained, there were two phases of the

study. Phase 1 of the study focused on developing the survey instrument and Phase 2 focused on

the exploratory factor analysis (EFA). In Phase 2, the researcher established construct validity

for the instrument. Yong and Pearce (2013) endorsed the application of EFA to validate

constructs hidden in a survey instrument. They even recommended confirmatory factor analysis

after the EFA to validate the factors after identification.

Description of the Sample

Phase 1 – Development of the Survey Instrument

Field study. A Delphi study that included anonymous participants leading towards

consensus was the approach for Phase 1. However, the Delphi study cannot begin without any

basis. Malhotra (1981) stressed the need for a field study before initiating the Delphi study. The

researcher, with the help of literature review and interaction with five experts in the field of

security analytics, built an initial questionnaire consisting of 20 questions focused on eliciting

opinion about the implementation of any security analytic tool or package. Of the five experts,

three were hands-on architects in the field of security analytics, and two were highly qualified

academicians specializing in IT. Curtis, Krasner, and Iscoe (1988) insisted that field study is a

viable way of soliciting initial opinion from experts. The researcher adopted some of these best

practices in this field study of security analytics. Even though it was not necessary, the

researcher maintained anonymity between the participants. Each field study participant

independently reviewed the initial set of questions, added his or her opinions on changes to

existing questions, and some added new questions to the questionnaire. The researcher avoided

bias among participants by executing the field study in parallel. One of the five experts was a

technology researcher, and he preferred to provide his feedback through a discussion process. At

the end of the field study, 32 questions in security analytics emerged from these five experts.

These 32 questions formed the input to the Delphi study (Round 1).

Delphi Panel. The researcher recruited the Delphi expert panel with the help of

purposive or criterion sampling. Hasson, Keeney, and McKenna (2000) delineated, in their

highly cited study, the need for a Delphi study to have a sample of experts. Unlike a

conventional quantitative study, where researchers commonly choose their samples using

random selection techniques, in a Delphi study, researchers choose experts based on predefined

criteria. Hence, the name for this type of recruiting is criterion sampling. The researcher

recruited experts based on the criteria explained in Chapter 3. After identifying 47 experts

through LinkedIn profile searches, the researcher sent invitations to 43 experts in security

analytics. It later emerged that four experts had conflicts of interest, and the researcher dropped

them from the panel shortlist. Two reminders were sent to the experts after the initial invitation

to send in their filled-in consent forms.

After waiting for a period of one month, the researcher received consent from 20 experts

to participate in the Delphi study. All these participants sent filled-in consent forms to the

researcher’s e-mail address. In this process, all the Delphi participants communicated from their

personal e-mail addresses to the researcher’s student e-mail address. The researcher never asked

any of the panel members to share their sensitive personal information. The researcher

communicated with the panel members throughout the Delphi study only to the e-mail addresses

that they voluntarily shared. The researcher did not coerce any of the participants into the study

at any stage of the Delphi process, thereby truly implementing the Belmont principles as

explained by Adams and Miles (2013). The initial Delphi panel of 16 members, who completed

the first round, belonged to the following diverse categories of security professionals: (a)

information security executive leaders, (b) engineering fellows, (c) security architects, (d)

security program managers, (e) security analytic product managers, and (f) security consultants.

The average experience of the 16 panel members was 20 years in the information security

industry (Table 2). Two notable participants in the panel were a chief information officer and a

chief information security officer (CISO). Every question in the questionnaire had five choices

using a Likert-type scale. Those choices were (a) highly relevant, (b) relevant, (c) neutral, (d)

less relevant, (e) irrelevant. Experts rated the questions based on how relevant those questions

were to the survey. As already discussed in earlier chapters, these Delphi experts contributed

their expertise in building a questionnaire which went on to Phase 2 for the EFA study. The

Phase 2 study went to a larger audience to collect enough data to perform factor analysis. The

Delphi study had four rounds before a solid instrument emerged.

Table 2

Delphi Panel Member Experience

# IT / Info security experience category Years of experience

1 Principal IT architect / Fellow 20

2 CISO 20

3 Security consultant 10

4 Senior DB security developer 15

5 IT consulting manager 30

6 Security operations consultant 12

7 CIO 30

8 Security services manager 12

9 Security program manager 22

10 Senior IT program manager 13

11 Senior IT and security architect 30

12 Senior IT program manager 23

13 Senior ERP and security consultant 28

14 Security product manager 18

15 Database product manager 22

16 Security managing consultant 20

All the Delphi participants received a complete orientation to the study and an

explanation of their roles using a waterfall model diagram (Figure 5). They had options to ask

questions and clarifications. Some of the participants obtained the necessary clarifications from

this researcher after going through the model described in Figure 5.

Phase 2 – Sample Description

Chapter 3 has explained the sample size choice of 200 participants for the EFA study

(Phase 2), and the rationale behind the sample size. The researcher used Qualtrics Research and

Audience Services as the survey tool to collect the sample from a broad audience of information

security professionals who had good exposure to security analytic tools. The researcher added a

total of six screening questions to the existing set of 50 questions. These questions had the twin

purpose of improving the quality of the survey and bringing in security professionals from

multiple business domains. The purposes of these questions were as follows: (Q1) obtaining

informed consent from the participant, as mandated by the institutional review board (IRB); (Q2)

providing an option to the participant to be a volunteer in the study without any coercion; (Q3)

procuring commitment from the survey participant to provide thoughtful answers to the

questions; (Q4) ensuring that any participant in the survey had a minimum of one year’s

exposure to security analytic tools; (Q5) ensuring active involvement of the participant in the

information security domain in the last year; and (Q6) ascertaining the business / industry of the

participant (finance, retail, healthcare, government, other).

Figure 5. Security analytics research – A waterfall approach to orient the Delphi study

participant.

Research problem Lack of assessment factors for security analytics tools in the currently

available and surveyed academic literature (not commercial assessments).

Research topic To assess such tools, the first step is to identify factors that impact and

influence a successful implementation.

Identify factors To identify factors it is very important that the user community of these

tools is surveyed – .

Security analytics questionnaire

To survey these users a security analytics questionnaire was developed

by the researcher based on his initial knowledge.

Field study of questionnaire A small group of experts reviewed the above questionnaire and

made necessary changes to it.

Delphi study by expert panel (Phase 1) A bigger group of security analytics experts will refine the

questionnaire and this process will go for many rounds (at least

three) till consensus

EFA study by actual participants (Phase 2) The questionnaire which is the outcome of Phase 1 (above)

will be modified (format wise) to suit the exploratory factor

analysis (EFA) study and will be distributed to cyber security

professionals in the industry within North America.

Statistical analysis performed on EFA study data Quantitative statistical techniques will be applied to

the data returned from EFA study to extract success

factors and make final corrections to the questionnaire.

You are one of

the Delphi

panel experts

and your role

ends with Phase

The researcher had established a contract with Qualtrics for 200 responses based on

sample size requirements (Chapter 3) and cost constraints. Qualtrics published the survey and

made it available for five days after recruiting the cybersecurity professionals from within its list

of professionals. A total of 2,202 professionals took the survey. Some 1,497 professionals did not

have any experience in cybersecurity, so they did not get beyond Q3. Only 705 participants

reached Q5. Of these 705 participants, 319 participants had one year’s experience in the security

analytics domain, 167 participants had two years of experience in security analytics domain, 219

participants had three years of experience in security analytics domain, and 87 more participants

were not able to proceed beyond Q5, as their cybersecurity experience was not current. A final

list of 373 completed the survey. Qualtrics provided the best 210 responses to the researcher

based on the contract agreement. Qualtrics provided 10 responses free of cost. The researcher

discarded four responses, as they contained repeated straight-line responses. Simple analytics

performed on the 206 responses revealed the following statistics (Tables 3 and 4).

Table 3

Participant Domain Experience

Security Analytic Domain Experience (Years) Participant Count

1 71

2 38

3 or more 97

Table 4

Participant Industry

Industry Participant Count

Finance 52

Healthcare 31

Retail 45

Government 21

Other 57

Table 3 indicates that 47% (97 counts) of the participants had used security analytics

tools for three or more years in their information security career. Some 66% (135 counts) of the

survey participants had more than two years of experience in using security analytic tools and

processes. Table 4 provides the count of participants from different industries. Finance, retail,

and other categories formed 75% of the survey participants.

Delphi Study

Round 1 of the Delphi Technique

The Round 1 questionnaire for the security analytic study went to 20 security analytic

experts who signed up to participate in the study. Sixteen of the experts returned the filled-in

questionnaire after two reminders went out. The researcher consolidated all 16 responses to the

32 questions. Skulmoski et al. (2007) opined that the first round of Delphi is mostly an initial

brainstorming round. In this round, the panel generated the maximum set of new ideas, as panel

members looked at the questionnaire for the first time. Based on the feedback, the researcher

took a few actions: (a) the researcher consolidated all the Round 1 feedback and communicated it

to the panel members, (b) the researcher computed Kendall’s coefficient to understand the extent

of convergence among panel members; (c) the researcher sent 25 new questions that emerged to

the panel members, (d) the researcher used the percent agreement method to assess consensus.

The focus areas of feedback for Round 1 were (a) tool service level agreements; (b) upgrades,

availability, and failover support; (c) log source management; (d) tool coverage, compliance, and

licensing; and (e) cloud environment and architecture. Kendall’s coefficient for Round 1 was

0.208 (Table 5).

Table 5

Results of the Delphi Rounds

Delphi

Round

# of

Panel

Experts

# of Items

(at the

Start of

the

Round)

Kendall’s

Coefficient

(W)

% Agreement

Among Panel

Members for

Relevancy of

Question/Item

Notes

1 16 32 0.208 81% Initial input to the panel

(output of the field study)

contained 32 questions.

The expert panel found

81% of the questions

relevant

2 15 57 0.148 77% Twenty-five questions

added based on expert

panel feedback before the

start of Round 2. The

expert panel found 77% of

the questions relevant

3 13 50 0.147 100% Seven questions removed

based on refinement

feedback from the panel

before the start of Round

4 11 50 0.212 100% No additions of new

questions were made.

Minor changes to existing

questions.

Applying the percent agreement method, panel members found that 81% of the questions

were either relevant or highly relevant, thereby approving those questions for Phase 2 of the

study. The researcher considered all the responses to every question except “neutral” and “not

applicable” responses in computing percent agreement. For every question, the researcher

computed an average of the responses. For 81% of the questions, the average rating was less than

2. On the Likert-type scale, the researcher assigned the following values to the Delphi survey

responses: (a) 1 for “highly relevant,” (b) 2 for “relevant,” (c) 3 for “neutral,” (d) 4 for “less

relevant,” and (e) 5 for “irrelevant.” Vonhof (2015) followed a similar process of employing the

Likert-type scale in Delphi survey. For the big data section of the survey only, the researcher

introduced an additional option (6: not applicable). The researcher did this for two reasons: (a)

many organizations may not have implemented big data security analytics, and (b) participants

may not want to commit to a new or emerging area of technology in a survey, and hence they

should have an option to ignore the question; a “neutral” option will not cover the above

conditions. The descriptive statistics are in Table 6.

Round 2 of the Delphi Technique

Round 2 started with 57 questions that emerged from the collective feedback from the

panel in Round 1. The researcher added 25 new questions and modified some others. Delphi

panelists reviewed the consolidated feedback after the completion of Round 1 and accepted

them. There was no feedback from the panel for the consolidated report after Round 1. Hence,

the researcher converted all the feedback from Round 1 into formal questions. Major feedback

for Delphi Round 2 focused on the following areas: (a) threat intelligence, (b) big data-based

security analytic tools, and (c) pruning of unsuitable questions. Consolidated feedback went to

the Round 2 participants. Kendall’s coefficient for Round 2 was 0.148. A reduced Kendall’s

coefficient was predictable for Round 2, as the number of questions nearly doubled. One

participant dropped out of the Delphi study and the number of participants for Round 2 was 15.

Table 6

Delphi Round 1– Descriptive Statistics

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q1 16 1.50 0.632 1 3

Q2 16 1.63 0.619 1 3

Q3 16 2.25 1.238 1 5

Q4 16 1.69 0.793 1 4

Q5 16 1.69 0.704 1 3

Q6 16 2.06 0.929 1 4

Q7 16 1.81 0.911 1 4

Q8 16 2.31 1.250 1 4

Q9 16 1.88 1.088 1 4

Q10 16 2.13 0.957 1 4

Q11 16 1.75 0.856 1 4

Q12 16 1.56 0.727 1 3

Q13 16 1.56 0.629 1 3

Q14 16 2.06 0.854 1 4

Q15 16 1.75 0.775 1 4

Q16 16 1.63 0.719 1 3

Q17 16 1.81 0.981 1 4

Q18 16 1.69 1.078 1 5

Q19 16 2.06 0.854 1 3

Q20 16 1.63 0.885 1 4

Q21 16 1.50 0.516 1 2

Q22 16 1.56 0.727 1 3

Q23 16 1.50 0.632 1 3

Q24 16 1.50 0.632 1 3

Q25 16 1.56 0.814 1 3

Q26 16 1.75 0.775 1 3

Q27 16 1.88 0.957 1 5

Table 6 (continued)

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q28 16 2.31 1.352 1 5

Q29 16 2.88 1.310 1 5

Q30 16 3.06 1.237 1 5

Q31 16 2.88 1.204 1 5

Q32 16 2.75 1.291 1 5

As per the percent agreement method, the number of questions that the participants found

relevant was 77% of the 57 questions. The researcher ignored “neutral” and “not applicable”

responses in computing the percentage of agreement. This drop in percentage triggered the need

to prune questions that were not relevant to the survey from the list. As a result, the researcher

removed seven questions from the list based on the feedback from the panel members. The

descriptive statistics for Round 2 are provided in Table 7.

Round 3 of the Delphi Technique

Round 3 started with 50 questions, and these questions went to the Delphi panel for

further feedback and review. Even though there were no new questions or new areas of security

analytics, feedback focus from the panel was on questions in big data security analytics. Panelists

suggested changes to these questions concerning some foundational aspects of big data.

Consequently, after applying these changes, big data questions were more focused and effective

in extracting answers from potential participants. Consolidated feedback on Round 3 went to the

participants just before the start of Round 4.

Table 7

Delphi Round 2 – Descriptive Statistics

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q1 15 1.33 0.488 1 2

Q2 15 1.53 0.516 1 2

Q3 15 1.93 0.961 1 4

Q4 15 1.73 0.458 1 2

Q5 15 1.73 0.458 1 2

Q6 15 1.73 0.594 1 3

Q7 15 1.93 1.033 1 4

Q8 15 1.80 0.941 1 4

Q9 15 1.80 0.862 1 4

Q10 15 2.47 0.834 1 4

Q11 15 1.73 0.594 1 3

Q12 15 2.13 0.990 1 4

Q13 15 2.47 0.834 1 4

Q14 15 1.53 0.516 1 2

Q15 15 2.40 0.910 1 4

Q16 15 1.73 0.594 1 3

Q17 15 2.00 1.000 1 4

Q18 15 1.60 0.507 1 2

Q19 15 2.13 0.834 1 4

Q20 15 1.53 0.640 1 3

Q21 15 2.27 0.961 1 4

Q22 15 2.00 1.000 1 4

Q23 15 2.00 0.926 1 4

Q24 15 1.80 0.941 1 4

Q25 15 1.53 0.516 1 2

Q26 15 1.67 0.617 1 3

Q27 15 1.93 0.704 1 3

Table 7 (continued)

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q28 15 1.60 0.632 1 3

Q29 15 1.87 0.834 1 3

Q30 15 2.00 0.756 1 3

Q31 15 2.13 0.743 1 3

Q32 15 2.20 0.941 1 4

Q33 15 1.67 0.724 1 3

Q34 15 2.07 0.884 1 4

Q35 15 2.40 0.910 1 4

Q36 15 2.00 0.756 1 3

Q37 15 2.40 0.910 1 4

Q38 15 2.33 0.900 1 4

Q39 15 2.00 0.926 1 4

Q40 15 2.13 0.915 1 4

Q41 15 2.00 1.134 1 5

Q42 15 2.27 1.033 1 5

Q43 15 1.93 0.704 1 3

Q44 15 2.27 0.884 1 4

Q45 15 2.27 1.163 1 6

Q46 15 2.47 1.246 1 6

Q47 15 2.60 1.242 1 6

Q48 15 2.67 1.234 1 6

Q49 15 2.47 1.187 1 6

Q50 15 2.47 1.356 1 6

Q51 15 2.20 1.320 1 6

Q52 15 2.80 1.320 1 6

Q53 15 2.53 1.246 1 6

Q54 15 2.67 1.234 1 6

Q55 15 2.47 1.246 1 6

Table 7 (continued)

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q56 15 2.53 1.125 2 6

Q57 15 2.47 1.246 1 6

Two participants dropped out of the study in Round 3. Only 13 participants participated

in Round 3 and provided responses within the waiting period of three weeks. Kendall’s

coefficient was 0.147, almost the same as Round 2. However, as per the percent agreement

method, the number of questions that the panel found relevant increased to 100%. The researcher

ignored “neutral” and “not applicable” responses in computing the percentage of agreement. As

per the original agreement, the researcher had only planned three rounds of Delphi study.

However, based on the feedback in Round 3, with the specific focus on big data security

analytics, and the lack of consensus, the researcher decided to conduct one more round of the

Delphi study. The researcher notified the participants of this and gave them the option to drop

out of Round 4 if they were not interested in proceeding to one more round. Descriptive statistics

for this round are provided in Table 8.

Round 4 of the Delphi Technique

Round 4 started with 50 questions that emerged from Delphi Round 3. The researcher

incorporated the changes and modifications the panel members suggested in Round 3 before the

questionnaire went out to the panel members for Round 4. The questionnaire went to the 13

members who were still in the study by the end of Round 3. After the waiting period of three

weeks, the researcher received feedback from 11 participants.

Table 8

Delphi Round 3 – Descriptive Statistics

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q1 13 1.46 0.660 1 3

Q2 13 1.62 0.961 1 4

Q3 13 1.62 0.768 1 3

Q4 13 1.54 0.519 1 2

Q5 13 1.69 0.751 1 3

Q6 13 1.85 0.689 1 3

Q7 13 1.31 0.630 1 3

Q8 13 1.69 0.947 1 4

Q9 13 1.54 0.519 1 2

Q10 13 1.85 0.801 1 3

Q11 13 1.69 0.751 1 3

Q12 13 1.69 0.630 1 3

Q13 13 1.85 0.689 1 3

Q14 13 1.54 0.660 1 3

Q15 13 1.62 0.650 1 3

Q16 13 1.46 0.660 1 3

Q17 13 1.85 0.689 1 3

Q18 13 1.46 0.519 1 2

Q19 13 2.00 1.000 1 4

Q20 13 1.69 0.480 1 2

Q21 13 1.54 0.660 1 3

Q22 13 1.31 0.480 1 2

Q23 13 1.54 0.660 1 3

Q24 13 1.85 0.689 1 3

Q25 13 1.54 0.519 1 2

Q26 13 1.46 0.519 1 2

Q27 13 1.77 0.599 1 3

Table 8 (continued)

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q28 13 1.31 0.480 1 2

Q29 13 1.77 0.832 1 3

Q30 13 1.62 0.768 1 3

Q31 13 1.23 0.439 1 2

Q32 13 1.69 0.751 1 3

Q33 13 1.62 0.650 1 3

Q34 13 1.92 0.641 1 3

Q35 13 1.77 0.725 1 3

Q36 13 1.92 0.954 1 4

Q37 13 1.85 0.801 1 3

Q38 13 1.77 0.927 1 4

Q39 13 1.69 0.751 1 3

Q40 13 1.62 0.768 1 3

Q41 13 2.31 1.377 1 6

Q42 13 2.69 1.251 1 6

Q43 13 2.46 1.266 1 6

Q44 13 2.62 1.660 1 6

Q45 13 2.08 1.382 1 6

Q46 13 2.23 1.363 1 6

Q47 13 2.46 1.266 1 6

Q48 13 2.38 1.261 1 6

Q49 13 2.38 1.325 1 6

Q50 13 2.23 1.363 1 6

Two Delphi participants dropped out of the study during Round 4, bringing down the

number of participants in the final round to 11. Except for a few very minor changes to

grammatical aspects of the questions, the panel did not propose any major changes in Round 4.

They did not add any new questions in Round 4. Kendall’s coefficient increased to 0.212 in

Round 4. As per the percent agreement method, for all the 50 questions, panel members came to

100% agreement that all the questions were relevant and were potential candidates for Phase 2 of

the research study. While Kendall’s coefficient of consensus increased by 33% in the final round,

the consensus built using the percent agreement method remained at 100%. These consensus

readings aided in the conclusion that Round 4 brought stability to the entire survey. Hence the

researcher decided to terminate the Delphi study at Round 4. At the end of the fourth round of

the survey, 50 questions remained, and these 50 questions became the input to the EFA phase

(Phase 2) of the study. Descriptive statistics for this round are provided in Table 9.

Table 9

Delphi Round 4 – Descriptive Statistics

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q1 11 1.09 0.302 1 2

Q2 11 1.27 0.467 1 2

Q3 11 1.36 0.505 1 2

Q4 11 1.36 0.505 1 2

Q5 11 1.18 0.405 1 2

Q6 11 1.55 0.522 1 2

Q7 11 1.27 0.467 1 2

Q8 11 1.36 0.505 1 2

Q9 11 1.36 0.505 1 2

Q10 11 1.73 0.905 1 3

Q11 11 1.36 0.505 1 2

Q12 11 1.36 0.505 1 2

Q13 11 1.36 0.505 1 2

Table 9 (continued)

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

Q14 11 1.27 0.467 1 2

Q15 11 1.64 0.674 1 3

Q16 11 1.64 0.674 1 3

Q17 11 1.73 0.647 1 3

Q18 11 1.64 0.674 1 3

Q19 11 1.82 0.603 1 3

Q20 11 1.36 0.505 1 2

Q21 11 1.27 0.467 1 2

Q22 11 1.55 0.688 1 3

Q23 11 1.27 0.467 1 2

Q24 11 1.55 0.522 1 2

Q25 11 1.36 0.505 1 2

Q26 11 1.55 0.522 1 2

Q27 11 1.64 0.674 1 3

Q28 11 1.36 0.674 1 3

Q29 11 1.64 0.809 1 3

Q30 11 1.45 0.688 1 3

Q31 11 1.27 0.647 1 3

Q32 11 1.45 0.688 1 3

Q33 11 2.00 0.894 1 4

Q34 11 2.00 1.000 1 4

Q35 11 1.45 0.688 1 3

Q36 11 1.64 1.027 1 4

Q37 11 1.73 0.786 1 3

Q38 11 1.36 0.674 1 3

Q39 11 1.55 0.688 1 3

Q40 11 1.45 0.688 1 3

Q41 11 2.18 1.471 1 6

Table 9 (continued)

Descriptive Statistics

N Mean Std. Deviation Minimum Maximum

`Q42 11 2.18 1.471 1 6

Q43 11 2.45 1.864 1 6

Q44 11 2.64 1.362 1 6

Q45 11 2.36 1.567 1 6

Q46 11 2.27 1.421 1 6

Q47 11 2.27 1.489 1 6

Q48 11 2.55 1.809 1 6

Q49 11 2.45 1.864 1 6

Q50 11 2.36 1.912 1 6

EFA

There was no hypothesis testing involved in this research study, but construct validity of

the survey instrument was performed. The researcher used the KMO measure of sampling

adequacy to validate the sample size. As per Field (2013), a value greater than or equal to 0.9 is

marvelous. SPSS produced a value of 0.926 for the KMO test on the above sample. The output

from the KMO test is included in Table 10.

Table 10

Sample Adequacy Test

KMO Measure of Sampling Adequacy .926

Bartlett’s Test of Sphericity Approx. Chi-Square 4700.035

df 946

Sig. .000

The researcher next performed a correlation of 50 items/questions as recommended by

Field (2013). The analysis procedure is described in Figure 6. Tabachnick and Fidell (2007)

suggested that any correlation between variables that is less than 0.3 is indicative of weak

correlation. Field further asserted that researchers should exclude any variable with many

occurrences of weak correlation and should consider them for factor analysis instead. Yong and

Pearce (2013) concurred on eliminating variables with many occurrences of correlation

coefficients less than 0.3. Based on the above citations, the researcher eliminated six questions

from the list of 50 questions: Q41, Q42, Q46, Q31, Q51, and Q12. This elimination left 44

questions for the next part of the data analysis.

Figure 6. Phase 2 – EFA study.

Factor extraction was the next step as given in Figure 6. Williams et al. (2010) asserted

that principal axis factoring is the most useful method to extract factors. Yong and Pearce (2013)

stated that scree plot is a method that many research studies have used to identify the number of

factors. The researcher applied both these methods to extract factors. Eigenvalues are the

deciding parameters in concluding the factors as per the principal axis factoring method. Kaiser

(1960) recommended that researchers should include factors with eigenvalues greater than 1.

Based on this approach, nine factors emerged for this study.

Table 11

Total Variance Explained (SPSS)

Total Variance Explained

Factor Initial Eigenvalues Extraction Sums of Squared

Rotation

Sums

of Squared

Loadings

Total % of

Variance

Cumulative

Total % of

Variance

Cumulative

1 15.937 36.221 36.221 15.457 35.129 35.129 10.472

2 2.182 4.96 41.181 1.731 3.934 39.063 18.423

3 1.862 4.149 45.33 1.404 3.191 42.257 25.755

4 1.345 3.057 48.387 0.868 1/972 44.226 32.221

5 1.266 2.878 51.265 0.788 1.791 46.017 38.443

6 1.2 2.727 53.991 0.719 1.634 47.652 43.542

7 1.149 2.612 56.603 0.676 1.537 49.189 47.051

8 1.061 2.412 59.015 0.595 1.351 50.54 49.708

9 1.043 2.371 61.386 0.571 1.299 51.389 51.839

As per Table 11, nine factors accounted for 61% of cumulative variance before rotation

and 52% after rotation. All nine factors have eigenvalues more than 1.

Based on the scree plot analysis in Figure 7, the inflection point happens just after 5 on

the x-axis. While there is a difference between the scree plot and an eigenvalue-based approach

for a given number of extracted factors, factor rotation methods are very effective in deciding the

number of factors based on the factor loadings. Da Veiga et al. (2007) recognized that a factor

loading of more than 0.3 is a meaningful value and variables that load more than 0.3 are

necessary for any given factor for this study. Rotated factor level details from the SPSS tool are

given in Table 12.

Figure 7. SPSS scree plot analysis.

Table 12

Rotated Factor Matrix – SPSS Output

Rotated Factor Matrixa

Factor

1 2 3 4 5 6 7 8 9

Q54 0.667 0.181 0.155 0.147 0.199 0.144 0.106

Q48 0.639 0.239 0.125 0.139 0.110 0.191 0.135

Q53 0.614 0.112 0.157 0.153 0.158 0.342

Q55 0.612 0.154 0.173 0.169 0.193

Q52 0.604 0.208 0.223 0.125 0.211

Q49 0.589 0.135 0.156 0.117 0.469 0.108 0.126

Q50 0.575 0.152 0.111 0.192 0.223 0.168

Q56 0.503 0.119 0.146 0.220 0.443

Q57 0.501 0.231 0.252 0.208 0.179 -0.183

Q18 0.185 0.612 0.126 0.153

Q19 0.131 0.608 0.154 0.153 0.232 0.187

Q23 0.606 0.276 0.250 0.142

Q30 0.187 0.424 0.271 0.307 0.211 0.177

Q36 0.203 0.402 0.198 0.314 0.182 0.142 0.309 0.277

Q16 0.140 0.366 0.236 0.265 0.191 0.322

Q20 0.238 0.362 0.216 0.182 0.184 0.176 0.215 0.136 0.212

Q7 0.154 0.204 0.746 0.112

Q8 0.150 0.710 0.124 0.122 0.108 0.103 0.215

Q9 0.227 0.177 0.611 0.158 0.145 0.121 0.107 -0.173

Q10 0.187 0.213 0.543 0.143 0.172 0.102 0.288 0.297

Q11 0.280 0.225 0.437 0.242 0.203 0.200 0.253

Q37 0.253 0.122 0.666 0.117 0.242 0.184

Q27 0.221 0.195 0.216 0.556 0.223 0.158 0.122

Q26 0.226 0.242 0.179 0.503 0.188 0.103 -0.112 0.286

Q43 0.132 0.160 0.419 0.185 0.278 0.281

Q38 0.253 0.227 0.101 0.384 0.220 0.133 0.105 0.114

Table 12 (continued)

Rotated Factor Matrixa

Factor

1 2 3 4 5 6 7 8 9

Q22 0.194 0.228 0.321 0.354 0.191 0.269 0.260 0.184

Q29 0.198 0.323 0.132 0.325 0.225 0.276 0.320 0.155 0.243

Q25 0.237 0.114 0.147 0.159 0.643 0.132

Q15 0.139 0.328 0.186 0.121 0.522 0.103 0.157

Q14 0.160 0.365 0.226 0.229 0.407 0.227 -0.110

Q24 0.187 0.287 0.173 0.151 0.401 0.210 0.187 0.109 0.116

Q39 0.199 0.256 0.170 0.162 0.401 0.301

Q28 0.246 0.278 0.210 0.375 0.390 0.214

Q35 0.235 0.177 0.141 0.184 0.289 0.220 0.263 0.218

Q44 0.188 0.162 0.234 0.194 0.676 0.110

Q34 0.199 0.268 0.330 0.327 0.421 0.146

Q45 0.240 0.211 0.182 0.131 0.391 0.151 0.201 0.112

Q40 0.276 0.236 0.320 0.381 0.222 0.101

Q32 0.182 0.239 0.112 0.257 0.282 0.362 0.260

Q33 0.306 0.342 0.193 0.169 0.342 0.109

Q17 0.156 0.331 0.321 0.251 0.108 0.132 0.382

Q13 0.141 0.273 0.255 0.174 0.280 0.131 0.343 0.330

Q21 0.209 0.247 0.182 0.205 0.252 0.220 0.507

a Rotation converged in nine iterations. Extraction method: Principal Axis Factoring.

Before interpreting extracted factors, it is necessary to rotate them. Factor rotation leads

to better interpretation since unrotated factors are ambiguous (Yong & Pearce, 2013). There are

two types of rotation: quartimax and varimax. As per Yong and Pearce (2013), varimax rotation

reduces the number of variables that load onto a factor, whereas quartimax rotation reduces the

number of factors. Two major aims of this research are relevant here. The first aim was to extract

the common underlying factors from the variables. The second aim was to optimize and distill

the questions in the survey to generate a meaningful instrument. Hence the researcher chose

varimax rotation. In Table 12, Factors 1 through 6 are easily identifiable with the help of strong

factor loadings. It includes all the loadings with a value of 0.3 and above. In Table 12, there is a

clear decrease in the loadings for Factor 1 after Q57. Similarly, for Factor 2, the loadings reduce

after Q20. Extending the same concept up to Factor 6, there are strong factor loadings in the

range of 0.4 to 0.6.

However, in the case of Factor 7, only two loadings were close to each other (for Q17

and Q13). But, there are strong loadings in Q49 and Q30 for Factor 7, and they also happen to be

cross-loadings. The researcher needed to decide whether to drop a cross-loading item from the

analysis (Costello & Osborne, 2005). As per Costello and Osborne (2005) usually researchers do

not consider cross-loadings if there are other strong loadings that are above 0.5. But in the case

of Factor 7, all the factor loadings were in the range of 0.3 to 0.46. The eigenvalue for Factor 7 is

1.149 (Table 11). These four questions dealt with closely connected areas of the security analytic

domain such as SOC, network anomalies, and tuning. Due to the interconnectedness of the four

loadings mentioned above, the researcher calculated Cronbach’s alpha for all these four factor

loadings to ascertain whether to include Factor 7 in the final list of factors based on the reliability

aspect of the factor. The result of Cronbach’s alpha gave a value of 0.69 for Factor 7 with the

four items mentioned above. Since the rounded value is 0.7, the minimum mark mandated by

Field (2013), the researcher included Factor 7 in the list of factors for this research study. Factors

8 and 9 listed above do not display strong factor loadings. Hence, they are not on the final list of

factors. To summarize, the researcher retained Factors 1 through 7 for further analysis. Factor 8

had only one factor loading and hence the researcher discarded it.

100

Reliability analysis. Reliability analysis is usually a proven technique to establish

instrument reliability. Cortina (1993) asserted that it is easy to ascertain the reliability of an

instrument using the value for Cronbach’s alpha. Field (2013) opined that if the value for

Cronbach’s Alpha is in the range of 0.7 to 0.8, then the instrument reliability is acceptable. Any

value less than 0.7 is not an accepted value for proving the reliability of an instrument. The

researcher performed a reliability test for the list of 44 variables or questions at both the

instrument level and the factor level, the results of which follow.

The reliability of the instrument demonstrates that the instrument measures what it should

measure and that the results are applicable and generalizable to a larger population (Cronbach,

1951). As Table 13 and Table 14 show, reliability at the instrument level is 0.959, suggesting

that the instrument is very cohesive in its reliability.

Table 13

Reliability – Overall Instrument Level

Reliability Statistics

Cronbach’s Alpha Cronbach’s Alpha Based on Standardized Items No, of Items

.958 .959 44

Table 14

Case Processing Summary

N %

Cases Valid 206 100.0

Excludeda 0 .0

Total 206 100.0

a. Listwise deletion based on all variables in the procedure. Scale: Overall_instrument_level.

101

All the 44 items are closely connected and measure the same set of underlying constructs. The

researcher also computed Cronbach’s alpha at factor level to ensure reliability at the individual

factor level. The results are in Table 15.

Table 15

Cronbach’s Alpha at Factor Level

Factor Number of Items Eigenvalue Cronbach’s Alpha

1 9 15.937 0.883

2 7 2.182 0.829

3 5 1.826 0.838

4 7 1.345 0.839

5 6 1.266 0.82

6 6 1.2 0.796

7 4 1.149 0.693

After factor rotation and removal of the double-counting items (Q49 and Q30), 42

questions formed the final output of this study, and they are included in the Appendix A.

Naming of factors. Based on the questions that the researcher categorized after factor

rotation, some common themes for every factor emerged, and the researcher used those common

themes to name the factors. Williams et al. (2010) recommended that factor naming is a

subjective task for the researcher, based on the context of the research and the results of data

analysis. After many rounds of thematic analysis of the questions, the researcher developed the

following table of factor names. More applications of these factors are provided in Chapter 5.

Table 16 provides the details on factor names.

102

Table 16

Factor Names

Factor # Themes/Keywords from Survey Questions Factor Name

Factor 1 Big data platform, incident management and reduction,

correlation, attack detection, network and user behavioral

anomalies

Large-scale security

event analysis

Factor 2 Tool usability, user interface, SOC productivity,

scalability, tuning and false positives, user training

Functional utilization

Factor 3 Correlation analysis, attack prevention, security incident

detection, and suspicious network activity/events

Incident detection and

correlation analysis

Factor 4 Security posture, security health, security process

improvement, reporting capabilities, compliance

management, vulnerability management

Governance and CISO

metrics

Factor 5 Use cases, log sources, categories of log sources, and use

case review

Log source and use

case management

Factor 6 Distributed denial of service (DDoS) attack detection,

authentication attack detection, monitoring, SOC

operational efficiency, actionable intelligence

Threat and operational

intelligence

Factor 7 Real-time attack detection, SOC and network anomaly Real-time attack and

anomaly detection

Summary

This chapter has presented and discussed the results of both Phase 1 and Phase 2 of the

research study. For Phase 1 of the study, four Delphi rounds took place, allowing the participants

to provide their inputs. The researcher applied Kendall’s coefficient and the percent agreement

method to assess the consensus, leading to a 50-question survey instrument. After the fourth

round of Delphi study, 50 questions went to Phase 2 of the research. Phase 2 (the EFA part of the

study) applied a five-step method to perform analysis and to filter out variables or questions that

did not fit into the overall survey instrument. The researcher eliminated six variables or questions

even before conducting factor analysis during the correlation analysis stage (R-matrix). Forty-

four questions formed the input to the factor extraction step. The researcher eliminated two

103

cross-loading questions after the factor rotation, resulting in 42 variables or questions. Thus, the

researcher achieved construct validity using the EFA technique. The researcher validated the

reliability of the instrument using Cronbach’s alpha. The researcher computed Cronbach’s alpha

for all seven factors and found that all these factors were within the acceptable limits, thereby

validating the entire instrument as reliable.

104

CHAPTER 5. DISCUSSION, IMPLICATIONS, AND RECOMMENDATIONS

Introduction

This chapter contains a summary of the results and a discussion of the impact of this

study on the future of research in the domain of security analytics. Cardenas et al. (2013)

summarized the evolution of the security analytics domain by defining three levels of maturity:

(a) Intrusion detection and prevention system (IDPS) related analytics, (b) security information

and event management (SIEM) analytics, and (c) big data-related security analytics. There were

three research questions to this study. The first omnibus question dealt with the determination of

factors that determine the successful implementation of security analytic tools as a whole. The

second dealt with the determination of factors that impact the successful implementation of non-

big data security analytic tools. The third dealt with the determination of factors that impact the

successful implementation of big data-based security analytic tools. Apart from addressing these

questions, the development of a survey instrument dedicated to security analytic domain,

grounded in theory and expert opinion, was another major goal of this research. There is a

summary of those results in the following sections.

Summary of the Results

Even though security analytics tools and products have been in existence for close to 10

years, there was a gap in the literature in terms of a lack of assessment of these tools, and there

was also a lack of expert-validated survey instrument with a theoretical foundation. Hinde (2005)

provided the first introduction to security analytics with respect to fraud detection but did not

105

build an assessment framework. The basis of this research was the understanding that an expert-

validated instrument in security analytics will form the trigger for more research questions in this

area. This study addressed the suggestion of Ferketich et al. (1993) that the development of a

survey instrument is the best way to establish foundational research in any new or unexplored

area. This research has successfully accomplished the above goals, as explained in the following

salient points: (a) the researcher has extracted factors that impact the implementations using

Phase 1 and Phase 2 of the study, (b) Phase 1 applied the Delphi research method to produce an

initial survey instrument, (c) Phase 2 applied exploratory factor analysis (EFA) to refine this

instrument and to arrive at factors, and (d) the researcher used Cronbach’s alpha to validate the

reliability of the survey instrument.

All the results were very satisfactory in producing quantitative results. For example, the

Delphi study concluded in four rounds, ending with good consensus among participants, and the

final round of Delphi had 11 participants complete the survey instrument. Stability emerged

between the third and fourth rounds. True to the opinions expressed by Heiko (2012) in his

consensus and stability research for Delphi studies, good consensus and stability emerged using

the percent agreement method and Kendall’s coefficient in this study. Very few changes

occurred between the third and fourth rounds of Delphi, thereby rendering the instrument more

stable.

Chapter 2 presented a literature analysis of the security analytics domain. It indicated a

gap in the literature in terms of a lack of expert-validated survey instrument to broadly assess

security analytics tools. The goal of developing a survey instrument is to identify factors that

determine a successful implementation of security analytic tools. The research questions are

applicable to all three generations of security analytic tools. These three generations of tools

106

were categorized by Cardenas et al. (2013). Chapter 2 discussed both the operational and the

strategic aspects of the security analytic tools. The literature review covered tool performance,

scalability, user interface, user training, actionable intelligence, correlational ability, SOC

interface, and incident management. The theoretical foundation for this research came from five

major theories: (a) game theory, (b) computational learning theory, (c) DS theory or evidence

theory, (d) the MapReduce programming model, and (e) TAM-3. Chapter 2 discussed

applications of these theories in the field of security analytics with examples. Questions in the

final survey instrument indirectly alluded to the above theories. Chapter 2 included a discussion

of commercial surveys of security analytics tools and their limitations. The researcher distilled

appropriate concepts from the security analytic domain towards the end of Chapter 2 for

inclusion in the survey instrument. These concepts took the form of seed questions for the field

study. The output of the field study consisted of 32 questions. These questions went to Delphi

Round 1 to initiate the Delphi study. Based on the opinions on quantitative surveys by

Pinsonneault and Kraemer (1993), the researcher wove these concepts in as survey questions

during the field study.

Chapter 3 discussed the methodology for both Phase 1 and Phase 2. It explained the

research design in detail using flow diagrams. It explained the quantitative Delphi method and its

usefulness in arriving at a consensus for emerging areas of research, based on Garson (2014). It

applied discussions on quantitative Delphi by Garson in the formulation of initial questions using

a five-point Likert-type scale. Chapter 3 also explained Delphi consensus methods based on the

elaborate work done by Heiko (2012). In addition to this, Chapter 3 discussed participant

selection for the Delphi study. The Delphi study was an extensive and complicated process that

the researcher executed with full adherence to Belmont principles and other ethical principles.

107

There was a discussion of data collection for both Phase 1 (Delphi) and Phase 2 (EFA) with

emphasis on the sample size and the rationale behind it. Finally, the researcher employed a

multistep process for establishing instrument validity using EFA. The researcher demonstrated

the reliability of the survey instrument using Cronbach’s alpha. Seven factors had emerged by

the end of Phase 2 of this research study. Those seven factors accounted for 57% of the total

variance before rotation and accounted for 47% after varimax rotation. Those factors were (a)

large-scale security event analysis, (b) functional utilization, (c) incident detection and

correlation analysis, (d) governance and CISO metrics, (e) log source and use case management,

(f) threat and operational intelligence, and (g) real-time attack and anomaly detection

Discussion of the Results

Security Analytics Implementation – Seven Success Factors

EFA was a major part of the second phase of this research study. This part fulfilled the

construct validity requirement of this study. Vonhof (2015) utilized a similar approach to identify

factors for his research study on telework adoption. This section explains seven factors that

impact the successful implementation of a security analytic tool (Table 17). Factors may not

come from the core part of the security analytic tool. There could be subfactors that are external

to the implementation. It is necessary to assess both intrinsic and extrinsic aspects of the tool

landscape to arrive at the name of the factors. For example, Factor 1 is large-scale security event

analysis. This factor explains the ability of the tool and its environs to perform large-scale event

analysis. Even though the tool primarily drives large-scale security event analysis, computing

power, and network bandwidth at the implementation site indirectly influence the

implementation success.

108

Table 17

Factor Definitions

Factor # Factor Name Factor explanation

Factor 1 Large-scale security event

analysis

This factor is the ability of the security

analytic tool to perform large-scale security

event analysis, and this factor is closely

related to big data tools.

Factor 2 Functional utilization Function utilization is the ability of the users

to effectively utilize the tool to achieve

higher levels of security.

Factor 3 Incident detection and

correlation analysis

Incident detection is the ability of the tool to

identify suspicious events by performing

correlation analysis.

Factor 4 Governance and CISO metrics Governance factor refers to the capability of

the tool to support CISO level metrics and

reporting.

Factor 5 Log source and use case

management

Log sources are identified by business

stakeholders in collaboration with tool

experts.

Factor 6 Threat and operational

intelligence

Threat intelligence refers to the ability of the

tool to provide input to the security

operations center resources.

Factor 7 Real-time attack and anomaly

detection

This factor refers to the tool’s ability to

detect an attack in real-time and report it.

Factor 1: Large-scale security event analysis. The ability of a security analytic tool to

perform large-scale security event analysis is an important factor. While many tools in the

current market have this ability, this factor points to all the sub-factors and the depth of the tool’s

scalability to perform large-scale security event analysis. Oltsik (2013) argued that big data-

based security analytic tools will provide better incident detection and incident response. Several

survey questions in the instrument address subfactors to this factor. RQ3 focuses on big data-

driven security analytics. All the big data-related questions in the survey loaded into Factor 1.

Nine survey questions cover this factor. The large-scale security event analysis factor completely

109

answers RQ3. As per the EFA (Phase 2) results, the majority opinion pointed to big data security

analytic tools being useful as supplementary tools to existing tools and products.

Factor analysis indicated that simulation of security incidents (e.g., malware infection)

and prediction of potential attack scenarios were also important subfactors in this study. The

predictive ability of big data security analytic tools is a major decision-making point for many

customers involved in the initial purchase cycle of these tools. Colbaugh and Glass (2011)

discussed the possibility of such tools predicting future attack situations and timelines. Big data

survey questions and underlying factors have vindicated the solid opinion of Colbaugh and

Glass. These tools exploit the historical data of past attacks and attack situations in detecting

attack scenarios. Wheelus et al. (2016) developed a unique solution for threat intelligence

analytics, and they used the HIVE tool to query the output of the security analytics. Bou-Harb et

al. (2016) utilized the concept of behavioral service graphs to detect user behavioral anomalies.

Based on the above discussions, it is clear that large-scale security event analysis factor has the

following sub-factors: (a) attack prediction, (b) attack simulation, (c) query capability, and (d)

anomaly detection

Factor 2: Functional utilization. RQ2 had six factors that emerged from the EFA.

Factors 2 through 7, as presented in Chapter 4, relate to RQ2. While Factor 1 was fully

representative of big data security analytics, all the other factors belonged to non-big-data

security analytics, thereby directly answering RQ2. In summary, one factor emerged for big data

security analytics (RQ3) and six factors for non-big data security analytics emerged from EFA

(Phase 2). Functional utilization refers to the ability of the security analytics tool to empower the

users and administrators of the tool in using its innate capabilities. Factor analysis strongly

pointed to the user’s ability to understand the tool within a one-year period is an important aspect

110

of the implementation. Venkatesh and Bala (2008) introduced the concept of user perception of

any new technology. Seven survey questions loaded into this factor. Some of the subfactors that

emerged from the factor analysis are as follows.

Shackleford (2014) opined that close to 48% of the users of security analytic tools were

unsatisfied by the training they received. Factor analysis indicated that both functional and

technical training on the tool is a significant influencer in user perception of security analytic

tools. The user interface was yet another aspect influencing the success of security analytic tool

implementation. Crespo and Garwood (2014) argued that the user interface plays a powerful role

in helping the user to understand the tool. Shackleford (2016) found that SOC analysts lack the

skills to deal with security analytic tools. Rules and scalability are other influencers in the

successful implementation of any security analytic tool. The survey instrument that underwent

EFA produced Factor 2, that is, functional utilization. The above-discussed influencers form the

following subfactors: (a) user understanding, (b) user training, (c) SOC interface, (d) rules and

scalability, and (e) user interface.

Factor 3: Incident detection and correlation analysis. EFA produced six factors for

RQ2. Factor 3 is incident detection and correlation analysis. Incident detection is a common

thread among the questions that belong to this factor. Crespo and Garwood (2014) researched the

problem of botnets and recommended that a deeper analysis of security incidents is necessary to

detect all types of security incidents. Correlation analysis of heterogeneous log sources is yet

another influencer in Factor 3. Alserhani (2016) presented an alert fusion approach using

correlational analysis. Five survey questions loaded into this factor. These influencers form the

subfactors incident detection and correlation analysis.

111

Factor 4: Governance and CISO metrics. The fourth factor that the EFA identified is

governance and CISO metrics. Most of the influencers in this group pointed to extrinsic

subfactors. For example, security analytic tools do not necessarily produce all the metrics

necessary for enterprise-level security governance. Hence outputs from security analytic tools, in

the form of reports or dashboard, is a major input to governance metrics. Gordas (2014)

discussed the relevant points in implementing a security analytic tool for small and medium-level

enterprises. Metrics was a major aspect of this discussion. Factor 4 directly addressed RQ2.

Compliance with laws and auditing requirements are very important influencers in the

implementation of security analytic tools. Many security analytic tools support both external

compliance with laws and internal compliance to IT mandates of the corporation. Van de

Moosdijk et al. (2015) discussed in detail the ability of SIEM tools to support compliance

requirements. O’Hara (2010) revealed that network-level attacks are a major concern to the

security posture and security health of even giants like Google Corporation. Many questions that

loaded into this factor focused on security posture and health. Quantitative measurements of

posture and health are regular inputs to CISOs. Reports and dashboards on governance regularly

go to CISOs to apprise the office about the security posture of the organization. Seven questions

from the survey instrument loaded into this factor. Based on the above citations and discussions,

the following subfactors apply to this factor: (a) security posture, (b) security metrics, (c)

compliance, and (d) executive reporting.

Factor 5: Log source and use case management. The fifth factor is log source and use

case management. Like Factor 4, this factor also included some extrinsic variables and

influencers. Some of the common threads for this factor are log sources and use cases. Log

sources are boarded into the security analytic tool from all the locations of any given

112

organization. Log sources also include heterogeneous logs from all the business groups within an

organization. Van de Moosdijk et al. (2015) provided extensive details about log sources in their

work on SIEM governance. Identifying and boarding log sources is a team effort and not

necessarily an intrinsic ability of a security analytic tool. Use cases are the driving points for

both rules and log sources.

Van de Moosdijk et al. (2015) discussed the application of use cases and rules in their

SIEM discussion. Use cases are the result of the requirements of business groups. For example, a

data loss prevention or firewall log may be a valid input to a security analytic tool. The

individual business group needs to identify the business case. This aspect drives home the point

that security analytic tools cannot attain success by their attributes alone. Many inputs from the

IT landscape of the organization are necessary to attain success in protecting the network and

data, using security analytic tools. Identifying and understanding the root causes of a cyber-

attack or a data breach is key to preventing future attacks. Root cause analysis performed on any

attack reveals the vulnerable parts of the organization’s network and data. This kind of analysis

feeds into the use case management process. Nicolett and Kavanagh (2013) categorized learning

from root cause analysis as a critical capability of security analytic tools. Six questions from the

survey instrument loaded into this factor. Based on the above discussions, the subfactors for this

factor are (a) use case management, (b) log source management, and (c) incident root cause

analysis process.

Factor 6: Threat and operational intelligence. The sixth factor is the threat and

operational intelligence. Many of the questions that loaded for this factor focus on faster

detection of specific types of cyber-attacks like Distributed denial of service (DDoS) attack after

implementation of the security analytic tool. In other words, the build-up of threat intelligence

113

and operational intelligence for SOC analysts is the common thread in these questions. Yen et al.

(2013) discussed the need for security analytic tools to have updated threat intelligence

capabilities. Bou-Harb et al. (2016) extracted actionable intelligence from the security analytic

tool they had built. Malekpour et al. (2014) expounded the ability of learning algorithms to

prevent fraudulent behavior. While actionable intelligence is more helpful in preventing future

attacks and even zero-day attacks, the operational intelligence security analytic tools provide to

SOC analysts will boost the analysts’ productivity by helping them in the monitoring of SOC

operations and log sources. Six questions loaded into this factor. Based on the above discussion,

the subfactors are (a) SOC monitoring, (b) actionable intelligence, and (c) faster attack detection.

Factor 7: Real-time attack and anomaly detection. The seventh factor is real-time

attack and anomaly detection. While there have been discussions of attacks in earlier factors, it is

important to understand that real-time attack detection is a slightly different aspect of security

analytic tools. Traffic behavioral analysis is necessary to detect attacks in real time. Raiyn (2014)

presented a survey cyber-attack detection strategy. SOC analysts increasingly rely on such tools

to understand anomalies and attacks. It is possible to understand anomalous behavior, both user

behavior, and network behavior, using such real-time tools. However, SOC analysts are in the

immediate field of a cyber-attack, and they will be the experts who conclude that an anomaly has

occurred. Elliott (2016) described the functionality of a typical security operation center (SOC)

and the SOC analyst. Four questions loaded into this factor; two of them were cross-loadings. In

Chapter 4, there was an extensive discussion on the reasons for including cross-loadings for

Factor 7. Based on the above discussion, the subfactors are real-time attack detection and

anomaly detection. Anomaly detection is a common influencer in both big data-based security

analytic tools and non-big-data-based tools.

114

Findings

Based on the detailed discussions above, the findings of the research in terms of the seven

factors are well aligned with the concepts presented in the literature review section of Chapter 2.

The survey instrument also included these concepts in the form of survey questions.

Development of the survey instrument focused on security analytics through two phases, along

with its validity and reliability tests, have proven that there indeed was a gap in the literature

regarding non-availability of a survey instrument and a lack of clarity in terms of factors to

consider when implementing a security analytic tool. All the questions at the end of Phase 2 still

maintained a strong connection to the underlying theories. For example, Q54 reads, “The big

data security analytic process in my organization could reduce large datasets into smaller ones

thereby helping incident the response team to focus better on the incident investigation.” This

question clearly portrays the underlying MapReduce model. In another example, Q11, whose

final version is “Correlation of log source data feeding into the security analytic tool has

significantly enhanced the incident detecting capabilities” has a direct connection to the DS

theory concepts explained in Chapter 2. Similarly, Q18, “Owners and users of the security

analytic tool could understand and use the tool fully within one year of its implementation”

alluded to the TAM-3 theoretical constructs explained in Chapter 2.

Conclusions Based on the Results

This research has remedied the lack of a survey instrument to probe research questions in

security analytics. Even though the survey instrument passed through many rounds of refinement

and validation, this research has answered the three basic research questions of this study: ORQ

– “What are the factors that impact the successful implementation of analytic tools?”, RSQ1 –

“What are the factors that determine the successful implementation of non-big data security

115

analytic tools?” and RSQ2 – “What are the factors that determine the successful implementation

of big data security analytic tools?” The final instrument contained 42 questions and these

questions covered seven factors. These seven factors along with their subfactors can help in the

assessment of any security analytic implementation. Cardenas et al. (2013) introduced the

concept of three maturity levels for security analytic tools and processes. These seven factors

apply to all three levels. At the first level of maturity, IDPS can apply some of the factors. For

example, attack detection and anomaly detection applies to IDPS devices and their analytics

landscape. At the second level of maturity, these factors apply to a very great extent to SIEM

devices and tools. In a typical low-end SIEM device with more focus on log processing, large-

scale security event analysis (Factor 1) is not fully applicable. However, other factors are

applicable. At the third level of maturity, a fully grown SIEM solution combined with big data

analytical capability, all factors are fully applicable. A summary of the seven success factors for

any security analytic implementation is in Table 18.

As explained in Chapter 1, these factors will immensely benefit the security program

managers responsible for implementing and measuring the success of security analytic tools and

processes. Based on the factors given above, any given security analytic program manager can

focus on measuring these factors, depending on the specifics of implementation. Managers can

establish the goals of any security analytic program with the help of the above factors. For

example, van de Moosdijk et al. (2015) discussed the different possible components of any

security analytic program implementation. Some financial organizations may focus more on

predicting cyber-attacks. In contrast, a small enterprise will try to establish a minimum program

for the safety of the corporate network. Some healthcare organizations will focus more on

compliance with health care laws.

116

Table 18

Factors and Subfactors

Factor # Factor Name Subfactors

Factor 1 Large-scale security event

analysis

Attack prediction, attack simulation, query

capability, anomaly detection

Factor 2 Functional utilization User understanding, user training, SOC interface,

rules and scalability, user interface

Factor 3 Incident detection and

correlation analysis

Incident detection, correlation analysis

Factor 4 Governance and CISO

metrics

Security posture, security metrics, compliance,

executive reporting

Factor 5 Log source and use case

management

Use case management, Log source management,

incident root cause analysis process

Factor 6 Threat and operational

intelligence

SOC monitoring, actionable intelligence, faster

attack detection

Factor 7 Real-time attack and

anomaly detection

Real-time attack detection, anomaly detection

Nicolett and Kavanagh (2013) produced the survey material for the Gartner research

report, but it lacked theoretical and research considerations. However, the use cases in this

research survey were very broad and detailed. The resulting survey instrument focuses on both

the strategic and the operational side of a security analytic implementation. Real experts in the

industry built the survey instrument in Phase 1 part of the study using the Delphi process. At the

end of the Delphi study, there were 11 experts who contributed to the survey instrument. In the

second phase (EFA), the researcher surveyed many security analytic professionals and refined

the survey instrument. Before managers decide to buy or build a security analytic product, the

factors this research has identified can help in the buying process. Security experts can also use

the factors this study has identified when they build the checklist for any implementation of a

security analytic tool. Crespo and Garwood (2014) described an ideal SIEM analytic engine.

Extending the concept of the analytic engine described by Crespo and Garwood, the seven

117

success factors that this study identified can be an ideal assessment standard for architects and

program managers in the security analytics domain.

One of the discussions in Chapter 2 focused on investment in security analytic tools.

Small and medium enterprises have limited IT budgets. Before investing in security analytic

tools, they ideally should become aware of the factors that impact the successful implementation

of such tools. The factors this research study has identified provide a structure to such enterprises

about the nature of these tools and the processes surrounding them. Chapter 2 discussed a

solution provided by Chari et al. (2013) in detail. The focus of this discussion was on compliance

by internal users to prevent insider attacks. One of the subfactors this study identified was

compliance, both external and internal. Yet another subfactor was anomalous behavior. Chari et

al.’s solution addressed both these subfactors.

Chapter 2 described many types of cyber-attacks and the ability of security analytic

solutions to identify and prevent them. Crespo and Garwood (2014) analyzed the ACDC project

and explained the ability of the project to dilute botnet attacks. Their conclusion was a match to

three subfactors identified in this study; attack prediction, attack simulation, and real-time attack

detection. Yen et al.’s (2013) discussion focused on advanced, persistent type of attacks. Some of

these analytic solutions omitted the reduction of false positives as a necessary goal. Reducing

false positives is in the survey instrument as an individual question. False positives are in Factor

2: Functional utilization. Factor 1 is large-scale security event analysis. Big data-based security

analytic tools related to RQ3. All the big data-related attributes loaded into a single factor –

large-scale security event analysis. Wheelus et al. (2016) discussed a very elaborate solution for

big data-based security analytics. Chapter 2 discussed many use cases of this solution. Attack

prediction, attack simulation, querying analytic results, and anomaly detection were the

118

subfactors of Factor 1. The solution built by Wheelus et al. addressed all these subfactors.

Network anomalies are major use cases for security analytic tools. Big data tools, with their

ability to process large datasets, can identify network anomalies. There are many questions in the

survey instrument dedicated to network analytics. Bhuyan et al. (2014) expounded the

fundamental aspects of network security analytics. Factor 1 covers most of the literature on big

data-related security analytics. Based on the discussions earlier in this section, it is clear that the

literature review, the results in terms of factors, and the survey instrument are in agreement to a

great extent. This agreement leads to the conclusion that the researcher has achieved the

dissertation goals.

Limitations

The researcher delimited the sample size to 200 participants. Chapter 3 explained in

detail that this was sufficient for the EFA. However, in any quantitative analysis involving

surveys, a higher sample size is always better. Yong and Pearce (2013) suggested that a sample

size of 300 or more will reduce the error in the data. Tabachnick and Fidell (2013) professed that

a minimum sample size of 500 is ideal for EFA. Fabrigar, Wegener, MacCallum, and Strahan

(1999) opined that sample sizes for EFA studies vary among researchers. The researcher

undertook this study to extract factors that determine the success of implementation of security

analytic tools and as a secondary goal to produce a survey instrument. Yong and Pearce (2013)

recommended that confirmatory factor analysis is the best way to validate the factors identified

in an EFA study. A confirmatory factor analysis was beyond the scope of this study. However,

future studies may employ confirmatory factor analysis to validate the factors. Any future studies

performed to revalidate the survey instrument produced by this research should consider a bigger

audience with the help of a neutral sponsor.

119

From a security analytic domain perspective, there is one limitation that needs a mention

in this section. Artificial intelligence (AI)-driven security analytics, or cognitive security, is the

current trend. Rao (2016) introduced the concept of Security-360, a cognitive and adaptive

approach to enterprise security, currently under research at IBM research labs. There are many

opinions on cognitive security by leading research labs. However, AI-driven security will have a

major influence in the immediate future. Cardenas et al. (2013) introduced three levels of

maturity for security analytics, with the third level being big data-driven security. Based on this

research study and the current trends in cognitive security, AI-driven security will become the

fourth level of maturity. Even though the survey instrument the researcher has produced in this

research focuses on cyber-attacks, the questions do not cover all the well-known cyber-attacks.

In future versions of this instrument, it will be possible to introduce many probing questions to

focus on the ability of the security analytic tools to handle emerging types of attacks (e.g.,

ransomware attacks). Also, there were not enough questions on cloud-driven security analytics.

Future studies need to address this area. Finally, all the participants in Phase 2 of the study

(EFA) came from the United States and Canada. Future researchers can overcome this limitation.

Implications for Practice

The survey instrument that resulted from this research study also identified seven factors after

the exploratory factor analysis (EFA) study (Phase 2). Information security and cybersecurity

professionals can benefit in many ways from this research study. Many implications of this study

are discussed in this section.

Implications for practitioners. Practitioners can use factors that this research has

generated to assess the implementation of different market leading products. Cerullo, Formicola,

Iamiglio, and Sgaglione (2014) compared leading products like QRadar, ArcSight, and enVision.

120

They provided a deep technical analysis. However, it will be ideal to use the seven factors this

research has generated to assess such tools, along with the subfactors. Security consulting

organizations may use the factors extracted during this research study to meet their clients’

needs. When a security analytic tool implementation takes place in any organization,

measurement of success of the implementation and its benefits is a key performance indicator

(KPI) to the executives. Metrics defined based on these factors and subfactors can be useful as

benchmarks to measure the implementation success. For example, in this study, incident

detection is part of Factor-3. Mateski et al. (2012) defined many metrics around incident data.

Average incident count, when repeatedly measured for every month, can help the organization

set some benchmark values. This benchmark will help to identify any outliers and its causes

thereby driving investigations on security incidents. Van de Moosdijk et al. (2015) examined

security governance using analytic tools. The components they discussed in their research can

blend with the factors extracted in this research to bolster security governance in any

organization.

Implications for researchers. Researchers can break down these factors and subfactors

further to serve the needs of security architects. For example, attack simulation is a subfactor.

Many algorithms and other technical data points make up this subfactor. It is necessary to

measure the effectiveness of such techniques to assess the success of this subfactor in any

implementation. It is also possible to extend factors from this study to technical aspects of

security analytics. Yen et al. (2013) discussed the implementation of their Beehive security

analytic tool. The processing speed of log sources was a major metric in their study. This metric

will roll up under Factor 5 (log management and use case management). All of these examples

may be of interest to the research community.

121

Recommendations for Further Research

There are many avenues for further research based on the results of this study. Factor 3 in

this research study is incident detection and correlation analysis. Incident detection is key to any

type of investigation that can confirm an attack or data breach. Security analytic tools can

produce inbuilt reports as discussed by van de Moosdijk et al. (2015) in their research on SIEM.

Researchers can further probe the impact of correlation analysis on identifying security incidents

using statistical techniques. For example, they can explore whether there is a causal relationship

between correlation and incident detection. There are many types of correlation tools and

engines. Rosa et al. (2015) compared different correlation engines and their speeds and

processing power.

Identifying factors that are useful to assess a successful implementation of security

analytic tool was the primary goal of this investigation. Measurement of those factors using

either a quantitative approach or a qualitative approach was not within the scope of this study.

Measurements are usually made using metrics that could be either quantitative or qualitative.

This research study ended with the identification of factors. Metrics for measuring the factors

and subfactors, once defined in a separate study, will extend this research to be a more

productive tool. Factor-3 included correlation analysis. For example, the speed of correlation

engine is measured by the number of security events processed per second. Mateski et al. (2012)

included security threat metrics as a part of their research at Sandia Labs. Similar operational

metrics can be produced based on the factors that are identified in this research study.

Cloud computing has emerged as the future of IT. Many organizations are migrating

towards the cloud. With the emergence of less expensive cloud infrastructures, security analytic

tools are migrating to the cloud. Yam, Baldwin, Shiu, and Ioannidis (2011) provided their

122

opinion on investment in cloud computing and its future. The survey instrument produced in this

research study can be easily customized to include questions on security analytics in a cloud

environment. Even though a security analytic tool and its functions remain almost the same in a

cloud environment, the infrastructure is not anymore a standalone infrastructure and hence

certain tool functionalities are impacted by the cloud environment. Exploring the entire study or

parts of the study for the cloud environment is yet another future research option. Many factors

identified in this research study are applicable to the cloud environment.

User acceptance of any new technology is not attained easily. Venkatesh and Bala (2008)

defined TAM-3 variables as a part of their enhancement to the technology acceptance model.

Perceived usefulness (PU) and perceived ease of use (PEOU) are two variables that are

commonly used to assess the user acceptance of any new technology. While this research study

focused on a broader goal of identifying success factors, there was no in-depth probing of the

user acceptance aspects of the security analytic products and tools. For example, the perceived

ease of use (PEOU) of any given architecture is a promising area to generate more research

questions in the field of security analytics. Functional utilization (Factor 2) is the factor suitable

for exploring further on the TAM-3 related research questions.

Log source and use case management (Factor 5) is an important area of security

analytics. Van de Moosdijk et al. (2015) discussed log sources and use cases in their work on

SIEM tools. Effect of log sources on security incidents is an area of research that can add value

to future studies. Business stakeholders identify use cases that need to be included for analysis in

a security analytic tool. For example, email server logs are a good candidate to be boarded into a

security analytic tool as a log source. Once a log source is boarded, there is a high potential for

security incident data to be impacted by the new log source.

123

Conclusion

This study investigated three research questions in the security analytics domain, in

addition to producing a survey instrument whose construct validity and reliability were fully

validated. The first question pertained to factors that impact the successful implementation of

security analytics tools as a whole. The second question pertained to factors that impact the

successful implementation of non-big data security analytics tools. The third question pertained

to factors that impact the successful implementation of big data security analytics tools. Phase 1

of this study used a Delphi iterative process to produce a survey instrument. The Delphi process

went through four rounds and produced a 50-question survey instrument. In Phase 2, this survey

instrument underwent further refinement by eliminating eight more questions. The final survey

instrument consisted of 42 questions dedicated to the area of security analytics. Phase 2

employed an EFA study that produced seven factors and 23 subfactors. This chapter has

discussed the results and limitations of this research study in detail. It has also presented future

research possibilities with a focus on implications for practice. Since the survey instrument has

undergone refinement through many rounds, future research in security analytics can easily

employ this instrument by performing a confirmatory factor analysis and by exploring many

other possible variable relationships and research questions based on the extracted factors.

124

REFERENCES

Adams, D. P., & Miles, T. P. (2013). The application of Belmont Report principles to policy

development. Journal of Gerontological Nursing, 39(1), 16–21. doi:10.3928/00989134-

20131028-07

Ahlan, A. R., & Ahmad, B. I. E. (2014). User acceptance of health information technology (HIT)

in developing countries: a conceptual model. Procedia Technology, 16, 1287–1296.

Alserhani, F. M. (2016). Alert correlation and aggregation techniques for reduction of security

alerts and detection of multistage attack. International Journal of Advanced Studies in

Computers, Science and Engineering, 5(2), 1–7.

Avella, J. R. (2016). Delphi panels: Research design, procedures, advantages, and challenges.

International Journal of Doctoral Studies, 11, 305–321.

Barron, E. N. (2013). Game theory: An introduction (Vol. 2). Hoboken, NJ: Wiley.

Bhuyan, M. H., Bhattacharyya, D. K., & Kalita, J. K. (2014). Network anomaly detection:

Methods, systems and tools. IEEE Communications Surveys & Tutorials, 16(1), 303–336.

Blum, A. L. (1994). Relevant examples and relevant features: Thoughts from computational

learning theory. In AAAI Fall Symposium on Relevance. Menlo Park, CA: AAAI Press.

Retrieved from https://aaai.org/Papers/Symposia/Fall/1994/FS-94-02/FS94-02-005.pdf

Bou-Harb, E., Scanlon, M., & Fackha, C. (2016). Behavioral service graphs: A big data approach

for prompt investigation of internet-wide infections. In 2016, 8th IFIP International

conference on new technologies, mobility and security (NTMS) (pp. 1–5). Larnaca,

Cyprus: IEEE. http://doi.org/10.1109/NTMS.2016.7792437

Boudreau, M.-C., Gefen, D., & Straub, D. W. (2001). Validation in information systems

research: A state-of-the-art assessment. MIS Quarterly,

25(1),1. http://doi.org/10.2307/3250956

Bowers, K., Hart, C., Juels, A., & Triandopoulos, N. (2013). Securing the data in big data

security analytics. IACR Cryptology ePrint Archive, 2013, 625.

Buczak, A. L., & Guven, E. (2016). A survey of data mining and machine learning methods for

cybersecurity intrusion detection. IEEE Communications Surveys & Tutorials, 18(2),

1153-1176

Burton, L. J., & Mazerolle, S. M. (2011). Survey instrument validity part I: Principles of survey

instrument development and validation in athletic training education research. Athletic

Training Education Journal, 6(1), 27–35.

Carroll, T. E., & Grosu, D. (2011). A game theoretic investigation of deception in network

security. Security and Communication Networks, 4(10), 1162–1172. doi:10.1002/sec.242

http://doi.org/10.2307/3250956

125

Cardenas, A. A., Manadhata, P. K., & Rajan, S. P. (2013). Big data analytics for security. IEEE

Security & Privacy, 11(6), 74–76. doi:10.1109/MSP.2013.138

Cerullo, G., Formicola, V., Iamiglio, P., & Sgaglione, L. (2014). Critical infrastructure

protection: Having SIEM technology cope with network heterogeneity. arXiv preprint.

arXiv:1404.7563.

Chari, S., Habeck, T., Molloy, I., Park, Y., & Teiken, W. (2013). A big data platform for

analytics on access control policies and logs. In Proceedings of the 18th ACM Symposium

on Access Control Models and Technologies – SACMAT ’13, USA, 1–264.

doi:10.1145/2462410.2462433

Charishma, P., & Venkatesh, K. (2015). Big data security analytic solution using Splunk.

International Journal of Engineering Research and Applications, 5(4 [Part 3]), 50–53.

Retrieved from http://www.ijera.com

Chaturvedi, M., Narain Singh, A., Prasad Gupta, M., & Bhattacharya, J. (2014). Analyses of

issues of information security in Indian context. Transforming Government: People,

Process and Policy, 8(3), 374–397.

Chen, L. (2009). Curse of dimensionality. In Encyclopedia of Database Systems (pp. 545-546).

Springer US.

Chen, P. C. (2013). Experience-based cyber security analytics (Doctoral dissertation). Available

from ACM Digital Library. (UMI No. AAI3576193)

Cheng, F., Azodi, A., Jaeger, D., & Meinel, C. (2013). Multi-core supported high performance

security analytics. In 2013 IEEE 11th International Conference on Dependable,

Autonomic and Secure Computing, USA, 621–626. doi:10.1109/DASC.2013.136

Chung, K., Kamhoua, C. A., Kwiat, K. A., Kalbarczyk, Z. T., & Iyer, R. K. (2016). Game theory

with learning for cyber security monitoring. In 2016 IEEE 17th International symposium

on high assurance systems engineering (HASE) (pp. 1–8). Orlando, FL:

IEEE. http://doi.org/10.1109/HASE.2016.48

Chuvakin, A. (2010). The complete guide to log and event management. Retrieved

from http://www.novell.com/docrep/2010/03/Log_Event_Mgmt_WP_DrAntonChuvakin

_March2010_Single_en.pdf

Chuvakin, A. (2016). Demystifying security analytics: Data, methods, use cases. Paper presented

at the RSA Conference (2016, March). San Francisco, CA.

Retrieved from https://www.rsaconference.com/writable/presentations/file_upload/air-

t09-demystifying-security-analytics-data-methods-use-cases-final.pdf

Colbaugh, R., & Glass, K. (2011). Proactive defense for evolving cyber threats. In Proceedings

of 2011 IEEE International Conference on Intelligence and Security Informatics, ISI

2011 (pp. 125–130). Washington, DC: IEEE. doi:10.1109/ISI.2011.5984062

http://doi.org/10.1109/HASE.2016.48

http://www.novell.com/docrep/2010/03/Log_Event_Mgmt_WP_DrAntonChuvakin_March2010_Single_en.pdf

126

Cooper, W. W., Gallegos, A., & Granof, M. H. (1995). A Delphi study of goals and evaluation

criteria of state and privately owned Latin American airlines. Socio-Economic Planning

Sciences, 29(4), 273–285.

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications.

Journal of Applied Psychology, 78(1), 98–104.

Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four

recommendations for getting the most from your analysis. Practical Assessment,

Research & Evaluation, 10(7), 1–9.

Crespo, B. G.-N., & Garwood, A. (2014). Fighting botnets with cyber-security analytics: Dealing

with heterogeneous cyber-security information in new generation SIEMs. In 2014 Ninth

International Conference on Availability, Reliability and Security (pp. 192–198).

Fribourg, Switzerland: IEEE. doi:10.1109/ARES.2014.33

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika,

16(3), 297–334.

Curtis, B., Krasner, H., & Iscoe, N. (1988). A field study of the software design process for large

systems. Communications of the ACM, 31(11), 1268–1287.

Cybenko, G., & Landwehr, C. E. (2012). Security analytics and measurements. IEEE Security &

Privacy Magazine, 10(3), 5–8. doi:10.1109/MSP.2012.75

Da Veiga, A., Martins, N., & Eloff, J. H. (2007). Information security culture–Validation of an

assessment instrument. Southern African Business Review, 11(1), 147–166.

Davis, F. D., Jr. (1986). A technology acceptance model for empirically testing new end-user

information systems: Theory and results (Doctoral dissertation). Retrieved from

http://www.researchgate.net/

Davidson, P. L. (2013). The Delphi technique in doctoral research: Considerations and rationale.

Review of Higher Education and Self-Learning, 6(22), 53–65.

Dawis, R. V. (1987). Scale construction. Journal of Counseling Psychology, 34(4), 481.

Dean, J., & Ghemawat, S. (2010). MapReduce: A flexible data processing tool. Communications

of the ACM, 53(1), 72–77.

Dittrich, D., & Kenneally, E. (2012). Applying ethical principles to information and

communication technology research: A companion to the Department of Homeland

Security Menlo Report. Retrieved from

http://www.dhs.gov/sites/default/files/publications/CSD-MenloPrinciplesCOMPANION-

20120103-r731.pdf

127

Duffield, C. (1993). The Delphi technique: A comparison of results obtained using two expert

panels. International Journal of Nursing Studies, 30(3), 227–237.

Ehsan, H., & Khan, F. A. (2012, June). Malicious AODV: implementation and analysis of

routing attacks in MANETs. In Trust, Security and Privacy in Computing and

Communications (TrustCom), 2012 IEEE 11th International Conference (pp. 1181-

1187). IEEE.

Elliott, T. (2016). Predominance of using security information and event management tools to

enhance security analytics for mitigating threats (Doctoral dissertation). Available from

Dissertations & Theses @ Utica College. (Order No. 10183247). Retrieved from

https://search.proquest.com/docview/1853118200?accountid=28902

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use

of exploratory factor analysis in psychological research. Psychological Methods, 4(3),

272.

Ferketich, S., Phillips, L., & Verran, J. (1993). Development and administration of a survey

instrument for cross‐cultural research. Research in Nursing & Health, 16(3), 227–230.

Field, A. (2013). Discovering statistics using IBM SPSS statistics. Thousand Oaks, CA: Sage.

Fielder A., Panaousis E., Malacaria P., Hankin C., & Smeraldi, F. (2014) Game Theory Meets

Information Security Management. In N. Cuppens-Boulahia, F. Cuppens, S. Jajodia, A.

Abou El Kalam, & T. Sans (Eds.), ICT Systems security and privacy protection. SEC

2014. IFIP Advances in Information and Communication Technology,

Vol 428 (pp. 15–29). Heidelberg, Germany: Springer.

Gantsou, D. (2015). On the use of security analytics for attack detection in vehicular ad hoc

networks. In 2015 International conference on cyber security of smart cities, industrial

control system and communications (SSIC), China. 1–6. doi:10.1109/SSIC.2015.7245674

Garrido-Moreno, A., & Padilla-Meléndez, A. (2011). Analyzing the impact of knowledge

management on CRM success: The mediating effects of organizational factors.

International Journal of Information Management, 31(5), 437–444.

Garson, G. D. (2014). The Delphi method in quantitative research. Asheboro, NC: Statistical

Associates.

Glass, K., & Colbaugh, R. (2011). Web analytics for security informatics. In Proceedings – 2011

European Intelligence and Security Informatics Conference, EISIC 2011, USA. 214–219.

doi:10.1109/EISIC.2011.66

Gliddon, D. G. (2006). Forecasting a competency model for innovation leaders using a modified

Delphi technique (Doctoral dissertation). Available from ProQuest Dissertations &

Theses Global. (Order No. 3292523)

128

Gold, R. (1967). Optimal binary sequences for spread spectrum multiplexing (corresp.). IEEE

Transactions on Information Theory, 13(4), 619–621.

Goldman, S. A. (1995). Computational learning theory. doi:10.1007/3-540-59119-2

Goodman, C. M. (1987). The Delphi technique: A critique. Journal of Advanced Nursing, 12(6),

729–734.

Gordas, V. (2014). Implementing information security management system in SMEs and

ensuring effectiveness in its governance. Egham, Surrey, UK: Information Security

Group. Retrieved from http://www.ma.rhul.ac.uk/static/techrep/2014/RHUL-MA-2014-

6.pdf

Grublješič, T., & Jaklič, J. (2015). Business intelligence acceptance: The prominence of

organizational factors. Information Systems Management, 32(4), 299–315.

Gunaydin, H. M. (2006). The Delphi method. Optimization Group. Retrieved from

http://www.iyte.edu.tr/~muratgunaydin/delphi.htm

Han, Z. (2012). Game theory in wireless and communication networks: theory, models, and

applications. Cambridge: Cambridge University Press.

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey

technique. Journal of Advanced Nursing, 32(4), 1008–1015.

Hathaway, O., Crootof, R., Levitz, P., Nix, H., Nowlan, A., Perdue, W., & Spiegel, J. (2012).

The law of cyber-attack. California Law Review, 100(4), 817–885. Retrieved

from http://www.jstor.org/stable/23249823

Heiko, A. (2012). Consensus measurement in Delphi studies: Review and implications for future

quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536.

Hinde, S. (2005). No state can exist without the confidence of the people. Computer Fraud &

Security, 2005(8), 18-20

Howarth, F. (2014). Security intelligence: Solving the puzzle for actionable insight. London, UK:

Bloor Research. Retrieved from http://www.bloorresearch.com

Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus.

Practical Assessment, Research & Evaluation, 12(10), 1–8.

Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014). Toward scalable systems for big data analytics: A

technology tutorial. IEEE Access, 2, 652–687. doi:10.1109/ACCESS.2014.2332453

Iivari, J. (1991). A paradigmatic analysis of contemporary schools of IS development. European

Journal of Information Systems, 1(4), 249.

129

Janssens, W., de Pelsmacker, P., & van Kenhove, P. (2008). Marketing research with SPSS.

Boston, MA: Pearson Education.

Jupp, V. (2006). The Sage dictionary of social research methods. Thousand Oaks, CA: Sage.

Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and

Psychological Measurement, 20(1), 141–151.

Kavanagh, K. M., Rochford, O., & Bussa, T. (2015). Magic quadrant for security information

and event management. Gartner, Tech. Rep

Kent, K. (2007). Guide to computer security log management. Gaithersburg, MD: National

institute of standards and technology. Retrieved

from http://logrhythm.com/Portals/0/resources/NIST Guide Log Mgmt SP800-

92.pdf%5Cnhttp://m.sagedatasecurity.com/pdfs/SP800-92-NIST-Guide-to-Log-

Management.pdf

Kohlas, J., & Monney, P.-A. (1994). Theory of evidence – A survey of its mathematical

foundations, applications and computational aspects. ZOR – Mathematical Methods of

Operations Research, 39(21), 35–68.

Leclerc, G., Lefrancois, R., Dubé, M., Hébert, R., & Gaulin, P. (1998). The self-actualization

concept: A content validation. Journal of Social Behavior and Personality, 13(1), 69.

Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003). The technology acceptance model: Past,

present, and future. Communications of the Association for Information Systems, 12(1),

752–780. Retrieved from http://aisel.aisnet.org

Liang, X., & Xiao, Y. (2013). Game theory for network security. IEEE Communications Surveys

& Tutorials, 15(1), 472–486.

Liu, P., Zang, W., & Yu, M. (2005). Incentive-based modeling and inference of attacker intent,

objectives, and strategies. ACM Transactions on Information and System Security, 8(1),

78–118. doi:10.1145/1053283.1053288

Lops, P., de Gemmis, M., Semeraro, G., Narducci, F., & Musto, C. (2011). Leveraging the

linkedin social network data for extracting content-based user profiles. In Proceedings of

the Fifth ACM Conference on Recommender Systems - RecSys ’11 (p. 293). New York,

NY: ACM Press. http://doi.org/10.1145/2043932.2043986

Lu, Y., & K. (Ram) Ramamurthy. (2011). Understanding the link between information

technology capability and organizational agility: An empirical examination. MIS

Quarterly, 35(4), 931–954. Retrieved from http://www.jstor.org/stable/41409967

http://logrhythm.com/Portals/0/resources/NIST

130

Mahmood, T., & Afzal, U. (2013). Security analytics: big data analytics for cybersecurity: A

review of trends, techniques and tools. In 2013 2nd National Conference on Information

Assurance (NCIA) (pp. 129–134). Rawalpindi, Pakistan:

IEEE. http://doi.org/10.1109/NCIA.2013.6725337

Malekpour, M., Khademi, M., & Minae-bidgoli, B. (2014). A hybrid data mining method for

intrusion and fraud detection in e-Banking systems. Journal of computational intelligence

and electronic systems, 3,1–6. http://doi.org/10.1166/jcies.2014.1068

Malhotra, N. (1981). A scale to measure self-concepts, person concepts, and product concepts.

Journal of marketing research, 18(4), 456–464. doi:10.2307/3151339

Mateski, M. E., Trevino, C. M., Veitch, C. K., Michalski, J. T., Harris, J. M., Maruoka, S., &

Frye, J. N. (2012). Cyber threat metrics. Albuquerque, NM: Sandia National

Laboratories. Retrieved from http://prod.sandia.gov/techlib/access-

control.cgi/2012/122427.pdf

Mayhew, M., Atighetchi, M., Adler, A., & Greenstadt, R. (2015). Use of machine learning in big

data analytics for insider threat detection. In MILCOM 2015 - 2015 IEEE Military

Communications Conference (pp. 915–922). Tampa, FL, USA:

IEEE. http://doi.org/10.1109/MILCOM.2015.7357562

Mensch, S., & Wilkie, L. (2011). Information security activities of college students: An

exploratory study. Journal of Management Information and Decision Sciences, 14(2), 91.

Mohammed, N., Fung, B. C. M., & Debbabi, M. (2011). Anonymity meets game theory: Secure

data integration with malicious participants. VLDB Journal, 20(4), 567–588.

doi:10.1007/s00778-010-0214-6

Nash, J. F. (1950). Equilibrium points in n-person games. Proceedings of the National Academy

of Sciences, 36(1), 48–49 (Original work published 1950)

National Institutes of Health, Office of Human Subjects Research. (1979). The Belmont Report:

Ethical principles and guidelines for the protection of human subjects of research.

Washington, DC: Government Printing Office.

Newsom, J. (2005). A quick primer on exploratory factor analysis. In SEM (pp. 1–4). Retrieved

from http://web.cortland.edu/andersmd/psy341/efa.pdf

Nicolett, M., & Kavanagh, K. M. (2013). Critical capabilities for security information and event

management. Stamford, CT: Gartner RAS Core Research. Retrieved from

http://dss.lv/f/Critical_Capabilities_for_Security_Information_and_Event_Management_-

_2013_Q1Labs_IBM_Security_Systems.pdf

O’Hara, G. (2010). Cyber-espionage: A growing threat to the American economy. CommLaw

Conspectus, 19, 241.

131

Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design

considerations and applications. Information & Management, 42(1), 15–29.

Oltsik, J. (2013, April 1). Defining big data security analytics [Blog post]. Retrieved from

https://www.csoonline.com/article/2224394/cisco-subnet/defining-big-data-security-

analytics.html

Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations:

Research approaches and assumptions. Information Systems Research, 2(1), 1–28.

Panigrahi, S., Sural, S., & Majumdar, A. K. (2009). Detection of intrusive activity in databases

by combining multiple evidences and belief update. In 2009 IEEE Symposium on

Computational Intelligence in Cyber Security (pp. 83–90). IEEE.

http://doi.org/10.1109/CICYBS.2009.4925094

Pathirage, C., Amaratunga, R. D., & Haigh, R. (2005). Knowledge management research within

the built environment: research methodological perspectives. In 5th international

postgraduate conference in the built and human environment. The Lowry, Salford

Quays,UK: University of Huddersfield. Retrieved from

http://eprints.hud.ac.uk/22699/%0AThe

Pierazzi, F., Casolari, S., Colajanni, M., & Marchetti, M. (2016). Exploratory security analytics

for anomaly detection. Computers & Security, 56, 28–49.

Pinsonneault, A., & Kraemer, K. L. (1993). Survey research methodology in management

information systems: An assessment (No. URB-022). UC Irvine: Center for Research on

Information Technology and Organizations. Retrieved from

http://escholarship.org/uc/item/6cs4s5f0

Pinto, A. (2014). Secure because math: A deep dive on machine learning based monitoring [PDF

document]. Retrieved from http://docs.huihoo.com/blackhat/usa-2014/us-14-Pinto-

Secure-Because-Math-A-Deep-Dive-On-Machine-Learning-Based-Monitoring-WP.pdf

Putnam, J. W., Spiegel, A. N., & Bruininks, R. H. (1995). Future directions in education and

inclusion of students with disabilities: A Delphi investigation. Exceptional Children,

61(6), 553–576.

Puri, C., & Dukatz, C. (2015). Analyzing and predicting security event anomalies: Lessons

learned from a large enterprise big data streaming analytics deployment. In 2015 26th

International Workshop on Database and Expert Systems Applications (DEXA) (pp. 152–

158). Valencia, Spain: IEEE. http://doi.org/10.1109/DEXA.2015.46

Qin, X., Wang, H., Li, F., Zhou, B., Cao, Y., Li, C., … Wang, S. (2012). Beyond simple

integration of RDBMS and MapReduce – Paving the way toward a unified system for big

data analytics: Vision and progress. Proceedings – 2nd International Conference on

Cloud and Green Computing and 2nd International Conference on Social Computing and

Its Applications, CGC/SCA 2012 (pp. 716–725). Washington, DC: IEEE.

doi:10.1109/CGC.2012.39

132

Raiyn, J. (2014). A survey of cyber attack detection strategies. International Journal of Security

and Its Applications, 8(1), 247–256.

Rao, J. R. (2016). Preface: Security intelligence. IBM Journal of Research and Development,

60(4), 1.

Razaq, A., Tianfield, H., & Barrie, P. (2016). A big data analytics based approach to anomaly

detection. In Proceedings of the 3rd IEEE/ACM International Conference on Big Data

Computing, Applications and Technologies - BDCAT ’16 (pp. 187–193). New York, NY:

ACM Press. http://doi.org/10.1145/3006299.3006317

Rosa, L., Alves, P., Cruz, T., Simoes, P., & Monteiro, E. (2015). A comparative study of

correlation engines for security event management. In Proceedings of the 10th

International Conference on Cyber Warfare and Security (p. 11). Kruger National Park,

South Africa: Acpil. Retrieved from citeulike-article-id:13744694

Roy, S., Ellis, C., Shiva, S., Dasgupta, D., Shandilya, V., & Wu, Q. (2010). A survey of game

theory as applied to network security. In 2010 43rd Hawaii International Conference on

System Sciences (pp. 1–10). Honolulu, HI: IEEE. http://doi.org/10.1109/HICSS.2010.35

Sadi, S., & Noordin, M. F. (2011). Developing and validating an instrument for measuring M-

commerce adoption: An exploratory analysis. International Journal of Network and

Mobile Technologies, 20(Sp.), 1–8.

Salih, A., Ma, X., & Peytchev, E. (2017). Implementation of hybrid artificial intelligence

technique to detect covert channels attack in new generation internet protocol IPv6.

In Leadership, Innovation and Entrepreneurship as Driving Forces of the Global

Economy (pp. 173–190). New York, NY: Springer International Publishing.

Sagiroglu, S., & Sinanc, D. (2013, May). Big data: A review. In Collaboration Technologies and

Systems (CTS), 2013 International Conference on (pp. 42–47). Washington, DC: IEEE.

Schales, D. L., Christodorescu, M., Rao, J. R., Sailer, R., Stoecklin, M. P., & Venema, W.

(2011). Stream computing for large-scale, multi-channel cyber threat analytics:

Architecture, implementation, deployment and lessons learned. IBM Research Report.

Retrieved from http://www.research.ibm.com/

Schinagl, S., Schoon, K., & Paans, R. (2015, January). A framework for designing a security

operations centre (SOC). In System sciences (HICSS), 2015 48th Hawaii International

Conference on (pp. 2253–2262). Washington, DC: IEEE.

Schrittwieser, S., Mulazzani, M., & Weippl, E. (2013). Ethics in security research which lines

should not be crossed? In 2013 IEEE Security and Privacy Workshops (pp. 1–4). San

Francisco, CA: IEEE. http://doi.org/10.1109/spw.2013.6914700

Shackleford, D. (2014). Analytics and intelligence survey 2014. A SANS survey. Retrieved from

http://www.sans.org

http://www.research.ibm.com/

133

Shackleford, D. (2016). SANS 2016 security analytics survey. A SANS Survey. Retrieved from

http://www.sans.org

Shafer, G. (1976). A mathematical theory of evidence (Vol. 1). Princeton, NJ: Princeton

University Press.

Singh, K., Guntuku, S. C., Thakur, A., & Hota, C. (2014). Big data analytics framework for peer-

to-peer botnet detection using random forests. Information Sciences, 278, 488–497.

Skulmoski, G., Hartman, F., & Krahn, J. (2007). The Delphi method for graduate research.

Journal of Information Technology Education: Research, 6(1), 1–21.

Spector, P. E., & Meier, L. L. (2014). Methodologies for the study of organizational behavior

processes: How to find your keys in the dark. Journal of Organizational Behavior, 35(8),

1109–1119.

Spitzner, L. (2003). Honeypots: Catching the insider threat. In Computer Security Applications

Conference, 2003. Proceedings. 19th Annual (pp. 170–179). Washington, DC: IEEE.

Štemberger, M. I., Manfreda, A., & Kovačič, A. (2011). Achieving top management support

with business knowledge and role of IT/IS personnel. International Journal of

Information Management, 31(5), 428–436.

Stroeh, K., Madeira, E. R. M., & Goldenstein, S. K. (2013). An approach to the correlation of

security events based on machine learning techniques. Journal of Internet Services and

Applications, 4(1), 7.

Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA:

Pearson Education.

Talabis, M., McPherson, R., Miyamoto, I., & Martin, J. (2014). Information security analytics:

Finding security insights, patterns and anomalies in big data. London, United Kingdom:

Syngress.

Taylor-Powell, E. (2002). Quick tips collecting group data: Delphi technique. Quick Tips #4

Program Data and Evaluation. Madison, WI: University of Wisconsin. Retrieved from

http://www.uwex.edu/ces/pdande/resources/index.html

TechAmerica Foundation: Federal Big Data Commission. (2012). Demystifying big data: A

practical guide to transforming the business of government. Washington, D.C.: Tech

America Foundation's Federal Big Data Commission. Retrieved from

https://breakinggov.com/documents/demystifying-big-data-a-practical-guide-to-

transforming-the-bus/

Thompson, M. A. (2013). Predicting threat potential using cyber sensors (Doctoral dissertation).

Available from ProQuest Dissertations & Theses Global. (Order No. 3573605)

134

Tripathi, S., Gupta, B., Almomani, A., Mishra, A., & Veluru, S. (2013). Hadoop based defense

solution to handle distributed denial of service (DDoS) attacks. Journal of Information

Security, 04(03), 150–164. doi:10.4236/jis.2013.43018

Turocy, T. L., & von Stengel, B. (2001). Game theory* (Report No. Rep. LSE-CDAM-2001-09).

Draft prepared for the Encyclopedia of Information systems. Department of Mathematics,

London School of Economics, London, UK.

Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27(11), 1134–

1142.

Leeuwen, J.V. (2004). Approaches in Machine Learning. In W. Verhaegh, E. Aarts, & J. Korst

(Eds.), Algorithms in ambient intelligence (pp. 151–166). New York, NY: Springer.

http://doi.org/10.1007/978-94-017-0703-9_8

Van de Moosdijk, J., Wagenaar, D., & Final, S. (2015). Addressing SIEM. Retrieved from

http://www.vurore.nl/images/vurore/downloads/scripties/2030-Def-scriptie-Jarno-van-de-

Moosdijk---daan-Wagenaar.pdf

Venkatesh, V., & Bala, H. (2008). Technology acceptance model 3 and a research agenda on

interventions. Decision Sciences, 39(2), 273–315. doi:10.1111/j.1540-5915.2008.00192.x

Vonhof, J. J. (2015). Middle management and telework adoption: Development of an instrument

using Delphi and exploratory factor analysis for the financial services industry (Doctoral

dissertation). Available from ProQuest Dissertations & Theses Global. (Order No.

3684229).

Von Neumann, J., & Morgenstern, O. (2007). Theory of games and economic behavior.

Princeton, NJ: Princeton University Press.

Wei, Z., Tang, H., Yu, F. R., Wang, M., & Mason, P. (2014). Trust establishment with data

fusion for secure routing in MANETs. In 2014 IEEE International Conference on

Communications (ICC) (pp. 671–676). Sydney, NSW, Australia: IEEE.

http://doi.org/10.1109/ICC.2014.6883396

Wheelus, C., Bou-Harb, E., & Zhu, X. (2016). Towards a big data architecture for facilitating

cyber threat intelligence. In 2016 8th IFIP International Conference on New

Technologies, Mobility and Security (pp. 1–15). Washington, DC: IEEE.

http://doi.org/10.1109/NTMS.2016.7792484

Williams, B., Onsman, A., & Brown, T. (2010). Exploratory factor analysis: A five-step guide

for novices. Journal of Emergency Primary Health Care, 8(3), 116–131. Retrieved from

http://www.tandfonline.com/doi/abs/10.1080/09585190701763982

Worrell, J. L., di Gangi, P. M., & Bush, A. A. (2013). Exploring the use of the Delphi method in

accounting information systems research. International Journal of Accounting

Information Systems, 14(3), 193–208.

135

Yam, C.-Y., Baldwin, A., Shiu, S., & Ioannidis, C. (2011). Migration to cloud as real option:

investment decision under uncertainty. In 2011 IEEE 10th International Conference on

Trust, Security and Privacy in Computing and Communications (pp. 940–949).

Washington, DC: IEEE. http://doi.org/10.1109/TrustCom.2011.130

Yen, T., Oprea, A., & Onarlioglu, K. (2013). Beehive: Large-scale log analysis for detecting

suspicious activity in enterprise networks. Proceedings of the 29th Annual Computer

Security Applications Conference (pp. 199–208). New York, NY: ACM.

doi:10.1145/2523649.2523670

Yong, A. G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory

factor analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79–94.

Yu, D., & Frincke, D. (2005). Alert confidence fusion in intrusion detection systems with

extended Dempster-Shafer theory. In Proceedings of the 43rd Annual Southeast Regional

Conference on - ACM-SE 43 (p. 142). New York, NY: ACM Press.

http://doi.org/10.1145/1167253.1167289

Zadrozny, P., & Kodali, R. (2013). Getting Data into Splunk. In Big Data Analytics Using

Splunk (pp. 9-30). Apress, Berkeley, CA.

Zomlot, L., Sundaramurthy, S. C., Luo, K., Ou, X., & Rajagopalan, S. R. (2011). Prioritizing

intrusion analysis using Dempster-Shafer theory. In Proceedings of the 4th ACM

Workshop on Security and Artificial Intelligence - AISec ’11 (p. 59). New York, NY:

ACM Press. http://doi.org/10.1145/2046684.2046694

136

STATEMENT OF ORIGINAL WORK

Academic Honesty Policy

Capella University’s Academic Honesty Policy (3.01.01) holds learners accountable for the

integrity of work they submit, which includes but is not limited to discussion postings,

assignments, comprehensive exams, and the dissertation or capstone project.

Established in the Policy are the expectations for original work, rationale for the policy,

definition of terms that pertain to academic honesty and original work, and disciplinary

consequences of academic dishonesty. Also stated in the Policy is the expectation that learners

will follow APA rules for citing another person’s ideas or works.

The following standards for original work and definition of plagiarism are discussed in the

Policy:

Learners are expected to be the sole authors of their work and to acknowledge the

authorship of others’ work through proper citation and reference. Use of another person’s

ideas, including another learner’s, without proper reference or citation constitutes

plagiarism and academic dishonesty and is prohibited conduct. (p. 1)

Plagiarism is one example of academic dishonesty. Plagiarism is presenting someone

else’s ideas or work as your own. Plagiarism also includes copying verbatim or

rephrasing ideas without properly acknowledging the source by author, date, and

publication medium. (p. 2)

Capella University’s Research Misconduct Policy (3.03.06) holds learners accountable for research

integrity. What constitutes research misconduct is discussed in the Policy:

Research misconduct includes but is not limited to falsification, fabrication, plagiarism,

misappropriation, or other practices that seriously deviate from those that are commonly

accepted within the academic community for proposing, conducting, or reviewing

research, or in reporting research results. (p. 1)

Learners failing to abide by these policies are subject to consequences, including but not limited to

dismissal or revocation of the degree.

http://www.capella.edu/assets/pdf/policies/academic_honesty.pdf

http://www.capella.edu/assets/pdf/policies/research_misconduct.pdf

137

Statement of Original Work and Signature

I have read, understood, and abided by Capella University’s Academic Honesty Policy (3.01.01)

and Research Misconduct Policy (3.03.06), including Policy Statements, Rationale, and

Definitions.

I attest that this dissertation or capstone project is my own work. Where I have used the ideas or

words of others, I have paraphrased, summarized, or used direct quotes following the guidelines

set forth in the APA Publication Manual.

Learner name

and date Sethuraman K Srinivas, 11 April 2018

http://www.capella.edu/assets/pdf/policies/academic_honesty.pdf

http://www.capella.edu/assets/pdf/policies/research_misconduct.pdf

138

APPENDIX A. RESEARCHER-DESIGNED SECURITY ANALYTICS SURVEY

This survey instrument may not be used without the expressed written consent of the author

Survey Instrument – Final list of 42 security analytic questions

Q7 A security analytic tool implemented in my organization has accelerated the detection and

prevention of cyber-attacks and data breaches

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q8 Implementation of a security analytic tool was able to uncover new categories of attacks /

anomalies that were previously undetected

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q9 Implementation of a security analytic tool immediately resulted in increased detection rate of

security incidents across the organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q10 Implementation of a security analytic tool increased awareness among security analysts

about suspicious network and system activities.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q11 Correlation of log source data feeding into the security analytic tool has significantly

enhanced the incident detecting capabilities.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

139

Q13 Tuning of rules in the security analytic tool is carried out on a regular basis (and also on as-

needed basis) to enhance detection of threats and attacks

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q14 Categories of log sources feeding into the security analytic tool cover all business units of

my organization. Example of categories are network logs, endpoint logs, critical application logs,

cloud related logs and referential data.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q15 Log sources feeding into the security analytics tool covers all locations of my organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q16 Out of box correlation rules provided by the security analytic tool were effective at

addressing potential attack scenarios

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q17 The security analytic tool provided real-time analytical capabilities that improved the

insights of Security Operational Center (SOC) security analysts within one year of its

implementation.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

140

Q18 Owners and users of the security analytic tool were able to fully understand and use the tool

within one year of its implementation

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q19 Both the technical and functional users of the security analytic tool were provided the

needed training on the tool to perform their functions using the tool

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q20 Tuning of the security analytic tool (including alert correlation) was done on a regular basis

to reduce the number of false positives.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q22 Implementation of the security analytic tool has resulted in more positive compliance

assessment results

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q23 The graphical user interface (GUI) aspect of the security analytic tool is user friendly for the

tool administrators

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q24 Use cases are periodically reviewed and updated based on changes to the threat landscape

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

141

Q25 Business process owners contribute to use cases for the security analytic tool in my

organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q26 In my opinion, the security analytic tool is faster at detecting breaches or attacks and raising

alerts compared to earlier processes used by my organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q27 In my opinion, implementation of the security analytic tool has improved the security health

of internal networks in my organization.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q28 Implementation of the security analytic tool has positively impacted the incident response

process within my organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q29 Metrics and reports from the security analytic tool provided sufficient actionable insights to

my organization’s CISO / CIO team

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

142

Q30 The security analytic tool brought about measurably improved results (within 6 months) for

the analysts in charge of security operation center (SOC)

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q32 The security analytic tool is able to leverage inputs from intrusion detection and prevention

systems (IDPS log sources) systems, in protecting my IT assets

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q33 Actionable intelligence to be followed up on by security analyst(s) was directly generated

by the security analytic tool within 6 months after its implementation.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q34 Identification and monitoring of critical log sources feeding into the security analytic tool

occurs on a regular basis

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q36 The security analytics tool implemented in my organization is able to scale to meet the

growing demands of my organization.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

143

Q37 In my opinion, the security analytic tool improved the overall security posture of my

organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q38 Reporting capabilities available inside the security analytic tool are sufficient to fulfill the

IT governance requirements of my organization

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q39 Root cause analysis is performed on high severity incidents created through alerts in the

security analytic tool.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q40 After implementing the security analytics tool, additional security operation center (SOC)

analysts were required to process the increased number of new offenses or tickets.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q43 The security analytic tool implemented in my organization is able to pro-actively discover

network and application level vulnerabilities.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

144

Q44 An increase in detection of authentication related security incidents was observed within 1

year of implementing the security analytic tool.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q45 An increase in detection of denial of service (DoS & DDoS) incidents was observed within

6 months of implementing the security analytic tool

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Some organizations employ big-data analytics to supplement and enhance a security information

and event management (SIEM) tool or an equivalent. Some organizations use big data tools as a

mainstream security analytics tool. Big data in this context refers to employing one or more

products like Hadoop, MapReduce, Pig, Mahout, IBM Infosphere BigInsights or equivalent to

enhance security analytic capabilities. Big data related questions are grouped together to help the

participant to focus on the context of big data.

Q48 A big data platform in my organization supplements existing security tools and products

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q49 A Big data security analytic platform/tool in my organization provided deeper insights into

anomalous network situations.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q50 In my opinion, big data security analytic platform/tool in my organization benefited from

past data experience which reduced the time to detect potential attack situations.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

145

Q52 A big data security analytic platform/tool provides users with flexibility to interact and

query from large volumes of security data.

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q53 A Big data security analytic platform/tool was able to successfully correlate structured and

unstructured data to enhance incident detection rate

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q54 Big data security analytic process in my organization was able to reduce large data-sets into

smaller ones thereby helping incident response team to better focus on incident investigation

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q55 Big data analytic functions were able to enhance the knowledge of security analysts by

simulating security incidents. (for example: malware infection)

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q56 Big data predictive functions were able to predict potential compromises

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)

Q57 Anomalous insider behavior across the organization was easily identified by big data based

security analytic tool

 Strongly agree (1)  Agree (2)  Neutral (3)  Disagree (4)  Strongly disagree (5)