Need Help For Below Research In The APA Format

1.pdf

Home >Information Systems homework help >Need Help For Below Research In The APA Format

A METHODOLODY TOWARD SECURITY EFFECTIVENESS FOR CRITICAL INFRASTRUCTURE AND DEPENDENT RESOURCES

by Latechia White

B.S. in Electrical Engineering, December 1995, Howard University M.S. in Electrical Engineering, May 2006, The George Washington University

A Praxis submitted to

The Faculty of The School of Engineering and Applied Science

of The George Washington University in partial satisfaction of the requirements for the degree of Doctor of Engineering

January 19, 2018

Praxis directed by

Timothy J. Eveleigh Professorial Lecturer of Engineering Management and Systems Engineering

Bereket Tanju

Professorial Lecturer of Engineering Management and Systems Engineering

ProQuest Number:

INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed,

a note will indicate the deletion.

ProQuest

Published by ProQuest LLC ( ). Copyright of the Dissertation is held by the Author.

Microform Edition © ProQuest LLC.

ProQuest LLC. 789 East Eisenhower Parkway

P.O. Box 1346 Ann Arbor, MI 48106 - 1346

10743940

2018

The School of Engineering and Applied Science of The George Washington University certifies

that Latechia White has passed the Final Examination for the degree of Doctor of Engineering as

of January 3, 2018. This is the final and approved form of the Praxis.

A METHODOLODY TOWARD SECURITY EFFECTIVENESS FOR CRITICAL INFRASTRUCTURE AND DEPENDENT RESOURCES

Latechia White

Praxis Research Committee:

Timothy J. Eveleigh, Professorial Lecturer of Engineering Management and Systems Engineering, Praxis Co-Director Bereket Tanju, Professorial Lecturer of Engineering Management and Systems Engineering, Praxis Co-Director E. Lile Murphree, Professor Emeritus of Engineering Management and Systems Engineering, Committee Member Amir Etemadi, Assistant Professor of Engineering and Applied Science, Committee Member Barry C. Ezell, Chief Scientist and Research Associate Professor, Committee Member

iii

DEDICATION

This research is dedicated to Critical Infrastructure (CI) owner/operators and national security and

medical professionals seeking to do more to protect and maintain our nation’s CIs and dependent

resources in an effort to preserve public health and demonstrate emergency preparedness.

ACKNOWLEDGMENTS

“If ye have faith as a grain of a mustard seed, ye shall say unto this mountain, Remove hence to

yonder place; and it shall remove; and nothing shall be impossible unto you.”

Matthew 17:20

To my Lord and Savior, because of You I was able. I would like to thank Dr. Timothy

Eveleigh, Dr. Thomas Holzer, and Dr. Bereket Tanju, my advisors at The George Washington

University (GWU), who guided me during my research journey. Their expert feedback and

suggestions helped me achieve one of my life dreams. I am also particularly grateful to Dr.

Shahram Sarkani and Dr. Thomas Mazzuchi, Professors of Engineering Management and

Systems Engineering at the GWU, for establishing such a challenging yet flexible program for full-

time working professionals.

I am thankful to my core group of friends and family who I affectionately call my Prayer

Warriors, whose support and encouragement have been crucial to my motivation and survival in

this program. To my employer, I am grateful for the flexible hours that allowed me to balance my

professional career and academic goals. Additionally, I would like to thank those that took the time

to review my research and provided valuable, constructive feedback. Special thanks to those that

took the time to complete the survey that provided insightful data to my research. Your feedback

will hopefully result in a more secure and resilient world whose reality has become filled with

relentless cyber and physical attacks.

To my husband, lover, and friend, who claims he suffered from my absence during my

journey, who had to fend for himself for breakfast, lunch, and dinner, who tolerated and accepted

my absence and neglect of my domestic responsibilities, who smiled when I offered peanut butter

and jelly as a dinner substitute – thank you for your prayers, patience, enduring love, and support.

Without you this dream would have not been possible. Forever grateful!

vii

ABSTRACT

A Methodolody Toward Security Effectiveness for Critical Infrastructure and Dependent Resources

A successful Denial of Service (DoS) attack on a Critical Infrastructure (CI) can indirectly have

devastating and irreversible effects to those that depend on its services. The mere possibility that

physical destruction or loss of human life can result (indirectly) from a successful attack on a CI

gives reason to re-assess the effectiveness of security measures in place to protect and provide

resiliency. Although existing literature describe numerous approaches to CI interdependency

analyses, it does not sufficiently identify or address a method to dynamically (through scenario

analysis) and proactively evaluate and quantify the relative effectiveness of implemented and/or

proposed security measures against multi-order cascading effects given a CI disruption.

To address the persistent challenge of protecting CIs and maintaining the essential services

that they provide, a method to evaluate security effectiveness with an operational framework is

offered to assist proactive, scenario-based interdependency analysis of CI Protection and

Resiliency (CIP/R). This methodology is provided for CI owners and stakeholders to evaluate their

posture and ultimately make provisions for a more proactive response before potential disaster.

The Bayesian Approach to Security Effectiveness through metrics, modeling and decision-making

(BASE m2d) conceptual framework was developed by this research to address this pervasive

problem. Specifically, this research illustrates the framework by examining multi-order effects on

hospital operations, thereby assessing the likelihood of impact to a patient’s health given a

successful cyber or physical (natural or man-made) attack on a dependent CI.

A survey was provided to medical professionals at 10 different hospitals to help identify current

risk management processes used by the medical professionals to understand, assess, and validate

patient impact given DoS to a dependent CI (Power, Water, and Communications). The

viii

probabilistic Bayesian module allowed for a scenario-based, what-if impact analysis, given limited

available data. This research revealed, despite the known dependence on CIs, no standardized

metrics or processes are used to assess patient impact for risk mitigation given a DoS. Also noted

was a lack of general preparedness, training, and methods of sharing information in the event of a

DoS on a dependent CI. Consequently, the findings of this research resulted in a hybrid,

hierarchical, multi-dimensional approach grounded in systems engineering principles.

TABLE OF CONTENTS

DEDICATION ....................................................................................................................................... IV

ACKNOWLEDGMENTS ........................................................................................................................... V

ABSTRACT ........................................................................................................................................ VII

TABLE OF CONTENTS.......................................................................................................................... IX

LIST OF FIGURES ............................................................................................................................... XII

LIST OF TABLES ............................................................................................................................... XIII

LIST OF ACRONYMS .......................................................................................................................... XIV

CHAPTER 1 - INTRODUCTION ................................................................................................................ 1

1.1 Research Background ....................................................................................................... 5

1.1.1 Overview of CI ............................................................................................................... 5

1.1.2 Healthcare and Public Health ........................................................................................ 7

1.2 Motivation ........................................................................................................................ 10

1.3 Research Problem .......................................................................................................... 11

1.4 Research Objectives ....................................................................................................... 12

1.5 Scope and Limitations ..................................................................................................... 13

1.6 Research Contribution .................................................................................................... 14

1.7 Significance/Implications of Research ............................................................................. 15

1.8 Definitions of Key Concepts ............................................................................................ 17

1.8.1 Security Effectiveness for the Operational Environment ............................................. 17

1.8.2 Hierarchical Holographic Modeling (HHM) .................................................................. 18

1.8.3 Bayesian Belief Network (BBN) ................................................................................... 20

1.8.4 Systems Security Effectiveness Index (SSEI) ............................................................. 21

1.9 Organization and Outline................................................................................................. 22

CHAPTER 2 - LITERATURE REVIEW ..................................................................................................... 24

2.1 Review of Related Works ................................................................................................ 24

2.2 Overview of CIP/R Analysis ............................................................................................ 25

2.3 Approaches to CI Protection & Resiliency (CIP/R) .......................................................... 28

2.3.1 Implementations of Decision Analysis Tools for CIP/R ................................................ 31

2.3.2 Implementations of Probabilistic Risk Modeling for CIP/R ........................................... 32

2.4 Approaches to Measuring Security Effectiveness for CI.................................................. 34

2.5 Summary ......................................................................................................................... 34

CHAPTER 3 - RESEARCH METHODOLOGY ............................................................................................ 35

3.1 Research Design ............................................................................................................. 36

3.2 Expert Elicitation ............................................................................................................. 37

3.3 Survey Scale ................................................................................................................... 40

3.4 Expert Elicitation Calibration ........................................................................................... 40

3.5 Survey Instrument ........................................................................................................... 41

3.6 Validity of Survey Instrument ........................................................................................... 43

3.7 Validity of Conceptual Framework ................................................................................... 44

3.8 Data Collection ................................................................................................................ 45

CHAPTER 4 – HOSPITAL CASE STUDY ................................................................................................. 46

4.1 Case Study Background.................................................................................................. 46

4.2 Case Study Application ................................................................................................... 47

4.2.1 Step 1 and 2: Define Operational Environment and Security Goals ........................... 49

4.2.2 Step 3: Identify Dependencies ................................................................................... 50

4.2.3 Step 4: Assess/Measure the Security Posture ........................................................... 50

4.2.4 Step 5: Assess multiple dimensions/perspectives [CI-HHM] ...................................... 53

4.2.5 Step 6: Assess Strength/Weakness [Calculate the SSEI] .......................................... 56

4.2.6 Step 7: Assess Impact Likelihood [Construct the BBN] .............................................. 59

4.2.7 Steps 8-10: Decision Analysis .................................................................................... 65

CHAPTER 5 - DATA ANALYSIS AND RESULTS ....................................................................................... 65

5.1 Analysis Objectives ......................................................................................................... 65

5.2 Demographic Data .......................................................................................................... 66

5.3 Metrics ............................................................................................................................ 68

5.4 Descriptive Data .............................................................................................................. 70

5.5 Reliability and Validity of Survey Instrument ................................................................... 72

5.6 Validity of Conceptual Framework ................................................................................... 74

5.7 Threats to Internal Validity............................................................................................... 75

5.8 Threats to External Validity ............................................................................................. 75

CHAPTER 6 – CONCLUSION ................................................................................................................ 76

6.1 Conclusion with Respect to Study Hypotheses ............................................................... 77

6.2 Conclusion with Respect to Study Questions .................................................................. 77

6.3 Discussion ....................................................................................................................... 78

CHAPTER 7 - FUTURE RESEARCH ....................................................................................................... 81

REFERENCES .................................................................................................................................... 83

APPENDIX A ...................................................................................................................................... 96

APPENDIX B .................................................................................................................................... 102

xii

LIST OF FIGURES

Figure 1. CI Interdependency Multi-order Effects ............................................................................ 9

Figure 2. HHM for Critical Infrastructure or Dependent Resources (CI/DR-HHM) ..........................19

Figure 3. Research Focus Areas ....................................................................................................27

Figure 4. General Flow of BASE m2d Framework .........................................................................48

Figure 5. Simplified BBN ................................................................................................................60

Figure 6. Representation of BASE m2d Model of DoS attack on CIs (w/o SSEI) ...........................62

Figure 7. Representation of SSEI Analysis given DoS Attack on Power CI ...................................63

Figure 8. Survey: Years of Experience in Medical Field ................................................................67

Figure 9. Survey: Medical Field Profession ...................................................................................67

Figure 10. Survey: Experience in Intensive Care Unit (ICU) ..........................................................67

xiii

LIST OF TABLES

Table 1. Healthcare and Public Health Sector CI Dependency (DHS.gov) ..................................... 8

Table 2. Literature Review CI Model Comparative Analysis ...........................................................30

Table 3. Reference Metrics for Critical Infrastructure/Dependent Resource Protection .................52

Table 4. Reference Metrics for Critical Infrastructure/Dependent Resource Resilience .................53

Table 5. CI/DR-HHM SSEI Scoring Scale (by Category) ...............................................................55

Table 6. CI/DR-HHM SSEI Scoring Scale (Total) ...........................................................................56

Table 7. Area of Improvement/Deficiency (Calculated SSEI) .........................................................57

Table 8. Exemplar Stakeholder Question Categories ....................................................................58

xiv

LIST OF ACRONYMS

AHP Analytical Hierarchy Process

BASE m2d Bayesian Approach to Security Effectiveness with metrics, modeling, and decision-

support

BBN Bayesian Belief Network

CI Critical Infrastructure

CIP/R Critical Infrastructure Protection and Resiliency

CPT Conditional Probability Table

DA Decision Analysis

DAG Directed Acyclic Graph

DHS Department of Homeland Security

DOS Denial of Service

DR Dependent Resource

EO Executive Order

GAO Government Accountability Office

HHM Hierarchical Holographic Model

ICU Intensive Care Unit

MAUT Multi-Attribute UtilityTheory

NIPP National Infrastructure Protection Plan

PDD Presidential Decision Directive

PPD Presidential Policy Directive

SCADA Supervisory Control and Data Acquisition

SSEI Systems Security Effectiveness Index

SSP Sector Specific Plan

CHAPTER 1 - INTRODUCTION

The definition of security is the state of being free from danger or threat. While this defined state of

being can never be achieved within the cyber domain, the knowledge of one’s security posture (as

it relates to exposure to danger or threat) is paramount to ultimately understanding cyber and/or

physical security and implementing appropriate, timely, and necessary security measures. To that

end, there has been an avalanche of standards, tools, and compliance guidance added to the

cadre of weapons for cyber-warfare; inevitably developed to afford a perceived sense of security

that the devices, networks, systems, and enterprises are secure. Still, organizations’ relative

security posture (i.e. assessed security effectiveness per defined security goals) remains unknown

despite Information Security Professionals having implemented the suggested/recommended

security configurations, tools, policies, and plans in an effort to defend against various known

threats. The risks of misinterpreting one’s security posture can have devastating consequences in

terms of misallocation of resources, loss of revenue, loss of competitive advantage, loss of privacy,

loss of trust, damage to reputation, destruction of property, and in some cases, potential loss of life.

These consequences are amplified when they are associated with the elements of the nation’s

Critical Infrastructures (CIs).

CIs, as defined by the Department of Homeland Security (DHS), are the assets, systems, and

networks, whether physical or virtual, so vital to the United States that their incapacitation or

destruction would have a debilitating effect on security, national economic security, national public

health or safety, or any combination thereof [DHS.gov 2013]. CIs are large, complex, adaptive,

and highly interconnected; they are ripe with potential areas where system knowledge can easily

go unexamined and thus undiscovered. Implementing security measures without considering the

multi-dimensional aspect of interdependencies that are both internal and external to CIs can lead to

undesirable consequences. This can ultimately imperil efforts to achieve CI protection goals.

Moreover, goals can be negatively impacted by providing an illusion of acceptable security where

there are actually areas of unacceptable and intolerable risk - in spite of operators having

employed every available tool, policy, and standard. Within the multi-dimensional regions of CIs

there are cyber-based, temporal, geo-spatial, legislative, societal, economic, and stakeholder

attributes that can be explored and considered to uncover unknown insight into a CI and/or

dependent resource’s true security posture.

According to Government Accountability Office (GAO) reports, over the past decade there

has been a sustained increase in malicious penetrations to government information systems [ICS-

CERT 2017, DHS 2016, GAO-15-573T 2015, GAO-12-666T 2012]. In 2012 alone, America's

power, water, financial institutions, nuclear systems, and other key resources have experienced a

52 percent increase in targeted attacks by cyber criminals seeking to gain (or deny) access to the

nation's CI [CERT 2013]. Parties aligned with the Russian government have developed a

cyberweapon (CrashOverride) specifically designed to destroy industrial control systems of CIs.

Considered Stuxnet 2.0, CrashOverride hackers briefly shut down one-fifth of the electric power

generated in Kiev (washingtonpost.com 2017). Recent ransomware cyber-attacks such as

WannaCry and Petya have affected more than 200,000 computers, causing chaos and disruption

for critical infrastructure and dependent resources (CI/DR), including major hospitals, a nuclear

disaster site, and electrical grids (CNET 2017). It was noted by Healthcare Informatics: “this attack

(WannaCry) shows that interconnected devices and systems are vulnerable to attack by nations,

non-state actors, and just plain crooks. An attack of this scope points to the potential for an entirely

different type of damage: shutting down entire businesses, hospital systems, banks, and critical

infrastructure” (2017). Cyber criminals are demonstrating their ability to access sensitive

information, disturb the integrity of personal data, and block the availability of information systems.

However, their ability to manifest the physical destruction traditionally associated with kinetic

warfare has yet to be fully realized. Dually noted by the GAO and U.S. adversaries, the nation’s

CIs and Key Resources remain vulnerable to the potential devastation of cyber-attacks. As

indicated in the Presidential Policy Directive 21 (PPD 21), U.S. CIs remain a high value target

[GAO-18-62, 2017]. According to these recent reports (EO, PPD, and GAO), implementations of

existing standards, policy, legislation, methodology, and tools have not provided sufficient

confidence, guidance, or rigor toward the effective protection against these increasingly frequent

and potentially destructive attacks [GAO-13-462T, 2013; GAO-17-518T, 2017].

As noted by these reports and the adversary’s ability to continually penetrate CIs, it is

critical not only to have the ability to assess the effectiveness of various security measures, but

also to have a method that cogitates (considers/includes) the complexity and interdependencies of

CIs. This research offers a case study to demonstrate such a method - the Bayesian Approach to

Security Effectiveness with metrics, modeling, and decision-support (BASE m2d) conceptual

framework. This comprehensive method is used to assess a CI’s security posture by evaluating

the effectiveness of not only implemented measures but proposed measures, as well; potentially

avoiding devastating (unintended or intended) consequences. As a result, the risk of not protecting

CIs or providing the resiliency required to maintain services to dependent key resources such as

hospitals is explored.

Specifically, this study examines the likelihood of impact to a patient’s health given a

successful cyber or physical (natural or man-made) Denial of Service (DoS) attack on a dependent

CI using the BASE m2d framework (construct).

Understanding the potential impacts to elements of the Healthcare and Public Heath (HPH)

sector given a successful Denial of Service (DoS) to CIs is imperative to ensure proper security

effectiveness measures are in place to avoid potentially irreversible consequences. Further, a

strategic, holistic construct is proposed to properly focus budget constrained resources based on

the assessment of potential weaknesses of CIs as defined per their security goals or protection

targets (patients, personnel, medical devices, etc.).

The objective of this study is to provide emergency management personnel a conceptual

framework and methodology to evaluate security effectiveness and effects of cascading risk that

may result from inadequate security measures. This research is presented as a proof of concept

using a combination of real and notional data. Elicitation of specific data from medical experts was

limited due to the acknowledged vulnerability of hospitals and the potential insight it may provide to

adversaries. This research revealed the troubling finding that of the 13 hospitals pursued for expert

elicitation, medical professionals from 10 distinct hospitals noted their current risk assessment

plans did not include CI interdependency patient impact metrics or analysis given a denial of

service on a dependent CI. Consequently, the disparate measures hospital engineers, IT

professionals, and physicians use today to protect and maintain services they provide for the

ultimate purpose of preserving life are discussed. Further, this framework is being offered as a

method that considers those distinct security measures and provides a holistic approach to ensure

better protection and resiliency toward patient care. The BASE m2d framework was validated with

data provided by medical professionals from 10 different hospitals.

1.1 RESEARCH BACKGROUND

1.1.1 OVERVIEW OF CI

As a result of devastating events (Oklahoma bombing, Katrina, 9-11, Stuxnet, etc.) and the nation’s

lack of preparedness, congressional attention has resulted in the government, academia, and

private industry taking action. The dedicated focus on Critical Infrastructure Protection (CIP) spans

across four presidential administrations. In 1996, President Clinton established the President’s

Commission on Critical Infrastructure Protection (PCCIP) (E.O. 13010). Although no immediate

threats to critical infrastructures were noted upon the release of the PCCIP in 1997, it did stress the

importance of CI interdependencies. In 1998 the Presidential Decision Directive, number 63 (PDD-

63) was released. This directive sought to protect, by the year 2003, the nation’s CIs from

deliberate attacks. PDD-63 was later updated by President Bush through the Homeland Security

Presidential Directive HSPD-7 to establish a national policy for Federal departments and agencies

to identify and prioritize United States Critical Infrastructure and Key Resources and to protect

them from terrorist attacks (HSPD-7, 2003). The E.O. 13130 established by President Clinton in

1999 and E.O. 13231 by President Bush in 2001, essentially instituted Information Sharing and

Analysis Centers (mostly facilitated by the private-sector) and a National Infrastructure Advisory

Council (NIAC). In 2002, the Department of Homeland Security (DHS) was established and

charged with the primary responsibilities of protecting the United States and its territories from -

and responding to - terrorist attacks, man-made accidents, and natural disasters (DHS.gov). In

2013, President Obama issued E.O. 13636 and PPD-21. These two directives aim to “enhance the

security and resilience of the nation’s critical infrastructure and to maintain a cyber environment

that encourages efficiency, innovation, and economic prosperity while promoting safety, security,

business confidentiality, privacy, and civil liberties.” (E.O. 13636, PPD-21) Most recently (2017),

http://en.wikipedia.org/wiki/Unorganized_territory

http://en.wikipedia.org/wiki/Terrorism

http://en.wikipedia.org/wiki/Accident

http://en.wikipedia.org/wiki/Natural_disaster

President Trump released an Executive Order on Strengthening the Cybersecurity of Federal

Networks and Critical Infrastructure. Together, these policies are intended to achieve the following

(DHS.gov):

• Encourage the adoption of effective measures across all critical infrastructure sectors to

improve security and resiliency and reduce risk from cyber-attacks to essential functions

and services by publishing a Cybersecurity Framework (the Framework, 2017) that will

provide owners and operators with a prioritized, flexible, repeatable, performance-based,

and cost-effective set of validated security controls based upon industry best practices.

• Enhance timely, relevant, and accurate information sharing on significant risks by

implementing a program to develop and rapidly share unclassified information with

critical infrastructure owners and operators, enabling the adoption of effective mitigations

to prevent or to reduce the consequences of significant incidents.

• Align responsibilities of public and private partners to efficiently allocate risk reduction

responsibilities by conducting an analysis of the existing critical infrastructure public-

private partnership model and recommending options for improving the effectiveness of

the partnership in managing both the physical and cyber risks.

• Promote innovation in novel risk-reduction solutions by developing a National Critical

Infrastructure Security and Resilience Research and Development (R&D) Plan to identify

priorities and guide R&D requirements and investments toward those solutions that will

help assure the provision of essential functions and services over time.

• Ensure that privacy, civil rights, and civil liberties are protected as a foundational part of

all risk management efforts by conducting an assessment of the privacy, civil rights, and

civil liberties implications of all EO 13636 and PPD-21 programs and recommending

revisions to proposed initiatives as required.

The Presidential Directive 21 (PPD 21) identified 16 CI sectors and designated specific federal-

agencies to facilitate/head protection and resiliency programs and activities. The sectors are

identified as follows:

1. Agriculture and Food 2. Banking and Finance 3. Chemical 4. Commercial Facilities 5. Communications 6. Critical Manufacturing 7. Dams 8. Defense Industrial Base 9. Emergency Services 10. Energy 11. Government Facilities 12. Healthcare and Public Health 13. Information Technology 14. Nuclear Reactors 15. Transportation Systems 16. Water and Sewage

It is commonly acknowledged that the interdependencies between critical infrastructure sectors

are of great importance to the protection of the nation.

1.1.2 HEALTHCARE AND PUBLIC HEALTH

Healthcare and public health facilities rely on various CIs in order to maintain daily operations. A

cyber or physical attack on any of those interdependent CIs can indirectly have a detrimental

impact on a patient’s health or their personal data. Depending on the motivation of the attacker,

i.e., to disturb the confidentiality, availability, or integrity of the hospital service or patient’s data, the

impact can be irreversible. Table 1 describes the critical dependencies between various sectors to

HPH. For examples, hospitals rely on critical services such as reliable power, clean water supply,

and available communications. An illustration of CI interdependency includes the following: Water

and Communication CIs depend on Energy (power) CIs, while Emergency Service (i.e.,

ambulances) CIs rely on Communication CIs. A DoS of critical services to a hospital from either a

cyber or physical attack can have a cascading effect if the proper protection and resiliency security

measures are not implemented.

Table 1. Healthcare and Public Health Sector CI Dependency (DHS.gov)

Understanding how and where to properly address and allocate security measures in a

budget-constrained environment will prove invaluable to ensure these critical resources/services

are uninterrupted (or have minimal or acceptable impact). A case is modeled in this research

where a patient in ICU is critically ill and depends on dialysis for survival. Given a DoS on a Water

CI, the hospital could be subsequently impacted if the proper back-up resources are not engaged

or available in a timely manner. The calculated System Security Effectiveness Index (SSEI)

defined in this study uses Systems Engineering/Systems Thinking concepts as a foundation to

inform the decision-maker of deficiencies internal and/or external to the infrastructure, enabling a

holistic, proactive assessment of their security effectiveness (of current or proposed security

measures) before an actual attack occurs.

A CI failure can be due to a successful attack, whether virtual or physical, that results in a

service interruption (partial shutdown or complete shutdown), ultimately having a 2nd, 3rd or 4th

order effect [Figure 1].

Figure 1. CI Interdependency Multi-order Effects

The National Association of County and City Health Officials (NACCHO), in summary, have

identified four categories of cyber-attack impact on healthcare and public health facilities

(NACCHO 2014):

1. Loss of integrity: Patients and practitioners may lose confidence in a healthcare

provider’s ability to maintain patient privacy due to perceptions of inadequate security.

2. Loss of availability: Cyber threats to data and operational systems can: take a facility off-

line, leading to disruption of care; create loss of access to health records, limiting the

provider’s ability to provide appropriate care, shelter, and medicine; disrupt emergency

telephone lines and EMS systems; and slow or disable emergency medical response

systems. Cyber-attacks can also prevent or impact production and manufacturing

of medical equipment or drugs.

3. Loss of confidentiality: The exposure of personal data can trigger ripple effects for

victims of cyber-crime, including theft or loss of a patient’s private information or discovery

of patient information on personal medical devices.

4. Physical destruction of systems: Cyber-attacks could damage physical systems used to

perform functions, such as regulate utilities critical to healthcare and public health and

could shut down or slow supply chains, impair patient care, and impede emergency

response, potentially leading to significant loss of life.

Public trust depends upon the sustainability, resilience, integrity, and availability of national HPH

critical infrastructure [NACCHO 2014]. Research results from this study reveal that hospital

engineers have performed due diligence to ensure if power is disrupted or water supplies have

been tainted or halted, backup generators and alternate water supplies are available to maintain

critical services. However, this research further revealed that most hospitals have not performed

additional risk assessments to understand or evaluate the potential impact of a DoS (i.e., water,

power) to the patient.

1.2 MOTIVATION

This study is motivated by the need to do more to protect and maintain the nation’s CIs and the

services they provide. In light of the mere possibility that physical destruction or loss of human life

can result (directly and/or indirectly) from a successful attack on a CI, is cause enough to pause

and re-assess the effectiveness of security measures in place to protect and provide resiliency.

1.3 RESEARCH PROBLEM

The indirect consequence of a DoS on a CI can be devastating to those that depend on its

services. The need to do more to protect and maintain the nation’s CIs and the services they

provide is paramount considering the dire consequences if neglected. Residential communities,

hospitals, banking, and government services are examples of resources that require sustained and

reliable provisions from CIs such as Energy, Water, and Communications for mere survival. Thus,

the ability to protect and/or maintain these critical services to dependent resources is crucial. As

CIs strive to institute protective and resiliency measures, the ability to assess the effectiveness of

those measures becomes more important not only to the CI, but also to the dependent resources.

This research explores the growing concern and significance of understanding potential

impacts of indirect consequences to dependent resources given a Denial of Service (DoS) to CIs.

To address the persistent challenge of protecting CIs and maintaining the essential services they

deliver, this research reveals that a methodology is needed to provide CI owners/stakeholders a

tool to evaluate their security posture while ultimately allowing for proactive provisioning before

potential disaster.

Current methodologies used to address the complex problem of improving cyber/physical

security protection of the enterprise, or specifically in this case, the nation’s CIs, must expand

beyond existing traditional approaches. Most security methods used today are considered from a

single-dimension. This is normally accomplished by protecting virtual or physical access to

architectural elements or components (network of routers, switches, servers, SCADA) from cyber-

attacks, e.g., by configuring firewalls, implementing policies, limiting access to data servers, and

training users. Although these techniques demonstrate a noble effort, they have proved to be

neither sufficient nor effective. Further, organizations, CI owners, and operators must have a

method to assess the effectiveness of the security program they put in place. A serious gap exists

in the tools available to assess the effectiveness of security measures which are designed to

mitigate disruptions to essential CI services (NIST 2014, GAO-16-152, GAO-17-518T 2017). Also

lacking are strategic methods to evaluate the subsequent impacts resulting from interruptions to

those services. Furthermore, a framework that empowers emergency management personnel to

reduce negative impacts of a CI DoS by strategically improving implemented security measures

does not readily exist today (NACCHO 2017). Existing literature describe numerous approaches to

CI interdependency analyses [Zio, Ouyang, Eusgeld, Haimes, DiMase, Borum, Kozik, Sikula,

Rinaldi, DiGiorgio]. However, there is limited research on estimating the likelihood of negative

impacts from those interdependencies or understanding the effectiveness of security measures

designed for their protection. This research fills a gap by providing a framework that allows

emergency management personnel to estimate the likelihood of impacts using a construct that

dynamically (through scenario analysis) and proactively addresses and evaluates the relative

effectiveness of implemented and/or proposed security measures designed to help minimize or

negate undesirable effects of CI service disruptions. We expand on knowledge, experience, and

recommendations offered by previously documented research of noted scholars.

1.4 RESEARCH OBJECTIVES

The objective of this research is to develop a solution to address the stated research problem.

Thus, a conceptual model that comprises a systematic, comprehensive (quantitative/quantitative),

scenario-based tool to proactively assess the effectiveness of implemented or proposed security

measures is offered. Specifically, the objectives are outlined as follows:

1) develop a strategic methodology to assess the effectiveness of security implementations

and aid in decision-making with a goal of proactively preparing for the inevitable occurrence of a CI

service interruption/disruption.

2) model how successful penetrations of CI vulnerabilities can have life-threatening

implications as an indirect or direct result of CI service disruptions;

3) identify exemplar metrics that could effectively be used to warn, prevent, or absorb CI

service interruptions.

Research Questions and Hypotheses

The following research questions are related to the problem statement and noted hypotheses.

Question #1: How can the assessment of security effectiveness of CI interdependencies and

vulnerabilities be modeled proactively (before occurrence or penetration) for improved

decision-making?

Question #2: How can combining probabilistic reasoning and a holistic, systems-thinking

based framework facilitate the assessment of relative effectiveness of CI and dependent

resources security measures?

Question #3: What metrics are used by hospitals to trigger auxiliary systems in the event of a

shutdown (partial or complete)? What metrics, if any are used to warn, prevent, or absorb CI

service interruptions?

Hypothesis 1a: Probabilistic reasoning and a systems-thinking based framework can be combined to assist in the overall evaluation (strength or weakness) of CI and/or dependent resources security effectiveness. Hypothesis 1b: Probabilistic reasoning and a systems-thinking based framework can be combined to quantifiably evaluate security effectiveness of Protection and Resiliency (P/R) for CIs and their dependent elements.

1.5 SCOPE AND LIMITATIONS

This research forms the basis and foundation for future research. The value of this research will

obtain its greatest return with the continuance of this research through studies in multi-order

dependency analysis of CI impacts towards the goal of improved security (protection and

resilience) effectiveness.

The bounds/limitations identified for this research include the examination of CIs as a black

box. It is important to note that the same CI/DR-HHM analysis can be performed looking internal to

the CI to assess the sub-components and the effects of any CI external influence it may cause.

While excluding a detailed assessment of each CI, it is asserted that the illustration and

demonstration of the methodology and conceptual framework is not lost.

Ten medical professionals (nurse, physician, or administrator) from ten different hospitals

provided expert knowledge based on the questions asked. Their responses are limited to their

individual experience and training. Conversely, those respondents that participated in this

research consisted of seasoned medical professionals appointed to speak on behalf of their

hospital’s practices.

1.6 RESEARCH CONTRIBUTION

This research seeks to contribute to the body of knowledge within multiple disciplines, as follows:

• Systems Engineering – provides a systematic, holistic, qualitative, and quantitative

approach to CI protection that considers the elements of the greater system, the element

interactions, and their emergent properties.

• Engineering Management – supports and informs decision-making using “what-if”

scenario analyses within the model to allow for proactive planning and better allocation of

resources based on security goals and implemented security measures.

• Systems Security Engineering/Information Security – combines security and systems

engineering best practices; this construct enables a comprehensive perspective of the

system and its interfaces/interdependencies to encourage better security in the appropriate

areas based on identified security goals.

• CI Protection/Resiliency – combines a framework for security professionals and decision-

makers to perform scenario-based, quantitative, and qualitative analysis of CIs and

dependent resources.

Specifically, the novelty expressed in this research expands upon existing CI interdependency

studies and analysis, while exploring a unique implementation of HHM and BBN through the

proposition of a hybrid multi-dimensional framework that generates quantitative and qualitative

results to assess security effectiveness.

1.7 SIGNIFICANCE/IMPLICATIONS OF RESEARCH

The ability to quantify the potential impacts to a critical hospital patient using numeric and

qualitative data can be instrumental for decision-makers seeking to preemptively execute

necessary security measures in order to demonstrate emergency preparedness, given a CI service

disruption. This conceptual framework allows CI owner/operators the capability to proactively

assess their situational awareness or security posture through scenario analysis, strategically

based on the organization’s security goals. For hospitals, maintaining public health by providing

continuity of services and ultimately preserving human life is of the highest priority; this is followed

closely by the goal of maintaining the confidentiality and integrity of patient records and billing

information. This methodology provides CI owners and dependent resources with a tool to assess

the exogenous (external) and endogenous (internal) nth order dependencies and impacts to ensure

emergency preparedness through various scenarios or what-if analysis [i.e. assess how the

degradation in power and/or water services may impact the ability of a hospital to provide

necessary services to a patient to sustain life]. The approach also allows for an analysis of

impact(s) between the CIs [i.e. how degradation in power may impact water resources and/or

communications]. The source of these impacts can easily go undetermined without multi-order

interdependency risk analyses.

Although this methodology and framework is demonstrated using a specific threat (Denial of

Service) to assess a specific purpose (impact to public health) given various possible vulnerabilities

(people, processes, tools, networks, or physical assets), the applications of this approach extend

beyond what is demonstrated here. This study approach could aid decision-makers assessing

various threats including the following:

• Availability of critical services to hospitals and patients;

• Integrity of the services provided by medical staff and vendors (medical devices);

• Confidentiality of patient records, billing, and pharmacy data.

Furthermore, this research suggests that in order for a complex, interdependent system of CIs to

effectively provide critical services to dependent resources, more effective and efficient standards

need to be implemented. Standards that mandate CIs communicate (share information) across

CIs and to those that depend on their services, in accordance with a security effectiveness

taxonomy understood by the impacted community of stakeholders. The SSEI concept and

construct offered in this research would allow CIs to communicate in a common language at a

confidential or sensitive information level, if necessary. This would allow appropriate security

measures to be considered from the perspective of their own internal security effectiveness

evaluation, as well as the external interdependent sources security effectiveness levels/evaluation.

As an example, during the stakeholder risk assessment, knowing the SSEI of other CIs as well as

your own SSEI, would aid decision-makers in a more efficient allocation of resources to effectively

maintain their security goal/target at an acceptable level of protection with the appropriate

resiliency, given a successful penetration or attack. Further, if a CI that is providing critical services

has a SSEI of 0.65, a dependent resource may want to ensure that his/her internal/individual SSEI

compensates for the weakness of that CI, potentially with a more immediate failover system for

power, or maintain a larger back-up water source or more robust filtration system.

1.8 DEFINITIONS OF KEY CONCEPTS

The proposed construct provides guidance on evaluating the security effectiveness of the

protective or resilient resources a CI operator/owner has chosen to implement - in a systematic,

quantitative, and performance metric-based approach. The following paragraphs describe the

essential elements of the combined framework and their relevance.

1.8.1 SECURITY EFFECTIVENESS FOR THE OPERATIONAL ENVIRONMENT

Relative effectiveness is best defined in its operational environment. In this study, the operational

environment is bounded by the components’ internal and external (interfaces) to the CIs and

dependent resources in question. The following definitions are important to note:

Security - the extent to which security measures provide protections that detect, deter, neutralize, and mitigate potential threats, while also providing resiliency measures to resist, respond, recover, absorb, and adapt to availing threats. (DHS NIPP, 2013)

Security Effectiveness - the degree to which security implementations provide adequate protective and resilient measures, allowing business operations to be maintained at an agreed upon level of service per the enterprise security goal.

Risk Management Effectiveness – determined by “whether and how much risk was actually reduced or whether risk was acceptable…” (Hubbard, 2009)

Security effectiveness, as defined above, implies that implemented security measures should not

impede, interrupt, or disturb critical operations of the enterprise, unless by design in order to

protect systems or persons from active attack.

Effective protection for one organization or CI may not apply to another organization or CI.

Measures applied for a specific threat may not be as effective for a different threat. The same

paradigm applies when measuring in different operational environments and for different security

goals. A more targeted solution considers what is relative or relational to the problem and specific

influences to the overall system. As a result, this research addresses security effectiveness more

appropriately as relative security effectiveness. Specifically, this research asserts relative security

effectiveness is best achieved by first defining the operational environment, understanding

associated dependencies, identifying the goal or target to be protected, and evaluating the problem

with a specific threat in mind.

1.8.2 HIERARCHICAL HOLOGRAPHIC MODELING (HHM)

HHM is one approach to multi-dimensional modeling (modeling from various/multiple perspectives).

The philosophy of HHM is grounded in the fundamental principle that complex, large-scale systems

such as CIs cannot be sufficiently appreciated or modeled in a planar or singular context. Haimes

[1981] states:

“The HHM approach (philosophy) recognizes that no single vision or perspective of a system is

adequate to represent a system and its component parts. Instead, the HHM approach identifies

and coordinates multiple, complementary decompositions of a complex system.”

HHM was chosen for this study to incorporate societal, legislative, environmental, spatial, and other

relevant dimensional perspectives that may contribute to the strength or weakness of security

posture. A CI/DR-HMM, developed for this research, is defined here as the HHM generated

specifically for the purposes of evaluating CIs or dependent resources. The HHM categories and

variables were extracted from a multitude of sources, to include the Department of Homeland

Security NIPP.

The HHM philosophy provides comprehensive, multi-dimensional insight into an otherwise

hidden problem/solution space to measure security effectiveness. To demonstrate the concept,

weights are distributed equally among the five (5) categories of the HHM (threats, vulnerabilities,

protection, resiliency, interdependencies), resulting in a sum of 20 percent for each hierarchical

category – totaling 100 percent for the entire critical infrastructure. Additionally, each category is

an aggregate of its components (i.e., protection includes detect, deter, neutralize and reduce.) See

Figure 2.

Figure 2. HHM for Critical Infrastructure or Dependent Resources (CI/DR-HHM)

This risk-assessment consists of input gathered from multiple stakeholders, such as engineering,

IT, and medical professionals. Each CI owner or decision-maker calculates their CI-HHM score as

described in subsequent sections. This information is used in the evaluation of the overall SSEI.

1.8.3 BAYESIAN BELIEF NETWORK (BBN)

BBNs are graphical illustrations of probabilistic dependencies (links) between variables (nodes).

The graph is a Directed Acyclic Graph (DAG) and the dependencies are such that any node given

its parents in the graph is independent of its non-descendants (Pearl, 1988). Like BBNs, attack

graphs are visual representations of physical and/or logical access into and within an enterprise,

network or CI.

An attack graph, as discussed in Frigault’s work [2014], can be represented as a DAG, coupled

with conditional probability tables (CPT) to constitute the BBN. A thorough implementation of

attack paths considers all paths that an attacker may exploit, both virtually and physically, to

access the CI/enterprise, network, or system. To demonstrate the BASE m2d concept the network

was modeled at the highest CI nodal hierarchy (black box) and the attack paths are notionally

identified via the DAG.

BBNs employ the fundamental premise of the Bayes Theorem:

The stated probability of an event or hypothesis is conditional based on the available/known

evidence in the relevant context. This condition can be made explicit by the notation P(H|E), which

reads as "the probability of event H given the evidence E." BBN is a method for understanding

evidence in the context of previous knowledge or experience [Pearl 1988]. The utility of BBNs has

become increasingly popular over the past decade in various fields of study to demonstrate

reliability, predictions, diagnosis, and decision analysis, among other uses. If it is accepted that

prior knowledge has intrinsic value, there is basis for using BBN.

In this study, a BBN is generated illustrating the dependencies and potential impacts of

successful penetrations originating from CIs. The BBN is used here for its ability to account for

uncertainty and limited available data of CI probability of attacks and interdependency/impact data.

For this study, the proof of concept is demonstrated by using historical and relative notional data

(prior probabilities), while the unknown values are calculated through a Bayesian software

simulation tool (Netica v5.15) to infer the CI interdependency impacts to dependent resources.

1.8.4 SYSTEMS SECURITY EFFECTIVENESS INDEX (SSEI)

The SSEI is a calculated value, resulting from risk-assessment performed by stakeholders to

understand (quantify) the relative security effectiveness and posture of CIs and/or dependent

resources. This index is designed to allow owners/operators the ability to assess and

communicate the strength and weakness of implemented and/or proposed security measures. The

SSEI serves as an evaluation of the risk mitigation steps a CI or dependent resource has taken to

protect against service disruptions. The BASE m2d framework is the vehicle developed to apply the

index. We show how an organization can use their self-evaluation of security effectiveness to

estimate the multi-order impact(s) of a CI service disruption.

https://www.norsys.com/

This research is intended to demonstrate how the SSEI (index) can be determined and used to

understand and improve security effectiveness. It ranges from 0 to 1 (0 – 100%), with a low index

indicating a weak or poor security effectiveness rating. Section 4.2.5 describes how the SSEI was

constructed to demonstrate the concept of this research. The BASE m2d conceptual framework is

the vehicle developed to exercise the index. The SSEI serves as an evaluation of the risk

mitigation steps a CI has taken to protect against DoS attacks or whatever threat is being

assessed. For example, based on the measures taken in various areas of security, per the CI/DR

HHM categories, a CI or DR would self-assess their overall security effectiveness (SSEI) to

determine their current posture or where they could improve. The BASE m2d framework is

provided for the CI owner to assess various scenarios given their current or objective SSEI.

Alternatively, an objective or threshold (minimum) SSEI can be obtained from a trade analysis to

determine the index required so as not to negatively impact the security goal (or have an impact of

an acceptable level). A general scale was developed for this study and used to illustrate the SSEI

concept.

1.9 ORGANIZATION AND OUTLINE

This document consists of nine sections: Introduction; Literature Review; Research Methodology;

Hospital Case Study; Data Analysis and Results; Conclusions; Recommendations for Future Work;

References; and Appendices. The Introduction details the research background, motivation,

hypotheses/questions, and the significance of the study. The Literature Review examines relevant

studies on critical infrastructure protection and resiliency, implementation of BBNs and HHMs for CI

analyses, and security effectiveness approaches. The Research Methodology describes the use of

survey research for studying existing CI interdependent patient-impact risk assessments performed

at hospitals, data collections, and sources and methods. The Hospital Case Study applies the

proposed methodology and implementation of the BASE m2d construct. The Data Analysis and

Results describe the statistical analysis of the data and results of the hypotheses testing. The

Conclusion section details the findings of this study from the perspective of the research questions

and hypotheses. The Recommendations for Future Work describes potential research directions

suggested to further this study. The References section contain a bibliography of the resources

used throughout this research. Finally, the Appendices provide supplemental material as a result of

this research.

CHAPTER 2 - LITERATURE REVIEW

The purpose of this chapter is to review, assess, synthesize, and critique existing literature with a

goal of furthering the body of knowledge in the field of evaluating and improving CIP/R security

effectiveness. To address the stated research problem in accordance with the established research

goals, existing tools (models, techniques, and approaches) developed for the purpose of

performing risk analysis of CI interdependencies were reviewed and compared against criteria

collected from current literature. This chapter evaluates how current approaches assess CIP/R

security effectiveness, given the complex, interdependent nature of CIs. Ultimately, models were

reviewed for their ability to assess multi-order (hidden) effects inherent in CI interdependencies

combined with the capability to insert mitigation, scenario-based modeling to potentially reduce

vulnerabilities. While compiling this literature review, it was necessary not only to express the gaps

noted in existing literature but to clearly articulate the distinctions of my research objectives, while

specifically stating how this study expands upon existing research. The hierarchical models chosen

for hybridization are stated up front, while justification for that selection is supported in subsequent

sections that detail the review of related works.

2.1 REVIEW OF RELATED WORKS

Review of literature from noted scholars (Ayyub, DiMase, Borum, Ryan, Di Giorgio, Pettigrew,

Bayuk, Haimes, Satumtira, Ghorbani, Rinaldi) identified relevant attributes/criteria to effectively

achieve the goal of generating a comprehensive framework, given complex adaptive systems, such

as CIs:

1. Align/trace to security goals

2. Strategic/systems engineering approach

3. Performance based metrics

4. Quantitative and qualitative assessment

5. Scenario or what-if analysis for decision making

6. Accommodates uncertainty

7. Allows for limited data

8. Assess security effectiveness of security measures

9. Extensible application

10. Assess indirect consequences/Interdependency analysis

Although these scholars have acknowledged the criteria above as imperative components to

model/provide/improve effective security - current models, methods, and techniques at most only

incorporate two or three components. Thus, the ability to assess the effectiveness of security

implementations on a holistic level has been limited.

2.2 OVERVIEW OF CIP/R ANALYSIS

Government agencies, private sectors and noted scholars have done a thorough job identifying the

importance of CIP (GAO, 2008-20014; PPD21; Zimmerman, 2001; Rinaldi, 2001, 2004; Moteff,

2005; Huang et.al.) Al., 2014; Cummings, 2014; Ezell, 2005; George, n.d.; Richard, 2008). Others

have gone further to note various ways to incorporate risk management in CIP analysis (Bensi,

2013; Haimes, 1995, Chittister, 2012; Kjølle, 2012). Some have emphasized that without

considering the interdependence of CIs, an analysis would be inadequate (Gheorghe 2005;

Haimes, 2004; Zimmerman, 2001; Santos, 2006; Zhang, 2011; Zio, 2013). Also noted is the fact

that more quantitative methods should be developed, implemented, and accompanied by

qualitative analysis (Kjølle, 2012; Ryan, 2005; Di Giorgio, 2012). It has been predominantly

discussed by those in academia and the government that modeling and simulation techniques are

effective in doing predictive, scenario-based analysis of CIP (Ouyang, 2014; P. Pederson, 2006; Di

Giorgio, 2012). Additionally, there has been a recent surge in adding resilience techniques based

on the realization that techniques or tools will never fully protect CIs or dependent resources

(Ouyang, 2012, 2014; Kahan, 2009; Vugrin, 2010, Little, 2013; PPD21, 2013; NIPP, 2012; Biringer,

2010).

Each of the aforementioned scholars acknowledge that more needs to be done to protect and

maintain our nation’s CIs; however, review of these related works primarily revealed that prevailing

methodologies do not provide a systematic, comprehensive approach towards assessing and

acquiring security effectiveness for CI protection and resiliency. In light of the heightened focus on

CIP and evidence of repeated penetrations, standards, protocols, and procedures have been

developed - only to provide a false sense of security. This study expands on

knowledge/experience and recommendations offered by previously documented research.

The following graphic depicts areas of existing research; while the area in grey denotes the

gaps that, if filled would constitute a comprehensive solution. This research targets the noted

gaps.

Figure 3. Research Focus Areas

Review of literature revealed both qualitative and quantitative methods are most effective when

combined to aid in reducing the likelihood of the undesired consequences of a successful attack on

a CI (Adar, 2005). Thus, the concept of combining HHM and BBN were explored; two hierarchical

models, one allowing for a qualitative analysis (HHM), while the other provides for a quantitative

assessment (BBN). The fundamental philosophy of HHM and the probabilistic backbone of BBN

were combined, with the “tuning” mechanism of a Systems Security Effectiveness Index (SSEI), in

an effort to develop a more comprehensive approach to ultimately evaluate and improve the

effectiveness of security measures. The BASE m2d conceptual framework is offered as a

contribution to the CIP/R crusade.

In this chapter we discuss the comparative analysis performed to select models chosen for this

research. Additionally, existing literature employing applications of HHM and BBN for CIP/R, and

other methods, their uses, and individual limitations are reviewed.

This literature review is organized as follows: Approaches to CI protection and resiliency;

Approaches to Measuring Security Effectiveness for CIs; Implementations of HHM for CIP/R;

Implementations of BBN for CIP/R; Literature Review Summary.

2.3 APPROACHES TO CI PROTECTION & RESILIENCY (CIP/R)

Successful attacks on CIs resulting in physical destruction have raised growing concerns regarding

the effectiveness of various techniques implemented for the purpose of providing protection. This

research examined existing methods, techniques, and strategies used to evaluate the

effectiveness of security measures for Critical Infrastructure Protection and Resiliency (CIP/R).

Specifically, this study sought to understand if and how these methods considered or extended

measures to protect dependent (2nd/3rd order) resources in a manner that is systematic, proactive,

and holistic, such that it assists in effective, efficient, and informed decision-making.

As noted by many scholars researching CI protection, there does not exist today a silver bullet

approach that completely protects and prevents the interruption or denial of CI services (Biringer

et. al., 2013). Instead, a more strategic and effective approach is needed, to include having the

ability to provide resiliency to maintain service in accordance with security metrics and stakeholder

goals. Biringer explains that security systems must be designed relative to the specific security

concerns of the infrastructure, the threat to the infrastructure, the security concerns of the

infrastructure, and the protection goals of the security system. They further elaborate that each

owner of the site, facility, or system must specify or describe the protection goals of its security

system to allocate sufficient financial resources and labor to meet goals with a clear understanding

of the level of consequences that are acceptable if the protection goals cannot be met (Biringer et.

al., 2013). Similarly, this study asserts that within the design, the CI owner/operator must consider

how a realized vulnerability of the infrastructure may affect the services provided by the

infrastructure, ultimately impacting dependent resources.

Over thirty-five models, techniques, and approaches were reviewed, with data sourced from

Ouyang (2014), Eusgeld (2010), Idaho National Labs (Pederson, 2006) and Satumtira (2010), and

Vugrin (2010), to assess how each technique synergized the multi-dimensional, multi-objective,

qualitative, stochastic, and hierarchical nature of CIs and dependent resources (CI/DR). Although

each model was in various stages of maturity (R&D, Internal-only, operational), they were each

designed for the purpose of analyzing CI interdependencies. Many were designed for a specific CI

(internal dependencies), others intended to manage cross-sector dependencies. It was unclear

which tools were designed to evaluate multi-order effects (indirect consequences of negative

events), which is of great interest to this study. In the models reviewed, resiliency was handled, at

most from the perspective of redundancy. Many of the tools had the ability to perform sensitivity

analysis and the ability to determine various “strength” of dependencies (i.e. which dependency

had the greatest impact on another CI). Various decision analysis techniques were built in to

determine or indicate priority and relative importance; however, very few of the models illustrated

the ability to assess dimensional interdependencies (legislative, societal, economic, stakeholder,

etc.). At most, a few models had the ability to incorporate temporal, spatial, and geographic data.

Five models were highly regarded by the DHS: Athena, CARVER, Critical Infrastructure Modeling

system (CIMS), Knowledge Display and Aggregation System (KDAS), and Maritime Security Risk

Analysis Model (MSRAM). Each of the five models are considered Model-Based Risk Analysis

(MBRA) tools, known for their ability to aid in risk-informed decisions (Lewis, 2012). CARVER,

Athena, KDAS and MSRAM were each designed specifically for military and government entities,

while CIMS targeted emergency planners and responders as end users. Of all the models

evaluated, no tool clearly articulated how or if effectiveness was assessed; and the extent to which

resiliency was addressed, it was limited to identifying redundant components/measures. No tool

addressed resiliency as it is defined by the NIPP (Ouyang, 2013). The ability to handle minimal

data and account for uncertainty was only managed by tools with a stochastic engine, however,

even those tools did not address effectiveness from the perspective of both protection and

resiliency, nor from various dimensions, as previously described.

Table 2. Literature Review CI Model Comparative Analysis

Although unable to physically manipulate the models evaluated for this research, the data available

served well in filtering various capabilities and limitations of each tool. While each tool appeared to

serve a valuable fit for its purpose, it did not appear evident that they lent themselves to trivial

modification for extensibility to incorporate additional/lacking features. A tool is most valuable and

effective when designed from its core to allow for modular growth that enhances or provides

Criteria/Features

B A S E m

2 d

B o

lo g

n a

N IP

P C

C IP

P P

D 6

B r ie

r e

R in

a ld

P e

e r e

n b

o o

H a

im e

Z im

m e

r m

a n

M e

n d

o n

c a

O s o

r io

B e

n s i

H a

im e

A IM

A t h

e n

C A

R V

E R

C I3

C IM

C IP

/D S

C IP

M A

C IS

D E

E M

C A

F A

F IN

S IM

F o

r t

F u

tu r e

IE IS

II M

K D

A S

K M

M IN

M S

R A

M U

N IC

IP A

N - A

B L

N E

M O

N e

t- C

e n

tr ic

G IS

N E

X U

S F

u s io

F r a

m e

w o

r k T

N g

to o

N S

R A

P F

N A

T R

A G

T R

A N

S IM

W IS

Conceptual Modeling and/or Simulation ToolsApplied

1 Align/trace to security goals x x x x x x

2 Strategic approach x x x x x x x x x

3 Performance-based metrics x x x x

4 Quantitative and qualitative x x x

5 Scenario or what-if analysis x x x

6 Accommodates uncertainty x x x x x

7 Allows for limited data x

8 Assess security effectiveness x x x x

9 Extensible application x x x

10 Interdependency Analysis x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

11 Protection x x x x x x x

12 Resiliency x x x x x x

Hybrid

additional capability, vice adding on to a tool that was not built to be dynamically modified.

Consequently, I sought to establish the BASEm2d conceptual framework, a tool that asserts to

provide what existing tools lack, a comprehensive approach to assess security effectiveness of

measures proposed/implemented for CI protection and resiliency; while also providing modularity

for future improvements and/or additional functionality.

2.3.1 IMPLEMENTATIONS OF DECISION ANALYSIS TOOLS FOR CIP/R

HHM has been used extensively in assessing and identifying sources of risk, complex

interdependencies of water resource CIs (Chittister e.t al. 2012; Haimes 1995), and Supervisory

Control and Data Acquisition (SCADA) networks (Haimes 2005). However, this research will be the

first implementation of HHM to assist in the identification of strength or weakness weights/index,

coupled with using BBN analysis to assess and improve security effectiveness of CI protections

and resiliency.

Haimes originally designed HHM to identify sources of risk. Although modified in this study

from its original design, I maintain and leverage the integrity of the HHM philosophy and concept

for its ability to not only identify sources of risk, but to assist in the identification of relevant

variables hierarchically, in various categories, and from multiple dimensions. Approaches such as

Analytical Hierarchy Process (AHP), Multi-Attribute Utility Theory (MAUT), or Multi-Criteria Decision

Analysis were evaluated as comparable models; and although each of these methods also assist in

decision making and allow for a structured way of framing the problem, the need to assign relevant

weights to each criteria to show importance was not necessary to meet the objectives of this

conceptual study. Instead, to demonstrate the preliminary concept, variables were elicited from

existing research previously collected from experts at the DHS and documented in the NIPP. CI

experts determined that the variables identified within the NIPP held equal weight at the highest

level of evaluation. In an effort to scope individual components of this framework I leverage

(extend) vetted research to focus this research on demonstrating the comprehensive framework.

This research acknowledges that criterion weights are a reality (all variables may not be equally

important), hence they may vary depending on budget constraints and available resources.

Variables should be re-evaluated/weighted on a case by case basis, which would suggest the use

of the aforementioned decision analysis models such as AHP and others. This is recommended as

a future enhancement to the BASE m2d preliminary framework. The HHM component of this tool is

a modular component which can be modified or replaced as the user desires.

2.3.2 IMPLEMENTATIONS OF PROBABILISTIC RISK MODELING FOR CIP/R

Graphical probabilistic models such as Markov Random Fields (Markov Networks) and Bayesian

Belief Networks were explored to address the uncertainty of complex system interdependencies

and the very real occurrence of limited data; specifically, a probabilistic model that allows for

scenario/what-if analysis, with a user-friendly interface, and one that does not require great

statistical knowledge. Markov Networks are known for their power and flexibility (undirected,

allowing cycles); BBNs are considered to be restricted by comparison (directed, acyclic). It was

discovered that both Markov Networks and BBNs would suffice for this study, however, BBN was

chosen simply based on its immediate capabilities, the researcher’s familiarity with the method,

and its ease of use.

A review of existing literature indicates BBNs have been implemented to model and analyze

the interdependencies of CI (Di Giorgio, 2011). BBNs have also been used to predict the likelihood

of terrorist attacks on CIs (Johan 2009; Haimes 2004). Kazak (2010) uses BBN with a security

ontology to assess the severity level of detected threats, while Faribault (2014) demonstrates the

utility of BBN to assess vulnerability in computer networks. Additionally, Queiroz (2013), et. al. use

BBN to evaluate information diversity within SCADA systems. There is also extensive research

available implementing BBNs to provide earthquake decision-support systems for seismic

infrastructure risk assessment, to include Bensi (2010), Bayraktarli (2005), and Kheun (2009). In

the medical field, BBNs are currently applied to assist in more accurate diagnoses (Forsberg

2011), to predict the occurrence of cancers (Burnside, n.d.), and to estimate patient survival given

the presence of certain cancer prognostic factors (Forsberg 2012).

Although it was determined through literature review that BBN is effective and most

appropriate to evaluate CI vulnerabilities and medical prognosis/diagnosis, gaps exist in its

implementation as a catalyst for stimulating decision analysis for the purpose of assessing and

quantifying security effectiveness toward improved CI protection and resiliency. Further, no

comprehensive approach that generates an effectiveness index based on existing implemented

security measures for the purpose of further improving security posture was found.

As previously discussed, various CI interdependency analyses/approaches have been

extensively explored by scholars such as Rinaldi, Haimes, Di Giorgio, Zhi-yan, Macaulay, and

others. However, this research extends upon that research with provisions for a holistic framework

to address relative security effectiveness, given CI interdependencies, to make informed decisions.

It is acknowledged that even the best efforts will not afford absolute protection, thus the

need to simultaneously prepare for resiliency (i.e., adaptability, recoverability, absorption, etc.) of a

cyber or physical attack. The BASE m2d framework is aimed at providing both proactive and

responsive measures toward better P/R. Subsequent sections of this paper detail the methodology

and the combined implementation of the associated models (HHM and BBN).

http://gw.summon.serialssolutions.com/search?s.dym=false&s.q=Author%3A%22Zhi-yan%2C+Liu%22

http://gw.summon.serialssolutions.com/search?s.dym=false&s.q=Author%3A%22Macaulay%2C+Tyson%22

2.4 APPROACHES TO MEASURING SECURITY EFFECTIVENESS FOR CI

Methods used today to evaluate security effectiveness include variations on vulnerability analysis,

penetration testing, threat analysis, and/or risk analysis. It would be expected, at a minimum, for

organizations to perform some aspect of each of these activities before allocating resources to

improve their security effectiveness. Conversely, review of literature has revealed that those

employing any one of the aforementioned analyses, most often do not necessarily do so in a

systematic, holistic manner (Biringer, 2013; Pettigrew, 2009; Ryan, 2008.). For example, the

assessment of various threats/vulnerabilities from the perspective of various stakeholders

predicated on the organization’s security goals, using qualitative or quantitative performance-based

metrics to determine the effectiveness of the protection and/or resilience measures they have in

place. Additionally, until recently, those efforts that focused on vulnerabilities, penetration testing,

threats, and risk did so only as it related to protection. More recently, there has been an exertion

toward the realm of implementing practices to ensure resilience. Currently, the review of related

works primarily identifies that existing methodologies do not provide an approach toward assessing

and acquiring security effectiveness for CI protection and resiliency.

2.5 SUMMARY

In Chapter 2 the problem currently faced by the nation to protect and maintain CIs from successful

cyber or physical attacks was discussed. Several models were reviewed to evaluate their ability to

address the multi-dimensional, multi-objective, quantitative/qualitative, and hierarchical nature of

CIs and dependent resources. Of the models reviewed, no tool clearly articulated how or if

effectiveness was assessed; and the extent to which resiliency was addressed was limited to

identifying redundant components/measures. The ability to handle minimal data and account for

uncertainty was only managed by tools with a stochastic engine, however, even those tools did not

address effectiveness from the perspective of both protection and resiliency, nor from various

dimensions, as previously described. Consequently, this research implemented the integration

(hybrid) of two hierarchical techniques that encompass each of the ten (10) aforementioned

criteria. It was hypothesized that by combining multidimensional modeling such as HHM and BBN,

a conceptual framework could be developed to enable scenario-based analysis to assess the

effectiveness of implemented or proposed security measures.

Chapter 2 also discussed a detailed overview of existing techniques and models used today to

provide protection and resiliency to CIs and those that depend on its services. As previously

discussed, various CI interdependency analyses/approaches have been extensively explored by

scholars such as Bloomfield, Eusgeld, Haimes, Min, Ouyang, Kjolle, Liberati, Satumtira,

Zimmerman, Rinaldi, Haimes, Di Giorgio, Zhi-yan, Macaulay, and others. However, this research

extends upon that research with provisions for a methodology and framework to address relative

security effectiveness, given CI interdependencies, to make informed decisions.

CHAPTER 3 - RESEARCH METHODOLOGY

This chapter describes the research methodology chosen to investigate how two hierarchical

modeling techniques can be combined to provide a quantitative and qualitative conceptual

framework (BASE m2d) that can be used to assess and improve the relative security effectiveness

of CIs and dependent resources. Described herein, is the approach taken to develop, test, and

validate the holistic framework. The research method chosen consists of a survey designed to

elicit probability distributions from experts, which are subsequently employed to validate the BASE

http://gw.summon.serialssolutions.com/search?s.dym=false&s.q=Author%3A%22Zhi-yan%2C+Liu%22

http://gw.summon.serialssolutions.com/search?s.dym=false&s.q=Author%3A%22Macaulay%2C+Tyson%22

m2d framework. To test the hypotheses, a hospital case study was designed to demonstrate the

frameworks’ utility, flexibility, and limitations.

3.1 RESEARCH DESIGN

This study utilized a focused cross-sectional survey to collect data from medical professionals

employed at 10 distinct hospitals. The survey was used to capture the knowledge and experience

of medical professionals regarding the use of metrics implemented by their respective hospital

facility to measure effectiveness and/or emergency preparedness. Data was also collected to

understand various impacts to a critically ill hospital patient (dependent resource) given a CI failure

or Denial of Service (DoS). Survey responses were collected via a web-based, electronic tool

(Survey Monkey). Using the expert elicitation method, the data was further used to validate the

BBN model and expected results. Elicitation of specific data from medical experts was limited due

to the acknowledged vulnerability of hospitals and potential insight it may provide to adversaries.

Thus, we use a combination of real and notional data and present this research as a proof of

concept.

A survey was used to evaluate various impacts to a hospital patient given a DoS to a CI and to

understand the knowledge of medical professionals regarding metrics implemented by their

employment facility (hospital). Surveys are often used to gather statistical data about

demographics, actions, techniques, perceived effectiveness, characteristics or attributes from a

selected or random community, or category of a specific population (Babbie, 2010; Creswell, 2009;

Salkind, 2009). Descriptive and inferential analysis was performed on the data collected using

Netica, SPSS, and Minitab.

The research survey process for this study maintained the strictest confidence of respondent

information and identities, as required by the GWU Institutional Review Board (IRB). No personally

identifiable information (PII) was acquired, and all metadata associated with each respondent was

securely discarded.

3.2 EXPERT ELICITATION

Expert elicitation (EE) is often used when it is not feasible to collect empirical data for statistical

analysis (Cooke 1991). According to Ryan, et. al., EE is designed to elicit, codify, and combine the

knowledge of people who have significant experience or expertise in a defined field in order to

assess unknown quantities or parameters (Ryan 2012). These scholars go on to state that the use

of “expert judgment is justified when quantitative data is missing, of dubious quality, or is

insufficient for obtaining reasonable statistical results.” Various methods are noted in scholarly

research of how to elicit, codify, and combine expert knowledge, as described by Cooke (1991).

For this research effort, methods detailed by Buede, Renooij (2001), and Pitchforth (2011) was

used based on its extensive application in Bayesian analysis. Bayesian Networks are often

created through the process of EE (Pitchforth 2012).

Studies have shown that experts tend to be overconfident about their judgments (Tversky

1974, Lin 2008, Flandoli 2011). Heuristics and biases are issues that often arise when eliciting

information from experts. To overcome these relevant concerns, scholars recommend a well-

structured elicitation process. According to Kahneman et. al., (1982) and Renooij (2001), bias is a

systematic tendency to take into account factors that are irrelevant to the task at hand, or to ignore

relevant facts, thereby failing to make an inference that any appropriate normative theory, for

example probability theory, would classify as necessary. There are two types of biases to

consider: motivational bias, which is caused by personal interest or circumstances experienced by

the expert; and cognitive bias which arise during the processing of expert information with the use

of heuristics (availability, anchoring, representativeness, and control) (Kahneman et. al., 1982).

Motivational bias can be mitigated by reassuring the respondents that their responses are for

information only and not a promise or commitment. Cognitive bias can be minimized by letting the

respondent know that they exist and by subjecting the experts to calibration (Hubbard, 2016). The

survey should consider these concerns in deciding upon the survey method used (Renooij 2001).

This research has taken specific steps to minimize bias, inconsistencies, potential errors, and

overconfidence. The following process was used to elicit relevant data from respondents:

Step 1: Expert Selection

Experts were selected by various representatives of each hospital solicited for data on this study

(Hospital Risk Assessment Office, Administrator of hospital, Chief Information Technology Officer,

etc.). Each respondent was selected based on their domain knowledge of hospital operations and

their ability to assess patient impacts. The respondents were advised that their individual

participation would remain anonymous and data provided would be voluntary, with no obligations

or retribution to their employment. They were also assured that the accuracy of their information

need only be based on their experience (Renooij 2001).

Step 2: Set Foundation for the Expert

During initial contact, the expert was advised of the purpose of the study, to include background

information, definitions, and scope. Expectations were detailed and the confidentiality of personal

information was emphasized. Special care was taken to ensure that the same contextual

information was provided to each participant in a common manner so as not to introduce bias in

judgment (Boring, 2005).

Step 3: Train the Expert

Participants were subjected to an electronic survey which included an introduction and described

the purpose of the study. The introduction described the basic process and flow of the survey, and

each section provided necessary instructions for completion. The goal of this activity was to

ensure the expert felt comfortable with the process and understood with clarity the objective of the

survey instrument. The multiple choice, five-point Likert scale questions eased the burden of the

respondents who may have not been comfortable with providing direct probability distributions.

Step 4: Elicitation of Judgments

Responses were collected via a web-based, electronic tool (Survey Monkey). Experts were asked

to provide minimal demographic information (i.e. years of experience, medical profession, and

experience in ICU or other emergency patient care). Expert responses were calibrated based on

absolute agreement in accordance with Intra-class Correlation Coefficient (ICC). Probability

distributions were indirectly elicited from each expert, capturing the likelihood CI service disruption

impacts to patients. The elicited data was used to validate the behavior of the BBN model.

Step 5: Aggregating the data for Consistency

Upon completion of the survey, verification consisted of checking whether the elicited probabilities

are coherent, obey the laws of probability, and are reliable (Fenton, 1998). According to Renooij,

an indication of the validity of the assessments can also be obtained by entering observations into

the belief network and computing the effect of the observations on the probabilities for certain

variables of interest. The outcomes for these variables can then be checked against available data

or presented to the expert (Renooij 2001).

3.3 SURVEY SCALE

The scale used for this survey instrument consisted of a five-point Likert scale. The questions

were rank ordered from lowest (very unlikely) to highest (very likely). As noted by Fowler, Hayes,

Nunnally, Punch, Weisberg, a graduated scale will maximize the degree of variability in the

responses and lend itself to an analysis of continuous variable data (Fowler, 1995; Hayes, 1998;

Nunnally, 1978; Punch, 2003; Weisberg, 1977). The chances of misclassifying the data due to

reverse coding will be significantly reduced with rank ordered responses (Punch, 2003; Weisberg,

1977). Each rank order corresponded/coded to a probability distribution as follows:

1 2 3 4 5

Very unlikely

Unlikely

Need More Info

Likely

Very Likely

(0-20%) (21-40%) (41-60%) (61-80%) (81-100%)

3.4 EXPERT ELICITATION CALIBRATION

Various methods exist on how to elicit, structure, combine, and calibrate an expert’s response to a

survey (Ryan 2012; Flandoli 2011; and Cooke 1991). Cooke’s Classical Method (1991) suggests

weighting expert’s opinions and experience with “seed questions,” to which the answers are

generally available and typically known by subject matter experts in the field to be studied. How

the experts respond are scored in accordance with the true answers and consequently weighted.

The weights are derived from a combination of the expert response to the seed questions and are

used to calibrate the “accuracy” of the experts’ opinion. Calibration measures the statistical

likelihood that a set of experimental results corresponds with the experts’ assessments (Cooke

2004). Alternatively, Hubbard (2016) discusses how he facilitates workshops to “calibrate the

expert” to improve one’s ability to subjectively assess odds. He teaches various methods to

reduce overconfidence and underconfidence in responses – known challenges to eliciting expert

judgment.

Expert responses used in this study were not calibrated as offered by Cooke, given no seed

questions were offered within the survey to calculate weights. To account for this, we attempted to

take extreme care during the expert training period, as preferred by Hubbard – calibrate the expert.

The probability distributions collected from experts were used to confirm BBN validation, i.e., does

the model behave as it should; are the simulated results as expected.

This overall elicitation approach was taken with caution and considered a sound way to

minimize biases and under/overconfidence. The following considerations were taken into account

to proceed with this method, when no other data exist (Johnson 2010):

• Detailed training of experts prior to the survey

• Clear, standardized instruction script

• Provided Likert scale with corresponding probability distributions

• Avoiding use of scenarios or anchoring data

• Allow for comments or feedback

3.5 SURVEY INSTRUMENT

This research leveraged the flexibility and convenience of a professional online tool (Survey

Monkey) used to collect data, extract, and compile the final results. Survey Monkey is a

convenient, web-based tool, available 24/7 to respondents during the data collection period.

Although many researchers find several advantages to using the online survey tool, such as being

cost efficient and providing quick results. Survey Monkey also allows for various design methods

to present/display questions and collect responses, it has a global (electronic) reach to extend

beyond places a researcher could physically be present, and it also tends to have a higher

response rate than other popular methods. Detailed benefits are described below:

Benefits of Electronic/Online Survey: http://writing.colostate.edu/guides/page.cfm?pageid=1406&guideid=68

• Cost-savings: It is less expensive to send questionnaires online than to pay for postage

or for interviewers.

• Ease of Editing/Analysis: It is easier to make changes to questionnaire and to copy and

sort data.

• Faster Transmission Time: Questionnaires can be delivered to recipients in seconds,

rather than in days as with traditional mail.

• Easy Use of Preletters: You may send invitations and receive responses in a very short

time and thus receive participation level estimates.

• Higher Response Rate: Research shows that response rates on private networks are

higher with electronic surveys than with paper surveys or interviews.

• More Candid Responses: Research shows that respondents may answer more honestly

with electronic surveys than with paper surveys or interviews.

• Potentially Quicker Response Time with Wider Magnitude of Coverage: Due to the

speed of online networks, participants can answer in minutes or hours, and coverage can

be global.

• Same strength as written survey

• Ability to consistently track responses

• Automatic randomization of questions and answers choices to remove potential biases

Drawbacks to Electronic/Online Survey:

• Sample Demographic Limitations: Population and sample limited to those with access to

computer and online network.

• Additional Orientation/Instructions: More instruction and orientation to the computer

online systems may be necessary for respondents to complete the questionnaire.

• Potential Technical Problems with Hardware and Software: Computers have a much

greater likelihood of "glitches" than oral or written forms of communication.

• Potential for SPAM

• Potential for survey fatigue

The survey questions were designed to be simple, straight forward, and elicit an intuitively positive

response (Punch, 2003; Weisberg, 1977). The questions were also designed to be without

negative statements, which have been known to confuse respondents and lead to non-normal,

diverged, and skewed distributions and result in weaker statistical correlations (Hayes, 1998).

3.6 VALIDITY OF SURVEY INSTRUMENT

Purpose: To collect relevant data for the purpose of understanding how hospitals assess,

measure, and/or utilize the following:

• Hospitals utilization/implementation of metrics to trigger alternate critical systems for

water, power, and communications

• Likelihood of Impact to hospital and/or critically ill patients, given a limited or complete

service disruption of power, water, or communications to the areas of cardiac care,

dialysis, and oxygen/ventilator – services to which the critically ill patient is dependent.

Approach: The results of the survey were used to validate the findings of the conceptual

framework. To determine if the results of the survey instrument are valid and reliable, the

respondent data was evaluated against the data produced by the tool. If an instrument is

unreliable, it is also invalid, because accurate findings cannot be obtained with inconsistent data

(Carmines, 1979). A valid survey instrument serves the purpose it is intended to serve and

provides correct information (Fink, 2003). This survey instrument was evaluated based on the

following criteria/tests:

RELIABILITY ▪ INTERNAL CONSISTENCY or Homogeneity using a Cronbach coefficient (α >0.7)

o Measures the extent to which a technique, experiment, or measuring procedure

assess the same characteristic or quality.

▪ INTER- AND INTRARATER RELIABILITY

o Measures the extent to which multiple respondents agree in their ratings of given

items.

VALIDITY

▪ CONTENT VALIDITY o Measures the extent to which the question/items thoroughly and appropriately

assess the characteristics or qualities it purports to measures.

▪ FACE VALIDITY

o Measures the appearance of the metric/question on the surface.

3.7 VALIDITY OF CONCEPTUAL FRAMEWORK

Purpose: The objective of the conceptual framework is to provide relevant information for the

purposes of providing proactive protection and resilience to CIs and dependent resources. In

doing so, the validity and reliability of the conceptual framework was evaluated. Specifically, the

following approach was used to measure the tools’ effectiveness for its purpose, and its ability to

perform as a hybrid model to measure and assess security effectiveness of CIs and dependent

resources.

Approach: Previous research performed by the DHS identified relevant variables used to

measure effectiveness for CI protection and resiliency. This research was furthered by using those

variables within the conceptual framework (BASE m2d) to demonstrate the ability to quantitatively

and qualitatively measure and assess security effectiveness of CIs and dependent resources.

Further, the data collected from medical professionals was used to validate the results of the hybrid

conceptual model.

3.8 DATA COLLECTION

Data was collected and provided from various sources as described herein. Ten medical

professionals from 10 distinct hospitals provided data through expert elicitation, to validate the BBN

model and expected results. DHS and various sources discovered through research provided the

relevant variables used for the SSEI and HHM. The specific CI categories and hierarchical

relationships used in this study were extracted from the Department of Homeland Security (DHS)

National Infrastructure Protection Plan (NIPP) and their associated Sector Specific Plans (SSP).

Historical and relational data was used as prior probabilities for the BBN analysis from sources

such as the medical professionals, DHS, Verizon 2013, 2014, and 2015 Breach Reports, Symantec

Annual Security Threat Report, and Verisign. Specific data for each CI was collected from

government sources such as DHS in collaboration with the Department of Energy (DOE), Water

Waste and Sewage (Environmental Protection Agency), and the Communications sector (National

Communications Systems). Previous research performed by the DHS identified relevant variables

for CI protection and resiliency (CIP/R). Patient impact data was collected per the Institutional

Review Board (IRB) guidelines, ensuring no personally identifiable information was acquired,

attributed, or misused. Survey Monkey, a web based renowned vendor often used in academic

research, was employed during this study as the survey instrument to facilitate the collection of

patient impact data from medical professionals. Results of the survey were used to validate the

BBN model and overall conceptual framework.

CHAPTER 4 – HOSPITAL CASE STUDY

4.1 CASE STUDY BACKGROUND

Public trust depends upon the sustainability, resilience, integrity, and availability of national

Healthcare and Public Health (HPH) critical infrastructure [NACCHO 2014]. Continuity of

healthcare and public health services are critical to response and recovery following a disaster or

emergency [NACCHO 2016]. Results from this research reveal that hospital engineers have

performed due diligence to ensure if power is disrupted or water supplies have been tainted or

halted, backup generators and alternate water supplies are available to maintain critical services.

However, this research further revealed that most hospitals have not performed additional risk

assessments to evaluate implemented security measures to allow them to understand, evaluate,

and reduce the potential impact of a CI DoS (e.g., water, power) to a patient or medical device.

Understanding how and where to properly address and allocate security measures in a

budget-constrained environment will prove invaluable to ensure these critical resources/services

are uninterrupted (or have minimal or acceptable impact). This case study models a critically ill

patient in ICU who depends on medical equipment (dialysis, defibrillator, etc.) for survival. Given a

DoS to a CI (power, water, etc.), the hospital could be subsequently impacted if the proper back-up

resources are not engaged or available in a timely manner

Multi-order, cascading effects are important to understand given they can occur as a result

of a direct or indirect attack or occurrence. A CI failure or DoS (resulting in a partial shutdown or

complete shutdown) can be due to a successful attack, whether virtual or physical, intentional or

unintentional, potentially having a 2nd, 3rd, or 4th order effect [Figure 2]. The case study modeled in

this study examines the 4th order effect to a patient that is depending on CI services to sustain life.

Figure 2. CI Interdependency Multi-Order Effects

4.2 CASE STUDY APPLICATION

The Bayesian Approach to Security Effectiveness with metrics, modeling and decision-support

(BASE m2d) framework uses a hospital case study to demonstrate the overall methodology.

Described herein are the steps to perform a risk-assessment using the CI/DR-HHM, calculate the

SSEI, and perform “what-if” analyses to determine the potential impact to a dependent resource

(hospital or patient) given a successful DoS to one or more CIs (water, power, communications).

Figure 4 represents the general flow of the BASE m2d framework.

Figure 4. General Flow of BASE m2d Framework

The following sections describe the ten-step methodology of the BASE m2d framework. The first

three steps define the problem-solution space; Steps 4-7 assess the current security posture using

metrics, the operational environment from multiple stakeholder perspectives using HHM, the

overall relative effectiveness using SSEI, and the likelihood of impact using BBN; while Steps 8-10

estimate the relative risk, and describe the iterative decision analysis process.

4.2.1 STEP 1 AND 2: DEFINE OPERATIONAL ENVIRONMENT AND SECURITY GOALS

The strategic implementation of this framework begins by establishing security goals within the

operational environment. The Department of Homeland Security National Infrastructure Protection

Plan (NIPP) has developed value propositions and/or security goals for each CI identified in the

plan. Establishing security goals includes documenting what is important to the organization and

considers perspectives of all stakeholders, to include CI owners/operators, dependent CI

owners/operators, vendors, consumers/customers, etc. Many decision-makers focus resources in

areas that have little to no impact on what they value, and without the full consideration of the

operational impact; thus, leading to security compliance with little security effectiveness. Once

goals are established within the operational environment, measures can be directly implemented to

monitor progress toward achieving those goals in accordance with the value proposition.

HPH Security Goal: For the HPH sector or hospitals, it is crucial that they maintain

(resilience) a certain level of business continuity to preserve human life and to protect

the confidentiality of information for their patients and personnel.

The HPH security goal identified above can be divided into two separate targets to protect: human

life and patient information. This case study focuses on preserving human life to demonstrate the

capabilities of the model; however, the model is extensible to include an analysis of the impact of a

successful attack on patients’ personal information, as well. That case is excluded here based on

the threat (DoS) in question, which is often employed for purposes other than to exploit personal

data.

4.2.2 STEP 3: IDENTIFY DEPENDENCIES

Internal and external dependencies (also known as potential attack paths) to the protection target

or security goal (hospital, patient), within the operational environment can indicate vulnerabilities.

Specifically, if there is a penetrable entry or exit point (link) to or from the target, strategic

consideration should be given to apply appropriate measures of protection or resiliency to prevent

undesired effects in the event of an attack or natural disaster. Links/dependencies may be

identified as having virtual and/or physical access and should be prioritized, especially in a budget

constrained environment. Identifying dependencies between CIs (water, power, communications)

and dependent resources (hospital, medical devices, patient) provides the topology to construct the

general Bayesian Belief Network.

THE METRICS

4.2.3 STEP 4: ASSESS/MEASURE THE SECURITY POSTURE

Relevant metrics should be identified and implemented to assess the current security posture of

the organization, particularly metrics that acknowledge and indicate internal/external degradation of

the infrastructure. Metrics chosen should pass the “so-what” test and should be selected in relation

to the security goal within its operational environment. They should be identified and defined

around the problem space. Without metrics, organizations will find it difficult to accurately gauge

effectiveness and articulate improvement. When metrics can be quantified as a number or

percentage, are contextually relevant, and measured consistently, they confer credibility to the

overall assessment (Jaquith, 2007). Jaquith goes on to state that good metrics should facilitate

discussion, insight, and analysis.

Security metrics are the servants of risk management, and risk management is about making

decisions. Therefore, the only security metrics we are interested in are those that support

decision making about risk for the purpose of managing that risk (Jaquith, 2007)

Getting the right metrics requires asking the right questions. Cyber-attacks may cause a temporary

disruption (minutes to hours), while natural disasters such as tornadoes or hurricanes may cause

long-term outages (weeks to months). A question such as “are failover settings for the backup

generator sufficient to not have a negative effect on the patient, in the event that power is disrupted

at the hospital’s main plant,” should lead to identifying the appropriate metrics that ensure the

ability to proactively monitor status and plan accordingly. Medical devices dependent on CI

services would benefit from metrics such as mean time between failure (MTBF) for monitoring or

alarm if a failover system (power, water) exceeds a certain value. Asking how long a backup

generator can support a critically ill patient surviving on a ventilator before asphyxiation or brain

damage occurs should also result in relevant metrics. Although the answer to this question may

depend on the severity of the patient’s illness, the reliability and sustainability of the medical

equipment should be measured, understood, and baselined accordingly, to assist in proper

planning.

Penetration testing should also be performed and results included in the overall

assessment to continuously monitor and capture anomalies due to unsolicited or unintentional

physical or cyber access. These combined techniques provide an understanding of the current

security posture and insight into the strength or weakness of the enterprise.

The BBN model developed for this research is structured such that if current measures

indicate an unacceptable impact to the patient, additional or alternative security measures should

be implemented and assessed. Threshold and objective parameters should be considered for

each metric (there may be cases where threshold parameters may be sufficient). Identifying these

parameters and monitoring for trends or outliers will allow emergency personnel to take appropriate

actions prior to an undesirable event. Such metrics would be useful to ensure resiliency. Step 4

results in the identification and assessment of relevant performance metrics by stakeholders that

further assist in understanding and monitoring the health of the enterprise.

Below, we provide an example set of performance metrics to be used for the protection

and resilience of CIs or dependent resources is provided in Tables 3 and 4.

Table 3. Reference Metrics for Critical Infrastructure/Dependent Resource Protection

Table 4. Reference Metrics for Critical Infrastructure/Dependent Resource Resilience

4.2.4 STEP 5: ASSESS MULTIPLE DIMENSIONS/PERSPECTIVES [CI-HHM]

[Steps 5 and 6 are closely linked. Step 5 details the risk-assessment process using HHM which is

required to calculate the Systems Security Effectiveness Index (SSEI), discussed in Step 6.]

This extensible CI/DR-HHM is provided with hierarchical and dimensional categories to

allow the CI/DR owner/operator to perform a risk-assessment of his/her infrastructure. Each

category (threats, vulnerability, protection, resiliency, and interdependency) is used by the

stakeholders to assess the security measures currently in place or proposed. Each stakeholder

responds to a series of the same questions from the perspective of their own discipline (e.g.

engineer, IT specialist, doctor/nurse, hospital administrator, etc.). This includes a combined

stakeholder threat and vulnerability analyses, for example, to determine the extent to which each

discipline effectively implements measures to reduce vulnerabilities, employs threat modeling,

plans for protection and resiliency, and considers interdependencies of other CIs and relevant

resources. The strength (or weakness) of security measures employed by an organization is

calculated based on a combined stakeholder risk-assessment of each CI/DR-HMM category. The

quantified results identify areas of deficiency, thus implying areas where improvements can be

made. The CI owner or dependent resource facilitator should evaluate (weight and priority) each

CI/DR-HHM category per their risk tolerance and goals.

Figure 2 illustrates the third order (hierarchical) graphical representation of the CI/DR-HHM,

with general weights uniformly distributed among its five categories. Adjacent to each sub-

category is the max weight a CI/DR owner/operator would self-assess the effectiveness of the

security measures they have in place. The decision-maker combines the results of the risk-

assessment completed by multiple stakeholders (internal and external), such as risk managers,

engineering, IT, and medical professionals to calculate the SSEI, described in Step 6.

Figure 2. HHM for Critical Infrastructure or Dependent Resources (CI/DR-HHM)

Using the CI/DR-HHM categories provided in Figure 2, relevant questions should be devised by

the organization and each question should have a measurable component for monitoring and

improvement. Qualitative responses to questions such as “do we have measures in place to

protect (blank)” (yes/no); “if so, what are they and are they sufficient?” Table 5 provides a general

CI/DR-HHM scale to score the risk-assessments for each category. A general SSEI scale was

developed for this study to illustrate the concept. [Table 5 and 6].

Table 5. CI/DR-HHM SSEI Scoring Scale (by Category)

Categories are identified as the main variables within the CI-HHM (threats, vulnerabilities, protections, resilience, and

interdependencies). Each of the five CI subset categories can have a maximum value of 20 percent (0.20).

A CI or enterprise can have a strength/weakness value per category in the range from 0 – 20

percent (0 – 0.20) as shown in Table 5. This weight or value implies that the enterprise has

implemented security measures (for that category) to a certain level of effectiveness in accordance

with their value proposition or goals. The results from the CI/DR-HHM category assessment

ultimately contribute to the overall SSEI “score” [Table 6] or level of the strength/weakness of an

organization’s security posture. Subsequently, the SSEI is used within the BBN analysis to

estimate impact. This step results in an understanding of specific areas of strength and weakness

of the enterprise and facilitates the ability to better allocate resources to security measures

identified as deficient so as to reduce negative consequences.

4.2.5 STEP 6: ASSESS STRENGTH/WEAKNESS [CALCULATE THE SSEI]

The SSEI is a calculated value that indicates/quantifies the risk management steps taken to reduce

the probability of a successful attack. Emergency management personnel or decision-makers are

to assess their effectiveness index based on the degree to which they implement measures and

reduce risk identified in the CI/DR-HHM categories [Figure 3]. Security measures should include

people, process, technology, e.g., “what, if any, security measures do we have in place, in the form

of people, processes, and technology, to detect, deter, neutralize, and reduce cyber/physical

attacks?”

Table 6. CI/DR-HHM SSEI Scoring Scale (Total)

The individual scores of the five categories are calculated from Figure 3 and Table 5, then combined for the overall CI-

HHM score. This total will be used to select the current effectiveness level of the SSEI in Table 6.

Table 7 illustrates scores elicited from a Department of Energy (power) engineering expert.

The CI/DR-HHM risk-assessment scores resulted in an overall SSEI score of 0.745 (75%) out of a

total possible score of 1.0 (100%). This particular CI owner, with appropriate stakeholders,

assessed their category relative effectiveness scores as follows: Threat (0.155 of max possible

0.20), Vulnerability (0.120 of max 0.20), Protection (0.178 of max 0.20), Resilience (0.156 of max

0.20) and Interdependency (0.136 of max 0.20). Table 4 denotes that a score of 0.745 falls in the

range of “good security posture.”

Table 7. Area of Improvement/Deficiency (Calculated SSEI)

Table 7 also identifies a difference score (column 3), which denotes area or room for

improvement. Given the overall score of 0.745, there is a total area of improvement of 0.255.

Specific areas can be identified to improve as the results are incorporated into the BBN model,

e.g., the Vulnerability category risk-assessment resulted in a difference score of 0.080 (having the

highest difference score). With an overall SSEI of 0.745, the BBN evaluation infers a “degraded”

impact to the patient. To improve this potentially unacceptable impact, the CI operator would use

the difference scores to take appropriate measures to increase overall security effectiveness,

ultimately improving their protection and resiliency. Further explanation is provided in BBN section

to follow (Step 7).

The examination of dimensional (category) elements of the CI/DR-HHM is critical to the overall

comprehensive security effectiveness evaluation. This research reveals that societal,

environmental, legislative, and stakeholder perspectives and actions contribute to either

strengthening or weakening protection and/or resiliency measures. An example to assess security

measures from a dimensional (stakeholder) perspective follows.

Example: HHM-Stakeholder Dimension: The questions in Table 8 should be asked from the

perspective of each stakeholder (engineer, IT professional, CI owner/operator, Physical security,

etc.).

Table 8. Exemplar Stakeholder Question Categories

A complete risk-assessment would follow the same logic and questioning from the perspective of

other dimensions, e.g., for the legislation/governance/policy dimension, “are there

laws/standards/policies in place (in the form of people, processes, or technology) that enable or

prevent our ability to detect, deter, neutralize, or reduce cyber/physical threats?” If so, what are

they and are they strategically employed to protect our security goal(s)? This step exits with an

SSEI score to be incorporated next into the BBN analysis.

THE MODEL

4.2.6 STEP 7: ASSESS IMPACT LIKELIHOOD [CONSTRUCT THE BBN]

In this study, a Bayesian Belief Networks (BBN) are generated illustrating the dependencies and

potential impacts of successful penetrations targeting CIs. The BBN is used here for its ability to

account for uncertainty and limited available data of CI probability of attacks and

interdependency/impact data. In general, BBNs can be used as visual representations of physical

and/or logical access into and within an enterprise, information system network, CI, or dependent

resource.

A BBN attack graph is provided at its highest level of nodal hierarchy, mapping the CI

interdependencies of an Healthcare and Public Health (HPH) CI sector element (e.g., hospital).

Figure 5 is offered as a simplified example to help understand and elucidate this research’s use of

BBN. The fundamental question for analysis is “what is the probability of a hospital patient’s

degraded health (with the possibility of death) given an attack on the power plant and/or the water

facility on which the patient ultimately depends?” This is determined by identifying through expert

judgment or historical data the marginal probability of an attack on nodes B, C, and D; and

subsequently calculating the joint probability to determine the potential impact to node E (using

Bayes Theorem). Upon calculating the probability of an attack on the hospital, that assessment is

propagated toward computing the probability of its associated links (F and G). The joint probability

of nodes F and G is then calculated to assess the vulnerability of H, the hospital patient’s degraded

health or death. Netica BBN software uses the Joint Tree algorithm to make inferences that

propagate calculated probabilites to adjoining nodes.

The assessment of each node is iteratively improved given updated data/knowledge for

that node, thus continuously reducing the uncertainty and adding more information fidelity for

decision making. Although this study considers each CI as a black box, it should be noted that one

will increase the fidelity of a node by examining the hierarchy within each node. Internal to each

node one would consider an aggregate of factors such as security measures currently in place,

historical attack data that may be available for that node, etc. – ultimately providing additional

insight into the “strength” weight or security effectiveness level for that node – assessing it to be

less/more vulnerable to penetration.

Figure 5. Simplified BBN

For this study, the proof of concept is demonstrated by using historical and relative notional data

(prior probabilities), while the unknown values are calculated through the Bayesian software

simulation tool Netica v5.15 to infer the CI interdependency impacts to dependent resources.

The case is considered where essential interdependent CIs to the hospital have performed

their risk-assessment of security effectiveness per the CI-HHM provided and have scored

themselves accordingly. The model structure and preliminary results were vetted by medical

professionals to confirm the model and various scenarios correctly represented their expectations.

The Conditional Probability Tables (CPT) and prior probabilities used in the hospital patient

healthcare BBN model were generated using data obtained from combined sources: medical

professionals, NIPP, DoE, DHS, Verisign, SSP, Verizon Breach Reports, Symantec, and EPA

along with best estimates and theoretical data from scenario analysis. Illustrated in the Figure 7

BBN, are various security effectiveness states of the CIs, given a successful DoS Attack or

No_Attack (on Power CI), resulting in either No_Effect, Partial_Shutdown, or Complete_Shutdown.

The following definitions provide further understanding of the “what-if” scenario analysis model:

No_Effect is defined as an event/attack having no significant impact/disruption, while a

Partial_Shutdown indicates that the main source has been impacted and services are only being

provided by the backup or a temporary alternate source. A Complete_Shutdown indicates that

both the main source and the backup are no longer providing service.

In Figure 7, the basic structure of the BBN with links is illustrated from the interdependent CIs

and their association to the hospital, subsequently noting the links from the hospital to the medical

devices on which a patient may be dependent to sustain life. Also noted, the network with nodes

that indicate a successful Attack or No_Attack on a CI, highlighted in red.

Figure 6. Representation of BASE m2d Model of DoS attack on CIs (w/o SSEI)

The CI_DoS Attack nodes in in Figure 6 are shown in red. Upon compilation, the dependent

nodes are then calculated to assess the potential impact to the hospital and subsequently to the

patient. The patient is assumed to be in the ICU and totally dependent on the medical device

(dialysis, ventilator, etc.). This model structure allows one to assess the impact to either the

hospital or patient target node of interest. Additional nodes can be added, ultimately adding to the

complexity of the interdependent nature of the CI, enterprise or organization.

The estimated risk-assessment from each CI is now folded into the SSEI nodes indicating the

CI strength or weakness. Until the SSEI rating/score is entered, the BBN model assumes a

uniform distribution, essentially stating that the score is unknown at the time of compilation. Upon

knowing/evaluating the respective CI SSEI or posture, that value is inserted as a Finding or as

Evidence in the model. However, the other CIs will be estimated or varied to perform an

appropriate “what-if” scenario analysis. As more evidence or data is known or observed, the

Water

No_Effect Partial_Shutdown Complete_Shutdown

100 0 0

Power

No_Effect Partial_Shutdown Complete_Shutdown

100 0 0

Communications

No_Effect Partial_Shutdown Complete_Shutdown

100 0 0

Hospital

None Limited_Service No_Service

100 0 0

DoS_Attack_Power

No_Attack Attack

100 0

DoS_Attack_Water

No_Attack Attack

100 0

Ventilator

None Degraded Out_of_Service

100 0 0

Dialysis_Machine

None Degraded Out_of_Service

100 0 0

DoS_Attack_Comm

No_Attack Attack

100 0

Ventilator_Patient

None Degraded Critical

100 0 0

Dialysis_Patient

None Degraded Critical

100 0 0

results of the model are improved. Performing this step results in estimating the likelihood of

impact to a patient given a cascading failure from a DoS attack on a CI.

Figure 7. Representation of SSEI Analysis given DoS Attack on Power CI

BASE m2d allows the decision-maker to assess from CI or dependent resource, i.e. “illustrate how

the CI’s SSEI strength/weakness potentially impacts the hospital or the patient.” An operator could

go further to assess or determine “what is the minimum SSEI one could have so as to not have a

critical impact?” The scenarios to be evaluated are numerous, each potentially enabling the

decision-maker to make more informed and proactive improvements for better protections.

Water

No_Effect Partial_Shutdown Complete_Shutdown

15.9 40.9 43.2

Power

No_Effect Partial_Shutdown Complete_Shutdown

47.9 31.3 20.8

SSEI_Water

Excellent Good Fair Poor

0 0 0

100

SSEI_Power

Excellent Good Fair Poor

16.7 25.0 50.0 8.33

SSEI_Communications

Excellent Good Fair Poor

0 0 0

100

Communications

No_Effect Partial_Shutdown Complete_Shutdown

39.8 28.9 31.3

Hospital

None Limited_Service No_Service

31.8 22.7 45.5

DoS_Attack_Power

No_Attack Attack

0 100

DoS_Attack_Water

No_Attack Attack

50.0 50.0

Ventilator

None

Degraded

Out_of_Service

17.3

23.2

59.5

Dialysis_Machine

None

Degraded

Out_of_Service

37.6

19.2

43.2

DoS_Attack_Comm

No_Attack Attack

50.0 50.0

Ventilator_Patient

None

Degraded

Critical

17.3

23.2

59.5

Dialysis_Patient

None

Degraded

Critical

37.6

19.2

43.2

Figure 7 illustrates how the use of SSEI HHM (blue nodes) assessments can be used to make

decisions to ultimately improve a CI element’s security posture. Upon determining the SSEI score

from the risk-assessment, the CI-HHM BBN model was used to evaluate impacts to a CI facility,

dependent hospital, or patient given a successful DoS. Figure 7 also illustrates how the Water CI

and Communication CI have experienced a successful penetration, which yields a “Poor” relative

effectiveness index. The model indicates that given a “Poor” index, the probability of

Complete_Shutdown on Water and Communications is 43.2% and 31.3%, respectively, given a

successful DoS attack on Power (due to their interdependence). Additionally, the impact to the

hospital is noted as having a 45.5% probability of providing “No_Service” to dependent resources

given the CI vulnerabilities. Subsequent actions by the hospital to improve the index should result

in a lower likelihood of providing “No_Services” to the hospital.

Further analysis revealed a 43% probability of potential “critical” impact to the patient. A

“critical” impact score to the patient indicates a life-threatening result due to the grave nature of the

patient’s condition and their total dependence on the medical device that is providing services.

Given an unsatisfactory potential patient impact score, the SSEI risk-assessment (difference

scores) that was previously performed is re-assessed. This re-assessment specifically seeks to

target/improve security measures that would strengthen or reduce the impact to the target goal

(patient). This evaluation is an iterative process with the goal of continual, targeted, security

effectiveness improvement.

Decisions can be made given various trades from the scenario analysis, taking into account

the security goal, risk priorities, and budget constraints. The “difference scores” indicated in the

SSEI table allow the CI operator to strategically target specific areas of improvement for better

protections.

4.2.7 STEPS 8-10: DECISION ANALYSIS

The decision-maker has now examined, based on his security goals, “What can go wrong and its

impact?” (steps 1-6), “How likely is it to go wrong?” (step 7), and “What are the possible

outcomes?” (Steps 1-7). From Step 7, we have learned the likelihood of impact to the patient in

the event of a CI DoS attack. Step 8 -10 uses this information to make strategic decisions based

on relevant goals. Upon estimating the risk to the patient, as categorized in the model (Step 8), the

decision-maker proceeds to Step 9 to determine if the risk is acceptable to sustain life until

alternative protective or resilient measures take effect. Step 10 responds to the respective answer:

If “yes,” re-assess periodically for new threats or assessments; If “No,” implement new/additional

measures based on self-assessment risk analysis (difference scores).

The BASE m2d methodology seeks to empower emergency management personnel with

the ability to make risk-informed decisions given the realities of the world we live in today –

uncertainty; sophisticated adversaries; limited data; and adaptive, complex, and interdependent

infrastructure. With the insight gained from using the multi-dimensional approach, we have

potentially uncovered otherwise hidden areas of risk. Coupled with BBN, the user can

operationalize the stakeholder information to make strategic decisions.

CHAPTER 5 - DATA ANALYSIS AND RESULTS

5.1 ANALYSIS OBJECTIVES

This section describes the analysis of data gathered and presented by this research. Data analysis

consisted of analyzing demographic, descriptive data, inferential data, response data, and reliability

data (i.e., Cronbach alpha). Statistical software (i.e., Minitab, SPSS and Netica) was employed to

perform the necessary analysis. There were two main objectives identified for the analyses

performed in this research: 1.) Reliability and Validity of Survey Instrument; 2.) Validity of

Conceptual Framework. Ten (10) respondents/representatives from ten distinct hospitals

participated in the online survey. The analysis revealed internal reliability and validity of the survey

instrument and the framework. The findings of each are detailed in the following sections.

5.2 DEMOGRAPHIC DATA

Demographic data were collected on respondents on specific job function in the medical

profession, with options of physician, nurse practitioner, or administrator. Research revealed that

any of these professions would provide relevant data regarding the objectives of this study. Years

of experience and experience in ICU data were also collected.

Thirteen hospitals were solicited for expert knowledge on risk management, metrics,

procedures, and policies regarding CI service interruption impacts to patients. Ten hospitals

responded by completing a 60-question electronic survey, to include comment sections of

voluntary information allowing for elaboration of selected answers or to provide additional details.

A majority (7 of 10) of the survey respondents were tenured medical professionals (10 years or

more experience in a medical field). All respondents were identified as personnel experienced and

qualified to speak on behalf of their hospital.

Figure 8. Survey: Years of Experience in Medical Field

Figure 9. Survey: Medical Field Profession

Figure 10. Survey: Experience in Intensive Care Unit (ICU)

In this research, experts are loosely defined as a nurse, physician, or administrator knowledgeable

in the hospital’s risk management practices, and having a clinical understanding of patient impact

given specific service interruptions. Respondents experience in ICU was relevant to understand

their appreciation of emergency equipment operations and significance to patient’s survival.

Although ten experts were surveyed, five experts are usually sufficient for most elicitation efforts.

Clemen and Winkler (1985) discussed that data saturation can occur and that diminishing returns

have been observed when including additional experts.

5.3 METRICS

Basic and publicly available metric data was elicited from respondents for two purposes: 1)

understand knowledge of medical professional staff of hospital protocols and procedures regarding

measures used to trigger alternate systems, and 2) Calibration of expert judgment information, if

desired. Questions and responses are summarized below.

What metric is used to trigger backup POWER generator?

What metric is used to trigger backup WATER supply?

What metric is used to trigger alternate COMMUNICATIONS?

In summary, it is interesting to note elicited responses regarding knowledge of metrics used: 50%

were not aware of metrics used to trigger alternate power sources or generators; a disparity in

knowledge existed for metrics used for water; and more than 70% indicated they had no

knowledge of alternate measures or metrics for communications given a service outage.

As indicated in this research, metrics are imperative to demonstrate emergency preparedness

for the public healthcare community; just as important is assessing those measures periodically

and ensuring that the staff is aware and trained to respond accordingly in the event alternate

comms are needed.

5.4 DESCRIPTIVE DATA

Data were collected from experts on likelihood of CI service disruption impacts to patients

depending on the following medical devices: cardiac equipment, dialysis systems, and

oxygen/ventilators. Ten medical professional respondents from different hospitals provided expert

judgment on likelihood of impact (critical, degraded, no impact) given a CI (power, water, comms)

service disruption.

A summary of analysis was performed on data collected from the respondents. The data was

grouped by CI service disruption (power, water, comms) against likelihood of impact (critical,

degraded, no impact) to patients depending on various medical equipment (cardiac care, dialysis,

ventilator). The results were used to validate the behavior of the BBN model.

The data was collected from the respondents used a 5 point Likert scale that correlated to

probability distributions as follows:

1 2 3 4 5

Very unlikely Unlikely Need More Info Likely Very Likely

(0-20%) (21-40%) (41-60%) (61-80%) (81-100%)

Power Disruption Impact: Respondents were mostly consistent in their responses regarding

the likelihood of “critical” impact of disrupted power to medical devices. Respondents tended to

agree (between likely and very likely) that any disruption of service to power (limited or complete

shutdown) would have a higher likelihood of having a “critical” impact on a patient depending on

dialysis, cardiac care, and/or ventilator. More variance was noted among respondents in regards

to assessing “degraded” impact to patients given a service disruption. Respondent ability to

assess “degraded” impact to patient given power disruption exhibited a large variance, indicating

responses in this category to be inconclusive. Nevertheless, it was noted that fifty percent of the

respondents selected “need more info” to determine the likelihood of impact to a cardiac care

patient given limited power. Respondents tended to agree that “no impact” to a patient would be

less likely (between very unlikely and unlikely) given service disruption in power to any of the

medical devices. This was consistent with the BBN model results.

Water Disruption Impact: Respondents were mostly consistent in their responses regarding

the likelihood of “critical” impact of disrupted or contaminated water supply. Respondents tended to

agree (between likely and very likely) that any disruption of service to water (limited or complete

shutdown) would have a higher likelihood of having a “critical” impact to a patient depending on

dialysis, cardiac care, and/or ventilator. Respondents had a general agreement and a tendency

toward unlikely “degraded” impact to patients given water disruption, although a variance spread

was dually noted in this category. A stronger agreement among respondents (between very

unlikely and unlikely) was noted in assessing likelihood of “no impact” to patients given a water

disruption/contamination. This was consistent with the BBN model results.

Communication Disruption Impact: Respondents were generally consistent in their

responses regarding the likelihood of “critical” impact of disrupted comms unlikely to very likely.

Meaning, each respondent was consistently explicit that disrupted comms would have some

measure of impact to patient care. This category may have been less intuitive to respondents

which is indicated by the disparity in their response. Comms is represented here as the electronic

response system used to communicate with emergency staff to provide the status of patients

connected to a medical device, such as an alarm to notify service has been disrupted. If

communication systems are down, medical professionals may be unaware that help is needed and

services are no longer being provided or have been degraded. Respondents tended to agree

(between unlikely and very unlikely) that any disruption of service to comms (limited or complete

shutdown) would have a higher likelihood of having a “degraded” impact to a patient depending on

dialysis, cardiac care, and/or ventilator. More variance was noted among respondents in regards

to assessing “no impact” to patients given a comms service disruption, indicating more info would

be needed to determine increase or decrease in “no impact” likelihood. This was consistent with

the BBN model results.

5.5 RELIABILITY AND VALIDITY OF SURVEY INSTRUMENT

To determine if the results of the survey instrument are reliable and valid, respondent data is

evaluated against the data produced by the tool (instrument). Specifically, did the survey measure

what it was intended to measure. In review, the instrument was evaluated as follows:

RELIABILITY ▪ INTERNAL CONSISTENCY or Homogeneity using a Cronbach coefficient (α >0.7)

o Measures the extent to which a technique, experiment, or measuring procedure

assess the same characteristic or quality

▪ INTRARATER RELIABILITY

o Measures the extent to which multiple respondents agree in their ratings of given

items.

Results revealed that the survey instrument tested reliable in its ability to assess the impact to a

patient given a service disruption in power, water, and communications. Each category

demonstrated a Cronbach alpha as follows:

➢ Critical α = 0.936

➢ Degraded α = 0.924

➢ No impact α = 0.931

These results indicate the strong internal consistency of the tools’ ability to measure the

aforementioned attributes.

To determine the intrarater reliability, the Intra-class Correlation Coefficient (ICC) was used.

This statistical measure the proportion of variance of an observation due to between-subject

variability in the true score (Fink 2003).

➢ Critical ICC = 0.932

➢ Degraded ICC =0 .929

➢ No impact ICC = 0.925

The results indicate consistent responses between the respondents.

VALIDITY

▪ CONTENT VALIDITY o Measures the extent to which the question/items thoroughly and appropriately

assess the characteristics or qualities it purports to measures.

▪ FACE VALIDITY

o Measures the appearance of the metric/question on the surface.

Content validity is a subjective measure of how appropriate items or scales seem to a set of

reviewers identified as Subject Matter Experts (SMEs) (Litwin 2003). It typically involves an

evaluation of the survey’s content to ensure it includes relevant information or questions. Content

validity is not quantified with statistics (Litwin 2003). Instead it is an opinion from subjective

experts. The survey was reviewed by administrators of each hospital for relevance before

appointing an appropriate Respondent. The survey was deemed relevant and appropriate for its

purpose.

Face Validity is determined by an observer with an untrained eye and is considered a casual

assessment of the appropriateness of the survey instrument (Litwin 2003).

5.6 VALIDITY OF CONCEPTUAL FRAMEWORK

A major goal of this study was to develop a model that is valid in assessing the ultimate purpose of

evaluating system security effectiveness of CIs and dependent resources, given noted

interdependencies. Events and actions that occur from a dependent or linked CI have the ability to

influence the likelihood of occurrence and impact on an adjoining dependent resource. It is

demonstrated here that this phenomena can be modeled using BBN nodes with various alternating

states. The direction of influence is identified with parent-child or CI-hospital and/or hospital-

medical device references as required by one of the rules of BBN using Directed Acyclic Graphs

(DAG). Although the direction of influence and dependence often flows from parent to child,

inferences can be made in the opposite direction. The model learns from this data and can

generate Conditional Probability Tables (CPTs) that are often used to evaluate the model

performance. The CPTs used in this research were generated from historical and relational data.

As mentioned previously, this conceptual framework is validated with the use of data identified in

the Data Collection section and correlated with data provided by experts.

5.7 THREATS TO INTERNAL VALIDITY

Threats to validity for this research include the following:

• Internal validity demonstrates a causal relationship between variables. Medical

professionals are influenced (bias and overconfidence) by many factors, and this is often

reflected in their responses. The approach used to mitigate this threat to internal validity

was to take care in training or calibrating the expert with the use of mixed method

techniques (as described in the Research Methodology section). Respondents were also

allowed to provide open comments to elaborate or ask questions, if needed. Clear

instructions and training are imperative to mitigating this threat.

• Reliability of the measurements can threaten validity. The data collection instruments

included: knowledge questions to establish the credibility of the respondent, consistent

measurement scales, clear and unambiguous questions, concise and efficient

questionnaire design, and sufficient time to fill-out the questionnaire (Babbie, 1990,

Sudman and Bradburn, 2004; Valerdi, 2005).

5.8 THREATS TO EXTERNAL VALIDITY

External validity refers to the ability to apply the research results to other contexts or domains.

The external validity of this study extends beyond CIs and hospitals. Another application within the

space community is to evaluate potential impacts to satellites given a successful attack to

interdependent ground stations. The general concept has extensibility to assess and evaluate how

a successful threat to a third party’s vulnerability can impact a primary or secondary source and

vice versa.

CHAPTER 6 – CONCLUSION

This research establishes a systematic, multi-dimensional, quantitative, metric-based framework

required to effectively assess, measure, and ultimately improve an organization’s security posture.

The utility of multi-dimensional models such as HHM and BBN and applying them to CI cyber-

physical security protection and resiliency is examined.

To achieve greater security effectiveness for CIs, a systematic approach, using Systems

Engineering principles, within the operational environment by first identifying security goals and

objectives based on what is to be protected was suggested. Second, identify all paths that lead to

the CIs that need protecting, including internal and external access and dependencies (via people,

processes, technology, infrastructure, i.e., the entire enterprise). Third, articulate what security

means are implemented from the perspective of all stakeholders. In the absence of this type of

systematic approach, organizations, enterprises, and infrastructures will remain vulnerable. This

research postulated that the understanding and influence of the strengths/weaknesses of the

enterprise are enhanced with the use of inferential statistics coupled with multi-dimensional,

metrics-based analysis. It also is important to position security professionals and other key decision

makers to measure the effectiveness of implemented security controls. Further, this framework

and methodology empowers leaders to make more informed decisions as to where resources

should be focused for more effective security.

The methodology used to address this complex problem of improving cyber-physical

security protection of the enterprise, or specifically in this case the nation’s CIs, must expand

beyond traditional approaches used today. Further, organizations, CI owners, and operators must

have the ability to assess the effectiveness of the security program they put in place. Measures of

effectiveness (MOE) identified in the proper context (based upon security goals and the operational

environment), using relevant performance metrics for proactive monitoring and vulnerability

assessment, can be used to not only understand security posture but also to highlight deficiencies

that would otherwise prevent the making of informed decisions toward the implementation of more

cost-effective solutions.

6.1 CONCLUSION WITH RESPECT TO STUDY HYPOTHESES

In review, the following hypotheses were asserted:

It was shown that HHM (systems-thinking based framework) and BBN (probabilistic reasoning)

could be combined to demonstrate the strength or weakness of CI security effectiveness by

implementing the HHM qualitative component to evaluate various elements of the CI from multiple

dimensions, and using the BBN module of the framework to dynamically assess the potential

impact of a threat in accordance with available data and identified dependencies. It was further

determined that the hybrid model demonstrated the ability to quantify the effectiveness of

protection and resiliency for CIs with the use of metrics, the SSEI, and the inherent nature of BBNs.

6.2 CONCLUSION WITH RESPECT TO STUDY QUESTIONS

In review, the following questions were asserted for this research:

Question #1: How can the assessment of security effectiveness of CI interdependencies and

vulnerabilities be modeled proactively (before occurrence or penetration) for improved decision-

making?

The use of BBNs allowed for the most appropriate assessment of indirect consequences of a direct

attack. The very nature of BBN development is to establish hierarchical dependencies, often to

multiple orders (3rd, 4th order, etc.). It was demonstrated through scenario analysis within the

framework and a hospital case study that a successful DoS attack on a CI (power, etc.) could be

modeled proactively (before occurrence or penetration) for improved decision-making

Question #2: How can combining HHM and BBN facilitate the assessment of relative

effectiveness of CI security measures?

The BASE m2d framework integrated both qualitative (HHM) and quantitative (BBN)

modules/components to exercise the SSEI concept, allowing a CI owner/operator the ability to self-

assess the current security posture of their infrastructure based on their security goals and

tolerance. This self-assessment with the conceptual framework enables the CI to ultimately make

improvements to prevent or mitigate (direct and indirect) negative impacts.

Question #3: What metrics are used by hospitals to trigger auxiliary systems in the event of a

shutdown (partial or complete)? What metrics, if any are used to warn, prevent, or absorb CI

service interruptions?

As a result of the Respondents survey, it was surprisingly noted that many medical professionals

were unaware of any metrics used to trigger auxiliary systems in their respective hospitals.

6.3 DISCUSSION

In this research, the question “how secure am I?” is not asked. Although a relevant question, a

more realistically attainable and measurable question is posed– “how effective are the security

measures I have put in place?” More specifically, “what systematic construct or metrics have we

employed to assess that effectiveness in accordance with our value proposition or security goals?”

For example, if you have removed unnecessary accesses (via physical hardware or personnel

(root access)), you have essentially reduced your attack surface; in turn, you can measure the

effectiveness of that security implementation. Although many organizations work diligently to

ensure they are in compliance with the latest standards, it has been proven by our adversaries that

being in compliance does not guarantee security.

To effectively achieve the stated research goals, this research proposed the integration of

two hierarchical techniques to enable a qualitative and quantitative assessment of security

measures. This framework provides a risk assessment via scenario or what-if analysis to inform

the decision-maker on where best to allocate proper resources that maintain a specific and

appropriate level of service to preserve human life in the event of a CI failure due to an attack

vector (partial shutdown or complete shutdown). In addition, it is asserted that metrics should be in

place to account for minimum mean time between failures to ensure resiliency measures are

effective. For example, a back-up generator must be triggered within a certain timeframe to ensure

a ventilator patient does not asphyxiate due to oxygen deprivation.

Additionally, the significance of attack graphs and understanding attack paths to protect an

enterprise security goal is a strategic and important step in effective and efficient use of resources.

Attack paths/graphs are to be produced from various scenarios within the CIs’ operational

environment, to include interdependencies with other CIs. BBNs are then generated from each

attack graph to identify potential areas of vulnerability. HHM has been used to identify dimensional

variables (temporal, geo-spatial, legislative, societal, stakeholders’ perspective) from which

strengths and weaknesses of each CI node are to be evaluated. Finally, a simplified BBN threat

scenario is provided that is used to demonstrate and evaluate the CI security effectiveness per the

identified metrics. The overall methodology is iterative to incorporate new data or knowledge (via

BBN), to produce results with greater accuracy and fidelity; this construct is scalable and extensible

in that this method can be used for the largest CI to the smallest architectural element (network or

enterprise) and various CI domains; and a general process is provided for repeatability.

This study expands on knowledge, experience, and recommendations offered by previously

documented research. The hybrid methodology assesses the potential impacts to patients,

medical devices and the emergency preparedness of a healthcare facility given a CI service

disruption. Additionally, it provides a holistic, strategic, quantitative, and measurable approach that

permits one to perform relative security effectiveness analysis in the face of uncertainty and where

limited data prevents necessary measures from being executed to ensure proper protection and

resiliency of the enterprise.

I bound and strategically target the problem-solution space by defining the operational

environment and identifying the security goals that are to be protected. This research

postulates that the understanding and influence of the strengths/weaknesses of the enterprise are

enhanced with the use of inferential statistics coupled with multi-dimensional, metrics-based

analysis. Standardized use of the SSEI would allow CIs and/or key resources to maintain and

communicate a certain “acceptable” index that merits or ensures an agreed-to level of service to its

dependent resources.

It is acknowledged that current methodologies used to address the complex problem of

improving cyber/physical security protection of the enterprise, or specifically in this case the

nation’s CIs, must expand beyond existing traditional approaches. Most security methods used

today are considered from a single-dimension. This is normally accomplished by protecting

architectural elements or components (network of routers, switches, servers) from cyber-attacks,

i.e., by configuring firewalls, implementing policies, and training users. Although these techniques

demonstrate a noble effort, they have proved to be neither sufficient nor effective. Further,

organizations, CI owners, and operators must have the ability to assess the effectiveness of the

security program they put in place. Measures of effectiveness (MOE) identified in the proper

context (based upon security goals and the operational environment), using relevant performance

metrics for proactive monitoring and vulnerability assessment, can be used to not only understand

security posture but also to highlight deficiencies that would otherwise prevent the making of

informed decisions toward the implementation of more cost-effective solutions. Finally, this

approach allows organizations, CI owners, and operators the ability to assess and reduce

vulnerabilities within and across the security domain to which they interface and are responsible.

CHAPTER 7 - FUTURE RESEARCH

The SSEI scale proposed in this research could be improved, verified, and validated by CI

owners/operators and further assessed against previous cases of enterprises or CIs that have

experienced DoS attacks to test predictability and reliability. This research also recommends that

for a complex, interdependent system of CIs to effectively provide critical services to dependent

resources, more effective and efficient standards need to be implemented. Standards that

mandate CIs communicate (share information) across CIs and to those that depend on their

services, in accordance with a security effectiveness taxonomy understood by the impacted

community of stakeholders. The SSEI concept introduced in this research, if further developed,

would allow CIs to communicate in a common language at a confidential or sensitive information

level, if necessary. This would allow appropriate security measures to be considered from the

perspective of their own internal security effectiveness evaluation, as well as the external

interdependent sources security effectiveness levels/evaluation.

This research has laid the ground work for future studies in the area of understanding,

assessing, and improving system security effectiveness for CIP/R. The following

recommendations are offered as potential areas of research:

• The Systems Security Effectiveness Index can be furthered developed by generating a

more relevant scale of measurement. This research provided a general scale to

demonstrate the concept of the hybrid framework by equally dividing each category. For

example, further research can determine if a scoring range of .95 – 100 percent should be

considered ‘Excellent’, or a range of 0 - 0.50 should be considered ‘Poor’ security

effectiveness.

• The CI-HHM provided in this research established an equal scale for the variables

identified from the NIPP. Additional research could establish priorities for a specific CI

function (Water, Power, etc.) or specific enterprise in question and place relevant weights

using models such as AHP or MAUT.

• This conceptual framework was demonstrated using a hospital case study however, the

researcher believes the concept is extensible to other disciplines with varying threats

• Establish an SSEI ontology that can be generalized to a relevant community and

mandated to be shared.

REFERENCES

Adar, E., & Wuchner, A. (2005). Risk management for critical infrastructure protection (CIP)

challenges, best practices, & tools. Paper presented at the 8 pp.

doi:10.1109/IWCIP.2005.18

Alden, J. (2006) Measuring the “unmeasurable.” Perf. Improv., 45: 7–11.

Alqahtani, Abdulrahman, (2015) "Towards a Framework for the Potential Cyber-Terrorist Threat to

Critical National Infrastructure." Information and Computer Security 23 (5): 532-569

Annex, A., (2010) Infrastructure, N., & Plan, P. Healthcare and Public Health Sector-Specific Plan

Amass F. S., Bhunia, A. K, Chaturvedi, A. L., Dolk, D. R., Peeta, S., and Atallah, M. J. Advances in

Homeland Security Series: The Science of Homeland Security (Volume 1), West Lafayette, IN:

Perdue University Press, 2006 227 pp, introduction, notes, bibliography, index (hardcover)

Assessment of Key Risks for Hospitals and Healthcare Systems – Spring 2010,

<http://www.kpmg.com/US/en/IssuesAndInsights/ArticlesPublications/Documents/heal

thcare--‐risk--‐assessment--‐spring--‐2010.pdf>

Ayyub, B. M. (2001). Elicitation of expert opinions for uncertainty and risks, CRC Press.

Babbie, E. (2010). The basics of social research (5th ed.). Belmont, CA: Wadsworth Publishing.

Bayraktarli, Y. Y., J. Ulfkjaer, et. al.. (2005) On the application of Bayesian probabilistic networks

for earthquake

Bayuk , Jennifer and Ali Mostashari. (2013) "Measuring systems security." Systems Engineering: 1-

14.

Bensi, Michelle Terese. (2010) A Bayesian Network Methodology for Infrastructure Seismic Risk

Assessment and Decision Support

Biringer, B., E. Vugrin, et. al.. (2013). Critical infrastructure system security and resiliency, CRC

Press.

Bloomfield R , Chozos N , Nobles P . (2009) Infrastructure interdependency analysis: introductory

research review. Adelard LLP;

Boring, R., Gertman, D., Joe, J., Marble, J., Galyean, W., Blackwood, L., & Blackman, H. (2005).

Simplified Expert Elicitation Guidelines For Risk Assessment Of Operating Events.

(INL/EXT-05-00433). Idaho Falls, Idaho: Department of Energy, U.S. Nuclear Regulatory

Commission, Office of Nuclear Regulatory Research, Division of Risk Analysis &

Applications.

Burnside ES, Rubin DL, Shachter RD. (2004) Using a Bayesian Network to Predict the Probability

and Type of Breast Cancer Represented by Microcalcifications on Mammography” in:

Fieschi, M., Coiera, E., and Li, Y.J., Eds., Medinfo 2004, Proceedings of the 11th World

Congress on Medical Informatics, Sept. 7-11, 2004, IOS Press, 13-18.

Carmines, E. and R. Zeller, Reliability and Validity Assessment. Quantitative Applications in the

Social Sciences, ed. M.S. Lewis-Beck. 1979, Thousand Oaks, California: SAGE

Publications.

Chittister, Clyde C., and Yacov Y. Haimes. (2012) "Risk to cyberinfrastructure systems served by

cloud computing technology as systems of systems." Systems Engineering 15: 213-224.

Clemen, R. T. and R. L. Winkler (1985). "Limits for the Precision and Value of Information from

Dependent Sources." Operations Research 33(2): 427-442.

Cooke, R.M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science

Cooke, R. M., & Goossens, L. H. (2004). Expert judgement elicitation for risk assessments of

critical infrastructures. Journal of risk research, 7(6), 643-656.

Cronbach L. J. (1971) "Test Validation," R.L. Thorndike (ed.) Educational Measurement

Washington, DC: American Council on Education

Department of Homeland Security, http://www.dhs.gov/healthcare-and-public-health-sector

Di Giorgio, Alessandro, and Francesco Liberati. (2011) "Interdependency modeling and analysis of

critical infrastructures based on Dynamic Bayesian Networks." In Control & Automation

(MED), 2011 19th Mediterranean Conference on, pp. 791-797. IEEE.

Dimase, D., Collier, Z. A., Heffner, K., & Linkov, I. (2015). Systems engineering framework for

cyber physical security and resilience. Environment Systems & Decisions, 35(2), 291-300.

Druzdzel, MJ and Van der Gaag, LC, 1995 “Elicitation of probabilities for belief networks:

combining qualitative and quantitative information,” Proceedings of the Eleventh Conference

on Uncertainty in Artificial Intelligence 141–148

Eusgeld, I., C. Nan, S. Dietz, (2011) ‘‘System-of-systems” approach for interdependent critical

infrastructures, Reliability Eng. Syst. Saf. 96 679–686.v

Eusgeld, I., D. Henzi, et. al.. (2008). "Comparative evaluation of modeling and simulation

techniques for interdependent critical infrastructures." Scientific Report, Laboratory for Safety

Analysis, ETH Zurich.

Executive Order 13636 (2013) Improving Critical Infrastructure Cybersecurity, President Barack

Obama

Executive Order 13010 (1996). Critical Infrastructure Protection. Federal Register. Vol. 61. No.

138.

Executive Order 13130 (1999). National Infrastructure Assurance Council, Federal Register, Vol.

64.

Executive Order 13231(2001) Critical Infrastructure Protection in the Information Age. Federal

Ezell, B. C. (2007). Infrastructure Vulnerability Assessment Model (I‐VAM). Risk Analysis, 27(3),

571-583.

Fink, A. (2003). The survey handbook, Sage.

Fink, A. (2003). The Survey kit:How to manage, analyze and interpret survey data. Thousand

Oaks, CA: SAGE Publications Ltd doi: 10.4135/9781412984454

Fink, A. (2003). The survey kit: How to manage, analyze, and interpret survey data, Thousand

Oaks, CA: Sage Publications, Inc.

Flandoli, F., E. Giorgi, et. al. (2011). "Comparison of a new expert elicitation model with the

Classical Model, equal weights, and single experts, using a cross-validation technique."

Reliability Engineering & System Safety 96(10): 1292-1310.

Forsberg JA, Eberhardt J, Boland PJ, Wedin R, Healey JH. (2011) Estimating Survival in Patients

with Operable Skeletal Metastases: An Application of a Bayesian Belief Network. PLoS

ONE 6(5): e19956.

Forsberg, J. A., Healey, J. H., & Brennan, M. F. (2012) A probabilistic analysis of completely

excised high-grade soft tissue sarcomas of the extremity: an application of a Bayesian

belief network. Annals of Surgical Oncology, 19(9), 2992–3001.

Fowler, F. J. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA:

Sage Publications.

Fowler, F. J. (2009). Survey research methods (4th ed.) Thousand Oaks, CA: Sage Publications.

Frigault, Marcel. (2010) Measuring network security using Bayesian network-based attack graphs.

Ph.D. diss., Concordia University (Canada),

http://search.proquest.com/docview/794810164?accountid=11243

Gass, Saul I. 2005. Model world: The great debate-MAUT versus AHP. Interfaces 35, (4) (Jul):

308-312, http://search.proquest.com/docview/217112431?accountid=11243 (accessed

February 27, 2015).

Ghorbani AA, BagheriE. (2013)The state of the art in critical infrastructure protection : a framework

for convergence. International Journal of Critical Infrastructures 2008;4:215–44.

Grover, Jeff. (2013) Strategic economic decision-making using Bayesian belief networks to solve

complex problems. New York, NY: Springer

Haimes, Yacov Y. (1981) "Hierarchical Holographic Modeling." IEEE Transactions On Systems,

Man, and Cybernetics: 606-617.

Haimes, Yacov Y. and Chittester, Clyde G. (2005) "A Roadmap for Quantifying the Efficacy of Risk

Management of Information Security and Interdependent SCADA Systems," Journal of

Homeland Security and Emergency Management: Vol. 2: Iss. 2, Article 12.

Haimes, Yacov Y. (2004) “Risk modeling, assessment, and management”. Hoboken, N.J.: Wiley-

Interscience

Haimes YY,Horowitz BM, Lambert JH, Santos JR,Crowther KG, and Lian C. Inoperability input–

output model for interdependent infrastructure sectors. II: ase studies. Journal of

Infrastructure Systems,11;80–92.

Haimes, Y.Y., J. Lambert, Duan Li Duan Li, R. Schooff, and V. Tulsiani. (1995) “Hierarchical

holographic modeling for risk identification in complex systems. 1995 IEEE International

Conference on Systems, Man, and Cybernetics”. Intelligent Systems for the 21st Century.

Harrison, Keith and White, Gregory B. (2010) "An Empirical Study on the Effectiveness of Common

Security Measures." Lecture, 2010 43rd Hawaii International Conference on System

Sciences, Koloa, Kauai, Hawaii

Hayes, N. (Ed.) (1998). Doing qualitative analysis in psychology. Hove: Psychology Press.

Hokstad, P., I.B. Utne, J. Vatn,( 2012) Risk and Interdependencies in Critical Infrastructures: A

Guideline for Analysis, Springer.

Hubbard, D. W. (2009). The failure of risk management: Why it's broken and how to fix it. John

Wiley & Sons.

Hubbard, Douglas W. (2010) How to measure anything finding the value of "intangibles" in

business. 2nd ed. Hoboken, N.J.: Wiley.

Hubbard, Douglas W. (2016) How to measure anything in cybersecurity risk. Hoboken, N.J.: Wiley.

http://ics-cert.us- cert.gov/sites/default/files/Monitors/ICS-CERT_Monitor_Oct-Dec2012.pdf

http://www.dhs.gov/xlibrary/assets/NIPP_Overview.pdf

HTTP://WWW.DHS.GOV/NEWS/2013/07/18/WRITTEN- testimony-nppd-house-homeland-security-

subcommittee-cybersecurity

https://www.washingtonpost.com/world/national-security/russia-has-developed-a-cyber-weapon-

that-can-disrupt-power-grids-according-to-new-research/2017/06/11/b91b773e-4eed-11e7-

91eb-9611861a988f_story.html?utm_term=.89e606950738 (2017)

https://www.whitehouse.gov/the-press-office/2017/05/11/presidential-executive-order-

strengthening-cybersecurity-federal (2017)

https://www.healthcare-informatics.com/article/cybersecurity/exclusive-report-what-can-us-

healthcare-it-leaders-learn-wake-wanna-cry (2017)

https://www.cnet.com/news/petya-goldeneye-wannacry-ransomware-global-epidemic-just-started/

(2017)

Jha, Manoj K.(2009) "Dynamic Bayesian network for predicting the likelihood of a terrorist attack at

critical transportation infrastructure facilities." Journal of Infrastructure Systems 15, no. 1:

31-39.

Jaquith, Andrew. (2007) Security metrics: replacing fear, uncertainty, and doubt. Upper Saddle

River, NJ: Addison-Wesley.

Kahan, Jerome H.; Allen, Andrew C.; and George, Justin K. (2009) "An Operational Framework

for Resilience," Journal of Homeland Security and Emergency Management: Vol. 6: Iss. 1,

Article 83.

Kjølle, G. H., I. B. Utne, et. al.. (2012). "Risk analysis of critical infrastructures emphasizing

electricity supply and interdependencies." Reliability Engineering & System Safety 105:

80- 89.

Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model

selection.

Kozik, Rafał, Michał Choraś, and Witold Hołubowicz. (2010) "Fusion of Bayesian and Ontology

Approach Applied to Decision Support System for Critical Infrastructures Protection." In

Mobile Lightweight Wireless Systems. : Springer Berlin Heidelberg.

Kuehn, N. M., Riggelsen, C., and Scherbaum, F. (2009). ―Facilitating Probabilistic Seismic

Hazard Analysis Using Bayesian Networks.‖ Seventh Annual Workshop on Bayes

Applications (in conjunction with UAI/COLT/ICML 2009).

Laconte, P., Y. Y. Haimes, et. al.. (1982). Water resources and land-use planning : a systems

approach: Proceedings of the NATO Advanced Study Institute on: "Water Resources and

Land-Use Planning," Louvain-la-Neuve, Belgium, July 3-14, 1978. The Hague ; Boston

Hingham, MA, USA, M. Nijhoff ; Distributors for the U.S. and Canada, Kluwer Boston.

Laskey, K. B., & Mahoney, S. M. (2000). Network Engineering for Agile Belief Network Models,

12(4), 487–498.

Lewis, T. G. (2006). Critical infrastructure protection in homeland security: defending a networked

nation, John Wiley & Sons.

Liberati, A. D. G. a. F. "<Bayesian Network-Based Approach to the Critical Infrastructure

Interdependencies Analysis.pdf>." IEEE SYSTEMS JOURNAL VOL. 6, (NO. 3): 510-519.

Little, Richard G. (2003) "Toward More Robust Infrastructure: Observations on Improving the

Resilience and Reliability of Critical Systems." Lecture, System Sciences. Proceedings of

the 36th Annual Hawaii International Conference on, Big Island, Hawaii.

Litwin, M. S. and Fink, A. (2003). How to assess and interpret survey psychometrics, Sage.

Mahoney, SM and Laskey, KB, 1996, “Representing and combining partially specified CPTs”

Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence 391–400.

Marcot, B. G., Steventon, J. D., Sutherland, G. D., & Mccann, R. K. (2006). Guidelines for

developing and updating Bayesian belief networks applied to ecological modeling and

conservation

McGee, Sibel, Jaime Frittman, Seongjin James Ahn, and Susan Murray. 2016. Implications of

cascading effects for the hyogo framework. International Journal of Disaster Resilience in

the Built Environment 7, (2): 144-157

Min, H.-S. J., W. Beyeler, et. al.. (2007). "Toward modeling and simulation of critical national

infrastructure interdependencies." Iie Transactions 39(1): 57-71.

Netica v5.15, http://www.norsys.com/index.html

National Association of County and City Health Officials (NACCHO), (2014) Cyber Attack on U.S.

Hospital Group Highlights Vulnerability of Critical Infrastructure

http://nacchopreparedness.org/

National Association of County and City Health Officials (NACCHO), (2015) The Role of Local

Public Health in Healthcare Critical Infrastructure Protection

http://nacchopreparedness.org/

National Institute of Standards and Technologies (NIST), (2014) ) Framework for Improving Critical

Infrastructure Cybersecurity

National Infrastructure Protection Plan (2013) http://www.dhs.gov/publication/nipp-2013-partnering-

critical-infrastructure-security-and-resilience

Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.

Ouedraogo, Moussa; Savola, Reijo M.; Mouratidis, Haralambos; Preston, David; Khadraoui,

Djamel and Dubois, Eric. (2013) "Taxonomy of quality metrics for assessing assurance of

security correctness." Software Quality Journal: 67-97.

Ouyang, M. (2014). "Review on modeling and simulation of interdependent critical infrastructure

systems." Reliability Engineering & System Safety 121: 43-60.

Ouyang, M., Dueñas-Osorio, L. (2012). “A three-stage framework resilience analysis framework for

urban infrastructure systems.” Structural Safety 2012; 36: 23–31.

Pearl, Judea. (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference.

San Mateo, Calif.: Morgan Kaufmann Publishers.

Pederson, P., D. Dudenhoeffer, et. al.. (2006). "Critical infrastructure interdependency modeling: a

survey of US and international research."

http://nacchopreparedness.org/?p=2854

Pettigrew, J., Ryan, J., Salous, K., Mazzuchi, T., & Dc, W. (2009). Decision-Making by Effective

Information Security Managers.

Pfleeger, Shari Lawrence and Cunningham, Robert K., (2010) "Why Measuring Security Is Hard."

IEEE Security & Privacy Magazine: 46-54.

PPD-21 (2013) Critical Infrastructure Security and Resilience, President Barack Obama

Punch, K. F. (2003). Survey research: The basics. London, England: Sage Publications.

Queiroz, C.; Mahmood, A.; Tari, Z., (2013) "A Probabilistic Model to Predict the Survivability of

SCADA Systems," Industrial Informatics, IEEE Transactions on , vol.9, no.4, pp.1975,1985

Radvanovsky, R. S. and A. McDougall (2013). Critical infrastructure: homeland security and

emergency preparedness, CRC Press.

Renooij, S. (2001). "Probability elicitation for belief networks: issues to consider." The

Knowledge Engineering Review 16(03): 255-269.

Riegel, C., Risk Assessment and Critical Infrastructure Protection in Health Care Facilities:

Reducing Social Vulnerability, Retrieved March 10, 2013 From

<http://www.ehs.unu.edu/file/get/3796>

Rinaldi , S. m., Peerenboom J. p. and Kelly, T.k. (2001) "Identifying, understanding, and analyzing

critical infrastructure interdependencies." IEEE Control Systems Magazine: 11-25.

Robert, Benoit (2008). Modelling interdependencies among critical infrastructures. International

journal of critical infrastructures. , 4 (4), p. 392. (ISSN: 1475-3219)

Roberts, Steven (2004) "Tips and Trends for Homeland Security and Critical Infrastructure

Protection," Journal of Homeland Security and Emergency Management: Vol. 1: Iss. 4,

Article 405

Ryan, J.J.C.; Ryan, D.J., (2008) "Performance Metrics for Information Security Risk

Management," Security & Privacy, IEEE , vol.6, no.5, pp.38,44.

Ryan, Julie J.c.h., Thomas A. Mazzuchi, Daniel J. Ryan, Juliana Lopez De La Cruz, and & Cooke,

Roger. (2012) "Quantifying information security risks using expert judgment elicitation."

Computers & Operations Research: 774-784.

Ryan, J.j.c.h. (2004) "Information security tools and practices: what works?." IEEE Transactions on

Computers: 1060-1063.

Salkind, N. J. (2009). Exploring research (7th ed.). Upper Saddle River, NJ: Pearson Education.

Sanders, W, (2014). “Quantitative Security Metrics: Unattainable Holy Grail or a Vital

Breakthrough Within Our Reach”, Security & Privacy, IEEE,vol 12 no.2, pp 67-69

Santos, Joost R., Haimes, Yacov Y. and Lian, Chenyang. (2007) "A Framework for Linking

Cybersecurity Metrics to the Modeling of Macroeconomic Interdependencies." Risk

Analysis: 1283-1297.

Santos JR (2006). Inoperability input–output modeling of disruptions to interdependent economic

systems. Systems Engineering pp20–34.

Satumtira G, Dueñas-Osorio L. (2010) Synthesis of modeling and simulation methods on critical

infrastructure interdependencies research. In: Gopalakrishnan K, Peeta S, editors.

Sustainable infrastructure systems: simulation, imaging, and intelligent engineering. New

York: Springer-Verlag.

Sikula, Nicole R., James W. Mancillas, Igor Linkov, and John A. McDonagh. "Risk management is

not enough: a conceptual model for resilience and adaptation-based vulnerability

assessments." Environment Systems & Decisions 35, no. 2 (2015): 219.

SPSS. (2007). SPSS survey tips guide. Retrieved from: http://www.spss.com

Symantec 2010 Critical Infrastructure Protection Study Global Results. (2010), (October).

The White House. (2013) "Executive Order -- Improving Critical Infrastructure Cybersecurity."

http://www.whitehouse.gov/the-press-office/2013/02/12/executive-order-improving-critical-

infrastructure-cybersecurity .

Tversky, A., D. Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science

185(4157): 1124-1131.

U.S. Government Accountability Office. (2011) "Cybersecurity: Continued Attention Needed to

Protect Our Nation's Critical Infrastructure and Federal Information Systems." GAO-11-

463T

U.S. Government Accountability Office. (2012) "Critical Infrastructure Protection: Cybersecurity

Guidance Is Available, but More Can Be Done to Promote Its Use." GAO-12-92

U.S. Government Accountability Office. (2015) "Critical Infrastructure Protection: Measures Needed

to Assess Agencies’ Promotion of the Cybersecurity Framework. GAO-16-152

U.S. Government Accountability Office. (2017) "Critical Infrastructure Protection: Needs to Better

Measure Cybersecurity Progress.” GAO-16-79

U.S. Government Accountability Office. (2017) "Information Security: DHS Needs to Continue to

Advance Initiatives to Protect Federal Systems." GAO-17-518T

U.S. Government Accountability Office. (2017) "Critical Infrastructure Protection: DHS Risk

Assessments Inform Owner and Operator Protection Efforts and Departmental Strategic

Planning.” GAO-18-62

Vira, C. and Y. Y. Haimes (1983). Multiobjective decision making : theory and methodology. New

York, North Holland.

Vugrin, E., D. Warren, et. al.. (2010). A Framework for Assessing the Resilience of Infrastructure

and Economic Systems. Sustainable and Resilient Critical Infrastructure Systems. K.

Gopalakrishnan and S. Peeta, Springer Berlin Heidelberg: 77-116.

Weisberg, H. F. (1977). An introduction to survey research and data analysis. San Francisco, CA:

Freeman & Co.

Wright, P. D., M. J. Liberatore, et. al.. (2006). "A Survey of Operations Research Models and

Applications in Homeland Security." Interfaces 36(6): 514-529.

Written testimony of NPPD Executive Order 13636 and Presidential Policy Directive 21

Integrated Task Force Director Robert Kolasky for a House Committee on Homeland

Security, Subcommittee on Cybersecurity, Infrastructure Protection, and Security

Technologies hearing titled “Oversight of Executive Order 13636 and Development of the

Cybersecurity Framework” (2013)

Zimmerman, R. (2004). Decision-making and the vulnerability of interdependent critical

infrastructure. Systems, Man and Cybernetics, 2004 IEEE International Conference on, IEEE.

Zio E, FerrarioE. (2013) A framework for the system-of-systems analysis of the risk for a safety-

critical plant exposed to external events. Reliability Engineering and System

Safety;114:114–25.

Zio, Enrico (2016) Challenges in the vulnerability and risk analysis of critical infrastructures, In

Reliability Engineering & System Safety, Volume 152, , Pages 137-150, ISSN 0951-8320,

https://doi.org/10.1016/j.ress.2016.02.009.

APPENDIX A

Survey on Critical Infrastructure Protection and Resiliency Vulnerability

Impact to Hospital/Patient

A successful Denial of Service (DoS) attack on a Critical Infrastructure (CI) can indirectly have

devastating and irreversible effects to those that depend on its services. Healthcare and public

health facilities rely on various CIs in order to maintain daily operations. A cyber or physical attack

on any interdependent CI (water, power, communications, etc.) can indirectly have a detrimental

and irreversible effect on a patient’s health.

This survey is constructed to identify metrics used by hospitals to protect and/or maintain patient

healthcare and to assess the impact to a patient given a DoS or interruption to power, water or

communications to a hospital. The following questions should be answered from the expert

knowledge of a physician/nurse or administrator caring for a critically/gravely ill patient in Intensive

Care Unit (ICU) totally dependent on a device serviced by power, water or communications. Thank

you in advance for sharing your knowledge/expertise. No personal information will be elicited

or shared in this survey.

Background Information (Anonymous)

1. What is your medical profession? a. Nurse b. Physician c. Administrator

2. How long have you practiced in the medical field? a. Less than 1 year b. 1-5 years c. 6-10 years d. 11-15years e. 16 or more years

3. Do you have experience in the Intensive Care Unit (ICU)? a. Yes b. No

Hospital Metrics

4. What metric is used to trigger backup POWER generator? a. Main source off for greater than 5 minutes b. Main source off for less than 5 minutes c. Unknown

d. No metric used 5. What metric is used to trigger backup WATER supply?

a. Main source off or contamination detected in greater than 5 minutes b. Main source off or contamination detected in less than 5 minutes c. Unknown d. No metric used

6. What metric is used to trigger alternate COMMUNICATIONS? a. Main source off for greater than 5 minutes b. Main source off for less than 5 minutes c. Unknown d. No metric used

INSTRUCTIONS: Assessing Impact to Patient

Answers to each of the following questions should add/total to 100%. Answers may be provided in the form of a

check mark or percentages. An example is provided below.

Example: Given a denial of service of POWER to a hospital, resulting in a complete shutdown (No power generator

or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely

(0-20%) Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical ✓

Degraded ✓

No Impact

This answer is interpreted as “given a complete shutdown to power, the likelihood of impact to a patient’s health is very

likely to have a critical impact and 0-20% likely to have a degraded impact”. The likelihood of no impact is

assumed zero, unless answers are provided in percentages. The answer must total 100%.

Denial of Service (DoS) - POWER

1. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

2. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

3. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only and no main source power), what is the likelihood of impact to a patient depending on dialysis (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

4. Given a DoS/interruption of POWER to a hospital, resulting in a complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on dialysis (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

5. Given a DoS/interruption of POWER to a hospital, resulting in limited services (power generator only and no main source power), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

6. Given a DoS/interruption of POWER to a hospital, resulting in a complete shutdown (No power generator or main source power), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

Denial of Service (DoS) - WATER

7. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

8. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

9. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on dialysis (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

10. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown ( No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on dialysis (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

100

11. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

12. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown ( No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

Denial of Service (DoS) - COMMUNICATIONS

13. Given a DoS/interruption of COMMS to a hospital, resulting in limited emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

14. Given a DoS/interruption of COMMS to a hospital, resulting in a complete shutdown of COMMS (No emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

15. Given a DoS/interruption of COMMS to a hospital, resulting in limited emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on dialysis (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

101

Degraded

No Impact

16. Given a DoS/interruption of COMMS to a hospital, resulting in a complete shutdown of COMMS (No emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on dialysis (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

17. Given a DoS/interruption of COMMS to a hospital, resulting in limited emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

18. Given a DoS/interruption of COMMS to a hospital, resulting in a complete shutdown of COMMS (No emergency communications (i.e. ambulance to hospital, patient to nurse/doctor, pharmacy to vendor), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)

Very unlikely (0-20%)

Unlikely (21-40%)

Need more info (41-60%)

Likely (61-80%)

Very Likely (81-100%)

Critical

Degraded

No Impact

102

APPENDIX B

Survey on Patient Impact due to Critical Infrastructure Service Interruption/Denial

1. Which of the following best describe your job function?

Answer Options

Response Percent Response Count

Physician 10.0% 1 Nurse 80.0% 8 Administrator 10.0% 1 answered question 10 skipped question 0

2. How long have you practiced in the medical field?

Answer Options

Response Percent Response Count

Less than 1 year

0.0% 0

1-5 years 20.0% 2 6-10 years 10.0% 1 11-15 years 20.0% 2 16 or more

years 50.0% 5

answered question 10 skipped question 0

3. Do you have experience in the Intensive Care Unit (ICU)?

Answer Options

Response Percent Response Count

Yes 66.7% 6 No 33.3% 3 Other (please specify) 2 answered question 9 skipped question 1

103

4. What metric is used to trigger backup POWER generator?

Answer Options

Response Percent Response Count

Main source off for greater than 5 minutes

0.0% 0

Main source off for less than 5 minutes

50.0% 5

Unknown 50.0% 5 No metric

used 0.0% 0

answered question 10 skipped question 0

5. What metric is used to trigger backup WATER supply?

Answer Options

Response Percent Response Count

Main source off or contamination detected in greater than 5 minutes

10.0% 1

Main source off or contamination detected in less than 5 minutes

20.0% 2

Unknown 70.0% 7 No metric

used 0.0% 0

answered question 10 skipped question 0

104

6. What metric is used to trigger alternate COMMUNICATIONS?

Answer Options

Response Percent Response Count

Main source off for greater than 5 minutes

30.0% 3

Main source off for less than 5 minutes

30.0% 3

Unknown 40.0% 4 No metric

used 0.0% 0

answered question 10 skipped question 0

7. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on cardiac care (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 1 1 2 2 3 9

Degraded 1 2 4 1 0 8

No Impact 3 0 1 0 0 4

answered question 10 skipped question 0

8. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 0 0 3 7 10

Degraded 2 0 1 1 1 5

No Impact 4 0 0 0 0 4

answered question 10 skipped question 0

105

9. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on dialysis (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 2 2 5 1 10

Degraded 1 2 2 1 0 6

No Impact 2 1 2 0 0 5

answered question 10 skipped question 0

10. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on dialysis (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 0 0 5 5 10

Degraded 1 2 1 1 1 6

No Impact 2 2 0 0 0 4

answered question 10 skipped question 0

11. Given a DoS/interruption of POWER to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 2 2 4 2 10

Degraded 1 1 4 1 0 7

No Impact 2 1 1 0 0 4

answered question 10 skipped question 0

106

12. Given a DoS/interruption of POWER to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 0 0 2 7 9

Degraded 2 0 2 2 1 7

No Impact 2 1 1 0 0 4

answered question 10 skipped question 0

13. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on cardiac care (ICU)?

Answer Options

Very Unlikely Unlikely Need More Info

Likely Very

Likely Response

Count

Critical 0 2 1 4 2 9

Degraded 1 3 1 3 0 7

No Impact 2 1 0 1 0 4

answered question 10 skipped question 0

14. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on cardiac care (ICU)

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 2 1 2 5 10

Degraded 2 3 0 1 1 7

No Impact 2 0 0 2 0 4

answered question 10 skipped question 0

107

15. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on dialysis (ICU)?

Answer Options

Very Unlikely Unlikely Need More Info

Likely Very

Likely Response

Count

Critical 0 2 1 4 3 10

Degraded 0 4 1 1 1 7

No Impact 1 1 1 1 0 4

answered question 10 skipped question 0

16. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on dialysis (ICU)?

Answer Options

Very Unlikely Unlikely Need More Info

Likely Very

Likely Response

Count

Critical 0 2 0 2 6 10

Degraded 2 2 0 1 1 6

No Impact 2 1 1 1 0 5

answered question 10 skipped question 0

17. Given a DoS/interruption of WATER to a hospital, resulting in limited water supply (backup water supply only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)?

Answer Options

Very Unlikely Unlikely Need More Info

Likely Very

Likely Response

Count

Critical 0 2 3 3 2 10

Degraded 1 4 1 0 1 7

No Impact 1 1 2 0 0 4

answered question 10 skipped question 0

18. Given a DoS/interruption of WATER to a hospital, resulting in a complete shutdown (No filtered/uncontaminated water or water from main source), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)?

Answer Options

Very Unlikely Unlikely Need More Info

Likely Very

Likely Response

Count

Critical 0 2 0 2 6 10

Degraded 3 3 0 0 1 7

No Impact 1 2 1 0 0 4

answered question 10 skipped question 0

108

19. Given a DoS/interruption of COMMS to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on cardiac care (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 2 1 5 2 10

Degraded 1 4 2 0 0 7

No Impact 0 2 1 1 0 4

answered question 10 skipped question 0

20. Given a DoS/interruption of COMMS to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on cardiac care (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 1 1 2 6 10

Degraded 2 1 3 0 0 6

No Impact 0 1 4 0 0 5

answered question 10 skipped question 0

21. Given a DoS/interruption of COMMS to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on dialysis (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 3 2 2 3 10

Degraded 1 5 1 0 0 7

No Impact 0 2 1 1 0 4

answered question 10 skipped question 0

109

22. Given a DoS/interruption of COMMS to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on dialysis (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 3 0 2 5 10

Degraded 2 4 1 0 0 7

No Impact 0 2 1 1 0 4

answered question 10 skipped question 0

23. Given a DoS/interruption of COMMS to a hospital, resulting in limited power (power generator only), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 2 2 3 3 10

Degraded 1 5 0 0 1 7

No Impact 1 2 0 1 0 4

answered question 10 skipped question 0

24. Given a DoS/interruption of COMMS to a hospital, resulting in complete shutdown (no power generator or main source power), what is the likelihood of impact to a patient depending on oxygen/ventilator (ICU)?

Answer Options

Very Unlikely (0- 20%)

Unlikely (21-40%)

Need More Info (41- 60%)

Likely (61- 80%)

Very Likely (81-

100%)

Response Count

Critical 0 1 1 2 6 10

Degraded 2 3 1 0 1 7

No Impact 1 2 1 0 0 4

answered question 10

110

skipped question 0