Predictors of the Ability of Cybersecurity Professionals to Implement the
National Institute of Standards and Technology Risk Management
Framework
CHAPTER 1: INTRODUCTION
The number of cyber threats the United States (U.S.) faces, both in the public and private
sectors, continues to grow, as does the level of sophistication of the cyber threat actors (Exec.
Order No. 14028, 2021; The White House, 2023; U.S. Department of Homeland Security [DHS],
2023; U.S. Government Accountability Office [GAO], 2023a; U.S. GAO, 2023b; U.S. Office of
Management and Budget [OMB], 2022). In fiscal year 2022 alone, U.S. federal civilian agencies
experienced 30,659 cybersecurity incidents (U.S. GAO, 2024). Strategically, the expanding
cyber-threat landscape positions cybersecurity as an existential threat facing the U.S. (Mayorkas,
2021; Russell & Jackson, 2018; U.S. DHS, 2023; U.S. GAO, 2023a). The need to implement
comprehensive cybersecurity programs to protect information systems is an increasing business
and operational imperative (Granneman, 2018; Hepfer & Powell, 2020; Perwej et al., 2021;
Schinagl & Shahim, 2020; U.S. GAO, 2024; Verdugo & Rodríguez, 2020).
Cybersecurity frameworks are valuable tools in architecting these foundational security
programs for organizations (Dedeke & Masterson, 2019; Perwej et al., 2021). Syafrizal et al.’s
(2020) research on cybersecurity standards and frameworks identified approximately 250 types
of standards and frameworks in use worldwide. Of the 250 types, the researchers discovered 33
cybersecurity frameworks, standards, and directives were predominately noted in academic
publications. Of the cybersecurity frameworks, there are several notable frameworks available
for organizations to use; for example, the International Organization for Standardization and the
International Electrotechnical Commission (ISO/IEC) 27001 framework, the National Institute of
Standards and Technology (NIST) Cybersecurity Framework (CSF), the NIST Risk Management
Framework (RMF), the Health Information Trust Alliance (HITRUST) Common Security
Framework, the Center for Internet Security’s Critical Security Controls, and the Control
Objectives for Information and Related Technology (de Sá Mussa, 2021; Dedeke & Masterson,
2019; Srinivas et al., 2019; Syafrizal et al., 2020).
Organizations may select a cybersecurity framework, or components from various
frameworks, best suited to serve their organization and security needs. However, the NIST RMF
is mandatory for organizations that manage U.S. federal information and information systems
(NIST, 2020a). With the numerous U.S. organizations and cybersecurity professionals required to
implement the comprehensive security and privacy controls prescribed in the NIST RMF, this
research explored human-capitol factors posited to constrain the efficacy of individuals
protecting U.S. government information and information systems.
Background of Study
Researchers have investigated challenges to accepting and implementing the NIST
Cybersecurity and Risk Management Frameworks (Dedeke & Masterson, 2019; Holmes, 2021;
Syafrizal et al., 2020; Yvon, 2020). Of the numerous obstacles noted in their research, two
general themes resonated as challenges continuing to hinder the implementation of the NIST
RMF. First, information system security effectiveness is inhibited by the complexity and depth of
cybersecurity programs (Dedeke & Masterson, 2019; Diesch et al., 2020; Goel et al., 2020; Hale
& Gamble, 2019; Salminen, 2019; Srinivas et al., 2019; Syafrizal et al., 2020). The extensiveness
of the NIST RMF remains prevalent. Since NIST introduced the initial RMF in 2005 with 241
security controls, the number of security and privacy controls and enhancements has grown to
1,189 (NIST, 2005; NIST, 2020a). Second, the lack of diverse cybersecurity knowledge
possessed by security teams was an obstacle to implementing the frameworks (Holmes, 2021;
Salminen, 2019; Syafrizal et al., 2020; Yvon, 2020). With the comprehensive expansion of the
NIST RMF, the expertise required of cybersecurity professionals to implement RMF controls
was hypothesized to fluctuate across the families of security and privacy controls.
Considering the massive size of the U.S. government, the importance of protecting U.S.
information, and the mandatory application of the in-depth NIST RMF across the U.S.
government, this research delved deeper into these previous research findings.
The first area for examination in this study was whether cybersecurity certifications of
persons were a predictor of the ability of cybersecurity professionals to implement RMF security
and privacy controls. Cybersecurity certifications were posited to help baseline the information
security knowledge of cybersecurity personnel (International Information System Security
Certification Consortium [(ISC)²], 2021). Consistent with these assertions, Fortinet (2022)
conducted a global research survey of information technology and cybersecurity managers and
reported 95% of respondents believed certifications were important for personnel in
technologyrelevant workforce roles.
The U.S. Office of Personnel Management (OPM) has not mandated certification
requirements for the cybersecurity workforce across the entire U.S. federal government (2018).
Alternatively, OPM (2018) recognized that U.S. federal agencies may elect to specify
certification requirements as part of an agency’s hiring practices. The U.S. Department of
Defense (DoD), as the federal government's largest agency (U.S. DoD, n.d.), has mandated a
certification requirement for their cybersecurity professionals. Certifications are mandatory for
U.S. DoD civilian, military, and support contractors assigned to cybersecurity workforce
positions (U.S. DoD, 2015; U.S. DoD, 2020). Notwithstanding these certification requirements,
commercial cybersecurity certifications are issued by different governing bodies (U.S. Defense
Information Systems Agency, n.d.) and assess varied bodies of knowledge.
The second area for investigation in this study was whether cybersecurity experience was
a predictor of the ability of cybersecurity professionals to implement RMF controls, particularly
in relationship to the unique cybersecurity certification held by these professionals. Syafrizal et
al.’s (2020) research into adopting cybersecurity frameworks and standards asserted governments
lacked the personnel with cybersecurity experience necessary to implement the frameworks.
Additionally, the ability to secure government systems depends on the knowledge, skills, and
abilities (i.e., experience) of the federal cybersecurity workforce securing the information
systems (U.S. GAO, 2021). Conversely, while work experience may be a force multiplier,
overstating desired job experience levels may limit the number of applicants eligible to compete
for cybersecurity positions (McQuaid & Cervantes, 2019).
Problem Statement
Organizations managing U.S. federal government information or information systems
must implement NIST RMF controls to protect the information and systems (NIST, 2020a). In
the most recent revision to the NIST RMF (2020a), the document lists 20 security and privacy
control areas, covering 1,189 controls and control enhancements. These NIST controls and
enhancements are further defined and assessed through sub-controls, known as assessment
procedures, exacerbating the amount of knowledge cybersecurity professionals must understand
to defend information systems from being breached. The general problem is U.S. information
and information systems are at risk of being compromised due to the inability to implement NIST
RMF security and privacy controls (U.S. Department of Education [ED], 2021; U.S. Department
of Veterans Affairs [VA], 2021; U.S. GAO, 2022; U.S. OMB, 2020; U.S. OPM, 2021).
The specific problem is the U.S. federal government lacks the cybersecurity workforce
with the competency (e.g., skills, abilities, or behaviors) to comprehensively implement NIST
RMF security and privacy controls to enable them to protect U.S. government information (U.S.
GAO, 2019b).
Purpose of Dissertation Study
The cybersecurity field needs suitably skilled individuals (Salminen, 2019; U.S. GAO,
2019a; U.S. GAO, 2019b; U.S. GAO, 2021) and there continues to be a shortfall of cybersecurity
talent. According to the (ISC)² Cybersecurity Workforce Study (2021), the global cybersecurity
personnel deficit is 2.72 million people. CyberSeek’s (2024) cybersecurity job heat map, a
project supported by the National Initiative for Cybersecurity Education (NICE) grant, placed the
overall U.S. cybersecurity job openings at 572,392 positions. Studies suggest there is a global
shortage of cybersecurity experts needed to fill positions to protect information systems from
threat actors (McQuaid & Cervantes, 2019; Noche, 2021; Oltsik & Lundell, 2021; Ramezan,
2023; U.S. GAO, 2021). This research pivoted to examining predictors of the abilities of the
existing U.S. cybersecurity workforce dedicated to implementing NIST RMF security and
privacy controls. The purpose of this quantitative, non-experimental survey was to examine the
relationship between select competencies of U.S. cybersecurity professionals and their perceived
ability to implement NIST RMF security and privacy controls for organizations managing U.S.
information or information systems.
Significance of Study
This study sought to extend and distinctively investigate new areas of similar, previously
conducted research. Dobrydney (2020) surveyed cybersecurity professionals who held DoD
8570.01-M Information Assurance Management Level III certifications and compared this
information to their perceived ability to implement RMF policy at the organizational level. As a
recommendation for future study, Dobrydney suggested collecting additional data from survey
participants, including experience, education, and training types. Building on this existing
research, this study examined the relationship between the certifications and experience of
cybersecurity professionals managing U.S. information and information systems and their
perceived ability to implement NIST RMF security and privacy controls.
Additionally, the scope of this study increased the organizational diversity of the targeted
study population and focused on a different step in the NIST RMF process. The population of
this study was expanded to all cybersecurity professionals who use the NIST RMF for securing
U.S. information or information systems. Furthermore, this research examined the competency
relationships from an implementation, system-level perspective versus a policy,
organizationallevel study. Lastly, an analysis by certification was completed instead of only
analyzing the number of certifications held. These study modifications were introduced to inject
more granularity into the ability assessments to investigate potential differences in the task
outcomes.
The theoretical benefit of this research was to inform further the measurement of the
competency component of the Self-Determination Theory (SDT) and to expand the use of the
SDT within the cybersecurity domain. Fundamentally, SDT is posited to emphasize the
psychological needs of relatedness, autonomy, and competence as the principal motivation of
behavior (Ryan & Deci, 2000).
Additionally, the significance of this research aimed to identify potential needs for change
in the content or applicability of commercial-based certifications. The contributions from this
study may also help to advocate for the resources to improve a fractured and ad-hoc approach to
providing RMF implementation practitioner support. For example, there are some siloed and
DoD-level knowledge resource centers (e.g., Risk Management Framework Knowledge Service)
to address RMF questions; however, the knowledge communities are informally moderated.
Furthermore, this research may justify the need to specialize further in one or more of the
three suggested security and privacy control areas (i.e., administrative, physical, and technical).
There may also be a complementary need to develop single/multi-area experts to provide
crossorganizational or intra-agency RMF security and privacy controls implementation support.
Finally, this study may provide information to support advocacy for the continued development
of automated tools to implement and assess security and privacy controls.
Overall, the human elements of cybersecurity are less researched than the technological
aspects of cybersecurity (Dalal et al., 2022). The practical significance of exploring these
cybersecurity-related human capital predictors may assist cybersecurity professionals, training
personnel, managers, and leaders in understanding these limitations and defining comprehensive
strategies to improve the abilities of cybersecurity professionals responsible for implementing the
NIST RMF controls to protect U.S. information systems.
Nature of Study
This study used a quantitative, survey-based design to enable the researcher to examine the
relationship between select competencies of U.S. cybersecurity professionals and their perceived
ability to implement NIST RMF security and privacy controls. The population for the research
was bounded to cybersecurity professionals implementing the NIST RMF to protect U.S.
information and information systems. An online questionnaire was used to collect responses.
The researcher did not uncover any empirical evidence to identify any of the NIST RMF
steps as more critical in the execution of the NIST RMF process. However, Porter (2019) posited
that the implementation of security controls (RMF Step 4) is one of the more essential steps
required to ensure mission assurance. NIST SP 800-53 also stated, “The selection, design, and
implementation of security and privacy controls are important tasks” (2020a, p. 1).
Research Questions
Spring and Illari (2019) asserted the information security field is ripe with challenges,
making it an attractive domain for philosophical study. This research analyzed the association
between select competency qualities of the U.S. cybersecurity workforce and the expertise of
these individuals to implement NIST RMF controls. The two predictor variables representing
cybersecurity competency were certification of persons and years of experience. The first
predictor, certification, is a specific cybersecurity certification (e.g., Certified Information
Systems Security Professional, Computing Technology Industry Association Advanced Security
Practitioner, Certified Chief Information Security Officer). The second predictor, experience, is
the years of experience in the information technology and/or cybersecurity fields.
The dependent variable is defined as the perceived ability to implement NIST RMF
security and privacy controls. More specifically, the perceived ability to implement controls were
assessed against the primary RMF Implementation step task (i.e., Task I-1 – Control
Implementation) outcomes (NIST, 2018).
The two RMF Task I-1 outcomes were separated into four areas to explicate the study
results: methodology use, administrative control implementation, physical control
implementation, and technical control implementation. The three control implementation areas
(i.e., administrative, physical, and technical) categorize the 20 NIST RMF security and privacy
control families. The categorization was based on previous versions of the NIST RMF (NIST,
2010) and researchers' overall classifications of security controls (Park et al., 2021; Zaini et al.,
2020).
The three categories are comprised of the following NIST RMF control families:
Administrative: Awareness and Training (AT); Assessment, Authorization, and
Monitoring (CA); Configuration Management (CM); Contingency Planning (CP); Incident
Response (IR); Planning (PL); Program Management (PM); Personnel Security (PS); Personally
Identifiable Information Processing and Transparency (PT); Risk Assessment (RA); System and
Services Acquisition (SA).
Physical: Maintenance (MA); Media Protection (MP); Physical and Environmental
Protection (PE); Supply Chain Risk Management (SR).
Technical: Access Control (AC); Audit and Accountability (AU); Identification and
Authentication (IA); System and Communications Protection (SC); System and Information
Integrity (SI).
Eight research questions were posited to inform this study. Two predictor variables (i.e.,
certifications and years of experience) were assessed against the outcomes of the primary NIST
RMF Implementation step (i.e., Task I-1). As shown in Figure 1, the task outcomes were divided
into four sub-outcome areas (i.e., administrative, physical, technical, and methodologies).
Figure 1
Relationship Between Survey Participant Demographics and Research Questions
The specific, proposed Research Questions (RQ) and hypotheses are as follows:
RQ1: Is there a relationship between a certification and a perceived ability to implement
NIST RMF administrative controls?
Hypothesis (Ha1) supporting RQ1: There is a positive relationship between a
certification and a perceived ability to implement NIST RMF administrative controls.
The null hypothesis (H01) for RQ1 is there is no relationship between a certification and a
perceived ability to implement NIST RMF administrative controls.
RQ2: Is there a relationship between a certification and a perceived ability to implement
NIST RMF physical controls?
Ha2: There is a positive relationship between a certification and a perceived ability to
implement NIST RMF physical controls.
H02: There is no relationship between a certification and a perceived ability to implement
NIST RMF physical controls.
RQ3: Is there a relationship between a certification and a perceived ability to implement
NIST RMF technical controls?
Ha3: There is a positive relationship between a certification and a perceived ability to
implement NIST RMF technical controls.
H03: There is no relationship between a certification and a perceived ability to implement
NIST RMF technical controls.
RQ4: Is there a relationship between a certification and a perceived ability to use system
security and privacy engineering methodologies to implement NIST RMF controls?
Ha4: There is a positive relationship between a certification and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF
controls.
H04: There is no relationship between a certification and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF
controls.
RQ5: Is there a relationship between experience and a perceived ability to implement
NIST RMF administrative controls?
Ha5: There is a positive relationship between experience and a perceived ability to
implement NIST RMF administrative controls.
H05: There is no relationship between experience and a perceived ability to implement
NIST RMF administrative controls.
RQ6: Is there a relationship between experience and a perceived ability to implement
NIST RMF physical controls?
Ha6: There is a positive relationship between experience and a perceived ability to
implement NIST RMF physical controls.
H06: There is no relationship between experience and a perceived ability to implement
NIST RMF physical controls.
RQ7: Is there a relationship between experience and a perceived ability to implement
NIST RMF technical controls?
Ha7: There is a positive relationship between experience and a perceived ability to
implement NIST RMF technical controls.
H07: There is no relationship between experience and a perceived ability to implement
NIST RMF technical controls.
RQ8: Is there a relationship between experience and a perceived ability to use system
security and privacy engineering methodologies to implement NIST RMF controls?
Ha8: There is a positive relationship between experience and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF
controls.
H08: There is no relationship between experience and a perceived ability to use system
security and privacy engineering methodologies to implement NIST RMF controls.
Theoretical Framework
Two prominent behavioral theories used in the security field are the Protection
Motivation Theory (PMT) and SDT (Haastrecht et al., 2021). This research was viewed through
the lens of SDT. The two predictor variables (i.e., certification of persons and years of
experience) were collected as part of each survey participant's demographic information (e.g.,
role, education level). The participant’s perceived ability to implement NIST RMF security and
privacy controls was constructed and measured using a Perceived Competence Scale (PCS)
consisting of four subordinate stub statements.
The PCS was designed by Williams and Deci (1996), and items on the instrument may be
adapted to provide relevance to the domain being studied. The four PCS subordinate statements
were assessed using an ordinal, 7-point Likert scale. Each participant was asked to annotate one
rating per RQ. For example, to inform RQ1, the survey participants read four stub statements and
then holistically rated their perceived ability to implement NIST RMF administrative controls on
a scale from one (e.g., not true at all) through seven (e.g., very true). A pilot study was conducted
with a small cohort of NIST RMF subject matter experts to evaluate the functionality and
intuitiveness of the survey tool.
The targeted population for this study included cybersecurity professionals who were
applying the NIST RMF to protect U.S. information or information systems. The survey was
administered using a hosted, online survey tool to facilitate its administration. Participants for the
survey were solicited using an email group consisting of approximately 2,200 cybersecurity
professionals principally involved with NIST RMF activities. Survey participants were also
solicited through LinkedIn RMF-specific groups and connections. Post-survey data analysis was
conducted using statistical analysis software.
Definitions
A common lexicon is foundational for a mutual understanding of terms presented in this
study. U.S. federal government definitions are primarily cited due to the study’s focus on the
federal government.
Ability: defined by the U.S. OPM (2019) as a “competence to perform an observable
behavior” (p. 199).
Certification: a credential issued by a certification body “based on an individual
demonstrating, through an examination process, that they have acquired the designated
knowledge, skills, and abilities to perform a specific job” (U.S. Bureau of Labor Statistics, 2017,
p. 1).
Competency: the blending of a person’s characteristics, including an individual’s
knowledge, skills, abilities, and behaviors, required to perform a workforce role (U.S. OPM,
2019).
Cybersecurity: “prevention of damage to, protection of, and restoration of computers,
electronic communications systems, electronic communications services, wire communication,
and electronic communication, including information contained therein, to ensure its availability,
integrity, authentication, confidentiality, and non-repudiation” (U.S. OMB, 2016, p. 28).
Experience: an “indicator of proficiency” attributed to the blend of an individual’s
personal and professional experiences related to the knowledge, skills, and abilities required to
perform in a workforce role (U.S. OPM, 2019, p. 204).
Information Security: a term often used synonymously for cybersecurity. Information
security and cybersecurity are implemented to provide confidentiality, integrity, and availability
(CIA) of a system (U.S. OMB, 2016). However, the subtle difference reasoned by von Solms and
von Solms (2018) and Perwej et al. (2021) is information security applies to information in all
mediums, whereas cybersecurity applies to information within a virtual environment. Based on
the nature and scope of this study, cybersecurity was predominately used in this study.
Security Control: a measure, mechanism, or safeguard employed to protect the CIA of a
system and the information contained in the system (U.S. OMB, 2016).
Assumptions
Several research assumptions were formulated during the development of this study. The
assumptions were based on a post-positivism paradigm and study-specific considerations.
Postpositivism allows for a broader belief of how knowledge is produced, permitting the role of
human perspective and error to be considered within this perspective (Godwin et al., 2021).
The first assumption was survey participants honestly responded to survey questions and
assessments (Goes & Simon, 2018). This is an essential assumption, especially regarding a
respondent’s assessment of their abilities. An individual’s abilities are likely to be overstated in a
social setting when confidence is exhibited by others in the group (Cheng et al., 2021). Similarly,
respondents may hesitate to select the lowest or highest rating on a psychometric scale to assess
their abilities (Theofanidis & Fountouki, 2019). However, the confidentiality and anonymity
provided by the implementation of the online survey may have further encouraged participants to
assess their abilities truthfully. Additionally, survey participants were volunteers and could
abandon the questionnaire at any time, without prejudice (Goes & Simon, 2018).
Second, and complementary to the first assumption, the study participants trusted the
anonymity provided by the researcher by virtue of the privacy methods extended by the online
survey tool. Third, survey participants were NIST RMF practitioners who were involved with the
implementation phase of the RMF in the protection of U.S. information or information systems.
Fourth, respondents fully understood the survey questions. Lastly, the meaning assigned to this
study was subjective and assigned by the researcher; reality itself is objective and discrete from
the researcher (Goes & Simon, 2018).
Scope
This quantitative research examined the correlation between the cybersecurity
certifications of persons and the years of experience possessed by U.S. cybersecurity
professionals, and their perceived ability to implement NIST RMF security and privacy controls
for organizations managing U.S. information or information systems. Using an online
questionnaire, primary study data were collected from voluntary, first-hand sources (i.e., NIST
RMF practitioners). Internet-based surveys provide many advantages; they are economical to
administer, provide an expanded population reach, foster expeditious data collection, ease data
summation, and regulate data response formatting (Ponchio et al., 2021). The survey was
projected to be available to participants for 3- to 4-weeks and extended, as needed, to ensure an
acceptable sample size.
A random sampling approach was employed to reach potential survey respondents. A
non-discriminative snowball recruitment method was also used to encourage participants to
recommend the survey to other potential respondents. The confidentiality and anonymity of the
online questionnaire may have alleviated any potential privacy or coercion concerns related to
the snowball sampling technique. Survey participants were solicited using LinkedIn professional
networking services and through an ad-hoc email distribution list.
Limitations
Study limitations are uncontrollable constraints outside the researcher's sphere of
influence (Theofanidis & Fountouki, 2019). However, limitations may influence the research
design and findings and should be acknowledged in a study’s report (Theofanidis & Fountouki,
2019). This study was limited by selecting a quantitative, non-experimental research design (i.e.,
one sample and no control groups). There was no presumption of a direct, causal relationship
(internal validity) between the predictor variables (i.e., certifications and years of experience)
and the dependent variable (i.e., ability to implement NIST RMF controls) because of the
nonexperimental design of the study. Alternatively, the purpose of the research was to examine
the association between the variables, to include determining if the variable relationships were
statistically significant and if any changes between the predictor variables were associated with
changes in the dependent variable. Additionally, the data collection procedure (i.e., survey) used
quantitative data. Consequently, the data collected was closed-ended and lacked descriptive
details to inform further analysis regarding the studied relationships (Siddiqui, 2019).
Lastly, the online survey distribution method may have constrained the potential sample
size and dissemination of the questionnaire. The researcher found the U.S. federal government
had holistically fallen short in accurately identifying and coding personnel positions performing
information technology, cybersecurity, or cyber-related roles (U.S. GAO, 2019a). This deficiency
was also acknowledged by the Cyberspace Solarium Commission (CSC) created by the U.S.
Congress. In their 2022 report, the CSC recommended the federal cybersecurity workforce be
appropriately identified, and the CSC called for updated federal cyber-specific occupational
classifications (Ma & Montgomery, 2022). The lack of a full accounting of cybersecurity
positions limited the ability to calculate the size of the study’s population. Finally, using a
voluntary sampling technique enabled potential participants to ignore the survey, further limiting
the sample size.
Delimitations
Delimitations of a study describe what the researcher did not elect to do in the
performance of the study (Theofanidis & Fountouki, 2019). This study was divided into two
primary areas to help focus the scope of the research. First, the NIST RMF 7-step process (i.e.,
Prepare step through Monitor step) was descoped to include only research applicable to the
Implementation step of the RMF process. Second, independent variables were decreased to two
predictor variables. Other variables, such as the size of the organization’s cybersecurity team or
the types of training courses completed, could have been included in the study. The researcher
introduced these delimitations to reduce the survey burden on potential participants and increase
the likelihood participants would complete the questionnaire and provide accurate
selfassessments.
Chapter Summary
This chapter presented an introduction to the research topic, and the researcher asserted
the U.S. federal government lacks the cybersecurity workforce with the competencies to
implement comprehensive NIST RMF security and privacy controls to enable them to protect
U.S. government information proactively. A quantitative study was described as the proposed
research method to examine the relationship between select competencies of U.S. cybersecurity
professionals and their perceived ability to implement NIST RMF controls. The significance and
nature of the research were stated, and eight research questions were proposed to inform this
research. Furthermore, an introductory overview of the theoretical framework was presented in
this chapter. Seven terms relevant to this study were also defined, with an emphasis on using U.S.
government definitions. Finally, the scope, limitations, and delimitations framing this research
were acknowledged in this chapter.
Chapter 2 provides a detailed literature review of the research topics. The review presents
additional background information on the research subject, references opposing viewpoints,
provides context to the study’s focus areas, and identifies research gaps. The literature landscape
is primarily organized into four general areas, including governmental guidance, the
implementation of controls, cybersecurity workforce certifications, and cybersecurity experience.
CHAPTER 2: LITERATURE REVIEW
Introduction
Foundational to defining the current knowledge of a research topic, a literature review
provides background information and context about the research topic (Machi & McEvoy, 2022).
The intention of this chapter is to present an exploration of the literature relevant to the study of
the cybersecurity professionals who implement NIST RMF security and privacy controls to
protect U.S. information and information systems. The chapter begins with a summary of the
objective of this study and a description of the methods used to search for scholarly documents.
After that, the first section of this literature review describes the governmental guidance
pertinent to the NIST RMF, with a specific emphasis on the implementation of security and
privacy controls. The second section of the review documents the current findings related to the
implementation of security and privacy controls, the certification of persons, and the relevancy of
cybersecurity workforce experience. The third section of this exploration identifies gaps in the
reviewed literature. Lastly, the chapter concludes with a chapter summary.
Restatement of Study Objective
The objective of this research was to examine the relationship between two attributes of
U.S. cybersecurity professionals and their perceived ability to implement NIST RMF security
and privacy controls for organizations. The researcher surveyed U.S. personnel who administer
the NIST RMF to protect U.S. federal government information and information systems. The
researcher sought to measure an individual's assessment of their ability to implement the NIST
RMF influenced by their cybersecurity certifications, including the possibility of not possessing a
cybersecurity certification. The researcher also measured the number of years of information
technology and cybersecurity experience that were possessed by the practitioners. The study
results may inform executive leadership, managers, and human-resource personnel regarding the
relevancy of cybersecurity certifications and experience in the context of the RMF implementer
role.
Literature Review Process
The systematic search for literature began with determining the sources to obtain
scholarly literature. Three primary sources were identified and predominantly employed to
conduct the literature review for this study. The three sources included Capitol Technology
University resources, Google Scholar, and U.S. government sites (e.g., nist.gov).
Of the three primary sources, the Capitol Technology University Virtual Library was used
to conduct extensive literature searches. The Virtual Library had a comprehensive collection of
over 15 searchable databases, including the Association for Computing Machinery (ACM)
Digital Library, Homeland Security Digital Library, Institute of Electrical and
Electronics Engineers (IEEE) Xplore collection, Wiley Online Decision Sciences Journal, and
ProQuest. The databases provided access to numerous publications, for example, conference
proceedings, research publications, standards, books, dissertations, and scholarly journals.
ProQuest was the primary source used in the Virtual Library portfolio of resources. ProQuest is a
search engine capable of querying 7 academic databases featuring 51 unique document
collections.
Advanced title and/or full-text keyword searches were conducted as a first step of the
literature review. Keywords for the searches were based on the focus areas of this research and
keywords commonly cited in related, published journal articles. Keywords included, but were not
limited to, certifications, computer security, cybersecurity, experience, implementation,
information assurance, information security standards, information technology, ISO/IEC 17024,
NIST, NIST SP 800-37, NIST SP 800-53, NIST SP 800-61, RMF, security controls, security
standards, workforce development, and combinations thereof.
The search objective was to keep the search terms as broad as possible to ensure the
discovery and inclusion of relevant literature. Also, searches were focused on peer-reviewed
journal articles published within the last 5 years (i.e., articles published in 2018 or thereafter).
Other exclusionary criteria included such documents as working papers, commentaries, and
nonEnglish publications.
The second step involved screening the articles for applicability to this study. Screening
entailed a deliberate review of titles, abstracts, and complete texts. Selected, relevant articles
were ingested and categorized in RefWorks for further consideration and summary. RefWorks
was also useful for identifying any duplicate references. RefWorks is a reference management
tool with the capability to manage citations, store full-text documents, and create reference lists.
The number of publications selected for further review, categorized by overall context, is shown
in Appendix A.
After the articles were stored and organized in RefWorks, the third step included reading,
understanding, and summarizing the articles. Drafted article summaries were stored as part of the
metadata associated with each respective article in RefWorks. The systematic collection of
summaries was helpful for the purposes of record keeping and organizing the documents. The
article summaries were used to form the substance of this chapter.
Additionally, peer-reviewed scholarly articles were obtained from Google Scholar, as a
resource for locating full-text publications referenced in other sources. Google Scholar was also
used to perform complimentary searches to ensure a comprehensive literature search.
Furthermore, due to the focus of the research in the context of protecting U.S.
information and information systems, online NIST resources (e.g., nist.gov, csrc.nist.gov) were
primarily used to collect relevant policy documents. Seminal documents and U.S. government
documents dated before 2018 germane to this study were also considered, and in some cases
presented, in this literature review. Finally, of note, the scope of this literature review was limited
to publicly available documents. The review did not include publications specific to the
protection of classified information or classified information systems (e.g., systems categorized
as U.S. national security systems or intelligence systems).
Overview of Germane Governmental Guidance
In an irregular, rapidly changing security environment, cybersecurity governance and the
role of the cybersecurity practitioner are unceasingly complex (Burdon & Coles-Kemp, 2019;
Schinagl & Shahim, 2020). This holds for the cybersecurity workforce protecting U.S. federal
information systems and the overarching U.S. cybersecurity regulations. With the underpinning
of a U.S. federal government-focused research topic, cybersecurity guidance specific to the
objective of this study and the U.S. government are summarized in this section. This overview
provides the contextual and governing landscape cybersecurity practitioners must navigate.
Federal Law and Office of Management and Budget Circular
Federal Information Security Management Act (FISMA)
FISMA is at the pinnacle of U.S. government laws relevant to information security.
Initially signed into law in 2002 and later amended in 2014, the statute was enacted to assign
authorities and address information security requirements, including, providing a comprehensive
framework for implementing security controls, developing, and sustaining a minimum set of
controls to protect federal information and information systems, and updating security practices
(FISMA, 2014). Additionally, FISMA introduced information security as protecting the
confidentiality (authorized access and disclosure), integrity (authorized modification or
destruction), and availability (timely and reliable access/use) of information and information
systems. A system's confidentiality, integrity, and availability are often referred to as the CIA
triad (Alsaqour et al., 2021; Zaini et al., 2020).
Federal Cybersecurity Workforce Assessment Act of 2015
This Law mandated the U.S. federal government perform human-resource planning for the
Government’s cybersecurity workforce. Specifically, the Act required the U.S. OPM to
implement a workforce position coding structure to identify civilian and non-civilian positions
conducting cybersecurity, cyber-related, or information technology job activities (Federal
Cybersecurity Workforce Assessment Act, 2015). The Act further directed OPM to coordinate
with NIST to develop the coding construct under the NICE framework.
OMB Circular A-130, Managing Information as a Strategic Resource
Atop the hierarchy of U.S. government policy for federal agencies, this Circular is central
to information security and privacy policy. The Circular is reflective of many statutes,
Presidential Directives, Executive Orders, and practices. It is applicable to all agencies of the
Executive Branch (e.g., executive agency, federal-controlled corporations, military department,
etc.) and non-federal entities that collect or manage information on behalf of the federal
government (U.S. OMB, 2016).
The main theme of Circular A-130 is to set policy for managing information resources.
However, inclusive of the guidance, Appendix I exclusively details the responsibilities and
minimum requirements for managing and protecting federal information and information
systems. The federal mandate to implement RMF (i.e., NIST SP 800-37, Risk Management
Framework for Information Systems and Organizations) and to implement security and privacy
controls are introduced at this level of the U.S. government. From a broad perspective, in
addition to the requirement to maintain a comprehensive privacy program, U.S. agencies are
required to protect information commensurate with risk, designate a senior agency information
security officer, and execute security policies issued by the DHS, the General Services
Administration (GSA), the Office of Personnel Management (OPM), and the Department of
Commerce (e.g., NIST publications).
DoD and Intelligence Community systems are exempt from OMB guidance regarding the
NIST RMF (FISMA, 2014). However, both organizations have agreed to follow the NIST RMF
guidance (Johnson, 2019). Within FISMA and OMB Circular A-130, this exemption was
specified in terms of the lack of applicability of the standards to national security systems (e.g.,
intelligence systems, classified systems). However, the DoD and Intelligent Community are
encouraged to apply the guidance as appropriate (U.S. OMB, 2016).
National Institute of Standards and Technology Publications
The U.S. NIST, a non-regulatory agency of the U.S. Department of Commerce, has a
broad profile of responsibilities across the physical sciences, including information technology
and cybersecurity. Within these two domains, NIST has published numerous information security
standards as part of their responsibility under OMB Circular A-130, FISMA,
Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure (Executive
Order 13800), and the Cybersecurity Enhancement Act of 2014, among other directives. Of
NIST’s six research laboratories, the Information Technology Laboratory is focused on the
security and privacy of federal information and information systems, excluding national security
systems (e.g., intelligence systems, weapon systems, command and control systems). Several of
these NIST cybersecurity standards and guidelines are germane to this research and are
summarized in this sub-section. On the international stage, the ISO/IEC provides similar
processes (e.g., ISO/IEC 27000 family of standards) for information security techniques and risk
management.
NIST SP 800-12, Rev 1, An Introduction to Information Security
This NIST publication provides an overview of information security principles,
highlighting the elements of information security, roles and responsibilities, policies, risk
management, and cryptography (NIST, 2017a). The guidance also provides a synopsis of each
security control family (i.e., Access Control, Contingency Planning, Media Protection, etc.). The
security controls therein are comprehensively expounded in NIST SP 800-53 and form the basis
of the dependent variable of this study (i.e., perceived ability to implement security and privacy
controls).
NIST SP 800-37, Rev 2, Risk Management Framework for Information Systems and
Organizations
NIST SP 800-37 is the cornerstone of the RMF process and provides the guidelines for
executing activities to manage security and privacy risks (2018). The guidance is technology
agnostic, enabling the Framework to be applied to any system. The publication provides a brief
overview of organization-wide risk management, as NIST SP 800-39 thoroughly describes
NIST’s recommended approach to managing risks.
Central to NIST SP 800-37 are the defined and detailed RMF steps and structure. As
shown in Figure 2, the seven RMF steps are identified as Prepare, Categorize, Select, Implement,
Assess, Authorize, and Monitor.
Figure 2
NIST RMF 7-Step Process
Note. Adapted from “Risk Management Framework for Information Systems and
Organizations (NIST SP 800-37, Rev 2),” by NIST, 2018, p. 9.
Consequent to this study’s focus, specifically on the Implementation step of the RMF
process, each RMF step is summarized to provide context.
The Prepare step was added as an additional RMF step in 2018. This initiating
organizational step informs the activities in the remaining six steps. Tasks include such activities
as establishing organizational-level risk management and continuous monitoring strategies,
identifying roles, and determining the risk tolerance for the organization.
During the Categorize step, the information system characteristics are documented, and
the system’s authorizing official finalizes and approves a categorization decision. For systems not
identified as national security systems, system categorization (i.e., low, moderate, or high impact)
is based on Federal Information Processing Standards (FIPS) Publications 199 and 200
requirements.
The Select step involves choosing a control baseline, tailoring out specific security and
privacy controls, developing a system-level continuous monitoring strategy, and authoring and
finalizing security and privacy plans documenting the control selections.
This study was framed within the context of Step 4, the Implement step. This step is
focused on two primary tasks. The first task is to implement the controls identified and
documented in the previous steps. The two outcomes of the first task are the application of
security and privacy controls and the employment of engineering methodologies to implement
the controls. The second task entails updating security and privacy control execution information.
The outcomes identified for this second task are ensuring plans are updated to reflect the reality
of the controls as the controls are implemented or changed during the RMF cyclical process.
NIST identified the System Owner and Common Control Provider as the primary
workforce roles responsible for the Implement step. Common Control Providers are officials
responsible for developing, implementing, and managing controls inheritable by other systems.
Additionally, NIST identified several roles supporting this step, including the Information
Owner, Security/Privacy Architect, Security Engineer, System Security Officer, System Privacy
Officer, and System Administrator.
The Assess step entails developing and approving assessment plans, evaluating
implementation of controls against the assessment plans, documenting the assessment results,
remediating any noted deficiencies, and authoring plan of actions for any remediation areas not
immediately resolved.
The Authorize step is a culminating inspection, of sorts. During this step, the system
security and privacy plan, assessment reports, and any ongoing Plan of Action and Milestones are
assessed to formulate a risk determination. Based on the holistic risk determination, the
authorizing official approves or denies the authorization for the information system.
During the Monitor step, the information system and its operating environment are
monitored consistent with the continuous monitoring plans developed in the Prepare and Select
steps. Ongoing evaluations of the effectiveness of controls are conducted during this step, as well
as the assessments of continuous monitoring activities to ensure the information system is
maintained within the organization's risk tolerance. Events, such as a major update to the system
or a change in risk, may require the organization to revisit one or more RMF steps.
NIST Cybersecurity Framework, Version 2.0
The NIST Cybersecurity Framework (CSF), the framework was first published in
February 2014, nearly a decade after the initial edition of NIST SP 800-37, RMF for Information
Systems and Organizations, was released in May 2004. There are a few notable differences
between the CSF and RMF.
First, the CSF is a voluntary framework and may be used by public- and private-sector
organizations (NIST, 2024). The RMF, conversely, must be used by the U.S. federal government.
Governments below the federal level and non-federal entities are encouraged to use the RMF
(NIST, 2018). Commercial and private enterprises may elect to use the RMF as well.
Second, the CSF applies to organizations dependent on traditional information technology
and moreover, operational technology (e.g., industrial control systems and cyberphysical
systems). The RMF primarily focuses on information systems and organizations.
Third, the CSF is grounded in six high-level ongoing and synchronous functions: Govern,
Identify, Protect, Detect, Respond, and Recover. By selecting categories and subcategories with
the six CSF functions, organizations may create current and target “Profiles” to help inform
cybersecurity continuous improvement activities. Four CSF “Tiers” further informs
organizational cybersecurity risk maturity level decisions (i.e., Partial through Adaptive).
Conversely, as previously noted, the RMF is based on a 7-step, cyclical construct (see Figure 2).
Fourth, specified personnel roles and responsibilities are largely absent within the CSF,
whereas the RMF identifies primary and supporting roles and responsibilities for each task.
NIST suggested the CSF and RMF may be used in a concerted effort to enable agencies
to meet their regulatory information technology and cybersecurity requirements (NIST, 2018). To
facilitate this integrated approach, both Frameworks include cross-references, as applicable,
between the RMF tasks and CSF subcategories.
NIST SP 800-39, Managing Information Security Risk: Organization, Mission, and
Information System View
Organizations experience risk in many areas (e.g., safety, financial, reputational, legal),
including risk related to information systems security. NIST (2011) authored this preeminent
publication to address information security risk from a distinctive organization-wide perspective.
NIST described a risk continuum, from strategic to tactical risk, across three views: organization,
business/mission processes, and information systems. Additionally, applied across these three
views, NIST identified four risk management components: framing, assessing, responding to, and
monitoring risks. These four components comprise the non-sequential steps in the NIST risk
management process.
Specific to information systems security controls, NIST noted common controls are
selected as part of the activities associated with framing risk at the organizational level (Tier 1).
At the mission-process level (Tier 2), security requirements are integrated into processes, and an
enterprise security architecture is developed. These Tier 1 and 2 activities help inform
tacticallevel decisions at Tier 3 (information system), including assigning, implementing, and
assessing security controls.
Figure 3
NIST Three Levels of Organization-Wide Risk Management
Note. Adapted from “NIST Risk Management Framework Overview,” by NIST, 2018,
https://www.nist.gov/system/files/documents/2018/03/28/vickie_nist_risk_management_framew
ork_overview-hpc.pdf, p. 5.
Within NIST SP 800-39, NIST further presented the idea of trustworthy information
systems. NIST noted trustworthy systems are information systems with the degree of
trustworthiness required by an organization to operate within its risk tolerance, regardless of the
security risks (e.g., breaches, hardware failures, and natural disasters). The two elements of
system trustworthiness are security functionality and assurance. Security functionality is
achieved by employing security and privacy controls. Security assurance is obtained by the
specific actions taken to design, develop, implement, and operate the security controls (i.e.,
security functionality), as well as conducting the required assessments to ensure the controls are
implemented correctly to meet the desired security results.
NIST SP 800-53, Security and Privacy Controls for Information Systems and Organizations
This publication is a collection of all the security and privacy controls recommended by
NIST to help secure information systems and organizations from a wide range of threats, from
natural disasters to state-sponsored threat actors (NIST, 2020a). NIST SP 800-53 was also a
precursor and blueprint for other seminal governance regulations, such as Gramm-Leach-Bliley
Act and Health Insurance Portability and Accountability Act (Granneman, 2018).
This exhaustive document spans 20 control areas, referred to as security and privacy
control families. Each family consists of defined base controls and control enhancements. Base
controls and enhancements are further separated into supporting and clarifying subordinate
assessment procedures. In total, there are 1,189 base controls and enhancements. Controls are
autonomously selected by an organization based on frameworks (e.g., RMF, CSF, Privacy)
decisions and risk management activities. Additionally, the controls may be adapted and
implemented to meet the needs of an organization’s risk tolerance; however, a minimum set of
security and privacy controls are required for federal information systems. Minimum control
baselines are identified in NIST SP 800-53B, Control Baselines for Information Systems and
Organizations.
Ascribable to the focus of this study, specifically the implementation of controls, the
security and privacy security control families are described in the following summaries.
Awareness and Training (AT). Specific activities focused on security training for
general system users, role-based training (e.g., system administrator, cybersecurity roles), and
subsequent training records.
Assessment, Authorization, and Monitoring (CA). Includes control assessments,
penetration testing, cybersecurity authorizations, management of system-related Plan of Action
and Milestones, and continuous monitoring activities.
Contingency Planning (CP). Controls related to the preparation and testing of
procedures and technical measures to recover and reconstitute information systems after a system
disruption or emergency.
Incident Response (IR). Policy and procedures are developed to enable an organization
to respond to cybersecurity threats, from malicious code risks to insider threats. This includes
response testing, handling, monitoring, reporting, and IR training activities.
Planning (PL). Controls related to establishing a system security plan, security and
privacy architectures, concept of operations, and system user agreements.
Program Management (PM). Activities focused on the comprehensive oversight of the
information security and privacy programs, including resource strategies, workforce
management, architecture, risk management, and continuous monitoring.
Personnel Security (PS). Guidance and procedures related to determining the eligibility
of personnel to access facilities, systems, and information and the succeeding management of
personnel to include access agreements, personnel transfers, and access terminations.
Personally Identifiable Information (PII) Processing and Transparency (PT).
Controls applicable to authorizing the use of PII, acquiring PII collection consent, providing
appropriate PII notices, and managing PII across the information life cycle.
Maintenance (MA). Controls relevant to the upkeep of security tools, procedures for
offsite maintenance of system components, authorization of personnel who may perform
maintenance activities, and documentation of maintenance actions.
Media Protection (MP). Procedures specific to managing the life cycle of media,
including the access, marking, use, storage, transport, and destruction of media.
Physical and Environmental Protection (PE): Safeguards employed to protect
facilities, systems, and infrastructure from PE threats. These controls enable access to emergency
power and lighting, maintenance of physical access to systems and facilities, and provisions for
fire and environmental control systems.
Supply Chain Risk Management (SR). Actions taken to minimize supply chain threats,
to include system component acquisitions strategies, verification of the authenticity of
components, diversity and resiliency of suppliers, and supply chain transparency and traceability.
Access Control (AC). Enforcement of policies and procedures, based on the least
privileged principle, dedicated to ensuring only authorized users have access to the system, and a
level of access to the system commensurate with their workforce role. Activities include
administering accounts, logging and maintaining access events, controlling remote and wireless
access, and managing the metadata of security attributes.
Audit and Accountability (AU). Systematic collection and review of information system
logs to ensure proper management of auditable information throughout its life cycle.
Configuration Management (CM). Policies and activities specific to the configuration
of a system, to include establishing and documenting a system baseline, analyzing security
impacts for proposed configuration changes, implementing component and software changes,
verifying configuration settings, and maintaining hardware and software inventories.
Identification and Authentication (IA). Management of the policies, procedures, and
credentials employed to authorize system users, devices, and services.
Risk Assessment (RA). Strategies and procedures corresponding to the categorization of
the system (i.e., impact levels), assessments of system risk, and scanning for system
vulnerabilities.
System and Services Acquisition (SA). Focused on acquiring a system, the system
development life cycle, engineering privacy and security concepts into the system, managing
external system services, and requiring the system developer to test and evaluate the
implementation of controls.
System and Communications Protection (SC). Securing a system through such
measures as boundary protection, denial-of-service protection, securing transmissions, secure
coding, wireless link protection, and securing data-at-rest.
System and Information Integrity (SI). Controls focused on remediating system
security flaws, continuously monitoring a system, verifying software and information integrity,
and protecting a system with best-practice tools, such as malicious code and spam protections.
Lastly, NIST mapped the controls to international security standards to help facilitate the
applicability and usability of the controls.
NIST SP 800-53A, Rev 5, Assessing Security and Privacy Controls in Information Systems and
Organizations
As a supplement to NIST SP 800-53, NIST (2022a) authored detailed assessment
objectives and methods for every security and privacy control to assist an organization with
evaluating their compliance with the controls. These assessment elements enable an organization
to comprehensively examine and test controls for effectiveness and inform system and
organizational risk determinations. The publication is particularly useful during Step 5 (i.e.,
assessing controls) of the NIST RMF process.
NIST SP 800-53B, Control Baselines for Information Systems and Organizations
Understanding the 20 security and privacy control families identified in NIST SP 800-53,
NIST (2020b) developed security and privacy control baselines to help the federal government
select security controls for implementation proportionate to each system based on organizational
risk tolerance. First published in 2005, NIST delineated the security control baselines into three
categories (i.e., low, moderate, and high) based on FIPS Publication 199 definitions. Generally,
the higher the risk categorization of the system, the greater the number of controls an
organization implements for a system.
Security control tailoring and security overlays may also add or subtract from the overall
number of controls implemented by an organization. Tailoring controls enable organizations to
remove controls based on mission requirements. Conversely, security overlays identify additional
security and privacy controls for implementation based on further categorizing the system (e.g.,
classified systems, intelligence-based systems).
NIST SP 800-128, Guide for Security-Focused Configuration Management of Information
Systems
In this guidance, NIST identified a basic configuration management cycle: planning,
identifying/implementing, controlling, and monitoring system configuration changes (NIST,
2019). The four-phase cycle concentrates on recommended security activities throughout the
continuous cycle. Consistent with NIST SP 800-37, the system administrator implements the
organizational-defined system secure baseline configurations.
NIST SP 800-181, Rev 1, Workforce Framework for Cybersecurity (NICE Framework)
The NICE Framework is a keystone cybersecurity workforce publication and provides a
common language to describe and discuss the cybersecurity workforce across all types of
organizations (2020c). The Framework was a collaborative effort between Government, industry,
and academia, and is an applicable reference across the cybersecurity workforce continuum, from
students to accomplished practitioners.
The essence of the Framework is the development of Tasks, Knowledge, and Skill (TKS)
statements to inform competencies. NIST-proposed Skills and Knowledge statements describe
learner actions, whereas Task statements are defined by an organization. For example, for the
system administrator work role, a NIST-developed Skill statement is “skill in configuring
software,” and a Knowledge statement is “knowledge of system availability measures” (NIST,
2022b, “Knowledge and Skills” spreadsheet). An organization may define a Task statement as,
“configure a firewall.” Of note, NIST revamped its previous construct of Knowledge, Skills, and
Abilities (KSA) statements into the TKS structure. NIST did this by refashioning Skills and
Ability statements into Skills statements.
Additionally, a collection of like TKS statements may be grouped together in the TKS
hierarchy to inform a competency. NIST suggested an organization could then assess a learner
based on these grouped competencies, and specifically the collection of Knowledge and Skills
statements, to form a credential. In the current version of the Framework, the competencies and
work roles were decoupled from the NICE Framework document into separate documents to
improve the management and currency of these resources. As a part of the NICE Framework
revisions, NIST authored National Institute of Standards and Technology Internal Report
(NISTIR) 8355 to complement the NICE Framework (Wetzel, 2023). The Report provides an
explanation of the TKS construct and an explanation of why TKS was introduced into the NICE
Framework, Revision 1.
Finally, on the world stage, other international entities have also found a need to define a
common lexicon and framework, like the NICE Framework, to understand and advance the
development of cybersecurity workforce roles, competencies, knowledge, and tasks. For
example, the European Union Agency for Cybersecurity (2022) developed the European
Cybersecurity Skills Framework, the Australian government constructed the Australian Defence
Force Cyber Skills Framework (2020), and the Chartered Institute of Information Security
(CIISec) (2019), a United Kingdom Royal Charter of Incorporation organization, developed the
CIISec Skills Framework.
Federal Information Processing Standards
FIPS Publication 199, Standards for Security Categorization of Federal Information and
Information Systems
This NIST guidance is central to assisting cybersecurity practitioners with categorizing
information and information systems. These categorization decisions are formulated during Step
2 (Categorize) of the RMF (NIST SP 800-37) process. U.S. government agencies use this
guidance to ensure a standardized categorization method. This approach also promotes effective
management and reporting to the U.S. OMB and Congress (NIST, 2004).
Security categorizations are based on the impact on an individual or organization, should
the information and information system be compromised. Categorizations are based on three
security objectives. FISMA (2014) defined the three objectives as confidentiality, integrity, and
availability (i.e., the CIA triad). NIST further identified three potential impact levels for
information and information systems. The three impact assessments include low, moderate, and
high (NIST, 2004).
In the event of an unauthorized disclosure, modification, or access to the information or
system, the impact would be considered low if the event would impose a limited adverse effect
on the individual or organization. The impact would be regarded as moderate if the unauthorized
event imposed a serious adverse effect on the individual or organization. Furthermore, an impact
would be considered high if the unauthorized event imposed a severe or catastrophic adverse
effect on the individual or organization. In totality, these information types and information
system categorizations, coupled with threat and vulnerability information, assist in determining
organizational risk.
FIPS Publication 200, Minimum Security Requirements for Federal Information and
Information Systems
FIPS PUB 200 also informs decisions made during Step 2 (Categorize) and, moreover,
Step 3 (Select) of the RMF process. This NIST guidance seeks to facilitate a consistent approach
to selecting controls to meet minimum security thresholds (NIST, 2006). Across the security and
privacy control families described in NIST SP 800-53, FIPS PUB 200 enumerates the
minimumsecurity requirements to be implemented for information systems. Additionally, and
essentially, the publication prescribed organizations must minimally implement a low, moderate,
or high baseline of controls for their information system. The baseline selection is consistent with
the impact categorization of the information system (i.e., low-, moderate-, or high-impact). The
baselines are identified by control in NIST SP 800-53.
Current Findings
As previously stated, this research focused on cybersecurity professionals protecting U.S.
federal government information and information systems. Given this center of interest, the
paucity of empirical research focused on the U.S. federal workspace was noted during the
literature review. Despite the limited literature concentrated within the federal government
domain, there were literary discoveries relevant and generalizable to the Government sector.
The findings identified are thematically organized in this section. The three primary focus
areas include the implementation of controls, cybersecurity certifications of persons, and
cybersecurity workforce experience. The implementation of security and privacy controls was a
literature review concentration area because the nucleus of the study resided within the context of
the NIST RMF Implementation step task (i.e., Task I-1 – Control Implementation) (NIST, 2018).
Furthermore, the cybersecurity certifications of persons and workforce experience were research
concentration areas, as these areas were the two predictor variables representing cybersecurity
competency.
Implementation of Security and Privacy Controls
The CIA triad is at the foundation of any security safeguard (Alsaqour et al., 2021;
FISMA, 2014). Zaini et al. stated adequate security and control procedures are required to sustain
the CIA of information systems (2020). Zaini et al. further remarked that the effective
implementation of security controls will help protect information systems, thus preserving the
CIA of system components, including software, hardware, networking devices, data, etc. (2020).
As early as 1972, resemblances of current security controls were already being advocated
as necessary elements of computer security. Broad security controls in the form of personnel
security, contingency planning, physical security, secure coding, and access controls were
inceptive security control considerations (Anderson, 1972). The review of the literature
confirmed security controls have increased in complexity and volume. A few solutions have also
been proposed to ease the complexity of implementing security controls.
Complexity of Implementation
While NIST advocated and suggested recommendations for simplifying, innovating, and
automating tasks to execute the RMF (NIST, 2018), Goel et al. noted the RMF, CSF, and similar
frameworks require an extensive collection of information to facilitate the use of the frameworks
(2020). Moreover, the authors believed the frameworks are primarily effective for day-to-day,
lower organizational-level risk management decisions. Due to these shortcomings, the
researchers alternatively developed a framework to assess an organization’s cybersecurity
posture at a strategic level. The proposed framework consists of five assessment components:
prioritize, resource, implement, standardize, and monitor (PRISM). In the context of the PRISM
framework, the implementation component was described as an organization having the skills
and assets available to defend its systems from present and future cyber threats. The implement
component also included security awareness training, malware detection and eradication, and
vulnerability management.
Hale and Gamble (2019) also recognized the regulatory requirements, business criticality,
and complexity of implementing cybersecurity standards (e.g., NIST SP 800-53 and ISO/IEC
Common Criteria) for systems. In their research, the authors proposed a process to derive
security requirements using logical relationship models and hierarchies. One of their two case
studies used NIST SP 800-53 (i.e., NIST RMF) and related security documents to evaluate their
proposed process. Based on the feedback from subject matter experts, the research results
indicated the process would be favorable for extracting security baseline security certification
requirements.
Analogous implementation challenges were identified during an academic review of
ISO/IEC 27001. ISO 27001 is a well-known international information security framework
comparable to the NIST RMF, COBIT, and HITRUST (Culot et al., 2021). Culot et al. reviewed
ISO/IEC 27001 and the literature review included 96 documents, downselected from 537 initial
records, published over 15 years. A content analysis of the 96 contributions led the authors to 5
overarching findings. Within one of the finding themes, framework implementation, the
researchers noted the ISO 27001 framework as broad, structured, comprehensive, and overall,
challenging to implement and assess the security controls. One-hundred and thirty-three controls
were identified in the 2005 revision of the ISO 27001 framework. This is half of the controls
prescribed in the 2005 version of the comparable NIST RMF, specifically NIST SP 800-53
(NIST, 2005). Moreover, the 2020 version of NIST SP 800-53 prescribes 20 security and privacy
control areas, covering 1,189 controls and control enhancements (2020a). This is an increase of
88% over the 2005 version of the ISO 27001 framework that Culot et al. (2021) noted as being
challenging to implement.
Salminen (2019) affirmed the similar findings Culot et al. (2021) identified in their
research on ISO/IEC 27001. Salminen noted the complexity of cybersecurity standards, and the
growing number of security controls create weaknesses in system certification programs and
managing security (2019). For example, employing auditors with the competence to audit a
standard for an entire enterprise is challenging. Often, cybersecurity professionals are well versed
in a few cybersecurity domains and do not possess the same high level of competence across all
areas.
The size and organizational complexity also contribute to the arduous task of
implementing security and privacy controls across an organization. Nifakos et al. (2021)
analyzed the considerations impacting the security posture of healthcare organizations. Their
study included a review of 70 journal articles. The authors highlighted the primary reason most
healthcare facilities struggled with implementing cybersecurity controls was the large and
complex nature of these healthcare organizations. Furthermore, Nifakos et al. identified the lack
of cybersecurity-related roles as a contributing factor to an increasing number of data breaches
experienced in the healthcare sector (2021).
Like the U.S. government, the healthcare sector also supports stand-alone systems and
legacy infrastructure. However, the challenge is not confined to legacy infrastructure or
standalone systems. As acknowledged in a 2022 U.S. Department of Health and Human Services
(HHS) Inspector General (IG) report, the IG uncovered the Indian Health Service failed to
implement select controls before the deployment of their telehealth system (U.S. Department of
HHS, 2022).
Overall, the literature review findings noticeably highlighted well-known control
frameworks are complex to implement. When discussing the hundreds of security controls
contained in some of the common control frameworks (e.g., NIST SP 800-53, ISO/IEC 27001,
COBIT), Blum (2020) commented, “Even larger organizations and experts struggle to see the
forest for the trees” (p. 161). This complexity was also evidenced in the varied approaches used
by organizations to implement cybersecurity programs and security controls.
Implementation Approaches
Overall, information security management (ISM) is implemented using process-based
guidance (Diéguez et al., 2020). Security controls, also illuminated by Diéguez et al., are
typically implemented using qualitative methods. In their research, Diéguez et al. summarized
cybersecurity is mainly approached using risk management methodologies, human behavior
practices (i.e., efforts focused on promoting and implementing a corporation’s security policies),
or security maturity models. Dedeke and Masterson (2019) also noted the gravitation towards
structuring cybersecurity frameworks based on risk management approaches when they analyzed
frameworks from Australia, the U.S., and the United Kingdom.
One of the process-based approaches analyzed by Alsaqour et al. (2021) was the
defensein-depth approach commonly used by organizations to secure information and
information resources. This multilayered security approach is credited mainly to the U.S.
military. Alsaqour acknowledged this defense-in-depth approach to security is challenging to
implement.
Organizations must possess the personnel with the requisite skills, experience, and knowledge to
integrate the necessary configurations across the security layers while adhering to security
policies and standards. Additionally, the authors noted that unsuccessful implementation at any
security layer may lead to introducing new security threat vectors.
Dombora (2019) also sought to understand the approach to administering a requisite,
process-based Information Security Management System (ISMS) effectively. Accordingly, the
researchers developed and tested guidelines to assist organizations with implementing an ISMS.
As a component of the study’s research methodology, Dombora noted several problems in using
an ISMS. Of the many challenges captured, the author identified a lack of professional
knowledge, exhaustive lists of security rules, and lengthy information security guidance as
obstacles to implementing an ISMS. Eight organizations implemented an ISO/IEC 27001-based
ISMS to evaluate the author's implementation recommendations with the author-developed
guidelines. Dombora concluded the guidelines were acceptable; however, the guidelines could
use some refinement.
In the same vein as Dombora’s research, Zammani et al. (2019) conducted a survey to
identify what factors were related to the successful implementation of ISM. The survey included
243 participants representing public- and private-sector organizations in Malaysia. Public
agencies employed 93% of the survey respondents. The authors’ results substantiated 14 main
components, comprising 45 sub-components, contributed to the successful implementation of
ISM. The highest-rated factor was “Top Management,” closely followed by policy, procedures,
and “ISM Team,” rounding out the top four success factors (p. 387). Additionally, within the sub-
components of the ISM Team, domain knowledge, and information security skills were also
strongly rated success factors. Granneman (2018) also acknowledged and suggested the need to
coalesce an organizational cybersecurity implementation team to execute the information
security strategy. Granneman highlighted this step is often overlooked.
Information Security Risk Management (ISRM) is another qualitative, process-based
approach to implementing cybersecurity. Lundgren (2020) explored what capabilities were
resident within ISRM. Lundgren’s study was grounded in a theoretical framework consisting of
four risk management components resembling the NIST RMF 7-step process. The components
consisted of identifying, prioritizing, implementing, and monitoring. The author believed the four
cyclical components were informed by capability, a function of intent (to do) and knowing (how
to do). The research consisted of a literature review comprising the analysis of 27 journal articles.
Regarding the implementation of control objectives, a general focus of this research, Lundgren
noted the intention for security control may largely vary. With implementation intent varying on
a spectrum, even within the same agency, there was an unbalanced applicability of knowledge
resulting in an impact on the efficacy of the implementation of security controls.
Park et al. (2021) also focused on qualitative and less technical approaches to
implementing cybersecurity. The authors investigated how elements of the 7S model (i.e., shared
values, strategy, structure, systems, staff, style, and skills) impacted the capabilities of an
organization to protect its technology. The authors surveyed employees of small, medium, and
large enterprises in the Republic of Korea. One of the hypotheses was formulated around the
skills attribute of the 7S model, and generally evaluated the competency and skills (i.e.,
expertise, professional certification, training) of the personnel responsible for technology
protection for their organization. Based on 435 surveys collected, the skills component was only
found to influence the protection of technology for medium- and large-sized organizations. The
authors posited this may be the case because small organizations do not perceive the need for
complex security technologies. Also, a noted limitation of the study was the survey population
included all employees, not just security practitioners.
Central to implementing process-based approaches to cybersecurity, Masrek et al. (2020)
studied the relationship between cybersecurity threats and the efficacy of information security
policy. Policy was their primary research focus, as Masrek et al. garnered from literature that
information security policy was a predominant form of administering security. Their study
involved surveying federal employees across the Malaysian government. From the analysis of
292 useful questionnaire responses, Masrek et al. confirmed their hypothesis that information
security policy was effective in reducing cybersecurity threats.
Diéguez et al. (2020) acknowledged the nascent quantitative approaches to the
management of information security. Consequently, Diéguez et al. proposed a model to tackle the
challenge of selecting and implementing security controls. The author’s model considered
variables such as nested controls, dependencies of security controls, etc. The authors posited a
quantitative and operational research-framed conceptual model to map variations for
implementing security controls. The approach was examined using security controls for a
Chilean government agency and proved to be a useful tool. Optimizing the selection of security
controls (i.e., RMF Step 3) may assist with implementing controls (i.e., RMF Step 4).
Ekelund and Iskoujina (2019) also considered a quantitative approach to cybersecurity. In
particular, the authors studied the optimization of cybersecurity investments and the protection of
assets. During their research to develop a new method to approach security expenditures,
Ekelund and Iskoujina discovered an increase in the research area of cybersecurity economics.
However, research was far less prolific regarding investments in new security safeguards to
strengthen security controls.
Lastly, Diesch et al. (2020) attested previous efforts to view and implement information
security solely through a technical lens were unsuccessful. In their research, the authors also
documented the absence of studies focused on the indicators of successful cybersecurity
implementation factors. These findings were consistent with the research of Angraini and
Okfalisa (2019) and Zammani et al. (2021).
Zammani et al. (2021) asserted the implementation of ISM within organizations remained
at a low readiness level based on increased cybersecurity intrusions, despite organizations
satisfying ISM requirements. The authors additionally posited that current maturity assessment
models did not take a comprehensive approach to assessing information security practices within
an organization. As such, Zammani et al. conducted a mixed-methods study of security maturity
models to ascertain the elements needed to evaluate the efficacy of ISM. A systematic literature
review (SLR) of the success factors of security maturity models confirmed models were not
comprehensive and often favored assessing technology factors moreover people factors.
Certification of Persons
Cybersecurity is a complex and broad discipline, requiring cybersecurity practitioners to have
specific skills and knowledge across a breadth of cybersecurity areas (Marquardson &
Elnoshokaty, 2020; Ramezan, 2023; Salminen, 2019; Zammani et al., 2019). Domains across the
cybersecurity field intersect, as do the skills, expertise, and knowledge of the individuals
assigned to the various cybersecurity roles (Jarocki & Kettani, 2019).
To validate an individual’s cognitive expertise, professional workforce certifications are
suggested to offer a level of confidence to an employer regarding an individual’s knowledge
(Ramamonjiarivelo et al., 2020; Salminen, 2019; Selamat & Ab Halim, 2021) and provide a
mechanism for assessing an individual’s knowledge (Harrack, 2021; James & Callen, 2018;
Kavosa et al., 2022). Professional certifications are also asserted to add credibility to the
organization and individual (Anderson et al., 2021; James & Callen, 2018; Ramamonjiarivelo et
al., 2020). Additionally, Mbise (2021) maintained information technology certifications provide a
means to remain updated on professional skills and certifying professionals is an organizational
best practice.
Moreover, cybersecurity certification of persons continues to be a pertinent workforce
topic. Noche (2021) aimed to determine the insistent challenges of developing the cybersecurity
workforce. Conducting a SLR of empirical evidence, the author filtered research articles to 24
relevant, peer-reviewed papers. The results of the study identified cybersecurity workforce
certifications as one of the top eight issues in cybersecurity employee development.
The relevancy of certifying the cybersecurity workforce was also noted worldwide. Teoh
and Mahmood (2018) explored what elements of the cybersecurity workforce were being
developed by leading countries around the globe. Specifically, countries poised to take advantage
of new technologies in the virtual economy. Using a World Economic Forum ranking to identify
these leaders, the researchers analyzed nine of the ten top countries in the world. Teoh and
Mahmood then analyzed the national cybersecurity strategy of each country. The authors
identified six main cybersecurity workforce development elements based on their analysis. One
of the main elements was certifications; notwithstanding, only four countries (Finland,
Singapore, the U.S., and the United Kingdom) had certifications as a notable element in their
national cybersecurity strategy.
Industry Cybersecurity Certifications
Are the organizations offering premium-priced cybersecurity certifications ‘self-licking
ice cream cones’? The efficacy and usefulness of cybersecurity certifications are debatable
(Kappers & Harrell, 2020; Salminen, 2019). Certifying organizations may benefit from the
imprimatur of the certification holders.
In 1989, Novell, a computer systems company, created the first information technology
certification. Novell introduced the vendor-specific Certified Novell Engineer certification to fill
a void created between the higher learning education systems and working information
technology practitioners (Adelman, 2000).
Since Novell introduced the first certification, the number of information technology and
cybersecurity-related workforce certifications have increased significantly. This growth was
evidenced by a review of certifications identified on the U.S. Credentials Center website. The
Credentials Center, hosted on the CareerOneStop website sponsored by the U.S. Department of
Labor, is a resource for anyone seeking information about approaches to achieve their training
and education goals (U.S. Department of Labor Employment and Training Administration,
2024). Searching the Credentials Center, using the Certification Finder tool, the “Information
Security Analysts” occupation returned a list of 147 certifications and related certifications
offered by 35 organizations. Of the 147 certifications, 39 were categorized as
“product/equipment” specific. The same type of search on the “Network and Computer Systems
Administrators” occupation, a work role noted as having a responsibility to implement security
and privacy controls, identified 217 certifications from 39 organizations. Of the 217
certifications, 187 were categorized as “product/equipment” specific.
Jarocki and Kettani (2019) asserted that vendor-agnostic certifications are perceived to be
more valuable to potential employers than their vendor-specific counterpart certifications. Jarocki
and Kettani also stated this may especially be the case in instances where an individual has
earned a vendor-specific certification, but the specific software or tool is not used within the
enterprise.
The proliferation of certifications was also identified by a pair of researchers in their
longitudinal study of the information technology workforce. For 15 years, Cummings and Janicki
have conducted a survey of information technology and system practitioners across the U.S.
every 2 years (2021). The aim of the surveys was to identify technological trends to assist
academia with revising or designing relevant college courses. Over 500 workers responded to the
author’s survey conducted in 2020. The number of survey respondents who held at least one
information technology/system certification grew over 27 percentage points over the last 2 years,
rising to 89% in 2020. The top information technology certification held by survey participants
was Microsoft Certified Solutions Expert (MCSE), followed by Computing Technology Industry
Association (CompTIA) A+ and Cisco Certified Network Associate (CCNA). The survey results
appeared to indicate an increasing value, perceived or otherwise, of earning certifications.
Jarocki and Kettani (2019) also recognized the vast number of cybersecurity certifications
available to cybersecurity practitioners. Given the broad certification landscape, Jarocki and
Kettani examined if commercial certifications could be an effective measure for hiring security
analyst candidates, specifically practitioners filling incident responder roles.
Based on their qualitative analysis, the authors concluded that a certification inclusive of all
incident response responsibilities was not supported. However, they acknowledge the “supposed
worth” of obtaining a certification for security analysts (p. 4).
Considering the spectrum of cybersecurity workforce roles, the numerous workforce
cybersecurity certifications available, and the subjectivity of ranking these certifications as more
or less valuable, the researcher did not find an empirical ranking of cybersecurity certifications.
However, as early adopters of mandating cybersecurity certifications for the DoD workforce, the
DoD maintained a list of certifications based on workforce role categories (U.S. DoD, n.d.; U.S.
DoD, 2015). Additionally, the U.S. Cybersecurity and Infrastructure Security Agency (CISA),
through its National Initiative for Cybersecurity Careers and Studies (NICCS) initiative,
compiled a list of industry cybersecurity certifications (U.S. CISA, 2021).
A sampling of workforce cybersecurity certifications is shown in Table 1. The sample is
based on the list of certifications compiled by the DoD, CISA, and frequently acknowledged
certifications noted during the study’s literature review (Marquardson & Elnoshokaty, 2020;
Peslak & Hunsinger, 2019; Ramezan, 2023; U.S. CISA, 2021; U.S. DoD, n.d.; U.S. DoD, 2015).
Table 1
Sampling of Cybersecurity Certifications
Certifying Exam Fee
Certification Entity (U.S.) Work Experience Required Source
Certified in
Governance, Risk
and Compliance
(CGRC)
(ISC)² $599 ≥ 2 years of cumulative, paid
work experience in ≥ one
domain in CGRC CBK
(ISC)², 2023
Certified Cloud
Security
Professional (CCSP)
(ISC)² $599 ≥ 5 years of cumulative, paid
work experience in
information technology; 3
years in information security
and ≥ 1 year experience in ≥
(ISC)², 2023
one domain in CCSP CBK
Certified
Information Systems
Security
Professional
(CISSP)
(ISC)² $749 ≥ 5 years of cumulative, paid
work experience in ≥ two
domains in CISSP CBK
(ISC)², 2023
Systems Security
Certified
Practitioner (SSCP)
(ISC)² $249 ≥ 1 year of cumulative, paid
work experience in ≥ one
domain in SSCP CBK
(ISC)², 2023
Cisco Certified
Network Associate
(CCNA)
Cisco $300 Recommended: ≥ 1 year of
experience implementing
and administering Cisco
solutions
Cisco, 2022
Security+ CompTIA $392 Recommended: ≥ 2 years of
experience in information
technology administration
(security focused)
CompTIA, n.d.
CompTIA Advanced
Security Practitioner
(CASP+)
CompTIA $494 ≥ 10 years of general
handson information
technology experience, with
≥ 5 years of broad hands-on
security experience
CompTIA, n.d.
Certified Network
Defender (CND)
EC-Council $550 ≥ 2 years of experience in
information security
EC-Council,
2022
GIAC Security
Essentials (GSEC)
GIAC $949 None GIAC, 2022
GIAC Security
Leadership (GSLC)
GIAC $949 None GIAC, 2022
Certified
Information Security
Manager (CISM)
ISACA $575
members;
$760
nonmembers
≥ 5 years of information
security management work
experience, in accordance
with CISM CBK
ISACA, 2022
Note. (CBK) Common Body of Knowledge; (CompTIA) Computing Technology Industry
Association; GIAC (previously Global Information Assurance Certification); (ISACA)
Information Systems Audit and Control Association; (ISC)² International Information System
Security Certification Consortium.
Obtaining and maintaining industry certifications can be expensive. As sampled and
identified in Table 1, certifying organizations typically assess a fee to attempt a certification
examination. Generally, there is also a nominal, non-refundable fee for applying for a
certification, for example, $50 (ISACA, 2022) to $100 (EC-Council, 2022). Additionally,
examination preparation options vary and depend on the learning method. For example, a printed
Security+ Study Guide was priced at $174 (CompTIA, n.d.), CND electronic courseware was
advertised at $550 (EC-Council, 2022), and an on-demand course for GSEC was priced at $8,275
(GIAC, 2023). Furthermore, certification organizations commonly charge certification holders a
yearly certification maintenance fee. The maintenance fees vary, for example, from $45 to $125
per year (ISACA, 2022; (ISC)², 2022).
Moreover, for those who obtain a certification, the certification holder commonly must
earn continuing professional or education units every year to actively maintain their certification.
The number of education credits or units a person must acquire varies by certification. For
example, CASP+ certification holders must earn at least 50 continuing education units within a
3-year period (CompTIA, n.d.). In contrast, a CISSP or CISM holder must earn a minimum of
120 continuing professional education hours over a 3-year reporting cycle (ISACA, 2022; (ISC)²,
2022).
Intermediate and advanced cybersecurity certifications frequently have an experience
requirement associated with the award of the certificate, as well. There is a range of experience
requirements. For instance, for the award of a CASP+ certification, the applicant must have had a
minimum of 10 years of general hands-on information technology experience, including at least
5 of those years applying to broad, hands-on security experience (CompTIA, n.d.). Another
certification, the (ISC)² CISSP, requires a minimum of 5 years of cumulative, paid work
experience in at least two of the CISSP domain knowledge areas (2022).
From a U.S. federal government perspective, NIST drafted supplementary guidance
(NISTIR 8193) to the NICE Cybersecurity Workforce Framework to develop work-role
indicators. Certifications, in part, were considered and adapted into the guidance. The indicators
are used to illustrate a worker can perform a particular cybersecurity role. Working with 31
subject matter experts from 16 U.S. federal agencies and departments, NIST identified and
defined 5 work role capability indicators: education, training, certification, experiential learning,
and continuous learning (NISTIR 8193, 2017b). The five capability indicators are defined across
three proficiency levels (i.e., entry, intermediate, and advanced).
For the 42 work roles with developed capability indicators in NISTIR 8193,
credentials/certifications are recommended for 36% of the entry-level roles, 74% of the
intermediate-level roles, and 67% of the advanced-level roles. Where credentials/certifications
are either recommended or noted as beneficial, NIST provided a list of topics the certification
should address. However, NIST did not identify specific certifications (e.g., CISSP, CISM,
Security+, etc.) a worker should strive to earn. Lastly, while certifications are recommended for
many of the work roles described in the NISTIR, this effort may still fall short of assisting
federal agencies with consistently defining a list of appropriate cybersecurity certifications for
their workforce.
This lack of guidance, coupled with federal agencies inconsistently defining their
cybersecurity positions, was a noted deficiency by the U.S. GAO (2019b). Additionally, the U.S.
GAO (2024) noted that the lack of clear and well-communicated cybersecurity roles and
responsibilities contributed to ineffective information security programs.
Furthermore, certification requirements vary for personnel assigned to U.S. federal
government agencies. The U.S. OPM has not mandated certification requirements for the
cybersecurity workforce across the U.S. government (U.S. OPM, 2018). Select organizations
across the federal government have elected to mandate certifications for their cybersecurity
workforce. For example, the U.S. DoD has mandated cybersecurity certifications for all U.S.
DoD civilian, military, and support contractors assigned to cybersecurity workforce positions
(U.S. DoD, 2015; U.S. DoD, 2020). Conversely, the U.S. National Aeronautics and Space
Administration, for example, does not require employees in positions performing cybersecurity,
cybersecurity-related, or information technology work roles to possess a professional
certification (U.S. GAO, 2019a).
Certification Practices
From a global vantage point, the International Organization for Standardization (ISO) and
the International Electrotechnical Commission (IEC) are non-governmental entities, and together
they develop international standards for a wide breadth of products and technology. Part of their
work involved working with national-member groups of the ISO or IEC to develop a
conformityassessment based standard detailing the general requirements for organizations
(certification bodies) who seek to certify people. The Standard (ISO/IEC 17024:2012,
Conformity assessment – General requirements for bodies operating certification of persons) was
developed to ensure certification bodies administer and issue certifications in a dependable,
consistent, and equivalent manner (ISO/IEC, 2012). As acknowledged in the certification
Standard, examinations are part of the assessment process and seek to measure a candidate’s
ability to apply the knowledge and skills (i.e., competence) to meet an objective.
Many organizations have adopted and complied with the conformity assessment approach
developed by the ISO/IEC and codified in the ISO/IEC 17024 Standard. Regardless of this
worldwide acceptance, Kavosa et al. posited certifications do not necessarily validate an
individual’s adherence to the competency standards within the certification’s corresponding
profession, domain, or specialty (2022). The authors further asserted certification bodies
predominately emphasize the assessment of knowledge and acquired experience. This is
evidenced in many cybersecurity certification examinations, as the examinations use
multiplechoice questions to assess the requisite cybersecurity knowledge. However, in the
perspective of Bloom’s Taxonomy of Educational Objectives (Bloom et al., 1956), certifications
based on multiple-choice question assessments generally measure a lower classification of
learning outcomes, for example, knowledge and comprehension (Salminen, 2019).
Additionally, Kavosa et al. (2022) postulated the widely accepted conformity assessment
approach should be remodeled into a process focused on the certification and continuous
monitoring of professional competence (i.e., knowledge, skills, and abilities). Based on a
qualitative literature analysis and a survey of 673 certified construction specialists, Kavosa et al.
proposed integrating competency-based activities into the certification process. Their suggested
process included trinity involvement, including the certification body (standardizes measures of
competencies), the applicable industry (stakeholder involvement; continuous monitoring of
competencies), and a supervisory authority (defines competencies), to produce a certified
individual. While part of the research involved the responses of construction specialists to
ascertain what competencies were important and necessary for their tradecraft, the authors noted
the literature analysis was widely inclusive across industries.
Ghosh and Francia (2021) also advocated for competency-based assessments, as the
authors noted the gap between the cybersecurity graduate needed in the field versus the skills and
knowledge imparted upon graduation from collegiate programs. James and Callen (2018) further
maintained cybersecurity certifications are reactive to industry requirements, offering
leadingedge skills training to the cybersecurity workforce.
The movement towards assessments of competencies has been noted. Kappers and
Harrell (2020) acknowledged the cybersecurity certification industry has improved the
robustness of their certifications by including hands-on skills testing as part of some certification
assessments. This hands-on testing, coupled with proven years of experience in
certificationrelevant professional roles, may boost the acceptance and credibility of cybersecurity
certifications.
Finally, other non-U.S. organizations have recognized the need for certification and
accreditation methods to validate the competencies of cybersecurity practitioners. For example,
in the United Kingdom, the Council for Registered Ethical Security Testers (CREST) organizes
cybersecurity examinates to confirm the competencies of security practitioners evaluated by the
United Kingdom’s Government Communications Headquarters (Caron, 2021). CREST (2022) is
an international non-profit organization; the membership body has accredited nearly 300
companies and has certified thousands of security professionals.
Influence of Cybersecurity Certifications
In the context of what appears to influence the employability of cybersecurity
professionals, and arguably a perceived value to the employer, is what skills, certifications, and
degrees are of interest to organizations. Analyzing the advantages of formal education compared
to cybersecurity certifications is not within the scope of this research. However, Marquardson
and Elnoshokaty (2020) explored what certifications, college degrees, and skills employers
sought in applicants who were seeking to start a career in cybersecurity. Of interest, the
researchers analyzed 11,938 cybersecurity-related job listings on a popular employment search
website. Of the entry-level cybersecurity jobs listed, three of every five advertised positions
required a college degree in an information technology-related field. Comparably, 29% of the
employers wanted candidates to possess a cybersecurity-related certification. In the study, the top
three cybersecurity-related certifications sought by prospective employers were CISSP,
Information Technology Infrastructure Library (ITIL), and Security+.
In a similar analysis of 487 unique job postings for a “cybersecurity analyst” in 2019,
Peslak and Hunsinger cited the top four cybersecurity certifications included in the job
advertisements were the CISSP, CISM, GIAC, and Certified Ethical Hacker (CEH) credentials.
Furthermore, these findings were consistent with a 2023 examination of 935 cybersecurityrelated
positions on a popular job website. The results indicated the four most frequently cited
credentials were the CISSP, Security+, CISM, and CEH certifications (Ramezan, 2023).
While college degrees remain relevant, skills, experience, and certifications remain
preferable qualifications for entry-level positions. Marquardson and Elnoshokaty (2020) also
acknowledged employers are increasingly seeking candidates with cybersecurity certifications.
Based on scholarly studies of cybersecurity-related job listings in 2017 and 2019, Marquardson
and Elnoshokaty noted that job requisitions requiring certifications increased by nine percentage
points between the two studies. Harrack (2021) asserted certifications positively impacted hiring
decisions and earning potential. Gough and Kyle (2019) also generally accepted certifications as
a supportive, differentiating measure and assist workforce candidates with securing an entrylevel
role.
Furthermore, Kappers and Harrell (2020) also recognized the disparity between collegiate
programs and the knowledge of security practitioners. In their research to propose a new
framework to examine information security skills, specifically C-Suite skills, the authors posited
many security practitioners can use security certifications to bridge their post-academic studies
with the skills and knowledge they need in the field. Furthermore, as illustrated by the authors’
research into the Chief Information Security Officer (CISO) role, both degree and cybersecurity
certifications continue to be listed as requirements in position requisitions.
In summary, Zahadat (2019) posited most people would find it reprehensible for a lawyer
or medical doctor not to be credentialed prior to practicing in their profession. James and Callen
asserted a similar viewpoint for certified public accountants and lawyers (2018). Perhaps, in the
future, U.S. federal or state credentialing of cybersecurity professionals may also become an
expectation of the public.
Cybersecurity Workforce Experience
Carlton et al. (2019) remarked that a system is only as secure as the system’s most
penetrable point. Extending this observation and recognizing the human is often referred to as the
weakest link in an organization’s cybersecurity armor (Ani et al., 2019; Hatzivasilis et al., 2020;
Jeong et al., 2019; Rahman et al., 2021), the ability to protect an information system may only be
as strong as the weakest cybersecurity skill possessed by an individual or team. The broad nature
of cybersecurity compounds the human-capital challenge.
Cybersecurity is an expansive discipline, encompassing many domains and a wide range
of workforce roles (Blažič, 2022; Peslak & Hunsinger, 2019; Ramezan, 2023). The breadth of the
cybersecurity discipline is substantiated by the 20 security and privacy control families
prescribed by NIST (2020a), ranging from personnel security to system and communications
protection. Diesch et al. (2020) also illustrated the diversity of the discipline during their search
to ascertain ISM factors. During their research, Diesch et al. identified 12 success factors through
a SLR and additional inputs by subject matter experts. The 12 success factors included physical
security, vulnerability, infrastructure, awareness, access control, risk, resources, organizational
factors, CIA triad, continuity, security management, and compliance/policy. Consequently,
experienced personnel are needed to implement security and privacy controls across a wide
breadth of cybersecurity domains (i.e., categorically administrative, physical, and technical
controls).
Beyond a Technical Challenge
Kör and Metin (2021) asserted focusing only on technology could not exclusively
safeguard an organization’s information and information systems. The authors further posited
human factors should also be considered in protecting information assets. Dalal et al. purported
the technological aspects of cybersecurity are researched more than the human elements of
cybersecurity (2022). Jeong et al. (2019) also believed cybersecurity must be researched not only
from a technical perspective but also from a social point of view.
From this perspective, Jeong et al. (2019) aimed to increase their understanding of
cybersecurity human factors. Conducting a SLR of publications over 10 years (2009-2019), the
authors selected and synthesized 27 peer-reviewed studies. The studies were analyzed based on
three focus areas, including personality, demographics, and culture. Within the context and
subcategory of demographics, the results revealed 5 of the 27 articles noted prior cybersecurity
experience had a positive impact on overall cybersecurity awareness.
Rahman et al. (2021) affirmed Jeong et al.’s (2019) research premise and remarked that
researchers often overlook human factors in cybersecurity. Rahman et al. also suggested the
computer science community also frequently focuses on technical aspects of cybersecurity,
moreover human factors.
As a specific example of the lack of focus on human factors within the cybersecurity
domain, Zwilling (2022) researched the preparedness of CISOs to mitigate cyber threats. The
study was conducted using a text analysis of the results of a SLR, opinion articles related to
cyber threats, and cybersecurity vulnerabilities documented by the Open Web Application
Security Project. An analysis of a corpus of 3,742 documents revealed the literature was
predominately focused on researching the risk-mitigation constructs rather than the knowledge
and skills of CISOs.
Dawson and Thomson (2018) further affirmed the lack of human factors research in the
cybersecurity domain. The authors reviewed literature focused on cybersecurity workforce
development and expertise. While noting the scarcity of quantitative studies relating to
nontechnical cybersecurity workforce success factors, the authors hypothesized six assumptions
for developing the future workforce. The six attributes posited included the need for team
players, systemic thinkers, continuous learners, articulate communicators, mindfulness for civic
duty, and the possession of both technical and social skills.
The NICE Framework is also technology focused. One of the limitations of the NICE
Framework is the focus on technical TKS statements versus non-technical (e.g., leadership,
teamwork, and communication) competencies (Sussman, 2021). Sussman conducted a qualitative
study focused on identifying non-technical KSAs valuable to developing cybersecurity
professionals. To inform the study, the author interviewed 43 professionals in the U.S. assigned
to various cybersecurity roles. The results indicated a strong agreement among the interviewees
for the need to develop non-technical KSAs across the cybersecurity workforce. The study
participants also identified three overarching, non-technical activities the cybersecurity
workforce should develop: communication skills, critical thinking skills for decision-making, and
teamwork to solve customer challenges.
Furthermore, recent U.S. government guidance (i.e., the NIST CSF) promoted measures
to increase the cybersecurity defense of operational technology and critical infrastructure (e.g.,
water treatment facilities, dams, oil refiners, etc.). Chowdhury and Gkioulos (2021) sought to
identify skills and competencies required by cybersecurity practitioners protecting critical
infrastructure systems. The authors conducted a SLR, summarizing a total of 37 articles. In their
analysis, Chowdhury and Gkioulos discovered a multitude of skills and competencies needed by
the cybersecurity workforce within the critical infrastructure industry. The identified skills were
categorized as managerial, technical, implementation, and soft skills. Of interest to this research,
the authors noted implementation skills generally differentiate senior cybersecurity practitioners
from their less experienced counterparts. The authors believed this lacuna in skills may impact
the identification and use of security controls. Furthermore, the authors posited the NICE
Framework, while encompassing the identification of requisite knowledge and skills for a range
of cybersecurity roles, the Framework lacks in the areas of job descriptions, competencies and
training guidance, and efficacy metrics, to name a few deficiencies.
It appeared NIST previously acknowledged some of these deficiencies and sought
improvements. In their draft supplementary guidance to the NICE Cybersecurity Workforce
Framework, NIST identified five work role capability indicators (NISTIR 8193, 2017b). The five
capability indicators are experiential learning, education, training, certification, and continuous
learning. The capability indicators are defined across three proficiency levels, including entry,
intermediate, and advanced. While experiential learning (i.e., hands-on/on-the-job experience) is
identified as a positive measure, a majority of the 42 work roles do not have a targeted number of
years recommended across the three proficiency levels.
Perceived Experience Relevancy
Comparable to cybersecurity workforce certifications, employers appeared to perceive
experience as a valuable attribute and discriminating hiring variable. Recognizing the expanding
cyber threat aperture, Peslak and Hunsinger (2019) acknowledged the need for cybersecurity
experts and sought to identify what cybersecurity skills and experience are required of these
professionals. Using a moderately popular online job search engine conducive to data scraping,
the authors searched for the job title “cybersecurity analyst.” The researchers collected 487
distinct job requirements. Analyzing the job data, the authors’ noted experience was singled out
as a top requirement among employers; 48 percent of the employers sought to hire professionals
with 3 to 10 years of experience. Furthermore, “experience” was listed 2,341 times in the 486 job
descriptions. Second, to experience, skills related to RMF were equally sought after by
employers; variants of the term RMF had a high-frequency word count.
In a comparable analysis of 935 cybersecurity-related job announcements, 4 of every 5
positions noted an experience requirement of 5 or fewer years of experience (Ramezan, 2023).
Of the 935 positions, 18 percent required 6 or more years of experience.
Additionally, prior-related experience is perceived to be a highly conveyable knowledge
and skill. Dokko et al. (2009) sought to understand further the transferability of an employee’s
career experiences in relationship to job performance. As the workforce became more nomadic in
the post-industrial era, Dokko et al. concentrated their research on career experiences and job
performance as employees transferred to other organizations. Prior research focused on
investigating the relationship between job performance and career experiences of employees as
they traversed jobs within an organization; the authors noted the previous empirical studies
resulted in inconsistent conclusions.
Dokko et al. (2009) posited the prior study results varied because the researchers
examined experience as a singular concept rather than viewing past experiences as a bifurcation
of task-relevant skill and knowledge. Within this task-relevant skill and knowledge construct, the
authors studied current and past employees of a major U.S. insurance company using human
resource records. The personnel records consisted of performance evaluations and competency
assessments; the researchers recorded 771 observations. The results suggested a positive
relationship between prior work experience in a related profession and task-relevant knowledge
and skill. However, the authors did note the diminishing impact of previous experience as an
employee accumulates experience within their current organization. Furthermore, results
suggested not all experience has a positive effect on performance. For example, inflexible
behaviors or thinking carried over from a past job at another organization may negatively affect
job performance.
Conversely, despite the constraints of prior or current experiences, Spring and Illari
(2019) aimed to illustrate how general knowledge could be gained within the information
security domain. The authors argued that building general information security knowledge was
better studied from the lens of philosophical means versus finding general laws of nature or
theories, as is done in scientific fields such as mathematics, chemistry, and physics. Analyzing
three case studies, the researchers concluded cybersecurity practitioners could gain and share
general knowledge in the absence of scientific laws. Additionally, during their research, the
authors remarked on the challenge of sharing security knowledge among practitioners.
Government and private-sector constraints, for example, non-disclosure agreements,
classification of data, and proprietary information, can acerbate the challenge of sharing
knowledge and experiences across industries, sectors, agencies, and even within organizations.
Lastly, while experience was generally perceived as a positive attribute, the researcher
found sparse mention of experience matrices for cybersecurity workforce roles, let alone
empirically researched standards for experience. NIST did document some experiential learning
(i.e., hands-on/on-the-job experience) recommendations in NISTIR 8193 (2017b). Zammani et
al. (2021) also introduced experience levels as a component of their proposed holistic security
maturity model. The author-developed model for measuring ISM was evaluated and improved
through interviews and questionnaires of security practitioners and experts. The resultant model
included 4 aspects (i.e., People, Organizational Document, Process, and Technology), 14 factors,
and 5 maturity levels. Of note, experience was included in their maturity model. Specifically, the
ISM team (one of the 14 success factors), categorized under People, was measured based on the
average of the team members' years of experience in implementing security operations. Using the
proposed model, if the average was less than 1 year, the maturity level was measured at the
lowest level, Level 1. An average experience level greater than 4 years earned the highest
maturity level, Level 5.
Gaps in the Literature
The researcher discovered several research gaps relevant to this study during the review
of the literature. The first gap in the literature was a contextual observation. In the same vein as
this study, there was a noticeable lack of empirical studies conducted using U.S. federal
government organizations as the research focus. Similarly, there was an absence of studies using
U.S. cybersecurity personnel as research subjects.
Second, previous research focused on the overarching complexity of implementing
cybersecurity frameworks (e.g., NIST RMF, COBIT, HITRUST, ISO/IEC Common Criteria) to
protect information and information systems (Blum, 2020; Culot et al., 2021; Goel et al., 2020;
Hale & Gamble, 2019; Salminen, 2019). Some researchers also proposed new implementation
approaches and processes (Dombora, 2019; Diéguez et al., 2020; Goel et al., 2020; Hale &
Gamble, 2019). However, further research is needed to examine the individual phases or steps
required to implement and sustain cybersecurity frameworks successfully. This may assist in the
discernment of the challenges facing this multifaceted discipline.
Third, the review revealed little guidance associated with identifying the specific
cybersecurity work roles responsible for implementing security and privacy controls to protect
information systems. Consequently, there was a sparseness of studies focused on analyzing RMF
implementer roles. Moreover, the factors associated with developing the best team to implement
and sustain protective system safeguards are not well reported. Research is needed to recognize
and study key cybersecurity roles within each step of the RMF process.
Fourth, cybersecurity studies have been largely focused on technology. There is a lack of
research on human factors in the cybersecurity domain (Dalal et al., 2022; Dawson & Thomson,
2018; Jeong et al., 2019; Rahman et al., 2021). Further research in this area may assist in a better
understanding of the human-factor challenges embroiled within this complex field.
Finally, based on the literature review, there was a noted paucity of empirical literature
focused on the efficacy of cybersecurity certifications of persons and experience. Certifications
and experience are generally posited to be valuable attributes; however, the benefits of obtaining
cybersecurity certifications are not well studied by researchers. Likewise, the importance of
experience is often purported but is not well researched by investigators. Overall, these
observations influenced the researcher to investigate two competencies of cybersecurity
professionals (i.e., certifications and experience) and explore these human aspects in relationship
to the ability to implement security and privacy controls to protect U.S. information systems.
Chapter Summary
This chapter presented a comprehensive summary of the scholarly research applicable to
the study of cybersecurity professionals who implement NIST RMF security and privacy controls
to protect U.S. information and information systems. The chapter began with a restatement of
this study’s objective and an overview of the literature review process employed for this study.
Thereafter, the literature pertinent to this study was presented in this chapter.
Foremost, an overview of the governmental guidance relevant to the NIST RMF was
provided, with a specific emphasis placed on the U.S. governance germane to implementing
security and privacy controls. NIST published a predominance of this applicable U.S.
government cybersecurity guidance. Additionally, comprehensive public information and
academic research were limited regarding the application of the NIST RMF within the context of
the U.S. federal government domain. Notwithstanding, some general observations were inferred
from public literature.
Next, an overview of the current findings summarized from a review of the literature was
organized into three concentration areas. Findings related to the implementation of security and
privacy controls were summarized first. Overall, implementing cybersecurity frameworks to
protect information systems was noted as a complex challenge (Blum, 2020; Culot et al., 2021;
Goel et al., 2020; Hale & Gamble, 2019; Salminen, 2019). The NIST RMF is one of many
internationally recognized frameworks, as are ISO/IEC 27001, NIST CSF, COBIT, and
HITRUST frameworks. Also presented were several alternative implementations to the security
frameworks.
The second concentration area centered on the cybersecurity certification of persons. The
cybersecurity workforce is required to have the skills and knowledge to practice across a breadth
of cybersecurity areas (Marquardson & Elnoshokaty, 2020; Ramezan, 2023; Salminen, 2019;
Zammani et al., 2019). Consequently, acquiring a cybersecurity certification is asserted to signify
a level of knowledge within a requisite certification area. The number of available information
technology and cybersecurity certifications has grown significantly since introducing the first
certification in 1989. Additionally, earning a cybersecurity certification is commonly posited as a
positive attribute. Certifications are frequently cited as position requirements in job
advertisements.
The last concertation area focused on the relevancy of cybersecurity workforce
experience. The researcher did not uncover research-based studies denoting a matrix of optimal
years of experience correlated to cybersecurity areas. However, experience was also widely
perceived as a useful, personal attribute.
Finally, five observed gaps in the reviewed literature were identified in this chapter. There
was a lack of empirical studies conducted using U.S. federal government organizations as the
focus of research, and research lacked the use of U.S. cybersecurity personnel as research
subjects. Altogether, a survey of the literature also found limited studies focused on the
relationship between the cybersecurity certifications and the years of experience possessed by
cybersecurity professionals, and their perceived ability to implement cybersecurity frameworks.
The next chapter restates and further examines the proposed research questions of this
study, particularly in the context of the literature gaps noted during the literature review.
Additionally, the associations between the theoretical framework of this study, the existing
literature explored in this chapter, and the research methodology are examined. Furthermore, a
general identification of the study’s population, the proposed data collection process, the design
of the quantitative survey, the statistical tools and methods, and the ethical considerations are
framed and documented in the following chapter.
CHAPTER 3: METHODOLOGY
Introduction
This chapter presents the research design and methodology for this study. The chapter
begins with a restatement of the study’s purpose. This is followed by a review and explanation of
the research questions, including the eight hypotheses and associated variables. Purposely, the
investigative research design approach and methods are explained, along with identifying the
study’s population, the data collection procedures, the concept and design of the quantitative
survey, and the statistical tool and methods employed to analyze the collected data. Ethical
considerations are acknowledged to conclude the chapter.
Research Questions
As stated previously, the purpose of this study was to examine the relationship between
select competencies of NIST RMF practitioners and their perceived ability to implement NIST
RMF security and privacy controls for organizations managing U.S. information or information
systems. Considering the gaps in the literature noted in the previous chapter, certifications of
persons and professional experience were the two competencies selected for examination in this
study. Predictor variables, also known as independent variables, can be measured or manipulated
and influence the outcomes of the study (Park et al., 2020). Alternatively, a dependent variable
(in this study, the ability to implement RMF controls) can only be measured and measure the
outcomes in the study (Park et al., 2020).
The first categorical predictor variable, certification of persons, was defined as the unique
cybersecurity or cybersecurity-related certification (e.g., (ISC)2 CISSP, CompTIA Security+,
ISACA CISM, etc.) held by an individual. A study participant may not possess a certification,
and this is also valid data for analysis. The second predictor variable, experience, was quantified
as the years of professional (i.e., paid) experience a person has obtained in the information
technology and/or cybersecurity fields. The dependent variable was the perceived ability to
implement NIST RMF security and privacy controls.
The following eight Research Questions (RQ) and hypotheses meet the purpose of this
study, within the overall framework described in Chapter 1:
RQ1: Is there a relationship between a certification and a perceived ability to implement
NIST RMF administrative controls?
Hypothesis (Ha1) supporting RQ1: There is a positive relationship between a
certification and a perceived ability to implement NIST RMF administrative controls.
The null hypothesis (H01) for RQ1 is there is no relationship between a certification and a
perceived ability to implement NIST RMF administrative controls.
RQ2: Is there a relationship between a certification and a perceived ability to implement
NIST RMF physical controls?
Ha2: There is a positive relationship between a certification and a perceived ability to
implement NIST RMF physical controls.
H02: There is no relationship between a certification and a perceived ability to implement
NIST RMF physical controls.
RQ3: Is there a relationship between a certification and a perceived ability to implement
NIST RMF technical controls?
Ha3: There is a positive relationship between a certification and a perceived ability to
implement NIST RMF technical controls.
H03: There is no relationship between a certification and a perceived ability to implement
NIST RMF technical controls.
RQ4: Is there a relationship between a certification and a perceived ability to use system
security and privacy engineering methodologies to implement NIST RMF controls?
Ha4: There is a positive relationship between a certification and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF
controls.
H04: There is no relationship between a certification and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF
controls.
RQ5: Is there a relationship between experience and a perceived ability to implement
NIST RMF administrative controls?
Ha5: There is a positive relationship between experience and a perceived ability to
implement NIST RMF administrative controls.
H05: There is no relationship between experience and a perceived ability to implement
NIST RMF administrative controls.
RQ6: Is there a relationship between experience and a perceived ability to implement
NIST RMF physical controls?
Ha6: There is a positive relationship between experience and a perceived ability to
implement NIST RMF physical controls.
H06: There is no relationship between experience and a perceived ability to implement
NIST RMF physical controls.
RQ7: Is there a relationship between experience and a perceived ability to implement
NIST RMF technical controls?
Ha7: There is a positive relationship between experience and a perceived ability to
implement NIST RMF technical controls.
H07: There is no relationship between experience and a perceived ability to implement
NIST RMF technical controls.
RQ8: Is there a relationship between experience and a perceived ability to use system
security and privacy engineering methodologies to implement NIST RMF controls?
Ha8: There is a positive relationship between experience and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF
controls.
H08: There is no relationship between experience and a perceived ability to use system
security and privacy engineering methodologies to implement NIST RMF controls.
The eight RQs were developed based on the approach described hereafter and as
summarized in Figure 4. Figure 4
Development of RQs
Note. Developed by author.
Foremost, the first four RQs were focused on the predictor variable, certifications of
persons. Whereas the RQs five through eight were focused on the second predictor variable,
professional experience. Then, for each predictor variable (i.e., certification and experience), four
RQs were designed based on the Implement step of the 7-step NIST RMF process.
Examining this more closely, NIST (2018) prescribed two primary Implement step tasks.
The two tasks are “Control Implementation” (Task I-1) and “Update Control Implementation
Information” (Task I-2) (NIST, 2018, p. 58). Two outcomes inform both implementation tasks.
The first outcome of Task I-1 is if the security and privacy controls are implemented as specified
in the organization’s plans. The second outcome is assessed against whether the controls were
implemented using system security and privacy engineering methodologies. The second
implementation task, Task I-2, is essentially focused on documenting any changes to the planned
implementation of the security and privacy controls. Consequently, to scope this study, the
researcher focused the RQs strictly on the predominant Implement step task, i.e., Task I-1 –
Control Implementation.
With the RQs bifurcated between the predictor variables, certification of persons (i.e.,
RQs 1 – 4) and experience (i.e., RQs 5 – 8), and the center of interest on Implement step Task I1,
the two Task I-1 outcomes were separated into four areas for each predictor variable to explicate
the study results. The first three areas (i.e., RQs 1 – 3 and RQs 5 – 7) are subdivisions of the
security and privacy controls (i.e., administrative, physical, and technical controls). The fourth
area (i.e., RQ 4 and RQ 8) is a linear adaptation of the use of engineering methodologies to
implement security and privacy controls.
The 3 control implementation areas (i.e., administrative, physical, and technical) are a
categorization of the 20 NIST RMF security and privacy control families. The researcher divided
the 20 NIST RMF control families into 3 categories to reduce the survey commitment for survey
respondents. The three researcher-defined categories are comprised of the following NIST RMF
control families:
Administrative: Awareness and Training (AT); Assessment, Authorization, and
Monitoring (CA); Configuration Management (CM); Contingency Planning (CP); Incident
Response (IR); Planning (PL); Program Management (PM); Personnel Security (PS); Personally
Identifiable Information Processing and Transparency (PT); Risk Assessment (RA); System and
Services Acquisition (SA).
Physical: Maintenance (MA); Media Protection (MP); Physical and Environmental
Protection (PE); Supply Chain Risk Management (SR).
Technical: Access Control (AC); Audit and Accountability (AU); Identification and
Authentication (IA); System and Communications Protection (SC); System and Information
Integrity (SI).
The security and privacy control categorizations were based on previous versions of the
NIST RMF and the overall classifications of security controls by other researchers. In NIST SP
800-53, Revision 3, and prior editions of NIST SP 800-53, the controls were categorized into
three groups, including management, operational, and technical (NIST, 2010). Fenz et al. (2014)
classified security defense measures into three groups to include physical (e.g., guards, locks),
technical (e.g., antivirus scanners, firewalls), and organizational (e.g., policies). Additionally,
from an international perspective, the Republic of Korea guidelines categorized cybersecurity
protection measures into four similar groups, specifically, administrative (e.g., security strategies,
guidance), physical (e.g., protected areas, facilities), technical (e.g., security technologies, access
privileges), and personnel security (e.g., roles, responsibilities, training) (Park et al., 2021).
Furthermore, Prislan et al. in their study of performance measures of information security,
delineated security into three areas: physical, logical, and technical security controls (2020).
Finally, Zaini et al. (2020), based on their SLR when investigating the relationship between
organizational agility and ISM, categorized security practices into three groups: administrative,
technical, and physical/environmental. Considering these security control categorizations, the
administrative, physical, and technical areas emerged as substantiated categories for the 20 NIST
RMF security and privacy control families.
Research Design
A quantitative, non-experimental survey design was employed for this study. This study
aimed to examine two cybersecurity-focused relationships. First, the researcher examined the
relationship between certifications of persons and the ability to implement NIST RMF controls.
Second, the researcher compared the relationship between professional experience and the ability
to implement NIST RMF controls.
A quantitative research design was favored for this study considering the research
objectives. A quantitative approach enables the testing of a theory by examining relationships
between variables (Creswell & Creswell, 2023). Additionally, the competency component of the
Self-Determination Theory was examined in this study.
Creswell and Creswell (2023) also stated study variables may be measured, and
quantitative research employs deductive reasoning. Deduction allows a theory to be narrowed
into specific hypotheses that a researcher may test. Furthermore, generalizability and objectivity
were also the purposive aims of this research design. The elements of generalizability and
objectivity, coupled with numerical data, are equated to quantitative research (Goes & Simon,
2018). Finally, the goal of a postpositivist research philosophy is prediction, and the approach to
this end is primarily quantitative in nature (Burkholder et al., 2019).
Within the stream of quantitative research, a non-experimental survey method was
selected for this study. Whereas cybersecurity may be argued to be interwoven into several
academic disciplines (e.g., formal sciences, applied sciences, etc.), the two predictor variables
examined in this study (i.e., certifications of persons and experience) are aligned with the social
sciences discipline. With this focus, research designs commonly associated with social science
research, e.g., laboratory experiments and surveys, enable researchers to design their studies and
select how to measure factors of interest (Qiu et al., 2018). Qiu et al. also stated designing social
science research with theoretically pertinent variables as predictors has been an established and
accepted practice (2018).
Additionally, maintaining the quantitative research design construct, a survey is a
valuable tool to examine trends in sizeable populations of individuals (Creswell & Guetterman,
2019). Rashid et al. also acknowledged the quantitative method is an empirical approach and
depends on the collection of data from a large population (2021). This population dependency
was congruent with the design of this research considering the significant size of the U.S. federal
cybersecurity workforce. Moreover, researchers typically measure a group's opinions, attitudes,
behaviors, or characteristics using surveys (Creswell & Guetterman, 2019; Rashid et al., 2021).
Lastly, Creswell and Creswell (2023) noted quantitative research is framed with closed-ended
questions (e.g., survey designs) rather than using open-ended questions (e.g., interview questions
in qualitative research designs).
Content Analysis
This section provides the details regarding the analysis of the content collected and
analyzed in this study. This phase includes the description of the units of analysis, the selected
study population and sampling method, and the data collection procedures.
Unit of Analysis and Observation
The content analysis in this study focused on two units of analysis, namely certifications
of persons and professional experience. Analysis of these independent variables aimed at
examining the relationship between unique certifications held by cybersecurity practitioners
applying NIST RMF controls, as well as the relationship between the years of work experience
held by cybersecurity experts implementing NIST RMF controls. The unit of observation in this
study was individual people. More specifically, the observations were a collection of data about
how cybersecurity practitioners rated their perceived ability to execute NIST RMF
implementation tasks.
Population and Sampling
Population
The broader population of this study is described as the cybersecurity practitioners who
implement NIST RMF security and privacy controls, as defined in NIST SP 800-53, to protect
U.S. information and information systems. The practitioner may be a U.S. federal employee (e.g.,
military or civilian service) or a U.S. contractor implementing privacy and security controls. The
researcher did not find any authoritative data sources to quantify explicitly the U.S. federal
cybersecurity workforce. The U.S. OPM maintains FedScope, a resource for U.S. federal
workforce data (2023). FedScope data is sourced from the U.S. OPM Enterprise Human
Resources Integration-Statistical Data Mart (U.S. OPM, 2023).
Within FedScope, quarter-to-quarter and year-to-year employment data were accessible.
However, exacting cybersecurity workforce data was not available from FedScope for a few
reasons. First, several agencies are excluded from the data. For example, the data does not
include the Defense Intelligence Agency, National Security Agency, and the U.S. Postal Service.
Second, while workforce data by occupational series is available, the employment data did not
specifically identify cybersecurity and cybersecurity-related roles. For instance, employment data
categorized under the occupational series “2210-Information Technology Management” and
series “0854-Computer Engineering” were accessible. However, these occupational series did not
exclusively indicate cybersecurity-role applicability. The categorization of cybersecurity,
information technology, and cybersecurity-related roles by federal agencies, as required by the
U.S. OPM, was a noted deficiency by the U.S. GAO (2019a). Third, data regarding the
contractor workforce who manage or assist with securing U.S. information and information
systems were not captured in the FedScope data.
Despite the lack of specific workforce data, the population of cybersecurity practitioners
who implement the NIST RMF security and privacy controls was posited to be significant. For
example, using the FedScope data alone, in March 2023, there were 91,667 individuals
categorized in the Information Technology Management occupation (series 2210) and 4,859
individuals in the Computer Engineering specialty (series 0854). The researcher postulated most
of these individuals participate in security and privacy control implementation activities, along
with other federal employees assigned to other occupation series.
Sampling
Various methods were employed to recruit and collect data from a random sample of
participants from the cybersecurity practitioner population. First, NIST RMF practitioners were
contacted through relevant LinkedIn RMF groups. Second, a nationwide cybersecurity
practitioner email group was leveraged to connect with potential respondents. Both
communication channels were employed to solicit survey respondents for this study.
Additionally, the researcher refrained from directly contacting any known cybersecurity
colleagues to collect survey responses to reduce any potential survey response biases.
Furthermore, since there was no single source of referrals to the entire population
(Ponchio et al., 2021), exponential, non-discriminative snowball recruitment was encouraged to
further reach potential survey respondents. Snowball sampling (a.k.a. chain sampling) is a
method where the researcher contacts an initial group of individuals, and those people, in turn,
may make referrals to others (Rashid et al., 2021).
Sample Size
Determining sample size has been widely deliberated in academic literature (Rashid et al.,
2021). Acknowledging the unknown size of the population in this study, some general rules were
applied to calculate an appropriate sample size. Roscoe (1975) proposed sample sizes between 30
and 500 are appropriate for most research. Gay (1996) noted similar guidelines for determining
sample size. Gray proposed in a large population size (approximately n = 5,000) a sample size of
400 is adequate. As the population size increases, a smaller percentage of respondents is required
to obtain a representative sample (Goes & Simon, 2018). Consistent with these guidelines, a few
hundred responses were sought for this study.
Data Collection Procedures
To meet the research objectives of this study, data were collected from primary sources
(i.e., cybersecurity practitioners) using an online questionnaire. Online data collection methods
are cost-effective and efficient (Ponchio et al., 2021). The online survey platform the researcher
selected to employ was SurveyMonkey. The platform is a cloud-based software services
company. SurveyMonkey services include worldwide accessibility to surveys, quality survey
templates, easy-to-use surveys, and a range of data analysis reporting and exporting functionality
(Abd Halim et al., 2018).
A link to the questionnaire was disseminated through two convenience sampling
distribution channels. The first channel was LinkedIn RMF-applicable groups. For example, the
LinkedIn RMF Resource Center group contained 4,599 members, and the Federal Cybersecurity
and Risk Management Forum contained 4,120 members. The other primary channel was an
adhoc, nationwide cybersecurity practitioner email group. The researcher asked the owner and
maintainer of the email group to distribute the survey link to the group. The cybersecurity
practitioner email group contained approximately 2,200 email addresses. Individuals are
voluntary members of these groups. Also, the members in each group may not be mutually
exclusive.
Voluntary survey respondents were recruited through these distribution channels using
river sampling. River sampling involves recruiting participants by inviting them to click on a link
within virtual environments where they are likely to see the announcement, e.g., an email or
social media post (Lehdonvirta et al., 2020).
Also, Sutton and Edlund (2019) acknowledged previous research demonstrated
selfselection bias as a threat to external validity when the title of the study directly influences the
dependent variables being measured (in this study, the ability to implement RMF controls).
Knowing this, the researcher carefully considered the titling and messaging of the survey to
potential respondents. However, Sutton and Edlund’s research also supported the idea that
selfselection bias may have less of an impact on electronic studies than in-person studies (2019).
Furthermore, snowball sampling was encouraged; that is, individuals may provide the
survey link to other potential respondents. All respondents had the opportunity to contact the
researcher. However, contact was voluntary and unnecessary to complete the questionnaire
and/or to recruit other subjects. Study participants were required to read and acknowledge an
Informed Consent Statement (see Appendix B) before the participant advanced to the
questionnaire section of the online survey. Lastly, subjects were offered no incentives or
compensation to participate in the survey.
No personally identifiable information (PII), for example, name, email address, phone
number, etc., was requested from respondents. Even so, SurveyMonkey employed encryption
techniques to safeguard the confidentiality and integrity of collected data. The data collected
were initially stored on the SurveyMonkey platform. All responses were checked for
completeness. For the purpose of accepting or rejecting a completed survey, respondent
qualification was assessed using the consent form and an employment category (i.e., U.S. federal
employee or a U.S. contractor) question. Specifically, the respondent had to be involved with
implementing NIST RMF security and/or privacy controls.
Next, the researcher exported the data to the researcher’s personal HP laptop. The laptop
was password and biometrically protected; the hard drive was also encrypted. After the study’s
data and results were processed and documented in the study, the data was archived onto a
password-protected and encrypted external Western Digital drive.
Finally, as the study involved human subjects, it was compliant with the U.S. Department
of Health and Human Services' rules and policies (i.e., “Common Rule”). No vulnerable groups
or persons were knowingly or purposefully recruited as subjects for this study. The study design
was one of marginal risks to participants. The likelihood of any harm was not any greater than
typically encountered in everyday life.
Instrumentation
Within the construct of a philosophy-based study, using an online survey is an effective
method to measure responses from study participants (Rashid et al., 2021). Considering the
philosophical assumptions of this study, the study’s objectives, the deductive inquiry approach,
and the cost-effectiveness of an online method, a survey was selected as the instrument of choice
to conduct this study. As previously stated, the online questionnaire for this study was
constructed and facilitated using the SurveyMonkey platform.
Following the consent statement, the questionnaire had two main sections. Section One
was used to collect demographic information, and Section Two was used to assess the
respondents' ability to implement NIST RMF security and privacy controls. The demographics
section captured years of experience, education level, work role, certification(s) held, and
employment categorization for each respondent. Section Two comprised eight questions:
basically, a question corresponding to each RQ. Additionally, the questionnaire was designed so
the participants could complete the survey in less than 10 minutes. The survey was anticipated to
remain active for 20 to 30 days to ensure a maximum number of respondents had an opportunity
to complete the survey.
The questionnaire encapsulated a previously vetted research instrument founded on
SelfDetermination Theory (SDT). The Perceived Competence Scale (PCS) instrument blends
SDT autonomy and competence constructs into an assessment of respondents’ feelings about
their competence in a topic. Fundamentally, a respondent is asked to provide a self-evaluation of
a given topic based on a holistic assessment after considering four PCS sub-statements regarding
the single topic.
Self-Determination Theory
SDT was the theoretical foundation for this survey. SDT is a functional theory of “human
development and wellness” (Ryan & Deci, 2020, p. 1). The theory primarily focuses on
“volitional or self‐determined behaviors and the social and cultural conditions that promote it”
(Ryan, 2009, p. 1). SDT emphasizes an individual’s inherent, motivational tendencies for
growing and learning (Ryan & Deci, 2020). Ryan and Deci also stated SDT adopts the premise
that individuals are predisposed toward psychological advancement and, subsequently, a
propensity towards connectedness with others, learning, and mastery of one’s activities (2020).
SDT maintains that for healthy human development to be nurtured, individuals require
support for three basic psychological needs, notably autonomy, competence, and relatedness
(Ryan & Deci, 2020). Autonomy refers to an individual's sense of initiative and choice; it is
“supported by experiences of interest and value” (Ryan & Deci, 2020, p. 1). Competence is
described as an individual’s belief in their capabilities. Relatedness refers to an individual's
interpersonal connections and belonging (Ryan & Deci, 2020).
These three needs and processes are considered “significant” within such domains as
education, work, and health care, to name a few (Ryan & Deci, 2000, p. 1). SDT has numerous
practical implications across these domains (Ryan, 2009). The theory is posited to be an approach
to predicting performance, learning, experience, and psychological health (Deci & Ryan, 2015).
Furthermore, Martela and Riekki (2018) noted an increase in SDT research in the context of
work environments.
Specific to competence within SDT, the Center for SDT (CSDT) stated the perceptions or
sense of competence within knowledge domains or activities are postulated to be important
because competency facilitates an individual’s goal attainment and provides them with a feeling
of satisfaction for the fundamental need for competence (2022). Additionally, the CSDT cited
perceived competence has been assessed in various studies and used, along with perceived
autonomy, to predict effective performance, among other factors.
Ryan and Deci (2000) also stated an individual’s feelings of competence can elevate
internal motivation, an important stimulus for performance. Ryan and Deci discerned this
increased innate motivation will not be felt unless the feeling of competence is complemented by
the perception of autonomy (i.e., self-determined behavior) (2000). In a study of 108 university
students, Szulawski et al. (2021) further acknowledged the importance of competence on
performance. The authors concluded competence, of the three fundamental needs, had the
strongest positive impact on performance. The result was, in part, effectuated by the task's
difficulty and intrinsic motivation (Szulawski et al., 2021).
Perceived Competence Scale
Perceived competence (i.e., knowledge, skills, ability, and capacity) was the principal lens
of this study. With this chosen focus, the researcher selected the PCS as the instrument to
facilitate this study. Williams and Deci (1996) designed the PCS. The PCS is a four-item
questionnaire purported to have a strong face validity amongst the various instruments
constructed to assess SDT elements (CSDT, 2022; Williams & Deci, 1996; Williams et al., 1998).
The researcher downloaded the copyrighted PCS questionnaires from the CSDT (2022) website
after agreeing to the terms and conditions. The CSDT permitted academic use of the
PCS questionnaire.
To operationalize the PCS, relevant domain or behavior statements are generally
appended to the PCS item and four sub-statements (CSDT, 2022). The respondent reads the PCS
item and assesses their believed competency in relation to the corresponding four sub-items (i.e.,
in essence, an averaging of their responses to the four sub-items). The self-assessment is then
measured using a Likert-type scale. This study used the PCS questionnaire construct to determine
the extent participants felt about their ability to implement RMF controls
(outcome/dependent variable) in relation to the certifications held and professional experience
(predictor/independent variables).
The researcher adapted the PCS and corresponding four sub-items to fit the study's
objectives. When considering RQ1 (i.e., Is there a relationship between a certification and a
perceived ability to implement NIST RMF administrative controls?), for instance, the four PCS
sub-statements related to RQ1 were appropriately revised to measure the respondent’s perceived
ability in the RQ’s specific topic area. Further extending this example, the first PCS substatement
for RQ1 was adapted from “I feel confident in my ability to learn this material” to “I feel
confident in my ability to implement NIST RMF administrative controls.” The second PCS sub-
statement was changed from “I am capable of learning the material in this course” to “I am
capable of implementing NIST RMF administrative controls.” The third PCS sub-statement was
revised from “I am able to achieve my goals in this course” to “I am able to implement NIST
RMF administrative controls.” Lastly, the fourth PCS sub-statement was modified from “I feel
able to meet the challenge of performing well in this course” to “I feel able to meet the challenge
of implementing NIST RMF administrative controls.” Subsequently, for each RQ, the four PCS
sub-statements were individually adapted to meet the measurement objectives of the RQ. The
fully adapted questionnaire is documented within the survey instrument recorded in Appendix C.
The participants were asked to rate each PCS item on their perceived ability within the
context of the PCS item (i.e., each RQ topic). The participants used a seven-point,
agreementlevel ordinal Likert scale from 1 (not at all true) to 7 (very true) to assess their
abilities. The Likert scale was not changed to retain the originally intended PCS rating scale.
Likert scales are a common self-report measurement method and facilitate the measurement of
unobservable concepts (Jebb et al., 2021).
Pilot Test
Prior to disseminating the survey to potential respondents, a pilot study was conducted to
validate the usability and effectiveness of the questionnaire. The small-scale preliminary analysis
involved a non-probability sampling of NIST RMF practitioners, as the researcher selected ten
experienced NIST RMF subject matter experts to participate in the pilot study. The subject matter
experts were asked to assess the survey instrument in the context of the aims of the study and to
evaluate the functionality of the survey instrument.
Feedback from the RMF practitioners was requested and evaluated by the researcher. The
pilot participants provided favorable comments and no major changes were recommended by the
participants. However, based on the survey completion times recorded during the preliminary
test, the estimated completion time was reduced on the consent statement by a few minutes to
further encourage potential participants to complete the survey.
Validity
Measurement validity is not assessed by a sole statistic. There is a plethora of literature
on psychometric validation and the varying types of evidence used in research to measure
validity. Sürücü and Maslakçi (2020) acknowledged 18 different types of validity assessments.
However, the authors also noted content validity and construct validity were the two types of
validity prominent within literature.
The content validity of a measuring instrument identifies the extent each item in the
instrument serves the aim of the study (Sürücü & Maslakçi, 2020). To address content validity
for the survey used in this study, a pilot group of experts was recruited to review the
questionnaire. The experts assessed the questionnaire by reading and contemplating each
question. Then, they were asked to indicate whether each question met the intent of the objective
of the question and the overall study.
Conversely, construct validity is concerned with the comprehensiveness that the
instrument measures the behavior, concept, or idea the instrument is supposed to measure
(Sürücü & Maslakçi, 2020). Acceptable construct and face validity have been acknowledged in
the PCS literature (CSDT, 2022; Williams & Deci, 1996; Williams et al., 1998).
Reliability
Cronbach’s alpha (α) is a measure of reliability. The alpha measure is used to evaluate the
internal consistency of an item and is often used for Likert and Likert-type scales (Simon &
Goes, 2018). The alpha score ranges between 0.0 and 1.0. The internal consistency is considered
reliable if the score is .70 or higher.
Previous studies (e.g., Williams & Deci, 1996; Williams et al., 1998) used the PCS
instrument, and the Cronbach alpha (α) was greater than 0.80 (CSDT, 2022). Additionally,
Dobrydney (2020) used the PCS, and the original PCS stub-statements, in a NIST RMF related
study construct. Dobrydney evaluated the relationship between DoD information assurance (i.e.,
cybersecurity) Level III certifications and a perceived ability to implement organizational-level
risk tasks within the Prepare Step of the RMF process. The results of Dobrydney’s study (n =
220; α = 0.90) indicated the questionnaire met the requirements of generally accepted internal
consistency based on Cronbach’s alpha test result parameters.
Quantitative Data Analysis
Respondents to this study’s PCS-based questionnaire used a Likert scale to rate their
ability to implement NIST RMF security and privacy controls. Likert scale data is predominantly
considered ordinal. Even so, there is controversy about whether to treat the Likert scale as a
measurement scale (i.e., ratio or interval) or as an ordinal scale (Joshi et al., 2015; Mircioiu &
Atkinson, 2017; Norman, 2010; Pimentel, 2019; Stratton, 2018). The basis of the debate is
whether the points on a Likert, or Likert-like, scale are considered equidistant and equivalent.
The consideration and decision of what the Likert scale measurements represent impact
whether parametric or nonparametric methods are used to analyze the data. Parametric methods
(tests for means) are accepted to be more powerful than nonparametric methods (tests for
medians). However, Kvam et al. (2022) argued that even when parametric assumptions are exact,
nonparametric methods “are only slightly less powerful than the more presumptuous statistical
methods” (p. 3).
Rensis Likert, the developer of the Likert-scale concept, considered the underlying
opinions or attitudes assessed with the scale to be at the interval level, at best (1932).
Acknowledging Likert’s concept and the steadfastness of researchers to consider Likert data as
ordinal, the author examined the study’s data using nonparametric methods. Additionally, the
researcher is not assuming the collected data were normally distributed.
The demographic section of the questionnaire was analyzed using descriptive statistics.
Additionally, the ordinal data were analyzed as numerical data by applying nonparametric
statistical methods. The data were analyzed using the International Business Machines
Corporation (IBM) Statistical Package for Social Science (SPSS), Version 29. SPSS was selected
based on the interoperability between the proposed survey platform (SurveyMonkey) and SPSS.
Nonparametric Tests
Nonparametric tests may be used to examine the study's data, including calculations of
the medians, modes, frequencies, Kruskal-Wallis H-test, Mann-Whitney U-test, and Spearman’s
correlation. Medians and modes test for central tendencies, and frequencies for variability.
The nonparametric Kruskal-Wallis H-test was run as an alternative to the parametric
oneway Analysis of Variance (ANOVA) test. The H-test was used to compare each predictor
variable (i.e., certification held or years of experience) against the four outcome variables, i.e.,
ability to implement RMF administrative controls, ability to implement RMF physical controls,
ability to implement RMF technical controls, and ability to use system security and privacy
engineering methodologies to implement NIST RMF controls.
If a significant result was noted from the Kruskal-Wallis H-test, the Mann-Whitney Utest
was considered to compare and examine two selected groups further. The nonparametric Utest is
equivalent to the two-sample t-test. The U-test does not compare the groups using the medians,
except if the distribution shapes of the two groups are the same and the location parameters for
the distribution are different (Schober & Vetter, 2020).
Additionally, to examine further the two variables of interest, the variables were plotted
one against the other using a scatterplot. If the relationship between the variables was determined
to be monotonic, the Spearman rank-order correlation coefficient (rs) was calculated. Spearman’s
correlation is a nonparametric test used to measure the direction of association and strength
between two variables. In monotonic relationships, variables increase in value together, or as one
variable value increases, the value of the second variable decreases.
Hypothesis Testing
Hypothesis testing evaluates the validity of what a researcher is postulating. An
evaluation of the hypothesis is usually made in comparison to a null hypothesis (H0). This null
hypothesis testing is the cornerstone of statistical analysis (Gwise et al., 2021). One-sample Sign
Tests, Spearman’s correlation, and Kruskal-Wallis H-test were used to test the hypotheses.
Additionally, the Mann-Whitney U-test was used, as needed, to test the null hypothesis further.
Furthermore, p-value tests were calculated and analyzed as tests of the null hypotheses.
The p-value is quantified as the probability of observing data as extreme or more extreme than
those observed, assuming a null hypothesis is true (Gwise et al., 2021). A small p-value indicates
a likely improbable event, and under such a calculation, the null hypothesis may be rejected in
favor of the alternative hypothesis. Conversely, a large p-value typically indicates the collected
data more closely aligns with the null hypothesis, indicating it is the more likely result. However,
Gwise et al. (2021) cautioned, “appropriate action upon rejecting a null hypothesis depends on
the context of the finding, the study’s design, and especially, the relevance of the null hypothesis
to specific research goals” (p. 57). For this study, a p-value of 0.05 was established as the
threshold for rejecting the posited null hypotheses. P-value thresholds are used extensively in the
biomedical, physical, life, and social sciences fields. Most of these fields use a p-value threshold
of 0.05 (Ioannidis, 2019).
Ethical Considerations
Expectations in society regarding ethical conduct in research proceedings, professional
settings, personal actions, etc., have grown (Fleming & Zegwaard, 2018). This research respected
the ethical standards set by overall ethics guidelines especially as human subjects were used to
collect the data for this research. This study was expected to present minimal risk to participants.
Potential survey respondents were presented with the nature of the study and what was asked of
them. Participants were informed that their participation was voluntary.
Furthermore, anonymity, confidentiality, and privacy are important ethical elements to
consider in the development of an online survey (Nayak & Narayan, 2019). The respondents'
data remained confidential, and there was no attempt to identify participants at any time. To this
end, no PII information was collected to help protect the anonymity of respondents. Furthermore,
the collected demographic information was anticipated to be too broad to deduce the identity of a
participant. Lastly, potential respondents were informed of the identity and affiliation of the
researcher before the study using the invitation message and informed consent statement.
Chapter Summary
This chapter presented the details of the research design and methodology for this study.
The eight hypotheses from Chapter 1 were re-introduced, with descriptions and rationale for
constructing the hypotheses. Broadly, the targeted population of this study were cybersecurity
practitioners who implemented NIST RMF security and privacy controls to protect U.S.
information and information systems.
The theoretical foundation of this study was grounded in SDT, focusing on the
competency pillar of SDT. The survey questionnaire was designed around the PCS, and the
survey construct addressed the objective of this study. RMF-applicable LinkedIn groups and an
ad-hoc RMF-focused email distribution group were the primary vectors for seeding the survey
for responses. The data collected were projected to be analyzed using descriptive statistics, and
SPSS was used as the analytical tool. Considering the primary questionnaire data was captured
using a 7-point Likert scale, the ordinal data were examined using nonparametric methods. Key
nonparametric tests include the one-sample Sign Test, Kruskal-Wallis H-test, Mann-Whitney
Utest, and Spearman’s correlation test.
The next chapter turns to presenting descriptions of the data collection activities,
including the employed pilot test of the survey instrument, distribution of the questionnaire, data
collection techniques, and results of the data collected from respondents. Then, Chapter 5
provides an in-depth interpretation of the study’s data and implications.
CHAPTER 4: RESULTS
Introduction
This chapter describes the outcomes of the study’s data collection procedures and
presents the results of the data collected from the survey respondents. The chapter begins with an
overview of how the survey was developed and how a pilot study was conducted to evaluate the
survey instrument. Next, described is the process used to recruit survey participants using two
primary online distribution channels. The respondents' demographic data are also presented to
understand the sample's representativeness. Moreover, descriptive statistics are summarily
presented with an analysis of the tests to reject or fail to reject the null of the eight hypotheses
posited in this study. This study aimed to examine the relationship between certifications of
persons and professional experience held by NIST RMF practitioners and their perceived ability
to implement RMF security and privacy controls.
Survey Development
The survey for this study was developed using SurveyMonkey, Advantage Individual Plan.
The Advantage Plan enabled the researcher to use several advanced survey features, including
customized survey variables, tailored survey web addresses (i.e., Uniform Resource Locator),
and the ability to export survey data into SPSS. SPSS was the primary tool used to analyze the
survey data.
Overall, the researcher selected the Heritage standard survey theme available within
SurveyMonkey to maximize survey accessibility. The Heritage theme was Section 508 and Web
Content Accessibility Guidelines (WCAG), Version 2, compliant. Section 508 is a U.S. federal
law requiring electronic and information technology used by federal agencies to be accessible to
people with disabilities. WCAG2 is a set of accessibility guidelines, including a collection of
technical criteria to test for accessibility issues. Selecting this theme formatted the survey to meet
recognized accessibility standards, including enabling respondents to use a screen reader, if
needed, to complete the survey.
The survey comprised three main content areas. The three areas consisted of the informed
consent statement, demographics section, and the core questionnaire.
Informed Consent
The survey landing page for every potential respondent was the electronic informed consent
statement. The statement consisted of seven brief paragraphs. Key elements of the statement
included an invitation to participate in the survey, the purpose of the survey, the confidentiality of
survey records, voluntary nature of consent, lack of compensation for participating, minimal
foreseeable risks, and the principal investigator’s contact information. The complete consent
statement is presented in Appendix B.
To proceed to the next section of the survey (i.e., demographic component), the respondent
had to agree to the consent statement. If the respondent disagreed, the participant advanced to a
closure confirmation page. If the respondent agreed to the informed consent statement, the
participant was forwarded to the demographics section of the survey.
Demographics
The demographics section of the survey contained five questions. The first question was
used to capture the current employment category of the respondent. The question was also
partially used as a qualification statement, i.e., was the respondent implementing security and
privacy controls to protect U.S. federal information or information systems. The second question
asked the respondent to identify the title of their current work role. The third question queried the
respondent about their current education level. The fourth question asked the respondent to
identify the cumulative years they had worked in the information technology and/or
cybersecurity fields. Lastly, selecting from a pre-populated list of certifications, respondents were
asked to indicate the current information technology and/or cybersecurity certifications they held.
Respondents could check all applicable certifications. Forty prominent certifications related to
the area of research were listed in this section of the survey.
As noted previously, the researcher did not find an empirical ranking of cybersecurity
certifications. Considering this and the numerous workforce cybersecurity certifications
available, the researcher compiled the list of 40 certifications available for selection in the survey
based on certifications regularly contained in lists of certifications compiled by the DoD, CISA,
and the peer-reviewed literature examined during this study (Marquardson & Elnoshokaty, 2020;
Peslak & Hunsinger, 2019; Ramezan, 2023; U.S. CISA, 2021; U.S. DoD, n.d.; U.S. DoD, 2015).
Furthermore, respondents were also provided the opportunity to add any other certifications not
listed or to check “None” if the respondent did not possess a certification.
Questionnaire
The core of the survey was the last section. This section encapsulated the questions to
inform this study and, specifically, the eight proposed research questions. The first four questions
focused on gathering data to examine the relationship between the respondent’s certifications and
their perceived ability to implement NIST RMF controls and use system security and privacy
engineering methodologies. The remaining four questions focused on collecting data to examine
the relationship between the respondent’s experience and their perceived ability to implement
NIST RMF controls and use system security and privacy engineering methodologies.
If the respondent did not possess at least one information technology or cybersecurity
certification, the respondent was directly advanced to the experience-related questions (i.e., the
last four questions) in the survey. In this scenario, the participant was not provided with an option
to view or answer the four certification-related questions. The survey was bifurcated with survey
logic to minimize confusion and improve data-collection accuracy.
Lastly, based on the survey’s overall content, the SurveyMonkey algorithm estimated that
it would take a respondent approximately 6 minutes to complete the survey. The demographics
and questionnaire sections of the survey are presented in Appendix C.
Pilot Study
A pilot study was administered following Capitol Technology University Institutional
Review Board approval to conduct the study and use human subjects. The purpose of the pilot
study was to validate the questionnaire by holistically evaluating the effectiveness, usability, and
functionality of the survey before a large-scale deployment of the survey. The researcher
recruited a small group of experienced NIST RMF practitioners to participate in the pilot study.
Overall, ten cybersecurity subject matter experts were asked to participate in the pilot study. The
experts had a range of industry and U.S. federal government experience.
Recruitment emails were individually sent to potential pilot-study participants.
Personalized salutations were used; however, each participant's core content and instructions
were identical. The core content contained the email's purpose and the study's objective. The
subject matter experts were asked to, minimally, evaluate the functionality of the survey, the
clarity of the survey, and the intent of the survey instrument in meeting the objective of the
research. A link to the online survey was also included in the email.
Overall, positive feedback was received from the pilot participants, and no substantive
changes were recommended to the researcher. The researcher made one administrative change to
the survey based on the test results. The estimated time to take the survey was revised on the
survey’s landing page (i.e., consent statement) from 10 minutes to 6 minutes based on the time
calculated by the survey algorithm and the average time used by pilot survey participants to
complete the survey. Also, this revision was made to encourage survey participation further.
The survey used by the pilot-study participants was closed to restrict any further access to
the survey. The revised survey was copied, and a new, custom survey link (i.e., URL) was created
for the operational survey. These measures were taken to prevent any comingling of survey
instrument testing data and the data collected for analysis and presentation in this study.
Additionally, multiple responses were not permitted for the operational study, as the survey
instrument parameters were set so the survey could only be taken once from the same device.
Data Collection
Survey data was collected directly from primary sources (i.e., cybersecurity practitioners).
The survey was methodically time-released through the two proposed distribution channels to
maximize visibility and increase potential response rates. For example, the recruitment email was
sent to the email distribution group during the work week. Responses were collected over 6
weeks.
Email Distribution Group
The ad-hoc email distribution group was targeted for the initial recruitment of survey
participants. This decision was based on the timing of the conclusion of the pilot study and the
capability to contact potential respondents at their business email addresses during the typical
work week. The email group administrator released the recruitment email.
Social Media
The second survey distribution channel used was LinkedIn. Considering social media
mediums, a concise recruitment message was used to announce the survey to LinkedIn members.
The posts and messages were limited to three sentences. Primarily, members of the LinkedIn
RMF Resource Center and LinkedIn Federal Cybersecurity and Risk Management Forum groups
were asked to complete the survey through individual messages.
Data Management
The researcher exported the raw survey data from the survey collection platform and
imported the data into SPSS for analysis after the online survey was closed to new responses.
SPSS, Version 29, was the statistical tool used to analyze the data.
The researcher performed some preliminary data cleaning to review the raw survey data.
Data cleaning is necessary before the data are analyzed to ensure inconsistent data, such as
missing data, extreme values, etc., do not adversely impact the research findings (Verma &
Abdel-Salam, 2019). Holistically, the researcher removed blank columns automatically generated
by the survey tool, and Internet Protocol addresses were also removed to avoid any privacy
concerns. Variables were renamed for easier reference.
Then, after reviewing the initial data, 289 individuals consented to participate in the
survey. The average time spent taking the survey by all respondents was 4.5 minutes. Of the 289
consensual participants, 256 completed the demographic section of the survey (e.g., employment
category, years of experience, etc.). Of those respondents, 224 participants followed on to
complete the entire survey.
The data for these 224 respondents was examined prior to beginning any statistical
analysis. Foremost, for this data set, the employment category of the respondents was reviewed
to assess the reasonableness the respondents met the target population of this study (i.e.,
practitioners who implement NIST RMF security and privacy controls to protect U.S.
information and information systems).
The first question in the demographics section of the survey (i.e., current employment
category of the respondent) was partly used as a qualification statement and used as an
assessment of a participant’s inclusion in the study. Consequently, for the 17 participants who
marked “Other” as their employment category, the researcher scrutinized the textual responses to
this question to determine if each respondent met the intended target population. Based on
examining the “Other” employment category responses, 9 of 17 respondents were excluded from
the final number of responses used for the survey data results. For instance, survey participants
who stated their employment category as “Canadian Private Sector,” “Non-Government
Employee,” “Non US,” or “Retail industry” were excluded from the final sample of respondents.
As a result, data from 215 respondents were retained for analysis.
After the sample size was determined (n = 215), the data set was comprehensively
reviewed and further cleaned. The researcher did not detect any missing values in the data. The
predominant checkbox style of the survey limited the participants to selecting responses within
the survey's parameters. Next, the researcher focused on examining responses to the work role
and certification demographic questions.
For respondents who indicated they had a work role title not presented in the list of
options, 10 of these “Other” work titles were standardized for data format consistency to aid data
analysis. For example, for five respondents who entered their work role as “ISSM,” the
researcher revised these work titles to “Information System Security Manager.” Additionally, for
respondents who indicated they had information technology or cybersecurity certifications not
presented in the list of options, certification titles were also standardized for consistency.
Of note, two respondents’ comments regarding certifications were removed. First, in the
certification open-ended question, i.e., “Other (please specify),” the respondent did not identify
any certifications they held and stated the certifications they held previously were no longer
valid. As such, the respondent’s response to the certification question was changed to “None; I do
not have an IT or cybersecurity certification,” and the responses to the four certification
questions were also removed from the data set. The researcher retained the participant’s
experience-related responses. Second, a respondent had two qualifying certifications. However,
under the context of the certification “Other” response, the respondent also listed a work-role
qualification as a certification. The work-role qualification title was removed as it was not a
qualifying certification, as was the data value indicating an “Other” certification was held by the
individual.
Survey Results
The survey results presented are based on a sample of 215 individuals. The final sample
size fell within an acceptable range for this study (see p. 79). The researcher measured the
internal consistencies of the questionnaire items, and the results are presented in this section.
Thereafter, the demographics of the individuals comprising the sample are summarized and
described, as are the results of the tests of the eight hypotheses.
Internal Consistency Tests
Cronbach’s alpha (α) is a measure of the internal consistency of a category of items and is
often used to test Likert-type scale items (Verma & Abdel-Salam, 2019). If the α score is .70 or
higher, the internal consistency of the group of items is considered reliable (Simon & Goes,
2018; Verma & Abdel-Salam, 2019). Previous researchers used the established PCS instrument,
and the α results were greater than 0.80 (Dobrydney, 2020; Williams & Deci, 1996; Williams et
al., 1998). The researcher calculated similar α scores to these cited studies. For this study, the α
score was .938 for the four PCS questions related to certifications of persons, and the α score for
the four PCS questions relevant to experience was .919 (see Table D1). These α scores may be
interpreted as excellent (George & Mallery, 2022; Verma & Abdel-Salam, 2019), indicating
internal consistency (i.e., reliability) of the questions.
Description of Study Sample
The first demographic question was partially used as a qualification question and ascertained
the employment category of the respondent. Nearly three-quarters (73.5%, n = 158) of the survey
participants identified their employment status as a U.S. contractor, and 22.8% (n = 49)
categorized their status as a U.S. federal employee. Of the few participants who indicated their
employment status as neither a U.S. contractor nor a U.S. federal employee (3.7%, n = 8), the
respondents used such descriptors as U.S. Federally Funded Research and Development Center
employee or government employee to express their employment status.
Considering the range of practitioners who implement security and privacy controls, the
second demographic question asked survey participants to select a work title describing their
current role using work role options presented in the survey. The 13 pre-populated work roles
were based on RMF roles in NIST Special Publication 800-37 (NIST, 2018). The work roles
noted by 158 respondents (73.5%) were diversely represented across 10 NIST RMF work roles
(see Table 2). The RMF work role with the highest frequency (n = 68) was System Security
Officer. Fifty-seven individuals (26.5%) decidedly reported their role as “Other.” Of these
reported roles, remarked work roles were grouped as Information System Security Manager (n =
10), Security Control Assessor (n = 8), Program Manager (n = 8), and Advisor (n = 3), inter alia.
Table 2 Work Role of Survey
Respondents
Description Frequency Percentage
System Security Officer 68 31.6
Systems Security Engineer 31 14.4
Chief Information Security Officer 22 10.2
Security Architect 15 7.0
Information Owner or Steward 8 3.7
System Administrator 4 1.9
Enterprise Architect 3 1.4
Common Control Provider 3 1.4
Chief Information Officer 3 1.4
System Owner 1 0.5
Other 57 26.5
Total 215 100.0
The third demographic question captured the highest level of education of the survey
respondents. At least 214 participants (99.5%) completed at least a high school level of
education. Many survey respondents (87%, n = 187) earned a bachelor’s degree or higher (see
Table 3). The largest number of survey participants (n = 94) had a graduate degree.
Table 3
Level of Education of Survey Respondents
Some High School 0 0.0
High School or equivalent (e.g., GED) 2 0.9
Trade School 2 0.9
Some college but no degree 15 7.0
Associate degree 8 3.7
Bachelor degree 78 36.3
Graduate degree 94 43.7
Ph.D. or equivalent degree, or higher 15 7.0
Other 1 0.5
Total 215 100.0
Description Frequency Percentage
As one of the study’s independent variables, respondents were queried about their
experience in the cybersecurity field. Nearly half (46.9%) of the survey respondents had 20 or
more years of professional experience in cybersecurity and/or a similar domain (see Table 4).
About 5% (n = 11) of participants had less than 5 years of experience.
Table 4
Experience of Survey Respondents
Description Frequency Percentage
Less than 3 years 5 2.3
3 to less than 5 years 6 2.8
5 years to less than 9 years 33 15.4
9 years to less than 15 years 42 19.5
15 years to less than 20 years 28 13.0
Greater than 20 years 101 47.0
Note. n = 215
On the final demographic-related question, and based on the study’s second independent
variable, participants were asked about their earned certifications. Data collection was facilitated
using a list of 40 pre-populated certification options. If a certification was not on the list of
options, the respondent was given the opportunity to enter the names of any other certifications
they held. Additionally, the participants were requested only to indicate certifications they were
in good standing (i.e., had a current certification) with the certification’s certifying body.
Of the 215 survey respondents, 202 individuals earned at least one information
technology or cybersecurity certification. When only considering the presented 40 pre-populated
certification options, the most certifications held by an individual was 8 certifications; two
respondents each had 8 certifications. When also considering “Other” respondent-specified
cybersecurity certifications to the overall count, the most certifications held by any individual
was 10 distinct certifications. Thirteen individuals did not have any certifications. The 10 most
frequently earned certifications held by survey respondents are noted in Table 5.
Table 5
Top Ten Certifications Held Survey by Respondents
Description Frequency
Security+ (Sec+) 126
Certified Information Systems Security Professional (CISSP) 116
Certified Information Security Manager (CISM) 45
Certified Ethical Hacker (C|EH) 36
Information Technology Infrastructure Library (ITIL) 27
Certified in Governance, Risk and Compliance (CGRC)a 23
Certified Cloud Security Professional (CCSP) 19
Certified Information Systems Auditor (CISA) 18
Advanced Security Practitioner (CASP+) 15
Cyber Security Analyst (CySA+) 14
a. In February 2023, (ISC)2 renamed the Certified Authorization Professional (CAP) certification
to CGRC.
Descriptive Statistics
Presented in Table 6 are the descriptive statistical results for the four RQs based on
certifications, namely the perceived ability of practitioners to implement NIST RMF
administrative controls, to implement physical controls, to implement technical controls, and to
use system security and privacy engineering methodologies to implement NIST RMF controls.
Table 6
Summary of Descriptive Statistics for Dependent Variable – Certification
Statistic
Certification:
Administrative
Controls
Certification:
Physical
Controls
Certification:
Technical
Controls
Certification: Use
of Methodologies
n 202 202 202 202
Median (Mdn) 6.000 6.000 6.000 6.000
Mode 7.00 7.00 7.00 7.00
Std. Deviation 1.43528 1.52861 1.60264 1.63550
Variance 2.060 2.337 2.568 2.675
Skewness -1.308 -1.153 -0.892 -0.830
Std. Error (SE) .171 .171 .171 .171
Kurtosis 1.528 0.933 0.138 -0.065
SE .341 .341 .341 .341
Range 6.00 6.00 6.00 6.00
Minimum 1.00 1.00 1.00 1.00
Maximum 7.00 7.00 7.00 7.00
Notes. Thirteen of the survey respondents had no certifications, resulting in 202 responses to the
certification related questions. SE of skewness is calculated based on sample size (i.e., 202),
resulting in the same SE of skewness across all certification categories. Similarly, SE of kurtosis
is calculated based on sample size and SE of skewness, resulting in the same SE of kurtosis
across all certification categories.
As discussed in Chapter 3, the data were presumed not to be normally distributed.
However, tests of normality were conducted to validate this assumption. A Shapiro-Wilk’s test (p
>.05) of the data (Shapiro & Wilk, 1965), as well as a visual inspection of the data box plots and
histograms, provided evidence that the perceived ability ratings were non-normally distributed
(see Table D2 and Figures D1 through D4). This was further evidenced by the reported skewness
calculations of -1.308, -1.153, -0.892, and -0.830 (SE .171) and kurtosis outcomes of 1.528,
0.933, 0.138, and -0.065 (SE .341). The four distributions were all leftskewed with varying
degrees of kurtosis, ranging from leptokurtic (i.e., administrative controls responses) to near
mesokurtic (i.e., methodologies responses).
Respondents rated their abilities across the spectrum of available ratings. Responses were
recorded in each of the seven-point Likert scale categories, ranging from 1 (not at all true) to 7
(very true). The most frequent response for the four certification-associated questions was the
same (mode = 7, very true), and the Mdn result for each question was identical (Mdn = 6).
Furthermore, Table 7 summarizes the descriptive statistical results for the four RQs based
on experience, namely the perceived ability of practitioners to implement NIST RMF
administrative controls, to implement physical controls, to implement technical controls, and to
use system security and privacy engineering methodologies to implement NIST RMF controls.
Table 7
Summary of Descriptive Statistics for Dependent Variable – Experience
Statistic
Experience:
Administrative
Controls
Experience:
Physical
Controls
Experience:
Technical
Controls
Experience: Use
of Methodologies
n 215 215 215 215
Mdn 7.000 7.000 7.000 6.000
Mode 7.00 7.00 7.00 7.00
Std. Deviation 1.12261 1.24174 1.17854 1.28965
Variance 1.260 1.542 1.389 1.663
Skewness -1.919 -2.096 -1.623 -1.469
SE .166 .166 .166 .166
Kurtosis 4.306 5.128 3.020 2.200
SE .330 .330 .330 .330
Range 6.00 6.00 6.00 6.00
Minimum 1.00 1.00 1.00 1.00
Maximum 7.00 7.00 7.00 7.00
Notes. SE of skewness is calculated based on sample size (i.e., 215), resulting in the same SE of
skewness across all experience categories. Similarly, SE of kurtosis is calculated based on
sample size and SE of skewness, resulting in the same SE of kurtosis across all experience
categories.
Again, the data were presumed not to be normally distributed, but tests of normality were
conducted to validate this assumption. A Shapiro-Wilk’s test (p > .05) of the data (Shapiro &
Wilk, 1965), as well as a visual inspection of the data box plots and histograms, provided
evidence that the perceived ability ratings were non-normally distributed (see Table D3 and
Figures D5 through D8). This was further evidenced by the reported skewness calculations of
-1.919, -2.096, -1.623, and -1.469 (SE .166) and kurtosis outcomes of 4.306, 5.128, 3.020, and
2.200 (SE .330). The four distributions were all left-skewed and leptokurtic.
For the set of experience-related questions, respondents again rated their abilities across
the spectrum of available ratings. Responses were recorded in each of the seven-point Likert
scale categories, except for the question posed regarding perceived ability and the
implementation of NIST RMF administrative controls. For this question, no rating of 2 was
annotated by any respondent (see Table D5 and Figure D5). The most frequent response and the
Mdn results for the four experience-associated questions were identical (mode = 7, Mdn = 7),
except for the responses for the perceived ability of practitioners to use system security and
privacy engineering methodologies The Mdn was 6 for these responses.
Hypotheses Tests
Hypothesis tests were conducted after calculating and recording the descriptive statistical
results. Hypothesis testing was bifurcated into two phases. The first phase focused on testing the
four hypotheses associated with cybersecurity certifications of persons (i.e., H01 – H04). The
second phase concentrated on testing the four hypotheses related to an individual’s professional
cybersecurity experience (i.e., H05 – H08).
Additionally, as previously stated, the data collected were not normally distributed based
on Shapiro-Wilk’s tests (p > .05), as well as visual inspections of the data box plots and
histograms. Presented with non-normally distributed ordinal data, only nonparametric tests were
considered and used for hypothesis testing. Different hypothesis tests were conducted for each
group of hypotheses (i.e., certifications and experience).
Certifications of Persons (RQ1 – RQ4)
RQ1 investigated the relationship between a certification held by a cybersecurity
practitioner and their perceived ability to implement RMF administrative controls. Similarly,
RQ2 and RQ3 examined the relationship between a certification held by a cybersecurity
practitioner and their perceived ability to implement RMF physical and technical controls. Lastly,
RQ4 examined a certification held by a cybersecurity practitioner and their perceived ability to
use system security and privacy engineering methodologies to implement RMF controls.
A central tendency approach was used to evaluate the hypotheses for RQ1 through RQ4
due to the design of the hypotheses, the structure of the survey instrument, and the non-normal
distribution of the study’s data. Additionally, since each certification evolved as a constant
predictor variable of a single sample, nonparametric tests such as the Mann-Whitney U-test,
Spearman’s r, and Chi-square were eliminated as testing methods for the first four RQs.
Specifically, one-sample Sign Tests were used to test the hypotheses. A Sign Test is a
nonparametric alternative to a one-sample T-Test (Verma & Abdel-Salam, 2019). A Sign Test
compares the Mdn of a single data set to an established value. For example, for this research, the
data set used to assess H01 consisted of the practitioners who had a specific certification and their
perceived ability to implement RMF administrative controls compared to the Mdn of 3. If the
collected responses had a Mdn greater than 3, then there was evidence to support the rejection of
the null hypothesis.
The Mdn comparison value was established as three because H01 through H04
hypothesized there was no relationship between a certification and a perceived ability to
implement NIST RMF administrative, physical, or technical controls, or the to use
methodologies to implement controls. An Mdn of 4 (Somewhat true response) or greater on the
7-point agreement level Likert scale questionnaire indicated a “positive” relationship, thus
supporting the alternate hypotheses (i.e., Ha1 through Ha4).
Considering the 40 pre-populated certifications available on the survey instrument, 27
certifications were selected at least once. These 27 certifications and two noted in the “Other
(please specify)” section of the survey were analyzed using Sign Tests. The two additional
“Other” certifications analyzed were the ISACA Certified Data Privacy Solutions Engineer
(CDPSE) (n = 8) and ISACA Certified in Risk and Information Systems Control (CRISC) (n =
5). These two additional certifications were included in the hypotheses testing because of the
applicability of the certifications to this study and the frequency of certifications was greater than
four, enabling sign tests to be conducted to assess the data.
Each certification (i.e., predictor variable) was tested against the dependent variable (i.e.,
abilities in each of the four categories). Decisions regarding whether to reject the null hypothesis
were based on the one-sample Sign Test for each certification (e.g., CISSP, CISM, Sec+, etc.).
There was evidence to support a positive relationship between cybersecurity/information
technology certifications and a perceived ability to implement NIST RMF administrative,
physical, and technical controls. Furthermore, there was evidence to support a positive
relationship between cybersecurity/information technology certifications and a perceived ability
to use system security and privacy engineering methodologies to implement NIST RMF controls.
Results of the individual Sign Tests are presented in Tables D6 through D9; however,
generalized summaries of the hypothesis (H) tests are listed in Table 8.
Table 8
Summary of Hypothesis Test Results (H1 – H4) – Sign Tests
H# Variable Relationships Sig. Decided Resulta
H1 Certification Perceived ability to
implement administrative controls
See Table D6 for
individual test results
Reject the null
hypothesis
H2 Certification Perceived ability to
implement physical controls
See Table D7 for
individual test results
Reject the null
hypothesis
H3 Certification Perceived ability to
implement technical controls
See Table D8 for
individual test results
Reject the null
hypothesis
H4 Certification Perceived ability to use
methodologies to implement controls
See Table D9 for
individual test results
Reject the null
hypothesis
a. Researcher’s subjective summation of individual Sign Test results, based on Tables D6 – D9.
Professional Experience (RQ5 – RQ8)
RQ5 examined the relationship between the years of professional experience a
cybersecurity practitioner had and their perceived ability to implement RMF administrative
controls. Similarly, RQ6 and RQ7 examined the years of professional experience a cybersecurity
practitioner had and their perceived ability to implement RMF physical and technical controls.
Lastly, RQ8 examined the years of professional experience a cybersecurity practitioner had and
their perceived ability to use system security and privacy engineering methodologies to
implement RMF controls.
Spearman Rank-Order Correlation (rs) tests were used to test the hypotheses for the
remaining RQs (i.e., RQ5 through RQ8). The Spearman Correlation test is a nonparametric
method useful for measuring the monotonic association between two ordinal or nominal
variables. Also, correlation measures the direction (i.e., positive or negative) and strength of the
relationship between the variables (Verma & Abdel-Salam, 2019).
The Spearman Correlation tests were calculated using the 6 groupings of years of
experience, from less than 3 years to greater than 20 years of experience (reference Table 4), and
the respondent’s perceived abilities. Data assumptions were met for the Spearman RankOrder
Correlation test (e.g., monotonic related, ordinal, non-normally distributed). Additionally, the
researcher visually inspected scatter plots of the data.
The Spearman Correlation test results indicated there were significant and positive, but
weak, associations between years of professional experience and the perceived ability to
implement RMF administrative controls, rs = .221, n =215, p < .001, to implement RMF physical
controls, rs = .284, n =215, p < .001, to implement RMF technical controls, rs = .296, n =215, p
< .001, and to use system security and privacy engineering methodologies to implement RMF
controls, rs = .244, n =215, p < .001. Weak relationships were assessed as correlation coefficients
less than .30. Summaries of the hypothesis tests are listed in Table 9.
Table 9
Summary of Hypothesis Test Results (H5 – H8) – Spearman Rank-Order Correlation
H# Variable Relationships Correlation
Coefficienta Sig. Decision
H5 Experience Perceived ability to implement
administrative controls
.221 <.001 Reject the null
hypothesis
H6 Experience Perceived ability to implement
physical controls
.284 <.001 Reject the null
hypothesis
H7 Experience Perceived ability to implement
technical controls
.296 <.001 Reject the null
hypothesis
H8 Experience Perceived ability to use
methodologies to implement controls
.244 <.001 Reject the null
hypothesis
Note. n = 215.
a. Correlation is significant at the 0.01 level (1-tailed).
Additionally, the Mdn of the responses associated with each of the four hypotheses (i.e.,
H5, Mdn = 7; H6, Mdn = 7; H7, Mdn = 7; H8, Mdn = 6; reference Table 7) subjectively
supported a “positive” relationship between the independent and dependent variables. A response
of 7 on the experience-related questions equated to a Very true answer selection.
Chapter Summary
This chapter described how the research data were collected, the statistical methods used
to analyze the data, and the results of the hypothesis testing associated with the eight research
questions. The survey questions were structured using the PCS to examine the relationship
between certifications and professional experience held by NIST RMF practitioners and their
perceived ability to implement RMF security and privacy controls. Survey participants were
recruited through two distribution channels, namely LinkedIn and an email distribution group.
A total of 289 cybersecurity practitioners consented to participate in the online survey,
and 215 responses were retained for analysis. Nearly three-quarters of the survey respondents
identified their employment status as a U.S. contractor, and the most frequently indicated work
role was System Security Officer. Additionally, 87% of the survey respondents held a bachelor’s
degree or higher. The education level of the survey participants appeared to be consistent with
the years of experience accumulated by the respondents, as 47% of individuals had greater than
20 years of professional experience in the cybersecurity domain. Moreover, research results
revealed there was evidence to support all eight alternate hypotheses. Next, Chapter 5 describes
the study's limitations and delves into the interpretations of the survey results presented in this
chapter.
CHAPTER 5: DATA ANALYSIS
Introduction
This chapter presents a summary of the research objectives and a synopsis of the
hypotheses testing. The limitations of the study are also readdressed in the context of post-data
collection. Additionally, an analysis and interpretation of the results presented in Chapter 4 are
advanced considering the eight research questions, literature, and applications to the research
problem.
Aim of Study
The purpose of this quantitative study was to measure cybersecurity workforce
competencies in the context of implementing NIST RMF security and privacy controls.
Specifically, this research examined the relationship between cybersecurity certifications and the
years of professional experience held by U.S. cybersecurity practitioners and their perceived
ability to implement RMF security and privacy controls for organizations managing U.S.
information or information systems. The research sought to determine if the relationships were
statistically significant and if any changes within the predictor variables (i.e., specific
certifications or years of experience) had an impact on the dependent variables (i.e., perceived
abilities).
This study used an online survey to gather and record responses. The statistical analysis
results of the collected survey data supported the alternative hypothesis for the eight research
questions proposed in this study.
The first four research questions focused on cybersecurity certifications of persons. There
was evidence to support a positive relationship between cybersecurity/information technology
certifications and a perceived ability to implement RMF administrative, physical, and technical
controls. Additionally, the results of the data suggested a positive relationship between
cybersecurity/information technology certifications and a perceived ability to use system security
and privacy engineering methodologies to implement RMF controls.
The remaining four research questions centered on the years of professional experience
held by an individual. Like the certification of persons’ results, there was evidence to support a
positive relationship between experience and a perceived ability to implement RMF
administrative, physical, and technical controls. Additionally, the results supported a positive
relationship between experience and a perceived ability to use system security and privacy
engineering methodologies to implement RMF controls.
Limitations
“All research involves compromise” (Sumpter et al., 2023, p. 1). As noted in Chapter 1,
this study was constrained by limitations. Considering the non-experimental research design of
the study, data were only collected from one sample of the research population and ordinal data
were collected to inform the research questions. These limitations narrowed the data analysis and
testing methods that could be performed on the data. This shortcoming was mitigated by shaping
the research objectives. For example, the researcher sought to examine relationships in a general
direction between variables (positive or a lack of relationship) rather than seeking evidence of
causality.
Also, employing an online data collection method introduced self-selection survey bias,
i.e., potential respondents could choose to participate in the survey. Furthermore, the two survey
recruitment methods (an email distribution group and social media) likely restricted the
dissemination of the questionnaire. For these reasons, using robust inferential statistics to infer
results beyond the collected sample would be challenging. However, considering the quantity of
responses and the qualifying data collected from the respondents, the researcher posited the
sample at least moderately reflected the characteristics of the targeted population of
cybersecurity practitioners. Also, nonparametric statistical calculations were conducted to
examine substantively the survey results.
Furthermore, the survey design was predominantly closed-ended and lacked descriptive
remarked responses to inform any further analysis regarding the studied relationships. This
limitation was likewise mitigated by considering the composition of the research objectives and
hypotheses. The survey design also, consequently, made it challenging to differentiate perceived
abilities to implement RMF controls and use methodologies when a survey respondent indicated
they held more than one cybersecurity certification. These conditions were moderated by
conducting additional statistical analysis to analyze and interpret the survey results holistically.
To extend the last point, it may have been challenging for respondents to cognitively
discern and primarily attribute where their knowledge originated (e.g., experience, certifications,
education, training) when responding to questions about their perceived abilities. In these
circumstances, the researcher proposed the respondents made their best effort to attribute their
abilities to their sources of personal learning and knowledge acquisition when responding to the
questionnaire.
Analysis and Interpretations of Results
The results presented in Chapter 4 are analyzed and interpreted in this section. Introduced
first are some general observations about the study sample. Then, interpretations are focused on
each of the eight research questions. Also, while this study was conducted in the context of
protecting U.S. federal government information and information systems, the results are
proposed to be similar across non-governmental agencies as well. This is purported because, for
example, the NIST 800-series of Special Publications may be applied to both government and
non-governmental agencies (NIST, 2018), and certifications and experience in this study’s
context are agnostic of organizational setting.
Study Sample
The sample of survey respondents (n = 215) appeared to embody the characteristics of an
acceptable research sample to meet the objectives of this study. While a majority of the U.S.
federal respondents identified as U.S. federal contractors (73.5%), there was a representative
composition of respondents. The respondents were also diversely represented across 10 of the 13
work roles contained in NIST SP 800-37, as well as other relevant cybersecurity roles (e.g.,
Information System Security Managers). Additionally, respondents had a range of experience and
appeared well educated.
Minimal guidance was found in the literature regarding NIST RMF implementer roles.
However, NIST SP 800-37 (2018) identified primary and supporting workforce roles responsible
for the Implement step of the NIST RMF 7-step process. The special publication included such
roles as System Owner, Information Owner, Common Control Provider, Security Engineer,
System Security Officer, and System Administrator. Consistent with the NIST list, respondent
demographic data supported identifying these cybersecurity work roles as involved in
implementing security and privacy controls.
Surprisingly, the survey respondents had more experience and higher levels of education
than anticipated by the researcher. Sixty percent of the participants had 15 years or more years of
professional cybersecurity-related experience, whereas 5.1% had less than 5 years of experience.
An experienced workforce sample may be related to the increased levels of education. Half of the
respondents (50.7%) held a graduate-level degree or higher.
Of the 215 respondents, 202 held at least one cybersecurity or information technology
certification. The commonly held certifications by the participants were consistent with the
frequently acknowledged certifications in other studies (Marquardson & Elnoshokaty, 2020;
Peslak & Hunsinger, 2019; Ramezan, 2023). This was interesting as at least 217 potential
certifications are available to cybersecurity professionals involved in implementing security and
privacy controls (U.S. Department of Labor Employment and Training Administration, 2024).
Certifications of Persons (RQ1 – RQ4)
Research Question 1
RQ1 sought to determine if there was a relationship between a certification and a
perceived ability to implement NIST RMF administrative controls (e.g., Awareness and Training,
Planning, Program Management, and Personnel Security controls). H01 posited there was no
relationship between a certification and a perceived ability to implement RMF administrative
controls. In all but one case, this study's results indicated a positive relationship between a
certification earned and a perceived ability to implement administrative controls.
Of the certification samplings large enough to calculate a one-sample Sign Test (n = 19),
18 certifications supported a positive relationship, and significance (p-values) ranged from <.001
to .031. The exception was the test result for the GIAC Security Leadership Certification
(GSLC). The GSLC results did not support a positive relationship (p = .109, n = 6). While the
GSLC sampling minimally met the requirements for calculating a Sign Test, caution must be
applied. Another possible explanation for the finding may be the focus of the GSLC. The GSLC
is primarily concentrated on management topics and technical security controls (GIAC, 2023).
These results supported the postulation that certifications may essentially be a predictor of the
ability of cybersecurity professionals to implement RMF administrative controls.
A complete summary of the results of the 29 certifications tested are listed in Table D6.
As noted in Chapter 4, the researcher established an Mdn of 4 (Somewhat true response) or
greater on the 7-point Likert scale questionnaire as indicating a “positive” relationship. An Mdn
of 4 was the benchmark for demonstrating support for the alternate hypotheses for RQs 1 through
4.
Of note, for RQ1, 10 specific certifications did not have enough valid cases for statistical
processing. In these cases, the researcher made a general interpretation of the data. However,
caution must be applied due to the small sample sizes. This person-centered analysis was taken
instead of eliminating the results with small sample sizes to let the data unfold writ large to assist
in the interpretations of the results independent of any assumptions (Godwin et al., 2021). This
same approach was used when resulting and analyzing RQs 1 through 4.
Research Question 2
RQ2 sought to determine if there was a relationship between a certification and a
perceived ability to implement NIST RMF physical controls (e.g., Maintenance, Media
Protection, Physical and Environmental Protection, and Supply Chain Risk Management
controls). H02 posited there was no relationship between a certification and a perceived ability to
implement RMF physical controls. The results of this study indicated there was generally a
positive relationship between a certification earned and a perceived ability to implement physical
controls.
Of the certification samplings large enough to calculate a one-sample Sign Test (n = 18),
17 certifications supported a positive relationships and significance (p-values) ranged from <.001
to .031. A complete summary of the results of the 29 certifications tested are listed in Table D7.
These results supported the postulation that certifications may largely be a predictor of the ability
of cybersecurity professionals to implement RMF physical controls.
Again, the only exception to the positive findings was the result for the GSLC. The
GSLC results, while representing a small sampling, did not support a positive relationship (p
= .109, n = 6). As noted in the interpretation for RQ1, the GSLC is primarily concentrated on
management topics and technical controls (GIAC, 2023), and may have been a contributing
factor to the non-significant results.
For consideration, one other certification, the GIAC Security Expert (GSE), was
interpreted as not supporting a positive relationship between the certification and the
practitioner’s ability to implement physical controls. The GSE sample size was small (n = 1) so a
one-sample Sign Test could not be conducted on the sample. As such, the test result was not
included in the 18 Sign Tests conducted for RQ2. The GSE-related response was 3 (i.e., a rating
of less than Somewhat True).
Research Question 3
RQ3 sought to determine if there was a relationship between a certification and a
perceived ability to implement NIST RMF technical controls (e.g., Access Control, Audit and
Accountability, Identification and Authentication, and System and Communications Protection
controls). H03 posited there was no relationship between a certification and a perceived ability to
implement RMF technical controls. This study's results suggested a positive relationship between
a certification earned and a perceived ability to implement technical controls. Of the certification
samplings large enough to calculate a one-sample Sign Test (n = 19), all the certifications
supported a positive relationships and significance (p-values) ranged from <.001 to .031. A
complete summary of the results of the 29 certifications tested are listed in Table D8.
For this hypothesis, and conversely to H1 and H2, GSLC responses did support a positive
relationship (p = .016, n = 6). As stated previously, this may be due to the focus on technical
controls as part of the GSLC learning objectives.
Contrary to expectations, results appeared to support the perceived ability to implement
technical-categorized controls. Salminen (2019) noted certifications based on multiple-choice
question assessments generally measure a lower classification of learning outcomes. With a
potential for practitioners to gain less knowledge and comprehension from these types of
assessments and considering the increased number and complexity of security and privacy
controls (NIST, 2005; NIST, 2020a), it appeared plausible to anticipate a lower perceived ability
to implement technical controls based on some of the certification types. However, the results
supported the idea certifications may largely be a predictor of the ability of cybersecurity
professionals to implement RMF technical controls.
Research Question 4
RQ4 sought to determine if there was a relationship between a certification and a
perceived ability to use system security and privacy engineering methodologies to implement
NIST RMF controls. H04 posited there was no relationship between a certification and a
perceived ability to use system security and privacy engineering methodologies to implement
RMF controls. The results of this study indicated there was generally a positive relationship
between a certification earned and a perceived ability to use system security and privacy
engineering methodologies. Of the certification samplings large enough to calculate a onesample
Sign Test (n = 18), all the certifications supported a positive relationships and significance (p-
values) ranged from <.001 to .031. A complete summary of the results of the 29 certifications
tested are listed in Table D9. The results supported the idea certifications may largely be a
predictor of the ability of cybersecurity professionals to use system security and privacy
engineering methodologies to implement RMF controls.
The researcher interpreted one certification, the Cisco Certified Network Professional –
Security (CCNP – Security), as not supporting a positive relationship between the certification
and the practitioner’s ability to use methodologies. The CCNP – Security sample size was small
(n = 2), so a Sign Test could not be performed on the sample. The CCNP – Security-related
responses were 2 and 5 (i.e., one negative sign test and one positive sign test), thus failing to
meet the Mdn threshold of 4 (Somewhat True).
Supplemental Analysis (RQ1 – RQ4)
To analyze the certification-related results further, Kruskal-Wallis H-tests were performed
using the survey data. The H-test is a nonparametric equivalent of the parametric oneway
analysis of variance (ANOVA) test. Additionally, the H-test is a method for testing whether the
ranks are equal in all the groups and is used to test data with three groups or more (Verma &
Abdel-Salam, 2019).
H-tests were calculated to examine the relationship between the number of certifications
held by a cybersecurity professional, regardless of the specific certification, and their perceived
ability to implement RMF administrative, physical, and technical controls, and to use system
security and privacy engineering methodologies to implement RMF controls. All respondents to
the certification-related questions had at least 1 certification, but not more than 10 certifications.
The test results revealed no significant relationships between the number of certifications
held by a respondent and their perceived abilities: Administrative controls – H(8, n = 202) =
8.331, p = .402; Physical controls – H(8, n = 202) = 9.592, p = .295; Technical controls – H(8, n
= 202) = 8.598, p = .377; Methodologies – H(8, n = 202) = 12.502, p = .130. See Table 10 for a
summary of the test results.
Table 10
Summary of Supplemental Hypothesis Test Results (Count of Certifications)
Supporting Null Hypothesis Test Sig.a, b Decision
RQ1 The distribution of perceived ability
to implement administrative controls
is the same across the number of
certifications held
Independent-
Samples
Kruskal-Wallis
Test
.402 Retain the null
hypothesis
RQ2 The distribution of perceived ability
to implement physical controls is the
same across the number of
certifications held
Independent-
Samples
Kruskal-Wallis
Test
.295 Retain the null
hypothesis
RQ3 The distribution of perceived ability
to implement technical controls is
the same across the number of
certifications held
Independent-
Samples
Kruskal-Wallis
Test
.377 Retain the null
hypothesis
RQ4 The distribution of perceived ability
to use methodologies to implement
controls is the same across the
number of certifications held
Independent-
Samples
Kruskal-Wallis
Test
.130 Retain the null
hypothesis
Note. n = 202.
a. The significance level is .050.
b. Asymptotic significance is displayed.
The lack of evidence to support the alternate hypotheses in these H-tests may be
interpreted as additional support for the alternate hypotheses stated for RQ1 through RQ4. The
results indicated abilities were consistent regardless of the number of certifications held by a
respondent. As previously stated, the survey design limited truly compartmentalized responses to
the questionnaire when comparing specific certifications to perceived abilities.
Hernandez Merced (2023) discovered similar results when conducting a study of the
perceived extrinsic and intrinsic values of prominent cybersecurity certifications of persons. The
target population of the research was cybersecurity professionals in the Washington, District of
Columbia, area. Of the 67 survey participants, 50 individuals (74.63%) indicated increased
confidence in their capabilities because of their earned certification (Hernandez Merced, 2023).
However, interestingly, Dobrydney (2020) found no correlation between the perceived
capability to perform organizational tasks related to the NIST RMF Prepare Step and the number
of earned or unearned U.S. DoD Information Assurance Management Level III certifications,
such as CISSP, CISM, or CCISO. Dobrydney’s research concluded the presence or absence of
these certifications did not predict the ability to implement organizational-level RMF Prepare
Step tasks. This differs from the results presented in this study. However, Dobrydney focused on
the first step of the RMF Process (Prepare), limited the study to a small set of certifications, and
centered the study on U.S. DoD personnel.
Lastly, de Sá Mussa et al. (2021) believed specific attributes of certifications, or more
broadly, certain certifications, may better align to an individual’s cybersecurity role and, thus, the
organization’s objectives. Alternatively, given the diversity of certifications and RMF roles
presented in the study’s sample, it appeared the certification type was agnostic of the
respondents’ role and ability to implement security and privacy controls.
Professional Experience (RQ5 – RQ8)
Research Question 5
RQ5 aimed to examine the relationship between experience and a perceived ability to
implement NIST RMF administrative controls. H05 posited there was no relationship between
experience and a perceived ability to implement RMF administrative controls. The results of this
study indicated there was a significant and positive, but weak, relationship between the years of
professional experience a cybersecurity practitioner had acquired throughout their career and
their ability to implement administrative controls, rs = .221, n =215, p < .001.
This finding suggested that as the years of cybersecurity-related professional experience
increased, so did the perceived ability of the respondents to implement administrative security
and privacy controls. The results are similar to Jeong et al. (2019) findings. The authors, in their
SLR to research their understanding of cybersecurity human factors, noted prior cybersecurity
experience had a positive impact on overall cybersecurity awareness. Overall, the results support
the idea that experience may essentially be a predictor of the ability of cybersecurity
professionals to implement RMF administrative controls.
Research Question 6
RQ6 aimed to examine the relationship between experience and a perceived ability to
implement NIST RMF physical controls. H06 posited there was no relationship between
experience and a perceived ability to implement RMF physical controls. The results of this study
indicated there was a significant and positive, but weak, relationship between the years of
professional experience a cybersecurity practitioner had acquired throughout their career and
their ability to implement RMF physical controls, rs = .284, n =215, p < .001.
This finding also suggested that as the years of cybersecurity-related professional
experience increased, so did the perceived ability of the sample population to implement physical
security controls. Overall, the results support the idea experience may largely be a predictor of
the ability of cybersecurity professionals to implement RMF physical controls.
Research Question 7
RQ7 aimed to examine the relationship between experience and a perceived ability to
implement NIST RMF technical controls. H07 posited there was no relationship between
experience and a perceived ability to implement RMF technical controls. The results of this study
indicated there was a significant and positive, but weak, relationship between the years of
professional experience a cybersecurity practitioner had acquired throughout their career and
their ability to implement RMF technical controls, rs = .296, n =215, p < .001.
This finding was consistent with those related to administrative and physical controls.
Again, the results suggested that as the years of cybersecurity-related professional experience
increased, so did the perceived ability of the study’s respondents to implement technical controls.
These findings reflect those of Chowdhury and Gkioulos (2021). The researchers noted
implementation skills are generally what differentiates senior cybersecurity practitioners from
their less experienced counterparts and believed this skill gap may impact the use of security
controls. Overall, the results support the idea that experience may largely be a predictor of the
ability of cybersecurity professionals to implement RMF technical controls.
Research Question 8
RQ8 aimed to examine the relationship between experience and a perceived ability to use
system security and privacy engineering methodologies to implement NIST RMF controls. H08
posited there was no relationship between experience and a perceived ability to use system
security and privacy engineering methodologies to implement RMF controls.
The results of this study indicated there was a significant and positive, but weak,
relationship between the years of professional experience a cybersecurity practitioner had
acquired throughout their career and their ability to use system security and privacy engineering
methodologies to implement RMF controls, rs = .244, n =215, p < .001.
This finding also suggested that as the years of cybersecurity-related professional
experience increased, so did the perceived ability to use system security and privacy engineering
methodologies to implement RMF security and privacy controls.
Supplemental Analysis (RQ5 – RQ8)
To analyze and interpret the experience-related results further, Kruskal-Wallis H-tests
were also performed using the survey data. H-tests were calculated to examine the pairwise
relationships across all six experience categories (e.g., less than 3 years, 3 to years to less than 5
years, etc.) and the cybersecurity professional’s perceived ability to implement RMF
administrative, physical, and technical controls, and to use system security and privacy
engineering methodologies to implement RMF controls.
First, an H-test related to RQ5 supported a difference in experience across all six
experience categories, H(5, n = 215) = 14.33, p = .014, when the focus area was the perceived
ability to implement administrative RMF controls. While there were differences in experience
across the years of experience categories, a pairwise comparison of the categories did not
indicate any significant differences (p < .05) when the Bonferroni correction adjusted the
significance values. Bonferroni corrects for the multiple comparisons being calculated (Verma &
Abdel-Salam, 2019).
Second, an H-test related to RQ6 supported a difference in experience across all six
experience categories, H(5, n = 215) = 19.13, p = .002, when the focus area was the perceived
ability to implement physical RMF controls. A pairwise comparison of the categories, when the
Bonferroni correction adjusted the significance values, did reveal one significant difference (p
< .05). The difference was between 5 years to less than 9 years and greater than 20 years of
experience categories (p = .014).
The difference between these two groups was further examined and tested using a
MannWhitney U-test. The test results revealed a significant difference in the rating of abilities
between the 5 years to less than 9 years (Mdn = 6, n = 33) and the greater than 20 years (Mdn =
7, n = 101) experience groups, U = 1076.00, p = < .001. The findings appeared to indicate the
cybersecurity practitioners in the greater than 20 years group felt their experience was a more
significant influence when rating their perceived ability to implement physical security controls,
than those in the 5 years to less than 9 years group.
Third, an H-test related to RQ7 supported a difference in experience across all six
experience categories, H(5, n = 215) = 23.15, p < .001, when the focus area was the perceived
ability to implement technical RMF controls. A pairwise comparison of the categories, when the
significance values were adjusted by the Bonferroni correction, did reveal two significant
differences (p < .05). The differences were between the less than 3 years and greater than 20
years (p = .020), and 5 years to less than 9 years and greater than 20 years (p = .004) of
experience categories.
The differences between the groups were examined and tested using a Mann-Whitney
Utest. Comparing the difference between the less than 3 years (Mdn = 5, n = 5) and the greater
than 20 years (Mdn = 7, n = 101) experience groups, the U-test results revealed a significant
difference in the rating of abilities between the experience groups, U = 64.00, p = .001.
Additionally, comparing the difference between the 5 years to less than 9 years (Mdn = 6, n =
33) and the greater than 20 years (Mdn = 7, n = 101) experience groups, the U-test results
revealed a significant difference in the rating of abilities between the experience groups, U =
1031.50, p = < .001. The findings again appeared to indicate the cybersecurity practitioners in the
greater than 20 years group felt their experience was a more significant influence when rating
their perceived ability to implement technical security controls, than those in both the less than 3
years and 5 years to less than 9 years groups.
Fourth, an H-test related to RQ8 supported a difference in experience across all six
experience categories, H(5, n = 215) = 17.50, p = .004, when the focus area was the perceived
ability to use system security and privacy engineering methodologies to implement RMF
controls. Conducting another pairwise comparison of the categories, when the significance
values were adjusted by the Bonferroni correction, revealed one significant difference (p < .05).
The difference was between the less than 3 years and greater than 20 years (p = .030) of
experience categories.
The difference between these two groups was examined and tested using a MannWhitney
U-test. The U-test results revealed a significant difference in the rating of abilities between the
less than 3 years (Mdn = 5, n = 5) and the greater than 20 years (Mdn = 7, n = 101) experience
groups, U = 63.50, p = .001. The findings appeared to indicate the cybersecurity practitioners in
the greater than 20 years group felt their experience was a greater influence when assessing their
perceived ability to use system security and privacy engineering methodologies to implement
RMF controls than those in the 5 years to less than 9 years group.
Summarily, based on the results of the Kruskal-Wallis H-tests and subsequent
comparisons using U-tests, there was additional evidence to support a positive relationship
between experience and a perceived ability to implement NIST RMF administrative, physical,
and technical controls. Additionally, there was evidence to support a positive relationship
between experience and a perceived ability to use system security and privacy engineering
methodologies to implement NIST RMF controls. Thus, experience may be considered a
predictor of the ability of cybersecurity professionals to implement RMF controls. Summaries of
the supplemental hypothesis tests are listed in Table 11.
Table 11
Summary of Supplemental Hypothesis Test Results (Abilities and Experience)
Supporting Null Hypothesis Test Sig.a, b Decision
RQ5 The distribution of perceived ability
to implement administrative
controls is the same across years of
professional cybersecurity/IT
experience categories
IndependentSamples
KruskalWallis Test
.014 Reject the null
hypothesis
RQ6 The distribution of perceived ability
to implement physical controls is
the same across years of
professional cybersecurity/IT
experience categories
IndependentSamples
KruskalWallis Test
.002 Reject the null
hypothesis
RQ7 The distribution of perceived ability
to implement technical controls is
the same across years of
professional cybersecurity/IT
experience categories
IndependentSamples
KruskalWallis Test
<.001 Reject the null
hypothesis
RQ8 The distribution of perceived ability
to use methodologies to implement
controls is the same across years of
professional cybersecurity/IT
experience categories
IndependentSamples
KruskalWallis Test
.004 Reject the null
hypothesis
Note. n = 215.
a. The significance level is .050.
b. Asymptotic significance is displayed.
Lastly, based on the research results, there appeared to be support for employers’
perception that experience is a valuable attribute and a discriminating hiring variable (Peslak &
Hunsinger, 2019; Ramezan, 2023). Experienced personnel are needed across a wide breadth of
cybersecurity domains (i.e., categorically administrative, physical, and technical controls) to
successfully implement a diverse set of security and privacy controls (Diesch et al., 2020). To
this end, Stephenson (1953) postulated that behavior and inner experience are similar.
Stephenson explained what is subjective, e.g., feelings, thoughts, and sensations, and what is
observable to others, e.g., running down the road or swimming in a pool, are not distinct for
psychological purposes. The association is interesting, considering the results of this study. With
a potential positive relationship between increased experiences and heightened perceived abilities
to implement security and privacy controls, this may equate to similar positive behavior or
outcomes.
Chapter Summary
This chapter presented a summary of the research objectives, a synopsis of the hypotheses
testing, a description of the limitations of the study, and moreover, interpretations of the eight
hypothesis test results. The research objectives of this study were to examine the relationship
between certifications of persons and years of professional experience held by NIST RMF
practitioners and their perceived ability to implement RMF security and privacy controls.
There were limitations of the study. Data were collected from a single research sample
using ordinal data and self-reported responses, limiting the scope of data analysis and inference
of results. The online survey method introduced self-selection bias, making it challenging to
extend results beyond the sample. Also, the survey mainly used closed-ended questions, lacking
textual, descriptive responses for in-depth analysis. These limitations were partially addressed by
adjusting research goals. Additionally, there were challenges in distinguishing the abilities of
respondents with multiple certifications. However, this was somewhat mitigated through
additional statistical analysis to comprehensively interpret results.
The results of the hypotheses were detailed in Chapter 4 and summarily presented in this
chapter. Nonparametric statistical tests were used to test the eight hypotheses, using one-sample
Sign Tests (H01 – H04) and Spearman Rank-Order Correlation tests (H05 – H08). Subsequent
Kruskal-Wallis H-tests and Mann-Whitney U-tests were conducted on the collected data to
analyze and interpret the research results further.
Generally, the predictor variables (i.e., specific certifications or years of experience) were
associated with positive relationships in the dependent variables (i.e., perceived abilities). The
dependent variables focused on four areas, including the ability to implement RMF
administrative, physical, and technical controls and the ability to use system security and privacy
engineering methodologies to implement RMF controls. Considering the results and
interpretations of the data, certifications of persons and experience could potentially serve as
predictors for the capability of cybersecurity professionals to implement RMF controls.
Next, Chapter 6 provides a research summary, specific recommendations for future
research, and concluding remarks.
CHAPTER 6: CONCLUSIONS
Introduction
Holistically considering the study, this chapter provides insights on recommendations,
contributions to the cybersecurity discipline, and presents a concluding summary. Practical
implications and recommendations for future research are then offered based on the results and
interpretations of the eight research questions. Then, thoughts are presented on how this research
contributes to the cybersecurity field of study. Lastly, the chapter concludes with an overview of
the study’s objectives and synopses of the preceding chapters.
Recommendations
Cybersecurity management is a formidable and necessary function in technology-ladened
organizations (Sallos et al., 2019). This section presents recommendations of the research.
Suggestions are offered as possible means to improve outcomes related to implementing the
NIST RMF. Thereafter, ideas are proposed on how the findings of this study can be implemented
to further this stream of research.
Foremost, the results of this research indicated cybersecurity certifications of persons are
positively associated with the ability to administer NIST RMF security and privacy controls, as
well as the ability to use system security and privacy engineering methodologies to implement
RMF controls. Given these encouraging results, the U.S. OPM should consider mandating
certification requirements for the cybersecurity workforce across the entire U.S. government. The
U.S. OPM has not directed cybersecurity personnel across all federal agencies to obtain a role-
based certification (U.S. OPM, 2018). Equally, non-governmental organizations using the RMF
to manage security and privacy risks should also contemplate mandating certification
requirements for cybersecurity personnel.
Congruently, executives or human resource managers should consider promoting and
funding cybersecurity role-based certifications. Ryan and Deci (2000) and Szulawski et al.
(2021) posited competence can elevate internal motivation, an important stimulus for
performance. Evidence from this study supports the positive relationship between certifications
of persons and RMF abilities. This finding, by extension of competence, may elevate employees'
internal motivation and outward performance.
The study's results also provided evidence that appeared to indicate that as the years of
cybersecurity-related professional experience increased, so did the perceived ability of the
respondents to implement RMF controls. Based on the positive relationships between years of
professional experience and the ability to administer RMF controls, the data regarding the
categorical groupings of years of experience may be used to develop targeted tiers of experience
for employee-hiring actions or outlining proficiency levels. For example, NIST identified
experiential learning (i.e., hands-on/on-the-job experience) as a positive measure and capability
indicator (NISTIR 8193, 2017b). However, a majority of the 42 cybersecurity-related work roles
noted in the NISTIR did not have a targeted number of years of experience recommended across
the three work role proficiency levels (i.e., entry, intermediate, and advanced). The results of this
study could help fill in the gaps.
Similarly, Zammani et al. (2021) introduced experience levels as a component of a
proposed holistic ISM maturity model. The resultant model included 4 aspects (i.e., People,
Organizational Document, Process, and Technology), 14 factors, and 5 maturity levels.
Implementation experience was incorporated into the maturity model and was categorized under
People. Using Zammani et al.’s proposed model, if the average experience of the security team
was less than 1 year, the maturity level was measured at the lowest level, Level 1. An average
experience level greater than 4 years earned the highest maturity level, Level 5. Based on the
results of this study, authors of such security maturity models may want to reconsider the number
of years required to be scored at the higher experience maturity levels. For example, to be rated
at the highest level (i.e., Level 5), the average level could be greater than 9 years. Nine or more
years is conceivable, given the data in this study suggested there continues to be a positive
relationship with each categorical increase in experience level.
Future Research
To better understand the implications of the results of this study, future research could
focus on exploring the criticality of individual NIST RMF process steps, responsibilities of
cybersecurity work roles, development of a cybersecurity practitioner research model, RMF
studies informed by control groups, and studying other work role predictors. These ideas are
expanded upon in this section.
From an overarching NIST RMF process perspective, two areas of study are
recommended for further research. First, additional research is suggested to identify what RMF
process step provides the most significant influence in the successful implementation of the RMF
process and, ultimately, the defense of information and systems. There is limited literature
dedicated to individual RMF steps. In time and resource-constrained environments, it may be
constructive to understand the critical RMF steps to maximize security outcomes. Second, the
literature revealed a paucity of guidance associated with identifying specific cybersecurity work
roles responsible for implementing security and privacy controls to protect information systems.
This study indicated a wide array of work roles participate in implementing RMF controls.
Research is recommended to identify and study not only implementer roles, but also key
cybersecurity roles within each step of the RMF process.
Considering RMF roles, Haney and Lutters (2021) introduced the novel concept of
creating a cybersecurity advocate role, essentially an ambassador for the cybersecurity
community. Yoo et al. (2020) suggested the collective efficacy of workgroups and the sharing of
security knowledge dramatically influences the relationship between individual and group
security effectiveness. As such, an emergent cybersecurity advocate role could be helpful in
bringing together cybersecurity practitioners to share their knowledge across the 20 NIST RMF
security control families, as well as champion cybersecurity support and resources from
executive leadership. This role warrants further study.
Additionally, it may be functional to develop a standardized research model to study
cybersecurity practitioners’ confidence in work roles and responsibilities. Existing research
models for similar domain-related research, such as the Self-Efficacy in Information Security,
Computer Self-Efficacy, and Cybersecurity Self-Efficacy models, were discovered during this
research. However, the researcher posits it may be beneficial to take elements of these models,
inter alia, to develop a Cybersecurity Practitioner Self-Efficacy research model.
Recognizing the single-sample limitations of this study, research could also be
constructed to examine further the relationships between certifications, experience, and abilities
to implement RMF controls by considering control groups and longitudinal studies. For example,
data could be collected on a group of cybersecurity professionals at three or more-time intervals.
The time intervals could be before obtaining a certification, after a short duration following the
certification award, and then again at 6 months to 1 year. Additionally, within the same study, a
group of cybersecurity professionals who elected not to obtain a cybersecurity certification could
be studied at the same time intervals to strengthen the rigor of the research.
Similarly, two comparable (e.g., size, like industries) organizations could be studied,
where one organization has mandated certifications, and the other has not. This research
construct could be used to examine the efficacy of the certifications on cybersecurity outcomes
(e.g., incidents, vulnerability scores) between the two organizations.
Furthermore, this study focused on the certification of persons and experience. While
education levels were collected as part of this study, education was not examined as a predictor
variable. Future studies in this discipline could measure the predictive nature of education,
training, or microcredentials (i.e., academic non-degree credentials and certificates) on RMF
implementation outcomes.
Lastly, considering the encouraging results presented in this study and in the context of
the existential cyber threats facing the U.S., further research is recommended to evaluate
mandating certifications for cybersecurity professionals employed in every sector (i.e., private,
public, and not for profit). For example, with new U.S. Securities and Exchange Commission
rules requiring companies to provide comprehensive cybersecurity disclosures, coupled with
potential organizational and personal liabilities associated with the disclosures, researchers are
already recommending cyber strategists (i.e., a corporate board level cyber position) possess
toplevel cybersecurity credentialing (e.g., a CISSP or CISM certification) (Abraham et al., 2024).
Furthermore, researchers have posited that most people would find it reprehensible for a medical
doctor, public accountant, or lawyer not to be certified prior to practicing in their profession
(James & Callen, 2018; Zahadat, 2019). Subsequently, at some time in the future, the public may
expect all cybersecurity practitioners to be credentialed in a similar manner as other
professionals. Moreover, further considering the promising certification of persons study results,
this credentialing investigation could be extended beyond cybersecurity professionals.
Contributions to Field of Study
Collectively, technological aspects of cybersecurity are researched more than human
elements of cybersecurity (Dalal et al., 2022; Dawson & Thomson, 2018; Jeong et al., 2019;
Rahman et al., 2021). Examining cybersecurity-related human capital predictors in this study
provided insight into the positive impacts of cybersecurity certifications of persons and years of
professional experience of practitioners implementing NIST RMF controls to protect U.S.
information and information systems.
Within this human-centric research construct, this study concentrated on the
Implementation step of the NIST RMF process to scope and inform this study. The researcher
believes this is the first attempt to empirically study and measure the relationships of specific
information technology or cybersecurity certifications, as well as years of professional
experience, in the context of the NIST RMF. This is conjected due to a lack of studies discovered
during the literature review. Additionally, there was an absence of studies using U.S.
cybersecurity personnel as research subjects in this context, as well.
Prior to this study, the practical significance of cybersecurity certifications of persons and
experience appeared anecdotal and lacked evidence-based research. This research may assist in
the increased understanding and appreciation of the positive impacts of certification of persons
and experience. Moreover, this study established empirical data and results to incrementally
inform research and add to the body of knowledge within the cybersecurity discipline.
Summary of Study
Organizations managing U.S. federal government information or information systems
must implement NIST RMF security and privacy controls to safeguard the information and
systems (NIST, 2020a). However, the RMF is complex and difficult to implement (Goel et al.,
2020; Hale & Gamble, 2019), making systems and information vulnerable to exploitation due to
the inability to employ NIST RMF controls (U.S. ED, 2021; U.S. GAO, 2022; U.S. OMB, 2020;
U.S. OPM, 2021; U.S. VA, 2021). Part of the RMF challenge is the cybersecurity workforce
lacks the competencies to administer thoroughly the vast array of RMF controls to protect federal
information (U.S. GAO, 2019b).
To examine some of these workforce competencies, this research focused on measuring
two relationships within the context of the Implementation step (Step 4) of the 7-step RMF
process. First, the research aimed to compare the relationship between cybersecurity
certifications held by cybersecurity professionals and their perceived ability to implement NIST
RMF security and privacy controls, as well as their perceived ability to use system security and
privacy engineering methodologies to implement controls. Four hypotheses were posited to
frame these research objectives. The second focus of this research was to compare the
relationship between the years of professional experience acquired by cybersecurity practitioners
and again, their perceived ability to implement NIST RMF security and privacy controls and
their perceived ability to use system security and privacy engineering methodologies. Similarly,
four hypotheses were posited to bind the experience-related research goals.
A quantitative, survey-based research method was employed to conduct this study. The
targeted population of the study was cybersecurity practitioners who administered security and
privacy controls to defend U.S. federal government information or systems. Social media and an
email distribution group were leveraged to distribute the survey. A total of 289 individuals
consented to participate in the 13-item survey. After scrutinizing the collected data, 215
individual responses were retained for analysis.
The survey sample (n = 215) appeared to be a diverse group of cybersecurity
practitioners. The group was predominately comprised of a mix of U.S. contractors and U.S.
federal employees and held a wide range of relevant RMF work roles. Additionally, the sample
exhibited a high level of education and extensive experience. Out of the 215 survey respondents,
202 individuals had obtained at least one relevant certification. The three most frequently held
certifications by respondents were Security+ (n = 126), CISSP (n = 116), and CISM (n = 45).
Overall, the findings reported support the alternative hypothesis (i.e., positive
relationships) for all eight hypotheses proposed in this study. Most results were expected by the
researcher. For example, a positive relationship between certifications of persons and a perceived
ability to implement NIST RMF administrative and physical controls. The less expected result
was the positive relationship between certifications and the ability to implement NIST RMF
technical controls. The results and interpretations indicated the predictor variables (i.e.,
certifications, years of experience) were associated with positive relationships in the dependent
variables (i.e., perceived abilities).
