Week 14

DEDICATION

I dedicate this thesis to the two most important women in my life, my mother and wife. For my wife, the time she sacrificed with me to ensure I had enough to “study” and the amount she had to endure my complaining about not enough time. For my mother, forever giving of confidence, encouragement, and pride now and throughout my life (not to mention coffee at just the right time). Without their confidence, patience, and continuous and unconditional support this work would not have been possible. Therefore, this work is as much a reflection of their efforts as it is mine.

ACKNOWLEDGEMENTS

I would like to acknowledge and thank all those who have supported me throughout this long, and sometimes, arduous journey. My professors throughout this program have provided essential feedback and challenged me in ways proving beneficial in the end (though not necessarily appearing so at the time). My colleagues at work provided with me with a sounding board for ideas when needed and for those who had been through this journey themselves helped reduced the anxiety of what was to come. Finally, I would like to thank my thesis advisor, Dr. Christopher Martinez who provided necessary clarification and accommodation where I needed it most and made this last part of the journey (that often seemed it would never end) a smooth, productive, and most importantly, enjoyable experience.

ABSTRACT OF THE THESIS

AVOIDING THE HOOK: THE EFFECT OF COMPETENCIES AND END USER RESPONSES TO PHISHING

by

XXXXXXXXXXXXXX

American Public University, January 26, 2019

Charles Town, West Virginia

Dr. Christopher Martinez, Thesis Professor

The purpose of this qualitative study was to examine competencies (knowledge, skills, and attitude) of information system (IS) end-users to understand their effect on information security policy (ISP) compliant behavioral responses to phishing, to what extent end users possess these competencies, and implications to organizations. A secondary aim was to evaluate the value of an end-user information security competency model for organizations. The study was conducted due to the continuing phishing threat to organizations IS through the exploitation of end-users despite ISP and anti-phishing measures. This study used a semi-systematic literature review of 28 primary phishing studies containing competencies using actual behavioral response as the dependent variable against a comprehensive IS security competency model. Better ISP compliant outcomes of avoiding phishing generally resulted from higher levels of competency, knowledge having the most powerful effect, except for self-efficacy skills where users over-estimating their knowledge were phished more. Implications to the organization were discussed and recommendations provided. Overall, end-users with higher levels of competencies had more ISP compliant responses to phishing consistent with wider ISP compliance research. Finally, a competency model is useful to enable organizations to define, assess, and improve end-user phishing competencies.

Keywords: phishing, end-user, competency, information security, IS, ISP, information security policy, compliance

THE EFFECT OF COMPETENCIES AND END-USER RESPONSES TO PHISHING 22

Table of Contents Introduction 11 Statement of the Problem 15 Purpose Statement 15 Research Questions 16 Literature Review 16 Introduction 16 Phishing Defined 17 How Phishing works 19 Why Phishing Works – User Susceptibility 21 Phishing Mitigation Measures 25 Phishing Summary 27 Phishing and Information Security (InfoSec) 28 Organizational Factors and End User ISP Compliance 29 End-User Factors Affecting ISP Compliant Behavior 33 Summary 36 Theoretical Framework 37 Introduction 37 Key definitions 38 Defining Competency Variables 39 Developing a Holistic Competency Framework 42 Comprehensive InfoSec Competency Framework for Phishing 45 Research Methodology 47 Semi-systematic Literature Review 48 Limitations and Bias 55 Results 57 Findings of Studies in Semi-Systematic Review 57 Findings, Analysis, and Recommendations 70 Findings and Analysis 70 Implications of Findings 74 Recommendations for Organizations 81 Recommendations for Future Research 87 Conclusion 88 References 90

Table

List of Figures

Changing security behavior is challenging, and it only takes one misstep to seriously compromise a system. (Caputo, Pfeeger, Freeman, & Johnson, 2014, p. 37)

Phishing continues to be a pervasive and continuously growing cyber threat. Phishing is a specific information system (IS) security problem where the goal of IS security for organizations is the protection of the confidentiality, integrity and availability (CIA) of their information (Kruger & Kearney, 2006). Despite attempts to combat the problem, the Anti-Phishing Working Group (APWG) report for 2019 indicates 182,465 total phishing sites detected up (51,000 unique) from 138,328 in 2018 (61,000 unique). Since first appearing in 1996 in American Online (AOL) accounts, phishing “evolved from being an almost a poorly constructed instant messaging attack…that may be easily detected” (Vayanasky & Kumar, 2018, p. 15) to more sophisticated attacks like spear-phishing targeting specific individuals by circumventing system filter detection (Chaudry, Chaudry, & Rittenthouse, 2016) and leveraging psychological principles to convince recipients of their legitimacy, relevance, and importance (Musuva, Chepken, & Getao, 2019).

Though email remains the prevalent phishing attack vector at 96% (Verizon, 2018), phishing now also delivers payloads, e.g. malware and Advanced Persistent Threats (APT), via legitimate mechanisms such as fake websites (Chaudry et al., 2016; Gupta, Arachchilage, & Psannis, 2018). Further, almost 60% of attackers now use HTTPS security protocol so that to the user the domain seems safe because the lock in the browser bar indicates secure comms between the user’s browser and the visited website (APWG, 2019) and deceive end-users. Even for individuals with higher levels of awareness, knowledge, and skill it is possible to deceive them as well given credential theft and social engineering (Arief & Adzmi, 2015). For example, in the 2016 presidential campaign of Hillary Clinton, a simple phishing email was sent to the campaign chairman’s Google account, supposedly by Russian hackers, prompting him to change his login credentials (CBS News, 2016). Ironically, despite suspicious indicators, he was assured by his own IT staff that what he received was legitimate and he complied by “clicking”. Thus, as Gupta, Tewari, Jain, and Agrawal (2017) note, a myriad of techniques or methods now exists that attackers can use to deceive and achieve their aims; far more than when the basic “scam” originated over 20 years ago and far more difficult to detect.

The organizational approach to dealing with IS security problems, such as phishing, generally consist of three elements: information security policies (ISP) detailing end user organizational expectations of appropriate, or compliant, security behavior; technical measures on both server and client (user) sides; and security education, training, and awareness (SETA) programs for users. While ISP are the mainstay of any IS security program, they are not enough to encourage security compliance on their own (Siponen, 2000) without understanding how organizational programs and security culture support them. Technical measures on the server side may or may not be successful and client-side tools rely on the user to heed the warnings. Further, phishers have continuously adapted and developed ways to defeat technological mechanisms, e.g. filters, developed to defend against them (Gupta et al., 2017). Based on one report, more than 25% more phishing emails each year get past company filters reaching the end-user (KnowBe4, 2018) demonstrating purely technological solutions are “far from foolproof” (Greene, Steves, & Theofanos, 2018, p. 86) leaving the end-user as the last opportunity to deal with the phishing attempt in an ISP compliant manner to protect the information. However, the focus of phishing research literature has been on technological solutions (Das, Kim, Tingle, &Nippert-Eng, 2019; Ferreira & Vierra-Marques, 2018) when it is generally noted long-term solutions rest with educating users because attackers now target end users instead of technical attacks.

The behavioral response of end user is directly implicated in phishing, i.e. clicking, as it either protects, or threatens, the CIA of an organizations IS regardless of the phishing technique or method involved; the user must make the decision to “click”. Any phishing attack scenario ultimately targets the end user and this decision thus making users the most vulnerable and exploitable link in IS security (Chaudry et al., 2016; Gupta et al., 2017). However, without the ability to understand and avoid the threat, the user may become the threat themselves (Van Niekerk & von Solms, 2010) as their non-compliant ISP response negates any defense. This makes understanding the factors influencing users’ decisions of critical importance. In the broader IS security domain, various factors such as the user’s level of awareness and knowledge, attitude and perceptions towards risk, use of technology, and motivation have been studied to determine their effect on ISP compliant behavior. Within the phishing domain, the predominance of studies has primarily focused on what factors makes users susceptible or more vulnerable to phishing, e.g. age, demographics and how phishing works in terms of exploiting those factors.

The most common approach to end-user IS problems in the wider IS security domain centers around security education, training, and awareness (SETA) programs. These programs are designed to ensure users are aware of threats, understand security concepts and ISP with the goal of influencing and improving positive security behavior through compliance being the desired outcome. However, some authors find many users aren’t aware of the phishing threat (Ba, 2018), can’t differentiate between spam and phishing emails, have unrealistic expectations of security features (Rajivan, Moriano, Kelley, and Camp (2017), or can’t detect it even if they are aware of the threat (Iuga, Nurse, & Erola, 2016). Further, even those users aware and successfully detect a phishing email, almost 75% delete it without notifying anyone (Alohali, Clarke, Li, & Furnell, 2018) defeating follow-on technical filter adjustments. According to one report these numbers are not surprising considering almost half of users only receive a rudimentary form of training once per year (often not containing phishing details) and at least 10% never receive such training or only once when they are first employed with the organization (KnowBe4, 2018). According to Cybersecurity Insiders (2018), over half of organizations view users as the biggest threat and believe inadequate levels of end user training and expertise is the largest barrier to addressing the threat. Ironically, over 80% of those same organizations also report they have policies and provide training that is very effective at addressing the phishing threat. Clearly, if this were the case phishing would not continue to be the significant and pervasive threat it remains to be.

RQ1: How do knowledge, skills, and, attitudes (competencies) affect IS end-user ISP compliant behavioral responses to phishing?

RQ2: What extent do IS end-users possess knowledge, skills, and attitudes (competencies) necessary to respond to phishing in an ISP compliant manner?

RQ4: What value does a competency model have for assessing end-user ISP compliance and how can it assist organizations in doing so?

Phishing is a specific security problem with the IS domain concerning organizations attempting to protect their information and users whose competencies to comply with ISP may put information at risk. Accordingly, this review covers two related streams pertaining to this problem, namely phishing and ISP compliance. First, the review covers current research on the state of the phishing threat using various media, devices and techniques to exploit the end user as target (or as an entry point to the organization) emphasizing the critical role of the user response to phishing both in terms of how it works on end users and what makes phishing possible. Current anti-phishing measures are reviewed in addition to limitations and gaps in phishing research. Second, the review covers phishing in the wider IS security context, the issue of ISP compliance, and the relationship between the organization and user in terms of factors influencing ISP compliance focusing on the end user. Finally, the review covers limitations of current ISP compliance and phishing research leading to the aim for the current study.

One common theme throughout the literature on phishing is the lack of standard definition. Lastdrager (2014) identified 113 distinct definitions many of which focused on emails and did not allow for different threat actors beyond a fraudster, such as cyber-spies, nor consider the increase of threat surface in the Internet of Things (IoT) (Steves, Greene, & Theofanos, 2019). Many definitions also omit the wide variety of attack mechanisms (e.g. email, HTTP, SMS, VoIP) and media that can now enable the deception (Aleroud & Zhou, 2017) using terminology missing the nuance of the victim and their role in the offence and, perhaps more importantly, the defense (Khonji, Iraqi, & Jones, 2013). This research uses the Khonji et al. (2013) definition that phishing is “a type of computer attack that communicates socially engineered messages to humans via electronic communication channels in order to persuade them to perform certain actions for the attacker’s benefit” (p. 2092).

The most concerning form of phishing currently is spear phishing which is phishing targeted to a specific individual(s) using personal information available publicly online or disclosed by on social networking sites (SNS) to increase the likelihood of deception (Halevi, Memon, & Nov, 2015; Nuha & Molok, 2011). Spear phishing has almost twice the success rate of basic phishing (Montoya, Junger, & Hartel, 2017), can yield 10 times the benefit from a quarter of the effort or investment (Caputo et al., 2014), and is an easy entry point for other types of more severe attacks, e.g. ransomware, identity theft (Goel, Williams, & Dincelli, 2017; Heartfield, Loukas, & Gan, 2016). The rise in spear phishing is problematic given the greater difficulty in detection (Halevi et al., 2015) and the adaptation of phishers to defeat technological defenses against them (Gupta et al., 2017) leaving a reliance on user’s detection abilities (Greene, Steves, & Theofanos, 2018). A current example of phishing attack sophistication is the use of legitimate sites using previously reliable security features to fool users. According to the APWG (2019), almost 60% of phishing sites are operating from hacked websites and are using SSL up from less than 2% in 2016. This has eroded current and longstanding research on rule-based approaches noting the presence (or lack thereof) of HTTPs and lock icons are two of the most reliable cues for the indication of phishing or legitimacy (Alsharnouby, Alaca, & Chiasson, 2015; Goel et al., 2017; Jensen, Dinger, Wright, & Thatcher, 2017; Musuva et al., 2019; Vishwanath, Harrison, & Ng, 2016).

Most phishing research can be categorized into what Vishwanath (2015) calls the antecedents of susceptibility. These are the techniques and methods used by the phisher in their communications to exert persuasion and influence on the recipient to deceive them and individual factors or attributes of the recipient causing the recipient to be susceptible to the deception and victimization.

Age and gender. Current research into age and gender remains inconclusive. The findings of Sheng et al. (2010) indicating females are more susceptible to phishing remains to be confirmed and has been found by other researchers (Halevi et al., 2015; Iuga, Nurse, & Erola, 2016). Others have found susceptibility to be conditionally affected by other factors. Oliveira et al. (2017), for example, found older women to be more susceptible but only under time constraints whereas younger adult males demonstrated less cautionary behavior and all subjects were susceptible in certain domains, e.g. legal regardless of age or gender. Several other major experiments have contradicted many age and gender findings such as those in Darwish et al. (2013) finding no effect of age (Sarno, Lewis, Bohil, & Neider, 2019) or gender (Montoya et al., 2017) on vulnerability; rather they are all equally effective at detecting phishing emails. The authors in both cases suggest these results could be explained by differing levels of risk aversion and technical acumen in the populations serve to equalize differences. So, age and gender are associated to phishing susceptibility but not as the cause as suggested by some but rather as contributing or influencing factors.

Personality. Individual differences in personality may also increase susceptibility as a contributing factor though results vary. The most widely used model for assessing personality traits is the five-factor model, consisting of neuroticism, extroversion, openness, agreeableness, and conscientiousness (known as the Big 5) (Moody et al., 2017). In some cases, openness, agreeableness, and extraversion have been correlated with detection abilities depending on whether the user was well-informed or not (Vishwanath, 2015) whereas in others there has been a negative correlation where an awareness element is not considered (Halevi et al., 2015). Alohali et al. (2018) examined 538 survey responses and found conscientiousness to have an impact but the relationship was stronger only when combined with other factors such as age, gender, technical acumen, and internet usage. Others have found the relationship between personality and susceptibility but with lower impact (Moody et al., 2017).

Cognitive. The nature of email usage and cognitive processes of users also affects susceptibility. Whether the user takes a systematic or heuristic approach to handling emails makes a difference. Users who systematically and deliberately attend to technical details in the email header first before proceeding are less likely to be tricked (Vishwanath et. al, 2016; Wang, Herath, Chen, Vishwanath, & Rao, 2012) whereas those users who take their cues from the body of the email first are more vulnerable to influence and more likely misled (Harrison et al., 2016; Vishwanath et al., 2011). Similarly, email and internet usage drive the amount of cognitive processing and vulnerability. Those who deal with large volumes of emails or use the internet extensively will be at higher risk because the maintenance of productivity levels means processing everything systematically is not possible (Harrison et al. 2016; Purkait, 2012b) or simply because the probability of avoiding all suspicious emails decreases as the volume increases (Khonji et al., 2013). Moreover, dispositional factors, such as risk tolerance, suspicion, and willingness to trust have also been implicated (Wright & Marett, 2010) where suspicious and risk avoidant users are less susceptible (Darwish et al., 2013). However, Harrison et al. (2016) found the role of these factors role in user vulnerability varied and were largely dependent on user knowledge and experience.

Knowledge and experience. Of all the factors associated with phishing vulnerability, knowledge and experience are consistently demonstrated to be related. In Wright and Marett (2010), experiential factors, consisting of computer self-efficacy (CSE), computer experience, and security knowledge were shown to be strong predictors of susceptibility. Users with more of each were less vulnerable and were better able to detect deception and phishing attempts. Their findings reinforced earlier findings (Dhamija et al., 2006; Downs et al., 2007; Jakobsson, 2007) linking higher technical knowledge and experience with reduced victimization. More recent research shows lower levels of CSE and experience leads to higher phishing victimization regardless of other factors (Halevi et al., 2015; Harrison et al., 2016). Further, Wright and Marett (2010) also found experiential factors have a greater effect on susceptibility than dispositional factors. The authors suggest this is because dispositional factors are more constant over time whereas knowledge and experience continue to develop over time. Purkait (2012b) found this experiential development by observing a greater ability to detect deceptive emails in subsequent follow-on phishing tests at a far higher rate. Only one study has ever found technical proficiency or experience was not related to increased deception detection (Alsharnouby et al., 2015). However, while higher technical skill, knowledge, and experience may lead to greater abilities to detect phishing attempts other research has noted phishers are sophisticated enough to take advantage of all levels of knowledge and experience (Arief & Adzmi, 2015) and victimize even IT/IS personnel (Dhamija et al., 2006; Khonji & Iraqi, 2013).

Phishing Mitigation Measures

There are several methods used by organizations to prevent or mitigate the phishing threat. These anti-phishing measures can be categorized into three areas: technical or automatic measures not involving the user (e.g. phishing email is filtered out by system), toolbars and browser-based indicators warning users of suspicious websites or emails, and anti-phishing training aiming to increase awareness and/or decrease susceptibility by improving detection (Hong, 2012; Purkait, 2012a). Technological methods to stop emails before reaching the end user have proven less than reliable (Thomas, 2018) thereby the leaving user options.

Toolbars. Security toolbars and indicators come in two different forms: active and passive. Passive tools provide an indication somewhere on the email or website indicates an issue but does not require any engagement whereas active tools stop the process and requires user engagement, i.e. agreement, before allowing the action, e.g. click. Passive tools enable heuristic processing whereas active tools are designed to encourage systematic processing. Active tools are more successful in preventing phishing attempts because they increase the odds of the user’s decision being slowed to be less heuristic (Furnell, 2013). However, for these tools to be effective, the user must make the decision to use or heed the warnings which empirical evidence suggests they don’t (Downs et al., 2007; Sheng et al., 2010). Despite improvements in technology that should increase effectiveness, Iuga et al. (2016) used eye tracking analysis and found users only spent 6% of their time attending to such indicators. Other research shows the same lack of attention (Dhamija et al., 2006; Wright & Marett, 2010) and/or users outright ignoring warnings and clicking through them anyway (Alsharnouby et al., 2015). Further, many users misunderstand the significance of warnings or security risks the tools are preventing let alone what their actions (or inaction) may be causing (Furnell, Esmael, Yang, & Li, 2018; Iuga et al., 2016).

Anti-phishing training. Training is the most common prevention method used and, given the growing sophistication and scope of the problem, forms the most important part of any phishing defense (Chaudry et al. 2016; Wash & Cooper, 2018). Rule-based approaches have been demonstrated to be effective in some cases (Kumaraguru et al., 2010) where users receiving training to identify phishing cues, using signal detection theory (SDT) and then follow certain prescribed steps, e.g. report, delete, etc. Other approaches use a SCAM (suspicion, cognition, automaticity) model to train users to examine emails and websites using a more systematic way (Canfield, Fischhoff, & Davis, 2016) or being more “mindful” when they examine emails and websites (Jensen et al., 2017). Awareness has been addressed though security notices, web-based training, and videos (Gupta et al., 2018) and through contextual training where users are given simulated phishing email examples along with other materials (Wright & Marett, 2010). The embedded approach, which subjects users to a controlled phishing attack followed by immediate feedback, has shown a strong effect in increasing phishing resilience (Darwish et al., 2013). Likewise, video game and comic approaches teaching users about phishing, such as PhishGuru (Kumaraguru et al., 2009) have shown increased detection scores and increased retention of information to 28 days (Kumaraguru et al. 2010). However, other researchers have not been convinced of either the efficacy of one approach over another or retention lasting beyond the short-term (Burns, Johnson, & Caputo, 2019; Caputo et al., 2014). Interestingly, Abawajy (2014) evaluated the effectiveness of user preferences and performance against a variety of training delivery types, e.g. web-based, contextual, embedded, and game-based. The author found all approaches successful in improving phishing detection performance to some extent, but the relationship was strongest where the type of training matched user preference.

Overall, despite considerable phishing research focusing on how and why phishing works, the results remain inconclusive. While some factors show strong, clear relationships, such as knowledge and experience, others such as personality, age and gender have often been contradictory. This has an impact on the development of effective mitigation strategies particularly where mitigation measures seem to be “one size fits all” when it appears this is not the case (Oliveira et al., 2017). If there is agreement on one thing in the current phishing literature, it is that users are not good at detection (Alsharnouby et al., 2015) and that anti-phishing measures affect different users in different ways (Jensen et al., 2017). Therefore, many researchers have recommended the individuality of user attributes be assessed and understood so targeted measures may be developed to assist those users in dealing with the increasing sophistication of phishing (Goel et al., 2017).

Limitations/Gaps. The literature also indicates several limitations and research gaps should be considered. First, many phishing research studies have lacked basic demographic detail, used small sample sizes and/or specific populations, e.g. students, and/or their methods have not been fully detailed or described (Das et al., 2019) thus potentially affecting their generalizability and/or validity. Second, many studies have lacked theoretical grounding (Harrison et al., 2016) thereby reducing their explanatory power. Third, although user-focused phishing studies are on the rise, they are still vastly out-numbered by technological (Ferreira & Vieira-Marques, 2018) representing 13.9% of relevant studies published between 2004 and 2018 (Das et al., 2019). Fourth, despite the existence of various types of anti-phishing methods, the discrepancy between end-user abilities and increasing threat creates a constant research gap requiring continuous evaluation of attack vectors and updating of anti-phishing methods by organizations to mitigate accordingly. Lastly, although various types of methods have been used to study phishing (Alsharnouby, 2015; Musuva et al., 2019), Das et al. (2019) notes most phishing studies are self-report and survey which only provides an idea of what users think they would do rather than what they would “actually” do (Montoya et al., 2017). Therefore, some authors (e.g. Burns et al., 2019; Jensen et al. 2017; Pattinson et al., 2012) have suggested more research is needed in studying phishing from a behavioral aspect of what makes some users more successful and other factors affecting user’s actual behavior such as their attitude, motivations, and commitment to the organization.

Phishing is a specific security problem within the information system (IS) domain. The goal of IS security for organizations is the protection of the confidentiality, integrity and availability (CIA) of their information (Kruger & Kearney, 2006). Within the IS domain, information security policies (ISP) are security controls organization should detail end user organizational expectations of appropriate security behavior (ISO 27000, 2018) where expected security behavior equals security compliance (Vroom & von Solms, 2004). Each organization is different and will have different specific policies, but it is these ISP, whether formal or not, should define the specific roles and responsibilities for end users and expectations for positive or compliant behavior (Bulgurcu, Cavusoglu, & Benbasat, 2010). In the phishing context, positive or compliant type security behavior would include cautionary or pre-cautionary behavior with emails, attachments, and websites (Alohali et al., 2018; Boss, Kirsch, Angermeier, Shingler, & Boss, 2009; Ng, Kankanhalli, & Xu, 2009; Wright & Marett, 2010). According to Safa, von Solms, and Furnell (2016), opening unknown emails and attachments or engaging with unknown or suspicious websites are avoidable user errors since they involve risky behavior. Thus, for phishing, end-user compliance with ISP means avoiding risky behaviors and avoiding phishing victimization. ISP research literature pertaining to end-user compliance (Siponen, Pahnila, & Mahmood, 2007; Herath & Rao 2009a; Herath & Rao 2009b; Bulgurcu et al., 2010) indicates organizations desiring to protect their IS from threats, like phishing, need to consider more than one factor. They should consider factors affecting compliance both at the organizational and user level.

The end user is directly implicated in this problem as it is their behavioral response to phishing, i.e. clicking, representing their decision to comply or not comply with ISP (Tsohou & Holtkamp, 2018) that either protects, or threatens, the CIA of an organizations IS. The behavior of an organizations users has been empirically demonstrated to be associated with the CIA of information (Kaur & Mustafa, 2013). The goal then for organizations is to influence end user behavior to make decisions in order to better protect the CIA of information and reduce information security risks (Lebek, Uffen, Neumann, Hohler, & Breitner, 2014). According to Frauenstein and von Solms (2014), to shape end user behavior in relation to IS an organization needs to make their expectations clear through ISPs, ensure a strong and positive relationship between the organization and user, and enable end users’ abilities and decision-making skills. Further, ISO 27000 (2018) details organizations should also have in place an effective security education, training, and awareness (SETA) program to ensure users know of such ISP and possess the necessary skills to achieve compliance and implement mechanisms to motivate employees to behave compliantly. Taken together, these elements compromise the basis of IS competencies (ISO 27001, 2013) for which an organization should also establish and define regarding what is necessary for them to perform in an ISP compliant manner.

While ISP are the foundation of the traditional approach (Siponen, 2000) they cannot achieve security compliance on their own or change end-user compliant behavior without SETA and a user-centric approach (Frauenstein & von Solms, 2014). To this end, information security awareness (ISA) is the most common and important aspect of SETA programs (Metalidou et al., 2014) and are critical to influencing cautionary security behaviors (Jansen & Van Schaik, 2019). There exists a wide range of historical literature to support this claim (Anderson & Agarwal, 2010; Bulgurcu et al., 2010; Siponen, 2000). According to Bulgurcu et al. (2010), organizations should deliver two types of knowledge in SETA programs: ISP awareness, where users are oriented towards knowledge of ISP expectations and user specific roles in ISP; and general awareness where users are oriented towards knowledge of the threats and the consequences of those threats.

ISP awareness. Organizations should provide users with an understanding of what is required of them as a primary requirement (Metalidou et al., 2014) and security compliance behavior is an expected part of their work (Frauenstein & von Solms, 2014). In the contemporary work environment, use of email systems and computers is a central requirement to most jobs and therefore security compliance is an essential work requirement (Miranda, 2018). Research suggests if this expectation is not articulated to the user and/or the user views the requirement as more than their job requires, users will likely ignore their role in IS protection (Boss et al., 2009). The more robust the ISP explanation, the greater the user understanding, and participation of ISP will be leading to better choices and ultimately increases in ISP compliant behavior (da Veiga, 2016; Furnell et al., 2018). However, according to Parsons, McCormac, Butavicius, Pattinson, and Jerram (2014), beyond detailing the ISP expectations and user roles, organizations must also ensure end users understand why they are important.

General awareness. Organizations must also ensure SETA programs make the user aware of the vulnerabilities of the organization (Siponen, Pahnila, & Mahmood, 2010) to threats, such as phishing, and the consequences of what ISP non-compliance may cause to occur, e.g. data breach. Further, end users need to be informed not only of the threats, consequences, risks, and costs to the organization but also as the same applies to themselves (Bulgurcu et al., 2010). Wash and Cooper (2018) studied training and found the most common methods for achieving these aims within SETA programs is using media such as posters, flyers, and notices but may accomplish very little in the way of understanding. Further, the author found evidence of briefings and stories being more successful depending on who was giving the information and that a group environment is more effective though any type of training needs to be conducted on a regular basis and in a targeted fashion. However, one of the fundamental problems with many compliance measures, such as SETA programs, is most lack any theoretical and/or empirical basis for the changing of behavior (Siponen et al., 2007; Sommestad, Hallberg, Lundholm, & Bengtsson, 2014). Regardless, according to Furnell (2013) knowledge of the threats and what to do alone are insufficient as users need to also be informed as to how, when, and why they should do it.

Skills. In order to enable user ISP compliance, the organization must also provide the necessary tools to do so understanding knowledge is a pre-requisite for the implementation of skills to apply technological security controls. Sherif, Furnell, and Clarke (2015b) noted a critical component in ISP compliance is user’s behavior and interaction with technological security controls. Gaps in IS protection will occur if either the user is not provided the knowledge and skills necessary or they believe someone or something else, e.g. IT personnel or filter, does it for them (Frauenstein & von Solms, 2014). A basic user is not likely a computer expert and needs practical training and skills to enable better security decisions either with the assistance of computer technology (Wash & Cooper, 2018) to use technological security controls competently to strengthen compliance with ISP (Padayachee, 2012). According to Parsons et al. (2014), most phishing victims only need a low level of expertise with basic computer hygiene skills when combined with knowledge which is strongly correlated to the intention or preparedness to use technology in relation to security problems (Wang, 2013). According to the Technology acceptance model (TAM) (Davis, 1989), user intention to comply and adopt the use of technological security controls also depends on both how effective and how useful they perceive the tools to be. For example, if the skills or tools are too difficult to use, it may lead the user to feeling less competent and demotivate them (Padayachee, 2012) achieving the opposite effect. However, knowledge and skills, or the “what” and “how”, together as competencies are not complete without consideration of the environment (Ebrio, 2018) that supports security compliant attitudes, motivation, and behavior.

Security culture. Beyond providing knowledge and skills, the organization must consider the influence on end user attitudes as part of a security culture. The organization must consider user attitude and the environment because even with necessary knowledge and skills, the compliance desired of the user may not result from negative attitude and lack of commitment to the organization (Tsohou & Holtkamp, 2018) which is linked to user errors; where commitment is high, errors are much lower and vice versa (Siponen, 2000). To this end, Sherif et al. (2015a) suggested security culture is influenced by multiple variables can lead to increase in user commitment and ultimately lead to compliant IS behavior. Very simply, the process starts with executive level support and commitment for ISP leading to effecive SETA program development which then affects IS acceptance by the user leading to compliant behavior.

When guided by a top-down approach to security culture, the process is self-reinforcing whereby users are more receptive to security controls, i.e. ISP and SETA programs, which have a positive effect on compliant behavior which, in turn, increases the security culture of the organization and so on (Sherif et al., 2015b). According to Padayachee (2012), receptiveness to security controls also occurs from knowledge and skills growth further motivate and produce positive attitudes required for compliant behavior and further the security culture. Users who lack motivation or possess a poor attitude towards security and compliance with ISP form the largest obstacle to the achievement of commitment connecting the organization and end user (Metalidou et al., 2014) because they undermine the role of ISP and SETA programs which negate IS acceptance thereby precluding the security culture. Further, the main reason why the traditional, stand-alone, rule-based ISP approach doesn’t work (Siponen, 2000) and why it is important users understand the “why” in SETA programs (Parsons et al., 2014) as mentioned earlier. Neither of these elements alone will influence attitudes in the desired direction, i.e. compliance, without the other.

Theoretical models in ISP compliance. According to Lebek et al. (2014), studies of ISP compliance have used behavioral theories including the Theory of Reasoned Action/Theory of Planned Behavior (TRA/TPB), General Deterrence Theory (GDT), Protection Motivation Theory (PMT) and Technology Acceptance Model (TAM). All four of these behavioral theories, which are also used in many phishing studies where theories have been used, account for behavior by utilizing various factors and variables differently. These theories all account for the antecedents of user intention to comply with ISP (Tsohou & Holtkamp, 2018) where the basic premise is attitude drives intention and intention to perform, or comply, drives actual behavior (Sommestad et al., 2014). Studies of ISP compliance using intention to comply rather than actual compliance have been the preferred method given the practical limitations of measuring actual behavior, organizational concerns given sensitivity of information, and ethical concerns (Anderson & Agarwal, 2010; Crossler et al., 2013; Musuva et al., 2019; Vroom & von Solms, 2004). Further, Lebek et al. (2014) notes the connection between behavioral intention and actual behavior has been theoretically grounded and tested and proven empirically by at least one research study (see Anderson & Agarwal, 2010). However, the reliance on intention to comply and validity has also been noted as both a limitation and research gap.

This review has covered the phishing problem in terms of what phishing is, how it works, why it works, and current approaches to phishing from a user perspective. The research shows the phishing threat continues to grow in severity and sophistication. At the same time, users are vulnerable to phishing attacks due to various individual factors causing them to be vulnerable in different ways. Similarly, current phishing mitigation measures have demonstrated various success rates depending on various individual and contextual factors. Overall, the results gleaned from phishing specific studies appear inconclusive as to how and why phishing works. However, the research does seem to indicate some relationships are stronger than others, e.g. knowledge and skills, and both the understanding of why it works and what to do about needs to be understood from the perspective of the end user and their unique individual attributes and what will work for them. In other words, there are many factors at play and no “one size fits all” and, thus, mitigation measures to increase the user’s resilience to phishing needs to be a targeted approach.

Within the wider IS domain, several organizational factors have a bearing on ISP compliant of the end user. Specific literature details the requirement for organizations to establish ISP as a security control detailing their expectations of the end user and ISP compliant behavior as it pertains to phishing. In addition, according to international guidelines, organizations should outline the competencies necessary for ISP compliant behavior and subsequently provide the necessary SETA programs to enable and shape the behavior through the provision of the knowledge and skills to do so and further build a security culture to encourage a positive attitude and motivate the end user. These, however, are guidelines and therefore only recommendations. Likewise, it is these same variables influencing end user compliant behavior along with several other individual factors affecting their compliance with ISP. These ISP compliance factors have been studied and, like phishing specific research noted above, the results have been inconclusive failing to find a singular most important factor or theoretical explanation for user abilities to act compliantly with ISP. In both phishing and ISP compliance research, each variable and theory has explained only part of the behavior leading most authors to conclude it is likely many variables and theories likely influence phishing susceptibility and ISP compliance.

Literature in both the ISP compliant behavior and the specific phishing domain share several common research gaps. First, is the tendency to focus on singular factors, influencing factors, or theories to the exclusion of others that may underpin or better explain end user ISP compliant behavior or phishing resiliency from a holistic perspective to comprehensively consider the competencies of end-users since they most directly influence and drive end-user responses to specific IS problems and safeguards (such as with phishing). Second, is the predominance in both streams of research using self-report and survey to measure behavioral intentions rather than actual behavior when there may exist a difference between what is said and what is done. This research attempts to address these gaps by comprehensively examining the relationship of end-user competencies in relation to ISP compliant behavior specific to phishing using actual behavior.

Information security policies (ISP) are the primary mechanism for organizations in the protection of their information systems (IS) from threats like phishing making ISP compliant behavior of the end-user critically important. Previous ISP compliance and phishing research focuses primarily on theories and methods using behavioral intention to comply with ISP as the dependent variable rather than actual behavior. Further, research in ISP compliance and phishing have focused on singular variables or theories. However, IS security (InfoSec) literature indicates end-user behavior is influenced by multiple internal, e.g. personality, and external, e.g. organization, factors. This research attempts to address these gaps by using a comprehensive model to study the effect of end-user ISP competency variables, defined in wider ISP compliance research, on the actual behavioral response of end-user to phishing. Further, this research uses a comprehensive competency framework allowing for other factors influencing, or being influenced by, end-user competencies and their behavioral response to phishing in an ISP compliant manner.

Competence in InfoSec context. What competence is and how it applies in various contexts has been defined and described differently throughout the literature. Several researchers note the importance of competencies in ISP compliant behavior to protect the confidentiality, integrity, and availability (CIA) of information but are not specific as to what they are (Chang, Chen, & Chen, 2011; Siponen, 2000). In the context of ISP compliant behavior, Tsohou and Holtkamp (2018) examined multiple definitions across various domains and scholarly references to conclude competence in the InfoSec context is the integration of knowledge, skill, and attitudinal characteristics directly influencing IS end-user behavior in deciding to comply (or not) with ISP. For this research, IS competencies are defined as the knowledge, skills, and attitudinal characteristics possessed by the basic end-user (e.g. employee) affecting their behavioral response to phishing in an ISP compliant manner.

ISP compliant behavior. Phishing is a specific IS security threat to the CIA of organizations IS. As noted earlier, ISP are security controls detailing organizational expectations of the user in meeting specific roles and responsibilities for positive, or compliant, IS behavior (Bulgurcu et al., 2010). Thus, for phishing, end-user ISP compliance means following ISP by applying cautionary behaviors, avoiding phishing victimization and/or following ISP and related procedures to report phishing attempts (attempted or actual). This ISP compliant behavior represents the dependent variable in this research.

This research seeks to examine the relationship between end-user competencies (knowledge, attitude, and skills) and their behavioral responses to phishing. Therefore, this research uses a competency model widely used in human resource and educational domains which would establish and define the desired knowledge, skills, and attitudes necessary for desired ISP compliant phishing performance (Markus, Thomas, & Allpress, 2005). The iceberg model in Figure 2 depicts the relationship between the knowledge, skills, and attitude components of competency (the independent variables and area of focus for this study in bold) “visible” above or just below the blue “water” line where knowledge and skills are the “what” and “how” and attitude is the driver behind them which are further influenced by deeper non-visible, internal factors such as personality and motives (Ebrio, 2018) where personality is stable over time like attitude whereas motivation is more dynamic (Siponen, 2000). According to Teodorescu (2006), the value of this model lies in the ability to define performance results of a user that are measurable and can be linked to programs, e.g. training. Therefore, phishing competencies can be defined, and behavior assessed to provide training or resources necessary to address deficiencies in competencies. Given the central role of email and IS as critical business functions, all employees must execute in most organizations and the role of the organization in defining expectations of competencies and training programs to achieve same (Miranda, 2018), this is considered an appropriate model. This model now requires definition of InfoSec competencies in relation to phishing (i.e. the independent variables) and ISP compliant behavior.

Knowledge. Knowledge is the foundation for traditional InfoSec competence and affects all other aspects of competence flowing from ISA and SETA programs. According to Tsohou and Holtkamp (2018), in the context of phishing, users must possess: knowledge of the cost of the phishing threat; knowledge of the organization’s ISP and procedures regarding phishing; and knowledge of security controls or measures they are to take in dealing with phishing attempts. This knowledge makes the user ready to practice ISP compliant behavior, is developed over time, and provides the user with the awareness to perceive the threat correctly and tools necessary to deal with a phishing threat in the desired way. Moreover, different levels of knowledge and skill are required depending on the InfoSec safeguard or security control. For example, there is a difference between what is required by an IT security specialist in dealing with a network intrusion threat and a basic user dealing with a phishing attempt. ISP compliance research confirms increased knowledge and awareness of IS threats and measures to deal with those threats are positively associated with ISP compliant behavior (Bulgurcu et al., 2010; Siponen, 2000; Siponen et al., 2010). Therefore, in relation to phishing, it is hypothesized that:

Skills. Self-efficacy is an essential element to competence and defined as user’s ability to assess their own ability to perform InfoSec behaviors based on their skill or experience. The connection between self-efficacy and ISP compliant behavior has been demonstrated to be positively related in wider ISP compliance research (Anderson & Agarwal, 2010; Bulgurcu et al., 2010; Ng. et al., 2009). While some authors have referred to users’ perceptions as part of the attitude component of competency (Bulgurcu et al., 2010; Lin & Kunnathur, 2013), Tsohou & Holtkamp (2018) detailed perceptions are also part of the skills component in competencies. Like self-efficacy, where users assess themselves about applying their own knowledge of security controls, users must assess their own perceptions of several variables. These include their perceptions of: the phishing security threat likelihood and vulnerability; the effectiveness of security controls; and perceptions of the phishing consequences; In ISP compliance research, higher levels of these skills have been positively associated with ISP compliant behavior (Boss et a., 2009; Herath & Rao, 2009a; Herath & Rao, 2009b; Lin & Kunnathur, 2013; Tsohou & Holtkamp, 2018). Therefore, it is hypothesized that:

H2 : Higher levels of end-user skill will be associated with an ISP compliant behavioral response to phishing.

Attitude. Though the ability to apply ISP knowledge and skills to achieve the desired end state of applying security controls are necessary to demonstrate competency they are not enough. According to Padayachee (2012), proper motivation and a positive attitude are also necessary competencies because they are the key influencers in user’s self-efficacy and confidence in using security controls and recognizing their value in the protection of the organizations IS. Bulgurcu et al. (2010) defined attitude toward ISP compliance as how positively the user views compliance with ISP. User attitude influences all other dimensions of end-user competence such as how they perceive risks, how they value security controls, and how their actions will help or not. When users think their actions are beneficial, they are more likely to do so and this, in turn, further motivates ISP compliant behaviors. Tsohou and Holtkamp (2018) argue a positive attitude towards ISP (i.e. that they are useful, beneficial, important, and or necessary) will lead to the expected level of behavior in complying with ISP. The link between positive user attitude towards ISP and their abilities to assess IS threats and cope with them has been empirically demonstrated in several studies showing increased intention towards ISP compliant behavior (Anderson & Agarwal, 2010; Herath & Rao, 2009a; Ng et al., 2009). This implies a user with a positive attitude towards ISP is more likely to also possess higher levels of knowledge and skill. Thus, regarding phishing, it is hypothesized that:

H3: The higher the positivity level of end-user attitude towards ISP will be associated with ISP compliant behavioral responses to phishing.

End-user holistic frameworks. With the key variables of InfoSec end-user competencies defined for phishing, a comprehensive model is developed to account for the interaction of these competencies (the independent variables) and actual ISP compliant behavior (the dependent variable) in the larger organizational context. Several frameworks including competency were considered for this research such as the human, organizational, technological (HOT) framework to combat phishing (Frauenstein & von Solms, 2009), an end-user characteristic framework for social engineering attacks (Albladi & Weir, 2018), and a theoretical computer-mediated communication (CMC) competence model (Spitzberg, 2006). These models are all user-focused, social engineering or phishing related, and contain the competence components of knowledge, skills, and attitude. However, the models either take a technological perspective where application to IS users for problems such as phishing is limited (Parsons et al., 2014) or their use in cyber-security domain may not be applicable (Halevi et al., 2015). Further, the models all evaluate competencies primarily from a vulnerability or susceptibility perspective rather than examining how those competencies result in InfoSec or ISP compliant behavioral responses of being resilient to being phished and, therefore, were not considered adequate for this research.

End-user InfoSec behavioral response models. The literature concerning InfoSec and ISP compliance indicates an increasing tendency towards the use of behavioral models. Kruger and Kearney (2006) developed the Knowledge, Attitude, Behavior (KAB) model to assess and measure the level of information security awareness (ISA) in an international gold mining organization. According to the model, KAB refers to what the user knows (K), what the user thinks (A), and what the user does (B) where it is posited growth in ISP knowledge causes users to have a more positive attitude towards security controls (i.e. ISP and procedures) which then translates into more ISP compliant behavior. Though lacking any theoretical foundation, the model demonstrated organizations need to continually measure and assess ISP, ISA, and user behavior relative to IS threats and vulnerabilities so appropriate changes to programs can be made. Ng et al. (2009) developed a similar KAB model using a health belief analogy based on Protection Motivation Theory (PMT) which states users’ threat responses are shaped directly by their own perceptions and attitudes of their ability to assess, and cope, with threats. The authors found user intention for ISP compliant security behavior was able to be predicted based on perceived susceptibility to IS security risks, perceived costs or consequences of IS security threats, and their self-efficacy in dealing with those IS threats. The application of PMT in other ISP compliance studies found similar findings of the behavioral intention to comply with ISP and competency related to IS security (Anderson & Agarwal, 2010; Herath & Rao, 2009a).

Khan, Alghathbar, Nabi, and Khan (2011) integrated the KAB model with the Theory of Reasoned Action (TRA) and Theory of Planned Behavior (TPB) in a five-step model to evaluate ISA training methods and demonstrate knowledge alone is not enough to produce ISP compliant behavior. According to these theories, behavioral intention is the dependent variable in user behavior, and it is the end user attitude towards compliant behavior and subjective norms about what others think with the accumulation of knowledge that leads to desired behavior. Thus, the authors proposed ISP knowledge influences attitudes leading to normative beliefs about InfoSec leading to compliance and finally InfoSec behavior. In doing so, the authors were able to reach several conclusions. First, the most effective of ISA methods covered most or all the five components including group discussions, email messages, and educational presentations while the least effective were video games, newsletters, and posters. Second, psychological and behavioral theories borrowed from other domains such as education and healthcare can be used to make training methods more effective. Lastly, methods and metrics need to be developed by organizations in order to measure the ISA and behavior of users to improve the competence and performance of users to inform SETA programs.

End-user InfoSec competency (EUSIC) model. Lin and Kunnathur (2013) constructed an end-user information security competence model adding a skills component to KAB models in an organizational context. Based on TRA, TPB, and PMT, InfoSec competence is an iterative process where users develop perceptions about InfoSec risks and the efficacy or benefits of security controls to address them which further influences user attitude and motivation in the development of knowledge and skills leading to ISP compliant behavior. The core concept behind the model is each variable influences the other leading to desired security behavior and only when the behavior is executed is competency demonstrated. Further, higher competence means higher end-user participation in security decision making increasing performance security control use, such as ISP, and increases alignment with organizational goals. Moreover, security, education, training, and awareness (SETA) programs are key enablers to increasing end-user InfoSec competencies as they promote user interaction and lead to increased motivation and personal attachment to organizational InfoSec goals and objectives in protecting IS (Siponen, 2000). The conclusion being InfoSec and ISP compliant behavior impacts, and is impacted by, every aspect of the organization, its security culture and security, education, training, and awareness (SETA) programs. Thus, it is also hypothesized that:

H4: End-users competency (knowledge, skills, and attitude) levels and the level to which they respond in an ISP compliant manner will have implications to the organization.

The final research model (presented at Figure 3) incorporates the competency model from Figure 1 and EUISC Model. The competency model defines the independent variables of knowledge, skills, and attitude applied to phishing in ISP compliance and expected outcome of end-user behavioral response to phishing (dependent variable) with the underlying assumption that motivation and personality also have an influence. The placement of competency variables within EUSIC model accounts for the level of competencies and ISP compliant behavior to be contextualized on the hypothesis the organization impacts and is impacted by end-user response to phishing. Thus, the competency model will be used as the basis for categorization of the data collected in the review method and the use of the overall integrated model will serve as the framework to analyze the data to provide findings and recommendations. The use of this comprehensive model attempts to address previous criticisms of focus on a single variable and include the potential impact of the organization and personality as two key factors of user behavior in the InfoSec domain (Parsons et al., 2014; Vroom & von Solms, 2004). Applying this model to existing phishing research enabled the relationship between user competencies and ISP compliant phishing responses to be examined holistically within an organizational context.

This research study used a qualitative, semi-systematic literature review to examine information system (IS) end-user’s knowledge, skills, and attitudes forming the basis of competence in their ISP compliant behavioral response to phishing. The study results led to an understanding of these competencies and their hypothesized relationship to stronger IS user’s phishing responses rather than what makes them vulnerable to phishing. When the aim of the research is to identify and synthesize previous research in very specific areas, such as phishing, literature reviews are considered an appropriate methodology (Ferrari, 2015). In the information security domain, these types of reviews can make use of past studies to analyze research outcomes about a hypothesized relationship and whether it is supported or not to make recommendations for further research or practical implications (King & He, 2005). Since this research hypothesized the relationship of competencies of the end-user and their behavior against a competency model, a qualitatively oriented narrative review was also considered appropriate to examine such a relationship (Snyder, 2019).

More Quantitative literature review methods, such as meta-analysis and systematic review (SR), were considered but were not chosen for several reasons. Meta-analytic and SR require effect sizes in the individual studies to be reported (or data included to compute effect sizes) for their aggregation and integration to practically occur (Baumeister & Leary, 1997; Siddaway, Wood, & Hedges, 2019). However, previous reviews have indicated a high level of methodological and theoretical heterogeneity in user-focused phishing research (Das et al., 2019; Ferreira & Marques, 2018) making such reviews impossible. The method used in this study is considered useful in addressing these diversity issues where meta-analysis and SR are not possible (Baumeister & Leary, 1997; Snyder, 2019). Further, these methods are time-consuming, typically involve multiple researchers, and require access to large amounts of resources (Haddaway, Woodcock, Macura, & Collins, 2015). This makes these methods impractical in cases such as the current study where there is a sole researcher, limited time and resources.

A semi-systematic review was used in this study given the potential to achieve the same value as more quantitative review methods since a review can examine and integrate many studies to reach broader conclusions not possible for any single empirical study (Baumeister & Leary, 1997). Reviewing all literature associated with phishing is not possible and thus a strategy for finding, selecting, analyzing literature in the review is required if a literature review is to be used as a research methodology (Snyder, 2019). However, unlike SR, there is no standard format or guidelines for the conduct of a narrative review leaving the researcher to decide themselves what articles to select, how to categorize them, and how to frame the results (Ferrari, 2015; King & He, 2005). Haddaway et. al (2015) note many of these freedoms are the limitations inherent in narrative reviews such as publication and selection bias. However, the authors also suggest these limitations can be reduced by applying SR principles to the narrative review in order to increase the depth, rigor, and reliability of the results by making them more transparent, objective, and replicable. Therefore, the narrative review method in this study followed the systematic review approach detailed by various authors (Ferrari, 2015; Siddaway et al. 2019; Snyder, 2019) using the lessons learned of Haddaway et al. (2015). What follows below details the two-stage process used in this study. First, how relevant studies were located and selected. Second, how the identified studies were synthesized and analyzed to test the hypotheses and answer the research questions.

Literature search. In order to increase reliability, four databases were searched: the APUS digital library, ACM, IEEE, and Google Scholar. According to Haddaway et al. (2015) this is enough databases; any more would likely be impractical if not redundant given the incidence of duplication. The keywords used in the searches were “phishing” and “social engineering” to determine if a publication contained a keyword in all fields including keyword, title, abstract, or text. Truncation and wildcard symbols (e.g. “$”, “*”) were also used to account for variations, e.g. spear-phishing, phish, etc. as recommended by Siddaway et al. (2019). In these respects, the literature search followed other comprehensive phishing literature reviews (Das et al., 2019; Ferreira & Marques, 2018; Purkait, 2012a).

Inclusion/Exclusion criteria . Several inclusion and exclusion criteria were defined to allows for the selection of relevant studies to analyze.

Publications. Only peer reviewed journal articles and papers accepted at international conferences were included in this study. Conference papers accepted at peer-reviewed conferences, e.g. Symposium on Usable Privacy and Security (SOUPS), Human Aspects of Information Security and Assurance (HAISA), were included given many user studies of information technology and security issues may only be found in such papers and may not reach journal publication (Das et al., 2019; Ferreira & Marques, 2018). Non-academic papers, such as white papers, and grey literature, such as dissertations, were excluded as their methodological rigor or level of peer review is unknown or are not widely accessible (Lebek et al., 2014; Purkait, 2012a). Further, any literature with only abstract available or not accessible to the author, incomplete (e.g. in progress), or not in English were also excluded.

Timeframe. The date range of studies considered for inclusion is January 2010 through December 2019. The reasoning for this is the significantly low levels of user-focused phishing studies prior to 2010 and 2012 through 2015 with significantly higher levels of research in 2010, 2011, and 2016 onward (Das et al., 2019). The use of a broader time range also allows for broader conclusions and observations to be made (Baumeister & Leary, 1997) by allowing for observation of the variables over time (Ferrari, 2015).

Table 1

An important point to note is some of the variables noted above have been defined or conceptualized differently from one another. For example, self-efficacy is categorized by Lin and Kunnathur (2013) as part of knowledge and skills alongside information security awareness (ISA) whereas Tsohou and Holtkamp (2018) define self-efficacy as part of skills only reflecting it is a self-assessment rather than an objective test of their knowledge. When it comes to knowledge and skills in the IT and cyber-security domain, objective tests as direct assessments or indirect assessments, such as self-evaluations to evaluate technical knowledge and skill have a high positive correlation with actual behavior particularly when conducted together (Wang, 2013). However, Harrison et al. (2016) define the two types of knowledge as subjective and objective. Objective knowledge evaluates actual knowledge or proficiency of the end-user against established test measures whereas subjective uses self-report measures to measure what users think their level of proficiency or knowledge is. In this research, the knowledge component refers to objective tests of knowledge whereas subjective phishing knowledge tests are used to refer to the skills component, i.e. the user’s ability to rate their self-efficacy. Likewise, threat and risk perceptions were categorized by Lin and Kunnathur (2013) as part of the attitude component of competency whereas Tsohou and Holtkamp (2018) these types of perceptions are part of skills as they involve the user’s making personal assessments and therefore, as a function of their actual behavior in ISP compliance, requires knowing the level to which they possess this competence, e.g. to correctly perceive the phishing threat. Notwithstanding differences in authors, the same key variables of the three competency components were included even if described in different dimensions.

Research method/design. Studies not including a detailed methodology, including details of their sample and methodological tools, were excluded from this research. This has been neglected in many user-focused phishing studies and diminishes the external validity of their results (Das et al., 2019). According to Musuva et al. (2019), the three key methods commonly used to assess end-user’s vulnerability and phishing behaviors are phishing knowledge/IQ tests, lab experiments, and naturalistic or field experiments. Knowledge/IQ tests are most beneficial to measure actual knowledge but also those items such as perceptions and attitudes which are difficult to observe and have been demonstrated to be highly correlated to actual behavior in the cybersecurity domain (Wang, 2013). However, these tests are subject to Hawthorne effects and are not generalizable. Therefore, only studies using these tests in conjunction with measurement of actual behavior were included. Similarly, lab-based phishing experiments simulating phishing are subject to the same disadvantages and not reflective of real-world phishing attacks, but they do allow for a much better understanding of user’s behavioral responses by using both qualitative and quantitative methods. Naturalistic or field experiments are the most preferred method for assessing end-user behavior. Notwithstanding the reluctance of organizations to conduct or facilitate them, field experiments allow for direct observation and measurement of user behavioral response to simulated phishing attack in a real-world setting giving these studies high ecological validity and making the results highly generalizable. Thus, studies were included where they measured at least one of the key variables of competency and the actual user response (i.e. clicking/not clicking) to phishing using at least one of these methods (lab or field experiment). Otherwise, they were excluded.

Borderline cases . Two articles were found fitting all the inclusion criteria except the direct measurement of actual behavior. According to Siddaway et al., 2019, borderline cases should be disclosed and considered for inclusion or exclusion based on their merits. The first by Conway et al. (2017), was a qualitative mixed method study using phishing knowledge/IQ tests of bank employees accompanied with follow-up interviews. The second by Thomas (2018) was a qualitative exploratory case study using interviews of CISSP certified information security professionals who have dealt with actual spear-phishing outcomes and end-users in their individual organizations. While not measuring actual behavior directly, these professionals have at least five years of experience observing end-users in terms of actual successful phishing attempts and investigation and remediation giving them first hand in-depth insights into actual end-user behavior as subject matter experts (SME). Baumeister and Leary (1997) note one critical requirement in literature reviews as a methodology is the methodological diversity because the more results show similar results over many different methods (as opposed to number), the less likely the hypothesis is false. Further, the methodological diversity of user-focused phishing studies generally has been criticized for lacking qualitative analysis, including interviews, thus missing greater understanding of user experience and behavior (Crossler et al., 2013; Das et al., 2019). Therefore, both studies were chosen to be included.

Final literature sample. After completing the selection process, the total number of articles included in the sample for analysis was N=28 (see Table 2 below). The years of the study included in the final sample replicated similar trends in distribution in publication as found in the recent systematic review of the ACM library of user phishing research completed by Das et al. (2019). In terms of sample size, the final number of 28 is like other social engineering systematic reviews where 30 was used as the number of studies needed to overcome bias in a systematic review over more than a decade time span (Aldawood & Skinner, 2019). Likewise, the number of studies located in journals (n=18) and conference papers (n=10) is like the ratio found in other literature reviews concerning phishing and information security (Ferreira & Marques, 2018; Lebek et al., 2014; Purkait, 2012a). Many studies found for inclusion studied more than one phishing competency variable and in more than one competency dimension.

Data collection, synthesis, and analysis. The following was extracted from each study to enable the research questions to be answered and test the hypotheses: sample frame and size, method used (lab or field experiment), key findings, relationships between variables (where such is examined), effect size (if reported), statistics (if reported), and limitations. This followed the best practices for systematic reviews as a research methodology (Ferrari, 2015; Snyder, 2019). Subsequently, the studies and data were categorized according to the competency components (Attitude, Skills, and Knowledge) to summarize and synthesize study findings within each component in relation to the theoretical framework, i.e. ISP compliance/non-compliance. According to Baumeister and Leary (1997), a systematic review (qualitative or quantitative) may reach one of four conclusions in relation to a hypothesis: based on the evidence found, it is correct; it is likely correct should be considered true unless contrary evidence exists; it is unknown whether the hypothesis is true or false; or it is false. The findings of the studies were categorized according to these outcomes to summarize the results of the hypothesized outcomes in this research. Finally, these findings were analyzed in order to answer the research questions.

Although this research was conducted using best-practice systematic review principles to overcome bias, several possible limitations of the sample located exist for this literature review method must be noted. First, is the use of a sole researcher where best practices of systematic reviews usually entail at least two reviewers for literature search and selection. However, as noted by Siddaway et al. (2019), this is a common occurrence where the researcher is a student and established guideline dictates the requirement and what is most important is the review provides enough information to demonstrate the review was completed according to best practices. Second, systematic reviews may be subject to publication bias by slanting their inclusion criteria towards statistically significant (p<.05) studies and exclude grey literature (both published and unpublished) from inclusion (Siddaway et al. 2019). Inclusion criteria for research studies made no such requirement for statistical requirement (all studies meeting the inclusion criteria were included regardless of statistics but have been reported where they exist) but did exclude grey literature for the reasons noted above (which may meet all other criteria).

Third, study quality tools were not used as recommended in other systematic reviews (Sommestad et al., 2014) to assess sensitivity of results and quality differences which may lead to bias. However, Siddaway et al. (2019) note study quality tools may not be useful as many authors believe as disagreement exists as to their value. In the place of a quality tool, this study instead followed the practice of reporting study attributes, so the results are transparent to the reader (Ferrari, 2015). Fourth, the range of articles found for inclusion in this review were limited by keywords used and selection criteria. Therefore, relevant studies exist may not have been located because they did not contain a keyword or were not accessible to the author.

The 28 phishing studies comprised 18,965 subjects across the three competency components where the competency component was observed in the study indicated by an X (knowledge, n = 22; skills, n = 28; attitude, n = 14) as shown in Table 3.

Table 3

A summary of the findings and limitations of each study is presented in Table 4. The findings of each study are categorized according to findings of higher levels of each competency and one of three possible outcomes: a) higher compliant ISP phishing responses (hypothesized relationship), b) non-compliant ISP phishing responses, or c) no relationship or inconclusive result. Table 5 provides an overview of these categorizations. In the sections that follow, the results of the review of phishing studies measuring the dimensions of competency (knowledge, skills, and attitude) and actual behavioral responses are detailed in turn.

Table 4

Notes. Correlations are provided where available in selected study.

(+) denotes subjects successfully phished more according to higher levels of the variable;

(-) denotes subjects were phished less with higher levels of the variable.

K = Knowledge; PC = perceived consequences; PCE = perceived control effectiveness; PT = perceived threat; PV= perceived vulnerability; SE = self-efficacy

Table 5

Table 6

Note. Authors in bold denote field experiment; authors in italics denote sample size > 200.

Similar results showing the relationship of knowledge and ISP compliant has been found also in smaller sample studies and those using qualitative methods. Harrison et al. (2016) found a significant correlation (.37, p < .01) between those with the highest levels of aspects of ISP knowledge and their ability to detect and avoid phishing. The case study by Thomas (2018) surveying seven CISSP professionals who have dealt with actual phishing cases affirmed this notion noting higher levels of basic knowledge coupled with an awareness of the size and complexity of the phishing problem are necessary (and possessed by most who successfully deal with phishing) but lack in most phishing victimization they encounter. Conway et al.’s (2017) study of bank employees revealed those with higher levels of knowledge of ISP, including direction on what to do and where to go with questions or observations had been better at avoiding phishing; those who did not, made up their own ideas of what to do and they were usually false. Yang et al. (2017) conducted controlled phishing attacks with website warnings on 63 students using a control group, pre/post-test design, and a follow-up interview. In the pre-test, almost one half of subjects had not heard of or knew the meaning of phishing, did not understand or ignored the warnings, or did not understand the nature of the phishing problem. The phish rate was 100% in the both control and experimental group in the first test. After phishing specific training including rules, procedures, and tools, the phished rate dropped to zero in the subsequent attack. Knowledge was considered by the participants to be the key factor in understanding the nature of the problem and the nature of the warnings. Greene et al. (2018) followed 70 employees in a natural workplace setting and their responses to repeated controlled phishing attacks over 4.5 years to understand who clicks phishing emails and who does not. They found even those subjects who only knew the definition of phishing and how it continues to be an evolving problem avoided being phished.

Contradictory results were found in two studies. Canfield et al. (2016) examined knowledge, confidence, and judgement in the context of phishing detection and subsequent behavior finding higher levels of knowledge associated with higher levels of phishing victimization in both cases. Komatsu et al. (2012) examined the difference between intention and behavior and found where intention to use higher levels of knowledge and IT competence was present in self-report surveys, it prevented compliant behavior. No relationship between knowledge and ISP compliant phishing responses existed in five studies examining knowledge. The largest of the studies found both the group phished least (at 3%) and the group phished most (66%), subjects’ levels of knowledge were proportionally equal in both groups (Abbasi et al., 2016). A similar result was found by Broadhurst et al. (2019) who subjected targets to generic, tailored, and spear-phishing emails. Although the phishing success rate increased with more specific approach, subject’s knowledge level was not related to their responses.

Skills. Studies examining at least one of the variables of the skills dimension of competency totaled 28 comprising all subjects included in the studies ( n = 18965) in at least one variable (though many studies covered more than one variable). Table 7 below depicts the categorization of each study including a skills variable according to whether higher levels of the skill variable resulted in ISP compliant responses or not.

Table 7

Note. Authors in bold denote field experiment; authors in italics denote sample size > 200.

Six studies ( n = 1,175) found no relationship between self-efficacy and phishing behavior or their findings were inconclusive. Abbasi et al. (2016) found equal proportions of high self-efficacy ratings were accurate against their actual knowledge were present in both the best and worst groups for avoiding phishing. In another study, self-efficacy did not correlate with actual phishing behavior regardless of users self-reported intentions on survey instruments (Flores et al., 2013). Halevi et al. (2013) also found no effect of knowledge and self-efficacy ratings but rather personality was the most important variable influencing the self-efficacy rating. The remainder of the studies no found effect of self-efficacy (Broadhurst et al., 2019; Nguyen et al., 2017).

Ten studies ( n = 4, 238) found higher ratings of self-efficacy resulted in non-compliant ISP behavior. In one field study, both control and experimental groups reported knowing what phishing looks like, how to detect it, and high confidence in their ability to do so (Jensen et al., 2017). However, of 13.2% of the experimental group phished, those with the highest rated self-efficacy were phished most. Diaz et al. (2019) found users overestimated their knowledge and awareness as 92% of users were phished at least once in three controlled phishing attacks. A similar finding occurred in another study where 89% rate their self-efficacy as high but 92% were phished successfully (Hong et al., 2013). In other studies, phishing attack results indicated users with high-self efficacy who should have been able to avoid phishing were not (Caputo et al., 2014). Purkait (2012b) found only 21% of those rating themselves expert were able to avoid being phished. In other studies, self-efficacy was influenced by other factors. Vishwanath et al. (2016) found users rating high self-efficacy were not successful at avoiding phishing if the perceived risk (threat and vulnerability) were assessed as low; they were phished far more. Thomas (2018) concluded too much or too little confidence in relation to their actual ability had the same effect on phishing outcomes; users are weaker and phished at a much higher rates.

Perceived threat and vulnerability . The effect of threat and vulnerability perceptions was examined in 17 studies ( n = 13, 617 subjects). The three largest field studies ( n = 11,932) found users are phished less where they perceive the threat of phishing and their vulnerability to phishing higher and more accurately (Burns et al., 2019; Jensen et al, 2017; Mohebzada et al., 2012). These studies found the majority of users successfully phished were not aware of the gravity of the phishing threat or how vulnerable they were. Several other smaller studies showed the same results (Arachchilage et al., 2016; Yang et al., 2017) with one showing almost all phished had underestimated the threat (Hong et al., 2013). Some studies found higher, more realistic threat and vulnerability perceptions led to better phishing outcomes because it caused users to be more suspicious and more systematic in their approach to emails relying less on institutional or external anti-phishing measures (Abbasi et al., 2016; Vishwanath et al., 2016). Other studies showed the same relationship between user vulnerability estimates and reliance on other measures like believing someone else will take care of the problem, e.g. IT department, because user’s knowledge of the threat is low and their vulnerability underestimated (Conway et al., 2017; Thomas, 2018). The result is users are less resilient, phished easier, and more often.

The specific relation of phishing knowledge to formation of correct perceptions of threat and vulnerability (and avoidance) was found in one study with a strong correlation (.64, p < .001) (Kleitman et al., 2018) contradicting earlier results showing no effect between knowledge and perceptions but rather personality as the factor (Halevi et al., 2013). A later study by Halevi et al. (2015) found users underestimating threat and vulnerability will be phished more but even more so where certain personality traits are present. Wright and Marett (2010) found no significant relationship between user threat and vulnerability perceptions and their ability to avoid phishing. One study did find contradictory findings. House and Raja (2019) studied the fear of phishing in a field experiment ( n = 223) and found a negative correlation (-.150, p < .001) between higher threat and vulnerability perceptions and phishing avoidance. From both survey data and interviews, they found users who had higher perceptions of threat and vulnerability were phished more because in addition to perceiving the threat as very severe (though more severe than present) they also perceived little could be done to effectively avoid the problem.

Perceived consequences . The effect of perceived consequences was examined in nine studies ( n = 3,720 subjects). Kim et al.’s (2019) study of 1248 employees in multiple controlled phishing attacks found a significant influence of perceived sanctions and consequences. Where the employees were made aware of personal sanctions and consequences that result for the organization through training a specific deterrent effect caused employees to be five times less likely to fall for phishing attempts in future attacks. Other large field studies also demonstrated the effect of users stronger at avoiding phishing were more aware of the consequences of non-ISP compliance both personally and to the organization (Caputo et al., 2014; Vishwanath et al., 2016; Wright & Marett, 2010). One study, by Greene et al. (2018), also found a difference between the nature of the consequences. People who “clicked” were more concerned about the consequences of not clicking, e.g. not responding to a legitimate request, whereas those who did not “click” were more concerned about the potential consequences of responding, e.g. phishing, virus, etc. Thomas (2018) found users who fall victim to phishing are both more unaware of, and underestimate, the real and actual consequences of phishing to themselves and their organization 98% of those successfully phished had underestimated the severity of the consequences (Hong et al., 2013). Two studies failed to find any relationship while other results found a relationship between accurate perceived consequences and non-compliant ISP responses.

Attitude. The effect of the attitude dimension of competency was examined in 9 studies ( n = 18207) Table 8 below depicts the categorization of each study including some aspect of attitude according to how a more positive attitude resulted in ISP compliant responses or not.

Table 8

Note. Authors in bold denote field experiment; authors in italics denote sample size > 200.

Several studies found users who realistically view the phishing risk and take the threat, ISP, and phishing controls seriously are phished less (Mohebzada et al., 2012; Vishwanath et al., 2016). Others found users with positive attitudes towards ISP, anti-phishing training, and tools were phished less versus those users who felt training and policies were not valuable (Caputo et al., 2014; Greene et al., 2014). In a multi-round phishing attack, Burns et al. (2019) found employees least phished were the most motivated with positive attitudes towards ISP programs and training framed towards them resulting in increased security culture buy-in. Lower self-efficacy and knowledge were related to attitude whereby those phished more frequently were lower in both because their attitude was negatively oriented towards protecting themselves (e.g. for fear of looking naïve) rather than the organization (Conway et al., 2017; Jensen et al., 2017). Similarly, Halevi et al. (2013) found users with more positive attitudes towards the organization ISP avoided phishing since they engaged in more protective type behaviors. Other experimental studies showed higher levels of skills had more positive attitudes towards protection from phishing (Arachchilage et al., 2016; House & Raja, 2019; Thomas, 2018). Two studies found equal attitudinal characteristics in both phished and not phished groups preventing any observation of relationship (Abbasi et al., 2016; Broadhurst et al. 2019). Komatsu et al. (2012) found similar results in those phished and not phished but that motivation and attitudes towards protection varied depending on the variable it was compared with, e.g. self-efficacy, perceived consequences. However, the variance was unpredictable and could not be accounted for.

Overall, the results of the literature review conducted above generally show, by magnitude or order of effect, that higher levels of compliant ISP responses (or avoiding phishing) result when users possess higher levels of competencies as shown in Table 9. However, the effects on phishing response of some components of competency, like knowledge, are clearer than others such as self-efficacy. As noted earlier, a review may reach one of four conclusions in relation to a hypothesis: based on the evidence found, it is correct; it is likely correct and should be considered true unless there exists evidence to the contrary; it is unknown whether the hypothesis is true or false; or it is false. The findings of the studies examined in this research are now analyzed in relation to their support of this research’s hypotheses.

Table 9

H1: Higher levels of end-user knowledge in relation to phishing related threats and measures will be associated with ISP compliant behavioral responses to phishing.

The knowledge dimension of competency was included in 22 of the 28 studies and included 96% of all subjects in all studies as noted in Table 9. Higher levels of knowledge were found present in ISP complaint phishing behavior in 15 of the 22 studies accounting for 94% of all subjects in the studies observing knowledge as a variable. A total of 11 of those studies were ecologically valid field experiments with the majority ( n = 8) being field experiments with large sample sizes over 200. Those studies showed no relationship between higher levels of knowledge and ISP compliant behavior totaled 5% comprised of five studies were mostly small field studies finding users with higher levels of knowledge were equally phished in proportion to those with lower levels of knowledge. The number of studies indicating higher levels of knowledge present in non-compliant ISP behaviors consists of two small lab experiments ( n = 255). Therefore, based on the evidence found, Hypothesis 1 is correct.

H2: Higher levels of end-user skill will be associated with an ISP compliant behavioral response to phishing.

Overall, the four skills variables (self-efficacy, perceived threat and vulnerability, perceived control effectiveness, and perceived consequences) were examined in all 28 studies.

Self-efficacy . This variable was included in 22 studies accounting for 35% of the total subjects. Higher levels of self-efficacy were related to ISP compliant behavior in studies with 19% of subjects whereas higher levels of self-efficacy were related to non-compliant phishing responses in studies with 64% of subjects with both categories having an equal proportion of field and lab experiments. The difference between the two is those users who were compliant accurately assessed their knowledge whereas non-compliant subjects had inaccurately assessed their knowledge which does not necessarily mean the hypothesis is untrue given some studies used subjective assessments of knowledge only. Inconclusive results or no relationship was found in five studies totaling 18% of subjects.

Perceived threat and vulnerability . This variable was included in studies accounting for almost three quarters of subjects in all studies with the studies containing 94% of subjects. These studies found accurate user’s perception (or assessment) of the phishing threat and/or their vulnerability is associated with higher levels of ISP compliant behavior. Studies finding no relationship or contrary findings contained few of the total subjects, but their effects were strong (Wright & Marett, 2010) or may have been the result of other moderating factors.

Perceived control effectiveness . This skills variable was examined in eight studies comprising 5% of the total subject population. However, the studies finding users who perceive controls as effective and avoid phishing comprised 98% of the sample population of this variable. Only one small ( n = 21) lab experiment found inconclusive results.

Perceived consequences . This variable was studied in nine studies comprising 20% of the total subject population. Users perceiving the consequences of phishing either to themselves and/or the organization was found to be associated with ISP compliant responses to phishing in seven of the nine studies comprising 92% of subjects. Of these, five were field studies and two were large samples. Only two studies found no effect of perceived consequences.

Although the results in three of the skills variables are high, the lower overall population sizes and the equivocal self-efficacy results serve to moderate these results. Therefore, Hypothesis 2 is likely correct and is considered true unless further evidence to the contrary exists.

The attitude dimension of competency was included in 14 of the 28 studies and included 75% of all subjects in all studies ( n = 14,293). The association of positive attitudes to ISP complaint phishing behavior was found in 11 of the 14 studies accounting for 95% of all subjects in the studies included some aspect of attitude as a variable. Most of these studies ( n = 8) were field experiments with the majority ( n = 6) being field experiments with large sample sizes over 200. The three studies showed no relationship between higher levels of attitude and ISP compliant behavior contained 5% of the subject population finding similar attitudinal characteristics in both ISP compliant and non-compliant groups preventing any conclusion. Therefore, based on the evidence found, Hypothesis 3 is also correct.

Hypothesis 4 stated that end user competency (knowledge, skills, and attitude) levels and the level to which they respond in an ISP compliant manner will have implications to the organization. Based on the findings above, the higher levels of knowledge, skills, and attitude that end-users possess, the better their performance in complying with ISP in response to phishing will be. In the aggregate total effect of all three competency dimensions, higher levels of knowledge, skills, and attitude were associated with compliant ISP responses to phishing in studies comprising 85% of all subjects as noted in Table 7. This is consistent with previous research indicating that knowledge and awareness, risk perceptions, attitudes and motivation levels have the most influence on actual behavior (Metalidou et al., 2014). The competencies showing the strongest effect on compliant ISP responses to phishing were knowledge, attitude, and the perceived threat/vulnerability and perceived control components of the skill competency dimension (Table 7 refers). This concurs with the other findings in the wider ISP compliance research domain finding these variables amongst the strongest predictors of ISP compliance generally (Sommestad et al., 2014). These findings will now be discussed, in turn, against wider ISP compliance and phishing research as well as the end-user competency model.

As noted earlier in this paper, the value of a competency model is that it defines the performance results expected of a user that can be assessed and linked to programs, e.g. training and policies (Teodorescu, 2006). In the phishing context then, phishing competencies can be defined, performance assessed or measured, and training or resources necessary provided to address any deficiencies in phishing competencies. Using a competency model, the higher levels noted above represent the organizational performance expectations of users with respect to phishing. Thus, organizations intent on preventing their information systems from being compromised by phishing through their users’ need to ensure that their users possess the desired levels of these competencies. The findings of this study also represent the measurement or assessment of end-users showing subjects in both compliant and non-compliant categories possessed different levels of competencies, representing deficiencies, as discussed below.

Perceptions of threat and vulnerability. Accurate perceptions of threat and vulnerability are skills associated with ISP compliant responses. However, most studies in this research found users victimized possessed low threat knowledge and vastly underestimated their vulnerability to being phished. This may be due to a false belief in the IT department or an unwarranted reliance on institutional measures (Conway et al., 2017; Thomas, 2018). Of course, the opposite may be true. Users may perceive the risk of phishing to be so high nothing can be done, by them or anyone else, which leads to the same effect of being phished (House & Raja, 2019). Thus, even where the same level of knowledge may have existed, users in these studies varied in their perceptions of the phishing threat and their vulnerabilities. This may have been due to other factors, such as personality noted in two of the studies (Halevi et al., 2013; Halevi et al., 2015), or it may be since different people perceive risk differently even with the same information (Jensen et al., 2017). Therefore, organizations must not only provide information for users to understand the phishing threat and vulnerability but also understand how their users perceive the phishing risks (to both the themselves and the organization).

Perceived control effectiveness. Similarly, the findings indicated users most phished had the lowest perceptions towards the effectiveness of controls. They underestimated the usefulness of tools they could use to cope with phishing and overestimated institutional controls, e.g. filters. This may be due to a lack of basic computer knowledge, e.g. what a filter does and what it does not, or a lack of awareness and understanding of the usefulness of the user-oriented controls. The problem with this is gaps in IS security against phishing occur because of users lack knowledge and skill about technical limitations (Frauenstein & von Solms, 2014) and misunderstand the significance what anti-phishing tools do preventing them from understanding the security risks their actions (or inactions) cause (Furnell et al.; 2018, Iuga et al., 2016). Moreover, if they do not perceive the security controls as useful or easy to use, they will not likely adopt their use and leave the user feeling less competent (Padayachee, 2012). The findings in these studies demonstrated users with the most ISP compliant response to phishing had a tempered view of phishing recognizing some phishing attempts will always get though organizational filters and perceived the usefulness of tools they could use to avoid it as beneficial. Considering this, organizations must not only provide users with knowledge and training of phishing controls but also understand how their users perceive the effectiveness of anti-phishing measures or controls correctly and as realistic, beneficial and effective.

Perceptions of consequences. The users most ISP compliant in their responses possessed accurate and realistic perceptions of the consequences of phishing. Those users most phished in these studies were unaware and underestimated the consequences of phishing both to themselves and to the organization. The effect of perceived consequences was shown to have an effect in two ways. First, individual perceived sanctions and consequences of phishing had a significant influence showing a specific deterrent effect in increasing ISP compliant responses to phishing when users were informed of such though training. This result is contrary to previous findings in wider ISP compliance research where sanctions were found to be one of the worst predictors of ISP compliance (Sommestad et al., 2014). Users also responded differently depending on how they perceived the nature of the consequences. Those users responding to phishing attempts were concerned about the consequences of not responding to legitimate emails whereas ISP compliant users were most concerned about the consequences of responding, i.e. malicious attachments. The difference between these two indicate the problem with imposing consequences on those who may be victims of phishing considering wider ISP compliance research. First, a well-intentioned user is unknowingly deceived into destructive behavior by a highly engineered phishing attack which is different than other intentional information security deviant behavior (Crossler et al., 2013). Given this, it would seem relatively unfair to punish or sanction users in any severe way given the nature of phishing and the fact that even highly knowledgeable and skilled users have also been victimized. Second, if consequences are perceived as severe then user productivity may suffer as users fear clicking on anything (Greene et al., 2018). Further, it is widely noted that even the best phishing defenses and programs will never be able to completely ameliorate the risk of phishing. Rather, organizations must be realistic about what anti-phishing training and policies can do (Greene et al, 2018; Jensen et al., 2017). Therefore, organizations must consider carefully how they wish to deal with users who fall victim to non-compliant ISP behavior and ensure they understand how their users perceive the consequences to themselves and the organization.

Attitude. A positive attitude is a necessary competence as demonstrated in those users best at avoiding phishing. For the most part, users with negative attitudes towards policies and training were more likely to be phished as they may not have accurately perceived the phishing threat or consequences and were the least motivated to engage in protective behaviors either for themselves or the organization. This is not surprising when considered against the Theory of Reasoned Action (TRA), Protection Motivation Theory (PMT) and Theory of Planned Behavior (TPB) contained within the End User Information Security Competence model which are predicated on the combination of knowledge, perception, and attitudes in shaping ISP compliant behaviors. In some studies, user’s non-ISP compliant behaviors to phishing were related to their interests in looking out for themselves versus protecting the organization. Although the subjects had the requisite knowledge and skills to respond compliantly, they did not; their actual behavior did not match their intended behavior as would be predicted by the TRA and TPB (Alsharnouby et al., 2015; Canfield et al., 2016; Flores et al., 2013; Komatsu et al., 2012). One possible reason for this is a lack of positive attitude.

This is important because, even with adequate knowledge and skills, negative attitude and lack of commitment to the organization will prevent the expected level of ISP compliance (Metalidou et al., 2014; Tshohou & Holtkamp, 2018). As a critical component of the competency model, attitude is an underlying driver of skills and knowledge and a negative attitude causes a gap between knowing and doing (Cox, 2012). Further, positive attitudes and motivation were found to increase security culture “buy-in” and possibly be dependent on other variables such as how they perceive the effectiveness of controls and self-efficacy. This is consistent with ISP compliance research that proper motivation and attitude influence knowledge and skills growth which, in turn produce more positive attitudes furthering a security culture (Padayachee, 2012) though over longer periods of time (Siponen, 2000) and helps to close the knowing-doing gap in user behavior. Therefore, organizations must create and encourage an environment supporting ISP compliant attitudes and motivation to deal with phishing in a consistent and ongoing manner.

In addition, the findings also indicate the effects of these competencies are not necessarily equivocal and the competencies above interact with one another or other factors, such as personality, thereby impacting ISP compliant responses to phishing. End-user behavior is influenced by many factors at the individual level and organizational and is a long-held precept in information security; however, it is not often considered (Vroom & von Solms, 2004). The current information security competency model accounts for this as an iterative process between knowledge, skills, and attitude of the user influenced by other psychological factors, such as personality, supported by the organization and its culture, ISP, and SETA programs. This is now considered to provide recommendations to the organization in addressing this end-user phishing problem though the improvement of their competencies.

Considering the above within an overall competency model, organizations need to determine how user’s competency levels can be raised to enable them to perform in an ISP complaint manner to phishing and to the way the organization must support the user to develop expected levels of competency through its ISP, SETA programs and security culture. Understanding each organization has different needs and will define competencies and their security culture differently, general recommendations are made drawing from wider ISP and phishing as well and the findings and implications noted above in this research as follows:

ISP. Organizations should ensure end-users are aware of what is expected of them regarding phishing in terms of what an ISP compliant response and the procedures they should follow in doing so. This is particularly important considering that many organizations do not have ISP that deal with phishing specifically (Kim et al., 2019). Further, for those organizations that do have ISP pertaining to phishing they may be too basic, out dated, and/or not account for how a security breach from phishing is vastly different that other forms of information security risks because of the deceptive nature of phishing and its effects on unsuspecting or ignorant users (Crossler et al., 2013). As a results, these policies should be clear and understandable to a wide range of users including those with only a basic understanding of computers and phishing to those with more advanced knowledge as reflected in the subjects throughout this study. This is critical because if users do not understand what is expected they will be less likely to comply and the policy will not be implemented and, therefore, be ineffective (Metalidou et al., 2014). Beyond just clear direction of expectations, ISP concerning phishing should also detail the “why” of the policies and how enforcement will occur. This enables both a “buy-in” attitude contributing to the overall security culture and, for those who cannot or will not comply with ISP and provides a mechanism for addressing the problem (Furnell et al., 2018). Moreover, this top-down approach sets the stage for users to be more receptive to ISP and subsequent training programs making compliant behavior more likely further enhancing the security culture (Sherif et al., 2015b). Findings in this study indicated these ISP were the basis of high knowledge competency levels and influenced all the components of skills and attitude as well either positively, to users who performed compliantly (knowledge of ISP), or negatively, to those who did not (little or no knowledge).

Training. In the general sense, basic training is an essential of any information security program. Security, Education, Training and Awareness (SETA) programs should explain not only what is expected in response to phishing but why it is important where the former is knowledge based and the latter affects attitude (Parsons et al., 2014). Further, end users must be made aware of the threats and consequences of phishing to both themselves and the organization. The most successful users at avoiding phishing in this research were those most able to perceive both these factors accurately consistent with ISP literature (Bulgurcu et al., 2010). The everchanging nature of phishing makes it imperative this training be continuously updated and delivered as many ISP compliant users in the studies had credited current and frequent training as the most beneficial to their success in avoiding phishing. This makes the practices currently used by many organizations such as provision of training on an annual or one-time basis inadequate. Likewise, training must include phishing specifically because it is different from other information security practices as discussed. Despite the single finding in this study showing a positive effect of sanctions, harsh consequences to the user who falls victim will likely have a counter-productive effect to work and reduce the likelihood users will act as an “ally” rather than a vulnerability by reporting phishing attempts making it through filters to them (Greene et al., 2018). This will also help to motivate and assist users in understanding their choices and guide their behavior (Wash & Cooper, 2018). Therefore, users should be trained to respond in a positive way by reporting them so organizational filters can be updated, controls updated, and users understand the vital role they play in protecting information from phishing threats. This may also increase the likelihood of more positive user attitude and further reinforce a culture of information security as a result.

Another aspect for organizations to consider is a targeted approach to training. Some of the subjects in this study possessed high levels of competency while others were lower, and this influenced their responses to phishing. Assessing individual end-user attributes and customizing training to their needs has been widely noted in the phishing literature as being beneficial in assisting users in keeping up with the changing phishing threat and addressing competency deficiencies through targeted measures (Abassi et al., 2016; Burns et al., 2019; Goel et al., 2017; Harrison et al., 2016; Jensen et al., 2017; Thomas, 2018). This is important considering the stable nature of those earlier noted user attributes that may influence phishing susceptibility, such as age and gender, that cannot change in contrast to user competencies that can. Further, the evidence has shown where anti-phishing training matches the user’s needs, attributes, and their preferences it will be most effective regardless of the type of training (Abawajy, 2014). With respect to the subjects in this study, targeted training could help the high self-efficacy group overestimating their knowledge in one way and use other training methods to assist the high-risk group of users underestimating their knowledge and skills. It may also assist in ensuring phishing controls are not misunderstood, or their effectiveness perceived inaccurately.

One method for training organizations should consider is the use of embedded training where the organization (or a third party) subjects its users to a controlled phishing attack followed by immediate feedback. Although organizations have been reluctant to conduct such attacks or many reasons (Musuva et al.,2019) they remain the most realistic and ecologically valid method for testing users’ actual responses to phishing as opposed to self-reported intention or lab experiments shown by a number of studies in this research to be different (Canfield et al., 2016; Flores et al., 2013, Komatsu et al., 2012). Many of the authors in the current research used this method in their research for this reason as they are not subject to the same limitations as surveys or questionnaires where users may not be honest fearing negative effects such as consequences to their job (Crossler et al., 2013). However, embedded training can accomplish several things. First, it can act as a training tool ensuring users are aware of real-world phishing threats enabling their ability to detect and avoid phishing attempts and make them aware of what actions to take and the reporting processes. Secondly, it can act as an assessment tool for the organization to ensure the reporting process is working and track any adjustments required. Lastly, embedded training provides an unbiased and accurate assessment of user response to phishing attempts enabling the organization to identify users who may require targeted additional training or resources provided to them.

Measurement. Regardless of the type of training provided, organizations need to have valid measurement tools to assess both the user, anti-phishing programs and their own security culture. From the individual perspective, measuring user in the information security domain is seen as no different than any other type of performance assessment (Vroom & von Solms, 2004). This type of assessment is particularly important given the effect of higher or lower levels of knowledge, self-efficacy, and perceptions on end user ISP phishing performance found in this study. Further, the effect of personality and the interrelated nature of the competencies also play a role and need to be considered. There are several information security assessment tools available recommended to be used. The Human Aspects of Information Security Questionnaire (HAIS-Q) is one tool measuring all these requirements. The HAIS-Q is a 63-item scale developed by Parsons et al. (2014), assessing end-user knowledge, attitude, behavior, and personality. Other assessment tools include the Security Behavior Intention Scale (SeBIS) (Egelman et al., 2016) measuring user attitudes of computer security; the Security SRK (security skills, rules, and knowledge) instrument measuring behavior towards security controls and warnings (Rajivan et al., 2017); and an IS vocabulary test that measures awareness (Kruger et al., 2010). All these tools were designed to provide organizations with a point of reference to evaluate their information security programs and strategies both in the short and long-term. Although not phishing specific, the tools could be helpful to ensure organizations understand the various competency levels of their users and enable adjustments to training or ISP accordingly.

From the organizational perspective, the effectiveness of their SETA programs, their information security culture, and their user’s behavior within it should be also be accounted for. Sherif et al. (2015b) detailed two such tools may be useful to organizations in accomplishing this. The Security Culture Review and Evaluation Tool (SeCuRE) measures and tracks changes in information security awareness, measures and assists information security behavior change, and assesses and assists changes in information security culture. The second tool, QinetiQ, also includes information security awareness, measures information security behavior but measures and enhances information security culture. Although not phishing specific, either of these survey and analysis tools should provide enough ability for an organization to evaluate its overall security culture and its programs in relation to its users and phishing.

This study contributed to IS security research in several ways. First, it demonstrated competencies positively affect ISP compliant behavioral end-user phishing responses. Overall, the higher the level of knowledge, skills, and attitude end-users possess, the better their performance in complying with Information security policies (ISP) in response to phishing (i.e. avoiding phishing). Second, this research confirmed wider ISP compliance research as applied to phishing indicating knowledge as a fundamental competency requirement in ISP compliant behavior because it underpins all competency dimensions and prepares users to practice ISP compliant behavior. Although the self-efficacy findings in this research contradicted wider ISP compliance research, they do indicate the critical role of accurate levels of end-user self-awareness and knowledge in responding to phishing. Either users did not have the requisite knowledge or skill and/or overestimated their knowledge and skill. The implication being that those with the least amount of actual knowledge then are not able to execute the skills of accurately assessing the phishing risk, the effectiveness of security controls, and will be less confident (or over-confident) in their abilities to be ISP compliant in response to phishing. So, organizations need to identify, through assessment, both end-users who have the least amount of knowledge and those who overestimate their knowledge and skill.

Lastly, a competency model was shown to be a useful method for organizations in dealing with phishing by enabling the organization to define, assess, and improve phishing competencies. Users may possess different levels of competency and respond differently to phishing differently because of them. Further, users may be susceptible to phishing because of factors that cannot be changed, e.g. age, gender. However, a competency model enables these differences to be accounted for and addressed against a standard benchmark in a structured, methodical, and deliberate manner to achieve the organizations desired end state of ISP complaint behavior from end users. Using this approach, phishing competency levels of end-users within organizations can be identified and shortfalls in competencies improved. In addition to the development of strong ISP, organizations must also ensure knowledge through targeted training and ongoing assessment of all competencies to ensure users perceive risks in the desired way and remain properly motivated through the security culture. Although not tested empirically as competencies would traditionally be measured, it was concluded a competency model has value to organizations in defining, measuring, and guiding users towards ISP compliant behavioral responses to phishing if used and applied as in other domains. Returning to the epigraph at the beginning, improving end user phishing competencies may enable organizations to protect their information from the everchanging phishing problem by addressing the challenge of changing phishing behavior helping users prevent missteps by avoiding the “hook”.

Anti-Phishing Working Group (2019). Phishing Activity Trends Report: 2nd Quarter Report.

Abbasi, A., Zahedi, F. M., & Chen, Y. (2016, September). Phishing susceptibility: The good, the bad, and the ugly. In 2016 IEEE Conference on Intelligence and Security Informatics (ISI) (pp. 169-174). IEEE.

Abawajy, J. (2014). User preference of cyber security awareness delivery methods. Behaviour & Information Technology, 33(3), 237-248.

Aleroud, A., & Zhou, L. (2017). Phishing environments, techniques, and countermeasures: A survey. Computers & Security, 68, 160-196.

Anderson, C. L., & Agarwal, R. (2010). Practicing safe computing: a multimedia empirical examination of home computer user security behavioral intentions. MIS quarterly, 34(3), 613-643.

Arachchilage, N. A. G., Love, S., & Beznosov, K. (2016). Phishing threat avoidance behaviour: An empirical investigation. Computers in Human Behavior, 60, 185-197.

Arief, B., & Adzmi, M. A. B. (2015). Understanding cybercrime from its stakeholders' perspectives: Part 2--defenders and victims. IEEE Security & Privacy, 13(2), 84-88.

Ba, L. (2018). The Biggest Email Security Challenge Facing Organizations Today. Security, 55(9), 62–65. Retrieved from http://search.proquest.com/docview/2109286978/

Boss, S. R., Kirsch, L. J., Angermeier, I., Shingler, R. A., & Boss, R. W. (2009). If someone is watching, I'll do what I'm asked: mandatoriness, control, and information security. European Journal of Information Systems, 18(2), 151-164.

Broadhurst, R., Skinner, K., Sifniotis, N., Matamoros-Macias, B., & Ipsen, Y. (2019). Phishing and Cybercrime Risks in a University Student Community. International Journal of Cybersecurity Intelligence & Cybercrime, 2(1), 4-23.

Bulgurcu, B., Cavusoglu, H. and Benbasat, I. (2010) Information security policy compliance: An empirical study of rationality-based beliefs and information security awareness, MIS Quarterly, 34(3), 523-548.

Burns, A. J., Johnson, M. E., & Caputo, D. D. (2019). Spear phishing in a barrel: Insights from a targeted phishing campaign. Journal of Organizational Computing and Electronic Commerce, 29(1), 24-39.

Canfield, C. I., Fischhoff, B., & Davis, A. (2016). Quantifying phishing susceptibility for detection and behavior decisions. Human factors, 58(8), 1158-1172.

CBS news (2016, October 28). The phishing email that hacked the account of John Podesta. Retrieved from: https://www.cbsnews.com/news/the-phishing-email-that-hacked-the-account-of-john-podesta/

Chang, S. E., Chen, S.-Y., & Chen, C.-Y. (2011). Exploring the relationships between it capabilities and information security management. International Journal of Technology Management, 54(2/3), 147-166.

Conway, D., Taib, R., Harris, M., Yu, K., Berkovsky, S., & Chen, F. (2017). A qualitative investigation of bank employee experiences of information security and phishing. In Thirteenth Symposium on Usable Privacy and Security ({SOUPS} 2017) (pp. 115-129).

Culnan, M. J., Foxman, E. R., & Ray, A. W. (2008). Why IT executives should help employees secure their home computers. MIS Quarterly Executive, 7(1), 49-56.

Cybersecurity Insiders. (2018). Insider threat 2018 report. Presentation sponsored by CA technologies in conjunction with Crowd Research Partners.

Da Veiga, A. (2016). Comparing the information security culture of employees who had read the information security policy and those who had not: Illustrated through an empirical study. Information & Computer Security, 24(2), 139-151.

Das, S., Kim, A., Tingle, Z., & Nippert-Eng, C. (2019). All About Phishing: Exploring User Research through a Systematic Literature Review. arXiv preprint arXiv:1908.05897.

Davis, FD. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart, 13(3), 319-40.

Dhamija, R., Tygar, J. D., & Hearst, M. (2006, April). Why phishing works. In Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 581-590). ACM.

Diaz, A., Sherman, A. T., & Joshi, A. (2019). Phishing in an academic community: A study of user susceptibility and behavior. Cryptologia, 1-15.

Downs, J. S., Holbrook, M., & Cranor, L. F. (2007, October). Behavioral response to phishing risk. In Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit (pp. 37-44). ACM.

Egelman, S., Harbach, M., & Peer, E. (2016, May). Behavior ever follows intention? A validation of the security behavior intentions scale (SeBIS). In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 5257-5261). ACM.

Ferrari, R. (2015). Writing narrative style literature reviews. Medical Writing, 24(4), 230-235.

Ferreira, A. M., & Marques, P. M. V. (2018). Phishing Through Time: A Ten-Year story based on abstracts. In ICISSP (pp. 225-232).

Flores, W. R., Holm, H., Svensson, G., & Ericsson, G. (2014). Using phishing experiments and scenario-based surveys to understand security behaviours in practice. Information and Computer Security, 22(4), 393.

Frauenstein, E. D., & von Solms, R. (2009). Phishing: How an organisation can protect itself. In Proceedings of the ISSA 2009 Conference (p. 253). ISSA.

Frauenstein, E. D., & von Solms, R. (2014). Combatting phishing: A holistic human approach. 2014 Information Security for South Africa, 1-10. IEEE.

Furnell, S., R. Esmael, W. Yang, and Ninghui Li. (2018) Enhancing security behaviour by supporting the user. Computers & Security 75, 1-9.

Greene, K., Steves, M., & Theofanos, M. (2018). No Phishing beyond This Point. Computer, 51(6), 86-89.

Greene, K. K., Steves, M. P., Theofanos, M. F., & Kostick, J. (2018, February). User Context: An Explanatory Variable in Phishing Susceptibility. In Proc. 2018 Workshop Usable Security (USEC 18).

Halevi, T., Lewis, J., & Memon, N. (2013, May). A pilot study of cyber security and privacy related behavior and personality traits. In Proceedings of the 22nd International Conference on World Wide Web (pp. 737-744). ACM.

Halevi, T., Memon, N., & Nov, O. (2015). Spear-phishing in the wild: A real-world study of personality, phishing self-efficacy and vulnerability to spear-phishing attacks. Phishing Self-Efficacy and Vulnerability to Spear-Phishing Attacks (January 2, 2015). http://dx.doi.org/10.2139/ssrn.2544742

Harrison, B., Svetieva, E., & Vishwanath, A. (2016). Individual processing of phishing emails: How attention and elaboration protect against phishing. Online Information Review, 40(2), 265-281.

Herath, T., & Rao, H. R. (2009a). Encouraging information security behaviors in organizations: Role of penalties, pressures and perceived effectiveness. Decision Support Systems, 47(2), 154-165.

Herath, T., & Rao, H. R. (2009b). Protection motivation and deterrence: a framework for security policy compliance in organisations. European Journal of Information Systems, 18(2), 106-125.

Hong, J. (2012). The state of phishing attacks.  Communications of the ACM,  55(1), 74-81.

Hong, K. W., Kelley, C. M., Tembe, R., Murphy-Hill, E., & Mayhorn, C. B. (2013, September). Keeping up with the Joneses: Assessing phishing susceptibility in an email task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 57, No. 1, pp. 1012-1016). Sage CA: Los Angeles, CA: SAGE Publications.

Iuga, C., Nurse, J. R., & Erola, A. (2016). Baiting the hook: factors impacting susceptibility to phishing attacks. Human-centric Computing and Information Sciences, 6(1), 8.

ISO 27001 (2013) Information Technology – Security Techniques – Information Security Management Systems – Requirements, International Organization for Standardization, Geneva.

Jakobsson, M. (2007). The human factor in phishing. Privacy & Security of Consumer Information, 7(1), 1-19.

Jansen, J., & van Schaik, P. (2019). The design and evaluation of a theory-based intervention to promote security behaviour against phishing. International Journal of Human-Computer Studies, 123, 40-55.

Jensen, M. L., Dinger, M., Wright, R. T., & Thatcher, J. B. (2017). Training to mitigate phishing attacks using mindfulness techniques. Journal of Management Information Systems, 34(2), 597-626.

Kaur, J., & Mustafa, N. (2013, November). Examining the effects of knowledge, attitude and behaviour on information security awareness: A case on SME. In 2013 International Conference on Research and Innovation in Information Systems (ICRIIS) (pp. 286-290). IEEE

Khan, B., Alghathbar, K. S., Nabi, S. I., & Khan, M. K. (2011). Effectiveness of information security awareness methods based on psychological theories. African Journal of Business Management, 5(26), 10862-10868.

Khonji, M., Iraqi, Y., & Jones, A. (2013). Phishing detection: a literature survey. IEEE Communications Surveys & Tutorials, 15(4), 2091-2121.

Kim, B., Lee, D. Y., & Kim, B. (2019). Deterrent effects of punishment and training on insider security threats: a field experiment on phishing attacks. Behaviour & Information Technology, 1-20.

KnowBe4. (2018). Best practices for implementing security awareness training. Retrieved from: https://info.knowbe4.com/whitepaper-osterman-implementing-security-awareness-training.

Komatsu, A., Takagi, D. and Takemura, T. (2013). Human aspects of information security: An empirical study of intentional versus actual behavior. Information Management & Computer Security, 21(1), 5-15.

Kruger, H. A., & Kearney, W. D. (2006). A prototype for assessing information security awareness. Computers & security, 25(4), 289-296.

Kumaraguru, P., Rhee, Y., Sheng, S., Hasan, S., Acquisti, A., Cranor, L. F., & Hong, J. (2007, October). Getting users to pay attention to anti-phishing education: evaluation of retention and transfer. In Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit (pp. 70-81). ACM.

Kumaraguru, P., Cranshaw, J., Acquisti, A., Cranor, L., Hong, J., Blair, M. A., & Pham, T. (2009, July). School of phish: a real-world evaluation of anti-phishing training. In Proceedings of the 5th Symposium on Usable Privacy and Security (p. 3). ACM.

Kumaraguru, P., Sheng, S., Acquisti, A., Cranor, L. F., & Hong, J. (2010). Teaching Johnny not to fall for phish. ACM Transactions on Internet Technology (TOIT), 10(2), 7.

Lastdrager, E. E. (2014). Achieving a consensual definition of phishing based on a systematic review of the literature. Crime Science, 3(1), 1-10.

Markus, L., Thomas, H. C., & Allpress, K. (2005). Confounded by competencies? An evaluation of the evolution and use of competency models. New Zealand journal of psychology, 34(2), 117.

Miranda, M. J. (2018). Enhancing Cybersecurity Awareness Training: A Comprehensive Phishing Exercise Approach. International Management Review, 14(2), 5-10.

Mohebzada, J. G., El Zarka, A., Bhojani, A. H., & Darwish, A. (2012, March). Phishing in a university community: Two large scale phishing experiments. In 2012 International Conference on Innovations in Information Technology (IIT) (pp. 249-254). IEEE.

Montoya, L., Junger, M., & Hartel, P. (2017). Spear phishing in organisations explained. Information and Computer Security, 25(5), 593–613.

Ng, B., Kankanhalli, A., Xu, Y. (2009). Studying users’ computer security behavior: A health belief perspective. Decision Support Systems 46(4), 815–825

Nguyen, K. D., Rosoff, H., & John, R. S. (2017). Valuing information security from a phishing attack. Journal of Cybersecurity, 3(3), 159-171.

Nuha, N., & Molok, N. N. A. (2011). Disclosure of organizational information by employees on Facebook: Looking at the potential for information security risks. In Proceedings of the Australasian Conference on Information Systems (ACIS), Sydney, Australia.

Parsons, K., Butavicius, M., Delfabbro, P., & Lillie, M. (2019). Predicting susceptibility to social influence in phishing emails. International Journal of Human-Computer Studies, 128, 17-26.

Parsons, K., McCormac, A., Butavicius, M., Pattinson, M., & Jerram, C. (2014). Determining employee awareness using the human aspects of information security questionnaire (HAIS-Q). Computers & security, 42, 165-176.

Pattinson, M., Jerram, C., Parsons, K., McCormac, A. and Butavicius, M. (2012). Why do some people manage phishing e-mails better than others? Information Management & Computer Security, 20(1), 18-28.

Purkait, S. (2012a). Phishing counter measures and their effectiveness–literature review. Information Management & Computer Security, 20(5), 382-420.

Purkait, S. (2012b). Exploring the factors that influence an internet user’s ability to correctly identify phishing websites. IUP Journal of Information Technology, 8(3), 7–38.

Rajivan, P., Moriano, P., Kelley, T., & Camp, L. J. (2017). Factors in an end user security expertise instrument. Information & Computer Security, 25(2), 190-205.

Safa, N. S., Von Solms, R., & Furnell, S. (2016). Information security policy compliance model in organizations. Computers & security, 56, 70-82.

Sarno, D. M., Lewis, J. E., Bohil, C. J., & Neider, M. B. (2019). Which Phish Is on the Hook? Phishing Vulnerability for Older Versus Younger Adults. Human factors, https://doi.org/10.1177/0018720819855570

Sheng, S., Holbrook, M., Kumaraguru, P., Cranor, L. F., & Downs, J. (2010, April). Who falls for phish?: a demographic analysis of phishing susceptibility and effectiveness of interventions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 373-382). ACM.

Sherif, E., Furnell, S., & Clarke, N. (2015b, December). Awareness, behavior and culture: The ABC in cultivating security compliance. In 2015 10th International Conference for Internet Technology and Secured Transactions (ICITST) (pp. 90-94). IEEE.

Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual review of psychology, 70, 747-770.

Siponen, M., Pahnila, S., & Mahmood, A. (2007). Employees’ Adherence to Information Security Policies: An Empirical Study. New Approaches for Security, Privacy and Trust in Complex Environments, 133-144.

Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104, 333-339. doi.org/10.1016/j.jbusres.2019.07.039

Spitzberg, B. H. (2006). Preliminary development of a model and measure of computer-mediated communication (CMC) competence. Journal of Computer-Mediated Communication, 11(2), 629-666.

Steves, M. P., Greene, K., & Theofanos, M. F. (2019, February). A Phish Scale: Rating Human Phishing Message Detection Difficulty. In Workshop on Usable Security (USEC) 2019; February 24, 2019; San Diego, California, United States.

Teodorescu, T. (2006). Competence versus competency: What is the difference? Performance improvement, 45(10), 27-30.

Thomas, J. (2018). Individual cyber security: Empowering employees to resist spear phishing to prevent identity theft and ransomware attacks, International Journal of Business Management, 12(3), 1-23.

Vayansky, I, & Kumar, S. (2018). Phishing–challenges and solutions. Computer Fraud & Security, 1 (2018), 15-20.

Verizon (2018). Data Breach Investigations Report. Verizon Business Ready

Vishwanath, A., Herath, T., Chen, R., Wang, J., & Rao, H. R. (2011). Why do people get phished? Testing individual differences in phishing vulnerability within an integrated, information processing model. Decision Support Systems, 51(3), 576-586.

Von Solms, B., & Von Solms, R. (2004). The 10 deadly sins of information security management. Computers & Security, 23(5), 371-376.

Vroom, C., & Von Solms, R. (2004). Towards information security behavioural compliance. Computers & security, 23(3), 191-198.

Wang, J., Herath, T., Rui Chen, Vishwanath, A., & Rao, H. (2012). Research article phishing susceptibility: An investigation into the processing of a targeted spear phishing email. IEEE Transactions on Professional Communication, 55(4), 345–362. https://doi.org/10.1109/TPC.2012.2208392

Wang, P. A. (2013, June). Assessment of cybersecurity knowledge and behavior: an anti-phishing scenario. In International Conference on Internet Monitoring and Protection (ICIMP) (pp. 1-7).

Williams, E. J., Hinds, J., & Joinson, A. N. (2018). Exploring susceptibility to phishing in the workplace. International Journal of Human-Computer Studies, 120, 1-13.

Workman, M. (2008). Wisecrackers: A theory‐grounded investigation of phishing and pretext social engineering threats to information security. Journal of the American Society for Information Science and Technology, 59(4), 662-674.