Quizz

Topics Start Learning Search 50,000+ courses, events, titles, … What's New

272  ◾  PRAGMATIC Security Metrics

of the extent to which the system is exposed to the threats of most concern, and the importance or criticality of the system to the business (which is a proxy for the impact component).*

In much the same vein, indicators are telltales, meaning observable character- istics or features that are associated with, rather than part of, the thing of interest. In relation to information security risk, again, one indicator is the number and gravity/severity of incidents and near misses that have happened. Absent other clear warning signs, a marked increase in the number of reported industrial espionage incidents strongly indicates an increase in the risk of industrial espionage. e link is not absolute,† but it takes a brave (or foolhardy!) manager to argue the converse and ignore the obvious warning signs.

Corporate security culture is another example. It is technically possible to mea- sure corporate security culture in a scientific manner using psychological/anthropo- logical research methods, but it is generally impracticable to do so (not least because of the difficulty of defining it precisely enough to study). Proxies include various, more readily observed, behaviors, such as the extent of employee compliance with policies, etc. and employee attitudes toward security as revealed by surveys. It is pretty obvious that a resentful, demoralized, indifferent staff is less likely to take a personal interest in carefully managing risk and compliance rules than one that is not. Absenteeism, rising levels of infractions, the general nature and tone of emails (a field of study called sentiment analysis), collective torpor, water-cooler mutter- ings, or the more contemporary version, griping on social media, are all potential indicators of the corporate security culture.

Indicators and proxies are useful because they correlate, to varying extents, with information security aspects of interest. We have more to say on correlation and other statistical methods in Chapter 11.

9.3 Key Indicators

9.3.1 Key Goal Indicators (KGIs)

We introduced the capability maturity model (CMM; Paulk et al. 1995) in Section 3.8 as one of several ways to define metrics requirements and described the defini- tion of information security goals supporting business outcomes. KGIs are metrics that relate to those goals.

For most organizations with critical or sensitive information assets, particularly highly regulated organizations in the financial services and telecoms sectors, CMM level 4 represents an adequate level of protection, though one not easy to attain. e

* While still relatively crude, we much prefer approaches of this nature over more theoretical objective risk analysis methods that appear to us somewhat divorced from reality, given the distinct lack of factual data for their projections.

† Perhaps the number of incidents is the same as ever, but more are being reported.

Advanced Information Security Metrics  ◾  273

version of CMM in the CISM Review Manual (ISACA 2012) states the following 15 requirements to achieve level 4:*

1. e assessment of risk is a standard procedure, and exceptions to following the procedure would be noticed by information security management.

2. Information security risk management is a defined management function with senior-level responsibility.

3. Senior management and information security management have determined the levels of risk that the organization will tolerate and have standard mea- sures for risk/return ratios.

4. Responsibilities for information security are clearly assigned, managed, and enforced.

5. Information security risk and impact analysis is consistently performed. 6. Security policies and practices are completed with specific security baselines. 7. Security awareness briefings are mandatory. 8. User identification, authentication, and authorization are standardized. 9. Certification of security staff is established. 10. Intrusion testing is a standard and formalized process leading to improvements. 11. Cost–benefit analyses, supporting the implementation of security measures,

are increasingly being utilized. 12. Information security processes are coordinated with the overall organization

security function. 13. Information security reporting is linked to business objectives. 14. Responsibilities and standards for continuous service are enforced. 15. System redundancy practices, including use of high-availability components,

are consistently deployed.

Each of the 15 statements can provide the basis for a KGI and, for that matter, one or more policy statements. KGIs may also need to be developed for intermedi- ate steps of multiyear projects that might coincide with budget cycles, for example.

9.3.2 Key Performance Indicators (KPIs)

KPIs are the measurable steps along the way to achieving a KGI, used to track prog- ress. e same CMM level used to set objectives can also serve as a KPI to deter- mine maturity progress toward the objective. Other performance indicators may be used to track budgets and project completion steps. KPIs are used to populate a project roadmap, and their sequence should be subject to critical path analysis to ensure various project steps don’t interfere with each other, for example, if one step

* ISACA’s straightforward narrative is comprehensible even by business types. In case the advan- tage isn’t blindingly obvious, try explaining a balanced business scorecard or a computational

model, such as VAR, to the uninitiated!

274  ◾  PRAGMATIC Security Metrics

in the roadmap requires the installation of a database, obviously the supporting infrastructure must be provided first. Conflicting resource requirements will also be highlighted by this process.

9.3.3 Key Risk Indicators (KRIs)

Careful post hoc analysis of serious information security incidents almost always shows a series of precursor events and changes that could or should have raised the red flag on the associated risks that eventually materialized. Some risk factors are glaringly obvious in hindsight, such as a dramatic rise in incidents in your industry sector or location (increasing threat), a stack of security patches waiting patiently on the side for their chance to be implemented (increasing vulnerability), or changes that led to business processes becoming critically reliant on certain information sources or systems (increasing impacts). Other risk factors may not be so clear, but it is unusual for major incidents to happen totally out of the blue.

Aside from investigating the causative factors in order to resolve them, post- incident reviews should also explore the question of why the warning signs (i.e., the KRIs) were missed at the time. What were the KRIs, and how come they either weren’t being monitored or failed to trigger appropriate responses that would have prevented or lessened the incident?*

Many information security metrics could be classed as risk indicators: the differ- ence with KRIs is simply a matter of degree; “key” implying “important” or “criti- cal” or “miss this and we’re sunk.” KRIs are therefore highly context- dependent, so the example KRIs below may not be applicable in your organization:

◾ Unusual numbers of change requests especially emergency changes

◾ Unusual numbers of change requests, especially emergency changes ◾ Substantial increases in employee turnover or absenteeism ◾ Substantial increases in policy infractions, noncompliance/exceptions, and

exemption requests

* If it appears that an incident was caused not by the lack of warning about the risk but by the failure of one or more security controls, that simply moves the focus of attention to the suit-

ability and effectiveness of the controls, indicating unmanaged risks in that specific area.

Tip: Seize the day! Incidents often highlight concerns over the risk analysis and risk monitoring/tracking processes that evidently missed the threats, vul- nerabilities, exposures, and impacts. In the aftermath of an incident, man- agers may argue over the details and dispute accountability, but that there

actually was a risk is an indisputable, unavoidable, iron-clad fact, whereas it appears not to have been noticed or addressed adequately beforehand.

Advanced Information Security Metrics  ◾  275

◾ Delays/inconsistencies in patching ◾ A series of highly critical audit reports

KRIs are about identifying and doing something to avert those accidents wait- ing to happen. Often the solution lies in an escalation path to senior management

for security issues that have been building for some while. Strategic-level Predictive and Relevant metrics are therefore likely to qualify as KRIs.

9.3.4 Critical Success Factors (CSFs)

ey may not have “key” in the name, but CSFs are conceptually related to KRIs, KPIs, and KGIs. CSFs are things deemed essential to achieve some goal, objec- tive, or outcome. Businesses, business units, departments/functions, projects, and activities can all have CSFs. Because they are critical, CSFs are self-evidently prime

did f

Tip: Despite the obvious advantages, organizations with immature approaches to information security seem curiously reluctant to undertake post-incident reviews. We suspect the ability to identify and, especially, to learn from one’s mistakes correlates strongly with an organization’s overall information secu- rity capability. Whether or not you agree with us, take a long cold look at Figure 7.2 and ponder the incident management metrics examples in Section 7.10. Are you doing enough to capture the lessons? Are your KRIs being actively tracked?

candidates for measurement. Information security CSFs obviously relate to information security depart-

ment/function, projects, and activities (particularly the information security goals shown in Figure 3.2) but also occur in related fields (such as risk, compliance, and audit) and in unrelated fields and business activities whenever they identify a criti- cal reliance on either information or information security (e.g., in relationships with third parties providing, processing, storing, and transporting vital information).

9.4 Targets, Hurdles, Yardsticks, Goals, Objectives, Benchmarks, and Triggers

As soon as something is measured, we have the possibility of comparing it ratio- nally against comparators, such as

◾ Previous measurements of the same metric (giving us historical trends and

self-improvement)

276  ◾  PRAGMATIC Security Metrics

◾ Projected or predicted measurements (giving assurance as to our predictive capability)

◾ Variously defined ideal values (see below) ◾ Other similar or related metrics (allowing us to cross-check or validate par-

ticularly important metrics)

Common factors are (1) having reference points against which to assess/com- pare the measurements and (2) the presumption, urge, desire, or demand for the subject of measurement to meet or exceed expectations.

e reference points and ideal values include the following:

◾ Targets, for example, reducing spam messages reaching employees’ inboxes below 1% of emails received

◾ Hurdles, for example, getting 80% of the security policies formally approved by management this year

◾ Yardsticks, for example, reaching CMM level 3 for our information security practices

◾ Goals and objectives, for example, supporting the safe, controlled introduction of an online sales platform

◾ Benchmarks, for example, being rated by a broad-based information security survey in the top quartile of organizations in our industry for all categories of concern

◾ Triggers, for example, if the rate of password resets exceeds 10% of help desk calls, review and repeat the associated security awareness activities

Notice that the comparisons, and thus the metrics, do not necessarily have to be

strictly numeric:* being certified compliant with ISO/IEC 27001 is a good example of a meaningful metric, a standard of achievement that is not itself a matter of earn- ing so many points from the certification assessors. ere is no simple pass mark for certification. It comes down to an accredited certification body’s informed opinion on whether the organization satisfies the mandatory requirements for an informa-

tion security management system that are explicitly stated in the standard, plus provides sufficient, credible evidence to prove the system both meets the organiza- tion’s information security objectives (as specified by management in the statement of applicability) and is operating correctly.†

* We beg to differ with those who insist that metrics generate numeric or number-and-unit outputs (e.g., 27 incidents or $3 million). Disregarding measurements that do not generate

numbers is a retrograde step, in our opinion. But what do you think? † Admittedly, satisfying or failing to satisfy the standard’s mandatory requirements is a binary

measure, but the other aspects are certainly analog. In practice, numbers really don’t come into

the equation: certification is a question of judgment, not mathematics.