Quizz

Chapter 3

The Art and Science of Security Metrics

PRAGMATIC security metrics

Chapter 1

Introduction Chapter 13

Conclusion Chapter 2 Why measure?

Chapter 12

Case study Chapter 3

Art and science Chapter 11

Using metrics Chapter 4 Audiences

Chapter 10

Appendices

Types

References

SMD

29

p

Downsides Chapter 5 Finding metrics

Chapter 9

Advanced metrics Chapter 6

PRAGMATIC Chapter 8

Measurement

system

Chapter 7 Example metrics

IT metrics are like art. No one can seem to agree on what constitutes a good metric, but everyone seems to know one when they see it.

Ann All

Security metrics is an evolving field of study, involving a combination of purely scientific and not-so-purely scientific approaches as the academics and practitioners feed off each other. While we appreciate the value of the scientific and mathemati- cal principles, theories, and models that underpin metrics and measurements, our particular contribution in writing this book lies far more on the practical side of

the fence. We study metrics not for the sake of science, but because they can help

30  ◾  PRAGMATIC Security Metrics

us resolve real-world situations that we face in information security management. Call it applied science if you will, state of the art, perhaps.

In recent years, a number of organizations and individuals have expressed their views and made suggestions on how information security can or should be mea- sured. In this chapter, we consider their advice, comparing and contrasting their approaches with the approach that we favor. If you are serious about security met- rics, we encourage you to check out the cited references for yourself (if you haven’t already) and draw your own conclusions. Although we are highlighting certain key sources specifically in this chapter, we encourage you to look at the bibliography toward the end of the book for further reading. You may not have the interest to delve too deeply into the field right now, but perhaps after finishing this book and starting to apply the techniques we suggest, you will feel the need for additional background and loftier expositions on security metrics.

3.1 Metrology, the Science of Measurement

Metrology, derived from the Greek word , is the science of measuring and metron quantifying things. “Metricians” are the practitioners of metrology. In this book, we are primarily concerned with one relatively narrow and specific form of applied metrology: the practical application of theoretical measurement science in the real world of information security. We also have an interest in the application of met- rics to the much broader fields of business management, governance, and risk management, although mostly in the areas where they intersect with information security.

y We don’t intend to go into detail, but, briefly, here are a few of the important

factors in metrology:

◾ Precision concerns the limit of details that can be measured and distinguished. ◾ Accuracy includes aspects such as repeatability. ◾ Integrity is a concern for the measurement data and the systems and processes

of measurement.

◾ Utility is about measuring things that matter.

You will find distinct echoes of these considerations and more in the PRAGMATIC method.

3.2 Governance and Management Metrics

Metrics are primarily a . Good metrics provide decision support tool for managementuseful, relevant information to help people—mostly, but not exclusively, manag- ers—make decisions based on a combination of historical events (the context),

The Art and Science of Security Metrics  ◾  31

what’s going on right now (including available resources and constraints), and what is anticipated to occur in the future (the change imperative).

Management metrics and measurement practices in general are continually evolving; for instance, the Balanced Scorecard (Kaplan and Norton 1996) was con- sidered state of the art when it was released well over a decade ago and still remains influential today. It has been adopted and adapted by many organizations and today finds application in a wider variety of ways and situations than its original authors probably conceived.* In contrast, executive information systems that became preva- lent a few years back when everything was going online have proven counterproduc- tive in some organizations by focusing management attention on the pretty graphics while, in some cases, hiding or distracting them from important details.

e executive dashboard concept remains intriguing and pops out of the wood- work again every few years. e latest incarnation we’ve noticed is security infor- mation and event management (SIEM). Most dashboards are promoted to the market as metrics systems by pushy software vendors who have found it easy to catch the eye of budget holders with their bright colors and fancy graphics. Most of them report metrics from automated security tools, such as firewalls, antivirus, and intrusion detection/prevention systems because certain technical statistics are readily obtained, but as to whether they are of any practical value for executive managers seems doubtful, in particular, because management needs information beyond the purely technical.

Information security managers find it difficult to justify security initiatives and investments objectively to upper management. is is, in large part, because they can’t provide metrics to support their cases because they don’t incorporate feedback processes indicating the extent of protection and the effectiveness of security con-

p g p y trols implemented previously. is is a systematic issue. Coupled with the inability to quantify information security risks and the effectiveness of proposed controls persuasively, it generally results in a lack of adequate or misdirected investment in security and hence significant underprotection of information resources.

Anything that causes an observable change can be measured by observing the change. e main issue in relation to security metrics is to determine the sort and nature of measurements that provide useful and meaningful information upon which to effectively base decisions—decisions about whether greater control is needed, whether controls are failing and need remediation, or whether existing controls are sufficient or, in fact, excessive.

* In the same way, we hope the PRAGMATIC approach and the concept of metametrics will be widely adopted. We don’t consider ourselves the sole guardians or bastions of Truth. We encourage you to consider and build on the ideas we are presenting, finding creative ways to

use them that make sense in your specific context. We would you to share your experiences loveand developments with others. Most of all, we want to give our fellow practitioners a handy tool to make your lives a little easier albeit one of many in your toolboxes.

32  ◾  PRAGMATIC Security Metrics

Metrics can also be used to influence and possibly bring change to a culture. For example, developing good and useful metrics around security and then pub- lishing the results, say, monthly to management may yield a number of beneficial results. For one thing, it will raise the profile of information security. For another, it will make it abundantly clear which way things are headed and can generate a great deal of pressure for improvement. It might serve to expose the security laggards and to reinforce those activities that yield good results.

e point is that information security metrics should serve to inform, facili- tate, and guide proper and appropriate decisions to achieve security objectives. at

translates into measuring the right things and reporting them to in the right people the right format at the right time:

◾ e right things are those that mean something to the recipients, causing them to make appropriate decisions.

◾ e right people are those that make or at least influence the decisions (see Chapter 4).

◾ e right format is one that effectively communicates—it gets the message across and motivates the recipients to act accordingly (see Chapter 11 for more on data analysis and presentation).

◾ e right time is, of course, before the final decision is made, and in practice, it usually includes the lead-up and preparation of analyses, reports, presenta- tions, and arguments that will influence the outcome.

3 3 Information Security Metrics

3.3 Information Security Metrics

ere are many different aspects to managing information security risks and, hence, many different elements that could be measured.

Compared to fields such as financial and operations management, metrics in information security are relatively immature. Few organizations have a coherent

suite or of information security metrics in place, and hardly any would claim system that their security metrics are comprehensive. A surprising number of organiza- tions still don’t use any at all! Others use a bewildering array of security metrics with little if any consensus on what generally ought to be measured—a core set of essential information security metrics, if you will. Worse still, while several sources

list things that could be measured, nobody really agrees on to go about chooshow - ing or developing metrics that are appropriate for a given organization: there is very limited guidance on the process of selecting or developing appropriate metrics and, in some cases, dubious pseudo-scientific advice about the methods of collecting, analyzing, and presenting data.Context is important: many of the information security metrics that are appro-

priate for your particular organization may differ from those needed by others, even your peers in the same industry. e process for determining the metrics your

The Art and Science of Security Metrics  ◾  33

organization actually needs (discussed in detail in Chapter 8) can be summarized by thinking through who needs to know what, when. In other words, measurements are made to provide information supporting managers and operational people in making decisions. Measurement information has to be relevant and meaningful to the recipient. Anything else is just noise.

We have no knowledge of your specific situation, so our advice has to be generic, but where possible, we will provide realistic examples to help you apply the tools and techniques to select suitable metrics.

3.4 Financial Metrics (for Information Security)

During the past decade or more, a variety of approaches have been developed in the effort to improve security metrics. Along the way, financially based security metrics spun away from the mainstream on something of a tangent of their own.

In Managing Cybersecurity Resources: A Cost–Benefit Analysis, two well-respected professors of economics and accounting* address the rhetorical question, “How can you know if your firm is committing too much money, or not enough, to protect itself against such unseen hazards?” (Gordon and Loeb 2006). e answer, apparently, involves a conventional cost–benefit analysis using standard accounting methods, such as net present value (NPV) or internal rate of return (IRR), to appraise investment options. In Gordon and Loeb’s highly rational world, managers justify and select cer- tain cybersecurity projects purely on the basis of their net value (benefits less costs).

While the cost part of the equation is straightforward, the benefits of security relatively require some creative thinking using techniques such as ALE and SLE:

q g g q

◾ Annualized loss expectancy (ALE) is simply the anticipated average annual loss from security failures, projected in line with the losses that have accu- mulated in the preceding years. Simply stated, if you have suffered historical losses totaling, say, $28,000 over a 10‐year period, and you anticipate the same level of loss to occur over the next 10, then the ALE is $2,800. ere are major assumptions inherent in ALE.† For a start, it is assumed that someone has been accurately identifying and diligently recording the costs of secu- rity incidents: if not, the ALE is pure guesswork. Second, it is assumed that you have enough knowledge and understanding of the situation (meaning the risks—the threats, vulnerabilities, exposures, and impacts) to predict the

* We suspect professors Gordon and Loeb might be a little embarrassed at being billed as “global leaders in the critical area of cybersecurity” in the book’s marketing blurb. It is pretty obvious from the way they use the term “cybersecurity” and refer to the Internet as if it had only just been invented, that they have nontechnical/non-IT backgrounds. We decline to say whether

that represents a benefit or a cost. † ese are not a million miles away from the theoretical physicist’s “First, assume a spherical

cow in a vacuum…”