COMPUTER SECURITY

These types of metrics are often created using online surveys. The major challenge in this approach is devising methods of identifying organizational structure and individuals within them that can be held accountable for survey answers. The state of the art in survey metrics collection utilities are systems wherein the accountable person can be linked to the control they are reporting on in such a way that when they log in to the risk application, their log- in is treated as a digital signature. This, combined with strict management policy that ques- tionnaires must be answered within strict timelines, allow the risk managers to report the results with firmwide authority. Figure 6A-5 provides an example of the type of online surveys an accountable manager may encounter.

From such a survey, metrics such as Figure 6A-6 may be derived. Risk management profes- sionals reduce accountability to self-reported task completion. From Figure 6A-6, it may be inferred that all of the new applications introduced into operation in the first quarter are using the firm’s single sign-on system for authentication, and are compliant with security pol- icy, design, and change control standards. However, there were a few introduced in the sec- ond quarter that did not make use of single sign-on and were not policy or standards compli- ant. The pie charts in Figure 6A-7 show this line of business (LOB) in comparison with the

Application detail record for SALESAPP

Application Acronym

SALESAPP CIO Owner: Doe, John L. (6043291)

Application Full Name Salesperson Data Entry Application

User Community Line of Business - Services (Internal)

Information Confidential (but not NPI) customer Classification

Information Class Definitions Log-in method Single Sign On Login Method Definitions

Is application design security policy compliant? Yes No

Is application implementation security policy compliant? Is application change control process security policy compliant?

Yes

No Yes No

Does the application encrypt PCIS data? Outsource Type IT Manager Business Owner

Status/DR Status

Figure 6A-5 Application Survey Courtesy Course Technology/Cengage Learning

Yes No Customized Third Party Application run in house

Jones, Janice Smith, Susan PROD

( 1638199 )

( 6400809 ) Always Hot, Redundant Across Alternate Sites

User Community Line of Business - Services (Internal)

Information Confidential (but not NPI) customer Classification

Information Class Definitions Log-in method Single Sign On Login Method Definitions

Is application design security policy compliant? Yes No

Is application implementation security policy compliant? Is application change control process security policy compliant?

Yes

No Yes No

Does the application encrypt PCIS data? Outsource Type IT Manager Business Owner

Status/DR Status

Figure 6A-5 Application Survey Courtesy Course Technology/Cengage Learning

Yes No Customized Third Party Application run in house

Jones, Janice Smith, Susan PROD

( 1638199 )

( 6400809 ) Always Hot, Redundant Across Alternate Sites

6A

50 40 30 20 10

0

1st QTR

2nd QTR 3rd QTR

4th QTR

LOB4

Percentage of applications that are not compliant

T o t a l

A p p s

A p p s

o n

S S O

P o l i c y

O K

D e s i g n

O K

T o t a l

A p p s

P o l i c y

O K

T o t a l

A p p s

A p p s

o n

S S O

A p p s

o n

S S O

P o l i c y

O K

D e s i g n

O K

C h n g

C n t r l

O K

T o t a l

A p p s

A p p s

o n

S S O

P o l i c y

O K

D e s i g n

O K

C h n g

C n t r l

O K

D e s i g n

O K

C h n g

C n t r l

O K

C h n g

C n t r l

O K

Figure 6A-6 Application Metrics Courtesy Course Technology/Cengage Learning

LOB1

Percentage of new applications that pass compliance surveys

LOB2 LOB3

Percentage of new applications that achieve compliance within one year

Figure 6A-7 Line of Business Application Metrics Courtesy Course Technology/Cengage Learning

other LOBs in the firm. By comparison, LOB1 seems to have superior security deployment processes.

However, given that the data is based, not on automated measures, independent audit, or repeatable processes, but instead on data collection from accountable managers, all that can be inferred from Figure 6A-7 is that LOB1 claims to have fully compliant applications. The risk manager running the report is reduced to clerk status. These scenarios exist because the management objective on which the metric is modeled does not easily conform to available security measurement techniques. Suppose instead, there were automated verification

(Whitman 223)

Whitman, Michael E. Readings & Cases in Information Security: Law & Ethics. Cengage Learning, 20100623. VitalBook file.

procedures that could check to see if an application was in compliance, and there was a com- plete and accurate inventory of machines deployed in support of a given application. The inventory, in combination with automated verification procedures and/or technical configura- tion audits, could be used to establish that applications met firmwide security objectives. However, pressure for results in short timeframes sometimes require even the most technical of metrics professionals to resort to a question-and-answer method of data gathering.

These scenarios almost always suffer from real or perceived ambiguity in the way the “com- pliance” officer phrases overall goals and statements to be affirmed. For example, suppose an executive of a given organization is asked to affirm whether or not that organization has a policy to achieve compliance with a given regulation. The executive may sincerely respond in the affirmative, yet be thinking of a policy that states that the organization is fully commit- ted to apply to all applicable regulation. At the same time, the executive may omit to mention that he or she does not believe that the regulation in question applies to that organization. Even when specific technology questions seem easy to verify, a respondent may find a ques- tion ambiguous enough for a misleading answer. For example, note the question in Figure 6A-5 that reads: “Does the application encrypt PCIS9 data?” A manager who encrypts some but not all such data may honestly and misleadingly answer “yes.” These examples and others like them leave a loophole of plausible deniability among those accountable for answering surveys while allowing the compliance officer to publish remediation metrics showing 100% compliance.

Case 2: The Risk Management Program

Risk assessment metrics with respect to security focus almost entirely on some adaptation of the risk management model prevalent in today’s security literature:10

1. Assets 2. Threats to those assets 3. Vulnerabilities that may be exploited to enact threats 4. Probabilities that the exploits occur 5. Cost of controls that will reduce the likelihood the vulnerabilities can be exploited

Through a sound step-by-step approach, it is rare that an InfoSec professional will identify assets in any manner that would make sense from a business perspective. Performed cor- rectly, the risk assessment would identify assets as products or services. However, there is overwhelming evidence in the security literature that assets are commonly construed to be computers themselves.11 This makes threat identification easy, as a generic set of people who abuse computers can be easily identified: disgruntled employees, hackers, terrorists, and natu- ral disasters. Vulnerabilities are also readily available as published sets of vulnerabilities from vendors as well as floods, fire, and hurricanes. Controls appear in this model as commercial off-the-shelf security products and services that will reduce the likelihood of threat.

So it is not too surprising that controls appearing as commercial off-the-shelf products and services are measured in similar ways by the InfoSec and risk management community. Automated tools can easily determine whether a given computer has a secure configuration. Figure 6A-8 demonstrates a metric commonly used in presentations of both InfoSec metrics and risk management metrics. It shows that there are a set of 27 computers in an organization all running the same operating system. There is an assumption that a secure

(Whitman 224)

Whitman, Michael E. Readings & Cases in Information Security: Law & Ethics. Cengage Learning, 20100623. VitalBook file.

30 25 20 15 10

5

0 7/1/08

Security Configurations for OS/400

6A

Risk manager see no dramatic changes

7/8/08

7/15/08

7/22/08

Security

manager sees systemic issue

7/29/08

Compliant Identified

Figure 6A-8 Operating System Security Metric Courtesy Course Technology/Cengage Learning

configuration of the operating system has been devised and is deployed on most of the com- puters in the set. However, it also shows that there are one or two machines that continually fall out of the secure configuration. To a risk manager, the lower line seems steady. The per- centage of computers that are not secure seems low. The security configuration for the oper- ating system as a whole seems stable. However, to the security manager, the same graph shows a target metric failed. The lower line indicates a weak link in the security operations support chain that could create a vulnerability to the entire platform. When this type of graph is presented to executives or auditors, it is almost always in the context of risk management.

Figure 6A-9 also demonstrates a metric commonly used in presentations of both InfoSec metrics and risk management metrics. It shows that some security control measure is in place for some type of platform that is deployed in different lines of business (LOB) in the organization. In both security management and risk metrics, it is intended to demonstrate that one business unit is more secure than another, to foster a sense of competition among business units. In the context of risk management, however, these metrics are instead used to explain different levels of risk and different approaches to the use of security controls as a risk reduction measure taken by different lines of business. Rather than walk away thinking that some LOB manager should be disciplined, executives are encouraged to compare the cost of security measures to the income produced by each line of business, and consider the probability that one of the known vulnerabilities indicated by the diagram may be exploited. A security manager seeing the same diagram sees a real and present danger.

It is an often glossed-over fact that the data presented in metrics like those in Figure 6A-8 reflect decisions based on judgments made by risk managers. An LOB may use Figure 6A-9 to determine that many of their noncompliance issues have to do with a single configuration parameter, that their machines are configured to “trust” others on their network. They may do a risk assessment on this configuration and decide that trust does not present a risk to

(Whitman 225)

their environment. Where a security manager has been told that a LOB need not fix a vulner- ability because their environment does not meet risk criteria necessary to fund a security fix, the group may mark that set of computers as the “equivalent” of having a compliant config- uration. This may be recorded in a database that is consulted in the “automated” report that produces Figure 6A-8. To be clear on the potential impact of this risk management approach, the “Configured with Trust” line in Figure 6A-9 shows a number of computers that are not security policy compliant, but have been declared to be safe anyway due to a “risk accep- tance” performed by line of business management. In Figure 6A-9, the 27 computers in Figure 6A-8 are represented as less than half policy compliant but half are not compliant due to a configuration that allows machines to trust each other. If a risk manager in the LOB finds the risk of trusting other computers acceptable in the course of their own risk assessment, the machines may be marked compliant. However, an auditor may only see Figure 6A-8 and not be told that different LOBs have different standards.

Moreover, even when configuration reports are produced with no exceptions, these risk man- agement scenarios almost always concentrate on reported results of generic security software deployment packages. They rarely check to see if assets are protected through appropriate applications controls, or question whether management has allowed information to be dis- seminated too broadly rather than restricted by job function.

Case 3: The Sphere of Control

Organizations of this type have a centralized security function for most technology, while allowing some business units to independently manage business-unit-specific security mea- sures. These distributed security functions usually have varying degrees of compliance with centrally published policies and procedures. However, when regulators and auditors come through, they are introduced to a person who is responsible only for security functions that

(Whitman 226)

overlap all business units, a central security officer, who proudly displays their metrics

program while not mentioning that it does not cover 100% of the organization. 6A

In contrast to the LOB-specific measures in Figure 6A-9, the central security function will gloss over technical differentiations in the data-gathering function. For example, measure- ments with respect to Windows operating systems are different than measurements concern- ing Linux. So normalizing these distinct measurements into industry-standard agreed-upon criteria for security measures seems like a commonsense exercise.

For example, Figure 6A-10 shows a set of security functions broken down by measureable features of operating systems that reflect them in a given environment. It does not necessarily matter that some operating systems implement security features better than others or that some instances of a given operating system are beyond the sphere of control of the central authority who generates the reports. It just demonstrates, whether, in the judgment of the central organization, these features are well implemented on the machines in scope. Such reports are meant to convey that the organization at least knows how to secure and measure things and that there should be no technology that requires security that would be beyond organizational expertise to deliver.

Central security organizations often feel justified in displaying this type of fuzzy target metric because they want to display true information but have been conditioned to allow the inde- pendent business units to evade their sphere of control. Where this is the case, they feel no responsibility to comment on the fact that their own management has allowed the indepen- dence. The lack of authority over LOBs could be perceived to indicate a lack of faith on the part of that management with respect to the security organization’s ability to handle the spe- cial needs of the suborganization in the centralized program. The suborganization is acknowledged to be a stepchild under special care, like a child that is acknowledged to require special education whose efforts should not be averaged into the rest of the classroom because it would bring down the ratings for the school as a whole.

(Whitman 227)

Whitman, Michael E. Readings & Cases in Information Security: Law & Ethics. Cengage Learning, 20100623. VitalBook file.

Conclusions

Let us suppose we are an InfoSec manager who needs to collect metrics to present to the board of directors (BOD). We are constrained on both time and level of detail. Suppose fur- ther we are new to the organization and not technical. We have no way to judge whether anything is secure or not for ourselves. In such a case, defining accountability and conducting surveys may be the only way to get the job done. The fact that the information is of dubious integrity is easily overlooked as the BOD meeting is short and is not expected to include tech- nical details of how information was collected.

Even without the supposition that we are technical enough to evaluate information security risk ourselves, we are still constrained on both time and level of detail. Even when we know exactly where the problems are in the organization, it does not mean we have the solutions, especially when our sphere of control does not extend to LOB deployment programs. In this case, a security manager may feel that the program progress metrics are best served by pre- senting metrics as evidence of program evolution, that is, as charts that are clearly climbing uphill to meet targets. These normally leave the execs beaming about continuous improve- ments rather than upset about not meeting state-of-the-art, well-known, 10-year-old best practices.

A continuous improvement report where we do not meet targets is actually less preferable to present to the board than a presentation in which we are not at all expected to report on pro- grams we do not control. Where we cannot even collect data from LOBs outside our control, we can simply omit mention of them and provide metrics on the deployment measures we fully control and endorse.

The metrics described in these cases will not tell the BOD whether the organization’s informa- tion is secure. You know your information is secure when you have a security program and corresponding systems architecture. The way you know your program is working is via activ- ity and monitoring metrics. The way you know your architecture is in place is via target metrics. When you find out you have a gap that leaves information exposed, you need reme- diation metrics. Metrics that correspond to a holistic security program demonstrate that man- agement objectives for the program are (or are not) being met (or to what degree the gap). Without the explanatory guidance provided by the security program and information architec- ture, metrics represent collections of disconnected attributes, none of which are security. Every aspect of the security program need not be continuously measured to provide adequate assur- ance that the program is working. However, enough control points must be capable of being monitored often enough so that the combined measures of control points directly map onto the organization’s requirements for information security. This mapping must be designed in such a way that there is capacity for demonstration that objectives are met.

Too often, target and remediation metrics absent program methodology and information architecture are mistaken for evidence of security program management. Executives on the receiving end of metrics often have a false sense of security. This type of blindspot, if it exists, does not extend to the rest of the organization nor even to its entire management team. Just as the product support team at Enron knew there was no broadband service offering,12 the InfoSec managers and other technical managers know fully

(Whitman 228)

Whitman, Michael E. Readings & Cases in Information Security: Law & Ethics. Cengage Learning, 20100623. VitalBook file.

The citation provided is a guideline. Please check each citation for accuracy before use.