Cyber Security

1 C O M M U N I C AT I O N S O F T H E A C M | J U N E 2 0 1 7 | V O L . 6 0 | N O . 6

V viewpoints

wrong, or may change over time—for example, as new types of threats are de- tected and exploited.

In addition to trustworthiness or untrustworthiness of people relevant to their interactions with computers in the above sense, trustworthiness and specifically personal integrity are also meaningful attributes of people and governments in their daily existence. In particular, truthfulness and honesty are typically thought of as trustworthi- ness attributes of people. The ques- tion of whether a particular computer system is honest would generally not be considered, because such a system has no moral compass to guide it. How- ever, truthfulness is another matter. A system might actually be considered dishonest or even untruthful if it con- sistently or even intermittently gives wrong answers just in certain cases— especially if it had been programmed explicitly to do exactly that. For exam- ple, such behavior has been associated with certain proprietary voting sys- tems—see Douglas W. Jones and Bar-

T R U S T W O R T H I N E S S I S A N at- tribute that is fundamental to our technologies and to our human relations. Overly trusting something that is

not trustworthy often leads to bad re- sults. Not trusting something that re- ally is trustworthy can also be harmful.

In many of the past 240 Inside Risks columns, we have been concerned ex- tensively with trustworthiness, which should be a basic requirement of all computer-related systems—particu- larly when used in mission-critical applications, but also in personal set- tings such as maintaining your own quality of life. Trustworthiness is abso- lutely essential to the proper behavior of computers and networks, and to the well-being of entire nations and indus- tries that rely on proper behavior of their computer-based enterprises.

Computer-Based Systems and People Trustworthy system behavior typically may depend on trustworthiness of

people—for example, system design- ers, hardware developers and program- mers, operational staff, and high-level managers. Many systems that might have some assessment of trustwor- thiness can nevertheless be seriously compromised by malicious malware, external adversaries, and insider mis- use, or otherwise disrupted by denial- of-service attacks. If such compromises arise unexpectedly, then those systems were most likely not so trustworthy as had been believed.

Thus, we need system designs and implementations that are tolerant of people who might usually be trustwor- thy but who make occasional errors, as well as systems that are resistant to and resilient following many other poten- tial adversities. More importantly, we need measures of assurance—which assess how trustworthy a system might actually be in certain circumstances (albeit typically evaluated only against perceived threats). Unfortunately, some the assumptions made prior to the evaluation process may have been

Inside Risks Trustworthiness and Truthfulness Are Essential Their absence can introduce huge risks …

DOI:10.1145/3084344 Peter G. Neumann

J U N E 2 0 1 7 | V O L . 6 0 | N O . 6 | C O M M U N I C AT I O N S O F T H E A C M 2

viewpoints

V with Ruin at the Virtual Casino,” The New York Times, Feb. 5, 2017).

! People who believe that elections based on Internet voting and propri- etary unauditable voting machines are inherently “fair” can be easily misled. People who continue to believe that Rus- sians had no influence on the November 2016 election in the U.S. or in the April preliminary elections in France are oblivious to real evidence in both cases. Furthermore, The Netherlands recently abandoned electronic voting systems, returning to paper ballots—wary of fur- ther ongoing Russian interference.

Risks of Believing in Human Truthfulness and Integrity Human creativity can have its down- sides. For example, opportunities for ransomware, cyberfraud, cybercrime, and even spam all seem to be not only increasing, but becoming much more sophisticated.

Social engineering is still a simple and effective way to break into other- wise secure facilities or computer sys- tems. It takes advantage of normal hu- man decency, helpfulness, politeness,

bara Simons, Broken Ballots, University of Chicago Press, 2012.

Systems can be untrustworthy be- cause of false assumptions by the programmers and designers. For ex- ample, sensors measure whatever they are designed to measure, which may not include the variables that should be of greatest concern. Thus, a system assessing the slipperiness of the road for a vehicle might rely upon a sensor that determines whether the road is wet. Sometimes that is done by check- ing whether the windshield wipers are on—which is a rather indirect mea- sure of slipperiness and can lead to false or imprecise recommendations or actions. At least one commercial aviation accident resulted from an in- direct and imprecise determination of runway slipperiness.

Risks of Believing in Computer Trustworthiness Many people believe computers are in- fallible and cannot lie. However, com- puters are created by people who are not infallible. Therefore, logically we might conclude that computers can-

not be infallible. Indeed, they cannot always perform exactly as expected, given the presence of hardware errors, power outages, malware, hacking at- tacks, and other adversities.

Indeed, computers can be made to lie, cheat, or steal. In such cases, of course, the faults may originate with or be amplified by people who commis- sion systems, or design them, or pro- gram them, or even just use them, but not with the computers themselves. However, even supposedly ‘neutral’ learning algorithms and statistics can be biased and untrustworthy if they are presented with a biased or untrust- worthy learning set. Unfortunately, the complexity of systems makes such be- havior difficult to detect. Worse, many statistical learning algorithms (for ex- ample, deep learning) and artificial in- telligence cannot specify how they ac- tually reached their decisions, making it difficult to assess their validity.

! People who believe that online gambling is “fair” are likely to be be easy victims. So can those who know it is not fair, but are nevertheless addict- ed (see Francis X. Clines, “Threatened CR

E D

I T

T K

3 C O M M U N I C AT I O N S O F T H E A C M | J U N E 2 0 1 7 | V O L . 6 0 | N O . 6

viewpoints

be confused with truth, even though that confusion appears remarkably common. People who believe every- thing they read on Facebook, Google, Amazon, Twitter, and other Internet sites are clearly delusional.

Conclusion People who are less aware of technol- ogy-related risks tend to overendow computers as perfect, while computers have little respect for people. Neither computer behavior nor human behav- ior is always perfect, and should not be expected to be so. There are significant risks in blindly believing in computer trustworthiness and human truthful- ness. We must not believe in computer infallibility, or in everything we read on the Internet in the absence of credible corroboration. But then we should also not believe people who pervasively dis- honor truthfulness.

Unfortunately, the trends for the fu- ture seem relatively bleak. Computer system trustworthiness and the impli- cations of its absence are increasingly being questioned. For example, a re- cent article by Bruce G. Blair (Hack- ing our Nuclear Weapons, The New York Times, Mar. 14, 2017) suggests “Loose security invites a cyberattack with possibly horrific consequences.” Semi- and fully autonomous systems, the seemingly imminent Internet of Things, and artificial intelligence are providing further examples in which increasing complexity leads to obscure and unexplainable system behavior. The concept of trustworthiness seems to becoming supplanted with people falsely placing their trust in systems and people that are simply not trust- worthy—without any strong cases be- ing made for safety, security, or indeed assurance that might otherwise be found in regulated critical industries such as aviation. However, the risks of false would-be “facts” may be the ulti- mate danger. An obvious consequence might be the extensive institutional loss of trust in what is neither trustwor- thy nor truthful. The truth and trust- worthiness may be even more impor- tant now than ever before.

Peter G. Neumann ([email protected]) moderates the ACM Risks Forum and is Senior Principal Scientist in SRI International’s Computer Science Lab. He is grateful to Donald Norman for considerable useful feedback.

Copyright held by author.

and altruism. A knee-jerk attempt to rein in social engineering could involve eliminating these very desirable social attributes (which might also eliminate civility and decency from our society).

How Does This All Fit Together? It should be fundamental to readers of Inside Risks articles that point solu- tions to local problems are generally in- sufficient, and that we have to consider trustworthiness in the total-system con- text that includes hardware, software, networking, people, environmental concerns, and more. On September 22, 1988, Bob Morris (then chief scientist of the National Computer Security Center at NSA) said in a session of the Nation- al Academies’ Computer Science and Telecommunications [now Technology] Board on problems relating to security, “To a first approximation, every com- puter in the world is connected with ev- ery other computer.” That quote is even more relevant today, almost 30 years lat- er. Similarly, all of the issues considered in this column involving computers and people may be intimately intertwined.

Science is never perfect or immu- table—it is often a work in progress. Hence, scientists can rarely if ever know they have the absolute final an- swer. However, scientific methods have evolved over time, and scientists gener- ally welcome challenges and disagree- ments that can ultimately be resolved through better theories, experimental evidence, and rational debate. Occa- sionally, we even find fake science and untrustworthy scientists, although these aberrations tend to be refuted eventually via peer pressure. Where science has strong credible evidence, it deserves to be respected—because in the final analysis reality should be able to trump fantasies (although this in fact may not work).

Truth is perhaps even more in flux than science, and certainly relative, not absolute—with many caveats. However, truth matters. We might paraphrase the oft-cited Albert Ein- stein quote as “Everything should be stated as simply as possible, but not simpler.” Oversimplifications, the lack of foresight, and a seriously non-ob- jective perspective are often sources of serious misunderstandings, and can result in major catastrophes. On the other hand, untruthfulness must not

AD TK