AI military weapons should be ban

D E C E M B E R 2 0 1 6 | V O L . 5 9 | N O . 1 2 | C O M M U N I C AT I O N S O F T H E A C M 27

news I

M A

G E

B Y

D I

G I

T A

L S

T O

R M

Society | DOI:10.1145/3005678 Keith Kirkpatrick

Can We Trust Autonomous Weapons? Nations consider using defense systems that can make their own lethal decisions.

control for target verification, though developers have given it the capability to use its sensors to detect, select, and shoot at targets autonomously.

It is the last capability that has watchdogs worried. Systems such as LRASM and the SGR-A1 are now only approved for use with a human approv- ing targets to be killed by the system, but there is considerable concern the U.S. and other world powers are on the fast track to developing machines able

M O S T R E A S O N A B L E P E O P L E can see the benefits of us- ing fully autonomous sys- tems, particularly to help prevent injuries or death,

as is the case with advanced driver as- sistance systems increasingly found in automobiles. When it comes to au- tonomous systems that are designed to take life rather than preserve it, there is significantly more debate.

Currently, the U.S. and other na- tions do not have any weapons systems that can operate fully autonomously, which is defined in military parlance as selecting, aiming, and firing at a target without a human being “in the loop,” or somehow in control of the weapon system. However, a variety of military weapons systems operate semiautono- mously, requiring some human con- trol or input to select or choose targets, but relying on pre-programmed algo- rithms to execute a strike.

A good example of this is the Lock- heed Martin Long Range Anti-Ship Missile (LRASM) system, slated to en- ter service in the U.S. defense system within the next two years. The LRASM can be fired from a ship or plane and autonomously travel through the air, avoiding obstacles outside of the tar- get area. Published reports indicate humans choose and program the algorithms to seek out and identify potential targets, thus keeping a hu- man in the loop. While the exact fac- tors that make up the target selection algorithm are classified, it is likely a weighting of elements such as the tar- get’s size, location, radar signature, heat profile, or other elements that positively identify the target.

Another example of a system with semiautonomous capabilities is Sam- sung’s SGR-A1, a military border sentry robot in development for deployment on the border between North and South

Korea. Essentially an unarmed guard tower, the system is designed to assist border guards by scanning the area for those who might try to cross the bor- der. The system is armed with a light machine gun that can dispense tear gas or rubber bullets, and is equipped with cameras, a laser range finder, and a pattern recognition algorithm designed to discern between people and animals. Currently, the system is designed to be operated under human

28 C O M M U N I C AT I O N S O F T H E A C M | D E C E M B E R 2 0 1 6 | V O L . 5 9 | N O . 1 2

news

to kill people independently. “I think it’s pretty clear that military

mastery in the 21st century is going to depend heavily on the skillful blending of humans and intelligent machines,” says John Arquilla, professor and chair of Department of Defense Analysis at the U.S. Naval Postgraduate School in Monterey, CA. “It is no surprise that many advanced militaries are investing substantially in this field.”

Indeed, in late 2014, U.S. Secretary of Defense Ash Carter unveiled the country’s so-called “Third Offset” strat- egy, essentially an attempt to offset the shrinking U.S. military force by incor- porating technologies that improve the efficiency and effectiveness of weapons systems. While many of the specific aspects of the strategy are classified, industry observers agree a key tenet is increasing the level of autonomy in weapons systems, which will improve warfighting capability and reduce the number of humans required to operate weapons systems.

“For over a decade, we were the big guy on the block,” explains Major Gen- eral (Ret.) Robert H. Latiff, an adjunct professor at the Reilly Center for Sci- ence, Technology, and Values at the University of Notre Dame, and Values Research Professor and an adjunct pro- fessor at George Mason University. “But with the emergence of China, and the reemergence of Russia, both of whom are very, very technically capable, and both of whom have invested fairly sig- nificantly in these same technologies, it goes without saying that the DoD (the U.S. Department of Defense) feels like they need to do this just to keep up.”

Military guidelines published by the U.S. Department of Defense in 2012 do not completely prohibit the develop- ment and use of autonomous weap- ons, but require Pentagon officials to oversee their use. That is why human rights groups such as the Campaign to Stop Killer Robots are actively lob- bying the international community to impose a ban on the development and use of autonomous weapons systems.

“We see a lot of investment happen- ing in weapons systems with various levels of autonomy in them, and that was the whole reason why we decided at Human Rights Watch back in 2012 to look at this,” explains Mary Ware- ham, advocacy director of the Arms

Division of the Human Rights Watch, and coordinator of the Campaign to Stop Killer Robots. “We’re seeking a preemptive ban on the development, production, and use of fully autono- mous weapons systems in the United States and around the world.”

“We’re focusing quite narrowly on the point at which critical functions of the weapons system become autonomous,” Wareham says. “The critical functions that matter to us are the selection and identification of the target, and the use of force.” Wareham’s main issue is that the fully autonomous weapons systems of the future may rely solely on algorithms to target and kill enemy targets, without a human in the loop to verify the system has made the right decision.

Human Rights Watch is pushing to have a negotiated international treaty limiting, restricting, or prohibiting the use of autonomous weapons systems written and ratified within the next two or three years. “If this becomes [a decade-long process], then we’re in big trouble,” she admits, noting that at present, there are no such treaties in process within the international com- munity. “At the moment, it is just talk,” Wareham acknowledges.

A key argument of groups such as Human Rights Watch is that these sys- tems, driven by algorithms, may make mistakes in target identification, or may not be able to be recalled once de- ployed, even if the scenario changes. Others with military experience point out that focusing on the potential for mistakes when using fully autono- mous weapons systems ignores the re-

alities of warfighting. “I think one of the problems in the

discourse is the objection that a ro- bot might accidentally kill the wrong person, or strike the wrong target,” Arquilla says. “The way to address is this is to point out that in a war, there will always be accidents where the in- nocents are killed. This has been true for millennia, it is true now, and it is true with all of the humans killed in the Medecins Sans Frontieres hospital in Afghanistan.”

Arquilla adds that while the use of artificial intelligence in weapons will not eliminate mistakes, “Autono- mous weapons systems will make few- er mistakes. They don’t get tired, they don’t get angry and look for payback, they don’t suffer from the motivated and cognitive psychological biases that often lead to error in complex military environments.”

Furthermore, military experts feel an outright ban would be impossible to enforce due to the secretive nature of most militaries, and likely would not be in the best interest of any military group or nation.

Indeed, even with today’s military technologies, getting the military or its contractors to discuss the exact al- gorithms used to acquire, select, and discharge a weapon is difficult, as dis- closing this information would put them at a distinct tactical disadvan- tage. Therefore, even if a ban were to be put in place, devising an inspections system similar to those used for chemi- cal and anti-personnel mines would be extremely complicated.

“A ban is typically only as good as the people who abide by it,” Latiff says, noting that those who will sign and fully abide by a ban make up “a pretty small fraction of the rest of the world.” In practice, he says, “When something becomes illegal, everything just goes underground. It’s almost a counter- productive thing.”

Work on autonomous weapons system has been going on for years, and experts insist expecting militaries to stop developing new weapons sys- tems that might provide an advantage is foolhardy and unrealistic. As such, “There is absolutely an arms race in autonomous systems underway,” Ar- quilla says. “We see this in both coun- tries that are American allies, and also

“We’re seeking a preemptive ban on the development, production, and use of fully autonomous weapons systems in the United States and around the world.”

D E C E M B E R 2 0 1 6 | V O L . 5 9 | N O . 1 2 | C O M M U N I C AT I O N S O F T H E A C M 29

news

the use of autonomous weapons, and whether they will be developed in ways that are consistent with accepted ideas about ethics and war, Saxon says.

“There are good arguments about increased moral hazard with autono- mous weapons systems, that they make killing too easy,” Saxon says. “I think they also have an effect on traditional military virtues that need to be examined. What does it mean to be courageous, for instance, when your machines take the risks and do the killing?”

For Latiff’s part, while he does not support a ban on autonomous weap- ons, he would support a non-prolifer- ation treaty allowing militaries to re- search and test these systems to ensure they can be made as reliable and safe as possible.

“At the end of the day, it’s kind of like nuclear weapons,” Latiff says. “Ev- erybody’s going to get them, and the people that don’t get them are going to want them. The best we can hope for is that we slow it down.”

Further Reading U.S. Department of Defense 2012 Directive on Autonomous Weapons: http://www.dtic. mil/whs/directives/corres/pdf/300009p.pdf

Campaign to Stop Killer Robots: https://www.stopkillerrobots.org/

Video: Scary Future Military Weapons Of War-Full Documentary: https://www.youtube.com/ watch?v=DDJHYEdKCBE

Keith Kirkpatrick is principal of 4K Research & Consulting, LLC, based in Lynbrook, NY.

© 2016 ACM 0001-0782/16/12 $15.00

among potential adversaries. In par- ticular, the Russians have made great progress. So have the British; they are putting together a fighter plane that can do everything a piloted fighter plane can, and can be built to higher performance characteristics, because you don’t have a human squeezed by G-forces in the cockpit.”

Others agree, even if they admit that at present, there are no significant ad- vantages to using fully autonomous weapons, versus the semiautonomous systems already in use.

“I would imagine that [as autono- mous weapons] become more capa- ble, they will be seen to operate more effectively than systems with humans in the loop,” says Lieutenant Colonel Michael Saxon, an assistant professor teaching philosophy at the U.S. Mili- tary Academy at West Point. “Once you introduce swarms or you have to respond to adversary systems that are autonomous, humans in the loop will create real disadvantages. This, of course, is all predicated on advances in these machines’ capabilities.”

Still, observers suggest while a ban on autonomous weapons may not be the right course of action, a deliberate approach to developing and incorpo- rating them into the military arsenal is prudent.

“I think we’re probably closer to the kind of capabilities we’re talking about than most people think, and Russia and China are, too,” Latiff says. “These are dangerous systems. In the wrong hands, these things could really be awful.”

A ban on autonomous weapons would likely have little impact on the development of weapons systems in the near future, which still will be over- seen by humans, even if the actual de- cision to select and strike a target is made by an autonomous system.

“An autonomous weapon operates without human control, but that does not mean that it is free from human in- put,” Arquilla says. “There are a lot of elements to the chain [of command]; the unfortunate term is the ‘kill chain.’ There will be people and machines in- termixed within the chain.”

Whether or not a ban is put into place, the international community is likely to be faced with significant mor- al and legal questions surrounding

“Once you introduce swarms or you have to respond to adversary systems that are autonomous, humans in the loop will create real disadvantages.”

Milestones

Ken Kennedy Award Goes To Gropp ACM and IEEE Computer Society named William D. Gropp, Thomas M. Siebel Chair of Computer Science at the University of Illinois at Urbana-Champaign, as recipient of the 2016 ACM/ IEEE Computer Society Ken Kennedy Award for his highly influential contributions to the programmability of high performance parallel and distributed computers.

Gropp was a leader in the development of the Message Passing Interface (MPI) standard, the de facto standard for programming distributed- memory computers in scientific applications. and a key author in the development of MPI-I, MPI-2, and MPI-3. Gropp also designed and developed MPICH, the first functional implementation of MPI.

Along with collaborators David Keyes and Xiao-Chuan Cai, Gropp developed and analyzed key scalable parallel algorithms for adaptive mesh refinement and domain decomposition methods, which are now widely used in parallel applications. This led to the development of the Portable, Extensible Toolkit for Scientific Computation (PETSc), which Gropp developed in collaboration with Barry Smith. PETSc has been used in a variety of applications, including nano-simulations, biology, fusion, geosciences, environmental modeling, fluid dynamics, and software engineering, among others.

In 2011, Gropp helped launch the ACM Special Interest Group on High Performance Computing (SIGHPC). He also served as the editor of CONNECT, the newsletter for SIGHPC; as chair of the Gordon Bell Prize Committee, and in various capacities for numerous conference committees.

Gropp, who also is Acting Director and Chief Scientist of the National Center for Supercomputing Applications, has been elected an ACM Fellow and an IEEE Fellow, and is a member of the U.S. National Academy of Engineering.

Copyright of Communications of the ACM is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.