Synthesis Paper (cyber security)

D E C E M B E R 2 0 1 5 | V O L . 5 8 | N O . 1 2 | C O M M U N I C AT I O N S O F T H E A C M 43

V viewpoints

There is the real danger that if even one nation acquires these weapons, others may feel they have to follow suit in or- der to defend themselves and to avoid falling behind in a robotic arms race.

The open letter signed by some of the most renowned AI experts stated, “If any major military power pushes ahead with AI weapon development, a global arms race is virtually inevitable … We therefore believe that a military AI arms race would not be beneficial for humanity. There are many ways in which AI can make battlefields safer for humans, especially civilians, with- out creating new tools for killing peo- ple … [M]ost AI researchers have no in- terest in building AI weapons—and do not want others to tarnish their field by doing so, potentially creating a major public backlash against AI that curtails its future societal benefits.”b

There is also the prospect that fully autonomous weapons could be ac- quired by repressive regimes or non- state armed groups with little regard for the law. These weapons could be perfect tools of repression and terror for autocrats.

Another type of proliferation con- cern is that such weapons would in- crease the likelihood of armed attacks, making resort to war more likely, as de- cision makers would not have the same concerns about loss of soldiers’ lives. This could have an overall destabiliz- ing effect on international security.

b http://bit.ly/1UQI7QE

DOI:10.1145/2835963 Point: Stephen Goose

D I P L O M A T S A N D M I L I TA R Y ex- perts from more than 90 countries gathered in Ge- neva in April 2015 for their second meeting on “lethal

autonomous weapons systems,” also known as fully autonomous weapons, or more colloquially, killer robots. Noted artificial intelligence expert Stu- art Russell informed delegates the AI community is beginning to recognize the specter of autonomous weapons is damaging to its reputation and indi- cated several professional associations were moving toward votes to take a po- sition on the topic.

On July 28, 2015, more than 1,000 AI professionals, roboticists, and others re- leased an open letter promoting a “ban on offensive autonomous weapons be- yond meaningful human control.”a

Thus, it is timely and appropriate that Communications is focusing on fully autonomous weapons, particular- ly the call for a preemptive prohibition on their development, production, and use as has been made by the Campaign to Stop Killer Robots, many in the AI community, more than 20 Nobel Peace Laureates, and many others.

There is no doubt that AI and greater autonomy can have military and hu-

a The Future of Life Institute organized the letter. The text and list of signatories can be found at: http://bit.ly/1UQI7QE

manitarian benefits. Most of those call- ing for a ban support AI and robotics research. But, weaponizing fully auton- omous robotic systems is seen by many as a step too far.

The Many Reasons to Ban Fully Autonomous Weapons The central concern is with weapons that once activated, would be able to select and engage targets without further human involvement. There would no longer be a human operator deciding whom to fire at or when to shoot. Instead, the weapon system it- self would undertake those tasks. This would constitute not just a new type of armament, but a new method of war- fare that would radically change how wars are fought, and not to the better- ment of humankind.

Many in the AI community have focused on the serious international security and proliferation concerns related to fully autonomous weapons.

Point/Counterpoint The Case for Banning Killer Robots Ban the bots? Considering both sides of the argument for and against.

Stephen Goose/Ronald Arkin

Weaponizing fully autonomous robotics systems is seen by many as a step too far.

44 C O M M U N I C AT I O N S O F T H E A C M | D E C E M B E R 2 0 1 5 | V O L . 5 8 | N O . 1 2

viewpoints

Rapid and Growing Support for a Ban Over the past two years, the question of what to do about fully autonomous weapons has rocketed to the top ranks of concern in the field of disar- mament and arms control, or what is now often called humanitarian disar- mament. Within this short period of time, world leaders including the Sec- retary-General of the United Nations, the U.N. disarmament chief, and the head of the International Committee of the Red Cross, have expressed deep concerns about the development of fully autonomous weapons and urged immediate action.

The Campaign to Stop Killer Ro- bots—an international coalition of non- governmental organizations (NGOs)— was launched in April 2013 calling for a preemptive ban on fully autonomous weapon systems.f Coordinated by Hu- man Rights Watch, it is modeled on the successful civil society campaigns that led to international bans on antiper- sonnel landmines, cluster munitions, and blinding lasers. A month after the campaign launched, the U.N. special rapporteur on extrajudicial killings pre- sented the Human Rights Council with a report that echoed many of the cam- paign’s concerns and called on govern- ments to adopt national moratoria on the weapons.

In October 2013, more than 270 prominent engineers, computing and artificial intelligence experts, roboti- cists, and professionals from related disciplines issued a statement call- ing for a ban. This was organized by the International Committee for Ro- bot Arms Control (ICRAC), which was founded in 2009 by roboticists, ethi- cists, and others.g As noted earlier, in July 2015 more than 1,000 AI experts signed an open letter supporting a prohibition; the letter was organized by the Future of Life Institute.

During 2014, the European Parlia- ment passed a resolution that calls for a ban, more than 20 Nobel Peace Laureates issued a joint statement in favor of a ban, and more than 70 prominent faith leaders from around the world released a statement call- ing for a ban. A Canadian robotics

f http://www.stopkillerrobots.org/ g http://icrac.net/call/

be especially difficult as this relies heavily on situational and contextual factors, which could change consider- ably with a slight alteration of the facts.

There are also serious concerns about the lack of accountability when fully autonomous weapons fail to com- ply with IHL in any particular engage- ment. Holding a human responsible for the actions of a robot that is acting autonomously could prove difficult, be it the operator, superior officer, pro- grammer, or manufacturer.

Scientists and military leaders have also raised a host of technical and op- erational issues with these weapons that could pose grave dangers to civil- ians—and to soldiers—in the future.d A particular concern for many is how “robot vs. robot” warfare would unfold, and how devices controlled by complex algorithms would interact.e

Taken together, this multitude of concerns has led to the call for a pre- emptive prohibition on fully autono- mous weapon systems—a new inter- national treaty that would ban the development, production, and use of fully autonomous weapons, and require there is always meaningful human con- trol over targeting and kill decisions.

d See, for example, “Autonomy in Weapons Sys- tems,” U.S. Department of Defense, Directive Number 3000.09, Nov. 21, 2012. This direc- tive, which guides U.S. policy on autonomous weapons, cites a multitude of technical issues that would have to be overcome before fielding fully autonomous weapons.

e An October 2013 joint statement calling for a ban on fully autonomous weapons signed by more than 270 computing experts said, “Such interactions could create unstable and unpredictable behavior, behavior that could initiate or escalate conflicts, or cause unjus- tifiable harm to civilian populations.” See http://icrac.net/call/

For many people, these weapons would cross a fundamental moral and ethical line by ceding life and death deci- sions on the battlefield to machines. Giv- ing such responsibilities to machines in such circumstances has been called the ultimate attack on human dignity.c The notion of allowing compassionless robots to make decisions about the ap- plication of violent force is repugnant to many. Compassion is a key check on the killing of other human beings.

In my extensive engagement with a variety of audiences on this issue, it has been striking how most people have a visceral negative reaction to the notion of fully autonomous weapons. The Martens Clause, which is articu- lated in the Geneva Conventions and elsewhere, is a key provision in interna- tional law that takes into account this notion of general repugnance on the part of the public. Under the Martens Clause, fully autonomous weapons should comply with the “principles of humanity” and the “dictates of pub- lic conscience.” They likely would not comply with either.

There are serious questions about whether fully autonomous weapons would ever be capable of complying with core principles of international humanitarian law (IHL) during com- bat, or international human rights law (IHRL) during law enforcement operations, border control, or other circumstances. There is of course no way of predicting what technology might produce many years from now, but there are strong reasons to be skeptical about compliance with in- ternational law in the future, includ- ing the basic principles of distinction and proportionality.

Could robots replicate the innately human qualities of judgment and in- tuition necessary to comply with IHL, including judgment of an individual’s intentions, as well as subjective deter- minations? Compliance with the rule of proportionality prohibiting attacks in which expected civilian harm out- weighs anticipated military gain would

c This point has been made repeatedly by the U.N. Special Rapporteur Christof Heyns. See, for example, U.N. Human Rights Council, Report of the Special Rapporteur on extra- judicial, summary or arbitrary executions, Christof Heyns, Lethal Autonomous Robotics, A/HRC/23/47, Apr. 9, 2013.

Many advanced militaries are rapidly marching toward ever greater autonomy in their weapons systems.

D E C E M B E R 2 0 1 5 | V O L . 5 8 | N O . 1 2 | C O M M U N I C AT I O N S O F T H E A C M 45

viewpoints

company, Clearpath, became the first commercial entity to support a ban and declare it will not work toward the development of fully autonomous weapons systems.

Governments have become seized with the issue of fully autonomous weapons since 2013, though few have articulated formal policy positions. Most importantly, the 120 States Par- ties to the Convention on Convention- al Weapons (CCW) agreed in Novem- ber 2013 to take up the issue, holding meetings in May 2014 and then again in April 2015. In the diplomatic world, the decision to take on killer robots was made at lightning speed.

It appears certain that nations will agree to continue their CCW delib- erations next year, though questions remain about the nature, content, and duration of the work. A small but growing number of states have already called for a preemptive ban, while most participating states have expressed interest in discussing the concept of meaningful human control of weap- ons systems, indicating they see a need to draw a line before weapons become fully autonomous.

The April 2015 CCW experts meet- ing was by far the richest, most in- depth discussion held to date on auton- omous weapons. Not a single state said it is actively pursuing them yet the week featured extensive discussion about the potential benefits of such weapons.

The U.S. and Israel were the only states to explicitly say they were keep- ing the door open to the acquisition of fully autonomous weapons but there are plenty of indicators there are many states that are contemplating them. Without question, many advanced mil- itaries are rapidly marching toward ev- er-greater autonomy in their weapons systems, and there are no stop signs in their path.

Why a Ban Is the Best Solution Some oppose a preemptive and com- prehensive prohibition, saying it is too early and we should “wait and see” where the technology takes us. Oth- ers believe restrictions would be more appropriate than a ban, limiting their use to specific situations and missions. Some say existing international hu- manitarian law will be sufficient to ad- dress the challenges posed.

The point of a preemptive treaty is to prevent future harm and with all the dangers and concerns associated with fully autonomous weapons, it would be irresponsible to take a “wait and see” approach and only try to deal with the issue after the harm has al- ready occurred. Once developed, they will be irreversible; it will not be possi- ble to put the genie back in the bottle as the weapons spread rapidly around the world.

The notion of a preemptive treaty has been done before. The best ex- ample is the 1995 CCW protocol that bans blinding laser weapons. After initial opposition from the U.S. and others, states came to agree the weap- ons would pose unacceptable dangers to soldiers and civilians. The weapons were seen as counter to the dictates of public conscience and nations came to recognize their militaries would be better off if no one had the weap- ons than if everyone had them. These same rationales apply to fully autono- mous weapons.

While some rightly point out that there is no “proof” there cannot be a technological fix to the problems of fully autonomous weapons, it is equal- ly true there is no proof there can be. Given the scientific uncertainty that exists, and given the potential benefits of a new legally binding instrument, the precautionary principle in inter- national law is directly applicable. The principle suggests the international community need not wait for scientific certainty, but could and should take action now.

Fully autonomous weapons repre- sent a new category of weapons that could change the way wars are fought

and pose serious risks to civilians. As such, they demand new, specific law that clarifies and strengthens existing international humanitarian law.

A specific treaty banning a weapon is also the best way to stigmatize the weapon. Experience has shown that stigmatization has a powerful ef- fect even on those who have not yet formally joined the treaty, inducing them to comply with the key provi- sions, lest they risk international con- demnation. A regulatory approach restricting use to certain locations or to specific purposes would be prone to longer-term failure as countries would likely be tempted to use them in other, possibly inappropriate, ways during the heat of battle or in dire circumstances. Once legitimized, the weapons would no doubt be mass produced and proliferate worldwide; only a preemptive international treaty will prevent that.

The call for a ban on development of fully autonomous weapons is not aimed at impeding broader research into military robotics or weapons au- tonomy or full autonomy in the civil- ian sphere. It is not intended to curtail basic AI research in any way. Research and development activities should be banned if they are directed at technol- ogy that can only be used for fully au- tonomous weapons or that is explicitly intended for use in such weapons.

Conclusion While there are at this stage still many doubters, my experience leads me to conclude a preemptive ban is not only warranted, but is achievable and is the only possible approach that would suc- cessfully address the potential dangers of fully autonomous weapons. Howev- er, the involvement, advice, and exper- tise of the AI community are needed both to get to a ban and to ensure it is the most effective ban possible.

The AI community has an important role to play in bringing about the ban on fully autonomous weapons. This is not a political issue to be avoided in the name of pure science, but rather an issue of humanity for which we are all responsible.

Stephen Goose ([email protected]) is the director of the Human Rights Watch Arms Division.

Copyright held by author.

It will not be possible to put the genie back in the bottle as the weapons spread rapidly around the world.

Copyright of Communications of the ACM is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.