AI military weapons should be ban

FEATURES

The Growing International Movement Against Killer Robots

S ix weapons have been inter­ nationally banned: poison gas, biological weapons, chemical weapons, blinding

lasers, antipersonnel land mines, and cluster munitions. The bans on the latter three were all achieved in the past 21 years, and, in each case, civil society was the driving force in bringing the issue to the forefront and convincing governments to agree to a ban. All three bans were rooted in the desire to better protect civilians during warfare and its af­ termath, and to strengthen interna­ tional humanitarian law.

These efforts gave birth to what is now often called “humanitarian dis­ armament,” which puts top priority on the welfare of civilians through the creation of strong international standards. Humanitarian disarma­ ment has also been characterized by the prominent role of non-govern­ mental organizations (NGOs) and civil society more broadly, as well as partnership between different

stakeholders to achieve their goals. The humanitarian disarmament

concept dates back to the award­ ing of the 1997 Nobel Peace Prize to the International Campaign to Ban Land mines together with its then-coordinator for the unprece­ dented way in which they worked together with governments of small and medium-sized countries to es­ tablish the Mine Ban Treaty, which the Nobel Committee described as “a convincing example of an effec­ tive policy for peace.”

Three prominent hum anitar­ ian disarmament campaigns are currently underway. One, led by the International Network on Ex­ plosive Weapons (INEW), aims to alleviate the impact on civilians of the military practice that is caus­ ing the most civilian casualties and suffering today: the use of explosive weapons in populated areas. In par­ ticular, INEW is calling for a halt in the use of explosive weapons with wide area effects in densely popu-

S tep h en D. G oose Executive Director, Arms Division

Human Rights Watch

Mary W areh am Advocacy Director, Arms Division

Human Rights Watch

lated areas. A second, led by the Internation­

al Campaign to Abolish Nuclear Weapons (ICAN), is aimed at elim­ inating the existing weapons that pose the gravest risk to the survival of the planet. ICAN focuses on the humanitarian impact of nuclear weapons and calls for a new inter­ national treaty banning these kinds of weapons.

The third effort, led by the Cam­ paign to Stop Killer Robots, is aimed at tackling the number one future threat to civilians and the most dan­ gerous military development now underway: fully autonomous weap­ ons systems, or “killer robots.” The Campaign is calling for an interna-

28 SUMMER 2016 Photo Courtesy Associated Press

tional treaty preemptively banning the development, production, and use of such weapons. These future weapons could select and attack targets without any human input or interaction.

Autonomous weapons are be­ coming widely acknowledged as the next technological revolution in warfare following the invention of gunpowder and the development of nuclear weapons. In 2013, it took just six months after the launch of the Campaign to Stop Killer Robots for states to agree to begin talks on the weapons at the Convention on Conventional Weapons (CCW). Yet the process is now seen as aiming rather low and growing slowly. A movement of various constituen­ cies is converging on key concerns, but governments need to act to push the diplomatic process towards a constructive and binding outcome — a new CCW protocol banning the weapons.

1. W h a t are k ille r r o b o ts? At issue are weapons systems

that would be capable of selecting targets and using force without any human input or interaction. These are often referred to as “human- out-of-the-loop” weapons. Sever­ al terms are used to describe the weapons, but since 2013 countries at the CCW have settled on “lethal autonomous weapons systems.”

Armed drones currently in op­ eration over Afghanistan, Iraq, Ye­ men, and other countries depend on a person to make the final de­ cision whether to fire on a target. However, the autonomy of these and other weapons that have been deployed or are under development is growing quickly. Newer military drones, such as the MQ-9 Reaper, already can take off, land, and fly to designated points without human intervention.

Low-cost sensors and advances in artificial intelligence are making it increasingly practical to design weapons systems that would target and attack without human inter­

vention. For example, South Korea has developed automated station­ ary gun towers and placed them in the Demilitarized Zone with North Korea. According to one of its de­ velopers, the “original version had an auto-firing system but all of our customers asked for safeguards to be implemented. Technologically it wasn’t a problem for us but they were concerned the gun might make a mistake.” The Samsung SGR-i sentry robot is equipped with sen­ sors that detect movement and — in accordance with its current “on- the-loop” setting — send a signal to a command center where human soldiers determine if the individual identified poses a threat and decide whether to fire.

If the trend towards autonomy continues, the fear is that humans will start to fade out of the deci­ sion-making loop, first retaining only a limited oversight role, and then no role at all. Most specialists acknowledge that fully autonomous weapons do not currently exist, but the capacity to develop them is ex­ pected to be available within a mat­

ter of years rather than decades. The United States affirmed in No­ vember 2015 that “there is broad agreement that lethal autonomous weapon systems do not exist” and do not refer to “remotely piloted drones, nor precision-guided muni­ tions or defensive systems.”

According to the Internation­ al Committee of the Red Cross (ICRC), most existing weapons sys­ tems are overseen in real time by a human operator and tend to be highly constrained in the tasks they are used for and the types of targets

FEATURES they attack. The ICRC describes an autonomous weapon system as one that has autonomy in its “critical functions,” meaning a weapon that can select (i.e., search for or de­ tect, identify, track) and attack (i.e., intercept, use force against, neu­ tralize, damage or destroy) targets without human intervention. The ICRC believes future autonomous weapon systems could have more freedom of action to determine their targets, operate outside tightly con­ strained spatial and temporal lim­ its, and encounter rapidly changing circumstances. They could include systems operating over extended periods of time with no possibility of human intervention or supervi­ sion whatsoever.

Some specialists describe po­ tential autonomous weapons that could be capable of human-level cognitive tasks, at least for more narrow problems, as artificial intel­ ligence weapons. According to Paul Scharre, a senior fellow and direc­ tor of the 20YY Future of Warfare Initiative at the Center for a New American Security, these sophis­

ticated future autonomous sys­ tems would exhibit some degree of “learning, adaptation, or evolution­ ary behavior.” Automated weapons operating according to more com­ plex, rule-based systems are, ac­ cording to Scharre, autonomous if the systems “exhibit goal-oriented behavior.”

A system may be autonomous in the sense that is operates without human supervision, but its func­ tionality may be quite simple and constrained, leading some to refer to it as an “automatic” or “automat-

^Autonomous weapons are becoming widely acknowledged as the next technological revolution in warfare following the invention of gunpowder and the development of nuclear weapons. "

SUMMER 2016 29

FEATURES ed” weapon rather than an autono­ mous weapon. Automatic systems follow specific pre-programmed commands with little room for variation in a “structured environ­ ment,” while autonomous weapons would have more freedom to deter­ mine their own actions in an “open and unstructured” environment.

For example, several countries employ weapons defense systems that are programmed to respond automatically to threats from in­ coming munitions. The Nether­ lands describes its ship-based Goal­ keeper close-in weapons system and Patriot surface-to-air missiles as examples of automatic systems that can to a large extent oper­ ate automatically, noting: “The degree to which these systems are set to ‘automatic’ by their op­ erators depends on the security environment and the threat situ­ ation. The greater the threat and the shorter the response time, the more automatically these systems need to operate in order to be effective, though they are con­ tinuously monitored by their oper­ ators.”

Land mines are generally consid­ ered automatic, not autonomous, weapons because they are com­ prised of a simple, threshold-based system with easily predictable re­ flexive responses to external input.

Other automatic or automated land-based precursor systems in­ clude defensive artillery and auton­ omous ground systems capable of collaborating with other entities. The Iron Dome system used by the Israel Defense Force senses and intercepts incoming rockets and projectiles by using a mechanism that is programmed to respond au­ tomatically. Such automatic weap­ ons are intended for defensive use against material targets as opposed to personnel, and sometimes allow a human to override its function with a human-on-the-loop.

The New York Times has report­ ed on the development of Lockheed Martin’s Long Range Anti-Ship

Missile (LRASM), which the De­ partment of Defense describes as “semi-autonomous” under the defi­ nitions established by its first policy directive on autonomy in weapons systems issued in November 2012. Yet, as John Markoff notes for the New York Times, the missile is controversial because, “although a human operator will initially se­ lect a target, it is designed to fly for several hundred miles while out of contact with the controller and then automatically identify and attack an enemy ship.”

UA robot cannot possibly be

programmed to deal with every situation

it might encounter on the field ”

2. Can ex istin g law a d d ress the k iller rob ots challenge?

Military policy documents, es­ pecially from the United States, reflect clear plans to increase the autonomy of weapons systems. In its first report on fully autonomous weapons issued in November 2012, Human Rights Watch identified “precursors” to fully autonomous weapons systems under develop­ ment in at least six countries.

Drivers towards fully autono­ mous weapons include increasing speed and distance of air, ground, and naval-based systems, concerns about insecure communications, the requirement for fewer person­ nel amid rising costs, the ability of such systems to intervene in diffi­ cult to access areas, and the need to stay as ahead as possible from ad­ versaries in terms of technology.

Proponents tout a range of poten­ tial benefits, including that fully au­ tonomous weapons could decrease the need for soldiers on the battle­ field and thereby save military lives.

These weapons could have the abil­ ity to detect and attack targets with greater speed and precision than weapons directed by human beings, and fully autonomous weapons would not experience pain, hunger, exhaustion, or other human feel­ ings.

These characteristics could en­ tice militaries to deploy fully au­ tonomous weapons despite their humanitarian drawbacks. Saving soldiers’ lives is a laudable goal, but it must be balanced against the like­ lihood that the danger for civilian casualties would increase with the

use of these weapons and shift the burden of conflict onto civilians.

It is clear that existing interna­ tional humanitarian law and hu­ man rights law will apply to fully autonomous weapons systems, but there are serious concerns the weapons would not be able to ful­ ly comply with those complex and subjective rules, which require human understanding and judg­ ment as well as compassion. There is no certainty that fully

autonomous weapons would have the capacity to distinguish between combatants, who would be targeted, and non-combatants, who should not be. It would also be difficult to program fully autonomous weap­ ons to carry out the proportionality test required for prohibiting attacks in which expected civilian harm outweighs anticipated military ad­ vantage. It would be particularly difficult to program a machine to determine that balance in every situation because there are an in­ finite number of possible situations. There are serious doubts that the weapons could exercise comparable judgment to assess proportionality of attacks in complex and evolving situations on the ground.

A robot cannot possibly be pro­ grammed to deal with every situa­ tion it might encounter on the field. So if it unlawfully killed a civilian instead of a soldier, who would be responsible? According to Human Rights Watch, the programmers,

30 SUMMER 2016

manufacturers, and military com­ manders would all likely escape liability. Under criminal law, if a military commander used the fully autonomous weapons intentional­ ly to kill civilians, he or she would be held accountable. But it would be unfair and legally impossible to hold commanders accountable for situations when the robot acted in an unpredictable manner that they could not foresee or prevent.

The potential for an accountabili­ ty gap has serious consequences for the future usage of these weapons systems. Without accountability there is no way to deter future vio­ lations of international law, and no

retribution for victims of past viola­ tions.

Scientists find it unlikely that au­ tonomous weapons will meet legal requirements for the use of force due to an “absence of clear scien­ tific evidence that robot weapons have, or are likely to have in the foreseeable future, the functionality required for accurate target identifi­ cation, situational awareness or de­ cisions regarding the proportional use of force.”

Under Article 36 of the First

Additional Protocol to the Gene­ va Conventions, state parties are obliged to determine whether new weapons and new methods of war­ fare are compatible with interna­ tional law. A common understand­ ing appears to be emerging that sees the acquisition or deployment of autonomous weapons system as prohibited if the relevant require­ ments of international law cannot be met.

The Campaign to Stop Killer Ro­ bots and others have found that the profound and fundamental ques­ tions about autonomous weapons replacing the role of humans in the use of force and the taking of hu­

man life cannot be left to national weapons reviews alone. The ICRC warns that leaving it up to each state to determine the lawfulness and acceptability of the specific au­ tonomous weapon systems they are developing or acquiring may risk inconsistent outcomes with, for ex­ ample, some states applying limits to the use of such systems and oth­ ers prohibiting their use altogether.

Fully autonomous weapons would threaten rights and princi­ ples under international human

FEATURES rights law as well as international humanitarian law (IHL). Funda­ mental guarantees, such as the right to life and the right to a remedy — which continue to apply during armed conflict as well as peacetime — could be at risk because the au­ tonomous weapons cannot be pro­ grammed to handle every situation in due manner. Fully autonomous weapons would also undermine hu­ man dignity because as inanimate machines they cannot understand or respect the value of life proper­ ly, yet they would have the power to determine when to take it away.

Increasingly, the Martens Clause, a customary rule of internation­

al humanitarian law that has appeared in international treaties and national military manuals for more than a century, is be­ ing cited in the debate over what to do about autonomous weapons. It states, “In cases not covered by this Proto­ col or by other inter­ national agreements, civilians and combat­ ants remain under the protection and authority of the prin­ ciples of internation­ al law derived from established custom, from the principles of humanity and from the dictates of public conscience.”

Eventually, states may have to determine whether fully autono­ mous weapons meet the dictates of public conscience and, if they do, public views should be taken into account.

3. A ffirm ing th e p rin cip le o f m ea n in g fu l h u m a n co n tro l via a p reem p tiv e ban

This matter of whether autono­ mous weapons are ethically accept­ able lies at the core of the debate over them. The fundamental ques-

People protest against the use of drones outside of the Iowa Air National Guard base. Un­ manned drones are currently in use but still depend on human actions. New technological developments could lead to completely autonomous drones.

Photo Courtesy Associated Press SUMMER 2016 31

FEATURES tion is not technological, but phil­ osophical: “Will machines be given the power and authority to kill a hu­ man being?”

History has shown that for some weapons that significantly threaten non-combatants, including poison gas, biological weapons, chemical weapons, blinding lasers, antiper­ sonnel mines, and cluster muni­ tions, responsible governments have found it necessary to supple­ ment the limits already in the in­ ternational legal framework. Fully autonomous weapons have the po­ tential to raise a comparable level of humanitarian concern.

Based on current and foreseeable technology, the ICRC says there are serious doubts about the ability of autonomous weapon systems to comply with IHL “in all but the nar­ rowest of scenarios and the simplest of environments.” Because of this, the ICRC says “it seems evident that overall human control over the se­ lection of, and use of force against, will continue to be required.”

The Campaign to Stop Killer Robots fundamentally objects to autonomous weapons because it rejects the notion that a machine should be permitted to take a hu­ man life on the battlefield or in policing, border control, or any cir­ cumstances. It seeks to affirm the positive principle that weapons sys­

tems, military attacks, and the use of violent force in policing should always be kept under meaningful human control. In order to do that, the Campaign to Stop Killer Robots calls on states and others to endorse its call for a preemptive ban on the development, production, and use of fully autonomous weapons sys­ tems.

The global coalition of non-gov­ ernmental organizations does not oppose military use of autonomy and technologies relating to arti­ ficial intelligence, but it draws the line at the development of machines that could select and fire on targets without human intervention. Re­ search and development activities should be banned if they are direct­ ed at technology that can be used exclusively for fully autonomous weapons or that is explicitly intend­ ed for use in such weapons.

A preemptive prohibition is both possible and desirable. The Con­ vention on Conventional Weapons Protocol IV banning blinding lasers provides precedent for the creation of a protocol on lethal autonomous weapons systems.

The campaign has urged that states ban rather than regulate ful­ ly autonomous weapons because a complete prohibition is clearer and easier to enforce than partial regula­ tions or restrictions and eliminates room for different interpretations. A complete prohibition creates greater stigma against the weapons and discourages their proliferation. Mere regulation would still allow governments to obtain them, and be tempted to use them illegally.

The serious international securi­ ty and proliferation concerns relat­ ing to fully autonomous weapons

include the real danger that if even one nation acquires these weapons, others may feel they have to follow suit in order to defend themselves and to avoid falling behind in a ro­ botic arms race.

There is also the prospect that fully autonomous weapons could be acquired by repressive regimes or non-state armed groups with little

regard for the law. These weapons could be perfect tools of repression and terror for autocrats and non­ state actors.

Another proliferation concern is that such weapons would increase the likelihood of armed attacks, making war more likely, as deci­ sion-makers would not have the same concerns about loss of sol­ diers’ lives. This could have a desta­ bilizing effect on international secu­ rity given the current moral limits.

Since 2013, a broad range of in­ dividuals, organizations, and states have come forward to endorse the call for a preemptive ban on fully autonomous weapons systems, in­ cluding:

•Bolivia, Cuba, Ecuador, Egypt, Ghana, the Holy See, Pakistan, the State of Palestine, and Zim­ babwe. •The European Parliament, which adopted a resolution by a vote of 534-49 calling for a ban on “de­ velopment, production and use of fully autonomous weapons which enable strikes to be carried out without human intervention.” •21 Nobel Peace Laureates, who expressed concern that “leaving the killing to machines might make going to war easier.” •More than 120 religious leaders and organizations of various de­ nominations, who view the weap­ ons as “an affront to human dig­ nity and to the sacredness of life.” •More than 270 scientists in 37 countries, who warned how inter­ actions by devices controlled by complex algorithms “could cre­ ate unstable and unpredictable behavior ... that could initiate or escalate conflicts, or cause unjus­ tifiable harm to civilian popula­ tions.” •More than 3,000 artificial intel­ ligence (AI) and robotics experts, who said they have “no interest in building AI weapons and do not want others to tarnish their field by doing so, potentially creating a major public backlash against AI that curtails its future societal

‘‘The fundamental question is not technological, but philosophical: ‘Will machines be given the power and authority to kill a human being?’ *

32 SUMMER 2016

benefits.” •Canadian technology company Clearpath Robotics, which said it had “chosen to value our eth­ ics over potential future revenue” by pledging not to manufacture “weaponized robots that remove humans from the loop.” •The ethics council of the US$830 billion Norwegian government pension fund, which is looking at whether investments in compa­ nies making lethal autonomous weapons systems would violate the fund’s fundamental humani­ tarian principles. •The UN Special Rapporteur on extrajudicial, summary or arbi­ trary executions and the Special Rapporteur on the rights to free­ dom of peaceful assembly and of association, who in a report rec­ ommended that, “Autonomous weapons systems that require no meaningful human control should be prohibited.”

4. F ind ing th e w ay forw ard, fa ster tow ard an o u tco m e

In May 2013, states first delib­ erated on autonomous weapons in a multilateral forum where a UN Special Rapporteur presented a re­ port to the Human Rights Council that called for an immediate mor­ atorium on these weapons. At the Convention on Conventional Weap­ ons, also in Geneva, representatives from more than 85 countries met in May 2014 and April 2015 for a total of nine days to discuss lethal au­ tonomous weapons systems, while a third meeting took place in April 2016.

Various UN agencies, interna­ tional organizations, the Red Cross, and civil society groups coordinat­ ed by the Campaign to Stop Killer Robots also participate throughout these deliberations and other meet­ ings, which have helped sharpen the focus on lethal autonomous weapons systems and the concerns that they raise.

This process is helping to estab­ lish a common base of knowledge

from which some nascent norms or principles are finding broad agree­ ment, most notably the notion that human control must be retained over the operation of weapons sys­ tems that can kill human lives.

However, the informal talks have been criticized for “treading water”: they lack ambition, demonstrate no sense of urgency, and reflect the CCW’s usual “go slow and aim low” approach.

The Campaign to Stop Killer Robots calls on states to agree to a more formal and substantive pro­ cess, with up to four weeks of work

dedicated in 2017, when they decide on the way forward at the end of the year. The CCW Review Conference in December 2016 should agree to begin formal negotiations and aim to complete them within one or two years by producing a preemptive prohibition on lethal autonomous weapons systems.

Pakistan — the first and probably the most ardent supporter of the call for a ban — will serve as president of the CCW Review Conference, where states must make crucial decisions about the process going forward and desired outcome.

Sustained and substantive na­ tional campaigning, including leg­ islative scrutiny in the form of de­ bate, questions, and inquiries, and new legislation to ban the weapons is required to make any headway reining in autonomous weapons. As Canada’s new Minister of Defense Harjit Sajjan commented to the Halifax Security Forum, legislative oversight is required as technology advances so that the burden of nav­ igating ethical dilemmas associated with their use is not placed on ser­ vice men and women.

The United States is the only

FEATURES country with a detailed written pol­ icy guiding it on fully autonomous weapons, which it says “neither encourages nor prohibits” develop­ ment of lethal autonomous weap­ ons systems. The United States has described fully autonomous weapons as a potential “force mul­ tiplier.” Yet, recent media inter­ views with top Pentagon officials provide some disturbing signs that the five-year policy may soon be replaced by guidelines allowing for the acquisition of human-out-of- the-loop weapons systems. Howev­ er, officials also express support for

retaining human control by stating, for example, that “we will not dele­ gate lethal authority to a machine to make a decision.”

Israel has also spoken about the desirability and potential benefits of autonomous weapons systems, urging other countries to “keep an open mind” because it is “difficult” to foresee how developments may look in 10 to 30 years from now.

No other state has publicly ex­ pressed interest in pursuing such systems and several have explicitly stated that they are not developing such systems and have no plans to do so. But there have been exten­ sive discussions about the potential benefits of such weapons, and many states are concerned that potential enemies will acquire fully autono­ mous weapons, making their con­ tinued development inevitable.

The trend toward ever-greater autonomy in warfare has been ac­ celerating greatly in recent years. A strong, inclusive, and unified ef­ fort is needed now to prevent the unconstrained development of fully autonomous weapons systems, iti

4<T h e United States is the only country with a detailed written policy guiding it on fully autonomous weapons ..."

SUMMER 2016 33

Copyright of Harvard International Review is the property of Harvard International Review and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.