Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Why those who care about catastrophic and existential risk should care about autonomous weapons , published by aaguirre on the Effective Altruism Forum.
(crossposted to Lesswrong here.)
Although I have not seen the argument made in any detail or in writing, I and the Future of Life Institute (FLI) have gathered the strong impression that parts of the effective altruism ecosystem are skeptical of the importance of the issue of autonomous weapons systems. This post explains why we think those interested in avoiding catastrophic and existential risk, especially risk stemming from emerging technologies, may want to have this issue higher on their list of concerns.
We will first define some terminology and do some disambiguation, as there are many classes of autonomous weapons that are often conflated; all classes have some issues of concern, but some are much more problematic than others. We then detail three basic motivations for research, advocacy, coordination, and policymaking around the issue:
Governance of autonomous weapon systems is a dry-run, and precedent, for governance of AGI. In the short term, AI-enabled weapons systems will share many of the technical weaknesses and shortcomings of other AI systems, but like general AI also raise safety concerns that are likely to increase rather than decrease with capability advances. The stakes are intrinsically high (literally life-or-death), and the context is an inevitably adversarial one involving states and major corporations. The sort of global coordination amongst potentially adversarial parties that will be required for governance of transformative/general AI systems will not arise from nowhere, and autonomous weapons offer an invaluable precedent and arena in which to build experience, capability, and best practices.
Some classes of lethal autonomous weapon systems constitute scalable weapons of mass destruction (which may also have a much lower threshold for first use or accidental escalation), and hence a nascent catastrophic risk.
By increasing the probability of the initiation and/or escalation of armed conflict, including catastrophic global armed conflict and/or nuclear war, autonomous weapons represent a very high expected cost that overwhelmingly offsets any gain in life from substituting autonomous weapons for humans in armed conflict.
Classes of autonomous weapons
Because many things with very different characteristics could fall under the rubric of “autonomous weapon systems” (AWSs) it is worth distinguishing and classifying them. First, let us split off cyberweapons – including AI-powered ones – as being an important but distinct issue. Likewise, we’ll set aside AI in other aspects of the military not directly related to the use of force, from strategy to target identification, where it serves to augment human action and decision-making. Rather, we focus on systems that have both (some form of) AI and physical armaments.
We now consider in turn these armaments’ target types, which we will break into categories of anti-personnel weapons, force-on-force (i.e. attacking manned enemy vehicles or structures) weaponry, and those targeting other autonomous weapon systems.
Anti-personnel AWSs can be further divided into lethal (or grossly injurious) ones versus nonlethal ones. While an interesting topic,[1] we leave aside here non-lethal anti-personnel autonomous weapon systems, which have a somewhat distinct set of considerations.[2]
We regard force-on-force systems designed to attack manned military vehicles and installations as relatively less intrinsically concerning. The targets of such weapons will, with considerably higher probability, be valid military targets rather than civilian ones, and insofar as they scale to mass damage, that damage will be to an adversary’s military. Of course if these weapon...
view more