Assignment #1

23 The Transhumanist FAQ

Nick Bostrom

GENERAL QUESTIONS ABOUT TRANSHUMANISM

What is transhumanism?

Transhumanism is a way of thinking about the future that is based on the premise that the human species in its current form does not represent the end of our development but rather a comparatively early phase. We formally define it as follows:

1) The intellectual and cultural movement that affirms the possibility and desirability of fun- damentally improving the human condition through applied reason, especially by devel- oping and making widely available technologies to eliminate aging and to greatly enhance human intellectual, physical, and psychological capacities.

2) The study of the ramifications, promises, and potential dangers of technologies that will enable us to overcome fundamental human limitations, and the related study of the ethical matters involved in developing and using such technologies.

Transhumanism can be viewed as an extension of humanism, from which it is partially derived. Humanists believe that humans matter, that individuals matter. We might not be perfect, but we can make things better by promoting rational thinking, freedom, tolerance, democracy, and concern for our fellow human beings. Transhumanists agree with this but also emphasize what we have the potential to become. Just as we use rational means to improve the human condition and the external world, we can also use such means to improve ourselves, the human organism. In doing so, we are not limited to traditional humanistic methods, such as education and cultural develop- ment. We can also use technological means that will eventually enable us to move beyond what some would think of as ‘‘human.’’

It is not our human shape or the details of our current human biology that define what is valuable about us, but rather our aspirations and ideals, our experiences, and the kinds of lives we lead. To a transhumanist, progress occurs when more people become more able to shape them- selves, their lives, and the ways they relate to others, in accordance with their own deepest values. Transhumanists place a high value on autonomy: the ability and right of individuals to plan and

From Nick Bostrom et al., The Transhumanist FAQ: A General Introduction, version 2.1 (2003). Copyright � 2003 by The World Transhumanist Association. Reprinted by permission of the author.

345

C o p y r i g h t 2 0 0 9 . R o w m a n & L i t t l e f i e l d P u b l i s h e r s .

A l l r i g h t s r e s e r v e d . M a y n o t b e r e p r o d u c e d i n a n y f o r m w i t h o u t p e r m i s s i o n f r o m t h e p u b l i s h e r , e x c e p t f a i r u s e s p e r m i t t e d u n d e r U . S . o r a p p l i c a b l e c o p y r i g h t l a w .

EBSCO Publishing : eBook Comprehensive Academic Collection (EBSCOhost) - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS AN: 336849 ; David M. Kaplan.; Readings in the Philosophy of Technology Account: s4264928.main.edsebook

346 Nick Bostrom

choose their own lives. Some people may of course, for any number of reasons, choose to forgo the opportunity to use technology to improve themselves. Transhumanists seek to create a world in which autonomous individuals may choose to remain unenhanced or choose to be enhanced and in which these choices will be respected.

Through the accelerating pace of technological development and scientific understanding, we are entering a whole new stage in the history of the human species. In the relatively near future, we may face the prospect of real artificial intelligence. New kinds of cognitive tools will be built that combine artificial intelligence with interface technology. Molecular nanotechnology has the potential to manufacture abundant resources for everybody and to give us control over the biochem- ical processes in our bodies, enabling us to eliminate disease and unwanted aging. Technologies such as brain-computer interfaces and neuropharmacology could amplify human intelligence, increase emotional well-being, improve our capacity for steady commitment to life projects or a loved one, and even multiply the range and richness of possible emotions. On the dark side of the spectrum, transhumanists recognize that some of these coming technologies could potentially cause great harm to human life; even the survival of our species could be at risk. Seeking to understand the dangers and working to prevent disasters is an essential part of the transhumanist agenda.

What is a posthuman?

It is sometimes useful to talk about possible future beings whose basic capacities so radically exceed those of present humans as to be no longer unambiguously human by our current standards. The standard word for such beings is ‘‘posthuman.’’ (Care must be taken to avoid misinterpretation. ‘‘Posthuman’’ does not denote just anything that happens to come after the human era, nor does it have anything to do with the ‘‘posthumous.’’ In particular, it does not imply that there are no humans anymore.)

Many transhumanists wish to follow life paths which would, sooner or later, require growing into posthuman persons: they yearn to reach intellectual heights as far above any current human genius as humans are above other primates; to be resistant to disease and impervious to aging; to have unlimited youth and vigor; to exercise control over their own desires, moods, and mental states; to be able to avoid feeling tired, hateful, or irritated about petty things; to have an increased capacity for pleasure, love, artistic appreciation, and serenity; to experience novel states of con- sciousness that current human brains cannot access. It seems likely that the simple fact of living an indefinitely long, healthy, active life would take anyone to posthumanity if they went on accumulat- ing memories, skills, and intelligence.

Posthumans could be completely synthetic artificial intelligences, or they could be enhanced uploads, or they could be the result of making many smaller but cumulatively profound augmen- tations to a biological human. The latter alternative would probably require either the redesign of the human organism using advanced nanotechnology or its radical enhancement using some combi- nation of technologies such as genetic engineering, psychopharmacology, anti-aging therapies, neu- ral interfaces, advanced information management tools, memory enhancing drugs, wearable computers, and cognitive techniques.

Some authors write as though simply by changing our self-conception, we have become or could become posthuman. This is a confusion or corruption of the original meaning of the term. The changes required to make us posthuman are too profound to be achievable by merely altering some aspect of psychological theory or the way we think about ourselves. Radical technological modifications to our brains and bodies are needed.

It is difficult for us to imagine what it would be like to be a posthuman person. Posthumans may have experiences and concerns that we cannot fathom, thoughts that cannot fit into the three-

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 347

pound lumps of neural tissue that we use for thinking. Some posthumans may find it advantageous to jettison their bodies altogether and live as information patterns on vast super-fast computer net- works. Their minds may be not only more powerful than ours but may also employ different cogni- tive architectures or include new sensory modalities that enable greater participation in their virtual reality settings. Posthuman minds might be able to share memories and experiences directly, greatly increasing the efficiency, quality, and modes in which posthumans could communicate with each other. The boundaries between posthuman minds may not be as sharply defined as those between humans.

Posthumans might shape themselves and their environment in so many new and profound ways that speculations about the detailed features of posthumans and the posthuman world are likely to fail.

What is a transhuman?

In its contemporary usage, ‘‘transhuman’’ refers to an intermediary form between the human and the posthuman. One might ask, given that our current use of, for example, medicine and informa- tion technology enable us to routinely do many things that would have astonished humans living in ancient times, whether we are not already transhuman? The question is a provocative one, but ultimately not very meaningful; the concept of the transhuman is too vague for there to be a definite answer.

A transhumanist is simply someone who advocates transhumanism. It is a common error for reporters and other writers to say that transhumanists ‘‘claim to be transhuman’’ or ‘‘call themselves transhuman.’’ To adopt a philosophy which says that someday everyone ought to have the chance to grow beyond present human limits is clearly not to say that one is better or somehow currently ‘‘more advanced’’ than one’s fellow humans.

TECHNOLOGIES AND PROJECTIONS

Biotechnology, genetic engineering, stem cells, and cloning—what are they and what are they good for?

Biotechnology is the application of techniques and methods based on the biological sciences. It encompasses such diverse enterprises as brewing, manufacture of human insulin, interferon, and human growth hormone, medical diagnostics, cell cloning and reproductive cloning, the genetic modification of crops, bioconversion of organic waste and the use of genetically altered bacteria in the cleanup of oil spills, stem cell research, and much more. Genetic engineering is the area of biotechnology concerned with the directed alteration of genetic material.

Biotechnology already has countless applications in industry, agriculture, and medicine. It is a hotbed of research. The completion of the human genome project—a ‘‘rough draft’’ of the entire human genome was published in the year 2000—was a scientific milestone by anyone’s standards. Research is now shifting to decoding the functions and interactions of all these different genes and to developing applications based on this information.

The potential medical benefits are too many to list; researchers are working on every common disease, with varying degrees of success. Progress takes place not only in the development of drugs and diagnostics but also in the creation of better tools and research methodologies, which in turn accelerates progress. When considering what developments are likely over the long-term, such improvements in the research process itself must be factored in. The human genome project was completed ahead of schedule, largely because the initial predictions underestimated the degree to which instrumentation technology would improve during the course of the project. At the same

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

348 Nick Bostrom

time, one needs to guard against the tendency to hype every latest advance. (Remember all those breakthrough cancer cures that we never heard of again?) Moreover, even in cases where the early promise is borne out, it usually takes ten years to get from proof-of-concept to successful commer- cialization.

Genetic therapies are of two sorts: somatic and germ-line. In somatic gene therapy, a virus is typically used as a vector to insert genetic material into the cells of the recipient’s body. The effects of such interventions do not carry over into the next generation. Germ-line genetic therapy is per- formed on sperm or egg cells, or on the early zygote, and can be inheritable. (Embryo screening, in which embryos are tested for genetic defects or other traits and then selectively implanted, can also count as a kind of germ-line intervention.) Human gene therapy, except for some forms of embryo screening, is still experimental. Nonetheless, it holds promise for the prevention and treat- ment of many diseases, as well as for uses in enhancement medicine. The potential scope of genetic medicine is vast: virtually all disease and all human traits—intelligence, extroversion, conscien- tiousness, physical appearance, etc.—involve genetic predispositions. Single-gene disorders, such as cystic fibrosis, sickle cell anemia, and Huntington’s disease are likely to be among the first tar- gets for genetic intervention. Polygenic traits and disorders, ones in which more than one gene is implicated, may follow later (although even polygenic conditions can sometimes be influenced in a beneficial direction by targeting a single gene).

Stem cell research, another scientific frontier, offers great hopes for regenerative medicine. Stem cells are undifferentiated (unspecialized) cells that can renew themselves and give rise to one or more specialized cell types with specific functions in the body. By growing such cells in culture, or steering their activity in the body, it will be possible to grow replacement tissues for the treat- ment of degenerative disorders, including heart disease, Parkinson’s, Alzheimer’s, diabetes, and many others. It may also be possible to grow entire organs from stem cells for use in transplanta- tion. Embryonic stem cells seem to be especially versatile and useful, but research is also ongoing into adult stem cells and the ‘‘reprogramming’’ of ordinary cells so that they can be turned back into stem cells with pluripotent capabilities.

The term ‘‘human cloning’’ covers both therapeutic and reproductive uses. In therapeutic cloning, a preimplantation embryo (also known as ‘‘blastocyst’’—a hollow ball consisting of 30– 150 undifferentiated cells) is created via cloning, from which embryonic stem cells could be extracted and used for therapy. Because these cloned stem cells are genetically identical to the patient, the tissues or organs they would produce could be implanted without eliciting an immune response from the patient’s body, thereby overcoming a major hurdle in transplant medicine. Reproductive cloning, by contrast, would mean the birth of a child who is genetically identical to the cloned parent: in effect, a younger identical twin.

Everybody recognizes the benefit to ailing patients and their families that come from curing specific diseases. Transhumanists emphasize that, in order to seriously prolong the healthy life span, we also need to develop ways to slow aging or to replace senescent cells and tissues. Gene therapy, stem cell research, therapeutic cloning, and other areas of medicine that have the potential to deliver these benefits deserve a high priority in the allocation of research monies.

What is molecular nanotechnology?

Molecular nanotechnology is an anticipated manufacturing technology that will make it possible to build complex three-dimensional structures to atomic specification using chemical reactions directed by nonbiological machinery. In molecular manufacturing, each atom would go to a selected place, bonding with other atoms in a precisely designated manner. Nanotechnology prom- ises to give us thorough control of the structure of matter.

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 349

Since most of the stuff around us and inside us is composed of atoms and gets its characteris- tic properties from the placement of these atoms, the ability to control the structure of matter on the atomic scale has many applications. As K. Eric Drexler wrote in Engines of Creation, the first book on nanotechnology (published in 1986):

Coal and diamonds, sand and computer chips, cancer and healthy tissue: throughout history, variations in the arrangement of atoms have distinguished the cheap from the cherished, the dis- eased from the healthy. Arranged one way, atoms make up soil, air, and water; arranged another, they make up ripe strawberries. Arranged one way, they make up homes and fresh air; arranged another, they make up ash and smoke.

Nanotechnology, by making it possible to rearrange atoms effectively, will enable us to trans- form coal into diamonds, sand into supercomputers, and to remove pollution from the air and tumors from healthy tissue.

Central to Drexler’s vision of nanotechnology is the concept of the assembler. An assembler would be a molecular construction device. It would have one or more submicroscopic robotic arms under computer control. The arms would be capable of holding and placing reactive compounds so as to positionally control the precise location at which a chemical reaction takes place. The assem- bler arms would grab a molecular (but not necessarily individual atoms) and add it to a workpiece, constructing an atomically precise object step by step. An advanced assembler would be able to make almost any chemically stable structure. In particular, it would be able to make a copy of itself. Since assemblers could replicate themselves, they would be easy to produce in large quantities.

Mature nanotechnology will transform manufacturing into a software problem. To build something, all you will need is a detailed design of the object you want to make and a sequence of instructions for its construction. Rare or expensive raw materials are generally unnecessary; the atoms required for the construction of most kinds of nanotech devices exist in abundance in nature. Dirt, for example, is full of useful atoms.

By working in large teams, assemblers and more specialized nanomachines will be able to build large objects quickly. Consequently, while nanomachines may have features on the scale of a billionth of a meter—a nanometer—the products could be as big as space vehicles or even, in a more distant future, the size of planets.

While it seems fairly well established that molecular nanotechnology is in principle possible, it is harder to determine how long it will take to develop. A common guess among the cognoscenti is that the first assembler may be built around the year 2018, give or take a decade, but there is a large scope for diverging opinion on the upper side of that estimate.

Because the ramifications of nanotechnology are immense, it is imperative that serious thought be given to this topic now. If nanotechnology were to be abused the consequences could be devastating. Society needs to prepare for the assembler breakthrough and do advance planning to minimize the risks associated with it.

What is superintelligence?

A superintelligent intellect (a superintelligence, sometimes called ‘‘ultraintelligence’’) is one that has the capacity to radically outperform the best human brains in practically every field, including scientific creativity, general wisdom, and social skills.

Sometimes a distinction is made between weak and strong superintelligence. Weak superin- telligence is what you would get if you could run a human intellect at an accelerated clock speed, such as by uploading it to a fast computer. If the upload’s clock-rate were a thousand times that of

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

350 Nick Bostrom

a biological brain, it would perceive reality as being slowed down by a factor of a thousand. It would think a thousand times more thoughts in a given time interval than its biological counterpart.

Strong superintelligence refers to an intellect that is not only faster than a human brain but also smarter in a qualitative sense. No matter how much you speed up your dog’s brain, you’re not going to get the equivalent of a human intellect. Analogously, there might be kinds of smartness that wouldn’t be accessible to even very fast human brains given their current capacities. Something as simple as increasing the size or connectivity of our neuronal networks might give us some of these capacities. Other improvements may require wholesale reorganization of our cognitive archi- tecture or the addition of new layers of cognition on top of the old ones.

However, the distinction between weak and strong superintelligence may not be clear-cut. A sufficiently long-lived human who didn’t make any errors and had a sufficient stack of scrap paper at hand could in principle compute any Turing computable function. (According to Church’s thesis, the class of Turing computable functions is identical to the class of physically computable func- tions.)

Many but not all transhumanists expect that superintelligence will be created within the first half of this century. Superintelligence requires two things: hardware and software. Chip-manufac- turers planning the next generation of microprocessors commonly rely on a well-known empirical regularity known as Moore’s Law. In its original 1965-formulation by Intel co-founder Gordon Moore, it stated that the number of components on a chip doubled every year. In contemporary use, the ‘‘law’’ is commonly understood as referring more generally to a doubling of computing power, or of computing power per dollar. For the past couple of years, the doubling time has hov- ered between eighteen months and two years.

Most experts, Moore included, think that computing power will continue to double about every eighteen months for at least another two decades. This expectation is based in part on extrap- olation from the past and in part on consideration of developments currently underway in labora- tories. Thus it appears quite likely that human-equivalent hardware will have been achieved within not much more than a couple of decades.

How long it will take to solve the software problem is harder to estimate. One possibility is that progress in computational neuroscience will teach us about the computational architecture of the human brain and what learning rules it employs. We can then implement the same algorithms on a computer. In this approach, the superintelligence would not be completely specified by the programmers but would instead have to grow by learning from experience the same way a human infant does. An alternative approach would be to use genetic algorithms and methods from classical AI. This might result in a superintelligence that bears no close resemblance to a human brain. At the opposite extreme, we could seek to create a superintelligence by uploading a human intellect and then accelerating and enhancing it. The outcome of this might be a superintelligence that is a radically upgraded version of one particular human mind.

The arrival of superintelligence will clearly deal a heavy blow to anthropocentric worldviews. Much more important than its philosophical implications, however, would be its practical effects. Creating superintelligence may be the last invention that humans will ever need to make, since superintelligences could themselves take care of further scientific and technological development. They would do so more effectively than humans. Biological humanity would no longer be the smartest life form on the block. The prospect of superintelligence raises many big issues and con- cerns that we should think deeply about in advance of its actual development. The paramount ques- tion is: What can be done to maximize the chances that the arrival of superintelligence will benefit rather than harm us? The range of expertise needed to address this question extends far beyond the community of AI researchers. Neuroscientists, economists, cognitive scientists, computer scientists, philosophers, ethicists, sociologists, science-fiction writers, military strategists, politicians, legisla-

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 351

tors, and many others will have to pool their insights if we are to deal wisely with what may be the most important task our species will ever have to tackle.

Many transhumanists would like to become superintelligent themselves. This is obviously a long-term and uncertain goal, but it might be achievable either through uploading and subsequent enhancement or through the gradual augmentation of our biological brains, by means of future nootropics (cognitive enhancement drugs), cognitive techniques, IT tools (e.g., wearable comput- ers, smart agents, information filtering systems, visualization software, etc.), neural-computer inter- faces, or brain implants.

What is uploading?

Uploading (sometimes called ‘‘downloading,’’ ‘‘mind uploading,’’ or ‘‘brain reconstruction’’) is the process of transferring an intellect from a biological brain to a computer.

One way of doing this might be by first scanning the synaptic structure of a particular brain and then implementing the same computations in an electronic medium. A brain scan of sufficient resolution could be produced by disassembling the brain atom for atom by means of nanotechnol- ogy. Other approaches, such as analyzing pieces of the brain slice by slice in an electron micro- scope with automatic image processing have also been proposed. In addition to mapping the connection pattern among the 100 billion-or-so neurons, the scan would probably also have to regis- ter some of the functional properties of each of the synaptic interconnections, such as the efficacy of the connection and how stable it is over time (e.g., whether it is short-term or long-term potenti- ated). Non-local modulators such as neurotransmitter concentrations and hormone balances may also need to be represented, although such parameters likely contain much less data than the neu- ronal network itself.

In addition to a good three-dimensional map of a brain, uploading will require progress in neuroscience to develop functional models of each species of neuron (how they map input stimuli to outgoing action potentials, and how their properties change in response to activity in learning). It will also require a powerful computer to run the upload, and some way for the upload to interact with the external world or with a virtual reality. (Providing input/output or a virtual reality for the upload appears easy in comparison to the other challenges.)

An alternative hypothetical uploading method would proceed more gradually: one neuron could be replaced by an implant or by a simulation in a computer outside of the body. Then another neuron, and so on, until eventually the whole cortex has been replaced and the person’s thinking is implemented on entirely artificial hardware. (To do this for the whole brain would almost certainly require nanotechnology.)

A distinction is sometimes made between destructive uploading, in which the original brain is destroyed in the process, and non-destructive uploading, in which the original brain is preserved intact alongside the uploaded copy. It is a matter of debate under what conditions personal identity would be preserved in destructive uploading. Many philosophers who have studied the problem think that at least under some conditions, an upload of your brain would be you. A widely accepted position is that you survive so long as certain information patterns are conserved, such as your memories, values, attitudes, and emotional dispositions, and so long as there is causal continuity so that earlier stages of yourself help determine later stages of yourself. Views differ on the relative importance of these two criteria, but they can both be satisfied in the case of uploading. For the continuation of personhood, on this view, it matters little whether you are implemented on a silicon chip inside a computer or in that gray, cheesy lump inside your skull, assuming both implementa- tions are conscious.

Tricky cases arise, however, if we imagine that several similar copies are made of your

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

352 Nick Bostrom

uploaded mind. Which one of them is you? Are they all you, or are none of them you? Who owns your property? Who is married to your spouse? Philosophical, legal, and ethical challenges abound. Maybe these will become hotly debated political issues later in this century.

A common misunderstanding about uploads is that they would necessarily be ‘‘disembodied’’ and that this would mean that their experiences would be impoverished. Uploading according to this view would be the ultimate escapism, one that only neurotic body-loathers could possibly feel tempted by. But an upload’s experience could in principle be identical to that of a biological human. An upload could have a virtual (simulated) body giving the same sensations and the same possibilities for interaction as a non-simulated boy. With advanced virtual reality, uploads could enjoy food and drink, and upload sex could be as gloriously messy as one could wish. And uploads wouldn’t have to be confined to virtual reality: they could interact with people on the outside and even rent robot bodies in order to work in or explore physical reality.

Personal inclinations regarding uploading differ. Many transhumanists have a pragmatic atti- tude: whether they would like to upload or not depends on the precise conditions in which they would live as uploads and what the alternative use. (Some transhumanists may also doubt whether uploading will be possible.) Advantages of being an upload would include:

• Uploads would not be subject to biological senescence. • Back-up copies of uploads could be created regularly so that you could be rebooted if

something bad happened. (Thus your lifespan would potentially be as long as the uni- verse’s.)

• You could potentially live much more economically as an upload since you wouldn’t need physical food, housing, transportation, etc.

• If you were running on a fast computer, you would think faster than in a biological imple- mentation. For instance, if you were running on a computer a thousand times more power- ful than a human brain, then you would think a thousand times faster (and the external world would appear to you as if it were slowed down by a factor of a thousand). You would thus get to experience more subjective time, and live more, during any given day.

• You could travel at the speed of light as an information patterns, which could be convenient in a future age of large-scale space settlements.

• Radical cognitive enhancements would likely be easier to implement in an upload than in an organic brain.

A couple of other points about uploading:

• Uploading should work for cryonics patients provided their brains are preserved in a suffi- ciently intact state.

• Uploads could reproduce extremely quickly (simply by making copies of themselves). This implies that resources could very quickly become scarce unless reproduction is regulated.

What is the singularity?

Some thinkers conjecture that there will be a point in the future when the rate of technological development becomes so rapid that the progress-curve becomes nearly vertical. Within a very brief time (months, days, or even just hours), the world might be transformed almost beyond recognition. This hypothetical point is referred to as the singularity. The most likely cause of a singularity would be the creation of some form of rapidly self-enhancing greater-than-human intelligence.

The concept of the singularity is often associated with Vernor Vinge, who regards it as one

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 353

of the more probable scenarios for the future. Provided that we manage to avoid destroying civiliza- tion, Vinge thinks that a singularity is likely to happen as a consequence of advances in artificial intelligence, large systems of networked computers, computer-human integration, or some other form of intelligence amplification. Enhancing intelligence will, in this scenario, at some point lead to a positive feedback loop: smarter systems can design systems that are even more intelligent, and can do so more swiftly than the original human designers. This positive feedback effect would be powerful enough to drive an intelligence explosion that could quickly lead to the emergence of a superintelligent system of surpassing abilities.

The singularity-hypothesis is sometimes paired with the claim that it is impossible for us to predict what comes after the singularity. A post-singularity society might be so alien that we can know nothing about it. One exception might be the basic laws of physics, but even there it is some- times suggested that there may be undiscovered laws (for instance, we don’t yet have an accepted theory of quantum gravity) or poorly understood consequences of known laws that could be exploited to enable things we would normally think of as physically impossible, such as creating traversable wormholes, spawning new ‘‘basement’’ universes, or traveling backward in time. How- ever, unpredictability is logically distinct from abruptness of development and would need to be argued for separately.

Transhumanists differ widely in the probability they assign to Vinge’s scenario. Almost all of those who do think that there will be a singularity believe it will happen in this century, and many think it is likely to happen within several decades.

SOCIETY AND POLITICS

Will new technologies only benefit the rich and powerful?

It is clear that everybody can benefit greatly from improved technology. Initially, however, the greatest advantages will go to those who have the resources, the skills, and the willingness to learn to use new tools. One can speculate that some technologies may cause social inequalities to widen. For example, if some form of intelligence amplification becomes available, it may at first be so expensive that only the wealthiest can afford it. The same could happen when we learn how to genetically enhance our children. Those who are already well off would become smarter and make even more money. This phenomenon is not new. Rich parents send their kids to better schools and provide them with resources such as personal connections and information technology that may not be available to the less privileged. Such advantages lead to greater earnings later in life and serve to increase social inequalities.

Trying to ban technological innovation on these grounds, however, would be misguided. If a society judges existing inequalities to be unacceptable, a wiser remedy would be progressive taxa- tion and the provision of community-funded services such as education, IT access in public librar- ies, genetic enhancements covered by social security, and so forth. Economic and technological progress is not a zero sum game; it’s a positive sum game. Technological progress does not solve the hard old political problem of what degree of income redistribution is desirable, but it can greatly increase the size of the pie that is to be divided.

Aren’t these future technologies very risky? Could they even cause our extinction?

Yes, and this implies an urgent need to analyze the risks before they materialize and to take steps to reduce them. Biotechnology, nanotechnology, and artificial intelligence pose especially serious risks of accidents and abuse.

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

354 Nick Bostrom

One can distinguish between, on the one hand, endurable or limited hazards, such as car crashes, nuclear reactor meltdowns, carcinogenic pollutants in the atmosphere, floods, volcano eruptions, and so forth, and, on the other hand, existential risks—events that would cause the extinction of intelligent life or permanently and drastically cripple its potential. While endurable or limited risks can be serious—and may indeed be fatal to the people immediately exposed—they are recoverable; they do not destroy the long-term prospects of humanity as a whole. Humanity has long experience with endurable risks and a variety of institutional and technological mechanisms have been employed to reduce their incidence. Existential risks are a different kind of beast. For most of human history, there were no significant existential risks, or at least none that our ancestors could do anything about. By definition of course, no existential disaster has yet happened. As a species we may therefore be less well prepared to understand and manage this new kind of risk. Furthermore, the reduction of existential risk is a global public good (everybody by necessity bene- fits from such safety measures, whether or not they contribute to their development), creating a potential free-rider problem, that is, a lack of sufficient selfish incentives for people to make sacri- fices to reduce an existential risk. Transhumanists therefore recognize a moral duty to promote efforts to reduce existential risks.

The gravest existential risks facing us in the coming decades will be of our own making. These include:

Destructive uses of nanotechnology. The accidental release of a self-replicating nanobot into the environment, where it would proceed to destroy the entire biosphere, is known as the ‘‘gray goo scenario.’’ Since molecular nanotechnology will make use of positional assembly to create nonbiological structures and to open new chemical reaction pathways, there is no reason to suppose that the ecological checks and balances that limit the proliferation of organic self-replicators would also contain nano-replicators. Yet, while gray goo is certainly a legitimate concern, relatively sim- ple engineering safeguards have been described that would make the probability of such a mishap almost arbitrarily small. Much more serious is the threat posed by nanobots deliberately designed to be destructive. A terrorist group or even a lone psychopath, having obtained access to this tech- nology, could do extensive damage or even annihilate life on earth unless effective defensive tech- nologies had been developed beforehand. An unstable arms race between nanotechnic states could also result in our eventual demise. Anti-proliferation efforts will be complicated by the fact that nanotechnology does not require difficult-to-obtain raw materials or large manufacturing plants, and by the dual-use functionality of many of the basic components of destructive nanomachinery. While a nanotechnic defense system (which would act as a global immune system capable of identi- fying and neutralizing rogue replicators) appears to be possible in principle, it could turn out to be more difficult to construct than a simple destructive replicator. This could create a window of global vulnerability between the potential creation of dangerous replicators and the development of an effective immune system. It is critical that nano-assemblers do not fall into the wrong hands during this period.

Biological warfare. Progress in genetic engineering will lead not only to improvements in medicine but also to the capability to create more effective bioweapons. It is chilling to consider what would have happened if HIV had been as contagious as the virus that causes the common cold. Engineering such microbes might soon become possible for increasing numbers of people. If the RNA sequence of a virus is posted on the Internet, then anybody with some basic expertise and access to a lab will be able to synthesize the actual virus from this description. A demonstration of this possibility was offered by a small team of researchers from New York University at Stony Brook in 2002, who synthesized the polio virus (whose genetic sequence is on the Internet) from scratch and injected it into mice that subsequently became paralyzed and died.

Artificial intelligence. No threat to human existence is posed by today’s AI systems or their

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 355

near-term successors. But if and when superintelligence is created, it will be of paramount impor- tance that it be endowed with human-friendly values. An imprudently or maliciously designed superintelligence, with goals amounting to indifference or hostility to human welfare, could cause our extinction. Another concern is that the first superintelligence, which may become very powerful because of its superior planning ability and because of the technologies it could swiftly develop, would be built to serve only a single person or a small group (such as its programmers or the corporation that commissioned it). While this scenario may not entail the extinction of literally all intelligent life, it nevertheless constitutes an existential risk because the future that would result would be one in which a great part of humanity’s potential had been permanently destroyed and in which at most a tiny fraction of all humans would get to enjoy the benefits of posthumanity.

Nuclear war. Today’s nuclear arsenals are probably not sufficient to cause the extinction of all humans, but future arms races could result in even larger build-ups. It is also conceivable that an all-out nuclear war would lead to the collapse of modern civilization, and it is not completely certain that the survivors would succeed in rebuilding a civilization capable of sustaining growth and technological development.

Something unknown. All the above risks were unknown a century ago and several of them have only become clearly understood in the past two decades. It is possible that there are future threats of which we haven’t yet become aware.

Evaluating the total probability that some existential disaster will do us in before we get the opportunity to become posthuman can be done by various direct or indirect methods. Although any estimate inevitably includes a large subjective factor, it seems that to set the probability to less than 20 percent would be unduly optimistic, and the best estimate may be considerably higher. But depending on the actions we take, this figure can be raised or lowered.

If these technologies are so dangerous, should they be banned? What can be done to reduce the risks?

The position that we ought to relinquish research into robotics, genetic engineering, and nanotech- nology has been advocated in an article by Bill Joy (2000). Joy argued that some of the future applications of these technologies are so dangerous that research in those fields should be stopped now. Partly because of Joy’s previously technophiliac credentials (he was a software designer and a cofounder of Sun Microsystems), his article, which appeared in Wired magazine, attracted a great deal of attention.

Many of the responses to Joy’s article pointed out that there is no realistic prospect of a worldwide ban on these technologies; that they have enormous potential benefits that we would not want to forgo; that the poorest people may have a higher tolerance for risk in developments that could improve their condition; and that a ban may actually increase the dangers rather than reduce them, both by delaying the development of protective applications of these technologies, and by weakening the position of those who choose to comply with the ban relative to less scrupulous groups who defy it.

A more promising alternative than a blanket ban is differential technological development, in which we would seek to influence the sequence in which technologies developed. On this approach, we would strive to retard the development of harmful technologies and their applications, while accelerating the development of beneficial technologies, especially those that offer protection against the harmful ones. For technologies that have decisive military applications, unless they can be verifiably banned, we may seek to ensure that they are developed at a faster pace in countries we regard as responsible than in those that we see as potential enemies. (Whether a ban is verifiable

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

356 Nick Bostrom

and enforceable can change over time as a result of developments in the international system or in surveillance technology.)

In the case of nanotechnology, the desirable sequence of development is that nanotech immune systems and other defensive measures be deployed before offensive capabilities become available to many independent powers. Once a technology is shared by many, it becomes extremely hard to prevent further proliferation. In the case of biotechnology, we should seek to promote research into vaccines, anti-viral drugs, protective gear, sensors, and diagnostics, and to delay as long as possible the development and proliferation of biological warfare agents and the means of their weaponization. For artificial intelligence, a serious risk will emerge only when capabilities approach or surpass those of humans. At that point one should seek to promote the development of friendly AI and to prevent unfriendly or unreliable AI systems.

Superintelligence is an example of a technology that seems especially worth promoting because it can help reduce a broad range of threats. Superintelligent systems could advise us on policy and make the progress curve for nanotechnology steeper, thus shortening the period of vul- nerability between the development of dangerous nanoreplicators and the deployment of effective defenses. If we have a choice, it seems preferable that superintelligence be developed before advanced nanotechnology, as superintelligence could help reduce the risks of nanotechnology but not vice versa. Other technologies that have wide risk-reducing uses include intelligence augmen- tation, information technology, and surveillance. These can make us smarter individually and col- lectively or make enforcement of necessary regulation more feasible. A strong prima facie case therefore exists for pursuing these technologies as vigorously as possible. Needless to say, we should also promote non-technological developments that are beneficial in almost all scenarios, such as peace and international cooperation.

In confronting the hydra of existential, limited, and endurable risks glaring at us from the future, it is unlikely that any one silver bullet will provide adequate protection. Instead, an arsenal of countermeasures will be needed so that we can address the various risks on multiple levels.

The first step to tackling a risk is to recognize its existence. More research is needed, and existential risks in particular should be singled out for attention because of their seriousness and because of the special nature of the challenges they pose. Surprisingly little work has been done in this area. The strategic dimensions of our choices must be taken into account, given that some of the technologies in question have important military ramifications. In addition to scholarly studies of the threats and their possible countermeasures, public awareness must be raised to enable a more informed debate of our long-term options.

Some of the lesser existential risks, such as an apocalyptic asteroid impact or the highly spec- ulative scenario involving something like the upsetting of a metastable vacuum state in some future particle accelerator experiment, could be substantially reduced at relatively small expense. Pro- grams to accomplish this—for example, an early detection system for dangerous near-earth objects on potential collation course with Earth, or the commissioning of advance peer review of planned high-energy physics experiments—are probably cost-effective. However, these lesser risks must not deflect attention from the more serious concern raised by more probable existential disasters.

In light of how superabundant the human benefits of technology can ultimately be, it matters less that we obtain all of these benefits in their precisely most optimal form, and more that we obtain them at all. For many practical purposes, it makes sense to adopt the rule of thumb that we should act so as to maximize the probability of an acceptable outcome, one in which we attain some (reasonably broad) realization of our potential; or, to put it in negative terms, that we should act so as to minimize net existential risk.

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 357

Shouldn’t we concentrate on current problems such as improving the situation of the poor, rather than putting our efforts into planning for the ‘‘far’’ future?

Many of the technologies and trends that transhumanists discuss are already reality. Biotechnology and information technology have transformed large sectors of our economies. The relevance of transhumanist ethics is manifest in such contemporary issues as stem cell research, genetically modified crops, human genetic therapy, embryo screening, end of life decisions, enhancement med- icine, information markets, and research funding priorities. The importance of transhumanist ideas is likely to increase as the opportunities for human enhancement proliferate.

An argument can be made that the most efficient way of contributing to making the world better is by participating in the transhumanist project. This is so because the stakes are enormous— humanity’s entire future may depend on how we manage the coming technological transitions—and because relatively few resources are at the present time being devoted to transhumanist efforts. Even one extra person can still make a significant difference here.

Will extended life worsen overpopulation problems?

Population increase is an issue we would ultimately have to come to grips with even if healthy life- extension were not to happen. Leaving people to die is an unacceptable solution.

A large population should not be viewed simply as a problem. Another way of looking at the same fact is that it means that many persons now enjoy lives that would not have been lived if the population had been smaller. One could ask those who complain about overpopulation exactly which people’s lives they would have preferred should not have been led. Would it really have been better if billions of the world’s people had never existed and if there had been no other people in their place? Of course, this is not to deny that too-rapid population growth can cause crowding, poverty, and the depletion of natural resources. In this sense there can be real problems that need to be tackled.

How many people the Earth can sustain at a comfortable standard of living is a function of technological development (as well as of how resources are distributed). New technologies, from simple improvements in irrigation and management, to better mining techniques and more efficient power generation machinery, to genetically engineered crops, can continue to improve world resource and food output, while at the same time reducing environmental impact and animal suf- fering.

Environmentalists are right to insist that the status quo is unsustainable. As a matter of physi- cal necessity, things cannot stay as they are today indefinitely, or even for very long. If we continue to use up resources at the current pace, without finding more resources or learning how to use novel kinds of resources, then we will run into serious shortages sometime around the middle of this century. The deep greens have an answer to this: they suggest we turn back the clock and return to an idyllic pre-industrial age to live in sustainable harmony with nature. The problem with this view is that the pre-industrial age was anything but idyllic. It was a life of poverty, misery, disease, heavy manual toil from dawn to dusk, superstitious fears, and cultural parochialism. Nor was it environmentally sound—as witness the deforestation of England and the Mediterranean region, desertification of large parts of the middle east, soil depletion by the Anasazi in the Glen Canyon area, destruction of farm land in ancient Mesopotamia through the accumulation of mineral salts from irrigation, deforestation and consequent soil erosion by the ancient Mexican Mayas, overhunt- ing of big game almost everywhere, and the extinction of the dodo and other big featherless birds in the South Pacific. Furthermore, it is hard to see how more than a few hundred million people

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

358 Nick Bostrom

could be maintained at a reasonable standard of living with pre-industrial production methods, so some 90 percent of the world population would somehow have to vanish in order to facilitate this nostalgic return.

Transhumanists propose a much more realistic alternative: not to retreat to an imagined past, but to press ahead as intelligently as we can. The environmental problems that technology creates are problems of intermediary, inefficient technology, of placing insufficient political priority on environmental protection as well as of a lack of ecological knowledge. Technologically less advanced industries in the former Soviet-bloc pollute much more than do their advanced Western counterparts. High-tech industry is typically relatively benign. Once we develop molecular nano- technology, we will not only have clean and efficient manufacturing of almost any commodity, but we will also be able to clean up much of the mess created by today’s crude fabrication methods. This would set a standard for a clean environment that today’s traditional environmentalists could scarcely dream of.

Nanotechnology will also make it cheaper to colonize space. From a cosmic point of view, Earth is an insignificant speck. It has sometimes been suggested that we ought to leave space untouched in its pristine glory. This view is hard to take seriously. Every hour, through entirely natural processes, vast amounts of resources—millions of times more than the sum total of what the human species has consumed throughout its career—are transformed into radioactive substances or wasted as radiation escaping into intergalactic space. Can we not think of some more creative way of using all this matter and energy?

Even with full-blown space colonization, however, population growth can continue to be a problem, and this is so even if we assume that an unlimited number of people could be transported from Earth into space. If the speed of light provides an upper bound on the expansion speed then the amount of resources under human control will grow only polynomially (�t3). Population, on the other hand, can easily grow exponentially (�et). If that happens, then, since a factor that grows exponentially will eventually overtake any factor that grows polynomially, average income will ultimately drop to subsistence levels, forcing population growth to slow. How soon this would hap- pen depends primarily on reproduction rates. A change in average life span would not have a big effect. Even vastly improved technology can only postpone this inevitably for a relatively brief time. The only long-term method of assuring continued growth of average income is some form of population control, whether spontaneous or imposed, limiting the number of new persons created per year. This does not mean that population could not grow, only that the growth would have to be polynomial rather than exponential.

Some additional points to consider:

• In technologically advanced countries, couples tend to have fewer children, often below the replacement rate. As an empirical generalization, giving people increased rational con- trol over their lives, especially through women’s education and participation in the labor market, causes couples to have fewer children.

• If one took seriously the idea of controlling population by limiting life span, why not be more active about it? Why not encourage suicide? Why not execute anyone reaching the age of seventy-five?

• If slowing aging were unacceptable because it might lead to there being more people, what about efforts to cure cancer, reduce traffic deaths, or improve worker safety? Why use double standards?

• When transhumanists say they want to extend lifespans, what they mean is that they want to extend healthspans. This means that the extra person-years would be productive and

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

The Transhumanist FAQ 359

would add economic value to society. We can all agree that there would be little point in living an extra ten years in a state of dementia.

• The world population growth rate has been declining for several decades. It peaked in 1970 at 2.1 percent. In 2003, it was 1.2 percent; and it is expected to fall below 1.0 percent around 2015 (United Nations 2002). The doomsday predictions of the so-called ‘‘Club of Rome’’ from the early 1970s have consistently turned out to be wrong.

• The more people there are, the more brains there will be working to invent new ideas and solutions.

• If people can look forward to a longer healthy, active life, they will have a personal stake in the future and will hopefully be more concerned about the long-term consequences of their actions.

How does transhumanism relate to religion?

Transhumanism is a philosophical and cultural movement concerned with promoting responsible ways of using technology to enhance human capacities and to increase the scope of human flour- ishing.

While not a religion, transhumanism might serve a few of the same functions that people have traditionally sought in religion. It offers a sense of direction and purpose and suggests a vision that humans can achieve something greater than our present condition. Unlike most religious believers, however, transhumanists seek to make their dreams come true in this world, by relying not on supernatural powers or divine intervention but on rational thinking and empiricism, through continued scientific, technological, economic, and human development. Some of the prospects that used to be the exclusive thunder of the religious institutions, such as very long lifespan, unfading bliss, and godlike intelligence, are being discussed by transhumanists as hypothetical future engi- neering achievements.

Transhumanism is a naturalistic outlook. At the moment, there is no hard evidence for super- natural forces or irreducible spiritual phenomena, and transhumanists prefer to derive their under- standing of the world from rational modes of inquiry, especially the scientific method. Although science forms the basis for much of the transhumanist worldview, transhumanists recognize that science has its own fallibilities and imperfections, and that critical ethical thinking is essential for guiding our conduct and for selecting worthwhile aims to work toward.

Religious fanaticism, superstition, and intolerance are not acceptable among transhumanists. In many cases, these weaknesses can be overcome through a scientific and humanistic education, training in critical thinking, and interaction with people from different cultures. Certain other forms of religiosity, however, may well be compatible with transhumanism.

It should be emphasized that transhumanism is not a fixed set of dogmas. It is an evolving worldview, or rather, a family of evolving worldviews—for transhumanists disagree with each other on many issues. The transhumanist philosophy, still in its formative stages, is meant to keep developing in the light of new experiences and new challenges. Transhumanists want to find out where they are wrong and to change their views accordingly.

Won’t it be boring to live forever in a perfect world?

Why not try it and see? ‘‘Perfection’’ is a vague and treacherous word. There is considerable disagreement among

transhumanists about what kind of perfection is attainable and desirable, either in theory or in prac- tice. It is probably wiser to speak of improving the world, rather than making it ‘‘perfect.’’ Would

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use

360 Nick Bostrom

it be boring to live for an indefinitely long time in a greatly improved world? The world could surely be improved over the way it is now, including becoming less boring. If you got rid of the pain and stress associated with, say, filling out annual tax returns, people would probably not sit around afterward saying: ‘‘Life feels meaningless now that I no longer have income tax forms to fill out.’’

Admittedly, material improvements to the environment may not, in themselves, be sufficient to bring about lasting happiness. If your accustomed fare is bread and water, then a box of cookies can be a feast. But if every night you eat out at fancy restaurants, such fine fare will soon seem ordinary and normal; and any lesser feast, such as a box of cookies, would be insulting by compari- son. Some cognitive scientists speculate that we each have a ‘‘set point’’ of happiness, to which we soon return regardless of changes in the environment. There may be considerable truth to the folk wisdom that an expensive new car does not make you happier (or rather, it makes you happier, but only temporarily). In some ways, human minds and brains are just not designed to be happy. Fortu- nately, there are several potential viewpoints from which to go about addressing this challenge.

Apes engage in activities that we, as humans, would find repetitive and dull. In the course of becoming smarter, we have become bored by things that would have interested our ancestors. But at the same time we have opened up a vast new space of possibilities for having fun—and the new space is much larger than the previous one. Humans are not simply apes who can obtain more bananas using our intelligence as a tool. Our intelligence enables us to desire new things, such as art, science, and mathematics. If at any point in your indefinitely long life you become bored with the greatly improved world, it may only indicate that the time has come to bump up your intelli- gence another increment.

If the human brain has a ‘‘set point’’ of happiness to which it returns, maybe this is a design flaw and should be fixed—one of those things that we will end up defining as human, but not humane. It would probably be unwise to eliminate boredom entirely, since boredom can serve to prevent us from wasting too much time on monotonous and meaningless activities. But if we’re doing new things, learning, growing more intelligent, and we still aren’t happy, for no better reason than that our cognitive architecture is badly designed, then perhaps it is time to redesign it. Present clinical mood-drugs are crude, but nonetheless they can sometimes restore interest and enthusiasm for life—sometimes tiredness and despair has no interesting reason behind it and is simply an imbalance of brain chemistry. Only by compartmentalizing our thinking to a high degree can we imagine a world where there is mature molecular nanotechnology and superhuman artificial intelli- gence, but the means are still lacking to control the brain circuitry of boredom. Fundamentally, there is no reason why pleasure, excitement, profound well-being and simple joy at being alive could not become the natural, default state of mind for all who desire it.

We should also consider some of the following points:

1) Ordinary life is sometimes boring. So what? 2) Eternal life will be as boring or as exciting as you make it. 3) Is being dead more exciting? 4) If eternal life becomes boring, you will have the option of ending it at any time.

Transhumanism is not about a fancier car, more money, or clever gadgetry, even though this is what the media presents to us as ‘‘science’’ and ‘‘advanced technology’’: transhumanism is about genuine changes to the human condition, including increased intelligence and minds better suited to the achievement of happiness.

EBSCOhost - printed on 2/25/2023 1:56 AM via UNIVERSITY OF MARYLAND GLOBAL CAMPUS. All use subject to https://www.ebsco.com/terms-of-use