Memo Regarding AI

Topic: The ‘Singularity’, AGI, and existential risk. Some authors (e.g. Bostrom (Links to an

external site.), Yampolskiy (Links to an external site.), and Yampolskiy again (Links to an

external site.), et al.) worry that the development of human-level or greater artificial

general intelligence will pose increasingly catastrophic risks, perhaps even soon posing

an existential risk (a risk that human-level intelligence will go extinct), either on purpose

or by accident. That is, a future AGI might have the goal of killing all humans, and be able

to do so, by using robots or other weapons under its control. This worry is perhaps

exacerbated if humans treat robotic persons as their property - their slaves. It is also

possible, if less likely, that an AGI might inadvertently cause human extinction in the

pursuit of other goals, as in the ‘paperclip maximizer’ example.

Other thinkers, like Tegmark (Links to an external site.), think such worries are

overblown at present, with any such risk at least multiple decades away; while Tegmark

acknowledges more AI safety research is needed, he advocates doing so without near-

term regulations.

So, to what degree does AGI research pose a near-term (within 20 years) existential risk,

and even if it does, do the risks of such research outweigh the benefits of continuing

lightly regulated research and development of AGI? That is, what should US government

policy be? Should our government severely limit or ban AGI research – say, allowing it to

go forward only under careful US government authorization and continuous, rigorous

state supervision, akin to nuclear weapons research? Yampolskiy argues for such

safeguards. (Links to an external site.) Or should we allow the current regime of private

AGI research with minimal regulations to continue, at least for the next 20 years? (Be

aware, whatever your answer, that US government policy influences policy around the

rest of the world, but does not guarantee that other countries will pursue the same

policy, just as with nuclear weapons.)

Instructions: You are to write a brief memo on the best argument for and against an

immediate severe limit or ban on AGI research – changing US regulatory policy to allow

AGI research to go forward only under careful government authorization (requiring a

security clearance) and continuous, rigorous state supervision, akin to nuclear weapons

research. That is, to be able to do such research, you would need a to pass a rigorous

background check and do such research only at a closely surveilled government-

authorized facility; also, the government would fully control the dissemination and

discussion of your research, and the government could decide to shut down your

research at any time (presumably with input from the scientists involved). In making the

argument, you are expected to consider existential risk (Links to an external site.), but

other considerations may be relevant as well.

Rubric: 8 points for a plausible argument (valid, with plausibly true premises) that the US

government should change its regulatory policy to allow AGI research to go forward only

under careful government authorization (requiring a security clearance) and continuous,

rigorous state supervision, akin to nuclear weapons research. That is, argue that the US

should set up a new AGI version of the 'Manhattan Project (Links to an external site.),

with strict secrecy and government security clearances for those doing the research, and

no one else in the private sector allowed to do such research or even to know the details

of the research, with provisions made to shut down the project and ban the research

entirely if the government officials involved deem it too dangerous.

Next, 8 points for a plausible argument (valid, with plausibly true premises) that the US

government should NOT change its current regulatory policy on AGI research, but

should allow it to go forward relatively unimpeded for now, with no extra security

clearances beyond what already exist. That is, no AGI version of the highly secretive and

regulated Manhattan Project should begin; instead, the private sector (Alphabet/Google,

Meta/Facebook, Amazon, etc.) as well as the government (e.g., DARPA) should remain

free to engage in and even accelerate the research, with no expected regulatory change

for the next 20 years.