Illustration: Petra P\u00e9terffy\/The Guardian<\/p>\n
\u201cTechnology happens because it is possible,\u201d OpenAI CEO, Sam Altman, told<\/a> the New York Times in 2019, consciously paraphrasing Robert Oppenheimer, the father of the atomic bomb.<\/p>\n Altman captures a Silicon Valley mantra: technology marches forward inexorably.<\/p>\n Another widespread techie conviction is that the first human-level AI \u2013 also known as artificial general intelligence (AGI) \u2013 will lead to one of two futures: a post-scarcity techno-utopia<\/a> or the annihilation of humanity<\/a>.<\/p>\n For countless other species<\/a>, the arrival of humans spelled<\/a> doom. We weren\u2019t tougher, faster or stronger \u2013 just smarter and better coordinated. In many cases, extinction was an accidental byproduct of some other goal we had. A true AGI would amount to creating a new species<\/a>, which might quickly outsmart<\/a> or outnumber<\/a> us. It could see humanity as a minor obstacle, like an anthill<\/a> in the way of a planned hydroelectric dam, or a resource to exploit<\/a>, like the billions of animals confined in factory farms.<\/p>\n Altman, along with the heads of the other top AI labs, believes that AI-driven extinction is a real possibility<\/a> (joining hundreds of leading AI researchers and prominent figures).<\/p>\n Given all this, it\u2019s natural to ask: should we really try to build a technology that may kill us all if it goes wrong?<\/p>\n Perhaps the most common reply says: AGI is inevitable. It\u2019s just too useful not to build. After all, AGI would be the ultimate technology \u2013 what a colleague of Alan Turing called<\/a> \u201cthe last invention that man need ever make\u201d. Besides, the reasoning goes within AI labs, if we don\u2019t, someone else will do it \u2013 less responsibly, of course.<\/p>\n A new ideology out of Silicon Valley, effective accelerationism<\/a> (e\/acc), claims<\/a> that AGI\u2019s inevitability is a consequence of the second law of thermodynamics and that its engine is \u201ctechnocapital\u201d. The e\/acc manifesto<\/a> asserts: \u201cThis engine cannot be stopped. The ratchet of progress only ever turns in one direction. Going back is not an option.\u201d<\/p>\n For Altman<\/a> and e\/accs, technology takes on a mystical quality \u2013 the march of invention is treated as a fact of nature. But it\u2019s not. Technology is the product of deliberate human choices, motivated by myriad powerful forces. We have the agency to shape those forces, and history shows that we\u2019ve done it before.<\/p>\n No technology is inevitable, not even something as tempting as AGI.<\/p>\n Some AI worriers like to point out<\/a> the times<\/a> humanity resisted and restrained valuable technologies.<\/p>\n Fearing novel risks, biologists initially banned<\/a> and then successfully regulated<\/a> experiments on recombinant DNA in the 1970s.<\/p>\n No human has been reproduced via cloning<\/a>, even though it\u2019s been technically possible<\/a> for over a decade, and the only scientist to genetically engineer humans was imprisoned<\/a> for his efforts.<\/p>\n Nuclear power can provide<\/a> consistent, carbon-free energy, but vivid fears of catastrophe have motivated<\/a> stifling regulations and outright bans<\/a>.<\/p>\n And if Altman were more familiar<\/a> with the history of the Manhattan Project, he might realize that the creation of nuclear weapons in 1945 was actually<\/a> a highly contingent and unlikely outcome, motivated<\/a> by a mistaken belief that the Germans were ahead in a \u201crace\u201d for the bomb. Philip Zelikow, the historian who led the 9\/11 Commission, said<\/a>: \u201cI think had the United States not built an atomic bomb during the Second World War, it\u2019s actually not clear to me when or possibly even if an atomic bomb ever is built.\u201d<\/p>\n It\u2019s now hard to imagine a world without nuclear weapons. But in a little-known episode, then president Ronald Reagan and Soviet leader Mikhail Gorbachev nearly agreed<\/a> to ditch all their bombs (a misunderstanding over the \u201cStar Wars<\/a>\u201d satellite defense system dashed these hopes). Even though the dream of full disarmament remains just that, nuke counts are less than<\/a> 20% of their 1986 peak, thanks largely to international agreements<\/a>.<\/p>\n These choices weren\u2019t made in a vacuum. Reagan was a staunch opponent<\/a> of disarmament before the millions-strong<\/a> Nuclear Freeze movement got to him. In 1983, he commented<\/a> to his secretary of state : \u201cIf things get hotter and hotter and arms control remains an issue, maybe I should go see [Soviet leader Yuri] Andropov and propose eliminating all nuclear weapons.\u201d<\/p>\n There are extremely strong economic incentives to keep burning fossil fuels, but climate advocacy has pried open<\/a> the Overton window and significantly accelerated our decarbonization efforts.<\/p>\n In April 2019, the young climate group Extinction Rebellion<\/a> (XR) brought<\/a> London to a halt, demanding the UK target net-zero carbon emissions by 2025. Their controversial civil disobedience prompted parliament to declare<\/a> a climate emergency and the Labour party to adopt a 2030 target<\/a> to decarbonize the UK\u2019s electricity production<\/a>.<\/p>\n The Sierra Club\u2019s Beyond Coal<\/a> campaign was lesser-known but wildly effective<\/a>. In just its first five years, the campaign helped shutter more than one-third of US coal plants. Thanks<\/a> primarily<\/a> to its move from coal, US per capita carbon emissions are now<\/a> lower than they were in 1913.<\/p>\n In many ways, the challenge of regulating efforts to build AGI is much smaller than that of decarbonizing. Eighty-two percent<\/a> of global energy production comes from fossil fuels. Energy is what makes civilization work, but we\u2019re not dependent on a hypothetical AGI to make the world go round.<\/p>\n Further, slowing and guiding the development of future systems doesn\u2019t mean we\u2019d need to stop using existing systems or developing specialist AIs to tackle important problems in medicine, climate and elsewhere.<\/p>\n It\u2019s obvious why so many capitalists are AI enthusiasts: they foresee a technology that can achieve their long-time dream of cutting workers out of the loop (and the balance sheet).<\/p>\n But governments are not profit maximizers. Sure, they care about economic growth, but they also care about things like employment, social stability, market concentration, and, occasionally, democracy.<\/p>\n It\u2019s far less clear how AGI would affect these domains overall. Governments aren\u2019t prepared for a world where most people are technologically unemployed.<\/p>\n Capitalists often get what they want, particularly in recent decades, and the boundless pursuit of profit may undermine any regulatory effort to slow the speed of AI development. But capitalists don\u2019t always get what they want.<\/p>\n At a bar in San Francisco in February, a longtime OpenAI<\/a> safety researcher pronounced to a group that the e\/accs shouldn\u2019t be worried about the \u201cextreme\u201d AI safety people, because they\u2019ll never have power. The boosters should actually be afraid of AOC and Senator Josh Hawley because they \u201ccan really fuck things up for you\u201d.<\/p>\n Assuming humans stick around for many millennia, there\u2019s no way to know we won\u2019t eventually build AGI. But this isn\u2019t really what the inevitabilists are saying. Instead, the message tends to be: AGI is imminent. Resistance is futile.<\/p>\n But whether we build AGI in five, 20 or 100 years really matters. And the timeline is far more in our control than the boosters will admit. Deep down, I suspect many of them realize this, which is why they spend so much effort trying to convince others that there\u2019s no point in trying. Besides, if you think AGI is inevitable, why bother convincing anybody?<\/p>\n We actually had<\/a> the computing power required to train GPT-2 more than a decade before OpenAI actually did it, but people didn\u2019t know whether it was worth doing.<\/p>\n But right now, the top AI labs are locked in such a fierce race that they aren\u2019t implementing<\/a> all the precautions that even their own safety teams want. (One OpenAI employee announced<\/a> recently that he quit \u201cdue to losing confidence that it would behave responsibly around the time of AGI\u201d.) There\u2019s a \u201csafety tax\u201d that labs can\u2019t afford to pay if they hope to stay competitive; testing slows<\/a> product releases and consumes company resources.<\/p>\n Governments, on the other hand, aren\u2019t subject to the same financial pressures.<\/p>\n An inevitabilist tech entrepreneur recently said<\/a> regulating AI development is impossible \u201cunless you control every line of written code\u201d. That might be true if anyone could spin up an AGI on their laptop. But it turns out that building advanced, general AI models requires<\/a> enormous arrays of supercomputers, with chips produced by an absurdly monopolistic<\/a> industry. Because of this, many AI safety advocates see \u201ccompute governance<\/a>\u201d as a promising approach. Governments could compel cloud computing providers to halt next generation training runs that don\u2019t comply with established guardrails<\/a>. Far from locking out upstarts or requiring Orwellian levels of surveillance, thresholds could be chosen to only affect players who can afford to spend more than $100m<\/a> on a single training run.<\/p>\n Governments do have to worry about international competition and the risk of unilateral disarmament, so to speak. But international treaties<\/a> can be negotiated to widely share the benefits from cutting-edge AI systems while ensuring that labs aren\u2019t blindly scaling up systems they don\u2019t understand<\/a>.<\/p>\n And while the world may feel fractious, rival nations have cooperated to surprising degrees.<\/p>\n The Montreal Protocol fixed<\/a> the ozone layer by banning chlorofluorocarbons. Most of the world has agreed to ethically motivated bans on militarily useful weapons, such as biological<\/a> and chemical weapons<\/a>, blinding laser weapons<\/a>, and \u201cweather warfare<\/a>\u201d.<\/p>\n In the 1960s and 70s, many analysts feared<\/a> that every country that could build nukes, would. But most<\/a> of the world\u2019s roughly three-dozen nuclear programs were abandoned<\/a>. This wasn\u2019t the result of happenstance, but rather the creation of a global nonproliferation norm through deliberate statecraft, like the 1968 Non-Proliferation Treaty<\/a>.<\/p>\n On the few occasions when Americans were asked if they wanted superhuman AI, large<\/a> majorities<\/a> said \u201cno\u201d. Opposition to AI has grown<\/a> as the technology has become more prevalent. When people argue that AGI is inevitable, what they\u2019re really saying is that the popular will shouldn\u2019t matter. The boosters see the masses as provincial neo-Luddites who don\u2019t know what\u2019s good for them. That\u2019s why inevitability holds such rhetorical allure for them; it lets them avoid making their real argument, which they know is a loser in the court of public opinion.<\/p>\n The draw of AGI is strong. But the risks involved are potentially civilization-ending. A civilization-scale effort is needed to compel the necessary powers to resist it.<\/p>\n Technology happens because people make it happen. We can choose otherwise.<\/p>\n