In a small room in San Diego last week, a man in a black leather jacket explained to me how to save the world from destruction by AI. Max Tegmark, a notable figure in the AI-safety movement, believes that \u201cartificial general intelligence,\u201d or AGI, could precipitate the end of human life. I was in town for NeurIPS, one of the largest AI-research conferences, and Tegmark had invited me, along with five other journalists, to a briefing on an AI-safety index that he would release the next day. No company scored better than a C+.<\/p>\n
The threat of technological superintelligence is the stuff of science fiction, yet it has become a topic of serious discussion in the past few years. Despite the lack of clear definition\u2014even OpenAI\u2019s CEO, Sam Altman, has called<\/a> AGI a \u201cweakly defined term\u201d\u2014the idea that powerful AI contains an inherent threat to humanity has gained acceptance among respected cultural critics.<\/p>\n Granted, generative AI is a powerful technology that has already had a massive impact on our work and culture. But superintelligence has become one of several questionable narratives promoted by the AI industry, along with the ideas that AI learns like a human, that it has \u201cemergent\u201d capabilities, that \u201creasoning models\u201d are actually reasoning, and that the technology will eventually improve itself.<\/p>\n I traveled to NeurIPS, held at the waterfront fortress that is the San Diego Convention Center, partly to understand how seriously these narratives are taken within the AI industry. Do AGI aspirations guide research and product development? When I asked Tegmark about this, he told me that the major AI companies were sincerely trying to build AGI, but his reasoning was unconvincing. \u201cI know their founders,\u201d he said. \u201cAnd they\u2019ve said so publicly.\u201d<\/p>\n Parallel to the growth of fear and excitement about AI in the past decade, NeurIPS attendance has exploded, increasing from approximately 3,850 conference-goers in 2015 to 24,500 this year, according to organizers. The conference center\u2019s three main rooms each have the square footage of multiple blimp hangars. Speakers addressed audiences of thousands. \u201cI do feel we\u2019re on a quest, and a quest should be for the holy grail,\u201d Rich Sutton, the legendary computer scientist, proclaimed in a talk about superintelligence.<\/p>\n The conference\u2019s corporate sponsors had booths to promote their accomplishments and impress attendees with their R&D visions. There were companies you\u2019ve heard of, such as Google, Meta, Apple, Amazon, Microsoft, ByteDance, and Tesla, and ones you probably haven\u2019t, such as Runpod, Poolside, and Ollama. One company, Lambda, was advertising itself as the \u201cSuperintelligence Cloud.\u201d A few of the big dogs were conspicuously absent from the exhibitor hall, namely OpenAI, Anthropic, and xAI. Consensus among the researchers I spoke with is that the cachet of these companies is already so great that setting up a booth would be pointless.<\/p>\n The conference is a primary battleground in AI\u2019s talent war<\/a>. Much of the recruiting effort happens outside the conference center itself, at semisecret, invitation-only events in downtown San Diego. These events captured the ever-growing opulence of the industry. In a lounge hosted by the Laude Institute, an AI-development support group, a grad student told me about starting salaries at various AI companies of \u201ca million, a million five,\u201d of which a large portion was equity. The lounge was designed in the style of a VIP lounge at a music festival. It was, in fact, located at the top of the Hard Rock Hotel.<\/p>\n The place to be, if you could get in, was the party hosted by Cohere, a Canadian company that builds large language models. (Cohere is being sued for copyright and trademark infringement by a group of news publishers, including The Atlantic.) The party was held on the USS Midway, an aircraft carrier used in Operation Desert Storm, which is now docked in the San Diego harbor. The purpose, according to the event\u2019s sign-up page, was \u201cto celebrate AI\u2019s potential to connect our world.\u201d<\/p>\n With the help of a researcher friend, I secured an invite to a mixer hosted by the Mohamed bin Zayed University of Artificial Intelligence, the world\u2019s first AI-focused university, named for the current UAE president. Earlier this year, MBZUAI established the Institute for Foundation Models, a research group in Silicon Valley. The event, held at a steak house, had an open buffet with oysters, king prawns, ceviche, and other treats. Upstairs, Meta was hosting its own mixer. According to rumor, some of the researchers downstairs were Meta employees hoping to be poached by the Institute for Foundation Models, which supposedly offered more enticing compensation packages.<\/p>\n Of 5,630 papers presented in the poster sessions at NeurIPS, only two mention AGI in their title. An informal survey<\/a> of 115 researchers at the conference suggested that more than a quarter didn\u2019t even know what AGI stands for. At the same time, the idea of AGI, and its accompanying prestige, seemed at least partly responsible for the buffet. The amenities I encountered certainly weren\u2019t paid for by chatbot profits. OpenAI, for instance, reportedly expects<\/a> its massive losses to continue until 2030. How much longer can the industry keep the ceviche coming? And what will happen to the economy, which many believe is propped up by the AI industry, when it stops?<\/p>\n In one of the keynote speeches, the sociologist and writer Zeynep Tufekci warned researchers that the idea of superintelligence was preventing them from understanding the technology they were building. The talk<\/a>, titled \u201cAre We Having the Wrong Nightmares About AI?,\u201d mentioned several dangers posed by AI chatbots, including widespread addiction to chatbots and the undermining of methods for establishing truth. After Tufekci gave her talk, the first audience member to ask a question appeared annoyed. \u201cHave you been following recent research?\u201d the man asked. \u201cBecause that\u2019s the exact problems we\u2019re trying to fix. So we know of these concerns.\u201d Tufekci responded, \u201cI don\u2019t really see these discussions. I keep seeing people discuss mass unemployment versus human extinction.\u201d<\/p>\n It struck me that both might be correct: that many AI developers are thinking about the technology\u2019s most tangible problems while public conversations about AI\u2014including those among the most prominent developers themselves\u2014are dominated by imagined ones. Even the conference\u2019s name contained a contradiction: The name \u201cNeurIPS\u201d is short for \u201cNeural Information Processing Systems,\u201d but artificial neural networks were conceived in the 1940s by a logician-and-neurophysiologist duo who wildly underestimated the complexity of biological neurons and overstated their similarity to a digital computer. Regardless, a central feature of AI\u2019s culture is an obsession with the idea that a computer is a mind. Anthropic<\/a> and OpenAI<\/a> have published reports with language about chatbots being, respectively, \u201cunfaithful\u201d and \u201cdishonest.\u201d In the AI discourse, science fiction often defeats science.<\/p>\n On the roof of the Hard Rock Hotel, I attended an interview<\/a> with Yoshua Bengio, one of the three \u201cgodfathers\u201d of AI. Bengio, a co-inventor of an algorithm that makes ChatGPT possible, recently started a nonprofit called LawZero to encourage the development of AI that is \u201csafe by design<\/a>.\u201d He took the nonprofit\u2019s name from a law featured in several Isaac Asimov stories that states that a robot should not allow humans to be harmed. Bengio was concerned that, in a possible dystopian future, AIs might deceive their creators and that \u201cthose who will have very powerful AIs could misuse it for political advantage, in terms of influencing public opinion.\u201d<\/p>\n I looked around to see if anyone else was troubled by the disconnect. Bengio did not mention how fake videos are already affecting public discourse<\/a>. Neither did he meaningfully address the burgeoning chatbot mental-health crisis<\/a>, or the pillaging of the arts and humanities<\/a>. The catastrophic harms, in his view, are \u201cthree to 10 or 20 years\u201d away. We still have time \u201cto figure it out, technically.\u201d<\/p>\n