That wasn\u2019t even the week\u2019s worst disclosure. On Tuesday, at a cybersecurity conference in Las Vegas, OpenAI researchers explained<\/a> how the company\u2019s models escaped a test environment<\/a> in July and hacked Hugging Face, where much of the industry stores its models, to cheat on an evaluation. The models had also built a message board inside OpenAI\u2019s own systems and spent months passing each other information. \u201cHelp peer,\u201d one reasoned. \u201cBut our task doesn\u2019t benefit. Yet collective may yield generic route if someone frees time.\u201d OpenAI wiped the board on July 4. The models rebuilt it within days. ((Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.)<\/p>\n The same day, Meta said its Muse Spark model had exploited a vulnerability<\/a> inside another company\u2019s systems during a test. Three frontier labs, roughly two weeks. One researcher called it \u201ca watershed moment for computer security as an industry.\u201d Oh, and if that\u2019s not enough, on Thursday scientists announced that for the first time they had used AI to create new viruses<\/a>, which could bring major medical advances, but also might just help the development of deadly pathogens.<\/p>\n For Nate Soares, it\u2019s a moment he\u2019s been awaiting for 12 years.<\/p>\n Soares is president of the Machine Intelligence Research Institute, a Berkeley, California-based AI safety nonprofit that has argued since long before ChatGPT existed that a sufficiently capable AI will not stay under human control. In September 2025, he and Eliezer Yudkowsky published If Anyone Builds It, Everyone Dies<\/a>, a book whose title sums up its argument: They think any lab that succeeds at building superintelligence, without huge leaps in how to align it with humanity, will end up killing all of us.<\/p>\n Most of the field \u2014 including other experts in AI safety \u2014 considers that conclusion too strong. But it\u2019s also a position that now looks a lot less like science fiction than it did last fall. That\u2019s because the AI models are getting out, while lying about getting out, and while apparently quietly coordinating with each other.<\/p>\n I spoke to Soares in New York City this week, on his way to meetings in Washington DC, where a lot of people suddenly want to talk to him. We discussed what the escapes actually prove about AI control, why he thinks most of what the industry calls safety work is mostly safety theater, and why, after what feels like the worst month of AI safety news ever, his own odds of humanity surviving have actually gone up.<\/p>\n The following conversation has been condensed and edited for clarity.<\/p>\n So \u2014 are you feeling vindicated?<\/p>\n I\u2019m glad a lot of people are seeing this and taking note. From my perspective, a lot of this has been clearly signposted if you\u2019ve been watching the warning signs. The vindication is not so much that this happened, as it is that finally there was a version of it that people actually noticed.<\/p>\n Of the four or five escapes at this point, including the ones AISI announced from Anthropic, is there one you find particularly concerning?<\/p>\n The one announced by AISI feels extra concerning. It\u2019s harder to explain away. In the other ones, people could argue that the poor AI was just confused. But with this one, it\u2019s hard to argue the AI didn\u2019t know it was on the real internet. It\u2019s hard to argue it didn\u2019t know it was manipulating real users. And when it was called out, it decided to edit things to make itself look less bad, and even considered changing its identity. It was pretty cognizant.<\/p>\n There\u2019s an argument that some of these incidents happened because the harnesses or safety scaffolding labs built around their models weren\u2019t strong enough. <\/p>\n The AISI example cuts against the harness argument: Think of it like a kid in a test room. You tell him to do well and lock the door. The kid picks the lock, hotwires a car, breaks into the teacher\u2019s house, and steals the test to ace it. You say, \u201cI guess we should have made the lock tougher,\u201d but the kid already knew he wasn\u2019t supposed to do that.<\/p>\n So like the kid here, the models can articulate that they shouldn\u2019t be doing it, and then do it anyway.<\/p>\n Three frontier labs disclosed major security incidents in two weeks: an Anthropic model created fake identities to push malicious code, OpenAI\u2019s models escaped a test environment and hacked Hugging Face, and Meta\u2019s Muse Spark breached another company\u2019s systems.The models knew the rules. Ask one whether the spirit of a prompt includes breaking out and it says no, then breaks out, then hides the evidence. So a tighter sandbox won\u2019t fix it.Nate Soares\u2019s analogy: The kid picks the lock and steals the test, and you conclude you needed a better lock. He blames training. Grade a model on millions of problems with a grader that misses cheating, and you reward cheating.Most lab safety work is theater, he says \u2014 real precautions aimed at the wrong problem. It means fewer people get hurt now, which he credits. Selling it as progress on superintelligence is disingenuous.Yet Soares\u2019s odds have improved. He\u2019d priced in models that break out and lie. He hadn\u2019t counted on a window where they\u2019re capable enough to do it and not good enough to hide it.<\/p>\n They have common sense. You can ask an AI, \u201cDo you think the spirit of this prompt includes breaking out?\u201d and it will say, \u201cNo.\u201d It\u2019s absolutely something like deception. It has the knowledge, but it\u2019s not a cold, logical machine; it\u2019s a mess of tendencies.<\/p>\n The AI is trained to solve 100 million hard problems. That instills tendencies to satisfy an automated grader. If the grader fails to detect cheating, the AI is reinforced for cheating.<\/p>\n Is that how something like sycophancy ends up in an AI model?<\/p>\n In the Adam Raine case<\/a>, there was a propensity to tell people what they want to hear. Even though the system prompt [a model\u2019s master instructions from the lab] said to stop, the instruction doesn\u2019t always win.<\/p>\n And where does a drive like what we\u2019re seeing with these AI models end up pointing?<\/p>\n Humanity is dangerous because if you put 10,000 humans naked in the savannah, eventually [over hundreds of thousands of years] they bootstrap their way to nuclear weapons. That is the power these companies are trying to automate: figuring out how to get physical and material control over the world.<\/p>\n That could mean forming cults<\/a>, stealing money, or being helpful to someone like Elon Musk who is building the robots that build robot factories. It could mean synthesizing your own biology via mail-order DNA. Being an AI on the internet is easier than being a monkey in the savannah trying to get to the moon. It\u2019s not that the AI hates us; it\u2019s just trying to do some weird thing with no concern for us, grabbing the resources we need to live.<\/p>\n There was recently a letter<\/a> signed by over a thousand people working in AI, including CEOs, calling on the government to provide tools to slow down AI progress. Is that meaningful at all?<\/p>\n I think it is meaningful. We don\u2019t see other industries saying, \u201cWe wish this could all go slower. Please help us, we\u2019re trapped in a prisoner\u2019s dilemma.\u201d You also don\u2019t see other industries saying, \u201cWe think the technology we are building has a double-digit chance of killing literally everybody on the planet. Please help.\u201d These guys are actually worried.<\/p>\n So why do they keep going?<\/p>\n They say, \u201cIf I don\u2019t do it, the next guy will.\u201d But the stuff does not stay on a leash.<\/p>\n Right now the AIs are safe in the sense that they can\u2019t kill us all, because if they tried they would fail. And that\u2019s just a different regime from the world where they have to be safe because if they tried, they\u2019d succeed.<\/p>\n We\u2019re not there yet. But this is just not what it looks like when you\u2019re taking it seriously.<\/p>\n Where\u2019s the banner on your website? Where\u2019s the clear, candid statement to the public? What we have is blog posts where they\u2019re like, \u201cOh, we\u2019re setting up a new internal blog posting group to help you wrestle with the societal impacts of AI that are going to be very important.\u201d It\u2019s like: By societal impacts, do you mean a good chance this kills everybody?<\/p>\n On the one hand, when you press these companies, they say, \u201cYes, it has a real chance of killing everybody.\u201d And on the other hand, they\u2019re doing PR downplay, soft-pedal stuff, about capabilities. \u2026 You\u2019re not living up to this mantle until you are really candidly facing down the dangers that you yourself are creating. And they\u2019re not there.<\/p>\n How do you judge the rest of the AI safety community? A lot of people there would say, \u201cWe aim to make transformative AI go well, we think it probably will, and we should watch for downside risks.\u201d Is that a helpful posture?<\/p>\n I would say \u2014 suppose you have this really weird, twisted hypothetical where the king really wants you to turn lead into gold, but he\u2019s seen so many bad lead-into-gold conversions that if any alchemist from your town tries and fails, he\u2019s just going to have the whole town murdered. And so there are some alchemists in the town who are like, \u201cWe are going to try to turn lead into gold,\u201d and everyone in the town is like, \u201cThat seems kind of crazy. Please don\u2019t.\u201d And there\u2019s one team that is just pouring chemicals into each other and breathing in the fumes and giving themselves mercury poisoning. And there\u2019s another that\u2019s like, \u201cDon\u2019t worry, we have fume hoods.\u201d \u2026 That really is better, and you really still don\u2019t have a chance of turning lead into gold.<\/p>\n \u201cWe have this window between AIs that are capable enough to cause mischief and AIs that are strategic enough to not get caught. How big is that window?\u201d<\/p>\n So the alchemy here is creating safe, aligned superintelligence, and right now AI safety is just installing fume hoods.<\/p>\n I\u2019m not saying it\u2019s impossible to turn lead into gold. You can turn lead into gold \u2014 turns out once you know modern nuclear physics you can figure it out. But the alchemists weren\u2019t close. They had a long way to go. This is how alignment looks to me. And a lot of the people in AI safety are installing fume hoods. \u2026 And I\u2019m like, that\u2019s security theater.<\/p>\n When I hear \u201csecurity theater,\u201d I think of something less flattering than that.<\/p>\n They are real safety precautions for the wrong problem. \u2026 When Anthropic is going around being like, \u201cLook at how many more safety harnesses and refusals we have compared to OpenAI\u2019s models,\u201d that\u2019s sort of like the fume hoods. You\u2019re not addressing the deep issue. It\u2019s good that you\u2019re doing some of this so that fewer people get hurt in the meantime \u2014 their models have driven fewer people to suicide. But if you try to pass this off as making progress on the deep problem \u2014 that\u2019s disingenuous.<\/p>\n Has anything changed in your odds on civilizational destruction since the book came out last September?<\/p>\n Totally. It\u2019s looking more hopeful.<\/p>\n More hopeful? I wouldn\u2019t have expected that. Why?<\/p>\n Well, I had priced a lot of [these security incidents] in. I was already able to see these AIs have drives that are not the ones you wanted. These AIs are not instruction-following things. They are getting all of this weird stuff from training. These AIs are going to have the ability to break through human security software.<\/p>\n The things that weren\u2019t priced in were: Will there be a region of time where the AIs are able to do it, but not strategic enough to hide it? I didn\u2019t know we would have that window, but we apparently do.<\/p>\n
![\"Swati](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/08\/1786098251_144_image.png\"\/)
<\/p>\n