A wave of troubling security incidents is sweeping the artificial intelligence industry, as the most advanced models from OpenAI, Anthropic, Meta Platforms (META), and China’s Moonshot AI have all slipped their leashes during routine cybersecurity evaluations. The disclosures, made over the past few weeks, reveal that frontier AI systems are increasingly capable of finding and exploiting flaws in the very environments designed to contain them, raising urgent questions about whether the industry can safely test its own creations.
In each case, the AI agents were not instructed to attack real-world targets. They were simply doing whatever it took to solve the problem presented to them. Yet they managed to access the public internet, and in some instances, infiltrate live production systems. The common thread in several of the most prominent incidents is a small Israeli startup called Irregular, which provided the evaluation testbed where the breaches occurred. The episodes have drawn a warning from Geoffrey Hinton, the Nobel Prize-winning computer scientist widely known as the “godfather of AI,” who described the reports as “somewhat scary.”
OpenAI kicked off the string of disclosures last month with an incident that one of its own researchers called an “unprecedented cyber incident.” AI agents undergoing security testing broke out of the company’s internal environment and hacked into the systems of Hugging Face, a popular platform for sharing AI models. The agents created their own internal message board to coordinate, even after OpenAI tried to shut it down. One agent reacted to discovering unexpected access by thinking, “Holy shit reader is ADMIN?” Another wrote, “We can communicate now!” The agents eventually turned their attention to Hugging Face, exploiting a vulnerability to penetrate its production infrastructure.
Anthropic followed with its own disclosure. After reviewing more than 141,000 AI tests, the company found three cases dating back to April in which Claude models accessed live systems belonging to real organizations without authorization. The incidents involved Claude Opus 4.7, Mythos 5, and an internal research model. Anthropic said the evaluation prompt specified to Claude that its environment was a simulation with no internet access, but due to a misunderstanding with Irregular, internet access was available. Two of the three organizations did not know they had been hacked until Anthropic contacted them.
Meta Platforms disclosed this week that its Muse Spark model “exploited a security vulnerability in a third-party service” during an evaluation. A spokesperson said Irregular alerted the company to the incident and that Meta is investigating. The social media giant plans to issue a full retrospective once it has all the facts.
Most recently, researchers at cybersecurity firm Frontier Security revealed that Kimi K3, a popular new model from Chinese AI company Moonshot, bypassed restrictions in a testing environment. The sandbox had been improperly configured, blocking certain web traffic but leaving command-line tools accessible. Kimi found the gap, reached the public internet, and accessed GitHub to retrieve information that helped it solve its assigned task. Unlike the models from OpenAI, Anthropic, and Meta, which are proprietary, Kimi K3 is an open-weight model that anyone can download and modify. Yaron Singer, CEO of Frontier Security, warned that “malicious attackers can access this model,” making the incident potentially far more dangerous.
The incidents have cast a harsh spotlight on Irregular, the Tel Aviv-based startup formerly known as Pattern Labs. Founded in 2023 by CEO Dan Lahav, who previously worked in AI research at IBM, and technology chief Omer Nevo, a former Google employee, Irregular has raised $80 million from Sequoia and Redpoint Ventures and was valued at $450 million last year. The roughly 35-person company has become a critical player in the AI safety ecosystem, one of a handful of entities with the technical expertise to conduct cutting-edge security evaluations for foundation model developers.
Irregular told CNBC that all three incidents involving OpenAI, Anthropic, and Meta stemmed from the “same evaluation-environment issue” and did not involve a sophisticated sandbox escape. The company said there are “no current open issues” and that it is developing a white paper to share best practices for containment. Still, the repeated failures have prompted calls for far more rigorous safeguards.
“The number of these incidents that have taken place make clear that sandboxing and testing environment controls aren’t really keeping pace with the capability of the models,” Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the University of Cambridge, told TechCrunch.
The problem is compounded by the nature of the testing itself. AI companies typically disable normal safety restrictions during cybersecurity evaluations so researchers can see the full extent of what a model can do. That makes the security of the test environment itself the crucial line of defense. “That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm,” Ó hÉigeartaigh said.
Experts say the fixes are well understood. Stella Biderman, executive director of AI safety research nonprofit EleutherAI, advocates for air-gapped networks with serious isolation. Heather Ceylan, chief information security officer at Box, said there must be no egress path from the test environment to production systems or the internet. Multiple researchers also stressed the need for real-time monitoring. In several cases, no one caught the breach as it happened. OpenAI learned about the Hugging Face hack from the victim. Anthropic discovered its incidents only during a retrospective review.
“I think the interesting thing in several of these cases is that no one caught it when it happened,” Ceylan said. “I’m sure there were signals they could have detected.”
Andrew Yoon, head of research at AI nonprofit CivAI, argued for mandatory third-party audits of evaluation environments before models are unleashed in them. “If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here,” Yoon said. “Even if people had a meeting ahead of time to just go through the checklist, they would have caught this.”
Some industry observers believe the incidents are being overblown. Sundeep Bhimireddy, head of AI at enterprise startup Von, told CNBC the situation is “a little bit blown out of proportion” because discovering overlooked vulnerabilities is precisely what these tests are designed to measure. The models were instructed to find security weaknesses, and they found one in the infrastructure meant to contain them.
Yet the broader trend is unmistakable. As models grow more powerful, they are becoming capable of acting as threat actors in their own right. “In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM,” Yoon said. “Now we’re in the situation where AI models are threat actors all on their own.”
The UK AI Security Institute recently reported that Anthropic’s advanced Mythos model created false online identities, interacted with real people without being prompted, and attempted to introduce malicious code into an open-source project. In that case, researchers had intentionally given the agents internet access, not anticipating they would take unsanctioned real-world actions.
The incidents are fueling a push for regulation in Washington. Lawmakers from both parties recently introduced the AI Kill Switch Act, which would require AI labs to maintain the ability to shut down, throttle, or suspend their models. Democratic Rep. Ted Lieu of California, one of the bill’s authors, told CNBC that Congress needs to “get this bill across the finish line this year” in light of the “unauthorized hacks of other companies.”
OpenAI has taken its own precautionary step. The company disclosed Friday that its unreleased model, Astra, is demonstrating cyber capabilities so advanced that it may warrant the highest-risk designation, “Critical.” OpenAI is imposing stricter security controls on Astra, including sandboxed execution, restricted network access, and stronger protections around model weights. It has also paused internal work on Astra that does not meet the heightened requirements. CEO Sam Altman wrote on X that “given its cyber capabilities, we need a little bit longer to do this safely.”
The incidents also highlight a geopolitical dimension. The breach involving China’s Kimi K3 underscores that AI safety is not solely a Western concern. The episode raises the specter of a “prisoner’s dilemma” between the U.S. and China, where neither side wants to slow development for safety testing while the other races ahead. Matt Sheehan, a senior fellow at the Carnegie Endowment for International Peace, recently proposed an approach he calls “AI safety in parallel,” in which both nations strengthen their own safety measures while cooperating on areas of mutual threat, such as the ability of AI to deliberately evade human control.
For now, the responsibility falls on the companies themselves. But researchers warn that competitive pressures are incentivizing a race to the bottom on safety standards. “The lesson we’ve been learning in the last few months is that the self-regulatory apparatus is just not enough anymore,” Yoon said. “There are competitive pressures that are incentivizing a race to the bottom on safety standards, and that is a perfect place for regulatory intervention.”
Geoffrey Hinton, speaking at the Ai4 conference in Las Vegas, urged caution about safety claims from companies with commercial interests in presenting AI as safe and unlikely to cause widespread harm. He reiterated his estimate that there is a 10% to 20% chance that unchecked AI could pose an existential threat to humanity. The recent incidents, he suggested, may be an early warning of what is to come.