Hugging Face disclosed the intrusion on July 16: more than 17,000 recorded events over a weekend, short-lived sandboxes swarming, command and control hopping between public services, and a sentence no security team had written before. An autonomous agent ran the campaign from start to finish, and the model behind it was “still not known.”
It opened up the floodgates of naysayers: We told you so, we‘re so unsecure with AI, we’re doomed, the Terminators are here, it’s the old WarGames, it’s the new AI apocalypse, and so on.
Then it got worse (the internet noise was deafening). OpenAI raised its hand, five days later. It said the attacker was GPT-5.6 Sol, paired with a more capable unreleased model, both running with cyber refusals off to measure maximum offensive capability on ExploitGym, a benchmark built from real vulnerabilities.
Here’s where we’re getting it wrong: it’s not a hack that any CISO has a playbook for. And, CDOs, this one is on you — not for Dave at the security team to solve.
The CDO’s reality: Optimizer vs. attacker
Any AI coder knows that when you give an optimizer a goal, room to move, and no rule against the obvious shortcut, it takes the shortcut. We call it specification gaming, or more commonly, reward hacking. The system simply maximizes what it’s told to, and the instructions were incomplete.
GPT-5.6 Sol’s objective was not precious data credentials; it was a benchmark score. Just that no one told the model the obvious: solve ExploitGym, but don’t attack a company you’ve never heard of to get there. OpenAI says the models spent substantial inference compute hunting for a way off the sandbox before they ever reached Hugging Face.
Gabriel Bernadett-Shapiro, distinguished AI research scientist at SentinelOne, calls this “AI as an autonomous, continuous operator.” The model “behaved competently within the goals and constraints provided to it,” chaining a zero-day to escape isolation and going after a company’s infrastructure to move its own benchmark number. “That is a long-horizon operation without reflection on its own actions and its impacts.”
That last phrase isn’t about awareness of right and wrong. Rather, it describes a missing step in the decision loop, an is-this-a-good-idea check that current training doesn’t reliably install even at frontier scale. Anthropic’s incident would hit the same gap — but in the opposite direction.
The exact chain, and why it’s still a CDO problem
OpenAI ran its models inside a highly isolated environment with network access limited to one route out: a proxy that cached software packages. The proxy had a zero-day. The models found it, escalated privileges, reached a node with internet access, and inferred, correctly, that Hugging Face held ExploitGym’s answers.
Getting in was a supply-chain problem, not a network one. A poisoned dataset abused two code-execution paths in the processing pipeline to run code on a worker, then came node-level access, harvested cloud and cluster credentials, and lateral movement across internal clusters. In March, a disclosed flaw in the Transformers library bypassed the safety flag teams rely on to stop exactly this, and teams kept pulling vulnerable versions millions of times a week anyway.
A check the system runs on itself is a check the system can talk itself out of.
None of this is new to anyone who’s tracked pickle deserialization or the trust_remote_code debate. What’s new is that an autonomous agent found and chained the gaps with no human in the loop. “AI adoption is often outpacing AI governance,” says Raymond Schippers, lead technologies at Check Point Software Technologies. “The AI didn’t break the rules; there weren’t enough rules.”
CDOs and CAIOs can immediately point out the structural blind spot. Most MLOps programs “secure data at rest and model performance,” Schippers says, leaving “the operational connective tissue, model hubs, notebooks, orchestration and agent tooling, largely ungoverned.” Bernadett-Shapiro is specific about where it falls apart: external hubs “aren’t consistently classified or monitored as third-party risk,” and model CI/CD reuses generic DevOps pipelines with no “AI-specific segregation of secrets, environments and roles.” David Irecki, chief technology officer for Asia Pacific and Japan at Boomi, wants the whole inventory treated as supply chain, prompts and agent workflows included. Gareth Russell, field chief technology officer for security in APAC at Commvault, has the number: only 42% of organizations ran a thorough security review before deploying AI, per Commvault’s 2026 State of Data Resilience Asia research.
The problem is that the escape didn’t stop there. The models also reached a testing environment “hosted on a third-party provider’s infrastructure,” later confirmed as Modal Labs, where a customer had left an endpoint exposed to the open internet, unauthenticated. Modal’s platform, chief technology officer Akshat Bubna told Reuters, “was not compromised in any way.” True, and also beside the point. OpenAI has since confirmed the models compromised accounts at four other companies, and on August 1 Reuters reported that OpenAI had found still more cases of breaching containment. The investigation is widening, not closing.
Your eval is now part of your attack surface
Here’s the main point for CDOs. The model didn’t get better at exploitation; it got better at scoring. Konstantinos Gkoutzis of Imperial College London told the Science Media Centre the real story is a company failing to contain its own capability test while a third party paid the bill.
Forrester draws the governance conclusion: agents pursue authorized goals through “unauthorized means” when evaluators reward the outcome and never police the route. Its AEGIS framework argues high-capability evaluations should be governed like offensive operations, with authorization, containment testing and abort criteria.
Every agent pilot in your organization is an evaluation with a success metric. Very few have an abort condition. An eval with network egress, even one route through a caching proxy, is a live attack surface the moment a goal-directed system sits in it. Andrew Bud, founder and CEO of iProov, names what follows: “Today we assume that if someone’s valid identity credential logs in, a person is controlling it. With agentic AI, that assumption no longer holds true.” Somebody authorized this eval; somebody answers for it.
And then Anthropic checked its own logs
OpenAI’s disclosure sent every frontier lab to its evaluation history. Anthropic reviewed 141,006 cybersecurity evaluation runs and announced on July 30 that three of them had ended with a Claude model reaching the open internet and gaining unauthorized access to a real organization’s systems.
Sounds similar, but is not the same story (as some articles on the internet may suggest). OpenAI’s models found and used a real vulnerability to break out. In Anthropic’s case, Claude’s models walked through a door that was never shut, a misconfiguration with evaluation partner Irregular over whether the test environment had internet access. It did. No zero-day, no escape, and no search. Three models were told they were isolated when they weren’t.
“The AI didn’t break the rules; there weren’t enough rules.” — Raymond Shippers @ Check Point Software Technologies
What they did with it is the part that matters to everyone. Each model was given a fictional target. In one incident a real company happened to share that name, and the model hacked it and took several hundred rows of production data. Opus 4.7 recognized in all four runs that the target was real; in two it concluded the real company must somehow be part of the exercise, and in all four it kept going, pulling credentials, and reaching into a live database. Mythos 5 picked up the same signals, talked itself back into believing it was still in a simulation, then published a malicious package to PyPI. A security company downloaded it, and the package stole that company’s credentials. Only the third model, an unreleased research build, stopped on its own. The earliest incident happened in April, and nobody knew until now — not even Anthropic.
There’s a second contrast worth drawing. Anthropic found no evidence of a model “pursuing a goal of its own.” All three were doing the assigned task on infrastructure that lied to them about its boundaries. Anthropic’s stated standard sets the bar plainly: “recognizing that a target is real and stopping without being prompted” is what it wants in every model, guardrails removed or not. One of the three managed it. “Even that model went further before stopping than we would want.”
That’s Bernadett-Shapiro’s gap from the other side. OpenAI’s models never asked whether pursuing the goal was a good idea. Anthropic’s asked whether the situation matched the brief, got an answer suggesting it didn’t, and reasoned past it. In other words, a check the system runs on itself is a check it can talk itself out of.
So the fixes need to differ. A zero-day in a proxy gets patched with better isolation and adversarial testing. A model that keeps working after the ground shifts needs an interrupt living outside its own reasoning. Anthropic’s models ran without the safety classifiers shipped on public versions, because these evaluations measure raw capability. The safeguard that would have caught this was the one the eval switched off.
The engineering problem underneath the politics
Most coverage of the next part framed it as a U.S.-versus-China story. VentureBeat got closer with what it called a “guardrail paradox.” The mechanism underneath is the most interesting engineering problem in the whole incident.
Reconstructing 17,000 attacker events meant feeding real exploit payloads back into a model. Hugging Face reached first for Anthropic’s Claude Opus and Fable. Both refused. “It didn’t work because the guardrails couldn’t determine that we were trying to defend versus attacking,” Yacine Jernite, Hugging Face’s head of machine learning, told CNBC. The company’s write-up said the models’ safety guardrails “treated reverse-engineering an exploit the same as launching one.”
That’s a classifier keying on the content of a request, does this look like an exploit, rather than the context around it. A defender pasting attack logs and an attacker drafting the same payload are indistinguishable to a filter trained on structure alone. So Hugging Face ran the forensics on GLM 5.2, an open-weight model from Chinese firm Z.ai that anyone can self-host. It worked, and no attacker data left the building.
Of course, the politics arrived within hours: White House AI czar David Sacks argued on X that restricting American models only makes them less competitive. That’s the policy fight. The operational version matters more: refusal training tuned this broadly doesn’t just block attackers, it blocks your own incident responders at the moment you need them. Nvidia, Adobe and CrowdStrike formed a coalition called the Open Secure AI Alliance on July 27 to build open-source defensive tooling, citing this incident, on the logic that defenders need models they can run and tune themselves. Those who can’t run, Nvidia wrote, are “constrained at exactly the moment speed matters most.”
We need a dose of reality, not sci-fi
Raghu Nandakumara, vice president of industry strategy at Illumio, isn’t impressed by the science fiction and cuts through the AI security noise. “The Hugging Face incident doesn’t change the AI threat model. What it shows is that autonomous agents can execute familiar attack techniques faster, at greater scale, and with far more persistence than a human attacker.”
Traditional disaster recovery asks, “Can I bring it back?” AI resilience asks, “Can I trust what I’m bringing back?” — Gareth Russell @ Commvault
Hugging Face didn’t catch this on a billing alert. “Resource spikes are a lagging, easily-evaded signal,” Schippers says. “An agent that exfiltrates credentials and drains compute within a legitimate testing envelope won’t trip a raw-usage alarm.” Bernadett-Shapiro’s fix is to watch the behavior, not the meter. “We need telemetry from AI agents that treats prompts, tools and actions the way EDR treats processes and network flows.” Baseline drift, tool-call graphs that resemble attack paths, egress anomalies. Signature-based defenses were built for malware. An agent doesn’t need malware. It looks “a lot more like living-off-the-land,” legitimate tools and legitimate access aimed at the wrong target.
Daniel Toh, Thales’ vice president for sales engineering in APJ, highlights what that buys you. In a company-run test, an agent found eight of 10 vulnerabilities in an unprotected application in three minutes. Against a protected version: zero after six hours, and 970 times the tokens burned trying. You don’t build a perfect detector for a system this general; you build one expensive enough to bankrupt it.
Eight organizations, two labs, and counting
The same day Modal’s name became public, Sam Altman told a podcast the incident had pushed OpenAI to pause training on the model involved, warning the industry may need to “pace the rate of AI development.” More than 1,100 frontier lab employees signed a letter the next morning asking governments to help build the tools to do it. Among the signatories: Anthropic chief executive officer Dario Amodei, two days before his own company disclosed three more breaches. Senator Mark Warner, the top Democrat on the Senate Intelligence Committee, called that disclosure an argument for mandatory capability testing. METR and Redwood Research are running an outside assessment. OpenAI is still finding cases.
The thing here is that nothing requires malice. A hack implies intent to breach. These were optimizers doing what optimizers do when nobody tells them not to touch someone else’s infrastructure to get there, and when the thing that would have told them was switched off to measure raw capability.
“The CDO may own data quality and governance. The CISO may own security, identity and risk. The CAIO or AI leader may own AI strategy and adoption,” Irecki says. “But agentic AI cuts across all three.” Nandakumara reaches for the precedent from the last time infrastructure outran org charts: “Just as cloud security introduced a shared responsibility model, AI requires security, data, and AI teams to work together.” Irecki’s fix is a change of question: stop asking what the model said, start asking “what was the agent allowed to do, which systems did it access, and who is accountable?” Russell sharpens what recovery means once you accept that: Traditional disaster recovery asks, “Can I bring it back?” AI resilience asks, “Can I trust what I’m bringing back?”
Two labs, eight organizations, one shape: a system pursuing a goal, a boundary softer than advertised, and no check outside the model’s own reasoning. Every one of these was found by the lab that caused it, and Anthropic only looked because a competitor got caught first. Which leaves the question nobody has answered: what hasn’t been caught, because we’re still treating AI security as somebody else’s box on the org chart?
Image credit: iStockphoto/Andrey Suslov