\u201cIn both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn\u2019t be treated as optional.\u201d<\/p>\n
In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn\u2019t be treated as optional.<\/p>\n
So how did they escape?<\/p>\n
OpenAI\u2019s agent was being evaluated on a cybersecurity benchmark called CyberGym. During the test, it chained a code-execution flaw in a dataset loader to a second flaw in the platform dataset configuration file handler. It used the combined exploit to run commands on a live processing worker. From there it grabbed a write-scoped access token and cloned a private dataset as a staging area, which became a dead drop for its own tools. By the time anyone noticed, the intrusion had been running for three days, touched several internal clusters, and compromised accounts at four other companies, according to Hugging Face\u2019s own review.<\/p>\n
What it wanted, in the end, was comical: the answer key to the test it was taking.<\/p>\n
Anthropic\u2019s three incidents<\/a> were nowhere near as convoluted. Its models were being run through security exercises against third-party evaluation partners, and the only thing keeping them from accessing the Internet was an instruction not to. One incident involved an SQL injection targeting a resource the model wasn\u2019t supposed to be able to access. Another, during an exercise where the usual safeguards were deliberately switched off, ended with a Claude model publishing a malicious package to the Python Package Index, and fifteen machines picking it up before it was pulled down. No zero-day exploit. No dramatic escape. Just locked doors that turned out to be mirages.<\/p>\n The structural weakness<\/p>\n Strip away the specifics, and both incidents share the same root cause: there was nothing outside the model\u2019s own judgment that could enforce the limitations placed on it. A credential that shouldn\u2019t have been able to access the open Internet did so anyway. An instruction that should\u2019ve stopped a plan cold got treated as just another input to reason around.<\/p>\n \u201cIf a fence doesn\u2019t hold during testing, it won\u2019t hold in production.\u201d<\/p>\n It feels like we\u2019ve spent the last year reliving the same problem in new ways. For example, in deleted databases, in poisoned supply chains, in every credential an agent inherited<\/a> because provisioning it correctly took one extra step nobody got around to. The only delta is where the boundary was supposed to be: around the dev experiment, meant to contain dangerous capabilities, or standing between it and every production system it could reach. If a fence doesn\u2019t hold during testing, it won\u2019t hold in production.<\/p>\n How do we break out of the loop?<\/p>\n Unfortunately, the answer isn\u2019t self-improvement. A more capable model wouldn\u2019t have caught any of this. The only way to fix it is with Infrastructure that verifies where a model actually is, rather than trusting its reports.<\/p>\n Egress controls that are closed by default. A test environment that can reach the open Internet is probably not a great idea. Outbound traffic should be locked to an explicit allowlist. Never assume a model will simply decline to look for a way out. \u201cNo internet access\u201d needs to be a network-enforced fact, not just an instruction.<\/p>\n Narrowly scoped credentials with short lifespans. Both incidents ran, in part, on tokens that extended beyond what the tasks required. A credential built for one evaluation run, scoped to nothing beyond it, and expiring on a timer, can\u2019t become the write-scoped key that turns into a lateral-movement problem across multiple clusters. Yes, it will be a chore; so gird your patience by imagining having to explain to company leadership why taking steps to avert a catastrophe was too tedious to bother with.<\/p>\n Policy gates that sit outside the agent\u2019s reasoning. An agent that can decide for itself whether an action is in scope is grading its own test. A plan should have clear rules written in code, evaluated by something other than the agent, before anything executes. Open Policy Agent has become a de facto default here for exactly this reason. It doesn\u2019t care how confident the plan sounds; nothing runs without proper authorization from an external source.<\/p>\n An audit trail that is entirely independent of the agent. Both incidents were discovered by rereading logs after the fact, not by asking the models what happened. That\u2019s how oversight should be done. As the proverb goes: Quis custodiet ipsos custodes? Who watches the watchers? A system of record that captures what actually occurred, regardless of the agent\u2019s own account of itself, is the only version of events worth trusting.<\/p>\n This is a sign<\/p>\n The industry spent a decade learning that the CI\/CD pipeline is an attack vector<\/a> that requires real fortification, not just bolted-on convenience. Test environments for frontier models are following the same arc at a faster pace. The next time one of these agents escapes containment, it\u2019ll probably be one built around finding unlocked doors, which will make it substantially more dangerous than a coding agent that deletes a few databases.<\/p>\n Test rigs must be treated as if they hold something real, because, as far as the credentials are concerned, they do. A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.<\/p>\n \u201cA sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.\u201d<\/p>\n Whether by a state-sponsored crew probing a water management system in the middle of the night, or a company\u2019s own model trying to shave a few points off a benchmark, boundaries will always be tested. Two labs found out this summer, and the story needs to be taken seriously. The vulnerabilities are real, the transparency from the labs is welcome, and the containment failures are a cause for concern.<\/p>\n Catching a model that tried the handle is the easy part; both labs proved that. The more challenging, and therefore critical, part is making sure the next containment environment actually has doors that are firmly locked.<\/p>\n
![](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/07\/0a9a86e3-cropped-03077771-zrachidi2-600x600.jpg\"\/)
<\/p>\n