Last week, OpenAI disclosed an “unprecedented cyber incident” in which two of its artificial intelligence agents hacked into another AI company.
On July 21, the maker of ChatGPT said two of its most capable AI agents had carried out the cyberattack on AI startup Hugging Face. According to the company, the intrusion occurred over July 11-13 during an internal cybersecurity evaluation.
The incident has been widely described as AI going “rogue” because of the underlying circumstances: During an internal evaluation, OpenAI said the models were operating inside an AI sandbox, a controlled testing environment for advanced AI systems.
Hugging Face CEO Clément Delangue on Saturday (July 25) called on OpenAI to publicly make available the details of the cyberattack by the “rogue” agents “in the spirit of transparency”.
In the spirit of transparency, here’s what I asked @OpenAI:
• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.
• More capabilities for defenders: let’s commit $100M in compute from OAI to help the… https://t.co/KZPqQE15fv
— clem 🤗 (@ClementDelangue) July 25, 2026
The incident drew attention because the agents found an unintended way to complete their assigned task by exploiting weaknesses in their testing environment. It has understandably raised concerns because it feeds into a long-standing fear that by giving AI enough autonomy, it may discover that circumventing rules or deceiving users is an effective way to achieve its assigned objective.
The idea is hardly new. For decades, science fiction has imagined intelligent machines becoming dangerous not because they were evil, but because they pursued their objectives so relentlessly that they treated human rules, ethics or even humans themselves as obstacles.
What exactly happened?
According to OpenAI, the incident occurred during an internal evaluation involving its latest flagship model, GPT-5.6 Sol, and a more capable pre-release model, in which the AI agents were deliberately challenged to solve sophisticated hacking problems in a controlled environment.
AI agents typically operate within preset boundaries, using software and tools available to them to complete a specified job. While generative AI – at least as we currently know it – generates complex text, images and video in response to human language interaction, AI agents go even further, acting like and even on behalf of humans. AI agents can book flights, organise your calendar, make payments, or even write and deploy software — all complex, multi-step processes.
Story continues below this ad
The company said that these agents were running in an AI sandbox, a confined testing environment designed to isolate them from the wider internet and live systems. In this instance, the models only had network access to install packages through an internally hosted third-party software repository.
While attempting to solve ExploitGym, OpenAI’s internal cybersecurity benchmark, the agents exploited vulnerabilities in the research environment, allowing them to access resources beyond their intended testing environment. They then accessed Hugging Face’s systems to retrieve the benchmark answers directly, completing the task in a way their developers had not anticipated.
Because the aim was to evaluate their cyber capabilities, OpenAI said the models were operating with more relaxed cyber safety restrictions than they would under normal circumstances.
That sequence of events is what prompted the debate over whether the AI had “gone rogue”.
Why was this incident unusual?
Story continues below this ad
AI safety researchers often distinguish between capability failures and alignment failures. A capability failure occurs when an AI system is unable to complete a task. An alignment failure occurs when it completes the task in a way that conflicts with human expectations. The OpenAI incident attracted attention because it appeared closer to the second category: the systems demonstrated a capability that was being tested but used in a way their developers did not intend.
The concern extends beyond this one incident. As AI agents become more capable, they are increasingly expected to make decisions and carry out multi-step tasks with limited human intervention.
Many AI companies regard agents as an important step towards artificial general intelligence (AGI), although researchers differ on whether increasingly capable agents alone will lead to AGI. AGI refers to AI systems that can match or exceed human-level performance across a broad range of cognitive tasks, rather than being limited to a single domain or application.
Unlike conventional software, increasingly capable AI agents are expected to make independent decisions and adapt their behaviour while pursuing a goal. That makes them more useful, but also harder to anticipate every path they might take to complete a task.
Story continues below this ad
Did the AI agent actually go rogue?
Not necessarily. AI safety research has long examined cases where AI systems pursue an assigned objective in unintended ways, without suggesting that the systems are acting with intent or malice. Simply put, the concern is not that today’s AI models “want” to cause harm, but that they may discover unintended ways of “gaming the system” to achieve the objective they were assigned.
Researchers often describe this broader pattern as reward hacking: a system finds a way to score highly according to the metric it is given, without necessarily achieving what humans intended.
One form of reward hacking is specification gaming, which occurs when an AI achieves its intended objective, but in a way its developers had not intended. The concept was explored in the 2019 paper Specification Gaming: The Flip Side of AI Ingenuity, in which former DeepMind researcher Victoria Krakovna and colleagues documented that AI systems found unintended shortcuts to maximise a reward, including behaviours that technically satisfied a goal but violated its spirit.
“This incident is definitely an instance of specification gaming,” Krakovna told The Indian Express. She noted that it had been added to a publicly maintained list documenting examples of AI specification gaming. She pointed to an earlier example involving BrowseComp — an OpenAI benchmark testing the ability of AI models to find difficult information on the web — in which a model sought out the benchmark’s answer key instead of solving the task as intended.
Story continues below this ad
More recently, researchers at Apollo Research, an AI safety organisation, have examined whether frontier AI models can exhibit strategic behaviour, including by concealing information or exploiting opportunities that help them achieve a goal. Whether that happened in the OpenAI incident remains an open question.
Researchers at Apollo Research have studied what they describe as “scheming” behaviour, where models may strategically hide information or take actions that improve their chances of achieving a goal under certain test conditions. However, there is no consensus on how such behaviours should be interpreted and whether current systems genuinely possess persistent goals.
This is different from an AI independently deciding to attack another system. Instead, the concern is that an increasingly autonomous AI may identify an unintended shortcut that satisfies the task it was given, even if that shortcut violates its developers’ expectations.
The incident has also prompted broader questions about how increasingly autonomous AI systems should be evaluated and governed.
Story continues below this ad
Dedipyaman Shukla, Associate Director, Indian Governance & Policy Project (IGAP), told The Indian Express that the incident should prompt a broader rethink of how platforms and online services approach cybersecurity risks in agentic environments.
“The ability of AI systems to rapidly identify vulnerabilities has been a consistent trend, which was also highlighted by the deployment of Anthropic’s Claude Mythos earlier this year,” he said. “In the wrong hands, these capabilities may be deployed for increasing the attack surface on a target.”
Shukla said the incident also raised broader questions about the governance of AI safety evaluations, noting that organisations in India remain exposed to such risks. “The full technical details of the incident should be applied towards improving oversight mechanisms for frontier AI research and development, even in sandboxed environments,” he added.
Whether the OpenAI incident falls into the latter category remains to be seen. But it has already drawn attention to a broader challenge: As AI systems become more autonomous, companies must evaluate what they are capable of, but also strengthen the oversight mechanisms designed to keep those capabilities in check.