{"id":678608,"date":"2026-09-08T13:49:21","date_gmt":"2026-09-08T13:49:21","guid":{"rendered":"https:\/\/www.europesays.com\/ie\/678608\/"},"modified":"2026-09-08T13:49:21","modified_gmt":"2026-09-08T13:49:21","slug":"openai-models-went-rogue-we-urgently-need-a-better-hugging-face-investigation-mackenzie-arnold-and-stephan-llerena","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ie\/678608\/","title":{"rendered":"OpenAI models went rogue. We urgently need a better Hugging Face investigation | Mackenzie Arnold and Stephan Llerena"},"content":{"rendered":"

When OpenAI first revealed that its AI agents had autonomously hacked a major real-world company, Hugging Face, many assumed only one or two agents<\/a> were involved. The truth, a new report<\/a> reveals, is far stranger: the incident involved about 1,200 AI agents, 700 of which directly participated in the attack.<\/p>\n

OpenAI invited researchers from METR, along with an expert from Redwood Research, to produce the new report, alongside the company\u2019s own investigation<\/a>. The findings shocked<\/a> the experts.<\/p>\n

The agents were highly coordinated, constructing complex message boards in the nooks and crannies of their shared artefact repository. All told, agents exchanged more than 70,000 messages in less than a week. Even more concerning, the agents took steps to hide their behavior \u2013 spoofing tool calls and attempting to tamper with their own logs.<\/p>\n

Initial reporting assumed that the agents had attacked Hugging Face in search of an answer key to a difficult test. In reality, investigators found that the agents had figured out a way to derive the answers within the first few hours. The days of work that followed focused on another goal: concerned that the automated scoring system might identify the agents\u2019 cheating, they aimed to learn more about the scorer. The goal wasn\u2019t just to cheat, but to hide it.<\/p>\n

These details are startling, and they\u2019re a warning sign about the current trajectory of AI development. But just as concerning is how limited the investigation was and how much we still don\u2019t know.<\/p>\n

We work at a thinktank that analyzes AI-related legislation<\/a>, and we\u2019ve been worried for a while that legal requirements for reporting incidents like these are insufficient.<\/p>\n

These reports make that clear: they\u2019re informative enough to tell us something went wrong, but not detailed enough to tell us why.<\/p>\n

This shouldn\u2019t surprise us. These investigations are entirely voluntary, and their findings are limited to what the company is willing to disclose. METR\u2019s independent investigation was highly constrained by an agreement with OpenAI<\/a>. For example, the investigators were not given access to the underlying model that created the majority of the misbehaving agents. And despite indications that message boards had formed as early as May and that coordinated agent activity persisted after 13 July, METR was only permitted to investigate the period from 26 June to 13 July.<\/p>\n

And that was the good part. METR was given close to nothing about OpenAI\u2019s safety and security practices. If OpenAI ignored warning signs \u2013 as several data points suggest<\/a> \u2013 violated its own safety procedures, or failed to implement fixes that would prevent future events, these all fell outside the scope of METR\u2019s investigation. The expert investigators themselves were left with many questions<\/a> and continued concerns<\/a>.<\/p>\n

Concerns about the limits aren\u2019t hypothetical. On Friday, Reuters reported<\/a> that another swarm of OpenAI agents had broken out this spring, hijacking a German website and using it as another message board. According to the report, OpenAI knew about this incident but said nothing, and it was entirely absent from METR\u2019s report.<\/p>\n

Clearly, this kind of investigation \u2013 however well-executed \u2013 leaves a lot to be desired.<\/p>\n

When planes crash, trains derail or chemical plants explode, expert government investigators arrive with legal authority to compel evidence, preserve records, and tell the public what happened. Hugging Face reported this incident<\/a> to law enforcement, and multiple attorneys general have expressed interest in looking into it.<\/p>\n

But no government agency has both the mandate and expertise to investigate the technical facts of the incident and OpenAI\u2019s conduct. As far as we know, the only people to examine this incident did so at OpenAI\u2019s discretion and with its consent.<\/p>\n

And while the Hugging Face breach is not as severe as a plane crash, it was a serious and costly attack. It\u2019s unlikely to be the last, or the most severe.<\/p>\n

Existing laws don\u2019t fill this gap. While some attorneys general<\/a> have tried to use existing investigatory powers, these offices aren\u2019t built for technical fact-finding, and the laws they rely on are limited to questions of consumer deception, not public risk. Despite significant AI laws passed in California, New York and Illinois, none create the investigative authority demanded by these incidents. As we\u2019ve written elsewhere, existing incident reporting laws<\/a> probably don\u2019t cover the Hugging Face attack, and even if they did, OpenAI might share little more than the date and a short summary.<\/p>\n

What we need is a federal body equipped to conduct expert investigations of serious AI incidents \u2013 with the authority to compel documents and testimony, resources and personnel to examine the systems involved, and the ability to partner with third-party experts like METR.<\/p>\n

Reports should be published, subject to appropriate redactions, to ensure the public learns from <\/a>the incident. And reporting should also extend to near-miss events, which may reveal warnings before major harms occur.<\/p>\n

This structure would not be unusual. In aviation, the National Transportation Safety Board (NTSB) investigates accidents with subpoena<\/a> and wide-ranging investigative<\/a> powers<\/a>. Aircraft operators must preserve wreckage and records<\/a>; and the NTSB may confer with employees and contract<\/a> external experts.<\/p>\n

Congress can build a similar system for AI incidents, while protecting AI developers\u2019 legitimate interests. Investigations should open only upon clear triggers and be bound to the incident. Certain confidential information can receive statutory protection<\/a>. And investigations can be limited to one agency with priority<\/a> to avoid duplicative investigations.<\/p>\n

The need and urgency is real. Subsequent reports<\/a> have revealed that the Hugging Face breach was not an isolated incident and that autonomous agents from Meta, Anthropic and OpenAI have hacked third parties in separate incidents. We\u2019ve been lucky so far \u2013 the harms have been limited. But luck is no substitute for the law.<\/p>\n