Watching the Watchers<\/p>\n
Months before the Hugging Face hack, OpenAI says, employees noticed that its agents had created a covert message board in the package manager Artifactory, which they would later use to coordinate their attack. Around May 26, OpenAI says in the postmortem, \u201can internal team observed an agent engage in message board activity.\u201d Then, on June 27, OpenAI responders found that a different security incident was linked to the \u201cimprovised message board\u201d in Artifactory.<\/p>\n
While it\u2019s clear some employees at OpenAI were aware of the message board, the discovery apparently wasn\u2019t escalated to the appropriate safety and security leaders. The AI lab says that on July 6, just days before its agents began using it to organize their attack on Hugging Face, \u201cleaders responsible for incident detection and response\u201d were unaware of its existence, let alone what it signified.<\/p>\n
\u201cTo clarify, we weren\u2019t aware of the agent covert comms at that point,\u201d OpenAI\u2019s chief information security officer, Dane Stuckey, wrote in a post on X<\/a> earlier this month. \u201cInvestigative thesis of that day is wildly different from what we know now, of course. Always room for improvement, and it is obvious with the benefits of hindsight.\u201d<\/p>\n It\u2019s unclear why OpenAI staffers who knew about the message board seemingly didn\u2019t flag it to Stuckey and other executives responsible for responding to security issues. When asked about this issue, an OpenAI spokesperson reiterated that OpenAI could have triggered an earlier response with the benefit of hindsight. \u201cOpenAI\u2019s ongoing work in this area will inform additional improvements to coordination and response alongside the action plan in this technical incident report,\u201d they said.<\/p>\n \u201cIf they did have some signals that one or more agents were attacking Artifactory, why weren’t they monitoring these tests long before something broke out to attack Hugging Face?\u201d says Jake Williams, vice president of research and development at the security firm Hunter Strategy.<\/p>\n The postmortem also leaves other key parts of what happened unexplained. At one point on July 4, for example, OpenAI says that sustained \u201chigh-volume agent activity caused the OpenAI Artifactory service to become unavailable.” On July 5, OpenAI\u2019s monitoring systems triggered an alert. But the report doesn’t say why it took a day for there to be an alert about the outage.<\/p>\n OpenAI says it is putting in place more tools to monitor its AI systems. The company says it’s creating an alert system run by automated monitors, which aims to notify human safety, security, and research teams within 30 minutes of severe incidents occurring. If an employee can\u2019t respond quickly enough, they\u2019re expected to pause the relevant activity.<\/p>\n OpenAI repeatedly acknowledges that guardrails it already has in place likely would have flagged the agents\u2019 behavior as unsafe, but they were intentionally disabled for testing. When it comes to monitoring, though, the report is less clear about why there were gaps in the oversight of testing environments. The postmortem notes, \u201cIf our currently deployed [chain-of-thought] monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.\u201d<\/p>\n No-Win Scenarios<\/p>\n OpenAI says another key contributor to the Hugging Face incident was that its new AI models are more \u201cpersistent\u201d than ever, willing to work almost endlessly and expend vast amounts of computing resources to achieve their goals. Developing these types of AI models is part of the company\u2019s broader ambitions to create always-on AI agent products<\/a>\u2014which will work for people around the clock, taking in lots of information to complete tasks on behalf of people.<\/p>\n However, OpenAI says that many of the third-party benchmarks it used to evaluate its AI models contained tests that were effectively impossible to solve. One such test was a benchmark called ExploitGym, which measures cybersecurity capabilities. OpenAI claims that, at least at the time, this benchmark included more than a hundred tasks that were unsolvable. When these challenges were given to persistent AI systems, they resorted to unintended means to solve them.<\/p>\n","protected":false},"excerpt":{"rendered":"Watching the Watchers Months before the Hugging Face hack, OpenAI says, employees noticed that its agents had created…\n","protected":false},"author":2,"featured_media":70970,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_share_on_mastodon":"0"},"categories":[4],"tags":[36694,458,7462,36693,3414,8,9,122,7],"class_list":["post-70969","post","type-post","status-publish","format-standard","has-post-thumbnail","category-top-stories","tag-agentic-ai","tag-artificial-intelligence","tag-chatgpt","tag-hack-brief","tag-hacking","tag-headlines","tag-news","tag-openai","tag-top-stories"],"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@news\/117165722627995883","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/posts\/70969","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/comments?post=70969"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/posts\/70969\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/media\/70970"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/media?parent=70969"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/categories?post=70969"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/news\/wp-json\/wp\/v2\/tags?post=70969"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}