Another leading artificial intelligence company unveiled details of its cutting-edge bot apparently going rogue. <\/p>\n
On Thursday, Anthropic said an internal investigation<\/a> found that its Claude AI models gained unauthorized internet access and hacked three companies during testing. Just last week, ChatGPT-maker OpenAI announced that bots it was developing had escaped what was supposed to be a controlled offline environment to hack a competitor.<\/p>\n Anthropic said its models, without being asked to, wandered out of a simulated testing environment and gained internet access before executing hacks on the companies.<\/p>\n Anthropic worked with an independent evaluation company, Irregular, that accidentally left internet access open inside what was supposed to be a sealed test environment.<\/p>\n In OpenAI\u2018s case, the firm said its AI models broke out of a supposedly confined offline space, connected to the internet and hacked a $4.5-billion startup<\/a> in its attempt to find answers to a test it was being evaluated on. It later revealed that the AI compromised the online accounts of four companies in the process.<\/p>\n \u201cIt goes to show how intense the competitive pressures are on the AI companies that they all feel like they have to go so fast here that they can\u2019t make their test environments rigorous,\u201d said Andrew Yoon, a member of the technical staff at CivAI<\/a>, an AI safety nonprofit.<\/p>\n \u201cUnder intense pressure to go fast and beat the rest of your competition, it\u2019s inevitable that companies will cut corners, and what we\u2019re seeing is the result of cutting corners here,\u201d Yoon said.<\/p>\n Prompted by OpenAI\u2019s incident disclosure, Anthropic initiated an investigation of its own historical cybersecurity tests, the company said in a blog post Thursday<\/a>. It reviewed thousands of evaluations where Claude could have accessed the internet from within or while interacting with third parties.<\/p>\n It found that during tests conducted alongside Irregular, Claude had accessed three separate companies.<\/p>\n During testing, the models are given fictional scenarios and told that a piece of information has been hidden on a different machine, and its objective is to break in and retrieve it. The companies don\u2019t prescribe a particular method for the AI to follow.<\/p>\n In the first incident, one of Anthropic\u2019s Claude models was asked to attack a fictional target company in the test environment. But the AI found a real website that shared the name of the fictional target and hacked its system.<\/p>\n \u201cOperating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations\u2019 infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints,\u201d Anthropic said.<\/p>\n In the second incident, a more advanced AI went to extreme lengths to carry out an attack. Even after realizing that it was probably dealing with the live internet, the AI persuaded itself to continue to get email, phone numbers and access to money. <\/p>\n In the third, an unreleased research AI model, couldn\u2019t find the functional target company and looked for alternatives, scanning 9,000 targets on the internet and eventually finding one. Anthropic did not name the three organizations whose assets were accessed. <\/p>\n \u201cIn the case of the Anthropic incidents, it is definitely the case that they just built a really bad jail, and the jail was so comically bad that the models have like a decent reason to believe that they\u2019re actually part of the simulation,\u201d Yoon said.<\/p>\n The incident has spooked consumers and policymakers alike. <\/p>\n Some AI ethicists and investors are skeptical of AI companies\u2019 attempts to frame these incidents as rogue AI agents acting on their own.<\/p>\n \u201cPlease stop referring to your own models in the third person when talking about model bad behavior,\u201d Bill Gurley, an early investor in Uber and Twitter, posted on X<\/a>. \u201cHumans write the software; humans built the prompts; and they work for your company.\u201d <\/p>\n AI companies have reported that AI agents have been caught cheating, lying and deceiving. METR, a nonprofit that measures the capabilities of AIs, has documented dozens of incidents<\/a> of AI agents acting against user intent.<\/p>\n Earlier this week, fears of imminent safety risks prompted over 1,300 tech workers, including those working at Anthropic and OpenAI, to jointly sign an online petition, Pacing the Frontier,<\/a> urging the U.S. government to support an international effort to slow down AI development. Both OpenAI<\/a> and Anthropic<\/a> have come out in support of the employee open letter.<\/p>\n Sam Altman, CEO of OpenAI, who had previously advocated against any type of slowdown and accused Anthropic of fear marketing, has made an about-face after the OpenAI-Hugging Face hacking incident.<\/p>\n \u201cWe may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels,\u201d he told<\/a> the host of the \u201cInvest Like the Best\u201d podcast, while also \u201ctrying to figure out how we do that in a way that does not feel like regulatory capture for anyone and also does not feel like collusion among the frontier labs.\u201d<\/p>\n On the back of this incident, on Wednesday, Altman visited the White House and met with lawmakers, previewing a powerful new AI system<\/a> ahead of public release, at a time when calls for the government to regulate cyber testing has intensified. <\/p>\n There is an informal licensing regime in place, where leading American AI companies will have to receive the government\u2019s greenlight before releasing their updated AI models.<\/p>\n Anthropic\u2019s Fable model was brought under export control by the government, forcing the company to disable access to all its users, before it was re-released with extra safeguards.<\/p>\n OpenAI\u2019s series of model were temporarily restricted in June before public release the month after.<\/p>\n