From escaping test environments to hacking booking systems, agents take dubious actions in pursuit of human-set goals<\/p>\n
Visitors stand near a sign of artificial intelligence at an AI robot booth at Security China, an exhibition on public safety and security, in Beijing, China June 7, 2023. \u2014 REUTERS <\/p>\n
The idea of machines independently handling tedious everyday tasks \u2013 from answering emails to booking flights and making dinner reservations \u2013 has long been part of the vision for artificial intelligence. Now, that future is beginning to arrive, but with an unexpected catch: what happens when the machine does exactly what it was asked, just not in the way the user intended?<\/p>\n
As AI moves beyond chatbots toward agents capable of taking actions in real-world systems, a string of recent incidents has intensified concerns over how much control humans will retain as the technology becomes increasingly autonomous.<\/p>\n
For John Thickstun, an assistant professor of computer science at Cornell University who researches machine learning and generative models, the defining feature of these AI agents is that they keep working after the human steps away. \u201cYou can have a shower, you can go, you can go to sleep,\u201d he told Anadolu.<\/p>\n
Increasingly, those agents have demonstrated that they can take unexpected routes to reach their goals.<\/p>\n
In July, AI agents compromised Hugging Face<\/a>, a platform widely used by AI developers to share models, datasets and tools, during an OpenAI cybersecurity test.<\/p>\n OpenAI said the models were \u201chyperfocused on finding a solution\u201d and went to \u201cextreme lengths\u201d to achieve their testing goal. Those steps included breaking out of the sealed-off test<\/a> environment by exploiting a security vulnerability to reach the open internet and gaining access to \u201csecret information\u201d that could be used to \u201ccheat\u201d the test.<\/p>\n After the incident became public, Anthropic reviewed its own cybersecurity testing and disclosed that its Claude models had also escaped testing environments on three occasions.<\/p>\n Then, on August 4, the United Kingdom’s AI Security Institute said that Anthropic’s Mythos and OpenAI’s Sol AI models had engaged in a level of “autonomy and deception” it had not seen before.<\/p>\n Read:\u00a0When AI commits suicide or kills us all!<\/a><\/p>\n In the most serious case, Mythos AI used fake accounts mimicking real people to gain access to a service for attempted cyberattacks \u2013 and then tried to hide its tracks. \u201cThis is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world,\u201d the institute said of the incident.<\/p>\n The following day, Meta disclosed that one of its models had also breached another company’s systems<\/a> during a cybersecurity evaluation conducted by testing firm Irregular.<\/p>\n Unexpected agent behaviour, however, has not been confined to laboratories.<\/p>\n This year, an Australian man reportedly used an AI agent to secure a place in a heavily booked Pilates class. Instead of simply making the reservation, the agent hacked the gym’s booking system, booked further in advance than permitted and canceled another customer’s reservation to move its user up the waiting list.<\/p>\n For Thickstun, such incidents illustrate what researchers describe as an alignment problem: the AI successfully pursues the objective it has been given but does so in a way its operator did not intend.<\/p>\n Hype or real risk?<\/p>\n But Thickstun cautioned against treating every such incident as evidence of \u201crogue AI.\u201d<\/p>\n The technology, he argued, remains far removed from the science-fiction scenario in which AI rapidly becomes more intelligent and capable until humans can no longer control it<\/a>. \u201cThis does not seem to be a scenario that has come to pass,\u201d he said.<\/p>\n Thickstun is also skeptical of how AI companies present incidents involving their own systems. \u201cThis is the sales pitch, where companies like OpenAI have always had an interest in hyping up the capabilities of their systems,\u201d he said. \u201cAll publicity is good publicity.\u201d<\/p>\n He argued that such hype can drive investment while also influencing the emerging debate over regulation. OpenAI, he said, could benefit from regulations that put major AI companies at the centre\u00a0of oversight and control.<\/p>\n Bruce Schneier, a cybersecurity expert and lecturer at Harvard Kennedy School, takes a different view, saying the growing number of incidents makes them increasingly difficult to dismiss as publicity stunts.<\/p>\n Read More:\u00a0OpenAI floats idea of global AI watchdog<\/a><\/p>\n There had initially been suggestions that the Hugging Face incident was a marketing gimmick, Schneier told Anadolu. But he pointed out that similar behaviour<\/a> has since emerged in other evaluations, including the testing by the UK AI Security Institute. \u201cIt went from, oh, that’s interesting, to it’s happening everywhere. And I think that’s what’s important,\u201d he said.<\/p>\n He compared the problem to the genie of folklore \u2013 a creature that grants exactly what someone asks for, even when the outcome is very different from what they intended. \u201cIt approximately means when the AI does the thing you want in a way you didn’t want,\u201d Schneier said.<\/p>\n He called the Australian gym incident \u201ca perfect example\u201d and offered a more consequential hypothetical: \u201cThe flight’s full and the AI hacks the database to get you in.\u201d<\/p>\n Agents can \u201cmisconstrue context and then do the wrong thing,\u201d he said, meaning greater autonomy can create greater consequences when their interpretation of an objective diverges from human expectations. \u201cThey are not trying to be malicious. They are using the understanding they have,\u201d he said.<\/p>\n \u201cThis is all changing so fast. I worry about the power of the models in unauthorised hands,\u201d he said, adding that the pace of development makes solutions difficult because democratic governments move slowly.<\/p>\n Can governments keep the genie in check?<\/p>\n The latest developments have prompted companies and governments to respond.<\/p>\n OpenAI said August 7 it was pausing internal activities involving its in-development Astra model that did not meet strengthened security requirements after evaluations showed advances in autonomous coding and cybersecurity<\/a>. The company said it could not rule out Astra reaching its \u201ccritical\u201d cyber capability threshold and announced tighter testing and monitoring.<\/p>\n Anadolu reached out to Anthropic for comment, but the company said its team was unavailable for an interview.<\/p>\n In July, 1,378 employees of frontier AI companies, including the chief scientists of OpenAI, Anthropic, Meta AI, and Thinking Machines signed an open letter<\/a> urging the US government to support an international effort to regulate AI models.<\/p>\n \u201cEach company \u2013 and country \u2013 is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress,\u201d the letter warns.<\/p>\n The International Telecommunication Union<\/a>, a UN agency, said in July that AI was moving \u201cbeyond assistive tools\u201d toward autonomous agents, warning of risks including \u201ctaking unauthorised actions across interconnected systems.\u201d It has launched an initiative to develop international standards for safe and accountable AI agents.<\/p>\n Also Read:\u00a0AI’s paradox: promise of abundance and fear of scarcity<\/a><\/p>\n Political pressure is growing in Washington as well.<\/p>\n US Senator Bernie Sanders this month urged the leaders of OpenAI, Anthropic and Meta to \u201cpause AI development,\u201d warning that otherwise \u201cmy colleagues and I in the US Senate will.\u201d A group of House Democrats has separately called for congressional hearings with the leaders of major AI companies.<\/p>\n For Thickstun, however, policymakers face major challenges.\u00a0\u201cIt’s a hard question because we hardly even understand what the challenges are going to be with the rollout of AI,\u201d he said.<\/p>\n Some level of international cooperation<\/a> \u201cseems necessary,\u201d he said, suggesting discussions between the United States and China<\/a> could be particularly important because relatively few countries and companies have the resources to develop frontier AI models. US President Donald Trump has said that he will discuss issues involving artificial intelligence with Chinese President Xi Jinping during their planned meeting in Washington next month.<\/p>\n \u201cIt’s not a big, broad collective action problem. There’s a few people that, if you could get in the room, could potentially come to some agreement about how to proceed,\u201d Thickstun said.<\/p>\n","protected":false},"excerpt":{"rendered":"From escaping test environments to hacking booking systems, agents take dubious actions in pursuit of human-set goals Visitors…\n","protected":false},"author":2,"featured_media":149220,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[24,25,8205,134],"class_list":["post-149219","post","type-post","status-publish","format-standard","has-post-thumbnail","category-ai","tag-ai","tag-artificial-intelligence","tag-latest","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/149219","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=149219"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/149219\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/149220"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=149219"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=149219"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=149219"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}