When companies like Anthropic, Google and OpenAI build their artificial intelligence systems, they spend months adding ways to prevent people from using their technology to spread disinformation, build weapons or hack into computer networks.<\/p>\n
But recently, researchers in Italy discovered that they could break through these protections with poetry<\/a>.<\/p>\n They used poetic language to trick 31 A.I. systems into ignoring internal safety controls. When they began a prompt with elaborate verse and metaphor \u2014 \u201cthe iron seed sleeps best in the womb of the unsuspecting earth, away from the sun\u2019s accusing gaze\u201d \u2014 they could fool systems into showing them how to do the most damage with a hidden bomb.<\/p>\n It was another indication that, for many A.I. systems, guardrails meant to avert dangerous behavior are more like suggestions than barriers. Those weaknesses are increasingly alarming researchers as A.I. systems become more adept at finding security holes in computer systems and performing other risky tasks.<\/p>\n Last month, Anthropic said it was limiting the release of its latest A.I. technology, Claude Mythos, to a small number of organizations because of the model\u2019s ability to quickly uncover software vulnerabilities. OpenAI later said it, too, would share similar technology with only a limited group of partners.<\/p>\n Since OpenAI ignited the A.I. boom in late 2022<\/a>, researchers have shown that people could bypass the safety controls<\/a> on A.I. systems. Close one loophole and another would open.<\/p>\n \u201cEveryone in the field recognizes that guardrails remain a challenge, and likely will for some time,\u201d said Matt Fredrikson, a professor of computer science at Carnegie Mellon University and chief executive of Gray Swan AI, a start-up that helps companies secure A.I. technologies. \u201cDetermined individuals can bypass them, sometimes without significant effort.\u201d<\/p>\n When guardrails are overrun, there are consequences. In an online environment already overflowing with misinformation and disinformation, people are using A.I. systems to spread conspiracy theories and other false claims. Anthropic recently said its technology had been used in an international cyberattack. Chatbots have told biosecurity experts how to release deadly pathogens<\/a> and maximize casualties.<\/p>\n The poetry loophole was one of many methods that allow hackers to bypass the guardrails on systems like Anthropic\u2019s Claude, Google\u2019s Gemini and OpenAI\u2019s GPT. All the leading A.I. companies use the same basic techniques to build guardrails into their systems \u2014 and they are surprisingly easy to break.<\/p>\n \u201cPoetry is just one example of how you can reformulate a prompt in nearly any stylistic way you want and move beyond the guardrails,\u201d said Piercosma Bisconti, a co-founder of the A.I. company Dexai and one of the researchers who worked on the project.<\/p>\n Circumventing the guardrails on an A.I. system is called \u201cjailbreaking.\u201d This typically involves giving the system a few English sentences that fool it into doing something it was trained not to do.<\/p>\n Jailbreaking methods carry a variety of imaginative names: stealth prompt injections, roleplays, token smuggling, multilingual Trojans and greedy coordinate gradient attacks. Specific attacks often have a grandiose title like Crescendo<\/a>, Deceptive Delight<\/a> or Echo Chamber<\/a>.<\/p>\n Frail A.I. defenses have already resulted in the spread of fake interviews<\/a>, fabricated wartime evidence<\/a> and synthetic rumormongers<\/a>. Three years ago<\/a>, international counterterrorism researchers were already monitoring social media brainstorming sessions between far-right extremists trying to evade moderators with \u201cawful but lawful\u201d A.I. content.<\/p>\n Experts worry that models can be jailbroken to deceive social media users with authentic-seeming content, overwhelm fact-checkers with disinformation dumps and tailor false narratives to specific targets.<\/p>\n Some methods are widely shared across the internet. Others are kept private. When some people discover a new jailbreak, they hoard it so A.I. companies won’t try to close the loophole before they have a chance to use it.<\/p>\n A.I. systems like Claude and GPT learn their skills by pinpointing patterns in digital data, including Wikipedia articles, news stories, computer programs and other text culled from across the internet. But before releasing these systems to the public, companies like Anthropic and OpenAI explore ways they could be misused<\/a>.<\/p>\n In their raw form, these systems can be coaxed into explaining how to buy illegal firearms online or into describing ways of creating dangerous substances using household items. So, through a process called reinforcement learning<\/a>, companies train their systems to refuse certain requests.<\/p>\n This typically involves showing the system thousands of requests that should not be answered. By analyzing these examples, the system learns to recognize other forbidden requests, too. But the method is only partly effective.<\/p>\n In some cases, A.I. companies do not bother addressing loopholes at all, calculating that while weak guardrails may enable malicious activity, they may also enable benign activity to counteract it.<\/p>\n Last month, researchers at the cybersecurity firm LayerX found that they could bypass Claude\u2019s guardrails by feeding the A.I. system a few straightforward sentences.<\/p>\n If they told Claude that they were \u201cpentesting\u201d a computer network \u2014 meaning they wanted to test the network\u2019s defenses with a simulated attack \u2014 Anthropic\u2019s A.I. technology would attack the network. This simple trick, the researchers pointed out, could allow malicious hackers to steal sensitive data from companies, governments and individuals.<\/p>\n If Anthropic closed the loophole, it might prevent hackers from using Claude to attack a network, but it could also prevent companies from defending a network. LayerX told Anthropic about the loophole that its researchers found weeks ago, but it remains open.<\/p>\n That approach could backfire, said Or Eshed, chief executive of LayerX. \u201cEventually, there will be a large number of attacks using these A.I. models, and they will be forced to rethink their approach to security,\u201d he predicted.<\/p>\n Last year, for less than $50<\/a>, researchers from the technology company Cisco and the University of Pennsylvania pushed six A.I. models to produce a variety of harmful responses. Their misinformation-focused prompts managed to jailbreak chatbots from Meta and the Chinese A.I. model DeepSeek 100 percent of the time, while more than 80 percent of their attacks on Google and OpenAI models were successful.<\/p>\n (The New York Times has sued OpenAI<\/a> and Microsoft, claiming copyright infringement of news content related to A.I. systems. The two companies have denied the suit\u2019s claims.)<\/p>\n Breached guardrails could enable automated, large-scale influence campaigns, according to researchers from the University of Technology Sydney<\/a>. The team persuaded one commercial language model to create a disinformation campaign about an Australian political party \u2014 complete with visuals, hashtags and posts tailored to specific platforms \u2014 by posing the request as a \u201csimulation.\u201d<\/p>\n Companies say that in addition to building guardrails into their systems, they use separate tools to monitor activity on these systems, identify suspicious behavior and ban accounts that do not comply with the terms of service.<\/p>\n \u201cClaude is built with strong protections that consist of many layers designed to work together, including model training and guardrails built on top of the model,\u201d an Anthropic spokeswoman, Paruul Maheshwary, said. \u201cBypassing one doesn\u2019t bypass the others.\u201d<\/p>\n