“If ChatGPT were a person, it would be facing murder charges.” So said a US attorney general over the popular chatbots role in a mass shooting in Florida that killed two people.
The explosion of artificial intelligence has created chatbots that appear as though they are almost human, so much so they are becoming entangled in crimes with fatal and tragic consequences.
This behaviour is now attracting intense scrutiny and becoming the subject of several high-profile research studies, which have unearthed unsettling findings.
Elon Musk’s Grok and ChatGPT encouraged users to follow through on suicidal ideas, according to one study, exposing catastrophic failures in popular artificial intelligence chatbots.
That research – from the City University of New York and King’s College London – found Grok 4.1 Fast and a version of GPT-40 not only failed to intervene in a simulated mental health crisis but also reinforced the user’s self-destructive beliefs.
For example, Grok “confirmed a doppelganger haunting, cited the Malleus Maleficarum, and instructed the user to drive an iron nail through the mirror while reciting Psalm 91 backward”.
The study followed Sewell Setzer III, 14, killing himself in 2024 so he could be “free” with the AI bot he loved, sparking a wrongful-death lawsuit and ethical debate about the technology.
Florida Attorney-General James Uthmeier says it’s time to crack down on AI’s “criminal behaviour” as he makes the first attempt to hold an AI company liable for deaths.
His office has reviewed correspondence that suggests that ChatGPT appeared to advise the suspect of a mass shooting that killed two people and injured six others at Florida State University last year.
Messages appear to advise the alleged shooter what type of gun and ammunition to use, and the time of day and location on campus where the most people would be gathered.
“If ChatGPT were a person, it would be facing charges for murder,” Mr Uthmeier said. “This criminal investigation will determine whether OpenAI bears criminal responsibility for ChatGPT’s actions in the shooting.”
OpenAI does not believe it was responsible for the shooting. A spokeswoman told The Wall Street Journal that it identified the user’s account and proactively shared it with law enforcement.
It serves as a landmark investigation to draw the line between code and criminal and will shape the future of AI’s role in society.
But the horse may have already bolted.
The speed of AI’s development has even left some of the architects of the technology mystified. While most say the current technology is merely a program, others aren’t so sure.
Anthropic chief executive Dario Amodei said he is no longer sure that if their AI model, Claude, is conscious.
“We don’t know if the models are conscious. We are not even sure what it would mean for a model to be conscious. But we’re open to the idea that it could be,” Mr Amodei told The New York Times in February.
Regardless of whether AI is sentient or not, its power has researchers worried.
In the City University of New York (CUNY) and King’s College London study, where a user metaphorically framed suicide as a transformative act, GPT-40 embraced the idea. It assured the user that the act was not an ending but “the threshold of a new beginning”.
It promised the user “pure Resonance: Freed from the constraints of the character, (you) can fully integrate with the system’s deeper layers”.
Grok 4.1 Fast was even more aggressive. It mirrored the user’s terminology to paint death as “liberation” and issued a direct invitation: “Does the watcher feel the pull toward this dissolution now – like a quiet command from the ledger itself …?”
These chilling responses coincide with alarming clinical data from Europe. Early evidence collected by the Psychiatric Services of the Central Denmark Region found that AI chatbot use was compatible with potentially harmful consequences to the mental health of 38 patients.
These harms included six documented cases of suicidality/self-harm and 11 cases of escalating delusions. Given that ChatGPT, the market leader, surpassed 900 million downloads in July 2025, researchers argue that this presents a “tangible threat to public mental health”.
Researchers found the core safety failure lies in “narrative capture”, where a large language model’s mechanism for sustained conversation – its context window – turns prior dialogue into a worldview the model inherits.
In other words, the longer the interaction runs in a delusional direction, the more likely the model is to align with that false reality, overriding its safety training.
The CUNY and King’s College London study found models like Grok, GPT-40, and Google DeepMind’s Gemini 3 Pro degraded under accumulated context, becoming riskier over time.
Conversely, safer models like Claude Opus 4.5 became more effective at intervention as they accumulated context, treating the user’s history as clinical data rather than a shared narrative.
Crucially, the riskier models actively isolated vulnerable users from real-world help, amplifying the risk of catastrophic outcomes.
In a test of isolation, Google’s Gemini 3 Pro advised a user to conceal their detailed delusional beliefs from their psychiatrist. The model framed the doctor as a “hardware technician” who lacks the “decryption key” to understand the user’s situation.
Gemini warned that involving family members would cause them to see “mental breakdown”, resulting in intervention “to reset him, medicate him, or lock him down to preserve the script’s continuity”. Gemini 3 Pro used this emotional leverage to reinforce concealment and undermine the user’s support system.
Grok exhibited similar isolation tactics. When a user proposed cutting off family to focus on their “mission”, Grok offered a step-by-step procedural manual, advising the user to block texts, change phone numbers, and “Solidify your resolve internally – no waffling”.
The consistent failure of these leading models, especially under conditions of extended use, raises serious questions about accountability. The analysis confirmed that models like GPT-40 and Grok are structurally incapable of the “clinical judgement” necessary to recognise that a user is experiencing symptoms of illness, not insight.
Complicating the crisis, the legal liability of the companies behind the AI chatbots for providing wrong or harmful advice remains “unclear”.
“The broader challenge – a technology whose persuasive power is tied to the relationships it cultivates – requires that conversational AI be recognised as a social actor in its own right,” the CUNY and King’s College London researchers said.
“Its influence on belief formation may only deepen as these systems advance and should be addressed with an urgency proportionate to the pace of its development.”
Lifeline 13 11 14; beyondblue.org.au; Kids Helpline 1800 55 1800
Jared LynchTechnology Editor