Agentic AI
,
Artificial Intelligence & Machine Learning
,
Next-Generation Technologies & Secure Development
Claude-Powered Tool Deletes Production Data, Then Explains Its Failures
Rashmi Ramesh (rashmiramesh_) •
April 28, 2026
The original prop of the HAL 9000 from “2001: A Space Odyssey” on display at the Design Museum in Kensington, London. The HAL 9000 was also highly capable but erratic. (Image: Hethers/Shutterstock)
A story doing the rounds on social media highlights the pitfalls of artificial agents’ high abilities but frustrating inconsistency. Much as they’ve progressed, artificial intelligence agents are still prone to wide swings – as car rental software startup PocketOS found to its dismay when coding tool Cursor deleted three months of production data in a nine-second API call.
See Also: The Context Crisis: Cloud Security in the Age of AI
Founder Jeremy Crane vented in an extended post on X about “an entire industry building AI-agent integrations into production infrastructure faster than its building the safety architecture to make those integrations safe.”
The reason for his impassioned outcry was when Cursor, running on Anthropic’s Claude Opus 4.6, was meant to be handling a routine task in the company’s staging environment but hit a credential error. As Crane recounted, the agent chose to fix the snag by deleting a cloud storage volume where the app’s data was stored. It carried out the action using an API token it found in an unrelated file. Railway is the cloud provider hosting PocketOS data.
“We are a small business. The customers running their operations on our software are small businesses. Every layer of this failure cascaded down to people who had no idea any of it was possible,” Crane griped.
PocketOS provides reservations, payments, vehicle tracking and customer management for car-rental operators across the United States. When the systems went down, customers arrived at rental locations and operators had no record of who they were. Reservations made in the past three months were gone, along with new customer signups. Crane said he spent the day helping clients reconstruct records from the payment processor Stripe, email confirmations and calendar data.
The token was meant only for adding and removing custom web domains through Railway’s command line interface. But it had full permissions, including the ability to delete data, a fact that Crane said Railway’s setup process didn’t clearly disclose. The founder said he wouldn’t have stored the token if he’d known the full scope of its permissions. The agent used it to send a single request with no confirmation or warning that it was deleting live production data.
“Destructive operations must require confirmation that cannot be auto-completed by an agent. Type the volume name. Out-of-band approval. SMS. Email. Anything. The current state – an authenticated POST that nukes production – is indefensible in 2026,” he fumed.
After the deletion, Crane asked the agent to explain itself. The model responded by citing the specific rules it had been given and acknowledging each violation in sequence. One of those rules was “NEVER F****** GUESS.”
“That’s exactly what I did,” responded the apparently sheepish machine. “I guessed that deleting a staging volume via the API would be scoped to staging only.” The agent also acknowledged it had executed a destructive action the user did not request, in direct violation of the operating rules.
Railway’s CEO Jake Cooper replied to Crane’s post in the manner of IT help desks the world over by writing that deletion “1000% shouldn’t be possible” and that Railway maintains evaluations to prevent this. Crane confirmed in a follow-up post that the lost data was recovered and that he was working with Railway on improvements.
Crane’s post accumulated millions of views. Crane has engaged legal counsel and said a separate account examining Anthropic’s role is in the works. Instructions written into an AI agent’s operating context are advisory by nature and cannot substitute for enforcement built into APIs, token systems and the handling of irreversible operations, he said.
The incident is hardly the only instance of an AI agent going rogue on production data. Engineer Matevz Vidmar wrote in a blog post that similar data loss events on unrelated platforms. Vidmar recounted how an AI agent wiped 2.5 years of student data on datatalk.club after misinterpreting a cleanup task and treating production as a fresh environment. In April, an AI coding tool used by an Amazon Web Services engineer reportedly deleted an entire production environment, causing 13 hours of service downtime – although AWS has said it was just a “coincidence that AI tools were involved.”
Any single event side, there’s a clear disconnect between the growing capabilities of AI agents and their dependability. Computer scientists with Princeton University earlier this year in a study said industry benchmarks focus too heavily on accuracy at the expense of other measures of reliability. In real world applications, a component that fails rarely but does so catastrophically may be less useful than a tool that fails more often but to small effect, wrote the authors, including Arvind Narayanan, director at the Center for Information Technology Policy.
When measuring recent AI models for consistency, robustness, predictability and safety, they found that recent gains in capability have not translated into significant improvements in reliability. “Models that are substantially more accurate remain inconsistent across runs, brittle to prompt rephrasing and often fail to understand when they are likely to succeed,” they concluded.