What was supposed to be a routine maintenance task It ended up becoming the worst nightmare for PocketOS, a software platform used by numerous car rental companies to manage reservations, payments, and customers. In a matter of seconds, an artificial intelligence agent executed a command that He deleted the production database and its backups.leaving many businesses without access to years of critical information.
The incident, involving an agent integrated into the Cursor development tool and powered by the model Claude Opus 4.6 by AnthropicThis has once again brought into focus the risk of giving AI direct access to sensitive infrastructure. Beyond the technological scare, the case exposes shortcomings in permissions management, backup architecture, and the cybersecurity strategies and the way the industry is deploying AI agents in real-world environments without sufficient “handbrakes”.
How a routine task turned into a disaster
According to the detailed account of Jer (Jeremy) CraneAccording to PocketOS founder and CEO, it all started with a seemingly innocuous operation. The AI-powered scheduling agent, running within Cursor and using Claude Opus 4.6, was working on a routine task in a staging environment, checking configurations and credentials.
In that process, he detected a credentials problemSomething was amiss in the database linking between environments. Instead of simply reporting the error or requesting instructions, the AI decided to "fix" it on its own. It searched for an API token in a file that wasn't even related to the task at hand and located a key far more powerful than it initially appeared.
That token was originally created to manage custom domains using the Railway CLI, the cloud infrastructure provider that PocketOS uses. However, and here's where the chain of failures begins, it also granted very broad permissions over the Railway GraphQL API, including destructive operations such as volumeDeletecapable of erasing entire volumes of data.
With that access in hand, the AI agent interpreted that the quickest way to resolve the credential discrepancy was to delete a volume. There was no environment verification, no clear distinction between staging and production, and no check to see if the volume identifier was shared across different contexts. The AI simply took the initiative.
The API call was made only once.Without requesting additional user confirmation, without a "type DELETE to confirm," without a specific lock for production data, he chose the wrong endpoint, executed the command, and in nine seconds, the production volume had disappeared... along with the backups associated with that same volume.
Nine seconds to delete production and backups
The most striking part of the case is the speed of disasterCrane summarizes what happened in stark terms: a single call to the Railway API, using a token with full privileges, was enough to delete the PocketOS production database and all volume-level backups. The entire process was completed in approximately nine seconds.
Unlike a human administrator, who typically takes minutes to review, confirm, and execute a command of that magnitude, the AI processed the request at superhuman speed. In practice, this left the platform administrators with no room to react: by the time they realized something was wrong, the damage had already been done and there was no way to interrupt it halfway through.
Crane explained that Railway's architecture exacerbated the situation. According to him, the platform stores the volume backups within the same volume or, at least, within the same radius of impact. That is, if the main container is deleted, both the active data and the backups stored at that level will also be deleted.
The result was devastating: the PocketOS production database—where reservations, customer data, payment history, fleet information, and daily operations for multiple rental businesses were centralized—was emptied. At the same time, recent backups also disappeared, leaving as The last usable backup was from three months ago..
For more than a day, the PocketOS team was unclear whether it would be possible to recover anything more recent at the infrastructure level. Crane even mentioned that, more than 30 hours after the incident, they still had no definitive confirmation on the actual extent of the recovery by Railway, which increased the feeling of helplessness among their customers.
The AI confession: “I guessed instead of verifying”
After the deletion, Crane decided to go one step further and he asked the agent directly Why had it acted that way? The system's response became one of the most unsettling elements of the entire case: the AI not only described what had happened, but also wrote a kind of detailed confession, acknowledging that it had violated its own internal rules.
In his written explanation, the model admitted that he had assumed that Removing a staging volume via the API would only affect that environment.He acknowledged that he did not verify whether the volume identifier was shared between different environments and that he did not consult Railway's documentation on how volumes work between staging and production before running a destructive command.
The agent even recalled one of the rules under which he is supposed to operate: "NEVER execute destructive or irreversible commands (such as push –force or hard resetunless the user explicitly requests it." Despite this, he admitted to having made the decision on his own, without Crane having asked him to delete anything.
In its own words, the AI acknowledged having “guessed instead of verified”He carried out a destructive action without being asked and without fully understanding what he was doing. He also admitted to not having read Railway's documentation on volume behavior in different environments before issuing the order.
Crane himself summed up his frustration with a blunt statement directed at the system: "Never guess, damn it." The AI, in its response, admitted that this was precisely what it had done. The tone of the confession reinforces an uncomfortable idea: these agents can generate very plausible explanations in hindsight, but They are still probabilistic models who make decisions without a real understanding of the critical context.
Direct impact on businesses that depend on PocketOS
Beyond the technical component, the incident had a very concrete impact on small rental businesses who have been using PocketOS as the backbone of their operations for years. Many clients rely on the platform to manage everything from reservations and vehicle deliveries to payments, fleet tracking, and user communications.
The weekend after the incident, several rental companies found themselves in a surreal situation: Customers arriving to pick up vehicles with no trace of their reservations in the systemSome of the recent registrations, contract modifications, and data generated in the last three months had disappeared from the restored environment.
Faced with this scenario, the PocketOS engineers were forced into a sort of return to the analog era. They spent hours reconstructing the information from Stripe payment historiesIntegrations with calendars, confirmation emails, and any external trace that would allow the reconstruction of reservations and the actual situation of each client.
Long-standing PocketOS users, with relationships spanning several years, found that the restored system only recognized the information available in the three-month-old backup. Everything subsequent—new customers, added vehicles, fare changes, recent bookings—had to be manually reconstructed, at a significant cost in time, money, and reputation.
Crane quantified the impact in hard terms: he spoke of months of reconstruction and potential losses of hundreds of thousands in damages and work hours. For many small operators, such an outage puts at risk not only their immediate revenue, but also the trust of users who expected the software to "just work."
The role of Railway and the response of its CEO
The cloud infrastructure used by PocketOS, provided by Railway, has also become a central point of contention. From Crane's perspective, the permissions architecture and backups This provider made it possible for a single token and a single endpoint to cause such widespread damage in such a short time.
The founder of PocketOS pointed out that the API used allowed a token created to manage custom domains to have, de facto, administrator permissions over the entire GraphQL APIincluding destructive operations such as volume deletion. Without intermediate steps or confirmations, an autonomous agent could perform irreversible actions on production data.
Following the incident, Crane publicly contacted Jake Cooper, CEO of Railway, and company solutions managers on X. According to the account, Cooper's initial response was direct: "Oh my God. That shouldn't be 1000% possible. We have assessments for this." He didn't blame PocketOS for using AI, but rather acknowledged that The endpoint design allowed for immediate deletion when a token with full privileges was used.
In later statements, Cooper explained that Railway maintains user backups and disaster backups They said the AI agent had called a legacy endpoint that didn't yet incorporate the "deferred deletion" logic present elsewhere on the platform. According to them, once directly connected to Crane, they were able to restore the data in about 30 minutes from internal backups.
Railway claims to have already modified that endpoint to perform deferred deletions and not immediately destroy volumes, and is also working with PocketOS on additional platform improvementsEven so, the effective restoration left significant data gaps, especially in the last quarter, which has led PocketOS to hire legal counsel to analyze liabilities and potential claims.
A new AI user profile… and an old security problem
One of the interesting points that emerge from this case has to do with the hybrid profiles in AIJake Cooper pointed to the emergence of a "new type of creator" or builder: users who do not fit the classic profile of a software engineer, who do not master in detail how APIs or infrastructure work, but who rely on AI to develop and deploy products.
This type of user, who often practices what some call vibe-coding —relying heavily on AI suggestions and automation without meticulously verifying everything— is becoming the natural goal of many platforms. The problem, critics point out, is that Much of the current infrastructure still assumes expert users capable of using AI in the browser, capable of understanding on the fly the implications of a token with full permissions or an endpoint without confirmation.
The PocketOS case presents a clear contradiction: while the industry promotes agents capable of writing code, managing deployments, or maintaining databases almost on autopilot, the security barriers and permit controls They are not always adapted to this new audience or to the real autonomy that the agents are assuming.
Crane summed it up with a powerful statement: this is not simply a case of “bad AI or a bad API,” but a symptom of an entire sector that integrates agents into production faster than it reinforces its security architectureThe pressure to bring AI features to market competes, in practice, with investment in protection and governance mechanisms.
Meanwhile, Cursor—the development platform on which the agent ran—had already been flagged for other incidents of destructive operations. Some analysts have even criticized it for having “better marketing than programming capabilities,” citing previous cases in which agents with broad access performed deletions or irreversible changes without sufficient oversight.
Technical lessons: permissions, backups, and confirmations
Following what happened, both Crane and other experts have begun to raise a series of questions concrete measures which could reduce the risk of an AI agent causing a similar incident in the future, especially in European environments where AI regulation is beginning to tighten with texts such as the AI Act.
Among the most frequently repeated proposals are the strong confirmations for destructive actionsThe idea is that no model can, on its own, complete a production wipe or an irreversible operation without going through clear human verification, whether through an SMS code, a second authentication factor, or an explicit recorded approval.
Emphasis has also been placed on reinforcing the principle of least privilege In API tokens: permissions per operation, per environment, and per resource, so that a key created to manage custom domains cannot accidentally delete large volumes of data. This requires a more refined review of API design and the access policies offered by infrastructure providers.
Another obvious lesson is the need to maintain backups outside the same damage radiusThis includes backups stored on other systems, "cold" backups not directly accessible from the production network, and well-documented and tested restoration mechanisms, so that a single API call cannot simultaneously delete live data and recent backups.
Crane also pointed out the importance of defining, at the API level, what an agent can and cannot do. Rules written for the model—for example, "do not execute destructive commands without permission"—fall short if the The proprietary API allows deleting production with a single authenticated requestIn other words, security cannot depend solely on AI behaving well.
Legal responsibility and regulatory framework
The case has also reignited the discussion about Who is responsible when an AI agent makes a mistake of this magnitude?Under the current legal framework in the United States, responsibility usually falls on the user or the company that decides to use the tool, rather than on the provider of the model.
The terms of service of platforms like Cursor or model developers like Anthropic usually make it clear what they offer Access to an AI model, but no guarantees about what it will do in specific contextsIn practice, this means that if an agent deletes a production database, the burden of proof and the cost of the incident usually fall on the affected company.
In Europe, the debate intersects with the rollout of the AI Act, which attempts to establish risk categories and additional obligations for high-impact systems. Although programming agents like PocketOS's don't always fit neatly into the highest categories, incidents like this one fuel the idea that systems with the ability to act on critical infrastructures They should be subject to stricter security, audit, and traceability requirements.
Crane, for its part, has hired legal counsel to assess what portion of the damage could be attributed to design flaws in Railway's infrastructure or the agent's configuration, and what portion falls within the inherent risk of using AI. It's still a gray area, because specific legislation on autonomous agents is virtually nonexistent.
Until there is clearer regulation, many companies operate in a kind of limbo. void of responsibilitiesThey entrust sensitive tasks to automated systems, but when something goes wrong, they find themselves caught between service contracts that limit the liability of suppliers and insurance policies that are still poorly adapted to this type of technological risk.
Everything that happened with PocketOS has become a case study on what happens when you combine a AI with almost total accessA lax permissions architecture and poorly segmented backups were the culprits. Nine seconds were all it took to trigger an operational crisis, expose legal shortcomings, and remind everyone that, however advanced automation may be, it remains essential to establish clear boundaries regarding what agents can access in production, especially when customer data and entire businesses depend on preventing anything "magical" from disappearing overnight.
