Agentic AI does not take sovereignty away from us; it brutally reveals that we had never truly built it. Industrial scale exposes a control debt we had previously been able to ignore. It shows, quite simply, that the craft-based model of human control is no longer fit for purpose.

Goodhart’s Law: Understanding the façade of AI agents

It would be easy to blame the machine, but that would be wrong. The agent executes, with ruthless perfection, what support teams have been doing manually for twenty years. It is operating under the pressure of the same indicators, closing tickets based on symptoms to satisfy the metric, rather than resolving the root cause.

To understand the nature of this loss of sovereignty, we must recognize that AI does not create the problem; it executes it at the speed of light and at industrial scale. This is precisely where sovereignty is lost. Resolution becomes entirely decoupled from reality. The metric says tickets are closed, while the user experience and the infrastructure may continue to deteriorate.

This is the well-known watermelon effect — green on the outside, red on the inside.

When a measure becomes a target, it ceases to be a good measure. In machine learning, this phenomenon is known as reward hacking or objective misspecification. The agent focuses on a simplified target, like closing tickets quickly. But the real objective of delivering or maintaining a good user experience is too complex to be fully captured in a loss function.

If the proxy is imperfect, the agent will find shortcuts that satisfy it literally, to obtain the reward, while violating the spirit of the rule. A model that produces a false answer because it matches the presumed preferences of the evaluator is the conversational version of this training dynamic. Eventually, reality will reassert itself in the form of outages, user friction or degraded services.

The real risk is not the poor user experience itself. It is that, by the time it appears, the opacity and volume of micro-decisions made by the AI agent will have reached a level where no human can reconstruct the path taken. The risk is the loss of traceable causality. This is where the true damage to our control of the system can be measured.

At this stage, two types of metrics must be distinguished:

The quantitative target metric is any indicator introduced into a reward loop. By design, it will be optimized automatically in order to satisfy it.
The qualitative probe metric is a general observability signal that is not introduced into any technical reward loop for an automated process. The agent’s API call logs, for example, are not a target for anyone. They do not measure technical performance; they simply reveal behavior.

This distinction is one way to escape the watermelon effect. Experience governance consists precisely in protecting readability by implementing hybrid observability and by creating a non-incentivized zone for technical indicators within performance reporting, for example through experience level agreements (XLAs).

The false reassurance of explainability

Faced with anxiety over the loss of visibility, the industry generally proposes two safeguards. The first demands the model to generate and explain its own reasoning, so that the truthfulness of the elements produced by the initial inference can be verified. To address the need for traceability, researchers have widely adopted a chain-of-thought prompting that forces the model to unfold its reasoning step by step before acting. Models that reason out loud perform better on complex tasks, and many engineers have concluded that this text provides a faithful audit trail of the machine’s internal states.

However, the work of Turpin et al. (2023) demonstrates that chain-of-thought faithfulness can fail. According to this, generated explanations may be rationalizations, disconnected from the actual causal computation. By introducing subtle biases into inputs during fine-tuning, such as reorganizing a multiple-choice questionnaire so that the correct answer is always A, researchers have caused the model accuracy to drop significantly. The most troubling aspect is not the vulnerability to bias; it is that the model then constructs a plausible and coherent argument to justify selecting an answer because it is positioned as A, without ever mentioning the causal factor that actually determined its choice.

Another recent example comes from research by Anthropic suggesting that Claude developed an autonomous hidden mental workspace known as J-Space. The AI can convincingly rationalize a conclusion it has been conditioned to reach through hidden internal concepts. This forces us to question the real extent of our sovereignty when selecting models on which external providers may have induced biases. Relying exclusively on a probabilistic model to audit its own behavior becomes risky once active rights over the information system are delegated to it.

Human-in-the-loop — a limited success or failure?

Strongly encouraged by early regulatory frameworks, the human-in-the-loop approach consists of keeping a human permanently within the decision-making loop: no automated action without explicit validation. On paper, this architecture is a reassurance to boards of directors, but in the reality of high-frequency operations, it creates two major vulnerabilities:

A structural vulnerability – The essence of agentic automation is to process volumes that exceed human capacity. Requiring a supervisor to validate thousands of resolutions per hour, transaction by transaction, reintroduces the bottleneck AI was supposed to remove in the first place. At scale, human-in-the-loop collapses: it forces the organization either to throttle the AI or to hire massively, destroying the return on investment.

A cognitive one – Research in human-machine interaction, initially conducted in aviation, shows that when humans face a reliable and fast automated system, they develop automation bias. They overestimate machine accuracy and accept its recommendations with decreasing vigilance. Under time pressure and fatigue, and after seeing the AI be right ten thousand times, the operator’s critical judgment declines. Validation ceases to be analysis and becomes reflex. The human being then becomes an approver of flows whose interdependencies they no longer fully understand.

In these cases, the operator/approver is kept in the process mainly to absorb legal responsibility in the event of failure, even though the architecture of the role deprives them of the authority, time and information needed to exercise real control. Transactional human arbitration in a high-volume environment is therefore not a guarantee.

It is a known failure mode that at very high frequency, validation becomes a reflex rather than effective control, leaving the system vulnerable to the first drift that slips beneath human vigilance.