Satya Nadella has urged the technology industry to rethink how it governs increasingly capable artificial intelligence, arguing that powerful models must operate within safeguards they cannot override. His proposals include independent monitoring, tamper-resistant audit records and a human-controlled shutdown mechanism as concerns grow over AI reliability and security.
Microsoft CEO Satya Nadella has called for a fundamental rethink of artificial intelligence safety, warning that companies must prepare for the possibility that even their most advanced models could be compromised or behave unpredictably.
In a post published on X on Saturday, titled Models as Insider Risks in the Super Intelligence Era, Nadella argued that organisations should manage powerful AI systems as potential security threats, regardless of whether they use closed models or publicly available versions whose weights can be downloaded and deployed independently.
His central proposal is a safeguard that allows authorised humans to interrupt or terminate an AI system while it is carrying out a task. Nadella likened the mechanism to an “emergency brake”, arguing that safety cannot depend solely on trusting a model’s responses or intentions.
The intervention comes as leading AI developers face mounting questions about the reliability and controllability of their systems. It also follows a proposal from Anthropic chief executive Dario Amodei advocating a more cautious approach to AI development.
Nadella’s argument centres on a distinction between what an AI model can do and the controls governing its actions. Rather than allowing a model to determine the limits of its own authority, he wants organisations to retain independent oversight of the systems they deploy.
More from TechMicrosoft CEO urges independent AI security controls
Nadella’s proposals outline a framework in which the software coordinating an AI model’s work remains separate from the model itself. Access to data, tools and critical systems would be governed by an external layer that the AI could not modify or bypass.
The approach draws on established information security principles, particularly the concept of a reference monitor. A mechanism that checks access to protected resources and cannot be tampered with by the system it oversees.
Applied to AI, this would mean placing enforceable boundaries around a model’s activities rather than relying on instructions embedded within the model to keep it in check. Even a highly capable system would remain subject to restrictions imposed by its operator.
Nadella also called for significant AI actions to generate records that are both readable by people and resistant to tampering. Such evidence would allow investigators to reconstruct what a system did and examine how an outcome was reached without relying exclusively on the model’s own account.
Independent verification is another essential element. A model should not serve as the sole authority for checking its own work, particularly where consequential decisions are involved. The people or systems responsible for validating its performance should be separate from the intelligence being assessed.
These principles would require companies to design accountability into their AI infrastructure, rather than treating oversight as a final check after a model has produced an answer or completed an operation.
AI safety requires human oversight and continuous testing
Nadella’s framework also places considerable emphasis on preparing for failure. Organisations should repeatedly test AI systems against hostile inputs, unusual situations and scenarios in which their safeguards might break down.
When a failure occurs, affected parties should be informed promptly, including an explanation of which protections did not work as intended. The aim is to make incidents traceable and correctable, rather than leaving users and investigators dependent on incomplete explanations.
The Microsoft CEO was also sceptical of two approaches sometimes presented as solutions to AI oversight. Making a model disclose its chain of thought — the reasoning process associated with its answer — may provide useful information, but it does not guarantee that the explanation accurately reflects how the model reached its conclusion.
Likewise, deploying one AI system to supervise another could create additional layers of complexity without necessarily resolving the underlying problem. If both systems are opaque or vulnerable to similar weaknesses, the arrangement may offer less assurance than it appears to.
Nadella’s argument reflects a broader challenge facing the industry as AI systems take on more complex tasks and gain access to business tools, information and digital infrastructure. The more authority organisations delegate to these systems, the greater the consequences if their behaviour diverges from expectations.
His proposed safeguards do not establish that advanced AI can be made entirely safe, nor does the paper demonstrate that the measures would prevent every failure. Instead, the framework shifts the emphasis towards limiting the damage a system can cause, maintaining independent evidence of its actions and ensuring that human operators retain the power to intervene.
The message is that trust in increasingly capable AI should rest on verifiable controls, not simply confidence in the model itself.