This is the first instalment in a two-part QA Financial series examining how Barclays is preparing autonomous AI agents for deployment inside one of the world’s most highly regulated banking environments.
Part one explores the testing, observability and governance disciplines shaping production-ready AI. Part two will examine why evaluation, production assurance and software quality are becoming central to enterprise AI strategy.
Banks have spent decades refining how they test software before it reaches production. Autonomous AI agents are now forcing them to rethink that discipline.
The challenge is no longer simply proving that a model produces the correct answer. Increasingly, it is about demonstrating that an AI system can be observed, governed, evaluated and, if necessary, stopped.
That shift is becoming increasingly visible across financial services. Earlier this year, the UK’s Financial Conduct Authority brought Barclays, UBS, Lloyds Banking Group and others into its AI Live Testing programme, signalling that regulators are looking beyond model accuracy towards governance, monitoring, human oversight and operational controls inside live environments.
Andy McMahon
Andy McMahon, Principal AI Engineer at Barclays, believes banks need to think the same way. Speaking on The Brave Technologist podcast last month, McMahon offered a rare glimpse into how one of Britain’s oldest banks is approaching autonomous AI inside one of the world’s most tightly controlled technology environments.
Yet rather than discussing ever more capable AI models, he repeatedly returned to subjects that software testing professionals know well: telemetry, observability, evaluation, permissions, production controls and software assurance.
It is a notable change in emphasis. Much of the conversation around agentic AI still revolves around capability, while McMahon’s focus was confidence.
Not confidence that an agent can complete a task in a demonstration, but confidence that it will continue behaving safely once deployed into production.
Observability before intelligence
Ask most organisations about AI agents and the conversation quickly turns to reasoning, planning and autonomy. McMahon starts somewhere else.
“I covered three different core concerns for productionizing agent systems,” he explained. “So one was observability.” That is not an accidental choice.
For decades, software teams have relied on logs, metrics and monitoring to understand how production systems behave. McMahon argued autonomous AI requires exactly the same engineering discipline, only at a much deeper level.
“So you know how do you capture telemetry metrics, logs, traces for your agent and agentic solutions,” he shared.
Those few words may prove more significant than any discussion about larger models or more sophisticated reasoning.
Without telemetry, engineers cannot understand why an agent reached a particular decision. Without traces, they struggle to reconstruct failures. Without observability, testing effectively ends the moment software reaches production.
That represents an important shift. Testing no longer stops when software is released. Instead, production itself becomes part of the testing lifecycle, continuously generating evidence about how autonomous systems behave under real operating conditions.
That philosophy mirrors Barclays’ broader engineering direction. Last year, the bank described how it had embedded testing earlier into development while expanding observability across its engineering practices.
Global Lead for Performance Engineering Manik Sikka argued that observability and shift-left testing had become essential for identifying issues before software reached customers.
McMahon’s comments suggest those same principles are now being extended into autonomous AI.
Testing behaviour, not just functionality
Traditional software testing has largely focused on whether applications perform the functions they were designed to perform.
Autonomous systems introduce a different challenge. Engineers also need confidence that agents will continue operating within acceptable boundaries when conditions change.
For McMahon, safety is not an abstract discussion about existential AI risk. It is a practical engineering problem. “How do you make sure what you’re deploying safe and you can understand how it’s going to behave in the wild?”
That phrase, ‘behave in the wild’, may become one of the defining testing challenges of enterprise AI. Laboratory evaluations reveal how systems respond to expected scenarios. Production introduces unexpected ones.
That is why McMahon places equal emphasis on operational controls. “How can you employ different levels of control over your agents? How do you do things like kill switches programmatically?”
Rather than viewing kill switches as emergency mechanisms, he presents them as another component of engineering design. This broadens the definition of software quality itself.
Quality is no longer measured solely by functional correctness or performance. It increasingly includes recoverability, controllability and the ability to intervene safely when autonomous software behaves unexpectedly.
The same thinking is beginning to appear among regulators. Britain’s FCA has made clear that AI assurance extends beyond the model itself, encompassing deployment context, governance, evaluation techniques and input and output controls.
In effect, testing is becoming evidence that an entire AI system can be trusted rather than simply proving that an algorithm performs well in isolation.
Limits of autonomy
McMahon was equally dismissive of the idea that production AI should operate without constraints. “I think we are,” he said when asked whether autonomous agents are ready. “I think though there is definitely a case to be made that it’s really autonomy within bounds.”
That distinction becomes especially important inside banking. “You can’t just give an AI agent like carte blanche to do what it wants,” he argued. “You can’t give it like access to all of your tools, all of your systems.”
Instead, autonomy has to be designed. “But you can bound these systems and employ that control I was talking about,” McMahon stated “there are different permission structures you can employ.”
“There’s just lots of ways of giving control so that the agent can act autonomously but within bounds and that’s important,” he continued.
“You can’t just give an AI agent like carte blanche to do what it wants. You can’t give it access to all of your systems.”
– Andy McMahon
These comments push QA into territory traditionally occupied by architecture, identity management and security. Testing no longer asks only whether an agent produces the right answer.
It also asks whether the agent accessed only authorised systems, remained inside defined permission boundaries, respected governance rules and generated sufficient evidence for every decision it made.
In many respects, the questions resemble those already familiar from cyber resilience and operational risk. The difference is that the software itself is now capable of planning and making decisions, increasing the importance of proving those controls continue to operate under real-world conditions.
‘Old engineering’ still matters
One of McMahon’s strongest arguments is also one of the simplest: despite the excitement surrounding agentic AI, software engineering fundamentals have not suddenly become obsolete.
Too many organisations, he believes, are treating AI as if it demands an entirely new technology stack. “The biggest thing for me is everyone thinking this is such a revolution,” he said, “that they forget there’s a huge piece of evolution as well.”
That observation should sound reassuring rather than disappointing. Banks have spent years investing in DevOps, MLOps, automated testing, identity management, containerisation, monitoring and observability. McMahon argued those disciplines remain every bit as relevant for AI systems.
“All of the practices from DevOps, all of the stuff from MLOps, all of the stuff from data science, all the stuff from just traditional good app development… all of that is still there.”
“We’re still containerising stuff. We’re still running in Kubernetes. We’re still using identity providers. We’re still using observability solutions,” he said.
AI introduces new concerns, he acknowledged, but they are additions rather than replacements. “It’s the same technologies with new spins.”
That applies equally to identity and permissions. “Now I have a concept of an agent identity and I have to think how does that have certain permissions and how does it act on behalf of other things.”
The temptation, McMahon warned, is to assume everything must be rebuilt from scratch. “I think the biggest danger is everyone thinks I have to throw everything out and start again.”
Instead, AI engineering should build on proven software engineering practices while extending them to accommodate autonomous behaviour.
That means existing investment in automation, continuous testing, observability and production monitoring becomes even more valuable. Rather than creating an entirely new QA discipline, agentic AI is expanding the scope of the one banks already have.
Can regulation produce better software?
Few developers enjoy additional controls or governance requirements. Yet McMahon argues that working inside a heavily regulated bank ultimately improves engineering quality. “There are a lot of cool problems there.”
More importantly, he continued, “it does at the end make a better product.”
The reason is straightforward. Banking systems face extraordinary levels of scrutiny before reaching customers. “They have to go through so much scrutiny that you’re sort of you’re really sure.”
McMahon added: “You go into production with this thing, you’re like, this has been battle tested.”
That phrase, battle tested’, neatly captures how AI assurance is evolving inside financial services. Banks are increasingly expected to demonstrate not only that AI systems work, but that they continue working safely once exposed to real users, changing conditions and live production data.
That thinking aligns closely with the direction regulators are taking.
Britain’s FCA’s AI Live Testing initiative explicitly moves beyond evaluating models in isolation. Instead, it looks at governance, deployment context, human oversight, evaluation techniques and operational controls, effectively treating AI as an operational system rather than simply an algorithm.
For software testing professionals, that represents an important shift. Testing is becoming less about checking outputs and more about generating evidence that complex systems remain safe, resilient and accountable throughout their lifecycle.
The next QA discipline
Perhaps the biggest lesson from McMahon’s interview is that autonomous AI is not making software testing less important. It is making it significantly broader.
Observability, telemetry, permissions, evaluations, governance and production monitoring are no longer operational nice-to-haves added after deployment. They are becoming core engineering requirements that influence how AI systems are designed from the outset.
That reflects a wider trend already visible across financial services. Barclays has spent the past two years strengthening shift-left testing, automation and observability across its engineering organisation. Meanwhile, regulators are asking firms to produce evidence that AI behaves safely in live environments rather than relying solely on laboratory validation.
Together, those developments point in the same direction. Quality engineering is evolving from verifying software into assuring autonomous systems.
The question is no longer simply whether an AI agent completes the task it was given. It is whether engineers understand why it made its decisions, whether those decisions remained inside approved boundaries, whether every action can be traced, and whether the system can be controlled if something goes wrong.
Those are questions software testers have been asking for years. Autonomous AI simply makes the answers far more important.
Tomorrow, in Part II, Andy McMahon will explain why too many organisations are deploying AI agents before properly testing and evaluating them, why telemetry should be designed in from day one, and why the most important question for engineers is no longer ‘can I build this?’ but ‘should I build this?’
REGISTER TODAY – SIMPLY CLICK HERE
Why not become a QA Financial subscriber?
It’s entirely FREE
* Receive our weekly newsletter every Wednesday * Get priority invitations to our Forum events *
REGULATION & COMPLIANCE
Looking for more news on regulations and compliance requirements driving developments in software quality engineering at financial firms? Visit our dedicated Regulation & Compliance page here.
READ MORE
QA FINANCIAL PODCASTS
