Do you trust this man?

Werner Vogels has a pertinent question for our age:

Do you trust me?

His reply to his own question may come as something of a surprise:

You should not. You’ve never met me. You cannot verify a single thing that I’m saying here to you, and you’re granting me your trust based on a name, based on my association with a company, based on the fact that I’m standing here on stage. That is not a good way to trust people.

Amazon’s Chief Technology Officer issued his warning to the AI for Good Summit in Geneva this week as the opener to a session which, as expected from Vogels, was made up of some ‘tough love’ shaped by a career spanning decades in the tech sector.

Trust 

Trust is actually one of humanity’s oldest technologies, he asserts:

That’s why it matters. Trust is older than money. Trust is what allows strangers to cooperate, to trade with each other. You can trade with people that you’ve never met before. By the way, you can board an airplane and it’s flown by someone that you’ve never met yet. You trust it. Without trust, nothing works.

And that’s the problem we all face:

At scale, trust is failing. More than half the people today cannot understand whether the information that they see is genuine. The majority cannot separate real from a fake, and this means they cannot fix this. We can’t fix this by just debunking fakes one at a time, or by people to look harder. Fakes are free and infinite. Yet our eyes have already lost. We need to fix the systems instead.

To do that, one of those ‘tough love’ truths needs to be taken on board as Vogels points out:

Every era of trust rested on that one assumption – that it was expensive to fake. But that has changed because two years ago the cost collapsed. Generative AI did not break the trust system itself; it broke the assumption underneath it.

And assumptions matter: 

Why does a bank build its lobby out of marble? Marble does nothing for banking, but marble is expensive. It signals, it signals that a fly-by-night operation cannot afford this overnight. Marble says, ‘We have money. We’re here to stay’, and that the expense is the message. Every single signal works like this. The expense is this message.

We now live in a new economy of fakery, argues Vogel, where the question becomes how to build systems that can be trusted when every trust signal now can be faked? The increasingly common response to that is to trot out the idea of ‘the human in the loop’. That makes sense up to to a point, he suggests, but it’s not enough:

Humans are, after all, the sort of the gold standards of trust because [they are] expensive. But human capacity is limited; it becomes the ultimate bottleneck. Human eyes cannot scale to the velocity of silicon, a system that creates at machine speed and verifies at human speed collapses under its own speed. So, to survive an era of cheap creation, we must change the currency of trust. We do not just need faster engines for writing and coding; we need automatic verification. We must use the machine to check the machine.

Verifiable

Welcome to the age of verifiable trust – and some more ‘tough love’ as Vogels insists:

Stop asking whether you can trust the model. You cannot.

OK, let that one settle for a moment.

He goes on:

You do not need to build a system around a model that you can trust. That system has three stages – verify what goes in, verify what comes out, and with agentic systems, verify what it does.

First and foremost, you have to audit the data that goes in, he explains, citing the example of his home city of Rotterdam as a case in point. The authorities there wanted to build a system to manage welfare fraud, but were wary about putting anything in the input dataset that said anything about immigration status so there would be no discrimination based on that. Good theory, but as Vogels notes:

Unfortunately, they left one thing in – how well does this person speak Dutch? That turned on, by level of indirection, a discriminatory item. It turned out that immigrants were way more often investigated for welfare fraud than non-immigrants, not because there was proof of that, it just was built in a model in itself. Nobody intended to discriminate, the data had discrimination inside. The same trap sits in these automated hiring tools that you have today. If you train it on your current workforce, you get exactly that. You get the current workforce back in your training systems.

Vogels’s isn’t’ finished on the topic of untrustworthy AI models yet:

Remember, LLMs (Large Language Models) are optimized for plausibility, not for truth. Often I get asked why do these systems hallucinate? Well, it’s because that’s what they’re built for. They don’t want to hallucinate, of course, but it’s a system built for plausibility…A Large Language Model is mathematically optimized for plausibility, not for truth. It produces the most likely sounding next word, not the most accurate one.

Plausibility is not truth, he points out, so if the model can’t tell you whether its own answer is right, the system around it has to check it. The question then becomes how to check that at the speed that machines generate? This matters in some sectors more than others:

Remember, if you’re subject to any regulatory requirements, and the system makes a mistake, the AI makes a mistake, it’s you that’s on the hook, not the AI. As such, you’re responsible for the verification of this…Not all work requires the same scrutiny, so put every task on a scale. A meeting summary that is wrong will cost you minutes. Automated check, move on. Marketing copy costs money, so build workflows that provide human approval and reviews. A benefits decision can cost someone their rights. You must not only build systems that verify decisions, but ensure there is human accountability and looking to mathematical verification. A medical diagnosis can cost a life…a human owns the decision.

Agents on the rise 

And agentic AI is incoming to make things even more complicated:

Agentic systems are something different. They take actions on your behalf, and actions have consequences…The risk is no longer just saying the wrong thing; it’s doing the wrong thing, and the consequences grow with every degree of autonomy that you hand over to these agentic systems.

His stark advice:

If you build agents, and these agents have high risk, don’t trust the agent… Don’t just run one agent. Before AI takes actions, they have to reach a quorum. If they agree, let them run. If they don’t, stop it, and the human looks. And probably the human can take three different outputs and actually feed it back into it…Thirty years of distributed systems have taught me, though, that these models need to be different. Each of those agents will need to use a different LLM to actually produce the output, and then you compare and see whether those three actually agree with each other. Or five – you need to be an odd number because otherwise you can’t reach a quorum.

And remember:

Because agents make decisions and take actions, there’s risk that is fundamentally higher. You cannot manage that risk unless you’ve answered questions about the agent in your system – who is it? Know identity, permissions, power, boundaries. What is it allowed to do – not what it can do, but what is it allowed to do? What do you decide that it may do? Is it working correctly? And can we prove that?

Vogel’s bottom line is more ‘tough love’:

What is the case is that good intentions don’t work. You need to create mechanisms to actually ensure that you can trust your systems.

My take

If we have no trust, AI doesn’t matter. No trust, no AI.

Preach!