(©dkeine – canva.com)

AI is now the fastest-growing expense in corporate technology budgets, according to Deloitte. That’s to be expected – businesses understand that the enterprise of the future is AI-driven, and they’re ramping up their investments in this crucial transformation.

The problem is, companies aren’t just spending more on AI. They’re often spending more on AI than they should.

It’s all too easy for businesses to treat large language models (LLMs) as universal decision-makers, rather than situationally appropriate tools, and watch their LLM token usage soar. Or, to put it another way, it’s all too easy for businesses to risk an LLM making a bad decision – while paying for the privilege.

When an agent does (and doesn’t) need to call an LLM

LLMs are probabilistic reasoning engines. When an AI agent needs to find the most likely correct answer using imperfect information (especially human language), an LLM can be hugely valuable. For example, imagine an AI agent that’s tasked with classifying a customer support request, submitted via email. The customer may have expressed their need in any number of ways, and calling an LLM could help the agent to infer the customer’s intent and correctly classify the request.

But what if an AI agent needs to make a simple, deterministic (if A is true, then B) decision? Let’s continue with the same example – say the customer believes they’ve been charged twice for a recent purchase. The agent now needs to decide whether to automatically issue the refund. In this scenario the range of inputs required to make the decision are finite, and all the necessary information is right there in the business’s systems. The correct next step, based on that information, is also clearly defined by the company’s business rules.

Here, the AI agent shouldn’t have to guess what to do. It should be able to act based on the available inputs and existing business logic – if the systems show a duplicate payment has been made, and the transaction is below a predefined value, then issue the refund.

Involve an LLM in this kind of deterministic decision and you introduce unnecessary risk. What’s more, you waste tokens and inflate your AI costs unnecessarily.

The LLM tax – a hefty price to pay

This hidden overspend – this “LLM tax” – can put a serious dent in a company’s AI budget. McKinsey estimates that a relatively sophisticated chatbot can cost a business up to $500,000 every year in model inference alone.

Ready for the good news? It is possible to reduce unnecessary LLM calls – make sure your AI agents understand the specific context in which they’re operating.

How to reduce the LLM tax

Generalized LLMs learn about businesses from public content on the internet. This means they know, in general terms, what an invoice is and how a supply chain works. But they don’t know how things work within any specific, real-world enterprise.

This is the context model that enterprise AI agents need, if they aren’t to rely on LLMs and guesswork. Built on process data and business knowledge from across systems, applications, devices, and interactions, a context model is a living, digital twin of operations that gives AI agents operational clarity of their business. And when combined with AI-driven decision intelligence, it allows agents to simulate scenarios, predict outcomes, and make informed recommendations.

Once agents have access to this context layer, they don’t need to guess how a business’s processes work. When they need to make a deterministic decision, they can see the steps and dependencies, and cost-effectively apply the existing business logic. And when they need to make a probabilistic decision, they’re able to ground their reasoning in the business’s unique, operational reality.

This leads to more efficient AI usage and stronger AI outcomes. Imagine asking an AI copilot whether you should reorder stock following a demand spike. Operational context is the difference between AI giving a generic answer – “Yes, demand spikes can lead to stock outs. You might want to reorder” – and enterprise AI coming back with a clear risk assessment, and a recommended supplier based on their historical performance.

For example, one of the world’s oldest technology companies is using this technology to gain full transparency into inventory levels and AI-enhanced recommendations about when to stock up, re-allocate, and refrain from buying materials. Doing so has enabled it to reduce excess inventory by 20%.

A leading global distributor of plastics, chemicals, and raw materials increased operational productivity by 20% for a $3 billion business unit by orchestrating AI agents. When a new sales order is created, an AI solution retrieves all the potential carriers and an agent prioritizes the top three options, based on business guidelines like cost and throughput time. Once a human approves the order, confirmations are automatically sent first to the carrier, and then to the customer and internal team members to ensure seamless pickup and delivery.

LLM costs are rising. It pays to address this now.

No business likes wasting money, and if a company is squandering LLM tokens today, the problem’s only going to worsen in the coming months and years. As multi-agent systems become more common, and AI-driven decisions increasingly rely on agent-to-agent conversations, the LLM tax will grow into an ever more painful levy.

The smart move? Focus on Token Taming not tokenmaxxing. Optimize your AI execution to minimize token consumption and maximize business outcomes.