{"id":143861,"date":"2026-08-18T17:22:13","date_gmt":"2026-08-18T17:22:13","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/143861\/"},"modified":"2026-08-18T17:22:13","modified_gmt":"2026-08-18T17:22:13","slug":"agentic-ai-has-a-latency-problem-that-more-compute-wont-solve","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/143861\/","title":{"rendered":"Agentic AI has a latency problem that more compute won’t solve"},"content":{"rendered":"

Half of enterprise AI deployments are missing their own latency targets at peak load. This is the headline finding of Akamai\u2019s State of AI Inference 2026 report<\/a>, which surveyed 200 AI practitioners and found that 82% of organizations say their most critical use cases require end-to-end response times of 500 milliseconds or less.\u00a0A total of 64% of organizations now require end-to-end response times of less than 250 milliseconds for their most important use cases, yet 50% of deployments are failing to meet these latency demands at peak load.<\/p>\n

My colleague Ari Weil, who leads product marketing for our cloud computing business and ran point on that research, summarizes the findings<\/a> well: \u201cThe enterprise AI honeymoon phase is over\u2026 they are hitting the latency wall.\u201d\u00a0<\/p>\n

Agentic workflows aren\u2019t a \u201csingle round trip\u201d<\/p>\n

The latency issue stems from the way agents work. It\u2019s an iterative process, somewhat like a king sending out knights, emissaries, and messengers to conduct the business of the kingdom. There are many comings and goings, not just one person sent on a single round trip.\u00a0<\/p>\n

For instance, when an agent built on a framework like LangChain, CrewAI, or Pydantic AI received a user request, it can fan out into dozens of sequential operations such as a reasoning call, a tool invocation, an API lookup, or a context retrieval. Then an agent may execute another reasoning call to decide what to do with what just came back. Every one of these operations or \u201chops\u201d that must cross a wide-area network to reach a centralized data center adds latency, and a chain of 50 hops can multiply that transport time into seconds on its own, regardless of how fast the model generates tokens.<\/p>\n

In fact, in a paper posted to arXiv in November 2025<\/a>, researchers found that CPU-side processing can account for up to 90.6% of total latency in agentic workloads. In other words, your GPU might finish a reasoning step in a few hundred milliseconds, but then it might have to wait on additional tool call runs to CPUs in distant data centers. This is what causes spikes in GPU idle time.\u00a0<\/p>\n

\u201cMore GPU capacity does nothing for this. You can\u2019t brute-force your way out of a wait state.\u201d<\/p>\n

More GPU capacity does nothing for this. You can\u2019t brute-force your way out of a wait state. This is the part of the conversation the industry keeps skipping, mostly because \u201cbuy more GPUs\u201d is a much quicker fix to suggest than \u201cfigure out where your CPU-bound work is actually executing and why it\u2019s so far from the data it needs.\u201d<\/p>\n

We need new benchmarks to fix the latency issue<\/p>\n

One reason the looming latency wall sneaks up on teams is that they are not looking at the right benchmarks for agentic workloads. Most LLM-serving benchmarks measure tokens per second and GPU utilization on a single box. That\u2019s great if the workload is indeed on a single box (i.e., one model answering one prompt), but that\u2019s not the case with agentic workloads. Those benchmarks don\u2019t address an agentic response that, say, makes a 50-hop chain cross a WAN 4 times to reach 4 separate services.\u00a0<\/p>\n

\u201cStaging may pass the benchmarks because it tests the model, but production tests the whole chain, including every hop your serving engine was never designed to see.\u201d<\/p>\n

That\u2019s where the gap lies: Staging may pass the benchmarks because it tests the model, but production tests the whole chain, including every hop your serving engine was never designed to see.<\/p>\n

The 500ms wall is not a soft target<\/p>\n

This is showing up at scale because agents are moving into production faster than most teams\u2019 architecture is evolving to support them. LangChain\u2019s State of Agent Engineering 2026 survey<\/a> of more than 1,300 professionals found that 57.3% of organizations now have agents running in production, up from 51% a year earlier. Among those builders, latency has become the second-most-cited barrier to production, behind only output quality.\u00a0<\/p>\n

This is a serious issue for application teams. The 500ms threshold in Akamai\u2019s survey isn\u2019t a performance goal teams can afford to miss. For a live customer interaction or a real-time compliance check, that 500ms determines whether the application works or it doesn\u2019t.\u00a0<\/p>\n

We\u2019ve solved this problem before<\/p>\n

There\u2019s a reason this feels familiar to anyone who was building for the web in 1999. Akamai exists because of a nearly identical problem. MIT researchers Tom Leighton and Danny Lewin founded the company to answer a challenge posed by Tim Berners-Lee: fix what the press had started calling the \u201cWorld Wide Wait,\u201d the crushing latency of pulling every request back to a small number of centralized servers. When the trailer for The Phantom Menace crashed sites across the internet in 1999, the culprit was distance: millions of browsers all reaching for the same far-away origin server at the same moment. The fix moved content to thousands of points closer to the people requesting it, instead of trying to build a faster origin.<\/p>\n

Agentic AI is running into the same wall, just in a different vehicle. AI works just fine on centralized inference if you\u2019re talking about running batch jobs overnight. But today\u2019s applications built on agentic AI are real-time loops sitting inside live transactions, and the fix for agentic lag is distribution. Instead of expanding racks of CPUs and GPUs at the center, we need to move agentic execution to where the model\u2019s tools, context data, and users actually live.<\/p>\n

Agentic AI needs a tiered architecture, not a bigger data center<\/p>\n

In practice, agentic AI requires a tiered architecture, one that includes a centralized core, regional GPU clusters, and CPUs at the Edge.\u00a0<\/p>\n

Centralized core\u2014perfect for heavy reasoning over large context windows, where the round trip to a large model matters less than the model\u2019s raw capability.<\/p>\n

Regional GPU clusters, increasingly built on hardware like NVIDIA\u2019s Blackwell platform\u2014ideal for localized inference, so the heaviest compute sits closer to where demand actually concentrates.<\/p>\n

Edge CPUs\u2014the essential component for speed. This is the nexus for tool execution, orchestration, and context retrieval, since these are the steps that happen most often in a chain and benefit most from sitting next to the data and APIs they call.<\/p>\n

We\u2019ve built Akamai Inference Cloud around this tiered framework. It\u2019s the same distribution logic behind our AI Grid Orchestrator. We route CPU-bound orchestration and tool calling to the edge, and keep GPU-bound reasoning where it makes sense, regionally or centrally.\u00a0<\/p>\n

What to demand before you commit<\/p>\n

The good news is you don\u2019t need to distribute every workload to the edge on day one. But before you commit to a production architecture, you should know which of your agent\u2019s dozens of hops are latency-sensitive and which aren\u2019t. Then build a defined performance budget for each one.<\/p>\n

\u201cThe teams that treat it as a GPU-shopping decision will be back here in six months, staring at the same four-second response time, wondering why more compute didn\u2019t help.\u201d<\/p>\n

My advice is this: Before signing off on a large-scale inference deployment, ask your infrastructure for four things:\u00a0<\/p>\n

Portability across regions and providers<\/p>\n

Elasticity to absorb peak load without falling over<\/p>\n

Data locality so tool calls aren\u2019t crossing oceans to reach the context they need<\/p>\n

A performance budget you\u2019ve actually tested against production traffic, not staging traffic.<\/p>\n

The teams that address this infrastructure decision now will be the ones whose agents still work when the benchmark environment transitions to real users. The teams that treat it as a GPU-shopping decision will be back here in six months, staring at the same four-second response time, wondering why more compute didn\u2019t help.<\/p>\n

\t<\/p>\n

\n\t\t\t\tYOUTUBE.COM\/THENEWSTACK\n\t\t\t<\/p>\n

\n\t\t\t\tTech moves fast, don’t miss an episode. Subscribe to our YouTube
\n\t\t\t\tchannel to stream all our podcasts, interviews, demos, and more.\n\t\t\t<\/p>\n

\t\t\t\tSUBSCRIBE<\/p>\n

\t<\/a><\/p>\n

Group
\n Created with Sketch.<\/p>\n

\t\t<\/p>\n

\t\t\t\t\t<\/p>\n

\n\t\t\t\t\t\t\tJon Alexander is Senior Vice President of Product for the Cloud Technology Group at Akamai. He is responsible for the strategy, roadmap, and success of the cloud computing and delivery products. Alexander joined Akamai in 2017 and led various product…\t\t\t\t\t\t<\/p>\n

\t\t\t\t\t\tRead more from Jon Alexander\t\t\t\t\t\t<\/p>\n

\t\t<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Half of enterprise AI deployments are missing their own latency targets at peak load. This is the headline…\n","protected":false},"author":2,"featured_media":143862,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[179,7493,21726,4898,15061],"class_list":["post-143861","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-agentic-ai","tag-agentic-artificial-intelligence","tag-akamai","tag-post","tag-sponsored"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/143861","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=143861"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/143861\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/143862"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=143861"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=143861"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=143861"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}