{"id":141521,"date":"2026-08-16T15:13:12","date_gmt":"2026-08-16T15:13:12","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/141521\/"},"modified":"2026-08-16T15:13:12","modified_gmt":"2026-08-16T15:13:12","slug":"agentic-ai-crunch-creates-cpu-comeback","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/141521\/","title":{"rendered":"Agentic AI Crunch Creates CPU Comeback"},"content":{"rendered":"

Earlier this year, leaders at Amazon Web Services<\/a> delivered a new mandate to their engineers: they need to conserve CPU cycles at all costs. AWS has reportedly<\/a> experienced an explosion in wait times for CPU server capacity as AI workloads strain the company\u2019s cloud infrastructure. <\/p>\n

The issue seemingly took AWS off-guard, and for good reason. The AI boom led to a surge in demand for GPUs<\/a> and, later, memory<\/a>. CPUs were mostly left out of the story, as their relative lack of parallelization made them a poor fit for AI model inference, the process of running and serving large language models<\/a> (LLM) to users. <\/p>\n

But the rise of agentic AI systems, which allow AI models<\/a> to operate autonomously and call on sub-agents, is changing the narrative.<\/p>\n

Matt Kimball<\/a>, vice president and principal datacenter analyst at Moor Insights & Strategy<\/a>, says 2026 has brought a spike in CPU demand, much of it due to agentic AI<\/a>. \u201cIt\u2019s one thing to have this agentic workload, and let\u2019s say, it spawns 100 agents. If I\u2019m going to roll this out across my enterprise, those 100 become tens of thousands, hundreds of thousands, or millions of agents,\u201d says Kimball. \u201cYou have agents spawning sub-agents, making [application programming interface (API)] calls, and talking to more agents through [Anthropic\u2019s] model context protocol.\u201d<\/p>\n

AI agents need to use computers, and computers need CPUs<\/p>\n

Kimball\u2019s comments refer in part to \u201ctool use,\u201d which is shorthand for an LLM\u2019s ability to access the internet, open files on a desktop, and generally use a variety of software to accomplish its task. <\/p>\n

LLMs trained for tool use learn how to call on other software. While the LLM\u2019s inference is still primarily executed on a GPU or similar AI accelerator, the tool calls the LLM makes are typically pushed to the CPU.<\/p>\n

\u201cMany components of an agentic AI task are inherently CPU based jobs,\u201d explains Souvik Kundu<\/a>, senior staff research scientist at Intel<\/a>. \u201cThe CPU does the job of parsing output, figuring out which tool to invoke, making the API call or running the code, collecting the result, and feeding it back.\u201d Madhu Rangarajan<\/a>, vice president of compute and enterprise AI at AMD<\/a>, makes a similar claim, saying \u201cin our testing, seven of the eight stages in realistic agentic AI pipelines run entirely on the CPU.\u201d<\/p>\n

An LLM tasked with programming software, for example, will likely make tool calls to write code to files, move or replace files, download required packages, and build the software once the LLM believes it\u2019s complete. <\/p>\n

Kundu co-authored a paper<\/a> on agentic AI optimization alongside researchers from the Georgia Institute of Technology. They found the CPU is often idle while LLM inference is executed on a GPU and that, conversely, the GPU is often idle when tool calls are executed on the CPU. To optimize this, Kundu and his colleagues propose scheduling optimizations that can cut end-to-end latency (the time between the start and finish of the agentic workload) by up to 1.8x under sustained load. <\/p>\n

It\u2019s a start, but the gains chase a moving target. Agentic systems generate work at machine speed and multiply it as they go. OpenAI\u2019s inadvertent hack<\/a> of Hugging Face saw its model fire off as many as 300 actions an hour, and a single agent can spawn sub-agents that make tool calls of their own. <\/p>\n

And there\u2019s one more important complication that may increase the workload on a CPU as models become more complex: safety guardrails.<\/p>\n

Safety and policy checks on an agent\u2019s actions are often specific rules that inspect syntax and log files, Kundu says. Guardrails may also use small models (under a billion parameters) to analyze task complexity or intent. Though they could be executed on a GPU, they often aren\u2019t, because their small size and the need to minimize latency keeps the work on the CPU.<\/p>\n

\"A Increasing the number of CPUs available significantly decreases the latency for Llama-8B responses over longer sequence lengths.Source: Euijun Chung, Yuxiao Jia, et al.<\/a><\/p>\n

Tokenization adds to bottlenecks<\/p>\n

Euijun Chung<\/a>, a PhD student at the Georgia Institute of Technology, recently co-authored another paper<\/a>, with findings that complement Kundu\u2019s work. Chung and his co-authors found that when a server has too few CPU cores, it falls behind on dispatching work to the GPUs<\/a>. That causes the GPUs to stall as they wait for instructions.<\/p>\n

In addition to that, the paper touches on another key element of LLM workloads: tokenization<\/a>.<\/p>\n

Tokenization is a key first step in LLM inference. It converts text into integer token IDs that can be processed by the model. Unlike the matrix math required for most LLM inference, tokenization is branchy, data-dependent sequential string manipulation. Though it can be parallelized by chunking text, it\u2019s not massively parallel in the same way as the bulk of LLM inference is.<\/p>\n

Tokenization of small prompts is a relatively trivial task and won\u2019t tax even an entry-level CPU. However, an agentic model that makes tool calls must parse and tokenize the results of the call. <\/p>\n

\u201cIf you have an ongoing sequence of, say, 100,000 tokens, and you have a tool result of a 1,000 tokens, the tokenizer will have to tokenize the whole sequence again. And you have to do tokenization at every agentic tool call,\u201d says Chung. This both increases the frequency of tokenization and increases the number of tokens involved. It\u2019s conceivable that future tokenizers will find ways to mitigate this, Chung says, but it remains a problem for modern LLM inference.<\/p>\n

The paper finds that time-to-first-token latency (the time required for the model to produce the first word of its reply) can increase dramatically as the sequence length grows. CPUs with more cores can reduce the problem. In test runs at longer sequence lengths, increasing CPU core counts can reduce time-to-first-token latency by roughly 1.5x to 7x. <\/p>\n

Chung and his colleagues were only able to test smaller models, such as Alibaba\u2019s Qwen 3-30B and Meta\u2019s Llama 3.1-70B, due to limitations of the hardware available for testing. He speculates that larger models will experience less dramatic bottlenecks due to their higher overall GPU demand, but also expects agentic AI will push token lengths far beyond the longest he and his co-authors tested.<\/p>\n

\u201cIf you think about something like Anthropic\u2019s Claude, you can easily hit 500,000, even a million tokens,\u201d says Chung. \u201cIn the world of agentic AI, the average sequence length will grow and grow, so I\u2019m expecting this problem to get worse in future workloads.\u201d <\/p>\n

Is a CPU crunch just getting started?<\/p>\n

Amazon\u2019s crackdown on use of CPU resources is one of several indicators that Kundu and Chung have identified issues with real-world relevance. <\/p>\n

Intel<\/a> has sold out<\/a> of server CPUs through at least the end of the year. AMD<\/a> has doubled<\/a> its server CPU forecast. Arm<\/a> and Qualcomm<\/a> have both announced new CPUs designed to accelerate agentic AI. Even Nvidia<\/a> has prioritized Vera<\/a>, its Arm-based CPU for agentic AI, which is part of Nvidia\u2019s Vera Rubin<\/a> platform.<\/p>\n

Kimball says these developments make it clear that the AI industry is placing more emphasis on CPU performance. He sees the surge in demand as an \u201cabsolute tell\u201d that CPUs are now considered a key part of an agentic AI system.<\/p>\n

Unfortunately, this may translate to broader CPU shortages and increased prices, much as has already occurred with GPUs and memory. <\/p>\n

\u201cYou\u2019re already seeing a CPU crunch to some degree. When you look at the constraints in the market, it even trickles down into the consumer space,\u201d says Kimball. He adds that Intel has cut production<\/a> of client CPUs in favor of server CPUs even as Intel\u2019s new 18A production process has grown the company\u2019s sales in the client segment. Kimball sees that as a sign that CPU makers will follow the money. <\/p>\n

From Your Site Articles<\/p>\n

Related Articles Around the Web<\/p>\n","protected":false},"excerpt":{"rendered":"Earlier this year, leaders at Amazon Web Services delivered a new mandate to their engineers: they need to…\n","protected":false},"author":2,"featured_media":141522,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[179,7493,4104,2010,2225],"class_list":["post-141521","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-agentic-ai","tag-agentic-artificial-intelligence","tag-cpu","tag-gpu","tag-llms"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/141521","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=141521"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/141521\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/141522"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=141521"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=141521"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=141521"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}