![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/08\/41b1b9dc2024a109_1787845187_inline_1.jpg\")
<\/p>\nMost conversations about AI infrastructure start with the model. Red Hat’s performance engineering team thinks that’s the wrong place to look. Speaking on the AI Engineer podcast, performance lead Ashish Kamra and product manager Yuchen Fama make a blunt case: the industry’s serving stacks were built for a workload that barely existed two years ago \u2014 and the mismatch is costing enterprises real money.<\/p>\n
The workload in question is the agentic loop: a model that doesn’t just answer one prompt and stop, but calls tools, reads results, and keeps going \u2014 sometimes for thousands of turns in a single session. Fama and Kamra spent the episode dissecting what that traffic does to a serving system, and walking through the architectural fixes Red Hat has shipped in LLMD, the CNCF-hosted inference framework where the company is a top contributor alongside AMD and NVIDIA.<\/p>\n
The thesis isn’t subtle. Agentic traffic is so dominated by cache reuse and phase interference that the classic “one pod does everything” model is economically obsolete. The fix involves separating prompt processing from token generation, routing requests by cache locality, and measuring success in P99 inter-token latency rather than raw throughput.<\/p>\n
What Actually Happens When Agents Talk to Models<\/p>\n
The episode opens with a taxonomy of agentic workloads drawn from real traces \u2014 SWE-bench sessions, cloud coding logs. The numbers are stark, and they invalidate most capacity-planning spreadsheets.<\/p>\n
Workload characteristicClassic inference assumptionAgentic realityTurn count per session1\u20132Up to 3,000 turnsSystem prompt reuseIncidentalCache hit rates often exceed 90%Input-to-output token ratioNear 1:1Often exceeds 100:1Context lengthStableHighly volatile, client-determinedScheduling priorityThroughputLatency and cache locality<\/p>\n
The consequence is that teams cannot plan around averages. Fama is explicit: capacity planning must account for P90 and full distributions, because the variance is so extreme. He also flags a subtle pathology he calls “sub-session panels” \u2014 patterns where context is partially reused across turns in ways that break naive prefix caching assumptions.<\/p>\n
To help the community study these patterns, Red Hat collaborated with Google and IBM (Red Hat’s parent company) to add a trace replay tool to the inference-perf benchmarking suite. The message is clear: if your benchmark doesn’t reproduce agentic behavior, your benchmark is lying to you.<\/p>\n
<\/p>\n
Layer One: KV Cache-Aware Routing<\/p>\n
The first lever Red Hat pulls is routing. The goal: get each request to the pod that already holds its prefix. LLMD implements this through an “endpoint picker” plugin system that continuously probes each pod’s VM metrics \u2014 running and waiting request counts, KV cache utilization, and prefix cache availability \u2014 then scores each pod on the combination of lowest load and highest probability of a cache hit.<\/p>\n
Fama demonstrated the mechanism live. A first request with a fresh system prompt takes roughly 3 seconds and populates the cache. A second request reusing that system prompt drops to about 1 second and lands on the same pod. A third request with a different system prompt resets to 3 seconds and routes to a different pod.<\/p>\n
The demo is simple. The economics are not. Anthropic’s API pricing, cited in the episode, shows a 10x cost differential between cached and non-cached tokens. That turns KV cache hit rate from a performance nicety into a balance-sheet line item.<\/p>\n
Below the router, LLMD is pushing KV cache management into multi-tier offloading \u2014 NVMe SSD and filesystem tiers, plus KV-centric stores like Mooncake \u2014 and implementing smarter eviction policies, including priority-based eviction and session pinning. The idea: critical agent contexts should persist exactly where they’re needed, not get evicted by a burst of unrelated traffic.<\/p>\n
![\"\"\/](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/08\/41b1b9dc2024a109_1787845268_inline_2.jpg\")
<\/p>\n
Layer Two: Prefill-Decode Disaggregation<\/p>\n
KV cache routing solves time-to-first-token. But the other half of the latency problem \u2014 inter-token latency stability \u2014 requires a more radical move.<\/p>\n
Kamra explains the physics. The prefill phase builds KV caches; it’s compute-hungry, bursty, and thrives on large batch parallelism. The decode phase generates one token at a time; it’s memory-bandwidth-bound, latency-sensitive, and requires high cache residency. Collocating them on one GPU creates what Kamra calls “phase interference” \u2014 a sudden influx of long prefill prompts stalls ongoing decode generation, causing jitter in streaming latency.<\/p>\n
The fix is prefill-decode (P\/D) disaggregation: run prefill and decode on separate worker pools.<\/p>\n
The experimental results are the episode’s strongest evidence. On a GPOSS 12B model with 16 H100s, comparing four aggregated replicas (tensor parallelism 4) against two prefill and two decode workers (also TP4), with multi-turn workloads of 10,000-token prefixes and 128-token turns, the P99 inter-token latency dropped from roughly 900 milliseconds to about 100 milliseconds \u2014 a 9x improvement with dramatically smoother variance.<\/p>\n
A second experiment on 64 H100s with a prefill-heavy workload (5,000 average input tokens, 500 output) showed the P\/D configuration dominating the aggregated configuration across the entire interactivity spectrum. But Kamra is careful not to oversell. As Yuchen puts it:<\/p>\n
“PD is essentially a separation phase separation trade-off and not a magic bullet.”<\/p>\n
When the Architecture Fails<\/p>\n
One of the episode’s most valuable contributions is its honesty about failure modes. Kamra lays out a decision matrix that belongs on every inference team’s whiteboard:<\/p>\n
Consider P\/D disaggregation when…Stick with aggregated serving when…Long context with high ISL-to-OSL ratiosShort or moderate contextLarge models amenable to rich model parallelismAny model size at low concurrencyOperating in the middle concurrency regimeStrict TTFT requirements (tunable on aggregated)Strict ITL streaming requirementsNo high-speed network fabric for KV transferHigh-speed network available (RDMA, RoCE)<\/p>\n
The hidden dependency is the network. KV cache transfer between prefill and decode workers requires advanced fabric \u2014 RDMA or RoCE. Yuchen is direct:<\/p>\n
“If you don’t have the network fabric to support those KV cache transfers, you might actually just want to stick with aggregated.”<\/p>\n
Kamra adds two more caveats. First, P\/D ratios can start static, but they must evolve dynamically with the autoscaler as traffic mixes change. Second, the prefill and decode pools must scale independently \u2014 otherwise you’re just swapping one rigidity for another.<\/p>\n
The GLM 5.2 Case Study: Making It Work on H200s<\/p>\n
The episode’s anchor is an ongoing effort to serve GLM 5.2 \u2014 a model whose impressive public benchmark numbers were achieved on B200 GPUs that most customers simply don’t have \u2014 on clusters of H200s.<\/p>\n
The architecture combines every technique discussed:<\/p>\n
Prefill pool: up to three workers optimized for high throughput with deep batching
\nDecode pool: one dedicated worker optimized for low latency
\nKV transfer: NVIDIA’s NIXL for efficient cache movement between pools
\nWithin-worker parallelism: leader-worker sets with TP1, DP8, and expert parallelism 8<\/p>\n
The modularity is the point. Throughput scales by adding prefill workers without reconfiguring the decode pool.<\/p>\n
The results so far are striking. On a dataset with a 45:1 input-to-output ratio, the prefill-heavy configuration delivered a 4x improvement in pass-through TTFT and 60% more requests served compared to a 2-prefill, 1-decode baseline.<\/p>\n
Fama also surfaced an unexpected finding: BF16 KV cache is actually faster than FP8 KV cache for longer prefill sequences. It’s counterintuitive \u2014 FP8 is supposed to be the efficiency play \u2014 and the team is still exploring why. But it’s a reminder that the gap between benchmark assumptions and production reality cuts in both directions.<\/p>\n
The work is explicitly in progress. Next steps include lowering TTFT further at the upper layer and adding more prefill replicas \u2014 a prediction Kamra makes with medium confidence, expecting further throughput gains from the same architectural direction.<\/p>\n
The Bigger Picture: Benchmarks Are Broken<\/p>\n
The episode’s deepest insight is that the industry’s measurement infrastructure has not caught up with its workloads. Public inference benchmarks show what Fama calls “very steady state, isolated, highly sanitized numbers” that conceal “the chaotic reality of multi-turn interactions, massive context fluctuations.”<\/p>\n
Every technique discussed \u2014 cache-aware routing, P\/D disaggregation, multi-tier KV offloading \u2014 is a response to that gap. And Red Hat’s broader roadmap includes session graph orchestration, program-aware scheduling, state reuse lifecycle management, and new agentic benchmarks. The company is explicitly betting on open collaboration \u2014 with CoreWeave, Google, IBM, and NVIDIA \u2014 as a competitive advantage in distributed inference.<\/p>\n
The tension that remains unresolved is economic and architectural at the same time. The 10x cached-vs-uncached pricing differential means cache hit rate is now a financial metric. But the hardware required to maximize it \u2014 high-speed fabrics for KV transfer, disaggregated pools \u2014 is itself expensive. The H200-vs-B200 gap for GLM 5.2 is a reminder that the frontier of model capability and the reality of enterprise hardware are diverging, and that software orchestration, not just silicon, is where the gap gets closed.<\/p>\n
For anyone tracking the inference market \u2014 investor or engineer \u2014 the episode is a useful corrective to vendor benchmarks. The real performance question is not “what can this model do on a B200?” but “what can this serving stack sustain on the hardware I actually own, under agentic traffic, at the 99th percentile?” The answer, increasingly, is not found in the model card. It’s found in the router, the cache policy, and the network fabric.<\/p>\n","protected":false},"excerpt":{"rendered":"Most conversations about AI infrastructure start with the model. Red Hat’s performance engineering team thinks that’s the wrong…\n","protected":false},"author":2,"featured_media":153732,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[179,7493,3125,53,74898,945,45062,132,74902,2618,74899,74903,74900,24729,74901,10194,74897],"class_list":["post-153731","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-agentic-ai","tag-agentic-artificial-intelligence","tag-amd","tag-anthropic","tag-ashish-kamra","tag-coreweave","tag-glm-5-2","tag-google","tag-gposs-12b","tag-ibm","tag-llmd","tag-mooncake","tag-nvidia-h100","tag-nvidia-h200","tag-nvidia-nixl","tag-red-hat","tag-yuchen-fama"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/153731","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=153731"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/153731\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/153732"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=153731"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=153731"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=153731"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}