Red Hat’s Yuchen Fama: Agentic AI Traffic Is Breaking the Old Rules of LLM Serving — BigGo Finance

Most conversations about AI infrastructure start with the model. Red Hat’s performance engineering team thinks that’s the wrong place to look. Speaking on the AI Engineer podcast, performance lead Ashish Kamra and product manager Yuchen Fama make a blunt case: the industry’s serving stacks were built for a workload that barely existed two years ago — and the mismatch is costing enterprises real money.

The workload in question is the agentic loop: a model that doesn’t just answer one prompt and stop, but calls tools, reads results, and keeps going — sometimes for thousands of turns in a single session. Fama and Kamra spent the episode dissecting what that traffic does to a serving system, and walking through the architectural fixes Red Hat has shipped in LLMD, the CNCF-hosted inference framework where the company is a top contributor alongside AMD and NVIDIA.

The thesis isn’t subtle. Agentic traffic is so dominated by cache reuse and phase interference that the classic “one pod does everything” model is economically obsolete. The fix involves separating prompt processing from token generation, routing requests by cache locality, and measuring success in P99 inter-token latency rather than raw throughput.

What Actually Happens When Agents Talk to Models

The episode opens with a taxonomy of agentic workloads drawn from real traces — SWE-bench sessions, cloud coding logs. The numbers are stark, and they invalidate most capacity-planning spreadsheets.

Workload characteristicClassic inference assumptionAgentic realityTurn count per session1–2Up to 3,000 turnsSystem prompt reuseIncidentalCache hit rates often exceed 90%Input-to-output token ratioNear 1:1Often exceeds 100:1Context lengthStableHighly volatile, client-determinedScheduling priorityThroughputLatency and cache locality

The consequence is that teams cannot plan around averages. Fama is explicit: capacity planning must account for P90 and full distributions, because the variance is so extreme. He also flags a subtle pathology he calls “sub-session panels” — patterns where context is partially reused across turns in ways that break naive prefix caching assumptions.

To help the community study these patterns, Red Hat collaborated with Google and IBM (Red Hat’s parent company) to add a trace replay tool to the inference-perf benchmarking suite. The message is clear: if your benchmark doesn’t reproduce agentic behavior, your benchmark is lying to you.

Layer One: KV Cache-Aware Routing

The first lever Red Hat pulls is routing. The goal: get each request to the pod that already holds its prefix. LLMD implements this through an “endpoint picker” plugin system that continuously probes each pod’s VM metrics — running and waiting request counts, KV cache utilization, and prefix cache availability — then scores each pod on the combination of lowest load and highest probability of a cache hit.

Fama demonstrated the mechanism live. A first request with a fresh system prompt takes roughly 3 seconds and populates the cache. A second request reusing that system prompt drops to about 1 second and lands on the same pod. A third request with a different system prompt resets to 3 seconds and routes to a different pod.

The demo is simple. The economics are not. Anthropic’s API pricing, cited in the episode, shows a 10x cost differential between cached and non-cached tokens. That turns KV cache hit rate from a performance nicety into a balance-sheet line item.

Below the router, LLMD is pushing KV cache management into multi-tier offloading — NVMe SSD and filesystem tiers, plus KV-centric stores like Mooncake — and implementing smarter eviction policies, including priority-based eviction and session pinning. The idea: critical agent contexts should persist exactly where they’re needed, not get evicted by a burst of unrelated traffic.

Layer Two: Prefill-Decode Disaggregation

KV cache routing solves time-to-first-token. But the other half of the latency problem — inter-token latency stability — requires a more radical move.

Kamra explains the physics. The prefill phase builds KV caches; it’s compute-hungry, bursty, and thrives on large batch parallelism. The decode phase generates one token at a time; it’s memory-bandwidth-bound, latency-sensitive, and requires high cache residency. Collocating them on one GPU creates what Kamra calls “phase interference” — a sudden influx of long prefill prompts stalls ongoing decode generation, causing jitter in streaming latency.

The fix is prefill-decode (P/D) disaggregation: run prefill and decode on separate worker pools.

The experimental results are the episode’s strongest evidence. On a GPOSS 12B model with 16 H100s, comparing four aggregated replicas (tensor parallelism 4) against two prefill and two decode workers (also TP4), with multi-turn workloads of 10,000-token prefixes and 128-token turns, the P99 inter-token latency dropped from roughly 900 milliseconds to about 100 milliseconds — a 9x improvement with dramatically smoother variance.

A second experiment on 64 H100s with a prefill-heavy workload (5,000 average input tokens, 500 output) showed the P/D configuration dominating the aggregated configuration across the entire interactivity spectrum. But Kamra is careful not to oversell. As Yuchen puts it:

“PD is essentially a separation phase separation trade-off and not a magic bullet.”

When the Architecture Fails

One of the episode’s most valuable contributions is its honesty about failure modes. Kamra lays out a decision matrix that belongs on every inference team’s whiteboard:

Consider P/D disaggregation when…Stick with aggregated serving when…Long context with high ISL-to-OSL ratiosShort or moderate contextLarge models amenable to rich model parallelismAny model size at low concurrencyOperating in the middle concurrency regimeStrict TTFT requirements (tunable on aggregated)Strict ITL streaming requirementsNo high-speed network fabric for KV transferHigh-speed network available (RDMA, RoCE)

The hidden dependency is the network. KV cache transfer between prefill and decode workers requires advanced fabric — RDMA or RoCE. Yuchen is direct:

“If you don’t have the network fabric to support those KV cache transfers, you might actually just want to stick with aggregated.”

Kamra adds two more caveats. First, P/D ratios can start static, but they must evolve dynamically with the autoscaler as traffic mixes change. Second, the prefill and decode pools must scale independently — otherwise you’re just swapping one rigidity for another.

The GLM 5.2 Case Study: Making It Work on H200s

The episode’s anchor is an ongoing effort to serve GLM 5.2 — a model whose impressive public benchmark numbers were achieved on B200 GPUs that most customers simply don’t have — on clusters of H200s.

The architecture combines every technique discussed:

Prefill pool: up to three workers optimized for high throughput with deep batching
Decode pool: one dedicated worker optimized for low latency
KV transfer: NVIDIA’s NIXL for efficient cache movement between pools
Within-worker parallelism: leader-worker sets with TP1, DP8, and expert parallelism 8

The modularity is the point. Throughput scales by adding prefill workers without reconfiguring the decode pool.

The results so far are striking. On a dataset with a 45:1 input-to-output ratio, the prefill-heavy configuration delivered a 4x improvement in pass-through TTFT and 60% more requests served compared to a 2-prefill, 1-decode baseline.

Fama also surfaced an unexpected finding: BF16 KV cache is actually faster than FP8 KV cache for longer prefill sequences. It’s counterintuitive — FP8 is supposed to be the efficiency play — and the team is still exploring why. But it’s a reminder that the gap between benchmark assumptions and production reality cuts in both directions.

The work is explicitly in progress. Next steps include lowering TTFT further at the upper layer and adding more prefill replicas — a prediction Kamra makes with medium confidence, expecting further throughput gains from the same architectural direction.

The Bigger Picture: Benchmarks Are Broken

The episode’s deepest insight is that the industry’s measurement infrastructure has not caught up with its workloads. Public inference benchmarks show what Fama calls “very steady state, isolated, highly sanitized numbers” that conceal “the chaotic reality of multi-turn interactions, massive context fluctuations.”

Every technique discussed — cache-aware routing, P/D disaggregation, multi-tier KV offloading — is a response to that gap. And Red Hat’s broader roadmap includes session graph orchestration, program-aware scheduling, state reuse lifecycle management, and new agentic benchmarks. The company is explicitly betting on open collaboration — with CoreWeave, Google, IBM, and NVIDIA — as a competitive advantage in distributed inference.

The tension that remains unresolved is economic and architectural at the same time. The 10x cached-vs-uncached pricing differential means cache hit rate is now a financial metric. But the hardware required to maximize it — high-speed fabrics for KV transfer, disaggregated pools — is itself expensive. The H200-vs-B200 gap for GLM 5.2 is a reminder that the frontier of model capability and the reality of enterprise hardware are diverging, and that software orchestration, not just silicon, is where the gap gets closed.

For anyone tracking the inference market — investor or engineer — the episode is a useful corrective to vendor benchmarks. The real performance question is not “what can this model do on a B200?” but “what can this serving stack sustain on the hardware I actually own, under agentic traffic, at the 99th percentile?” The answer, increasingly, is not found in the model card. It’s found in the router, the cache policy, and the network fabric.