Part 1<\/a> left off. In the SwarmKV post I had two agents reading one document, and I bragged about prefilling once and fanning the KV cache out. Then, in the caveats, I admitted the embarrassing part: every branch\u2019s actual GPU work still ran behind one global mutex. The orchestration fanned out; the\u00a0compute\u00a0lined up single file. Two agents, two turns. Fifty agents, fifty turns. I hand-rolled a lock and called it a day.<\/p>\nThat is fine for a demo. It is a disaster for production, where \u201can agent swarm\u201d means a dozen small specialized models \u2014 a router, a summarizer, a safety checker, a retriever, a pile of tool-callers \u2014 all awake at once, all wanting the same accelerator. You cannot buy each of them an H100 (unless your name is Jensen Huang). You pack them onto one shared GPU and\u00a0hope the scheduler sorts it out.<\/p>\n
So I wanted to answer one blunt question:\u00a0when two agents share one GPU, what does each one actually pay \u2014 and will anything in my cluster tell me?<\/p>\n
Spoiler alert: it costs real milliseconds, it lands almost entirely on the small fast agent, and no, nothing in your cluster will tell you. So I built a tool that does.<\/p>\n
2. Two agents with opposite personalities<\/p>\n
The repo behind this post runs two containerized PyTorch workers that stand in for the two types which you find in basically almost every agent swarm: <\/p>\n
A small, twitchy, latency-sensitive agent\u00a0(fft_worker.py). It runs a continuous loop of big 2-D complex FFTs. Think of it as the router \/ guardrail \/ tool-caller class \u2014 the agents that must answer\u00a0now\u00a0or the whole world starts falling apart.<\/p>\n
A big, steady, compute-hungry agent\u00a0(matmul_worker.py). It runs a continuous stream of large square matrix multiplies \u2014 the GEMM at the heart of a transformer forward pass. This is the heavyweight actually doing the model\u2019s thinking.<\/p>\n
Their entire workload is quite simple for each.\u00a0The FFT worker pre-allocates a 4096\u00d74096 complex tensor and beats on it:<\/p>\n
# —– Pre-allocate tensors —–
\n# Single allocation keeps cuFFT plan creation and allocator traffic out of the per-iteration “elapsed_time“ window on GPU.
\n# “complex64“ matches typical PHY IQ data width; real-only FFT would under-report memory traffic relevant to DRAM contention with GEMM tenants.
\ndata = torch.randn(MATRIX_SIZE, MATRIX_SIZE, device=device, dtype=torch.complex64)
\n# First launches pay JIT\/plan costs; five iterations is a small fixed count\u2014formal steady-state trimming still happens in “generate_results“ \u00a71.4.
\n# Throwaway “fft2“ calls prime instruction and constant caches so timed iterations see repeatable SM occupancy, not driver one-shot spikes.
\nfor _ in range(5):
\n # Assignment to “_“ discards output tensor handle immediately; we only need kernel execution side effects on device resident “data“.
\n torch.fft.fft2(data)
\n# Final sync guarantees no warmup kernel overlaps the first timed iteration\u2019s event pair\u2014critical for CUDA event timing validity \u00a73.
\nsync()<\/p>\n
The GEMM worker pre-allocates two FP32 matrices and multiplies them forever:<\/p>\n
# Matmul needs two operands resident on device; allocating once keeps allocator and paging out of the timed cuBLAS path each iteration.
\n# FP32 is the default training\/inference dtype on Pascal-class GPUs without Tensor Cores; this matches the \u201cGEMM on 1080\u201d narrative in README.
\nA = torch.randn(MATRIX_SIZE, MATRIX_SIZE, device=device)
\nB = torch.randn(MATRIX_SIZE, MATRIX_SIZE, device=device)
\n# cuBLAS autotuning can pick different algorithms across first launches; warmup iterations absorb that non-determinism before “KTS_APP“ lines.
\n# Five repeats mirror FFT worker so cross-tenant comparisons in papers do not confound different warmup depths with silicon interference effects.
\nfor _ in range(5):
\n # Result discarded; peak memory stays flat because output tensor is freed each iteration before timed loop allocates nothing new per iter.
\n torch.matmul(A, B)
\n# Sync closes the warmup window so first “_ev_start.record“ does not overlap trailing warmup kernels on the same default CUDA stream semantics.
\nsync()<\/p>\n
The point was never to build a clever model \u2014 it was to build two GPU citizens with opposite manners and watch them share one room. One finishes in about 3.6 ms and wants to go again immediately; the other takes about 20 ms and just wants to grind. Now put them on the same GPU and ask the only interesting question: who blinks first?<\/p>\n
Both workers are configured by environment variables, so a pod spec can re-tune them without rebuilding the image:<\/p>\n
# —– Configuration (overridable via env vars so pod specs can tune per experiment) —–
\n# “ITERATIONS“ default matches FFT worker so DF numerators\/denominators use comparable sample counts without env overrides in YAML.
\n# Raising iterations lengthens shared-GPU “kubectl wait“; lowering spikes variance in p99 tails used for contention storytelling in “results.md“.
\nITERATIONS = int(os.environ.get(“ITERATIONS”, 800))
\n# “MATRIX_SIZE“ dominates FLOPs per iteration; env override lets you downshift VRAM when MatMul shares 8 GB with FFT co-tenant allocations.
\n# Time-slicing does not partition memory\u2014both pods\u2019 peak allocations must fit one physical card or the slower OOMKill path invalidates the experiment.
\nMATRIX_SIZE = int(os.environ.get(“MATRIX_SIZE”, 4096))
\n# “SLEEP_MS“ defaults slightly above FFT\u2019s 100 ms so two tenants rarely wake in lockstep, spreading scheduler quanta for more realistic interference.
\n# Same caveat as FFT: sleep is between measured iterations and is excluded from “latency_ms_device“\u2014only GPU matmul time is in the sample list.
\nSLEEP_MS = int(os.environ.get(“SLEEP_MS”, 150))<\/p>\n
I guess by this point you realize that nothing here is domain specific. The numbers happen to come from a signal-processing workload next to a matmul, but swap in your own two agents \u2014 one light and deadline-driven, one heavy and steady \u2014 and the story holds. This is a post about workload personalities colliding on one accelerator, not about any one application. <\/p>\n
Timing it without being illusioned<\/p>\n
There is a classic way to benchmark a GPU and get a beautiful yet completely wrong number: time how long it takes Python to launch the kernel. CUDA is asynchronous, so torch.matmul(A, B) returns almost instantly while the GPU is still sweating. Measure that and you\u2019ll be satisfied that your matmul takes only 50 microseconds, and then you\u2019ll start to bang your head wondering why the production is slow.<\/p>\n
The workers don\u2019t do that. They wrap each operation in CUDA events and force a torch.cuda.synchronize() so the clock stops after the kernels actually retire on the SMs:<\/p>\n
# Start epoch immediately before “record“ minimizes gap between \u201cintent to launch\u201d and queue submission for join alignment studies.
\nepoch_ns_start = time.time_ns()
\n_ev_start.record()
\n_ = torch.fft.fft2(data)
\n_ev_end.record()
\ntorch.cuda.synchronize()
\nepoch_ns_end = time.time_ns()
\nlatency_ms_device = float(_ev_start.elapsed_time(_ev_end))<\/p>\n
elapsed_time reads the GPU\u2019s own timeline \u2014 sub-microsecond resolution, no host-side jitter. That synchronize() is the difference between measuring \u201chow long did the GPU work\u201d and \u201chow long did Python take to ask.\u201d Then every iteration coughs up one structured line and flushes it, so Kubernetes log streaming sees it immediately:<\/p>\n
print(
\n f”KTS_APP,v1,FFT,{i},{epoch_ns_start},{epoch_ns_end},{latency_ms_device:.6f},{phase_optional}”
\n)
\n# “flush“ forces line-buffered container stdout through CRI before the next sleep\u2014without it, tail -f can batch lines and scramble join order.
\nsys.stdout.flush()<\/p>\n
Raw silicon execution time goes in; a structured log comes out. A downstream parser aggregates these into exact percentiles, creating a strict measurement contract that strips away all host-side noise.<\/p>\n
3. How two pods end up on one GPU (explained)<\/p>\n
This is the part which will feel like magic for people who are new to K8s. For others, you can safely skip this section and move on to the next one.<\/p>\n
By default, Kubernetes treats nvidia.com\/gpu as a whole, indivisible thing: one GPU, one claimant, no sharing. The NVIDIA device plugin\u2019s time-slicing feature changes the bookkeeping. You hand it a ConfigMap that says, essentially, \u201cpretend each physical GPU is several\u201d:<\/p>\n
apiVersion: v1
\nkind: ConfigMap
\nmetadata:
\n name: time-slicing-config
\n namespace: nvidia-device-plugin
\ndata:
\n any: |-
\n version: v1
\n flags:
\n migStrategy: “none”
\n failOnInitError: true
\n sharing:
\n timeSlicing:
\n failRequestsGreaterThanOne: true
\n renameByDefault: false
\n resources:
\n – name: nvidia.com\/gpu
\n replicas: 4<\/p>\n
replicas: 4 is Kubernetes for \u201clie to the scheduler four times.\u201d After this, one physical GTX 1080 advertises four allocatable nvidia.com\/gpu slots to the API. Four pods can each request “1” and all get scheduled, quite happily.<\/p>\n
Here is the catch, in bold because the entire post depends on it: this does not physically partition the hardware. It is not MIG. There is no memory fence and no compute fence. The four \u201cGPUs\u201d are the same silicon, and the pods take turns on it through CUDA time-slicing \u2014 the GPU context-switches between them like a single barista serving four lines by sprinting between registers. More schedulable slots, exactly zero isolation.<\/p>\n
![\"Two](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/ktp_architecture.png\")
<\/p>\n
The experiment is three Kubernetes Jobs: each agent alone (the baselines), and then both at once. The \u201cboth at once\u201d manifest is the whole ballgame \u2014 two Jobs, each innocently asking for one GPU, deliberately landing on the same card:<\/p>\n
containers:
\n – name: worker
\n image: localhost\/kts-worker:v1
\n imagePullPolicy: Never
\n resources:
\n limits:
\n nvidia.com\/gpu: “1”
\n requests:
\n nvidia.com\/gpu: “1”<\/p>\n
Neither pod knows the other exists. Neither asked to share. The scheduler put them in the same room because, as far as it knows, there were four rooms. The baselines tell you how fast each agent runs when it owns the GPU; the shared run tells you what it pays for company. The gap between them is the entire story.<\/p>\n
4. The rig, in one sentence<\/p>\n
Everything below runs on a seven-year-old NVIDIA GTX 1080 (8 GB, Pascal) on a single-node K3s with the stock NVIDIA device plugin and CUDA time-slicing. No H100, no MIG, no datacenter rack \u2014 just the card half the people reading this still have under their desk.<\/p>\n
I am using this antique on purpose. Bad scheduling doesn\u2019t magically vanish on an H100; it just executes its bottlenecks at a higher clock speed. If your agents are fighting over a memory bus on a $150 card, throwing $30,000 at the problem won\u2019t prevent the traffic jam\u2014it just makes the crash more expensive. Throwing an H100 at an orchestration flaw doesn\u2019t fix the contention; it just lets you execute bad architecture in fewer milliseconds. The physics of cache eviction do not care what year your silicon was minted.<\/p>\n
(Driver, containerd, and toolkit versions are pinned in the repo for anyone reproducing this; they are boring on purpose and you do not need them to follow the story.)<\/p>\n
5. The receipts (i.e., the numbers)<\/p>\n
Now the whole story in one picture:<\/p>\n
![\"Throughput](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/results_grid-1024x645.png\")
<\/p>\n
Four panels, one punchline. The medians (the left pair of bars in each latency chart) are basically untouched. The throughputs (bottom row) lost a measly 7.3% and 1.4% \u2014 the kind of number you\u2019d report up the chain and get a thumbs-up emoji for. And then there\u2019s that top-right corner of the top-left chart: the small agent\u2019s p99 jumped by 66%. Same dashboard, same Running pods, same boring throughput graph \u2014 and one of your two agents is now occasionally, unpredictably 66% slower than it was yesterday. Welcome to GPU sharing.<\/p>\n
The actual numbers, so nobody has to squint their eyes at the bars:<\/p>\n
MetricSoloSharedChangeFFT (latency-sensitive) p503.598 ms3.593 msinsignificantFFT p953.645 ms5.868 ms1.61\u00d7FFT p993.679 ms6.101 ms1.66\u00d7FFT jitter (p99\/p50)1.021.70tail blows outGEMM (heavy) p5020.677 ms20.669 msinsignificantGEMM p9520.896 ms24.505 ms1.17\u00d7GEMM p9920.985 ms24.690 ms1.18\u00d7GEMM jitter (p99\/p50)1.011.20slightFFT throughput (iter\/s)278.1257.9\u22127.3%GEMM throughput (iter\/s)49.148.3\u22121.4%<\/p>\n
Read those FFT rows twice. The median did not move. If you were staring at a p50 dashboard you\u2019d swear nothing happened, sign off, and go to lunch. But one in every hundred FFT calls now takes 66% longer, and the gap between a typical iteration and a bad one nearly doubled. You didn\u2019t slow the agent down on average \u2014 you made it occasionally, unpredictably late. Which is worse, because now it\u2019s a flaky agent and nobody can reproduce it on a Friday afternoon.<\/p>\n
This is the key asymmetry, and it is not a coincidence: the small, latency-sensitive agent degrades first and worst. The big GEMM is a bulldozer \u2014 it grabs its quantum and grinds through. The little FFT keeps getting tapped on the shoulder mid-stride, shoved off the SMs, and told to wait for its next turn. When two workloads share a single line, the one that needed to be quick is the one that suffers. This has huge implications in the telecom domain: if it keeps happening, calls start to drop and worse comes to worst, even emergency service numbers may also stop functioning. Just let that thought sink in!<\/p>\n
To make this comparable across any pair of agents, the tool computes a degradation factor (DF) = shared_p99 \/ baseline_p99. DF = 1.0 means sharing was free. Higher means it hurt. For this run it\u2019s 1.66 for the FFT and 1.18 for the GEMM. That 1.66 is the entire post compressed into a number you can put on a slide to show to your manager.<\/p>\n
And here\u2019s the part that should be illegal: the throughput barely moved. If your SLO (Service Level Objective) is written in terms of average throughput, you\u2019d look at \u201cFFT down 7%, GEMM down 1%\u201d and declare victory. Meanwhile your tail-sensitive agent is silently missing one deadline in fifty. Averages are where contention goes to hide. The mean is a kind soul who rounds your worst moments away. The p99 is the friend who remembers everything.<\/p>\n
One sanity check, then we move on. The profiler also scrapes DCGM GPU-utilization counters every 100 ms and joins them to each iteration. In the shared window, the FFT worker\u2019s SM and DRAM activity rise sharply (its execution cycles now overlap with a GEMM hammering the same memory system); in the solo window, they don\u2019t. So the contention shows up at two completely independent layers \u2014 application latency and hardware counters \u2014 which is how you know this is real and not a stopwatch artifact.<\/p>\n
6. This is about agent swarms, not any one workload<\/p>\n
One could easily label section 5 as \u201can FFT and a matmul fought over a GPU, which surprises absolutely no one who has ever written a CUDA kernel\u201d, but that misses the point entirely. The two workers are just convenient, measurable stand-ins for a pattern that shows up the instant you put a real agent swarm on shared hardware: <\/p>\n
The light, deadline-driven agents \u2014 routers, guardrails, classifiers, tool-callers, small fast models. Cheap individually, constantly running, and the whole pipeline waits on them. (The FFT worker is one concrete example of this personality.) <\/p>\n
The heavy, steady agents \u2014 the big transformer forward passes, the GEMM-bound model calls that dominate compute. (The GEMM worker is one concrete example of that one.)<\/p>\n
Put any two agents with those shapes on one time-sliced GPU and you get exactly what I measured: medians barely twitch, but the small, latency-critical agent eats the tail. It does not matter what the agents do; it matters how they behave on the SMs \u2014 one needs to finish fast and often, the other just wants to grind. Time-slicing hands out turns. It does not hand out deadlines. So the agent that lives or dies by its deadline is the one that suffers when its turn keeps getting interrupted.<\/p>\n
That is the systems thread running through this whole series. Part 1 was about not repeating work across agents (share the KV cache). This part is about not lying to yourself about what sharing the GPU costs those agents. Time-slicing buys you capacity \u2014 more schedulable slots on one card \u2014 and gives you zero isolation. Watch only averages and your most deadline-sensitive agent breaks first, silently, in the p99, while every pod keeps flashing Running.<\/p>\n
7. \u201cSo\u2026 how do I actually run it?\u201d<\/p>\n
The pipeline is deliberately boring, because in systems engineering, \u2018exciting\u2019 usually means production is on fire. It\u2019s a linear build \u2192 cluster \u2192 logs \u2192 metrics graph driven from the repo root:<\/p>\n
run.py builds the worker image with Podman, imports it into K3s\u2019 containerd, makes the namespace, optionally starts a DCGM scrape thread, applies the Jobs, waits, and collects logs into logs\/run-\/.<\/p>\n
The workers emit those per-iteration KTS_APP lines you saw above.<\/p>\n
generate_results.py parses the logs, trims warmup, computes p50\/p95\/p99, the throughput proxy, the degradation factor, and the DCGM join, then writes data\/summary.{csv,json}, the plots, and a docs\/results.md.<\/p>\n
On a node that already has K3s, the NVIDIA driver, the Container Toolkit, the device plugin, and the nvidia RuntimeClass, the whole thing is three commands:<\/p>\n
# 1. Install the time-slicing ConfigMap and reload the device plugin
\nkubectl apply -f time-slicing-config.yaml<\/p>\n
# 2. Build the worker image and run the full benchmark (build, import, Jobs, logs)
\npython3 run.py<\/p>\n
# 3. Turn the logs into summaries, plots, and a results page
\npython3 generate_results.py<\/p>\n
The repo link? well, you can find it near the top of the article. And congratulations that you made it this far \u2013 I hardly thought anyone would ever do!<\/p>\n
8. Honest caveats (because the comments are coming)<\/p>\n
This is a small, deliberate study, not a datacenter capacity model. Here is exactly what it is not, before someone posts it for me:<\/p>\n
It is two agents, not fifty. The config exposes four logical slots; the highlighted run pairs one FFT worker with one GEMM worker. That\u2019s the smallest interesting contention case, picked for clarity. Filling all four slots (the full contention matrix) is on the roadmap, not in these numbers. I am not reporting fifty-agent results, because I did not measure fifty agents.<\/p>\n
Throughput is a mean-rate proxy. 1000 \/ mean latency is an iteration rate, not request-serving throughput under a real arrival process. It earns its keep for the \u201caverages hide the tail\u201d point and nothing fancier.<\/p>\n
The workloads are synthetic. A looping FFT and a looping matmul are honest stand-ins for a light, latency-sensitive agent and a heavy inference agent, but they are not a fully served model behind real traffic. The interference shape generalizes; the absolute milliseconds do not.<\/p>\n
DCGM activity is a low-magnitude proxy. The workers pace themselves with sleeps, so the GPU idles a lot and the SM\/DRAM means look small. Treat them as relative, within-study signals \u2014 they corroborate the latency story, they don\u2019t claim full saturation.<\/p>\n
Time-slicing is not the only sharing mode. As \u00a77 lays out, this study deliberately measures the default path \u2014 the one most people get the moment they flip on GPU sharing. A head-to-head with MPS and MIG is a separate post.<\/p>\n
One GPU class, one run highlighted. Numbers come from a single Pascal GTX 1080. Newer GPUs context-switch faster and the absolute tails shrink; the direction \u2014 small latency-sensitive agent degrades first \u2014 is the durable result.<\/p>\n
None of this moves the takeaway. It just keeps me honest about its scope \u2014 and the moment a benchmark post hides its caveats is the moment its numbers stop being worth anything.<\/p>\n
9. Wrap (and the setup for Part 3)<\/p>\n
Kubernetes time-slicing is a wonderful illusion. It tells your scheduler that one GPU is four, lets four pods report Running, and then quietly locks them in a room to fight over the memory bus. For throughput-bound, deadline-relaxed work, that illusion is harmless and genuinely useful. For the latency-sensitive members of an agent swarm, the illusion hides exactly where you are not looking: the p99.<\/p>\n
The solution isn\u2019t to ban GPU sharing\u2014you have to share hardware, unless you have an infinite budget. The solution is to stop using a green YAML checkmark as a substitute for microarchitectural reality. Measure the tail, attribute the degradation, and schedule with actual silicon limits in mind. Kube-TimeSlice-Profiler is a step towards the right direction: it turns the vague feeling of \u201cthe GPU seems slow today\u201d into a measurable Degradation Factor with receipts.<\/p>\n
If you came here as a beginner who just wanted to know why \u201cboth pods are Running\u201d doesn\u2019t mean \u201cboth agents are happy\u201d: congratulations, you now understand GPU sharing better than the green checkmark does. Go ahead and distrust your averages, you are ready!<\/p>\n
Coming up next: The PCIe Walk of Shame<\/p>\n
![\"\"](\"https:\/\/www.europesays.com\/ai\/wp-content\/uploads\/2026\/06\/part-3-overview-option-2-1024x683.png\")
<\/p>\n
We just survived two agents fighting over a single GPU without lying to ourselves about the latency tail. But there\u2019s another silent tax buried in every RAG pipeline: the PCIe commute.<\/p>\n
Right now, every time an agent needs to retrieve context, it pauses, leaves the accelerator, crawls across the PCIe bus back to Python, runs a vector search on the CPU, and trudges all the way back. <\/p>\n
In Part 3, we are killing that commute. We will build a custom CUDA Top-K kernel to keep the entire retrieval loop trapped on the GPU hardware\u2014no Python round-trips, no host-side delays. Same budget GPU. Same \u201cstop wasting hardware\u201d philosophy. <\/p>\n
See you in Part 3.<\/p>\n
Disclaimer: The illustrations in this article were generated using AI (Claude Opus 4.8). They are illustrative, not photographic, and any labels visible inside the images are stylized rather than authoritative \u2014 refer to the article body and the code itself for precise function names, metric values, and architecture details. <\/p>\n","protected":false},"excerpt":{"rendered":". Production agents fight over the same GPU \u2014 and on one shared card, a latency-sensitive agent\u2019s p99…\n","protected":false},"author":2,"featured_media":73602,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[179,7493,23817,12673,23820,40588],"class_list":["post-73601","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-agentic-ai","tag-agentic-artificial-intelligence","tag-deep-dives","tag-kubernetes","tag-mlops","tag-system-architecture"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/73601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=73601"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/73601\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/73602"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=73601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=73601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=73601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}