You make GPU decisions on one benchmark run. I ran the same one 14 times.
If you run open models in production, you make real decisions from benchmark numbers: which server settings ship, whether a change regressed latency, how many GPUs to buy. Nearly every such number, including the ones in public benchmarks and vendor blogs, comes from running the thing once. So I took one production-like setup, a 32B open model on two RTX 4090s under vLLM, fed it the same fixed stream of 192 chat-style requests, and ran that identical benchmark 14 times in a row, restarting the server each time. Then I compared the 14 answers. For the number that tells you how long your slowest users wait, up to 18% of any single answer is luck.
- For whom
- Anyone who benchmarks vLLM or reads its benchmarks: platform teams picking serving configs, engineers gating CI on latency numbers, anyone sizing GPU capacity from a test run.
- Question
- When a serving benchmark reports a number, how much of it survives running the exact same thing again, on the same machine, with the same arguments?
- Method
- Pre-registered repeat batches: 14 repeats of two configs, 8 of two more, full restart and strict VRAM drain per run, no outlier rejection. Plus 29 instrumented boots and a client-dispatch calibration.
- Findings
- Throughput and cache metrics reproduce to 0.02%: trust them from one run. Sub-second tail latency needs 4 to 9 repeats before the number means anything. One config turned out not to be noisy at all: it is bistable, with two discrete outcomes and nothing in between. And the server's own memory pool varies by ~5% across identical boots.
- Use it
- Five concrete rules at the end of this page. If you change nothing else: never gate a decision on a sub-second p95 from a single run, and read your granted KV pool from the log of the run that actually executed.
The run that started it
Here is the decision this article is actually about. A team changes a serving config, benchmarks it once, sees p95 latency move 15%, and ships or reverts on that number. A CI pipeline gates a release on the same kind of single measurement. A capacity plan divides a GPU budget by a throughput figure from one test run. Every one of those decisions assumes the number would come out the same if you ran it again. I had the same assumption, until one setup broke it in my face.
During my simulator-validation work, one setup refused to cooperate. Two server setups matter for this story, and I will call them by the labels from my series: A, the baseline, a comfortable memory allocation; and J, the identical server with the memory it reserves for its cache trimmed by about 4%. J's first-token tail (the time the slowest 5% of requests wait before their reply starts) measured 0.705 s in the canonical run and 0.834 s in a re-run of exactly the same thing: an 18% swing, while throughput, cache efficiency and total latency reproduced to 0.14% on the same pair of runs. One published FAIL verdict would have been a PASS against the re-run. Either the benchmark was untrustworthy, and with it every verdict in the series, or something specific was going on with that one number.
So I pre-registered a plan (analyses fixed before any run, no outlier rejection, published verdicts stand regardless) and ran config A and config J fourteen times each, alternating, with a full server restart, a strict VRAM drain and an asserted memory pool before every single run.
Run your own benchmark
The fourteen measured values of config J's ttft_p95 are loaded below. Each press of the button draws one of them at random: one press is one single-run benchmark, the kind most published numbers come from. Watch what your report would have said at each point, and what shape emerges if you keep going.
The toy reproduces two properties of the real data. A single run lands up to 14% from the reference mean, in either direction, with total confidence. And with enough draws the histogram refuses to fill in the middle: the section on bimodality below explains why the middle stays empty.
The numbers: what reproduces and what does not
| metric | config A CV | config J CV | single run trustworthy? |
|---|---|---|---|
| throughput_tok_s | 0.02% | 0.02% | yes |
| prefix_cache_hit_rate | 0.00% | 0.00% | yes |
| e2e_p50_s | <0.2% | <0.2% | yes |
| e2e_p95_s | <0.2% | <0.2% | yes |
| ttft_p50_s | low | low | mostly |
| ttft_p95_s | 0.69% | 6.07% | no |
| you want a ttft_p95 claim within | config A needs | config J needs |
|---|---|---|
| ±10% | 2 repeats | 4 repeats |
| ±5% | 3 repeats | 9 repeats |
There is also a subtler trap I hit and named the repeatable-but-weakly-estimating paradox. A bootstrap confidence interval on the p95 computed within one run (182 requests) is an order wider than the run-to-run spread. The benchmark repeats beautifully and still estimates the true tail poorly: the fixed trace pins the sampling noise in place rather than eliminating it. Practical consequence: comparing two configs on the same trace is sound, quoting an absolute p95 as the truth about your service is not.
Noise vs pressure: the ladder
The obvious hypothesis: noise grows with memory pressure. I pre-registered it, extended the ladder to two more pool sizes, and the data declined the hypothesis.
The spike: a bistable system, not noise
Sorted, config J's fourteen tail values are two clusters with an empty 126 ms gap between them: 0.700 and 0.702 on one side, twelve values between 0.828 and 0.842 on the other. Nothing in the gap, ever. Variance statistics describe this as CV 6.07%. The distribution describes it as a coin with two faces.
Per-request forensics on the fourteen dumps found the switch: 3 requests out of 182 carry the difference between the modes, the largest moving by 0.308 s, while the remaining 179 sit flat within 4 ms. Config J is the memory setting where cache eviction just begins to bite: with 4% less cache, the server sometimes has to discard a session's stored context to make room. A borderline request then either finds its cached history or recomputes it from scratch, a fixed chunk of extra work, and the race between arriving requests and eviction resolves that coin on each run. Two runs in fourteen it landed heads.
This reframes what a tail number near a capacity threshold even is. It is a distribution over modes with probabilities, and any tool reporting a single value there, mine included through run 8, is reporting one flip of the coin. That sentence is now in my audit spec, and it is the single most transferable finding of the batch.
The boot lottery
While tightening the repeat protocol I caught the server itself varying. Identical arguments, identical machine: the KV pool (the slab of GPU memory the server sets aside at startup for its attention cache, its main capacity parameter) came out ~5% different between boots. The button below replays the measured behavior.
Mechanism: the first boot of a new (batch-budget, sequence-cap) shape runs torch compilation and CUDA-graph capture while the memory profiler is sizing the pool, so the profiler sees a busier GPU and grants less. The compile cache persists on disk; every later boot of that shape is granted the larger pool. It is silent, it survives VRAM draining, and anyone sizing capacity from a first deployment inherits an uncontrolled ~5% term. Workaround: boot each new shape once, discard it, then measure. I drafted this as an upstream issue with reproduction steps.
The small print of the same batch: client dispatch granularity measured at 0.1 ms per request, meaning "simultaneous" arrivals in a trace reach the server as a 0.1 ms staircase. Negligible against a 0.85 s prefill step, and I know that because I measured it before it could become an excuse.
Did any of part 1's verdicts sit inside the noise?
After measuring the noise I went back and scored every claim my part-1 series had published against the measured noise band of its own metric, with the plan pre-registered before the repeats ran. The same audit works on any benchmark suite you rely on.
| outcome | rows | reading |
|---|---|---|
| failures inside the noise band | 0 of 71 | every recorded MISS exceeds the harness's own reproducibility |
| passes inside the noise band | 11 of 82 | gaps too small to resolve; not evidence of accuracy |
| failures clearing the band by <2× | 2 | run-5 H (1.54×) and run-6 J (1.34×) rest on differences this benchmark can barely resolve |
The errors that turned out to be real
A clean instrument is what lets you convict the model. Once the noise floor was measured (the baseline setup's tail reproduces to 0.69%), the 20-50% tail errors that survived part 1 could no longer be blamed on measurement, and two more rounds found their true cause: my simulator issued every request's first token one engine step later than the real server does. Removing that one accounting error collapsed the worst prediction from +52.9% error to +4.8%, moved a case that had sat bit-identical through four earlier fixes, and produced two consecutive blind-predicted setups with every number inside 5.1% of measured.
And measured honestly, the fix also overshot four cases that were already nearly right, so the residual ±10% scatter is the next, smaller object of study, resolvable only because the noise floor beneath it is 0.69%. None of that was resolvable before the noise floor under it was measured.
How this changed the way I optimize
Everything above is about measurement, but the reason to measure is to change things. Here is the working order these results pushed me into, different in three places from how I would have optimized a serving stack a month ago.
1. Noise floor first, optimization second. Any "improvement" smaller than your run-to-run spread is a story you told yourself. My floor turned out to be 0.02% on throughput and up to 6% on the tail, which means a 3% throughput win is real from one run and a 10% tail win might be nothing from three. Measuring the floor took one night and now prices every subsequent experiment.
2. Hunt order-of-magnitude switches before turning dials. In my sweep of 256 configurations, the entire feasible field of memory and batching settings spanned 2.6% in cost, and the default sat within 0.6% of the best. One binary switch, prefix caching, was worth 2x throughput on the same workload. The expensive mistake is not a mistuned dial; it is a week spent on dials while a switch sits unflipped. Check the switches (caching, quantization, engine version), then decide whether the dials are even worth your time.
3. Ask whether your bottleneck is configurable at all. My workload turned out to be arrival-bound: the server finished work faster than requests arrived, so no configuration could improve cost, and the sweep proved it in an evening of CPU time. The question that actually had money in it was capacity: how much more load fits in the same hardware inside the SLO. If your GPUs sit under 90% utilisation, you are likely in the same position, and config tuning is the wrong aisle of the store.
4. Write the expected effect down before flipping anything. Not for ceremony. Across ten rounds, three of my confident causal stories about my own system were wrong, and I only know that because the expectation was on record before the result. A one-line prediction in the PR description ("this should cut p95 roughly 20% and leave throughput alone") costs nothing and converts every change into an experiment that can teach you something when it disagrees.
5. Iterate at the speed of the metric. Cost metrics reproduce from single runs, so cost experiments can run as fast as you can restart the server. Tail claims need 4 to 9 repeats, so tail experiments belong in overnight batches, not in interactive loops. Splitting the two cadences, instead of running everything at the cautious speed, is where most of my throughput as an experimenter came from.
If you benchmark vLLM, take these five
- Repeat, then report the distribution. Sub-second tail percentiles need 4–9 repeats before the number means anything. Medians and throughput you may trust from one run.
- Assert the granted pool, never assume it. Read KV capacity from the startup log of the run that actually executed, and reject boots that got a different pool than asserted.
- Discard the first boot of any new shape. The compile cache makes boot one a different machine than boot two.
- Drain VRAM between runs and gate on real idle. Crashed engines leak workers that poison the next measurement.
- Near capacity thresholds, expect modes, and report them as modes. A bistable tail averaged into one number is a fiction with error bars.