Field notes · part 3 · what actually matters · 2 × RTX 4090 · 480 simulations · 15 live runs

I measured four famous LLM serving speedups. The best moved 2.6%. A config mismatch moved 92%.

Tensor parallelism, speculative decoding, a faster attention backend, a different quantization: the standard menu of inference optimizations. I tested each one on a production-like setup, a 32-billion-parameter open model on two RTX 4090s under vLLM, against a fixed stream of 192 recorded requests. None of them cleared even a 20% bar. Then I took a server tuned for one type of traffic and gave it another type, and the cost of serving doubled. This page shows the numbers, and the mechanics behind them, so you can check which side of that mismatch your own stack is on.

For whom: Anyone running open models on their own GPUs, and anyone about to spend a week tuning an inference stack.
Question: Where does serving cost actually live: in the famous switches, in the config dials, or somewhere else?
Method: Pre-registered experiments, predictions committed to git before measuring, negative results reported as negative. Four switch tests, a 96-configuration sweep across five workload types, six live cross-check runs.
Findings: Switches: at most 2.6%, and that one negative. Dials within one workload: a 2.6% spread across the whole grid. Matching the config to the traffic type: up to 2x, and my replay simulator predicted the measured +92% penalty within 7 points before the run.
Use it: A decision guide at the end, plus a memory calculator that answers "will this model even run on one GPU" without booting anything.

Why I ran this. I build a replay simulator that predicts what a vLLM config change will do before it touches production. Parts 1 and 2 validated the predictions and measured the benchmark noise floor. This part uses the validated tool to ask the question everyone actually has: what is worth changing. The page stands alone; the earlier parts are the receipts.

Round one: the four switches

Each of these is a real recommendation from real deployment guides, and each promises a lot. Success bar, fixed before any run: at least 20% throughput change. Press a card to see what the hardware said. Every number is a live measurement against a same-day baseline; throughput reproduces to 0.02% run-to-run on this rig, so these are not noise.

Measured throughput change vs baseline. Orange line = the 20% bar. Press each card. TP=1 is a special case: it produced no number at all, for a reason worth its own section below.

What this means at your desk: on a 24 GB card serving a quantized 32B model, the famous switches are not where your money is. Before copying any "we got 3x with X" post, check what model size, hardware, and traffic it was measured on. Mine is a common self-hosted setup, and on it the whole menu moved low single digits.

Round two: the dials

Maybe the wins hide in the continuous settings: how much GPU memory the cache gets, how many requests run at once, how big a chunk of prompt one step processes. I swept 96 combinations of those dials for each of five traffic types, using the simulator (validated in part 1: cost predictions land within a few percent), and checked the top candidates live.

Within any single traffic type, the frontier is flat. On my agent-style workload the entire feasible field spanned 2.6% in cost, and the stock default sat within 0.6% of the best possible setting. On the chat workload all 96 configurations tied within 0.1%. Tuning dials for a workload you already serve correctly is polishing a coin.

What this means at your desk: if your GPUs are not saturated, your config is probably already fine, and a week of dial-tuning will buy you low single digits. The question with money in it is different, and it is the next section.

Round three: the match

Two kinds of traffic dominate real deployments. Agent traffic: long sessions where every request re-sends the same system prompt and a growing history, so most of each prompt is text the server has already seen. Chat traffic: independent one-shot requests, every prompt new. Servers exploit the first kind with a prefix cache: memory that stores computed prompt state so repeated text is reused instead of recomputed. The catch is that this cache competes for the same GPU memory as everything else, so a config tuned for chat gives it very little. Run the demo both ways and watch the cost meter.

prompt tokens recomputed0 reused from cache0 relative serving cost–

shared prefix in prompt unique text reused (cache hit) recomputed (cache too small or empty)

A toy with real proportions: agent prompts here are ~85% repeated content, like my measured trace (cache hit rate 0.86). Try all four combinations; the mismatch quadrant is the expensive one.

Now the real numbers. I took the config that a chat workload picks (small cache, since chat has nothing to reuse) and served agent-style traffic on it, live. Throughput fell by half: the server spent its time recomputing the same system prompt over and over. Serving cost: +92.3%. And the simulator had predicted +99.6% before the run, 7.4 points off, by the right mechanism: it predicted the cache hit rate would collapse from 0.86 to 0.59, and the measured collapse was 0.86 to 0.60.

What this means at your desk: the expensive failure mode is not a badly tuned server, it is a correctly tuned server whose traffic changed. A product that launched as a chatbot and grew agent features, a shared pool serving a new tenant, a migration that copied last year's config: each of these can quietly double cost with zero errors in any log. Check when your config was last matched against what your traffic looks like now.

The memory wall, predicted to the gibibyte

The tensor-parallelism switch deserves its own story. TP=2 splits the model across both GPUs; TP=1 would run it on one, freeing the other for a second replica. Before booting TP=1 I did the arithmetic and froze it in git: the 18 GB of model weights would consume 90% of the memory budget, leaving less than the minimum the cache needs, so the server should refuse to start. It refused, citing the same numbers: 2.0 GiB of cache needed, my prediction 2.00 GiB. The lesson generalizes, so here it is as a calculator instead of a story.

model weights on disk: 18 GB GPU memory per card: 24 GB memory utilization target: 0.85

weightsruntime overhead (~1.5 GB)left for KV cache

Rough single-GPU arithmetic; the same sum matched vLLM's own refusal message to the gibibyte on my box. KV appetite varies by model family; the verdict bands here assume a 32B-class model needing ~2 GiB for an 8K context.

What this means at your desk: "does it fit on one card" is answerable on paper before any download. And when weights eat most of the card, remember what the leftover sliver was for: the cache from round three. A model that technically boots with a starved cache will serve agent traffic at recompute prices.

The speculation asterisk

One switch deserves an honest footnote instead of a clean null. N-gram speculative decoding (the server guesses the next few tokens from patterns already in the context, then verifies the guesses in one step) did nothing for throughput, and there is a solid reason: my workloads are prefill-dominated, with 7 to 17 tokens of prompt processing per generated token, and speculation accelerates only the generation side. But it cut median response latency roughly in half in my runs. That number carries an asterisk: my synthetic prompts made the guesser artificially perfect, so half is the ceiling, and real traffic with 60-80% guess accuracy will see less. The transferable part: on prefill-heavy traffic, speculation is a latency lever, and it is a throughput lever only where generation dominates. Decide which of those you are buying before you pay the setup cost.

Deciding for your own stack

Find out which regime you are in. GPUs mostly idle between requests: you are arrival-bound, config changes will not move cost, and the useful question is capacity headroom. GPUs saturated: continue.
Profile your traffic before touching settings. One number does most of the work: what share of your prompt tokens repeat across requests (shared system prompts, histories, templates). High share means the prefix cache is your economy; protect its memory. Near zero means cache memory is dead weight you can spend on concurrency.
Match, then stop. Give the cache what your reuse share justifies, verify the hit rate in production telemetry, and leave the remaining dials alone: measured spread across my whole grid was 2.6%.
Re-check the match on a schedule, and after every product change. The 92% failure needs no bug and no alert to happen. Traffic drift is the whole risk; one telemetry graph (cache hit rate over time) watches it for free.
Treat switch claims as claims about someone else's setup. Model size, card memory, and prefill share decide whether TP, speculation, or a backend swap can even matter. Do the memory arithmetic first; it is free and it was exact on my box.