Predicting vLLM config changes from request logs
Changing a production serving config is a decision most teams still make blind: try it on live traffic and watch what breaks, or leave a working config alone and pay for the headroom. Over two days I tested a third option: replay request logs through an offline model of the engine and ask it what a change will do before anything touches the cluster. This page is the full log of that test, including every prediction that turned out wrong.
- Goal
- Find out whether a config change's effect on cost and latency can be predicted from request logs alone, accurately enough to act on. And where prediction breaks, find out exactly why.
- Method
- Replay a fixed workload through a calibrated model of vLLM. Predict configs the model has never seen, commit the predictions to git, then run them for real. Seven rounds, eleven configurations, seven of them held out.
- Result
- Cost and cache efficiency: predictable within a few percent, zero misses across every held-out config. The first-token latency tail: resisted every fix for four rounds.
- Diagnosis
- Per-request forensics traced the entire tail residual to one small group of requests and two specific causes. Both are understood; neither is fixed yet.
- Takeaway
- The physics model was never the problem. The harness that measured its coefficients was, three separate times. Calibrate on the exact path your traffic takes.
The setup
Every team serving open models in production faces the same decision problem. The serving stack has a dozen consequential knobs: how much memory the cache gets, whether prefixes are reused, how many requests run at once, how big a chunk of prompt one step may process. Each knob moves cost and latency, sometimes by a factor of two. And the only trustworthy way to learn what a knob does is to turn it on production traffic, because a staging cluster at production scale doubles the GPU bill and a smaller one lies about contention. So most configs are set once, by folklore and defaults, and then left alone while the headroom quietly costs money.
The experiment's goal: determine whether an offline replay of request logs can stand in for the production experiment. Concretely, whether a simulator can predict the effect of a config change it has never seen, on every metric an operator cares about, within 15 points of the measured change. And where it cannot, the goal is a named mechanism: which requests are mis-priced, and why. A predictor earns trust once its failure modes have names.
The simulator is a discrete-event model of the engine in a few hundred lines of Python: chunked prefill under a token budget, continuous batching, paged KV blocks, a refcounted prefix cache, preemption by recompute. Step time is a calibrated linear model with four coefficients. The workload is a deterministic agentic trace: 192 requests, 24 sessions, a shared 1,200-word system prompt, histories growing over eight turns. The same trace replays against the simulator and against a live server (Qwen3-32B-AWQ on two RTX 4090s, tensor parallel 2, vLLM 0.28).
The machinery, in plain terms
If you have never looked inside an LLM serving engine, this section is enough to follow everything below. Readers who live in vLLM internals can skip to the run ledger.
Requests arrive
Each request is a prompt (often thousands of tokens: a system prompt, tools, chat history) plus a reply to generate.
In agent workloads most of the prompt repeats: every request in a session starts with the same system prompt and the same growing history.
The engine loop
GPU memory
The saved state is the KV cache. It lives in fixed-size blocks (16 tokens each) in a pool that shares the GPU with model weights.
The trick that makes agent serving affordable is the prefix cache: if a new request starts with token-for-token the same content as blocks already in memory, the engine reuses those blocks instead of recomputing them. A cached system prompt means a request pays prefill only for what is new. But cached blocks occupy the same pool that active requests need, so when memory runs short the engine evicts cold blocks, and an evicted prefix must be recomputed by whoever needs it next. Every axis of this experiment pushes on that trade somewhere.
What a config change actually turns
How much GPU memory the KV pool gets. Bigger pool: more cached prefixes survive. Smaller: eviction, recompute, queueing.
configs C, E, K, F, J, A, HTurning reuse off entirely: every request pays full prefill. The most expensive switch in this workload.
config B (throughput fell by half)How many requests may decode at once. Predicted, correctly, to change nothing at this load.
config DHow many prompt tokens one engine step may chew. Bigger chunks: fewer, longer steps.
config GTight pool and big chunks together, built as a worst case. It passed; the quiet configs failed.
config IHow the simulator fits in
The simulator never runs the model itself. It plays through the engine's bookkeeping and charges each step from the calibrated price list, which is why a full config comparison takes seconds on a laptop instead of GPU-hours.
Try the trade yourself
A toy version of the block pool, deliberately tiny: four sessions, three turns each, arrivals interleaved. Shared prefix in blue, each session's history in green, generated tokens as stripes. Run it with a large pool, then a small one, and watch what eviction does to later turns. The pool axis of the experiment, and the "eviction cliff" the forensics section ends with, are this mechanism at production scale.
Small glossary
- TTFT
- Time to first token: how long a request waits before the reply starts. Its p95 is the "first-token tail" this log keeps fighting.
- p95
- The value the slowest 5% of requests exceed. Tails, not averages, are what SLOs are written against.
- KV cache
- The saved attention state of processed tokens. Re-read on every decode step, so it is the memory that costs money.
- block
- A 16-token page of KV cache. The pool is counted in blocks; the whole experiment varies that count.
- prefix cache
- Reuse of blocks whose content exactly matches a new request's beginning. Hit rate = share of prompt tokens served from cache.
- eviction
- Dropping cold cached blocks when the pool runs short. The evicted content is recomputed on next use.
- preemption
- Pausing a running request and recomputing it later because memory ran out mid-generation.
The run ledger
Each run changed one thing and pre-registered what should happen. Select a run.
Three hypotheses, three funerals
The fixes mattered less than the funerals. Each dead hypothesis was replaced by a measurement, and the measurement pointed somewhere no one was looking.
"The missing context term explains the e2e gap"
Recovering c_kv bought 1.0 point of an expected 20+. The real cause: a constant 5.6 ± 0.4 ms per decode step, invariant across pools and prefill budgets. Constants do not come from mis-fitted coefficients.
"Queue build-up behind long prefill steps"
Tested with a config built to maximize it: 8,192-token prefill budget crossed with a tight pool. The config built to fail passed at −10.6%. Two quiet configs kept failing. Wrong hypothesis, by its own test.
"Eviction order causes the tail overshoot"
Leaf-first eviction reached the cache measurably (hit rate exact to three decimals on half the set) and never reached the tail: four configs bit-identical under two opposed cache changes. The tail is not a cache phenomenon.
What got solved: cost
Seven held-out configurations across five axes: pool size, scheduler limits, prefill granularity, and axes crossed. Zero cost misses in the series. The chart below is the cache model against reality across every pool size tested; hover the points.
| config | axis | throughput sim/real | err | hit-rate gap |
|---|
What resisted: the first-token tail
One metric carried every remaining miss. Toggle the simulator version: two opposed cache changes between v0.5 and v0.7, and the failing rows do not move.
| config | real ttft p95 | abs error | moved vs prev |
|---|
The forensics
After run 7 the right move was to stop changing physics and look at individual requests. Per-request instrumentation on the four resistant configs answered the first question immediately: the top-5% slow sets match between simulator and engine, 9 of 9 exactly on config K. Same requests, wrong price. A magnitude problem, not a missing scheduling mechanism.
Then the residual collapsed onto a single cohort that identified itself with a number. Every mis-predicted request had its prefix match capped at exactly 1,200 tokens, and the trace's sessions share exactly 1,200 words of system prompt before diverging. These were requests that matched the shared prefix and nothing else. Everything outside the cohort was accurate to within 14 milliseconds.
| mean ttft err (s), by turn | F | J | K | H |
|---|
Cold start
Turn-0 requests legitimately match only the shared prompt. The simulator over-charges their cold prefill by a flat 0.13–0.17 s at every pool size. This is the whole of config H's failure, the row that was bit-identical across three versions.
Eviction cliff
Later-turn requests that should match their session history fall back to 1,200: the simulator evicts inter-turn history more aggressively than the engine. The count is perfectly monotonic in pool size: 0, 4, 11, 14 requests at 5,811 → 4,246 blocks.
Methods, and the criterion
Two things kept this honest. First, the freeze: predictions committed before measurements, superseded exactly once with the reason recorded, absolute afterwards. Run 4 re-predicted against already-seen measurements and says so in its own header, because it tested three pre-registered hypotheses and nothing else. Second, the criterion: the original 15-point relative bar amplifies sub-second baseline errors into hundred-point gaps, so a v2 criterion (relative gap or absolute error within 15%) was adopted in writing between rounds, prospective only. The whole series scores under both:
| run | sim | rows | v1 | v2 | held-out | v1 | v2 |
|---|
The meta-lesson, which I now believe generalizes: the serving physics was the easy part. A linear step model with four coefficients survived seven rounds. If you build anything that predicts serving behavior, calibrate on the exact path your traffic takes, at the concurrency it takes it, and treat your calibration harness as the component most likely to be wrong.
The eviction-retention fix and the cold-start term are next. If you run vLLM in production and the shape of this problem looks familiar, I want to compare notes.