Vllm

Open RFCs in vLLM and SGLang propose predicting when an agent returns from a tool call and prefetching its KV cache. Tested on 744,000 real tool calls and a real vLLM server, the prediction does not pay. Host tier size, the store path and one scheduler flag move the tail.

Tensor parallelism, speculative decoding, a faster attention backend, a different quantization: none cleared a 20% bar on a 32B model across two RTX 4090s. Then I served agent traffic on a chat-tuned config and cost went up 92%.

Nearly every serving benchmark number, published or internal, comes from running the thing once. I ran one identical vLLM benchmark 14 times: throughput reproduces to 0.02%, the latency tail swings 18%, and one config turned out to be bistable rather than noisy.

Changing a serving config is a decision most teams make blind. Two days testing a third option: replay request logs through an offline model of the engine.