Vllm

Tensor parallelism, speculative decoding, a faster attention backend, a different quantization: none cleared a 20% bar on a 32B model across two RTX 4090s. Then I served agent traffic on a chat-tuned config and cost went up 92%.

Nearly every serving benchmark number, published or internal, comes from running the thing once. I ran one identical vLLM benchmark 14 times: throughput reproduces to 0.02%, the latency tail swings 18%, and one config turned out to be bistable rather than noisy.

Changing a serving config is a decision most teams make blind. Two days testing a third option: replay request logs through an offline model of the engine.