Speculative decoding shifted our output distribution and evals missed it

Short summary

Speculative decoding in vLLM can diverge from greedy decoding due to float16 arithmetic differences between batched and sequential verification steps. At Nexus Labs, 1.2% of outputs shifted (e.g., 50→500 in tool arguments), undetected because evals routed through HuggingFace API while production used vLLM. Solutions: route all evals through the exact production endpoint, pin serving config in CI with fingerprints, and run nightly divergence canaries to catch updates.

• •Speculative decoding's batched verification can produce different tokens than sequential decoding when logits are close, especially in structured output
• •Eval frameworks that don't match the production serving path won't catch these divergences
• •Solutions: evaluate through production endpoints, assert serving-config fingerprints in CI, and run divergence canaries

Generated with AI, which can make mistakes.