DFlash Speculative Decoding Drafts Whole Token Blocks in Parallel for Up to 15x Higher Throughput on NVIDIA Blackwell

Short summary

UC San Diego's DFlash replaces autoregressive drafting with a lightweight block diffusion model for speculative decoding, enabling whole token blocks to be drafted in parallel through KV injection. The technique achieves up to 6.08x lossless speedup on Qwen3-8B and 15x throughput on NVIDIA Blackwell GPUs, shipping with 20 pre-trained checkpoints and integration support for SGLang, vLLM, and TensorRT-LLM.

• •Block diffusion model for speculative decoding drafts multiple tokens in parallel via KV injection
• •Achieves up to 6.08x speedup on Qwen3-8B, 15x throughput on NVIDIA Blackwell
• •Ships with 20 checkpoints and integrates with SGLang, vLLM, and TensorRT-LLM

Generated with AI, which can make mistakes.