What 90% Line-Rate Utilization on a Single 100GbE Port Means: Analyzing Network Bottlenecks in Inference Storage

Short summary

Mingxin FX100 achieves 90% line-rate utilization on a single 100GbE port (~11.25 GB/s), eliminating network as the bottleneck for KV Cache loading in LLM inference. Combined with LMCache parallel read patching in vLLM, TTFT dropped by 26-32% under 480B parameter workloads. The article details how network bandwidth contention limits inference throughput and how all-flash NVMe-oF arrays resolve it.

• •FX100 achieves 11.25 GB/s effective bandwidth on a single 100GbE port, 60-80% faster than local PCIe Gen4 NVMe
• •LMCache parallel read patching with FX100 reduced TTFT from 37.97s to 9.30s (4.1x improvement) in cold-read scenarios
• •Network bandwidth, not storage medium, is the primary bottleneck for KV Cache loading in multi-GPU inference clusters

Generated with AI, which can make mistakes.