Hybrid Mamba-Transformer MoEs Hide Their Stalls in Places Dashboards Do Not Look

Short summary

Hybrid Mamba-Transformer MoE models hide performance problems in per-layer latency tails—MoE all-to-all collective stalls are 69x slower in tail (p99) despite only 1/9th the kernel calls, yet aggregate GPU metrics report 95% utilization. Use eBPF tracing (Ingero) to decompose runtime per-layer type and expose bottlenecks invisible to vLLM/SGLang dashboards. Three operational fixes suggested: penalize unbalanced expert routing, avoid co-scheduling MoE collectives on same NCCL stream, and tune per-expert capacity.

• •MoE all-to-all collective communication dominates tail latency (p99 12.4ms, 69x tail ratio) despite low call count, hidden by aggregate GPU utilization metrics
• •eBPF kernel tracing (Ingero tool) exposes per-layer-type decomposition that dashboards like vLLM/SGLang cannot provide
• •Three operational fixes: imbalance-penalized batch routing, NCCL stream co-scheduling avoidance, and per-expert capacity tuning

Generated with AI, which can make mistakes.