AirLLM Runs a 70B Model on a 4GB GPU. It's True, and That's Not the Interesting Part

Short summary

AirLLM streams transformer layers one at a time from disk to GPU, so peak VRAM equals the largest single layer (~1.75GB for a 70B model at FP16) instead of the full model (~140GB). The claim of running 70B on 4GB is technically true and preserves full precision without quantization. The catch is speed: generating one token requires reading the entire model off disk, yielding roughly 5-35 seconds per token on NVMe — making it useful for experimentation, not production serving. MoE models benefit disproportionately since only active experts need streaming.

• •AirLLM loads one transformer layer at a time, reducing peak VRAM to the size of the largest single layer
• •Full-precision 70B models genuinely run on 4GB GPUs, but at 5-35 seconds per token — disk I/O is the bottleneck
• •MoE models like Kimi K3 (2.8T params) need less VRAM than dense 671B models because only active experts are streamed

Generated with AI, which can make mistakes.