Running an 80B Qwen Model in 4.3GB: Quantization, Pruning, and Apple Silicon Explained

Short summary

Developers in 2026 demonstrated running an 80B Qwen model in just 4.3GB of RAM on Apple Silicon using hybrid quantization, structural pruning, weight sharing, and dictionary coding. Apple's M5 Max delivers 900GB/s memory bandwidth yielding ~4.7 tokens/sec, while the AMX-3 coprocessor accelerates sparse 2-bit operations. iOS 2026 introduced llmCache in CoreML, enabling 35B models to run on iPhones by paging weights from NVMe flash into DRAM.

• •80B Qwen model runs in 4.3GB RAM via hybrid 2.5-bit quantization, pruning, and dictionary coding
• •Apple M5 Max provides 900GB/s bandwidth; AMX-3 coprocessor accelerates sparse 2-bit dot products
• •iOS llmCache API pages model weights from NVMe flash, enabling 35B models on iPhone 18 Pro

Generated with AI, which can make mistakes.