Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models

Short summary

This paper proposes a compound sparsity framework for LLM compression that combines static parameter pruning (low-rank approximation + channel pruning) with dynamic per-token layer skipping via lightweight routers. Experiments show compound sparsity outperforms single-mechanism compression at the same total sparsity budget, delaying performance decay on language understanding and modeling benchmarks. The authors find near-balanced allocation between parameter and computation sparsity is most effective, and identify a cross-dimensional sparsity boundary that limits further compression.

• •Combines static parameter pruning with dynamic token-level layer skipping for LLM compression
• •Compound sparsity outperforms single-mechanism approaches under equal total sparsity budgets
• •Near-balanced allocation between parameter and computation sparsity is optimal; code available on GitHub

Generated with AI, which can make mistakes.