Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression

Short summary

This paper combines neuron importance and data-aware low-rank approximation into a single objective for LLM compression via SVD, addressing limitations of prior work that studied these perspectives in isolation. The authors also propose a computationally efficient algorithm for dynamic compression rate allocation across layers, replacing uniform distribution or expensive heuristic search. Experiments show the approach performs on par or substantially better than previous state-of-the-art, especially at high compression ratios.

• •Unifies neuron importance and data-aware low-rank approximation into a single compression objective
• •Proposes efficient dynamic compression rate allocation across layers instead of uniform distribution
• •Outperforms prior SOTA especially under high compression ratios

Generated with AI, which can make mistakes.