LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Short summary

LISA (Linear-Indexed Sparse Attention) is a plug-and-play attention replacement that reduces inference complexity from O(n²) to O(nM) for long-context reasoning models. It combines a linear attention module for long-range memory with a Lightning Indexer for dynamic top-M token selection, fused via gating. On DeepSeek-distilled-Qwen models, LISA achieves 50% inference speedup at 16K-token context while improving average reasoning performance by 5.6% on benchmarks like AIME and MATH-500.

• •Plug-and-play module reduces attention complexity from O(n²) to O(nM) without pretraining from scratch
• •Combines linear attention for long-range memory with dynamic token selection via Lightning Indexer
• •50% inference speedup at 16K context with 5.6% average performance improvement on reasoning benchmarks

Generated with AI, which can make mistakes.