r/MachineLearning
6/25/2026
Attention pathologies stem from norm
Original: [R] All Routes Lead to Collapse: attention sinks, representation collapse, and norm stratification are what content-based routing does under a norm-blind metric
Short summary
Attention pathologies—sinks, representation collapse, norm stratification—aren't separate issues but symptoms of a single geometric problem: standard attention routes using metrics blind to key magnitude. The author demonstrates this across five architectures (Transformers, GATs, Mamba, RWKV, AttnRes) with empirical validation, causal ablations, and open-source code.
- •Unified explanation for multiple attention pathologies as symptoms of norm-blind routing
- •Pattern validated empirically across 5 different architectures and 9+ pretrained models
- •Proposes that positional mechanisms (RoPE, time decay) control timing/severity of collapse
Generated with AI, which can make mistakes.
Is this a good recommendation for you?
