Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks

Short summary

Researchers show that grokking—the sudden delayed onset of generalization in deep neural networks—occurs when stochastic gradient descent noise drives models trapped in metastable states across energy barriers, following physics-inspired first-order phase transitions. In linear networks, escape times follow Arrhenius scaling laws. The findings suggest this mechanism applies to nonlinear networks and opens routes toward more efficient learning schemes.

• •Grokking explained as noise-driven escape from metastable phases via energy barriers
• •Phase transitions and Arrhenius scaling characterize the phenomenon
• •Mechanism likely extends to nonlinear networks, enabling efficient training strategies

Generated with AI, which can make mistakes.