Back to feed
arXiv cs.LG
arXiv cs.LG
6/19/2026
Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

Short summary

Researchers decompose how transformer weights evolve during AdamW training using a Weibull framework, finding alignment forces dominate early training (88-94%) before transitioning to decay. A spline-based method recovers these forces from sparse checkpoints with 92-94% accuracy, enabling analysis of real models. Results suggest weight-scale evolution is data-dependent.

  • •Weibull framework decomposes AdamW weight-scale growth into alignment, injection, and decay forces
  • •Alignment dominates early training (88-94% of force budget) before balancing with decay at saturation
  • •Novel spline method recovers alignment forces from sparse checkpoints with 92-94% accuracy

Generated with AI, which can make mistakes.

#research-breakthrough#open-source
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Explore more

A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks
arXiv cs.LG
Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks
arXiv cs.LG
Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking
arXiv cs.LG
Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models
arXiv cs.LG