Back to feed
arXiv cs.LG
arXiv cs.LG
6/25/2026
Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

Short summary

Looped language models decode hidden states for prediction and feed results back into computation, but dense cross-entropy loss only supervises readout-exposed variables, not all state components. Scale-invariant readouts like RMSNorm hide radial scale from the loss, allowing it to grow uncontrolled during recurrence. The fix is to make scale visible to the loss or remove it from the recurrent loop entirely, improving perplexity at matched inference depths.

  • •Dense supervision controls only readout-exposed variables, not all hidden state components
  • •Scale-invariant readouts (RMSNorm, LayerNorm) hide radial scale from loss, causing uncontrolled growth
  • •Solution: make scale visible to loss or remove it from recurrence architecture

Generated with AI, which can make mistakes.

#research-breakthrough#certification-education
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Explore more

Perfect Detection, Failed Control: The Geometry of Knowing vs. Steering in Language Models
arXiv cs.CL
Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test
arXiv cs.LG
When Does a Language Model Commit? A Finite-Answer Theory of Pre-Verbalization Commitment
arXiv CS.AI
Context Recycling for Long-Horizon LLM Inference
arXiv cs.CL