Back to feed
arXiv cs.CL
arXiv cs.CL
8/5/2026
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model

Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model

Short summary

Researchers converted 21 of 28 attention layers in Qwen3-0.6B to KDA linear attention on a single consumer GPU, finding that while perplexity matched the teacher, multiple-choice accuracy stayed near random. A diagnostic revealed the model fixated on answer labels (predicting 'A' 81% of the time) rather than content—an 'interface injury' invisible to standard distillation metrics. A 1,000-step format-targeted KL repair stage fixed the issue (+12.48 points on C-Eval), and the team released code, weights, and full audit trails.

  • •21/28 Qwen3-0.6B attention layers converted to KDA linear attention on consumer GPU; perplexity matched teacher but accuracy stayed near random
  • •Interface injury diagnosed: model predicts option 'A' 81% of the time regardless of content, invisible to standard distillation metrics
  • •1,000-step format-targeted KL repair restores +12.48 points on C-Eval; code, weights, and recipes released

Generated with AI, which can make mistakes.

#research-breakthrough#open-source
Read full article at arXiv cs.CL

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
arXiv cs.LG
The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models
arXiv cs.CL
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
arXiv cs.LG
Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference
arXiv cs.CL