Challenge: Hand coding weights for efficient sequence memorisation

Short summary

Researchers hand-coded weights for single-layer MLPs that memorize labels for two-token input sequences. The hand-coded models scale roughly linearly in memorization capacity with parameter count, matching trained models' scaling behavior. However, the scaling prefactor for hand-coded models still falls short of trained models by a significant factor, suggesting trained models use more efficient memorization strategies.

• •Hand-coded MLP weights memorize two-token sequence labels with 90% accuracy
• •Memorization capacity scales linearly with parameter count, matching trained models
• •Hand-coded models' scaling prefactor is lower than trained models' by a notable factor

Generated with AI, which can make mistakes.