Back to feed
arXiv cs.LG
arXiv cs.LG
7/23/2026
Bayesian Wind Tunnels for Model Selection

Bayesian Wind Tunnels for Model Selection

Short summary

This paper introduces 'Bayesian wind tunnels' to test whether transformers can perform Bayesian model selection, not just filtering. A 2.8M-parameter transformer achieves near-optimal Bayesian model selection on involutions, but fails when the discriminative statistic requires arithmetic with opaque symbols—a boundary that persists even at 316M parameters. Frontier LLMs show qualitative Bayesian behavior but a large calibration gap (~55x).

  • •Transformers can perform Bayesian model selection on relational tasks but fail when arithmetic is needed with opaque symbols
  • •This failure boundary persists under 112x parameter scaling (2.8M to 316M)
  • •Frontier LLMs show qualitative Bayesian behavior but a ~55x calibration gap

Generated with AI, which can make mistakes.

#research-breakthrough
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment
arXiv CS.AI
Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models
arXiv CS.AI
Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Models
arXiv cs.CL
Scaling Point-in-Time Language Models
arXiv cs.CL