Building Fast, Evaluating Slow: Pipeline Choices Dominate Autointerpretability Score Variance

Short summary

This paper shows that autointerpretability scores for sparse autoencoders are dominated by methodological pipeline variance rather than architectural differences. Across four metrics, two models, and four variation axes, pipeline choices explain more score variance than the SAEs being compared. The authors contribute a variance decomposition approach, Stability Check, and Minimum Reporting Checklist to improve evaluation reliability.

• •Autointerpretability score variance is driven more by evaluation pipeline choices than by SAE architecture differences
• •Detection is the most stable metric while fuzzing is unreliable across all conditions
• •Authors propose variance decomposition, Stability Check, and Minimum Reporting Checklist for reliable evaluation

Generated with AI, which can make mistakes.