Identifying Interactions at Scale for LLMs

TL;DR

Berkeley researchers introduce SPEX and ProxySPEX, algorithms that efficiently identify feature interactions driving LLM predictions at scale using signal processing. These methods maintain interpretability across thousands of features while remaining computationally tractable, enabling practical model attribution for debugging and validation.

• •SPEX exploits sparsity and low-degree interactions to efficiently discover what drives LLM predictions
• •ProxySPEX improves efficiency 10x by leveraging hierarchical relationships between interactions
• •Methods scale to thousands of features while maintaining faithfulness that simpler attribution approaches lose

Generated with AI, which can make mistakes.