My LLM drift tracker flagged four regressions this week. All four were wrong.

Short summary

An LLM drift tracker flagged four model regressions in four days — all false positives. Two were caused by rate-limited API calls scoring as zero accuracy, and two were single-question flips on a 35-task suite too small to resolve sub-3-point noise. The author argues sensitivity is a feature, but only if a human checks reliability metrics before publishing. The key missing metric: the share of alerts that survive verification.

• •Four flagged regressions were all false: two from rate limits scored as 0%, two from single-question flips on a 35-task suite
• •Reliability (share of calls that returned) must sit beside accuracy on every chart or the chart lies
• •Automation should notice; humans should check — four notices, zero posts this week

Generated with AI, which can make mistakes.