Can You Tell When an LLM API Swaps in a Cheaper Model?

Short summary

Detecting when LLM API providers swap in cheaper models requires measuring log-probability of fixed token sequences, not output quality. Genuine models assign higher probability to their own outputs; a cheaper impostor's signal emerges statistically over 10–15 checks, becoming reliable only through accumulation. The method works for open-weight models but not closed APIs.

• •Log-probability scoring on fixed sequences detects model swaps more reliably than quality assessment
• •Signal requires statistical accumulation over 10-15 prompts to distinguish 2x-cheaper impostors
• •Only practical for open-weight models; closed-API detection remains unsolved

Generated with AI, which can make mistakes.