The original title is: "Detecting model substitution in LLM API relays: A verification playbook"

Short summary

A verification playbook for detecting when LLM API relays silently swap in cheaper, quantized, or context-truncated models while billing for flagships. Methods include tokenizer fingerprinting via crafted probe strings, capability floor tests, needle-in-haystack context truncation detection, and determinism/repeatability analysis. The author discloses working on daoxe, a gateway, and provides an open-source tool that automates the verification process.

• •Tokenizer fingerprinting: send crafted strings and compare token counts against trusted endpoints to detect model family substitution
• •Capability floor tests and needle-in-haystack probes catch downgraded models and silent context truncation
• •Determinism testing at temperature=0 reveals routing instability and backend switching by dishonest relays

Generated with AI, which can make mistakes.