The $2,300 Weekend: When Fallback Routing Goes Wrong in AI Gateways

Short summary

A misconfigured fallback route in LiteLLM caused a $2,300 bill when the cheap primary model (DeepSeek-V3) rate-limited and traffic routed to 18x-more-expensive GPT-4o. The anti-pattern: falling back from cheaper to expensive models; the solution: tier-based fallback chains, per-key budget caps, and monitoring fallback rates at 5% threshold. Includes production config templates and monitoring options.

• •Misconfigured fallback routed 5% of traffic from cheap DeepSeek to expensive GPT-4o when rate-limited, resulting in $2,300 unexpected charges
• •Anti-pattern: capability-based fallback (cheap → expensive); solution: tier-based fallback (cheap → cheap, mid → mid) with hard per-key budget limits
• •Monitor fallback rate at 5% threshold; use circuit breakers, cooldown delays, and cache to prevent cost bleed

Generated with AI, which can make mistakes.