Harvesting a regression test set from gateway logs with a plugin

Short summary

Nexus Labs replaced stale hand-written eval cases with a production-derived test set by capturing AI model traffic through a Bifrost gateway plugin. They sample 5% of production requests, deduplicate via clustering, oversample failures, and curate ~250 high-signal cases weekly—avoiding the drift that happens when test cases are invented at a desk. The approach caught regressions their old suite missed and requires choosing between self-hosted gateways like Bifrost (low overhead, more engineering) and managed services like Portkey (faster setup, data residency trade-offs).

• •Real production traffic via gateway plugin beats hand-written test cases that diverge from actual usage patterns
• •5% sampling + embedding-based clustering + human curation yields 250 high-signal cases weekly from 9,000 daily calls
• •Bifrost vs LiteLLM vs Portkey: trade off self-hosted control, setup speed, and data residency based on team constraints

Generated with AI, which can make mistakes.