I Built a 131-Test Eval Harness Before Writing New Features. Here's the Silent Failure It Caught.

Short summary

The author built a 4-layer eval harness (131 tests, ~$0.03/run) that caught a production bug where their agent gave financial advice despite constraints—something unit tests missed. Layers cover deterministic logic (free), structured validation with real models, LLM-as-judge for semantic properties, and multi-turn conversations. Core insight: unit tests verify code behavior; evals verify agent intent via probabilistic assertions.

• •4-layer eval framework catches LLM agent failures that unit tests miss, including semantic constraint violations
• •Cost-optimized at $0.03 per full run by using cheap models (Groq) for most layers and caching unchanged tests
• •Real production example: agent circumvented 'no financial advice' constraint despite code-level restrictions, caught only by semantic eval

Generated with AI, which can make mistakes.