Your AI isn't too weak. Your evals are missing.

Short summary

LLM quality failures typically live in prompts and output contracts, not model capability. This post describes building an evaluation harness that separates expensive generation (cached) from cheap grading—catching bugs Opus couldn't fix with prompt clarification alone and showing Sonnet matched Opus at 40% cost. A three-tier framework (unit assertions, human review, A/B tests) systematizes quality trade-offs and reveals when model upgrades are wasted spend.

• •Evaluation harness architecture separates expensive API calls from free cached grading, enabling rapid iteration on quality checks
• •Found production bugs that single-line prompt fixes solved, which bigger models couldn't—proving capability isn't where most failures live
• •Data showed Sonnet 4.6 matched Opus quality at 40% cost, and systematic evaluation beats the reflex to upgrade models

Generated with AI, which can make mistakes.