Eval-Driven Agent Development: How I Stopped Tuning Prompts on Vibes

Short summary

The author built an evaluation framework to objectively measure whether changes to their AI agent harness improve performance. They run multiple trials combining deterministic code graders and an LLM judge to catch different failure modes. Using pass@k (capability) vs pass^k (consistency) reveals reliability problems that average scoring hides.

• •Use eval frameworks to measure agent harness improvements objectively instead of relying on vibes
• •Combine deterministic code graders and LLM judges to catch different types of failures
• •Track pass@k (can it do it?) vs pass^k (can it do it reliably?) instead of means to reveal reliability issues

Generated with AI, which can make mistakes.