The original title is "How to evaluate an LLM agent: evals, golden sets, and LLM-as-judge"

Short summary

Evals replace deterministic unit tests with scored, repeatable checks across multiple runs — measuring whether an LLM agent reaches acceptable outcomes at an acceptable rate. The core toolkit includes a curated golden set grown from real failures, offline CI runs plus online production monitoring, and task-appropriate scorers (assertion, golden set, LLM-as-judge, or human review). LLM-as-judge scales subjective grading but has real failure modes — position bias, verbosity bias, inconsistency, and gameability — so the judge itself must be validated against human-labeled data before trust.

• •Evals reframe LLM testing as a measurement problem (SLO-style) rather than pass/fail unit tests
• •Golden sets should be grown from real production failures, kept version-controlled and representative
• •LLM-as-judge scales subjective grading but needs validation against human labels due to bias and inconsistency

Generated with AI, which can make mistakes.