I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Short summary

Most LLM-as-judge tools optimize for scoring speed rather than validation against human labels. None of the six tested (DeepEval, Evidently, Braintrust, Promptfoo, Confident AI, Future AGI) ship human-agreement metrics as a default workflow. Proper evaluation requires hand-labeling 200 examples, computing Cohen's kappa, and deploying judges only when kappa reaches ~0.6.

• •Most LLM-as-judge tools prioritize score production over human-agreement validation
• •Tested 6 tools; none default to confusion-matrix workflows or kappa computation
• •Concrete methodology: hand-label examples, compute Cohen's kappa, deploy at ≥0.6 with confusion-matrix review

Generated with AI, which can make mistakes.