Getting Evals Right for LLM Applications | Interrupt 26

Short summary

Shreya Shankar and Hamel Husain share five critical mistakes teams make when evaluating LLM applications, from over-relying on generic metrics to automating evals without understanding failure modes. They teach a data-science approach: treat LLM judges as imperfect classifiers, design custom metrics through exploration, and systematically validate your data labeling. This 18-minute talk is essential for anyone shipping production LLM products.

• •Avoid generic metrics like 'helpfulness'—build custom evaluation interfaces through data exploration
• •Treat LLM judges as imperfect classifiers requiring train/dev/test splits and calibration
• •Don't fully automate evals; manual review of failure modes is critical for catching product issues

Generated with AI, which can make mistakes.