AI Dev 26 x SF | Ara Khan: Evals Are Broken Use Them Anyway

Short summary

Ara Khan from Cline explains why systematic evaluation is essential for AI agents, sharing practical heuristics for designing and interpreting evals despite their limitations. The team moved from skepticism to treating evals as core infrastructure for scaling agent capabilities. Evaluation beats iteration-by-feel.

• •Evals evolved from dismissible overhead to essential optimization loop
• •Practical heuristics for designing, running, and interpreting evaluation frameworks
• •Systematic evaluation outperforms intuition-based iteration at scale

Generated with AI, which can make mistakes.