How Lyft Builds Evals That Actually Matter in Production | Interrupt 26

Short summary

Lyft's ML lead shares how their team built an evaluation system for 270K monthly AI agent interactions using offline simulation, LLM-as-judge rubrics, and task-based failure modes. Key lesson: offline evals often diverge from production—real success requires continuous annotation loops. LangSmith unifies evaluation, feedback, and model improvement.

• •Offline simulation with LLM-as-judge and task-based rubrics for production AI evals
• •Real-world lessons: offline evals often fail; need continuous annotation cycles to stay aligned
• •LangSmith workflow integrates eval management, annotation feedback, and improvement loops

Generated with AI, which can make mistakes.