Observing And Testing CX Agents | Interrupt 26

Short summary

Cisco's Customer Experience team shared how they scale AI agents to handle 16M interactions yearly by closing the feedback loop from production signals directly back to code merges. The system uses AI triage agents for diagnostics and human oversight only for write decisions, not reads, addressing the bottleneck at scale. Key takeaway: treat evals as infrastructure and use MCPs to integrate feedback signals with code systems.

• •Production feedback (thumbs-down, errors, routing confusion) feeds into AI triage agents that diagnose issues and route to code agents for fixes
• •Human-in-the-loop only on write decisions, not reads—reducing team bottleneck while processing 16M interactions yearly
• •Observability is infrastructure: evals treated as tests, LangSmith traces capture all signals, MCPs integrate with Jira and Splunk

Generated with AI, which can make mistakes.