Your agent returned 200 OK. Was it actually right?

Short summary

Modern agent observability stacks trace what an agent did but don't verify whether its output is actually correct in the moment. The author found a weak model scored 69% accuracy on a structured task but looked right far more often; wrapping outputs in grounded constraint checks and re-rolling failures brought it to 100%. The key insight: consistency isn't correctness, and the agent stack needs a runtime certification layer between logging and offline evals.

• •Agent observability tools show what happened, not whether the answer was correct
• •A weak model hit 100% accuracy when outputs were verified against constraints and failures re-rolled
• •The gap is a runtime certification layer that checks correctness in production, not just offline

Generated with AI, which can make mistakes.