Pydantic passed. Types matched. The downstream system still got garbage.

Short summary

A contract-extraction agent using Pydantic and Claude passed schema validation on three production failures because schema validation enforces grammar, not meaning. The failures were: paraphrased (not verbatim) clause text, uncontrolled retry loops causing billing incidents, and semantic drift after a model upgrade. Solution requires a separate semantic validation layer, capped retries with escalation, and shadow evaluation on model changes.

• •Schema validation only checks grammar (syntax), not meaning (semantics)
• •Three production failures masked by valid JSON: paraphrases, retry billing, model drift
• •Solution: add semantic evaluation layer, cap retries, shadow-eval on model upgrades

Generated with AI, which can make mistakes.