JSON Schema Doesn't Prevent AI Hallucinations (And That's Okay)

Short summary

The author argues that JSON Schema and Structured Outputs solve format correctness, not information correctness—a model can return perfectly valid JSON with hallucinated values. Proposes a two-layer validation model: structural validation (schema) and trust validation (evidence, source attribution, confidence scores). Compares how OpenAI, Groq, Ollama, and Anthropic implement structured outputs with different guarantee levels, and pitches the open-source ShapeCraft library.

• •JSON Schema guarantees structure, not correctness—valid JSON can still contain hallucinated values
• •Two independent validation layers needed: structural (schema) and trust (evidence, source attribution)
• •Providers implement structured output differently: native, constrained, or best-effort approaches

Generated with AI, which can make mistakes.