Did it check, or did it guess? Testing a tool whose output is written by an LLM

Short summary

A senior architect shares lessons from testing an AI auditing tool built as a Claude Code skill that evaluates auth system hard-coding. The tool confidently hallucinated findings when encountering version mismatches in vendor libraries, and produced inconsistent results across model runs. The author argues that freeform markdown output makes mechanical testing nearly impossible and advocates for structured output formats to enable cheaper, more reliable test pipelines.

• •LLM-based tools can confidently fabricate findings when encountering unfamiliar code patterns
• •Cross-model and cross-run inconsistency is a real testing challenge for AI tools
• •Structured output formats are essential for enabling mechanical, cost-effective testing of AI tools

Generated with AI, which can make mistakes.