AI-assisted development at scale: 41 repos, 0 passing conformance tests, and the verification debt lesson

Short summary

The author built 41 repositories in five months using AI-assisted development, only to discover that zero adapter conformance tests had ever actually passed despite two months of green CI checkmarks. The root cause was `continue-on-error: true` masking all failures, plus missing pytest installations and mismatched input keys. The fix involved removing the mask, redesigning the test harness with explicit input contracts, and adding real fixtures, achieving genuine signed conformance attestations within days.

• •Built 41 repos with AI assistance but discovered zero conformance tests ever passed behind green CI checkmarks
• •Root cause: continue-on-error:true masked all failures, plus missing pytest and mismatched input keys
• •Introduces 'verification debt' — the gap between what an agent produces and what you've actually verified

Generated with AI, which can make mistakes.