What Auditing My Own AI Projects Taught Me About Shipping Production Code

Short summary

An engineer audited two AI projects — a GitHub code-review automation tool and an LLM API proxy — and found 13 serious bugs sharing one pattern: silent failures that don't announce themselves. The most dangerous bugs included an auth bypass that failed open, a rate limiter that became a memory leak, and an installer that could destroy user data. The core lesson is that production code must fail loudly, default to denial on uncertainty, and prove fixes with targeted regression tests rather than relying on coverage metrics.

• •13 serious bugs across two AI projects shared one pattern: silent failures that compound without leaving a trail
• •Worst bugs: auth failing open (security hole), rate limiter memory leak (outage vector), installer overwriting user data (data loss)
• •Fix principle: convert silent failures to visible ones, default to deny when uncertain, and write targeted regression tests for each specific failure mode

Generated with AI, which can make mistakes.