A Flaky Test Is a Corrupted Reward Signal

Short summary

Flaky tests are a minor annoyance for human developers but a catastrophe for AI agents running TDD loops. Humans filter noise using team folklore and contextual judgment; agents lack this out-of-band knowledge and treat every red bar as a real signal. The result is that flaky tests corrupt the agent's reward function, forcing it to chase phantom failures or ignore real defects if told to skip them.

• •Flaky tests corrupt the binary reward signal AI agents rely on in TDD loops
• •Humans filter flaky failures via team folklore; agents cannot access this context
• •Ignoring flaky tests for agents creates permanent blind spots in the specification

Generated with AI, which can make mistakes.