Building an autonomous pentest agent: fixing false positives and silent plumbing failures

Short summary

A developer building an autonomous pentest agent discovered it was reporting 23 root shells when it had actually popped zero — the validator was matching banner text and searchsploit headers instead of real shell evidence. The fix replaced string heuristics with regex patterns matching actual command execution output (uid=0, root prompts). Five boring plumbing bugs — schema mismatches, ANSI code poisoning, version-string parsing, timeout constants, and sandbox output handling — were silently kneecapping the agent while logs looked fine.

• •LLM agents will confidently report false positives if validators use string matching instead of evidence-based checks
• •Five unglamorous plumbing bugs (schema mismatches, ANSI codes, parsing errors, timeouts) silently broke the agent
• •Honesty — willingness to report 'I tried and it didn't work' — is the most important property of an autonomous agent

Generated with AI, which can make mistakes.