Testing AI Coding Agents Beyond Code Generation: A Real-World Benchmark

Short summary

This benchmark tests AI coding agents on real-world maintenance tasks, not just greenfield generation. Two Codex runs on a full-stack task manager are documented: a 34-minute initial build and a 42-minute auth-and-migration refactor with an interruption recovery. The evaluation framework checks behavior preservation, data integrity, security enforcement, and workflow recovery — dimensions missing from typical demos.

• •Codex completed a greenfield full-stack build in ~34 min and a refactor in ~42 min
• •Benchmark evaluates maintenance tasks: auth changes, migrations, test preservation, recovery
• •Proposes five-question framework for assessing agent performance on stateful systems

Generated with AI, which can make mistakes.