Let the Compiler Argue With the Demo: A Repeatable C++ Grader for Coding Models

Short summary

The author built a repeatable C++ grader for coding AI models that evaluates each answer against three mechanical gates: strict build with project flags and sanitizers, hidden behavior check against expected output, and patch-boundary scope verification. Five to ten tasks sourced from real bugs are sufficient, and each candidate costs roughly one command to judge. The approach makes evaluation economic enough to re-run repeatedly rather than relying on a single lucky completion.

• •Three gates: strict build, hidden behavior check, and patch-scope boundary enforcement
• •5-10 tasks from real bugs are enough; each candidate costs one command to judge
• •Replaces vibe-based model comparisons with deterministic, scriptable pass/fail verdicts

Generated with AI, which can make mistakes.