The original title is: "Stop Guessing: A Reproducible Harness for Evaluating Free AI Coding Models on Your Own Repo"

Short summary

A practical guide to building a reproducible evaluation harness for comparing AI coding models on your own repository. Instead of relying on public benchmarks or vibes, you define tasks as directories with prompts, code snapshots, and verification scripts. The harness applies model-generated patches and checks them against your own test suite, type checker, and lint rules — no LLM-as-judge needed. Includes a minimal Python runner (~120 lines) that separates generation from verification so you can diff outputs across models.

• •Define eval tasks as directories with prompt, code snapshot, and verify.sh
• •Runner applies model patches and checks exit codes — no scoring model needed
• •Separates generation from verification for cross-model comparison

Generated with AI, which can make mistakes.