I built an eval harness to prove an LLM worked. It proved the opposite.

Short summary

The author built an eval harness to test whether an LLM could classify integration failures, but deterministic rules beat the LLM (89.7% vs 87.9%). The biggest classification improvement came from correlating log lines with regex before any model saw the data. The LLM's only useful role turned out to be writing plain-English incident summaries, run locally via Ollama at zero cost.

• •Deterministic rules outperformed LLM on classification (89.7% vs 87.9%)
• •Correlated log-line stitching via regex was the single biggest quality improvement
• •LLM relegated to summary generation only; cost $0 via local Ollama

Generated with AI, which can make mistakes.