How I A/B test LLM prompts without fooling myself

Short summary

Testing LLM prompts requires sufficient sample size—thirty examples can't detect improvements under 10 points. Use paired testing (same questions for both versions) to control for question difficulty, report confidence ranges rather than averages, and deploy multi-armed bandits when testing 3+ versions to minimize exposure to weaker prompts. Automated prompt generation combined with careful A/B testing finds better solutions than hand-written iteration alone.

• •Small sample sizes (30 examples) mask luck as improvement; required size grows quadratically with desired sensitivity
• •Use paired testing on identical questions to isolate prompt quality from question difficulty
• •Report confidence intervals, not averages; ship only if entire range exceeds the minimum worthwhile improvement threshold

Generated with AI, which can make mistakes.