LLM Self-Preference Bias: How Anonymized Peer Review Fixes It

Short summary

LLMs systematically prefer outputs matching their own writing style, with GPT-4 favoring its own generations over 90% of the time in head-to-head comparisons. This self-preference compounds with verbosity bias (longer = better) and position bias (first = better), creating multiple evaluation distortions. Anonymizing outputs before ranking—replacing model names with neutral labels while keeping a private mapping—eliminates identity signals and restores fair scoring.

• •LLMs exhibit self-preference bias: favor outputs matching their writing style
• •Research shows GPT-4 prefers its own output >90% of the time
• •Solution: anonymize outputs before evaluation to eliminate identity signals

Generated with AI, which can make mistakes.