How We Actually Measure Whether an LLM's Output Is Good - BLEU, COMET and BLEURT

Short summary

Machine translation evaluation evolved from BLEU (2002), which counts word overlaps, to learned metrics like BLEURT and COMET using neural networks to judge meaning and quality. These modern metrics handle paraphrasing and creativity better than simple lexical matching. Evaluation remains expensive (150K outputs ≈ $25K in expert labor), so researchers today combine automatic metrics with strategic human review.

• •BLEU (2002) replaced expensive human evaluation by measuring n-gram overlap, enabling rapid iteration in machine translation research
• •BLEURT and COMET use pretrained transformers to evaluate meaning, handling paraphrasing and multiple correct answers that BLEU misses
• •Modern LLM evaluation combines automatic metrics with targeted human feedback due to cost constraints ($25K+ for expert review of 150K outputs)

Generated with AI, which can make mistakes.