Your RAG faithfulness check is measuring copy-paste, not faithfulness

Short summary

Token-overlap metrics for RAG faithfulness are fundamentally broken—they miss hallucinated content hidden by stopwords and penalize good paraphrasing. Semantic entailment-based approaches fix both failure modes but add cost and complexity. The trade-off between cheap-but-wrong and expensive-but-accurate metrics matters for production systems.

• •Token overlap fails bidirectionally: false positives hide hallucinated key facts among matching stopwords, false negatives punish good paraphrases
• •Semantic entailment via NLI models or LLM judges fixes both problems but requires per-example model calls
• •Production RAG systems must consciously choose between cheap-but-wrong metrics and accurate-but-expensive ones

Generated with AI, which can make mistakes.