RAGAS vs DeepEval faithfulness metrics: a controlled comparison reveals divergent scoring on fabricated RAG outputs

Short summary

A controlled comparison of RAGAS and DeepEval faithfulness metrics reveals they measure different things: RAGAS scored a fabricated answer 0.0 while DeepEval scored it 1.0. The author built OpenGATE, a deterministic checker that catches omitted facts (where LLM judges fail) but misses semantic inversions (where judges excel). Conclusion: use both approaches in combination for robust RAG evaluation, especially in healthcare where silent failures are dangerous.

• •RAGAS and DeepEval 'faithfulness' metrics disagree fundamentally — one catches fabrication, the other doesn't
• •Deterministic checks catch omitted facts (5/5) where LLM judges fail (0-1/5); judges catch meaning inversions (4-5/5) where deterministic checks fail (0/5)
• •OpenGATE offers free, deterministic gating ($0 vs $8-11 per 1k evals) but should complement, not replace, LLM judges

Generated with AI, which can make mistakes.