How Do You Know Your RAG Actually Works?

Short summary

A dialogue-style article explaining why manual testing of RAG systems is unreliable and how to build proper offline evaluation. It introduces the concept of a golden dataset — fixed questions paired with expected answers, chunks, and citations — and argues for four question types including answerable, unanswerable, and edge cases. The core message: without systematic evaluation, you cannot distinguish retrieval failures from LLM hallucinations or detect silent regressions.

• •Manual RAG testing is biased — you unconsciously ask easy questions and grade generously
• •A golden dataset pairs questions with expected answers, chunks, and citations written before the system runs
• •Offline evaluation catches regressions from prompt/embedding/chunk changes before users do

Generated with AI, which can make mistakes.