PathReportEval: A Systematic Benchmark for Pathology Report Generation

Short summary

PathReportEval introduces a standardized benchmark and evaluation framework for pathology report generation from whole-slide images across three datasets and three pathology foundation encoders. The key contribution is the Clinical Report Quality Score (CRQS), which measures clinical fact coverage, key information recall, hallucination rate, and clinical discordance. Conventional NLG metrics like BLEU and ROUGE were weakly aligned with clinical correctness, while CRQS revealed meaningful differences between models that lexical metrics missed.

• •Standardized benchmark evaluates four methods across TCGA, HistAI, and REG 2025 datasets with three pathology encoders
• •CRQS metric measures clinical fact coverage, recall, hallucination rate, and discordance — unlike BLEU/ROUGE which overestimate quality
• •Conventional NLG metrics fail to detect clinically consequential errors; CRQS captures them

Generated with AI, which can make mistakes.