Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts

Short summary

This study benchmarks 5 LLM watermarking schemes across 11 LLMs and 7 VLMs on clinical reasoning tasks, finding that watermarking can cause lexical corruption, hallucinated terminology, and misattribution of image findings. The authors introduce a human-expert-validated pipeline for auditing medical reasoning quality and show that standard aggregate metrics mask clinically consequential failures. Domain-specific evaluation is essential before deploying watermarked models in medicine.

• •Watermarking degrades LLM performance in medical contexts via lexical corruption and hallucinated terminology
• •Standard benchmarks obscure clinically significant watermark-induced failures
• •Human-expert-validated pipeline introduced for auditing medical reasoning quality in watermarked models

Generated with AI, which can make mistakes.