Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events

Short summary

A retrieval-augmented, multi-agent LLM framework with human-in-the-loop review was evaluated for detecting cutaneous immune-related adverse events from clinical notes. The LLM-assisted workflow improved F1 score from 0.77 to 0.88, increased inter-rater agreement (Cohen's kappa 0.82 vs 0.50), and halved average review time. The framework demonstrates how LLMs can enable scalable, accurate adverse event data extraction across organ systems.

• •LLM-assisted workflow improved F1 from 0.77 to 0.88 for detecting skin adverse events
• •Inter-rater agreement rose from kappa 0.50 to 0.82 with LLM assistance
• •Average clinical note review time reduced by approximately half

Generated with AI, which can make mistakes.