Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization
Summary
- Researchers developed a clinically grounded adversarial evaluation framework to assess large language models generating patient-friendly summaries of radiology reports.
- The study evaluated 18,000 pairs using controlled term substitutions and impression removals across CT and MRI reports using six LLMs.
- GPT-5-mini achieved the highest factual faithfulness and certainty fidelity, though term substitutions still caused hallucinations in 28.1% to 30.1% of its summaries.
- Across top models, adversarial testing revealed specific vulnerabilities, including omitted severity descriptors, altered diagnostic certainty, and missed anatomical modifiers.
Why it matters
Adversarial evaluation highlights critical reliability and safety gaps in LLMs applied to medical document summarization, showing that even top-performing models struggle with clinical modifiers and diagnostic certainty.