Source-linked AI summary
Improving Health Literacy through Lay Summarization of Radiological Reports: An Evaluation of BioNER and Retrieval-Augmented Generation
Egecan Çelik Evgin, İlknur Karadeniz, Olcay Taner Yıldız
TL;DR
Patients may struggle to understand specialized radiology reports and increasingly use LLMs despite risks of inaccurate or hallucinated explanations. This study compares NER- and RAG-enhanced summarization across models and learning settings, finding that BioNER consistently improves readability and overall quality, while fine-tuned BioBART with BioNER performs best overall.
Problem
Specialized radiology language is difficult for patients, while the comparative effectiveness of RAG and NER for radiology lay summarization remains underexplored.
Method
The study compares RAG-based and NER-based summarization across few-shot and fine-tuned variants of Qwen and BioBART using four radiology datasets.
Results
BioNER consistently improves readability and overall performance, whereas RAG is inconsistent; fine-tuned BioBART with BioNER achieves the best overall results.
Takeaways & Limitations
Explicit biomedical entity information is a simple, effective enhancement for readable lay summaries, while retrieval effectiveness depends on retrieved-knowledge quality.
Takeaways & Limitations
RAG can retrieve irrelevant or mismatched information that introduces unsupported content, so more reliable knowledge sources and careful checking are needed before practice use.
Abstract
from arXiv · showhide
Radiology reports are written primarily for clinicians, and their specialized terminology often makes them difficult for patients to interpret. As a result, many patients turn to publicly available Large Language Models (LLMs) to help explain their reports, despite well-documented risks of factual inaccuracies and hallucinations. Automated lay-summary generation has emerged as a promising alternative, yet the effectiveness of retrieval-enhanced and clinically informed approaches for radiology-specific communication remains underexplored. This study investigates the extent to which Retrieval-Augmented Generation (RAG) and Named Entity Recognition (NER) improve the quality, factual consistency, and readability of automatically generated lay summaries compared with standard LLM-based generation. We develop a framework combining NER-based extraction of clinically relevant findings with a RAG mechanism for contextual grounding, evaluated across few-shot and fine-tuned variants of two models (Qwen, BioBART). Results show that NER consistently improves readability and overall quality, while RAG alone offers no benefit and can introduce hallucinations from irrelevant retrieved terms. Combining RAG with NER degrades performance in few-shot settings but improves readability when fine-tuned. Fine-tuned BioBART with NER achieves the best overall performance, highlighting entity-aware extraction as the primary driver of improved patient-friendly summaries.
1 Introduction
Radiology reports use specialized clinical language that can be difficult for patients to understand, creating a need for patient-friendly lay summaries. This study compares retrieval-based and entity-aware approaches across models and training settings to address an underexplored research gap.
- Radiology reports primarily communicate imaging findings among healthcare professionals but are difficult for patients to interpret because of specialized terminology.
- Patients increasingly use LLM chatbots for health information, although these systems can generate inaccurate or hallucinated content.
- RAG and NER have been explored to improve lay-summary quality, factual consistency, and relevance, but their comparative effectiveness in radiology remains underexplored.
- The study evaluates a framework combining RAG-based and NER-enhanced radiology summarization across four public datasets spanning diverse clinical settings and imaging modalities.
- The experiments compare a general-purpose LLM with a biomedical small language model and assess both few-shot baselines and fine-tuned variants.
2 Related Work
Prior radiology lay-summarization work has explored RAG, NER, prompting, and fine-tuning, but these strategies have rarely been compared within one framework across substantially different models and training regimes.
- BioLaySumm shared tasks target relevant, readable, and factual biomedical lay summaries, including a radiology-report task.
- Fine-tuning-based approaches: Fine-tuned systems have performed strongly, including AEHRC’s supervised T5-Large system and KHU LDI’s QLoRA-based Qwen pipeline.
- Prompting-based approaches: Prompting-based systems have used large models and selected few-shot radiology examples, while open-weight models have produced more high-risk errors than GPT-4o in one comparison.
- RAG-based approaches: RAG approaches have retrieved medical definitions from resources such as Wikipedia after extracting terms with biomedical tools.
- NER-based approaches: NER-based systems have used biomedical entity labels or definitions to improve technical-term handling, readability, and factuality, sometimes at a small ROUGE cost.
- Few studies directly compare RAG and NER within a unified framework or across models differing in scale and training regime, which this study addresses.
3 Methodology
The methodology compares general-purpose and biomedical language models on radiology lay summarization using multiple datasets, learning settings, enhancement strategies, and evaluation dimensions. BioNER guides generation with clinically relevant entities, while RAG supplies retrieved definitions as contextual grounding.
- The study compares Qwen3.5-0.8B, a general-purpose small model, with BioBARTv2-large, pretrained on biomedical text.
- Four public datasets—PadChest, BIMCV-COVID19+, Open-i, and MIMIC-CXR—cover diverse clinical settings and imaging modalities.
- Evaluation Metrics: The evaluation uses relevance, readability, and factuality dimensions with min-max normalization; lower FKGL, DCRS, and SLE values indicate more readable text.
- Evaluation Metrics: Relevance uses ROUGE, METEOR, and BERTScore, while factuality uses SummaC, FENICE, and CheXbert-F1.
- Learning Settings: Few-shot baselines use 0-shot, 1-shot, and 3-shot prompting, while both models also receive LoRA fine-tuning.
- Enhancement Strategies: BioNER extracts five clinically relevant entity classes and uses them to guide generation toward clinically relevant content.
- Enhancement Strategies: RAG extracts medical terms, retrieves local or Wikipedia descriptions, and provides the definitions as contextual grounding during generation.
- Enhancement Strategies: The combined condition uses BioNER-extracted terms to query the RAG pipeline, yielding baseline, BioNER, RAG, and BioNER+RAG conditions per model and learning setting.
4 Results
Across few-shot and fine-tuned experiments, BioNER consistently improved readability and overall performance, whereas RAG alone was inconsistent and could introduce hallucinations. Fine-tuned BioBART with BioNER achieved the strongest overall results across model configurations.
- BioNER improved overall relevance, readability, and factuality over the 0-shot baseline for Qwen, while RAG did not outperform it.
- In few-shot experiments, combining BioNER with RAG lowered scores across all evaluation dimensions for both Qwen and BioBART.
- After fine-tuning, BioNER improved both models’ overall performance, particularly readability, while RAG improved readability but reduced relevance.
- Fine-tuned BioBART with BioNER outperformed Qwen with 0-shot BioNER across most evaluation metrics, achieving the best overall configuration.
- Explicit biomedical entities helped models identify important concepts and generate clearer lay summaries, while RAG benefits were limited by retrieved-knowledge quality.
- RAG sometimes retrieved irrelevant Wikipedia entries for terms with identical surface forms, and FENICE scores indicated hallucinated information in some summaries.
5 Conclusion
Across few-shot and fine-tuning settings, the study evaluates BioNER- and RAG-based strategies with nine equally weighted metrics spanning relevance, readability, and factuality. BioNER consistently improves baseline models, especially readability, whereas RAG gains are inconsistent and depend on retrieved-knowledge quality.
- The study compares BioNER and RAG strategies for radiology lay summarization under few-shot inference and fine-tuning.
- Nine equally weighted metrics assess relevance, readability, and factuality.
- BioNER consistently improves baseline models, particularly in readability, while maintaining competitive relevance and factuality.
- RAG does not consistently improve performance, and combining it with BioNER does not yield the expected gains.
- Providing explicit biomedical entity information is a simple, effective way to improve readability in automatically generated lay summaries.