Source-linked AI summary
RadEval: A framework for radiology text evaluation
Justin Xu, Xi Zhang, Javid Abderezaei, Julie Bauml, Roger Boodoo, Fatemeh Haghighi, Ali Ganjizadeh, Eric Brattain, Dave Van Veen, Zaiqiao Meng, David Eyre, Jean-Benoit Delbrouck
TL;DR
Radiology report generation requires evaluation of clinical correctness as well as language, yet standardized metrics remain limited for reliable comparison across studies. RadEval unifies radiology text metrics, expert annotations, and benchmarking tools; its analysis identifies GREEN as the most reliable single metric for tracking clinically significant errors.
Problem
Method
Results
GREEN was the most reliable single metric for tracking clinically significant errors, followed by SRR-BERT; within-study analyses found no metric tracked insignificant errors.
Source: §6 · p. 7Takeaways & Limitations
RadEval’s modular architecture is intended to support robust, reproducible, and clinically grounded evaluation of radiology AI.
Source: §7 · p. 7Current evaluations focus primarily on English chest X-ray reports from U.S. institutions, limiting generalizability across languages and regions.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
We introduce RadEval, a unified, open-source framework for evaluating radiology texts. RadEval consolidates a diverse range of metrics, from classic n-gram overlap (BLEU, ROUGE) and contextual measures (BERTScore) to clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1) and advanced LLM-based evaluators (GREEN). We refine and standardize implementations, extend GREEN to support multiple imaging modalities with a more lightweight model, and pretrain a domain-specific radiology encoder, demonstrating strong zero-shot retrieval performance. We also release a richly annotated expert dataset with over 450 clinically significant error labels and show how different metrics correlate with radiologist judgment. Finally, RadEval provides statistical testing tools and baseline model evaluations across multiple publicly available datasets, facilitating reproducibility and robust benchmarking in radiology report generation.
1 Introduction
Radiology report generation needs evaluation that captures clinical correctness as well as language quality, but factuality metrics lack a single reproducible implementation. RadEval combines standardized metrics with expert annotations, ready-made baselines, and statistical testing.
1 Introduction
- Evaluation has progressed from BLEU and ROUGE through disease classification and NLI, semantic graphs, and LLM-based approaches.These approaches represent successive developments in radiology report evaluation.Source: §1 · p. 1
- RadEval re-implements factuality-oriented metrics in one open-source repository, addressing score variation caused by differing implementation choices.Earlier implementations varied in BERTScore settings and in how F1RadGraph judged entity and relation correctness.Source: §1 · p. 1
- Refined metrics aim to provide more faithful estimates of clinical correctness.The paper describes these as corrected and improved versions of existing metrics.Source: §1 · pp. 1–2
- The updated RadEval-expert dataset uses radiologist annotations to support fine-grained study of automatic-metric alignment with human judgments.The annotations cover clinically significant and insignificant errors across different categories.Source: §1 · p. 2
- Published predictions from widely cited models provide out-of-the-box baselines, while permutation and bootstrap tests assess whether score differences are statistically significant.The tests are described as mirroring best practices in image captioning.Source: §1 · p. 2
2 Existing Radiology Report Metrics
Radiology report metrics span lexical overlap, clinical concepts, learned composites, and LLM-based evaluation, but each family has distinct limits in what it measures or how it is implemented.
2.1 Lexical Overlap Metrics
- N-gram metrics such as ROUGE and BLEU are simple baselines, but can underrate reports with paraphrasing or clinically equivalent wording because they are insensitive to meaning and clinical correctness.BERTScore instead uses contextual embeddings to compare semantic similarity, though it remains a lexical-family approach in this overview.Source: §2.1 · p. 2
2.2 Clinical Concept-Based Metrics
- Clinical concept metrics include disease-label F1, entity-relation graphs, synonym-aware diagnostic weighting, and measures of temporal progression or stability.F1CheXbert uses 14 disease labels; SRR-BERT expands the label space to 55 and supports structured reports.Source: §2.2 · p. 2
- F1RadGraph depends on parser accuracy and training-data quality, and simplified entity-relation matching may overestimate alignment in complex cases.Its graph representation captures anatomical and observational entities and their relations.Source: §2.2 · p. 2
2.3 Composite and Learned Metrics
- RadCliQ combines BLEU, BERTScore, CheXbert vector similarity, and F1RadGraph in a linear model trained on radiologist-labeled error counts, improving correlation with expert assessment but reducing interpretability.Source: §2.3 · p. 3
2.4 Generative and LLM-Based Evaluation Metrics
- LLM-based evaluators assess correctness, style, or completeness; examples include error categorization, line-level severity scoring, and sentence-level factuality analysis.CheXprompt, FineRadScore, and RadFact illustrate these approaches.Source: §2.4 · p. 3
- GPT-based evaluation can involve high computational cost, deployment barriers, inconsistent outputs, and privacy concerns; RadEval supports local, privacy-preserving evaluation with open-source GREEN.Source: §2.4 · p. 3
3 RadEval
Radiology report generation lacks standardized evaluation metrics, hindering reliable comparison across studies. RadEval consolidates diverse metrics in a modular system to address fragmentation and support reproducible benchmarking.
3 RadEval
- A lack of standardized evaluation metrics hinders reliable analysis and comparison across studies and limits the real-world applicability of research progress.Source: §3 · p. 3
- Metrics often require separate installations and preprocessing, while some lack public code or use proprietary APIs, and outputs vary in format and interpretability.Source: §3 · p. 3
- RadEval consolidates RRG metrics from n-gram baselines to LLM evaluators in a modular system designed for plug-and-play integration of new metrics.Source: §3 · p. 3
4 Benchmarking
RadEval benchmarks a representative set of radiology report generation systems on public datasets, evaluating their Findings and Impression sections under dataset- and model-dependent image-input settings.
4 Benchmarking
- Benchmarking covers publicly available datasets, using official test splits for MIMIC-CXR and ReXGradient-160K and the public validation set for CheXpert Plus.Source: §4 · p. 3, §4.1 · p. 3
4.1 Datasets
- Evaluation retains associated images when available, uses one representative image or all images depending on model support, and excludes reports missing Findings or Impression.Source: §4.1 · pp. 3–4
4.2 Baselines
- The baseline set represents systems from different institutions and architectures.Source: §4.2 · p. 4
5 RadEvalBERTScore
RadEvalBERTScore is a radiology-domain text encoder evaluated for zero-shot report retrieval. The experiments assess ranking on single- and multi-label datasets using complementary retrieval metrics.
5 RadEvalBERTScore
- A ModernBERT-base encoder is contrastively pretrained with SimCSE on Findings and Impression text from MIMIC-CXR, CheXpert, and ReXGradient-160K.Source: §5 · p. 4
- Zero-shot retrieval encodes query and candidate reports, ranks candidates by cosine similarity, and treats reports sharing at least one label as relevant.Source: §5 · p. 4
mean Average Precision (mAP)
- Average Precision averages precision at ranks containing relevant reports, stops at the last relevant rank, and mAP averages AP across queries.Source: mean Average Precision (mAP) · p. 5
CheXbert Test Set
- The single-label CheXpert experiments cover five categories, while the CheXbert test set uses 14 possible findings and reports Precision@{5, 10}, nDCG@{5, 10}, and mAP.Source: §5.2 · p. 5, CheXbert Test Set · p. 5, Table 1 · p. 5
HOPPR 8×200
- HOPPR 8×200 is an out-of-domain, single-label dataset used to assess generalization, with Precision@{5, 10, 50, 100} and mAP reported.Source: HOPPR 8×200 · p. 5, Table 1 · p. 5
5.1 Metrics
- The protocol uses Precision@K for top-K exact-hit rate, mAP for early ranking, and nDCG@K for sensitivity to the number of shared labels.Source: HOPPR 8×200 · p. 5, §5.1 · p. 5, mean Average Precision (mAP) · p. 5
6 RadEval Expert Dataset
The expert dataset extends radiologist error annotation to Findings as well as Impression, and the metric analysis identifies GREEN as the most reliable single tracker of clinically significant errors.
6 RadEval Expert Dataset
- The updated RadEval-expert dataset annotates clinically significant and insignificant errors, extending coverage to Findings and adding inarticulate report/grammar as an error category.Source: §6 · pp. 5, 7
- GREEN is the most reliable single metric for tracking clinically significant errors, followed by SRR-BERT; within-study analyses also align GREEN, BERTScore, and BLEU with significant-error totals.Source: Table 2 · p. 6, §6 · p. 7
- Metric agreement is measured with pooled and within-study Kendall’s τb analyses, with block-bootstrap confidence intervals for blocked comparisons.Source: §6 · p. 7
7 Conclusion
RadEval unifies and standardizes diverse radiology report evaluation metrics, addressing reproducibility and benchmarking challenges. The framework also includes an expert-annotated test set, model benchmarks, a domain-specific encoder, and an updated lightweight GREEN metric.
7 Conclusion
- RadEval consolidates lexical, clinical concept, structured graph, and LLM-based metrics in a unified framework for reproducible radiology report evaluation.The authors describe its modular architecture as supporting robust, reproducible, and clinically grounded evaluation.Source: §7 · p. 7
- The work releases a high-fidelity expert-annotated test set and benchmarks state-of-the-art report-generation systems across multiple public datasets.It also introduces statistical testing tools and baseline evaluations, as stated in the paper context.Source: §7 · p. 7
- A new domain-specific sentence encoder demonstrates utility in a zero-shot retrieval task.The conclusion does not report retrieval results here.Source: §7 · p. 7
- An updated lightweight GREEN metric supports cross-modality evaluation, and the authors frame RadEval as helping accelerate safe deployment of radiology AI.The conclusion describes the framework as modular and clinically grounded.Source: §7 · p. 7
Limitations
RadEval does not implement some recently proposed LLM metrics, and some included metrics rely on potentially noisy upstream parsers. Current evaluations are concentrated on English chest X-ray reports from U.S. institutions.
Limitations
- CheXprompt, FineRadScore, and RadFact are not implemented because they rely on LLM APIs or lack standardization.The authors identify them as possible future additions.Source: Limitations · p. 7
- Some metrics depend on upstream parsers that may introduce noise.Source: Limitations · p. 7
- Current evaluations focus primarily on English chest X-ray reports from U.S. institutions, limiting generalizability across languages and regions.The authors also plan to expand the expert-labeled dataset and compute more detailed metric-correlation analyses.Source: Limitations · p. 7
A Score table
Table 3 covers model benchmarks on standard datasets under default system prompts; the supplied entries include dataset names and numerical values.
A Score table
- Table 3 benchmarks multiple models on standard datasets under default system prompts.Source: Table 3 · p. 11
- One listed entry pairs CheXpert-Plus with 23.7 and Med-CXRGen-I with 26.6.Source: A Score table · p. 11
- Another listed entry pairs CheXpert-Plus with 25.0 and Med-CXRGen-I with 20.8.Source: A Score table · p. 11
- A further listed entry pairs CheXpert-Plus with 3.8 and Med-CXRGen-I with 22.3.Source: A Score table · p. 11
B Refined GREEN Metric
The refined GREEN metric uses a compact Gemma-2B model trained on the original GREEN data and 50,000 annotated report pairs across multiple imaging modalities. It reduces inference time while maintaining performance comparable to larger models.
B Refined GREEN Metric
- The refined GREEN metric fine-tunes Gemma-2B on the original GREEN dataset and 50,000 additional annotated report pairs spanning CT, MRI, and ultrasound.The original implementation focused exclusively on chest X-rays.Source: B Refined GREEN Metric · p. 12
- Inference averages 2–3 seconds per report, with performance comparable to larger models such as Llama and Phi.Source: B Refined GREEN Metric · p. 12
- The authors say the reduced resource footprint makes the updated model more practical for large-scale or real-time deployment.They also propose exploring more diverse imaging contexts and lighter architectures without sacrificing clinical alignment or interpretability.Source: B Refined GREEN Metric · p. 12