Source-linked AI summary
GREEN: Generative Radiology Report Evaluation and Error Notation
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, Jean-Benoit Delbrouck
TL;DR
Radiology report metrics need to assess factual correctness and uncertainty while producing interpretable feedback for expert review. GREEN uses a language-model evaluator to quantify and explain clinically significant errors through an interpretable score and summaries. The paper reports stronger alignment with expert evaluations than existing metrics, while noting that fine-tuning may be needed for new imaging modalities and that error counts retain some randomness.
Problem
Radiology report metrics must capture factual correctness and uncertainty while remaining interpretable, but existing approaches have limitations in these areas.
Method
GREEN uses a language-model evaluator to identify and categorize clinically significant errors, compute a score from matched findings and error counts, and generate free-text explanations.
Results
GREEN aligns more closely with expert evaluations than existing metrics while providing a score and interpretable error explanations.
Takeaways & Limitations
GREEN offers a lightweight open-source approach for clinically relevant, interpretable radiology report evaluation across imaging modalities.
Takeaways & Limitations
Fine-tuning GREEN for new imaging modalities may be required, and error quantification retains some randomness due partly to expert disagreement.
Abstract
from arXiv · showhide
Evaluating radiology reports is a challenging problem as factual correctness is extremely important due to the need for accurate medical communication about medical images. Existing automatic evaluation metrics either suffer from failing to consider factual correctness (e.g., BLEU and ROUGE) or are limited in their interpretability (e.g., F1CheXpert and F1RadGraph). In this paper, we introduce GREEN (Generative Radiology Report Evaluation and Error Notation), a radiology report generation metric that leverages the natural language understanding of language models to identify and explain clinically significant errors in candidate reports, both quantitatively and qualitatively. Compared to current metrics, GREEN offers: 1) a score aligned with expert preferences, 2) human interpretable explanations of clinically significant errors, enabling feedback loops with end-users, and 3) a lightweight open-source method that reaches the performance of commercial counterparts. We validate our GREEN metric by comparing it to GPT-4, as well as to error counts of 6 experts and preferences of 2 experts. Our method demonstrates not only higher correlation with expert error counts, but simultaneously higher alignment with expert preferences when compared to previous approaches.
1 Introduction
Radiology report evaluation must capture factual correctness, uncertainty, and clinically meaningful errors while remaining interpretable for expert review. GREEN addresses these gaps with a validated score, human-readable error summaries, lightweight open-source models, privacy-oriented applicability, multimodality evaluation, and shared datasets.
- Motivation: Radiology report metrics must assess factual correctness, uncertainty, and interpretability to support expert review and clinical communication.Existing metrics are described as insufficient for the nuanced requirements of radiology reports.
- Contributions: GREEN introduces a score ranging from 0 to 1 that assesses factual correctness and uncertainty beyond prior approaches.The paper states that the score is validated and designed for radiology language.
- Contributions: GREEN generates test-set-size-independent, human-readable summaries that categorize and explain errors for system improvement.The summaries are intended to help practitioners and experts pinpoint areas for improvement.
- Contributions: GREEN uses an open-source language model with fewer than 7B parameters, reducing GPU requirements and improving processing speed.The paper presents this as a practical alternative with similar evaluation abilities to larger counterparts.
- Contributions: GREEN is designed for confidential datasets without patient privacy concerns because it is free and open-source.This addresses privacy requirements associated with commercially available LLM services.
- Contributions: GREEN is evaluated for generalized medical-language understanding across imaging modalities using zero-shot abdominal CT reports, and the authors share 150,000 annotations for further research.The shared resources include 100,000 chest-X-ray annotations and 50,000 annotations across diverse imaging modalities.
2 Related Work
Radiology report generation metrics range from lexical similarity to factuality-focused measures, but prior methods remain limited in scope or interpretability. GREEN emphasizes clinical relevance and interpretability while reporting stronger alignment with expert judgments.
- Prior Metrics: Existing metrics evaluate reports using n-gram overlap, general language similarity, pathology labels, entity graphs, or commercial language models.Examples include ROUGE-L, BLEU, F1CheXbert, and F1RadGraph.
- Limitations: F1CheXbert covers only 14 specific chest-X-ray finding classes, limiting its pathology scope.It evaluates disease-label accuracy against a narrow reference.
- GREEN: GREEN emphasizes clinical relevance and interpretability while showing higher alignment with expert error counts and preferences than previous metrics.The comparison retains an open-source model below 7B parameters.
3 GREEN
GREEN combines a generative evaluator, a score based on matched findings and clinically significant errors, and free-text error summaries. It is trained on large synthetic and real report-pair resources and validated against expert assessments and preferences.
- Overview: GREEN has three components: a generative model for six error categories, the GREEN score, and a text-form summary.The evaluation pipeline also validates the method quantitatively and qualitatively.
- Model and Data: GPT-4 generates GREEN analyses from prompts containing judgment criteria, reference reports, synthetic candidates, and a response template.The analyses provide the basis for numerical scores and free-text summaries.
- Model and Data: 100,000 reference-candidate pairs were compiled from six publicly available de-identified chest-X-ray datasets.The resulting collection used 174,329 unique reports as references or candidates.
- Model and Data: Five pairing heuristics generated 20,000 unique pairs per heuristic, including random matching, report modification, RRG outputs, BERTScore matching, and RadGraph permutations.Pairs formed under one heuristic were excluded from the others to preserve uniqueness.
- Model and Data: The evaluator was built by pretraining and fine-tuning open-source models, including domain-adapted RadLLaMA-2 and RadPhi-2.The approach avoids relying on API-based models for a local evaluator.
- GREEN Score: The GREEN score rewards matched findings and penalizes clinically significant errors, producing a value bounded between 0 and 1.The score becomes harder to increase as reports become more concise and accurate.
- GREEN Summary: GREEN summaries explain errors across six subcategories and cluster representative explanation sentences for interpretable reporting.The model response includes explanations, clinically significant and insignificant error counts, and matched findings.
- Validation: Validation uses expert error counts and pairwise preferences, including ReXVal assessments from six radiologists and preferences from two radiologists.The ReXVal set contains 200 report pairs from 50 MIMIC-CXR cases, while the preference data contains 100 pairwise judgments.
4 Experiments
GREEN was evaluated against expert error counts, GPT-4, classical metrics, and expert preferences. It showed competitive correlation with radiologists and stronger preference alignment than summed error counts, while anatomical-location errors remained its largest deviation.
- Expert Error Counts: 1.54 ± 1.36 significant-error difference for RadLLaMA-2 approached GPT-4’s 1.51 ± 1.29 difference from the mean radiologist.Against individual experts, RadLLaMA-2’s average difference was 1.02 ± 0.27, within the inter-expert difference of 0.83 ± 0.13.
- Expert Error Counts: 0.63 correlation placed GREEN within the inter-expert range of 0.48 to 0.64 on the same examples.This enabled cross-metric and cross-study comparisons, although the authors noted clinical limitations of total error counts.
- Metric Correlations: 0.79 correlation between GREEN’s unweighted total error count and radiologists’ total error counts was statistically indistinguishable from experts and GPT-4-based systems.The comparison used the sum of clinically significant and insignificant errors.
- Expert Error Counts: Misidentification of a finding’s anatomic location or position produced GREEN’s largest deviation compared with all six experts and GPT-4.The fine-grained analysis categorized significant errors across six subcategories.
- Expert Preferences: GREEN and GREEN-GPT-4 achieved the highest preference accuracy, outperforming summed error counts, while direct GPT-4 preferences reached 0.23 (95% CI, 0.13–0.36).GREEN’s preference alignment improved as radiologist confidence increased, unlike total error counts or direct GPT-4 preferences.
5 Multimodality Generalizability
The multimodality evaluation tested GREEN beyond chest X-rays using synthetically modified reports from diverse imaging modalities. Zero-shot performance was weaker on external out-of-distribution data, while multimodality fine-tuning improved significant-error differences and text similarity.
- Dataset Construction: 50,000 candidate reports were generated from diverse imaging modalities using sentence rearrangement, sentence removal, random pairing, and GPT-4-injected errors.The resulting data were split into training, validation, and test sets at an 80/10/10 ratio and combined with the chest X-ray dataset.
- Fine-Tuning: Further fine-tuning on multimodality data improved significant-error differences and text-similarity metrics for in-distribution and out-of-distribution settings.The model was fine-tuned for eight epochs with a batch size of 80 using the stated GREEN hyperparameters.
6 Conclusion
The paper concludes that GREEN improves radiology-report evaluation by combining factual-error assessment with expert-aligned scoring. Its open, lightweight, privacy-preserving design and out-of-distribution performance support broader use across imaging modalities.
- Contributions: GREEN is presented as a metric that assesses factual correctness and uncertainty while aligning closely with expert evaluations.The conclusion characterizes GREEN as a novel metric for enhancing radiology-report evaluation.
- Practicality: GREEN’s open-source and lightweight design supports privacy-preserving use, lower computational demands, and collaborative improvement across settings.The authors also describe adaptability across imaging modalities and extensive datasets as supporting broader applicability.
- Generalizability: Robust performance on out-of-distribution data indicates GREEN’s versatility for future automated radiology-reporting developments.The conclusion frames this performance as supporting potential use as a standard for future developments.
7 Limitations
GREEN’s limitations include nontrivial computational cost, possible fine-tuning requirements for new imaging modalities, and some uncontrollable variability in error quantification.
- 3.75 seconds per sample on one A100 GPU makes GREEN slower than ROUGE-L at 0.015 seconds per sample.Batching reduces processing to 1.06 seconds per sample for four samples in 4.22 seconds.
- GREEN remains faster than GPT-4, which takes up to 22.0 seconds per sample.
- Fine-tuning GREEN for new imaging modalities might be required to ensure satisfactory performance.The paper also describes adapting GREEN to different imaging techniques without an initial RRG model for each technique.
- Although GREEN operates deterministically, error quantification remains partly uncontrollable and introduces randomness into error counts.The paper associates this variability with uncertainty in the task and disagreement among experts on fine-grained error counts.
A.1 GPT-4 Prompt Template for Generation of Training Data
The GPT-4 training-data prompt asks the model to compare candidate and reference radiology reports, count clinically significant and insignificant errors, classify error types, and return structured explanations and matched findings.
- The prompt’s objective is to evaluate a candidate radiology report against a reference report written by expert radiologists.
- GPT-4 receives judgment criteria, reference and candidate reports, and a required assessment format.
- The assessment counts clinically significant and clinically insignificant errors.
- Six error categories cover false findings, missing findings, anatomic misidentification, severity errors, unsupported comparisons, and omitted change comparisons.
- The prompt prioritizes clinical findings over writing style and evaluates only findings appearing in both reports.
- The required output separately reports explanations, clinically significant errors, clinically insignificant errors, and matched findings.
A.2 GPT-4 Response used as Training Data
A sample GPT-4 response illustrates how GREEN training data records a candidate’s clinically significant error while distinguishing matched findings and zero insignificant errors.
- The example uses a reference report and a candidate report differing in the anatomic description of infiltrates.
- The candidate changes the infiltrates’ location from the upper middle right field to the upper dorsal right field.
- This change is labeled one clinically significant error involving misidentification of anatomic location or position.
- The response assigns zero clinically insignificant errors across all six error categories.
- Three findings are matched: doubtful retrocardiac infection, clear costophrenic sinuses, and no other notable findings.
A.3 GPT-4 Prompt Template for Preference Evaluation
The appendix specifies GPT-4 prompts for preference evaluation, report modification across modalities, summary visualization, and statistical comparisons of expert and model error counts.
- A.3 GPT-4 Prompt Template for Preference Evaluation: The preference prompt compares two AI-generated reports against a radiologist-written reference report.
- A.3 GPT-4 Prompt Template for Preference Evaluation: Preference judgments prioritize clinical accuracy, especially positive findings, over report length and style.
- A.3 GPT-4 Prompt Template for Preference Evaluation: GPT-4 first compares each generated report with the reference, then labels one assistant better, worse, or equal.
- A.4 Algorithm for Modifying Radiology Reports: The report-modification algorithm generates prompts that introduce specified error types into reports, sometimes changing only one word subtly.
- A.5 Visualization of the GREEN Summary Clustering Technique: The GREEN summary visualization clusters sentences separately for each error subcategory using t-SNE.
- A.6 Test for difference in location of the mean errors: Table 10 tests whether experts’ error counts differ significantly from the mean expert.
- A.6 Test for difference in location of the mean errors: Table 11 tests whether model error counts differ significantly from the mean expert.
- A.7 Fine-grained interexpert correlation: Figure 6 presents a Kendall’s Tau correlation matrix for mean-expert and inter-expert fine-grained error counts on the RexVal external validation set.