PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards

Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, Curtis P. Langlotz

arXiv:2210.12186v1cs.CLcs.AI

TL;DR

Radiology report generators can perform well on NLG metrics while remaining factually incomplete or inconsistent. This paper uses RadGraph annotations to create reinforcement-learning rewards, reporting factual-metric improvements on two chest X-ray datasets.

  • Problem

    Radiology report generators can score highly on NLG metrics while producing reports that are factually incomplete or inconsistent, and more complex architectures have yielded only marginal NLG gains.

    Source: §1 · p. 1

  • Method

    The authors use RadGraph’s annotated chest X-ray entities and relations to define semantic rewards that they optimize with reinforcement learning.

    Source: §1 · p. 2

  • Results

    Up to +14.2% on MIMIC-CXR and +25.3% on Open-i on metrics evaluating factual correctness and completeness.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The authors report that directly optimizing these rewards outperforms approaches prioritizing traditional NLG metrics or using unsupervised out-of-domain factual metrics.

    Source: §8 · p. 9

    Optimizing rewards based only on entities, or on entities and relations, can come at the cost of grammar; the authors combine an RG metric with BERTScore and NLL.

    Source: §9 · p. 9

Abstract

from arXiv · show

Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. These systems have achieved promising performance as measured by widely used NLG metrics such as BLEU and CIDEr. However, the current systems face important limitations. First, they present an increased complexity in architecture that offers only marginal improvements on NLG metrics. Secondly, these systems that achieve high performance on these metrics are not always factually complete or consistent due to both inadequate training and evaluation. Recent studies have shown the systems can be substantially improved by using new methods encouraging 1) the generation of domain entities consistent with the reference and 2) describing these entities in inferentially consistent ways. So far, these methods rely on weakly-supervised approaches (rule-based) and named entity recognition systems that are not specific to the chest X-ray domain. To overcome this limitation, we propose a new method, the RadGraph reward, to further improve the factual completeness and correctness of generated radiology reports. More precisely, we leverage the RadGraph dataset containing annotated chest X-ray reports with entities and relations between entities. On two open radiology report datasets, our system substantially improves the scores up to 14.2% and 25.3% on metrics evaluating the factual correctness and completeness of reports.

1 Introduction

Radiology report generation could reduce repetitive drafting and improve clinical communication, but strong NLG scores do not ensure factual completeness or consistency. The paper proposes RadGraph-based semantic rewards and a simple architecture for training report generators.

1 Introduction

  • Systems with high BLEU and CIDEr scores may still be factually incomplete or inconsistent, while architectural complexity has brought only marginal NLG improvements.The authors attribute these shortcomings to inadequate training and evaluation.Source: §1 · p. 1
  • Prior factual methods encourage reference-consistent entities and inferentially consistent descriptions, but rely on weak supervision or biomedical entity recognition not specific to chest X-rays.The paper proposes RadGraph-based rewards using radiologist-annotated chest X-ray reports and their entities and relations.Source: §1 · p. 2
  • The proposed simple report-generation architecture is fast to train, suitable for reinforcement learning, and performs equally well as earlier, more complex architectures.The paper also proposes semantic rewards to evaluate generated reports’ factual correctness and completeness.Source: §1 · p. 2
  • On two datasets, directly optimizing the proposed rewards outperforms prior approaches that prioritize traditional NLG metrics.The introduction presents this as the paper’s central result.Source: §1 · p. 2

2 Factually-oriented Graph-Based Reward

This section introduces RadGraph’s chest X-ray report annotations and describes how the authors use them to construct semantic rewards. The rewards compare report annotations represented as graphs.

2.1 RadGraph

  • RadGraph contains radiologist-annotated chest X-ray reports, with entities and relations structured using an information-extraction schema.Its MIMIC-CXR annotations cover 500 reports, 14,579 entities, and 10,889 relations; its test set contains 100 reports from MIMIC-CXR and CheXpert.Source: §2.1 · pp. 2–3
  • RadGraph entities are anatomy or observations labeled as definitely present, uncertain, or definitely absent; relations encode links such as inference, location, or modification.The paper’s examples include “lung” as anatomy and “effusion” as an observation.Source: §2.1 · p. 3
  • The authors use a PubMedBERT model fine-tuned on RadGraph to annotate the reports in their experiments.Source: §2.1 · p. 3

2.2 RadGraph reward

  • The rewards use F-score-style comparisons between generated and reference annotation sets.The paper defines the variants’ scores as the harmonic mean of precision and recall.Source: §2.2 · pp. 3–4
  • The authors represent report annotations as directed graphs, with entities as labeled nodes and relations as labeled edges, and apply a fine-tuned RadGraph model to generated and reference reports.The graph direction distinguishes (v1, v2) from (v2, v1).Source: §2.2 · p. 3
  • The three reward variants compare entity labels, entity labels plus relation presence, or entities and their relations between generated and reference reports.Their sets are scored using the harmonic mean of precision and recall.Source: §2.2 · pp. 3–4

3 Model

The model encodes X-ray visual features and generates reports with a transformer using cross-attention. It is trained first with negative log-likelihood and then with reinforcement learning that optimizes a factually oriented reward.

3 Model

  • X-ray features from DenseNet-121 condition language generated by a one-layer BERT with cross-attention.The visual feature representation has size 49 × 1024.Source: §3 · p. 4
  • Training first minimizes the report’s negative log-likelihood, then uses reinforcement learning to optimize a factually oriented reward.The reward-based loss replaces the negative log-likelihood objective during the reinforcement-learning stage.Source: §3 · p. 4
  • SCST approximates the expected gradient of the non-differentiable reward, using sampled expected reward as a variance-reduction baseline.The reward is assigned to the generated report.Source: §3 · p. 4
  • The model uses one 768-size Transformer block, a 3072-size feed-forward layer, Adam at 3e−4, batches of 128, and beam search of size 3.Source: §3 · p. 4

4 Datasets

The experiments use MIMIC-CXR for training, validation, and testing, while Open-i is reserved for testing because its filtered report set is too small for both training and testing.

4 Datasets

  • MIMIC-CXR contributes 152,173 training, 1,196 validation, and 2,347 test samples after reports without Findings are discarded.Source: §4 · p. 4
  • Open-i is used only for testing because filtering out reports without Findings leaves the dataset too small to train and test a report-generation model.The filtered dataset contains 3,335 reports, excluding image multiplicity within studies.Source: §4 · p. 4
  • CheXbert labels are generated from each reference Findings section to provide diagnostic labels for calculating F1 CheXbert.The labels cover 14 possible observations and their degree of presence.Source: §4 · p. 4

5 Metrics

Evaluation covers both conventional NLG similarity and factual correctness and completeness. The section defines entity- and diagnosis-based measures and contrasts RadGraph reward with factENT and factENTNLI.

5 Metrics

  • The evaluation separates widely reported NLG measures from metrics aimed at factual correctness and completeness of generated Findings.Source: §5 · pp. 4–5, §5.2 · p. 5

5.1 NLG-oriented

  • BLEU, CIDEr, and ROUGE-L measure NLG performance, with ROUGE-L included for consistency with prior work.Source: §5.1 · p. 5

5.2 Factually-oriented

  • factENT computes the harmonic mean of precision and recall between entities extracted from the generated report and its reference.The described implementation uses Stanza’s clinical NER.Source: §5.2 · p. 5
  • factENTNLI extends entity matching by rejecting an extracted entity when its sentence contradicts the highest-BERTScore counterpart sentence in the reference.The NLI model determines whether the reference counterpart contradicts the generated sentence.Source: §5.2 · p. 5
  • F1CheXbert compares CheXbert predictions for generated and reference reports, calculating F1 over five observations.CheXbert predicts abnormalities across fourteen chest X-ray classes.Source: §5.2 · p. 5
  • The RadGraph rewards described earlier are also used as evaluation scores.Source: §5.2 · p. 5

5.3 RadGraph vs factENT and factENTNLI

  • RGE combines factENT and factENTNLI concepts, evaluates entities and their labels with chest-X-ray-specific NER, and avoids an external model for sentence alignment.Entity labels represent anatomy and observation status, enabling penalties for contradictions with the reference.Source: §5.3 · p. 5

6 Results

RGER improves factual-metric results on MIMIC-CXR and out-of-domain Open-i, while radiologists favor its reports over factENTNLI findings. The analyses also reveal uneven coverage across entity and relation labels.

6.1 Quantitative analysis

  • The NLL model performs similarly on NLG metrics but exceeds Miura et al. (2021) by 2.3% on factENT and 8.0% on factENTNLI on MIMIC-CXR.It is better in BLEU but worse in ROUGEL than Chen et al. (2021).Source: §6.1 · p. 5
  • Optimizing any presented reward improves RadGraph scores, and radiologists on average favor RGER over factENTNLI findings.Two board-certified radiologists compared findings for up to a hundred studies.Source: §6.1 · p. 5, §6.2 · p. 7
  • On MIMIC-CXR, RGER improves factENTNLI by +14.2% over Miura et al. (2021), and on out-of-domain Open-i it improves factENTNLI by 16.7% (55.0 vs 47.1).Table 1 compares models across NLL/RL and MIMIC-CXR/Open-i categories.Source: §6.1 · p. 6, Table 1 · p. 6
  • Generated relations were correct at 20.4% under the stricter criterion and 38.7% under the relaxed criterion, which counts relation presence regardless of label or target.The authors hypothesize that the relaxed constraint encourages relation generation and more factually correct and complete reports.Source: §6.1 · pp. 6–7

6.2 Qualitative analysis

  • On MIMIC-CXR, RGER generates 20% more Anatomy entities than reference reports but barely generates the under-represented OBS-U and suggestive_of labels.For the four most frequent labels, generated entities and relations range from 12% fewer to 24% more than references.Source: §6.2 · p. 7
  • RGER findings have median word length 7% below references, compared with 25% below references for NLL findings on MIMIC-CXR.In cherry-picked examples, RGER output length is closer to reference length than NLL output length.Source: §6.2 · pp. 7–8, Table 2 · p. 7

7 Related work

Prior radiology-report work explores architectural changes, reinforcement learning, and retrieval. This paper distinguishes its graph-based approach by using radiologist-annotated, chest-X-ray-specific graphs as an optimizable evaluation reward rather than model input.

Architectural novelties

  • Earlier architectures use relational or cross-modal memories, prior-and-posterior knowledge, or unsupervised knowledge graphs to support report generation.Examples include memory-driven Transformers, cross-modal memory networks, and a Knowledge Graph Auto-Encoder.Source: §7 · p. 8
  • Unlike the cited graph-input studies, this work uses a graph generated from high-quality human-annotated data as an evaluation metric directly optimized with reinforcement learning.The reward captures chest-X-ray-specific entities and relations, including anatomy, observation, absent, and present labels.Source: §7 · p. 8

Reinforcement learning

  • Prior RL work optimized CheXpert output correctness, while Miura et al. (2021) introduced factENT and factENTNLI metrics for factual correctness and consistency.The paper includes the latter metrics in its evaluation.Source: §7 · p. 8

Other works

  • Retrieval approaches either select training-set reports or combine template retrieval with sentence generation, while Yang et al. (2021) supplied RadGraph annotations as model input.The paper contrasts its use of the graph as an optimizable evaluation metric with using it as input.Source: §7 · p. 8

8 Conclusion

The paper uses radiologist-annotated RadGraph entities and relations to design a reward for factual correctness and completeness. Across two datasets, directly optimizing these rewards outperforms approaches prioritizing traditional NLG metrics or using unsupervised out-of-domain factual metrics.

8 Conclusion

  • On MIMIC-CXR and Open-i, the best model reports up to +14.2% and +25.3% improvements on factual metrics.The paper compares direct reward optimization with approaches prioritizing traditional NLG metrics or using unsupervised out-of-domain factual metrics.Source: §8 · p. 9

9 Limitations

The limitations include trade-offs from reward optimization, weak entity-relation performance, and expensive RL training; the appendices also provide code and supplementary representations.

9 Limitations

  • Entity-only or entity-and-relation rewards can harm grammar, so training combines an RG metric with BERTScore and NLL using weights 0.495, 0.495, and 0.01.Source: §9 · p. 9
  • Figure 4 compares per-label entity and relation counts from RGER with counts from RadGraph on MIMIC-CXR test references.Source: Figure 4 · p. 9
  • Excluding two under-represented labels, macro-averaged recall is 40.9% for entities but 18.7% for relations.Source: §9 · p. 9
  • OBS-U recall is 0; 30% of reference OBS-U words are generated as OBS-DP or OBS-DA, while most relation errors reflect absent relations.Source: §9 · p. 9
  • RL training takes 7–10 hours per MIMIC-CXR epoch, versus 50 minutes for NLL training, although the model fits on a 12 GB GPU.Source: §9 · p. 9

A Code release

  • The authors release code for the factually oriented metrics and SCST implementation through ViLMedic to support reproducibility and fairer comparisons.Source: A Code release · p. 12

B Set ¯V of Figure 3

C Qualitative Study

In a radiologist comparison, RGER reports were preferred on average to factENTNLI reports, though not systematically; the section also describes the comparison protocol and model configuration.

C Qualitative Study

  • Radiologists compared paired reports for factual correctness, completeness, and consistency on a −2-to-2 scale; the study selected 100 clinical cases with at least 10 instances per abnormality.Source: C Qualitative Study · p. 12
  • Both radiologists preferred RGER reports on average to factENTNLI reports, although the reports were not systematically better.Source: C Qualitative Study · p. 12

D Model

  • The model uses DenseNet121 features with a BERTGeneration decoder and has 25.8 million learnable parameters, fitting on a single 12 GB GPU.Source: D Model · p. 13
  • Training uses batch size 128 and RAdam, with ROUGEL monitored during evaluation for plateau scheduling and early stopping.Source: D Model · p. 13

E Results on the impression section

The passage identifies Table 4 as reporting RGER results when trained to generate MIMIC-CXR Impression sections rather than Findings.

E Results on the impression section

  • Table 4 reports results for RGER trained on MIMIC-CXR Impression sections instead of Findings.Source: Table 4 · p. 13
  • The table concerns the Impression-generation setting rather than the Findings-generation setting.Source: Table 4 · p. 13
  • The caption specifies the model and training-section comparison but gives no metric values.Source: Table 4 · p. 13
Loading 2210.12186v1…