PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation

Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P. Langlotz, Dan Jurafsky

arXiv:2010.10042v2cs.CL

TL;DR

Radiology report generators can score highly on standard NLG metrics while remaining factually incomplete or inconsistent. The paper introduces entity-matching and NLI-based rewards, combines them with BERTScore, and optimizes them through reinforcement learning; on two open datasets, the resulting system achieves substantially higher clinical-metric performance and more complete, consistent reports than baselines. Its reports nevertheless remain limited in handling comparisons with prior studies and uncertainty.

  • Problem

    Radiology report generation systems can achieve high CIDEr or BLEU scores while still producing factually incomplete and inconsistent reports.

  • Method

    The paper combines entity-based completeness and NLI-based consistency rewards with BERTScore and directly optimizes them in a Transformer-based generator using reinforcement learning.

  • Results

    ∆+ 64.2% clinical-metric performance was achieved on two publicly available datasets, with reports more factually complete and consistent than baselines.

  • Takeaways & Limitations

    Entity and NLI signals can be integrated into reinforcement-learning-based report generation to improve factual completeness and consistency in radiology reports.

  • Takeaways & Limitations

    The model cannot capture comparisons with prior studies because it does not take patients’ past reports as input, and it may turn uncertainty into definite findings.

Abstract

from arXiv · show

Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural language generation metrics such as CIDEr or BLEU, still suffer from incomplete and inconsistent generations. Here we introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate radiology domain entities consistent with the reference, and one that uses natural language inference to encourage these entities to be described in inferentially consistent ways. We combine these with the novel use of an existing semantic equivalence metric (BERTScore). We further propose a report generation system that optimizes these rewards via reinforcement learning. On two open radiology report datasets, our system substantially improved the F1 score of a clinical information extraction performance by +22.1 (Delta +63.9%). We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines.

1 Introduction

Radiology report generation can reduce repetitive reporting work, but systems scoring well on standard NLG metrics may still produce factually incomplete or inconsistent reports. The paper proposes entity- and inference-based rewards, combined with BERTScore and reinforcement learning, to improve clinical report quality.

  • Motivation: Radiology report generation uses X-ray images to produce textual descriptions of clinical observations and may reduce radiologists’ repetitive work.The task is framed as an assistive application of natural language generation.
  • Problem: High CIDEr and BLEU scores do not ensure factual completeness or consistency in generated radiology reports.Existing systems can omit image findings or describe absent findings, producing clinically poor reports.
  • Proposed approach: factENT measures generated coverage of radiology disease and anatomy entities against a reference report.The reward is designed to capture disease and anatomical knowledge encoded in domain entities.
  • Proposed approach: factENTNLI extends entity matching with natural language inference to assess whether generated entities are inferentially consistent with reference descriptions.The NLI component is intended to control disease overestimation when optimizing entity coverage.
  • Proposed approach: The report generation model optimizes factENT, factENTNLI, and BERTScore with reinforcement learning on two publicly available radiology datasets.The approach also includes a weakly supervised method for adapting an NLI model to the radiology domain.
  • Results: The proposed approach substantially improves clinical metrics, with gains as high as ∆+ 64.2% on two publicly available datasets.The authors report that optimizing the proposed rewards with RL improves clinical information extraction performance relative to prior approaches.

2 Related Work

Prior radiology report systems combined image understanding with disease classification, retrieval, structure, clinical rewards, or memory-driven Transformers. Related work also addressed factual consistency with NLI and QA, while Transformer architectures increasingly incorporated attention and semantic or visual information.

  • Radiology report generation: Early radiology systems jointly generated reports and classified disease labels from chest X-rays.Later extensions incorporated multiple images, hybrid retrieval-generation, and structural information.
  • Radiology report generation: Clinical consistency and accuracy became a focus through Clinically Coherent Reward, clinical-metric evaluations, and memory-driven Transformer models.The paper identifies Liu et al. (2019) as especially related but notes dependence on a rule-based chest-X-ray information extraction system.
  • Factual consistency: NLI, QA, and related approaches were used to assess or control consistency in summarization, dialogue, headline generation, and data-to-text tasks.The paper notes that some prior approaches showed negative results or were not directly applicable to radiology reports.
  • Transformer architectures: Image captioning research progressed from CNN–RNN systems toward Transformer architectures with attention over semantic and visual information.The authors use Meshed-Memory Transformer as their base architecture after finding it more effective for radiology report generation than RNN- and traditional Transformer-based models.

3 Methods

The system combines a multi-image Transformer report generator with entity-based factual rewards, an NLI model adapted to radiology, and reinforcement learning to optimize factual and language objectives.

  • Image-to-Text Report Generation: The base M2 Trans architecture processes multiple images and generates a textual report describing their clinical observations.Image regions are extracted by a CNN, encoded with memory-augmented attention, and processed by a meshed decoder.
  • Factual Rewards: The factENT reward measures generated-report coverage of reference radiology entities to encourage factual completeness.Entities are recognized in generated and reference reports, and their precision and recall are combined using a harmonic mean.
  • Factual Rewards: The factENTNLI reward extends entity matching with natural language inference to encourage inferential consistency and control disease overestimation.Entailment and contradiction labels determine whether matched entities contribute to the reward, while BERTScore supplies text similarity.
  • Joint Optimization: The report generator optimizes factual rewards with reinforcement learning while combining language-model and NLG losses through scaling factors.The RL loss uses sampled and greedily decoded texts; the joint objective includes NLL, an NLG-metric RL loss, and a factual-reward RL loss.
  • Radiology NLI: A weakly supervised approach constructs radiology NLI training pairs by filtering report sentence pairs with semantic-similarity and entity-based rules.The resulting rules cover entailment, neutral, and contradiction relations, and the NLI model is trained using radiology data together with MedNLI data.
  • Radiology NLI: Adding the radiology NLI training set increased NLI accuracy on radiology data by +24.5%.The reported accuracies are averages over five runs.

4 Experiments

The experiments train and evaluate M2 Trans with multiple joint losses on MIMIC-CXR and Open-i, using general NLG, clinical, and factual-reward metrics. Results compare baselines with objectives incorporating CIDEr-D, BERTScore, factENT, and factENTNLI.

  • 4.1 Data: 152173 MIMIC-CXR reports were used for training, 1196 for validation, and separate MIMIC-CXR and Open-i sets for testing.The test sets contained 2347 MIMIC-CXR reports and 3335 Open-i reports.
  • 4.2 Evaluation: Clinical evaluation uses CheXbert to compare generated and reference presence statuses for atelectasis, cardiomegaly, consolidation, edema, and pleural effusion.Micro-average accuracy, precision, recall, and F1 are calculated over these five observations.
  • 4.3 Model Variations: M2 Trans uses DenseNet-121 as its image encoder and is evaluated with NLL, CIDEr-D, BERTScore, factENT, and factENTNLI objectives.The joint-loss variants include NLL, NLL+CDr, NLL+BS, NLL+BS+fcE, and NLL+BS+fcEN.
  • 4.3 Model Variations: TieNet, CNN-RNN2, and R2Gen provide comparison baselines alongside M2 Trans.The baselines use CNN-RNN or memory-driven Transformer architectures with their respective training objectives.

5 Results and Discussions

The proposed rewards and BERTScore improved clinical evaluation of generated radiology reports, while qualitative and human analyses showed gains alongside remaining factual limitations.

  • Reward Optimization: +3.6 factENT and +4.9 factENTNLI were observed on MIMIC-CXR with M2 Trans compared with M2 Trans w/ BS.The best result for a metric or reward occurred when that metric or reward was directly included in the optimization objective.
  • Clinical Metrics: +22.1 increase (Δ+63.9%) in F1 score was achieved by M2 Trans using factENT against the best baseline R2Gen.The best recalls and F1 scores were obtained with factENT, whereas factENTNLI produced higher precision and accuracy with lower recall.
  • Baselines: M2 Trans w/ NLL showed strong improvements in clinical metrics against R2Gen, despite both models using Transformer-based architectures and NLL loss.The authors attribute these improvements to possible architecture differences, such as memory matrices in the encoder.
  • Baselines: The difference between NLL and NLL+CDr indicated that NLL and CIDEr were unreliable for factual completeness and consistency.The results without the proposed rewards and BERTScore also revealed the strength of M2 Trans and the inadequacy of NLL loss and CIDEr for these properties.
  • Estimating Clinical Accuracy: factENTNLI showed the strongest correlation with clinical accuracy, although the correlations remained limited and left room for improvement.This result suggests factual rewards may help estimate factual completeness and consistency when a strong clinical information extraction system is unavailable.
  • Qualitative Analysis: Qualitative examples showed BERTScore correctly generated atelectasis and suppressed left pleural effusion, while factENTNLI correctly generated edema and suppressed atelectasis.These examples were presented as evidence of improved factual completeness and consistency from integrating the three metrics.
  • Limitations: The model remained incomplete: it could not capture comparisons with previous studies and rendered uncertain edema as definite mild pulmonary edema.The model was not designed to use patients’ past reports as input.

6 Conclusion

The paper combines entity-based factual rewards, weakly supervised NLI, and semantic equivalence optimization to improve radiology report generation. On two open datasets, the resulting system produced reports that were more factually complete and consistent than baselines.

  • The two rewards use radiology entities and weakly supervised NLI to target factual completeness and consistency.The entity reward captures reference coverage, while the NLI-based reward evaluates inferential consistency.
  • The Transformer-based system directly optimizes these rewards with self-critical reinforcement learning.
  • The system generated reports with substantially higher clinical metric scores and greater factual completeness and consistency than baselines.The conclusion also suggests that combining entities and NLI may extend beyond radiology reports to other data-to-text tasks.
  • The weakly supervised radiology NLI is trained from sentence-pair rules that classify entailment, neutral, or contradiction relations.The rules use semantic similarity, named entities, negation, anatomy modifiers, and observation keywords.
  • The NLI rules include examples based on entity inclusion, semantic similarity, negation, antonyms, and differing observation keywords.

A.2 Validation and Test Datasets of Radiology NLI

The radiology NLI dataset was built from MIMIC-CXR sentence pairs selected for semantic similarity and balanced MedNLI labels, then expert-annotated in both directions. The resulting bidirectional pairs were divided into validation and test sets.

  • 480 sentence pairs were sampled from the MIMIC-CXR validation section for radiology NLI development.
  • The sampled pairs had BERTScore(s1, s2) ≥0.5 and equally distributed MedNLI labels across entailment, neutral, and contradiction.
  • Two experts annotated each pair after swapping premise and hypothesis, with disagreements resolved through discussion with an additional NLP expert.
  • The 960 resulting bidirectional pairs were split equally into 480 validation pairs and 480 test pairs.

A.3 Configuration of Radiology NLI Model

The radiology NLI model starts from pre-trained BERT, adapts it to radiology reports, and trains it for classification with selected optimization settings. Validation accuracy determines the checkpoint evaluated on the test set.

  • The model uses bert-base-uncased and is further fine-tuned on MIMIC-III radiology reports with masked language modeling.The masked language modeling stage lasts 8 epochs before classification optimization.
  • The model is optimized on training data with classification negative log likelihood using Adam.
  • The configuration uses β1 = 0.9, β2 = 0.999, batch size 16, gradient clipping norm 5.0, and learning rate lr = 1e−5.
  • Training runs for at most 20 epochs, with validation accuracy selecting the checkpoint used for test evaluation.Training takes approximately 2 hours on a single Nvidia Titan XP.

B.1 M2 Trans

The M2 Trans configuration uses a DenseNet-121 image encoder and GloVe text embeddings within a Transformer-based architecture. The model dimensionality is 512, with 8 attention heads and 40 memory vectors.

  • DenseNet-121 is used as the CNN image feature extractor and pre-trained on CheXpert with 14-class classification.
  • GloVe pre-trains text embeddings on the training set with embedding size 512.
  • The model dimensionality is 512, with 8 attention heads and 40 memory vectors.
  • The optimization takes approximately 10 days on a Titan XP.

B.2 TieNet

TieNet uses ResNet-50 image features, GloVe text embeddings, and a jointly optimized NLL and multi-label classification objective.

  • TieNet uses ResNet-50 with default ImageNet-pretrained weights as its CNN image feature extractor.
  • TieNet uses GloVe to pre-train text embeddings.
  • The model uses an LSTM dimension of 256 and five global attentions.
  • TieNet combines NLL loss with multi-label classification loss using α = 0.85.
  • Training uses a linear scheduler starting at 1e−4, halving every eight epochs, with batch size 32.

B.3 CNN-RNN2

CNN-RNN2 uses DenseNet-121 image features and trains with a joint CIDEr and Clinically Coherent Reward objective after NLL pretraining.

  • CNN-RNN2 uses DenseNet-121 with default ImageNet-pretrained weights as its CNN image feature extractor.
  • The model uses GloVe to pre-train text embeddings and an LSTM dimension of 256.
  • CNN-RNN2 modifies CheXpert into CheXbert to improve the information extraction system’s training speed.
  • The joint loss combines CIDEr with Clinically Coherent Reward using λ = 10.0.
  • The model is first trained with NLL loss using a linear scheduler starting at 1e−4 and halving every eight epochs.

C Detailed Result of Clinical Metrics

Clinical metrics compare R2Gen with M2 Trans variants using BERTScore alone or augmented with factual entity rewards across radiology observations and datasets. Factual rewards usually produce the best F1 scores, while infrequent observations show less stable extraction performance.

  • Table 5 compares clinical metrics for R2Gen and M2 Trans with BERTScore, factENT, or factENTNLI.The compared systems are R2Gen, M2 Trans w/ BS, M2 Trans w/ BS+fcE, and M2 Trans w/ BS+fcEN.
  • In most cases, the best F1 scores occur when factENT or factENTNLI is included in the joint loss.
  • Consolidation is an exception, with the best precision, recall, and F1 varying across joint losses.The authors attribute this variation to consolidation’s infrequent appearance in MIMIC-CXR and Open-i.
  • CheXpert-based evaluations in Table 6 follow trends similar to Table 5 because CheXbert is at least as accurate for most observations.
  • The remaining nine CheXpert observations have relatively weaker and unstable extraction performance because many are infrequent.
Loading 2010.10042v2…