PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Attention for Automatic Chest X-ray Report Generation

Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Yuexian Zou, Ping Zhang, Yuexian Zou, Xu Sun

arXiv:2106.06965v5cs.CVcs.CL

TL;DR

Chest X-ray report-generation models may overlook abnormal regions because normal images, regions, and descriptions dominate the data. The paper proposes Contrastive Attention, which compares each input with known normal images to distill abnormal-region information, and reports state-of-the-art results on IU-X-ray and MIMIC-CXR.

  • Problem

    Normal images and regions dominate chest X-ray datasets and reports, potentially preventing learning-based methods from capturing rare but important abnormalities needed for clinical decision-making.

  • Method

    Contrastive Attention compares an input image with known normal images to obtain contrastive information and can be incorporated into existing report-generation models.

  • Results

    The method achieves state-of-the-art results on IU-X-ray and MIMIC-CXR, with equipped existing models outperforming baselines across most metrics and human evaluations.

  • Takeaways & Limitations

    Contrastive Attention helps existing models better capture and depict abnormalities, supporting more accurate descriptions and potentially assisting radiologists’ clinical decision-making.

  • Takeaways & Limitations

    The authors identify future improvements in applying contrastive attention to path features and using feature maps from multiple convolutional layers.

Abstract

from arXiv · show

Recently, chest X-ray report generation, which aims to automatically generate descriptions of given chest X-ray images, has received growing research interests. The key challenge of chest X-ray report generation is to accurately capture and describe the abnormal regions. In most cases, the normal regions dominate the entire chest X-ray image, and the corresponding descriptions of these normal regions dominate the final report. Due to such data bias, learning-based models may fail to attend to abnormal regions. In this work, to effectively capture and describe abnormal regions, we propose the Contrastive Attention (CA) model. Instead of solely focusing on the current input image, the CA model compares the current input image with normal images to distill the contrastive information. The acquired contrastive information can better represent the visual features of abnormal regions. According to the experiments on the public IU-X-ray and MIMIC-CXR datasets, incorporating our CA into several existing models can boost their performance across most metrics. In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis. Specifically, we achieve the state-of-the-art results on the two public datasets.

1 Introduction

Chest X-ray report generation must describe both normal and abnormal regions, but dataset and image-level bias can make models overlook clinically important abnormalities. The proposed Contrastive Attention model compares input images with normal images, improving abnormal-region capture and report-generation performance.

  • Medical reports describe normal and abnormal chest X-ray regions, but accurate interpretation requires specialized radiological expertise.
  • Automatic report-generation systems are needed because they can reduce radiologists’ workload, while existing methods may fail to support clinical decision-making.
  • Normal images and regions dominate the data and reports, which may prevent models from capturing rare but important abnormalities.
  • Contrastive Attention compares input images with known normal images to identify suspicious abnormal regions and can be integrated into existing models.
  • The approach is evaluated on IU-X-ray and MIMIC-CXR using automatic metrics and human evaluations, where equipped baselines outperform baseline systems.
  • State-of-the-art performance is reported on both IU-X-ray and MIMIC-CXR, with clinicians also invited to assess usefulness for clinical practice.

2 Related Works

Prior work applies encoder-decoder and attention-based methods to image captioning and chest X-ray report generation, while contrastive learning provides the closest conceptual basis for comparing similar and dissimilar images.

  • Image captioning combines image understanding with language generation, typically producing short descriptive sentences through encoder-decoder systems.
  • Chest X-ray report-generation research includes encoder-decoder frameworks, medical-concept enrichment, visual attention, and entity-matching rewards for factual accuracy.
  • Contrastive learning learns similar or dissimilar image representations from paired data, providing the closest related mechanism to Contrastive Attention.

3 Approach

The approach equips existing chest X-ray report generators with Contrastive Attention, which compares an input image against normal images to isolate differentiating visual information. Its Aggregate Attention finds similar normal references, while Differentiate Attention removes shared information so downstream models can use contrastive features.

  • Visual Feature Extraction: The implementation uses ResNet-50 features with 49 spatial vectors of dimension 512 and applies average pooling for a global visual feature.A linear transformation reduces the ResNet output dimension from 2,048 to 512.
  • Contrastive Attention: Contrastive Attention compares each input image with normal images to capture differentiating properties for report generation.The resulting features replace the original image features before being fed into existing models.
  • Contrastive Attention: The normality pool contains 1,000 normal training images represented by global visual features.Each normal image contributes a feature vector to the pool used for comparison.
  • Aggregate Attention: Aggregate Attention weights normal images by similarity to the input, then repeats attention n times to find close references across different image parts.This addresses noise from equally weighting mismatched normal images and captures references similar to specific regions.
  • Attention Mechanism: Dot-product attention was selected because preliminary experiments found it performed better than additive and bilinear attention.The comparison is reported as a preliminary implementation choice.
  • Differentiate Attention: Differentiate Attention summarizes common information between the input and selected normal images, then subtracts it from the input to obtain contrastive information.The contrastive feature is subsequently used to update the original image features.

4 Experiments

Experiments evaluate Contrastive Attention on IU-X-ray and MIMIC-CXR using established baselines, automatic metrics, clinical efficacy measures, and human judgments. Across these evaluations, CA improves baseline performance and reaches state-of-the-art results while improving abnormality descriptions.

  • Datasets and settings: Experiments use IU-X-ray and MIMIC-CXR, with official or established dataset splits and MIMIC-CXR providing 377,110 images and 227,835 reports.IU-X-ray contains 3,955 reports and 7,470 images; MIMIC-CXR uses 368,960 training, 2,991 validation, and 5,159 test instances.
  • Baselines and metrics: The evaluation compares CA-equipped systems with image-captioning and chest X-ray report-generation baselines using BLEU, METEOR, and ROUGE-L.The report-generation baselines include HLSTM, HLSTM+att+Dual, Co-Attention, and Multi-Attention.
  • Automatic evaluation: 14% and 17% are the maximum BLEU-4 improvements on MIMIC-CXR and IU-X-ray, respectively, after incorporating Contrastive Attention into baselines.The improvements are reported on test sets, with Setting (g) achieving the greatest improvements.
  • Automatic evaluation: CA-equipped Multi-Attention outperforms existing state-of-the-art models on major metrics across IU-X-ray and MIMIC-CXR.The comparison includes SentSAT + KG and R2Gen, with results reported from their original papers.
  • Clinical efficacy: CA improves clinical efficacy across all reported metrics and exceeds R2Gen by 6%, 9%, and 10% in Precision, Recall, and F1, respectively.These metrics use CheXpert labeling to measure the accuracy of descriptions for clinical abnormalities on MIMIC-CXR.
  • Human evaluation: Human evaluation finds CA-equipped reports stronger in fluency, comprehensiveness, and faithfulness, with maximum comprehensiveness gains of 41 and 61 points on MIMIC-CXR and IU-X-ray.The evaluation compares CA-equipped systems with HLSTM and Multi-Attention baselines.

5 Analysis

The analysis shows that Contrastive Attention improves baseline report generation by extracting contrastive information and using Aggregate Attention to select useful normal images while filtering noise. Qualitative examples further indicate better alignment between detected abnormal regions and generated abnormal descriptions.

  • Effect of Contrastive Attention: Contrastive Attention comprises Differentiate Attention and Aggregate Attention, with both components improving baseline performance across their evaluated metrics.Differentiate Attention extracts contrastive information, while Aggregate Attention identifies useful normal images and filters noisy ones.
  • Effect of Contrastive Attention: Up to 5% gain in BLEU-4 makes HLSTM with Differentiate Attention equally competitive with HLSTM+att+Dual.The result indicates that the contrastive information extracted by Differentiate Attention contains accurate abnormal information useful for report generation.
  • Effect of n: Aggregate Attention performance increases as n grows below 6, because multiple attention weights identify similar normal images from different aspects.The analyzed aspects can include organs or tissues, supporting more accurate and robust contrastive information.
  • Qualitative Analysis: In Figure 3, CA contrasts the input with normal images, highlights abnormal regions, selects closest normal images, and filters noisy images.Red boxes mark abnormal regions; blue and green boxes show selected normal images from different attention weights, while purple boxes show noisy images.
  • Qualitative Analysis: HLSTM tends to repeat normal findings, whereas Multi-Attention can describe abnormalities but sometimes generates incorrect ones.The analysis attributes these behaviors to dataset deviation for HLSTM and imperfect abnormality descriptions for Multi-Attention.

6 Conclusion

The paper concludes that Contrastive Attention helps chest X-ray report generation models capture and describe abnormalities by contrasting input images with normal images. Experiments and human evaluations support improved performance and assistance for radiologists, while future work identifies feature-representation extensions.

  • Conclusion: Contrastive Attention captures abnormal regions by contrasting the input image with normal images in chest X-ray report generation.The approach is designed to be incorporated into existing report generation models.
  • Conclusion: Experiments on two public datasets show that incorporating Contrastive Attention boosts existing models under most metrics.The conclusion also reports state-of-the-art results on both datasets.
  • Conclusion: Clinical efficacy scores and human evaluations indicate that Contrastive Attention helps existing models capture and depict abnormalities.The reported results include the best human preference on the two datasets.
  • Future Work: Future improvements include applying contrastive attention to path features and using feature maps from multiple convolutional layers.The current discussion contrasts these possibilities with using global features and only the last convolutional layer.

Ethical Considerations

The paper evaluates report-generation systems partly through clinical efficacy and human evaluation, focusing on whether abnormalities are omitted or incorrectly added.

  • Ethical Considerations: The evaluation measures omission of abnormalities and descriptions of abnormalities that do not exist.These analyses use clinical efficacy metrics and human evaluation to assess abnormality-related reporting behavior.
Loading 2106.06965v5…