PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal Prototype Driven Network for Radiology Report Generation

Jun Wang, Abhir Bhalerao, Yulan He

arXiv:2207.04818v1cs.CVcs.CL

TL;DR

Radiology report generation faces complex long-form text and dataset biases, while existing methods often focus on single-modal features. XPRONET learns cross-modal prototypes and uses them to enrich image and text features. It outperforms prior methods by a notable margin on IU-Xray and leads on most MIMIC-CXR metrics.

  • Problem

    Radiology report generation must handle long, semantically complex reports and normal-case-heavy datasets, while many methods focus on single-modal features rather than cross-modal interactions.

    Source: §1 · p. 2

  • Method

    XPRONET learns class-related cross-modal prototypes and uses prototype querying, responding, and an improved multi-label contrastive loss to enrich image and text features.

    Source: §1 · p. 3, §3.2 · p. 5, Cross-modal Prototype Querying · p. 7

  • Results

    XPRONET surpasses R2GenCMN by 4.3% on BLEU-1, 3.1% on BLEU-4, and 4.1% on RG-L on IU-Xray, and achieves the best MIMIC-CXR performance except on BLEU-1.

    Source: Comparisons with Previous Studies · p. 11

  • Takeaways & Limitations

    The authors conclude that class-related prototype querying and responding addresses data bias in radiology report generation.

    Source: §5 · p. 14

Abstract

from arXiv · show

Radiology report generation (RRG) aims to describe automatically a radiology image with human-like language and could potentially support the work of radiologists, reducing the burden of manual reporting. Previous approaches often adopt an encoder-decoder architecture and focus on single-modal feature learning, while few studies explore cross-modal feature interaction. Here we propose a Cross-modal PROtotype driven NETwork (XPRONET) to promote cross-modal pattern learning and exploit it to improve the task of radiology report generation. This is achieved by three well-designed, fully differentiable and complementary modules: a shared cross-modal prototype matrix to record the cross-modal prototypes; a cross-modal prototype network to learn the cross-modal prototypes and embed the cross-modal information into the visual and textual features; and an improved multi-label contrastive loss to enable and enhance multi-label prototype learning. XPRONET obtains substantial improvements on the IU-Xray and MIMIC-CXR benchmarks, where its performance exceeds recent state-of-the-art approaches by a large margin on IU-Xray and comparable performance on MIMIC-CXR.

1 Introduction

Radiology report generation could help reduce the workload of radiologists, but long, semantically complex reports and dataset biases make accurate generation challenging. XPRONET uses cross-modal prototypes to model interactions between image and text, and the paper reports improvements over state-of-the-art methods on two benchmarks.

1 Introduction

  • Radiology report generation could rapidly produce reports and assist radiologists, potentially reducing workload and saving medical resources.Manual reporting often takes more than five minutes and can be affected by fatigue and distraction.Source: §1 · p. 1
  • Longer reports, sophisticated linguistic and semantic patterns, and normal-case-heavy datasets challenge accurate report generation.Abnormal regions may occupy only small image areas, while most statements can describe normal findings.Source: §1 · p. 2
  • XPRONET learns cross-modal prototypes as intermediate representations to enrich single-modal features and explicitly model cross-modal information flow.The work introduces a shared prototype matrix and a prototype network for embedding cross-modal information into visual and textual features.Source: §1 · pp. 2–3
  • A class-related prototype querying and responding module makes single-modal features interact with cross-modal prototypes to address imbalance.The proposed improved multi-label contrastive loss accommodates label differences through an adaptive controller term.Source: §1 · p. 3
  • The paper reports that XPRONET outperforms several state-of-the-art methods on IU-Xray and MIMIC-CXR.The authors also report ablation studies of individual components.Source: §1 · p. 3

2 Related Work

Prior report-generation work adapted encoder-decoder methods from image captioning, while XPRONET focuses on cross-modal pattern interactions. Its closest comparison is R2GenCMN, which also uses memory for cross-modal patterns but differs in prototype learning, label-conditioned querying, and feature distillation.

Image Captioning

  • Image-captioning methods often use encoder-decoder architectures and Transformers, but short-description methods are less suited to long reports.Some long-text methods may miss specific medical observations and abnormal image regions.Source: Image Captioning · p. 3

Radiology Report Generation

  • Radiology report-generation methods commonly use encoder-decoder architectures, often emphasizing single-modal features over cross-modal pattern interactions.Prior approaches include hierarchical report generation, topic conditioning, knowledge graphs, and reinforcement learning,,,,.Source: Image Captioning · p. 4
  • Compared with R2GenCMN, XPRONET uses a shared class-related prototype matrix, an improved multi-label contrastive loss, and a different cross-modal information distillation approach.R2GenCMN randomly initializes its memory, uses cross-entropy loss, queries all prototypes, and averages features.Source: Image Captioning · p. 4

3 Methods

XPRONET explicitly models cross-modal interactions for radiology report generation using class-related prototypes, then generates reports with a Transformer. Its method combines visual feature extraction, prototype learning and interaction, and an improved multi-label contrastive loss.

3.1 Image Feature Extractor

  • ResNet-101 extracts convolutional image features, which are linearized into region tokens for subsequent processing.The features come from the last convolution layer, before final average pooling.Source: §3.1 · pp. 4–5

3.2 Cross-modal Prototype Network

  • CheXbert generates pseudo class labels, while pretrained ResNet-101 and BERT image-text features are concatenated and class-wise K-Means cluster averages initialize the shared prototype matrix.The matrix stores class-related cross-modal patterns for subsequent querying and responding.Source: Pseudo Label Generation · p. 5, Prototype Matrix Initialization · p. 6
  • For each image, the querying module selects the top γ similar cross-modal prototypes sharing its label to interact with its single-modal representation.The associated pseudo labels determine which category-specific prototype sets are queried.Source: Cross-modal Prototype Querying · pp. 7–8
  • The responding module forms weighted sums of transformed prototypes, and a learned feature-interaction layer fuses these responses with visual and textual features.The learned fusion is proposed to address potentially noisy responses or non-discriminative single-modal features.Source: Cross-modal Prototype Responding · p. 8, Feature Interaction Module · pp. 8–9

3.3 Reports Generation with Transformer

  • A Transformer encodes responded visual features and decodes report words using the current fused textual representations.The process repeats to obtain the complete report.Source: §3.3 · p. 9

3.4 Improved Multi-Label Contrastive Loss

  • The improved contrastive loss treats samples sharing at least one nonzero label as positives and samples with no common labels as negatives.Its label-difference term allows some dissimilarity between positive pairs in the multi-label setting.Source: §3.4 · pp. 9–10
  • The contrastive loss is applied to visual and textual responses, then combined with report cross-entropy for joint optimization.Only negative pairs whose similarity exceeds margin α contribute to the visual response loss.Source: Objective Function · p. 10, §3.4 · p. 10

4 Experiments

Experiments evaluate XPRONET on IU-Xray and MIMIC-CXR using common report-generation metrics. XPRONET achieves larger gains over prior methods on IU-Xray, while leading on all MIMIC-CXR metrics except BLEU-1.

Experimental Setup

  • Evaluation uses BLEU-1–4, ROUGE-L, METEOR, and CIDEr across IU-Xray and MIMIC-CXR; IU-Xray uses a 70/10/20 split and MIMIC-CXR its official split.IU-Xray contains 7,470 images and 3,955 reports; MIMIC-CXR contains 473,057 images and 206,563 reports.Source: §4 · p. 10, Datasets · p. 11

Comparisons with Previous Studies

Ablation Analysis

  • Removing prototype initialization lowers BLEU-4 from 0.199 to 0.160 on IU-Xray and from 0.105 to 0.100 on MIMIC-CXR; removing any of the three components significantly lowers performance.The passage also reports lower BLEU-2 and ROUGE-L when the improved multi-label contrastive loss is removed.Source: Ablation Analysis · p. 12, Table 2 · p. 12
  • Without the full cross-modal prototype network, BLEU-1 falls from 0.525 to 0.467 on IU-Xray and from 0.344 to 0.321 on MIMIC-CXR; example reports show other models may omit abnormalities.The authors attribute the stronger report generation to learned cross-modal prototypes and class-related querying and responding.Source: Ablation Analysis · p. 13, Figure 4 · p. 13

Sensitivity Analysis

  • On IU-Xray, BLEU-4 peaks at 0.199 when γ=15 and falls to 0.171 at γ=17; on MIMIC-CXR, performance is best at θ=1.750.The authors associate excessive or insufficient responding prototypes with performance drops and note the need to balance prototype learning with tolerance of positive-pair dissimilarity.Source: Ablation Analysis · pp. 13–14, Figure 5 · p. 14

5 Conclusions

XPRONET explicitly models cross-modal pattern learning for medical report generation, and experiments on two public benchmarks support its superiority. The conclusion also identifies prototype count and initialization clustering as potential improvement directions.

5 Conclusions

  • Figure 6 examines how varying the tolerance-rate control term θ affects BLEU-4 score.Source: Figure 6 · p. 14
  • XPRONET explicitly models cross-modal pattern learning through a cross-modal prototype network for medical report generation.Its class-related querying and responding module distills cross-modal information into single-modal features and addresses data bias.Source: §5 · p. 14
  • Experiments on two publicly available benchmarks verify XPRONET's superiority, while ablations demonstrate the effectiveness of its component parts.Source: §5 · p. 14
  • The authors suggest that more cross-modal prototypes for larger datasets and improved clustering during prototype-matrix initialization could further improve XPRONET.Source: §5 · p. 14

1 Implementation Details

Implementation uses ResNet-101 for visual features and a Transformer encoder-decoder, with prototype settings and optimization choices specified for the two datasets.

1 Implementation Details

  • ResNet-101 pretrained on ImageNet extracts visual features, with both IU-Xray images used and one MIMIC-CXR image used per patient.Training images are resized and randomly cropped; testing images are resized directly to 224 × 224.Source: §1 · p. 19
  • The Transformer backbone has three layers, eight attention heads, and 512-dimensional hidden states, while the prototype matrix contains 280 vectors.The model uses 14 pseudo-label categories with 20 clusters each and selects the top 15 prototypes to respond to single-modal representations.Source: §1 · p. 19
  • XPRONET is optimized with Adam under cross-entropy and improved multi-label contrastive losses, using dataset-specific learning rates.Learning rates decay by 0.8 per epoch; batch size is 16, beam size is three, and hyperparameters are selected on validation sets.Source: §1 · p. 19

2 More Example Visualizations

Figure 1 presents prediction visualizations from XPRONET and identifies GT as Ground Truth.

2 More Example Visualizations

Loading 2207.04818v1…