PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation

Kang Liu, Zhuoqi Ma, Xiaolu Kang, Yunan Li, Kun Xie, Zhicheng Jiao, Qiguang Miao

arXiv:2502.20056v1cs.CVcs.AI

TL;DR

Existing chest X-ray report-generation methods often use single or fixed-view images, limiting representation of current disease conditions and disease progression. MLRG combines multi-view longitudinal contrastive learning with tokenized absence encoding for prior knowledge, and reports state-of-the-art performance across three datasets.

  • Problem

    Existing methods focus mainly on single or fixed-view images, limiting current multi-view disease modeling and overlooking disease progression.

  • Method

    MLRG integrates current multi-view and longitudinal information through report-supervised contrastive pre-training and uses tokenized absence encoding for missing patient-specific prior knowledge.

  • Results

    MLRG outperforms existing state-of-the-art methods across MIMIC-CXR, MIMIC-ABN, and Two-view CXR in generating coherent and clinically accurate reports.

  • Takeaways & Limitations

    Multi-view longitudinal information, report-based spatiotemporal supervision, and available prior knowledge together support MLRG’s report-generation performance across three datasets.

Abstract

from arXiv · show

Automated radiology report generation offers an effective solution to alleviate radiologists' workload. However, most existing methods focus primarily on single or fixed-view images to model current disease conditions, which limits diagnostic accuracy and overlooks disease progression. Although some approaches utilize longitudinal data to track disease progression, they still rely on single images to analyze current visits. To address these issues, we propose enhanced contrastive learning with Multi-view Longitudinal data to facilitate chest X-ray Report Generation, named MLRG. Specifically, we introduce a multi-view longitudinal contrastive learning method that integrates spatial information from current multi-view images and temporal information from longitudinal data. This method also utilizes the inherent spatiotemporal information of radiology reports to supervise the pre-training of visual and textual representations. Subsequently, we present a tokenized absence encoding technique to flexibly handle missing patient-specific prior knowledge, allowing the model to produce more accurate radiology reports based on available prior knowledge. Extensive experiments on MIMIC-CXR, MIMIC-ABN, and Two-view CXR datasets demonstrate that our MLRG outperforms recent state-of-the-art methods, achieving a 2.3% BLEU-4 improvement on MIMIC-CXR, a 5.5% F1 score improvement on MIMIC-ABN, and a 2.7% F1 RadGraph improvement on Two-view CXR.

1. Introduction

Radiology report generation aims to reduce the expertise-dependent and time-consuming burden of manually documenting chest X-ray interpretations. MLRG addresses limitations of single-image methods by combining current multi-view images, longitudinal data, and patient-specific prior knowledge.

  • Chest X-rays support diagnosis of pneumonia, fracture, pneumothorax, pleural effusion, and cardiomegaly, but manual reporting is expertise-dependent and time-consuming.
  • Clinical reporting integrates current multi-view images, medical histories, and patient-specific prior knowledge to track disease progression and support diagnosis.
  • Most existing RRG methods rely on single images and struggle to distinguish PA, AP, lateral, and left anterior oblique views.
  • MLRG uses multi-view longitudinal contrastive learning supervised by spatiotemporal information in radiology reports to pre-train visual and textual representations.
  • MLRG introduces tokenized absence encoding so the text generator can use available patient-specific prior knowledge when some information is missing.

2. Related Work

Related work spans radiology report generation, medical vision-language pre-training, and multi-view medical image analysis. MLRG builds on these directions by combining longitudinal progression modeling with current multi-view representation learning and report supervision.

  • Radiology report generation: Radiology report generation combines vision encoders with text generators to produce detailed content using specialized medical terminology.
  • Medical vision-language models: Medical vision-language models learn generalized medical visual representations by maximizing agreement between images and reports.
  • Medical vision-language models: Prior work includes BioViL-T for disease progression modeling from longitudinal data, while MLRG supervises visual and textual pre-training with reports’ spatiotemporal information.
  • MLRG overview: MLRG’s pipeline uses RAD-DINO, CXR-BERT, and DistilGPT2 to learn multi-view longitudinal features before generating reports with patient-specific prior knowledge.
  • Enhancing medical image analysis via multi-view data: Multi-view learning derives shared and complementary information from multiple views of the same subject, but some methods depend on additional annotated disease labels.

3. Method

MLRG uses a two-stage pipeline combining multi-view longitudinal contrastive learning with tokenized encoding for missing patient-specific prior knowledge. The method aligns visual and textual spatiotemporal representations before report generation.

  • Stage 1: Stage 1 jointly learns visual and textual representations using multi-view longitudinal contrastive learning supervised by radiology reports’ spatiotemporal information.The approach combines current multi-view images with longitudinal data and report-derived supervision.
  • Multi-view contrastive learning: Multi-positive contrastive learning treats images from the same visit as positives, accommodating visits with varying numbers of current views.Visits with only one image are excluded from this loss but remain available for subsequent cross-modal alignment.
  • Multi-view longitudinal fusion: The multi-view longitudinal fusion network integrates current views with the most recent previous image using view and temporal positional embeddings.Processing one sample at a time allows adaptation to variable view counts and absent previous images.
  • Cross-modal alignment: Cross-modal alignment uses spatiotemporal visual features and textual features, treating same-visit pairs and different visits with identical reports as positives.This design incorporates report content that may compare current findings with medical history.
  • Stage 2: Stage 2 represents missing “INDICATION” and “previous report” inputs with special tokens while cleaning and serializing available prior information.The resulting patient-specific context is combined for the text generator, which is initialized from Stage 1 components and trained with cross-entropy loss.

4. Experiments

Experiments evaluate MLRG across multiple datasets, metrics, ablations, and qualitative analyses. Results show benefits from multi-view images, longitudinal data, patient-specific prior knowledge, and report-supervised learning, with stronger performance than existing methods.

  • Main Results: MLRG achieves state-of-the-art performance across most metrics, particularly B-4, RG, and F1, while slightly trailing B-LLM in Recall.The comparison includes 14 state-of-the-art methods.
  • Ablation Study: MLRG significantly exceeds the variant without Stage 1, indicating that multi-view longitudinal contrastive learning improves generated-report accuracy and coherence.LG and LMP C also positively affect performance in the ablation study.
  • Ablation Study: MLRG significantly outperforms the variant without patient-specific prior knowledge, while “INDICATION” and previous reports each contribute positively.The results emphasize the importance of incorporating available patient-specific prior knowledge.
  • Ablation Study: Current multi-view images outperform single-image inputs for report generation, improving the modeling of current disease conditions.This comparison isolates the contribution of current multi-view images.
  • Ablation Study: MLRG outperforms the comparison variant using the MLF network without previous images, indicating effective integration of previous images for tracking disease progression.The reported benefit is linked to more clinically accurate reports.
  • Data-Component Analysis: Including current multi-view images, multi-view longitudinal data, and indications significantly improves NLG and RG metrics.The authors associate these gains with richer visual representations and the use of indications by the text generator.
  • Clinical Accuracy: MLRG outperforms SEI on most of 14 CheXpert observations, including challenging Pneumothorax and Fracture observations.The comparison is reported across three datasets.
  • LLM Evaluation: LLM-based evaluation finds that MLRG surpasses R2Gen, CMN, CGPT2, and SEI in both #Matched Findings and GREEN score.GREEN is used to identify clinically significant errors and count matched findings.

5. Conclusion

The conclusion presents MLRG as a two-part approach combining multi-view longitudinal contrastive learning with tokenized absence encoding. Experiments across three datasets show stronger generation of coherent and clinically accurate reports, while future work targets reliability.

  • Contributions: MLRG uses report-supervised multi-view longitudinal contrastive learning to capture view differences, current spatial features, and longitudinal temporal features.The approach guides visual and textual representation pre-training.
  • Contributions: Tokenized absence encoding lets the fusion network adapt to scenarios with or without patient-specific prior knowledge.The text generator uses available prior knowledge when it exists.
  • Conclusion: Experiments on MIMIC-CXR, MIMIC-ABN, and Two-view datasets show that MLRG outperforms existing state-of-the-art methods in generating coherent and clinically accurate reports.The conclusion describes MLRG as a strong contender for chest X-ray report generation.
  • Future Work: Future work will use saliency maps for region-based features and uncertainty prediction to improve model reliability.These directions are presented as extensions of the current method.

A.1. Implementation Details

Training uses separate two-stage settings for MIMIC-CXR and fine-tuning settings for MIMIC-ABN and Two-view CXR.

  • Training Settings: MIMIC-CXR training uses 50 epochs in each stage, with batch sizes of 32 and 14 for Stages 1 and 2, respectively.Stage 2 uses learning rates of 5e-6 for Stage 1 parameters and 5e-5 for remaining parameters.
  • Training Settings: MIMIC-ABN and Two-view CXR are fine-tuned from the Stage 2 MIMIC-CXR model using a learning rate of 5e-6.The supplied implementation passage also specifies a batch size for this fine-tuning procedure.

A.2. Clinical Accuracy of 14 Observations

MLRG outperforms SEI on most of 14 CheXpert observations across the evaluated datasets, including challenging Pneumothorax and Pleural Other observations.

  • MLRG outperforms SEI on most of 14 CheXpert observations across the MIMIC-CXR, MIMIC-ABN, and Two-view CXR datasets.The model slightly surpasses the baseline on challenging observations including Pneumothorax and Pleural Other.

A.3. Performance of Generating “FINDINGS” and “IMPRESSION” Sections

The study evaluates generating both the “FINDINGS” and “IMPRESSION” sections, using special tokens to distinguish the two sections before combining them into final reports.

  • Radiology reports contain “INDICATION”, “FINDINGS”, and “IMPRESSION” sections, with “FINDINGS” describing observations and comparisons and “IMPRESSION” summarizing conclusions.
  • MLRG generates both sections by prepending “[FINDINGS]” or “[IMPRESSION]” special tokens before the corresponding section content.
  • The generated sections are combined to form final reports, while other generation settings remain identical to those used for “FINDINGS” generation.

A.4. Qualitative Analysis

Qualitative and GREEN-based analyses indicate that MLRG generates clinically accurate findings and impressions, especially temporal comparisons, while retaining weaknesses in lesion location and severity descriptions.

  • Qualitative Analysis: MLRG generates both “FINDINGS” and “IMPRESSION” sections competitively, as well as “FINDINGS” alone, on the MIMIC-CXR test set.
  • GREEN Evaluation: GREEN evaluates six clinical error categories, including false or missing findings, location and severity errors, and incorrect or omitted prior-study comparisons.
  • Evaluation Setup: Tables A5 and A6 report clinical accuracy for 14 CheXpert observations, while Table A7 reports performance for generating “FINDINGS” and “IMPRESSION” sections.
  • GREEN Evaluation: MLRG achieves the highest “#Matched Findings” and GREEN score with the fewest total clinically significant errors among the compared systems.
  • GREEN Evaluation: MLRG performs best at avoiding omissions of comparisons that detail changes from prior studies, indicating effective extraction of temporal features.
  • Limitations: MLRG has higher error counts than baselines for anatomical location and severity assessment, motivating future use of saliency maps and MIMIC-CXR-VQA data.
  • Qualitative Analysis: Qualitative examples show discrepancies including an unsupported aortic valve replacement, omitted cardiomegaly, and no explicit statement that the lungs are clear.
Loading 2502.20056v1…