PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing

Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, Ozan Oktay

arXiv:2301.04558v2cs.CVcs.CL

TL;DR

Biomedical VLP commonly aligns reports with single images even when reports refer to prior studies, leaving temporal semantics underused. BioViL-T incorporates prior images and reports through a CNN–Transformer multi-image encoder and joint text supervision, achieving state-of-the-art results across temporal and static tasks while introducing MS-CXR-T for temporal evaluation.

  • Problem

    Biomedical reports often reference prior images, but existing VLP methods mainly align single images and reports, leaving temporal content underused.

  • Method

    BioViL-T jointly trains a hybrid CNN–Transformer multi-image encoder and text model, incorporating prior images and reports when available during training and fine-tuning.

  • Results

    BioViL-T achieves state-of-the-art performance across report generation, temporal image classification, phrase grounding, and additional static tasks.

  • Takeaways & Limitations

    Temporal image and text content provides useful learning signals for richer biomedical vision-language representations and broader downstream evaluation.

  • Takeaways & Limitations

    Further evaluation on diverse datasets is required to determine which tasks benefit from temporal modeling and this methodology.

Abstract

from arXiv · show

Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-supervision through existing temporal content in the data. In this work, we explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model. It is designed to be versatile to arising challenges such as pose variations and missing input images across time. The resulting model excels on downstream tasks both in single- and multi-image setups, achieving state-of-the-art performance on (I) progression classification, (II) phrase grounding, and (III) report generation, whilst offering consistent improvements on disease classification and sentence-similarity tasks. We release a novel multi-modal temporal benchmark dataset, MS-CXR-T, to quantify the quality of vision-language representations in terms of temporal semantics. Our experimental results show the advantages of incorporating prior images and reports to make most use of the data.

1. Introduction

Biomedical VLP reports often contain temporal references to prior images, but existing methods generally align each report with only one image. BioViL-T explicitly uses prior images and temporal report content to improve representation learning and downstream performance.

  • Motivation: Clinical reports routinely compare current findings with prior imaging, creating ambiguity when single-image methods align reports without prior images.Temporal statements such as “Pneumonia is improving” may describe any image containing pneumonia.
  • Motivation: Existing biomedical VLP methods often remove temporal report content, despite its potential to provide additional self-supervision without new data.The approach instead exploits temporal structure already present in paired image–report datasets.
  • Contributions: BioViL-T introduces temporal VLP pre-training that makes prior images available and uses a multi-image encoder robust to missing images and spatial misalignment.The framework explicitly accounts for temporal information rather than ignoring or removing it.
  • Results: BioViL-T achieves state-of-the-art performance on temporal image classification, report generation, and phrase grounding, while also improving non-temporal tasks.Reported non-temporal gains include pneumonia detection and phrase grounding, alongside broader applicability across downstream tasks.
  • Contributions: MS-CXR-T is a new multimodal benchmark for evaluating chest X-ray vision-language representations through temporal semantics.The benchmark is curated by an expert radiologist and targets temporal content in image and text data.

2. Related work

Related biomedical VLP methods adapt general-domain contrastive, masking, and multimodal objectives, while longitudinal medical-image modeling has largely remained unimodal. BioViL-T addresses temporal semantics without relying on standard registration or dense-video assumptions.

  • Biomedical VLP: Biomedical VLP has adapted CLIP-style contrastive learning, multimodal masking, and domain-specific data strategies from general-domain vision-language research.These approaches use paired medical images and reports for self-supervised representation learning.
  • Longitudinal modelling: Prior longitudinal medical-image studies use previous images for supervised analysis, whereas temporal structure has not been directly used for self-supervision.Some related methods use patient metadata to select contrastive examples in unimodal learning.
  • Longitudinal modelling: Existing temporal image models rely on late global fusion or explicit spatial correspondence, including fixed grids, bounding boxes, and registration.For chest X-rays, registration is difficult because 2D projections represent 3D anatomy and retain residual misalignment.
  • Temporal self-supervision: BioViL-T uses visual-token self-attention across time to handle spatial misalignment without explicit image registration.Its temporal setting differs from densely sampled video because medical images are sparse and sporadic and still require text supervision.

3. BioViL-T training framework

BioViL-T jointly trains multi-image and text encoders using temporal image features, contrastive alignment, and image-guided language modeling. Its design supports missing prior images, pose variation, and downstream temporal conditioning.

  • Image encoder: BioViL-T extracts spatio-temporal features with a hybrid CNN–Transformer multi-image encoder and adapts the resulting models to uni- or multimodal tasks.The CNN provides image-token features, while the transformer models interactions across images over time.
  • Image encoder: The multi-image encoder incorporates prior images when available and handles spatial misalignment through cross-time self-attention rather than image registration.It is designed for sequences containing either single images or current-plus-prior image examples.
  • Image encoder: The encoder decomposes representations into current-image features and temporal progression features, then concatenates them into the final image representation.The progression representation aggregates patch-level information anchored on the current image.
  • Text supervision: Text is encoded with contextualized token representations, while image and text features are projected into a shared latent space for global and local InfoNCE objectives.Local contrastive learning is reported as important for phrase grounding and temporal image classification.
  • Text supervision: Cross-attention supplies image information for masked language modeling, helping disambiguate masked findings and attributes that text alone may not predict.Training uses image-guided MLM as an auxiliary source of cross-modal supervision.
  • Downstream adaptation: For report generation, BioViL-T conditions current-report decoding on prior reports alongside image information.A separation token distinguishes prior and current report tokens in the multimodal encoder–decoder input.

4. Datasets & experiments

BioViL-T is evaluated across temporal and static biomedical vision-language tasks using the MS-CXR-T benchmark and established datasets. Results show that temporal pre-training improves downstream performance, including report generation, temporal classification, phrase grounding, disease classification, and sentence-level temporal sensitivity.

  • Datasets: MS-CXR-T evaluates temporal image classification and sentence similarity, with progression labels spanning Improving, Stable, and Worsening.The benchmark contains 1,326 multi-image and ground-truth label pairs across five findings, plus 361 sentence pairs.
  • Results: BioViL-T achieves state-of-the-art performance on report generation and temporal image classification, while using prior images and reports improves report-generation metrics.Temporal context particularly improves the TEM metric, which evaluates accurately describing change while avoiding hallucinations.
  • Results: Up to 20 percentage points: BioViL-T pre-training improves previously reported temporal image-classification results and outperforms the non-temporal BioViL baseline.Registration-based baseline methods underperform the proposed approach; errors tend to occur when radiologists disagree.
  • Results: BioViL-T also establishes a new state of the art on zero- and few-shot single-image pneumonia classification without restricting gains to temporal tasks.Including prior images does not impair phrase-grounding performance, while feature decomposition preserves localized current-image information.
  • Results: BioViL-T pre-training increases text-embedding sensitivity to temporal content while better capturing static content on RadNLI.The image encoder requires static/progression feature decomposition for single-image tasks and temporal positional encodings to disambiguate scan order.
  • Ablations: Ablations identify local contrastive learning and image-guided MLM as important pre-training components, while prior reports provide a clearer signal than prior images for report generation.Prior images remain useful because they contain granular details that reports may omit; progression terms rely heavily on prior images during image-guided MLM.

5. Conclusion

BioViL-T aligns text with multiple images by decomposing static and temporal features, yielding richer representations and improved performance across downstream tasks. Its attention analyses support temporal grounding and robustness to pose variation, while broader evaluation remains necessary.

  • Conclusion: Prior-image information improves prediction of progression tokens, including “persist,” “improving,” “remains,” “unchanged,” and “residual.”Figure 4 measures the increase in masked-language-modeling loss when the prior image is discarded.
  • Conclusion: BioViL-T decomposes static–temporal features and augments current-image representations with prior-image information to ground temporal references in text.The framework explicitly leverages temporal content rather than discarding it.
  • Conclusion: BioViL-T achieves state-of-the-art results on report generation, temporal image classification, few/zero-shot pneumonia detection, and phrase grounding.The model performs strongly on both static and temporal tasks.
  • Conclusion: Further evaluation on diverse datasets is required to characterize which tasks benefit from temporal modeling and from the proposed methodology.This is an author-stated scope boundary for the conclusions.
  • Conclusion: Attention rollout maps agree with radiologist-annotated pathology regions across current and prior images, including cases with vertical shifts.The reference patch attends to corresponding regions in both images.
  • Conclusion: Under rotations of the prior image, BioViL-T continues attending to the corresponding anatomical region, demonstrating that image registration is not needed.The comparison uses rollout maps computed from the same reference patch in the current image.

A.4. Data curation of imaging datasets

BioViL-T can rank and identify problematic imaging instances in MIMIC-CXR, including mislabeled views, noise, incorrect anatomy, incomplete fields of view, and unsuitable post-processing. Qualitative comparisons also show more accurate temporal reporting than a current-image-only model.

  • A.4. Data curation of imaging datasets: BioViL-T selects representative images affected by diverse acquisition or processing issues through pairwise ranking in MIMIC-CXR.The analysis uses pairwise ranking with δ = 0.2.
  • A.4. Data curation of imaging datasets: The dataset contains lateral acquisitions labeled as frontal, noisy or non-human samples, incorrect anatomy, incomplete fields of view, and contrast-enhanced scans.These examples illustrate the range of imaging-quality and labeling problems detected.
  • A.4. Data curation of imaging datasets: BioViL-T generates more accurate reports with more precise progression details than BioViL, which uses only the current image.The comparison is made against radiologist reports and includes examples involving pleural-effusion improvement and no relevant change.
  • A.4. Data curation of imaging datasets: Rollout maps compare attention from a reference patch across current and prior images, including pathology examples with spatial transformations.The figures examine pleural effusion and consolidation, with rotations applied to prior images.

A.5. Phrase-grounding on external data

The external-data evaluation uses a small expert-labeled Open-Indiana CXR subset for phrase grounding, under distributional differences from MS-CXR. Results are reported as micro-averages because some classes contain few samples.

  • A.5. Phrase-grounding on external data: The Open-Indiana CXR phrase-grounding evaluation uses 137 expert-labeled bounding-box–caption pairs.The abnormalities match those evaluated in the MS-CXR benchmark.
  • A.5. Phrase-grounding on external data: The external dataset differs from MIMIC-CXR in text token distribution, demographics, and disease prevalence.The experiment uses the same methods and setup as the corresponding MS-CXR evaluation.
  • A.5. Phrase-grounding on external data: Phrase-grounding results are reported as micro-averages because some classes contain limited numbers of samples.The table summarizes multi-modal results for image-text pairs.

B. Temporal aspects of the MIMIC-CXR v.2 dataset

MIMIC-CXR v2 contains substantial longitudinal structure: many subjects have multiple studies, and a large fraction of reports explicitly describe progression. These properties support evaluating temporal vision-language modeling.

  • B. Temporal aspects of the MIMIC-CXR v.2 dataset: Radiologists assess pathology progression by comparing images from different studies, with each study accompanied by a radiologist-authored report.Images may also be referred to as scans or series.
  • B. Temporal aspects of the MIMIC-CXR v.2 dataset: 67 % of MIMIC-CXR subjects have at least two associated studies, and therefore at least two images acquired at different disease stages.A study is one or more images taken during a single medical-facility visit.
  • B. Temporal aspects of the MIMIC-CXR v.2 dataset: Around 40 % of MIMIC-CXR reports have a progression label for at least one finding defined by Chest ImaGenome.These labels are extracted from reports when radiologists explicitly describe changes.

C. MS-CXR-T benchmark

MS-CXR-T provides temporal benchmarks for chest X-ray image progression and sentence-level disease-progression semantics. Its construction combines curated image pairs, progression labels, and paraphrase or contradiction sentence pairs.

  • Temporal image classification: The temporal image classification task covers Consolidation, Edema, Pleural Effusion, Pneumonia, and Pneumothorax across Improving, Stable, and Worsening classes.Progression labels are automatically derived from radiology reports through Chest ImaGenome resources.
  • Temporal image classification: Report-based progression labels are matched to specific image pairs, with radiologist selection of the best frontal scan when a study contains multiple frontal scans.The pipeline includes a second curation step to create the image dataset and a quality-control step for diagnostic image quality.
  • Temporal image classification: The image-classification distribution skews toward Stable and Worsening classes rather than Improving.The passage attributes this pattern to patients being more likely to receive chest X-rays when stable or deteriorating.
  • Temporal sentence similarity: The sentence-similarity benchmark constructs progression-related paraphrase and contradiction pairs using RadGraph matching and temporal-keyword swaps.Sentences are filtered for pathology mentions and temporal keywords before pair construction.
  • Temporal report evaluation: Temporal entity matching evaluates generated reports by extracting radiology-reviewed temporal entities and combining global precision and recall with their harmonic mean.The metric uses entities tagged as observations or observation modifiers and a curated list of progression-indicating temporal entities.

E.2. Training infrastructure

The implementation supports mixed single- and multi-image training through distributed sampling and a shared CNN pass. Downstream procedures adapt BioViL-T to report generation, temporal classification, retrieval, and entity-based evaluation.

  • Training infrastructure: A custom sampler separates single-image and multi-image batches so distributed training can process each batch efficiently.The approach concatenates current and prior images along the batch dimension for one CNN pass.
  • Training infrastructure: The custom sampler has negligible effects on training metrics relative to fully random sampling.With eight GPUs and cross-GPU negatives, each update still averages a representative mixture of single-image and multi-image samples.
  • Downstream adaptation: Report generation fine-tuning uses autoregressive cross-entropy initialized from pretrained image and text encoders.A projection layer maps image patch embeddings to the text encoder’s hidden dimension, with cross-attention additionally trained.
  • Downstream adaptation: BioViL-T pre-training consistently improves report-generation decoding, with further gains when prior images and reports are used.The reported evaluation uses named entity metrics and compares nearest-neighbour and autoregressive approaches.
  • Downstream adaptation: Temporal image classification predicts progression for five pathologies using supervised or few-shot heads attached to the BioViL-T image encoder.The few-shot setting freezes the encoder and trains a single-layer linear head initialized from text prompts.
  • Downstream adaptation: Zero-shot temporal classification prompts the autoregressive decoder with “[FINDING] is” and maps predicted tokens to Improving, Stable, or Worsening.Posterior values are normalized across the three classes, and the highest-scoring class is selected.

F.5. Further analysis of image-guided MLM

The image-guided MLM analysis estimates how prior images affect token prediction in report sentences. It evaluates token-level changes across test reports while controlling for cross-sentence information leakage and low-count variance.

  • Analysis procedure: The estimated prior-image effect is averaged over reports containing each token, using N_m as the number of such sentences.The estimate can have high variance for small N_m, so Figure 4 filters to tokens with N_m ≥ 10.
  • Analysis procedure: Each report sentence is evaluated independently because different sentences can refer to the same image finding.This sentence-level treatment prevents information leakage from other sentences when masking one token at a time.
  • Analysis procedure: The analysis computes image-guided MLM loss for predicting a masked token with and without the prior image.A token’s contribution is zero when it does not appear in the sentence under evaluation.
  • Analysis procedure: The filtered validation set contains 931 tokens manually annotated by a board-certified radiologist into radiology-specific semantic categories.The categories are shown in Figure 4 and described in Table F.4.

F.6. Sentence similarity experiment

The sentence-similarity experiment probes text representations without task-specific fine-tuning. It uses cosine similarity between sentence embeddings to distinguish progression-related paraphrases from contradictions.

  • Evaluation protocol: The text models are evaluated zero-shot by testing whether their sentence representations separate paraphrase and contradiction pairs.The experiment uses BERT output token embeddings rather than fine-tuning a traditional NLI classifier.
  • Evaluation protocol: Cosine similarity between sentence representations supplies logits for binary paraphrase-versus-contradiction classification.For RadNLI, entailment and contradiction pairs are retained while neutral pairs are discarded.
  • Evaluation protocol: Performance is reported with ROC-AUC and binary accuracy, using validation-derived thresholds for the latter.Threshold tuning uses ten-fold cross-validation with step size 0.005.
  • Evaluation data: The benchmark’s image-pair examples span different pathologies and progression labels.Figure F.1 presents examples of image pairs from MS-CXR-T.
Loading 2301.04558v2…