PaperlayerView exact revision on arXiv

Source-linked AI summary

Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing

Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, Ozan Oktay

arXiv:2204.09817v4cs.CVcs.CL

TL;DR

Biomedical vision–language models need better handling of the complex semantics in radiology text, especially when paired images and reports are used as supervision. The paper introduces CXR-BERT and BioViL, alongside the MS-CXR benchmark, and reports state-of-the-art performance across many downstream tasks, including segmentation despite using only global alignment during training.

  • Problem

    Insufficiently adapted language models lack domain-specific radiology understanding, limiting biomedical vision–language modelling despite the availability of paired images and reports.

  • Method

    The paper develops CXR-BERT with radiology-specific semantic pretraining and uses it in BioViL, a self-supervised joint vision–language approach, while releasing the radiologist-annotated MS-CXR dataset.

  • Results

    State-of-the-art performance is reported across a wide range of public benchmarks, including improved radiology natural language inference, phrase grounding, and segmentation.

  • Takeaways & Limitations

    Principled textual-semantic modelling can improve contrastive self-supervised biomedical vision–language learning and support fine-grained image–text evaluation.

Abstract

from arXiv · show

Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.

1 Introduction

Biomedical vision–language learning can exploit routinely paired radiology images and reports without requiring extensive manual annotation. The paper argues that domain-specific text modelling is central to improving joint representations and downstream performance.

  • Motivation: Paired clinical images and expert reports offer routinely collected data for self-supervised and weakly supervised biomedical vision–language learning.These approaches address the expense and time required to obtain detailed ground-truth annotations.
  • Motivation: Self-supervised vision–language processing jointly learns image and text representations for applications including classification, report generation, error detection, and disease localisation.
  • Motivation: Insufficient vocabulary adjustment, fine-tuning, and language grounding can degrade joint latent representations, leaving substantial performance gains available from specialised text models.Free-text descriptions provide a semantically dense learning signal compared with image-only contrastive methods and supervised classification.
  • Contributions: CXR-BERT introduces an improved vocabulary and specialised pretraining, while BioViL combines improved text modelling, grounding, augmentation, and regularisation.
  • Contributions: MS-CXR provides radiologist-verified local phrase-grounding annotations for evaluating complex semantic reasoning in biomedical image–text models.The benchmark includes domain-specific location references, complex negations, and reporting-style variation.

2 Making the Most of Free-Text Supervision

The paper adapts language modelling to radiology semantics and uses the resulting text encoder within a jointly trained vision–language system. Its design combines specialised pretraining, text augmentation, regularisation, and continued language modelling.

  • CXR-BERT pretraining: CXR-BERT uses a domain-specific vocabulary and three-phase pretraining to capture dense semantics in radiology reports.The vocabulary is built from PubMed abstracts, MIMIC-III clinical notes, and MIMIC-CXR reports.
  • CXR-BERT pretraining: Radiology section matching contrasts Findings and Impression representations from the same report against unmatched section pairs.The objective is combined with masked language modelling and increased dropout during specialisation.
  • Augmentation and regularisation: Sentence shuffling within Findings and Impression sections provides text augmentation because their sentences are usually permutation-invariant.The augmentation is used both during CXR-BERT pretraining and joint model training.
  • BioViL: BioViL projects CNN image features and CXR-BERT text features into a joint space, retaining grid-level image embeddings for fine-grained tasks such as segmentation.
  • BioViL: Joint training combines global image–text alignment with continued masked language modelling and uses image/text augmentations plus higher text-encoder dropout.The authors report that the combination of these components improves downstream performance across the board.

3 Evaluating Self-Supervised Biomedical VLP

The evaluation targets a limitation of global image–text alignment: clinical reports may describe multiple findings corresponding to distinct image regions. MS-CXR is introduced as a challenging benchmark for testing this local, language-dependent correspondence.

  • Evaluation motivation: Local image–text alignment matters because radiology images and reports often contain multiple findings mapped to distinct regions.Global alignment can overfit to spurious correlations, such as chest tubes co-occurring with pneumothorax mentions.
  • Evaluation motivation: Existing public chest X-ray benchmarks may lack paired free-text descriptions, sufficient samples, diverse abnormalities, or curated phrases.
  • MS-CXR: MS-CXR is designed to test grounding of radiology phrases to image regions while requiring location parsing, reporting-style bias handling, and complex-negation understanding.
  • MS-CXR: MS-CXR contains 1153 image–sentence bounding-box pairs across eight cardiopulmonary findings, with annotations verified by two board-certified radiologists.Its phrases are genuine descriptions from radiology reports and dictated transcripts rather than short captions.
  • Evaluation design: The evaluation compares the coverage of recent CXR image–text alignment studies and includes BioViL among the assessed approaches.

4 Experiments

The experiments evaluate CXR-BERT and BioViL across text understanding, phrase grounding, global alignment, and segmentation benchmarks. Results indicate that specialised text modelling improves joint biomedical vision–language performance, including local reasoning and segmentation.

  • Evaluation scope: The evaluation covers CXR-BERT intrinsic text properties, RadNLI, MS-CXR phrase grounding, RSNA classification, and RSNA segmentation.The benchmarks assess text, image, and shared latent-space performance across global and local tasks.
  • Text model evaluation: 65.21% RadNLI accuracy is achieved using only MedNLI training samples, outperforming the reported comparison models and improving further after joint image training.RadNLI measures entailment, contradiction, and neutral relations in radiology report pairs.
  • Text model evaluation: CXR-BERT achieves higher radiology mask prediction accuracy than ClinicalBERT and PubMedBERT, indicating that radiology text differs from general clinical and biomedical literature text.The metric is Top-1 mask prediction accuracy on the MIMIC-CXR validation set.
  • Text model evaluation: Keeping the masked language modelling objective during joint fine-tuning produces substantial gains in downstream and NLI performance.The ablation reports improvements in entailment alignment and contradiction discrimination.
  • Local alignment evaluation – phrase grounding: BioViL achieves superior phrase grounding across radiological findings, while adding local losses in BioViL-L improves performance for almost all findings.Phrase similarity grids are evaluated against ground-truth bounding boxes on MS-CXR.
  • Global and local alignment evaluation: BioViL improves zero-shot and fine-tuned pneumonia classification and outperforms related methods on segmentation with the same labelled-data budget.The framework uses simple prompts and no local loss term, separate object detector, or segmentation network for the reported segmentation setup.

5 Related Work

Related work studies self-supervised vision–language learning and local image–text objectives for radiology data. Existing local datasets provide incomplete support for phrase grounding with free-text descriptions.

  • Biomedical vision–language processing: Prior radiology vision–language studies learn joint representations from paired images and reports using instance-level, local, or combined objectives.The cited approaches include contrastive instance-level learning, local terms, and local-only mutual-information approximations.
  • Biomedical vision–language processing: Related medical vision–language work commonly uses BioBERT, ClinicalBERT, BioClinicalBERT, or PubMedBERT with masked-language-model pretraining or continual pretraining.Specialised corpora include PubMed abstracts, PubMed Central full texts, and clinical notes.
  • Local alignment datasets: Existing chest X-ray datasets provide bounding boxes but lack free-text descriptions, limiting phrase-grounding evaluation.VinDr, RSNA Pneumonia, and the NIH Chest X-ray Dataset are listed as examples.
  • Local alignment datasets: The released MS-CXR benchmark addresses this gap with phrase-grounding annotations for radiology findings verified by board-certified radiologists.Its purpose is to evaluate local image–text reasoning and domain-specific language understanding.
  • Local alignment datasets: Other resources offer gaze locations, small phrase-annotation sets, or limited abnormalities rather than broad, complete phrase matches to image regions.The cited limitations concern REFLACX, MIMIC-CXR phrase annotations, and ImaGenome.

6 Conclusion

The paper concludes that improved text modelling benefits self-supervised biomedical vision–language processing, while BioViL achieves strong performance across downstream tasks and MS-CXR supports fine-grained evaluation.

  • CXR-BERT improves radiology language understanding and downstream vision–language performance through an improved vocabulary and specialised text modelling.The model improves radiology natural language inference, masked token prediction, and downstream VLP task performance.
  • BioViL is a simple self-supervised baseline that achieves state-of-the-art performance across many downstream tasks involving global and local image, text, and joint representations.It also reduces sensitivity to text-prompt engineering and outperforms related work on segmentation without a local loss or additional region-proposal model.
  • MS-CXR is a radiologist-verified chest X-ray phrase-grounding dataset designed to evaluate local image–text reasoning over domain-specific language and image regions.The benchmark contains curated regions annotated with descriptions of eight radiology findings.

A Additional Experiments

The additional experiment examines sensitivity to text-prompt choices in RSNA pneumonia zero-shot classification and finds BioViL more robust than the baseline.

  • BioViL shows much lower sensitivity to text prompts than the baseline on RSNA pneumonia zero-shot classification.Its performance remains high for relatively long queries, whereas the baseline is more affected by prompt choice.
  • The results suggest that CXR-BERT makes prompt engineering easier and less necessary for achieving high zero-shot classification performance.
  • Models without the proposed text-modelling improvements produce inconsistent latent text embeddings for synonym phrases.This inconsistency can make baseline methods reliant on post-hoc text-prompt engineering heuristics.

A.2 Qualitative Results – Phrase Grounding

Qualitative MS-CXR examples indicate that improved text modelling enhances phrase grounding across findings with varied sizes, types, and anatomical locations, while smaller structures remain difficult.

  • BioViL produces significantly enhanced phrase-grounding similarity maps compared with ClinicalBERT, ConVIRT, and GLoRIA.The examples span clinical findings differing in size, type, and anatomical location.
  • BioViL remains limited when grounding descriptions involving smaller structures such as rib fractures and pneumothorax.Some cases also fail to disassociate location modifiers from the entities corresponding to abnormalities.

A.3 Additional Evaluation Metrics

The appendix reports additional evaluation metrics for the classification and phrase-grounding experiments, extending the main tables with sensitivity, specificity, and mean IoU.

  • Table A.2 adds sensitivity and specificity for RSNA pneumonia zero-shot and fine-tuned classification experiments.Classification thresholds are set to maximise F1 scores for each method, and the comparison includes GLoRIA and BioViL training sizes.
  • Table A.4 provides mean IoU scores for the pretrained BioViL phrase-grounding experiments on MS-CXR.

A.4 Ablations on Training Dataset Size & Use of Raw Input Images

The ablation evaluates whether more training pairs and raw DICOM inputs affect BioViL phrase grounding, alongside qualitative failure cases on complex queries and small regions.

  • Ablation design: Training pairs increased from 146.7k to 176k, while a separate ablation replaced compressed JPEG images with raw DICOM images.Both interventions were evaluated for their impact on phrase grounding performance on MS-CXR.
  • Qualitative comparisons: BioViL qualitative outputs are compared with ClinicalBERT, ConVIRT, and GLoRIA on MS-CXR phrase grounding examples.The comparison uses latent vector similarity model outputs.
  • Failure cases: Failure cases include missed atelectasis despite a bilateral-bases location phrase and missed small pneumothorax despite an apical modifier.These examples show errors involving spatial specifications and small findings.
  • Failure cases: The model also failed to recognise loculated pleural fluid despite explicit apical and right-hemithorax modifiers.The query combines pathology, anatomical location, and spatial language.
  • Scope: The appendix notes that improved results and pre-training models were neither reported nor used in the main experiments.The authors present these findings as potentially useful for future research.

B.1 Key NLP and Dataset Challenges in Radiology

Radiology reports present domain-specific language challenges that general-domain vision–language models may not capture adequately, including modifiers, uncertainty, negation, and class imbalance.

  • Complex sentence structure: Radiology reports frequently use negation to indicate absent findings and describe changes in a patient’s health.This linguistic pattern requires text models to distinguish findings from their absence.
  • Use of modifiers: Spatial modifiers such as “medial,” “apical,” “bilateral,” and “basilar” are central to describing findings relative to anatomy.Severity modifiers including “mild,” “moderate,” and “severe” are also common.
  • Expressions of uncertainty: Radiology reports often express uncertainty through long descriptions with alternative diagnoses and long-range dependencies.The example combines competing interpretations with relative likelihoods.
  • Class imbalance: CXR VLP faces imbalanced latent entities because reports roughly divide scans into normal and abnormal findings.Normal and anomalous findings are described with substantially different clinical content.

C MS-CXR Dataset Details

MS-CXR is a radiologist-verified chest X-ray phrase-grounding benchmark that locally aligns clinical descriptions with image regions across eight pathologies.

  • General overview: MS-CXR contains 1,047 images and 1,153 bounding-box and sentence pairs.Each sentence describes a single pathology, with multiple boxes possible for one finding.
  • Pathology coverage: The dataset covers atelectasis, cardiomegaly, consolidation, edema, lung opacity, pleural effusion, pneumonia, and pneumothorax.These classes were selected based on overlap with existing datasets and the CheXbert classifier.
  • Annotation structure: Images may contain multiple pathologies, with separate bounding-box sets for each phrase describing a unique pathology.This supports locally aligned reasoning over distinct findings within one study.
  • Differences to existing annotations: MS-CXR extends and curates REFLACX and MIMIC-CXR-Annotations by adding studies and reviewing existing annotations.The cited passage identifies these as differences from the source annotation sets.
  • Annotation variability: The dataset includes variability in spatial extent, spatial modifiers, and the number of boxes and sentence annotations per image.These properties are illustrated in representative examples.

C.1 Label Collection and Review

MS-CXR candidate annotations are extracted from radiology reports and transcripts, filtered for suitable single-finding descriptions, and reviewed for image–phrase correctness. The resulting dataset is balanced across pathologies while remaining broadly similar to MIMIC-CXR.

  • Candidate extraction and filtering: Candidate phrases are extracted from MIMIC reports and REFLACX transcripts, then filtered with CheXbert to retain target-pathology descriptions.The procedure requires one target-pathology sentence per study and excludes sentences describing multiple findings.
  • Candidate extraction and filtering: Review excludes absent findings, unrelated or uncertain descriptions, incorrect bounding boxes, unsuitable images, and sentences exceeding 30 tokens.Differential diagnoses and longitudinal information are also excluded because they hinder grounding to a single paired image.
  • Candidate extraction and filtering: Images may contain multiple findings, but each separately described finding generates its own phrase–bounding-box instance.This preserves distinct annotations when separate report descriptions correspond to separate abnormalities in one image.
  • Review and adjudication: 222 of 817 automatically extracted candidate annotations were rejected during review and omitted from MS-CXR.The raw text was algorithmically processed to generate caption candidates before annotation-pair adjudication.
  • Balancing and dataset characteristics: Additional studies are randomly sampled conditional on underrepresented pathologies to increase dataset size and balance classes.Descriptions are extracted from MIMIC-CXR and ImaGenome using the established methodology.
  • Balancing and dataset characteristics: MS-CXR phrases average 7.49 tokens, compared with 6.76 in and 8.49 in transcription-derived sentences.These averages characterize phrase length across the benchmark datasets.
  • Balancing and dataset characteristics: MS-CXR subjects are older on average than the full MIMIC-CXR population, while the dataset remains broadly close to the MIMIC-CXR distribution.The age difference is attributed to excluding healthy studies without anomalous findings; the sampling is not expected to introduce gender bias.
Loading 2204.09817v4…