Source-linked AI summary
Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, Lequan Yu
TL;DR
Existing medical image-text methods provide limited supervision across the disease-, instance-, and pathological-region-level correspondences available in paired images and reports. MGCA aligns these granularities through instance-wise, token-wise, and disease-level objectives, and experiments across seven downstream datasets show substantial performance with limited annotated data. The framework is evaluated on classification, detection, and segmentation, but not image-image or image-text retrieval.
Problem
Existing medical image-text joint learning methods are limited to instance or local supervision and do not exploit disease-level semantic correspondences.
Method
MGCA combines instance-wise alignment, bidirectional-cross-attention token-wise alignment with contrastive learning, and disease-level alignment through cross-modal cluster assignment consistency.
Results
Across seven downstream datasets covering image classification, object detection, and semantic segmentation, MGCA achieves stable and superior performance with limited annotated data.
Takeaways & Limitations
MGCA learns generalized and discriminative medical visual representations for versatile downstream tasks while reducing the annotation burden.
Takeaways & Limitations
The study does not evaluate image-image or image-text retrieval downstream tasks.
Abstract
from arXiv · showhide
Learning medical visual representations directly from paired radiology reports has become an emerging topic in representation learning. However, existing medical image-text joint learning methods are limited by instance or local supervision analysis, ignoring disease-level semantic correspondences. In this paper, we present a novel Multi-Granularity Cross-modal Alignment (MGCA) framework for generalized medical visual representation learning by harnessing the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels, i.e., pathological region-level, instance-level, and disease-level. Specifically, we first incorporate the instance-wise alignment module by maximizing the agreement between image-report pairs. Further, for token-wise alignment, we introduce a bidirectional cross-attention strategy to explicitly learn the matching between fine-grained visual tokens and text tokens, followed by contrastive learning to align them. More important, to leverage the high-level inter-subject relationship semantic (e.g., disease) correspondences, we design a novel cross-modal disease-level alignment paradigm to enforce the cross-modal cluster assignment consistency. Extensive experimental results on seven downstream medical image datasets covering image classification, object detection, and semantic segmentation tasks demonstrate the stable and superior performance of our framework.
1 Introduction
Large-scale labeled medical datasets are expensive to assemble, motivating representation learning from paired radiology reports and images. MGCA addresses the underused disease-, instance-, and pathological-region-level correspondences across modalities.
- Large annotated medical-image datasets are expensive and time-consuming to assemble, motivating learning from paired radiology reports without extra manual annotation.
- Medical images and radiology reports naturally exhibit disease-level, instance-level, and pathological region-level semantic correspondences.
- Existing image-text joint learning methods explore correspondence at only part of these levels, limiting use of image/report information and representation capability.
- MGCA combines instance-wise, token-wise, and disease-level alignment, including bidirectional cross-attention for local image-text matching and cluster assignment consistency for disease-level alignment.
- The framework targets generalized medical visual representations transferable to image- and pixel-level downstream tasks with limited annotated data.
2 Related Work
Prior work learns medical visual representations from reports through extracted disease labels or vision-language contrastive learning, while dense prediction methods target fine-grained visual correspondence. These approaches provide context for MGCA’s cross-modal alignment framework.
- Report-supervised medical representation learning uses either human-designed rules to extract disease labels or vision-language contrastive learning.Rule-based extraction requires human labor and expert knowledge, while extracted labels can be noisy.
- MGCA organizes cross-modal correspondence at disease, instance, and pathology-region levels within a unified representation-learning framework.
- Contrastive learning brings positive instances closer and negative pairs farther apart in an embedding space, with medical-image applications achieving substantial performance.
- Fine-grained semantic correspondence is important for dense prediction tasks such as object detection and semantic segmentation.
3 Method
MGCA learns generalized medical visual representations by aligning image-report pairs at instance, pathology-region/token, and disease/prototype levels. Its modules combine global contrastive pairing, bidirectional local token matching, and cross-modal cluster consistency.
- MGCA jointly exploits disease-level, instance-level, and pathology-region token-level correspondences between medical images and reports.
- Instance-wise Cross-modal Alignment: Instance-wise alignment contrasts true image-report pairs against random pairings to preserve cross-modal agreement in the latent space.The objective averages symmetric image-to-text and text-to-image losses over image-report pairs.
- Token-wise Cross-modal Alignment: Token-wise alignment uses bidirectional cross-attention to softly match visual and text tokens, then applies contrastive losses to align them.Visual tokens attend to text tokens, while token importance weights emphasize pathology-relevant regions during local alignment.
- Token-wise Cross-modal Alignment: The token module performs per-token InfoNCE over local similarities rather than contrasting aggregated instance-level similarities.This distinguishes MGCA’s token-wise alignment from the cited local contrastive loss.
- Disease-level Cross-modal Alignment: Cross-modal prototype alignment addresses disease-level semantics by enforcing consistency between image and report cluster assignments.Soft assignments are obtained through clustering, and cross-modal prediction uses each modality’s assignment code as the other modality’s pseudo-label.
4 Experiments
MGCA is pretrained on roughly 217k MIMIC-CXR image-report pairs and evaluated across seven datasets spanning classification, detection, and segmentation. It achieves strong transfer performance, including improvements in low-data settings, while ablations support complementary benefits from its alignment components.
- Experimental Setup: The evaluation covers seven downstream datasets across medical image classification, object detection, and semantic segmentation.Classification uses CheXpert, RSNA Pneumonia, and COVIDx; detection uses RSNA Pneumonia and Object CXR; segmentation uses SIIM and RSNA.
- Experimental Setup: MGCA is pretrained on roughly 217k MIMIC-CXR image-report pairs after removing lateral views and short or empty reports.The framework uses BioClinicalBERT with ViT-B/16 by default and also reports ResNet-50 results for comparison.
- Results: MGCA with ViT-B/16 achieves the best performance in all nine reported classification settings and remains competitive with ResNet-50.With 1% training data, it exceeds GLoRIA-MIMIC by 1.7% AUROC on CheXpert, 2.1% AUROC on RSNA, and 8.3% ACC on COVIDx.
- Results: MGCA outperforms ConVIRT, GLoRIA, and GLoRIA-MIMIC by a large margin across the reported object-detection settings.The comparison uses the same ResNet-50-YOLOv3 architecture and includes fine-tuning with 1%, 10%, and 100% of the training data.
- Results: MGCA obtains higher Dice scores than the compared methods in five of six segmentation settings.With 1% training data, Dice improves by 12.3% on SIIM and 2.7% on RSNA over GLoRIA-MIMIC.
- Analysis of Our Framework: Jointly training ITA, CTA, and CPA gives the best ablation performance, while CTA and CPA provide complementary gains.CTA produces a larger segmentation improvement than CPA, supporting its role in learning fine-grained information.
5 Discussion and Conclusion
MGCA learns generalized medical visual representations from free-text radiology reports by aligning image–report semantics across multiple granularities. Experiments span seven downstream datasets, while the authors identify retrieval evaluation and broader pre-training integration as future work.
- MGCA harnesses multi-granularity semantic correspondences between medical images and reports to learn generalized and discriminative visual representations.The framework uses free-text radiology reports and targets versatile downstream tasks with limited annotation.
- The framework is evaluated on seven downstream datasets and achieves substantial performance with limited annotated data.
- The work does not evaluate image-image or image-text retrieval downstream tasks, which the authors identify as a limitation.
- MGCA may assist abnormality diagnosis from chest X-rays with limited annotated data, potentially reducing radiologists’ workload and supporting health in poor areas.The authors also caution that medical data may contain private information or harmful texts, requiring careful analysis before practical deployment.
Checklist
The checklist records empirical-work disclosures covering claims, limitations, societal impacts, assumptions, reproducibility, assets, licensing, consent, privacy, and human-subject involvement.
- Research reporting: The paper states that its main claims, limitations, contributions, scope, and potential negative societal impacts were addressed in the manuscript.The checklist points to the introduction and Section 5 for these disclosures.
- Research reporting: The work is described as mainly empirical, with theoretical assumptions and complete proofs marked not applicable.The checklist explicitly says the work mainly involves empirical contributions.
- Reproducibility: The paper reports reproduction materials and training details, including datasets, model information, data splits, supplementary experiments, and code.Error bars were listed as an experiment-reporting item, but the supplied passage does not show whether they were provided.
- Reproducibility: The checklist reports that compute resources and their type were included in the second paragraph of Section 4.1.The supplied text records this disclosure as affirmative.
- Assets and data: The experiments use publicly available datasets, cite their creators, and state that the datasets are under non-commercial licenses.The checklist also says dataset and model assets are described, with code and pretrained models to be released later.
- Assets and data: Consent and participant-risk items are marked not applicable because the experiments use public datasets and involve no human subjects.The paper states that no personally identifiable information is involved.