Source-linked AI summary
Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training
Zhihong Chen, Yuhao Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, Tsung-Hui Chang
TL;DR
Medical vision-and-language pre-training remains limited by few domain-specific studies and the need to learn from medical image-text data. M3AE addresses this gap with self-supervised masked reconstruction using modality-specific designs, and the authors report state-of-the-art performance across downstream tasks.
Problem
Medical vision-and-language pre-training has received limited study despite the need for effective representations from medical images and texts.
Method
M3AE self-supervisedly reconstructs missing pixels and tokens from randomly masked medical images and texts using modality-specific masking, representations, and decoders.
Results
M3AE achieves state-of-the-art results on all downstream tasks in the constructed medical vision-and-language benchmark.
Takeaways & Limitations
The approach provides an effective, simple way to learn cross-modal medical knowledge without fine-grained image or text annotations.
Abstract
from arXiv · showhide
Medical vision-and-language pre-training provides a feasible solution to extract effective vision-and-language representations from medical images and texts. However, few studies have been dedicated to this field to facilitate medical vision-and-language understanding. In this paper, we propose a self-supervised learning paradigm with multi-modal masked autoencoders (M$^3$AE), which learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts. There are three key designs to make this simple approach work. First, considering the different information densities of vision and language, we adopt different masking ratios for the input image and text, where a considerably larger masking ratio is used for images. Second, we use visual and textual features from different layers to perform the reconstruction to deal with different levels of abstraction in visual and language. Third, we develop different designs for vision and language decoders (i.e., a Transformer for vision and a multi-layer perceptron for language). To perform a comprehensive evaluation and facilitate further research, we construct a medical vision-and-language benchmark including three tasks. Experimental results demonstrate the effectiveness of our approach, where state-of-the-art results are achieved on all downstream tasks. Besides, we conduct further analysis to better verify the effectiveness of different components of our approach and various settings of pre-training. The source code is available at~\url{https://github.com/zhjohnchan/M3AE}.
1 Introduction
Medical vision-and-language pre-training addresses the need for transferable image-text representations despite limited labeled data and domain-specific studies. M3AE uses masked multimodal reconstruction and achieves strong downstream performance.
- Motivation: Medical vision-and-language pre-training learns transferable representations from large-scale medical image-text data for multiple downstream tasks.These include medical visual question answering, image-text classification, and image-text retrieval.
- Research gap: Existing general-domain VLP models underperform in medical image-text classification without integrated domain-specific information.Prior medical evaluation also covered only a limited setting, motivating broader Med-VLP design and evaluation.
- Approach: M3AE reconstructs missing image pixels and text tokens from randomly masked inputs using purely Transformer-based models.The method is designed to learn cross-modal medical knowledge self-supervisedly without fine-grained image or text annotations.
- Approach: M3AE uses modality-specific masking ratios, layer-specific reconstruction features, and separate vision and language decoder designs.Images receive a higher masking ratio than texts; vision uses a Transformer decoder and language uses an MLP decoder.
- Evaluation: The authors construct a medical vision-and-language benchmark and report that M3AE outperforms previous studies on all downstream tasks.The benchmark is intended to support comprehensive evaluation and further research.
2 The Proposed Approach
The proposed approach uses a Transformer-based multimodal backbone to encode medical images and texts, fuse their representations, and reconstruct masked image and text signals. Its masked-autoencoder design accounts for modality-specific information density and abstraction levels.
- Backbone Model Architecture: The backbone comprises vision and language encoders plus a multimodal fusion module for jointly processing medical images and texts.The fusion module uses self-attention within modalities and cross-attention across modalities.
- Vision Encoder: Images are divided into patches, linearly projected with position information, and processed by a Transformer vision encoder.A learnable aggregation token is prepended to the patch sequence.
- Language Encoder: Text is tokenized into subwords, projected into embeddings with special boundary tokens and position information, and processed by a Transformer language encoder.The language encoder follows BERT-style WordPiece tokenization.
- Multimodal Fusion Module: The multimodal fusion module stacks Transformer layers containing self-attention, cross-attention, and feedforward sub-layers.Self-attention models within-modality interactions, whereas cross-attention integrates information across modalities.
- Multi-Modal Masked Autoencoders: M3AE masks random image patches and text tokens, then reconstructs missing pixels and tokens using modality-specific designs.Images and texts use different masking ratios, representations from distinct layers support reconstruction, and image and language reconstruction losses use MSE and negative log-likelihood, respectively.
3 Experiments
The experiments evaluate M3AE on three medical image-text understanding tasks using ROCO and MedICaT pre-training data. M3AE achieves state-of-the-art results across downstream datasets, while ablations and qualitative cases examine its components and effects.
- Pre-Training Setup: The pre-training data comprise over 81,000 ROCO image-text pairs and over 217,000 MedICaT medical images with captions and inline textual references.ROCO uses official splits; MedICaT reserves 1,000 images each for validation and testing.
- Evaluation Setup: M3AE is evaluated on Med-VQA, medical image-text classification, and medical image-text retrieval.The evaluation uses accuracy for Med-VQA and classification, and Recall@K for retrieval.
- Ablation Study: M3AE achieves the best VQA-RAD ablation result when both MIM and MLM are used as pre-training objectives.The analysis reports that MIM alone does not improve performance, whereas adding MLM produces considerably better results.
- Representation Analysis: Layer-0 representations yield the worst MIM performance, indicating that textual information is important for performing MIM.The study compares representations from layers 0 through 6.
- Qualitative Analysis: In four VQA-RAD cases, MLM+MIM answers yes/no, organ-related, and disease-related questions correctly where MLM does not.The qualitative analysis suggests that MLM+MIM might aid finer-grained image-text mappings.
4 Conclusion
The paper proposes M3AE for pre-training on medical image-text pairs and evaluates it across a three-task benchmark. It reports superior performance and state-of-the-art results across the evaluated datasets.
- Conclusion: M3AE pre-trains on medical image-text pairs using designs for masking ratios, representation selection, and decoder designs.The benchmark includes Med-VQA, medical image-text classification, and medical image-text retrieval.
- Conclusion: M3AE achieves state-of-the-art results on all evaluated downstream tasks and datasets.