Source-linked AI summary
Multimodal Contrastive Training for Visual Representation Learning
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, Baldo Faieta
TL;DR
Existing visual pre-training either depends on costly fine-grained labels or requires extremely large datasets or long training. This paper jointly preserves intra- and inter-modal similarities with contrastive learning, yielding visual representations that transfer across classification, tagging, retrieval, detection, and instance segmentation.
Problem
Supervised visual pre-training requires prohibitively heavy fine-grained labeling, while self-supervised methods require extremely large training sets or longer training epochs.
Method
A unified multimodal framework combines self-supervised intra-modal learning with inter-modal similarity preservation through multiple contrastive training paths.
Results
The learned visual representations transfer effectively across image classification, tagging, cross-modal retrieval, object detection, and instance segmentation.
Takeaways & Limitations
Combining intrinsic within-modality data properties with cross-modal semantic information produces high-quality visual features with scalability and transferability across tasks.
Takeaways & Limitations
For images with only tags or captions, feature learning is guided by modality-specific combinations of the available contrastive losses.
Abstract
from arXiv · showhide
We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of $55.3\%$ on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.
1. Introduction
The paper proposes unified multimodal visual representation learning that combines self-supervised intra-modal structure with cross-modal semantic correlations. Extensive evaluations target transfer across classification, tagging, retrieval, detection, and segmentation tasks.
- Visual representation learning supports classification, tagging, detection, and segmentation, but supervised labeling is costly and self-supervised methods often require huge datasets or long training.
- Unlike VirTex’s single-path cross-modal proxy task, the method also exploits intrinsic data properties within each modality through self-supervision.
- Additional tags and captions are easier to acquire than fine-grained labels and provide higher-level concepts for connections across modalities.
- The framework jointly exploits unlabeled within-modality structure, cross-modal similarity in a common semantic space, and extensibility to incoming modalities.
- The method combines intra-modal and inter-modal training paths with contrastive losses, learning intrinsic modality patterns alongside cross-modal interactions.
- The symmetric framework can incorporate new modalities, while pretrained visual representations transfer across many downstream computer vision tasks.
- The approach is evaluated through comparisons with supervised, self-supervised, and text-learning methods across classification, tagging, retrieval, detection, and segmentation.
2. Related Work
Related work spans contrastive self-supervised visual learning and joint visual-textual pretraining. The paper positions its approach as using multimodal data through multiple contrastive training paths.
- Self-supervised learning: Contrastive self-supervised methods compare instances to learn visual representations, using mechanisms such as memory banks, momentum encoders, and queues.
- Joint visual-textual pretraining: Joint visual-textual pretraining methods use multimodal information for computer vision tasks, while this method aligns visual features with semantic information through multiple training paths.
3. Method
The method unifies intra-modal self-supervised contrastive learning with inter-modal similarity preservation, using separate paths and losses to learn visual features from images, captions, and tags. It combines modality-specific structure with cross-modal semantic alignment through a weighted multimodal objective.
- Framework: The framework combines intra-modal self-supervised learning with inter-modal contrastive learning in a common semantic space.Intra-modal paths learn from individual modalities, while inter-modal paths preserve similarity bidirectionally across modalities.
- Intra-modality Contrastive Learning: Intra-modal visual and textual training preserves similarity between augmented variants using MoCo-v2-style encoders, momentum updates, and dynamic feature queues.The visual path uses image augmentations, query-key pairs, momentum encoders, and a queue; the textual path applies analogous processing to augmented captions.
- Intra-modality Contrastive Learning: Tag supervision adds images sharing tags as positive visual pairs, encouraging semantically aligned features beyond pairs originating from the same image.The tag-based positive set includes queue keys whose tag overlap exceeds threshold ϵ, and Jtag reduces to Jii when no additional semantic positives exist.
- Inter-modality Contrastive Learning: Inter-modal training embeds images and captions into a common space and optimizes image-to-caption and caption-to-image similarity against negative queue features.Distinct MLP branches map features for cross-modal learning so the intra-modal and inter-modal objectives do not interfere with each other.
- Overall Objective: The final objective weights visual, tag, textual, and bidirectional cross-modal contrastive losses through five trade-off parameters.The loss is J = λiiJii + λtagJtag + λccJcc + λicJic + λciJci, and the framework can train with incomplete tag or caption information.
4. Experiments
Experiments evaluate COCO- and Stock-pretrained visual backbones across classification, detection, segmentation, tagging, and cross-modal retrieval. The multimodal approach consistently benefits transfer performance, with ablations attributing gains to separate intra- and inter-modal representations and additional tag supervision.
- Evaluation setup: The evaluation spans ImageNet classification, PASCAL VOC detection, COCO instance segmentation, COCO retrieval, and Stock tagging and retrieval.COCO and Stock provide image-caption-tag tuples for pretraining, while downstream protocols detach the pretrained ResNet-50 backbone and evaluate task-specific performance.
- ImageNet classification: 2.1 p.p. and 3.0 p.p. are the reported gains over VirTex and ICMLM, respectively, for the caption-trained model.The pretrained visual backbone also outperforms self-supervised methods under the common linear-classification transfer protocol.
- Object detection and instance segmentation: 5× fewer pre-training epochs still produce comparable PASCAL VOC detection APs to VirTex trained for 1000 epochs.The method also consistently outperforms VirTex at matched 200- and 1000-epoch settings.
- Object detection and instance segmentation: The method significantly outperforms COCO-pretrained self-supervised methods on PASCAL VOC detection and COCO instance segmentation.These evaluations use fine-tuned Faster R-CNN and Mask R-CNN backbones under the stated transfer protocols.
- Stock evaluation: The method consistently outperforms supervised baselines on Stock image tagging and cross-modal retrieval.Stock retrieval is evaluated on a 10K-image test set, while tagging uses a 50K-image test set.
- Ablation study: Separate MLPs improve ImageNet transfer over unified MLPs, with the final 128-d intra-modal and 1024-d inter-modal design reaching 54.9%.The ablation reports 49.6–52.3% for unified MLPs and 53.6–53.9% for distinct MLPs with matched dimensions.
- ImageNet classification: 55.3% top-1 accuracy is achieved on ImageNet-1K with the final model trained using all five losses.The tag-supervision ablation reaches 50.2%, while the model using only the intra-modal term reaches 49.3%.
5. Conclusion
The paper presents a unified multimodal framework that learns visual representations through intra-modal and inter-modal contrastive learning paths. Its symmetric design supports scalability and transferability across datasets, modalities, and downstream tasks.
- The framework combines two intra-modal and inter-modal learning paths with carefully designed contrastive losses.It exploits intrinsic patterns within modalities and semantic information from cross-modal interactions.
- The method demonstrates better scalability and transferability across extensive experiments on various datasets and tasks.
- Because the framework is symmetric across modalities, it can be extended beyond images and captions to video and audio.