Source-linked AI summary
Vision-Language Pre-Training with Triple Contrastive Learning
Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, Junzhou Huang
TL;DR
Existing vision-language pre-training can overlook intra-modal similarity and localized structure, with noisy data further degrading representations. TCL combines cross-modal and intra-modal contrastive objectives with local MI maximization, and achieves state-of-the-art results across vision-language benchmarks. The paper also notes that learned representations may perform worse for underrepresented groups.
Problem
Cross-modal alignment alone may fail to preserve meaningful within-modality similarity and localized structure, especially when pre-training data is noisy.
Method
TCL combines cross-modal alignment, intra-modal contrastive learning, and local MI maximization between global representations and local image or text regions.
Results
TCL achieves a new state of the art across vision-language benchmarks, including image-text retrieval and VQA; on MSCOCO zero-shot retrieval, mean recall is 79.5% versus ALIGN's 70.9%.
Takeaways & Limitations
Combining cross-modal and intra-modal supervision, together with local structure information, provides a competitive approach for vision-language representation learning.
Takeaways & Limitations
Learned representations may reflect available data, causing worse performance for underrepresented groups.
Abstract
from arXiv · showhide
Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text. However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations. For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by. This problem can get even worse when the pre-training data is noisy. In this paper, we propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning. To take advantage of localized and structural information from image and text input, TCL further maximizes the average MI between local regions of image/text and their global summary. To the best of our knowledge, ours is the first work that takes into account local structure information for multi-modality representation learning. Experimental evaluations show that our approach is competitive and achieves the new state of the art on various common down-stream vision-language tasks such as image-text retrieval and visual question answering.
1. Introduction
Vision-language pre-training addresses multimodal tasks but cross-modal alignment alone can overlook intra-modal structure and localized information, especially in noisy data. TCL combines cross-modal and intra-modal self-supervision with local MI maximization, achieving strong benchmark performance.
- Vision-language pre-training supports multimodal tasks including visual question answering, image-text retrieval, and visual entailment.
- Cross-modal alignment can produce degraded representations because it does not ensure similar inputs within each modality remain close, particularly with noisy web data.
- Global MI maximization can overlook localized and structural information, allowing unrelated image patches or text tokens to dominate learning.
- TCL combines cross-modal alignment, intra-modal contrastive learning, and local MI maximization to learn semantically meaningful multimodal representations.The three modules maximize global image-text MI, agreement between augmented views, and MI between global representations and local regions.
- TCL maximizes local MI between local image or text regions and their global summaries to exploit localized and structural information.
- 79.5% mean recall versus 70.9% for ALIGN on MSCOCO zero-shot image-text retrieval demonstrates TCL's reported improvement despite using 5M rather than 1.8B image-text pairs.Pre-training on 14M pairs produced a further performance boost.
2. Related Work
Prior vision-language pre-training methods use multimodal objectives and contrastive alignment, while MI-based methods motivate local information modeling. TCL extends these ideas to multimodal representation learning with local MI and reports strong empirical performance.
- CLIP and ALIGN use large-scale image-text data, while later VLP methods learn joint representations for tasks such as VQA and visual reasoning.
- ALBEF aligns image and text features before joint modeling, whereas TCL additionally uses intra-modal self-supervision to address feature degeneration.
- CODIS uses teacher-student distillation and cluster representations for feature alignment, distinguishing it from TCL's approach.
- InfoNCE identifies positive samples among negatives and provides a lower bound on MI, making contrastive loss a practical route to MI maximization.
- TCL extends local MI maximization from intra-modal methods to multimodal representation learning by relating global representations to image patches and text tokens.
- Patch embeddings from the transformer's last layer outperform intermediate-layer patches in the reported experiments.
3. Method
TCL uses vision, text, and fusion encoders with momentum counterparts, aligning representations through cross-modal and intra-modal contrastive objectives. It combines CMA, IMC, and LMI to support joint embeddings while incorporating global and localized information.
- Model Architecture: The framework uses vision, text, and fusion encoders, each paired with a momentum encoder updated by a moving average.The vision and text encoders learn unimodal features before an alignment module prepares them for fusion.
- Uni-modal Representation Learning: Two augmented views of each image provide correlated inputs for unimodal representation learning.The views are produced by separate augmentations sampled from the same augmentation family.
- Alignment Module: The alignment module contains CMA, IMC, and LMI, which provide complementary cross-modal and intra-modal supervision before fusion.CMA aligns matched image-text pairs using InfoNCE, while IMC adds within-modality contrastive supervision.
- Cross-Modal Alignment (CMA): CMA pulls matched image-text embeddings together and pushes unmatched pairs apart to maximize their mutual information.The method minimizes InfoNCE as a lower bound on mutual information, using negative examples maintained in queues.
- Intra-Modal Contrastive (IMC): IMC contrasts positive pairs within vision and language, using augmented image views and independently dropped-out text representations.It encourages agreement within each modality and promotes uniformity across the representation space.
- Local MI Maximization (LMI): LMI maximizes mutual information between global representations and local image patches or text tokens, preserving localized and structural information.This objective uses local positives and negative patch embeddings so representations encode information shared across regions rather than only dominant patches.
4. Experiments
Experiments evaluate TCL on retrieval, VQA, visual entailment, and visual reasoning, including zero-shot, fine-tuned, and ablation settings. TCL generally outperforms comparable methods, while larger pre-training data further improves performance.
- Experimental setup: The 4M dataset contains 5.1M image-text pairs, while the larger setting reaches 16M pairs across 14.97M unique images.The experiments use COCO, Visual Genome, Conceptual Captions, SBU Captions, and CC12M.
- Zero-shot image-text retrieval: TCL improves zero-shot retrieval over ViLT by +9.5% on COCO and +12.2% on Flickr30K.The comparison supports conducting cross-modal alignment before fusion.
- Zero-shot image-text retrieval: TCL adds +2.7% TR/R@1 and +3.4% IR/R@1 over ALBEF on MSCOCO (5K).The reported gains come from explicitly leveraging intra-modal information from global and local perspectives.
- Vision+language tasks: TCL achieves state-of-the-art results on five of six criteria across VQA, visual entailment, and NLVR2.These tasks require joint multi-modal embeddings from image and text inputs.
- Ablation studies: Larger-scale pre-training provides a significant performance boost, while m = 0.5 achieves the best performance in the momentum-coefficient study.The authors hypothesize that still larger datasets could improve results further.
5. Limitations
Learned representations may reflect features present in the available data, creating a risk of bias against underrepresented groups.
- Underrepresented groups may receive worse model performance when their features are insufficiently represented in the available data.
6. Conclusion
TCL extends cross-modal alignment with intra-modal supervision and local mutual-information maximization. Experiments show that it outperforms existing state-of-the-art methods on widely used benchmarks.
- TCL combines cross-modal alignment, intra-modal supervision, and local MI maximization in a single vision-language pre-training framework.The local MI objective links global representations with local information from image patches or text tokens.
- Intra-modal supervision is intended to make representations meaningful within each modality while benefiting cross-modal alignment and joint embedding learning.
- TCL maximizes mutual information between global representations and local image-patch or text-token information to incorporate localized and structural cues.
- TCL outperforms existing state-of-the-art methods by a large margin on widely used vision-language benchmarks.