Source-linked AI summary
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, Jing Shao
TL;DR
Text-image retrieval is limited by approaches that independently embed images and sentences without sufficiently modeling their interactions. CAMP adaptively passes and gates messages across modalities, predicts scores from fused features, and trains with hardest-negative binary cross-entropy. Experiments on COCO and Flickr30k validate the approach and report results that significantly surpass state-of-the-art methods.
Problem
Text-image retrieval methods commonly embed images and sentences independently, leaving their fine-grained interactions insufficiently explored for matching.
Method
CAMP uses cross-modal message aggregation and gated fusion to adaptively pass information between image regions and sentence words, then predicts matching scores from fused features with hardest-negative binary cross-entropy.
Results
Experiments on COCO and Flickr30k significantly surpass state-of-the-art methods.
Takeaways & Limitations
CAMP provides an alternative to conventional visual-semantic embedding by incorporating comprehensive, fine-grained interactions while handling negative pairs and irrelevant information through adaptive gating.
Abstract
from arXiv · showhide
Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In this paper, we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively controls the information flow for message passing across modalities. Our approach not only takes comprehensive and fine-grained cross-modal interactions into account, but also properly handles negative pairs and irrelevant information with an adaptive gating scheme. Moreover, instead of conventional joint embedding approaches for text-image matching, we infer the matching score based on the fused features, and propose a hardest negative binary cross-entropy loss for training. Results on COCO and Flickr30k significantly surpass state-of-the-art methods, demonstrating the effectiveness of our approach.
1. Introduction
CAMP addresses the limits of independent image-sentence embedding by modeling fine-grained cross-modal interactions and adaptively controlling message flow. It fuses cross-modal messages to predict matching scores and uses hardest-negative binary cross-entropy training.
- Prior methods independently embed images and sentences into a joint space, but do not exploit their interactions before measuring similarity.
- Fine-grained matching requires attending to salient regions and words, filtering irrelevant information, and identifying cross-modal misalignments.Examples include a missing blue shirt or an absent railing near a brick wall.
- CAMP comprises Cross-modal Message Aggregation and Cross-modal Gated Fusion to adaptively explore interactions between images and sentences.
- Adaptive gating controls message fusion because negative pairs and irrelevant background information can confuse matching, while residual connections preserve original features when gates are low.
- The model predicts matching scores from fused features and trains with hardest-negative binary cross-entropy instead of conventional joint-embedding ranking.
2. Related Work
Related work primarily learns image-sentence similarities through joint embeddings, while other approaches use attention and fusion to model language-vision interactions. CAMP differs by using symmetric adaptive message passing to extract salient cross-modal features for retrieval.
- Most text-image retrieval methods embed image and sentence features into a joint space and train similarities with ranking-based objectives.
- Stacked cross attention models correspondences between words and regions, but attends asymmetrically to either words or regions rather than both modalities.
- CAMP uses adaptive message passing to exploit interactions between language and vision and extract salient features for text-image matching.
3. Cross-modal Adaptive Message Passing
CAMP enables deep image-sentence interactions through cross-modal message aggregation and gated fusion, then predicts matching from fused features rather than joint-space distances. Its gates encourage fusion for matched pairs while suppressing mismatched information and irrelevant content.
- CAMP combines Cross-modal Message Aggregation with Cross-modal Gated Fusion to enable deep interactions between images and sentences.
- Cross-modal Message Aggregation: Cross-modal attention uses region cues to attend to words and word cues to attend to regions, producing modality-specific aggregated messages.The resulting messages represent visual features corresponding to words and textual features corresponding to regions.
- Cross-modal Gated Fusion: The model gates fusion between original features and cross-modal messages because directly fusing mismatched pairs can produce meaningless representations and impede training and inference.The gate controls region-word fusion while preserving original features when intensive fusion is inappropriate.
- Cross-modal Gated Fusion: For matched regions and sentences, high gate values encourage fusion, whereas low values suppress mismatched information.Figure 3 illustrates fusion for positive and negative region-message pairs.
- Fused Feature Aggregation: CAMP aggregates fused regional and word features into whole-image and whole-sentence representations before matching.Linear projections and SoftMax normalization provide attention weights for aggregating the fused features.
- Fused Feature Matching: Unlike conventional visual-semantic embedding, CAMP predicts matching from fused features, avoiding the requirement that image and language features share one embedding space.The paper motivates this formulation as better suited to comprehensive and fine-grained cross-modal interactions.
4. Experiments
Experiments evaluate CAMP on Flickr30K and COCO using standard retrieval splits and recall metrics, and compare its interaction-based design with conventional and ablated alternatives. Qualitative and ablation analyses examine fine-grained matching and adaptive gating.
- Experimental settings: Flickr30K and COCO provide the evaluation benchmarks, with COCO tested on both 5-fold 1K and full 5K test-image settings.Flickr30K uses 1,000 validation and 1,000 test images; COCO uses the Karpathy split with 5,000 validation and 5,000 test images.
- Experimental settings: Recall at K, reported as R@1, R@5, and R@10, measures the proportion of correct matches among the top-k retrieved results.These are the stated evaluation metrics for text-image retrieval.
- Qualitative results: The gated fusion model retrieves ground-truth images and ranks results with finer agreement to details such as a forest setting and a red collar.The qualitative example reports that all top-five results relate to the query, while the top-ranked image matches more details.
- Quantitative results: CAMP compares favorably with conventional approaches that independently embed images and sentences, while ablations test the contribution of cross-modal interaction.The ablation baseline removes cross-modal interactions and uses cosine-distance matching with hardest-negative ranking loss.
- Ablation study: Adaptive gates intensify fusion for strongly aligned words or regions and suppress uninformative content, while experiments remove gates or residual connections to assess their necessity.The study also compares concatenation and product fusion, and reports that conventional fusion causes a significant performance decline.
- Ablation study: Matching scores are predicted with an MLP on fused features, and hardest-negative cross-entropy training is compared against joint-embedding and ranking-loss alternatives.The joint-embedding ablation calculates cosine distance between fused features, whereas the MLP+ranking-loss variant changes the scoring or supervision choice separately.
5. Conclusion
The conclusion presents CAMP as a model that adaptively passes messages between textual and visual modalities. It emphasizes fine-grained interactions and adaptive handling of mismatched or irrelevant information, supported by benchmark experiments and analysis.
- Conclusion: CAMP adaptively passes messages across textual and visual modalities to incorporate cross-modal interactions in text-image retrieval.The model is introduced as a Cross-modal Gated Fusion approach based on this observation.
- Conclusion: The adaptive gating scheme handles negative pairs and irrelevant information while supporting comprehensive and fine-grained cross-modal interactions.The conclusion identifies these properties as central aspects of the approach.
- Conclusion: Extensive experiments and analysis on benchmarks are used to demonstrate the effectiveness of CAMP.The conclusion states this evidence without specifying a numerical result in the supplied passage.