Source-linked AI summary
Contrastive Learning for Weakly Supervised Phrase Grounding
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, Derek Hoiem
TL;DR
Weakly supervised phrase grounding must learn word–region associations from paired image-caption data without explicit grounding annotations. The paper trains word-region attention by maximizing an InfoNCE lower bound on mutual information and uses language-model-guided, context-preserving negative captions. This approach improves grounding performance over random negative sampling and achieves a reported gain on Flickr30K Entities.
Problem
Phrase grounding requires associating caption words with image regions, but learning this mapping without explicit region-word annotations remains challenging.
Method
The model optimizes word-region attention with an InfoNCE mutual-information objective and context-preserving negative captions generated by plausible noun substitutions.
Results
The model achieves a 5.7% absolute gain in pointing accuracy over Align2Ground on Flickr30K Entities and contextually plausible negatives outperform random sampling by ≥8% in recall@1 and pointing accuracy.
Takeaways & Limitations
Language-model-generated, context-preserving negatives substantially improve contrastive learning for weakly supervised phrase grounding compared with randomly sampled negatives.
Takeaways & Limitations
The Llang objective with context-preserving negatives is not a valid mutual-information lower bound because it oversamples words related to a caption.
Abstract
from arXiv · showhide
Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a $\sim10\%$ absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of $5.7\%$ to achieve $76.7\%$ accuracy on Flickr30K Entities benchmark.
1 Introduction
The paper frames weakly supervised phrase grounding as learning word–region associations from paired image-caption data without explicit grounding annotations. It replaces conventional multiple-instance learning with mutual-information-based contrastive training that learns attention to align contextualized words with image regions.
- Motivation: Weakly supervised phrase grounding aims to associate caption words with image regions for downstream vision-language tasks, but explicit region-word annotations are unavailable.The paper motivates learning from paired image-caption data instead of laborious grounding annotations.
- Negative captions: The framework introduces context-preserving negative captions by replacing caption nouns with plausible but image-untrue alternatives.This provides a contrastive training signal beyond standard paired image-caption supervision.
- Related work: Existing approaches commonly cast weakly supervised phrase grounding as multiple instance learning over image regions and aggregate region-level features or scores.Reported aggregation methods include max or mean pooling, noisy-OR, and attention.
- Proposed formulation: The proposed formulation maximizes a lower bound on mutual information between image-region features and contextualized caption-word representations.The model optimizes word-region attention while using pretrained visual and textual representations.
- Contrastive training: InfoNCE training uses image- and language-based contrastive objectives to distinguish true image-word pairs from mismatched images or caption words.The two objectives contrast the same caption word across images and the same image across captions.
- Compatibility function: The compatibility function uses query-key-value attention to compute word-specific soft alignments between contextualized words and image regions.Queries and keys determine attention over regions, while attended visual and word representations determine compatibility.
2 Method
The method learns weakly supervised phrase grounding by maximizing an InfoNCE lower bound on mutual information between image regions and contextualized caption words. It uses word-specific query-key-value attention for soft region selection and language-model-guided context-preserving negative captions for contrastive training.
- Mutual-information objective: Word-region attention learns a soft grounding by downweighting nuisance regions and attending to regions sharing information with the word representation.The model optimizes attention parameters while using pretrained visual and textual representations.
- Mutual-information objective: The method maximizes an InfoNCE lower bound on mutual information between image region features and each contextualized caption word.The objective uses minibatches containing positive region-word pairs and negative pairs.
- Word-region attention: The compatibility function maps regions to keys and values, maps words to queries and values, and compares the attended visual representation with the word representation.Query-key attention produces word-specific region weights, whose weighted values form the attended visual representation.
- Contrastive objectives: The image-based objective contrasts positive region-word pairs with pairs formed by replacing image regions from a positive pair with regions from another minibatch instance.The formulation does not require every region to correspond to a caption word because captions describe only part of an image.
- Negative captions: Context-preserving negative captions retain the original context while substituting a noun that is plausible in context but makes the caption untrue for the image.BERT proposes candidates from the masked context, and reranking reduces synonyms or hypernyms and retains the top 25 captions as negatives.
- Negative captions: The proposed negative-caption procedure is empirically effective because selected candidates match the context while rejected candidates are often synonyms or hypernyms of the true noun.The procedure uses language-model probabilities and a proxy for how true a candidate remains given the original noun and context.
3 Experiments
The experiments compare the proposed approach with weakly supervised phrase-grounding methods, evaluate language-model components and negative-caption construction, and examine InfoNCE as a grounding proxy. The model gains 5.7% absolute pointing accuracy over Align2Ground, while contextually plausible negatives and pretrained word representations substantially improve performance.
- Experimental setup: The experiments compare the approach with state-of-the-art weakly supervised phrase-localization methods and analyze its language-model components and InfoNCE proxy.Evaluation uses COCO training data and Flickr30K Entities for validation and testing, with recall@k and pointing accuracy as metrics.
- Performance on Flickr30K Entities: 5.7% absolute gain in pointing accuracy over Align2Ground is achieved using the same training data and visual feature architecture.The comparison is made on the Flickr30K Entities test set.
- Benefits of language modeling: ∼10% absolute gain in both recall@1 and pointing accuracy results from using pretrained BERT word representations.BERT (Pretrained) supplies contextualized word features without finetuning, unlike the randomly initialized comparison.
- Benefits of language modeling: Contextually plausible negative captions outperform random sampling by ≥8% in recall@1 and pointing accuracy, while excluding near-synonyms and hypernyms adds ∼3 points.Negative construction first samples context-conditioned noun candidates and then filters likely synonyms or hypernyms that may also describe the image.
- InfoNCE as a proxy: InfoNCE lower-bound maximization correlates with phrase-grounding performance, but the best grounding occurs before the bound reaches its maximum.One- and two-layer MLP key-value-query modules peak faster and perform better than linear functions.
- Qualitative results: The learned word-region attention localizes related objects, distinguishes instances, identifies object parts, handles occlusion, and covers uncommon categories.These abilities are illustrated qualitatively in Fig. 5.
4 Conclusion
The paper formulates weakly supervised phrase grounding as mutual-information estimation between image regions and caption words, using attention to ground words in images. Language-model-generated context-preserving negative captions improve learning over random negatives.
- 4 Conclusion: The framework estimates mutual information between image regions and caption words from paired image-caption data.It optimizes a lower bound with respect to a region-word attention mechanism.
- 4 Conclusion: Attention learned by maximizing the mutual-information lower bound grounds words in images.
- 4 Conclusion: Language models generate context-preserving negative captions that improve learning compared with randomly sampled training captions.
- 4 Conclusion: Attention visualizations show attended regions for highlighted caption words alongside all detected regions and top-3 attention scores.
A.1 Limitations and Future Works
The authors identify reliance on pretrained representations, a fully labeled validation set, and a non-valid language-side mutual-information bound as limitations.
- Pretrained representations: The approach relies on pretrained object detectors and language models for region and caption-word representations.The authors would ideally learn or improve these representations directly from image-caption data.
- Need for fully-labeled validation set: Model selection requires early stopping based on a fully labeled validation set for phrase grounding.The Flickr30K Entities validation set is 80× smaller than the COCO training set, but still supplies full supervision for some images.
- Bounds on MI: The language-side objective log(K) −Llang is not a valid mutual-information lower bound because it oversamples caption-related negative words.Random caption sampling would give a valid bound, but context-preserving negatives perform better.
A.2 Advantages of Context-Preserving Negative Sampling
Randomly sampled negatives often differ too much in context to provide informative training signals, whereas single-word, plausible substitutions create harder negatives and improve pointing accuracy.
- Negative sampling: Random negative captions showed no significant gains over training without negative captions because their contexts were often entirely different.Such negatives are easy for the model to assign low compatibility scores.
- Context-preserving negatives: 76.74% vs. 66.89% pointing accuracy favors context-preserving over random negative sampling.The method substitutes one plausible, non-synonymous or non-hypernym word while preserving the positive caption’s context.
- Context-preserving negatives: Context-preserving construction can produce effective negatives for any positive caption, unlike random sampling, which depends on informative captions occurring in training data.
A.3 Relation between our query-key-value attention and self-attention in Transformers
The paper uses query-key-value attention for word-region alignment, contrasting it with Transformer self-attention, which contextualizes words through word-to-word attention.
- Self-attention in Transformers: Transformers compute queries, keys, and values for each input word and use attention scores for contextualization.
- Word-region alignment: The proposed mechanism computes queries for contextualized words, keys for regions, and values for region-word alignment.
A.4 Comparison to Align2Ground
Compared with Align2Ground, the model uses the same visual features but replaces its bi-GRU textual features with BERT. Pretraining substantially improves BERT-based grounding performance.
- A.4 Comparison to Align2Ground: The model matches Align2Ground’s visual features but uses BERT instead of Align2Ground’s bi-GRU textual features.BERT is described as a transformer-based language model, contrasting with RNN-based approaches.
- A.4 Comparison to Align2Ground: 66.89% grounding performance is achieved with pretrained BERT, compared with 57.37% for randomly initialized BERT.The comparison estimates the gain from pretrained language representations.
- A.4 Comparison to Align2Ground: The experiment isolates the contribution of pretrained language representations by comparing pretrained and randomly initialized BERT.