Source-linked AI summary
Grounding of Textual Phrases in Images by Reconstruction
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, Bernt Schiele
TL;DR
Grounding arbitrary textual phrases in images is difficult because spatial annotations are scarce and costly. GroundeR learns phrase localization through attention and phrase reconstruction across multiple supervision settings, and its supervised variant outperforms state-of-the-art on Flickr 30k Entities and ReferItGame.
Problem
Grounding arbitrary natural-language phrases in images is challenging, while many paired image-text datasets lack costly bounding-box annotations.
Method
GroundeR learns to ground phrases by attending to image regions and reconstructing the phrases from those automatically localized regions.
Results
GroundeR operates across unsupervised, semi-supervised, and supervised settings, with the supervised variant outperforming state-of-the-art on Flickr 30k Entities and ReferItGame.
Takeaways & Limitations
Phrase reconstruction provides a grounding approach that uses unlabeled data while incorporating available bounding-box supervision.
Abstract
from arXiv · showhide
Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text reference resolution. Few datasets provide the ground truth spatial localization of phrases, thus it is desirable to learn from data with no or little grounding supervision. We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly. During training our approach encodes the phrase using a recurrent network language model and then learns to attend to the relevant image region in order to reconstruct the input phrase. At test time, the correct attention, i.e., the grounding, is evaluated. If grounding supervision is available it can be directly applied via a loss over the attention mechanism. We demonstrate the effectiveness of our approach on the Flickr 30k Entities and ReferItGame datasets with different levels of supervision, ranging from no supervision over partial supervision to full supervision. Our supervised variant improves by a large margin over the state-of-the-art on both datasets.
1 Introduction
The paper targets grounding arbitrary natural-language phrases in images without costly bounding-box annotations. GroundeR learns attention by reconstructing phrases and supports unsupervised, semi-supervised, and supervised training.
- 1 Introduction: GroundeR addresses grounding arbitrary natural-language phrases rather than constrained sets of nouns.The approach focuses on fine-grained localization of phrases in images.
- 1 Introduction: The method can learn localization from phrases paired with images without bounding-box annotations.It also incorporates bounding-box supervision when annotations are available.
- 1 Introduction: During training, GroundeR attends to a bounding-box proposal and reconstructs the phrase from the selected region.Correct phrase prediction depends on attending to the relevant box, enabling weakly supervised learning.
- 1 Introduction: GroundeR operates with no, few, or all grounding annotations available.The paper evaluates these supervision settings on Flickr 30k Entities and ReferItGame.
- 1 Introduction: The unsupervised variant is better than prior work, while the supervised variant significantly outperforms state-of-the-art on both datasets.The semi-supervised variant can exploit small amounts of labeled data and multiple losses.
2 Related work
Prior grounding methods often target restricted nouns, object classes, or specially designed queries. GroundeR instead combines attention with phrase reconstruction to handle arbitrary and complex phrases.
- 2 Related work: Earlier image-grounding methods include graphical models, conditional random fields, and multiple-instance learning for aligning language with image regions.Several approaches use restricted representations such as nouns, dependency-tree relations, or scene-graph queries.
- 2 Related work: Object co-localization and web-trained detectors use weak image-level supervision but focus on discrete classes or concepts.These approaches do not address complex phrases such as “a blond boy on the left.”
- 2 Related work: GroundeR differs by targeting arbitrary phrases and allowing localization of novel phrases rather than only a few predefined classes.Its reconstruction objective is complementary to prior ranking objectives.
- 2 Related work: Attention mechanisms have been applied to captioning, video description, and action labeling by selecting relevant visual content over time or regions.GroundeR applies soft attention to select a bounding-box proposal for a given phrase.
3 GroundeR: Grounding by Reconstruction
GroundeR grounds arbitrary phrases by attending to image-region proposals and reconstructing the phrase from the attended visual features. The same framework supports unsupervised, semi-supervised, and fully supervised localization, with grounding attention used at test time.
- Core idea: GroundeR learns phrase grounding from bidirectional correspondence: a correctly selected image region should support reconstruction of the phrase describing it.The model trains grounding and reconstruction jointly, using reconstruction likelihood to guide attention when localization labels are unavailable.
- Architecture: During training, the grounding component attends to one or multiple bounding-box proposals, while the reconstruction component generates the input phrase from the attended boxes.Attention weights aggregate proposal visual features before an encoding layer and an image-description LSTM generates the phrase word sequence.
- Supervision modes: The framework spans no, little, and full localization supervision: semi-supervised training combines attention and reconstruction losses, while full supervision removes reconstruction.At test time, only the grounding part is used to select a region, whose overlap with the ground-truth box is evaluated using intersection over union.
- Architecture: The phrase encoder represents multi-word queries with an LSTM, and proposal regions are encoded with a CNN before a two-layer perceptron computes attention.The attention architecture combines the phrase representation with each proposal’s visual feature, using a ReLU nonlinearity.
- Learning to ground: Softmax converts proposal scores into normalized attention weights that can be interpreted as probabilities of regions being correct.When ground-truth localization is available, an attention loss directly optimizes the predicted correct proposal; otherwise reconstruction supplies the weak supervision.
4 Experiments
Experiments evaluate GroundeR across Flickr 30k Entities and ReferItGame under varying supervision, using proposal-based localization accuracy and qualitative comparisons. Results show strong gains from supervision, semi-supervision, and suitable visual features, while complex phrases and regions remain challenging.
- Experimental setup: GroundeR is evaluated on Flickr 30k Entities and ReferItGame using automatically generated box proposals and phrase-localization accuracy at more than 0.5 IOU.Flickr 30k Entities uses 100 Selective Search proposals per image, while ReferItGame uses 100 Edge Boxes proposals; test accuracy counts correctly overlapping attended boxes.
- Flickr 30k Entities: On Flickr 30k Entities, unsupervised accuracy reaches 29.02% with VGG-DET after sentence-level post-processing, compared with 24.66% for VGG-CLS.The corresponding unprocessed VGG-DET result is 28.94%, and the sentence constraint prevents selecting the same box twice for phrases from one sentence.
- Flickr 30k Entities: 47.81% accuracy with supervised VGG-DET improves over 43.89% from the prior VGG-DET approach on Flickr 30k Entities.GroundeR with VGG-CLS reaches 41.56%, while earlier methods and SCRC reach 27.42% and 27.80%, respectively.
- Flickr 30k Entities: Using only 3.12% annotated data raises accuracy from 28.94% to 42.32% with VGG-DET, while all annotations reach 48.38%.The semi-supervised model consistently improves over the supervised model using the same fraction of annotations, and reconstruction remains beneficial with all supervision.
- Qualitative results: Qualitative results show supervised localization visibly improves over unsupervised predictions, while ReferItGame examples include failures involving complex language and object relations.The paper identifies expressions such as “hut to the nearest left of the person on the right” as requiring additional relation modeling.
5 Conclusion
GroundeR addresses unconstrained phrase grounding across none, little, and full supervision by attending to box proposals and reconstructing phrases. It outperforms state-of-the-art on Flickr 30k Entities and ReferItGame.
- GroundeR localizes unconstrained natural phrases by attending to box proposals and reconstructing the phrase across multiple supervision settings.The settings include no, little, and full bounding-box supervision.
- 4.5% and 10.6%: GroundeR outperforms state-of-the-art on Flickr 30k Entities and ReferItGame, respectively.
- Semi-supervised GroundeR uses unsupervised data to outperform purely supervised training with the same amount of labeled data.
- GroundeR can extend beyond bounding-box proposals to other regions such as segmentation proposals.