Source-linked AI summary
Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
Raymond A. Yeh, Jinjun Xiong, Wen-mei W. Hwu, Minh N. Do, Alexander G. Schwing
TL;DR
Textual grounding needs to locate image regions described by language, but proposal-based systems depend on limited candidate boxes. The paper uses interpretable image concepts with efficient branch-and-bound search over all possible boxes, and reports improvements on two benchmarks while deriving interpretable word embeddings. Its fixed pretrained feature extractor remains a scope limitation relative to methods that fine-tune the full network.
Problem
Textual grounding is important for human-computer interaction, robotics, and knowledge mining, but existing systems are limited by their dependence on bounding-box proposals.
Method
The method combines image concepts with an energy formulation and efficiently searches all possible bounding boxes using branch-and-bound inference.
Results
7.77% and 3.08% improvements were reported on ReferItGame and Flickr 30k Entities, respectively, using IoU against state-of-the-art techniques.
Takeaways & Limitations
The learned parameters provide interpretable word embeddings that capture relationships between phrase words and image concepts.
Takeaways & Limitations
The approach uses a fixed pretrained network for image-feature extraction, whereas later methods fine-tune the entire network or train more feature detectors.
Abstract
from arXiv · showhide
Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.
1 Introduction
Textual grounding links phrases to image regions but is limited by proposal-based systems. This work instead searches bounding boxes globally using interpretable image concepts and reports stronger benchmark performance.
- Motivation: Textual grounding must connect phrases describing objects and relations to corresponding image bounding boxes for human-computer interaction, robotics, and knowledge mining.
- Motivation: Proposal-based methods are constrained because grounding quality depends on the automatically generated bounding boxes.
- Approach: The proposed method scores image concepts, including semantic segmentations, detections, and object priors, then searches all possible boxes for the highest-scoring region.
- Approach: Efficient subwindow search provides the branch-and-bound inference scheme, while learned weights can also function as word embeddings.
- Results: 7.77% and 3.08% improvements on ReferItGame and Flickr 30k Entities, respectively, were reported using IoU against state-of-the-art techniques.
2 Related Work
Prior grounding work spans retrieval, multimodal embeddings, structured language grounding, and proposal scoring. This paper distinguishes itself by using efficient subwindow search instead of relying primarily on region proposals.
- Prior grounding methods: Visual grounding research includes image retrieval, joint visual-semantic models, structured relation grounding, and proposal-based phrase localization.
- Datasets: ReferItGame and Flickr 30k Entities are common visual-grounding datasets, with Flickr 30k Entities providing noun-phrase bounding-box annotations.
- Efficient subwindow search: Unlike methods largely based on region proposals, this work proposes efficient subwindow search as the inference engine for textual grounding.
- Efficient subwindow search: Efficient subwindow search uses branch and bound and has been applied to object localization, detection, weakly supervised localization, and related vision tasks.
3 Exact Inference for Grounding
The method formulates textual grounding as globally optimal energy minimization over all bounding boxes, using interpretable image-concept score maps and efficient branch-and-bound inference. Its model combines query words with priors, geometry, segmentation, and detection features, while structured learning trains parameters against IoU-based loss.
- Exact inference: Global inference searches the full bounding-box product space and uses branch and bound to obtain a globally optimal prediction without exhaustive minimization.The output space is recursively decomposed, lower bounds guide subspace selection, and the process terminates when one bounding box remains.
- Energy function: The energy combines word-query indicators with image-concept score maps, using sparse word-to-concept weights to score accumulated evidence inside each candidate box.Image concepts include word priors, geometric cues, semantic segmentations, and detections; the learned weights connect words to concepts and speed inference through sparsity.
- Score maps: Word priors, aspect ratio, area, segmentation, and detection maps provide complementary information, while word priors and geometry remain independent of image specifics.Word priors are computed by averaging annotated box coverage over training occurrences, whereas image-based features come from deep neural networks.
- Inference: Branch-and-bound inference recursively splits coordinate intervals, evaluates lower energy bounds for subspaces, and selects the subspace with the smallest bound.Global convergence requires bounds that lower-bound every candidate energy and become exact for singleton bounding boxes; integral images enable constant-time bound evaluation.
- Learning the parameters: Structured SVM learning enforces lower energy for ground-truth boxes than alternatives with an IoU task loss, while fixing lower-layer parameters reduces training to a structured SVM.Loss-augmented inference remains structurally similar to energy inference, allowing the same bound computations.
- Learning the parameters: The approach can backpropagate through nonlinear neural-network parameters in principle, although those parameters were not trained this way for the reported results.This leaves the top-layer decomposition as a design choice for efficient inference rather than a fundamental restriction.
4 Experimental Evaluation
Experiments compare the approach with recent baselines on two grounding datasets, assess phrase types and learned word embeddings, and examine inference efficiency and limitations. The method outperforms competing methods overall while revealing interpretable word–image relationships and several practical constraints.
- Implementation: The experiments use restricted vocabularies covering about 90% of training phrases, with segmentation, detection, and pose-estimation maps supplying image features.The vocabularies contain the top 200 ReferItGame words and top 1000 Flickr 30k Entities words; remaining words share an additional token.
- Implementation: Integral images make lower-bound computation a lookup operation, supporting efficient search over the ReferItGame and Flickr 30k Entities datasets.The datasets contain more than 99,000 regions from 20,000 images and more than 275k bounding boxes from 31k images, respectively.
- Quantitative evaluation: Around 3% on Flickr 30k Entities and around 7% on ReferItGame, the approach outperforms recent state-of-the-art baselines at more than 0.5 IoU.The evaluation compares word priors, geometric information, segmentation maps, detection maps, and body-part detection variants.
- Phrase-type analysis: The system outperforms the state-of-the-art in every Flickr 30k Entities phrase type except clothing.
- Limitations: Performance is limited when stronger competitors fine-tune the full feature network, while word priors can hurt and additive linear scoring can fail for phrases such as “dirt bike.”
- Learned parameters + word embedding: Large word–concept weights and clustered word vectors show that learned embeddings capture relationships between phrase words, image concepts, and spatially interchangeable objects.Examples include ship–boat weights and similarity groups such as bicycle–bike, camera–cellphone, and snowboarder–skier.
- Computational Efficiency: Inference speed is comparable to CCA and much faster than GroundeR, with one image-feature pass, negligible language lookup time, and 1.05 ms score computation on a CPU.The reported image-feature extraction time is 142.85 ms for the approach, while GroundeR requires a forward pass for each image region.
5 Conclusion
The paper presents textual grounding as an interpretable, extensible mechanism with globally optimal inference by searching over all possible bounding boxes rather than a small proposal set.
- Searching over all possible bounding boxes enables globally optimal inference instead of restricting grounding to a small proposal set.
- The mechanism links words with image concepts, making textual grounding interpretable and supporting learned parameters as word embeddings for spatial-image relationships.
- The approach is designed to be easy to extend while avoiding dependence on successful first-stage region-proposal hypotheses.