Source-linked AI summary
A Fast and Accurate One-Stage Approach to Visual Grounding
Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, Jiebo Luo
TL;DR
Existing two-stage visual grounding methods are limited by their region candidates, motivating an end-to-end one-stage alternative. The paper fuses text and spatial features into YOLOv3, achieving faster and more accurate grounding across phrase localization and referring expression comprehension. The authors therefore advocate one-stage grounding as a new baseline and paradigm.
Problem
Two-stage grounding can fail when its region proposals miss the ground-truth region, while much computation is spent generating and ranking candidates.
Method
The model fuses a text query embedding and spatial features into the YOLOv3 object detector for end-to-end grounding.
Results
About 10 times faster than state-of-the-art two-stage methods and more accurate on phrase localization and referring expression comprehension.
Takeaways & Limitations
The authors advocate one-stage visual grounding as a paradigm shift and a new strong baseline for future work.
Takeaways & Limitations
The model is insensitive to some attributes, fails on very difficult cases, and can disagree with user annotations for ambiguous queries.
Abstract
from arXiv · showhide
We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight. The performances of existing propose-and-rank two-stage methods are capped by the quality of the region candidates they propose in the first stage --- if none of the candidates could cover the ground truth region, there is no hope in the second stage to rank the right region to the top. To avoid this caveat, we propose a one-stage model that enables end-to-end joint optimization. The main idea is as straightforward as fusing a text query's embedding into the YOLOv3 object detector, augmented by spatial features so as to account for spatial mentions in the query. Despite being simple, this one-stage approach shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension, according to our experiments. Given these results along with careful investigations into some popular region proposals, we advocate for visual grounding a paradigm shift from the conventional two-stage methods to the one-stage framework.
1. Introduction
The paper argues that conventional propose-and-rank visual grounding is limited by region proposals and computationally wasteful. It introduces an end-to-end one-stage model that fuses language and spatial information into YOLOv3, reporting faster and more accurate grounding across two tasks.
- Motivation: Visual grounding localizes a natural-language query about an image onto a correct image region.The framework covers phrase localization and referring expression comprehension as variations of visual grounding.
- Results: The model is reported to be about 10 times faster than state-of-the-art two-stage methods while achieving greater accuracy on phrase localization and referring expression comprehension.The authors attribute its advantages to one-pass inference and end-to-end optimization.
- Limitations of two-stage methods: Existing two-stage methods generate region candidates and rank them by similarity to the language query.Their performance depends on the first-stage proposals before ranking occurs.
- Limitations of two-stage methods: 200 Edgebox proposals per image hit only 68% of ReferItGame ground-truth regions at IoU ≥ 0.5.If no candidate reaches the ground-truth region, the ranking stage cannot recover the correct region.
- Proposed approach: The proposed one-stage model fuses a text query embedding into YOLOv3 and augments feature maps with spatial features for spatial mentions.A softmax output enforces generation of one image region per query.
- Implications: The authors advocate a shift toward one-stage visual grounding as a new strong baseline for future work.They propose extending the framework with domain-specific cues such as attributes, phrase relationships, and spatial configurations.
2. Approach
The approach replaces candidate generation and ranking with an end-to-end YOLOv3-based grounding network. It fuses visual, text, and spatial features across multiple resolutions and predicts one grounding box using anchor-based outputs.
- Two-stage methods: Conventional visual grounding uses a first stage to generate candidate regions and a second stage to rank them conditioned on a language query.Candidate regions may come from unsupervised proposal methods or pretrained object detectors.
- Two-stage methods: The two-stage framework incurs heavy computation and is capped by the quality of its candidate regions.The paper motivates a one-stage alternative to enable end-to-end optimization.
- One-stage model: The one-stage model fuses query embeddings into YOLOv3, adds spatial features, replaces sigmoid with softmax, and trains with YOLO’s loss.The softmax selects one region from the model’s predicted boxes.
- Feature encoding: The network takes an image and text query, encodes text with BERT, and extracts visual features using Darknet-53 with feature pyramids at three resolutions.Visual feature maps are mapped to a common 512-dimensional representation.
- Spatial encoding: Spatial features explicitly encode coordinate information because visual features primarily capture appearance and lack position information.Eight spatial features describe grid coordinates at each of the 8×8, 16×16, and 32×32 resolutions.
- Grounding output: The grounding module follows YOLOv3’s output layer, recalibrates anchor boxes, and uses a softmax over 4032 boxes.The target anchor is selected by highest IoU with the ground-truth region and trained against a one-hot label.
- Related approaches: Compared with related one-stage methods, the proposed framework is designed for supervised grounding with YOLOv3-style anchor-based prediction.The paper contrasts it with IGOP, MATN, and segmentation-oriented one-stage frameworks.
3. Experiments
Experiments evaluate the one-stage model on Flickr30K Entities and ReferItGame, compare architectural and proposal baselines, and analyze region-candidate coverage and qualitative successes and failures.
- Evaluation: The experiments evaluate visual grounding on Flickr30K Entities and ReferItGame, with additional RefCOCO results reported in supplementary materials.The evaluation covers phrase localization and referring expression comprehension.
- Baselines and variants: The study compares state-of-the-art methods, similarity-network variants, and one-stage model variations using different region proposals, visual features, and language embeddings.The comparisons include a Darknet-based similarity network and ablations such as removing spatial features.
- Visual grounding results: ResNet-based similarity features outperform VGG-based features, whereas Darknet features produce poor two-stage grounding results despite comparable ImageNet performance to ResNet-101.The paper attributes the Darknet result to difficulties extracting discriminative region features without a separate proposal network.
- Visual grounding results: The one-stage method and its variations outperform the two-stage approaches with large margins on Flickr30K Entities and ReferItGame.On Flickr30K Entities, spatial features improve the no-spatial variant by about 1.6%, while language embeddings have only a small effect.
- Efficiency: The one-stage approach grounds a query in under 40ms, with the Fisher-vector variant requiring 16ms and potentially supporting real-time applications.Two-stage methods spend substantial computation generating region candidates and extracting their features.
- Qualitative analysis: Qualitative failures of the one-stage model include insensitivity to attributes, difficult instances, and ambiguous queries with annotations that differ from its predictions.The two-stage method commonly fails on multiple-object, stuff-region, and challenging-region queries that the one-stage method can correct.
4. Conclusion
The paper concludes that its one-stage visual grounding model combines language and spatial features with YOLOv3, achieving faster inference and superior grounding accuracy while motivating a shift from two-stage methods.
- The model merges language queries and spatial features into YOLOv3 to create an end-to-end trainable visual grounding system.
- 10 times faster inference than state-of-the-art two-stage methods is reported for the proposed approach.
- The approach achieves superior grounding accuracy and provides a framework intended for modeling additional visual grounding cues.
Appendix A. Results on RefCOCO
On RefCOCO, one-stage grounding outperforms most state-of-the-art methods, but the smaller improvement reflects unusually strong COCO-trained region proposals; end-to-end proposal finetuning improves two-stage methods without surpassing the proposed approach.
- Ours-LSTM outperforms state-of-the-art methods except MAttNet, which uses extra supervision including proposal attributes and class labels.
- 1.4% improvement on RefCOCO is smaller than the 25.0% and 7.8% improvements reported on ReferItGame and Flickr30K Entities.
- 92.4% detection and 98.5% region-proposal hit rates on RefCOCO indicate that COCO-trained proposals cover almost all referred objects.
- RefCOCO’s shared COCO images and objects make its strong proposal quality a special case that does not directly extend to other datasets.
- 44.07% for QRC Net versus 59.30% for Ours on ReferItGame, and 65.14% versus 68.69% on Flickr30K Entities, show that end-to-end RPN finetuning improves two-stage methods but remains below the proposed approach.
Appendix B. Cross-Sample Relationships
The extended model uses relationships among image-query samples by grouping same-object queries as positives and different-region or different-image queries as negatives, then applies feature regularization.
- Cross-sample relationships are added to the vanilla framework to exploit multiple phrases and queries associated with visual grounding samples.
- Positive bags contain different queries describing the same object in the same image, while negative bags contain different regions or different images.
- The method assumes fused visual-textual features should be more similar to positive than negative samples.
- A triplet loss enforces this relationship using sampled positive and negative image-query pairs and a fused visual-textual feature.
- Performance improves on all three datasets with margin m = 1 and regularization weight w_reg = 1, while hard triplet generation produces no major change.