Source-linked AI summary
Improving One-stage Visual Grounding by Recursive Sub-query Construction
Zhengyuan Yang, Tianlang Chen, Liwei Wang, Jiebo Luo
TL;DR
One-stage visual grounding methods can miss detailed information in long and complex queries when the query is compressed into a single sentence vector. The paper introduces recursive sub-query construction, which repeatedly reasons between image and query to refine visual-text features and reduce ambiguity. It reports absolute gains of 5.0%, 4.5%, 7.5%, and 12.8% over the state-of-the-art one-stage baseline across four datasets while maintaining real-time speed.
Problem
Single-vector query representations in existing one-stage methods can overlook important referring details, particularly in long and complex queries.
Method
The framework alternates a sub-query learner and a sub-query modulation network across multiple rounds to refine text-conditional visual features.
Results
5.0%, 4.5%, 7.5%, and 12.8% absolute improvements over the state-of-the-art one-stage method are reported on ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg, respectively.
Takeaways & Limitations
Recursive query modeling improves one-stage grounding on multiple datasets while maintaining real-time inference speed.
Abstract
from arXiv · showhide
We improve one-stage visual grounding by addressing current limitations on grounding long and complex queries. Existing one-stage methods encode the entire language query as a single sentence embedding vector, e.g., taking the embedding from BERT or the hidden state from LSTM. This single vector representation is prone to overlooking the detailed descriptions in the query. To address this query modeling deficiency, we propose a recursive sub-query construction framework, which reasons between image and query for multiple rounds and reduces the referring ambiguity step by step. We show our new one-stage method obtains 5.0%, 4.5%, 7.5%, 12.8% absolute improvements over the state-of-the-art one-stage baseline on ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg, respectively. In particular, superior performances on longer and more complex queries validates the effectiveness of our query modeling.
1 Introduction
Previous one-stage visual grounding methods struggle with long and complex queries because single-vector query representations can overlook detailed referring information. The paper proposes recursive sub-query construction to reduce referring ambiguity through repeated image-query reasoning, improving results across multiple datasets.
- Motivation: Existing one-stage methods encode the entire query as a single embedding vector, increasing representation ambiguity.Examples include BERT’s [CLS] embedding and aggregated LSTM hidden states.
- Motivation: Single-vector modeling can focus on some words while overlooking important descriptions, especially in long and complex queries.The paper identifies this as a source of lost referring information.
- Approach: The framework recursively constructs sub-queries and refines text-conditional visual features through multiple rounds of image-query reasoning.A sub-query learner and a sub-query modulation network operate alternately to reduce referring ambiguity.
- Results: 5.0%, 4.5%, 7.5%, and 12.8% absolute improvements are obtained over the state-of-the-art one-stage method on ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg, respectively.The method also runs at 38 FPS (26ms).
- Results: The authors report significantly improved results on multiple datasets while maintaining real-time inference speed.Extensive experiments and ablations are presented as validation of the method.
2 Related Work
Visual grounding methods are commonly divided into phrase localization and referring expression comprehension, with one-stage methods offering dense prediction without explicit region proposals. Query attention provides an alternative query model, but the paper reports stronger performance from its multi-round solution.
- Two-stage methods: Visual grounding includes phrase localization and referring expression comprehension, with many earlier methods using proposal generation followed by query-region similarity scoring.Two-stage systems may use EdgeBox, selective search, or pretrained detectors for proposals.
- One-stage methods: One-stage methods densely fuse visual-text features at image locations and directly predict bounding boxes instead of explicitly extracting proposal-region features.This paradigm targets proposal sparsity and slow region-feature computation in two-stage methods.
- Query modeling: The paper compares multi-round sub-query modeling with attention-based query alternatives and observes better performance for the proposed solution.The comparison refers to the “Single/ Multi-head attention” and “Sub-query learner (ours)” alternatives in Table 3.
3 Approach
The framework recursively constructs visual-conditioned sub-queries and uses them to refine text-conditional visual features over multiple rounds before predicting grounding boxes.
- Query Modeling: Previous one-stage methods encode the query as one vector, which can overlook important words and increase referring ambiguity.This weakness is especially relevant to long and complex queries.
- Recursive Framework: The framework starts from the image feature v(0) and recursively refines v(k) through K rounds before box prediction.Each round produces a new attended sub-query and updates the text-conditional visual feature.
- Sub-query Learner: The sub-query learner uses query-word features, the current pooled visual feature, and attention history to predict word-attention scores α(k).Referencing the current visual feature makes sub-query construction responsive to the image, while history helps avoid overemphasizing keywords.
- Sub-query Learner: Diversity and coverage regularization encourages different rounds to attend to different words while collectively examining the query.Ldiv discourages repeated word focus, whereas Lcover encourages attention to all query words.
- Sub-query Modulation: The modulation network converts q(k) into scaling and shifting vectors that refine v(k−1), whose final representation feeds the grounding module.The grounding module predicts bounding boxes from v(K); the design uses shared modulation vectors across spatial locations.
4 Experiments
Experiments compare the proposed one-stage grounding method with existing methods across multiple datasets, query lengths, and query-modeling alternatives. The method improves accuracy, remains fast, and benefits from recursive sub-query construction for detailed and ambiguous queries.
- The evaluation covers RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities, with results reported using Acc@0.5%.
- Quantitative results: 38 FPS (26ms) inference speed provides a real-time efficiency advantage for the proposed method.The larger model configuration improves accuracy but slightly slows inference speed.
- Quantitative results: Query-length breakdowns compare Ours-Base with One-Stage-BERT across four query-length subsets on four datasets.Ours-Base is used because its inference speed is more comparable with One-Stage-BERT.
- Qualitative results: Qualitative visualizations show successive sub-queries refining predictions from head nouns toward location and relationship details, suppressing distracting objects.In one example, “persons head,” “in the middle,” and “with drill” progressively narrow the prediction to the referred person.
5 Conclusions
The paper proposes recursive sub-query construction to address complex-query limitations in one-stage visual grounding, with strong accuracy gains while maintaining real-time inference speed.
- The recursive framework constructs sub-queries and refines visual-text features for grounding box prediction.
- The method significantly outperforms state-of-the-art one-stage methods by over 5% absolute accuracy across multiple datasets.
- The framework maintains a real-time inference speed despite its improved grounding performance.
A Qualitative Results
Qualitative results show recursive sub-queries progressively disambiguate targets, including challenging scenes and attribute-based queries, while also revealing failures on missed objects and rare attributes.
- Recursive disambiguation: The text-conditional visual feature progressively produces more accurate and confident predictions through recursive disambiguation.Better modeling of head-noun modifiers corrects some previous failures.
- Recursive disambiguation: The method handles tiny targets, visually similar distractors, and queries with complex attributes and relationships.
- Attributes: Attribute reasoning can shift the heatmap peak from a distracting trolley to the referred red suitcase using the sub-query “red.”
- Failure cases: Failure cases arise when the model misses related objects or fails to understand rarely appearing attributes such as “plaid.”Distracting objects can retain high final-round heatmap responses, even when the predicted bounding box is correct.
B Ablation Studies
Ablation studies examine the number of reasoning rounds, larger model settings, attribute-focused performance, and qualitative visualizations of recursive sub-query behavior.
- Number of rounds: Increasing the number of rounds stops improving accuracy after a dataset-specific threshold, so K = 3 balances efficiency and accuracy by default.Different datasets may benefit slightly from another K or achieve lower inference time.
- Ours-Large: Ours-Large improves accuracy through larger input images and ConvLSTM integration of intermediate text-conditional visual features, with increased complexity or slower inference.
- Performance break-down with attributes: Attribute-subset analysis evaluates queries containing color, location, or size keywords against the full dataset.
- Performance break-down with attributes: Relative gains on the attribute subsets are around 15% and exceed the gains reported for the full datasets.
- Qualitative visualizations: Figure 6 visualizes constructed sub-queries and intermediate text-conditional visual features across rounds, comparing predicted regions with ground truths and highlighting targets versus distractors.
- Qualitative visualizations: Figure 7 provides additional qualitative results.