Source-linked AI summary
Query-guided Regression Network with Context Policy for Phrase Grounding
Kan Chen, Rama Kovvuri, Ram Nevatia
TL;DR
Phrase grounding localizes image objects referred to by phrases, while proposal-ranking approaches can be limited by proposal coverage and fail to use description context. QRC Net combines proposal generation, query-guided regression, and context-policy rewards; it reports accuracy increases on both Flickr30K Entities and Referit Game.
Problem
Proposal-ranking methods can be bounded by independent proposal generators’ coverage and leave description context unused for reducing semantic ambiguity.
Source: §1 · p. 1Method
QRC Net combines PGN proposal generation, QRN query-conditioned regression, and a CPN that uses reinforcement-learning rewards based on context.
Source: §1 · p. 2Results
QRC Net reports accuracy increases of 14.25% on Flickr30K Entities and 17.14% on Referit Game.
Source: §5 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Given a textual description of an image, phrase grounding localizes objects in the image referred by query phrases in the description. State-of-the-art methods address the problem by ranking a set of proposals based on the relevance to each query, which are limited by the performance of independent proposal generation systems and ignore useful cues from context in the description. In this paper, we adopt a spatial regression method to break the performance limit, and introduce reinforcement learning techniques to further leverage semantic context information. We propose a novel Query-guided Regression network with Context policy (QRC Net) which jointly learns a Proposal Generation Network (PGN), a Query-guided Regression Network (QRN) and a Context Policy Network (CPN). Experiments show QRC Net provides a significant improvement in accuracy on two popular datasets: Flickr30K Entities and Referit Game, with 14.25% and 17.14% increase over the state-of-the-arts respectively.
1. Introduction
Phrase grounding localizes objects referred to in image descriptions, but proposal-ranking methods can miss objects not covered by independent generators and leave description context unused. QRC Net combines query-guided regression with context-based rewards, and is reported to improve accuracy on Flickr30K Entities and Referit Game.
1. Introduction
- Independent proposal-ranking methods may miss mentioned objects and do not use description context to reduce semantic ambiguity.Because they select from a fixed proposal set, retrieval methods are bounded by the proposal generator’s coverage.Source: §1 · p. 1
- QRC Net predicts object locations through query-guided regression and uses context-dependent rewards, estimated with reinforcement learning, to refine predictions.It assumes different phrases in a sentence refer to different visual objects and down-weights predictions covering objects named by other phrases.Source: §1 · p. 2
- The framework jointly trains PGN to generate proposals, QRN to score and regress them conditioned on the query, and CPN to evaluate predictions using context rewards.At test time, CPN is fixed while trained PGN and QRN ground objects for different queries.Source: §1 · p. 2
- QRC Net outperforms state-of-the-art methods, with more than 14% accuracy increase on Flickr30K Entities and 17% on Referit Game.The evaluation uses both grounding datasets.Source: §1 · p. 2
- The paper identifies query-guided regression and reinforcement learning for context as its two contributions.The stated aims are to overcome independent proposal-generation limits and reduce semantic ambiguity.Source: §1 · p. 2
2. Related Work
Related work frames phrase grounding around matching language with visual regions, generating proposals, and using reinforcement learning to guide selection. QRC Net draws on proposal generation and regression while adding query conditioning and context-based policy rewards.
2. Related Work
- Prior grounding methods learn language–visual correlations through subspace matching, recurrent ranking, attention, or structured matching, but rely on external proposal systems.The cited examples include Karpathy et al. [16], Hu et al. [14], Rohrbach et al. [29], Plummer et al. [24] [25], and Wang et al..Source: §2 · p. 2
- Proposal-generation work spans Selective Search, EdgeBoxes, and integrated systems such as RPN [28]; QRC Net regresses proposals conditioned on the query.The cited progression also includes Fast R-CNN, Redmon et al. [27], and Liu et al. [20].Source: §2 · p. 2
- Prior reinforcement-learning applications include sampling discriminative expressions [37] and traversing semantic action graphs [18]; QRC Net uses policy-gradient rewards to leverage context during training.The related work also cites DQN and deep deterministic policy gradients [19].Source: §2 · p. 2
- Figure 2 presents QRC Net as PGN proposal generation, QRN query-conditioned regression, and CPN rewards based on foreground, background, or context.PGN uses RoI pooling [28], while QRN encodes query semantics with an LSTM [13].Source: Figure 2 · p. 3
3. QRC Network
QRC Net combines proposal generation, query-guided regression, and a context policy to localize objects from image-query pairs. Its framework trains the components to rank and refine proposals while using other phrases in the description as context.
3. QRC Network
- QRC Net combines PGN candidate generation, QRN regression and ranking, and CPN context-based refinement of ranking results.The model grounds one query phrase at a time and treats other phrases in the sentence as context referring to different objects.Source: §3 · p. 3
3.1. Framework
- The framework has PGN generate proposals, QRN predict their query relevance and regression parameters, and CPN evaluate them against context-mentioned objects using reward-based policy gradients.The objective combines PGN generation loss, QRN classification and regression losses, and CPN's expected reward.Source: §3.1 · p. 3
3.2. Proposal Generation Network (PGN)
- PGN uses an image feature map and anchors with multiple scales and aspect ratios to generate proposals, then selects top anchors by objectness and extracts proposal features for QRN.Training labels proposals positive above IoU 0.7 and negative below IoU 0.3.Source: §3.2 · pp. 3–4
3.3. Query guided Regression Network (QRN)
- QRN encodes the query with an LSTM, combines it with each proposal's visual feature, and predicts both query relevance and bounding-box regression parameters.Its regression parameters specify offsets from the proposal to the query-mentioned object's location.Source: §3.3 · p. 4
3.4. Context Policy Network (CPN)
- CPN rewards top-ranked proposals that overlap the query object more than proposals overlapping context objects, then uses policy gradients to update QRN.The reward is evaluated on QRN's top-ranked proposals; reward prediction loss trains CPN while CPN's policy gradients optimize QRN.Source: §3.4 · pp. 4–5
3.5. Training and Inference
- Training proceeds from PGN pretraining through QRN and CPN training, followed by one iteration of PGN tuning; at test time, the highest-relevance regressed proposal is selected.The described setup uses 100 PGN proposals and assigns rewards to QRN's top 10; CPN is fixed during testing.Source: §3.5 · p. 5
4. Experiment
QRC Net is evaluated on Flickr30K Entities and Referit Game, with comparisons covering proposal generators, model variants, and qualitative grounding examples. The reported results include a 14.25% accuracy increase over CCA embedding on Flickr30K Entities and incremental gains from QRN and PGN on Referit Game.
4.1. Datasets; 4.2. Experiment Setup
- Flickr30K Entities has 29,783 training images and 276K annotated boxes, while Referit Game has 19,894 images and 130,525 query phrases.Accuracy is the ratio of phrases whose regressed box overlaps the mentioned object by more than 50% IoU; comparisons include GroundeR, CCA embedding, MCB, Structured Matching, and SCRC.Source: §4.1 · p. 5, §4.2 · pp. 5–6
4.3. Performance on Flickr30K Entities; 4.4. Performance on Referit Game
- 14.25% increase over CCA embedding [25] in accuracy is reported for QRC Net on Flickr30K Entities; on Referit Game, QRN and PGN also improve accuracy over comparison models.On Referit Game, QRN improves accuracy by 5.28% over EB+GroundeR, PGN+QRN brings an 11.36% increase, and CPN adds 0.50%.Source: Table 1 · p. 6, §4.3 · p. 6, Table 7 · p. 8
4.3. Performance on Flickr30K Entities
- PGN improves both proposal coverage and the average number of proposals per ground-truth object, while QRN regression can surpass PGN’s upper-bound performance.Upper-bound performance is the proportion of ground-truth objects covered by proposals; coverage is counted at IoU > 0.5.Source: §4.3 · pp. 6–7, Table 2 · p. 6
- On Flickr30K Entities, performance fluctuates when the regression-loss weight is small and declines when it becomes large; reward values near 1.0 also reduce performance.Performance changes with multimodal dimension m are small below 1000 and decline at larger values; near β = 1.0, CPN rewards for ground-truth and context objects become similar and confuse QRN.Source: Table 3 · p. 6, Table 4 · p. 6, Table 5 · p. 6
4.4. Performance on Referit Game
- On Referit Game, performance fluctuates at small regression-loss weights and decreases at large weights, while it changes little for m < 1000 and declines for m > 1000.At large λ, regression loss can outweigh classification loss, leading to selection of a wrong seed proposal and incorrect grounding.Source: §4.4 · p. 8, Table 8 · p. 8, Table 9 · p. 8
4.5. Qualitative Results
- Qualitative examples show successful grounding of people and clothing, but an ambiguous query such as “hat on the right” can produce an incorrect result.The examples distinguish ground-truth boxes, selected proposal boxes, and regressed boxes by blue, green, and red, respectively.Source: Figure 3 · p. 7, §4.5 · p. 8
5. Conclusion
QRC Net uses regression and context information for phrase grounding, improving accuracy on Flickr30K Entities and Referit Game.
5. Conclusion
- 14.25% and 17.14% accuracy increases over prior methods are reported on Flickr30K Entities and Referit Game, respectively.The conclusion attributes these gains to QRC Net's regression mechanism and use of context information.Source: §5 · p. 8
- QRC Net adopts a regression mechanism for phrase grounding.Source: §5 · p. 8
- QRC Net leverages context information in phrase grounding.Source: §5 · p. 8