Source-linked AI summary
Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, Anton van den Hengel
TL;DR
The paper targets referring-object localization when fixed class labels cannot specify the subject, including cases requiring interactive disambiguation. It introduces PLAN, which recurrently combines global image attention with proposal-level attention over stepwise language, and reports state-of-the-art performance across benchmark datasets. The recurrent process also makes the reasoning updates visualizable and explainable.
Problem
Fixed class labels and pre-prepared training images restrict object detection when subjects must be specified at test time or clarified interactively.
Method
PLAN recurrently processes variable-length language stepwise while attending to global image context and candidate proposal regions in parallel.
Results
PLAN produces state-of-the-art performance on several benchmarked datasets spanning phrase- and dialog-based referring expressions.
Takeaways & Limitations
Recurrent parallel attention supports object discovery across short queries and long dialogs while making the referring process visualizable and explainable.
Takeaways & Limitations
RefCOCOg comparisons are not directly comparable because the paper uses the standard “google” split while prior work uses a random object partition.
Abstract
from arXiv · showhide
Recognising objects according to a pre-defined fixed set of class labels has been well studied in the Computer Vision. There are a great many practical applications where the subjects that may be of interest are not known beforehand, or so easily delineated, however. In many of these cases natural language dialog is a natural way to specify the subject of interest, and the task achieving this capability (a.k.a, Referring Expression Comprehension) has recently attracted attention. To this end we propose a unified framework, the ParalleL AttentioN (PLAN) network, to discover the object in an image that is being referred to in variable length natural expression descriptions, from short phrases query to long multi-round dialogs. The PLAN network has two attention mechanisms that relate parts of the expressions to both the global visual content and also directly to object candidates. Furthermore, the attention mechanisms are recurrent, making the referring process visualizable and explainable. The attended information from these dual sources are combined to reason about the referred object. These two attention mechanisms can be trained in parallel and we find the combined system outperforms the state-of-art on several benchmarked datasets with different length language input, such as RefCOCO, RefCOCO+ and GuessWhat?!.
1. Introduction
The paper addresses referring-object discovery when fixed class labels are insufficient, proposing PLAN to resolve variable-length expressions through recurrent, region-wise and step-wise reasoning.
- 1. Introduction: The framework processes language stepwise and images region-wise, progressively narrowing candidate regions as additional description becomes available.For example, “woman” considers multiple candidates, “in the middle” removes border regions, and “wearing brown jacket” leaves one region.
- 1. Introduction: PLAN recurrently discovers referred objects from short phrases through long multi-round dialogs, rather than relying on a fixed object-class inventory.This supports test-time specification of instance-level subjects and subjects not anticipated during training.
- 1. Introduction: PLAN combines image-level attention over global context with proposal-level attention over candidate regions to ground expressions involving scene context and multiple objects.The two attention sources are used together to localize the referred region and select the relevant proposal.
- 1. Introduction: The recurrent referring process makes intermediate updates across expression or dialog steps visualizable and explainable.The paper presents this interpretability as a side contribution of the recurrent design.
2. Related work
Related work distinguishes referring-expression comprehension from generation and situates it among methods using local, global, and relational visual information, alongside dialog-based vision-language tasks.
- 2. Related work: Referring-expression comprehension localizes regions described by an expression, whereas generation produces an expression for a given region.Comprehension scores extracted candidate regions against the referring expression and selects the highest-scoring grounding.
- 2. Related work: Prior comprehension methods use local or global visual features and increasingly model relationships or appearance comparisons among candidate objects.These approaches address inter-object relationships and surrounding-candidate information when grounding expressions.
- 2. Related work: The paper extends sentence- or phrase-based referring expressions toward dialogs, connecting the task to visual question answering and Visual Dialog.Visual Dialog requires an agent to participate intelligently in a dialog about an image.
3. The PLAN Model
PLAN is a unified referring-expression model that recurrently combines image-level and proposal-level attention to localize a referred object from language descriptions.
- 3. The PLAN Model: PLAN takes an image, object proposals, and sequential language units as input, then outputs a probability distribution over proposals and selects the highest-probability grounding.Language units are words for sentence descriptions or question-answer pairs for dialogs; GuessWhat?! additionally uses candidate categories.
- 3.1. Feature encoding: PLAN supports words, sentence descriptions, and dialog units, using an additional LSTM encoder for question-answer pairs and directly mapping single words through an MLP.Proposal features combine CNN appearance, normalized eight-dimensional box geometry, and category information only for GuessWhat?! experiments.
- 3. The PLAN Model: The model recurrently discovers the target by updating attended visual information as it listens to successive expression units.This stepwise process narrows potential regions as more language information is incorporated.
- 3.2.1 Image-level attention: Image-level attention uses expression features and the previous LSTM state to attend over spatial regions of the global convolutional image feature.The attended image feature is fed back with the current language representation to update the LSTM state, whose final state represents image-level attention.
- 3.2.2 Proposal-level attention: Proposal-level attention recurrently weights candidate object regions using the language sequence, with final attention weights producing attended proposal features.Unlike image-level attention over dense spatial regions, proposal-level attention operates only on regions containing candidate objects.
- 3.2.3 Referring process: The two attended representations are fused by a dot product and softmax to estimate each proposal’s referring probability, optimized with cross-entropy loss.The framework’s two branches therefore combine global contextual cues with candidate-specific evidence.
4. Experiments
The experiments evaluate PLAN and its components on referring-expression and dialog datasets, showing improvements from recurrent image- and proposal-level attention and stronger gains from their joint optimization.
- Experimental setup: The evaluation covers RefCOCO, RefCOCO+, RefCOCOg, and GuessWhat?!, using annotated entities as proposal boxes for fair comparison with previous methods.RefCOCO, RefCOCO+, and RefCOCOg use different expression lengths and evaluation splits, while all datasets are collected on MS-COCO images.
- Evaluation on RefCOCO, RefCOCO+ and RefCOCOg datasets: Adding image-level and proposal-level attention separately improves the baseline, while jointly optimizing both attention mechanisms yields further gains.The authors attribute the joint improvement to consistency between the attended image region and attended proposals; proposal-level attention has the more obvious separate effect.
- Evaluation on RefCOCO, RefCOCO+ and RefCOCOg datasets: Nearly 5% improvement on RefCOCO+ TestA over the listener model in [45] is reported for PLAN, while the final single model exceeds [45]’s ensemble.The RefCOCOg comparison is not directly comparable because the paper uses the standard ‘google’ split, whereas [45] randomly partitions objects into training and validation splits.
- Evaluation on the GuessWhat?! dataset: PLAN improves steadily over baseline methods on GuessWhat?!, with both image-level and proposal-level attention contributing to the final performance increase.The GuessWhat?! evaluation reports results with category features for fair comparison against the guesser model in [6].
- Evaluation on the GuessWhat?! dataset: 5.0% relative test-set improvement is observed for the recurrent encoding baseline over the best LSTM model in [6] when category features are removed.The comparison targets a flat-sequence dialog encoder and is reported in the no-category-feature setting.
- Recurrent parallel attention visualization: Qualitative visualizations show attention progressively shifting toward the referred object as dialog clues accumulate, although proposal-level attention can fail to track the target in a failure case.The image-level attention and final prediction can remain correct even when the proposal-level attention trend becomes less aligned with the target.
5. Conclusion
PLAN recurrently discovers referred objects from variable-length language descriptions using parallel attention, achieving state-of-the-art performance on several benchmark datasets while making reasoning visualizable.
- The proposed parallel attention network recurrently discovers referred objects from phrases to dialogs and achieves state-of-the-art performance on several benchmark datasets.The framework localizes the referring object using two-way attention over global contextual features and region proposals.
- Figure 3 visualizes recurrent parallel-attention changes during inference, including image-level attention maps and proposal-level attention over the top-5 proposals.The ground-truth answer is marked, and the last row is identified as a failure case.