Recursive Visual Attention in Visual Dialog

1. Introduction

• The introduction situates the problem within a broader field that has advanced across vision-language tasks, while comprehension in continuous human-agent interaction remains unresolved.The cited task examples include visual relation detection, image captioning, referring expression grounding, and VQA.Source: §1 · p. 1
• Visual co-reference resolution is a key challenge in multi-round visual dialog: in the large-scale VisDial dataset, 98% of dialogs and 38% of questions have at least one pronoun.Source: §1 · p. 1
• Earlier approaches store prior image attentions or recognized entities and softly attend over them, whereas people tend to review only topic-related history when confused.The cited approaches operate at sentence level with attention memory or at word level with neural module networks.Source: §1 · p. 2
• A parser that flags pronouns would miss pronouns that need no resolution and ambiguous fragments without pronouns, making exhaustive case handling impractical.The introduction illustrates these cases with “Is it sunny?” and “What color?”.Source: §1 · p. 2
• RvA is proposed for visual co-reference resolution, recursively reviewing topic-related history when the current question cannot be grounded and refining attention until confident or history is exhausted.The approach uses end-to-end trainable discrete decisions and separates reference-aware features for grounding and history review from answering-aware features for selecting image attributes.Source: §1 · p. 2
• The authors report state-of-the-art results on VisDial v0.9 and v1.0, alongside qualitative evidence of reliable visual and language attention during history-aware recursion.The qualitative examples include ambiguous questions both with pronouns and without them, which a natural-language parser would be hard to exhaustively handle.Source: §1 · p. 2

2. Related Work

• VisDial uses a free-form chat setting, unlike GuessWhat’s closed-ended yes/no/NA questions about locating a selected object.The paper adopts the VisDial setting.Source: §2 · p. 2
• Visual co-reference resolution links expressions such as pronouns and noun phrases to the same entity and grounds that referent in visual content.Prior work applies co-reference resolution to visual grounding, action recognition, and scene understanding.Source: §2 · p. 2
• Unlike soft attention over all stored attention maps or grounded entities, RvA predicts discrete attention over topic-related history for a more intuitive, explainable process.Related approaches also include history-conditioned attention and entity recognition with neural module networks.Source: §2 · p. 2

3. Approach

• Visual dialog asks an agent to rank 100 candidate answers or generate a sentence for the current question, given the image and dialog history.The history includes the image caption and earlier question-answer pairs.Source: §3 · p. 2

3.1. Recursive Visual Attention

• RvA recursively pairs an ungrounded question with related history, combines that turn’s attended feature with the current attention using a learnable weight, and stops at a confident question or the caption.The process backtracks from present to past, then unrolls to form the recursive visual feature.Source: §3.1 · p. 3

3.2. Neural Modules

• INFER decides whether to terminate recursion and sets the attention-fusion weight, while PAIR selects the related history turn and ATT computes question-guided visual attention.PAIR accounts for both question-history matching and sequential distance; its discrete output selects one history.Source: §3.2 · pp. 3–4

3.3.1 Feature Representation

• Reference-aware question attention emphasizes nouns and pronouns for grounding, while answering-aware attention emphasizes property words to activate relevant visual attributes.The model uses separate question representations for recursion termination and visual grounding versus question answering.Source: §3.3.1 · p. 5, Figure 4 · p. 5
• The attended visual feature is filtered to deactivate information unrelated to answering, then combined with the answering-aware question feature and history embedding for the answer decoder.The model uses top-36 region proposals and attends over history to form a fact embedding.Source: §3.3.1 · p. 5

3.3.2 Gumbel Sampling

• RvA uses Gumbel-Max sampling with a continuous softmax relaxation to make its discrete decisions differentiable for end-to-end training.Training uses a one-hot sample in forward propagation and gradients through the relaxed operation; testing greedily selects the largest probability.Source: §3.3.2 · pp. 5–6

4.1. Datasets and Setup

• VisDial v0.9 contains 83k MS-COCO training dialogs and 40k validation dialogs; v1.0 adds Flickr dialogs and uses random-length dialogs for its test split.Source: §4.1 · p. 6

4.2. Metrics

• Evaluation ranks 100 candidate answers, using Mean, R@k, and MRR for v0.9 and adding NDCG for v1.0.Source: §4.2 · p. 6

4.3. Implementation Details

• Text is lowercased, digit-normalized, and tokenized, with vocabulary cutoffs, 300-dimensional GloVe embeddings, 512-dimensional LSTMs, and Adam training.The vocabulary contains 9,795 words for v0.9 and 11,336 for v1.0.Source: §4.3 · p. 6

4.4. Comparing Methods

• Comparisons include fusion-, attention-, and visual co-reference-based methods, plus ablations of region proposals, recurrent encoders, recursive attention, and visual feature filtering.Source: §4.4 · p. 6

4.5. Quantitative Results

• RvA outperforms state-of-the-art methods across all discriminative VisDial v0.9 metrics, with approximately 2-point R@k and 2% MRR improvements; similar conclusions hold on v1.0.Source: §4.5 · p. 7, Table 1 · p. 7, Table 2 · p. 7
• Recursive attention improves every tested ablation, and RvA ranks third on the VisDial v1.0 leaderboard while DL-61 records NDCG 0.5788.The cited leaderboard result describes DL-61's two-stage approach.Source: §4.5 · p. 7, Table 3 · p. 8
• In generative VisDial v0.9 retrieval, RvA gains approximately 2 R@k points over CorefNMN but trails reinforcement-learning-trained CoAtt.Source: §4.5 · p. 7, Table 4 · p. 8

4.6. Qualitative Results

• Qualitative analyses report reasonable recursive histories and reference-aware attention; 89% of recursive attention maps versus 56% of question-guided maps are reasonable for ambiguous questions.The PAIR example skips irrelevant history under the ground-truth dialog, while a substituted answer changes the pairing and affects attention and prediction.Source: Figure 5 · p. 7, §4.6 · p. 8, Figure 6 · p. 8

• On VisDial v0.9 and v1.0, RvA achieves state-of-the-art performance and obtains explainable recursion and attention maps.Source: §5 · p. 8
• RvA formulates visual co-reference resolution as a model with three neural modules that determine recursion at run-time.Source: §5 · p. 8
• The dialog agent recursively reviews topic-related history to refine visual attention and can be end-to-end trained while making discrete module-assembly decisions.Source: §5 · p. 8
• The authors plan to incorporate in-depth language parsing modules into RvA for more accurate recursive decisions.Source: §5 · p. 8