Source-linked AI summary
Visual Coreference Resolution in Visual Dialog using Neural Module Networks
Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, Marcus Rohrbach
TL;DR
Visual dialog requires resolving image-grounded references across questions and dialog history, a capability prior work handled implicitly or coarsely. The paper introduces Refer and Exclude modules in a neural module network for explicit, grounded, word-level resolution, achieving near-perfect MNIST Dialog accuracy and outperforming prior VisDial approaches while improving interpretability, grounding, and consistency.
Problem
Visual dialog needs explicit word-level resolution of noun phrases and pronouns to image entities, whereas prior approaches model coreference implicitly or at question-level granularity.
Method
A neural module network uses Refer and Exclude modules to track entities, resolve seen references through a reference pool, and ground novel entities in the image.
Results
The model achieves near-perfect accuracy on MNIST Dialog and outperforms prior work on both MNIST Dialog and VisDial, with VisDial gains across all metrics except NMN on NDCG.
Takeaways & Limitations
Explicit word-level coreference makes the model more interpretable, grounded, and consistent by construction across visual dialog rounds.
Abstract
from arXiv · showhide
Visual dialog entails answering a series of questions grounded in an image, using dialog history as context. In addition to the challenges found in visual question answering (VQA), which can be seen as one-round dialog, visual dialog encompasses several more. We focus on one such problem called visual coreference resolution that involves determining which words, typically noun phrases and pronouns, co-refer to the same entity/object instance in an image. This is crucial, especially for pronouns (e.g., `it'), as the dialog agent must first link it to a previous coreference (e.g., `boat'), and only then can rely on the visual grounding of the coreference `boat' to reason about the pronoun `it'. Prior work (in visual dialog) models visual coreference resolution either (a) implicitly via a memory network over history, or (b) at a coarse level for the entire question; and not explicitly at a phrase level of granularity. In this work, we propose a neural module network architecture for visual dialog by introducing two novel modules - Refer and Exclude - that perform explicit, grounded, coreference resolution at a finer word level. We demonstrate the effectiveness of our model on MNIST Dialog, a visually simple yet coreference-wise complex dataset, by achieving near perfect accuracy, and on VisDial, a large and challenging visual dialog dataset on real images, where our model outperforms other approaches, and is more interpretable, grounded, and consistent qualitatively.
1 Introduction
Visual dialog requires reasoning over images and dialog history, with visual coreference resolution as a central challenge. The paper proposes explicit, grounded, word-level entity tracking through neural modules and reports strong performance and interpretability benefits.
- Visual dialog agents answer image-grounded question sequences by reasoning over both the image and previous dialog history.
- Visual coreference resolution links noun phrases and pronouns to the same image entity, enabling grounded interpretation of references such as “it.”
- Prior VQA work largely ignores visual coreference, while vision-language models often perform poorly at grounding entities in correct image pixels.
- Explicit entity tracking improves interpretability by exposing known entities and phrase-to-entity resolutions, while promoting consistency across dialog rounds.
- The model introduces Refer and Exclude modules for explicit, grounded coreference resolution, plus word-level caption handling within a neural module network.
- The approach achieves near-perfect accuracy on MNIST Dialog and outperforms other approaches on VisDial while being more interpretable, grounded, and consistent.
2 Related Work
Prior visual dialog approaches commonly represent whole questions and dialog history, whereas this work operates at word-level granularity to resolve and ground distinct referring expressions.
- Earlier approaches store attention maps using representations of entire questions, answers, and dialog history.
- Word-level processing lets the model resolve different phrases within one question and ground them to different image regions.
- For anaphora such as “it,” “he,” or “she,” the model first resolves the reference to a known entity before borrowing that entity’s visual grounding.
3 Approach
The model answers visual-dialog questions by generating and executing neural-module programs that explicitly resolve phrase-level coreferences through a history-based reference pool. It combines program generation, dynamic module execution, and candidate-answer decoding to ground references in image regions and produce answers.
- The task is formulated as ranking 100 candidate answers for the current question given the image and dialog history.
- The model explicitly resolves visual coreferences by identifying entity phrases, distinguishing novel from previously seen entities, and retrieving visual groundings from dialog history.Novel entities are localized in the image, while previously seen entities retrieve the grounding associated with their earlier reference.
- Its three components are program generation, dynamic program execution with neural modules, and answer decoding from the resulting context vector.The generated program determines the module layout, execution produces context vector c_t, and decoding scores candidate answers.
- 3.1 Neural Modules for Visual Question Answering: Neural modules receive spatial image embeddings and question-derived text embeddings, producing either spatial attention maps for subsequent modules or context vectors for answer prediction.Module text features are weighted sums of question-word embeddings using program-generated soft attention.
- 3.2 Neural Modules for Coreference Resolution: The reference pool stores text features and attention maps from earlier Find modules, forming an interpretable dictionary of entities and their visual groundings.The pool contains entries for entities mentioned in previous questions and supports later reference resolution.
- 3.2 Neural Modules for Coreference Resolution: Refer matches a current text query to reference-pool entries, uses dialog distance in scoring, and transfers the best match’s image attention as the phrase grounding.Softmax contributions are computed across candidate entities after similarity scoring.
- 3.2 Neural Modules for Coreference Resolution: Exclude combines Find, Not, and And to locate objects or attributes other than those covered by an input attention map.This supports questions such as asking for other red objects.
- 3.3 Program Generation: The program generator predicts a module sequence and a question-word attention distribution for every module, enabling distinct phrases to drive distinct operations.History-aware question encoding supports coreference decisions based on earlier dialog, including pronouns referring to entities introduced in previous answers.
4 Experiments
Experiments evaluate CorefNMN on MNIST Dialog and VisDial, testing explicit coreference resolution in synthetic and real-image settings. The model achieves near-perfect MNIST accuracy and outperforms competing approaches on VisDial retrieval metrics, with memory augmentation improving performance.
- 4.1 MNIST Dialog Dataset: MNIST Dialog uses 4 × 4 grids of MNIST digits and ten-round dialogs designed to require tracking target digits across references.Questions query digit attributes and counts, making coreference resolution central to evaluation.
- Qualitative Analysis: The model’s qualitative reasoning tracks entities through a coreference pool and resolves later phrases by reusing earlier visual grounding.Examples show Refer resolving “them” or “the head” from prior dialog context.
- 4.1 MNIST Dialog Dataset: 99.3% accuracy: CorefNMN outperforms all other models on the MNIST Dialog test set.Most remaining errors involve qualitatively difficult examples from the original MNIST dataset.
- 4.1 MNIST Dialog Dataset: 10.6% improvement: incorporating dialog sequence information benefits CorefNMN on MNIST Dialog.The corresponding improvement for AMEM is 7.2%.
- 4.2 VisDial v0.9 Dataset: CorefNMN outperforms all other approaches across MRR, recall@k, and mean rank on VisDial v0.9 validation data.The evaluation uses discriminative models and reports R@1, R@5, R@10, mean rank, and MRR.
- 4.2 VisDial v0.9 Dataset: Removing memory augmentation hurts performance uniformly across VisDial v0.9 metrics because the model cannot use history to decide when to invoke Refer.Caption modules reduce R@1 but have marginal or no effect on other metrics.
- 4.3 VisDial v1.0 Dataset: On VisDial v1.0, CorefNMN outperforms all other approaches on every reported metric except NDGC against the neural module baseline.The comparison is reported on the test-standard split.
5 Conclusions
The paper presents CorefNMN as a neural module network that explicitly links visual coreferences to image regions at word level. It reports near-perfect MNIST performance and improved VisDial performance with interpretability, grounding, and consistency by construction.
- 5 Conclusions: CorefNMN explicitly links visual coreferences and grounds them in the image at word level, rather than implicitly or at sentence level.The model is based on neural module networks.
- 5 Conclusions: The model outperforms prior work on both MNIST Dialog and VisDial while being more interpretable, grounded, and consistent by construction.MNIST Dialog performance is described as close to perfect.
Overview of Supplement
The supplement presents additional decoder, feature, implementation, and architectural details, including the Refer module and word-level caption handling.
- A Discriminative Decoder Experiments: Section A reports results using a discriminative decoder with ImageNet-pretrained ResNet-152 image features.
- B Generative Decoder Experiments: Section B introduces a generative answer decoder for the VisDial v0.9 experiments.
- C Implementation Details: Section C provides implementation details for the experiments.
- Supplementary Visualizations: The supplement includes schematics of Refer, the caption auxiliary task, and another qualitative VisDial example.
A Discriminative Decoder Experiments
The supplement evaluates CorefNMN with ResNet-152 image features on discriminative VisDial v0.9 retrieval experiments.
- Feature Setup: CorefNMN is evaluated with pretrained ResNet-152 features, alongside comparisons against retrained or author-provided baselines.The comparison is intended to make feature usage fair across models where possible.
B Generative Decoder Experiments
The generative-decoder experiments show that CorefNMN improves several VisDial retrieval metrics, while the NMN baseline remains strongest on R@1 and MRR.
- Generative Decoder: The generative decoder is a multi-layer LSTM initialized with the context vector and trained by minimizing negative log-likelihood.At test time, candidate answers are ranked by model log-likelihood for VisDial evaluation.
- Ablations: The full model uses memory augmentation and caption-training ablations to test the contributions of dialog history and auxiliary supervision.The compared variants remove memory augmentation, caption auxiliary loss, or both.
- Results: CorefNMN outperforms other generative approaches on R@5, R@10, and mean rank.The model’s mean rank is 15.69, a 4% improvement over the NMN baseline.
- Results: The NMN baseline achieves higher R@1 and, possibly as a consequence, the best MRR.
- Ablations: The combined ablation is better than the no-memory ablation across all metrics, while the full model is uniformly better than the combined ablation.
C Implementation Details
The experiments use TensorFlow, Adam optimization, gradient clipping, and standardized lowercase tokenization with a frequency-filtered vocabulary.
- Optimization: The models are implemented in TensorFlow v1.0 and optimized with Adam at a learning rate of 0.0001.
- Optimization: Gradients are clamped to [−2.0, 2.0] to avoid gradient explosion.
- Text Preprocessing: Questions and answers are lowercased, tokenized with NLTK, and restricted to words appearing at least five times in training.
C.1 MNIST Dialog Dataset
MNIST Dialog uses compact recurrent and embedding representations suited to synthetic, low-variability dialog, while the Refer module retrieves attention from a coreference pool to interpret pronouns such as “it”.
- Synthetic dialog uses a single-layer LSTM with hidden size 64 for questions and history.
- The vocabulary contains 73 words, each represented by a learned 32-dimensional embedding.
- The Refer module uses keys x_p to retrieve attentions a_p from the coreference pool P_ref.
C.2 VisDial Dataset
VisDial experiments use deeper, larger recurrent representations and spatially enriched ResNet-152 image features to support reasoning over real-image dialog.
- VisDial questions and histories are encoded with two-layer LSTMs having hidden size 1000.
- Image representations use convolutional features from an ImageNet-pre-trained ResNet-152 before final mean pooling.
- Two additional dimensions encode X-column and Y-row locations to facilitate spatial reasoning.
- With a vocabulary of around 8k words, word and module embeddings are 300-dimensional vectors.
C.3 Document Changelog
The document changelog records an ECCV 2018 camera-ready version and an initial arXiv version that added experiments on VisDial v1.0.
- The changelog is provided to help readers track changes to the document.
- Version 0 is identified as the ECCV 2018 camera-ready version and was not on arXiv.
- Version 1 is identified as the initial arXiv version and added experiments on the VisDial v1.0 dataset.