Visual Coreference Resolution in Visual Dialog using Neural Module Networks

1 Introduction

• Visual coreferences include noun phrases and pronouns that refer to the same image entity, and their meaning can vary with context, so pronouns must be linked to antecedents and grounded visually.The proposed approach reuses information associated with previously mentioned entities, analogously to referring back to stored variables in a program.Source: §1 · p. 2
• Because isolated VQA questions rarely contain coreferences, prior VQA work largely ignored the challenge; empirical studies also found vision-language models weak at grounding entities to pixels.The paper positions explicit reasoning over past dialog interactions as a way to make entity tracking inspectable and model behavior more consistent.Source: §1 · pp. 2–3
• The neural module network introduces Refer and Exclude for explicit, grounded word-level coreference resolution, and handles captions at finer-than-sentence granularity.The model tracks entities across dialog turns so later phrases can reuse their earlier visual grounding.Source: §1 · pp. 2–3
• The model achieves near-perfect accuracy on MNIST Dialog and outperforms other approaches on VisDial, while enabling word-level coreference resolution.The paper also qualitatively presents the model as more interpretable, grounded, and consistent across dialog rounds.Source: §1 · p. 3

Visual Dialog

• Visual dialog datasets formalized by differ in purpose: elicits free-form questions about hidden scenes, while uses goal-driven yes/no questions to identify a secret object.Follow-up works used reinforcement learning [14]; other approaches transfer knowledge to generative dialog, use attention for visual coreferences, or model dialogs with conditional variational autoencoders.Source: §2 · p. 3
• Unlike, which uses global visual attention maps stored against whole-question and answer representations, this model resolves and grounds different phrases at word level.The authors identify as the closest prior work, while are complementary approaches to visual dialog.Source: §2 · pp. 3–4

Grounding language in images and video

• Visual grounding work localizes referring expressions in images or video; this model first resolves anaphora to a known entity, then borrows its grounding.Source: §2 · p. 4

Coreference resolution

• Visual coreference resolution links phrases that refer to the same entity in visual data; unlike and, this work predicts a program to resolve coreferences and answer image questions.Jointly generating language while grounding and resolving coreferences is described as a future direction, especially for generating questions in visual dialog.Source: §2 · p. 4

Neural Module Networks

• Neural Module Networks compose instance-specific architectures from shared modules, and this work extends from VQA to visual dialog with a module for explicit visual coreference resolution.Prior NMN applications include VQA in real images and linguistic databases, and complex reasoning on synthetic datasets.Source: §2 · p. 4

3 Approach

• The model identifies entity phrases, distinguishes novel from previously mentioned entities, grounds novel entities in the image, and reuses stored groundings to resolve later coreferences.Its reference pool accumulates unique entities and their visual groundings as dialog proceeds.Source: §3 · p. 5

3.1 Neural Modules for Visual Question Answering

• The approach builds on attentional neural module networks, whose modules use image features, question-derived text features, and spatial attention inputs to produce attention maps or context vectors.The module repertoire supports localization, spatial relocation, attention-map intersection or union, and attribute, existence, or count reasoning.Source: §3.1 · p. 6

3.2 Neural Modules for Coreference Resolution

• Refer matches a question phrase to entities in the reference pool, using dialog distance in its candidate scores and combining candidate attention maps into the phrase’s visual grounding.The pool stores text features and Find-module attention maps from earlier questions, creating an interpretable dictionary of entities and groundings.Source: §3.2 · pp. 6–8
• Exclude finds image regions matching a phrase while removing regions covered by an input attention map, enabling questions about other objects or attributes.It combines Find, Not, and And modules.Source: §3.2 · p. 8

3.3 Program Generation

• The history-aware program generator predicts a module sequence in reverse Polish notation and a question-attention map for each module, using dialog history to inform both predictions.Module-specific attention produces the text features used to instantiate modules, while access to history lets the predictor determine whether entities have appeared before.Source: §3.3 · p. 8
• Caption entities enter the reference pool through an auxiliary caption–image alignment task, while prior answers become candidates paired with an attention map from the reasoning that produced them.For answer-derived candidates, the model encodes the preceding question–answer pair and stores the last module’s image attention map.Source: §3.3 · pp. 9–10

3.4 Other Model Components

• Program execution dynamically assembles modules from the predicted layout, and answer decoding scores candidate answers against the resulting context vector.At test time, the decoder selects the candidate with the maximum score.Source: §3.4 · p. 10

4.1 MNIST Dialog Dataset

• CorefNMN achieved 99.3% accuracy on MNIST Dialog, outperforming all models; text-only Q, image-only I, and NMN scored 36.6%, 20.2%, and 23.8%.Adding dialog sequence information improved CorefNMN by 10.6%; the paper notes that ambiguous phrases often refer to the most recent entity.Source: §4.1 · pp. 11–12, Table 2 · p. 12

4.2 VisDial v0.9 Dataset

• On VisDial v0.9, CorefNMN outperformed other approaches across all metrics, with R@k (k = 1, 2, 5) at least 1 point above AMEM+SEQ-QI and almost 2 points above NMN.Removing memory augmentation hurt performance across all metrics; caption-trained modules had varied effects, decreasing R@1 but marginally increasing or not affecting other metrics.Source: Table 3 · p. 13, §4.2 · p. 14

4.3 VisDial v1.0 Dataset

• CorefNMN beat all compared models on VisDial v1.0 test-standard retrieval metrics except NDCG, where the neural module baseline NMN performed better.The comparison used ResNet-152 features for CorefNMN and NMN, while the other compared models used VGG features.Source: Table 4 · p. 15, §4.3 · p. 15

5 Conclusions

• CorefNMN outperforms prior work on MNIST Dialog, achieving close to perfect accuracy, and on VisDial.The conclusion also characterizes the model as more interpretable, grounded, and consistent by construction.Source: §5 · p. 16
• CorefNMN explicitly links visual coreferences and grounds them in the image at the word level, rather than implicitly or at the sentence level.The passage describes this as introspective reasoning about visual coreferences.Source: §5 · p. 16
• The model is more interpretable, grounded, and consistent by construction.Source: §5 · p. 16

Sec. A

• Sec. A reports superior performance for CorefNMN with a discriminative decoder and ImageNet-pretrained ResNet-152 features.The section presents these results in the supplement.Source: Overview of Supplement · p. 17

Sec. B

• Sec. B details experiments using a generative answer decoder.Source: Overview of Supplement · p. 17

Sec. C

• Sec. C provides implementation details for the experiments.Source: Overview of Supplement · p. 17

Supplementary figures

• Figures 5–7 show Refer-module schematics, a caption-module auxiliary task, and another qualitative VisDial example.The auxiliary task enables modules to run on captions and handles captions at fine word-level granularity.Source: Overview of Supplement · p. 17

A Discriminative Decoder Experiments

• For VisDial v0.9, ResNet-152 features replace the VGG features used in the main-paper models; the comparison includes author-provided metrics for MNQIH-G and LF-QH-G and a retrained NMN.Results are presented in Table 5.Source: A Discriminative Decoder Experiments · p. 17

B Generative Decoder Experiments

• A multilayer LSTM initialized with context vector c_t is trained using the ground-truth answer’s negative log-likelihood; evaluation ranks answer-pool candidates by model log-likelihood.Ranking candidates follows the VisDial v0.9 evaluation protocol, rather than being a limitation of the generative decoder.Source: B Generative Decoder Experiments · p. 17
• Unlike a discriminative decoder, the generative decoder can potentially produce novel answers through language generation.At test time it ranks the answer pool, but that ranking is required by the evaluation protocol rather than imposed by the decoder.Source: B Generative Decoder Experiments · p. 17
• Caption-module auxiliary loss boosts full CorefNMN but hurts the no-memory ablation; removing it improves that ablation across metrics, while removing it from full CorefNMN is uniformly worse.The passage compares CorefNMN\Mem\Laux_C with CorefNMN\Mem across all metrics and CorefNMN\Laux_C with CorefNMN.Source: B Generative Decoder Experiments · p. 18
• CorefNMN achieved the best R@5, R@10, and mean rank; its mean rank was 15.69 (a 4% improvement over NMN), while NMN led R@1 and MRR.The reported results are on the VisDial v0.9 validation set.Source: B Generative Decoder Experiments · p. 18, Table 6 · p. 19

C Implementation Details

• Models use TensorFlow v1.0, Adam [24] at 0.0001, gradients clamped to [−2.0, 2.0], and lowercase NLTK-tokenized text; vocabulary keeps training words occurring at least five times.The gradient clamp is intended to avoid gradient explosion.Source: C Implementation Details · p. 19

C.1 MNIST Dialog Dataset

• A single-layer LSTM with hidden size 64 encodes questions and history, using 32-dimensional embeddings for the 73-word vocabulary and modules.The dialog’s synthetic nature yields low text variability and a small vocabulary.Source: §C.1 · p. 19
• The image CNN follows, with four 3 × 3 convolutional layers, batch normalization, ReLU, and 2 × 2 max pooling; early stopping uses 10k validation images.The first two convolutional layers have 32 channels each, and the last two have 64.Source: §C.1 · p. 19
• Figure 5 illustrates Refer using keys xp to retrieve attention ap from the coreference pool Pref to understand “it.”The figure depicts sentence/question, text, image, and spatial attention with colored lines and boxes.Source: Figure 5 · p. 20

C.2 VisDial Dataset

• VisDial experiments used two-layer, 1000-unit LSTMs, spatially augmented ResNet-152 features, 300-dimensional embeddings, and MRR early stopping on 3k held-out images.Images used pre-final-mean-pooling features from ImageNet-pretrained ResNet-152, augmented with X/Y location dimensions. The approximately 8k-word vocabulary used GloVe-initialized word embeddings; the 3k-image validation subset was set aside from 83k training images.Source: §C.2 · p. 20

• The v0 version was the ECCV 2018 camera-ready version, and v1 added experiments on the VisDial v1.0 dataset.Source: §C.3 · p. 20