Reasoning Visual Dialogs with Structural and Partial Observations

1. Introduction

• Prior visual-dialog methods largely neglect relational information, using concatenated multimodal features and feed-forward networks that do not iteratively reason over dialog structure.The paper notes that co-attention methods capture cross-modal correlations but retain limited reasoning ability.Source: §1 · p. 1
• The model represents dialog entities as graph nodes and semantic dependencies as edges, jointly inferring graph structure and unobserved values such as the queried answer.The paper frames dialog history as a partial observation of the graph.Source: §1 · pp. 1–2
• Because dialog structures lack labels, the proposed approach treats them as unknown parameters in an MRF and uses EM, approximated by a GNN, to infer structure and missing node values.Node values and graph structure can inform each other iteratively.Source: §1 · p. 2
• Experiments on VisDial v0.9, VisDial v1.0, and VisDial-Q report promising performance and automatic dialog-structure parsing.The model is presented as a unified framework for diverse dialog settings with post-hoc structural interpretability.Source: §1 · p. 2

2. Related Work

• Image-captioning work moved from template retrieval to CNN-RNN generation and attention-based methods, while later work explored specialized captioning settings.Examples include stylized, personalized, and context-aware captioning.Source: §2 · p. 2
• Graph-based VQA models apply graph reasoning to visual content or factual questions, whereas this paper treats dialog relations as unknown.The authors contrast their inference setting with propagation over a fixed graph structure.Source: §2 · p. 2
• Earlier visual-dialog work explored attention, dialog memory, image-guessing games, sequence generation, and joint question generation and answering.The cited literature also includes reinforcement-learning and GAN-based answer generation.Source: §2 · pp. 2–3
• The paper frames visual dialog as partial observation of nodes and automatically infers graph structure, with the answer represented as an unobserved node.The proposed GNN approximates an EM solution to this inference problem.Source: §2 · p. 3

3. Our Approach

• The overall approach treats the dialog history and current question as observed nodes and the current answer as missing, using EM to infer both answer and graph structure.The input includes an image, caption, dialog history, and current question; the task ranks 100 candidate answers.Source: §3 · p. 3

3.1. Dialog as Markov Random Field

• The dialog is represented as a Markov Random Field whose nodes are questions and answers and whose soft, weighted edges encode semantic dependencies.Unary and pairwise potential functions specify node-level and pairwise terms in the joint distribution.Source: §3.1 · p. 3

3.2. Inference with Partial Observation

• The EM procedure alternates an E-step that computes expected likelihood with an M-step that optimizes parameters; a MAP plug-in surrogate addresses the intractable answer-space expectation.The method applies this alternating inference to estimate missing values and graph parameters.Source: §3 · p. 3, §3.2 · p. 4

3.3. MRF with Partial Observations

• The partial-observation formulation jointly infers the missing queried-answer value and unknown edge weights from observed dialog nodes.The observed nodes are the history questions and answers and current question; the queried answer is unobserved.Source: §3.3 · p. 4

3.4. GNN with Partial Observations

• The GNN alternates neural message passing to update unobserved node states with link functions that estimate edge weights from node embeddings.Observed variables and edge weights are fixed during message passing; the hidden states are initialized from node features.Source: §3.4 · p. 5

3.5. Network Architecture

• Caption, history, and queried-answer nodes are initialized by fusing image features with language embeddings, with the queried-answer node using the current question embedding.The final node embedding is scored against candidate answers by dot product, followed by softmax and a multiclass cross-entropy loss.Source: §3.5 · pp. 5–6, Figure 3 · p. 5

4.1. Performance on VisDial v0.9 [9]

• On VisDial v0.9, the model's Recall@k is at least 0.4 point higher than SF, while SCA performs slightly better.The comparison covers R@1, R@5, and R@10; Table 1 reports quantitative results on the validation split.Source: §4.1 · p. 7, Table 1 · p. 7

4.2. Performance on VisDial v1.0 [9]

• On VisDial v1.0 test-standard, the model outperforms all other models across all metrics.The comparison is presented in Table 2.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Performance on VisDial-Q Dataset [9, 22]

• On VisDial-Q, the full model with 3 EM iterations outperforms comparative methods in all evaluation metrics.The evaluation includes retrieval metrics Recall@k, MRR, and Mean Rank.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4. Diagnostic Experiments

• Without EM iterations, the model performs worst among the tested variants; 3 message-passing iterations perform best.The ablation compares constant-graph, no-EM, and n-EM variants on VisDial v0.9.Source: §4.4 · p. 8, Table 4 · p. 8
• Dynamic edge weights outperform a constant graph because they filter misleading messages while allowing information flow.With constant edge weights, all incoming messages are treated equally.Source: §4.4 · p. 8

• The GNN represents dialog entities as nodes and their semantic dependencies as edge weights.Source: §5 · p. 8
• An EM-style inference algorithm estimates latent relations between nodes and missing values of unobserved nodes.Source: §5 · p. 8
• Experiments on VisDial and VisDial-Q report that the method finds and uses dialog structures for inference in both tasks.The authors describe this as demonstrating the method's generality and effectiveness.Source: §5 · p. 8