PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Dialog

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, Dhruv Batra

arXiv:1611.08669v5cs.CVcs.AIcs.CLcs.LG

TL;DR

Visual Dialog addresses the challenge of enabling meaningful, multi-turn language interaction about images rather than single-round visual question answering. The paper introduces a role-specific chat protocol and VisDial dataset, neural encoder-decoder models, and retrieval-based evaluation; the resulting benchmark shows substantial room for improvement toward visual intelligence.

  • Problem

    Visual Dialog asks how an AI agent can answer image-grounded questions while using dialog history for context and maintaining conversational consistency.

  • Method

    The paper builds VisDial with two-person live chats, develops three encoder architectures with generative and discriminative decoders, and evaluates candidate-answer ranking.

  • Results

    The proposed encoder-decoder settings outperform sophisticated baselines, and human studies quantify the machine-human performance gap.

  • Takeaways & Limitations

    Visual Dialog provides a testbed for measuring progress toward visual intelligence through image-grounded conversational responses.

  • Takeaways & Limitations

    Caption-seeded collection was chosen because blurred-image seeds produced blob-recognition questions rather than natural conversations.

Abstract

from arXiv · show

We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being grounded in vision enough to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial v0.9 has been released and contains 1 dialog with 10 question-answer pairs on ~120k images from COCO, with a total of ~1.2M dialog question-answer pairs. We introduce a family of neural encoder-decoder models for Visual Dialog with 3 encoders -- Late Fusion, Hierarchical Recurrent Encoder and Memory Network -- and 2 decoders (generative and discriminative), which outperform a number of sophisticated baselines. We propose a retrieval-based evaluation protocol for Visual Dialog where the AI agent is asked to sort a set of candidate answers and evaluated on metrics such as mean-reciprocal-rank of human response. We quantify gap between machine and human performance on the Visual Dialog task via human studies. Putting it all together, we demonstrate the first 'visual chatbot'! Our dataset, code, trained models and visual chatbot are available on https://visualdialog.org

1. Introduction

Visual Dialog extends visual-language interaction from single-turn questions to meaningful, history-aware conversations grounded in images. The paper introduces the task, VisDial dataset, encoder-decoder models, and retrieval-based evaluation.

  • Task and motivation: VQA lacks follow-up questions, memory of prior questions, and consistency across previous answers.Visual Dialog is designed to address these limitations of single-round visual question answering.
  • Task and motivation: Visual Dialog requires answering a natural-language follow-up using an image and previous question-answer history.The task includes grounding the question in visual content and maintaining conversational context.
  • Task and motivation: Visual Dialog balances general machine-intelligence testing with vision grounding that supports objective response evaluation.Its design avoids dependence on a specific downstream task while retaining measurable visual grounding.
  • Contributions: VisDial uses two-person chat collection to provide 10 question-answer pairs per image at large scale, with approximately 140k COCO images and 1.4M pairs planned.Compared with VQA, the dataset supports richer dialog, free-form longer answers, and avoids visual priming because the questioner does not see the image.
  • Contributions: The paper introduces Late Fusion, Hierarchical Recurrent, and Memory Network encoders with generative and discriminative decoders.The models encode image, question, and dialog history using distinct fusion, recurrent, or memory-based mechanisms.
  • Contributions: The proposed systems outperform sophisticated baselines, while retrieval-based evaluation ranks candidate answers using metrics such as mean-reciprocal-rank.The paper also measures human performance and demonstrates a visual chatbot.

2. Related Work

Visual Dialog builds on vision-language, visual Turing test, reading-comprehension, and chatbot research while targeting open-ended, multi-turn visual interaction. Its role asymmetry and image grounding support more realistic conversations and objective response evaluation.

  • Vision and Language: Most prior vision-language tasks involve at most one natural-language interaction and do not support dialog.The related work includes captioning, video description, grounding, storytelling, and VQA.
  • Visual Turing Test: Compared with a restrictive visual Turing test, VisDial uses free-form open-ended questions collected through two-person AMT conversations.The paper also contrasts VisDial’s broader scene variety with a dataset limited to street scenes.
  • Text-based Question Answering: VisDial combines reading-comprehension-style dialog-history understanding with visual question answering.The machine must comprehend prior dialog and understand the image to answer each question.
  • Conversational Modeling and Chatbots: Unlike symmetric text chat, VisDial assigns one participant to ask about an unseen image and the other to answer from the visible image.This role structure gives the interaction a defined purpose and enables objective evaluation of individual responses.

3. The Visual Dialog Dataset (VisDial)

VisDial is collected through live, role-specific conversations between paired AMT workers viewing different information about COCO images. The protocol is designed to produce temporally continuous, image-grounded, natural exchanges with unconstrained answers.

  • Data collection: VisDial uses COCO images because their visual complexity supports engaging and diverse conversations.The dataset construction begins with images containing multiple objects in everyday scenes.
  • Data collection: Two AMT workers chat in real time, with one assigned as questioner and the other as answerer.The live interface targets temporal continuity, image grounding, and natural conversational exchange.
  • Role assignment: The questioner sees only a COCO caption and asks about the hidden image, while the answerer sees the image and caption and responds naturally.Unlike VQA, answers are not restricted to short or concise forms.
  • Protocol: Conversations allow 20 exchanged messages, corresponding to 10 question-answer pairs, before workers may end the interaction.The questioner waits for each answer before posting the next question.
  • Protocol choice: Caption-seeded conversations were selected over blurred-image seeds because blurred images produced blob-recognition questions.The authors report that captions generated more natural questions and better matched intended applications.
  • Implementation: The AMT implementation required custom Redis and Node.js infrastructure because AMT did not support multi-user HITs.Abandoned or disconnected sessions were discarded and relaunched to obtain genuine two-person conversations.

4. VisDial Dataset Analysis

VisDial v0.9 provides a large conversational dataset whose questions and answers are more diverse and context-dependent than earlier visual question-answering data. Its analysis highlights visual priming reduction, longer answers, coreference, topical continuity, and retrieval-based response evaluation.

  • VisDial v0.9 contains 1,232,870 question-answer pairs across approximately 123k COCO-train/val images.
  • Questioners do not see the image, reducing the visual priming bias that can make language-only models perform strongly on VQA.
  • VisDial answers average 2.9 words, compared with 1.1 in VQA, 2.0 in Visual 7W, and 2.8 in Visual Madlibs.
  • The top 1,000 answers cover approximately 63% of VisDial answers versus approximately 83% in VQA, with 337,527 unique VisDial answers.
  • Answer types include uncertainty expressions such as “I think so,” “I can’t tell,” and “I can’t see,” reflecting questions that may not be answerable with certainty.
  • Pronouns occur in 38% of questions, 19% of answers, and nearly all dialogs, making coreference a central challenge for successful Visual Dialog systems.
  • The task evaluates individual responses by ranking 100 candidate answers using rank, recall@k, and mean reciprocal rank.

5. Neural Visual Dialog Models

The paper develops encoder-decoder models that combine the image, current question, and dialog history in different ways. The encoders range from simple fusion to hierarchical recurrence and memory-based attention, paired with generative or discriminative decoders.

  • All Visual Dialog models use an encoder-decoder framework that maps image, history, and question inputs into a representation before producing an answer.
  • Decoders: The model family includes generative LSTM and discriminative decoders, with generative models ranking candidates by answer log-likelihood.
  • Late Fusion Encoder: The Late Fusion encoder separately encodes the concatenated history and question, then combines their representations with the image representation.
  • Hierarchical Recurrent Encoder: The Hierarchical Recurrent Encoder represents words, question-answer rounds, and dialog context through recurrent blocks and a dialog-level RNN.
  • Hierarchical Recurrent Encoder: HRE attention selects the history round relevant to the current question before producing the joint representation used by the decoder.
  • Memory Network Encoder: The Memory Network stores previous question-answer pairs as facts, attends to them and the image, and forms a context vector for answering.

6. Experiments

Experiments evaluate VisDial models on a 40k-image validation set against frequency, nearest-neighbor, and adapted VQA baselines. Learning-based, discriminative, history-aware, and image-aware models generally perform better, with memory-network models achieving the strongest reported scores.

  • The VisDial v0.9 test evaluation uses 40k COCO-validation images, while the dataset split includes 80k training and 3k validation dialogs.
  • Baselines: The baselines include answer-prior and nearest-neighbor methods, along with adapted SAN and HieCoAtt VQA models.
  • Results: Learning-based models significantly outperform non-learning baselines, and discriminative models significantly outperform generative models.
  • Results: 0.526 MRR is achieved by MN-QIH-G, while MN-QIH-D achieves 0.597 MRR as the best generative and discriminative models, respectively.
  • Results: Naively incorporating history provides little benefit or can hurt, whereas models with stronger history encoding, such as MN and HRE, perform better than corresponding Late Fusion models.
  • Results: Models using the image outperform corresponding blind models without image input.
  • Human Studies: Human studies find that without the image, people perform better with dialog history, while access to the image narrows this gap.

7. Conclusions

VisDial is presented as a dialog dataset whose answers, references, and topic transitions exhibit conversational structure rather than independent image-question pairs. The supplementary analyses report longer, more diverse answers and continuity across question rounds.

  • Answer characteristics: VisDial answers average 2.9 words, compared with 1.1 in VQA, 2.0 in Visual 7W, and 2.8 in Visual Madlibs.
  • Answer characteristics: 337,527 of 1,232,870 VisDial answers are unique, and the top 1,000 cover approximately 63% of answers versus approximately 83% in VQA.
  • Conversational structure: 38% of questions, 19% of answers, and 98% of dialogs contain at least one pronoun, compared with 9% of questions and 0.25% of answers in VQA.
  • Conversational structure: VisDial averages 4.55 ± 0.17 topics across ten rounds, while three-question sliding windows average 2.14 ± 0.05 topics versus 2.53 ± 0.09 for VQA.
  • Conversational structure: In-order VisDial has a 0.61 topic-transition probability versus 0.76 ± 0.02 after permutation, compared with 0.80 versus 0.83 ± 0.02 for VQA.

A.4. VisDial has the statistics of an NLP dialog dataset

Experiments comparing natural and permuted sequences find that VisDial has measurable temporal and linguistic structure characteristic of dialog datasets. Perplexity and classification results distinguish VisDial from VQA and place it closer to Cornell Movie-Dialogs.

  • Evaluation setup: The analysis compares VisDial, VQA, and Cornell Movie-Dialogs using sequence structure in natural versus permuted test orderings.
  • Evaluation setup: A Seq2Seq model predicts each VisDial question from the preceding question-answer pair and evaluates temporal structure with perplexity.
  • Perplexity: Shuffling significantly increases perplexity in all three datasets, indicating that permutation disrupts linguistic sequence structure.
  • Perplexity: The absolute perplexity increase is 3.0 for Cornell, 0.7 for VisDial, and 0.35 for VQA; relative increases are 3.64%, 10.13%, and 4.21%, respectively.
  • Classification: The perplexity-based classifier reaches 73.3% accuracy on VisDial, versus 61.0% on Cornell and 52.8% on VQA.
  • Conclusion: The experiments conclude that VisDial is more dialog-like than VQA and behaves more like the Cornell Movie-Dialogs corpus.

A.5. VisDial eliminates visual priming bias in VQA

VisDial reduces visual priming bias by having questioners ask without seeing the image, producing uncertainty and answer patterns unlike VQA. Its questions therefore reflect conversational information gathering rather than image-conditioned prompting.

  • Visual priming bias: VisDial questioners do not see the image, reducing the visual priming bias present in VQA and other earlier image-question-answering datasets.
  • Answer uncertainty: VisDial answers include expressions such as “I think so,” “I can’t tell,” and “I can’t see,” reflecting doubt or lack of information.
  • Answer forms: VisDial distinguishes binary questions from binary answers, allowing answers to include clarification or answer without explicitly saying “yes” or “no.”

C. Human-Machine Comparison

Human studies compare people and models under different access to images and dialog history, revealing a substantial performance gap and distinct benefits from contextual information.

  • Study design: Humans ranked candidate answers with varying access to images and dialog history, using ten options and three subjects per task.The study covered all combinations of image and history access across 100 images at each of ten rounds.
  • Human performance: 30.31 vs. 25.10 R@1 shows that without the image, humans perform better with dialog history than with the question alone.With the image available, the corresponding gap narrows to 48.03 vs. 46.12 R@1.
  • Human-machine comparison: 83.76% vs. 69.39% R@5 shows a large gap between humans with full information and the best discriminative model.The generative model comparison is wider: Human-QIH reaches 83.76% versus 61.61% for HREA-QIH-G.
  • Human-machine comparison: 70.53% vs. 69.39% R@5 shows that humans with history but no image outperform the best machine evaluated with full information.Humans with the image but no history also outperform the machine, reaching 82.54% R@5.
  • Role of history: Dialog history provides scene context, response-style cues, and disambiguation among likely image interpretations for human answer selection.These roles help eliminate choices, identify answers among similar candidates, and resolve ambiguity.

E. Additional Analysis of VisDial

Additional analyses characterize how question and answer forms vary across dialog rounds and question types in VisDial.

  • Question and answer lengths: Question length by type remains consistent across rounds, while questions beginning with “any” tend to be shortest.Examples include “any people?” and “any other fruits?”.
  • Question and answer lengths: Answer length varies substantially by question type and round, with binary-question answers tending to be short.Answers to “how” and “what” questions tend to be longer.
  • Question-type coverage: As conversations progress, “is”, “what” and “how” questions decrease while “can”, “do”, “does” and “any” questions increase.Questions beginning with “Is” are the most popular overall.

F. Performance on VisDial v0.5

On VisDial v0.5, learning-based and history-aware models perform best, with Memory Networks leading both generative and discriminative settings while dialog-level accuracy remains limited.

  • Model performance: MN-QIH-G reaches 0.44 MRR and MN-QIH-D reaches 0.53 MRR, outperforming sophisticated baselines in generative and discriminative settings.Memory Network has the best performance in both settings.
  • Model performance: All learning-based models significantly outperform non-learning baselines, while discriminative models significantly outperform generative models.The discriminative advantage reflects tuning to answer-option biases but sacrifices the ability to generate responses.
  • Model inputs: Models using history outperform question-only models, image-aware models improve recall@1 by at least 2%, and models using both history and image perform best.These comparisons identify history and image access as important inputs for VisDial performance.
  • Dialog-level evaluation: 7.01 versus 5.37 rounds are correct under R@5 for MN-QIH-D and MN-QIH-G respectively.Their mean first-failure rounds are 3.23 and 2.39, respectively.
  • Memory Network: The Memory Network learns to attend to history facts relevant to the current question, though the selected examples are not always interpretable.Examples include attending to kite-related text for a color question and bus-related text for a bus question.

G.2. Training

Training uses standardized VisDial splits and preprocessing, with model selection based on validation MRR and shared neural-network settings.

  • Data preparation: The 83k COCO-train dialogs are split into 80k training and 3k validation examples, while 40k COCO-val dialogs form the test set.These splits are used for VisDial v0.9.
  • Data preparation: Questions and answers are spell-corrected, lowercased, normalized, tokenized, and restricted to a vocabulary of roughly 7.5k words appearing at least five times.Digits are converted to words and contractions are removed before tokenization.
  • Training setup: Hyperparameters are selected by early stopping on validation MRR, using two-layer 512-dimensional LSTMs and 300-dimensional word and image embeddings.Word embeddings are shared across question, history, and decoder LSTMs.
  • Evaluation: Table 5 evaluates methods using MRR, recall@k for k = {1, 5, 10}, and mean rank, with Memory Network best in both decoder settings.Higher is better for MRR and recall@k, while lower is better for mean rank.
Loading 1611.08669v5…