Source-linked AI summary
GuessWhat?! Visual object discovery through multi-modal dialogue
Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, Aaron Courville
TL;DR
GuessWhat?! addresses the challenge of grounding natural-language descriptions in visual scenes through a cooperative guessing game. It introduces a large-scale human-played dataset, associated oracle and questioner tasks, and neural baselines, with oracle performance improving when object information is provided.
Problem
Machines need to understand natural-language descriptions and ground them in the visual world to support conversational scene understanding.
Method
The paper introduces a cooperative two-player game in which a questioner uses yes-no questions to locate an oracle’s hidden object in a rich image scene, and defines tasks for both roles.
Results
The work provides the first large-scale image-dialogue dataset, three task formulations with neural baselines, and lower oracle error when object information is supplied.
Takeaways & Limitations
GuessWhat?! offers a testbed for studying visual grounding, spatial reasoning, and dialogue through a game that supports large-scale data collection.
Takeaways & Limitations
The question-generator baseline fixes the dialogue length at five questions and does not address when to stop asking and start guessing.
Abstract
from arXiv · showhide
We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task. Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images. We explain our design decisions in collecting the dataset and introduce the oracle and questioner tasks that are associated with the two players of the game. We prototyped deep learning models to establish initial baselines of the introduced tasks.
1. Introduction
GuessWhat?! brings computer vision and dialogue together in a goal-directed game where agents must ground natural-language interaction in rich visual scenes. The paper introduces a large-scale dataset and baseline tasks for studying this challenge.
- Motivation: Natural-language interaction can require multiple exchanges before a referred object is uniquely identified in a visual scene.The listener’s state of mind and scene context affect whether an expression identifies one object.
- Motivation: Grounding natural-language descriptions in the visual world remains a fundamental challenge for conversational computer-vision systems.Such systems could support more transparent and interpretable interaction through clarifying questions.
- Contribution: GuessWhat?! proposes a goal-directed two-player multi-modal dialogue task that extends ReferIt from one expression to interactive questioning.The challenge is to acquire natural language by interaction on a visual task while understanding relations between objects.
2. GuessWhat?! game
GuessWhat?! is a cooperative game in which a questioner identifies an oracle-selected object by asking yes-no questions about a rich image scene. Its question strategies use spatial, visual, and categorical groupings, while the task targets higher-level visual reasoning.
- Game rules: The oracle privately receives an object, while the questioner asks yes-no questions and eventually selects the hidden object from the scene.The questioner sees the whole picture but not the object list until guessing.
- Player roles: Oracle answers are limited to Yes, No, and N/A, with N/A covering ambiguous or visually indeterminate questions.The questioner’s harder role is to generate questions that progressively narrow the candidate objects.
- Question strategies: Questioners can group objects using absolute or relative spatial relations, visual properties such as size, shape, and color, and hierarchical object categories.Examples include locating an object in the bottom left, left of a car, or within the vehicle category.
- Task rationale: The intended reasoning challenge emerges in scenes with enough objects; otherwise, people often enumerate objects by category.The questioner ideally minimizes questions through a binary-search-like strategy that eliminates half the candidates at each step.
3. Related work
GuessWhat?! extends visual-language research toward interactive, goal-directed dialogue. Its dataset is designed to address limitations of single-question or non-grounded conversational tasks by requiring iterative object identification.
- Connections to vision: GuessWhat?! builds on MS COCO’s images and object segmentations while adding dialogue-based interaction around visual scenes.The related-work discussion positions the dataset as an extension of existing image-captioning resources.
- Visual question answering: Unlike VQA systems that may exploit question-answer correlations, GuessWhat?! questions favor spatial understanding and object attributes because the questioner must locate a hidden object.Its binary answers are described as balanced, and it provides more questions per picture on average.
- Referring expressions: Compared with ReferIt’s single referring expression, GuessWhat?! iteratively narrows the target through positive and negative feedback across images containing 3–20 objects.The dataset contains three times more images than the ReferIt datasets.
4. GuessWhat?! Dataset
GuessWhat?! combines carefully filtered MS COCO scenes with interactive crowd-sourced dialogues and analyzes their structure, vocabulary, and success factors. The resulting dataset contains 155,280 dialogues and shows that scene complexity, object properties, and search behavior shape dialogue outcomes.
- Data collection: 77,973 MS COCO images containing 609,543 objects were retained after removing objects smaller than 500px^2 and restricting scenes to 3–20 objects.The authors verified that this selection did not significantly alter the original dataset distribution.
- Data collection: More than 10K AMT workers collected the dialogues through separate questioner and oracle tasks, qualification rounds, bonuses, reporting, and bans.Only dialogues from qualified workers and successful qualification-round games were retained.
- Data analysis: Question counts decrease exponentially within dialogues, while questions per image grow between logarithmically and linearly with the number of objects.Human questioning therefore falls between simply enumerating objects and optimal binary search; additional questions may confirm a choice when oracle errors are possible.
- Data analysis: Questioners use vocabulary covering abstract object properties, spatial locations, visual features, and prepositions expressing relationships between objects.Figure 3c displays word frequencies in a clustered word cloud after manually removing uninformative words.
- Data analysis: 95% success with 3 objects falls to around 70% with 20 objects, while the smallest objects are found only 60% of the time.Objects near image borders and objects from commonly grouped categories, such as bananas or books, are also harder to identify.
5. Baselines
The paper evaluates GuessWhat?! with neural baselines for the oracle and questioner tasks. Oracle performance improves when object information is supplied, while guesser and question-generator results expose the difficulty of dialogue-based object localization.
- Oracle baselines: The oracle predicts a Yes, No, or N/A answer from an image, question, and selected object information.The baseline embeds the image, crop, spatial features, category, and question, concatenates them, and uses an MLP with a softmax output.
- Oracle baselines: The best oracle model uses the question, selected object category, and spatial features.Training keeps VGG parameters fixed and optimizes the LSTM, lookup tables, and MLP with early stopping.
- Oracle baselines: 50.8% error results from always predicting No, while using only the question lowers error to 41.2%.Image-only and crop-only features barely improve over the majority-answer baseline.
- Oracle baselines: 25.7% error is achieved by Question+Crop, compared with 29.2% for Question+Category and 24.7% for their combination.The paper reports that category and crop information are partly redundant, while crop features may contain additional visual information.
- Questioner baselines: The guesser combines image and dialogue representations with category-spatial object embeddings to predict among the image’s objects.Shared MLP parameters allow the model to handle a variable number of objects; LSTM and HRED encode dialogue differently.
- Questioner baselines: 38.7% error is obtained by the guesser on human-generated dialogues, while question-generator models perform between random guessing and human-dialogue performance.Using the oracle’s answers during question-generator training significantly deteriorates performance, and the baseline fixes dialogue length at five questions.
6. Discussion
The paper presents GuessWhat?! as a game-based framework and dataset for multimodal dialogue, with neural baselines across three tasks. It also identifies extensions to other learning and dialogue settings.
- GuessWhat?! introduces a novel framework for multimodal dialogue and a large-scale dataset collected through an engaging game.The authors connect the game format with reduced collection costs for datasets needed by modern algorithms.
- Three tasks based on the questioner and oracle roles receive initial neural baselines and quantitative dataset analysis.
- The dataset could support one-shot learning of new object categories, transfer learning on line-drawing images, and questions in another language.The authors also suggest estimating confidence intervals before the final guess.
A. User interface
The interface supports the two player perspectives through example games and an iterative workflow for correcting and validating questions.
- User interface: The oracle and questioner receive separate instructions before beginning their first game.Figures 8 and 9 present example games from the oracle and questioner perspectives.
- User interface: The examples depict the game interface from both player perspectives rather than a single shared view.
- User interface: Question correction alternates with validation until all questions have been corrected and validated.Validation displays the difference between each original question and its proposed correction.
B. GuessWhat?! samples
The samples illustrate varied scenes, objects, dialogues, and the distinction between GuessWhat?! and ReferIt games using shared MS COCO images and objects.
- GuessWhat?! samples: Examples show that the same picture can be paired with different target objects.
- GuessWhat?! samples: A long dialogue example demonstrates interaction in a visually rich environment.
- GuessWhat?! samples: GuessWhat?! and ReferIt samples use identical objects and images selected from MS COCO for comparison.
C. Additional database statistics
The additional statistics describe word usage, object and spatial distributions, dialogue structure, and success ratios in the GuessWhat?! dataset.
- Word statistics: The 50 most frequent words are compared using a row-normalized co-occurrence matrix and hierarchical clustering with Euclidean distance.
- Object statistics: Thirty prominent object categories account for 71.3% of GuessWhat?! objects, excluding the person category from the visualization.The category and size distributions are compared with those of MS COCO.
- Dialogue statistics: Question lengths follow a Poisson-like distribution, while questioners tend to reuse words during dialogues.
- Dialogue statistics: Topic proportions evolve across a six-question dialogue, with two prominent topics and manually selected topic titles reported.
- Performance statistics: Success ratios are broken down by object category, spatial location, and dialogue length.The statistics include both a spatial heatmap and a dialogue-length histogram.
D. All oracle baselines
This section presents classification errors for oracle baselines and illustrates both correct and incorrect object predictions from the QGen+GT model.
- Table 6 reports classification errors for all oracle baselines.
- Figure 22 shows three QGen+GT dialogue samples in which the correct object was predicted.
- Figure 23 shows three QGen+GT dialogue samples in which the wrong object was predicted.