GuessWhat?! Visual object discovery through multi-modal dialogue

1. Introduction

• Referring expressions can identify one object in a rich scene, but whether they do so unambiguously depends on the listener’s state of mind and scene context.The introduction notes that real-life situations may require multiple exchanges before the intended object becomes clear.Source: §1 · p. 1
• A fundamental challenge for conversational vision systems is understanding natural-language descriptions and grounding them in the visual world.The introduction also motivates conversation as a natural way for people to interact with systems and ask clarifying questions about what they perceive.Source: §1 · p. 1
• Prior visual-language tasks include image captioning, single-question VQA, and ReferIt’s generation of one referring expression.GuessWhat?! brings visual understanding and dialogue together by extending ReferIt into an interactive setting.Source: §1 · pp. 1–2
• Goal-directed dialogue tasks offer task success as an automatic evaluation metric, addressing the lack of dialogue metrics that correlate well with human evaluation.The introduction cites bAbI, movie-dialog recommendation, and restaurant-finding tasks as examples of this approach.Source: §1 · p. 2
• GuessWhat?! extends the ReferIt game into a two-player dialogue task in which success requires understanding object relations and how natural language expresses them.The challenge is to acquire natural language through interaction on a visual task, rather than searching an external database.Source: §1 · p. 2
• The dataset contains 155,280 dialogues, 831,889 question/answer pairs, and 66,537 MS COCO images; the paper defines three sub-tasks and prototypes deep-learning baselines.The dataset is the paper’s key contribution, and the baselines are intended to establish task difficulty.Source: §1 · p. 2

2. GuessWhat?! game

• The questioner asks questions about a shared image, then guesses the hidden object after the object list is revealed; a small per-question penalty encourages informative questions.The oracle is randomly assigned an object, and the game succeeds only if the questioner selects it correctly.Source: §2 · p. 2
• The oracle answers Yes, No, or N/A, while the questioner must generate questions that progressively narrow possible objects, ideally by eliminating half at each step.N/A covers ambiguous questions or answers that cannot be determined from the image.Source: §2 · p. 2
• Questions can group objects by absolute or relative location, or by visual properties such as size, shape, and color.Examples include asking whether an object is at the bottom left, left of a blue car, big, square, or blue.Source: §2 · pp. 2–3
• Questions can also group objects by category taxonomy, such as vehicles, or by interaction, such as whether an object can be driven.Taxonomic questions can refer jointly to categories such as cars and trucks.Source: §2 · p. 3
• The task targets natural-description grounding and higher-level reasoning in sufficiently rich scenes; with fewer objects, people often fall back to linear enumeration by category.Source: §2 · p. 3

Image captioning

• Unlike captioning’s coherent image descriptions and conventional VQA’s question-answering setup, GuessWhat?! requires asking useful questions to locate an object in a scene.The work builds on MS COCO [26], which provides 120k images, more than 800k object segmentations, and five captions per image.Source: §3 · p. 3

VQA datasets

• GuessWhat?! questions favor spatial understanding and object attributes; its binary answers are balanced, and it has twice more questions on average per picture.The paper contrasts this with VQA models that can exploit question-answer correlations and report the same answer regardless of the image.Source: §3 · p. 3

Goal-directed dialogue

• GuessWhat?! is presented as the first large-scale dataset that is both goal-oriented and multi-modal, with dialogues grounded in the current image rather than an external database.The paper contrasts this with dialogue datasets that are small, template-generated, or simulated, and with systems reliant on static databases or external resources.Source: §3 · p. 3

Human computation games

• GuessWhat?! follows human-computation games such as ESP and Peekaboom, which gathered image tags and object segmentations, but addresses a different purpose.Those games were developed when object recognition was in its infancy.Source: §3 · p. 3

ReferIt

• Unlike ReferIt [30] [48], which uses one referring expression to locate an object, GuessWhat?! iteratively narrows it down through positive and negative question feedback.GuessWhat?! selects images with 3–20 objects without class restrictions and has three times more images than ReferIt datasets; its dialogues also favor abstract concepts.Source: §3 · pp. 3–4

4.1. Data collection

• The authors retained 77,973 MS COCO images [26] with 609,543 objects after excluding objects below 500px2 and images with fewer than three or more than twenty objects.They report that this selection did not significantly alter the original dataset distribution.Source: §4.1 · p. 4
• More than 10K AMT participants played the game, with qualification, bonuses, reporting, and dialogue-success requirements used to support cooperative, quality-controlled collection.Questioners and oracles completed separate tasks, and questioners were rewarded slightly more.Source: §4.1 · p. 4

4.2. Data analysis

• The raw dataset contains 155,280 dialogues, 821,889 question-answer pairs, 66,537 images, and 134,073 objects; 84.6% of dialogues are successful.Answers are 52.2% no, 45.6% yes, and 2.2% N/A; dialogues average 5.2 questions, and the dataset distinguishes full, finished, and successful subsets.Source: §4.2 · p. 4
• Question counts per dialogue decrease exponentially, while questions per image object count grow between logarithmically and linearly, consistent with human search strategies between binary and linear search.The authors suggest limited access to the full object list, a preference for linear search, and confirmation questions as possible reasons.Source: §4.2 · pp. 4–5
• Questioners move from abstract terms such as “human” and “furniture” early in dialogues to spatial and visual terms such as “left,” “white,” and “table” later.Prepositions are also heavily used to express relationships between objects.Source: §4.2 · p. 5
• Success falls from 95% with three objects to around 70% with twenty; large and central objects are easier to find than small and border objects, and grouped categories fare worse.The smallest objects are found only 60% of the time.Source: §4.2 · p. 5
• Short dialogues tend to end with a final yes, whereas long dialogues begin with more no answers that gradually decrease.Source: §4.2 · p. 5

4.3. Dataset release

• The dataset assigns 70% of images and their dialogues to training and 15% each to validation and test, ensuring test images were unseen during training.The dataset is available for download at guesswhat.ai/download.Source: §4.3 · p. 5

5.1. Oracle baselines

• The oracle embeds the image, candidate-object crop, spatial features, category, and question, then predicts the answer with an MLP and softmax.The baseline uses fixed VGG features, an LSTM question embedding, and ignores prior question-answer history; it is trained with cross-entropy and early stopping.Source: §5.1 · p. 6
• The best oracle baseline combines the question, object category, and spatial features; including object information reduces error below 31%.Question+Category achieves 29.2% error, Question+Crop 25.7%, and Question+Category+Crop 24.7%; the table identifies Question+Category+Spatial as best.Source: §5.1 · p. 7, Table 2 · p. 7

5.2. Questioner baselines

• The question-generator baseline produces five questions and is evaluated through a pretrained oracle and guesser, with ground-truth answers in training but oracle answers at test time.The authors also train a QGEN+ORACLE variant using oracle answers during training to address this mismatch.Source: §5.2 · pp. 7–8
• On human-generated dialogues, adding VGG image features does not improve LSTM or HRED guessers, and LSTMs perform slightly better than HRED models.The guesser combines image and dialogue representations with object embeddings derived from category and spatial features; shared MLP weights support variable object counts.Source: §5.2 · pp. 7–8, Table 3 · p. 7
• 38.7% error is achieved by the guesser on human-generated dialogues; question-generator models fall between random and human-dialogue performance, and oracle-answer training significantly worsens performance.The reported QGEN evaluation uses the guesser’s error on objects predicted after five generated question-answer rounds.Source: §5.2 · p. 8, Table 4 · p. 8

Discussion

• GuessWhat?! introduces a multi-modal dialogue game and, to the authors’ knowledge, the first large-scale dataset involving images and dialogue.The game greatly decreases the cost of collecting a large dataset for modern algorithms.Source: §6 · p. 8
• The authors introduce three tasks based on the questioner and oracle roles, prototype a neural architecture as a first baseline for each, and quantitatively describe the dataset.They also analyze the baseline results.Source: §6 · p. 9
• The dataset could support applications including confidence intervals before a final guess, one-shot learning of new object categories, transfer learning on line-drawing images [10], and questions in another language.These possibilities include applications based on the game or extensions of the database to other tasks.Source: §6 · p. 9

A. User interface

• Before their first game, oracle and questioner workers received instructions, and the paper shows example games from each role’s perspective.The examples are presented from the oracle’s and questioner’s perspectives, respectively.Source: A. User interface · p. 11, Figure 8 · p. 11, Figure 9 · p. 12
• Figure 8 shows an example game from the oracle’s perspective.The figure is presented left to right and top to bottom.Source: Figure 8 · p. 11
• Figure 9 shows an example game from the questioner’s perspective.The figure is presented left to right and top to bottom.Source: Figure 9 · p. 12
• Workers iteratively corrected questions and validated each proposed correction by comparing it with the original question.The tasks alternated until all questions were corrected and validated.Source: Figure 10 · p. 13

C. Additional database statistics

• The 50 most frequent words are shown in a row-normalized co-occurrence matrix, while word proportions are also tabulated by question depth and dialogue size.The co-occurrence rows are hierarchically clustered using Euclidean distance; the table covers the ten most common words at each depth.Source: Figure 15 · p. 16, Table 5 · p. 20
• 30 prominent object categories account for 71.3% of GuessWhat?! objects, and the figures visualize category and object-size distributions against MS COCO.The person category is omitted for clarity; its counts are 273469 for MS COCO and 188204 for GuessWhat?!.Source: Figure 17 · p. 17, Figure 16 · p. 17
• Question lengths follow a Poisson-like distribution, and questioners tend to reuse the same words during dialogues.The figures also show the percentage of new words appearing along dialogues and word counts per question.Source: Figure 18 · p. 18
• For dialogues of size 6, the topic-model visualization tracks the relative evolution of two prominent topics and their key words.The topics are detected with Data Topic Models, and their titles are manually assigned.Source: Figure 19 · p. 18
• Success ratios are visualized by object category, spatial location in the image, and dialogue length.The supplied captions describe these breakdowns but do not provide specific success-ratio values.Source: Figure 20 · p. 19, Figure 21 · p. 19

D. All oracle baselines

• Table 6 reports classification errors across all oracle baselines.Source: Table 6 · p. 21

• Figure 22 shows three QGen+GT dialogue samples in which the correct object was predicted.Source: Figure 22 · p. 22
• Figure 23 shows three QGen+GT dialogue samples in which the wrong object was predicted.Source: Figure 23 · p. 23