Source-linked AI summary
Generating Natural Questions About an Image
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, Lucy Vanderwende
TL;DR
The paper addresses the gap between literal image descriptions and natural questions involving commonsense inference and abstract events. It introduces VQG, builds three datasets, and evaluates generative and retrieval models; the models produce reasonable questions but remain well below human performance.
Problem
Existing vision-language tasks mainly describe visible image content, leaving questions about inferred events, states, and commonsense concepts less explored.
Method
The paper defines VQG as generating a natural, engaging, non-visually-verifiable question from an image and constructs three datasets spanning object-centric and event-centric images.
Results
Models generate reasonable questions across varied images, while end-to-end deep neural models outperform other approaches on the most challenging dataset and still trail human performance.
Takeaways & Limitations
VQG establishes a new challenge for connecting visual input with commonsense knowledge and pragmatics through natural question generation.
Takeaways & Limitations
The paper notes that visual features may detect familiar objects but fail to make sense of combinations of previously unseen concepts.
Abstract
from arXiv · showhide
There has been an explosion of work in the vision & language community during the past few years from image captioning to video transcription, and answering questions about images. These tasks have focused on literal descriptions of the image. To move beyond the literal, we choose to explore how questions about an image are often directed at commonsense inference and the abstract events evoked by objects in the image. In this paper, we introduce the novel task of Visual Question Generation (VQG), where the system is tasked with asking a natural and engaging question when shown an image. We provide three datasets which cover a variety of images from object-centric to event-centric, with considerably more abstract training data than provided to state-of-the-art captioning systems thus far. We train and test several generative and retrieval models to tackle the task of VQG. Evaluation results show that while such models ask reasonable questions for a variety of images, there is still a wide gap with human performance which motivates further work on connecting images with commonsense knowledge and pragmatics. Our proposed task offers a new challenge to the community which we hope furthers interest in exploring deeper connections between vision & language.
1 Introduction
The paper introduces Visual Question Generation (VQG) to move beyond literal image descriptions by generating natural, engaging questions about inferred events and states. It contributes datasets and evaluation methods, while finding promising model-generated questions still lag human performance.
- Task motivation: Natural questions often concern inferred abstract events or states, such as an accident or injury, rather than directly visible objects.This contrasts with automatically generated captions, which provide literal descriptions of image objects.
- Task motivation: VQG asks a system to generate a natural and engaging question from an image, excluding questions answerable by visually inspecting the image alone.The task targets questions that can engage a human in conversation rather than questions designed only to test visual recognition.
- Contributions: The paper argues that asking relevant questions indicates understanding and can support dynamic learning and conversation.Question generation is presented as more than syntactic transformation because deciding what to ask demonstrates understanding.
- Contributions: The paper creates three datasets containing 75,000 questions across object-centric and event-centric images, including abstract terms such as events and states.The datasets are intended to broaden VQG beyond the object-focused data common in image captioning.
- Results: End-to-end deep neural models outperform other approaches on the most challenging dataset, but generated questions remain substantially behind human performance.The paper also reports that ΔBLEU strongly correlates with human judgments for automatic evaluation.
2 Related Work
Prior vision-language datasets largely emphasize objects, visually verifiable answers, or human-oriented actions, whereas VQG targets natural questions about less directly observable events and concepts.
- Vision-language datasets: Image-captioning datasets primarily focus on objects, while event datasets often lack textual event descriptions and cover relatively few human-oriented actions.The paper instead targets static-image events such as fire, explosion, and snowing.
- Visual question answering: VQA asks systems to answer image questions, whereas VQG asks systems to generate questions that interest people in conversation.CQA automatically transforms captions into questions whose answers are limited to objects, numbers, colors, or locations.
- Textual question generation: Existing question-generation work studies grammaticality and content focus from textual input using templates, fill-in-the-blank methods, or crowdsourcing.VQG extends the question-generation setting to visual input.
3 Data Collection Methodology
The paper constructs three VQG datasets by crowdsourcing natural, potentially conversational questions over object-centric and event-centric images, then analyzes their linguistic and task properties. The datasets total 15,000 images and 75,000 questions, while the Bing images also expose limitations of existing captioning systems.
- Task definition: VQG excludes visually verifiable questions and instead targets natural questions that can engage a human in conversation.Questions about image-visible attributes, such as object counts or colors, are outside the task scope.
- Image and question collection: Questions were crowdsourced on Amazon Mechanical Turk, beginning with 5,000 MS COCO images shared with VQA and CQA for comparison.This dataset is named V QGcoco−5000; the paper also uses Flickr and Bing image collections to broaden coverage.
- Annotation analysis: VQG questions show greater formulation diversity than VQA and CQA, frequent verbs, more abstract concepts, and high inter-annotator textual similarity.The analysis compares object mentions, vocabulary size, verb part of speech, abstract terms, and BLEU-based inter-annotation similarity.
- Annotation analysis: The MS COCO-derived dataset remains conceptually limited by its 91 predefined object categories, with event terms such as ‘wedding’, ‘injured’, and ‘accident’ rare.This limitation motivated collecting V QGFlickr−5000 from images embedded in story-like photo albums.
- Event-centric collection: The Bing dataset was built from 1,200 event-centric search queries, producing 5,000 filtered images whose frequent vocabulary includes ‘happen’, ‘work’, and ‘cause’.Queries combined WordNet event and process hyponyms, TimeBank events, and manually curated stereotypical events; graphics and cartoons were filtered by crowdsourcing.
- Dataset statistics: Together, the three datasets contain 15,000 images and 75,000 questions, with an average VQG question length of 6 tokens.The datasets cover a broad range of visual concepts and events.
- Captioning evaluation: Captioning performance drops substantially on Bing images compared with MS COCO, confirming that Bing covers a new image class for existing captioning systems.The comparison uses the MSR captioning system and standard BLEU and METEOR metrics on CaptionsBing−5000 and MS COCO.
- Dataset release: The paper releases an extended VQG dataset to encourage research on more end-goal-oriented vision-and-language tasks.The release is presented as a resource for the research community.
4 Models
The paper compares generative and retrieval approaches for VQG, including neural generation from image features and nearest-neighbor question reuse. Retrieval uses distance and semantic-similarity controls to select candidate questions.
- Generative models: GRNN transforms 4096-dimensional fc7 image features into a 500-dimensional initial state for a 500-dimensional GRU.The decoder produces the question sequentially and uses beam search of size 8.
- Generative models: Three generative models are presented for VQG: MELM, MT sequence-to-sequence translation, and GRNN recurrent generation.MELM predicts candidate words before language modeling, MT translates image descriptions into questions, and GRNN generates questions word by word from transformed visual features.
- Retrieval models: Retrieval models construct candidate pools from K nearest-neighbor images using fc7 features, then select semantically representative questions.The one-best question for each retrieved image is the question with the highest semantic similarity to the other four questions.
- Retrieval models: Dynamic retrieval uses max-distance to exclude noisy neighbors and min-distance to let a highly similar image become the sole candidate.The reported frequencies of a training image within distance 0.1 are 7.68% for COCO, 8.4% for Flickr, and 3.0% for Bing.
- Retrieval models: The final retrieval settings include 1-NN with K=1 and K-NN+min with K=30, max-distance = 0.35, and min-distance = 0.1.Candidate-question selection also uses Smoothed-BLEU and Average-Word2Vec similarity measures.
5 Evaluation
The evaluation combines human ratings with automatic similarity metrics across three VQG datasets. The authors report that ∆BLEU strongly correlates with human judgment, while BLEU is a practical proxy when per-reference ratings are unavailable.
- Human evaluation: Human evaluation uses three crowd workers who rate candidate questions on a three-point semantic scale.Human evaluation is presented with multiple system hypotheses simultaneously to support calibrated judgments.
- Automatic evaluation: Automatic evaluation measures hypothesis-reference similarity with BLEU, METEOR, and ∆BLEU.BLEU uses equal weights through 4-grams, while METEOR uses its default version 1.5 setting.
- Metric results: ∆BLEU strongly correlates with human judgment, and BLEU also shows strong correlations across the reported Pearson, Spearman, and Kendall analyses.The authors recommend ∆BLEU as the main evaluation metric and BLEU for benchmarking or optimization when per-reference ratings are unavailable.
- Experimental setup: The three VQG-5000 datasets are split randomly into train, validation, and test sets at 50%, 25%, and 25%.Models are trained both on all training sets and on the corresponding dataset alone.
- Model results: Humanconsensus consistently scores higher than Humanrandom, indicating shared intuition about the most natural question for an image.GRNNall performs best in 2/3 of human-evaluation runs, while GRNNX outperforms other models on Bing under all three automatic metrics.
6 Discussion
The paper positions VQG as a task for generating natural questions about images, especially questions involving event-centric concepts. It identifies generalization to unseen concept combinations as a limitation and points toward conversational extensions.
- Discussion: VQG asks systems to generate natural questions from images, with the Bing dataset emphasizing event-centric concepts such as cause, event, and happen.The authors encourage reporting results on the Bing test set using ∆BLEU.
- Discussion: The Bing dataset is described as the most challenging and as difficult for state-of-the-art captioning systems.Its challenge centers on generating questions about events and other abstract concepts rather than only literal visual content.
- Limitations: The authors identify difficulty generalizing to previously unseen combinations of concepts as a limitation of current systems.They note that visual features may detect individual objects without making sense of their combination.
- Future directions: Future work includes larger training data, models that generalize to unseen concepts, and conversational question generation using context and history.The paper frames VQG as a step toward systems that can naturally start conversations with engaging questions.