Source-linked AI summary
Creativity: Generating Diverse Questions using Variational Autoencoders
Unnat Jain, Ziyu Zhang, Alexander Schwing
TL;DR
Visual question generation seeks diverse, plausible questions for an image because conventional approaches often do not address creativity and engagement across education, assistants, and entertainment. The paper combines variational autoencoders with LSTM cells, sampling latent representations and decoding them with image features. Across VQG-COCO, VQG-Flickr, and VQG-Bing, the framework generates remarkably diverse questions from a single image, while remaining limited by small question datasets and co-occurrence-based errors.
Problem
Visual question generation needs methods that produce diverse, plausible questions rather than only a single or reference-like question for each image.
Method
The method combines variational autoencoders with LSTM cells, sampling a latent vector and decoding it with the image representation to generate questions.
Results
The framework generates a series of remarkably diverse questions from single images across VQG-COCO, VQG-Flickr, and VQG-Bing.
Takeaways & Limitations
The framework produces literal-to-inferential and unseen questions, including questions requiring scene understanding and prior information about objects.
Takeaways & Limitations
The VQG dataset is very small, making training larger generative models for the problem challenging.
Abstract
from arXiv · showhide
Generating diverse questions for given images is an important task for computational education, entertainment and AI assistants. Different from many conventional prediction techniques is the need for algorithms to generate a diverse set of plausible questions, which we refer to as "creativity". In this paper we propose a creative algorithm for visual question generation which combines the advantages of variational autoencoders with long short-term memory networks. We demonstrate that our framework is able to generate a large set of varying questions given a single input image.
1. Introduction
The paper frames creativity as important when existing representations and rules are insufficient, motivating visual question generation that produces diverse questions from images. It proposes a VAE-based method that samples a latent representation and decodes it with image features to generate novel questions.
- Creativity matters when existing knowledge structures, logical rules, observations, or representations cannot fully specify the desired outcome.
- Visual question generation supports education, AI assistants, and entertainment by exposing users to novel aspects rather than repeating handcrafted questions.
- The proposed method embeds questions and corresponding image features into a low-dimensional latent space, then samples and decodes with image features during inference.
- The framework combines variational autoencoders with LSTM cells to generate diverse questions from a single input image.
- The approach is evaluated on VQG-COCO, VQG-Flickr, and VQG-Bing, where it generates a series of remarkably diverse questions from images.
2. Related Work
Prior visual question generation methods either produce one question per image or rely on complicated sampling procedures for diversity. This paper instead samples in an encoding space and maps those samples to question representations with LSTM networks.
- Visual question generation complements visual question answering and requires creativity because multiple reasonable questions can correspond to one image.
- Earlier work generated a single question per image, while diverse beam search produced varied questions by sampling from a complicated energy landscape.
- The proposed approach samples from a distribution in encoding space and maps the encodings to high-dimensional representations using LSTM networks.
- Classical generative models can struggle with the high ambiguity inherent in images, motivating more flexible deep generative approaches.
- GANs and VAEs are successful deep generative modeling approaches, but their training often depends on multiple practical tricks.
3. Approach
The approach uses LSTM-based VAE encoder and decoder networks to map image-question pairs into a latent space and generate questions by sampling and decoding latent representations with image features.
- VAE background: The VAE objective uses a lower bound on log-likelihood and approximates expectations with samples from the encoder distribution.The reparameterization trick permits differentiation through sampling when jointly optimizing encoder and decoder parameters.
- Learning: The model relies on variational autoencoders with LSTM encoder and decoder modules for visual question generation.The encoder models q and the decoder models p, with parameters learned jointly.
- Q-distribution: The encoder combines an image feature and question word embeddings, then maps the final hidden state to Gaussian mean and log variance.The image is provided first, followed by word embeddings; two linear transformations produce the latent distribution parameters.
- Evaluation: Uniform latent-space sampling yields better oracle scores, whereas normal sampling yields better average BLEU and METEOR metrics.These results are reported for the VQG-COCO accuracy evaluation; corresponding Flickr and Bing tables are in the supplementary material.
- P-distribution: The decoder reconstructs questions from image features, a sampled latent vector, and sequential word embeddings.During inference, the latent sample is drawn from N(0, 1); during training, it is shifted and scaled using encoder outputs.
- P-distribution: The decoder transforms each hidden representation into vocabulary logits and applies softmax to predict the next word.Training maximizes the predicted log-probability of the next word across the LSTM sequence.
4. Experiments
The experiments evaluate visual question generation across VQG datasets using accuracy, oracle, and diversity metrics, alongside qualitative diversity and failure-case analyses. The study also addresses the small size of VQG data through combined VQA and VQG training data.
- Datasets: The VQG datasets contain roughly 5,000 images with five questions per image, split into 50% training, 25% validation, and 25% test.
- Datasets: The small VQG dataset makes training larger generative models for natural question generation challenging.
- Datasets: The combined VQA+VQG training set supplies more data for the LSTM-based VAE while retaining additional diversity from VQG questions.
- Metrics: Evaluation uses BLEU, METEOR, oracle metrics, and a proposed diversity score to assess accuracy and diversity of generated questions.
- Results: The model generates diverse questions across COCO, Flickr, and Bing, including literal and inferential questions, while sunburst plots visualize word-level diversity.
- Results: Failure cases arise when visual recognition misidentifies objects or frequent training-set co-occurrences prompt questions about absent greens, birds, or other concepts.
5. Conclusion
The paper combines variational autoencoders with LSTM cells to generate diverse questions from a single image. It demonstrates applicability across varied images and identifies computational education, entertainment, and driving assistants or chatbots as potential domains.
- The framework combines variational autoencoders with LSTM cells to generate a diverse set of questions from one input image.
- The authors demonstrate the framework on a diverse set of images and envision applications in computational education, entertainment, driving assistants, and chatbots.