Source-linked AI summary
Visual Question Generation as Dual Task of Visual Question Answering
Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang
TL;DR
VQA and VQG were explored separately despite their complementary question-answer relationships in images. iQAN unifies them through invertible fusion, shared parameters, and joint training, improving results across VQA and VQG tasks on CLEVR and VQA2. The framework also supports question augmentation from cheaply labeled answers, although answer-only inputs such as yes/no constrain VQG usefulness.
Problem
VQA and VQG were studied separately despite both reasoning between image content, questions, and answers.
Method
iQAN jointly trains VQA and VQG using invertible cross-modal fusion, shared visual and sequence-model components, and duality regularization.
Results
iQAN outperforms baseline models on VQA2 and CLEVR, generalizes to other VQA architectures, and consistently improves VQA and VQG results.
Takeaways & Limitations
VQG can augment VQA training data by generating questions from cheaply labeled answers, extending the framework beyond dual-task training.
Takeaways & Limitations
On full VQA2, gains are small or negative because yes/no answers provide too little information for expected-question generation, allowing VQG loss to dominate training.
Abstract
from arXiv · showhide
Recently visual question answering (VQA) and visual question generation (VQG) are two trending topics in the computer vision, which have been explored separately. In this work, we propose an end-to-end unified framework, the Invertible Question Answering Network (iQAN), to leverage the complementary relations between questions and answers in images by jointly training the model on VQA and VQG tasks. Corresponding parameter sharing scheme and regular terms are proposed as constraints to explicitly leverage Q,A's dependencies to guide the training process. After training, iQAN can take either question or answer as input, then output the counterpart. Evaluated on the large-scale visual question answering datasets CLEVR and VQA2, our iQAN improves the VQA accuracy over the baselines. We also show the dual learning framework of iQAN can be generalized to other VQA architectures and consistently improve the results over both the VQA and VQG tasks.
Introduction
VQA and VQG address inverse image-based reasoning between questions and answers, but were previously studied separately. iQAN unifies them through shared components, invertible fusion, and joint dual-task training.
- VQA answers image-based questions, whereas VQG generates questions from an image and a given answer.
- Both tasks share visual input and an encoder-fusion-decoder pipeline while using questions and answers in reverse order.
- The proposed iQAN jointly trains VQA and VQG as dual tasks in a unified framework that can perform either task at test time.
- iQAN uses an invertible Dual Mutan fusion module to infer answer features from image-plus-question or question features from image-plus-answer.
- Shared visual, encoder, and decoder parameters plus duality regularization explicitly exploit question-answer complementarity during joint training.
Related Work
Prior work developed VQA, VQG, and dual-learning methods separately. This paper positions iQAN as the first framework applying dual learning to VQA and VQG together.
- VQA research has advanced encoder-decoder models, visual and language attention, multimodal fusion, and benchmark datasets such as VQA2 and CLEVR.
- VQG research introduced image-question datasets and models using CNN-LSTM generation or region captions to guide visually grounded questions.
- Earlier dual-learning work used cycle consistency or mutual guidance in machine translation, question answering, and image translation.
- The paper identifies no prior work applying dual learning to VQA and presents its VQA-VQG formulation as the first such attempt.
(iQAN)
iQAN unifies VQA and VQG as inverse processes, using shared multimodal components, a Dual Mutan, and duality regularization to jointly learn question–answer representations.
- Framework overview: iQAN contains VQA and VQG components that process questions and answers in reverse directions using shared visual, attention, and fusion modules.The VQA branch predicts answers from questions, while the VQG branch generates questions from answers and images.
- VQA component: The VQA branch encodes questions with an LSTM, attends to image features, fuses them with Mutan, and predicts answer features with a classifier.The Mutan fusion module models interactions among transformed question and visual features before answer classification.
- VQG component: The VQG branch retrieves answer embeddings, combines them with attended image features, and decodes predicted question features into word sequences with an LSTM-based model.Beam search is used during inference, and the branch shares the visual CNN and Mutan-based attention with VQA.
- Parameter sharing: The model shares parameters across the primal and dual transformations, including visual, question, answer, and core-tensor projections.This sharing unifies the two Mutan modules while preserving their different dimension arrangements.
- Dual Mutan: Dual Mutan performs question-to-answer and answer-to-question inference with one shared kernel by alternating the mode-1 input.The two forms share projection matrices and the core tensor, while the output projections are integrated with the decoders.
- Duality Regularizer: Duality regularization minimizes smoothL1 discrepancies between predicted and target question or answer representations, linking VQA and VQG during joint training.The regularizer also supplies soft targets for question and answer feature learning.
Experiments
The experiments evaluate iQAN and its dual-training components on filtered and full VQA2, CLEVR, other VQA architectures, and VQG-based data augmentation. Results show strongest VQA gains on filtered VQA2 and CLEVR, generalization across architectures, and a trade-off between VQA improvement and VQG quality when generated questions augment training data.
- Evaluation Setup: The experiments use filtered VQA2 and CLEVR, with Acc@1 and Acc@5 for VQA and sentence-level BLEU for VQG.The implementation uses a fixed ResNet-152 visual backbone and evaluates generated questions with BLEU.
- Component Analysis: 0.77% on top-1 accuracy: the full model improves over the separated baseline on filtered VQA2.Adding Dual Mutan, duality regularization, and encoder/decoder sharing progressively improves performance.
- Component Analysis: Full VQA2 yields little improvement, while intermediate models can underperform the baseline because yes/no answers provide insufficient information for question generation.The resulting VQG loss can dominate training and deteriorate VQA performance.
- Component Analysis: 1.33% gain on overall Acc@1: dual training improves VQA reasoning performance on CLEVR.CLEVR is designed to diagnose VQA reasoning ability, and the authors connect the gain to training the model to understand and generate related questions.
- Dual Learning for Other VQA Models: Dual training generalizes to other VQA architectures and brings concordant improvements across the evaluated models.The scheme requires shared encoders and decoders plus a duality regularizer; improvement for VQG can remain limited when feature representations differ or baseline VQG is already strong.
- Augmenting VQA with VQG: VQG-generated questions can augment answer-only data to improve VQA, but they can deteriorate VQG performance because generated questions follow a different distribution.Pretraining with dual training on the augmented data and fine-tuning on paired data significantly outperforms vanilla dual-trained models.
Conclusion
The paper presents iQAN as a dual training scheme that unifies VQA and VQG in one model. Experiments report gains over baselines on VQA2 and CLEVR and improvements across several VQA architectures.
- iQAN jointly trains VQA and VQG as conjugate tasks in a single model.
- The dual training scheme reconstructs VQA models to perform VQG and extends beyond the Mutan architecture.
- Dual-trained models outperform baselines on both VQA2 and CLEVR datasets.
- The method consistently brings gains to several recent VQA models.
- VQG-generated questions can leverage cheaply labeled answers to augment training and improve VQA and VQG models.
Supplementary Materials: Qualitative Results
The qualitative results show jointly trained VQA and VQG models recovering image-question-answer relationships, while also revealing failures from incorrect generation, visual recognition, and difficult fine-grained questions.
- Both VQA and VQG are jointly trained with the iQAN dual training scheme on the VQA2 validation set.
- Figure 4 marks correct counterpart predictions in green and failure cases in red.
- The VQA model associates questions and images to find answers, while VQG generates questions corresponding to given answers.
- Failures include mismatched generated questions, incorrect visual answers, alternative answers, and questions involving fine-grained categories or tiny objects.
- Generated questions are generally similar to human annotations and correspond to their answers, supporting their use as augmented VQA data.