Source-linked AI summary
Yin and Yang: Balancing and Answering Binary Visual Questions
Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, Devi Parikh
TL;DR
The paper addresses binary VQA where language priors can support superficial answers without detailed visual understanding. It balances abstract-scene data with complementary pairs and verifies question-derived visual concepts using tuple-based language and image attention. The approach outperforms language-only and state-of-the-art VQA baselines on the balanced dataset while matching the state-of-the-art approach on the unbalanced dataset.
Problem
Binary VQA can appear strong because language priors provide answers without requiring detailed visual understanding.
Method
The paper balances abstract VQA with complementary scene pairs and converts each question into a tuple whose visual concept is verified in the image.
Results
The approach matches a state-of-the-art VQA approach on the unbalanced dataset and outperforms it on the balanced dataset.
Takeaways & Limitations
Balanced complementary scenes make language priors insufficient and require models to use visual information to answer binary questions.
Takeaways & Limitations
The dataset is based on abstract scenes, whose fine-grained semantic differences trade off against the coarser differences available in real images.
Abstract
from arXiv · showhide
The complex compositional structure of language makes problems at the intersection of vision and language challenging. But language also provides a strong prior that can result in good superficial performance, without the underlying models truly understanding the visual content. This can hinder progress in pushing state of art in the computer vision aspects of multi-modal AI. In this paper, we address binary Visual Question Answering (VQA) on abstract scenes. We formulate this problem as visual verification of concepts inquired in the questions. Specifically, we convert the question to a tuple that concisely summarizes the visual concept to be detected in the image. If the concept can be found in the image, the answer to the question is "yes", and otherwise "no". Abstract scenes play two roles (1) They allow us to focus on the high-level semantics of the VQA task as opposed to the low-level recognition problems, and perhaps more importantly, (2) They provide us the modality to balance the dataset such that language priors are controlled, and the role of vision is essential. In particular, we collect fine-grained pairs of scenes for every question, such that the answer to the question is "yes" for one scene, and "no" for the other for the exact same question. Indeed, language priors alone do not perform better than chance on our balanced dataset. Moreover, our proposed approach matches the performance of a state-of-the-art VQA approach on the unbalanced dataset, and outperforms it on the balanced dataset.
1. Introduction
The paper targets binary VQA under language priors that can produce superficially strong answers without detailed visual understanding. It uses abstract scenes and complementary scene pairs to make visual verification essential.
- Dataset design: Abstract scenes enable balanced VQA data by pairing visually similar scenes with opposite answers to the same question.This design controls language priors while preserving fine-grained visual differences.
- Motivation: Language priors can yield strong image-captioning and VQA performance without models understanding the relevant visual details.Examples include stereotyped captions and frequent answers such as “tennis” or “white.”
- Problem formulation: Binary questions can be treated as visual verification of a concept described by the question.The concept is represented as a concise tuple; its presence yields “yes,” otherwise “no.”
- Contributions: The paper proposes balancing an existing abstract binary VQA dataset and representing questions with tuples that describe the visual concept to verify.These are presented as the paper’s main contributions.
2. Related work
Prior work spans visual question answering across constrained, free-form, synthetic, and real-image settings, while this paper uses abstraction to control semantic data distributions.
- Visual question answering: VQA research includes constrained and free-form questions and answers across image, video, synthetic, and real-world settings.The cited work covers datasets and methods for several forms of multimodal question answering.
- Visual abstraction + language: Abstract scenes have been used to study high-level semantics and links between language, common sense, interactions, and visual features.Prior studies include scene description, generation, common-sense capture, interaction modeling, and semantic feature importance.
- Dataset balancing: This work augments an abstract VQA dataset with human-collected scenes to equalize “yes” and “no” answers where possible.The approach is framed as semantic data augmentation, motivated by the difficulty of balancing real-image VQA concepts.
3. Datasets
The dataset uses an abstract clipart library and human-created counterfactual scenes to balance binary VQA answers through minimal visual changes.
- Abstract library: The abstract scene library contains human models, objects, animals, adjustable poses, and separate indoor and outdoor scene types.The backgrounds distinguish indoor from outdoor scenes.
- Complementary scenes: The balancing task asks workers to modify an abstract scene so its associated binary answer changes from “yes” to “no” or vice versa.Workers are instructed to alter the clipart scene as little as possible.
- Complementary scenes: Minimal modifications create complementary scenes that function as hard negatives or positives for learning subtle visual differences.The paper also connects this contrastive construction to changing one thing at a time in pedagogy.
- Dataset limitations: 5.93% of scene-question pairs could not be modified to flip the answer, while 14.55% had disagreement or unsuccessful intended labels.These exclusions reflect limitations of scene modification and human annotation consistency.
4. Approach
The approach parses each binary question into a primary object, relation, and secondary object, aligns those arguments to scene objects, and verifies the resulting concept using fused language and image features.
- 4. Approach: The pipeline has two stages: language parsing into a <P, R, S> tuple and visual verification of whether that concept appears in the image.The primary object is always present, while relation or secondary-object arguments may be absent.
- 4.1. Tuple extraction: Tuple extraction parses questions, removes stop words and leading material, and divides the resulting summary into primary, relation, and secondary arguments.The intended arguments are noun phrases for P and S and a verb phrase or preposition for R.
- 4.2. Object alignment: Object alignment selects image objects for P and S using mutual information between argument words and clipart-object occurrences.For multiple instances, the method assigns an instance randomly; absent referenced objects require a fallback object assignment.
- 4.3. Visual verification: The Q-model encodes the full question with an LSTM, whereas the Tuple-model encodes P, R, and S word embeddings as alternative language representations.Both models share image features and form an ensemble.
- 4.3. Visual verification: Image and language features are mapped into a common 256-dimensional space, multiplied elementwise, and classified into “yes” and “no” scores.The Q-model and Tuple-model predictions are multiplied for the final prediction.
5. Experiments
The experiments compare language-only, holistic-vision, and attention-based models on unbalanced and balanced VQA settings. Results show that visual information and question-directed attention matter most when language biases are controlled.
- Evaluation on the balanced dataset: On the balanced dataset, Q+Tuple+A-IMG outperforms all baselines by a large margin.The balanced setting reduces language priors and makes visual understanding essential.
- Evaluation on the balanced dataset: Models using visual information tend to outperform blind models when trained on the balanced dataset.Balanced training forces models to focus on visual understanding rather than language priors.
- Evaluation on the balanced dataset: Complementary-scene pairs require both answers to be correct, making language-only models score zero because they predict the same answer for both scenes.The pair-level evaluation directly tests whether models distinguish subtle visual differences associated with opposite answers.
- Evaluation on the balanced dataset: The attention model outperforms Q+Tuple+H-IMG by attending to relevant image regions rather than using generic holistic descriptors.This comparison tests whether question-directed regions and object interactions improve performance.
- Analysis: 79.2% VQA accuracy is achieved with imperfect tuple extraction and alignment, versus 81.06% with perfect intermediate stages.Imperfect tuple extraction and alignment account for approximately a 2% loss in VQA accuracy.
- Ablation Study: 78.9% VQA accuracy for Q+Tuple exceeds 77.87% for Q-only and 77.54% for Tuple-only on unbalanced data.Combining question and tuple language features performs better than either feature type alone in this ablation.
6. Discussion
The paper highlights a trade-off between abstract and real images for balanced VQA: abstract scenes simplify low-level recognition but enable finer-grained semantic differences.
- Abstract scenes make complementary pairs fine-grained, forcing models to learn subtle visual differences.Real-image pairs would generally differ more coarsely and therefore be easier for visual models.
- Real images pose greater low-level recognition difficulty, whereas abstract scenes make those recognition tasks trivial.
- Balancing VQA datasets can be generalized to real images by finding images with different answers for the same question.
7. Conclusion
The paper balances binary VQA on abstract scenes with complementary scene pairs and proposes visual verification using question tuples and relevant image regions. The approach outperforms language-prior and state-of-the-art baselines on the balanced dataset.
- The balanced dataset gives nearly every question a “yes” scene and a closely related “no” scene, requiring image understanding.The authors plan to make the balanced dataset publicly available.
- The proposed model summarizes questions as tuples, identifies relevant scene regions, and verifies the corresponding visual concept.
- The approach outperforms both a language-prior baseline and a state-of-the-art VQA approach by a large margin on the balanced dataset.Qualitative results show attention to relevant scene parts during answering.
A. Role of language priors
The abstract VQA dataset contains language and clipart-library biases that enable answer prediction without full visual reasoning. N-gram and language-only baselines achieve high accuracy on the unbalanced data.
- 75.11% VQA accuracy is achieved by the n-gram baseline, versus 68.67% for the PRIOR “Yes” baseline.The n-gram model memorizes the most common answer for question-starting n-grams and defaults to “yes” when an n-gram is unseen.
- The clipart library makes “no” always correct for “is the door open?” and “is it raining?”, while “is it daytime?” receives 100% “yes” accuracy.
- 78.9% VQA accuracy is achieved by the language-only LSTM model, which can remember meaningful question words and dataset biases.
- The abstract library contains indoor and outdoor scene types with setting-specific clipart objects, including humans, pillows, TVs, eagles, bikes, and footballs.Indoor scenes use living/dining-room backgrounds, while outdoor scenes use park backgrounds.
C. Dataset collection
The dataset-collection section documents the AMT interface and supporting examples used to create complementary scenes for the balanced dataset.
- The AMT interface provided full worker instructions for collecting complementary scenes.
- Workers were shown good and bad examples to clarify the complementary-scene collection task.
- The paper presents sample complementary scenes from the balanced dataset in Figure 9.
D. Qualitative results
The paper presents qualitative results of its approach, including identified failure cases.
- Figure 10 presents qualitative results of the approach.
- The qualitative results include failure cases.
- Primary and secondary objects are marked with red and blue boxes, respectively.
E. Issue of Negation
The approach has difficulty handling negation because it focuses on meaningful words and may map positive and negative questions to the same tuple. The image representation includes object, spatial, and scene-level features.
- Issue of Negation: Negative questions are poorly handled because the approach focuses on meaningful words in the question.
- Issue of Negation: “Is the cat on the ground?” and “Is the cat not on the ground?” produce the same tuple, although their answers should be opposite.
- Image Features: The image features comprise primary-object, secondary-object, relative-location, and scene-level features.
- Image Features: Primary- and secondary-object features each have 563 dimensions, while relative-location and scene-level features have 48 and 258 dimensions.
G.1. Pre-processing
The preprocessing pipeline normalizes questions, removes nonmeaningful phrases, parses grammatical structure, and extracts summaries before splitting them into primary objects, relations, and secondary objects.
- Question normalization: The pipeline keeps letters and digits, removes nonmeaningful question phrases, standardizes capitalization, and appends a question mark.
- Syntactic parsing: Processed questions are parsed with the Stanford parser, which assigns grammatical entities to words.
- Summary extraction: When a nominal subject exists, the pipeline uses noun and pronoun tags to select the meaningful question fragment, excluding stop-list pronouns.
- Summary extraction: Without a nominal subject, the pipeline finds the first noun or non-stop-list pronoun in root-, nmod-, or nummod-initial questions.
- Syntactic parsing: The method retains words from a defined entity list and removes words whose entities are not included.
- Tuple extraction: Extracted summaries are split into primary object, relation, and secondary object components using grammatical entities, noun phrases, object lists, and location lists.
- Tuple extraction: The method includes adjectives with secondary-object nouns when a noun follows, while otherwise retaining the nouns as secondary objects.