GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Drew A. Hudson; Christopher D. Manning

Source-linked AI summary

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

Drew A. Hudson, Christopher D. Manning

arXiv:1902.09506v3Feb 25, 2019cs.CLcs.AIcs.CVcs.LG

TL;DR

VQA benchmarks can be unreliable because answer priors and simple questions allow models to avoid genuine visual reasoning. GQA constructs a large, scene-graph-grounded dataset with functional programs, balanced answer distributions, and complementary behavioral metrics. It reports low baseline performance relative to human performance and positions the benchmark for studying deeper, more robust reasoning.

Problem
Existing VQA benchmarks contain question-conditional biases and simple questions that can let models circumvent true visual scene understanding.
Method
GQA combines scene graphs, linguistic structure, and functional programs to generate diverse grounded questions, balance answer distributions, and evaluate consistency, grounding, plausibility, and related properties.
Results
Baseline models achieve 17.82%–41.07% on GQA, while the paper reports fine-grained analyses of model behavior and performance.
Takeaways & Limitations
GQA is presented as a benchmark for deeper semantic understanding, sound reasoning, enhanced robustness, and improved consistency in VQA.
Takeaways & Limitations
GQA focuses on factual, image-answerable questions and multi-hop reasoning, excluding some intention, OCR, and external-knowledge question types.

Abstract

from arXiv · show

We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages scene graph structures to create 22M diverse reasoning questions, all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate question biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. An extensive analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%, offering ample opportunity for new research to explore. We strongly hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding for images and language.

1. Introduction

Existing VQA benchmarks can reward answer-distribution guesses and simple recognition rather than visual understanding. GQA addresses these weaknesses with diverse, compositional questions grounded in scene graphs, balanced answer distributions, and richer evaluation metrics.

1. Introduction: Existing VQA benchmarks contain strong real-world priors and simple questions that let models bypass visual scene understanding.These vulnerabilities undermine the reliability of benchmark scores as measures of visual understanding.
1. Introduction: GQA generates over 22 million diverse questions by combining Visual Genome scene graphs with a linguistic grammar and functional programs.The programs specify question contents and semantics while grounding questions in image scene graphs.
1. Introduction: GQA uses semantic representations to reduce question biases and creates a 1.7M balanced dataset with tunable smoothing for open and binary answers.The balancing strategy makes educated guessing less rewarding while preserving varied reasoning skills and multi-step inference.
1. Introduction: The dataset adds consistency, validity, plausibility, grounding, and distribution metrics alongside standard accuracy.These measures target anomalous behaviors such as ignoring question words or attending to irrelevant image regions.
1. Introduction: GQA combines crisp semantic representations with the semantic and visual richness of real-world images as a resource for visual reasoning.Its stated contributions include the dataset, a scene-graph-based question-generation method, and metrics for analyzing model success and failure.

2. Related Work

Prior approaches address VQA bias through synthetic data, paired images, or adversarial answer distributions, but each leaves trade-offs in realism, diversity, or fairness. GQA instead targets a balanced benchmark over real images with broad compositional reasoning.

2. Related Work: Prior bias-mitigation approaches use constrained synthetic images or altered train-validation answer distributions, limiting realism or potentially penalizing salient learned properties.The related work motivates a balanced test that is more resilient to educated-guessing strategies.
2. Related Work: Blind models already perform strongly on earlier VQA benchmarks, reaching 50% on VQA1.0 and 67% of binary questions on VQA2.0.These figures illustrate the extent to which answer priors can support performance without image input.
2. Related Work: Unlike CLEVR’s synthetic, low-diversity setting, GQA uses real images and a large semantic space to make compositional reasoning more challenging.CLEVR’s limited object classes and properties make memorization of combinations more feasible.
2. Related Work: Earlier visual question-generation datasets are either small or restricted to few objects and non-compositional templates, while neural alternatives pursue engaging rather than necessarily accurate questions.GQA differs by targeting large-scale, semantically grounded compositional questions.

3. The GQA Dataset

GQA is a large real-world visual reasoning dataset built from scene graphs and structured question semantics. Its generation, balancing, and evaluation design target compositional reasoning while reducing answer-distribution shortcuts.

22M questions over 113K images cover object and attribute recognition, relation tracking, spatial reasoning, logical inference, and comparisons.
GQA pairs images and questions with scene graphs, functional programs, and textual and visual answer justifications to support grounded analysis.These structured annotations enable balanced question construction and metrics beyond standard accuracy.
3.2. The Question Engine: The question engine combines normalized scene graphs with 524 structural patterns to generate diverse, grammatical questions with varying compositionality.It uses graph content, linguistic patterns, object references, and decoys to instantiate questions, yielding 22M examples.
3.3. Functional Representation and Entailment: Functional programs provide structured semantics that support fine-grained assessment by question type, topology, textual and semantic length, and entailment relations.The representations also support recursive entailment computation for measuring model coherence and consistency.
3.4. Sampling and Balancing: The dataset uses functional-program labels at global and local granularities to smooth conditional answer distributions while retaining tunable real-world tendencies.This two-level balancing increases answer-distribution entropy by 72%.

4. Analysis and Baseline Experiments

GQA analysis shows a large, diverse dataset that supports fine-grained evaluation across question types, transfer settings, and behavioral metrics. Models remain substantially below humans and exhibit weaknesses in transfer, consistency, plausibility, distribution modeling, and grounding.

4.1. Dataset Analysis and Comparison: 22,669,678 questions over 113,018 images span varied reasoning skills, question lengths, and inference-step counts, with 1,878 possible answers.The dataset covers 88.8% of VQA questions and 70.6% of VQA answers.
4.2. Baseline Experiments: Baseline models achieve only 17.82%–41.07%, while stronger Bottom-Up and MAC models surpass them but remain well below human performance.The blind LSTM reaches 22.7% on open queries and performs near chance on binary questions.
4.3. Transfer Performance: 52.1% on VQA before fine-tuning and 60.5% afterward are achieved by MAC trained on GQA, compared with 39.8% and 46.5% when MAC transfers from VQA to GQA.The results support the realism and diversity of GQA questions while indicating that GQA poses a greater challenge for VQA-trained models.
4.4. New Evaluation Metrics: 98.4% consistency is achieved by people, whereas even the best models are inconsistent about one-fifth of the time and LSTM models nearly half the time.Consistency evaluates whether answers remain compatible across semantically entailed questions.
4.4. New Evaluation Metrics: Models fail to produce plausible or valid answers 5–15% of the time, while the distribution metric reveals whether predictions capture less frequent answers beyond the most common ones.The distribution metric uses a Chi-Square statistic, with lower values indicating closer agreement with the true conditional answer distribution.
4.4. New Evaluation Metrics: About 80% grounding scores are obtained by models using object-based features, versus 43% with spatial features.The comparison indicates that object-informed features provide finer-grained regions for attending to relevant image content.

5. Conclusion

The paper presents GQA as a benchmark for real-world visual reasoning and compositional question answering. It concludes that structured data, behavioral metrics, and richer reasoning tasks can support more compositional, interpretable, and consistent models.

5. Conclusion: GQA combines dataset generation, baseline experiments, and behavioral measures to provide deeper insight into visual question answering models.The authors position the benchmark as supporting research on deeper semantic understanding, sound reasoning, robustness, and consistency.
5. Conclusion: The paper identifies closer integration between visual knowledge extraction and question answering as a potential avenue toward more grounded and transparent reasoning.This is presented as a potential direction rather than an established result.

GQA TYPE COMPOSITION (DETAILED)

GQA organizes questions by structural and semantic composition and provides richer, more connected image-based questioning than conventional VQA examples. Its examples emphasize multi-step relations, attributes, comparisons, and repeated coverage of image content.

GQA TYPE COMPOSITION (DETAILED): Figure 7 summarizes question-prefix frequencies, question-type composition, frequent objects and relations, and word clouds for objects, attributes, and relations.The top-left arcs encode counts by first four words, while the top-right presents question-type distribution.
GQA TYPE COMPOSITION (DETAILED): GQA questions are categorized by structural type, semantic type, and functional-program templates, supporting analysis of reasoning composition.The structural categories include verify, query, choose, logical, and compare questions.
GQA TYPE COMPOSITION (DETAILED): GQA examples ask linked questions about the same image, including relations, attributes, object identity, spatial position, and comparisons.Examples include identifying what an object holds, locating objects relative to one another, and checking combined attributes.
GQA TYPE COMPOSITION (DETAILED): GQA questions tend to involve more image elements, longer compositional wording, and more questions per image than VQA examples.The comparison also describes VQA questions as more ambiguous or subjective in some cases.

8. Dataset Balancing

GQA balances answer distributions by iteratively reducing head dominance while preserving real-world tendencies. The procedure spreads probability toward less frequent answers across question groups and granularity levels.

8. Dataset Balancing: Figure 10 compares conditional answer distributions before balancing across global and local question groups, with column heights representing relative answer frequencies.Global groups include broad categories such as colors and animals, while local groups include relations such as apple-color and table-material.
8. Dataset Balancing: The balancing process reduces the original 22M auto-generated questions to a 1.7M balanced dataset through answer-distribution balancing, similarity reduction, and type-based sampling.Balancing iterates through answers within each question group while applying frequency bounds.
8. Dataset Balancing: The resulting distribution is pushed away from frequent answers toward the tail while intentionally retaining original real-world tendencies.This preserves authenticity rather than making all answers uniformly distributed.

9. Baselines Implementation Details

The experiments standardize model components and training conditions to enable fair baseline comparisons, while evaluating performance across question and training-set variations.

9. Baselines Implementation Details: All models use the same LSTM, CNN, classifier, and 300-dimensional GloVe embeddings, differing primarily in their model architectures.CNN models use ImageNet-pretrained spatial features, while BottomUp and MAC use faster R-CNN object-based features.
9. Baselines Implementation Details: Figure 11 evaluates accuracy against textual question length, semantic question length, training-set size, and MAC network length.Semantic length is measured by the number of operations in the functional program, while training subsets range from 10K to 10M.
9. Baselines Implementation Details: 15 training epochs with Adam provide the common optimization setup, while MAC uses four cells and BottomUp is re-implemented and validated on standard VQA.Each epoch takes about an hour, and the BottomUp reimplementation achieved 67% on the standard VQA dataset.

10. Further Diagnosis

The diagnosis shows that GQA stresses compositional reasoning: semantic complexity lowers accuracy, while deeper MAC networks, larger datasets, and more semantic representations improve performance.

10. Further Diagnosis: Longer MAC networks perform better on GQA, supporting the dataset’s compositional character.The analysis varies the number of recurrent reasoning cells and reports the results in Figure 11.
10. Further Diagnosis: 1-step questions are easier than multi-step questions, although textual question length correlates positively with model accuracy.Longer wording may provide exploitable cues, whereas greater semantic length requires more compositional reasoning operations.
10. Further Diagnosis: Most GQA questions require about 2–3 reasoning steps involving relations, attributes, or logical operations.Figure 13 characterizes semantic length by the number of computation steps needed to reach an answer.
10. Further Diagnosis: The dataset’s logical representations enable formal entailment and equivalence relations that support consistency analysis across question types.A consistent model should not answer incompatible colors for equivalent or entailed questions.
10. Further Diagnosis: 53.8% accuracy and 81.59% consistency on textual questions rise to 59.7% accuracy and 85.85% consistency on functional programs.The textual questions contain synonyms, diverse structures, and long-range dependencies absent from the smaller operation vocabulary of functional programs.
10. Further Diagnosis: Performance improves with larger training sets and more semantically informed visual representations, from CNN features toward direct scene-graph embeddings.The training-set analysis has not reached saturation, suggesting benefit from larger datasets.

11. Comparison between GQA and VQA 2.0

Compared with VQA 2.0, GQA uses longer, more compositional questions that require a broader range of reasoning skills, while deliberately excluding several question types.

11. Comparison between GQA and VQA 2.0: GQA questions are longer on average than VQA questions and contain more verbs, prepositions, nouns, and adjectives.These linguistic differences are presented as evidence of increased compositionality.
11. Comparison between GQA and VQA 2.0: GQA demands more spatial, logical, relational, and comparative reasoning and contains significantly more compositional questions than VQA 2.0.The comparison is summarized in Table 3.
11. Comparison between GQA and VQA 2.0: GQA focuses on factual, image-answerable, multi-hop questions rather than covering all VQA question types.It excludes intention questions and questions requiring OCR or external knowledge.

12. Scene Graph Normalization

GQA normalizes Visual Genome scene graphs into a clean, semantically structured representation by curating vocabulary and relations, validating object properties, and adding spatial and global information.

12. Scene Graph Normalization: The normalization starts from Visual Genome scene graphs covering 113K COCO and Flickr images, where nodes represent bounded objects with attributes.The dataset is extended with 5K hidden scene graphs collected through crowdsourcing.
12. Scene Graph Normalization: Free-form annotations are converted into a unified ontology by cleaning vocabulary, consolidating synonyms, filtering rare concepts, and adding linguistic features.The resulting ontology contains 1,740 objects and 620 attributes, with additional semantic and grammatical annotations.
12. Scene Graph Normalization: Unnatural or inadequate graph edges are pruned using category rules, n-gram frequencies, co-occurrence statistics, and manual curation.Examples include relations such as a woman being in a shirt or a hand hugging a bear.
12. Scene Graph Normalization: Object detectors conservatively validate object absence or uniqueness when Visual Genome annotations cannot guarantee complete coverage.This validation supports the generation of correct and unambiguous questions.
12. Scene Graph Normalization: The graphs are enriched with absolute and relative positions, shared attributes, and directly inferable global information such as location or weather.The resulting scene graphs have clean, unified, rich, and unambiguous node and edge semantics.