GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

1. Introduction

• Existing VQA benchmarks contain real-world answer priors and simple questions that let models exploit statistical biases instead of demonstrating visual scene understanding.The paper also identifies missing annotations of question structure and content as an obstacle to explaining model behavior and errors.Source: §1 · pp. 1–2
• GQA uses scene graphs and linguistic methods to generate over 22 million diverse, visually grounded questions with functional programs, establishing a resource for visual reasoning.The programs specify each question’s content and semantics.Source: §1 · p. 2
• GQA downsamples questions to a 1.7M balanced dataset and uses tunable smoothing to balance open and binary questions and make answer distributions more uniform.The approach uses semantic representations to control question-type composition and reduce dataset biases.Source: §1 · p. 2
• GQA supplements accuracy with consistency, validity, plausibility, grounding, and distribution metrics, and provides answer explanations with word-level visual pointers.The explanations and pointers can also supervise the development of more transparent models.Source: §1 · p. 2

2. Related Work

• Earlier bias-mitigation methods either use constrained synthetic images [39] or leave open questions and residual binary-answer bias; Goyal et al. [11] omit 29% of questions.The passage notes that the annotation limits leave answer distributions largely unbalanced even among the covered questions.Source: §2 · p. 2
• Agrawal et al. [3] create adversarial train–validation answer distributions, which can penalize legitimate learning; GQA instead seeks a balanced test resilient to gaming.The passage frames educated guessing from training-data properties as a valid strategy for both machines and people.Source: §2 · p. 2
• 50% of VQA1.0 questions, and 67% of binary and 27% of open VQA2.0 questions, are answered correctly by blind models [11].These figures illustrate the dataset-bias problem described in the section.Source: §2 · p. 2
• Unlike CLEVR’s synthetic images and handful of object classes and properties, GQA operates over real images and a large semantic space, making it more challenging.GQA’s questions are not natural as in other VQA datasets [11] [41], but may provide a cleaner benchmark for more controlled and comprehensive evaluation.Source: §2 · p. 3
• Earlier question-generation datasets are small [25] or use restricted objects and non-compositional templates [24], while neural alternatives target engaging, potentially inaccurate questions about image context [28] [14] [40].Those alternatives may generate questions about subjective feelings or speculative events related to depicted scenes [28].Source: §2 · p. 3

3. The GQA Dataset

• Scene graphs, question programs, and answer justifications enable answer-distribution control, behavioral metrics, and assessment by question type and topology.Image graphs represent objects, attributes, and relations; programs encode reasoning steps, while answers include textual and visual justifications.Source: §3 · p. 3

3.1. Scene Graph Normalization

• Scene graph normalization consolidates 2,690 ontology classes, prunes unreliable edges, and enriches graphs with positional and semantic properties.The starting graphs cover 113K COCO and Flickr images and represent objects with bounding boxes, attributes, and relation edges.Source: §3.1 · p. 4

3.2. The Question Engine

• The engine combines scene graphs with 524 patterns across 117 question groups, using templated and probabilistic variations and nested references to increase compositional diversity.It begins with 250 manually constructed patterns and adds 274 natural patterns derived from VQA1.0.Source: §3.2 · pp. 4–5
• Scene-graph decoys and careful reference selection support negative and inference questions while avoiding answer-revealing wording and ambiguous relations.Distractors are selected for plausibility in context and filtered to exclude candidates considered too similar.Source: §3.2 · p. 5
• 22M diverse, challenging, grammatical questions are generated from scene content using GQA’s question engine.The dataset covers 113K images and questions spanning varied types and degrees of compositionality.Source: §3 · p. 3, §3.2 · p. 5

3.3. Functional Representation and Entailment

• Functional programs encode question semantics as chained operations and support performance analysis by question type and topology, as well as entailment-based consistency evaluation.The programs also support dataset balancing and identification of equivalent or entailing questions.Source: §3.3 · p. 5

3.4. Sampling and Balancing

• Two-level answer smoothing retains real-world tendencies while increasing answer-distribution entropy by 72%.Questions are grouped using global answer-type and local subject-answer labels, then downsampled to produce more uniform distributions at both granularities.Source: §3.4 · pp. 5–6

4.1. Dataset Analysis and Comparison

• GQA contains 22,669,678 questions over 113,018 images, spans 3,097 words and 1,878 answers, and covers 88.8% of VQA questions and 70.6% of its answers.Questions vary in structural and semantic types and in required reasoning steps.Source: §4.1 · p. 6, Figure 6 · p. 7

4.3. Transfer Performance

• GQA-trained MAC scores 52.1%/60.5% on VQA before/after fine-tuning, versus 51.6% for LSTM+CNN and 68.3% for VQA-trained MAC; VQA-trained MAC scores 39.8%/46.5% on GQA.The transfer results suggest GQA is a realistic proxy for human-like questions while posing a further challenge for models trained on VQA.Source: §4.3 · p. 7

4.4. New Evaluation Metrics

• People achieve 98.4% consistency, while even the best models contradict themselves about one in five times and LSTM nearly half the time.Consistency checks whether answers to a question agree with answers to entailed questions; it can expose contradictions that accuracy misses.Source: §4.4 · p. 7
• Models fail to give valid or plausible answers 5–15% of the time; validity checks question scope, while plausibility checks whether an answer is reasonable for its subject.Plausibility uses whether an answer occurs with the question’s subject anywhere in the dataset, so data scarcity for infrequent objects can limit precision.Source: §4.4 · pp. 7–8

4.2. Baseline Experiments

• Baselines score 17.82%–41.07%; bottom-up attention and MAC outperform them but remain below human performance.The question-only LSTM reaches 22.7% on open query questions and performs only slightly above chance on binary questions.Source: §4.2 · p. 7, Table 1 · p. 8

4.4. New Evaluation Metrics

• The distribution metric compares predicted and true answer distributions using Chi-Square, and leading state-of-the-art models score lower than baselines, with lower scores better.It measures whether models capture less frequent answers as well as the most common ones.Source: §4.4 · p. 8
• Attention-based models attain grounding scores of about 80%, versus 43% with spatial features, indicating object-based features better focus attention on relevant image regions.Grounding measures attention probability over a visual region referred to by the question or answer.Source: §4.4 · p. 8

• GQA combines a dataset-generation process, baseline experiments, and new measures for analyzing models’ behavior and performance.Source: §5 · p. 8
• More intimate integration between visual knowledge extraction and question answering may advance deeper semantic understanding, sound reasoning, robustness, and consistency.Source: §5 · p. 8
• The authors hope GQA will motivate compositional, interpretable, and cogent reasoning models and advance scene understanding and visual question answering.Source: §5 · p. 8

GQA TYPE COMPOSITION (DETAILED)

• GQA visualizations show question-prefix and question-type distributions, plus frequent objects, categories, attributes, and relations, excluding left/right.Source: Figure 7 · p. 11
• GQA questions have a diverse range of lengths compared with other VQA datasets, except synthetic CLEVR.Source: Figure 8 · p. 12
• GQA’s structural and semantic question-type distributions are visualized alongside a catalog of types, program templates, and example questions.Source: Figure 8 · p. 12, Table 2 · p. 12
• The dataset visualizations show an object class hierarchy created as part of GQA’s construction.Source: Figure 8 · p. 12
• Compared with VQA questions about the same images, GQA questions tend to involve more image elements and be longer and more compositional.Source: Figure 9 · p. 13
• VQA questions tend to be more ambiguous and subjective than GQA questions, sometimes lacking a clear, conclusive answer.Source: Figure 9 · p. 13
• The examples show that GQA provides more questions per image than VQA, covering each image more thoroughly.Source: Figure 9 · p. 13

Dataset Balancing

• After balancing, answer distributions are more uniform with heavier tails, while intentionally retaining original real-world tendencies to a tunable degree.Before balancing, the displayed distributions are heavily biased; afterward, frequency is pushed away from the head and spread over the tail to retain authenticity.Source: Figure 10 · p. 14, §8 · p. 14
• 22M auto-generated questions are reduced to a 1.7M balanced dataset through answer-distribution balancing, similarity reduction, and type-based sampling.The passage describes this as the result of the combined dataset-processing steps.Source: §8 · p. 14
• Balancing iteratively down-samples frequent answers from most to least frequent under a bound b, while limiting frequency ratios between consecutive answers.For each question group, the process iterates over answer-count pairs and applies minimum and maximum bounds to successive-answer frequency ratios.Source: §8 · p. 14

9. Baselines Implementation Details

• All models use 300-dimensional GloVe embeddings and the same LSTM, CNN, and classifier components, while CNN and state-of-the-art models use different feature sources.CNN models use ImageNet-pretrained spatial features; BottomUp [5] and MAC [12] use object features from a Faster R-CNN detector [31].Source: §9 · p. 14
• All models use a sigmoid-based classifier and Adam [19] for 15 epochs, with each training run taking about an hour.Source: §9 · p. 15
• MAC [12] uses its official code with 4 cells, while BottomUp [5] is reimplemented and validated at 67% on standard VQA, matching Anderson et al. [5].The BottomUp reimplementation follows details in [5] [34].Source: §9 · p. 15

Further Diagnosis

• The diagnosis varies question word and semantic length, MAC cell count, training-set size, and input representation to probe model behavior.MAC is a recurrent attention network that reasons in multiple concurrent steps over questions and images.Source: §10 · p. 15
• Accuracy rises with textual question length but falls as semantic reasoning grows; one-step questions are easiest, while most GQA questions require 2–3 steps.Semantic steps can involve following relations, identifying attributes, or applying logical operations.Source: §10 · p. 15, Figure 13 · p. 15
• Accuracy improves as visual inputs become more semantic, from CNN spatial features to detector-based object features and scene-graph embeddings.The scene-graph condition provides direct semantic access, described as “perfect sight”; Figure 12 also compares textual questions with functional programs.Source: Figure 12 · p. 15, §10 · p. 16
• People achieve 98.4% consistency across entailed question answers, while deep learning models reach only 69%–82%.The analysis argues that improving consistency is important because inconsistent answers cast doubt on models’ reliability and visual understanding.Source: Figure 14 · p. 16, §10 · p. 16
• MAC achieves 53.8% accuracy and 81.59% consistency on textual questions, versus 59.7% accuracy and 85.85% consistency on functional programs.Programs use a small operations vocabulary, while natural-language questions include synonyms, varied structures, and long-range dependencies.Source: §10 · p. 16
• More-cell MAC networks perform better, and increasing training-set size significantly improves performance, with no evidence that data size has saturated.The section notes that larger datasets may therefore benefit models.Source: §10 · p. 16

11. Comparison between GQA and VQA 2.0

• GQA questions are longer on average and contain more verbs, nouns, adjectives, and prepositions than VQA questions.Source: §11 · p. 16, Table 3 · p. 17
• GQA questions are significantly more compositional than VQA’s and involve spatial, logical, relational, and comparative reasoning more often.Source: §11 · p. 17, Table 3 · p. 17
• GQA excludes some VQA question types, including intention, OCR, and external-knowledge questions, focusing on objective, unambiguous factual questions answerable from images and multi-hop reasoning.Source: §11 · p. 17

12. Scene Graph Normalization

• GQA starts from 113k Visual Genome images [20] from COCO [23] and Flickr [36], adding 5k crowdsourced hidden graphs; nodes represent boxed, attributed objects connected by relation edges.Relations capture actions, spatial relationships, and comparisons.Source: §12 · p. 17
• Vocabulary cleanup and categorization produce a curated hierarchy with 1,740 objects, 620 attributes, and 330 relations across 60 categories and subcategories.The process removes stop words, fixes typos, consolidates synonyms, filters rare or amorphous concepts, and uses embedding distances followed by manual curation; semantic and linguistic features support grammatical question generation.Source: §12 · p. 17
• Edges unsuitable for question generation are pruned using category rules, n-gram frequencies, co-occurrence statistics, and manual curation.Examples include “(woman, in, shirt)” and “(hand, hugging, bear).”Source: §12 · p. 17
• Low-threshold object detectors [31], trained on Visual Genome, conservatively confirm object absence or uniqueness when annotation coverage is insufficient.This validation supports correct, unambiguous question generation.Source: §12 · p. 17
• Graphs gain image-margin, relative-position, shared-property, and inferable location or weather annotations, yielding clean, unified, rich, and unambiguous node and edge semantics.Position and shared color, material, or shape annotations are added only when they can be safely determined; global information is included when directly inferable from the objects.Source: §12 · p. 17