PaperlayerView exact revision on arXiv

Source-linked AI summary

AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

Madeleine Grunde-McLaughlin, Ranjay Krishna, Maneesh Agrawala

arXiv:2103.16002v1cs.CVcs.CL

TL;DR

Existing video question answering benchmarks can conflate distinct reasoning failures and contain biases that permit superficial answer guessing. AGQA addresses these gaps with a large, programmatically generated benchmark and controlled evaluation splits, revealing that current models remain weak on compositional spatio-temporal reasoning.

  • Problem

    Existing video question answering benchmarks conflate multiple reasoning errors and contain answer-distribution biases that complicate capability analysis.

  • Method

    AGQA generates controlled question-answer pairs from spatio-temporal scene graphs and provides balanced data plus splits for novel compositions, indirect references, and additional reasoning steps.

  • Results

    Current models barely outperform linguistic-only baselines, and none generalize to novel compositions; accuracy decreases as reasoning steps increase.

  • Takeaways & Limitations

    AGQA’s controlled generation and evaluation expose weaknesses in current visual reasoning systems across compositional reasoning abilities.

  • Takeaways & Limitations

    Scene-graph quality is constrained by visually unclear videos, missing annotations, and inconsistencies between human and AGQA term definitions.

Abstract

from arXiv · show

Visual events are a composition of temporal actions involving actors spatially interacting with objects. When developing computer vision models that can reason about compositional spatio-temporal events, we need benchmarks that can analyze progress and uncover shortcomings. Existing video question answering benchmarks are useful, but they often conflate multiple sources of error into one accuracy metric and have strong biases that models can exploit, making it difficult to pinpoint model weaknesses. We present Action Genome Question Answering (AGQA), a new benchmark for compositional spatio-temporal reasoning. AGQA contains $192M$ unbalanced question answer pairs for $9.6K$ videos. We also provide a balanced subset of $3.9M$ question answer pairs, $3$ orders of magnitude larger than existing benchmarks, that minimizes bias by balancing the answer distributions and types of question structures. Although human evaluators marked $86.02\%$ of our question-answer pairs as correct, the best model achieves only $47.74\%$ accuracy. In addition, AGQA introduces multiple training/test splits to test for various reasoning abilities, including generalization to novel compositions, to indirect references, and to more compositional steps. Using AGQA, we evaluate modern visual reasoning systems, demonstrating that the best models barely perform better than non-visual baselines exploiting linguistic biases and that none of the existing models generalize to novel compositions unseen during training.

1. Introduction

AGQA addresses limitations in video question answering by generating controlled questions for compositional spatio-temporal reasoning and evaluating generalization beyond training compositions. The benchmark exposes substantial gaps between current visual reasoning systems and the intended capabilities.

  • Motivation: Existing video question answering benchmarks combine object recognition, relationship identification, and temporal reasoning failures into a single accuracy metric.This makes individual model capabilities difficult to diagnose.
  • Motivation: Biases in answer distributions and visual-event frequencies allow models to guess answers without learning compositional reasoning.AGQA therefore targets more granular control over question composition and concept distributions.
  • Benchmark: AGQA provides 3.9M balanced and 192M unbalanced question-answer pairs associated with 9.6K videos.Questions are generated by handcrafted programs operating over spatio-temporal scene graphs.
  • Evaluation: The benchmark tests generalization to novel compositions, indirect references, and increasing numbers of compositional reasoning steps.These abilities are controlled through the question-generation process and dedicated training/test splits.
  • Results: 47.74% accuracy is achieved by the highest-performing model, while modern systems barely outperform linguistic-only baselines.All models lose accuracy as compositional steps increase, and none generalize to novel compositions.

2. Related Work

AGQA extends compositional question answering from images and limited video settings to real-world videos with controlled spatio-temporal reasoning. Its design targets reasoning types and biases that existing benchmarks do not adequately measure.

  • Existing benchmarks: Image question answering benchmarks support spatial compositional reasoning but generally do not measure temporal reasoning beyond common-sense actions.AGQA targets video understanding using language for question answering.
  • Existing benchmarks: Synthetic video benchmarks provide granular control for abilities such as causality or counting, but use short clips, few objects, and limited visual diversity.Some also include questions requiring commonsense or external knowledge.
  • Existing benchmarks: Human-generated or descriptive video benchmarks can suffer from biases and simplicity that limit compositional reasoning analysis.These limitations motivate controlled question generation for real-world video understanding.
  • AGQA’s relation to prior work: AGQA includes duration, action sequencing, relationship-action interactions, and logical combinations absent from existing real-world video-only corpora.The benchmark focuses on visual understanding rather than external knowledge.
  • AGQA’s relation to prior work: AGQA generalizes GQA’s compositional pipeline from images to temporal localizations, indirect action references, and changing subject-object relationships.Its programs operate over Action Genome’s spatio-temporal scene graphs to generate question-answer-video pairs.

3. The AGQA benchmark

AGQA constructs a large, balanced benchmark by turning annotated spatio-temporal scene graphs into controlled question-answer pairs. Its programs track reasoning requirements, semantic categories, and question structures while balancing distributions to reduce exploitable biases.

  • Generation pipeline: The pipeline consolidates Action Genome scene graphs and Charades action localizations into symbolic video representations before generating questions with handcrafted programs.Questions are produced using probabilistic grammar rules and then balanced by answer and question-structure distributions.
  • Scene graphs: AGQA’s augmented ontology contains 36 objects, 44 relationships, and 157 actions across 7,787 training and 1,814 test scene graphs.Entailments, synonym merging, and temporal heuristics improve the semantic consistency of the scene graphs.
  • Question categories: Questions are categorized by reasoning ability, semantic class, and structure, including object, relationship, and action subjects and open or binary answer formats.This organization supports analysis across reasoning steps, semantic categories, and question forms.
  • Question generation: 269 natural-language question frames and 28 programs generate 192M question-answer pairs, including over 45M unique questions and 174 unique answers.Programs associate each question with its required reasoning skills and number of reasoning steps.
  • Balancing: Rejection sampling balances answer distributions and question structures, reducing the original 192M pairs to a 3.9M balanced benchmark.The procedure addresses model exploitation of dataset imbalances and increases representation of difficult query questions.
  • Evaluation splits: AGQA introduces training/test splits for evaluating generalization to novel compositions, indirect references, and more compositional steps.These splits use the benchmark’s controlled question generation to isolate particular compositional abilities.

4. Experiments and analysis

Experiments evaluate model performance across reasoning abilities, question structures, and generalization settings. Models generally struggle with visual compositional reasoning, especially novel compositions and increasing reasoning depth.

  • Human validation: Human validation labeled 86.02% of AGQA answers as correct, indicating that about 13.98% of generated questions contain errors.The validation used majority votes from three subjects on at least 50 randomly sampled questions per type.
  • Performance across reasoning abilities: HME and HCRN outperform PSAC across reasoning categories, with HME strongest on superlatives and HCRN strongest on sequencing and activity recognition.These comparisons are reported across the benchmark’s reasoning categories.
  • Performance across reasoning abilities: HCRN exceeds its language-only counterpart by more than 1.5% only for activity recognition and object-action interaction, while sometimes performing worse.HCRN performs worse without the language-only comparison on sequencing and action-duration questions.
  • Question structures: Open-ended questions are the most challenging structural category, and HCRN improves over its language-only variant by only 0.68%.PSAC struggles most with open-ended questions, while HME performs best on choosing questions.
  • Generalization: All models struggle with novel compositions; only HME exceeds 50% on binary questions, reaching 52.39% accuracy.No model performs much better than the Most-Likely baseline on open questions.
  • Generalization: Models fail on at least nearly a fifth of indirect-reference questions even when answering the corresponding direct questions correctly.HCRN generalizes best to indirect object and temporal references, while HME is strongest for indirect relationship and action references.
  • Generalization: With more compositional steps, models outperform the Most-Likely baseline on open questions but remain below 50% accuracy on binary questions.HCRN performs best on open-ended questions, whereas HME generalizes best overall in this setting.
  • Reasoning depth: Accuracy decreases as the number of compositional reasoning steps increases, with weak linear-regression R2 scores for all three models.The reported R2 scores are HCRN .43, HME .24, and PSAC .51.

5. Discussion and future work

AGQA is a large benchmark for evaluating visual compositional spatio-temporal reasoning across novel compositions, indirect references, and increasing reasoning depth. Experiments show that current systems barely exceed language-based alternatives and do not generalize to novel compositions, motivating several future research directions.

  • Contribution: AGQA is a real-world compositional spatio-temporal benchmark three orders of magnitude larger than existing work.The benchmark evaluates visual compositional reasoning across multiple dimensions.
  • Contribution: AGQA is the first benchmark described here to use language to evaluate generalization to novel compositions, indirect references, and more compositional steps.These splits target distinct compositional reasoning desiderata.
  • Findings: Modern visual systems barely outperform linguistic-bias variants, and no evaluated model generalizes to novel compositions unseen during training.Although models show some ability to handle more compositional steps, accuracy decreases as reasoning steps increase.
  • Future work: The authors propose neuro-symbolic or semantic-parsing approaches because AGQA supplies ground-truth scene-graph annotations and executable-question structure.They also suggest meta-learning and multi-task learning for discovering shared compositional rules.
  • Future work: AGQA’s multidimensional benchmark structure provides the computer vision community with multiple directions for future work.The discussion specifically highlights approaches targeting systematic compositional rules and novel-composition generalization.

6. Supplementary

The supplementary analyses separate binary and open-answer performance and test generalization to novel compositions, indirect references, and additional compositional steps. Models perform worse on open answers and struggle especially with novel compositions and longer reasoning chains.

  • Evaluation setup: The Most-Likely baseline differs between binary and open-answer categories because answer-choice structure and answer distributions affect guessing accuracy.Balanced Yes/No binary questions have a 50% Most-Likely baseline, whereas open-answer baselines use the frequency of the category’s most common answer.
  • Binary versus open-answer results: Models perform much worse on open-answer questions than binary questions across reasoning categories.HCRN is generally strongest on open-answer questions, while HME leads binary object questions by over 5%.
  • Novel compositions: Models struggle to generalize to novel compositions in every category except duration.Novel object-relationship compositions are especially difficult; HCRN leads open-answer questions and HME leads binary questions.
  • Indirect references: Open-answer questions show a larger Recall-to-Precision accuracy increase than binary questions for indirect references.Among direct counterparts answered correctly, HME performs best on binary questions and HCRN on open-ended questions.
  • Compositional steps: Models struggle to generalize to questions with more compositional steps after training on questions with fewer steps.HME performs best overall and on binary questions, while it performs better on open-answer questions.

6.2. Scene graph augmentation details

AGQA augments noisy, inconsistent, and sparse Action Genome scene graphs before generating questions. The process merges duplicate annotations, standardizes relationships, propagates information across frames, and excludes unsupported spatial questions.

  • Source annotations: Action Genome scene graphs annotate five sampled frames per Charades action with object contact, spatial, and attention relationships.AGQA generates questions and answers from these spatio-temporal annotations.
  • Duplication: Duplicate object and action references are merged or replaced with more specific annotations to reduce artificial ambiguity.Objects with overlapping but nonsynonymous annotations can be manually reannotated.
  • Inconsistency: Beneath and above annotations are filtered or flipped to enforce a consistent head-level relationship definition.Objects with at most 95% intra-class consistency are removed before correction.
  • Sparsity: Sparse annotations are propagated to surrounding frames using heuristic rules.Questions using spatial relationships are omitted from videos with sparse annotations because those relationships lack entailments.

6.3. Question quality checks

AGQA audits generated questions to remove rare, implausible, ambiguous, or difficult-to-interpret cases. Its question inventory is organized by reasoning types and model-evaluation categories.

  • Filtering: Questions with object-relationship pairs occurring fewer than 10 times are removed as rare combinations.The filtering process also excludes manually blacklisted pairs likely to occur but not be annotated.
  • Evaluation organization: AGQA reports model results across binary and open questions in reasoning, semantic, and structural categories.Table 9 presents the binary/open split, while the benchmark also includes overall reasoning-type organization.
  • Question types: The benchmark’s question types include activity recognition, such as determining what action occurs before or after another action.The activity-recognition examples use temporal relations around a referenced action.
  • Filtering: Decoy questions reuse verbs or objects present in the video rather than introducing entirely unrelated actions.For example, a video containing opening a door may generate questions about opening a refrigerator or fixing a door.
  • Filtering: Questions involving difficult attention or negative-contact relationships are excluded after human evaluation found them hard to answer correctly.Examples include looking at, not looking at, and not contacting.

6.4. Templates

AGQA uses 28 question templates paired with executable programs that combine discrete reasoning steps over scene-graph information. Multiple natural-language variants produce diverse question-answer pairs.

  • Templates: AGQA’s 28 templates generate diverse question-answer pairs from scene-graph information.Each template has multiple natural-language options for expressing the same underlying structure.
  • Templates: Each template is associated with a program that automatically generates answers from the spatio-temporal scene graph.The programs combine discrete reasoning steps in numerous ways.

6.5. Balancing

AGQA balances answer distributions and question-structure proportions to reduce answer guessing from question wording and increase challenging open-answer questions. The resulting benchmark has less answer skew and a larger share of open-answer questions.

  • Answer and structure balancing: The balancing process first smooths answer distributions, then adjusts question-structure proportions to create a more diverse benchmark.Binary categories are balanced so answers are equally plausible, while open-answer distributions are smoothed before structural balancing.
  • Answer and structure balancing: Across balancing steps, categories with only one possible answer are removed, and questions are further split by temporal-localization effects when multiple answers remain.This process is applied to specified categories such as exists-paper and preserves the intended answer distribution within the resulting groups.
  • Answer and structure balancing: Binary questions are balanced within specific content categories, deleting questions from the more frequent answer until alternatives occur equally often.Balancing categories rather than binary questions overall reduces guessing based on the question itself.
  • Answer and structure balancing: Open-answer distributions are smoothed by deleting head questions until no more than 30% of pairs use the most frequent 20% of answer types.The process moves a splitting index down the ordered distribution while preserving frequency ordering where possible.
  • Answer and structure balancing: Structural balancing increases the proportion of open-answer questions because initial answer balancing leaves binary questions overrepresented.Questions are deleted across structural, template, content, and answer categories while retaining the original answer distribution within each category.
  • Outcome: After both balancing rounds, AGQA contains a larger percentage of open-answer and challenging questions with less skew in answer distributions.The benchmark’s balancing is intended to reduce, though not eliminate, answer guessing from question wording.

6.6. Novel compositions

AGQA evaluates novel compositional reasoning by withholding selected temporal and object-relationship combinations from training and placing them in test sets. The split includes both direct questions and indirect references to held-out combinations.

  • Novel-composition split: Novel-composition splits withhold selected combinations from training and evaluate them in test questions.The temporal combinations include six before-action phrases selected with varied frequency.
  • Novel-composition split: The object-relationship split holds out pairs such as table-wiping, paper-carrying, and chair-leaning on from training.Questions directly asking about these pairs and questions using indirect references are removed from training and retained for testing.
  • Novel-composition split: The object-relationship novel-composition test set contains 24,005 questions.The held-out test questions include both direct object-relationship queries and indirect references such as “the object they were carrying.”

6.7. Human study

AGQA’s human studies identify annotation and evaluation errors arising from unclear videos, missing or inconsistent scene-graph labels, and differing task demands. These findings motivate qualification procedures, constrained question design, and improved annotation practices.

  • Human validation: Human validation used answer verification and dropdown selection to examine correctness and task-specific cognitive demands.The two formats helped identify question types associated with larger performance gaps.
  • Annotation errors: Missing, incorrect, and inconsistent scene-graph annotations introduced errors into AGQA’s question-generation process.Events at video boundaries, unannotated relationships, and visually unclear objects limited the underlying representation.
  • Question design: AGQA excluded relationship-answer questions because scene graphs frequently omitted other relevant relationships, creating potentially multiple valid answers.Some contact relationships could be added by entailment, but not all missing relationships were recoverable.
  • Annotation errors: Incorrect Charades timestamps propagated to Action Genome and left some sampled annotation frames unrelated to the intended action.Overlapping timestamps for different action families could not always be corrected.
  • Annotator disagreement: Annotators disagreed about object identity, action segmentation, filming-phone interactions, animal actions, and transition durations.The benchmark avoided questions about transition-verb duration and merged some semantically similar terms, but these adjustments added cognitive effort.
  • Future annotation practices: The annotation process is difficult and expensive because AGQA’s specific questions depend on detailed, thorough scene-graph representations.Recommended practices include full-video coverage, globally sequenced timestamps, explicit definitions, and one annotation per object with hierarchical specificity.

6.8. Conclusion

AGQA provides a large balanced benchmark for complex spatio-temporal reasoning and evaluates models across several generalization abilities. State-of-the-art models struggle across these abilities, with performance declining as question complexity increases.

  • Conclusion: AGQA produces a large balanced video-question-answer dataset requiring complex spatio-temporal reasoning despite annotation challenges.The benchmark is designed to expose model strengths and weaknesses across reasoning skills.
  • Conclusion: Current state-of-the-art models struggle to generalize to novel compositions, indirect references, and more compositional steps.The benchmark contributes metrics targeting each of these abilities.
  • Conclusion: Models’ performance decreases as question complexity increases, whereas humans perform similarly on simple and complex questions.This contrast highlights a gap in compositional reasoning ability measured by AGQA.
  • Conclusion: AGQA opens avenues for models that more effectively perform compositional reasoning.Its evaluations can distinguish relative strengths and weaknesses across reasoning skills.
Loading 2103.16002v1…