PaperlayerView exact revision on arXiv

Source-linked AI summary

Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, Alan Yuille

arXiv:2212.00259v2cs.CVcs.CL

TL;DR

VQA models are commonly evaluated under an assumption of shared training and test distributions, while performance on other domains can be difficult to analyze across intertwined factors. Super-CLEVR isolates four domain-shift factors for controlled testing, and P-NSVQA, which adds uncertainty reasoning to NSVQA, outperforms existing methods on three of the four factors.

  • Problem

    VQA methods are typically developed assuming training and test data share a distribution, although algorithms trained on one domain often fail to generalize to others.

    Source: §1 · p. 1

  • Method

    Super-CLEVR uses controllably generated data to isolate visual complexity, question redundancy, concept distribution, and concept compositionality, while P-NSVQA adds uncertainty reasoning to NSVQA.

    Source: §1 · pp. 1–2

  • Results

    P-NSVQA significantly outperforms existing models on three of four domain-shift factors, and the conclusion reports better performance than existing methods in both in-domain and out-of-domain testing.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The study concludes that disentangled reasoning and perception together with explicit uncertainty modeling yield a more robust VQA model.

    Source: §1 · p. 2

    The benchmark is synthetic; the authors identify better-controlled, balanced real datasets as future work and say models should also be tested on real data.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.

1. Introduction

The paper introduces Super-CLEVR to isolate four factors of VQA domain shift and diagnose their effects. It evaluates existing models and proposes P-NSVQA, which outperforms existing models on three of the four factors.

1. Introduction

  • The paper’s central motivation is that VQA domain shifts combine intertwined visual and linguistic factors, yet their individual contributions lack systematic analysis.Standard benchmarks often assume test data shares the training distribution, while models can struggle on out-of-distribution tests.Source: §1 · p. 1
  • Super-CLEVR independently varies visual complexity, question redundancy, concept distribution, and concept compositionality to diagnose their effects.Its generated data allows test distributions to differ from training distributions along these factors.Source: §1 · p. 1, Figure 1 · p. 2
  • Neural modular methods are more robust to question redundancy, non-modular methods to visual complexity, and NSVQA to concept distribution shifts.The evaluation covers FiLM, mDETR, NSCL, and NSVQA; all show varying sensitivity to domain shifts.Source: §1 · p. 2
  • P-NSVQA significantly outperforms existing models on three of the four domain-shift factors.It extends NSVQA by adding probabilities to its symbolic executor to account for uncertainty in scene understanding.Source: §1 · p. 2

2. Related work

Prior VQA work includes real-image benchmarks, synthetic reasoning datasets, methods addressing dataset bias, and approaches to domain shift. Super-CLEVR differs by introducing complex visual scenes and controllability for studying isolated shift factors.

Visual question answering

  • VQA methods include two-stream feature-fusion models, neural-symbolic parse-then-execute systems, and pretrained transformer models.The paper selects FiLM, NSCL, and mDETR as category representatives.Source: §2 · p. 2

VQA datasets

  • Real-image VQA datasets can contain strong priors and biases; CLEVR uses synthetic images and multi-step questions to assess reasoning.Dataset-balancing and counterfactual examples are among the approaches used to address bias.Source: §2 · p. 2

Domain shift in VQA

  • Super-CLEVR adds complex visual scenes and controllability to study domain robustness along isolated factors, unlike prior VQA domain-shift work.Prior work includes adaptation, cross-dataset analysis, synthetic-to-real transfer, and out-of-distribution approaches.Source: §2 · p. 3

3. Super-CLEVR

Super-CLEVR is a controllable synthetic benchmark designed to study four factors of VQA domain shift independently. Its vehicle scenes, generated questions, and dataset variants enable controlled changes in visual complexity, redundancy, concept frequency, and concept composition.

3.1. Motivation: domain shift factors

  • Visual complexity covers object variety and appearance factors, which Super-CLEVR varies using vehicle parts and optional textures.The paper notes that complexity can also relate to size, background, lighting, occlusion, and viewpoint.Source: §3.1 · p. 3
  • Question redundancy is over-specified attribute or relationship information; the benchmark varies it because redundant details can distract VQA models.An example is asking for a large bus behind a cyan car when only one bus is present.Source: §3.1 · p. 3
  • Concept distributions vary across datasets, and long-tailed frequencies can limit rare-concept learning and introduce answer biases.Super-CLEVR varies concept distributions to study model robustness to such shifts.Source: §3.1 · p. 3
  • Concept compositionality is the co-occurrence of concepts, such as shapes and colors; the benchmark tests whether models generalize to changed combinations.For example, it asks whether models can recognize blue bicycles and red cars after training on red bicycles and blue cars.Source: §3.1 · p. 3

3.2. Dataset generation

  • Super-CLEVR follows CLEVR’s generation pipeline but uses vehicle models with annotated parts and attributes that can differ between a vehicle and its parts.Scenes contain 3–10 objects, and the pipeline saves object and part bounding boxes and segmentation masks.Source: §3.2 · pp. 3–4
  • Question templates instantiate reasoning programs, allowing redundancy to be controlled by adding or removing redundant reasoning steps.Questions are generated from programs operated on scene graphs.Source: §3.2 · p. 4

3.3. Controlling the dataset

  • Visual-complexity variants share layouts and questions: harder versions add varied part attributes and textures, isolating visual changes.The variants are easy, mid, and hard.Source: §3.3 · p. 4
  • Redundancy variants remove redundant attributes, use default CLEVR-style redundancy, or add possible attributes and relationships; all questions remain valid.Concept distributions also range from balanced to long-tailed, with head, tail, and opposite-distribution test variants for frequency analysis.Source: §3.3 · p. 4

4. Evaluated methods

The evaluated methods range from simple answer-prediction baselines to two-stream, transformer-based, and neural-symbolic systems. P-NSVQA adds probabilistic reasoning over scene-parser confidence to NSVQA and significantly improves over its deterministic version.

4.1. Simple baselines

  • Random and majority-answer predictors, a question-only LSTM, and a CNN-plus-LSTM image-question model provide simple baselines.The question-only model predicts answers from the final LSTM state; the CNN-plus-LSTM model combines image and question features.Source: §4.1 · p. 5

4.2. Existing models

  • FiLM represents two-stream feature fusion, mDETR is a text-conditioned transformer detector, and NSCL and NSVQA are neural-symbolic methods.NSVQA separates its scene parser, question parser, and program executor, which can be learned independently.Source: §4.2 · p. 5

4.3. Probabilistic NSVQA (P-NSVQA)

  • P-NSVQA augments NSVQA’s deterministic executor with scene-parser likelihoods and step-wise confidence, selecting the program with the largest joint likelihood; it significantly improves over deterministic NSVQA [59].The likelihood model includes object classes, attributes, and spatial relations.Source: §4.3 · p. 5

4.4. Implementation details

  • NSVQA and P-NSVQA use ground-truth programs for training, and question-to-program parsing exceeds 99% accuracy; default-split experiments were repeated three times with different seeds, with standard deviations ±0.10 for P-NSVQA and ±0.40 for NSVQA, while other experiments ran once.All models were trained with 200,000 questions.Source: §4.4 · p. 5

5. Results and analysis

The results compare model accuracy in-domain and under four domain shifts, then examine transfer to GQA and Super-CLEVR’s potential for part-based reasoning. P-NSVQA leads on three of four shift factors, while non-modular methods are more robust to visual complexity shifts.

5.1. In-domain results

  • On Super-CLEVR, P-NSVQA beat other models and NSVQA by 2.96%, while every model scored lower there than on CLEVR.Neural modular methods outperformed non-modular methods; mDETR reached 82.7% and FiLM 53.2%.Source: Figure 3 · p. 6, §5.1 · p. 6

5.2. Out-of-domain results

  • P-NSVQA outperformed other models on three of four domain-shift factors, while non-modular methods were more robust to visual complexity shifts.The comparison uses Relative Degrade, where lower values indicate better robustness.Source: §5.2 · pp. 6, 8, Table 2 · p. 7
  • Modular methods had less than 2% RD under question-redundancy shifts, compared with around 20% for non-modular methods.The authors attribute the difference to step-by-step reasoning, which makes each reasoning step independent and reduces reliance on spurious question-answer correlations.Source: §5.2 · pp. 6–7
  • FiLM and mDETR had less than 10% RD on visual complexity, while NSCL and (P-)NSVQA exceeded 12%; P-NSVQA lowered RD by 4.04% relative to NSVQA.The authors suggest simple reasoning modules may process visual signals less effectively than dense non-modular models.Source: §5.2 · p. 7
  • All four existing models exceeded 20% RD on concept distribution; NSVQA beat the other three by more than 7.5%, and P-NSVQA reduced NSVQA’s RD from 21% to 14%.The authors connect NSVQA’s robustness to disentangling reasoning and visual understanding, limiting reliance on distributional priors.Source: §5.2 · p. 7
  • On concept compositionality, non-modular methods were more robust than NSCL and NSVQA, but P-NSVQA outperformed the non-modular methods.The authors identify improved robustness of modular methods as a potential benefit of further model improvements.Source: §5.2 · p. 8

5.3. More analysis and future work

  • On real GQA, mDETR’s accuracy dropped more than that of neuro-symbolic methods as redundant operations were removed; Super-CLEVR can also support part-object hierarchy questions.The paper notes that current VQA models lack the ability to understand part-object hierarchies.Source: §5.3 · p. 8, Table 3 · p. 8
  • The benchmark is synthetic, so the authors recommend also testing models on real data and propose collecting better-controlled, balanced real datasets for diagnosis.The paper frames Super-CLEVR’s purpose as model diagnosis.Source: §5.3 · p. 8

6. Conclusion

The paper introduces Super-CLEVR to diagnose visual-reasoning domain shifts by independently studying distinct factors with controlled data generation. It reports that P-NSVQA outperforms existing methods in in-domain and out-of-domain testing, and identifies disentangled perception and reasoning with probabilistic uncertainty as a promising combination.

6. Conclusion

  • P-NSVQA outperforms all existing methods in both in-domain and out-of-domain testing.P-NSVQA extends NSVQA with uncertainty in the reasoning modules.Source: §6 · p. 8
  • All four evaluated existing methods struggle with domain shifts, highlighting the importance of out-of-domain testing.Source: §6 · p. 8
  • Neural modular methods are more robust to question redundancy, while NSVQA shows better robustness to distribution and compositionality shifts.Source: §6 · p. 8
  • The study suggests that disentangling perception and reasoning, combined with probabilistic uncertainty, forms a strong VQA model that is more robust to domain shifts.Source: §6 · p. 8

A. Dataset statistics

Figure 4 presents the distribution of question types in Super-CLEVR, with types determined by the last operation in each question program.

A. Dataset statistics

B. List of objects

Super-CLEVR contains 21 objects from five categories: airplane, bicycle, bus, car, and motorcycle.

B. List of objects

C. Dataset controlling

Dataset variants control concept distribution and concept compositionality: the former uses bal, slt, long, head, tail, and oppo, while the latter uses co-0, co-1, and co-2.

C. Dataset controlling

D. Definition of Relative Degrade

For concept compositionality, Relative Degrade denotes accuracy when a model trained on variant i is tested on variant j.

D. Definition of Relative Degrade

  • For concept compositionality, Relative Degrade uses A_i^j for accuracy after training on variant i and testing on variant j.The passage identifies this definition under concept compositionality.Source: D. Definition of Relative Degrade · pp. 12–13

E. More details about P-NSVQA

P-NSVQA maintains object-selection probabilities and updates them through reasoning operations. The section also describes the dataset’s object categories and concept-distribution variants.

E. More details about P-NSVQA

  • P-NSVQA maintains an n-object probability vector, where each entry is the probability that its object is selected, and updates it stepwise during reasoning.The scene operation initializes all entries to 1.Source: E. More details about P-NSVQA · p. 13
  • A filter operation uses the visual scene parser’s predicted probability that an object has the specified attribute.The examples include filters for attributes such as color.Source: E. More details about P-NSVQA · p. 13
  • Spatial relate operations return probabilities that objects lie on a specified side of the input object; the reported hyperparameters are a=20 and b=0.02.The examples include relate behind, front, right, and left.Source: E. More details about P-NSVQA · p. 13
  • Same-attribute operations return probabilities that each object shares the input object’s color, shape, size, or material.The passage illustrates the operation with color.Source: E. More details about P-NSVQA · p. 13
  • Intersection and union operate on two probability vectors, and ⊙ denotes pointwise product.Source: E. More details about P-NSVQA · p. 14
  • The dataset figure captions identify 21 objects across 5 categories, concept-distribution variants, and co-distribution matrices for co-0, co-1, and co-2.Source: Figure 5 · p. 15, Figure 6 · p. 16, Figure 7 · p. 17

Related papers

Loading 2212.00259v2…