PaperlayerView exact revision on arXiv

Source-linked AI summary

MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering

Tejas Gokhale, Pratyay Banerjee, Chitta Baral, Yezhou Yang

arXiv:2009.08566v2cs.CVcs.CL

TL;DR

VQA models can exploit spurious dataset correlations rather than generalize out of distribution. MUTANT trains on semantically structured input mutations with consistency constraints, achieving state-of-the-art VQA-CP-v2 accuracy and reducing the VQA-v2 performance gap.

  • Problem

    VQA models may rely on dataset biases and spurious correlations, limiting evidence that same-distribution test performance measures out-of-distribution generalization.

  • Method

    MUTANT automatically generates semantic mutations of image or question entities and trains models with pairwise consistency between original and mutant answer representations.

  • Results

    10.57% improvement over the previous best establishes new state-of-the-art accuracy on VQA-CP-v2, with gains across question categories.

  • Takeaways & Limitations

    MUTANT achieves state-of-the-art VQA-CP-v2 accuracy and reduces the performance gap between VQA-CP-v2 and VQA-v2.

Abstract

from arXiv · show

While progress has been made on the visual question answering leaderboards, models often utilize spurious correlations and priors in datasets under the i.i.d. setting. As such, evaluation on out-of-distribution (OOD) test samples has emerged as a proxy for generalization. In this paper, we present MUTANT, a training paradigm that exposes the model to perceptually similar, yet semantically distinct mutations of the input, to improve OOD generalization, such as the VQA-CP challenge. Under this paradigm, models utilize a consistency-constrained training objective to understand the effect of semantic changes in input (question-image pair) on the output (answer). Unlike existing methods on VQA-CP, MUTANT does not rely on the knowledge about the nature of train and test answer distributions. MUTANT establishes a new state-of-the-art accuracy on VQA-CP with a $10.57\%$ improvement. Our work opens up avenues for the use of semantic input mutations for OOD generalization in question answering.

1 Introduction

The introduction argues that i.i.d. evaluation can mask spurious dataset correlations, motivating OOD evaluation for robust VQA generalization. MUTANT addresses this by training on semantically distinct mutations with pairwise consistency objectives, achieving 69.52% accuracy on VQA-CP-v2.

  • Motivation: i.i.d. test performance cannot measure generalization when models exploit dataset correlations, motivating evaluation on out-of-distribution samples.The introduction distinguishes useful positive bias from spurious negative bias in datasets.
  • Mutant paradigm: MUTANT exposes VQA models to perceptually similar yet semantically dissimilar mutations of questions and images.Mutations alter the input and consequently change the answer, helping models identify input changes critical to answering.
  • Training objective: The method trains predictions for original and mutant samples to remain pairwise consistent while using Noise-Contrastive Estimation on projected cross-modal features and answer embeddings.The projections map features and true answers onto a learned shared manifold.
  • Results: 69.52% accuracy is achieved on the VQA-CP-v2 benchmark, establishing a new state of the art.The introduction frames the approach as amplifying positive bias and implicitly attenuating spurious correlations rather than explicitly debiasing.

2 MUTANT

MUTANT trains VQA models with semantically meaningful input mutations and consistency constraints, addressing the limitations of answer-class correlations and linguistic priors. Its components project inputs and answers into a shared space, expose question types, and preserve semantic shifts between original and mutant pairs.

  • Mutant Inputs: MUTANT creates mutant inputs through addition, removal, or substitution transformations that alter the image or question and produce a new answer.Image mutations include adding or removing objects and changing object attributes.
  • Motivation: Answer-class classification can learn correlations between question features and labels without encoding answer meaning, harming generalization to visually different instances.The paper illustrates this with banana-color questions, where models may associate the question with “yellow” without representing yellowness or greenness.
  • Answer Projection: Answer Projection maps image-question features and answers to a shared manifold, enabling similarity-based training beyond answer-class prediction.The method uses Noise Contrastive Estimation over projected input features and answer embeddings.
  • Type Exposure: Type Exposure teaches models to identify question types and mask answer candidates to those valid for each type, rather than removing linguistic bias entirely.This targets spurious priors such as “tennis” for sport questions and “two” for counting questions.
  • Pairwise-Consistency: Pairwise Consistency trains on original-mutant pairs so predicted answer-vector distances match ground-truth distances, regularizing semantic shifts caused by mutations.For example, an image mutation changing the answer from “two” to “one” should produce a corresponding shift in answer space.

3 Generating Input Mutations for VQA

MUTANT automatically generates semantic mutations in images and questions that reliably alter or preserve answers, without using test-set knowledge. Image mutations remove or recolor critical objects, while question mutations negate, substitute, or mask critical words.

  • Mutation framework: Mutations transform semantic entities—objects in images and words in questions—to create samples with predictable answer changes or invariances.The automated process does not use knowledge about the test set or its answer distribution.
  • Image mutations: Image mutations identify critical objects and either remove object instances or invert their RGB colors.Removed instances are selected using COCO polygon annotations; color mutations replace the answer with the objects’ new color without relying on real-world color knowledge.
  • Image mutations: Removed-object mutations sample m instances from M object instances, then use GAN-based inpainting to produce photorealistic masked images.The sampling allows m ∈ {0, ..., M}, and inpainting prevents models from exploiting removal shapes as cues.
  • Question mutations: Question mutations use negation, antonym or adversarial object-word substitution, and word masking to introduce ambiguity.Questions with indeterminate new answers receive broad category labels, while mutations of non-critical objects or words preserve the answer.
  • Dataset construction: 679k mutant samples are generated from VQA-CP-v2 training data, averaging 1.5 mutants per original sample without changing the question-type distribution.The generated samples are categorized by mutation type in Table 2.

4 Experiments

Experiments evaluate MUTANT on VQA-CP-v2, an OOD benchmark with differing answer distributions across train and test, and VQA-v2. MUTANT improves strong baselines, benefits both UpDn and LXMERT, and achieves gains across question categories and component ablations.

  • Benchmark Results: MUTANT improves LXMERT by 23.29% on VQA-CP and beats the previous best by 10.57%, establishing a new state of the art.Gains include 8.78% on Yes-No, 17.75% on Number-based, and 9.57% on other questions.
  • Benchmark Results: MUTANT improves two state-of-the-art models and outperforms the evaluated baselines, demonstrating that the paradigm is model-agnostic.The experiments incorporate MUTANT into both UpDn and LXMERT rather than modifying only UpDn.
  • Benchmark Results: 70.24% accuracy is achieved on balanced i.i.d. VQA-v2, the best performance among methods designed specifically for OOD generalization.A model trained on VQA-CP and mutant data also reaches 67.63% on VQA-v2 without retraining, including 88.56% on yes-no, 50.76% on number-based, and 54.56% on other questions.
  • Effect of Training with Mutant Samples: 10.42% and 13.46% are the respective improvements for UpDn and LXMERT after augmentation with mutant samples.UpDn gains 23.94% on numeric questions, while both models show marked improvements on yes-no and number categories.
  • Ablation Studies: Question mutations outperform image mutations on yes-no and other questions, whereas image mutations perform better on numeric questions; combining both types performs best.Ablations also find Answer Projection improves yes-no performance, Type Exposure improves other questions, and Pairwise Consistency boosts numeric and yes-no performance.

5 Related Work

Related work addresses VQA dataset bias, robustness, data augmentation, and answer embeddings. MUTANT extends these directions by generating semantically distinct samples with different answers and training with a consistency loss between original and mutant inputs.

  • De-biasing of VQA datasets: VQA-v2 mitigated VQA-v1’s question–answer imbalances and language priors by pairing each question with complementary images yielding different answers.
  • De-biasing of VQA datasets: VQA-CP reorganized train and test splits so answer distributions differed significantly across question types, targeting models’ superficial answer correlations.
  • Robustness in VQA: Prior robustness methods use cycle-consistency to generate complementary questions with the same answer or provide perception-related sub-questions, whereas MUTANT generates questions with different answers.
  • Data Augmentation: Counterfactual-image augmentation and minimal edits such as masking or inpainting have improved VQA out-of-distribution generalization, but prior work focused primarily on language or visual priors.
  • Data Augmentation: MUTANT enables VQA samples with different answers and couples a novel architecture with consistency loss between original and mutant samples.
  • Answer Embeddings: Unlike prior answer-embedding methods with a single objective, MUTANT combines classification and NCE loss on projected answer vectors.

6 Discussion and Conclusion · Appendix

MUTANT trains VQA models with structured input mutations and consistency-oriented components to improve out-of-distribution generalization. The paradigm frames semantic changes as enabling “what if” decision making beyond observed inputs.

  • 6 Discussion and Conclusion: MUTANT uses input mutations to train VQA models for out-of-distribution generalization.The method is presented as a training approach centered on mutated inputs.
  • 6 Discussion and Conclusion: The answer projection module minimizes distance between answer and input projections while complementing canonical VQA classification.This module is described as a novel component of the training method.
  • 6 Discussion and Conclusion: Type Exposure treats all valid answers for each question type as equally probable candidates, reducing reliance on negative question-answer linguistic priors.The model uses this exposure to move away from linguistic priors.
  • 6 Discussion and Conclusion: Unlike random adversarial perturbations, MUTANT uses structured perturbations that cause semantic input changes and deterministic output changes.The distinction is made against robust-learning methods using random adversarial perturbations.
  • 6 Discussion and Conclusion: Carefully designed input perturbations or manipulations are described as potentially benefiting generalization and improving performance.The discussion connects this perspective to concurrent image-classification work.
  • 6 Discussion and Conclusion: MUTANT aims to let models imagine changes in a scene or language query and predict outputs for the imagined input.This supplements “what” decision making from observed inputs with “what if” decision making from imagined inputs.

A Datasets · A.1 VQA-CP · A.2 COCO

The paper evaluates OOD generalization on VQA-CP, a VQA reorganization that changes answer distributions across question types between training and testing. Its images come from COCO, which contains complex natural scenes with object annotations and captions.

  • A.1 VQA-CP: VQA-CP reorganizes VQA so answer distributions differ between training and test splits across 65 question types.Question types are identified by prefixes such as “how many,” “what color,” and “which.”
  • A.1 VQA-CP: Unlike VQA-v2’s random independent assignment, VQA-CP greedily resplits samples to prevent train-test sharing of questions with the same type and answer.The passage also states that VQA-CP has no leakage between train and test splits compared with the original VQA splits.
  • A.1 VQA-CP: 121k images, 245k questions, and 2.5M answers comprise VQA-CP-v2’s training set, compared with 98k images, 220k questions, and 2.2M answers in testing.These figures describe the VQA-CP-v2 train and test sets, respectively.
  • A.2 COCO: VQA and VQA-CP source their images from MS-COCO.The dataset relationship is stated explicitly in the passage.
  • A.2 COCO: COCO contains natural images depicting complex real-world scenes with common objects from 91 categories.Examples include person, chair, fork, horse, and sports-ball.
  • A.2 COCO: Each COCO image includes 5 captions, bounding boxes, and polygon annotations for each object instance.These annotations provide textual and object-level information for the images used by VQA and VQA-CP.

B Image Mutant Generation Process · B.1 Selection of Objects

The mutant-generation process starts from original VQA question-image-answer triplets and applies linguistic preprocessing alongside object selection from COCO annotations. Object selection distinguishes critical from non-critical objects, while removal is illustrated through bounding-box or polygon annotations and inpainting.

  • B Image Mutant Generation Process: Mutant samples are generated from original question-image-answer triplets in the VQA-CP dataset.
  • B Image Mutant Generation Process: Linguistic operations combine SpaCy with LemmInflect for lemmatization and inflection.
  • B Image Mutant Generation Process: COCO bounding-box and polygon annotations support object removal, with inpainting used to produce the resulting images.
  • B.1 Selection of Objects: For each VQA sample, the process builds a word list from the ground-truth answers and the question.
  • B.1 Selection of Objects: All nouns in the word list are converted to their singular form.
  • B.1 Selection of Objects: For yes-no, numeric, and object-color questions, candidate objects are obtained from COCO.
  • B.1 Selection of Objects: Background and crowd objects are filtered out before separating critical and non-critical objects.
  • B.1 Selection of Objects: Critical objects are defined as image objects that are manipulated or removed in the mutation process.

B.2 Object Removal and In-painting … C.2 Adversarial Words and Masking

MUTANT creates photorealistic image mutations through object removal, in-painting, and color inversion, while generating question mutants through negation, adversarial-word substitution, and masking. It derives new answers from mutation type and uses broad categories or uncertainty labels when exact answers cannot be determined.

  • B.2 Object Removal and In-painting: B.2 Object Removal and In-painting: Objects are removed by setting their pixels to white, then a GAN-based inpainting network fills the masked region.The inpainting network is trained on ImageNet and produces photorealistic images.
  • B.2 Object Removal and In-painting: B.2 Object Removal and In-painting: Photorealistic inpainting is preferred because masked images differ from natural-image distributions and mask boundaries can reveal the missing object’s outline.The method uses an off-the-shelf blind image inpainting model.
  • B.3 Color Inversion Process: B.3 Color Inversion Process: Color mutations invert every pixel within the object mask without relying on prior knowledge of valid object colors.This avoids restricting mutations to typical colors, such as yellow, green, or black for bananas.
  • B.4 Answer Generation: B.4 Answer Generation: Answer updates depend on question type: complete removal flips yes-no answers, critical-object counts change from n to n−m, and color answers are generated by inverting HEX values.For partial removal or non-critical objects, yes-no and counting answers remain unchanged; inverted colors are matched to the nearest CSS-21 color.
  • C Question Mutant Generation Process: C Question Mutant Generation Process: Question mutants use three operators—negation, substitution with antonyms or adversarial words, and masking critical words.These operators modify question semantics while preserving the mutation-based training framework.
  • C.1 Negation: C.1 Negation: Template-based negation inserts words such as “not” or “no” into yes-no and color-based questions, flipping yes to no or no to yes.The negative word is placed before a preposition, noun phrase, or verb.
  • C.2 Adversarial Words and Masking: C.2 Adversarial Words and Masking: Object words are replaced with BERT-ranked synonyms absent from the image, or critical object words are replaced with “MASK”.When exact answers are indeterminate, the method assigns a broad category such as “size,” or uses “can’t say” or “don’t know” when no broad category is ascertainable.
  • C.2 Adversarial Words and Masking: C.2 Adversarial Words and Masking: Answer categories are constructed by clustering GloVe vectors with k-means and manually annotating the categories at K = 50.Clustering uses Euclidean distance with a manually tuned number of clusters.

D Dataset Analysis

The dataset analysis examines answer distributions by question type, mutation sample counts, and the dataset’s final answer-type distribution.

  • D Dataset Analysis: The analysis considers how answers are distributed across question types.
  • D Dataset Analysis: It examines the number of samples associated with each mutation type.
  • D Dataset Analysis: The dataset’s answer-type distribution is also reported.

D.1 Distribution by Question Type · D.2 Distribution by Mutation Type · D.3 Distribution by Answer Type

The paper analyzes answer distributions across question types, mutation categories, and answer types. Mutant data remain broadly aligned with VQA-CP training distributions while introducing small, characterized changes without test-answer leakage.

  • D.1 Distribution by Question Type: Answer distributions are shown for the three question categories “How many,” “What sport,” and “What color,” using the top-10 answers.
  • D.1 Distribution by Question Type: The distributions are distinct across the examined question types.
  • D.1 Distribution by Question Type: Mutant distributions are close to VQA-CP training data, except for categorical answers such as “number” and “sports” introduced during mutation.
  • D.1 Distribution by Question Type: The mutation method does not leak test-set answer information into training data.
  • D.2 Distribution by Mutation Type: Table 10 reports the number of generated samples for each mutation type.
  • D.3 Distribution by Answer Type: VQA-CP and Mutant datasets share three answer types: yes/no, number, and other.
  • D.3 Distribution by Answer Type: Creating mutant samples causes a small change in the answer-type distribution shown in Table 9.
Loading 2009.08566v2…