PaperlayerView exact revision on arXiv

Source-linked AI summary

Cycle-Consistency for Robust Visual Question Answering

Meet Shah, Xinlei Chen, Marcus Rohrbach, Devi Parikh

arXiv:1902.05660v1cs.CV

TL;DR

VQA models are brittle to linguistic variations, and existing evaluation lacks a benchmark for measuring this robustness. The paper introduces VQA-Rephrasings and trains models with answer-conditioned cycle consistency without additional annotation, achieving greater robustness and state-of-the-art VQA and VQG performance. Its training scheme assumes generated questions are semantically and syntactically correct, an assumption that does not always hold.

  • Problem

    VQA models are brittle to linguistic variations in input questions, while the field lacks a benchmark for quantitatively evaluating robustness across rephrasings.

  • Method

    A model-agnostic cycle-consistent framework jointly answers questions and generates diverse answer-conditioned rephrasings, enforcing the original ground-truth answer on generated questions without additional annotation.

  • Results

    Models trained with the approach are significantly more robust on VQA-Rephrasings and achieve state-of-the-art performance on VQA and VQG on VQA v2.0.

  • Takeaways & Limitations

    Cycle-consistent joint question answering and generation provides a model-agnostic route to robustness across linguistic rephrasings while improving performance on both tasks.

  • Takeaways & Limitations

    The training scheme assumes generated questions are always semantically and syntactically correct, but this assumption does not always hold in practice.

Abstract

from arXiv · show

Despite significant progress in Visual Question Answering over the years, robustness of today's VQA models leave much to be desired. We introduce a new evaluation protocol and associated dataset (VQA-Rephrasings) and show that state-of-the-art VQA models are notoriously brittle to linguistic variations in questions. VQA-Rephrasings contains 3 human-provided rephrasings for 40k questions spanning 40k images from the VQA v2.0 validation dataset. As a step towards improving robustness of VQA models, we propose a model-agnostic framework that exploits cycle consistency. Specifically, we train a model to not only answer a question, but also generate a question conditioned on the answer, such that the answer predicted for the generated question is the same as the ground truth answer to the original question. Without the use of additional annotations, we show that our approach is significantly more robust to linguistic variations than state-of-the-art VQA models, when evaluated on the VQA-Rephrasings dataset. In addition, our approach outperforms state-of-the-art approaches on the standard VQA and Visual Question Generation tasks on the challenging VQA v2.0 dataset.

1. Introduction

VQA models are brittle to reasonable linguistic variations in questions, motivating a benchmark and an annotation-free cycle-consistent framework for robustness. The approach generates answer-conditioned rephrasings and enforces consistent correct answers, while also improving standard VQA and VQG performance.

  • Motivation: VQA models remain far from robust to reasonable variations in users’ image-question language.This matters for both real-world applications and VQA as a test of multimodal understanding.
  • Approach: The proposed model-agnostic framework trains VQA models to answer questions and generate diverse answer-conditioned semantic rephrasings without additional annotation.It enforces that the generated rephrasing receives the original question’s ground-truth answer.
  • Approach: Consistent correctness across generated rephrasings targets unseen semantically equivalent questions at test time.The framework generates linguistically diverse rephrasings on the fly during training.
  • Benchmark: VQA-Rephrasings provides 3 human rephrasings for approximately 40k questions on approximately 40k VQA v2.0 validation images.The dataset and associated consensus score enable quantitative robustness evaluation across linguistic variations.
  • Results: Models trained with the approach outperform state-of-the-art models on standard VQA and VQG and are significantly more robust on VQA-Rephrasings.The reported comparisons cover VQA v2.0 and the proposed rephrasing benchmark.

2. Related Work

Prior work studied VQA robustness to visual, answer-distribution, and adversarial changes, while this work targets linguistic and compositional question variations. It builds on question generation and cycle-consistency research to connect answer-conditioned generation with robust multimodal question answering.

  • Visual Question Answering: The proposed method is model-agnostic and can be applied with existing VQA architectures.This broadens its compatibility beyond a single VQA model family.
  • Robustness: Prior VQA robustness studies examined semantic image changes, answer-distribution shifts, and adversarial image perturbations.The paper positions linguistic and compositional question rephrasings as an additional robustness dimension.
  • Visual Question Generation: Earlier visual question generation methods produced diverse or informative image questions, with some methods conditioning generation on an answer.The related work distinguishes answer-agnostic generation from answer-specific question generation.
  • Cycle-Consistent Learning: Cycle consistency has regularized transformations in tracking, machine translation, image translation, and text-based question answering.The paper applies this idea to the multimodal connection between questions, images, and answers.

3. Approach

The framework trains VQA and answer-conditioned question generation jointly so generated rephrasings preserve the original answer, improving robustness to linguistic variation.

  • Cycle-consistent framework: The model predicts an answer for the original question, generates a semantically similar question from the predicted answer and image, then re-answers the generated question.The cycle links VQA transformation F with VQG transformation G.
  • Cycle-consistent framework: Cycle consistency enforces that the generated question receives the same answer as the original question’s ground truth answer.The approach assumes generated questions are valid semantic and syntactic rephrasings.
  • Question generation module: Attention supplies additional supervision because answer-to-question generation maps a low-information modality to a higher-information modality.The question generator encodes attended image features and answer distributions, adds noise, and reconstructs the original question with an LSTM.
  • Training objective: The total objective combines VQA answer loss, question-generation loss, and cycle-consistency loss with tunable weights.The VQA and cycle losses are cross-entropy terms, while the generation loss is a sequence-generation loss.
  • Training safeguards: A gating mechanism filters incoherent generated questions, while late activation delays cycle-consistency training to reduce failures such as mode collapse.The authors identify question-generation design, gating, and late activation as crucial and support this with ablations.
  • Training effects: Shared VQA weights and online generation expose the model to diverse rephrasings during training, making the framework an online data-augmentation technique.The authors report improved performance, robustness, consistency, and prediction of the model’s own failures.

4. VQA-Rephrasings Dataset

VQA-Rephrasings is a human-annotated benchmark for consistency across semantically equivalent questions, paired with a consensus score that measures joint correctness across rephrasings.

  • Dataset: VQA-Rephrasings is introduced as a dataset for evaluating VQA robustness and consistency across different rephrasings with the same meaning.It addresses evaluation of linguistic robustness rather than image-side variation.
  • Dataset construction: 40,504 sampled VQA v2.0 validation questions, each paired with human-collected rephrasings, span 40,504 images.The base validation split contains 214,354 questions over 40,504 images.
  • Dataset construction: The final dataset contains 162,016 questions spanning 40,504 images, averaging approximately three rephrasings per original question.The collection used two human stages to produce and validate rephrasings.
  • Consensus Score: Consensus Score CS(k) is the fraction of size-k rephrasing subsets in which every answer is correct according to non-zero VQA accuracy.It is an all-or-nothing measure of joint correctness within each question group.
  • Consensus Score: At k = |Q|, a model must answer the original question and all rephrasings correctly to obtain non-zero consensus.Higher average consensus at large k indicates greater robustness to linguistic variation.
  • Results: Cycle-consistent models consistently outperform their baseline counterparts at every tested consensus-score value.Table 1 compares original-question and rephrased-question splits using CS(k).

5. Experiments

Experiments evaluate cycle-consistent VQA models on standard VQA, rephrasing consistency, question generation, and failure prediction tasks. The framework improves robustness and standard VQA performance, while attention-based question generation yields larger gains than unattended features.

  • Experimental Setup: The experiments train on VQA v2.0 splits and never explicitly train on VQA-Rephrasings, using that dataset purely for evaluation.The implementation uses publicly available backbone implementations and fixed cycle-consistency hyperparameters.
  • Consistency Performance: 39.49% consensus score at k = 4 is achieved by baseline Pythia, with consistency dropping as more rephrasings are included.MUTAN, BAN, and BUTD show similar consistency losses; BAN + CC outperforms BAN and all other models at every tested k.
  • Visual Question Answering Performance: Cycle-consistent versions of BUTD, Pythia, and BAN outperform their corresponding baselines on VQA v2.0 validation and test-dev splits.The ablation table adds question consistency, answer consistency, and gating incrementally, with the complete framework consistently outperforming baselines.
  • Visual Question Answering Performance: 8.08% consistency CS(k = 4) and 0.5% VQA accuracy gains with attention exceed the 3.58% and 0.2% gains from unattended image features.The unattended configuration also improves over baseline, but generates poorer questions and produces smaller overall gains.
  • Visual Question Generation Performance: The VQG component is evaluated on VQA v2.0 validation using CIDEr, BLEU, METEOR, and ROUGE-L against answer-conditional VQG models.The compared approaches include iVQA and iQAN.
  • Failure Prediction: Failure prediction uses either answer-confidence thresholding or a binary classification module that predicts whether the VQA answer is correct.The module receives the image, question, and predicted answer, using image and answer encoders similar to those in question generation.

6. Conclusion

The paper proposes cycle-consistent, model-agnostic VQA training and the VQA-Rephrasings benchmark to address linguistic robustness. It reports robustness gains together with state-of-the-art VQA and VQG performance on VQA v2.0.

  • 6. Conclusion: The paper introduces a model-agnostic cycle-consistent training strategy, VQA-Rephrasings, and a consensus metric for evaluating robustness to linguistic variation.The framework is intended to make VQA models robust to linguistic variations and self-aware of failures.
  • 6. Conclusion: Models trained with the proposed strategy are reported to be robust to linguistic variations and achieve state-of-the-art performance in VQA and VQG on VQA v2.0.

A. Dataset Details

VQA-Rephrasings preserves the linguistic intent of original questions while varying their wording, with part-of-speech distributions similar to VQA v2.0. Its questions are slightly longer on average.

  • Statistics: 7.15 is the average question length in VQA-Rephrasings, compared with 6.32 in VQA v2.0.The rephrasing questions are therefore slightly longer on average.
  • Statistics: Part-of-speech distributions in VQA-Rephrasings and VQA v2.0 follow similar trends, indicating rephrasings are not formed merely by adding adjectives or adverbs.
  • Interface: The human-annotation interface showed only invalid rephrasing examples and explanations in the final setup after A/B testing four example configurations.Manual inspection found this setup produced higher-quality data.
  • Examples: Rephrasings maintain the original question’s intent while introducing linguistic variation.

B. Attention Analysis

Attention visualizations compare baseline Pythia with the same model trained using cycle consistency across question rephrasings. The cycle-consistent model maintains relevant region attention and correct answers, while the baseline fails on some variants.

  • Attention Analysis: Across four rephrasings, cycle-consistent Pythia attends to relevant image regions and answers all variants correctly, unlike baseline Pythia on some variants.
  • Attention Analysis: Both baseline and cycle-consistent Pythia perform poorly on counting questions because the baseline Pythia model lacks a counting module.

C. Attention Consistency

The framework adds an attention consistency loss intended to align image-region attention across original and generated rephrasings. However, this intervention reduced VQA performance and was excluded from the final model.

  • Attention Consistency: The VQA model attends over image regions for an original question, while the VQG model generates a question from the predicted answer and image.The generated question is then answered by the VQA model.
  • Attention Consistency: The added attention consistency loss minimizes the L2 distance between attention distributions for the original and generated questions.
  • Results: 1.34% lower VQA accuracy resulted when attention consistency was added to a cycle-consistent Pythia model on the VQA v2.0 validation split.The model was trained on the train split only.
  • Results: The authors suspect that jointly optimizing multiple diverse losses is difficult and leave the precise cause of the degradation for future work.The final model omits naive attention matching because it was not effective in the reported setting.

D. Question Generation

The VQG module generates questions conditioned on answers across varied answer types, including numbers, colors, and yes/no responses.

  • Question Generation: The VQG model correctly generates answer-conditioned questions for numbers, colors, and yes/no answers.

E. Hyperparameters

The experiments use default hyperparameters for the base VQA models and fixed settings for the VQG module, alongside qualitative visualizations of dataset examples, attention, and generated questions.

  • Hyperparameters: MUTAN, BUTD, Pythia, and BAN use the default hyperparameters from their publicly available implementations.Cycle-consistent variants retain the same parameters for their VQA models.
  • Hyperparameters: The VQG module uses Tsim=0.9, λG=1.0, λC=0.5, and Aiter=5500.
  • Hyperparameters: The VQG module is always trained with a fixed learning rate of 0.0005.
  • Qualitative Analysis: Figure 6 presents original VQA v2.0 validation questions alongside human-provided rephrasings from VQA-Rephrasings.
  • Qualitative Analysis: Figure 7 compares predicted answers and textual and image-region attention for rephrasings under baseline Pythia and Pythia trained with cycle consistency.
  • Qualitative Analysis: Figure 8 shows qualitative examples of answer-conditioned questions generated by the VQG module.
Loading 1902.05660v1…