Source-linked AI summary
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Linjie Li, Jie Lei, Zhe Gan, Jingjing Liu
TL;DR
It remains unclear whether strong VQA models are robust to examples encountered in the wild, despite their progress on standard benchmarks. This paper introduces Adversarial VQA, collected iteratively through human-and-model-in-the-loop attacks, and finds that models perform substantially worse on it than on VQA v2 and other robust benchmarks.
Problem
Despite progress on VQA, it remains unclear whether strong models remain robust on examples encountered in the wild, and existing robustness benchmarks have several design limitations.
Source: §1 · pp. 1–2Method
The authors introduce Adversarial VQA, an iterative benchmark that uses human annotators to attack current models and adds collected examples to train stronger models.
Source: §1 · p. 2Results
SOTA models show significant and universal performance drops on AVQA versus VQA v2 and other robust benchmarks, with particular weakness on counting and reasoning questions.
Source: §1 · p. 2Takeaways & Limitations
AVQA provides a new challenge for designing VQA models intended for real-life applications.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Benefiting from large-scale pre-training, we have witnessed significant performance boost on the popular Visual Question Answering (VQA) task. Despite rapid progress, it remains unclear whether these state-of-the-art (SOTA) models are robust when encountering examples in the wild. To study this, we introduce Adversarial VQA, a new large-scale VQA benchmark, collected iteratively via an adversarial human-and-model-in-the-loop procedure. Through this new benchmark, we discover several interesting findings. (i) Surprisingly, we find that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully. (ii) Both large-scale pre-trained models and adversarial training methods achieve far worse performance on the new benchmark than over standard VQA v2 dataset, revealing the fragility of these models while demonstrating the effectiveness of our adversarial dataset. (iii) When used for data augmentation, our dataset can effectively boost model performance on other robust VQA benchmarks. We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.
1. Introduction
VQA models have advanced, but existing robustness benchmarks leave open whether they handle diverse, real-world examples. Adversarial VQA uses iterative human-and-model collection to test robustness and finds broad vulnerability across models and question types.
1. Introduction
- Existing robustness benchmarks often rely on heuristics, focus on one robustness type, or use VQA v2 images or questions, leaving real-world robustness unclear.Source: §1 · pp. 1–2
- AVQA is collected iteratively: annotators create questions that current models answer incorrectly, those examples train stronger models, and collection repeats across rounds.The dataset draws images from Conceptual Captions, Fakeddit, and VCR, and is intended to remain a moving target rather than a static benchmark.Source: §1 · p. 2
- SOTA models show significant and universal performance drops on AVQA versus VQA v2 and other robust benchmarks, with particular weakness on counting and reasoning questions.The authors also report that examples collected against one set of models challenge other models.Source: §1 · p. 2
- Non-expert annotators can successfully attack SOTA VQA models within two trials on average.Source: §1 · p. 2
2. Related Work
Prior robust VQA benchmarks commonly reuse VQA v2 data or target a single robustness type, while this work collects new questions by having annotators attack current SOTA models. Its iterative collection draws on the HAMLET approach to keep a benchmark challenging as models improve.
2. Related Work
- Prior benchmarks cover distinct targets, including language bias, distribution shift, question rephrasings, image manipulations, and reasoning abilities.Source: §2 · p. 2
- Earlier robust VQA benchmarks often use VQA v2 examples or transformations and focus on a single robustness type, whereas AVQA uses different image sources and direct human attacks on SOTA models.Source: §2 · p. 2
- HAMLET updates models across collection rounds using previously collected examples, allowing a dataset's challenge to evolve as model capabilities advance.The approach was used for ANLI and also for video-and-language future prediction.Source: §2 · pp. 2–3
- In each AVQA collection round, annotators write tricky questions, receive model predictions, judge answers, and verify successful attacks for further answers.Source: Figure 2 · p. 3
3. Adversarial VQA Dataset
AVQA is a large, dynamically collected VQA benchmark built through iterative human attacks and model retraining. Its statistics and comparison with prior datasets highlight both the ease of attacking models and the benchmark’s broader, non-overlapping image and question coverage.
3.1. Data Collection Pipeline
- In HAMLET, annotators repeatedly challenge a model, other workers verify successful attacks, and models are retrained across three rounds to elicit harder questions.Questions exceeding five tries are stopped; attacked questions receive nine additional answers, and items without six confident answers and agreement among ten annotators are removed.Source: §3.1 · p. 3
- 243.0K questions over 37.9K images make up AVQA, with 142.1K/8.7K/26.4K images in the train/val/test split.The dataset combines data collected in R1, R2 and R3.Source: §3.1 · p. 4
3.2. Data Statistics
- R1 and R2 yield about 6.8 and 5.4 questions per image, respectively; authors suspect workers identify vulnerabilities faster than models learn from adversarial examples.Workers increasingly ask about categories such as counting, OCR and commonsense reasoning, where models are more likely to fail.Source: §3.2 · p. 4
- With more diverse R3 images, model error rate drops from 49.0% to 34.4%; authors suspect human-centric VCR images, while higher R1/R2 rates may reflect annotator gains or limited model strengthening.The R1/R2 interpretation assumes constant image-domain difficulty.Source: §3.2 · p. 4
- Fewer than 2 tries suffice on average to attack a model, and average time per successful attack decreases by 15 seconds as collection progresses.This holds even though models used in later rounds were trained with more data.Source: §3.2 · p. 4
3.3. Comparison with Other Datasets
- AVQA spans multiple robustness types and uses non-overlapping, diverse-domain natural images with human-written questions, unlike prior sets based on rule-based questions or manipulated images.Many earlier robust VQA datasets use VQA v2 validation images, which SOTA models may have used for training or hyper-parameter tuning.Source: §3.3 · p. 5
- Unlike ANLI, AVQA’s open-ended VQA construction requires workers to judge model answers before supplying ground truth, and the authors report VQA attacks are harder to defend against with data augmentation.AVQA collects ten answers to evaluate performance under the same criteria as VQA v2.Source: §3.3 · p. 5
4. Experiments and Analysis
The experiments evaluate AVQA across models, robustness factors, other benchmarks, question types, and human versus automatic attacks. Across these analyses, AVQA exposes weaknesses in current VQA models and can improve performance on other robustness benchmarks.
4.1. Model Evaluation
- UNITER-L gains only +2.00 on AVQA over BUTD when trained on “ALL” data, and training on more rounds generally improves performance without making later rounds easier.With the same training data, later-round performance is comparable or lower; nearly doubling the data with examples workers thought models answered correctly did not significantly improve results.Source: §4.1 · pp. 5–6
4.2. Key Factor Analysis
- Language-only models perform poorly on AVQA and similarly on VQA v2, while UNITER-B is not much better than language-only on AVQA.AVQA-only training slightly raises both models’ AVQA performance, but the VQA v2 performance drop is out of proportion to that gain.Source: §4.2 · p. 6
- Random-image regularization that helps on VQA-CP hurts BUTD and UNITER-B on AVQA, and is less effective for UNITER-B on VQA-CP.The method replaces the image with a randomly sampled one during training.Source: §4.2 · p. 6
- ClipBERT performs comparably to UNITER-B on AVQA despite significantly underperforming it on VQA v2, indicating AVQA examples transfer across image representations.VILLA adversarial training brings only slight improvement, leaving a significant AVQA–VQA v2 performance gap.Source: §4.2 · pp. 6–7
4.3. Evaluation on Other Datasets
- AVQA training improves performance on other robustness benchmarks, including reasoning across three datasets and counting on IV-VQA.The paper suggests exposure to more diverse question templates may explain gains on VQA-Rephrasings.Source: §4.3 · p. 7
4.4. Analysis on Question Types
- Counting questions comprise nearly 30% of sampled attacks; later rounds include harder small or occluded-object counts and more relational and commonsense reasoning.The analysis groups questions into counting, OCR, reasoning, and visual concept recognition; OCR and counting together approach 50% in later rounds.Source: §4.4 · pp. 7–8
4.5. Why Human-in-the-Loop?
- Human annotators take fewer tries and succeed more often than automatic attacks, whose generated questions can change the answer or fail to make the model wrong.Word similarity may also fail when synonyms must be grounded in the image.Source: §4.5 · p. 8, Table 8 · p. 8
5. Conclusion
AVQA is an iteratively collected, three-round benchmark built with a human-and-model-in-the-loop procedure across images from different domains. State-of-the-art models perform poorly on it despite large-scale pretraining, adversarial training, sophisticated architectures, and stronger visual features.
5. Conclusion
- AVQA’s diverse robustness types and image domains expose a challenge for developing VQA models suited to real-life applications.The benchmark is collected iteratively across three rounds using a human-and-model-in-the-loop training paradigm.Source: §5 · p. 8
A. More Discussions on AVQA
The discussion frames AVQA as complementary to VQA v2 and other robustness benchmarks, with collection choices and quality controls shaping its scope.
A. More Discussions on AVQA
- AVQA is intended to complement, not replace, VQA v2; the paper recommends reporting results on both alongside other robust VQA benchmarks.The authors expect AVQA to evolve as models grow more robust, helping address saturation and overfitting on static benchmarks.Source: A. More Discussions on AVQA · p. 11
- Question types are not constrained during collection to avoid unconscious dataset-creator bias, and models make more mistakes on Count/OCR/Relation/Commonsense questions.The discussion also notes that robustness benchmarks tailored to individual types can provide useful analysis alongside AVQA’s broader coverage.Source: A. More Discussions on AVQA · p. 11
- Collection rules require objective, image-relevant questions with exact answers; repetitive patterns are manually filtered and answers are checked by annotation.The process validates questions to some extent, making them answerable by humans but not always by models.Source: A. More Discussions on AVQA · p. 11
- The current model selection assumes newer models are more likely to use transformer architectures and proven effective features; broader model coverage is planned.Source: A. More Discussions on AVQA · p. 11
B. Data Statistics
AVQA includes varied question forms and answer distributions, while collection statistics show that examples were generally obtained quickly and answer agreement tracked annotator confidence.
Type of Questions
- About 50 points separate model performance from the upper bounds or estimated human performance on AVQA.Human performance was estimated by using one of ten answers as a prediction and the other nine as references, averaging over ten repetitions.Source: B. Data Statistics · p. 11
Question Types and Lengths
- Most AVQA questions were four to ten words long, and their opening-word patterns included common forms such as “What is,” “How many,” and “Is there.”Questions were also grouped as Y/N, Num, OOV, or Other; OOV questions cannot be answered using VQA v2 answer vocabularies.Source: B. Data Statistics · p. 11, Figure 4 · p. 11, Table 9 · p. 12
Dataset Properties Across Rounds
- Over 80% of verified examples across all three rounds were collected within two tries, and later rounds increasingly produced examples within 100 seconds.Round 3 also had more “not sure” judgments of model answers, indicating that the task was getting harder.Source: B. Data Statistics · p. 11
Answer Confidence and Inter-human Agreement
- More than nine annotators were confident on over 60% of questions on average, and inter-annotator agreement increased with confidence across rounds.Workers were more confident in Rounds 2 and 3 than in Round 1.Source: B. Data Statistics · p. 12
Answer Distribution
- Yes/no answers are typical for “Is,” “Can,” and “Does” questions, while “What is/are” and “What kind/type” questions have diverse answers.Color questions tend to receive color answers, and “Which” questions include specialized responses such as “left” and “right,” similar to VQA v2.Source: B. Data Statistics · p. 12
C. More Visualizations
The visualized examples span several adversarial question categories, while the paper contrasts human-collected questions with automatically generated textual attacks.
More Visualizations
- AVQA examples cover counting, OCR, reasoning, and visual concept recognition, and a question may involve multiple categories.Reasoning examples include positional, commonsense, and comparative questions; visual concepts range from color and shape to action and relations.Source: C. More Visualizations · p. 12
- Automatically generated textual attacks are often invalid or limited to linguistic variations of existing questions, requiring human validation and offering less comprehensive analysis.The illustrated methods include Sears, Textfooler, and Sememe+PSO; the examples include both invalid and valid cases.Source: C. More Visualizations · p. 12
- Figures 6 and 7 plot the number of tries and time spent per verified example across the three collection rounds.The supplied figure descriptions identify the distributions but do not report their values.Source: Figure 6 · p. 13, Figure 7 · p. 13
D. More Results
The domain-transfer evaluation finds that training on Conceptual Captions images improves performance on Fakeddit and VCR, though VCR results are higher across training settings.
More Results
- Including more Conceptual Captions training data improves UNITER-B performance on both Fakeddit and VCR despite image-domain differences.The evaluation uses a split with Conceptual Captions images for training and Fakeddit/VCR images for evaluation.Source: D. More Results · p. 13, Table 10 · p. 13
- VCR performance is higher than performance on the original R3 validation and Fakeddit splits across all training settings, possibly because VCR frames are more human-centric.The paper suggests these frames may be easier than complex or abstract Conceptual Captions and Fakeddit scenes.Source: D. More Results · p. 13
- The appendix also reports detailed results for BUTD, ClipBERT, VILLA-B, and VILLA-L under different settings.Table 11 defines AVQA as R1+R2+R3 and ALL as VQA v2+VGQA+AVQA.Source: D. More Results · p. 13
E. Data Collection Interface
The appendix illustrates the interfaces and supplementary materials used for question and answer collection, alongside examples and distribution visualizations.
Data Collection Interface
- Supplementary figures show examples across rounds and textual attack methods, plus answer distributions by question type.The answer-distribution bars are proportional to each answer’s percentage within its question type; the example visualizations distinguish correct, wrong, and verified adversarial questions.Source: Figure 10 · p. 15, Figure 11 · p. 16, Figure 12 · p. 16
- The question-collection interface lets annotators submit a tricky question against a model and immediately review its response.Annotators then judge the answer as definitely correct, not sure, or definitely wrong; those marking it definitely wrong are prompted to provide a correct answer.Source: Figure 13 · p. 17, Figure 14 · p. 18
- The answer-collection interface asks annotators to provide a concise answer and rate their confidence as yes, no, or maybe.The appendix includes full question-collection instructions and examples shown to annotators.Source: E. Data Collection Interface · p. 13, Figure 15 · p. 19, Figure 16 · p. 20