NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples

Collecting natural adversarial samples

• NaturalBench samples are collected semi-automatically from natural image-text datasets using failed VLM image-caption matches, ChatGPT-generated questions, and human filtering.The procedure does not target a specific VQA model or perturb images or questions.Source: §1 · p. 2, Figure 2 · p. 3

Avoiding “blind” solutions

• Each test sample pairs two images with two questions whose answers alternate, preventing solutions that repeat answers regardless of image or question from succeeding.This balanced protocol is designed to make evaluation depend on visual inputs.Source: §1 · pp. 2–3

NaturalBench

• Even the best closed-source models lag human performance by more than 50% on NaturalBench, whose human accuracy exceeds 90%.The benchmark includes 5,800 yes-or-no and 1,800 multiple-choice samples in its initial collection.Source: §1 · p. 3

What are the challenges?

• The benchmark examines diverse compositional skills and exposes VLMs’ tendency to repeat answers across images or questions.Its tags cover applicable skills rather than assigning just one tag per sample.Source: §1 · p. 3

Dynamic evaluations

• The first release extends to long DOCCI captions and non-English Chinese and Hindi captions, bringing NaturalBench to 10,000 samples.The paper presents this expansion as adaptable to future dynamic evaluations.Source: §1 · pp. 3–4

Benchmarks for vision-language models

• Popular VQA benchmarks assess varied visio-linguistic abilities, but their construction requires substantial human effort to design skills, source images, and create QA pairs.Examples include MMStar [5], MMMU [85], MME [18], and MMBench [51].Source: §2 · p. 4

Biases in vision-language benchmarks

• Benchmarks can contain commonsense-solvable questions and spurious answer patterns, enabling blind systems to answer without viewing images.For VQAv1, questions beginning “Do you see a...” receive “Yes” 87% of the time.Source: §2 · p. 4

Adversarial samples for dynamic model evaluation

• Unlike human-in-the-loop adversarial benchmarks that repeatedly solicit hard examples, this method avoids targeting specific models and requires only single-step human verification.Prior approaches include Adversarial NLI [59], Dynabench [32], and adversarial VQA benchmarks.Source: §2 · p. 4

Challenges in designing VQA benchmarks

• Earlier benchmarks can include commonsense-answerable questions and imbalanced answers, allowing blind systems to exploit knowledge or answer patterns.MME [18] includes examples with highly skewed yes/no answers, while MMStar [5] has skewed answer frequencies for emotion and color questions.Source: §3 · pp. 4–5

Efficient construction of balanced VQA samples (ours)

• The pipeline finds similar image-caption pairs that CLIP mismatches, then uses ChatGPT to generate questions with different answers for the two images.The mismatch criterion compares cross-pair similarity scores against correct-pair scores.Source: §3 · p. 5

Step 1: Collecting pairs of image-text samples

• For Flickr30K, the authors randomly retain one mismatch per sample, collect about 2,000 unique pairs, and discard around 800 pairs whose captions can describe both images.The retained pairs also form an image-text retrieval benchmark challenging even SigLIP.Source: §3 · p. 5

Step 2: Generating questions and answers

• Two annotators judge each generated sample against candidate answers and “Unanswerable”; only unanimous judgments are retained, with up to three question resampling attempts.The combined Flickr30K and DOCCI dataset contains 7,600 English VQA samples.Source: §3 · p. 6

Challenges in designing VQA benchmarks

• On earlier VQA benchmarks, blind GPT-3.5 finetuned without images can outperform chance and sometimes match image-finetuned LLaVA-1.5-7B.The comparison uses equal-sized train and test splits and reports zero-shot and finetuned results.Source: Figure 4 · p. 6

NaturalBench is more robust against blind solutions

• Blind GPT-3.5 does not exceed random chance on NaturalBench, while vision finetuning raises LLaVA-1.5 by 9% and GPT-4o by 10% G-Acc.The English subset was split into equal-sized training and test sets; GPT-4o finetuning used the ‘auto’ setting.Source: Figure 5 · p. 7, §4 · p. 7

Evaluation setup

• G-Acc awards a point only when all four image-question pairs in a sample are answered correctly; Q-Acc and I-Acc measure correctness across both images or both questions, respectively.Models use their default decoding strategy, or can be evaluated deterministically by comparing candidate-answer likelihoods with VQAScore.Source: §4 · p. 7

NaturalBench challenges all state-of-the-art VLMs

• Q-Acc is consistently lower than I-Acc, primarily because models often repeat the same answer for a question regardless of the image.This pattern motivates the paper’s analysis of debiasing VLMs.Source: §4 · p. 7
• Latest VLMs including LLaVA-OneVision, Cambrian-1, Llama3.2-Vision, Molmo, and Qwen2-VL lag humans by 55% to 70% in G-Acc; OpenAI o3 trails by 30%.The comparison covers 55 leading VLMs.Source: Table 1 · p. 8

Compositionality

• NaturalBench samples combine skills such as object recognition, attribute binding, relations, logic, comparison, counting, and world knowledge; each sample receives all applicable skill tags.Its taxonomy includes 8 object types, 8 attribute types, 3 relation types, and 5 reasoning types.Source: §5 · pp. 8–9, Figure 6 · p. 9
• Abstract attributes and orientation are harder than color, size, proximity, or projectivity, and even GPT-4o is challenged by advanced comparison reasoning.The taxonomy supports fine-grained skill analysis rather than assigning each sample a single skill tag.Source: §5 · p. 9, Figure 6 · p. 9

Biases

• Many VLMs favor answers such as “Yes” or “B”; threshold-adjusted debiased metrics outperform original metrics by 35% to 40%.The threshold is tuned within each four-pair sample to enforce specified patterns of different answers.Source: §5 · p. 9
• Debiased metrics assess whether correct image-question-answer combinations rank above incorrect ones, but violate the original task of answering a single image-question pair.The authors therefore examine alternative debiasing techniques in the Appendix.Source: §5 · p. 9

Expanding NaturalBench

• Updating benchmarks with new data sources is important because existing benchmarks can leak into foundation models’ training data.The paper presents the curation method as adaptable to new image-text datasets.Source: §6 · pp. 9–10
• Table 3 reports statistics for the collected benchmarks, with model performance on each dataset reported in the Appendix.The supplied passage does not provide the table’s values.Source: Table 3 · p. 10
• The curation method expands NaturalBench with DOCCI captions over 100 words and XM3600 captions in Chinese and Hindi.The authors use longCLIP for long text sequences and NLLB-CLIP for non-English captions.Source: §6 · p. 10

• Collected samples may inherit biases from web-scraped datasets and foundation models, making human verification crucial.Source: §7 · p. 10
• The analysis focuses on individual skill tags; future work may study combinations of skills.Source: §7 · p. 10
• NaturalBench evaluates VLMs on natural adversarial samples that challenge models more than humans, while penalizing solutions that ignore images.Source: §7 · p. 10
• The benchmark provides comprehensive skill tags for assessing compositional reasoning and highlights biases in VLMs.Source: §7 · p. 10

Outline

• The supplement covers collection, VQA and retrieval performance, skill definitions and analysis, and debiasing techniques.Source: Outline · p. 16

A Collection Details

• The pipeline finds image-text pairs with mismatches in discriminative VLM similarity scores, then uses the pairs to generate questions with different answers.Source: A Collection Details · p. 16
• The paired samples also form an image-text retrieval task, and can be semantically similar without being visually similar.Source: A Collection Details · p. 16
• For DOCCI, captions are shortened to under 230 characters for processing; the authors expect long-context CLIP advances could streamline this step.Source: A Collection Details · p. 16
• Two human annotators choose between two candidate answers and Unanswerable, and samples are retained only if both agree; collection took around 500 annotator hours at 14 dollars per hour.Source: A Collection Details · p. 17
• The supplement includes additional NaturalBench examples, including examples with skill tags.Source: A Collection Details · p. 17, Figure 7 · p. 17

• The supplement reports G-Acc on NaturalBench subsets and multilingual subsets, with the latter restricted to models claiming multilingual capabilities.Source: Table 4 · p. 18, B NaturalBench Performance · p. 18, Table 5 · p. 19
• Translating Chinese or Hindi questions into English often boosts performance, except for top models InternVL-Chat-V1.2-Plus and GPT-4o; NaturalBench-Hindi remains particularly challenging for open-source models.Source: B NaturalBench Performance · p. 18, Table 5 · p. 19
• The adversarial collection procedure produces a much more challenging dataset than alternative Flickr30K collection methods.Source: Table 6 · p. 19, B NaturalBench Performance · p. 19
• NaturalBench-Retrieval evaluates CLIP and SigLIP on 1,200 human-verified paired Flickr samples using text, image, and group scores; the task challenges leading discriminative models.Source: B NaturalBench Performance · p. 19, Table 7 · p. 20

• NaturalBench provides skill definitions and examples, with additional tagging examples intended to support fine-grained analyses such as combinations of skills.Source: C Skill Analysis · p. 21, Figure 8 · p. 21, Table 8 · p. 22
• The skill analysis reports Q-Acc by Object and Attribute tags, and separately by Relation and Reasoning tags.Source: C Skill Analysis · p. 21, Table 9 · p. 23, Table 10 · p. 23
• The definitions span visual properties, part-whole relationships, quantity judgments, and physical arrangements among entities.Source: C Skill Analysis · p. 22
• Examples cover attribute binding, actions, spatial relations, and comparisons across people, animals, and objects.Source: C Skill Analysis · p. 22
• Other tagged skills include comparing characteristics, logical operators limited to negation and universality, and reasoning from external commonsense knowledge.Source: C Skill Analysis · p. 22

D Debiasing Analysis

• VQAScore predicts between two candidate answers by comparing their generative likelihoods, producing deterministic results almost consistent with stochastic decoding.The threshold τ can be adjusted from −1 to 1 for debiasing.Source: D Debiasing Analysis · p. 24
• Global-level debiasing searches for one τ that maximizes G-Acc across all samples, rather than optimizing τ separately for each image-question pair.The main paper’s sample-level approach optimizes τ within each of the four image-question pairs.Source: D Debiasing Analysis · p. 24
• Post-Hoc debiasing compares image-conditioned answer likelihoods with question-only likelihoods, where P(a|q) is estimated without image tokens.The appendix describes this post-hoc technique as equivalent to the displayed likelihood-ratio rule.Source: D Debiasing Analysis · p. 24
• Global-optimal τ and Post-Hoc debiasing perform significantly worse than oracle sample-level τ; global tuning yields only slight gains, and post-hoc debiasing reduces performance for some models.The reported reductions include Bunny, InterVL-Chat-V1.2, and GPT-4o.Source: Table 11 · p. 24, D Debiasing Analysis · p. 24