Source-linked AI summary
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz
TL;DR
Interpreting unusual image compositions requires connecting visual cues to everyday, physical, social, and cultural knowledge, beyond object recognition. WHOOPS! pairs synthetic commonsense-violating images with annotations and evaluates explanation generation, captioning, cross-modal matching, and visual question answering. State-of-the-art vision-and-language models lagged humans; fine-tuned BLIP2-XXL [25] reached 27% acceptability, while a predicted-image-description GPT-3 (davinci-003) pipeline reached 33%, versus humans’ 95% on explanation generation.
Problem
Interpreting unusual image compositions requires connecting visual cues to everyday, physical, social, and cultural knowledge, beyond object recognition.
Source: §1 · p. 1Method
WHOOPS! pairs synthetic commonsense-violating images with annotations and evaluates explanation generation, captioning, cross-modal matching, and visual question answering.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The dataset, benchmark tasks, evaluation code, and leaderboard provide a resource for studying commonsense and compositionality and tracking replicated results.
Source: §7 · p. 9The authors focus on providing a challenging resource, not establishing the ultimate upper bound on model performance.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field. Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. Data, models and code are available at the project website: whoops-benchmark.github.io
1. Introduction
Humans can interpret unusual image compositions by connecting visual cues to everyday, physical, social, and cultural knowledge, while AI models struggle. WHOOPS! introduces synthetic commonsense-violating images, four evaluation tasks, and an explanation-generation metric.
1. Introduction
- WHOOPS! comprises 500 synthetic images and 10,874 annotations designed to test commonsense and compositionality.Designers modify plausible scenes into implausible combinations and annotate them with captions and explanations.Source: §1 · pp. 1–2
- The benchmark tests explanation generation, image captioning, cross-modal matching, and visual question answering in zero-shot and supervised settings.The explanation task asks models to identify and explain what makes an image weird.Source: §1 · p. 2
- On WHOOPS!, fine-tuned BLIP2-XXL achieved 27% acceptability, while a pipeline feeding predicted image descriptions to GPT3 (davinci-003) reached 33%; humans achieved 95% on explanation generation, and models lagged humans across all tasks.Source: §1 · p. 2
- For automated evaluation of explanation-of-violation tasks, the authors introduce a GPT4-based model metric that compares ground-truth and predicted explanations, achieves accuracy over 81%, and aligns well with human ratings.Source: §1 · p. 2
2. Collecting Weird Images
WHOOPS! uses designer-generated images to probe a range of commonsense violations. Its collection process refines images and explanations, while annotations span diverse categories of weirdness.
2. Collecting Weird Images
- The collection aims to make weirdness understandable across audiences, although the concept is subjective, ambiguous, and culture-dependent.Designers and annotators come from different countries and continents.Source: §2 · p. 2, §2.1 · p. 3
2.1. Human Generated Synthetic Images via Text-to-Image Models
- Designers create synthetic images by replacing one element in a plausible co-occurring pair with an implausible alternative, yielding 500 images.For example, a smartphone replaces the notebook in an imagined scene of Einstein.Source: §2.1 · pp. 2–3
- Expert designers spend around 25 iterations per image to clarify its intended weirdness and remove extraneous elements.Each image is presented to a small control group; images that fail are returned for refinement or alternative concepts.Source: §2.1 · p. 3
2.2. Commonsense Categorization of Weird Images
- WHOOPS! images were manually annotated across 26 commonsense categories, covering temporal, physical, nutritional, environmental, activity, symbolic, and folklore-related violations.The category analysis is intended to represent the diversity and complexity of reasoning skills needed for the benchmark.Source: Figure 3 · p. 3, §2.2 · p. 4
3. Related Work
Prior work covers visual commonsense, compositionality, abductive reasoning, and multimodal humor, among other tasks. WHOOPS! differs by using synthetic images deliberately designed to violate expectations.
3. Related Work
- Related vision-and-language work addresses visual understanding, commonsense association and analogy, compositionality, visual abduction, and multimodal humor.Examples include VCR, WinoGAViL, VASR, Winoground, and Sherlock.Source: §3 · p. 4
- Recent instruction-following and in-context or zero-shot models provide approaches for complex vision-and-language reasoning tasks.The passage names BLIP2, Flamingo, and MLLM among these developments.Source: §3 · p. 4
- WHOOPS! uses synthetic images designed to challenge commonsense and compositional reasoning, unlike prior work focused on pre-existing images.Image generation makes unique, complex scenes difficult or impossible to obtain otherwise.Source: §3 · p. 4
4. Using Weird Images to Create V&L Tasks
WHOOPS! supports four vision-and-language tasks, including explanation generation that tests recognition of violated commonsense rules and relations among image elements. Its annotations and filtering procedures are designed to make the tasks usable and to check their quality.
4.1. Explanation-of-violation Generation
- Explanation-of-violation generation tests whether models can identify the commonsense rule an image violates and explain relations between its elements.The task includes identifying whether an image is weird and explaining what makes it weird.Source: §4 · p. 4, §4.1 · p. 4
- Humans achieved 92% accuracy when classifying 100 paired weird and normal images, with three annotators agreeing in 70% of cases.Final classifications used majority vote; the reported results suggest humans generally agree on weirdness in WHOOPS!'s setting.Source: §4.1 · p. 4
4.2. Established Tasks
- The established tasks are captioning, cross-modal matching, and VQA, with matching requiring detailed captions to rank above accurate but underspecified alternatives.Captioning describes the unusual scene without explaining its incongruity; VQA questions are generated from captions and filtered to remove instances answerable by text alone.Source: §4.2 · p. 5
- Human verification agreed with the automatically generated VQA labels in 97% of sampled cases, supporting the quality of the QA instances.Three annotators reached full agreement in 94% of the sample.Source: §4.2 · p. 5
4.3. Toxic Content Filtering
- Only 0.4% of captions, 0.1% of explanations, and 0.4% of underspecified captions contained toxic language; those instances were removed and replaced.The authors manually screened images and used the Perspectives API to filter toxic language.Source: §4.3 · p. 5
5. Experiments
Experiments evaluate explanation generation, captioning, cross-modal matching, and VQA in zero-shot and supervised settings. Models lag human performance on explanation tasks, and the paper also assesses an automatic explanation metric against human judgments.
5. Experiments
- Models were evaluated zero-shot and with five-fold supervised cross-validation, while explanation pipelines passed predicted or oracle captions to GPT3.The supervised setup fine-tuned BLIP2's Qformer parameters and jointly trained one model across tasks.Source: §5 · pp. 5, 7, Figure 6 · p. 7
5.2. Results
- Fine-tuned BLIP2 FlanT5-XXL generally performed best on captioning, matching, and VQA, but supervised VQA peaked at a BEM score of 57%.Despite a relatively high CIDEr score, zero-shot BLIP2 captions received a 49% human rating; the text-only supervised baseline performed poorly on captioning and VQA.Source: Table 2 · p. 6, §5.2 · p. 8
- 73% was the best end-to-end fine-tuned BLIP2 FlanT5-XXL accuracy on identifying unusual images, while oracle-caption explanation reached 68% versus 95% for humans.The oracle system receives a ground-truth human-authored image description; humans outperform models on both explanation subtasks.Source: Table 1 · p. 6, §5.2 · p. 7
5.1. Automatic Evaluation for Explanation-of-violation
- Automated explanation ratings achieved accuracy higher than 81%, with a maximum 9% deviation from human ratings for Predicted Caption →GPT3.The GPT4-based metric compares ground-truth and candidate explanations for equivalence.Source: §5.2 · pp. 7–8, §5.1 · p. 7
6. Analysis
The analysis finds that BLIP2 captioning errors are concentrated on weird images rather than arising mainly from synthetic images. Performance also varies across commonsense categories, with strengths and weaknesses differing by category.
6.1. Main Challenge: Weirdness, not Synthesis
- The text-only model performed well on matching at 94%, so the authors recommend using matching only as a zero-shot evaluation.The authors suggest it may learn to prefer more detailed captions without seeing the image.Source: §6.1 · p. 8
- BLIP2 caption accuracy was 89% on natural and normal images but 49% on weird images, pointing to weirdness rather than synthesis as the main challenge.For 40% of image triplets, incorrect captions occurred only for the weird image; naturalness-related errors affected 3% of cases.Source: §6.1 · pp. 8–9
6.2. Performance by Commonsense Categories
- Predicted Caption →GPT3 outperformed supervised BLIP2-XXL in 46% of categories, underperformed in 23%, and performed similarly in 31%.The comparison spans commonsense categories rather than a single aggregate score.Source: §6.2 · p. 9
- Both models performed poorly on temporal discrepancy and art knowledge, but well on Unsuitable Environment.Examples include a smartphone in a Renaissance-era scene, The Girl with a Pearl Earring wearing a golden hoop, and a snowman on a sunny beach.Source: §6.2 · p. 9
7. Conclusions
The conclusion presents WHOOPS! as a benchmark for commonsense and compositional reasoning, with explanation generation showing a substantial human–model gap. It also provides evaluation code and a leaderboard for tracking and replicating results.
7. Conclusions
- WHOOPS! combines synthetic images with annotations for explanation, captioning, underspecified captions, and VQA across four benchmark tasks.The dataset is intended to challenge AI models' commonsense and compositional reasoning.Source: §7 · p. 9
- State-of-the-art models struggled especially with explanation generation, where a substantial gap from human performance remains.Source: §7 · p. 9
- Evaluation code and a leaderboard support methodical tracking and replication across the four benchmark tasks.Source: §7 · p. 9
8. Limitations
The authors note potential content objections despite filtering, a smaller dataset than other benchmarks, and that their focus is a challenging resource rather than an ultimate model-performance upper bound.
8. Limitations
- Despite filtering potentially harmful or offensive images and texts, some individuals may still find content objectionable.The authors invite reports of harmful cases for removal from the dataset.Source: §8 · p. 9
- WHOOPS! has fewer images than other benchmarks, with expansion planned to enhance its usefulness.The authors say its smaller size enabled efficient manual annotation and evaluation while supporting data quality and reliability.Source: §8 · p. 9
- The work prioritizes a challenging commonsense and compositionality resource over achieving the ultimate upper bound on model performance.Source: §8 · p. 9
A. Appendix
The appendix provides further details and examples relevant to reproducibility, and credits the title icon to Freepik via Flaticon.
A. Appendix
- The title’s magic icon is credited to Freepik via Flaticon.Source: A. Appendix · p. 12
- The following sections provide reproducibility details and examples about model evaluation, training, and data collection.Source: A. Appendix · p. 12
A.1. Model versions
The authors use the official implementations of the listed vision-and-language models, with versions specified in the paper and default hyperparameters unchanged.
A.1. Model versions
- The experiments use official implementations of CLIP, OFA, CoCa, BLIP, and BLIP2, retaining the paper-specified versions and default hyperparameters.Source: §A.1 · p. 12
A.2. Supervised Data Details
The appendix documents supervised training details, image-generation guidelines, commonsense categories, and human annotation procedures.
A.2. Supervised Data Details
- Supervised tasks use a multitask BLIP-2 Flan-T5 sequence-to-sequence setup trained on image and prompt inputs to predict textual targets; the authors recommend WHOOPS! primarily as a test set.A single run of 15 epochs takes around 2 hours on 8xA6000 GPUs; the full training schedule described takes approximately 72 hours, or 576 GPU hours.Source: §A.2 · p. 12
A.3. Image Generation Designers Guidelines
- Image prompts replace a plausible element X1 with a similar X2 in a normal real-world pairing, creating a conceptual commonsense violation while avoiding unrelated visual distortions.The images should be relatively realistic and include one weird element that requires logic and general knowledge.Source: §A.3 · p. 12
A.4. Commonsense Categories
- WHOOPS! organizes images into 26 commonsense categories, including temporal discrepancy, physics rules, inability to execute, and unsuitable or unnatural environments.For example, trees blocking wind from a turbine illustrate inability to execute, while car racing in the Colosseum illustrates an unsuitable environment.Source: §A.4 · p. 13
A.5. Human Annotation
- Human annotation includes qualification screening, VQA answer verification, and selection of acceptable explanations that identify why an image is unusual.Explanations are rejected if they omit the reason, rely on generic statements, contain incorrect information, or require search-engine verification.Source: §A.5 · p. 15, Figure 13 · p. 15