Source-linked AI summary
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross
TL;DR
Vision-and-language models need to reason compositionally when identical caption words are reordered, but existing evidence about this ability is limited. The paper introduces Winoground, a hand-curated, tagged matching task, and finds that diverse models generally perform no better than chance, motivating targeted future work while leaving scope limitations around language, size, and curation.
Problem
It asks whether vision-and-language models can conduct visio-linguistic compositional reasoning when captions contain identical words in different orders.
Method
The paper introduces Winoground, a hand-crafted, fine-grained-tagged task matching two images with two captions and evaluates transformers and RNN-based models.
Results
Models rarely—and only barely—outperform chance on Winoground, indicating weak visio-linguistic compositional reasoning.
Takeaways & Limitations
Winoground is intended as a probing evaluation set to guide research toward more robust vision-and-language models.
Takeaways & Limitations
Winoground is English-only, limited in size, and costly to curate with experts.
Abstract
from arXiv · showhide
We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground. Given two images and two captions, the goal is to match them correctly - but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set of fine-grained tags to assist in analyzing model performance. We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance. Evidently, these models are not as skilled at visio-linguistic compositional reasoning as we might have hoped. We perform an extensive analysis to obtain insights into how future work might try to mitigate these models' shortcomings. We aim for Winoground to serve as a useful evaluation set for advancing the state of the art and driving further progress in the field. The dataset is available at https://huggingface.co/datasets/facebook/winoground.
1. Introduction
Winoground tests whether vision-and-language models can match images with captions whose identical words appear in different orders. Despite broad multimodal capabilities, evaluated models rarely exceeded chance, motivating further analysis and model development.
- Task and motivation: The task targets a gap in models’ understanding of word order and visual grounding, although humans readily distinguish the resulting scene differences.Transformers have been reported to be relatively insensitive to word order.
- Task and motivation: Winoground matches two images with two captions containing the same words in different orders, testing visio-linguistic compositional reasoning.Success requires compositional sensitivity within text and images, plus synthesis across modalities.
- Dataset: Expert annotators hand-crafted Winoground and added fine-grained tags so it can function as a targeted probing task for model analysis.The tags support more detailed examination of what models learn.
- Evaluation: The study evaluates state-of-the-art vision-and-language transformers and RNN-based models on Winoground.The evaluated families include both transformer and sequence-based architectures.
- Findings: Models rarely—and only barely—outperform chance, indicating that their visio-linguistic compositional reasoning falls well short of expectations.The paper analyzes model performance to identify directions for more robust systems.
2. Related Work
Winoground extends multimodal stress testing and probing by holding caption words constant while changing their order. This design isolates compositional reasoning more directly than prior stress tests and connects to evidence that transformers may rely on co-occurrence statistics.
- Visio-linguistic stress testing: Existing multimodal stress tests cover implausible scenes, modality ablations, word mismatches, hate speech, and memes, but do not keep caption vocabulary constant.Winoground removes biases from using altogether different words.
- Visio-linguistic stress testing: Because Winoground holds the word set fixed, unstructured bag-of-words models cannot succeed through vocabulary matching alone.The task therefore targets compositional distinctions rather than simple word presence.
- Probing: NLP probing work reports that word order has limited impact on large transformer language-model pretraining performance across multiple metrics.This finding motivates testing whether apparent word-order understanding reflects higher-level co-occurrence statistics.
- Winograd schemas: Winoground applies the twin-sentence idea from Winograd schemas to image grounding, using identical words in different orders.Winograd-style formats have been used across several language-related tasks.
3. Winoground
Winoground is a hand-curated probing dataset built from image-caption pairs whose captions share words but differ in order. It uses fine-grained linguistic and visual tags, and evaluates matching with text, image, and group scores.
- Dataset construction: Each Winoground example pairs two captions and two images, with annotators preferring the correct pairings and captions differing only in word or morpheme order.
- Tagging: The dataset contains 70 swap-dependent linguistic tags covering object swaps, relation swaps, and swaps involving both relations and objects.
- Tagging: Examples are additionally tagged by main-predicate count and by three nonmutually exclusive visual reasoning tags, including Pragmatics and Series.Every example has one or two main predicates; visual tags apply only when the corresponding phenomena occur.
- Dataset construction: 1600 image-text pairs comprise 400 examples, including 800 correct and 800 incorrect pairings with 800 unique captions and images.
- Metrics: The text score tests whether each image receives its ground-truth caption a higher score than the alternative caption.
- Metrics: The image score reverses the retrieval direction, while the group score requires all four image-caption combinations in an example to be scored correctly.
4. Experimental Setup
The experiments compare diverse multimodal transformers and RNN-based models using image-text alignment scores, alongside a crowdworker human baseline. The setup also accounts for differences in architectures, pretraining objectives, and model inputs.
- Models: The evaluation includes CLIP, FLAVA, LXMERT, UniT, UNITER, VILLA, VinVL, ViLT, VisualBERT, and ViLBERT, plus VSE++ and VSRN.
- Model configurations: The transformer comparison distinguishes single-stream from dual-stream encoders and documents pretraining datasets, architectures, and intermodal attention mechanisms.
- Model configurations: Models are selected because they use an image-text matching classification head or produce a similarity score between image and text.
- Multimodal RNNs: VSE++ and VSRN provide sequence-based baselines with hardest-negative losses, GRU language representations, and separate image-embedding pipelines.
- Human baseline: Crowdworkers label all 1600 image-caption combinations, and the human image-caption score is the proportion judging each pair to match.
5. Results
Winoground exposes a large human–model gap in visio-linguistic compositional reasoning, with models struggling across linguistic and visual phenomena and often performing near or below chance.
- Compared to humans: All models fall below random chance on group score, whereas human group performance is not appreciably lower than their text and image scores.Confidence intervals for these results are reported in Appendix A.
- Linguistic tags: Models perform poorly across linguistic tags, generally favoring shorter, simpler captions over longer, more compositional ones.Humans perform better on examples with two main predicates, while models largely show the opposite pattern.
- Linguistic tags: CLIP reaches human-comparable text-score performance on the both tag, whose 26 examples contain short, weakly compositional captions.The examples include morpheme-level swaps such as “presenting the watch” versus “watching the present.”
- Visual tags: Humans underperform on pragmatics partly because the task presentation may not have provided both images and captions simultaneously.The authors hypothesize that more simultaneous information or more annotators would substantially raise human scores.
6. Discussion
The discussion links stronger text-score performance to richer features and identifies visual discrimination and caption complexity as important challenges, without finding a clear architecture advantage.
- 6.1. Capabilities of Encoders: Only UNITER, VILLA, VinVL, ViLT, FLAVA, and CLIP exceed chance, and only on text score, possibly because of richer unimodal or multimodal features.The proposed explanations include large-scale pretraining, object-detector training data, and ViT image features.
- 6.1. Capabilities of Encoders: Most models receive 0% group score on the same image series tag, consistent with choosing one image or caption regardless of the caption.The authors suggest weak visual encoders may fail to discriminate substantially similar images, encouraging fallback to unimodal priors.
- 6.1. Capabilities of Encoders: Longer captions correlate with lower performance for the best models, while the correlation is reversed for humans.The shortest captions are also the least compositional, often involving part-of-speech or morpheme-level swaps.
- 6.1. Capabilities of Encoders: Models’ image-pair scores typically show a weak correlation with captions having low perplexity.Perplexity was computed using the standard-size GPT2 checkpoint.
- 6.2. By Architecture & Type of Attention: Single-stream and dual-stream models both perform significantly worse than humans, with no distinct performance difference established by architecture or attention type.At least one model from each architecture is above chance in most experiments, but co-attention models remain below chance.
- 6.2. By Architecture & Type of Attention: After excluding CLIP and FLAVA as outliers, multimodal pretraining dataset size correlates highly significantly with model scores.Unimodal training data for image backbones or initialized text encoders is excluded from these calculations.
7. Conclusion
Winoground exposes substantial shortcomings in state-of-the-art models’ visio-linguistic compositional reasoning. The dataset also has practical scope and curation limitations that constrain its use.
- Models generally perform no better than chance on Winoground’s visio-linguistic compositional reasoning task.
- Future work could investigate single-stream models, more pretraining data, improved image encoding, and objectives emphasizing similar but incorrect images.
- Winoground is English-only, limited in size, and expensive to curate because it relies on expert annotation.
A. Confidence Intervals
The paper reports 95% confidence intervals for aggregate Winoground results to quantify uncertainty around model performance.
- The dataset was divided into four equal-sized groups, producing four scores per model and score type for Student’s t-distribution intervals.
- 95% confidence intervals are provided for aggregate Winoground results.Results above chance are shown in bold.
B. Impact of Pretraining Data Size and Model Type on Model Performance
Model performance correlates significantly with pretraining data size, while model-type analyses distinguish single-stream and dual-stream behavior. Figures visualize these relationships while excluding CLIP and FLAVA as outliers.
- Impact of Pretraining Data Size: Correlations between pretraining data size and model performance are highly significant in every case.
- Model Type: Most single-stream models perform slightly above chance on the text score.
- Model Type: CLIP and FLAVA are the only dual-stream models above chance, with drastically more training data than the other models.
- Figures: Performance is plotted against numbers of pretraining images and captions, with CLIP and FLAVA removed as outliers.Backbone pretraining data is not included in the plotted counts.
D. Heatmaps for the Word-Region Alignment Models
The appendix provides word-region alignment heatmaps for UNITER, ViLLA, and ViLT, focusing on models trained with alignment objectives.
- Heatmaps are provided for UNITER, ViLLA, and ViLT because they were trained with a word-region alignment objective.
- The figures visualize word-region alignment scores between image and text features for ViLLAbase and UNITERbase.
E. Mechanical Turk Interface
The Mechanical Turk validation process required qualified English-speaking crowdworkers to pass onboarding, while Winoground’s annotation process addressed representation and identity-related concerns.
- Worker Qualification: Crowdworkers had to be English speakers with 98% prior HIT approval, 1,000 completed HITs, and a passing onboarding test.The onboarding test contained ten independent image-caption matching questions and used the same interface as the task.
- Validation Interface: The validation interface was implemented on Amazon Mechanical Turk.
- Annotation Considerations: Annotators avoided gendered terms and swaps involving gender, race, or ethnicity because image-only identity judgments could reflect cultural assumptions.
- Representation: Annotators were encouraged to select images representing varied perceived race, gender, disability, and related dimensions.Getty Images metadata was searched for gendered and gender-neutral terms as a rough proxy for gender representation.