PaperlayerView exact revision on arXiv

Source-linked AI summary

When and why vision-language models behave like bags-of-words, and what to do about it?

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, James Zou

arXiv:2210.01936v3cs.CVcs.AIcs.CLcs.LG

TL;DR

It is unclear whether VLMs’ strong benchmark performance reflects rich compositional understanding. The paper introduces ARO and analyzes retrieval and contrastive training, finding that models can succeed without composition or order information; composition-aware hard negatives improve understanding.

  • Problem

    It is unclear whether VLMs’ strong benchmark performance indicates rich compositional understanding of text and images.

  • Method

    The paper introduces ARO and evaluates retrieval and contrastive pretraining under perturbations that remove composition and order cues.

  • Results

    VLMs struggle with relation, attribution, and order understanding, while retrieval can remain high without compositional structure.

  • Takeaways & Limitations

    Composition-aware hard-negative mining is a simple training modification that improves understanding of attributes, relations, order, and compositionality.

  • Takeaways & Limitations

    Contrastive objectives do not inherently incentivize learning order structure unless datasets or algorithms are carefully designed for it.

Abstract

from arXiv · show

Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We show where state-of-the-art VLMs have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity. VLMs are predominantly trained and evaluated on large datasets with rich compositional structure in the images and captions. Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency. To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures. We demonstrate that it is possible to perform well on retrieval over existing datasets without using the composition and order information. Given that contrastive pretraining optimizes for retrieval on datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information. This finding suggests a natural solution: composition-aware hard negative mining. We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring understanding of order and compositionality.

1 INTRODUCTION

The paper questions whether strong VLM benchmark performance reflects rich compositional understanding. It introduces ARO, critiques retrieval-based evaluation and contrastive pretraining, and proposes composition-aware hard negatives as a simple fix.

  • Training critique: Contrastive pretraining can likewise succeed without learning compositional information because it optimizes retrieval.This makes ignoring composition a viable shortcut under existing training and evaluation setups.
  • ARO benchmark: ARO evaluates VLMs’ fine-grained understanding of relations, attributes, and word order.The benchmark includes Visual Genome Attributions, Visual Genome Relations, COCO Order, and Flickr30k Order.
  • Findings: State-of-the-art VLMs fail on simple relational, attributive, and caption-order distinctions.Examples include confusing “the black jacket and the blue sky” with “the blue jacket and the black sky” and accepting permuted captions.
  • Evaluation critique: Standard retrieval evaluations can conceal compositional deficiencies because models perform well after order and composition cues are removed.The experiments show that retrieval remains strong even when these cues are removed from captions or images.
  • Proposed remedy: Composition-aware hard-negative mining improves model understanding of attributes and relations.The modification introduces nearby-image negatives and captions with perturbed word order.

2 ATTRIBUTION, RELATION, AND ORDER (ARO) BENCHMARK: WHEN DO MODELS BEHAVE LIKE A BAG-OF-WORDS?

The ARO benchmark targets compositionality through relation, attribution, and order tests built from large-scale datasets. Evaluations reveal severe relational weaknesses and weak sensitivity to correct caption ordering across VLMs.

  • Benchmark motivation: ARO introduces large-scale, targeted evaluation of VLMs’ relation, attribution, and order understanding.The benchmark addresses limitations of smaller tests that can depend on common-sense reasoning or world knowledge.
  • Visual Genome Relation: Visual Genome Relation tests whether models distinguish X relation Y from Y relation X.Examples include “the dog is behind the tree” versus “the tree is behind the dog.”
  • Visual Genome Attribution: Visual Genome Attribution tests whether models assign properties to the correct objects.A test may contrast “the crouched cat and the open door” with “the open cat and the crouched door.”
  • Dataset scale: The Visual Genome tests contain 23,937 relation cases and 28,748 attribution cases.They cover 48 relations and 117 unique attribute pairs, respectively.
  • Results: Most models perform near or below chance on relations, while attribution performance varies substantially across models.BLIP reaches 66% macro accuracy for spatial relations and 56% for verbs; attribution ranges from 62% for CLIP to 88% for BLIP.
  • Order sensitivity: Models show little or no consistent preference for correctly ordered captions in COCO and Flickr30k tests.BLIP’s order performance is closer to chance than its earlier task results, while Flava is below chance on both order tasks.

3 WHY DO MODELS BEHAVE LIKE BAG-OF-WORDS? A CRITIQUE OF RETRIEVAL AND CONTRASTIVE PRETRAINING

Standard retrieval evaluations can remain strong even when captions or images lose order and compositional information. The paper argues that retrieval objectives and datasets therefore permit models to succeed without representing composition.

  • 3.1 LIMITATIONS OF RETRIEVAL AS AN EVALUATION: The evaluation uses COCO and Flickr30k test splits, reporting Recall@1 and Recall@5 for text-image retrieval.COCO contains 5k test images and Flickr30k contains 1k test images under the stated splits.
  • 3.1 LIMITATIONS OF RETRIEVAL AS AN EVALUATION: Caption permutations and image-patch permutations remove substantial order and compositional cues before retrieval is recomputed.Images are split into nine equally sized patches and permuted, while caption words are shuffled using specified strategies.
  • 3.1 LIMITATIONS OF RETRIEVAL AS AN EVALUATION: Most models lose only marginal retrieval performance when caption order or image patch order is perturbed.The experiments evaluate multiple perturbation strategies with repeated runs and standard error bars.
  • 3.2 LIMITATIONS OF RETRIEVAL AND CONTRASTIVE PRETRAINING AS AN OBJECTIVE: Contrastive pretraining optimizes image-text retrieval, so the paper questions what would incentivize models to learn order structure without carefully designed datasets or algorithms.The authors connect this concern to the dominant contrastive training paradigm for VLMs.
  • 3.2 LIMITATIONS OF RETRIEVAL AND CONTRASTIVE PRETRAINING AS AN OBJECTIVE: Retrieval can achieve high performance without composition information because existing datasets lack enough similar-word alternatives that require order-based differentiation.Under these dataset conditions, behaving like a bag-of-words becomes a high-reward shortcut.

4 A SIMPLE FIX: COMPOSITION-AWARE HARD NEGATIVES

The paper extends CLIP with composition-aware hard negatives that force discrimination among similar images and between correct and perturbed captions. This finetuning improves compositional and order-sensitive evaluations while preserving downstream performance.

  • 4 A SIMPLE FIX: COMPOSITION-AWARE HARD NEGATIVES: Composition-aware hard-negative mining adds strong alternative images and word-order-perturbed captions to contrastive learning.Nearest-neighbor images expose fine-grained scene differences, while negative captions target incorrect ordering.
  • 4 A SIMPLE FIX: COMPOSITION-AWARE HARD NEGATIVES: Negative captions swap noun phrases, nouns, adjectives, adverbs, or verb phrases to create compositionally targeted alternatives.The examples swap agents and actions while retaining much of the original caption content.
  • 4 A SIMPLE FIX: COMPOSITION-AWARE HARD NEGATIVES: Strong alternative images are sampled from the K = 3 nearest neighbors computed with CLIP image similarity.The alternatives, their captions, and negative captions are added to each training batch.
  • 4 A SIMPLE FIX: COMPOSITION-AWARE HARD NEGATIVES: NegCLIP improves VG-Relation from 63% to 81%, VG-Attribution from 62% to 71%, COCO Order from 46% to 86%, and Flickr30k Order from 59% to 91%.The model is evaluated on four composition-sensitive tasks and five downstream classification or retrieval tasks.
  • 4 A SIMPLE FIX: COMPOSITION-AWARE HARD NEGATIVES: NegCLIP does not substantially hurt downstream performance while providing substantial gains on order-sensitive tasks.The authors present targeted negative mining as an avenue for improving compositional representation without claiming it is the best possible relational model.

5 RELATED WORK

Prior work has documented compositional weaknesses in vision-language systems, but existing evaluations often use small or broader benchmarks. This paper positions ARO as a large-scale, fine-grained complement focused on relations, attributes, and order.

  • Visio-linguistic compositionality: Winoground evaluates compositional and pragmatic language understanding with 400 image-text matching test cases, where VLMs perform around chance level.The dataset is carefully curated but relatively small, limiting the statistical strength of fine-grained conclusions.
  • Visio-linguistic compositionality: ARO complements Winoground by isolating fine-grained relation and attribution understanding at larger scale.The paper describes ARO as a 50,000-test-case dataset, compared with Winoground’s 400 examples.
  • Visio-linguistic compositionality: Related benchmarks probe compositional generalization, visual question answering, linguistic phenomena, abstract-shape attributes, and text-to-image faithfulness.These include VG-based VQA, GQA, NVLR, VALSE, Zerroug et al.’s abstract-shape benchmark, and DrawBench.
  • Order information in language and vision: Research on language models has also shown that some downstream tasks can be solved while disregarding word order.Prior studies report marginal downstream effects from shuffled-word pretraining and identify GLUE tasks solvable without word order.
  • Negative mining and contrastive learning: Hard negatives have improved representation learning and contrastive learning, while this work applies them specifically to vision-language compositionality.The paper distinguishes its approach by exploring hard negatives with contrastive learning in the vision-language setting.

6 CONCLUSION

The paper finds that VLMs struggle to encode relation, attribution, and order structure, while retrieval performance can remain high without those cues. It reports improved compositional and order understanding from composition-aware hard negative mining and highlights evaluation and training choices as important.

  • 6 CONCLUSION: Large-scale evaluations show that VLMs struggle with relation, attribution, and order understanding.The conclusion frames these evaluations as producing fine-grained and statistically strong insights into model limitations.
  • 6 CONCLUSION: VLMs can achieve high cross-modal retrieval performance without learning order and composition information.The paper argues that retrieval-focused contrastive pretraining can explain why models need not encode these cues.
  • 6 CONCLUSION: Composition-aware hard negative mining improves the compositional and order understanding of VLMs.The modification generates composition-aware hard negatives during model training and is evaluated across several settings.
  • 6 CONCLUSION: The interaction between pretraining objectives and the large datasets used to train VLMs is important for understanding their behavior.The paper identifies composition-aware contrastive pretraining and other pretraining objectives as directions for future study.
  • 6 CONCLUSION: Fine-grained evaluations are needed alongside standard tasks to illuminate model strengths and deficiencies across contexts and goals.The paper connects focused evaluation with understanding where models can be used.

ETHICAL STATEMENT

The paper uses existing large-scale image datasets while addressing ethical concerns about privacy, stereotypes, and resource use. It also emphasizes that compositional failures can matter in downstream generative systems and may contribute to stereotypical outputs.

  • ETHICAL STATEMENT: Existing image datasets have been criticized for insufficient attention to privacy, stereotypical representations, and the resources required for training and evaluation.These concerns are inherited from prior literature on standard image datasets.
  • ETHICAL STATEMENT: The paper introduces no new images but perpetuates use of existing datasets to facilitate comparison with prior work.This limits new data concerns while retaining concerns associated with the standard datasets.
  • ETHICAL STATEMENT: Understanding VLM strengths and deficiencies matters because models such as CLIP increasingly support text-to-image generation used by millions of people.The paper frames this as an ethical dimension of evaluating downstream consequences.
  • ETHICAL STATEMENT: Compositional failures in generated images may include misrepresenting properties assigned to entities and defaulting to stereotypes.The paper cites an example involving a prompt about a disabled woman leading a meeting.

REPRODUCIBILITY STATEMENT

The authors release code for experiments and dataset generation, use released model checkpoints, and document reproducibility procedures. Caption-perturbation experiments use three seeds, while CLIP and NegCLIP experiments use fixed seeds but are not repeated because of computational costs.

  • REPRODUCIBILITY STATEMENT: Code for reproducing experiments and generating the proposed datasets and tasks is released publicly.The statement provides a GitHub repository for the released code.
  • REPRODUCIBILITY STATEMENT: Experiments on caption perturbation use three different seeds to account for randomness in permutation methods.The models are obtained from checkpoints released by their respective repositories.
  • REPRODUCIBILITY STATEMENT: The experiments use checkpoints released for BLIP2, X-VLM, CLIP, and Flava.These checkpoints were obtained from the corresponding repositories.
  • REPRODUCIBILITY STATEMENT: CLIP and NegCLIP experiments were not run multiple times because of computational requirements but used fixed seeds for replication.NegCLIP is implemented as a fork of OpenCLIP and will be released under the same license.

A.1 GENERATING THE DATASETS

The datasets are generated from annotated scene graphs by selecting suitable object pairs, isolating their joint regions, and creating true-versus-swapped relation or attribute captions. The resulting Visual Genome datasets contain 23,937 relation and 28,748 attribution test cases.

  • Candidate objects are identified in GQA scene graphs and filtered using minimum relative width and height criteria.
  • Visual Genome Relation pairs objects from different categories, while Visual Genome Attribution pairs objects with distinct attributes.
  • The smallest bounding box containing each object pair is extracted to reduce distraction from the surrounding Visual Genome scene.
  • True and false captions swap relation direction or exchange attributes between the paired objects.
  • Symmetric relations such as “near” and “next to” are removed during Visual Genome Relation post-processing.
  • 23,937 relation and 28,748 attribution test cases result from the dataset-generation process.

B MODELS

The experiments use several pretrained VLMs and evaluate retrieval and caption-selection behavior under shuffled text or image conditions. The reported tables summarize averaged retrieval or caption-selection performance and fine-grained relation results.

  • The evaluated models include CLIP ViT-B/32, BLIP Base, X-VLM Base, and FLAVA-full.
  • Text-shuffled retrieval performance is averaged over three seeds with standard deviations reported beside each mean.
  • Image-shuffled retrieval performance is averaged over three seeds with standard deviations reported beside each mean.
  • Pick-the-Right-Caption performance is averaged over three seeds with standard deviations reported beside each mean.
  • Table 6 reports accuracy, retrieval recall, macro accuracy, and pick-the-right-caption accuracy across classification, retrieval, relation, attribution, and order tasks.
  • COCO captions are modified by swapping pairs of nouns, adjectives, adverbs, verb phrases, or sufficiently long noun phrases to create negative captions.
  • For each COCO image, three most similar images are collected and one sampled negative image is added to each training batch.

C.3 FINE-TUNING DETAILS

The fine-tuning procedure trains CLIP-FT and NegCLIP on COCO for five epochs, selects learning rates using validation retrieval, and uses a specified optimizer schedule and batch size. The authors could not train an entire model from scratch and expect larger batches to improve results.

  • CLIP-FT and NegCLIP are fine-tuned on COCO for five epochs, with model selection based on COCO validation retrieval performance.
  • Learning rates are swept over {1e−5, 5e−6, 1e−6}, with 50 warmup steps, AdamW, cosine annealing, and N = 1024 batch size.
  • The study could not train an entire model from scratch because of computational-resource constraints.
  • The authors expect larger batch sizes to yield further improvements, noting their N = 1024 setup versus Radford et al.’s N = 32,000.
Loading 2210.01936v3…