PaperlayerView exact revision on arXiv

Source-linked AI summary

CREPE: Can Vision-Language Foundation Models Reason Compositionally?

Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, Ranjay Krishna

arXiv:2212.07796v3cs.CLcs.CV

TL;DR

Vision-language models lack large-scale evidence of compositional reasoning, despite compositionality’s importance in vision, language, and downstream applications. CREPE benchmarks systematicity and productivity with Visual Genome-based retrieval datasets and hard negatives, finding consistent failures across unseen compositions and increasing complexity. The results hold across model and training-dataset sizes, while the benchmark remains limited in model coverage and includes noisy productivity negatives.

  • Problem

    Large-scale benchmarks had not directly tested whether vision-language models can reason compositionally, despite their widespread use in tasks requiring compositional reasoning.

  • Method

    CREPE evaluates systematicity and productivity using Visual Genome scene graphs, training-dataset-specific compositional splits, varying-complexity captions, and hard negatives.

  • Results

    Vision-language models struggle with both systematicity and productivity: performance drops on unseen compositions and retrieval performance decays as caption complexity increases.

  • Takeaways & Limitations

    CREPE provides a more systematic benchmark for tracking compositionality in future vision-language models and supplies hard-negative generation methods for evaluation or training.

  • Takeaways & Limitations

    CREPE covers a limited set of publicly available contrastively trained vision-language models, and approximately 70% of HN-SWAP and HN-NEG productivity negatives were rated correct.

Abstract

from arXiv · show

A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, we find that: across 7 architectures trained with 4 algorithms on massive datasets, they struggle at compositionality. To arrive at this conclusion, we introduce a new compositionality evaluation benchmark, CREPE, which measures two important aspects of compositionality identified by cognitive science literature: systematicity and productivity. To measure systematicity, CREPE consists of a test dataset containing over $370K$ image-text pairs and three different seen-unseen splits. The three splits are designed to test models trained on three popular training datasets: CC-12M, YFCC-15M, and LAION-400M. We also generate $325K$, $316K$, and $309K$ hard negative captions for a subset of the pairs. To test productivity, CREPE contains $17K$ image-text pairs with nine different complexities plus $183K$ hard negative captions with atomic, swapping and negation foils. The datasets are generated by repurposing the Visual Genome scene graphs and region descriptions and applying handcrafted templates and GPT-3. For systematicity, we find that model performance decreases consistently when novel compositions dominate the retrieval set, with Recall@1 dropping by up to $12\%$. For productivity, models' retrieval success decays as complexity increases, frequently nearing random chance at high complexity. These results hold regardless of model and training dataset size.

1. Introduction

CREPE addresses the lack of large-scale, controlled evaluations of compositional reasoning in vision-language models by testing systematicity and productivity. Across broad experiments, models struggle to generalize compositionally, with performance declining for unseen compositions and increasing complexity.

  • Compositionality combines meanings of parts into a meaningful whole across language and vision.In language, parts are words; in vision, they include objects, attributes, and relationships.
  • Large-scale benchmarks had not directly evaluated whether vision-language models reason compositionally, despite their use in retrieval, text-to-image generation, and open-vocabulary classification.The gap is especially important because these models increasingly serve as foundations for other systems.
  • Prior retrieval evaluations lacked controlled negatives or varied negatives along only one axis, and did not test unseen compositions or increasing complexity.CREPE targets both systematic generalization and productivity under controlled retrieval conditions.
  • CREPE evaluates systematicity and productivity using Visual Genome scene graphs and datasets aligned with CC-12M, YFCC-15M, and LAION-400M.Systematicity examines seen and unseen atoms and compounds, while productivity examines increasingly complex expressions.
  • Across 7 architectures and 4 training algorithms, models struggle with both systematicity and productivity.The experiments report consistent performance drops between seen and unseen compositions and degradation as caption complexity increases.
  • Up to 12% Recall@1 decrease occurs for models trained on LAION-400M as compositions become less familiar.Productivity performance also decays with increased caption complexity, while dataset size and model size show no clear relationship to compositional reasoning.

2. Related Work

Related work evaluates vision-language models through contrastive pretraining, compositionality benchmarks, and hard negatives. CREPE extends these approaches by scaling controlled hard-negative evaluation and separating compositionality into systematicity and productivity.

  • Contrastively trained models such as CLIP have enabled many computer vision and natural language tasks with notable zero-shot capabilities.CREPE evaluates whether these models capture compositionality, a property present in human vision and language.
  • Compositionality: Compositionality supports systematic combinatorial generalization and can provide benefits such as controllability.Prior systematic-generalization benchmarks largely focused on computer vision or machine learning rather than vision-language models.
  • Compositionality: Productivity had mainly been studied in specialized tasks or toy domains before CREPE’s image-text retrieval evaluation with varying compositional complexity.CREPE uses three datasets to address the lack of benchmarks complementary to large-scale vision-language training data.
  • Evaluation with hard negatives: Hard negatives are commonly used to isolate model capabilities, including atomic replacements, distractor images, and swapped atoms.These strategies target phenomena such as existence, cardinality, actions, spatial relationships, and bag-of-words behavior.
  • Evaluation with hard negatives: CREPE complements Winoground by scaling hard-negative evaluation by three orders of magnitude and testing systematicity, productivity, and multiple foil types.This expands evaluation beyond a small set of human-edited negatives and a single compositionality focus.

3. Compositional evaluation

CREPE operationalizes compositionality as systematicity and productivity, evaluating vision-language representations through retrieval with controlled hard negatives across seen-unseen combinations and increasing caption complexity.

  • 3.1. Compositional language of visual concepts: CREPE defines a scene-graph compositional language in which atoms are objects, relationships, or attributes, and compounds connect multiple atoms.Compounds include attribute-object attachments such as “black dog” and relationship-based object connections such as “man hugs child.”
  • 3.2. Systematicity: Systematicity tests recombination using Seen Compounds, Unseen Compounds, and Unseen Atoms splits defined by training-data exposure.Seen Compounds contains captions whose compounds have been seen; Unseen Compounds keeps atoms seen while withholding at least one compound; Unseen Atoms includes at least one unseen atom.
  • 3.3. Productivity: Productivity evaluates matching images to captions with increasing complexity, measured by the number of atoms in each caption’s corresponding scene graph.The benchmark uses productivity complexities n = 4, 5, . . . , 12.
  • 3.4. Compositional evaluation via retrieval: CREPE uses zero-shot image-to-text and text-to-image retrieval with hard negatives that introduce minimal atomic shifts to expose specific compositional failures.Hard negatives are captions that do not faithfully represent the image; the benchmark includes retrieval sets with ground-truth pairs and generated negatives, plus sets containing only ground-truth pairs.
  • 4.2. Creating productivity datasets: Productivity datasets are generated by random walks over scene graphs, producing subgraphs and ground-truth captions across complexities before adding three hard-negative types.Simple subgraphs use handcrafted templates, while larger subgraphs use GPT-3-generated captions filtered for omitted atoms and low quality.
  • 3. Compositional evaluation: Manual verification rated 87.9% of sampled productivity ground-truth captions faithful to images, while 83.7% of productivity and 86.0% of systematicity hard negatives were genuine negatives.Pairwise annotator agreements were 84.3% and 83.7% for productivity and systematicity hard negatives, respectively.

5. Experiments

Across models and training settings, CREPE experiments show systematicity and productivity failures: performance falls on unseen compositions and increasingly complex captions, without clear benefits from dataset or model scale.

  • Systematicity: Models’ systematicity performance decreases consistently when compounds are unseen, with the largest drops occurring for LAION-400M-trained models.The decrease reaches 6% for ViT-B-32 overall and 12.5% on HN-ATOM.
  • Systematicity: HN-COMP shows little to no seen–unseen difference, likely because its longer foils provide more opportunities to distinguish the ground truth.Recall@1 is overall higher on HN-COMP than HN-ATOM despite larger retrieval sets.
  • Productivity: Retrieval performance decreases with caption complexity on HN-ATOM and HN-SWAP, nearing random chance or falling below it at high complexity.The downward trend also appears in FLAVA and ALBEF, including models trained on Visual Genome images.
  • Productivity: All evaluated foundation models struggle on productivity hard negatives, while training dataset size shows no clear effect on productivity performance.Negation hard-negative performance varies across training sets, with CC-12M models outperforming larger-dataset models in the reported comparison.
  • Scaling: Model size has no clear relationship with systematicity or productivity, although LAION-trained models achieve significantly better absolute performance.The reported models are evaluated across multiple architectures, training algorithms, and datasets.
  • Correlations: ImageNet zero-shot accuracy strongly correlates with absolute hard-negative retrieval performance, but not with systematic or productive generalization.The exception is productivity HN-SWAP, where correlations are weak to nonexistent.

6. Discussion

CREPE provides hard-negative image-to-text retrieval datasets for evaluating systematicity and productivity, and the paper identifies limitations in negative quality, model coverage, and evaluation modalities.

  • Limitations: Approximately 70% of HN-SWAP and HN-NEG negatives were rated correct, creating noise that may hinder future evaluations as models improve.The authors state that this limitation does not invalidate the key productivity result.
  • Limitations: The evaluation covers a limited set of publicly available vision-language models trained with contrastive loss.The computational requirements of foundation-model training constrained the model selection.
  • Limitations: The systematicity analysis lacks text-to-image datasets with hard negatives, leaving counterfactual negative-image generation for future work.The authors report similar trends between text-to-image and image-to-text retrieval in their systematicity experiments.
  • Discussion: CREPE evaluates pretrained vision-language models’ systematicity and productivity using image-to-text retrieval datasets with hard negative texts.The benchmark is intended to support more systematic evaluation as future models improve.

A. Additional details on dataset generation

The dataset-generation pipeline builds compositional image-text examples from Visual Genome scene graphs, then creates captions and hard negatives to probe specific failure modes across complexity.

  • Hard-negative design: CREPE uses hard negatives designed to target common noncompositional failure modes in vision-language models.The benchmark enumerates negative types for both systematicity and productivity evaluations.
  • Systematicity data: A rule-based Stanford Scene Graph Parser creates systematicity splits, achieving high precision but lower recall, especially on noisier LAION-400M captions.The authors argue that parser recall errors would make the systematicity task easier, yet observed SC–UC drops remain.
  • Productivity data: Productivity examples begin with an n-node Visual Genome subgraph sampled by random walk and paired with an image region formed from its atoms’ bounding boxes.Ground-truth captions and hard negatives are generated after quality filtering.
  • Caption generation: Templates generate captions for complexity n = 4, while GPT-3 generates coherent captions for higher complexities using object attributes, relationships, and few-shot examples.Figure 9 illustrates the subgraphs, prompts, and generated captions.
  • Data verification: Generated productivity ground-truth captions remain consistently high in image faithfulness across complexities.Manual verification reports stable faithfulness for complex captions with n ≥ 7.
  • Hard-negative generation: HN-SWAP negatives exchange selected subjects, objects, attributes, or object assignments, while HN-NEG negatives negate captions or individual atoms after graph-based correctness checks.For low complexity, same-type atom permutations are also allowed.

A.5. Test dataset sizes, examples, and additional verification

The evaluation datasets are documented through retrieval-set coverage, hard-negative availability, and human verification of productivity negatives.

  • DRAW_test is a superset of DHN_test ground-truth captions because some captions lacked enough generated hard negatives.The missing hard negatives prevented those captions from supporting DHN_test retrieval.
  • Rare atoms limited valid hard-negative generation for the UA split, so evaluation focuses on the SC and UC splits.Models have seen all atoms in both retained splits, making their comparison the more informative analysis.
  • HN-ATOM had the highest accuracy and pairwise agreement, substantially exceeding HN-SWAP and HN-NEG.A hard negative is accurate when it contains facts that are incorrect about the image.

A.6. Systematicity hard negative dataset details

The systematicity hard-negative splits contain comparable atom-frequency scales, reducing concern that unseen-composition results mainly reflect atom rarity.

  • Training-set atom counts remain on the same scale across SC and UC splits for each training dataset.The analysis uses this comparison to assess whether UC atoms are substantially rarer or more difficult than SC atoms.
  • UC atoms are not substantially rarer or more difficult than SC atoms.

B.1. Full retrieval results on hard negative datasets

Full hard-negative retrieval results preserve the systematicity gap, while relaxing Recall@1 to Recall@3 reduces the difference between seen and unseen splits.

  • Systematicity: Full results cover DHN_test configurations using both HN-ATOM and HN-COMP, HN-ATOM alone, and HN-COMP alone.
  • Systematicity: The SC–UC performance difference decreases when the retrieval metric is relaxed from R@1 to R@3.
  • Productivity: Productivity results are reported for the full DHN_test sets in separate tables.

B.2. Retrieval results on raw datasets

Raw-dataset retrieval reproduces a systematicity drop for unseen compounds, while productivity retrieval becomes easier at higher caption complexity, likely because simple captions create false negatives.

  • Evaluation setup: Each retrieval task pairs one image with every caption in a split, or one caption with every image, and reports mean and standard deviation of Recall@1.Results use K-fold retrievals with K = min(20, floor(|DRAW_test| / N)).
  • Systematicity: Models show a monotonic performance decrease when compounds are unseen in the raw systematicity retrieval sets.
  • Systematicity: The raw systematicity results show similar LAION-model drops for both image-to-text and text-to-image retrieval.
  • Systematicity: 13% is the largest reported drop for ViT-L/14 LAION models across the SC → UC and UC → UA splits.
  • Productivity: Recall@1 generally increases with caption complexity in DRAW_test retrieval.The authors hypothesize that low-complexity captions are often true for multiple images, creating false negatives; higher complexity yields more true negatives.
  • Dataset reporting: The appendix organizes systematicity and productivity counts by hard-negative type and split, with examples of ground-truth and hard-negative captions.

B.3. Retrieval results with all hard negatives at once

With all three hard-negative types combined, models’ Recall@1 decreases as caption complexity increases. The productivity evaluation uses retrieval sets containing one ground-truth caption and fifteen hard negatives.

  • Evaluation setup: Each productivity retrieval set contains one ground-truth caption and fifteen hard negatives, with five from each hard-negative type.
  • Productivity: Recall@1 performance decreases with caption complexity when retrieval sets contain hard negatives of all three types.The hard negatives are HN-ATOM, HN-SWAP, and HN-NEG.
  • Evaluation setup: Figure 11 illustrates ground-truth captions across complexities and gives examples of atomic, swapping, and negation hard negatives.
  • Evaluation setup: The productivity retrieval evaluation includes datasets whose sizes are summarized alongside the systematicity retrieval sets.

B.4. Qualitative analysis on systematicity evaluation

Qualitative and dataset analyses show systematicity gaps between seen and unseen compositions, especially for some LAION-400M models, while productivity performance generally worsens with complexity. The examples indicate that relationship substitutions and atomic foils contribute to systematicity errors, although larger models can sometimes retain correct retrievals.

  • Qualitative analysis: ViT-B-16 often replaces the relationship “on” with “off” or substitutes an object with an atomic foil.For the example “plants on a building,” it retrieves a caption containing “plants on bob and plants off building.”
  • Qualitative analysis: ViT-L-14 continues to retrieve the correct caption on the more challenging Unseen Compounds split, while ViT-B-16 often ranks the ground truth within the top three.This produces a narrower ViT-B-16 Recall@3 gap between Seen Compounds and Unseen Compounds.
  • Systematicity results: Models’ performance decreases from Seen Compounds to Unseen Compounds, particularly for LAION-400M models.
  • Productivity results: Productivity Recall@1 decreases as caption complexity increases for atomic hard negatives and remains near random chance for swapping and negation negatives.The negation condition has exceptions for some OpenAI CLIP models.
  • Systematicity results: Systematicity performance further decreases from Unseen Compounds to Unseen Atoms in the raw-data analysis.
  • Productivity results: Raw productivity results show Recall@1 increasing with caption complexity, contrasting with the hard-negative productivity analyses.
Loading 2212.07796v3…