PaperlayerView exact revision on arXiv

Source-linked AI summary

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

Martha Lewis, Nihal V. Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H. Bach, Ellie Pavlick

arXiv:2212.10537v3cs.CVcs.AIcs.CL

TL;DR

Whether vision-and-language models encode grounded compositional meaning remains open, especially for compositions requiring abstract, syntax-sensitive binding. We compare CLIP variants and compositional distributional models across composition tasks; CLIP handles single-object compositions but struggles when binding is required, while the harder tasks remain difficult across models.

  • Problem

    Whether large neural models provide compositional and grounded semantic representations remains unresolved, including whether they bind meanings to abstract syntactic roles.

    Source: §1 · p. 1

  • Method

    The study evaluates CLIP and fine-tuned variants alongside composition-aware models on single-object, two-object, and relational visual composition tasks.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    CLIP performs well on single-object adjective–noun compositions but struggles with variable-binding tasks, where compositional models also approach chance performance.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The results suggest that vision-and-language model pretraining should account for binding and order to support better compositional generalization.

    Source: §7 · p. 9

    The findings do not settle normative claims about the semantics learned by vision-and-language models; further work and debate are needed.

    Source: §1 · p. 1

Abstract

from arXiv · show

Large-scale neural network models combining text and images have made incredible progress in recent years. However, it remains an open question to what extent such models encode compositional representations of the concepts over which they operate, such as correctly identifying "red cube" by reasoning over the constituents "red" and "cube". In this work, we focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts and to bind variables in a structure-sensitive way (e.g., differentiating "cube behind sphere" from "sphere behind cube"). To inspect the performance of CLIP, we compare several architectures from research on compositional distributional semantics models (CDSMs), a line of research that attempts to implement traditional compositional linguistic structures within embedding spaces. We benchmark them on three synthetic datasets - single-object, two-object, and relational - designed to test concept binding. We find that CLIP can compose concepts in a single-object setting, but in situations where concept binding is needed, performance drops dramatically. At the same time, CDSMs also perform poorly, with best performance at chance level.

1 Introduction

Whether vision-and-language models provide grounded compositional representations remains open, especially for language requiring abstract variable binding. This work tests CLIP and compositional distributional models in controlled visual tasks and finds that binding remains difficult across models.

1 Introduction

  • Semantic representations are expected to combine word meanings according to syntax and to be grounded, yet whether large neural models meet these criteria remains unresolved.Evidence and arguments support opposing claims about whether models are compositional and grounded.Source: §1 · p. 1
  • The benchmark tests adjective–noun composition, two adjective–noun pairs, and relations between nouns, with the latter tasks requiring binding meanings to abstract syntactic roles.Examples include “red cube” and “cube behind sphere.”Source: §1 · p. 1
  • Meaningful distractors are intended to avoid “hackable” label artifacts and make the benchmark require both the image and labels for prediction.The authors contrast their controlled setting with datasets where incorrect labels may not require using the image.Source: §1 · pp. 1–2
  • CLIP outperforms compositional distributional models on single adjective–noun compositions, but binding tasks leave all models essentially at chance in the best case.CDSMs sometimes improve performance in binding settings, but not consistently.Source: §1 · p. 2
  • The work adapts traditional compositional distributional semantics architectures to grounded language and compares them with CLIP text encoders.These architectures represent linguistic composition within vector spaces while retaining traditional compositional structure.Source: §1 · p. 2

2 Models

The paper compares CLIP-based methods with compositional distributional semantics models (CDSMs) for representing phrases grounded in images. Across a shared setup, it tests pretrained and fine-tuned CLIP variants and several explicitly compositional architectures.

2.1 Setup

  • Phrase embeddings are evaluated against image embeddings in a shared setup, with CLIP supplying image vectors and either its text encoder or a CDSM supplying phrase vectors.All models except frozen CLIP are trained on the phrase data; compositional models update the word embeddings used to form phrases.Source: §2.1 · pp. 2–3

2.2 CLIP and Variants

  • CLIP predicts by choosing the text vector with the highest cosine similarity to an image vector, using prompts for adjective–noun and subject–relation–object phrases.The pretrained model was trained contrastively on 400 million image–text pairs; the paper also considers frozen and fine-tuned variants.Source: §2.2 · p. 3
  • Compositional soft prompting (CSP) fine-tunes adjective and object token embeddings on seen classes while keeping the text and image encoders frozen, then recombines tokens for zero-shot inference.The paper evaluates CSP on adjective–noun and subject–relation–object tasks.Source: §2.2 · p. 3

2.3 Compositional Distributional Semantics Models (CDSMs)

  • CDSMs learn word-level vectors, matrices, or tensors and compose them into phrase embeddings aligned with frozen CLIP image embeddings.The compared models include architectures designed to represent linguistic composition within vector spaces.Source: §2 · p. 2, §2.3 · p. 3

Syntax Insensitive Models (Add, Mult, Conv)

  • Add, Mult, and circular convolution compose word vectors without encoding word order.Add uses vector addition, Mult uses pointwise multiplication, and Conv uses circular convolution.Source: Syntax Insensitive Models (Add, Mult,… · p. 3
  • The type-logical model composes adjective–noun phrases by matrix–vector multiplication and relational phrases by matrix–vector multiplication followed by pointwise multiplication.Nouns are vectors, while adjectives and relations are matrices.Source: Syntax Insensitive Models (Add, Mult,… · p. 4
  • The role-filler model represents a phrase as role–filler bindings combined with circular convolution.For red cube, red and cube are fillers bound to adjective and noun roles, respectively.Source: Syntax Insensitive Models (Add, Mult,… · p. 4

3 Concept Binding Benchmark

The concept-binding benchmark uses synthetic image-and-label datasets to test compositional generalization in vision-and-language models. Its three tasks vary from single-object adjective–noun labels to two-object and relational compositions, with held-out classes and challenging distractors.

3 Concept Binding Benchmark

  • The benchmark comprises single-object, two-object, and relational datasets with training, validation, and generalization sets that have no overlap in true class labels.All individual nouns, adjectives, and relations occur in training, while validation and generalization test held-out combinations; the datasets include semantically meaningful and hard labels.Source: §3 · pp. 4–5
  • Single-object images contain one colored shape and use adjective–noun labels, with four distractors sampled from other possible combinations.The dataset uses cubes, spheres, and cylinders with eight colors, and holds out specified combinations for validation and generalization.Source: §3 · p. 5
  • Two-object images contain differently colored objects, and distractors include swapped adjective–object pairings such as red sphere for a red cube label.The splits use the same shape–color combinations as the single-object task.Source: §3 · p. 5
  • Relational images pair two objects with a spatial relation, and distractors include reversed object order, the opposite relation, and alternative object types.Validation and generalization hold out distinct object-relation combinations; object shapes can appear on either side of the image.Source: §3 · p. 5

4 Experiments and Results

The experiments benchmark CLIP variants and compositional models on single-object, two-object, and relational datasets. Single-object composition is stronger than settings that require binding, where generalization is poor.

4 Experiments and Results

  • Across the benchmark, performance is strongest for single-object composition, while two-object and relational tasks show weak generalization when binding is required.The reported two-object models overfit or underfit, and relational models generally fail to generalize beyond chance.Source: §4.1 · p. 5, §4.2 · pp. 6–7, Table 4 · p. 6

4.1 Single Adjective-Noun Composition

  • Frozen CLIP reaches 97.75% validation and 92.39% generalization accuracy on single-object composition, outperforming the compositional models.Add and RF reach 80.64% and 85.16% on generalization, respectively; Conv, Mult, and TL do not generalize to validation or generalization sets.Source: §4.1 · pp. 5–6

4.2 Two-Object Adjective-Noun Binding

  • CLIP-FT overfits two-object training data, reaching 86.91% training accuracy but 6.31% validation and 0.25% generalization accuracy.CSP performs somewhat better at its validation-selected epoch, but it too overfits as training progresses; additive Add and RF underfit and achieve random validation and generalization accuracy.Source: §4.2 · pp. 6–7, Table 4 · p. 6
  • Two-object errors are similar across models, but Conv often gets both adjective and noun wrong, while Add and RF show a slight bias toward adjective errors.For most models, errors are split between adjective-only and noun-only mistakes, with only a small proportion getting both wrong.Source: Table 5 · p. 6, §4.2 · p. 7

4.3 Relational Composition

  • Frozen CLIP is slightly above chance on relational training but worse on validation and generalization; fine-tuned CLIP, CSP, and TL also fail to generalize.Other CDSMs are slightly above chance, while the commutative Add, Mult, and Conv models cannot represent order.Source: §4.3 · p. 7, Table 6 · p. 7

5 Discussion

CLIP can compose object properties in single-object settings, but its performance does not extend reliably to abstract binding. The authors caution that their experiments concern CLIP and that existing benchmarks may not test binding adequately.

5 Discussion

  • CLIP disassociates objects and adjectives in the single-object setting, but appears to lack the richer structure needed for syntax-sensitive variable binding.Fine-tuning CLIP or training compositional models does not improve generalization on unseen two-object and relational classes.Source: §5 · pp. 7–8
  • CLIP-FT overfits, CSP sometimes improves validation or generalization, and Add, Mult, Conv, TL, and RF show task-dependent limits.Add generalizes to held-out single-object classes, while the syntax-insensitive models are near chance on relational data and RF is at chance on two-object and relational data.Source: §5 · p. 8
  • The study focuses only on CLIP; newer visual encoders may perform better, and progress in visual and text encoding may both be needed.The authors motivate pretraining methods that account for binding, while treating broader conclusions conservatively.Source: §5 · p. 8
  • Common compositional zero-shot benchmarks test adjective-noun combinations with one object, not abstract order-aware syntax, so their scores do not establish binding ability.The paper reports high CLIP and CSP accuracy on its single-object dataset but a dramatic drop on two-object and relational datasets.Source: §5 · p. 8

6 Related Work

The related work connects the paper to linguistic composition models and compositional vision benchmarks. It frames the study as a controlled investigation of concept binding and order in vision-language models.

Compositionality in Language

  • Language composition research spans elementwise, neural, type-logical, and role-filler models; this work focuses on type-logical and role-filler approaches.The paper extends type-logical models to learn from images and text and handle a wider range of compositions.Source: §6 · p. 8

Compositionality in Language; Compositionality in Vision

  • The work compares CLIP image embeddings to a “bag of concepts” account, paralleling prior work showing sentence encoders can be approximated by a “bag of words” model.Concurrent work found Imagen can outperform frozen CLIP in some two-object binding cases.Source: §6 · pp. 8–9

Compositionality in Vision

  • Many compositional zero-shot vision benchmarks are restricted to adjective-noun settings, while CREPE, ARO, and Winoground study binding but may lack faithful, controlled conditions.The paper positions its synthetic setup as a controlled alternative without potential confounders from real-world images.Source: §6 · p. 9

7 Conclusion

The paper tests CLIP and its variants on compositional visual reasoning, finding strong performance on single adjective–noun compositions but difficulty when tasks require variable binding. Across models, binding-dependent performance is near chance at best, motivating attention to binding and order in VLM pretraining.

7 Conclusion

  • CLIP performs well on single adjective–noun compositions but struggles on tasks requiring variable binding.Some CDSMs also perform well on single adjective–noun compositions, while their two-object and relational performance is closer to chance.Source: §7 · p. 9
  • The findings suggest that VLM pretraining should account for binding and order to support compositional generalization.Source: §7 · p. 9

8 Limitations and Risk

The limitations concern the model set, synthetic data, and language scope; the risk discussion describes the investigation as limited in domain and based on abstract datasets.

8.1 Models

  • Only CLIP is evaluated as a major VLM, and results may differ for other VLMs.The compositional comparison covers key type-logical and role-filler models but omits some types, including Recursive Neural Networks.Source: §8.1 · p. 9

8.2 Datasets

  • The benchmark uses artificially generated shapes, so findings may differ with more realistic images.The authors note that dataset simplicity strengthens their findings.Source: §8.2 · p. 9

8.3 Language

  • The study is limited to English and uses simple label language; more complex image descriptions remain untested.Source: §8.3 · p. 9

8.4 Risk

  • The authors describe risk as limited because the datasets are abstract and the investigation domain is narrow.Previously existing artefacts were used within the limits of their original purpose.Source: §8.4 · p. 9

9 Ethical Considerations

The ethical discussion characterizes the abstract datasets and narrow modeling domain as presenting minimal ethical implications, while noting language and computational considerations.

9 Ethical Considerations

  • The authors consider ethical implications minimal given the abstract datasets, while noting that the proposed CDSM methods could apply to other languages.Source: §9 · p. 9
  • Fine-tuning uses a VLM with many parameters, but frozen image embeddings can be saved and used to train CDSMs, minimizing use of that model.Source: §9 · p. 9
  • The methods use English, although the CDSM methods could also be applied to other languages.Source: §9 · p. 9

A Training Details

Training uses pretrained CLIP image embeddings and either CLIP’s text encoder or a CDSM to generate phrase embeddings. The appendix also specifies training settings and an algorithm that aligns caption and image vectors.

A Training Details

  • Training ran for 20 epochs on one available RTX 3090, A40, or V100 GPU, taking about 6–20 minutes per epoch depending on dataset.Table 8 reports the number of trainable parameters for each experiment.Source: A Training Details · p. 13, Table 8 · p. 13
  • All models use pretrained CLIP ViT-L/14; CDSMs are trained with cross-entropy and Adam, while CLIP and CSP use their specified fine-tuning setups.Validation accuracy selects the final model for testing; other settings include batch size 32 and weight decay 10^-5.Source: A Training Details · p. 13

B Training Algorithm

  • The training algorithm aligns caption vectors with image vectors for adjective–noun phrases, with a similar procedure for relational phrases.Source: B Training Algorithm · p. 13
  • Algorithm 1 takes the training data, encoders, parameters, concept sets, weight decay, distractor count, and epochs as inputs, and outputs model parameters.The objective is entropy loss with weight decay.Source: B Training Algorithm · p. 13

C Calibrated Stacking

Calibrated stacking improves CSP’s single-object generalization, but the benefit does not extend to the two-object setting, and relational calibration is not applicable under the evaluated split. BLIP likewise performs well on single-object data but drops sharply on two-object and relational datasets.

C Calibrated Stacking

  • Calibration lowers cosine similarity scores for seen classes, with its coefficient selected on validation data to maximize the harmonic mean of seen- and unseen-class accuracy.The setup reserves 10% of the training data as seen-class validation data and trains on the remaining 90%.Source: C Calibrated Stacking · pp. 13–14
  • CSP with calibration improves generalization accuracy by 8 points and outperforms CLIP by 4 points on the single-object setting.The authors interpret the comparison as evidence that CSP learned to generalize to unseen adjective-noun compositions.Source: C Calibrated Stacking · p. 14, Table 10 · p. 14
  • Calibration does not improve two-object generalization because validation hard distractors are unseen, and it is omitted for relational data because the two generalization classes are equivalent.For the relational split, setting distractor probabilities to zero would yield 100% accuracy, so the authors defer that calibration experiment.Source: C Calibrated Stacking · p. 14

D Experiments with BLIP

  • The BLIP evaluation tests BLIP-Base and BLIP-Large on the concept-binding benchmark using CLIP’s evaluation procedure.BLIP uses unimodal image and text encoders alongside image-grounded text encoding and decoding.Source: D Experiments with BLIP · p. 14
  • BLIP performs well on single-object splits, but both BLIP model sizes drop dramatically on two-object and relational datasets, mirroring CLIP.The evaluation follows the procedure used for CLIP.Source: D Experiments with BLIP · p. 14, Table 9 · p. 14

Related papers

Loading 2212.10537v3…