PaperlayerView exact revision on arXiv

Source-linked AI summary

How can embedding models bind concepts?

Arnas Uselis, Darina Koishigarina, Seong Joon Oh

arXiv:2605.31503v1cs.CVcs.LG

TL;DR

CLIP recognizes individual concepts but struggles to bind them into objects in multi-object scenes, despite object information being recoverable within each modality. This paper analyzes the complexity of CLIP’s binding function and shows that controlled transformers can learn low-complexity, multiplicative bindings that generalize to unseen combinations.

  • Problem

    CLIP recognizes individual concepts but struggles to represent which concepts form which objects, despite object information being recoverable from unimodal embeddings.

  • Method

    The paper analyzes scene-embedding geometry and binding-function complexity, then trains controlled transformer dual encoders on synthetic multi-object data to test binding generalization.

  • Results

    CLIP’s binding is high-complexity and combination-specific, whereas controlled transformers learn low-complexity multiplicative interactions that generalize to unseen objects with sufficient data coverage.

  • Takeaways & Limitations

    Generalizable concept binding is possible when embedding models learn a low-complexity, systematic concept-to-object mapping that can be shared across modalities.

  • Takeaways & Limitations

    The analysis uses synthetic datasets, and its complexity notion depends on the chosen approximator family.

Abstract

from arXiv · show

Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP's binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization. Code is publicly available at https://github.com/oshapio/binding-concepts-complexity.

1. Introduction

The paper argues that CLIP fails at concept binding not because object information is absent, but because its high-complexity binding function does not generalize across modalities. Controlled transformer models show that sufficient data coverage can produce low-complexity binding functions that generalize to unseen objects.

  • The binding problem: Humans bind colors to shapes in multi-object scenes, whereas CLIP often recognizes individual concepts but represents their combinations incorrectly.CLIP behaves like a bag-of-concepts system in such settings.
  • What CLIP represents: CLIP scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover and manipulate object-level information.The additive geometry does not by itself explain whether concepts are correctly bound to objects.
  • Why CLIP fails: CLIP’s binding function is high-complexity, which prevents image and text encoders from learning a shared mechanism that generalizes to unseen concept combinations.The paper identifies binding-function complexity, rather than missing object structure, as the source of CLIP’s failures.
  • A path to generalization: Controlled transformer dual encoders trained on synthetic multi-object data develop binding generalization with sufficient data coverage.Generalizing models learn low-complexity binding functions that reuse structure across concept combinations and extend to unseen objects.

2. Related work

Related work frames binding as associating features within objects, documents CLIP’s stronger unimodal than cross-modal binding, and studies compositional generalization and representation geometry. This paper extends these lines to binding and generalization in multi-object scenes.

  • Binding: Binding research studies how models associate features belonging to the same object and has identified position-independent binding IDs in autoregressive language models.Surveys and empirical studies also examine binding limits and emergent symbolic mechanisms.
  • Binding in embedding models: CLIP exhibits unimodal binding but fails cross-modally, with proposed explanations including encoder weaknesses and over-reliance on concept-level information.Reported encoder weaknesses include difficulty with fine-grained details, negations, and spatial reasoning.
  • Compositional generalization: Compositional-generalization work either injects generalization into neural networks or studies the conditions under which models generalize to novel combinations of known parts.Existing literature largely focuses on single-object settings with factorized concept spaces.
  • Compositional generalization: For CLIP, concept recognition can generalize to novel combinations, but binding and object-level recognition remain difficult, especially beyond single-object settings.The paper instead studies binding in multi-object scenes.
  • Geometry of learned representations: Representation-geometry studies report linearity, disentanglement, and additive decomposition across language and vision models, including CLIP encoders.The cited analyses characterize the shape of learned features and representations.

3. Formalisation of binding

The paper formalizes binding as recognizing both which concept values are present in a scene and which values belong together in objects. It distinguishes cross-modal and uni-modal binding and defines binding functions that map scenes to modality-specific embeddings.

  • Formal setup: Scenes are modeled as tuples of objects, with each object represented by one value for every concept in the Cartesian-product concept space.An object is o = (c1, ..., cC), and a scene contains one or more such objects, up to a maximum number Omax.
  • Binding ability: Binding requires both concept recognition and object recognition: identifying present concept values and determining which values belong together.Concept recognition captures individual-value presence, whereas object recognition tests whether the model identifies the combinations forming present objects.
  • Modalities: Cross-modal binding evaluates recognition directly through the dual encoder, whereas uni-modal binding evaluates whether a trained probe can recover binding within one modality.The framework studies independently encoded scene inputs and queries matched by cosine similarity.
  • Binding functions: Binding functions Bimg : S → Rd and Btxt : S → Rd map scenes to image- and text-modality embeddings.Cross-modal binding requires these functions to produce compatible embeddings for the same scenes, including novel scenes.

4. Geometry of embedding models’s binding

CLIP scene embeddings decompose additively into object components, which explain most variance, support counterfactual object edits, and make object and concept information recoverable by unimodal probes. Object components jointly encode concept combinations, explaining why probes observe binding despite cross-modal misalignment.

  • 4.1 Geometry: CLIP scene embeddings are well approximated by sums of object components, supporting an additive object-level decomposition.This decomposition also extends to three-object CLEVR scenes, including occlusions, and to natural images generated by Gemini Nano Banana 2.
  • 4.1 Geometry: Replacing one object component with another produces embeddings that behave like intended counterfactual scenes under probing and retrieval.CLIP achieves 1.00 retrieval on CLEVR with scene-averaged object embeddings, while single-object embeddings achieve 0.97 retrieval on CLEVR and CLEVR-2D.
  • 4.1 Geometry: Bag-of-concepts representations explain up to 84% of text-encoder variance and 71% of image-encoder variance, while bags of objects explain roughly 10% more.Thus, concepts capture most variance, but object representations contribute additional structure beyond concepts alone.
  • 4.2 Component attribution: For CLIP-B/32, subtracting concept components reduces concept decoding from 1.00 to 0.06 for text and 0.94 to 0.05 for image while largely preserving object decoding.Subtracting object components instead collapses both object and concept decoding, showing that corresponding components drive recognition.

5. Binding generalization

Pretrained embedding models fail to generalize binding to unseen concept combinations, whereas controlled transformers trained from scratch can achieve near-perfect held-out object recognition with sufficient coverage. This generalization corresponds to a low-complexity, multiplicative binding structure in the embedding space.

  • 5.1 Binding complexity in pretrained models: For pretrained embedding models, even high-capacity MLPs fail to generalize binding to unseen concept combinations.Concept recognition reaches ≥80% when data fraction exceeds 0.3, but object recognition remains substantially lower even at large widths and high coverage.
  • 5.2 Binding generalization: Controlled dual-encoder transformers vary object-space coverage from 0.1 to 0.9 and test exclusively on held-out objects.Scenes contain up to two objects, each defined by C concepts with V values, yielding |O| = V^C possible objects.
  • 5.2 Binding generalization: Object recognition rises sharply with coverage, surpassing the bag-of-concepts baseline and approaching perfect accuracy despite entirely unseen test objects.Concept recognition generalizes earlier, while binding requires substantially more coverage; larger object spaces require a smaller fraction of observed objects for reliable recognition.
  • 5.3 Binding-function complexity: In generalizing transformer models, even small nonlinear MLPs approximate the binding map well, indicating low computational complexity.Performance gaps widen as the object space grows with larger C and V.
  • 5.4 Functional form of binding: The additive bag-of-concepts probe performs well for concepts but fails for objects, whereas multiplicative interactions recover binding generalization.The Global product probe performs best, especially in larger object spaces.
  • 5.4 Functional form of binding: Generalizable binding corresponds to multiplicative structure: simple multiplicative composition systematically combines concept representations and supports unseen combinations.This multiplicative form remains sufficient to capture concept combination even though its correlation weakens slightly at |O| = 125k.

6. Discussion … B.4. Optimization and controlled scenarios

The paper formalizes concept binding as mapping concepts to objects in scene embeddings: embeddings decompose additively, but CLIP’s high-complexity mappings hinder cross-modal generalization. Controlled transformer models instead learn multiplicative, low-complexity binding when trained with sufficient coverage.

  • 6. Discussion: Scene embeddings exhibit additive object-level structure, supporting uni-modal object decodability and direct embedding edits despite brittle cross-modal alignment.The concept-to-object mapping remains high-complexity and combination-specific, blocking generalization.
  • 6. Discussion: The analysis relies on synthetic datasets because no real-world dataset currently provides the required controlled, combinatorially complete structure.The paper also notes that complexity depends on the chosen approximator family because Kolmogorov complexity is not computable in practice.
  • Appendix: Controlled models are trained from scratch with separate scene and query transformers using a CLIP-like retrieval objective over procedurally generated multi-object scenes.The setup varies object-space size and scene complexity through controllable generation parameters.
  • B.1. Scene space and tokenization: Each object is a C-tuple of discrete concept values, and scenes are ordered lists of 1 to Omax independently sampled objects.Train/test splits can restrict sampling to allowed object lists.
  • B.1. Scene space and tokenization: Objects are tokenized with concept/value IDs plus SOO and EOO markers, while scenes concatenate objects, append EOS, and pad to Lmax = (C + 2)Omax + 1.The token mapping is t(c, v) = (c −1)V + v.
  • B.2. Encoders: Independent scene and probe encoders share the same transformer architecture and map padded token sequences to single d_out-dimensional embeddings.Each encoder uses token embeddings, positional encoding, and an L-layer Transformer with H attention heads while masking PAD tokens.
  • B.3. Queries, labels, and loss: Concept queries test single-attribute retrieval, whereas object queries test multi-attribute binding using unique batch objects and hard negatives such as attribute swaps.When both query types are enabled, their losses are summed and backpropagated through both encoders and the temperature parameter.
  • B.4. Optimization and controlled scenarios: Training uses AdamW with 10% linear warmup followed by linear decay, optionally taking multiple optimizer steps per sampled minibatch.Default hyperparameters and numerical-stability choices are listed in Table 9.

C. Experimental details · C.1. Dataset details

The experiments use three two-object real or synthetic image datasets, including the introduced CLEVR-2D adaptation. These datasets vary object concepts such as color, shape, or species and provide paired scene captions.

  • C.1. Dataset details: The study uses PUG:SPARE, CLEVR, and the newly introduced CLEVR-2D dataset.CLEVR-2D adapts CLEVR by replacing 3D shapes with flat 2D counterparts.
  • C.1. Dataset details: CLEVR-2D contains 576 two-object scenes varying across 8 colors and 3 shapes.Its shape substitutions are cube → square, sphere → circle, and cylinder → triangle.
  • C.1. Dataset details: Each CLEVR-2D scene contains two objects, and paired captions follow the same template as CLEVR.The caption format describes the two objects and their concepts.
  • C.1. Dataset details: 7392 scenes comprise PUG:SPARE, with two photorealistic objects varying across 12 colors and 8 animals.The dataset varies color and species in two-object scenes.
  • C.1. Dataset details: 576 scenes comprise CLEVR, with two 3D-rendered objects varying across 8 colors and 3 shapes.CLEVR provides the original synthetic 3D scene structure.
  • C.1. Dataset details: Across the synthetic datasets, the shared concept structure is 8 colors × 3 shapes and 576 scenes.CLEVR-2D preserves CLEVR’s concept structure while changing the visual shape representations.

C.2. CLIP geometry experiments … D.1. Interventions

The paper evaluates CLIP’s scene geometry through probing, retrieval, reconstruction, binding-function approximation, and component interventions. These experiments use controlled synthetic data, over-parameterized MLPs, held-out object combinations, and permuted-ID controls to test whether embeddings preserve concept and object binding.

  • C.2. CLIP geometry experiments: Frozen linear probes measure concept and object recognition from image or text embeddings, using multi-hot targets for scenes containing multiple objects.Concept probes predict values for each concept, while the object probe predicts concept tuples.
  • C.2. CLIP geometry experiments: Retrieval uses nearest-neighbor top-1 accuracy against intended scenes, treating any positional variant with the correct object tuple as correct.Predicted embeddings can come from object-mean reconstructions or interventions.
  • C.2. CLIP geometry experiments: The R2 reconstruction score measures how much embedding variance is explained by adding two object embeddings across all two-object scenes.It evaluates how well V2 vectors reconstruct the V4 scene embeddings.
  • C.3. Text encoder decomposition: The text decomposition experiments generate 10,000 two-object scenes and 100 single-object scenes from adjective and noun concept spaces.Each object is represented by a pair of integer indices, and captions linearize two objects as “a ⟨attr⟩⟨obj⟩and a ⟨attr⟩⟨obj⟩”.
  • C.4. Details on approximating pre-trained models’ binding function: Binding-function approximators use ReLU MLPs over concatenated one-hot concept indices, with hidden widths from 64 to 4096 and output dimension equal to encoder dimension.The architectures are deliberately broad to distinguish binding-function complexity from insufficient model capacity.
  • D. Additional results: Training varies object coverage from 10% to 90%, while test scenes contain only objects whose concept configurations were never seen during training.Models use Adam, learning rates in {10^-2, 10^-3, 10^-4}, batch size 4096, and 10,000 steps; evaluation probes predicted embeddings for functional binding.
  • D.1. Interventions: Interventions subtract estimated concept or object components and evaluate the resulting embeddings with the same probing and retrieval metrics against counterfactual target scenes.The intervention experiments test how concept and object components contribute to recognition in scene embeddings.
  • D.1. Interventions: Permuted-ID controls randomly remap concept values or objects before forming residual embeddings, testing whether ablation effects exceed those of subtracting arbitrary same-sized signals.The permuted residuals are evaluated with the same probes for image encoders and random-init counterparts.

D.2. Editing scene embeddings with object embeddings … CLEVR-2D PUG:SPARE

Object embeddings can be reused through arithmetic to edit counterfactual scenes, although shared concepts cause interference and reduce performance. Additive decomposition extends to three-object and more naturalistic scenes, while MDS projections reveal approximate compositional geometry across datasets and models.

  • D.2. Editing scene embeddings with object embeddings: Object-replacement interventions generally produce counterfactual scene embeddings supporting intended object probing and retrieval, with AVG and AVG+POS performing similarly.The experiments test whether object embeddings transfer across scenes through simple arithmetic; results are reported at k = 1.0.
  • D.2. Editing scene embeddings with object embeddings: Shared-concept scenes substantially reduce performance, especially retrieval, because overlapping representation directions interfere with selective object removal and insertion.The effect is consistent across datasets and encoders.
  • D.2. Editing scene embeddings with object embeddings: Performance typically improves as intervention strength k increases before saturating, whereas shared-concept scenes often work best with smaller k that partially mitigates interference.k = 0 leaves the embedding unchanged, and larger k increases the replacement contribution.
  • D.3. Three-object scenes and occlusions: The additive decomposition extends to three-object CLEVR scenes: probing accuracy remains 0.91–0.93 with occlusions, while object-based reconstruction exceeds concept-based reconstruction.Occlusion lowers retrieval from 0.76 →0.65, but probing remains high.
  • D.4. Extension to natural images: On 625 generated naturalistic images spanning 5 object types and 5 patterns, decomposition and editing retrieval results remain above chance despite greater visual complexity.The images vary in size, shape, color, and pattern realization; larger samples may improve object-embedding estimates.
  • D.5. MDS visualization: MDS projections show a partial grid-like structure reflecting color and shape composition, strongest for text embeddings and more distorted for CLIP image embeddings.DINOv2 separates objects with different shapes while keeping similarly colored objects relatively close.
  • CLEVR-2D PUG:SPARE: Figure 18 visualizes an approximate additive structure in CLIP and DINOv2 embeddings across CLEVR, CLEVR-2D, and PUG.Object and concept embeddings are estimated by averaging scenes containing the corresponding components, and two-object scenes are represented by concatenated labels.

D.6. Robustness of the binding-complexity diagnostic to the approximator family · D.7. Multiplicative probe applied to CLIP and DINOv2

The binding-complexity diagnosis is robust across MLP, XGBoost, and Random Forest approximators: concept recognition remains high, but object-level binding fails. A multiplicative probe likewise recovers concepts with more data yet cannot capture CLIP’s or DINOv2’s binding structure, leaving object recognition near zero.

  • D.6. Robustness of the binding-complexity diagnostic to the approximator family: The MLP choice is motivated by the expectation that SGD favors simple, compressible, compositional solutions, which should reveal a simple compositional binding function if one exists.This motivates testing whether the observed conclusion depends on the approximator family.
  • D.6. Robustness of the binding-complexity diagnostic to the approximator family: XGBoost and Random Forest were fit with the same scene-embedding prediction setup and hyperparameter sweeps used to test robustness across approximator families.The reported result uses the best accuracy across configurations.
  • D.6. Robustness of the binding-complexity diagnostic to the approximator family: Across MLP, XGBoost, and Random Forest approximators, concept recognition is high while object recognition stays near zero on held-out objects.This indicates that the bottleneck is binding structure rather than approximator choice.
  • D.6. Robustness of the binding-complexity diagnostic to the approximator family: All three approximator families predict individual concepts successfully but fail at object-level binding, preserving the qualitative conclusion that CLIP does not generalize to unseen objects.MLPs perform slightly better than XGBoost and Random Forest, although the search was not exhaustive.
  • D.7. Multiplicative probe applied to CLIP and DINOv2: For CLIP and DINOv2, concept recognition recovers with more training data, but object recognition remains near zero across encoders under the multiplicative Global product probe.Neither encoder admits a multiplicative binding function, unlike the from-scratch models.
  • D.7. Multiplicative probe applied to CLIP and DINOv2: The multiplicative probe cannot capture CLIP’s or DINOv2’s binding functions, whose complexity is consistent with their cross-modal concept-binding failures.The figure contrasts these encoders with from-scratch pixel-trained models, where multiplicative structure holds.

E. Vision encoders trained from scratch

From-scratch vision encoders reproduce the paper’s binding findings when trained on pixel inputs rather than multi-hot sequences. Across increasing visual complexity, they learn low-complexity binding functions and multiplicative structure that supports held-out object recognition.

  • Setup: The controlled pipeline replaces the multi-hot scene input with a convolutional front-end and an eight-layer transformer while retaining the original training, evaluation, and binding analyses.Each object combines square color and border color, with V = 50 values and up to 6.5×10^6 object combinations.
  • Results: The pixel-input experiments replicate the two main findings from the multi-hot setting.Binding maps are well approximated by shallow MLPs, and the multiplicative Global product probe recovers object recognition on held-out objects.
  • Results: The Global product probe recovers object recognition on held-out objects despite speckled noise and overlapping objects.This contrasts with pretrained CLIP and DINOv2 encoders, where the same probe fails.
  • Results: Even small MLPs approximate the binding function across three levels of visual complexity, confirming that from-scratch vision encoders learn low-complexity binding functions.The levels range from noise-free, non-overlapping objects to speckled noise and noisy, overlapping objects.
Loading 2605.31503v1…