PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models

Arnas Uselis, Andrea Dittadi, Seong Joon Oh

arXiv:2602.24264v2cs.CVcs.LG

TL;DR

Vision models must generalize to unseen combinations of familiar concepts, but the representational structure required for this remains unclear. The paper formalizes compositional generalization through three desiderata and shows that it requires linear factorization with cross-concept orthogonality, a pattern partially observed in modern vision models and correlated with generalization.

  • Problem

    Vision systems are expected to recognize novel recombinations of familiar concepts despite training on only a small, biased subset of the combinatorial space.

    Source: §1 · p. 1

  • Method

    The paper formalizes compositional generalization through divisibility, transferability, and stability to derive necessary representational geometry and dimension constraints.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Representations are theoretically forced into linear factorization with cross-concept orthogonality, while modern vision and self-supervised models partially exhibit this structure, whose degree correlates with unseen-combination accuracy.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results provide a concrete geometric target for compositional generalization and a representation-space diagnostic for assessing compositional capability.

    Source: §1 · p. 2, §6 · p. 9

    The theory assumes a fixed encoder retrained across different supports, unlike practical training that typically uses one dataset and one encoder.

    Source: §6 · p. 9

Abstract

from arXiv · show

Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Although modern models are trained on massive datasets, they still cover only a tiny fraction of the combinatorial space of possible inputs, raising the question of what structure representations must have to support generalization to unseen combinations. We formalize three desiderata for compositional generalization under standard training (divisibility, transferability, stability) and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. This provides theoretical grounding for the Linear Representation Hypothesis: the linear structure widely observed in neural representations is a necessary consequence of compositional generalization. We further derive dimension bounds linking the number of composable concepts to the embedding geometry. Empirically, we evaluate these predictions across modern vision models (CLIP, SigLIP, DINO) and find that representations exhibit partial linear factorization with low-rank, near-orthogonal per-concept factors, and that the degree of this structure correlates with compositional generalization on unseen combinations. As models continue to scale, these conditions predict the representational geometry they may converge to. Code is available at https://github.com/oshapio/necessary-compositionality.

1. Introduction

The introduction frames compositional generalization as reliable performance on unseen recombinations of familiar visual concepts, despite models’ frequent failures under train/test composition mismatch. It proposes divisibility, transferability, and stability, arguing that under standard linear prediction these desiderata require linearly factorized, near-orthogonal representations.

Motivation

  • Vision models often struggle when test images contain unusual combinations of familiar concepts, despite strong zero-shot transfer on many tasks.Training data cover common compositions, while models must answer the same queries on rare or unseen combinations.Source: §1 · p. 1

Desiderata

  • The paper formalizes compositional generalization through three desiderata: divisibility, transferability, and stability.They require accessible parts, transfer of readouts from small diverse subsets, and robust generalization from any valid subset.Source: §1 · pp. 1–2

Structural necessity

  • Under standard linear prediction, these desiderata necessitate linear factorization with near-orthogonal concept directions.The scope includes CLIP-style zero-shot classifiers, linear probing, and fixed nonlinear heads folded into the encoder.Source: §1 · p. 2
  • Under gradient descent with cross-entropy, embeddings decompose into per-concept sums with orthogonal difference directions.This provides theoretical grounding for the Linear Representation Hypothesis as a consequence of compositional generalization.Source: §1 · p. 2

Empirical grounding

  • Across CLIP and SigLIP families, representations show factorization, near-orthogonality, low-rank per-concept geometry, and correlation with compositional generalization accuracy.The findings suggest current state-of-the-art vision models are converging toward the necessary geometric properties.Source: §1 · p. 2

2. Related work

Prior work identifies sufficient conditions, observed geometric structure, and proposed disentanglement schemes for compositional generalization; this paper instead derives necessary embedding conditions under practice-driven desiderata and standard training.

Compositional generalization

  • The paper contrasts prior sufficient-condition analyses with its question of what embeddings must satisfy for transfer from restricted training subsets to the full compositional space.Prior work studies sufficient conditions under assumptions on data-generating processes or representations, whereas this work studies necessary conditions.Source: §2 · p. 2

Geometry of learned representations

  • Although prior studies report linear subspaces, linear representations, near-orthogonal encodings, and Minkowski-sum structure, this paper argues that linearity and orthogonality are necessary under standard training.The related work notes that geometric structure may arise from architecture as well as downstream demands, while this paper derives necessity from practice-driven desiderata.Source: §2 · p. 2

Disentangled and object-centric representations

  • Disentangled and object-centric work proposes desiderata and training schemes, but evidence linking them to compositional generalization is mixed; this paper asks what successful compositional generalization requires of embeddings.The complementary question is discussed further in Section A (Dittadi et al., 2022).Source: §2 · p. 2

3. Setup: A framework for compositionality

The framework models inputs as tuples in a concept space and formalizes compositional generalization through embedding models, valid training supports, and three desiderata: divisibility, transferability, and stability. It then instantiates these requirements for CLIP-like probe-based classification to study constraints on image-representation geometry.

3.1. Setup: concept spaces and data collection process

  • Compositionality is defined over a Cartesian concept space C, with valid training supports specifying which concept combinations are observable and collectible.With k concepts of n values each, C = [n]^k and |C| = n^k; each tuple c corresponds to an input x_c.Source: §3.1 · p. 3

3.2. Compositional representations and models

  • The framework studies non-contextual embedding encoders, with compositionality evaluated through readouts that are linear in the representation.The model scope includes CLIP and SigLIP (Zhai et al., 2023), supervised models, and DINO (Siméoni et al., 2025).Source: §3.2 · p. 3

3.3. Compositional generalization and desiderata

  • Compositional generalization requires divisibility, transferability to the full concept grid from every valid support, and stability of per-concept predictions across valid retrainings.Stability is treated as an idealization that may hold only approximately when supports are sufficiently informative.Source: §3.3 · p. 4
  • The complete setup is Π = (f, H, A, T), which exhibits compositional generalization when all valid supports satisfy the three desiderata on the full grid.Linear compositional generalization is the special case in which the readout hypothesis class is linear.Source: §3.3 · p. 4

3.4. Instantiating the framework with CLIP

  • In CLIP-like dual encoders, text prompts supply shared linear probe vectors, so the central question is what image-representation geometry enables compositional classification across all concept combinations.Image and text embeddings are typically ℓ2-normalized, making their inner products cosine similarities.Source: §3.4 · pp. 4–5

4. Implications of compositionality on representations

Under gradient descent with cross-entropy, stable compositional generalization requires—and in the binary regime is guaranteed by—linear factorization with orthogonal factors across concepts. Supporting all combinations requires embedding dimension at least equal to the number of concepts, while fixed-dimensional spaces force increasingly low-rank factors as concepts grow.

4.1. Geometry of f under common training settings

  • The representation decomposes a combination space of nk elements into n · k factors, with orthogonality required across concepts but not among values within one concept.Synthetic classification experiments find that linearity and orthogonality emerge, especially as the number of concepts increases.Source: §4.1 · pp. 5–6
  • Stable compositional generalization under gradient descent with cross-entropy requires linear factorization and orthogonality between factors from different concepts.These conditions hold under both near-complete random sampling and a minimal anchor-plus-one-change dataset, and are also sufficient in the binary regime.Source: §4.1 · pp. 5–6

4.2. Packing and minimum dimension

  • Linear probes that classify all nk combinations necessarily require embedding dimension d ≥ k, independent of the number n of values per concept.The lower bound applies to discrete and continuous factors when distinct values remain distinguishable.Source: §4.2 · p. 6
  • As the number of concepts grows at fixed dimension, per-concept subspaces become increasingly low-rank and approach near-collinearity.Empirically, Euclidean cross-entropy approaches the lower bound most closely, binary cross-entropy typically requires approximately 2k dimensions, and spherical geometry adds roughly one dimension.Source: §4.2 · p. 6

5. Surveying necessary conditions in pretrained models

Across CLIP, SigLIP, and DINO models, embeddings partially satisfy the predicted linear-factorization and cross-concept-orthogonality conditions. Greater linear factorization correlates with better compositional generalization, while factor directions are low-rank and more orthogonal than in randomly initialized encoders.

5. Surveying necessary conditions in pretrained models

  • DINOv3 embeddings on dSprites exhibit near-constant concept directions, grid-like additive structure, low-rank factors typically at most 3D, and near-orthogonality across concepts.The broader survey evaluates CLIP, SigLIP, and DINO families on PUG-Animal, dSprites, MPI3D, and ImageNet-AO using a multivalued additive extension of the binary theory.Source: §5 · pp. 6–7

5.1. Linear factorization in pre-trained models

  • Across evaluated pretrained models and datasets, Projected R2 is consistently above random baselines at 0.4-0.6 versus 0.12-0.42, indicating partial but non-perfect linear factorization.The score measures whitened variance explained after projection onto the probe span, with 1.0 denoting perfect linearity.Source: §5.1 · p. 8

5.2. Compositional generalization and linear factorization

  • Higher Projected R2 consistently coincides with higher compositional accuracy on unseen combinations across datasets and model families, including zero-shot settings and varied train fractions.The randomly initialized OpenCLIP ViT-L/14 baseline occupies the low-R2/low-accuracy corner, and the relationship persists on the full 32×32 dSprites grid.Source: Figure 8 · p. 8, §5.2 · p. 9

5.3. Orthogonality of factors

  • Pretrained encoders show substantially greater within-concept than cross-concept direction similarity, with values around 0.53-0.55 versus 0.09-0.12, compared with 0.32 across concepts for random initialization.This indicates partial cross-concept orthogonality, also observed in zero-shot text-probe experiments.Source: §5.3 · p. 9, Figure 10 · p. 9
  • Per-concept factors are typically low-dimensional, with ordinal and continuous concepts captured by ≤4 PCs, often ≤2, while discrete concepts have higher effective rank across model families.Factor geometry is similar across CLIP, OpenCLIP, SigLIP, and SigLIP2.Source: §5.3 · p. 9

6. Discussion and conclusion · Appendix · A. Extended related work

The paper argues that compositional generalization necessitates linearly factorized, cross-concept orthogonal representations with embedding dimension d ≥ k, while empirical structure correlates with unseen-combination performance. The extended related work positions this result as a necessary-condition account complementary to empirical, sufficient-condition, disentanglement, geometry, and training-effect studies.

6. Discussion and conclusion

  • The desiderata of divisibility, transferability, and stability force linear factorization, cross-concept orthogonality, and minimum embedding dimension d ≥ k.This reframes the Linear Representation Hypothesis as a consequence of compositional generalization rather than merely an empirical observation.Source: §6 · p. 9
  • Across vision-language and DINO models, partial factorization correlates with compositional generalization on unseen combinations, while R2–accuracy offers a representation-space diagnostic.The remaining gap from perfect factorization may help explain current benchmark failures and predicts geometry that models may approach as they scale.Source: §6 · p. 9
  • Future work should relax worst-case stability, characterize how training supports shape geometry, and assess the fixed-encoder assumption against single-dataset training.These directions could connect necessary conditions to practical data and model design while testing their applicability beyond retraining across supports.Source: §6 · p. 9

A. Extended related work

  • Prior compositional-generalization work studies training objectives, architectures, kernels, disentangled representations, data scaling, and CLIP’s limitations on unlikely attribute-object combinations.Examples include Compositional Feature Alignment, Compositional Risk Minimization, and CLIP analyses by Abbasi et al. (2024).Source: A. Extended related work · p. 16
  • Unlike prior sufficient-condition results for restricted model or data classes, this work derives necessary conditions without imposing a specific data-generating process or learned representation.The analysis asks what transfer from restricted supports to the full space implies under its stated desiderata.Source: A. Extended related work · p. 16
  • Where prior studies document linear, orthogonal, disentangled, or task-dependent embedding geometry, this work argues that linearity and orthogonality are necessary under practice-driven desiderata.Related evidence includes Trager et al. (2023), Jiang et al. (2024), Park et al. (2023), Engels et al. (2025), Abbasi et al. (2024), and Fel et al. (2025).Source: A. Extended related work · p. 16
  • Related work shows that data distributions, objectives, binding, explicit structure injection, and concept-identification methods can shape representational geometry and compositional behavior.These studies span concept frequency, BCE versus CE, contrastive objectives, hyperbolic embeddings, entailment learning, causal concept identification, and object-centric pipelines.Source: A. Extended related work · p. 16
  • Disentanglement and object-centric research proposes structured codes, metrics, and inductive biases for generalization, while this work asks what embeddings must contain when discriminative models generalize compositionally.Abbasi et al. (2024) report a correlation between CLIP disentanglement and compositional OOD accuracy, but their word-level decomposition assumption is not required here.Source: A. Extended related work · p. 17

B. Necessary conditions (proof of Proposition 1) · C. Sufficiency of linear factorization for compositional generalization · C.1. Binary valued-case: sufficiency of SVMs

This section establishes necessary and sufficient conditions for compositional generalization under the stated training-support regimes. It proves that gradient-based and SVM-based methods recover concept-specific linear geometry and transfer stably across full-grid and sufficiently structured partial datasets.

B. Necessary conditions (proof of Proposition 1)

  • Proposition 1 shows that valid compositional generalization requires z_c = Σ_i u_i,c_i and (u_i,1 − u_i,0) ⊥ (u_j,1 − u_j,0) for distinct concepts.The result applies to binary grids under either |T| = 2k−1 + 1 or anchor-plus-one-flip supports, assuming the three desiderata and linear heads.Source: B. Necessary conditions (proof of… · p. 20
  • The proof uses stability and SVM support-vector geometry to show each concept’s representation difference is independent of other concept variations, yielding linear factorization.Cross-datasets isolate one-concept flips, while invariance makes each concept predictor depend only on its own value; the resulting concept differences are orthogonal to the span of other concept differences.Source: B. Necessary conditions (proof of… · pp. 20, 22

C. Sufficiency of linear factorization for compositional generalization

  • The broader sufficiency discussion distinguishes the binary GD+CE case from multi-valued settings, where recoverable linear factors can support constructed readouts even without orthogonality or GD+CE.For general multi-valued concepts, GD+CE need not satisfy the desiderata automatically, although classifiers can be constructed in principle because the factors are recoverable.Source: C. Sufficiency of linear factorization… · p. 23
  • Linear factorization with cross-concept orthogonality is sufficient under binary GD+CE: supports with |T| = 2k−1 + 1 or a cross dataset of size 1 + k recover every concept value and remain stable across valid supports.Together with Proposition 1, this establishes a necessity–sufficiency characterization for the binary setting.Source: C. Sufficiency of linear factorization… · p. 23, §C.1 · p. 23

C.1. Binary valued-case: sufficiency of SVMs

  • Proposition 4 assumes binary representations that are linear sums of concept-specific factors with pairwise orthogonal concept differences, and considers both large random supports and cross datasets.The specified training sets are any |T| = 2k−1 + 1 support or an anchor with all k one-value flips, of size 1 + k.Source: §C.1 · p. 23
  • Gradient descent with cross-entropy satisfies transferability and stability across the entire binary concept grid.The proof uses convergence to the max-margin SVM solution in the binary-class case (Soudry et al., 2024).Source: §C.1 · p. 23
  • SVM weights align with individual concept differences because counterfactual pairs realize the shortest separation between the two class convex hulls.Orthogonality makes every convex-hull separation at least as large as the corresponding concept-factor difference, identifying the support vectors and exact weights.Source: §C.1 · pp. 23–25
  • The resulting classifier is correct for every concept assignment because other concepts’ contributions are absorbed into the bias.Centered factors sum to zero, while orthogonality removes cross-concept terms from the classification score.Source: §C.1 · p. 25

C.2. General case: linearly factored embeddings and sufficiency of recovering the factors

For linearly factored embeddings, factor recovery is unique up to concept-wise shifts, while centering removes this ambiguity. When the one-hot design matrix reaches rank 1 + k(n −1), recovered factors suffice to reconstruct embeddings and linearly read out concept values across all combinations.

C.2. General case: linearly factored embeddings and sufficiency of recovering the factors

  • Any two factor families generating the same embeddings differ only by concept-wise shift vectors whose sum is zero.Thus, the recovered factors generate the same embeddings despite this shift ambiguity.Source: §C.2 · p. 26
  • Centering each concept’s factors makes the factorization unique, so any recovered factorization matches the true centered factors.The concept-wise shifts cancel when subtracting each factor family’s within-concept mean.Source: §C.2 · p. 27
  • In the general case, recovery requires the one-hot design matrix induced by observed tuples to have sufficient rank, even when concept directions are not linearly independent.Once factors are recovered, the embeddings can be reconstructed and linear probes can recover concept values under linear separability.Source: §C.2 · p. 27
  • rank(A) = 1 + k(n −1) uniquely determines centered factors and enables reconstructing grid embeddings and fitting linear readouts for every combination.This sufficiency holds under the stated linear-separability assumption, with shift ambiguity handled by centering.Source: §C.2 · pp. 27–28

D. Packing and minimum dimension (proof of Proposition 2) · E. Examples of linearly compositional models and their geometries · E.1. Case 1: Ideal “LRH” concept classifier

The paper proves that linear probes realizing all combinations of k concepts require embedding dimension d ≥ k, with this bound tight, and illustrates ideal LRH geometries through orthogonal concept-factor grids and exact linear classification. These toy models clarify probe geometry without claiming full compositional generalization or learning stability.

D. Packing and minimum dimension (proof of Proposition 2)

  • Figure 13 illustrates the bound: three affine hyperplanes in d = 2 produce at most 7 regions, fewer than the 2^3 = 8 binary combinations.For non-binary concepts, restricting each concept to two values induces the same binary obstruction.Source: Figure 13 · p. 29, D. Packing and minimum dimension… · p. 29
  • Restricting every concept to two values yields 2^k binary combinations, whose k affine probe hyperplanes must create at least 2^k regions; when d < k, hyperplane arrangements cannot do so.The binary restriction proves the general n-valued lower bound, with the region-count argument formalized using Zaslavsky’s bounds.Source: D. Packing and minimum dimension… · pp. 29–30
  • For k concepts with n values, correctly classifying all n^k combinations using linear probes necessarily requires d ≥ k, and d = k is sufficient.The lower bound follows from hyperplane-region counting, while a standard-basis construction achieves all combinations at dimension k.Source: D. Packing and minimum dimension… · pp. 29–30

E. Examples of linearly compositional models and their geometries

  • The paper presents two toy geometries spanning ideal co-linear LRH factors and maximally dimensional representations with factors orthogonal to every other factor.These examples explain probe behavior and dimensionality but do not model the learning algorithm, stability, or full compositional generalization.Source: E. Examples of linearly compositional… · p. 31

E.1. Case 1: Ideal “LRH” concept classifier

  • The ideal LRH case assumes representations are strict sums of concept directions, with distinct scalar coefficients for each value and orthogonal directions that imply d ≥ k.The directions are treated as linearly independent for simplicity, and the coefficients are ordered within each concept.Source: §E.1 · p. 31
  • Under the stated assumptions, orthogonal concept directions make representations form a regular grid in z inside V, with each coordinate axis corresponding to one concept.Source: §E.1 · p. 31
  • Nearest-neighbor prototype probes correctly classify every point under orthogonality, whereas correctness is not guaranteed without it; Figure 14 visualizes the resulting decision regions for k = 2 and n = 30.The figure compares mildly non-orthogonal and closer-to-orthogonal concept directions, whose probe weights are near-parallel.Source: §E.1 · p. 32, Figure 14 · p. 32

E.2. Case 2: Ideal “on-off” concept classifier

The ideal “on-off” classifier assigns α to matching concept values and β to mismatches, but realizing this pattern requires at least 1 + k(n −1) dimensions. When distinguishability holds, the representations admit an additive linear factorization that preserves every probe response.

Classifier definition

  • The on-off classifier uses exactly two probe scores: α for a concept match and β for a mismatch, with α > β.This applies independently to every concept, value, and tuple.Source: §E.2 · p. 33, Figure 15 · p. 33

Minimal dimensionality

  • Any model realizing the on-off pattern requires at least 1 + k(n −1) embedding dimensions, and this bound is tight.The logit matrix has rank 1 + k(n −1), with one shared direction and n−1 independent directions per concept block; orthogonal constructions attain the bound.Source: §E.2 · pp. 33–35

Factorization conditions

  • Linear factorization holds exactly when embeddings have dimension 1 + k(n −1), and otherwise holds at least after projection onto the probe span.The construction uses averaged representations and centered concept factors, while avoiding the zero-global-mean normalization condition α + (n −1)β = 0.Source: §E.2 · pp. 35–36

Additive factorization

  • Under the on-off pattern, each representation can be reconstructed in additive form from a global mean and centered concept-value factors.The reconstruction is indistinguishable from the original representation to all probes, so it preserves the complete on-off response pattern.Source: §E.2 · pp. 35–36

F. Discussion on stability · G. Additional information · G.1. Testing linear factorization

Stability is necessary to prevent linearly separable representations from developing arbitrarily degenerate, non-factorized geometries across valid training supports. The paper measures linear factorization by projecting embeddings onto per-concept probe spans and evaluating whitened projected R2.

F. Discussion on stability

  • Stability fixes probe weights and decision boundaries across valid supports, forcing the representational structure established by the main results.Without it, retraining can produce different partitions and no robustness is guaranteed.Source: F. Discussion on stability · p. 37
  • Linear separability alone does not imply additive factorization, even in d = 2 and k = 2 as n →∞.Perfect classification can coexist with point clouds that admit no decomposition into per-concept components.Source: F. Discussion on stability · p. 37
  • Figure 16 constructs n2 convex cells for concept-value pairs and shows that ε-perturbed boundaries can create arbitrarily thin regions while preserving perfect separability.For n = 8 and n = 14, the arrangements yield 64 and 196 regions, respectively, while no linear factorization is possible.Source: Figure 16 · p. 37, F. Discussion on stability · p. 37
  • Robust compositional generalization therefore requires preventing degenerate configurations through exact Stability or an approximate alternative.Minimum-area constraints could address such cases in principle, but the framework avoids them directly through Stability.Source: F. Discussion on stability · p. 37

G. Additional information

  • The additional information section expands the treatment of linear factorization, including its non-triviality, whitening-based measurement, illustrative intuition, and counterexamples under perfect classification.These materials explain why linearly compositional models need not have representations decomposable into per-concept sums.Source: G. Additional information · p. 38

G.1. Testing linear factorization

  • To isolate conceptual structure, the method trains per-concept linear probes and projects embeddings onto the joint span of their probe vectors.For each concept i and value j, probe weights form W, from which the projection matrix P_W is computed.Source: §G.1 · p. 38
  • Projected R2 is computed after PCA-whitening the embeddings to unit covariance, preventing dominant directions from producing trivially high scores.The same whitening transform is applied to the projected data and reconstructions.Source: §G.1 · p. 38

G.2. Whitening in measuring linear factorization … H.4. Experiments using text encoders as probes (zero-shot results)

Across theoretical examples and supplementary experiments, whitening exposes misleading linearity scores, while pretrained vision representations show compositional structure through correlated held-out accuracy, orthogonal factors, and low-dimensional concept subspaces. The analyses also show that perfect linear classification alone does not guarantee linear factorization, and compare external probes with text-encoder zero-shot probes.

G.2. Whitening in measuring linear factorization

  • Whitening prevents dominant directions from inflating R2: illustrative scores fall from 0.813 to 0.509 and from 0.991 to 0.564 in non-ideal representations.Whitening applies PCA and unit-covariance normalization; it leaves R2 unchanged when representations are already linearly factored.Source: §G.2 · pp. 38–39

G.3. Intuition of linear factorization

  • R2 measures decomposition into per-concept components, with R2 = 1 indicating perfect factorization; examples reconstruct perfectly at R2 = 1 but not at R2 = 0.53.For two concepts with 24 values each, 576 combinations are generated from 48 component vectors in the perfectly linear case.Source: §G.3 · p. 39

G.4. Non-triviality of linear factorization of linearly compositional models

  • Perfect linear classification does not imply linear factorization: examples achieve 100% accuracy while R2 ranges from 0.32 to 0.83, or equals 0.53.Extra embedding-space degrees of freedom allow perturbations that preserve decision regions while violating additive per-concept structure.Source: §G.4 · pp. 39–40

G.5. Models and probing

  • Experiments span CLIP, OpenCLIP, MetaCLIP, MetaCLIP2, SigLIP, SigLIP 2, and DINOv1–v3 on PUG-Animal, dSprites, MPI3D, and ImageNet-AO.The model set covers contrastive vision-language and self-supervised vision encoders.Source: §G.5 · p. 42

H.1. Linearity and compositional generalization

  • Higher projected R2 consistently predicts higher held-out compositional accuracy across split regimes, although the relationship flattens near ceiling on dSprites and MPI3D.The analysis trains one linear probe per concept and evaluates average accuracy on unseen concept combinations.Source: §H.1 · pp. 42, 45

H.2. Orthogonality of factors

  • Across orthogonality matrices and aggregate summaries, across-concept factor similarities are consistently lower than within-concept similarities, matching the predicted orthogonality pattern.Orthogonality is measured using mean absolute cosine, where lower cross-concept values indicate greater orthogonality.Source: §H.2 · pp. 46–47, Figure 22 · p. 47, Figure 23 · p. 47

H.3. Dimensionality of factors

  • Recovered semantic factors are usually low-rank and cross-model similar: at least 95% of variance is typically captured by one or two principal components, while discrete concepts can be higher-rank.dSprites factors are effectively below three dimensions, with size and y-position near one-dimensional and orientation occupying a smooth two-dimensional curve.Source: §H.3 · p. 48, Figure 24 · p. 48, Figure 25 · p. 48

H.3.1. FULL PER-MODEL RESULTS

  • Full per-model dimensionality results report the minimum PCA or SVD components needed to explain 95% of recovered-factor variance across representative vision backbones and datasets.The complete plots cover dSprites, PUG-Animal, and MPI3D across models including OpenCLIP and MetaCLIP variants.Source: §H.3.1 · p. 49, Figure 26 · p. 50

H.4.1. EXPERIMENTS ON PUG-ANIMAL

On PUG-Animal, text-encoder probes evaluate character, background, scale, and texture concepts. Higher representation linearity corresponds to higher full-dataset accuracy, while factors are more orthogonal across concepts than within them.

Setup

  • The evaluation exposes character, background, scale, and texture through descriptive prompt templates whose text embeddings are generated by the text encoder.For each concept value, the encoder produces an ℓ2-normalized probe vector w_i,j.Source: §H.4.1 · pp. 51, 53

Linearity of factors and generalization

  • Higher projected R2 generally coincides with higher average accuracy across all PUG-Animal concept combinations, while the random baseline has low projected R2 and accuracy.The comparison uses text-encoder probes across models and matches observations from the main experiments.Source: §H.4.1 · p. 53, Figure 28 · p. 53

Orthogonality of the factors

  • Across all evaluated models, factors are more orthogonal across concepts than within concepts, whereas the random baseline shows average cosine similarity around 0.5 in both cases.The evaluated concepts are character, world, size, and texture; the orthogonality pattern is shown in pairwise cosine-similarity heatmaps.Source: §H.4.1 · p. 53, Figure 29 · p. 53

Qualitative examples

  • For SigLIP2, high-R2 samples generally depict clean scenes where character, size, and texture are easier to discern than in low-R2 samples.The qualitative examples report size values 0.7 as small, 1.0 as medium, and 1.3 as large.Source: §H.4.1 · p. 53, Figure 30 · p. 54

H.4.2. EXPERIMENTS ON IMAGENET-AO … H.7. Empirical evaluation on synthetic data

Across ImageNet-AO and synthetic and controlled analyses, compositional structure is associated with linear factorization, cross-concept orthogonality, and improved generalization, while robustness checks qualify how these patterns emerge and are measured. Qualitative and residual analyses further show low-rank factor geometry and broadly distributed unexplained variance.

H.4.2. EXPERIMENTS ON IMAGENET-AO

  • Higher projected R2 coincides with higher full-dataset accuracy on sparse ImageNet-AO, while the random baseline remains below 0.1 projected R2.The analysis uses text-encoder probes and a linear system to recover noun and adjective factors because data are sparse and insufficient for trained linear probes.Source: §H.4.2 · pp. 55–56, Figure 32 · p. 56
  • Across ImageNet-AO models, factors are more orthogonal across concepts than within concepts, whereas the random baseline averages 0.49 cosine similarity in both comparisons.For CLIP ViT-L/14, within-concept noun and adjective similarities average 0.10 and 0.14, versus 0.07 across concepts.Source: §H.4.2 · p. 56, Figure 33 · p. 56

H.5.1. CORRELATION ACROSS TRAIN FRACTIONS (|S|/N ∈{0.001, . . . , 0.9})

  • The positive projected-R2–compositional-accuracy correlation persists across all tested train fractions, including the extreme low-data regime and the full 32×32 dSprites grid.The robustness analysis spans train fractions from 0.001 to 0.999 over three seeds.Source: §H.5.1 · p. 58, Figure 35 · p. 58, Figure 36 · p. 59

H.5.2. PROJECTED R2 AS A FUNCTION OF TRAIN FRACTION

  • Projected R2 increases with train fraction but plateaus, preserves relative model rankings, and shows its major jump only as the train fraction approaches N.This indicates that the main-text values are not trivially inflated by the selected data split.Source: §H.5.2 · p. 60, Figure 37 · p. 60

H.5.3. ORTHOGONALITY WITH ERROR BARS AND FULL DSPRITES GRID

  • The within-concept versus across-concept orthogonality gap remains consistent across models and datasets, including the complete dSprites grid with uncertainty estimates.Error bars are standard deviations across concept pairs, and pretrained models consistently show higher within-concept than across-concept similarity.Source: §H.5.3 · p. 60, Figure 38 · p. 60

H.5.4. SPECTRAL ANALYSIS OF RESIDUALS

  • Residual variance after subtracting the recovered linear factorization spreads across many components without dominant singular values across both tested models and all three datasets.This indicates that the unexplained variance is not concentrated in a low-dimensional structured subspace.Source: §H.5.4 · p. 60, Figure 39 · p. 61

H.6. Qualitative results

  • Qualitative visualizations across DINOv3, OpenCLIP, and MetaCLIP2-bigG show low-rank factor structure and near-orthogonality for dSprites and MPI3D, while PUG-Animal factors are higher-rank and shown only through top principal components.Pairwise projections are less informative for PUG-Animal because many concepts, including character and background, are discrete with many values.Source: §H.6 · p. 62, Figure 40 · p. 62, Figure 42 · p. 64

H.7. Empirical evaluation on synthetic data

  • The synthetic evaluation varies Euclidean versus spherical geometries, CE versus BCE losses, concept counts, value counts, and embedding dimensions, but reports properties of standard-loss compositional models rather than guarantees of generalization.Combinations are capped at 100,000, making reported R2, orthogonality, and dimension estimates approximate in larger regimes.Source: §H.7 · p. 65

H.7.1. RESULTS: LINEARITY.

In the from-scratch setting, embeddings exhibit substantial linearity across representation spaces and loss types, with most cases achieving linear-factorization R2 ≥0.7.

RESULTS: LINEARITY.

  • Most from-scratch cases achieve linear-factorization R2 ≥0.7.This result holds across the representation-space and loss-type settings evaluated in Figure 43.Source: §H.7.1 · p. 65, Figure 43 · p. 66
  • Figure 43 evaluates linear-factorization R2 while varying the representation space.Source: Figure 43 · p. 66
  • Figure 43 also evaluates linear-factorization R2 across different loss types.Source: Figure 43 · p. 66

H.7.2. RESULTS: ORTHOGONALITY. · H.7.3. RESULTS: DIMENSIONALITY.

The results support the theory’s geometric predictions: cross-concept factors become increasingly orthogonal as concept count grows, while the dimension required for accurate linear readout scales primarily with the number of concepts and depends on the training objective and geometry.

H.7.2. RESULTS: ORTHOGONALITY.

  • Cross-concept factor cosine similarity is consistently lower than within-concept similarity and decreases as k grows across losses and geometries.The pattern matches the theory’s prediction of orthogonal cross-concept factor differences without requiring orthogonal within-concept directions.Source: §H.7.2 · p. 66, Figure 44 · p. 67
  • Figure 44 reports within-concept cosine similarity above cross-concept similarity, with cross-concept similarity declining as the number of concepts increases.This provides the visual evidence for stronger cross-concept orthogonality in the from-scratch setting.Source: Figure 44 · p. 67

H.7.3. RESULTS: DIMENSIONALITY.

  • The dimensionality lower bound d≥k is independent of the number of values n and specifies what is necessary in principle, not what training must attain.Thus, the bound is a capacity result rather than a guarantee for any particular objective.Source: §H.7.3 · p. 67
  • The dimensionality experiments estimate the minimum d achieving ≥0.99 per-concept accuracy across varying k and n, using CE or BCE with Euclidean or spherical geometry.The comparison covers CLIP-like and SigLIP-like objectives and dot-product or cosine-normalized representations.Source: §H.7.3 · p. 67
  • CE in Euclidean space typically approaches the theoretical minimum d≈k, whereas BCE often requires roughly 2k and spherical geometry adds about one dimension.These results support concept count as the primary dimensional driver while showing that objective choice affects whether the lower bound is attainable.Source: §H.7.3 · p. 67, Figure 45 · p. 68
  • BCE generally needs larger dimension because each class is trained as an independent one-vs-rest problem, while CE separates classes through relative margins.The objective therefore better or worse matches the inference rule and affects attainability of the geometric capacity bound.Source: §H.7.3 · p. 67

Related papers

Loading 2602.24264v2…