PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models

Puru Vaish, Meelad Dashti, Christoph Brune, Pascal Mettes, Tatiana Tommasi, Nicola Strisciuglio

arXiv:2610.29811v1cs.CV

TL;DR

The paper asks whether OW-CZSL needs exhaustive pair recombination or can instead rely on scores from single concept primitives. The paper extends GDE across geometries and scores standalone tangent-space or intrinsic attribute and object primitives directly against test images. GDE primitives consistently outperform pair-based compositions for open-world CZSL, while intrinsic primitives are particularly effective in hyperbolic spaces.

  • Problem

    The paper asks whether OW-CZSL needs exhaustive pair recombination or can instead rely on scores from single concept primitives.

    Source: §3.2 · p. 4

  • Method

    The paper extends GDE across geometries and scores standalone tangent-space or intrinsic attribute and object primitives directly against test images.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    GDE primitives consistently outperform pair-based compositions for open-world CZSL, while intrinsic primitives are particularly effective in hyperbolic spaces.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Primitive representations emerge in pretrained models across Euclidean, spherical, and hyperbolic spaces and can support CZSL without explicit pair recomposition.

    Source: §6 · p. 8

    UT-Zappos50K is a consistent exception to the primitive-based advantages, in a narrow domain with strongly coupled attribute–object relationships.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Compositional zero-shot learning (CZSL) aims to recognize novel combinations of previously observed objects and attributes. Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization. The Geodesically Decomposable Embeddings (GDE) framework formalizes this by identifying directions in the tangent space at the Frechet mean of the embedding distribution, which encode primitive visual concepts. Unseen attribute-object compositions can be synthesized through vector addition of these primitives. We revisit the assumption that compositional generalization requires explicit attribute-object pair re-composition. By means of new compositional scoring functions and extensive evaluations, we show that directly using the decomposed primitives suffices for recognizing unseen compositions. In that, we gain insights on how the geometry of the embedding manifold (Euclidean, spherical, or hyperbolic) and dataset structure affect compositional reasoning. Across models (CLIP, DINOv2, MERU, HyCoCLIP), we show that compositional structure consistently emerges as primitive representations, enabling inference on CZSL benchmarks (MIT-States, VAW-CZSL, UT-Zappos50K, Clothing16K) and group robustness on Waterbirds and CelebA. Code: https://github.com/nis-research/Compositionality-with-Primitives.

1. Introduction

The paper asks whether pretrained vision-language models can recognize unseen attribute–object compositions without explicitly rebuilding each pair. It studies primitive-based inference across manifold geometries and reports compositional and group-robustness benefits.

1. Introduction

  • The open question is whether GDE extends beyond spherical CLIP embeddings and whether unseen-pair recomposition is necessary for compositional generalization.Earlier GDE work recombined tangent-space primitives to synthesize unseen pairs.Source: §1 · p. 1
  • Primitive-based inference supports open-world generalization across Euclidean, spherical, and hyperbolic models without explicit attribute–object pair construction.The study evaluates tangent-space GDE and manifold-native intrinsic primitives.Source: §1 · p. 2
  • Compositional structure emerges across several CZSL benchmarks, with stronger evidence on datasets whose samples have higher semantic variance.The benchmarks include MIT-States, VAW-CZSL, Clothing16K, and UT-Zappos50K.Source: §1 · p. 2
  • Using object primitives alone substantially and consistently increases worst-group accuracy on Waterbirds and CelebA.The authors report that primitive-based reasoning can remove attribute-induced biases and spurious correlations.Source: §1 · p. 2

2. Related Work

Prior work explored compositional learning through shared spaces, prompt tuning, conditional representations, and GDE. This paper instead studies primitive structure in pretrained embeddings across geometries and directly scores individual primitives.

2. Related Work

  • Earlier CZSL approaches used shared embedding spaces, prompt tuning, or conditional attribute–object representations, while contrastive VLMs can behave like a “bag of words” on text composition.Benchmarks have also highlighted reliance on statistical shortcuts.Source: §2 · p. 2
  • Prior GDE factorized primitive directions at the embedding distribution’s Fréchet mean but focused on spherical embeddings and explicit pair construction [2].Its tangent-space primitives were recombined to form unseen compositions.Source: §2 · p. 2
  • This work scores independent object and attribute primitives directly, extending GDE across Euclidean, spherical, and hyperbolic latent spaces.The approach studies pretrained encoders without adding learnable components, unlike fine-tuning and conditional-learning methods [17] [22] [35] [36].Source: §2 · pp. 2–3
  • For hyperbolic models [5] [23] [27], the paper investigates how hierarchical geometry shapes compositional reasoning and reports a role for intrinsic primitives in group robustness.It reports that intrinsic primitives provide more semantically coherent representations than tangent-space approximations in hyperbolic settings.Source: §2 · p. 3
  • Applying GDE to DINOv2 shows compositional structure in a vision-only encoder’s Euclidean latent space, without language supervision.Talk2DINO [1] is described as aligning text embeddings to DINO patch features through a learned projection.Source: §2 · p. 3

3. Preliminaries

The preliminaries establish the manifold and tangent-space setup for GDE, then define open-world CZSL and pair-based scoring as the comparison point for primitive-based inference.

3. Preliminaries

  • The preliminaries introduce compositional embeddings, GDE, open-world CZSL, pair-based scoring, and the alignment of DINOv2 patch embeddings with CLIP text embeddings.The DINOv2–CLIP alignment is done through Talk2DINO.Source: §3 · p. 3

3.1. Geodesically Decomposable Embeddings

  • GDE computes a dataset Fréchet mean, maps embeddings into its tangent space with Logµ, and uses Expµ as the local inverse map.Concept primitives are weighted averages of projected tangent vectors associated with each attribute or object.Source: §3.1 · p. 3
  • The illustration contrasts standalone tangent-space primitive scoring and intrinsic concept means with pair-based tangent-sum and textual-pair alternatives.The examples use unseen Blue Squares and their attribute and object concepts.Source: Figure 2 · p. 4

3.2. Open-World CZSL and Pair-based Scoring

  • Open-world CZSL tests every attribute–object combination, and the section asks whether single-concept scores can replace exhaustive primitive-pair recombination.The conventional zero-shot baseline scores text prompts for the full candidate label space.Source: §3.2 · p. 4

4. Primitive-based CZSL

The section presents two ways to represent attributes and objects as standalone primitives and score test images against them without explicitly recomposing attribute-object pairs. Intrinsic primitives are computed on the manifold, while GDE primitives are tangent-space directions used for scoring in a shared tangent space.

4. Primitive-based CZSL

  • Both intrinsic and GDE approaches score test images directly against standalone attribute and object primitives, eliminating explicit primitive re-composition.The approaches operate in their native mathematical domains.Source: §4 · p. 4

4.1. Intrinsic Primitives via Fréchet Means

  • Intrinsic primitives are Fréchet means of samples sharing an attribute or object label, computed with Riemannian gradient descent on non-Euclidean manifolds and arithmetic means in Euclidean spaces.Separate primitives are computed for attributes and objects.Source: §4.1 · pp. 4–5
  • Intrinsic primitive similarity is computed on the underlying manifold with geometry-specific similarity functions, and the individual primitive scoring function does not rely on text-prompt embeddings.Source: §4.1 · p. 5

4.2. GDE Primitives on any Manifold

  • GDE uses logarithmic and exponential maps to extend tangent-space primitives across spherical, Euclidean, and Lorentz geometries; in Euclidean space, these reduce to subtraction and addition.Source: §4.2 · p. 5
  • GDE primitive directions and projected test-image representations share a tangent space, where attribute and object scores use cosine similarity and inference bypasses manifold projections.Source: §4.2 · p. 5

5. Experiments

Across four CZSL benchmarks and three embedding geometries, primitive-based inference generally improves open-world compositional recognition over explicit pair composition. The experiments also relate performance to dataset variance, test the separation of GDE primitives, and evaluate object primitives for group robustness.

5.1. Setup

  • The evaluation spans Euclidean, spherical, and hyperbolic models and compares pair prompts, primitive-level inference, and composed GDE pairs using HM and UPA.HM balances seen-pair and unseen-pair accuracy, while UPA measures generalization to novel compositions.Source: §5.1 · pp. 5–6

5.2. Results: CZSL via Primitives

  • UT-Zappos has substantially lower intrinsic variance than MIT-States, VAW-CZSL, and Clothing16K across geometries; whitening raises its variance toward their regime.The authors link the low-variance setting to weaker primitive decomposition and stronger pair-based exploitation of dataset-specific co-occurrence.Source: §5.2 · p. 6, Figure 3 · p. 6
  • Primitive-based inference consistently improves HM and UPA over GDE pair composition across models and datasets, particularly on unseen pairs.On MIT-States, GDE primitives raise HM by +1.53 to +3.66 across five models; UPA gains include +3.85 for MERU-B and +5.04 for HyCoCLIP-B.Source: §5.2 · p. 6, Table 1 · p. 7
  • After whitening, intrinsic primitives achieve the highest HM for every model and highest UPA for four of five models on UT-Zappos.Their remaining UPA shortfall is 0.05 points below the best MERU-B pair-based result.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3. Statistical Separation of GDE Primitives

  • Object-attribute primitive dependence is lower than dependence between either primitive and the full pair representation across datasets and model families.On MIT-States, primitive CORR is 0.444–0.473 versus 0.825–0.895 for object primitive and pair; MI is 13.0 versus 119.4–320.7.Source: §5.3 · p. 7

5.4. Group Robustness via Primitives

  • Object primitives improve worst-group accuracy over zero-shot object inference on Waterbirds and CelebA, largely preserving or improving average accuracy without retraining or group annotations.The gains are especially pronounced for hyperbolic models MERU and HyCoCLIP.Source: Table 3 · p. 8, §5.4 · p. 8

6. Conclusion

Primitive representations support compositional zero-shot inference without explicitly rebuilding attribute–object pairs, with GDE primitives robust across open-world CZSL and intrinsic primitives offering a geometry-aware complement. The appendix describes Fréchet-mean optimization and Talk2DINO feature routing, and reports that native Euclidean DINO embeddings outperform spherical ones.

6. Conclusion

  • GDE and intrinsic primitives support CZSL inference without explicit pair recomposition; GDE primitives outperform pair-based compositions in open-world CZSL, while intrinsic primitives are particularly effective in hyperbolic spaces.The conclusion also reports object-primitive inference benefits for group robustness without model retraining.Source: §6 · p. 8

A. Riemannian Gradient Descent

  • The Fréchet mean is optimized by averaging log-mapped tangent vectors as a gradient direction, then mapping updates back to the manifold with the exponential map.The procedure uses a step size and iterates over the embeddings.Source: A. Riemannian Gradient Descent · p. 11

B. Dense Feature Extraction via Talk2DINO

  • Talk2DINO uses a text query to route over DINO patch embeddings and produce one unified Euclidean image representation while preserving DINO’s native geometry.The query-conditioned routing uses a temperature parameter.Source: B. Dense Feature Extraction via… · p. 11

C. DINO Embeddings: Euclidean or Spherical?

D. GDE Primitive Concept Separation

GDE object and attribute primitives are comparatively disentangled from each other, while each retains stronger association with pairwise structure. Non-Euclidean whitening uses tangent-space operations, with spherical whitening constrained by the sphere’s finite cut locus.

D. GDE Primitive Concept Separation

  • Object–attribute primitive pairs have the lowest correlation and mutual information, indicating greater separation than object–pair or attribute–pair representations.Object–pair estimates exceed attribute–pair estimates across datasets and models, suggesting embeddings align more with object-level than attribute-level information.Source: D. GDE Primitive Concept Separation · pp. 11–12

E. Non-Euclidean Whitening

  • For non-Euclidean features, whitening is performed in a tangent space after computing the intrinsic mean; spherical whitening maps embeddings there, whitens, then maps them back.On the sphere, rescaling keeps tangent-vector norms below π to avoid exponential-map wrapping that may destroy semantic information.Source: E. Non-Euclidean Whitening · p. 12, §E.1 · p. 12

E.2. Hyperboloid Whitening

Hyperboloid whitening first moves the intrinsic mean to the canonical origin with a Lorentz isometry, then applies tangent-space whitening and maps the vectors back. Unlike spherical space, hyperbolic space has no finite cut locus.

E.2. Hyperboloid Whitening

  • Hyperbolic whitening maps the intrinsic mean to the canonical origin with a distance-preserving Lorentz isometry, whitens tangent coordinates, then maps them back; unlike the sphere, hyperbolic space has no finite cut locus.At the origin, tangent coordinates represent the local Riemannian metric as an ordinary Euclidean representation.Source: §E.2 · pp. 12–13

F. Primitive Weighting Ablation

The ablation reports minor performance variations when noise estimation is disabled, consistent with GDE weights carrying confidence rather than directional information.

F. Primitive Weighting Ablation

  • Disabling noise estimation causes only minor GDE performance variations across models, validating that scalar weights carry confidence but no directional information.The weighting design is inherited from the baseline GDE framework.Source: F. Primitive Weighting Ablation · p. 13

G. Extended Open-World CZSL Metrics

The extended open-world evaluation reports primitive-level object and attribute accuracy alongside pair performance. Primitive-based inference generally improves recognition of both underlying semantic factors, particularly on datasets with greater semantic diversity.

G. Extended Open-World CZSL Metrics

  • Object-level and attribute-level accuracies supplement harmonic mean and composed-pair performance in the open-world evaluation.These additional metrics provide a more detailed view of compositional behavior.Source: G. Extended Open-World CZSL Metrics · p. 13
  • Object and attribute accuracies isolate recognition of underlying semantic primitives, distinguishing primitive identification from compositional reasoning.Source: G. Extended Open-World CZSL Metrics · p. 13
  • Primitive-based inference generally improves object and attribute recognition, particularly on semantically diverse MIT-States and VAW-CZSL.The reported gains extend beyond pair retrieval to separability of the underlying semantic factors.Source: G. Extended Open-World CZSL Metrics · p. 13

H. Hierarchy of Hyperbolic Intrinsic Primitives

In hyperbolic models, attribute-object compositions occupy larger radii than their constituent primitives, while primitive radius ordering varies by dataset. The analysis associates intrinsic primitives with denser representation regions and improved robustness under distribution shifts.

H. Hierarchy of Hyperbolic Intrinsic Primitives

  • Attribute-object compositions occupy larger radii than constituent primitives, consistent with compositions being more specific in the hyperbolic hierarchy.The analysis compares attribute, object, and composition radius distributions for HyCoCLIP and MERU.Source: H. Hierarchy of Hyperbolic Intrinsic… · p. 13
  • Attributes lie closer to the origin than objects in MIT-States, VAW-CZSL, UT-Zappos, and Clothing16K, but occupy larger radii in Waterbirds and CelebA.The authors suggest that attributes such as backgrounds or facial characteristics add complexity and may dominate semantic structure in the latter datasets.Source: H. Hierarchy of Hyperbolic Intrinsic… · p. 13
  • Intrinsic primitive embeddings occupy denser, better-covered regions, while compositions correspond to deeper, sparser concepts.The authors report that intrinsic primitive means remain closer to the training distribution than locally estimated GDE primitives, supporting more stable compositional representations and improved robustness under distribution shifts.Source: H. Hierarchy of Hyperbolic Intrinsic… · pp. 13–14

I. Example Images for Datasets

Test-split examples in Figure 6 illustrate the diversity of attribute-object compositions across datasets. The datasets differ in visual and semantic variation, with Clothing16K broader and UT-Zappos more constrained.

I. Example Images for Datasets

  • Figure 6 shows random attribute-object examples drawn from each dataset’s test split.The samples illustrate the diversity of compositions across domains.Source: §I · p. 14, Figure 6 · p. 17
  • Most datasets show substantial variation in attributes and objects, reflecting the task’s combinatorial structure.Source: §I · p. 14
  • Clothing16K spans varied garment categories, textures, appearances, poses, and viewpoints, creating a more diverse and challenging generalization setting.Source: §I · p. 15
  • UT-Zappos is limited to footwear and relatively consistent object presentation, aligning with lower compositional variability and different behavior in the results.Source: §I · p. 15

Related papers

Loading 2610.29811v1…