PaperlayerView exact revision on arXiv

Source-linked AI summary

Closing the Modality Gap via Simplex-Constrained Representations

Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi, Cem Subakan

arXiv:2610.32890v1cs.AIcs.CV

TL;DR

Contrastive multimodal models can retain systematic offsets between modality-wise embedding distributions despite objectives designed to align matched pairs. The paper uses lightweight adapters to project outputs onto the probability simplex, reporting 97–99% centroid ℓ2 gap reductions across five benchmarks while matching or improving retrieval.

  • Problem

    Contrastive multimodal models can retain systematic offsets between modality-wise embedding distributions despite objectives designed to align matched pairs.

    Source: §1 · p. 1

  • Method

    The approach changes output geometry to simplex-constrained representations using lightweight adapters, with softmax projection and negative Total Variation scoring.

    Source: §1 · p. 1, §5.1 · p. 6

  • Results

    Across five benchmarks, simplex projections reduce centroid ℓ2 gap by 97–99% while matching or improving retrieval.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Matched-capacity comparisons indicate that simplex constraints, rather than additional architectural capacity, drive the observed gap closure.

    Source: §1 · p. 2

Abstract

from arXiv · show

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent \emph{modality gap}. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, $S^{d-1}$, to the probability simplex, $\Delta^{d-1}$. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid $\ell_2$ gap by 97--99\% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.

1 Introduction

The introduction frames modality gap as a persistent separation between modality-wise distributions despite matched-pair objectives, with debated consequences. It proposes simplex-constrained outputs and reports large gap reductions while preserving retrieval.

  • The practical effects of modality gap remain debated: it may miscalibrate similarity and harm retrieval, while collapsing it may disrupt within-modality geometry and downstream performance.Source: §1 · p. 1
  • Replacing spherical outputs with simplex constraints imposes nonnegative, unit-mass embeddings that allocate a shared probability budget across the same coordinates.Softmax adapters also make representations invariant to global logit shifts.Source: §1 · p. 1
  • The approach freezes LAION-CLAP or FLAVA, attaches shared lightweight adapters, and trains with in-batch InfoNCE while scoring simplex outputs by negative Total Variation distance.Source: §1 · p. 1
  • Matched-capacity comparisons include Euclidean and simplex Linear/MLP adapters, cosine-scored XAttn, ReTreever, and MRL.These comparisons are designed to isolate output geometry from adapter capacity and shared cross-attention.Source: §1 · p. 2
  • Simplex-constrained adapters reduce centroid ℓ2 modality gap by 97–99% across five benchmarks while matching or improving NDCG@10.Source: §1 · p. 2

2 Related Work

Related work examines the modality gap’s geometric and practical implications, approaches for managing cross-modal separation, and simplex representations in retrieval and other settings.

3 Method

The method compares spherical and simplex outputs using shared lightweight adapters and a symmetric contrastive objective. Its geometric analysis characterizes symmetry and distance behavior, while matched baselines separate output geometry from adapter and cross-attention choices.

3.1 Adapters and output domains

  • Frozen multimodal embeddings pass through shared adapters into either L2-normalized, cosine-scored spherical outputs or softmax-projected simplex outputs scored by negative Total Variation.The adapters are linear or two-layer MLPs.Source: §3.1 · p. 3

3.2 Training objective

  • A symmetric DPR-style in-batch contrastive loss trains on paired text and audio or image examples, using cosine or negative Total Variation according to output domain.Source: §3.2 · p. 3

3.3 Sphere vs. simplex: symmetries and distance scaling

  • Simplex nonnegativity and unit mass remove most spherical rotational and sign-flip freedom, while softmax outputs are invariant to global logit shifts.Source: §3.3 · p. 3
  • Expected simplex distances depend on density: fixed-α vectors contract as O(1/d), whereas α=c/d yields a plateau.The analysis estimates concentration by moment matching and uses RelGap to normalize centroid separation by intra-modality spread.Source: §3.3 · p. 3

3.4 Baselines

  • Matched Linear/MLP pairs isolate output geometry; XAttn versus ReTreever controls for cross-attention, while MRL versus ReTreever compares coarse-to-fine retrieval across domains.Source: §3.4 · p. 4

4 Experimental Setup

The experiments test bidirectional retrieval on five paired audio–text and image–text benchmarks using frozen encoders with lightweight adapters. They compare Euclidean and simplex outputs using gap and retrieval metrics.

4 Experimental Setup

  • Five paired retrieval benchmarks cover audio–text (Clotho, AudioCaps, SoundDescs) and image–text (MS-COCO Captions, Flickr30k).Retrieval is evaluated in both directions for each modality pair.Source: §4 · p. 5
  • Frozen LAION-CLAP and FLAVA backbones are evaluated with adapter configurations including Euclidean outputs, softmax-projected simplex outputs, and cross-attention models.Adapters are trained for 200K steps with fewer than 1% trainable parameters.Source: §4 · p. 5
  • Modality gaps are measured with centroid ℓ2, silhouette, and RelGap, which normalizes centroid separation by intra-modality spread for cross-domain comparison.Retrieval quality is measured with NDCG@10 in both directions.Source: §4 · pp. 5–6
  • Figure 4 compares RelGap for adapter types on SoundDescs and AudioCaps, with non-softmax methods in red and softmax methods in blue.Lower values are better, and the caption reports that softmax methods consistently reduce RelGap.Source: Figure 4 · p. 6

5 Results

Across five benchmarks, simplex outputs sharply reduce modality gaps while maintaining retrieval quality. Comparisons, scaling tests, and analysis of learned embeddings support output geometry as a central factor in the observed gap reduction.

5.1 Simplex projection collapses the gap while preserving retrieval

  • 97–99%: softmax simplex projection reduces centroid ℓ2 gaps across datasets, while retrieval is preserved and sometimes improved.Clotho Lin.+Sfx raises T2A NDCG@10 from 0.259 to 0.382; SoundDescs ReTreever records retrieval scores of 0.535/0.542 with a small gap.Source: §5.1 · p. 6
  • RelGap, which normalizes centroid separation by intra-modality spread, is consistently lowest for simplex methods, supporting alignment beyond uniform distance rescaling.SoundDescs RelGap is 0.521 frozen, 0.179 for MLP+Sfx, and 0.097 for ReTreever; AudioCaps values are 0.380, 0.180, and 0.113, respectively.Source: §5.1 · p. 6

5.2 Separating cross-attention from output geometry

  • Shared cross-attention alone does not close the gap: XAttn retains substantial gaps, while simplex-output ReTreever achieves much smaller centroid gaps and RelGap.On SoundDescs, XAttn has centroid ℓ2 0.309 and RelGap 0.331, versus ReTreever at 0.012 and 0.097.Source: §5.2 · p. 7

5.3 Coarse-to-fine scaling: Euclidean gaps grow with dimension, simplex gaps stay small

  • 17×: MRL’s AudioCaps centroid gap grows from 0.032 at 2-d to 0.534 at 256-d, while ReTreever remains below 0.06 across resolutions.The comparison reports increasing gaps for Euclidean MRL and small gaps for simplex ReTreever across dimensions.Source: §5.3 · p. 7

5.4 Learned simplex embeddings lie in the sparse regime

  • ∼18 to ∼0.06: SoundDescs’ estimated Dirichlet concentration falls as dimension grows, entering the sparse regime below α = 1 by d = 16.Despite baseline distances plateauing in this regime, the modality gap remains very low and RelGap remains lowest for simplex methods, supporting an alignment rather than distance-scale explanation.Source: §5.4 · p. 7

6 Conclusions

The paper presents simplex projection as a geometric approach to reducing multimodal modality gaps. Across five benchmarks, it reports large gap reductions with retrieval maintained or improved, and identifies simplex projection as central to the observed reductions.

6 Conclusions

  • Simplex constraints use nonnegativity and unit mass to impose a shared coordinate system as cross-modal calibration.Across five benchmarks, simplex projections reduce centroid ℓ2 gaps by 97–99% while matching or improving retrieval; comparisons with Euclidean cross-attention baselines indicate shared attention alone does not reliably eliminate the gap.Source: §6 · p. 7

Related papers

Loading 2610.32890v1…