PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Minimizers of Sigmoid Contrastive Loss

Kiril Bangachev, Guy Bresler, Iliyas Noman, Yury Polyanskiy

arXiv:2509.18552v2cs.LGcs.AI

TL;DR

Representation-synchronization theory lacks coverage of practical dimensions and flexible cross-modal geometries. The paper characterizes sigmoid-loss global minimizers with trainable inverse temperature and bias as (m, b_rel)-Constellations, then uses this geometry to analyze retrieval, modality gaps, and relative-bias training. The theory explains perfect retrieval and linear modality separation under stated conditions, while synthetic experiments show faster convergence and greater flexibility for relative-bias parameterization.

  • Problem

    Prior theory does not adequately cover the practical regime d ≪ N ≪ 2^d or flexible synchronized representations that need not align across modalities.

  • Method

    The paper characterizes sigmoid-loss global minimizers with trainable inverse temperature and bias using (m, b_rel)-Constellations, then studies their capacity, retrieval, modality separation, and relative-bias reparameterization.

  • Results

    Sigmoid loss admits a rich family of zero-loss configurations; these yield perfect retrieval, explain linear modality separation in SigLIP and CLIP, and support faster synthetic-data convergence with relative-bias parameterization.

  • Takeaways & Limitations

    Synchronization can preserve modality-specific separation rather than alignment, while trainable relative bias provides a way to control geometry and support locked or multiple encoders.

  • Takeaways & Limitations

    The paper does not prove performance guarantees for first-order methods, fully resolve constellation capacity, or evaluate relative-bias parameterization on real data.

Abstract

from arXiv · show

The meta-task of obtaining and aligning representations through contrastive pretraining is steadily gaining importance since its introduction in CLIP and ALIGN. In this paper we theoretically explain the advantages of synchronizing with trainable inverse temperature and bias under the sigmoid loss, as implemented in the recent SigLIP and SigLIP2 models of Google DeepMind. Temperature and bias can drive the loss function to zero for a rich class of configurations that we call $(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations. $(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations are a novel combinatorial object related to spherical codes and are parametrized by a margin $\mathsf{m}$ and relative bias $\mathsf{b}_{\mathsf{rel}}$. We use our characterization of constellations to theoretically justify the success of SigLIP on retrieval, to explain the modality gap present in SigLIP and CLIP, and to identify the necessary dimension for producing high-quality representations. Finally, we propose a reparameterization of the sigmoid loss with explicit relative bias, which improves training dynamics in experiments with synthetic data.

1 Introduction

The paper studies representation synchronization with trainable inverse temperature and bias under sigmoid loss, addressing gaps in prior theory about practical dimensions and flexible cross-modal geometries. It characterizes zero-loss configurations, connects them to retrieval and modality gaps, and proposes relative-bias parameterization for locked or multiple modalities.

  • 1 Introduction: The paper targets the practical regime d ≪ N ≪ 2^d, where prior analyses either assume d ≥ N or let N grow with fixed d.This regime matters because phenomena such as modality gaps arise when N > d.
  • 1 Introduction: Trainable temperature and bias make sigmoid loss attain zero on a rich family of configurations characterized by margin m and relative bias b_rel.These (m, b_rel)-Constellations satisfy positive-pair inner products above a threshold and mismatched-pair inner products below it.
  • 1 Introduction: The inner-product separation condition is necessary and sufficient for global minimization, and the resulting configurations are also global minima of triplet loss.The condition is nearly satisfied by SigLIP trained on real images and text.
  • 1 Introduction: Sigmoid-loss minimizers impose globally thresholdable geometry, unlike InfoNCE minimizers, which use row-wise thresholds that vary by anchor.This distinguishes the solution geometries of the two contrastive objectives.
  • 1 Introduction: Constellation capacity is formulated as a combinatorial problem related to spherical codes, supporting analysis of the dimension needed to store many synchronized pairs.The margin also controls retrieval robustness, motivating bounds on constellation size for fixed dimension and sample count.
  • 1 Introduction: Nearest-neighbor search on any constellation gives perfect retrieval, while larger margins improve robustness to approximate-nearest-neighbor errors.This connects the geometric characterization directly to practical retrieval systems.
  • 1 Introduction: When |b_rel| < m and N > d, zero-loss configurations are linearly separable across modalities; all eight tested Hugging Face SigLIP models showed perfect separation on ImageNet.The paper interprets this modality gap as synchronization without requiring alignment of image and text representations.
  • 1 Introduction: Relative-bias parameterization supports locked encoders, multiple modalities, and deliberate control of representation geometry through trainable or fixed relative bias.The parameterization implicitly adds modality-dependent linear adapters and can guide solutions toward more diverse configurations.

2 Background and Prior Work

Representation synchronization covers multimodal alignment, distillation, and augmentation matching, with loss choice governed by minimizing geometry and computational efficiency. Prior work characterizes InfoNCE and sigmoid-loss solutions in restricted regimes, often favoring rigid alignment or simplex structures.

  • Representation synchronization includes image-text alignment, teacher-student distillation, self-distillation, and matching augmented views.
  • The formal setup uses unit-norm encoders whose pairwise embedding inner products are optimized through a loss with temperature and bias hyperparameters.
  • Parameter updates may train both encoders or preserve one trusted or locked encoder while adapting the other or adding a small adapter.
  • Loss selection depends on the geometry of desired minimizing representations and the computational efficiency of evaluating the loss with respect to batch size.
  • InfoNCE applies a soft-max across rows and columns, while sigmoid loss processes pairs separately without a batch normalization term, enabling parallel computation.
  • Prior solution-geometry results favor simplex or aligned configurations in regimes such as d ≥ N, d ≥ N + 1, or N → +∞, while sigmoid analyses often yield aligned or antipodal representations.

3 Main Results

This section characterizes zero-loss sigmoid configurations as $(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations, including their existence, dimensional limits, and extensions to multiple modalities. It also examines how jointly trained inverse temperature and relative bias support synchronization and retrieval-oriented representations, and evaluates explicit relative-bias parameterization and alternative training approaches in synthetic experiments.

  • 3.1 Geometric Characterization of Zero Loss Representations: All global minima are exactly (m, b_rel)-Constellations, and every such constellation is a global minimizer whose optimal margin controls convergence speed.Practically trained models are also approximately constellations after removing 5% outliers from positive and negative pairs.
  • 3.1 Geometric Characterization of Zero Loss Representations: A positive-margin zero-loss constellation gives perfect nearest-neighbor image-text retrieval, while larger margins improve robustness to approximate nearest-neighbor errors.Under the proposition’s batch-size conditions, nearest-neighbor search is correct for at least a 1 − Nξ/[B(B−1)] fraction of examples.
  • 3.2 Constructions of (d, m, b_rel)-Constellations And Cardinality Bounds: Constellations can be constructed from spherical codes, exist in exponential size under m + b_rel < 1 and 3m < 1 + b_rel, and these inequalities are necessary up to o(1).Upper bounds also constrain constellation size and therefore inform the embedding dimension needed for a desired number of pairs.
  • 3.3 The Modality Gap in SigLIP: For N ≥ d + 2 with positive matching and negative nonmatching inner products, a separating hyperplane places at least N − d representations from the opposite modality on one side.For SigLIP2 with N ≈ 10^10 and d ≈ 10^3, this leaves at most 0.0000001% of text embeddings outside the guaranteed separation; the theorem is essentially tight.
  • 3.4 Experiments: Sigmoid Loss with Explicit Relative Bias Parameterization: Trainable explicit relative bias converges faster than the original parameterization, while both outperform fixed temperature and bias on synthetic synchronization.The reparameterization also permits freezing relative bias; in SigLIP2 B/16 at 384 × 384, t ≈ 117.8, b ≈ −12.9, and b_rel ≈ −0.11.
  • 3.4 Experiments: Sigmoid Loss with Explicit Relative Bias Parameterization: For multiple modalities synchronized over a graph, zero loss requires a common margin and relative bias satisfying the constellation condition on every synchronized modality pair.This yields a direct extension from pairwise synchronization to complete or star graphs and motivates constructions for more than two modalities.
  • 3.4 Experiments: Sigmoid Loss with Explicit Relative Bias Parameterization: For k > 2 modalities, adapters append modality-dependent suffixes that enforce synchronization, while relative bias can capture the same structure through constant cross-modal inner products.The construction uses ⟨w_i, w_j⟩ = −1/(k−1) for distinct simplex vertices.
  • 3.5 Ablation Studies: Training temperature and bias drives the loss toward zero across the wide class of (m, b_rel)-Constellations relevant to retrieval.The experiments evaluate inner-product separation and convergence to zero loss because separation determines success under approximate nearest-neighbor retrieval.
  • 3.5 Ablation Studies: Trainable inverse temperature and relative bias outperform fixed-temperature and bias-parameterization alternatives in retrieval-relevant separation, convergence, or margin.Low fixed temperature yields smaller margins or violates separation; high fixed temperature fails to approach zero loss and produces unusable retrieval representations, while the alternative bias parameterization converges more slowly and tends toward smaller margins.

4 Limitations and Future Directions

The paper identifies (m, b_rel)-Constellations as configurations where trainable temperature and bias can drive sigmoid loss toward zero. It also notes unresolved theoretical and empirical limitations, including missing first-order guarantees, an incomplete dimension problem, and no real-data evaluation of the relative-bias parameterization.

  • 4 Limitations and Future Directions: The paper does not prove performance guarantees for first-order methods, fully resolve the dimension-selection problem, or test relative-bias reparameterization on real data.Experiments show faster convergence for the reparameterization only on synthetic data.
  • 4 Limitations and Future Directions: The authors report no known negative or direct societal impacts, while noting possible indirect relevance to large-scale machine-learning systems.
  • A.1 Global Minimizers of The Sigmoid Loss are (m, brel)-Constellations: The constellation characterization constrains matched similarities above a bias-adjusted margin and mismatched similarities below it.The proof uses these inequalities to establish the loss bound and characterize minimizing limits.
  • A.1 Global Minimizers of The Sigmoid Loss are (m, brel)-Constellations: Trainable temperature drives sigmoid loss toward zero on (m, b_rel)-Constellations with positive margin m.The proof bounds the loss by an exponentially decaying term in temperature.

A.2 Robustness of Nearest-Neighbor Retrieval: Proof of Proposition 1

The retrieval analysis shows that low sigmoid-loss configurations yield nearest-neighbor recovery for most data indices under suitable similarity thresholds. The proof obtains this by bounding the fractions of indices violating matched and mismatched similarity conditions.

  • For all but a bounded fraction of indices, matched similarities exceed the bias-to-temperature threshold.This is the first threshold condition used in the retrieval argument.
  • The proof separately bounds the fraction of index pairs whose mismatched similarities cross the same threshold.Combining the matched and mismatched bounds yields the nearest-neighbor result.
  • Nearest-neighbor search succeeds for the indices satisfying the derived matched-versus-mismatched similarity inequalities.The conclusion follows after optimizing the auxiliary violation parameters.

A.3 Global Minimizers of InfoNCE Loss

The InfoNCE analysis characterizes sufficient and necessary similarity-margin conditions for global minimizers, including a one-sided variant. These conditions require matched pairs to exceed relative-bias thresholds while mismatched pairs remain below them.

  • InfoNCE global minimizers satisfy matched-pair similarities at least b_rel(i)+m and mismatched similarities at most min(b_rel(i), b_rel(j))−m.
  • A one-sided InfoNCE sufficient condition requires matched similarities at least b_rel(i)+m and mismatched similarities at most b_rel(i)−m.
  • The necessary-condition analysis shows that inverse temperature diverges and limiting cross-similarities become negative under the stated iterative sequence assumptions.

A.4 Triplet Loss

The paper relates sigmoid-loss constellations to triplet-loss and InfoNCE minimizers through their margin conditions. It concludes that sigmoid minimizers form only a subset of the configurations minimizing InfoNCE and triplet loss.

  • A (m, b_rel)-Constellation also satisfies the triplet-loss constraints for any margin α ≤ 4m.The triplet hinge term is bounded above by α−2m under the constellation inequalities.
  • InfoNCE global minimizers imply triplet-loss optimality for margins α ≤ 2m.
  • A triplet-loss global minimizer also yields a constellation for suitable relative biases whenever m ≤ α/2.
  • Global minimizers of InfoNCE and triplet loss coincide, whereas sigmoid-loss global minimizers are only a subset of those configurations.The paper states that Fig. 2 illustrates this distinction.

B Proof of Theorem 3.5: Dimension vs Size tradeoff

The appendix proves the dimension–size tradeoff using spherical inner-product bounds and develops convex-geometric tools for the modality-gap analysis.

  • B Proof of Theorem 3.5: Dimension vs Size tradeoff: With H ∼ Unif(Sd−1), selecting C(H) = {i : ⟨ci,H⟩ > δ} yields a realization satisfying ∥UC + VC∥2 > 4N′2δ2, combined with (29) using AC ≥ (m + brel)N′ and BC ≤ (brel − m)N′(N′ − 1).Here ci = (Ui + Vi)/2 and δ ∈ (0, 1).
  • B Proof of Theorem 3.5: Dimension vs Size tradeoff: The proof derives a sufficient inequality involving δ, m, and b_rel by combining norm bounds and dropping the nonnegative ξ_C term.The resulting condition is stated as 2δ^2 > 1 − 3m + b_rel.
  • C.1 Preliminaries from Convex Geometry: The modality-gap proofs rely on convexity, convex hulls and cones, dual cones, and separation theorems.The appendix introduces Helly’s theorem, Carathéodory’s theorem, and the hyperplane separation theorem as its main geometric tools.
  • C.1 Preliminaries from Convex Geometry: Helly’s theorem is stated as reducing global intersection to checking every d + 1 convex sets, while Carathéodory’s theorem reduces convex-hull representations to at most d + 1 points.These results provide the finite-dimensional reductions used in the subsequent modality-gap arguments.

C.2 Combinatorics of Modality Gap

This section establishes combinatorial constraints on modality-gap configurations and constructs examples showing that the resulting separation bound is nearly tight.

  • C.2 Combinatorics of Modality Gap: When (32) holds and N ≥ d + 2, a vector h exists with positive inner product with every U_i and lying in their convex hull.The proof obtains h using Helly’s theorem, projection onto the convex hull, and normalization.
  • C.2 Combinatorics of Modality Gap: A separating direction can have negative inner product with all but at most d vectors V_j, yielding separation of at least N − d vectors.Carathéodory’s theorem reduces the relevant convex representation to at most d vectors, while the remaining vectors have negative inner products.
  • C.2 Combinatorics of Modality Gap: A construction separates all but at least d − 1 vectors with a hyperplane, making the preceding bound almost tight.The construction preserves positive matching and negative nonmatching inner products, while every direction leaves at least d − 1 pairs with matching signs.
  • C.2 Combinatorics of Modality Gap: The construction extends from d = 3 to higher dimensions by embedding the three-dimensional configuration in an orthogonal complement of a simplex.A sufficiently small perturbation preserves the required dot-product signs.
  • C.2 Combinatorics of Modality Gap: No configuration satisfying (32) exists when d = 2 and N ≥ 4.After rotation, all U_i lie in a right half-plane; ordering their angles forces U_2 into cone(U_1,U_3), contradicting the preceding proposition.

D.1 Experiments on ImageNet

The ImageNet experiments evaluate original SigLIP representations on captioned validation images and show near separation of matching and nonmatching pairs, with some semantically understandable errors.

  • D.1 Experiments on ImageNet: The study uses 50,000 captioned ImageNet validation images, 1,000 distinct captions, and eight trained models evaluated with the B/16 encoder.The authors use original SigLIP rather than SigLIP 2 to isolate the core sigmoid contrastive loss from SigLIP 2’s auxiliary training losses.
  • D.1 Experiments on ImageNet: Inner-product separation is nearly satisfied, although the “American chameleon” image matches “African chameleon” more closely than its correct caption.The authors characterize this as an expected practical error because both captions refer to related species.
  • D.1 Experiments on ImageNet: 84.5% ImageNet accuracy is reported for large models in [ZMKB23b].

D.2 Experiments with Locked Representation

Locked-representation experiments compare fixed and trainable sigmoid-loss parameterizations, finding that trainable bias and inverse temperature enable separation and zero-loss convergence.

  • D.2 Experiments with Locked Representation: Trainable temperature and relative bias are evaluated by fixing one modality, synchronizing the other, and optimizing the respective loss with gradient descent.The experiments include fixed-temperature baselines, trainable temperature and bias, and trainable temperature with relative bias.
  • D.2 Experiments with Locked Representation: The trainable-temperature experiments parameterize t as e^t′ to ensure positive temperature.This reparameterization is used in the trainable settings and follows [ZMKB23b].
  • D.2 Experiments with Locked Representation: The adapter and no-adapter relative-bias variants have comparable performance and similar inner-product separation, but adapter training appears less stable.Because the adapter is invertible, inner products are reported both before and after removing it.
  • D.2 Experiments with Locked Representation: Only variants learning both bias and inverse temperature reach zero loss, while the relative-bias parameterization converges fastest.Figure 16 compares six parameterizations using score histograms and loss trajectories over 10,000 iterations.

D.3 Experiments with Multiple Modalities

Experiments show that synchronizing more modalities and using relative bias can improve separation margins, while initialization and parameterization materially affect optimization and final representations.

  • D.3 Experiments with Multiple Modalities: The trainable-temperature experiments use t = et′ to ensure positive temperature, with Adam and initial learning rate 0.01.The reported four-modality experiment uses d = 10 and N = 100.
  • D.3 Experiments with Multiple Modalities: Synchronizing more modalities generally increases the final separation margin, suggesting more robust representations.The experiment uses N = 100 and d = 10.
  • D.5 Initializing Fixed Relative Bias: A fixed relative bias near 0.7 achieves the largest final margin among the tested values.The experiment trains representations and inverse temperature while holding relative bias fixed.
  • D.4 Bias Parameterization Leads to Zero Relative Bias: LRB−Sig generally converges faster to zero loss and produces larger margins than LSig, while LSig drives relative bias toward zero.The larger margin is associated with greater retrieval robustness.
  • D.6 Initializing Learnable Temperature and Relative Bias: Final margins are best with low initial temperature or intermediate temperature near 10 paired with relative bias near 0.6.High initial temperature with low relative bias can yield small or negative margins.

E Connection to Linear Representation Hypothesis Across Modalities

The paper connects the quantity ξ to cross-modal linear representation structure: small ξ indicates a nearly constant shift between paired image and text embeddings. Direct representation training can drive ξ toward zero, whereas real SigLIP models retain substantial ξ on ImageNet validation.

  • E Connection to Linear Representation Hypothesis Across Modalities: ξ measures how closely a constellation satisfies the Linear Representation Hypothesis and is required to be near zero for the upper bound in Theorem 3.5 to be tight.Thus ξ links representation geometry to the tightness of the paper’s cardinality bound.
  • E Connection to Linear Representation Hypothesis Across Modalities: Small ξ means the differences Ui − Vi are nearly identical across pairs, so one concept-shift vector approximately maps text representations to their matching image representations.If ξ = 0, all Ui − Vi are identical.
  • E Connection to Linear Representation Hypothesis Across Modalities: Directly training representations drives ξ toward zero across a range of dimensions in the reported experiments.This behavior is illustrated in Figure 21 for d = 10 and 100 paired representations.
  • E Connection to Linear Representation Hypothesis Across Modalities: For real models on ImageNet validation, ξ remains far from zero, and paired norm statistics are closer to random pairing than to the norm of means.The authors suggest that the model dimension d ≈1000 may be far from optimal for requiring ξ = 0.
Loading 2509.18552v2…