PaperlayerView exact revision on arXiv

Source-linked AI summary

Expressivity In Multimodal Contrastive Learning

Andrew Stuart, Florian Wolf

arXiv:2608.17203v1stat.MLcs.LGmath.ST

TL;DR

The paper asks whether multimodal contrastive parameterizations can universally approximate joint distributions, beyond finite-sample, optimization, and generalization questions. It analyzes existing architectures and proposes Hadamard-CLIP, showing that two-tower CLIP is universal for two modalities, sum-of-pairs CLIP is not for three or more, and Hadamard-CLIP restores universality.

  • Problem

    Existing work has paid comparatively little attention to whether contrastive parameterizations can universally approximate underlying joint distributions rather than only certain conditionals.

  • Method

    The paper adopts a population-level density-estimation viewpoint, analyzes induced tilting functions, and introduces Hadamard-CLIP by adding one learned weight vector.

  • Results

    Two-tower CLIP is universal for two modalities; sum-of-pairs CLIP matches pairwise conditionals but not arbitrary joints for three or more, while Hadamard-CLIP restores joint universality.

  • Takeaways & Limitations

    Hadamard-CLIP closes the expressivity gap for any number of modalities while preserving fast retrieval with precomputable embeddings.

  • Takeaways & Limitations

    The guarantees are population-level and leave the effects of empirical approximation, optimization, and latent-dimension accuracy tradeoffs open.

Abstract

from arXiv · show

Contrastive learning has become a cornerstone of modern representation learning, powering CLIP-style models that underpin text-to-image generation, vision-language models, and retrieval across a rapidly growing range of modalities. Despite this empirical success, the expressive power of these architectures remains poorly understood. To gain insight, we study expressivity by adopting a population-level, density-estimation viewpoint: each architecture comprises a parameterized set of densities whose parameters may be chosen to approximate the joint distribution of the modalities. This isolates a question of pure representational capacity: which joint distributions can a given contrastive family of parameterizations approximate to arbitrary accuracy? We show that expressivity is sharply architecture-dependent. For two modalities, the simple two-tower CLIP architecture is a universal approximator. A natural generalization of CLIP, widely used in practice when three or more modalities are present, is based on a loss found by summing over all pairwise similarities. This provably cannot represent arbitrary joint distributions, although we prove that it remains expressive enough to match all pairwise conditionals. Motivated by this gap, we propose Hadamard-CLIP, which adds a single learned weight vector on top of the existing encoders and restores universal approximation of the joint for any number of modalities while preserving CLIP's fast, precomputable-embedding retrieval.

1 Introduction

This paper studies the expressivity of multimodal contrastive parameterizations in the population limit, treating them as density estimators of joint modality distributions. It proves sharp architecture-dependent results: bimodal CLIP and Hadamard-CLIP are universal, whereas sum-of-pairs CLIP is limited to pairwise conditionals for m ≥ 3.

  • Bimodal CLIP: For m = 2, two-tower CLIP universally approximates any joint distribution and matches a general joint network in approximation power.This holds for both unnormalized encoders and normalized encoders with learnable temperature.
  • Beyond two modalities: For m ≥3, the sum-of-pairs CLIP generalization universally approximates all pairwise conditionals simultaneously but cannot approximate the joint.Because it models only pairwise interactions, it cannot represent the full conditional xi | x−i or the joint.
  • Hadamard-CLIP: Hadamard-CLIP restores universal approximation of the joint for any m by adding one learnable vector ω to the element-wise product of encoder outputs.The modification preserves CLIP’s inference-time efficiency and precomputable embeddings.
  • Population-level formulation: The analysis isolates representational capacity by using the population limit, where contrastive learning estimates the joint law through a learnable tilt of the product of marginals.The tilt is the log-density ratio relative to independence and vanishes when modalities are independent.
  • Motivation: The paper focuses on expressiveness itself, which prior contrastive-learning analyses generally presuppose rather than characterize.The stated theoretical contributions are universal approximation theorems and failure examples for multimodal contrastive algorithms.

2 Probabilistic Formulation

This section formulates multimodal contrastive learning as approximation of a joint measure through a tilting function relative to the product of modality marginals. It connects population-level divergences to conditional contrastive losses, establishes KL-based control results, and identifies incomplete multimodal data as a practical limitation.

  • 2.2 Population-Level Multimodal Constrastive Loss Function: Population-level conditional losses are motivated by matching the conditional distributions of µ and µa.The framework is intended to connect population-level theory with contrastive learning algorithms and finite-sample objectives.
  • 2.1 Problem Formulation: The data consist of m ≥2 modalities with marginal measures absolutely continuous relative to restricted Lebesgue measures.Each marginal µi has density fi ∈L1(Xi, λi).
  • 2.1 Problem Formulation: The joint measure µ is represented as a density tilt r relative to the product of marginals, with finite mutual information ensured by log r ∈L1(X, µ).The log density Φ := log(r) is the tilting function used in the theoretical analysis.
  • 2.1 Problem Formulation: The approximation goal is to select a finite-valued tilting function Φa so that the induced measure µa approximates µ under a specified metric or statistical divergence.The analysis studies how approximating Φ by Φa affects D(µ, µa).
  • 2.2 Population-Level Multimodal Constrastive Loss Function: For m = 2, separately encoded and normalized modalities yield a population conditional objective whose empirical approximation exactly recovers the standard CLIP loss.The empirical objective becomes a sum of two cross-entropy losses for conditional distributions.
  • 2.2 Population-Level Multimodal Constrastive Loss Function: For m ≥3, full conditional losses are limited by the rarity of datasets containing every modality for every sample.Large-scale multimodal datasets commonly aggregate heterogeneous sources, motivating objectives that work beyond the complete-data setting.
  • 2.3 Multimodal Contrastive Learning: Choosing D as a KL divergence makes small joint divergence control the conditional distributions and the full-conditional and pairwise contrastive losses.The results apply because µ ≪µa by construction.
  • 2.3 Multimodal Contrastive Learning: Minimizing DKL(µ ∥µa) is strictly stronger than minimizing averaged full-conditional losses and also controls pairwise conditional losses.Theorems 2.5 and 2.6 state these controls for every modality and every distinct modality pair, respectively.

3 General Universal Approximation

Under a C0-universal approximation assumption for the tilting-function family, the paper establishes universal approximation of the target measure in both compact-support and tightness settings. The abstract result reduces model expressivity to verifying this assumption for the architecture, which holds for bimodal CLIP encoders but fails for the natural m≥3 extension.

  • Abstract universal approximation: If MNN universally approximates continuous functions, the induced tilting-measure family can approximate the data-generating measure µ with respect to KL divergence.This is the abstract content of Theorems 3.2 and 3.3.
  • Architectural verification: At the architectural level, universal approximation reduces to verifying UA Assumption 3.1 for modality-specific encoders; this succeeds for bimodal CLIP but fails for the natural m≥3 extension.The failure of the multi-modal extension motivates the later architectural analysis.
  • Universal approximation assumption: UA Assumption 3.1 requires approximating every continuous Φ ∈ C0(X; R) arbitrarily well on compact K under the sup norm.For every ϵ > 0 and compact K ⋐ X, some parameterized function achieves the required approximation.
  • Compact support: Theorem 3.2 establishes joint-network universal approximation when X is compact and µ satisfies Assumption 2.1.For every ϵ > 0, an appropriate parameter and tilting function exist within the assumed function family.
  • Tightness: Theorem 3.3 extends joint-network universal approximation to potentially non-compact X under the same function-class assumption.The extension is stated for measures satisfying Assumption 2.1 and uses tightness rather than compactness of the full domain.

4 CLIP: Contrastive Learning m = 2

In the bimodal setting, CLIP is a universal approximator for compactly supported measures with either unnormalized or normalized encoders, provided the normalized case has learnable temperature. However, universal approximation does not guarantee dimensionality reduction: some distributions require latent dimension at least that of the joint domain.

  • Universal approximation: CLIP achieves universal approximation for bimodal compactly supported measures with unnormalized encoders.For every ϵ > 0, suitable latent dimension, parameter count, and encoder parameters exist.
  • Universal approximation: Normalized encoders also achieve universal approximation when the temperature τ > 0 is learnable.A sufficiently small τ overcomes the bounded inner-product range imposed by normalization.
  • Universal approximation: The bimodal CLIP function is as expressive as the more general joint network ΦNN.This establishes equal representational power in the classical two-modality setting.
  • Dimensionality limitation: CLIP does not guarantee latent dimensionality below the original joint-domain dimension, even when encoder architectures and parameter dimensions vary.The lower bound applies uniformly over temperatures, parameterizations, and normalized or unnormalized encoders.
  • Dimensionality limitation: Four Fourier features yield an exact representation with normalized encoders and τ = 1/2, making the latent-dimension threshold sharp.The construction uses X1 = X2 = [−1, 1] and demonstrates the claim at d = 1 < D.

5 Contrastive Learning m > 2

For m ≥ 3 modalities, the standard pairwise-similarity CLIP generalization cannot universally approximate arbitrary joint distributions because it captures only pairwise interactions, though it can match all pairwise conditionals. Hadamard-CLIP restores universal approximation by adding a learned weighted Hadamard-product fusion layer while preserving precomputable-embedding retrieval.

  • Failure of pairwise CLIP: For every m ≥ 3, (Φm-clip-Tilting) fails to universally approximate arbitrary joint distributions, even after ranging over latent dimensions, parameter spaces, and encoder parameterizations.The limitation is structural: its separable form captures only pairwise interactions and excludes higher-order multimodal effects.
  • Pairwise conditional approximation: (Φm-clip-Tilting) remains expressive enough to universally approximate all pairwise conditionals simultaneously under the stated compactness and neural-network universality assumptions.The proof represents consistent pairwise conditionals using sums of two-variable functions within an exponential family, then approximates those representatives.
  • Hadamard-CLIP: Hadamard-CLIP restores universal approximation by applying a learned linear functional to the element-wise product of all modality encoders.This fusion generates degree-m multilinear interactions and adds only a single learnable weight vector on top of the existing encoders.
  • Efficient retrieval: Hadamard-CLIP preserves CLIP’s retrieval efficiency by storing Hadamard-product embeddings and reducing inference to one query-encoder pass followed by N dot products.The stored vectors are dLat-dimensional, reducing storage requirements in large-scale applications.

6 Conclusion and Future Work

The paper studies contrastive learning as population-level approximation of joint distributions and shows that expressivity depends sharply on architecture. It establishes universality for bimodal two-tower CLIP, identifies limitations of sum-of-pairs models, and proposes Hadamard-CLIP to close the gap.

  • The paper frames contrastive learning as learning approximations of joint probability distributions from data, focusing on population-level expressivity.
  • Bimodal two-tower CLIP is as expressive as a general joint network, while sum-of-pairs models fail to represent arbitrary joints for three or more modalities.Sum-of-pairs models still match every pairwise conditional.
  • Hadamard-CLIP closes this expressivity gap through a minimal modification that adds a single learned weight vector.
  • Future work: Open questions include sharp latent-dimension bounds, the dimension–accuracy tradeoff, and statistical-learning analysis beyond the population limit.Proposition 4.3 shows that compression below the original dimension is generally impossible.

Appendix A. Technical Lemmas and Proofs

Appendix A collects the technical results and full proofs supporting the main text. It develops loss-control, approximation, bimodal CLIP, and joint-universality results.

  • The appendix first proves that joint KL divergence controls both full and pairwise conditional losses.
  • It then develops general function-to-measure approximation tools used throughout the universal approximation arguments.
  • A subsequent subsection establishes bimodal CLIP results, including normalized-encoder construction and latent-dimension obstruction.
  • The appendix also proves the failure of joint universal approximation.

A.1 Contrastive Learning Population-Level Loss Functions

The subsection proves the population-level loss comparisons underlying Theorems 2.5 and 2.6. It establishes that the relevant divergences and conditionals are well-defined, then applies KL-divergence chain rules to bound full and pairwise conditionals.

  • Well-definedness and absolute continuity: The proofs first verify that all KL divergences and conditional distributions in Theorems 2.5 and 2.6 are well-defined.This follows from the required absolute-continuity relations and disintegration arguments.
  • Theorem 2.5: For Theorem 2.5, absolute continuity enables the Radon-Nikodym chain rule even when Φ takes values in R∪{±∞}.The proof uses Φa's finite-valuedness, µ ≪ µP, and marginal absolute continuity to establish the needed relations.
  • Theorem 2.5: Theorem 2.5 applies the KL divergence chain rule to decompositions into a marginal and a full conditional distribution.The argument fixes i and decomposes µ and µa using x−i and xi|x−i.
  • Theorem 2.6: Theorem 2.6 repeats the absolute-continuity argument for pairwise conditionals and applies the KL chain rule after separating (xi, xj) from the remaining variables.The proof then applies the resulting argument to the bivariate marginal µij and its corresponding approximation.

A.2 General Universal Approximation Results

This subsection develops architecture-independent tools that convert approximations of tilting functions into approximations of measures. It establishes compact-support and tightness-based bounds for the resulting KL divergence, including truncation and tail-error controls.

  • Tightness extension: The non-compact extension restricts the measures and tilting function to a compact set obtained through tightness.The auxiliary measure ν(A) := ∫A |Φ| dµ supplies an additional tail-control condition for the approximation.
  • Continuous approximation: The continuous-representative lemma approximates any integrable log-density by a continuous function on a compact domain.The construction uses double-sided truncation, Lusin’s theorem, and Tietze extension while controlling the L1 approximation error.
  • Compact support: The compact-support function-to-measure proposition bounds KL divergence when an approximating tilting function is uniformly close to the continuous representative.The bound is DKL(µ ∥µa) ≤ϵ + δ · µ(X) + δ + log(µ(X)) + ϵ µ(X).
  • Tightness extension: The tightness-based proposition controls the integral approximation error by ϵ + δ + (1 + B)η.Its assumptions include compact-set approximation, tail bounds, and constants δ > 0 and B ≥ 0.
  • Tightness extension: The normalization error in the tightness argument is bounded by δ + ϵ + exp(B)η.This follows by separately bounding contributions inside and outside the compact truncation set.

A.3 CLIP: Contrastive Learning m = 2

For two modalities, Stone–Weierstraß yields universal approximation for CLIP with unnormalized encoders, and normalization with a learnable temperature preserves this guarantee. However, normalized CLIP has a sharp latent-dimension obstruction for low-dimensional approximations.

  • Bimodal Stone–Weierstraß approximation: For compact modality domains, separable encoder products form a dense subalgebra of continuous bimodal functions, establishing the core approximation lemma.The construction is closed under addition, scalar multiplication, and multiplication, contains constants, and separates points.
  • Theorem 4.1: Unnormalized two-tower CLIP satisfies the universal-approximation assumption by approximating the separable factors with its encoders and controlling the resulting inner-product error.The proof invokes the bimodal approximation lemma and concludes via Theorems 3.2 and 3.3.
  • Theorem 4.2: Normalized CLIP exactly recovers the unnormalized approximation guarantee by orthogonal padding and setting the learnable temperature to τ := α^2τ0.The padded encoders are normalized by construction, while orthogonality preserves the relevant inner product up to the temperature rescaling.
  • Proposition 4.3: 4 non-zero singular values equal to 1/2: the target bimodal tilting operator has rank four, whereas normalized CLIP with latent dimension d has score rank at most d.The rank bound is unaffected by τ because temperature rescales singular values without changing their number.
  • Proposition 4.3: d ∈ {1, 2, 3}: localized Eckart–Young–Mirsky arguments produce a measure-level obstruction to approximating the target log-density with normalized CLIP.The contradiction uses a target grid matrix M0 = (5/2)I4 and shows that at least one involved log-density error must exceed the prescribed threshold.
  • Proposition 4.3: The latent-dimension threshold is sharp for normalized CLIP, as shown by identical normalized encoders mapping [−1, 1] into S3.This construction provides the matching positive direction at the threshold.

A.4 Contrastive Learning m > 2

For more than two modalities, the standard pairwise extension of CLIP cannot universally approximate the joint distribution. The proof constructs a distribution with matching pairwise marginals yet a uniform positive approximation error, while motivating a representative that preserves pairwise conditionals.

  • A.4 Contrastive Learning m > 2: The standard pairwise extension of CLIP cannot universally approximate arbitrary joint distributions when m > 2.This is established by Theorem 5.1.
  • A.4 Contrastive Learning m > 2: The subsection develops a consistent exponential-family representative that preserves all pairwise conditionals for use in the next subsection.This construction addresses the gap identified for the pairwise extension.
  • A.4 Contrastive Learning m > 2: The counterexample has all univariate and pairwise marginals equal to those of a product reference measure, despite containing a three-way interaction.Integrating over any of the first three variables eliminates the three-way interaction, so the pairwise marginals coincide with the product marginals.
  • A.4 Contrastive Learning m > 2: 1/288 is a uniform positive lower bound on the model’s approximation error across latent dimensions, parameter dimensions, encoder parameterizations, parameters, and temperatures.The proof derives the bound using conditional KL divergences and Pinsker’s inequality.

A.4.2 Consistent L1 Representatives for Pairwise Conditionals

The section constructs a unique consistent representative by projecting onto measures with the data distribution’s pairwise marginals, then shows this representative is an exponential-family measure that interpolates all pairwise conditionals.

  • Consistent representative: The representative µ⋆ is defined as the KL-minimizing measure whose pairwise marginals match those of the data distribution.The constraint set is Cµ := {ν ∈ P(X) | νij = µij for all i < j}.
  • Consistent representative: The constraint set is non-empty, convex, and closed under total-variation limits, ensuring the projection problem has a unique solution µ⋆.Non-emptiness follows because µ ∈ Cµ; linearity gives convexity, and marginalization preserves total-variation convergence.
  • Exponential-family form: The unique solution µ⋆ has a density relative to the product reference measure with pairwise additive log-potentials and a normalization constant.The supplied passages identify the pairwise potentials as Φij and the normalization constant as Z⋆ > 0.
  • Pairwise conditionals: µ⋆ belongs to the exponential family representable by the CLIP architecture and perfectly interpolates every pairwise conditional of the original measure µ.The construction also establishes µ ≪ µ⋆ before applying the KL chain rule.

A.5 Proof of Theorem 5.2 · A.6 Proof of Theorem 5.3

Theorem 5.2 proves simultaneous universal approximation of all pairwise conditionals through a proxy target and block-orthogonal normalized encoders. Theorem 5.3 proves Hadamard-CLIP universal approximation by combining Stone–Weierstrass density, encoder approximation, and normalization.

  • A.5 Proof of Theorem 5.2: Theorem 5.2 constructs a proxy target with separable structure, applies bimodal CLIP approximation pairwise, and translates the result into simultaneous approximation of all pairwise conditionals.The proof proceeds through a consistent pairwise representative, bimodal approximation, and a block-orthogonal normalized-encoder construction.
  • A.5 Proof of Theorem 5.2: The block-orthogonal encoder construction makes each cross-modal inner product recover exactly the corresponding pairwise encoder interaction while preserving unit-norm encoders.A shared temperature is defined as τ := α2 > 0, and orthogonal padding enforces ∥¯gi(xi; ˆθi)∥2 = 1.
  • A.5 Proof of Theorem 5.2: µm-clip(·; ˆθ) universally approximates the proxy target µ⋆, and the proxy’s matching pairwise marginals yield the desired pairwise conditional approximation.The translation uses µ⋆ij = µij, disintegration, strict positivity of the approximating density, and Theorem 2.6.
  • A.6 Proof of Theorem 5.3: Theorem 5.3 establishes density of the Hadamard subalgebra in C0(X; R) using Stone–Weierstrass, because it contains constants and separates points.Closure under addition and scalar multiplication is direct, while multiplication produces sums of fully separable monomials.
  • A.6 Proof of Theorem 5.3: For arbitrary ϵ > 0, continuous target functions are approximated by weighted fully separable monomials whose factors are approximated by the encoder families.A telescoping product bound controls each monomial, and summing weighted errors gives ϵ-closeness in the supremum norm.
  • A.6 Proof of Theorem 5.3: Normalization is enforced by rescaling and orthogonal padding with dLat := K + m, producing unit-norm encoders without altering the represented Hadamard interactions.For k ≤ K the products recover the unnormalized approximation, while for k > K at least one padding factor is zero.
  • A.6 Proof of Theorem 5.3: Absorbing the weights into the normalized construction exactly recovers the unnormalized approximation, and Theorem 3.2 or Theorem 3.3 yields the final approximation result for measures.The construction therefore completes the universal-approximation proof for Hadamard-CLIP.
Loading 2608.17203v1…