Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

Weixin Liang; Yuhui Zhang; Yongchan Kwon; Serena Yeung; James Zou

Source-linked AI summary

Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, James Zou

arXiv:2203.02053v2Mar 3, 2022cs.CLcs.AIcs.CVcs.LGcs.MM

TL;DR

Multi-modal models place different modalities in separate regions of their shared representation space, but the origins and implications of this modality gap are not fully understood. The paper combines theoretical analysis and experiments to attribute the gap to initialization-induced cone effects and contrastive optimization, finding that changing its distance affects downstream zero-shot performance and fairness.

Problem
The paper investigates why multi-modal models embed different modalities in separate regions and how this gap relates to downstream performance and fairness.
Method
The paper combines theory and experiments to explain the gap through narrow initialization cones from two encoders and contrastive learning that preserves separation.
Results
The modality gap appears across diverse multi-modal models and modalities, while changing its distance affects downstream zero-shot performance and fairness.
Takeaways & Limitations
Modality-gap distance is a consequential geometric property, with larger gaps helping some fairness and zero-shot learning applications.
Takeaways & Limitations
How mismatched data affect the contrastive loss landscape and modality gap remains an open direction for future research.

Abstract

from arXiv · show

We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized. During optimization, contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness. Our code and data are available at https://modalitygap.readthedocs.io/

1 Introduction

The paper identifies a pervasive modality gap in which different modalities occupy separate regions of shared representation spaces, even under random initialization. It explains the gap through neural-network cone effects, differing encoder initializations, and contrastive learning, and shows that modifying the gap can improve zero-shot performance and fairness.

Modality gap phenomenon: The modality gap appears across multi-modal models and modalities, with image and text embeddings occupying separate regions even when encoders have random weights.The reported modalities include text, natural images, videos, medical images, and amino-acid sequences.
Explanation of the gap: Deep neural architectures restrict embeddings to narrow cones, while independently initialized encoders create different cones that separate modalities at initialization.The paper mathematically characterizes the contraction mapping induced by linear layers with ReLU nonlinearities and reports agreement between theory and experiments.
Explanation of the gap: Contrastive learning preserves the modality gap during optimization, with the separation influenced by the temperature parameter in the loss.The paper supports this explanation with theoretical analysis and experiments.
Downstream implications: Modifying the gap distance improves CLIP’s downstream zero-shot performance and fairness.The paper presents changing the gap distance as a simple intervention with downstream benefits.

2 The Cone Effect Induces A Modality Gap

The modality gap arises because encoder embeddings concentrate within narrow cones, even at random initialization, separating representations before modality-specific data or contrastive training can explain the gap. Nonlinear activations drive this concentration, while the gap can persist when encoders process identical data.

Cone Effect: Encoder embeddings occupy an extremely narrow cone, creating separated representation regions and inducing a modality gap already at random initialization.A cosine similarity of 0.56 corresponds to less than 1/2^512 of a unit hypersphere’s surface area in 512 dimensions.
Random Initialization: Randomly initialized models retain the cone effect, with average cosine similarity exceeding that of trained models; the gap therefore does not require different modalities.The same real MSCOCO Caption data were used for both encoders, yet their embeddings remained separated across random initializations.
Activation Functions: Nonlinear activation functions are crucial: deeper nonlinear MLPs rapidly increase average cosine similarity, reaching 0.99 for a 2-layer Sigmoid MLP, whereas linear MLPs show little cone effect.The study used randomly initialized 512-dimensional MLPs with varied nonlinearities or no nonlinearities.
Normalization and Architecture: The cone effect persists despite normalization layers, and the analyzed architectures do not use ReLU as the final layer before embedding extraction.The models include batch normalization or layer normalization, so nonnegative final-layer outputs alone do not explain the observation.

3 Theoretical analysis

The theoretical analysis explains the cone effect by showing that layerwise computation increases cosine similarity and that intermediate-output variance is mainly driven by random initialization. These results account for representations becoming increasingly aligned and initialization-dependent in deep networks.

Each network layer increases cosine similarity: Each feedforward layer increases cosine similarity with high probability when the output layer is sufficiently wide, matching the observed increase across intermediate layers.The theorem applies to a linear transformation followed by ReLU and holds with probability at least 1 − O(1/dout).
Each network layer increases cosine similarity: The monotonicity result requires the input vectors not to be too directionally close, because random bias addition can otherwise slightly reduce cosine similarity.The condition is considered plausible because intermediate-layer output norms are close to one with high probability.
Effect of random initialization: The variance of an intermediate output decomposes into data-induced variance and variance across random weights and biases, with the latter becoming dominant in deep networks.Theorem 2 lower-bounds the initialization-variance fraction by β, the average cosine similarity of previous-layer outputs, which approaches one as depth increases.

4 Contrastive learning preserves modality gap

Contrastive learning preserves the modality gap: at CLIP’s learned temperature, the default gap minimizes contrastive loss, while higher temperatures and mismatched data alter the gap’s strength. Training from different initial gaps shows that both initialization and optimization shape the final separation.

Contrastive objective: CLIP maximizes cosine similarity for N aligned image-text pairs while minimizing it for the N^2 − N incorrect pairs using image-to-text and text-to-image losses.The embeddings are L2-normalized, and τ is a learned temperature parameter scaling the logits; CLIP’s final value is τ = 1/100.
Loss landscape: At τ = 1/100, the default gap distance 0.82 achieves the global minimum of the contrastive loss, whereas closing the gap increases loss.A local minimum also appears when text embeddings are shifted to the opposite, back-to-back position.
Temperature: The gap distance decreases monotonically as temperature increases, with high temperatures significantly reducing or closing the gap and low temperatures preserving it.This result comes from fine-tuning CLIP across six temperatures on MSCOCO Caption with batch size 64.
Mismatched data: With mismatched image-text pairs, the simulation reproduces a temperature-dependent repulsive optimization structure, which disappears when mismatches are removed.Mismatched data therefore contributes to modality-gap formation under low temperatures; such mismatches can arise from difficult captions or annotation errors.
Initialization and optimization: 1.2991 ± 0.0389 and 0.7457 ± 0.0633 were the final gaps from initial gaps of 1.1891 ± 0.0017 and 0.0388 ± 0.0351, respectively.The models were trained on MSCOCO Caption with batch size 64 and τ = 1/100; values are 95% confidence intervals over three runs.

5 Modality Gap Implications

Shifting CLIP embeddings to modify the modality gap improves performance on several downstream zero-shot tasks. Increasing the gap from 0.82 to 0.97 consistently reduces denigration harms across races, with only a 0.0008 top-1 accuracy drop.

Downstream performance: The study evaluates gap changes on CLIP zero-shot transfer using CIFAR10, CIFAR100, EuroSAT, SVHN, and HatefulMemes [28].Task-specific metrics and prompts are provided in Supplementary Table 3.
Downstream performance: Modifying the modality gap by shifting embeddings improves different downstream tasks compared with CLIP’s original unshifted gap.The evaluation covers coarse-grained classification, fine-grained classification, and optical character recognition tasks.
Fairness implications: Increasing the gap from 0.82 to 0.97 consistently reduces denigration harms across all races, while causing only a minor 0.0008 top-1 accuracy drop.Making the gap too small or too large exacerbates crime-related and non-human biases, respectively.

6 Related Work

Prior work established contrastive learning across vision, language, and graphs, while multi-modal models aligned modalities in shared spaces using contrastive pretraining. Existing cone-effect research focused mainly on language models, whereas this work broadens the phenomenon to deep neural networks generally.

Contrastive Representation Learning: Contrastive representation learning brings similar objects closer than dissimilar ones and has succeeded in vision [7] [6] [9], language [16], and graph domains.Despite this success, comprehensive theoretical and empirical understanding of why contrastive learning works remains limited.
Multi-modal Contrastive Representation Learning: Multi-modal models map inputs from different modalities into shared representation spaces [24] and are commonly pretrained with contrastive loss.Contrastive learning was reported as 12× more efficient than generative approaches.
Cone Effect: Prior cone-effect studies examined trained language representations such as BERT and GPT-2 [12] [15], while post-processing and modified objectives alleviate anisotropy and improve downstream performance [15] [16].Existing work attributed the cone effect to optimization under unbalanced word-frequency distributions [15].
Cone Effect: This work broadens cone-effect analysis by demonstrating that the phenomenon holds beyond the previously studied language-representation setting.The paper presents the cone effect as a general phenomenon for deep neural networks.

7 Discussion

The discussion identifies modality gap as arising from model initialization and optimization, and as consequential for downstream performance and fairness. It also highlights temperature’s influence on contrastive geometry and larger gaps’ potential benefits for some applications.

Discussion: The work attributes modality gap to the joint effects of model initialization and optimization, while showing that the gap can affect downstream performance and fairness.The paper’s main objective is to demonstrate the phenomenon and explain how contraction mapping contributes to it.
Discussion: Embedding shifting, simulation, and fine-tuning experiments show that temperature heavily influences the contrastive loss landscape, complementing findings on unimodal representation geometry.Prior work links temperature to uniformity and affinity in unimodal representation spaces.
Discussion: Changing the modality gap affects downstream performance and fairness, with larger gaps helping some fairness and zero-shot learning applications.Systematic analysis of the gap’s impact on applications is identified as an important direction for future work.

Reproducibility Statement

The authors release an open-source implementation that supports reproducing the reported modality gap, analyzing additional cross-modal models, and generating the paper’s figures.

Reproducibility Statement: The open-source implementation enables reproduction of the modality gap, analyses on additional cross-modal models, and generation of the paper’s figures.Available at https://github.com/Weixin-Liang/Modality-Gap.

Checklist · A Contrastive learning preserves modality gap

The checklist confirms that the paper addresses claim scope, limitations, societal impacts, theoretical assumptions and proofs, reproducibility, training details, uncertainty, compute, asset documentation, data privacy, and participant considerations.

Checklist: The authors affirm that the paper’s abstract and introduction accurately reflect its claims and scope, and that limitations are described.They also confirm discussion of potential negative societal impacts and compliance with ethics-review guidelines.
Checklist: For theoretical results, the checklist states that all assumptions are specified and complete proofs are included.
Checklist: For experiments, the authors confirm that code, data, and reproduction instructions are provided, alongside complete training details.
Checklist: The experiments report error bars from repeated runs and document total compute and the resources used.
Checklist: The paper cites existing assets, mentions their licenses, and provides new assets through supplemental material or a URL.
Checklist: The checklist addresses consent, personally identifiable or offensive content, and human-subject requirements, which are marked not applicable where relevant.The paper confirms discussion of consent and data risks, while participant instructions, risks, approvals, and compensation are marked N/A.

A.1 Simulating Mismatched Data · B Modality Gap Implications · B.1 Zero-shot Performance

Simulations show that mismatched data creates a temperature-dependent repulsive structure that disappears when the mismatch is removed. The paper also evaluates how increasing the modality gap affects zero-shot performance across coarse- and fine-grained classification and optical-recognition tasks.

A.1 Simulating Mismatched Data: Mismatched data enables the simulation model to reproduce a temperature-dependent repulsive structure in the optimization landscape.This structure appears in Figure 3 (e-g).
A.1 Simulating Mismatched Data: Removing the mismatch eliminates the repulsive structure in the simulation.The comparison uses Supplementary Figure 9.
A.1 Simulating Mismatched Data: The theoretical simulations place embeddings on a 3D unit sphere, fix image vectors on the equator, and shift text embeddings toward closing the gap.The setup uses r = 1, Δφ = 15°, and θ = 90° for the fixed image embeddings.
B Modality Gap Implications: The modeling identifies low temperature and hard samples or annotation errors as important factors in forming the observed structure.The supplied passage states that both factors are important forming factors.
B.1 Zero-shot Performance: Increasing the modality gap in CLIP can improve downstream performance on several zero-shot learning tasks.Performance versus gap distance is reported in Appendix Figure 10.
B.1 Zero-shot Performance: The evaluated tasks span coarse-grained classification, fine-grained classification, and optical character recognition.They include CIFAR10 and CIFAR100, EuroSAT, and SVHN and HatefulMemes [28].

B.2 Fairness · C The bigger picture: Why studying the modality gap is important

Modifying CLIP’s modality gap consistently reduces racial bias with only a minor accuracy cost, while extreme gap settings amplify distinct bias types. The broader evidence shows that modality gaps arise from narrow-cone representations, persist across training conditions, and affect downstream performance and optimization.

B.2 Fairness: A simple gap-offsetting approach consistently reduces CLIP’s bias across races while causing only a 0.0008 top-1 accuracy drop, from 0.5817 to 0.5739.Making the gap too small or too large instead exacerbates crime-related or non-human biases, respectively.
C The bigger picture: Why studying the modality gap is important: Studying modality gaps fits broader work on neural-network inductive bias, including fitting random labels and optimization preferences for flatter minima or solutions near initialization [13] [27].This context motivates analyzing how initialization and optimization shape multimodal representation geometry.
C The bigger picture: Why studying the modality gap is important: The modality gap appears in pretrained cross-modal models and already exists at initialization, before multimodal training.SVD visualizations show paired image-text embeddings separated across models and during initialization.
C The bigger picture: Why studying the modality gap is important: Randomly initialized neural networks map inputs into narrow cones, with distinct random seeds producing distinct cones that can remain different after ImageNet training.Average pairwise cosine similarity is substantially above zero on COCO-Captions and ImageNet, indicating a restricted effective embedding space.
C The bigger picture: Why studying the modality gap is important: Mismatched image-text pairs create a repulsive loss-landscape structure, whereas correcting the mismatches removes that structure.The simulation shifts text embeddings toward closing the modality gap and compares matched with mismatched pairings.
C The bigger picture: Why studying the modality gap is important: Modifying the modality gap improves zero-shot downstream performance, with task-specific trends as embeddings shift toward the center between image and text embeddings.The reported improvements are statistically significant, and optimal performance can require either increasing or decreasing the gap.
B.2 Fairness: A gap of d = 0.07 exacerbates crime-related bias, whereas d = 1.29 exacerbates non-human bias consistently across races.Larger values indicate more denigration bias under the original CLIP definition.
C The bigger picture: Why studying the modality gap is important: The modality gap persists when CLIP input normalization and normalization layers are disabled, and it arises across embedding dimensions 64, 128, 256, and 512.These results indicate that the phenomenon is not dependent on the tested normalization setup or a particular embedding dimensionality.

D Proofs

The proofs establish concentration and cosine-similarity bounds for randomly initialized ReLU networks, using symmetry, Gaussian moments, and probabilistic inequalities. They also formalize how the trace parameter β captures preceding-layer cosine similarity and constrains the final-layer variance.

Lemma 3: Lemma 3 compares inner products before and after a random affine layer with symmetric, variance-scaled weights and biases, providing the basis for the subsequent ReLU similarity inequalities.The proof relies on independence of W and b, then decomposes each coordinate into positive and negative parts; mixed-sign terms are nonpositive.
Theorem 1: Theorem 1 handles nonpositive input inner products separately because ReLU outputs have positive cosine similarity almost surely, while the main proof addresses the positive-inner-product case.This establishes the scope of the concentration argument used for the theorem.
Theorem 1: Theorem 1 is proved by treating neuron preactivations as i.i.d. Gaussian variables, bounding rectified-Gaussian moments, and applying Chebyshev concentration to obtain a high-probability cosine-similarity bound.The argument uses finite fourth moments and combines two concentration inequalities, yielding failure probabilities of order O(1/(d_outϵ_1^2)) and O(1/(d_outϵ_2^2)).
Theorem 4: Theorem 4 bounds the conditional variance of each final-layer coordinate under Gaussian initialization, assuming unit-norm inputs and tr(Var[h^(L−1)(U) | Θ^(L−1)]) = 1 − β.The proof proceeds through conditional Gaussian variance, symmetry-based variance bounds, and the law of total variance.
Theorem 4: β approximates the expected cosine similarity of independent (L−1)-th-layer outputs and is plausibly close to one at large depth because the cone effect reduces conditional variance.This connects the theorem’s variance parameter to the geometric concentration of deep-network representations.