UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport

Zonglin Yang; Huilan Ma; Xudan Zheng; Yuejun Xie

Source-linked AI summary

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport

Zonglin Yang, Huilan Ma, Xudan Zheng, Yuejun Xie

arXiv:2609.10224v1Sep 9, 2026cs.CV

TL;DR

Vision-language modality gaps remain difficult to diagnose because distributional mismatch and broken image–caption correspondence are not cleanly separated. UOT-Gap uses frozen embeddings and unbalanced entropic optimal transport to decompose these effects, and its pair-aware residual tracks retrieval degradation while barycentric objective descent can reduce retrieval quality.

Problem
Existing accounts of the modality gap do not resolve its distributional and pairwise contributions to retrieval.
Method
UOT-Gap applies unbalanced entropic optimal transport to frozen embeddings, decomposes distributional terms, and compares observed pairs with soft UOT matching.
Results
Across tested datasets and encoders, the pair-aware residual tracks caption-induced retrieval degradation more consistently than the mean-gap baseline.
Takeaways & Limitations
UOT-Gap supports screening caption quality, tracking modality alignment, and auditing frozen VLM retrieval robustness under low-compute constraints.
Takeaways & Limitations
The scope is limited to frozen global embeddings and 1K retrieval subsets; the pair-aware residual also requires paired image–caption data.

Abstract

from arXiv · show

Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing accounts connect this modality gap to initialization, contrastive dynamics, and information imbalance, while its distributional and pairwise contributions to retrieval remain unresolved. We introduce UOT-Gap, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT). The UOT optimum separates transport, coupling complexity, and marginal mass variation; a complementary pair-aware residual compares observed image-caption pairs with the UOT soft matching. On Flickr8K and COCO-1K with frozen CLIP, OpenCLIP, and SigLIP encoders, caption degradation reduces Flickr8K Recall@1 from 0.559 to 0.003. Across six dataset-model conditions, the pair-aware residual tracks retrieval degradation with mean absolute Spearman 0.973, compared with 0.392 for the mean gap. The association remains stable across five random COCO-1K subsets at $0.954\pm0.026$, with a minimum of 0.943. UOT barycentric updates reduce the transport objective while degrading retrieval, distinguishing geometric objective descent from task improvement. These results establish UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.

1 Introduction

UOT-Gap frames the modality gap as an unbalanced transport residual and adds a pair-aware diagnostic to separate distributional mismatch from broken image–caption correspondence. Using frozen embeddings, it supports training-free analysis of caption quality, modality alignment, and retrieval robustness.

The diagnostic separates transport, coupling, and marginal components from a pair-aware residual comparing observed pairs with the best soft UOT matching.
Unbalanced transport represents modality-specific information asymmetry by penalizing mass deletion or creation rather than enforcing complete mass conservation.
A random caption permutation preserves the text marginal distribution while disrupting pairwise retrieval, isolating pair correspondence from distributional mismatch.
Using frozen embeddings, UOT-Gap screens caption quality, monitors modality alignment, and audits retrieval robustness without encoder fine-tuning.
UOT-Gap formulates the modality gap as the residual of an unbalanced entropic optimal transport problem between frozen image and text embeddings.
The study validates the diagnostic across datasets, encoders, controlled data, random subsets, caption degradation, hyperparameter sweeps, and correction baselines.

2 Related Work

Prior work explains modality gaps through initialization, optimization dynamics, mismatched pairs, and information imbalance. UOT-Gap complements these accounts with a post-hoc distributional and pair-identity diagnostic for frozen vision-language encoders.

Earlier studies connect modality separation to initialization, contrastive optimization, gradient-flow dynamics, mismatched pairs, temperature, and information imbalance.
Vision-language models progressed from cross-modal encoders to large-scale dual encoders and frozen-backbone models with greater alignment capacity.
Optimal transport compares distributions geometrically, while entropic regularization enables fast scaling and unbalanced transport relaxes marginal constraints with divergence penalties.
Multimodal identifiability motivates relaxed mass matching because modality-private factors can remain unresolved while shared factors align.

3 Method

The method applies unbalanced entropic optimal transport to frozen image and text embeddings, decomposes its optimum into distributional terms, and adds pair-aware and barycentric diagnostics. These components distinguish marginal geometry, observed pair identity, and post-hoc geometric correction.

3.1 Embeddings and UOT Objective: The mean gap captures centroid displacement, leaving higher-order geometry, neighborhood structure, and pair identity outside the statistic.
3.1 Embeddings and UOT Objective: The UOT objective combines transport cost, entropic coupling regularization, and penalties for deviations of row and column marginals.
3.1 Embeddings and UOT Objective: The optimized plan decomposes into Cost for cross-modal matching, Coupl for complexity relative to independent matching, and Marg for mass deletion or creation.
3.2 Pair-Aware UOT Residual: Gappair compares observed paired cost with UOT cost, becoming large when observed pairs are worse than the best soft cross-modal matching.
3.2 Pair-Aware UOT Residual: For retrieval degradation, Gappair is the primary UOT statistic, while distributional components should be inspected separately.
3.3 Barycentric Correction: The UOT plan induces a tangent-space barycentric correction on normalized embeddings to test whether matching geometry reduces transport residuals and modality separability.

4 Theory

The theory decomposes the modality gap into transport and marginal variation, proves that missing modality-specific mass incurs unavoidable UOT cost, and establishes a first-order descent direction for UOT barycentric updates.

Mean-gap certificates: Low cosine transport cost certifies small centroid separation, while global mean-gap bounds do not capture local mismatch or missing modality-specific factors.The balanced transport certificate applies to global displacement; the relaxed UOT decomposition addresses additional mismatch sources.
Mean-gap certificates: Transport and marginal deviations jointly certify the classical mean gap, grounding UOT residuals in measurable geometric quantities.The decomposition separates normalized coupling displacement from marginal deviations, with KL terms controlling the latter.
Missing-attribute bound: For a uniformly distributed private attribute with k states and one text-alignable state, the unavoidable private mass is η = 1 − 1/k = 1 − e^−H(U).The lower bound follows by minimizing the pointwise trade-off between transporting and deleting private mass.
Missing-attribute bound: Private visual mass contributes a positive UOT residual through transport cost or marginal KL when captions omit or underspecify the corresponding attributes.Under the missing-attribute assumption, retained mass incurs transport cost while deleted mass incurs a generalized-KL penalty.
Barycentric correction: UOT barycentric updates provide a first-order descent direction for the optimized UOT objective, but retrieval response is left to empirical evaluation.The guarantee relies on the envelope theorem and local stability of the entropic UOT optimum; it does not itself guarantee retrieval improvement.

5 Experiments

Experiments show that pair-aware UOT statistics consistently track caption-induced retrieval degradation across datasets, frozen encoders, controlled shifts, and hyperparameters, while correction experiments separate geometric objective descent from retrieval improvement.

5.2 Retrieval Degradation: 0.973 mean absolute Spearman: Gappair tracks retrieval degradation more reliably than Gapmean's 0.392 across fixed-parameter dataset–model conditions.On Flickr8K with CLIP ViT-B/32, Recall@1 falls from 0.559 for full captions to 0.003 for random captions, while Gappair rises from −0.126 to 0.017.
5.2 Retrieval Degradation: 0.954±0.026 mean Spearman across five COCO-1K subsets: Gappair remains stable, with a minimum of 0.943, as caption degradation reduces retrieval.Full-caption Recall@1 is 0.514 ± 0.009, quarter-caption Recall@1 is 0.078 ± 0.005, and random-caption Recall@1 is 0.001 ± 0.001.
5.3 Synthetic Data and Hyperparameter Sensitivity: Concentration imbalance increases UOT marginal and transport terms even when centroid separation remains modest, whereas the mean gap primarily reflects global displacement.Synthetic normalized Gaussian sphere samples vary mean separation and concentration imbalance independently.
5.3 Synthetic Data and Hyperparameter Sensitivity: 0.947 mean absolute Spearman across 150 hyperparameter–dataset–model combinations, with 142/150 reaching |ρs| ≥0.8.The pair-aware association remains stable over the tested ε and ρ grid, although raw decomposition values vary with those parameters.
5.4 Model Sweep: CLIP and OpenCLIP show the clearest pair-aware relationship with retrieval degradation, while SigLIP remains model-dependent and motivates reporting raw and composite statistics.The composite UOT-Gap score can be weaker for SigLIP because it combines marginal, coupling, and pair-residual terms.
5.5 Post-hoc Correction: 0.829 R@1 after Procrustes versus 0.349 after UOT barycentric correction, despite paired cost falling from 0.681 to 0.324 under the latter.The result distinguishes reduction of the UOT objective from improvement in retrieval performance.

6 Discussion and Limitations

The discussion presents UOT-Gap as a two-level diagnostic separating distributional mismatch from broken pair identity, while bounding its conclusions to frozen embeddings, 1K subsets, paired data, and parameter-sensitive decompositions.

6 Discussion and Limitations: UOT-Gap separates distributional transport, coupling complexity, and marginal mass variation from pairwise mismatch between observed pairs and UOT soft matching.Random caption permutation preserves the text distribution, collapses retrieval, and increases the pair-aware residual.
6 Discussion and Limitations: Pair-aware residuals track caption-induced retrieval degradation more consistently than Gapmean, while distributional UOT terms answer a complementary and more encoder-sensitive question.The discussion recommends component-level reporting: distributional terms for distribution comparison and the pair-aware residual for paired retrieval diagnosis.
6 Discussion and Limitations: UOT barycentric steps reduce paired cost and UOT objective terms while retrieval declines, whereas Procrustes preserves paired supervision and performs better as a retrieval correction.This comparison frames UOT-Gap as an analysis tool and barycentric updates as a diagnostic intervention.
6 Discussion and Limitations: The scope is limited to frozen global embeddings and 1K retrieval subsets; the pair-aware residual additionally requires paired image–caption data.Full COCO evaluation and broader VLM families remain extensions, while UOT hyperparameters affect decomposition values.
6 Discussion and Limitations: Task metrics remain necessary because semantic errors can persist after distributional alignment.The implementation pairs geometric corrections with retrieval evaluation rather than treating objective reduction as task improvement.

7 Conclusion

UOT-Gap interprets the modality gap through unbalanced optimal transport, linking transport and marginal variation to modality imbalance and missing modality-specific mass. Across varied conditions, its pair-aware residual tracks retrieval degradation, supporting low-compute diagnostics for frozen VLMs.

UOT-Gap connects transport and marginal variation to the mean gap and missing modality-specific mass, while UOT barycentric updates provide a first-order descent direction.
The pair-aware residual tracks retrieval degradation across caption conditions, encoder families, hyperparameter settings, and random COCO subsets.The correction study distinguishes geometric objective descent from retrieval improvement.
These results support screening caption quality, tracking modality alignment, and auditing frozen VLM retrieval robustness under low-compute constraints.