PaperlayerView exact revision on arXiv

Source-linked AI summary

Binding Multiple Modalities via Multimodal Wasserstein Barycenter

Xiaole Tang, Jiayi Xu, Xiang Gu, Yan Yang, Jian Sun

arXiv:2609.33800v1cs.LG

TL;DR

Multimodal learning beyond two modalities often uses a specific modality as an anchor, which can produce imbalanced representations rather than a shared semantic space. BaryBind aligns modalities to a Wasserstein barycenter and uses simplex volume for global alignment; it reports balanced performance across tasks and robustness to modality absence and scaling beyond three modalities.

  • Problem

    For three or more modalities, pairwise alignment is challenging, and designated modality anchors can bias the shared representation away from shared semantics.

    Source: §1 · p. 2

  • Method

    BaryBind transports a specific-modality initializer to a multimodal Wasserstein barycenter and aligns embeddings around it using a barycenter-simplex volumetric objective.

    Source: §1 · p. 2

  • Results

    BaryBind reports a 3.1% reduction in the text-to-video/video-to-text retrieval gap versus VAST and balanced state-of-the-art performance across retrieval, generation, classification, and videoQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that barycenter-aligned representations are scalable beyond three modalities and robust in missing-modality settings.

    Source: §1 · p. 2, §5 · p. 10

    Iterative barycenter optimization adds a marginal training-computation overhead relative to simple pairwise alignment, which the authors describe as negligible.

    Source: §B.16 · p. 27

Abstract

from arXiv · show

Multimodal learning beyond two modalities commonly leverages a specific modality (e.g., text) to bind other modalities. However, how to establish a more balanced representation space that approximates shared semantics while respecting the holistic geometry of $n$-modal data remains challenging. In this work, we present BaryBind, which aims to transport the specific modality towards the Wasserstein barycenter (WB) optimized across all modalities and introduces a volumetric alignment objective to establish a unified semantic space around the WB embedding. Specifically, we project specific modalities to the WB, which minimizes the average Wasserstein distances to multimodal distributions and serves as the anchor for subsequent alignment. We then construct a barycenter simplex, whose volume is taken as a similarity metric for global alignment centered at the WB. Experiments show that BaryBind achieves competitive performance in text-video-audio retrieval, classification, videoQA, and cross-modal generation tasks, along with robustness under modality absence and scalability to more than three modalities. Code is released at https://github.com/xl-tang3/BaryBind.

1 Introduction

Pairwise alignment and modality-specific anchors leave open how to represent the shared semantics and holistic geometry of three or more modalities without bias. BaryBind uses a multimodal Wasserstein barycenter as a shared anchor and a simplex-volume objective for alignment.

1 Introduction

  • Modality-specific anchors can bias multimodal spaces away from shared semantics, while pairwise contrastive losses are typically formulated for two modalities.The introduction notes a large text-to-video versus video-to-text Recall@1 gap for VAST.Source: §1 · p. 2
  • BaryBind transports a specific-modality initializer to a Wasserstein barycenter optimized across modalities, then aligns embeddings to that anchor using simplex volume.The barycenter minimizes the average Wasserstein distances to multimodal distributions and is intended to mitigate modality bias while preserving holistic geometry.Source: §1 · p. 2, Figure 3 · p. 3
  • BaryBind develops an optimal-transport duality framework with error bounds and uses barycenter-simplex volume as a global similarity metric for n-modal alignment.Source: §1 · p. 2
  • BaryBind reports balanced state-of-the-art performance across retrieval, generation, classification, and videoQA, with robustness to missing modalities and scalability beyond three modalities.Source: §1 · p. 2

2 Related Works

Prior multimodal representation work extends contrastive alignment from image–text and audio–text to additional modalities, often using a designated anchor. BaryBind instead uses a Wasserstein barycenter as the anchor for a more balanced shared space.

2 Related Works

  • ImageBind aligns modalities to a vision pivot, while later methods extend alignment to language anchors, volume-based losses, pairwise losses, or text-centered supervision.Source: §2 · p. 3
  • BaryBind transforms the specific anchor into a barycenter optimized across multimodal distributions, then uses barycenter-anchored volumetric contrastive loss for global alignment.Source: §2 · p. 3
  • A Wasserstein barycenter minimizes weighted Wasserstein distances to input measures while preserving mass structure and optimal-transport geometry.Prior work estimates high-dimensional barycenters with deep dual formulations, neural OT maps, and energy-guided potentials.Source: §2 · p. 3

3 Method

BaryBind learns a multimodal Wasserstein barycenter as a shared anchor and aligns modality embeddings through the volume of a barycenter simplex. Its training combines barycenter optimization, volumetric contrast, and data-anchor matching.

3.1 Preliminaries

  • The Wasserstein barycenter is a distribution minimizing weighted Wasserstein distances to modality distributions, and BaryBind uses it as the joint representation space.The paper applies this formulation to encoded features in a multimodal latent space.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 Multimodal Wasserstein Barycenter Optimization

  • The multimodal barycenter map is learned through a dual reformulation, with alternating optimization of the map and modality-specific potentials.The initializer is empirically the arithmetic mean of available modality features, and the resulting embedding serves as the alignment anchor.Source: §3.2 · pp. 4–5

3.3 Barycenter-anchored Volumetric Alignment

  • The barycenter simplex is spanned by the barycenter embedding and modality-to-barycenter gap vectors, and its volume reflects alignment to the barycenter and inter-modality discrepancy.Its squared volume is computed from the determinant of the vectors’ Gram matrix.Source: §3.3 · p. 5
  • The BVC loss contrasts small simplex volumes for positive pairs against large volumes for negative pairs to bind modalities to the shared barycenter.The volume is formed from the WB embedding and modality-to-barycenter gap vectors.Source: §3.3 · p. 6

3.4 Training Loss

  • Training combines multimodal Wasserstein-barycenter, barycenter-volume-contrastive, and data-anchor-matching losses, with the barycenter potentials optimized in alternation.Data-anchor matching distinguishes matched from mismatched data pairs.Source: §3.4 · p. 6
  • In the VGGSound visualization, BVC yields compact class-wise clusters around Wasserstein barycenters, unlike scattered unaligned embeddings or modality-separated pairwise-aligned embeddings.Source: §3.4 · p. 6, Figure 4 · p. 6

4 Experiments

The experiments evaluate BaryBind across multimodal classification, retrieval, missing-modality settings, and loss and batch-size ablations. Results report strong zero-shot performance, balanced retrieval, and robustness to missing modalities and moderate batch-size variation.

4.1 Experimental Setup

  • Experiments use a VAST backbone with BERT-B, BEATs, and EVA-CLIP-ViT-G encoders across three- and four-modality datasets and VGGSound.The setup also adds depth to MSR-VTT and VATEX using ChronoDepth and a lightweight head attached to the vision encoder.Source: §4.1 · p. 7

4.1.1 Zero-shot video/audio classification

  • 55.6%/83.4% top-1/top-5 accuracy on VGGSound5K video+audio exceeds VAST's 48.1%/79.6%.BaryBind also raises audio-only Acc@1 from VAST's 40.3% to 45.7%.Source: §4.1.1 · p. 7, Table 1 · p. 7

4.1.2 Cross-modal retrieval

  • 56.1/54.0 R@1 on DiDeMo T2V/V2T under T-VA leads prior methods, while T-VAS sets records on MSR-VTT and VATEX.On MSR-VTT under T-VA, the T2V/V2T performance gap is 2.8 versus VAST's 5.6.Source: §4.1.2 · pp. 7–8, Table 2 · p. 8

4.2 Robustness to Missing Modalities

  • BaryBind preserves robust, graceful degradation when audio is absent during AudioCaps training or video is absent during VGGSound inference.In both settings, the barycenter is optimized using only accessible modalities.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.3 Ablation Studies

  • BaryBind's training loss drops faster and stabilizes earlier than VAST's, while retrieval R@1 stays higher and the T2V/V2T gap smaller.The reported comparison is from training-from-scratch dynamics on MSR-VTT.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Adding MWB improves audio-only classification from 40.3% to 44.3% and V2T retrieval from 43.7% to 48.8%; MWB+BVC yields 55.6% audio Acc@1 and 56.1/53.6 T2V/V2T R@1.The study compares these losses with pairwise cosine contrastive losses; Table 5 reports VGGSound5K Acc@1 and MSR-VTT R@1.Source: §4.3 · p. 9, Table 5 · p. 9
  • Performance remains stable across batch sizes, with only a slight decrease at batch size 16 and highly consistent results above 32.The evaluation uses MSR-VTT retrieval and VGGSound classification under T-VA, training on two A100 GPUs without gradient accumulation.Source: §4.3 · p. 10, Table 6 · p. 10

5 Conclusion

BaryBind aligns modalities to a multimodal Wasserstein barycenter through volumetric alignment, and the paper reports scalability beyond three modalities and robustness to missing modalities. The appendix excerpts describe the barycenter optimization and assumptions used in its error analysis.

5 Conclusion

  • BaryBind aligns modalities to a multimodal Wasserstein barycenter through volumetric alignment, with reported scalability beyond three modalities and robustness under modality absence.The authors identify incorporating reconstruction to support joint understanding and generation as future work.Source: §5 · p. 10

A.1 Proof of theorem 3.1

  • The practical barycenter objective replaces integrals with empirical expectations, using the norm distance between modality features and the barycenter.The barycenter is sampled from a distribution Q, with expectations taken over modality features and barycenter samples.Source: §A.1 · p. 16

A.2 Proof of Theorem 3.2

  • The approximation-gap analysis assumes gk(mk, b) is β-strongly convex with respect to b.The proof separately analyzes two gaps, E1 and E2.Source: §A.2 · pp. 17–18

A.3 Analysis on barycenter simplex volume

The barycenter simplex volume captures multimodal geometry around the Wasserstein barycenter, including both alignment to the barycenter and interactions among modalities. The paper relates smaller volume to stronger alignment and reports a negative correlation with retrieval performance.

A.3 Analysis on barycenter simplex volume

  • For three modalities, the angles measure deviations from the barycenter and the gap between non-anchor modalities; small angles correspond to smaller volume.The gap-vector angle γ reflects structural inter-modality gaps, while mutually orthogonal directions increase volume.Source: §A.3 · pp. 18–19
  • The volume captures each modality’s alignment toward the barycenter while preserving inter-modal interactions, unlike pairwise cosine similarity.It is proposed as a metric for holistic n-modality alignment.Source: §A.3 · p. 19
  • ρ = −0.9844: simplex volume is negatively correlated with downstream retrieval performance.The analysis describes zero-shot MSR-VTT retrieval R@1 as volume varies.Source: Figure 6 · p. 19, §A.3 · p. 19

B.3 Broader Multimodal Understanding and Generation

  • BaryBind addresses broader multimodal representation challenges by using a multimodal Wasserstein barycenter as a shared anchor while retaining complementary structures.The discussion situates this approach in settings involving heterogeneous modalities, granularities, and objectives.Source: §B.3 · p. 21

B.4 Finetuning Cross-modal Retrieval Results

Fine-tuning results report gains over prior models, while the paper distinguishes BaryBind through Wasserstein barycenter optimization and a barycenter-anchored simplex. The reported evaluations also describe performance across tasks and increasing modality counts.

B.4 Finetuning Cross-modal Retrieval Results

  • 60.3 R@1 on MSR-VTT and 72.1 on ActivityNet for T2V under T-VA surpass VAST by +4.5 and +2.3 points, respectively.Under T-VAS, BaryBind reaches 65.7 R@1 on MSR-VTT.Source: §B.4 · p. 21

B.5 Clarification on Distinction from Prior Anchor-based Approaches

  • BaryBind’s WB optimization transports modality-specific distributions toward a barycenter before volumetric alignment, which the paper identifies as its distinction from pure anchor-based alignment.The barycenter is defined as closest to all modality-specific distributions under Wasserstein distance.Source: §B.5 · p. 21
  • The WB-centered simplex is spanned by the barycenter anchor and gap vectors, contrasting with GRAM’s parallelotope and Triangle’s origin-centered triangle.The volume-based contrastive alignment guides paired multimodal data toward a shared representation.Source: §B.5 · p. 21
  • Results across retrieval, classification, and video-QA include smaller retrieval gaps and robustness under missing-modality settings; removing WB leads to consistent performance drops.The ablations identify WB as the primary contributor.Source: §B.5 · pp. 21–22

B.6 Scalability of BaryBind with Increasing Modality Number

  • Recall@1 improves as MSR-VTT inputs expand from T-V through T-VA, T-VAS, and T-VASD, and BaryBind outperforms VAST across configurations.The depth modality is derived using ChronoDepth and added through a lightweight head attached to the vision encoder.Source: §B.6 · p. 22

B.7 Robustness on different WB initializers

The initializer ablation reports that WB-driven optimization improves performance across tested initializations. The normalization analysis finds slightly lower absolute retrieval scores but stable overall performance.

B.7 Robustness on different WB initializers

  • WB-driven optimization boosts vision, text, and T+V+A initializations; text-driven WB is comparable to fused- and video-driven WB on V+A VGGSound classification.The reported Acc@1 values are 55.6%, 55.8%, and 55.2%, respectively.Source: §B.7 · p. 22

B.8 Normalization Strategies for Polytope Volume Metric

  • Normalization slightly reduces absolute MSR-VTT zero-shot retrieval scores, while (V/n!)^1/n remains a reliable, interpretable metric without significantly sacrificing accuracy.The passage attributes the slight reduction likely to a weaker gradient signal from the normalized volume metric.Source: §B.8 · p. 23, Table 12 · p. 23

B.9 Hyperparameter Sensitivity Analysis

The sensitivity analysis finds best retrieval performance at α1 = 1 and α2 = 0.1 on the MSR-VTT validation set. Larger or poorly chosen weights can weaken alignment or instance-level discrimination.

B.9 Hyperparameter Sensitivity Analysis

  • α1 = 1 yields the best MSR-VTT validation retrieval performance; increasing it further leads to decline, possibly weakening instance-level discrimination.The reported trend is that increasing the BVC-loss weight gradually improves performance up to this value.Source: §B.9 · p. 23, Table 13 · p. 24
  • α2 = 0.1 yields the strongest retrieval performance; excessively small or large weights cause suboptimal alignment.The passage links the extremes to underconstrained or modality-biased instance matching.Source: §B.9 · p. 23, Table 14 · p. 24
  • α1 = 1 and α2 = 0.1 deliver the highest multimodal alignment on the MSR-VTT validation set, reflecting complementary strengths of BVC and DAM.Source: §B.9 · p. 24

B.10 Training from Scratch Evaluation

BaryBind remains competitive when trained from scratch, and its optimization and learned WB space show stable adaptation and cross-modal semantic consistency. The analyses also report a moderate training-time increase.

B.10 Training from Scratch Evaluation

  • On MSR-VTT, BaryBind improves over VAST by +6.4% (T2V) and +3.8% (V2T); on ActivityNet, it reaches 30.8% (T2V) and 32.4% (V2T).BaryBind achieves the highest R@1 across the reported settings, outperforming VAST, GRAM, and Triangle.Source: Table 15 · p. 24, §B.10 · p. 24

B.10.1 Effect of Relaxing the Congruence Constraint

  • The soft penalty for the congruence constraint achieves comparable performance with only marginal degradation, while the hard parameterization is described as enabling stable optimization.With η = 10, potential optimization stabilizes after approximately 3,000 iterations despite early fluctuations.Source: §B.10.1 · p. 24, Table 16 · p. 24, Table 17 · p. 25

B.10.2 Stability of WB Map Adaptation under Encoder Distribution Shifts

  • The WB map adapts to evolving encoder distributions, with loss decreasing substantially between 500 and 2,000 iterations and variation shrinking after approximately 3,000.The reported analysis tracks WB mapping loss during encoder fine-tuning.Source: §B.10.2 · p. 25, Table 18 · p. 25

B.11 Computational Overhead Comparison During Training

  • BaryBind adds computation mainly through WB optimization and auxiliary alignment losses, while the increase in per-step training time remains moderate.The comparison was run on 2×NVIDIA A100 80GB GPUs with mixed precision.Source: §B.11 · p. 25, Table 19 · p. 25

B.12 Semantic Consensus in the Learned WB Space

  • Cross-modal retrieval through the WB space returns samples sharing high-level concepts such as cooking and basketball while retaining modality-specific variations.The examples are reported in Tables 20 and 21.Source: §B.12 · p. 25

B.13 More Quantitative Results on Scaling to More Modalities

The evaluation progressively adds modalities from text and video to a five-modality configuration, keeping text as the anchor. BaryBind and VAST both benefit from additional modalities, while BaryBind consistently outperforms VAST across the reported configurations and datasets.

B.13 More Quantitative Results on Scaling to More Modalities

  • The scaling evaluation increases input modalities stepwise from two to five to assess their effects on retrieval performance and alignment quality.Text remains the anchor throughout.Source: §B.13 · p. 25
  • BaryBind consistently outperforms VAST on MSR-VTT and VATEX for both T2V and V2T retrieval across the evaluated modality configurations.Both methods benefit from adding modalities; configurations progress from text and video to text, video, audio, subtitle, and depth, with text as anchor.Source: §B.13 · pp. 25–26, Table 22 · p. 26

B.14 Additional Evaluations on VideoQA and Cross-modal Generation

BaryBind is evaluated on VideoQA and cross-modal generation in addition to its other tasks. It achieves competitive VideoQA accuracy and the generation examples illustrate semantic consistency.

VideoQA

  • BaryBind achieves competitive accuracy on MSRVTT-QA and MSVD-QA and outperforms recent global alignment methods.The authors report that its VideoQA performance highlights representations capturing fine-grained semantics.Source: §B.14 · p. 26

Cross-modal generation

  • The VGGSound generation evaluation covers text-to-image, audio-to-image, and joint text-audio-to-image generation.It uses approximately 200K video clips annotated with 309 sound classes, with 1024 videos reserved for testing.Source: §B.14 · p. 26
  • Qualitative generation examples in Figure 1 illustrate semantic consistency in BaryBind's results.The figure caption describes TA2I generation as preserving both text and audio semantics.Source: §B.14 · p. 26

B.15 Visualization of Top-1 Retrieval Results

This section presents qualitative top-1 retrieval examples and related paper disclosures, including optimization overhead, future work, impact and reproducibility statements, and the checklist’s LLM-usage designation.

B.15 Visualization of Top-1 Retrieval Results

  • BaryBind’s top-1 text-to-video examples show more relevant matches than two strong baselines across scenes and emotional queries.Examples include a beach party matched to semantic and acoustic mood, gameplay matched using visual and subtitle cues, and a fearful cat-dog chase for “I’m scared.”Source: §B.15 · pp. 26–27, Figure 8 · p. 28

B.16 Limitations and Future Work

  • The iterative barycenter optimization adds a marginal training-compute overhead over pairwise alignment, which the authors call negligible relative to retrieval gains.The authors point to Table 19 for this cost and the retrieval-accuracy gains.Source: §B.16 · p. 27
  • Future work includes integrating BaryBind into multimodal LLMs and adding reconstruction for multimodal understanding and generation.The authors aim to explore a balanced token space not centered on language, potentially addressing hallucinations and supporting perception beyond linguistic reasoning.Source: §B.16 · p. 27

B.17 Impact Statement

  • The authors describe potential positive impact through more efficient and robust multimodal understanding, and state that they know of no notable ethical or societal risks.They frame the work as supporting responsible development of scalable, balanced multimodal learning methods and theories.Source: §B.17 · p. 27

NeurIPS Paper Checklist

  • The checklist reports that data are publicly available, code is not included in the supplement, and error bars are omitted because experiments use fixed seeds and deterministic configurations.The authors say implementation details and the algorithm are provided elsewhere in the paper and appendix for reproducibility.Source: NeurIPS Paper Checklist · p. 31

16. Declaration of LLM usage

  • The authors answer N/A for LLM usage, stating that LLMs are not an important, original, or non-standard component of the core method.The checklist explains that N/A means the core method does not involve LLMs in those roles.Source: NeurIPS Paper Checklist · p. 35

Related papers

Loading 2609.33800v1…