PaperlayerView exact revision on arXiv

Source-linked AI summary

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Qun Dai, Liangjian Wen, Jiang Duan, Yong Dai, Dongkai Wang, Maolin Wang, Mingjie Wang, Jianzhuang Liu, He Yan, Zhao Kang

arXiv:2610.12393v1cs.AIcs.LG

TL;DR

Task-relevant synergy can arise only from joint multimodal observation, yet pairwise contrastive learning may fail to preserve it. HRIL models higher-order interactions with a cross-moment tensor, Tucker decomposition, and a synergy-aware regularizer. The paper reports consistent benchmark improvements, with pronounced gains on synergy-heavy tasks.

  • Problem

    Pairwise contrastive learning centers shared information, while task-relevant synergy may be accessible only from joint observation and can be discarded by pairwise-minimal representations with more than two modalities.

    Source: §1 · p. 2

  • Method

    HRIL forms an empirical cross-moment tensor over modality embeddings, applies Tucker decomposition, and regularizes the core to preserve higher-order interaction capacity.

    Source: §1 · p. 2, §3.1 · pp. 4, 6

  • Results

    Experiments on real-world benchmarks and controlled synthetic evaluations show consistent improvements over state-of-the-art methods, with pronounced gains on synergy-heavy tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis identifies higher-order dependence as the information-theoretic carrier of pure synergy and a criterion for when higher-order modeling is necessary.

    Source: §1 · p. 2, §7 · p. 10

    The latent-factor framework assumes shared, modality-specific, and synergistic factors are mutually independent and mediate all task-relevant information.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

1 Introduction

Task-relevant multimodal synergy can emerge only from joint observation, but pairwise contrastive approaches center shared information and may not preserve it. HRIL explicitly models higher-order interactions and reports consistent improvements, especially on synergy-heavy tasks.

1 Introduction

  • Synergy can carry task-relevant information that emerges only from combining modalities and cannot be inferred from any single modality.The paper gives multimodal genre prediction as an example: genre semantics can emerge from the combination of plot text and poster imagery.Source: §1 · p. 2
  • Pairwise contrastive learning under the multiview redundancy assumption primarily captures shared information, and minimal pairwise-sufficient representations can discard synergy with more than two modalities.Prior higher-order methods either lack explicit guarantees for synergy preservation or do not formally connect modeled dependence to task-relevant synergy.Source: §1 · p. 2
  • HRIL models multi-way interactions with an empirical cross-moment tensor, Tucker decomposition, and a regularizer that preserves diverse higher-order interaction patterns.The regularizer discourages degenerate interaction structures during training.Source: §1 · p. 2
  • Experiments across healthcare, affective-computing, and controlled synthetic evaluations show consistent improvements over state-of-the-art methods, with pronounced gains on synergy-heavy tasks.The work also identifies pure synergistic information with a conditional total-correlation difference as a criterion for when higher-order modeling is necessary.Source: §1 · p. 2

2 Theoretical Analysis

The theoretical analysis uses a latent-factor framework to examine when pairwise-minimal representations lose task-relevant synergy, focusing on settings with more than two modalities. It links pure synergy to higher-order statistical dependence through conditional total correlation.

2 Theoretical Analysis

  • The analysis formalizes pairwise contrastive insufficiency for more than two modalities, focusing on minimal sufficient representations and deriving a conditional-total-correlation criterion for higher-order modeling.The paper states that its pairwise-learning analysis concerns minimal sufficient representations.Source: §2 · p. 2

2.1 A Latent-Factor Framework for Multimodal Representations

  • The latent-factor framework assumes shared, modality-specific, and synergistic factors mediate task-relevant information and are mutually independent.The complete multimodal observation is treated as containing the task-relevant information represented by these factors.Source: §2.1 · p. 3
  • A pure synergistic factor contains information given the shared factor but none recoverable from any proper subset of modalities given that factor.The paper characterizes it as task-relevant dependence emerging only through joint multimodal observation.Source: §2.1 · p. 3

2.2 Minimal Sufficient Representation Analysis

  • Under the stated latent-factor and pure-synergy conditions, pairwise-minimal sufficiency leaves an irreducible task-information gap when the synergistic factor is task-relevant.The theorem separates this synergistic gap from a unique-information gap.Source: §2.2 · pp. 3–4
  • Theorem 2 identifies pure synergistic information with the increase in conditional total correlation, which reflects higher-order dependence inaccessible to pairwise objectives.Conditioning on the synergistic factor leaves single- and pairwise marginals unchanged while increasing total correlation by the synergistic information.Source: §2.2 · p. 4

3 Method

HRIL combines multimodal fusion contrastive learning with higher-order interaction modeling and cross-view alignment. It uses a cross-moment tensor and a synergy-aware regularizer to preserve capacity for higher-order dependencies while aligning shared and modality-specific subspaces.

3 Method

  • The final objective combines modality-fusion contrastive learning, synergy-aware regularization, and auxiliary cross-view alignment.The components extract cross-modal task-relevant signals, preserve capacity for higher-order interactions, and stabilize shared and modality-specific subspaces.Source: §3 · p. 4

3.1 Higher-Order Dependence Modeling for the Synergistic Subspace

  • The empirical cross-moment tensor aggregates cross-modal dependencies across interaction orders, and Tucker decomposition yields a core tensor representing their multilinear complexity.The tensor entries estimate mixed moments of modality embeddings.Source: §3.1 · pp. 4–5
  • When shared factors dominate embeddings, low-rank prediction constrains the core tensor; extreme energy concentration signals dominance by a single separable component.Penalizing concentration preserves capacity for higher-order dependence, which carries pure synergistic information.Source: §3.1 · pp. 5–6
  • HRIL projects embeddings into a lower-dimensional subspace before computing the cross-moment tensor and Tucker core for each augmented view.The projection is used to avoid the cost of computing the full-dimensional tensor.Source: §3.1 · p. 6
  • The synergy-aware loss penalizes core-energy concentration above the 1−ξ margin in each view, discouraging one-sparse degeneration and preserving multi-way coupling capacity.ξ controls the minimum energy dispersion.Source: §3.1 · p. 6
  • Contrastive alignment pairs each unimodal representation with the fused representation, while the synergy regularizer preserves the fused representation’s capacity for higher-order interactions.The same contrastive loss can then extract synergistic components; cross-view alignment is computed from unimodal representations in both views.Source: §3.1 · pp. 5–6

3.2 Contrastive alignment for shared and unique subspaces

  • Cross-view agreement and a variance penalty encourage shared-factor consistency while preserving modality-specific variation and discouraging coordinate-wise collapse.The variance threshold γ is 0.5 by default.Source: §3.2 · pp. 6–7

4 Experiments

HRIL is evaluated with linear probing on a controlled synthetic task and diverse Multibench datasets. It outperforms CoMM on the reported Trifeature, trimodal, and bimodal comparisons.

4 Experiments

  • The experiments use linear probing on synthetic and real-world benchmarks, including Multibench datasets spanning different domains and modality combinations.The Trifeature compliance task has a 50% chance baseline.Source: §4 · p. 7, §4.2 · p. 8, §4.1 · p. 8

4.1 Experiments on Synthetic Datasets with Trifeature

  • 5.8% on uniqueness and 10.9% on synergy: HRIL exceeds CoMM and achieves the highest performance on both measures in the Trifeature task.The task evaluates redundancy, uniqueness, and synergy with linear probing.Source: Table 1 · p. 7, §4.1 · p. 8

4.2.1 Experiments with 3 Modalities: UR-FUNNY, MOSI, and MOSEI

  • 0.7% on UR-FUNNY, 1.2% on MOSI, and 0.6% on MOSEI: HRIL outperforms CoMM and all other baselines across the three trimodal datasets.ConFu, which adds a higher-order contrastive loss to Tri_CLIP, yields only limited gains.Source: Table 2 · p. 8, §4.2.1 · p. 8

4.2.2 Experiments with 2 Modalities

  • On bimodal Multibench classification tasks, HRIL improves accuracy over strongest baseline CoMM on all four: 1.6 percentage points on MIMIC, 2.1 on MOSI, 0.6 on UR-FUNNY, and 2.2 on MUSTARD, averaging 1.63 percentage points.Source: Table 3 · p. 8, §4.2.2 · pp. 8–9

5 Ablation Studies

The ablations test loss composition, interaction attribution, and fusion capacity, with reported gains in synergy-related measures.

Loss functions

  • The full objective raises synergy accuracy from 78.0% to 82.3% and uniqueness accuracy from 85.1% to 92.6% over LInfoNCE + Lalign.Redundancy remains nearly saturated.Source: Table 4 · p. 9, §5 · p. 9

Interaction attribution

  • Mean InterSHAP increases from 5.32% to 6.87% with Lsyn across the evaluated datasets.Gains are most pronounced on MOSI and UR-FUNNY, with a smaller mean change on MOSEI.Source: Table 5 · p. 9, §5 · p. 9

Fusion module

  • The linear fusion module fails to capture synergy, whereas HRIL with attention-based fusion substantially improves synergy performance.The result suggests synergy-aware regularization works only when fusion has sufficient capacity for higher-order interactions.Source: Table 6 · p. 9, §5 · p. 9
  • If information in one modality can be linearly approximated from another, the resulting representation is unlikely to encode genuinely synergistic information.Source: §5 · p. 10

6 Related Work

Related work spans pairwise multimodal contrastive learning, methods for higher-order dependence, and tensor-based fusion; HRIL explicitly regularizes cross-moment structure for synergy.

Multimodal contrastive learning

  • Pairwise contrastive frameworks primarily capture shared information, often at the expense of modality-unique features, while explicit synergy modeling remains open [46].HRIL builds on CoMM’s fusion and alignment paradigm by adding an explicit higher-order structural regularizer.Source: §6 · p. 10

Higher-order multimodal dependence

  • Methods including Symile, ConFu [23], M3G, GRAM, and TRIANGLE [8] model joint dependence or alignment, whereas HRIL regularizes cross-moment structure to preserve synergy capacity.Source: §6 · p. 10

Tensor-based multimodal fusion

  • Tensor fusion methods model multiplicative interactions; TFN incurs exponential complexity, while LMF uses low-rank factorization, and HRIL uses Tucker decomposition as a structural regularization prior.Source: §6 · p. 10

7 Conclusion

HRIL targets synergy left by pairwise contrastive learning through regularization of a Tucker-decomposed cross-moment core. Experiments report gains, especially on the controlled synergy task, while the analysis assumes cleanly separated latent factors.

7 Conclusion

  • Under pure synergy, the paper identifies synergistic information with a conditional total-correlation difference recoverable only through higher-order dependence, motivating regularization that preserves multi-way interaction capacity.The proposed framework combines that regularization with multimodal contrastive objectives.Source: §7 · p. 10
  • Experiments on synthetic and real-world benchmarks show consistent gains, especially on the controlled synergy task.Source: §7 · p. 10
  • The analysis assumes cleanly separated latent factors; future work will relax this assumption and extend the framework to large multimodal models.Source: §7 · p. 10

A Limitations and Future Work

The theoretical results rely on pure synergy, while mixed interaction structures occur in real multimodal data; robustness beyond that idealized regime remains open.

Limitations

  • The theory assumes pure synergy, where the synergistic factor has no mutual information with any proper modality subset after conditioning on the shared factor.Real data may mix synergy with residual pairwise dependence, partially redundant cues, or modality-specific shortcuts; systematic robustness study remains open.Source: A Limitations and Future Work · p. 14

Future Work

  • Future work includes extending the framework to approximate or mixed synergy, developing data-dependent diagnostics, and testing larger modality sets and large multimodal models.Source: A Limitations and Future Work · p. 14

B Experimental Details

HRIL’s implementation differentiates through Tucker decomposition and uses a thresholded synergy regularizer, alongside Transformer fusion and fixed projection matrices.

Experimental Details

  • Autograd propagates gradients through the executed HOOI trajectory, with truncated-SVD initialization, a 10^-3 convergence tolerance, and at most 15 iterations.The differentiable Tucker decomposition uses TensorLy’s backend and PyTorch autograd.Source: B Experimental Details · p. 14
  • The projection matrix is created once by QR decomposition of a fixed-seed random Gaussian matrix, cached across views, and kept constant during training.Dataset-specific encoders, augmentations, and latent converters follow CoMM configurations.Source: B Experimental Details · pp. 14–15
  • The synergy hinge activates only when a view’s core-energy concentration exceeds its threshold; implementation clamps ratios and averages the two view-wise hinges.When inactive, the direct hinge gradient is zero, although Tucker-decomposition operations are still evaluated.Source: B Experimental Details · p. 15
  • Fusion uses a Transformer with a prepended learnable [CLS] token: one layer for bimodal settings and two for trimodal settings, each with eight attention heads.For ConFu, the subset fusion module is replicated as a two-layer MLP.Source: B Experimental Details · p. 15

C Dataset Details

The appendix describes the datasets and evaluation measures, then reports interaction-score, hyperparameter, and batch-size analyses. HRIL leads the tested interaction comparisons, while sensitivity results identify effective settings and batch-size behavior.

C.1 Trifeature

  • Trifeature contains 1,000 shape, color, and texture compositions; its standard split uses 800 for training and 200 for evaluation, with augmented instances forming bimodal pairs.The resulting pairs comprise 10,000 training and 4,096 test examples.Source: §C.1 · p. 15

C.2 Multibench

  • The Multibench datasets are systematically de-identified, and the benchmark provides extracted features and labels for each dataset..Source: §C.2 · p. 15
  • Multibench datasets cover clinical diagnosis, sentiment, humor, sarcasm, and robotic contact prediction, using combinations of temporal, tabular, visual, acoustic, textual, depth, and proprioceptive inputs.The described tasks include binary disease classification, sentiment classification, humor and sarcasm detection, and next-step contact prediction.Source: §C.2 · pp. 15–16

C.3 MM-IMDb

  • MM-IMDb is a 23-category multilabel movie-genre task using 25,959 movies; HRIL is evaluated on original posters and plot descriptions with weighted and macro F1.The experiments train on the original image and text inputs rather than extracted MultiBench features.Source: §C.3 · p. 16

D.1 Evaluations with InterSHAP

  • InterSHAP summarizes the share of absolute Shapley Interaction Index attributions due to cross-modal interactions; the rank control penalizes core singular values instead of concentration in one coordinate.Higher InterSHAP scores indicate greater reliance on interaction terms than unimodal effects.Source: §D.1 · p. 16
  • HRIL exceeds the multilinear-rank control by 1.08 and 4.03 percentage points in mean InterSHAP on UR-FUNNY and MOSI, respectively.HRIL has the highest mean InterSHAP score on both datasets; the control slightly raises the no-synergy variant on UR-FUNNY but lowers it on MOSI.Source: Table 7 · p. 17, §D.1 · p. 17

D.2 Hyperparameter Sensitivity

  • A Tucker rank of 24 and projection dimension of 24 each give the highest mean accuracy in their respective sweeps; increasing projection dimension to 32 brings no further mean improvement.Across the margin sweep, mean accuracy varies by 0.3 percentage points on UR-FUNNY and 0.5 on MOSEI, with ξ = 0.5 among the best settings on both.Source: Table 8 · p. 17, §D.2 · p. 17

D.3 Batch Size Robustness

  • Batch size 64 gives HRIL its best performance, and further increases bring no consistent gains; robustness is more pronounced on larger-sample datasets.The authors attribute the trend to more reliable mini-batch estimates of cross-modal moments with larger datasets.Source: §D.3 · p. 17, Figure 3 · p. 18

D.4 Experiments on MM-IMDb

On MM-IMDb, HRIL is evaluated with linear probing on posters and plot descriptions. It exceeds CoMM on both reported F1 measures, while CLIP results indicate that the combined inputs outperform either modality alone.

D.4 Experiments on MM-IMDb

  • For CLIP, combining posters and plot descriptions improves over using either modality alone.The authors interpret this as additional predictive value from the joint input.Source: §D.4 · p. 18
  • On MM-IMDb, under the same-backbone comparison, HRIL achieves weighted and macro F1 scores of 62.30% and 55.44%, exceeding CoMM by 1.01 and 1.65 percentage points, respectively.Table 9 reports linear-probing F1 scores; the experiment applies multilabel classification directly to raw image and text inputs.Source: Table 9 · p. 18, §D.4 · p. 18

D.5 Experiments on Vision&Touch

HRIL improves Vision&Touch contact-prediction accuracy over the strongest listed baseline, with additional training computation but no regularizer use at inference. The appendix also describes the training algorithm and broader-impact considerations.

D.5 Experiments on Vision&Touch

  • 88.6% accuracy: HRIL improves on Cross+Self by 1.0 percentage point and on CoMM by 1.6 points for Vision&Touch contact prediction.Cross+Self is the strongest listed baseline at 87.6%.Source: §D.5 · p. 18, Table 10

D.6 Computational Complexity and Runtime Analysis

  • HRIL’s per-step runtime rises from roughly 90 ms to 156–159 ms versus CoMM, chiefly from added loss computation, especially cross-moment tensor regularization.For batch size B and M modalities projected to r_p dimensions, explicit tensor construction takes O(Br_p^M) time; the added computations are confined to training, and the regularizer is not used at inference.Source: Table 11 · p. 19, §D.6 · p. 19

E Broader Impact

  • The paper identifies clinical decision support as a potential beneficiary of multimodal fusion and calls for privacy, fairness, and computational-sustainability safeguards in downstream deployment..Source: E Broader Impact · p. 19

F Pseudo code

  • Algorithm 1 takes multimodal data, transformations, encoders, batch and Tucker settings, and loss weights as inputs; its training loop applies InfoNCE, alignment, and synergy losses before updating encoders and fusion.The pseudocode includes Tucker decomposition of the cross-moment tensor.Source: F Pseudo code · p. 20

G Proof

This section presents proofs about synergistic information, conditional total correlation, and tensor structure, alongside checklist statements on limitations, reproducibility, ethics, and documentation.

G Proof

  • Theorem 1 shows that pairwise-minimal representations capture at most shared task information, while the total information gap includes unique and strictly positive synergistic terms.The synergistic term is positive under the theorem’s stated hypothesis.Source: G Proof · pp. 20–21
  • Conditional total correlation decomposes by the chain rule into successive conditional mutual-information terms, with the three-modality form valid under permutations.For M = 3, one form is TC(X1, X2, X3 | qr) = I(X1; X2 | qr) + I(X3; X1, X2 | qr).Source: G Proof · pp. 21–22
  • Theorem 2 identifies synergistic information with the reduction in conditional total correlation after conditioning on the synergistic factor.Conditioning on that factor leaves every proper-subset marginal unchanged, including all pairwise marginals when M ≥ 3.Source: G Proof · p. 22
  • Proposition 2 bounds the optimal cross-modal linear predictor’s rank by the shared-subspace dimension and shows reconstruction error vanishes as both shared-to-total energy ratios approach one.Under shared-factor dominance, the embeddings exhibit redundant low-order structure.Source: G Proof · pp. 22–23
  • Lemma 1 shows that redundant low-order structure yields a low-multilinear-complexity approximation of the cross-moment tensor, with at least one Tucker-core mode having bounded effective rank.The rank bound is stated for an approximation of the tensor and its truncated core.Source: G Proof · p. 24
  • Lemma 2 shows that when at least 1 − δ of the Tucker core’s energy is concentrated, the core lies within relative Frobenius error δ of a one-sparse tensor.For small δ, this corresponds to dominance by a single separable component.Source: G Proof · pp. 24–25

NeurIPS Paper Checklist

  • The checklist states that the paper characterizes synergistic gaps for more than two modalities and proposes HRIL to preserve synergy information capacity, with experiments on diverse real-world datasets.The checklist answers yes that the abstract and introduction accurately reflect the paper’s contributions and scope.Source: NeurIPS Paper Checklist · p. 26
  • The authors say limitations are discussed in Appendix A and that experimental analyses and results are in Section 4, with training details in Appendix B.They state that training code and key hyperparameters will be open-sourced after the anonymity period.Source: NeurIPS Paper Checklist · pp. 26–28
  • The authors state that main-text theorems, formulas, propositions, and proofs are numbered and cross-referenced, with key proofs included in Appendix G.Source: NeurIPS Paper Checklist · p. 27
  • For all experiments, the authors report means and standard deviations across five independent runs using random seeds 42–46.Source: NeurIPS Paper Checklist · p. 28
  • The authors report using a single NVIDIA RTX 4090 GPU with 24GB of VRAM for all experiments.Further compute details are in Appendix B.Source: NeurIPS Paper Checklist · p. 29
  • The authors say experiments use publicly available benchmark datasets under their licenses, involve no human subjects or personally identifiable information, and require no IRB approval.They also state that the preprocessed MultiBench datasets are anonymized.Source: NeurIPS Paper Checklist · pp. 29–30
  • The authors state that they address both potential positive and negative societal impacts in Appendix E.Source: NeurIPS Paper Checklist · p. 29
  • The authors say sources and citations for models and frameworks are provided, and new code and dataset assets are documented with training, protocol, and licensing information.Source: NeurIPS Paper Checklist · pp. 30–31
  • The authors state that LLMs were used only for writing and editing, not as a core research component.Source: NeurIPS Paper Checklist · p. 32

Related papers

Loading 2610.12393v1…