PaperlayerView exact revision on arXiv

Source-linked AI summary

I$^2$-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis

Xiaoting Xie, Feng Qiao, Xianwei Zhang, Hailong Chen, Kaiwen Zheng, Joemon M. Jose, Xuri Ge, Yujun Li

arXiv:2610.31087v1cs.CV

TL;DR

Multimodal fusion can make it difficult to distinguish a modality’s standalone predictive utility from its interaction-conditioned utility, which need not match. I2-KAN keeps intra-modal and pairwise inter-modal information in separate pathways, refines them, and fuses them with structured KANs. Across three benchmarks, analyses show that weak standalone utility does not necessarily imply weak interaction-conditioned utility.

  • Problem

    Fusion methods can make it difficult to trace whether predictive utility comes from standalone modalities or pairwise interactions, although a modality’s utility in isolation may not reflect its utility in multimodal contexts.

    Source: §I · p. 2

  • Method

    I2-KAN separately maintains intra-modal and pairwise inter-modal pathways, refines each with TDL-GDB, and fuses them through structured KANs that retain pathway associations.

    Source: §I · p. 2

  • Results

    Across CMU-MOSI, CMU-MOSEI, and CH-SIMS v2, analyses show that low standalone predictive utility does not necessarily imply low interaction-conditioned predictive utility.

    Source: §I · p. 2, §V · p. 10

  • Takeaways & Limitations

    The findings support characterizing multimodal utility from both standalone and interaction-conditioned perspectives.

    Source: §I · p. 2

Abstract

from arXiv · show

Multimodal sentiment analysis (MSA) aims to infer sentiment by integrating linguistic, acoustic, and visual information. However, many fusion methods integrate modality-specific representations and cross-modal interactions into joint representations, making it difficult to determine whether predictive utility arises from a modality itself or from its interactions with others. More fundamentally, a modality's standalone predictive utility may not faithfully reflect its interaction-conditioned predictive utility. To address this issue, we propose I$^2$-KAN, an interpretable multi-pathway framework that explicitly maintains intra-modal and pairwise inter-modal information in separate structural pathways. Each pathway is refined by a Task-Decoupled Linear-Gated Denoising Bottleneck (TDL-GDB) to learn compact and task-relevant representations, followed by structured Kolmogorov-Arnold Network fusion that retains explicit pathway associations and supports multi-level interpretation. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 demonstrate strong overall performance. Ablation and interpretability analyses consistently show across the three benchmarks that low standalone predictive utility does not necessarily imply low interaction-conditioned predictive utility: a modality that appears weak in isolation can remain informative through pairwise interactions. These findings highlight the interaction-dependent nature of multimodal utility and suggest that modality utility should be characterized from both standalone and interaction-conditioned perspectives.

I. INTRODUCTION

Multimodal sentiment methods can make it difficult to distinguish standalone modality contributions from pairwise interactions, even though a modality weak in isolation may remain useful through interactions. I2-KAN preserves these information sources in separate pathways, refines them for task-relevant prediction, and supports multi-level interpretation.

I. INTRODUCTION

  • Prior fusion methods often use cross-modal relations as intermediate representations, making standalone modality and pairwise interaction roles difficult to trace in a unified decision structure.The paper also notes that existing interpretability approaches mainly provide modality-level importance or analyze nonlinear fusion behavior.Source: §I · p. 2
  • Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 report competitive performance and show that low standalone utility need not mean low interaction-conditioned utility.Interaction-pathway and component-wise ablations support the proposed representation, TDL-GDB, and KAN-based fusion.Source: §I · p. 2
  • I2-KAN preserves standalone and pairwise inter-modal information in separate pathways, enabling both forms of utility to be modeled and analyzed together.The paper reports that these forms of predictive utility can differ substantially across three benchmarks.Source: §I · p. 2
  • TDL-GDB combines denoising compression, an information-preserving shortcut, and task-specific gated projections to retain complementary information for different sentiment objectives.It learns compact, task-relevant pathway representations before global fusion.Source: §I · p. 2
  • Structured KAN fusion retains pathway associations and supports pathway-level, compressed-feature, and functional-edge analyses of information contributing to the final decision.These analyses trace where sensitivity is concentrated, which compact features are sensitive, and how features are nonlinearly transformed.Source: §I · p. 2

II. RELATED WORK

Prior MSA work has developed joint fusion, modality-specific representations, and explicit relational modeling, while interpretability methods have examined modality importance and interaction effects. I2-KAN distinguishes itself by retaining intra-modal and pairwise inter-modal pathways through refinement and nonlinear fusion, enabling their roles to be traced at multiple levels.

A. Multimodal Fusion and Interaction Modeling

  • Fusion research progressed from tensor and recurrent methods to cross-modal attention, then explored modality-specific representations and explicit intra- and inter-modal relational structures.Examples include TFN and LMF, MulT, MISA, Self-MM, HGraph-CL, Semi-IIN, TEMSA, and I2RL.Source: A. Multimodal Fusion and Interaction… · pp. 2–3

B. Modality Imbalance and Robust Representation Learning

  • Modality-imbalance and robust-learning methods address unequal information density, noise, optimization difficulty, and redundant or unstable information, but do not explicitly tailor preservation strategies to separate standalone and interaction pathways.The paper positions TDL-GDB as independently refining each pathway and using task-decoupled gated projections to preserve task-relevant information before global fusion.Source: B. Modality Imbalance and Robust… · p. 3

C. Interpretability in Multimodal Sentiment Analysis

  • Existing interpretability approaches provide modality scores, interaction decompositions, or KAN edge mappings, whereas I2-KAN retains pathway associations to trace standalone and pairwise information at pathway, feature, and functional-edge levels.KANs replace scalar connection weights with learnable univariate functions on network edges; KAN-MCP uses original modalities as inputs.Source: C. Interpretability in Multimodal… · p. 3

III. PROPOSED METHOD

I2-KAN organizes modality-specific representations into three intra-modal and three pairwise inter-modal pathways, independently refines them, and fuses the compact representations while retaining their source associations. Pairwise concatenation preserves modality pairs for later pathway-specific nonlinear learning rather than treating concatenation itself as the interaction.

A. Overall Framework

  • The framework processes multimodal evidence through feature extraction, semantic pathway construction, TDL-GDB refinement, and Interaction-Aware KAN Fusion, which produces sentiment predictions while preserving pathway associations.The six compact pathway representations are integrated in a fixed pathway order.Source: A. Overall Framework · p. 3, Figure 2 · p. 4

B. Feature Extraction and Semantic Pathway Construction

  • Textual, acoustic, and visual encoder outputs are organized into three intra-modal and three pairwise inter-modal semantic pathways.The intra-modal pathways retain single-modality sources, while pairwise pathways concatenate the corresponding modality representations.Source: B. Feature Extraction and Semantic… · p. 4
  • Pairwise concatenation preserves paired modality sources; each pair is then processed by its own TDL-GDB to learn pathway-specific nonlinear interactions before global fusion.The construction introduces no additional learnable interaction operator at the concatenation stage.Source: B. Feature Extraction and Semantic… · p. 4

C. Task-Decoupled Linear-Gated Denoising Bottleneck

TDL-GDB independently refines each semantic pathway into compact representations while decoupling continuous, polarity, and fine-grained sentiment objectives. It combines denoising compression, an information-preserving shortcut, and task-specific gated projections before global fusion.

  • Each semantic pathway has an independently parameterized TDL-GDB, with task branches sharing compressed candidates before task-specific selection.Source: C. Task-Decoupled Linear-Gated… · p. 4
  • TDL-GDB normalizes pathway inputs and applies Gaussian perturbation during training, with perturbation disabled at inference.Source: C. Task-Decoupled Linear-Gated… · p. 4
  • A nonlinear denoising encoder produces a compact representation, while a parallel linear shortcut preserves a direct projection of the normalized input.The nonlinear representation captures noise-robust information; the shortcut provides an information-preserving projection.Source: C. Task-Decoupled Linear-Gated… · p. 5
  • Independent task-specific gates select between the nonlinear and shortcut representations for regression, binary polarity, and fine-grained sentiment tasks.Source: C. Task-Decoupled Linear-Gated… · p. 5
  • Task-specific representations receive pathway-level prediction supervision, while the regression representation also uses reconstruction and sparsity regularization.The regression-specialized representation is forwarded to KAN fusion; binary and fine-grained branches provide auxiliary supervision during training.Source: C. Task-Decoupled Linear-Gated… · p. 5

D. Interaction-Aware KAN Fusion

I2-KAN fuses six refined intra-modal and pairwise inter-modal pathways while retaining their source associations. Its structured KAN fusion supports tracing pathway contributions and examining learned nonlinear transformations.

  • The six refined pathway representations are concatenated in a fixed order, preserving each representation’s source association through nonlinear fusion.Source: D. Interaction-Aware KAN Fusion · p. 5
  • Unlike MLPs with scalar connection weights, KAN assigns each edge a learnable univariate function, while pathway assignments remain explicit during fusion.Source: D. Interaction-Aware KAN Fusion · p. 5
  • The final KAN prediction is supervised by the continuous sentiment label, with pathway-level objectives providing auxiliary supervision during training.Source: D. Interaction-Aware KAN Fusion · p. 5
  • Retained pathway associations support pathway-level, compressed-feature, and functional-edge analyses of how features transform toward the final prediction.Source: D. Interaction-Aware KAN Fusion · p. 6

A. Datasets and Evaluation Metrics

The experiments use CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 with their stated official or standard splits. Evaluation uses classification and regression metrics matched to each dataset’s sentiment labels.

A. Datasets and Evaluation Metrics

  • CMU-MOSI contains 2,199 English opinion utterances with sentiment scores from −3 to +3, split into 1,284 training, 229 validation, and 686 test utterances.The standard split is used.Source: A. Datasets and Evaluation Metrics · p. 6
  • CMU-MOSEI is an English multimodal sentiment dataset, and the standard partition contains 16,326 training, 1,871 validation, and 4,659 test utterances.Only sentiment annotations are considered.Source: A. Datasets and Evaluation Metrics · p. 6
  • CH-SIMS v2 is a Chinese multimodal sentiment dataset with continuous scores from −1 to 1 and an official split of 2,722 training, 647 validation, and 1,034 test samples.Source: A. Datasets and Evaluation Metrics · p. 6
  • CMU-MOSI and CMU-MOSEI use Acc7, Acc2, weighted F1, MAE, and Corr, while CH-SIMS v2 uses Acc3, Acc2, weighted F1, MAE, and Corr.Acc7/Acc3, Acc2, F1, and Corr are better when higher; MAE is better when lower.Source: A. Datasets and Evaluation Metrics · p. 6

B. Implementation Details

The implementation uses dataset-specific modality encoders and model dimensions, with a three-dimensional compact representation for every semantic pathway. Results for I2-KAN and its variants are averaged over five random-seed runs.

B. Implementation Details

  • Each semantic pathway has compact dimension 3, while TDL-GDB intermediate dimensions are 1024, 518, and 1024 and KAN hidden-node counts are 4, 4, and 3 for CMU-MOSI, CMU-MOSEI, and CH-SIMS v2, respectively.Source: B. Implementation Details · p. 6
  • CMU-MOSI and CMU-MOSEI use DeBERTa-v3-base, COVAREP, and FaceNet512; CH-SIMS v2 uses Chinese BERT, OpenSMILE, and OpenFace for text, audio, and vision.Temporal mean pooling is applied when needed to obtain utterance-level representations.Source: B. Implementation Details · p. 6
  • All models use AdamW with weight decay 0.01, and I2-KAN and variant results are averaged over five runs with different random seeds.Batch sizes, training epochs, learning rates, and auxiliary supervision differ by dataset.Source: B. Implementation Details · p. 7

C. Performance Comparison

I2-KAN performs strongly across the three benchmarks, while pairwise-pathway ablations show consistent gains from retaining all interactions. The analyses also characterize dataset-specific pathway behavior and how information is traced through the structured representation and KAN.

C. Performance Comparison

  • I2-KAN achieves the best results across all five metrics on CMU-MOSI, the best Acc2, F1, MAE, and Corr while remaining competitive in Acc7 on CMU-MOSEI, and the best Acc2, F1, MAE, and Corr on CH-SIMS v2.The reported results span classification and regression metrics across the three benchmarks.Source: C. Performance Comparison · p. 7

D. Analysis of Pairwise Interaction Pathways

  • Adding all three pairwise pathways improves over the intra-modal-only configuration: Acc7 and Acc2 rise by 5.4 and 3.5 points on CMU-MOSI, 5.2 and 1.5 on CMU-MOSEI, and Acc3 and Acc2 by 6.3 and 4.2 on CH-SIMS v2.MAE also decreases consistently across all three datasets.Source: D. Analysis of Pairwise Interaction… · p. 7
  • The most effective individual interactions vary by dataset and metric, while the Full configuration provides the most consistent overall performance across the three benchmarks.Text-centered pathways generally provide larger gains than VA on CMU-MOSI, whereas VA is comparatively more effective on CH-SIMS v2.Source: D. Analysis of Pairwise Interaction… · pp. 7–8

E. Interpretability Analysis

  • The six-pathway structure supports tracing prediction dependence from pathway-level information through compact features to their nonlinear propagation through KAN.Each pathway contributes three compressed dimensions to the KAN input.Source: E. Interpretability Analysis · p. 8

1) KAN Functional-Edge Analysis:

KAN functional responses concentrate in selected routes, and pathway sensitivity differs across datasets. Within pathways, compressed dimensions also have differentiated predictive roles rather than a fixed pattern.

1) KAN Functional-Edge Analysis:

  • Learned KAN responses concentrate in selected nonlinear routes, with stronger hidden-to-output responses through H3 and H4 on CMU-MOSI, H2 and H4 on CMU-MOSEI, and H1 and H2 on CH-SIMS v2.Because models are trained independently, hidden-node indices should not be interpreted as semantically aligned across datasets; routes adapt to dataset-specific multimodal characteristics.Source: 1) KAN Functional-Edge Analysis · p. 8

2) Pathway-Level Predictive Sensitivity Analysis:

  • Low sensitivity of standalone A can coexist with considerably higher sensitivity for TA and VA, while dominant pathways differ across datasets.CMU-MOSEI has normalized sensitivity 0.398 for T; CMU-MOSI is strongly interaction-oriented, and CH-SIMS v2 is more sensitive to TV and VA.Source: 2) Pathway-Level Predictive… · pp. 8–9

3) Compressed-Feature Predictive Sensitivity Analysis:

  • Compressed dimensions within a pathway show differentiated predictive-sensitivity patterns that vary by pathway and dataset rather than following a fixed pattern.Text-related dimensions are more prominent on CMU-MOSEI, while interaction-related dimensions are more prominent on CMU-MOSI and CH-SIMS v2; dimension indices do not have fixed semantic meanings across models.Source: 3) Compressed-Feature Predictive… · p. 9

F. Ablation Study

On CMU-MOSEI, removing pairwise pathways, replacing TDL-GDB, or replacing KAN each lowers performance in the reported ablations. The conclusion emphasizes that standalone predictive utility may not reflect interaction-conditioned utility.

F. Ablation Study

  • Removing the three pairwise pathways leads to clear performance degradation across all metrics on CMU-MOSEI.The comparison retains the three intra-modal pathways with TDL-GDB and KAN.Source: F. Ablation Study · pp. 9–10
  • Replacing TDL-GDB with a conventional bottleneck degrades performance, indicating that dimensionality reduction alone is insufficient for these heterogeneous pathways.The authors attribute the comparison to task-decoupled denoising and gated information selection.Source: F. Ablation Study · p. 10
  • Replacing KAN with a comparable-width MLP lowers performance even when the six pathways and TDL-GDB representations are retained.The authors suggest KAN supplies nonlinear fusion while retaining the functional interpretability analyzed in Section IV-E.Source: F. Ablation Study · p. 10

V. CONCLUSION

  • I2-KAN’s multi-level analyses show that standalone predictive utility does not necessarily reflect interaction-conditioned predictive utility.The conclusion suggests considering modality utility from both standalone and interaction-conditioned perspectives.Source: §V · p. 10
Loading 2610.31087v1…