PaperlayerView exact revision on arXiv

Source-linked AI summary

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition

Pengfei Shao, Jisheng Dang, Jiawen Fang, Ning Liu, Wencan Zhang, Bimei Wang, Jingwen Zhao, Jianhuang Lai, Qi Tian, Tat-Seng Chua

arXiv:2609.06188v1cs.AI

TL;DR

Multimodal affective computing requires integrating textual, acoustic, and visual signals, while adapting LLMs efficiently without losing complementary textual cues. MVFA addresses this with multi-view text-guided fusion and ENQF-based pseudo-tokens for frozen LLMs. Across three datasets and multiple backbones, it achieves strong, stable performance while updating only a small fraction of parameters.

  • Problem

    Existing multimodal methods and lightweight adapters do not sufficiently preserve complementary textual cues before multimodal compression, while full-model fine-tuning is computationally prohibitive.

  • Method

    MVFA constructs max-, mean-, and attention-pooled text views to guide audio-visual fusion, then compresses the fused representation into learnable pseudo-tokens through ENQF.

  • Results

    MVFA achieves strong and consistent performance across CH-SIMS V2.0, MELD, and CHERMA while showing portability across frozen LLM backbones.

  • Takeaways & Limitations

    Multi-view text-guided fusion is presented as an effective, parameter-efficient approach for adapting frozen LLMs to multimodal affective computing.

Abstract

from arXiv · show

Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic, and visual modalities. Although large language models (LLMs) offer powerful language understanding, adapting them to multimodal affective computing remains challenging: full-model fine-tuning is computationally prohibitive, while many existing lightweight adapters fail to preserve rich textual cues during cross-modal fusion. To address these limitations, we propose the multi-view text-guided multimodal fusion adapter (MVFA), a parameter-efficient framework that augments frozen LLMs with strong multimodal reasoning capability. MVFA first constructs complementary text views via max pooling, mean pooling, and attention pooling; these views then guide cross-modal interactions with audio and visual features. The fused multimodal representations are subsequently compressed into a compact set of learnable pseudo-tokens through an Enhanced Q-Former Fusion Module. Using ChatGLM3-6B-base as the primary backbone, we further validate MVFA on LLaMA2-7B and Qwen3-8B to examine its portability across multiple frozen LLM backbones. MVFA is evaluated on three challenging datasets: CH-SIMS V2.0, MELD, and CHERMA. Experimental results demonstrate that MVFA achieves state-of-the-art performance on key metrics while updating only a small fraction of parameters. Specifically, it attains 84.62\% Acc2 and 84.59\% F1 on CH-SIMS V2.0, 67.36\% Acc and 66.03\% WF1 on MELD, and 74.66\% Acc on CHERMA. These findings establish multi-view text-guided fusion as an effective and scalable paradigm for parameter-efficient multimodal LLM adaptation in affective computing. The code is publicly available at https://github.com/Overwhelm1208/MVFA.

I. INTRODUCTION

MVFA addresses limitations in multimodal affective computing by preserving complementary textual cues during fusion and adapting frozen LLMs through a lightweight architecture. It combines multi-view text modeling with compact pseudo-token injection and is evaluated across datasets and backbones.

  • MVFA adapts frozen LLMs to multimodal sentiment analysis and emotion recognition through a parameter-efficient framework.
  • Three complementary text views—max, mean, and attention pooling—guide cross-modal alignment with audio and visual features.
  • The Enhanced Q-Former Fusion Module compresses fused multimodal information into compact pseudo-tokens for injection into frozen LLMs.
  • MVFA is evaluated on CH-SIMS V2.0, MELD, and CHERMA, with portability tested on ChatGLM3-6B-base, LLaMA2-7B, and Qwen3-8B.
  • Experiments report strong and consistent results across Chinese and English datasets while updating only a small fraction of parameters.

A. Conventional Multimodal Fusion for MSA and ERC

Prior work progresses from explicit multimodal fusion toward adaptive interaction and parameter-efficient adaptation. MVFA differs by preserving complementary text views before cross-modal compression while operating with a frozen LLM.

  • TFN models unimodal, bimodal, and trimodal interactions with tensor products, while LMF factorizes high-dimensional fusion tensors for efficiency.
  • MAG, CHFN, and GraphCFC introduce adaptive mechanisms that condition language representations or model cross-modal complementarity.
  • MSE-Adapter is closely related because both methods inject compact pseudo-tokens into frozen LLMs for multimodal affective tasks.
  • MVFA constructs complementary text views, guides cross-modal interaction, and compresses multimodal information through ENQF before token injection.
  • The method uses pre-extracted standardized feature sequences rather than raw speech waveforms, video frames, or text.

B. Modality Encoding

The modality-encoding stage standardizes multimodal inputs and builds contextual audio, visual, and text representations. Its text branch begins by preserving complementary information through multiple aggregation strategies.

  • Audio and visual temporal features are contextually modeled with unidirectional LSTMs and projected into fixed-dimensional global summaries.
  • The MTF module uses raw text embeddings with intermediate representation dimension h = 256 to preserve complementary textual cues for alignment.
  • The text branch constructs three complementary views using different aggregation strategies.
  • The max-pooled text view applies an independent MLP before pooling to capture salient local activations.
  • The mean-pooled view applies an independent MLP and averages across the sequence dimension to summarize overall semantic tendency.

3) Attention-Pooled Text View:

The attention-pooled text view weights transformed tokens with a parameter-free attention mechanism, after which the three views are progressively fused with residual connections.

  • 3) Attention-Pooled Text View:: An independent MLP transforms the text embeddings before each token is weighted by a parameter-free attention mechanism.
  • 3) Attention-Pooled Text View:: The attention score for each token is computed by averaging over the hidden dimension.
  • 4) Progressive Fusion:: The max, mean, and attention views are fused in two stages using residual connections.
  • 4) Progressive Fusion:: Independent MLP parameters allow the aggregation views to learn complementary representations.
  • 4) Progressive Fusion:: The fused text feature interacts with global audio and visual representations through two independent cross-attention operations.

5) Text-Audio Cross-Attention:

MVFA performs text-guided cross-attention with both audio and visual features, then fuses the resulting representations before Enhanced Q-Former processing. The design uses feature enhancement and learnable pseudo-queries to retain salient multimodal information efficiently.

  • Text-Audio Cross-Attention: Text features serve as queries while audio and visual features provide keys and values for separate cross-attention operations.The audio and visual outputs are then combined along the feature dimension.
  • Cross-Modal Fusion: The two cross-attention outputs are concatenated and projected through a two-layer MLP with ReLU activation.The projection returns the fused representation to the original dimension.
  • Enhanced Q-Former Fusion: ENQF applies learnable pseudo-queries through cross-attention to extract context-aware semantic representations from the fused multimodal feature.The default configuration uses four pseudo-tokens with hidden dimension 256.
  • Feature Enhancement: A feature-enhancement subnetwork uses two fully connected layers, GELU activation, and a sigmoid gate to emphasize discriminative channels.The projected feature and enhanced feature are subsequently fused.
  • Enhanced Q-Former Fusion: By avoiding complex self-attention stacking, ENQF reduces parameters and computational overhead while helping queries capture key input information.The module uses explicit feature enhancement to support this extraction.

C. LLM Inference and Loss Function

MVFA combines pseudo-tokens with raw and prompt-augmented text embeddings before passing them to a frozen LLM. The model generates sentiment or emotion labels autoregressively, while optimization updates only MVFA parameters.

  • LLM Input Construction: Pseudo-tokens, raw text embeddings, and prompt-augmented text embeddings are concatenated sequentially to form the unified LLM input.
  • LLM Inference: The frozen LLM generates sentiment or emotion label sequences autoregressively during inference.
  • Loss Function: Gradients backpropagate only through MVFA, while the LLM parameters remain frozen.

A. Experimental Settings

MVFA is evaluated across multilingual multimodal sentiment and emotion datasets using established classification and regression metrics, multiple frozen LLM backbones, and comparisons with reproduced and prior baselines.

  • Datasets and backbones: ChatGLM3-6B-base is the primary frozen backbone, with MVFA additionally instantiated on LLaMA2-7B and Qwen3-8B.The backbone variants are used to examine portability across frozen LLMs.
  • Datasets and backbones: Experiments cover CH-SIMS V2.0, MELD, and CHERMA across Chinese and English multimodal affective-computing settings.The datasets support evaluation across sentiment analysis and emotion recognition tasks.
  • Evaluation metrics: CH-SIMS V2.0 evaluation includes classification metrics Acc2, F1, and Acc2-Weak, plus Pearson correlation, R2, and MAE for regression.Acc2 and F1 use non-positive/positive binarization, while Acc2-Weak uses a ground-truth-score subset.
  • Benchmark results: 84.62% Acc2 and 84.59% F1 are achieved on CH-SIMS V2.0, while MVFA also obtains 0.736 correlation and 0.284 MAE.The reported CH-SIMS results include 77.27% Acc2-Weak and R2 of 45.34.
  • Benchmark results: 74.66% accuracy is achieved on CHERMA, with the best F1 scores on six of seven emotion categories for MVFA-ChatGLM3-6B.MVFA-Qwen3-8B achieves the best result on Sadness, while the ChatGLM3-6B variant leads overall.
  • Benchmark results: 67.36% Acc and 66.03% WF1 are achieved on MELD, exceeding the reproduced MSE-ChatGLM3-6B baseline by 1.48 and 1.36 points.The ChatGLM3-6B variant achieves the strongest overall MELD performance, while LLaMA2-7B remains competitive and Qwen3-8B is weaker.

D. Efficiency Analysis

MVFA combines a very small trainable footprint with ablation evidence that its fusion modules, three text views, and multimodal inputs each contribute to performance.

  • Parameter efficiency: 6.05M–6.26M trainable parameters represent only 0.0968%–0.1002% of the ChatGLM3-6B-base model.Variation across datasets mainly reflects dataset-specific modality-encoder dimensions.
  • Module ablations: Removing MTF or ENQF degrades CH-SIMS V2.0 performance, while removing both causes a further overall drop.The full model achieves the best overall trade-off across metrics, although individual ablations can remain competitive on some regression measures.
  • Module ablations: The full model performs best on MELD and CHERMA, and removing both fusion modules produces the largest degradation.The relative impact of MTF and ENQF varies by task, indicating dataset-dependent contributions.
  • Text-view ablations: Removing any max-, mean-, or attention-pooling branch consistently degrades the main CH-SIMS V2.0 classification metrics.Some single-branch variants remain competitive on individual regression metrics, but the three views provide complementary information overall.
  • Text-view ablations: 67.36% accuracy and 66.03% weighted F1 are recovered by the full MELD model, while removing any text branch lowers performance most when mean pooling is removed.The three text views contribute complementary rather than redundant information.
  • Modality ablations: 47.21% Acc and 35.35% WF1 result without text, substantially below all text-involved MELD variants.Joint fusion of text, visual, and audio modalities reaches 67.36% Acc and 66.03% WF1.

F. Hyperparameter Sensitivity

The analysis identifies moderate ENQF settings as the most effective: four pseudo-tokens and an enhancement factor of 0.3 provide the strongest overall trade-offs on CH-SIMS V2.0.

  • Pseudo-token number n: 4 pseudo-tokens improve every main evaluation metric over n = 2, including Acc2, F1, Acc3, Acc5, Corr, and R2, while reducing MAE.The comparison uses CH-SIMS V2.0 results averaged over five random seeds.
  • Pseudo-token number n: Larger settings of n = 8 or 16 do not yield consistent gains, with n = 16 producing the weakest overall results.The findings suggest that too few tokens limit compression capacity, whereas too many may introduce redundancy.
  • Enhancement factor α: α = 0.3 achieves the highest Acc2, F1, Acc3, and Corr among the tested enhancement factors.The sensitivity analysis keeps all other settings unchanged and evaluates α from 0.1 to 0.5.
  • Enhancement factor α: Smaller enhancement factors lead individual metrics, while increasing α to 0.4 or 0.5 degrades performance on most metrics.α = 0.2 is best for Acc5, whereas α = 0.1 achieves the best R2 and MAE.

G. Qualitative Case Analysis on CH-SIMS V2.0

The qualitative cases show MVFA handling both implicit contrastive sentiment and explicit negative sentiment by combining textual, acoustic, and visual evidence. These examples provide an intuitive complement to the reported quantitative results.

  • Contrastive and figurative sentiment: MVFA correctly handles a challenging case where positive wording contrasts with negative metaphorical sentiment.The example includes “good news” and “my heart is frozen,” requiring joint interpretation across modalities.
  • Contrastive and figurative sentiment: The first case illustrates that the final prediction is not determined by isolated lexical cues alone.Text-guided multimodal fusion combines textual, acoustic, and visual signals in interpreting the utterance.
  • Explicit negative sentiment: Explicit negative cases produce strongly negative predictions aligned with multimodal annotations when non-textual modalities provide consistent supporting evidence.These examples show effective aggregation of consistent cross-modal evidence.
  • Overall qualitative findings: Together, the cases cover direct, implicit, and contrastive affective expressions and offer an intuitive view of multimodal contributions.The qualitative observations are described as consistent with the quantitative improvements in the main experiments.
Loading 2609.06188v1…