PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis

Shanshan Lin, Yuesheng Wu, Chao Chen, Yizhe Yang, Zhihao Chen, Zexian Yang, Xiangwen Liao

arXiv:2608.16201v1cs.LG

TL;DR

Many LLM-based MSA methods coarsely summarize audio and visual signals, risking loss of sentiment-relevant temporal structure. MGSI preserves multiple temporal granularities, refines non-text features with text guidance, and conditions a frozen LLM through compact pseudo-tokens. Across four public benchmarks, it consistently improves frozen-LLM baselines and remains competitive with strong multimodal methods.

  • Problem

    Many MSA methods summarize non-text modalities before fusion or use uniform temporal abstraction, potentially missing sentiment evidence across different time spans.

  • Method

    MGSI encodes audio and visual streams at multiple temporal granularities, applies text-guided refinement and sentiment calibration, then compresses them into pseudo-tokens for frozen-LLM conditioning.

  • Results

    Across four public benchmarks, MGSI consistently improves frozen-LLM baselines and remains competitive with strong multimodal methods.

  • Takeaways & Limitations

    Ablation and sensitivity analyses indicate that the gains reflect complementary effects from temporal modeling, polarity-aware supervision, and adaptive calibration.

  • Takeaways & Limitations

    MGSI relies on pre-extracted audio and visual features, and its small pseudo-token budget creates an efficiency–fidelity trade-off for long or highly expressive inputs.

Abstract

from arXiv · show

Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.

1 Introduction

MSA must jointly model text, audio, and vision because affective meaning emerges from their complementary interactions and evolves across temporal scales. MGSI addresses this challenge by preserving multi-granularity audio-visual structure, refining it with text, and compressing it into pseudo-tokens for a frozen LLM, outperforming frozen-LLM prompting across four benchmarks.

  • 1 Introduction: MSA is challenging because sentiment depends jointly on lexical content, vocal prosody, facial expressions, and body movements, whose interactions can modify one another.
  • 1 Introduction: Existing approaches may compress audio-visual streams too early or use a single temporal abstraction, obscuring short-, medium-, and long-range affective cues before fusion.
  • 1 Introduction: MGSI treats multimodal adaptation as structured temporal-semantic compression, encoding audio and visual streams at multiple temporal granularities before pseudo-token compression for a frozen LLM.
  • 1 Introduction: MGSI further applies text-guided non-text refinement, auxiliary polarity-aware supervision, and adaptive residual calibration to make compressed representations more sentiment-discriminative.
  • 1 Introduction: MGSI consistently outperforms frozen-LLM prompting across four benchmarks and remains competitive with strong multimodal methods.

2 Related Work

Prior multimodal sentiment work addresses temporal dynamics, cross-modal fusion, representation learning, and adaptation of multimodal features to LLMs. MGSI instead organizes audio-visual signals across multiple temporal granularities and refines them before conditioning a frozen LLM, targeting information organization rather than only feature compatibility.

  • Multimodal Sentiment Analysis: Earlier MSA methods model temporal dynamics and cross-modal interactions through tensor fusion, low-rank fusion, cross-modal attention, representation decomposition, self-supervision, and efficient temporal attention.
  • Multimodal Sentiment Analysis: Many existing methods summarize non-text modalities before fusion or use uniform temporal abstraction, risking loss of short-, medium-, and long-range sentiment evidence.
  • LLM-Based Multimodal Sentiment Analysis: Recent LLM-based approaches textualize non-text content or use adapters and mixtures of adapters to map multimodal features into frozen LLMs.
  • LLM-Based Multimodal Sentiment Analysis: Unlike prior adapters that mainly address non-text-to-LLM embedding compatibility, MGSI structures audio-visual information across temporal granularities and applies text-guided sentiment refinement before compression.
  • Robust Multimodal Sentiment Analysis: Robust MSA studies address missing, unreliable, or weakly aligned modalities, whereas MGSI targets temporal and semantic organization before conditioning a frozen LLM rather than incomplete-input prediction.

3 Method · 3.1 Task Definition · 3.2 Overview

MGSI formulates multimodal sentiment analysis as intensity prediction from text, audio, and visual sequences, using text as a semantic anchor and non-text modalities as complementary evidence. Its framework preserves temporal structure, integrates cross-modal sentiment cues, compresses them into pseudo-tokens, and conditions a frozen LLM for prediction.

  • 3.1 Task Definition: MGSI predicts a real-valued sentiment intensity score from utterance-level text, audio, and visual inputs, with positive, zero, and negative values denoting positive, neutral, and negative sentiment.The multimodal input is X = (X(t), X(a), X(v)), where text is tokenized and audio and visual data are feature sequences.
  • 3.1 Task Definition: The method embeds text through the input layer of a frozen LLM and projects each non-text modality into a shared latent space.For each non-text modality q ∈ {a, v}, MGSI applies a projection after extracting text embeddings E(t) = Embed(X(t)).
  • 3.2 Overview: The integration stage combines text-guided alignment, a non-neutral classifier, an adaptive sentiment calibrator, and a multi-scale fusion adapter.These components refine and combine non-text representations before decoding.
  • 3 Method: MGSI encodes audio and visual streams with multi-granularity temporal encoders to preserve sentiment-relevant temporal structure before compression.The framework retains temporal information in non-text modalities rather than immediately reducing them to a coarse representation.
  • 3.2 Overview: MGSI first extracts modality-specific representations from audio and visual streams before performing cross-modal sentiment integration and frozen-LLM decoding.The three stages are modality-specific feature extraction, cross-modal sentiment integration, and frozen-LLM decoding for sentiment prediction.
  • 3 Method: Text serves as the semantic anchor, while audio and visual streams provide complementary sentiment evidence that is compressed into pseudo-tokens for efficient frozen-LLM conditioning.The generated pseudo-tokens are injected into the frozen LLM for sentiment prediction.

3.3 Multi-Granularity Temporal Encoder (MGT)

The multi-granularity temporal encoder captures audio and visual sentiment cues across short-, medium-, and long-range temporal scales. It summarizes and fuses these branches into a temporally enriched representation.

  • 3.3 Multi-Granularity Temporal Encoder (MGT): MGT uses three parallel branches for each non-text modality to model sentiment cues appearing at different temporal ranges.This design targets temporal structure in both audio and visual streams.
  • 3.3 Multi-Granularity Temporal Encoder (MGT): The short-term branch captures local patterns with 1D convolution, while the medium-term branch expands its receptive field using dilated convolution.The dilation rate controls the medium-term receptive field.
  • 3.3 Multi-Granularity Temporal Encoder (MGT): The long-term branch models longer-range dependencies with a Transformer and applies attention pooling to produce a branch summary.Positional encoding supports the Transformer, and attention pooling uses a learnable vector over hidden states.
  • 3.3 Multi-Granularity Temporal Encoder (MGT): The three branch summaries form a three-token sequence, undergo multi-head self-attention, and are mean-pooled into the temporally enriched representation h(q).MeanPool averages the three output tokens after fusion.

3.4 Text-Guided Alignment (TGA)

Text-Guided Alignment uses text as the clearest semantic anchor to refine audio and visual representations before multimodal fusion through modality-specific gating and projection.

  • 3.4 Text-Guided Alignment (TGA): TGA uses text to refine audio and visual representations before multimodal fusion, leveraging text as the clearest semantic anchor.
  • 3.4 Text-Guided Alignment (TGA): The method computes a global text summary and maps it to a modality-specific gate for each non-text modality.
  • 3.4 Text-Guided Alignment (TGA): Each gate modulates its corresponding audio or visual representation through a Hadamard product.
  • 3.4 Text-Guided Alignment (TGA): The aligned audio and visual representations are fused through a linear projection.

3.5 Non-Neutral Classifier (NNC)

The Non-Neutral Classifier sharpens polarity discrimination by applying an auxiliary binary objective to samples whose sentiment magnitude exceeds a threshold. It predicts the resulting labels from h(f)_i using a lightweight MLP and binary cross-entropy loss.

  • Non-Neutral Classifier (NNC): The auxiliary binary objective is defined over non-neutral instances satisfying |y_i| > τ, with larger τ restricting supervision to more clearly polarized samples.
  • Non-Neutral Classifier (NNC): A lightweight MLP predicts the auxiliary binary label from h(f)_i, and the corresponding training objective is binary cross-entropy.

3.6 Adaptive Sentiment Calibrator (ASC)

The Adaptive Sentiment Calibrator refines ambiguous or near-neutral sentiment representations through sample-adaptive residual correction. It uses an input-dependent coefficient while limiting residual scaling to avoid over-correction.

  • ASC targets near-neutral or weakly polarized samples, whose subtler sentiment evidence makes them harder to model.
  • ASC applies an input-dependent correction coefficient to adaptively refine the fused representation for ambiguous sentiment.

3.7 Multi-Scale Fusion Adapter · 3.8 Training Objective

MGSI interfaces its calibrated multimodal representation with a frozen LLM by compressing it into a small set of projected pseudo-tokens. Training uses an autoregressive sentiment-token objective with an auxiliary non-neutral supervision term weighted by λnnc.

  • 3.7 Multi-Scale Fusion Adapter: The adapter applies K parallel MLP branches to the calibrated representation, stacks their outputs, and compresses them with a 1D convolution.
  • 3.7 Multi-Scale Fusion Adapter: The compressed features are projected into the frozen LLM’s embedding space as P ∈ R^M×d, where M is the pseudo-token count and d is its hidden size.
  • 3.7 Multi-Scale Fusion Adapter: All experiments use M = 4 pseudo-tokens, maintaining a small conditioning budget for the frozen LLM.The passage states that this budget balances r, but the supplied text does not provide the remainder of that condition.
  • 3.7 Multi-Scale Fusion Adapter: The frozen LLM receives the pseudo-token representation together with prefix and suffix prompt embeddings, denoted E(pre) and E(suf).
  • 3.8 Training Objective: Training models the target sentiment score as a token sequence s = (s1, ..., sN) using an autoregressive log-probability objective.
  • 3.8 Training Objective: The full training objective includes auxiliary non-neutral supervision, whose contribution is controlled by λnnc.

3.9 Complexity Analysis

MGSI preserves multi-scale temporal information with tractable modality encoders and reduces LLM-side self-attention through pseudo-token compression. With M = 4, the method replaces non-text feature steps before decoding and trains efficiently on MOSI.

  • Temporal Encoder Complexity: Short- and medium-term convolutional branches scale linearly with sequence length, while one lightweight Transformer branch per modality keeps long-term temporal encoding tractable despite self-attention.The encoder preserves multiple temporal scales without using multiple Transformer branches per modality.
  • LLM Interface Complexity: MGSI reduces LLM self-attention from O((L_t + L_a + L_v)^2d) to O((L_t + M)^2d) by compressing non-text features into M pseudo-tokens, typically M ≪ L_a + L_v.This compression is the main efficiency gain at the LLM interface.
  • Empirical Efficiency: With M = 4, MGSI replaces L_a + L_v non-text feature steps with four pseudo-tokens before LLM decoding and requires approximately two minutes per MOSI training epoch on one NVIDIA A30 GPU.These measurements come from historical training logs.

4 Experiments

MGSI is evaluated on four multilingual benchmarks using frozen LLM backbones and standardized multimodal features, metrics, splits, and training procedures. It improves frozen-LLM performance broadly, while ablations and sensitivity analyses support multi-granularity temporal modeling, adaptive refinement, and dataset-dependent fusion choices.

  • Experimental Setup: Experiments use MOSI, MOSEI, SIMS, and SIMS-V2 with official splits, three-seed averages, dataset-specific metrics, and frozen ChatGLM3-6B or Qwen2.5-7B backbones.The same pre-extracted audio and visual features are used across compared LLM-adapter variants.
  • Overall Results: MGSI improves most metrics over frozen-LLM baselines across MOSI, MOSEI, SIMS, and SIMS-V2, with ChatGLM3-6B Acc-2 gains of 25.88, 27.46, 6.74, and 10.06 points, respectively.Relative to direct prompting, ChatGLM3-6B MAE also decreases by 0.406, 0.279, 0.112, and 0.103 on the four datasets.
  • Ablation Study: Removing any core component degrades performance, with ASC most influential on MOSI and MGT contributing most to Acc-2 and F1 on SIMS.The full model provides the strongest overall balance across metrics, while NNC contributes smaller but consistent benefits.
  • Ablation Study: No temporal scale is uniformly optimal: the mid-term branch matters most on MOSI, while short-term affects MOSI Acc-2 and long-term influences SIMS F1 and MAE.These branch-specific effects support retaining short-, medium-, and long-term temporal modeling.
  • Sensitivity and Design Analysis: Sensitivity analyses show dataset-dependent design choices: MOSI favors kernel size 3 and dynamic fusion, whereas SIMS favors kernel size 7 and attention fusion.Dynamic fusion improves MOSI Acc-2 and F1 by 1.31 and 1.38 points, while attention fusion improves SIMS by 3.37 and 4.87 points; moderate λnnc generally offers the best trade-off.

5 Conclusion

MGSI preserves sentiment-relevant temporal structure in audio and visual streams through multi-granularity encoding, text-guided refinement, and compact pseudo-token adaptation before conditioning a frozen LLM. It improves frozen-LLM baselines across four public benchmarks while retaining efficiency, but relies on pre-extracted features and faces an efficiency–fidelity trade-off.

  • MGSI combines multi-granularity temporal encoding, text-guided refinement, and compact pseudo-token adaptation to preserve richer multimodal affective information while efficiently conditioning a frozen LLM.
  • MGSI consistently improves frozen-LLM baselines on four public benchmarks and remains competitive with strong multimodal methods.Ablation and sensitivity analyses attribute the gains to complementary effects including multi-granularity temporal modeling and polarity-aware auxiliary supervision.
  • The framework relies on pre-extracted audio and visual features, while compressing multimodal information into few pseudo-tokens creates an efficiency–fidelity trade-off for long or highly expressive inputs.Future work may explore adaptive temporal-scale selection, uncertainty-aware calibration, and dynamic pseudo-token allocation.
Loading 2608.16201v1…