Source-linked AI summary
Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis
Xiaoran Kou, Jingyi Wu, Peng Sun, Yang Liu, Hong Chen
TL;DR
Text-centered sentiment analysis can miss visual affect, while late fusion and globally shared prompts limit how visual evidence adapts to a sample and layer. VG-TPT injects facial cues into frozen BERT through adaptive prompts routed using text state and visual guidance. It performs competitively on CMU-MOSI and CMU-MOSEI with 2.4M trainable parameters.
Problem
Late-stage fusion limits vision's role in forming text representations, and globally shared prompts do not adapt to sample- and layer-dependent visual guidance.
Source: §I · pp. 1–2Method
VG-TPT uses a co-guided router to compose layer-wise prompts from a trainable prompt bank using the evolving text state and visual guidance feature.
Source: §I · p. 2Results
Takeaways & Limitations
VG-TPT achieves competitive performance with only 2.4M trainable parameters while keeping the visual encoder and BERT backbone frozen.
Source: §I · p. 2
Abstract
from arXiv · showhide
Multimodal sentiment analysis requires effective modeling of both verbal semantics and non-verbal affective cues. A central challenge is to calibrate text-centered sentiment understanding with visual facial evidence in a controlled, adaptive, and parameter-efficient manner. Text usually serves as the semantic anchor, whereas visual cues provide complementary evidence for ambiguous or implicit expressions; however, indiscriminate fusion may introduce visual noise and distort textual semantics. Moreover, fully fine-tuning large visual and textual encoders is costly and prone to overfitting on limited and scenario-dependent MSA benchmarks. To address these issues, we propose Vision-Guided Text Prompt Tuning (VG-TPT), which formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations. VG-TPT injects visual affective cues into a frozen BERT encoder through layer-wise adaptive prompts, rather than relying on late-stage feature fusion or full backbone tuning. A co-guided router composes prompts from a trainable prompt bank according to both the evolving text state and the visual guidance feature, enabling sample-specific and layer-specific modulation. Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines and achieves competitive or superior performance compared with several full-modality methods, while updating only 2.4M trainable parameters. The code is available at https://github.com/ma-tubu/VG-TPT.
I. INTRODUCTION
VG-TPT addresses limits in text-centered multimodal sentiment analysis by using facial affect to calibrate textual representations. It combines layer-wise, co-guided prompts with frozen backbones for parameter-efficient adaptation.
I. INTRODUCTION
- Facial expressions can reveal decisive sentiment that is not explicitly encoded in an utterance's words.The paper frames text as the semantic anchor and facial cues as evidence for refining or overturning its interpretation.Source: §I · p. 1
- Late-stage fusion restricts vision to a parallel auxiliary stream, limiting its role in the formation of textual semantics.The paper argues that visual evidence should intervene in text representations across Transformer layers.Source: §I · pp. 1–2
- Visual guidance reliability varies across samples and textual layers, while globally shared prompts do not accommodate those variations.VG-TPT conditions prompt composition on both the evolving text state and visual guidance.Source: §I · p. 2
- VG-TPT injects facial affective cues into internal text-encoder layers through co-guided prompts composed from a trainable prompt bank.The router uses both the evolving text state and visual guidance to support sample-specific and layer-specific modulation.Source: §I · p. 2
- With 2.4M trainable parameters, VG-TPT outperforms text-only baselines and remains competitive with several full-modality methods.The visual encoder and BERT backbone are frozen.Source: §I · p. 2
II. RELATED WORK
The related work describes multimodal sentiment methods that generally fuse modalities after independent encoding, then positions VG-TPT as text-guided visual prompting. It also distinguishes VG-TPT from globally shared prompt fusion through adaptive, co-guided routing.
A. Multimodal Sentiment Analysis
- Prior MSA approaches commonly encode modalities independently and fuse afterward, whereas VG-TPT injects facial cues into the textual encoder.The related work describes progress from feature-level fusion to attention-based fusion, multimodal Transformers, disentanglement, and dynamic modality weighting,,, [13], –.Source: A. Multimodal Sentiment Analysis · p. 2
B. Prompt Tuning for Multimodal Fusion
- Unlike globally shared prompt fusion, VG-TPT routes prompts using the current text state and visual guidance to enable sample-specific and layer-specific modulation.Prompt tuning freezes pretrained backbones and optimizes a small number of continuous prompt tokens,.Source: B. Prompt Tuning for Multimodal Fusion · p. 2
A. Overview
VG-TPT treats text as the semantic anchor and uses facial affect as guidance to refine text-centered sentiment understanding. It injects visual guidance through adaptive prompts within the BERT-based encoder, then uses the resulting representation for sentiment prediction.
A. Overview
- Text serves as the semantic anchor, while visual facial cues refine and disambiguate text-centered sentiment understanding.A frozen visual encoder extracts a guidance feature that is injected into the BERT-based textual encoder through layer-wise adaptive prompts.Source: A. Overview · p. 2, Figure 2 · p. 3
- The resulting vision-guided text representation is fed to a sentiment prediction head.This avoids directly concatenating visual guidance with the final text feature.Source: A. Overview · p. 2
B. Layer-wise Vision-guided Text Encoding
- Prompts are inserted after [CLS], processed by each Transformer layer, then removed to preserve the original text sequence structure.The prompts interact with textual tokens through self-attention and progressively modulate representations at different semantic levels.Source: B. Layer-wise Vision-guided Text… · p. 3
C. Co-guided Prompt Routing
Co-guided routing determines prompt activation jointly from the current text state and visual guidance. This lets prompt selection adapt to each sample and layer.
C. Co-guided Prompt Routing
- Joint text and visual guidance determine prompt activation, adapting prompt selection to each sample and prompted layer.The current text state reflects what the encoder has captured, while visual guidance supplies complementary affective evidence.Source: C. Co-guided Prompt Routing · p. 3
- The text state and visual feature are projected into separate routing subspaces and combined into a joint routing embedding.The text-side and vision-side embeddings have dimensions dT and dV, respectively, with the joint dimension dr equal to dT + dV.Source: C. Co-guided Prompt Routing · p. 3
- The joint embedding is compared with frozen routing anchors to produce a distribution whose weights set the activation strength of each prompt basis.The number of prompt bases is K, and τ is the temperature coefficient.Source: C. Co-guided Prompt Routing · p. 3
D. Routing-guided Adaptive Prompt Composition
VG-TPT forms layer-specific prompts by softly combining trainable prompt bases according to routing weights and adding a visual instruction token. Together, these components provide sample-specific and layer-specific modulation.
D. Routing-guided Adaptive Prompt Composition
- Routing weights softly activate and combine trainable prompt bases to form an adaptive prompt for each prompted layer.The prompt basis bank is maintained separately for each layer.Source: D. Routing-guided Adaptive Prompt… · p. 3
- The co-guided mechanism uses text state and visual guidance to produce routing scores over the prompt basis bank and compose a layer-specific prompt.The routing scores softly activate different prompt bases.Source: Figure 3 · p. 4
- A projected visual feature is inserted as a visual instruction token alongside the routing-guided adaptive prompt.The final inserted prompt has M + 1 tokens: the adaptive prompt and one visual instruction token.Source: D. Routing-guided Adaptive Prompt… · p. 4
- The adaptive prompt provides sample- and layer-specific modulation, while the visual instruction token injects compact affective evidence into text encoding.Together, they support controlled visual guidance across diverse text-visual affective relations.Source: D. Routing-guided Adaptive Prompt… · p. 4
E. Training Objective and Parameter-efficient Tuning
VG-TPT trains with a regression loss for sentiment intensity and a prompt-balance regularizer, while freezing the visual encoder and BERT backbone. For classification, the task loss can instead be cross-entropy.
E. Training Objective and Parameter-efficient Tuning
- For sentiment-intensity prediction, the task loss is the absolute difference between predicted and ground-truth scores.For classification settings, the task loss can be replaced by cross-entropy.Source: E. Training Objective and… · p. 4
- The prompt activation balance loss addresses possible underuse of prompt bases by favoring uniform activation across bases at the batch level.The loss is averaged over prompted layers, and λbal controls its strength.Source: E. Training Objective and… · p. 4
- VG-TPT freezes the visual encoder and BERT backbone, training only prompt-related and task-specific modules.Trainable modules include the prompt basis bank, co-guided routing projections, visual instruction projection, and sentiment prediction head.Source: E. Training Objective and… · p. 4
- Visual affective cues are injected into text encoding without full-model fine-tuning.The backbones remain frozen during training.Source: E. Training Objective and… · p. 4
A. Datasets and Evaluation Metrics
VG-TPT is evaluated on CMU-MOSI and CMU-MOSEI, using utterances from online videos annotated for sentiment intensity. The evaluation covers regression error, seven-class sentiment, and binary polarity.
A. Datasets and Evaluation Metrics
- The evaluation uses CMU-MOSI and CMU-MOSEI, whose online-video opinion utterances have sentiment-intensity scores from −3 to +3.The study uses the vision-to-text setting, with text as semantic anchor and vision as affective guidance.Source: A. Datasets and Evaluation Metrics · p. 5
- Visual guidance is extracted from the middle frame sampled from each utterance-level video segment.CMU-MOSEI uses 17,016 training, 1,813 validation, and 4,098 test utterances; CMU-MOSI uses 1,283, 214, and 686, respectively.Source: A. Datasets and Evaluation Metrics · p. 5
- Reported metrics are MAE, Acc-7, Acc-2, and F1, covering continuous prediction error, seven-level sentiment, and binary polarity.Acc-7 maps continuous predictions to seven discrete sentiment levels; Acc-2 and F1 evaluate binary polarity prediction.Source: A. Datasets and Evaluation Metrics · p. 5
B. Main Results
On CMU-MOSEI, VG-TPT improves over text-only baselines and is competitive or superior to several full-modality methods; on CMU-MOSI, it leads on Acc-7 and remains competitive on other reported metrics.
B. Main Results
- The full-modality reference methods use acoustic information in addition to vision and text, unlike VG-TPT’s V+T setting.Source: B. Main Results · p. 5
- On CMU-MOSEI, VG-TPT exceeds text-only baselines on Acc-7, Acc-2, and F1, and is competitive or superior to full-modality methods on several metrics.The full-modality methods also use acoustic information, so they are not strictly comparable to VG-TPT’s V+T setting.Source: Table I · p. 5, B. Main Results · p. 5
- On CMU-MOSI, VG-TPT achieves the best Acc-7 and competitive Acc-2, F1, and MAE results among compared methods.Source: B. Main Results · p. 5, Table II
C. Ablation Study
VG-TPT updates a small fraction of model parameters while outperforming full BERT fine-tuning on two reported metrics. Ablations support adaptive, layer-wise prompt updates and routing that uses the evolving text state; the conclusion identifies future extensions.
C. Ablation Study
- Removing adaptive prompts, layer-wise updating, text-state routing, or selective activation weakens CMU-MOSEI performance.The variants use fixed prompts, share prompts across layers, omit the current text state, or uniformly average prompt bases.Source: Table III · p. 5, C. Ablation Study · p. 5
D. Parameter Efficiency
- VG-TPT updates 2.4M parameters, or 1.2% of its total, and outperforms full BERT fine-tuning on Acc-7 and Acc-2.Training updates the prompt basis bank, routing projections, visual instruction projection, and prediction head.Source: Table IV · p. 6, D. Parameter Efficiency · p. 6
V. CONCLUSION
- VG-TPT injects facial cues through layer-wise adaptive prompts and co-guided routing; future work targets audio-visual-text chains and finer temporal visual reasoning.Source: §V · p. 6