Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

1 INTRODUCTION

• Multimodal fusion commonly assumes all modalities are available during training and testing, although visual, acoustic, or textual input may be missing in practice.The introduction gives camera non-coverage, ambient noise, and privacy as examples.Source: §1 · p. 1
• Most existing missing-modality methods assume only one modality is absent, leaving the more general case of multiple uncertain absences insufficiently addressed.The paper notes that such methods may require a new model for each missing-modality case.Source: §1 · pp. 1–2
• TATE uses tag encoding to mark missing modalities and guide attention, with a two-by-two projection pattern mapping features into a common space for joint representation learning.A full-modality pretrained network supervises the encoded vectors before Transformer encoder outputs are used for sentiment prediction.Source: §1 · p. 2
• TATE is reported to achieve significant improvements over several benchmarks on CMU-MOSI and IEMOCAP.The introduction presents this as validation of the model; it gives no specific metric values here.Source: §1 · p. 2

2.1 Multimodal Sentiment Analysis

• Multimodal fusion is commonly organized as early, late, or intermediate fusion, but few of the reviewed fusion models handle partially missing modalities.The approaches combine modalities before classification, aggregate modality scores, or use shared layers to fuse features.Source: §2.1 · p. 2

2.2 Missing Modalities Methods

• Missing-modality approaches are grouped into generative methods, which generate or impute data, and joint-learning methods, which learn representations from inter-modal relations.The review gives auto-encoders and GANs as generative examples, and cyclic translation and Transformer-based reconstruction as joint-learning examples.Source: §2.2 · p. 2
• Most reviewed methods handle only one missing modality and may require a separate model for each case; TATE instead proposes tags for uncertain missing cases and a new common-space mapping method.The stated distinction is the ability to cover multiple missing modalities without training a new model for every case.Source: §2.2 · pp. 2–3

3.1 Problem Definition and Notations

• TATE classifies sentiment under uncertain missing modalities, including cases where visual, acoustic, or textual inputs are absent.The task is to classify sentiment as positive, neutral, or negative.Source: §3.1 · p. 3

3.2 Overall Framework

• The framework masks missing inputs, processes features through pretrained and tag-assisted branches, then uses encoder outputs for classification and auxiliary reconstruction and tag-recovery supervision.The forward similarity loss compares pretrained vectors with encoder outputs.Source: §3.2 · p. 3

3.3 Multi-Head Attention

• Transformer multi-head attention generates contextual representations for each modality, using attention to model intra-modal and inter-modal information flow.The method uses multiple attention heads to obtain information from different semantic spaces.Source: §3.3 · p. 3, §3.6 · p. 4

3.5 Common Space Projection

• A two-by-two projection pattern maps modality features into common vectors, which are concatenated with the encoded tag to form joint representations.The paper motivates this pattern as a more general space than using the same feed-forward projection parameters when there are more than two modalities.Source: §3.5 · p. 4, Figure 4 · p. 4

3.6 Transformer Encoder-Decoder

• A Transformer encoder-decoder processes the joint representations to model long-term dependencies in intra-modal and inter-modal information.The encoder outputs are accessed by multi-head attention and feed-forward networks; the decoder produces decoded outputs.Source: §3.6 · p. 4

3.7 Training Objective

• The training objective includes classification, forward differential, backward reconstruction, and tag recovery losses.Forward loss compares pretrained and encoder outputs; backward loss compares decoder outputs with updated joint representations; tag recovery uses MAE against the last four decoder-output digits.Source: §3.7 · pp. 4–5

Datasets

• Experiments use CMU-MOSI and IEMOCAP, multimodal benchmarks with visual, acoustic, and textual modalities.CMU-MOSI has 2,199 segments from 93 opinion videos; IEMOCAP has five sessions with about 30 videos per session.Source: §4 · p. 5
• CMU-MOSI results use negative, neutral, and positive classes, while IEMOCAP results use negative and positive classes.CMU-MOSI sentiment scores are mapped to three classes; IEMOCAP combines frustration, anger, sadness, fear, and disappointment as negative, and happy and excited as positive.Source: §4 · p. 5

Evaluation Metric

• Model performance is measured using Accuracy and Macro-F1.The model is tuned with five-fold validation and grid search on the training set.Source: §4 · p. 5

4.2 Baselines

• Tables 2 and 3 report baseline results for missing single and multiple modalities, respectively.Source: Table 2 · p. 6, Table 3 · p. 6

4.1 Feature Extraction

• Visual features are extracted with OpenFace 2.0, while BERT-base supplies 768-dimensional text vectors and Librosa-derived audio features are concatenated into 33 dimensions.The visual representations are 709-dimensional and include face, head, and eye-movement features.Source: §4.1 · pp. 5–6

4.2 Baselines

• The comparison set spans autoencoder methods AE and CRA, translation methods MCTN2 and TransM, and missing-modality recovery method MMIN3.Source: §4.2 · p. 6
• TATE is included as the proposed model.Source: §4.2 · p. 7

• With single-modality missing rates from 0 to 0.5, results generally decline as the missing rate increases; TATE is best in other settings except full-modality CMU-MOSI M-F1, about 2.02% below MMIN.The authors also report that the backward decoder can assist the forward encoder and improve overall performance.Source: §4.3 · p. 7
• TATE improves M-F1 by about 0.89% to 3.10% and ACC by about 1.31% to 4.81% over other baselines when one or two modalities are randomly discarded.The authors attribute retained semantic-relevant information to the forward differential loss and tag assistance.Source: §4.3 · p. 7

• The ablation settings include single- and two-modality inputs as well as removing the tag encoding, common space projection, and loss components.Source: §4.4 · p. 7
• Performance drops sharply when text is missing, whereas similar reductions are not observed when the visual modality is removed.The authors suggest text dominates and conjecture that subtle facial changes may limit visual feature extraction.Source: §4.4 · p. 7
• Removing the forward differential module reduces M-F1 by about 4.52% to 6.38% and ACC by about 3.69% to 9.09% compared with the whole model.The authors describe the forward guidance as important and the tag encoding module as a slight improvement.Source: §4.4 · p. 7

4.5 Effects of the Tag Encoding

• At a 30% missing rate on CMU-MOSI, adding tag encoding improves AE and TransM by about 2.54% to 3.69% on M-F1 and 1.04% to 4.08% on ACC.The authors report that tags guide the network to focus on missing modalities.Source: §4.5 · p. 7, Table 5 · p. 7

4.6 Effects of the complete modality

• On IEMOCAP, the gap between full- and incomplete-modality test data is smallest at a 0.3 missing rate and grows above 0.3.The authors link increasing training-time missing samples to harder modality correlation capture and suggest too many absent samples can impair joint representation learning.Source: §4.6 · p. 8, Figure 5 · p. 8

4.7 Multi-classes on IEMOCAP

• M-F1 and ACC decrease as IEMOCAP expands from 2 to 4 to 7 classes, with overall performance dropping sharply at 7 classes.The authors attribute the 7-class drop to confusion among multiple classes and resulting convergence difficulties.Source: Table 7 · p. 8, §4.7 · p. 8
• The IEMOCAP experiments use happy, angry, sad, and neutral for 4 classes, then add frustration, excited, and surprise for 7 classes.Source: §4.7 · p. 8
• The gaps among missing-modality rates are smaller in the 7-class results than in the 2-class and 4-class results.Source: §4.7 · p. 8

4.8 Effects of different losses

• TATE’s JS-divergence and MAE combination achieves the best performance among four loss settings on both datasets.JS divergence alone ranks second; the authors note that MAE’s absolute function is less sensitive to outliers and suits the tag’s binary digits.Source: Table 8 · p. 8, §4.8 · p. 8

4.9 Visualization

• Through missing rate 0.4, the overall joint-representation distribution is similar to the full-modality condition.Figure 6 panels (a)–(e) span full modalities and missing rates 0.1–0.4.Source: §4.9 · p. 8, Figure 6 · p. 9
• Representations become more discrete as missing rates rise, especially above 0.3, and neutral samples are harder to classify.Source: §4.9 · p. 8
• At larger missing rates, outliers widen; at rate 0.5, the decision boundary is closer to them, which the authors suspect reflects absent samples dominating training.The authors also link too many absent samples to the model’s inability to converge.Source: §4.9 · p. 9, Figure 6 · p. 9

5 CONCLUSION

• TATE uses tag encoding to cover uncertain missing cases, a tag recovery loss to supervise joint representation learning, and common-space vectors processed by a Transformer encoder.The final objective loss also directs learning of missing modalities.Source: §5 · p. 9
• The authors identify alternative common-space fusion methods and Transformer structures as future directions for improvement.They currently use the original Transformer sublayer as the basic semantic encoder.Source: §5 · p. 9