Fine-Grained Visual Preprocessing and Dual-Stream Temporal Modeling for Multimodal Sentiment Analysis on Social Media

• The study evaluates its framework on CH-SIMS v2.0S, whose natural conversational clips include varied speakers, poses, lighting, and spontaneous expressions.Source: §1 · p. 2
• NAPS removes raw-video noise at the source using face tracking, identity representation, and normalized lip-motion analysis to localize target speakers and improve visual-input reliability.Source: §1 · p. 2
• DS-TANet jointly models static facial appearance and dynamic motion with EfficientNetB2, RAFT optical flow, motion-guided attention, and GRU-based temporal modeling.Source: §1 · p. 2
• DS-TAFNet combines DS-TANet with a prompt-tuned MacBERT text encoder and compares fusion strategies under small-dataset conditions.Source: §1 · pp. 2–3

2.1. Multimodal Sentiment Analysis

• On weak-sentiment samples, visual modality reaches Acc2_weak of 68.79%, exceeding textual modality's 62.49%; CH-SIMS v2.0 increases linguistic ambiguity to encourage non-text cues.Source: §2.1 · p. 3

2.2. Visual Affective Feature Extraction

• Single-frame features miss continuous facial dynamics, while RAFT's use in temporal facial affect modeling remains limited, motivating dual-stream visual designs.Source: §2.2 · pp. 3–4

2.3. Multimodal Fusion Strategies

• Fusion ranges from feature concatenation to prediction combination and cross-modal attention, but complex fusion may lack stable gains when data or unimodal features are weak.Source: §2.3 · p. 4

2.4. Chinese Pretrained Language Models

• MacBERT uses a correction-oriented synonym-substitution objective to reduce pretrain–finetune input mismatch and is presented as a robust Chinese MSA backbone.Source: §2.4 · p. 4

2.5. Research Gaps

• The paper identifies three gaps: neglected visual preprocessing, static-frame modeling that underuses facial dynamics, and limited fusion-strategy analysis on small Chinese datasets.Source: §2.5 · p. 5

3.1. CH-SIMS v2.0S Dataset

• 3,866 valid samples remain after excluding 533 neutral samples and four clips with only profile-face views; the task is positive–negative binary classification.Balanced sampling and Focal Loss address the slight excess of negative samples.Source: §3.1 · p. 5

3.2. NAPS Preprocessing Pipeline

• NAPS is a seven-stage automated pipeline with low-light recovery that converts raw videos into 224 × 224 speaker-centered facial clips.Source: §3.2 · p. 6
• Face tracking combines spatial overlap and appearance embeddings, then merges fragmented identity trajectories and retains up to five candidates.Face embeddings support cross-frame association, while candidate selection ranks merged identities by frame presence.Source: §3.2 · pp. 6–7
• NAPS filters profile-face candidates using mouth-corner width and classifies scenes by candidate-frame overlap, invoking active-speaker identification when overlap exceeds 30%.Source: §3.2 · pp. 8–9
• To address sensitivity to mouth shape and head pose, NAPS normalizes vertical lip distance by mouth-corner width using facial geometry alone.Source: §3.2 · pp. 9–10
• Norm-Soft reduces variation from individual mouth shapes and corrects a misidentification when unnormalized lip-motion scores are too close.Samples 1–3 had identical identification results under both variants when sufficient frontal frames were available; in Sample 4, Norm-Soft separated two fully frontal candidates.Source: §3.2 · p. 11
• 90.1% identification accuracy across 81 valid multi-person samples, 3.7 percentage points above Abs-Hard’s 86.4%, validates Norm-Soft’s normalized lip-motion analysis and pose-aware weighting.Table 2 compares the two variants. In a pose-constrained example, retaining mildly profile frames increased the valid-frame ratio from 54.5% to 65.2% and restored a non-zero speaking score.Source: Table 2 · p. 12, §3.2 · p. 12
• In non-multi-person scenes, NAPS takes the candidate with the longest frame presence; in multi-person co-occurrence scenes, final speaking score is CV × DR, with Tier 1 requiring CV ≥0.30 and DR ≥0.04.Source: §3.2 · pp. 12–13
• Stage 7 smooths crop position and scale to suppress frame-level detection jitter while preserving frame-rate continuity and temporal alignment with the original clip.The crop is resized to 224 × 224, and black frames are inserted when the target face is absent from its trajectory.Source: §3.2 · pp. 13–14
• A low-light recovery mode lowers the face-detection confidence threshold, removes skin-color filtering, and relaxes the minimum trajectory length to capture weak facial signals.Recovered identities use EMA-smoothed cropping and manual target-speaker selection.Source: §3.2 · pp. 14–15
• Seven samples were discarded for persistent profile-face views, severe blur, or the absence of a valid facial region.Source: §3.2 · p. 15
• 4,396 samples remained as valid localization and cropping outputs; automatic localization accuracy was approximately 95%, with the remaining approximately 5% manually reviewed and corrected.Source: §3.2 · p. 15

4.1. Overall Framework

• DS-TAFNet concatenates 256-dimensional visual and textual representations, then projects them and classifies sentiment polarity.The visual representation comes from DS-TANet; the text representation comes from MacBERT-Base with a sentiment-guided prompt.Source: §4.1 · p. 16

4.2. Dual-Stream Temporal Visual Encoder (DS-TANet)

• The visual encoder processes 14 uniformly sampled NAPS facial frames at 224×224 resolution using separate appearance and motion streams, motion-guided enhancement, and Bi-GRU temporal modeling.Its static branch uses EfficientNetB2 pretrained on AffectNet; the dynamic branch estimates inter-frame optical flow with RAFT.Source: §4.2 · p. 17
• RAFT-derived optical-flow maps are compressed into frame-level motion features, while EfficientNetB2 independently encodes each frame’s appearance.A zero vector is inserted for the first frame, which has no preceding frame for optical-flow estimation.Source: §4.2 · pp. 17–18
• Motion features modulate static features channel-wise before a Bi-GRU models the enhanced sequence’s global temporal dependencies.The EfficientNetB2 backbone is pretrained on AffectNet.Source: §4.2 · pp. 17–18

W2 ReLU

• Motion-guided attention uses dynamic features to modulate static features channel-wise, with a residual term preserving the static representation when motion is weak.Enhanced features from invalid black frames are zeroed before the sequence is processed by a bidirectional GRU.Source: W2 ReLU · p. 18
• The Bi-GRU concatenates its 128-dimensional forward and backward states into a 256-dimensional representation and selects the last valid frame’s state as the video representation.The valid-frame mask prevents selection of a fixed final step that may be an invalid or black frame.Source: W2 ReLU · p. 18

4.3. Text Encoder

• Prompted MacBERT uses the [MASK] position’s final hidden state as a sentiment-aware semantic anchor, then projects it to a 256-dimensional textual representation.The prompt is “Overall, the sentiment tendency of this text is [MASK].”Source: §4.3 · pp. 18–19

4.4. Multimodal Fusion and Classification

• The fusion study compares concatenation and bidirectional cross-modal attention to examine how data scale constrains fusion-strategy selection.Concat Fusion is the adopted lightweight strategy; it concatenates visual and textual representations before a fusion head and two-layer classifier.Source: §4.4 · p. 19
• Cross-modal attention lets each modality query the other’s original representation, then concatenates the enhanced representations for classification.The attention strategy uses approximately 1.4M trainable fusion parameters and does not add Modal Dropout or auxiliary classification heads.Source: §4.4 · pp. 20–21

5.1.1. Dataset and Split

• Experiments use the official CH-SIMS v2.0S supervised subset after removing neutral samples, leaving 3,866 samples.The training-set positive-to-negative ratio is approximately 1:1.20, so balanced sampling and Focal Loss are used.Source: §5.1.1 · p. 21

5.1.2. Evaluation Metrics

• Macro F1 is the primary metric because it averages class-wise F1 scores equally under class imbalance; Acc2 is supplementary.The positive-to-negative ratios are approximately 1:1.20, 1:1.41, and 1:1.15 in training, validation, and test sets.Source: §5.1.2 · p. 22

5.1.4. Implementation Details

• MacBERT-Base training freezes its embedding layer and bottom four Transformer layers while leaving the top eight layers trainable.The experiments run in PyTorch on an NVIDIA RTX 4080 GPU.Source: §5.1.3 · p. 22, §5.1.4 · p. 23

5.1.3. Training Configuration

• DS-TAFNet training freezes both unimodal backbones while training fusion, then jointly fine-tunes the top two MacBERT layers and fusion module with the visual encoder frozen.The visual variants use shared hyperparameters, data augmentation, balanced sampling, and Focal Loss; EfficientNetB2 uses AffectNet-pretrained weights and RAFT remains frozen.Source: §5.1.3 · p. 22, §5.1.4 · p. 23

5.2.1. Comparison with Baselines Reported in the Original Dataset Paper

• 87.49% Acc2 exceeds AV-MC(Semi) (83.46%) and AV-MC (82.50%) by 4.03 and 4.99 percentage points, respectively, without additional data.AV-MC(Semi) uses 10,161 additional unlabeled videos.Source: §5.2.1 · pp. 23–24, Table 3 · p. 24

5.2.2. Comparison with Recent Related Methods

• The proposed method outperforms CMC and MC-Teacher by 3.06 and 3.16 percentage points, respectively.The comparison uses the official split while additionally removing neutral samples and samples without valid facial sequences after NAPS quality control.Source: §5.2.2 · p. 24, Table 5 · p. 25
• 87.49% Acc2 and 87.48% Macro F1 are reported for the proposed method on the 911-sample test set.The method uses text and vision without acoustic input or additional data.Source: §5.2.1 · p. 23, Table 5 · p. 25, §5.2.2 · p. 25

5.3. Ablation Study of the Visual Encoder

• The ablation starts with an EfficientNetB2 static-stream baseline, adds RAFT motion, and compares concatenation with motion-guided attention, each with and without Bi-GRU.Source: §5.3 · p. 25
• The full visual model improves Acc2 and Macro F1 by 1.76 and 1.60 percentage points over the static baseline.Adding the RAFT dynamic stream alone accounts for 0.89 points of F1.Source: §5.3 · pp. 25–26, Table 6 · p. 26
• Without temporal modeling, motion-guided attention falls 0.87 points below simple concatenation; with Bi-GRU, it exceeds concatenation by 0.46 points.The Bi-GRU gain is +1.58 along the attention path and +0.25 along the Concat path.Source: §5.3 · p. 26

5.4. Unimodal Baseline Performance

• Exp1's 80.98% Macro F1 is 0.43 percentage points above the text baseline's 80.55%, indicating that NAPS-processed static facial sequences perform comparably.Exp1 uses the EfficientNetB2 static stream without the dynamic stream.Source: §5.4 · pp. 26–27
• Exp5 reaches 82.58% Macro F1, 1.60 percentage points above Exp1's 80.98% Macro F1 after adding RAFT motion, motion-guided attention, and Bi-GRU temporal modeling.The comparison supports a two-level progression from static visual features to the complete temporal visual encoder.Source: §5.4 · pp. 26–27

5.5. Comparison of Fusion Strategies

• Cross-Modal Attention raises test-set Macro F1 from 0.8748 to 0.8814 (+0.66 percentage points), but has approximately 3.5 times Concat Fusion’s parameters and greater validation instability with approximately 2.4k training samples.Concat Fusion has lower model complexity, a more stable training process, and stronger generalization.Source: §5.5 · p. 27

6. Conclusion

• NAPS is a seven-stage visual preprocessing pipeline with face tracking, identity association, normalized lip-motion analysis, and pose-aware soft weighting, reaching approximately 95% localization accuracy.A small number of complex samples still require manual review, leaving room to improve automatic localization.Source: §6 · pp. 28–29
• The NAPS-processed EfficientNetB2 static baseline reaches 80.98% Macro F1, comparable to the textual baseline's 80.55%.The conclusion describes this as a static baseline using NAPS-processed sequences.Source: §6 · p. 28
• DS-TANet combines EfficientNetB2 static appearance and frozen RAFT motion streams through motion-guided attention and Bi-GRU temporal modeling, achieving 82.58% Macro F1.The conclusion states that static appearance and inter-frame motion are complementary, and temporal modeling is essential for effective sentiment representations.Source: §6 · p. 28
• Without acoustic modality or additional data, DS-TAFNet achieves 87.49% Acc2 and 87.48% Macro F1, outperforming the study's trimodal baselines.The conclusion states that greater fusion complexity does not reliably yield performance gains under limited training data.Source: §6 · p. 28
• The authors prioritize unimodal representation quality before complex cross-modal interaction as a design principle for data-limited multimodal pattern recognition.They also describe NAPS as a reusable modular visual front end and DS-TANet as a reference for strengthening weak visual modalities.Source: §6 · p. 29
• Transfer to other language settings remains unvalidated and requires adapted text encoders; the authors plan evaluation on English and multilingual datasets.The conclusion also identifies future cross-dataset pretraining and data augmentation to reassess higher-capacity fusion at larger data scales.Source: §6 · p. 29
• The acoustic modality is not incorporated; the authors plan to add it once its representation quality is sufficiently improved to investigate complementarity with optimized visual representations.The stated motivation is that acoustic prosodic and intonational information may complement visual expressive cues.Source: §6 · p. 29

Generative AI Disclosure

• Generative AI tools were used for language refinement and to generate simple icons used in figures.Source: Generative AI Disclosure · p. 29
• The authors state that the research content, figures, and analyses are their own.Source: Generative AI Disclosure · p. 29
• The authors reviewed and edited all AI-assisted output and take full responsibility for the published article.Source: Generative AI Disclosure · p. 29

CRediT authorship contribution statement

• Su Li is credited with original-draft writing, visualization, validation, methodology, data curation, conceptualization, and investigation.Source: CRediT authorship contribution statement · p. 30
• Yigong Zhang is credited with conceptualization, original-draft writing, review and editing, supervision, formal analysis, and funding acquisition.Source: CRediT authorship contribution statement · p. 30
• Lei Xiong and Chune Li are each credited with review and editing, visualization, and formal analysis.Source: CRediT authorship contribution statement · p. 30

Declaration of competing interest

• The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.Source: Declaration of competing interest · p. 30