Source-linked AI summary
Temporal Multimodal Fusion for Video Emotion Classification in the Wild
Valentin Vielzeuf, Stéphane Pateux, Frédéric Jurie
TL;DR
The paper studies emotion classification from short videos recorded in the wild, where limited labeled data and complex, diverse emotions make generalization difficult. It combines improved visual descriptors with temporal and multimodal fusion, and its model ranked fourth in the 2017 Emotion in the Wild challenge with 58.81% accuracy.
Problem
Emotion classification from videos recorded in the wild must handle difficult conditions and only 773 labeled short video clips relative to standard deep-model complexity.
Method
The paper combines audio and visual modalities with 2D CNN and C3D face descriptors, temporal fusion, and multimodal fusion methods including hierarchical feature-and-score combination.
Results
58.81% accuracy ranked the proposed method 4th at the EmotiW’17 challenge.
Takeaways & Limitations
The study indicates that temporal and multimodal fusion, alongside carefully chosen model capacity and transfer learning, are central design directions for this small-data setting.
Takeaways & Limitations
The differing Train/Validation/Test class distributions can make validation performance differ from test performance, and selected temporal windows may miss the emotion because videos are labeled as wholes.
Abstract
from arXiv · showhide
This paper addresses the question of emotion classification. The task consists in predicting emotion labels (taken among a set of possible labels) best describing the emotions contained in short video clips. Building on a standard framework -- lying in describing videos by audio and visual features used by a supervised classifier to infer the labels -- this paper investigates several novel directions. First of all, improved face descriptors based on 2D and 3D Convo-lutional Neural Networks are proposed. Second, the paper explores several fusion methods, temporal and multimodal, including a novel hierarchical method combining features and scores. In addition, we carefully reviewed the different stages of the pipeline and designed a CNN architecture adapted to the task; this is important as the size of the training set is small compared to the difficulty of the problem, making generalization difficult. The so-obtained model ranked 4th at the 2017 Emotion in the Wild challenge with the accuracy of 58.8 %.
1 INTRODUCTION AND RELATED WORK
Emotion classification in short videos remains challenging outside controlled laboratory settings, motivating multimodal representations and fusion methods that handle temporal and cross-modal information. The paper also addresses limited training data by adapting model capacity and using transfer learning.
- Emotion recognition supports applications including advertising, psychological-disorder understanding, video summarization, and face normalization, but remains difficult in videos recorded in the wild.
- The task assigns each short video clip one emotion from six basic categories plus neutral using audio and visual content.
- Prior work represents audio and video with handcrafted or learned features, including Action Units, deep convolutional networks, and C3D spatio-temporal face descriptors.
- Recent multimodal systems fuse signals with approaches such as two-stream ConvNets, ModDrop, and Multiple Kernel Fusion, commonly using face and audio or speech modalities.
- The paper investigates hierarchical fusion that combines features and scores at different levels to preserve unimodal information while exploiting cross-modal information.
- The paper proposes a C3D-LSTM descriptor and studies temporal information in visual descriptors, while reducing overfitting through model-size choices and transfer learning for 773 labeled clips.
2 PRESENTATION OF THE PROPOSED APPROACH
The proposed approach combines audio with VGG-LSTM and C3D-LSTM visual modalities, using temporal descriptors and multiple fusion strategies. It also introduces improved face processing and a score-weighted C3D method to address temporal variability in emotion expression.
- Overview: The pipeline combines audio, VGG-LSTM, and C3D-LSTM modalities, with feature dimensions balanced to trade off parameters and performance.Each modality produces feature vectors and scores that can be fused at different stages.
- Face and temporal representations: Faces are detected, tracked, aligned, smoothed, and represented using VGG-based 2D features alongside C3D-based temporal features.The pipeline selects the longest face sequence and extracts modality-specific descriptors for subsequent temporal modeling.
- 2D convolutional representation: The fine-tuned VGG-face model uses FER 2013 pretraining, augmentation, and strong dropout to improve robustness and limit over-fitting.It achieved 71.2% on the FER private test set and 45.2% on SFEW without retraining on SFEW.
- 3D convolutional representation: The C3D method trains on all 16-frame windows and weights windows by their scores so more meaningful temporal segments contribute more during learning.The weighting uses a decreasing temperature parameter and is normalized within each video.
- Temporal and multimodal fusion: The approach experiments with temporal overlap and four fusion methods, including ModDrop, score fusion, and the novel Score Trees architecture.Score Trees combine high-level scores with lower-level features, while adapted weight decay preserves unimodal information during fusion.
3 EXPERIMENTAL VALIDATION AND RESULTS
The experimental section evaluates the proposed method on AFEW, first by modality and then through fusion, including the EmotiW’17 challenge submission.
- The experiments cover AFEW dataset preparation, individual modality performance, multimodal fusion, and the final EmotiW’17 challenge evaluation.
3.1 The Acted Facial Emotion in the Wild dataset
AFEW 7.0 contains labeled video clips for seven discrete emotions, with separate training, validation, and test sets whose class distributions differ. The authors also augment training with externally collected and manually annotated movie clips.
- 773 training, 383 validation, and 653 test videos are labeled as angry, disgust, fear, happy, sad, neutral, or surprise.
- Different Train/Val/Test class distributions can make test performance differ from validation performance because some emotions are more challenging.
- External training data: 380 non-overlapping movie clips were collected, processed around detected face timestamps, and manually selected and annotated as additional training data.Annotators achieved 60%–80% validation performance, depending on the annotator.
- External training data: Table 1 reports the number of AFEW 7.0 video sequences for each emotion class.
3.2 Experiments on Single Modalities
Single-modality experiments evaluate VGG-LSTM, C3D-LSTM, C3D, and audio representations. Temporal modeling improves visual descriptors, while the best architectures use one-layer unidirectional LSTMs.
- VGG-LSTM and C3D-LSTM: The best VGG-LSTM and C3D-LSTM architectures use one-layer unidirectional LSTMs because bidirectional and deeper variants over-fit the training data.
- VGG-LSTM: 41.4% validation accuracy is obtained by the VGG model without LSTM using maximum sequence scores.
- VGG-LSTM: The VGG-LSTM experiments improve performance by 3% over Fan et al., with whole-sequence inputs and data augmentation identified as contributing factors.
- C3D-LSTM: The C3D implementation trained on random windows and evaluated on central windows underperforms Fan et al., while central-window training improves results but remains below state of the art.
- C3D-LSTM: Weighted C3D performs better without overlapping windows, and adding LSTM produces the highest scores for this modality.
- C3D-LSTM: The proposed C3D descriptor performs significantly better than the descriptor of Fan et al.
- Audio: 36.5% audio accuracy is below the 39.8% state-of-the-art result, although a perceptron enables high-level audio features during fusion.
3.3 Experiments on Fusion
Fusion experiments compare simple score combinations with ModDrop and Score Trees. The best validation result substantially exceeds the organizer baseline, while the proposed complex fusion methods do not outperform weighted means on the test set.
- Majority voting and mean-score fusion perform worse than the VGG-LSTM modality alone.
- ModDrop and Score Trees achieve promising validation results but underperform simple weighted means on the test set.The authors relate this to their larger parameter counts and parameters cross-validated on the validation set.
- 52.2% validation accuracy is significantly higher than the organizer baseline’s 38.81% accuracy.
- Table 4 compares the different fusion methods on both validation and test sets.
3.4 Our participation to the EmotiW’17 challenge
The EmotiW’17 submissions compared multimodal fusion configurations, with the best run training on both training and validation data. The method ranked fourth, while confusion patterns revealed severe class-specific weaknesses.
- Submission variants: Submission 7 used the same method as Submission 6 but trained models on both training and validation sets, improving accuracy by 1.2%.This improvement was also observed in earlier challenge editions.
- Submission variants: Submission 6 used weighted-mean fusion of all preceding modalities, gaining 1% on the test set while losing 1% on validation.The authors interpreted this divergence as highlighting generalization issues.
- Challenge result: The proposed method ranked 4th in the competition, where top accuracy improved by only 1.1% over the previous edition.The authors suggested methods may be saturating toward assumed human performance around 60%, although top human annotators exceed 70%.
- Confusion patterns: Disgust and surprise were never predicted, while happy, neutral, and angry were well recognized.Neutral produced the largest number of false positives; rows denote true classes and columns predicted classes.
4 CONCLUSIONS
The paper presents a multimodal video emotion-classification approach combining VGG and C3D descriptors with temporal fusion, and ranks fourth at EmotiW’17 with 58.81% accuracy. The authors identify validation–test discrepancy and overfitting as continuing challenges, while suggesting more data, fewer parameters, pretrained fusion models, and contextual modalities as future directions.
- Method: The proposed approach combines VGG and C3D image descriptors with temporal fusion architectures for video emotion classification.The paper also experimentally compares multimodal fusion strategies on AFEW validation and test sets.
- Result: 58.81% accuracy ranked the method 4th at the EmotiW’17 challenge, 1.5% below the competition winners.
- Limitations and future work: Validation performance did not guarantee comparable test performance, and reducing model parameters could help limit overfitting.The paper also proposes pretrained fusion models and larger datasets as ways to address this problem.
- Future work: Future work could add contextual information such as scene descriptions, voice recognition, or movie type as extra modalities.