Source-linked AI summary
Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
Liam Schoneveld, Alice Othmani, Hazem Abdelkawy
TL;DR
In-the-wild emotion recognition is difficult because spontaneous behavior varies across audio, visual appearance, and timing. The paper addresses this with distilled visual and deep audio feature extractors, model-level fusion, and recurrent temporal modeling, achieving strong results across audiovisual and facial-expression benchmarks, including state-of-the-art valence prediction on RECOLA.
Problem
In-the-wild emotion recognition remains challenging because spontaneous behavior varies widely in audio profile, visual aspects, and timing.
Method
The approach uses a knowledge-distilled visual CNN, a modified VGGish audio backbone, model-level fusion, and recurrent neural networks for temporal dynamics.
Results
The method substantially outperforms existing approaches for RECOLA valence prediction and achieves state-of-the-art results on AffectNet and Google FEC facial-expression evaluation.
Takeaways & Limitations
The results support deep modality-specific representations, knowledge distillation, and multimodal fusion as effective components for audiovisual emotion recognition.
Takeaways & Limitations
The one-million-image unlabeled PowderFaces dataset provides only marginal benefit, so more unlabeled images are needed to assess transfer from large-scale unlabeled-data methods.
Abstract
from arXiv · showhide
Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation and high-performing deep architectures. The deep feature representations of the audio and visual modalities are fused based on a model-level fusion strategy. A recurrent neural network is then used to capture the temporal dynamics. Our proposed approach substantially outperforms state-of-the-art approaches in predicting valence on the RECOLA dataset. Moreover, our proposed visual facial expression feature extraction network outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets.
1. Introduction
Emotion recognition spans facial, vocal, and gestural behavior, but spontaneous in-the-wild expressions vary more widely in audio, appearance, and timing. This paper therefore applies deep learning to audiovisual emotion recognition using separately pretrained modality networks and fusion.
- Formal studies reported universality for six basic emotions alongside cultural differences in spontaneous emotional expressions.The six emotions are anger, disgust, fear, happiness, sadness, and surprise.
- Affect is expressed through facial, vocal, or gestural behaviors and can be represented categorically or along continuous arousal-valence dimensions.The categorical view treats affect as discrete states, whereas the dimensional view uses a continuous arousal-valence space.
- Emotion recognition has expanded from separate speech and facial-expression tasks toward multimodal affective computing applications.Applications include human-computer interaction, health informatics, and assistive technologies.
- In-the-wild recognition is harder because spontaneous behavior varies more widely in its audio profile, visual aspects, and timing.Real-world databases also include illumination variation, noise, occlusion, and non-frontal head poses.
- The paper proposes audiovisual emotion recognition using separately pretrained audio and visual deep CNN modules followed by a dataset-specific fusion module.The fusion module is trained using generic emotion-recognition features extracted by the pretrained components.
2. Related Works and paper contributions
Multimodal emotion recognition combines modality-specific information through several fusion strategies. This paper contributes independent audio and facial feature extractors, self-distillation for facial expression recognition, and recurrent modeling of audiovisual spatio-temporal dynamics.
- 2.1. Related Work: Model-level fusion concatenates modality feature representations and passes them through a model that learns an internal joint representation.The literature also describes feature-level, decision-level, and hybrid fusion strategies.
- 2.2. Contributions of this work: The proposed AVER system combines a knowledge-distilled deep CNN for facial expression recognition with a modified, fine-tuned VGGish model for speech emotion recognition.A model-level fusion approach combines the audio and visual representations, while recurrent neural networks model temporal dynamics.
- 2.2. Contributions of this work: The paper learns two independent emotion-specialized feature extractors, one for audio and one for face images, for downstream audiovisual emotion-recognition tasks.The contribution is framed as learning reusable modality-specific representations.
- 2.2. Contributions of this work: Knowledge distillation, specifically self-distillation, is applied alongside additional unlabeled data for facial expression recognition.Self-distillation trains a student network using predictions from a teacher network of the same size or larger.
- 2.2. Contributions of this work: A recurrent neural network learns the spatio-temporal dynamics required for audiovisual emotion recognition.The recurrent model processes spatial and temporal representations after audio-visual fusion.
3. Proposed multimodal deep CNN architecture
The proposed multimodal architecture combines specialized visual and audio emotion embeddings, knowledge distillation, model-level fusion, and recurrent temporal modeling. Its visual branch is trained across complementary facial-expression datasets, while the audio branch adapts VGGish to produce emotion-oriented embeddings.
- Overall architecture: The architecture has three components: visual facial-expression encoding, audio emotion encoding, and multimodal fusion with recurrent temporal modeling.Visual and audio representations are extracted separately, fused at the model level, and processed with recurrent neural networks.
- Visual facial expression embedding network: Knowledge distillation trains a facial-expression embedding network through teacher and student phases, with the student trained to mimic teacher outputs.The teacher is trained on facial-expression tasks, and a second CNN learns from its predictions.
- The teacher network: The teacher starts from pre-trained FaceNet and is trained jointly on AffectNet categories and Google FEC triplet-based embedding supervision.AffectNet provides eight expression categories, while Google FEC supplies triplets whose similar face-expression pair should be close in embedding space.
- Visual facial expression embedding network: Two teacher networks with different random seeds and penultimate dimensions are ensembled by concatenating their outputs as distillation targets.The teachers use Dface = 128 and Dface = 256, respectively.
- Multimodal fusion: Each modality is independently transformed into a (9, 64) tensor, concatenated into (9, 128), and passed through a two-layer 256-dimensional LSTM before arousal and valence prediction.A final fully connected layer and tanh activation produce two outputs between -1 and 1.
- Audio emotion encoding: Audio inputs are represented as Mel-spectrogram tensors computed from 16 kHz signals using 40-millisecond windows, 40-millisecond hops, and 128 Mel filters spanning 125–7500 Hz.For each audio file, the preprocessing produces a tensor with shape [R, l, M_f].
- Audio emotion encoding: The audio branch modifies pre-trained VGGish by replacing its final convolution and pooling layers, then adds fully connected layers to produce a 128-dimensional emotion embedding.The modified backbone uses global average pooling followed by layers of sizes 4096, 4096, and 128.
4. Experimental results
Experiments evaluate the visual, audio, and multimodal components on AffectNet, Google FEC, and RECOLA. The multimodal model achieves its strongest reported result on RECOLA valence, while distillation and unlabeled data have different effects across facial-expression benchmarks.
- Visual facial expression embedding network performance: 61.6% accuracy on AffectNet and 86.5% triplet accuracy on Google FEC outperform state-of-the-art results for the visual embedding network.AffectNet uses eight facial-expression classes; Google FEC is evaluated with triplet accuracy.
- Ablation and discussion: 58.8% AffectNet accuracy and 85% Google FEC accuracy result when distillation is removed, showing a substantial drop from the distilled model.The paper reports that self-distillation benefits Google FEC more markedly than AffectNet.
- Visual-only and audio-only performance: 0.55 and 0.57 validation CCCs for visual-only valence and arousal rise to 0.66 and 0.57 on the RECOLA test set.The visual-only model achieves state-of-the-art valence performance despite using only visual features.
- Visual-only and audio-only performance: 0.52 valence CCC and 0.70 arousal CCC are obtained by the audio-only model on the RECOLA test set.The arousal result matches existing state-of-the-art performance among audio-only methods.
- Fusion model performance and comparison with state-of-the-art approaches: 0.740 valence CCC and 0.719 arousal CCC are achieved by multimodal fusion on the RECOLA test set.The model substantially exceeds the previous best valence CCC of 0.612, while remaining below the reported arousal state of the art of 0.796.
- Discussion and limitations: The unlabeled PowderFaces dataset provides only marginal performance benefit, if any, and the authors state that many more unlabeled images are needed to assess transfer from large-scale unlabeled-data results.PowderFaces contains approximately one million images, about twice the combined number of faces in AffectNet and Google FEC.
5. Conclusion and Future work
The paper presents a high-performing deep neural network approach for audio-visual emotion recognition, combining distilled visual features, a modified VGGish audio backbone, and model-level fusion. It reports effective facial representation learning and state-of-the-art valence prediction on RECOLA, while identifying continuous emotion encoding as future work.
- The AVER system fuses a distilled visual feature extractor, modified VGGish backbone, and model-level fusion architecture.
- Joint end-to-end training on AffectNet and FEC effectively learns robust facial expression representations.
- Knowledge distillation further improves facial expression recognition.
- The multimodal fusion approach outperforms state-of-the-art methods for valence prediction on RECOLA.
- Future work will investigate continuous emotion encoding through coarse classification, regression, label distribution learning, or ranking.