Source-linked AI summary
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Panagiotis Tzirakis, George Trigeorgis, Mihalis A. Nicolaou, Björn Schuller, Stefanos Zafeiriou
TL;DR
Emotion recognition is challenging because emotions vary in expression and span multiple modalities. The paper proposes an end-to-end audiovisual system combining learned speech and visual representations with temporal modeling, and reports improved performance over traditional handcrafted-feature approaches on RECOLA.
Problem
Emotion recognition is challenging because human emotions lack temporal boundaries and individuals express them differently across speech and visual modalities.
Method
The system extracts speech features with a CNN and visual features with ResNet-50, fuses both streams, models context with LSTM networks, and trains end to end.
Results
The experiments report that the proposed models outperform designed features and other models on several RECOLA emotion-prediction comparisons, including test-set arousal performance.
Takeaways & Limitations
End-to-end multimodal learning can exploit speech and visual information while modeling contextual temporal information for spontaneous emotion prediction.
Abstract
from arXiv · showhide
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
I. INTRODUCTION
Emotion recognition is difficult because emotions have no fixed temporal boundaries and vary across individuals. The paper proposes end-to-end audiovisual affect sensing using learned speech and visual representations, evaluated on RECOLA.
- Emotion recognition supports human–machine interaction and applications such as fatigue monitoring and caller-state detection.
- Human emotions lack temporal boundaries, and individuals express them differently, making recognition challenging across modalities.
- The paper replaces predominantly hand-engineered representations with learned representations derived directly from raw input signals.
- The proposed system uses a CNN for speech, ResNet-50 for visual information, fusion of both streams, and an LSTM for affective-state modeling.
- On RECOLA, the multimodal model benefits from both modalities, while unimodal models produce the best results for their respective speech and visual modalities.
II. RELATED WORK
Prior emotion-recognition studies applied deep networks to audio, visual, and audiovisual data, but commonly relied on handcrafted features or separate training stages. This paper emphasizes an end-to-end multimodal model that also captures temporal context.
- Deep learning research in pattern recognition includes CNNs, DBNs, autoencoders, and LSTMs, with LSTMs designed to enhance memory for sequential information.
- Earlier multimodal systems extracted or learned audio and video features separately before combining them for recognition.
- Speech emotion systems commonly used handcrafted features, probability statistics, or transformed inputs before final classification.
- Visual emotion studies used deep networks with hypergraphs or CNN–RNN pipelines for image- and video-based categorical emotion recognition.
- AVEC 2016 evaluated audio, video, and physiological modalities, including systems based on relevance-vector machines and handcrafted geometry or acoustic features.
- The proposed study addresses prior reliance on handcrafted features and incomplete temporal modeling through an end-to-end multimodal model with contextual temporal information.
III. PROPOSED METHOD
The proposed method learns spatial representations from audio and visual signals using convolution, models their temporal structure with LSTM cells, and trains the system through backpropagation.
- The model uses one-dimensional convolution for audio signals and two-dimensional convolution for visual signals.
- Learned kernel parameters perform task-specific spatial modeling that reduces background noise and enhances relevant signal components.
- LSTM cells model the temporal structure of both speech and video after spatial processing.
- The complete model is trained subsequently with backpropagation using the stated objective function.
A. Visual Network
The visual pathway uses ResNet-50 to process cropped-face pixel intensities through residual learning and a sequence of convolutional bottleneck blocks.
- The visual network uses a 50-layer deep residual network with cropped-face pixel intensities as input.
- Residual learning combines a learned residual function with either an identity mapping or a linear projection matching the input dimensions.
- ResNet-50 begins with a 7x7 convolutional layer with 64 feature maps followed by 3x3 max pooling.
- The remaining architecture uses four bottleneck stages, each containing 1x1, 3x3, and 1x1 convolutions with shortcut connections.
- Table I reports the replication of each bottleneck architecture and the resulting convolutional feature-map sizes.
B. Speech Network
The speech network learns features directly from normalized raw waveforms using temporal convolutions, pooling, and dropout regularization. Its recurrent layers model temporal structure while limiting overfitting.
- B. Speech Network: The model jointly learns speech feature extraction and regression rather than using separately engineered acoustic features.
- B. Speech Network: 6 s raw-waveform segments are normalized to zero mean and unit variance, producing 96000-dimensional inputs at 16 kHz.
- B. Speech Network: Twenty 5 ms temporal filters extract fine-scale spectral information from the high-sampling-rate signal.
- B. Speech Network: Forty 500 ms filters capture longer-term speech characteristics and roughness after temporal pooling downsamples the responses to 8 kHz.
- B. Speech Network: Channel max-pooling reduces dimensionality while preserving necessary convolved-signal statistics, and dropout with probability 0.5 regularizes the model.
C. Objective function
The system trains its networks using concordance correlation as the objective, aligning optimization with the evaluation criterion. Modality-specific networks are pretrained before their features are fused and refined in a multimodal recurrent model.
- C. Objective function: The objective minimizes concordance-correlation cost L_c instead of training with MSE while evaluating with ρ_c.
- C. Objective function: Minimizing L_c, equivalently maximizing ρ_c, is performed by backpropagating its gradient through the final-layer weights.
- D. Network Training: Each modality-specific network is trained separately before multimodal training to speed up the procedure.
- D. Network Training: Speech and visual features are concatenated into a 1920-dimensional vector feeding two 256-cell LSTM layers, while the modality networks are fine-tuned.
- D. Network Training: The recurrent networks process 6 s sequences as 150 subsequences matching the 40 ms annotation frequency.
IV. DATASET
The study investigates time-continuous prediction of spontaneous and natural arousal and valence from speech and visual data using the RECOLA database.
- IV. DATASET: RECOLA provides multimodal recordings for studying spontaneous and natural arousal and valence prediction from speech and visual data.
- IV. DATASET: The corpus includes audio, video, electro-cardiogram, and electro-dermal activity modalities, with 9.5 hours of multimodal recordings reported.
V. EXPERIMENTS & RESULTS
Experiments train the models with Adam, dropout, augmentation, and prediction post-processing. The architecture combines speech and visual feature extraction with recurrent layers, and sequence length is selected for multimodal training based on development results.
- V. EXPERIMENTS & RESULTS: Adam optimization uses a fixed learning rate of 10^-4, with dropout probability 0.5 applied to nonrecurrent layers for regularization.
- V. EXPERIMENTS & RESULTS: Video inputs use 96 × 96 images, small batches constrained by hardware, resizing and cropping augmentation, and random brightness and saturation changes.
- V. EXPERIMENTS & RESULTS: Development-set predictions undergo median filtering, centering, scaling, and time-shifting as post-processing steps.
- V. EXPERIMENTS & RESULTS: The network extracts and concatenates raw speech and visual features before two LSTM layers capture contextual information.
- V. EXPERIMENTS & RESULTS: 60 training epochs are used for all model experiments.
- V. EXPERIMENTS & RESULTS: The multimodal model uses sequence length 150 because visual-network performance differs little between sequence lengths 150 and 300.
B. Speech Modality
The speech modality models are evaluated on arousal and valence using concordance correlation, including sequence-length and optimization comparisons. Learned speech representations capture acoustic and prosodic cues, while comparisons show strong test performance against designed features and prior methods.
- B. Speech Modality: Optimizing all networks directly for ρc provides a more elegant objective and consistently better results across test-runs.The authors state that direct optimization matches the evaluation metric.
- B. Speech Modality: The model outperforms designed features in terms of ρc, although eGEMAPS performs closely on valence.The authors note that valence is more difficult to predict from speech than arousal.
- B. Speech Modality: The experiments report arousal and valence performance using ρc, with results organized by sequence length and optimization objective.Table II varies sequence length; Table III contrasts optimization with MSE versus ρc.
- B. Speech Modality: The model outperforms other literature models on the arousal test set despite lower validation ρc than the challenge baseline.The comparison covers methods submitted to the AVEC 2016 challenge.
- B. Speech Modality: Speech-model gate activations are sensitive to acoustic and prosodic cues, including RMS-energy range, loudness, and fundamental-frequency mean.These cues are described as relevant to affective state recognition and are reflected in gate-activation analyses.
C. Visual Modality
The visual modality is evaluated on RECOLA for arousal and valence using appearance and geometric comparisons. Visual information benefits valence prediction, and the proposed model achieves the best valence test-set performance among the compared models.
- C. Visual Modality: Visual modality is reported to benefit prediction of the valence dimension, with an exception for Video CNN-L4 using appearance features.The comparison uses appearance and geometric features alongside the proposed network’s raw face pixels.
- C. Visual Modality: The proposed model outperforms all other compared models on the valence test set.Table V presents the RECOLA valence results used for this comparison.
D. Multimodal Analysis
On RECOLA, the proposed system is compared with other audiovisual models and examined through arousal and valence predictions. It outperforms competing models on valence, while arousal remains stronger for OA RVM-SR.
- The proposed model outperforms the two other audiovisual RECOLA models on valence with high magnitude.
- OA RVM-SR achieves the best result for arousal among the compared audiovisual models.
- Figure 3 illustrates arousal and valence predictions for a single RECOLA test subject.
- The comparison uses RECOLA results measured by concordance correlation coefficient, with development-set performance shown in parentheses.
- The broader experiments report that unimodal models significantly outperform other RECOLA and AVEC2016-challenge models, while the multimodal model is stronger for valence than arousal.