Source-linked AI summary
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Samira Ebrahimi Kahou, Xavier Bouthillier, Pascal Lamblin, Caglar Gulcehre, Vincent Michalski, Kishore Konda, Sébastien Jean, Pierre Froumenty, Yann Dauphin, Nicolas Boulanger-Lewandowski, Raul Chandias Ferrari, Mehdi Mirza, David Warde-Farley, Aaron Courville, Pascal Vincent, Roland Memisevic, Christopher Pal, Yoshua Bengio
TL;DR
Emotion recognition in short movie clips is difficult because few labeled examples must cover varied subjects, lighting, poses, and multimodal signals. The paper combines specialist models for faces, audio, mouth-region appearance, and motion, then aggregates their predictions, obtaining a marked performance increase over simpler model combinations on validation and test sets.
Problem
Emotion recognition in the wild must assign seven emotion labels to short movie clips despite limited samples and substantial variation in subjects, lighting, and poses.
Method
The paper combines modality-specific models for facial frames, audio, spatio-temporal motion, and mouth-region visual features, using supervised and unsupervised deep-learning procedures and prediction aggregation.
Results
The resulting aggregation produced a marked performance increase on both challenge validation and test sets, while simple model averaging did not outperform the SVM approach.
Takeaways & Limitations
Simple weighted combinations of complex specialist models provided a low-complexity way to explore many configurations without retraining the neural networks.
Takeaways & Limitations
Small training sets caused severe ConvNet overfitting, with 96.73% training accuracy versus 35.32% validation accuracy on AFEW2.
Abstract
from arXiv · showhide
The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convolutional neural network, focusing on capturing visual information in detected faces, a deep belief net focusing on the representation of the audio stream, a K-Means based "bag-of-mouths" model, which extracts visual features around the mouth region and a relational autoencoder, which addresses spatio-temporal aspects of videos. We explore multiple methods for the combination of cues from these modalities into one common classifier. This achieves a considerably greater accuracy than predictions from our strongest single-modality classifier. Our method was the winning submission in the 2013 EmotiW challenge and achieved a test set accuracy of 47.67% on the 2014 dataset.
1 Introduction
The paper addresses seven-way emotion classification in short, realistic film clips by combining specialist models for different modalities. Its multimodal combination outperformed the strongest single-modality classifier and achieved strong challenge results.
- The EmotiW task assigns one of seven emotion labels to 1–2-second clips from feature films with varied subjects, lighting, poses, and audio.
- The approach combines modality-specific models for facial frames, audio, spatio-temporal features, and mouth-region visual features.The models include a ConvNet, deep belief net, relational autoencoder, and shallow mouth-feature network.
- 41.03% accuracy was achieved on the 2013 challenge test set, exceeding the next best competitor’s 35.89%.
2 Related work
Related work treats multimodal fusion as a suitable strategy for emotion recognition in short video clips, alongside the growing use of deep learning in related fields.
- Emotion recognition in short video clips is well suited to methods combining features from different modalities.
- Successful EmotiW approaches include Multiple Kernel Learning for fusing visual and audio features.
- Deep learning’s success in computer vision, language modeling, and speech recognition has extended to emotion recognition.
3.1 A convolutional network approach for faces
The face modality trains a ConvNet on external facial-expression images, converts video-frame predictions into fixed-length descriptors, and classifies clips with an SVM. Registration, illumination normalization, augmentation, and temporal face-box smoothing support this pipeline, while extra-data training addresses severe overfitting.
- ConvNets extract spatially structured visual features through convolution and pooling, with pooling reducing dimensionality and adding slight translation invariance.
- 96.73% training accuracy versus 35.32% validation accuracy exposed severe overfitting when ConvNets were trained on the small competition dataset.The authors therefore trained the deep network on a separate facial-expression dataset.
- The face pipeline trains on extra-data faces, extracts seven-class frame probabilities, aggregates them into fixed-length video descriptors, and classifies clips with an SVM.Aggregation uses expansion or contraction to produce descriptors for competition videos.
- 3.1.1 Additional Face Dataset: 35,887 Google images and 4,178 Toronto Face Dataset images provide external facial-expression training data covering the seven emotion classes.The Google images are grayscale 48 × 48 pixels, while the Toronto dataset mainly contains fully frontal poses.
- 3.1.1 Additional Face Dataset: Registration and illumination normalization integrate the external and competition face images despite differences in subjects, lighting, poses, and cropping.Registration uses 51 facial keypoints, while isotropic smoothing provides illumination normalization.
- 3.1.2 Extracting frame-wise emotion probabilites: The ConvNet uses 48 × 48 inputs, random 40 × 40 crops, horizontal flips, four stages, and seven softmax outputs for the emotion labels.Random cropping and flipping expand the limited training set and help avoid overfitting; test-time classification uses center crops.
3.2 Audio & Deep Belief Networks
The audio specialist uses a pretrained deep belief network to represent emotion-related information from movie soundtracks, followed by supervised fine-tuning and temporal feature pooling.
- DBN pretraining: The audio model uses a deep belief network pretrained unsupervisedly with greedy layerwise restricted Boltzmann machine training before supervised fine-tuning.The network is a probabilistic generative model whose layers are trained as RBMs.
- Audio preprocessing: Movie audio features include 29 Yaafe descriptors and three MFCC feature types because speech, background noise, and soundtrack information may indicate emotion.The features were extracted at a 48 kHz sampling rate.
- DBN pretraining: The DBN contains three RBM layers: a Gaussian noisy-ReLU first layer followed by two Gaussian-Bernoulli layers trained with stochastic maximum likelihood and CD-1.Each RBM layer had 350 hidden units.
- Temporal pooling: Temporal pooling sorts multi-timescale activations, selects the top N rows, and summarizes the most active features with a weighted average before the final softmax.The method backpropagates through pooling during supervised fine-tuning and performed best with bounded nonlinearities such as sigmoid or tanh.
- Supervised fine-tuning: Early stopping on the competition validation set yielded an accuracy of 32.90% during supervised fine-tuning.The training used momentum and learning-rate decay when validation error continued increasing.
3.3 Activity recognition using a relational autoencoder
The activity-recognition specialist captures local spatio-temporal motion patterns with a relational autoencoder and classifies quantized descriptors, but motion alone is weak because it can overfit and may lack emotion-correlated cues.
- Motivation: The model incorporates temporal evolution because emotion labels are assigned to video sequences rather than isolated frames.The approach is based on local spatio-temporal feature computation.
- Learned filters: Figure 6 visualizes a subset of SAE filters learned on the AFEW2 training set across Frames 1, 3, 5, 7, and 9.The filters are shown from left to right in temporal order.
- Activity-recognition pipeline: The method uses a relational autoencoder as a local motion-feature extractor within a bag-of-words activity-recognition pipeline.The pipeline includes motion extraction, K-means vector quantization, and a χ2 kernel SVM.
- Motion features: Local descriptors are computed from sub-blocks of a 14×20×20 super block, concatenated, and reduced with PCA.The spatial combination of smaller regions follows prior findings that local features can improve representations.
- Limitations: The motion classifier overfit the training set, and attempted remedies such as affine data augmentation did not help.The authors suggest that the videos may contain few motion cues strongly correlated with emotion labels, while motion remains useful for disambiguating uncertain cases.
3.4 Bag of mouth features and shallow networks
The bag-of-mouths specialist extracts local visual patterns from a fixed mouth region, pools them into frame descriptors, and averages frame-level predictions for video classification.
- Motivation: Mouth-region features target emotion cues such as smiles for happiness and O-shaped open mouths for surprise.The mouth crop was globally selected by visualizing many training images.
- Feature encoding: Each patch becomes a 400-dimensional representation using triangle activations based on distances to learned centroids.A feature is positive when its centroid is closer than the mean centroid distance.
- Classification: Pooled frame features train a regularized logistic regression classifier, and test-video predictions are averaged across frames.The pooling generally uses average activation, with standard deviation also evaluated.
4 Experimental results
Experiments compare modality-specific models and fusion strategies across AFEW2 and AFEW4. ConvNet1 was the strongest individual model, while weighted fusion and random search produced the best reported test performance.
- Single-modality models: ConvNet1 achieved the highest validation accuracy on both AFEW2 and AFEW4 among the individual models.On AFEW2, its test accuracy was 35.58%; on AFEW4, audio reached 34.20% and bag of mouths 27.42% after ConvNet1.
- Averaged predictions: 40.15% validation accuracy came from averaging all models on AFEW2, producing 37.17% test accuracy in the 2013 competition.The same validation performance was obtained without the second convolutional network, which was omitted from subsequent AFEW4 strategies.
- Averaged predictions: 39.90% validation accuracy came from averaging ConvNet1 and audio, the strongest two-model simple average on AFEW2.This result motivated a more sophisticated combination of those two modalities.
- Weighted fusion: 49.49% validation and 41.03% test accuracy were obtained with random search on weighted averages for the 2013 submissions.The simple average reached 40.15% validation and 37.17% test accuracy, while the weighted approach performed better on both measures.
- SVM combination: 44.71% test accuracy resulted from an SVM combining ConvNet1 and audio, improving over ConvNet1 alone at 37.35%.Training the combination model on predictions from data unseen during sub-model training may have reduced bias, although validation data alone was too small.
5 Conclusions and discussion
The authors attribute their results partly to training deep models with large-scale additional imagery rather than relying only on challenge data. They conclude that simple, constrained weighted aggregation can outperform more flexible combination strategies when component models overfit.
- Conclusions and discussion: Large-scale mining of Google image search imagery was used to train the deep neural network and was believed to help avoid overfitting the challenge dataset.The authors present this as an insight from experiments on both competition datasets.
- Conclusions and discussion: Additional labeled face images produced higher validation accuracy than training the convolutional network directly on extracted challenge faces.The authors attribute this partly to clearer correspondence between still-image labels and emotional content than isolated video frames labeled at clip level.
- Conclusions and discussion: Random search over simple per-model and per-class weighted averages was developed after flexible aggregation methods overfit the data.The resulting low-complexity technique achieved marked gains on both challenge validation and test sets.