Source-linked AI summary
Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
Mariana-Iuliana Georgescu, Radu Tudor Ionescu, Marius Popescu
TL;DR
Facial expression recognition research has largely emphasized deep models, leaving limited attention to handcrafted features and their combination with deep representations. The paper fuses CNN and BOVW features and applies local SVM learning, achieving state-of-the-art results across FER 2013, FER+, and AffectNet. It also identifies additional datasets and video as directions for future evaluation.
Problem
Most recent facial expression recognition work focuses on deep neural networks, while handcrafted models have received little attention and local learning had not been combined with deep features.
Method
The approach fuses CNN-learned automatic features with BOVW handcrafted features and uses local learning with nearest-neighbor selection followed by a one-versus-all SVM.
Results
The approach reports state-of-the-art results across three datasets, including 75.42% on FER 2013, 87.76% on FER+, 59.58% on AffectNet 8-way, and 63.31% on AffectNet 7-way classification.
Takeaways & Limitations
Combining deep and handcrafted features with local learning is reported to surpass the best methods on FER 2013, FER+, and both AffectNet classification settings.
Takeaways & Limitations
The authors identify evaluation on additional datasets and adaptation of the method for video as future work.
Abstract
from arXiv · showhide
We present an approach that combines automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model in order to achieve state-of-the-art results in facial expression recognition. To obtain automatic features, we experiment with multiple CNN architectures, pre-trained models and training procedures, e.g. Dense-Sparse-Dense. After fusing the two types of features, we employ a local learning framework to predict the class label for each test image. The local learning framework is based on three steps. First, a k-nearest neighbors model is applied in order to select the nearest training samples for an input test image. Second, a one-versus-all Support Vector Machines (SVM) classifier is trained on the selected training samples. Finally, the SVM classifier is used to predict the class label only for the test image it was trained for. Although we have used local learning in combination with handcrafted features in our previous work, to the best of our knowledge, local learning has never been employed in combination with deep features. The experiments on the 2013 Facial Expression Recognition (FER) Challenge data set, the FER+ data set and the AffectNet data set demonstrate that our approach achieves state-of-the-art results. With a top accuracy of 75.42% on FER 2013, 87.76% on the FER+, 59.58% on AffectNet 8-way classification and 63.31% on AffectNet 7-way classification, we surpass the state-of-the-art methods by more than 1% on all data sets.
1. Introduction
The paper combines CNN-learned automatic features with BOVW handcrafted features and local learning to improve facial expression recognition. Experiments across three benchmark datasets report state-of-the-art results.
- 1. Introduction: The approach combines automatic CNN features with handcrafted BOVW features and applies local learning during training.The study also evaluates multiple CNN architectures, pre-training choices, and Dense-Sparse-Dense fine-tuning.
- 1. Introduction: The combined models are evaluated with global and local SVM alternatives on FER 2013, FER+, and AffectNet.The experiments compare the proposed models with recent state-of-the-art approaches.
- 1. Introduction: 75.42% accuracy on FER 2013 surpasses the state-of-the-art accuracy by 2.02%.The result is obtained with the combined features and local SVM approach.
- 1. Introduction: 87.76% accuracy on FER+ surpasses the best reported method by 2.77%.The paper reports this as its best FER+ accuracy.
- 1. Introduction: The approach surpasses prior methods by 1.58% in AffectNet 8-way classification and by 1.20% in 7-way classification.The 7-way setting excludes the contempt class.
- 1. Introduction: The paper contributes CNN training with Dense-Sparse-Dense, feature fusion with local learning, extensive evaluation, and state-of-the-art results on three datasets.The authors identify these four contributions explicitly.
2. Related Work
Facial expression recognition has shifted from handcrafted features toward deep learning, while some work combines both feature types or uses local learning. This paper distinguishes itself by combining deep and handcrafted features with local learning.
- 2. Related Work: Early facial expression recognition work mostly used handcrafted features, whereas recent work has predominantly adopted deep learning.The shift followed the success of AlexNet in the ImageNet challenge.
- 2. Related Work: The top-scoring FER 2013 Challenge system used a deep CNN, while the best handcrafted model ranked fourth.This comparison illustrates the performance gap motivating later deep-learning approaches.
- 2. Related Work: Recent methods include CNN ensembles, combinations of deep features with SIFT or HOG, and approaches for static images or video.The related work spans multiple architectures, feature types, and input modalities.
- 2. Related Work: Closer prior work either combined deep and handcrafted features or used local learning with handcrafted models.The cited approaches provide the two main precedents for the paper’s method.
3. Approach
The approach combines CNN-derived automatic features with spatial BOVW handcrafted features, then applies local learning with linear SVMs to classify each test image.
- Deep Models: Three CNN models provide automatic features, with VGG-13 trained from scratch and VGG-face and VGG-f used in pre-trained or fine-tuned forms.The CNNs use data augmentation and Dense-Sparse-Dense training to reduce overfitting.
- Handcrafted Model: The BOVW representation quantizes dense SIFT descriptors into visual words and concatenates binary spatial-pyramid feature vectors across image regions.Spatial bins preserve region-specific facial information, such as muscle contractions near the eyes.
- Model Fusion and Learning: Deep and BOVW feature vectors are concatenated and L2-normalized before the learning stage.Deep features come from the last remaining fully connected layer after removing each CNN's softmax layer.
- Model Fusion and Learning: Local learning selects nearby training samples, trains a classifier on them, and predicts the label for the associated test sample.Neighbor selection uses cosine similarity, and the local classifier is a linear SVM.
- Model Fusion and Learning: A local linear SVM produces a non-linear overall decision function because each test sample receives a separately trained classifier.The framework can correct cases misclassified by a single global linear classifier.
4. Experiments
Experiments evaluate combined deep and handcrafted features with global and local SVMs across FER 2013, FER+, and AffectNet. Local SVM generally provides the strongest results, while ablations support combining both feature types.
- Experimental setup: The experiments compare combined deep and handcrafted models using global or local SVMs across FER 2013, FER+, and AffectNet.Results are reported with and without data augmentation and against state-of-the-art approaches.
- Results on FER 2013: 75.42% accuracy on FER 2013 is achieved by the local SVM with flipped-image augmentation, exceeding the global SVM by 2.17%.Without augmentation, the local SVM reaches 74.92%; local prediction takes 40.28 seconds for 3,589 test images versus 23.93 seconds globally.
- Results on FER 2013: 70.33% accuracy is obtained by k-NN with 200 neighbors, while the local SVM reaches 75.42% on FER 2013.The authors conclude that local SVM provides a considerable improvement over k-NN and surpasses the state-of-the-art accuracy by 2.02%.
- Results on FER+: 87.76% accuracy on FER+ is achieved by the local SVM without augmentation, improving 1.08% over the global SVM and 2.77% over the state-of-the-art method.Data augmentation does not improve performance on FER+, although local SVM remains better than global SVM.
- Results on AffectNet: 59.58% accuracy on AffectNet 8-way classification surpasses the compared approach by 1.58%, while 63.31% on 7-way classification is 1.20% above the reported state of the art.The 8-way result uses local SVM with down-sampling; the best 7-way result uses local SVM with data augmentation.
- Ablation results: The combined model outperforms each individual deep or handcrafted component across the ablation results.The best individual models include fine-tuned VGG-face on FER 2013 and AffectNet, while fine-tuned VGG-f reaches 86.01% on FER+.
5. Conclusion
The paper presents a state-of-the-art facial expression recognition approach that combines deep and handcrafted features with local learning. It reports top accuracies across four benchmarks and identifies additional datasets and video as future evaluation directions.
- The approach combines deep and handcrafted features and applies local learning during training.
- 75.42% top accuracy is reported on FER 2013, 87.76% on FER+, 59.58% on AffectNet 8-way classification, and 63.31% on AffectNet 7-way classification.
- The authors plan to evaluate the approach on additional data sets and adapt it for video.