Source-linked AI summary
Facial Expression Recognition Using a Hybrid CNN-SIFT Aggregator
Mundher Al-Shabi, Wooi Ping Cheah, Tee Connie
TL;DR
Facial-expression recognition is challenging, and CNNs often require substantial training data despite limited expression datasets. This paper combines CNN features with SIFT or Dense SIFT and aggregates models; the aggregator outperforms state-of-the-art methods on FER-2013 and CK+.
Problem
CNNs often need substantial training data to generalize, while facial-expression datasets are limited.
Source: §1 · p. 2Method
Results
The aggregator outperforms state-of-the-art methods on FER-2013 and CK+, and CNN with Dense SIFT outperforms CNN-only and CNN-with-SIFT models.
Source: §6.1 · p. 8, §6.2 · p. 9, §7 · p. 10Takeaways & Limitations
The authors conclude that Dense SIFT and CNN features can complement one another in facial-expression recognition.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Deriving an effective facial expression recognition component is important for a successful human-computer interaction system. Nonetheless, recognizing facial expression remains a challenging task. This paper describes a novel approach towards facial expression recognition task. The proposed method is motivated by the success of Convolutional Neural Networks (CNN) on the face recognition problem. Unlike other works, we focus on achieving good accuracy while requiring only a small sample data for training. Scale Invariant Feature Transform (SIFT) features are used to increase the performance on small data as SIFT does not require extensive training data to generate useful features. In this paper, both Dense SIFT and regular SIFT are studied and compared when merged with CNN features. Moreover, an aggregator of the models is developed. The proposed approach is tested on the FER-2013 and CK+ datasets. Results demonstrate the superiority of CNN with Dense SIFT over conventional CNN and CNN with SIFT. The accuracy even increased when all the models are aggregated which generates state-of-art results on FER-2013 and CK+ datasets, where it achieved 73.4% on FER-2013 and 99.1% on CK+.
1 Introduction
Facial expression recognition matters for human-computer interaction, but CNNs often need more training data than limited expression datasets provide. The paper combines regular or Dense SIFT features with CNN features and aggregates models to address this setting.
1 Introduction
- Facial expression recognition is challenging and has applications in human-computer interaction, call centers, and interactive games.The introduction also cites applications in building robots that understand human emotions.Source: §1 · p. 1
- Earlier approaches used hand-crafted features such as local binary patterns, Haar features, SIFT, and Gabor filters with classifiers.CNNs instead learn features from the training database and can be trained end-to-end with a feedforward classifier.Source: §1 · pp. 1–2
- The proposed approach combines CNN features with regular SIFT or Dense SIFT and aggregates models to improve facial expression recognition on limited data.The CNN and SIFT features are combined during training and testing, with the CNN and SIFT fully connected layers trained at the same time.Source: §1 · p. 2
- CNNs can learn features directly from training data, but facial-expression datasets are limited and CNNs often need substantial data to generalize.SIFT features can generalize without extensive training data, although fixed transformations limit their modeling capacity.Source: §1 · p. 2
- The paper investigates combining SIFT and Dense SIFT with CNN features and designing an aggregator for FER-2013 and CK+.The stated contributions are evaluating the feature combinations and an aggregated classifier intended to achieve state-of-the-art results on both datasets.Source: §1 · p. 2
2 Related Work
Prior facial-expression work includes hand-crafted feature methods, CNNs, and ensembles. The paper distinguishes its hybrid approach by jointly training a custom CNN and fully connected layers after SIFT extraction.
2 Related Work
- Earlier work explored hand-crafted features, CNN architectures and losses, transfer features, and preprocessing for facial-expression recognition.Examples include a CNN reporting 98.3% accuracy on CK+ [8] and a FER-2013 challenge method reporting 71.2% accuracy on a private test [22].Source: §2 · p. 3
- Ensemble approaches fused CNN decisions using learned weights, committee structures, or combinations of video and audio predictors.The cited EmotiW2015 winner combined committee decisions through a hierarchical architecture with exponentially weighted decision fusion.Source: §2 · pp. 3–4
- Unlike prior SIFT-CNN fusion work, this method jointly trains a custom CNN and the fully connected layers after SIFT extraction.The authors also describe their model as smaller and less complex, which they consider favorable when an aggregator requires multiple models.Source: §2 · p. 4
3 Pre-Processing
Preprocessing standardizes images and augments each image with transformations intended to improve robustness to noise and slight changes.
3 Pre-Processing
- Each 48x48 image is augmented tenfold using transformations intended to make the model more robust to noise and slight transformations.The listed operations include horizontal flips, rotations from -30 to 30 degrees, center-area skewing, and zooming at four image corners.Source: §3 · p. 4
- The preprocessing transformations include horizontal flipping, random-angle rotation, center skewing, and corner zooming.Figure 2 depicts examples of the transformations.Source: §3 · p. 4, Figure 2 · p. 4
- All images are normalized to zero mean and unit variance after augmentation.Source: §3 · p. 4
4 Deep CNN Architecture
The authors build a CNN from scratch with convolutional, pooling, and dense layers, then use dropout and Leaky ReLU alongside categorical cross-entropy and Adam.
4 Deep CNN Architecture
- The custom CNN has six convolution layers, three Max-Pooling layers, and two dense fully connected layers.Convolution filters are 3x3, with counts of 64, 128, and 256 as pooling is added; the dense layer has 2048 neurons.Source: §4 · p. 5
- Max-Pooling follows every two convolution layers, while dropout follows each pooling and dense layer to reduce overfitting risk.Pooling uses a 2x2 stride and is described as down-sampling that can provide translation invariance and reduce deeper-layer computation.Source: §4 · p. 5
- The network uses Leaky ReLU, categorical cross-entropy, and Adam, with hyperparameters tuned on the FER-2013 validation set.The Leaky ReLU threshold is selected using that validation set; the authors choose it over ordinary ReLU to address dying ReLU.Source: §4 · p. 5
5 SIFT AND BAG OF KEY-POINTS
Regular SIFT descriptors are converted into a fixed-size bag-of-key-points vector, while Dense SIFT describes a grid of image regions; these features are merged with CNN outputs and the three models can be aggregated.
5 SIFT AND BAG OF KEY-POINTS
- Regular SIFT extracts key-points, computes local gradient directions and magnitudes, and forms a 128-value descriptor from sixteen 8-value cell vectors.Source: §5 · p. 5
- Each image’s SIFT key-point descriptors are clustered with K-means, and cluster counts form a fixed-size feature vector with K = 2048.The descriptor vector is passed through a 4096-unit fully connected layer with dropout and L2 regularization of 0.01 before merging with the CNN model.Source: §5 · pp. 5–6
- Dense SIFT skips key-point extraction, dividing the image into sixteen 12x12-pixel regions and computing a 128-feature descriptor per region.This yields 2048 features for the whole image.Source: §5 · p. 6
- The CNN-only, CNN-with-SIFT, and CNN-with-Dense-SIFT outputs are aggregated using average sum to estimate expression probabilities.The best-match expression is the one with the highest probability.Source: §5 · p. 7
6 Experimental Results
The experiments compare CNN, SIFT-enhanced CNN, and an aggregator on FER-2013 and CK+, finding that Dense SIFT improves on regular SIFT and that aggregation achieves the strongest reported outcomes.
6 Experimental Results
- FER-2013 has 28,709 training images across seven expression labels, while CK+ contributes 309 images from six labels after contempt examples are removed.CK+ originally contains 327 images from 123 subjects; FER-2013 images vary in pose, and Table 1 presents label distributions across the datasets.Source: §6 · p. 7, Table 1 · p. 7
6.1 Experimental results for FER 2013
- On FER-2013, the aggregator surpasses state-of-the-art methods; on CK+, all tested models outperform state-of-the-art techniques.The reported comparisons are shown in the FER-2013 and CK+ accuracy tables.Source: §6.1 · p. 8, Table 2 · p. 9, §6.2 · p. 9
- Dense SIFT improves on CNN alone and CNN with regular SIFT on both datasets, while regular SIFT does not improve FER-2013 accuracy over CNN alone.On FER-2013, Dense SIFT raises CNN accuracy by about 1% and runs faster than regular SIFT because keypoint locations are fixed and Bag of Key Points is unnecessary.Source: §6.1 · p. 8, Figure 5 · p. 9, §6.2 · p. 9
6.2 Experimental Results for CK+ Dataset
- For CK+, the model is pretrained on FER-2013, tuned on CK+, and evaluated using 10-fold cross-validation on all 309 images.Networks are trained for 20 epochs to prevent overfitting on the limited training set.Source: §6 · p. 7, §6.2 · pp. 8–9
7 Conclusion
The conclusion presents a hybrid CNN–Dense SIFT aggregator for facial expression recognition. It reports that combining the models outperforms state-of-the-art methods on both evaluated datasets.
7 Conclusion
- The proposed method combines CNN and Dense SIFT models in an aggregator for facial expression recognition.The authors describe the hand-crafted and deep-learning approaches as complementary.Source: §7 · p. 10
- Dense SIFT shows an advantage over regular SIFT, and aggregating the models yields a noticeable performance gain.Source: §7 · p. 10
- The aggregated approach outperforms state-of-the-art methods on FER-2013 and CK+.Source: §7 · p. 10