Source-linked AI summary
Going Deeper in Facial Expression Recognition using Deep Neural Networks
Ali Mollahosseini, David Chan, Mohammad H. Mahoor
TL;DR
FER methods often struggle to generalize from limited, database-specific training data to unseen or uncontrolled images. This paper proposes a deep architecture combining convolutional and Inception layers, evaluates it across seven databases, and reports performance comparable to or better than state-of-the-art methods, with improved accuracy and training efficiency over conventional CNNs.
Problem
Existing FER approaches often use engineered features and database-specific tuning, limiting generalization to unseen or uncontrolled images; FER datasets also provide limited subject and expression diversity.
Method
The paper proposes a single-component deep FER network with two convolutional layers followed by max pooling and four Inception layers, using registered facial images as input.
Results
Across subject-independent and cross-database evaluations on seven public datasets, the architecture was comparable to or better than state-of-the-art methods and conventional CNNs.
Takeaways & Limitations
The proposed Inception-based architecture supports FER across multiple databases while reducing the operations required for training compared with conventional CNN methods.
Takeaways & Limitations
Training the complex architecture across databases can reduce subject-independent performance because data vary in pose and lighting and the available FER data are limited.
Abstract
from arXiv · showhide
Automated Facial Expression Recognition (FER) has remained a challenging and interesting problem. Despite efforts made in developing various methods for FER, existing approaches traditionally lack generalizability when applied to unseen images or those that are captured in wild setting. Most of the existing approaches are based on engineered features (e.g. HOG, LBPH, and Gabor) where the classifier's hyperparameters are tuned to give best recognition accuracies across a single database, or a small collection of similar databases. Nevertheless, the results are not significant when they are applied to novel data. This paper proposes a deep neural network architecture to address the FER problem across multiple well-known standard face datasets. Specifically, our network consists of two convolutional layers each followed by max pooling and then four Inception layers. The network is a single component architecture that takes registered facial images as the input and classifies them into either of the six basic or the neutral expressions. We conducted comprehensive experiments on seven publically available facial expression databases, viz. MultiPIE, MMI, CK+, DISFA, FERA, SFEW, and FER2013. The results of proposed architecture are comparable to or better than the state-of-the-art methods and better than traditional convolutional neural networks and in both accuracy and training time.
1. Introduction
Automated FER is important for richer human–machine interaction but remains difficult to generalize beyond the databases and conditions used for training. The paper therefore investigates deep neural networks for cross-database FER across seven public datasets.
- Facial expression recognition provides a key nonverbal channel for recognizing internal emotions in human–machine interaction.
- Traditional FER methods often rely on engineered features and database-specific tuning, limiting their performance on novel or uncontrolled data.
- Neural networks can learn features from training data that generalize to scenarios not represented during training, motivating cross-database evaluation.
- FER datasets commonly contain few subjects, limited examples per expression, or little variation between sets, making deep-network training difficult.FER2013 contains 35,887 images but only 547 disgust examples, while MultiPIE has about 750,000 images from 337 subjects.
- The paper evaluates a deep FER architecture on MultiPIE, MMI, CK+, DISFA, FERA, SFEW, and FER2013, reporting results comparable to or better than competing methods in accuracy and learning time.
2. Background and Related Work
FER systems typically register faces, extract features, and classify expressions, with evaluation commonly performed within databases or across databases. Related work spans engineered features, conventional and specialized deep networks, and cross-database benchmarks.
- Traditional FER pipelines perform face registration, feature extraction, and classification, using geometric, appearance, or motion features.
- Neural networks can replace hand-designed feature selection by learning statistically useful representations from input data.
- Message-based FER predicts labeled emotions, whereas sign-based FER detects facial Action Units and converts them into emotion categories.
- Subject-independent evaluation holds out subjects within one database, while cross-database evaluation trains on one database and tests on another.
- Deep neural networks learn multiple representation levels for complex patterns, and are motivated for FER because hand-crafted features may not generalize across databases.
- Conventional CNNs achieve strong visual-recognition results but can be computationally expensive; AlexNet requires roughly 100M operations per iteration.
- Prior FER systems include AU-aware networks, multimodal video architectures, and 3D CNNs, with reported performance on posed and spontaneous datasets.A multimodal system achieved 41.03% on EmotiW 2013, while the 2014 challenge winner achieved 50.40%.
3. Proposed Method
The proposed FER model combines convolutional modules with Inception-style parallel convolutions and uses registered, augmented face images as input. Registration and architectural choices are intended to improve recognition while controlling computational cost.
- Inception layers increase network depth and width while keeping the computational budget constant, and smaller convolutions improve local-feature recognition.
- The network uses two convolution–max-pooling modules followed by Inception-style modules and fully connected classification layers.
- Each Inception module applies 1 × 1, 3 × 3, and 5 × 5 ReLU convolutions in parallel before concatenating their outputs.
- Facial landmarks from AAM and IntraFace are used to affine-register faces to an average face before extracting a fixed face region.
- Face registration increased FER accuracy by 4–10%, indicating that normalization is a substantial component of the pipeline.
- Registered images are resized to 48×48 pixels, and corner, center, and horizontally flipped crops augment the training data.
- The model uses a polynomially decreasing learning rate because testing found this policy superior to fixed, step, and exponential alternatives.
4. Experimental Results
The experiments evaluate the proposed FER architecture across seven databases using subject-independent and cross-database protocols. Results indicate strong performance, including improved top-2 accuracy, cross-database generalization, and lower computational cost than AlexNet.
- Datasets: Experiments covered seven public facial-expression databases spanning posed, spontaneous, and in-the-wild imagery.The databases were MultiPIE, MMI, DISFA, CK+, FERA, SFEW, and FER2013.
- Evaluation Protocol: Subject-independent evaluation used strict subject-level splits and five-fold cross-validation, except for FERA and SFEW release-defined test sets.Training, validation, and test images were separated by subject where cross-validation was applied.
- Subject-Independent Results: 15% higher top-2 accuracy than top-1 accuracy was observed in most cases, especially on FERA, SFEW, and FER2013.The authors motivate top-2 prediction because a single expression can be ambiguous during transitions or before peak expression.
- Cross-Database Results: Cross-database evaluation trained on one database and evaluated on another, addressing the differing lighting, pose, and emotion characteristics across databases.The experiments also included CK+↔MMI comparisons and reported average accuracy for six basic expressions plus neutral.
- Overall Findings: The authors conclude that the network generalizes well for FER and can serve as a baseline for cross-database evaluation on challenging datasets.They note that prior cross-database studies had not evaluated FERA, SFEW, and FER2013 in the same way.
- Comparison and Efficiency: The proposed architecture performed better than AlexNet on MMI and FER2013, comparably on the remaining databases, and required about 25M rather than more than 100M operations.Both networks were trained from scratch using the same protocol.
5. Discussion
Subject-independent results were comparable to or better than the state of the art, with significant gains on MultiPIE and SFEW but exceptions on MMI and FERA. The authors attribute discrepancies partly to cross-database training that mixes pose and lighting conditions.
- Subject-independent results were comparable to or better than the current state of the art.
- The proposed method performed significantly better than the state of the art on MultiPIE and SFEW, with MMI and FERA as exceptions.
- Cross-database training mixes images with differing pose and lighting conditions, contributing to performance discrepancies on subject-independent databases.
6. Conclusion
The paper introduces an Inception-based deep neural network for facial expression recognition that classifies registered faces into seven expression categories. Evaluated across subject-independent and cross-database settings, it improves accuracy while reducing training operations relative to conventional CNNs.
- The network uses two convolutional layers with max pooling followed by four Inception layers to classify registered faces into six basic expressions or neutral.
- Across seven databases and both evaluation settings, the proposed network outperformed several state-of-the-art methods using engineered features and tuned classifier parameters.
- The proposed architecture increases classification accuracy while reducing the operations required for training compared with conventional CNNs.