Source-linked AI summary
Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
Behzad Hasani, Mohammad H. Mahoor
TL;DR
Practical FER systems still lack sufficient generalization, especially for dynamic video expressions and limited training databases. This paper combines 3D Inception-ResNet, LSTM temporal modeling, and facial landmarks, and evaluates the approach across four databases in subject-independent and cross-database tasks. The authors report outperforming many state-of-the-art methods in both tasks.
Problem
Existing FER methods often lack generalization in practical applications, while video expressions require modeling rapid temporal changes and available databases may contain few or unrelated samples.
Method
The method combines 3D Inception-ResNet layers, an LSTM unit, and facial landmarks to model spatial and temporal information while emphasizing expressive facial components.
Results
The method outperforms many state-of-the-art methods in subject-independent and cross-database FER tasks across CK+, MMI, FERA, and DISFA.
Takeaways & Limitations
The evaluated architecture provides a general solution for facial expression recognition in videos across subject-independent and cross-database settings.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) have shown to outperform traditional methods in various visual recognition tasks including Facial Expression Recognition (FER). In spite of efforts made to improve the accuracy of FER systems using DNN, existing methods still are not generalizable enough in practical applications. This paper proposes a 3D Convolutional Neural Network method for FER in videos. This new network architecture consists of 3D Inception-ResNet layers followed by an LSTM unit that together extracts the spatial relations within facial images as well as the temporal relations between different frames in the video. Facial landmark points are also used as inputs to our network which emphasize on the importance of facial components rather than the facial regions that may not contribute significantly to generating facial expressions. Our proposed method is evaluated using four publicly available databases in subject-independent and cross-database tasks and outperforms state-of-the-art methods.
1. Introduction
Facial expression recognition remains difficult in practical video settings because existing systems often lack generalization and must capture rapid temporal changes. The paper proposes combining 3D convolutions, LSTM modeling, and facial landmarks, then evaluates the method across four databases and cross-database tasks.
- Existing FER methods often perform undesirably in practical applications or lack generalization because they are developed under controlled conditions.
- Video FER is challenging because expressions progress through onset, peak, and offset phases and often occur very quickly.
- Traditional approaches commonly analyze still images independently, ignoring temporal relations between consecutive frames that reveal subtle expression changes.
- FER neural-network training is constrained by databases with few samples and unrelated still images rather than consecutive onset-to-offset sequences.
- The proposed method combines 3D convolutional networks and LSTM to extract temporal relations, incorporates facial landmarks, and tests four databases plus cross-database classification.
2. Related work
Prior FER systems use engineered features, CNNs, temporal models, and recurrent architectures, but standard CNNs omit temporal relations and ordinary RNNs struggle with long sequences. These limitations motivate sequence-aware architectures combining spatial and temporal modeling.
- Traditional FER pipelines use registration, feature extraction, and classification, often relying on hand-crafted descriptors such as LBP, HOG, LPQ, optical flow, landmarks, and PCA.
- Traditional temporal approaches include Bayesian models, CRFs, LD-CRFs, and HCRFs for modeling dynamic facial-expression transitions.
- CNN architectures such as AlexNet and GoogLeNet learn visual features through convolutional, pooling, activation, and multiscale Inception layers.
- Ordinary CNNs extract spatial relations but ignore temporal information in sequences, whereas 3D-CNNs operate across temporal and spatial dimensions.
- RNNs model temporal dynamics but can struggle with long-term sequences because of vanishing or exploding gradients, motivating memory-based mechanisms such as LSTMs.
- Prior sequence-labeling work combines LSTMs with CNNs, including architectures that process visual inputs with spatially and temporally deep recurrent convolutional networks.
3. Proposed method
The proposed FER network combines 3D Inception-ResNet processing, facial-landmark weighting, and an LSTM to model spatial and temporal information in video sequences. Landmark filters emphasize expressive facial components during feature extraction.
- 3. Proposed method: The network uses a 3D Inception-ResNet architecture followed by an LSTM, fully connected layer, and softmax activation.The 3DIR component extracts spatial and temporal features, while the LSTM captures temporal relations in the resulting feature map.
- 3.1. 3D Inception-ResNet (3DIR): The proposed 3D Inception-ResNet is slightly shallower than the original architecture and was selected after investigating several module variations.The authors report that this configuration achieved better recognition rates than their other attempts across several databases.
- 3.1. 3D Inception-ResNet (3DIR): Input videos contain 10 frames of size 299 × 299 with 3 color channels and pass through stem, reduction, 3DIR, pooling, dropout, and fully connected layers.Reduction-A changes the grid size from 38 × 38 to 18 × 18, and Reduction-B changes it from 18 × 18 to 8 × 8.
- 3.2. Facial landmarks: Facial landmarks are incorporated by weighting pixels according to their distances from detected expressive facial components.The implementation detects 66 facial landmark points, and the authors found a Manhattan-distance linear weight function effective across databases.
- 3.2. Facial landmarks: Landmark filters assign the highest weights at landmarks and progressively lower weights to surrounding pixels within a 7 × 7 window.Landmarks are not incorporated with the third 3DIR module because the feature map becomes too small for calculating the filter.
- 3.3. Long Short-Term Memory unit: The LSTM receives the 3DIR feature map vectorized along its sequence dimension, preserving the input time order for sequence classification.The authors investigated 200 hidden units as a reasonable amount for FER.
4. Experiments and results
The experiments evaluate the proposed facial-expression model on four video databases using subject-independent and cross-database protocols. The method generally compares favorably with existing approaches, while database composition and expression characteristics affect recognition difficulty.
- 4.1. Face databases: The evaluation uses MMI, extended CK+, GEMEP-FERA, and DISFA, selected because they contain videos of annotated facial expressions.Still-image databases with unrelated frames are excluded because the method classifies input sequences.
- 4.2. Results: The network extracts 66 facial landmarks, resizes faces to 299×299 pixels, and trains the same settings from scratch separately for each database.The landmarks are obtained by a face-alignment algorithm using regression local binary features.
- 4.2. Evaluation protocols: The experiments use subject-independent five-fold cross-validation and cross-database testing, where each test database is withheld entirely from training.Subject-independent results average recognition rates over five folds; cross-database training uses the remaining databases.
- 4.2.1. Subject-independent task: Sequential 3D processing considerably improves recognition over the 2D Inception-ResNet, while landmarks emphasize expressive changes in components such as lips and eyes.The improvement is reported as especially apparent on the MMI and FERA databases.
- 4.2.1. Subject-independent task: In subject-independent comparisons, the method outperforms state-of-the-art results on FERA and DISFA and achieves comparable results on CK+ and MMI.DISFA recognition varies across folds partly because inactive frames and neutral faces can bias recognition toward the neutral category.
- 4.2.2. Cross-database task: In cross-database evaluation, the method outperforms state-of-the-art results on CK+, FERA, and DISFA but does not improve over others on MMI.The authors note that the comparison methods may train only on CK+, whereas this method trains with additional databases having different settings and subjects.
5. Conclusion
The paper presents a 3D Deep Neural Network for facial expression recognition in videos and evaluates it across subject-independent and cross-database tasks. Its 3D Inception-ResNet, LSTM, and facial-landmark design is reported to outperform many state-of-the-art methods across four databases.
- The proposed 3D Inception-ResNet network processes image sequences and extracts spatial relations between frames.
- An LSTM follows the 3D Inception-ResNet to use temporal relations for sequence classification.
- Facial landmarks are incorporated to distinguish expressive facial components from other facial regions.
- The method is evaluated in subject-independent and cross-database tasks using CK+, MMI, FERA, and DISFA.
- The experiments report that the proposed method outperforms many state-of-the-art methods in both evaluation tasks.