Source-linked AI summary
Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network
Shervin Minaee, Amirali Abdolrashidi
TL;DR
Facial expression recognition remains challenging because expressions vary within classes and important clues are concentrated in only a few facial regions. The paper uses an attentional convolutional network to focus on salient face areas, achieving promising results across multiple databases and visualizing regions associated with different expressions.
Problem
Facial expression recognition remains difficult because of high intra-class variation and the need to identify informative facial regions.
Method
The paper proposes an end-to-end attentional convolutional network trained from scratch that focuses on salient facial regions, together with visualization of important face areas.
Results
The model achieves promising results across four facial expression recognition databases, including around 70.02% accuracy on FER-2013 and around 99.3% on FERG.
Takeaways & Limitations
Attention can enable a network with fewer than 10 layers to compete with or outperform deeper networks for emotion recognition, while visualization highlights regions crucial for different expressions.
Abstract
from arXiv · showhide
Facial expression recognition has been an active research area over the past few decades, and it is still challenging due to the high intra-class variation. Traditional approaches for this problem rely on hand-crafted features such as SIFT, HOG and LBP, followed by a classifier trained on a database of images or videos. Most of these works perform reasonably well on datasets of images captured in a controlled condition, but fail to perform as good on more challenging datasets with more image variation and partial faces. In recent years, several works proposed an end-to-end framework for facial expression recognition, using deep learning models. Despite the better performance of these works, there still seems to be a great room for improvement. In this work, we propose a deep learning approach based on attentional convolutional network, which is able to focus on important parts of the face, and achieves significant improvement over previous models on multiple datasets, including FER-2013, CK+, FERG, and JAFFE. We also use a visualization technique which is able to find important face regions for detecting different emotions, based on the classifier's output. Through experimental results, we show that different emotions seems to be sensitive to different parts of the face.
I. INTRODUCTION
Facial expressions are important and accessible signals for emotion recognition, but useful clues are concentrated in specific facial regions. The paper proposes attention-based convolutional recognition to focus on those salient regions.
- Facial expressions are widely used for emotion recognition because they are visible, informative, and easier to collect at scale than other signals.
- CNN-based facial expression recognition learns useful features, but emotion clues often come mainly from the mouth and eyes rather than ears or hair.
- The proposed framework uses attention to focus on salient facial regions during expression recognition.
- An attentional convolutional network is designed to focus on feature-rich face regions while achieving high accuracy with fewer than 10 layers.
- The paper visualizes regions with the strongest impact on classifier output, with examples shown for different emotions.
II. RELATED WORKS
Earlier facial-expression systems commonly extracted hand-crafted features before classification, while later CNN systems improved performance. The paper identifies attention to important face regions as a remaining gap.
- Basic emotion-recognition datasets commonly represent six cardinal emotions plus neutral.
- Earlier facial-expression systems used a two-step pipeline combining hand-crafted image features with classifiers such as SVMs, neural networks, or random forests.
- Deep learning and CNN-based models produced significant improvements over traditional emotion-recognition approaches.
- The paper identifies insufficient attention to important facial regions as a missing component in prior emotion-detection systems.
III. THE PROPOSED FRAMEWORK
The proposed end-to-end model combines a shallow convolutional network with spatial attention to focus on relevant face regions. It uses an affine spatial transformer and regularized classification training.
- The framework uses an attentional convolutional network trained from scratch with fewer than 10 layers for facial-expression classification.
- A spatial transformer focuses the model on important facial regions by estimating an attended sampling area and warping the input with an affine transformation.
- The feature extractor contains four convolutional layers, pooling and ReLU operations, dropout, and two fully connected layers.
- Training minimizes classification cross-entropy together with an ℓ2 weight-regularization term using the Adam optimizer.
- Dropout and ℓ2 regularization support training from scratch on small datasets, while deeper networks of more than 50 layers did not improve accuracy substantially.
IV. EXPERIMENTAL RESULTS
The experiments evaluate the model on four facial-expression databases, compare its performance with prior work, and visualize the salient regions detected by the trained model.
- The experimental analysis covers database descriptions, performance comparisons on four facial-expression databases, and visualization of detected salient regions.
A. Databases
The experiments use four facial-expression datasets spanning real faces, posed and spontaneous expressions, and stylized characters. Their sizes and image conditions vary substantially across FER2013, CK+, JAFFE, and FERG.
- FER2013: FER2013 contains 35,887 48x48 images, mostly captured in the wild, labeled with six expressions plus neutral.Its original split includes 28,709 training images and 3,589 images each for validation and testing.
- CK+: CK+ contains 593 sequences across 123 subjects and includes both posed and spontaneous expressions.Previous works commonly use the last frame of each sequence for image-based recognition.
- JAFFE: JAFFE contains 213 images of seven posed facial expressions from 10 Japanese female models.Each image was rated on six emotion adjectives by 60 Japanese subjects.
- FERG: FERG contains 55,767 annotated face images of six stylized characters, grouped into seven expression types.The database evaluates performance on cartoonish characters modeled using MAYA.
B. Experimental Analysis and Comparison
The proposed model is evaluated with dataset-specific training, validation, and test procedures across four databases. Reported accuracies are 70.02% on FER2013, around 99.3% on FERG, and around 92.8% on JAFFE, with comparisons to prior work.
- Evaluation procedure: The evaluation trains on a subset of each dataset, validates on a validation set, and reports accuracy over the test set.One model is trained per dataset while the architecture and hyperparameters are kept similar across models.
- FER2013: 70.02% accuracy is achieved on the FER2013 test set.The model uses all 28,709 training images and 3,589 test images; errors are more frequent for undersampled classes such as disgust and fear.
- Comparison with prior work: The paper compares its results with previous works on FER2013, FERG, JAFFE, and CK+.These comparisons are presented in Tables I–IV.
- FERG: Around 99.3% accuracy is achieved on the FERG test set.Approximately 34k images are used for training, 14k for validation, and 7k for testing, with 1k test images randomly selected per expression.
- JAFFE: Around 92.8% overall accuracy is achieved on JAFFE.The split uses 120 training images, 23 validation images, and 70 test images, or 10 test images per emotion.
C. Model Visualization
The visualization method identifies face regions that influence expression predictions by occluding image patches. Saliency varies across emotions and images: mouth regions are often important, while neutral expressions may require information across the entire face.
- Visualization method: Occluding square image regions and checking prediction changes identifies candidate regions important for classifying a specific expression.A region is considered potentially important when its removal causes an incorrect expression prediction.
- Visualization method: The examples compare how zeroing different regions affects predictions for happy and angry faces in JAFFE images.
- Emotion-specific regions: Neutral-expression saliency can cover the entire face, indicating that eyes, lips, eyebrows, and forehead all contribute to classification.
- Emotion-specific regions: For happiness and fear, regions around the mouth are more important than other facial areas.
- Image variation: Saliency maps can differ between images with the same expression because gestures and image variation alter the detected important regions.
- Image variation: Fear images show similar mouth-focused regions, but one image also highlights the forehead, plausibly because of visible forehead lines.
V. CONCLUSION
The paper proposes an attentional convolutional network for facial expression recognition and argues that attention helps shallow networks compete with deeper ones. It also reports experiments across four databases and visualization of salient facial regions.
- The framework uses an attentional convolutional network to focus on salient facial regions for expression recognition.
- Attention is presented as enabling networks with fewer than 10 layers to compete with and outperform deeper emotion-recognition networks.
- Experiments on four facial expression recognition databases produced promising results.
- A visualization method highlights face regions that are most crucial for detecting different facial expressions.