Source-linked AI summary
DeXpression: Deep Convolutional Neural Network for Expression Recognition
Peter Burkert, Felix Trier, Muhammad Zeshan Afzal, Andreas Dengel, Marcus Liwicki
TL;DR
Facial expression recognition can improve machine communication and safety-related systems, but earlier approaches often rely on handcrafted features. DeXpression uses a fully automatic CNN with parallel feature extraction and evaluates it on CKP and MMI, achieving high reported recognition accuracy while mainly confusing neutral or visually similar expressions.
Problem
Facial expression recognition is important for interpreting nonverbal communication and supporting applications such as human-computer interaction and driver safety.
Method
DeXpression is a fully automatic CNN that uses the FeatEx parallel feature-extraction block, combining convolutional, pooling, and ReLU layers to learn image representations.
Results
99.6% CKP and 98.36% MMI average recognition accuracy are reported from 10-fold cross-validation, with performance described as competitive with state-of-the-art approaches.
Takeaways & Limitations
DeXpression performs very well on both tested datasets when an emotion is present, supporting its reported applicability to emotion recognition.
Takeaways & Limitations
Misclassifications usually occur in early sequence frames depicting neutral expressions, while visually similar emotions such as Surprise and Disgust can also be confused.
Abstract
from arXiv · showhide
We propose a convolutional neural network (CNN) architecture for facial expression recognition. The proposed architecture is independent of any hand-crafted feature extraction and performs better than the earlier proposed convolutional neural network based approaches. We visualize the automatically extracted features which have been learned by the network in order to provide a better understanding. The standard datasets, i.e. Extended Cohn-Kanade (CKP) and MMI Facial Expression Databse are used for the quantitative evaluation. On the CKP set the current state of the art approach, using CNNs, achieves an accuracy of 99.2%. For the MMI dataset, currently the best accuracy for emotion recognition is 93.33%. The proposed architecture achieves 99.6% for CKP and 98.63% for MMI, therefore performing better than the state of the art using CNNs. Automatic facial expression recognition has a broad spectrum of applications such as human-computer interaction and safety systems. This is due to the fact that non-verbal cues are important forms of communication and play a pivotal role in interpersonal communication. The performance of the proposed architecture endorses the efficacy and reliable usage of the proposed work for real world applications.
1 INTRODUCTION
Facial expression recognition supports richer human–machine communication because emotions convey important nonverbal information. The paper motivates fully automatic CNN-based recognition for applications including human-computer interaction, surveillance, and driver safety.
- Motivation: Correctly interpreting emotions is important for understanding encoded feelings in human communication.
- Applications: Emotion recognition can support more natural communication between people and machines, including chat systems that insert emotion-based emoticons.The paper also identifies surveillance and driver-safety applications, where detected mood may inform system adaptation.
- Existing approaches: Handcrafted facial landmarks and Action Units provide established cues, but landmark detection can be difficult because facial geometry varies across people.Texture changes offer another cue for expression recognition.
- Proposed direction: The proposed approach uses neural networks to learn image features with less manual intervention and supports raw-image processing.The architecture is inspired by CNN-based image feature extraction and is evaluated on CKP and MMI datasets.
2 RELATED WORK
Prior expression-recognition methods combine facial patches, handcrafted descriptors, classifiers, and deep CNNs. Reported performance varies across approaches and datasets, with CNNs reaching 99.2% accuracy on CKP.
- Patch-based methods: A facial-patch pipeline extracts features from high-variance regions and classifies them with an SVM, reaching 94.09% average accuracy under 10-fold cross-validation.
- CNN approaches: A five-layer deep CNN with convolutional, pooling, local-filter, and fully connected layers achieves 99.2% accuracy on CKP.Dropout is used to reduce overfitting.
- Video recognition: A video-based three-dimensional CNN using groups of five consecutive frames achieves 95% accuracy on a database containing 10 people.
- Handcrafted and matching methods: Handcrafted descriptor methods report 95.1% CKP accuracy with boosted LBP features, while robust NCC reports 100% on the Extended Cohn-Kanade Database.
3 CONVOLUTIONAL NEURAL NETWORKS
CNNs transform image inputs through learned convolutions, pooling, nonlinear activations, fully connected layers, and class outputs. These components extract representations while reducing dimensionality and producing class predictions.
- Convolutional layers: Convolutional layers apply learned filters to local input regions, and shared neighboring weights reduce the number of parameters that must be trained.Multiple filters provide a richer representation of the input.
- Pooling: Max pooling reduces an input by taking local maxima and provides translational invariance relative to the filter size.
- Activation: ReLU computes a nonnegative activation, creating sparse representations with inexpensive computation and avoiding the nonlinear-gradient behavior associated with vanishing gradients.The passage contrasts ReLU with sigmoid and binary units.
- Fully connected layers: Fully connected layers connect every neuron in one layer to every neuron in the next through a weight matrix.
- Output and softmax: The output layer represents the input image’s class as a one-hot vector whose dimensionality equals the number of classes.The network propagates error through a softmax layer mapping inputs to class probabilities.
4 DATASETS
The paper evaluates facial-expression recognition on MMI and CKP, which contain varied depictions of six basic emotions; CKP also includes Contempt.
- MMI Dataset: MMI contains over 2,900 videos and images from 75 people, annotated with action units and six basic emotions.The emotions are Anger, Disgust, Fear, Happiness, Sadness, and Surprise.
- CKP Dataset: CKP contains 593 emotion-labeled sequences from 210 people aged 18–50, covering seven emotions including Contempt.The dataset includes both grayscale and colored images.
- MMI Dataset: MMI examples show substantial within-emotion variation in eyebrows, eyes, mouth, forehead, lips, and chin across people.The figure illustrates differing expression strength and facial-region configurations for the six emotions.
- CKP Dataset: CKP examples likewise vary across facial regions, and Contempt is omitted because no annotated image depicts it.The CKP figure compares expression differences while excluding the unavailable Contempt example.
5 PROPOSED ARCHITECTURE
The proposed compact CNN automatically preprocesses facial images, extracts multiscale features through two FeatEx blocks, and classifies emotions while visualizing learned representations.
- Architecture: The compact four-part CNN preprocesses images, extracts features with two FeatEx blocks, and classifies inputs through a fully connected layer.Its design explicitly considers training speed, real-time use, and resource consumption.
- Architecture: Table 1 reports the output dimensionality produced by each layer of the architecture.
- FeatEx Block: FeatEx uses parallel convolutional and pooling paths with different filter scales, concatenating them into a more diverse facial representation.The block uses 1 × 1 and 3 × 3 convolutions, ReLU activation, and max pooling; using it twice yields good results.
- Feature Visualization: The first network stages create low-level edge features, while the first FeatEx block forms multiscale paths and the second refines and reduces dimensionality.The visualization presents concatenated FeatEx features as an abstract representation.
6 EXPERIMENTS AND RESULTS
The architecture is evaluated with 10-fold cross-validation on CKP and MMI, achieving high average accuracy on both datasets and outperforming the cited MMI comparison.
- CKP Results: 99.6% average accuracy is achieved on CKP using all 5,870 annotated images in 10-fold cross-validation.The accompanying comparison lists state-of-the-art approaches, while a cited 100% result used hand-picked images without cross-validation.
- CKP Results: The CKP confusion matrix shows that some emotions are recognized perfectly.
- MMI Results: 98.63% average accuracy is achieved on MMI with 10-fold cross-validation, exceeding the accuracies reported by Wang and Yin.The evaluation uses 3,740 automatically extracted representative frames from the videos.
7 DISCUSSION
DeXpression shows strong recognition on CKP and MMI, with most errors concentrated in early, often neutral frames and a few visually similar emotions.
- Error analysis: Most CKP misclassifications occur in the first frames of emotion sequences, where neutral expressions are often still present.The authors characterize these errors as partly reflecting data selection rather than a major model failure.
- Error analysis: 0.045% is the highest reported CKP confusion rate, occurring when Surprise is confused with Disgust.Fear is also confused with Surprise and Anger in described examples because their facial regions can look similar.
- Overall assessment: DeXpression produces very few errors on images containing an emotion across both tested datasets.The discussion attributes remaining confusions mainly to images without clearly displayed emotions.
8 CONCLUSION AND FUTURE WORK
DeXpression is a fully automatic, computationally light CNN architecture that uses the FeatEx structure for feature representation. It achieves high cross-validated accuracy on CKP and MMI, while future applications could model more nuanced emotions and intensities.
- Conclusion: 99.6% CKP and 98.36% MMI recognition accuracy are obtained with 10-fold cross-validation.The conclusion presents these results as evidence that the architecture can compete with current state-of-the-art approaches.
- Conclusion: DeXpression works fully automatically and uses FeatEx, a composed structure of convolutional, max-pooling, and ReLU layers.FeatEx is described as creating a rich feature representation of the input.
- Future Work: Future applications could distinguish emotions such as Nervousness and Panic, composed emotions, and different levels of emotional strength.The paper connects these extensions to real-environment applications, including early panic detection and evaluating reactions.