Source-linked AI summary
Real-time Convolutional Neural Networks for Emotion and Gender Classification
Octavio Arriaga, Matias Valdenegro-Toro, Paul Plöger
TL;DR
The paper addresses the difficulty of deploying accurate CNN-based facial analysis in real-time robot systems. It proposes compact fully convolutional architectures and an integrated vision pipeline, reporting 96% gender accuracy on IMDB and 66% emotion accuracy on FER-2013. The system also includes real-time guided back-propagation visualization and was deployed on a Care-O-bot 3 robot.
Problem
Large CNN architectures are difficult to deploy in real-time robot systems, while learned CNN features often remain hidden.
Method
The paper builds fully convolutional real-time CNNs using Global Average Pooling, residual modules, depth-wise separable convolutions, and guided back-propagation visualization.
Results
The system reports 96% accuracy for IMDB gender classification, 66% for FER-2013 emotion classification, and integrated deployment on Care-O-bot 3.
Takeaways & Limitations
The proposed models support integrated face detection, gender classification, and emotion classification in a real-time robot vision pipeline.
Takeaways & Limitations
The trained gender-classification CNNs were empirically found to be biased toward western facial features and facial accessories.
Abstract
from arXiv · showhide
In this paper we propose an implement a general convolutional neural network (CNN) building framework for designing real-time CNNs. We validate our models by creating a real-time vision system which accomplishes the tasks of face detection, gender classification and emotion classification simultaneously in one blended step using our proposed CNN architecture. After presenting the details of the training procedure setup we proceed to evaluate on standard benchmark sets. We report accuracies of 96% in the IMDB gender dataset and 66% in the FER-2013 emotion dataset. Along with this we also introduced the very recent real-time enabled guided back-propagation visualization technique. Guided back-propagation uncovers the dynamics of the weight changes and evaluates the learned features. We argue that the careful implementation of modern CNN architectures, the use of the current regularization methods and the visualization of previously hidden features are necessary in order to reduce the gap between slow performances and real-time architectures. Our system has been validated by its deployment on a Care-O-bot 3 robot used during RoboCup@Home competitions. All our code, demos and pre-trained architectures have been released under an open-source license in our public repository.
I. INTRODUCTION
The paper targets robust, computationally efficient facial analysis for real-time robot interaction, where conventional CNNs are difficult to deploy. It proposes a real-time CNN framework validated through integrated face, gender, and emotion classification.
- Real-time robot interaction requires facial analysis that can extract emotional state and gender from faces despite high sample variability.
- Millions of parameters in state-of-the-art CNNs make direct deployment on hardware-constrained robot platforms and real-time systems unfeasible.
- The proposed framework designs real-time CNNs and validates them in a facial-expression system combining face detection, gender classification, and emotion classification.
- The integrated system was deployed on a Care-O-bot 3 robot and extended to general robot platforms and RoboCup@Home challenges.
- Real-time guided-gradient back-propagation visualization was implemented to validate the features learned by the CNN.
II. RELATED WORK
Prior CNN-based facial-expression systems achieved strong FER-2013 accuracy but commonly relied on large fully connected layers or CNN ensembles. These architectures motivate reducing parameter counts for efficient deployment.
- VGG16 contains approximately 90% of its parameters in the final fully connected layers.
- Global Average Pooling reduces each feature map to a scalar by averaging all elements, reducing reliance on large final layers.
- 71% accuracy on FER-2013 was reported for a square-hinged-loss CNN using approximately 5 million parameters.
- 98% of the parameters in that FER-2013 architecture were located in its final fully connected layers.
- The second-best methods in the cited comparison achieved 66% accuracy using an ensemble of CNNs.
III. MODEL
The paper develops fully convolutional architectures that remove fully connected layers and use compact convolutional designs for real-time facial classification. The mini-Xception model combines residual and depth-wise separable convolutions to reduce parameters while retaining reported accuracy.
- Sequential fully-CNN: The sequential fully-CNN uses nine convolution layers, ReLUs, batch normalization, and Global Average Pooling, with approximately 600,000 parameters.Its final convolution layer matches the number of feature maps to the number of classes before softmax activation.
- Sequential fully-CNN: 96% accuracy was achieved on the IMDB gender dataset, while 66% accuracy was achieved on FER-2013 emotion classification.The IMDB dataset contains 460,723 RGB images, and FER-2013 contains 35,887 grayscale images across seven emotion classes.
- Mini-Xception: Depth-wise separable convolutions separate spatial cross-correlations from channel cross-correlations using depth-wise and pointwise convolution layers.
- Mini-Xception: Depth-wise separable convolutions reduce computation relative to standard convolutions by a factor of 1/N + 1/D^2.
- Mini-Xception: Mini-Xception contains four residual depth-wise separable convolutions and approximately 60,000 parameters, a 10× reduction versus the initial implementation and 80× versus the original CNN.
- Runtime evaluation: The combined face-detection, gender-classification, and emotion-classification pipeline takes 0.22 ± 0.0003 ms on an i5-4210M CPU.The reported runtime corresponds to a 1.5× speedup compared with Tang’s original architecture.
- Visualization: Guided back-propagation reconstructs input-image derivatives with respect to higher-level feature-map elements while filtering negative gradients.
IV. RESULTS
The system demonstrates integrated real-time face, gender, and emotion inference, while visualizations reveal learned features and recurring classification errors.
- The complete pipeline integrates face detection, emotion classification, and gender classification in the Care-O-bot 3 robot.
- Common emotion errors include predicting “sad” instead of “fear” and “angry” instead of “disgust”.
- Guided back-propagation visualizations show activations associated with frowns, teeth, eyebrows, and widened eyes across emotion classes.
- The visualizations help explain misclassification of people wearing glasses as “angry”, because dark frames can resemble frowning features.
- The mini-Xception model learns more interpretable features than the sequential fully-CNN, while the provided materials include real-time emotion and combined gender-emotion demonstrations.
V. FUTURE WORK
The paper identifies training-data and accessory-related biases that constrain gender and emotion classification, while guided visualizations support further bias analysis.
- Gender-classification CNNs were empirically biased toward western facial features and facial accessories.
- The authors hypothesize that predominantly western training images contributed to these gender-classification misclassifications.
- Glasses may interfere with emotion classification and gender classification, potentially because many glasses-wearing images carried the “man” label.
- Figure 8 compares FER-2013 emotion samples with guided back-propagation visualizations from the mini-Xception and sequential fully-CNN models.
VI. CONCLUSIONS
The paper concludes that compact CNN designs can support integrated real-time face, gender, and emotion classification while retaining interpretable feature visualizations.
- The proposed architectures reduce parameters by 80× while obtaining favorable classification results.
- The integrated module performs face detection, gender classification, and emotion classification in a single system.
- The complete pipeline was successfully integrated into a Care-O-bot 3 robot.
- Guided back-propagation visualizes high-level features learned by the models.