Source-linked AI summary
Facial expression and attributes recognition based on multi-task learning of lightweight neural networks
Andrey V. Savchenko
TL;DR
Lightweight facial-analysis models must balance accuracy with deployment practicality, while identity features require adaptation for emotion recognition. The paper develops a sequential multi-task pipeline using lightweight CNNs and detector-based crops, achieving near-state-of-the-art facial-attribute and emotion results, including strong video-recognition performance.
Problem
Complex facial-analysis models are often impractical for mobile and edge devices, and face-identification features cannot be directly used for emotion recognition.
Method
The paper sequentially trains MobileNet, EfficientNet, and RexNet-based networks from face identification through facial attributes and fine-tuned emotion recognition, using cropped faces without margins.
Results
The models achieve near-state-of-the-art results across age, gender, ethnicity, and emotion tasks, while video features provide state-of-the-art single-model performance on AFEW and VGAF.
Takeaways & Limitations
The trained models provide lightweight facial representations suitable for fast decision-making in embedded and mobile intelligent systems.
Takeaways & Limitations
The study uses traditional classifiers on extracted features, so some results do not reach state-of-the-art performance.
Abstract
from arXiv · showhide
In this paper, the multi-task learning of lightweight convolutional neural networks is studied for face identification and classification of facial attributes (age, gender, ethnicity) trained on cropped faces without margins. The necessity to fine-tune these networks to predict facial expressions is highlighted. Several models are presented based on MobileNet, EfficientNet and RexNet architectures. It was experimentally demonstrated that they lead to near state-of-the-art results in age, gender and race recognition on the UTKFace dataset and emotion classification on the AffectNet dataset. Moreover, it is shown that the usage of the trained models as feature extractors of facial regions in video frames leads to 4.5% higher accuracy than the previously known state-of-the-art single models for the AFEW and the VGAF datasets from the EmotiW challenges. The models and source code are publicly available at https://github.com/HSE-asavchenko/face-emotion-recognition.
1 Introduction
The paper addresses the need for practical facial analytics by developing lightweight multi-task CNNs that retain high accuracy across facial analysis tasks. Its simplified training procedure targets mobile and edge-device deployment.
- Modern facial analytics increasingly uses complex deep CNNs for age, gender, ethnicity, and emotion prediction.
- Complex facial-analysis methods can be impractical for mobile applications and edge devices.
- The paper proposes a simplified training procedure for lightweight, accurate CNNs supporting multiple facial analysis tasks.
- The paper proceeds from related literature and datasets to the proposed approach, experiments, and conclusions.
2 Related Works
Prior work spans image and video emotion recognition, facial-attribute prediction, and multi-task learning, but many high-performing CNNs remain too computationally demanding for practical applications. The paper therefore focuses on lightweight architectures.
- AffectNet is a major benchmark for static-image emotion classification, with strong results from large or specialized models.
- AFEW and VGAF evaluate video-based and group-level emotion recognition in EmotiW challenges.
- Age, gender, and ethnicity recognition commonly uses CNNs trained on IMDB-Wiki or UTKFace.
- Large-scale face-recognition pre-training has been reported to benefit facial-attribute prediction by learning efficient face representations.
- Existing multi-task approaches improve facial analysis, but their best CNNs can have running times too high for many practical applications.
- The paper concentrates on lightweight CNN architectures, including EfficientNet and RexNet.
3 Proposed Approach
The proposed system adapts a face-identification network into a sequential multi-task pipeline for attributes, emotion, and video recognition. It uses detector-based crops without margins, lightweight backbones, task-specific heads, and aggregated facial features.
- 3.1 Multi-task networks: The base CNN is pre-trained on face identification with VGGFace2, then adapted using MTCNN crops without margins.
- 3.1 Multi-task networks: MobileNet, EfficientNet, and RexNet serve as lightweight backbone architectures for face recognition.
- 3.1 Multi-task networks: The shared feature vector feeds simple classifiers for gender and ethnicity, while age prediction adds layers before its final fully connected layer.
- 3.1 Multi-task networks: Face-identification features are fine-tuned on emotion data because identity features cannot be directly reused for emotion recognition.
- 3.2 Training details: Training proceeds sequentially from face identification to separate age, gender, and ethnicity heads, followed by emotion fine-tuning.
- 3.2 Training details: Emotion training uses weighted categorical cross-entropy, initially learning a new head with frozen weights and then optimizing all weights.
- 3.3 Video-based facial attribute recognition: For video recognition, fine-tuned networks extract frame-level facial features that are aggregated into video descriptors.
- 3.3 Video-based facial attribute recognition: Only 2,619 training and 741 validation videos contain at least one detected face, so videos without detected faces are excluded or assigned zero descriptors.
4 Experimental Results
The experiments evaluate the proposed lightweight pipeline for face identification, facial-expression recognition, video emotion classification, and age, gender, and ethnicity recognition. Results show that detector-cropped faces without margins and task-specific fine-tuning are important for expression recognition, while the models achieve competitive accuracy with low computational cost.
- Face identification: The LFW experiment evaluates rank-1 face-identification accuracy for models trained by the proposed pipeline.The models trained by the pipeline are marked in the comparison table.
- Face identification: 1–2% higher accuracy is obtained when detected faces replace center crops for the proposed descriptors.For the conventional SENet descriptor, detector-cropped faces instead produce a 0.5% higher error rate than loosely cropped faces.
- Facial expression recognition on single images: Identity features from pre-trained CNNs are not sufficient for reliable facial-expression recognition, whereas MTCNN-cropped faces perform better.The comparison includes models whose base networks remain from face-recognition pre-training and models further trained for expression classification.
- Facial expression recognition on single images: EfficientNet-based models improve the known state-of-the-art accuracy on AffectNet for both 8 and 7 emotional classes.The experiments also emphasize selecting detector bounding boxes without added margins.
- Facial expression recognition on single images: The trained CNNs have relatively small parameter counts and running times for emotion prediction, although ResNet-18 has comparable speed.Performance is reported in Table 3 using measurements from an Intel Core i7-8750H laptop.
- Video-based facial expression recognition: 59.89% and 59.63% validation accuracy are obtained by EfficientNet-B0 and EfficientNet-B2, respectively, on 379 AFEW videos containing detected faces.EfficientNet-B0 is reported as practically reaching the 59.42% best-known ensemble accuracy, while the full validation set was not processed because of face-detection mistakes.
- Video-based facial expression recognition: 70.31% and 68.29% accuracy are obtained by MobileNet and EfficientNet-B0, respectively, on 741 VGAF videos with at least one detected face.The proposed approach is reported as the best known single model for VGAF, while the challenge-winning audio-video ensemble reaches 74.28%.
- Facial attributes recognition: The models achieve at least 2.5% higher gender-classification accuracy and 0.6 lower age MAE than DEX on UTKFace.The MobileNet-based multi-task model uses substantially fewer parameters than two VGG-16 DEX models and reaches 97.5% gender accuracy with 5.39 age MAE on a specified test subset.
5 Conclusion
The proposed training pipeline produces lightweight CNNs with high accuracy, speed, and model compactness for facial expression recognition and other facial-analysis tasks. However, traditional classifiers limited some results compared with state-of-the-art methods.
- The novel training pipeline achieves state-of-the-art accuracy for lightweight neural networks in facial expression recognition across several image and video datasets.
- Pre-training on VGGFace2 and using detector-defined face crops without margins provide high accuracy alongside excellent speed and small model size.
- The resulting models are suitable for fast decision-making in embedded systems, including mobile intelligent systems.
- Using only traditional classifiers means that some results do not reach state-of-the-art performance.