Source-linked AI summary
Suppressing Uncertainties for Large-Scale Facial Expression Recognition
Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, Yu Qiao
TL;DR
Large-scale FER datasets contain uncertainty from ambiguous or low-quality images and subjective or incorrect annotations, creating a challenge for deep-learning-based recognition. The paper proposes SCN, which weights and ranks training samples and carefully relabels low-ranked samples. SCN is evaluated on synthetic data, WebEmotion, and public benchmarks, where it reports 88.14% on RAF-DB, 60.23% on AffectNet, and 89.35% on FERPlus.
Problem
Uncertain facial images and noisy annotations make large-scale FER datasets difficult to annotate reliably and can cause deep networks to over-fit uncertain samples.
Method
SCN combines self-attention importance weighting, ranking regularization, and careful relabeling of samples in the lowest-ranked group.
Results
SCN reports 88.14% on RAF-DB, 60.23% on AffectNet, and 89.35% on FERPlus, described as new records on these benchmarks.
Takeaways & Limitations
Synthetic and WebEmotion experiments validate SCN's effectiveness for reducing the impact of uncertainty in large-scale FER.
Abstract
from arXiv · showhide
Annotating a qualitative large-scale facial expression dataset is extremely difficult due to the uncertainties caused by ambiguous facial expressions, low-quality facial images, and the subjectiveness of annotators. These uncertainties lead to a key challenge of large-scale Facial Expression Recognition (FER) in deep learning era. To address this problem, this paper proposes a simple yet efficient Self-Cure Network (SCN) which suppresses the uncertainties efficiently and prevents deep networks from over-fitting uncertain facial images. Specifically, SCN suppresses the uncertainty from two different aspects: 1) a self-attention mechanism over mini-batch to weight each training sample with a ranking regularization, and 2) a careful relabeling mechanism to modify the labels of these samples in the lowest-ranked group. Experiments on synthetic FER datasets and our collected WebEmotion dataset validate the effectiveness of our method. Results on public benchmarks demonstrate that our SCN outperforms current state-of-the-art methods with \textbf{88.14}\% on RAF-DB, \textbf{60.23}\% on AffectNet, and \textbf{89.35}\% on FERPlus. The code will be available at \href{https://github.com/kaiwang960112/Self-Cure-Network}{https://github.com/kaiwang960112/Self-Cure-Network}.
1. Introduction
Large-scale in-the-wild FER is difficult because ambiguous, low-quality facial images and subjective annotation create uncertainty. SCN addresses this by weighting samples, ranking their importance, and carefully relabeling low-ranked samples.
- 1. Introduction: Large-scale Internet FER datasets are difficult to annotate consistently because facial ambiguity, image quality, and annotator subjectivity create uncertain labels.These uncertainties can produce inconsistent or incorrect annotations and encourage over-fitting to uncertain samples.
- 1. Introduction: SCN is proposed to reduce uncertainty in large-scale facial expression recognition through self-attention importance weighting, ranking regularization, and noise relabeling.The method is designed to reduce the impact of uncertain samples during training.
- 1. Introduction: Self-attention weighting assigns sample importance, ranking regularization separates high- and low-importance samples, and relabeling modifies labels in the lowest-ranked group.The ranking mechanism encourages reliable samples to receive higher importance and uncertain samples to receive lower importance.
- 1. Introduction: The paper evaluates SCN on synthetic FER data and the real-world uncertain WebEmotion dataset.WebEmotion was collected from the Internet to investigate SCN under extreme uncertainty.
2. Related Work
FER systems combine face detection, feature extraction, and expression recognition, using engineered or learned representations. Prior work addresses noisy labels, while ambiguity and image-quality uncertainty receive less attention.
- 2. Related Work: A typical FER system includes face detection, feature extraction, and expression recognition.Detected faces may also be aligned before feature extraction.
- 2. Related Work: Engineered FER features include texture-based, geometry-based, and hybrid representations.Examples include SIFT, HOG, LBP, Gabor coefficients, and facial landmark features.
- 2. Related Work: Learning-based FER methods use shallow or deep CNNs, facial action units, and region-based attention to model expression-related features.Region-based attention methods address pose and occlusion awareness.
- 2. Related Work: FER uncertainty arises from ambiguous expressions, low-quality images, inconsistent annotations, and incorrect annotations, with noisy-label learning receiving greater prior attention.The passage states that the other uncertainty aspects are rarely explored.
3. Self-Cure Network
Self-Cure Network combines self-attention weighting, ranking regularization, and relabeling to reduce the impact of uncertain facial-expression samples during training.
- 3. Self-Cure Network: SCN uses a backbone CNN, self-attention importance weighting, ranking regularization, and relabeling as its three core modules.The attention module learns sample weights, ranking regularization separates high- and low-importance groups, and relabeling targets selected low-importance samples.
- 3.2. Self-Attention Importance Weighting: The self-attention module assigns each image an importance weight from facial features for loss weighting, with uncertain samples expected to receive lower weights.Weights are generated from facial features using a fully connected layer and sigmoid activation, then used to weight logits.
- 3.3. Rank Regularization: Rank regularization sorts attention weights into high- and low-importance groups and enforces a margin between their mean weights.The regularization loss uses the difference between the group means and a margin parameter δ1; the total loss combines RR-Loss and LW CE.
- 3. Self-Cure Network: SCN mainly suppresses uncertainty through re-weighting, while relabeling modifies only some uncertain samples.The method describes a possible progression from low weighting to low-importance grouping and subsequent relabeling, with corrected samples potentially receiving higher weights later.
- 3.4. Relabeling: The relabeling module examines only low-importance samples and assigns a pseudo-label when the maximum predicted probability exceeds the given-label probability by a threshold.The comparison uses Softmax probabilities, with δ2 controlling the required difference.
- Implementation: The default training setup divides each batch into 70% high-importance and 30% low-importance samples and jointly optimizes RR-Loss with WCE-Loss.The implementation uses end-to-end training with a batch size of 1024; the two losses are combined at a 1:1 ratio by default.
4. Experiments
Experiments evaluate SCN under synthetic and real-world annotation uncertainties, then analyze its learned weights, modules, and hyperparameters. Across these settings, SCN improves FER performance and suppresses or corrects uncertain samples.
- Experimental setup: Experiments cover synthetic label noise, WebEmotion pretraining, original FER datasets, qualitative weight visualizations, ablations, and hyperparameter evaluations.The evaluated datasets include RAF-DB, FERPlus, AffectNet, and the collected WebEmotion dataset.
- Synthetic uncertainties: 13.80%, 1.07%, and 1.91% are SCN’s gains over the baseline at 30% noise on RAF-DB, FERPlus, and AffectNet under scheme i), respectively.Under scheme ii) at the same noise ratio, gains are 2.20%, 2.47%, and 3.12%; improvements become more apparent as noise increases.
- Real-world uncertainties: 6.31%, 0.7%, and 1.02% are SCN’s gains from scratch on RAF-DB, AffectNet, and FERPlus, while pretrained gains are 2.83%, 1.73%, and 1.21%, respectively.The reported improvements suggest that the public datasets contain annotation uncertainties, including ambiguity and low-quality images.
- Real-world uncertainties: Low-importance RAF-DB images are associated with low quality and occlusion, which are described as making annotation difficult.Figure 4 presents ten examples for each basic emotion and includes their ground-truth labels in the text.
- Ablation studies: 76.26% is the RAF-DB result after adding WCE-Loss to the 72% baseline, and RR-Loss plus relabeling further improve WCE-Loss by 2.15%.The ablation identifies re-weighting as the most contributing module; equal weighting of RR-Loss and WCE-Loss performs best, while increasing RR-Loss weight from 0.5 to 0.8 degrades performance.