Source-linked AI summary
Suppressing Uncertainties for Large-Scale Facial Expression Recognition
Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, Yu Qiao
TL;DR
Large-scale FER datasets contain uncertainty from ambiguous or low-quality images and subjective or incorrect annotations, creating a challenge for deep-learning-based recognition. The paper proposes SCN, which weights and ranks training samples and carefully relabels low-ranked samples. SCN is evaluated on synthetic data, WebEmotion, and public benchmarks, where it reports 88.14% on RAF-DB, 60.23% on AffectNet, and 89.35% on FERPlus.
Problem
Uncertain facial images and noisy annotations make large-scale FER datasets difficult to annotate reliably and can cause deep networks to over-fit uncertain samples.
Source: §1 · p. 1, §3.4 · p. 4Method
SCN combines self-attention importance weighting, ranking regularization, and careful relabeling of samples in the lowest-ranked group.
Source: §1 · pp. 1–2Results
SCN reports 88.14% on RAF-DB, 60.23% on AffectNet, and 89.35% on FERPlus, described as new records on these benchmarks.
Source: §1 · p. 2Takeaways & Limitations
Synthetic and WebEmotion experiments validate SCN's effectiveness for reducing the impact of uncertainty in large-scale FER.
Source: §1 · p. 2, §4.2 · p. 5, §4.3 · p. 7
Abstract
from arXiv · showhide
Annotating a qualitative large-scale facial expression dataset is extremely difficult due to the uncertainties caused by ambiguous facial expressions, low-quality facial images, and the subjectiveness of annotators. These uncertainties lead to a key challenge of large-scale Facial Expression Recognition (FER) in deep learning era. To address this problem, this paper proposes a simple yet efficient Self-Cure Network (SCN) which suppresses the uncertainties efficiently and prevents deep networks from over-fitting uncertain facial images. Specifically, SCN suppresses the uncertainty from two different aspects: 1) a self-attention mechanism over mini-batch to weight each training sample with a ranking regularization, and 2) a careful relabeling mechanism to modify the labels of these samples in the lowest-ranked group. Experiments on synthetic FER datasets and our collected WebEmotion dataset validate the effectiveness of our method. Results on public benchmarks demonstrate that our SCN outperforms current state-of-the-art methods with \textbf{88.14}\% on RAF-DB, \textbf{60.23}\% on AffectNet, and \textbf{89.35}\% on FERPlus. The code will be available at \href{https://github.com/kaiwang960112/Self-Cure-Network}{https://github.com/kaiwang960112/Self-Cure-Network}.
1. Introduction
Large-scale in-the-wild FER is difficult because ambiguous, low-quality facial images and subjective annotation create uncertainty. SCN addresses this by weighting samples, ranking their importance, and carefully relabeling low-ranked samples.
1. Introduction
- Large-scale Internet FER datasets are difficult to annotate consistently because facial ambiguity, image quality, and annotator subjectivity create uncertain labels.These uncertainties can produce inconsistent or incorrect annotations and encourage over-fitting to uncertain samples.Source: §1 · p. 1
- SCN is proposed to reduce uncertainty in large-scale facial expression recognition through self-attention importance weighting, ranking regularization, and noise relabeling.The method is designed to reduce the impact of uncertain samples during training.Source: §1 · pp. 1–2
- Self-attention weighting assigns sample importance, ranking regularization separates high- and low-importance samples, and relabeling modifies labels in the lowest-ranked group.The ranking mechanism encourages reliable samples to receive higher importance and uncertain samples to receive lower importance.Source: §1 · p. 2
- The paper evaluates SCN on synthetic FER data and the real-world uncertain WebEmotion dataset.WebEmotion was collected from the Internet to investigate SCN under extreme uncertainty.Source: §1 · p. 2
2. Related Work
FER systems combine face detection, feature extraction, and expression recognition, using engineered or learned representations. Prior work addresses noisy labels, while ambiguity and image-quality uncertainty receive less attention.
2. Related Work
- A typical FER system includes face detection, feature extraction, and expression recognition.Detected faces may also be aligned before feature extraction.Source: §2.1 · p. 2
- Engineered FER features include texture-based, geometry-based, and hybrid representations.Examples include SIFT, HOG, LBP, Gabor coefficients, and facial landmark features.Source: §2.1 · p. 2
- Learning-based FER methods use shallow or deep CNNs, facial action units, and region-based attention to model expression-related features.Region-based attention methods address pose and occlusion awareness.Source: §2.1 · p. 2
- FER uncertainty arises from ambiguous expressions, low-quality images, inconsistent annotations, and incorrect annotations, with noisy-label learning receiving greater prior attention.The passage states that the other uncertainty aspects are rarely explored.Source: §2.2 · p. 2
3. Self-Cure Network
Self-Cure Network combines self-attention weighting, ranking regularization, and relabeling to reduce the impact of uncertain facial-expression samples during training.
3. Self-Cure Network
- SCN uses a backbone CNN, self-attention importance weighting, ranking regularization, and relabeling as its three core modules.The attention module learns sample weights, ranking regularization separates high- and low-importance groups, and relabeling targets selected low-importance samples.Source: §3.1 · p. 3, §3 · p. 3, §3.3 · p. 3
3.2. Self-Attention Importance Weighting
- The self-attention module assigns each image an importance weight from facial features for loss weighting, with uncertain samples expected to receive lower weights.Weights are generated from facial features using a fully connected layer and sigmoid activation, then used to weight logits.Source: §3.2 · p. 3, §3.1 · p. 3
3.3. Rank Regularization
- Rank regularization sorts attention weights into high- and low-importance groups and enforces a margin between their mean weights.The regularization loss uses the difference between the group means and a margin parameter δ1; the total loss combines RR-Loss and LW CE.Source: §3.3 · p. 3
3. Self-Cure Network
- SCN mainly suppresses uncertainty through re-weighting, while relabeling modifies only some uncertain samples.The method describes a possible progression from low weighting to low-importance grouping and subsequent relabeling, with corrected samples potentially receiving higher weights later.Source: Figure 2 · p. 4, §3.4 · p. 4
3.4. Relabeling
- The relabeling module examines only low-importance samples and assigns a pseudo-label when the maximum predicted probability exceeds the given-label probability by a threshold.The comparison uses Softmax probabilities, with δ2 controlling the required difference.Source: §3.4 · pp. 3–4
Implementation
- The default training setup divides each batch into 70% high-importance and 30% low-importance samples and jointly optimizes RR-Loss with WCE-Loss.The implementation uses end-to-end training with a batch size of 1024; the two losses are combined at a 1:1 ratio by default.Source: §3.5 · pp. 4–5
4. Experiments
Experiments evaluate SCN under synthetic and real-world annotation uncertainties, then analyze its learned weights, modules, and hyperparameters. Across these settings, SCN improves FER performance and suppresses or corrects uncertain samples.
Experimental setup
- Experiments cover synthetic label noise, WebEmotion pretraining, original FER datasets, qualitative weight visualizations, ablations, and hyperparameter evaluations.The evaluated datasets include RAF-DB, FERPlus, AffectNet, and the collected WebEmotion dataset.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 5
Synthetic uncertainties
- 13.80%, 1.07%, and 1.91% are SCN’s gains over the baseline at 30% noise on RAF-DB, FERPlus, and AffectNet under scheme i), respectively.Under scheme ii) at the same noise ratio, gains are 2.20%, 2.47%, and 3.12%; improvements become more apparent as noise increases.Source: §4.2 · p. 5
Real-world uncertainties
- 6.31%, 0.7%, and 1.02% are SCN’s gains from scratch on RAF-DB, AffectNet, and FERPlus, while pretrained gains are 2.83%, 1.73%, and 1.21%, respectively.The reported improvements suggest that the public datasets contain annotation uncertainties, including ambiguity and low-quality images.Source: §4.3 · p. 7
- Low-importance RAF-DB images are associated with low quality and occlusion, which are described as making annotation difficult.Figure 4 presents ten examples for each basic emotion and includes their ground-truth labels in the text.Source: §4.3 · p. 7
Ablation studies
- 76.26% is the RAF-DB result after adding WCE-Loss to the 72% baseline, and RR-Loss plus relabeling further improve WCE-Loss by 2.15%.The ablation identifies re-weighting as the most contributing module; equal weighting of RR-Loss and WCE-Loss performs best, while increasing RR-Loss weight from 0.5 to 0.8 degrades performance.Source: §4.4 · p. 7