Source-linked AI summary
Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition
Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, Tao Mei
TL;DR
Annotation ambiguity in FER arises because facial expressions are subjective and visually similar, making single labels unreliable for some images. DMUE mines latent label distributions and estimates pairwise uncertainty to guide learning, and it reports leading results on real-world and synthetic benchmarks without extra inference burden.
Problem
FER annotation ambiguity matters because people may assign inconsistent expression categories, while large datasets make image-level label distributions expensive to obtain.
Method
DMUE uses auxiliary multi-branch learning to mine latent label distributions and pairwise semantic-feature relationships to estimate each sample's ambiguity.
Results
89.42% on RAF-DB and 63.11% on AffectNet were reported as best performance, with evaluations also covering synthetic noisy datasets.
Takeaways & Limitations
DMUE describes ambiguous instances more finely, flexibly integrates with existing backbones, and imposes no extra burden during inference.
Abstract
from arXiv · showhide
Due to the subjective annotation and the inherent interclass similarity of facial expressions, one of key challenges in Facial Expression Recognition (FER) is the annotation ambiguity. In this paper, we proposes a solution, named DMUE, to address the problem of annotation ambiguity from two perspectives: the latent Distribution Mining and the pairwise Uncertainty Estimation. For the former, an auxiliary multi-branch learning framework is introduced to better mine and describe the latent distribution in the label space. For the latter, the pairwise relationship of semantic feature between instances are fully exploited to estimate the ambiguity extent in the instance space. The proposed method is independent to the backbone architectures, and brings no extra burden for inference. The experiments are conducted on the popular real-world benchmarks and the synthetic noisy datasets. Either way, the proposed DMUE stably achieves leading performance.
1. Introduction
FER faces annotation ambiguity because people may assign inconsistent expression categories, while label distributions can better represent uncertain visual content. DMUE addresses this through latent distribution mining and pairwise uncertainty estimation, achieving leading benchmark results without extra inference burden.
- Motivation: Annotation ambiguity remains an obstacle because different people may assign inconsistent categories to the same facial image.A user study found an image annotated as Anger was most often judged Neutral, while another had only a 20% gap between its top two classes.
- Motivation: Label distributions can describe the possibility of belonging to each expression class more faithfully than a single deterministic label.The motivation is especially strong when the most and secondary possible classes are close or disagree with the original annotation.
- Approach: DMUE mines latent label distributions online with temporary auxiliary branches and estimates ambiguity using pairwise relationships between samples.The mined distributions provide semantic features, while the uncertainty module combines annotations with relationship statistics.
- Approach: The uncertainty module guides the model to dynamically adjust learning focus between mined distributions and original annotations at the sample level.This design targets differing ambiguity extents across samples rather than applying a fixed weighting.
- Results: 89.42% on RAF-DB and 63.11% on AffectNet were reported as best performance, while the approach adds no extra inference burden.The method was evaluated on real-world benchmarks and synthetic noisy datasets.
2. Related Work
Earlier FER methods used handcrafted or learned features, while newer work addresses ambiguity through noisy-label learning, uncertainty estimation, latent-truth discovery, and label enhancement. FER ambiguity is particularly subjective for compound expressions, whose descriptions vary among users.
- FER Methods: FER algorithms span handcrafted features based on expression-related folds and geometry changes and learning-based methods that dominate more recent work.Examples of learning-based strategies include decoupling identity information and exploiting differences between expressive images.
- Ambiguity in FER: Recent ambiguity-focused FER methods use multiple training phases, semi-online label-distribution learning, or confidence and latent-truth estimation.These approaches target annotation inconsistency or harmful effects from ambiguous data through different training mechanisms.
- Learning with Ambiguity: General ambiguity handling includes noisy-label learning, uncertainty estimation, latent-truth discovery, and label enhancement for low-quality images.Prior methods use model predictions, auxiliary embeddings, temperature softmax, or artificial uncertainty to improve supervision.
- Learning with Ambiguity: Compound-expression ambiguity in FER is subjective because different users provide varied label descriptions for the same image.This distinguishes the FER setting from approaches designed primarily for mislabelled annotations or low data quality.
3. Method
DMUE addresses FER annotation ambiguity by mining latent label distributions and estimating sample-level uncertainty from pairwise semantic relationships. Its multi-branch architecture jointly uses mined distributions and annotations while dynamically adjusting their learning influence.
- Latent Distribution Mining: DMUE represents each image’s ambiguity with a latent distribution over all classes except its annotated class.The latent distribution is normalized and jointly learned with the deterministic annotation.
- Latent Distribution Mining: C auxiliary branches mine latent distributions online, while a same-architecture target branch produces the final C-class prediction.The auxiliary branches share lower-level features and each predicts a (C −1)-class distribution for samples annotated to another class.
- Latent Distribution Mining: The target branch matches sharpened latent distributions with bounded L2 loss, which is less sensitive to inaccurate predictions than cross-entropy.Gradients are not backpropagated through the computation of the mined distributions.
- Overall Loss Function: The confidence-based modulator reduces cross-entropy for more ambiguous samples, allowing latent distributions to provide greater guidance.Training first emphasizes auxiliary branches and later shifts attention to the target branch after the mined distributions stabilize.
- Pairwise Uncertainty Estimation: Pairwise uncertainty estimation concatenates class-wise average cosine similarities with an anchor’s one-hot annotation to characterize its ambiguity.A two-layer module maps the resulting 2C-dimensional batch features to one confidence scalar per image.
4. Experiments
DMUE is evaluated on synthetic noisy annotations and real-world FER benchmarks, with ablations and visual analyses examining its components and behavior. It consistently improves ambiguity-aware recognition, including leading benchmark performance and robustness to noisy labels.
- Experimental setup: Experiments cover synthetic noisy annotations and four in-the-wild FER benchmarks, with ablations across components and backbone architectures.The study also evaluates performance on RAF-DB, AffectNet, FERPlus, and SFEW.
- Experimental setup: Only the target branch is retained during testing, so auxiliary branches add no inference-time burden.Training uses shared low-level layers with separated auxiliary branches, while testing keeps only the target branch.
- Evaluation on Synthetic Ambiguity: At 30% synthetic label noise, DMUE improves accuracy by 4.29% on RAF-DB and 4.21% on AffectNet.The experiments compare DMUE with baseline models and SCN, repeating each experiment three times and reporting mean accuracy and standard deviation.
- Component Analysis: Latent distribution mining is the strongest individual component, while uncertainty estimation adds 0.74% on AffectNet and 0.91% on RAF-DB over latent distribution mining alone.Similarity preserving also improves results, but its contribution is smaller than those of the other components.
- Visualization Analysis: Visualizations show latent distributions qualitatively consistent with human intuition and pairwise uncertainty scores stable across different batches.Figure 5 illustrates that excluding a misleading Neutral class can produce a prediction that better reflects a Sad image’s truth.
5. Conclusion
DMUE addresses FER annotation ambiguity through latent distribution mining and pairwise uncertainty estimation. It describes ambiguous instances more finely, estimates ambiguity from sample relationships, adds no inference burden, and integrates with existing architectures.
- DMUE combines latent distribution mining with pairwise uncertainty estimation to address annotation ambiguity in facial expression recognition.
- Mined latent distributions describe ambiguous instances in a fine-grained way and guide the model.
- Pairwise relationships between samples are exploited to estimate each instance’s ambiguity degree.
- DMUE imposes no extra inference burden and can be integrated with existing network architectures.
- Experiments on popular benchmarks and synthetic ambiguous datasets show DMUE’s effectiveness.
6. Appendix
The appendix analyzes DMUE’s training regularization, latent distribution mining, backbone flexibility, and uncertainty estimation. These analyses report more stable training, evolving and noise-correcting distributions, improvements across architectures, and uncertainty scores aligned with subjective ambiguity.
- Similarity-preserving regularization: Lsp regularizes branches to predict consistent similarity values for image pairs, making training more stable.The loss rises slightly around the 20k-th iteration as ramp functions increase the target-branch loss weight.
- Latent distribution mining: DMUE can correct noisy annotations by mining latent distributions from synthetically mislabelled training samples.
- Different backbone networks: DMUE stably improves performance across ShuffleNetV1 and MobileNetV2 on AffectNet and RAF-DB.
- Latent distribution mining: LD-A averages predictions from auxiliary branches trained on the whole image batch, while LD-N denotes the original mining approach.
- Latent distribution mining: The latent distribution evolves from random initial predictions toward representations reflecting a sample’s visual features.
- Latent distribution mining: Auxiliary branches disentangle naturally entangled expression connections by mapping images to latent classes.
- Mathematical analysis: For a sample feature f, the decomposition f = f1 + f2 separates its class-center component from an orthogonal component f2.The appendix states that f2 is orthogonal to the class center and that E{⟨x, f2⟩} = 0 under the modeled distribution.
- Uncertainty estimation: Lower estimated uncertainty scores correspond to more ambiguous images and align with subjective perception.The uncertainty module is intended to suppress adverse influence from ambiguous data while encouraging use of semantic features and latent distributions.