Source-linked AI summary
Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution
Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, Zhengyou Zhang
TL;DR
Crowd-sourced labels are inexpensive but noisy, creating a challenge for learning facial-expression classifiers. The paper re-labels FER images with 10 taggers, trains DCNNs under four schemes, and finds that methods using the label distribution outperform majority voting. FER+ provides multiple labels per image for this setting, although the authors note an unresolved training–testing mismatch for multi-label learning.
Problem
Crowd-sourced labels are cheap but noisier than expert annotations, motivating methods that learn facial-expression recognition from noisy labels.
Method
The paper re-labels each FER image with 10 crowd taggers and compares four DCNN training schemes for using the resulting label distributions.
Results
Using multiple labels improves classification accuracy over the traditional single-label majority-voting approach, with probabilistic label drawing and cross-entropy loss performing best.
Takeaways & Limitations
FER+ supplies multiple labels per face image for evaluating training methods that leverage crowd-sourced label distributions.
Takeaways & Limitations
The authors hypothesize that multi-label learning may underperform because training uses label distributions while testing uses only the majority emotion, and further work is needed.
Abstract
from arXiv · showhide
Crowd sourcing has become a widely adopted scheme to collect ground truth labels. However, it is a well-known problem that these labels can be very noisy. In this paper, we demonstrate how to learn a deep convolutional neural network (DCNN) from noisy labels, using facial expression recognition as an example. More specifically, we have 10 taggers to label each input image, and compare four different approaches to utilizing the multiple labels: majority voting, multi-label learning, probabilistic label drawing, and cross-entropy loss. We show that the traditional majority voting scheme does not perform as well as the last two approaches that fully leverage the label distribution. An enhanced FER+ data set with multiple labels for each face image will also be shared with the research community.
1. INTRODUCTION
Facial-expression recognition supports human-computer interaction, neuroscience, and computer-vision research, but crowd-sourced labels are cheaper and noisier than expert coding. The paper evaluates four DCNN training schemes using multiple crowd labels to address this noise.
- Facial and body cues convey emotions important to daily communication and social interaction.
- FACS provides a standard facial-expression coding scheme, but requires professionally trained coders and has limited unconstrained-image datasets.
- Crowd-sourcing holistic facial-emotion labels costs about 1–2 US cents, but produces noisier labels because emotions are subjective and workers are low-paid.
- The paper re-labels each FER image with 10 crowd taggers and compares majority voting, multi-label learning, probabilistic label drawing, and cross-entropy loss in DCNN training.
- Probabilistic label drawing and cross-entropy loss are reported as the most effective schemes for training emotion-recognition classifiers from noisy labels.
2. RELATED WORK
Prior work addresses facial-expression recognition with handcrafted features and DCNNs, while related emotion research considers multiple simultaneous or distributed labels. Crowd-sourcing research mainly focuses on improving annotation quality through monitoring, filtering, and gold-standard controls.
- Crowd-sourcing quality-control methods include gold-standard items, annotator filtering, and real-time monitoring of accuracy and fatigue.
- FER and FER+ examples compare original FER labels with FER+ labels obtained after majority voting.
- Facial-expression recognition has used handcrafted features such as Gabor Wavelets, LBP-TOP, PHOG, and LPQ, followed more recently by DCNNs.
- Related emotion-recognition studies model non-exclusive emotions through multi-label classification, emotion mixtures, and emotion distribution learning.
3. THE FER+ DATA SET
FER+ re-tags FER images with crowd-sourced emotion labels, collecting distributions rather than relying only on one label. The dataset uses 10 annotators per image and supports four training schemes that exploit these multiple labels.
- The original FER dataset contains web-crawled face images filtered by human labelers, but its label accuracy is not very high.
- Each FER+ image is assigned one of eight emotion types by crowd taggers required to choose a single emotion.
- 10 taggers label each image after stopping at two-agreement tagging produced unsatisfactory quality, yielding an emotion distribution for every face.
- With 3 taggers agreement is 46%, with 5 it is about 67%, and with 7 it exceeds 80% against the 10-label majority reference.
- The resulting 10-annotator distributions support majority voting, multi-label learning, probabilistic label drawing, and cross-entropy-loss training.
4. DCNN LEARNING
The paper uses a custom VGG13 DCNN for eight-class facial-expression recognition and tests four ways to train it from crowd-sourced label distributions. These schemes range from majority-vote targets to methods that retain or sample the distribution across training.
- Network architecture: The custom VGG13 model processes 64×64 grayscale faces and outputs eight emotion classes through convolutional, pooling, dropout, dense, and soft-max layers.The model has 10 convolution layers and two 1024-node dense layers; dropout is used throughout the architecture.
- Training setup: The network is trained from scratch on FER+ using the original FER train-validation-test split and on-the-fly affine data augmentation.The augmentation applies transforms similar to prior work and targets robustness to translation, rotation, and scaling.
- Label-distribution schemes: The study compares majority voting, multi-label learning, probabilistic label drawing, and cross-entropy loss for exploiting crowd-sourced label distributions.The four schemes differ in whether they collapse the distribution, accept multiple sufficiently supported emotions, sample labels, or directly match the distribution.
- Majority voting: Majority voting converts each crowd-label distribution into a single target emotion by selecting the emotion with the largest vote share.The resulting target distribution is one-hot, with the selected emotion receiving the target mass.
- Multi-label learning: Multi-label learning treats multiple emotions as acceptable when each has sufficient annotator support, using a 30% threshold in the experiments.The method chooses among supported emotions according to the network output and applies a lower cost when the output agrees with any accepted emotion.
- Probabilistic label drawing: Probabilistic label drawing samples an emotion from each image’s label distribution during every training epoch, allowing different labels across epochs.Across multiple epochs, the authors expect this repeated sampling to approach the underlying label distribution on average.
5. EXPERIMENTAL RESULTS
Experiments on FER+ compare four training schemes using crowd-sourced label distributions, with PLD and CEL achieving the strongest accuracy and outperforming majority voting. Performance remains weaker for disgust and contempt, which have few labeled training examples, while the explanation for ML's lower performance remains unverified.
- The FER+ label distribution was formed from 10 taggers per image, with frequencies below one reset to zero and the remaining frequencies normalized.The resulting distribution was used to test all four training schemes.
- Test accuracy was measured against the majority emotion as the single label for each image.
- PLD and CEL achieved the best test accuracy, exceeding MV by more than 1%, with a t-value around 3.1 and estimated statistical significance of 99%-99.5%.The difference between PLD and CEL was within the standard deviation.
- ML performed worse than PLD and CEL, possibly because majority-emotion testing creates a training-testing mismatch; this hypothesis requires further verification.
- The best-performing network performed well on most emotions but struggled with disgust and contempt because FER+ contains very few examples labeled with them.
6. CONCLUSIONS
The paper concludes that using multiple crowd-sourced labels per image improves facial-expression classification accuracy over majority voting. It also makes the enhanced FER+ dataset available for download.
- Using multiple labels per image improves classification accuracy compared with the traditional single-label majority-voting approach.
- The FER+ dataset is available for download through the Microsoft FERPlus repository.