Source-linked AI summary
The Devil of Face Recognition is in the Noise
Fei Wang, Liren Chen, Cheng Li, Shiyao Huang, Yanjie Chen, Chen Qian, Chen Change Loy
TL;DR
Large-scale face-recognition datasets contain label noise, but its sources, effects, and optimal annotation strategies remain insufficiently understood. The paper cleans subsets of MegaFace and MS-Celeb-1M, builds noise-controlled IMDb-Face, and studies noise types and labeling. It finds that clean data can match full noisy datasets with far fewer samples, while label flips are more damaging than outliers and IMDb-Face is competitive despite its smaller size.
Problem
Large-scale face-recognition datasets inevitably contain label noise because well-annotated data are prohibitively expensive and time-consuming to collect.
Method
The paper manually cleans MegaFace and MS-Celeb-1M subsets, builds IMDb-Face from IMDb images, injects controlled label corruption, and evaluates noise types and annotation strategies.
Results
32% of cleaned MegaFace or 20% of cleaned MS-Celeb-1M matched respective full-dataset performance; label flips were more harmful than outliers, and IMDb-Face was competitive despite its smaller size.
Takeaways & Limitations
Data source and cleaning can improve face-recognition accuracy, while IMDb-Face provides a relatively clean resource for training and future noise studies.
Takeaways & Limitations
Human verification results estimate performance on face pairs likely to share an identity, not arbitrary face-pair verification.
Abstract
from arXiv · showhide
The growing scale of face recognition datasets empowers us to train strong convolutional networks for face recognition. While a variety of architectures and loss functions have been devised, we still have a limited understanding of the source and consequence of label noise inherent in existing datasets. We make the following contributions: 1) We contribute cleaned subsets of popular face databases, i.e., MegaFace and MS-Celeb-1M datasets, and build a new large-scale noise-controlled IMDb-Face dataset. 2) With the original datasets and cleaned subsets, we profile and analyze label noise properties of MegaFace and MS-Celeb-1M. We show that a few orders more samples are needed to achieve the same accuracy yielded by a clean subset. 3) We study the association between different types of noise, i.e., label flips and outliers, with the accuracy of face recognition models. 4) We investigate ways to improve data cleanliness, including a comprehensive user study on the influence of data labeling strategies to annotation accuracy. The IMDb-Face dataset has been released on https://github.com/fwang91/IMDb-Face.
1 Introduction
Large-scale face-recognition datasets make deep learning effective but inevitably introduce costly label noise. This paper studies its sources and consequences, releases cleaned resources, and evaluates data-cleaning and annotation strategies.
- Large-scale face-recognition datasets inevitably contain label noise because well-annotated data are prohibitively expensive and time-consuming to collect.
- MegaFace and MS-Celeb-1M include both easy-to-remove and difficult incorrect identity labels.
- 32% of cleaned MegaFace or 20% of cleaned MS-Celeb-1M matched the respective full dataset’s performance, while noisy training required a few orders more samples.
- IMDb-Face contains 1.7M images of 59K celebrities collected from IMDb movie screenshots and posters.
- Label-flip noise harmed face-recognition accuracy more than outliers, and accuracy decreased rapidly and nonlinearly as label noise increased.
- The study combines cleaned datasets, noise-controlled IMDb-Face, and annotation experiments to understand noise and balance labeling cost against accuracy.
2 How Noisy is Existing Data?
The paper profiles noise in representative face-recognition datasets using sampled manual categorization and analyzes how noise is distributed across large public datasets. MegaFace and MS-Celeb-1M show long-tailed identity sizes, with different noise patterns and dataset-specific artifacts.
- Figure 2 visualizes dataset size and estimated noise percentages, then shows noise distributions for MS-Celeb-1M(v1) and MegaFace with lower- and upper-bounds.
- The study estimates signal-to-noise ratios by manually categorizing random samples as correct, doubtful, or wrong identity assignments.Exact noise counts are infeasible at full dataset scale, so the analysis uses sampled subsets.
- MS-Celeb-1M and MegaFace have long-tailed identity-size distributions, with most identities containing very few images.
- MegaFace’s automatically formed clusters can split the same identity across clusters, while its clean portion contains many duplicated images.
- The study reports fewer noises across all identity-size groups in MegaFace than in MS-Celeb-1M.
3 Building a Noise-Controlled Face Dataset
IMDb-Face is constructed from structured IMDb imagery to provide more diverse face data, then cleaned through a human-annotation study and large-scale labeling process. The study compares annotation schemes and uses the results to guide dataset cleaning.
- Data collection: IMDb provides diverse celebrity imagery, including official photos, lifestyle photos, and movie snapshots that are rarely returned by search engines.The paper motivates IMDb as a source that can broaden pose and scene variation beyond search-engine results.
- Dataset diversity: IMDb-Face exhibits larger pose variations and diversity in age, gender, and race than the compared face datasets.Figure 4 compares yaw distributions with MS-Celeb-1M and MegaFace and shows demographic distributions for IMDb-Face.
- User study: The user study evaluates three annotation schemes: drawing a box around the target face, choosing one face from three, and selecting the target face.Each query face was annotated nine times across the schemes, and agreement by at least four annotations determined the assigned identity.
- User study results: Scheme I achieves the highest F1 score and recalls more than 90% of faces with under 10% false positive samples.Scheme II maintains a high true positive rate at low false positive rates, while Scheme III performs worse in that regime.
- Cleaning strategy: The authors recommend combining Schemes I and II for high precision and recall, but used only Scheme I to clean IMDb-Face because of budget limitations.The paper reports that Scheme I was the scheme used for the full dataset cleaning process.
- Dataset cleaning: Fifty annotators cleaned IMDb-Face for one month, producing 1.7M clean facial images from 2M raw images with an estimated noise level under 2%.The estimated noise level combines an approximated raw-data noise level of 2.7 ± 4.5% with Scheme I’s 8.7% false positive rate.
4 Experiments
The experiments isolate how label noise affects face recognition, compare noisy and cleaned public datasets, and evaluate IMDb-Face against established training sources and state-of-the-art methods.
- 4.1 Investigating the Effect of Noise on IMDb-Face: Label flips severely deteriorate model performance more than outliers, whose effect is less abrupt across losses.These results come from experiments that gradually introduce different noise types into IMDb-Face.
- 4.1 Investigating the Effect of Noise on IMDb-Face: A-Softmax becomes worse than Center loss and Softmax in the high-noise region despite performing better on clean data.
- 4.2 The Effect of Noise on MegaFace and MS-Celeb-1M: 4.14% average accuracy improvement is obtained by comparing cleaned and sampled datasets on MegaFace.Clean subsets even surpass raw datasets that are four times larger on average.
- 4.2 The Effect of Noise on MegaFace and MS-Celeb-1M: Over 5% accuracy improvements on MegaFace are observed for A-Softmax and Center-loss when trained on clean datasets.The authors attribute this benefit to focusing more on hard examples rather than meaningless noise.
- 4.3 Comparing IMDb-Face with other Face Datasets: IMDb-Face is competitive as a training source despite being smaller than several established datasets.The comparison uses CelebFaces, CASIA-WebFace, MS-Celeb-1M(v1), and MegaFace under the same training conditions across three losses.
- 4.4 Comparisons with State-of-the-Arts: A single model trained on IMDb-Face achieves state-of-the-art performance on LFW, MegaFace, and YTF against published methods.The final model is also comparable to a few private methods on MegaFace.
5 Conclusion
The paper studies face recognition from the data perspective by analyzing label noise, collecting IMDb-Face, and evaluating data sources and cleaning. Its experiments show that cleaner data improves accuracy and supports controlled noise studies and benchmarking.
- The study analyzes label-noise sources and consequences while collecting IMDb data as a naturally cleaner source than search engines.
- User studies identify an effective and accurate way to clean the collected data.
- Extensive experiments show that both data source and cleaning improve face recognition accuracy.
- A clean dataset supports controlled studies of training performance as a function of noise level and benchmarking of automatic data-cleaning methods.