Source-linked AI summary
WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition
Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, Jie Zhou
TL;DR
Face recognition research lacks sufficiently large public training data and practical time-constrained evaluations. The paper constructs WebFace260M and cleaned WebFace42M with CAST, alongside the FRUITS protocol and test set; WebFace42M improves IJB-C performance and achieves third place among 430 NIST-FRVT submissions. The benchmark provides million-scale data and evaluations spanning deployment time constraints, while attribute bias remains a limitation.
Problem
Public face-recognition training data are limited and evaluation protocols are nearly saturated, while time-constrained benchmarking remains difficult to access.
Method
The paper collects WebFace260M, cleans it with iterative CAST self-training, and constructs the FRUITS time-constrained protocol and attribute-rich test set.
Results
WebFace42M reduces relative IJB-C error by 40% and achieves overall third place among 430 NIST-FRVT submissions.
Takeaways & Limitations
WebFace42M and the accompanying benchmark substantially expand public data and practical evaluation coverage for million-scale face recognition.
Takeaways & Limitations
WebFace260M may contain attribute bias that can cause performance drops over certain attributes in real-world applications.
Abstract
from arXiv · showhide
In this paper, we contribute a new million-scale face benchmark containing noisy 4M identities/260M faces (WebFace260M) and cleaned 2M identities/42M faces (WebFace42M) training data, as well as an elaborately designed time-constrained evaluation protocol. Firstly, we collect 4M name list and download 260M faces from the Internet. Then, a Cleaning Automatically utilizing Self-Training (CAST) pipeline is devised to purify the tremendous WebFace260M, which is efficient and scalable. To the best of our knowledge, the cleaned WebFace42M is the largest public face recognition training set and we expect to close the data gap between academia and industry. Referring to practical scenarios, Face Recognition Under Inference Time conStraint (FRUITS) protocol and a test set are constructed to comprehensively evaluate face matchers. Equipped with this benchmark, we delve into million-scale face recognition problems. A distributed framework is developed to train face recognition models efficiently without tampering with the performance. Empowered by WebFace42M, we reduce relative 40% failure rate on the challenging IJB-C set, and ranks the 3rd among 430 entries on NIST-FRVT. Even 10% data (WebFace4M) shows superior performance compared with public training set. Furthermore, comprehensive baselines are established on our rich-attribute test set under FRUITS-100ms/500ms/1000ms protocol, including MobileNet, EfficientNet, AttentionNet, ResNet, SENet, ResNeXt and RegNet families. Benchmark website is https://www.face-benchmark.org.
1. Introduction
Face recognition progress is driven by architectures, loss functions, and benchmarks, but academia remains constrained by limited training data and saturated evaluation protocols. The paper addresses this gap with a million-scale benchmark and time-constrained evaluation framework.
- Motivation: Academia is restricted by limited training data and nearly saturated test protocols despite advances in networks, losses, and benchmarks.Public datasets include MegaFace2 with 4.7M faces from 672K subjects and MS1M with 10M faces from 100K celebrities but around 50% noise.
- Motivation: Face benchmarks empower researchers to train and evaluate high-performance face-recognition systems.
- Contribution: The benchmark introduces million-scale WebFace data and a time-constrained evaluation protocol for practical face-recognition scenarios.The contribution includes extensive experiments and baselines under the FRUITS protocol.
2. WebFace260M and WebFace42M
WebFace260M is collected from large celebrity name resources and Internet image searches, while WebFace42M is the cleaned, attribute-annotated subset. WebFace42M substantially exceeds prior public datasets in scale and has a lower estimated noise ratio.
- Data Collection: The celebrity name list combines 1M names from MS1M with names collected from IMDB, while commercial search engines provide ranked identity-specific images.The IMDB website contains nearly 4M celebrity names, though some subjects have no public images.
- WebFace42M: WebFace42M contains 2M subjects and 42M faces, with 3 to more than 300 faces per identity and 21 faces per identity on average.
- Attributes: WebFace42M includes seven face-attribute annotations and covers broad ranges of pose, age, and major races.The annotations include pose, age, race, gender, hat, glass, and mask.
- Scale and Quality: WebFace42M has 2M identities and 42M images versus 672K and 4.7M for MegaFace2, and 100K and 10M for MS1M.Its estimated noise ratio is lower than 10%, compared with more than 30% for MegaFace2 and 50% for MS1M.
3. Cleaning Automatically by Self-Training
CAST cleans noisy open-set face data through iterative self-training, using teacher-generated embeddings for intra-class and inter-class filtering. Replacing each teacher with a better-trained student progressively improves data quality and yields 2,059,906 identities with 42,474,558 faces.
- Motivation: Manual and semi-automatic cleaning pipelines require expensive labor, while uncleaned web data can contain near 50% noise and degrade trained-model performance.
- CAST Design: CAST applies self-training to open-set face recognition data, where direct pseudo-label generation is impractical, and emphasizes embedding quality during cleaning.
- CAST Design: CAST trains an initial teacher on MS1MV2, cleans 260M images, trains a student on the cleaned data, and iteratively switches student and teacher.The process continues until high-quality 42M faces are obtained.
- Cleaning Procedure: Intra-class cleaning clusters 512-dimensional teacher embeddings with DBSCAN and retains the largest cluster containing more than two faces per subject.
- Cleaning Procedure: Inter-class cleaning merges folders above cosine similarity 0.7 and deletes the smaller folder between 0.5 and 0.7.
- Output: 2,059,906 identities and 42,474,558 faces remain in WebFace42M after duplicate and test-set-overlap removal.
4. FRUITS Protocol
FRUITS addresses the limitations of accuracy-focused and impractical time-constrained evaluations by measuring complete face-recognition systems under deployment-oriented inference budgets. It combines 100-, 500-, and 1000-millisecond tracks with an attribute-rich test set and FNMR-at-FMR evaluation.
- Motivation: Existing protocols largely pursue accuracy, while FLOPs- and model-size constraints do not directly represent inference time or include face detection and alignment.NIST-FRVT uses time constraints but limits submissions to one every four calendar months.
- Protocol design: FRUITS measures complete face-recognition systems on a single Intel Xeon CPU core, including detection, alignment, feature embedding, and matching.The protocol is designed to support comprehensive academic evaluation of face matchers.
- Tracks: FRUITS-100 targets lightweight systems for mobile devices, FRUITS-500 targets local surveillance networks, and FRUITS-1000 targets capable models deployed on clouds.The tracks impose 100-, 500-, and 1000-millisecond limits, respectively.
- Test set: The manually constructed test set contains 38,578 faces from 2,225 identities with annotated age, race, gender, and controlled-or-wild attributes.Multi-ethnic annotators select familiar celebrities and are encouraged to gather attribute-balanced faces.
- Metrics: FRUITS evaluates 1:1 verification across attribute subsets using FNMR at specified FMR, where lower FNMR at the same FMR is better.The protocol includes comparisons across age, race, gender, and scenario subsets.
5. Experiments of Million-level Recognition
The experiments show that distributed training makes million-scale recognition practical, while larger and cleaned WebFace data improves performance across models, datasets, and time-constrained tracks. The benchmark also exposes distinct accuracy and deployment trade-offs among architectures.
- 5.2. Distributed Training: Parallelizing feature X and center W with FP16 reduces training cost while preserving comparable performance.With 32 nodes and 256 GPUs, scaling efficiency exceeds 80%, reducing ResNet-100 training from 233 hours to 9 hours on WebFace42M.
- 5.3. Comparisons of Training Data: WebFace42M consistently improves recognition over public training sets, and WebFace4M outperforms MS1M families and MegaFace2 despite using only 10% of the full data.The comparison spans standard verification, RFW, MegaFace, IJB-C, and the authors’ attribute-rich test set.
- 5.3. Comparisons of Training Data: 40% relative error reduction on IJB-C raises TAR from 96.03% to 97.70% at 10^-4 FAR versus MS1MV2.On the authors’ test set, WebFace42M produces a relative promotion near 70%.
- 5.3. Comparisons of Training Data: Performance improves consistently as training data increases from WebFace4M to WebFace12M to WebFace42M, benefiting both ResNet-100 and lightweight ResNet-14 models.The ResNet-14 experiment uses 10%, 30%, and 100% portions of WebFace42M.
- 5.4. Cleaning Ablations: CAST becomes purer and more effective through successive iterations, with accuracy increasing through the third iteration and saturating at the fourth.The authors therefore set CAST’s iteration number to three.
- 5.4. Cleaning Ablations: DBSCAN reaches 96.55% TAR at FAR=1e-4 on IJB-C, exceeding K-Means and slightly surpassing the evaluated GCN-based strategies.The authors select DBSCAN for intra-class cleaning because GCN methods can be sub-optimal on extremely noisy folders.
- 5.5. Baselines under FRUITS Protocols: Under FRUITS-100, the best lightweight baseline obtains 5.88% FNMR at FMR=1e-5, leaving substantial room for improvement.The best baseline is RegNet-800MF with lightweight detection and recognition modules.
- 5.5. Baselines under FRUITS Protocols: Under FRUITS-500, ResNet-100 performs best overall, ResNet-50 with flip testing leads Wild and Male attribute indicators, and ResNeXt ranks first in Cross-scene.These results show that the leading architecture depends on the evaluation dimension.
6. Discussion and Conclusion
The discussion acknowledges possible attribute bias in WebFace260M while describing balancing, attribute-based evaluation, and access controls. The conclusion presents the benchmark, cleaned data, protocol, distributed framework, and reported recognition results as the paper’s main contributions.
- Discussion: WebFace260M may contain attribute bias, although the authors address it through diverse sources, balanced test-set collection, and attribute-based evaluation.The discussion covers race, gender, and age attributes.
- Discussion: Bias may reduce performance on certain attributes, while balanced sampling and recent de-biasing research may help alleviate the issue.The authors state that balanced training data can be sampled from the extremely large WebFace260M.
- Conclusion: The paper concludes with a cleaned dataset of 42M images from 2M identities, a rich-attribute test set, FRUITS, distributed training, baselines, and a final model.The conclusion also reports a 40% IJB-C failure-rate reduction and third place among 430 NIST-FRVT entries.
- Discussion: Dataset access is restricted to applicants who sign a license, with research-purpose use as the stated goal.The authors describe strict access rules on the benchmark website.