Source-linked AI summary
Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not?
Erjin Zhou, Zhimin Cao, Qi Yin
TL;DR
The paper asks how large web-collected datasets affect face recognition and studies this through a simple CNN trained on a large labeled face database. The resulting system reaches 99.50% accuracy on LFW, while real-world security evaluation exposes a gap between benchmark performance and application requirements.
Problem
The paper investigates how big data affects recognition performance and whether high benchmark accuracy reflects real-world security performance.
Method
The authors collect labeled web data, train a convolutional network on the MFC database, and evaluate it on LFW and the CHID security benchmark.
Results
99.50% recognition accuracy is achieved on LFW, while the true positive rate is 66% at a 10^-5 false positive rate on the real-world security benchmark.
Takeaways & Limitations
High LFW accuracy does not imply acceptable performance in a domain-specific real-world security application, and the paper highlights data bias, low false-positive criteria, and cross factors as continuing challenges.
Takeaways & Limitations
Web-collected celebrity faces differ substantially from daily-life images, so LFW performance still hardly meets real-world application requirements.
Abstract
from arXiv · showhide
Face recognition performance improves rapidly with the recent deep learning technique developing and underlying large training dataset accumulating. In this paper, we report our observations on how big data impacts the recognition performance. According to these observations, we build our Megvii Face Recognition System, which achieves 99.50% accuracy on the LFW benchmark, outperforming the previous state-of-the-art. Furthermore, we report the performance in a real-world security certification scenario. There still exists a clear gap between machine recognition and human performance. We summarize our experiments and present three challenges lying ahead in recent face recognition. And we indicate several possible solutions towards these challenges. We hope our work will stimulate the community's discussion of the difference between research benchmark and real-world applications.
1. INTRODUCTION
The paper examines how accumulated web-collected data affects face recognition, builds a Megvii system from these observations, and contrasts high LFW accuracy with weaker real-world security performance.
- The LFW benchmark evaluates face recognition in unconstrained environments and has become a de-facto standard for in-the-wild performance.
- Large amounts of labeled web data and a convolutional network framework reveal that data distribution and size influence recognition performance.
- The Megvii system uses straightforward convolutional networks and a large web-collected labeled dataset to achieve 99.50% recognition accuracy on LFW.
- 66% true positive rate at a 10^-5 false positive rate does not meet the real security application’s requirement.
- The paper identifies data bias, very low false positive criteria, and cross factors as three persistent face-recognition challenges.
2. A DATA PERSPECTIVE TO FACE RECOGNITION
The paper frames LFW’s historical gains as coinciding with expanding training data and asks how large web-collected datasets affect recognition performance.
- Training data expanded 100 times from 2010 to 2014, alongside major performance improvements in face recognition.Examples range from about 10 thousand training samples in Multiple LE to 4 million images in DeepFace.
- The central question is how big data, especially large amounts of web-collected data, impacts recognition performance.
3. MEGVII FACE RECOGNITION SYSTEM
The Megvii system is trained using a large celebrity-face database and a simple deep network, with data-distribution effects examined through controlled training subsets.
- The MFC database contains 5 million labeled faces covering about 20,000 individuals, with LFW identities removed manually.
- The system uses a ten-layer multi-class CNN trained with softmax supervision, extracts hidden-layer features, applies PCA reduction, and compares images with L2 distance.
- LFW accuracy rises linearly as training data increases when individuals are randomly sampled from the MFC database.
4. CRITICAL OBSERVATIONS
Experiments show that more training data improves LFW performance, but web-data long-tail effects limit gains and sophisticated methods add little benefit at larger scale.
- Continued performance improvement: Large amounts of training data considerably improve system performance, with randomly sampled data preserving the original distribution.
- Continued performance improvement: The LFW performance improves linearly as the amount of accumulated training data increases.
- Long tail effect: The best performance occurs with the 10,000 individuals having the most instances; adding individuals with few instances does not help and can harm performance.
- Traditional tricks fade as data increasing: Sophisticated methods provide little gain compared with the simple CNN architecture and PCA reduction when applied to the MFC database.These methods introduce extra hyperparameters that make training harder.
5. PERFORMANCE EVALUATION
The system combines four face-region representations with PCA reduction and evaluates recognition on both LFW and a real-world security benchmark. It reaches 99.50% on LFW, but performance remains substantially weaker in the application setting, where cross-factor cases challenge recognition.
- System design: Four face regions are used for representation extraction, and their concatenated features are reduced with PCA before pairwise comparison.The regions are centered at the eyebrow, eye center, nose tip, and mouth corner; testing uses an L2 norm.
- Results on the LFW benchmark: 99.50% accuracy is achieved on the LFW benchmark, exceeding reported human performance.The remaining failures are mostly hard cases involving pose variation, make-up, glass wearing, or occlusion.
- Results on the real-world application: At a false positive rate of 10^-5, the system reaches a 66% true positive rate on the Chinese ID benchmark, below the application requirement.The benchmark targets real-world security certification and differs from LFW through its domain-specific setting.
- Results on the LFW benchmark: The LFW failures are grouped into easy cases potentially addressable by better training and hard cases involving special cross factors.The paper states that many hard cases are difficult even for humans and difficult for computers without additional priors.
- Results on the real-world application: Human testing solved 90% of the system’s failed Chinese ID cases, indicating a remaining machine–human performance gap in this scenario.Age variation is typical in CHID and is absent from the web-collected MFC training database.
6. CHALLENGES LYING AHEAD
The paper identifies data bias, very low false-positive criteria, and cross-factor variation as persistent face-recognition challenges. High LFW accuracy does not ensure strong performance in real-world settings, where age variation remains difficult and humans solve most failed cases.
- Data bias: Data bias limits generalization because web-collected faces are dominated by celebrities and differ from daily-life images.The paper characterizes web data as young, smiling, made-up, and beautiful, making it a baseline rather than a complete representation.
- Very low false positive rate: Very low false-positive requirements expose a gap between benchmark accuracy and real-world security performance.The authors report that their high LFW accuracy still falls far short of human performance in such settings.
- Cross factors: Pose, occlusion, and age variation are common factors affecting recognition performance, but they lack sufficiently comprehensive handling methods.The paper specifically notes insufficient investigation of these cross factors.
- Cross factors: 90% of failed CHID cases were solved by humans, highlighting a substantial gap between machine recognition and human-level performance.The failed cases involve both same-person age changes and age differences between people.
7. FUTURE WORKS
The paper treats large-scale data as a starting point for moving face recognition beyond laboratory benchmarks, while emphasizing that real-world applications still require substantial work. Future directions focus on data-centric improvement, including domain-specific mining, synthesis, and transfer.
- 7. FUTURE WORKS: Large amounts of web-collected data produce state-of-the-art LFW performance, but real-world applications remain an open challenge.The authors frame the LFW result as a new starting point rather than a completed solution.
- 7. FUTURE WORKS: Data is an important factor in recognition systems and motivates future research from an industrial perspective.The paper presents this conclusion after summarizing its experiments.
- 7. FUTURE WORKS: Mining domain-specific data more efficiently and synthesizing additional data are proposed directions for improving recognition performance.The paper gives video with weakly labeled faces and 3D face reconstruction as examples of these directions.
- 7. FUTURE WORKS: The long-tail effect in multi-class classification and transfer from generic systems to domain-specific applications remain open questions.The authors identify effective use of long-tail web data and domain transfer as future issues.
- 7. FUTURE WORKS: The report aims to stimulate academic and industrial discussion and further improve face-recognition techniques.