Source-linked AI summary
Level Playing Field for Million Scale Face Recognition
Aaron Nech, Ira Kemelmacher-Shlizerman
TL;DR
Million-scale face recognition shows large accuracy variation, raising questions about whether algorithms or training data drive performance. The paper creates MF2, a shared 672K-identity benchmark, and finds comparable results to prior challenges using private training data, while age-invariant recognition remains weaker.
Problem
Large-scale benchmarks show that face recognition remains unsolved globally and that accuracy varies dramatically across algorithms, leaving the roles of algorithm design and training data unclear.
Method
The paper constructs MF2, automatically labels a large Flickr-derived dataset, requires training on the shared data, and evaluates algorithms at million scale.
Results
MF2-trained algorithms achieve results comparable to MegaFace and MS-Celebs-1M, including a maximum accuracy of about 75%, while FG-Net age-invariant performance is slightly worse than private-data results.
Takeaways & Limitations
A common large public training set can level the playing field across algorithms, while age variation remains an important weakness for face recognition.
Takeaways & Limitations
The VGG baseline could not produce predictions for all 670K identities because its fully connected softmax output layer did not scale to the full dataset.
Abstract
from arXiv · showhide
Face recognition has the perception of a solved problem, however when tested at the million-scale exhibits dramatic variation in accuracies across the different algorithms. Are the algorithms very different? Is access to good/big training data their secret weapon? Where should face recognition improve? To address those questions, we created a benchmark, MF2, that requires all algorithms to be trained on same data, and tested at the million scale. MF2 is a public large-scale set with 672K identities and 4.7M photos created with the goal to level playing field for large scale face recognition. We contrast our results with findings from the other two large-scale benchmarks MegaFace Challenge and MS-Celebs-1M where groups were allowed to train on any private/public/big/small set. Some key discoveries: 1) algorithms, trained on MF2, were able to achieve state of the art and comparable results to algorithms trained on massive private sets, 2) some outperformed themselves once trained on MF2, 3) invariance to aging suffers from low accuracies as in MegaFace, identifying the need for larger age variations possibly within identities or adjustment of algorithms in future testings.
1. Introduction
MF2 addresses dramatic million-scale variation in face-recognition accuracy by fixing the training data and benchmarking algorithms at scale. The benchmark introduces a 672K-identity dataset and compares algorithm performance with prior large-scale challenges.
- Motivation: Million-scale benchmarks show that face recognition remains unsolved globally, with dramatic accuracy differences across algorithms.Smaller benchmarks have reached phenomenal verification and identification accuracies, but large-scale tests distinguish millions or billions of identities.
- Motivation: MF2 investigates whether algorithmic differences or access to large, high-quality training data explains variation in face-recognition accuracy.Unlike MegaFace and MS-Celebs-1M, MF2 requires groups to train only on a common publicly available dataset.
- Contributions: 672K identities are labeled automatically using compact face embeddings to cluster millions of photos.The approach relies on assumptions about the data structure, whereas ImageNet and Google Open Images include 10K classes.
- Contributions: MF2 provides baselines trained on the shared dataset and examines approaches for handling 672K identity classes during training.
- Contributions: MF2 benchmarks algorithms while contrasting results with MegaFace and MS-Celebs-1M challenges.
2. Related Work
Related work frames large-scale face recognition as a data, labeling, and training-scaling problem. Existing benchmarks permit arbitrary training data, while public datasets remain smaller and potentially biased toward celebrity imagery.
- Training data: Private face datasets can contain up to 8M identities and 200M+ photos, while the largest public dataset has 100K identities and 10M photos.
- Training data: Public celebrity-based datasets may bias algorithms toward professional-camera imagery and away from photographs of children or non-celebrities.
- Benchmarks: MegaFace and MS-Celebs-1M allow training on any data, report highest scores around 75%, and show dramatic accuracy differences across algorithms.
- Large-scale labeling: Manual labeling of million-scale data is costly, error-prone, and requires retesting and validation, motivating automated or semi-automated cleaning.
- Large-scale labeling: Prior work studies automated cleaning through near-duplicate removal, age and gender consistency, caption text, and constrained optimization.
- Large-scale training: Scaling the number of identity classes is distinct from scaling samples per class, and fully connected softmax networks scale poorly to hundreds of thousands of identities.
3. Data Collection for MF2
MF2 is constructed from a massive Creative Commons Flickr collection by detecting and extracting faces from roughly 100M photos across more than 550K accounts.
- Source collection: Roughly 100M Creative Commons Flickr photos from over 550K accounts provide the source for MF2.
- Face extraction: Faces are detected and extracted from the Flickr collection because not all source photographs contain faces.
4. Automatic Identity Labeling
The paper builds identities from unlabeled Flickr faces by clustering within accounts using relative embedding distances and explicit structural assumptions, then improves purity through robust outlier detection and inner-cluster purification.
- Clustering algorithm: An automatic labeling pipeline uses a pre-trained VGG-Face model with a triplet projection layer to produce 1024-dimensional embeddings for large-scale clustering.The embeddings provide Euclidean-distance-comparable features across the collected faces.
- Clustering considerations and assumptions: The clustering assumptions are imperfect: celebrities may recur across Flickr accounts and collage images may contain multiple copies of one identity.The authors nevertheless report that these assumptions hold well in practice.
- Clustering considerations and assumptions: The approach clusters faces within Flickr accounts because current methods generalize poorly across accounts when embedding distances vary.The construction assumes identities cannot cross accounts, cannot occur more than once per photo, and must contain more than Z photos.
- Clustering algorithm: A modified relative-distance clustering algorithm represents faces as a graph and retains connected components of at least Z = 3 faces as identities.Pairwise links use distances below βD, where D is the account-level average pairwise Euclidean distance, subject to a no-link constraint matrix.
- Parameter tuning and validation: 98% purity was achieved on both LFW and FGNET after applying two noise-reduction mechanisms.Parameter validation selected β = 5.5 and α = 1.5, yielding 98% purity while retaining 35% of faces in the artificial-cluster experiment.
- Clustering optimization: Median Absolute Deviation identifies robust outlier thresholds, replacing standard deviation for detecting clusters whose distances deviate from the median.MAD is computed as the median of absolute deviations from the median.
- Clustering optimization: Inner-cluster purification removes faces contributing most to pairwise-distance averages after impure clusters are detected.The method constructs an L2 distance matrix for each cluster, sums distances row-wise, and applies MAD to identify faces for removal.
- Parameter tuning and validation: Post-clustering optimization maintained over 98% purity as MegaFace distractors increased, whereas purity without optimization degraded from 90%.The comparison evaluates impure-cluster detection and inner-cluster purification under increasing noise ratios.
5. Final Dataset Statistics
The finalized MF2 dataset contains 4,753,320 faces from 672,057 identities, with substantial variation in capture conditions and measurable age diversity within identities. Its construction trades dataset quantity against noise removal, excluding non-faces, sparse identities, and impure clusters.
- Dataset scale: 4,753,320 faces and 672,057 identities comprise the clustered and optimized MF2 dataset.Each identity has at least 3 photos, averaging 7.07 and reaching a maximum of 2469.
- Dataset variation: MF2 identities span varied lighting, expressions, and camera conditions in the loosely cropped dataset.Figure 4 shows a randomly selected identity that was not flagged as impure and remained in the final dataset.
- Demographics: The dataset has a 16.1-year average age range within identities, with a 12-year median and 41.1% female versus 58.8% male subjects.The median gender variance within identities was 0.
- Purification trade-offs: The purification process balances noise against data quantity and removed faces below the three-photo threshold, impurities, and clusters remaining impure after purification.The audit attributes 69% of removed faces to the minimum-photo threshold, 4% to cluster impurities, and 27% to clusters still impure after purification.
6. MF2 Benchmark
MF2 standardizes large-scale face-recognition evaluation by requiring participants to train on the same public dataset and test with up to 1M distractions. This design targets algorithm comparisons rather than differences in training-data access.
- Benchmark protocol: Participants must train on MF2 and test under the MegaFace challenge’s up-to-1M-distraction probe scheme.The shared training and evaluation setup is intended to level the playing field between algorithms.
- Evaluation data: Participants extract Euclidean-comparable features from a trained algorithm for a 1M-image distraction set, FaceScrub, and FG-Net.FaceScrub and FG-Net serve as probe sets because their identities are known.
7. Baseline Training Algorithms
The baseline study uses four VGG-based models to examine feature learning and training strategies under MF2’s large identity count. Because fully connected softmax could not cover all 670K identities, the models use restricted or rotating classification and triplet tuning, evaluated with million-scale distractions.
- Baseline design: Four VGG networks provide the initial MF2 baseline models.The baseline representation follows VGG-Face’s feature extraction and triplet-mapping approach, using a 1024-dimensional triplet representation.
- Large-class training: A fully connected softmax output could not produce predictions for all 670K identities, so the baselines use alternative training configurations.The reported models include restricted-identity softmax, triplet tuning, rotating softmax, and triplet tuning applied to the rotating-softmax model.
- Triplet tuning: Model B triplet-tunes Model A across all 670K identities while retaining the underlying 4096-dimensional feature layer and producing 1024-dimensional triplet features.Model D applies the same triplet-tuning strategy using a frozen trained Model C.
- Evaluation: Rank-1 identification rates are measured as distractions increase to 1M, using FaceScrub and FG-Net as probe sets.The baselines use a disjoint 1M distraction set and report rates under varying distraction levels in Figure 5.
8. Competition Results
MF2 results show that fixed public training data can produce million-scale performance comparable to benchmarks permitting private data, while age-invariant recognition remains weaker. Performance declines as distractors increase, and some teams improve substantially when trained on MF2.
- Age invariance: MF2 algorithms performed slightly worse on FG-Net’s age-invariant probe set than when private training sets were allowed.The paper suggests insufficient within-identity age variation or the need to model age explicitly as possible explanations.
- Cross-benchmark comparison: 15% to 35% on FG-Net and 35% to 57% on FaceScrub were the 3DIVI group’s performance increases after using MF2 training data.The comparison covers groups participating in both MegaFace and MF2.
- Scale effects: Performance decreases as distractors increase from 10 to 1M, although all algorithms perform very well with 10 distractors.MF2 shows the same large-scale degradation pattern reported by MegaFace.
- Benchmark ceiling: About 75% was the maximum accuracy reported by all three recent large-scale benchmarks.The paper states that no state-of-the-art method exceeded this level.
9. Summary
The paper presents MF2 as a public, large-scale benchmark and training dataset intended to compare face-recognition algorithms under controlled data access. Its preliminary results are comparable to benchmarks allowing private data and expose remaining large-scale training and age-variation issues.
- Contribution: MF2 provides over 0.5M identities and a benchmark designed to level the playing field across algorithms.The paper also discusses large-scale labeling and training challenges.
- Findings: MF2 produced results comparable to MegaFace and MS-Celebs-1M, despite those benchmarks allowing private training datasets.The paper frames these as preliminary competition results.