Source-linked AI summary
Different Defects, Different Effects: A Controlled Audit of Face Recognition Training Data
Ana Dias, João Ribeiro Pinto, Hugo Proença, João C. Neves
TL;DR
Large face-recognition datasets can contain distinct defects, and evidence about which defects remain in curated data and how correcting them affects training has been limited. This paper audits MS1MV3 with complementary detection strategies and controlled interventions, finding that effects depend on defect type. Targeted duplicate removal gives the clearest benefit, while some other corrections also reduce steps to baseline performance.
Problem
Curated face-recognition datasets may retain distinct defects, but prior work does not directly compare how correcting them affects recognition and training behavior.
Source: §1 · pp. 1–2Method
The authors audit five MS1MV3 defect types with complementary detection strategies and evaluate separate and selected combined interventions under a controlled training protocol.
Source: §1 · p. 2Results
Takeaways & Limitations
The results indicate that removing more images or correcting more defect categories does not necessarily produce a better training set.
Source: §6 · p. 8
Abstract
from arXiv · showhide
The development of face recognition has been strongly supported by the availability of large-scale training datasets. Among them, MS-Celeb-1M has played an important role by providing millions of facial images from nearly 100K identities. However, the size and heterogeneity of such datasets increase the likelihood of annotation errors, redundant samples, and other forms of data inconsistency. This motivated the introduction of refined versions, such as MS1MV2 and MS1MV3, aimed at improving the quality of the training data. Despite these cleaning efforts, we show that several types of training-data defects remain in MS1MV3. By proposing complementary data-auditing strategies, we identify five defect categories: exact and near-duplicate images, split identities, crop errors, multiple-face ambiguity, and invalid content. We then assess the effect of correcting these defects under a controlled face-recognition training protocol based on IResNet-50 and ArcFace. The results show that targeted interventions can preserve, and in some cases improve, recognition performance while requiring fewer optimization steps to reach the performance of the original training set. Duplicate removal provides the clearest overall benefit, while split-identity merging and invalid-content removal also improve training efficiency. Overall, the results indicate that the effect of dataset cleaning depends strongly on the type of defect being addressed and provide further evidence that training-data quality remains an important and comparatively underexplored factor in large-scale face recognition. The training manifests and code used in this work will be released upon acceptance.
1. Introduction
Large web-collected face-recognition datasets can retain distinct defects that generic noise labels obscure. A controlled MS1MV3 audit finds that correction effects vary by defect, with duplicate removal delivering the clearest benefit.
1. Introduction
- The audit identifies five defects and uses complementary detection strategies, then tests separate and selected combined interventions under the same IResNet-50 and ArcFace protocol.Evaluations consider recognition accuracy, steps to baseline performance, and held-out verification on IJB-B and IJB-C.Source: §1 · p. 2
- 15.44% image removal raises average validation accuracy from 97.20±0.02 to 97.27±0.01 and reaches baseline performance with 23.7±5.0% fewer steps.The matched random-removal control reaches only 97.18±0.01; duplicate removal is compared with the baseline in Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
- Merging split identities largely preserves accuracy while reaching baseline performance with 21.8±5.0% fewer steps, and invalid-content removal shows a similar step reduction.Joint identity merging and duplicate removal preserves validation performance with 24.3±2.9% fewer steps, while extending the cleaning pipeline adds no improvement.Source: §1 · p. 2
- Filtering in-valid content, crop errors, and multiple-face ambiguity together does not show the same benefit as targeted interventions.The strongest validation variants remain competitive with the baseline on held-out IJB-B and IJB-C at earlier checkpoints.Source: §1 · p. 2
- The study’s contribution is a controlled, defect-specific comparison of training-data interventions rather than a single cleaned-dataset evaluation.It evaluates recognition performance, training efficiency, and held-out verification.Source: §1 · p. 2
2. Related Work
Prior face-recognition work addresses label noise, image quality, and data redundancy through cleaning, robust training, and sample reduction. This work distinguishes specific identity- and image-level defects and isolates perceptual duplicates as a training-data defect.
2.1. Training-Data Quality in Face Recognition
- Prior label-quality work detects noisy samples, merges similar identity classes, or iteratively cleans datasets, while other methods reduce noisy supervision during training.Examples include Global-Local GCN, Adaptive Label Noise Cleaning, WebFace260M [32], Co-Mining, and Sub-center ArcFace.Source: §2.1 · pp. 2–3
- Image-quality methods estimate face-image quality, whereas this study examines split identities and image-validity defects while retaining challenging but valid facial variations.Retained variations include pose, illumination, age, blur, and occlusion.Source: §2.1 · p. 3
2.2. Training-Data Redundancy in Face Recognition
- Earlier deduplication work examined effects on pretrained-model outputs, while this study measures duplicate removal during model training and its effect on recognition and efficiency.Schlett et al. [23] studied deduplication effects on face-recognition and quality-assessment results produced by pretrained models.Source: §2.2 · p. 3
- Unlike model-dependent sample-selection methods, this study removes perceptually duplicated images and compares targeted removal with matched random removal.The control matches total and per-identity image counts to separate duplicate removal from simple dataset-size reduction.Source: §2.2 · p. 3
3. Dataset Defect Detection
The audit covers identity-label inconsistencies, redundant images, and image-validity defects while retaining valid but challenging facial variation. It combines identity- and image-level detection criteria to identify and assess these defect types.
3. Dataset Defect Detection
- Five defect categories span image-level redundancy or validity and identity-level label inconsistency; valid but challenging pose, severe blur, and occlusion are retained.Figure 2 illustrates duplicate groups, split identities, crop errors, multiple-face ambiguity, and invalid content.Source: §3 · p. 3, Figure 2 · p. 4
3.1. Split-Identity Detection
- Split identities are sought through identity centroids and similar-centroid candidate search, then verified with multiple nonredundant cross-folder image matches.Identity centroids average the top 80% most centroid-consistent embeddings; cross-folder verification requires supporting matches in both directions.Source: §3.1 · pp. 3–4
- Verified split-identity groups use a complete-link requirement, and a random sample of detected pairs is manually inspected for precision.Every pair within a group must pass cross-folder verification; overlapping groups that fail this condition are resolved by prioritizing stronger similarity evidence.Source: §3.1 · p. 4
3.2. Duplicate Image Detection
- Duplicate candidates are detected within each identity using perceptual-hash Hamming distance, with exact and near duplicates handled together and candidate links grouped transitively.Images assigned to different identities are not compared during this duplicate-detection stage.Source: §3.2 · p. 4
3.3. Image-Validity Defect Detection
- Crop errors are flagged by padding overlap with the face or central facial region, while multiple-face ambiguity is flagged when the second-largest face exceeds 15% of the largest face’s area.Crop thresholds are 10% of the central facial region, 20% of the face bounding box, or 10% of the face bounding box with at least two landmarks covered.Source: §3.3 · p. 5
- Invalid content is flagged only when BLIP rejects the real-person photograph category and accepts at least one non-real category.A random sample of detections is manually inspected to assess precision.Source: §3.3 · p. 5
4. Experimental Setup
The experiments use controlled manifests to apply separate interventions to MS1MV3 defects while keeping the training protocol fixed. Recognition is evaluated on five validation benchmarks, with held-out verification on IJB-B and IJB-C.
4.1. Dataset and Controlled Interventions
- MS1MV3 supplies 5,179,510 images from 93,431 identities; separate manifests specify each intervention’s images and labels without changing source files.Table 1 summarizes intervention sizes and removal distributions; Table 2 summarizes duplicate and split-identity group structures.Source: §4.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6
- Duplicate variants retain one representative per detected group; split identities are either merged under a canonical label or reduced to the canonical folder.D7 also has a random-removal control matched for total and per-identity image counts; validity filtering removes invalid content, crop errors, and multiple-face samples, while crop and multiple-face defects are not tested separately.Source: §4.1 · p. 5
4.2. Implementation Details
- The audit uses perceptual hashing for duplicates, face embeddings and cross-folder verification for split identities, facial landmarks for crop errors, OFIQ for multiple-face ambiguity, and BLIP-VQA for invalid content.Split-identity candidates are retrieved with HNSW and screened using a centroid-similarity threshold and cross-folder support requirements.Source: §4.2 · pp. 5–6
4.3. Training and Evaluation Protocol
- All variants train an IResNet-50 with ArcFace under the same protocol, differing only in training manifest and, for identity merging, class labels.Two seeds are used; checkpoints are selected by mean accuracy across five validation benchmarks, and efficiency is the first step reaching the original run’s best validation performance.Source: §4.3 · p. 6
- Held-out IJB-B and IJB-C verification uses intervention checkpoints selected around baseline-matching performance, with TAR measured at FARs of 10−6, 10−5, 10−4, and 10−3.Table 3 defines the validation and step-reduction statistics, reported only when the baseline is reached in both seeds.Source: §4.3 · pp. 6–7, Table 3 · p. 7
5. Results
Targeted defect corrections produce different recognition and training-efficiency outcomes: duplicate removal yields the clearest validation benefit, while several interventions preserve performance with fewer steps. The strongest variants remain competitive on held-out IJB-B and IJB-C at earlier checkpoints.
5.1. Recognition performance
- 97.27 average validation accuracy and 23.7±5.0% fewer steps were achieved by D7; ID-M→D7 matched or surpassed the original mean at most held-out operating points.The held-out evaluation compares IJB-B and IJB-C results at earlier checkpoints selected around baseline-matching performance.Source: §5.1 · p. 7, §5.2 · p. 7, §5.3 · p. 8
- 97.23 average accuracy was reached by both split-identity merging and folder removal; merging retains all images while addressing label inconsistency.Invalid-content removal also reaches 97.23, compared with 97.19 for filtering invalid content, crop errors, and multiple-face ambiguity together.Source: §5.1 · p. 7
- 97.26 average accuracy was reached by ID-M→D7, while adding invalid-content removal reached 97.25, showing that combined interventions did not accumulate validation gains.Across interventions, targeted duplicate removal is strongest, while additional defect corrections mainly preserve rather than further improve validation performance.Source: §5.1 · p. 7
5.2. Training Efficiency
- 21.8 fewer steps were required by ID-M, compared with 11.3 for folder removal; invalid-content removal also reduced steps by 21.8.Image-validity filtering reached the target in only one seed, whereas invalid-content removal showed a clearer benefit than additionally filtering crop errors and multiple-face ambiguity.Source: §5.2 · pp. 7–8
- 24.3 fewer steps were required by ID-M→D7, the largest reduction among combined interventions; adding invalid-content removal yielded a smaller, more variable reduction of 19.0.The reported gains depend on the intervention rather than the amount of data removed or the number of defect types addressed.Source: §5.2 · p. 8
5.3. Held-Out IJB Evaluation
- At earlier checkpoints, the strongest validation variants remain close to or outperform the baseline across many IJB-B and IJB-C operating points; the full pipeline is less consistent at the lowest FARs.On IJB-C at FARs of 10−4 and 10−3, all evaluated interventions are equal to, above, or within 0.05 of the original baseline.Source: §5.3 · p. 8
6. Conclusion
A controlled audit finds that distinct training-data defects remain in MS1MV3 and affect training differently. Targeted correction can reduce training steps while preserving recognition performance, but broader cleaning is not necessarily better.
6. Conclusion
- The study provides one of the first controlled comparisons of how distinct defect types affect recognition performance and training efficiency.It introduces complementary strategies for auditing five defect categories in large-scale face-recognition data.Source: §6 · p. 8
- 15.44% duplicate removal improved average validation accuracy and reached baseline performance with 23.7% fewer training steps; matched random removal did not reproduce these gains.The random-removal control was matched in total and per-identity image counts.Source: §6 · p. 8
- Merging split identities with duplicate removal reached baseline with 24.3% fewer steps while maintaining comparable recognition performance.This combination produced the largest reduction in training steps.Source: §6 · p. 8
- Split-identity merging and invalid-content removal also reduced steps to baseline, whereas filtering all image-validity defects did not provide the same benefit.Source: §6 · p. 8
- The strongest variants remained competitive with baseline on held-out IJB-B and IJB-C at earlier checkpoints, while cleaning effects depended strongly on defect type.Removing more images or correcting more defect categories did not necessarily produce a better training set.Source: §6 · p. 8