Source-linked AI summary
DenseFace: Bias Mitigation in Face Recognition via Density-Aware Probabilistic Matching
Mansur Bultygov, Vadim Seliutin, Dmitry Nekhaev, Ivan Laptev
TL;DR
Face recognition models remain demographically biased, while existing mitigation methods can require retraining or reduce recognition accuracy. DenseFace models face embeddings with von Mises–Fisher distributions and adjusts probabilistic matching for density differences; experiments show bias reduction while preserving accuracy and avoiding retraining.
Problem
Face recognition models remain demographically biased, while existing mitigation approaches can require retraining or degrade recognition accuracy.
Source: §1 · p. 1Method
DenseFace models face embeddings as von Mises–Fisher distributions and adjusts probabilistic matching to account for embedding-density differences.
Source: §1 · p. 2, §3.1 · p. 4Results
Takeaways & Limitations
Bias mitigation can be applied to pre-trained face recognition models while leaving the underlying recognition model fixed.
Source: §1 · p. 2The reported overhead evaluation used an NVIDIA A100 GPU with batch size 8.
Source: §10 · p. 12
Abstract
from arXiv · showhide
Despite steady progress in face recognition, current face recognition models still suffer from significant demographic biases. While approaches for bias mitigation have been proposed, existing methods often impose constraints on the training procedure and result in the degradation of recognition accuracy. To address this issue, we here introduce a method that reduces racial bias in pre-trained face recognition models without compromising their accuracy. To this end, we model face embeddings of each person by von Mises-Fisher (MF) distribution. We next observe the dependency between demographic attributes and the density of MF distributions, and propose DenseFace, a probabilistic face matching procedure that accounts for differences in MF distributions. Our extensive experiments demonstrate DenseFace to consistently reduce racial bias in strong face recognition models varying in network architectures, training datasets and loss functions. Notably, DenseFace preserves recognition accuracy and requires no retraining of the underlying face recognition model. Our work also investigates previously adopted bias measures and makes suggestions.
1. Introduction
DenseFace uses density-aware probabilistic matching to mitigate racial bias in face recognition. Experiments report bias reduction across varied models while preserving accuracy and avoiding retraining.
1. Introduction
- Prior bias-mitigation methods using balanced datasets can lose accuracy, while other approaches often require retraining and may also reduce performance.The introduction frames post-training calibration as an alternative to those constraints.Source: §1 · p. 1
- DenseFace models embeddings with von Mises-Fisher distributions and adjusts matching scores to account for density differences associated with demographic attributes.The method uses a probabilistic matching procedure to account for differences in MF distributions.Source: §1 · p. 2
- The authors use fixed-threshold false match rates as the primary bias metric and assess verification accuracy on multi-racial and cross-racial pairs.They note that RFW classification accuracy may not suit common recognition scenarios and does not account for cross-racial matching.Source: §1 · p. 2
- DenseFace consistently reduces bias across strong models varying in architecture, training data, and loss function while preserving recognition accuracy without retraining.The underlying face recognition model remains fixed.Source: §1 · p. 2
- Local distortion estimates inter-class embedding densities, and the method identifies balanced anchor sets as important for those estimates.The introduction also describes a margin-based method for computing densities of nearly orthogonal embeddings.Source: §1 · p. 2
2. Related Work
Related work covers face-recognition models, competing bias measures, and data-, architecture-, loss-, and calibration-based mitigation approaches.
2.1. Face Recognition
- Modern face recognition uses large-scale identity datasets and convolutional or attention encoders, with softmax-based losses and extensions now dominant.Earlier systems used triplet losses; training losses incorporate margins to improve class separability and reduce within-class spread.Source: §2.1 · pp. 2–3
2.2. Measuring Bias in Face Recognition
- Bias measures include subgroup accuracy variation, ROC values, and error-rate combinations; the authors argue fixed-distance FMR better reflects racial-cohort distance dependencies than accuracy or ROC values.The section also reviews RFW, RB-WebFace, and synthetic-dataset evaluation.Source: §2.2 · p. 3
2.3. Bias Mitigation in Face Recognition
- Mitigation work spans balanced-data approaches, architectural changes, loss functions, and post-hoc calibration [22].DenseFace differs by estimating local inter-class density and incorporating it into a probabilistic matching score, without retraining.Source: §2.3 · pp. 3–4
3. Methodology
DenseFace represents face embeddings probabilistically, estimates demographic-sensitive local densities through angular-margin distortion, and uses those densities in probabilistic face matching. The method focuses on inter-class density because intra-class density distributions do not represent racial bias.
3.1. Probabilistic face embedding representation
- Face embeddings are modeled as von Mises–Fisher distributions on the unit hypersphere, with mean direction set to the embedding and density estimated from nearest anchor embeddings.The anchors are identity-averaged and demographically balanced.Source: §3.1 · p. 4
3.2. Local distortion of embedding space
- Intra-class embedding densities do not represent racial bias, so DenseFace estimates inter-class density using identity-level anchor embeddings.Near-identical images within one identity can yield extremely high density regardless of the person's race or gender.Source: §3.2 · p. 4, Figure 4 · p. 4
- An angular margin inside the cosine function addresses near-equal density estimates caused by near-orthogonal anchor embeddings.The resulting margin-based density is denoted κ(m); margin insertion is described as a local squeezing operation that increases anchor cosine similarities.Source: §3.2 · p. 5
- Non-Caucasian identities tend to have higher margin-based local densities, which correlate with greater recognition difficulty for state-of-the-art models.The density distributions also show demographic differences across racial groups and genders.Source: §3.2 · p. 5, Figure 6 · p. 5
3.3. Density-aware probabilistic matching
- DenseFace uses mutual likelihood between two probabilistic face embeddings as a density-aware matching score.The pipeline represents each embedding probabilistically, estimates local inter-class densities, then matches the distributions.Source: §3 · p. 4, §3.3 · p. 5
4. Experiments
Across evaluated models, DenseFace reduces racial bias while preserving or improving verification performance. The experiments also examine its evaluation protocol, ablations, deployment variant, and density patterns.
4.1. Datasets; 4.2. Implementation Details
- The experiments use RFW and RB-WebFace, several pretrained-model training datasets, and balanced identity-averaged anchors constructed offline.The balanced anchor set contains 54,000 identities with equal subgroup counts across four race groups and gender.Source: §4.1 · p. 5, §4.2 · p. 6
4.3. Performance Metrics
- The evaluation uses a single global threshold set to Caucasian FPR=10−3 and expands RFW testing to all possible image pairs.RFW has approximately 14 thousand positive and 50 million negative pairs per group, compared with 3,000 of each in vanilla RFW.Source: §4.3 · p. 6
4.4. Debias of SOTA models
- Models trained on large-scale skewed datasets show lower or comparable RFW accuracy-standard-deviation bias than models or methods trained on BUPT-Balancedface.This comparison concerns standard deviation across racial groups under the RFW protocol.Source: §4.4 · p. 6
- DenseFace significantly reduces racial bias for all non-Caucasian domains versus cosine similarity while preserving or improving verification performance.The comparison covers RFW and RB-WebFace bias tables and corresponding verification-performance tables.Source: §4.4 · p. 6, Table 3 · p. 7, Table 4 · p. 7
4.5. Ablation Study
- The ablation reports that identity-averaged anchors slightly reduce bias, balanced image anchors reduce it drastically, and the probabilistic score completes DenseFace.K=128 is reported as generally good; m=π/3 is optimal among considered models, while smaller margins inadequately address orthogonality and larger ones saturate f(x) at 1.Source: §4.5 · p. 7, Table 7 · p. 7
4.6. Learning-based DenseFace†
- DenseFace† improves bias mitigation over anchor-based DenseFace on both reported datasets, adding 0.2% memory and 1.75% end-to-end latency over cosine matching.The variant uses a regression network for faster matching inference.Source: §4.6 · pp. 7–8, Table 9 · p. 8
4.7. Multi-racial test
- On the multi-racial RFW test, DenseFace and DenseFace† consistently improve results over the cosine-similarity baseline.The test combines all RFW images for all-vs-all comparisons.Source: §4.7 · p. 8, Table 10 · p. 8
4.8. Quantitative Analysis
- For each RFW race, most nearest neighbors belong to the same race, enabling local-density estimation without ground-truth race labels at inference.The figure description notes cross-racial neighbors too; the prose attributes some to effects of attributes such as gender or age.Source: §4.8 · p. 8, Figure 7 · p. 8
5. Conclusion
DenseFace mitigates racial bias in pretrained face-recognition models without retraining, with experiments showing substantial bias reduction while preserving recognition accuracy.
5. Conclusion
- DenseFace mitigates racial bias in pretrained face-recognition models without retraining, while experiments show substantial bias reduction and preserved recognition accuracy.The experiments span architectures, training datasets, and loss functions.Source: §5 · p. 8
Supplementary Material
The supplement provides additional training details, comparisons, benchmark evaluations, limitations, and anchor-set and image-filtering details.
Supplementary Material
- The supplement covers BUPT-BalancedFace training, additional comparisons, conventional benchmarks, RFW evaluation, limitations, and anchor-set and image-filtering details.These topics are organized in Sections 6–12.Source: Supplementary Material · p. 11
6. Training on BUPT-BalancedFace
On BUPT-Balancedface-trained ResNet-50, DenseFace lowers fixed-threshold FPR for African and Indian domains despite the baseline model remaining highly biased.
6. Training on BUPT-BalancedFace
- 0.00615 to 0.00143 for African and 0.00401 to 0.00103 for Indian: DenseFace reduces fixed-threshold FPR approximately fourfold in each domain.The model is ResNet-50 trained on BUPT-Balancedface with CosFace and evaluated on RFW under the NIST protocol.Source: §6 · p. 11, Table 11 · p. 11
7. Comparison with other de-biasing methods
DenseFace reduces residual racial bias in a de-biased CIFP model, and reported comparisons with other methods are limited by unavailable materials for several approaches.
7. Comparison with other de-biasing methods
- DenseFace mitigates the remaining FPR differences across racial groups in a CIFP-trained ResNet-34 model on RFW.The passage reports that CIFP has less bias than CosFace-R34-Glint360K, but still shows a large FPR gap among racial groups.Source: §7 · p. 11, Table 12 · p. 11
- Reimplemented comparisons with EM-FAR and SN-M3 [22] in the AdaFace-R100-WebFace12M setup report significant improvements for DenseFace.The comparison results are reported in Table 13 for RB-WebFace.Source: §7 · p. 11, Table 13 · p. 11
8. Evaluation on Standard Face Recognition Benchmarks
On standard face-recognition benchmarks, DenseFace maintains the strongest AdaFace-R100-WebFace12M model’s original accuracy while improving the baseline on a racially representative test set.
8. Evaluation on Standard Face Recognition Benchmarks
- DenseFace maintains original accuracy on five standard benchmarks while significantly improving the baseline on the racially representative test set.The evaluated benchmarks are LFW, CFP-FP, AgeDB, CPLFW, and CALFW.Source: §8 · p. 11, Table 14 · p. 11
- The evaluation uses the strongest AdaFace-R100-WebFace12M model on LFW, CFP-FP, AgeDB, CPLFW, and CALFW.Source: §8 · p. 11
9. DenseFace on original RFW protocol
On the original RFW protocol, DenseFace slightly reduces bias without lowering average accuracy; its computational-cost analysis reports negligible overhead for the regression-based variant.
9. DenseFace on original RFW protocol
- On the original RFW protocol, DenseFace slightly reduces bias measured by accuracy standard deviation without reducing average accuracy.Source: §9 · p. 11, Table 15 · p. 12
10. Computational Cost Analysis and Limitations
- The DenseFace† regression network adds negligible memory and latency overhead compared with the baseline.The cost comparison covers latency and memory for cosine matching, anchor-set DenseFace, and regression-network DenseFace†.Source: §10 · p. 12, Table 16 · p. 12
- A lookup table for the modified Bessel function trades 40 MB of memory for reduced matching cost, without affecting bias or verification performance.Source: §10 · p. 12
11. Anchor set identity overlap
The paper addresses possible anchor-identity overlap and examines whether filtering high-density samples changes verification errors across racial groups.
11. Anchor set identity overlap
- Anchor-set identity overlap could inflate κ(m) density; duplicate identities are excluded from nearest-neighbor sets using a cosine-similarity threshold of 0.6.Source: §11 · p. 12
12. Local density as image filter
- Removing high-density test samples helps reduce FPR for African and Indian groups, while Caucasian and Asian false-positive errors are less sensitive to density.Figure 9 plots Error vs. Reject Curves for RFW.Source: §12 · p. 12, Figure 9 · p. 12
- The paper also examines high-density sample removal on a multi-racial test, comparing FPR at a fixed threshold and FNR at a fixed FPR.Figure 10 presents the multi-racial Error vs. Reject Curve comparison.Source: §12 · pp. 12–13, Figure 10 · p. 13