Source-linked AI summary
Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning
Mei Wang, Weihong Deng
TL;DR
Face-recognition fairness is limited by racial bias associated with both training data and algorithmic margins. The paper introduces ethnicity-aware datasets and RL-RBN, which adaptively selects race-specific margins through deep reinforcement learning. Experiments report that RL-RBN mitigates racial bias and produces more balanced performance across races.
Problem
Racial bias produces less balanced face-recognition performance across races, while large-margin methods can still yield inferior non-Caucasian generalization under a uniform margin.
Method
RL-RBN formulates race-specific margin selection as a Markov decision process and uses deep Q-learning to learn adaptive margin policies, alongside two ethnicity-aware training datasets.
Results
RL-RBN successfully mitigates racial bias and learns more balanced performance across races in the reported experiments.
Takeaways & Limitations
Combining balanced training data with the debiased algorithm obtains the fairest performance across races in the paper’s experiments.
Abstract
from arXiv · showhide
Racial equality is an important theme of international human rights law, but it has been largely obscured when the overall face recognition accuracy is pursued blindly. More facts indicate racial bias indeed degrades the fairness of recognition system and the error rates on non-Caucasians are usually much higher than Caucasians. To encourage fairness, we introduce the idea of adaptive margin to learn balanced performance for different races based on large margin losses. A reinforcement learning based race balance network (RL-RBN) is proposed. We formulate the process of finding the optimal margins for non-Caucasians as a Markov decision process and employ deep Q-learning to learn policies for an agent to select appropriate margin by approximating the Q-value function. Guided by the agent, the skewness of feature scatter between races can be reduced. Besides, we provide two ethnicity aware training datasets, called BUPT-Globalface and BUPT-Balancedface dataset, which can be utilized to study racial bias from both data and algorithm aspects. Extensive experiments on RFW database show that RL-RBN successfully mitigates racial bias and learns more balanced performance for different races.
1. Introduction
The paper frames racial bias in face recognition as a fairness problem arising from both training data and algorithmic design, then proposes datasets and RL-RBN to learn more balanced performance.
- Racial bias is important because face-recognition systems can produce substantially higher error rates for non-Caucasian people.The cited example reports an error rate up to 39% for non-Caucasian people.
- Training data can encode ethnic underrepresentation, motivating the construction of BUPT-Globalface and BUPT-Balancedface.The datasets are designed to represent ethnic groups according to world population ratios or with equal sample counts.
- Uniform-margin face-recognition algorithms may favor overrepresented groups, while non-Caucasians can remain less separable even with balanced training data.The paper attributes this gap to differing feature separability and group representation.
- RL-RBN formulates race-specific margin selection as a Markov decision process and uses deep Q-learning to learn adaptive margin policies.The agent maximizes rewards based on the skewness of intra- and inter-class distances between races.
- The paper contributes two ethnicity-aware datasets, a reinforcement-learning debiasing algorithm, and experiments reporting fairer performance across races.The stated evaluation uses ethnicity-aware training datasets and the RFW database.
2. Related work
Prior work established racial bias in face recognition and explored fairness-oriented strategies, but comparatively few studies addressed racial bias in deep face recognition.
- Early studies found racial bias in non-deep face-recognition algorithms, including poorer recognition of Black cohorts and performance drops across racial domains.The cited work includes NIST FRVT studies and cross-race training and matching experiments.
- RFW provided a test dataset for validating racial bias in commercial APIs and state-of-the-art face-recognition algorithms.Prior work also used deep unsupervised domain adaptation to alleviate the observed bias.
- Computer-vision fairness methods have included unbalanced-training strategies, attribute suppression, domain adaptation, and reweighting data points.The related work categorizes these approaches by how they address network or data bias.
- Reinforcement learning uses experience within a Markov decision process to select agent behaviors and has been applied to high-dimensional computer-vision problems.The cited example combines reinforcement learning with CNNs for Atari-game decisions.
3. Ethnicity aware training datasets
The paper constructs two ethnicity-aware training datasets to address racial underrepresentation, using different distribution goals and reporting their scale and composition.
- BUPT-Globalface follows the approximate ethnic distribution of the world’s population, whereas BUPT-Balancedface approximately equalizes representation across four racial categories.The categories are Caucasian, Indian, Asian, and African.
- BUPT-Globalface contains 2M images from 38K celebrities, while BUPT-Balancedface contains 1.3M images from 28K celebrities.BUPT-Balancedface has approximately 7K identities per race.
- The datasets are built from race-selected images using celebrity nationality information and Face++ API assistance.The source material describes selection from MS-Celeb-1M and additional downloading to address insufficient Indian and Asian coverage.
4. Our method
The method investigates race-dependent feature separability and introduces adaptive margins learned through deep reinforcement learning. An agent adjusts non-Caucasian margins using skewness-based rewards, while Caucasians retain a fixed margin.
- 4.1. Investigation and observation: Large-margin losses improve discrimination but can yield less balanced representations for non-Caucasians under a uniform margin.The paper attributes this gap to insufficient samples and greater difficulty for non-Caucasians.
- 4.1. Investigation and observation: The study measures intra-class and inter-class angles on set-1 and RFW for different races using ResNet-34 models trained with ArcFace and Softmax.Intra1 and Intra2 denote intra-class angles on set-1 and RFW; Inter1 and Inter2 denote inter-class angles on those datasets.
- 4.1. Investigation and observation: Non-Caucasians show weaker intra-class compactness and inter-class discrepancy than Caucasians, especially for inter-class angles.Their test performance remains inferior even when training uses a uniform margin across races.
- 4.1. Investigation and observation: RL-RBN replaces the fixed margin with a race- and training-step-dependent margin for non-Caucasian groups while keeping the Caucasian margin unchanged.The adaptive margin is intended to minimize angular skewness between races and learn more balanced performance.
- 4.2. Adaptive margin policy learning: The margin-selection process is formulated as an MDP in which deep Q-learning chooses actions from the current race, margin, and inter-class skewness state.Actions keep the margin unchanged, shift it upward, or shift it downward by a constant ϵ.
- 4.2. Adaptive margin policy learning: The agent’s reward reflects skewness in intra- and inter-class distances between each non-Caucasian group and Caucasians.The DQN estimates Q-values for possible actions and is trained by minimizing a temporal-difference loss.
5. Experiments
Experiments evaluate racial fairness on RFW using ethnicity-aware training data, varying racial distributions, image degradations, and comparisons with fixed- or standard-margin methods. Across these settings, RL-RBN adaptively selects race-specific margins and generally produces more balanced performance, including lower fairness disparities.
- Experimental settings: RFW evaluation uses Caucasian, Asian, Indian, and African verification subsets, with average accuracy, standard deviation, and skewed error ratio measuring performance and fairness.A separate validation set contains 500 identities per race without subject overlap with the training or RFW datasets.
- Cause of racial bias: Increasing blur or Gaussian noise widens the performance gap between Africans and Caucasians, even after balanced training.Both groups are sensitive to image degradations, with blur and noise applied at increasing levels.
- Experiments with varying racial distributions: When the racial distribution is 4:2:2:2, Norm-Softmax accuracy is 89.69% for Caucasians and 84.17% for Africans.Increasing the Caucasian training ratio from 2/5 to 7/10 raises its Norm-Softmax accuracy from 89.67% to 90.67%.
- Experiments with varying racial distributions: For the 7:1:1:1 distribution, RL-RBN(soft) decreases SER from 1.83 to 1.35 compared with Norm-Softmax.The reported results indicate more balanced performance as the racial distribution becomes more uneven.
- Results on BUPT-Balancedface: Balanced training reduces racial bias but does not remove poorer non-Caucasian performance; combining balanced data with RL-RBN obtains the fairest performance.Manual margins based only on sample counts remain inferior because the most difficult race may require a larger margin despite its representation.
6. Conclusion
The paper provides two ethnicity-aware training datasets and proposes RL-RBN, which uses deep reinforcement learning to adaptively find margins for non-Caucasian races. Comprehensive experiments support the effectiveness of RL-RBN.
- Two ethnicity-aware training datasets, BUPT-Globalface and BUPT-Balancedface, are provided for studying racial bias.These datasets support research on reducing racial bias from data and algorithm perspectives.
- RL-RBN introduces a Markov decision process to adaptively find optimal margins for non-Caucasian races using deep reinforcement learning.The method learns an adaptive margin policy to support more balanced features across races.
- Comprehensive experiments demonstrate the effectiveness of RL-RBN.