Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy

1 INTRODUCTION

• Deep learning relies on large datasets that may contain contributors’ sensitive information, and models can memorize training data that attacks exploit.The paper notes that attacks can extract sensitive attributes from models in both black- and white-box settings.Source: §1 · p. 1
• Differential privacy adds noise to computations to protect against inferring an individual’s contribution, but can reduce accuracy and disproportionately harm minority subgroups.Prior work reports that stricter privacy can widen the utility gap between groups.Source: §1 · p. 1
• On CelebA, with a 30% male subgroup, ε=16.2 and ε=4.98 yielded 5.1% and 5.2% more female-vs-male accuracy disparity; at 0.1%, disparity rose to 19% and 52%, respectively.The paper also reports that both fairness metrics worsen as imbalance increases, across privacy levels.Source: §1 · p. 2
• The study spans minority-subgroup shares of 0.1%–30% and privacy budgets ε=1.15 and ε=16.2, beyond the ranges examined in earlier work.Bagdasaryan et al. studied minority shares of 0.01%–5.00% and ε values from 3 to 10.Source: §1 · p. 2

2 RELATED WORK

• Bagdasaryan et al. report lower accuracy for underrepresented groups under DP-SGD across face classification, sentiment analysis, species classification, and language modeling.The cited examples include lower accuracy classifying Black faces than white faces in two face datasets.Source: §2 · p. 2
• Kuppam et al. [19] propose fairness measures for randomized DP algorithms and identify causes of outcome disparities.Source: §2 · p. 2
• This paper varies data-imbalance ratios and measures equality of opportunity and demographic parity, extending the range of prior experiments.These are presented as common fairness metrics.Source: §2 · p. 2

3.1 Differentially Private SGD (DP-SGD)

• Differential privacy compares an algorithm’s output probabilities on datasets differing by at most one element, approximating the effect of individual opt-outs.Source: §3.1 · p. 2
• DP-SGD bounds gradient sensitivity through clipping and adds random noise to clipped gradients before updating model parameters.A moments accountant tracks privacy expenditure across training updates.Source: §3.1 · p. 2

3.2 Fairness & Bias In Machine Learning

• Fairness concerns disparate treatment across groups; bias may be historical, arise from population representation, or be generated by the algorithm.The background also distinguishes individual, group, and subgroup fairness.Source: §3.2 · p. 2

4 EXPERIMENTAL RESULTS

• The experiments use a pre-trained ResNet18 on 30,000 CelebA images, vary gender balance and DP noise, and include non-DP baselines.The primary smile attribute stays balanced; privacy settings use δ=10^-6, with ε=1.15, 4.98, and 16.2, and clipping is also varied on the 70%-female, 30%-male dataset.Source: §4 · p. 3

4.1 Impact of DP on Accuracy at Test Time

• Overall model accuracy decreases as stronger privacy guarantees are applied.The authors report this inverse relationship between model utility and information leakage and note that it agrees with Alvim et al..Source: §4.1 · p. 3
• Subgroup accuracy differences stay similar across privacy levels until the 99.9%–0.1% split, where the deltas diverge.Up to that extreme imbalance, ε=1.15 most closely matches the non-DP baseline; experiments with different clipping levels also show the strictest privacy guarantee has the smallest subgroup accuracy difference, 4.24%.Source: §4.1 · p. 3, Figure 1 · p. 3, Figure 2 · p. 3

4.2 Impact of DP on Accuracy During Training

• DP-SGD accuracy falls across overall, female, and male groups as privacy becomes stricter, and even ε=16.2 slows convergence and lowers final utility.Without privacy, training accuracy is highest across groups and time steps.Source: §4.2 · pp. 3–4, Figure 3 · p. 4
• During DP-SGD training, female–male accuracy gaps diverge, more under looser privacy; the non-DP gap stays relatively constant.The authors hypothesize that clipping disproportionately limits minority-group learning because its fewer examples produce gradients more likely to be clipped; they propose this effect compounds during training.Source: §4.2 · p. 4
• More imbalanced datasets show larger subgroup-accuracy divergence during training under (ε=4.98, δ=10^-6)-DP.The authors relate this pattern to smaller minority groups being more quickly affected by gradient clipping.Source: §4.2 · p. 4, Figure 4 · p. 4

4.3 Impact of DP on Fairness Metrics

• Greater imbalance worsens demographic parity and equality-of-opportunity differences across privacy levels.The authors report that the trend resembles subgroup-accuracy differences, especially for equality of opportunity; they assume the high-privacy model's better demographic parity but worse equality of opportunity in highly imbalanced cases reflects worsening precision.Source: §4.3 · p. 4

• Disparate accuracy impacts can occur with slight class imbalance and even loose privacy guarantees.Source: §5 · p. 4
• Tighter privacy does not always increase disparate impacts: severe utility degradation can make the model more random and therefore fairer.Source: §5 · p. 4
• Future work will examine imbalanced NLP tasks and mitigation through bias discovery and reduction using recourse or semi-supervised learning.Source: §5 · p. 4
• Figure 5 plots ΔDemP and ΔEO against dataset imbalance across varying DP levels.Source: Figure 5 · p. 5