Source-linked AI summary
Differential Privacy Has Disparate Impact on Model Accuracy
Eugene Bagdasaryan, Vitaly Shmatikov
TL;DR
The paper asks whether differential privacy’s accuracy cost is distributed equally across model subgroups. It studies DP-SGD across image, text, species, and federated language tasks, finding larger accuracy losses for underrepresented or complex groups and identifying clipping and noise as mechanisms behind the disparity.
Problem
The paper addresses limited evidence about how DP-SGD’s accuracy reduction affects different classes and subgroups.
Method
The paper evaluates DP-SGD across image classification, sentiment analysis, species classification, and federated language modeling, while analyzing clipping and noise effects.
Results
DP-SGD disproportionately reduces accuracy for underrepresented or complex groups, and worsens disparities already present in non-DP models.
Takeaways & Limitations
Gradient clipping and noise addition can exacerbate model unfairness, motivating further work combining fairness and privacy in practical deep-learning models.
Takeaways & Limitations
The federated-learning experiments used N = 80,000 participants and C = 100 per round rather than N = 800,000 and C = 5,000 because of computational constraints.
Abstract
from arXiv · showhide
Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.
1 Introduction
The paper shows that DP-SGD’s accuracy cost disproportionately affects underrepresented and complex subgroups. Gradient clipping and noise addition reduce these groups’ influence during training.
- DP-SGD clips gradients, adds random noise, and tracks privacy loss with a moments accountant.These mechanisms enable deep models with single-digit ϵ at a modest overall test-accuracy cost.
- DP-SGD disproportionately reduces accuracy for underrepresented and relatively complex subgroups.The authors report this effect across gender, age, sentiment, species, and language-modeling tasks.
- DP-SGD amplifies the model’s bias toward the most popular elements of the learned distribution.
- Underrepresented subgroup data produces larger gradients, so clipping reduces its learning rate and influence on the model.
- Random noise has the biggest impact on underrepresented inputs.
2 Related Work
Prior work studies DP, privacy leakage, fairness, and accuracy balancing separately or through post-processing, but has not established practical fair DP neural-network training.
- The paper focuses on DP-SGD because it supports practical deep-model training, federated learning, open-source implementation, iterative procedures, and tighter Rényi-DP bounds.
- Prior privacy-leakage studies examine underrepresented groups’ vulnerability but not DP’s impact on model accuracy.
- Existing subgroup-fairness methods can require at least 8,000 iterations, which would incur very high privacy loss if directly combined with DP.
- Oversampling, overweighting, and synthetic-data approaches cannot be directly combined with DP-SGD because its sensitivity bounds would not remain valid or the methods are incompatible with DP.
- Recent fairness-and-DP approaches have not yet yielded a practical procedure for training fair DP neural networks.
3 Background
The paper formalizes deep-learning training, differential privacy, DP-SGD, and federated learning, then evaluates disparate impact using accuracy parity across imbalanced classes and subgroups.
- 3.1 Deep learning: Deep-learning training minimizes a loss over inputs by updating model parameters with stochastic gradients.
- 3.1 Deep learning: In language modeling, recurrent networks process token vectors and compare outputs with labels or extended sequences.
- 3.2 Differential privacy: Differential privacy bounds output changes between adjacent datasets and limits computation through a privacy budget charged by ϵ.
- 3.2 Differential privacy: DP-SGD clips gradients above norm S and adds Gaussian noise σ, with z = σ/S linking the noise and clipping hyperparameters.
- 3.3 Federated learning: Federated learning aggregates locally trained participant models at a global server, while DP-FedAvg clips updates and adds Gaussian noise.
- 3.4 Measuring disparate impact: The evaluation uses accuracy parity, including long-tail accuracy and accuracy across subgroups defined by indirect attributes.
4 Experiments
Across facial, text, species, and federated language-model experiments, DP models generally lose more accuracy on smaller or underrepresented groups, with task-specific exceptions and computational constraints.
- 4.1 Gender and age classification on facial images: DP facial gender classification loses more accuracy on darker-skinned faces than on lighter-skinned faces relative to non-DP training.
- 4.1 Gender and age classification on facial images: DP age classification tends to be less accurate on smaller attribute-intersection subgroups, while already lower-accuracy classes suffer the largest drops.
- 4.2 Sentiment analysis of tweets: DP sentiment models learn SAE almost perfectly but lose substantially more accuracy on AAE than the non-DP model.
- 4.3 Species classification on nature images: DP species classification nearly matches non-DP accuracy on well-represented classes but performs significantly worse on smaller classes.
- 4.4 Federated learning of a language model: For federated language modeling, DP models have worse accuracy on participants with 500–1000-word vocabularies, similar accuracy on large vocabularies, and better accuracy on extremely small vocabularies.
- 4.4 Federated learning of a language model: The federated experiment used N = 80,000 and C = 100 instead of N = 800,000 and C = 5,000 because of computational constraints.
5 Effect of Hyperparameters
MNIST experiments show that clipping and noise jointly widen accuracy gaps against underrepresented classes, while hyperparameter changes trade accuracy, fairness, convergence, and privacy loss.
- With 500 examples, the non-DP model reaches 97% accuracy on “8” versus 99% on “2”, while the DP model reaches 77% versus 98%.
- Gradient clipping and noise addition: Clipping alone or noise alone yields 93% accuracy on the underrepresented class, whereas their combination prevents sufficiently effective updates.Without clipping, large gradients can compensate for noise; clipping without noise still updates model parts unaffected by other classes.
- Gradient clipping and noise addition: Combining clipping and noise with S = 1 and σ = 0.8 reduces well-represented-class accuracy from 99% to 98% but underrepresented-class accuracy from 93% to 77%.The underrepresented gradient reaches 170 and only drops to 110 after 60 epochs, leaving the model far from convergence.
- Noise multiplier z: Across equal-privacy settings, the DP accuracy gap is at least 20% versus under 3% for the non-DP model.Larger noise multipliers require smaller clipping bounds, while smaller noise multipliers permit larger bounds and higher accuracy.
- Batch size b: Increasing batch size decreases the accuracy gap but increases privacy loss, while overall accuracy still drops.Larger batches mitigate noise and are recommended for tuning model performance.
- Number of epochs T: Longer training can saturate DP accuracy without matching non-DP accuracy, and 50 underrepresented-class images prevent learning with ϵ < 10.The paper attributes the convergence problem to clipping slowing learning and noise blocking fine-grained minima.
6 Conclusion
The paper concludes that DP-SGD can disproportionately reduce accuracy for underrepresented and complex groups, worsening existing accuracy unfairness across image and language tasks.
- Gradient clipping and random noise disproportionately affect underrepresented and complex classes and subgroups.
- When non-private accuracy differs across subgroups, applying DP-SGD exacerbates that unfairness.
- The effect was demonstrated across several image-classification and natural-language tasks, motivating further research combining fairness and privacy.