Source-linked AI summary
Poisoning Attacks on Algorithmic Fairness
David Solans, Battista Biggio, Carlos Castillo
TL;DR
Poisoning attacks have been studied mainly for their effects on misclassification, while their effects on algorithmic fairness remain a concern. This paper formulates poisoning attacks against fairness and evaluates them in white-box and black-box settings. The results show that attacks can worsen disparate impact and disparate mistreatment, including through black-box transfer.
Problem
Poisoning attacks have typically targeted misclassification, while the effects of such attacks on algorithmic fairness are also a concern.
Source: §1 · pp. 1–2Method
The paper formulates a bilevel poisoning objective that optimizes a poisoning sample against validation loss, using disparate impact to target fairness while limiting accuracy effects.
Source: §2.1 · pp. 3–4Results
The attacks increase disparate impact and disparate mistreatment, including when a surrogate model is used for black-box transfer.
Source: §5 · p. 14Takeaways & Limitations
Studying adversarial attacks on algorithmic fairness can help make machine-learning systems more robust.
Source: §5 · p. 14The experiments use a specific poisoning technique; other models, datasets, and attack techniques remain future directions.
Source: §5 · p. 14
Abstract
from arXiv · showhide
Research in adversarial machine learning has shown how the performance of machine learning models can be seriously compromised by injecting even a small fraction of poisoning points into the training data. While the effects on model accuracy of such poisoning attacks have been widely studied, their potential effects on other model performance metrics remain to be evaluated. In this work, we introduce an optimization framework for poisoning attacks against algorithmic fairness, and develop a gradient-based poisoning attack aimed at introducing classification disparities among different groups in the data. We empirically show that our attack is effective not only in the white-box setting, in which the attacker has full access to the target model, but also in a more challenging black-box scenario in which the attacks are optimized against a substitute model and then transferred to the target model. We believe that our findings pave the way towards the definition of an entirely novel set of adversarial attacks targeting algorithmic fairness in different scenarios, and that investigating such vulnerabilities will help design more robust algorithms and countermeasures in the future.
1 Introduction
Algorithmic fairness concerns whether algorithms disadvantage groups, yet fairness methods often assume benevolent developers. This work proposes poisoning attacks that target fairness and reports that limited training-data perturbations can introduce inequities without large effects on overall accuracy.
1 Introduction
- Algorithmic fairness concerns designing algorithms and decision-support systems that do not impose unjustified disadvantage on groups, particularly already disadvantaged groups.In supervised classification, many methods have focused on parity criteria.Source: §1 · p. 1
- Traditional poisoning attacks have generally sought to increase misclassification, while training-data tampering poses a threat to fairness as well.The paper frames malicious data tampering as a challenge to the assumption that developers act to reduce discrimination.Source: §1 · p. 2
- The proposed poisoning attack aims to create or increase disadvantage for a specific group rather than simply reduce model accuracy.The motivation includes deliberately biased sampling and unintentional labeling errors, such as systematic mistakes on images of people with a certain skin color.Source: §1 · p. 2
- Carefully perturbing a limited number of training examples can introduce group inequities without large effects on overall accuracy, making the attacks harder to detect.The experiments examine black-box and white-box scenarios and use impact quantification metrics.Source: §1 · p. 2
2 Poisoning Fairness
The section formulates fairness poisoning as a bilevel optimization: a poisoning point is added to training data, and its effect is measured through loss on untainted validation samples. It develops a projected gradient-based attack and considers both full-knowledge and substitute-model transfer settings.
2.1 Attack Formulation
- The bilevel objective optimizes a poisoning point added to training data to maximize loss on untainted validation samples, accounting for the trained classifier’s dependence on that point.The formulation uses a box constraint on feature values observed in training data; optimizing one point at a time is greedy and can be suboptimal for multiple points.Source: §2.1 · pp. 3–4
- The attack targets disparate impact by maximizing a difference between mean group losses, using labels that encourage unprivileged samples to be negative and privileged samples to be positive.The objective is described as a smoother approximation of disparate impact; maximizing it amounts to minimizing that ratio.Source: §2.1 · p. 5, Figure 1 · p. 6
2.2 Gradient-Based Attack Algorithm
- Projected gradient ascent optimizes the poisoning point, with its gradient combining the explicit validation-loss derivative and the indirect effect through the trained parameters.The parameter derivative is obtained from the inner optimization’s KKT conditions; initialization with a randomly sampled point whose label is flipped is recommended to reduce the risk of a flat, zero-gradient start.Source: §2.2 · pp. 5–7
- In a bi-dimensional example, the attack slightly tilts a linear classifier’s boundary, increasing negative predictions for unprivileged samples and positive predictions for privileged samples.The section assumes full system knowledge for correct gradient computation, while black-box attacks optimize against a substitute trained on data from the target’s distribution and test transfer to the target.Source: §2.2 · p. 7, §2.3 · pp. 7–8, Figure 2 · p. 8
2.3 White-Box and Black-Box Poisoning Attacks
- The black-box setting evaluates transfer from a substitute model trained on data sampled from the target model’s distribution, without querying the target during attack optimization.The white-box setting assumes access to the target’s training data, feature representation, and learning and classification algorithms.Source: §2.3 · pp. 7–8
3 Experiments
Across synthetic and COMPAS experiments, the authors compare white-box and surrogate-trained black-box poisoning and report increased group disparities with modest accuracy effects. Transferability and attack stability vary with the target model and poison fraction.
3 Experiments
- Experiments assess white-box and black-box attacks on a 2,000-sample synthetic dataset and COMPAS (6,167 offenders); COMPAS transferability tests Logistic Regression-optimized attacks on five other model types.Source: §3 · p. 8, §3.2 · pp. 11–13
3.1 Experiments with synthetic data
- On synthetic datasets, poisoned models increase disparities across settings; the effect becomes more pronounced at larger class separations, with privileged-group FPR rising more than unprivileged-group FNR falls.The comparison covers demographic parity and average odds difference, while accuracy is affected moderately.Source: §3.1 · pp. 10–11, Figure 4 · p. 11
3.2 Experiments with real data
- Both attacks increase unfairness on COMPAS with a more modest accuracy effect, but black-box behavior becomes noisy and accuracy loss detectable above about 20% poisoned samples.White-box behavior remains stable as the number of poisoned samples grows; effects also increase with poison count, including higher FNR for unprivileged samples and higher FPR for privileged samples.Source: Figure 5 · p. 12, §3.2 · pp. 12–13
- Compared with the Error-Generic Poisoning Attack, the fairness-targeting attacks affect classifier fairness more, while the generic attack is designed to decrease accuracy.Source: §3.2 · p. 12
- Logistic Regression-optimized attacks transfer more strongly to Logistic Regression, Naive Bayes, and SVMs than to Decision Trees or Random Forests.The attacks still introduce demographic disparity and average odds difference on Decision Trees and Random Forests, but with more limited effects.Source: Figure 6 · p. 13, §3.2 · p. 13
4 Related Work
The section situates the work in gradient-based poisoning and fairness measurement, alongside methods that mitigate unfairness by changing training data or learning algorithms.
Adversarial Machine Learning Attacks
- Gradient-based optimization crafts poisoning points by solving a bilevel problem and iteratively optimizing one attack point at a time.The framework requires computing the gradient of the classifier's learned function.Source: §4 · p. 13
Measuring Algorithmic Fairness
- Fairness measures include individual consistency metrics and group parity metrics, including disparate impact, equalized odds, and disparate mistreatment.Group metrics can be computed from counts of privileged and unprivileged samples receiving positive or negative outcomes.Source: §4 · p. 14
Optimization-Based Approaches to Increase Fairness
- Other approaches mitigate unfairness by adding antidote samples or modifying the learning algorithm, with some methods balancing accuracy and fairness through covariance-based definitions.Examples include matrix-factorization recommender systems, gradient-based optimization, and classifiers that modify the learning algorithm.Source: §4 · p. 14
5 Conclusions and Future Work
The attacks can increase disparate impact and disparate mistreatment in system predictions, including when disparities already exist, and can transfer from a surrogate model to a target model. The authors identify broader models, datasets, and attack techniques as future directions for studying fairness vulnerabilities and robustness.
5 Conclusions and Future Work
- The attacks can increase disparate impact and disparate mistreatment in system predictions, including when disparities already exist in training data.They can introduce unfairness or increase existing unfairness.Source: §5 · p. 14
- A surrogate model can support a black-box transfer attack without access to the specific target model.The authors report that the attack can alter model fairness properties in this setting.Source: §5 · p. 14
- The experiments use a specific poisoning technique; future work could extend the study to neural networks, other datasets, and other techniques, including causality-based approaches.The authors also state that studying fairness attacks can help make machine-learning systems more robust.Source: §5 · p. 14