PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Privacy Risks of Algorithmic Fairness

Hongyan Chang, Reza Shokri

arXiv:2011.03731v4stat.MLcs.CRcs.CYcs.LG

TL;DR

The paper asks whether enforcing group fairness carries a privacy cost, a question that matters because fairness and privacy both matter in sensitive machine-learning settings. Using membership-inference attacks to compare fair and unconstrained models, it finds that privacy risk rises disproportionately for unprivileged subgroups, particularly when the underlying data are more unfair.

  • Problem

    The paper asks whether enforcing group fairness changes privacy risk, since fair treatment across groups and privacy of sensitive training records are both needed in machine learning.

    Source: §1 · p. 1

  • Method

    The authors use membership-inference attacks to compare information leakage from fair and unconstrained models, with a separate attack distinguisher for each subgroup.

    Source: §1 · p. 1, §6 · p. 10

  • Results

    Fair models increase privacy risk disproportionately for unprivileged subgroups, with greater risk when the underlying data and unconstrained model are more unfair.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    In sensitive decision settings, fairness and privacy need joint consideration because training data may contain sensitive personal information.

    Source: §1 · p. 1, §5.3 · p. 9

    The analysis focuses on supervised classification and, for its fairness setting, binary classification.

    Source: §2.1 · p. 2, §2.2 · p. 2

Abstract

from arXiv · show

Algorithmic fairness and privacy are essential pillars of trustworthy machine learning. Fair machine learning aims at minimizing discrimination against protected groups by, for example, imposing a constraint on models to equalize their behavior across different groups. This can subsequently change the influence of training data points on the fair model, in a disproportionate way. We study how this can change the information leakage of the model about its training data. We analyze the privacy risks of group fairness (e.g., equalized odds) through the lens of membership inference attacks: inferring whether a data point is used for training a model. We show that fairness comes at the cost of privacy, and this cost is not distributed equally: the information leakage of fair models increases significantly on the unprivileged subgroups, which are the ones for whom we need fair learning. We show that the more biased the training data is, the higher the privacy cost of achieving fairness for the unprivileged subgroups will be. We provide comprehensive empirical analysis for general machine learning algorithms.

1. Introduction

The paper asks whether enforcing group fairness increases models’ privacy risk, measuring that risk through membership inference. It finds that the cost falls disproportionately on unprivileged subgroups, where fair models can memorize training data.

1. Introduction

  • The paper measures privacy risk with membership-inference attacks and proposes a separate member-versus-nonmember distinguisher for each subgroup.The attacker observes predictions in a black-box setting; the subgroup-specific strategy achieves higher attack accuracy than a single global distinguisher.Source: §1 · p. 1
  • Fairness increases privacy risk disproportionately for unprivileged subgroups, and the increase is greater when the underlying data and unconstrained model are more unfair.The paper also reports higher privacy risk on these subgroups as models become fairer.Source: §1 · pp. 1–2
  • When an unprivileged subgroup is small or complex, fair models may memorize its training data rather than learn a general pattern, increasing membership-inference risk.Fairness constraints barely change memorization and privacy risk for other subgroups, so the privacy cost is low for them.Source: §1 · p. 2
  • The empirical analysis uses synthetic data and real-world Law School, Bank Marketing, and COMPAS datasets.The experiments examine how, when, and why fair models leak information about their training data.Source: §1 · p. 2

2. Background

The background sets up classification, group-fairness, and membership-privacy concepts used in the paper. It focuses on equalized odds while also noting other fairness definitions.

2. Background

  • The background introduces the definitions of group fairness and data privacy that underpin the analysis.Source: §2 · p. 2

2.1. Machine Learning

  • The paper studies supervised classification, modeling each example with features, a protected attribute, and a label; standard models trained without fairness requirements are called unconstrained.The setup assumes examples are i.i.d. from a data distribution.Source: §2.1 · p. 2, §2.2 · p. 2

2.2. Fairness

  • Group fairness seeks to prevent discrimination across protected groups; the paper mainly uses equalized odds, with equal opportunity and false-positive parity also evaluated.Its relaxed δ-equalized-odds condition bounds differences in false-positive and false-negative rates across groups by δ.Source: §2.2 · p. 2

2.3. Membership Privacy

  • The paper treats privacy risk as an adversary’s ability to infer whether an individual’s record was used as input to a learning algorithm.Membership inference attacks serve as a measure of information leakage.Source: §2.3 · p. 2

3. Privacy Analysis

The privacy-analysis framework defines risk through membership-inference success and compares fair with unconstrained models. It estimates risk using subgroup-specific loss thresholds under black-box access.

3. Privacy Analysis

  • The framework quantifies risk using membership-inference attack accuracy and defines the privacy cost of fairness as the difference between fair and unconstrained models’ risks.The paper’s framework evaluates individual and subgroup privacy risk.Source: §3.1 · p. 3

3.1. Definition of Privacy Risk

  • Individual privacy risk is the probability that the strongest adversary wins the membership-inference game for a data point; subgroup risk aggregates risk for points sharing a protected attribute and label.The game randomly chooses whether the point is included in the training set, runs the learning algorithm, and tests whether the adversary can infer membership.Source: §3.1 · p. 3

3.2. Quantifying Privacy Risk

  • A subgroup-specific loss threshold classifies an input as a member when its model loss falls below that threshold.The evaluation selects the threshold that best separates members from non-members in each subgroup; this uses knowledge of the training set and estimates risk under the strongest attack.Source: §3.2 · p. 3

4. Empirical Analysis

The empirical analysis examines membership-inference privacy costs of fairness across synthetic and real-data experiments, comparing fair and unconstrained models and exploring subgroup disparities, relevant conditions, and—in some settings—the relationship between privacy costs and unprivileged-subgroup accuracy.

4.1. Experimental Results on Synthetic Data

  • Fairness constraints increase privacy risk disproportionately for the unprivileged subgroup G−0, including a vulnerable point whose risk rises from 0.63 to 0.93.The most vulnerable points are mainly from G−0, and members are more distinguishable from non-members on fair models in that subgroup.Source: Figure 2 · p. 4, §4.1 · p. 5
  • Using a separate loss threshold for each subgroup increases membership-inference attack accuracy and improves privacy-risk estimation.This stronger attack uses known group membership and labels to set subgroup-specific thresholds.Source: Table 2 · p. 4, §4.1 · p. 4
  • For G−0, fairness raises training accuracy from 50.1% to 78.8% while increasing memorization to 0.58, twice the unconstrained model’s level.The paper attributes the subgroup’s increased privacy risk to fair models memorizing its training points rather than learning a general pattern; other subgroups’ privacy risk and memorization change little.Source: Figure 4 · p. 5, §4.1 · p. 5
  • As the enforced fairness gap δ decreases, accuracy and privacy risk rise for G−0, alongside increased memorization.Smaller δ means the fair model is less discriminatory on its training data.Source: Figure 6 · p. 6, §4.1 · p. 6, Figure 7 · p. 6
  • A larger fairness gap in the unconstrained model is associated with a larger privacy cost for G−0 when fairness constraints are imposed.The paper links greater underlying discrimination—and the greater need for fairness intervention—to increased privacy disparity for unprivileged groups.Source: Figure 8 · p. 6, §4.1 · p. 6
  • A smaller unprivileged subgroup or group increases its privacy cost, while increasing training-set size decreases privacy risk when subgroup proportions stay fixed.The paper links smaller subgroup sizes to a larger underlying fairness gap and further memorization; training-set size has a similar effect on fair and unconstrained models.Source: Figure 9 · p. 7, §4.1 · p. 7
  • With post-processing, unprivileged subgroup G−1 accuracy falls from 85.6% to 48.4% to satisfy fairness constraints; the fair model fits the subgroup no better than unconstrained models, and its privacy risk is lower.Across different group-fairness metrics, similar patterns are observed.Source: §4.1 · p. 7

4.2. Experimental Results on Real Data

  • The real-data experiments use Law, Bank Marketing, and COMPAS datasets with decision-tree models trained using the reductions approach.The experiments average results over 20 runs; the supplied passages do not include the reported performance results.Source: §4.2 · p. 8
  • On Bank, fairness widened the subgroup privacy-risk difference from 0.131 for the unconstrained model to 0.186 for the fair model.This real-data result shows that the privacy-risk disparity can increase after imposing a fairness constraint.Source: Table 7 · p. 8, §4.2 · p. 8
  • COMPAS (gender) had a smaller privacy cost than COMPAS (race), which the authors attribute to its smaller unconstrained-model fairness gap of 0.095 and consequently less memorization.Source: §4.2 · p. 8
  • On Law (gender), the unconstrained model’s fairness gap was 0.031, yet subgroup G−1 had a privacy cost of 0.064; the authors suggest its small size impedes generalization and encourages memorization.Source: §4.2 · p. 8
  • On Law (race), increasing the enforced fairness level can improve unprivileged-subgroup accuracy, but at the cost of privacy.Table 8 reports prediction accuracy and privacy risk across enforced fairness-gap levels.Source: §4.2 · p. 8, Table 8 · p. 9
  • For decision trees with max depth 5, imposing fairness reduced test accuracy on G− from 33.4% to 13.3%; low-capacity models have negligible privacy cost because they fit poorly.Random-guessing accuracy is 50%, and privacy risk is low for both fair and unconstrained low-capacity models.Source: §4.2 · pp. 8–9, Table 9 · p. 9

5. Related Work

Related work spans fairness methods, membership inference as a measure of information leakage, and privacy-fairness interactions; this paper focuses on individual-record and subgroup privacy under fairness constraints.

5.1. Algorithmic Fairness

  • Fairness methods include metric equality, individual fairness, and causality-based measures, implemented through pre-processing, in-processing, or post-processing.Pre-processing changes data representations, in-processing incorporates fairness into training, and post-processing adjusts predictions.Source: §5.1 · p. 9

5.2. Membership Inference Attacks

  • Membership inference tests whether a record was in a model’s training set and is used to measure information leakage across several machine-learning settings.Prior work studies classification, robust learning, explanations, embeddings, and reinforcement learning [13, 14, 16, 17, 32].Source: §5.2 · p. 9

5.3. Privacy and Fairness

  • Prior work finds privacy risks vary across groups and records [38], while this paper focuses on how fairness constraints affect subgroup and individual privacy.Related work also examines dataset balance in unconstrained models [40].Source: §5.3 · p. 9

5.4. Fair Learning with Differential Privacy

  • Differentially private methods can bound membership-inference risk [43], but some fairness-and-privacy work protects sensitive attributes rather than membership [45].Cummings et al. show pure differential privacy and exact equality of opportunity cannot both be achieved.Source: §5.4 · pp. 9–10

6. Conclusions

The paper presents a framework for measuring privacy risks under group fairness algorithms and evaluates it with membership inference on synthetic and real data. It reports that fair models tend to memorize under-represented groups, increasing information leakage about unprivileged groups.

6. Conclusions

  • The paper presents a framework for analyzing privacy risks of group-fairness algorithms using membership inference on synthetic and real data.The evaluations examine how and why fair models leak information about training data.Source: §6 · p. 10
  • Fair algorithms tend to memorize data from under-represented subgroups while trying to equalize model error across protected groups.The paper attributes this tendency to fitting group-specific data rather than learning general patterns.Source: §6 · p. 10
  • This memorization increases models’ information leakage about unprivileged groups.Source: §6 · p. 10

Appendix A. Additional Results

The appendix includes a decision-tree accuracy table for models with maximum depth 10, covering all subgroups.

Appendix A. Additional Results

Loading 2011.03731v4…