PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, Somesh Jha

arXiv:1709.01604v5cs.CRcs.LGstat.ML

TL;DR

Privacy risks from machine-learning models trained on sensitive data are known, but their underlying causes remain unclear. The paper formally and empirically analyzes membership and attribute inference in relation to overfitting and influence. It finds overfitting can enable these attacks but is not required for membership leakage, while attribute risk also depends on influence.

  • Problem

    Models trained on sensitive data can leak private information, but the factors behind specific privacy risks in standard machine-learning algorithms are not well understood.

    Source: §1 · p. 1

  • Method

    The paper formalizes membership and attribute inference advantage and analyzes these attacks in terms of generalization error and influence, with empirical checks on real-world datasets.

    Source: §1 · p. 2

  • Results

    Overfitting is sufficient but not necessary for membership vulnerability, while attribute-inference risk depends on both overfitting and target-attribute influence.

    Source: §1 · p. 2, §8 · p. 22

  • Takeaways & Limitations

    Assessing privacy risk requires considering influence as well as overfitting, since even stable learning algorithms can leak precise membership information.

    Source: §8 · p. 22

    Membership-advantage bounds apply under specified conditions and target-specific background knowledge; characterizing stable adversaries remains future work.

    Source: §8 · p. 23

Abstract

from arXiv · show

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.

1 Introduction

The paper addresses the poorly understood sources of privacy risk in machine-learning models trained on sensitive data. It analyzes how overfitting and influence relate to membership and attribute inference, and examines connections between the attacks.

1 Introduction

  • The analysis formalizes membership and attribute inference advantage, relates both to generalization error and influence, and tests findings on real-world datasets.The authors control for overfitting through regularization and model structure.Source: §1 · p. 2

Membership inference

  • A straightforward attack on deep CNNs performs almost as well as a recent membership attack with less computation and adversarial background knowledge.The experiments also find membership advantage depends on generalization error, sometimes in direct proportion.Source: Membership inference · p. 2
  • Overfitting is sufficient but not necessary for membership vulnerability: a subverted stable learner can reveal exact membership with little predictive-accuracy penalty.The construction assumes conditions commonly satisfied in practice.Source: Membership inference · p. 2

Attribute inference

  • The paper defines attribute advantage relative to inference on general-population points, distinguishing training-data risk from prior definitions.It argues that an alternative definition based on a simulator without model access does not isolate risk to training-set individuals.Source: Attribute inference · p. 2
  • Attribute inference is sensitive to overfitting, but training-specific risk also depends on target-feature influence and is greatest when the factors are in balance.As influence increases, the attacker’s ability to learn more about training data than the general population vanishes.Source: Attribute inference · pp. 2–3

Connection between membership and attribute inference

  • The attacks are linked by reductions; at high generalization error, a reduction-based attribute attack can outperform Fredrikson et al.’s attack.The paper reports reductions in both directions, while suggesting attribute inference may be harder than membership inference.Source: Connection between membership and… · p. 3

2 Background

The background establishes the learning setting and notation used throughout the paper, then introduces stability and generalization concepts. It relates changes in training data to model behavior and defines overfitting through generalization error.

2 Background

  • The paper studies privacy risks in machine-learning algorithms, beginning with learning-theory notation and concepts.Unless stated otherwise, a model is produced by applying a training algorithm to a dataset sampled from a data distribution.Source: §2 · p. 3, §2.1 · p. 3

2.1 Notation and preliminaries

  • A data point has features x and response y; datasets contain n i.i.d. samples, and models map features to responses while minimizing expected loss.The text uses 0-1 loss for discrete responses and squared-error loss for continuous responses.Source: §2.1 · p. 3

2.2 Stability and generalization

  • Stability limits how much replacing one training point changes an algorithm’s output; the paper defines on-average-replace-one stability and relates stability to differential privacy.The cited connection equates an algorithm’s ability to achieve a given generalization error with its ARO-stability rate.Source: §2.2 · p. 4
  • Overfitting means expected loss on distribution samples substantially exceeds expected loss on the training set, while average generalization error averages across training sets.The paper notes this differs from another common definition based on a particular training set.Source: §2.2 · p. 4

3 Membership Inference Attacks

The section defines membership inference and analyzes how differential privacy and generalization behavior constrain or enable an attacker’s ability to identify training points. It also shows that a model can reveal membership even when its generalization behavior remains close to that of the original learning rule.

3 Membership Inference Attacks

  • The membership experiment compares a challenge point drawn from the training set with one drawn from D, and measures the adversary’s distinguishing advantage.The definition does not credit an adversary for a point drawn from D that also happens to occur in the training set, so maximum advantage is 1 − μ(n, D).Source: §3 · p. 5

3.1 Bounds from differential privacy

  • Differential privacy bounds membership advantage by e^ϵ − 1, while the section’s attacks relate advantage to generalization error, including cases where it is directly proportional.The formal assumptions are consistent with many natural learning techniques, and experiments are reported to confirm the generalization connection on real data even when those assumptions do not hold.Source: §3.1 · p. 6, §3.2 · p. 6

3.2 Membership attacks and generalization

  • For bounded loss B, membership advantage equals generalization error divided by B; for classification with 0-1 loss, it equals generalization error.Thus high generalization error necessarily results in privacy loss for classification models.Source: §3.2 · p. 7
  • Membership advantage is 0 when σS = σD and approaches 1 as σD/σS grows without bound under the known-Gaussian-error attack.The ratio σD/σS measures generalization error for squared-error regression because error magnitude depends on response scale.Source: §3.2 · pp. 7–8

3.3 Unknown standard error

  • When the adversary does not know both standard errors, it can assume σS ≈ σD and set the residual threshold to σS, or estimate the errors by retraining the known algorithm.The threshold-based advantage is graphed as a function of σD/σS.Source: §3.3 · p. 8

3.4 Other sources of membership advantage

  • A bounded-loss learning rule on a finite domain can be modified into a vulnerable rule that is nearly as stable, yet permits membership advantage nearly as large as possible on the data distribution.The construction assumes each feature value uniquely determines its label; its stability rate increases by at most knB2^-d + μ(n, D).Source: §3.4 · p. 9

4 Attribute Inference Attacks

The section defines attribute inference as recovering a sensitive feature from partial information and measures how much better an adversary performs on training points than on population points. Its model-inversion analysis relates this advantage to generalization error and feature influence, while noting that alternative advantage definitions can conflate training-specific leakage with general trends.

4 Attribute Inference Attacks

  • Attribute advantage compares target-feature inference on training points with inference on points from the population, using only the adversary’s partial information about each point.The comparison is intended to measure information about the target that the trained model leaks specifically about the training set.Source: §4 · p. 11
  • The alternative simulator-based definition can reward an adversary for deliberately performing poorly when it believes the point is outside the training set.Because its advantage can also rise with general population accuracy, the paper uses Definition 5 to separate training-data overfitting from trends in the distribution and leaves fuller analysis of Definition 6 for future work.Source: §4 · pp. 11–12

4.1 Inversion, generalization, and influence

  • For model inversion, Adversary 4 tests each possible sensitive-attribute value, updates its prior using an approximated error distribution, and selects the most likely value.The analysis assumes the attacker can reasonably approximate the error distribution; for linear models, influence is the absolute normalized coefficient of the target attribute.Source: §4.1 · pp. 12–13
  • With a uniform binary target, the advantage depends on the training and population error scales and on τ, the output difference associated with changing the target attribute.When σS = σD the advantage is zero; as overfitting increases, advantage increases monotonically for fixed τ.Source: §4.1 · p. 13
  • At both τ extremes, Adversary 4’s advantage approaches zero; it gains advantage when τ and the ratio σD/σS are in balance.At τ = 0 the target has no effect on model output, while sufficiently large τ overwhelms error noise regardless of training-set membership.Source: §4.1 · p. 13, Figure 1 · p. 14
  • With a nonuniform prior over target values, the attacker weights candidate error likelihoods by their prior frequencies; the maximum advantage is reached as σS → 0 and σD → ∞.In that limit, it always guesses correctly for training points and is reduced to random guessing for population points.Source: §4.1 · p. 13

5 Connection between membership and attribute inference

The section relates membership and attribute inference by constructing adversaries that use one attack type as an oracle to perform the other. Attribute advantage transfers exactly to membership advantage in one direction, while the reverse reductions give lower bounds and require reconstruction assumptions or additional queries.

5.1 From membership to attribute

  • Using an attribute adversary as an oracle yields a membership adversary whose membership advantage exactly equals the oracle’s attribute advantage.Thus, effective inference of attributes for training-set individuals implies the ability to infer their membership.Source: §5.1 · p. 14

5.2 From attribute to membership

  • The reverse reduction reconstructs a point from its partial information and varies the sensitive attribute before querying a membership adversary.The reconstruction function need only preserve the partial information; because it is lossy, the reconstructed point need not equal the original.Source: §5.2 · p. 15
  • The uniform-choice reduction can return a correct attribute only when its random guess matches the true target value.Its choice is motivated by the possibility that membership advantage is concentrated on some target values rather than others.Source: §5.2 · p. 15
  • The uniform-choice reduction gives an attribute advantage proportional to a modified membership advantage, within a constant factor, without using associations between the known information and target attribute.The modified advantage accounts for the reconstructed challenge-point distribution; for model inversion, it equals the unmodified membership advantage.Source: §5.2 · p. 15
  • The multi-query reduction tests every possible target value in order of prior probability, then selects the most probable value among those accepted by the membership adversary.This construction is intended to improve on the uniform-choice reduction’s limitation and is evaluated experimentally in Section 6.3.Source: §5.2 · p. 16

6 Evaluation

The evaluation tests membership and attribute inference on linear, tree, and CNN models, and examines a colluding attack that can expose membership even when predictive accuracy remains largely intact.

6 Evaluation

  • The evaluation compares theoretical predictions with empirical adversary performance on real data using linear regression, tree, and deep convolutional neural network models.It tests whether overfitting predicts privacy risk in practice.Source: §6 · p. 16

6.1.1 Linear and tree models

  • The linear and tree experiments trained Ridge regression or decision trees on random 75–25% splits and averaged results over 100 repetitions.They used Eyedata, IWPC warfarin-dose data, and Netflix ratings data.Source: §6.1.1 · pp. 16–17

6.1.2 Deep convolutional neural networks

  • CNN experiments used subsets of MNIST, CIFAR-10, and CIFAR-100, with equal-sized training and test sets to match prior work.The VGG-based architecture varied layer width with size parameter s to control generalization error.Source: §6.1.2 · p. 17

6.2 Membership inference

  • For linear and tree models, theoretical and experimental membership results agreed when the adversary knew σS and σD; unknown σD sometimes favored a boundary at |ϵ| = σS.The training-error distributions had higher peaks at zero than a Gaussian, which could make boundaries closer to zero advantageous.Source: §6.2 · pp. 17, 19
  • On CNNs, threshold membership results were less closely aligned with theory than for linear and tree models, but did not diverge as much as the non-Gaussian-error assumption might suggest.Compared with Shokri et al., the one-query attack had the same recall and slightly lower precision, while requiring less computation and only the average training loss.Source: §6.2 · p. 19, Table 1 · p. 19

6.3 Attribute inference and reduction

  • On IWPC, attribute-inference advantage increased with overfitting; on Netflix, none of the attacks effectively inferred whether a user rated the target movie.The study could not control both σD/σS and τ on Netflix to test the predicted influence effect.Source: §6.3 · p. 19
  • For IWPC, multi-query reduction outperformed both uniform reduction and direct attribute inference, while on Netflix it was often slightly worse than uniform reduction but better than direct inference.Both reduction adversaries use the threshold membership adversary as their oracle.Source: §6.3 · p. 20, Figure 3 · p. 20

6.4 Collusion in membership inference

  • Accuracy fell by only 0.014 on MNIST, 0.047 on CIFAR-10, and 0.031 on CIFAR-100, while colluding parties achieved high membership advantage.The advantage rose rapidly around s ≈16, and the experiments found no discernible relationship between generalization error and membership advantage.Source: §6.4 · p. 20

7 Related Work

The related work spans privacy attacks on statistical summaries and machine-learning models. This paper distinguishes its contribution through explicit training-data advantage measures and analyses connecting overfitting and influence to attack risk.

7.1 Privacy and statistical summaries

  • Prior statistical-summary attacks include GWAS membership inference, recommendation-change attacks, and linear or nonlinear reconstruction.The paper focuses on machine-learning settings where generalization error and influence are relevant.Source: §7.1 · p. 21

7.2 Privacy and machine learning

  • Earlier machine-learning privacy work showed leakage from model structure and linked privacy notions to generalization, while this paper constructs an attacker that leverages overfitting directly.The cited work includes differential privacy and related stability notions,, and.Source: §7.2 · p. 21

7.2.1 Membership inference

  • Shokri et al. used shadow models and a learned attack model; this paper’s threshold adversary came surprisingly close with lower complexity and fewer adversarial assumptions.Prior explanations of their attack hypothesized overfitting and training-data diversity, while this paper makes the overfitting connection precise and shows model structure can also be exploited.Source: §7.2.1 · pp. 21–22

7.2.2 Attribute inference

  • The paper defines attribute advantage as the difference between an attacker’s accuracy on training and test data, measuring exposure specific to the training set.Prior inversion-attack advantage compared model-assisted accuracy with the best accuracy without the model; prior work also connected influence to model inversion.Source: §7.2.2 · p. 22

8 Conclusion and Future Directions

The conclusion connects membership and attribute inference through reductions and highlights overfitting, stability, and influence as factors in privacy risk. It also identifies unresolved boundaries for colluding attacks and membership bounds.

8 Conclusion and Future Directions

  • Both theoretical and experimental results indicate that overfitting increases vulnerability to membership and attribute inference, while stable algorithms can still leak precise membership information.The target attribute’s influence on model output also plays a key role in attribute inference.Source: §8 · p. 22
  • The colluding membership attack uses a paired adversary and learning rule; further work should test whether malicious algorithms can produce models indistinguishable from normal ones and how to mitigate them.The scenario may be suited to the framework of algorithm substitution attacks.Source: §8 · pp. 22–23
  • The membership bounds address adversaries targeting specific individuals with arbitrary background knowledge, not necessarily adversaries seeking limited facts about many training-set individuals.Characterizing such stable adversaries could yield tighter bounds or relax conditions on the learning rule.Source: §8 · p. 23
Loading 1709.01604v5…