Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Liwei Song; Reza Shokri; Prateek Mittal

Source-linked AI summary

Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Liwei Song, Reza Shokri, Prateek Mittal

arXiv:1905.10291v3May 24, 2019stat.MLcs.CRcs.LG

TL;DR

Prior work largely separates machine-learning security from privacy, leaving unclear whether adversarial defenses affect membership-inference risk. This paper evaluates six robust defenses, adds two attacks tailored to robust models, and finds greater membership-inference susceptibility than natural training. The authors also identify a scope limitation concerning whether the privacy–robustness conflict is fundamental.

Problem
Previous research typically considers machine-learning security and privacy separately, leaving the cross-domain impact of security defenses unclear.
Method
The paper measures membership-inference attacks against six adversarial defense methods and proposes two attacks exploiting structural properties of robust models.
Results
All six robust training methods make models more susceptible to membership-inference attacks than naturally undefended training.
Takeaways & Limitations
Adversarially robust training can increase the membership-inference privacy risk of machine-learning models.
Takeaways & Limitations
The paper leaves unresolved whether the privacy–robustness conflict is a fundamental principle, noting limitations of the perturbation constraints used to model adversaries.

Abstract

from arXiv · show

The arms race between attacks and defenses for machine learning models has come to a forefront in recent years, in both the security community and the privacy community. However, one big limitation of previous research is that the security domain and the privacy domain have typically been considered separately. It is thus unclear whether the defense methods in one domain will have any unexpected impact on the other domain. In this paper, we take a step towards resolving this limitation by combining the two domains. In particular, we measure the success of membership inference attacks against six state-of-the-art defense methods that mitigate the risk of adversarial examples (i.e., evasion attacks). Membership inference attacks determine whether or not an individual data record has been part of a model's training set. The accuracy of such attacks reflects the information leakage of training algorithms about individual members of the training set. Adversarial defense methods against adversarial examples influence the model's decision boundaries such that model predictions remain unchanged for a small area around each input. However, this objective is optimized on training data. Thus, individual data records in the training set have a significant influence on robust models. This makes the models more vulnerable to inference attacks. To perform the membership inference attacks, we leverage the existing inference methods that exploit model predictions. We also propose two new inference methods that exploit structural properties of robust models on adversarially perturbed data. Our experimental evaluation demonstrates that compared with the natural training (undefended) approach, adversarial defense methods can indeed increase the target model's risk against membership inference attacks.

1 INTRODUCTION

This introduction connects machine-learning security and privacy by asking whether defenses against adversarial examples alter membership-inference risk. The paper evaluates robust models and finds that adversarial defenses increase susceptibility to membership inference compared with natural training.

Security and privacy defenses are usually studied separately, leaving their cross-domain effects unclear.
The paper evaluates membership-inference attacks against adversarially robust models designed to mitigate adversarial examples.Membership inference determines whether a data point was used for training and reflects information leakage about training data.
Two new attacks exploit predictions on adversarial examples and verified worst-case predictions, achieving higher inference accuracy than confidence-based inference on benign inputs.
4.5×, 2×, and 3.5× are the membership-inference advantages of robust models relative to naturally undefended models on Yale Face, Fashion-MNIST, and CIFAR10, respectively.Membership-inference advantage is defined as the increase in inference accuracy over random guessing, multiplied by 2.
Experiments cover six state-of-the-art adversarial defense methods and examine robustness generalization, perturbation constraints, model capacity, prior knowledge, and countermeasures.The defenses include three empirical and three verifiable methods.
The paper extends earlier workshop analysis with two new inference attacks, four additional defenses, and a broader investigation of privacy-risk factors.

2 BACKGROUND AND RELATED WORK: ADVERSARIAL EXAMPLES AND MEMBERSHIP INFERENCE ATTACKS

This background introduces adversarial examples, robust defenses, and membership inference. It distinguishes empirical defenses that approximate robust loss from verifiable defenses that compute upper bounds under perturbation constraints.

Adversarial examples and defenses: Adversarial examples perturb benign test inputs to induce incorrect predictions, while poisoning attacks maliciously modify training data.
Adversarial examples and defenses: Adversarial robustness seeks unchanged predictions throughout a perturbation region, commonly modeled as an l∞-ball around each input.
Adversarial examples and defenses: Robust training combines natural loss with robust loss, where robust loss maximizes prediction loss over allowed perturbations.
Adversarial examples and defenses: Empirical defenses approximate robust loss by generating adversarial examples during training, whereas verifiable defenses use verified upper bounds on worst-case prediction loss.
Empirical defenses: PGD-based adversarial training generates perturbations through projected gradient steps and trains on adversarial examples, while distributional training uses a Lagrangian relaxation.
Empirical defenses: The empirical distributional defense lacks its stated l2 robustness guarantee here because the required smoothness condition is not satisfied by the setting’s loss function.
Membership inference attacks: Membership inference distinguishes training members from non-members using model predictions, including shadow models trained to simulate the target model.

3 MEMBERSHIP INFERENCE ATTACKS AGAINST ROBUST MODELS

This section explains why robust training can increase membership-inference risk and develops attacks that exploit benign, adversarial, and verified worst-case predictions.

Motivation: Robust training may increase membership-inference susceptibility by magnifying training-data influence while guaranteeing local prediction stability mainly around training examples.The paper links this increased sensitivity to easier distinction between members and non-members.
Membership inference performance: Membership inference determines whether an input belongs to the training set, and accuracy measures how often the adversary correctly distinguishes members from non-members.The evaluation samples equally from training and non-overlapping test sets.
Exploiting benign predictions: The baseline confidence-thresholding strategy infers membership when benign-example prediction confidence exceeds a selected threshold.The threshold is chosen to maximize inference accuracy by separating training and test confidence distributions.
Exploiting adversarial predictions: The first new strategy generates an untargeted adversarial example within Bϵ and thresholds prediction confidence on that perturbed input.PGD is used to generate xadv, with the threshold selected to maximize separation between adversarial train and test confidence distributions.
Targeted attacks: Targeted adversarial examples can expose distances to multiple label decision boundaries and may leak more membership information than untargeted examples.The targeted attack uses class-specific binary inference classifiers because each input has multiple targeted examples.
Verified worst-case predictions: A verification-based strategy uses worst-case prediction confidence over the perturbation region to infer membership for verifiably robust models.The attack assumes knowledge of the target model’s verification method and perturbation constraint.

4 EXPERIMENT SETUP

The experiments use image-classification datasets and convolutional or residual neural networks under l∞ perturbation constraints.

Perturbation setting: The study focuses throughout on the l∞ perturbation constraint Bϵ(x) = {x′ | ∥x′ −x∥∞≤ϵ}.This defines the allowed adversarial neighborhood around each input.
Yale Face: Yale Face contains 2,414 aligned grayscale images from 38 subjects, with 1,900 used for training and 514 for testing.The model is a CNN with four convolutional blocks and 3 × 3 kernels.
Yale Face: The Yale Face robust models use an l∞ perturbation budget of ϵ = 0.1.The architecture includes two fully connected layers after the convolutional blocks.
CIFAR10: The CIFAR10 classifier uses a wide ResNet with three residual groups and an l∞ perturbation budget of ϵ = 8/255.Each residual group contains five residual units, followed by a 10-neuron fully connected layer.

5 MEMBERSHIP INFERENCE ATTACKS AGAINST EMPIRICALLY ROBUST MODELS

The evaluation compares membership inference against natural and empirically robust models across datasets and inference strategies, then analyzes why robust training increases privacy leakage. Robust models are more susceptible to membership inference, with leakage linked to training-data sensitivity, robust-loss coverage, perturbation budget, and model capacity.

5.1 Overall Results: Three empirical defenses increase membership inference advantage by up to 3.2× on Yale Face, 2× on Fashion-MNIST, and 3.5× on CIFAR10 versus natural models.The evaluation covers PGD-based, distributional, and difference-based adversarial training.
5.1 Overall Results: Robust-model attacks using adversarial-example prediction confidence outperform benign-confidence attacks in most cases.Adversarial perturbations enlarge the confidence-distribution gap between robust training points and test points; the Dist-Based Adv-Train CIFAR10 classifier is an exception.
5.2.1 Sensitivity Analysis: Robust CIFAR10 models are more sensitive to training data than natural models when individual training points are removed and the models are retrained.Sensitivity is measured as the prediction-confidence difference between the original and retrained models; 60 points are evaluated across six retraining runs.
5.2.3 Privacy risk with model perturbation budget: Membership inference accuracy increases as more training points contribute to robust loss, alongside a larger gap between adversarial-training and adversarial-test accuracy.The experiment mixes robust-loss computation on one subset with natural-loss computation on the remaining training points.
5.2.3 Privacy risk with model perturbation budget: Larger adversarial perturbation budgets and higher model capacity are associated with greater membership inference accuracy in robust CIFAR10 classifiers.A larger ϵ makes models more robust and causes them to rely on a larger l∞ ball around each training point.
5.1 Overall Results: Targeted adversarial-example inference achieves the highest accuracy among five strategies because targeted examples encode distances to every incorrect class’s decision boundary.Untargeted adversarial examples reveal distance to only a nearby decision boundary.

6 MEMBERSHIP INFERENCE ATTACKS AGAINST VERIFIABLY ROBUST MODELS

The evaluation examines membership inference against three verifiable defenses across Yale Face, Fashion-MNIST, and CIFAR10 settings. Verifiable defenses increase privacy leakage when models fit training data, but limited capacity or loose verified losses can reduce attack accuracy.

Overall Results: Three verifiable defenses—Dual-Based Verify, Abs-Based Verify, and IBP-Based Verify—are evaluated under dataset-specific l∞ perturbation budgets.The budgets are 8/255 for Yale Face and 0.1 for Fashion-MNIST.
Overall Results: On Yale Face, all three verifiable defenses leak more membership information, with IBP-Based Verify exceeding 75% inference accuracy and reaching a 4.5× advantage over the natural model.Verified prediction confidence strategy IV performs best because verification enlarges confidence differences between training and test data.
Overall Results: On Fashion-MNIST, verifiable defenses do not increase membership inference accuracy because benign train accuracy falls below 90% and verified train accuracy below 80%.The models fit the training set poorly in this setting.
Overall Results: 52.10% is the highest reported inference advantage for verifiably robust Yale Face models, compared with 11.70% for the natural model.These values are reported for Table 8 under an l∞ constraint of 8/255.
Capacity and Training Fit: Increasing model capacity initially improves robustness and membership inference accuracy, but excessive capacity decreases both when the verified robust loss becomes too loose.The original architecture corresponds to capacity scale 8 in the Yale Face experiment.
Capacity and Training Fit: Reducing CIFAR10 training data to 20% enables verifiable models to fit better and increases membership inference accuracy from 71.50% to 78.50% as ϵ rises to 1/255.Beyond ϵ = 1/255, the reported inference accuracy begins to decrease.

7 DISCUSSIONS

The discussion tests attack knowledge, mitigation strategies, and the relationship between robustness and training-data privacy. It reports that robust training generally increases membership leakage, while calibration and improved robustness generalization can reduce it, leaving the fundamental conflict unresolved.

Attacker Knowledge: Approximate knowledge of a robust model’s l∞ perturbation budget suffices for high membership inference accuracy, and shadow training can tune attack parameters.Attack accuracy is generally highest when the attack budget is close to the model’s exact constraint.
Temperature Scaling: Temperature values approaching infinity make predictions nearly uniform and remove membership information, but also make the model useless for prediction.This establishes a direct privacy–utility limitation of extreme temperature scaling.
Temperature Scaling: Increasing softmax temperature decreases membership inference accuracy for robust Yale Face and Fashion-MNIST classifiers while maintaining the same classification accuracy.Temperature scaling reduces the prediction-confidence gap exploited by the attacks.
Regularization: DA-based regularization reduces the gap between adversarial train and test accuracy, thereby reducing membership inference risk in robust models.The approach aligns benign and adversarial logit distributions during training.
Privacy Versus Robustness: Robust training can magnify training-data influence by minimizing loss over an l_p ball around each training point, leading to greater memorization.The discussion links this mechanism to poor robustness generalization and elevated privacy leakage.

8 CONCLUSIONS

The paper connects machine-learning security and privacy by evaluating membership-inference risk in robust training. Across six state-of-the-art defenses, robust training increases susceptibility relative to natural undefended training, with leakage related to robustness generalization, perturbation constraints, and model capacity.

Six robust training methods made models more susceptible to membership inference attacks than naturally undefended training.
The study proposes two inference methods that exploit structural properties of adversarially robust defenses beyond benign-input prediction confidence.
Privacy leakage is related to the target model’s robustness generalization, adversarial perturbation constraint, and capacity.
The analysis examines the relationship between privacy and robustness and discusses adversary knowledge and potential countermeasures.

A FINE-GRAINED ANALYSIS OF PREDICTION LOSS OF THE ROBUST CIFAR10 CLASSIFIER

The analysis separates secure and insecure test inputs by prediction-loss distributions. Insecure inputs tend to have larger prediction loss and lower confidence, linking confidence-based membership inference risk to robustness generalization.

Secure examples are correctly classified for all adversarial perturbations within the constraint Bϵ.
Figure 7 compares prediction-loss histograms for secure and insecure test examples of the robust CIFAR10 classifier.
Insecure test inputs are very likely to have large prediction loss, corresponding to low confidence values.
Because membership-inference strategies use confidence, privacy risk is strongly related to robustness generalization even for benign inputs.

B MODEL ARCHITECTURE

The appendix presents the neural-network architectures used for Yale Face, Fashion-MNIST, and CIFAR10 experiments, including convolutional, residual, and fully connected layers.

Table 13 lists model architectures used on Yale Face, Fashion-MNIST, and CIFAR10.
The architecture notation specifies convolution channels, kernel dimensions, strides, residual units, and fully connected neuron counts.
All layers except the final fully connected layer use ReLU activations, and softmax produces the final prediction.

C EXPERIMENT MODIFICATIONS FOR THE DUALITY-BASED VERIFIABLE DEFENSE

The duality-based verifiable defense required architectural and input-dimension modifications for compatibility with its backpropagation-based dual analysis. The modified natural models retained similar accuracy and privacy performance in validation.

A 3 × 3 convolution with stride 2 was changed because its output-size computation prevented the defense’s dual backpropagation analysis.
The Yale Face input dimension was changed to 192 × 192 by adding zero padding for the same compatibility reason.
Validation found similar accuracy and privacy performance between modified and unmodified natural models.

D MEMBERSHIP INFERENCE ATTACKS WITH VARYING PERTURBATION CONSTRAINTS

This section evaluates membership inference attacks against robust Fashion-MNIST and CIFAR10 classifiers when perturbation constraints vary or are unknown to the adversary. The experiments use the IA strategy and report results in Figures 8 and 9.

The section also evaluates attacks when the adversary does not know the robust model’s l∞ perturbation constraint.
The evaluation considers attacks against robust Fashion-MNIST and CIFAR10 classifiers with varying perturbation budgets.The classifiers use robust-training constraints of 0.1 for Fashion-MNIST and 8/255 for CIFAR10.
The privacy leakage is evaluated using IA based on adversarial examples generated with varying perturbation budgets.
Fashion-MNIST classifiers are trained with an l∞ perturbation constraint of 0.1.
CIFAR10 classifiers are trained with an l∞ perturbation constraint of 8/255.

E PRIVACY RISKS OF OTHER ROBUST TRAINING ALGORITHMS

This section evaluates the privacy risks of a robust training algorithm that adds noise to the model architecture. The algorithm is based on a connection between differential privacy and model robustness.

The evaluation focuses on Lecuyer et al.’s robust training algorithm based on differential privacy and model robustness.
The method adds a noise layer to the model architecture for adversarial robustness.
The added noise layer uses either a Laplace or Gaussian distribution.