PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

Milad Nasr, Reza Shokri, Amir Houmansadr

arXiv:1812.00910v2stat.MLcs.CRcs.LG

TL;DR

The paper asks how much individual training data leaks from deep learning models, including in federated training where parameters or updates may be visible. It develops white-box membership inference attacks based on SGD-related gradients and evaluates passive and active attackers across centralized and federated settings. The attacks reveal substantial membership leakage even for well-generalized models, with active federated attackers achieving particularly high accuracy.

  • Problem

    The paper investigates the privacy risk of deep learning algorithms to individuals whose potentially sensitive data is used to train neural networks.

  • Method

    The paper designs white-box membership inference attacks using model parameters, updates, and SGD-derived gradient features across centralized and federated learning scenarios.

  • Results

    The attacks find substantial membership leakage in well-generalized models and federated settings, including 74.3% accuracy on CIFAR100 DenseNet and up to 87.3% for an active central attack.

  • Takeaways & Limitations

    Model prediction accuracy and generalization do not by themselves prevent white-box membership inference from exposing information about training records.

Abstract

from arXiv · show

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.

I. INTRODUCTION

The paper develops a comprehensive white-box privacy analysis of deep learning, showing that model parameters and training updates can reveal membership in training data even when predictions generalize well.

  • Privacy framing: The paper defines privacy-sensitive leakage as information about particular training samples that cannot be inferred from models trained on other data from the same distribution.This separates population-level utility from privacy loss about individual records.
  • Privacy framing: Membership inference is used as the privacy-analysis task because it tests whether a particular record was included in training and directly measures training-data leakage.This is distinguished from reconstruction attacks, which infer record attributes.
  • Attack framework: The framework analyzes white-box membership inference across fully trained models and model updates, covering centralized and federated learning, passive and active attackers, and varied prior knowledge.The attackers share the objective of inferring membership despite differences in knowledge, observations, and actions.
  • Attack framework: Analyzing activation functions as a direct extension of black-box attacks is ineffective because activation representations generalize quickly and provide little additional membership information beyond outputs.Later-layer activations contain more complex features, but their training-set information is described as largely similar to output leakage.
  • Attack mechanism: The proposed attacks exploit SGD by using target-record gradients, whose high-dimensional distributions can remain distinguishable between members and non-members.The attack uses gradients across model parameters and processes features from different layers to estimate membership probability.
  • Centralized evaluation: 74.3% white-box inference accuracy on CIFAR100 DenseNet exceeded 54.5% for black-box inference, despite the model achieving 82% test accuracy.The result indicates that well-generalized models can still leak substantial information about their training data.
  • Federated evaluation: 72.2% and 79.2% membership inference accuracy were achieved by a local participant and parameter server, respectively, in federated learning on CIFAR100 DenseNet.Repeated updates over epochs boost attack accuracy because they expose training-set-related changes over time.
  • Federated evaluation: 76.7%, 82.1%, and 87.3% accuracies were obtained by active attacks from an adversarial participant, central server, and isolated central attacker, respectively.The active attack uses gradient ascent on target points before uploading updates, magnifying SGD’s response to member records.

II. INFERENCE ATTACKS

The paper categorizes inference attacks by observation, adversary knowledge, training architecture, and attack mode, then develops white-box attacks that exploit model internals and SGD behavior.

  • Attack observations: White-box attackers access model parameters and can compute outputs of hidden layers and other functions of the model and input.This provides substantially richer attack inputs than black-box access, which exposes only predictions.
  • SGD-based leakage: The attack exploits SGD because gradients on training records are pushed toward zero after successive training rounds.The resulting high-dimensional gradient distributions may remain distinguishable between members and non-members even when predictions generalize well.
  • White-box attack architecture: Layer-specific attack components split inference across model layers and combine their outputs, reducing attack-model capacity and background-data requirements.The architecture is designed around differences in how much information each layer carries about individual records.
  • White-box attack architecture: The attack model separately processes layer gradients, activation vectors, model outputs, labels, and loss values through CNN and fully connected components.For federated learning and fine-tuning, corresponding features from multiple model versions are stacked before processing.

B. Inference Target: Stand-alone vs. Federated Learning

The paper considers inference against centralized and federated training, including final models, fine-tuned models, and model updates observed over time.

  • Training architectures: Stand-alone training places all training data in one location, whereas federated learning distributes data among multiple parties that jointly train a global model.In federated learning, participants locally update downloaded global parameters and a server averages their uploaded parameters.
  • Inference targets: Inference may target a final trained model or an updated model after fine-tuning on sensitive private labeled data.The fine-tuning attack is treated as a special case of the federated-learning membership-inference model.
  • Adversary positions: In federated learning, the parameter server can be curious or malicious, while a participant can observe global parameters and craft adversarial updates.These attacker positions expose different views of participant data and support passive or active inference strategies.
  • Temporal observations: Because successive model parameters can contain dependencies over time, the attack processes corresponding features from multiple observed models jointly rather than attacking each version independently.This single inference model is applied to sequences of gradients and other attack features.

C. Attack Mode: Passive vs. Active Inference Attack

The paper distinguishes passive attacks that only observe training from active attacks that manipulate federated updates, and describes supervised and unsupervised ways to train the inference model.

  • Attack mode: Passive attacks observe the learning process without modifying it, including attacks on models after stand-alone training is complete.Active attacks instead influence the target model during training to extract additional information.
  • Attack mode: Active attacks exploit SGD by increasing loss on a target record, after which local training reduces the gradient if that record is present.Repeated active attacks can produce high-confidence membership inference in federated learning.
  • Prior knowledge: Supervised inference training uses known members and non-members, or shadow models whose data share the target distribution without known overlap.The attack output in the shadow supervised setting is a membership probability.
  • Prior knowledge: Unsupervised training assumes access to a partially overlapping dataset without knowing which records overlap, and learns embeddings that separate members from non-members through clustering.The method uses an encoder-decoder architecture and predicts the cluster with the larger gradient norm as non-members.
  • Prior knowledge: The unsupervised decoder reconstructs loss, correctness, confidence, entropy, and gradient norms, then is discarded during membership inference.These features are used to maximize information in the membership embedding.

III. EXPERIMENTAL SETUP

The experiments evaluate the attacks on image-recognition and tabular datasets using a PyTorch implementation run on four Titan X GPUs.

  • Implementation: The implementation uses PyTorch and runs on a PC equipped with four Titan X GPUs, each with 12 GB of memory.These details describe the experimental computing environment.
  • Datasets: The evaluation uses CIFAR100, Purchase100, and Texas100 datasets.CIFAR100 contains 60,000 RGB images of size 32 × 32 organized into 100 object classes.

B. Target Models

The experiments use several target architectures, datasets, optimizers, and federated-learning procedures, with attack evaluation based on membership-prediction performance and prediction uncertainty.

  • Pretrained targets: The study also evaluates publicly available pretrained CIFAR100 models tuned for high testing accuracy with different regularization techniques.These models allow evaluation beyond models trained directly by the authors.
  • Federated targets: In federated learning, each party sends epoch-level parameter updates to a central server, which averages and redistributes them.The experiments use averaging aggregation and equal-sized party datasets selected uniformly.
  • Attack-model training: Attack models use ReLU activations, small normally initialized weights, batch size 64, Adam optimization, and up to 100 training epochs.The selected attack model is the one with the highest testing accuracy.
  • Evaluation metrics: Attack evaluation measures membership attack accuracy, true-positive and false-positive rates, and prediction uncertainty computed from normalized prediction entropy.Attack accuracy uses balanced member and non-member evaluation sets.

IV. EXPERIMENTS

Experiments examine white-box attacks on fully trained models, emphasizing gradients, layer combinations, model architecture, attacker-data size, output classes, and prediction uncertainty.

  • A. Stand-Alone Setting: Attacking Fully-Trained Models: The white-box attacker can use hidden-layer outputs and gradients, with later layers generally exposing more membership information than earlier activation layers.The last model layer leaks the most among the examined layer outputs.
  • Gradient features: Gradients leak significantly more membership information than layer outputs across the evaluated datasets and models.Combining gradients from all layers provides only a slight additional accuracy increase.
  • Architecture and leakage: Models with similar generalization errors can exhibit different white-box membership leakage, indicating that generalization error does not quantify this leakage adequately.DenseNet has more distinguishable member/non-member gradient norms and higher attack accuracy than ResNet.
  • Training data and gradient norms: Larger attacker training datasets improve membership-inference accuracy, while member gradient norms decrease during training and non-member norms do not show the same pattern.These observations link attack performance to gradient behavior and attacker background data.
  • Experimental comparisons: The experiments compare attack accuracy using individual activation layers, parameter gradients, and combinations of gradients with outputs in stand-alone CIFAR100-AlexNet models.They also vary the attacker training-set size while keeping the target training set at 50,000 examples.
  • Class-dependent leakage: Attack accuracy varies across output classes and is higher for classes with larger member/non-member gradient-norm differences or higher prediction uncertainty.The evaluated CIFAR100-AlexNet analyses compare classes with small, medium, and large differences.

B. Stand-Alone Setting: Unsupervised Attacks

The unsupervised stand-alone attack clusters encoded representations when member labels are unavailable and achieves noticeably higher accuracy than shadow-model training.

  • Attack setup: The unsupervised attacker uses data from the same underlying distribution without member or non-member labels, then classifies records into two clusters.This setting removes the need for labeled attack-model training data.
  • Results: The unsupervised attack achieves noticeably higher accuracy than the Shadow models approach across the evaluated datasets and architectures.The encoded values have different distributions for member and non-member samples.

C. Stand-Alone Setting: Attacking Fine-Tuned Models

The paper evaluates membership leakage from fine-tuned models and federated training observations, including passive attackers with varying access to model updates. Attack accuracy remains substantial in several settings, while aggregation and limited epoch access affect performance.

  • Fine-tuned models: Fine-tuned-model attacks distinguish members of the initial dataset, fine-tuning dataset, and non-members.The attacker has access to model snapshots before and after fine-tuning.
  • Fine-tuned models: Attackers distinguish initial-training members from fine-tuning members with reasonably high accuracy, while also separating members from non-members.The initial dataset D contains 60% of the training data, with the remainder used for D∆.
  • Passive federated attacks: Federated experiments evaluate passive global and local attackers across datasets, participant configurations, and training epochs.The global attacker aggregates parameters, whereas the local attacker observes only aggregate model parameters.
  • Passive federated attacks: Later training epochs substantially increase passive global attack accuracy on CIFAR100-Alexnet.Earlier epochs contain generic dataset features that leak less membership information.
  • Passive federated attacks: Access to several federated training epochs yields high CIFAR100 membership accuracy, but lowers accuracy on Texas100 and Purchase100 relative to stand-alone attacks.The reported datasets use different epoch sets for the federated attack.
  • Passive federated attacks: Local federated attacks achieve lower accuracy than global attacks because local attackers observe aggregate parameters.Averaging reduces the impact of each individual party.

E. Federated Learning Settings: Active Inference Attacks

Active federated attackers manipulate training with gradient ascent to create a stronger distinction between target members and non-members. This manipulation noticeably improves membership inference over passive global attacks, especially when the target participant is isolated.

  • Active attack: The active gradient-ascent attacker manipulates the learning process to improve membership inference accuracy.The attack evaluates membership for sampled member and non-member instances from the target model.
  • Active attack: Gradient ascent makes target-member gradient norms resemble those of non-target members, unlike target non-members.Figure 7 compares last-layer gradient norms during training.
  • Active attack: Training counteracts ascent on member instances by descending their gradients, while non-member gradients continue increasing.Members influence the training loss; non-members do not explicitly change it.
  • Active attack: Active gradient ascent creates a marked member–non-member gradient-norm distinction and noticeably improves accuracy over passive global attacks.The attacker forces the target model to behave differently on member and non-member instances.
  • Active attack: Isolating a target participant increases stored information about its training data and further improves gradient-ascent attack accuracy.Without aggregation with other parties, the target model retains more information about its own dataset.

V. RELATED WORK

Prior work established black-box membership inference and related attacks, while this paper extends privacy analysis to white-box neural-network settings. The paper reports attacks across stand-alone and federated scenarios, including passive and active adversaries, and identifies theoretical privacy bounds as future work.

  • Black-box membership inference: Black-box studies link membership leakage to training-data distribution and model generalizability, with overfitted models more susceptible than generalized models.These attacks observe only model predictions.
  • Collaborative learning: Earlier collaborative-learning attacks target small-batch updates and explicit word embeddings, unlike the broader neural-network settings studied here.The cited attack assumes participant updates after each mini-batch and models with explicit word embeddings.
  • This paper: The paper evaluates white-box membership attacks on standard learning mechanisms and target architectures, including pre-trained state-of-the-art models.The evaluations cover stand-alone and federated settings with passive and active attackers and different prior knowledge.
  • Defenses and related attacks: Differential privacy and adversarial training are established defenses, but the cited adversarial-training work focuses on black-box membership inference.Other related attacks include input, attribute, parameter, and side-channel inference.
  • Open problem: The paper leaves theoretical bounds on white-box deep-learning privacy leakage for future research.Its attacks are designed by exploiting vulnerabilities in stochastic gradient descent.

APPENDIX A ARCHITECTURE OF THE ATTACK MODEL

Appendix A reports the layer sizes used in the attack model architecture. The table serves as an architectural specification for the attack model.

  • Attack model architecture: Table XIV lists the layer sizes of the attack model architecture.The supplied passage identifies the table’s purpose but not the individual sizes.
  • Attack model architecture: The attack model architecture is documented separately from the attack-accuracy evaluations.The table is labeled as an appendix architecture description.
  • Attack model architecture: The appendix provides a compact reference for reproducing the attack model’s layer configuration.The supplied caption identifies layer sizes as the configuration being reported.
Loading 1812.00910v2…