PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning

Briland Hitaj, Giuseppe Ateniese, Fernando Perez-Cruz

arXiv:1702.07464v3cs.CRcs.LGstat.ML

TL;DR

The paper examines whether collaborative deep learning can protect private training data while avoiding centralized access. It develops a GAN-based active inference attack that exploits the real-time training process, finding that users can extract and induce disclosure of sensitive information despite parameter sharing restrictions and record-level differential privacy. The authors conclude that collaborative learning can allow any participant to violate another participant’s privacy.

  • Problem

    Centralized deep learning exposes sensitive training data to the service operator, while the privacy protection offered by collaborative learning remains uncertain.

  • Method

    The paper proposes a GAN-based active inference attack that observes and influences collaborative training to reconstruct sensitive information from a victim’s device.

  • Results

    The attack can extract sensitive information without compromising the service operator and remains effective against convolutional neural networks and record-level differential privacy.

  • Takeaways & Limitations

    Collaborative learning may let any participating user violate another user’s privacy, making it less desirable than the centralized approach it was intended to replace.

  • Takeaways & Limitations

    The assessment of federated learning is based on announcements and research papers rather than access to an actual system implementation.

Abstract

from arXiv · show

Deep Learning has recently become hugely popular in machine learning, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level DP applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).

1 INTRODUCTION

Deep learning’s strong performance and centralized training create privacy risks when models process sensitive user data. Collaborative learning limits direct data sharing, but this paper shows that GAN-based active attacks can still extract private information and remain effective against parameter obfuscation.

  • Motivation: Deep learning has improved performance across structured-data tasks and applications, but its models can incorporate and expose sensitive training information.Applications include computer vision, speech recognition, natural language processing, games, and disease diagnosis.
  • Motivation: Centralized training gives the service operator direct access to users’ private datasets, motivating collaborative schemes where participants share only model parameters.Collaborative participants train locally and share a fraction of parameters rather than uploading their datasets.
  • Collaborative learning: Collaborative learning requires repeated parameter exchanges during training, even when only a small percentage is shared or parameters are truncated or obfuscated with differential privacy.The learning process involves users updating parameters at each epoch.
  • Attack basis: GANs generate samples resembling a training distribution without accessing original samples, interacting instead with the discriminative network.The generator learns when the discriminator cannot distinguish generated samples from training samples.
  • Contributions: The paper introduces a GAN-based attack that exploits real-time collaborative training to infer sensitive information and deceive victims into releasing more accurate data.The attack is framed as a novel active inference mechanism for collaborative deep neural networks.
  • Contributions: The attack applies to convolutional neural networks and remains effective when shared parameters are obfuscated with record-level differential privacy.The authors distinguish this from an attack against differential privacy itself, targeting its proposed use in collaborative learning.

2 REMARKS

The paper’s GAN attack extracts class-level or prototypical information from collaborative deep-learning participants during training, including when parameters use record-level differential privacy. The resulting samples may reveal sensitive properties without reproducing exact training records.

  • Attack mechanism: The attack uses a GAN against collaborative deep learning to generate samples that appear to come from a private training distribution.The GAN pits a generator against the collaborative model’s discriminator and succeeds when the discriminator cannot distinguish generated from training samples.
  • Attack mechanism: The attacker can observe model evolution and influence honest participants during training, unlike attacks applied only after learning ends.The paper describes deceiving victims into releasing more relevant information and notes that this ability is unique to the attack.
  • Privacy impact: Privacy impact depends on the data: generated medical records may reveal a cancer-related class even when individual records are fictitious.The paper contrasts generic medical records with records whose shared distribution indicates cancer patients.
  • Privacy impact: For speech data, generated audio may reveal language or speaker gender despite producing fictitious word-like sounds.The paper says babbling alone may not violate privacy, while inferred language or gender may constitute a violation.
  • Differential privacy: Record-level differential privacy protects specific training elements but does not prevent GAN samples that are indistinguishable from individual examples.The paper explicitly distinguishes reconstructing a similar X′ from recovering the exact X, and calls record-level DP inadequate against active adversaries.
  • Examples: The attack’s broader consequence is that collaborative learning can let any participant violate another participant’s privacy without involving the service provider.This contrasts with centralized learning, where the service provider is the identified privacy risk.
  • Examples: The paper presents a face reconstruction that is essentially indistinguishable from Alice’s image even though it was not in the training set.The reconstruction is presented as similar rather than identical to the original face.

3 IMPACT

The paper argues that its attack extends to federated learning because devices collaboratively update shared models while keeping data local. It also distinguishes the authors’ threat model from secure aggregation’s focus on the service provider as adversary.

  • Federated learning: Federated learning appears vulnerable to the same GAN attack as collaborative learning while local models continue learning.The paper’s assessment is based on federated-learning descriptions and states that the GAN can operate through downloaded shared models.
  • Federated learning: Secure aggregation protects individual updates from the server by computing weighted averages through secure multiparty computation.The server can decrypt the aggregate only after several users participate.
  • Threat model: The secure-aggregation security model considers Google as the adversary but not casual users attacking other participants.The paper identifies this omitted insider threat as potentially making federated learning more dangerous than centralized learning.
  • Broader impact: The paper frames its warning as relevant to Apple’s reported plans to apply differential privacy in crowdsourced learning.The authors state that the adversary can be an ordinary user targeting another user, such as a celebrity or politician.

4 RELATED WORK

Related work spans model-extraction and inversion attacks, distributed and privacy-preserving learning, and differential-privacy defenses. The paper positions its contribution within collaborative training that shares selected or obfuscated parameters rather than raw data.

  • Model privacy attacks: Prior attacks inferred sensitive attributes from trained models, including ethnicity or gender from voice-recognition systems.Ateniese et al. used a meta-classifier to extract information or patterns from training sets.
  • Model privacy attacks: Model inversion attacks reconstructed images associated with labels by exploiting confidence information revealed by machine-learning models.The cited facial-recognition work targets images about a label known to the adversary.
  • Model privacy attacks: Other prior work demonstrated model stealing from predictions and membership inference from black-box model access.These attacks address model replication or whether a particular record was in the training set.
  • Model privacy attacks: The literature also includes identity inference from blurred images and adversarial inputs that induce model misclassification.These examples broaden privacy and security concerns beyond collaborative training.
  • Collaborative learning: Distributed deep learning was proposed to protect training data by having participants share gradients through a parameter server.The related approach relies on collaborative model training rather than explicitly pooling all raw data.
  • Privacy-preserving learning: Privacy-preserving alternatives include local smartphone training, encrypted inference, trusted processors, and differential privacy.The cited systems address data transfer, encrypted computation, data-oblivious training, or parameter protection.
  • Differential privacy: Differential privacy aims to provide provable record privacy without significant query-accuracy loss, but its protection depends on granularity and available auxiliary information.The paper cites prior work showing limitations when strongly correlated real-world data or outside information is accessible.
  • Collaborative learning: Collaborative learning shares only selected local gradients, with parameters optionally thresholded, range-limited, or randomized under differential privacy.Participants update locally and asynchronously while uploading and downloading a fraction of recent gradients.

5 BACKGROUND

Deep learning models summarize training data in their parameters, enabling accurate classification but also allowing adversaries to recover class information. GANs address a limitation of model inversion by generating samples intended to resemble the training distribution.

  • Supervised learning: Supervised learning estimates a classifier from labeled samples by minimizing prediction loss with regularization.The learned function predicts labels for previously unseen inputs.
  • Deep learning: Deep learning extracts relevant input features internally and performs well on large, high-dimensional, correlated datasets.This has driven gains across image, speech, video, and language applications.
  • Privacy implications: Model parameters summarize the training database and can reveal prototypical examples or class characteristics to an adversary with model access.The paper describes this leakage as intrinsic when a model learns accurate classification.
  • Model inversion: Model inversion follows network gradients to recover representative examples for represented and previously unknown classes.This demonstrates that trained deep models can leak information about their training classes.
  • Model inversion: Model inversion may produce meaningless, out-of-distribution images that receive accurate labels despite resembling neither the actual training data nor the intended class.Flexible deep networks can classify broad regions of input space with high confidence.
  • Generative adversarial networks: GANs are introduced to generate more training-like images, with adversarial training ending when generated and original samples become indistinguishable in distribution.The generator maps random inputs to images while the discriminator distinguishes real from generated samples.
  • Generative adversarial networks: The GAN theorem states that the global minimum occurs when the generated-image distribution equals the original data distribution.This distributional matching motivates using GAN outputs to reveal more sensitive information than ordinary model inversion.

6 THREAT MODEL

The threat model assumes an active insider who joins the collaborative learning protocol as an apparently honest participant. The insider seeks information about a class it does not own while leaving the central parameter server uncompromised.

  • Adversary: The adversary is an active insider who pretends to be an honest participant while influencing learning to extract information from a victim.This influence distinguishes the threat from applying model inversion only after final training.
  • Adversary: The adversary targets meaningful information about a class it does not own and does not compromise the central parameter server.The parameter server continues collecting and distributing participant parameters.

7 PROPOSED ATTACK

The proposed attack embeds a GAN inside collaborative training: an insider generates samples resembling a victim’s private class, labels them as a fake class, and uses the victim’s training response to improve reconstruction. The attack extends to multiple users and can remain effective under parameter obfuscation, although stronger restrictions may reduce model utility.

  • Attack setup: The adversary and victim share one label, while the victim owns a second private label that the adversary seeks to infer.In the basic scenario, the victim declares [a,b] and the adversary declares [b,c].
  • Attack procedure: The insider trains a GAN against a replica of the victim’s updating model to generate samples resembling the victim’s unknown class.The generated samples are labeled as the adversary’s fake class before being added to its local training data.
  • Attack procedure: The fake samples force the victim to distinguish the private class from the adversary’s fake class, causing the victim to reveal more information during collaborative learning.The insider uses the victim’s responses to improve knowledge of the class it previously lacked.
  • Attack procedure: The procedure alternates victim and adversary training, parameter exchange, GAN updates, fake-sample labeling, and local-model updates until convergence.The attack’s extra work occurs while the collaborative protocol continues iterating.
  • Scope: The attack generalizes from two participants to multiple users and classes, and the adversary need not begin with true samples.Figure 4 depicts the two-class case and states these generalizations.
  • Robustness: The attack works while the adversary’s local model improves and can remain effective when differential privacy or other obfuscation techniques are used.The authors distinguish this from attacking differential privacy itself and acknowledge possible degradation in result quality.

8 EXPERIMENTAL SETUP

Experiments implement the attack in a distributed collaborative learning system using CNNs, GAN generators, and the MNIST and AT&T face datasets. The setup follows an algorithm that alternates parameter exchange with adversarial sample generation and local training.

  • Datasets: Experiments use MNIST handwritten digits and the AT&T face dataset, with 60,000 MNIST training records and 400 face images.MNIST images are 32 × 32 pixels; the AT&T set contains 40 people with 10 images each.
  • Architectures: The experiments use Torch7 and convolutional neural network architectures for both datasets.The MNIST model has two convolution layers, while the AT&T model has three convolution layers and three max-pooling layers.
  • Collaborative protocol: Algorithm 1 begins with participants agreeing on the architecture, labels, learning parameters, upload and download fractions, thresholds, gradient bounds, and training procedure.These settings define the collaborative protocol before adversarial training starts.
  • Attack implementation: During the attack, the generator targets the victim’s unknown class, produces samples, assigns them a fake label, merges them with adversary data, and updates the local model with SGD.The adversary then computes and uploads parameter changes for the next collaborative iteration.
  • Outcome: At the end of training, the adversary obtains prototypical examples of a class known only to the victim.The implementation records the resulting collaborative model after the adversarial iterations.
  • Hyperparameters: MNIST experiments use learning rate 1e-3 and batch size 64, whereas AT&T experiments use learning rate 0.02 with batch sizes 32 or 1.The AT&T multi-participant setting uses batch size 1.

9 EXPERIMENTS

Experiments show that GAN-based attacks recover targeted information in collaborative learning across MNIST and AT&T settings, including distributed participation and differentially private parameter sharing. The attack benefits from real-time model access and adversarial influence, while stronger privacy can prevent learning altogether rather than specifically stopping the GAN.

  • MNIST experiments: GAN attacks outperform model inversion against CNNs on MNIST, where model inversion fails to produce clear results.The comparison gives the adversary access to fully trained models and all data.
  • MNIST experiments: Real-time access lets the GAN train jointly with the discriminative model, unlike model inversion, which accesses the model only after training.This real-time interaction is what makes the attack applicable to collaborative deep learning.
  • Collaborative experiments: With 41 participants, the adversary reconstructs a victim’s face despite having no local data and despite differential privacy being enabled.Forty honest users each train on one face class, while the adversary trains only on generated images.
  • Influence on collaborative learning: Adversarial fake-label influence makes reconstructions clearer and improves retrieved information, with good results appearing once discriminator accuracy reaches 80%.The influence also makes learning faster; clearer images appear after 50 epochs per participant.
  • Differentially private learning: Record-level differential privacy does not protect against the GAN when the discriminator continues learning, although tighter privacy can stop the underlying collaborative model from learning.Under tighter privacy bounds, the GAN fails because the local and server models cannot learn and collaboration does not proceed.

10 CONCLUSIONS

The paper presents active GAN-based inference attacks against collaborative deep learning and concludes that insiders can violate other users’ privacy without involving the service provider. It reports no effective countermeasure and identifies cryptographic or alternative-granularity privacy approaches as possible directions with remaining concerns.

  • Conclusions: The paper proposes and implements a novel class of active inference attacks using GANs in collaborative deep learning.The authors describe the approach as more effective and general than existing information-extraction mechanisms.
  • Conclusions: Collaborative learning allows any participating user to violate other users’ privacy without involving the service provider.This is presented as making collaborative learning less desirable than the centralized approach it was intended to replace.
  • Conclusions: The authors were unable to devise effective countermeasures against the attack.They discuss secure multiparty computation, homomorphic encryption, and differential privacy at different granularities as possible approaches.

A SYSTEM ARCHITECTURE

The experiments use separate architectures for MNIST and AT&T data, pairing collaborative or discriminative models with GAN generators. The MNIST CNN architecture is also reused for the adversary’s local discriminator.

  • MNIST architecture: The MNIST experiments use one convolutional neural network architecture for both the collaboratively trained model and the adversary’s local discriminator.The architecture is identified as the CNN used in MNIST-related experiments.
  • MNIST architecture: MNIST experiments use a dedicated generator model architecture.The passage identifies it as the generator architecture used in the MNIST experiments.
  • AT&T architecture: AT&T experiments use an architecture for the collaborative model and discriminator.The figure describes both components as being used for the AT&T Dataset experiments.
  • AT&T architecture: AT&T experiments use a separate generator architecture.The architecture is identified as the generator used in AT&T Dataset experiments.
Loading 1702.07464v3…