PaperlayerView exact revision on arXiv

Source-linked AI summary

When the Curious Abandon Honesty: Federated Learning Is Not Private

Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, Nicolas Papernot

arXiv:2112.02918v2cs.LGcs.CRcs.DC

TL;DR

Federated learning can leak user training data through shared gradients, challenging its presentation as privacy-preserving. This paper introduces trap weights that let an active central party amplify that leakage through inconspicuous weight manipulation. The attack achieves perfect reconstruction for a substantial fraction of data, including more than 50% of ImageNet points from mini-batches of up to 100 examples.

  • Problem

    Existing FL reconstruction attacks are expensive, low-fidelity, limited to small mini-batches, or require detectable model modifications, while shared updates already leak user information.

  • Method

    The paper adversarially initializes shared model weights with trap weights that amplify gradient leakage and extracts individual inputs through direct projection rather than complex optimization.

  • Results

    More than 50% of ImageNet training points are perfectly reconstructed from mini-batches of up to 100 examples, and the attack also perfectly extracts more than 65% of IMDB points at batch size 100.

  • Takeaways & Limitations

    An active dishonest central party can recover substantial portions of private training data efficiently, even from same-class mini-batches and across multiple local-training rounds.

  • Takeaways & Limitations

    The attack focuses on victim models containing a ReLU-based fully-connected layer, with extensions demonstrated for convolutional layers and token embeddings.

Abstract

from arXiv · show

In federated learning (FL), data does not leave personal devices when they are jointly training a machine learning model. Instead, these devices share gradients, parameters, or other model updates, with a central party (e.g., a company) coordinating the training. Because data never "leaves" personal devices, FL is often presented as privacy-preserving. Yet, recently it was shown that this protection is but a thin facade, as even a passive, honest-but-curious attacker observing gradients can reconstruct data of individual users contributing to the protocol. In this work, we show a novel data reconstruction attack which allows an active and dishonest central party to efficiently extract user data from the received gradients. While prior work on data reconstruction in FL relies on solving computationally expensive optimization problems or on making easily detectable modifications to the shared model's architecture or parameters, in our attack the central party makes inconspicuous changes to the shared model's weights before sending them out to the users. We call the modified weights of our attack trap weights. Our active attacker is able to recover user data perfectly, i.e., with zero error, even when this data stems from the same class. Recovery comes with near-zero costs: the attack requires no complex optimization objectives. Instead, our attacker exploits inherent data leakage from model gradients and simply amplifies this effect by maliciously altering the weights of the shared model through the trap weights. These specificities enable our attack to scale to fully-connected and convolutional deep neural networks trained with large mini-batches of data. For example, for the high-dimensional vision dataset ImageNet, we perfectly reconstruct more than 50% of the training data points from mini-batches as large as 100 data points.

1. Introduction

Federated learning keeps data on users’ devices but can still expose training information through shared model updates. This paper introduces trap weights, enabling an active central party to amplify leakage and reconstruct data efficiently, including from large same-class mini-batches.

  • Motivation: FL keeps user data local while a central party aggregates model updates, but those updates can leak membership, properties, and training data.Prior work shows naive FL offers little to no guarantees against leakage to other users or the central party.
  • Limitations of prior attacks: Existing reconstruction attacks are computationally expensive, low-fidelity, limited to small mini-batches, or require detectable architecture or parameter changes.The paper targets these limitations with inconspicuous shared-weight manipulations.
  • Trap weights: Trap weights adversarially initialize the shared model to amplify leakage while preserving unmodified model architectures and making the attack more stealthy.The method mainly rescales components of the model’s weight matrix before dispatching weights to users.
  • Attack efficiency: The attack extracts individual training inputs by projecting appropriate gradient portions onto the input domain, without complex optimization objectives.The extraction requires only a single-step computation over received model updates.
  • Results: The attack remains effective when gradients contain data from only one class and when users perform multiple local-training rounds or send model updates.These conditions are highlighted as settings where optimization-based attacks often struggle or where the protocol uses Fed-Avg.
  • Results: More than 50% of ImageNet training points are perfectly reconstructed from mini-batches containing up to 100 data points, while IMDB exceeds 65% under the same batch size.The reported reconstruction error is zero for the extracted points.

2. Background: Neural Networks, Federated Learning, and Differential Privacy

Neural networks learn by propagating mini-batch inputs through parameterized layers, computing loss gradients, and updating weights. Federated learning distributes this process across users, who compute local gradients that a central party aggregates into the shared model.

  • Neural Networks: A neural network classifier maps inputs to labels through layers combining linear operations with nonlinear activations such as ReLU.The paper considers fully-connected and convolutional layers.
  • Attack placement: The attack intervenes before user computation by manipulating shared weights, then extracts individual data from received gradients before aggregation.Figure 2 places the two attack points around model dispatch and gradient averaging.
  • Neural Networks: Mini-batch SGD samples data, computes predictions and loss, obtains weight gradients, and updates the model weights.Weights are initialized before this repeated training sequence.
  • Federated Learning: FL lets users keep data locally while a central party coordinates training by aggregating gradients from decentralized users.At each iteration, a selected subset of users receives the current shared model and contributes locally computed updates.
  • Federated Learning: Each selected user computes a gradient on a local mini-batch, uploads it, and contributes to the central party’s averaged update.The protocol represents decentralized mini-batch SGD.

3. Existing Data Reconstruction Attacks

Existing federated-learning reconstruction attacks use passive gradient observation or active model manipulation, but optimization-based methods can be costly and unreliable, while some active attacks are detectable or require repeated updates. The paper positions its approach against these limitations.

  • Passive attacks: Passive attacks observe received gradients and recover information through GAN training or second-order optimization.These attacks can target data reconstruction without maliciously manipulating the FL protocol.
  • Optimization-based attacks: Optimization-based attacks jointly or separately reconstruct inputs and labels by iteratively matching dummy gradients to users’ gradients.DLG optimizes both dummy data and labels, while iDLG analytically obtains labels from last-layer gradients.
  • Optimization-based attacks: DLG often fails to recover high-fidelity data and ground-truth labels because the optimization may not converge.Related methods improve convergence or regularization but retain the same underlying pathology.
  • Optimization-based attacks: Optimization-based attacks are computationally expensive because they require GAN training or second-order gradient optimization.The paper contrasts this with extracting exact data points directly from gradients without optimization or GAN training.
  • Active attacks: Active attacks can require detectable architecture or parameter modifications, including repeated weight patterns, zeroed parameters, or many updates from one user.These requirements constrain stealthiness or applicability in cross-device FL with randomly sampled participants.
  • Active attacks: The paper’s trap weights avoid highly regular weight matrices while making an output neuron likely to activate for only one mini-batch input.This design is presented as a less noticeable alternative to prior active attacks.

4. Threat Model and Assumptions

The threat model treats the central party as an active attacker that controls shared model weights and receives user gradients. The attack assumes local gradient computation on mini-batches, scaled features, limited attacker-side domain data, and victim architectures containing ReLU-based fully connected layers, with extensions to convolutional and token-embedding models.

  • 4.1. The Attacker: The attacker is the untrusted central party, or a third party controlling it, and targets individual training points from selected users.The central party is the entity orchestrating FL and is the vantage point for the attack.
  • 4.2. Assumptions and FL Setup: The central party chooses the task, instantiates the shared model, controls its weights, and can read users’ gradient updates.These capabilities provide the attacker with knowledge of the data domain and dimensionality and control over protocol models.
  • 4.2. Assumptions and FL Setup: Users compute gradients locally on one potentially large mini-batch and send the resulting gradients directly to the central party.The setup assumes features are scaled to [0, 1]; users may average gradients over multiple mini-batches when they have abundant data.
  • 4.2. Assumptions and FL Setup: The attacker is assumed to possess a small amount of data, such as one mini-batch, from the users’ private data domain.The paper treats this assumption as compatible with the central party’s task and model-selection responsibilities.
  • 4.2. Assumptions and FL Setup: The attack is designed for victim models containing a ReLU-based fully connected layer, rather than being architecture-agnostic.The paper also extends it to networks with convolutional layers and evaluates extraction with a token-embedding layer.
  • 4.2. Assumptions and FL Setup: The central party can target selected users in chosen rounds by sending different manipulated models to attacked and other users.After receiving attacked users’ gradients, it projects appropriate gradient portions onto the input domain for extraction.

5. Passive Analytical Extraction for FC-NNs

Fully connected layers can leak individual inputs directly through ordinary gradients, including gradients averaged over large mini-batches. The leakage can be extracted analytically, including in FedAvg model updates.

  • 5.1. Single-Input Gradients Directly Leak Input: A first fully-connected ReLU layer can directly reveal its input because the gradient of a weight row contains a scaled version of that input.This applies when the layer has a bias and at least one relevant output-gradient entry is non-zero.
  • 5.1. Single-Input Gradients Directly Leak Input: For B = 1, a non-zero bias gradient enables perfect reconstruction of the input from the corresponding weight and bias gradients.The weight gradient contains a scaled input, and the bias gradient provides the scaling factor for rescaling it at zero optimization cost.
  • 5.2. Mini-batch Gradients Directly Leak Some Individual Inputs: B = 100 gradients for CIFAR10 can visibly distinguish some individual training points, showing leakage is not limited to single-example batches.The leakage arises because different inputs contribute differently to the averaged gradient.
  • 5.2. Mini-batch Gradients Directly Leak Some Individual Inputs: In larger mini-batches, the weight gradient is a loss-weighted overlay of all inputs, but ReLU can zero out every contribution except one.When only x∗ contributes non-zero gradients, extraction reduces to the B = 1 case and can reconstruct x∗ exactly.
  • 5.3. Individual Inputs still Leak from Mini-batch Gradients computed in FedAvg: In FedAvg, the central party can recover local gradient information by comparing the model sent to a user with the returned model and rescaling by the learning rate.This exposes the same gradient structure after local training iterations, enabling the analytical extraction mechanism in the FedAvg setting.

6. Active Adversarial Initialization of the First Fully-Connected Layer

An active central party amplifies accidental gradient leakage by adversarially initializing the first fully-connected layer with trap weights. The method preserves the model architecture, improves extraction from large batches, and extends to convolutional and text models.

  • 6.2. Adversarial Weight Initialization: Trap weights re-scale components within each weight row, preserving unmodified model architectures while making the attack less conspicuous.The initialization increases the chance that only one input activates a neuron, causing other inputs to receive zero gradients.
  • 6.2. Adversarial Weight Initialization: The scaling factor s controls how aggressively negative weight components suppress neuron activations, and its ideal value depends on the dataset.The attacker can tune s using a small amount of domain data, passive leakage, or same-dimensionality data.
  • 6.2. Adversarial Weight Initialization: 51.4% of active neurons isolated individual ImageNet inputs with trap weights, versus 4.4% with Gaussian random weights at σ = 0.5.The comparison used 1000 neurons and shows that adversarial initialization increases single-input activations by more than a factor of 10.
  • 6.2. Adversarial Weight Initialization: 45.7% of data points were individually extracted from ImageNet mini-batches of B = 100 with trap weights, versus 21.8% with random model weights.Different neurons tend to activate on different individual inputs, allowing multiple points to be recovered from one batch.
  • 6.3. Extending the Attack to Other Architectures: The attack requires a ReLU-based fully-connected layer at the model’s beginning, while malicious weight manipulations can extend it to CNN architectures.For text models, extracted embeddings can be mapped back to tokens using a lookup dictionary.

7. Experimental Evaluation

Experiments evaluate passive and active gradient-based extraction across image and text datasets, architectures, initialization settings, and mini-batch sizes. Trap-weight initialization substantially improves exact individual-point reconstruction while remaining applicable to unmodified architectures and large batches.

  • Experimental Setup: The evaluation covers MNIST, CIFAR10, ImageNet, and IMDB using fully connected and convolutional neural networks.The experiments instantiate the attack across three image datasets and one text dataset, testing both FC-NNs and CNNs.
  • Active Manipulations: 54.0% extraction-recall is achieved on MNIST with trap weights, approximately nine times the passive attack’s result.The best MNIST scaling factor is s = 0.7, averaged over ten random adversarial initializations.
  • Active Manipulations: For CIFAR10 and ImageNet, trap weights perfectly reconstruct 54.0% and 45.7% of individual points, respectively, from batches of 100.The active attack is more than twice as successful as the passive attack on these datasets.
  • Active Manipulations: 65.4% extraction-recall is obtained on IMDB with s = 0.99, about 2.5 times the passive attack’s result.This result uses 1000 neurons and mini-batches of 100 data points.
  • Robustness and Comparisons: Extraction succeeds even when CIFAR10 mini-batches contain data from only one class, and the attack requires no back-propagation or complex optimization.The method extracts individual points perfectly with zero ℓ2 error in the reported comparison and same-class setting.
  • Scaling Factors and Batch Size: Smaller mini-batches and more extraction neurons improve reconstruction; with 3000 neurons, up to 50% of MNIST points are perfectly extracted from batches of 200.Mini-batches of 20 MNIST points are entirely extractable in this setting, while pooling compresses extracted CNN inputs.
  • Scaling Factors and Batch Size: Averaging gradients over 20 mini-batches of 100 reduces MNIST extraction-recall from 54.0% to 2%.The passage attributes this reduction to multiple data points overlaying in the gradients and notes extraction must occur before averaging.

8. Defending Privacy in Federated Learning

The paper discusses defenses against trap-weight attacks, combining differential privacy with empirical mitigations such as trusted hardware, local averaging, large mini-batches, and lossy layers. Each defense has important limitations, including trust assumptions, overhead, weak privacy–utility trade-offs, or reduced usability.

  • Differential privacy is presented as the gold standard for bounding private-information leakage from model gradients.
  • Differential privacy: Centralized differential privacy cannot protect against a malicious central party that extracts user data before adding noise or omits the noise entirely.
  • Differential privacy: Local differential privacy reduces trust in the central party but has poor privacy–utility trade-offs and is unpopular in practical applications.
  • Differential privacy: Distributed differential privacy uses local noise and aggregate privacy guarantees, but secure aggregation adds computational overhead and can be eluded in federated learning.
  • Attack-specific mitigations: Large mini-batches, local gradient averaging, lossy layers, and trusted execution environments can mitigate extraction, but provide empirical protection, reduce fidelity, or retain side-channel and deployment risks.
  • Attack-specific mitigations: Model-based protection is limited when the central party is untrusted because it controls the shared model’s architecture and hyperparameters.

9. Discussion and Future Directions

The discussion examines how trap weights may be detected and integrated into training, while arguing that weight manipulation creates a broader attack surface. Detection remains difficult because legitimate training is stochastic and users cannot reliably assess model convergence or update integrity.

  • Detectability: Users can seek trap weights by analyzing shared-model weights across one or multiple iterations or by testing the model’s behavior on their data.
  • Detectability: Weight-based detection can inspect first-layer distributions and asymmetries, but distinguishing malicious from legitimate updates across iterations is difficult.
  • Detectability: Training stochasticity, hardware non-determinism, and invisible user data prevent users from reliably verifying whether parameter changes resulted from legitimate optimization.
  • Detectability: Model-performance detection is unreliable because users’ examples may come from distribution tails and therefore may not achieve improved accuracy during federated training.
  • Integration into training: The attack can be intermittently deployed while the central party maintains shared-model utility between adversarial rounds.
  • Future directions: Manipulating neural-network weights creates a potential attack surface beyond federated learning by transmitting inputs through fully-connected and convolutional layers.
  • Privacy implications: Federated learning avoids directly sharing data, but its protocol does not itself guarantee protection against private-data leakage.

10. Conclusion

The paper presents an active privacy attack that maliciously manipulates shared-model weights to reconstruct user data from federated-learning updates. It reports highly efficient, near-perfect extraction at large mini-batch sizes and argues that additional privacy protections are necessary.

  • The attack uses malicious manipulation of shared-model weights to reconstruct a significant portion of users’ private training data.
  • Roughly 50% of individual ImageNet data points are perfectly extracted from mini-batches as large as 100, including batches containing data from one class.
  • The extraction is computationally highly efficient and achieves perfect reconstruction of individual points from same-class mini-batches.
  • Federated learning requires appropriate mitigations to provide meaningful privacy protection, despite keeping raw data on user devices.

A. Extended Related Work on Passive Data Reconstruction Attacks

The related-work section situates the paper among data-reconstruction attacks, including class-wise representation methods and optimization-based instance reconstruction. It also identifies comparison criteria for evaluating these approaches against the proposed attack.

  • Table 8 compares data-reconstruction attacks using dimensions including attacker role, server or user knowledge, class information, individual-point recovery, mini-batch size, optimization, training, and representation type.
  • Class-wise representation reconstruction attacks: Class-wise attacks reconstruct generic or per-user class representations rather than individual data points and require data from the users’ distribution.
  • Optimization-based instance reconstruction attacks: Optimization-based instance attacks reconstruct individual points while relaxing the assumption that attackers know the labels, but formulate reconstruction as an optimization problem.

B. Generalization of Data Extraction Attack to Convolutional Neural Networks

The attack is extended to convolutional networks by initializing convolutional filters to forward input features into a fully connected layer, while suppressing non-transmitting noise before reconstruction.

  • Convolutional layers are initialized to transmit model inputs unchanged to subsequent fully connected layers, reducing CNN extraction to the previously handled setting.
  • The forwarding design must preserve every input feature and prevent different features from being overlaid across feature maps.
  • Two Dimensional Input: For two-dimensional inputs, stride one, zero-padding, and filters with a central one preserve spatial size while forwarding features.
  • Three Dimensional Input: For three-dimensional inputs, separate filters copy each input channel, while unused filters may be randomly initialized and ignored by the attacker.
  • Before flattening, noise-producing filters are driven negative through ReLU so only transmitted feature maps remain for fully connected reconstruction.
  • Size-reducing convolutions distribute features across multiple output maps and require a number of filters proportional to the reduction factor squared.
  • Standard filter counts and random initialization of unused filters make the adversarial CNN architecture less conspicuous, while the method applies when layers have enough parameters for input features.

D. Additional Experimental Results

Additional experiments examine passive extraction from randomly initialized fully connected networks and introduce the architectures used for IMDB experiments.

  • The additional experiments include extraction from a randomly initialized fully connected network using the architecture specified in Table 9.
  • Table 10 specifies the model architecture used for experiments on IMDB, including vocabulary size, embedding size, activation function, and neuron count.
  • The passive baseline extracts CIFAR10 data from gradients of the first 30 weight rows in the first fully connected layer.

D.2. Trap Weights and Active Extraction

Experiments characterize trap-weight extraction across datasets, batch settings, model effects, and federated-learning benchmarks, including partial leakage when individual recovery fails.

  • Partial leakage occurs when an extracted gradient overlays only a few data points but still preserves distinguishable individual signals.
  • Trap weights amplify naturally extractable points and create redundancy by making some points recoverable from multiple weight-row gradients.
  • Non-IID CIFAR10 experiments directly compare active and passive extraction when each user holds data from a single class.
  • The attacker can tune scaling factor s from valid-range points passively extracted during an initial federated-learning round, without auxiliary data.
  • Dropout and pooling are evaluated for batch sizes 1 and 20; a later dropout layer has little effect because extraction occurs before its information loss.
  • Even when lossy layers reduce reconstruction fidelity, individual data points often remain recognizable.
  • Over 50 users, EMNIST reaches 0.32 ± 0.07 extraction-recall and 0.05 ± 0.02 extraction-precision, versus 0.10 ± 0.05 and 0.02 ± 0.01 in the baseline.
  • For CIFAR100, the attack reaches 0.44 ± 0.05 extraction-recall and 0.22±0.06 extraction-precision, versus 0.20±0.04 and 0.04±0.01 in the baseline.
Loading 2112.02918v2…