PaperlayerView exact revision on arXiv

Source-linked AI summary

Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models

Liam Fowl, Jonas Geiping, Wojtek Czaja, Micah Goldblum, Tom Goldstein

arXiv:2110.13057v2cs.LGcs.CR

TL;DR

Federated learning aims to train models without exchanging local data, yet prior attacks recover only limited or approximate information and weaken on aggregated batches. This paper introduces minimal malicious changes to shared model architectures that let servers directly recover user data from updates, including data from large batches. The authors conclude that large-scale averaging alone does not guarantee user privacy.

  • Problem

    Prior attacks reveal information from federated updates but degrade on moderately large batches, often recovering only approximations and relying on favorable model or data structure.

  • Method

    The paper modifies shared model architectures and parameters minimally so update patterns encode fixed subsets of user data for direct recovery.

  • Results

    The imprint method recovered almost perfect reconstructions of most data in batches of 64 ImageNet images, achieving 75.75 average PSNR versus 12.93 for prior batched reconstruction at batch size 8.

  • Takeaways & Limitations

    Large-scale averaging does not guarantee user privacy when a malicious server can distribute minimally modified models and parameters.

  • Takeaways & Limitations

    If aggregation is the only security source, the attack can uncover private samples but may not directly connect them to a specific user; local differential privacy is proposed as mitigation.

Abstract

from arXiv · show

Federated learning has quickly gained popularity with its promises of increased user privacy and efficiency. Previous works have shown that federated gradient updates contain information that can be used to approximately recover user data in some situations. These previous attacks on user privacy have been limited in scope and do not scale to gradient updates aggregated over even a handful of data points, leaving some to conclude that data privacy is still intact for realistic training regimes. In this work, we introduce a new threat model based on minimal but malicious modifications of the shared model architecture which enable the server to directly obtain a verbatim copy of user data from gradient updates without solving difficult inverse problems. Even user data aggregated over large batches -- where previous methods fail to extract meaningful content -- can be reconstructed by these minimally modified models.

1 INTRODUCTION

Federated learning trains models across user devices by exchanging model updates rather than local data, but those updates can still expose user information. This work studies malicious servers that make minimal model changes to recover private data despite aggregation defenses.

  • 1 INTRODUCTION: Federated learning distributes model training across user devices while exchanging model updates instead of local data.A central server sends model states, users compute local updates, and aggregated updates train the model over multiple rounds.
  • 1 INTRODUCTION: Data privacy is not guaranteed because it depends on many interdependent settings and design choices in each federated learning system.The paper focuses on systems where the central server cannot directly view user data.
  • 1 INTRODUCTION: Model updates can sometimes be inverted to recover input user information, especially when updates use a single data point or small batch.Aggregation over many local data points and multiple users is described as a strong defense against these attacks.
  • 1 INTRODUCTION: Prior inversion attacks generally assume an honest-but-curious server that examines updates without modifying the federated learning protocol.The paper argues that this assumption is contrived when the server can alter model behavior.
  • 1 INTRODUCTION: A malicious API participant may change model architecture and parameters without forcing edge devices to send user data directly.This threat model targets suspicious behavior in the shared model rather than changes to client code or learning behavior.
  • 1 INTRODUCTION: Minimal architectural changes create structured updates whose components reveal fixed subsets of data points exactly, even after large aggregation.For models with large linear layers, changing only layer parameters can suffice.

2 LIMITATIONS OF EXISTING ATTACK STRATEGIES

Existing privacy attacks can recover information from federated updates, but their success degrades for batched data and often depends on favorable model or data structure. The paper positions these limitations as motivation for examining small malicious architectural modifications.

  • 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES: Analytic attacks recover a single input from an affine-layer gradient, but multiple inputs generally yield only their average.Recursive and linear-equation extensions improve scope but retain substantial recovery limits for batched inputs.
  • 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES: Optimization-based attacks reconstruct inputs by solving gradient-matching objectives, with later work adding target recovery, similarity objectives, and generative priors.Additional regularizers can further improve image reconstruction.
  • 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES: Previous attacks cover fedSGD and some fedAVG settings, but image-focused methods benefit from strong image structure and priors unavailable for weaker-structure data such as tabular data.The paper states that its approach is more data-agnostic because it does not rely on such tricks.
  • 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES: 28% of user data was recovered by a state-of-the-art attack for a batch of 48 ImageNet images on ResNet-50, under unlikely label collisions.Recovery drops sharply with larger batches; another attack leaked only a few samples at batch size 128 on ResNet-32-10.
  • 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES: Prior reconstructions are approximations that may omit user data or replace it with likely but unrelated information when strong image priors dominate.This limitation concerns both incomplete recovery and fidelity to the actual data.
  • 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES: Existing studies often use privacy-sensitive architectural choices, motivating the question of the smallest neural-network modification that can break privacy.Examples include large gradient vectors, special activations, wide models, and representation-learning models.

3 MODEL MODIFICATIONS

The paper introduces imprint modules: minimal architectural modifications that structure gradients into bins, enabling analytic recovery of individual data points even within larger networks and batches.

  • 3.2 A SIMPLE EXAMPLE: For a batch with unique labels, multinomial logistic regression can theoretically recover every data point exactly, up to the number of output classes.The ImageNet example has 1,000 classes, allowing up to 1,000 unique data points in the optimal case.
  • 3.2 A SIMPLE EXAMPLE: Naive linear reconstruction can recover batched data only under restrictive conditions and becomes impractical for large datasets.Repeated labels cause averaging, placement before another network mixes gradient entries across labels, and ImageNet-scale recovery would require more than 150M parameters.
  • 3.3 IMPRINTING USER INFORMATION INTO MODEL UPDATES: Imprint modules preserve analytic reconstruction while overcoming the limitations of naive linear models through commonly used architectural components and modified parameters.The construction can also modify parameters of existing large linear layers rather than adding an entirely separate module.
  • 3.3 IMPRINTING USER INFORMATION INTO MODEL UPDATES: An imprint module uses a linear layer and ReLU to partition a scalar data statistic into equal-mass bins whose gradient differences isolate data points.The server estimates the statistic’s distribution or CDF, duplicates its measurement across channels, and chooses biases to define the bins.
  • 3.3 IMPRINTING USER INFORMATION INTO MODEL UPDATES: When a data point alone activates a bin, subtracting successive gradient rows isolates the inputs whose statistic lies between the corresponding bin boundaries.The isolation follows because higher-threshold bins exclude the uniquely activating point while retaining points above the next threshold.
  • 3.3 IMPRINTING USER INFORMATION INTO MODEL UPDATES: The expected number of perfectly reconstructed samples depends on batch size and the number of imprint bins, which the server can increase to improve recovery.Figure 1(a) shows the recovered proportion for batches of 64 ImageNet images as bins are added; 156 bins yield an expected recovery above 50%.

4 EXPERIMENTS

The experiments show that imprint modules recover user data from large aggregated updates, including nearly perfect recovery for 64-image batches and one-shot leakage from industrial-sized batches. Variants remain effective across local updates, model positions, and data modalities.

  • 4.1 FULL BATCH RECOVERY: 75.75 average PSNR was achieved when recovering data from updates aggregated over 64 ImageNet images.The modified ResNet-18 used an imprint module with 128 bins, while prior optimization-based reconstruction achieved 12.93 average PSNR for batches of only 8 images.
  • 4.2 PRIVACY BREACHES IN INDUSTRIAL-SIZED BATCHES – ONE-SHOT ATTACKS: A fused two-bin variant can leak one complete data point on average once every four incoming batches, regardless of batch size.For a ResNet-18 on ImageNet, this requires about 1% additional parameters.
  • 4.2 PRIVACY BREACHES IN INDUSTRIAL-SIZED BATCHES – ONE-SHOT ATTACKS: 161.36 PSNR produced a machine-precision copy from averages aggregated over 16,384 datapoints.The recovered image included a license plate that was blanked out in the figure to preserve privacy.
  • 4.3 VARIANTS: With multiple local updates, a sparse-gradient imprint variant threatens federated averaging by enabling direct inversion without pairwise differences.Scaling the associated linear function reduces bin shifts, making total data used in an update the key variable again.
  • 4.3 VARIANTS: The attack remains stable across a range of batch sizes and positions within a ResNet-18 model.The figure evaluates identification success against bin size and model position using the IIP score.
  • 4.3 VARIANTS: The imprint module is data-agnostic and can target image, text, or tabular data, including models with existing linear layers.The experiments focus on images, but the method does not use assumptions unique to vision.

5 POTENTIAL DEFENSE AND MITIGATION STRATEGIES

The proposed attack can defeat aggregation-based protection, recovering private samples from arbitrarily large batches. Local differential privacy may mitigate the attack, but effective protection can require substantial noise.

  • Aggregation alone does not prevent the attack from uncovering private samples in arbitrarily large batches.The One-shot mechanism is identified as especially effective against aggregation-based security.
  • Secure aggregation may conceal the affected users’ identities but does not prevent the underlying data breach.When imprint modules reveal securely aggregated data, secure aggregation is reduced to secure shuffling.
  • Local differential privacy can mitigate the attack without requiring central-server coordination or consent.Adding sufficient gradient noise can cause potentially unbounded reconstruction errors because the attack divides by a learned quantity.
  • Even when the correct noise scale is unknown, practical privacy can remain broken unless substantial noise is added.Private data remains visibly leaked in an appendix example with σ = 0.01.

6 CONCLUSIONS

Federated learning does not guarantee user privacy, even with large-scale averaging. The paper shows that malicious servers can use common, inconspicuous modules and minimally modified models and parameters to breach privacy.

  • Large-scale averaging in federated learning does not guarantee user privacy.
  • A malicious server can breach privacy by sending minimally modified models and parameters in a federated setup.The paper frames these modifications as common and inconspicuous machine-learning modules.
  • The examples are intended to clarify current privacy limitations and inform discussions of future federated-learning API design.

ETHICS STATEMENT

The paper presents an attack with potential to compromise user privacy while emphasizing that its fundamental purpose is to inform the community about federated learning’s privacy limitations.

  • The work uncovers an attack with the potential to compromise user privacy.
  • The authors state that the research is intended to inform the community about the state of privacy in federated learning.
  • The paper aims to help users and technical experts understand federated learning’s limitations for preserving user privacy.

REPRODUCIBILITY STATEMENT

The paper provides appendix technical details, a proof of Proposition 1, and open-source implementations of all investigated attacks to support reproducibility.

  • The appendix provides additional technical details and the proof of Proposition 1.
  • Open-source implementations of all attacks investigated in the work are provided.
  • The listed repositories include a minimalistic attack implementation and a broader federated-learning privacy-attack framework.
  • The experiments require no GPU.

A.1 PROOF OF PROPOSITION 1

The proposition’s proof counts expected perfectly recovered samples by partitioning a scalar data quantity into equal-mass imprint bins and analyzing bin occupancies. Singleton bins yield exact recovery, while residual terms account for configurations and tail data not covered by bins.

  • Proof strategy: The imprint construction perfectly recovers every sample in a bin containing exactly one batch element.Equal-mass bins are formed using the known continuous CDF of a scalar quantity measured by a linear function.
  • Proof strategy: The expected recovered amount is obtained by summing configurations weighted by the number of singleton bins.Configurations with i singleton bins contribute weight i, while the remaining bins contain either zero or at least two elements.
  • Occupancy counting: At most floor((n−i)/2) remaining bins can contain data when i bins are singletons, because occupied non-singleton bins require at least two elements.This constraint converts the occupancy count into an integer-orientation problem.
  • Occupancy counting: The occupancy constraints reduce to a stars-and-bars count after selecting singleton bins and bins containing at least two elements.The transformed variables are nonnegative and distribute the remaining elements among the selected bins.
  • Expected recovery: The final expectation includes residual terms r(n, k) for all-singleton configurations and tail samples outside the covered CDF bins.The cited derivation identifies these terms as corrections to the perfectly reconstructed amount.

A.2 OTHER CHOICES OF LINEAR FUNCTIONS AND DISTRIBUTIONS

The attack can use different linear queries and assumed distributions, with accuracy depending on how well the assumed distribution matches the data. Laplacian modeling improves image recovery, while Gaussian random projections generalize beyond images.

  • Distribution choices: Average brightness is approximately normally distributed on ImageNet, although the approximation is imperfect.The approximation is motivated by normalized color channels, many pixels, and limited pixel correlation.
  • Distribution choices: A Laplacian distribution with scale 1/2 slightly improves accuracy over the normal approximation for image brightness.The ground-truth distribution is better described by a Laplacian distribution.
  • Image queries: Using the 32nd DCT coefficient with a Laplacian distribution produces the strongest attack against image data.This choice relies on attacker knowledge that users train on natural images.
  • General-purpose queries: A Gaussian random vector with independent N(0, 1/√m) entries approximates a normal distribution better and applies to data modalities with bounded variance.This variant is less optimal than the Laplacian model for higher-frequency image features.
  • Distribution estimation: 0.1% of ImageNet data suffices for the server to obtain a close approximation to the brightness distribution.The estimate is evaluated through the density of image brightness.

A.4 OTHER DATA MODALITIES - TEXT

The imprint attack extends beyond images to text by inserting a malicious block after transformer word embeddings. With 512 bins, it reconstructs most sentences in a batch exactly or with high text similarity.

  • Scope: The attack is data-agnostic and can target tabular features, text, or even random signals.The authors verify this property experimentally on text data.
  • Method: The text experiment inserts the imprint module after word embeddings in a transformer and uses a Gaussian random linear function.The architecture is maliciously modified because standard transformer feedforward blocks do not span the entire sequence.
  • Results: 110 of 128 Wikitext sentences are perfectly recovered with 512 bins, yielding 86.33% reconstruction accuracy.The batch size is 128 and sequence length is 32.
  • Results: The text results also achieve BLEU 88% and ROUGE-L 87%.These metrics accompany the reported 86.33% reconstruction accuracy.
  • Recovery pattern: Recovered text shows the same pattern as image recovery: a subset of examples is exact, while other sentences are not recovered.The authors describe this as analogous to the image reconstructions in Figure 7.
  • Implementation: The implementation requires only a few tensor operations, making reconstruction faster and more portable than optimization-based techniques.The experiments ran on laptop and machine CPUs.

B REMARK ON RECOVERY

Equation (2) requires a downstream averaging operation or a linear layer with identical rows so that the relevant gradient entries remain aligned. The figures illustrate naive reconstruction behavior for ImageNet and CIFAR10 batches.

  • Equation conditions: Recovery from Eq. (2) uses averaging or a linear layer with identical row elements after W.This preserves the equality of the relevant bias gradients.
  • Equation conditions: The required gradient relation is preserved when the operation preceding the imprint module can be expressed as a matrix with identical rows.The paper states that averaging satisfies this sufficient condition.
  • Figure comparisons: For 64 ImageNet images, analytic reconstruction reaches PSNR 36.45 for separate classes but 13.84 for 64 white-shark images.The figure compares reconstruction against the true user data under the same recovery algorithm.
  • Figure comparisons: Figure 12 contrasts a 64-image CIFAR10 batch with its naive reconstruction using Eq. (2).The panels show raw data beside the reconstruction.
Loading 2110.13057v2…