PaperlayerView exact revision on arXiv

Source-linked AI summary

Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification

Yuxin Wen, Jonas Geiping, Liam Fowl, Micah Goldblum, Tom Goldstein

arXiv:2202.00580v2cs.LGcs.CRcs.CV

TL;DR

Federated-learning updates may reveal private user data, while prior inversion attacks have been limited by small batches or model modifications. The paper uses maliciously chosen parameters to fish for individual examples from large aggregates, and reports that query needs scale logarithmically with batch size. It also notes that available defenses have practical limitations.

  • Problem

    Prior gradient-inversion attacks largely addressed small batches or settings requiring modified architectures, limiting their coverage of realistic aggregated data.

    Source: §1 · p. 1

  • Method

    The approach uses maliciously chosen parameter vectors to magnify a target example’s gradient and reduce the contributions of other data.

    Source: §1 · p. 2

  • Results

    The one-shot binary attack’s query count scales as log(n), averaging 8 queries to retrieve one image from an aggregation of 256 images.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The authors conclude that untrusted server updates let attackers fish for separate data points from arbitrarily large batches without modifying the model.

    Source: §7 · p. 8

    Detecting sparse-gradient attacks may also flag benign examples, while secure median aggregation remains uncertain and may still expose a target as the median.

    Source: §5 · p. 8

Abstract

from arXiv · show

Federated learning (FL) has rapidly risen in popularity due to its promise of privacy and efficiency. Previous works have exposed privacy vulnerabilities in the FL pipeline by recovering user data from gradient updates. However, existing attacks fail to address realistic settings because they either 1) require toy settings with very small batch sizes, or 2) require unrealistic and conspicuous architecture modifications. We introduce a new strategy that dramatically elevates existing attacks to operate on batches of arbitrarily large size, and without architectural modifications. Our model-agnostic strategy only requires modifications to the model parameters sent to the user, which is a realistic threat model in many scenarios. We demonstrate the strategy in challenging large-scale settings, obtaining high-fidelity data extraction in both cross-device and cross-silo federated learning.

1. Introduction

Gradient inversion can expose private data from federated-learning updates, but prior attacks were limited by small batches or modified architectures. The paper introduces malicious server parameters to fish out individual examples from large aggregates.

1. Introduction

  • Earlier attacks were demonstrated mostly on small batches under an “honest-but-curious” server; prior work suggested batch size 32 could deter attacks for unmodified architectures, whereas fishing compromised one example from arbitrarily large batches with malicious parameters.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The paper frames malicious server parameters as a distinct threat model in which users cannot trust that server-sent updates are benign.The authors argue that users should regard server updates as untrusted because they are outside users’ control.Source: §1 · p. 2
  • This threat model is presented as realistic because a company may support federated learning while still seeking private user information through attacks.The passage gives regulatory compliance as an example of public support for federated learning.Source: §1 · p. 2
  • The attack magnifies a target example’s gradient while reducing others’ contributions, extending existing optimization-based and analytic attacks to large aggregates.The authors consider both cross-device and cross-silo settings.Source: §1 · p. 2

2. Background and Related Work

Prior gradient-inversion methods struggle with realistic aggregated batches, and malicious-parameter attacks can face architectural or protocol constraints. This work extends fishing strategies to arbitrary models and large aggregates.

2. Background and Related Work

  • Cross-device FL involves many unreliable users with small data holdings, whereas cross-silo FL involves a small, identifiable group with larger private datasets.Cross-device users are selected at random; cross-silo users can be queried each round.Source: §2 · p. 2
  • Analytic attacks can recover inputs for specific layers and architectures, but aggregated gradients mix data from multiple examples rather than revealing single inputs.The cited examples include linear layers, vision transformers, and recursively attacked CNNs.Source: §2 · p. 2
  • Optimization-based attacks handle somewhat larger batches than analytic attacks, but still cannot invert large-batch gradients.The passage cites small batch sizes such as 8 and notes that aggregation can therefore serve as a defense.Source: §2 · p. 2
  • Earlier malicious-parameter attacks could recover data from arbitrary batch sizes, but often required conspicuous architecture changes, individually targeted users, or side-channel knowledge.Some methods were restricted to large early linear layers or users with few data points.Source: §2 · p. 2
  • Several prior malicious-parameter attacks exploit gradient sparsity, making aggregated entries attributable to individual examples or users.Fowl et al. use a cumulative-sum structure; other cited attacks make most relevant gradients zero.Source: §2 · p. 3
  • The paper extends fishing to arbitrary models and arbitrarily large aggregates without architecture changes, individualized server updates, or side-channel knowledge.It also applies the approach to attacks that previously did not work beyond one data point.Source: §2 · p. 3

3. Method

The method uses maliciously chosen server parameters to reduce aggregated updates toward gradients from selected examples, under a threat model allowing modified updates. Class fishing selects a class, feature fishing separates examples within it, and cross-device and cross-silo strategies adapt these mechanisms to their query settings.

3. Method

  • The threat model assumes a malicious server can send modified updates; cross-device users may be queried once, whereas cross-silo users can be queried over successive rounds.The method focuses on multi-class classification and federated learning protocols, principally fedSGD.Source: §3 · p. 3

3.1. Class fishing strategy

  • Class fishing lowers prediction confidence for a target class by changing classification-layer parameters, making that class's gradient contribution dominate the aggregate update.For cross-entropy loss with bounded per-example gradients, the proposition states that a suitable parameter choice can make the feature-extractor update arbitrarily close to the target-class contribution.Source: §3.1 · p. 3

3.2. Feature fishing strategy

  • Feature fishing shifts a decision boundary along one feature to suppress selected same-class examples, isolating a target image when class fishing would leave a mixed update.The server can estimate the feature mean from gradients; the strategy requires a cutoff that separates the target class along that feature without increasing loss for non-targeted data.Source: §3.2 · pp. 4–5

3.3. Cross-device

  • In cross-device FL, the one-shot feature attack estimates a feature distribution from some users, then applies feature fishing to retrieve one data point from each remaining user.It chooses a feature with a more normal distribution because that feature can be estimated effectively from limited measurements.Source: §3.3 · p. 5

3.4. Cross-silo

  • Cross-silo binary attacks need O(n log n) queries on average to recover all per-example gradients, while the one-shot binary attack needs O(log n).In the worst case, the respective query counts are O(n^2) and O(n); the analysis assumes n target-class examples and a uniformly distributed selected feature.Source: §3.4 · p. 6

4. Experiments

Image-classification experiments evaluate fishing attacks with feature estimation, binary isolation, and APRIL. Across these settings, fishing lets existing attacks operate on larger aggregates, including cases where the original attack fails.

4. Experiments

  • The experiments assess whether fishing attacks breach privacy on large user aggregates and compare them quantitatively with prior work.The evaluation focuses on image classification.Source: §4 · p. 6

4.1. Experimentation details

  • Experiments use ImageNet ILSVRC 2012 image classification, applying both fishing strategies to the last linear layer of a pretrained ResNet-18 except in the APRIL experiment.ImageNet has 1,000 classes; image recovery uses the optimization method of Geiping et al. (2020).Source: §4.1 · p. 6

4.2. One-shot feature attack

  • 37% of the time, accurate estimation of the chosen feature distribution yields recovery of a user image.A larger number of users improves the estimated feature distribution; choosing the most normal feature helps, but even a random feature can recover private data.Source: §4.2 · p. 7

4.3. Binary attack

  • 8 queries recover an individual data point from an aggregation of 256 images on average, and the necessary query count scales as log(n).The experiments sample users whose data share a label, maximizing label collisions.Source: §4.3 · p. 7

4.4. Fishing with closed-form APRIL attacks

  • On a modified ViT-Small, original APRIL fails at batch size 2 as patches mix, whereas the proposed fishing attack remains consistent at larger batch sizes, scaling to arbitrary sizes.Source: §4.4 · p. 7

5. Defenses and Mitigation Strategies

Under the malicious-parameter threat model, further aggregation does not guarantee protection, and defenses have practical limits. Local differential privacy can break these attacks, but stronger privacy guarantees tend to reduce model accuracy.

5. Defenses and Mitigation Strategies

  • Further aggregation does not guarantee protection against the attack, and users have few defenses under the proposed threat model.The attack and related methods produce sparse gradients, with some returned parameters dominated or entirely contributed by single examples.Source: §5 · p. 7
  • Median aggregation may be difficult to achieve in secure aggregation, and parameter modifications could still make the targeted data point the median.Detecting sparse-gradient patterns could also flag benign examples.Source: §5 · pp. 7–8
  • Sufficient local differential privacy breaks these attacks, but model accuracy tends to diminish as privacy guarantees increase.Local differential privacy clips per-example gradients and adds noise on the user side.Source: §5 · p. 8

6. Disparate Impact

The fishing attack creates non-uniform privacy exposure by targeting feature-space outliers first, which may include underrepresented groups or different data domains. The authors identify this disparate impact as a concern for future investigation.

  • The attack generally targets feature-space outliers first, which can correspond to underrepresented groups or different data domains.Source: §6 · p. 8
  • In cross-silo learning, recovering a median-feature point takes n log(n) queries on average, versus log(n) for the smallest-feature outlier.Source: §6 · p. 8
  • In cross-device learning, the attack most easily fishes out each user’s or user-group’s point with the lowest feature value.Source: §6 · p. 8
  • The authors identify disparate impact in privacy attacks as a concern for future investigation.Source: §6 · p. 8

7. Conclusions

The paper reports that malicious server updates let attackers fish for individual data points from arbitrarily large batches without modifying the model. The attacks apply across cross-device and cross-silo settings, and aggregation alone does not suffice as a defense.

  • The attacks fish for separate data points from arbitrarily large batches without modifying the model.Source: §7 · p. 8
  • Figure 5 evaluates the one-shot binary attack through query counts across batch sizes and results on large ImageNet batches.Source: Figure 5 · p. 9
  • The attacks can break user privacy in both cross-device and cross-silo scenarios, while aggregation alone is insufficient protection.Source: §7 · p. 9

A. Proofs.

The proofs formalize how class and feature fishing can isolate target data contributions within aggregated updates. They also state conditions for suppressing non-target gradients and identify the high-magnitude target gradient as useful for reconstruction.

  • The proof frames the class-strategy calculation as showing that non-target-class images contribute minimally to the feature-extractor update.Source: A. Proofs · p. 12
  • Target-class image gradients are not suppressed in the same way as non-target image gradients.Source: A. Proofs · p. 13
  • Proposition 3.2 assumes a known continuous target-class feature CDF and bounded per-example cross-entropy gradients.Source: A. Proofs · p. 13
  • The proof constructs equal-mass feature bins and shows that when exactly one target-class image falls in a bin, parameter β can make its softmax probability arbitrarily small while making other images’ probabilities approach one.Source: A. Proofs · p. 13
  • Combining feature fishing with class fishing yields the stated gradient result, while a high-magnitude gradient for the fished image improves reconstruction.Source: A. Proofs · p. 14

B. Technical Details

The experiments use a PyTorch implementation, ImageNet validation data partitioned among 100 users, and specified assumptions about batch normalization and label metadata.

  • The implementation uses PyTorch; optimization-based attacks run on single 2080ti GPUs, while analytic attacks use APRIL on CPUs with an SVD solver for under-determined problems.Source: B. Technical Details · p. 14
  • ImageNet validation data are partitioned among 100 users, with users assigned either distinct classes or random images without replacement.Source: B. Technical Details · p. 14
  • Optimization experiments fix batch-normalization layers in evaluation mode, and the experiments assume label metadata is present while noting label-recovery methods can succeed at large aggregation sizes.Source: B. Technical Details · p. 14

C. Additional Results

The additional results show recoveries from 50 users at batch sizes 128 and 256, and illustrate feature fishing and a shark-only batch.

C. Additional Results

Loading 2202.00580v2…