PaperlayerView exact revision on arXiv

Source-linked AI summary

Cascading Gradient Inversion via LT-Code Inspired Peeling in Federated Learning

Saeed Shariati, Mohsen Alambardar Meybodi

arXiv:2609.09659v1cs.LGcs.AIcs.CR

TL;DR

Federated-learning updates can be inverted to reconstruct client data, while prior single-round analytic attacks degrade with batch size and face isolation-based limits. The paper connects gradient inversion to LT-code peeling, adds soliton-shaped activation control, and reports exact, certified recovery of large labeled batches from one FedSGD round. Across eight image and tabular datasets, recovery reaches near-complete passive performance for image batches up to 128 and exceeds 90% in active settings at batch sizes of several hundred.

  • Problem

    Shared federated-learning updates can leak training data, while prior single-round analytic attacks recover only a fraction of large batches and face isolation-based upper bounds.

  • Method

    The paper uses an iterative recover-and-subtract cascade analogous to LT-code peeling and shapes first-layer neuron degrees toward the robust soliton distribution.

  • Results

    Across eight image and tabular datasets, the attacks recover large batches exactly with labels from one FedSGD round, achieving 94–100% passive recovery up to batch size 128 and more than 90% active recovery at batch sizes of several hundred.

  • Takeaways & Limitations

    The results show that federated-learning privacy leakage is greater than prior single-round attack results suggested.

Abstract

from arXiv · show

Federated learning shares model updates rather than raw data, yet these updates can be inverted to reconstruct the clients' training data. Analytic reconstruction attacks, which invert a gradient in closed form, degrade as the batch grows: prior single-round attacks recover only about half of a batch of size $100$ even when the attacker fully controls the network parameters, and known upper bounds limit what any such method can recover. We establish a connection between gradient inversion and the theory of erasure-correcting codes, and use it to construct attacks that exceed these bounds. Our attacks recover batches exactly, together with every sample's label, from a single FedSGD round, and certify each recovery without ground-truth data. On eight image and tabular benchmarks they outperform prior single-round attacks by a wide margin. Even a passive attacker who only observes an honestly trained network recovers $94$--$100\%$ of ImageNet batches at sizes up to $128$, more than prior single-round attacks achieve even with active manipulation of the model, and in the active setting more than $90\%$ is recovered at batch sizes of several hundred. These results show that the privacy leakage of federated learning has been underestimated.

I. INTRODUCTION

Federated-learning updates can leak client training data, while prior single-round analytic attacks are limited on large batches. This paper introduces cascading recovery and soliton-shaped activation degrees to recover substantially larger batches in active and passive settings.

  • Federated learning shares model updates instead of raw data, but those updates can enable data reconstruction and other privacy attacks.
  • Isolation-based analytic attacks are bounded by the batch's convex-hull vertices and recover only roughly half of 100 samples under active manipulation.The recoverable fraction decreases as batch size grows, while multi-round alternatives require repeated controlled interaction.
  • The cascading procedure recovers isolated samples, subtracts them from the gradient, and uses the residual to expose further samples, paralleling LT-code peeling.
  • Soliton-Free and Soliton-Data shape first-layer neuron degrees toward the robust soliton distribution, with the former requiring no auxiliary data and the latter using one auxiliary batch.
  • The attacks recover exact samples with correct labels and certify every recovery without ground-truth data, applying to specified fully connected ReLU classification networks on image and tabular data.
  • In passive experiments, recovery reaches 94–100% for image batches up to 128 and a complete tabular batch at B = 64, outperforming prior passive and active single-round results.

A. Neural Networks for Classification

The paper analyzes classification networks with a fully connected ReLU first layer and derives how first-layer gradients encode samples, especially when neurons isolate them.

  • The attacks assume a classification network whose input passes through a fully connected first layer followed by ReLU, while later layers may be arbitrary.
  • For a batch, first-layer gradients accumulate contributions from every sample that activates each neuron.
  • A neuron’s activation set contains samples with positive pre-activation, and its degree is the number of samples in that set.
  • A degree-one neuron, called a singleton, isolates its unique activating sample.
  • When a sample is isolated, the corresponding weight-gradient and bias-gradient ratio recovers its input exactly.

C. Luby Transform Codes

The paper maps gradient recovery onto LT-code peeling: degree-one neurons reveal samples, and subtracting recovered contributions can create further singletons. Degree distributions are shaped toward the robust soliton distribution to sustain this cascade.

  • LT-code peeling: LT decoding repeatedly recovers message bits from degree-one checks and subtracts their contributions, reducing neighboring degrees and creating new degree-one checks.
  • Gradient-recovery analogy: In gradient inversion, batch samples act as message nodes and first-layer neurons act as check nodes.
  • Gradient-recovery analogy: A degree-one neuron permits exact sample recovery from weight- and bias-gradient entries; subtracting that sample from residual gradients propagates the cascade.
  • LT encoding: LT encoding selects check-node degrees from a prescribed distribution and connects each check to a uniformly chosen subset of message bits.
  • Robust soliton distribution: The robust soliton distribution adds perturbation mass near degree B/R, supplying higher-degree checks that sustain peeling after initial degree-one recoveries.
  • Robust soliton distribution: The attack deliberately shapes first-layer activation degrees toward the robust soliton distribution to maximize recovery.

III. RELATED WORK AND POSITIONING

The paper positions its method as a single-round exact reconstruction attack that extends beyond isolated samples through cascading subtraction, while also recovering labels and certificates. It targets limitations of isolation-based bounds and prior multi-round or low-scale methods.

  • The paper distinguishes sparsity-based isolation, recovery bounds, exact batch reconstruction, and label recovery as four positioning areas.
  • Sparsity-based analytic attacks: Unlike prior singleton-only attacks, the proposed method iteratively subtracts recovered samples to create new singletons and shapes degrees toward the robust soliton distribution.
  • Recovery bounds: Isolation-based attacks are bounded by the batch’s convex-hull vertices, but cascading subtraction can expose interior points after outer samples are removed.
  • Recovery bounds: Soliton-Data recovers HARUS batches in full at sizes up to 512.
  • Exact batch reconstruction: The attack reconstructs exact batches from one FedSGD round in passive and active settings, unlike methods requiring multiple controlled rounds or optimization.
  • Label recovery: Each recovered sample receives an individual label and a certificate from the first-layer bias residual via the logit Jacobian.

IV. THREAT MODEL AND ARCHITECTURAL ASSUMPTIONS

The threat model treats the server as the adversary in single-round FedSGD, distinguishing passive observation from active parameter manipulation. The attack requires only a fully connected ReLU first layer; later architecture affects recovery through sensitivities.

  • Threat model: In single-round FedSGD, each client returns the gradient of a local batch, and the server observes first-layer gradient components.
  • Threat model: The passive server follows the protocol, sends an unmodified model, and observes only returned gradients or updates.
  • Threat model: The active server may modify model weights and biases, using trap weights or Soliton-Data and Soliton-Free bias manipulation.
  • Threat model: Passive experiments use a standard random initialization of the first-layer weights.
  • Architectural assumptions: The architectural requirement is a fully connected layer followed by ReLU at the network input; subsequent layers may be arbitrary.
  • Architectural assumptions: The remainder of the network enters the attack through per-sample, per-neuron sensitivities, which the server computes after recovering a candidate sample.

V. METHOD

Peeling maintains the current residual gradient and uses singleton neurons to recover samples, while residual-bias consistency identifies their labels. The procedure also provides identifiability guarantees and avoids auxiliary data or per-class backward passes.

  • Residual gradients: The residual (G, h) is repeatedly updated after subtracting contributions from recovered samples.Initially it equals the observed gradient; subsequent ratios and certifications operate on the current residual.
  • Singleton extraction: A singleton neuron yields its activating sample because the unknown sensitivity coefficient cancels in the gradient-to-bias ratio.For neurons activating multiple samples, the same ratio instead produces a linear combination.
  • Singleton extraction: Non-singleton ratios can lie outside the data range because their coefficients may have different signs, limiting direct recovery to isolated samples.This is the convex-hull limitation of isolation-based extraction.
  • Label recovery: Labels are recovered by testing candidate classes and selecting the one whose residual-bias consistency error vanishes.The fit uses the neurons that isolate the recovered sample and relies on the cross-entropy model structure.
  • Computation: The label-recovery procedure requires one forward pass, one backward pass, and |Si| Jacobian–vector products per sample, with no auxiliary data.When Ji is constant, the procedure reduces to arithmetic.
  • Identifiability: For continuous random weights, distinct labels are identifiable with probability zero of ambiguity whenever at least one neuron isolates the sample.Duplicate labels elsewhere in the batch do not interfere because the fit uses only isolating neurons.

C. Certification: Distinguishing Genuine Candidates

Certification separates genuine samples from linear-combination candidates before destructive subtraction. Certified samples are labeled, batched into one gradient computation, subtracted, and used to expose further singleton neurons.

  • Candidate certification: Candidate ratios may represent invalid linear combinations, so certification is required before subtraction to prevent residual corruption.Subtracting an invalid candidate injects error into every subsequent recovery.
  • Candidate certification: A genuine candidate passes a sensitivity-consistency check computed from the server’s own forward and backward passes.The observed residual bias is compared with the predicted sensitivity on the neurons that produced the candidate.
  • Candidate certification: Linear combinations fail certification because their simulated activation masks and sensitivities do not correspond to any single summand or label.No label makes the certification residual small for such candidates.
  • Deduplication: Duplicate candidate vectors are merged, their neuron sets pooled, and candidates corresponding to already recovered samples are dropped.This consolidates multiple isolations of the same sample before certification.
  • Iterative peeling: Each iteration certifies distinct candidates, assigns labels, subtracts their combined contribution, and updates the residual.The implementation performs one batched backward pass for all samples certified in that iteration.
  • Iterative peeling: The loop terminates when no new sample is certified; otherwise subtraction can expose degree-1 neurons and continue the cascade.A neuron of original degree k yields its final sample after the other k − 1 samples are peeled away.

E. The Loop

The loop aims to make neuron degrees favorable for peeling by shaping first-layer activation patterns. Sparse rows seed cascades, but the LT-code analogy is limited because activation edges are data-dependent rather than uniformly random.

  • The Loop: The complete iterative procedure is summarized in Algorithm 1 and terminates when no remaining neuron isolates an unrecovered sample.Complete recovery occurs only if this terminal state is avoided until all B samples are extracted.
  • Degree shaping: The attack increases cascade length by shaping first-layer neuron degrees toward a robust soliton distribution.The construction begins with sparsity-inducing trap weights and then targets robust-soliton degrees.
  • LT-code caveat: The LT-code analogy is limited because neuron–sample edges are fixed by the data rather than selected uniformly at random.Samples inside the unrecovered convex hull may remain unable to become sole activators until peeling shrinks that hull.
  • Activation patterns: Sparse activation rows make singleton neurons possible, whereas uniform firing would make a singleton at B = 100 occur with probability roughly 10^-28.The stripe pattern combines a neuron-specific baseline with sample-specific deviations.
  • Mirrored construction: The mirrored construction can lose sparse rows when data features concentrate around a common value, causing recovery to collapse toward zero.Its approximately zero column sums eliminate the baseline term and stripe structure in that regime.
  • Passive setting: At s = 1, the independent construction matches standard Gaussian initialization and therefore represents a passive honest-but-curious setting.This provides a no-tampering reference for the attack.

B. Soliton-Free (SF)

Soliton-Free shapes neuron activation degrees without client data by approximating pre-activations as Gaussian and selecting bias thresholds for desired degrees. Sampling degrees from the robust soliton distribution targets high peeling success probability.

  • Gaussian approximation: Soliton-Free assumes approximately uniform, weakly dependent features so pre-activations can be modeled with a Gaussian approximation.The approximation uses feature means near 1/2, variances near 1/12, and negligible covariances.
  • Bias design: A neuron intended to activate on dj of B samples receives a bias threshold at the Gaussian (1 − dj/B)-quantile.The threshold is chosen so firing occurs when the pre-activation exceeds −b1[j].
  • Degree distribution: Degrees dj are sampled from the robust soliton distribution to shape activation patterns for peeling.The target distribution is chosen to maximize the cascade’s success probability.

C. Soliton-Data (SD)

Soliton-Data uses one auxiliary batch to set first-layer activation degrees exactly, targeting the robust soliton distribution and supporting the recovery cascade. The procedure certifies candidate samples before jointly subtracting their contributions from the residual gradient.

  • Soliton-Data: Soliton-Data uses one auxiliary batch to set each neuron’s activation degree exactly by thresholding sorted pre-activations.The threshold is placed midway between the d_j-th and (d_j + 1)-th largest auxiliary pre-activations.
  • Soliton-Data: The auxiliary batch can come from a test split, provided it is statistically similar to client data so the true-batch degrees approximate the robust soliton distribution.The auxiliary data share no samples with clients’ training data in the experiments.
  • Recovery procedure: Candidate ratios are deduplicated and certified with forward and backward passes before recovered samples are accepted.Candidates outside the data range or with near-zero bias entries are discarded for efficiency.
  • Recovery procedure: The label fit tests candidate labels using residuals that serve as recovery certificates without ground-truth data.After the joint Jacobian computation, label evaluation reduces to table lookups for neurons associated with each candidate.
  • Computational cost: Joint subtraction makes the number of backward passes equal the number of cascade iterations rather than the number of recovered samples.Memory is dominated by the residual gradient matrix of size O(FN).

A. Recovery Rate

The peeling cascade recovers samples by repeatedly finding singleton activation rows and subtracting their contributions, with recovery concentrated on atypical samples. Across benchmarks, the proposed soliton-based attacks substantially improve recovery, including near-complete passive recovery at batch sizes up to 128.

  • Recovery results: Soliton-Data achieves the highest recovery rate across the evaluated methods, followed by Soliton-Free.In passive settings, recovery remains a majority for batch sizes up to 128 on CIFAR-100, ImageNet, and several other datasets.
  • Cascade mechanism: Each activation-grid singleton row identifies a recoverable sample, and subtracting it removes one column while potentially creating new singleton rows.The grid has size N × B, and each weight-gradient column combines the samples activating its neuron.
  • Cascade mechanism: 61/256 = 0.238 of the batch was recovered in the illustrated five-iteration cascade before it stalled.The cascade removed 17, 18, 13, 10, and 3 samples across successive iterations.
  • Cascade mechanism: Recovery begins predominantly with atypical samples that activate sparse neurons and appear darker or off-distribution.Peeling concentrates on sparse bottom rows and atypical columns in the activation grid.
  • Recovery results: 94–100% passive recovery is achieved at batch sizes up to 128, while active recovery exceeds 90% at batch sizes of several hundred.The attacks recover exact samples and labels from a single FedSGD round and certify every recovery without ground-truth data.
  • Limitations and future work: Future work includes extending the cascade to FedAvg, proving guarantees for realistic data distributions, and preventing degree-1 neurons with lightweight defenses.Applicability to convolutional first layers is also identified as an open question.
Loading 2609.09659v1…