Provable Defense against Privacy Leakage in Federated Learning from Representation Perspective

Jingwei Sun; Ang Li; Binghui Wang; Huanrui Yang; Hai Li; Yiran Chen

Source-linked AI summary

Provable Defense against Privacy Leakage in Federated Learning from Representation Perspective

Jingwei Sun, Ang Li, Binghui Wang, Huanrui Yang, Hai Li, Yiran Chen

arXiv:2012.06043v1Dec 8, 2020cs.LGcs.AIcs.CV

TL;DR

Federated learning limits direct data sharing but remains vulnerable because model updates can leak private information, while existing defenses may impose overhead or accuracy costs. This paper identifies data-representation leakage as the essential cause, perturbs representations to degrade reconstructions while preserving FL performance, and derives robustness and convergence guarantees. On MNIST and CIFAR10 against DLG and GS attacks, the defense increased reconstruction MSE by more than 160× without sacrificing accuracy compared with baseline defenses.

Problem
Federated learning remains vulnerable to inference attacks through shared model updates, while existing defenses can incur computational overhead or accuracy loss and the leakage cause is insufficiently explained.
Method
The paper analyzes representation leakage and learns perturbations that preserve true representations for FL while making reconstructed data dissimilar to raw data.
Results
More than 160× higher reconstruction mean squared error was achieved for both DLG and GS attacks without sacrificing accuracy on MNIST and CIFAR10, compared with baseline defenses.
Takeaways & Limitations
Representation perturbation provides a targeted defense that significantly improves FL privacy while maintaining accuracy, with certified robustness and FedAvg convergence guarantees.
Takeaways & Limitations
Further work is needed to study other p-norm and q-norm choices and extend representation-leakage analysis to convolutional layers.

Abstract

from arXiv · show

Federated learning (FL) is a popular distributed learning framework that can reduce privacy risks by not explicitly sharing private data. However, recent works demonstrated that sharing model updates makes FL vulnerable to inference attacks. In this work, we show our key observation that the data representation leakage from gradients is the essential cause of privacy leakage in FL. We also provide an analysis of this observation to explain how the data presentation is leaked. Based on this observation, we propose a defense against model inversion attack in FL. The key idea of our defense is learning to perturb data representation such that the quality of the reconstructed data is severely degraded, while FL performance is maintained. In addition, we derive certified robustness guarantee to FL and convergence guarantee to FedAvg, after applying our defense. To evaluate our defense, we conduct experiments on MNIST and CIFAR10 for defending against the DLG attack and GS attack. Without sacrificing accuracy, the results demonstrate that our proposed defense can increase the mean squared error between the reconstructed data and the raw data by as much as more than 160X for both DLG attack and GS attack, compared with baseline defense methods. The privacy of the FL system is significantly improved.

1 Introduction

FL avoids transferring local data but remains vulnerable because shared updates leak private information. The paper identifies representation leakage as the essential cause and proposes a defense that preserves accuracy while degrading reconstructions.

Motivation: Shared model updates expose FL to property-inference and model-inversion attacks despite keeping local data on devices.Existing defenses based on differential privacy, secure multi-party computation, and data compression can incur computational overhead or accuracy loss.
Representation leakage: Class-wise data representations are embedded in local model updates and can be inferred to reconstruct training data.The paper analyzes how representations are embedded, particularly in fully connected layers, and reports inferred-to-real representation correlation as high as 0.99 during local training.
Defense: The defense perturbs data representations to keep them close to true representations while making reconstructions dissimilar to raw data.This design targets privacy leakage while maintaining federated-learning performance.
Theory: The paper derives certified robustness for FL and convergence guarantees for FedAvg after applying the defense.These guarantees accompany the representation-perturbation defense.
Evaluation: More than 160× higher mean squared error was achieved for both DLG and GS attacks without sacrificing accuracy, compared with baseline defenses.Experiments were conducted on MNIST and CIFAR10, and the authors report significantly improved FL privacy.
Contributions: The work explicitly identifies representations embedded in model updates as the essential source of private-information leakage and develops reconstruction and defense algorithms.The contribution is framed as an explanation of leakage followed by a targeted defense.

2 Related work

Related work covers privacy leakage attacks and privacy-preserving distributed-learning methods. The paper situates its approach among reconstruction, inference, differential-privacy, secure-computation, and compression techniques.

Privacy leakage: Distributed-learning attacks include data reconstruction, class-representative inference, membership inference, and attribute inference.Data reconstruction evaluates recovered samples by their similarity to original data; DLG reconstructs samples by optimizing inputs to match client gradients.
Privacy leakage: Class-representative attacks can use GANs to construct representatives, but one cited technique is evaluated only when same-class samples are virtually similar.The examples include handwritten digits and faces.
Privacy preservation: Existing privacy-preserving distributed-learning methods are categorized as differential privacy, secure multi-party computation, and data compression.The cited differential-privacy approaches include aggregating locally trained models or using sparse vectors.
Privacy preservation: Differential-privacy methods may require many participants to converge and achieve a desirable privacy-performance tradeoff, while MPC uses distributed private computation.A cited MPC example uses two non-colluding servers to train a global model.

3 Essential Cause of Privacy Leakage in FL

Privacy leakage in FL is essentially caused by class-wise data representations embedded in model updates. In non-IID settings, these representations are less entangled across classes, enabling inference and reconstruction from gradients.

Data representations embedded in communicated model updates are identified as the essential cause of privacy leakage in FL.
For a fully connected layer, each sample’s gradient is the product of a loss-gradient column vector and the transposed data representation.
Class-wise partitioning causes representations from different classes to occupy different gradient rows, while many classes can entangle representations in centralized batches.
Small local datasets, few batches, and few local epochs can leave class representations less entangled in practical FL, despite greater entanglement across many batches.
The inference algorithm identifies training classes from the last-layer gradients and iteratively infers corresponding representations through earlier fully connected layers.
In CIFAR10 non-IID experiments, representation correlation reached 0.99 and remained above 0.8 in almost all settings, indicating substantial leakage.The experiment used 100 devices, with 10 participating per round; each device held two classes and 20 samples per class.
Using inferred representations reconstructed inputs as effectively as whole-model gradients, whereas convolutional-layer gradients alone could not successfully reconstruct them.

4 Defense Design

The defense perturbs data representations to make reconstructed inputs dissimilar while keeping perturbed representations close to clean ones. It derives a perturbation solution and certified robustness guarantee under an invertibility assumption.

Defense Formulation: The defense maximizes distance between raw and reconstructed inputs while bounding the distance between clean and perturbed representations.These objectives respectively reduce privacy leakage and preserve FL performance.
Defense Formulation: The formulation assumes the feature extractor f has an inverse on clean and perturbed representations within the perturbation bound.This assumption supports the inverse-function-based derivation.
Defense Solution: With p = 2 and q = 0, the method maximizes MSE and selects the ϵ largest values of ||r_i(∇_Xf(r_i))^-1||_2.The q = 0 choice yields an analytical solution and a relatively sparse perturbed representation that improves communication efficiency.
Certified Robustness: The paper derives a certified robustness guarantee defined as the certified minimum Lp distance between raw and reconstructed inputs.A larger defense bound indicates a more effective defense.

5 Convergence Guarantee

The paper analyzes FedAvg with the proposed defense under smoothness, strong-convexity, stochastic-gradient, and bounded-gradient assumptions. It presents a convergence theorem for the defended algorithm using specified learning-rate parameters.

Convergence Guarantee: The section derives a convergence guarantee for FedAvg when the proposed defense is applied.The analysis first describes defended FedAvg and then states its convergence theorem.
FedAvg with Defense: The defended FedAvg process repeatedly broadcasts the global model, performs I local updates on selected devices, and averages their local models.Devices are sampled with replacement according to probabilities p_1, ..., p_N.
Assumptions: The analysis assumes L-smooth and µ-strongly convex local objectives, bounded stochastic-gradient variance, bounded expected gradient norms, and bounded output-layer gradient norms.These include Assumptions 2–6 used for the convergence result.
Convergence Theorem: The convergence theorem chooses κ = L/µ, γ = max{8κ, I}, and learning rate η_t = 2/[µ(γ+t)].Under the stated assumptions and parameter choices, the theorem applies to FedAvg with the defense.

6 Experiments

Experiments evaluate the defense on non-IID MNIST and CIFAR10 against DLG and GS attacks using accuracy and reconstruction MSE. The defense maintains utility while substantially degrading reconstruction quality and exhibits good defended FedAvg convergence.

Experimental Setup: The evaluation compares the proposed defense with gradient compression, DP-Gaussian, and DP-Laplace against DLG and GS model inversion attacks.DLG optimizes Euclidean gradient distance, whereas GS optimizes cosine similarity.
Experimental Setup: Experiments use non-IID MNIST and CIFAR10 distributed across 100 devices, with each device holding two random classes and 100 samples per class.Default training uses 1000 communication rounds for CIFAR10 and 200 for MNIST.
Evaluation Metrics: Accuracy measures FL utility, while MSE between reconstructed and raw images measures privacy leakage; higher MSE indicates stronger reconstruction degradation.The experiments seek high accuracy together with high MSE.
Defense Results: Utility-Privacy Tradeoff: At the human-unrecognizable reconstruction threshold, the proposed method shows no accuracy drop, while baselines lose up to 6% under DLG and 9% under GS.The defense maintains accuracy until MSE reaches 0.8, whereas baselines decline at smaller MSE values.
Defense Results: Utility-Privacy Tradeoff: 160x MSE improvement is achieved over baseline defenses without sacrificing accuracy across the evaluated DLG and GS settings.The result is reported for both attack methods and the evaluated datasets.
Defense Results: Visual Comparison: With GS on a CIFAR10 sample, the proposed defense makes reconstructions unrecognizable after pruning 50%–60% of fully connected-layer parameters, while GC remains recognizable after pruning 80% of the whole model.Human unrecognizability is treated as a meaningful, but not ultimate, privacy indicator.
Convergence Results: Loss-versus-round results show that LR+FedAvg with the defense converges well, supporting the convergence analysis.The experiment uses non-IID MNIST with 100 devices and varied local epochs and sampled-device counts.

7 Conclusions and Future Work

The paper identifies data-representation leakage from gradients as the essential cause of privacy leakage in FL and proposes a defense based on perturbing those representations. It also reports theoretical guarantees, empirical privacy improvements, and future extensions to other norms and layers.

Data-representation leakage from gradients is identified as the essential cause of privacy leakage in FL.
The defense perturbs data representations to degrade reconstructed-data quality while maintaining FL performance.
The paper derives certified robustness for FL and convergence guarantees for FedAvg under the defense.
Future work includes studying alternative p-norms and q-norms and extending leakage analysis to convolutional layers.

A Method of Inferring Data Representations

The inference method exploits class-wise structure in local updates, first identifying training classes from large last-layer row norms and then estimating their representations. These estimates are propagated backward through fully connected layers.

The method uses low entanglement of class-specific gradients to identify rows in local updates associated with each training class.
The method approximates batch gradients using similar bias derivatives and representation transposes across samples in a class batch.
For the last classification layer, the ground-truth class creates a unique peak in the loss-gradient bias vector, producing a larger corresponding weight-gradient row norm.
A malicious server computes last-layer row norms and selects significantly larger rows to infer the device’s training classes.
For each inferred class, the representation is approximated by scaling the corresponding weight-gradient row, with the scale determined by local training steps.

A.2 Inferring features in previous layers

Previous-layer representations are inferred by exploiting shared structure and sparsity across nonlinear transformations. The algorithm selects large-magnitude features and averages corresponding rows of the layer’s weight updates, enabling backward reconstruction across fully connected layers.

Figure 8 depicts the inference process used to recover representations across layers.
Despite nonlinear transformation, adjacent representations share similar structure and sparsity, enabling gradient-based feature selection.
The method selects M largest-magnitude elements in the later representation to identify unique gradient coordinates.
It estimates the preceding representation by fetching and averaging weight-update rows indexed by the selected coordinates.
The inference procedures are presented in Algorithms 3 and 4, covering last-layer and previous fully connected layers.
Combining previous-layer inference with last-layer estimates allows the server to recover class-wise representations across fully connected layers by backpropagation.

B Experiment Setup

The experiments use specified convolutional and fully connected architectures for representation inference and attack evaluation. DLG and GS experiments use LeNet and ConvNet models, respectively, with the listed optimization setup for reconstruction experiments.

Representation-inference experiments use Conv3-6→Maxpool→Conv6-16→Maxpool→FC–120→FC–84→FC–10.Kernel size is 5 for convolutional layers and 2 for max-pooling layers.
Representation-leakage experiments use Conv3-12→FC–10 with an L-BFGS reconstruction optimizer run for 300 iterations.
The DLG attack uses LeNet, while the GS attack uses ConvNet.
Table 3 presents the model architectures used for the DLG and GS attack experiments.

C Proof of Theorem 1

The proof uses a sub-multiplicative norm inequality to derive a lower bound on the distance between X and X′.

For a sub-multiplicative norm, ||AB||p ≤ ||A||p||B||p.
Under Assumption 1 and Lemma 1, the distance between X and X′ is expressed before establishing its lower bound.
The proof applies ||C−1D||p ≥ ||D||p/||C||p to lower-bound ||X−X′||p.

D Proof of Theorem 2

The proof establishes convergence for FedAvg with the defense by deriving new gradient and variance bounds, then adapting an existing convergence theorem under stated assumptions and parameter choices.

Proof overview: The proof derives post-defense bounds analogous to Assumptions 4 and 5, then adapts Theorem 2 from using those bounds.
Gradient bounds: The defense perturbs the input representation in the selected layer before analyzing the distance between perturbed and raw gradients.
Gradient bounds: Because the defense applies only to the s-th layer, the whole-model perturbed-gradient distance is treated as the distance in that layer.
Stochastic-gradient bounds: The proof bounds stochastic-gradient variance using the norm triangle inequality and the previously established bounds.
FedAvg convergence: The analysis defines Γ from the minima of F and Fk and assumes each device performs I local updates over T total iterations.
FedAvg convergence: With Assumptions 2–6 and specified parameter choices, FedAvg using the defense has a convergence guarantee.The choices include κ = L µ, γ = max{8κ, I}, and ηt = 2 µ(γ+t).