Source-linked AI summary
LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy
Lichao Sun, Jianwei Qian, Xun Chen
TL;DR
LDP-FL addresses the privacy and utility limitations of applying local differential privacy to federated deep learning. It combines adaptive-range perturbation with parameter shuffling, and empirical studies report superior performance to related methods while preserving a formal privacy guarantee.
Problem
Existing federated-learning privacy methods can leak information through model weights, while prior local-differential-privacy approaches face a privacy–performance trade-off in complex deep-learning settings.
Method
LDP-FL uses adaptive-range data perturbation and parameter shuffling to improve the privacy–utility trade-off and mitigate privacy degradation from high dimensionality and repeated updates.
Results
Empirical studies on MNIST, Fashion-MNIST, and CIFAR-10 report superior performance to previous related works, while the proposed mechanism satisfies ϵ-LDP under a bounding range.
Takeaways & Limitations
LDP-FL provides a practical local-differential-privacy solution for federated learning with deep neural networks.
Takeaways & Limitations
With fixed range settings on CIFAR-10 at ϵ = 10, model accuracy stays around 20% across tested r values, suggesting fixed ranges may fail for deeper networks.
Abstract
from arXiv · showhide
Train machine learning models on sensitive user data has raised increasing privacy concerns in many areas. Federated learning is a popular approach for privacy protection that collects the local gradient information instead of real data. One way to achieve a strict privacy guarantee is to apply local differential privacy into federated learning. However, previous works do not give a practical solution due to three issues. First, the noisy data is close to its original value with high probability, increasing the risk of information exposure. Second, a large variance is introduced to the estimated average, causing poor accuracy. Last, the privacy budget explodes due to the high dimensionality of weights in deep learning models. In this paper, we proposed a novel design of local differential privacy mechanism for federated learning to address the abovementioned issues. It is capable of making the data more distinct from its original value and introducing lower variance. Moreover, the proposed mechanism bypasses the curse of dimensionality by splitting and shuffling model updates. A series of empirical evaluations on three commonly used datasets, MNIST, Fashion-MNIST and CIFAR-10, demonstrate that our solution can not only achieve superior deep learning performance but also provide a strong privacy guarantee at the same time.
1 Introduction
The paper targets practical local differential privacy for federated deep learning, where model weights can leak sensitive data and existing methods face accuracy and privacy-budget challenges. LDP-FL combines adaptive perturbation ranges with parameter shuffling and reports strong privacy-utility results on three image datasets.
- Federated learning shares model weights instead of sensitive data, but those weights can still leak privacy and enable recovery of original data.
- Existing LDP approaches struggle with the trade-off between privacy budget and performance in deep learning on complex datasets.
- LDP-FL adapts perturbation ranges to different DNN layers and shuffles client parameters to mitigate dimensionality- and iteration-related privacy degradation.
2 Preliminary
Federated learning trains models by exchanging client-side model updates rather than private data. Local differential privacy formalizes protection for each client input, with stronger guarantees corresponding to smaller privacy budgets.
- Each client updates a model locally, sends gradients or weights to the cloud, and receives an updated central model for another training round.
- An ϵ-LDP mechanism provides a privacy guarantee for every pair of possible inputs and any mechanism output.
- A smaller privacy budget ϵ indicates a stronger privacy guarantee.
3 Overview of LDP-FL
LDP-FL combines federated local training with local perturbation designed for model-weight ranges. Its mechanism uses adaptive ranges and supports privacy preservation for client updates before aggregation.
- LDP-FL has cloud and local update stages in which selected clients optimize models from cloud-provided weights.
- Clients are assumed to remain anonymous to the cloud, unlike approaches that assume the aggregator knows user identities.
- The mechanism generalizes LDP perturbation by modeling each weight within a range centered at c with radius r.
- The algorithm randomizes each model-weight dimension and returns a perturbed tuple W∗ for transmission.
1 CloudUpdate
The cloud initializes the model and layer ranges, repeatedly selects clients, aggregates their returned updates, and refreshes the model and range information for subsequent rounds.
- The algorithm initializes model weights W0 and layer ranges (C0, R0), then sends them to clients before iterative training begins.
- For each round l = 1, 2, . . ., the cloud coordinates client selection and receives updates through SendToCloud.
- The cloud determines each weight identifier and computes the mean of local models when updating model weights.
- After processing each layer, the cloud updates C and R from Wl and sends the revised model and ranges to clients.
15 LocalUpdate(Wl, kl, Cl, Rl)
LDP-FL combines adaptive, layer-wise weight perturbation with parameter shuffling to protect federated updates. Randomized upload timing and anonymous splitting make updates harder to link while retaining practical utility.
- Local perturbation: Each client perturbs local weights using the proposed LDP mechanism with its layer-specific range and privacy budget.Algorithm 2 takes original local weights, layer ranges C_l and R_l, and privacy budget ϵ as inputs.
- Cloud update: The cloud aggregates the shuffled, perturbed updates and redistributes updated weights and range parameters for subsequent local optimization.Federated learning repeatedly sends client updates to the cloud, which aggregates them and updates the central model.
- Adaptive range setting: Adaptive range settings assign layer-specific weight bounds to reduce utility loss when weight ranges differ across deep neural network layers.The ranges C and R are calculated layer-wise, and updated ranges are distributed to clients.
- Parameter shuffling: Parameter shuffling splits weight updates and mixes them among clients so the cloud cannot associate multiple updates with one client.The design targets linkage through repeated uploads and side-channel timing information.
- Randomized timing: Random delays make parameter response times uniformly distributed and preserve client anonymity while limiting additional delay to T.The design accommodates heterogeneous training and communication times, with iteration delay determined by the slowest client.
4 Privacy and Utility Analysis
The analysis establishes privacy and utility properties for the proposed perturbation and parameter-shuffling mechanisms. It shows unbiased average estimation, bounded variance behavior, and protection against dimensionality-driven privacy costs.
- Local differential privacy: The proposed mechanism satisfies ϵ-LDP for any weight within a specified bounding range [c − r, c + r].Here c is the center and r is the radius of the weight range.
- Utility analysis: The proposed mechanism introduces zero bias when estimating average weights.Lemma 1 states E[M(w)] = w̄.
- Utility analysis: The estimated average weight has lower and upper variance bounds, with larger client counts enabling low variance even for wider perturbation ranges.The analysis connects the variance of estimated averages to the number of clients, range radius, and privacy budget.
- Adaptive range setting: Adaptive per-weight ranges preserve LDP and zero bias while allowing variance analysis when clients choose different radii.Clients may select different centers and radii for weights and communication rounds.
- Parameter shuffling: Parameter shuffling reduces the dimensionality-related privacy cost from Tdϵ without shuffling and dϵ with model shuffling to per-weight ϵ-LDP protection.Anonymous splitting prevents the cloud from linking weights from the same client across parameters or iterations.
5 Experiments
Experiments evaluate LDP-FL across MNIST, FMNIST, and CIFAR-10, including parameter sensitivity, privacy-budget effects, and adaptive-versus-fixed weight ranges. Performance generally improves with more clients, larger client fractions, and more communication, while adaptive ranges are especially beneficial for CIFAR-10.
- Experimental Setup: LDP-FL is evaluated on MNIST and FMNIST with two-layer CNNs and on CIFAR-10 with a redesigned small VGG network.Experiments use ten independent runs to average perturbation randomness.
- Parameter Analysis: Increasing the client fraction toward 1 brings performance close to noise-free results, while too-small fractions significantly affect FMNIST and CIFAR-10.The fraction has little effect on MNIST when it is too small.
- Parameter Analysis: More communication rounds improve training across datasets, with CIFAR-10 requiring more rounds because its data and task are more complex.The experiments analyze communication rounds alongside the fraction of clients.
- Performance Analysis and Comparison: 96.24% accuracy with ϵ = 1 on MNIST, 86.26% with ϵ = 4 on FMNIST, and 61.46% with ϵ = 10 on CIFAR-10 are reported for selected client counts.The corresponding client counts are n = 100, n = 200, and n = 500, respectively.
- Privacy-Budget Analysis: LDP-FL maintains high accuracy for ϵ > 0.3 on MNIST, ϵ > 2 on FMNIST, and ϵ > 5 on CIFAR-10, while accuracy drops below lower thresholds.The paper attributes the greater privacy cost needed for complex tasks to larger networks and wider weight ranges.
6 Related Work
Differential privacy provides a formal privacy framework, while local differential privacy perturbs raw data before it reaches the curator. Existing LDP-FL approaches remain impractical for deep learning because of limited task scope, strong assumptions, or poor performance under small privacy budgets.
- Differential privacy offers a mathematically provable framework for designing and evaluating privacy-protection schemes.
- Local differential privacy perturbs raw data before it is sent to the cloud data curator.
- Existing LDP-FL methods are difficult to apply practically to deep learning, especially on complex datasets and under limited privacy budgets.
- Bhowmick et al. used ϵ = 500 for MNIST and ϵ = 5000 for CIFAR-10 and assumed attackers had little prior knowledge about user data.
7 Conclusion
The paper presents an LDP mechanism for federated learning with DNNs that combines adaptive perturbation ranges with parameter shuffling. Empirical studies report superior performance to previous related work on the same image-classification tasks.
- LDP-FL uses generalized perturbation for arbitrary input ranges and adaptive range settings to improve the privacy/utility trade-off.
- Parameter shuffling mitigates privacy degradation caused by high data dimension and many query iterations.
- LDP-FL achieves superior performance to previous related works on the same image-classification tasks in empirical studies.
A Proof
The proof establishes privacy for the proposed mechanism over an arbitrary bounded weight range, then characterizes unbiasedness and variance-related bounds for estimated weights. These results provide the mechanism’s formal privacy and estimation guarantees.
- For any w ∈ [c − r, c + r], the proposed mechanism M satisfies ϵ-LDP with respect to that range.Here c is the range center and r is its radius.
- Algorithm 2 introduces zero bias when estimating average weights, satisfying E[M(w)] = w̄.
- The proof analyzes the variance of each perturbed weight and derives lower and upper bounds for the estimated average weight.