Source-linked AI summary
LDP-Fed: Federated Learning with Local Differential Privacy
Stacey Truex, Ling Liu, Ka-Ho Chow, Mehmet Emre Gursoy, Wenqi Wei
TL;DR
Federated learning must protect repeated uploads of high-dimensional model parameters, a setting existing single-value LDP protocols do not directly address. LDP-Fed combines formal local differential privacy with budget-aware perturbation and selective parameter sharing, reaching 86.85% final accuracy among privacy-preserving approaches in the reported evaluation.
Problem
Existing LDP protocols target single values, whereas federated DNN training repeatedly uploads high-dimensional, continuous parameter vectors requiring cumulative privacy accounting.
Method
LDP-Fed perturbs model updates under participant-defined local privacy settings and selectively shares updates across training rounds and parameters.
Results
86.85% final accuracy was achieved by α-CLDP-Fed, the highest among the privacy-preserving approaches in Figure 3.
Takeaways & Limitations
LDP-Fed extends local differential privacy to high-dimensional, continuous model updates while avoiding uncontrolled noise through utility-aware perturbation and selective sharing.
Takeaways & Limitations
Privacy accounting must include repeated training rounds and the multiple parameters in each model update.
Abstract
from arXiv · showhide
This paper presents LDP-Fed, a novel federated learning system with a formal privacy guarantee using local differential privacy (LDP). Existing LDP protocols are developed primarily to ensure data privacy in the collection of single numerical or categorical values, such as click count in Web access logs. However, in federated learning model parameter updates are collected iteratively from each participant and consist of high dimensional, continuous values with high precision (10s of digits after the decimal point), making existing LDP protocols inapplicable. To address this challenge in LDP-Fed, we design and develop two novel approaches. First, LDP-Fed's LDP Module provides a formal differential privacy guarantee for the repeated collection of model training parameters in the federated training of large-scale neural networks over multiple individual participants' private datasets. Second, LDP-Fed implements a suite of selection and filtering techniques for perturbing and sharing select parameter updates with the parameter server. We validate our system deployed with a condensed LDP protocol in training deep neural networks on public data. We compare this version of LDP-Fed, coined CLDP-Fed, with other state-of-the-art approaches with respect to model accuracy, privacy preservation, and system capabilities.
1 INTRODUCTION
Federated learning keeps raw data distributed, but independent ownership, legal restrictions, and business competition limit centralized data sharing. LDP-Fed addresses privacy risks in collaborative DNN training with formal, participant-specific local privacy guarantees.
- Distributed data ownership and regulations such as HIPAA, CCPA, and GDPR can prevent sensitive data from being shared with a trusted central party.
- Federated learning lets participants train locally on private data while sharing model parameters with a central parameter server.
- Shared model updates can leak private information to aggregators and participants, while final prediction outputs create another leakage point.
- Existing privacy solutions may require trusted aggregators, heavy cryptography, or focus on low-dimensional models, limiting their suitability for high-dimensional FL updates.
- LDP-Fed provides formal local differential privacy for joint DNN training and lets each participant define a local privacy setting.
2 PRELIMINARIES
The preliminaries describe DNN optimization, federated training, privacy leakage from shared updates, and local differential privacy mechanisms. They introduce CLDP as a distance-aware extension and explain that repeated training rounds require privacy accounting.
- 2.1 Deep Neural Network Training: DNN training minimizes a loss function over training examples, typically using iterative stochastic gradient descent on minibatches.
- 2.2 Federated Learning: In federated learning, participants train on local datasets and repeatedly exchange model parameters with a central aggregator.
- 2.2 Federated Learning: Federated data locality does not prevent inference attacks because shared parameter updates encode private data and can support membership inference.
- 2.3 Local Differential Privacy: Local differential privacy perturbs each value before upload, with a lower ε indicating stronger privacy.
- 2.3.1 Condensed Local Differential Privacy: CLDP additionally controls indistinguishability using a distance metric, requiring smaller α as the distance between values increases.
- 2.3.2 Privacy Accounting: Repeated queries across training rounds consume privacy budget, so composition and sampling amplification are needed to account for cumulative cost.
3 FEDERATED LEARNING WITH LDP-FED
LDP-Fed integrates local privacy perturbation and client selection into iterative federated DNN training. Its design distributes privacy budget across rounds and parameters while selectively uploading updates to make high-dimensional protection practical.
- Each participant initializes a locally configured privacy module, computes gradients on private data, perturbs them, and anonymously submits updates for probabilistic client selection.
- The parameter server initializes the model, receives k selected updates, averages them, broadcasts updated parameters, and repeats until a stopping condition is reached.
- LDP-Fed adds a Local Differential Privacy Module and a k-Client Selection Module to conventional federated learning.
- The LDP Module accepts high-dimensional model-update vectors and outputs perturbed vectors according to each participant’s privacy context.
- Because existing CLDP protocols address single-item uploads, LDP-Fed partitions privacy budget across iterations and parameters for repeated high-dimensional updates.
- Only k participants upload updates per round, and discarded updates incur no privacy cost under sampling amplification.
4 EXPERIMENTAL RESULTS
The experiments compare privacy-preserving federated learning approaches under a common privacy budget and show that selective, layer-aware updates improve accuracy. Cycle-based α-CLDP-Fed achieves the strongest reported stable performance.
- Experimental setup: The evaluation uses 50 participants, randomly selects 9 updates per round, trains for 80 rounds, and uses a convolutional DNN on FashionMNIST.FashionMNIST contains 60,000 training and 10,000 testing examples across ten fashion classes.
- Experimental setup: All private methods use a total privacy budget equivalent to α = 1.0, enabling comparison across CLDP-based settings and other privacy-preserving approaches.The comparison includes six existing federated learning scenarios plus CLDP-Basic, CLDP-Single Layer, and α-CLDP-Fed.
- Limited updates: CLDP-Basic performs worst, falling below the 10% random-guess baseline because its privacy budget is divided across all 29,034 parameters.This result motivates allocating budget to selected parameter updates instead of perturbing every parameter at every round.
- Limited updates: 84.89% final accuracy is achieved by CLDP-Single Layer, which sends updates for only one DNN layer per round and thereby increases the budget per parameter.Layers receive evenly divided rounds and privacy-budget slices in this approach.
- Limited updates: 86.85% accuracy is achieved by α-CLDP-Fed, which allocates rounds and budget proportionally to layer size and exceeds the other privacy-preserving approaches.It further improves on CLDP-Single Layer and outperforms Local Learning, DPSGD, and Hybrid in the reported comparison.
- Perturbation cycles: 86.93% average accuracy with a standard deviation of 0.12 results when α-CLDP-Fed uses five perturbation cycles, which is the default setting.Cycles allow layers to be revisited during training.
5 SYSTEM FEATURE COMPARISON
The system comparison highlights that LDP-Fed combines formal privacy protection with practical federated-learning capabilities for complex models. It avoids heavy cryptography and lets participants set local privacy levels.
- System capabilities: LDP-Fed does not require heavy cryptographic protocols, which may be unsuitable for edge devices participating in federated learning.This distinguishes its system design from approaches relying on heavy cryptographic techniques.
- System capabilities: Participants can locally define their own privacy levels through the LDP Module.The paper connects this flexibility to differing vulnerability levels across participants’ datasets and the desire for stricter guarantees for some participants.
- System capabilities: LDP-Fed provides formal protection from known privacy inference attacks while maintaining good accuracy for large, complex models.The statement summarizes the reported privacy and utility properties of the system.
6 RELATED WORK
The related-work discussion situates LDP-Fed between federated learning and privacy-preserving machine learning. Prior approaches rely on trusted aggregation, cryptography, or protocols whose guarantees and evaluations are limited for large models.
- Federated learning approaches: Some privacy-preserving federated learning systems require a trusted aggregator to add noise or provide a differentially private query interface.These designs include distributed data mining with differential privacy and teacher–student training with public unlabelled data.
- Federated learning approaches: Cryptographic protocols protect individual updates before aggregation but represent a separate privacy-preserving federated-learning approach discussed by the paper.The supplied passage introduces these protocols as related prior work.
- Privacy-preserving ML: A distributed learning system without a central trusted party offers a per-parameter differential-privacy guarantee that becomes meaningless for models with many parameters.The paper identifies high parameter dimensionality as the limiting condition for that guarantee.
- Privacy-preserving ML: Another LDP protocol supports multidimensional continuous data but was evaluated with 4 million users and fewer than 20 features for smaller-dimensional models.The comparison emphasizes the narrower model dimensionality of that evaluation relative to LDP-Fed’s target setting.
7 CONCLUSION
LDP-Fed provides formal local differential privacy for federated training while allowing participants to customize privacy budgets and controlling perturbation noise in complex model updates.
- LDP-Fed lets participants customize their LDP privacy budgets locally according to their own preferences.
- Its privacy-preserving collaborative training approach uses utility-aware perturbation to prevent uncontrolled noise from overwhelming training with large, complex parameter updates.
- The α-CLDP-Fed design extends traditional LDP theory from single categorical values to high-dimensional, continuous, large-scale model updates.
- LDP-Fed is compared analytically and empirically with state-of-the-art privacy-preserving federated learning approaches on accuracy and system features.