Source-linked AI summary
Federated Learning with Local Differential Privacy: Trade-offs between Privacy, Utility, and Communication
Muah Kim, Onur Günlü, Rafael F. Schaefer
TL;DR
Federated learning decentralizes training but shared weight updates can still reveal sensitive user information. This paper applies Gaussian mechanisms with tighter privacy accounting to FedSGD and derives utility and transmission bounds for heterogeneous users. The bounds yield significantly larger utility and smaller transmission rates than existing privacy accounting methods for the same privacy constraint.
Problem
Shared weight updates in federated learning can leak sensitive user information despite decentralized data storage, motivating local differential privacy.
Method
The paper analyzes Gaussian LDP for FedSGD using tighter privacy accounting, variable query sensitivity, heterogeneous-user utility bounds, and differential-entropy transmission metrics.
Results
The proposed bounds provide significantly larger utility and smaller transmission rates than existing privacy accounting methods under the same privacy constraint.
Takeaways & Limitations
Stronger privacy requires greater noise, which lowers the utility bound and raises the transmission-rate bound in the analyzed FedSGD model.
Takeaways & Limitations
The analysis assumes sufficiently large datasets so the difference between true and empirical loss is negligible, and its comparison includes a condition requiring σ_k ≥ 1.
Abstract
from arXiv · showhide
Federated learning (FL) allows to train a massive amount of data privately due to its decentralized structure. Stochastic gradient descent (SGD) is commonly used for FL due to its good empirical performance, but sensitive user information can still be inferred from weight updates shared during FL iterations. We consider Gaussian mechanisms to preserve local differential privacy (LDP) of user data in the FL model with SGD. The trade-offs between user privacy, global utility, and transmission rate are proved by defining appropriate metrics for FL with LDP. Compared to existing results, the query sensitivity used in LDP is defined as a variable and a tighter privacy accounting method is applied. The proposed utility bound allows heterogeneous parameters over all users. Our bounds characterize how much utility decreases and transmission rate increases if a stronger privacy regime is targeted. Furthermore, given a target privacy level, our results guarantee a significantly larger utility and a smaller transmission rate as compared to existing privacy accounting methods.
1. INTRODUCTION
Federated learning protects raw data through decentralization, but shared weight updates can still leak user information, motivating local differential privacy. This paper develops tighter privacy accounting and utility/transmission bounds for Gaussian LDP in FedSGD.
- Federated learning avoids collecting user data at an aggregator, but information can still be inferred from shared local-network updates.
- The paper analyzes FedSGD privacy using an enhanced moments accountant and defines a utility metric with variable query sensitivity.
- The utility bound accommodates heterogeneous dataset sizes, sampling probabilities, and target privacy levels across users.
- The bounds characterize privacy–utility–transmission trade-offs and show gains over existing methods in noise power, utility, and transmission rate.
- Transmission rate is measured through the differential entropy of noisy gradients for lossless communication.
2. SYSTEM MODEL
The system uses FedSGD with a central server and K users that sample local data, compute clipped gradients, add Gaussian noise, and send updates for aggregation. LDP protects each user’s dataset across repeated rounds of weight updates.
- 2.1. Federated SGD (FedSGD): At each of T time steps, the server distributes aggregated weights, users initialize matching networks, sample local datasets, and compute gradients.
- 2.1. Federated SGD (FedSGD): Users may have distinct dataset sizes and sampling probabilities, while local gradients are estimated from sampled data and assumed close to true loss gradients for sufficiently large datasets.
- 2.1. Federated SGD (FedSGD): Each user clips its local gradient using threshold C, and the server aggregates the clipped updates to optimize the global loss.
- 2.1. Federated SGD (FedSGD): The weighted average of local gradients produces the same weight update as centralized optimization on the combined sampled data.
- 2.2. Local differential privacy (LDP): Gaussian LDP adds noise with per-dimension variance proportional to C^2σ^2 before noisy gradients replace clipped gradients in every round.
3. TRADE-OFFS BETWEEN PRIVACY, UTILITY, AND TRANSMISSION RATE
The paper characterizes how Gaussian LDP noise affects utility and transmission rate over repeated FedSGD updates, including heterogeneous users and varying query sensitivity. Tighter privacy accounting can reduce the required noise and improve both utility and transmission rate under the same privacy target.
- Metrics: The utility metric is the multiplicative inverse of the convergence rate, and transmission rate is the differential entropy of each user’s noisy gradient.The transmission model assumes lossless communication, although quantization could further reduce the rate.
- Theoretical analysis: Theorem 1 bounds Gaussian noise variance, utility, and transmission rate after T rounds of FedSGD under per-user LDP constraints.The mechanism uses noise variance C^2σ_k^2 per dimension, while utility is analyzed through convergence and transmission rate through differential entropy.
- Theoretical analysis: The utility bound allows heterogeneous dataset sizes, sampling probabilities, and target privacy levels across users.The bound is derived for a smooth, strongly convex loss with an adaptive learning rate depending on the clipping threshold.
- Trade-offs: Increasing noise variance for stronger privacy lowers the utility lower bound and raises the transmission-rate upper bound, while utility can remain finite as T tends to infinity.When the required noise variance grows linearly with T, the utility-bound denominator also grows linearly, so convergence to the minimum loss is not guaranteed.
- Comparisons: Across the evaluated iteration counts, the proposed method requires the smallest noise variance for the same ϵ_k, followed by MA, AC2, and AC1.The required noise variance increases with T because repeated updates deteriorate individual privacy.
- Comparisons: For ϵ_k = 0.3 after T = 7×10^4 iterations, the proposed bound gives utility 25.83 and transmission rate 5.81×10^3 bits/symbol, outperforming MA, AC1, and AC2 on both reported bounds.The corresponding utility values are 10.79, 0.22, and 1.40, while transmission rates are 6.44, 9.26, and 7.91×10^3 bits/symbol for MA, AC1, and AC2, respectively.
4. CONCLUSION
The paper proves privacy–utility–transmission-rate trade-offs for FedSGD using Gaussian LDP and tight composition. Its bounds support heterogeneous users and improve noise, utility, and transmission-rate guarantees over existing composition methods.
- The paper proves trade-offs between privacy, utility, and transmission rate for FedSGD with a Gaussian LDP mechanism.
- The proposed noise variance bound guarantees a given LDP level after multiple weight-update rounds through a tight composition theorem.
- The utility bound supports distinct user parameters and clipped, noisy gradients.
- The proposed bounds yield significantly smaller required noise, larger utility, and smaller transmission rates than existing privacy composition methods.
A. PROOF OF THEOREM 1
The proof analyzes Gaussian-mechanism privacy for sampled, clipped gradients, composes privacy across iterations, and derives utility and transmission-rate bounds from the resulting noise variance.
- Clipping gradient norms at C limits query sensitivity to 2C for neighboring datasets.
- The privacy cost is computed for sampled datasets, composed linearly across T iterations, and converted from RDP to (ϵ(α, δ), δ)-DP.
- The proof reduces the neighboring-gradient analysis to one dimension and evaluates the Gaussian mechanism using Rényi divergence.
- Under smoothness, strong convexity, and a specified learning rate, the proof bounds the expected mean-square error and obtains a convergence-rate lower bound.
- The transmission-rate bound uses the differential entropy of clipped gradients plus Gaussian noise, with covariance and noise-entropy bounds combined to prove the result.