PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy

Xinwei Zhang, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, Jinfeng Yi

arXiv:2106.13673v1cs.LGcs.CRcs.DC

TL;DR

Client-level differential privacy in federated learning requires clipping transmitted updates before adding noise, but the optimization effects of this operation have been poorly understood. This paper analyzes clipping strategies and DP FedAvg, finding that performance and clipping bias relate to the distribution of client updates.

  • Problem

    How clipping model updates affects federated-learning optimization under differential privacy remains insufficiently understood, particularly with heterogeneous data and multiple local steps.

    Source: §1 · p. 2, §1.1 · p. 3

  • Method

    The paper analyzes model and difference clipping, studies how network, dataset, and data-distribution choices affect clipped FedAvg, and provides convergence analysis for DP FedAvg.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    Clipping-related accuracy drops track update-magnitude concentration, and clipped FedAvg can have negligible performance drops when network structure induces concentrated client updates.

    Source: §1 · p. 2, §2.2 · p. 8

  • Takeaways & Limitations

    The distribution of client updates is a key factor affecting clipped-FL performance, and clipping bias is related to that distribution.

    Source: §3.1 · p. 11, §5 · p. 13

    Direct model clipping requires a relatively large threshold and therefore larger perturbation.

    Source: §2.1 · p. 5

Abstract

from arXiv · show

Providing privacy protection has been one of the primary motivations of Federated Learning (FL). Recently, there has been a line of work on incorporating the formal privacy notion of differential privacy with FL. To guarantee the client-level differential privacy in FL algorithms, the clients' transmitted model updates have to be clipped before adding privacy noise. Such clipping operation is substantially different from its counterpart of gradient clipping in the centralized differentially private SGD and has not been well-understood. In this paper, we first empirically demonstrate that the clipped FedAvg can perform surprisingly well even with substantial data heterogeneity when training neural networks, which is partly because the clients' updates become similar for several popular deep architectures. Based on this key observation, we provide the convergence analysis of a differential private (DP) FedAvg algorithm and highlight the relationship between clipping bias and the distribution of the clients' updates. To the best of our knowledge, this is the first work that rigorously investigates theoretical and empirical issues regarding the clipping operation in FL algorithms.

1 Introduction

Client-level differential privacy in federated learning requires clipping transmitted updates before adding noise, but clipping’s effects on optimization under heterogeneous client data remain unclear. The paper studies clipping strategies and their implications for performance, convergence, and privacy guarantees.

1 Introduction

  • FL aims to reduce communication complexity and provide privacy protection, while inference attacks have shown that training can still leak local information.FedAvg reduces communication cost through periodic averaging and client sampling.Source: §1 · p. 1
  • The paper analyzes model and difference clipping, reports that difference clipping outperforms model clipping, and connects clipping-enabled FedAvg performance to network structure and client-update concentration.It also provides convergence analysis linking clipping bias to the distribution of client updates.Source: §1 · p. 2

1.1 Preliminaries & Related Work

  • Heterogeneous client data and multiple local updates complicate DP-enabled FL, while their effects on algorithm design and the privacy–optimization trade-off remain unclear.With Q > 1, data heterogeneity can degrade FedAvg even without clipping or perturbation.Source: §1.1 · pp. 3–4
  • Prior approaches clip local steps or transmitted models, while difference clipping has shown better numerical performance but lacked a convergence proof.Local-step clipping can significantly degrade performance; model clipping also reduces accuracy and has no theoretical convergence guarantee.Source: §1.1 · p. 4

2 Clipping Issues in FL

This section compares model clipping with update-difference clipping, then examines how clipping affects FedAvg’s optimization behavior across theoretical examples and neural-network experiments.

2.1 Model clipping versus Difference Clipping

  • For convex quadratics, model clipping can miss the global optimum for any fixed Q and local stepsize, while difference clipping can converge to it with suitable parameters.The difference-clipping result also gives a case where it converges to a lower-loss solution than unclipped FedAvg.Source: §2.1 · p. 5

2.2 Empirical Results

  • Compared with FedAvg, CE-FedAvg accuracy drops were 0.10% (ResNet-18) and 3.60% (AlexNet) on EMNIST, and 1.55% and 7.30% on Non-IID Cifar-10.The experiments did not add privacy perturbation.Source: §2.2 · pp. 7–8, Table 2 · p. 8
  • The experiments compared CE-FedAvg with FedAvg using AlexNet and ResNet-18 on EMNIST and Cifar-10 under IID and Non-IID client data settings.The EMNIST and Cifar-10 client sample distributions differed across the two settings.Source: §2.2 · pp. 7–8
  • Clipping’s accuracy impact varied with data distribution and model; larger losses aligned with less-concentrated update magnitudes, including for AlexNet and Cifar-10.IID data showed less impact from clipping; the update-concentration relationship is reported as a correlation.Source: §2.2 · p. 8

3 Convergence Analysis of Clipping-Enabled FedAvg

The section analyzes DP-FedAvg, which adds noise to locally clipped model differences, and relates its convergence to clipping bias and client-update distributions. It also gives a client-level privacy guarantee and a convergence-rate comparison with DP-SGD.

3 Convergence Analysis of Clipping-Enabled FedAvg

  • DP-FedAvg adds random perturbation to locally clipped model differences; the analysis assumes the attacker observes only the aggregated update, as with secure aggregation or a secure client-server uplink.The section frames this analysis as examining the trade-off between FedAvg convergence and differential-privacy guarantees.Source: §3 · p. 9

3.1 Convergence Analysis

  • The DP-FedAvg convergence bound adds privacy-noise and clipping-bias terms to standard FedAvg terms, recovering standard FedAvg bounds up to a constant when clipping bias and noise vanish.With the stated learning-rate choices, the standard terms yield a convergence rate of O(1/√PQT + 1/T).Source: §3.1 · p. 11
  • Clipping bias includes a local-update-variance component that can be small when stochastic local updates have similar variance or norm magnitudes, and vanishes when σl = 0.The analysis identifies a separate cross-client component: it is small when expected local-update norms are similar, but can remain even with full-batch local updates under heterogeneous data.Source: §3.1 · p. 11

3.2 Differential Privacy Guarantee

  • Theorem 3.2 gives DP-FedAvg a client-level (ϵ, δ)-differential-privacy guarantee under its stated conditions on T, client sampling, and noise.The privacy guarantee is characterized using standard Gaussian-mechanism privacy theorems.Source: §3.2 · p. 11
  • DP-FedAvg’s big-O convergence rate matches DP-SGD’s in d, ϵ, and N, under the corollary’s clipping-threshold and privacy conditions.The corollary sets c = ηlQc′ with c′ ≥ G and uses the noise level specified by Theorem 3.2.Source: §3.2 · p. 11

4 Numerical Experiments

The experiments compare FedAvg, CE-FedAvg, and DP-FedAvg across models and datasets. The reported observations link model structure to clipping performance and model size or depth to privacy-noise sensitivity.

4 Numerical Experiments

  • Experiments compare FedAvg, CE-FedAvg, and DP-FedAvg on non-IID EMNIST and IID Cifar-10 using 1,920 clients, 80 participants per round, 32 local iterations, and mini-batches of 64.Across all experiments, the privacy budget δ is fixed at 10−5.Source: §4 · p. 12
  • Structured models can concentrate FedAvg update differences, yielding better clipping performance.The authors relate this observation to clipping-related terms in Theorem 3.1.Source: §4 · p. 12
  • Larger or deeper models may be more sensitive to privacy noise because the theorem’s error term depends on model dimension and the square of the Lipschitz constant.The passage notes that the Lipschitz constant increases exponentially with the number of layers.Source: §4 · p. 12
  • The authors conjecture that good DP-FedAvg performance requires networks structured enough for clipping but not too large or deep.This is presented as a conjecture, not a demonstrated rule.Source: §4 · p. 12
  • Tables 3 and 4 report accuracy drops for FedAvg versus clipped FedAvg and clipped FedAvg versus DP-FedAvg.The clipping threshold is 0.5 of average update magnitude; the listed privacy budgets vary by model.Source: Table 3 · p. 12, Table 4 · p. 12

5 Conclusion

The conclusion frames clipping in FL as a convergence–privacy trade-off shaped by the distribution of client updates.

5 Conclusion

  • The paper combines empirical and theoretical study of FL clipping to seek a trade-off between convergence and differential privacy guarantees.Its numerical results corroborate the theory and identify client-update distribution as a key performance factor.Source: §5 · p. 13
  • Numerical results corroborate the paper’s theory about clipping-enabled FL.The conclusion does not specify particular metrics or values in this passage.Source: §5 · p. 13
  • The distribution of client updates is a key factor affecting clipping-enabled FL performance.Source: §5 · p. 13

A.1 Proof of Theorem 3.1

The proof analyzes clipping bias through a first-order term and bounds additional terms before combining the estimates across rounds.

A.1 Proof of Theorem 3.1

  • The proof treats the first two terms on the right-hand side of its decomposition as bias caused by clipping.It analyzes the first-order term to study that bias.Source: §A.1 · pp. 17–18
  • The proof bounds intermediate terms using Jensen’s inequality, smoothness, and stated assumptions on gradient quantities.The cited derivation also invokes a bound identified as [32, Lemma 3].Source: §A.1 · pp. 19–20
  • After combining intermediate inequalities, the proof sums over communication rounds and rearranges to obtain its bound.The supplied passage describes this final proof step without stating the resulting bound.Source: §A.1 · pp. 21–22

A.2 Additional Numerical Experiments

The appendix section introduces additional numerical results.

A.2 Additional Numerical Experiments

  • The section presents additional numerical results.Source: §A.2 · p. 22

A.2.1 Update Distributions

Across the plotted models and datasets, update-magnitude distributions are more concentrated for IID than corresponding Non-IID cases, and for EMNIST than Cifar-10.

A.2.1 Update Distributions

  • Update-magnitude distributions are more concentrated in IID than in corresponding Non-IID cases across models and datasets.Source: §A.2.1 · p. 22
  • For the same model, update-magnitude distributions are more concentrated on EMNIST than on Cifar-10.Source: §A.2.1 · p. 22

Related papers

Loading 2106.13673v1…