PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentially Private Federated Learning with Local Regularization and Sparsification

Anda Cheng, Peisong Wang, Xi Sheryl Zhang, Jian Cheng

arXiv:2203.03106v3cs.LGcs.CRcs.CV

TL;DR

Existing user-level DP methods can severely reduce federated-learning accuracy. This paper proposes Bounded Local Update Regularization and Local Update Sparsification to reduce local-update norms before DP operations. The authors report improved utility over state-of-the-art methods, with theoretical convergence analysis and privacy guarantees.

  • Problem

    Existing methods for user-level DP in federated learning can severely harm model utility through clipping and noise perturbation of local updates.

    Source: §3.3 · p. 3

  • Method

    BLUR regularizes local objectives to bound update norms, and LUS removes low-impact parameter updates before clipping to further reduce their magnitude.

    Source: §1 · p. 1, §4.1 · p. 4, §4.2 · pp. 4–5

  • Results

    On EMNIST, BLUR and LUS improve DP-FedAvg test accuracy by 3% ∼4% for CNN-2-Layers and 4% ∼5% for ResNet-18.

    Source: §7 · p. 7

  • Takeaways & Limitations

    The framework combines improved model utility with theoretical convergence analysis and rigorous privacy guarantees for federated learning under user-level DP.

    Source: §8 · p. 8

    The convergence analysis assumes bounded variance for local gradient estimators and global gradients.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

User-level differential privacy (DP) provides certifiable privacy guarantees to the information that is specific to any user's data in federated learning. Existing methods that ensure user-level DP come at the cost of severe accuracy decrease. In this paper, we study the cause of model performance degradation in federated learning under user-level DP guarantee. We find the key to solving this issue is to naturally restrict the norm of local updates before executing operations that guarantee DP. To this end, we propose two techniques, Bounded Local Update Regularization and Local Update Sparsification, to increase model quality without sacrificing privacy. We provide theoretical analysis on the convergence of our framework and give rigorous privacy guarantees. Extensive experiments show that our framework significantly improves the privacy-utility trade-off over the state-of-the-arts for federated learning with user-level DP guarantee.

1. Introduction

The introduction frames user-level privacy as necessary because federated learning remains vulnerable to inference attacks, while existing DP methods can substantially reduce model utility. It proposes local regularization and sparsification to bound update norms before DP operations, alongside theoretical analysis and experiments.

1. Introduction

  • Federated learning alone does not prevent inference attacks that can reveal subgroup properties, individual identities, or social security number completion.The introduction cites demonstrations of these attacks with high confidence.Source: §1 · p. 1
  • Gaussian-mechanism methods clip local updates and add noise, creating a small-threshold bias versus large-threshold variance trade-off that can damage global-model performance.The authors also identify limited interaction between DP operations and FL learning as a source of utility degradation.Source: §1 · p. 1
  • BLUR regularizes local objectives to bound update norms, while LUS zeros low-impact update values before clipping to reduce update magnitude.The stated goal is to lessen clipping's effect without sacrificing local-model accuracy.Source: §1 · pp. 1–2
  • The framework is accompanied by convergence analysis and rigorous privacy guarantees.The authors also report extensive experiments validating the proposed methods.Source: §1 · p. 2

2. Related Work

Related work covers noise-based user-level DP, a voting-based alternative, and FL regularization and sparsification methods. The paper distinguishes its use of these techniques to improve utility under user-level DP.

2. Related Work

  • DP-FedAvg and discrete-Gaussian approaches ensure user-level DP through clipping and noise, while AE-DPFL uses voting and avoids clipping but assumes server access to unlabeled global-distribution data.The authors characterize that data assumption as difficult to satisfy in practice.Source: §2 · p. 2
  • Earlier FL regularization methods address data or device heterogeneity, whereas this work uses regularization to bound local-update sensitivity.The cited methods are and [2].Source: §2 · p. 2
  • Prior privacy-preserving FL sparsification targets instance-level DP and communication costs, while this work targets model utility with user-level DP.The cited prior works are [11].Source: §2 · p. 2

3. Preliminary

The preliminary section sets out the weighted FL objective and FedAvg rounds, introduces differential privacy and sensitivity, then describes user-level DP and DP-FedAvg's clipping-and-noise procedure.

3.1. Federated Learning (FL)

  • FL minimizes a sample-size-weighted average of client losses; FedAvg has clients train locally and the server aggregate their updates in communication rounds.Client losses are weighted by their sample counts relative to the total dataset size.Source: §3.1 · p. 2

3.2. Differential Privacy (DP)

  • Differential privacy bounds output-distribution changes for adjacent datasets, and the Gaussian mechanism adds noise scaled to function sensitivity.Sensitivity is defined using the maximum L2 distance between function outputs on adjacent datasets.Source: §3.2 · pp. 2–3

3.3. Differential Privacy for Federated Learning

  • User-level DP treats datasets differing by one agent and all that agent's records as adjacent; DP-FedAvg clips each agent update and adds scaled Gaussian noise before upload.The paper notes that DP-FedAvg ensures user-level DP but severely harms model utility.Source: §3.3 · p. 3

4. Methodology

Clipping and Gaussian noise can distort local updates, so the method first reduces their norms. BLUR regularizes local training, while LUS sparsifies updates before clipping.

4. Methodology

  • Reducing local-update norms before clipping addresses the distortion from clipping and Gaussian noise that motivates the method.The analysis identifies clipping deviation and noise perturbation as separate sources of error; simply lowering the clipping threshold can worsen clipping distortion.Source: §4 · p. 3

4.1. Bounded Local Update Regularization (BLUR)

  • BLUR adds a norm constraint to local optimization through regularization, making updates more adaptive to clipping at threshold S.When an update exceeds S, BLUR discounts the local learning rate; when it is below S, the step is not limited.Source: §4.1 · p. 4

4.2. Local Update Sparsification (LUS)

  • LUS preserves parameter updates with greater estimated impact on local-model performance and zeros out the others, reducing the update norm.The number of preserved values controls sparsity and the resulting norm reduction.Source: §4.2 · pp. 4–5

5. Theoretical Results

The framework uses Gaussian noise for user-level privacy and analyzes privacy-cost accumulation with a moments accountant. Its convergence analysis separates FedAvg and privacy-operation terms and compares their dependence on the number of participants with DP-FedAvg.

5.1. Privacy Analysis

  • With the same noise scale, each communication round has the same privacy guarantee as DP-FedAvg, and the moments accountant composes privacy cost across rounds.The paper uses the moments accountant for tighter privacy bounds than the cited strong-composition theorem.Source: §5.1 · p. 5

5.2. Convergence Analysis

  • Under Assumptions 1–4, Theorem 2 bounds convergence with a FedAvg term and a separate term from operations for DP.The stated bound includes a privacy-operation term of O(η_gη_lQTd ln(1/δ)/(N^2ε^2)).Source: §5.2 · pp. 5–6
  • The privacy-operation term has quadratic speedup in participant count P, compared with linear speedup for DP-FedAvg.The paper presents this comparison as part of its convergence and privacy/utility analysis.Source: §5.2 · p. 6

6. Experiment Settings

The experiments compare the method with DP-FedAvg and other private-FL baselines on EMNIST and CIFAR-10. They vary privacy budgets and data heterogeneity, using two model architectures and specified training settings.

6. Experiment Settings

  • The evaluation compares DP-FedAvg with DDGauss and AE-DPFL, which provide user-level privacy through different mechanisms.DP-FedAvg is the baseline; DDGauss uses discrete Gaussian noise and secure aggregation, while AE-DPFL uses private voting and secure aggregation.Source: §6 · p. 6
  • Experiments use EMNIST and CIFAR-10 with CNN-2-Layers and ResNet-18 models.EMNIST groups handwritten characters by writer across 3,400 clients; CIFAR-10 data heterogeneity is modeled using a Dirichlet distribution.Source: §6 · p. 6
  • Training uses 30 local iterations, with dataset-specific rounds, agent-selection probabilities, and minibatch sizes.The default BLUR regularization parameter is λ = 0.4 and LUS sparsity is c = 0.7.Source: §6 · p. 6
  • The settings vary privacy budgets on EMNIST and data heterogeneity on CIFAR-10.The captions identify smaller ε as stronger privacy and smaller α as higher heterogeneity.Source: Table 1 · p. 7, Table 2 · p. 7

7. Experimental Results

The experiments report improved accuracy and convergence from local-update regularization and sparsification under user-level DP. They examine benchmark performance, the techniques’ effects on update norms, and robustness across data heterogeneity, communication frequency, and agent sampling.

Performance under different privacy budgets

  • BLUR and LUS consistently outperform prior private-FL methods on EMNIST; they improve DP-FedAvg accuracy by 3% ∼4% on CNN-2-Layers and 4% ∼5% on ResNet-18.The reported gains are larger for ResNet-18 than CNN-2-Layers, and larger at smaller privacy budgets.Source: §7 · p. 7

Effectiveness of BLUR

  • BLUR consistently speeds convergence and improves DP-FedAvg test accuracy across the tested regularization settings.Figure 1 identifies λ = 0 as vanilla DP-FedAvg.Source: §7 · p. 7, Figure 1 · p. 7

Effectiveness of LUS

  • LUS adds up to 2.07% more performance gain to DP-FedAvg + BLUR, with results indicating that BLUR and LUS are synergistic.Table 3 summarizes LUS effectiveness across sparsity settings.Source: Table 3 · p. 7, §7 · p. 8

Effects of bounding local update norms

  • With a 0.1 clipping bound, the proposed method’s pre-clipping update norms are closer to the bound and show less variance than DP-FedAvg’s.The authors report that clipping distorts less information in their framework.Source: §7 · p. 8, Figure 2 · p. 8

Impacts of data heterogeneity

  • Accuracy gains on CNN-2-Layers are 5.11% at α = 0.1 and 2.32% at α = 100; the method consistently outperforms baselines across heterogeneity levels.The authors attribute larger gains at higher heterogeneity to larger local-update norms.Source: §7 · p. 8

Impacts of communication frequency

  • On CIFAR-10, the approach is robust to different local-step counts, while DP-FedAvg degrades when Q is large, such as Q = 40.The authors relate this robustness to limiting local-update norms and reducing accuracy loss from clipping.Source: §7 · p. 8, Figure 3 · p. 8

Impacts of active agents

8. Conclusion

The conclusion identifies pre-clipping control of local-update norms as the key to addressing utility degradation in differentially private federated learning. The framework combines local regularization and sparsification, with convergence and privacy analysis and experiments showing improved utility over state-of-the-art methods.

Conclusion

  • The paper identifies naturally bounding local-update norms before clipping as key to addressing utility degradation under DP.Source: §8 · p. 8
  • The proposed approach uses local regularization and sparsification to address the identified problem.Source: §8 · p. 8
  • The framework is analyzed for convergence and privacy, and experiments show improved model utility over state-of-the-art DP federated-learning methods.Source: §8 · p. 8

Related papers

Loading 2203.03106v3…