PaperlayerView exact revision on arXiv

Source-linked AI summary

Curvature-Aware Clipping: Mitigating Bias in DPSGD via Approximate Second-Order Information

Xincheng Xu, Thilina Ranbaduge, Thierry Rakotoarivelo, David Smith

arXiv:2610.32175v1cs.LGcs.CR

TL;DR

DPSGD’s utility is constrained by clipping bias and DP noise, and existing variants can face tuning, cost, or guarantee challenges. CA-DPSGD constructs a diagonal clipping geometry from noise-corrected privatized second moments and pairs it with anisotropic noise. Across vision and language tasks, it improves utility over evaluated clipping baselines, with the diagonal geometry leaving cross-parameter correlations uncaptured.

  • Problem

    DPSGD’s per-sample clipping and calibrated noise can degrade performance versus non-private training, while existing variants may require complicated tuning, incur computational or memory costs, or lack theoretical guarantees.

    Source: §1 · p. 1

  • Method

    CA-DPSGD constructs a scalable diagonal clipping geometry from previously privatized gradients, corrects the DP-noise contribution to its second-moment estimate, and uses anisotropic Gaussian noise.

    Source: §4.2 · p. 4

  • Results

    CA-DPSGD achieves the highest mean accuracy across evaluated methods on nine datasets, including a 1.30% gain over vanilla DPSGD on ImageNette.

    Source: Experimental Setting · p. 6

  • Takeaways & Limitations

    Experiments across vision and language tasks show improvements over evaluated clipping baselines, and CA-DPSGD can be incorporated into existing DPSGD variants.

    Source: §7 · p. 6, Experimental Setting · p. 6

    The diagonal geometry does not capture cross-parameter correlations, and the convergence analysis assumes smoothness, bounded stochastic-gradient variance, and a positive lower bound on Gt.

    Source: §5 · p. 5, §7 · p. 6

Abstract

from arXiv · show

Differentially Private Stochastic Gradient Descent (DPSGD) has become the de facto standard for training deep neural networks with rigorous privacy guarantees. However, the utility of DPSGD is constrained by gradient clipping bias and DP noise. While existing work has tried to address these issues, the geometric misalignment caused by clipping bias remains a critical challenge. To bridge this gap, we propose a novel framework, Curvature-Aware Differentially Private Stochastic Gradient Descent (CA-DPSGD), which leverages approximate second-order information to construct an optimal, curvature-aware clipping space. CA-DPSGD integrates a rectified estimator for the curvature matrix and is coupled with an anisotropic noise mechanism to ensure privacy guarantees. Extensive empirical experiments demonstrate that our approach outperforms state-of-the-art clipping methods.

1. Introduction

DPSGD is widely adopted for private deep-model training, but clipping and calibrated Gaussian noise are associated with performance degradation versus non-private training. CA-DPSGD addresses clipping bias geometrically, deriving a clipping space and using privatized gradient information with noise correction and anisotropic noise.

1. Introduction

  • Existing DPSGD variants target noise variance or clipping bias, but can require complicated tuning, high computational or memory costs, and lack theoretical guarantees.Source: §1 · p. 1
  • CA-DPSGD derives a clipping space that minimizes clipping bias while maintaining the same amount of DP noise as vanilla DPSGD.The clipping space is constructed each iteration using a private approximation of gradient second moments.Source: §1 · pp. 1–2
  • A noise-correction mechanism and adaptive scaling construct a scalable diagonal clipping geometry using only previously privatized gradients.Source: §1 · p. 2
  • The paper establishes privacy and convergence guarantees, evaluates image and sentence classification benchmarks, and reports compatibility with existing DPSGD variants.Source: §1 · p. 2

2. Related Work

Prior DPSGD improvements address noise variance, clipping bias, or curvature, but can incur tuning, resource, or privacy challenges. The paper positions CA-DPSGD as a lightweight diagonal alternative that corrects noise in its second-moment estimate.

2. Related Work

  • Existing utility methods may require extra tuning, historical states, structured covariance, or specialized architectures; CA-DPSGD instead builds diagonal geometry from privatized gradients and corrects DP-noise contributions.Source: §2 · p. 2
  • Private second-order methods face difficulty protecting per-sample statistics, while CA-DPSGD uses approximate curvature to set clipping geometry before noise injection rather than modifying the optimizer update.Source: §2 · p. 2

3. Preliminaries

The paper studies supervised deep learning as empirical risk minimization over a dataset. Its preliminaries define the Fisher Information Matrix as a local second-order characterization and describe DPSGD’s per-sample clipping and noisy update.

3. Preliminaries

  • The supervised-learning objective is empirical risk minimization over N data points, with f(x; ξ) denoting the loss for one data point and d the parameter dimension.Source: §3 · p. 2
  • The Fisher Information Matrix is the expected outer product of per-sample gradients and, under negative log-likelihood, locally characterizes the optimization landscape to second order.Source: §3 · p. 2
  • DPSGD clips each mini-batch per-sample gradient, averages the clipped gradients, adds Gaussian noise, and updates model parameters.Source: §3 · pp. 2–3

4. Curvature-Aware Clipping

The method chooses a clipping geometry by balancing a clipping-bias bound against a fixed total noise variance. CA-DPSGD approximates that geometry with a diagonal matrix built from privatized gradients, correcting the known noise contribution and using anisotropic noise.

4.1. Optimal Clipping Geometry

  • Vanilla DPSGD clips in a Euclidean ball, whereas a bounded positive-definite matrix generalizes clipping to account for anisotropic gradient distributions.Source: §4.1 · p. 3
  • The optimal clipping matrix minimizes the clipping-bias upper bound subject to a constraint that preserves total noise variance.The resulting geometry assigns smaller clipping weights to directions with larger second moments.Source: §4.1 · p. 3
  • Directly computing the optimal matrix can be computationally prohibitive in high dimensions and may require additional privacy budget, motivating a scalable diagonal approximation.Source: §4.1 · p. 3

4.2. Our Proposed Algorithm: CA-DPSGD

  • CA-DPSGD constructs a diagonal clipping matrix from previously privatized batch gradients, adding no privacy cost through post-processing.Source: §4.2 · p. 4
  • Clipping in the transformed space and mapping back induces anisotropic noise, with the trace constraint preserving the total variance of isotropic DPSGD.Source: §4.2 · p. 4
  • The algorithm maintains an exponential moving average of squared privatized gradients and corrects the known contribution of anisotropic DP noise in the second-moment estimate.Source: §4.2 · p. 4

5. Theoretical Analysis

The theoretical analysis establishes a differential-privacy guarantee and a convergence bound under stated assumptions. The convergence bound separates optimization error, stochastic and DP noise, and a geometry-dependent clipping-bias term.

5. Theoretical Analysis

  • Theorem 3 states a conditional (ϵ, δ)-DP guarantee for Algorithm 1 with sampling probability q = B/N, constants c1 and c2, any ϵ ≤ c1q²T, and δ > 0; the condition after “if” is absent.Source: §5 · p. 5
  • The convergence analysis assumes an L-smooth loss, bounded stochastic-gradient variance, and a clipping matrix Gt bounded below by µId.The theorem also requires η ≤ B 2(2B+1)L.Source: §5 · p. 5
  • The convergence bound separates optimization error, stochastic and DP noise, and a geometry-dependent clipping-bias term.The choice of Gt affects the clipping-bias term under anisotropic gradient second moments.Source: §5 · p. 5

6. Experiments

Across nine vision and language datasets, CA-DPSGD achieves the highest mean accuracy among the evaluated methods. It also integrates with existing DPSGD variants.

Experimental Setting

  • The evaluation covers five image-classification datasets and four GLUE sentence-classification tasks, comparing CA-DPSGD with vanilla DPSGD, Auto-S, and DP-PSAC.The image datasets are MNIST, Fashion-MNIST, CIFAR-10, ImageNette, and CelebA; the GLUE tasks are MNLI, QQP, QNLI, and SST-2.Source: Experimental Setting · p. 5

Main Results

  • CA-DPSGD achieves the highest mean accuracy across the evaluated methods, with gains of 1.30% over vanilla DPSGD on ImageNette and 0.50% on QNLI.The reported results indicate improved utility across both vision and language tasks.Source: Experimental Setting · p. 6
  • CA-DPSGD can also be incorporated into IOM, KFSGD, and DPAdamBC.Additional ablations include the adaptive λ mechanism.Source: Experimental Setting · p. 6

7. Conclusion

The conclusion presents CA-DPSGD as a private curvature-aware clipping framework with a noise-corrected second-moment estimate and diagonal geometry. The surrounding appendices discuss related methods, privacy definitions, and proof steps for clipping-bias and geometry optimization.

7. Conclusion

  • CA-DPSGD uses a noise-corrected second-moment estimate to construct scalable diagonal clipping geometry with anisotropic Gaussian noise, and the paper reports privacy and convergence guarantees and improvements across vision and language tasks.The diagonal geometry does not capture cross-parameter correlations, which the authors identify as motivation for more expressive geometries.Source: §7 · p. 6

Appendix A. Additional Discussion of Related Work

  • Prior clipping-bias approaches include inner-outer momentum and error feedback, while the cited methods have costs including historical buffers, added variance, or specialized architectures.GeoClip [16] minimizes DP noise, whereas CA-DPSGD targets an upper bound on clipping bias; its full-rank released implementation does not scale to the paper's high-dimensional experiments.Source: Appendix A · p. 10

Appendix B. Differential Privacy Background

  • The appendix introduces (ϵ, δ)-DP through adjacent datasets differing by one element and describes Gaussian noise calibrated to ℓ2 sensitivity.ϵ and δ denote the privacy budget and failure probability, respectively.Source: Appendix B · p. 10

Appendix C. Proof of Lemma 1

  • The clipping-bias proof defines a clipping residual and uses Jensen’s inequality, then treats separately the cases where the gradient norm is at most or greater than the clipping threshold.For x ≤ C, the left-hand side is zero; for x > C, the proof bounds x − C by x^2/C.Source: Appendix C · p. 11

Appendix D. Proof of Theorem 2

  • The geometry-optimization proof introduces λ1 > 0 as a Lagrange multiplier for an equality constraint and uses that Gt and Ft are symmetric positive definite.The supplied proof passages show these steps but do not provide the complete optimization expression.Source: Appendix D · pp. 11–12

Appendix E. Proof of Lemma 7

The appendices analyze temporal stability of the privatized second-moment estimate, privacy, convergence assumptions, and the convergence-bound derivation. They also describe the experiment datasets and Figure 1’s comparison scope.

Appendix E. Proof of Lemma 7

  • As β approaches 1, the expected difference between successive privatized second-moment estimates approaches zero, but a large β may over-rely on historical gradients and slow convergence.The paper reports empirical examination of β’s effect in Appendix L.Source: Appendix E · p. 12

Appendix F. Privacy Analysis

  • The privacy analysis uses the Gaussian mechanism and subsampling amplification, then composes privacy over T iterations with the moments accountant to obtain an (ϵ, δ)-DP guarantee.The noisy batch-averaged gradient is protected using post-processing of quantities derived from previously privatized gradients.Source: Appendix F · p. 13

Appendix G. Assumptions for Convergence Analysis

  • The convergence analysis assumes L-smoothness, bounded stochastic-gradient variance, and a positive-definite clipping matrix with bounded eigenvalues.The stated matrix condition includes Gt ⪰ µI for some µ > 0.Source: Appendix G · p. 14

Appendix H. Proof of Theorem 4

  • The proof of the convergence bound defines bt as the difference between the expected per-sample clipped gradient and the true gradient, and combines expectation and variance terms in its upper bound.It also uses a lower bound on Gt to apply the Rayleigh quotient inequality.Source: Appendix H · pp. 14, 16, 17

Appendix I. Experimental Setting

  • Experiments cover five image-classification datasets and four GLUE sentence-classification tasks; Figure 1 compares test accuracy for IOM, KFSGD, and DP-AdamBC with and without CA-DPSGD on five image datasets.The “CA-” prefix denotes integration with CA-DPSGD.Source: Appendix I · p. 17, Figure 1 · p. 18

Appendix J. Additional Experiments: Compatibility with Other DPSGD Variants

CA-DPSGD was tested as a clipping adapter for three DPSGD variants and improved utility on image-classification datasets in most cases. Its gains were not uniform: CA-IOM had a marginal drop on Fashion-MNIST.

Appendix J. Additional Experiments: Compatibility with Other DPSGD Variants

  • The evaluation integrated CA-DPSGD with IOM, KFSGD, and DP-AdamBC, respectively addressing clipping bias, DP-noise variance, and second-order optimization.The experiments used a 4-layer CNN for MNIST and Fashion-MNIST, pretrained SimCLRv2 for CIFAR-10, and ResNet9 for ImageNette and CelebA; results report means and standard deviations over five runs.Source: Appendix J · pp. 18–19
  • On Imagenette, adding CA-DPSGD improved KFSGD test accuracy by approximately 2.0% and DP-AdamBC test accuracy by approximately 1.8%.Across five image-classification datasets, CA-DPSGD improved the utility of the three integrated methods in most cases.Source: Appendix J · p. 19
  • CA-IOM had a marginal performance drop on Fashion-MNIST, where momentum smoothing amplified low-curvature dimensions and information loss outweighed geometric rectification benefits.The passage attributes this outcome to Fashion-MNIST's simple landscape and CA-DPSGD's stricter clipping of low-curvature dimensions.Source: Appendix J · p. 19

Appendix K. Additional Experiments: Ablation Study

The ablations examine the Noise Correction (NC) module and adaptive λ. NC usually improves utility on MNIST and Fashion-MNIST, while adaptive λ outperforms the tested fixed values in most scenarios.

Appendix K. Additional Experiments: Ablation Study

  • Noise Correction improved utility on MNIST and Fashion-MNIST in most cases, while its performance gap was negligible on CIFAR-10.The passage attributes the CIFAR-10 result to fine-tuning a pretrained model with small injected noise variance, making correction less critical in that setting.Source: Appendix K · p. 19
  • The adaptive λ mechanism outperformed fixed λ2 values of 10−6, 10−7, and 10−8 in most scenarios, reducing the need for manual tuning across datasets and privacy budgets.The passage describes a bias–variance trade-off: larger λ suppresses DP-noise variance but adds clipping bias, while smaller λ reduces clipping bias but has higher DP variance.Source: Appendix K · p. 19

Appendix L. Additional Experiments: Parameter Study of β

The β study reports a stability–convergence trade-off: larger β stabilizes momentum updates, but excessive stability can slow optimization. The appendix also points readers to a table of notation.

Appendix L. Additional Experiments: Parameter Study of β

  • For Imagenette, increasing β reduced the difference between successive v_t values, while β values 0.1 and 0.5 degraded convergence.The divergence of v_t is theoretically bounded by O(1−β), so larger β enforces stricter stationarity.Source: Appendix L · p. 19
  • Figure 2 compares CA-DPSGD with and without Noise Correction and compares adaptive λ against fixed λ2 values on three image-classification datasets.The caption names MNIST, Fashion-MNIST, and CIFAR-10 and identifies test accuracy as the measure.Source: Figure 2 · p. 20
  • β = 0.999 was used as a stability–convergence trade-off; β = 0.9999 converged more slowly on Imagenette.The authors describe β = 0.999 as providing sufficient stability for the correction mechanism while maintaining efficient convergence.Source: Appendix L · p. 20
  • On Fashion-MNIST, larger β improved stability while smaller β accelerated convergence; β = 0.9999 eventually reached performance comparable to β = 0.999, more slowly.The passage attributes eventual convergence to the simplicity of the Fashion-MNIST landscape.Source: Appendix L · p. 21

Appendix M. Notations

  • The appendix states that Table 2 summarizes the paper's notation.The supplied table caption identifies it as a table of notations.Source: Appendix M · p. 21, Table 2 · p. 22

Related papers

Loading 2610.32175v1…