Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger

1 Introduction

• DP training privately releases gradients to address privacy risks such as membership leakage and memorized personal information in generated text.The introduction contrasts this with standard optimizers’ use of per-sample gradients and identifies R and σ as additional DP-specific hyperparameters.Source: §1 · pp. 1–2
• ImageNet ResNet18 accuracy can fall from 45% to 31% when R doubles and to 0.1% when R quadruples, making threshold selection consequential.Unlike the noise multiplier, R cannot be derived from the privacy budget; tuning R with the learning rate requires an expensive 2D search.Source: §1 · p. 2
• Existing adaptive clipping can replace tuning R with tuning new hyperparameters, while data-dependent threshold tuning can incur additional privacy loss.The cited alternatives include selecting a gradient-norm quantile or conducting multiple cheaper one-dimensional searches.Source: §1 · p. 2
• Automatic clipping removes R from general DP optimizers, and the authors report comparable privacy and efficiency, non-convex convergence guarantees, and strong vision and language results.They also report reducing large-scale ablation cost by 5×2 and provide code examples for switching from Abadi’s clipping.Source: §1 · p. 2

2.1 Differential Privacy

• Subsampled Gaussian DP training samples examples independently, clips per-sample gradients to bound sensitivity, and adds Gaussian noise scaled by R and σ.Privacy accounting tools can derive σ from the privacy budget and training parameters.Source: §2.1 · p. 3

2.2 Differentially Private optimizers with general clipping operations

• DP optimizers use privately released stochastic gradients, while clipping research either adapts Abadi’s threshold or proposes alternative clipping functions.Automatic clipping is presented as an alternative clipping method that works orthogonally to optimizers, architectures, and gradient-norm computation.Source: §2.2 · p. 3

3.1 Small clipping threshold often works best

• Small thresholds are reported as best for several large vision and language models, while clipping affects 100% of GPT2 generation gradients and about 20–60% in QQP, QNLI, and MNLI.Figure 1 compares threshold and learning-rate ablations for GPT2/E2E with DP-AdamW and ResNet18/ImageNet with DP-SGD.Source: Figure 1 · p. 3, §3.1 · p. 4

3.2 Per-sample gradient normalization as new clipping

• When per-sample gradients satisfy the stated nonnegative dot-product condition, AUTO-V is optimal under the paper’s criterion and significantly magnifies similarity relative to Abadi’s clipping.The comparison is shown for RoBERTa-base with DP-Adam on SST2 at ε = 3.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Stability constant breaks scale-invariance and remains stationary

• AUTO-V loses gradient magnitudes and can enter a lazy region, whereas AUTO-S’s positive stability constant preserves magnitude information and allows gradients to vanish as they approach zero.The authors state that this property supports convergence of AUTO-S, but not AUTO-V, to zero gradient norms; Figure 3 illustrates the lazy-region issue when class gradients cancel.Source: §3.3 · p. 5, Figure 3 · p. 5

4 Automatic DP Training

• Automatic clipping avoids tuning R and σ, leaving regular optimizer hyperparameters; γ=0.01 is the default because common optimizers are insensitive to γ.For RoBERTa-base with DP-Adam on SST2, QNLI, and MNLI, searching only over η instead of (R, η) saves tuning cost by 5×.Source: §4 · pp. 5–6, Figure 4 · p. 6

4.1 Non-adaptive optimizer couples clipping threshold with learning rate

• For non-adaptive optimizers, increasing R has the same effect as increasing η; the R-dependent method is equivalent to an R-independent one with learning rate ηR.With weight decay λ, the equivalent R-independent optimizer uses λ/R.Source: §4.1 · p. 6

4.2 Adaptive optimizer can be insensitive to clipping threshold

• Adaptive optimizers cancel R rather than coupling it to η; the equivalence covers AdaGrad, Adam-family methods, and other adaptive optimizers.For coupled weight decay, λ maps to λ/R; with decoupled weight decay, automatic DP-AdamW retains the same η and λ.Source: §4.2 · p. 6

4.3 Automatic clipping is equally private and maximizes utility

• AUTO-S has the same privacy guarantee as existing DP optimizers and, for γ>0, preserves per-sample gradient magnitude ordering that AUTO-V and Abadi clipping can violate.For small γ, AUTO-S allows more signal through for the same noise; as gradients approach zero, its clipped gradients approach standard SGD rather than showing AUTO-V’s instability.Source: §4.3 · p. 7, Figure 5 · p. 7

5.1 Convergence theory of DP-SGD to stationary points

• AUTO-S DP-SGD achieves an O(T^-1/4) rate for the minimum expected true-gradient norm, matching standard SGD; this guarantee uses symmetric gradient noise, unlike AUTO-V.The theorem also requires lower-bounded loss and smoothness assumptions.Source: §5.1 · pp. 7–8
• For AUTO-V, the convergence bound’s residual G is at least ξ, while for AUTO-S with positive γ it can reduce to zero.Real-data evidence also shows that positive γ reduces the gradient norm significantly.Source: §5.1 · p. 8

5.2 Analysis of factors affecting the convergence

• The stability constant γ and learning rate affect convergence but not privacy; small γ helps initially, whereas larger γ converges faster asymptotically.The learning rate is selected by minimizing the analyzed bound.Source: §5.2 · p. 8
• At fixed expected batch size, increasing iterations decreases the analyzed convergence quantity; at fixed iterations or epochs, a larger batch also decreases it.The authors consequently favor larger noise multipliers, while noting that these choices must account for the privacy–accuracy tradeoff.Source: §5.2 · p. 8
• Pretraining can improve DP accuracy through a smaller initial loss and a smoother, flatter initialization with smaller ξ.The analysis links pretraining to both the initial loss and the initialization’s smoothness and gradient-noise scale.Source: §5.2 · p. 8
• The analyzed optimal learning rate is larger for smaller models, weaker privacy, or a smaller iteration budget.The stated dependencies are model dimension, privacy level, and number of iterations.Source: §5.2 · p. 8

6.1 Image classification

• AUTO-S clipping outperforms existing clipping with statistical significance across all reported image datasets.Automatic DP optimizers also have smaller standard deviations across runs, indicating better reproducibility and stability.Source: Table 1 · p. 9, §6.1 · p. 9

6.2 Sentence classification

• Full-parameter finetuning with AUTO-S outperforms or matches state-of-the-art performance on all reported sentence-classification tasks.The comparison uses RoBERTa models and the same hyperparameters as the cited prior setup.Source: Table 2 · p. 9, Table 3 · p. 9, §6.2 · p. 9

6.3 Table-to-text generation

• AUTO-S full-parameter finetuning has the best overall performance on E2E table-to-text generation and scales with stronger performance on larger GPT-2 models.The experiments also report that AUTO-S can be combined with LoRA.Source: Table 4 · p. 10, §6.3 · p. 10

• Adaptive threshold methods use data-adaptive information such as gradient-norm quantiles, but may replace tuning R with tuning quantiles or privacy-budget allocation.The paper proposes eliminating R rather than searching for an appropriate threshold.Source: §7 · p. 10
• The authors distinguish small-threshold clipping from AUTO-S, arguing that it retains R tuning and does not reveal AUTO-S’s stability constant.They also note concurrent work with the same AUTO-S algorithm but fundamentally different theoretical analysis and experiment design.Source: §7 · p. 10

8 Discussion

• AUTO-S is a drop-in replacement for per-example clipping and combines non-convex convergence guarantees with strong empirical performance on vision and language tasks.The discussion describes it as advancing DP learning on these tasks.Source: §8 · p. 10

A Proof of differential privacy

• Automatic clipping preserves bounded global sensitivity, so matching the noise multiplier gives it the same Gaussian-mechanism privacy accounting as Abadi’s clipping.The accounting represents both approaches as adaptive compositions of T Poisson-sampled Gaussian mechanisms with sampling probability B/n and noise multiplier σ.Source: A Proof of differential privacy · p. 16

B.1 Non-adaptive DP optimizers

• For non-adaptive optimizers, R-dependent AUTO-S is equivalent to R-independent AUTO-S after scaling the learning rate by R and weight decay by 1/R.The proof gives this equivalence for DP-SGD and says HeavyBall and NAG can be handled similarly.Source: §B.1 · p. 16

B.2 Adaptive DP optimizers

• For DP-AdamW, R-dependent AUTO-S is equivalent to R-independent AUTO-S with unchanged learning rate and weight decay scaled by 1/R.The appendix identifies this as the most complicated adaptive case and says other adaptive optimizers can be analyzed similarly.Source: §B.2 · pp. 16–17

B.3 Automatic per-layer clipping

• Per-layer AUTO-S supports thresholds tuned proportionally, but changing one threshold independently can break the coupling with the learning rate.Uniform per-layer clipping requires tuning only one norm rather than L separate norms.Source: §B.3 · p. 17

C.2 Main proof of convergence for DP-SGD (the non-envelope version)

• Theorem 6 gives an upper bound on the minimum expected gradient norm, derived under Assumptions 5.1–5.3.The proof uses the symmetry assumption and analyzes AUTO-V and AUTO-S separately; for AUTO-S, r > 1 matters for convergence of the gradient norm to zero.Source: §C.1 · p. 18, §C.2 · pp. 21, 23

C.1 Main proof of convergence for DP-SGD (the envelope version)

• Under r ≥ 1 and γ > 0, automatic DP-SGD has an asymptotic convergence rate of O(T^-1/4).The analysis focuses on this regime because it is the only one in which the bound can converge to zero; the rate follows by minimizing over r.Source: §C.1 · p. 18

C.3 Proof of Lemma C.1

• The Lemma C.1 proof lower-bounds an expected improvement using properties of f(c, S; Γ), including its decrease in S and its behavior as the threshold ratio r varies.The analysis defines r as a measure of the minimum gradient norm at convergence, informally ξ/r.Source: §C.3 · p. 23

C.4 Proof of Theorem 8

• Theorem 8 distinguishes the clipping regimes: AUTO-V cannot compress the gradient norm to ξ/r for r ≥ 1, while AUTO-S can do so for r > 1.For AUTO-V, the lower bound becomes trivial at r ≥ 1; AUTO-S uses a positive stability constant.Source: §C.4 · p. 24

D Convergence rate of standard SGD

• Theorem 9 considers standard non-DP SGD run for T iterations under Assumptions 5.1–5.3, and the analysis does not require Assumption 5.3’s symmetry assumption.Source: D Convergence rate of standard SGD · p. 25

E.1 Proof of Theorem 7

• Theorem 7 establishes that f(c, S; Γ) decreases in S and that its minimum over c decreases in S; it also decreases in c when S > 1.The appendix visualizes these properties in Figures 10 and 11.Source: §E.1 · pp. 25–26, Figure 10 · p. 26, Figure 11 · p. 27

E.2 Proof of Lemma E.1

• Lemma E.1 supplies the positivity step used to establish monotonicity of f with respect to S.The proof reduces the claim to positivity of AΓ² + BΓ + C.Source: §E.1 · p. 26, §E.2 · p. 27

E.3 Proof of Lemma E.2

• For all 0 < c < 1, Γ > 0, and S > 1, the cited derivative satisfies dc < 0, so the result is negativity, not positivity.The proof says the case S² − 3cS + 2 ≤ 0 is settled directly, then proceeds by contradiction otherwise.Source: §E.1 · p. 26, §E.3 · p. 27

E.4 Proof of Fact E.3

• Fact E.3 states that a quadratic Ax² + Bx + C with A, B, C > 0 is positive for x ≥ 0, with minimum C at x = 0.Source: §E.4 · p. 28

E.5 Assumption of symmetric gradient noise

• The paper says its symmetry assumption is less strict than assumptions used in non-DP literature and can be relaxed to mirror symmetry about the hyperplane normal to the gradient.Source: §E.5 · p. 28

F.1 Balanced binary classification

• The balanced binary-classification experiment uses 10,000 positive and 10,000 negative samples drawn from class-conditional Gaussian distributions and a logistic regression model.Source: §F.1 · p. 28

F.2 Mean estimation on Gaussian mixture data

• In the Gaussian-mixture mean-estimation example, AUTO-V and Abadi’s clipping lose motivation to update the estimator on (−1, 1), while the regular and AUTO-S gradients vanish at the minimum.Source: §F.2 · p. 29

G.1 Image classification settings

• The vision settings cover MNIST, FashionMNIST, pretrained CIFAR10, ImageNette, and CelebA classification tasks.Source: §G.1 · p. 29

G.2 Sentence classification settings

• Sentence-classification experiments use the same hyperparameters for automatic clipping as for Abadi’s clipping, and the authors report robustness to learning rates around the optimum.Source: §G.2 · p. 30

G.3 Table-to-text generation settings

• The E2E table-to-text experiments evaluate multiple GPT2 model sizes using the same setup and clipping hyperparameters as the cited prior work.Source: §G.3 · p. 30

H.1 Frequency of clipping

• Abadi’s clipping clips a large proportion of per-sample gradients across sentence-classification tasks, and clips 100% of GPT2 gradients at every iteration.For the GPT2 models, the authors say this makes its performance equivalent to AUTO-V clipping.Source: §H.1 · pp. 31–32

H.2 Stability constant helps AUTO clipping reduce gradient norm

• The gradient-norm analysis plots norms across iterations for different automatic clipping methods on SST2 and MNLI using RoBERTa-base.Source: §H.2 · p. 32, Figure 14 · p. 32

H.3 Choice of stability constant is robust

• In the ablation study, with learning rate 0.0005, values 0.0001 < γ < 1 give similar accuracy, and the largest good γ is 1000 times bigger than the smallest good γ.Source: §H.3 · p. 32

I Full table of GPT2 generation task on E2E dataset

• Automatic clipping consistently performs better than other methods for GPT2, GPT2-medium, and GPT2-large on E2E generation.Table 8 reports test performance on E2E; the passage notes that LoRA with Abadi’s clipping is also strong in some cases.Source: I Full table of GPT2 generation task… · p. 33, Table 8 · p. 33

J Further experiments on CelebA dataset

• CelebA experiments show comparable test accuracy to Abadi’s DP optimizers without tuning the clipping threshold for each task or label.Learning labels separately gives better accuracy than joint learning, but requires more computation.Source: J Further experiments on CelebA dataset · p. 33

J.1 Multi-label classification

• Joint CelebA classification uses one ResNet9 with 40 sigmoid cross-entropy outputs whose losses are summed.Table 9 compares accuracy under the joint multi-label setting.Source: §J.1 · p. 34, Table 9 · p. 34

J.2 Multiple binary classification

• Separate CelebA classification trains 40 ResNet9 binary models, one per label.Table 10 compares accuracy for the separate-label setting.Source: §J.2 · p. 35, Table 10 · p. 35

K Code implementation of automatic clipping

• The paper says automatic clipping can be implemented by setting R to 1 or another constant.The passage points to Theorems 1 and 2 for this choice.Source: K Code implementation of automatic… · p. 36

K.1 Opacus

• Opacus implementations change clipping-factor lines for both all-layer and per-layer automatic clipping.The appendix gives instructions for current and older Opacus versions.Source: §K.1 · p. 36

K.2 ObJAX

• For ObJAX 1.6.0, the automatic-clipping implementation changes lines 92 and 145 of its gradient code.The appendix identifies these as the locations to modify.Source: §K.2 · p. 36

K.3 Private-transformers

• The private-transformers 0.1.0 codebase is modified at line 349 for the paper’s sentence-classification and table-to-text experiments.The appendix provides alternative clipping-factor expressions for the change.Source: §K.3 · p. 36

L More on related works of per-sample clipping

• The authors distinguish their non-convex convergence analysis from [74], which assumes relaxed Lipschitz smoothness instead of their symmetric-gradient-noise assumption.They report experiments on over 10 tasks, including DP-GPT2, compared with two smaller models in [74].Source: L More on related works of per-sample… · p. 37

L.1 Clarifications

• Per-sample clipping accuracy is insensitive to R only within a sufficiently small region; outside it, accuracy can be unacceptably low, and the suitable threshold varies by task, dataset, and optimizer.The required tuning can be expensive; R = 1 is not universally suitable, as [41] uses R = 0.1 for language models.Source: §L.1 · p. 37
• AUTO-V/S is presented as fully automatic for tuning R, while AUTO-V approximates per-sample gradient normalization; γ remains stable across a reported 10,000-fold range.At learning rate 0.0005, accuracy is reported as similar from γ = 0.001 to 10 in Figure 15.Source: §L.1 · p. 37
• With learning-rate adjustment proportional to R, AUTO-V/S maintains the same accuracy and is described as completely insensitive to R.The cited prior approaches say proportional adjustment can maintain similar accuracy or make performance less sensitive to the clipping norm [38] [17].Source: §L.1 · p. 37
• For DP-SGD, the learning rate absorbs a factor of R, whereas adaptive optimizers such as Adam and AdaGrad cancel R, producing different performance landscapes.The contrast is visualized as row-wise for DP-Adam and diagonal for DP-SGD in Figure 1.Source: §L.1 · p. 37
• Re-parameterized clipping can raise weight decay 64-fold when R changes from 1 to 2^-6, potentially worsening accuracy; AUTO-V/S avoids this through its analyzed decoupling.The passage attributes the increase to incomplete decoupling and points to the blue curve in [17, Figure 8].Source: §L.1 · p. 38

L.2 Connections to normalized optimisation

• Unlike per-sample normalization or clipping, batch-gradient normalization is unbiased in direction; per-sample normalization is biased because a sample gradient need not align with the mini-batch gradient.The paper says this distinction also holds when normalization is replaced by clipping.Source: §L.2 · p. 38
• Theorem 4 indicates that bias from per-sample clipping is not fatal to asymptotic convergence, while Table 11 notes that R-based clipping requires tuning and γ has theoretical and empirical benefits.Table 11 compares clipping functions for computer vision and natural language processing.Source: Table 11 · p. 37, §L.2 · p. 38