PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Hyperparameter Transfer for Differentially Private Optimization

Tianze Wang, Zhiqi Bu, Linjun Zhang

arXiv:2610.22976v2cs.LGcs.CRstat.ML

TL;DR

Large-model hyperparameter tuning is costly, and DP optimization complicates transfer because privacy noise and clipping affect layer-wise updates. DP-µP combines spectral scaling with an SNR condition to derive optimizer-specific rules, and experiments report zero-shot transfer across architectures that matches or exceeds individually tuned baselines while reducing computational costs.

  • Problem

    Although µP transfers hyperparameters across widths in standard training, DP noise and clipping make the spectral condition alone insufficient, while large-model tuning is costly.

    Source: §1 · p. 1, §3.2 · p. 5

  • Method

    DP-µP combines spectral scaling with a layer-wise SNR condition to derive optimizer-specific learning-rate and clipping rules for transfer.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 7

  • Results

    DP-µP transfers optimal hyperparameters from inexpensive proxies zero-shot across MLP, ViT, and GPT-2, matching or exceeding individually tuned baselines while reducing computational costs.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Small proxy models provide zero-shot learning rates and clipping thresholds for large targets, offering an alternative to repeated DP tuning or reliance on auxiliary public datasets.

    Source: §5 · p. 10

    The framework covers width and batch-size scaling, not depth; its analysis assumes a noise-dominant regime and leaves transition-regime adjustments open.

    Source: §6 · p. 10

Abstract

from arXiv · show

Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization ($\mu$P) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP-$\mu$P, a theoretically grounded framework that extends $\mu$P to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP-$\mu$P enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance of individually tuned baselines while significantly reducing computational costs.

1 Introduction

The introduction frames model-size-driven tuning costs as a challenge and argues that DP optimization requires layer-wise stabilization under privacy noise. It introduces DP-µP as a framework for deriving scaling rules and transferring hyperparameters across model widths.

1 Introduction

  • Hyperparameter search becomes computationally intensive as models grow, with grid and random search costs increasing exponentially with model size.Source: §1 · p. 1
  • Standard µP [42] preserves update magnitudes across widths, enabling zero-shot transfer of hyperparameters such as learning rate from small to larger models.Source: §1 · p. 1
  • The paper describes a privacy–utility tension: larger models can perform better but are more prone to training-data memorization, while DP is used to protect privacy.Source: §1 · pp. 1–2
  • The workflow aligns proxy-model SNR, selects its base learning rate, then applies layer-wise scaling to the large model for O(1) loss updates.Source: Figure 1 · p. 2
  • DP optimization is studied with layer-wise clipping in a noise-dominated regime, where privacy noise can overwhelm the gradient signal.The passage notes that noise dominance does not preclude meaningful utility.Source: §1 · p. 2
  • DP-µP derives spectral and SNR conditions for scaling rules and reports zero-shot learning-rate transfer that matches or exceeds individually tuned baselines.The rules cover DP-SGD, DP-Adam, and DP-Muon.Source: §1 · p. 2

2 Preliminary

The preliminaries establish notation for asymptotic and spectral scaling, describe gradient-based updates and µP’s feature-learning conditions, and define the paper’s DP optimization setup. Together, these concepts set up the later derivation of layer-wise scaling rules.

2.1 Notations

  • Θ denotes asymptotic scaling, while the spectral norm and ≍ notation describe operator size and equivalence up to constant factors.Source: §2.1 · pp. 2–3

2.2 Deep Learning and Gradient Descent

  • Training updates layer weights using a layer-specific learning rate and an optimizer-transformed mini-batch gradient, with SGD, Adam-like methods, and Muon given as examples.Source: §2.2 · p. 3

2.3 Maximal Update Parametrization (µP)

  • µP stabilizes activations for feature learning through width-dependent weight scaling and initialization; its spectral condition links layer-wise learning rates to transfer across widths.The analysis uses this condition to derive rules for different optimizers.Source: §2.3 · pp. 3–4

2.4 Differentially Private Optimization

  • DP optimization privatizes gradients through per-sample clipping and Gaussian noise, with automatic layer-wise clipping chosen to avoid tuning thresholds and support scalable distributed training.The setup applies optimizer-specific transformations to the clipped, privatized gradient.Source: §2.4 · p. 4

3 µP for DP optimizers

DP transfer requires both feature-scale spectral alignment and layer-wise SNR alignment; standard µP alone fails when privacy noise dominates. The section derives these conditions and adjusts layer-wise clipping to satisfy them.

3 µP for DP optimizers

  • DP transfer requires the µP spectral condition and layer-wise SNR alignment together; the SNR condition is necessary and sufficient when combined with the layer-wise learning rate.The analysis assumes per-sample gradients are rank-1 and uses a geometric alignment assumption for mini-batch gradients.Source: §3 · p. 4, §3.2 · p. 6

3.1 Standard Optimizers with Per-Sample Gradient Clipping

  • Clipping-only updates retain standard µP transfer: derived scaling rules yield successful learning-rate transfer across widths, and Figure 2a shows aligned loss curves.The figure compares a 5-layer MLP on CIFAR10.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 DP Optimizers with Per-Sample Gradient Clipping and Noise

  • Under noise-dominated DP-SGD, the signal contribution scales as Θ(1/√n), so spectral scaling alone cannot preserve feature learning as width grows.Figure 2b shows that scaling by the noisy update's spectral norm fails to align loss curves across widths.Source: §3.2 · pp. 5–6, Figure 2 · p. 5
  • The layer-wise SNR condition preserves effective noise intensity relative to signal across model scales, enabling hyperparameters tuned on a proxy to transfer to the target.The condition applies in the noise-dominated regime and must hold at every layer.Source: §3.2 · p. 6

3.3 Layer-wise Clipping Parameter

  • Because input and output dimensions are fixed while hidden widths scale, noise scaling alone cannot match boundary-layer SNR; layer-wise clipping coefficients supply the adjustable parameters.The constraints determine target and proxy clipping parameters for a target privacy noise level.Source: §3.3 · p. 7
  • In practice, calibrate target noise, use uniform proxy clipping, solve the SNR constraints, then tune the proxy learning rate and map it to target layer-wise rates.The procedure uses Lemma 2.1 for the learning-rate mapping.Source: §3.3 · p. 7

4 Experiments

Experiments test DP-µP transfer across image and text models, with coordinate checks used to verify width-consistent activations. Across these settings, results support transfer across model widths, including on GPT-2.

4 Experiments

  • The experiments assess transfer across width and batch size, comparing DP-µP with standard parametrization using SGD, Adam, and Muon.Unless specified otherwise, DP experiments use ϵ = 2 and δ = 10−5; the proxy learning-rate search cost is excluded from this budget.Source: §4 · p. 7, §4.1 · p. 7

4.1 Image Classification Pretraining

  • Across optimizers, µP produced more stable training dynamics and lower final loss than standard parametrization on CIFAR-10.The section presents SGD and Muon as representative examples and defers Adam results to Appendix C.1.Source: §4.1 · pp. 7–8
  • Larger ViTs achieved lower training loss across widths when the SNR condition held, while using the same learning rate as smaller models in the DP setting.Figure 5 reports ViT pretraining on CIFAR-100; CIFAR-10 results are deferred to Appendix C.3.Source: §4.1 · p. 8, Figure 5 · p. 8

4.2 Coordinate Check

  • MLP and ViT coordinate checks showed stable activation scales across widths, supporting the correctness of the DP-µP implementation.The checks track feature-activation l1 norms over initial training steps; SP activation norms typically explode as width increases.Source: §4.2 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9

4.3 Text Generation Pretraining

  • 90.35% lower theoretical compute and training runs reduced from approximately 30 to 11 minutes accompanied zero-shot learning-rate transfer across three GPT-2 scales.The experiments use a 5-head proxy to select the learning rate for a 36-layer GPT-2 Large target, with a 10-head intermediate model.Source: §4.3 · p. 9, Figure 8 · p. 9

5 Discussion

The discussion presents DP-µP as a framework for learning-rate transfer in differentially private deep learning. Its layer-wise scaling rules target consistent SNR across model scales, and experiments report transfer from inexpensive proxies to larger targets.

5 Discussion

  • DP-µP derives layer-wise learning-rate scaling to keep SNR consistent across model scales, addressing the interaction of gradient clipping and feature scaling.The framework is described as theoretically grounded and empirically validated for differentially private deep learning.Source: §5 · p. 10
  • Across MLP, ViT, and GPT-2, optimal learning rates and clipping thresholds identified on small proxy models transferred zero-shot to larger targets.Source: §5 · p. 10
  • Proxy-based transfer offers an alternative to repeated DP hyperparameter searches or reliance on auxiliary public datasets, reducing computational costs.Source: §5 · p. 10

6 Limitations and Future Work

The framework addresses width and batch-size scaling, but not depth scaling; accumulated privacy noise may challenge the layer-wise SNR condition in deeper networks. The appendix argues that bias terms and standard nonlinear activations do not change the order of width scaling.

6 Limitations and Future Work

  • The framework covers width and batch-size scaling, not depth scaling, where accumulated privacy noise may violate the layer-wise SNR condition.The paper identifies deriving a depth-scaling mechanism or establishing an impossibility result as future work.Source: §6 · p. 10

A.1.1 Bias Terms.

  • Treating each bias vector as a weight matrix connected from a width-one auxiliary layer lets the spectral analysis apply to biases.Under the stated spectral scaling condition, the bias norm is Θ(√n_l).Source: §A.1.1 · p. 14
  • The bias formulation accommodates layer-wise clipping because the weight matrix and bias share a clipping coefficient and noise scale.Source: §A.1.1 · p. 14

A.1.2 Nonlinear Activations

  • The nonlinear-activation analysis assumes activations preserve signal-magnitude order; ReLU and GELU are cited as examples that do not make order-one inputs explode or uniformly vanish.Under this assumption, nonlinearities do not alter the asymptotic width scaling, allowing the spectral analysis to use a linear approximation.Source: §A.1.2 · p. 14

A.2 Proof of Lemma 2.1

The proof links feature learning to the spectral scale of each layer's weight update. It then uses the optimizer update equation to derive the layer-wise learning rate that meets that scale.

A.2 Proof of Lemma 2.1

  • The proof starts from the feature-learning desideratum that feature changes remain appropriately scaled after a gradient step.Source: §A.2 · p. 14
  • Because ΔW_l = −η_l · g_l, the proof imposes a spectral scaling requirement on the weight update and relates its norm to the resulting activation change.When the input aligns with the update's top singular vector, ∥Δa_l∥₂ ≍ ∥ΔW_l∥_* · ∥a_(l−1)∥₂; the required update norm makes feature changes scale comparably to activations.Source: §A.2 · p. 15
  • Substituting the required update scale into the gradient-update equation yields the layer-wise learning rate η_l.Source: §A.2 · pp. 14–15

A.3 Derivation of Layer-wise Learning Rate Scaling Rules

The appendix derives layer-wise learning-rate scaling from update spectral norms across non-private, clipping-only, and noise-dominated regimes. It extends the constraints to input and normalization layers, including Transformer LayerNorm.

A.3 Derivation of Layer-wise Learning Rate Scaling Rules

  • The derivation sets layer-wise learning rates by matching update spectral norms to feature-learning requirements, with Table 2 summarizing rules across privacy settings.It uses rank-1 gradient and batch-alignment facts to obtain the spectral norms used in the derivations.Source: §A.3 · p. 15, Table 2 · p. 15

A.3.1 Non-Private Case (Standard µP)

  • In standard µP, the derivation treats SGD, sign-based Adam, and Muon separately, with Muon orthogonalization constraining update singular values to one.The resulting scaling rules use the ratio of the desired update spectral norm to the gradient spectral norm.Source: §A.3.1 · p. 16

A.3.2 Clipping-Only Case

  • With clipping but negligible noise, clipping preserves the rank and spectral properties of sign-based Adam and Muon updates, while SGD's Frobenius normalization changes its scaling.The SGD derivation invokes the sufficient batch-alignment assumption.Source: §A.3.2 · p. 16

A.3.3 Differentially Private Case (Noise-Dominated)

  • In the noise-dominated regime, Gaussian-noise spectral norms scale as √n_l + √n_(l-1); Adam reflects this geometry, while Muon's orthogonalization keeps its update-direction norm constant.Unlike aligned signal gradients, noise does not align across coordinates; Muon approximates orthogonalization with Newton–Schulz iterations.Source: §A.3.3 · p. 17

A.4 Boundary Condition at the Input Layer

  • The input layer receives fixed data samples, so its feature norm depends on input dimension; non-unit-variance preprocessing requires adjusting its learning-rate scale.The stated boundary condition uses the input-weight spectral norm and input-vector norm.Source: §A.4 · p. 17

A.5 SNR for Layer Normalization Parameters

  • For LayerNorm gain parameters, the update is diagonal, so its spectral norm is the maximum absolute gradient entry; under the stated tail assumption, this maximum scales logarithmically with width.The scaling is based on independent, identically distributed sub-Gaussian entries and is used to maintain the SNR condition.Source: §A.5 · p. 17

A.6 Scaling Rules for Transformers with LayerNorm

  • The Transformer constraints separate linear and normalization layers: LayerNorm spectral norms scale as ln n, rather than the √n scaling of linear layers.The extension explicitly accounts for both sets of layer indices.Source: §A.6 · p. 18

B Empirical Support for Assumption 3.2

The empirical checks support the predicted spectral-norm scaling for DP-SGD and DP-Adam across batch sizes and widths. A GPT-2 check also supports the DP-Adam pattern, while the analysis requires only comparable gradient alignment between proxy and target models.

B Empirical Support for Assumption 3.2

  • The MLP evaluation varies batch sizes 125–1000 and widths 256–1024 at fixed noise multiplier and learning rate; DP-Muon is omitted because its preconditioning maps updates to unit-spectral-norm matrices.The study evaluates DP-SGD and DP-Adam.Source: B Empirical Support for Assumption 3.2 · p. 18
  • For DP-SGD, spectral norms approximately halve when batch size doubles and grow proportionally to √n_l + √n_(l-1) across widths.Table 3 reports the mini-batch average-gradient spectral norms across batch sizes and network widths.Source: Table 3 · p. 18, B Empirical Support for Assumption 3.2 · p. 19
  • For DP-Adam, empirical spectral norms remain stable across batch sizes and grow proportionally to √n_l + √n_(l-1) as width increases.Table 4 reports the mini-batch average-gradient spectral norms across batch sizes and network widths.Source: Table 4 · p. 19, B Empirical Support for Assumption 3.2 · p. 19
  • On GPT-2, DP-Adam spectral norms are independent of batch size and grow with √n_l + √n_(l-1); exact gradient alignment is unnecessary if proxy and target alignment are comparable.The check varies widths 320, 640, and 1280 and batch sizes 64, 128, and 256.Source: B Empirical Support for Assumption 3.2 · p. 19, Table 5 · p. 19

C.1 Additional MLP Results for Adam

The supplementary experiments compare SP and µP with Adam on CIFAR-10 MLPs and examine batch-size transfer on MLPs and ViTs. The batch-size comparisons pair changes in batch size with noise multiplier changes and hold iteration count fixed.

C.1 Additional MLP Results for Adam

  • Figure 9 compares SP and µP training loss for a 5-layer CIFAR-10 MLP using Adam in non-private and differentially private settings.For the DP comparison, clipping coefficient and noise are adjusted to satisfy the SNR condition.Source: §C.1 · p. 19, Figure 9 · p. 20

C.2 Batch Size Transfer

  • The batch-size transfer comparison scales noise multiplier σ with batch size, keeping σ/B—and therefore the noise-to-signal ratio—unchanged.Fixing iterations approximately holds the privacy budget constant; transfer uses the noise multiplier returned by the privacy accountant for the chosen batch size.Source: §C.2 · p. 20
  • On CIFAR-10, MLP loss curves align across batch sizes when plotted against iterations for SGD, Adam, and Muon.Figures 10 and 11 report the corresponding batch-size transfer comparisons for MLP and ViT, respectively.Source: Figure 10 · p. 20, Figure 11 · p. 20

C.3 ViT Supplementary Results

On CIFAR-10 ViT, the supplementary results contrast unstable or diverging width-dependent dynamics under SP with perfect hyperparameter transfer under DP-µP. The DP-µP comparison includes SGD, Adam, and Muon.

C.3 ViT Supplementary Results

  • On CIFAR-10 ViT, DP-µP demonstrates perfect hyperparameter transfer, while SP fails to maintain consistent training dynamics across widths under DP-SGD and DP-Adam.Figure 13 shows the DP-µP optimizer comparison; Figure 12 shows SP dynamics becoming unstable or diverging as width increases.Source: §C.3 · p. 20, Figure 13 · p. 21, Figure 12 · p. 21
  • Under SP, ViT training dynamics become unstable or diverge as model width increases, unlike the transfer observed with µP.The cited comparison identifies the failure as occurring under Standard Parametrization.Source: Figure 12 · p. 21
  • The DP-µP ViT optimizer configuration includes SGD, Adam, and Muon, with SGD boundary updates for Muon.Source: Figure 13 · p. 21

C.4 Final Utility and Robustness across Seeds

The appendix evaluates test-accuracy transfer across widths and seeds, contrasts standard parametrization’s activation scales, illustrates DP-µP parameter derivation, and specifies end-to-end privacy accounting.

C.4 Final Utility and Robustness across Seeds

  • Under µP, the optimal learning rate stays nearly constant across widths, and at the largest width test accuracy is slightly higher than directly tuned SP for SGD and Adam; the ViT shows the same stability across widths and seeds.The comparison uses matched privacy and five fixed seeds; the test-accuracy-optimal rates differ from those that minimize training loss.Source: Table 6 · p. 21, §C.4 · p. 21, Table 7 · p. 22

C.5 Standard Parametrization Coordinate Check

  • Standard parametrization has inconsistent activation scales across widths in the MLP and ViT, including when the SNR condition is applied to the MLP.The coordinate checks contrast with the stable µP behavior shown in the main text.Source: §C.5 · p. 21, Figure 14 · p. 22, Figure 15 · p. 22

C.6 Illustrative Example of Parameter Derivation

  • The illustrative transfer uses a 5-layer CIFAR-10 MLP, mapping a proxy with hidden widths 128 to a target with hidden widths 8192.The example fixes privacy, batch size, and training steps before the search; at batch size 500 for 10 epochs, the target noise multiplier is approximately 1.0229, rounded to 1.Source: §C.6 · pp. 22–23
  • The example derives proxy noise multiplier σ′ ≈ 6.91 and target layer-wise scaling coefficients to preserve SNR across widths, then scales the proxy learning rate by spectral-norm ratios for the target.It assumes a proxy global learning rate of 1 for numerical stability and reuses the rest of the proxy training recipe unchanged on the target.Source: §C.6 · p. 23
  • If batch size is also selected, the noise multiplier must be recomputed for each candidate and the proxy-selection steps repeated.Source: §C.6 · p. 23

C.7 End-to-End Privacy Accounting

  • Constructing the DP-µP-aligned proxy configuration uses privacy parameters and architectures without accessing the private dataset, after which proxy search and deterministic transfer map the selected rate to target layer-wise rates.The paper describes this transfer as having no privacy cost.Source: §C.7 · p. 23
  • End-to-end accounting must include the full search-and-selection mechanism as well as target training; basic sequential composition adds their privacy budgets, while a common accountant can provide tighter guarantees.Search over multiple learning rates on the same private dataset must account for all evaluations and selection; public-data search has zero search privacy cost.Source: §C.7 · pp. 23–24
Loading 2610.22976v2…