PaperlayerView exact revision on arXiv

Source-linked AI summary

Clipped or Unclipped? Finite-Sample Trade-offs for Averaged SGD under Heavy-Tailed Noise

Alexandra Suvorikova, Egor Gladin, Darina Dvinskikh, Artem Agafonov, Mohammad Alkousa, Yuriy Dorn, Vladislav Matyukhin, Alexander Gasnikov

arXiv:2609.32962v1cs.LGmath.OC

TL;DR

Averaged SGD clipping suppresses extreme updates but also changes the mean update, so its statistical benefit depends on the requested accuracy and confidence. The paper compares clipped and unclipped methods under finite conditional moments and finds that clipping’s confidence improvement may affect only a lower-order correction for p > 2, while fixed thresholds can also alter asymptotic variance or target.

  • Problem

    Clipping controls large stochastic updates, but its effect on the final averaged-SGD estimator must be assessed because it also changes the mean update and can attenuate the optimization signal.

    Source: §1 · p. 1

  • Method

    The paper derives finite-sample comparisons of clipped and unclipped Polyak–Ruppert averaged SGD under finite conditional p-th moments, with explicit accuracy-confidence conditions and bounds that track clipping bias and moment scales.

    Source: §1 · p. 1

  • Results

    For p > 2, the Gaussian term can dominate the unclipped bound so clipping need not improve its leading order, while balanced clipping gives logarithmic confidence dependence for the heavy-tail correction; at p = 2, that improvement affects the leading bound.

    Source: §1 · p. 1, §7 · p. 9

  • Takeaways & Limitations

    Clipping policy should be compared at the intended accuracy, confidence, and horizon because its benefits and costs depend on the regime and threshold.

    Source: §7 · p. 9

    The sample-size comparison requires horizons satisfying the stated remainder conditions; outside that regime, the full bounds must retain squared-gradient contributions.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional $p$-th moments, $p\ge2$. Our main result gives explicit accuracy and confidence conditions under which, for $p>2$, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At $p=2$, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.

1 Introduction

Clipping controls large updates but also changes the mean update, so its value for averaged SGD depends on the accuracy and confidence required. The paper compares clipped and unclipped methods, identifies when clipping changes the leading guarantee, and analyzes finite-horizon and asymptotic costs.

1 Introduction

  • For p > 2, the unclipped heavy-tail correction decays faster than the Gaussian term at fixed confidence, so clipping need not improve the leading order once the Gaussian term dominates.Balancing clipping bias and concentration changes the correction’s confidence dependence from polynomial to logarithmic; at p = 2, the improvement affects the leading bound.Source: §1 · p. 1
  • The comparison gives explicit accuracy and confidence conditions and retains the distinct moment scales of whole-gradient clipping and noise-centered clipping.Whole-gradient clipping acts on signal and noise; the oracle noise-centered benchmark separates those effects, while projected convex bounds also account for initialization and squared-gradient terms.Source: §1 · p. 1
  • A rare-outlier construction in a scalar quadratic recursion attains the unclipped heavy-tail dependence, with the p = 2 confidence obstruction scoped to the specified single-run averaging mechanism.Robust final aggregation and confidence amplification are outside this lower-bound claim.Source: §1 · p. 2
  • Every finite fixed threshold strictly increases asymptotic variance on a scalar Gaussian quadratic, while fixed whole-gradient clipping can shift the limiting target under asymmetric noise.Suitable fixed thresholds can reduce variance for some symmetric heavy-tailed laws; for p > 2 in the linear noise-centered setting, a sufficient growing-threshold window preserves the unclipped first-order limit.Source: §1 · p. 2
  • In the off-policy click-prediction experiment, the validation-selected clipping threshold improves upper-tail target error at an intermediate budget but loses to unclipped SGD at a larger budget.Importance weights remain intact, and the threshold is selected on separate source-validation runs and held fixed during evaluation.Source: §1 · p. 2
  • The paper positions its comparison as distinguishing when clipping’s confidence improvement affects the leading stochastic term versus a lower-order correction while tracking moment scale and clipping bias.It connects finite-sample accuracy-confidence trade-offs to the limiting target and covariance of the averaged estimator.Source: §1 · p. 2

2 Problem Setting and Concentration Tools

The section sets up projected convex optimization with conditionally noisy stochastic gradients and finite conditional p-th moments. It distinguishes the scales and information requirements of clipping rules, then compares concentration bounds for clipped and unclipped averages.

2 Problem Setting and Concentration Tools

  • The setting assumes a nonempty solution set, convex objective, and bounded distance from each iterate to a fixed optimizer.The distance bound is typical for bounded feasible sets, projections onto a known ball, or stopping-time arguments.Source: §2 · p. 2
  • Each stochastic gradient decomposes into a predictable subgradient and fresh conditional noise, whose conditional p-th moments are bounded for p ≥ 2.For p = 2, the p-moment scale is set equal to the second-moment scale.Source: §2 · pp. 2–3

2.1 Clipping and the relevant moment scales

  • Whole-gradient clipping requires moments of the full stochastic gradient, whose scale can exceed the centered-noise scale because it includes the optimization signal.At a boundary solution the objective gradient need not vanish, so the full-gradient moment scale may exceed the noise scale.Source: §2.1 · p. 3
  • Noise-centered clipping is an oracle benchmark because the stochastic gradient alone does not reveal its predictable signal and centered noise separately.Difference clipping instead uses an anchor and gradient estimate, with guarantees requiring anchor-accuracy and difference-moment conditions.Source: §2.1 · p. 3

2.2 Fuk–Nagaev decomposition and clipping

  • For p > 2, the unclipped deviation bound separates a variance-controlled Gaussian term from a rare-increment correction that is lower order than N^-1/2 at fixed confidence.The cited boundary identifies when the heavy-tail correction is no larger than the Gaussian term.Source: §2.2 · p. 3
  • The clipped-average bound has variance, bounded-increment, and clipping-bias terms; balancing the latter two yields logarithmic confidence dependence.For noise-centered clipping the correction uses the noise moment scale, while whole-gradient clipping uses the full-gradient scale, which includes signal.Source: §2.2 · p. 4
  • At p = 2, Chebyshev gives a polynomial-confidence bound for an ordinary average, whereas the noise-centered clipped bound has logarithmic confidence dependence.The cited rare-outlier lower-bound result shows the ordinary empirical mean’s δ^-1/2 dependence cannot be uniformly removed under a variance bound alone.Source: §2.2 · p. 4

3 Finite-Sample Optimization Guarantees

The section derives finite-sample guarantees for projected convex SGD and extends the comparison to strongly convex quadratics. The bounds separate Gaussian and heavy-tail contributions, with the comparison depending on confidence, moment scale, and conditions that absorb squared-gradient remainders.

3 Finite-Sample Optimization Guarantees

  • Projected U-SGD and whole-gradient Clip-SGD use a common projected recursion, under convexity, bounded iterates, and the stated moment assumptions.The deterministic inequality and martingale bounds yield optimization, Gaussian, and heavy-tail contributions, with an additional squared-gradient remainder.Source: §3 · p. 4

3.1 Second-moment calibrations

  • At p = 2, the confidence-dependent U-SGD calibration and the whole-gradient clipping calibration give high-probability bounds, while the clipping correction retains a slower N^-1/2 rate.The supplied passages do not provide the full displayed bounds’ numeric expressions.Source: §3.1 · p. 5

3 Finite-Sample Optimization Guarantees

  • For p > 2, the calibrated U-SGD and whole-gradient Clip-SGD bounds share optimization and Gaussian terms, but their heavy-tail corrections differ in moment scale and confidence dependence.The unclipped correction uses σp and δ^-1/p; the clipped correction uses Gp and logarithmic confidence dependence.Source: §3.2 · p. 5, §3 · p. 5

3.2 Comparison and sample requirements

  • The correction comparison requires sufficient conditions to absorb squared-gradient remainders; outside the stated regime, sample requirements must retain those contributions.Under the stated conditions, the unclipped correction is no larger in order when the accuracy-confidence boundary is met.Source: §3.2 · p. 6
  • The unclipped heavy-tail confidence dependence also appears in an exact quadratic Polyak–Ruppert recursion, but the claim does not rule out independent runs combined by robust aggregation or confidence amplification.The cited recursion supports a specific single-run mechanism rather than all possible procedures.Source: §3.2 · p. 6

3.3 Extension to strongly convex quadratics

  • For unconstrained strongly convex quadratics, the epoch-averaged extension separates deterministic contraction, Gaussian variance, and a heavy-tail correction.At fixed p > 2 and confidence, the correction is lower order as target error decreases, so ordinary SGD and the noise-centered benchmark have the same leading accuracy dependence; at p = 2, clipping’s confidence improvement affects the leading statistical term.Source: §3.3 · p. 6
  • The clipped quadratic guarantees also cover certified inputs, while difference clipping requires controlled anchor error and difference moments.The corresponding construction includes additional oracle work.Source: §3.3 · p. 6

4 Polyak–Ruppert Asymptotics

The section compares limiting behavior under fixed and growing clipping thresholds. Fixed clipping can alter the target and covariance, while a suitable growing threshold in the linear noise-centered setting preserves the unclipped first-order limit.

4 Polyak–Ruppert Asymptotics

  • For unclipped quadratic SGD under the stated step-size and moment-decay conditions, the averaged error is, to first order, the empirical mean of gradient noise transformed by −H^-1.The limit also requires the stated martingale central-limit conditions.Source: §4 · p. 6
  • With a fixed threshold, clipping can change both the limiting target and covariance, and a shifted stable root produces persistent bias.The fixed-threshold clipped limit is characterized by the clipped mean field’s root and the covariance of clipped gradients.Source: §4 · p. 7
  • For symmetric noise in the scalar model, the target is preserved, but every finite threshold strictly increases asymptotic variance for nondegenerate Gaussian noise.Suitable finite thresholds can reduce variance for some symmetric heavy-tailed distributions.Source: §4 · p. 7
  • In the linear additive-noise model with p > 2, a growing noise-centered threshold can make clipping bias and bounded-increment concentration negligible on the N^-1/2 scale and preserve covariance H^-1ΣH^-1.This sufficient result uses a threshold throughout the run; online thresholds and whole-gradient clipping require separate analysis.Source: §4 · p. 7

5 When Does Clipping Help?

The comparison asks when the heavy-tail correction is no larger than the Gaussian component at a requested accuracy and confidence. For p>2, the bounds eventually share a Gaussian leading order; at p=2, clipping’s confidence improvement affects the leading statistical term.

5 When Does Clipping Help?

  • For p>2, at N ≍ ρ^-2ℓδ, the heavy-tail corrections are at most of Gaussian order under the stated accuracy-normalized condition.Here ρ is the normalized accuracy for the relevant convex or strongly convex quadratic setting; remaining optimization terms require separate control.Source: §5 · p. 7
  • When pρ^(p-2) tends to zero, the bounds eventually share a Gaussian leading order, leaving balanced clipping to change a lower-order correction.At p=2, clipping’s confidence improvement affects the leading statistical term.Source: §5 · p. 7

6 Off-Policy Click Prediction

The experiment tests whether a source-validation-selected gradient-clipping threshold remains beneficial as the optimization horizon grows, while leaving importance weights intact. The frozen clipped method improves target error at an intermediate budget but later loses to unclipped SGD.

6 Off-Policy Click Prediction

  • The study uses the full OBD all-campaign log, with BTS as behavior policy and Uniform Random over 80 items as target policy; importance weights are not truncated.The experiment uses 12,357,200 BTS impressions and a separately collected Random-policy log of 1,374,327 impressions.Source: §6 · p. 7
  • The structured Brier click scorer gives an exact quadratic objective, and the independent Random-policy log supplies target-policy evaluation risk.The comparison is between projected U-SGD and fixed whole-gradient clipping; all methods return the complete post-update Polyak–Ruppert average.Source: §6 · p. 8
  • At the threshold-selection horizon, the frozen clipped method has ratio RClip = 0.63 with paired 95% interval [0.60, 0.65] on fresh evaluation trajectories.The selected threshold λval = 5 was chosen on separate source-validation runs and held fixed across evaluation horizons.Source: §6 · p. 8
  • At N = 33554432, the clipped-to-unclipped target-error ratio is 1.89 [1.81, 1.95], after the frozen clipped and unclipped curves cross.The first displayed checkpoint after which the pointwise paired lower bound remains above one is N = 8388608; the result is consistent with the shifted mean-field target becoming the dominant error.Source: §6 · p. 9
  • At N = 4194304, the smallest error in the displayed finite threshold grid occurs at λ = 10, with ratio 0.67 [0.64, 0.70] relative to U-SGD.The threshold sweep is a separate diagnostic, not the tuning procedure.Source: Figure 1 · p. 8, §6 · p. 9

7 Conclusion

The conclusion is that clipping’s benefit depends on the requested accuracy, confidence, and clipping bias. The bounds, exact examples, and off-policy experiment support comparing policies at the intended budget while accounting for confidence gains and effects on the target.

7 Conclusion

  • For p>2, unclipped Polyak–Ruppert averaging enters a finite-sample Gaussian-dominated regime, so clipping need not improve the leading order.Under a finite second moment alone, robustification improves the confidence dependence of the ordinary untrimmed average, and bias-balanced clipping gives logarithmic confidence bounds.Source: §7 · p. 9
  • Fixed whole-gradient clipping can attenuate the signal, shift the limiting target under asymmetric noise, and change asymptotic efficiency.In the linear noise-centered setting, a growing threshold can recover the standard first-order Polyak–Ruppert limit.Source: §7 · p. 9

Guide to the Appendices

The appendices collect concentration tools, crossover calculations, and lower constructions, then provide proofs and supporting results across the paper’s optimization, asymptotic, and experimental analyses.

Guide to the Appendices

  • The appendix guide assigns concentration tools, crossover calculations, and lower constructions to Appendix A, with proofs and supporting results following the main statements or in corresponding proof sections.The appendices also cover projected convex optimization, quadratic epochs and restarts, Polyak–Ruppert limits, the OBD protocol, and accelerated-method proof targets.Source: Guide to the Appendices · p. 11, Appendix Contents · p. 11

A.1 A martingale Fuk–Nagaev bound

  • The martingale Fuk–Nagaev bound separates Gaussian-scale accumulation from a power-term contribution corresponding to a single large increment.For fixed confidence, the heavy-tail term for equal-weight empirical averages is lower order in N exactly when p>2; its dependence is sharp for the empirical-mean mechanism, but not uniformly over classes with both moments fixed.Source: §A.1 · pp. 11–12
  • At p=2, Chebyshev’s inequality gives the endpoint bound, and under a variance bound alone the δ^-1/2 dependence cannot be uniformly replaced by logarithmic dependence for a single empirical average.The limitation is stated to hold without a stronger assumption or robust transformation.Source: §A.1 · p. 12

A.2 Bias and concentration after clipping

  • Clipping centered noise replaces the raw moment scale Mp with the central scale σp.The appendix derives clipped-average concentration by combining bias and variance bounds with bounded-increment martingale concentration.Source: §A.2 · pp. 12–13
  • Clipping controls both conditional bias and concentration; balancing the non-Gaussian terms uses λ ≍ Mp(N/ℓδ)^(1/p).For noise-centered clipping, Mp = σp; for whole-gradient clipping, the valid uniform choice is Mp = Gp.Source: §A.2 · p. 13

A.3 Crossover boundaries and their interpretation

The appendix identifies when the unclipped heavy-tail correction is no larger than the Gaussian term, and gives a matching lower construction for untrimmed averaging. Its proofs also establish projected-SGD bounds, with different moment scales for whole-gradient and centered-noise clipping.

A.3 Crossover boundaries and their interpretation

  • For fixed confidence and moment ratio, the Gaussian-dominance condition eventually holds as N grows when p > 2, but increasing N does not improve the comparison at p = 2.When p > 2, the unclipped correction’s importance relative to the Gaussian core decays with N; balanced clipping can still improve the correction in the very-high-confidence region.Source: §A.3 · p. 14

A.4 The endpoint p = 2 and matching lower constructions

  • At p = 2, a one-outlier construction makes the ordinary-mean heavy-tail term sharp for untrimmed Polyak–Ruppert SGD on a scalar quadratic.The lower-bound statement does not cover median-of-means, Catoni-type final aggregation, or confidence boosting across independent runs.Source: §A.4 · p. 15

B Proofs for Convex Optimization

  • The projected-SGD analysis uses a common inequality that accounts for initialization, predictable bias, noise, and squared-gradient contributions.The proofs distinguish whole-gradient clipping, whose bias bound uses full-gradient moments, from centered-noise clipping as an oracle benchmark.Source: §B.1 · p. 16

B.1 The common projected inequality

  • For U-SGD, the projected inequality is paired with concentration bounds on noise and squared-gradient terms to obtain the stated high-probability guarantee.The proof combines three concentration events with total failure probability at most δ.Source: §B.2 · pp. 16–17

B.2 Proof of Theorem 3.1

  • The proof of Theorem 3.1 combines the projected inequality with bounds on martingale terms and squared noise to establish its arbitrary-stepsize guarantee.The provided passages describe the proof’s concentration steps rather than the full theorem statement.Source: §B.2 · pp. 16–17

B.3 Proof of Theorem 3.2

  • Theorem 3.2’s proof controls clipping bias and concentration separately before combining them in the projected bound.It uses the fact that clipping does not increase the gradient norm and bounds the squared-gradient term by Bernstein–Freedman concentration.Source: §B.3 · p. 18, §B.5 · pp. 19–20

B.4 Proof of (19)

  • The proof of (19) applies Chebyshev’s inequality to the scalar martingale differences before substituting the resulting bound into the projected inequality.The supplied passages do not state the complete bound’s formula.Source: §B.4 · p. 19

B.5 Proof of (20)

  • For the p = 2 clipping bound, the proof uses the raw second-moment scale G2, regardless of the stated value of σ2.It then chooses a threshold and substitutes it into the general clipping bound.Source: §B.5 · pp. 19–20

B.6 Remainder conditions and sample requirements

The appendix states remainder conditions for comparing finite-sample bounds, then develops an epoch-averaged scheme for strongly convex quadratics. Its analysis specifies the oracle assumptions, burn-in and averaging procedure, and concentration tools for unclipped and certified clipped inputs.

B.6 Remainder conditions and sample requirements

  • The sample requirements yield the stated conditional comparison only when the remainder conditions hold; otherwise the full bounds (16) and (17) must be used.The appendix identifies the heavy-tail correction scales and gives sufficient budgets for controlling the optimization, Gaussian, and correction terms.Source: §B.6 · pp. 20–21

C Epoch-Averaged SGD on Strongly Convex Quadratics

  • The quadratic appendix analyzes an explicit unprojected epoch scheme under conditional noise bounds, allowing noise to depend on the iterate and across steps.Its supplied directions include ordinary SGD and a noise-centered oracle benchmark that clips noise while leaving the exact mean gradient outside the clipping map.Source: C Epoch-Averaged SGD on Strongly… · p. 21, §C.1 · p. 21

C.1 Quadratic model and oracle

  • The quadratic model has a unique minimizer and assumes strong convexity and smoothness, with conditional mean-zero noise along the entire unprojected run.For certified inputs, the centered component is distinguished from its predictable bias, which remains in the bounds.Source: §C.1 · p. 21

C.2 Epoch algorithm and restarts

  • Each epoch burns in for its first b steps, averages the remaining iterates, and uses geometric restarts with allocated failure probabilities.The epoch length must be at least ⌈4κ⌉.Source: §C.2 · p. 22

C.3 Concentration and clipping

  • The one-epoch analysis combines operator-weighted concentration with clipping-bias control to bound the next point’s error for unclipped and certified inputs.Noise-centered clipping is certified with variance scale σ2 and moment scale σp; at p = 2, the unclipped bound uses a second-moment argument.Source: §C.3 · pp. 23–25

C.4 Epoch lengths and restart complexity

The quadratic scheme uses epoch lengths and restarts to reduce the objective gap under finite-moment or certified-input conditions. The appendix also gives a difference-clipping certificate and analyzes fixed-threshold Polyak–Ruppert behavior, including variance and target-shift costs.

C.4 Epoch lengths and restart complexity

  • For p > 2, the epoch schedule halves the objective gap with probability at least 1 − δ, while its heavy-tail terms are lower order than the variance term for fixed confidence.At p = 2, both statistical terms have order 1/ε and their confidence dependence distinguishes the displayed bounds; these are not minimax comparisons with all robust estimators or multi-run aggregations.Source: §C.4 · pp. 25–26

C.5 Difference clipping and its oracle cost

  • Difference clipping is certified only when the anchor accuracy and coupled gradient-difference moment bounds meet the stated conditions.Its total oracle cost includes anchor construction and refreshes, plus any paired evaluations beyond the core updates.Source: §C.5 · pp. 26–27

D Polyak–Ruppert Asymptotics: Assumptions and Proofs

  • The unclipped Polyak–Ruppert average has an exact decomposition, and under the stated stepsize and moment-decay assumptions its remainder is negligible on the root-N scale.The resulting Gaussian limit follows from a martingale central limit theorem and Slutsky’s theorem.Source: §D.1 · pp. 27–28

D.1 The exact linear identity and the unclipped limit

  • The exact Polyak–Ruppert decomposition is pathwise and requires neither symmetry nor stability of the drift matrix.Under the additional stepsize and moment-decay assumptions, the remainder is o(N^-1/2).Source: §D.1 · pp. 27–28

D.2 Fixed-threshold clipped stochastic approximation

  • With fixed-threshold clipping, the averaged iterate’s root-N limit is centered at the clipped mean-field root xλ, not necessarily the optimizer.If hλ(x⋆) ≠ 0, the root-N-scaled error centered at the optimizer is not tight.Source: §D.2 · pp. 29–30

D.3 A quadratic counterexample for fixed whole-gradient clipping

  • For asymmetric noise on a scalar quadratic, fixed whole-gradient clipping can shift the equilibrium, and the iterates and their arithmetic averages converge to that shifted point.The example uses a fixed threshold between the two noise magnitudes.Source: §D.3 · p. 30

D.4 Symmetric noise and the covariance comparison

Under symmetric noise, every finite clipping threshold strictly increases asymptotic variance for scalar Gaussian noise, though sufficiently small thresholds can reduce it for some symmetric heavy-tailed laws. The appendix also gives sufficient growing-threshold conditions, quantifies clipping’s deterministic transit cost, and states limits for extending its constant-step covariance analysis.

D.4 Symmetric noise and the covariance comparison

  • For scalar Gaussian noise, every finite threshold gives asymptotic variance strictly above the unclipped value σ^2/µ^2; the variance converges to that value as λ grows.The comparison assumes a continuous symmetric noise law with finite variance and the stated regularity and stability conditions.Source: §D.4 · p. 31
  • For symmetric noise with density continuous and positive at zero, sufficiently small positive thresholds can give lower asymptotic variance than no clipping.The Polyak–Ruppert interpretation relies on the regularity and stability assumptions of Corollary D.6.Source: §D.4 · p. 31

D.5 Growing thresholds in the linear additive model

  • Sufficient growing-threshold conditions yield the stated high-probability bound; for ρ_N = δ_N, the required condition is δ_N N^(1−γ) → ∞.For bounded or polylogarithmic ℓ_N, a sufficient threshold is λ_N = N^β with 1/[2(p−1)] < β < 1/2; at p = 2 these exponents meet.Source: §D.5 · pp. 32–33

E.1 The deterministic transit cost of clipping

  • Whole-gradient clipping changes deterministic convergence from geometric contraction to a linear transit phase when the iterate starts above λ/µ.Difference clipping avoids this cost only after controlling its anchor error, difference-moment certificate, refresh schedule, and auxiliary oracle requirements.Source: §E.1 · p. 34

E.2 The O(η) covariance correction

  • The constant-step averaged process has a multivariate CLT with asymptotic covariance given by the stated expansion, under the theorem’s i.i.d., moment, independence, initialization, and contraction conditions.The expansion has an O(η^2) remainder, uniform for sufficiently small positive η.Source: §E.2 · p. 34

E.2.1 Proofs for the covariance result

  • The local clipped-score substitution identifies a multiplicative-noise contribution to the O(η) correction, but other terms prevent it from being a full nonlinear clipped expansion.A full expansion requires a stationary-bias theorem and Poisson-equation or Markov-chain analysis uniform in the clipping parameter.Source: §E.2 · p. 35

F Experimental Protocol

  • The appendix identifies its OBD experiment material as a protocol appendix; the supplied passage does not state protocol details.Source: F Experimental Protocol · p. 36

F.1 Source-only selection and paired evaluation

The OBD experiment separates source-only threshold selection from paired evaluation on fresh trajectories and reports uncertainty intervals and a tail diagnostic. The accelerated-method discussion offers a weighted regime map as a hypothesis, while identifying unresolved proof and implementation requirements.

F.1 Source-only selection and paired evaluation

  • The clipping threshold is selected on separate source-validation trajectories, frozen, and evaluated against unclipped SGD on fresh paired trajectories.A separate predeclared-grid sweep is used only as a threshold-sensitivity diagnostic.Source: §F.1 · p. 36, §F.2 · p. 36

F.2 Uncertainty and threshold sensitivity

  • Figure 1’s pointwise intervals are distribution-free 95% order-statistic intervals conditional on the fixed logs; paired-bootstrap intervals quantify the reported error ratios.Source: §F.2 · p. 36

F.3 Tail diagnostic

  • Figure 1(a) reports that observations at or above the empirical 0.999-quantile of gradient norm account for 99.6% of its empirical second moment.The plot’s vertical value is the fraction of observations whose weight or gradient norm exceeds the horizontal value times its median.Source: §F.3 · p. 36

G Directions for Accelerated Methods

  • The accelerated-method discussion says a direct counterpart of the paper’s three-level comparison is still missing and records weighted quantities a future theorem would need to control.Source: G Directions for Accelerated Methods · p. 36

G.1 Weighted Nagaev geometry

  • Momentum can concentrate stochastic weights, reducing N_p while increasing maximum influence; the proposed accelerated regime map is conditional on a predictable representation.For p > 2, the hypothesis predicts matching leading clipped order only in the stated region; outside it clipping may improve confidence dependence while attenuating signal.Source: §G.1 · p. 37

G.2 Thresholds, centering, and first-order efficiency

  • Concentrated accelerated weights can shrink or eliminate the sufficient threshold interval; a difference-clipped method also needs a charged anchor estimator and control through later momentum states.The discussion identifies predictable representation after momentum, clipping, and localization as the hard step.Source: §G.2 · p. 37, §G.1 · p. 37
Loading 2609.32962v1…