PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating Population-Risk Curves Along Nonconvex Gradient Flows from the Training Sample

Mingzhi Song

arXiv:2608.30261v1stat.MLcs.LG

TL;DR

Estimating a fitted model’s population-risk curve from its training sample avoids setting observations aside for validation, while exact leave-one-out requires n additional training trajectories. The paper introduces Flow-ALO and establishes a high-probability finite-sample comparison to the realized full-sample flow’s conditional risk, uniformly over compact training times and fixed-dimensional hyperparameters.

  • Problem

    The paper asks whether the fitting sample can estimate a fitted model’s population-risk curve without external validation, given that exact leave-one-out requires n additional training trajectories.

    Source: §1 · p. 1, §2 · p. 6

  • Method

    Flow-ALO propagates a deletion response along the full path, averages omitted losses at response paths, and replaces the unknown secant operator with the deleted Hessian on the observed path.

    Source: §1.2 · p. 3, §3 · p. 7

  • Results

    The paper establishes a high-probability finite-sample comparison between the Flow-ALO score and the realized full-sample flow’s sample-conditional risk, uniformly over compact training times and fixed-dimensional hyperparameters.

    Source: §1.1 · p. 1

  • Takeaways & Limitations

    Under the stated comparison, a measurable approximate Flow-ALO minimizer has bounded excess risk.

    Source: §1.1 · p. 1

    The long-time obstruction is demonstrated for a deterministic balanced-sample pitchfork; an iid fixed-confidence lower bound requires a separate argument.

    Source: §5 · p. 10

Abstract

from arXiv · show

We estimate the conditional population-risk curve of a realized smooth nonconvex gradient flow from the training sample. Flow approximate leave-one-out (Flow-ALO) propagates a deletion response and evaluates omitted observations at approximate deleted paths. The risk-curve error decomposes into response approximation, exact-LOO fluctuation, and deletion-to-full risk transfer. On each fixed finite horizon, bounded centered training-loss gradients, a one-sided Hessian lower bound, locally Lipschitz Hessians, and a strict tube-closure condition yield an explicit $(n-1)^{-2}$ bound for the deletion-response error. Bounded evaluation-loss gradients transfer the deletion-response bound to the score without requiring the Hessian to be invertible. Direct first-order jackknife cancellation and exact-LOO concentration control deletion-to-full risk transfer and fluctuation, respectively, completing recovery of the conditional population-risk curve. For bounded smooth two-layer mean-field networks training both layers, the score-error bound is uniform in width.

1. Introduction.

The paper asks whether the training sample can estimate the sample-conditional population-risk curve of a realized smooth nonconvex gradient flow without external validation, and develops Flow-ALO with response-approximation, concentration, and deletion-to-full analyses.

1.1. Contributions.

  • Theorem 7.6 compares Flow-ALO with the realized full-sample flow’s sample-conditional risk with high probability, uniformly over predetermined compact training-time and hyperparameter sets, without external validation.The paper also gives excess-risk bounds for measurable approximate Flow-ALO minimizers; absolute exact-LOO concentration recovers risk levels, while chained concentration identifies the curve up to a common sample-dependent shift.Source: §1.1 · p. 1
  • With deleted-objective Hessians uniformly lower-bounded, locally Hessian-Lipschitz, bounded centered forcing, and strict tube closure, Flow-ALO scores have uniform order-(n-1)^-2 error relative to exact LOO under Assumption 4.1 and evaluation-gradient bound (27).Source: §1.1 · p. 2
  • Exact-LOO concentration controls deletion-risk fluctuations, while first-order cancellation and a uniform second-derivative bound yield order (n-1)^-2 deletion-to-full transfer.Theorem 7.4 gives a uniform high-probability absolute bound; Theorem 7.5 instead controls anchored increments up to a common sample-dependent shift.Source: §1.1 · p. 2
  • For bounded smooth two-layer mean-field networks training both layers, an O(n^-1) dynamic-kernel difference with width-uniform constants yields an O(n^-2) Flow-ALO score error and a finite-width-family risk-curve bound.The result applies on each common finite horizon and at each common finite width.Source: §1.1 · p. 2

1.2. Problem formulation.

  • Flow-ALO propagates a linear response along the full path and averages omitted losses at response paths; the target risk integrates the realized full fit against P0 with the sample held fixed.The formulation also defines the analogous deletion-fit risk.Source: §1.2 · p. 3

1.3. Related work.

  • Unlike endpoint ALO [39] and iterative full-batch-GD approximate CV [25], this work targets population risk along a nonconvex continuous-time path rather than fitted endpoints or exact iterative LOO scores.The cited iterative-GD analysis gives approximate-to-exact LOO error bounds but no population-risk concentration or oracle for the selected iterate.Source: §1.3 · pp. 3–4

2. Setup and targets.

The setup defines measure-based training objectives and gradient-flow paths sharing a deterministic initialization. Its primary target is the uniform error of the training-sample risk-curve estimator relative to the conditional population-risk curve, with or without a common sample-dependent shift.

2. Setup and targets.

  • Training objectives are defined for admitted mass-one finite signed measures when the training loss is measurable and integrable, and population evaluation losses are integrable under the population distribution.The evaluation loss may be expressed through one common measurable evaluation rule.Source: §2 · p. 5
  • The full and deleted flows share the deterministic initial parameter and are defined over the predetermined finite horizon.For each hyperparameter, the full-sample trajectory is denoted by θλ.Source: §2 · p. 6
  • The primary target is the uniform error of the training-sample risk-curve estimator relative to the population-risk curve, measured either in absolute level or up to one common sample-dependent shift.Approximate minimization over the candidate set is a consequence of this target.Source: §2 · p. 6

3. Individual-deletion response and Flow-ALO.

Flow-ALO approximates an individual deletion response using the deleted Hessian along the full-sample path and propagates a centered-Hessian correction throughout. Its same-sample score retains higher-order effects, including when the training-point evaluation gradient vanishes at interpolation.

3. Individual-deletion response and Flow-ALO.

  • Proposition 3.1 characterizes exact deletion dynamics through a secant operator obtained by integrating the deleted-objective Hessian along the segment between paths.It assumes both flows exist and the deleted objective is twice continuously differentiable on the relevant line segments.Source: §3 · p. 6
  • Flow-ALO replaces the unknown secant operator with the deleted Hessian evaluated on the observed full-sample path, propagating a centered-Hessian correction throughout.The accumulated centered-Hessian propagation is termed resummation.Source: §3 · pp. 7–8
  • A simultaneous-stage forward-Euler discretization of Flow-ALO matches the full-batch IACV recursion after accounting for objective normalization.The cited recursion is from Luo, Ren and Barber [25, Section 2.3, Equation (12)].Source: §3 · p. 7
  • The conventional scaled empirical-weight path derivative obeys a full-Hessian influence ODE, whereas Flow-ALO incorporates the centered-Hessian correction during propagation.The empirical-weight path is differentiated at the empirical distribution Pn.Source: §3 · pp. 7–8
  • The same-sample Flow-ALO score retains quadratic and higher-order effects even when the training-point evaluation gradient vanishes at interpolation.This distinguishes it from a first-order loss correction.Source: §3 · p. 8

4. Deterministic approximation theory.

Section 4 develops a deterministic finite-horizon guarantee for the deletion response by localizing deleted and response paths within a full-path tube. It controls their approximation error under regularity and strict tube-closure conditions.

Deterministic approximation theory

  • Equation (14) approximates the exact deletion displacement by replacing the secant Hessian with B_lambda evaluated along the full path.Theorem 4.3 bounds the resulting error uniformly over the finite horizon.Source: §4 · pp. 8–9
  • Uniform Hessian and third-derivative bounds, together with a strict tube margin, keep deleted paths and responses inside the full-path tube for comparison.Lemma 4.2 localizes the exact and response curves there, enabling full-horizon comparison.Source: §4 · p. 8
  • Under Assumption 4.1, Theorem 4.3 gives a continuous-time finite-horizon counterpart under one-sided tube curvature, with dependence on Jκ(T) explicit.The parameter and score radii are dimension-independent whenever L3, M, Gout, and Jκ(T) are common scalar envelopes.Source: §4 · p. 9
  • The comparison radius in (29) is O(n^-2) when T, kappa, KC, L3, and M remain bounded; resummation removes the separate KC premise.The resummed response is also used for the width-uniform output-space verification in Section 6.Source: §4 · p. 9
  • When every deleted gradient is affine in theta, the approximation is exact on the common flow interval, independently of propagator size.Corollary 4.4 gives equality of the approximate and exact leave-one-out scores on the stated time-parameter domain.Source: §4 · p. 9

5. A long-time obstruction.

A one-dimensional smooth pitchfork shows that deletion perturbations can amplify over logarithmic time, obstructing long-time risk-curve control. In a deterministic balanced sample, deleted paths separate from the full path enough to create a fixed risk gap, motivating the theorem’s explicit finite horizon.

5. A long-time obstruction.

  • Example 5.1 constructs a one-dimensional pitchfork that exhibits logarithmic-time instability despite smoothness.The setup uses a deterministic sample with equal numbers of +1 and −1 observations.Source: §5 · p. 10
  • The full path stays at the unstable equilibrium 0, while deleting either sign changes the forcing by ∓(n −1)−1 and moves the deleted path at least ϑ by logarithmic time.The separation occurs at T λ n,ϑ.Source: §5 · p. 10
  • At O(log n) time, the balanced-sample example has a risk gap of at least ϑ^2/4, illustrating why Theorem 4.3 uses an explicit horizon.This is a deterministic balanced-sample statement; an iid fixed-confidence lower bound would require a separate random-sample argument.Source: §5 · p. 10

6. Exact output-space error representations.

This section replaces potentially width-dependent parameter-space derivative bounds with output-space dynamic-kernel control. An exact identity links prediction perturbations to kernel mismatch, and the resulting criterion is verified for two-layer mean-field networks uniformly in width.

Exact dynamic-kernel mismatch identity

  • Theorem 6.1 gives an exact output-space dynamic-kernel mismatch identity for prediction perturbations along the full and perturbed flows.The section characterizes the nonlinear deleted and response trajectories through this identity rather than relying on parameter-space derivative envelopes.Source: §6 · pp. 10–12
  • Sun and Valaee [35] reduce an endpoint-removal influence system for a ridge-regularized linearized model under convexity and rank conditions; here the identity compares nonlinear deleted and response trajectories.The section distinguishes this exact trajectory comparison from the earlier parameter-to-dataset–output dimension reduction.Source: §6 · p. 12

Output-space deletion bound

  • Theorem 6.2 turns bounds on the perturbation response and dynamic-kernel mismatch into an output-space deletion bound over specified times, regularization values, and evaluation inputs.Its premises require finite constants controlling the response and kernel difference, and can apply uniformly across deletions and evaluation inputs.Source: §6 · p. 12
  • The kernel-mismatch premise admits a factorwise check, with Ddyn = Jtr{GchEJ + JdEU}; a common Kdyn envelope instead permits Ddyn = 2Kdyn.Under the stated Corollary 4.4 conditions with T′ = T, the passage also permits Ddyn = 0.Source: §6 · p. 13

Two-layer mean-field verification

  • O(n^-1) finite-horizon dynamic-kernel mismatch holds for smooth two-layer mean-field networks with both layers trained, uniformly over width under the stated boundedness and flow-existence conditions.The bound’s constants depend on the horizon, regularization range, input and initialization bounds, and derivative envelopes, not width.Source: §6 · pp. 12, 14

7. Estimating population-risk curves from the training sample.

This section develops training-sample recovery results for population-risk curves, with a finite-sample bound separating response approximation, LOO fluctuation, and cross-size terms. It also gives uniform results for a deterministic finite-width family of two-layer mean-field models under finite-horizon regularity and deletion-chord conditions.

7.1. Second-order comparison of deletion and full-sample risks.

  • Under twice-differentiable deletion chords and averaged first-order cancellation, Theorem 7.1 bounds the realized deletion-to-full risk discrepancy at O((n−1)^−2).The curvature condition applies uniformly along deletion chords.Source: §7.1 · pp. 15–16
  • When sample variance is O(1), the quadratic mean-flow risk discrepancy is O(n−2) uniformly on fixed horizons, and second- and fourth-moment bounds cover unbounded quadratic loss.The example and corollary require no global loss or stability bound under their moment conditions.Source: §7.1 · pp. 16–17

7.2. Concentration of the exact LOO path.

  • Theorems 7.4 and 7.5 provide uniform concentration for exact-LOO levels and chained relative concentration for anchored curve increments.For single-model flows, finite-horizon sufficient conditions yield βn−1 = O((n−1)−1) and ωn−1(r) = O(r).Source: §7.2 · pp. 19–20

7.3. Training-sample recovery of population-risk curves.

  • Theorem 7.6 gives a high-probability finite-sample curve bound that separates response approximation, exact-LOO fluctuation, and cross-size terms, permitting absolute recovery or recovery up to a common shift.The bound applies on a predetermined compact continuous-time domain for a smooth, possibly nonconvex flow.Source: §7.3 · pp. 20–21
  • Corollary 7.7 gives observable time-zero centering and bounds the excess risk of a measurable approximate Flow-ALO minimizer by 2rn(δ) + ζn.The centering result applies when the chained route uses a deterministic time-zero anchor and flows share a common deterministic initialization.Source: §7 · p. 15, §7.3 · p. 21

7.3. Training-sample recovery of population-risk curves

  • Corollary 7.8 addresses two-layer mean-field risk curves for a deterministic finite-width family.The corollary considers configurations that differ in at most one coordinate.Source: §7.3 · p. 22
  • The evaluated loss lies in one deterministic interval of length B_T, uniformly over widths, sample sizes, parameters, paths, and evaluation points.The finite constant B_T depends only on the listed model and horizon quantities.Source: §7.3 · p. 22
  • Uniformly over widths and sample sizes, β_N,m and β_mix_N,m are each bounded by C/N, with locally Lipschitz Hessians and deletion-chord hypotheses of curvature C_PP,T.The Hessian modulus is ω_n−1(r) = C_hyp,T r; the constants are finite and deterministic.Source: §7.3 · p. 22

APPENDIX S.A: PROOFS FOR DELETION DYNAMICS AND DETERMINISTIC APPROXIMATION

The appendix proves bounds for deletion dynamics and their deterministic linear-response approximation using curvature, forcing, and Hessian regularity controls. It also establishes tube-based existence, an affine-gradient case, and a scalar example separating deleted and full paths.

Proof of Lemma 4.2

  • The linear response exists uniquely and has norm at most M(n−1)^−1ϕκ(T), by scalar comparison under curvature and forcing bounds.The response-norm argument uses an upper-right Dini derivative and extends the bound to the horizon endpoint by continuity.Source: Appendix S · p. 23
  • The deleted path exists throughout [0,T] when its displacement bound stays strictly inside the tube, preventing the stopping boundary from being reached.Local Lipschitz continuity gives a local solution, while the continuation criterion rules out termination before T.Source: Appendix S · p. 23

Proof of Theorem 4.3

  • Subtracting (14) from (11), the derivation uses the integral Taylor formula and Hessian Lipschitz bound (22), takes norms, then a supremum over 1≤i≤n, 0≤t≤T, and λ∈Λ to obtain Equation (26).Source: Appendix S · pp. 24–25

Proof of Corollary 4.4

  • For affine deleted-objective gradients, the response initial-value problem has a unique solution throughout the stated interval [0,T′].Source: Appendix S · p. 25

Verification of Example 5.1

  • In the balanced scalar example, the full path stays at zero while every deleted path eventually has magnitude at least ϑ and remains bounded by 1.The deleted drift points inward at both endpoints of [0,1], ensuring existence for all time; below ϑ it has a positive growth rate.Source: Appendix S · pp. 26–27

APPENDIX S.B: PROOFS FOR THE OUTPUT-SPACE AND TWO-LAYER MEAN-FIELD RESULTS

The proofs establish output-space bounds by restricting scalar output weights while leaving hidden weights unrestricted, then use these bounds to obtain a width-independent two-layer mean-field score-error result.

APPENDIX S.B: PROOFS FOR THE OUTPUT-SPACE AND TWO-LAYER MEAN-FIELD RESULTS

  • The output-weight strip bounds each scalar output weight while leaving hidden weights unrestricted, enabling uniform Hessian bounds in RMS and block-maximum norms.The bounds are uniform over network width and the specified probability measures.Source: Appendix S · pp. 28–29
  • The response chord Jacobian has operator norm at most (n−1)^−1 A∆D+.The response and exact-deletion segments lie in the enlarged output-weight strip; A∆ is a finite deterministic deletion-displacement scale.Source: Appendix S · pp. 30, 32
  • The kernel formulas yield a width-independent uniform bound in Equation (48) and a kernel-mismatch constant for Equation (49).The bounds use the Jacobian and propagator envelopes along the full and response paths.Source: Appendix S · p. 33
  • Equations (50) and (51) follow with Atr = Aev = L1 and kernel-difference scale Ddyn = Cker,T/(n−1).The evaluation domain is Xev = {x ∈ X : ∥x∥ ≤ X}.Source: Appendix S · p. 33

APPENDIX S.C: PROOFS FOR POPULATION-RISK-CURVE RECOVERY AND SELECTION

The appendix develops proofs for population-risk-curve recovery and selection, deriving deletion-to-full risk control and exact-LOO concentration, establishing sufficient curvature conditions via continuous measure responses, verifying response behavior in Example 7.2, and proving risk-transfer and measurable near-minimizer results for approximate selectors.

S.C.1. Deletion-to-full risk and exact-LOO concentration

  • O((n-1)^-2) deletion-to-full risk error follows from cancellation of the averaged linear Taylor terms and deletion directions’ total-variation norm bound of 2.The proof applies Taylor’s formula along each deletion chord and bounds the remaining curvature contribution.Source: Appendix S · p. 33
  • In Example 7.2, the response equals the exact deleted-path displacement, so held-out losses agree term by term and the scores coincide.Source: Appendix S · p. 34

S.C.1.1. Continuous measure responses

  • For continuous measure perturbations, twice-differentiable affine flow responses that commute with time differentiation and satisfy uniform response-gain bounds yield a sufficient deletion-chord curvature condition.The corollary’s conditions support the curvature bound used in deletion-to-full risk control.Source: Appendix S · pp. 35–36

S.C.1.2. Exact-LOO concentration

  • For Theorem 7.4, the process has expectation zero, and each other LOO-score and population-risk summand changes by at most βn−1/n; for n>0 and x>0, McDiarmid’s bounded-differences inequality applies [26, Lemma 1.2].Source: Appendix S · p. 38
  • For Theorem 7.5, a bounded-difference increment MGF and finite-level chaining control the supremum over H, anchored at zero contrast at (t0,λ0).The increment argument uses McDiarmid’s method [26].Source: Appendix S · pp. 39–42

Proofs for population-risk-curve recovery and selection

  • For Theorem 7.6, both the absolute and chained routes provide measurable events with probability at least 1−δ.Source: Appendix S · p. 43
  • RS( bt, bλ) ≤ RS(t, λ) + 2rn(δ) + ζn bounds the risk of an approximate selector by the target risk plus two error terms.The inequality follows from the displayed comparison of the empirical criterion and the selector’s approximation error.Source: Appendix S · p. 43
  • If the criterion is jointly measurable and samplewise continuous on a compact metric space, a measurable selector within deterministic tolerance ζ exists.The construction takes the first point in a deterministic dense sequence whose criterion is within ζ of the infimum.Source: Appendix S · p. 44
  • With labels in {−1,1}, tanh activation, and logistic outer loss, condition (47) holds.Source: Tanh–logistic example · p. 44

Tanh–logistic example.

For the tanh–logistic specification, the section gives fixed-horizon Flow-ALO risk-curve error and a contraction-based condition for a horizon-independent risk-curve radius. It also establishes finite-horizon stability, parameter-modulus, and mixed-increment bounds for gradient-flow evaluation losses under the stated neighboring-sample, candidate-index, smoothness, and flow-containment conditions.

Tanh–logistic example

  • Under Corollary 7.8’s support, initialization, and measurability hypotheses, (88) is O(n^-1/2) + O(n^-2) at fixed horizon and confidence for fixed-in-n width and candidate families.The tanh–logistic specification has (S0,S1,S2,S3) = (1,1,2,6), (L1,L2,L3) = (1,1/4,1/4), cφ = log 2, and L0 = 0, meeting the smoothness premise.Source: Tanh–logistic example · p. 44
  • The tanh–logistic proof supplies width-uniform replace-one stability βN,m ≤ Cβ,T/N and places evaluated losses in a common deterministic interval of length BT.The bounds are uniform in width m and sample size N; the interval is [cφ − L1S0AT, cφ + L0 + L1S0AT].Source: Tanh–logistic example · p. 45
  • Direct cancellation at Pn and the simultaneous bounds on Flow-ALO–LOO, LOO–deletion-risk, and deletion–full-risk differences yield (88).The cancellation follows from the linear-functional sufficient condition; taking the maximum over widths and supremum over Hm gives the stated result.Source: Tanh–logistic example · p. 50

S.C.3. Uniformly contractive risk curves

  • Under the uniformly contractive conditions, Theorem S.C.2 gives a risk-curve radius independent of Tn on an event of probability at least 1 − δ.Positive curvature of every empirical and deletion-chord objective is a model-independent sufficient condition; the curvature requirement is global over the sample class because exact-LOO concentration uses uniform stability.Source: Tanh–logistic example · pp. 50–52

Finite-horizon stability

  • Removing the finite-net logarithmic factor requires the additional mixed-stability premise; positive curvature alone provides only ordinary stability.The stated finite-net bound retains a log n/n term alongside n^−2.Source: Tanh–logistic example · p. 54
  • At fixed horizon and uniform constants, β_(n−1)=O((n−1)^−1), ω_(n−1)(r)=O(r), and 2ω_(n−1)(n^−1/2)=O(n^−1/2).These bounds follow from the finite-horizon neighboring-flow and candidate-index conditions.Source: Tanh–logistic example · pp. 55, 58

Mixed increments

  • Uniform mixed-increment constants yield O(1/N) neighboring-sample evaluated-loss increments at fixed horizon.The mixed-increment result assumes smooth loss extensions on a common open set and flows that remain in a deterministic convex set.Source: Tanh–logistic example · pp. 58, 61

Related papers

Loading 2608.30261v1…