PaperlayerView exact revision on arXiv

Source-linked AI summary

When is early stopping beneficial under label noise? A phase diagram for kernel gradient flow

Fabrizio Boncoraglio, Lenka Zdeborová

arXiv:2610.33846v1cs.LGcond-mat.dis-nnstat.ML

TL;DR

The paper asks whether extensive label noise necessarily makes early stopping preferable, and studies that question through high-dimensional kernel gradient flow on a synthetic noisy classification task. It finds that kernel geometry can make interpolation optimal across the considered family and all training times, while the neural-network experiments leave the origin of analogous learned representations unresolved.

  • Problem

    The paper asks whether intrinsically unpredictable training labels necessarily make the best test error occur before interpolation.

    Source: §1 · pp. 1–2

  • Method

    The authors derive high-dimensional learning curves for unregularized kernel gradient flow in the RAF task, where labels combine a shared rule with input-independent labels.

    Source: §1 · p. 3

  • Results

    At a critical kernel geometry, the interpolating endpoint attains the lowest generalization error across the considered kernel families and all training times, despite extensive label noise.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    Within this setting, fitting unpredictable labels need not sacrifice generalization, and whether early stopping helps is controlled by representation geometry.

    Source: §5 · p. 12

    The kernel theory does not establish that neural networks learn the same geometry or explain how feature learning, architecture, and optimization produce the observed behavior.

    Source: §4 · p. 10, §5 · p. 12

Abstract

from arXiv · show

Early stopping is commonly used to prevent fitting label noise from degrading generalization. Does this mean that, when a positive fraction of the training labels is intrinsically unpredictable, the best test error must occur before interpolation? Experiments on a CIFAR10 classification task with randomly labeled examples motivate the opposite possibility: although fixed-feature and shallow models exhibit late-time deterioration of the test error, the need for early stopping essentially disappears in larger, deeper convolutional networks even after the random labels have all been classified correctly. We determine when late-time deterioration disappears entirely in a solvable setting. For kernel gradient flow on a synthetic noisy classification task, we derive sharp high-dimensional learning curves and obtain a phase diagram controlled by the balance between the linear kernel component that learns the shared relation between inputs and labels and the higher-order components that enable fitting the label noise. In one regime, generalization is optimized at finite time and early stopping is necessary. In another, generalization improves until interpolation, but the endpoint is suboptimal across kernels. At the boundary, the interpolating endpoint attains the lowest generalization error across the considered kernel family and all training times. Thus, even with extensive irreducible label noise, interpolation can outperform early stopping. The neural-network experiments leave open how learned representations acquire an analogous favorable geometry.

1 Introduction

The introduction challenges the expectation that unpredictable labels make early stopping necessary, using CIFAR10 experiments to motivate a theoretical comparison of stopping and interpolation. It proposes kernel gradient flow as a solvable setting where kernel geometry determines whether the best test error occurs at finite time or convergence.

1 Introduction

  • The CIFAR10 experiment uses cats and dogs for the shared classification task and fixed randomly labeled automobiles to track fitting unpredictable labels during training.Test error is measured on new cats and dogs, while memorization error tracks misclassified randomly labeled training automobiles.Source: §1 · p. 1
  • Larger, deeper convolutional and residual networks show little late-time test-error deterioration, with their best observed performance after random-label classification error reaches zero.Fixed-feature and smaller or shallower models exhibit the familiar deterioration as memorization error falls.Source: §1 · pp. 1–2
  • The theoretical question is whether training to interpolation can analytically attain the lowest test error, rather than merely whether an interpolating estimator generalizes despite label noise.The introduction distinguishes this trajectory comparison from benign overfitting and noiseless grokking.Source: §1 · p. 2
  • The RAF analysis follows an unregularized kernel-gradient-flow trajectory whose behavior depends on the balance between the rule-aligned kernel component and higher-order components that fit unpredictable labels.The work identifies a transition between finite-time and convergence-optimal test error that prior analyses did not characterize across kernels.Source: §1 · p. 3, Related work · p. 4
  • At a critical kernel geometry, the interpolating predictor attains the lowest generalization error across the considered kernel families and all training times, despite extensive label noise.Other regimes have either a finite-time minimum or improving error up to an interpolating endpoint that remains suboptimal across kernel geometries.Source: §1 · p. 3

2 Data & kernel dynamics

The paper studies a controlled noisy-label task with a shared rule and independently assigned labels, learned using fixed-kernel gradient flow. In the high-dimensional reduction, kernel components that support rule learning and noise fitting determine the trajectory’s generalization behavior.

2 Data & kernel dynamics

  • The RAF task separates a shared prediction rule from labels independent of the input, with a fixed fraction of randomized labels retained throughout training.This construction lets the analysis track fitting unpredictable labels separately from generalization of the shared rule.Source: §2 · p. 4
  • The analysis measures rule generalization on fresh rule-labeled inputs and uses memorization error to diagnose fitting of unpredictable labels.In the kernel analysis, interpolation means zero empirical square loss, which is stronger than zero memorization error.Source: §2 · pp. 4–5
  • Kernel gradient flow uses a fixed kernel and takes gradients with respect to the function in its RKHS, rather than the dual coefficient vector.For a positive-definite kernel with zero ridge, the trajectory converges to zero training loss and an interpolating endpoint.Source: §2 · p. 5
  • Kernel linearization reduces the relevant geometry to an input-dependent component for learning the shared rule and a diagonal higher-order component for fitting unpredictable labels.Their relative strength governs whether continued training improves or harms generalization; overall scale sets the time scale.Source: §2 · p. 6

3 Learning curves and the early-stopping boundary

In the proportional high-dimensional limit, the paper derives generalization and memorization trajectories and identifies a sharp kernel-geometry threshold for whether the minimum test error occurs at finite time or convergence. For a fixed kernel, the boundary determines whether stopping benefits generalization, while the sample ratio affects the magnitude of that benefit.

3 Learning curves and the early-stopping boundary

  • The derived high-dimensional learning curves give complete generalization and memorization trajectories, while the finite-time-versus-convergence optimum is governed by one kernel-geometry parameter.Trajectory shape and time scale also depend on sample ratio, initialization, and overall kernel scale.Source: §3 · pp. 6–7
  • At interpolation, different kernel geometries can fit every training label exactly yet produce different limiting test errors.The limiting test-prediction filter depends on γ, so interpolation alone does not determine generalization.Source: §3 · p. 7
  • For zero ridge, generalization decreases to interpolation when γ ≤ γc(ε), but eventually rises and reaches its minimum at finite time when γ > γc(ε).This criterion holds for every positive sample ratio, fixed finite initialization scale, and random-label fraction 0 < ε < 1; the boundary depends only on ε.Source: §3 · p. 7
  • For γ = π/5, the stopping-benefit diagram has εc ≃ 0.2633: benefit is zero below and on the boundary, and finite-time stopping helps above it.The plotted colors show the cost of training to interpolation; the sample ratio changes that cost but does not move the boundary.Source: §3 · p. 8, Figure 2 · p. 8

4 Optimal interpolation at the critical geometry

The critical kernel geometry reaches the lowest generalization error across the considered kernels and training times at interpolation. Which stopping time is optimal depends on kernel angle, and the critical boundary is the only geometry whose interpolating endpoint attains the global bound.

4 Optimal interpolation at the critical geometry

  • At the critical geometry, interpolation attains the lowest generalization error across all considered kernel geometries and training times.No finite-time iterate for another unregularized kernel matches or improves on this endpoint.Source: §4 · p. 9
  • The critical angle γc(ε)/π ≃ 0.2550 is the unique global optimum; at larger angles, even finite optimal stopping remains strictly above it.The gap is more visible for δ0 = 1, and the plotted curves use α = 1.5, ε = 0.1, and λ = 0.Source: Figure 3 · p. 9, §4 · p. 9
  • For γ > γc(ε), generalization eventually deteriorates and is minimized at finite time; for γ ≤ γc(ε), it decreases until interpolation.Among the kernels that improve until interpolation, only the critical geometry reaches the global bound.Source: §4 · pp. 10, 12
  • At γ ≤ γc(ε), gradient flow and optimally tuned ridge share the interpolating endpoint, but only the critical kernel attains the bound.For γ > γc(ε), optimally stopped gradient flow is strictly worse than converged ridge at its optimal regularization.Source: §4 · pp. 11–12

5 Discussion and Conclusion

The noisy CIFAR10 experiments support the possibility that larger, deeper networks do not benefit from early stopping, while fixed-feature and shallower models show late-time deterioration. The kernel analysis gives a solvable fixed-representation counterpart, but does not explain how neural networks acquire analogous representations.

5 Discussion and Conclusion

  • Larger and deeper networks attain their best observed test performance after memorization error reaches zero, unlike fixed-feature and shallower models that show late-time deterioration.The observation persists across additional architectures and training recipes, including cross-entropy, data augmentation, and scheduled learning rates.Source: §5 · p. 12
  • Kernel gradient-flow learning curves show that geometry determines whether the lowest test error occurs at finite time or convergence.At the critical geometry, the interpolating predictor has the lowest test error across the considered kernel family and all training times.Source: §5 · p. 12
  • The fixed-representation theory does not establish that neural networks learn the same geometry or explain how feature learning, architecture, and optimization produce the observed behavior.How larger and deeper feature-learning architectures realize analogous behavior remains open.Source: §5 · p. 12

AI Use Statement

The authors report using generative AI to assist with mathematical derivations, literature searches, numerical-method feedback, and writing, while retaining responsibility for the final work.

AI Use Statement

  • Generative AI tools assisted with checking and polishing derivations, literature searches, numerical-method feedback, and writing.The authors state that they reviewed the AI-assisted work and take responsibility for the final content.Source: AI Use Statement · pp. 12–13

A From a nonlinear kernel to an effective linear predictor

The appendices reduce fixed nonlinear dot-product kernel flow to an effective linear predictor in the proportional high-dimensional limit, then use label moments to derive prediction errors.

A From a nonlinear kernel to an effective linear predictor

  • The analysis takes n, d →∞ at fixed n/d, with fixed kernel, regularization, initialization variance, and positive training time.The population inputs are spherical; general covariance is treated separately.Source: A From a nonlinear kernel to an… · p. 16
  • For spherical inputs, expanding the kernel around small off-diagonal overlaps yields a matrix approximation with constant and linear components and a diagonal correction.The kernel function is assumed smooth near zero and finite at one.Source: A From a nonlinear kernel to an… · p. 16
  • Labels split into a rule-correlated part and a residual; their covariance simplifies asymptotically, while the residual need not be Gaussian or independent across examples.The residual has zero cross-moment with the rule vector, and only its covariance and cross-moment enter the overlap and squared-norm calculations.Source: A From a nonlinear kernel to an… · pp. 17–18
  • At fixed positive time, the constant prediction vanishes, and removing the constant kernel component leaves the limiting generalization and memorization errors unchanged.The coefficient difference need not vanish; its contribution to training and new-input predictions does.Source: A From a nonlinear kernel to an… · pp. 19–20
  • In the high-dimensional limit, a new input sees the linear predictor vτ, while the diagonal kernel component still changes the fitted training coefficients.The diagonal component has no direct diagonal contribution to the test vector.Source: A From a nonlinear kernel to an… · pp. 18, 21
  • After rescaling time and fixing initialization variance, kernel trajectories depend on the ratio of linear to higher-order kernel components, summarized by angle γ.Kernels with the same angle have identical error trajectories in reduced time.Source: A From a nonlinear kernel to an… · pp. 16, 21

B Derivation of the order parameters’ equations

The appendix solves reduced eigenmode dynamics and converts their spectral averages into formulas for generalization and memorization, including the limiting factual-margin distribution.

B Derivation of the order parameters’ equations

  • Reduced eigenmode dynamics and label moments yield overlaps, prediction variances, and factual margins, which are then expressed through one-dimensional spectral integrals.The calculations follow standard square-loss learning-curve methods.Source: B Derivation of the order parameters’… · pp. 21, 23
  • The test error concentrates to Egen(τ) = π^-1 arccos(mτ/√qτ), where mτ measures rule alignment and qτ measures prediction variance.The formula uses the angle between the learned and rule weight vectors for a new spherical input.Source: B Derivation of the order parameters’… · pp. 22–23
  • The factual training margin has limiting mean and variance given by the order parameters, with the zero-eigenvalue atom included when α > 1.For α > 1, the diagonal kernel also fits training directions absent from the linear predictor on new inputs.Source: B Derivation of the order parameters’… · p. 24
  • A typical factual margin converges to a Gaussian law, and the empirical memorization error converges to its single-fact Gaussian tail probability.The concentration argument uses that the number of facts grows proportionally with n.Source: B Derivation of the order parameters’… · p. 26
  • At zero ridge and convergence, every factual margin tends to one, so memorization error is zero, while generalization still depends on kernel angle.At this endpoint h∞ = 1 and r∞ = 0.Source: B Derivation of the order parameters’… · p. 26

C Why the critical kernel separates the stopping regimes

The stopping boundary follows from how kernel geometry balances rule-aligned signal against prediction variance: it separates finite-time optima from convergence-optimal interpolation.

C Why the critical kernel separates the stopping regimes

  • At zero ridge, γ ≤ γc(ε) gives decreasing generalization error through convergence, whereas γ > γc(ε) yields a finite-time minimum followed by deterioration.The threshold holds for every sample ratio, finite initialization scale, and random-label fraction in the stated range.Source: C Why the critical kernel separates… · pp. 27–28
  • When β ≥ bc, the signal-to-variance balance improves through convergence; when β < bc, the slowest-relaxing low-eigenvalue modes eventually drive generalization error upward.The initial error decreases for every kernel angle, so the late increase places the minimum at a finite positive time.Source: C Why the critical kernel separates… · pp. 27–28
  • The stopping proof compares stationary candidates with endpoint errors; a finite observation window alone cannot establish that the optimum occurs at infinity.The argument does not assume a unique turning point.Source: C Why the critical kernel separates… · pp. 27–28
  • The critical kernel’s interpolating endpoint attains the lowest test error across kernel angles and training times, and no finite-time filter attains the same bound.Equality requires zero surviving initialization variance and a filter proportional to s/(s + bc).Source: C Why the critical kernel separates… · pp. 28–29
  • With positive ridge, β = bc can reach the same optimal test error when b < bc, but the resulting predictor no longer interpolates training scores.The optimality statements apply to the specified smooth dot-product kernel family and training trajectories.Source: C Why the critical kernel separates… · p. 29
  • Euclidean coefficient flow has the same stopping boundary as RKHS flow, although their learning curves and optimal times generally differ.Their converged empirical-risk-minimizer errors coincide because both flows converge to the same coefficients.Source: C Why the critical kernel separates… · p. 29

D Extension to a general input covariance

The extension handles anisotropic input covariance while keeping the target independent and isotropic, and finds that the phase boundary survives under stated spectral conditions. It also adjusts test-error calculations to account for variance across feature directions.

D Extension to a general input covariance

  • Anisotropic test errors use a positive measure ν that weights feature directions by their variance on new inputs; for isotropic inputs, it recovers the main formulas.In general, ν differs from both the ordinary feature eigenvalue law and αρ.Source: D Extension to a general input… · p. 30
  • For a nondegenerate positive spectrum, the stopping boundary remains b + ℓ = bc, and the best endpoint is attained by the unregularized critical kernel.An interval of positive spectral support excludes finite-time equality; discrete spectra require checking filter equality at their supported eigenvalues.Source: D Extension to a general input… · pp. 30–31
  • The extension retains b > 0 and finite initialization variance, but does not cover targets aligned with selected covariance eigenvectors, growing covariance spikes, or arbitrary structured inputs.These changes can give different spectral weights to rule learning and fitting random labels, and can alter the boundary.Source: D Extension to a general input… · p. 31

E.1 Numerical evaluation of the gradient-flow and ridge comparison

  • The numerical checks compare analytical learning curves for kernel gradient flow at λ = 0 and the τ → ∞ limit for kernel ridge regression at positive λ.The curves are evaluated using Eqs. (9)–(11).Source: §E.1 · p. 31

E.2 Validation with the full RBF kernel

Full RBF-kernel simulations on synthetic RAF data reproduce the predicted learning curves across four stopping regimes. The regimes differ in when generalization improves, factual memorization completes, and late overfitting occurs.

E.2 Validation with the full RBF kernel

  • The simulations retain the kernel’s nonzero constant term and use a 0.01 factual-error threshold for memorization time, while exact interpolation is reached asymptotically.Figure 6 uses λ = 0 and compares 12 independent realizations per panel.Source: §E.2 · p. 31, Figure 6 · p. 32
  • Across four regimes, RBF simulations agree with theory and reproduce delayed generalization, late overfitting, finite-time optima, and improvement toward the critical interpolating endpoint.Figure 6 compares theoretical curves with simulation means; the four regimes arise by varying initialization, sample ratio, and kernel angle.Source: §E.2 · pp. 31–32, Figure 6 · p. 32
  • Panels (a) and (b) show facts memorized before generalization finishes improving; in (b), fitting noise along small-eigenvalue modes is followed by substantial test deterioration.Both panels have a finite test optimum, though late deterioration in (a) is weak.Source: §E.2 · p. 32
  • In panel (c), optimal stopping occurs before factual memorization completes, while in panel (d) the critical kernel keeps improving until its interpolating endpoint has the best theoretical test error.For α > 1 in panel (c), null-space directions affect factual memorization without directly contributing to test overlaps.Source: §E.2 · p. 32

E.3 Kernel-dependent stopping on CIFAR10

The CIFAR10 kernel experiments compare stopping behavior across RBF bandwidths, then describe further neural-network experiments with varied training procedures and label assignments. The kernel trajectories differ in whether generalization deteriorates near interpolation, while the described neural-network experiments compare finite training trajectories.

E.3 Kernel-dependent stopping on CIFAR10

  • The CIFAR10 kernel task uses balanced cat and dog training examples and approximately 20% randomly labeled automobiles, evaluating generalization on test cats and dogs and memorization on training automobiles.The experiment uses five independently sampled training sets and factual label assignments.Source: §E.3 · p. 33
  • Some RBF bandwidths show test error rising as factual memorization completes, while others reach interpolation with little observed generalization loss; endpoint errors still differ substantially.A kernel with little stopping benefit need not have the best test performance.Source: §E.3 · p. 33, Figure 7 · p. 33
  • The CIFAR10 kernel trajectories qualitatively match the theory’s distinction between harmful late training and little deterioration, but the illustrated bandwidths were selected for illustration.The image experiments are described as a qualitative comparison with theoretical regimes.Source: §E.3 · p. 33

F Neural-network experiments on CIFAR10

  • The neural-network experiments vary training recipe, include augmented cross-entropy and optimizer or residual-network controls, and also randomize cat and dog labels and vary other training conditions.The experiments compare finite training trajectories.Source: F Neural-network experiments on CIFAR10 · pp. 33–34

F.1 Data, observables and square-loss training

The square-loss CIFAR10 experiments define clean-test, factual-memorization, and training-loss observables and compare convolutional and residual architectures under specified training protocols. Larger deep networks retain good test performance as factual error approaches zero, though factual classification does not imply exact interpolation.

F.1 Data, observables and square-loss training

  • The dataset contains 2,800 cats, 2,800 dogs, and 1,400 automobiles with independent uniform labels; test error uses 1,000 new cats and 1,000 new dogs.The training set has n = 7,000 and ε = 0.2.Source: §F.1 · p. 34
  • Memorization error is the fraction of misclassified randomized automobiles, whereas zero square loss requires model scores to equal their labels.Thus, zero memorization error means all randomized automobiles are classified correctly, not that training MSE is zero.Source: §F.1 · p. 34
  • The architecture comparison spans a fixed-feature random-feature CNN, a trainable two-layer CNN, deeper CNNs, and residual networks including ResNet-16 and WRN-28.The first two models use a convolutional feature map and affine readout; the random-feature model keeps its convolutional weights fixed.Source: §F.1 · p. 34
  • Trainable networks use full-MSE constant-rate Nesterov SGD for 1,500 epochs, with batch size 128 and 28–30 completed runs per curve.The dataset is fixed across runs while initialization and minibatch order vary; the random-feature CNN instead follows full-batch gradient descent.Source: §F.1 · pp. 34–35
  • Larger deep networks retain good test performance as factual error approaches zero, while the two-layer CNN remains less accurate.The observed training MSE remains positive and continues decreasing, so zero memorization error should not be conflated with exact interpolation.Source: §F.1 · p. 35, Figure 8 · p. 35
  • Figure 8 reports unsmoothed test error, memorization error, and training MSE through 1,500 epochs across model depths and capacities, with pointwise 95% confidence intervals.Crosses mark each raw mean test curve’s minimum and corresponding values of the other metrics; squares mark the last observation.Source: Figure 8 · p. 35, §F.1 · p. 35

F.2 Augmented cross-entropy training

With augmented cross-entropy, larger convolutional and residual networks maintain test performance after memorizing random labels. That pattern remains stable across optimizer and dataset controls, and appears when random labels are assigned within the same image classes.

F.2 Augmented cross-entropy training

  • Across five architectures, test error plateaus near its best level after memorization reaches zero, while training cross-entropy continues to fall.The reported trajectories extend to 2000 epochs, except WRN-28, which is shown through 1000 epochs.Source: §F.2 · p. 36, Figure 10 · p. 37

F.3 Optimizer and residual-network controls

  • Across optimizer protocols, larger deep models show a smaller final-to-minimum test-error gap, and their qualitative comparison remains similar across the tested changes.The comparisons include two- and four-layer CNNs and ResNet-16 at intermediate and high capacity.Source: §F.3 · p. 38, Figure 11 · p. 38
  • Across both augmented cross-entropy and square-loss controls, test performance stays near its lowest recorded value through training after random labels are memorized.The controls vary training-set size, batch size and random-label fraction; plateau level varies across conditions.Source: §F.3 · p. 39, Figure 12 · p. 39, Figure 13 · p. 40

F.4 Random labels on cats and dogs

  • Under square loss on same-class randomized data, deeper trainable networks at the two larger capacities reach their lowest mean test error after memorization, with smaller subsequent deterioration than frozen features and the two-layer CNN.Some architectures still retain a visible gap after reaching zero memorization error.Source: §F.4 · p. 40, Figure 14 · p. 41, Figure 15 · p. 41
  • With augmented cross-entropy, all five architectures reach zero memorization error while keeping test performance close to its lowest observed mean error on same-class randomized data.Training cross-entropy continues to decrease after memorization is complete.Source: §F.4 · p. 40, Figure 17 · p. 42

Related papers

Loading 2610.33846v1…