Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

Stéphane d'Ascoli; Maria Refinetti; Giulio Biroli; Florent Krzakala

Source-linked AI summary

Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, Florent Krzakala

arXiv:2003.01054v2Mar 2, 2020cs.LGcond-mat.dis-nnstat.ML

TL;DR

The paper asks why overparameterized neural networks can generalize despite interpolation and develops a quantitative lazy-regime theory using random-feature regression. It decomposes test error into bias and distinct variance sources, finding that label-noise and initialization variances generate the interpolation peak and decline with overparametrization, while ensembling suppresses them. The analysis is supported by numerical checks in realistic lazy-learning settings but relies on a non-rigorous Replica Method.

Problem
The paper addresses why interpolating, overparameterized neural networks can exhibit double descent rather than the classical U-shaped bias-variance curve.
Method
The paper analyzes high-dimensional random-feature regression as an analytically tractable model of neural networks in the lazy regime and decomposes test error into bias and separate variance sources.
Results
Label-noise and initialization variances generate the interpolation overfitting peak and account for the test-error reduction beyond interpolation, while bias and dataset-sampling variance remain constant there.
Takeaways & Limitations
Ensembling reduces the variance contributions responsible for the interpolation peak, and the paper compares this effect with overparametrization and regularization.
Takeaways & Limitations
The analytical results rely on the non-rigorous Replica Method, and deterministic ridge regression cannot capture stochastic-gradient-descent noise.

Abstract

from arXiv · show

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly. Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization. In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression. We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant. We disentangle the variances stemming from the sampling of the dataset, from the additive noise corrupting the labels, and from the initialization of the weights. Following up on Geiger et al. 2019, we first show that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization. We then quantify how they are suppressed by ensemble averaging the outputs of K independently initialized estimators. When K is sent to infinity, the test error remains constant beyond the interpolation threshold. We further compare the effects of overparametrizing, ensembling and regularizing. Finally, we present numerical experiments on classic deep learning setups to show that our results hold qualitatively in realistic lazy learning scenarios.

1 Introduction

The paper studies double descent in the lazy regime by decomposing test error into bias and multiple variance sources. Its analysis identifies which variances produce the interpolation peak and explains how ensembling and overparametrization affect them.

Motivation: Double descent replaces the classical U-shaped error curve with a peak at interpolation followed by decreasing test error under overparametrization.The interpolation peak is also called the jamming peak when regularization is absent.
Approach: The lazy regime linearizes neural networks around initialization, making them analytically tractable through random-feature or kernel-like models.In this regime, weights remain close to their initial values and the model becomes a linear fitting problem with random feature vectors.
Contributions: The analysis separates test-error contributions from dataset sampling, label noise, random-feature initialization, and bias.This disentanglement targets the distinct randomness sources underlying generalization in the lazy regime.
Main findings: Ensembling attenuates the interpolation peak, and the paper compares its effects with overparametrization and optimal regularization.The authors derive a sharp asymptotic formula for averaging predictions from independently initialized estimators.
Main findings: The overfitting peak is entirely due to label-noise and initialization variances, while bias and dataset-sampling variance remain constant beyond interpolation.Thus, the benefit of ensembling and overparametrization beyond interpolation comes solely from reducing noise and initialization variances.
Validation and scope: Numerical experiments indicate that the random-feature findings remain qualitatively relevant for correlated random features and realistic data in lazy-learning setups.The analytical calculations use a non-rigorous Replica Method, whose asymptotic predictions are checked numerically at moderately small sizes.

2 Model

The model is a two-layer random-feature network: fixed Gaussian first-layer weights generate features, while ridge regression trains the second layer. It provides an analytically tractable approximation to lazy neural-network learning and connects to kernel ridge regression.

Model: The random-feature model is a two-layer neural network with fixed random first-layer weights and ridge-regressed second-layer weights.The first-layer weights are initialized as independent centered unit-variance Gaussian variables.
Kernel connection: As the number of random features tends to infinity, the model becomes equivalent to kernel ridge regression with the corresponding kernel.The random-feature construction can also be viewed as a randomized approximation to kernel ridge regression.
Model motivation: The model separates input dimension from parameter count and represents a structured projection of latent features into a higher-dimensional ambient space.This separation makes the setting useful for studying overparametrization analytically.
Data-generating model: The input matrix contains i.i.d. Gaussian features, and labels follow a linear ground truth corrupted by additive Gaussian noise.The training inputs are represented by X, while the labels combine ⟨β, Xμ⟩ with noise ϵμ.

3 Analytical results

The paper derives an asymptotic decomposition of random-features test error into bias, three variance sources, and Bayes error, then evaluates these terms across overparametrization. The analysis uses high-dimensional random-matrix and replica methods, with numerical checks showing strong agreement at moderate sizes.

Error decomposition: The test error separates into noise, initialization, and sampling variances, bias, and a Bayes-error term that is set to zero thereafter.Noise variance comes from label noise, initialization variance from random feature vectors, and sampling variance from the training-data sample.
Asymptotic setting: The analytical result targets the high-dimensional limit where N, P, and D grow while their ratios remain fixed.The analysis considers P/D = O(1) and N/D = O(1).
Analytical method: The authors derive the decomposition by mapping the random-features problem to random-matrix and statistical-physics models, then applying mean-field and replica methods.The sequence includes Gaussian-covariate replacement, replica-based integration, and overlap-matrix order parameters with quadratic fluctuation corrections.
Behavior at the interpolation threshold: The noise and initialization variances diverge at the interpolation threshold and then decrease, while sampling variance and bias show a kink followed by a plateau.Regularization smooths these singular behaviors.
Numerical validation: Finite-size trace evaluations agree extremely well with the asymptotic expressions even at D = 200.These numerical estimates support the accuracy of the analytical predictions at moderate sizes.
Overparametrization and ensembles: At vanishing regularization, vanilla terms diverge at P = N, whereas ensemble and divide-and-conquer terms behave differently.The vanilla terms describe a single random-features model; ensemble terms describe averaging learners trained on the same dataset, while divide-and-conquer terms use different data splits.

4 Analysis of Bias and Variances

The bias-variance decomposition identifies noise and initialization variances as the source of the interpolation peak and declining test error after overparametrization. Bias and sampling variance instead remain finite or nearly constant beyond the threshold.

Interpolation Threshold: The interpolation peak is entirely caused by noise and initialization variances, which diverge at P = N without regularization.Sampling variance and bias remain finite and show a phase transition at the threshold.
Overparametrized regime: Beyond P > N, test-error reduction comes entirely from decreasing noise and initialization variances, while sampling variance and bias remain nearly constant.As P/N →∞, initialization variance vanishes but irreducible noise variance remains.
Discussion: The double-descent curve originates in noise and initialization variances rather than the bias.The paper distinguishes this result from analyses that include initialization variance within the bias term.
Intuition: At P = N, label noise and activation nonlinearity are overfit; increasing P spreads these effects across more random features and reduces test error.The model gains expressivity while P < N, then averages the overfit noise as P increases beyond N.
Bias: Increasing P beyond N does not change the bias because extra features lie in the null space of the data matrix and add no expressivity.With nonzero regularization, the threshold shifts slightly beyond P = N because regularization lowers expressivity.

5 On the effect of ensembling

Ensembling averages predictions from independently initialized random-feature networks trained on the same data. It suppresses noise and initialization variances, reducing the interpolation peak and producing effects that overlap with overparametrization, regularization, and kernel-limit behavior.

Ensembling procedure: Ensembling averages predictions from K random-feature networks trained on the same data with independently drawn first-layer weights.In the lazy regime, these independent feature initializations correspond to independent neural-network initializations.
Effect on double descent: Increasing K diminishes the overfitting peak at the interpolation threshold, including in moderate-size numerical simulations.The analytic expression agrees with simulations at D = 200.
Variance suppression: Ensembling affects only noise and initialization variances, suppressing their interpolation-threshold divergence as 1/K.For P > N, sending K →∞ has the same effect as sending P/N →∞, approaching the kernel limit.
Ensembling vs. overparametrization: For K = 2, doubling features and ensembling have similar effects, but overparametrization shifts the interpolation threshold and can yield greater asymptotic improvement.At P ≫D, both methods provide similar improvement, while ensembling is more efficient because it avoids a larger matrix inversion.
Ensembling vs. regularization: An infinitely ensembled model performs better than an optimally regularized single model, while the optimal regularization decreases as K increases.Ensembling works best when the individual predictors are under-regularized.

6 Numerical experiments on neural networks

Experiments on PCA-reduced CIFAR-10 show that ensembling behaves like the random-features theory near the lazy regime, while moving toward feature learning changes the post-interpolation curve.

Experimental setup: A 5-layer fully connected network was trained on binary 10-PCA CIFAR-10 with full-batch Adam, comparing single predictors and K = 20 ensembles.The task keeps the first ten PCA components and groups images by label parity.
Experimental setup: For α ≫1, output scaling keeps weights close to initialization and induces the lazy-learning regime.The parameter displacement scales as θ−θ0 ∼1/α.
Results: The reported error is the fraction of misclassified images in this binary classification experiment.
Results: At α = 10, the ensemble curve develops a dip near interpolation and increases for P > N, unlike the flatter lazy-regime behavior.The increase is discussed as potentially associated with feature learning.

A.1 Asymptotic scalings

The asymptotic quantities approach their limits with inverse scaling in P/N, while the noise and initialization variances diverge near interpolation according to an inverse-power law.

Asymptotic scalings: The quantities entering the generalization-error decomposition approach their asymptotic values at rate (P/N)^−1 when P/N →∞.This scaling agrees with a corresponding law reported for real neural networks when P is replaced by network width.
Asymptotic scalings: Near the interpolation threshold, the noise and initialization variances follow inverse scaling in (P/N − 1)^−1 at vanishing regularization.

A.2 Divide and Conquer approach

Divide and conquer averages learners trained on disjoint data splits, suppressing label-noise effects as K grows but reducing the effective sample size and potentially shifting the optimum underparameterized.

A.2 Divide and Conquer approach: The divide-and-conquer method partitions the dataset into K equal splits and trains each differently initialized learner on a distinct split.
A.2 Divide and Conquer approach: Because each learner uses N_eff = N/K data points, increasing K shifts the effective problem toward the underparametrized regime.
A.2 Divide and Conquer approach: As K →∞, divide and conquer completely suppresses the effect of additive label noise, unlike ordinary ensembling.
A.2 Divide and Conquer approach: At K = 2, the kernel-limit error differs from the usual kernel-limit error because the effective dataset is two times smaller.
A.2 Divide and Conquer approach: The optimal regime shifts underparameterized when K increases, SNR decreases, or N/D decreases, especially for small noisy datasets.These effects are illustrated across the three panels of Figure 12.

B Statement of the Main Result

The main result is stated under regularity, high-dimensional asymptotic, and linear-teacher assumptions, with nonlinear teacher components treated as additional noise.

B. Statement of the Main Result: The activation σ is weakly differentiable, with σ and σ′ bounded by an exponential-growth condition.
B. Statement of the Main Result: The analysis takes D, P, and N to infinity while keeping P/D and N/D fixed at O(1).
B. Statement of the Main Result: Labels follow a linear teacher f_d(x) = ⟨β, x⟩ with ||β|| = F and additive Gaussian noise ϵ_μ ∼ N(0, τ).
B. Statement of the Main Result: A nonlinear teacher component is not captured by the student when N/D = O(1) and instead acts as an additional noise term.

B.2 Results

This section defines the quantities used in the main result and outlines how the random-features predictor and generalization error are computed, including their ensemble counterparts.

The main-result expressions distinguish vanilla, ensembling, and divide-and-conquer terms through the index a ∈{v, e, d}.
The quantities Sv, Se, and Sd are defined as functions of the order parameters (q, r, ˜q, ˜r), with distinct correction terms for each setting.
The derivation averages over the dataset X, label noise W, and random feature layers Θ in that order.
Replica representations and Gaussian integral identities are used to evaluate inverse-matrix quantities appearing in the calculations.
The random-features predictor and generalization error are written explicitly for both a single estimator and ensembles of K independently sampled random-feature vectors.
In the high-dimensional limit, the random-features model is mapped to a Gaussian covariate model with a transformed activation function.

C.3.1 Averaging over the dataset

This section derives the relevant terms by Gaussian integration and saddle-point analysis, first for vanilla estimators and then for ensemble and divide-and-conquer settings.

Vanilla terms: The vanilla calculation uses replica identities, order parameters, and a saddle-point method to evaluate the action and its fluctuations.
Vanilla terms: The saddle-point equations are solved using a replica-symmetric ansatz for the order parameters q, r, ˜q, and ˜r.
Vanilla terms: Quadratic fluctuations around the saddle point are required because the leading prefactor vanishes when ˜q = ˜r = 0.
Ensembling terms: For ensembling, the two inverse matrices differ, so the derivation introduces two distinct replica-variable systems and 2 × 2 block order parameters.
Divide-and-conquer terms: The divide-and-conquer calculation uses two independent data matrices and follows the ensemble derivation, with the Hessian taken with respect to Sd.