PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, Yi Ma

arXiv:2002.11328v3cs.LGstat.ML

TL;DR

Classical bias-variance theory does not fully explain why larger over-parameterized neural networks often generalize better. The paper measures bias and variance across widths, architectures, datasets, losses, depths, and distribution shifts, and analyzes a two-layer linear model. It finds decreasing bias with width but unimodal variance, yielding multiple risk-curve shapes including double descent, while OOD accuracy drops are driven mainly by increased bias.

  • Problem

    Classical theory predicts increasing variance with model complexity, conflicting with observations that larger over-parameterized neural networks often generalize better.

  • Method

    The paper measures bias and variance across neural-network settings, evaluates in- and out-of-distribution behavior, and analyzes a two-layer linear network with a random first layer.

  • Results

    Bias decreases monotonically with width while variance is unimodal across considered settings; their combination produces decreasing, bell-shaped, or double-descent risk curves.

  • Takeaways & Limitations

    Out-of-distribution accuracy drops mainly reflect increased bias, while deeper models reduce bias and increase variance.

  • Takeaways & Limitations

    The two-layer analysis is not rigorously generalized to deeper networks with nonlinear activations, which may involve additional complexity-regularization interactions.

Abstract

from arXiv · show

The classical bias-variance trade-off predicts that bias decreases and variance increase with model complexity, leading to a U-shaped risk curve. Recent work calls this into question for neural networks and other over-parameterized models, for which it is often observed that larger models generalize better. We provide a simple explanation for this by measuring the bias and variance of neural networks: while the bias is monotonically decreasing as in the classical theory, the variance is unimodal or bell-shaped: it increases then decreases with the width of the network. We vary the network architecture, loss function, and choice of dataset and confirm that variance unimodality occurs robustly for all models we considered. The risk curve is the sum of the bias and variance curves and displays different qualitative shapes depending on the relative scale of bias and variance, with the double descent curve observed in recent literature as a special case. We corroborate these empirical results with a theoretical analysis of two-layer linear networks with random first layer. Finally, evaluation on out-of-distribution data shows that most of the drop in accuracy comes from increased bias while variance increases by a relatively small amount. Moreover, we find that deeper models decrease bias and increase variance for both in-distribution and out-of-distribution data.

1. Introduction

Modern neural networks challenge the classical prediction that variance rises with complexity: their bias decreases with width, while variance typically rises and then falls. This variance shape explains several risk-curve patterns, including double descent, and is supported by experiments and a two-layer linear-network analysis.

  • Classical motivation: Classical theory predicts decreasing bias, increasing variance, and a U-shaped risk curve as model complexity grows.This framework favors models with intermediate capacity.
  • Main empirical finding: Increasing neural-network width produces monotonically decreasing bias but unimodal variance that first increases and then decreases.The pattern is observed across architectures, datasets, squared loss, and cross-entropy loss.
  • Risk-curve implications: Expected risk combines bias and variance, so their relative scales yield decreasing, bell-shaped, or double-descent risk curves.Double descent appears when bias and variance dominate in different width regimes.
  • OOD and depth: On out-of-distribution data, the accuracy drop is attributed primarily to increased bias, while deeper models reduce bias and increase variance.Both bias and variance increase under corruption, but bias dominates the resulting generalization gap.
  • Theoretical analysis: A two-layer linear network with a random Gaussian first layer reproduces decreasing bias and unimodal variance in an asymptotic analysis.The variance peak occurs when hidden-neuron count is approximately half the input dimension.

2. Preliminaries

The paper uses random-design bias-variance decompositions and estimates them by averaging predictions across models trained on multiple data splits. It also contrasts this setting with fixed-design analysis and extends the decomposition to cross-entropy loss.

  • Design settings: The learned predictor depends on the random training set, motivating separate random-design and fixed-design performance assessments.The paper focuses on random-design rather than the fixed-design setting used in several prior studies.
  • Random-design decomposition: Random-design evaluation averages prediction error over training-set realizations and test examples.The learned function is written as f(x; T) to make its training-set dependence explicit.
  • Fixed-design decomposition: Fixed-design analysis holds covariates fixed and attributes training randomness to response noise around a ground-truth function.Under this setting, the covariates are fixed while yi is generated from f0(x) plus random noise.
  • Estimation procedure: The bias-variance estimator approximates expectation over training sets by averaging results from multiple random splits.There is a trade-off between larger split sizes and more splits because n × N equals the total number of training samples.
  • Cross-entropy decomposition: For cross-entropy, the paper uses a generalized Bregman-divergence decomposition, but the resulting variance estimator is not unbiased.Increasing the number of random splits improves estimate quality, while bias is overestimated and variance underestimated.

3. Measuring Bias and Variance for Neural Networks

Experiments measure neural-network bias and variance across widths, architectures, losses, datasets, and label-noise levels. Bias decreases with width, while variance is typically unimodal, producing risk curves that can decrease, peak, or double-descend.

  • Bias decreases monotonically with width, whereas variance first increases and then decreases, forming a unimodal curve.
  • In the ResNet34 mainline experiment, the variance peak is small relative to bias, so overall risk decreases monotonically with width.
  • Robustness across settings: The same bias-decreasing and variance-unimodal pattern appears across ResNext29, cross-entropy-trained ResNet34, and fully connected MNIST networks.
  • Robustness across settings: Additional experiments on CIFAR100, VGG, different sample sizes, and regularization observe the same qualitative pattern.
  • Risk-curve shapes: Changing the relative scale of bias and variance yields monotonically decreasing, unimodal, or double-descent risk curves.
  • Risk-curve shapes: Adding label noise increases variance and can produce double-descent risk when the variance peak overwhelms bias.
  • Estimator reliability: For MSE, split-sample comparisons show 0.6% relative bias difference and 3% relative variance difference, indicating limited sensitivity to finite-sample effects.
  • Estimator reliability: For cross-entropy, the estimator overestimates bias and underestimates variance, although the overall curve behaviors remain consistent.

4. What Affects the Bias and Variance?

The paper examines distribution shift and model depth using bias-variance decomposition. Out-of-distribution degradation is mainly associated with increased bias, while deeper models reduce bias but modestly increase variance.

  • Out-of-distribution examples: On CIFAR10-C, both bias and variance increase relative to CIFAR10, but increased bias contributes most of the generalization gap.
  • Effect of depth: Across ResNet18, ResNet34, and ResNet50, bias decreases and variance increases as depth increases.
  • Effect of depth: The variance differences across depths are small, around 0.01, compared with the changes in bias.
  • Effect of depth: Overall risk typically decreases with depth, suggesting that deeper-model generalization improvements are mainly attributed to lower bias.
  • Effect of depth: The qualitative depth trends remain similar when ResNet basic blocks are replaced by bottleneck blocks.
  • Scope and caveat: At high width, ResNet50 has slightly higher bias than ResNet18 and ResNet34, attributed to training difficulties without bottleneck blocks.

5. Theoretical Insights from a Two-layer Linear Model

The paper analyzes a two-layer linear network with a random first layer in the random-design setting. Its theory shows decreasing bias, unimodal variance, and risk behavior that changes with sample size.

  • Model and setting: The analysis extends fixed-design theory to random-design data using a two-layer linear network with a random first layer.The model uses Gaussian inputs, independently sampled training data, random hidden-layer weights, and ridge regression for the second-layer parameters.
  • Model and setting: The asymptotic bias, variance, and risk are expressed in terms of input dimension d, sample size n, hidden units p, and regularization λ.The analysis introduces an asymptotic regime for n, p, and d and parameterizes the resulting expressions by γ and λ.
  • Theoretical results: As sample size increases, the theoretical risk changes from unimodal to monotonically decreasing, while bias remains monotonically decreasing and variance remains unimodal.These patterns are shown across multiple values of n/d and plotted in Figure 6.
  • Theoretical results: The limiting expected bias is monotonically nonincreasing as a function of γ when λ0 ≥ 0.This matches the monotonic bias behavior predicted by classical theory.
  • Theoretical results: For sufficiently small λ0, the variance is maximized near p = d/2, with regularization shifting the peak to d/2 − λ0d.The result identifies the width at which the variance peak occurs in the analyzed asymptotic model.
  • Analysis: The variance calculation uses asymptotic expectations of products of Wishart matrices characterized through non-crossing-partition combinatorics.This provides the technical route for analyzing random-design variance.

6. Conclusion and Discussion

The paper concludes that decreasing bias combined with unimodal variance explains several observed risk-curve shapes in neural networks. It supports this account experimentally and theoretically while identifying extensions to deeper nonlinear models as future work.

  • Main conclusions: Experiments find that bias decreases monotonically with width whereas variance is unimodal, and a two-layer linear-network analysis corroborates these observations.The variance first increases and then decreases as width grows.
  • Main conclusions: The combination of monotonic bias and unimodal variance produces three typical risk-curve patterns observed in practice.The paper presents these behaviors as consistent with classical bias-variance analysis when applied to the measured curves.
  • Interpretation: The paper conjectures that regularization reduces variance beyond the data dimension by improving empirical-covariance conditioning and alignment with the regularizer.This is presented as an account of the variance decrease in highly complex models, not as a proved general explanation.
  • Open questions: The two-layer model has not been rigorously generalized to deeper networks with nonlinear activations, leaving additional interactions with explicit or implicit regularization for future study.The discussion identifies this as a direction for extending the paper’s theoretical account.

A. Summary of Experiments

The appendix summarizes the paper’s experiments and links each experiment to one or more figures. It also records supplementary results and implementation details for the neural-network studies.

  • Experiment summary: Table 1 summarizes the experiments, with each row corresponding to one experiment and the final column identifying related figures.Some rows include several independent data splits.
  • Supplementary results: The appendix provides additional experimental results, including results referenced in Sections 3 and 4.
  • Implementation: The deep-network implementation is mainly adapted from the kuangliu/pytorch-cifar repository.
  • Implementation: Training schedules vary by dataset and sample size, using specified epoch counts and learning-rate decays for CIFAR10, CIFAR100, MNIST, and FMNIST.CIFAR schedules depend on sample size, while MNIST and FMNIST use 200 epochs with a learning-rate decay every 100 epochs.

B.1. Architecture

Supplementary architecture experiments evaluate ResNext29 and VGG on CIFAR10, including cross-entropy-loss results. The VGG study varies filter width while retaining the mainline optimization setup.

  • Architecture experiments: Additional experiments evaluate ResNext29 and VGG on CIFAR10, with results reported in Figures 7 and 8.The VGG experiment uses VGG11 with filters scaled by width k as [k, 2k, 4k, 4k, 8k, 8k, 8k, 8k].
  • Experimental setup: The VGG CIFAR10 experiment uses 10,000 training samples per split and the same optimization setup as the ResNet34 mainline experiment.
  • Loss-function experiment: A separate supplementary experiment studies cross-entropy loss, with results shown in Figure 9.

B.3. Dataset

Additional experiments show that the bias decreases monotonically and variance is often bell-shaped across datasets, architectures, losses, sample sizes, regularization, and label noise.

  • CIFAR100 experiments with cross-entropy loss show a bell-shaped variance curve and monotonically decreasing bias.
  • Using fewer training samples increases both bias and variance in ResNet34 experiments on CIFAR10.
  • Larger weight decay decreases variance for ResNet34 trained on CIFAR10.
  • For 0-1 loss on CIFAR10, bias remains monotonically decreasing while variance remains unimodal.
  • Adding label noise produces a double-descent risk curve.

B.8. Sources of Error for Mean Squared Error (MSE)

The paper examines estimator reliability, cross-entropy estimation, and depth effects, finding that deeper models reduce bias while increasing variance, including on out-of-distribution data.

  • Variance estimates are similar across two data portions when averaged over multiple random splits.
  • For cross-entropy loss, small numbers of random splits overestimate bias and underestimate variance, although the overall curve behavior remains consistent.Larger k gives a better estimate, but the paper does not obtain an unbiased cross-entropy estimator.
  • Increasing depth decreases bias and increases variance for ResNet models evaluated on CIFAR10-C.Deeper ResNet models also generalize better on this out-of-distribution dataset.
  • Deeper ResNet models with bottleneck blocks have lower bias and higher variance.
  • Deeper VGG models likewise have lower bias and higher variance.

B.13. Additional Synthetic Experiment

A synthetic two-layer linear-network experiment tests the theoretical prediction that bias and variance vary systematically with normalized hidden-layer width.

  • The experiment regresses synthetic Gaussian data with a two-layer linear network whose first layer is random Gaussian and whose second layer uses gradient descent with weight decay 0.1.
  • The horizontal axis is hidden-layer parameter count normalized by data dimension, p/d.
  • Theoretical predictions align well with experimental results, including the location of the peak.

C. Proof of Theorems in §5

This section proves the paper’s two-layer linear-network results by reducing the model to matrix expressions and analyzing their asymptotic spectral behavior.

  • The training-set-dependent matrix M is defined using the random first-layer matrix W, data covariance XX⊤, and regularization λ.
  • Expected bias and variance are expressed through statistics of the matrix M.
  • The proof shows that M converges to f_M(W) almost surely under the asymptotic assumptions.The approximation is established by bounding spectral-norm perturbation terms.
  • The resulting propositions provide asymptotic formulas for risk and bias under the stated high-dimensional assumptions.
  • The asymptotic analysis uses eigenvalues and the spectral measure of a normalized Wishart-type matrix, together with the Marchenko-Pastur law.
Loading 2002.11328v3…