PaperlayerView exact revision on arXiv

Source-linked AI summary

A Modern Take on the Bias-Variance Tradeoff in Neural Networks

Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, Ioannis Mitliagkas

arXiv:1810.08591v4cs.LGstat.ML

TL;DR

The paper asks whether increasing neural-network width follows the traditional bias-variance tradeoff, especially given that wider over-parameterized networks often avoid U-shaped test-error curves. It directly measures bias and variance, decomposes variance into optimization and sampling components, and analyzes simplified theoretical settings. The results show that both bias and variance can decrease with width, while optimization variance decreases and sampling variance levels off.

  • Problem

    Traditional theory predicts falling bias and rising variance with capacity, but wider over-parameterized networks often show decreasing rather than U-shaped test error, and direct neural-network bias-variance evidence is limited.

  • Method

    The paper measures prediction bias and variance in fully connected neural networks, decomposes variance into optimization and training-set-sampling components, and develops simplified theoretical analyses.

  • Results

    Both bias and variance can decrease with network width; optimization variance decreases in the over-parameterized regime, while sampling variance increases slowly and levels off.

  • Takeaways & Limitations

    The findings provide evidence against a universal strict bias-variance tradeoff with network width in neural networks.

  • Takeaways & Limitations

    The study focuses on optimization randomness from initialization and does not analyze stochastic mini-batching randomness.

Abstract

from arXiv · show

The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve. However, recent empirical results with over-parameterized neural networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks. This suggests that there might not be a bias-variance tradeoff in neural networks with respect to network width, unlike was originally claimed by, e.g., Geman et al. (1992). Motivated by the shaky evidence used to support this claim in neural networks, we measure bias and variance in the modern setting. We find that both bias and variance can decrease as the number of parameters grows. To better understand this, we introduce a new decomposition of the variance to disentangle the effects of optimization and data sampling. We also provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

1 INTRODUCTION

The paper revisits the traditional bias-variance tradeoff because wider neural networks often lack the expected U-shaped test-error curve. Direct measurements instead find that bias and variance can decrease together as network width increases.

  • The traditional view predicts falling bias, rising variance, and a U-shaped test-error curve as model capacity increases.
  • Wider neural networks increasingly generalize better, without exhibiting the classic U-shaped test-error curve.
  • Test-error trends alone cannot determine whether bias and variance trade off, as boosting can have decreasing test error while variance still grows.
  • Earlier neural-network experiments did not support the claim that variance increases with hidden units, despite attributing the discrepancy to convergence issues.
  • Both bias and variance can decrease as network width increases in common classification and regression settings.
  • The paper decomposes total variance into optimization and training-set-sampling components, finding optimization variance decreases with width while total variance becomes dominated by sampling variance.
  • Simplified theoretical analyses support bounded prediction variance in fixed-design linear models and diminishing optimization variance in neural networks under strong assumptions.

2 RELATED WORK

Related work explains wider-network generalization through mechanisms such as implicit regularization, optimization, and alternative capacity measures, but usually studies test error rather than bias and variance directly. This paper distinguishes its direct variance measurements from concurrent double-descent analyses.

  • Prior work proposed implicit regularization, optimization effects, and new capacity measures to explain why wider networks can generalize better.
  • This paper directly measures bias and variance, addressing whether lower bias must be accompanied by higher variance.
  • Its finer decomposition finds optimization variance vanishes with width while sampling variance increases and levels off.
  • Concurrent double-descent work focuses on test-error shape, whereas this paper measures bias, variance, and variance components in the over-parameterized regime.
  • The paper differs from theoretical linear-model analyses by testing neural networks on complex, real data.

3 PRELIMINARIES

The paper extends the standard training-set bias-variance setup by modeling optimization randomness separately and decomposing total prediction variance into sampling and optimization terms. This framework supports direct analysis of how network width affects each source.

  • 3.1 Set-up: Neural-network predictors are parameterized by weights, and performance is averaged over training sets of fixed size.
  • 3.1 Set-up: Unlike the traditional decomposition, the setup includes both training-set randomness and optimization randomness such as initialization.
  • 3.1 Set-up: For fixed training data and optimization randomness, the learning algorithm produces network weights; averaging over training sets yields a marginal weight distribution.
  • 3.2 Bias-variance decomposition: The standard squared-loss decomposition separates intrinsic error, bias, and expected variance of predictions.
  • 3.2 Bias-variance decomposition: Classification losses do not generally admit the same clean bias-variance decomposition, so the paper uses squared-error quantities while plotting 0-1 classification error.
  • 3.3 Further decomposing variance into its sources: Variance due to sampling measures variation across training sets after averaging over optimization randomness.
  • 3.3 Further decomposing variance into its sources: Variance due to optimization averages prediction variance over optimization randomness for each fixed training set.
  • 3.3 Further decomposing variance into its sources: The law of total variance provides the decomposition used to analyze these two sources separately.

4 EXPERIMENTS

Across classification and regression experiments, increasing network width reduces both bias and variance rather than producing the classic tradeoff. A variance decomposition attributes this pattern to decreasing optimization variance, while sampling variance increases slowly and levels off.

  • Experiments: Across all studied tasks, bias and variance both decrease as network width increases.Experiments cover MNIST, SVHN, CIFAR10, small MNIST, and sinusoid regression.
  • 4.2 Decreasing variance in full data setting: Full-data MNIST, CIFAR10, and SVHN show a clear decreasing variance trend with width.The trend persists with or without early stopping, but does not necessarily persist when varying depth.
  • 4.3 Testing the limits: decreasing variance in the small data setting: With only 100 training examples, both bias and variance still decrease with width, despite increased sensitivity to step size.Step sizes are tuned using validation data in this small-data setting.
  • 4.3 Testing the limits: decreasing variance in the small data setting: The decreasing variance trend also appears with batch gradient descent and LBFGS, not only SGD.These results test whether the phenomenon depends specifically on stochastic gradient descent.
  • Variance decomposition: Variance due to optimization decreases with width, whereas variance due to sampling increases slowly and levels off after sufficient over-parameterization.The decomposition separates these contributions using the law of total variance.
  • 4.3 Testing the limits: decreasing variance in the small data setting: In noisy sinusoid regression, wider networks learn increasingly similar functions, suggesting lower variance across learned predictors.The visualization compares 100 learned functions at widths 15, 1000, and 10000.

5 DISCUSSION AND THEORETICAL INSIGHTS

The theoretical discussion explains the empirical variance trends through over-parameterized linear models and a simplified neural-network setting. It shows that variance need not grow with parameter count and can decline under assumptions about task-relevant and irrelevant parameter directions.

  • 5.1.2 Over-parameterized setting: When the parameter dimension exceeds the sample count, gradient descent selects the solution closest to initialization while leaving the data-matrix null space unlearned.The learned solution combines a data-dependent row-space component with an initialization-dependent null-space component.
  • 5.1 Discussion of linear models: In over-parameterized linear models, variance does not grow with the number of parameters and instead scales with the data dimension or rank.Replacing the inverse covariance with the Moore–Penrose pseudoinverse changes the relevant scale from parameter count to rank.
  • 5.2.1 Variance due to initialization: Under strong assumptions, variance due to initialization decreases to zero as width increases when the prediction Lipschitz constant grows more slowly than the square root of dimension.The result assumes optimization is invariant in task-irrelevant directions and consistently reaches the same task-relevant component.
  • 5.2 Simplified neural-network setting: With fixed initialization, prediction variance depends on the dimensionality of the task-relevant subspace and remains constant if that dimension does not grow.This simplified setting separates relevant parameter directions from orthogonal directions that are essentially irrelevant to the task.
  • 5.2.2 Variance due to sampling: The analysis focuses on initialization randomness and does not address stochastic mini-batching because decreasing variance also appeared with batch gradient descent.The theoretical variance result therefore has a narrower optimization-randomness scope than the empirical decomposition.

6 CONCLUSION AND FUTURE WORK

The paper concludes that neural-network width can reduce both bias and variance, challenging the generality of the classic tradeoff while preserving its relevance for other model classes. It proposes variance decomposition and theory as useful directions, but leaves training-time bias and variance dynamics for future work.

  • Conclusion: Both bias and variance decrease with network width, providing evidence against the claim that achieving low bias requires high variance in neural networks.The paper notes that the classic claim remains correct in several other settings, including kNN, kernel regression, and splines.
  • Conclusion: Variance due to sampling levels off after sufficient over-parameterization, while variance due to optimization decreases with width.The authors connect this decomposition to theoretical analysis inspired by linear models.
  • Future work: The study does not examine how bias and variance change over the course of training, leaving their training dynamics as future work.The authors also identify further theoretical treatment of variance as an open direction.

Appendix A Probabilistic notion of effective capacity

The paper defines an ϵ-hypothesis class that accounts for which learned functions are likely under a data distribution and learning algorithm, not merely which functions are possible. This probabilistic view supports the intuition that variance can decrease with network width.

  • The effective hypothesis class accounts for functions that the learning algorithm can learn, but not how likely those functions are to be learned.
  • The ϵ-hypothesis class contains hypotheses that are at least ϵ-likely for some ϵ > 0 under a data distribution and learning algorithm.
  • Its probabilistic formulation uses the distribution over learned functions induced by training-data sampling and algorithmic randomness.
  • The paper argues that this view can produce different intuitions about model complexity and variance than traditional worst-case analysis.
  • The accompanying experiments include bias-variance plots for CIFAR10, SVHN, and MNIST, alongside corresponding train or test-error plots.

B.5 Fixed learning rate results for small data MNIST

Fixed-learning-rate experiments in the small-data setting show that apparent U-shaped variance can arise from using a step size that is suitable for some network widths but unsuitable for others. Additional experiments examine variance across optimizers and regression settings.

  • A fixed step size of 0.01 is used for all networks in the small-data variance experiment.
  • The apparent U curve without step-size tuning is explained by the constant step being good for some networks and bad for others.
  • Tuning the step size keeps step-size optimality comparable across network widths rather than keeping the numerical step size constant.
  • Variance decreases with width in the small-data setting even with batch gradient descent or a strong optimizer such as PyTorch’s LBFGS.
  • In the noisy sinusoid task, visualizations of learned functions across widths do not show the high-variance caricature as width increases.
  • The sinusoid regression setting exhibits the same trends in decomposed variance and test error, while dynamical isometry produces decreasing bias and only a small variance increase with depth.

C.2 Discussion on need for careful experimental design

Depth experiments show that training and architectural confounds can obscure bias-variance behavior, motivating careful controls. In over-parameterized linear models, variance depends on rank rather than parameter count, while initialization variance can vanish as width grows.

  • Experimental confounds: Very deep networks can retain high test error despite low training loss, with vanishing or exploding gradients and signal-propagation problems complicating depth studies.Skip connections, improved initialization, and batch normalization facilitate low training loss, but high test error can remain.
  • Experimental confounds: Vanilla deep networks exhibit nonmonotonic bias and roughly linearly increasing variance with depth, indicating uncontrolled training effects.Networks deeper than 20 take a long time to reach the target training loss of 5e-5.
  • Experimental confounds: Skip connections push the test-error minimum from 3 layers to 21 layers but do not prevent degradation at greater depths or nonmonotonic bias.The corresponding bias, variance, and test-error curves remain checkmark-shaped.
  • Experimental confounds: Dynamical isometry appears to make bias decrease monotonically with depth, further showing that architectural and optimization choices affect measured trends.The comparison is presented alongside controls intended to isolate depth effects.
  • Linear-model analysis: In over-parameterized linear models, variance depends explicitly on r = rank(X), not N, whereas the classic parameterization yields an increasing dependence on N.The over-parameterized analysis considers N > m, with N parameters and m training examples.

D.3 Proof of Theorem 1

The proof uses concentration results for Gaussian initialization under assumptions about optimization invariance and initialization independence. It concludes that prediction variance vanishes with width when the prediction function’s Lipschitz constant grows sufficiently slowly.

  • Concentration argument: Gaussian concentration converts Lipschitz control of predictions into concentration bounds, with a dimension-free bound in the Gaussian case.The argument uses Gaussian variables and a Lipschitz prediction function.
  • Theorem setup: The proof assumes optimization is invariant in θ_M⊥ and produces solutions independently of initialization along M.These are stated as Assumptions 1 and 2.
  • Theorem setup: The theorem analyzes prediction concentration for a fixed dataset under initialization randomness.The result is stated for parameters at the end of learning in the setting of Section 5.2 and Assumptions 1 and 4.
  • Risk connection: Classification risk is bounded by four times regression risk: Rclassif ≤ 4Rreg.This proposition connects the regression-risk analysis to classification risk.

E.1 Quotes from influential papers

Influential accounts commonly describe model complexity as reducing bias while increasing variance, producing a bias-variance tradeoff. These quotations frame the conventional view as a broad principle applied to neural networks and other flexible models.

  • Neural-network claims: Geman et al. describe small networks as biased because their available function class is limited, while overparameterization is associated with a competing concern.The excerpt is presented as an influential statement of the neural-network bias-variance dilemma.
  • Neural-network claims: Vapnik links avoiding overfitting with constructing networks having small VC-dimension.This quotation frames model capacity as relevant to confidence intervals and overfitting.
  • General tradeoff: Bousquet and Elisseeff characterize learning as balancing a large model’s approximation ability against the difficulty of estimating its best function from limited data.The passage presents this as a general bias-variance tradeoff.
  • General tradeoff: Standard textbook formulations state that increasing flexibility tends to decrease bias and increase variance, with test error eventually rising after an initial decline.The cited formulation explicitly describes the resulting U-shaped expected test MSE.
  • General tradeoff: Other textbook statements summarize the same pattern as low-bias, high-variance behavior for flexible models and decreasing squared bias with increasing complexity.These quotations appear in the paper’s review of the conventional account.
Loading 1810.08591v4…