Source-linked AI summary
Deep Double Descent: Where Bigger Models and More Data Hurt
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever
TL;DR
The paper investigates why model size, training time, and training data can produce non-monotonic test performance in deep learning. It introduces effective model complexity to unify these effects and empirically validates generalized double descent across varied settings. The results include model-wise and epoch-wise double descent, plus regimes where increasing training data worsens test performance.
Problem
Existing intuitions disagree about whether larger models, longer training, and more data improve test performance across modern deep learning settings.
Method
The paper defines effective model complexity as the maximum number of samples on which a training procedure achieves approximately zero training error and tests generalized double descent across datasets, architectures, and optimization methods.
Results
The experiments show robust model-wise and epoch-wise double descent, and identify settings where increasing training samples by a factor of 4.5 worsens test performance.
Takeaways & Limitations
Double descent can make slightly changing model size or training procedure unexpectedly hurt test performance near the critical regime where fitting the training set is marginal.
Takeaways & Limitations
The paper has not observed settings where more data hurts under optimal early stopping and leaves the behavior of double descent in that setting open.
Abstract
from arXiv · showhide
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
1 INTRODUCTION
Modern deep learning can reconcile classical and practitioner intuitions through double descent: increasing complexity may first worsen and then improve test performance, while training time and data can show related non-monotonic effects.
- Classical theory predicts a U-shaped test-error curve, after which increasing complexity worsens performance because variance dominates.
- Modern neural networks can fit random labels yet often outperform smaller models, motivating a reassessment of the claim that larger models are worse.
- After models reach near-zero training error, further complexity decreases test error, producing model-wise double descent across under- and over-parameterized regimes.
- The effective model complexity measures the maximum sample count on which a training procedure achieves close to zero training error, incorporating architecture, data distribution, and training procedure.
- Double descent also appears with training time: test error follows a U-like curve while underfitting, then improves once effective complexity exceeds the sample count.
- Increasing training samples can sometimes hurt test performance because the peak shifts with the interpolation threshold; one example worsens after multiplying samples by 4.5.
2 OUR RESULTS
The paper proposes effective model complexity to unify double descent across model size, training time, and sample count. Experiments across datasets, architectures, optimizers, and training procedures show peaks near interpolation, including regimes where bigger models or more data hurt test performance.
- Generalized Double Descent: Effective model complexity is the maximum number of samples on which a training procedure achieves approximately zero training error.It is defined relative to a data distribution and training procedure, rather than only parameter count.
- Generalized Double Descent: The generalized hypothesis predicts that increasing effective complexity lowers test error when it is sufficiently smaller or larger than the sample count.The critically parameterized regime near equality is the exception, where increasing effective complexity may increase or decrease test error.
- Limitations: The critical interval around the interpolation threshold is not formally specified, and its width depends on the distribution and training procedure in ways the authors do not yet understand.The experiments heuristically use ϵ = 0.1.
- Model-wise Double Descent: Model-wise double descent occurs across CIFAR-10, CIFAR-100, and IWSLT‘14, using CNNs, ResNets, Transformers, SGD, Adam, data augmentation, and regularization.The test-error peak systematically occurs at the interpolation threshold, with realistic settings where bigger models are worse.
- Epoch-wise Double Descent: Epoch-wise double descent makes a large model’s test error decrease, increase around the critical regime, and decrease again later in training.This means training longer can correct overfitting.
- Sample-wise Non-monotonicity: For fixed models and training procedures, the critical sample regime can produce a plateau or peak in test error, so more data can hurt performance.The paper reports this behavior for Transformers on IWSLT‘14 and notes that label noise strengthens double descent, while peaks also occur without label noise.
3 RELATED WORK
Prior work established model-wise double descent and developed theoretical analyses, especially for linear least-squares regression.
- Model-wise double descent was proposed as a general phenomenon by Belkin et al. (2018).
- Related observations appeared in earlier studies, including Opper (1995; 2001), Advani & Saxe (2017), Spigler et al. (2018), and Geiger et al. (2019b).
- A growing theoretical literature analyzes double descent in the tractable setting of linear least-squares regression.
4 EXPERIMENTAL SETUP
The experiments vary architecture families, optimization methods, training schedules, and label-noise conditions to study double descent across modern learning settings.
- The study evaluates ResNets, standard CNNs, and Transformers with architectures parameterized by model width.
- The experimental data and raw results are described as covering the stated architecture and optimization setup, with raw data made available online.
- ResNets and CNNs are trained with cross-entropy using Adam or SGD under distinct learning-rate and training-duration schedules.
- Transformers are trained for 80K gradient steps with 10% label smoothing and no dropout.
- Label noise independently replaces the correct label with a uniformly random incorrect label with probability p, sampled once rather than per epoch.
5 MODEL-WISE DOUBLE DESCENT
Model-wise double descent appears across architectures, datasets, optimizers, and noise conditions, with test-error peaks near interpolation and shifts caused by changes that raise the interpolation threshold.
- Test error often peaks near the interpolation point, and label noise makes this peak more prominent.
- Increasing label noise, adding data augmentation, or increasing training samples shifts the interpolation threshold and test-error peak toward larger models.
- The authors note that the mechanisms behind model-wise double descent in deep neural networks remain incompletely understood.
- In critically parameterized models, the interpolating solution is described as sensitive to training noise or model misspecification.
6 EPOCH-WISE DOUBLE DESCENT
Training time can produce double descent because increasing epochs increases effective model complexity, moving sufficiently large models from under- to over-parameterized regimes.
- Increasing training time increases effective model complexity, allowing a sufficiently large model to transition from under- to over-parameterized.
- Sufficiently large models can show test error that decreases, increases near interpolation, and decreases again as training continues.
- Medium-sized models may follow a classical U-shaped test-error curve over training time, making early stopping beneficial.
- Models too small to reach the approximation threshold show monotonically decreasing test error with increased training time.
- The experiments challenge a two-phase account of training by showing that test error may fall again after an initial increase and reach a lower final value.
- Epoch-wise double descent is observed for ResNet18 and CNN models under the stated optimizer and learning-rate settings.
7 SAMPLE-WISE NON-MONOTONICITY
Varying the number of training samples shifts the critical regime where effective model complexity matches sample count, producing plateaus and sometimes worse test performance with more data.
- Increasing samples shifts the test-error peak toward larger model complexity while generally shrinking the area under the curve.
- 4× more training samples does not improve test performance for some critically parameterized model sizes when training to completion.For sufficiently under- or over-parameterized models, more samples help.
- For 5-layer CNNs on CIFAR-10 with 20% noise, the high-test-error ridge follows the interpolation threshold.Slices show that more data helps small and large models but not near-critically-parameterized models.
- More data can hurt test performance in some model-size regimes, including settings beyond deep neural networks.The paper reports this phenomenon for linear models as well.
8 CONCLUSION AND DISCUSSION
The paper proposes effective model complexity as a unifying lens for double descent and argues that critical regimes can make modest training or data changes produce atypical test behavior.
- Effective model complexity is proposed as a unifying measure for model-wise, epoch-wise, and sample-wise double descent.The paper defines atypical behavior as occurring when EMC is comparable to the number of training samples.
- Near the critical regime, small changes to model size, regularization, or training procedure may hurt test performance.
- Optimal early stopping often prevents the reported double-descent phenomena by preventing the model from reaching zero training error.The authors report one exception: model-wise double descent can still occur for ResNets on noiseless CIFAR-100.
- The authors observed no setting where more data hurts with optimal early stopping and leave its behavior as an open question.
- EMC depends on true labels and the training procedure, unlike classical complexity measures based only on model family and data distribution.
B.2 IMAGE CLASSIFICATION: EXPERIMENTAL SETUP
The experimental setup trains CNNs and ResNets with standard cross-entropy optimization choices, while the broader experiments span neural translation and prior double-descent studies.
- CNN and ResNet experiments use cross-entropy loss unless otherwise stated, with no explicit weight decay or dropout by default.
- Data augmentation uses random crops and horizontal flips, assigning augmented versions of each noisy sample the same label.
- Adam uses a constant learning rate of 1e−4 unless specified otherwise, while SGD uses inverse-square-root decay with γ0 = 0.1 and L = 512.
- The stated optimizer and learning-rate choices were selected because they optimize well, avoid experiment-specific tuning, and support consistent use across experiments.
- Experiments use batch size 128 and PyTorch default layer initialization.
- Prior work reported double descent in fully connected networks, linear regression with random features, and random forests across several datasets.
D RANDOM FEATURES: A CASE STUDY
Random Fourier Features reproduce model-wise and sample-wise double descent, with the error peak tracking equality between embedding dimension and sample count.
- Random Fourier Features use fixed random first-layer features and train a zero-initialized second layer with MSE loss.The embedding dimension d controls model size.
- When EMC = d, the test-error peak follows the path n = d.
- Horizontal and vertical crossings of the test-error grid capture model-wise and sample-wise double descent, respectively.
- The Fashion MNIST sample-wise slice uses an embedding dimension of 1000 random features.
E.1 EPOCH-WISE DOUBLE DESCENT: ADDITIONAL RESULTS
Epoch-wise double descent persists across optimizers and learning-rate schedules, with its peak shifting near the interpolation point. The broader results also show that training longer, regularization, data scale, and ensembling can substantially alter test-error dynamics.
- Optimizer and schedule robustness: Double descent occurs reliably across Adam, SGD, and SGD+Momentum with constant, inverse-square-root, and dynamic-drop learning-rate schedules.The peak shifts with the interpolation point.
- Training-time dynamics: Test error can decrease again after an intermediate maximum, making the final value lower than the earlier minimum.This means stopping as soon as test error rises is not always optimal.
- Regularization: Weight decay produces a generalized double-descent pattern in which regularization controls the effective model complexity.The reported weight-decay experiments use ResNet18 on CIFAR-10 with 20% label noise and SGD with an inverse-square-root learning rate.
- Data and task scale: With optimal early stopping, more samples are always better in the subsampled IWSLT’14 experiment, while the larger translation task has much lower test error.The figures compare model-wise test-error dynamics across sample sizes and translation datasets.