Source-linked AI summary
A Brief Prehistory of Double Descent
Marco Loog, Tom Viering, Alexander Mey, Jesse H. Krijthe, David M. J. Tax
TL;DR
The letter challenges the claim that double descent was overlooked by pointing to earlier theoretical and experimental findings. It connects those findings across minimum norm linear regression, pseudo-Fisher linear discriminants, and related learning settings, while noting that some losses may not exhibit double descent.
Problem
The letter addresses the historical gap surrounding claims that double descent was not previously observed, emphasizing earlier findings relevant to contemporary machine learning.
Source: pp. 1–2Method
The letter surveys earlier experimental and theoretical studies of double descent across minimum norm linear regression, pseudo-Fisher linear discriminants, and related learning settings.
Source: pp. 1–2Results
Earlier work experimentally demonstrated double descent for classifiers trained through minimum norm linear regression and showed similar feature curves for pseudo-Fisher linear discriminants on real-world data.
Source: p. 1Takeaways & Limitations
The historical record indicates that double descent is not universal across losses, motivating further study of this learning behavior.
Source: p. 2Results from earlier theoretical and experimental studies suggest that some losses may not exhibit double descent, including linear SVMs and closely related perceptron models.
Source: p. 2
Abstract
from arXiv · showhide
In their thought-provoking paper [1], Belkin et al. illustrate and discuss the shape of risk curves in the context of modern high-complexity learners. Given a fixed training sample size $n$, such curves show the risk of a learner as a function of some (approximate) measure of its complexity $N$. With $N$ the number of features, these curves are also referred to as feature curves. A salient observation in [1] is that these curves can display, what they call, double descent: with increasing $N$, the risk initially decreases, attains a minimum, and then increases until $N$ equals $n$, where the training data is fitted perfectly. Increasing $N$ even further, the risk decreases a second and final time, creating a peak at $N=n$. This twofold descent may come as a surprise, but as opposed to what [1] reports, it has not been overlooked historically. Our letter draws attention to some original, earlier findings, of interest to contemporary machine learning.