On the Implicit Bias in Deep-Learning Algorithms

• Gradient-based algorithms seem to prefer generalizing solutions even though overparameterized networks have many training-fit solutions, most of which do not generalize.Source: §1 · p. 1
• A degree-8 polynomial fits training data more accurately than a degree-3 polynomial but overfits, illustrating the concern with excess model expressiveness.Source: §1 · p. 1, Figure 1 · p. 2
• Neural networks seem to generalize well without explicit regularization, motivating the belief that gradient methods induce implicit bias [57].Source: §1 · pp. 1–2
• The article provides a high-level overview of fundamental results on implicit bias, but it is not a comprehensive survey.Source: §1 · p. 2

• In the classical U-shaped picture, test risk first falls and then rises as parameter count increases, producing a sweet spot between underfitting and overfitting.Source: §2 · p. 2
• Double descent is believed to reflect gradient methods implicitly minimizing network complexity, whose measure remains challenging to characterize across settings.Source: §2 · p. 2
• The double-descent curve adds a second test-risk descent beyond perfect training fit, unlike the classical U-shaped curve.Source: §2 · p. 2, Figure 2 · p. 3

3.1 Logistic regression

• For linearly separable data, gradient flow in logistic regression converges in direction to the ℓ2 maximum-margin predictor, which supports margin-based generalization bounds even when d > n.In that setting, infinitely many directions can fit the training data, while the cited bounds imply that large-margin predictors generalize well.Source: §3.1 · pp. 3–4

3.2 Linear networks

• If gradient flow reaches zero loss, a fully connected linear network’s predictor converges in direction to the ℓ2 maximum-margin predictor, while its earlier weight matrices converge to rank 1.The section notes that this result holds despite dynamics differing from those of a linear predictor.Source: §3.2 · p. 4
• For non-fully-connected linear networks, the implicit bias can change: diagonal networks favor sparse predictors through an ℓ2/k quasi-norm margin, while convolutional networks favor frequency-domain sparsity.The diagonal-network result concerns the ∥·∥2/k quasi-norm.Source: §3.2 · p. 4

3.3 Homogeneous neural networks

• For homogeneous networks reaching sufficiently small loss, gradient flow converges toward a parameter-space margin KKT point, not necessarily an actual maximum-margin optimum.KKT conditions are necessary for optimality in this setting but are not sufficient even for local optimality.Source: §3.3 · p. 5
• Whether parameter-space margin results explain practical generalization remains uncertain because known neural-network margin bounds may be loose, and implicit bias in non-homogeneous networks remains poorly understood.The section also identifies whether implied sample complexity is small enough to reflect practice as an open question.Source: §3.3 · p. 5
• Nonlinear-network results include variation-norm margin bias in infinite-width two-layer homogeneous networks and other setting-specific findings for Leaky-ReLU and univariate ReLU networks.The cited results include linear-classifier convergence under assumptions and bias toward fewer linear regions in univariate two-layer ReLU networks.Source: §3.3 · p. 5

3.4 Extensions

• Many results extend beyond gradient flow to gradient descent, steepest descent, SGD, and multiclass cross-entropy, but convergence to maximum margin can be very slow.For linear predictors, the normalized predictor’s distance to the maximum-margin predictor is generally O(1/log(T)); nevertheless, smaller iteration counts can yield sufficiently large margins on enough data for good generalization.Source: §3.4 · p. 6
• The implicit bias depends on the loss tail, and finite-accuracy training can make initialization decisive: in a kernel-regime setting, the bias may be ℓ2 rather than ℓ2/k.Exponential-tail losses generally extend the logistic-loss results, whereas polynomial-tail losses can induce a different bias; the initialization result is for certain linear diagonal networks.Source: §3.4 · p. 6

4 Implicit bias in regression

• In overparameterized square-loss regression, gradient flow seeks a zero-loss solution among many, motivating an implicit regularizer that describes its selection.The section focuses on gradient flow and the square-loss objective.Source: §4 · p. 6

4.1 Linear regression

• Linear regression selects the global minimum closest to initialization in ℓ2 distance, a result that supports standard norm-based generalization bounds.The implicit regularizer is R_w(0)(w) = ∥w − w(0)∥2.Source: §4.1 · p. 7

4.2 Linear networks

• Linear-network regularizers depend on architecture and initialization; in shared-weight diagonal networks, increasing initialization scale shifts the bias from ℓ1 toward ℓ2.Exact expressions can depend on initialization scale and shape.Source: §4.2 · p. 7
• With infinitesimal initialization, fully connected linear networks minimize the ℓ2 norm; under mirror descent, the bias is a Bregman divergence from initialization.The deep-network result holds under certain initialization assumptions.Source: §4.2 · p. 7

4.3 Matrix factorization as a test-bed for neural networks

• Matrix-factorization bias was conjectured to minimize nuclear norm, but evidence refuted that general account and instead supports viewing it as a heuristic for rank minimization.The nuclear-norm conjecture was proved for commuting observations before being formally refuted.Source: §4.3 · p. 8

4.4 Nonlinear networks

• For a single-neuron ReLU network, no nontrivial function of the weights characterizes the bias, although its solution norm is within a factor of 2 of the minimum ℓ2 norm.The bound assumes gradient flow converges to a zero-loss solution.Source: §4.4 · pp. 8–9
• Two-layer single-neuron Leaky-ReLU networks have an explicit regularizer, while ReLU results depend on setting and do not support a general rank-minimization tendency.For two-layer ReLU networks, an ℓ2 parameter-norm result requires orthogonal training examples and certain initialization assumptions; rank bias was absent in a simple small-data case.Source: §4.4 · p. 9
• Overall, implicit bias in nonlinear regression remains poorly understood, even for simple models, unlike the useful characterization available for nonlinear homogeneous classification models.The survey identifies improving this understanding as a major challenge.Source: §4.4 · p. 9

5 Dynamical stability

• Finite-step gradient descent and SGD introduce discrete and stochastic implicit biases absent from gradient flow, relevant to understanding their behavior in practice.Empirical results suggest that increasing step size and decreasing batch size may improve generalization.Source: §5 · p. 9
• Stable minima have maximal Hessian eigenvalue at most 2/η, so step size can rule out sharper minima; small batches also encourage flat minima.Flat minima are believed to generalize better.Source: §5 · p. 9
• Parameter-space flatness is hard to interpret in function space because different network parameterizations can represent the same function with sharp or flat minima.The section notes this as a challenge for understanding the implications of bias toward flat minima.Source: §5 · p. 9
• Studies connect flat minima to smoothness, input-gradient regularization, Sobolev seminorm regularization, and balanced layer magnitudes, with the cited works focused on square-loss regression.The examples span univariate ReLU networks, diagonal linear networks, SGD, linear networks, and matrix factorization.Source: §5 · pp. 9–10
• In the Edge of Stability regime, sharpness hovers around 2/η while training loss decreases non-monotonically over long timescales.Empirically observed across several architectures and tasks with cross-entropy and square loss.Source: §5 · p. 10

• With label noise, SGD training of linear diagonal networks favors sparse solutions even at large initialization scales, unlike noise-free training, which yields implicit ℓ2 regularization.Source: §6 · p. 10
• Small-step gradient descent and SGD iterates lie close to gradient flow on modified losses with explicit regularizers.Source: §6 · p. 10
• For homogeneous activations trained by gradient flow, layer squared Frobenius-norm differences remain invariant; small initialization keeps norms roughly balanced, including in sparse or shared-weight networks.Source: §6 · p. 10
• In linear neural networks and matrix factorization, large-step gradient descent favors balanced layers through its implicit bias toward flat solutions.Source: §6 · p. 10
• Function-space implications of parameter-space norm minimization have been studied for univariate and multivariate two-layer networks, linear architectures, and broader convolutional linear networks [21], [58], [26], [31] [60].Source: §6 · p. 11
• The review mostly covers the rich regime, leaving NTK-regime implicit bias outside its scope; the NTK regime does not capture feature learning.Source: §6 · p. 11

• Implicit bias may help illuminate phenomena beyond generalization because gradient-based algorithms tend to prefer specific solutions.Source: §7 · p. 11
• Gradient methods' tendency to learn non-robust networks, even when robust networks exist, can be viewed as an implication of implicit bias [78].Source: §7 · p. 11
• Using implicit-bias characterization in homogeneous networks, researchers showed that training data can be reconstructed from trained networks, with possible negative privacy implications [27], [43] [35].Source: §7 · p. 11

• Why deep-learning algorithms generalize despite having far more parameters than training examples remains poorly understood, and implicit bias is believed to be a main ingredient.Source: §8 · p. 11
• The review says understanding of implicit bias has improved dramatically but remains far from satisfactory.Source: §8 · p. 11
• The authors believe further progress on implicit bias will eventually shed light on the mystery of deep-learning generalization.Source: §8 · p. 11