PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel and Rich Regimes in Overparametrized Models

Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, Nathan Srebro

arXiv:2002.09277v3cs.LGstat.ML

TL;DR

The paper asks when overparametrized models follow kernel-like training dynamics versus develop non-RKHS implicit biases. It analyzes homogeneous models across initialization scales and shows a scale-controlled transition, with width also affecting certain matrix factorizations and related behavior appearing empirically in deeper models.

  • Problem

    Overparametrized models can exhibit either RKHS-norm or richer non-RKHS implicit biases, motivating analysis of when the kernel and rich regimes occur and transition.

  • Method

    The paper analyzes simple D-homogeneous models under gradient flow across initialization scales, then studies asymmetric matrix factorization and deep ReLU networks.

  • Results

    Initialization scale controls the transition from kernel behavior at α →∞ to rich behavior at α →0, while width can also induce kernel behavior in certain asymmetric matrix factorizations.

  • Takeaways & Limitations

    The analysis links initialization scale, model depth, and selected implicit bias, including transitions between ℓ2- and ℓ1-type behavior and qualitatively similar effects in deeper models.

  • Takeaways & Limitations

    The detailed exact analysis is for a family of simple depth-D models, while matrix-factorization and deep-network results are presented as extensions or empirical demonstrations.

Abstract

from arXiv · show

A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstrate how gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. Building on an observation by Chizat and Bach, we show how the scale of the initialization controls the transition between the "kernel" (aka lazy) and "rich" (aka active) regimes and affects generalization properties in multilayer homogeneous models. We also highlight an interesting role for the width of a model in the case that the predictor is not identically zero at initialization. We provide a complete and detailed analysis for a family of simple depth-$D$ models that already exhibit an interesting and meaningful transition between the kernel and rich regimes, and we also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.

1 Introduction

The paper examines when overparametrized models behave like kernel methods versus developing richer implicit biases, focusing on initialization scale as the control for their transition and generalization.

  • Kernel and rich regimes: Kernel-regime training behaves like a kernel method and selects the minimum-norm solution in the corresponding RKHS.This gives the model the inductive bias and generalization behavior of that RKHS.
  • Kernel and rich regimes: Other deep overparametrized models can induce non-Hilbert implicit biases, including ℓ1, total-variation, and nuclear-norm penalties.These biases cannot be represented by any kernel or RKHS norm.
  • Research question: The central question is when kernel and rich regimes hold and how training transitions between them.The paper frames this as determining whether deep learning remains kernel-like or exploits multilayer structure for richer inductive biases.
  • Control parameter: Initialization scale controls the kernel–rich transition in homogeneous models, rather than width alone.The cited prior observation indicates that sufficiently large initialization scale yields kernel behavior for any width.
  • Contributions: The paper exactly characterizes a simple 2-homogeneous model: the bias moves from ℓ2 as α →∞ to ℓ1 as α →0.It also studies generalization, initialization shape, depth, matrix-factorization width, and analogous behavior in deep ReLU networks.

2 Setup and preliminaries

The paper formalizes parameterized predictors and studies gradient-flow solutions in overparameterized homogeneous models, emphasizing which zero-loss minimizer the initialization selects.

  • Model setup: Models map parameters w and examples x to predictions, with the implemented predictor denoted F(w).For models linear in x, F(w) can be represented by a vector βw, although the parameterization remains crucial.
  • Homogeneity: D-homogeneous models satisfy F(c·w) = c^D F(w) for positive c, including ReLU networks, layered linear networks, and matrix factorization.Here D corresponds to network depth for these model classes.
  • Optimization: The analysis uses squared loss and gradient flow with infinitesimally small stepsizes on overparameterized training problems.The supplied setup describes many global minimizers with zero training loss.
  • Initialization scale: Initialization scale is represented by α through the path starting at wα,w0(0) = αw0.The analysis asks which global minimizer, or predictor, gradient flow reaches as time tends to infinity.

3 The Kernel Regime

The kernel regime is characterized by an approximately fixed tangent kernel during optimization. For homogeneous models, large initialization scale leads to this regime, while small scale can produce richer biases.

  • Tangent-kernel view: Gradient flow is locally governed by the model’s first-order approximation in parameter space.This yields an affine approximation with tangent features given by parameter gradients.
  • Tangent-kernel view: The tangent kernel is the inner product of parameter-gradient features, with the initialization tangent kernel serving as a central reference.The feature map is φw(t)(x) = ∇w f(w(t), x).
  • Kernel regime: The kernel regime assumes K_w(t) ≈ K_w(0) throughout optimization, so training is equivalent to kernelized gradient flow with an initialization bias term.For D-homogeneous initialization αw0, the initial kernel scales as α^{2(D−1)}K0.
  • Initialization: With unbiased initialization satisfying F(w0) = 0, the kernel-regime bias term vanishes.One construction uses units with opposite signs at initialization.
  • Kernel–rich transition: As α →∞, homogeneous-model dynamics converge to kernel gradient flow, whereas increasing width can also induce kernel behavior in certain asymmetric matrix factorizations.The width effect can occur even when α →0 at an appropriately slow rate.
  • Rich regime: The α →0 limit often yields rich, active, adaptive, or feature-learning behavior with biases such as sparsity or low-rank structure.These behaviors can support generalization settings where kernel methods would not.

4 Detailed Study of a Simple Depth-2 Model

The depth-2 model's implicit bias continuously transitions from ℓ1 in the small-initialization rich limit to a shape-dependent quadratic norm in the large-initialization kernel regime, with generalization and optimization trading off across scales.

  • Model and setup: The squared parameterization is an unbiased diagonal linear network whose two-weight construction represents all signed linear functions and permits zero predictor initialization without a trapping saddle point.The model uses 2d units and can equivalently be expressed through shared input-output weights under symmetric initialization.
  • Kernel-to-rich transition: As α→∞, gradient flow approaches the minimum-ℓ2 solution, whereas as α→0 it approaches the minimum-ℓ1 solution.These limits represent the kernel and rich regimes, respectively, and the ℓ1 bias is not an RKHS norm.
  • Kernel-to-rich transition: For every α>0, the solution is characterized as the minimum of an implicit regularizer Qα over interpolating solutions, and Qα interpolates between ℓ1 and ℓ2 as α varies.The paper develops a method for deriving this regularizer rather than assuming its form in advance.
  • Initialization shape: The transition is asymmetric: polynomially large α approximates the ℓ2 solution accurately, whereas exponentially small α may be required to approximate the ℓ1 solution uniformly.An algebraic alternative regularizer approaches ℓ1 polynomially, showing that parameter-space distance alone does not capture the bias transition's complexity.
  • Generalization and optimization: In sparse regression, the rich limit can generalize with N = Ω(r∗log d) samples, while the kernel solution requires N = Ω(d) samples and does not generalize at the smaller sample size.Thus, smaller initialization can improve generalization, although approaching zero also slows escape from the saddle point w = 0.
  • Generalization and optimization: Generalization and optimization are in tension: the largest α supporting generalization becomes extremely small near the minimum sample size for ℓ1 recovery, but larger α suffices with slightly more data.Experiments suggest standard neural-network initializations operate near the boundary between rich and kernel behavior.
  • Initialization shape: Initialization shape affects the kernel and intermediate biases but not the rich-limit ℓ1 bias, a pattern also observed for wide neural networks with different initialization distributions.In the large-scale limit, Qα,w0 is a weighted quadratic norm whose weights depend on initialization shape.

5 Higher Order Models

The analysis extends the kernel-to-rich transition from depth-2 models to D-homogeneous models with D ≥ 3. While the ℓ2 and ℓ1 limits remain unchanged, increasing depth sharpens the transition and makes rich-limit behavior easier to reach.

  • Model generalization: The model generalization corresponds to higher-order homogeneous parametrizations, including depth-D matrix factorization with commutative measurement matrices.The paper motivates this extension by noting that depth-D ReLU networks are D-homogeneous.
  • Empirical illustration: Figure 3 includes q_D(z), a norm-ratio diagnostic, and sparse-regression simulations comparing different order models.The norm ratio captures the transition from ℓ2 behavior, where it equals 1, toward ℓ1 behavior, where it equals 1/d.
  • Empirical illustration: In the sparse-regression simulation, the y-axis reports the largest initialization scale α^D that recovers the planted predictor to accuracy 0.025.A vertical dashed line marks the sample count needed for the ℓ1 solution to approximate the planted predictor.
  • Higher-order transition: For any depth D ≥ 2, the implicit bias approaches ℓ2 as α →∞ and ℓ1 as α →0.The rich-limit solution remains ℓ1 across depths, unlike the depth-dependent bias associated with explicit regularization.
  • Higher-order transition: The intermediate regime changes with homogeneity order D, especially in the sharpness of the transition between the two limiting biases.This depth dependence is illustrated in Figures 3(a)–3(c).
  • Higher-order transition: At D = 3, polynomially small α suffices to approximate ℓ1, whereas the depth-2 case requires exponentially small α.Increasing D therefore hastens the transition to the rich limit.

6 The Effect of Width

Width matters alongside initialization scale: wide asymmetric factorizations can enter the kernel regime through the lifted model even when the predictor itself remains bounded or vanishes. For commuting measurements, the resulting implicit bias continuously spans rich trace-norm-like behavior and kernel Frobenius-norm-like behavior.

  • Width and scale: The kernel regime can arise naturally from increasing width, not only from initialization scales diverging.The relevant scale is that of the lifted or symmetrized model, whose magnitude can grow with width.
  • Matrix factorization setup: For matrix factorization, the predictor is M_U,V = UV^T with width k and squared loss optimized by gradient flow.This framework includes matrix completion, matrix sensing, and two-layer linear networks.
  • Lifted dynamics: The lifted model, rather than the predictor alone, determines the dynamics because asymmetric factorization dynamics depend on the specific U,V factorization.A symmetric lifting makes the dynamics depend on both the observed product and unobserved diagonal blocks.
  • Commutative measurements: For commuting measurements, the relevant lifted scale transitions from rich behavior at zero scale to kernel behavior at infinite scale.At the transition, the implicit bias resembles Q_μ.
  • Implicit bias: The wide parallel model’s implicit bias minimizes a Schatten-Q_μ norm, approximating the trace norm for small μ and the Frobenius norm for large μ.In the diagonal case, this is the minimum Q_μ solution and requires no unbiasing trick.
  • Non-commutative measurements: For non-commutative measurements, the full intermediate regime remains difficult to characterize, although the kernel limit is proved for sufficiently small predictor scale.The paper reports empirical evidence that the commutative transition may extend beyond this setting.

7 Neural Network Experiments

Experiments across synthetic regression, matrix completion, MNIST, CIFAR10, and univariate ReLU networks demonstrate transitions between kernel and richer behaviors as initialization changes. Small initialization can leave the kernel regime and improve test performance, while the exact transition depends on architecture and parameter scaling.

  • Synthetic experiments: Synthetic ReLU networks remain near the kernel regime at large initialization and exit it as initialization becomes smaller.The experiments track both test error and tangent-kernel feature-map change.
  • Matrix completion: In matrix completion, small lifted scale yields nearly minimized nuclear norm, whereas large lifted scale leaves unobserved entries unchanged in the kernel regime.The heatmaps measure excess nuclear norm and movement on unobserved entries.
  • MNIST and CIFAR10: A VGG11-like CIFAR10 network shows similar initialization-dependent behavior to MNIST.The paper presents this as evidence that the transition also appears in a more realistic deep convolutional network.
  • Initialization trade-offs: Near α ≈ 1, models combine good test error with proximity to the kernel regime, coinciding with standard Uniform He initialization.The paper connects this region to a learning-versus-optimization trade-off.
  • Univariate ReLU networks: In univariate two-layer ReLU networks, the rich limit is associated with linear-spline interpolation, while the kernel limit gives different cubic-spline interpolations.The kernel interpolation depends on relative layer-weight scaling.

A Neural Network Experiment Details

The experiments vary initialization scale across synthetic, MNIST, and CIFAR10 settings while using standard optimization procedures and fixed architectures. They compare learned networks with explicitly trained kernel predictors and document optimization difficulties at smaller scales.

  • Synthetic setup: Synthetic experiments use 10 unit-circle points, depth-2, 3, and 5 ReLU networks, and 30 units per layer.Full gradient descent with stepsize 0.01 trains until squared loss falls below 10^-9.
  • MNIST setup: MNIST uses a single-hidden-layer ReLU network with 5000 units, cross-entropy loss, SGD, and learning-rate tuning separately for each α.Generalization is measured on 10,000 test samples after training on 60,000 examples.
  • Observed outcomes: MNIST networks reach 100% training accuracy and less than 10^-5 cross-entropy loss, while test error changes sharply with α.The output-layer-only baseline has 2.7% test error.
  • CIFAR10 setup: CIFAR10 uses a VGG11-like convolutional architecture, Uniform He initialization scaled by α, batch size 128, learning rate 0.0001, and 2000 epochs.No data augmentation is used, and all experiments reach 100% training accuracy.
  • Kernel comparison: Kernel-regime test error is approximated by freezing bottom layers and training only the output layer.These results correspond to the solid curves in the experiment figures.
  • Optimization: Smaller α creates optimization difficulties documented by the CIFAR10 training curves.The paper identifies these difficulties as a limitation of using smaller initialization.
  • Weight dynamics: Equal-magnitude input and output weights preserve equal magnitudes and prevent sign changes during training.This permits a shared-weight parametrization for each hidden unit.

C Proof of Theorem 1

The proof analyzes gradient-flow dynamics for the model’s parameters and connects their limiting behavior to the resulting linear predictor. It uses explicit trajectory expressions and residual integrals to establish Theorem 1 for arbitrary positive initialization scales.

  • Theorem scope: Theorem 1 applies for every 0 < α < ∞ and every initialization w0 with no zero entries.This establishes the general-case scope of the theorem.
  • Proof strategy: The proof begins by calculating gradient-flow dynamics on the parameter vector w.The limiting predictor is obtained by applying F to the limit of these weight dynamics.
  • Proof strategy: The residual r_α(t) is defined from the prediction error and appears in the explicit solution of the parameter dynamics.Element-wise multiplication structures the coordinatewise evolution.
  • Trajectory analysis: Because positive and negative parameter components share the same initialization, the predictor trajectory can be expressed explicitly over time.The proof then integrates the resulting expression to characterize the limit.

D Proof of Theorem 2

The proof analyzes Qα under constrained interpolation and shows that sufficiently large ℓ1- or ℓ2-norm solutions cannot satisfy the relevant minimization behavior. It also establishes that Qα need not be proportional to either norm up to O(ϵ).

  • Qα(β) = Qα(|β|), because g(x) = x arcsin(x/2) is even.
  • For suitable α, at least one pair of vectors demonstrates that Q is not proportional to the ℓ1 norm up to accuracy O(ϵ).
  • Theorem 2 states the corresponding result for any 0 < ϵ < d under the specified setting with w0 = 1.
  • A solution with sufficiently large ℓ1 norm cannot be selected, even when Qα,1(β) is not approximately proportional to ∥β∥1.
  • A solution with sufficiently large ℓ2 norm cannot be selected, even when Qα,1(β) is not approximately proportional to ∥β∥2.

E Proof of Theorem 3

The proof characterizes the rich and kernel limits of the depth-D model through gradient-flow dynamics and KKT conditions. It also extends the analysis to matrix factorization, where width and initialization jointly determine kernel behavior.

  • For D > 2, Lemma 4 provides the central characterization used in analyzing the D-homogeneous model.
  • Rich Limit: In the rich limit α → 0, the limiting solution satisfies the KKT conditions for ℓ1 minimization.
  • Kernel Regime: In the kernel regime, the limiting solution satisfies the KKT conditions for ℓ2 minimization for any D.
  • Matrix Factorization: For the matrix-factorization extension, the limiting diagonal prediction minimizes Qµ(δ) subject to Xδ = y.
  • Matrix Factorization: With width k, the matrix-factorization model can enter the kernel regime even as σ → 0, provided σ approaches zero more slowly than 1/k.

G.1 Proof of Theorem 5

The proof compares gradient flow with a tangent-kernel model for matrix factorization without requiring zero initial prediction. Under spectral and initialization conditions, it establishes controlled agreement and derives a width-dependent kernel regime.

  • The resulting corollary connects kernel-regime behavior to initialization scale α and width k, with sufficiently large width allowing α to decrease slower than 1/k.
  • The proof controls the deviation by restricting the parameter trajectory within a radius R around initialization and showing that this restriction persists.
  • Theorem 5 establishes conditions under which the factorized gradient-flow path W(t) remains close to the tangent-kernel path WTK(t).
  • The theorem assumes k ≥ d, bounded measurement spectrum λI ⪯ XX⊤ ⪯ ΛI, and µ > 4Λγ.
  • The analysis does not require that the initial predictor satisfy F(W(0)) = 0.
Loading 2002.09277v3…