On Lazy Training in Differentiable Programming

Lenaic Chizat; Edouard Oyallon; Francis Bach

Source-linked AI summary

On Lazy Training in Differentiable Programming

Lenaic Chizat, Edouard Oyallon, Francis Bach

arXiv:1812.07956v5Dec 19, 2018math.OCcs.LG

TL;DR

The paper asks whether lazy training is specific to over-parameterized neural networks and desirable in practice. It analyzes scaling, linearization, and optimization paths, finding that lazy training degrades performance in nonlinear convolutional networks.

Problem
It was unclear whether lazy training was specific to over-parameterized neural networks and whether this theoretically tractable regime was desirable in practice.
Method
The paper analyzes rescaled models theoretically, bounds their distance from linearized optimization paths, and evaluates lazy training in neural-network experiments.
Results
Performance of nonlinear deep convolutional neural networks degrades as the scaling increases and the models enter the lazy regime.
Takeaways & Limitations
These results make it unlikely that lazy training explains the many successes of neural networks on difficult high-dimensional tasks.
Takeaways & Limitations
Some quantitative convergence statements involve the smallest positive singular value of Dh(w0), which is hard to control.

Abstract

from arXiv · show

In a series of recent theoretical works, it was shown that strongly over-parameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to over-parameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit various situations where this phenomenon arises in non-convex optimization and we provide bounds on the distance between the lazy and linearized optimization paths. Our numerical experiments bring a critical note, as we observe that the performance of commonly used non-linear deep convolutional neural networks in computer vision degrades when trained in the lazy regime. This makes it unlikely that "lazy training" is behind the many successes of neural networks in difficult high dimensional tasks.

1 Introduction

The paper argues that lazy training is a general consequence of scaling that keeps optimization close to the model’s initialization linearization, rather than a phenomenon specific to over-parameterized neural networks. It develops theoretical criteria and bounds for this regime, then shows that laziness can make practical CNN performance no better than classical linear methods.

Core concept: The paper argues that scaling, rather than over-parameterization itself, makes models behave like their linearizations and equivalent positive-definite kernel methods.This behavior is presented as general beyond neural networks and as recovering a linear method.
Core concept: Lazy training occurs when the optimization paths of a nonlinear model and its initialization linearization remain close throughout training.The paths are automatically close only at the beginning; laziness means this persists until training stops.
General criterion: For rescaled models αh, κ_αh decreases as α^-1 when the scaled initial output remains bounded, producing lazy training for large α.The quantity κ_h(w0) controls how much square-loss training dynamics differ from linearized dynamics.
General criterion: Increasing initialization variance drives homogeneous neural networks into the lazy regime when their initial output has bounded norm.For two-layer models, laziness is guaranteed when mα(m) →∞, whereas α(m)=1/m yields the mean-field limit.
Contributions: The paper combines gradient-flow theory and experiments to show that lazy models track linearized dynamics, while practical CNN performance does not improve beyond some classical linear methods as laziness increases.For strongly convex losses, lazy gradient flows converge linearly to global or local minimizers depending on parameterization.

2 Analysis of Lazy Training Dynamics

For large scaling α, lazy gradient-flow dynamics remain close to the linearized model, with parameter movement typically of order O(1/α) under suitable conditions. The analysis establishes finite-horizon and uniform-time results, including global convergence in over-parameterized settings and only local-minimum convergence in under-parameterized ones.

General lazy training: Lazy training depends only on local properties of h around initialization, while finite-horizon bounds become less informative as time increases.The time dependence can grow as K^2, but the estimate remains useful for large or ill-conditioned problems stopped before convergence.
Uniform-time convergence: When R is strongly convex, the linearized flow converges linearly to the unique global minimizer, and for h(w0) = 0 its trajectory remains O(1/α) from initialization uniformly in time.These properties provide the basis for uniform-time comparisons with the nonlinear flow.
Over-parameterized case: If Dh(w0) is surjective, lazy training globally converges; when h(w0) = 0, its parameter displacement is uniformly O(1/α) as α →∞.The result assumes a positive smallest singular value of Dh(w0)^⊺ and a sufficiently large scaling factor α.
Under-parameterized case: In the under-parameterized case, strong convexity, separability, h(w0) = 0, and locally constant rank imply linear convergence for sufficiently large α.The theorem is qualitative and covers population-loss settings where F may be infinite-dimensional.
Under-parameterized case: In under-parameterized lazy training, the limiting parameters are generally a strict local minimizer rather than a global minimizer because Dh(w0)'s image may not contain R's global minimizer.The supplied discussion contrasts this finite-parameter result with global optimality obtained in an infinite over-parameterization limit under sufficiently diverse initialization.

3 Numerical Experiments

The experiments show that lazy training can harm generalization and optimization in two-layer networks, while standard nonlinear CNNs substantially outperform their lazy or linearized counterparts on CIFAR-10. Increasing the scaling factor drives CNNs toward a linearized regime with degraded performance.

Experimental settings: The experiments comprise synthetic-data two-layer networks and convolutional networks evaluated on CIFAR-10.The two experiment sets use distinct synthetic and image-classification settings.
Two-layer networks: The 1/√m scaling induces lazy training and poor generalization as width increases, whereas 1/m keeps test error relatively close to 0 for large m.The comparison averages results over five experiments with d = 100.
Two-layer networks: In the under-parameterized case, SGD converges to an a priori local minimum for large τ in the lazy regime but behaves well when τ is small.The experiment uses d = 100, m = 50, symmetrized initialization, batch-size 200, and five runs.
Convolutional networks: On CIFAR-10, increasing α for VGG-11 causes a significant performance drop followed by a plateau, indicating convergence toward the lazy regime.The linearized model at large α is insufficient to explain the model’s good performance at α = 1.
Convolutional networks: Widening VGG slightly improves the linearized model, but it remains substantially behind other non-learned representations and even farther behind their non-lazy counterparts; the same pattern occurs for ResNet.The comparison concerns widened CNNs trained in the lazy regime against linear methods without learned representations.

4 Discussion … A.2 Two-layer neural networks

The paper interprets lazy training as a scaling-induced regime in which nonlinear models behave like their linearizations, while showing that the resulting tangent-kernel perspective has concrete consequences for supervised learning and two-layer networks.

4 Discussion: Lazy training is an implicit-bias phenomenon in which a nonlinear parametric model behaves like a linear one when model scale becomes large.The scale can increase implicitly through choices of normalization, initialization, and iteration count.
A.1 Differentiable models and their linearization: The first-order approximation around initialization w0 produces a linearized or tangent model with an affine hypothesis class.For neural networks, linearity is in initialization-dependent features rather than necessarily in the input x.
A.1 Differentiable models and their linearization: For neural networks, the linearized model can resemble a non-parametric method because its feature dimension may greatly exceed the input dimension.The features are D_wf(w0, x), which generally depend non-linearly on x.
A.1 Differentiable models and their linearization: With square loss, training the affine linearized model is equivalent to training a linear model in transformed variables, and for k = 1 it becomes a tangent-kernel method.The tangent kernel is K(x, x′) = D_wf(w0, x)D_wf(w0, x′)^T.
A.2 Two-layer neural networks: In two-layer networks, the tangent kernel decomposes into one kernel contribution from each layer.The model has hidden-layer size m, parameter count p = m(d + 1), and scaling α(m) > 0.
A.2 Two-layer neural networks: With α(m) = 1/√m, the two-layer tangent kernel has a non-degenerated limit as m →∞ and can be represented through random feature kernels.Independent initial weights generate random feature kernels that converge to limiting kernels.
A.2 Two-layer neural networks: For deeper neural networks, the link with independent random sampling is lost, although tangent kernels still converge for certain architectures as network size increases.For ReLU networks with rotation-invariant weights, the two-layer kernel also admits explicit formulae based on the angle between inputs.

A.3 Generalization for the lazy model

This section asks whether lazy and tangent models generalize similarly beyond the training data. It gives a proposition establishing this under parameter-distance and uniform derivative regularity assumptions.

Generalization question: The central question is whether, at training time T, the lazy and tangent models produce approximately equal outputs on points outside the training set.The models are written as f(w(T), x) and f( w̄(T), x) for x ∈ R^d.
Scaling: The section notes that the scaling reflects use of the Euclidean metric on parameter space and becomes important for training.Because gradient definitions depend on the metric, the scaling is not intrinsically important by itself.
Generalization question: Proposition A.1 assumes the lazy and tangent parameters satisfy ∥wα(T) − w̄(T)∥ ≤ C log(α)/α^2.This parameter-closeness condition is the first hypothesis of the generalization result.
Generalization question: The proposition also requires uniformly bounded input Jacobians and uniformly Lipschitz Jacobians over a set X.Specifically, M1 := sup_x∈X ∥D_w f(w0, x)∥ and M2 := sup_x∈X Lip(w ↦ D_w f(w, x)) must be finite.

B Proofs of the theoretical results

The proofs analyze output trajectories y(t) and their linearized counterparts using scaled model notation and the tangent kernel. The nonlinear dynamics remain coupled to the parameter trajectory w_α(t).

Notation and dynamics: The proofs define y(t) = αh(w_α(t)) and ȳ(t) = αh̄(w̄_α(t)) for the original and linearized dynamics.Although suppressed in the notation, both trajectories also depend on α.
Notation and dynamics: The tangent kernel is defined as Σ(w) := Dh(w)Dh(w)ᵀ, a quadratic form on F.This kernel enters the differential-equation formulation obtained through the chain rule.
Notation and dynamics: The trajectories satisfy differential equations with shared initial condition y(0) = ȳ(0) = αh(w_0).The first differential equation is coupled with the parameter trajectory w_α(t).

B.1 Proof for Theorem 2.2 (finite horizon, non-quantitative)

The finite-horizon proof tracks dependence on α and bounds the deviation between nonlinear and linearized trajectories. A differential-inequality argument yields an O(1/α) deviation, with exponential dependence on the final time and other problem characteristics.

B.1 Proof for Theorem 2.2 (finite horizon, non-quantitative): The proof tracks only the dependence on α, with C denoting quantities independent of α.The constant C may vary from line to line.
B.1 Proof for Theorem 2.2 (finite horizon, non-quantitative): The reference trajectory remains uniformly controlled on [0,T], with bounded displacement and bounded gradient of R.Specifically, supt∈[0,T] ∥y(t) − y(0)∥ ≤ C and supt∈[0,T] ∥∇R(y(t))∥ ≤ C.
B.1 Proof for Theorem 2.2 (finite horizon, non-quantitative): The trajectory difference Δ(t) starts at zero and is bounded using a scalar differential inequality.The proof introduces Δ(t) := ∥y(t) − ȳ(t)∥ with Δ(0) = 0.
B.1 Proof for Theorem 2.2 (finite horizon, non-quantitative): Δ(t) ≤ C1 α^C2 (exp(C2t) − 1) ≤ C/α, revealing exponential dependence on the final time and other problem characteristics.This follows because Δ(t) is a sub-solution of u′(t) = C1/α + C2u(t), u(0) = 0.

B.2 Proof of Theorem 2.3 (finite horizon, square loss)

For square loss, the proof establishes that the nonlinear trajectory remains within the initialization neighborhood through the finite horizon and bounds its deviation from the linearized dynamics. The argument relies on a monotonicity property specific to square loss and yields a dimensionless relative-error bound.

Step 1: The parameter displacement satisfies ∥wα(t) − w(0)∥ ≤ tα^-1∥y(0) − y⋆∥Lip(h), giving a tighter small-time bound.The proof uses this estimate to control the trajectory relative to initialization.
Step 1: The scaling assumption α ≥ k∥y(0) − y⋆∥/(rLip(h)) ensures the exit time exceeds the finite horizon T.The resulting estimate gives ∥wα(t) − w0∥ ≤ r for the relevant time interval.
Step 1: Square loss provides the key monotonicity inequality used to control the difference between nonlinear and linearized outputs.This property is specific to the square loss and is used after taking norms and dividing by the trajectory difference.
Step 2: Evaluating the upper bound at T = K/Lip(h)^2 yields the bound stated in the theorem.The proof obtains the theorem’s finite-horizon result by substituting this specific terminal time.
Step 3: The final estimate is dimensionless and measures the relative difference between nonlinear and linearized dynamics against the scale introduced in Section 1.2.The normalization makes the inequality invariant under simple rescaling of the norms on F or R^p.

B.3 Proof of Theorem 2.4 (over-parameterized case) · B.4 Stability Lemma

The proof establishes linear convergence while the trajectory remains near initialization, then shows conditions ensuring it never leaves that neighborhood. A stability lemma bounds the discrepancy between nonlinear and linearized paths, supporting their equivalence in the lazy regime.

B.3 Proof of Theorem 2.4 (over-parameterized case): B.3 Proof of Theorem 2.4 (over-parameterized case): Within radius r0, Σ(w) remains uniformly positive definite, enabling linear convergence of y(t) until the trajectory exits that neighborhood.The radius is defined as r0 := σmin/(2Lip(Dh)), with Σ(w) ⪰ σ2_min Id/4 while ∥w − w0∥ < r0.
B.3 Proof of Theorem 2.4 (over-parameterized case): B.3 Proof of Theorem 2.4 (over-parameterized case): The exit time is infinite when the initial residual satisfies ∥y(0) − y∗∥ ≤ 2αC0.Under this condition, the parameter-variation bound stays below r0, so T = ∞.
B.3 Proof of Theorem 2.4 (over-parameterized case): B.3 Proof of Theorem 2.4 (over-parameterized case): When h(w0) = 0, the parameters remain lazy, with supt≥0 ∥wα(t) − w0∥ = O(1/α).Because y(0) is then independent of α, the bound directly yields the O(1/α) parameter variation.
B.3 Proof of Theorem 2.4 (over-parameterized case): B.3 Proof of Theorem 2.4 (over-parameterized case): The integrated discrepancy between nonlinear and tangent gradient flows is controlled by terms of order O(log α/α) and O(1/α).The first term integrates to O(1/α); the second contributes O(log α/α) on [0,t0] and O(1/α) afterward.
B.3 Proof of Theorem 2.4 (over-parameterized case): B.3 Proof of Theorem 2.4 (over-parameterized case): Strong convexity and a uniformly positive metric yield exponential decay, with ∥y(t) − y∗∥² ≤ (M/m) exp(−2mλt) ∥y(0) − y∗∥².The estimate follows from Grönwall’s Lemma combined with strong-convexity and smoothness inequalities.
B.4 Stability Lemma: B.4 Stability Lemma: The lemma compares a time-varying-metric path y(t) with a frozen-metric path ¯y(t), forming the basis for lazy-training and linearized-model equivalence.The paths solve y′(t) = −Σ(t)∇R(y(t)) and ¯y′(t) = −Σ(0)∇R(¯y(t)), with Σ(t) ⪰ λId.
B.4 Stability Lemma: B.4 Stability Lemma: The path difference is bounded by controlling a concave differential inequality for h(t), which remains below K²/(2λ³µ²).Since h(0) = 0 and the right-hand side becomes negative above this threshold, the bound holds for all t ≥ 0.

B.5 Proof of Theorem 2.5 (under-parameterized case)

The proof uses the rank theorem to straighten the model image locally, reducing the under-parameterized dynamics to a strongly convex problem on an r-dimensional subspace. It then establishes exponential convergence there and shows that, for sufficiently large α, the trajectory remains in the local neighborhood for all time.

Step 1: The rank theorem provides local diffeomorphic coordinates in which h becomes the projection π_r onto an r-dimensional subspace Π_r.The coordinates are centered at w0 and h(w0), with the remaining output coordinates set to zero.
Step 1: For sufficiently large α, the rescaled objective G_α is smooth and strongly convex on the local output domain, with a unique interior minimizer g∗.Its Hessian combines a uniformly positive-definite term with a perturbation that vanishes as α increases.
Step 2: The transformed trajectory g(t) remains in Π_r, and its distance to g∗ decays exponentially as ∥g(t) − g∗∥≤C1∥g(0) − g∗∥exp (−C2t).The constants C1 and C2 are positive and independent of α.
Step 3: Choosing α sufficiently large keeps w_α(t) inside W0 for every t≥0, so the local argument applies globally and T=∞.This completes the theorem’s under-parameterized case.

C Experimental details and additional results · C.1 Many neurons dynamics visualized · C.2 Stability of activations

With more neurons, the experiments continue to show good behavior in the non-lazy regime for small τ, while the appendix defines activation stability by tracking whether ReLU pre-activation signs remain unchanged between initialization and the final epoch.

C.1 Many neurons dynamics visualized: With m = 200 and n = 200, Figure 6 visualizes training dynamics for a larger number of neurons.The setting matches panels (a)–(b) of Figure 1 except for the increased neuron counts.
C.1 Many neurons dynamics visualized: Small τ produces good behavior in the non-lazy regime, as in Figure 1.This observation is reported for the larger-neuron setting as well.
C.1 Many neurons dynamics visualized: Symmetrized initialization enforces f(w0, ·) = 0 in panel (c), whereas panel (b) uses non-zero initialization.The comparison isolates the effect of initialization on the displayed dynamics.
C.1 Many neurons dynamics visualized: Without symmetrized initialization, neurons move slightly more to compensate for the non-zero initialization.This explanation applies to panel (b) relative to panel (c).
C.1 Many neurons dynamics visualized: In the two-layer ReLU network, changing τ^2 is equivalent to changing α by the same amount because the setting is 2-homogeneous.The figure caption states that lazy training therefore occurs for large τ.
C.2 Stability of activations: Activation stability is defined for a ReLU layer ℓ of size nℓ using test input data (xi)N.The appendix introduces this quantity as the “stability of activations” referenced in Section 3.2.
C.2 Stability of activations: For each data sample i and neuron index j, zij(T) is the pre-activation value entering the ReLU at epoch T.The definition evaluates these values using the network parameters at the specified epoch.
C.2 Stability of activations: The stability score counts indices whose pre-activation signs agree between Tlast and Tinit, normalized by nℓ × N.The supplied definition specifies sign(zij(Tlast)) = sign(zij(Tinit)) as the criterion.

C.3 Spectrum of the tangent kernel

For the VGG11 model, the linearized regime with α ≥ 10 retains far-from-zero training loss, alongside an ill-conditioned tangent-kernel spectrum. The spectrum has a one-order-of-magnitude gap after the largest 0.2% eigenvalues and then decreases approximately as O(1/i) on both CIFAR10 and random inputs.

C.3 Spectrum of the tangent kernel: Training loss remains far from 0 for all α ≥10 after 70 epochs, despite the model being highly over-parameterized.Figure 7(a) reports the corresponding train and test losses in the setting of Figure 3(a).
C.3 Spectrum of the tangent kernel: A gap of 1 order of magnitude separates the 0.2% largest eigenvalues from the remaining eigenvalues, causing ill conditionning.This spectral pattern is observed for both CIFAR10 and completely random inputs.
C.3 Spectrum of the tangent kernel: After the spectral gap, the eigenvalues decrease in order approximately O(1/i) for both CIFAR10 and completely random inputs.Figure 7 presents the tangent-kernel spectrum for the VGG11 model on these two data sets.