PaperlayerView exact revision on arXiv

Source-linked AI summary

A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability

Quan Nguyen, Robin Yadav, Param Mody, Mark Schmidt

arXiv:2610.23007v1cs.LG

TL;DR

The paper examines why gradient descent can optimize neural networks at learning rates that violate classical smoothness conditions, amid edge-of-stability behavior and non-monotone loss. It analyzes a two-neuron ReLU network through an active cone and establishes max-margin convergence and large-step catapult behavior.

  • Problem

    How can gradient descent behave at learning rates violating η < 2/β when neural-network training can exhibit edge-of-stability dynamics and non-monotone loss?

    Source: §1 · p. 1

  • Method

    The analysis identifies an active cone containing every perfect classifier, where training becomes piecewise logistic regression on time-varying data.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    Constant-step GD diverges along an explicit max-margin vector while its orthogonal component converges, and sufficiently large steps can yield exponentially small loss after two to five updates following a spike.

    Source: §1 · p. 2

  • Takeaways & Limitations

    This ReLU setup provides a tractable model for studying large-step neural-network optimization beyond linear predictors and separable data.

    Source: §8 · p. 8

    The theory is limited to one-dimensional data and a two-neuron network with fixed output weights; extending it to higher dimensions, trainable output layers, greater width, or depth remains open.

    Source: §8 · p. 8

Abstract

from arXiv · show

Gradient descent (GD) with large learning rates can successfully optimize neural networks even though classical smoothness-based learning rate conditions are violated. Yet theory for this regime largely covers linear models or linearly separable data. We analyze a minimal setting outside both regimes. Specifically, a two-neuron ReLU network trained with logistic loss on 1D non-linearly separable data. The objective is nonconvex and nonsmooth, and GD exhibits edge-of-stability (EoS) behavior. We identify an active cone in parameter space that contains every perfect classifier, where the problem becomes piecewise logistic regression with a varying effective dataset. Within this cone, we prove a finite-time loss bound for constant learning rates and show that GD converges in direction to an explicit max-margin vector, while the orthogonal component converges to the unique minimizer of a non-strongly convex potential. We further establish catapult results where exponentially small loss is reached after a single loss spike. Furthermore, we propose an adaptive schedule that lets learning rates grow to infinity without leaving the active cone. We prove that this schedule achieves a loss rate of $O(1/(T\log{T})$, which is faster than the standard $O(1/T)$ rate. Our theoretical results extend to local convergence on more general network and are validated on MNIST dataset. Overall, we show that this setup provides a tractable neural-network model for studying large and adaptive step sizes beyond linear predictors and separable data.

1 INTRODUCTION

The paper asks whether large and adaptive-step GD can be characterized throughout training on a neural-network classification problem beyond linear predictors and linearly separable data. It analyzes a minimal two-neuron ReLU model and gives results on its active cone, constant-step dynamics, catapults, adaptive steps, and a less symmetric model.

Motivation

  • The analysis targets a gap between large-step theory for neural networks, often restricted to linearly separable data, and nonseparable linear models that may cycle.The paper asks whether a neural-network classification problem beyond linear predictors and linearly separable data can have exactly characterized large- and adaptive-step trajectories.Source: §1 · pp. 1–2

Minimal example

  • The model uses three non-linearly separable points and a two-neuron ReLU network with logistic loss, whose objective is nonconvex and nonsmooth and whose GD exhibits EoS behavior.The paper studies the full trajectory of the parameters, including the phase before all data are fit.Source: §1 · pp. 1–2

Active cone

  • The active cone contains every perfect classifier and turns training into piecewise logistic regression with an effective dataset that switches with the hidden-bias sign.In the feature basis, reflected-iterate coordinates are non-decreasing in the cone; GD either stays there or enters the dead region.Source: §1 · p. 2

Constant steps

  • For constant steps, reflected iterates diverge along max-margin vector (2, 0, −1), while the orthogonal component converges to a unique minimizer of a convex, nonsmooth, non-strongly-convex potential.The paper also gives a finite-time loss bound for constant steps.Source: §1 · p. 2

Beyond EoS

  • For certain initializations, sufficiently large constant steps produce a loss spike of order η followed by exponentially small loss after two to five updates, while iterates remain in the active cone.The adaptive schedule η_t = κ|w_t| preserves the cone while η_t grows without bound; for 0 < κ ≤ 1/4, the paper proves convergence to the same max-margin direction as constant-step GD, with an additional log log t term in parameter growth.Source: §1 · p. 2

Breaking the symmetry

  • In the unsymmetric model, an explicit region around the max-margin ray is preserved by GD for any step size η ≤ 1, with a finite-time loss bound.The active cone is no longer invariant in this setting, even under gradient flow.Source: §1 · p. 2

2 RELATED WORK

Related work studies large-step GD, edge-of-stability behavior, feature learning, and implicit bias in settings that differ from this paper’s nonseparable ReLU classification problem. The paper positions its analysis around activation changes and nonseparable inputs.

EoS and catapults

  • EoS research has examined phenomena including progressive sharpening, self-stabilization, sharpness reduction, effective stochasticity, and minimal models.The passage cites Wang et al. (2022b), Damian et al. (2023), Arora et al. (2022), Kreisler et al. (2023), Kong and Tao (2020), Zhu et al. (2023), Chen and Bruna (2023), and Ahn et al. (2023).Source: §2 · p. 3

Logistic regression

  • Large-step logistic-regression results cover separable data, while nonseparable linear models may cycle; this paper extends an orthogonal-component approach to switching features and a nonsmooth potential.The cited results include max-margin convergence for separable data at arbitrary constant steps and acceleration with large or adaptive steps.Source: §2 · p. 3

Feature learning and ReLU dynamics

  • Related feature-learning analyses cover XOR learning, square-loss regression, or high-dimensional random data, whereas this paper studies logistic loss in a two-layer network under constant and increasing steps.The cited works include minibatch ReLU SGD on XOR, quadratic activations with linear loss, and square-loss settings (Lai et al., 2026).Source: §2 · p. 3

Neural implicit bias

  • Prior neural implicit-bias results include gradient-flow guarantees, smoothness-restricted GD results, or large-step results for smooth, nearly homogeneous predictors; this paper handles ReLU activation changes and nonseparable inputs directly.The cited large-step guarantee for smooth, nearly homogeneous predictors additionally uses input linear separability and activation derivatives bounded away from zero.Source: §2 · p. 3

3 PROBLEM SETUP

The paper studies a symmetric two-neuron ReLU classifier on three nonseparable one-dimensional examples. Its active cone yields a piecewise logistic-regression view with a switching effective dataset and a common max-margin vector.

Model and data

  • The data are (−1, +1), (0, −1), and (+1, +1), which are not linearly separable in the input features; the network has shared input weights ±w, shared bias b, unit output weights, and output bias g.The parameter vector is θ = (w, b, g), and the feature analysis uses reflected vector ϑ = (|w|, b, g).Source: §3 · p. 3

Active cone and landscape

  • The active cone is C = {|w| > |b|}; outside C the loss is bounded below by log 27/4 and at least one example has nonpositive margin.The dead region D has zero gradients in w and b, and a trajectory entering D cannot escape; a trajectory with loss converging to zero must eventually remain in C.Source: §3 · p. 3

Piecewise logistic regression

  • Within C, the endpoint examples activate one hidden neuron each, while the middle example activates both when b > 0 and neither when b ≤ 0.The resulting effective dataset switches between {x_e, x_e, x_−} for b_t ≤ 0 and {x_e, x_e, x_+} for b_t > 0.Source: §3 · pp. 3–4

Margins and max margin

  • In reflected coordinates, the loss is 2ℓ(u) + ℓ(v), and the common max-margin vector for all three candidate features is (2, 0, −1).Only two candidate features are active at any time, so the update dataset is degenerate in every round; no two-feature subset spans R^3.Source: §3 · p. 4

Trajectory implications

  • A trajectory can leave C only through b = −|w| into the dead region, where it becomes trapped; changing activation patterns make the transformed feature representations non-stationary.The paper identifies these changing representations as a key component of its analysis.Source: §3 · p. 4

4 CONSTANT STEP SIZES

For constant learning rates, the paper gives conditions for cone preservation, finite-time loss bounds, and convergence in direction to a max-margin vector. It also treats entry from outside the cone, characterizes the orthogonal limit, and extends cone preservation to regularization and momentum.

4.1 Cone Invariance and Loss Convergence

  • For constant steps, GD has an O((log T)^2/T) average-iterate loss bound when iterates stay in the active cone, and the sufficient cone-preservation conditions do not require steps below the EoS threshold.The stated fixed-rate conditions include 0 < η < |w_0|/2 or 0 < η ≤ 1 and 2g_0 < |w_0| + b_0.Source: §4.1 · p. 4
  • GD can enter the active cone in finite time from certain initializations in the linear region, after which the stated conditions ensure it remains there and the loss bound applies.The entry result assumes b_0 > |w_0| > 0 and g_0 − b_0/2 + 5|w_0|/2 > log 2, together with one of two step-size conditions.Source: §4.1 · pp. 4–5

4.2 Implicit Bias

  • With constant η and iterates in C, GD converges in direction to the max-margin vector, while the orthogonal component converges to the unique minimizer of a convex, coercive potential that is nonsmooth and not strongly convex.The theorem also gives exact asymptotics for parameters, loss, and the learned classifier; the potential’s minimizer lies on its kink.Source: §4.2 · p. 5
  • The learned classifier’s endpoint margins are log(ηt) + o(1), while its middle margin is log(ηt/3) + o(1); the middle margin lags by log 3.The kink locations converge to the middle point, and the decision boundary converges to |x| = 1/2.Source: §4.2 · p. 5

4.3 Regularization and momentum

  • With quadratic regularization, cone preservation holds for 0 < η_t ≤ 1/(1 + λ); with zero-initial-velocity heavy-ball momentum, it holds for 0 < η_t ≤ (1 − √β)^2/(1 + λ), under the stated initialization.Regularization contracts the dynamics: a step size that preserves the cone without regularization can cause exit after regularization is added.Source: §4.3 · p. 5

5 BEYOND EDGE OF STABILITY

The section shows that GD can operate beyond the edge of stability in this model: large constant steps can produce a loss spike followed by rapid recovery, while an adaptive schedule supports convergence with growing steps. The adaptive schedule also yields a faster loss rate and the same max-margin direction as constant-step GD.

5.1 Loss Spikes and Catapults

  • For suitable initializations, sufficiently large constant steps produce a loss spike of order η and exponentially small loss after two updates; an expanded set reaches that loss within five updates.The initialization is fixed as η grows, and the iterates remain in the active cone through the two-step result.Source: §5.1 · pp. 5–6

5.2 Adaptive Learning Rate

  • For 0 < κ ≤ 1/2, ηt = κ|wt| preserves the active cone and grows without bound; for 0 < κ ≤ 1/4, the analysis proves convergence from every initialization in the cone.The extension of the asymptotic theorem to 1/4 < κ ≤ 1/2 remains open.Source: §5.2 · pp. 6–7
  • With ηt = κ|wt| and 0 < κ ≤ 1/4, the loss is asymptotically 5/(2κt log t), gaining a factor log t over every constant step size.The schedule is horizon-free; for constant steps, tL(θt) → 5/η.Source: §5.2 · p. 6
  • Under the same adaptive schedule, GD has the same reflected max-margin direction and transverse limit as constant-step GD, while adding a log log t term along the max-margin vector.The decision boundary again converges to |x| = 1/2, and the middle example contributes three fifths of the asymptotic loss.Source: §5.2 · p. 6

6 BREAKING THE SYMMETRY

The paper decouples the two hidden weights, making the active cone no longer invariant and complicating convergence analysis. It gives a local convergence result around the max-margin ray, while noting that smoother activations may require new analyses.

6 BREAKING THE SYMMETRY

  • The decoupled model is only one step toward realism; smooth activations such as GELU may require new analyses because they produce time-varying data on a continuous subset of R3.Source: §6 · p. 7
  • Decoupling the hidden weights yields a more realistic model, but successful optimization from same-sign initial weights requires at least one weight to change sign.The model is f(x) = [w1x + b]+ + [w2x + b]+ + g.Source: §6 · p. 7
  • Without symmetry, GD can leave the active cone even under gradient flow and become trapped with positive limiting loss in another region.Additional activation regions can have nonseparable feature datasets, complicating analysis of trajectories from general initializations.Source: §6 · p. 7
  • For initializations sufficiently aligned with the four-feature max-margin vector, the paper establishes local convergence in the decoupled model.The four-feature max-margin vector represents the same classifier as the symmetric model’s max-margin vector.Source: §6 · p. 7

7 EMPIRICAL VALIDATION

The empirical validation studies binary MNIST training with a two-layer ReLU network and examines loss convergence alongside hidden-unit activation patterns. Across runs, many fits remain in active cones, while others end with all hidden units inactive.

Loss Convergence

  • With 16 hidden neurons, trainable-output training has asymptotic loss O(1/(t ln t)), compared with O(1/t) for fixed-output training.The validation uses binary MNIST classification on digits 3 and 5.Source: §7 · p. 7, Figure 4 · p. 8
  • Activation patterns continue switching late in training, mapping each image to different effective feature vectors and producing piecewise logistic-regression dynamics.With fixed output weights, pattern changes correspond to the model’s switching behavior for bt < 0 and bt > 0.Source: §7 · p. 8, Figure 4 · p. 8
  • Across 30 MNIST runs, 23 fit the data and remained in active cones, while seven ended with all hidden units inactive.The runs support the active-cone/dead-region dichotomy at the end of training despite transient visits to non-active cones.Source: §7 · p. 8

8 CONCLUSION

The paper concludes that this minimal ReLU classifier reproduces several large-step training phenomena and supports convergence and implicit-bias analysis through a piecewise-logistic view. Its theory remains limited to a one-dimensional, two-neuron model with fixed output weights, while the authors report AI assistance in drafting, proof work, and experiment code.

8 CONCLUSION

  • Piecewise logistic regression on time-varying features is the key modeling insight behind the paper’s finite-time loss bounds and implicit-bias results in the active cone.The cone contains all perfect classifiers, and the paper studies convergence within it.Source: §8 · p. 8
  • The one-dimensional, two-neuron model with fixed output weights bounds the scope of the convergence and max-margin theory; extending it to rotating hidden directions or trainable, wider, deeper networks remains open.MNIST experiments with 16 hidden units and a trainable output layer exhibit similar switching and loss decay, but extending the theory to those settings remains open.Source: §8 · p. 8

AI use statement

  • The authors used generative AI for drafting and editing, parts of proof derivations and checks, and experiment and figure code, then verified proofs and reran experiments.They state that AI was not used for research ideation and that they take responsibility for the final content.Source: AI use statement · pp. 8–9

CHECKLIST

The checklist points readers to the paper’s stated assumptions, proofs, experiments, and supplementary material. The supplements establish notation, while the related-work discussion distinguishes the paper’s cone-based trajectory results from prior analyses with different assumptions.

CHECKLIST

  • The checklist reports that each gradient-descent iteration costs O(1) operations and directs readers to the theorems and supplementary sections for convergence analyses, assumptions, and proofs.It also points to supplementary material for code and Section 7 and Section H for training details.Source: CHECKLIST · p. 11

A Minimal Two-Layer ReLU Classifier on Non-Linearly Separable Data: Supplementary Materials

  • The supplements define reflected coordinates ϑ = (|w|, b, g) and state that, while a trajectory stays in C, reflection is a fixed orthogonal coordinate change for computing updates and sharpness.Table 1 collects the symmetric model’s notation, including indices for the three signed features.Source: A Minimal Two-Layer ReLU Classifier on… · p. 12, Table 1 · p. 12

A DETAILED COMPARISON WITH RELATED WORK

  • The related-work discussion distinguishes identifying a KKT direction from proving trajectory entry and convergence: cone conditions support the latter, including admissible constant steps above the EoS threshold.For the increasing schedule ηt = κ|wt|, the paper reports the same limiting direction and t log t L(θt) → 5/(2κ) for 0 < κ ≤ 1/4; these results concern a specific minimal ReLU model.Source: A DETAILED COMPARISON WITH RELATED WORK · p. 13

B PROOFS FOR THE PROBLEM SETUP

The proofs characterize gradient descent inside and outside the active cone, establishing conditions for cone entry and invariance and describing when entry can fail or leaving the cone traps iterates at loss bounded away from zero. They also derive a finite-time loss bound within the cone using a comparator argument.

B PROOFS FOR THE PROBLEM SETUP

  • Inside the active cone, the loss is the sum of logistic losses on the endpoint margin and a middle margin determined by the sign of b.The proof also establishes that the extended loss is convex and that GD’s direction is a subgradient.Source: B PROOFS FOR THE PROBLEM SETUP · pp. 14–15
  • Leaving the active cone from within sends GD into the invariant dead region, where the loss is at least log(27/4).The exit condition is 2η_t(A_t − 2B_t) ≥ W_t + b_t when b_t > 0; otherwise the iterate remains in the cone.Source: B PROOFS FOR THE PROBLEM SETUP · p. 15

C.1 Cone Invariance and the Finite-Time Bound

  • A fixed learning rate preserves the active cone if η ≤ W_0/2, or if η ≤ 1 and 2g_0 < W_0 + b_0.The appendix also gives examples where the iterates remain in the cone despite sharpness exceeding 2/η.Source: §C.1 · pp. 16–17
  • Under either stated step-size condition, an initialization in the linear region reaches the cone boundary in finite time and enters the active cone without overshooting into the dead region.After entry, an invariance condition keeps subsequent iterates in the cone.Source: §C.1 · pp. 17–18
  • After the iterates enter C, cone invariance keeps all subsequent iterates there; before the entry time they lie in E, and a possible preceding iterate lies on the face {b = |w|}.Source: §C.1 · p. 18
  • Some initializations never enter the active cone: the stated boundary condition gives a stationary point, and w0 = 0 keeps wt = 0, so no positive step-size sequence guarantees entry from every non-dead initialization.The corollary does not claim that the sufficient conditions for entry are necessary.Source: §C.1 · p. 19
  • Within C, the proof treats GD as subgradient descent on the convex function eL and shifts a comparator along the common separating direction to absorb the second-order step term.A comparator with margin R on every feature has eL loss 3ℓ(R) ≤ 3e^-R; telescoping the comparator inequality yields the finite-time bound.Source: §C.1 · pp. 19–20

C.2 Progressive Sharpening

This section analyzes progressive sharpening and associated dynamics, including an EoS crossing, cone preservation conditions and counterexample under regularization and momentum, and a max-margin vector in rotated coordinates. It also studies a convex, coercive, non-strongly convex potential, establishing bounds and convergence of the orthogonal component to its unique minimizer despite summable perturbations.

C.2 Progressive Sharpening

  • Under Theorem C.4’s hypotheses, sharpness strictly increases through N and S_N > 2/η as margins approach zero through the first EoS crossing, raising logistic curvature while activation remains fixed.The iterates remain in the active cone, where the activation pattern is fixed.Source: §C.2 · p. 20

C.3 Regularization and momentum

  • At η=3/4, λ=1 makes the first update leave C despite ηλ<1, whereas λ=0 stays in C; zero-velocity momentum shares that first update.Source: §C.3 · p. 23
  • Under Theorem C.8 and Corollary C.9 conditions, regularized GD and momentum keep iterates in C, preserve 2g_t ≤ W_t+b_t, and never change the sign of w_t.Source: §C.3 · p. 23

D.1 Max-Margin Vector, Rotated Coordinates, and the Potential

  • The unique max-margin solution is ˆϑ=(2,0,-1), with all three constraints tight and KKT multipliers (2,2,1).Source: §D.1 · p. 25
  • In rotated coordinates, each feature shares the same e1 component, margins split into progress and orthogonal parts, and p_t is non-decreasing.Source: §D.1 · p. 25
  • The convex potential G is coercive but not strongly convex, and its minimal active feature has inner product at most -m||ϑ̄||.At b = 0, G is nondifferentiable; the active middle feature is selected by the smaller of its two inner products.Source: §D.1 · p. 26

D.2 Growth of p, Boundedness, and Expansions

  • When p < 0, each step increases p by at least ηγ/2; once p ≥ 0, bounded step sizes keep the orthogonal component bounded.For p ≥ 0 and bounded orthogonal component, E(2B + A) approximates G within 3e^(2HM)/E, and E L(θ) approximates G within 3e^(2HM)/(2E), where E = e^(γp).Source: §D.2 · pp. 27–28

D.3 A Perturbed Subgradient Method

  • For bounded iterates, summable perturbations in subgradient descent on a convex function with a unique minimizer still yield convergence to that minimizer.Uniqueness replaces strong convexity; equivalently, for a finite convex function, convergence of values to the minimum implies convergence to its unique minimizer.Source: §D.3 · p. 28

D.4 Proof of Theorem 4.5

The proof establishes convergence by first trapping the iterates, then showing linear growth of a scale variable and convergence of the orthogonal component to the unique minimizer of G. This convergence supports the theorem’s parameter and margin asymptotics.

D.4 Proof of Theorem 4.5

  • After finite time, the proof traps the iterates with p_t ≥ 0 and bounded orthogonal component, then shows E_t grows linearly.The argument bounds one-step growth using the update and establishes E_t ≍ t.Source: §D.4 · p. 29
  • The subgradient used by the orthogonal dynamics remains valid at b_t = 0, where G is nondifferentiable.At the kink, the selected gradient belongs to the subdifferential of G.Source: §D.4 · p. 29
  • The orthogonal component converges to the unique minimizer of G, using uniqueness in place of strong convexity.The iterates act as a perturbed subgradient method with step sizes of order 1/t.Source: §D.4 · p. 29
  • The proof derives parameter and margin asymptotics from convergence of the orthogonal component and the growth of E_t.The learned-function calculation also uses the ReLU 1-Lipschitz property and the stated uniform margin expression over |x| ≤ 1.Source: §D.4 · pp. 29–30

E LOSS SPIKES AND RECOVERY WITH LARGE STEP SIZES

For specified initializations and sufficiently large constant step sizes, the loss can spike and then recover exponentially quickly while iterates remain in the active cone. The proof gives two-step recovery for one initialization range and five-step recovery for an expanded range.

E LOSS SPIKES AND RECOVERY WITH LARGE STEP SIZES

  • For sufficiently large η, two-step recovery applies when ν > 1/3, while the expanded case 0 < ν < 1/3 remains in the cone through five updates.The initialization is fixed as η grows, and ν = 2B0 − A0 with ν ≠ 1/3.Source: E LOSS SPIKES AND RECOVERY WITH LARGE… · pp. 30–31
  • The recovery iterates stay in the active cone, whose gaps remain positive; W − b does not decrease while the iterates are active.The proof establishes cone membership through the relevant recovery steps for sufficiently large η.Source: E LOSS SPIKES AND RECOVERY WITH LARGE… · pp. 30–32
  • The loss after recovery is exponentially small because both final margins grow linearly with η, with the smaller margin setting the exponential rate.The stated rate limits are 3ν − 1 for the two-step case and 1 − ν for the five-step case.Source: E LOSS SPIKES AND RECOVERY WITH LARGE… · p. 32

F PROOFS FOR THE ADAPTIVE SCHEDULES

The proofs analyze adaptive increasing-step-size schedules, establishing bounds on endpoint-weight growth, cumulative loss, and the behavior of the scale and orthogonal component. They also show boundedness of the iterates with transformed growth proportional to t log t, and address initialization constraints for NTK separability and gradient-flow exit from the active region.

F PROOFS FOR THE ADAPTIVE SCHEDULES

  • For θ_0 ∈ C and 0 < κ ≤ 1/2, the schedule η_t = κW_t keeps iterates in C and has W_t → ∞ and η_t → ∞.The divergence follows by contradiction: bounded W_t would force the positive-label margin both to diverge and to remain bounded above.Source: F PROOFS FOR THE ADAPTIVE SCHEDULES · p. 33

F.1 Finite-Time Convergence with Increasing Step Sizes

  • The cumulative loss is controlled by the endpoint weight through progress along the common separating direction, without a squared-distance comparator estimate.The schedule η_t = κW_t makes this linear coordinate control the unweighted cumulative loss.Source: §F.1 · pp. 34–35
  • For κ ≤ 1/4, the endpoint weight satisfies W_T = O(log(T + 1)), yielding a logarithmic growth bound uniform in κ.The proof uses a potential comparable to W whose upward drift becomes exponentially small as the parameter scale grows.Source: §F.1 · pp. 36–37

F.2 Implicit Bias with Increasing Step Sizes

  • For 0 < κ ≤ 1/4, the scale p_t diverges while the orthogonal component remains bounded under the increasing-step schedule.The proof addresses the growing second-order update term by showing the effective orthogonal step size tends to zero and using an invariant sublevel set of a smooth potential.Source: §F.2 · p. 38
  • After a finite time, coercivity and an induction bound keep the transformed iterates ¯ϑ_t bounded under the increasing-step-size schedule.Once W_t ≥ 5, the proof uses η_t = κW_t ≥ 1 and controls F(¯ϑ_t) by K for all later iterations.Source: §F.2 · p. 40

Conclusion.

  • E_t ≍ t log t while η_t ≍ log t, and the normalized parameter direction converges toward (2, 0, −1).The margins satisfy u_t = R_t + o(1) and v_t = R_t − log 3 + o(1).Source: Conclusion · p. 41

G.1 NTK Separability and Necessary Alignment

  • For same-sign initialization with b_0 ≤ 0, the frozen tangent model has loss at least log 4, while any GD path whose loss tends to zero must change a weight’s sign and eventually remain in an opposite-orientation cone.After possibly exchanging neurons, strictly positive margins require w_1 > |b| and −w_2 > |b|.Source: §G.1 · p. 42

G.2 Finite-Time Exit under Gradient Flow

  • Under the lemma’s symmetric gradient-flow initialization, b catches r within time τ ≤ 2δ, then crosses into the linear region b > r and never returns.At the boundary, the derivative of b − r is positive on both sides, and in the linear region z increases toward log 2.Source: §G.2 · p. 43

G.3 An Invariant Region

This subsection constructs a smaller region around the max-margin ray that remains invariant under GD, even though the larger active cone need not be invariant for arbitrary initializations. Within the region, the dynamics admit a convex surrogate and a finite-time loss bound.

G.3 An Invariant Region

  • The invariant region is a cylinder around the max-margin ray, since the larger active cone is not invariant for arbitrary initializations.The construction targets a smaller region that GD never leaves.Source: §G.3 · pp. 43–44
  • The four signed features share the max-margin vector (2, −2, 0, −1), which has unit margin on each feature.The vector represents the same classifier as in the symmetric model.Source: §G.3 · p. 44
  • R(24.1,45.1) remains invariant under GD for every η∈(0,1].Corollary G.5 gives a loss-convergence bound for starts in this region.Source: §G.3 · pp. 44–46
  • Inside the active cone, GD is a subgradient method on a convex surrogate that agrees with the loss along the trajectory.The surrogate uses the active features and their margins.Source: §G.3 · pp. 44, 46
  • The finite-time loss bound uses the common margin-one comparator along the max-margin direction.The comparator is R times the max-margin vector and has surrogate loss at most 3e^-R.Source: §G.3 · p. 47

H.1 MNIST experimental details

The MNIST experiments examine active-cone behavior, gate-pattern dynamics, and finite-time loss curves, alongside constant-step and catapult behavior. Results include rapid fitting in many runs, persistent gate switching, and comparisons with predicted rates.

H.1 MNIST experimental details

  • Of 30 MNIST runs, 23 fit by step 32 and remained in active cones; the other seven ended with every hidden unit inactive.Nine fitting runs passed through non-active cones only during steps 4–14; no run ended with a misclassified image and a live unit.Source: §H.1 · p. 47
  • The MNIST setup used 256 training images of digits 3 and 5, no data augmentation, and one matched pair of runs with identical initialization and training examples.The reported results describe one matched pair and do not quantify variability across initializations; Table 3 reports final losses and classification accuracies.Source: §H.1 · pp. 47, 49, Table 3 · p. 49
  • The fitted loss reference curves had constants 4.5678 for the fully trainable network and 5.5892 for the fixed-output network, but do not establish asymptotic convergence laws.The curves provide finite-time comparisons with proposed rates.Source: §H.1 · p. 48
  • All four displayed examples switched gate patterns on every one of 149 transitions, and none had an exact period of at most 75 iterations.The authors characterize these as finite-time recurrence and switching, not infinite-time periodicity.Source: §H.1 · p. 48
  • For fixed output weights, recurrent gate patterns correspond to recurrent fixed parameter-space features; repeated gates alone do not establish fixed features when output weights evolve.In the fully trainable model, the output weights also change.Source: §H.1 · p. 49

H.2 Implicit bias

  • After a catapult, runs with η≥64 remained in the active cone through 3000 steps, with log10 loss at step 2 of −13.80 for η=256 and −55.85 for η=1024.For η=16, the sign of b kept switching while the iterates stayed in the cone, and log10 L(θ3000)=−4.06.Source: §H.2 · p. 49
  • For ηt=|wt|/2, the median and largest final losses coincided to two decimals, consistent with empirical scaling 5/(t log t); this κ is outside Theorem 5.3’s range.The two failures of ηt=1/L(θt) began from initializations that entered D within two steps, whereas both stayed in C with η0=0.25.Source: §H.2 · p. 49
  • With constant steps, Figure 6 shows tL(θt)→5/η and endpoint and middle margins approaching their predicted growth; the reflected iterates diverge along the max-margin vector.For the stated initialization and η values, the figure compares rotated coordinates with the limits from Theorem 4.5(a).Source: Figure 6 · p. 50
Loading 2610.23007v1…