Source-linked AI summary
Removing spurious minima for planar features by skip connections
Jakob Paul Zimmermann, Moritz Grillo, Andrei Balakin, Georg Loho
TL;DR
Loss landscapes of neural networks remain incompletely understood, including whether width or architectural changes exclude spurious minima. This paper analyzes Gaussian population loss for shallow ReLU teacher–student networks and proves that a learned linear skip removes spurious minima under planar-teacher conditions, while width alone need not. It also identifies limits on student feature directions and on the empirical transfer result.
Problem
It remains unclear when increasing width or adding a linear skip excludes spurious local minima in neural-network loss landscapes.
Source: §1 · pp. 1–2Method
The paper analyzes Gaussian population loss for shallow ReLU teacher–student networks, comparing plain networks with networks that include a learned linear skip.
Source: §1 · p. 2Results
With positive-mass planar teacher features and student width at least the teacher width, every local minimum with non-negative student masses is an exact fit with a learned linear skip, whereas without a skip spurious minima persist at every width of at least three in a two-dimensional example.
Source: §1.1 · p. 2, §4 · p. 7Takeaways & Limitations
Under the stated conditions, a learned linear skip rules out spurious minima that can persist under arbitrary overparameterization without the skip.
Source: §1.1 · p. 2
Abstract
from arXiv · showhide
Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.
1 INTRODUCTION
The paper studies when width and a learned linear skip connection remove spurious minima in shallow ReLU teacher–student networks. It shows that the skip rules out such minima under stated planar-teacher conditions, while width alone need not, and establishes further landscape and finite-sample results.
1 INTRODUCTION
- The analysis addresses open questions about whether width or architecture removes spurious minima and whether subspace learning is a property of the loss landscape, beyond prior gradient-based studies.For rank-at-most-two teachers, the paper characterizes the full learned-skip landscape without the small-loss or separation assumptions used in Zhou et al. (2021).Source: §1 · pp. 1–2, §1.2 · p. 3
1.1 OUR CONTRIBUTIONS
- With positive teacher masses, planar teacher features, and student width at least the teacher width, every local minimum with non-negative student masses is an exact fit when both networks have a learned linear skip.The result characterizes all such local minima without a small-loss initialization or teacher-separation assumption.Source: §1.1 · p. 2, §1.2 · p. 3
- Without a skip, a fixed two-dimensional teacher with three positive-mass hidden units has a spurious local minimum at every student width of at least three.Adding a linear skip excludes these minima under the main theorem’s assumptions.Source: §1.1 · p. 2
- With or without a skip, at local minima with non-negative student masses, every positive-mass student feature direction lies in the span of the teacher directions.Source: §1.1 · p. 2
- For plain networks in two dimensions, every critical point with positive student masses has at most twice the teacher width in distinct student feature directions, and this bound is sharp.Source: §1.1 · p. 2
- The main skip result extends to empirical loss for sufficiently accurate teacher-generated samples when local optimality is considered on a radius-dependent neighborhood.The empirical conclusion is zero loss under the stated accuracy and locality conditions.Source: §1.1 · p. 2
2 PRELIMINARIES
The preliminaries express shallow ReLU networks in mass–direction coordinates and define the loss landscapes and parameter classes used in the analysis. A residual representation then links the Gaussian population loss and its derivatives to student and teacher features.
2 PRELIMINARIES
- The paper distinguishes plain, centered, and skip networks; a learned linear skip decomposes into a centered network plus a linear map that the skip can absorb.Source: §2 · pp. 4–5
- A neuron is represented by a mass and a unit feature direction; their product is its generator, and zero-mass units contribute nothing to the network output.The mass–direction parameterization removes positive-scaling redundancy present in raw weights.Source: §2 · p. 4
- Critical points and local minima are defined in mass–direction parameter space, with the skip vector included for skip loss; when student width is at least teacher width, zero loss is attainable.In that width regime, a local minimum is spurious exactly when its loss is positive.Source: §2 · p. 5
- The Gaussian loss and its first derivatives are expressed through a residual function built from student and teacher potentials, using pair-moment kernels for the features.The centered and plain kernels differ only in the degree-one harmonic.Source: §2 · pp. 5–6
3 PROPERTIES OF LOCAL MINIMA
Local minimality imposes structural constraints on student features with non-negative masses. In particular, positive-mass directions lie in the teacher span, and a zero-mass student unit forces exact fit under the stated dimension and teacher-mass conditions.
3 PROPERTIES OF LOCAL MINIMA
- At any local minimum of the centered or plain loss with non-negative student masses, every positive-mass student direction lies in the span of the teacher directions.Source: §3 · p. 6
- The feature-span result follows by decomposing each student direction into teacher-span and orthogonal components: local minimality forces the mass-weighted orthogonal components to vanish.The proof compares normal and radial motions and uses the mixed second derivative of the loss.Source: §3 · p. 6
- For local minima with unrestricted masses and a zero-mass unit, exact fit holds for LC in d ≥ 1 and LR in d ≥ 3; teacher masses must be non-negative only when d = 2.Source: §3 · p. 6
- The non-negative-teacher-mass condition in the planar zero-mass result cannot be dropped: a negative-mass teacher example has a spurious minimum with a zero-mass student unit.Source: §3 · p. 6
4 BENIGN LOSS LANDSCAPE FOR COPLANAR TEACHER NETWORKS
For positive coplanar teachers, a skip network with at least the teacher’s width has only exact-fit local minima with non-negative student masses. The proof reduces the problem to the plane and uses angular residuals to exclude spurious minima.
4 BENIGN LOSS LANDSCAPE FOR COPLANAR TEACHER NETWORKS
- Every local minimum is an exact fit when the teacher has positive masses and planar features and the skip student is at least as wide.The result applies to non-negative student masses and any teacher linear skip.Source: §4 · p. 7
- The proof reduces the skip loss to a centered planar loss, then uses interlacing and a negative second variation to rule out spurious minima.When student width exceeds teacher width, interlacing alone yields a contradiction; the second-variation argument handles matched width.Source: §4 · p. 7
- Between feature lines, the angular residual follows an ODE and can be represented as a sum of Green functions, yielding the interlacing property.The residual is also described as the response of a bent ring to paired loads at the feature lines.Source: §4 · p. 7
- Figure 2a shows two Green-function contributions summing to the residual in a student gap containing two teacher units.Panel (b) depicts the non-negative Green function with zero value and slope at both ends and a unit third-derivative jump.Source: Figure 2 · p. 8
5 SPURIOUS LOCAL MINIMA FOR RELU
Without a skip, a three-unit planar teacher yields spurious local minima at every student width of at least three. The section also bounds the number of distinct student directions at critical points.
5 SPURIOUS LOCAL MINIMA FOR RELU
- For the planar teacher with unit masses at angles 0, 5π/6, and 4π/3, a spurious local minimum with positive student masses exists for every width n ≥ 3.The minimum is strict at n = 3 and flat for n > 3.Source: §5 · p. 8
- The construction establishes a strict spurious minimum at width three using strong convexity near a rational candidate and a separate positive-loss argument.Positive residual curvature permits splitting a student unit into multiple units with the same direction without losing local minimality.Source: §5 · p. 8
- At positive-mass critical points in two dimensions, there are at most 2m distinct student directions for a teacher of width m.For every odd m ≥ 3, the bound is attained.Source: §5 · p. 8
6 FINITE SAMPLE CONSEQUENCE
The population benignity result extends to empirical minima that persist on a fixed-radius parameter ball, provided the inputs meet an accuracy condition. This is a stability-based transfer, not a general guarantee for empirical local minima.
6 FINITE SAMPLE CONSEQUENCE
- The empirical loss is the sample-average squared error for the skip student with teacher-generated labels.Input accuracy is measured using the average squared input radius and a Gaussian approximation condition.Source: §6 · p. 9
- For every radius r > 0, sufficiently accurate finite datasets have zero empirical loss at every r-stable local minimum with non-negative masses, when the teacher is positive and coplanar and n ≥ m.In dimension one, the conclusion holds for every finite dataset without an accuracy requirement.Source: §6 · p. 9
- The transfer covers only minima stable on a prescribed parameter ball, not all empirical local minima.The proof uses uniform convergence and fixed-radius minimizer stability to derive a contradiction from a positive-loss stable minimum.Source: §6 · p. 9
7 CONCLUSION AND LIMITATIONS
The paper conjectures that the skip-network result extends beyond planar teachers, but its proof relies on angular analysis specific to the plane. It also identifies deeper networks, ordinary empirical minima, and optimization dynamics as open scopes.
7 CONCLUSION AND LIMITATIONS
- The authors expect the planarity assumption to be removable, but state this as a conjecture rather than a proved result.They identify reduction to the circle as the place where planarity enters.Source: §7 · p. 9
- The angular interlacing and descent arguments have no known higher-dimensional replacement, and the analysis is limited to one hidden layer.The residual representation relies on closed-form pair moments that have no counterpart for deeper networks.Source: §7 · p. 9
- The empirical result addresses minima stable on a prescribed parameter ball; ruling out ordinary positive-loss empirical minima remains open.The paper also states that its landscape results do not imply convergence of a particular optimization algorithm.Source: §7 · p. 9
A PROOFS
The appendix develops proof machinery for the paper’s main results, including auxiliary loss identities and proofs of local-minimum properties, the planar landscape result, and the finite-sample consequence.
A PROOFS
- The appendix organizes proofs of the auxiliary lemmas, local-minimum properties, planar benignity, persistent plain-network minima and effective-width bound, and finite-sample transfer.These proof sections correspond respectively to results in Sections 2 through 6 of the main paper.Source: A PROOFS · p. 13
A.1 AUXILIARY LEMMAS
- The reparameterization transfers local and global minimality between corresponding raw and normalized parameters, and transfers criticality when the raw weights are nonzero.The chain-rule gradients show that vanishing raw gradients is equivalent to vanishing mass derivatives and spherical feature gradients.Source: §A.1 · p. 13
- The skip loss separates into centered-network loss plus a squared linear residual, and local minima require the residual coefficient to vanish.The centered-output difference is even in the input, while the linear residual is odd; their expected cross term is therefore zero under the Gaussian law.Source: §A.1 · p. 13
- For non-negative student masses in two dimensions, the plain and centered networks correspond to translated zonotopes, with the linear term given by the translation.Centering removes the first angular Fourier harmonic while leaving even harmonics unchanged.Source: §A.1 · pp. 13–14, Figure 3 · p. 14
A.2 PROPERTIES OF LOCAL MINIMA
- At local minima with non-negative student masses, every positive-mass student feature lies in the span of the teacher directions.The proof decomposes student directions into teacher-span and orthogonal components, then uses curvature and second-moment comparisons to rule out a nonzero orthogonal component.Source: §A.2 · pp. 15–16
- A local minimum with a zero-mass unit is an exact fit under the stated dimension and loss conditions.The conditions are centered loss for d ≥ 1, plain loss for d ≥ 3, or centered loss for d = 2 with non-negative teacher masses; the proof propagates residual vanishing from an open set across the sphere.Source: §A.2 · p. 17
A.3 BENIGN LOSS LANDSCAPE FOR COPLANAR TEACHER NETWORKS
This section establishes that centered ReLU loss for coplanar teachers with positive masses has no spurious local minima with nonnegative student masses when student width is at least teacher width. The proof reduces the loss landscape to the teacher span and uses angular-residual and line-structure arguments.
A.3 BENIGN LOSS LANDSCAPE FOR COPLANAR TEACHER NETWORKS
- Confinement places every student direction in the teacher span, reducing the analysis to a plane when the teacher features are planar and n ≥ m.The proof then treats the centered loss; the result is transferred to the skip network.Source: §A.3 · p. 17
- On any gap between positive-net-mass lines, the centered angular residual is zero if the gap contains no negative-net-mass line, and strictly negative otherwise.This gap property follows from representing the piecewise residual as a sum of nonnegative Green functions.Source: §A.3 · pp. 19–20
- When there are at least two positive-net-mass lines and student units are at criticality, every gap contains a negative-net-mass line, so |S+| ≤ |S−|.The interlacing result uses the fact that student lines are double zeros of the centered angular residual.Source: §A.3 · p. 21
- A positive-output-weight local minimum with at most one positive-net-mass line must be an exact fit.If two positive-mass student units collide on a line where the residual has negative second derivative, their collision direction gives strictly negative curvature.Source: §A.3 · p. 21
- At a positive-output-weight critical point with at least two positive-net-mass lines, the positive lines equal the student lines; at a spurious local minimum, all n student directions occupy distinct lines.The distinct-line condition follows because colliding student units would yield a negative-curvature direction.Source: §A.3 · p. 22
- A hypothetical spurious local minimum must have matched width and line-separated, strictly interlacing student and teacher directions; in that case a critical point admits a strictly negative-curvature direction.Thus overparameterized cases are excluded by the interlacing condition, while the matched-width case is excluded by the descent certificate.Source: §A.3 · pp. 23–24
- The matched-width descent certificate uses an exact trigonometric inequality whose full derivation is supplied by the accompanying Lean development.The paper presents the negative-curvature direction, curvature decomposition, energy identity, and summed inequality, while Lean supplies omitted auxiliary identities and calculations.Source: §A.3 · pp. 24–25
- Every nonnegative-mass local minimum is an exact fit when the positive-mass teacher is planar and student width is at least teacher width.For d ≥ 2, the proof reduces to a two-dimensional subspace; if all student masses are positive, their directions lie in the teacher span. The d = 1 case follows by residual scaling.Source: §A.3 · p. 25
- Zero-mass student units remain a boundary exception: spurious minima occur in d = 1 for LR and, with negative teacher masses, in d = 2 for LR and LC.The d = 2 example uses a single zero-mass student unit; the passage leaves open whether the corresponding LR result holds for nonnegative teacher masses.Source: §A.3 · pp. 25–26
A.4 SPURIOUS LOCAL MINIMA FOR RELU
The appendix constructs positive-loss local minima for a fixed planar ReLU teacher and shows how splitting preserves such minima at larger widths. It also bounds the number of distinct student directions at critical points by twice the teacher width, with a sharp bound for odd teacher widths.
A.4 SPURIOUS LOCAL MINIMA FOR RELU
- Positive residual curvature lets a student unit split into finitely many same-angle children without changing the represented function or loss, preserving local minimality.For two or more children, transferring mass between children gives a non-constant constant-loss curve, so the split minimum is non-strict.Source: §A.4 · pp. 27–28
- A rational box certificate gives a strict, positive-loss local minimum for the three-unit planar teacher, while the teacher itself attains zero loss.The certificate combines a Hessian lower bound, a small gradient bound, and a differentiability argument establishing positive loss throughout the box.Source: §A.4 · pp. 26, 29
- The strict three-unit seed and finite splitting yield a spurious minimum at every student width at least three for the fixed planar teacher.At widths above three the constructed minima are non-strict; at width three the strict seed is used.Source: §A.4 · p. 29
- For planar plain ReLU networks, every critical point with positive student masses has at most 2m distinct student directions, and this bound is attained for odd m ≥ 3.The counting argument uses positive humps of the residual-related function and teacher atoms in student-angle gaps.Source: §A.4 · p. 30
A.5 FINITE SAMPLE CONSEQUENCE
The appendix transfers the population exact-fit result to empirical minima on parameter balls when teacher-generated samples meet a radius-dependent accuracy condition. Its proof uses compactness and uniform convergence, and the radius must be fixed before refining the approximation.
B.1.1 THE MODEL
- The result is stated for the Gaussian population framework, with the empirical loss expressed using the student–teacher residual and a mismatch coordinate for the linear skip.The mismatch coordinate is u = b_s − b_match(s, w).Source: §A.5 · p. 30, §B.1.1 · p. 33
A.5 FINITE SAMPLE CONSEQUENCE
- The radius must be fixed before approximation is refined, because minima with shrinking neighborhoods can converge to a saddle.The appendix distinguishes this setting from results requiring uniformly strict local minima or smoothness with gradient and Hessian control.Source: §A.5 · p. 31
- For sufficiently accurate teacher-generated samples, a student locally optimal on a prescribed parameter ball has zero empirical loss under the main theorem’s assumptions.The required sampling accuracy depends on the locality radius.Source: §A.5 · p. 32
- The transfer proof places stable minima in a dataset-independent compact set, then uses uniform objective convergence and population minimizer stability to rule out positive empirical loss.The compactness argument bounds the minima independently of the dataset and its radial scale.Source: §A.5 · pp. 31–32
B LEAN CERTIFICATES
- The appendix reports that every theorem, proposition, lemma, equation, and named proof step has a counterpart in a Lean 4 development that compiles without admitted proofs.The accompanying build certificate limits the paper statements’ axioms to propositional extensionality, choice, and quotient soundness.Source: B LEAN CERTIFICATES · p. 33
B.1.2 CONTRIBUTION 1: BENIGN LOSS LANDSCAPE WITH LINEAR SKIPS
With a learned linear skip, non-negative local minima are exact fits under the stated planar-teacher and width assumptions. Without a skip, a fixed three-unit planar teacher has positive-loss minima at every student width of at least three.
B.1.2 CONTRIBUTION 1: BENIGN LOSS LANDSCAPE WITH LINEAR SKIPS
- A learned linear skip makes every non-negative local minimum an exact fit when the teacher has positive weights and planar features and the student is at least as wide.The theorem concerns Gaussian population loss and unit feature directions.Source: §B.1.2 · p. 34
B.1.3 CONTRIBUTION 2: SPURIOUS MINIMA AT ARBITRARY OVERPARAMETERIZATION
- Without a skip, the fixed planar teacher with three unit-mass neurons has a positive-mass spurious minimum at every width of at least three.The width-three witness is strict; witnesses at larger widths are non-strict and have positive loss.Source: §B.1.3 · p. 34
B.1.4 CONTRIBUTION 3: LEARNING THE TEACHER SUBSPACE
At local minima with non-negative student output weights, every positive-mass student feature direction lies in the teacher-feature span. In two dimensions, positive-mass plain-ReLU critical points also have a teacher-width-controlled bound on their distinct oriented directions.
B.1.4 CONTRIBUTION 3: LEARNING THE TEACHER SUBSPACE
- Every student direction with positive output mass at a local minimum lies in the teacher-feature span, for either activation model; this is containment, not equality.The theorem assumes non-negative student masses and unit directions.Source: §B.1.4 · p. 35
B.1.5 CONTRIBUTION 4: EFFECTIVE WIDTH
- The effective-width result's critical-point condition requires a genuine zero Fréchet derivative in mass-angle coordinates, not merely the convention that fderiv is zero at nondifferentiable points.The plain loss is globally C2 in these coordinates.Source: §B.1.5 · p. 35
- In dimension two, a positive-mass plain-ReLU critical point has at most 2m distinct oriented student directions, with equality attained for odd teacher widths m ≥ 3.Opposite directions count separately.Source: §B.1.5 · p. 35
B.1.6 CONTRIBUTION 5: TRANSFER TO EMPIRICAL LOSS
The paper transfers its benign-landscape conclusion to empirical loss minimization over a prescribed joint parameter ball. Under a Gaussian-approximation condition on the data and the stated teacher–student assumptions, such candidates fit every observed label exactly.
B.1.6 CONTRIBUTION 5: TRANSFER TO EMPIRICAL LOSS
- Data accuracy compares the unweighted dataset with the Gaussian functional on continuous degree-two homogeneous tests, using the observed average squared input radius for normalization.Teacher-generated labels are exact, without observation noise.Source: §B.1.6 · pp. 36–37
- The joint parameter ball uses the maximum of the mass, direction, and matching-skip-mismatch distances, and minimization is required against every feasible point in that ball.The ball allows signed mass perturbations, although the candidate itself must have nonnegative masses.Source: §B.1.6 · p. 37
- A non-negative-mass candidate that minimizes empirical loss over the prescribed ball has zero empirical loss when labels come from the teacher and the student is at least as wide as the teacher.The theorem assumes a planar teacher-feature span, positive teacher masses, and unit directions.Source: §B.1.6 · p. 37
- For each prescribed positive radius, one positive accuracy tolerance suffices uniformly over nonempty datasets and is chosen before the sample, labels, skips, or student parameters.The tolerance depends on the radius; dimension one needs no accuracy hypothesis.Source: §B.1.6 · pp. 36–37
B.2 STATEMENTS AND PROOFS
The statements-and-proofs section records the scope of the principal theorems and their formal proof contracts. The accompanying account also reports substantial AI involvement in discovery and formalization, while describing the authors' role in reviewing and reconstructing the exposition.
B.2 STATEMENTS AND PROOFS
- For both activation models, active student directions at local minima lie in the teacher span, under non-negative student masses and the stated unit-direction assumptions.The proposition applies in input dimension at least two and permits signed teacher masses.Source: §B.2 · p. 38
- For a fixed five-unit signed teacher network, under either activation, every zero-mass one-student parameter with angle in (-pi/6, pi/6) is a positive-loss local minimum on the unit-direction space despite a lower-loss one-student competitor.The formal statement also records an explicit feasible same-width competitor with strictly smaller loss, establishing genuine non-global spuriousness.Source: §B.2 · p. 39
- For odd teacher widths m ≥ 3, sharpness is witnessed by the printed explicit family; this is not a claim of attainment for every m.Source: §B.2 · p. 40
- The empirical-transfer theorem makes the accuracy threshold uniform over nonempty datasets, teacher skips, and non-negative fixed-radius minima; its conclusion is zero empirical loss.Dimension one requires no accuracy hypothesis.Source: §B.2 · p. 40
C AI USE STATEMENT
- Generative AI contributed substantially to mathematical discovery and proof search, while the authors reviewed claims and largely reconstructed the main-paper proof ideas manually.The paper reports project usage in weekly service allowances rather than complete token counts.Source: C AI USE STATEMENT · pp. 40–41
C.1 SCOPE OF THE CONTRIBUTION AND REPRODUCIBILITY
The paper presents mathematical results rather than an empirical evaluation of an AI research system, and its workflow account describes an evolving, human-guided research process. It also delineates what its verification materials and released skills establish—and what they do not document.
C.1 SCOPE OF THE CONTRIBUTION AND REPRODUCIBILITY
- The paper does not separately evaluate its agent orchestration or claim autonomous discovery, comparative model performance, proof-search success rates, or research-efficiency gains.Its contribution is mathematical results, not an empirical evaluation of an AI research system.Source: §C.1 · p. 41
C.2 PROGRESSION OF THE RESEARCH WORKFLOW
- The later workflow combined auto-formalization, shared-library proof search, numerical exploration, and fresh searches for simpler proofs, with human guidance on questions and proposed claims.The phases describe project progression, not a single fixed serial pipeline.Source: §C.2 · pp. 41–42
- Early text-only proof search relied on informal agent verification, and hallucinations and verification failures prompted a move to formal proof checking.That phase also surfaced the ring-equation analogy described for Theorem 19.Source: §C.2 · p. 41
- Generated conventional proof drafts remained difficult to read, so the authors substantially reorganized the arguments and wrote the main-paper proof ideas largely manually before re-formalization.The authors caution that this project experience is not a systematic evaluation of models' mathematical writing capabilities.Source: §C.2 · p. 42
C.3 VERIFICATION AND AUTHOR RESPONSIBILITY
- Certified-development admission required complete Lean proofs without sorry/admit, project-specific axioms, or unchecked computational shortcuts, with formal checks supplemented by semantic review.The authors remain responsible for the exposition and for claims about the certificates.Source: §C.3 · p. 42
C.4 RELEASED SKILLS AND LIMITS OF THE WORKFLOW RECORD
- Released agent skills are final versions, not a complete chronological record of prompts or interactions; without workflow benchmarks or ablations, their superiority or sufficiency for autonomous discovery is unsupported.The authors distinguish that limitation from verification of the mathematical results.Source: §C.4 · pp. 42–43