PaperlayerView exact revision on arXiv

Source-linked AI summary

Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus

Akash Kumar

arXiv:2609.39408v1cs.LGmath.OCstat.ML

TL;DR

Population loss can stay nearly constant even as a network’s representation improves, raising the question of whether subspace recovery and predictive gains can precede loss reduction. The paper analyzes simultaneous population gradient descent and complementary SwiGLU dynamics. It proves that these forms of progress can occur during a loss plateau, under the stated settings and assumptions.

  • Problem

    The paper asks whether nonlinear training can recover every direction of a target subspace and produce more predictive features while population loss remains high.

    Source: §1 · p. 2

  • Method

    The paper studies simultaneous population gradient descent on ReLU and leaky-ReLU networks, tracking predictor AGOP alignment and same-budget feature refits, and analyzes leading-AGOP alignment for SwiGLU.

    Source: §1 · p. 2, §6 · p. 13

  • Results

    During the plateau, ReLU and leaky-ReLU minimum AGOP alignment improves by at least 1/2 and same-budget refit MSE decreases by more than 0.399; the same trajectory later lowers trained loss.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    Loss alone can miss substantial representational progress, so tracking AGOP geometry and a controlled refit separately distinguishes acquired subspaces from predictive information accessible in the features.

    Source: §6 · p. 13, §7 · p. 14

    The quantitative ReLU prescriptions are sufficient existence conditions, population gradients require further work for finite-sample guarantees, and accessible-scale experiments do not verify all theorem hypotheses.

    Source: §6 · p. 14

Abstract

from arXiv · show

Population loss can remain nearly constant while a neural network learns a substantially more predictive representation. We establish this separation for two-layer ReLU and leaky-ReLU networks trained on Gaussian inputs by simultaneous fixed-step population gradient descent on all parameters. For structured additive teachers whose links are positive mixtures of Gaussian-damped cubics in $H^1(\gamma)$, we give explicit conditions under which small IID Gaussian initialization yields a high-probability guarantee: at a checkpoint during a high-loss plateau, minimum alignment between the rank-$r$ teacher subspace and the leading $r$-dimensional eigenspace of the predictor's average gradient outer product (AGOP) increases by at least $1/2$, and the minimum refit MSE under unchanged coefficient budgets decreases by more than $0.399$, both relative to initialization. The same trajectory subsequently attains a trained loss below every value in the plateau window. A complementary result treats unequal-weight cubic teachers and small additive Sobolev perturbations using projected-feature refits. For SwiGLU networks with an exactly fitted intercept, we prove leading-AGOP alignment during a loss plateau at fixed width and dimension as Gaussian initialization vanishes, for square-integrable teachers with nonzero Hermite content of degree one, two, or three. A rank-one cubic specialization also gives simultaneous unrestricted-refit gains at a prescribed width. An approximation lower bound further shows that certain interaction targets retain nonzero error when ridge neurons are restricted to shared orthogonal axes within the teacher subspace. Population-moment experiments with ReLU students across 21 teachers and 50 initializations per teacher complement the analysis.

1 INTRODUCTION

The paper asks whether nonlinear training can recover every direction of a target subspace and improve predictive features while population loss remains high. It studies this separation with simultaneous population gradient descent and reports guarantees for ReLU and leaky-ReLU students, alongside complementary SwiGLU results and experiments.

1 INTRODUCTION

  • The analysis tracks the weakest teacher direction through the predictor’s leading AGOP eigenspace and measures predictive information by refitting the feature bank under unchanged budgets.Refitting is a diagnostic and does not enter the training updates.Source: §1 · p. 2
  • The positive learning results require structure beyond Gaussian Sobolev regularity alone; the main ReLU result uses positive mixtures of Gaussian-damped cubic links, including nonpolynomial links.The stated function class requires the link and its weak derivative to be square-integrable under the Gaussian measure.Source: §1 · p. 2
  • With high probability, ReLU and leaky-ReLU students improve minimum AGOP alignment by at least 1/2 and same-budget refit MSE by more than 0.399 during a high-loss plateau.The same trajectory later lowers its trained loss.Source: §1 · p. 2
  • For square-integrable teachers with nonzero Hermite content of degree one, two, or three, SwiGLU’s leading AGOP direction can approach the teacher subspace as initialization shrinks, while loss remains near its initial value.The result holds at fixed width, dimension, and GD step with the intercept fitted exactly.Source: §1 · p. 2
  • For the normalized SiLU(X1)X2 teacher, mean minimum-AGOP alignment rises from 0.028 to 0.986 and mean refit MSE falls from 0.829 to 0.455 by update 200 while loss remains near one.The observed refit improvement is not monotone, and the interaction teacher lies outside the ReLU plateau theorems.Source: §1 · p. 2, Figure 1 · p. 3
  • The experiments distinguish recovery of a teacher subspace from alignment of individual neurons to its axes; the quadratic teacher serves as a rotation-invariant symmetry control.Figure 2 varies teacher rank, while Figure 1’s axis-projection diagnostic retains outside-plane components and biases.Source: §1 · p. 4, Figure 2 · p. 4

2 RELATED WORK

The paper situates its results among work on early feature learning, plateaus, Gaussian multi-index learning, and gradient-based subspaces. It emphasizes a joint guarantee for simultaneous training that links AGOP recovery, bounded refitting, and later loss reduction.

Feature changes before loss reduction

  • The paper connects early progress to every-direction AGOP recovery, a bounded-readout gain, and later loss decrease on one simultaneous-GD trajectory.Related settings include early alignment (Maennel et al., 2018), silent tangent-kernel alignment, and hidden Fourier-feature amplification.Source: §2 · p. 5

Plateaus, scale, and escape

  • Plateau and scaling analyses span deep-linear dynamics, diagonal linear networks, and ReLU networks, while initialization and GD step remain substantive hypotheses here.The cited settings include delayed transitions (Saxe et al., 2014), small-initialization limits, and scaling-dependent lazy training.Source: §2 · p. 5

Gaussian multi-index learning

  • Unlike Gaussian multi-index analyses with decoupled neurons or faster link fitting, this setting trains coupled neurons, biases, and heads jointly under square loss and studies the current predictor’s AGOP.The paper describes its width prescriptions as sufficient existence bounds, not improved coverage rates.Source: §2 · p. 5

Hermite structure

  • The teacher-mixture and Sobolev-neighborhood hypotheses control directional signal and approximation by learned profiles; regularity alone is insufficient.This contrasts with Hermite information exponents studied for high-dimensional online search (Ben Arous et al., 2021).Source: §2 · p. 5

Gradient-based subspaces and AGOP

  • The paper studies AGOP eigenspace evolution during nearly constant population loss and controls the full current-head AGOP and cutoff eigengap without assuming proportionality to a weight Gram matrix.Bounded refitting separately measures predictive information; related active-subspace work includes Constantine et al. (2014) and Yuan et al. (2025).Source: §2 · p. 5

3 POPULATION TRAINING AND REPRESENTATION DIAGNOSTICS

The paper defines a population-training setup on Gaussian inputs and separates representation geometry from predictive information. It measures geometry with the predictor’s full AGOP and predictive information with a fixed-budget refit of normalized features.

Population training setup

  • The teacher depends on an unknown r-dimensional subspace of Gaussian inputs, with its projector and orthogonal complement defined by the orthonormal subspace basis.The Gaussian function norm includes the function’s squared L2 norm and squared gradient norm.Source: §3 · p. 5

Network and optimization

  • The students are two-layer ReLU or leaky-ReLU networks trained by simultaneous Euclidean gradient descent on all parameters, using exact population gradients and a fixed force step.Initialization entries are IID Gaussian with variance s^2/d; the raw step is η = mh/2.Source: §3 · p. 6

AGOP representation diagnostic

  • The geometry diagnostic is the leading rank-r eigenspace of the full predictor AGOP, which includes cross-neuron terms and current heads; minimum and mean principal-angle scores measure its alignment with the teacher subspace.The minimum score tests recovery of every teacher direction, and eigenspace alignment is separated from output amplitude.Source: §3 · p. 6

Predictive-information diagnostic

  • The normalized-feature refit measures information available in the feature bank using identical coefficient caps at each checkpoint; it does not replace the trained heads or affect training.The feature normalization is invariant when a weight and its bias are scaled together by the same positive factor.Source: §3 · p. 6

Teacher family

  • The main teacher family consists of Gaussian-damped cubic links from a positive-mixture construction, with unit-norm targets and links in the Gaussian Sobolev class.The pure h3 link is included; non-atomic mixtures can yield infinite Hermite expansions starting at degree three.Source: §3 · p. 6

SwiGLU setup

  • For SwiGLU, the intercept is fitted exactly and the profiled loss measures residual variation; all other parameters train jointly from Gaussian initialization with a fixed raw GD step.The reported SwiGLU AGOP alignment concerns its leading direction.Source: §3 · p. 7

4 SUBSPACE ACQUISITION DURING A LOSS PLATEAU

Under explicit parameter conditions, the paper proves subspace acquisition during a loss-controlled window for the stated ReLU and leaky-ReLU teacher family. A later checkpoint on the same trajectory has lower trained loss than every value in that window.

Theorem conditions

  • The theorem assumes Gaussian inputs, exact population gradients, the specified teacher, and explicit size, initialization, and step conditions.Its appendix prescription requires more than m ≥ r and d ≥ 16r/δ; compatible widths may exceed dimension, and the bounds assert no practical complexity or sample-size guarantee.Source: §4 · p. 7

Acquisition and later loss decrease

  • At an acquisition checkpoint inside the loss-controlled window, the features support a better bounded refit; later, the original predictor beats every loss value in that window.The two improvements quantify representation and trained prediction separately along the same trajectory.Source: §4 · p. 7

Subspace recovery

  • The full AGOP has a positive rank-r cutoff gap at initialization and at the acquisition checkpoint, permitting its leading subspace to be compared at both times.The supplied theorem passage states that its principal-angle scores satisfy a bound but does not include the bound’s expression.Source: §4 · p. 7

3. Refit improvement. For the identical bounded diagnostic (6),

The section quantifies refitting gains during high-loss plateaus, first for the bounded diagnostic and then for a complementary perturbed-cubic setting. It also records subsequent loss release and broader teacher conditions.

3. Refit improvement

  • The same unchanged gradient-descent trajectory later lowers its loss below every value in the plateau window.A full-energy continuation estimate supports this loss release; the simultaneous claim follows by a union bound over the stated events.Source: §3 · pp. 7–8
  • The acquisition proof tracks actual gradient-descent updates and then uses acquired biased profiles to construct a readout satisfying both coefficient caps.The signal is shown to dominate the total signed defect, including adverse heads and components outside U.Source: §3 · p. 8
  • During the plateau, the same-budget bounded refit MSE falls by more than 0.399 relative to initialization.Acquired biased profiles admit a readout satisfying both coefficient caps.Source: §3 · p. 8
  • The broader mixture theorem permits different directional measures and positive coefficients, but excludes arbitrary signed mixtures and arbitrary H1 teachers.For a common link, the coefficient ratio is at most 6/5.Source: §3 · p. 8
  • For unequal positive cubic coefficients and a small additive Sobolev perturbation, the projected diagnostic improves by more than 1/2 at both stated coefficient budgets.The teacher must remain additive on U, and the extension uses a different feature class and budget from the bounded diagnostic.Source: §3 · p. 8

5 POPULATION ILLUSTRATIONS ACROSS TEACHER LINKS

Population illustrations examine ReLU students across teacher links, while the SwiGLU results establish leading-direction alignment during a low-loss-movement window and a rank-one cubic case with simultaneous refit gains. The guarantees have specific scope: the SwiGLU alignment result need not recover every direction for higher-rank teachers, and the numerical illustrations do not test the theorem’s sufficient parameter assumptions.

5 POPULATION ILLUSTRATIONS ACROSS TEACHER LINKS

  • The ReLU illustrations use 21 normalized teachers, 50 predetermined seeds per teacher, and a 20,000-update population-GD budget; their exploratory interactions lie outside the plateau theorems.The reported ReLU loss is full population MSE, with a unit baseline from teacher normalization.Source: §5 · p. 9
  • The quadratic-gate target remains at squared L2 distance at least 1/4 from every orthonormal two-axis additive class, and a small snapping penalty alone does not prove specialization.The axis-snapped refit projects each teacher-plane component onto its nearest frame axis while preserving bias, outside-plane component, and normalization.Source: §5 · p. 9

5.1 SWIGLU: FEATURE LEARNING THROUGH A SMOOTH GATE

  • At n1 = floor(τ1/(h ε^(k−2))), loss changes by at most C ε^k across the prefix while Atop reaches at least 1−δ for the stated SwiGLU teacher class.A nonzero degree-one or degree-two Hermite component gives k = 3; when those vanish and cubic signal remains, k = 4.Source: §5.1 · pp. 9–10
  • For rank-one cubic teachers at the prescribed width, the joint SwiGLU result gives simultaneous alignment and unrestricted-refit gains at one plateau checkpoint.For pure h3 with d = 16 and m = 153, the gains can be at least 0.5 and 0.7, with asymptotic probability lower bound approximately 0.9519.Source: §5.1 · p. 10
  • Theorem 5.1 proves leading-direction alignment, not recovery of every teacher direction when r > 1; the later-loss decrease remains open for SwiGLU.Its result concerns fixed finite width and dimension as initialization vanishes, with a profiled intercept and an initialization-dependent cutoff.Source: §5.1 · p. 10
  • The SwiGLU proof attributes alignment to a growing neuron whose teacher-subspace contribution dominates the leading AGOP, while its orthogonal spatial components remain fixed.For the joint refit result, cancellation removes a quadratic contribution before normalization and a linear correction approximate h3 along the teacher direction.Source: §5.1 · p. 10

6 DISCUSSION

A loss plateau can conceal feature-geometry change and improved predictive information, but the results hold under specific teacher and training conditions. The discussion distinguishes subspace acquisition from prediction, relates the findings to prior alignment and lazy-training work, and identifies limits on the guarantees.

What a plateau can conceal

  • The plateau is a finite-window condition, not necessarily an exactly flat loss or a positive error floor.The same trajectory later decreases its trained loss.Source: §6 · p. 13

Mechanism

  • Scaling all ReLU or leaky-ReLU parameters together scales the output quadratically, allowing geometry and output amplitude to evolve on different scales.This is a dynamical mechanism, not a normalization that hides loss decrease; the fixed-step result also uses restrictive initialization and step prescriptions.Source: §6 · p. 13

Geometry and predictive information

  • The weakest-direction alignment score tests recovery of every teacher direction, while same-budget refitting separately tests whether the features support better prediction.Refitting does not change the heads used by gradient descent, and the complementary projected-feature and unrestricted-refit results use different budgets.Source: §6 · p. 13

Relation to early alignment and lazy training

  • The joint all-direction, bounded-refit, and later-loss guarantee is distinct from prior early alignment and silent alignment results.A flat loss or moving AGOP alone does not determine whether training is lazy; separation from the full initial tangent-feature model requires further analysis.Source: §6 · p. 13

Teacher regularity and neuron mixing

  • The ReLU guarantees cover explicit structured families within H1, not every Sobolev link.The extension permits controlled directional-signal changes and small Sobolev residuals; extra Hermite components can alter directional forces and affect prediction.Source: §6 · p. 13
  • For the h2 interaction example, shared orthogonal teacher-plane axes retain MSE of at least 1/4, but the bound does not establish that training selects those axes.The interaction teachers described are outside the ReLU plateau theorems, and subspace recovery need not entail specialization to teacher axes.Source: §6 · p. 13

Why study a gate?

  • The SwiGLU theorem gives leading-direction alignment at fixed width and dimension as initialization vanishes, but not all-direction recovery or later loss release.Its profiled intercept and initialization-dependent cutoff differ from the ReLU result; the rank-one refit readout can grow without bound as initialization shrinks.Source: §6 · p. 13

Limits and next questions

  • The ReLU prescriptions are sufficient existence conditions, not practical width or dimension bounds, and population-gradient guarantees leave finite-sample analysis for future work.Accessible-scale experiments illustrate mechanisms rather than verify all theorem hypotheses; broader Hermite mixtures, sharper parameter costs, and persistence after acquisition remain open problems.Source: §6 · p. 14

7 CONCLUSION

Loss alone can miss substantial representational progress. Separately tracking loss, all-direction AGOP geometry, and controlled refitting makes that progress measurable and distinguishes acquired subspaces from predictors that use them effectively.

7 CONCLUSION

  • Tracking loss, all-direction AGOP geometry, and a controlled refit separately measures representational progress and distinguishes acquired subspaces from predictors that use them effectively.The results establish this separation in explicit population regimes and identify additional conditions needed to connect feature acquisition to prediction.Source: §7 · p. 14

AI USE

AI tools assisted across formulation, modeling, proofs, experiments, interpretation, literature discovery, and manuscript preparation. The authors retain responsibility for the final work, and AI-assisted checks are not formal verification.

AI USE

  • AI tools assisted across research and writing tasks, while the authors take responsibility for the final content and note that AI-assisted checks are not formal verification.Assisted tasks included mathematical modeling, proof development, experimental design, implementation, and figure preparation.Source: AI USE · p. 15

A NUMERICAL PROTOCOL AND COMPLETE TEACHER COMPARISON

This section records the population-moment experiment protocols, a geometric approximation bound for axis-restricted features, and denser followups that reveal partial alignment and prediction gains during loss plateaus.

A NUMERICAL PROTOCOL AND COMPLETE TEACHER COMPARISON

  • Baseline experiments covered 21 teacher functions with 50 fixed seeds each, using a ReLU student with r = 2, d = 16, and m = 32.All parameters trained for 20,000 updates, and the same seed reused the same initial parameter arrays across teachers.Source: A NUMERICAL PROTOCOL AND COMPLETE… · p. 18, Table 1 · p. 20

2 Var(SiLU(S)) . (21)

  • The protocol defines per-seed loss plateaus, resolves AGOP scores using eigenvalue and eigengap thresholds, and numerically brackets constrained-refit risks.Unresolved AGOP scores contribute the interval [0, 1], while reported seed percentile bands are variability bands rather than confidence intervals.Source: §2 · p. 19

A.1 NEURON AXES AND FRAME INTERVENTIONS

  • Axis-snapping refits preserve each row’s bias, outside-plane component, and augmented denominator while keeping the original coefficient caps.A small refit gap indicates preserved refit quality, not that the original neurons were already specialized; a tested rotation grid is not exhaustive.Source: §A.1 · pp. 20–21
  • The orthogonal-frame approximation floor is 1/4 for h2 interactions and zero for the bilinear link, with the h2 minimum attained at 45°.The proposition bounds error over orthonormal frames; it is an approximation result, not a learning-dynamics theorem.Source: §A.1 · p. 21

A.2 DENSER FOLLOWUPS FOR FOUR SHORT-PREFIX TEACHERS

  • Because original rank-two spectra were unresolved for four short-prefix cohorts, followups used d = 64, m = 32, s = 0.01 and denser observations.The pilot did not establish reliable full-subspace learning; the followup configuration was selected after inspecting it and was not an independent replication.Source: §A.2 · pp. 21–22
  • All 200 followup initial states, endpoints, and saved in-prefix spectra were resolved, but weakest-direction gains were smaller than mean-alignment and refit gains.The followups therefore show partial feature and prediction improvement on the plateau, not uniform recovery of both teacher directions.Source: §A.2 · p. 22
  • A median 90–98% of the original-loss decrease during the followup plateaus came from reduced mean-plus-linear squared prediction error.Thus refit gains should not be read as isolated learning of higher Hermite components.Source: §A.2 · p. 22
  • Median ∆R/L20000 pairs were 0.43/0.054 (raw SiLU), 0.589/0.00056 (centered SiLU), 0.313/1.2e-06 (ReLU), and 0.607/0.062 (sine).Table 2 reports medians of individual-run endpoint changes; later loss is the original model at update 20,000.Source: §A.2 · p. 22, Table 2 · p. 22

B HIGHER-RANK COMPARISONS AND NEURON GEOMETRY

This section develops higher-rank comparisons of loss windows and neuron geometry, alongside conditions and arguments for minimal teacher rank and combining guarantees on a single trajectory. It also clarifies that random cutoffs do not yield finite-scale confidence bounds.

B HIGHER-RANK COMPARISONS AND NEURON GEOMETRY

  • The four high-rank SwiGLU columns end their shared 5% windows at 310, 332, 271, and 237 updates, with mean losses 0.967, 0.963, 0.962, and 0.963, respectively.The corresponding 1% window endpoints are 233, 248, 182, and 149 updates; applying the stricter absolute 10^-3 rule gives 119, 128, 64, and 41 updates on the same trajectories.Source: B HIGHER-RANK COMPARISONS AND NEURON… · p. 31, Figure 15 · p. 35
  • For nonquadratic ReLU links, subspace entry precedes teacher-axis concentration; quadratic nearest-axis statistics are rotation-invariant controls, so these diagnostics do not establish that mixing is necessary.The rank-eight comparison distinguishes entry into the teacher subspace from alignment with an individual projected axis.Source: B HIGHER-RANK COMPARISONS AND NEURON… · p. 32, Figure 14 · p. 34

C.1 POSITIVE MIXTURES OF GAUSSIAN THIRD DERIVATIVES

  • C.1 specifies public parameter choices for positive mixtures of Gaussian-damped cubics, with all raw coordinates independently Gaussian-initialized and the same fixed step applied to full-MSE gradient descent.The prescription is made before the initialization draw; smaller positive step, force allowance, and raw scale are permitted, with downstream quantities recomputed.Source: §C.1 · pp. 40–41

C.2 ADDITIVE CUBIC NEIGHBORHOOD: COMPLETE PARAMETER PACKAGE

  • C.2 allows additive targets on U whose mean, linear component, and higher tails remain in training comparisons; under the stated residual bound, U is the target’s true minimal index subspace.The parameter-choice order is acyclic, but the passage does not assert a practical scale.Source: §C.2 · p. 42

D EXACT UPDATE AND MINIMAL TEACHER RANK

  • The cubic-tensor argument shows that perturbations within the stated bounds preserve mode-one rank at least r, establishing U as the actual target’s minimal index subspace.A target measurable on a smaller linear subspace would contradict that rank bound.Source: D EXACT UPDATE AND MINIMAL TEACHER RANK · p. 43
  • For 6 ≤ N_i < 9 and ∥g_i/N_i∥_H1 < 13, every odd-degree coefficient 2j + 3 is nonzero if μ_i([1/3, 1)) > 0, with no truncation in updates or proofs.These are the stated conditions and coefficient property for the exact-update, minimal-teacher-rank result.Source: D EXACT UPDATE AND MINIMAL TEACHER RANK · p. 43

D.1 COMMON ORIGINAL-LAW EVENT ASSEMBLY

  • The common-event rule combines conclusions on the original Gaussian draw using a union bound without independence, and common-endpoint claims require the same endpoint on the same trajectory.For a coupled scale ε and positive random cutoff c, the rule also gives a limiting probability bound as ε vanishes; it does not supply a numerical finite-scale confidence bound.Source: §D.1 · p. 43

E FULL MIXTURE THEOREM AND PROOF

The full-mixture theorem gives simultaneous endpoint geometry and refit conclusions under its stated conditions. The same gradient-descent run later reaches a loss below the plateau window.

E FULL MIXTURE THEOREM AND PROOF

  • With probability at least 1 − 7δ/8, the theorem’s conclusions hold simultaneously for the original Gaussian initialization.The theorem specifies a normalized, unprojected, full-bank diagnostic with ℓ2 budget 32/J and ℓ1 budget 64√r/J.Source: E FULL MIXTURE THEOREM AND PROOF · p. 44
  • A finite later checkpoint N2 satisfies L_N2 ≤ 1 − 2G_c, so every n ≤ N_c has L_n − L_N2 ≥ G_c under the same update.The theorem also states physical-clock bounds for the trajectory.Source: E FULL MIXTURE THEOREM AND PROOF · p. 44
  • At the endpoint, the full current-head AGOP has a positive rank-r cutoff gap, establishing the theorem’s spectral conclusion.The proof applies a spectral lemma with a positive cutoff gap.Source: E FULL MIXTURE THEOREM AND PROOF · pp. 44–45
  • Initialization has R(0) ≥ 1 − χ0/8, while a legal terminal witness has L2 error below 3/5 + C1e* + ζ.The initial bound uses the central mixture’s Hermite rank of at least three; the terminal witness retains the full infinite Hermite tail.Source: E FULL MIXTURE THEOREM AND PROOF · p. 45
  • The proof’s probability guarantee combines acquisition, initial-projector, and initial-refit events on the same run and public checkpoint N.Their failure bounds are 3δ/8, δ/4, and δ/4, respectively.Source: E FULL MIXTURE THEOREM AND PROOF · p. 45

F THE ADDITIVE CUBIC-NEIGHBORHOOD EXTENSION

The additive cubic-neighborhood extension establishes actual AGOP acquisition and subsequent loss decrease under its stated conditions. It also sets out the normalized additive-teacher conventions and a Gaussian Sobolev trace bound.

F THE ADDITIVE CUBIC-NEIGHBORHOOD EXTENSION

  • At the common endpoint, the minimum principal-angle improvement is at least 3/4 − ε ≥ 1/2, and the endpoint AGOP has a positive rank-r cutoff gap.The theorem states that the endpoint is public and that the leading rank-r projector is genuine.Source: F THE ADDITIVE CUBIC-NEIGHBORHOOD… · pp. 45–46
  • The projected-feature refit is a population prediction-risk comparison, not a classification-accuracy or finite-sample guarantee, and it does not update trained heads.Its coefficient budget scales as 1/s and is identical at both checkpoints.Source: F THE ADDITIVE CUBIC-NEIGHBORHOOD… · p. 46
  • An acquired dictionary time J satisfies ηJ ≥ (m/8) log(1024/5), with complete A_sub > .9 persisting through N.The minimum-AGOP claim is at the public endpoint N, not an earlier hitting time or a whole-suffix minimum-angle claim.Source: F THE ADDITIVE CUBIC-NEIGHBORHOOD… · p. 46
  • The unchanged full-MSE update later reaches L ≤ 1 − 2G_rel, at least G_rel below every prefix checkpoint’s loss.This bound uses the complete trained bank and full teacher.Source: F THE ADDITIVE CUBIC-NEIGHBORHOOD… · p. 48

G.1 CONVENTIONS AND THE NORMALIZED ADDITIVE TEACHER

  • The normalized additive-teacher conventions define Gaussian L2 and derivative-based sizes, while the weak Gaussian Sobolev space uses the H_y norm.The passage gives H_y = (A^2 + S)^(1/2) and defines A and S for the teacher.Source: §G.1 · p. 48

G.2 A GAUSSIAN HILBERT-SPACE TRACE BOUND

  • The Gaussian Hilbert-space trace result gives a pointwise bound for H1 functions and a canonical L2 trace for Gaussian Sobolev functions along a unit direction.The trace statement applies to y in H1(γ_d).Source: §G.2 · p. 48

G.3 UNIFORM FULL-PARAMETER CURVATURE, INCLUDING PURE-BIAS ROWS

This section develops dimension-free curvature, regularity, and spectral tools for ReLU features and coupled population-gradient dynamics, then uses them to establish refit gains and the acquisition and persistence of mixture-specific profiles under full-parameter training. It also gives full-horizon profile and loss bounds, a static ceiling for the full model, and bias-weight control of signed components, including pure-bias rows.

G.3 UNIFORM FULL-PARAMETER CURVATURE, INCLUDING PURE-BIAS ROWS

  • For every y in H1(γd), Cy is degree-one homogeneous and C2 away from zero, with dimension-free value, gradient, and Hessian bounds, including pure-bias rows.The Hessian bound scales as By = 4(1 − α)Hy; no lower spatial-radius bound, bound on |b|/||w||, or exclusion of aligned teacher directions is needed.Source: §G.3 · p. 49

G.4 FULL UNCENTERED HERMITE KERNEL AND MEANING OF DIFFERENTIATION

  • The full uncentered Hermite series represents the original correlation and its actual parameter derivatives through order two converge uniformly on compact sets away from zero, including aligned directions.Under H1 alone, this does not imply existence of artificial off-sphere partial derivatives at aligned directions.Source: §G.4 · p. 50

G.5 INTERFACES FOR COUPLED EUCLIDEAN POPULATION GD

  • The raw coupled-bank interface bounds Fx, its gradient, the feature norm, Vx, and the quotient gradient on retained full coordinates; for β = 2, the interpolation-segment quotient constant is 8 + 2By.Rows with both head and hidden parameters zero may be omitted; the interface does not claim C2 regularity at a zero hidden row with nonzero head.Source: §G.5 · p. 51

H.1 POPULATION MODEL AND GENUINE INITIAL EIGENDIRECTIONS

  • With probability one, the initial spatial AGOP has rank min(m, d) and pairwise distinct positive eigenvalues, giving a unique leading rank-q projector whenever q < d.This is qualitative nondegeneracy and does not give a lower bound on the boundary gap.Source: §H.1 · p. 52

H.2 HAAR LAW, MEAN/MIN STATISTICS AND INITIAL HEADROOM

  • The initial leading rank-r AGOP projector is Haar distributed, and an explicit size regime gives mean and minimum teacher-subspace alignment at most 1/4 with probability at least 1 − δ0.The regime provides initial headroom but does not show that a half-unit improvement is attained; no upper width restriction occurs in the stated bounds.Source: §H.2 · p. 53

H.3 HERMITE CORRELATION AND BOUNDED READOUTS

  • For a teacher with Hermite content only from degree k0 onward, any square-integrable one-dimensional readout has correlation with y bounded by its L2 norm times t^k0 + ζ, where t is its direction's teacher-subspace projection norm.A pure-bias feature has t = 0.Source: §H.3 · p. 53
  • R(0) − R(1) ≥ 1 − 2B1ϵ − ρ2 when a legal endpoint readout has L2 error at most ρ.The initial risk is at least 1 − 2B1ϵ; the bound requires no feature independence or Gram conditioning.Source: §H.3 · p. 54

H.4 THE ORIGINAL GAUSSIAN BANK

  • With probability at least 1 − δI, the original Gaussian bank has R(0) ≥ 1 − 2B1{(2QI/d)^(k0/2) + ζ}, with no width ceiling.The obstruction covers adaptive feasible readouts while retaining all existing ℓ2 caps; it adds no intercept or projection.Source: §H.4 · p. 54

I.1 FULL LINKS AND UNCHANGED ACTUAL DYNAMICS

  • The full links retain their actual mean, linear part, and tail coefficients, and the raw-GD argument applies away from the augmented origin, including pure-bias states.The optional residual does not cover arbitrary order-one mean or linear components; at µi = δ1/3 the normalized link is more than one L2 unit from h3.Source: §I.1 · p. 55

I.2 EXACT SELECTED-CONE DYNAMICS, INCLUDING CANDIDATE EXITS

  • Selected-axis candidates preserve their cone and upper-bias barriers through full updates, while the radial profile obeys O+ ≤ (1 − t/3)O + tM²ν/k.The argument pays for force displacement and repeated bias transitions; no favorable sign or cone is assumed for other rows.Source: §I.2 · pp. 56–57

I.3 FAVORABLE GROWTH AND A LOWER BIAS BARRIER

  • Every original row satisfies Q+ ≥ Q − hν², and selected rows maintain favorable growth and a lower bias barrier under the stated force bounds.The selected-row estimates preserve the head sign and give p+ ≥ p + 3hk/4; the radius bound is persistent rather than assumed monotone.Source: §I.3 · p. 57

I.4 ACTUAL CONVERGENCE TO THE LINK-DEPENDENT BIAS

  • The selected-row bias converges to its link-dependent target B*i within an explicit error bound at the actual integer checkpoint.The bound includes intermediate bias transitions and uses a radius bound acquired before the late phase.Source: §I.4 · p. 58

I.5 ORIGINAL GAUSSIAN EVENT AND FULL COUPLED-FORCE CLOSURE

  • The unscreened original Gaussian draw supplies distinct designated rows for every required category with high probability, without restricting the remaining coordinates.The selected rows meet the stated initial cone, balance, and correlation conditions, while Gaussian and chi-square tail bounds control all original rows.Source: §I.5 · p. 58

I.6 ONE FULL-HORIZON INDUCTION AND ITS PERSISTENT PROFILES

  • One full-horizon induction closes the coupled-force bounds and preserves the selected-row profiles throughout the interacting-bank trajectory.The induction keeps the signs, curvature quotient, cone, and bias barriers, with the cumulative quadratic bounds holding for every original row.Source: §I.6 · p. 59
  • The full-horizon induction keeps late profiles within ε of their targets and preserves angular, loss, head, and spatial-motion bounds across the stated horizon.It also yields p/ρ > 19/20 and B ≥ 1/16 at n0 and thereafter, without requiring monotonicity of ρ.Source: §I.6 · p. 59

J.1 FULL MODEL AND THE SHARP STATIC CEILING

  • The full-model static ceiling is sharp uniformly over the family, with equality for a single cubic axis at B = a and p^2 = (1 + a)ρ^2.The stated numerical bound includes Θ(29/20) = 0.821163… < 411/500.Source: §J.1 · p. 60

J.2 A BOUNDED BIAS WEIGHT FOR EVERY SIGNED COMPONENT

  • The cited result states L ≥ Δ and R ≥ Δ, and gives their sum as L + R = (3 + B)t − Bt² (SB12).On [0, 3/2], 1/5 ≤ c ≤ 1/4 and c is Lipschitz with constant two; the negative coordinate itself pays its potentially expanding radial contribution.Source: §J.2 · p. 61

J.3 THE ACTUAL CANDIDATE, INCLUDING EVERY ERROR AND CROSSING

This section analyzes the actual coupled training trajectory through finite-step bounds, profile constructions, and deterministic arguments, accounting for update errors, sign crossings, and complete-bank entry thresholds. It connects signal acquisition and conditional AGOP alignment to prediction bounds and later loss release, while also treating cubic row growth and distinguishing loss improvement or parameter motion from alignment and refit guarantees.

J.3 THE ACTUAL CANDIDATE, INCLUDING EVERY ERROR AND CROSSING

  • For the actual force, compatible displacements hν∥θold∥ and hνpold cost less than 20hνV, proving SB18 for all coordinate sign crossings; teacher and full student/residual terms remain in the same dynamics.Only the pre-state must be in SB10; the candidate can cross its bias or energy boundary.Source: §J.3 · p. 62

J.4 A DISCRETE ENTRY ENVELOPE FOR THE COMPLETE BANK

  • The entry-aware envelope bounds every row across marking and threshold crossings, with X_j,l ≤ A_j,l throughout and Z_j,l ≤ A_j,l after marking and reaching 480.Selected rows retain Q_j > .832K*; rows are marked at Q_j ≥ .826K* and thereafter have Q_j ≥ .825K*.Source: §J.4 · p. 63

K THE COMPLETE SYMMETRIC DERIVATIVE FRAME

  • The complete symmetric derivative frame uses normalized student rows and an orientation-dependent defect combining distance outside the teacher subspace with a bias contribution.The frame is defined for the complete student, rather than a selected subset of rows.Source: K THE COMPLETE SYMMETRIC DERIVATIVE… · p. 64

K.1 A SYMMETRIC SIGNED FRAME FROM FIXED QUADRATIC TESTS

  • Centered quadratic tests yield an exactly symmetric current-head frame, whose negative signed-row contributions are paid by the BF24 debit without angular-separation or common-head-sign assumptions.Groups of near-axis profiles give a positive least-eigenvalue contribution when the scalar lower bound is positive.Source: §K.1 · p. 64

K.2 FULL AGOP, OUTSIDE GRADIENT, AND A CONDITIONAL SPECTRAL JOIN

  • When the outside trace β is below the inside bound a, the full AGOP has a cutoff gap of at least a − β and minimum teacher-subspace alignment at least 1 − β/a.The comparison uses the full AGOP, including cross blocks and all trained rows; when r = d, the angle assertions hold automatically.Source: §K.2 · p. 65

L.1 THE FULL MIXTURE AND ITS EXACT SCALAR PROJECTION

  • For every mixture measure and eligible link-dependent bias, the full nonconstant projection energy satisfies E_b(y_μ) > 2/5.The unexplained infinite tail has exactly squared norm 1 − E_b(y_μ); the result uses the full link rather than a truncation.Source: §L.1 · p. 66

L.2 A FINITE EXACT-RATIONAL CERTIFICATE FOR MP8

  • The reported rational checks give 39/1000 < Vb < 13/250 and pb < 57/100, with lower-expression check at b = .51 and upper-expression and P+ checks at b = .787.Source: §L.1 · p. 66, §L.2 · p. 67

L.3 ONE EXACT FOUR-PROFILE IDENTITY

  • Four biased profiles yield an exact representation with an ℓ1 coefficient budget of 2/J.The construction uses D_τ(z) = (1 + α)z + τJ clip(z, −b, b); squaring its four coefficients cancels the mixed terms.Source: §L.3 · pp. 67–68

L.4 NORMALIZED, UNPROJECTED MIXTURE APPLICATION

  • The normalized full central mixture is unit norm, centered, and of Hermite rank at least three; with nonzero coefficients its minimal linear index space is span(U).The application retains the full target, charges rank and leak explicitly, and gives the diagnostic class budgets a bound of 64√r/J; no uniform bound holds as α approaches 1.Source: §L.4 · pp. 68–69

M.1 ACTUAL SIMULTANEOUS UPDATE AND TWO COMPLETE-ROW INEQUALITIES

  • For every row sign, the exact simultaneous update preserves balance and nonzero hidden states, bounds parameter motion and energy growth, and controls the energy increment relative to 4hΔH.The bounds include D⁺ ≥ (1 − h²∥T + E∥²)D and V⁺ ≤ (1 + 2h)²V.Source: §M.1 · pp. 69–70

M.2 A PUBLIC SUFFIX THAT PAYS ALL UNMARKED ENERGY

  • Given an actual signal row at entry, a public suffix keeps the full force capped and rows trained and balanced, then pays for all unmarked energy to establish a complete-bank central signal.The selected signal row grows throughout the suffix; rows crossing the threshold join the marked set, while the comparison is needed only for rows that never cross.Source: §M.2 · pp. 70–71

M.3 FULL ORIGINAL LOSS, UNCHANGED RATE, AND EXPLICIT GAIN

  • At a later first radius hit K, the full-MSE result gives L_K ≤ 1 − 2G_sig and L_n − L_K ≥ G_sig for every n ≤ M.The continuation uses the original full-MSE rate after the suffix; it asserts no retained alignment or refit accuracy during continuation.Source: §M.3 · p. 71

N.1 RAW COORDINATES, FULL TEACHER, AND EXACT ROW BALANCE

  • The raw-coordinate setup retains the full teacher in every expectation, uses one shared Euclidean step, and transports exact row balance through a preceding actual loss band under its stated step-size condition.The averaged model uses κ = 1/m; the update’s full-MSE raw rate is η/2.Source: N SAME-STEP RELEASE OF THE ORIGINAL… · p. 72, §N.1 · p. 72

N.2 THE SHARED REGULARITY INTERFACE

  • The full-teacher kink kernel is C2 away from zero rows and supplies global bounds on F, its gradient, the feature norm, and V.The Hessian bound applies where every nonzero row has |q_j| ≤ 2∥θ_j∥; fixed zero rows are omitted and remain zero.Source: §N.2 · p. 72

N.3 A STRONGER SAME-STEP DETERMINISTIC RELEASE

  • At the permitted maximum radius, positive balanced H1 entry yields an original-risk gain of a²/(2κ²) at a finite first crossing, without a reset or selected sub-bank.The result continues the identical simultaneous raw gradient-descent updates.Source: §N.3 · p. 73
  • At the radius hit, the release gives 1 − L ≥ aR_*^2 and joint motion of at least R_* − R_0, but does not guarantee spatial motion or alignment.For nonzero-mean targets, the retained head/bias bound is |K(w, b)| ≤ ||w|| + |μ||b|; a constant target can improve with W = 0.Source: §N.3 · p. 74

O CUBIC ROW GROWTH AND SELECTED-CONE INVARIANCE

  • The cubic selected-cone analysis treats one category representative with |C_0| ≥ k, q_0C_0 > 0, ρ_0 ∈ [1/2, 2], and initial joint norm below 3.It also assumes |q_0| ≤ ||θ_0|| and B_0 ≤ 1/16; the passage identifies the temporary radius cap as the public envelope M and the category milestone as R.Source: O CUBIC ROW GROWTH AND SELECTED-CONE… · p. 75

O.1 DETERMINISTIC ACTUAL COMPATIBLE-FORCE THEOREM

  • The deterministic compatible-force theorem allows an adaptive force of norm at most ν without assuming force derivatives or independence across rows.The supplied passage introduces the exact normalized original coupled recurrence but does not provide its formula.Source: §O.1 · p. 75

O.2 DIRECT PHASE, BIAS, AND OUTSIDE-ENERGY PROOF

This section develops discrete invariants and conditional bounds for simultaneous updates, tracking row correlation, energy, phase, bias, projected-cone retention, and full-bank correlation. It also specifies the checkpoint and step-size conditions required for the spatial/bias and sign-free energy and normalized-correlation arguments.

O.2 DIRECT PHASE, BIAS, AND OUTSIDE-ENERGY PROOF; O.3 INTRINSIC FORCE SUMS

  • The direct phase argument keeps p and K positive and the augmented norm nondecreasing, while force-sum bounds preserve bias sign and control reliable-row energy.The radius remains above 1/3; the reliable-row energy bound is O_n ≤ 2, and the passage cautions that other rows remain included in complete-bank scores.Source: §O.2 · pp. 75–76, §O.3 · p. 76

O.4 DIRECT INVARIANT PROJECTED CONE

  • The selected projected cone is retained through horizon N for every original cone representative, with the selected cubic term dominating the other coordinates.The proof excludes candidate failures directly rather than assuming a favorable sign or relabeling a reference basin.Source: §O.4 · p. 76

O.5 DIRECT BIAS CALIBRATION AND RETENTION AFTER A MILESTONE

  • After a row reaches 2RD, its radius remains above RD through N, while its original bias sign and affine-profile error bound persist.The radius decrease over any prefix is bounded by the full force budget, at most 1/(16d).Source: §O.5 · p. 77

P.1 ACTUAL MODEL AND COMPATIBLE FORCE

  • The actual model trains every original row, bias, and student cross term simultaneously, and exact compatible balance is its only row-ratio hypothesis.Raw coordinates use the same full-MSE step η = mh/2; initial head signs and teacher-detector lower bounds are not assumed.Source: §P.1 · pp. 77–78

P.2 CORRELATION INVARIANT AND COMPATIBLE-FORCE DEBIT

  • Under a compatible force bound ν, Q_n + 2νhn is nondecreasing; with zero force, Q itself is nondecreasing at the fixed discrete step.Source: §P.2 · pp. 78–79

P.3 A COMPENSATED ENERGY INEQUALITY VALID AT NEGATIVE CORRELATION

  • For updates with Q_l ≤ K, the energy bound is V_n ≤ V_a exp{(4K + 5ν)t + 4h(Q_n − Q_a)}, including a threshold-crossing candidate step.This compensated inequality retains energy decreases from negatively correlated rows and requires no restriction on terminal Q_n.Source: §P.3 · p. 79

P.4 PUBLIC-HORIZON IRREVERSIBLE CLASSIFICATION, INCLUDING ALL CROSSINGS

  • Permanent marking prevents a row from returning below κ, while unmarked rows retain at most (8/7) times their initial total energy.Marked rows acquire same-sign nonzero head and correlation, but the proof does not presume they are aligned; all original rows remain in the bank.Source: §P.4 · p. 80

P.5 CONDITIONAL FULL-BANK CORRELATION FROM ACTUAL ENERGY GROWTH

  • If actual bank energy reaches at least 2V0 within the public horizon, the proof derives a conditional full-bank correlation conclusion.The supplied passage states this energy-growth condition but omits the resulting inequality, so its quantitative bound is not reported here.Source: §P.5 · p. 81
  • Under t≤T and h≤1/512, the debit is at most 1/32, yielding a numerator greater than 5/8 and a denominator at most 5T; energy doubling remains a checkpoint requirement, not a proved exit time.The endpoint-force condition is also required because a first energy-crossing state may lie outside an unpadded pre-update tube.Source: §P.5 · p. 81

P.6 SPATIAL/BIAS INVARIANT FROM THE SAME ACTUAL RECURRENCE

  • Using the same history-dependent compatible error E for head and hidden updates, M=max{O^2,(b^2−I^2)} satisfies O^2≤M and b^2≤ρ^2+M.No independence or derivative of E is assumed, and the construction imposes no spatial-radius lower bound.Source: §P.6 · p. 81

P.7 SIGN-FREE ENERGY AND NORMALIZED CORRELATION

  • For ν≤1 and h≤1/128, the sign-free bound is (V−4hH)_+−(V−4hH)≤4hH+3hνV, including zero correlation and zero head.The stated scope also assumes θ≠0 and |q|≤∥θ∥; the step-size range is justified by the explicit t≤1/127 segment check, not widened by the theorem's h≤1/512 condition.Source: §P.7 · pp. 81–82
  • When |q|≤2∥θ∥, the four terms in the quotient Hessian D^2(H/V) are bounded by 17/V, 4/V, 1/V, and 4/V; these are calculus identities, not trajectory comparisons.The passage gives the component bounds as quotient calculus results.Source: §P.7 · p. 82

P.8 A MAXIMUM OF QUADRATICS HANDLES EVERY BIAS TRANSITION

This section develops controls for bias and radius transitions, original-row trajectories, and the outside AGOP, including bounds on force integrals and balance defects. It also treats simultaneous maturation of cubic-category representatives, signed cubic-frame ordering, and projected-feature refit bounds with an initial top-r baseline for acquired cubic students.

P.8 A MAXIMUM OF QUADRATICS HANDLES EVERY BIAS TRANSITION

  • When H ≥ 0, every actual candidate step raises M − 2hH by at most 3hνV, without restrictions on B or the next sign of H.This bias-energy barrier remains valid through repeated crossings of normalized bias one or three and when a spatial row vanishes.Source: §P.8 · pp. 82–83

P.9 A PUBLIC POLYNOMIAL BUDGET JOINS ALL ORIGINAL ROWS

  • Under a public step, noise, and time budget, every original row remains controlled across marking, including rows with either initial head sign.The result does not require a nonzero initial detector; an identically zero initial joint row contributes zero throughout.Source: §P.9 · p. 83

Q SIMULTANEOUS MATURATION OF THE CUBIC CATEGORIES

  • 4r distinct original category rows mature simultaneously by checkpoint J, yielding Sn > .24Vn and substantial movement from initialization.On an original Gaussian event of probability at least 1 − δ, their common maturation follows a controlled actual trajectory; the stated movement bound is ∥Wn − W0∥F > 8s√m and its ratio to ∥W0∥F exceeds 4.Source: Q SIMULTANEOUS MATURATION OF THE CUBIC… · pp. 85–86

R.1 ACTUAL RUN AND THE STRONGER FORCE INTEGRAL ALREADY PAID BY AC

  • On the original-probability event, the actual run is controlled at every integer through N, and its cone-force cutoff is stronger than the earlier coarse force-integral bound.The unchanged model retains all original means, linear terms, tails, and student interactions; no centering or resetting is used.Source: §R.1 · p. 86

R.2 PERMANENT MARKING AND A NEW BALANCE-DEFECT UPPER BOUND

  • Permanent marking preserves the same nonzero sign for a row’s head and cubic correlation, while bounding its balance defect at every marked state.The defect bound includes the marking state and is separate from the preserved nonnegative balance lower bound; the all-bias quadratic estimate also covers unmarked rows and branch crossings.Source: §R.2 · p. 87
  • At a fixed absolute entry radius, acquired all-row bounds place the first quantity above 16/25 and the second below 34/25.These are bounds on selected trajectory rows, not assumptions about unselected rows.Source: §R.2 · p. 87

R.3 THE EXACT HEAD–OUTSIDE PRODUCT CONTRACTS ABOVE THAT RADIUS

  • Above radius four, the exact outside-teacher-subspace product contracts with a favorable quadratic term retained, even under adaptive coupled errors.The estimate charges both hidden-force and head-force errors.Source: §R.3 · p. 88

R.4 ALL LOW STATES, CANDIDATE CROSSINGS, AND EVERY ORIGINAL ROW

  • Every original row is controlled across all low states and candidate crossings, yielding a complete-bank head-weighted outside-energy estimate.The induction combines high-radius bounds with bounds at other pre-step states; it does not require a single radius crossing or discard unmarked rows.Source: §R.4 · p. 88

R.5 EXACT FULL OUTSIDE AGOP AND THE REMAINING INSIDE OBLIGATION

  • The full outside-AGOP operator norm stays below 1522s^4 throughout the actual horizon, independent of M and later row-radius growth.The bound uses no cancellation and retains the linear part of leaky ReLU.Source: §R.5 · p. 89

S.1 THE EXACT MAP AND ORDERING WITHOUT A HEAD-SIGN ASSUMPTION

  • The pure map preserves coordinate signs and strict weighted-coordinate order for either head phase; forced updates preserve them when cumulative force debits stay below initial gaps.The identities apply to the actual history, including the final candidate step.Source: §S.1 · p. 90

S.2 ORIGINAL-COORDINATE ANCHORS AND THE COMPLETE SIGNED FRAME

  • The original-coordinate event has probability at least 1−δAC−ζ/2, and actual-history bounds preserve original signs and strict order through candidate endpoints.For each coordinate pair, the force-free anchor has at most a second-order pure discrete defect; paying the actual force incurs the exponential cost.Source: §S.2 · pp. 91–92
  • Every favorable row’s oriented maximum is its original extremal axis, yielding a uniform bound on head-weighted inside transverse norm across all rows.The bound is independent of the radius envelope M and clock T.Source: §S.2 · p. 93
  • The complete signed inside gradient-Hermite frame retains every row’s contribution as nonnegative diagonal mass plus an explicitly bounded error.The construction does not select a favorable subbank or change head signs.Source: §S.2 · p. 93
  • Repeated winner axes alone do not ensure coverage of every axis; acquired AC categories and a complete outside-AGOP estimate are needed before stating a full spectral theorem.Such a theorem must specify its radius choice, eigenvalue floor, gap, and angle comparison.Source: §S.2 · p. 94

T.1 EXACT INPUTS AND THE DIAGNOSTIC BEING COMPARED

  • The cubic application analyzes 4r acquired original rows under explicit alignment and eigengap assumptions, while retaining unchosen rows in training and the diagnostic bank.The target includes its actual mean, linear part, and higher tails; the result does not extend the acquired dictionary to an unrestricted H1 teacher.Source: §T.1 · p. 94

T.2 RAW, PROJECTED CUBIC APPLICATION

  • For the structured cubic witness, a projected-feature refit can use selected acquired rows while keeping biases unchanged and setting other diagnostic coefficients to zero.The full original bank remains the comparator, and the projection estimate bounds activation error using the teacher-direction alignment condition.Source: §T.2 · p. 95

T.3 A DIMENSION-FREE DISTORTION BOUND FOR THE STRUCTURED WITNESS

  • A dimension-free distortion bound controls projected-feature error through the alignment parameter µ, and the resulting endpoint risks include the full original perturbation.The argument applies to the observed PN and full trained bank; it assumes neither a later Haar law nor independence between PN and learned rows.Source: §T.3 · pp. 95–96

T.4 INITIAL TOP-R BASELINE WITHOUT A WIDTH CEILING

  • Under qr,d ≤ δI/32 and ν/2 ≤ 1/128, the initial risk is at least 1 − 9/256 for the normalized cubic benchmark.The bound holds with probability at least 1 − δI and compares the full target and complete projected bank at initialization and the later checkpoint.Source: §T.4 · pp. 96–97

U SWIGLU STUDENTS: LEADING-DIRECTION LEARNING DURING A PLATEAU

  • The SwiGLU appendix treats leading-direction alignment during a near-constant-loss interval separately from unrestricted-refit approximation and later trained-loss decrease.For teacher subspaces larger than one, leading-direction alignment is weaker than minimum top-r alignment, and a general theorem joining plateau learning to later loss release remains open.Source: U SWIGLU STUDENTS: LEADING-DIRECTION… · p. 98

U.1 WHICH TEACHER LINKS ARE COVERED?

  • SwiGLU alignment requires nonzero teacher Hermite content in degree one, two, or three; pure degree-at-least-four links are excluded, and the rank-one refit theorem requires a nonzero cubic with lower degrees absent.Arbitrary square-integrable additions supported in degrees at least four preserve the classification when a lower-degree coefficient is nonzero.Source: §U.1 · p. 98

U.2 SETUP AND LEADING-DIRECTION THEOREM

This section studies small-initialization SwiGLU dynamics through leading-AGOP alignment and population refit guarantees, including fixed-width and rank-one results with conclusions scoped by width, limit order, and refit setting. It also reports finite-initialization outcomes and numerical checks, clarifying the plateau gains’ interpretation and distinguishing the adaptive-flow extension from the fixed-step guarantee.

U.2 SETUP AND LEADING-DIRECTION THEOREM

  • On an initial plateau with loss variation O(ε^k), the leading AGOP eigenspace enters the teacher subspace at fixed width as initialization vanishes.The theorem treats both gradient flow and fixed-step population gradient descent; for k = 4, the interval scales as ε^-2 with loss variation O(ε^4), and for k = 3 as ε^-1 with variation O(ε^3).Source: §U.2 · pp. 99–100
  • The alignment result requires nonzero teacher signal in Hermite degrees one through three, but places no smallness restriction on higher-degree components.It does not cover teachers with no signal in degrees one through three, and it does not claim recovery of all teacher directions when the teacher rank exceeds one.Source: §U.2 · p. 99
  • For rank-r teachers, the theorem guarantees leading-direction alignment rather than full-subspace recovery; its r-dimensional conclusion is only an average-alignment lower bound.The stated bound is tr(PP_r)/r ≥ (1 − δ)/r, not a lower bound on the smallest principal-angle cosine.Source: §U.2 · p. 103

U.3 UNRESTRICTED POPULATION REFITTING OF SMALL SWIGLU FEATURES

  • The unrestricted-refit subsection establishes an initial-plateau population risk bound, while comparison with initialization is asserted separately only at the boundary width.All readout coefficients and the intercept are unrestricted and may diverge as initialization vanishes; the claims do not concern bounded readouts, ridge regression, or finite-sample estimators.Source: §U.3 · p. 104
  • For rank-one cubic teachers and m ≥ D, unrestricted refitting can approach the higher-Hermite tail risk arbitrarily closely at late fixed pre-escape times, with escape probability at least 1 − 2^-m.For a pure cubic teacher, the attained refit risk can be made at most any δ > 0; this is not a finite-ε confidence bound.Source: §U.3 · p. 106
  • With m ≤ D − 1, the unrestricted refit risk satisfies 0 ≤ 1 − R = Oτ(ε^2) almost surely at every deterministic fixed pre-escape time.The result does not cover adaptive times approaching escape as ε shrinks.Source: §U.3 · p. 107
  • At the boundary width m = D, the rank-one theorem yields a strictly positive, initialization-dependent asymptotic improvement from initial to endpoint refit risk, without a deterministic positive lower bound.The result holds on the escape event and assumes d ≥ 2.Source: §U.3 · pp. 107–108
  • For general-rank type-three teachers with m ≥ D, a late one-winner feature captures a teacher cubic contraction, but the bound does not establish axis entry or recovery of the full teacher subspace.The result is an asymptotic single-direction refit guarantee, not a full-subspace guarantee.Source: §U.3 · p. 109
  • The refit analysis fixes a late pre-escape time before shrinking initialization; wider networks may admit further cancellation, and no subsequent fixed-size trained-loss decrease is proved.The rank-one bound retains the higher-Hermite remainder, while the unrestricted refit witness may require coefficients growing as ε^-3.Source: §U.3 · p. 109

U.4 A JOINT RANK-ONE CONSEQUENCE

  • At boundary width m = D, a rank-one result combines plateau alignment and refit conclusions at a common endpoint; the endpoint time scales as ε^-2 and prefix loss variation is O(ε^4).The probability statement is asymptotic as ε ↓ 0 and retains sample-dependent endpoint and initialization cutoffs.Source: §U.4 · p. 110
  • For the pure h3 example with d = 16 and m = 153, the joint rank-one consequence gives gains of at least 0.5 and 0.7, with an asymptotic probability lower bound of approximately 0.9519.The bound gives neither a practical numerical initialization scale nor a finite-sample guarantee; trained loss remains near one in the certified window, and subsequent decrease is separate.Source: §U.4 · p. 110

U.5 WHAT IS NOT ESTABLISHED BY THE PLATEAU THEOREM

  • The candidate one-neuron loss-release mechanism applies only to pure h3, and the compact-time comparison does not prove that the plateau ends at the first substantial loss decrease.A transfer would require control to an initialization-independent parameter norm and characterization of the trajectory leaving the degenerate origin; higher Hermite components and finite-amplitude GD discretization may alter the dynamics.Source: §U.5 · p. 111

U.6 POPULATION-QUADRATURE EXPERIMENTS WITH A SWIGLU STUDENT

  • Experiments use population quadrature and select the plateau from loss history before evaluating alignment or refit; adaptive flow approximations do not establish a fixed-step guarantee.The integrals are numerical approximations, not certified exact values, and separate fixed-step runs test ordinary gradient descent.Source: §U.6 · pp. 111–112

U.7 FINITE-INITIALIZATION OUTCOMES

  • Leading AGOP alignment rose from 0.128, 0.022, 0.000 at initialization to 0.998, 0.994, 0.994 at primary loss-prefix endpoints for rank-one h3 seeds 0–2; all later reached ℓstop = 0.5.The threshold-crossing losses are beyond the plateau and are not unconstrained final-loss outcomes or consequences of the alignment theorem.Source: §U.7 · p. 113, Figure 21 · p. 116, Table 3 · p. 117
  • Rank-two leading-direction alignment reached 0.994–0.997 while weakest-direction alignment ranged from 0.004 to 0.976, so leading-direction recovery did not ensure full-subspace recovery.For c = (1, .6), weakest-direction alignment was 0.008, 0.067, 0.004; for c = (1, 1), it was 0.976, 0.151, 0.178.Source: §U.7 · pp. 113–114, Figure 23 · p. 118
  • At the validated width-256 reference checkpoint, corrected prediction risk fell from approximately 0.82 to 0.232107528 at loss movement 1.17993 × 10^-5; this replay was not a primary endpoint.The checkpoint replay validated the reference decrease rather than adding a seed or selecting a minimum.Source: §U.7 · p. 113, Figure 24 · p. 119
  • The fixed ∆t = 0.3 control diverged to loss approximately 6.11 × 10^237, and the retained validation script reported four failed checks; neither supports a stability or release conclusion.The failures included two finite-difference gradient discrepancies and two Monte Carlo comparisons.Source: §U.7 · p. 114, Figure 25 · p. 120
  • Table 5 records terminal observations under Section U.6 stopping rules; loss-stop values may undershoot targets because of update-grid spacing, and refit risks are numerical attained risks.The fixed .1 endpoint risk changes by .01133 when quadrature is doubled.Source: Table 5 · p. 123

U.8 TWENTY-SEED RANK-ONE EXTENSION

  • The rank-one extension covers seeds 0–19 across four configurations (80 outcomes), retains the original 12 runs, adds 68, and records 80 loss-threshold stops.The other 12 smoke, rank-two, wide-student, and fixed-step control conditions remain unchanged, bringing the combined study to 92 conditions.Source: §U.8 · p. 125
  • Runs use d = 16, m = 64, κ = 1, and ε = .05 with adaptive Euler steps and a profiled intercept; they do not instantiate the fixed-step theorem at a quantified practical initialization scale.The cubic stopping loss is .5 and the other links stop at .7, subject to ceilings of 20,000 updates and mean-field time 106.Source: §U.8 · p. 125
  • Across 207 resolved comparisons, quadrature-doubling changes were at most 1.09 × 10^-6 in normalized loss, 3.85 × 10^-8 in leading alignment, and 2.88 × 10^-3 in normalized refit risk, with 0 unresolved.These checks apply to stored states, not certified integration-error bounds; an unresolved seed would suppress its ensemble statistic rather than be dropped.Source: §U.8 · p. 125
  • Per seed, the primary plateau is the full-update initial prefix |ℓ_n − ℓ_0| ≤ 10^-3; gains use the last saved diagnostic inside it, not interpolated threshold crossings.Later loss changes are measured outcomes, not conclusions of a trained-loss release theorem, and the unrestricted numerical readout differs from the ReLU theorem’s bounded-refit comparison.Source: §U.8 · p. 125

Related papers

Loading 2609.39408v1…