PaperlayerView exact revision on arXiv

Source-linked AI summary

Flat Loss, Evolving Features: Grokking in Modular Addition

Akash Kumar, Karthik Sankaranarayanan

arXiv:2610.22593v1cs.LGstat.ML

TL;DR

The paper asks whether a network can continue changing its features and improve held-out accuracy while fitted training CE stays relatively flat and positive. It studies modular addition with normalized updates, proving coexistence for one-hidden-layer networks from constructed initializations and testing it experimentally. In a fixed-window experiment, all 20 runs retain perfect training accuracy as held-out accuracy rises above 90%.

  • Problem

    After fitting, positive CE reflects prediction confidence as well as correctness, so a flat loss does not reveal whether hidden features or held-out predictions are still changing.

    Source: §1 · p. 1

  • Method

    The paper proves finite-window coexistence for 1HL networks from constructed initializations and tests feature change and recovery with experiments using supported-Polar hidden updates and normalized head updates.

    Source: §1 · p. 2, §3.2 · p. 4, §4 · p. 6

  • Results

    On a window fixed in advance, every 1HL run keeps all training pairs correct while held-out accuracy climbs from zero to above 90% and raw training CE varies by less than 8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A flat training loss is not by itself evidence that learning has stopped; Fourier structure and a train-only refit can reveal representation progress the loss does not show.

    Source: §6 · p. 14

    The train-only refit establishes earlier decodability under its specified ridge rule, but does not show that the network’s actual head follows it or separate feature direction from scale.

    Source: §6 · p. 14

Abstract

from arXiv · show

Can a network keep learning while its training cross-entropy stays nearly flat at a positive level? We show that it can. On modular addition, a standard testbed for grokking, ReLU networks trained with supported-Polar hidden updates, normalized output-layer updates and weight decay can keep 100% training accuracy while their hidden features change and held-out accuracy rises from zero to above 90%, at relatively flat training cross-entropy. Correct classification need not minimize cross-entropy, which still depends on prediction confidence. We prove this coexistence for one-hidden-layer networks in two finite-window theorems from constructed initializations: one gives a constant gain in the Fourier fraction of the hidden features, and the other holds at arbitrarily low positive loss, where a readout refitted on the training data generalizes before the network. Experiments from random initialization show the same coexistence on a window fixed in advance, and in deeper networks held-out accuracy also rises over post-hoc windows with a net increase in training cross-entropy. With a last-layer rate set per training fraction and head decay scaled with the prime, deeper networks at p=47 and 61 meet a criterion on post-hoc windows, fixed in advance, in nearly all fresh runs. A one-block transformer trained with the same updates and a slowly decaying output gain keeps every training pair correct at every logged checkpoint while its training cross-entropy rises and its held-out accuracy approaches one; with two blocks this holds through the held-out rise, after which training pairs begin to flip. A plateau in training loss need not be a plateau in learning. Code: https://github.com/akashkumar-d/flat-loss-evolving-features.

1 Introduction

The paper asks whether features and held-out accuracy can improve while a fitted network’s positive training cross-entropy (CE) remains relatively flat. It answers yes with finite-window theorems and experiments on modular addition, tracking both feature structure and predictive usefulness.

1 Introduction

  • Because CE measures confidence as well as correctness, a fitted network can have positive loss without that loss revealing whether its features or held-out predictions are changing.The introduction situates this question in grokking, where networks fit small algorithmic training sets before generalizing [1].Source: §1 · p. 1
  • The Fourier fraction measures task-aligned structure, while a ridge readout trained only on training pairs tests whether the features already support held-out prediction.The refit is not installed in the network.Source: §1 · p. 2
  • On a window fixed in advance, all 20 runs keep every training pair correct while held-out accuracy rises from zero to above 90% and training CE varies by less than 8%.The train-only refit reaches 90% more than 3,000 updates before the network, while the Fourier fraction rises from about 0.02 to 0.12.Source: §1 · p. 2, Figure 1 · p. 2
  • Constructed initializations yield finite-window theorems for 1HL networks that jointly guarantee flat positive CE, increasing Fourier fraction, and held-out recovery.One theorem works near log p on a structured split; the other works at an arbitrarily small prescribed positive loss on a random split, with refit-first recovery.Source: §1 · p. 2
  • The theorems start from constructed open sets of initializations; entry into those regions from iid initialization is not proved.The experiments probe behavior beyond the constructions.Source: §1 · p. 2

2 Related Work

Prior work studies grokking through representation formation, weight dynamics, circuit competition, implicit bias, and Fourier structure. This paper distinguishes its focus on feature change and held-out recovery during finite windows of positive, relatively flat CE.

Grokking and modular arithmetic

  • Prior grokking explanations include representation formation [10], weight-norm dynamics [5], competing circuits [11], transitions between training regimes [6], and implicit biases [7].For shallow modular addition, prior work studies margin maximization [8], sample complexity [15], and training dynamics [16] [17] [18].Source: §2 · p. 3

Initialization scale and weight decay

  • Earlier weight-decay studies follow weight norms and test loss as training loss becomes small, whereas this paper tracks features and positive training CE on finite windows.The paper also shares weight decay with prior approaches; its one-hidden-layer experiments start from a large first layer, and its transformer output gain relaxes from twice equilibrium.Source: §2 · p. 3

Fourier features and margin maximization

  • Prior Fourier work describes circuits, explicit solutions, maximum-margin structure, or feature-matrix deviation; this paper tracks task-aligned hidden-activation energy along training at positive loss.Its Fourier fraction can be nonzero even at the maximum-margin network, where AH = 2/5.Source: §2 · p. 3

Hidden progress and plateaus

  • Prior hidden-progress results include sparse parity and teacher-student regression, while this paper studies post-fit classification recovery under CE and proves joint finite-window bounds.The constructions control CE variation, Fourier fraction, and predictions together; the random-split theorem certifies that a train-only refit recovers before the network without becoming its head.Source: §2 · p. 3

Orthogonalized updates

  • The feature update uses a rescaled supported Polar factor without momentum, connecting it to spectral descent and Muon [9].Unlike work on shortening grokking delays [29] [30], this paper studies representation change and held-out recovery during relatively flat loss.Source: §2 · p. 3

3 Problem Setup

The setup defines modular addition, the network and update rules, and measures for fitting, loss flatness, feature structure, and predictive usefulness. These measures distinguish training correctness and confidence from representation change and held-out recovery.

3.1 Task, model, and training objective

  • The task is addition modulo an odd prime p: each pair is encoded by concatenated one-hot vectors, and a fixed training subset is held out from the p^2 input pairs.The theory uses a bias-free one-hidden-layer ReLU network and mean cross-entropy; the hidden activation table records features over the full grid without enlarging the training set.Source: §3.1 · p. 4

3.2 Fixed normalized updates

  • Hidden weights take supported-Polar steps and the output head takes normalized-gradient steps, both based on negative CE gradients at the same state.For nonzero gradients, both normalizations have unit Frobenius norm; the Polar step moves equally along gradient singular directions, while the head step preserves their ratios. Fixed-length steps can keep weights moving during flat loss, but do not guarantee flat loss or useful motion.Source: §3.2 · p. 4

3.3 Fitting, plateaus, and delayed recovery

  • Training is fitted when every training margin is positive, while a 10% plateau bounds raw CE’s relative variation at every update without asserting stationarity or optimality.Delayed recovery means held-out accuracy becomes high after fitted training with low held-out accuracy; coexistence claims concern feature change and recovery on the same window only.Source: §3.3 · p. 5

3.4 The Fourier fraction

  • The Fourier fraction AH is the share of centered hidden-activation energy in mean-zero functions of a + b, the task-aligned addition subspace.It is scale-invariant, can rise through increased task-aligned energy or reduced remaining energy, and can change without a change in feature span; a 0.10 gain means ten percentage points.Source: §3.4 · p. 5

3.5 The train-only refit

  • A separate ridge head is fitted on training pairs alone and evaluated on held-out pairs without being installed in the network or changing training.It measures what ridge regression extracts from the features, which can differ from the network head’s predictions.Source: §3.5 · p. 6

4 Main Theoretical Results

The theorems establish that one-hidden-layer networks from constructed initializations can retain perfect training accuracy while held-out accuracy recovers during a CE plateau. They do so for a structured split near log p loss and a random split at arbitrarily small positive loss.

4 Main Theoretical Results

  • The guarantees apply to constructed initialization sets; entry into those sets from iid weights is not established.The random-split construction also builds its initialization around the realized training data.Source: §4 · p. 6, §4.2 · p. 8

4.1 A constant Fourier gain on a structured half-grid

  • Helper and query unit decay keeps training logits fixed while suppressing harmful held-out logits; helper decay also removes energy outside Fadd.These mechanisms support held-out recovery and the net Fourier-fraction gain.Source: §4.1 · p. 7
  • On a structured half-grid, training accuracy stays 1 with v<0.0045 as held-out accuracy rises from zero to one and AH gains >0.0998.The theorem places these changes on a single finite training window.Source: §4.1 · p. 7

4.2 Random splitting and refit-first recovery

  • On independent 4/5 pair splits, training accuracy stays 1 with v<0.1 at arbitrarily small positive CE; the refit is accurate from update 1, while the network recovers only at J.The refit is fitted on training data and is not installed in the network.Source: §4.2 · pp. 7–8
  • Because initial features already span every grid function, refit improvement comes from selecting a better predictor, not from gaining new feature span.The prescribed ridge fit is accurate from the first update.Source: §4.2 · p. 8

5 Experiments

Experiments test whether the proved coexistence also appears from iid initialization at practical widths, across one-hidden-layer and deeper ReLU networks and transformers. The results show held-out recovery alongside changing features and relatively flat or rising training cross-entropy, with outcomes depending on architecture and evaluation window.

5 Experiments

  • Experiments extend the theorem’s constructed-initialization setting to iid initialization at practical widths, but the theory’s sufficient conditions for head decay do not cover the cumulative decay used in Figure 1.The Figure 1 cumulative decay is 45, outside the construction-specific sufficient conditions described for the theory.Source: §5 · p. 9
  • The primary one-hidden-layer evaluation uses a window and configuration locked before its 20 fresh initializations and splits; deeper-network primary windows are selected by a loss-only rule, while other shaded windows are post-hoc.The post-hoc windows contain the largest rise in cycle-mean loss after the knee while staying within 3% of their running maximum; they do not make a flatness claim except in the Figure 4 criterion.Source: §5.1 · p. 10, §5 · p. 10, §5.2 · p. 11

5.1 One-hidden-layer ReLU networks

  • Across 20 one-hidden-layer runs on a window fixed in advance, training accuracy stays at 100% while cross-entropy varies by 6.33–7.81%, AH rises from 0.018–0.020 to 0.115–0.116, and held-out accuracy reaches 91.7–98.4%.The training-only refit first reaches 90% accuracy at update 6,500 in every run, before the network reaches 90% at updates 9,631–11,665.Source: §5.1 · p. 10

5.2 Extension to deeper ReLU networks

  • At p = 31, depths 3, 4, and 5 meet the primary endpoint in 40/40, 39/40, and 38/40 runs, respectively, while depth 2 passes in 34/40.All 120 runs at depths 3–5 finish with perfect held-out accuracy; with a normalized-gradient feature update, networks generalize but their loss keeps falling during recovery.Source: §5.2 · p. 12
  • With the last-layer multiplier retuned by training fraction and head decay scaled with p, depths 3 and 4 meet a post-hoc-window criterion in 155 of 160 fresh runs at p = 47 and 61.These windows were fixed before the runs, but the primary endpoint holds in only 9 of 160 runs; at depth 3, p = 47, no training pair is misclassified inside any window.Source: Figure 4 · p. 11, §5.2 · pp. 12–13

5.3 Transformers

  • For one-block transformers, all 10 runs pass at p = 113: training cross-entropy rises by a median 33% across the no-mirror held-out rise, and final held-out accuracy is at least 0.999.AH of the normalized head input gains 0.78–0.83 inside the window; with the output gain fixed at 1.5, 2, or 3, no run passes.Source: §5.3 · p. 13
  • For the transformer, the full train-only ridge refit trails the network at its 50% crossing in every run, whereas restricting the refit to the eight frequencies with most training energy makes it lead in all 10 runs.The unrestricted refit weights all 56 label frequencies equally, while the normalized head input carries 7–13 frequencies at the network’s 50% crossing.Source: §5.3 · p. 13
  • The one-block transformer also passes in 10/10 runs at p = 89, but two-block runs pass in none of the planned settings and eventually show training errors after held-out accuracy first reaches 0.99.For two blocks, first training errors are logged 1.4k–5.7k updates later, after which held-out accuracy falls back; six of 10 runs end below 0.99.Source: Figure 5 · p. 12, §5.3 · p. 13

6 Discussion

Relatively flat positive training cross-entropy can coexist with evolving features and delayed generalization, though the theoretical guarantees cover constructed initial regions for one-hidden-layer networks. The discussion distinguishes what the feature diagnostics reveal from what they cannot establish and notes limits in extending the results to deeper networks and transformers.

Discussion

  • A flat training loss alone does not show that learning has stopped; Fourier structure and train-only refits can reveal progress it misses.Once all training pairs are correct, cross-entropy measures confidence rather than correctness.Source: §6 · p. 14
  • The Fourier fraction tracks representation structure, but ignores decoder signs and class coverage, so it cannot alone guarantee better margins or test accuracy.A train-only refit tests predictive usefulness, but earlier refit recovery does not show that the network head follows it or separate feature direction from scale.Source: §6 · p. 14
  • The theory gives only qualitative bounds on its initialization basin, positive head step, and decay allowance, and its width extension uses inactive padding.Neither theorem explains how iid training reaches the constructed regions.Source: §6 · p. 14
  • Deeper-network configurations rely on hyperparameter search and tuning, and at p = 47 and 61 the specified setup meets a post-hoc criterion but not the primary endpoint.It uses a last-layer rate set per training fraction and head decay scaled with the prime.Source: §6 · p. 14
  • The one-block transformer depends on a slowly decaying output gain; with two blocks, training fit is lost after the held-out rise, and these cases lack theory.Without the output gain decay, training cross-entropy falls during the one-block held-out rise.Source: §6 · p. 14

AI use statement

Large language model assistants supported several stages of the work, and the authors state that they reviewed the assisted work and take responsibility for the final content.

AI use statement

  • Large language model assistants supported development of initial ideas, mathematical claims and hypotheses, proof development, and writing.The authors also used them for experimental design, implementation, and interpretation.Source: AI use statement · p. 14
  • Generative AI tools also assisted with literature and reference checks and drafting and editing parts of the paper.The authors say they reviewed all AI-assisted work.Source: AI use statement · p. 14
  • The authors take responsibility for the final paper content, including text, claims, and artifacts produced with AI assistance.Source: AI use statement · p. 14

Appendices

The appendices define feature-structure and readout diagnostics, then distinguish their information from the signed quantities used to reason about classification. They also specify observation cadence and cautions for interpreting refit and margin results.

A.1 Fourier features, projection, and energy

  • The addition-Fourier fraction measures how much centered hidden-activation energy lies in mean-zero functions of a + b.The appendix relates the addition modes to a projector and distinguishes absolute addition-energy growth from a fraction gain caused by decreasing other energy.Source: §A.1 · p. 18
  • Hidden-span availability measures how much of the rule space the centered hidden span contains, without weighting feature amplitudes.It can be one while the Fourier fraction and readout performance change; the appendix gives 1/(p + 1) as a geometric reference for isotropic hidden-feature directions.Source: §A.1 · p. 18

A.2 Actual prediction, train-only refitting, and signed rule margins

  • The train-only ridge diagnostic scores a unique refitted solution using all hidden columns, with empirical penalty α = 10^-3 in mean-loss units.The empirical diagnostic uses a fixed α, while the theoretical penalties scale with training-feature energy.Source: §A.2 · p. 19
  • Legacy deep-development refits center training features and targets differently, and their fixed penalty is not invariant to feature scale.A better training refit objective alone does not imply better held-out accuracy.Source: §A.2 · p. 19
  • The signed rule and adverse residual contrasts bound the margin, while a gain in Fourier fraction alone guarantees neither larger margins nor better test accuracy.The residual can help classification, and the theorems use separate signed-prediction arguments rather than a universal alignment threshold.Source: §A.2 · p. 19
  • Headline refit observations every 500 updates can miss crossings or dips between observations, and a first 90% crossing need not persist.Network accuracy is saved every update in the headline study.Source: §A.2 · p. 19

A.3 Windows and endpoints of the experiments

The appendix defines locked, loss-selected, strict, and post-hoc windows with distinct pass criteria. It also describes the one-hidden-layer experiment protocol and clarifies that the fresh evaluation tests a locked recipe selected using held-out accuracy, not an untuned optimizer family.

A.3 Windows and endpoints of the experiments

  • Fresh one-hidden-layer cohorts use windows fixed on development runs and locked before evaluated pairs are drawn, reporting loss variation, training correctness, Fourier gain, and held-out accuracies.The window is assessed against a declared 10% tolerance.Source: §A.3 · p. 20
  • The primary endpoint requires a captured share of the held-out rise, final held-out accuracy of at least 0.99, Fourier gain of at least 0.02, and positive logged minimum training margins.A depth succeeds when the lower end of its two-sided 95% Wilson interval reaches 0.80, or at least 37 of 40 passes.Source: §A.3 · p. 20
  • The stricter first-fit-anchored endpoints require delayed held-out recovery, a first 90% crossing inside the window, and continued training fit through recovery.For deeper networks, the strict endpoint also requires bounded raw band variation and sufficient last-layer Fourier gain.Source: §A.3 · p. 20
  • The post-hoc non-falling rule selects a maximal post-knee interval for the largest cross-entropy rise and passes on a 0.2 held-out-accuracy increase by the network or refit.It does not require captured share, final accuracy, training fit, or Fourier gain.Source: §A.3 · p. 20

B One-hidden-layer experiments

  • The one-hidden-layer experiments are evidence for coexistence, not replications of either initial-region theorem.Earlier runs are reported descriptively when reanalyzed from saved states or replayed.Source: B One-hidden-layer experiments · pp. 20–21

B.1 Protocol and selection

  • The headline protocol tracks every state for cross-entropy and accuracy, but records Fourier fraction every 100 updates and refit every 500.Its refit uses raw features and fixed α = 10^-3; uniform feature shrinkage strengthens that penalty.Source: §B.1 · p. 21
  • A recipe and the [4500, 12000] window were selected by a rule using held-out accuracy, then locked before 20 fresh pairs were drawn.The evaluation tests that locked choice, not an untuned optimizer family.Source: §B.1 · p. 21

B.2 The headline cohort

In the headline cohort, hidden-feature structure and held-out accuracy improve during a predeclared positive-loss plateau, with the train-only refit reaching 90% before the network. A separate preregistered test confirms recovery to 50%, but not to 90%, under its no-loss-decrease criterion.

B.2 The headline cohort

  • Across window I, AH gains 0.0955–0.0972 (median 0.0965); under a fixed rule on a 500-update grid, the refit first records 90% at update 6,500, with the previous checkpoint below it.Source: §B.2 · p. 21
  • AH gains 0.0955–0.0972 over the window, while cycle-mean CE has median relative variation 7.06%, within the predeclared 10% tolerance.At the refit’s first 90% observation, update 6,500, the previous checkpoint is below 90% in every run; this is earlier availability on a 500-update grid, not proof that the network head tracks the refit.Source: §B.2 · p. 21
  • In the headline cohort, cycle-mean CE falls 1.5–3.4% between the network’s first 10% and 90% held-out crossings, so recovery is not wholly non-decreasing in loss.No run meets the strict first-fit-anchored endpoint: training pairs intermittently misclassify after first fit, and held-out accuracy remains zero through that delay.Source: §B.3 · p. 22, §B.2 · p. 22

B.3 Pre-registered tests of recovery without CE decrease

  • The joint endpoint requiring recovery to 90% without more than 0.5% drawdown passed in 10 of 20 runs, below the required 18.Every run met the other four clauses; failures involved a late or missing 90% crossing, or a small loss decrease before it.Source: §B.3 · p. 22
  • The separately locked endpoint passed in 20 of 20 runs for recovery to 50%, with CE ending 3.8–5.9% above its starting value.Held-out accuracy rose from at most 1.9% to 50%, while CE never fell more than 0.05% below its running maximum; the confirmed result is recovery to 50%, not 90%.Source: §B.3 · p. 22

B.4 Results across primes

Across primes, separately tuned recipes produce varied plateau and recovery outcomes, while a single fixed recipe does not work consistently across moduli. Replay and quadratic-activation studies provide additional results, with windows and optimizer differences limiting their interpretation.

B.4 Results across primes

  • At p = 17, 23, and 29, all 60 per-prime recipe runs reach 90% held-out accuracy, but the strict joint endpoint passes in 15/20, 0/20, and 0/20 runs.The 10%-plateau counts are 16/20, 20/20, and 20/20; these plateau and recovery counts are not joint successes.Source: §B.4 · p. 25
  • On the common window, the fixed recipe meets the 10% plateau criterion in 17 of 20 runs at p = 17, but in 0, 0, 2, 0, and 0 runs at p = 11, 13, 19, 23, and 29.Thus success with a retuned family is not evidence that one practical recipe works across moduli.Source: §B.4 · p. 25

B.5 Replays and other one-hidden-layer studies

  • In replayed p = 23 and p = 31 trajectories, all ten descriptive windows show 10% plateaus and recovery, with median ∆AH of 0.038 and 0.049.Replayed raw CE stays within 2 · 10−14 of the originals through 15k updates; the windows are descriptive and the runs also belong to the deeper study’s development inventory.Source: §B.5 · p. 25
  • In the quadratic-activation contrast, all 40 runs reach 90% and gain at least 0.02 in Fourier fraction, but none has a 10% CE plateau.Both activation and optimizer differ, so this is not an activation-only ablation; tested 50%-training-fraction recipes and later p = 31 settings also did not recover within their tested horizons or selection rules.Source: §B.5 · p. 25

B.6 Theorem-inspired split and update settings from iid initialization

The p = 101 iid-initialization experiments test theorem-inspired settings, but do not replicate the theorem’s initialization, width, or certified windows. Deeper-network experiments use development-selected settings and recurring seed indices across configurations.

B.6 Theorem-inspired split and update settings from iid initialization

  • With head step ηV = 3 · 10−4, all four criteria hold in 10/10 runs; with ηV = 10−3, criteria (b)–(d) hold in every run, but none meets the 1% loss-variation criterion.For ηV = 10−3, v([1000, 40000]) is 1.34–1.37%; the displayed arms’ own refit-accuracy 0.1-to-0.9 intervals have loss variation at most 0.05% and 0.17%.Source: §B.6 · p. 25, Figure 8 · p. 26
  • During the displayed refit-accuracy intervals, AH rises from about 0.009 to 0.025, while the network’s held-out accuracy stays zero at recorded checkpoints.By update 40,000 the refit reaches one in every run, while the network reaches 0.25–0.47.Source: §B.6 · pp. 25–26
  • The displayed p = 101 runs do not reproduce theorem clauses (i) and (iii): training accuracy is 0.76–0.84 when the refit reaches 0.1, and 2–14 training pairs remain misclassified at update 40,000.The loss stays within about 1.5% of log p, a plateau near uniform prediction rather than a fitted one; the experiment also uses width 512 rather than the stated minimum 515,302 and iid rather than engineered initialization.Source: §B.6 · pp. 25–26

C Deeper-network experiments

  • The deeper-network study evaluates 164 runs across depths and settings, using 64 seed indices distinct from 199 development runs; seed indices recur across configurations, so observations are not independent.The development runs selected settings using held-out accuracy and windows maximizing captured accuracy rise under a flat cycle-mean-loss condition.Source: C Deeper-network experiments · p. 27

C.1 Network settings and learning-rate selection

  • For deeper networks, the last-layer shrink-rate multiplier is selected by training to a target held-out accuracy, then held fixed throughout the restarted run.For depths 3, 4, and 5, the stated starting multipliers are 0.50, 0.35, and 0.35, respectively.Source: §C.1 · p. 27

C.2 Depths three to five

At depths 3–5, the primary endpoint was met by most evaluated runs; changing the modulus, width, or feature-update rule did not meet it. With last-layer rates set per training fraction, a post-hoc-window criterion held in nearly all fresh runs at p = 47 and 61, while the primary endpoint held in only 9.

C.2 Depths three to five

  • Against the recipe’s 20/20 passes, fixed-multiplier runs passed 19/20, 5/20, and 19/20 at depths 3, 4, and 5; the benefit was clear at depth 4, not every depth.The reported exact McNemar p-values were 1, 6.1 · 10^-5, and 1, respectively.Source: §C.2 · p. 27
  • None of the changed settings met the primary endpoint: p = 23, width 512, and normalized-gradient feature updates had 0/20, 0/4, and 0/20 passes.The width-512 runs reached final held-out accuracy of 0.92–0.95 at 100k updates; NGD runs generalized, but their CE kept falling during recovery.Source: §C.2 · p. 27
  • 40/40, 39/40, and 38/40 runs met the primary endpoint at depths 3, 4, and 5, respectively.The per-depth Wilson lower bounds were at least 0.80; the three misses failed only the share threshold.Source: §C.2 · p. 27, Figure 9 · p. 28
  • 155 of 160 fresh runs at p = 47 and 61 met the post-hoc-window criterion with rates set per training fraction, while only 9 met the primary endpoint.The criterion was fixed in advance; p = 23 was not rerun with this rate-setting rule.Source: §C.2 · p. 28
  • Secondary checks passed in 49 of 120 runs for the persistent-fit variant and in none for the strict first-fit joint or either representation-first flag.These checks did not alter the primary verdict; the experiments do not establish a uniform refit-before-head chronology or a causal role for scalar Fourier growth.Source: §C.2 · p. 28

C.3 Results across primes

  • Across other tested primes, only depth 3 at p = 47 passed among depths 2–4; these settings were development-tuned rather than pre-specified evaluations.At p ≥ 47, some training pairs were misclassified inside the post-hoc window; under the looser post-hoc rule, every run of these settings passed.Source: §C.3 · pp. 28–29

C.4 Depths three and four across training fractions

In deeper networks at p = 47 and 61, the post-hoc criterion passes in nearly all runs, though the rise occurs at different training-loss levels across training fractions. Further analyses show broad post-hoc-window passes, mixed effects from hidden biases, and a specified transformer evaluation protocol.

C.4 Depths three and four across training fractions

  • All 16 deeper-network settings passed the post-hoc criterion, with 155 of 160 runs passing; results are reported per setting because runs share seeds.Three failures at fraction 0.5 and p = 47 ended before reaching 0.99 held-out accuracy; two at fraction 0.8 and depth 4 had windows that captured too little of the rise.Source: Table 3 · p. 29, §C.4 · p. 30
  • At fraction 0.8, CE is 0.52–0.67 log p and changes −2.9% to +9.5%; at 0.5, it is 0.007–0.045 log p at held-out 50%.At fraction 0.5, the relative CE change from the refit’s 20% crossing to the network’s 90% crossing is 136% to 1312%.Source: §C.4 · p. 30

C.5 Post-hoc window counts

  • Post-hoc windows pass across many settings, including all 170 runs across 17 development-selected deeper settings and all 120 primary runs.At depths 3 to 5, raw CE varies by orders of magnitude inside these windows, and at p ≥ 47 nearly every run misclassifies some training pairs.Source: §C.5 · p. 33

C.6 Biases in the hidden layers

  • With hidden biases, the one-hidden-layer median loss decrease is 5.3% versus 2.6% without bias, while depth-2 primary-endpoint passes fall from 34/40 to 18/40.The bias runs have median q(IC) = 0.62, below the required 0.75; 20 of 22 misses fail the share threshold.Source: §C.6 · p. 33, Figure 13 · p. 34

D.1 Model, updates and readouts

  • Transformer evaluation reports accuracy on all held-out pairs and on no-mirror pairs, whose mirror is also absent from training.About a fraction ϕ of held-out pairs have their mirror in training; at p = 113, these pairs are much more often correct at first full fit than other held-out pairs.Source: §D.1 · p. 35

D.2 Window and pass criterion

  • The transformer pass rule requires gains in Fourier-filtered-refit accuracy and AH within a post-fit window, no training errors, and final all-pairs accuracy of at least 0.99.The window is selected post hoc to capture the largest share of held-out 10%-to-90% rise, using a 201-update averaged CE.Source: §D.2 · p. 36

D.3 The gain and the recipe at p = 113

This section describes the p = 113 recipe for nearly flat-loss learning and the static constructions used to analyze controlled reference dynamics. It covers source covariances and normalization, positive margins and training-silent query features, prediction- and geometry-preserving lifts, and a late-refit bound.

D.3 The gain and the recipe at p = 113

  • At p = 113, g0 = 2 or 3 with τ = 2,500–10,000 passes in 18/18 runs, while τ = 20,000 passes in 6/9.The recipe uses gain 2–3 relaxing to 1, supported-Polar feature steps, and a training fraction that places the held-out rise after the CE minimum; changing the feature rate to 0.2 yields only a 10% CE rise.Source: §D.3 · p. 36, Table 5 · p. 37
  • Across four settings, two-block runs pass in none of 25 runs; at ϕ = 0.2, held-out accuracy on all pairs reaches 0.99 at 14.9k–18.2k updates, yet 6 of 10 runs end below 0.99.At ϕ = 0.2, training pairs begin to flip at 17.8k–22.0k updates, 1.4k–5.7k updates after reaching 0.99.Source: §D.3 · pp. 36–37

D.4 Other primes and mirror pairs

  • At p = 61, 89, and 149, the fixed-rule settings pass in 5/10, 10/10, and 7/10 runs, respectively; p = 113 passes in 10/10.At p = 149, all ten runs also meet the loose criterion; at p = 61, all ten pass on no-mirror pairs, while five fail on mirror pairs.Source: §D.4 · p. 37
  • At p = 61, a mirror-free split passes in 9/10 runs, versus 5/10 for the random split; the mirror-free window contains the full no-mirror rise.The mirror-free split has 1,038–1,043 distinct unordered training pairs, similar to 1,021–1,039 in the random split; with the random split, held-out pairs whose mirrors are training pairs are learned early.Source: §D.4 · p. 38

D.5 Readouts

  • The Fourier-filtered refit leads the network at the 50% crossing in 49/50 runs and at the 90% crossing in all 30 runs at p ≥ 89, while the all-frequency refit trails in all 50 runs.At p = 61, the filtered refit levels off near 0.96 and crosses 90% after the network; the top-16 choice was post hoc.Source: §D.5 · p. 38, Table 8 · p. 39
  • The readout’s lead depends on frequency selection: at the network’s 50% crossing, only 6–13 frequencies carry held-out signal, while the all-frequency refit weights every frequency equally.In the transformer, frequencies outside the top 8 have negative held-out R2; the network’s own logits restricted to the top 8 lead by about as much as the Fourier-filtered refit. The cited prior solutions use every frequency [8] or five [2].Source: §D.5 · pp. 38–39

E.1 Statement and conventions

  • For p ≥ 1764, the appendix adds a late complete train-only refit bound, Eref(j) < 121/400 at every j ∈ Ilate; the public late-refit guarantee concerns nonsquare differences, not diagonal inputs.The common theorem range is p ≥ 101, and the positive head step and full-dimensional initialization-ball radius are existential.Source: §E.1 · p. 43

E.2 Claim I: construction and uniformly controlled reference dynamics

  • The construction first bounds the uniform source and Polar rows, then uses those bounds to select a static initializer and verify an exact frozen-head trajectory.The static construction uses a prime p ≥ 101 and a fixed arithmetic-mask dictionary with reflection-paired rows sharing masks, classes, and masses.Source: §E.2 · p. 43, §E.2.1 · p. 44

E.2.1 The static construction

  • The character operators satisfy Hχ^2 = Pc and C−C+ = C+C− = pPc, while T is Hermitian with T^2 = pI and ∥T∥ = √p.These identities provide the character/Fourier structure used in the construction.Source: §E.2.1 · p. 45
  • The exact source covariance separates into nonconstant Fourier blocks M± and a constant block with eigenvalues 2c0 and zero.For a uniform source, the constant line within the supported subspace has eigenvalue 2c0.Source: §E.2.1 · pp. 47–48
  • For p ≥ 97, the supported inverse-root normalization satisfies ∥Ae − I∥ < 3/p, with Ae real symmetric.The covariance blocks M± are positive definite in this range.Source: §E.2.1 · p. 48
  • For the uniform source, σmin,E(F0) > 1/(5p^2), establishing a lower bound on its action over the supported subspace.The proof derives this bound from the nonconstant Fourier covariance blocks and the constant-line eigenvalue.Source: §E.2.1 · p. 50
  • For p ≥ 97, the constructed row has a positive bracket on the active mask, bounded below by 11p/80 − 3√p/8 − 27/4.The argument uses count and error bounds, including ν∞ < p/3, to control the inverse-root row.Source: §E.2.1 · pp. 51–52
  • Each query witness is inactive on training inputs but active on its held-out set Du, with logits zero on S and equal to (24ϵ/p)Tc(u) on Du.Its held-out label witness c(u) differs from the true label, while c(u) is the training label for (u, u + 1).Source: §E.2.1 · p. 53
  • Positive lifts preserve logits, example-row Gram matrices, and prescribed ridge penalties; with positive penalty, they also preserve train-only refit predictions, errors, and accuracies.The result applies to block-preserving lifts and to appended zero feature columns.Source: §E.2.1 · p. 53

E.2.2 Explicit initialization and uniformly controlled reference dynamics

This section develops finite-window guarantees for explicit initializations, controlling frozen-head dynamics, Fourier-fraction growth, training margins and cross-entropy, and held-out predictions and refits across early and late arrivals. It also presents a separate random-split, low-cross-entropy construction with refit-first recovery, under specified conditions on primes, widths, splits, and parameters.

E.2.2 Explicit initialization and uniformly controlled reference dynamics

  • The prescribed clock yields 32 early-window indices and at least 16 late-window indices, with every early index preceding every late index.These windows are separated while the clock parameter remains above its prescribed lower bound.Source: §E.2.2 · p. 54
  • The static program has a unique supported-partial-isometry minimizer, and its frozen-head reference trajectory keeps training logits and cross-entropy constant throughout the horizon.The trajectory’s core weights and features are exact at every arrival.Source: §E.2.2 · p. 55

E.3.1 The Fourier-fraction bound

  • The full-mask common-component identity establishes orthogonal projections for the grid and addition components, yielding the stated energy decompositions by Pythagoras.The accompanying centered masked-map bound is part of the identity’s result.Source: §E.3.1 · pp. 57–58
  • For primes p ≥97, the binary-comparison estimate gives a constant-component residual orthogonal to the comparison Polar factor, with norm ε_pν_c.The result also specifies the comparison source’s behavior on constants and the antisymmetric mode.Source: §E.3.1 · p. 58
  • The Fourier-fraction bound gives δ2 Q < 19/10 + 1/192 < 2.The constant block contributes exactly 2[1 −(1 + εp)−1/2] ≤εp, where εp <1/192.Source: §E.3.1 · p. 59

E.3.2 Actual-arrival table and Fourier transfer

  • The centered table’s Fourier fraction is below 0.004 at early arrivals and above 0.100 at late arrivals, with positive centered energy throughout.The bounds hold for every actual arrival when p ≥101 and ϵ is sufficiently small.Source: §E.3.2 · p. 60

E.3.3 Margins, cross-entropy, and prediction chronology

  • For p ≥101, the reference path keeps training accuracy at one with CE relative variation below 0.0045, while held-out accuracy is zero for ρj ≥9/10 and one for ρj ≤ϵ4.The path has positive core margins for every rival at every actual arrival.Source: §E.3.3 · p. 61

E.3.4 The complete refit at early arrivals

  • At early arrivals, the complete train-only refit has zero held-out accuracy, including diagonal inputs, and normalized held-out error greater than one.Its true-class score is negative, and the named rival exceeds it by more than 1/2(d + 1/32).Source: §E.3.4 · p. 62

E.3.5 The complete refit at late arrivals

  • For p ≥1764, the late complete-ridge analysis reduces the QR-anchor error to two representative Fourier alias problems and bounds predictions on held-out inputs with nonsquare differences.The supplied passage states the error and prediction bounds but does not provide their values.Source: §E.3.5 · p. 64
  • For p≥1764, the binary alias geometry gives positive Gram eigenvalues, and defect and penalty bounds control source mismatch and the actual QR table’s regularization.The actual penalty obeys ℓ/ℓbin < 1 + 2p/((p −1)√p + 1), identified as ℓ≤ℓ+.Source: §E.3.5 · pp. 65–66
  • For odd primes p≥1764, adjacent Fourier coefficients satisfy a1 > p^2/π^2, a2 < 1, and C1/a1 < 5/2.These bounds control the adjacent-interval correlation used by the sparse comparison teacher.Source: §E.3.5 · pp. 66–67
  • The complete fit uses every column, while a two-slope q=±k comparison coefficient has regularized target energy uniformly below 4/5.The sparse comparison is only a proof device; the full refit still minimizes over all columns.Source: §E.3.5 · p. 67, Its actual QR teacher f ′ · p. 67

Its actual QR teacher f ′

  • For every prime p≥1764, Proposition E.51 bounds the complete error of the actual QR teacher, comparing the full ridge minimizer with a corrected sparse-coefficient trial.The proof converts the residual and sampling estimates into held-out error through the alias identity.Source: Its actual QR teacher f ′ · p. 69
  • At every prescribed late arrival for p ≥1764, the complete train-only refit has margin >11/50 on nonsquare-difference held-out inputs, normalized error <121/400, and accuracy ≥(p −1)/(p + 1).The accuracy guarantee counts nonsquare-difference inputs; no diagonal correctness is asserted, although diagonal inputs remain in the full error denominator.Source: Its actual QR teacher f ′ · p. 71

E.4.1 Finite-horizon assembly at a fixed width

  • For each fixed p ≥101 and admitted width, an open initialization ball preserves Theorem 4.1’s prescribed-window guarantees; at p ≥1764 it also preserves the late-refit guarantee.The finite horizon and windows are fixed before initialization; the constants may depend on the fixed width, with no uniformity over widths asserted.Source: §E.4.1 · pp. 71–73

F.1 Statement and construction

  • The separate random-split construction fixes its update rates before sampling and gives a nonempty open ball in all 3pm weight coordinates on the split event.Both matrices remain trainable and untied, the dictionary adds no memories, and the construction does not assert accuracy after J.Source: §F.1 · p. 73
  • The random-split theorem keeps training margins strictly positive while its Fourier fraction remains O(1/p) through J; the claimed error gain is relative to each trajectory’s own initial error.The theorem applies on the specified split event with fixed updates and a training-data-built center.Source: §F.1 · p. 73
  • The engineered initial head and separate complete diagnostic both fail to predict held-out inputs, and refitted heads are never installed in the learner.The initial head uses two one-time centered train-only calibrations; the construction is not a proved outcome of earlier training.Source: §F.1 · p. 73

F.2.1 Split model, dictionary and the initial-data event

This section develops the split-data dictionary and calibrated initializations, establishing high-probability geometric and kernel conditions for finite-horizon feature evolution. It then presents finite-window conclusions on Fourier gains, refit and held-out behavior, and positive-loss trajectories under specified initialization and decay conditions.

F.2.1 Split model, dictionary and the initial-data event

  • The normalized training Gram matrix of the constructed dictionary equals B0,SS, and its centered initial feature table has mean feature hµ² at each grid input.Here B0 = I + Orow + Ocol + 11⊤.Source: §F.2.1 · pp. 74–75
  • Each of the four count/arm events has probability at least 1−δ; for p≥5, each sampled-force event has probability greater than 1−4δ.The count/arm result includes |n̂c| ≤ 2D* on Edes and Hyc ≥ Hmin on EH.Source: §F.2.1 · p. 76
  • With probability greater than 1−2/√p, the initial-data event gives a proper split with training and held-out inputs in every operand row, column, and class.For sufficiently large p, every class also has at least one training input.Source: §F.2.1 · p. 77
  • On the sampled design, n ≥ n− ≥ ρp²/2 and s0I ⪯ K ⪯ (ρp + K*)I, with κ* ≤ 1/2.The design identities include Z⊤Z = pI_E◦ and ZΥ−1Z⊤ = Orow + Ocol + 11⊤.Source: §F.2.1 · p. 78

F.2.2 The dictionary coefficient of the rates

  • For p≥64, τc > p^(3/2)/32 and τ0 ≥ p²/24, so ΣF has rank 2p−1, range Evis, and smallest positive eigenvalue greater than p³/1024.The covariance bounds establish nondegeneracy across the visible space.Source: §F.2.2 · p. 79
  • For J ≥ 30 and p ≥ 64, the pre-sampling rate choices define an increasing clock from 0 to 9/50 with increments greater than 1/(12J).The reference path is a decayed fixed-direction recursion used only in the proof.Source: §F.2.2 · p. 80

F.2.3 Train-only calibration of the initial head

  • The unique centered train-only offset makes each training example’s softmax probabilities equal the class-count proportions; canonical training margins are A + O(ω).For the canonical head, the normalized residual has zero row and column sums.Source: §F.2.3 · pp. 82–83
  • The train-only calibrated center has positive training margins and negative rival margins on every held-out input for sufficiently large p.The bounds are A(1 − e0) > 0 on training inputs and at most −A(ξ − e0) < 0 on held-out inputs.Source: §F.2.3 · p. 83
  • The train-only calibrated head has positive training margins of at least A and a held-out rival margin at most −Aξ, with e0=O(ω) and Ecal=O(Aω).The calibration establishes these strict signs on the split event.Source: §F.2.3 · p. 84

F.2.4 The reference direction and kernels

  • The reference force has bounded coordinates and a singular-value floor; its kernel separates rule, axis, constant, and mixed components, with the rule eigenvalue growing in z.The reference-direction estimates also establish the force anchor’s Polar identity and control perturbations of that direction.Source: §F.2.4 · pp. 84–86

F.2.5 Uniform finite-horizon comparison

  • Acc_D(0)=0 exactly, Acc_D(j)→0 for j<J, and Acc_D(J)→1 while L_j=L*+o(1).The comparison holds for initial states within a prescribed radius and head sequences with bounded increments; training margins are eventually at least ν_jA/25.Source: §F.2.5 · p. 87
  • The finite-horizon comparison establishes mean-loss and delayed held-out-accuracy limits, while a separate operator-motion bound controls every training margin.The initial held-out accuracy is exactly zero under the stated margin and logit-error conditions.Source: §F.2.5 · p. 91

F.3.1 Full refit and Fourier fraction

  • The reference feature table’s Fourier fraction increases with z, while sufficiently nearby tables retain the reference predictions and can have zero held-out refit accuracy initially.The stated growth bound is d/dz AH(F(z)) > A0; the initial-failure guarantee applies under the specified feature-distance condition.Source: §F.3.1 · pp. 93–94

F.4 Assembly: the prescribed ball and one common threshold

  • The assembled trajectories have strict training margins and satisfy AH(Hj) < 7/(3p + 1), with the required loss and accuracy conclusions.These include exactly zero initial held-out accuracy; positive-time refit accuracy rises, and the trajectory’s own initial refit error tends to one.Source: §F.4 · p. 94
  • With probability at least 1 − 2/√p, the construction yields a prescribed nonempty open ball and common finite threshold, with no accuracy claim after J.For finitely many states, actual Fourier-fraction increments eventually exceed A0/(24J), while AH(Hj) < 7A0; one finite p0 makes all bounds simultaneous.Source: §F.4 · p. 95

G Small decoupled readout decay

  • Positive readout-decay certificates preserve finite-horizon conclusions for the half-grid and specified event, ball, and η-based budgets for the random split.The half-grid late-refit clause is outside this positive-decay conclusion; for the random split, data step η/2 and a pre-sampling decay cap ensure head increments have norm at most η. Decay is outside normalization, raw training-CE sources use the departing state, and decayed trajectories may differ from undecayed ones.Source: G Small decoupled readout decay · p. 95

G.1 The structured half-grid construction

The appendix extends the structured half-grid and random-split trajectory results to readout decay. The half-grid retains its arrival windows, while the random-split guarantee uses a pre-sampling decay choice and a halved data step.

G.1 The structured half-grid construction

  • For the half-grid, every initialization in BM retains Theorem 4.1(i)–(iii) on the same arrival windows when 0 < dV < min{d0, 1/(20J)}.The head step is η0/2, and JdV < 1/20.Source: §G.1 · p. 95
  • The half-grid decay cap d0 is a fixed positive finite-horizon continuity bound with no stated formula, and the late-refit conclusion is not covered for dV > 0.The cited late-refit result assumes dV = 0.Source: §G.1 · p. 96

G.2 The random-split construction

  • With readout decay, the random-split construction retains Theorem F.1’s trajectory conclusions through state J with probability at least 1 − 2/√p.The guarantee covers every initialization in the prescribed ball; the actual data step is η/2, while radius and error formulas retain η.Source: §G.2 · p. 96
  • The random-split extension proves no practical decay magnitude, equal two-layer shrinkage, or positive decay lower bound uniform in p; its trajectories need not match the undecayed ones.The decay choice uses pre-sampling bounds and reserves motion budget by reducing the data step from η to η/2.Source: §G.2 · p. 97
Loading 2610.22593v1…