PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking

Jianliang He, Leda Wang, Siyu Chen, Zhuoran Yang

arXiv:2602.16849v1cs.LGmath.OCstat.ML

TL;DR

How finite two-layer networks combine learned Fourier features into a global modular-addition solution remains unclear. This paper analyzes the mechanism and training dynamics, showing that diversification supports majority voting while frequency competition and weight decay explain feature emergence and grokking.

  • Problem

    Prior work does not fully explain how finite-neuron networks combine single-frequency, phase-aligned features into a global modular-addition solution.

    Source: §1 · p. 4, §4 · p. 13

  • Method

    The paper combines systematic experiments with rigorous analyses of diversification, frequency competition, phase coupling, and three-stage grokking dynamics in two-layer networks.

    Source: §1 · p. 4

  • Results

    The network implements Fourier-domain majority voting, while initial magnitude and phase misalignment determine competing frequency winners and weight decay prunes non-feature frequencies.

    Source: §1 · p. 4, §7 · p. 22

  • Takeaways & Limitations

    The analysis provides an end-to-end mechanistic account linking feature learning, training dynamics, and grokking in modular addition.

    Source: §1 · p. 4, §7 · p. 22

    Under quadratic activation, uneven neuron growth can cause a few phase-aligned neurons to dominate, leaving insufficient growth among others.

    Source: §6.2 · p. 21

Abstract

from arXiv · show

We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each neuron, with the "winner" determined by its initial spectral magnitude and phase alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.

1 Introduction

The paper uses modular addition as a tractable setting to explain how Fourier features emerge, combine into a global solution, and support grokking. It addresses finite-neuron mechanisms through diversification, lottery-ticket competition, and a three-stage memorization-to-generalization account.

1 Introduction

1.1 Related Work

1 Introduction

  • Frequency diversification and phase symmetry let neurons combine noisy Fourier signals into a flawed indicator function that identifies modular sums.Frequency diversification covers the necessary spectrum across neurons, while phase symmetry within frequency groups enables noise cancellation.Source: §1 · p. 4
  • Gradient-flow competition makes the frequency with greatest initial magnitude and least phase misalignment each neuron’s dominant feature.The ODE comparison lemma establishes that this winner grows exponentially faster, explaining how random initialization selects single-frequency representations.Source: §1 · p. 4
  • Grokking proceeds through memorization and two generalization stages, as weight decay removes residual noise and refines features into a sparse Fourier representation.The process is driven by competition between loss minimization and weight decay, with memorization arising from a perturbed lottery-ticket mechanism.Source: §1 · p. 4

2 Preliminaries

The paper studies modular addition as a classification task over all pairs in Z_p, using disjoint training and test sets to evaluate prediction on unseen pairs. It models the task with a bias-free two-layer network whose embedded inputs pass through nonlinear hidden units, output weights, softmax, and cross-entropy loss.

Modular Addition

  • The complete modular-addition dataset contains every pair (x, y) in Z_p with label (x + y) mod p, then splits these examples into disjoint training and test sets.Test performance measures accuracy on unseen input pairs; this setup is used to study grokking and feature learning in modular arithmetic tasks.Source: §2 · pp. 5–6

Two-Layer Neural Network

  • The model is a bias-free two-layer neural network with M hidden neurons that represents each input x using either a canonical p-dimensional embedding or a trainable d-dimensional embedding.Each hidden neuron has a parameter vector θ_m, and its corresponding output-layer weight ξ_m produces the network output.Source: §2 · p. 6
  • The experiments primarily use ReLU activations, while the theoretical analysis uses the quadratic activation σ(x) = x^2.This separates the activation used for empirical experiments from the one used for theoretical interpretations.Source: §2 · p. 6
  • For classification, the network logits are passed through softmax and trained with cross-entropy against the one-hot label for (x + y) mod p.Each pair is mapped to hidden features σ(⟨h_x + h_y, θ_m⟩), combined linearly using ξ_m, and converted into a categorical distribution.Source: §2 · p. 6

3 Empirical Findings

The experiments identify structured Fourier features, phase coupling, frequency diversification, and activation robustness in trained networks, while tracking how initialization and weight decay shape feature emergence and grokking.

3.1 Mechanistic Pattern: Experimental Observations on Learned Weights

  • Trained neurons encode single Fourier frequencies with trigonometric parameters, doubled phases, approximately uniform within-frequency phases, and comparable magnitudes across neurons.These observations hold across training runs and collectively explain the learned modular-addition structure.Source: §3.1 · pp. 7–8
  • Perfect accuracy persists when ReLU is replaced by |x|, x2, or x4, but falls close to zero for x or x3, implicating even-order activation components.The learned parametrization therefore depends on dominant even-order components rather than ReLU specifically.Source: Table 1 · p. 9, §3.1 · p. 9

3.2 Dynamical Perspective: Phase Alignment and Feature Emergence

  • Under small random initialization, frequencies compete independently within each neuron, and the eventual winner is selected by larger initial magnitude and smaller phase misalignment.All frequencies initially grow in parallel, but favorable components accelerate and dominate the neuron’s final feature.Source: §3.2 · pp. 9–11

3.3 Grokking: From Memorization to Generalization

  • Grokking proceeds through memorization and two generalization stages: loss minimization first overfits, then weight decay sparsifies frequencies and finally drives test accuracy toward perfection.The first generalization stage preserves growth of parameter norms while pruning non-dominant components; around step 10,000, weight decay becomes dominant.Source: §3.3 · pp. 11–12
  • During memorization, the model prioritizes common symmetric examples before rare examples, suppressing rare-example accuracy before later fitting them.Training accuracy peaks around step 1000 while test accuracy falls from 5% to 0% during this common-to-rare progression.Source: §3.3 · p. 12

4 Mechanistic Interpretation of Learned Model

The learned model approximates modular-addition indicators by aggregating frequency-phase views across neurons: phase alignment creates a correct-sum signal, while diversification enables majority voting to suppress residual noise. Under full diversification and sufficient magnitude, the true-signal peak dominates spurious peaks, with fully diversified parameterization also being the most parameter-efficient.

Single-Neuron Contribution and Majority Voting

  • Each phase-aligned neuron supplies a primary Fourier signal peaking at j = (x+y) mod p, alongside frequency-, phase-, and input-dependent residual noise.The primary term is cos(ωk(x+y−j)); remaining terms create biased individual frequency-phase views.Source: §4 · p. 13

Majority-Voting Approximates Indicator via Overparameterization

  • When neuron count is sufficiently large, full diversification balances every frequency and imposes uniform phase symmetry across frequency groups.Its exact frequency balance is an idealization that holds approximately under random initialization.Source: §4 · p. 13
  • Fully diversified neurons collectively approximate the indicator 1(x + y mod p = j), with the correct-sum peak exceeding spurious peaks by aNp/8.The softmax therefore concentrates on x + y mod p when the learned magnitude becomes sufficiently large.Source: §4 · pp. 13–14

Ablation Studies

  • Fully diversified parameterization is the most parameter-efficient ablation, producing the largest logit gap between the ground-truth index and incorrect labels.The comparison evaluates limited frequencies and non-uniform phases under the same neuron budget.Source: §4 · p. 14

5 Training Dynamics for Feature Emergence

This section analyzes feature emergence by tracking two-layer network parameters in Fourier space under gradient flow. It shows that small-initialization dynamics decouple across neurons and frequencies, preserve single-frequency features, and drive phase alignment toward symmetry.

5 Training Dynamics for Feature Emergence

  • Gradient-flow analysis tracks parameter evolution directly in Fourier space to explain how single-frequency and phase-alignment features emerge during training.The analysis uses quadratic activation, complete-data training, and the continuous-time limit of gradient descent.Source: §5 · p. 14, §5.2 · p. 15

5.1 Background: Discrete Fourier Transform

  • The DFT projects input- and output-layer parameters onto Fourier basis components, which are reparameterized by frequency magnitudes and phases.The analysis assumes prime modulus p > 2 for convenience and interprets each component through its magnitude-phase representation.Source: §5.1 · pp. 14–15

5.2 A Dynamical Perspective on Feature Emergence

  • Under specialized small random initialization, each neuron starts with one frequency, allowing its evolution to be analyzed within an orthogonal frequency subspace.The single-frequency initialization avoids chaotic frequency competition and is later extended to multiple-frequency initialization in §6.1.Source: §5.2 · p. 15

5.3 Properties at the Initial Stage

  • During the initial stage, small parameters make outputs nearly uniform, yet neurons decouple and learn feature directions through gradients; minimizing cross-entropy is approximately average-margin optimization.Each neuron’s dynamics depend only on its own Fourier magnitudes and a phase-related parameter, while the small-scale loss approximation connects the analysis to Morwani et al. (2023).Source: §5.3 · pp. 15–17

5.4 Preservation of Single-Frequency Pattern

  • Non-feature frequencies remain negligible throughout the initial stage, so the single-frequency pattern is preserved under sufficiently small initialization.The theorem requires log M/M ≲ c^-1/2 · (1 + o(1)), κinit = o(M^-1/3), and Cend ≍ κinit; constant-order parameter growth suffices for later phase alignment.Source: §5.4 · pp. 17–18

5.5 Neuron-Wise Phase Alignment

  • Phase differences converge to zero because misalignment drives opposite phase rotations, while aligned phases stop rotating; after alignment, parameter scales remain near initialization order.The phase-alignment time depends on initial misalignment, tolerance, initialization scale, and modulus, and the mean-field limit yields phase symmetry.Source: §5.4 · p. 18, §5.5 · p. 19

6 Theoretical Extensions

Theoretical extensions explain lottery-ticket frequency selection under multi-frequency initialization and characterize how ReLU preserves dominant Fourier features while producing structured leakage. They show that phase alignment and frequency competition remain central beyond the quadratic setting.

6.1 Theoretical Underpinning of Lottery Ticket Mechanism

  • Each frequency evolves independently, while better phase alignment accelerates magnitude growth and larger magnitudes accelerate alignment, creating a self-reinforcing competition.This decoupled ODE structure provides the competitive environment underlying the lottery-ticket mechanism.Source: §6.1 · p. 20
  • The ODE comparison analysis formalizes that initialization determines the winning frequency: with equal initial magnitudes, the frequency having the smallest initial phase misalignment ultimately dominates rapidly.The dominance guarantee holds under mild conditions with high probability at least 1 −eΘ(p−c).Source: §6.1 · pp. 20–21

6.2 Dynamics Beyond Quadratic Activation

  • Quadratic activation can produce unstable, highly imbalanced neurons, whereas ReLU approximately preserves each neuron’s initialized single-frequency pattern with small leakage into other frequencies.For θm, leakage appears at odd multiples such as 3k⋆ and 5k⋆; for ξm, it appears at 2k⋆ and 3k⋆, with magnitudes decaying across multiplicative factors.Source: §6.2 · pp. 21–22
  • For ReLU, leakage to the rk-th harmonic decays as 1/r^2, only odd harmonics leak into ξm, and the dominant-frequency gradient reinforces the feature without phase rotation.The dominant-frequency component is proportional to the parameter vector, explaining single-frequency emergence and phase alignment.Source: §6.2 · p. 22

7 Conclusion · A Additional Experimental Details and Results

The paper reverse engineers how two-layer networks learn modular addition, from training dynamics to a final Fourier-domain majority-voting solution. It explains feature emergence through competing frequencies and characterizes grokking as weight-decay-driven cleanup toward generalization.

7 Conclusion

  • The paper provides an end-to-end reverse engineering of modular-addition learning, covering both training dynamics and the final learned model.Source: §7 · p. 22
  • Trained networks implement Fourier-domain majority voting through phase alignment and model symmetry.Source: §7 · p. 22
  • Within each neuron, competing frequencies are selected by their initial magnitude and phase misalignment through a lottery-like mechanism.Source: §7 · p. 22
  • Grokking proceeds through three stages in which weight decay prunes non-feature frequencies, converting a perturbed Fourier representation into a clean, generalizable solution.Source: §7 · pp. 22–23
  • These findings provide insight into feature-learning dynamics in neural networks and may extend to more general tasks.Source: §7 · p. 23

A.1 Detailed Interpretation of Grokking Dynamics in Section 3.3

The paper characterizes grokking as three phases: symmetry-based memorization, loss-driven feature cleanup, and slow weight-decay cleanup that yields near-perfect generalization. Progress is tracked through model outputs and Fourier-domain parameter concentration, including inverse participation ratio measurements.

Progress Measures

  • Inverse participation ratio quantifies Fourier-coefficient concentration and parameter sparsity by averaging measurements across the θ_m and ξ_m parameters.The metric is applied to all neurons’ parameter vectors.Source: §A.1 · p. 26

Three-Phase Dynamics of Grokking

  • The grokking trajectory is governed by loss minimization and weight decay across one memorization phase followed by two generalization phases.The three-phase account is examined using progress measures over model outputs and parameters.Source: §A.1 · p. 26

Phase I: Memorization

  • During memorization, the model reaches 100% training accuracy and roughly 70% test accuracy by exploiting symmetry, while failing on truly unseen held-out data.Feature frequencies become roughly aligned and sparsity increases, producing a perturbed Fourier solution that overfits the incomplete training set.Source: §A.1 · p. 26, Figure 10 · p. 27

Phase II: Loss-Driven Norm Growth with Rapid Feature Cleanup

  • After perfect training accuracy, loss minimization continues increasing parameter scale while weight decay prunes unnecessary frequencies and slows norm growth.This phase combines loss-driven norm growth with rapid feature cleanup.Source: §A.1 · p. 26

Phase III: Slow Cleanup Driven Solely by Weight Decay

  • Near-zero training loss lets weight decay dominate, slowly reducing parameter norms while test accuracy converges to 100%.This final cleanup is extremely slow because the loss gradient has diminished.Source: §A.1 · p. 27

A.2 Ablations Studies for Fully-Diversified Parametrization · A.3 Training Dynamics with Quadratic Activation · B Proof of Results in Section 4 and 5

The ablations show that full frequency and phase diversification maximizes parametrization efficiency under fixed computational constraints. Training dynamics preserve each neuron’s initial single-frequency structure during the initial stage under sufficiently small initialization.

A.2 Ablations Studies for Fully-Diversified Parametrization

  • Ablations vary frequency count using |K| = {1, 2, 4, 8} while keeping selected-frequency phases uniform over [0, 2π).Source: §A.2 · p. 28
  • Phase-uniformity ablations use the full set of frequencies while restricting each frequency’s phase uniformly to [0, ιπ) for ι ∈ {0.4, 0.8, 1.2, 1.6}.All predictors share M = 128 neurons, and models are evaluated under a fixed, equivalent computational budget.Source: §A.2 · p. 28
  • 7.41 × 10−15 CE loss is achieved only with the full frequency set and phases spanning [0, 2π), establishing maximal efficiency under fixed constraints.The loss falls from 1.64 at |K| = 1 to 7.41 × 10−15 with all frequencies, while restricting phases to [0, 0.4π) yields loss 4.82.Source: §A.2 · p. 28, Table 2 · p. 29
  • The fully diversified parametrization produces the largest logit gap between the ground-truth label and incorrect alternatives, giving the highest prediction confidence.Source: Figure 13 · p. 29, §A.2 · p. 29

A.3 Training Dynamics with Quadratic Activation

  • The quadratic-activation dynamics experiment uses p = 23, width M = 512, SGD step size η = 10−4, and initial scale κinit = 0.02.Source: §A.3 · p. 30
  • Under sufficiently small initialization, each neuron preserves its initial single-frequency structure throughout the initial training stage.The neurons remain fully decoupled, with the main flow dominating and only the initial Fourier frequency evolving while other coefficients stay zero.Source: §A.3 · p. 30, Figure 14 · p. 30

B.1 Proof of Proposition 4.2 … B.3.2 Proof of Lemma B.3: Main Flow of Decoupled Neurons

The appendices prove that phase-aligned, balanced, phase-symmetric neurons produce logits whose softmax approximates the correct modular sum, while early gradient-flow dynamics simplify under small parameter scaling and nearly decouple across neurons. Fourier-basis calculations then characterize the resulting main flow and show cancellation of constant-frequency terms.

B.1 Proof of Proposition 4.2

  • Under phase alignment, each neuron’s logit decomposes into frequency-dependent cosine terms, which sum across balanced frequency groups and phase-symmetric neurons into the modular-sum indicator structure.The derivation uses homogeneous scaling, equal frequency-group sizes, balanced frequencies, and high-order phase symmetry.Source: §B.1 · p. 31
  • The phase-aligned, balanced, phase-symmetric network achieves softmax error at most p · exp(−aNp/8), requiring a ≳ (Np)^−1 · log(p/ϵ) for error ϵ.The correct class margin is at least aNp/8, and Lemma B.1 converts this margin into the stated softmax bound.Source: §B.1 · p. 31

B.2 Preliminary: Gradient Computation

  • The gradient-analysis setup decomposes cross-entropy into two parts and computes entry-wise derivatives with respect to each neuron’s ξ_m and θ_m parameters over the full modular-addition dataset.The theoretical analysis assumes quadratic activation σ(x) = x^2.Source: §B.2 · pp. 32–33

B.3 Main Flow Approximation under Small Parameter Scaling

  • With small initialization and sufficiently small constant a, the softmax components in the initial-stage gradients are uniformly approximated by a uniform vector with controllable error.Lemma B.2 formalizes the approximation using bounds on parameter scales and explicitly tracks error terms that vary along gradient flow.Source: §B.3 · pp. 33–34

B.3.1 Proof Overview: Simplified Dynamics under Approximation

  • Under the specialized initialization, early training has a neuron-wise decoupled loss landscape and preserves a single-frequency Fourier structure for each neuron.Table 3 summarizes the approximate parameter dynamics while omitting perturbations from the approximation errors.Source: §B.3.1 · p. 34, Table 3 · p. 35
  • The proof roadmap derives approximate dynamics first for the original parameters, then for discrete Fourier coefficients, and finally for Fourier magnitudes and phases to analyze alignment between θ_m and ξ_m.The discrete Fourier coefficients and signal magnitudes and phases are defined in §5.1.Source: §B.3.1 · p. 35, §B.3.2 · p. 35

B.3.2 Proof of Lemma B.3: Main Flow of Decoupled Neurons

  • Lemma B.3 shows that each neuron’s θ_m(t) and ξ_m(t) dynamics depend only on that neuron up to small approximation errors, yielding almost full decoupling.The derivation switches to the Fourier basis, evaluates gradient terms using trigonometric identities and modular periodicity, and finds that constant frequencies cancel.Source: §B.3.2 · pp. 36–38

B.4 Proof of Theorem 5.2: Single-Frequency Preservation · B.4.1 Proof of Auxiliary Lemma B.5

The proof establishes single-frequency preservation during an initial small-parameter stage: non-feature frequencies remain controlled while the feature frequency can grow despite approximation errors. Auxiliary results provide the forced linear ODE solution and high-probability phase alignment needed for the argument.

B.4 Proof of Theorem 5.2: Single-Frequency Preservation

  • The initial stage is defined by the first hitting time at which parameters reach the threshold C_end, while parameters remain substantially small beforehand.The analysis later bounds this hitting time using the growing feature coefficient and the condition C_end = Θ(κ_init).Source: §B.4 · pp. 38, 39, 42
  • Under the theorem’s stated initialization, scale, threshold, and phase-alignment conditions, non-feature frequencies remain controlled while the feature coefficient can grow rapidly during the initial stage.The phase event holds with probability greater than 1 − M^-c, and the proof uses uniformly bounded approximation errors over the initial interval.Source: §B.4 · p. 39
  • A well-aligned neuron persists throughout the initial stage because phase differences decrease consistently, and real dynamics stay close to the central flow with κ_init-dependent error.Continuity transfers the controlled central-flow phase behavior to the actual dynamics.Source: §B.4 · pp. 39, 42
  • The proof derives frequency dynamics with approximation-induced errors, then bounds non-feature growth and shows that zero initialization removes their leading contribution.The resulting bounds apply over the restricted initial-stage interval and support single-frequency preservation.Source: §B.4 · pp. 39–41
  • For a well-aligned feature, the relevant Riccati dynamics are monotone increasing under the stated large-M condition, yielding exponential growth until the threshold is reached.The proof uses the auxiliary ODE solution to obtain the growth lower bound and then upper-bounds the hitting time.Source: §B.4 · p. 42

B.4.1 Proof of Auxiliary Lemma B.5

  • Lemma B.5 gives the unique integrating-factor solution for ∂_t x(t) = ιx(t) + ζ(t), including the explicit expression when ζ(t) is constant.The proof follows by differentiating x(t) exp(−ιt) and integrating from 0 to t.Source: §B.4.1 · p. 43
  • Under random initialization, Lemma B.6 shows with probability greater than 1 − M^-c that at least one neuron has cosine phase alignment within O(M^-2(log M)^2) of one.The result follows from uniform phase distribution, a cosine lower bound, and the inequality (1 − x)^M ≤ exp(−xM).Source: §B.4.1 · p. 43

B.5 Proof of Theorem 5.3: Phase Alignment

The proof reduces the phase-alignment dynamics to a low-dimensional, symmetric ODE and establishes convergence of phase differences to zero. It then shows that diverging magnitudes preserve uniform phase marginals while enforcing the limiting relation 2ϕ = ψ.

B.5 Proof of Theorem 5.3: Phase Alignment

  • Theorem B.7 formalizes phase alignment under the main flow dynamics and the initialization in Assumption 5.1 for sufficiently small tolerance δ.The proof analyzes the central flow while omitting error terms caused by unpredictable drift.Source: §B.5 · p. 44
  • Fourier-domain single-frequency initialization reduces the analysis from a 2p-dimensional system to a three-particle dynamical system, with conserved quantities further reducing it to one dimension.The resulting trajectory also has a symmetry that relates evolutions from different initializations.Source: §B.5 · p. 44
  • The simplified dynamics agree well with the full dynamics for the same initialization, while closer initial phase differences yield shorter convergence times.Figure 15 illustrates monotonic phase-difference decay and the associated magnitude dynamics for two initializations.Source: §B.5 · p. 45, Figure 15 · p. 45
  • The phase difference D⋆_m(t) remains in its initialized half-space and approaches zero monotonically, with symmetry reducing convergence-time analysis to two initial-condition regimes.The proof separately treats initial differences in (0, π/2) and (π/2, π), with the latter requiring twice the corresponding travel time.Source: §B.5 · pp. 45–46
  • As parameter magnitudes diverge, phases converge to initialization-determined fixed values, enforcing the limiting alignment relation 2ϕ = ψ while preserving uniform phase marginals.For every time and neuron, the phase variables remain independently uniform, and the limiting joint measure is concentrated on the periodic line 2ϕ = ψ.Source: §B.5 · pp. 47–48

B.5.1 Proof of Auxiliary Lemma B.8, B.9 and B.10 · C Proof of Results for Theoretical Extensions in Section 6

The appendix proves auxiliary lemmas by reducing the Fourier-domain gradient flow from an exact single-frequency pattern to lower-dimensional dynamical systems. It also establishes phase invariance, half-space preservation, and symmetry properties needed to analyze the resulting dynamics.

B.5.1 Proof of Auxiliary Lemma B.8, B.9 and B.10

  • The proof of Lemma B.8 establishes an exact single-frequency pattern by driving the approximation error to zero.The result is stated under Assumption 5.1 and is supported by a matrix ODE argument.Source: §B.5.1 · p. 48
  • With initialization and preservation of the single-frequency, the 2p-dimensional dynamical system can be tracked via a four-particle system involving α⋆.Source: §B.5.1 · p. 49
  • Lemma B.9 proves that α⋆_m(t) remains constant throughout gradient flow and that an initialization-dependent constant C_diff exists.These conclusions follow from the simplified Fourier-domain flow and subsequent chain-rule calculations.Source: §B.5.1 · p. 50
  • The phase difference D⋆_m(t) remains in its initialized half-space and never reaches an integer multiple of π.Consequently, no jump occurs in the phase representation, allowing its dynamics to be tracked directly.Source: §B.5.1 · p. 50
  • Lemma B.10 reduces the main flow to a one-dimensional dynamical system characterized by β⋆_m(t).The proof uses the nonnegativity of magnitudes and analyzes the trajectory around the hitting time t_π/2 through backward and forward processes.Source: §B.5.1 · p. 51
  • Uniqueness of the locally Lipschitz ODE solution yields symmetry of β⋆_m around the hitting time t_π/2.The backward and forward trajectories coincide when parameterized by equal time offsets from the hitting point.Source: §B.5.1 · p. 51

C.1 Proof of Corollary 6.1: Phase Lottery Ticket · C.1.1 Proof of Auxiliary Lemma C.3 · C.2 Proof of Proposition 6.3: Dynamics of ReLU Activation

This section develops the proofs of the phase lottery-ticket result and the ReLU activation dynamics, using gradient-flow analysis and asymptotic Fourier calculations. It also derives the stated ReLU implication by evaluating cosine and sine integrals and specializing to even frequencies.

C.1 Proof of Corollary 6.1: Phase Lottery Ticket

  • Under random multiple-frequency initialization, the winning frequency is the one with the smallest initial phase-gap statistic eD_k^m, and it achieves dominance with high probability.The formal corollary defines k⋆ = argmin_k eD_k^m(0) and gives a high-probability dominance-time bound under a condition on p.Source: §C.1 · p. 51

C.1.1 Proof of Auxiliary Lemma C.3

  • Within each neuron, frequency magnitudes preserve their initial rank throughout gradient flow, with larger magnitudes corresponding to smaller initial phase differences.The proof uses frequency orthogonality, neuron decoupling, and order preservation to identify the lottery-ticket winner and compare it with the runner-up.Source: §C.1 · p. 52
  • The auxiliary lemma follows by transforming the dynamics into a cooperative system, whose monotonicity preserves the initial ordering of frequency magnitudes.The argument combines phase-reflection symmetry, standardized phase initialization, and cooperative-system rank preservation.Source: §C.1.1 · p. 56

C.2 Proof of Proposition 6.3: Dynamics of ReLU Activation

  • For ReLU, the analysis uses max{x, 0} = (x + |x|)/2 for the ξ_m gradient and applies DFT to the ξ_m and θ_m gradients in the asymptotic regime p →∞.Source: §C.2 · pp. 57–58
  • The proof combines the evaluated cosine and sine components to obtain the result stated in (C.18).The cosine calculation uses the integral involving |sin(u)| and cos(h_1u), followed by a corresponding sine-component calculation.Source: §C.2 · p. 59
  • For even r_k, the derived expression implies the stated specialization of the ReLU activation dynamics.This implication follows after combining equations (C.19) and (C.20).Source: §C.2 · p. 59

D Comparison with Existing Results

This section situates the work relative to Tian (2024) and Wang and Wang (2025), contrasting their modified-loss and mean-field analyses with the present standard-cross-entropy dynamics. It also clarifies the Fourier-coordinate correspondence and connects prior mean-field results to phase alignment and single-frequency sparsity.

Loss Landscape

  • Unlike the modified ℓ2-loss of Tian (2024) and Wang and Wang (2025), standard cross-entropy with small initialization decouples frequency dynamics, while the constant frequency has zero gradient and remains zero under zero-initialization.This comparison identifies the loss formulation and initialization conditions behind the differing frequency behavior.Source: D Comparison with Existing Results · p. 59

Notation Clarifications

  • The complex Fourier coefficients used by Tian (2024) are one-to-one equivalent to the paper’s real cosine-sine DFT basis through conjugate symmetry.A single real component determines the complex coefficients at paired frequencies k and p − k.Source: D Comparison with Existing Results · p. 60

Loss Landscape within Fourier Domain

  • Tian (2024) decomposes the modified loss into per-frequency quadratic terms involving third-order Fourier-coefficient monomials, then derives heuristic mean-field gradient dynamics under Gaussian initialization as M →∞.The resulting ODE evolves each coefficient according to a constant-order factor times the difference between a diagonal indicator and the coefficient.Source: D Comparison with Existing Results · p. 60

Mean-Field Dynamics

  • The prior mean-field ODE supports the emergence of phase alignment and single-frequency sparsity, while Wang and Wang (2025) rigorously formulate such dynamics as Wasserstein gradient flow.For diagonal frequency interactions, convergence requires vanishing imaginary parts; for off-diagonal interactions, phase differences vanish, corresponding to the observed single-frequency structure.Source: D Comparison with Existing Results · p. 60

Related papers

Loading 2602.16849v1…