PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Networks Provably Learn Spectral Representations for Group Composition

Jianliang He, Leda Wang, Fengzhuo Zhang, Siyu Chen, Zhuoran Yang

arXiv:2606.02993v2cs.LGmath.OCmath.RTmath.STstat.ML

TL;DR

The paper asks whether neural networks learning finite-group composition discover irreducible representations and low-rank structure. Using Fourier-domain analysis of gradient flow, it proves representation selection, rank-one cross-layer alignment, and uniform representation-phase behavior for Abelian groups.

  • Problem

    The paper asks whether networks learning finite-group composition discover the group’s irreducible representations and associated alignment or low-rank structure.

    Source: §1 · p. 3

  • Method

    The authors lift projected gradient flow for a two-layer group-composition network into the Fourier domain and analyze its Riemannian dynamics.

    Source: §1 · pp. 3–4

  • Results

    Gradient flow drives neurons toward single irreducible representations and rank-one cross-layer alignment, while Abelian groups exhibit uniform representation and Haar-uniform phase distributions.

    Source: §1 · pp. 4–5, §6 · p. 25

  • Takeaways & Limitations

    The theory characterizes feature learning and low-rank compression for finite-group representations, with Abelian ensembles achieving exact noise cancellation and perfect accuracy.

    Source: §1 · p. 4, §6 · p. 25

    The analysis focuses on population training, leaving rigorous train-test analysis and grokking-related delayed generalization open.

    Source: §6 · p. 26

Abstract

from arXiv · show

Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.

1 Introduction

The paper shows that gradient flow on finite-group composition tasks learns irreducible representations, with rank-one cross-layer alignment and refined diversification and phase dynamics for Abelian groups. It situates these results within mechanistic interpretability and learning-dynamics research, and defines notation for subsequent analysis.

1 Introduction

  • For arbitrary finite groups, gradient flow almost surely selects one non-trivial irreducible representation per neuron, while surviving Fourier coefficients become rank-one and rotationally align across layers.The analysis uses Fourier-lifted projected gradient flow and establishes convergence by showing non-target equilibria are measure-zero traps or strict saddles avoided under continuous random initialization.Source: §1 · p. 4
  • For Abelian groups, uniform spherical initialization diversifies neurons uniformly across non-trivial irreducible representations and Haar-uniform phases, producing a closed-form majority-vote predictor with exponential alignment and competition.The initially dominant representation wins almost surely while competing representations are exponentially suppressed.Source: §1 · p. 5

1.1 Related Work

  • The work extends Fourier-structured modular-addition analyses to arbitrary finite groups and provides exact, falsifiable predictions about emergent internal organization through representation-theoretic training dynamics.Unlike related sequential-composition work, it studies two-input composition with one-hot embeddings and standard gradient flow.Source: §1.1 · pp. 5–6

1.2 Summary of Notation

  • The notation section defines norms, softmax, asymptotic comparisons, complex-number operations, Hermitian adjoints, the unit circle, positive real proportionality, and group constructions.These conventions support the paper’s Fourier and group-theoretic analysis.Source: §1.2 · p. 6

2 Learning Group Composition with Neural Network

This section defines group composition prediction, specifies the quadratic two-layer network used to learn it, and describes full-data cross-entropy training. A two-stage procedure separates directional feature learning from later scale optimization.

Group Composition Task

  • The task predicts g_1 ⋆ g_2 for every pair in a finite group, whose operation satisfies associativity, identity, and inversion, with commutativity defining the Abelian case.Modular addition on Z_p is given as a canonical Abelian example.Source: §2 · p. 6

Network Architecture

  • The model is a two-layer fully connected network with separate operand embeddings, output embeddings, positive neuron scales, and quadratic activation σ(x) = x^2.Separate input embeddings are needed to distinguish operand order in non-Abelian groups; mean-field scaling uses a_m/M = Θ(1/M).Source: §2 · pp. 6–7

Training Data and Loss Function

  • Training uses the complete composition table and minimizes cross-entropy, balancing the correct-label logit against the log-partition penalty over all output classes.The complete dataset contains every (g_1, g_2) ∈ G × G with label g_1 ⋆ g_2.Source: §2 · p. 7

Training Algorithm

  • A two-stage algorithm first trains unit-sphere directions by projected gradient flow with scales fixed, then freezes those directions and optimizes neuron scales to sharpen correct predictions.Stage I uses a sufficiently small fixed scale to maintain a small-logit regime, while Stage II performs scale optimization after directional learning.Source: §2 · pp. 7–8

3 Warmup: Learning Generalized Modular Addition

This section generalizes modular addition from cyclic groups to arbitrary finite Abelian groups using group representation theory. It shows that training produces sparse, phase-aligned, diversified spectral features whose ensemble approximates the correct-answer indicator.

3.1 Generalized Modular Addition

  • Every finite Abelian group decomposes into cyclic factors, reducing generalized addition to component-wise modular addition and encompassing cyclic addition and bitwise XOR.The cyclic case has d = 1, while XOR sets every n_j = 2.Source: §3.1 · p. 8
  • The induced discrete Fourier transform decomposes functions into one-dimensional harmonic representations, while real-valued parameters impose conjugate-paired coefficients.The analysis focuses on the cleaner non-self-conjugate case where all n_j are odd; self-conjugate representations are deferred to §F.2.Source: §3.1 · pp. 8–9

3.2 Learned Patterns for Generalized Modular Addition

  • Each neuron concentrates on one nontrivial frequency and its conjugate, while the surviving input and output Fourier phases satisfy an additive alignment.Figure 1 verifies both sparse spectral support and concentration of output phase against summed input phases.Source: §3.2 · pp. 9–10, Figure 1 · p. 9
  • Across neurons, surviving frequencies are uniform over nontrivial frequencies, and input phases are independently Haar-uniform on [0, 2π).The frequency and phase diversification pattern is proved as part of Theorem 5.1 in §5.Source: §3.2 · p. 10, Figure 2 · p. 10
  • Combining single-frequency computation, phase alignment, and diversification yields an ensemble logit that peaks at the correct answer while exhibiting secondary ghost peaks.The construction generalizes the indicator-function mechanism shown by He et al. (2026) for Zp to arbitrary representations beyond product form.Source: §3.2 · p. 11

4 Main Results for General Group Learning

This section establishes that, for general finite groups, projected gradient flow learns sparse spectral structure, with neurons selecting single non-trivial irreducible representations and achieving rank-one rotational alignment. It analyzes the underlying Riemannian dynamics, rules out nongeneric equilibria, and characterizes the associated scale growth and empirical training behavior.

4.1 Background: Harmonic Analysis on Finite Group

  • The group DFT decomposes functions into matrix-valued blocks indexed by irreducible representations, generalizing scalar Fourier components from Abelian groups and enabling spectral analysis of non-Abelian structure.Non-Abelian irreducible representations can have dimension dρ > 1, producing dρ-by-dρ complex Fourier coefficients.Source: §4.1 · pp. 11–12

4.2 Stage I: Group Representation Learning

  • Under sufficiently small initialization scale a, the approximate small-logit risk tracks the original gradient-flow trajectories over finite horizons, with early-time discrepancy growing at most linearly in time.The linearized estimate permits horizons of order t = O(M|G|^-1/2a^-2ε) when the corresponding exponent remains small.Source: §4.2 · pp. 12–13
  • Each neuron almost surely converges to one non-trivial irreducible representation and its dual, while all other Fourier components vanish and active coefficients become rank-one and mutually aligned.The result holds as t → ∞ under random Stage I initialization and extends the single-frequency pattern to non-Abelian groups.Source: §4.2 · pp. 13–14

4.3 Proof Sketch for the Emergence of Spectral Patterns

  • Lifting projected gradient flow through the group DFT yields Riemannian gradient ascent on an energy functional; critical-point classification and saddle avoidance leave only positive-energy, single-representation, rank-one equilibria for random initialization.Negative- and zero-energy alternatives either require measure-zero initializations or are strict saddles, whose global stable set has zero Riemannian volume.Source: §4.3 · pp. 14–16
  • The proof excludes measure-zero initial basins and strict saddles, leaving only positive-energy equilibria with total effective rank one as attractors almost surely.Cases 1 and 2 require initialization in a zero-volume set, whereas Cases 3 and 4 are avoided because the flow escapes strict saddles.Source: §4.3 · p. 17

4.4 Further Discussion on Representation Selection in Stage I

  • Despite higher attainable energy for low-dimensional irreducible representations, random spherical initialization favors higher-dimensional representations, with learned proportions positively correlated with dimension.Initialization projections scale with representation dimension, and competition dynamics amplify this advantage.Source: §4.4 · p. 18

4.5 Stage II: Growth of Scaling Parameter

  • Stage I learns single irreducible representations with rank-1 Fourier alignment, while Stage II grows scale logarithmically and drives cross-entropy loss to zero at O(1/T).The Stage II guarantee requires the mean-field perfect accuracy condition (µ-PA), and the general-group mechanism extends Abelian frequency and phase alignment with novel rank-one compression.Source: §4.5 · p. 19

4.6 Experimental Results

  • Figure 6 shows each neuron concentrating Fourier coefficients in one representation block, while Figure 7 verifies perfect accuracy, alignment approaching 1, and singular-value ratios approaching 0.These observations support single-representation structure, cross-layer synchronization, and rank-one compression.Source: Figure 6 · p. 20, §4.6 · p. 21, Figure 7 · p. 21
  • Figure 8 shows cross-entropy plateauing during Stage I before dropping toward 0, while tied and untied scaling factors exhibit the predicted logarithmic growth.The experiments support the transition from structural alignment to scale-driven loss minimization.Source: §4.6 · p. 21

5 Mechanism and Training Dynamics of Abelian Group

For Abelian groups without self-conjugate representations, training yields diversified, phase-aligned spectral neurons whose majority vote produces a flawed indicator with the correct label dominant. The dynamics separate into exponentially convergent phase alignment and representation competition, with initialization determining each neuron’s winning representation.

5 Mechanism and Training Dynamics of Abelian Group

  • Figure 8 verifies the predicted two-stage dynamics: loss stays nearly constant during Stage I, then rapidly approaches 0 in Stage II as tied and untied scales grow logarithmically.The untied gradient is amplified by M for comparison with the tied case’s cross-neuron gradient accumulation.Source: Figure 8 · p. 22

5.1 Diversification, Perfect Accuracy, and Mechanics

  • Theorem 5.1 gives full diversification: nontrivial representations are uniformly distributed, phases are Haar-uniform, and representation choice is independent of phase.This mean-field limit satisfies (µ-PA) under uniform spherical initialization.Source: §5.1 · pp. 22–23
  • Averaging diversified neurons cancels phase- and representation-dependent noise while coherently accumulating signal, yielding a flawed indicator whose correct label always has positive logit margin.The logarithmic scale growth sharpens the softmax toward a one-hot prediction.Source: §5.1 · p. 23

5.2 Convergence Rate of the Spectral Patterns

  • Training decouples into phase alignment and representation competition: phases lock into the required relation, while the initially largest Fourier magnitude wins and competitors decay.Both processes converge exponentially fast, and the winning irrep is determined by random initialization; under uniform initialization, all nontrivial irreps are equally likely to win.Source: §5.2 · pp. 24–25
  • Theorem 5.3 establishes phase alignment toward one and winner selection by the largest initial irrep magnitude under scale-matching initialization, with convergence speed depending on initial phase and spectral gap.Phase alignment reaches epsilon-accuracy in time eO(M/(a|G|^1/2) · log(1/epsilon)) under the stated setup.Source: §5.2 · pp. 24–25

6 Conclusion · A Flow Approximation under Small-Logit Regime · A.1 Preparation: Gradient Computation

The paper establishes spectral learning dynamics for group composition, including irreducible-representation learning for general groups and uniform representation-phase distributions for Abelian groups. The appendices justify a small-logit risk approximation and derive the gradient correspondence underlying that analysis.

6 Conclusion

  • For general finite groups, gradient flow makes each neuron learn one irreducible representation with rank-one cross-layer alignment.This spectral structure is established through Riemannian flow analysis on the spectral manifold.Source: §6 · p. 25
  • For Abelian groups, learned representations and phases are independently uniform, yielding perfect accuracy through exact noise cancellation.Phase alignment and representation competition converge exponentially.Source: §6 · p. 25
  • The theory leaves open limiting distributions for general groups with high-dimensional irreducible representations and Abelian groups with self-conjugate representations.These cases are identified as unresolved theoretical problems.Source: §6 · p. 25
  • A rigorous train-test analysis remains open, including the delayed generalization phenomenon known as grokking.The paper focuses on the population case rather than characterizing train-test dynamics.Source: §6 · p. 26

A Flow Approximation under Small-Logit Regime

  • When the scaling factor a is small, near-zero logits make the softmax nearly uniform, so the exact and approximate risks are nearly identical.This establishes the first-order risk approximation used in Stage I.Source: A Flow Approximation under Small-Logit… · p. 30

A.1 Preparation: Gradient Computation

  • The gradient-computation appendix derives parameter gradients for both the exact cross-entropy risk R and its approximation R_ap to control their flow discrepancy.These expressions support the proof of Proposition 4.2 using the network architecture from (2.1).Source: §A.1 · p. 30
  • The two risks share the correct-label negative-logit term, differing only between log-sum-exp in R and mean logits in R_ap.Gradients are computed separately for these two components.Source: §A.1 · p. 30
  • The exact gradients use softmax weights, whereas the approximate gradients replace them with uniform weights 1/|G|, simplifying the expressions.This replacement is the sole difference between the compared gradient formulas.Source: §A.1 · pp. 30–31

A.2 Proof of Proposition 4.2 · B Proof of Theoretical Results for General Group · B.1 Equivalent Dynamics on the Spectral Manifold

This section establishes the approximate-flow comparison and develops the spectral-manifold dynamics underlying the general-group theoretical results. It characterizes the projected gradient flow as Riemannian gradient ascent on a real-valued energy functional and derives the resulting Fourier-coefficient evolution within each representation space.

A.2 Proof of Proposition 4.2

  • The proof controls the discrepancy between the exact and approximate flows by bounding projection, gradient, and Lipschitz-drift terms, then applying Grönwall’s lemma.Small scaling keeps the softmax distribution near uniform, making the two gradients close enough for the coupled differential inequality.Source: §A.2 · pp. 31, 32, 37

B Proof of Theoretical Results for General Group

  • The general-group appendix organizes its results around spectral dynamics, single-irreducible-representation convergence with rank-1 alignment, and Stage II scaling-factor growth.This passage previews the roles of Sections B.1–B.3 and the corresponding propositions and theorems.Source: B Proof of Theoretical Results for… · p. 37

B.1 Equivalent Dynamics on the Spectral Manifold

  • Using the group DFT and Schur orthogonality, Proposition B.1 derives equivalent projected-gradient-flow dynamics for each neuron’s matrix-valued Fourier coefficients on the spectral manifold.The derivation substitutes Fourier expansions into the gradient expressions, combines the resulting terms in the projected-flow ODE, and lifts the equations through the chain rule.Source: §B.1 · pp. 37–38
  • Cancellation of trivial-representation terms leaves only nontrivial representations in the projected dynamics.The same cancellation mechanism applies across the relevant Fourier coefficients, restricting the surviving terms to ρ ∈ Irr(G)̸=1.Source: §B.1 · p. 42
  • The energy functional Ω_m is real-valued and governs training because the projected gradient flow is precisely its Riemannian gradient ascent.The dynamics therefore seek to maximize Ω_m.Source: §B.1 · p. 43
  • Lifting the element-wise ODEs to Fourier coefficients and simplifying representation-index sums yields equations that characterize the dynamics within the representation space.The indicator terms collapse the sums to matching representations, while excluding the trivial representation.Source: §B.1 · p. 43

B.2 Proof of Theorem 4.3: Representation Learning in Stage I · B.2.1 Step 1: Riemannian Lifting to the Spectral Manifold

The proof lifts projected gradient flow into Fourier space, identifies it with Riemannian ascent on a spectral manifold, and then uses critical-point classification and almost-sure saddle avoidance to establish convergence to rank-one positive-energy equilibria. The lifting relies on unit-sphere constraints, reduced Fourier coordinates, and representation-wise Hessian decoupling.

B.2 Proof of Theorem 4.3: Representation Learning in Stage I

  • The proof classifies critical points by equilibrium energy and algebraic structure, ruling out zero-energy points supported only on the trivial representation except from an embedded strict submanifold.This excludes Cases 1 and 2 for generic initialization, while Cases 3 and 4 are removed by strict-saddle analysis.Source: §B.2 · p. 44
  • The only remaining critical points after excluding the other cases are total rank-one equilibria with positive energy Ω_m†.Critical points with positive energy and total rank at least two have positive-eigenvalue Hessian directions and are strict saddles.Source: §B.2 · p. 44
  • Under absolutely continuous random initialization, the Riemannian gradient flow avoids all strict saddles almost surely and cannot converge to the measure-zero bad cases.It therefore converges to the remaining rank-one equilibria, completing the proof of Theorem 4.3.Source: §B.2 · p. 44

B.2.1 Step 1: Riemannian Lifting to the Spectral Manifold

  • Riemannian gradients are obtained by projecting ambient Hilbert-space gradients onto tangent spaces, while the Riemannian Hessian adds the geometric correction from changing tangent spaces.The framework is developed for embedded manifolds and then specialized to the product sphere.Source: §B.2.1 · pp. 45–46
  • Real-valued parameters allow the Fourier dynamics to be represented using one representative from each conjugacy class of irreducible representations.The omitted coefficients are uniquely determined by the conjugacy relation, so the reduced spectral half-space captures the full dynamics.Source: §B.2.1 · p. 45
  • The projected gradient flow coincides with Riemannian gradient ascent on the energy functional Ω_m over a constrained Fourier-space manifold.The manifold encodes unit-norm constraints on single-neuron Fourier coefficients after applying the group DFT and Plancherel’s theorem.Source: §B.2.1 · pp. 45–46
  • After absorbing a positive multiplicative constant into time, the normalized flow has the form ∂t bΘ_m = grad_M Ω, without changing trajectory convergence conclusions.The Hessian projection on the product sphere uses Π_bν(U) = U − ⟨U, bν⟩ bν.Source: §B.2.1 · p. 46
  • In the representation basis, the ambient Hessian is block-diagonal and decouples across irreducible representations.Each ρ contributes an independent block, enabling the subsequent representation-wise critical-point and saddle analysis.Source: §B.2.1 · p. 46

B.2.2 Step 2: Critical Point Classification · B.2.3 Step 3: Saddle Avoidance for Riemannian Gradient Flow

Critical-point analysis excludes all equilibria except positive-energy, rank-one, single-representation points, while Riemannian gradient flow almost surely avoids the excluded saddles under absolutely continuous random initialization.

B.2.2 Step 2: Critical Point Classification

  • Only positive-energy, rank-one, single-representation equilibria remain as non-excluded convergence points after classifying the critical points.The first four cases are excluded, leaving the equilibria identified as convergence points in Theorem 4.3.Source: §B.2.2 · p. 47

Case 1. Negative Equilibrium Energy Ω†m < 0

  • Negative-energy equilibria attract trajectories only from a proper lower-dimensional submanifold of zero Riemannian volume, so absolutely continuous initialization reaches them with probability zero.The dynamics converge to the negative-energy equilibrium only when initialization lies in the exceptional set M_init.Source: Case 1. Negative Equilibrium Energy… · p. 47

Case 2. Zero Equilibrium Ω†m = 0 with only Trivial Representation.

  • Zero-energy equilibria supported only on the trivial representation likewise require initialization in the same exceptional set and are therefore excluded almost surely.The trivial-representation-only pattern can emerge and persist only from M_init.Source: Case 2. Zero Equilibrium Ω†m = 0 with… · p. 47

Case 3. Zero Equilibrium Ω†m = 0 with Non-trivial Representation Support.

  • Every zero-energy critical point with non-trivial representation support belongs to the strict-saddle set.Its equilibrium equations impose triple-annihilation relations, pairwise rank constraints, and a positive Hessian direction.Source: Case 3. Zero Equilibrium Ω†m = 0 with… · p. 48

Case 4. Positive Equilibrium Energy Ω†m > 0 with Higher-rank

  • Positive-energy equilibria with total rank at least two are strict saddles, leaving only rank-one, single-representation positive-energy equilibria.Their Fourier coefficients admit a shared-rank partial-isometry factorization, and tangent directions yield positive Hessian eigenvalues.Source: Case 4. Positive Equilibrium Energy… · pp. 48–49

B.2.3 Step 3: Saddle Avoidance for Riemannian Gradient Flow

  • For smooth gradient flow on a compact Riemannian manifold, the global stable set of saddle points has Riemannian volume measure zero.The result applies to functionals in C^r with r ≥ 2 and extends discrete first-order saddle-avoidance theory to continuous Riemannian flows.Source: §B.2.3 · p. 49
  • With an initialization law absolutely continuous relative to Riemannian volume, the flow avoids all saddle points with probability one.The center-stable manifolds are lower-dimensional null sets, and their countable union remains null.Source: §B.2.3 · p. 49

B.2.4 Assembly: Proof of Theorem 4.3 · B.2.5 Proofs of Lemma B.3 and B.4 · B.2.6 Proof of Lemma B.5, B.6, B.7: Cases 1 & 2

These appendices establish convergence of the Riemannian gradient flow, derive the geometric derivatives governing the dynamics, and exclude Cases 1 and 2 except for measure-zero initializations. Together, they complete Theorem 4.3 by showing that almost every trajectory reaches a positive-energy, rank-one, single-representation equilibrium.

B.2.4 Assembly: Proof of Theorem 4.3

  • Real-analytic gradient-flow theory upgrades vanishing gradients to convergence of each neuron’s trajectory to a single critical point.Compactness gives accumulation points and bounded energy, while the Łojasiewicz inequality yields finite trajectory length and rules out drifting along a continuum.Source: §B.2.4 · pp. 49–50
  • Theorem 4.3 follows by decoupling the dynamics across neurons and applying the analytic convergence result to the compact manifold M = S(H)⊗3 and polynomial energy Ω.The remaining classification of critical points identifies which equilibria can be reached and supplies the rotational alignment and positive proportionality conclusions.Source: §B.2.4 · p. 50
  • Almost surely, the only reachable equilibria are positive-energy, rank-one, single-representation states, yielding the theorem’s single-representation and rank-alignment properties.Cases 3 and 4 are strict saddles avoided almost surely, whereas Cases 1 and 2 require initialization in the measure-zero set Minit.Source: §B.2.4 · p. 50

B.2.5 Proofs of Lemma B.3 and B.4

  • Lemma B.3 derives the Riemannian gradient by projecting the ambient Fourier-domain gradient block-wise onto the three sphere factors.The resulting projected flow matches Proposition B.1 after scaling by 2a|G|^2/M.Source: §B.2.5 · p. 51
  • Lemma B.4 obtains the Riemannian Hessian by differentiating both the projected ambient Hessian and the curvature term induced by the sphere constraints.The ambient Hessian follows from the product rule, while the product-manifold projection acts block-wise and yields the claimed decomposition.Source: §B.2.5 · pp. 51–52

B.2.6 Proof of Lemma B.5, B.6, B.7: Cases 1 & 2

  • Lemma B.5 solves the relevant scalar linear evolution equations explicitly, providing the coefficient dynamics used in the Case 1 and Case 2 exclusions.The subsequent arguments use these solutions together with monotonicity of Ωm along the Riemannian gradient flow.Source: §B.2.6 · pp. 52–53
  • Convergence to either Case 1 or Case 2 requires the initial Fourier coefficients to satisfy the defining equal-norm constraints of the measure-zero set Minit.For Case 2, the energy remains nonpositive and the explicit dynamics force the corresponding nontrivial representation coefficients to vanish at equilibrium.Source: §B.2.6 · p. 53

B.2.7 Proof of Lemma B.8: Case 3 · B.2.8 Proof of Lemma B.9 and B.10: Case 4

These proofs characterize two equilibrium cases through block structure and Hessian analysis. Case 3 yields a positive-curvature saddle direction, while Case 4 derives shared-rank factorizations and tangent-space Hessian eigenvectors.

B.2.7 Proof of Lemma B.8: Case 3

  • Case 3 coefficient matrices have mutually orthogonal row and column spaces, with every pair of ranks summing to at most d_ρ.Partial-isometry bases and singular-value decompositions place the coefficients into separate diagonal blocks.Source: §B.2.7 · p. 54
  • When Ω†_m = 0, the Riemannian Hessian equals the ambient Hessian, enabling positive-eigenvalue directions built from leading singular vectors.The tangent construction branches on r2,ρ◦ + r3,ρ◦ < dρ◦ versus equality.Source: §B.2.7 · pp. 55–56
  • Case 3 equilibria are saddle points because an explicitly constructed tangent vector has a positive Hessian eigenvalue.The construction covers both whether a free block exists and the full-rank complementary case.Source: §B.2.7 · pp. 55, 57

B.2.8 Proof of Lemma B.9 and B.10: Case 4

  • Case 4 forces all three Fourier coefficients to have the same rank and consistent shared column and row spaces.The resulting common subspaces admit partial-isometry bases and a factorization through invertible core matrices.Source: §B.2.8 · p. 57
  • Case 4 equilibrium equations imply that the first two core matrices are proportional to unitaries, after which the third core is determined by Ω†_m[ρ].Unitary factors can be absorbed into the partial-isometry bases to complete the factorization.Source: §B.2.8 · pp. 57–58
  • For Case 4, the prescribed tangent directions satisfy the tangent-space constraints through trace-zero orthogonality and the shared factorization.The verification applies cyclically to θ1_m, θ2_m, and ξ_m.Source: §B.2.8 · p. 58
  • Every nonzero Case 4 tangent vector is a Riemannian Hessian eigenvector with eigenvalue λ = Ω†_m, while otherwise the tangent space is trivial.The ambient-Hessian calculation transfers to the Riemannian Hessian because projection preserves the Hessian action under the trace-zero constraint.Source: §B.2.8 · p. 59

B.2.9 Proof of Theorem B.11: Saddle Avoidance

The proof establishes that Riemannian gradient-flow trajectories avoid strict saddles almost surely under absolutely continuous random initialization. It combines global flow regularity with center-stable manifold geometry to show the saddle-convergent set has zero volume.

Lemma B.13

  • Under F ∈ C^r with r ≥ 2 on a compact Riemannian manifold, the gradient-flow map exists for all times and is a diffeomorphism.Completeness gives global existence, while uniqueness yields the inverse flow ϕ−t and smooth dependence on initial conditions.Source: §B.2.9 · p. 59

Theorem B.14

  • The center-stable theorem states that any trajectory remaining in a sufficiently small neighborhood of a critical point must lie on its local center-stable manifold.The manifold is locally invariant under forward flow, and trajectories that eventually remain in the neighborhood are captured by it.Source: §B.2.9 · p. 60

Local Flow Organization via Hessian Splitting

  • The local linearization splits tangent directions into nonrepelling stable/center directions and repelling unstable directions according to Hessian eigenvalues.The unstable subspace corresponds to positive Hessian eigenvalues, while the center-stable manifold integrates the non-positive eigenspace.Source: §B.2.9 · p. 60

Proof of Theorem B.11: Saddle Avoidance

  • For a strict saddle, the local center-stable manifold is lower-dimensional because the Hessian has a nonempty positive-eigenvalue unstable subspace.Its dimension is strictly less than dim(M), so it has zero volume measure on M.Source: §B.2.9 · p. 61
  • With probability 1, gradient flow initialized from an absolutely continuous distribution does not converge to any strict saddle.The stable set of all strict saddles is contained in a countable union of lower-dimensional embedded submanifolds and therefore has zero Riemannian volume.Source: §B.2.9 · p. 61

B.3 Proof of Theorem 4.5: Growth Rate of Scaling Factor in Stage II · C Proof of Results for Abelian Group in §5 · C.1 Spectral Dynamics for Abelian Groups

The proofs establish logarithmic scaling growth and cross-entropy convergence in Stage II, while specializing the spectral dynamics to Abelian groups through a closed magnitude–phase system. This reduction isolates relative phase alignment and supports the broader Abelian-group analysis.

B.3 Proof of Theorem 4.5: Growth Rate of Scaling Factor in Stage II

  • Theorem 4.5 follows by deriving scalar scale dynamics, transferring the population margin to finite width, and converting scale growth into a cross-entropy bound.The proof uses a positive population logit margin and finite-width approximation control to obtain logarithmic growth of a(t).Source: §B.3 · pp. 61–62
  • Under the tied constraint, the output factorizes as f = a · fbµ, and ∂ta becomes a softmax-weighted sum of correct-versus-incorrect logit gaps.This scalar gradient-flow identity is the first step of the proof.Source: §B.3 · p. 62
  • With high probability, finite-width logits retain an incorrect-label gap of at least Δfµ/4, yielding a positive lower bound on ∂ta and logarithmic growth of a(t).The bound holds simultaneously for all input pairs and incorrect labels after controlling the approximation error by Hoeffding’s inequality and a union bound.Source: §B.3 · p. 62
  • To reach R(T) ≤ ϵ, it suffices that a(T) ≳ log(|G|^2 · (|G| − 1)/ϵ), achieved when T ≳ |G|/ϵ · (1 + (|G| − 1)^−2).This establishes both the scale-growth and loss-convergence parts of Theorem 4.5.Source: §B.3 · p. 63

C Proof of Results for Abelian Group in §5

  • For finite Abelian groups, the appendix derives a magnitude–phase decomposition of the Fourier-domain ODE and uses it to analyze the limiting distribution, perfect accuracy, and convergence rates.The decomposition underlies Theorems 5.1 and 5.3 through subsequent lemmas on the population measure and convergence.Source: C Proof of Results for Abelian Group… · p. 63

C.1 Spectral Dynamics for Abelian Groups

  • Lemma C.1 shows that complex Fourier dynamics split into magnitude and phase equations, with both depending on individual phases only through the relative alignment φm[ρ].Magnitudes depend on ℜ(φm[ρ]), while phase evolution depends on ℑ(φm[ρ]).Source: §C.1 · pp. 63–64
  • The Abelian spectral system reduces to the closed variables (αθ,m[ρ], αξ,m[ρ], φm[ρ]), so individual phases need not be tracked separately.Lemma C.2 gives a closed-form ODE for φm[ρ] depending only on the two magnitudes and φm[ρ] itself.Source: §C.1 · p. 65

C.2 Proof of Theorem 5.1: Verification of (µ-PA) for Abelian Group

The proof verifies (µ-PA) for Abelian groups in two steps: it identifies the limiting parameter distribution and shows that its mean-field predictor achieves perfect accuracy despite structural ghost-label weights.

C.2 Proof of Theorem 5.1: Verification of (µ-PA) for Abelian Group

  • The limiting measure µ is the inverse-Fourier pushforward of Unif(Irr(G)\1) ⊗ Haar(D), using permutation and rotational invariances.This establishes µ = T_idft#π for π = Unif(Irr(G)\1) ⊗ Haar(D).Source: §C.2 · p. 66
  • The resulting predictor gives the correct label coefficient 2, ghost labels g2_1 and g2_2 coefficient 1, and all remaining labels baseline −4/|G|.Thus the predictor is a flawed indicator: the correct label remains largest despite nonzero ghost-label weights.Source: §C.2 · p. 66
  • Together, the distributional identification and coefficient calculation prove that the limiting measure satisfies the perfect-accuracy condition (µ-PA).Lemma C.3 identifies the limiting distribution, while Lemma C.4 verifies (µ-PA).Source: §C.2 · p. 66

C.2.1 Proof of Lemma C.3: Limiting Distribution · C.2.2 Proof of Lemma C.4: The Limiting Distribution Satisfies µ-PA

The proofs characterize each neuron's limiting state as an independent representation–phase pair, with uniform nontrivial representation selection and Haar-uniform phase. Averaging the resulting limiting measure makes the correct group-composition label achieve the largest expected logit.

C.2.1 Proof of Lemma C.3: Limiting Distribution

  • Each neuron converges to one nontrivial irreducible representation with rank-one rotational alignment, so its limit is determined by a representation–phasor pair.The limiting pair is (ˇρm, um), where ˇρm is the winning representation and um is its limiting phasor.Source: §C.2.1 · p. 67
  • The winning representation and limiting phasor are independent, and i.i.d. neuron initialization makes the limiting pairs i.i.d. with law Unif(Irr(G)̸=1) ⊗ Haar(D).The winner is determined by the reduced initial state, while the conditional phasor law remains Haar(D), independent of that state.Source: §C.2.1 · pp. 67, 70
  • Under uniform spherical initialization, Fourier phases are Haar-uniform and independent of the magnitudes and relative phases governing the reduced dynamics.This invariance allows the limiting phase to remain Haar-uniform after the dynamics apply a state-dependent deterministic rotation.Source: §C.2.1 · pp. 67–68
  • Permutation symmetry of initialization and equivariance of the ODE dynamics force every nontrivial irreducible representation to be equally likely to win.The symmetry acts transitively on conjugate pairs, yielding ˇρm ∼ Unif(Irr(G)̸=1).Source: §C.2.1 · pp. 68–69

C.2.2 Proof of Lemma C.4: The Limiting Distribution Satisfies µ-PA

  • The limiting measure's mean-field predictor is computed by expanding neuron logits and averaging over the uniformly selected representation and Haar-uniform phase.Phase integrals eliminate all nonzero powers of the unit-circle phase, leaving only representation-dependent terms.Source: §C.2.2 · pp. 70–71
  • The correct label j = g1 ⋆g2 achieves the largest expected logit after applying character orthogonality to the averaged expression.The resulting comparison includes the indicator terms for j = g1 ⋆g2 and the group-size correction −4/|G|.Source: §C.2.2 · p. 71

C.3 Proof of Theorem 5.3: Convergence Rate for Abelian Groups … F Additional Results for Generalized Modular Addition

This section develops supporting analyses and proofs for generalized modular addition and Abelian-group dynamics, including exponential convergence of phase alignment and representation competition. It also provides Fourier-domain visualizations, treatment of self-conjugate representations, technical representation-theoretic identities, and finite-group Fourier-analysis background.

C.3 Proof of Theorem 5.3: Convergence Rate for Abelian Groups

  • Theorem 5.3 separates exponentially convergent phase alignment from representation competition, with each rate established when the other mechanism is initialized in its limiting state.Scale-matching preserves equal input/output Fourier magnitudes, while the reduced dynamics couple phase alignment toward 1 with magnitude competition among representations.Source: §C.3 · pp. 71–72
  • With single-frequency initialization, the active conjugate representation pair is preserved and its relative phase converges exponentially to alignment.All other representation magnitudes remain zero throughout training under this initialization.Source: §C.3 · p. 73
  • With perfectly phase-aligned initialization, alignment remains invariant, the initially dominant nontrivial representation stays dominant, and magnitude ratios diverge so competing representations are suppressed.The convergence analysis uses log-ratio dynamics, monotonic growth of the initially dominant magnitude, and a separate bound for suppression of the trivial representation.Source: §C.3 · pp. 74–75

D Technical Lemmas

  • The technical appendices supply the analytic estimates and representation identities required by the trajectory and spectral-dynamics proofs, including softmax bounds, linear ODE solutions, dual-representation conjugacy, and trace-level Schur orthogonality.These results support proofs across the main and Abelian-group analyses.Source: §D.1 · pp. 75–76, §D.2 · p. 77

D.2 Technical Identities for Irreducible Representations

  • Schur orthogonality, unitarity, and Plancherel identities establish the technical Fourier tools used for finite-group representation calculations.The appendix derives matrix-entry identities from Schur orthogonality and states Plancherel’s theorem for functions on finite groups.Source: §D.2 · p. 78

E Background: Group Representation Theory

  • The representation-theory background defines finite and Abelian groups, unitary representations, duals, and examples including cyclic, product, symmetric, and Frobenius groups.It explains that cyclic-group representations recover the standard Fourier basis and that the Frobenius group has nontrivial three-dimensional representations used in the experiments.Source: E Background: Group Representation… · pp. 79–80

F Additional Results for Generalized Modular Addition

  • The appendix records full Fourier-domain heatmaps, interprets self-conjugate representations separately from the odd-order setting, and clarifies connections to cyclic modular addition [He et al. (2026)].These additions concern the generalized modular-addition experiment in G = Z3 ⊕ Z5 and its extension to self-conjugate cases.Source: F Additional Results for Generalized… · p. 81

F.1 Full Experimental Results in §3 · F.2 Modular Addition with Self-Conjugate Irreps · F.3 Comparison with Results in He et al. (2026)

The experiments show shared single-frequency sparsity across all learned parameter blocks, while self-conjugate representations reduce conjugate-pair structure and complex phases to real coefficients and signs. The framework subsumes the cyclic results of He et al. (2026) and extends them to arbitrary finite Abelian products.

F.1 Full Experimental Results in §3

  • Figure 10 shows conjugate-pair activation consistently across the three Fourier-transformed parameter families on G = Z3 ⊕ Z5.The heatmaps cover the top 20 neurons and display real and imaginary Fourier coefficients by frequency tuple.Source: §F.1 · p. 81, Figure 10 · p. 82
  • Each neuron concentrates on one shared non-trivial frequency tuple and its conjugate, producing the same sparsity pattern across θ1, θ2, and ξm.Hermitian symmetry gives equal real parts and opposite imaginary parts for the active conjugate coefficients.Source: §F.1 · p. 81

F.2 Modular Addition with Self-Conjugate Irreps

  • For G = Z2 ⊕ Z3 ⊕ Z5, the 29 non-trivial representations comprise one self-conjugate frequency, k = (1, 0, 0), and 14 conjugate pairs.A frequency tuple is self-conjugate when each even-order coordinate is 0 or nj/2.Source: §F.2 · p. 81
  • Most neurons retain single-frequency selection with a conjugate partner, but neurons selecting k = (1, 0, 0) learn one real coefficient with vanishing imaginary part.The self-conjugate representation has no distinct conjugate partner, creating an isolated middle-sector coefficient.Source: §F.2 · pp. 81–82, Figure 11 · p. 83
  • Self-conjugate modes replace the continuous phase exp(iϕm), with ϕm ∼ Unif(0, 2π], by the discrete Rademacher sign set {±1}.This is a boundary case of the same spectral mechanism rather than a new learning mechanism.Source: §F.2 · p. 82

F.3 Comparison with Results in He et al. (2026)

  • The cyclic-group results of He et al. (2026) are the one-dimensional special case, identifying single-frequency sparsity, phase alignment, and diversification with 1D irreducible representations.For odd n, characters occur in conjugate pairs, and sinusoidal features are real parts of phase-weighted characters.Source: §F.3 · p. 82
  • Our framework extends He et al. (2026) from cyclic odd-order groups to arbitrary finite Abelian products and includes self-conjugate sectors through Rademacher signs.Prior modular-addition analyses effectively avoided self-conjugate sectors by using odd p.Source: §F.3 · p. 83

Related papers

Loading 2606.02993v2…