PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu

arXiv:2610.36113v1cs.LGcs.AIhep-thquant-ph

TL;DR

Grokking is delayed generalization after neural networks have overfit, and applying RG to it requires mapping training observables and constructing spatial structure for hidden neurons. The paper builds an effective RG framework and reports that its exponent estimates agree with direct measurements within 1–10%, supporting RG as a quantitative tool for analyzing grokking.

  • Problem

    Grokking has been interpreted as a transition, but applying RG requires operational physical observables and a way to spatially order permutation-interchangeable hidden neurons.

    Source: §1 · p. 1

  • Method

    The framework maps training observables to effective thermodynamic quantities, constructs a Fiedler-ordered neuron geometry, and applies local blocking to derive scaling exponents.

    Source: §5 · p. 8

  • Results

    Direct and RG exponent estimates agree within 1–10%, while an independently measured correlation-length exponent across seven task moduli agrees with the RG prediction within 3%.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The findings support RG as a quantitative tool for analyzing grokking in neural networks.

    Source: §5 · p. 8

    Ensemble fluctuations combine optimization dynamics and data-sampling variability, so the thermodynamic quantities are effective descriptions rather than thermal averages at fixed disorder.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.

1 INTRODUCTION

Grokking presents a delayed transition from training-set overfitting to generalization, motivating a quantitative account of its sharp transition. The paper makes an RG description operational by mapping neural observables to physical quantities and constructing a geometry for hidden neurons.

1 INTRODUCTION

  • Grokking describes delayed generalization after neural networks have overfit their training sets.Training accuracy rapidly saturates while test performance remains suppressed before rising abruptly in a narrow epoch window.Source: §1 · p. 1
  • The RG application must define physical counterparts for training observables and a notion of spatial proximity among permutation-interchangeable hidden neurons.The statistical ensemble consists of independent training runs rather than a Gibbs distribution.Source: §1 · p. 1
  • The physical dictionary treats training epoch as an effective inverse temperature, weight decay as the field conjugate to squared weight norm, and cross-run variances as fluctuation proxies.The dictionary is motivated by an effective Gibbs canonical ensemble of the loss objective.Source: §1 · p. 2
  • A Hessian-derived coupling graph and its Fiedler ordering place neurons on a chain where mean coupling decays with distance, unlike under random ordering.Adjacent neurons on this chain are merged into blocks for the local RG construction.Source: §1 · p. 2
  • Direct and RG estimates of the critical exponents agree within 1–10%.The framework obtains estimates through complementary routes: ensemble-fluctuation fits and a local RG map.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Related work explains grokking through phase-transition, representation, and optimization accounts, alongside studies of abrupt changes and scaling in other learning settings. These accounts generally do not derive scaling behavior from physically coarse-graining neural representations.

2 RELATED WORK

3 METHODS

The methods map training epoch and weight decay to effective thermodynamic coordinates, then construct a neuron geometry from loss-derived couplings. Block-spin coarse-graining and a local RG fit yield scaling exponents and derived critical exponents.

3.1 Setup and Observables

  • The 30-run modular-addition ensemble uses squared weight norm as an order parameter and cross-seed variances of norm and loss as susceptibility and heat-capacity proxies.Epoch and weight decay serve as effective thermal and field coordinates; the run-to-run fluctuations combine optimization and data-sampling variability.Source: §3.1 · pp. 3–4
  • The susceptibility peaks at epoch 118, while test accuracy has its maximum slope at epoch 145, which anchors the scaling analysis.The analysis aligns its effective transition point with generalization at ec = 145.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Constructing the Real Space

  • Per-neuron norm coordinates define spins, and second derivatives of the loss with respect to those coordinates define pairwise couplings.The off-diagonal couplings are computed by rescaling neuron activations; absolute couplings form a similarity graph whose Fiedler ordering defines a one-dimensional distance.Source: §3.2 · pp. 4–5
  • After Fiedler ordering, coupling strength decays monotonically with distance; the nearest 2.5% of pairs couple 2.1× more strongly than the farthest 2.5%.Random ordering yields a flat distance profile with no spatial dependence.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3 Renormalization Group

  • The RG step groups adjacent neurons in Fiedler order into pairs and tracks first- and second-off-diagonal mean coupling strengths as two coordinates.This maps 500 neurons to 250 blocks; the local transformation is fitted from 65 configurations near the grokking center.Source: §3.3 · p. 5
  • The local RG fit gives yt = 0.6923 and yh = 0.6515; because both eigenvalues have |Λ| > 1, both scaling fields are relevant under the RG flow.Source: §3.3 · p. 6

4 DIRECT MEASUREMENTS AND COMPARISON

Direct estimates and local-RG predictions are compared across five critical exponents, with the direct and RG values differing by 1.2–9.7%. The direct estimates also provide a check of scaling identities and effective dimension.

4 DIRECT MEASUREMENTS AND COMPARISON

  • The correlation-length exponent ν controls transition sharpening with system size, with Δt ∝ L^-1/ν; for modular addition, p is the size and width is the relative spread of e50 across runs.e50 is the epoch at which a run first reaches 50% test accuracy.Source: §4 · p. 7
  • Direct estimates give Rushbrooke 2.119 versus 2, Josephson 1.400 versus 1.451, and Widom 0.546 versus 0.480.The Josephson relation gives an effective dimension of 1.04, close to the adopted one-dimensional value.Source: §4 · p. 7
  • The direct-fit figure covers α, β, and γ, ν, and δ; the table compares direct estimates with local-RG values.The supplied table caption defines relative difference as |direct − RG|/RG.Source: Figure 5 · p. 8, Table 1 · p. 8
  • 1.2–9.7% separates the direct and RG values across the five critical exponents.Table 1 presents the exponent estimates and their relative differences.Source: §4 · p. 8, Table 1 · p. 8

5 CONCLUSION

The paper makes RG operational for neural networks by defining observables and an effective neuron geometry, then fitting a local RG map. Its exponent estimates agree with direct measurements, while finite-size scaling and data collapse provide further support.

5 CONCLUSION

  • An effective Gibbs dictionary maps training epoch to inverse temperature, weight decay to the field conjugate to squared weight norm, and ensemble fluctuations to heat-capacity and susceptibility proxies.The squared weight norm is the order parameter.Source: §5 · p. 8
  • Exact second derivatives of loss in per-neuron norm coordinates define a coupling graph whose Fiedler ordering forms a one-dimensional chain with distance-decaying coupling.The decay is monotonic, unlike under random neuron ordering.Source: §5 · p. 8
  • Blocking adjacent neurons and fitting the local RG map yields two relevant scaling directions, from which five critical exponents follow through standard scaling relations.The direct estimates also imply an effective dimensionality close to that of the constructed chain.Source: §5 · p. 8
  • Local-RG scaling exponents agree with direct measurements within 1–10%, and the finite-size correlation-length exponent agrees with the RG prediction within 3%.Rescaling transition curves for seven moduli by the RG exponent yields data collapse.Source: §5 · p. 8

Supplementary Materials

The supplementary notation table defines symbols used for the RG map, scaling quantities, finite-size analysis, and statistical-physics background.

A NOTATION

  • RG notation includes coupling vectors before and after blocking, the RG map and fixed point, and the fitted map’s Jacobian and eigenvalues.The table also names thermal and magnetic scaling fields and RG eigenvalues.Source: A NOTATION · p. 12
  • Critical-exponent notation covers α, β, γ, ν, and δ, alongside correlation length, system size, transition width, and pseudo-transition epochs.The table also defines the shift exponent and limiting mean pseudo-transition epoch.Source: A NOTATION · p. 12
  • The statistical-physics notation lists temperature, magnetization, energy, specific heat, susceptibility, Hamiltonian, block-spin projector, and connected correlation function.It also includes Gibbs, free-energy, and finite-size-scaling quantities.Source: A NOTATION · p. 12

B CONSTRUCTION OF THE REAL-SPACE NEURON NETWORK

The section constructs an effective one-dimensional geometry for hidden neurons from their loss-based couplings, then uses adjacent-neuron blocking to connect that geometry to real-space RG. Fiedler ordering reveals coupling locality absent under random ordering, while the RG framework relates coarse-graining flows to critical exponents.

B CONSTRUCTION OF THE REAL-SPACE NEURON NETWORK

  • The neuron coupling Jij is the second derivative of loss with respect to per-neuron norm coordinates, computed exactly by rescaling each neuron's activation.Absolute couplings form the weighted graph used for Fiedler ordering.Source: B CONSTRUCTION OF THE REAL-SPACE… · p. 13
  • Adjacent neurons in Fiedler order are paired into blocks, and mean absolute couplings on the first two off-diagonal bands define the fine- and coarse-chain coordinates.The method uses block size b = 2; when the number of active neurons is odd, the last neuron is dropped.Source: Figure 6 · p. 13, B CONSTRUCTION OF THE REAL-SPACE… · p. 15
  • Fiedler ordering produces a diagonal band in block-averaged couplings and a mean |Jij| profile that decays with distance, unlike the flat profile under random permutation.The distance profile uses 40 equal bins of normalized distance.Source: B CONSTRUCTION OF THE REAL-SPACE… · p. 15, Figure 7 · p. 15

C.1 Critical Exponents and Universal Scaling

  • Near a continuous transition, heat capacity, order parameter, susceptibility, critical-isotherm response, and correlation length follow power laws in reduced temperature and conjugate field.The correlation length characterizes the spatial decay of connected spin correlations.Source: §C.1 · p. 16

C BACKGROUND: REAL-SPACE RENORMALIZATION

  • Real-space RG groups spins into blocks, integrates out fast modes, and rescales lengths; near a fixed point, linearized coupling flows identify relevant scaling directions.Relevant directions have positive RG eigenvalues.Source: §C.2 · p. 16

C.2 Block-Spin Coarse-Graining and Linearized Maps

  • With two relevant scaling fields, RG eigenvalues determine standard critical exponents through scaling relations derived from the singular free energy.The paper identifies thermal and magnetic scaling fields with eigenvalues yt and yh.Source: §C.2 · p. 16

C.3 Finite-Size Scaling and Task Modulus

The paper treats task modulus p as the finite-size scale for modular addition and tests whether transition widths and accuracy trajectories collapse under RG-inspired rescaling. It also specifies how physical variables and the empirical coupling map correspond to neural-network observables.

C.3 Finite-Size Scaling and Task Modulus

  • The finite-size parameter is task modulus p, mapped to the infrared cutoff qmin = 2π/p, under hypotheses linking modulus variation and run-to-run transition fluctuations to a shared scaling regime.Under these assumptions, pseudo-transition width scales as ∆t ∝ p^-1/ν.Source: §C.3 · pp. 16–17
  • With νRG = 1.444, test-accuracy trajectories collapse across moduli; collapse costs minimize at ν = 1.50 for mean accuracy curves and ν = 1.39 for t50 distributions.The scan covers 350 validation runs; Figure 8 describes 350 confirmation runs.Source: §C.3 · p. 17, Figure 8 · p. 17

C.4 Dictionary Mapping to Neural Networks

  • The neural dictionary maps Fiedler-ordered neurons to one-dimensional spins, adjacent pairs to blocks, epoch and weight decay to control fields, and the linearized coupling map to RG eigenvalues.The 2 × 2 map yields yt and yh, from which critical exponents follow.Source: §C.4 · pp. 17–18

D DERIVATION OF CRITICAL EXPONENT RELATIONS

The derivation uses the RG homogeneity relation to express singular free energy under repeated coarse-graining. Because the accumulated rescaling factor is free, choosing it to normalize the thermal argument isolates the temperature dependence.

D DERIVATION OF CRITICAL EXPONENT RELATIONS

  • Repeated coarse-graining transforms the singular free energy by rescaling system size and the thermal and field arguments according to their RG eigenvalues.The relation can be iterated for any integer number of blocking steps.Source: §D.1 · p. 18, D DERIVATION OF CRITICAL EXPONENT… · p. 18

D.1 From One Blocking Step to a Free Scale Factor

  • Choosing the free accumulated scale factor so the thermal argument has unit magnitude isolates the non-analytic dependence on reduced temperature.The scaling functions differ above and below the critical temperature.Source: §D.1 · p. 18

D.2 Thermodynamic Exponents

The section obtains thermodynamic exponents by differentiating the singular free energy with respect to temperature and field, and obtains the correlation-length exponent from spatial rescaling. The resulting exponent identities include a hyperscaling condition tied to short-range interactions.

D.2 Thermodynamic Exponents

  • Differentiating the singular free energy with respect to reduced temperature and field yields the specific-heat, order-parameter, susceptibility, and critical-isotherm exponent relations.On the ordered side at zero field, spontaneous magnetization scales as (−t)^((d−yh)/yt).Source: §D.2 · pp. 18–19

Correlation length, ν.

  • Choosing the rescaling factor as |t|^-1/yt connects spatial coarse-graining to the correlation length and gives its temperature scaling.This choice corresponds to coarse-graining until the effective block size reaches the correlation length.Source: Correlation length, ν · p. 19

D.4 Scaling Identities

  • The exponent identities include relations derived from free-energy properties and a hyperscaling relation that involves spatial dimension d and assumes short-range microscopic interactions.Source: §D.4 · p. 19

D.5 Numerical Values

The theoretical values used in Section 3.3 are obtained from d = 1 and the unrounded empirical RG eigenvalues. Full-precision inputs are used before rounding, while Figure 9 shows the local regression fit behind the RG map.

D.5 Numerical Values

  • Theoretical values used in Section 3.3 follow from d = 1 and the unrounded empirical RG eigenvalues (yt, yh) = (0.6923, 0.6515).Numerical evaluations use full-precision empirical inputs before rounding to preserve internal mathematical consistency.Source: §D.5 · p. 19
  • Figure 9 shows the local affine fit relating observed block coordinates K′1 and K′2 to c + TKn for the weighted 65-point cloud.Points far from the grokking center receive negligible weight; the identity line is shown for comparison.Source: Figure 9 · p. 20

E LOCAL REGRESSION DETAILS

The local RG map is fitted around the grokking center using a weighted cloud of coupling coordinates, while control tangents are estimated separately by unweighted regression. The Gibbs dictionary is an operational analogy: epoch and weight decay act as effective controls, with explicit caveats about equilibrium and AdamW.

E LOCAL REGRESSION DETAILS

  • The local RG fit weights the grokking center (145, 15) fully and nearby epochs and weight decays less, using normalized coordinates and ridge regularization.Epoch neighbors at 140 and 150 receive weight 0.21, while weight-decay neighbors at 14 and 16 receive 0.05; the ridge penalty is 10^-5∥T∥2 F.Source: E LOCAL REGRESSION DETAILS · p. 20
  • Control tangents ∂K/∂e and ∂K/∂λ come from unweighted linear regression of the 65 coupling coordinates on reduced epoch and weight-decay offset.Figure 10 locates these tangents alongside the right eigendirections of T at the grokking center.Source: E LOCAL REGRESSION DETAILS · p. 20, Figure 10 · p. 21

F GIBBS FORM OF THE DICTIONARY AND THE TEMPERATURE-LIKE COORDINATE

  • The Gibbs dictionary treats the weight-decay perturbation −hS as analogous to a magnetic coupling, with increasing h favoring larger weight norms.The dictionary entries are operational definitions motivated by a Gibbs description that the network does not explicitly instantiate.Source: F GIBBS FORM OF THE DICTIONARY AND THE… · p. 21, Table 3 · p. 21
  • In the Gibbs formulation, the partition function defines Pβ,h, and the free energy, internal energy, and thermal entropy satisfy F = U − TSth.The response functions are expressed per neuron in the paper's scaling dictionary.Source: F GIBBS FORM OF THE DICTIONARY AND THE… · p. 22
  • Because h = −(λ − λc)/2, the Gibbs response relation gives (∂m/∂λ)T = −χ/2.Source: F GIBBS FORM OF THE DICTIONARY AND THE… · p. 22
  • Empirical response proxies replace Gibbs averages with seed ensembles and use loss variance instead of full-Hamiltonian variance, omitting dimensional prefactors that leave exponents invariant.The loss-variance proxy is presented as operational in the critical regime; epoch as effective inverse temperature is motivated by gradient-descent dynamics, not equilibrium sampling, and AdamW decay is not an explicit loss term.Source: F GIBBS FORM OF THE DICTIONARY AND THE… · p. 22
Loading 2610.36113v1…