Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

1. Introduction

• A 384-configuration sweep fits a quantitative scaling theory for when networks cross from memorization to generalization.The study targets transition timing, relative hyperparameter influence, and whether a phase boundary exists.Source: §1 · p. 1
• The fitted law is Tgrok ∝ H^-0.27 D^-2.04 η^-0.50 λ^-0.64, with R^2 = 0.821 when interactions are included.The exponent hierarchy identifies data complexity as the dominant driver relative to model capacity.Source: §1 · p. 1
• A sharp boundary near λ ≳1.0 separates configurations that generalize from those whose generalization is structurally suppressed.The boundary is described in (η, λ)-space.Source: §1 · p. 1
• Weight norms compress monotonically during the transition, reaching a median ratio of 0.42 between generalization onset and memorization.This trajectory is presented as a measurable proxy for regime state.Source: §1 · p. 1
• Data coverage and regularization strength primarily control escape from memorization, whereas model capacity has a weaker role.This conclusion reframes the transition around data and regularization rather than parameter count.Source: §1 · p. 1

2. Experimental Setup

• The study evaluates addition mod 113 and division mod 97, benchmarks with known discrete-Fourier generalizing solutions.The tasks contain 12,769 and 9,312 examples, respectively.Source: §2 · p. 2
• The model is a two-hidden-layer ReLU MLP with learned embeddings and widths H ∈ {128, 256, 512}.The models contain approximately 100K–960K parameters.Source: §2 · p. 2
• The sweep varies data fraction, learning rate, and weight decay across 384 configurations using full-batch AdamW training up to 150K steps.Each configuration uses one random seed; 28 runs diverged and 297 of 356 completed runs grokked.Source: §2 · p. 2
• Tmem is the first step above 99% training accuracy, while Tgrok is the first step above 95% test accuracy; Tgrok > 150,000 defines non-grokking.The resulting grokking gap ranges from 100 to over 100,000 steps.Source: §2 · p. 2

3.1. Exponent Hierarchy: Data Dominates Capacity

• Doubling the training fraction cuts Tgrok by about 4.1×, reflecting the steepest fitted exponent, D^-2.04.The paper interprets this as data strengthening the generalizing signal while weakening the memorizing solution.Source: §3.1 · p. 2, Table 1 · p. 2
• Weight decay has exponent λ^-0.64 and the highest rank correlation with Tgrok, indicating a role in both transition speed and whether grokking occurs.Its reported Spearman correlation is ρ = −0.52.Source: §3.1 · p. 2, Table 1 · p. 2
• Doubling width reduces Tgrok by only about 1.2×, making H^-0.27 the weakest main effect.Width also has the lowest reported Spearman correlation, ρ = −0.08.Source: §3.1 · p. 2, Table 1 · p. 2

3.2. Interaction Effects

• Adding pairwise interactions and a task indicator raises R^2 from 0.732 to 0.821, with significant effects involving data, width, learning rate, and weight decay.Larger datasets partially substitute for strong regularization, while wider models benefit more from higher learning rates.Source: §3.2 · pp. 2–3

3.3. Robustness

• Figure 1 relates Tgrok to width, FLOPs, regularization, and data across 356 runs, including a compute-cost shift for wider models.Its panels jointly show faster grokking in steps for wider models and higher FLOP cost per step.Source: Figure 1 · p. 3, Figure 3 · p. 4
• The scaling law is supported across tasks and thresholds, while a Weibull model gives concordance index 0.71 and shape k = 1.4.The increasing hazard is consistent with acceleration once transition begins.Source: §3.3 · pp. 3–4

Phase boundary

• At λ ≥1.0, nearly all configurations grok, whereas at λ = 0.1 fewer than 60% do, with the transition occurring within a factor of 3× in λ.The boundary is sharp rather than gradual and becomes sensitive to learning rate below the higher-decay regime.Source: §4 · p. 4
• The proposed critical strength λ∗≈1.0 structurally suppresses generalization below the threshold and destabilizes memorizing solutions above it.The conjecture attributes this distinction to weight decay driving convergence toward lower-norm generalizing solutions.Source: §4 · p. 4

Interpretation

• The phase diagram quantitatively locates the weight-norm threshold associated with access to the generalizing basin.This connects the observed boundary to the Omnigrok framework while extending it across hyperparameter space.Source: §4 · p. 4

Transition timing

• The grokking gap ΔT = Tgrok − Tmem falls by more than an order of magnitude as λ increases from 0.1 to 3.0.The gap measures computation spent in the memorizing regime after the training objective is satisfied.Source: §4 · p. 4

Empirical dynamics

• The norm at generalization is lower than at memorization in all 14 analyzed runs, with median ratio ∥θ(Tgrok)∥/∥θ(Tmem)∥ = 0.42.The pattern spans fast, medium, and slow grokking strata and both tasks, though the sample contains only 20 configurations.Source: §5 · p. 4

Compute trade-off

• The compute-optimal frontier balances faster grokking in wider models against higher per-step FLOP cost, with total cost defined as Tgrok × Cstep(H).The figure specifies Cstep ∝ H2 and frames width as a steps-versus-compute trade-off.Source: Figure 5 · p. 5

Interpretation

• The proposed norm-compression threshold places transition near a ratio r∗ in the range 0.3–0.5 if that threshold is approximately configuration-independent.Under this conjecture, λ controls compression rate while D controls the amount of compression required.Source: §5 · p. 5

Regime indicator

• Faster norm decrease is associated with earlier grokking, whereas absolute norm at Tmem does not predict Tgrok (ρ = 0.06, p = 0.83).This supports tracking the trajectory through weight space rather than relying on the initial norm level.Source: §5 · p. 5

Grokking

• Prior grokking studies characterize Fourier circuits, norm-based mechanisms, hidden progress, and what happens during memorization-to-generalization transitions.The paper distinguishes its contribution by targeting when the transition occurs through a quantitative scaling law.Source: §6 · p. 5

Implicit regularization and regime transitions

• The λ−0.64 exponent quantifies how explicit weight decay modulates the implicit-regularization transition in practice.This extends theoretical work on max-margin solutions and lazy-to-rich transitions with an empirical regularization effect.Source: §6 · p. 5

Phase transitions in learning

• The phase diagram identifies an analogous boundary in (η, λ)-space governing delayed generalization, complementing interpolation-threshold phase-transition work.The boundary concerns whether delayed generalization occurs across training configurations.Source: §6 · pp. 5–6

Scaling laws

• The paper extends power-law scaling from final-loss prediction to the timing of a qualitative regime transition.This positions Tgrok as a transition-time quantity within the broader scaling-law framework.Source: §6 · p. 6

Memorization in generative models

• The D−2.04 exponent quantifies a data-complexity effect on memorization persistence analogous to dataset-scale effects reported for diffusion-model replication.The comparison is motivated by prior observations of memorization in generative models.Source: §6 · p. 6

• Doubling the training fraction reduces Tgrok by about 4.1×, whereas doubling width reduces it by only about 1.2×.The discussion describes data as the cheaper lever for accelerating generalization.Source: §7 · p. 6
• The methodology offers a template for studying memorization-to-generalization timing in larger generative models, while quantitative data-fraction exponents for attention architectures remain unmeasured.Source: §7 · p. 6
• The scaling-law exponents may differ for transformers or natural-language tasks because the study uses two-hidden-layer MLPs on modular arithmetic.Source: §7 · p. 6
• The phase boundary is empirically observed rather than derived from first principles, and future work proposes theoretical explanations involving solution complexity and norm dynamics.Source: §7 · p. 6
• The D−2 exponent remains near −2 across 85%–99% generalization thresholds, while the weaker width exponent varies more.Source: §A.2 · p. 8, Table 3 · p. 8

• Leave-one-level-out cross-validation achieves R2 = 0.67–0.76 with median multiplicative prediction errors of 1.4–1.6×.Source: §A.3 · p. 8
• Data fraction is the hardest hyperparameter level to extrapolate because the power law must span a wider dynamic range.Source: §A.3 · p. 8
• The log H × log D and log η × log λ interactions are not statistically significant, with |t| < 2.Source: §A.3 · p. 8

• Total FLOPs to grok scale as H^1.73, so wider models are less FLOP-efficient despite grokking in fewer steps.The compute-optimal width for budget F scales as H* ∝ F^0.58.Source: C. Compute-Optimal Frontier · p. 8

• Fourier-feature onset is predicted to scale as D−2 when the data fraction changes.Source: D. Falsifiable Predictions · p. 8
• Larger-width models are predicted to produce higher-effective-rank memorization solutions at Tmem.Source: D. Falsifiable Predictions · p. 8
• Norm-compression rate is predicted to correlate more strongly with Tgrok than absolute norm, with |ρ| > 0.5 versus an observed ρ = 0.06.Source: D. Falsifiable Predictions · p. 9