PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Anish Kataria

arXiv:2609.10657v1cs.AIcs.LG

TL;DR

The paper asks which hyperparameters govern when overparameterized networks transition from memorization to generalization. Across 384 modular-arithmetic MLP configurations, it fits a scaling law and identifies a phase boundary, finding that data complexity dominates transition timing.

  • Problem

    The paper addresses the missing quantitative theory of which hyperparameters govern grokking transition time and whether hyperparameter-space phase boundaries exist.

  • Method

    The study sweeps 384 two-hidden-layer MLP configurations on modular arithmetic, operationalizes memorization and generalization onset, and fits a power-law scaling relation.

  • Results

    Data complexity dominates model capacity, with Tgrok ∝ H^-0.27 D^-2.04 η^-0.50 λ^-0.64 and a sharp phase boundary at λ ≳1.0.

  • Takeaways & Limitations

    Data is the cheaper lever for accelerating generalization, while sufficient weight decay separates grokking from configurations that remain non-groking.

  • Takeaways & Limitations

    The scaling law is limited to two-hidden-layer MLPs on modular arithmetic, and the empirically observed phase boundary is not derived from first principles.

Abstract

from arXiv · show

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on \emph{why} this transition occurs, the quantitative structure of \emph{when} it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time: $T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}$ ($R^2 = 0.732$; $0.821$ with interactions). The exponent hierarchy reveals that data complexity ($D^{-2.04}$) is the dominant driver of regime transition, not model capacity ($H^{-0.27}$): doubling data accelerates generalization by ${\sim}4\times$, while doubling width yields only ${\sim}1.2\times$. A sharp phase boundary at weight decay $λ\gtrsim 1.0$ separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

1. Introduction

Grokking exposes a delayed memorization-to-generalization transition whose timing is not predicted by existing qualitative frameworks. This paper quantifies that timing across hyperparameters, identifying data complexity, regularization, and a phase boundary as central controls.

  • 1. Introduction: A 384-configuration sweep fits a quantitative scaling theory for when networks cross from memorization to generalization.The study targets transition timing, relative hyperparameter influence, and whether a phase boundary exists.
  • 1. Introduction: The fitted law is Tgrok ∝ H^-0.27 D^-2.04 η^-0.50 λ^-0.64, with R^2 = 0.821 when interactions are included.The exponent hierarchy identifies data complexity as the dominant driver relative to model capacity.
  • 1. Introduction: A sharp boundary near λ ≳1.0 separates configurations that generalize from those whose generalization is structurally suppressed.The boundary is described in (η, λ)-space.
  • 1. Introduction: Weight norms compress monotonically during the transition, reaching a median ratio of 0.42 between generalization onset and memorization.This trajectory is presented as a measurable proxy for regime state.
  • 1. Introduction: Data coverage and regularization strength primarily control escape from memorization, whereas model capacity has a weaker role.This conclusion reframes the transition around data and regularization rather than parameter count.

2. Experimental Setup

The experiments use two modular-arithmetic grokking tasks and two-hidden-layer MLPs across a structured hyperparameter sweep. Grokking is operationalized by separate training- and test-accuracy thresholds, distinguishing memorization from delayed generalization.

  • 2. Experimental Setup: The study evaluates addition mod 113 and division mod 97, benchmarks with known discrete-Fourier generalizing solutions.The tasks contain 12,769 and 9,312 examples, respectively.
  • 2. Experimental Setup: The model is a two-hidden-layer ReLU MLP with learned embeddings and widths H ∈ {128, 256, 512}.The models contain approximately 100K–960K parameters.
  • 2. Experimental Setup: The sweep varies data fraction, learning rate, and weight decay across 384 configurations using full-batch AdamW training up to 150K steps.Each configuration uses one random seed; 28 runs diverged and 297 of 356 completed runs grokked.
  • 2. Experimental Setup: Tmem is the first step above 99% training accuracy, while Tgrok is the first step above 95% test accuracy; Tgrok > 150,000 defines non-grokking.The resulting grokking gap ranges from 100 to over 100,000 steps.

3. Scaling Law for Generalization Onset

A log-linear analysis shows that all four swept hyperparameters reduce generalization-onset time, with data fraction exerting the strongest scaling effect and weight decay having the strongest rank association. Interaction terms improve fit quality and reveal trade-offs among data, regularization, learning rate, and width.

  • 3.1. Exponent Hierarchy: Data Dominates Capacity: Doubling the training fraction cuts Tgrok by about 4.1×, reflecting the steepest fitted exponent, D^-2.04.The paper interprets this as data strengthening the generalizing signal while weakening the memorizing solution.
  • 3.1. Exponent Hierarchy: Data Dominates Capacity: Weight decay has exponent λ^-0.64 and the highest rank correlation with Tgrok, indicating a role in both transition speed and whether grokking occurs.Its reported Spearman correlation is ρ = −0.52.
  • 3.1. Exponent Hierarchy: Data Dominates Capacity: Doubling width reduces Tgrok by only about 1.2×, making H^-0.27 the weakest main effect.Width also has the lowest reported Spearman correlation, ρ = −0.08.
  • 3.2. Interaction Effects: Adding pairwise interactions and a task indicator raises R^2 from 0.732 to 0.821, with significant effects involving data, width, learning rate, and weight decay.Larger datasets partially substitute for strong regularization, while wider models benefit more from higher learning rates.
  • 3.3. Robustness: Figure 1 relates Tgrok to width, FLOPs, regularization, and data across 356 runs, including a compute-cost shift for wider models.Its panels jointly show faster grokking in steps for wider models and higher FLOP cost per step.
  • 3.3. Robustness: The scaling law is supported across tasks and thresholds, while a Weibull model gives concordance index 0.71 and shape k = 1.4.The increasing hazard is consistent with acceleration once transition begins.

4. Phase Structure of the Grokking Boundary

The paper identifies a sharp grokking boundary in (η, λ)-space and links it to a quantitative transition threshold and delayed-computation gap.

  • Phase boundary: At λ ≥1.0, nearly all configurations grok, whereas at λ = 0.1 fewer than 60% do, with the transition occurring within a factor of 3× in λ.The boundary is sharp rather than gradual and becomes sensitive to learning rate below the higher-decay regime.
  • Phase boundary: The proposed critical strength λ∗≈1.0 structurally suppresses generalization below the threshold and destabilizes memorizing solutions above it.The conjecture attributes this distinction to weight decay driving convergence toward lower-norm generalizing solutions.
  • Interpretation: The phase diagram quantitatively locates the weight-norm threshold associated with access to the generalizing basin.This connects the observed boundary to the Omnigrok framework while extending it across hyperparameter space.
  • Transition timing: The grokking gap ΔT = Tgrok − Tmem falls by more than an order of magnitude as λ increases from 0.1 to 3.0.The gap measures computation spent in the memorizing regime after the training objective is satisfied.

5. Weight Norm Dynamics as a Regime Indicator

Weight norms compress monotonically before generalization, and the rate of compression predicts transition timing more reliably than the starting norm.

  • Empirical dynamics: The norm at generalization is lower than at memorization in all 14 analyzed runs, with median ratio ∥θ(Tgrok)∥/∥θ(Tmem)∥ = 0.42.The pattern spans fast, medium, and slow grokking strata and both tasks, though the sample contains only 20 configurations.
  • Compute trade-off: The compute-optimal frontier balances faster grokking in wider models against higher per-step FLOP cost, with total cost defined as Tgrok × Cstep(H).The figure specifies Cstep ∝ H2 and frames width as a steps-versus-compute trade-off.
  • Interpretation: The proposed norm-compression threshold places transition near a ratio r∗ in the range 0.3–0.5 if that threshold is approximately configuration-independent.Under this conjecture, λ controls compression rate while D controls the amount of compression required.
  • Regime indicator: Faster norm decrease is associated with earlier grokking, whereas absolute norm at Tmem does not predict Tgrok (ρ = 0.06, p = 0.83).This supports tracking the trajectory through weight space rather than relying on the initial norm level.

6. Related Work

The paper situates its contribution among work explaining grokking mechanisms, implicit regularization, phase transitions, scaling laws, and memorization in generative models.

  • Grokking: Prior grokking studies characterize Fourier circuits, norm-based mechanisms, hidden progress, and what happens during memorization-to-generalization transitions.The paper distinguishes its contribution by targeting when the transition occurs through a quantitative scaling law.
  • Implicit regularization and regime transitions: The λ−0.64 exponent quantifies how explicit weight decay modulates the implicit-regularization transition in practice.This extends theoretical work on max-margin solutions and lazy-to-rich transitions with an empirical regularization effect.
  • Phase transitions in learning: The phase diagram identifies an analogous boundary in (η, λ)-space governing delayed generalization, complementing interpolation-threshold phase-transition work.The boundary concerns whether delayed generalization occurs across training configurations.
  • Scaling laws: The paper extends power-law scaling from final-loss prediction to the timing of a qualitative regime transition.This positions Tgrok as a transition-time quantity within the broader scaling-law framework.
  • Memorization in generative models: The D−2.04 exponent quantifies a data-complexity effect on memorization persistence analogous to dataset-scale effects reported for diffusion-model replication.The comparison is motivated by prior observations of memorization in generative models.

7. Discussion

The discussion identifies data fraction as the main lever for shortening grokking, while extending the framework to generative models and highlighting limits of the current evidence.

  • Doubling the training fraction reduces Tgrok by about 4.1×, whereas doubling width reduces it by only about 1.2×.The discussion describes data as the cheaper lever for accelerating generalization.
  • The methodology offers a template for studying memorization-to-generalization timing in larger generative models, while quantitative data-fraction exponents for attention architectures remain unmeasured.
  • The scaling-law exponents may differ for transformers or natural-language tasks because the study uses two-hidden-layer MLPs on modular arithmetic.
  • The phase boundary is empirically observed rather than derived from first principles, and future work proposes theoretical explanations involving solution complexity and norm dynamics.
  • The D−2 exponent remains near −2 across 85%–99% generalization thresholds, while the weaker width exponent varies more.

A.3. Cross-Validation

Cross-validation indicates that the interaction model generalizes reasonably across held-out hyperparameter levels, although data fraction is the hardest level to extrapolate and some interactions are unsupported.

  • Leave-one-level-out cross-validation achieves R2 = 0.67–0.76 with median multiplicative prediction errors of 1.4–1.6×.
  • Data fraction is the hardest hyperparameter level to extrapolate because the power law must span a wider dynamic range.
  • The log H × log D and log η × log λ interactions are not statistically significant, with |t| < 2.

C. Compute-Optimal Frontier

Because per-step compute grows approximately quadratically with width while grokking steps decline only weakly, wider models require more total FLOPs to grok.

  • Total FLOPs to grok scale as H^1.73, so wider models are less FLOP-efficient despite grokking in fewer steps.The compute-optimal width for budget F scales as H* ∝ F^0.58.

D. Falsifiable Predictions

The paper turns its scaling law and norm dynamics into three falsifiable predictions about Fourier-feature timing, memorization rank, and norm-compression rate.

  • Fourier-feature onset is predicted to scale as D−2 when the data fraction changes.
  • Larger-width models are predicted to produce higher-effective-rank memorization solutions at Tmem.
  • Norm-compression rate is predicted to correlate more strongly with Tgrok than absolute norm, with |ρ| > 0.5 versus an observed ρ = 0.06.
Loading 2609.10657v1…