Unifying Grokking and Double Descent

1 Introduction

• Grokking follows near-perfect training performance and near-chance test performance with later improvement to perfect test accuracy, while double descent involves test performance worsening and then improving as capacity increases.Both dynamics are framed as generalization behaviors that can emerge during learning.Source: §1 · p. 1
• The paper argues both phenomena reflect inductive biases favoring slower, better-generalizing patterns, producing a transition from poorly to well-generalizing patterns.The transition is proposed to occur across training time and model size.Source: §1 · p. 1
• The authors propose a conceptual framework unifying grokking and double descent, a simple supporting model, and evidence that grokking can occur as model size varies.Figure 2 depicts pattern predictiveness and resulting train and test accuracy curves for both phenomena.Source: §1 · p. 2, Figure 2 · p. 2

2.1 Core Claims

• Grokking and epoch-wise double descent occur when fast, poorly generalizing patterns precede slower, well-generalizing patterns ultimately favored by training.Source: §2.1 · p. 2
• The framework extends these dynamics across model size as well as training time, and proposes a model to explain them.Source: §2.1 · p. 2

2.2 A Model of Learning Dynamics

• The model treats a network as n patterns whose correctness probabilities develop during training, with sigmoid learning curves parameterized by predictiveness, inflection point, and learning speed.Source: §2.2 · p. 2
• Test accuracy weights successful-pattern subsets by their generalization ability, with a preferred pattern receiving extra weight.Source: §2.2 · p. 3

2.3 Double Descent & Grokking with Pattern Types

• A fast, well-generalizing heuristic, a slower poorly-generalizing overfitting pattern, and a slow well-generalizing pattern can produce both dynamics, interpolated by changing the heuristic and slow pattern’s maximum predictiveness.Source: §2.3 · p. 3
• The model makes classification depend on whether an example is in the training or test set, a deliberate simplification used to represent overfitting.Source: §2.3 · p. 3

3.1 Model-Wise Grokking

• The experiments train decoder-only transformers on division mod 97 and vary parameter count to regulate effective model complexity.Source: §3.1 · p. 4
• Across a range of settings, delayed generalization appears along both training-step and model-size paths, with corresponding double descent in accuracy.Source: §3.1 · p. 4

3.2 A Type 1 Pattern in the Grokking Setting

• In modular division, 0/b=0 mod n yields an early above-chance spike; later Type 2 features likely push remaining-data accuracy below chance through train/test anti-correlation.Source: §3.2 · p. 4

Pattern Learning at Different Speeds

• Earlier accounts explain epoch-wise double descent through patterns or network parts learned at different speeds, with some findings showing that aligning or accelerating slow features can avoid it.Heckel & Yilmaz (2020), Pezeshki et al. (2021), and Stephenson & Lee (2021) are discussed.Source: §4 · p. 4

• Figure 5 depicts model-wise grokking across embedding dimensions, with delayed accuracy generalization and test loss rising before a fast descent.Each point is a transformer trained for 400K optimization steps.Source: Figure 5 · p. 5

Pattern Learning at Different Speeds

• The framework distinguishes pattern strength from learning speed and introduces per-pattern generalization parameters and preferred patterns.Source: §4 · pp. 4–5

Double Descent

• Double descent describes test performance that improves, worsens, then improves again as training capacity increases, including when optimization steps vary.Belkin et al. (2018) and Nakkiran et al. (2021) are cited.Source: §4 · p. 5

Grokking

• Prior work distinguished grokking from double descent, but accelerated grokking and double-descent findings are presented as consistent with similar pattern-development speeds.Power et al. (2022), Liu et al. (2022), and Heckel & Yilmaz (2020) are discussed.Source: §4 · p. 5

Weight Decay

• Weight decay is reported to constrain capacity and impede slower features in double descent, but speed generalization in grokking; the authors speculate it hinders memorization-based solutions.Source: §4 · p. 5

• The paper argues that grokking and double descent are best understood as instances of the same underlying learning dynamics.Source: §5 · p. 5
• The authors identify unpredictable out-of-distribution generalization as a concern when assessing model safety before scaling beyond controllable capability levels.Source: §5 · p. 5
• They state that a robust theory of generalization and learning in neural networks may be necessary to address this safety problem.Source: §5 · p. 5

• A pattern is treated as a function from a subset of the input distribution to the model output, rather than from the entire distribution.Source: A An alternative intuition for… · p. 8
• A pattern classifies a training example successfully only when that example lies within its domain.Source: A An alternative intuition for… · p. 8
• For test data, allocation is uniform among patterns whose domains include the point, and the assigned pattern classifies it successfully with probability g_j.Source: A An alternative intuition for… · p. 8
• A preferred pattern has a test-set domain disjoint from every other pattern's domain.Source: A An alternative intuition for… · p. 8

• All experiments use decoder-only transformers with causal attention masking, and evaluate loss and accuracy only on the equation answer.Source: B Experiment Details · p. 8
• Unless stated otherwise, the model has two layers, width 128, and a single attention head.Source: B Experiment Details · p. 8
• Typical training runs last 400,000 optimization steps using AdamW, learning rate 1e-3, and weight decay 1e-5.The reported AdamW parameters are β1 = 0.9 and β2 = 0.98.Source: B Experiment Details · p. 8

C Discussion: how relevant is the science of deep learning for AI safety?

• Understanding how models scale and behave is one aspect of making systems well-understood, which the paper identifies as important for safe AI.Examples include scaling laws, infinite-width behavior, and training curves across input types.Source: C Discussion: how relevant is the… · p. 8
• A science of deep learning could help predict when and how neural networks acquire capabilities or behaviors, and may improve interpretability tools.The paper gives examples involving systems that model humans or deceive operators after gaining relevant capabilities.Source: C Discussion: how relevant is the… · p. 8
• Scientific or empirical breakthroughs could accelerate potentially dangerous capabilities, and the paper says its argument for prioritizing safety is speculative.The authors tentatively argue that safe human-level AI may require a more robust theory than building human-level AI alone, while acknowledging that this argument may be wrong.Source: C Discussion: how relevant is the… · p. 8

D Additional Figures

• Figure 6 shows faint double descent in test accuracy and delayed generalization across training steps and model size.Delayed generalization appears vertically across epochs and horizontally across model sizes.Source: Figure 6 · p. 9
• 96 correct examples mark a brief peak when the model has perfect accuracy on 0/b = 0 and chance accuracy (1/97) on other data.The figure attributes this early performance to learning 0/b = 0 mod n.Source: Figure 7 · p. 9
• After a brief peak of 96 correct examples—perfect performance on 0/b = 0 and chance performance (1/97) on other data—test accuracy on all other examples falls below chance due to memorization.Source: Figure 7 · p. 9