PaperlayerView exact revision on arXiv

Source-linked AI summary

An Exploratory Replica-Overlap Probe of the Grokking Transition

A. C. Opus, J. Q. Lu

arXiv:2609.25634v1cs.LGcond-mat.dis-nncs.AI

TL;DR

The paper asks whether pairwise replica-overlap distributions change across grokking, a question its registered test cannot resolve because the alignment is not function-preserving. The study’s registered outcome is UNDETERMINED; exploratory frac40 results include a dip interval containing zero and an overlap-standard-deviation increase, but the interval is uninformative.

  • Problem

    The registered question is whether replica overlap changes across grokking, but the test does not yield a confirmatory null.

    Source: §8 · p. 11

  • Method

    The study applies a registered overlap-distribution test subject to checkpoint-completeness and alignment requirements.

    Source: §3 · pp. 3–4

  • Results

    The registered outcome is UNDETERMINED; for frac40, the post-hoc Hartigan-dip interval contains zero while overlap standard deviation increases by about 5.6 times.

    Source: §8 · p. 11

  • Takeaways & Limitations

    A valid follow-up requires function-preserving alignment, matched-epoch controls, and pre-specified power calibration, while also handling remaining network symmetries.

    Source: §8 · p. 11

    The non-function-preserving alignment, failed precision requirement, and protocol deviations prevent confirmatory inference from the registered test.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We trained 64 independently seeded networks in four configurations, continuing each to sustained convergence or a 40,000-epoch ceiling. We then asked whether an RSB-inspired distribution of pairwise weight overlaps changes across the grokking transition. It is the alignment step, not the overlap statistic, that determines what this registered probe can report. The registered implementation permutes hidden units without the corresponding bias and head-internal permutations and therefore does not preserve the network function. Every q_wt value computed through this alignment inherits the defect; q_fn does not, because it is computed from predictions of the unpermuted models. The numerical-precision requirement also failed, and an audit found protocol deviations. Consequently, the pre-registered rule gives no verdict: registered outcome UNDETERMINED (reason code C0_INSTRUMENT_INVALID). These data provide neither a confirmatory null nor a validated reading of the Parisi order parameter. Only frac40 cleared the 12/16 checkpoint-completeness requirement. For this configuration, a post-hoc criterion applied to the same data gave a Hartigan-dip interval containing zero (95% CI for Delta dip = [-0.017, 0.034]), whereas the overlap standard deviation increased by a factor of about 5.6. A post-hoc calibration assigns the dip test zero power at the simulated separations; the interval is therefore uninformative, not evidence of no change. The standard-deviation ratio is the only statistic here with power at the observed effect. Ensemble loss was near-flat only under the pre-specified 1% threshold. Finally, grokking rates of 0/16, 11/16 and 16/16 remain descriptive because train fraction is confounded with split identity.

1 Introduction

Prior grokking accounts describe individual-network mechanisms, while the replica method offers a way to study agreement among independently trained solutions. This pre-registered probe asks whether pairwise-overlap distributions change across grokking, but its registered test is invalid and its exploratory findings do not establish an RSB-consistent change.

1 Introduction

  • Prior grokking measures track one network or trajectory, not agreement among independently trained solutions.Nanda et al. [2023] identify Fourier-basis progress measures, while Cullen et al. [2026] and Xu [2026] use geometric quantities.Source: §1 · p. 1
  • Replica-symmetry-breaking theory predicts a potentially nontrivial distribution of pairwise overlaps among replicas sharing the same disorder.In the replica-symmetric phase, the distribution is a single delta function; with RSB, it may become multimodal [M´ezard et al., 1987].Source: §1 · p. 1
  • Prior neural-network applications of RSB have not asked whether pairwise-overlap shape changes across grokking for genuinely independent replicas.The cited applications include perceptron storage, associative memories, restricted Boltzmann machines, and feedforward networks.Source: §1 · p. 1
  • The registered test cannot provide a confirmatory null; exploratory analysis of the eligible configuration finds no evidence of an RSB-consistent change in overlap-distribution shape.Grokking frequency is descriptively associated with train fraction across three combinations, but fraction and split identity are confounded.Source: §1 · p. 2

2 Related Work

Prior work addresses grokking through single-trajectory mechanisms, phase-transition theories, and multi-network comparisons, but with different objects and replica constructions. The surveyed studies do not combine independently trained replicas, a full pairwise-overlap distribution, and a decision rule tied to grokking.

2 Related Work

  • Local entropy measures overlap-constrained low-loss volume around one reference solution, unlike this study’s pairwise overlaps between independent replicas across a transition.Bae and Jeong [2026] relate local entropy to dataset complexity; their work does not study grokking.Source: §2 · p. 2
  • Lin [2026] report dose-ordered shifts in generalization time and collapsing test-loss barriers, but their barriers compare checkpoints from one trajectory.Their study does not directly test the independent-replica alignment premise used here.Source: §2 · p. 2
  • Grokking theories use tensor networks, adaptive kernels, or single-trajectory geometry rather than replica-overlap distributions.ˇZunkoviˇc and Ilievski [2022] derive critical exponents and grokking-time distributions; Rubin et al. [2023] identify an analogy to a mixed phase.Source: §2 · p. 2
  • Kataria [2026] map grokking boundaries across 384 configurations, while Wang et al. [2026] bound prediction variation along a training trajectory; neither forms replica overlaps.Kataria [2026] identify a sharp weight-decay boundary, but this study does not externally check it.Source: §2 · p. 2
  • Mode-connectivity work finds low-loss paths between trained optima, while permutation alignment and weak-versus-strong alignment studies address whether networks can be jointly aligned.Liao et al. [2024] report a hierarchy via clustering rather than an overlap distribution; lottery-ticket work motivates ensemble size.Source: §2 · pp. 2–3
  • No surveyed study combines independently seeded replicas, a full pairwise-overlap distribution, and a registered decision rule tied to grokking.Li [2025] uses thermal samples from one trained network, while Chan et al. [2026] branch replicas from a shared checkpoint.Source: §2 · p. 3

3 Method

The study trains independently seeded replicas across four configurations and plans an overlap-distribution comparison after checkpoint eligibility and permutation alignment. The alignment implementation does not preserve network function, invalidating the weight-overlap instrument; the function-space measure saturates after grokking, and the ensemble-size probe is separately defined.

3 Method

  • The study uses 16 independently seeded replicas in each of four configurations on modular addition with p = 113, stopping at sustained 99% test accuracy or the 40,000-epoch ceiling.Frac25 and nogrok replicas all reached the ceiling; frac30 was mixed, and all frac40 replicas stopped early, with event-aligned checkpoints.Source: §3 · p. 3
  • Only configurations with at least 12 of 16 replicas reaching both required checkpoints are eligible for overlap analysis; this gate is not a power calculation.Configurations that miss the gate are reported as RAISE rather than as a null verdict.Source: §3 · p. 3
  • Pairwise weight overlap is aligned using residual-stream and MLP-hidden-layer permutations, but omitted bias and head-internal permutations make the transformation non-function-preserving.The audit found nonzero biases in analyzed checkpoints; the defect can create or erase structure in P(q), invalidating the registered instrument.Source: §3 · pp. 3–4
  • The primary statistic is aligned weight-space overlap qwt; Hartigan’s dip and a bootstrap assess shape change, with a failed fp32/fp64 precision rail requiring an UNDETERMINED verdict.The pre-registration left open whether overlap structure would expand or collapse across grokking.Source: §3 · p. 4
  • For frac40, prediction agreement qfn saturates at 1.0000 after grokking because every replica reaches perfect test accuracy, motivating replacement with qwt.The post-grokking qfn standard deviation is 0.0000, compared with 0.023 at the plateau.Source: §3 · p. 4
  • The independent ensemble-size probe measures the smallest k whose averaged loss is within 1% of full-ensemble loss, using exhaustive enumeration of the pre-registered 8+8 split.The exhaustive calculation replaces the registered bootstrap for k > 4 and is recorded as deviation D7.Source: §3 · p. 4

4 Protocol deviations

Protocol departures collectively reduce the study from a confirmatory test to an exploratory one. Checkpoint timing and missing intermediate archives prevent reconstruction of the registered comparison.

4 Protocol deviations

  • The authors state that the combined effect of protocol departures reduces the study from confirmatory to exploratory.Source: §4 · p. 4
  • The pre-registration body was committed before any replica finished training, but two later commits amended it after data existed.The initial body was committed on 2026-08-22 at 18:15; the amendments followed on 2026-08-23.Source: §4 · p. 4
  • Saved plateau and post-grokking checkpoints are late by 499 and 999 epochs, respectively, and the required every-500-epoch archive was not implemented.The missing intermediate checkpoints prevent reconstruction of the registered analysis.Source: §4 · p. 5

D2. The analysed configuration stopped early, so “deep in the grokked phase” does not apply

Every frac40 replica stopped early, so the analyzed overlap comparison cannot be described as deep in a post-transition phase. The audit also records limitations affecting the registered controls and interpretations.

D2. The analysed configuration stopped early, so “deep in the grokked phase” does not apply

  • All frac40 replicas stopped early, at a median well below one third of the training budget; the data do not support describing them as deep in the post-transition phase.The only configuration analyzed for overlap stopped before reaching the 40,000-epoch ceiling.Source: D2. The analysed configuration stopped… · p. 5
  • The registered matched-epoch overlap control for nogrok was not analyzed, so it cannot exclude continued optimization and weight decay as causes of overlap movement.The control therefore shows only that random labels do not generalize; excluding the alternative would require retraining.Source: D2. The analysed configuration stopped… · p. 5
  • The dip-test calibration was post hoc and gave zero power at simulated separations of 0.00, 0.01, and 0.02, making the interval uninformative rather than null-supporting.The calibration is provisional because it uses route-A frac40 values from the defective alignment; 16 independent replicas also limit the information available.Source: D2. The analysed configuration stopped… · p. 5
  • The alignment implementation permutes weights without corresponding biases and leaves head-internal symmetries untreated, invalidating the instrument for the registered question.The defect affects the registered overlap probe rather than merely reducing its power.Source: D2. The analysed configuration stopped… · p. 5
  • The nogrok implementation used shared i.i.d. uniform labels instead of the registered label permutation, a documented deviation judged to better serve the control’s intent.A permutation of modular-sum targets would remain a learnable function, defeating the control’s purpose of supplying no generalisable structure.Source: D2. The analysed configuration stopped… · pp. 5–6
  • O2 exhaustively enumerated subsets where the registration required a 100-subset bootstrap for k > 4; the resulting 70 draws fell below that requirement.The registration required exhaustive enumeration only through k = 4.Source: D2. The analysed configuration stopped… · p. 6

5 Results

Only frac40 clears the checkpoint-completeness gate, while the registered overlap analyses remain UNDETERMINED. The reported grokking rates, overlap changes, and ensemble-loss pattern are descriptive or exploratory within the supplied analyses.

5.1 Grokking reliability across train fraction

  • Only frac40 cleared the checkpoint gate; observed grokking rose from 0/16 to 11/16 to 16/16 across configurations, but split identity is confounded with train fraction.The gate makes frac25 and frac30 ineligible for overlap analysis, not null evidence about RSB.Source: Table 1 · p. 6, §5.1 · pp. 6–7
  • The 11 grokked frac30 replicas transitioned at epochs 28,861–37,838 (mean 33,765), spanning about 9,000 epochs and leaving 2,161 epochs at the latest transition.The five frac30 replicas that did not grok exhausted the 40,000-epoch budget.Source: Figure 1 · p. 6, §5.1 · p. 7
  • Frac25 and nogrok held-out accuracy stayed near the 1/113 chance floor (≈0.885%); two frac25 outliers (4.6% and 5.8%) are descriptive, not partial generalization.No test or equivalence margin was specified for this comparison.Source: Figure 2 · p. 7, §5.1 · p. 7

5.3 O2: ensemble-size diminishing returns

  • For O2, eight group-A replicas yielded MA = 1 across 28 post-grokking pairs; the shared precision rail left registered O2 UNDETERMINED, and its sample differs from O1’s.The mode count labels the supplementary result but does not validate it.Source: §5.3 · pp. 8–9

5.2 O1: overlap-distribution shape across the transition

  • Frac40’s registered O1 verdict was UNDETERMINED (C0); post-hoc qwt standard deviation rose 5.6×, while the dip interval [−0.017, 0.034] included zero and had zero calibrated power at simulated separations.The overlap-width change is detectable in this sample, but the dip interval does not establish absence of a distributional change.Source: §5.2 · pp. 7–8, Figure 3 · p. 9

5.3 O2: ensemble-size diminishing returns

  • Mean ensemble loss fell from 9.6839×10^-5 at k = 1 to 9.6746×10^-5 at k = 8, a 0.0969% decrease within the pre-specified 1% near-flat threshold.The curve decreases monotonically; it is not constant.Source: §5.3 · p. 9
  • O2 cannot test whether ensemble returns track overlap modality: it produced only one (MA, k*) point, without individual member losses and their variance or the registered 200-split companion.The stored loss values are means at each ensemble size.Source: §5.3 · p. 9

6 Discussion

The registered study yields no confirmatory null because its instrument is invalid. Exploratory findings for frac40 are limited: the dip result is uninformative, while overlap standard deviation rises, and the discussion identifies design and scope boundaries.

6 Discussion

  • The registered rule yields no confirmatory null: the instrument is invalid, and the primary outcome is UNDETERMINED.The dip analysis is post hoc, and its calibrated power is zero at the simulated separations, making its interval uninformative rather than null-supporting.Source: §6 · p. 9, Limitations · p. 10
  • The Hartigan dip criterion detects no change for frac40, while aligned-overlap standard deviation rises by about 5.6× across the transition.The study does not test whether RSB effects appear transiently or closer to the transition boundary.Source: §6 · pp. 9–10
  • Group-B ensemble loss decreases by 0.0969%, within the pre-specified 1% near-flat threshold; the discussion infers no mechanism.The earlier interpretation as a forced floor effect is retracted.Source: Figure 4 · p. 10
  • The near-boundary RSB hypothesis is exploratory, not derived from spin-glass theory; the cited zero-temperature result instead has infinitely many support points.Establishing the near-boundary regime would require a model-specific derivation, and testing it requires new training because frac30 fails the C1 gate.Source: §6 · p. 10
  • O2 is retained as a design lesson: replacing q_fn with cross-entropy to avoid an accuracy ceiling can encounter a similar ceiling after accuracy saturates.The study does not interpret O2 in either direction.Source: §6 · p. 10

Limitations

  • The study’s scope is limited to one small modular-addition task, one pinned architecture, and one split seed per configuration.Several registered controls did not run; the dip tests unimodality only, and no equivalence margin was specified, so other RSB-consistent changes are not excluded.Source: Limitations · pp. 10–11

7 Successor hypothesis (not pursued in this study)

The successor hypothesis is that RSB effects, if present, may concentrate near the grokking reliability boundary. Testing it requires new training and a better-powered design.

7 Successor hypothesis (not pursued in this study)

  • Testing the exploratory near-boundary hypothesis requires 2–4 new train-fraction configurations clustered near frac30 and an earlier, not-fully-converged checkpoint for an O2-style ensemble test.The proposed configurations are frac28–frac33; this hypothesis is recorded for future work rather than pursued in this study.Source: §7 · p. 11

8 Conclusion

The registered overlap test is UNDETERMINED because its alignment instrument is invalid, while the exploratory frac40 dip interval is uninformative despite an increase in overlap standard deviation. A valid follow-up needs function-preserving alignment and additional controls, including a gauge-invariant order parameter.

8 Conclusion

  • The registered overlap test is UNDETERMINED; for frac40, the dip interval contains zero but is uninformative, while overlap standard deviation increases about 5.6-fold.Grokking rates across the three fraction-by-split combinations remain confounded by split identity.Source: §8 · p. 11
  • A valid test requires function-preserving alignment, matched-epoch controls, and pre-specified power calibration.Discrete permutation alignment alone leaves ReLU positive-rescaling symmetry and within-head Q/K and V/O basis freedom intact.Source: §8 · p. 11
Loading 2609.25634v1…