PaperlayerView exact revision on arXiv

Source-linked AI summary

Progress measures for grokking via mechanistic interpretability

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt

arXiv:2301.05217v3cs.LGcs.AI

TL;DR

Abrupt emergent behavior is difficult to explain mechanistically, so this paper reverse-engineers grokking in small transformers trained on modular addition. It finds that grokking reflects gradual circuit formation followed by cleanup of memorizing components, rather than a sudden shift.

  • Problem

    Existing progress measures describe smoother changes around emergence but do not explain the mechanisms underlying abrupt transitions such as grokking.

  • Method

    The paper reverse-engineers transformer weights and activations, validates the Fourier-based addition circuit with Fourier-space ablations, and derives two mechanism-specific progress measures.

  • Results

    Training separates into memorization, circuit formation, and cleanup, with progress measures improving continuously before grokking and perfect test accuracy emerging during cleanup.

  • Takeaways & Limitations

    For this modular addition task, mechanistic interpretability provides a proof of concept for explaining grokking through continuous progress toward a learned algorithm.

  • Takeaways & Limitations

    The progress measures are specific to small transformers solving one algorithmic task, and extending them to larger realistic models requires automated, task-independent analysis.

Abstract

from arXiv · show

Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous \textit{progress measures} that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverse-engineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of ``grokking'' exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.

1 INTRODUCTION

The paper uses mechanistic interpretability to explain grokking in modular addition transformers and identify continuous progress measures. The learned algorithm maps inputs to Fourier features on a circle, while training progresses through memorization, circuit formation, and cleanup.

  • Motivation and approach: Mechanistic explanations reverse engineer network circuits to uncover hidden progress measures underlying grokking’s transition from overfitting to generalization.The paper frames progress measures as metrics that precede and are causally linked to phase transitions while varying smoothly.
  • Learned algorithm: The transformer maps inputs to sine and cosine features at sparse key frequencies, combines them with trigonometric identities, and computes addition as rotation on a circle.The output matrices shift and combine frequencies to represent a+b modulo P.
  • Mechanistic validation: The authors validate the algorithm through periodic weights and activations, sinusoidal neuron-logit structure, readable trigonometric identities, and polynomial approximations of circuit components.These constitute four lines of evidence presented for the reverse-engineered mechanism.
  • Progress measures: Restricted loss ablates non-key frequencies, whereas excluded loss ablates key frequencies; both improve continuously before grokking occurs.These metrics are constructed from the learned algorithm for the modular addition task.
  • Training dynamics: Training divides into memorization of training data, circuit formation, and cleanup, making grokking a gradual process rather than a sudden shift.The supplied passage introduces these three phases as the training dynamics revealed by the progress measures.

2 RELATED WORK

Prior work documents emergent phase changes and grokking, develops mechanistic-interpretability approaches, and introduces progress measures for studying smooth precursors to qualitative behavior.

  • Phase Changes: Neural networks can rapidly develop novel qualitative behaviors as they scale or train longer, including AlphaZero’s acquisition of chess concepts and opening theory.AlphaZero learned many human chess concepts between 10k and 30k training steps and reinvented human opening theory between 25k and 60k steps.
  • Grokking: Grokking was first reported as a sharp rise in test accuracy long after two-layer transformers reached perfect training accuracy on algorithmic tasks.Millidge attributed this to SGD’s random walk on the optimal manifold, whereas Barak et al. argued networks continuously progress toward the generalizing algorithm.
  • Circuits-style mechanistic interpretability: The paper’s post-hoc mechanistic-interpretability approach is heavily inspired by the Circuits framework developed in prior work.The cited Circuits approach includes work by Cammarata et al., Elhage et al., and Olsson et al.
  • Progress measures: Progress measures are metrics that improve smoothly before emergent behavior, and this work discovers them empirically through mechanistic interpretability rather than theory alone.Barak et al. theoretically showed training would amplify a certain mechanism and heuristically defined a progress measure.

3 SETUP AND BACKGROUND

The paper studies transformers trained to add numbers modulo P, using a mainline one-layer architecture and a 30% training subset. These networks first memorize the training data and later generalize through a Fourier multiplication algorithm.

  • Task and model setup: The mainline model performs modular addition on inputs “a b =” with P = 113, d = 128 embeddings, 4 attention heads, and n = 512 MLP units.Inputs a and b are P-dimensional one-hot vectors, and the output c is read above the special “=” token.
  • Training setup: Training uses 30% of the 113 · 113 possible input pairs, full-batch AdamW with γ = 0.001 and λ = 1, for 40, 000 epochs.Test loss and accuracy are evaluated on every pair excluded from training.
  • Grokking behavior: Around 10, 000 epochs, test accuracy increases to near 100% after training accuracy reaches 100% while test accuracy initially remains low.The networks first overfit the training set before generalizing, and robustness experiments find grokking across other architectures and prime moduli.
  • Learned algorithm: The proposed Fourier multiplication algorithm maps inputs to sine and cosine features, computes sum-angle features with trigonometric identities, and scores each candidate output c.Attention and MLP layers compute the sum features, while the output and unembedding matrices implement the candidate-dependent cosine calculation and aggregate frequencies.

4 REVERSE ENGINEERING A ONE-LAYER TRANSFORMER

The transformer implements Fourier multiplication through periodic, frequency-sparse representations and trigonometric computations of input sums. Weight and activation analyses, low-dimensional logit approximations, and targeted ablations consistently validate this mechanism.

  • 4.2 Mechanisms in model weights: The unembedding matrix is approximately rank 10, with directions corresponding to cosine and sine at each key frequency, while MLP projections compute cos(w_k(a + b)) and sin(w_k(a + b)).The ten directions reconstruct W_L with residual Frobenius norm under 0.55%, and the MLP representation explains over 90% of variance.
  • 4.1 Periodicity and Fourier structure: Fourier analyses reveal periodic network representations concentrated at five key frequencies, with embeddings sparse in the Fourier basis and logits forming twenty significant components.The key frequencies are k ∈ {14, 35, 41, 42, 52}; the logit components correspond to sine-cosine products at these frequencies.
  • 4.2 Mechanisms in model weights: The logits are well approximated by five terms cos(w_k(a + b − c)), explaining 95% of variance and improving test loss from 2.4 · 10^-7 to 4.7 · 10^-8.This shows that the high-dimensional output is captured by the five key-frequency directions predicted by the interpretation.
  • 4.3 Individual neurons: 433 of 512 MLP neurons, or 84.6%, have over 85% of their variance explained by a degree-2 polynomial of a single key frequency.The corresponding neuron-to-logit maps are frequency-localized, with each frequency cluster mapping through sine and cosine components at that frequency.
  • 4.4 Ablations: Ablations show that key frequencies and the ten W_L directions are necessary, while removing other Fourier components improves performance.Restricting MLP activations to the predicted trigonometric components improves loss by 77% to 5.54 · 10^-8, and removing all non-key Fourier components drops loss 70% to 7.24 · 10^-8.

5 UNDERSTANDING GROKKING BEHAVIOR USING PROGRESS MEASURES

Mechanistic ablations yield restricted and excluded loss as progress measures for tracking training. These measures reveal three continuous phases—memorization, circuit formation, and cleanup—in which Fourier structure forms before grokking and weight decay removes memorizing components.

  • Progress measures: Restricted and excluded loss translate Fourier-space ablations into progress measures that track how training shifts between memorization and generalization.Restricted loss retains the constant and 20 terms for five key frequencies; excluded loss removes those frequencies while measuring training loss.
  • Three phases: Training splits into three phases: memorization, circuit formation, and cleanup.The phases are identified using excluded loss, restricted loss, Gini coefficient, and sum of squared weights.
  • Memorization: During memorization (Epochs 0k–1.4k), train and excluded loss decline while test and restricted loss remain high, indicating unused final-model frequencies.The Gini coefficient also stays relatively flat during this phase.
  • Circuit formation: During circuit formation (Epochs 1.4k–9.4k), excluded loss rises and restricted loss falls while train and test loss stay flat, indicating a smooth transition toward Fourier multiplication.The sum of squared weights falls, suggesting weight decay contributes to circuit formation, which occurs well before grokking.
  • Cleanup: During cleanup (Epochs 9.4k–14k), restricted loss and test performance improve as weight magnitude drops sharply, while excluded loss plateaus.Weight decay favors the lower-weight completed Fourier multiplication circuit over the memorization circuit, helping remove memorized components.
  • Role of weight decay: Weight decay appears important for grokking: smaller amounts cause networks to take significantly longer to grok.The paper relates this to progress toward the generalizing solution and to inflection points in the ℓ2-norm of the weights.

6 CONCLUSION AND DISCUSSION · A MATHEMATICAL STRUCTURE OF THE TRANSFORMER

The paper uses mechanistic interpretability to identify continuous progress measures for a transformer that computes modular addition through rotations in R2 and trigonometric identities. It also specifies the model’s mathematical structure while noting limits in scaling the analysis and predicting phase-transition timing.

  • 6 CONCLUSION AND DISCUSSION: The transformer embeds inputs as rotations in R2 and composes them with trigonometric identities to compute a + b mod 113.This reverse-engineered algorithm underlies the paper’s progress measures.
  • 6 CONCLUSION AND DISCUSSION: Two progress measures increase continuously toward the final algorithm before the grokking phase change.The measures support interpreting grokking on this task as continuous progress before the transition.
  • 6 CONCLUSION AND DISCUSSION: The analysis required substantial manual effort, and its progress metrics are specific to small networks solving one algorithmic task.The paper identifies larger models and harder tasks as requiring larger and more numerous circuits.
  • 6 CONCLUSION AND DISCUSSION: The measures increase relatively smoothly before the phase transition, but no general notion of criticality predicts when that transition will occur ex ante.Future work is proposed to develop theory and practice for predicting emergent-behavior timing.
  • A MATHEMATICAL STRUCTURE OF THE TRANSFORMER: The model has dvocab = 113, dmodel = 128, dhead = 32, and dmlp = 512.These denote the vocabulary size, residual-stream width, per-head query/key/value width, and MLP neuron count, respectively.
  • A MATHEMATICAL STRUCTURE OF THE TRANSFORMER: The parameterization includes embedding, positional-embedding, attention, MLP, and unembedding matrices, with biases absent from embedding, attention, and unembedding layers.The embedding and unembedding matrices are not tied.
  • A MATHEMATICAL STRUCTURE OF THE TRANSFORMER: Loss is computed from final-token logits, while information moves between tokens only during attention.The notation tracks token inputs and positional embeddings, attention scores, residual streams, MLP activations, and final logits.

A.1 EMPIRICAL MODEL SIMPLIFICATIONS

The model admits two empirical simplifications: self-attention from ‘=’ is negligible, and the MLP skip connection can be ignored without changing accuracy. These simplify attention to a two-element softmax and make the composed matrix W_L = W_UW_out the meaningful object.

  • Self-attention from ‘=’ to itself averages only 0.1%–0.4% per head, and ablating it does not affect model performance.
  • The MLP skip connection is computationally unimportant: zero or mean ablation leaves accuracy unchanged, while loss rises from 2.4 · 10−7 to 9.12 · 10−7 or 7.25 · 10−7.
  • With self-attention removed, attention becomes a softmax over 2 elements, equivalently a sigmoid over their difference.
  • Because the MLP components compose linearly, Logits ≈ W_UW_out^MLP is represented by the meaningful matrix W_L = W_UW_out rather than either constituent alone.

B WHY USE CONSTRUCTIVE INTEREFERENCE? … C.1.1 PERIODICITY IN THE ACTIVATIONS OF OTHER ATTENTION HEADS

The model combines multiple periodic frequencies so constructive interference sharply favors x = 0 mod 113, while attention patterns across all four heads show striking periodicity with shared and distinct frequencies.

  • B WHY USE CONSTRUCTIVE INTEREFERENCE?: Multiple frequencies are added to create a function whose value at x = 0 mod 113 is significantly larger than at other x.This is the stated motivation for constructive interference in small transformers trained on the task.
  • B WHY USE CONSTRUCTIVE INTEREFERENCE?: The resulting function has period 113 and is maximized at x = 0 mod 113, but f14(8) = f14(105) = 0.998 and f14(16) = f14(89) = 0.994.A single frequency leaves several nonzero inputs close to the maximum value.
  • B WHY USE CONSTRUCTIVE INTEREFERENCE?: Adding f35 makes f35(8) = f35(105) = −0.990, reducing the peak near x = 8 when combined with f14.Additional frequencies destructively interfere at non-target inputs.
  • B WHY USE CONSTRUCTIVE INTEREFERENCE?: Adding a third frequency reduces the peak at x = 16 because f35(16) = 0.961 and f52(16) = −0.56.The passage gives this as a second example of suppressing non-target peaks through frequency addition.
  • B WHY USE CONSTRUCTIVE INTEREFERENCE?: Figure 8 shows the constructive interference produced by the cosine waves for the five frequencies used by the mainline model.The figure directly visualizes the combined-frequency mechanism.
  • C SUPPORTING EVIDENCE FOR MECHANISTIC ANALYSIS OF MODULAR: The supporting analysis examines additional evidence from the specific mainline training run.This section introduces the further analysis that follows.
  • C.1 FURTHER ANALYSIS OF THE SPECIFIC TRAINING RUN DISCUSSED IN THE PAPER: Figure 9 shows attention patterns from the final ‘=’ token to the first token a for all four heads as heatmaps over inputs a and b.The analysis treats this attention pattern as a scalar for each head.
  • C.1.1 PERIODICITY IN THE ACTIVATIONS OF OTHER ATTENTION HEADS: All four attention heads exhibit striking periodicity, with heads 1 and 3 sharing one frequency while heads 0 and 2 use different frequencies.The figure caption independently describes periodicity across every head, and the paragraph identifies the frequency-sharing pattern.

C.1.2 APPROXIMATING ATTENTION HEADS WITH SINES AND COSINES … C.2 ADDITIONAL RESULTS FROM DIFFERENT RUNS

The paper reverse-engineers attention heads as Fourier-structured computations: some form degree-2 trigonometric polynomials, while others amplify key frequencies. Additional activation, grokking, and cross-run results show periodic structure, phase-linked inflection points, and variation in the frequencies used across models.

  • C.1.2 APPROXIMATING ATTENTION HEADS WITH SINES AND COSINES: The attention pattern weights are well approximated by a 0.5 intercept plus differences of sine and cosine waves at one frequency per head.Fitting single-frequency waves to Cj explains 99.02%, 95.21%, 99.10%, and 92.42% of variance for the four heads, respectively.
  • C.1.2 APPROXIMATING ATTENTION HEADS WITH SINES AND COSINES: The softmax attention pattern is effectively linear over the heads’ input range, explaining 97.5% of its variance.Replacing the sigmoid with its best linear fit decreases test loss from 2.41 · 10^-7 to 2.12 · 10^-7.
  • C.1.2 APPROXIMATING ATTENTION HEADS WITH SINES AND COSINES: Attention heads 0 and 2 approximately compute degree-2 polynomials of sines and cosines at single frequencies, while heads 1 and 3 amplify key frequencies.The attention patterns are well approximated by differences of sine and cosine waves, and the OV circuits of heads 1 and 3 contain all five key frequencies.
  • C.1.4 PERIODICITY IN THE ACTIVATIONS OF ADDITIONAL NEURONS: Four additional MLP neurons exhibit activation patterns that are periodic in both input variables.The same periodicity is observed across neurons 1, 2, 3, and 4 as in neuron 0.
  • C.1.5 ADDITIONAL GROKKING FIGURES FOR MAINLINE RUN: Restricting the model to the five key Fourier frequencies improves performance during training and shows generalization before noise removal.The associated accuracy, restricted-loss, and excluded-loss figures have inflection points corresponding to the phases of grokking.
  • C.1.5 ADDITIONAL GROKKING FIGURES FOR MAINLINE RUN: The mainline grokking figures contain inflection points corresponding to the relevant grokking phases.This applies to the three figures discussed in the subsection.
  • C.2 ADDITIONAL RESULTS FROM DIFFERENT RUNS: Across additional runs, models learn variants of the modular arithmetic algorithm but use varying numbers of different key frequencies.Key frequencies for excluded and restricted loss are identified by applying a DFT to the neuron-logit map WL and selecting frequencies with non-

C.2.1 ADDITIONAL RESULTS FOR DIFFERENT RUNS WITH THE SAME ARCHITECTURE

Four additional random-seed runs with the same architecture consistently learned the Fourier multiplication algorithm and exhibited the same three grokking phases as the mainline model. Key-frequency ablations and Fourier-based progress measures supported this shared mechanism, while the timing of circuit formation and cleanup varied across runs.

  • Confirming the Fourier Multiplication Algorithm: All 4 additional runs used sparse Fourier representations in both the embedding matrix WE and neuron-logit map WL, identifying 3 or 4 key frequencies per seed.The sparsity of WE and WL in the Fourier basis provided evidence that these networks operated in that basis.
  • Confirming the Fourier Multiplication Algorithm: The MLP activations in directions corresponding to the key frequencies approximated the trigonometric identities for cos(w_k(a + b)) and sin(w_k(a + b)).The activations were approximated with single cosine or sine terms associated with the key frequencies.
  • Confirming the Fourier Multiplication Algorithm: Ablating all key frequencies reduced performance to worse than chance, whereas retaining only the key frequencies improved test performance.For seed 1, test loss was 2.07 · 10−7 before ablation, 6.5 · 10^0 with key frequencies removed, and 5.7 · 10−8 with all other frequencies removed.
  • Progress measures and grokking: The four additional runs reproduced the memorization, circuit formation, and cleanup phases, with restricted loss declining before train loss.All models completed memorization by around 1400 epochs, but circuit formation and cleanup occurred at different times.

C.2.2 RESULTS FOR OTHER EXPERIMENTAL SETUPS · C.2.3 GENERALIZING MODELS CONSISTENTLY USE THE FOURIER MULTIPLICATION ALGORITHM · D ADDITIONAL RESULTS ON GROKKING

Across varied modular-addition setups, grokking depends on data fraction, architecture, modulus, and weight decay, while correctly generalizing models consistently implement a Fourier multiplication algorithm. Generalization is immediate in some high-data settings, delayed or absent in others, and dropout models show weaker Fourier-structure evidence.

  • C.2.2 RESULTS FOR OTHER EXPERIMENTAL SETUPS: 1-layer models across these setups converge to the Fourier multiplication algorithm, while 2-layer models also grok but are complicated by the slingshot mechanism.The excluded loss of a 2-layer model improves relatively smoothly before grokking.
  • C.2.2 RESULTS FOR OTHER EXPERIMENTAL SETUPS: For P = 53, models do not generalize with λ = 1 but grok with λ = 5, plausibly because the smaller memorization solution requires more aggressive weight decay.There are only 53 · 53 total input pairs in the smaller-prime case.
  • C.2.2 RESULTS FOR OTHER EXPERIMENTAL SETUPS: For P = 113, grokking occurs with 30−50% of training data, lower fractions slow it, and ≥60% produces immediate generalization.Using 10% or 20% does not produce grokking within 40k epochs.
  • C.2.2 RESULTS FOR OTHER EXPERIMENTAL SETUPS: Train loss reaches below 10−6 immediately for every data fraction, whereas generalization takes significantly longer with less training data.Other loss thresholds show qualitatively similar behavior.
  • C.2.2 RESULTS FOR OTHER EXPERIMENTAL SETUPS: For P = 401, models immediately learn the generalizing solution across λ ∈{0.3, 0.5, 1, 3, 5, 8}, with the increased data volume favoring generalization from the start.For P = 109, behavior matches the mainline model.
  • C.2.3 GENERALIZING MODELS CONSISTENTLY USE THE FOURIER MULTIPLICATION ALGORITHM: Every weight-decay-trained model that generalizes correctly implements some variation of the Fourier multiplication algorithm.The analysis compares key frequencies, Gini coefficients, and relevant Fourier variance explained values.
  • C.2.3 GENERALIZING MODELS CONSISTENTLY USE THE FOURIER MULTIPLICATION ALGORITHM: Dropout models are less sparse in the Fourier basis, and p = 0.2 logits are less well explained by cosine sums, likely reflecting insufficient training or redundancy requirements.Dropout models appear to take much longer to grok, while p = 0.2 models may simply perform worse.
  • C.2.3 GENERALIZING MODELS CONSISTENTLY USE THE FOURIER MULTIPLICATION ALGORITHM: Generalizing models incur no significant performance penalty when the final MLP skip connection is zero or mean ablated.The final skip connection is therefore ignored in the analysis.

D.1 BOTH REGULARIZATION AND LIMITED DATA ARE NECESSARY FOR GROKKING … E.1 AN INTUITIVE EXPLANATION OF GROKKING

Grokking requires limited data together with suitable regularization, while slingshots are unnecessary and the pattern extends across several algorithmic tasks. The paper’s speculative explanation frames grokking as competition between memorization and Fourier-based generalization, shaped by phase transitions and regularization.

  • D.1 BOTH REGULARIZATION AND LIMITED DATA ARE NECESSARY FOR GROKKING: With weight decay λ = 1, smaller data fractions delay grokking despite approximately identical train losses.
  • D.1 BOTH REGULARIZATION AND LIMITED DATA ARE NECESSARY FOR GROKKING: Grokking takes around 3k epochs at λ = 0.3, 5-10k at λ = 1.0, and 20k at λ = 3.0; dropout can induce it, whereas ℓ1 regularization cannot.Dropout with p = 0.2 or p = 0.5 produced grokking for all three seeds, while ℓ1 regularization produced none.
  • D.1 BOTH REGULARIZATION AND LIMITED DATA ARE NECESSARY FOR GROKKING: Without weight decay, 1-layer transformers show no grokking and excluded loss does not increase.
  • D.2 THE SLINGSHOT MECHANISM OFTEN OCCURS, BUT IS UNNECESSARY FOR GROKKING: Slingshots are unnecessary for grokking when weight decay or other regularization is present, although they may act as implicit regularization without weight decay.
  • D.3 ADDITIONAL EVIDENCE FROM OTHER ALGORITHMIC TASKS: Across 5-digit addition, repeated subsequences, and skip trigram, restricting data was necessary and sufficient for grokking, while freshly sampled data prevented it.
  • 5 Digit Addition: With approximately infinite randomly generated data, 5-digit addition showed coincident train and test loss and no grokking because repeated pairs were almost never seen.Small bumps corresponded to learning each of the 6 output tokens.
  • E.1 AN INTUITIVE EXPLANATION OF GROKKING: The proposed explanation contrasts memorization, which has low train and high test loss, with generalization, which has low train and test loss.The authors explicitly state that this evidence is suggestive and a promising direction for future research.
  • E.1 AN INTUITIVE EXPLANATION OF GROKKING: The Fourier Multiplication Algorithm is presented as the generalization solution, while limited data, regularization, and phase transitions jointly produce grokking.The speculation argues that memorization complexity scales with training-set size, whereas generalization complexity is constant, and weight decay favors simpler solutions.

E.2 HYPOTHESIS: PHASE TRANSITIONS ARE INHERENT TO COMPOSITION

The section argues that phase transitions can arise inherently when neural networks form composed circuits: individual components may be ineffective alone but become increasingly useful together. This lottery-ticket-style composition explains why circuit formation can produce nonlinear, accelerating progress rather than smooth linear improvement.

  • Circuit composition: Mechanistic interpretability identifies circuits as subnetworks implementing discrete algorithms, including curve detectors and induction heads.These circuits are clean, interpretable algorithms formed from model subnetworks.
  • Circuit composition: An induction circuit combines a previous-token head with an induction head to continue repeated subsequences.The previous-token head copies earlier context, while the induction head uses that information to predict what follows.
  • Composition problem: Because each induction-head component is ineffective without the other, gradients cannot initially directly encourage formation of the complete circuit.Before either head is present, neither component receives a gradient incentive to form the circuit; the circuit improves loss only through the other heads.
  • Candidate explanations: The lottery-ticket hypothesis is the leading explanation: networks begin with many partial circuit components, and useful components gain advantage through composition despite interference from others.Evolution and random walks may contribute, but the modular-addition task shows hidden pre-grokking progress, arguing against chance discovery alone.
  • Phase-transition hypothesis: As circuit components improve, their mutual usefulness should make gradients grow nonlinearly, producing acceleration as the circuit approaches completion.Early rough components weakly affect loss, while increasingly developed components make one another more useful, yielding a phase-transition-like training dynamic.

F FURTHER DISCUSSION ON USING MECHANISTIC INTERPRETABILITY AND PROGRESS MEASURES FOR STUDYING EMERGENT PHENOMENA

Mechanistic interpretability offers a promising route to defining progress measures for emergent phenomena, but its scalability to large models remains uncertain. The authors therefore view future work as necessary for applying this approach to existing large language models, while remaining cautiously optimistic.

  • Limitations: The scalability of existing mechanistic interpretability approaches—and the availability of human-understandable explanations for all sophisticated behavior—remains uncertain.This uncertainty limits confidence in directly extending the approach to large models.
  • Potential methods: Mechanistic interpretability could support progress measures for emergent phenomena if it scales to understanding mechanisms behind significant portions of large models’ behavior.The authors note that this would enable analyses similar to the paper’s case study, but it is unclear whether mechanistic interpretability can scale that far.
  • Future directions: Future work outlined in Section 6 is necessary to apply this approach to predicting and understanding emergent behavior in existing large language models.Despite this requirement, the authors remain cautiously optimistic about the approach’s prospects.
Loading 2301.05217v3…