PaperlayerView exact revision on arXiv

Source-linked AI summary

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

Michael Hanna, Ollie Liu, Alexandre Variengien

arXiv:2305.00586v5cs.CLcs.AIcs.LG

TL;DR

The paper asks how pre-trained language models implement mathematical abilities that they acquire without explicit training. Using mechanistic interpretability, it identifies and analyzes a GPT-2 small circuit for greater-than, finding a complex mechanism that activates across contexts but remains limited in scope.

  • Problem

    How pre-trained language models implement acquired mathematical abilities remains unclear, because existing work is mainly behavioral or lacks causal explanations.

  • Method

    The authors use mechanistic interpretability, path patching, and causal ablations to identify and explain a GPT-2 small circuit for year-span greater-than prediction.

  • Results

    The circuit is mostly sufficient and necessary for the task: the larger-dataset probability difference is 72.7% after circuit-only patching, versus -36.6% when circuit nodes receive the alternate dataset.

  • Takeaways & Limitations

    GPT-2’s greater-than behavior uses a complex circuit that activates across contexts, adding nuance between memorization and generalization.

  • Takeaways & Limitations

    The conclusions are limited by GPT-2’s small model and dataset, the simple phenomenon studied, and the study’s model-centric design.

Abstract

from arXiv · show

Pre-trained language models can be surprisingly adept at tasks they were not explicitly trained on, but how they implement these capabilities is poorly understood. In this paper, we investigate the basic mathematical abilities often acquired by pre-trained language models. Concretely, we use mechanistic interpretability techniques to explain the (limited) mathematical abilities of GPT-2 small. As a case study, we examine its ability to take in sentences such as "The war lasted from the year 1732 to the year 17", and predict valid two-digit end years (years > 32). We first identify a circuit, a small subset of GPT-2 small's computational graph that computes this task's output. Then, we explain the role of each circuit component, showing that GPT-2 small's final multi-layer perceptrons boost the probability of end years greater than the start year. Finally, we find related tasks that activate our circuit. Our results suggest that GPT-2 small computes greater-than using a complex but general mechanism that activates across diverse contexts.

1 Introduction

The paper addresses the unclear mechanisms behind mathematical abilities acquired by pre-trained language models. It studies GPT-2 small with mechanistic interpretability to identify and explain a circuit computing greater-than across contexts.

  • Existing research largely describes what pre-trained language models can do mathematically, rather than causally explaining how they do it.
  • The authors study GPT-2 small because its compact size enables low-level investigation of its mathematical abilities.
  • They use a circuits perspective and causal ablations to identify a minimal computational-graph subset responsible for the task.
  • The case study elicits greater-than by asking GPT-2 to complete year-span sentences with end years greater than the start year.
  • The resulting circuit activates across contexts, exceeding simple memorization while falling short of full mathematical competence.

2 Year-Span Prediction in GPT-2

GPT-2 small is evaluated on a naturally phrased year-span task requiring higher probability for two-digit end years greater than the start year. Across qualitative and quantitative analyses, it exhibits a sharp, start-year-dependent preference for valid end years, though the preference weakens 15–20 years later.

  • Task and Dataset: The task asks GPT-2 to complete “The <noun> lasted from the year XXYY to the year XX” by favoring end years greater than YY.Sentences use nouns drawn from a 120-noun pool, with years sampled from 1000 to 1899.
  • Qualitative Evaluation: Figure 2 maps start year YY against two-digit output candidates, with each cell giving GPT-2’s mean probability for candidate X given start year Y.The figure also reports GPT-2’s average output distribution when YY=41.
  • Qualitative Evaluation: The evaluation uses 10,000 examples with randomly sampled nouns and years, averaging predicted-year distributions separately for each start-year suffix YY.The qualitative analysis considers YY values from 2 through 98 and visualizes the resulting distributions in Figure 2.
  • Qualitative Evaluation: GPT-2 creates a sharp cutoff between invalid end years ≤YY and valid end years >YY, but probabilities decline 15–20 years after YY.The span receiving elevated probability may reflect patterns in GPT-2’s training data.
  • Quantitative Evaluation: The probability-difference metric measures whether GPT-2 assigns more probability to years >YY, while cutoff sharpness measures the sharpness of the valid–invalid boundary.Probability difference ranges from -1 to 1, whereas cutoff sharpness tests whether outputs depend on YY rather than remaining constant.
  • Quantitative Evaluation: 81.7% probability difference (SD: 19.3%) and 6.0% cutoff sharpness (SD: 7.2%) indicate greater-than behavior on the 10,000-element dataset.Both qualitative and quantitative results support the conclusion that GPT-2 performs greater-than on this task.

3 A Circuit for Year-Span Prediction

The authors use causal path patching to isolate and validate a GPT-2 circuit for year-span prediction, then interpret how its attention heads and MLPs implement greater-than.

  • Path Patching: Path patching tests a component by replacing inputs along a specified path and observing the resulting change in model behavior.Unlike interventions that alter broad component effects, it constrains the intervention to particular edges and interactions.
  • Circuit Components: Iterative path patching identifies MLPs 8–11 and attention head a9.h1 as major direct contributors to the logits.Because MLPs cannot attend to the earlier start-year token, the analysis searches for upstream components that provide this information.
  • Circuit Components: The circuit adds attention heads that feed MLPs 8 and 9, including a9.h1, a8.h11, a8.h8, a7.h10, a6.h9, a5.h5, and a5.h1, together with their direct logit connections.These heads contribute more weakly to logits than the MLPs but provide upstream inputs needed by the circuit.
  • Circuit Validation: The circuit is mostly sufficient: probability difference reaches 72.7% (89.5% of the original), while assigning the 01-dataset to circuit nodes yields -36.6%.The evaluation gives most of the model misleading inputs while preserving the standard dataset only along circuit paths.
  • Circuit Validation: Performance on similarly sized alternative circuits improves as more paths overlap the original, with MLPs—especially MLPs 9 and 10—having the largest impact.When no attention head or MLP overlaps the original circuit, performance is very low.
  • Circuit Semantics: Attention heads communicate the start year YY, while MLPs 9, 10, and 11 compute greater-than in tandem; MLP 8 mainly influences downstream MLPs.MLPs 9 and 10 show upper-triangular logit patterns for years greater than YY, while MLP 11 appears to limit the duration.

4 Explaining Greater-Than in the Year-Span Prediction Circuit

The paper explains greater-than as a distributed computation: structured year information reaches several MLPs, whose neurons compose their logit contributions to approximate the operation.

  • Input Structure: PCA reveals that inputs to the relevant MLPs and outputs of relevant attention heads are ordered by the sentence’s start year.Irrelevant attention-head outputs lack this structure, suggesting that relevant heads transmit the information used downstream.
  • Input Structure: Structured year representations may support greater-than, but causal tests provide little evidence that they fully explain the behavior.Removing PCA dimensions had little effect, while LEACE reduced probability difference to 64.7% from the 81.7% baseline.
  • Neuron-Level Processing: Most of MLP 10’s 3072 neurons have near-zero causal effects on task performance, indicating sparse neuron contributions.The authors identify important neurons by path-patching each neuron’s direct contribution to the logits.
  • Neuron-Level Processing: No individual MLP 10 neuron computes greater-than because each neuron scales a fixed logit-distribution shape across inputs.The correct logit shape changes with the start year, so one neuron cannot provide the complete operation.
  • Neuron-Level Processing: The top 10 neurons produce an imperfect greater-than pattern when their contributions are summed, with sharper approximations from more neurons.This shows that the operation is composed across neurons rather than localized in one neuron.
  • Neuron-Level Processing: Across MLP 10 and the other relevant MLPs, multiple neuron outputs sum to form the greater-than computation.The neuron-level composition complements the circuit-level finding that several important MLPs jointly compute the operation.

5 Does The Circuit Generalize?

The greater-than circuit activates on several related prompts, but its generalization is partial: similar tasks use overlapping rather than identical circuits, and the circuit can misfire.

  • Related Greater-Than Tasks: Across three increasingly different prompt formats, GPT-2 performs greater-than with at least a 69% probability difference.The tested formats include year spans, luxury-good prices, and number sequences.
  • Related Greater-Than Tasks: The first two related tasks recover 98.8% and 88.9% of original performance when edges outside the year-span circuit are ablated.These results support substantial circuit reuse across contexts.
  • Related Greater-Than Tasks: The related tasks depend on MLPs 8–11, which receive information about YY from attention heads.This identifies a shared downstream mechanism and shared operand-transmission pathway.
  • Circuit Variation: Similar tasks use similar but nonidentical circuits: the number-sequence task recovered 67.8% with the original circuit and 90.3% after adding MLP 7 and two attention heads.Circuit overlap therefore does not imply exact circuit identity.
  • Failure Modes: GPT-2 sometimes uses entirely different heads and MLPs for other mathematical operations, even when the greater-than circuit could solve the task.It also produces unusual outputs on tasks suggesting less-than or exact-answer operations.
  • Failure Modes: The circuit overgeneralizes by predicting numbers greater than YY when prompts require smaller numbers, such as reversed year-order or BC-duration prompts.These errors arise because GPT-2 activates the same greater-than circuit in inappropriate contexts.
  • Interpretation: The evidence does not resolve whether cross-context activation reflects memorization or generalized mathematical ability.The authors characterize the mechanism as intermediate between the two, while noting failures on less-than and equal-to.

6 Conclusion

The paper presents an interpretable GPT-2 circuit that activates across contexts, while emphasizing that its generality and interpretation remain limited and unresolved.

  • Conclusion: The study outlines a GPT-2 circuit with interpretable structure and semantics, bridging analyses of mathematical abilities in toy models and larger pre-trained language models.The circuit is coarser-grained than toy-model findings but finer-grained than prior work on mathematics in pre-trained LMs.
  • Conclusion: The circuit activates across contexts on other greater-than-adjacent tasks, but whether this reflects generalization or memorization remains open.This uncertainty applies broadly to interpreting cross-context circuit activation.
  • Limitations and Future Work: The conclusions are limited by the small model and dataset, the simple phenomenon studied, and the model-centric methodology.The authors propose diverse-task evaluation, data-driven interpretability, and studies of larger models as future work.
  • Implications: The study suggests that circuits may help investigate memorization and generalization, including through joint analysis of attention heads, MLPs, and neurons.This is presented as groundwork for future circuits research rather than a definitive account of mathematical competence.

A More Behavioral Study of Greater-Than in GPT-2

Additional behavioral tests show strong top predictions and a default greater-than tendency, while circuit analysis links even some incorrect behavior to the same circuit.

  • Behavioral Evaluation: Without the output prefix XX, GPT-2 is evaluated on whether its century or full-year continuation remains at least as large as the input year.The test uses the same dataset as earlier experiments with the final XX removed.
  • Behavioral Evaluation: 100% of GPT-2’s top-1 continuations and 98.6% of top-5 continuations were correct on the year-span dataset.These results concern predicted YY continuations in the behavioral evaluation.
  • Default Behavior: GPT-2 predicts greater-than even in contexts where less-than is appropriate, so behavioral success on greater-than does not establish reliable operation selection.This motivates testing scenarios where neither greater-than nor less-than is required.
  • Default Behavior: On random nonmonotone number sequences, GPT-2’s continuation behavior depends on sequence length, with values below YN increasing smoothly to 50% by N = 20.At low sequence lengths, it produces mostly numbers greater than YN.
  • Default Behavior: Circuit analysis finds that the greater-than circuit also underlies this default behavior, linking the tendency to the circuit’s flawed identification of greater-than situations.The authors leave open why the circuit activates in these incorrect scenarios.

B The Full Year-Span Prediction Circuit

The full year-span circuit is assembled by tracing how early MLPs and attention heads supply information to later components, especially the year token YY. The patched circuit reproduces most of the model’s greater-than behavior.

  • Patching queries and keys with bad data reduces performance by 15% relative to the partial circuit, showing they also matter.Attention patterns vary in intensity across YY, especially at YY=01.
  • Attention heads’ values at the YY position are the most important inputs to the attention heads’ outputs.Patching values with 01-input entirely disrupts circuit performance, unlike patching keys or queries.
  • The main upstream influences on attention-head values are MLPs 0-3 and attention heads a0.h5, a0.h3, and a0.h1.MLP 2 is an exception because it does not rely on MLP 1, while MLP 0 depends primarily on token embeddings.
  • The full circuit achieves a probability difference of 71.5% and cutoff sharpness of 10.5%.These values are 98.3% of the earlier probability-difference result and sharper than pre-patching.

C Circuit Finding, Step by Step

The circuit-finding procedure proceeds downstream from direct logit contributors, then traces their upstream dependencies and adds heads that identify the starting year YY.

  • Circuit Finding, Step by Step: Direct path patching identifies MLPs 8-11 and attention head a9.h1 as connections to the logits.The procedure then examines which nodes influence these downstream components.
  • MLP 10 Neuron Contributions: MLP 10 neuron contributions are visualized with blue for upweighting predicted years and red for downweighting them.The neurons are ordered by importance from left to right and top to bottom.
  • Circuit Finding, Step by Step: The analysis skips a9.h1 because attention heads and MLPs perform separate roles and head queries, keys, and values require separate analysis.This keeps the step-by-step search focused on MLP-mediated paths.
  • Circuit Finding, Step by Step: Tracing MLP 8 reveals the heads that identify YY, completing the initial circuit investigations.This is the final upstream step described in the circuit-finding section.

D MLP 10 Neuron Contributions

MLP 10’s neuron contributions combine into the greater-than pattern: later years receive increasing support relative to the starting year YY.

  • Many important neurons contribute patterns that are relatively constant across starting years YY or predicted years.For example, the fourth-ranked neuron always upweights later years, while the tenth downweights years near the end of the candidate range.
  • Combining the top-100 or top-200 of MLP 10’s 3072 neurons more strongly reproduces the MLP’s overall contribution pattern.The logit-lens magnitude consistently increases between YY and YY+1 for a given start year YY.

E Logit Lens vs. Direct Effects via Path Patching

The paper checks logit-lens interpretations against direct effects measured by path patching. The two approaches yield largely similar insights despite differences in magnitude.

  • The logit lens may be misleading because layer normalization introduces a nonlinear transformation before residual-stream values become logits.This is a methodological limitation of interpreting intermediate outputs through the unembedding matrix alone.
  • Direct effects compare original year logits with logits after patching a component’s direct path using the 01-dataset.Their difference estimates the component’s direct effect on the logits.
  • Patching the top-3 MLP 10 neurons and summing their effects produces results essentially identical to logit-lens results, differing mainly in magnitude.The same comparison is also performed for the top-10 neurons as a group.
  • All logit-lens results, including those for entire MLPs and attention heads, can be reproduced using direct effects.This supports the conclusion that concerns about the logit lens are not dire for the reported insights.

F Year-Span Circuit Generalization

GPT-2’s greater-than circuit generalizes successfully across several year and price contexts, but fails on some sequences and comparisons. Successful transfer preserves the circuit’s structure, while less-than tasks show that the same greater-than mechanism can be activated incorrectly.

  • Tasks Failed: GPT-2 fails on tested sequence and explicit less-than prompts, producing roughly uniform predictions, often YY+1, or always outputting YY.For the sequence with increments of two, the correct increase must vary to avoid the single-token number 1700.
  • Tasks Completed Correctly: GPT-2 successfully completes year-span, price-range, and numerical-sequence tasks, although the price task has a 75% probability difference versus 90% for the others.The first two tasks recover more than 90% of performance through circuit evaluation; the sequence task initially does not.
  • Tasks Completed Correctly: Accounting for MLP 8’s dependence on MLP 7 and two additional attention heads restores more than 90% loss recovery on the numerical-sequence task.The extra heads are a7.h11 and a6.h1.
  • Tasks Completed Incorrectly: The circuit also activates for reversed year-order and BC-duration prompts, with both tasks completed successfully, though the BC task is completed less well.For these less-than tasks, impeding circuit components improves performance because the circuit performs greater-than.
  • Tasks Completed Incorrectly: The less-than tasks achieve performance recoveries greater than 90% when the greater-than circuit is patched or impeded according to the task direction.The same iterative path-patching plots resemble those for year-span prediction.
  • Method and Scope: The experiments use path patching to test whether the circuit’s components influence task behavior, with task design requiring clearly delimited answers and consistent model behavior.The approach is intended for tasks solvable in one forward pass; smaller, simpler tasks are easier to interpret precisely.
Loading 2305.00586v5…