PaperlayerView exact revision on arXiv

Source-linked AI summary

Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B

Rahul Chowdhury, Timothy A Rupprecht, Senhao Cao, Jiahao Liu, Octavia Camps, David Bau, Pu Zhao, Yanzhi Wang

arXiv:2608.18419v1cs.LGcs.AI

TL;DR

It remains unclear whether LLMs extrapolate numerical sequences by memorizing patterns or by inferring underlying structure. Studying LLaMA 3.1–8B with probing and activation patching, the paper finds evidence for localized first-difference computation, retrieval, and arithmetic extrapolation.

  • Problem

    It remains unclear whether LLMs extrapolate numerical sequences by memorizing patterns or by inferring underlying structure.

  • Method

    The paper tests LLaMA 3.1–8B on repeated-offset numerical sequences using linear probing and activation patching to analyze structural computations.

  • Results

    LLaMA computes and locally stores first differences, retrieves the relevant offset through induction, and adds it to the current value for extrapolation.

  • Takeaways & Limitations

    The study identifies an internal mechanism for induction over latent numerical structure in an LLM without explicit supervision.

  • Takeaways & Limitations

    The experiments use only LLaMA 3.1–8B, selected partly because its tokenizer represents integers from 0 to 999 as single tokens.

Abstract

from arXiv · show

Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they accomplish time-series prediction remain unclear. Specifically, whether they truly understand the underlying structure, which at a minimum requires reasoning over first differences in the sequence of numbers. To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view. Mechanistic interpretability is an emerging field concerned with the reverse engineering of the algorithms learned by neural networks such as LLMs. To assess Llamas' numerical sequence modeling capabilities and to facilitate our mechanistic interpretability analysis, we create a sequence modeling task that cannot be solved without picking up structural cues. Specifically, we sample n random numbers and repeat them with an offset. We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure. To understand the mechanisms that allow it to do so, we perform probing experiments and activation patching based counterfactual analysis. Probing reveals that the model computes and stores first differences in its internal representations without explicit supervision, indicating that it tracks structural information about the sequence. Activation patching reveals that Llama retrieves the relevant first-difference with a mechanism similar to an induction circuit and subsequently adds it to the current value. Notably, our work represents one of the first studies to identify this form of concept induction in LLMs.

1 Introduction

The introduction asks whether LLM sequence extrapolation reflects general structural reasoning rather than memorized pattern-specific rules. It presents a mechanistic study of LLaMA 3.1–8B showing that numerical extrapolation involves locally represented first differences and arithmetic operations over retrieved latent structure.

  • 1 Introduction: The study asks whether LLMs infer underlying numerical sequence structure generally or merely extrapolate with rules learned for familiar patterns.This question distinguishes abstract structural reasoning from memorization and motivates the investigation.
  • 1 Introduction: The authors introduce a non-trivial sequence-modeling task that cannot be solved through token-level copying via induction heads, and analyze LLaMA 3.1–8B with probing and activation patching.The task is designed to probe structural reasoning rather than simple copying.
  • 1 Introduction: First-difference representations are locally stored and linearly decodable across layers, indicating that the model internally represents structural information about numerical sequences.The introduction describes these representations as part of the study’s probing results.
  • 1 Introduction: The model identifies the repeated first-difference pattern, retrieves the delta following the final observed one, and adds that offset to the last number.The computations are described as precise, localized, and structurally grounded.
  • 1 Introduction: The work claims to be the first mechanistic interpretability study uncovering an LLM mechanism for induction over latent numerical-sequence structure.The mechanism performs extrapolation through arithmetic operations over internally retrieved representations.

2 Related Work

Prior work established that LLMs can complete numerical sequences and forecast time series, but left unresolved whether these abilities reflect memorization, structural understanding, or identifiable mechanisms. Existing studies also examined in-context learning and simple sequence-continuation circuits without testing inference of more abstract sequential structure in untrained tasks.

  • Prior sequence modeling and forecasting: LLMs have demonstrated sequence completion and strong time-series forecasting performance, but prior work did not determine whether this reflects memorization or temporal-structure understanding.Mirchandani et al. attributed sequence completion to in-context learning, while LLM-Time explored adapting pretrained LLMs for time-series forecasting.
  • Prior sequence modeling and forecasting: Toy-model studies of in-context learning did not test whether LLMs can infer sequential structure in tasks for which they received no explicit training.Akyürek et al. and Kantamneni et al. examined in-context learning using toy models.
  • Mechanistic interpretability: Mechanistic-interpretability work identified shared circuits for simple sequence continuation, but focused on familiar patterns such as short increasing number sequences.That work did not address whether models can reason over more abstract sequential structure.

3 Problem Setting

This section defines a numerical sequence task in which a random-walk segment repeats its first-difference pattern, requiring the model to uncover and track latent structure. Experiments use LLaMA 3.1–8B to predict the 30th number, achieving MAE 4.2748 and R^2 0.9958.

  • Dataset construction: The task uses a random-walk first segment and a second segment that reuses the same deltas in the same order, exposing structure only through differencing.The design tests whether the LLM can identify and track the latent delta pattern.
  • Model and evaluation: The study evaluates LLaMA 3.1–8B, whose tokenizer represents every integer from 0 to 999 as a single token, simplifying interpretability interventions.The model has 32 layers and 32 attention heads per layer.
  • Model and evaluation: The dataset contains 10,000 comma-separated integer sequences, and the model predicts the 30th number from the first 29 elements.Each instance ends with a comma, indicating that the next token should be a number.
  • Results: 4.2748 MAE and 0.9958 R^2 are achieved by LLaMA 3.1–8B on the evaluation dataset.The reported R^2 corresponds to approximately 99.58% of variance explained.

4 Is Numerical Information Represented Locally in the Model?

Probing experiments show that LLaMA 3.1-8B linearly encodes the first difference and final number in distinct token positions. The first difference becomes decodable after layer 7, while control probes support position-specific localization rather than general numerical information.

  • Layerwise decodability: Final-number representations gradually lose decodability across layers, contrasting with the comparatively stable encoding of first differences.Figure 3 tracks both mean absolute error and R^2 layer by layer.
  • Control conditions: Control probes perform substantially worse: shuffled-label controls and cross-position controls show that the observed decodability is position-specific rather than spurious or globally distributed.First-difference controls use position-57 activations, while final-number controls use position-27 activations; the former achieve negative R^2 and high MAE.
  • First-difference representation: After layer 7, first-difference decoding improves sharply, with R^2 stabilizing above 0.8 and performance remaining relatively stable across the first 16 layers.Although performance slightly decays with depth, first-difference representations remain relatively stable.
  • Interpretation: Together, the results indicate that the first difference and final number needed for arithmetic composition are linearly encoded in specific locations.This addresses whether numerical components are represented in structured, localized hidden states.

5 Is the Next Number Computed by Adding a Retrieved First Difference?

This section tests whether the model functionally computes the next number by retrieving a previously seen first difference and adding it to a new number. The analysis uses activation patching to transplant activations from a clean sequence into a corrupt one and evaluate the hypothesized retrieval position.

  • 5 Is the Next Number Computed by Adding a Retrieved First Difference?: The section asks whether localized representations of first differences and final numbers are functionally used for arithmetic composition.It specifically tests retrieval of a previously seen first difference from a particular token position.
  • 5 Is the Next Number Computed by Adding a Retrieved First Difference?: Activation patching transplants activations from a clean sequence into a corrupt one to test the hypothesized retrieval mechanism.The patching methodology is used to determine whether the model retrieves the delta following the latest one.
  • 5 Is the Next Number Computed by Adding a Retrieved First Difference?: The test evaluates whether the retrieved delta is added to a new number from an unseen sequence.This directly probes the proposed arithmetic operation on the retrieved first difference.

Clean vs. Corrupt Sequence

The study contrasts a structured clean sequence with a flattened corrupt control to test whether the model retrieves and applies a first difference. Activation patching then measures causal influence on the counterfactual prediction and identifies localized induction at the relevant sequence position.

  • Sequence design: The corrupt sequence consists entirely of 100s, making every first difference zero and removing intrinsic input variation.This flattened sequence serves as a control lacking the clean sequence’s algorithm, testing retrieval from an alternate source.
  • Activation patching: The experiment patches hidden states across layers while excluding the beginning token, final number token 57, and final token 58.Downstream-layer patching preserves transient causal signals, while the final token’s query vector is patched from the clean pass across layers l+1 through L.
  • Evaluation: Causal influence is measured by the probability difference ΔP for the counterfactual label ycf = 100 + Δ* between patched and corrupt forward passes.Higher ΔP indicates a stronger causal contribution from the patched components to predicting the correct label.
  • Patching results: On 100 zero-MAE instances, ΔP increased significantly at token 27 from layer 14 onward, indicating causal interaction with the final token’s query from layer 15 onward.The result supports induction over first differences: locating and copying a locally stored delta, then adding it to the number at position 57 to generate the next token.

6 Can the Model Identify Functionally Critical Tokens for Delta Retrieval and Extrapolation?

This section tests whether the model identifies the repeating-pattern onset, aligns it with the final observed difference, and selects the correct delta for extrapolation. Patching isolates causal contributions and reveals that the final query attends to token 27 to retrieve the locally stored first difference.

  • 6 Can the Model Identify Functionally Critical Tokens for Delta Retrieval and Extrapolation?: The model must identify the repeating-pattern onset, align it with the final observed difference, and select the correct delta for extrapolation.These requirements go beyond storing and retrieving first differences for arithmetic composition.
  • 6 Can the Model Identify Functionally Critical Tokens for Delta Retrieval and Extrapolation?: Token 27 shows a sharp causal effect, indicating that the final query attends there to retrieve the first difference following the last observed one.The effect appears in probability-difference heatmaps across layers and token positions after activation patching.
  • 6 Can the Model Identify Functionally Critical Tokens for Delta Retrieval and Extrapolation?: Patching isolates each attention head’s causal contribution to successful extrapolation by routing value from appropriate positions.The method assesses which heads drive the model’s extrapolation behavior.

Patching Setup

The patching setup transfers a single attention head’s output from a clean repeating-difference sequence into a corrupt flat sequence at the final token. The analysis combines causal effects with value-weighted attention to identify functionally relevant tokens, highlighting positions involved in pattern alignment and delta extrapolation.

  • Patching Setup: The counterfactual setup patches one head’s output at final position 58 from the clean run into the corrupt run.The clean sequence contains a repeating first-difference pattern, whereas the corrupt sequence is flat at 100s; the counterfactual label is y_cf = 100 + Δ*.
  • Patching Setup: This single-head intervention isolates its causal contribution while preserving the rest of the corrupt context.Only the head output at the final token is substituted from the clean run.
  • Token Importance: The token-importance profile combines each head’s causal effect with value-weighted attention mass across token positions.Causal effect is measured by the change in predicted probability for the counterfactual label, while value-weighted attention measures how much value enters the residual stream.
  • Token Importance: Token importance weights causal influence with value-aware attention to measure token-level functional relevance to the model’s output.The metric integrates the two signals rather than relying on attention alone.
  • Causal Attention Results: Across 100 zero-MAE sequences, the five most relevant positions were 57, 33, 27, 55, and 25.These positions correspond to the final number, the onset of the repeating delta pattern, the delta added to the final value, and other critical pattern-identification or extrapolation points.
  • Causal Attention Results: The attention pattern supports first-difference alignment, with token 27 especially important for extrapolation and the final first difference serving as a retrieval anchor.The analysis also suggests phase alignment and positional encoding guide delta selection after the repeating structure is recognized.

7 Does the Model Prioritize Delta Induction Over Positional Cues?

The model’s delta retrieval depends partly on positional and phase-based cues, but redirecting those cues does not make it prefer the counterfactual delta over the ground-truth delta.

  • Intervention: Key swapping between the 27th and 25th tokens tests whether delta retrieval follows semantic identity or positional and phase-based alignment.The positions had high and comparable importance scores.
  • Intervention: The intervention redirects key-based positional cues while preserving value representations and their first-difference content across subsequent layers.This separates positional and phase-based cues from delta content.
  • Results: Over 100 zero-MAE instances, key swapping significantly increased ΔPcf and decreased ΔPgt, showing that phase- or position-based cues contribute to delta retrieval and composition.The counterfactual label is ycf = x29 + Δ25, whereas the ground-truth label is ygt = x29 + Δ27.
  • Results: Despite the intervention, Ppatched(ycf) did not overtake Ppatched(ygt), so positional redirection influenced behavior without reversing the model’s preference for the ground-truth delta.The absolute post-intervention probabilities capture the response magnitude as well as its direction.

8 Conclusion

The study shows that LLaMA performs unsupervised structural extrapolation in numerical sequences by recognizing trends, inferring position-dependent first-difference rules, and composing them for prediction. Probing indicates these behaviors reflect functional computations over structured internal representations, while the findings motivate extending the analysis to other reasoning tasks involving latent structure.

  • Conclusion: The LLM performs structural extrapolation in numerical sequences without explicit supervision or training for high-precision numerical tasks.It recognizes trends and generates accurate predictions over evolving input sequences.
  • Conclusion: The model infers position-dependent rules encoded as distinct first differences and systematically composes them to generate predictions.The model later retrieves these differences during prediction.
  • Conclusion: Probing confirms that the observed effects reflect functional computations over structured internal representations rather than incidental correlations.This supports the interpretation that the model computes and stores structural information for sequence prediction.
  • Conclusion: Future work could extend these insights to reasoning tasks in which latent structure plays a critical role.The paper identifies this as a direction motivated by its findings.

Appendix · A Illustrating Patching Experiments

The appendix illustrates patching interventions used to isolate attention-head functions and distinguish positional phase cues from delta-based representations. The experiments patch clean activations into corrupt sequences and swap keys while preserving values to test first-difference selection during extrapolation.

  • A Illustrating Patching Experiments: Head-level patching isolates the functional role of individual attention heads in pattern recognition and delta retrieval.The intervention patches one attention head’s output at the final token position from a clean forward pass into a corrupt one.
  • A Illustrating Patching Experiments: The clean sequence contains structured first-difference patterns, whereas the corrupt sequence is flat-valued.This contrast provides the clean and corrupt conditions for the patching intervention.
  • A Illustrating Patching Experiments: At each layer l, only one attention head’s final-token output is patched from the clean forward pass into the corrupt sequence.All other heads and [the passage text is truncated].
  • A Illustrating Patching Experiments: Key swapping disentangles reliance on phase, or positional, cues from delta-based representations during extrapolation.The intervention swaps keys from token positions 25 and 27, which are both identified as causally important.
  • A Illustrating Patching Experiments: Keys from token positions 25 and 27 are swapped from layer l onward while their respective value vectors are preserved.The setup uses orange and gray arrows to indicate the intervention.
  • A Illustrating Patching Experiments: The key-swapping setup tests whether the model selects the correct first difference for addition based on the [passage text is truncated].The figure describes this as a test of the basis for selecting the first difference during extrapolation.
Loading 2608.18419v1…