PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs

Sharath Naganna, Tanvir Ahmed Sijan, Uddipta Kalita

arXiv:2607.16693v1cs.CL

TL;DR

The paper investigates whether arithmetic heuristic neurons are shared across symbolic arithmetic, word problems, and Python code, and whether cross-format failures reflect distinct circuits or differing activation states. Using causal neuron identification, intervention, and cross-format activation transfer, it finds a compact shared circuit that is necessary and sufficient for late-layer arithmetic computation. Shared-activation transfer recovers most failures, exceeding 97% for addition and subtraction, supporting largely form-invariant arithmetic computation at the neuron level.

  • Problem

    It remains unclear whether equivalent arithmetic problems in different formats recruit the same neurons or distinct arithmetic circuits, despite strong formulation sensitivity in LLMs.

  • Method

    The study identifies arithmetic heuristic neurons separately across three formats, intersects them, tests necessity and sufficiency with interventions, and transfers shared activations between successful and failed executions.

  • Results

    A compact shared neuron set is necessary and sufficient for late-layer arithmetic computation, while cross-format activation transfer exceeds 97% recovery for addition and subtraction.

  • Takeaways & Limitations

    Arithmetic computation is largely form-invariant at the neuron level, with shared neurons retaining consistent heuristic-family roles across representations.

  • Takeaways & Limitations

    The study covers elementary integer arithmetic in base Llama models and three controlled formats, leaving broader form invariance unclear.

Abstract

from arXiv · show

Large language models often succeed on one formulation of a problem while failing on an equivalent formulation. Whether these failures arise from distinct internal circuits or different activation states of a shared circuit remains unknown. Recent mechanistic interpretability studies suggest that arithmetic in LLMs emerges from a "bag of heuristics," encoded by a sparse set of MLP neurons that represent distinct arithmetic strategies. We investigate whether arithmetic heuristic neurons are form-invariant across symbolic arithmetic, natural language word problems, and Python code in three Llama-3 models. In each format, we identify arithmetic heuristic neurons using a two-stage pipeline combining attribution patching and activation patching. A compact set of neurons is shared across all three formats, and targeted interventions show this shared circuit is both necessary and sufficient for late-layer arithmetic computation. Transferring the shared neurons' activations from a successful execution in one format to a failed execution in another recovers most incorrect predictions, exceeding 97% for addition and subtraction, indicating that cross-format failures arise from activation states rather than distinct circuits. Moreover, shared neurons consistently belong to the same heuristic families across formats, demonstrating that arithmetic computation in LLMs is largely form-invariant at the neuron level.

1 Introduction

This paper asks whether arithmetic neurons generalize across symbolic equations, word problems, and code, motivated by strong formulation sensitivity and limited prior analysis beyond canonical expressions.

  • Experimental framing: Experiments use three equivalent prompt formats—symbolic arithmetic, Python code, and natural-language word problems—with answers generated as a single final-position token.The paper also provides code and data for reproducing the experiments.
  • Motivation: Prior work identifies a sparse “bag of heuristics” in which distinct neurons encode arithmetic strategies, but mainly studies canonical symbolic expressions.This leaves cross-format generalization of arithmetic neurons unresolved.
  • Motivation: LLMs can suffer substantial performance degradation when mathematically equivalent problems are paraphrased or reformulated, while performance also varies across instantiations.These behavioral sensitivities motivate examining the underlying neuron-level mechanism.
  • Research questions: The central question is whether symbolic equations, word problems, and code recruit the same arithmetic heuristic neurons.The paper frames this as RQ1 on form invariance.
  • Research questions: A second question asks whether cross-format failures reflect distinct arithmetic circuits or different activation states within a shared circuit.This is the paper’s RQ2.

2 Related Work

Mechanistic interpretability studies analyze transformer circuits using causal mediation methods, including MLP neurons and attention heads, while recent arithmetic work motivates testing whether heuristic neurons transfer across formats.

  • Mechanistic interpretability: Mechanistic interpretability reverse-engineers language-model computations through learned weights and interacting components such as MLP neurons and attention heads.These components are treated as circuits implementing specific functions.
  • Mechanistic interpretability: Causal mediation methods used to localize model behavior include activation patching, path patching, attribution patching, and logit attribution.These methods connect internal components to model behavior.
  • Arithmetic reasoning: Arithmetic circuit studies identify causal components and suggest that arithmetic emerges from collections of heuristic neurons rather than one coherent algorithm.Alternative explanations include symbolic pattern matching and dual-pathway computation.
  • Research gap and approach: This paper extends that line of work by systematically identifying and causally validating whether equivalent computations recruit the same neurons across symbolic arithmetic, word problems, and code.Its pipeline also isolates shared neurons through layer-wise intersection and tests them with interventions and activation transfer.

3 Background

The background presents transformer MLPs as key-value memories whose neurons write value vectors when inputs match learned key patterns, and describes arithmetic as a sparse cooperation of heuristic neurons.

  • MLPs as key-value memories: Transformer layers combine attention, which routes information across tokens, with MLPs, which perform nonlinear transformations independently at each token position.The paper analyzes arithmetic largely through the MLP component.
  • MLPs as key-value memories: An MLP maps input representation x to output y through up- and down-projection matrices and a nonlinear activation, then decomposes into individual neurons.The neuron decomposition supports component-level causal analysis.
  • MLPs as key-value memories: Each neuron has a key vector that detects a learned input pattern and a value vector that determines information written to the residual stream.The MLP output is a weighted sum of the value vectors of activated neurons.
  • Arithmetic neurons and heuristics: Prior work finds a sparse subset of arithmetic neurons whose ablation sharply degrades arithmetic performance while comparatively preserving many non-arithmetic capabilities.These neurons are concentrated mainly in middle and later transformer layers.
  • Arithmetic neurons and heuristics: Arithmetic neurons implement diverse heuristics over operands or expected results, including range, modulo, digit-pattern, identical-operand, and multi-result strategies.The resulting computation is described as a “bag of heuristics” in which specialized neurons cooperate on the final prediction.

4 Experimental Setup

Experiments compare arithmetic circuits across symbolic arithmetic, Python code, and word problems in three Llama-3 models, using final-token, late-layer analyses. A two-stage neuron-ranking pipeline and causal interventions test circuit faithfulness and cross-format activation transfer.

  • Prompt Formats: Three Llama-3 models are evaluated on semantically equivalent arithmetic in symbolic, Python-code, and natural-language formats, with answers generated as single final-position tokens.The formats cover addition, subtraction, multiplication, and division using Python integer-floor semantics.
  • Dataset Construction: Division results require caution because valid single-token integer quotients have limited answer diversity, with unique answers saturating at approximately 96, 163, and 164 across the three formats.The evaluation uses correctly answered prompts, split into training and evaluation sets for estimating importance and conducting interventions.
  • Computation Onset: Linear probes identify a consistent late-layer computation onset across formats, motivating analyses at the final token and layers ℓ≥16.The answer first becomes reliably linearly decodable at the final input token for Llama 3 8B, with pooled probes recovering a common answer direction within each format.
  • Arithmetic-Neuron Identification: Attribution patching first selects the top 2000 candidate neurons per layer, after which exact activation patching retains the top 200 neurons per layer.Importance scores are averaged across training prompts and ranked by mean plus one standard deviation before exact reranking.
  • Circuit Faithfulness Evaluation: The faithfulness evaluation compares the unmodified model, complete mean ablation, and ablation of only non-circuit MLP neurons to quantify the correct-answer signal preserved by the circuit.Attention heads and MLP neurons before the computation-onset region remain unmodified, while outside-circuit neurons are replaced with aligned mean-cache activations.
  • Cross-Format Transfer: Cross-format activation patching replaces shared-neuron activations in failed target executions with activations from successful donor executions and compares matched, mismatched, and random interventions.The analysis reports corrected target predictions and the normalized logit of the correct answer across operator and donor-to-target format pairs.

5 Results

Across symbolic arithmetic, code, and word problems, a compact shared neuron circuit supports arithmetic computation, with causal and activation-transfer results indicating form-invariant mechanisms.

  • 5.1 Sparsity: The three-way neuron overlap peaks with approximately 50–100 neurons per layer, then decreases as lower-ranked, increasingly format-specific neurons are added.This non-monotonic pattern holds across addition, subtraction, multiplication, and division.
  • 5.2 Causality: The shared neurons are causally necessary and sufficient for late-layer arithmetic computation across symbolic arithmetic, code, and word problems.Keeping only the shared circuit recovers performance, whereas ablating it rapidly degrades faithfulness; attention and earlier layers remain intact.
  • 5.3 Cross-format activation transfer: Activation transfer recovers over 97% of addition and subtraction failures at approximately k = 50 neurons per layer, while multiplication recovers 80–85%.Division reaches approximately 90% for word→arithmetic transfer but is more variable for code→arithmetic transfer.
  • 5.3 Cross-format activation transfer: Shared-neuron activation restoration recovers most failed predictions, indicating that cross-format failures primarily reflect activation states rather than distinct arithmetic circuits.Random-neuron activation patching yields near-zero recovery across operators and transfer directions.
  • 5.4 Mechanistic consistency: Shared neurons receive consistent heuristic-family assignments across formats, with average pairwise and three-way Jaccard similarities of 0.75 and 0.64.Exact agreement averages 30.6%, while 94–100% of shared neurons receive a heuristic label in every format for addition, subtraction, and multiplication.

6 Conclusion

The study finds that arithmetic computation across symbolic, word-problem, and code formats relies on a compact shared neuron circuit. Interventions and activation transfer indicate that cross-format failures mainly reflect activation states rather than distinct circuits.

  • The shared arithmetic circuit is both necessary and sufficient for computation across formats.Keep-only interventions recover nearly all arithmetic performance, while knockout interventions test the circuit’s necessity.
  • Cross-format failures mainly arise from different activation states of a shared circuit, not distinct arithmetic circuits.The shared neurons also preserve consistent functional roles across representations.
  • Transferring shared-neuron activations from successful executions recovers most failed cross-format predictions.The transfer experiments directly test whether activation states, rather than separate circuits, explain format-specific failures.

Limitations

The study’s conclusions are bounded by its focus on elementary integer arithmetic, base Llama models, controlled formats, final-token MLP analyses, and prompts the models already solved correctly.

  • The evidence is limited to elementary integer arithmetic in base Llama models and three carefully controlled input formats.Whether form invariance extends to complex reasoning, instruction-tuned models, or broader language tasks remains unclear.
  • The causal analysis examines only MLP neurons at the final input token, leaving cross-format invariance of attention-based routing unresolved.Prior work suggests attention heads route operand information to this position, but that mechanism is not analyzed here.
  • Shared neurons are identified from prompts answered correctly by the model, so broader reasoning failures and unconstrained prompts remain outside the analysis.The transfer experiments cover cases where the shared circuit succeeds in one format but fails in another.

A Additional Results on Llama Models

Experiments on Llama-3.2-3B and Llama-3.2-1B reproduce the main model’s qualitative findings, including shared late-layer computation, necessary and sufficient circuits, and successful activation transfer.

  • Both additional models show the same qualitative trends as Llama-3-8B across the analyses.These results support robustness across the three investigated model scales.
  • Answer representations become linearly decodable at a consistent late layer across symbolic arithmetic, code, and word problems.The matching onset pattern is reported for both Llama-3.2 models.
  • Three-way circuit overlap in both additional models follows the same non-monotonic pattern observed for Llama-3-8B.The overlap peaks at relatively small circuit sizes.
  • The shared circuit remains both necessary and sufficient for arithmetic computation in the two additional models.Figures 13–15 present the corresponding keep-only and knockout interventions.
  • Transferring shared heuristic-neuron activations substantially improves accuracy on failed target executions in both additional models.Figures 16 and 17 report representative cross-format transfer experiments.

B Dataset Statistics and Linear Probe Evaluation

The appendix reports dataset constraints and probe results for the additional models, with identical computation-onset layers across formats and smaller valid pools for multiplication.

  • At least 20,000 correct executions are available for addition, subtraction, and division, while multiplication is the primary dataset bottleneck.Multiplication is constrained by the requirement that answers remain at most 999.
  • The computation onset layer is strictly identical across symbolic, code, and word formats within each model.Table 3 measures the first layer where pooled-operator probe accuracy exceeds 50%.

C Additional Heuristic Consistency Results

Heuristic consistency remains high across three Llama-3 models and arithmetic operators, though exact family agreement is stricter and division is weaker.

  • Pairwise and three-way heuristic agreement remain high across all arithmetic operators and the three evaluated models.Exact family agreement is more stringent than broader agreement measures.
  • Division shows lower heuristic agreement, likely because valid integer quotients are fewer and less diverse.

D Distribution of Heuristic Families

The analysis refines broad heuristic categories into functional families defined by scoring mechanisms and targets, then tests whether shared-neuron agreement exceeds chance. Observed family consistency is significantly above randomized expectations across models and operators.

  • Distribution of Heuristic Families: Heuristic families intersect four scoring mechanisms with targets on one operand, both operands, or the expected result.The scoring mechanisms are range, modulo, pattern, and exact value.
  • Distribution of Heuristic Families: Families include operand range, result range, both modulo, result value, and identical operands, with one neuron potentially assigned multiple families.The result-value family activates for exact, zero-padded target numbers.
  • Distribution of Heuristic Families: The family distributions are summarized for shared neurons in Llama-3-8B, Llama-3.2-3B, and Llama-3.2-1B addition circuits.
  • Additional Heuristic Consistency Results: Observed exact family matches and pairwise Jaccard similarities significantly exceed randomized null expectations across all models and operators.Heuristic labels were randomly permuted independently within each format, with observed agreement exceeding the null maximum at p < 10−4.
  • Additional Heuristic Consistency Results: Table 7 reports shared-circuit size, labelled-neuron counts, three-way common assignments, exact matches, and pairwise or three-way family Jaccard similarities.
Loading 2607.16693v1…