PaperlayerView exact revision on arXiv

Source-linked AI summary

Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics

Yaniv Nikankin, Anja Reusch, Aaron Mueller, Yonatan Belinkov

arXiv:2410.21272v2cs.CL

TL;DR

The paper asks whether LLMs solve arithmetic with reusable algorithms or memorize training examples. Using causal analysis of arithmetic circuits and their neurons, it identifies a bag of heuristics as the mechanism behind arithmetic answers. The authors report that this mechanism is also the main source of arithmetic performance early in training.

  • Problem

    The paper asks whether LLMs implement robust reusable algorithms for arithmetic or memorize prompts from their training data, a distinction linked to generalization and scaling beyond limited problem sets.

    Source: §1 · p. 1

  • Method

    The authors use causal analysis to identify arithmetic circuits and examine individual neurons, classifying their input-pattern rules as heuristic types.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    The authors find that sparse MLP neurons act as memorized heuristics whose combination supports arithmetic answers, and that this mechanism explains most model behavior even in early training.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The authors conclude that arithmetic performance lies between robust algorithms and exhaustive memorization, relying instead on a combination of simpler rules and patterns.

    Source: §7 · p. 10

    The heuristic-type abstractions rely on human-identifiable definitions, and the analysis focuses on LLMs whose tokens can contain multiple digits.

    Source: §8 · p. 10

Abstract

from arXiv · show

Do large language models (LLMs) solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data? To investigate this question, we use arithmetic reasoning as a representative task. Using causal analysis, we identify a subset of the model (a circuit) that explains most of the model's behavior for basic arithmetic logic and examine its functionality. By zooming in on the level of individual circuit neurons, we discover a sparse set of important neurons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons is the mechanism used to produce correct arithmetic answers. To test this, we categorize each neuron into several heuristic types-such as neurons that activate when an operand falls within a certain range-and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a "bag of heuristics".

1 INTRODUCTION

The introduction asks whether arithmetic performance reflects reusable algorithms or memorized training examples, and uses arithmetic to investigate that distinction. It proposes that sparse neuron-level heuristics, rather than a single algorithm or exhaustive memorization, explain arithmetic answers and are present throughout training.

1 INTRODUCTION

  • The authors frame arithmetic as a test of whether LLMs use robust generalizable algorithms or memorize prompts from their training data.They note that memorization may suffice for limited problem sets, whereas algorithmic comprehension allows generalization and efficient scaling to new problems.Source: §1 · p. 1
  • The paper uses causal analysis of arithmetic circuits and individual neurons to examine how models generate correct answers.It builds on circuit-localization work that had not yet explained the circuit’s mechanism, and on Fourier-feature findings for fine-tuned addition models that the authors describe as only part of a more complex mechanism.Source: §1 · p. 1
  • Sparse circuit neurons implement distinct input-pattern heuristics whose combination supports correct arithmetic answers, rather than a single cohesive algorithm or memorization of every input-output pair.Some heuristics depend on operand patterns and boost logits for corresponding result tokens.Source: §1 · p. 2
  • The bag-of-heuristics mechanism explains most model behavior even in early training, with heuristics gradually converging toward those in the final checkpoint.The introduction reports that this mechanism is the main source of arithmetic-prompt performance from early stages.Source: §1 · p. 2
  • The paper presents its neuron-level account as a way to understand the sources and limitations of arithmetic capabilities, with possible relevance to other reasoning tasks.The authors also investigate when and why the mechanism fails to generalize.Source: §1 · p. 2

2 ARITHMETIC CIRCUIT DISCOVERY

The section identifies and evaluates an arithmetic circuit, a minimal subset of interconnected model components, then probes where correct answers become extractable. In Llama3-8B, the circuit captures most arithmetic performance, with later MLPs implicated in writing answers at the final position.

2.1 CIRCUIT DISCOVERY AND EVALUATION

  • Activation patching replaces a component’s activation with one from a counterfactual prompt, measuring changes in the probabilities of the original and counterfactual answers.Prompts use two operands and one of four basic operators; the evaluation set matches the discovery set in size, with 100 prompts per operator.Source: §2.1 · p. 3
  • Patching indicates MLPs affect output probabilities more than attention heads; the first MLP affects operand and operator positions, while middle and late MLPs influence the final position.Few attention heads are important, and those identified copy information from an operand or operator position to the final position.Source: §2.1 · p. 3, Figure 2 · p. 3
  • The Llama3-8B arithmetic circuit has 0.96 average faithfulness across four operators, accounting for 96% of the model’s arithmetic performance.Faithfulness measures the circuit’s contribution relative to the full model and a fully mean-ablated model.Source: §2.1 · p. 4

2.2 IDENTIFYING ANSWER-PROMOTING COMPONENTS

  • Linear probes extract the correct answer with high accuracy from the final position starting at layer 16, implicating MLPs in layers 16–32 in writing the answer representation.The answer is not extractable with high accuracy at earlier positions or layers.Source: §2.2 · p. 4, Figure 3 · p. 4

3 MLP NEURONS IMPLEMENT ARITHMETIC HEURISTICS

Causal analysis identifies a sparse set of circuit neurons sufficient for accurate arithmetic, with important neurons differing across operators. These neurons activate for numerical patterns and contribute either answer tokens directly or features for downstream processing.

3.1 DECOMPOSING CIRCUIT MLPS TO INDIVIDUAL NEURONS

  • Only 200 neurons (roughly 1.5%) per layer are needed for high faithfulness and correct arithmetic prompts after non-circuit components and lower-effect neurons are mean-ablated.The most important neurons are selected separately for each operator.Source: Figure 4 · p. 5, §3.1 · p. 5
  • Roughly 45% of important neurons for each operator are unique, motivating analysis of four separate operator circuits.Source: §3.1 · p. 5

3.2 MLP NEURONS ACT AS MEMORIZED HEURISTICS

  • The analysis treats MLP neurons as key-value memories: numerical input patterns trigger activations, and associated value vectors encode plausible answer tokens.Source: §3.2 · p. 5
  • Some neurons show human-identifiable activation patterns; one example activates for subtraction prompts whose results fall between 150 and 180.The Figure 5 heatmaps encode neuron activation strength for operand pairs under addition prompts.Source: §3.2 · p. 5, Figure 5 · p. 6
  • Direct heuristics activate based on both operands and encode expected results, while indirect heuristics depend on one operand and often encode features for downstream processing.Source: §3.2 · p. 6

4 ARITHMETIC PROMPTS ARE ANSWERED WITH A BAG OF HEURISTICS

The paper argues that arithmetic answers arise from combinations of prompt-matched heuristic neurons, whose small logit contributions can jointly promote correct answers. Targeted ablations support this account, while analyses of failures point to weak promotion of correct-answer logits rather than too few associated neurons.

4 ARITHMETIC PROMPTS ARE ANSWERED WITH A BAG OF HEURISTICS

  • Each heuristic neuron matches an input pattern, and some increase logits for relevant result tokens; several such neurons can jointly promote the correct answer.For example, the prompt “226 −68 =” matches several heuristic neurons that slightly increase the logit of result token 158.Source: §4 · p. 6

4.1 CLASSIFYING NEURONS TO HEURISTIC TYPES

  • The neuron-matching procedure compares activated prompts with prompts meeting a heuristic’s conditions, classifying a neuron when their intersection exceeds threshold t = 0.6.A neuron can be assigned to multiple heuristic types, and matching is conducted separately for each arithmetic operator.Source: §4.1 · p. 6
  • The procedure classifies 91% of the 3,200 top neurons for each operator as arithmetic heuristics; the remaining neurons show patterns that are not clearly identifiable.Manual inspection found no clear match to the defined heuristics for those remaining neurons.Source: §4.1 · pp. 6–7

4.2 HEURISTIC TYPES ARE COMBINED TO ANSWER ARITHMETIC PROMPTS

  • Across most heuristic types, ablating their neurons reduces accuracy more on associated prompts than on unassociated prompts, supporting their targeted functional role.The comparison uses 100 correctly completed prompts in each group.Source: §4.2 · p. 7, Figure 7 · p. 7
  • Ablating one heuristic type causes an average 29% accuracy drop from 95% pre-ablation accuracy, with low recall and other heuristics limiting the drop.The second ablation tests the combined contribution of the several heuristic types associated with each prompt.Source: §4.2 · p. 7
  • Ablating neurons from heuristics associated with a prompt reduces accuracy more than ablating the same number from unassociated heuristics, supporting their combined role in correct completion.Each heuristic may slightly boost the correct-answer logit; together they can produce a correct answer with high probability.Source: §4.2 · p. 7, Figure 8 · p. 8

4.3 FAILURE MODES OF THE BAG OF HEURISTICS

  • For Llama3-8B, failures are associated with weaker total logit contribution to the correct answer, not a shortage of associated heuristic neurons.Incorrect prompts had more associated heuristic neurons on average, while correct prompts had a slight advantage in their total correct-answer logit contribution.Source: §4.3 · p. 8, Figure 9 · p. 8

5 TRACKING HEURISTICS DEVELOPMENT ACROSS TRAINING STEPS

In Pythia-6.9B, heuristic neurons remain causally important from the earliest checkpoint examined, while the final checkpoint’s heuristics emerge gradually over training. Those final heuristics account for most heuristic contribution at intermediate checkpoints.

5 TRACKING HEURISTICS DEVELOPMENT ACROSS TRAINING STEPS

  • The model develops its final heuristic mechanism gradually, with final-checkpoint heuristic neurons appearing from an early checkpoint onward.The analysis begins at 23K steps, the earliest checkpoint with good arithmetic performance.Source: §5 · p. 8, Figure 10 · p. 9
  • Final-checkpoint heuristics explain an average of 79% of total heuristic contribution to accuracy at intermediate checkpoints, while other heuristics become less important.The analysis compares faithfulness using only neurons mutual with the final checkpoint against faithfulness using all heuristic neurons.Source: §5 · p. 9, Figure 10 · p. 9
  • Targeted ablation of prompt-associated heuristic neurons substantially reduces accuracy even at early checkpoints, and ablating 25 neurons per layer brings accuracy near zero across training stages.The targeted ablations have a larger effect than removing a similar number of randomly chosen important neurons.Source: §5 · p. 9, Figure 10 · p. 9

6 RELATED WORK

Prior work has identified arithmetic circuits and specialized mathematical algorithms, but how general-purpose LLM circuits combine operand information remains incompletely explained. This paper situates its neuron-level analysis at the intersection of arithmetic interpretability and the study of memorization versus generalization.

Memorization and generalization in LLMs

  • Research on memorization and generalization has often studied model behavior rather than internal mechanisms, with some work examining circuits or activations.The paper places its arithmetic analysis among studies of internal mechanisms and reports that heuristic sets are used even for arithmetic.Source: §6 · p. 10

Arithmetic reasoning interpretability

  • Earlier studies mapped arithmetic circuits and information flow, but did not fully explain how those circuits combine operand information to produce correct answers.Stolfo et al. (2023) and Zhang et al. (2024) characterize circuit structure; Zhou et al. (2024) report Fourier-space features for addition.Source: §6 · p. 10
  • Algorithmic mechanisms have been demonstrated in specialized toy models, but whether those findings extend to larger general-purpose LMs or other operators remains unclear.The cited examples concern modular addition and binary arithmetic.Source: §6 · p. 10

7 CONCLUSIONS

The paper concludes that LLM arithmetic relies on a combination of memorized heuristics rather than a robust algorithm or simple memorization. These heuristics emerge gradually during training, and the findings suggest mathematical improvement may require changes to training or architecture.

7 CONCLUSIONS

  • LLMs perform arithmetic with a bag of memorized heuristics, implemented by sparse neurons that activate for input patterns and together support correct answers.The mechanism evolves gradually during training rather than appearing abruptly or replacing another mechanism.Source: §7 · p. 10
  • The authors suggest improving mathematical abilities may require fundamental changes to training and architectures rather than post-hoc activation steering.They identify whether regularization can improve the mechanism as an open question for future research.Source: §7 · p. 10

8 LIMITATIONS AND DISCUSSION

The analysis depends on human-defined heuristic categories and covers models whose tokens can contain multiple digits; single-digit tokenization could yield different conclusions.

8 LIMITATIONS AND DISCUSSION

  • Heuristic categories rely on human-identifiable abstractions, leaving open whether they can be identified without human bias.The authors suggest developing a less human-biased method for identifying these abstractions.Source: §8 · p. 10
  • The analysis focuses on models with tokens containing multiple digits, so single-digit-tokenized models might produce different conclusions.The authors note that human arithmetic algorithms rely on separating larger numbers into single digits.Source: §8 · p. 10

A LLAMA3-8B CIRCUIT EVALUATION RESULTS

In Llama3-8B, a small number of attention heads preserves high arithmetic-circuit faithfulness for addition, subtraction, and division, while multiplication requires more heads. The appendix also introduces analysis of MLP0 and high-effect attention heads.

A LLAMA3-8B CIRCUIT EVALUATION RESULTS

  • 6 attention heads suffice for high circuit faithfulness on addition, subtraction, and division (97% on average), while multiplication requires 20 heads to exceed 90%.Faithfulness is assessed separately by operator; the full circuit's Pareto-optimal head counts and faithfulness are documented in Table 1.Source: A LLAMA3-8B CIRCUIT EVALUATION RESULTS · p. 14, Figure 11 · p. 14

B LLAMA3-8B CIRCUIT ADDITIONAL COMPONENTS

  • The additional-component analysis examines Llama3-8B's first MLP layer, MLP0, and its high-effect attention heads.The passage introduces these components without reporting their results.Source: B LLAMA3-8B CIRCUIT ADDITIONAL… · p. 14

B.1 MLP0

The appendix examines sparse MLP0 contributions, attention-head roles, operator-specific neuron overlap, heuristic categories, and the procedures and settings used in these analyses.

B.1 MLP0

  • As little as 1% of MLP0 neurons, alongside the top 1% in middle and late layers, is sufficient for high circuit faithfulness.Few MLP0 neurons have high intervention effects, similar to middle- and late-layer MLPs.Source: §B.1 · p. 14, Figure 12 · p. 15

B.2 ATTENTION HEAD PATTERNS

  • Three general arithmetic heads copy the first operand, operator, and second operand representations to the final position for further processing by heuristic MLP neurons.L16H21 attends to the first operand, L2H2 to the operator, and L15H13 to the second operand; other heads are operator-specific.Source: §B.2 · p. 15, Figure 14 · p. 16

C MULTI-LAYER PERCEPTRON IMPLEMENTATION DETAILS

  • Pythia-6.9B and GPT-J use two-matrix-multiplication MLPs, whereas Llama3-8B and Llama3-70B use gated MLPs; the key-value view treats neuron activation as a key and its output weights as a value.The gated form adds a parameter matrix and uses a Hadamard product.Source: C MULTI-LAYER PERCEPTRON… · pp. 16–17

D NEURON INTERSECTION BETWEEN OPERATORS

E HEURISTIC TYPES DESCRIPTIONS

  • The heuristic taxonomy includes range, modulo, pattern, identical-operands, and division-only multi-result heuristics, with types matched using parameterized definitions.Evenly subsampling parameters can make classifications approximate; for example, an actual range [105, 205] may be classified as [100, 200].Source: E HEURISTIC TYPES DESCRIPTIONS · pp. 17–18

F HEURISTIC NEURON CLASSIFICATION ALGORITHM

  • Neuron classification compares each neuron's top-k contributed prompts with prompts associated with a heuristic and assigns a match when normalized overlap exceeds the threshold.The threshold described is 0.6, and a neuron may match multiple heuristic types.Source: F HEURISTIC NEURON CLASSIFICATION… · pp. 18–19

G ADDITIONAL IMPLEMENTATION DETAILS

  • Arithmetic prompts use operands from 0 to 300, with prompts excluded when results are negative or exceed the model's single-token limit; circuit evaluation uses mean ablation of non-circuit components.The classification procedure uses last-position activations and a 0.6 score threshold.Source: G ADDITIONAL IMPLEMENTATION DETAILS · p. 19

H MODEL ACCURACIES ON ARITHMETIC PROMPTS

  • Accuracy is evaluated on single-token results with operands in [0, 300]; smaller models' high division accuracy is attributed to integer division's non-uniform answers, with half of legal prompts yielding 0.The passage gives the evaluation scope but no model accuracy values.Source: H MODEL ACCURACIES ON ARITHMETIC PROMPTS · p. 20, Table 3 · p. 20

I RESULTS ON ADDITIONAL MODELS

Across Llama3-70B, Pythia-6.9B, and GPT-J, the analyses largely reproduce the findings reported for Llama3-8B. In these models, sparse heuristic neurons in middle- and late-layer MLPs contribute to arithmetic outputs.

I RESULTS ON ADDITIONAL MODELS

  • Results across the three additional models are similar to Llama3-8B, with Llama3-70B showing the greatest similarity and a more pronounced knockout effect.The authors hypothesize that its larger size and modern training methodology may facilitate more sophisticated heuristics.Source: I RESULTS ON ADDITIONAL MODELS · p. 20
  • Ablating prompt-associated heuristic neurons reduces accuracy more than ablating an equally sized random neuron set.This pattern was observed in Llama3-70B, Pythia-6.9B, and GPT-J.Source: I RESULTS ON ADDITIONAL MODELS · p. 20
  • The arithmetic circuit includes sparse attention heads that project operand and operator information, along with all MLP layers that process operands and then their combined information.Early-layer MLPs process operand and operator positions; middle- and late-layer MLPs process the combined information at the final position.Source: I RESULTS ON ADDITIONAL MODELS · p. 20
  • Correct answers are extractable with high accuracy only at the final position, after processing by middle-layer MLPs.Source: I RESULTS ON ADDITIONAL MODELS · p. 20

J ADDITIONAL EXAMPLES FOR ARITHMETIC HEURISTICS

The supplementary material presents sampled heuristic-neuron examples for each arithmetic operator and examples of causally significant neurons that did not fit the defined heuristic types.

J ADDITIONAL EXAMPLES FOR ARITHMETIC HEURISTICS

Related papers

Loading 2410.21272v2…