Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs

Yao Yan

Source-linked AI summary

Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs

Yao Yan

arXiv:2506.07824v2Jun 9, 2025cs.AI

TL;DR

The paper asks how LLaMA-3-8B-Instruct internally processes multi-digit addition and addresses this with layer-wise linear probing and logit-lens inspection. It finds an ordered trajectory from formula structure through computation and numerical abstraction to output organization, while limiting causal and task-general claims.

Problem
The paper investigates whether LLaMA-3-8B-Instruct performs multi-digit addition through structured algorithmic computation or non-algorithmic pattern extraction.
Method
The study combines linear probes and logit-lens inspection of layer-wise hidden states to decode formula structure, arithmetic features, result digits, and output organization.
Results
The model exhibits a coherent sequential evolution in which formula structure, core arithmetic features, numerical abstractions, and output organization become decodable at different depths.
Takeaways & Limitations
The findings provide empirical evidence for an ordered, computation-like internal information representation path during addition.
Takeaways & Limitations
The correlational probes and logit lens establish decodability but do not prove that the forward pass uses those attributes or implements a specific carry-propagation algorithm.

Abstract

from arXiv · show

Multi-digit addition is a clear probe of the computational power of large language models. To dissect the internal arithmetic processes in LLaMA-3-8B-Instruct, we combine linear probing with logit-lens inspection. Inspired by the step-by-step manner in which humans perform addition, we propose and analyze a coherent four-stage trajectory in the forward pass:Formula-structure representations become linearly decodable first, while the answer token is still far down the candidate list.Core computational features then emerge prominently.At deeper activation layers, numerical abstractions of the result become clearer, enabling near-perfect detection and decoding of the individual digits in the sum.Near the output, the model organizes and generates the final content, with the correct token reliably occupying the top rank.This trajectory suggests a hierarchical process that favors internal computation over rote memorization. We release our code and data to facilitate reproducibility.

1 Introduction

The paper asks whether LLaMA-3-8B-Instruct performs multi-digit addition through structured computation or non-algorithmic pattern extraction. Using linear probes and the logit lens, it identifies an ordered four-stage progression from formula structure to output generation, while cautioning that decodability is diagnostic rather than causal.

The study examines whether LLaMA-3-8B-Instruct performs multi-digit addition through digit-wise summation and carry propagation or through non-algorithmic pattern extraction.
The logit lens is most informative late because the gold next token tends to reach rank one only near the end, suggesting intermediate reorganization rather than simple confidence accumulation.
The proposed trajectory makes formula structure, core computations, numerical abstractions, and output organization explicit and testable with layer-wise probes.
Linear probes reveal these signal families in sequence: formula structure appears early, digit-wise sums and carries in middle layers, numerical abstractions later, and output alignment near generation.
The analysis provides operational evidence for a structured information-processing pipeline but does not establish causal stage transitions or necessity of the decoded attributes.

2 Related Work

Related work frames LLM arithmetic around algorithmic computation, heuristic pattern matching, and abstract numerical representation. This paper combines linear probing with the logit lens to connect these perspectives through a layer-resolved account of addition in LLaMA-3-8B.

Algorithmic computation: Prior studies report algorithm-like arithmetic mechanisms, including modular addition, carry-routing components, and layer-wise specialization for column summation and carry propagation.
Heuristic pattern matching: A contrasting literature attributes arithmetic success to heuristic strategies and documents failures under digit permutations, novel symbols, or simple reorderings.
Abstract numerical representation: Other work finds abstract numerical structure, including linearly recoverable digits, low-dimensional numerical subspaces, and Fourier-like or group-structured representations.
Our position: This paper positions its contribution as a layer-resolved trajectory from formula recognition through core computation and numerical abstraction to output generation.

3 Methodology

The methodology tracks how arithmetic information evolves across LLaMA-3-8B-Instruct layers by combining logit-lens projections with linear probes on the last input-token state. The study prioritizes interpretable layer-wise diagnostics over leaderboard optimization and evaluates probe decodability on held-out data.

Overview: For each addition prompt, the study extracts the last input-token hidden state at every layer and applies logit-lens and linear-probe diagnostics.
Layer indexing: The analysis uses layer states L0–L32, with L0 as the embedding state and L1–L32 as outputs after Transformer blocks.
Dataset construction: The benchmark uses standardized short addition prompts, disjoint instance-level splits, controlled operand distributions, and preprocessing checks for tokenization artifacts.
Model selection: LLaMA-3-8B-Instruct is the primary subject because it provides clear, consistent probe signals, although other models largely preserve the ordering with shifted depths.
3.2 Logit Lens: The logit lens projects intermediate states to next-token logits, allowing the researchers to track gold-token rank and distribution sharpness across depth.
3.3 Linear Probes: Separate layer-specific linear classifiers decode arithmetic attributes such as carry bits, per-digit sums, and result digits, with accuracy measured on held-out data.

4 Experiments and Results

Layer-wise probes and logit-lens analysis reveal a staged progression from formula structure, through carry and sum information, to digit-level abstractions and late token selection.

4.1 Formula structure representation: Formula-structure probes rise from a 1/3 baseline to high accuracy in early-to-mid layers, with narrowing in-domain and OOD gaps across operand lengths.The task distinguishes a+b, b+a, and a+a, indicating increasingly accessible representations of operand order and self-addition before answer generation.
4.2 Emergence of core computational features: Sum-range accuracy rises sharply around L16–L19 from a 0.10 baseline and remains high thereafter across contiguous numerical ranges.Each range is evaluated as a 10-way classification task, and the common profile indicates similar emergence timing across ranges.
4.2 Emergence of core computational features: Carry probes begin increasing around ∼L14 and approach ceiling by ∼L19 for ones, tens, and hundreds positions.The timing aligns with the sum-range transition, supporting mid-layer establishment and consolidation of carry information before output formation.
4.3 Numerical abstraction of results: Digit-wise probes improve from near-baseline and reach a stable high-accuracy plateau after about ∼L28 across ones, tens, and hundreds positions.The similar depth profiles indicate that digit-level numerical abstractions consolidate late in the forward pass.
4.4 Logit lens: Organization and generation of output content: Hundreds-digit representations trained on addition generalize above 0.10 chance to subtraction and multiplication, with stronger transfer to subtraction.The reported transfer peaks near ∼0.9 for subtraction and ∼0.8 for multiplication.
4.4 Logit lens: Organization and generation of output content: All test prompts first reach gold-token top-1 in late layers L23–L32, peaking at L30 for about 45.4% of samples, with no never-top-1 cases.This places decisive output selection after the mid-layer transitions seen in sum-range and carry probes.

5 Analysis

The experiments indicate a sequential internal trajectory: structure appears first, core arithmetic signals emerge in middle layers, and deeper layers encode result digits before final output selection.

Sequential trajectory: The model’s representations progress from formula structure to core arithmetic features, then to result-level numerical abstractions and output generation.This ordering is presented as a four-stage layer-wise trajectory in LLaMA-3-8B-Instruct.
Formula structure: Formula structure becomes decodable in shallow layers, including a+b, b+a, and a+a patterns, with generalization across operand lengths.The analysis characterizes this as early recognition of the expression’s structural type.
Core computation: Carry signals become prominent roughly from layers 14–18 and approach near-perfect decodability around layer 19 across digit positions.This identifies the middle layers as the stage where key arithmetic-operation features become clearly accessible.
Numerical abstraction: After roughly layer 25, deeper states support accurate decoding of the individual hundreds, tens, and units digits of the sum.Logit-lens analysis then places final token organization and generation nearer the output.
Sum-range consistency: Sum-range probes reach high accuracy at similar depths around layers 16–19, showing no substantial range-dependent gradient in when these problems become solvable.The comparison covers ranges from 500–509 through 900–909.

6 Discussion

The discussion emphasizes dynamic, transferable representations and an arithmetic trajectory that resembles progressively decomposed human-style computation, while remaining diagnostic rather than causal.

Dynamic representations: Probe decodability can peak and later decline, indicating that representations are dynamically reorganized rather than monotonically refined.Examples include formula-structure curves peaking around layers 10–14 and carry or digit curves peaking near layers 18–20 before dipping.
Generalizability of representation: Addition-trained digit probes generalize more strongly to subtraction than multiplication, with peaks near 90% and 80%, respectively.The pattern supports shared numerical structure alongside operation-specific representational components.
Analogical computation: The sequential trajectory resembles human arithmetic decomposition, moving from expression structure through carries and sum calculation to digit confirmation and expression.The discussion frames this as an analogical characterization of the observed computation process.

7 Conclusion

The study finds a coherent, sequential trajectory of arithmetic information in LLaMA-3-8B-Instruct, from formula structure through computation and numerical abstraction to output organization. This ordering supports a computation-like account, while appearing qualitatively across architectures with model-specific layer differences.

The model’s internal arithmetic signals emerge in sequence: formula structure, core operations such as carries and sums, numerical abstractions, and output organization.
At different network depths, carries, sum values, and individual digits become clear and linearly decodable, with representation strength changing dynamically during processing.
The sequential feature emergence, numerical linkage, dynamic representations, and cross-task generalization support a computation-like process over rote memorization for in-distribution addition.
Comparative analyses found the trajectory largely consistent in Qwen2.5-Math-7B-Instruct, although feature-emergence layers and numerical-representation strategies vary by architecture.
The study provides empirical evidence for an ordered internal information-representation path underlying LLM arithmetic capabilities.
Although math-optimized models can achieve higher raw addition accuracy, the four-stage trajectory is robust across architectures and most clearly expressed in LLaMA-3-8B-Instruct.

Limitations

The study’s conclusions are limited by correlational diagnostics, a narrow in-distribution task setting, and model-dependent layer behavior. These constraints bound the claims to structured information processing rather than causal algorithms or universal layer locations.

The paper presents a layer-wise picture of multi-digit addition, but its caveats temper the strength of the broader claims.
Linear probes and the logit lens show decodability but do not establish that the model uses those attributes or implements a specific carry-propagation algorithm.Causal interventions such as activation patching, targeted ablations, or circuit-level surgery are left for future work.
The experiments study in-distribution base-10 multi-digit addition with short prompts and exact-match evaluation, not subtraction, multiplication, mixed bases, chain-of-thought, or adversarial perturbations.Claims are therefore scoped to base-10 addition under short prompts.
The main analyses focus on LLaMA-3-8B-Instruct, while the qualitative ordering recurs across models but exact layer indices and effect sizes vary by architecture and checkpoint.Broader generalization may depend on architecture, scale, tokenizer, instruction tuning, and math-oriented fine-tuning.

Ethical Considerations

The study investigates a benign arithmetic task using synthetic, non-sensitive data and aims to improve understanding of LLM internals. Its direct ethical scope is therefore limited to mechanistic analysis rather than broader societal risks.

The research studies internal information processing for basic multi-digit addition rather than developing new applications or models.
Multi-digit addition is a fundamental, well-defined task without sensitive content, personal data, or socially charged issues that would introduce bias.
The datasets use synthetically constructed arithmetic prompts and contain no personally identifiable or other sensitive information.
The research aims to improve understanding of LLM internals, contributing to model interpretability and transparency.
The work does not directly address broader LLM concerns such as bias, misuse, or environmental impact because its scope is a mechanistic study of arithmetic capabilities.

C.1 Formula Structure Representation (Addition-Order Experiments)

The appendix describes probes for recognizing arithmetic-expression structure and related addition attributes, using controlled datasets and digit-position analyses across several models. The supplied passages specify tasks and figure scopes but do not report numerical figure outcomes.

C.1 Formula Structure Representation (Addition-Order Experiments): The formula-structure task distinguishes a + b, b + a, and a + a for two-digit integers with a > b.
C.1 Formula Structure Representation (Addition-Order Experiments): The three formula classes use 5,000 random samples for the first two types and 80% of 90 unique pairs for the a + a type.
C.1 Formula Structure Representation (Addition-Order Experiments): Generalization tests mirror the formula-structure task with one-, three-, and four-digit addends.
C.2 Emergence of Core Computational Features (Arithmetic-Results Experiments): The arithmetic-results task predicts the exact sum, with each class corresponding to one sum value and examples drawn from five ranges between 500–509 and 900–909.
C.3 Carry-Signal Experiment: The carry-signal task performs binary detection of carries in the units, tens, or hundreds places using balanced datasets of 1,000 problems per decimal position.
C.4 Numerical Abstraction of Results: The numerical-abstraction task predicts the hundreds, tens, and units digits of three-digit sums through separate 10-way classifications, with an additional test on subtraction and multiplication results.
C.5 Logit Lens Analysis: The logit-lens analysis identifies the earliest layer whose output distribution ranks the correct sum token first, using 1,000 randomly generated three-digit addition problems.
D.1 Arithmetic Structure Recognition Experiments: Appendix figures for Mistral-7B-Instruct, Qwen2.5-Math-7B, and AceMath show probe accuracy by digit position for probes trained on two-digit addition.

D.2 Arithmetic Results Experiments

The experiments examine layer-wise arithmetic detection, carry recognition, sum-digit recognition, cross-operation transfer, and the emergence of correct top-1 predictions across three models.

Arithmetic detection: Layer-wise arithmetic detection is evaluated across numerical ranges for Mistral-7B-Instruct, Qwen2.5-Math-7B, and AceMath.The displayed ranges run from 500–509 through 900–909; the “a + b = 500” slice is treated as anomalous because correct samples comprise only ∼10%.
Carry detection: Carry recognition is measured separately for the ones, tens, and hundreds positions at each layer in all three models.
Sum-digit recognition: Layer-wise recognition accuracy is also measured for the ones, tens, and hundreds digits of the sum across the three models.
Cross-operation evaluation: Hundreds-digit probes trained on addition are evaluated on addition, multiplication, and subtraction test sets for each model.
Logit lens: Logit-lens analyses identify the distribution of layers where the correct sum token first becomes top-1 across 1,000 samples or the final ten layers.The analyses cover layers 23–32 for Mistral-7B-Instruct and Qwen2.5-Math-7B, and the last ten layers for AceMath.