Source-linked AI summary
"Close enough?" An investigation into multiplication errors in LLMs
William Bolton, Bezawit Tilaye, Huw Day
TL;DR
The paper asks whether an addition-based account of carry-boundary errors also explains multiplication, a question relevant to understanding LLM arithmetic errors. It evaluates small LLMs on multiplication problems and finds that multiplication errors differ from the addition pattern, with lower-order parity preservation and higher-order off-by-one errors in small Qwen models.
Problem
Whether the addition-based noisy-quantisation account of carry-boundary errors generalizes to multiplication remains an open question about LLM arithmetic errors.
Source: §1 · p. 1, §5.1 · p. 7Method
Results
Qwen3.5-2B errors in the tens column preserve parity 92.7% of the time versus a 44.4% null baseline, while higher-order errors shift toward off-by-one failures exceeding 64%.
Source: §5.1 · p. 7Takeaways & Limitations
Multiplication errors do not follow the addition-style noisy-quantisation pattern, and error regimes differ across digit order in small Qwen models.
Source: §5.1 · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) frequently fail at multi-digit arithmetic, but the underlying mechanisms governing these failures remain poorly understood. While recent work demonstrates that addition errors follow a "noisy quantisation" model localised around carry boundaries, whether this framework generalises to multiplication remains an open question. In this work, we evaluate multiplication capabilities across several small base LLMs on a dataset of 16,991 problems spanning varied operand lengths. We show that multiplication errors do not conform to the addition-style noisy quantisation model. Instead, we uncover a novel error regime in small Qwen models: errors in lower-order columns predominantly preserve parity. Conversely, higher-order digit errors shift toward off-by-one carry failures. Our findings suggest that arithmetic computation in LLMs relies on distinct, multi-periodic internal representations where parity is tracked more robustly than exact digit identity, providing new mechanistic insights into numerical reasoning in transformers.
1 Introduction
Arithmetic offers a controlled way to study how LLMs handle precise information, despite their broader mathematical capabilities and persistent fallibility. This work investigates multiplication errors in small LLMs and whether an addition-error model transfers.
1 Introduction
- Despite helping solve difficult Olympiad problems and some open research problems, LLMs remain fallible and notoriously unreliable at arithmetic.Production systems may use tools for larger calculations to guarantee a correct answer, but are still fallible.Source: §1 · p. 1
- Arithmetic is an easy, verifiable, controlled task for studying how LLMs struggle with precise information.Understanding arithmetic-error mechanisms may support development of models better able to handle precise information.Source: §1 · p. 1
- The paper investigates multiplication error modes in small LLMs and tests whether addition models such as noisy quantisation transfer.It proposes future mechanistic-interpretability work to validate its hypotheses.Source: §1 · p. 1
2 Related Work
Prior work describes periodic numerical representations and digit-wise arithmetic in LLMs. For addition, a noisy-quantisation account links errors to carry potentials near digit boundaries, motivating a test of whether that account extends to multiplication.
2 Related Work
- A 2024 study found Fourier-feature number representations and distinct roles for MLP layers, which approximate answer magnitude, and attention layers, which use high-frequency modular-arithmetic features [11].The study examined fine-tuned GPT-2-XL, prompted GPT-J and Phi-2, and GPT-3.5 and GPT-4.Source: §2 · p. 2
- Baeumel et al. found that LLMs perform arithmetic digit by digit using circuits for units, tens, and hundreds, even when a three-digit number is one token [13].Source: §2 · p. 2
- Kantamneni and Tegmark found that numbers lie on a helix causally implicated in addition and relevant to multiplication and division; addition constructs a + b by manipulating the helices for a and b [14].Source: §2 · p. 2
- Wen et al.'s noisy quantisation model attributes addition's off-by-one digit errors to noise near carry boundaries, prompting the paper to test whether a similar effect occurs in multiplication.Štefánik et al. reported universal sinusoidal number representations and a relation between output-representation fidelity and arithmetic accuracy [15].Source: §2 · p. 2
3 Experiments
The experiments evaluate multiplication in small open-weight LLMs using varied operand lengths, minimal prompts, and exact and per-digit accuracy measures. They also adapt carry potential as a way to examine possible carry-related errors.
3 Experiments
- The evaluation covers five small open-weight models: Qwen3.5-2B, Llama-3.2-3B, Gemma-4-E2B, Qwen3.5-0.8B, and Qwen2.5-Math-1.5B.Source: §3 · p. 2
- The 16,991-problem dataset spans seven operand-size tiers; the three smallest enumerate every pair, while the four largest use uniform random samples without replacement.The largest-tier samples are capped at 3,000 pairs for 2-digit × 3-digit and 3-digit × 3-digit, and 1,000 pairs for 2-digit × 4-digit and 3-digit × 4-digit.Source: §3 · p. 2, Table 1 · p. 3
- Models complete minimal multiplication prompts with greedy decoding, and evaluation measures both whole-answer accuracy and errors per digit.Answers are extracted with regular expressions; per-digit analysis follows prior work on digit-wise arithmetic.Source: §3 · p. 2
- Carry potential Φ_p sums digit contributions to the right of position p and represents the thrust to carry from previous columns.The experiments consider whether noise near integer boundaries in this quantity could produce off-by-one digit errors, as proposed for addition.Source: §3 · pp. 2–3
4 Results
Multiplication errors do not follow the addition-style noisy-quantisation pattern, although accuracy declines as carry increases. Small Qwen models show parity-preserving errors in lower-order columns, while the pattern varies across models and higher-order columns.
4 Results
- The noisy-quantisation model does not fit multiplication errors: its Figure 2 fit has R2 = −1.74, worse than a flat baseline.The authors suggest that summed partial products and more variable carry potentials may impede the addition-style bathtub pattern; a 2-digit × 2-digit carry potential can reach 17, compared with addition’s cap of 3.Source: §4 · p. 3, Figure 2 · p. 4
- Accuracy decreases monotonically with more carry columns, except that 3-digit × 4-digit cases have too few no-carry examples to be informative.Incorrect Qwen3.5-2B answers have a mean relative error of 1.1%.Source: §4 · pp. 3–4, Figure 5 · p. 5
- 92.7% of Qwen3.5-2B’s incorrect tens-column digits preserve parity, versus a 44.4% data-derived chance baseline; higher columns shift toward off-by-one errors.The parity test rejects its model-based null for all columns except the thousands column; in columns 10^4 and 10^5, 64% and 85% of errors are off by one, respectively.Source: §4 · pp. 5–6, Figure 6 · p. 5
4.1 Other Models
- The parity pattern is strongest in Qwen3.5-2B: its tens-column rate is 92.7%, compared with 69.5% for Qwen3.5-0.8B, despite the smaller model’s higher overall accuracy.Other model families essentially sit at chance; Gemma’s pattern may be obscured by low multiplication accuracy.Source: §4.1 · p. 6, Table 2 · p. 6
5 Discussion and Future Work
Multiplication errors in small language models do not follow addition’s noisy-quantisation pattern; instead, error behavior differs across digit order and model family. The discussion proposes mechanistic and cross-architecture studies to investigate these patterns.
5.1 Discussion
- Multiplication errors remain distributed across carry potentials rather than clustering near boundaries, unlike addition errors under the noisy-quantisation model.The authors suggest that generating and summing multiple partial products may contribute to this difference.Source: §5.1 · p. 7
- Lower-order Qwen errors, especially in the tens column, preserve ground-truth parity more often than expected, suggesting parity is represented more robustly than exact digit identity.Qwen3.5-2B shows 92.7% parity preservation versus a 44.4% null baseline in the tens column.Source: §5.1 · p. 7
- In higher-order columns, parity preservation drops sharply and off-by-one errors exceed 64%.Source: §5.1 · p. 7
- The strong parity-preservation effect appears in Qwen models but not similarly sized Llama and Gemma models.Tokenization granularity, baseline accuracy, architecture, and pre-training dynamics are discussed as possible factors.Source: §5.1 · p. 7
5.2 Future Work
- The authors propose activation patching, targeted ablation, and intermediate-layer probes to test whether mod 2 remains decodable as mod 10 degrades.Source: §5.2 · p. 7
- Cross-architecture studies are proposed to examine how tokenization strategies and training regimes relate to the emergence of parity-preserving circuits.Source: §5.2 · p. 7
- The authors propose testing whether answer-agnostic activation steering can restore sinusoidal coherence and mitigate arithmetic failures at inference time.Source: §5.2 · p. 7