PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating the Limitations of Transformers with Simple Arithmetic Tasks

Rodrigo Nogueira, Zhiying Jiang, Jimmy Lin

arXiv:2102.13019v3cs.CLcs.AIcs.LG

TL;DR

The paper asks whether number surface form affects how sequence-to-sequence transformers learn simple arithmetic and studies this question across multiple representations. It finds that explicit position tokens enable near-perfect arithmetic at longer lengths, while models still fail to extrapolate across changed length distributions.

  • Problem

    The paper investigates how transformer surface representations affect learning simple arithmetic, a capability relevant to more complex reasoning tasks.

  • Method

    The authors train sequence-to-sequence T5 models on generated addition and subtraction tasks using multiple number representations.

  • Results

    Explicit position-token representations achieve accuracy close to 100% through 15-digit addition, while character and underscore formats reach zero accuracy at 15 digits.

  • Takeaways & Limitations

    Simple surface manipulations can make arithmetic easy for pretrained transformers and suggest improving tokenizers and positional encodings.

  • Takeaways & Limitations

    Models cannot extrapolate arithmetic when evaluated on number lengths whose distribution differs from training, despite explicit semantic representations.

Abstract

from arXiv · show

The ability to perform arithmetic tasks is a remarkable trait of human intelligence and might form a critical component of more complex reasoning tasks. In this work, we investigate if the surface form of a number has any influence on how sequence-to-sequence language models learn simple arithmetic tasks such as addition and subtraction across a wide range of values. We find that how a number is represented in its surface form has a strong influence on the model's accuracy. In particular, the model fails to learn addition of five-digit numbers when using subwords (e.g., "32"), and it struggles to learn with character-level representations (e.g., "3 2"). By introducing position tokens (e.g., "3 10e1 2"), the model learns to accurately add and subtract numbers up to 60 digits. We conclude that modern pretrained language models can easily learn arithmetic from very few examples, as long as we use the proper surface representation. This result bolsters evidence that subword tokenizers and positional encodings are components in current transformer designs that might need improvement. Moreover, we show that regardless of the number of parameters and training examples, models cannot learn addition rules that are independent of the length of the numbers seen during training. Code to reproduce our experiments is available at https://github.com/castorini/transformers-arithmetic

1 INTRODUCTION

The paper examines how transformer models’ exclusive reliance on surface text affects simple arithmetic learning. It argues that explicit representations can make arithmetic easy, while performance remains limited when number lengths differ from training.

  • Transformers expose only surface text and supplemental embeddings, without explicitly materialized intermediate linguistic representations.
  • The paper uses simple arithmetic as a case study of how representation determines whether a task is nearly impossible or easy to learn.
  • Explicitly enumerating digit-position semantics can inject useful representations into transformer inputs without retraining the pretrained model.
  • The authors argue that appropriate representations remain crucial despite the power of end-to-end neural training.
  • Models fail to extrapolate arithmetic when evaluation length distributions differ from training, even with explicit semantic representations.The paper states that larger models, more computation, and more data do not solve this problem.

2 METHODOLOGY

The study formulates addition and subtraction as sequence-to-sequence generation and compares number representations, sampling strategies, and model configurations. Its representations range from ordinary decimal and character forms to explicit position-token encodings.

  • Task and evaluation: Addition and subtraction are framed as sequence-to-sequence tasks mapping two-number questions to exact numeric answers.Accuracy is one for an exact match and zero otherwise.
  • Data generation: Training, development, and test examples are programmatically generated with balanced or random number-length sampling.Balanced sampling distributes examples across digit lengths, while random sampling emphasizes the largest lengths.
  • Models: The experiments use pretrained T5 sequence-to-sequence models, with additional experiments involving vanilla transformers and multiple model sizes.The reported T5 sizes are 60M, 220M, and 770M parameters.
  • Representations: The study compares six number representations, including decimal, character, fixed-character, underscore, words, and position-token formats.The representations differ in how they expose digit identity and significance to the model.
  • Representations: 10-BASED representations separate digits with powers of 10, allowing digit significance to be inspected from neighboring position tokens.10E-BASED uses scientific notation to represent large powers of ten more compactly.

3 RESULTS

Arithmetic accuracy depends strongly on number representation: subword-like DECIMAL representations fail quickly, while explicit position tokens enable near-perfect performance through 15 digits. Other representations improve over DECIMAL but eventually degrade as digit length increases.

  • 0% accuracy occurs for DECIMAL addition at 5 digits or more.Inconsistent subword tokenization can make the same token represent different numbers of digits.
  • CHARACTER and UNDERSCORE outperform DECIMAL but both reach 0% accuracy at 15-digit addition.Their declining performance is attributed to the increasing difficulty of counting digits to determine significance.
  • Above 12 digits, FIXED-CHARACTER outperforms CHARACTER and UNDERSCORE, but reaches approximately 20% accuracy at 15 digits.The memorization strategy eventually breaks down because relative positional embeddings are difficult to learn precisely.
  • 40-60% accuracy remains stable for WORDS from 5 to 15 digits, but performance fails at 20 digits or more.Intrinsic position tokens such as “hundred” and “thousand” may help locate and combine distant digits.
  • Accuracy remains close to 100% through 15 digits for both 10-BASED and 10E-BASED representations.Explicit position tokens between digits let the model determine each digit’s significance from neighboring tokens.
  • Additional experiments show that representation matters with scarce data, although all representations can reach at least 97% accuracy with enough training data.Larger models perform better, but even 3B-parameter models cannot learn simple arithmetic rules; length-distribution mismatch also causes problems.

4 CONCLUSION

The paper shows that making numerical semantics explicit through simple surface-representation changes helps neural models learn arithmetic. It also identifies tokenizer and positional-encoding improvements as promising directions, while leaving broader transfer of the approach unresolved.

  • Simple surface-representation changes can help neural models learn simple arithmetic tasks.The paper presents this as a contribution to a problem that has remained challenging despite progress in neural networks.
  • Improving tokenizers and positional encodings is identified as a promising direction for future exploration.The paper does not establish whether the same approach applies to other tasks.

A RELATED WORK

Prior work shows that neural models often capture numerical magnitude more readily than numeracy and struggle to extrapolate, while specialized architectures and input representations offer alternatives. This paper isolates transformer failure cases using simple arithmetic to examine tokenization and positional encoding.

  • Recent studies have examined numerical capabilities learned from large text corpora, with a survey providing a broader overview.
  • Neural embeddings often capture magnitude relationships but fail to capture numeracy, with character-level models showing stronger numeracy than subword models.Similar-valued numbers can receive very different subword tokenizations.
  • Pretrained word embeddings have difficulty extrapolating to numbers unseen during training.Proposed alternatives include numerical-text augmentation and scientific-notation representations.
  • Specialized arithmetic architectures can sum numbers larger than those seen during training, unlike many general-purpose architectures.The cited contrast concerns extrapolation on numerical tasks.
  • Neural-symbolic hybrids combine neural vector conversion with symbolic rule application, while other evidence supports reasoning with suitable input representations.The paper focuses instead on a general-purpose architecture applicable to many natural-language-processing tasks.
  • The paper evaluates tokenizer and positional-encoding limitations with simple arithmetic rather than relying on complex numerical-reasoning benchmarks.This isolates transformer failure cases in a controlled task setting.
  • Complex numerical-reasoning datasets test comparison, sorting, and other mathematical tasks, whereas this work studies basic arithmetic as a prerequisite for more complex reasoning.

B POSITION EMBEDDINGS

The position-embedding study compares small unpretrained transformers using several representations and training conditions. Position-masked embeddings address the failure of standard sinusoidal encoding, while target-position training is generally less effective than omitting target positions.

  • Small transformers are fine-tuned without pretraining to study position embeddings on addition from 2 to 9 digits.The experiments compare 10E-BASED, 10-BASED, and CHARACTER representations.
  • The original sinusoidal positional encoding fails to learn addition effectively in the reported experiment.The authors attribute this possibly to correlation between token embeddings and absolute positional encodings.
  • Position-wise masked embeddings assign digit-specific embedding regions based on each digit’s significance.Source positions receive this encoding during training and inference, while target positions are either provided during training or set to zero.
  • 10E-BASED performs best under the same position encoding and training strategy.
  • Training with target position encoding generally performs worse and creates fluctuations across digit lengths than training without target position encoding.Whether target positions are provided is less important than using the same representation and encoding conditions consistently.

C EXPERIMENTS ON EXTRAPOLATION

The experiments distinguish interpolation from extrapolation by testing arithmetic beyond the digit lengths seen during training. Models interpolate successfully, but extrapolation remains sensitive to model size, generation order, sequence length, and training conditions.

  • Interpolation and extrapolation: Interpolation succeeds across model sizes, whereas extrapolation is substantially harder when evaluation lengths exceed training lengths.Interpolation uses training and testing up to 60 digits; extrapolation trains up to 50 digits and tests on 60-digit numbers.
  • Model size: T5-3B succeeds on almost all extrapolation tasks, while smaller models fail more often.For T5-220M, extrapolation accuracy reaches 0.862 for addition and 0.641 for subtraction in regular order, but T5-3B performs substantially better.
  • Generation order: Regular and inverse orders perform similarly on interpolation, but regular order yields higher extrapolation accuracy.T5-220M has zero extrapolation accuracy for both tasks with inverse order, compared with 60–90% using regular order.
  • Variance and data size: 20–50% accuracy is observed when T5-770M trained on up to 30 digits is evaluated on 60-digit numbers, with substantial variation across data-generation seeds.Extrapolation accuracy can range from 20% to 50% across training runs and oscillate by 20–40 percentage points across seeds.
  • Extrapolation boundary: Zero accuracy occurs when T5-220M, T5-770M, and T5-3B trained on 15 digits are evaluated on 20 digits.Extrapolation is hardly achieved when models are trained on fewer than 50 digits.
  • Training data: Additional training data does not improve extrapolation beyond a critical amount, while interpolation continues reaching 100% accuracy.For T5-770M evaluated on 60-digit numbers after training on 30 digits, accuracy remains 20%–50% with 100K, 1M, or 10M examples.
  • Failure behavior: Models can skip intermediate position tokens during extrapolation even though they later emit the correct end-of-sequence token.A T5-770M model trained on 30-digit numbers skipped position tokens 10e37 through 10e28 when evaluated on 60-digit numbers; this behavior was consistent across settings.

D IMPACT OF DATA SIZE

The amount of training data changes how strongly number representations affect addition accuracy. Position-aware representations lead with scarce data, while most representations exceed 99.9% accuracy at large dataset sizes, except DECIMAL.

  • Representation ranking: 1,000 examples rank 10E-BASED highest, followed by 10-BASED, WORDS, UNDERSCORE, CHARACTER, and DECIMAL for 30-digit addition.The experiment trains and evaluates T5-220M on addition of up to 30-digit numbers.
  • Large datasets: 10M examples produce more than 99.9% accuracy for almost all representations.The models are evaluated on addition of 30-digit numbers.
  • Large datasets: 2.1% error remains for DECIMAL after training with 10M examples.DECIMAL is the exception to the near-universal high accuracy at the largest dataset size.
  • Conclusion: With enough training data, models learn addition regardless of representation, while representation limitations are exposed mainly with small datasets.The conclusion applies to the representations evaluated in this experiment.

E PRETRAINED VS. FROM SCRATCH MODELS

The study compares pretrained and from-scratch models on addition of numbers up to 60 digits. Pretrained T5 models reach perfect accuracy with roughly ten times fewer examples and less compute.

  • Pretraining advantage: Approximately ten times fewer training examples and compute let pretrained T5-220M and T5-3B reach 100% accuracy than models trained from scratch.The comparison concerns addition of 60-digit numbers, with no repeated training examples.
  • Experimental comparison: Figure 4 compares pretrained and from-scratch model accuracy as the number of training examples increases for numbers up to 60 digits.The figure directly presents the training-example relationship for both model types.

F ACCURACY ON DIFFERENT BASES

The paper tests whether pretraining helps models learn addition in binary, ternary, decimal, and base 19. Pretrained models handle the common bases and degrade only slightly on base 19, whereas from-scratch models fail with all bases.

  • Experimental setup: The base-comparison experiment evaluates 15-digit addition using 10E-BASED orthography.The tested bases are binary, ternary, decimal, and base 19.
  • Pretrained models: Pretrained T5-220M learns binary, ternary, and decimal addition without difficulty, with only slight accuracy degradation on base 19.The experiment uses 1,000 examples and numbers equivalent to 15 decimal digits.
  • From-scratch models: Models trained from scratch fail to learn the addition task regardless of the numerical base.These models use 220M parameters but only 1,000 training examples.

G IMPACT OF DIFFERENT LENGTH DISTRIBUTIONS

The experiments examine whether mismatched training and test length distributions impair addition. Models trained on balanced lengths generalize across evaluation distributions, whereas random-length training does not generalize to shorter-number evaluation.

  • Balanced-distribution training succeeds on both random and balanced evaluation sets.The experiment trains T5-220M models on 100,000 examples, selects checkpoints with 10,000 development examples, and evaluates on 10,000 examples.
  • Random-distribution training succeeds on random evaluation but fails on balanced evaluation.The results concern accuracy on 60-digit addition under balanced and random sampling.
  • When 90% of training numbers have 60 digits, the model does not learn to add numbers with fewer digits.The reported finding indicates difficulty performing addition on sequences shorter than those seen during training.
Loading 2102.13019v3…