PaperlayerView exact revision on arXiv

Source-linked AI summary

Positional Description Matters for Transformers Arithmetic

Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, Yi Zhang

arXiv:2311.14737v1cs.CLcs.AIcs.LG

TL;DR

Arithmetic is challenging for transformers in part because its positional regularities differ from typical language tasks and may not extend to unseen digit positions. The paper explores positional-encoding and data-format changes, reporting that small models can perform intricate arithmetic tasks.

  • Problem

    Arithmetic has distinctive regularities and token significance, and it is unclear how transformers relying on absolute positions handle digits at unseen positions.

    Source: §1 · pp. 1–2

  • Method

    The paper investigates modifying positional encodings and arithmetic data formats to use positional information more efficiently.

    Source: §1 · p. 2

  • Results

    GPT2-small-sized 124M-parameter models can adeptly execute intricate arithmetic tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A small subset of model parameters can integrate arithmetic proficiency without affecting capacity on other tasks.

    Source: §1 · p. 2

    At a maximum of 20 digits, results for larger numbers were unsatisfactory, and the experiment did not fine-tune parameters.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Transformers, central to the successes in modern Natural Language Processing, often falter on arithmetic tasks despite their vast capabilities --which paradoxically include remarkable coding abilities. We observe that a crucial challenge is their naive reliance on positional information to solve arithmetic problems with a small number of digits, leading to poor performance on larger numbers. Herein, we delve deeper into the role of positional encoding, and propose several ways to fix the issue, either by modifying the positional encoding directly, or by modifying the representation of the arithmetic task to leverage standard positional encoding differently. We investigate the value of these modifications for three tasks: (i) classical multiplication, (ii) length extrapolation in addition, and (iii) addition in natural language context. For (i) we train a small model on a small dataset (100M parameters and 300k samples) with remarkable aptitude in (direct, no scratchpad) 15 digits multiplication and essentially perfect up to 12 digits, while usual training in this context would give a model failing at 4 digits multiplication. In the experiments on addition, we use a mere 120k samples to demonstrate: for (ii) extrapolation from 10 digits to testing on 12 digits numbers while usual training would have no extrapolation, and for (iii) almost perfect accuracy up to 5 digits while usual training would be correct only up to 3 digits (which is essentially memorization with a training set of 120k samples).

1 Introduction

Arithmetic remains a challenge for transformers, whose positional strategies may not generalize to longer numbers. The paper examines arithmetic-specific challenges and approaches that use positional information more efficiently.

1 Introduction

  • Arithmetic differs from natural language in its concise vocabulary, predictable patterns, and equal significance across tokens, while often requiring intricate intermediate processing.These properties make it uncertain whether architectures tailored mainly to language can handle arithmetic efficiently and accurately.Source: §1 · p. 1
  • The paper identifies three challenges: direct large-number calculation, addition length extrapolation, and integration of arithmetic with natural language.Directly recording intermediate steps can be verbose, while arithmetic and language data differ substantially.Source: §1 · p. 2
  • For length extrapolation, models may rely on absolute positions to align digits, leaving the handling of digits at unseen positions unclear.The paper illustrates this with an addition task containing a fourth digit not seen during training.Source: §1 · p. 2
  • The authors investigate two ways to use position information more efficiently: changing positional encoding or changing arithmetic data representation.They explore randomized embedding and data formats that may lead models to learn arithmetic differently.Source: §1 · p. 2
  • The study focuses on 124M-parameter GPT2-small-sized models, which the authors report can handle intricate arithmetic tasks.They state that arithmetic proficiency can be integrated without affecting the model’s capacity on other tasks.Source: §1 · p. 2
  • The authors report that their approach outputs 15-digit multiplication products and is essentially perfect up to 12 digits.They state they are not aware of prior work achieving this result.Source: §1 · p. 2

2 Large Number Multiplication

The section tests whether a small transformer can multiply large numbers directly, without a scratchpad. Basic training on mostly small multiplication examples performs poorly on larger numbers; padding and reversing products improve results, though 20-digit performance remains unsatisfactory.

2 Large Number Multiplication

  • The 300k-example dataset samples factor lengths uniformly from 1 to n, yet contains fewer than 0.0002% of 5-digit-by-5-digit cases.It includes around 8k single-digit products despite only 100 possible single-digit multiplications.Source: §2 · p. 3
  • Basic-format training performs well on 1–2-digit multiplication but poorly when the factors’ combined digit count exceeds 5.For combined lengths below 5, the training set covers more than 10% of possible combinations, leaving memorization as an alternative explanation for smaller-number proficiency.Source: §2 · pp. 3–4, Table 2 · p. 3

2.1 Padding

  • Padding gives factors and products fixed lengths, and reversing the product lets generation proceed from least to most significant digits.This makes the least-significant-digit calculation more straightforward; padding and reversal are evaluated together and separately.Source: §2.1 · p. 4, Table 4 · p. 4
  • At a 20-digit maximum, results for larger numbers remain unsatisfactory; the experiment did not fine-tune parameters.The authors note that larger training data, a more optimal digit distribution, or a more finely tuned learning rate could potentially improve accuracy.Source: §2.1 · p. 4
  • Padding plus reversed products supports accurate multiplication up to 15 × 15, while non-padded formatting struggled even with 4 × 4 multiplication.The result is reported for a 12-layer transformer producing the product directly, without a scratchpad.Source: §2 · p. 3, §2.1 · p. 4, Table 5 · p. 5

3 Length Extrapolation

The section tests ways to reduce reliance on absolute position in addition, using alternative data formats and positional embeddings to improve length extrapolation. It reports gains from both approaches, with outcomes depending on the format and model initialization.

3.1 Data format

  • Random spaces make fixed-position retrieval harder, while repeating information at each digit-wise step is intended to support learning the steps rather than positional patterns.The section also evaluates a chain-of-thought scratchpad format, which lists the addends, digit-wise summation steps, and final sum but struggles to generalize to longer numbers.Source: §3.1 · p. 5
  • Basic-format training through 10 digits fails beyond 10, random spaces extend accurate prediction by one digit, and increased per-step information improves generalization further.The model trained on the basic format omits digits on longer inputs; the passage describes the one-digit extension as a progression from no generalization to some generalization.Source: §3.1 · pp. 5–6
  • Recursive scratchpad generalization requires pretraining, random-space results change little with initialization, and reversing addend digits helps recursive scratchpad more than reversing both addends and sum.Basic format does not generalize to additional digits for either pretrained or randomly initialized models.Source: §3.1 · p. 6

3.2 Positional Embedding

  • The positional-embedding experiments include digit reversion on numbers longer than training examples and assess both regular and repetitive inputs.The section uses digit reversion as a simpler task for examining positional embedding before testing addition.Source: §3.2 · p. 6
  • Removing positional embedding improves regular-data generalization by approximately two digits for both model types, but repetitive data yields perfect 16-digit accuracy only for the pretrained model.The randomly initialized model has low accuracy even on 9-digit repetitive data.Source: §3.2 · p. 6
  • Random embedding improves generalization on digit reversion and addition, with addition performance reported as about the same as recursive scratchpad.The reversion tests include regular and repetitive data.Source: §3.2 · p. 8

4 Addition in Natural Language Setting

The section tests whether arithmetic-only data can support addition in natural-language dialogue, where difficult examples are scarce. It finds that data-format or positional-embedding changes help integrate the two formats, while simply mixing them does not significantly improve accuracy.

Addition and language integration

  • Naively mixing arithmetic and natural-language data does not significantly boost accuracy because their different formats make integration challenging.Predictable arithmetic patterns can invite positional reliance that does not carry over to arithmetic in natural-language context.Source: §4 · p. 8
  • 4–5-digit dialogue addition remains difficult for models trained only on dialogue, which lacks enough correct examples at those lengths.Adding arithmetic data helps only with suitable data formats or positional-embedding choices.Source: §4 · p. 8
  • Random spaces, removing absolute positional embeddings, and randomized embeddings improve dialogue-based addition when arithmetic data is mixed with dialogue.For models without positional embeddings or with randomized embeddings, Basic and Random space reach similar accuracy after sufficient training.Source: §4 · p. 8
  • Models without positional embeddings do slightly better in dialogue contexts, while randomized embeddings outperform them in pure-addition contexts.The comparisons are reported in Tables 9a–9d.Source: §4 · p. 9, Table 9 · p. 9
  • The section concludes that integration requires changing the data format or positional embeddings; randomized embeddings can offset stability drawbacks from removing absolute positional embeddings.Random spaces are a data-format example; removing absolute embeddings or adding randomized embeddings are positional approaches.Source: §4 · pp. 9–10

A.1 n × 1-Multiplication

The appendix examines whether exposing models to single-digit-factor products, and explicitly linking those products to larger multiplications, improves multiplication performance. The explicit first-step format yields a further improvement over adding more single-digit-factor examples alone.

First-step multiplication

  • The first-step multiplication format further improves accuracy over adding n × 1 examples alone, though its gains may be less pronounced than padding’s.Table 10 reports testing accuracies for both data augmentations.Source: Table 10 · p. 12, §A.1 · pp. 12–13

n × 1 augmentation

  • 1/3 of training datapoints are changed to n × 1 multiplication by replacing the second factor with its final digit; this somewhat improves accuracy.The models use reversed products without padding, and the comparison is against Table 4.Source: §A.1 · p. 12

First-step multiplication

  • The % operation defines a × (b mod 10) and labels it as guidance for the first step of a × b.The operation is substituted for * in every third training datapoint to connect n × 1 and n × m multiplication.Source: §A.1 · p. 12

Implications

  • The authors argue that presenting subproblems as parts of the harder task can connect simple cases to complex ones more effectively than simple cases alone.They suggest constructing datasets from hard problems’ subproblems may be more practical than collecting incrementally harder tasks.Source: §A.1 · p. 14

B Additional Details for Section 2

This appendix section specifies the multiplication training setup and identifies the testing-accuracy table for the 10-maximum-digit Basic-format condition.

Setup

Additional results

  • Table 13 reports testing accuracy for models trained on a 10-maximum-digit dataset in Basic format, evaluated on 100 samples.Source: Table 13 · p. 15

C.1 Additional Details for Section 3.1

This appendix details addition scratchpad formats, data generation, and fine-tuning conditions for Section 3.1. It describes Basic, Random Space, and Recursive Scratchpad representations and how their training and test data are constructed.

Data formats

  • Basic records each digit’s sum with its carry; Random Space inserts spaces between characters with probability 0.3; Recursive Scratchpad repeats remaining digits and accumulated sums while reversing addends.Source: §C.1 · p. 15

Data Generation

  • 120k addition examples are generated using addend lengths sampled uniformly from 2 to 10 digits, with each addend sampled uniformly from 0 to 10^n−1.Test addends are sampled independently for each digit length.Source: §C.1 · p. 15

Setup

  • Pretrained GPT2-small (124M) is fine-tuned for five epochs at learning rate 2e−5, then tested on 100 additions per length for n = 9 through 13.Source: §C.1 · p. 15

C.2 Additional Details for Section 3.2

The appendix describes how addition examples and test sets are generated, and identifies a testing-accuracy table for multiplication models trained with padding and/or reversed products.

C.2 Additional Details for Section 3.2

  • Training examples use a digit length sampled uniformly from 2–10, with 120k samples generated independently.For each length, numbers are sampled from the stated uniform range.Source: §C.2 · p. 15
  • Testing uses both regular numbers sampled for each digit length and numbers formed by repeating a randomly sampled digit.The passage describes these as two types of testing data.Source: §C.2 · p. 15
  • Table 14 reports testing accuracy for models trained with padding and/or reversed products, with a maximum of 10 digits.Source: Table 14 · p. 16

D Additional Details for Section 4

The appendix details two addition-data formats, dialogue-data construction, and the evaluation setup for comparing positional-embedding variants.

D Additional Details for Section 4

  • Table 15 gives testing accuracy for 20-maximum-digit data with padding and reversed product, while Table 16 gives sample failure cases for the three data formats.Source: Table 15 · p. 17, Table 16 · p. 17
  • GPT-3.5-generated dialogue data uses 50 random numbers; the appendix reports about 12,200 dialogues for 2–3-digit addition and 1,040 for 4–5-digit addition.The dialogue data contains more 2–3-digit examples; the authors leave prompt-format errors uncorrected.Source: D Additional Details for Section 4 · pp. 17–18
  • The two arithmetic formats are Basic, with operands followed by their sum, and Random Space, which inserts independently sampled spaces of 1–5 between number strings.The appendix identifies these as its two types of arithmetic data.Source: D Additional Details for Section 4 · pp. 16, 18
  • The setup compares dialogue-only with mixed dialogue-and-addition training, testing dialogue and arithmetic prompts across positional-embedding variants.Models are trained for 100 epochs; accuracy is checked on 100 samples per digit length.Source: D Additional Details for Section 4 · p. 18

Related papers

Loading 2311.14737v1…