PaperlayerView exact revision on arXiv

Source-linked AI summary

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

Aaditya K. Singh, DJ Strouse

arXiv:2402.14903v1cs.CLcs.LG

TL;DR

Number-tokenization strategies vary across frontier models, while their effects on arithmetic reasoning have lacked systematic comparison. This paper tests tokenization directions on arithmetic tasks and analyzes model errors and a repetition-based mitigation. R2L tokenization improves accuracy over L2R, and the effect is stronger in smaller models.

  • Problem

    Frontier models use varied number-tokenization strategies, but their effects on numerical reasoning have not been systematically compared on the same arithmetic tasks.

    Source: §1 · p. 2

  • Method

    The authors compare L2R and R2L tokenization on GPT arithmetic tasks, analyze error patterns, and test prompting models to repeat inputs in a different tokenization.

    Source: §1 · pp. 1, 3, §5.1 · p. 6

  • Results

    R2L tokenization yields accuracy up to 20% higher than L2R, and tokenization effects are stronger in smaller models.

    Source: §1 · p. 3, §8 · p. 8

  • Takeaways & Limitations

    The findings support careful ablations of number-tokenization choices when developing models for numerical reasoning.

    Source: §1 · p. 3, §8 · p. 8

    The gold comparison—training the same architecture and dataset with varied number tokenizations—is expensive, and token-controlled comparisons would expose models to different amounts of data.

    Source: §8 · p. 8

Abstract

from arXiv · show

Tokenization, the division of input text into input tokens, is an often overlooked aspect of the large language model (LLM) pipeline and could be the source of useful or harmful inductive biases. Historically, LLMs have relied on byte pair encoding, without care to specific input domains. With the increased use of LLMs for reasoning, various number-specific tokenization schemes have been adopted, with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and GPT-4 have separate tokens for each 1-, 2-, and 3-digit numbers. In this work, we study the effect this choice has on numerical reasoning through the use of arithmetic tasks. We consider left-to-right and right-to-left tokenization for GPT-3.5 and -4, finding that right-to-left tokenization (enforced by comma separating numbers at inference time) leads to largely improved performance. Furthermore, we find that model errors when using standard left-to-right tokenization follow stereotyped error patterns, suggesting that model computations are systematic rather than approximate. We show that the model is able to convert between tokenizations easily, thus allowing chain-of-thought-inspired approaches to recover performance on left-to-right tokenized inputs. We also find the gap between tokenization directions decreases when models are scaled, possibly indicating that larger models are better able to override this tokenization-dependent inductive bias. In summary, our work performs the first study of how number tokenization choices lead to differences in model performance on arithmetic tasks, accompanied by a thorough analysis of error patterns. We hope this work inspires practitioners to more carefully ablate number tokenization-related choices when working towards general models of numerical reasoning.

1. Introduction

Tokenization can impose inductive biases, and the paper examines how number-tokenization choices affect numerical reasoning in frontier LLMs. Because models use varied number-tokenization strategies, the authors call for systematic comparisons on arithmetic tasks.

1. Introduction

  • Tokenization groups input bytes into discrete tokens, and different groupings may impart different inductive biases on numerical reasoning.The study focuses on GPT-3.5 and GPT-4 arithmetic problems.Source: §1 · p. 1
  • BPE can assign dedicated tokens inconsistently to neighboring numbers, producing idiosyncratic segmentations of numbers with the same digit length.For example, 710 may receive a dedicated token while 711 does not.Source: §1 · pp. 1–2
  • Prior arithmetic work used spaces or commas to separate digits so that each digit maps to a single token (Nye et al., 2021; Zhou et al., 2022).The paper contrasts such choices with the varied strategies used in current frontier models.Source: §1 · p. 2
  • GPT-3.5 and GPT-4 use tokens for 1-, 2-, and 3-digit strings and default to left-to-right chunks of three digits.PaLM, LLaMa, and Mistral instead use single-digit tokenization.Source: §1 · p. 2
  • The paper compares tokenization choices on few-shot arithmetic tasks, addressing the lack of convergence on number-tokenization practices across frontier models.The authors describe this as the first systematic comparison of performance on the same numerical-reasoning tasks with varied tokenization.Source: §1 · pp. 2–3

2. Methods

The experiments evaluate few-shot addition with GPT models and vary number-tokenization direction through formatting at inference time.

2.1. Experiment setup

  • The setup evaluates GPT models on few-shot addition problems with 7- to 9-digit addends, using 90 random problems for most experiments and 1-, 2-, 4-, or 8-shot prompts.Shots match the query’s digit lengths and tokenization direction; decoding is greedy at temperature 0.Source: §2.1 · p. 3

2.2. Varying L2R vs. R2L tokenization

  • Commas inserted every three digits from the right enforce right-to-left tokenization because the tokenizer lacks tokens combining numbers and commas.The API accepts input text rather than input tokens, so this formatting varies segmentation at inference time.Source: §2.2 · p. 3

3. Right-to-left tokenization improves model performance

Right-to-left tokenization improves arithmetic accuracy over the default left-to-right direction. Controls indicate that delimiter choice and added predictable tokens do not explain the performance advantage.

3.1. Main results

  • At 1 to 8 shots, L2R accuracy rises from 68.5% to 75.6%, while R2L accuracy rises from 95.6% to 97.8%.The authors report larger gains for L2R as shots increase, suggesting in-context learning may slightly mitigate its harmful tokenization bias.Source: §3.1 · p. 3, Figure 4 · p. 4

3.2. Controlling for comma-based semantic priors

  • At 8 shots, performance is largely agnostic to the R2L separator, indicating tokenization direction matters more than using commas specifically.The compared separators include spaces, periods, dollar signs, and hash signs.Source: §3.2 · p. 4, Figure 5 · p. 4

3.3. Controlling for “thinking tokens”

  • Neither L2R separator controls nor added spaces around operators recover the R2L performance boost, arguing against the extra-computation explanation.With L2R separators, the model sometimes auto-corrected inputs by hallucinating trailing zeros.Source: §3.3 · p. 4, Figure 6 · p. 4

4. Error analysis reveals stereotyped patterns

The error analysis finds that L2R tokenization particularly harms addition when the answer has more digits than the addends, with errors concentrated at a token boundary rather than tracking carry count.

4.1. L2R tokenization is significantly worse when answer is longer than addends

  • When answer length mismatches the addends, L2R accuracy drops to 8.25%, while length-match performance is similar across tokenization directions.The mismatch occurs when a final carry makes the answer longer than the inputs.Source: Figure 7 · p. 5, §4.1 · p. 5

4.2. Errors do not seem correlated to number of carries

  • Accuracy shows no clear trend with carry count, indicating performance is not strongly affected by the number of carries.The analysis includes only carry-count subsets with at least 50 problems.Source: Figure 8 · p. 5, §4.2 · p. 5

4.4. Off-by-one errors at token boundaries account for nearly all remaining errors

  • Nearly all remaining off-by-one errors occur in the last digit of an output token, suggesting errors are more likely across token boundaries than within a three-digit token.The counts are 24 of 25 missed R2L problems and 53 of 56 missed L2R length-match problems.Source: §4.4 · pp. 5–6

4.3. Length mismatch problems yield stereotyped “digit 4” error pattern

  • In length-mismatch L2R errors, the model always gets digit 4 wrong while the first three digits are correct, a tokenization-dependent pattern suggestive of systematic processing.Digit 4 errors show a slight preference for off-by-one mistakes, but substitutions are otherwise haphazard; the pattern is absent with R2L tokenization.Source: §4.3 · p. 5, Figure 9 · p. 6

5. Models are able to convert from L2R to R2L tokenization, improving performance

The models can improve arithmetic on L2R inputs by repeating the problem in R2L form before answering. Simply answering in R2L form without repeating the problem does not yield the same improvement.

5.1. Main results

  • Models perform nearly as well converting L2R inputs to R2L themselves as when given R2L inputs initially, with performance improving as few-shot examples encourage repetition.The figure compares few-shot accuracy when a problem is presented in one direction and then repeated and answered in another.Source: Figure 10 · p. 6, §5.1 · p. 6

5.2. Controlling for output tokenization

  • Answering in R2L form alone improves performance less than also repeating the problem in R2L form, showing repetition matters to the gain.The control removes the instruction to repeat the problem while still prompting answers in a different tokenization direction.Source: §5.2 · p. 6

6. Tokenization-dependent effects mostly extend to future models

Tokenization-dependent performance differences persist across newer OpenAI models, though their strength varies by model version. The answer-length mismatch effect also recurs in newer GPT-3.5 and GPT-4 models.

6. Tokenization-dependent effects mostly extend to future models

  • For R2L-to-L2R prompts, formatting adherence rises from 13.3% with one shot to 98.9% with eight shots, while accuracy worsens as adherence increases.With fewer shots, models often ignore the prompt and answer in their preferred R2L format.Source: Figure 11 · p. 7
  • Tokenization effects persist: they remain strong in newer GPT-3.5, weaken in GPT-4’s March and June versions, then strengthen again in GPT-4 Turbo.GPT-4 performs better across both tokenization directions, while GPT-4 Turbo shows a slight overall regression.Source: Figure 12 · p. 7, §6 · p. 7
  • The authors caution that, without knowing what the model updates entail, they cannot explain why GPT-4 improved from March to June while GPT-3.5 did not.Source: §6 · p. 7
  • Answer-length mismatch remains the main reason for the L2R performance drop in both GPT-3.5 and GPT-4 models.The result is reported for answer-length-controlled problems.Source: Figure 13 · p. 7, §6 · p. 7

7. Related work

Related work situates this study among research on tokenization choices and artifacts, as well as arithmetic tasks used to probe numerical reasoning.

7. Related work

  • Modern LLMs commonly use BPE, but vary in hard-coded tokenization choices; the paper studies effects from OpenAI models’ use of one-, two-, and three-digit number tokens.The discussion also notes tokenizer-free approaches and continuous number encoding, whose cited approach is limited to purely numerical outputs.Source: §7 · p. 7
  • Prior work documents tokenization-related artifacts in sorting, completions, and extractive question answering; this paper adds a systematic study of tokenization-direction effects on numerical reasoning.The paper also connects its repetition-and-response prompts to token healing and rephrase-and-respond approaches.Source: §7 · p. 8

Arithmetic tasks as a testbed for numerical reasoning

  • Arithmetic studies have linked numerical performance to training-data frequency and task generalization; other work finds reversed-order addition can simplify the learned algorithm and improve performance.The authors report no strong frequency effect using BPE merge ranks as a proxy, and focus instead on tokenization-dependent effects.Source: Arithmetic tasks as a testbed for… · p. 8

8. Discussion

The discussion concludes that tokenization can shape numerical reasoning, while noting unresolved tradeoffs in tokenization choice and controlled comparisons. It also identifies tokenization-dependent error patterns as targets for interpretability research.

8. Discussion

  • Tokenization choices in GPT-3.5 and GPT-4 produce stereotyped error patterns, and the discussion concludes that tokenization can significantly influence numerical reasoning.The authors propose repeating answers in the model’s preferred tokenization as a mitigation and find the effect stronger in smaller models.Source: §8 · p. 8
  • GPT-3.5 and GPT-4’s 1–3-digit tokens may improve compression and length generalization, but L2R input-output misalignment can sharply reduce accuracy, especially in smaller models.The discussion notes that single-digit tokens would avoid this particular misalignment.Source: §8 · p. 8
  • A decisive comparison would train the same architecture on the same dataset with different number tokenizations, but cost and compute control are major obstacles.With up-to-3-digit tokens, a token-controlled experiment would expose some models to more data because of their better compression ratio.Source: §8 · p. 8
  • Three-digit number tokens can reduce inference-time compute when models produce numbers, potentially mattering for serving at scale.Source: §8 · p. 8
  • The robust GPT-3.5 “digit 4” error offers interpretability researchers a target for investigating mechanisms behind multi-token arithmetic errors.The discussion contrasts this with prior circuit work restricted to single-token operands.Source: §8 · p. 9

A. Tokenization differences between frontier LLMs

  • Claude’s number tokenization lacks systematicity, suggesting pure BPE rather than a bespoke number scheme.Its token coverage is generally higher than p50k base, likely partly because Claude’s vocabulary has 65k tokens versus 50k for OpenAI’s p50k base.Source: Figure 14 · p. 15, Figure 15 · p. 15

B. Experiments with other system prompts

The authors tested an alternate system prompt and found small performance gains at low shot counts that diminished by eight shots. They retained the default prompt for reproducibility and applicability.

B. Experiments with other system prompts

  • The alternate system prompt instructed MathGPT to repeat the problem statement and append only the answer.The authors compared it with the default “You are a helpful assistant” prompt.Source: B. Experiments with other system prompts · p. 15
  • The alternate prompt gave small performance improvements at low shot counts, such as 1-shot, but these diminished at 8-shots.Because most reported results use 8 shots, the authors doubted the prompt would have a large effect on their findings.Source: B. Experiments with other system prompts · p. 15
  • Figure 16 compares R2L and L2R tokenization across shot counts and two system prompts.Source: Figure 16 · p. 16

C. Frequency effects

The frequency analyses find only mild evidence that GPT-3.5 favors more frequent output tokens, and do not identify frequency as the dominant account of its error patterns. The authors note that merge rank is an imperfect proxy and call for follow-up with open pretraining data.

C. Frequency effects

  • Of GPT-3.5’s errors, 60% of 25 under R2L and 56% of 425 under L2R substituted a more frequent token; the authors judge the effect too small to dominate the error patterns.The L2R result had p = 0.005; the R2L result had p = 0.115, using a binomial null distribution assuming chance is 50%.Source: C. Frequency effects · p. 16
  • For “digit 4” errors, the most likely token appeared roughly 16% of the time versus 10% by chance, with no clear overall rank trend.Figure 17 shows the distribution of relative ranks for substituted tokens in the length-mismatch condition.Source: C. Frequency effects · p. 16, Figure 17 · p. 17
  • In off-by-one errors, the most frequent token was chosen 7 of 24 times under R2L and 17 of 53 times under L2R, roughly chance-level rates.The authors conclude that output-token frequency is not a dominant factor in these errors.Source: C. Frequency effects · p. 16
  • The frequency test uses BPE merge ranks as a proxy, but the authors lack GPT-3.5 and GPT-4 pretraining data and describe the proxy as imperfect.They suggest further study in newer, larger models with open pretraining data.Source: C. Frequency effects · pp. 16–17

D. Stereotyped patterns in model log probabilities

Log-probability analyses reveal distinct arithmetic error patterns across tokenization conditions: length-mismatch L2R errors look like magnitude-biased fourth-digit guesses, while other errors are mostly off by one. In shared correct cases, models are slightly more confident with R2L tokenization.

D. Stereotyped patterns in model log probabilities

  • The analysis compares top-five token log probabilities and per-output-token entropy across L2R length-mismatch, L2R length-match, and R2L conditions.Entropy is reported in nats, using a lower bound because only the top five log probabilities are available.Source: D. Stereotyped patterns in model log… · p. 17
  • Entropy is similar for incorrect and correct digit-4 cases (2.066 and 2.061), and the correct answer appears among the top five tokens in 49.6% of errors.The authors suggest the model may be guessing a second output token with the correct tens and ones digits but a random hundreds digit.Source: D. Stereotyped patterns in model log… · p. 17
  • For 53 L2R and 25 R2L off-by-one errors on length-match problems, the correct answer was always the second most likely token, with entropy around 0.45 ± 0.05.Correct outputs in these conditions had much lower entropy, approximately 0.03 averaged across dataset and tokens.Source: D. Stereotyped patterns in model log… · p. 18
  • On length-match problems answered correctly with both tokenizations, the model was slightly more confident with R2L, as shown by the answer-logprob difference histogram.Figure 18 compares summed answer log probabilities for this shared-correct subset.Source: D. Stereotyped patterns in model log… · p. 18, Figure 18 · p. 18
  • L2R length-mismatch outputs show magnitude-biased fourth-digit guesses, while other conditions are mostly confidently correct or uncertain between an off-by-one answer and the true answer.The authors interpret the latter errors as considering mainly those two outputs.Source: D. Stereotyped patterns in model log… · p. 18

E.1. Length control for error analysis

The appendix describes a 1,300-problem length-controlled error analysis, documents API access and model-availability caveats, and provides examples of the multi-turn prompts used in experiments.

E.1. Length control for error analysis

  • The length-controlled error analysis sampled 100 problems for each of 13 addend-and-answer digit-length triplets, totaling 1,300 problems.The triplets covered 7- to 9-digit inputs and answers.Source: §E.1 · p. 18

E.2. Access dates

  • The authors report API non-determinism even at temperature 0 and say gpt-4-0314 became inaccessible before its stated June 13, 2024 deprecation date.They provide access dates for the experiments.Source: §E.2 · pp. 18–19
  • The entropy footnotes compare observed lower bounds with random-guessing references: about 2.3 for ten tokens and about 0.69 for two tokens.The ten-token comparison aligns with an observed lower bound of about 2.06; the two-token comparison aligns with about 0.45.Source: §E.2 · p. 18

E.3. Example prompts

  • The prompt appendix displays two-shot multi-turn examples, while noting that most main-text experiments used eight shots and sampled length-matched shots randomly for each query.For Section 4 and Appendix E.1, examples were also controlled to match the query's answer length.Source: §E.3 · p. 19
Loading 2402.14903v1…