PaperlayerView exact revision on arXiv

Source-linked AI summary

TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar

Yinxi Li, Yuntian Deng, Pengyu Nie

arXiv:2510.14972v1cs.CLcs.AIcs.LGcs.PLcs.SE

TL;DR

Subword tokenization follows frequency statistics rather than programming-language grammar, so semantically identical code can receive different tokenizations. TOKDRIFT tests the effects of those changes through semantics-preserving rewrites, finding substantial behavior shifts across code LLMs. The findings motivate research into tokenizers that better reflect program structure.

  • Problem

    Subword-token boundaries often diverge from programming-language grammar, raising the question of whether this misalignment limits LLMs’ ability to understand and generate code.

    Source: §1 · p. 1

  • Method

    TOKDRIFT applies semantic-preserving rewrites that alter tokenization and measures output behavior across nine code LLMs and three programming tasks.

    Source: §1 · p. 2

  • Results

    Across evaluated models, minor tokenization changes substantially affect behavior; Qwen2.5-Coder-32B-Instruct changes predictions 6.09% of the time, up to 60% under one rewrite rule.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings motivate future research on grammar-aware or domain-adaptive tokenizers that more faithfully reflect programming-language structure.

    Source: §7 · p. 8

    The study tests a targeted set of common formatting and naming rewrites, may not generalize to fundamentally different architectures or tokenization strategies, and does not explore mitigation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently depending on superficial factors such as whitespace or identifier naming. To measure the impact of this misalignment, we introduce TokDrift, a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. Across nine code LLMs, including large ones with over 30B parameters, even minor formatting changes can cause substantial shifts in model behavior. Layer-wise analysis shows that the issue originates in early embeddings, where subword segmentation fails to capture grammar token boundaries. Our findings identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation, highlighting the need for grammar-aware tokenization for future code LLMs.

1 Introduction

Subword tokenization can diverge from programming-language grammar, so semantically identical code variants may yield different predictions. TOKDRIFT measures this behavior through semantics-preserving rewrites.

1 Introduction

  • Subword-frequency tokens can split identifiers arbitrarily or combine unrelated symbols, unlike programming-language grammar tokens such as keywords, identifiers, and operators.Source: §1 · p. 1
  • A factorial-function example changes from an incorrect prediction naming it “comb” and later “combin” to a correct prediction.Source: §1 · p. 1
  • TOKDRIFT creates semantically equivalent but differently tokenized program pairs using rewrites such as whitespace changes or identifier casing, then evaluates output functional equivalence across nine models and three tasks.Source: §1 · p. 2
  • Minor tokenization variations substantially alter model behavior; Qwen2.5-Coder-32B-Instruct changes predictions 6.09% of the time, and up to 60% under one rewrite rule.The study reports that the effect persists across evaluated models and with scaling.Source: §1 · p. 2
  • Layer-wise analysis locates the effect in early layers, where subword segmentation fails to align with grammatical token boundaries.Source: §1 · p. 2
  • The paper formalizes misaligned tokenization as a problem for code LLMs and releases its framework and data for future research.Source: §1 · p. 2

2 Background

LLM tokenizers learn subword units statistically, while programming-language tokenization follows grammar. The resulting token boundaries can fragment semantically identical programs inconsistently.

2.1 LLM Tokenization

  • LLM tokenization converts raw text into discrete units, with learned merges based on corpus frequency; tokenizers also vary by pre-tokenization rules, learning algorithms, and training corpora.Source: §2.1 · p. 2
  • Figure 2’s vocabulary-distance comparison accompanies the finding that even same-family models such as Llama 3 and Llama 4 often share less than half their vocabulary.Source: §2.1 · p. 2, Figure 2 · p. 2

2.2 PL Tokenization

  • Programming-language tokenization (lexing) follows grammar, producing stable tokens that formatting changes do not alter when code remains syntactically valid; Table 1 lists the experiment benchmarks.Source: §2.2 · pp. 2–3, Table 1 · p. 3
  • LLM tokenizers can break code structure into inconsistent or fragmented representations even when programs are semantically identical.Source: §2.2 · p. 3

3 TOKDRIFT Framework

TOKDRIFT compares model outputs for original inputs and semantically rewritten variants. The framework specifies its benchmarks, models, rewrite rules, and measures of model sensitivity.

3 TOKDRIFT Framework

  • TOKDRIFT compares LLM outputs for baseline inputs and rewritten variants across experiments tied to a benchmark, model, and rewrite rule.The rewrites are designed to preserve semantics while altering tokenization.Source: §3 · p. 3, §3.3 · p. 3

3.1 Benchmarks

  • Eight benchmarks cover bug fixing, code summarization, and code translation in Java and Python; all use tests to evaluate output correctness.Benchmarks were selected for code inputs and automated functional-correctness evaluation.Source: §3.1 · p. 3

3.2 Models

  • Three instruction-tuned model families cover small (~1B), medium (~7B), and large (>30B) parameter sizes.The families are Llama-3, Qwen2.5-Coder, and DeepSeek-Coder; decoding is greedy.Source: §3.2 · p. 3

3.3 Rewrite Rules

  • Semantic-preserving rules are applied one at a time: N rules alter identifier casing, while S rules insert spaces between selected adjacent tokens, including wildcard cases.The spacing rules target frequent token-type combinations involving operators and identifiers.Source: §3.3 · pp. 3–4

3.4 Metrics

  • Accuracy is the share of correct outputs, while Δaccuracy subtracts baseline from variant accuracy; sensitivity measures correctness flips among changed inputs, with lower values indicating greater robustness.Sensitivity avoids cancellation between improvements and degradations and excludes samples unaffected by a rewrite.Source: §3.4 · p. 4

4 Evaluation

The evaluation finds that semantic-preserving rewrites produce accuracy and sensitivity shifts, with effects varying by rewrite, model size, and identifier fragmentation.

4.1 Results

  • -8.27% ∆accuracy occurred for Llama-8B on Java under S18, dropping accuracy from 43.15% to 34.88%; across models, rule effects ranged from -2.90 to +0.32 percentage points.The paper notes these deltas are non-negligible against performance advances sometimes claimed with around a 1 percentage point margin.Source: §4.1 · pp. 4–5, Table 4 · p. 5
  • 9.26% average sensitivity for naming rewrites and 8.29% for spacing rewrites, with wildcard rules S17 and S18 exceeding 10%.All models had sensitivity of at least 5.71%; Llama-3 models were more sensitive than the other model series on average.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 Impact of Model Size

  • Large models were usually less sensitive than smaller counterparts, except Qwen-32B on naming rewrites; size differences were significant for spacing but not naming rules.The spacing-rule differences were not significant between small and medium models for the Qwen2.5-Coder and DeepSeek-Coder series.Source: §4.2 · pp. 5–6, Table 5 · p. 6

4.3 Impact of Identifier Fragment Changes

  • Samples with changed identifier fragments showed higher sensitivity than unchanged samples, with naming-rule sensitivity of 10.82% versus 6.61%.An identifier fragment change means the token-fragment list, ignoring spaces and underscores, differs before and after a rewrite.Source: Table 6 · p. 6, §4.3 · p. 6

5 Root Cause Analyses

The root-cause analyses examine rewrite-side word frequencies and changes in model hidden states to investigate sensitivity to tokenization changes.

5 Root Cause Analyses

  • The analyses relate rewrite-side word frequencies to sensitivity and compare pre- and post-rewrite hidden states, using GitHub counts and Qwen-32B states.The hidden-state analysis extracts the last token’s state for each changed sequence; GitHub search limitations restrict frequency analysis to a subset of spacing rules.Source: §5 · p. 6, §5.1 · pp. 6–7, §5.2 · p. 7

5.1 Word Frequency Analysis

  • Every measured RHS/LHS word-frequency ratio is below 100%, and low-ratio rule S14 also exhibits high sensitivity.Table 7 reports the ratios; GitHub search was used to measure frequencies.Source: §5.1 · p. 7, Table 7 · p. 7

5.2 Hidden State Analysis

  • For S14 and S3 spacing rewrites, middle-layer hidden-state similarity is low, matching rules associated with high sensitivity.Similarity starts near zero at the input layer, generally rises and stabilizes in middle layers, then drops at the output layer.Source: §5.2 · p. 7, Figure 4 · p. 7
  • At the medium layer, t-SNE visualizations distinguish hidden-state differences for naming and spacing rewrites.The spacing-rule visualization excludes S17 and S18 because they are supersets of other rules.Source: §5.2 · p. 7, Figure 5 · p. 8

6 Related Work

Prior work explores tokenizers aligned with code structure, model robustness to representation changes, and syntax-aware generation. TokDrift measures effects of semantics-preserving tokenization changes in programming languages.

Tokenization

  • Tokenizer studies examine syntax-aligned tokenization and show tokenization choices can influence code understanding and generation [Liu et al. (2025); Chirkova and Troshin (2023)].One approach allows merges across whitespace boundaries; another aligns tokenization more closely with programming-language syntax while reducing token counts and preserving model performance.Source: §6 · p. 8

Robustness to Representation Variations

  • Robustness studies report both retained performance under unconventional tokenization and measurable drops or inconsistent outputs, including for semantically equivalent molecular representations [Zheng et al. (2025); Wang et al. (2025); Yan et al. (2025)].TokDrift focuses this line of inquiry on programming languages.Source: §6 · p. 8

Syntax-Aware Code Modeling

7 Conclusions

TokDrift measures how semantic-preserving tokenization changes affect code LLMs. Across nine models and three tasks, even minor formatting or naming changes can substantially shift outputs, motivating research into tokenizers that better reflect programming-language structure.

7 Conclusions

  • TokDrift applies semantic-preserving rewrites to show that minor whitespace or identifier changes can substantially shift outputs across nine code LLMs and three tasks.The tasks are code fixing, summarization, and translation.Source: §7 · p. 8
  • The findings motivate future research on grammar-aware or domain-adaptive tokenizers that better reflect programming-language structure.Source: §7 · p. 8

Limitations

The study limits its scope to selected semantic-preserving rewrites and nine evaluated code LLMs, and does not test mitigation strategies. The accompanying background describes tokenizer differences, while the authors disclose using an LLM-based writing assistant for polishing.

Limitations

  • The rewrites cover common formatting and naming conventions, not every source of tokenization drift; findings may not generalize to different architectures or tokenization strategies, and mitigation is untested.The authors suggest tokenizer retraining, ensemble decoding, or architectural modifications as future research directions.Source: Limitations · p. 9

A Use of LLMs

  • The authors used an LLM-based writing assistant to polish the paper and state that the ideas, analyses, experiments, and scientific claims are their own.Source: A Use of LLMs · p. 11

B Additional Background: Tokenizer Differences Between LLMs

  • The background compares vocabulary distances across 19 open-source model tokenizers from eight families and describes their differing pre-tokenization choices.Source: B Additional Background: Tokenizer… · p. 11
  • More aggressive pre-segmentation tends to tolerate superficial symbol spacing but can increase sensitivity to numeric boundaries; lightly pre-segmented designs make underscore and identifier edits more likely to alter token boundaries.Source: B Additional Background: Tokenizer… · p. 11

C.1 Benchmarks Normalization

The supplementary methods normalize benchmark inputs, apply semantics-preserving naming and spacing rewrites, and measure model accuracy and sensitivity to those rewrites.

C.1 Benchmarks Normalization

  • Benchmark normalization canonicalizes inputs and fixes compatibility or test issues while applying edits uniformly to baseline and variant inputs.For translation benchmarks, some unsalvageable or pathological samples were pruned, including six Avatar Python-to-Java tasks and four Java-to-Python tasks.Source: §C.1 · p. 11

C.2 Rewrite Algorithms

  • Naming rewrites convert eligible identifiers between casing styles, while spacing rewrites insert one space at configured adjacent code-token type pairs.The algorithms track edits and aim to preserve compilation and test behavior; HumanEval renamings are also propagated to tests, entry points, and declarations.Source: §C.2 · pp. 11–12

C.3 Metrics Computation Algorithm

  • Accuracy measures correctness on a set, while Δaccuracy compares a rewrite assignment with the baseline and sensitivity measures correctness flips among inputs actually affected by the rewrite.Δaccuracy can hide offsetting gains and losses, whereas sensitivity isolates changes on modified inputs.Source: §C.3 · p. 13

C.4 Experimental Environment

Experiments use deterministic decoding and a standardized evaluation harness across Java and Python, with model-specific precision and task-specific generation limits.

C.4 Experimental Environment

  • Experiments use an NVIDIA H100 cluster, the bigcode-evaluation-harness, deterministic decoding, batch size 1, Java 21.0.1, and Python 3.8.Most models run in fp32; three named 32B–70B models run in fp16.Source: §C.4 · p. 13

D Additional Results and Analysis

Additional analyses compare sensitivity across rewrite rules, fragment-change groups, and model families, and report how many samples each rule modifies.

D Additional Results and Analysis

  • Sensitivity varies between fragment-change groups across model families, with Llama generally more sensitive overall.Figures 7–12 show rule-level and model-series patterns, including naming and spacing transformations.Source: D Additional Results and Analysis · p. 13, Figure 7 · p. 15, Figure 8 · p. 16
  • Table 8 counts samples modified by each rewrite rule across model series, while Table 9 breaks sensitivity down by fragment-change category.The accompanying discussion cautions that Δaccuracy can obscure robustness patterns because gains and losses cancel and many samples are unaffected.Source: D Additional Results and Analysis · p. 14, Table 8 · p. 15, Table 9 · p. 17
  • Fragment classification compares old and new LLM-token boundaries: lost boundaries indicate merging, gained boundaries indicate splitting, and both indicate mixed change.The classification masks selected edit-site boundaries before comparing the boundary sets.Source: D Additional Results and Analysis · p. 16

Related papers

Loading 2510.14972v1…