PaperlayerView exact revision on arXiv

Source-linked AI summary

TokEval: A Tokenizer Evaluation Suite

Clara Meister

arXiv:2608.18062v1cs.CLcs.LG

TL;DR

Tokenizer selection is guided by limited intrinsic evaluation, while the links between tokenizer properties and downstream abilities remain unclear. TokEval introduces broader metrics and validates them in controlled pretraining experiments, finding that information-theoretic metrics predict language-modeling efficiency whereas structure-sensitive metrics predict task accuracy.

  • Problem

    Tokenizer design is typically evaluated with a small set of heuristics, while evidence linking tokenizer properties to downstream abilities remains limited.

  • Method

    TokEval combines intrinsic metrics spanning text, mathematics, code, and multilingual fairness with controlled pretraining experiments that vary tokenizer configurations while holding other variables constant.

  • Results

    Information-theoretic metrics significantly predict FLORES BPB, with Rényi efficiency reaching Spearman ρ = 0.80, while task-accuracy prediction is limited to structure-sensitive metrics.

  • Takeaways & Limitations

    Intrinsic metrics can screen tokenizers for encoding efficiency, but selecting among close candidates for task accuracy still requires training runs.

  • Takeaways & Limitations

    The main results use a single 1.27B-parameter decoder-only architecture, so tokenizer–performance relationships may differ at larger scales or under other architectures.

Abstract

from arXiv · show

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers' training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.

1 Introduction

TokEval addresses limited attention to tokenizer design by introducing a broader suite of intrinsic metrics and controlled pretraining experiments to examine their relationship with downstream performance. The framework targets text, mathematics, code, and multilingual fairness while supporting tokenizer visualization and integrity checks.

  • Motivation: Subword tokenization underpins nearly all contemporary large language models, yet tokenizer design receives limited attention despite its central role in model training and evaluation.Modern pipelines commonly use BPE or UnigramLM to map raw text into discrete token sequences.
  • Motivation: Testing tokenizer variants is expensive because each design choice requires reconfiguring the pipeline from text processing through output formatting.Intrinsic evaluation is performed on the tokenizer and corpus alone, without training a downstream model.
  • Motivation: Existing intrinsic metrics are narrow, and their relationship with downstream performance remains contested.Compression rate measures compression but not preservation of task-relevant structural properties; prior work linked Rényi efficiency to machine translation quality.
  • Contributions: TokEval introduces an open-source suite of intrinsic tokenizer metrics spanning text, math, code, and multilingual fairness, with visualizations and integrity checks.The paper also proposes novel mathematical and code metrics targeting specific downstream capabilities and evaluates tokenizer configurations through controlled pretraining experiments.

2 Background and Related Work

This section introduces tokenization as the mapping from raw symbols to finite-vocabulary tokens and reviews evidence that tokenizer choices affect model performance, cost, and cross-lingual fairness. It distinguishes intrinsic tokenizer metrics from extrinsic evaluations of models trained with those tokenizers.

  • Tokenization Algorithms: Tokenization maps raw text symbols to a sequence of tokens from a finite vocabulary, which are then associated with real-valued vectors.This positions tokenization as a foundational preprocessing step for machine learning models.
  • Impacts of Tokenization: Tokenizer choices significantly affect language-model downstream performance and training efficiency, while vocabulary inclusion can raise assigned string probability by up to 17×.The reviewed studies isolate tokenizer settings or compare subword vocabulary inclusion against representing strings with two tokens.
  • Impacts of Tokenization: Splitting natural words into multiple tokens can reduce model performance, while tokenization also creates systematic cross-lingual unfairness and unequal economic costs.These effects are described as a token tax on under-represented languages.
  • Intrinsic Tokenizer Evaluation: Intrinsic metrics evaluate a tokenizer using only the tokenizer and a text corpus, whereas extrinsic metrics measure effects on a model trained with that tokenizer.Bits-per-byte and task accuracy are examples of extrinsic evaluation measures.

3 The TokEval Framework

TokEval is a Python framework that computes intrinsic tokenizer metrics over user-specified, language-labeled corpora, reporting results per language and globally. Its metrics cover information-theoretic efficiency, fairness, round-trip fidelity, digit structure, and code syntax boundaries.

  • Framework and reporting: TokEval encodes each language-labeled corpus text, computes metrics per language, and aggregates them globally, with bytes as the default length unit.Users can choose bytes, characters, words, or lines depending on the analysis.
  • Information-theoretic metrics: Compression Rate uses total corpus length divided by total token count, avoiding the short-text bias of averaging per-text ratios.Unigram Entropy measures empirical token-frequency Shannon entropy, while Rényi Efficiency extends entropy to other orders through α.
  • Linguistic and fairness metrics: Fertility captures tokenization-induced cross-lingual variation, while MorphScore measures alignment between token and morpheme boundaries.The framework identifies cross-lingual fertility variation as an indicator of tokenization-induced unfairness.
  • Fidelity metrics: Round-trip fidelity is characterized by Reconstruction Exact Match and Character Error Rate, while UTF-8 Token Completeness below 1 can be valid under byte fallback.CER can exceed 1 when reconstruction is substantially longer than the original, such as through byte-fallback expansion.
  • Structure-sensitive metrics: TokEval introduces structure-sensitive metrics for arithmetic and code, including Three-digit Boundary Alignment for right-aligned digit grouping and AST Boundary Alignment for syntactic constructs.AST-based evaluation uses parsed source code to assess whether token boundaries respect grammar-derived regions.

4 Experimental Setup

The experiments train 1.27B-parameter models across 46 tokenizer configurations while holding architecture, training data, and hyperparameters constant within each experimental line. Tokenizers vary across algorithm, pretokenization, and training-data composition, and models are evaluated on language, mathematical, and code-oriented outcomes.

  • Experimental design: 46 tokenizer configurations are compared using 1.27B-parameter models, including 44 custom tokenizers and Mistral-Nemo and LLaMA-3 references.Each experimental line holds architecture, model training data, and hyperparameters constant, making the tokenizer the sole experimental variable.
  • Tokenizer variation: Three tokenizer design axes are ablated while vocabulary size remains fixed at ∼128K entries: algorithm, normalization/pretokenization, and training-data composition.Different analyses use overlapping tokenizer–model subsets organized into families sharing the three design-axis values.
  • Tokenizer variation: Five tokenizer algorithms are tested: BPE, UnigramLM, SuperBPE, parity-aware BPE, and MinGram.BPE and UnigramLM use HuggingFace tokenizers, while the other three use their authors’ published codebases.
  • Tokenizer variation: The main pretokenization strategies include a punctuation-only baseline and GPT-4o regex, whose rules determine which text spans can become learned vocabulary tokens.Pretokenization is treated as an influential design choice because it splits text before vocabulary learning.
  • Evaluation: Models are evaluated with BPB on FLORES+ across 215 languages and the 31-language training subset, plus BLiMP and MultiBLiMP agreement benchmarks.The supplied setup passage specifies these evaluations for natural-language-focused models.

5 Results

Results show that intrinsic tokenizer metrics can predict downstream performance, but their relationships depend on the task, modeling specification, and measurement corpus. Information-theoretic metrics are strongest for language-modeling outcomes, while structure-sensitive metrics and fertility reveal task- and language-dependent effects.

  • Aggregate intrinsic–downstream correlations: Rényi efficiency has the strongest aggregate correlation with FLORES BPB, with Spearman |ρ| up to 0.80 across five significant information-theoretic predictors.Across trained languages, significant information-theoretic correlations range from |ρ| = 0.49 for unigram entropy to |ρ| = 0.80 for Rényi efficiency.
  • Dependence on the measurement corpus: Changing the measurement corpus alters the significance status of 7 of 98 aggregate correlations, while 91 retain their status.Several nonsignificant coefficients move or change sign; five of the seven status changes involve metrics dependent on the differing length unit.
  • Fertility and FLORES(all) BPB: Fertility has opposite associations across FLORES outcomes: its coefficient is +0.018 BPB per SD for FLORES(trained) BPB, where greater fragmentation predicts worse BPB.The passage states that fragmentation’s direction differs depending on which languages’ BPB is measured.
  • Within-language predictors of FLORES BPB: Within-language mixed-effects analysis preserves strong information-theoretic predictiveness and additionally identifies UTF-8 character split rate and fertility as significant FLORES BPB predictors.The analysis controls for per-language complexity with a random intercept, changing the predictiveness of several metrics relative to aggregate correlations.
  • Within-language predictors of FLORES BPB: Compression rate is not a significant predictor of MultiBLiMP accuracy, whereas five of nine significant metrics retain the same direction as their FLORES BPB effects.This supports the stated implication that language and the corpus used to measure an intrinsic metric affect its downstream relationship.

6 Discussion and Conclusion

TokEval reveals that intrinsic metrics have distinct predictive roles: information-theoretic and compression-based measures help optimize encoding efficiency, while task accuracy requires additional structure-sensitive evidence. The conclusions are bounded by protocol choices and experiments using one relatively small architecture.

  • What intrinsic metrics can and cannot predict: Five information-theoretic or compression-based metrics significantly predict FLORES(trained) BPB, making them useful for narrowing tokenizer searches before pretraining.These metrics are not a sole proxy for task accuracy.
  • What intrinsic metrics can and cannot predict: Fertility’s coefficient is not significant when using clustered standard errors.This qualification is reported in §C.1.2.
  • How to measure: Aggregate multilingual correlations can mask or reverse within-language relationships because of baseline differences across languages.The discussion identifies fertility as an illustration and motivates language-specific tokenizer sub-vocabularies.
  • Limitations: The main results use a single nanochat decoder-only transformer architecture with 1.27B parameters.Tokenizer–performance relationships may differ at larger scales or under other architectures, while the models may be too small for some abilities to emerge.
  • Conclusion: TokEval is an open-source suite spanning text, mathematics, code, and multilingual fairness, validated through controlled pretraining across tokenizer configurations and two data mixtures.The conclusion proposes intrinsic measurement as a replacement for pretraining sweeps where the two agree.

Ethics Statement … A.2 Training Data: Ablation Configurations

TokEval addresses tokenizer inequities affecting low-resource languages and non-Latin scripts while documenting pretokenization choices and multilingual training-data ablations. These configurations determine token boundaries, vocabulary granularity, and which languages tokenizer training represents.

  • Ethics Statement: Tokenizer choices can increase inference costs and degrade performance for low-resource languages and non-Latin scripts, motivating disparity measures such as fairness Gini and per-language fertility.The stated goal is to help practitioners identify and mitigate these inequities.
  • A.1 Pretokenization Configurations: Pretokenization deterministically splits raw text before BPE or UnigramLM, preventing learned merges or segmentation candidates from crossing pretoken boundaries.Thus, a token can be at most one pretoken long, and pretokenization sets the maximum vocabulary granularity.
  • A.1 Pretokenization Configurations: The Punctuation strategy isolates punctuation characters and otherwise uses the GPT-2 byte-level encoder’s default regex for split points.It is implemented as Sequence([Punctuation(behavior="Isolated"), ByteLevel(use_regex=True)]).
  • A.1 Pretokenization Configurations: GPT-4o uses CamelCase-aware splitting, left-aligned 3-digit groups, contraction handling, and preceding-punctuation attachment, whereas Claude separates whitespace types and omits that attachment.The Claude regex is only an approximation inferred from Claude API token counts, not a definitive reconstruction.
  • A.1 Pretokenization Configurations: Right-aligned digits group digit spans by place value, tokenizing “123456” as [“123”, “456”] rather than the left-aligned [“12”, “345”, “6”].Its lookahead-based regex aligns groups to thousands boundaries.
  • A.1 Pretokenization Configurations: Clean avoids camelCase splitting and explicit contraction attachment while making each digit its own pretoken; plus2 additionally attaches leading ASCII or curly apostrophes.These are multilingual-oriented alternatives to the other pretokenizers.
  • A Tokenizer Training and Configuration Details: The tokenizer panels group configurations sharing three design-axis values while varying only a training hyperparameter or preset, with family membership recorded in Table 5.This defines the comparison structure for the configuration panels.
  • A.1 Pretokenization Configurations: plus3 extends plus2 by attaching trailing apostrophes not followed by letters, targeting French, Italian, Catalan, and Maltese contraction patterns.Examples include l’arbre and Maltese ta’.

A.3 Full Tokenizer Panel Specification · B Model Architecture and Training Details · B.1 Architecture

The tokenizer panel spans 46 configurations varying algorithm, pretokenization, data, NFC handling, and evaluation panels, while the models use a fixed nanochat d24 architecture with structured attention and value embeddings. Under a ∼128K vocabulary, this architecture totals 1.27B parameters.

  • A.3 Full Tokenizer Panel Specification: 46 tokenizer configurations comprise 44 trained tokenizers and 2 off-the-shelf references, varying algorithm, pretokenization, data mixture, NFC handling, and panel membership.The panel includes BPE, Unigram, SuperBPE, PA-BPE, and MinGram variants, alongside controls and multilingual-data configurations.
  • A.3 Full Tokenizer Panel Specification: The panel includes GPT-4o, GPT-2, Claude, punctuation, right-aligned, SCRIPT-encoding, clean-multibyte, and whitespace-only pretokenization strategies.These strategies are represented through the panel’s Pretok codes and associated tokenizer variants.
  • B.1 Architecture: The model uses nanochat d24 with 24 layers, hidden dimension 1536, 12 attention heads, head dimension 128, and context length 2048.It uses RoPE with θ = 100,000, squared-ReLU activations, parameterless RMSNorm, QK-norm scale 1.2, logit soft-capping, and Flash Attention.
  • B.1 Architecture: The sliding-window “SSSL” pattern assigns three layers local attention over ⌈L/4⌉ tokens and one layer full-context attention.Alternating layers add gated value embeddings, while learnable scalars control residual scaling and skip-connection blending.
  • B.1 Architecture: 1.27B parameters comprise 682M transformer matrices, 197M input embeddings, 197M output projection, 197M value embeddings, and 74 scalar parameters.These counts assume a ∼128K vocabulary; embedding parameters therefore scale with vocabulary size.
  • B.1 Architecture: Mistral-Nemo’s 131,072-token vocabulary adds 12,976,128 parameters, equal to 1.0% of the 1.27B total.The other off-the-shelf tokenizer uses a 128,256-token vocabulary with no parameter-count difference from the trained model weights.

B.2 Training Hyperparameters … C.1.1 Tokenizer Panel Inclusion Criteria

The experiments fix detailed optimization and data-mixture settings, evaluate models with tokenizer-comparable and task-specific procedures, and construct tokenizer panels using explicit eligibility, family, and independence criteria. The primary panel contains 29 near-independent representatives, while the extended panel contains 39 family members and excludes two held-out reference tokenizers.

  • B.2 Training Hyperparameters: Natural-language-focused models train for 9.23B tokens, using a fixed budget derived from 10.5 × Nscale with Nscale = 879M.The budget corresponds to 8,800 steps at batch size 220 = 1,048,576 and was held fixed after a five-tokenizer continuation check.
  • B.2 Training Hyperparameters: Optimization uses Muon for weight matrices and AdamW for embeddings, projections, and scalars, with µP-transferred learning rates and gradient accumulation of 8 micro-batches.Base learning rates are 0.02 for Muon, 0.30 for input embeddings, 0.008 for output projection, and 0.50 for scalar parameters; training uses torch.compile(dynamic=False).
  • B.2 Training Hyperparameters: The training schedule warms up linearly for 40 steps, stays constant through 35% of training, then decays linearly to 5% of peak over the remaining 65%.Muon momentum changes from 0.85→0.97 over 400 steps and later decays to 0.90; weight decay is 0.050 with cosine decay to zero.
  • B.3.1 Natural Language-Focused Track: The natural-language-focused mixture contains 36.9% English web, 33.4% multilingual data, 16% math, and 13.6% code, measured using text bytes.The multilingual component spans 30 languages and 11 scripts, with a 266× scarcity gradient from Russian to Tamil.
  • B.3.2 Math+Code-Focused Track: The math+code mixture draws math from MegaMath-Web-Pro and code from The Stack v2’s educational subset, covering seven proportionally weighted programming languages.The listed code languages are JavaScript, Java, Python, C++, TypeScript, Go, and Rust, totaling approximately 50 GB and 13M documents.
  • C.1 Downstream Evaluation Configurations: BLiMP and MultiBLiMP use empty-context minimal-pair loglikelihood scoring, while GSM8K, HumanEval, and MBPP use specified few-shot or zero-shot generation protocols.GSM8K uses 8-shot chain-of-thought and exact match; HumanEval uses greedy 0-shot pass@1; MBPP uses greedy 3-shot pass@1.
  • C.1 Downstream Evaluation Configurations: BPB divides negative loglikelihood by UTF-8 byte length, making language-model evaluation comparable across tokenizers despite differing vocabularies and compression rates.The formula is BPB = ∑i NLLi/(ln 2 · B), where ln 2 converts nats to bits.

C.1.2 Statistical Testing Protocol

The protocol accounts for tokenizer-family dependence through family-level inference and clustered per-language uncertainty, while applying prespecified multiplicity corrections. Robustness checks, held-out family folds, and alternative reference distributions assess how stable the reported associations are.

  • Dependence-aware inference: Tokenizer families, rather than individual near-duplicate tokenizers, define the dependence-aware unit for extended inference and held-out cross-validation folds.Extended analyses average each family before ranking, while held-out folds exclude an entire family from training and scoring.
  • Dependence-aware inference: B = 10,000 fixed-seed reshuffles assess extended-panel significance by permuting outcomes among intact family means.The permutation p-value is the fraction of reshuffles whose absolute correlation reaches the observed value.
  • Per-language regressions: Per-language β estimates use mixed-effects fits, while standard errors and significance tests come from language-fixed-effects regressions clustered on tokenizer.Clustering allows residuals contributed by one tokenizer to remain dependent across languages.
  • Per-language regressions: 1.05 to 2.77 clustering factors change one cell from adjusted p = 0.00026 unclustered to 0.063 clustered: fertility against MultiBLiMP accuracy.Six additional cells retain significance only at a weaker star level.
  • Multiple comparisons: Benjamini–Hochberg correction is applied within prespecified metric-outcome families, while frozen membership and leave-one-out or design-axis removal checks provide robustness disclosures.The protocol reports results regardless of direction and labels leverage-removal analyses as diagnostics after the full-panel result.
  • Multiple comparisons: ρ = 0.61 and ρ = 0.65 report AST boundary alignment against MBPP pass@1 on nested primary and full math+code panels, respectively.The values describe the same relationship across the 19-member and 20-member rosters.

D Intrinsic Evaluation Results … E.2 Language Model Evaluation Results (Pilot-Scale; 300M)

The intrinsic results cover nearly all trained tokenizers and two off-the-shelf references, while downstream results report absolute benchmark scores for every tokenizer. Pilot-scale evaluation additionally measures validation, FLORES, and code BPB for a subset of initial-panel tokenizers.

  • D Intrinsic Evaluation Results: Tables 10 and 11 report intrinsic metric values for 43 of 44 trained tokenizers and two off-the-shelf references.Measurements use FLORES+ and FineWeb-Edu/FineWeb2, respectively, with length-unit conventions specified in the captions.
  • D Intrinsic Evaluation Results: Unigram Whitespace tuned is absent from both intrinsic tables because it has no intrinsic measurement and serves as a minimal-structure control.It belongs to no panel, so no correlation or ranking depends on it.
  • E.1 Language Model Evaluation Results (Full-Scale; 1.27B): Table 12 reports absolute downstream benchmark scores for every tokenizer named in the paper.The evaluated set is a superset of P and includes extended-panel variants, defective English-only retrains, the file-cap null control, and two minimal-structure controls.
  • E.1 Language Model Evaluation Results (Full-Scale; 1.27B): Table 5 provides each tokenizer row’s panel membership and configuration.These details contextualize the broader tokenizer set evaluated in Table 12.
  • E Language Model Results: The language-model results include full-scale 1.27B-parameter evaluations reported in Table 12 for every tokenizer named in the paper.The table’s tokenizer set includes controls and retrain variants beyond the main panel.
  • E.2 Language Model Evaluation Results (Pilot-Scale; 300M): 300M-parameter pilot models report validation BPB, FLORES BPB, and code BPB in Table 13.They use nanochat d16, 3.5B tokens, and the same 10.5× scaling ratio as the nanochat library.
  • E.2 Language Model Evaluation Results (Pilot-Scale; 300M): The pilot models cover only a subset of tokenizers, particularly those included in the initial panel.Their purpose is to provide a fast iteration loop between the 300M and 1.27B scales.

E.3 Ranking Stability Across Training Scales … F.1 Held-Out Prediction of the Reference Tokenizers

Tokenizer rankings are fairly stable across training scales but shift notably at the largest scale, while held-out reference prediction shows mixed calibration and training-mixture-specific intrinsic predictors. Cross-model code-BPB comparisons require rank-based interpretation because the underlying data mixtures differ.

  • E.3 Ranking Stability Across Training Scales: Adjacent-scale val BPB rank correlations are 0.882, 0.922, and 0.752, with full-versus-smallest-scale τ = 0.765 across the four training scales.The lowest adjacent-scale agreement occurs between d16 (596M) and d24 (1.27B).
  • E.3 Ranking Stability Across Training Scales: The final scale doubling changes roughly one tokenizer pair in eight, cautioning against using smaller-model results for larger-model tokenizer development.Rank agreement does not tighten monotonically as model scale grows.
  • E.3 Ranking Stability Across Training Scales: BLiMP is compared only within a single scale because unnormalized total-log-likelihood scoring lowers 1.27B aggregate accuracy relative to 300M pilots through collapsed quantifier-subtask accuracy.This cross-scale caveat is specific to the stated BLiMP scoring procedure.
  • E.4 Cross-Model Rank Correlation of Code BPB: Code-BPB levels across natural-language and math+code model families are not comparable because their training data mixtures differ; only per-tokenizer rank transfer is meaningful.The cross-family comparison retains all 25 eligible tokenizers, including family variants and labeled controls.
  • F.1 Held-Out Prediction of the Reference Tokenizers: Held-out prediction fits use leave-one-family-out validation to predict Mistral-Nemo and LLaMA-3, which share architecture and training data with the panel but were excluded from every fit.The evaluation covers val BPB, trained-31 FLORES BPB, code BPB, and MBPP pass@1.
  • F Additional Analyses: 6 of 8 univariate reference predictions fall inside their 90% intervals, whereas multivariate fits improve in-sample R2 and CV-RMSE but place 4 of 8 predictions outside.The two univariate exceptions are LLaMA-3 on trained FLORES BPB (+2.10 standardized errors) and math+code code BPB (−2.01).
  • F.1 Held-Out Prediction of the Reference Tokenizers: Predictive intrinsic metrics differ by training mixture: natural-language BPB favors identifier-based measures, while math+code code BPB favors FineWeb Gini rather than entropy metrics.The math+code association appears under FineWeb measurement, not FLORES+ measurement.

F.2 Intrinsic–Downstream Correlations Robustness Checks

Robustness checks largely preserve the reported intrinsic–downstream correlations when controlling for tokenizer-family dependence and testing individual-tokenizer influence. However, group-based exclusions and joint multiple-testing correction reveal sensitivity, especially for AST alignment and several digit-boundary correlations.

  • Extended-panel correction: 31 tokenizer families from the 39-tokenizer extended panel were collapsed to family means before Spearman correlations, with significance assessed by permutation tests.This prevents same-family tokenizers from counting as independent evidence; adjusted p-values are not directly comparable to Table 2’s.
  • Extended-panel correction: padj = 0.016 and 0.029: Rényi efficiency and digit-boundary-F1 correlations remained significant, while unigram entropy narrowly missed significance at padj = 0.062.Four of five information-theoretic correlations with FLORES(trained) BPB remained significant under the extended-level check.
  • Single-tokenizer sensitivity: Removing Unigram Claude pushed three of four correlations above the jointly corrected significance threshold, while 12 of 29 removals affected at least one correlation.The affected correlations were digit boundary F1 with BLiMP, digit boundary F1 with Code BPB, and AST alignment with MBPP.
  • Axis-based sensitivity: The AST-alignment/MBPP correlation reversed sign and became small and nonsignificant after removing tokenizer groups defined by non-default algorithm, pretokenization, or training-data composition.Three other correlations weakened but retained their sign under these axis-based removals.

F.3 Math and Code Generation (math+code Models) · F.4 External-Model Validation of Per-Example Tokenization Metrics

In math+code models, code-generation performance varies substantially across tokenizers, with line-break handling and AST boundary alignment associated with MBPP success, while external per-example tokenization metrics show no significant predictive relationship. These results indicate that tokenizer-metric associations depend on the evaluation setting and training distribution.

  • F.3 Math and Code Generation (math+code Models): MBPP pass@1 ranges from 0.000 to 0.250 across 20 custom tokenizers, whereas HumanEval pass@1 remains between 0.134 and 0.177.The models use identical training data, architecture, and budget.
  • F.3 Math and Code Generation (math+code Models): 1,024 fused punctuation-line-break tokens occur in the GPT-4o BPE vocabulary, while punct, Claude, and clean-multi vocabularies contain none.The GPT-4o vocabulary has 1,091 tokens containing a line break overall, including examples such as )\n and ;\n.
  • F.3 Math and Code Generation (math+code Models): ρ = +0.593 links the fraction of MBPP generations containing a line break with MBPP pass@1, with p = 5.8 × 10−3.Models with many fused punctuation-line-break tokens emit fewer line breaks, and MBPP evaluates Python, where line breaks are syntactic.
  • F.3 Math and Code Generation (math+code Models): ρ = +0.65 is the significant correlation between AST boundary alignment and MBPP pass@1, the only significant code-structure correlation on the 20-tokenizer roster.The result has padj = 0.036 and n = 20, but the passage cautions that it is not separable from line-break handling.
  • F.3 Math and Code Generation (math+code Models): Operator isolation correlates positively with MBPP pass@1 in prose, code, and math at ρ = 0.52, 0.34, and 0.28, respectively, but none survives BH correction.The smallest adjusted p-value is 0.058 for the prose corpus, across n = 20 custom tokenizers.
  • F.4 External-Model Validation of Per-Example Tokenization Metrics: The external validation covers five independently trained code models, five tokenizer classes, vocabularies from 49,152 to 256,000 tokens, HumanEval’s 164 problems, and MBPP’s 500 problems.Per-example metrics are computed on prompts and reference solutions and analyzed with problem-stratified conditional logistic regression, alongside a Bayesian mixed-model check.
  • F.4 External-Model Validation of Per-Example Tokenization Metrics: No per-example AST-boundary or identifier-fragmentation metric is FDR-significant for either task on the external five-model panel or the internal 20-tokenizer panel.The smallest within-task BH-adjusted p-values are 0.219 externally and 0.087 internally.
Loading 2608.18062v1…