Source-linked AI summary
Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result
Christos Koutsiaris
TL;DR
The paper asks whether prefix-nested byte-level vocabularies can let one model serve multiple vocabulary sizes without sacrificing modelling quality. It trains and evaluates shared-cap models with pre-registered comparisons, then finds exact and practical slicing but a specialist-performance penalty, harmful output restriction, and robustness gains from multi-granularity training.
Problem
The paper examines whether one model can serve coarse and fine byte-level vocabularies instead of requiring separate models.
Method
It trains shared models across three prefix-nested caps while varying cap control tokens and output restriction in a pre-registered design.
Results
Exact slicing enables multi-cap deployment, but shared models trail specialists, output restriction costs 0.47% to 1.19% BPB, and multi-cap training improves robustness under corruption.
Takeaways & Limitations
Multi-granularity training provides robustness without conditioning machinery, while shared modelling and output restriction do not deliver the intended modelling benefit.
Takeaways & Limitations
The study uses small 3.1M- and 10.6M-parameter bodies, roughly one epoch of training, English-centric data, and one tokenizer and data mixture.
Abstract
from arXiv · showhide
A byte-level BPE tokenizer is an ordered list of merge rules, so applying only a prefix yields a vocabulary whose token identifiers are the first rows of the full vocabulary. This prefix nesting allows one language model to operate at several vocabulary sizes, use a control token to indicate the active size, and be deployed at any trained size by slicing its embedding and output head. We pre-registered five claims, including margins, seeds, contrasts, and a stop rule, and trained 30 models with 3.1M- and 10.6M-parameter bodies on 200M tokens each. Slicing is numerically exact: across 76 checks, a sliced model reproduces the restricted full model's logits bit for bit and removes 66% of deployed weights without changing latency. However, the shared model trails a fixed-cap specialist by 3.64% bits per byte at 32k against a 1% margin, and by 2.96% at 8k against a 2% margin. A 2x2 ablation separating the control token from output restriction finds that the token changes performance by +0.07% to +0.13%, with all intervals crossing zero, while output restriction costs +0.47% to +1.19%; the factors are substitutes rather than complements. Multi-cap training nevertheless improves robustness: under typographical noise, the same checkpoint degrades 12.5--15.4 points less in its fine mode and outperforms each fixed-cap specialist at that specialist's vocabulary size. A control with neither cap token nor output restriction is equally robust, attributing this benefit to multi-granularity training rather than conditioning. The per-cap penalty tracks each cap's share of training rows, yielding a falsifiable prediction for future work.
I. INTRODUCTION
Nested byte-level vocabularies make multi-size deployment structurally possible, but the paper asks whether sharing one model across caps is worth its modelling cost. The study pre-registers that question and deliberately prioritizes statistical resolution at small scales.
- Motivation: Systems needing both coarse and fine granularity typically train two models because vocabulary size trades sequence cost against character-level structure.Coarse vocabularies shorten sequences, while fine vocabularies expose orthography but lengthen them.
- Motivation: Byte-level BPE merge ordering makes different vocabulary sizes prefix-nested, allowing one embedding matrix to contain every cap.Applying only a merge-table prefix preserves token identifiers across caps.
- Motivation: Unlike hidden-width or representation nesting, vocabulary nesting already exists in the ordered merge table before model training.The paper tests whether exploiting this existing structure can be effectively free.
- Research question: The paper pre-registers five claims, fixing evaluation protocols, margins, and compute-time reporting before experiments.It explicitly reports outcomes that contradict the motivating intuitions.
- Research question: The experiments use 3.1M- and 10.6M-parameter bodies with 30 models, paired seeds, a crossed ablation, and bit-exact deployment checks.This design trades absolute performance for statistical resolution and does not establish behavior at eventual deployment scales.
II. METHOD
The method trains one byte-level BPE model with three truncated, prefix-nested caps and varies cap conditioning and output restriction in a crossed design.
- Nested vocabularies: A 32,768-rank byte-level BPE tokenizer is truncated to caps of 2,048, 8,192, and 32,768 merges.Preserved rank order means each smaller vocabulary is a prefix of the full vocabulary.
- Training design: Each training row samples a cap with probabilities 0.2, 0.3, and 0.5, records the cap seed, and optionally carries an informative control token.The output distribution can also be restricted to the active cap's lexical rows plus end-of-text.
- Training design: A 2 × 2 design independently varies whether the control token is informative and whether output restriction is applied.Paired seeds share initialization and data order, isolating the two factors.
C. Metric
The study evaluates language modeling with bits per byte on decontaminated held-out data, using two compact model tiers and a pre-registered stopping rule.
- Metric: Bits per byte is total target negative log-likelihood in bits divided by decoded target UTF-8 byte length.Perplexity is not compared across tokenizers because tokens per byte differ by construction.
- Data and models: Held-out sets are decontaminated from FineWeb-Edu by removing 6,473 documents sharing any 64-byte window, with zero shared windows remaining at stride 1.This establishes the held-out split before tokenizer training.
- Data and models: Models use 3.1M- and 10.6M-parameter decoder-only bodies trained on a fixed 200M-token budget.Embedding and output head are untied so nested input and output capacity can be sliced independently.
- Stopping rule: A pre-registered stop rule halted the larger tier after both initial shared-model seeds exceeded a 3% deficit against the coarse specialist.The factorial corners continued at the smaller tier to preserve interpretability.
IV. RESULTS
The shared model fails the specialist-equivalence target and output restriction harms BPB, while the crossed conditioning results reject the intended reinforcing mechanism.
- Specialist equivalence: 3.64% BPB at the 4× smaller body remains above the specialist margin, matching the larger-tier comparison to three digits.Both intervals lie wholly above their pre-registered margins, and the gap is far larger than specialist seed spread.
- Explicit conditioning: 0.47% to 1.19% BPB is the measured cost of restricting outputs to the active vocabulary, with every interval excluding zero.The worst cost occurs at the finest cap.
- Explicit conditioning: The control token changes performance by no measurable amount without output restriction, because every interval crosses zero.The point estimates also have the wrong sign for the preregistered claim.
- Explicit conditioning: The token and output restriction are substitutes rather than complements: their interaction is negative at all three caps.The token recovers part of the masking cost when restriction is already present, contrary to the expected reinforcement.
- Explicit conditioning: 0.29% to 1.09% worse than telling the model nothing is the full treatment's ordering relative to the blind control.The four corners order as blind < token-only < restriction-only < both.
C. Granularity dial (C4)
The granularity dial succeeds for typographical robustness but fails to deliver orthographic accuracy gains, while the control machinery contributes little to robustness.
- Robustness: 12.5–15.4 points less degradation under typographical corruption makes the shared checkpoint outperform specialists at their own caps.The improvement is reported for fine mode and is the strongest positive result.
- Orthographic accuracy: 8–11 accuracy points separate the shared model from specialists on orthographic accuracy.The dial is marginal at the smaller tier and absent at the larger tier.
- Attribution: A blind control without a cap token or output restriction matches the shared model’s robustness, indicating that multi-cap training supplies the benefit.The reported robustness values are +0.161 for the blind control and +0.154 for the shared model.
D. Deployment (C5)
Slicing preserves the restricted model exactly and substantially reduces deployed memory without changing latency, but smaller granularity increases tokenization cost and runtime.
- Exactness: 0.000e+00 worst-case logit difference across 76 checks establishes bit-identity between sliced and restricted full models.The comparison uses a pre-registered tolerance of 10−5.
- Mechanism: Slicing works because it selects embedding and output rows while restricted computation forms the same inner products.The deployment path therefore changes the artifact’s size without approximating its logits.
- Memory: 66% of weights disappear when reducing the artifact to cap 2,048, with latency unchanged within measurement noise.The result is reported for the tier S deployment setting.
- Throughput: 34% longer runtime at cap 2,048 accompanies 59% more tokens for the same fixed text than at cap 32,768.The trade-off is attributed to granularity and throughput rather than sharing versus specialist deployment.
V. DISCUSSION
The discussion attributes the per-cap penalty to data dilution: performance follows each cap’s share of training rows, motivating a fixed-per-cap-token experiment.
- A. The penalty is data dilution: 1.3333, 1.3818, and 1.4228 BPB correspond to cap shares 1.00, 0.50, and 0.33 at cap 32,768.The penalty increases monotonically as the cap receives less data.
- A. The penalty is data dilution: +6.72% versus +3.64% and +2.53% versus +4.89% show alternative mixtures perform according to row share rather than condition.The alternative mixture is worse for cap 32,768 when it receives fewer rows and better for cap 2,048 when it receives more.
- A. The penalty is data dilution: The control token and output restriction cannot recover the deficit because neither adds data, while restriction removes gradient signal from excluded rows.The passage identifies output restriction as an active cost.
- Future work: Holding per-cap tokens fixed rather than total tokens is predicted to shrink the gap toward zero at proportionally greater compute.This prediction is explicitly described as falsifiable and untested.
B. Two results of independent interest
The shared model’s robustness under typographical noise comes from training across granularities rather than conditioning labels, while output restriction is measurably harmful.
- 12.5–15.4 points less degradation under typographical noise lets the shared model outperform each specialist at its own vocabulary size.The robustness advantage appears in the fine mode and is unique in these experiments.
- The same robustness appears without a cap token or output restriction, attributing the benefit to multi-granularity training rather than conditioning.The blind control degrades 0.161 points versus 0.154 for the shared model.
- Training across caps is hypothesized, post hoc, to act as segmentation-noise augmentation because the model encounters the same text under multiple segmentations.The paper explicitly labels this mechanism as a hypothesis formed after seeing the results, not a pre-registered claim.
- Multi-granularity training makes the model degrade less under corruption, but it still trails specialists by 3.64% BPB on clean text and by 8–11 orthographic-accuracy points.The gain is a flatter response to corrupted input, not higher clean-text or orthographic performance.
D. Comparison with stochastic tokenization
Multi-cap training and BPE-dropout provide robustness at different costs: BPE-dropout is cheaper near the native cap, while multi-cap training wins farther away from it.
- 0.47 BPB is the multi-cap model’s lead over BPE-dropout at cap 2,048.At that cap, the plain specialist scores 2.3748 BPB, BPE-dropout 1.9271, and the multi-cap model 1.4609.
- 1.40% against the specialist is BPE-dropout’s native-cap cost, versus 3.44% for multi-cap training.Near the native cap, BPE-dropout is the cheaper robustness intervention.
- The preferred method crosses over with deployment distance: BPE-dropout suits one fixed granularity, whereas multi-cap training suits one artifact spanning granularities.The paper frames this as a trade-off rather than a universal winner.
- Prefill latency repeats to 0.2%, generated bytes per second to about 5%, while the isolated head matmul fails its repeatability tolerance.The head matmul is reported flagged rather than quoted.
VII. CONCLUSION
Nested vocabularies satisfy the engineering goal of exact, low-memory multi-cap deployment but not the modelling goal of matching fixed-cap specialists. Multi-cap training does improve robustness to input corruption without conditioning machinery.
- Bit-identical slicing across trained vocabulary sizes removes 66% of weights without changing latency, making multi-granularity deployment practical.The conclusion contrasts this engineering success with a modelling shortfall.
- 3.6% BPB is the shared model’s cost against a specialist, and the visible cap label recovers none of it.The conclusion also states that restricting the output distribution worsens performance.
- Under input corruption, the shared model beats every specialist, and the same robustness appears in a model told nothing.The benefit requires neither a cap token nor output restriction.
- The released repository contains code, configurations, pre-registration, results tables, and per-run manifests.The paper provides the repository URL for these materials.