Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

• Byte sequences are longer than subword sequences and lack explicit text abstractions, while specialized hierarchical architectures can make gains harder to attribute to joint tokenization.The paper retains a flat Transformer to investigate whether longer sequences help and how Transformers recover text abstractions.Source: §1 · p. 1
• Byte Transformers outperform subword Transformers as parameter counts scale; they achieve around 40% relative improvements on CUTE scores and 20% on OCRBench.The scaling result uses token-superposition training and hash embeddings.Source: §1 · pp. 1–2
• Copying probes find that byte Transformers read local context from segmentation-like positions, and restricting up to 25% of intermediate layers to those representations preserves downstream performance.The positions need not coincide with whitespace.Source: §1 · p. 2
• Byte-model uncertainty concentrates near learned local-structure boundaries, and speculative decoding yields 3.4× more accepted tokens than in subword Transformers.Many byte positions have near-zero loss, while substantially larger losses cluster near those boundaries.Source: §1 · p. 2
• The results frame sequence length, learned abstraction, and computation allocation as connected dimensions for language-model design.Source: §1 · p. 2

2.1 BUILDING BYTE MODELS

• BYTE models pair token-superposition training with hashed n-gram embeddings to address long byte sequences and limited local context.TST provides 1.9× total text exposure under the same training-token count; hash embeddings add nearby multi-byte pattern information to byte embeddings.Source: §2.1 · pp. 2–3

2.2 PARAMETER AND COMPUTE EFFICIENCY UNDER ISOFLOPS

• At equal optimal parameter counts BYTE has lower BPB, but at equal training compute SUBWORD has lower BPB; BYTE optima allocate 3.2× more source bytes per parameter.The passage reports that this allocation implies BYTE can reach lower BPB if training data continues to scale.Source: Figure 1 · p. 3, §2.2 · pp. 3–4

2.3 FINE-GRAINED CAPABILITIES IN TEXT-ONLY AND VISION–LANGUAGE TASKS

• BYTE consistently outperforms SUBWORD on fine-grained tasks across model scales, with a clear word-manipulation advantage; broader text-only tasks favor BYTE, while vision-language performance is on par.The table caption describes fine-grained task results across model scales; the broader task patterns are reported separately.Source: Table 1 · p. 4, §2.3 · p. 4

2.2 PARAMETER AND COMPUTE EFFICIENCY UNDER ISOFLOPS

• Combining TST and hash embeddings improves BPB over raw training in both model families, with larger combined gains for BYTE.These components are ablated at matched depth and training compute within each family.Source: §2.2 · p. 4

2.4 CONTROLLING COMPUTATION WITH SHARED DEPTH AND SPARSE EXPERTS

• Shared-depth SUBWORD closes most of BYTE’s BPB gap, while sparse BYTE MoE improves BPB with less activated computation and can reach comparable QA RC at 1B versus 3B dense SUBWORD.Looped SUBWORD also achieves consistently higher QA RC than standard dense BYTE across evaluated scales.Source: §2.4 · pp. 4–5, Figure 2 · p. 5

3.1 COPYING AS A PROBE OF LOCAL INFORMATION STRUCTURE

• During copying, multiple consecutive target bytes repeatedly attend to a small set of aggregated source positions, including when input spaces are removed.Attention concentrates near chunk boundaries, which need not be whitespace separators.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 TESTING LOCAL ABSTRACTIONS WITH A FROZEN LAYER INTERVENTION

• Whitespace may provide shortcuts through repeatedly appearing tokens, so separator ablation is relevant when interpreting the aggregation result.The paper points to Figure 9 for the separator ablation.Source: §3.2 · p. 5, §3.1 · p. 5
• The intervention compresses local spans to mean-pooled representations for selected intermediate layers, then returns to byte resolution without updating model parameters.A frozen, architecturally matched 200M BYTE model supplies the spans; the intervention compresses non-image text tokens by approximately 4×.Source: §3.2 · pp. 5–6
• With G = [21, 27], BYTE retains downstream performance near baseline while 7 of 28 layers (25%) operate on compressed local representations.The same intervention substantially degrades SUBWORD across nearly all layer intervals.Source: §3.2 · pp. 6–7, Figure 4 · p. 6

4.1 AN EMPIRICAL BPB DISTRIBUTION LAW

• The BPB variable is byte-weighted: each token’s score is weighted by the number of original text bytes it spans.The main comparison uses 1B BYTE and SUBWORD models and extends the analysis to public models.Source: §4.1 · p. 7
• BYTE BPB has more near-zero and high-BPB mass than SUBWORD; fitted near-zero exponents are −0.97 to −0.91 for byte models and −0.8 to −0.6 for subword models.The byte-weighted distribution law has a near-zero power-law region and an exponential tail over finite ranges.Source: §4.1 · pp. 7–8, Table 2 · p. 8
• Averaging BYTE losses over subword token spans closely matches the native SUBWORD distribution, indicating that token grouping largely explains the distribution gap.Subword grouping combines high-BPB bytes with predictable continuations into fewer tokens while preserving total source information.Source: §4.1 · p. 8
• BYTE takes multiple modeling steps within spans handled in one step by SUBWORD, retaining high-BPB bytes separately and allocating more computation to difficult tokens.This also adds modeling steps for the many low-BPB continuations.Source: §4 · p. 7, §4.1 · p. 8

4.2 DRAFTING LOW-BPB CONTINUATIONS

• 3.4× as many tokens per target forward pass are accepted by BYTE as by SUBWORD at both draft sizes, with nearly identical accepted bytes per pass for 200M drafts.With 50M drafts, accepted bytes per forward pass are slightly higher for BYTE; the 200M drafts yield longer accepted text per forward in both families.Source: §4.2 · p. 8, Table 3 · p. 9

Tokenizer-free language models

• Tokenizer-free models operate on bytes or characters; some use hierarchical mechanisms to compress or aggregate inputs, whereas this work keeps byte resolution throughout a Transformer with hybrid attention.Examples include ByT5, MambaByte, and hierarchical approaches such as BLT and H-Net.Source: §5 · p. 9

Compute scaling and allocation

• Compute scaling can vary through tokenization or architecture, including repeated depth for more computation and sparse MoE for more parameter capacity without proportional activated computation.The paper uses shared depth for SUBWORD and sparse experts for BYTE to examine the byte–subword computation trade-off.Source: §5 · p. 9
• Earlier work studies how token granularity changes data, sequence length, and compute; this paper examines that trade-off using shared depth and sparse experts.The comparison concerns computation at a given parameter capacity.Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

• Byte Transformers develop segmentation-like positions that aggregate local text information, and intermediate layers can use these representations without downstream performance loss.The authors connect these results to hierarchical architectures while noting that the hierarchy is learned implicitly rather than imposed.Source: §6 · p. 9
• The authors argue that sequence length, learned abstraction, and computation should be viewed as coupled design dimensions for byte language models.They identify adaptive computation and learned computational granularity as directions for future work.Source: §6 · p. 9
• The study is limited to vanilla Transformer backbones and evaluated training and task scales; generalization to other architectures, larger models, and broader modalities remains open.Source: §6 · p. 10

STATEMENT ON THE USE OF GENERATIVE AI

• Generative AI tools assisted experimental design, method implementation and debugging, training, and mathematical exposition.Source: STATEMENT ON THE USE OF GENERATIVE AI · p. 11
• The tools also assisted literature retrieval, figure and table preparation, and manuscript drafting and editing.Source: STATEMENT ON THE USE OF GENERATIVE AI · p. 11
• The authors take full responsibility for the final manuscript and associated artifacts, including AI-assisted text, code, analyses, and claims.Source: STATEMENT ON THE USE OF GENERATIVE AI · p. 11

ETHICS STATEMENT

• The authors state that, to their knowledge, the work raises no specific ethical concerns.Source: ETHICS STATEMENT · p. 11

A.1 TOKENIZERS

• BYTE uses direct UTF-8 bytes with 13 special-token IDs forbidden in valid UTF-8; SUBWORD uses a 49,152-entry tokenizer averaging about 4.56 bytes per token.Source: §A.1 · p. 16

A.2 TOKEN-SUPERPOSITION TRAINING (TST)

• TST averages embeddings in four-token bags and predicts the next bag; its first-30%-of-updates schedule yields 1.9× total text exposure at the same training-token count.The remaining 70% uses ordinary next-token training on the same parameters.Source: §A.2 · p. 16

A.3 HASHED EMBEDDINGS

• Hashed n-gram embeddings add local context to token embeddings, using four 32,768-slot tables for BYTE and one 131,072-slot table for SUBWORD.Both model families use 131,072 hash slots in total.Source: §A.3 · p. 16

A.4 DENSE MODEL CONFIGURATION; A.5 MOE CONFIGURATION

• Dense models scale a Qwen3.5 text architecture; MoE models activate eight of 64 routed experts and one shared expert per layer.The reported MoE active parameter count includes routers, selected routed experts, and shared experts, but excludes inactive routed experts.Source: §A.4 · pp. 16–17, §A.5 · p. 17, Table 5 · p. 17

A.5 MOE CONFIGURATION; A.6 LOOPED MODEL CONFIGURATION

• MoE load balancing updates expert-selection biases from batch assignment counts, while the looped SUBWORD control applies a shared decoder stack for four fixed passes.The MoE biases are frozen during the final 5% of training updates; the looped model trains and evaluates with the same recurrence.Source: §A.5 · p. 17, §A.6 · p. 18

B MAIN MODEL SUITE: TRAINING DETAILS; B.1 PRETRAINING HYPERPARAMETERS; B.2 TEXT SUPERVISED FINE-TUNING

• The main BYTE and SUBWORD suite uses ClimbMix with approximately matched source-text exposure; pretraining uses µP-style hyperparameter transfer and includes separate text SFT.The suite spans 200M–3B models; the SFT mixture includes Smol-SmolTalk, MMLU, GSM8K, and CUTE-style tasks.Source: B MAIN MODEL SUITE: TRAINING DETAILS · p. 18, §B.1 · pp. 18–19, Table 7 · p. 18

B.3 VISION–LANGUAGE TRAINING DETAILS; C.1 SWEEP SETTINGS

• Vision–language models share a frozen SigLIP2 encoder and projector architecture, trained in two stages; the IsoFLOPs sweep compares raw, hash-only, TST-only, and TST+hash recipes.The sweep adjusts update counts to each FLOP budget and presents results across evaluated model sizes and budgets.Source: §B.3 · p. 19, §C.1 · p. 19

C.2 PARAMETER, FLOP, AND DATA ACCOUNTING

• Training FLOPs per token account for full-attention and Gated DeltaNet recurrent operations, while excluding embedding lookups, hashing, bag averaging, loss evaluation, and optimizer updates.Source: §C.2 · p. 19

C.2 PARAMETER, FLOP, AND DATA ACCOUNTING; C.3 GLOBAL FIT AND COMPUTE-OPTIMAL ALLOCATION

• TST+hash lowers fitted optimal BPB in both model families at shared FLOP budgets, with larger gains for BYTE; at matched N, BYTE TST+hash improves over raw SUBWORD as source-byte exposure grows.The figure surfaces cover only sampled parameter–data regions.Source: Figure 6 · p. 20, Figure 7 · p. 21

C.2 PARAMETER, FLOP, AND DATA ACCOUNTING

• Source-byte exposure is V = sD, with s averaged over the training schedule; TST raises exposure at fixed training-token count while leaving per-token compute unchanged.Ordinary next-token training uses s = 1 for bytes and s = 4.561464 for subwords; four-token TST bags for the first 30% multiply either value by 1.9.Source: §C.2 · p. 20

C.3 GLOBAL FIT AND COMPUTE-OPTIMAL ALLOCATION

• The authors fit a loss law over per-token compute and source bytes, then derive compute-optimal allocations whose data-to-parameter ratio depends on learned exponents and architectural compute costs.Reported optima are clipped to each recipe’s observed size range before Pareto frontiers are formed.Source: §C.3 · pp. 20–21, Table 8 · p. 21

D.1 VALIDATION BITS PER BYTE

• Language-modeling BPB normalizes next-token cross-entropy by text bytes and is evaluated on a held-out perplexity validation set of 18,523 documents.Source: §D.1 · pp. 21–22

D.2 QUESTION ANSWERING (QA RC); D.3 CUTE: CHARACTER AND WORD MANIPULATION

• QA RC is a percentage macro-average across 21 task-family scores, while CUTE evaluates 19 spelling, containment, and character/word manipulation variants after text SFT.Task-level QA RC and CUTE scores are tabulated separately from the aggregate CUTE scores for the main models.Source: §D.2 · p. 22, Table 9 · p. 22, §D.3 · p. 22

D.4 VISION–LANGUAGE EVALUATION

• Vision–language differences remain task-dependent, but byte models generally gain relative ground with scale, surpassing subword models on several largest-model benchmarks.The largest dense BYTE models surpass SUBWORD on ScienceQA-IMG, MMBench, and POPE and nearly match it on GQA; the largest BYTE MoE model scores higher on several broader benchmarks.Source: §D.4 · p. 24, Table 13 · p. 25, Table 14 · p. 25

D.5 PARAMETER AND COMPUTE EFFICIENCY ACROSS MODEL FAMILIES

• Dense BYTE achieves lower BPB and higher QA RC than depth-matched dense SUBWORD, with greater training compute.The figure reports BPB and QA RC for base models and CUTE for SFT models.Source: Figure 8 · p. 26

E.1 INPUT-COPYING TASK SETTING

• The input-copying probe tests retrieval from an earlier occurrence of the same text using ten six-word phrases whose words each have five ASCII bytes.Local retrieval persists when word separators are changed from spaces to |, ;, or and.Source: §E.1 · p. 26, Figure 9 · p. 27

E.2 FROZEN LAYER INTERVENTION

• The frozen-layer intervention pools loss-defined local spans for selected intermediate layers, then maps global updates back while preserving byte-level states outside the compressed stage.The appendix evaluates downstream performance relative to uncompressed baselines across interval starts, ends, and lengths, without updating parameters or additional training.Source: §E.2 · pp. 27–28, Figure 10 · p. 28

F.1 SOURCE ENTROPY AND TOKEN GRANULARITY

• For uniquely decodable deterministic tokenizers, source entropy is invariant; byte prediction exposes intermediate subword prefixes, while model BPB also reflects approximation error and available context.The proof uses the one-to-one correspondence between each source sequence and its encoding under either tokenizer.Source: §F.1 · p. 29

F.2 EXPERIMENTAL SETTINGS FOR BPB DISTRIBUTION ANALYSIS

• Byte models show more near-zero and high-BPB mass than subword models in byte-weighted distributions.The comparison uses a 2 MiB subset of the evaluation dataset.Source: §F.2 · p. 29, Figure 11 · p. 30
• Finite-range fits find byte exponents nearer -1 and comparable tail scales across granularities; H-Net’s five formatting-character targets are excluded from its statistics because preprocessing may distort their BPB.The fit criteria require stable ranges and bounded global and local log-density errors.Source: §F.2 · p. 29, Table 15 · p. 30

F.3 BPB AGGREGATION ABLATIONS

• SUBWORD, Qwen3, and Qwen3.5 produce similar aggregated BPB distributions, while SUBWORD, Llama, Qwen3, and Qwen3.5 share high boundary Jaccard similarity.The shared boundaries group largely the same bytes together, helping explain the consistency of aggregation across tokenizers.Source: §F.3 · p. 30, Figure 14 · p. 32
• Uniform four-byte grouping leaves more near-zero mass and less far-tail mass than tokenizer grouping for both BYTE-1B and BLT-1B.Tokenizer grouping reduces variation, and the differences highlight the role of tokenizer boundaries in shaping aggregated BPB distributions.Source: §F.3 · p. 30, Figure 13 · p. 32

G.1 MODELS AND EVALUATION SETTINGS

• Drafts of 50M or 200M parameters are evaluated against same-family 1B targets on validation documents within each family’s native context window.The 200M drafts use 16 layers; the 50M drafts use 8-layer models.Source: §G.1 · p. 32, Figure 15 · p. 33

G.2 ORACLE SIMULATION AND ACCEPTANCE METRICS

• The oracle simulation measures consecutive draft–target agreements on given evaluation text under teacher forcing, accepting a draft token whenever both models select the same token—even if it differs from the reference.Each verification step ends at the first rejection or document boundary.Source: §G.2 · p. 33
• The evaluation reports accepted draft tokens per simulated target forward pass and accepted bytes per forward, including steps with no accepted tokens.Correction and bonus outputs are excluded, and special tokens contribute no bytes.Source: §G.2 · p. 33
• Unlike diagnostics based on generated histories or expected single-token overlap, this measure tracks consecutive agreements and their byte coverage on fixed reference text.The related approaches described use generated histories or expected single-token acceptance on fixed prefixes.Source: §G.2 · p. 33
• Both families have high acceptance rates in their lowest-BPB intervals, where a much larger share of BYTE tokens lies and most are accepted.The passage also reports longer byte acceptance lengths in Table 3 and presents small drafts as a proof of concept for supplying low-BPB continuations for joint target verification.Source: §G.2 · p. 33
• Teacher-forced agreement lengths are not a general upper bound on generated-history acceptance; practical speedups require finite-proposal decoding and measurement of drafting and verification costs.Generated histories may differ from the reference text, and speculative decoding also includes correction or bonus outputs.Source: §G.2 · p. 34