PaperlayerView exact revision on arXiv

Source-linked AI summary

Byte Latent Transformer: Patches Scale Better Than Tokens

Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srinivasan Iyer

arXiv:2412.09871v1cs.CL

TL;DR

Large language models traditionally rely on fixed-vocabulary tokenization, while direct byte-level training has been costly at scale. BLT addresses this by dynamically grouping bytes into entropy-based patches and processing them with a byte-and-latent architecture. At scales up to 8B parameters and 4T bytes, it matches tokenization-based performance, improves robustness and efficiency, and enables better fixed-inference-budget scaling.

  • Problem

    Direct byte-level training has been prohibitively costly at scale because large feed-forward layers would run on every byte.

  • Method

    BLT dynamically groups bytes into patches using next-byte entropy and processes them with lightweight byte-level modules plus a large latent transformer.

  • Results

    BLT achieves training flop-controlled parity with Llama 3 up to 8B scale and can trade minor evaluation losses for up to 50% lower inference FLOPs.

  • Takeaways & Limitations

    BLT enables model size and patch size to grow together under a fixed inference budget while improving robustness to input noise and byte-level awareness.

  • Takeaways & Limitations

    The study often uses scaling laws derived for BPE transformers, which may yield suboptimal data-to-parameter ratios for BLT.

Abstract

from arXiv · show

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.

1 Introduction

BLT replaces fixed-vocabulary tokenization with dynamically sized byte patches, allocating computation according to prediction complexity. At scale, it matches token-based performance while improving inference efficiency, robustness, and scaling flexibility.

  • BLT is a tokenizer-free byte-level architecture that learns directly from raw bytes without a fixed vocabulary.
  • Tokenization remains costly at scale because large feed-forward layers run on every byte when models are trained directly on byte sequences.
  • BLT segments bytes by next-byte entropy, assigning more computation to difficult predictions and less to predictable continuations.
  • Up to 8B parameters and 4T training bytes, BLT matches Llama 3 on training flop-controlled performance and can use up to 50% fewer inference flops.
  • BLT improves robustness to noisy inputs and supports character-level abilities in orthographic knowledge, phonology, and low-resource machine translation.
  • BLT enables simultaneous growth of model and patch size under a fixed inference-flop budget, producing better scaling trends than tokenization-based architectures.

2 Patching: From Individual Bytes to Groups of Bytes

BLT dynamically groups bytes into patches using next-byte entropy, so expensive latent-transformer computation can follow contextual prediction difficulty rather than fixed token or stride boundaries.

  • BLT combines lightweight byte-level encoders and decoders with a large latent transformer operating on patch representations.
  • Patches are the primary units of expensive transformer computation, so their number directly determines most of the FLOP cost.
  • A global threshold ends patches when H(x_i) exceeds θ_g, while an approximate monotonic constraint detects increases relative to the previous entropy.
  • Entropy patching uses next-byte uncertainty to place boundaries, concentrating computation near high-entropy predictions and grouping low-entropy continuations.
  • Unlike fixed-vocabulary tokens, patches are dynamically grouped byte sequences, preserving direct access to underlying byte features.
  • Incremental patching requires boundary decisions to use only the already generated prefix, so future bytes cannot determine segmentation.

3 BLT Architecture

BLT combines a large global transformer operating on latent patch representations with lightweight local models that encode bytes into patches and decode patches back into bytes. Its patch-dependent cross-attention and byte-level components support computation over dynamically grouped raw bytes.

  • Global and local models: BLT uses a large global autoregressive transformer over patch representations and two smaller local models for byte-to-patch encoding and patch-to-byte decoding.The global model consumes most pretraining and inference FLOPs, so patch selection controls when it is invoked.
  • Local encoder: The local encoder maps input bytes into patch representations through lightweight transformer layers and cross-attention that pools byte information into patches.Its byte-level attention uses a local block-causal mask and can cross dynamic patch boundaries without crossing document boundaries.
  • Byte representations: Hash n-gram embeddings augment byte embeddings with preceding byte context before the representations enter the local encoder.The paper uses n-gram sizes from 3 through 8 and studies their effects in ablations.
  • Encoder cross-attention: BLT represents patches as variable latent representations rather than a fixed set of latent vectors, with each patch attending only to its constituent bytes.Patch queries are initialized by pooling the corresponding byte representations and applying a linear projection.
  • Local decoder: The local decoder is a lightweight transformer that uses reversed cross-attention roles to decode global patch representations into raw bytes.Byte representations serve as queries, patch representations as keys and values, and decoder transformer layers operate on the resulting byte sequence.

4 Experimental Setup

The experiments evaluate BLT and tokenization-based baselines across controlled compute, scaling, and architectural comparisons. They use matched data and context-related controls while measuring tokenizer-independent bits-per-byte and model FLOPs across multiple scales.

  • Data: The models are pretrained on the Llama 2 dataset and BLT-1T, a dataset containing one trillion tokens from public sources and Datacomp-LM data.The Llama 2 dataset contains two trillion collected tokens that were cleaned and filtered.
  • Patching controls: Entropy-based patching uses a threshold calibrated to a desired average patch size while keeping the expected number of bytes per batch constant.Training packs patches to keep the number of patches per batch constant and pads or truncates byte sequences to control memory spikes.
  • Compute accounting: FLOP accounting includes local encoder, global latent transformer, local decoder, and encoder and decoder cross-attention computations.The calculation adjusts attention FLOPs for whether each component operates on byte or patch sequences and uses k = 2 for the patch-to-byte dimension ratio in Figure 5.
  • Metrics: The study reports bits-per-byte rather than perplexity because bits-per-byte is tokenizer-independent and can compare byte-level with token-level models.Bits-per-byte normalizes cross-entropy over the data by the total number of bytes and a constant.
  • Evaluation design: Experiments compare BLT with tokenization-based models through controlled studies of scaling trends and downstream task performance at 400M, 1B, 2B, 4B, and 8B parameters.The study reports architectural hyperparameters and uses max-pooling for the first local-encoder cross-attention queries.

5 Scaling Trends

BLT matches or exceeds BPE scaling trends at compute-optimal training and achieves better fixed-inference scaling by increasing patch and model size together. Larger patches reduce inference cost, allowing compute to shift toward a larger global transformer.

  • Parameter-Matched Compute-Optimal Scaling: BLT models match or outperform BPE counterparts as model size and training FLOPs scale.This holds in the compute-optimal regime and validates comparable scaling trends with BPE-based models.
  • Parameter-Matched Compute-Optimal Scaling: BLT matches state-of-the-art tokenizer-based models at scale through architectural changes combined with dynamic patching.Space patching alone remains far behind Llama 3, whereas the full BLT architecture performs on par with it.
  • Parameter-Matched Compute-Optimal Scaling: Llama-3-tokenizer models outperform Llama-2-tokenizer models on the same training data, showing tokenizer choice affects tokenizer-based scaling.BLT can achieve similar scaling trends with average patch sizes of 6 or 8 bytes, compared with average BPE token sizes of 3.7 and 4.4 bytes.
  • Patches Scale Better Than Tokens: An average patch size of 8 bytes can yield nearly 50% inference-FLOP savings, while larger patch sizes improve relative performance at larger model scales.The 8-byte model starts worse than BPE at 1B parameters but becomes better than BPE at 7B scale.
  • Patches Scale Better Than Tokens: BLT can increase patch and model size together while maintaining the same training and inference FLOP budgets and training-data quantity.Longer patches save compute because the global latent transformer runs less often, enabling that compute to be reallocated toward model growth.
  • Patches Scale Better Than Tokens: The fixed-inference study finds BLT scaling trends better than tokenization-based architectures across both inference-FLOP classes.BPE models lead at small training budgets, but BLT surpasses them not far beyond the compute-optimal regime.

6 Byte Modeling Improves Robustness

BLT improves robustness to character-level noise and strengthens character, phonological, and low-resource multilingual understanding relative to tokenizer-based models. These gains include strong performance on sequence manipulation and long-tail byte sequences, while conversion from a pretrained tokenizer model remains imperfect.

  • Robustness to Noisy Inputs: BLT outperforms tokenizer-based models across five character-level noise strategies, with an average advantage of 8 points on noised HellaSwag.The evaluation applies noise to prompts, completions, or both.
  • Character-Level Tasks: BLT outperforms the Llama 3 tokenizer-based baseline on grapheme-to-phoneme prediction in a 5-shot Phonology Bench evaluation.The task maps character sequences to word-pronunciation phonemes.
  • Character-Level Tasks: BLT-Entropy outperforms both BPE Llama 3 models by more than 25 points on CUTE, including 99.9% on both spelling tasks.Word deletion and insertion are the only CUTE tasks where BPE performs better.
  • Low-Resource Machine Translation: BLT outperforms Llama 3 by 2 points when translating into English and by 0.5 points when translating from English on FLORES-101.The evaluation covers six language families and 21 lower-resource languages with varied scripts.
  • Low-Resource Machine Translation: These translation results support stronger generalization to long-tail byte sequences in lower-resource language families.The comparison uses SentencePiece BLEU across translations into and out of lower-resource languages.
  • Training BLT from Llama 3: Initializing BLT’s global transformer from pretrained Llama 3.1 parameters improves performance over both Llama 3 and BLT baselines at the same FLOP budget.The initialized model still outperforms BLT-Entropy on MMLU despite BLT-Entropy using a substantially larger training dataset.
  • Training BLT from Llama 3: Converting Llama 3.1 into BLT causes a slight drop on MMLU and HumanEval and larger drops on other tasks.The authors identify data mixtures and other hyperparameters as areas requiring further optimization.

7 Ablations and Discussion

The ablations examine patching strategies, cross-attention, entropy-model choices, and hash embeddings, finding that dynamic or space-based patching and key architectural components improve BLT efficiency or performance.

  • Entropy model: Scaling performance improves with larger entropy models and longer context windows, with diminishing returns beyond 50M parameters and 512-byte context.The study varies entropy-model sizes from 1M to 100M parameters and context windows from 64 to 512 bytes.
  • Patching schemes: All non-static patching schemes outperform static patching, with space patching closely approaching dynamic entropy-based patching.The comparison controls sequence length and the number of bytes seen during training and inference.
  • Patching schemes: Entropy-based patching gains carry over from scaling trends to downstream benchmark tasks despite the additional entropy-model computation.The evaluations compare tokenizer-based models, space patching, and entropy-based patching on the Llama 2 dataset.
  • Cross-attention: Decoder cross-attention is most effective, while encoder cross-attention provides only a slight improvement when queries use pooling initialization.Encoder cross-attention also helps particularly on Common-Crawl and with larger patch sizes.
  • Hash embeddings: Hash embeddings improve all domains, with larger gains on Wikipedia and Github; increasing hashes from 300K to 500K changes performance by only 0.001 bpb after 15k steps.The reported domain differences are 0.04 bpb versus 0.01 bpb after 15k steps at 8B, and hash gains are complementary with cross-attention.
  • Local model: When paired with hash n-gram embeddings, BLT works with a one-layer local encoder and a heavier decoder.The ablation reallocates capacity toward decoding while retaining performance.

8 Related Work

Related work established byte- and character-level modeling, patching, and compression-based alternatives, but often at smaller scale or higher compute. BLT extends these directions with dynamic patching and architectural changes aimed at closing the compute-controlled gap to tokenized models.

  • Character- and byte-level models: Character- and byte-level models offer flexible input representations and can improve selected language or domain tasks, but prior systems often required substantially more compute.The cited work includes character RNNs, convolutional models, hierarchical LSTMs, and byte-level transformers.
  • Patching-based approaches: Patching reduces the inflated computation of byte-level language models, with earlier approaches demonstrating success mainly at smaller model and training-data scales.These approaches include static downsampling and upsampling, whitespace patching, and compression-based windows.
  • Patching-based approaches: MegaByte uses fixed static patches and can match tokenizer-based models at 1B parameters on 400B bytes, but static patching lags behind compute-optimally trained tokenized models in the authors’ flop-controlled comparison.The paper positions BLT as a dynamic-patching extension intended to bridge this gap.

9 Limitations and Future Work

The paper identifies limitations in its training-ratio choices, implementation efficiency, and separately trained patching model. It also leaves larger-scale architectural optimization and byte-ifying pretrained tokenized models for future work.

  • Scope and scaling: BLT’s architectural choices use scaling laws derived for BPE transformers, which may be suboptimal for BLT’s data-to-parameter ratios.Many experiments were conducted up to 1B parameters, so optimal choices may change at 8B and beyond.
  • Implementation efficiency: The reported flop-matched implementations may not yet match tokenizer-based models in wall-clock time because existing libraries favor tokenized transformer architectures.Further implementation optimization may improve practical runtime parity.
  • Future work: BLT uses a separately trained entropy model for patching rather than learning the patching model end-to-end.End-to-end learning of the patching model is identified as a future direction.
  • Future work: Initial experiments suggest that pretrained tokenizer-based models such as Llama 3 may be byte-ified by initializing and freezing the global transformer.Further work is needed to determine whether this can retain byte-level benefits or exceed tokenizer-based performance without training from scratch.

10 Conclusion

BLT replaces fixed-vocabulary tokenization with dynamic byte patching, matching tokenization-based models while improving efficiency, robustness, and scaling flexibility. The conclusion reports results across models reaching 8B parameters and 4T training bytes.

  • BLT dynamically groups raw bytes into patches, allocating computation according to data complexity instead of relying on fixed-vocabulary tokenization.
  • BLT matches tokenization-based models at scales up to 8B parameters and 4T bytes, while allowing up to 50% fewer inference flops.
  • The flop-controlled experiments use distinct architectural hyper-parameter settings across BLT model sizes.

B FLOPs Equations

The appendix defines operation-level FLOP formulas for Transformer and BLT forward passes. It combines feed-forward, attention, embedding, and cross-attention costs, with stated parameter meanings and a separate byte n-gram hash definition.

  • The appendix reports operation costs per token or byte for the FLOP accounting framework.
  • The operation table includes feed-forward, de-embedding, attention, and cross-attention terms, with patch size and query-to-key ratio governing BLT costs.
  • Transformer FLOPs combine feed-forward and attention costs using layers, hidden dimensions, context length, attention heads, and vocabulary size.
  • BLT total FLOPs use the listed primitives together with the BLT-specific FLOPs equation, while backward computation is assumed to cost twice the forward pass.
  • A rolling polynomial hash represents byte n-grams, with its multiplier chosen as a 10-digit prime number.

D Frequency-based n-gram Embedddings

The frequency-based n-gram embedding approach stores frequent byte-grams by length and routes unseen n-grams through hash embeddings. Because table vocabularies omit infrequent n-grams, the final architecture moved to hash-based embeddings.

  • Frequency-based embeddings use separate matrices for byte-gram lengths from 1 through 8.
  • Each frequency table stores the 100,000 most frequent byte-grams for its length, while infrequent byte-grams receive hash-based embeddings.
  • Frequency-based tables cannot represent infrequent n-grams, motivating the subsequent move to hash-based n-gram embeddings.
  • The ablation table compares frequency-based and hash-based n-gram embedding tables for a 1B BLT trained on 100B bytes.

E Entropy Patching Example from MMLU

The MMLU example illustrates entropy-based patching over prompts, few-shot examples, and questions. Repeated answer-choice phrases form larger patches, reducing global-model invocations, but entropy drift can make repetitive reasoning patterns overly aggregated.

  • Full-context entropy patching can repeatedly merge later occurrences of a phrase into large patches, which may be undesirable for reasoning.
  • The MMLU visualization distinguishes prompt, few-shot examples, and answer questions by color while showing entropy-based patch boundaries.
  • Repeated phrases in answer choices form much larger patches, so the global model is invoked fewer times than in tokenization-based inference.
Loading 2412.09871v1…