PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Byte Latent Transformer

Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz, Gargi Ghosh, Luke Zettlemoyer, Christopher Potts, Xiaochuang Han, Srinivasan Iyer

arXiv:2605.08044v1cs.CLcs.AIcs.LG

TL;DR

Byte-level language models avoid subword vocabularies but remain limited by slow byte-by-byte generation. The paper introduces BLT-D for parallel byte generation and BLT-S and BLT-DV for verification-based decoding. Across translation and code generation, these methods reduce estimated memory-bandwidth cost by over 50% versus BLT, with different speed–quality trade-offs.

  • Problem

    Byte-level models retain advantages over subword models but remain inefficient because generating equivalent text requires many sequential byte-level decoder passes.

  • Method

    BLT-D combines BLT’s hierarchical latent tokenization with block-wise diffusion, while BLT-S and BLT-DV use decoder or diffusion drafts followed by autoregressive verification.

  • Results

    Over 50% lower estimated memory-bandwidth cost than BLT is achieved across translation and code generation tasks, with BLT-D the fastest method.

  • Takeaways & Limitations

    The methods narrow the inference-efficiency gap between byte-level and subword-level models while offering distinct speed–quality trade-offs.

  • Takeaways & Limitations

    Inference efficiency is evaluated using NFEs and estimated memory bandwidth rather than a highly optimized implementation benchmark.

Abstract

from arXiv · show

Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte autoregressive generation. We address this bottleneck in the Byte Latent Transformer (BLT) through new training and generation techniques. First, we introduce BLT Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise diffusion objective alongside the standard next-byte prediction loss. This enables an inference procedure that generates multiple bytes in parallel per decoding step, substantially reducing the number of forward passes required to generate a sequence. Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality: BLT Self-speculation (BLT-S), in which BLT's local decoder continues generating past its normal patch boundaries to draft bytes, which are then verified with a single full-model forward pass; and BLT Diffusion+Verification (BLT-DV), which augments BLT-D with an autoregressive verification step after diffusion-based generation. All methods may achieve an estimated memory-bandwidth cost over 50% lower than BLT on generation tasks. Each approach offers its own unique advantages, together removing key barriers to the practical use of byte-level LMs.

1 Introduction

Byte-level models avoid subword vocabularies but remain costly to decode because they generate bytes sequentially. This paper introduces BLT-D and verification-based extensions that generate multiple bytes per step while retaining BLT’s hierarchical design, reducing estimated inference cost.

  • Motivation: Byte-level models operate directly on raw bytes, but their longer sequences make naïve autoregressive training and inference substantially more expensive than subword models.A typical subword token spans several bytes, increasing the number of operations required for equivalent text.
  • BLT-D: BLT-D combines BLT’s hierarchical latent tokenization with block-wise discrete diffusion to generate fixed-size blocks of future bytes in parallel.Its training uses dynamically segmented patches while allowing diffusion blocks to extend beyond individual patch boundaries.
  • Verification extensions: Verification accepts drafted bytes through the first mismatch, and under greedy decoding produces outputs identical to standard autoregressive decoding.The full model re-encodes the candidate sequence and replaces the first mismatching byte with its own prediction.
  • Verification extensions: BLT-S drafts bytes beyond normal patch boundaries with BLT’s lightweight decoder, then verifies the draft using a full forward pass.This exploits BLT’s existing decoder rather than requiring a separate draft model.
  • Verification extensions: BLT-DV combines diffusion-based block drafting with autoregressive next-byte verification to improve generation quality relative to unverified diffusion generation.BLT-D is trained with both diffusion and next-byte prediction objectives, enabling causal autoregressive verification.
  • Results: Over 50% lower estimated memory-bandwidth cost versus BLT is achieved by BLT-D across translation and code generation tasks.BLT-D may reach up to 92% reduction with larger blocks, BLT-DV up to 81%, and BLT-S up to 77% with no loss in task performance.

2 Background and Related Work

This section introduces BLT’s byte-level, dynamically patched architecture and reviews discrete diffusion as a basis for non-autoregressive text generation. It also describes the local decoder’s cross-attention and the masking-based diffusion process used in this setting.

  • Byte Latent Transformer: BLT operates directly on raw bytes and dynamically groups them into variable-length patches that control computation across local and global components.High-entropy regions use shorter patches, while predictable spans use longer patches.
  • Architecture overview: BLT comprises a local encoder, global Transformer, and local decoder that transform byte representations into latent token representations and output bytes.The encoder produces latent representations, the global Transformer processes them, and the decoder generates the output sequence.
  • Local decoder: The local decoder autoregressively generates bytes while updating byte hidden states through cross-attention to latent token representations and Transformer layers.The decoder uses lightweight Transformer layers; its cross-attention projections are defined through query, key, value, and output matrices.
  • Discrete diffusion: Discrete diffusion models corrupt data through a forward noising process and learn an iterative reverse process that removes noise for generation.Text diffusion extends this framework to discrete token sequences and is typically non-autoregressive.
  • Absorbing discrete diffusion: The reviewed diffusion process independently masks sequence positions at a sampled noise level, trains denoising predictions, and treats [MASK] as an absorbing state.The timestep is not embedded directly; the input corruption is assumed to encode it implicitly.

3 BLT Diffusion

BLT-D adapts BLT for block-wise diffusion decoding, generating fixed-size future-byte blocks through masked iterative unmasking while retaining hierarchical latent processing. Its training data, attention patterns, and combined objectives support predictions beyond typical patch boundaries and reduce forward passes during generation.

  • BLT-D retains BLT’s encoder and global model while adding block diffusion decoding over fixed-size future-byte blocks.The decoder processes a clean prefix together with a masked block and iteratively reconstructs the block.
  • 3.1.2 Block Unmasking Strategy: Entropy-bounded sampling ranks masked positions by entropy and selects the largest subset whose cumulative entropy stays below a threshold.This provides an alternative unmasking strategy for controlling the trade-off between parallelism and prediction uncertainty.
  • 3.1.2 Block Unmasking Strategy: Confidence-based unmasking decodes masked positions whose maximum predicted probability exceeds α in parallel, unmasking the highest-confidence position when none qualifies.The strategy prioritizes high-certainty predictions while ensuring progress at every step.
  • 3.1 BLT-D Inference: BLT-D generates a block of size B in s unmasking steps rather than B sequential decoder passes, usually with s < B.The encoder and global model are also invoked once per block, and cached representations reduce recomputation.
  • 3.2 BLT-D Training: Training blocks begin at dynamically determined patch boundaries, use fixed size B, preserve original byte positions, and can extend beyond average patch lengths.Future-byte blocks are independently masked at a sampled timestep before reconstruction training.

4 Pre-training and Generation Experiments

The experiments evaluate BLT and BLT-D variants across translation and coding tasks using task-performance and efficiency metrics. BLT-D consistently improves efficiency, with larger diffusion blocks offering greater savings but potentially lower coding performance.

  • Metrics: Efficiency is measured with decoder NFEs, encoder/global NFEs, and estimated memory bandwidth dominated by parameter loading.The bandwidth estimate uses 16-bit parameters and assumes small KV caches and batch sizes.
  • Results: Across all evaluated tasks, BLT-D variants consistently outperform BLT in efficiency by reducing decoder and encoder/global NFEs.These reductions produce large memory-bandwidth decreases.
  • Results: BLT-D-4 nearly matches BLT’s task scores while requiring less than half the NFEs and memory bandwidth, especially benefiting translation tasks.BLT-D-8 also combines strong task performance with substantial efficiency gains.
  • Results: 87–92% reduction in memory bandwidth makes BLT-D-16 the fastest evaluated model, but its coding-task performance is lower.Its translation performance remains competitive, illustrating a speed–quality trade-off as block size increases.

5 Extensions: BLT-S and BLT-DV

BLT-S and BLT-DV reuse existing BLT components to trade inference efficiency against generation quality. BLT-S drafts with the local decoder, while BLT-DV verifies diffusion drafts autoregressively.

  • Design rationale: The extensions require no architectural changes or additional training, because they use existing model components as drafting and verification mechanisms.BLT-S uses BLT’s decoder, while BLT-DV combines BLT-D diffusion with causal next-byte prediction.
  • BLT-S: BLT-S drafts beyond normal patch boundaries with BLT’s local decoder, then verifies the draft using a full E, G, and D forward pass.Only the prefix through the first mismatch is accepted.
  • BLT-DV: BLT-DV uses diffusion to draft byte blocks and autoregressive next-byte prediction to verify them with the same model parameters.Block size and unmasking strategy control the balance between speed and verification acceptance.
  • Quality–efficiency trade-off: BLT-DV’s verification step prevents the rapid quality degradation typically associated with one-step diffusion alone.The verification step adds computation while recovering some generation quality.
  • Generation-task results: Verification improves BLT-D task performance but increases encoder/global NFEs and memory bandwidth, whereas BLT-S reduces encoder/global NFEs with competitive task performance.Unverified BLT-D-8 and BLT-D-16 remain the fastest methods, though their task performance is somewhat diminished.
  • Limitations and future work: The experiments use a relatively small decoder, leaving decoder scaling and improved BLT-DV training objectives as directions for future work.A larger decoder may improve BLT-D and BLT-DV efficiency, while BLT-S incurs smaller overhead with a lightweight decoder.

6 BLT-D Generation Analysis

BLT-D’s block diffusion decoding exposes a trade-off between generation diversity and computational efficiency. More decoder calls yield more diverse text, while fewer calls produce more repetitive outputs.

  • Analysis setup: The analysis evaluates unconditional BLT-D generation with entropy-bounded and top-p sampling while varying parallelism in block diffusion decoding.Diversity is measured by word-level type-token ratio after whitespace tokenization.
  • Diversity–efficiency trade-off: As decoder calls increase, type-token ratio also increases, indicating more diverse generated text.The reported relationship connects additional decoder passes with greater lexical diversity.
  • Diversity–efficiency trade-off: Fewer decoder calls correspond to repetitive, predictable text with lower uncertainty and entropy.Block diffusion therefore provides a controllable diversity–efficiency trade-off.

7 Conclusion

The paper introduces BLT-D and two speculative-decoding extensions to accelerate byte-level generation while retaining BLT’s latent representations and dynamic patching. Its main evaluation limitation is reliance on NFEs and estimated memory bandwidth as proxies rather than highly optimized inference measurements.

  • BLT-D combines BLT’s hierarchical latent tokenization with a block-wise diffusion objective and generates multiple future bytes in parallel.The semi-autoregressive decoder preserves dynamic patching and latent token representations.
  • BLT-S drafts bytes beyond normal patch boundaries with BLT’s decoder, while BLT-DV verifies diffusion drafts using autoregressive next-byte prediction.
  • Each proposed method substantially reduces total model calls, narrowing the inference-efficiency gap between byte-level and subword-level models.
  • The evaluation uses NFEs and estimated memory bandwidth as proxy metrics, leaving optimized implementations that account for kernels, hardware, batching, and KV-cache management for future work.

A.1 Architecture Implementation Details

The BLT and BLT-D models retain the original BLT’s core Transformer implementation choices while using different attention implementations for fixed and patch-dependent masks. FlashAttention handles fixed causal attention, whereas FlexAttention handles custom masks that vary with patch structure.

  • BLT and BLT-D use SwiGLU feed-forward layers, RoPE with θ = 500000, and RMSNorm across their Transformer components.
  • FlashAttention is used for encoder and global-model self-attention with fixed causal masks and a window size of 512.
  • FlexAttention is used for cross-attention and decoder self-attention because their custom masks depend on patch structure and vary by example.

A.2 Pre-training Optimization and Hyperparameter Settings

The training setup uses separate step counts and byte-scale batch sizes for 1B and 3B models, with AdamW, cosine learning-rate decay, warm-up, weight decay, and gradient clipping.

  • 1B models train for 240,000 steps with a batch size of 2^19 tokens, approximately 2 million bytes per step, while 3B models train for 480,000 steps with 2^20 tokens, approximately 4 million bytes.
  • All models use AdamW with β1 = 0.9, β2 = 0.95, and ϵ = 10^-8.
  • The cosine learning-rate schedule warms up to 4 × 10^-4 and decays to zero, with 2,000 warm-up steps for 1B models and 4,000 for 3B models.
  • Training applies weight decay of 0.1 and global gradient clipping at a threshold of 1.0.

B All 1B Model Results

The 1B-model evaluation covers likelihood-based benchmarks and multiple generation tasks across BLT, BLT-D, BLT-S, and BLT-DV. It varies diffusion thresholds, speculation windows, verification settings, and block sizes while tracking task performance, NFEs, and memory bandwidth.

  • Evaluation scope: The 1B-model evaluation reports generation-task results for BLT, BLT-D, BLT-S, and BLT-DV, alongside likelihood-based results in Table 2.
  • Inference settings: BLT-D is evaluated with confidence-based unmasking thresholds α ∈ {0.5, 0.7} and EB-sampling thresholds γ ∈ {0.8, 1.0}.
  • Inference settings: BLT-DV uses more permissive settings, including α = 0.3, γ ∈ {1.5, 2.0}, and one-step diffusion that unmasks all byte positions at once.
  • Inference settings: BLT-S is tested with speculation windows k ∈ {4, 8, 16}, while Figures 9 and 10 report selected generation settings for 1B variants.
  • Reported comparisons: Figure 9 compares task performance, NFEs, and memory bandwidth against BLT-D block sizes 4, 8, and 16 and a matched-size BPE model.
  • Tasks: The evaluation includes French-to-English and German-to-English translation, HumanEval, and MBPP generation tasks.

C All 3B Model Results

This section reports a larger sweep of inference hyperparameters for 3B BLT-D, BLT-DV, and BLT-S models across generation tasks. It includes full results for French-to-English, German-to-English, HumanEval, and MBPP under various generation settings.

  • Inference hyperparameter sweep: The experiments sweep inference hyperparameters for 3B BLT-D, BLT-DV, and BLT-S models on generation tasks.BLT-D settings include confidence-based unmasking thresholds α ∈ {0.5, 0.7} and EB-sampling thresholds γ ∈ {0.8, 1.0}.
  • Inference hyperparameter sweep: BLT-DV uses more permissive settings designed to unmask more bytes per step.The tested settings include α = 0.3 and γ ∈ {1.5, 2.0}.
  • Generation-task results: Full French-to-English translation results are reported for 3B-parameter models across various generation settings.
  • Generation-task results: Full German-to-English translation results are reported for 3B-parameter models across various generation settings.
  • Generation-task results: Full HumanEval and MBPP results are reported for 3B-parameter models across various generation settings.
Loading 2605.08044v1…