PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Multi-Byte Prediction With Hierarchical Language Models

Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant, Tomasz Limisiewicz, Sachin Kumar

arXiv:2608.15454v1cs.AI

TL;DR

Byte-level hierarchical language models remain slow at inference because they decode one byte at a time. The paper introduces dynamic multi-byte prediction aligned with learned latent segments, placing on the Pareto front for performance and throughput on three of four downstream tasks.

  • Problem

    Hierarchical byte-level models reduce sequence-processing cost but still decode one byte at a time, while fixed-offset multi-token prediction adds heads and misses variable local structure.

  • Method

    LCA uses a single decoder head and boundary-aware latent causal attention to predict a variable number of bytes per step without violating causality.

  • Results

    Across four downstream tasks, LCA lies on the Pareto front for performance and throughput on three of four.

  • Takeaways & Limitations

    Latent tokens learned by hierarchical byte-level models can serve as units of parallel generation without separate per-token multi-byte prediction heads.

  • Takeaways & Limitations

    Experiments use one 373M-parameter model and corpus across four English-centric or English-paired tasks, leaving larger-scale and broader-language generalization unverified.

Abstract

from arXiv · show

Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.

1 Introduction

Byte-level language models avoid limitations of fixed-vocabulary subword tokenization but incur longer sequences and slower autoregressive decoding. This paper introduces Latent Causal Attention, using learned byte segments and a boundary-aware causal mask for dynamic multi-byte generation.

  • Introduction: Subword tokenizers can overfragment rare words, encode language- and script-specific biases, and generalize poorly across domains, non-English settings, and evolving scenarios.
  • Introduction: Byte-level modeling avoids these tokenizer limitations but is computationally expensive because byte sequences are substantially longer than subword sequences.Longer sequences increase Transformer attention cost and slow autoregressive decoding.
  • Introduction: Hierarchical byte-level models compress byte streams into shorter latent sequences, yet inference still happens one byte at a time.Their latent tokenization primarily improves representation efficiency before decoding back to bytes.
  • Introduction: The paper proposes Latent Causal Attention, which uses learned byte segments as the unit of multi-byte generation instead of assigning one head to each future byte.A single multi-byte decoder uses a boundary-aware causal mask that permits attention to previous segments while preventing dependence on other bytes.

2 Background

Section 2 reviews multi-token prediction as parallel future-token generation and motivates dynamic prediction by highlighting its rigidity and parameter cost. It also introduces the hierarchical byte-level architecture underlying dynamic multi-byte prediction, including learned segment boundaries and latent-token compression.

  • Multi-token prediction: Multi-token prediction improves sample efficiency and inference speed by predicting multiple future subword tokens in parallel rather than one token at a time.It uses n dedicated prediction heads operating independently on a shared hidden representation.
  • Multi-token prediction: Fixed-length prediction cannot adapt to varying local surprisal, while each additional future token requires a separate prediction head, increasing parameters linearly with n.These limitations motivate a variable-length prediction window and parameter-efficient design.
  • Hierarchical byte-level model: The hierarchical byte-level model comprises an encoder, language modeling module, boundary predictor, and decoder, extending the FlexiTokens design for dynamic multi-byte prediction.The architecture is presented as a viable alternative to subword-based language models and is described as generalizable to other settings.
  • Hierarchical byte-level model: The encoder’s causal hidden states support boundary prediction and downstream residual connections, while mean-pooled latent tokens provide the model’s compressed hierarchical representation.The boundary count k is constrained by upper and lower bounds on k/T through hyperparameters α and β.
  • Hierarchical byte-level model: The boundary predictor identifies segment ends, and Gumbel Sigmoid sampling keeps these discrete boundary decisions differentiable during training.Mean pooling combines preceding positions into latent tokens, producing a shortened representation with sequence length strictly less than the encoder output.

3 Multi-Byte Prediction with Latent Causal Attention

LCA-MBP augments a hierarchical byte-level LM with a standard next-byte head and a single multi-byte head that predicts variable-length latent tokens in parallel. Its Latent Causal Attention mask preserves causality while shared transformer layers and a shared unembedding matrix support training and inference with left-to-right acceptance of sufficiently confident candidates.

  • Architecture: LCA-MBP decouples decoding into a standard next-byte prediction head and a single multi-byte prediction head.The MBP head predicts all bytes in each latent token in parallel except the first, which remains the next-byte head’s responsibility.
  • Latent Causal Attention: Latent Causal Attention lets each query byte attend to itself, earlier bytes in its segment, and all bytes in the immediately preceding segment while masking later and more distant positions.This mask preserves autoregressive causality while enabling parallel byte prediction.
  • Architecture: The MBP head handles variable-length latent tokens with transformer layers rather than a fixed set of independent prediction heads.Variable latent-token byte counts make the fixed-head design typical of MTP impractical.
  • Training: The training objective combines next-byte and multi-byte prediction losses, while both decoder heads share one unembedding matrix and the MBP pathway uses shared transformer layers.Unlike standard MTP, LCA obtains prediction independence through the latent attention mask and byte-segment pooling.
  • Inference: During inference, MBP candidates are generated from the current and previous latent-token window, then accepted left to right only while confidence exceeds threshold τ.The next-byte output is always accepted; the first candidate below τ and all subsequent candidates are discarded.

4 Experimental Setup

The experiments compare LCA with five parameter-matched flat, hierarchical, and multi-byte baselines across four generative tasks. Training and evaluation use fixed model settings, specified pretraining and finetuning procedures, and task-specific decoding.

  • Baselines: LCA is compared against five baselines spanning flat and hierarchical byte-level architectures.The baselines are LlamaByte, SpaceByte, FlexiTokens, MLP-MBP, and Efficient FlexiTokens.
  • Baselines: The baselines include vanilla byte-level LlamaByte, heuristic-segmented SpaceByte, learned-segmentation FlexiTokens, MLP-MBP, and Efficient FlexiTokens.MLP-MBP uses independent MLP heads for parallel future-token prediction, while Efficient FlexiTokens reuses cached LM outputs away from predicted boundaries.
  • Controlled comparison: All models are parameter-matched to the same total parameter count, increasing FLOPs for flat models such as LlamaByte and SpaceByte.The number of layers used across methods is reported in Appendix A, Table 2.
  • Model configuration: Models use dmodel = 1024, dinner = 4096, a 4096-token context window, and otherwise fixed architectural and optimization settings across runs.Additional fixed settings include RMSNorm, rotary embeddings, 8 attention heads, 8 KV heads, dropout of 0.1, and attention scaling of 1.0.
  • Training and evaluation: 373M-parameter models are pretrained on 50B bytes from FineWeb-edu and evaluated on summarization, question answering, machine translation, and instruction following.Pretraining uses 1,048,576 bytes per gradient step, while inference uses temperature 0.7 and top_p 0.9 for IFEval and greedy decoding for the other tasks.

5 Results and Analyses

LCA-MBP lies on the Pareto front in three of four downstream tasks, matching or exceeding competing throughput while retaining near-best task performance. Its advantage over MLP-MBP arises from more coherent predictions enabled by conditional transformer-based byte conditioning.

  • Overall results: LCA-MBP lies on the Pareto front in three of four tasks, including IFEval, CoQA, and CNN/DailyMail summarization.Across these tasks, LCA matches or exceeds other methods’ throughput while keeping performance close to FxT and Eff-FxT.
  • Overall results: On Spanish–English translation, FxT and Eff-FxT achieve slightly higher COMET scores but substantially lower throughput than LCA.LCA therefore trades a small amount of performance for a meaningful throughput advantage.
  • Head comparison: Despite higher acceptance rates, MLP-MBP consistently underperforms LCA-MBP on downstream metrics at comparable throughput.This suggests MLP-MBP makes confident speculative predictions that are frequently incorrect.
  • Head comparison: LCA improves prediction coherence by conditioning each byte on the previous segment’s context and its position within the current segment.In contrast, MLP heads independently predict future tokens from a shared hidden state without conditioning on other parallel predictions.
  • Threshold ablation: Lowering the acceptance threshold τ admits more speculative tokens, increasing throughput and acceptance rate.Figure 4 averages this ablation across DailySum, es-en, and fr-en.

6 Discussion and Ablations

The ablations show that acceptance thresholds and candidate counts control a task-dependent quality–throughput trade-off, while speculative decoding preserves performance as throughput increases. LCA also accelerates an external baseline by verifying parallel latent-token byte predictions without reducing downstream performance.

  • Threshold Sensitivity: As τ decreases from 0.9 to 0.7, acceptance rises from 50.1% to 56.7% and throughput improves by roughly 10%, while average performance declines by about 3 points.The marginal throughput gain no longer justifies the quality loss around τ = 0.75.
  • Candidate-Count Ablation: Under probability-threshold acceptance at τ = 0.75, optimal candidate count is task-specific: es-en and fr-en peak at n = 7, while DailySum peaks at n = 6.es-en and fr-en gain about +3 points over n = 3, whereas DailySum gains +2.11 points; performance and throughput decline beyond each peak.
  • Candidate-Count Ablation: With speculative decoding acceptance, downstream performance remains constant across n, while throughput grows monotonically by +29-37% between n = 3 and n = 7-8.LCA also overtakes the MLP-MBP baseline on all three tasks.
  • Accepted-Byte Analysis: Across 100 decoding steps, LCA accepts a mean of 3.05 of 6 candidates per step, with 15% of steps accepting the entire window.Accepted bytes range from 0 to all 6 candidates, and no candidate byte is accepted at the first decoding step across multiple runs.
  • External-Model Acceleration: Using LCA as a drafter and FxT as an external verifier matches FxT’s downstream performance on all three tasks while achieving a 1.4-1.7× speedup.Both models are the same size; the speedup comes from predicting an entire latent token’s bytes in parallel rather than using a smaller draft model.

7 Related Work

Related work contrasts byte-level and hierarchical language models with multi-token prediction methods. LCA combines hierarchical segment compression with variable-length, segment-aligned multi-byte decoding using one decoder head and a causality-preserving mask.

  • Byte-Level and Hierarchical Models: Byte-level models avoid fixed-tokenizer limitations for rare words, domain shift, and evolving vocabularies, but produce substantially longer sequences.Their longer sequences increase attention costs.
  • Byte-Level and Hierarchical Models: Hierarchical models compress byte or character sequences into shorter latent tokens using fixed-size pooling or dynamically predicted boundaries.These approaches primarily target sequence-length reduction or representation learning.
  • Byte-Level and Hierarchical Models: Most hierarchical models still decode autoregressively one byte at a time, leaving the decoding bottleneck unresolved.Their latent tokenization reduces byte-sequence processing cost but does not change standard autoregressive decoding.
  • Multi-Token Prediction: Multi-token prediction accelerates generation by predicting multiple future tokens from one context, but common methods add separate heads for future positions.These approaches can improve sample efficiency and accelerate generation, while their prediction-head design introduces additional parameters.
  • LCA’s Related-Work Distinction: LCA uses one decoder head and a learned mask to predict all bytes within each variable-length segment in parallel, conditioned only on previous segments.This removes per-token parameter overhead and replaces fixed-offset prediction with segment-aligned prediction.

8 Conclusion

The paper introduces LCA, a dynamic multi-byte prediction method for hierarchical byte-level language models. LCA uses one decoder head and boundary-aware masking to predict variable-length byte sequences, placing on the Pareto front for three of four downstream tasks.

  • LCA is a dynamic multi-byte prediction method for hierarchical byte-level language models.
  • A single decoder head and boundary-aware attention mask predict a variable number of bytes per step while aligning generation with the models’ hierarchical structure.
  • Across four downstream tasks, LCA lies on the Pareto front of performance and throughput on three of four.
  • LCA reuses latent tokens as parallel-generation units, eliminating separate per-token multi-byte prediction heads.

Limitations · Appendix

The study evaluates LCA at only one model scale and on English-centric or English-paired tasks. Testing whether its gains persist at larger scales and across low-resource or morphologically rich languages remains future work.

  • Limitations: The experiments use a single model scale, 373M parameters, and one training corpus.This enables comparison with parameter-matched baselines under identical conditions.
  • Limitations: The evaluation covers four English-centric or English-paired tasks but excludes low-resource and morphologically rich languages.The passage notes that LCA inherits its tokeniza…
  • Limitations: Verifying whether the gains persist at larger scales remains future work because larger-scale experiments require unavailable compute resources.The paper frames this as a limitation of the current experimental scope.

A General Architecture, Training Configurations, and Hyperparameters · A.1 Hyperparameters

The paper uses a consistent visual encoding to depict byte processing, segmentation, latent-token pooling, and RoPE positions. Architectural variants share hidden dimensions, training hyperparameters, and optimizer settings unless explicitly noted, with one compression-specific decoder-depth change.

  • A General Architecture, Training Configurations, and Hyperparameters: Uniform red boxes represent the processed byte sequence throughout the figures.
  • A General Architecture, Training Configurations, and Hyperparameters: Multi-colored bands in the boundary predictor show tokenization into segments, with same-colored bytes belonging to one segment.
  • A General Architecture, Training Configurations, and Hyperparameters: Deeper green, orange, blue, and yellow shades denote latent tokens formed by pooling bytes within each segment.
  • A General Architecture, Training Configurations, and Hyperparameters: Rotated color blocks indicate positions where rotary positional embeddings have been applied.
  • A.1 Hyperparameters: All main byte-level model variants use the same optimizer family and training hyperparameters unless explicitly noted elsewhere.
  • A.1 Hyperparameters: For MLP-MBP at compression rate 5, the MB decoder uses 5 layers instead of 3.
  • A.1 Hyperparameters: Architectural configurations generally share the same hidden dimension, training hyperparameters, and optimizer settings unless otherwise stated.

B Extended Results

Extended results show that LCA-MBP remains Pareto-efficient on French→English translation, while larger speculative windows reduce throughput. LCA-MBP can outperform MLP-MBP despite lower acceptance rates, and external verification confirms most speculative bytes.

  • French Translation Results: LCA lies on the Pareto front for French→English translation, achieving substantially higher BLEU than MLP-MBP at similar throughput.The trend mirrors the Spanish→English result.
  • Throughput for Candidate Bytes n Ablation: Throughput decreases monotonically with candidate count n across all three tasks because larger speculative windows cause more rejected candidates per step.The steepest decline is on DailySum, at -22% from n = 3 to n = 7.
  • Performance and Acceptance Rate with Speculative Decoding: At n = 3, LCA-MBP outperforms MLP-MBP on multiple tasks despite achieving a lower acceptance rate.These acceptance-rate and performance results are reported for speculative decoding.
  • Throughput and Acceptance rate for LCA with External Verification: LCA Self-verify and LCA FxT-verify accept 78-84% of speculative bytes across the three tasks under speculative decoding.FxT and LCA NBP accept every token by construction because they use standard next-byte decoding without speculation.
Loading 2608.15454v1…