PaperlayerView exact revision on arXiv

Source-linked AI summary

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard, Besnik Fetahu, Nasser Zalmout, Xian Li

arXiv:2603.03583v1cs.CLcs.LG

TL;DR

Fixed subword tokenization forces language models to use static segmentation and introduces a non-learnable pipeline stage. ByteFlow Net learns compression-based boundaries directly over raw bytes through a hierarchical architecture, and experiments report stronger performance than BPE-based and other byte-level models.

  • Problem

    Fixed subword tokenization applies static segmentation across inputs and creates a rigid, non-learnable stage in language-modeling pipelines.

  • Method

    ByteFlow Net hierarchically self-tokenizes raw byte streams by selecting boundaries with a coding-rate compression objective integrated into model computation.

  • Results

    ByteFlow Net consistently outperforms strong BPE-based transformers and other byte-level models, with 50.89% average accuracy versus LLaMA’s 49.15% at 600M parameters.

  • Takeaways & Limitations

    The results support end-to-end tokenizer-free modeling as feasible and more effective within the evaluated experiments.

  • Takeaways & Limitations

    The coding-rate computation depends on a noise variance parameter ε^2 that controls its sensitivity.

Abstract

from arXiv · show

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce \textbf{ByteFlow Net}, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top-$K$ selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.

1. Introduction

ByteFlow Net replaces fixed tokenization with end-to-end, compression-driven segmentation of raw bytes. Its hierarchical design dynamically selects meaningful units and concentrates computation on high-level representations.

  • Motivation: Fixed tokenization imposes one segmentation logic across inputs, constraining vocabulary, sequence length, representation granularity, and end-to-end learning.The paper links this static design to brittle behavior in counting, arithmetic, structured data, and multilingual text.
  • Contribution: ByteFlow Net learns to self-tokenize directly from raw byte streams instead of applying a fixed vocabulary.Segmentation is integrated into the forward computation rather than performed by a separate tokenizer.
  • Method: Boundary placement is formulated as an online coding-rate compression problem that adjusts token granularity to input complexity.The model estimates the representational cost of promoting positions to a higher level.
  • Architecture: The hierarchy uses local encoding, dynamic chunking, global transformation, and decoding to byte-level predictions.The global transformer models abstract patterns over dynamically produced tokens, while local components process byte-level information.
  • Results: ByteFlow Net outperforms strong LLaMA and other byte-level architectures on pre-training loss and downstream tasks, while preserving latent geometry.The paper attributes this success to coding-rate segmentation maintaining a coherent latent manifold instead of fragmenting representations.

2. Related Work

Tokenizer-free language-modeling research has developed several alternatives to conventional subword tokenization. These approaches differ mainly in whether byte sequences are processed directly or grouped using heuristic or learned chunking.

  • Tokenizer-free Architectures: Pure byte-level models perform language modeling directly on raw byte sequences.Because full attention is costly for long byte sequences, approaches such as MambaByte target computational efficiency.
  • Tokenizer-free Architectures: Heuristic hierarchical methods group bytes with fixed, rule-based strategies such as strides, word boundaries, or regular expressions.These methods impose segmentation rules before or outside learned adaptive chunking.
  • Tokenization in Language Modeling: Subword tokenization represents text with a fixed vocabulary of word pieces, creating a rigid non-learnable pipeline stage.The paper associates this fixed structure with brittle and unexpected behaviors.

3. ByteFlow Net

ByteFlow Net hierarchically transforms byte sequences into compressed global representations and reconstructs them for byte-level prediction. Coding-rate chunking selects informative boundaries, while efficient local mixing and compressed global attention control computation.

  • 3. ByteFlow Net: The architecture comprises local encoding, coding-rate downsampling, global modeling, upsampling, and decoding.These stages map raw bytes to compressed representations and back to the original sequence length.
  • 3.1. Local Encoder: The local encoder converts byte embeddings into contextualized representations using causal transformer blocks with sliding-window attention and Canon layers.Sliding-window attention reduces attention complexity from O(T^2) to O(T·w_local).
  • 3.1. Local Encoder: Canon layers provide efficient token mixing with negligible parameter overhead, avoiding the very deep local encoder otherwise required by sliding-window attention alone.The paper describes them as highly optimized operators that improve efficient processing of long sequences.
  • 3.2. Downsampling: Coding-rate chunking promotes positions with high information gain as boundaries and compresses positions with low coding rates.The criterion uses contextualized representations and treats diverse, high-information directions as worth preserving.
  • 3.2. Downsampling: Top-K selection chooses the K−1 positions with the largest marginal coding rates after reserving the BOS position, preserving a static computation graph.The fixed target length avoids the variable global sequence lengths produced by threshold-based chunking.
  • 3.3. Global Transformer and Decoder: Selected representations are projected into global space, processed by full causal attention, and then reconstructed to byte-level length.The global transformer is deep and wide because K≪T, while upsampling and decoding return predictions to the original sequence.

4. Experiments

Experiments evaluate ByteFlow Net across scaling, downstream accuracy, character-level performance, chunking strategies, latent representations, and training efficiency. Across these evaluations, coding-rate chunking and ByteFlow Net show stronger performance or scaling than the compared baselines.

  • Experimental setup: Experiments use matched-FLOPs training setups, FineWeb-Edu-100B pretraining, BPB loss, and selected zero-shot downstream tasks.Scaling experiments use GPT-3 XL-scale settings, while ablations use GPT-3 Large-level matched FLOPs.
  • Scaling experiments: ByteFlow Net surpasses the LLaMA baseline around 25B tokens at 600M parameters and maintains the advantage through 50B tokens.At 1.3B parameters, it exhibits the most favorable scaling trajectory among the tested architectures.
  • Downstream tasks: 50.89% average accuracy versus 49.15% for LLaMA gives ByteFlow Net a 1.74-point advantage at the 600M scale.The comparison covers six zero-shot downstream tasks evaluated at 0.6B and 1.3B scales.
  • Character-level performance: ByteFlow Net 1.3B substantially outperforms Llama 3 variants on CUTE despite using 20–32× less training data, with near-perfect Spelling Inverse performance.The reported result concerns character-level benchmarks and orthographic capability.
  • Chunking ablation: Coding-rate chunking achieves the lowest validation BPB loss, 0.86, and the highest average task accuracy, 50.89%, in the chunking ablation.The ablation replaces ByteFlow Net’s chunking module with seven alternative strategies at 0.6B parameters and 50B training tokens.
  • Representation and efficiency: ByteFlow Net preserves a coherent latent manifold and achieves the best BPB and downstream accuracy while training competitively among hierarchical byte models.The efficiency comparison uses controlled runs on 8×A100-80GB with matched FLOPs budgets.

5. Conclusion

ByteFlow Net reframes tokenizer-free language modeling as dynamic compression-based segmentation of raw data. The authors report that this approach outperforms BPE-based transformers and other byte-level models while preserving latent-manifold geometry.

  • ByteFlow Net learns meaningful semantic units from raw data using a coding-rate objective rather than a fixed vocabulary.The objective frames segmentation as a dynamic compression task.
  • ByteFlow Net consistently outperforms strong BPE-based transformers and other byte-level models across experiments.The conclusion also reports a superior scaling trajectory as model size increases.
  • Ablations identify the coding-rate criterion as the key to success, surpassing other dynamic chunking strategies.The authors attribute this advantage to preserving the underlying geometry of the data’s latent manifold.
  • The model allocates computation to a compressed stream of informative content, supporting tokenizer-free modeling as an effective and robust paradigm.

Ethics Statement

The work studies tokenizer-free language-model architecture without human subjects, personally identifiable information, or sensitive data. Experiments use publicly available curated datasets filtered to reduce privacy and harmful-content risks.

  • The research does not involve human subjects, personally identifiable information, or sensitive data.
  • Experiments use publicly available curated datasets, including FineWeb-Edu-100B.The datasets were filtered to minimize privacy violations and harmful-content exposure.
  • The authors acknowledge potential misuse risks involving bias, misinformation, and malicious generation, and encourage responsible downstream use.
  • The authors report no conflicts of interest or external sponsorship influencing the work.

Reproducibility Statement.

The paper documents the coding-rate formulation, its streaming approximation, model architectures, and standardized experimental setup. It also describes the covariance assumptions and implementation details needed to interpret or replicate the experiments.

  • Lossy coding-rate derivation: The coding-rate method models local representations with a multivariate Gaussian distribution and structured covariance matrix.The covariance uses a Kronecker-product structure based on the empirical covariance of local representations.
  • Lossy coding-rate derivation: The rate-distortion formulation estimates the minimum bits needed to encode representations under a reconstruction-error constraint.The determinant captures effective representation dimensionality, while ε^2 controls coding sensitivity.
  • L2 norm approximation: For streaming decisions, the exact coding-rate computation is approximated by a quantity proportional to the representation L2 norm.This approximation is intended for relative comparisons and is most accurate under stated noise, conditioning, and ranking assumptions.
  • Experimental design: The evaluation compares six model families at 600M and 1.3B scales under standardized optimization settings.The protocol includes transformer, byte-level, and hierarchical chunking-aware architectures.

C.4.3. Infrastructure and Implementation

The implementation uses mixed-precision distributed training with compilation and sharding for efficiency. Training configurations vary by model while following documented optimizer, precision, clipping, and positional-encoding settings.

  • All architectures use BF16 mixed precision and Fully Sharded Data Parallel training with model-specific optimizations.
  • ByteFlow Net trains for up to 1.95M optimizer steps, while baselines train for up to 950K steps using AdamW and cosine learning-rate decay.The peak learning rate is 4 × 10^-4, with different warmup and clipping settings for ByteFlow Net and baselines.
  • Training uses BF16 precision, disables TF32 matrix multiplications, and enables torch.compile for reproducibility and kernel fusion.
  • Models are trained on 8 NVIDIA A100 80GB GPUs using full-shard FSDP and pure data parallelism.Compiled graphs are cached to reduce startup overhead.
  • ByteFlow Net and the baseline use different RoPE settings, while the rate-distortion objective schedules λ toward a desired compression ratio.

D. Ablation Studies

The ablation studies isolate ByteFlow Net’s Canon layer integration and compression-ratio choices to examine their effects on efficiency and performance.

  • The studies evaluate Canon layer integration and compression ratio as key architectural design decisions.These experiments aim to identify sources of performance gains and characterize efficiency–performance trade-offs.
  • The ablations examine how efficient token mixing and global sequence length affect ByteFlow Net’s behavior.The supplied passage frames both components as central variables in the architecture.
  • The experiments are designed to reveal trade-offs between computational efficiency and model performance.

E. Ablation Studies

This section introduces the Canon layer as an efficient local token-mixing component and measures its effect through ablation. Removing it substantially reduces accuracy at both tested model scales.

  • Canon layers enable efficient local token mixing through causal convolution with minimal computational overhead.They use optimized CUDA kernels and a 4-token kernel size instead of traditional quadratic attention mechanisms.
  • At 600M parameters, removing Canon layers lowers average accuracy by 1.85 points, from 50.89% to 49.04%.ARC-c accuracy also decreases from 28.36% to 26.73%.
  • At 1.3B parameters, removing Canon layers decreases average accuracy by 2.13 points, from 63.19% to 61.06%.

E.1. Canon Layer Integration Analysis

The Canon layer is presented as a low-overhead mechanism for propagating information across positions in ByteFlow Net’s compressed hierarchical representations. Ablations report accuracy losses when it is removed, with degradation at both model scales.

  • Canon Layer Integration Analysis: Canon layers become increasingly important as models grow larger and process longer sequences.The passage attributes this pattern to more critical information propagation across positions for maintaining coherent compressed representations.

E.2. Compression Ratio Analysis

ByteFlow Net’s compression ratio controls a trade-off between computational efficiency and model performance. Lower compression performs best, while stronger compression substantially reduces global attention computation with manageable degradation.

  • Compression Ratio Analysis: The compression settings vary global sequence length from 4096, corresponding to 2.0× compression, to 1600, corresponding to 5.12× compression.The local sequence length remains fixed at 8192 bytes.
  • Compression Ratio Analysis: 51.74% average accuracy is achieved at global sequence length 4096, improving 0.85 points over the default setting of 3200.This setting incurs greater computational overhead because of larger global-transformer operations.
  • Compression Ratio Analysis: 48.48% average accuracy at global sequence length 1600 is only 2.41 points below the default setting.The result is described as graceful degradation at the highest compression setting.
  • Compression Ratio Analysis: Reducing global sequence length from 4096 to 1600 cuts quadratic global-attention operations by 6.6×.The passage presents this reduction as substantial computational savings with manageable performance trade-offs.
  • Compression Ratio Analysis: The ablations report that information-theoretic chunking maintains performance across a wide range of compression settings.Together with the Canon analysis, these results support the architecture’s stated design philosophy.
Loading 2603.03583v1…