PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

Sukjun Hwang, Brandon Wang, Albert Gu

arXiv:2507.07955v2cs.LG

TL;DR

Tokenization remains a handcrafted barrier to fully end-to-end foundation models, especially for raw data with weak segmentation cues. H-Net addresses this with recursively learned dynamic chunking and hierarchical processing, matching or surpassing tokenized Transformers while improving robustness, scaling, and data efficiency. The paper supports the approach across language, code, Chinese, and DNA, though validation beyond 1.3B-equivalent scale remains open.

  • Problem

    Tokenization remains a handcrafted preprocessing step, motivating end-to-end architectures that learn chunking while addressing efficiency, learnability, and stability.

  • Method

    H-Net combines recursive hierarchical processing with routing, smoothing, ratio regularization, and gradient-based learning to compress raw data into learned chunks.

  • Results

    A two-stage byte-level H-Net overtakes a strong tokenized Transformer after 30B training bytes, matches a Transformer twice its size downstream, and improves DNA data efficiency 3.6× over isotropic models.

  • Takeaways & Limitations

    H-Net learns semantically coherent boundaries without heuristics or supervision, with especially pronounced improvements for Chinese, code, and DNA.

  • Takeaways & Limitations

    The largest models are FLOP-matched to a 1.3B-parameter Transformer, so performance and stability at 3B, 7B, and larger scales remain to be validated.

Abstract

from arXiv · show

Major progress on language models (LMs) in recent years has largely resulted from moving away from specialized models designed for specific tasks, to general models based on powerful architectures (e.g. the Transformer) that learn everything from raw data. Despite this trend, pre-processing steps such as tokenization remain a barrier to true end-to-end foundation models. We introduce a collection of new techniques that enable a dynamic chunking mechanism which automatically learns content- and context- dependent segmentation strategies learned jointly with the rest of the model. Incorporating this into an explicit hierarchical network (H-Net) allows replacing the (implicitly hierarchical) tokenization-LM-detokenization pipeline with a single model learned fully end-to-end. When compute- and data- matched, an H-Net with one stage of hierarchy operating at the byte level outperforms a strong Transformer language model operating over BPE tokens. Iterating the hierarchy to multiple stages further increases its performance by modeling multiple levels of abstraction, demonstrating significantly better scaling with data and matching the token-based Transformer of twice its size. H-Nets pretrained on English show significantly increased character-level robustness, and qualitatively learn meaningful data-dependent chunking strategies without any heuristics or explicit supervision. Finally, the H-Net's improvement over tokenized pipelines is further increased in languages and modalities with weaker tokenization heuristics, such as Chinese and code, or DNA sequences (nearly 4x improvement in data efficiency over baselines), showing the potential of true end-to-end models that learn and scale better from unprocessed data.

1 Introduction

H-Net replaces handcrafted tokenization with end-to-end dynamic chunking that learns content- and context-dependent segmentation from raw data. Its recursive hierarchy improves modeling, robustness, and performance across settings where tokenization is weaker.

  • Tokenization remains a handcrafted preprocessing barrier because fixed-vocabulary chunks limit character-level understanding and degrade performance on complex languages and modalities.
  • H-Net combines hierarchical processing with dynamic chunking to learn content-aware compression jointly with the rest of the model.The mechanism uses routing, smoothing, auxiliary ratio loss, and gradient-based learning of discrete choices.
  • A single-stage byte-level H-Net matches the perplexity and downstream performance of a strong BPE-tokenized Transformer at sizes exceeding 1B parameters.It compresses inputs to a similar resolution as BPE tokenizers, averaging 4.5-5 bytes per chunk, without external supervision or heuristics.
  • A two-stage H-Net overtakes a strong tokenized Transformer in perplexity after 30B training bytes and matches the downstream evaluations of a Transformer twice its size.The recursive hierarchy also yields steeper training curves and better scaling with data.
  • Pretrained H-Nets show greater robustness to textual perturbations and learn semantically coherent chunk boundaries without explicit supervision.The robustness result is reported on the noisy HellaSwag benchmark suite.
  • On Chinese, code, and DNA, H-Net’s gains increase where tokenization heuristics are weaker, including 59.9 →66.3 on XWinograd-zh and 3.6× improved DNA data efficiency.

2 H-Net Architecture

H-Net is a recursive U-Net-like architecture that compresses sequences into learned chunks, processes them hierarchically, and reconstructs the original resolution. Dynamic routing, smoothing, regularization, and signal-propagation techniques support efficient and stable end-to-end learning.

  • Hierarchical architecture: H-Net consists of encoder networks, a main network, and decoder networks built from sequence-mixing layers, with the main network recursively able to contain another H-Net.An S-stage model uses encoders and decoders at stages 0 ≤ s < S and a main network only at stage S.
  • Hierarchical architecture: The hierarchy compresses inputs into fewer, richer representations, processes them in the main network, and decompresses them back to the original resolution.Unlike fixed-size pooling, chunk boundaries are determined dynamically.
  • Dynamic Chunking: The router’s confidence vector p_s indicates how likely each token is to pass into the main stage and is used by both chunking and dechunking.
  • Signal propagation and efficiency: Mamba-2 state-space layers serve as encoder and decoder building blocks because they efficiently process long, uncompressed sequences and fine-grained inputs.Ablations report that SSM-based encoders and decoders outperform Transformer layers at byte and coarser input levels.
  • Dynamic Chunking: Dynamic Chunking predicts boundaries with a routing module, smooths uncertain boundaries, and uses an auxiliary ratio loss to learn end-to-end compression.The smoothing module attenuates uncertain-boundary effects, while the ratio loss targets desired downsampling ratios.
  • Signal propagation and efficiency: Signal-propagation techniques place projections and normalization carefully and adjust optimization parameters by layer dimensionality and effective batch size.The chosen residual design preserves gradients from deeper stages while allowing learned adaptation of the residual contribution.
  • Autoregressive computation: At inference, H-Net allocates different amounts of computation to individual tokens based on its learned decisions.

3 Experiments

Across matched experiments, H-Net’s learned hierarchical chunking improves language modeling, robustness, interpretability, and performance on languages and modalities with weaker tokenization cues. Two-stage dynamic chunking consistently strengthens results over one-stage and heuristic or fixed-width alternatives.

  • English language modeling: H-Net (2-stage) outperforms H-Net (1-stage), while H-Net (1-stage) improves over learned space-based chunking and fixed-width pooling.These comparisons support the value of iterated dynamic chunking and context-dependent segmentation.
  • English language modeling: 30B bytes marks the XL crossover where H-Net (2-stage) overtakes the tokenized Transformer, with its advantage widening as training continues.The corresponding crossovers are 100B bytes for H-Net (1-stage) and 200B bytes for H-Net (space).
  • English language modeling: 2.6% is H-Net (2-stage)’s XL improvement over the Transformer baseline, while its Large improvement is 2.2%.The Large H-Net (2-stage) also matches the average downstream performance of the XL BPE Transformer.
  • Robustness and interpretability: H-Net (2-stage) shows substantially improved robustness to textual perturbations using checkpoints trained only on clean data.The improvement exceeds the gaps observed on standard benchmarks.
  • Robustness and interpretability: H-Net learns boundaries at whitespace and word starts, then forms content-aware chunks such as words, numberings, and semantic multi-word groups.It often preserves semantic boundaries even when whitespace is removed.
  • Alternate language datasets: 3.6× more data is sufficient for H-Net models to achieve similar DNA performance to isotropic models during stable training.The finding holds for both examined main-network architectures.
  • Ablation studies: Mamba layers improve byte-sequence compression and modeling, with pure Mamba encoder-decoder configurations outperforming pure Transformer configurations in ablations.The pure Transformer configuration performs worst despite using more FLOPs.

4 Discussion

The discussion positions H-Net as a dynamic hierarchical architecture with promising scaling and compute-allocation properties, while identifying evaluation, efficiency, and scaling limitations. It also highlights deeper hierarchies and layer choices as open directions.

  • Limitations: Dynamic computation complicates efficiency comparisons and batching because sequence lengths, memory use, and per-token compute vary across examples.The paper identifies specialized kernels and engineering improvements as practical responses, while treating these issues as non-fundamental.
  • Deeper hierarchies: Iterating H-Net from zero to one and then two stages consistently improves performance, while three-stage hierarchies remain untested.Testing deeper recursion is identified as an immediate direction for future work.
  • Architecture and computation: H-Net is a global architectural design that may interact with, but is orthogonal to, the choice of individual sequence-model layers.The paper suggests possible interactions with recurrent layers, dynamic state allocation, and long-context retrieval, but does not study them here.
  • Limitations: The largest models were FLOP-matched to a 1.3B-parameter Transformer, leaving validation at 3B, 7B, and larger sizes unresolved.The authors observed no instabilities at their tested sizes but note that larger-scale investigation may be needed.
  • Scaling: Formal scaling-law coefficients were not estimated because of resource constraints; losses during constant learning-rate phases were used as a data-scaling proxy.The authors regard formal scaling-law analysis as important future work.
  • Limitations: Standard BPB for BPE models is inexact because it assumes a string’s probability equals the probability of its greedy tokenization.Exact BPB is intractable because models may assign nonzero probability to exponentially many alternative tokenization sequences.

5 Conclusion

The conclusion presents H-Net as a tokenizer-free, end-to-end approach that learns dynamic chunking and hierarchical abstractions directly from raw data. It emphasizes tokenizer limitations and the broader potential of recursive architectures across language and other modalities.

  • End-to-end modeling: H-Net removes tokenizer preprocessing by learning content- and context-dependent discrete segmentation through gradient-based optimization.The model replaces handcrafted tokenization with an end-to-end dynamic chunking mechanism.
  • Tokenization limits: Traditional fixed-vocabulary tokenizers are reported to be biased toward high-resource languages, fragile on adversarial inputs, and weaker on character-level tasks.Tokenizer-free methods are motivated by these limitations.
  • Scope of related work: The conclusion summarizes tokenizer-free architectures across autoregressive, non-autoregressive, and non-language modalities, with related methods organized in Table 6.The appendices distinguish non-autoregressive work because differing motivations affect design and downstream evaluation.

A.1 Autoregressive Tokenizer-free Architectures

Autoregressive tokenizer-free architectures span isotropic byte models and hierarchical models with static or dynamic chunking. Static methods are efficient but can split meaningful language units because their boundaries ignore content.

  • Autoregressive tokenizer-free models include non-hierarchical isotropic architectures and hierarchical architectures with static, external, or end-to-end dynamic chunking.
  • Static Chunking: Fixed-width hierarchical models reduce long byte sequences through pooling, but their chunk boundaries are independent of input context.
  • Static Chunking: Static chunking can split words, morphemes, or phrases, limiting expressivity when meaningful units do not align with fixed windows.
  • Hierarchical Architectures: Autoregressive U-Nets extend multiscale processing while maintaining causality, and have also been used successfully in vision and audio.

A.1.3 External Chunking

External chunking uses content-aware boundaries supplied by delimiters or entropy heuristics, but these approaches are constrained by modality, tuning, and hierarchical extensibility. H-Net instead targets jointly learned content- and context-aware segmentation.

  • External chunking identifies content-aware boundaries using surface delimiters or statistical properties such as autoregressive conditional entropy.
  • Delimiter-based methods: Delimiter methods provide meaningful chunks for English and code but fail on inputs without explicit separators, including many non-European languages and DNA.
  • Delimiter-based methods: Delimiter-based chunking also cannot be extended cleanly to multiple hierarchical levels because higher-level natural delimiters are ambiguous.
  • Entropy-based methods: BLT-style entropy chunking requires domain-specific threshold tuning, adds 3B parameters at the 8B scale to match BPE Transformer baselines, and cannot support multi-stage latent hierarchies.
  • End-to-end dynamic chunking: H-Net replaces external heuristics with dynamic chunking learned jointly from data, while prior end-to-end DPT variants were unstable and limited to single-stage chunking.

A.2 Non-Autoregressive Tokenizer-free Architectures

Non-autoregressive tokenizer-free work includes isotropic, hierarchical, and modality-specific approaches that alter sequence resolution or tokenization. These methods address efficiency and representation granularity across language and other data types.

  • Isotropic: ByT5 directly models bytes with a bidirectional encoder-decoder and improves performance with small models and on spelling-sensitive tasks.
  • Hierarchical: Non-autoregressive hierarchical models use U-Net-like or convolution-based static downsampling, while Charformer pools inputs at multiple resolutions.
  • Implicit supervision: Some non-autoregressive methods use external tokenizers for implicit supervision during training, without requiring those tokenizers at inference time.
  • Hierarchical (Dynamic): Dynamic non-autoregressive methods include MANTa’s learned segmentation and MrT5’s inference-time token merging, which reduces sequence lengths by up to 80%.
  • Other modalities: Other modalities use patchification, latent-space variable-length tokenization, heuristic token merging, or learned semantic segmentation to shorten raw perceptual sequences.
  • Related directions: Recent work also studies vocabulary scaling, cross-tokenizer transfer, tokenization effects on arithmetic and data leakage, and tokenization theory.

C Learning Rate Modulation

H-Nets use stage-specific learning-rate modulation to account for differing batch sizes and dimensionalities across hierarchy levels. The proposed scaling factors are empirical heuristics rather than established principles.

  • Stage-specific learning rates multiply a base rate by λ_s, with example multipliers λ_0 = 2.0, λ_1 = 1.5, and λ_2 = 1.0.
  • The learning-rate scheme is tied to stage-dependent batch size and dimensionality in the hierarchical architecture.
  • The authors characterize these signal-propagation scaling factors as heuristics that may be improved through further research.

D.1 Robustness Score

The robustness score measures how much of a model’s original performance remains under textual perturbations. For HellaSwag, it adjusts for the 0.25 random-guess baseline.

  • For HellaSwag, the score subtracts 0.25 from perturbed and unperturbed accuracy.
  • The robustness score measures the percentage of unperturbed performance retained under perturbation.
  • The 0.25 adjustment treats four-choice chance performance as zero retained capability.

E.1 Different Downsampling Methods in the Chunking Layer

The chunking layer compares H-Net’s default downsampling with pooling and cross-attention alternatives within predicted boundaries. The default method performs on-par with these alternatives despite its simpler design.

  • H-Net compares default downsampling with channel-wise max pooling, mean pooling, and cross-attention within each predicted boundary.
  • The default H-Net compression performs on-par with channel-wise pooling and cross-attention.
  • Adding Transformer layers near dynamic-chunking modules can alleviate instabilities in a pure Mamba-2 network.

E.2 Details of Chinese and Code Experiments

The Chinese and code experiments compare H-Net architectures with Transformer and H-Net baselines under matched model settings. They evaluate scaling behavior, compression ability, and Chinese language understanding.

  • H-Net (2-stage) is evaluated against Transformer and H-Net (space) on Chinese and code, with superior scaling reported for H-Net (2-stage).
  • The experiments measure scaling behavior, final-checkpoint bits-per-byte compression, and Chinese XWinograd performance.
  • Chinese models primarily match GPT-3 XL settings, including d_model and encoder-decoder architecture, while adjusting main-network depth for compression differences.

E.3 DNA Architecture Ablations

DNA architecture ablations compare encoder and main-network designs. M3T1 improves data efficiency, while a principled Transformer-containing main network substantially improves training stability.

  • 3.6× data efficiency is achieved by H-Net (1-stage) with an M3T1 encoder versus an isotropic architecture.
  • Transformer layers in the encoder do not appear helpful for text, suggesting their usefulness may be modality-specific.
  • Replacing the pure Mamba-2 main network with a T1M13T1 architecture greatly improves stability.

F Distilling Token-Level Models to Byte-Level

The paper distills a pretrained token-level model into a byte-level H-Net using auxiliary supervision for dynamic boundaries and hidden-state alignment. The resulting model approximately matches the teacher across benchmarks, while revealing both the benefits and limits of reproducing tokenization behavior.

  • Distillation setup: H-Net distills a BPE-tokenized model into a byte-level architecture with a frozen main network initialized from the token model’s backbone.The outer stages replace the tokenizer, embedding module, and LM head around the main network.
  • Auxiliary losses: Boundary-prediction supervision trains the router to pass each real token’s start through the main network.The binary cross-entropy loss has equal weight to the main loss.
  • Auxiliary losses: Hidden-state matching aligns post-adapter representations with the corresponding token embeddings using an L2 loss weighted by 0.02.The matching target depends on whether a byte begins its token or belongs to a later byte.
  • Auxiliary losses: Boundary supervision is crucial for learning a good matching, while embedding matching helps speed training; excessive embedding-loss weight can harm language modeling.The two auxiliary objectives cannot generally reach zero because BPE boundaries may require future bytes to determine.
  • Learned boundaries: Above 99.5% boundary prediction accuracy, versus below 95% for space-only tokenization, indicates that the distilled H-Net recognizes some subword boundaries.The model therefore does not simply tokenize at whitespace positions.
  • Distillation results: H-Net approximately matches the tokenized teacher across almost all benchmarks but remains slightly worse because it cannot exactly reproduce the teacher’s behavior.The distilled model is nevertheless much stronger than an H-Net trained from scratch on 189B bytes.
Loading 2507.07955v2…