Source-linked AI summary
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel
TL;DR
Token-based NLP models remain vulnerable to vocabulary and spelling issues, motivating models that process raw bytes directly. ByT5 minimally adapts T5 for byte inputs and achieves competitive or superior results in several settings, despite using less pretraining text. The design increases computational cost, including slower pretraining and inference.
Problem
Tokenization can mishandle out-of-vocabulary words, spelling variation, and unknown characters, motivating token-free models that process arbitrary text directly.
Method
ByT5 minimally adapts multilingual T5 with byte inputs, longer mask spans, and a heavier encoder.
Results
ByT5 outperforms mT5 in five reported scenarios, including smaller models, generative and multilingual tasks, spelling-sensitive tasks, and noisy inputs.
Takeaways & Limitations
ByT5 provides a token-free pretrained alternative that remains competitive despite pretraining on 4× less text than mT5.
Takeaways & Limitations
The approach costs +33% pretraining time and can be up to 10× slower during inference in the worst case.
Abstract
from arXiv · showhide
Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: they can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.
1 Introduction
Tokenization introduces out-of-vocabulary and spelling-related weaknesses, while byte-level processing offers broader text coverage but faces longer sequences. ByT5 adapts a standard Transformer to bytes and remains competitive with a subword baseline.
- Fixed word vocabularies cannot distinguish different out-of-vocabulary words when they are mapped to the same <UNK> token.
- Subword tokenizers reduce out-of-vocabulary problems but can change representations substantially under typos, spelling, capitalization, morphology, or unknown characters.
- Byte-level models process arbitrary text sequences directly, avoid vocabulary building and tokenization, and require only 256 byte embeddings.
- Byte sequences are significantly longer than token sequences, increasing computational cost and motivating efficient architectures in prior work.
- ByT5 straightforwardly adapts the Transformer for byte sequences and is competitive with a subword-level baseline across diverse English and multilingual tasks.The models were pre-trained on 4× less text than the baseline, and the paper releases the models, code, and data.
2 Related Work
Related work spans character-aware and token-free models, but many approaches retain tokenization or target specific tasks. ByT5 instead develops a general-purpose byte-level pretrained model by minimally modifying an existing token-based model.
- Early neural language models operated directly on character sequences, while later work used character-level modeling to benchmark neural architectures.
- Character-aware architectures add character features but often still rely on tokenizers to identify word boundaries.
- Recent token-free pretrained models include multilingual character-level models and CANINE, although CANINE-S uses tokenization to define pretraining targets.
3 ByT5 Design
ByT5 replaces SentencePiece tokenization with UTF-8 bytes and modifies masking and encoder-decoder depth. The design reallocates vocabulary parameters while confronting longer byte sequences and their computational costs.
- Changes from mT5: ByT5 feeds UTF-8 bytes directly into the model, using 256 byte values plus three reserved special-token IDs instead of a SentencePiece vocabulary.
- Changes from mT5: The pretraining task masks spans averaging 20 bytes and reuses the final 100 byte IDs as sentinel identifiers.
- Changes from mT5: A 3:1 encoder-to-decoder depth ratio gives ByT5 a heavier encoder, which performs better on both classification and generation tasks in the reported ablation.
- Comparing the Models: Byte-level design shifts parameters away from vocabulary matrices, whose mT5-Base input and output matrices comprise about 66% of total parameters.
- Comparing the Models: Longer byte sequences raise Transformer computation because self-attention has quadratic time and space complexity in sequence length.
- Comparing the Models: For fixed input length and training steps, byte-level pretraining exposes the model to roughly 4× less actual text than subword-level pretraining.
4 Core Results
ByT5 is competitive with mT5 across English and multilingual benchmarks, with particular strengths in generation and word-level tasks. Its relative advantage varies by model size and language, while it consistently improves on several spelling- and pronunciation-sensitive evaluations.
- Overall results: ByT5 is competitive with mT5 across standard English and multilingual benchmarks, outperforming mT5 at small model sizes.The comparison covers a wide range of downstream tasks.
- English classification: ByT5 beats mT5 on GLUE and SuperGLUE at Small and Base sizes, whereas mT5 has the advantage at larger sizes.The models are parameter-matched, and the authors attribute the smaller-model pattern partly to ByT5’s larger dense-parameter allocation.
- Cross-lingual benchmarks: In the in-language XTREME setting, ByT5 surpasses mT5 on all tasks and model sizes, while larger-model translate-train results are mixed.At smaller sizes, ByT5 also beats mT5 in the translate-train setting.
- Cross-lingual benchmarks: Across languages, ByT5 is better on TyDiQA-GoldP while mT5 is consistently better on XNLI; higher compression rates tend to favor mT5.Lower-compression languages such as Indonesian and Vietnamese tend to favor ByT5, whereas Thai and Telugu tend to favor mT5.
5 Experiments on Synthetic Noise
The noise experiments compare ByT5 and mT5 under several synthetic corruptions, including noise seen during training and noise introduced only at evaluation. ByT5 generally degrades less, although some unseen noise remains highly damaging.
- Noise settings: The experiments apply Drop, Repetitions, Antspeak, Uppercase, and Random case corruptions to inputs.Drop removes characters probabilistically, Repetitions appends repeated characters, and Antspeak capitalizes and spaces characters.
- Learnable noise: ByT5 degrades less than mT5 across all learnable noise conditions on XNLI and TyDiQA-GoldP.Degradation is measured as the task-metric difference between clean and noisy settings.
- Learnable noise: Under random case, mT5 loses −25.7 and −14.3 points, while ByT5 drops only −1.5 and −0.2 points.These are the largest reported contrasts in the learnable-noise setting.
- Unseen noise: With noise unseen during training, ByT5 is again more resilient, showing only minor degradation for casing noise.Some unseen noise, including Antspeak, is highly detrimental despite the overall advantage.
- Interpretation: The results align with the broader conclusion that token-free models are more robust to noise across many tasks.The paper relates its findings to prior character-level comparisons against BPE and word-based models.
6 Ablation Study
The ablations show that ByT5’s performance is not explained solely by wider Transformer layers and benefits from a heavy encoder. They also examine depth allocation, masking span length, vocabulary choice, and inference speed.
- Model size: A 668M ByT5 model with mT5-Large-matched layer widths remains competitive and beats similarly sized mT5-Base by a large margin.Its size is approximately 54% of ByT5-Large and mT5-Large.
- Encoder-decoder depth: Balanced 24/24 and heavy-decoder 12/36 ByT5 configurations underperform the default heavy-encoder setup on all tasks.The alternative configurations also contain more parameters than the default heavy-encoder model.
- Encoder-decoder depth: mT5 gains +0.4, +1.8, and +0.7 from a heavier encoder, compared with ByT5 gains of +2.9, +4.8, and +5.2.The comparison spans XNLI zero-shot, TyDiQA-GoldP, and GEM-XSum.
- Pre-training objective: ByT5 uses a mean pre-training span length of 20 bytes because shorter byte spans would often reconstruct only part of a word.The ablations compare mean span lengths of 3, 20, and 40 bytes.
- Vocabulary choice: CharT5 is slightly worse than ByT5 on all three ablation tasks despite matching its overall parameter count.The paper attributes this partly to capacity reserved for rare characters and to UTF-8 sequence-length effects for non-Latin scripts.
7 Speed Comparisons
ByT5 incurs higher training and fine-tuning costs than mT5, with inference slowdowns that depend strongly on input and target length. The authors nevertheless judge the added cost worthwhile for non-latency-sensitive applications.
- ∼1.2× more operations and ∼0.75× as many sequences per second summarize ByT5’s pre-training cost across model sizes.These measurements use fixed hardware and sequence length 1024.
- ByT5 is similarly fast to mT5 on SIGMORPHON 2020 Grapheme-to-Phoneme but 1.5 to 2.6 times slower on Dakshina transliteration.
- ByT5 is 3.7 to 6.4 times slower on GEM-XSum and 6.4 to 9.5 times slower on XNLI, where inputs are longer or targets are short.
- ByT5-Large required 1.2×, 2.6×, and 4.5× as many fine-tuning steps as mT5-Large on XNLI, TyDiQA-GoldP, and GEM-XSum, respectively.ByT5-Large instead took 2.5× fewer steps on Dakshina.
- The authors consider roughly +33% wall time and some additional fine-tuning cost justified for non-latency-sensitive applications.
8 Conclusion
ByT5 is a token-free multilingual T5 variant that removes vocabulary building, preprocessing, and tokenization while remaining competitive with mT5. Its benefits include robustness, broad task quality, and data efficiency, but computation and some large English classification settings remain limitations.
- ByT5 removes vocabulary building, text preprocessing, and tokenization while remaining competitive with parameter-matched mT5 models.
- ByT5 outperforms mT5 under five conditions: smaller than 1 billion parameters, generative tasks, in-language multilingual labels, spelling- or pronunciation-sensitive tasks, and noisy inputs.
- ByT5 slightly underperforms mT5 on English classification tasks for model sizes over 1 billion parameters.
- A heavier encoder and masking of longer ID sequences improve byte-level encoder-decoder models in the reported ablations.
- ByT5 achieves its observed gains despite being pre-trained on 4× less text than mT5, suggesting greater data efficiency.
- These gains cost +33% pre-training time and up to 10× slower inference in the worst case.
- The authors identify local attention, down-sampling, hash embeddings, and sparse computation as possible ways to reduce latency.