Source-linked AI summary
Neural Machine Translation with Byte-Level Subwords
Changhan Wang, Kyunghyun Cho, Jiatao Gu
TL;DR
Character-based vocabularies can waste capacity on rare characters, while pure byte representations are computationally costly. The paper studies byte-level BPE with contextualized embeddings and finds compact vocabularies with competitive or improved translation results, including transfer across disjoint character sets.
Problem
Character-based vocabularies can waste vocabulary slots on rare characters, while pure byte representations are computationally costly for machine translation.
Method
The paper builds BBPE vocabularies from UTF-8 bytes and contextualizes BBPE embeddings with convolutional or recurrent layers.
Results
BBPE matches or outperforms BPE across the reported settings, uses much smaller vocabularies, and transfers across languages with non-overlapping character sets.
Takeaways & Limitations
BBPE maximizes vocabulary sharing across languages and supports model transfer without out-of-vocabulary tokens for unseen character sets.
Takeaways & Limitations
The paper identifies source-target sentence-length imbalance as a remaining issue and leaves broader translation settings and improved segmentation for future work.
Abstract
from arXiv · showhide
Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.
Introduction
The paper investigates byte-level subwords, especially BBPE, as a compact, language-agnostic alternative to character-level BPE for neural machine translation. It examines BBPE in bilingual and multilingual settings and in transfer learning to languages with non-overlapping character sets.
- Introduction: Byte representation offers compactness and language agnosticism, but character-based BPE remains standard because text is often naturally represented as characters.The byte vocabulary has up to 256 possible values.
- Introduction: BBPE tokenizes text into variable-length byte n-grams rather than character n-grams.The paper focuses specifically on applying byte-pair encoding to byte-level representations.
- Introduction: The study evaluates compact BBPE vocabularies in bilingual and multilingual translation settings.
- Introduction: Figure 1 contrasts character-level BPE and BBPE tokenization of a Japanese sentence, displaying byte fragments as hexadecimal digits.
- Introduction: The paper also introduces transfer learning to a new language whose character set does not overlap with the source language.
Byte Level Text Representation
The paper represents text as UTF-8 bytes, groups bytes into variable-length BBPE subwords to reduce sequence length, and contextualizes their embeddings because byte symbols may split characters ambiguously. It also uses dynamic programming to recover valid Unicode characters during decoding.
- Byte Level Text Representation: UTF-8 encodes each Unicode character into 1 to 4 bytes, allowing sentences in many languages to be modeled as byte sequences.The representation uses 248 of the 256 possible bytes.
- Byte Level Text Representation: BBPE learns BPE merges over UTF-8 bytes to create variable-length byte n-grams that are more efficient than processing pure byte sequences.Pure byte sequences can be up to four times longer than character sequences.
- Byte Level Text Representation: Because BBPE symbols may be partial characters, the model contextualizes embeddings with either a depth-wise convolution or a bidirectional GRU.The added context helps disambiguate symbols and learn character boundaries.
- Byte Level Text Representation: During decoding, invalid byte sequences are handled by recovering as many valid Unicode characters as possible through dynamic programming.The recurrence maximizes recovered characters over candidate UTF-8 character spans.
- Byte Level Text Representation: UTF-8 structure makes the recovery solution unique because trailing bytes of a multibyte character cannot independently form valid UTF-8 characters.
Experimental Settings
Experiments cover three bilingual corpora and a 59-language many-to-English corpus, using jointly learned source-target vocabularies and Transformer models trained and evaluated with standardized settings. The datasets span English-German, Japanese-English, Sinhala-English, and multilingual translation.
- Datasets: The experiments use English-German, Japanese-English, and Sinhala-English bilingual datasets.The Japanese-English corpus combines KFTT, TED, and JESC, while Sinhala-English uses FLoRes data.
- Datasets: The many-to-English dataset contains parallel data for 59 languages, with English as target and 58 languages as sources.A 22K-example sample from the 135K development set is used for validation.
- Datasets: Source and target sentences are used jointly to learn BPE and BBPE vocabularies with SentencePiece.
- Models and Learning: The models are Transformers trained with Fairseq using the original learning-rate schedule and experiment-specific dropout settings.Convolutional layers use kernel size 5 with padding 2 on both sides.
- Inference and Evaluation: Evaluation generates predictions with beam search and reports case-sensitive tokenized BLEU computed with sacreBLEU.Beam width is 4 for English-German and 5 for the other settings.
Results and Analysis
BBPE provides compact byte-level vocabularies with broad cross-lingual sharing, while contextualization improves performance on fine-grained representations. Across bilingual, multilingual, and transfer settings, it offers a practical balance between translation quality, vocabulary size, and sequence efficiency.
- Vocabulary analysis: BBPE distributes vocabulary symbols more evenly than BPE and can decompose rare characters into byte n-grams, freeing slots for frequent symbols.This is especially useful for noisy text and character-rich languages.
- Vocabulary analysis: BBPE symbols are shared across languages with different writing systems, increasing parameter sharing in multilingual models.The shared units are byte n-grams rather than necessarily shared characters.
- Sequence lengths: BBPE produces longer tokenized sequences than BPE but significantly shorter ones than character vocabularies, even with much smaller vocabularies.Longer sequences increase training and inference time relative to BPE, but BBPE remains more efficient than character-level tokenization.
- Contextualization: Over 4% validation BLEU gain comes from Bi-GRU contextualization for BBPE in all tested vocabulary-size cases.The experiments compare no contextualization, depth-wise convolution, and Bi-GRU; contextualization reduces difficulty learning attention over multiple source tokens.
- Translation quality: BBPE is comparable to BPE on Ja-En and beats the BPE baseline on overall and most per-language BLEU in multilingual X-En experiments.In X-En, BBPE vocabularies used 12.5% and 25% of the baseline BPE vocabulary; byte and character models performed better, but BBPE was described as the most practical balance of quality and speed.
- Transfer learning: 0.9-1.8 BLEU points are gained by transferring a pretrained X-En BBPE 4K model to Si-En with a disjoint Sinhala character set.BBPE's complete UTF-8 byte coverage avoids out-of-vocabulary tokens and permits transfer without replacing the vocabulary or retraining embeddings from scratch.
Related Work
Prior NMT work explored subword, character-level, byte-level, and hybrid Transformer architectures, but BBPE specifically extends BPE using bytes as basic units.
- Subword vocabularies, including morpheme-based, BPE, and unigram-language-model vocabularies, generally outperform word-level vocabularies in many settings.
- Character-level NMT progressed from composing word representations from characters to unrestricted decoding and fully character-level multilingual models.
- Character-level models generally outperform subword-level models when given sufficient model capacity.
- GPT-2 used byte-level BPE, but relied heavily on hard-coded merging rules and did not analyze its effect on language-model quality.
- Convolutional or recurrent layers have produced performance gains alongside Transformers in machine translation, speech recognition, and language modeling.
Conclusion
The paper concludes that BBPE provides compact, transferable multilingual representations, while retaining efficiency advantages over character vocabularies and leaving several evaluation challenges for future work.
- BBPE builds byte-level subword vocabularies that are more compact than character-based vocabularies without losing performance.
- In multilingual settings, BBPE often outperforms character-based vocabularies.
- Because BBPE has no out-of-vocabulary tokens, models can be transferred between languages with non-overlapping character sets.
- At the same vocabulary size, BBPE produces shorter sequences than character-based methods, enabling faster training and inference.
- The tokenization example orders granularity from fine byte representations through BBPE and character vocabularies to coarser BPE representations.
- Future work includes reducing source-target sentence-length imbalance and evaluating BBPE in one-to-many and many-to-many translation settings.