Source-linked AI summary
CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting
TL;DR
Modern NLP models still commonly depend on explicit tokenization and fixed vocabularies, which can be poorly suited across languages and domains. CANINE instead encodes character sequences directly, using downsampling, a deep transformer, and character- or subword-based pre-training losses. It is reported to surpass mBERT on TyDi QA while using fewer parameters, with limitations identified in rare-item and long-entity recognition.
Problem
Common NLP models still require explicit tokenization, while fixed vocabularies can be poorly suited to languages, domains, and orthographic variation.
Method
CANINE encodes Unicode character sequences directly and combines downsampling with a deep transformer stack and either character- or subword-based MLM losses.
Results
CANINE surpasses mBERT on TyDi QA, with CANINE-S improving by 2.8 F1 while using about 30% fewer parameters.
Takeaways & Limitations
CANINE provides a pre-trained tokenization-free, vocabulary-free encoder that surpasses models built on heuristic tokenizers.
Takeaways & Limitations
The final upsampling layer cannot be straightforwardly combined with block-wise local attention and is left as future work; CANINE-C also struggles with rare lexical items and long entities.
Abstract
from arXiv · showhide
Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.
1 Introduction
CANINE is introduced as a tokenization-free encoder that processes Unicode character sequences directly, while using downsampling and pre-training objectives to make character-level modeling practical.
- Motivation: Explicit tokenization remains a common preprocessing requirement despite the shift toward end-to-end neural NLP models.Tokenization is described as one of the few holdovers from traditional NLP pipelines.
- Motivation: Transformer computation grows quadratically with input length, making direct character-level replacement substantially slower than subword modeling.Standard subword models average roughly four characters per subword, creating an approximately 4x longer character input.
- CANINE: CANINE encodes raw Unicode character sequences directly, without requiring an input vocabulary or explicit tokenization.Its inputs are sequences of Unicode characters, and tokenization is absent from the encoder input.
- Architecture: CANINE uses strided convolutions to downsample character sequences before applying a deep transformer stack.Downsampling reduces the sequence length and avoids the slowdown associated with longer character inputs.
- Pre-training: CANINE retains MLM and NSP pre-training, with MLM losses based either on autoregressive character prediction or masked subword prediction.When subwords are used, they affect only the pre-training loss; tokens are never input to the encoder.
- Contributions: The paper contributes a pre-trained tokenization-free encoder, an efficient long-character-sequence architecture, and input processing without a lossy tokenization bottleneck.The stated contributions emphasize both efficiency and avoiding information loss during preprocessing.
2 Motivation
The paper motivates tokenization-free modeling by identifying linguistic and practical weaknesses in fixed, mostly word-splitting vocabularies across languages, domains, and orthographic variation.
- Linguistic pitfalls of tokenization: Subword tokenizers mainly perform simple word-splitting operations, which can mishandle agglutinative and non-concatenative morphology.Arabic conjugation can interleave root letters with inflection, so the root cannot be separated by a contiguous split.
- Domain and orthographic variation: Informal text containing typos, spelling variation, transliteration, or emoji can expose vocabulary coverage and corruption sensitivity in subword models.The passage specifically describes BERT as sensitive to natural and adversarial input corruptions.
- Linguistic pitfalls of tokenization: Whitespace and punctuation heuristics are problematic for languages without spaces between words or languages that use punctuation as letters.Examples include Thai and Chinese, and Hawaiian and Twi.
- Domain and orthographic variation: Fixed vocabularies force tradeoffs between preserving orthographic information and preventing variant forms from becoming too infrequent for vocabulary inclusion.The paper gives accents and casing as examples, including mBERT’s removal of diacritics and resulting Spanish information loss.
- Downstream adaptation: Pre-training vocabularies can constrain downstream adaptation when their segmentation is poorly suited to a target domain such as medical text.Excessive segmentation of medical data is reported to diminish the benefit of pre-training.
- Research motivation: The paper seeks models that can learn tokenization and generalize beyond orthographic forms encountered during pre-training.Its motivation includes balancing composition of words with memorization where memorization is useful.
3 CANINE
CANINE is a tokenization-free character encoder that combines vocabulary-free embeddings, efficient downsampling and upsampling, and a deep transformer stack. Its pre-training can use character-level or subword-based losses, while tokenization remains only a discarded soft inductive bias.
- Architecture: CANINE combines vocabulary-free character embeddings, downsampling, a primary transformer encoder, and task-dependent upsampling.The model is composed as DOWN, ENCODE, and UP; upsampling restores character-level outputs for sequence prediction tasks.
- Efficiency: Downsampling reduces the character sequence before the deep transformer stack, avoiding the linear feed-forward and quadratic attention costs of full-length character processing.The architecture uses local attention and strided convolutions before the main encoder.
- Architecture: CANINE directly encodes Unicode character sequences without requiring a fixed vocabulary or tokenization algorithm during model use.Inputs are represented as Unicode codepoints, and hashing supports embeddings across the full codepoint space.
- Embeddings: Character hash embeddings concatenate representations from multiple hash functions, reducing the chance that collisions produce identical codepoint embeddings.Each hash function contributes a portion of the final representation, so collisions across all functions are unlikely to coincide.
- Modularity: The tokenization-free design permits fine-tuning on previously unseen characters and processing data produced by different tokenizers.Pre-training tokenization components are not required during fine-tuning or prediction, avoiding lifecycle mismatches between training phases.
- Pre-training: CANINE pre-training retains MLM and supports either autoregressive character prediction or masked subword-token prediction.Subword tokenization is used only to compute the pre-training loss; tokens are never supplied to the encoder and the vocabulary can be discarded afterward.
4 Experiments
Experiments compare CANINE with mBERT on multilingual question answering and named entity recognition, then test architectural choices through ablations. CANINE improves substantially on TyDi QA and supports strong multilingual and tokenization-free behavior, while some design changes reduce quality or speed.
- TYDI QA Results: 2.8 F1 improves CANINE-S over mBERT on TYDI QA SELECTP with about 30% fewer parameters.CANINE-C improves over mBERT by 2.5 F1; adding vocabulary-free character n-grams increases the gain to +3.8 F1 on SELECTP and +6.9 F1 on MINSPAN.
- TYDI QA Results: A pure character-vocabulary baseline is 10X slower than subword BERT and suffers greatly in quality, while adding subword-based masking improves quality.In CANINE, subwords affect only the pre-training loss and are not encoder inputs.
- TYDI QA Results: CANINE performs particularly well on morphologically rich Kiswahili examples where mBERT’s WordPiece segmentations misalign related forms.Examples include matching milikiwa with iki-milikiwa and i-sambaza-yo with u-sambaza-ji.
- Ablations: More aggressive downsampling brings substantial speed gains but substantially degrades passage-level quality, especially minimal-span predictions.The reported comparison uses 5X or 6X downsampling instead of 4X.
- Ablations: Removing position-limited MLM computation causes a large speed reduction, while reducing FLOPS through another attention change does not materially improve pre-training throughput and substantially degrades quality.The position-limited MLM model is theoretically equivalent in operations; the other change reduces attention computation.
- NER Results: On NER, CANINE-C performs significantly worse than mBERT, but tokenization-free n-gram features restore performance and enable inexpensive memorization while remaining tokenization-free.NER evaluates sequence labeling across European and African-language datasets, including CoNLL and MasakhaNER.
5 Related Work
Related work spans improved subword tokenization, character-aware and tokenization-free models, and character-level approaches for language understanding and generation. Multilingual NLP has largely remained centered on shared subword vocabularies.
- Subword tokenization: Subword regularization and BPE-dropout address limitations of deterministic segmentation by sampling among multiple tokenizations.Unigram LM was reported to align with morphology better than BPE on pre-trained encoders.
- Hybrid models: Hybrid models combine characters with tokens or multiple subword vocabularies rather than eliminating tokenization entirely.
- Character-level representations: Character-level features remain important for morphologically rich and low-resource languages, despite generally lagging behind word-level counterparts.
- Character language modeling: Earlier character language models used RNNs, while later work explored hierarchical segmentation, hybrid generators, and external resources to improve performance.
- Character Transformers: Byte-level vanilla Transformers significantly underperformed word-level counterparts, leaving subword Transformers as the language-modeling status quo.
- Applications: Character-level methods have supported machine translation and many downstream tasks, including classification, tagging, parsing, and machine reading comprehension.
- Multilingual NLP: Multilingual NLP has been dominated by deeply pretrained models that share subword vocabularies across more than 100 languages.
6 Conclusion
The conclusion presents CANINE as a tokenization-free, vocabulary-free pretrained encoder for language understanding that surpasses models built on heuristic tokenizers. It also highlights reduced engineering pitfalls and new research directions.
- CANINE is presented as the first pretrained deep encoder for language understanding that is both tokenization-free and vocabulary-free.
- CANINE surpasses the quality of models built on heuristic tokenizers.
- The authors state that CANINE eliminates many engineering pitfalls for practitioners and opens new research directions.
A Appendix
The appendix provides language-wise breakdowns for TYDI QA and named entity recognition. It notes that English is excluded from TYDI QA’s overall score and that mBERT scores zero on Amharic because its vocabulary lacks Amharic-script entries.
- Table 7 reports TYDI QA primary-task results broken down by language, with English parenthesized because it is excluded from the overall score.
- Table 8 reports labeled F1 for NER on CoNLL and MasakhaNER, including mBERT’s zero Amharic score caused by no vocabulary entries in the Amharic script.