Source-linked AI summary
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa
TL;DR
Multilingual models have kept vocabulary size largely fixed as their parameters and depth have grown, limiting vocabulary capacity across more than 100 languages. XLM-V constructs a one-million-token vocabulary by reducing sharing between lexically dissimilar languages and allocating capacity across languages. It outperforms XLM-R across tested multilingual tasks, with particularly strong reported gains on low-resource tasks.
Problem
Multilingual models have kept vocabulary size largely unchanged as they scale, leaving limited capacity to represent the more than 100 languages that share a vocabulary.
Source: §1 · p. 1Method
XLM-V constructs a one-million-token vocabulary by de-emphasizing sharing between languages with little lexical overlap and allocating vocabulary capacity to individual languages.
Source: §1 · p. 1Results
3.5 points absolute was the average by which XLM-V outperformed XLM-R across tested multilingual language-understanding tasks.
Source: §1 · p. 2Takeaways & Limitations
XLM-V also produces semantically meaningful tokenizations, reduces average sequence length, and serves as an efficient form of conditional compute.
Source: §8 · p. 9Scaling vocabulary can increase pretraining time through the full-vocabulary softmax and can significantly increase model memory footprint.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This \textit{vocabulary bottleneck} limits the representational capabilities of multilingual models like XLM-R. In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), to named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 11.2% and 5.8% absolute on MasakhaNER and Americas NLI, respectively.
1 Introduction
The paper argues that multilingual models have not expanded their vocabularies alongside their parameter counts, leaving limited vocabulary capacity for many languages. It introduces a vocabulary construction approach and XLM-V, which outperforms XLM-R across tested multilingual tasks, with especially large gains on low-resource tasks.
1 Introduction
- Multilingual models with far more parameters have continued to use a 250K-token vocabulary, including models sharing it across more than 100 languages.The paper notes that this amounts to an average of 2,500 unique tokens per language, and cites prior work linking the vocabulary bottleneck to question-answering and sequence-labeling performance.Source: §1 · p. 1
- The proposed vocabulary construction de-emphasizes sharing between languages with little lexical overlap and allocates capacity to represent individual languages.The paper reports more semantically meaningful tokenizations and reduced over-tokenization for low-resource languages.Source: §1 · p. 1
- The paper improves language clustering representations and vocabulary-capacity assignments to construct large multilingual vocabularies.XLM-V is a one-million-token multilingual language model trained on 2.5TB of Common Crawl data.Source: §1 · p. 1
- XLM-V outperformed comparable baselines with the same vocabulary size on XNLI.The paper also reports evidence that expanding the vocabulary beyond 1M tokens can degrade downstream performance.Source: §1 · pp. 1–2
- 3.5 points absolute was the average by which XLM-V outperformed XLM-R across tested multilingual understanding tasks, including XNLI, WikiAnn, MLQA, XQuAD, and TyDiQA.On MasakhaNER, XLM-V outperformed XLM-R by 11.2% absolute F1; on AmericasNLI, it outperformed XLM-R by 5.8% absolute accuracy.Source: §1 · p. 2
2 Background
This section introduces subword vocabulary construction, multilingual language clustering, and a method for allocating vocabulary capacity across languages. It presents prior approaches that the paper builds on.
2.1 Sentencepiece
- The Unigram Language Model (ULM) prunes an initial vocabulary to maximize training-corpus likelihood, then uses Viterbi decoding to find the most probable tokenization.The paper uses ULM for both XLM-R and its own work.Source: §2.1 · p. 2
2.2 Clustering
- Chung et al. (2020) cluster languages using binary representations of their per-language vocabularies, then combine vocabularies built for each cluster.Their approach balances cross-lingual subword sharing against robust representation of individual languages.Source: §2.2 · p. 2
2.3 Vocabulary allocation
- Zheng et al. (2021) introduce average log probability (ALP) to assess how well a vocabulary represents a language and propose using a greedy algorithm to allocate vocabulary capacity.They show that ALP is highly correlated with downstream task performance.Source: §2.3 · p. 2
3 Methodology
The paper constructs a multilingual vocabulary by representing and clustering languages, allocating capacity to clusters, and combining their vocabularies. It then pretrains XLM-V and notes computational and memory trade-offs associated with large vocabularies.
3.1 Building the vocabulary
- Per-language sentencepiece vocabularies provide lexical fingerprints for clustering, with token likelihoods used instead of binary presence vectors.The method samples one billion CC100 lines with temperature 2 and trains a 30,000-token sentencepiece model for each language.Source: §3.1 · pp. 2–3, Figure 1 · p. 3
- The method uses Zheng et al. (2021) capacity allocations and K-Means with k = 8 to build clusters and assign vocabulary capacity.For cluster c2, capacity increased from 28,593 tokens under the prior proportional-allocation method to 102,722; uncovered tail-end languages receive a 2,000-token budget.Source: §3.1 · p. 3
- The final vocabulary contains 901,629 tokens, with 98,371 tokens overlapping between the eight cluster vocabularies.Thus, over 90% of the tokens learned in each cluster are unique.Source: §3.1 · p. 3
3.2 Training the model
- XLM-V is pretrained on CC100 using the XLM-R procedure, masked language modeling, and 1.5 million iterations.Training uses a sampling temperature of 0.3, a batch size of 8,192 across 256 A100 GPUs, and sequences up to 512 tokens.Source: §3.2 · pp. 3–4
- Without approximation tricks, the authors report that pretraining may not be feasible when vocabulary size exceeds 2M because the full-vocabulary softmax is computationally intensive.Adaptive softmax and adaptive inputs required substantial tuning and led to slower convergence and increased training instability in their experiments.Source: §3.2 · p. 4
4 Experiment setup
The experiments compare vocabulary and model baselines and evaluate cross-lingual transfer across multilingual datasets.
4.1 Baselines
- Baselines included reused XLM-R (250K), an XLM-R (1M) vocabulary, and a 1M Chung et al. (2020) clustering baseline; comparison encoders had 12 layers.The XLM-R (250K) experiments reused the public SentencePiece model and pretrained checkpoint.Source: §4.1 · p. 4
4.2 Datasets
- CC100 contains 2.5 TB across 116 languages and was used exclusively for vocabulary construction and model pretraining.Source: §4.2 · p. 4
- XNLI evaluates entailment, contradiction, or neutrality across translated premise–hypothesis pairs, while Americas NLI extends XNLI to 10 indigenous languages.Source: §4.2 · pp. 4–5
- The question-answering evaluations included MLQA, XQuAD, and typologically diverse TyDiQA, whose questions have less lexical overlap than those in MLQA or XQuAD.Source: §4.2 · pp. 4–5
- For QA, they retrieved only n-best spans with start and end indices in the passage context, improving performance for both baseline models and XLM-V.Source: §4.2 · p. 4
- Table 2 compares XLM, XLM-R, and XLM-V after English fine-tuning, summarizing task results with accuracy or F1.EM denotes exact match.Source: Table 2 · p. 5
- Named-entity recognition evaluation included WikiAnn across 48 languages and MasakhaNER across 10 African languages.Source: §4.2 · p. 5
5 Results
XLM-V improves on XLM-R in the reported full evaluations and in matched-vocabulary XNLI experiments. The strongest reported gains include low-resource language tasks.
5.1 Comparisons using partial training
- 1.34% and 1.11% absolute accuracy are XLM-V’s XNLI gains over XLM-R (1M) and Chung et al. (2020), respectively, in partial-training comparisons.The models were trained for 300K iterations with batch size 2,048 on CC100.Source: §5.1 · p. 5, Figure 2 · p. 5
5.2 Fully trained model
- XLM-V outperforms the re-implemented XLM-R on all evaluated datasets, with an average gain of 3.5 points absolute.The evaluations include XNLI, question answering, WikiAnn, and low-resource tasks.Source: §5.2 · p. 6
- 11.2% F1 is the average gain on MasakhaNER; XLM-V also improves Americas NLI by 18.2% absolute F1 on Quechua and 17.2% on Guaraní.Americas NLI uses zero-shot cross-lingual transfer; the Quechua and Guaraní results are reported in Table 4.Source: §5.2 · p. 6, Table 4 · p. 6
6 Analysis
The analysis examines vocabulary-size limits, tokenization quality and length, and training speed. It reports diminishing returns beyond 1M tokens alongside qualitative and efficiency benefits of XLM-V’s vocabulary.
6.1 The Zipf ceiling
- Beyond 1M tokens, vocabulary utility diminishes, and a 1.5M-token model performs worse than its 1M-token counterpart with equivalent training data.On FLoRes-200, 99% of content is covered by 197,817 unique tokens from the 1M vocabulary, versus 243,832 from the 2M vocabulary.Source: §6.1 · pp. 6–7, Figure 3 · p. 7
6.2 Qualitative improvements in tokenization
- XLM-V often produces semantically meaningful tokenizations, including Chinese segments corresponding to Cambridge University, undergraduates, and postgraduates.The paper also reports separating shared roots in English, French, and Spanish.Source: §6.2 · pp. 7–8
6.3 Over-tokenization
- 11.5% fewer tokens on average: XLM-V produces shorter sequences than XLM-R on FLoRes-200, though Chung et al. (2020) returns the fewest tokens for low-resource languages.The comparison covers high-, medium-, and low-resource languages.Source: Table 6 · p. 7, §6.3 · p. 8
6.4 Speed vs. size
- 3.3x more trainable parameters accompanies a 25% increase in training time when vocabulary size grows from 250K to 1M tokens.The paper frames the larger vocabulary as conditional compute because only a small fraction of the embedding matrix is used for any input.Source: §6.4 · p. 8, Figure 4 · p. 8
7 Related work
The related work contrasts larger subword vocabularies with vocabulary-free models and prior multilingual vocabulary construction. It notes speed trade-offs and data and scale constraints in earlier large-vocabulary work.
7.1 Vocabulary-free models
- Vocabulary-free models such as ByT4 and CANINE have demonstrated on-par or better performance than subword-tokenization counterparts, but one consistent drawback is slower training and inference.ByT5 is 6.4 to 9.5 times slower than mT5 on classification tasks like XNLI; CANINE remains approximately 1.6 times slower than a comparable BERT baseline.Source: §7.1 · p. 8
7.2 Building larger vocabularies
- Earlier large-vocabulary methods addressed over-tokenization and coverage, but were constrained by Wikipedia-scale pretraining data and vocabularies no larger than 500K tokens.The paper positions XLM-V as combining and improving these methods while using larger pretraining data and exploring vocabularies of 1M tokens and beyond.Source: §7.2 · p. 8
8 Conclusion
The conclusion presents XLM-V as a multilingual model with a 1M-token vocabulary and reports performance, tokenization, and efficiency benefits. It identifies several directions for further research.
8 Conclusion
- XLM-V, with a 1M-token vocabulary, outperforms XLM-R and shows outsized gains on low-resource tasks.The conclusion also reports semantically meaningful tokenizations, shorter average sequences, and use as conditional compute.Source: §8 · p. 9
- The authors propose testing vocabularies beyond 2M tokens with more data to investigate the Zipf ceiling.Source: §8 · p. 9
- Other future directions include larger multilingual vocabularies for autoregressive models and alternative clustering methods such as hierarchical clustering.Source: §8 · p. 9
Limitations
The section identifies scalability costs of a larger vocabulary: slower pre-training from the full-vocabulary softmax and a larger memory footprint.
Limitations
- A full-vocabulary softmax can increase pre-training time as vocabulary size grows.The authors suggest adaptive softmax and adaptive inputs as possible ways to address this computational cost.Source: Limitations · p. 9
- Scaling the vocabulary can significantly increase model memory footprint.The authors expect this to matter less for larger models, where non-embedding parameters outweigh the vocabulary embedding matrix.Source: Limitations · p. 9
- The authors propose adaptive softmax and adaptive inputs as potential remedies for the softmax cost.They believe these approximation techniques can solve the increased pre-training-time issue.Source: Limitations · p. 9
B Appendix
The appendix reports per-language evaluation results and the fine-tuning hyperparameters used for reproducibility.
B Appendix
- The appendix provides per-language results and task-specific fine-tuning hyperparameters for reproducibility.Source: B Appendix · p. 10
- MLQA, TyDiQA-GoldP, and XQuAD tables report question-answering results using EM/F1 metrics.The captions specify two fine-tuning epochs for MLQA and XQuAD and eight for TyDiQA-GoldP.Source: Table 7 · p. 11, Table 8 · p. 11, Table 9 · p. 11
- NER tables report results, including zero-shot MasakhaNER transfer trained on English and evaluated on unseen languages.Source: Table 10 · p. 11, Table 11 · p. 11