PaperlayerView exact revision on arXiv

Source-linked AI summary

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei

arXiv:2109.07306v1cs.CL

TL;DR

Limited shared vocabulary capacity can leave languages under-represented, while expanding the vocabulary slows pre-training. The paper introduces VOCAP for language-specific capacity allocation and k-NN target sampling for softmax approximation; experiments report improved vocabulary-construction results and faster pre-training with comparable performance.

  • Problem

    Shared multilingual vocabularies may under-represent languages, and it is difficult to determine whether each language has enough capacity.

    Source: §1 · p. 1

  • Method

    VOCAP allocates vocabulary capacity using language-specific average log probability and pre-training corpus size, while k-NN target sampling approximates the expensive softmax.

    Source: §1 · pp. 1–2

  • Results

    On XTREME’s three tasks across seven datasets, VOCAP consistently outperforms previous vocabulary construction methods, while k-NN target sampling accelerates pre-training with comparable performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Average log probability is a valid indicator of language-specific vocabulary capacity and correlates with downstream task performance on that language.

    Source: §6 · p. 9

Abstract

from arXiv · show

Compared to monolingual models, cross-lingual models usually require a more expressive vocabulary to represent all languages adequately. We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity. To this end, we propose an algorithm VoCap to determine the desired vocabulary capacity of each language. However, increasing the vocabulary size significantly slows down the pre-training speed. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax. Our experiments show that the multilingual vocabulary learned with VoCap benefits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mitigates the side-effects of increasing the vocabulary size while achieving comparable performance and faster pre-training speed. The code and the pretrained multilingual vocabularies are available at https://github.com/bozheng-hit/VoCapXLM.

1 Introduction

Cross-lingual models share vocabularies that may under-represent some languages, while it is difficult to determine each language’s needed capacity. The paper proposes VOCAP for language-specific allocation and k-NN target sampling to reduce the pre-training cost of larger vocabularies.

1 Introduction

  • State-of-the-art cross-lingual models use 250K subword units for more than 100 languages, with no more than 2.5K language-specific units allocated per language on average.Multilingual segmentation tends to select shared units for languages with common scripts, making the required capacity for each language difficult to determine.Source: §1 · p. 1
  • VOCAP uses each language’s average log probability and pre-training corpus size to estimate required vocabulary capacity, since excess capacity for low-resource languages is inefficient.The paper reports that average log probability correlates with downstream task performance.Source: §1 · pp. 1–2
  • Increasing vocabulary size slows pre-training and increases model size; k-NN target sampling approximates the expensive softmax, while halving embedding dimensions addresses model size.The paper reports that increasing vocabulary from 250K to 500K can improve performance with fixed model capacity.Source: §1 · p. 2
  • VOCAP allocates appropriate vocabulary capacity for each language in the shared multilingual vocabulary of cross-lingual models.This is the paper’s stated vocabulary-construction contribution.Source: §1 · p. 2
  • On XTREME’s three tasks across seven datasets, VOCAP consistently outperforms previous vocabulary construction methods, while k-NN target sampling accelerates pre-training with comparable performance.The evaluation covers seven datasets.Source: §1 · p. 2

2 VOCAP: Language-Specific Vocabulary Capacity Allocation

The section investigates language-specific vocabulary capacity and proposes VOCAP to allocate shared vocabulary using ALP and pretraining corpus size. It reports that ALP correlates with downstream performance and that VOCAP500K outperforms JOINT500K across resource groups, especially mid- and low-resource languages.

2.1 Investigating Language-Specific Vocabulary Capacity

  • ALP quantifies a language's vocabulary capacity using average log probability on its monolingual corpus, penalizing low-frequency subword units.The investigation evaluates downstream performance of monolingual models trained with vocabularies of differing ALP.Source: §2.1 · p. 2

2.1.1 Setup

  • The experiments use four languages from different families and corpus-size ranges, with monolingual vocabularies sized from 1K to 30K.Models are evaluated on NER and POS; vocabularies are learned from reconstructed CommonCrawl data, while Wikipedia is used for pretraining.Source: §2.1.1 · pp. 2–3

2.1.2 Observations

  • ALP correlates positively with downstream task performance and is a better indicator than vocabulary size, while reflecting language-specific characteristics.The reported task comparisons cover POS and NER.Source: §2.1.2 · pp. 3–4, Figure 2 · p. 3, Figure 3 · p. 3

2.2.1 VOCAP Implementation

  • VOCAP allocates language-specific vocabulary to maximize overall ALP while accounting for pretraining corpus size and language sampling probabilities.Because ALP gains from increasing vocabulary size diminish monotonically, the allocation can be solved greedily.Source: §2.2.1 · p. 4

2.2.2 Intrinsic Analysis

  • VOCAP's multilingual-vocabulary comparison evaluates ALP on each language's monolingual corpus against vocabularies learned directly with SentencePiece.The section distinguishes low-resource languages as having less than 1GB of pretraining corpus and high-resource languages as having more than 10GB; other languages are mid-resource.Source: §2.2.2 · p. 4
  • VOCAP500K consistently outperforms JOINT500K across resource groups, especially for mid- and low-resource languages.The comparison covers 86 languages with more than 0.1 GB of corpus data; the vocabulary-learning corpus uses α = 0.7.Source: §2.2.2 · pp. 4–5

3 Accelerate Large-Vocabulary Language Model Pre-Training

Large vocabularies benefit cross-lingual models but make pretraining slower and models heavier; this section presents softmax and embedding-dimension strategies to address those costs.

3.1 k-NN-Based Target Sampling

  • k-NN target sampling approximates the full-vocabulary masked-language-modeling softmax by restricting each masked target’s loss to a subset of its nearest embedding neighbors.Neighbor sets are refreshed every n steps, with n chosen so |V| ≪ n × |W|; similar subwords provide difficult noise samples.Source: §3.1 · p. 5

3.2 Reducing the Embedding Dimension

  • When vocabulary size doubles, halving input and output embedding dimensions and projecting to the Transformer hidden dimension keeps parameter count fixed.This rebalancing slightly degrades model performance while improving pretraining speed and decreasing model size.Source: §3.2 · p. 5

4 Experiments

Experiments compare vocabulary construction and softmax-acceleration strategies across cross-lingual understanding tasks. The analysis examines their performance, cost, and effects across language-resource levels.

4.1 Setup

  • Experiments evaluate seven XTREME datasets spanning classification, span extraction, and sequence labeling, using 12-layer models trained for 1 million updates.Fine-tuning results are averaged over five random seeds.Source: §4.1 · p. 6

4.2 Results

  • VOCAP500K improves over VOCAP250K on six datasets except PAWS-X, while JOINT500K does not improve over JOINT250K.VOCAP250K improves question answering but degrades on PAWS-X, POS, and NER versus XLM-R250K and JOINT250K.Source: §4.2 · p. 6

4.3 Analysis and Discussion

  • NCE and negative sampling perform significantly worse than k-NN and target sampling with VOCAP500K.The authors attribute this to NCE and negative sampling needing more training steps to converge; shared negatives are more efficient on GPUs.Source: §4.3 · p. 7, Table 2 · p. 7
  • With k=5, downstream performance does not significantly degrade versus original softmax, while smaller k enables faster pre-training.The experiments use k=50, balancing performance, speed, and memory for storing k-NN indices.Source: §4.3 · p. 7, Table 3 · p. 7
  • Increasing β improves XNLI but degrades NER; the experiments use β=0.7 after considering overall performance.β controls the number of selected language-specific subword units, and higher-resource languages predominate in XNLI while NER includes more low-resource languages.Source: §4.3 · pp. 7–8, Table 4 · p. 7
  • Combined k-NN sampling and half embedding dimensions yield a 1.35-times speed-up and more than 1 point improvement at similar model size versus 250K-vocabulary models.VOCAP500K+k-NN achieves the best performance in comparisons of downstream performance and pre-training cost.Source: §4.2 · p. 7, Figure 5 · p. 8, §4.3 · p. 8
  • Models with 500K vocabularies consistently outperform 250K models on XNLI, and MLQA performance degrades with continued training only for the 250K model.VOCAP500K+k-NN achieves the best performance versus pre-training cost.Source: §4.3 · p. 8, Figure 5 · p. 8
  • VOCAP significantly outperforms SentencePiece vocabularies on mid- and low-resource languages on XNLI and NER.The authors report that VOCAP provides more vocabulary-capacity benefit to low-resource than high-resource languages.Source: §4.3 · p. 8, Figure 6 · p. 8

5 Related Work

Prior work constructs multilingual vocabularies from combined corpora or language clusters, while softmax approximations use architectural changes or sampling. VOCAP quantifies language-specific capacity, and k-NN target sampling belongs to the sampling-based family.

Multilingual Vocabulary Construction

  • mBERT, XLM, and XLM-R learn multilingual vocabularies on combined corpora using WordPiece, BPE, and SentencePiece unigram modeling, respectively.Chung et al. (2020b) instead cluster languages, learn cluster-specific vocabularies, and combine them.Source: §5 · p. 8
  • VOCAP differs from clustered vocabulary construction by quantifying each language’s capacity needs using average log probability and pre-training corpus size.Source: §5 · p. 8

Softmax Approximation

  • Softmax approximations include architectural methods and sampling-based losses; k-NN-based target sampling is sampling-based.The passage describes architectural changes as unsuitable for GPU or multilingual settings.Source: §5 · p. 9

6 Conclusion

The paper proposes VOCAP for multilingual vocabulary construction and k-NN target sampling for faster pre-training with large vocabularies. Experiments support the vocabulary and acceleration methods, including reduced embedding dimensions to control parameter count.

6 Conclusion

  • VOCAP allocates multilingual vocabulary capacity using each language’s average log probability and pre-training corpus size.Source: §6 · p. 9
  • k-NN-based target sampling approximates the expensive softmax to accelerate pre-training with a large multilingual vocabulary.Source: §6 · p. 9
  • Reducing embedding dimension is reported as an effective way to retain large-vocabulary improvements without increasing model parameters.Source: §6 · p. 9

A Correlation between Language-Specific Vocabulary Capacity and Task Performance

The analysis compares how average log probability (ALP) and vocabulary size relate to downstream task performance. ALP correlates better with that performance than vocabulary size does.

A Correlation between Language-Specific Vocabulary Capacity and Task Performance

C Fine-tuning Settings

Fine-tuning settings vary by task and use multilingual development results where available. For XQuAD and TyDiQA-GoldP, the paper specifies alternate development-set choices.

C Fine-tuning Settings

  • POS uses average-pooled subword representations, with the model selected by best average development results across languages.For XQuAD, hyperparameters are tuned on MLQA development data because the datasets share training data and have greater language overlap; TyDiQA-GoldP uses the English test set as development data.Source: C Fine-tuning Settings · pp. 11–12
  • 10 epochs are used for XNLI, PAWS-X, POS, and NER; XQuAD uses 4, while TyDiQA-GoldP uses 6 or 8 selected against English test performance.Learning rates are selected from [7e-6, 1e-5] for XNLI and PAWS-X, [1e-5, 2e-5] for POS and NER, and [2e-5, 3e-5] for XQuAD, MLQA, and TyDiQA-GoldP.Source: C Fine-tuning Settings · p. 12

D Pre-Training Data

The paper learns vocabularies from reconstructed CommonCrawl and conducts pre-training on the smaller Wikipedia corpus. It includes only languages shared by both corpora.

D Pre-Training Data

  • Vocabulary learning uses the reconstructed CommonCrawl corpus from Chi et al. (2021b).The paper pre-trains on Wikipedia instead because tokenizing the pre-training data is time-consuming and Wikipedia is smaller.Source: D Pre-Training Data · p. 12
  • Only languages shared by reconstructed CommonCrawl and Wikipedia are considered for pre-training.Source: D Pre-Training Data · p. 12
  • Table 8 lists statistics for the Wikipedia corpus used for pre-training, while Table 7 lists statistics for the reconstructed CommonCrawl corpus used for learning vocabularies.Table 9 reports statistics of allocated vocabulary size for each language.Source: D Pre-Training Data · p. 12, Table 7 · p. 12, Table 8 · p. 13
Loading 2109.07306v1…