PaperlayerView exact revision on arXiv

Source-linked AI summary

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych

arXiv:2012.15613v2cs.CL

TL;DR

The paper asks whether multilingual models underperform monolingual counterparts on monolingual tasks and what explains any gap. It performs controlled comparisons across nine languages and five tasks, then disentangles data size from tokenizer choice. The results identify tokenizer specialization as an important factor and show that replacing multilingual tokenizers generally improves multilingual-model performance.

  • Problem

    The paper addresses the lack of compelling, rigorous comparisons establishing whether multilingual models outperform or underperform monolingual counterparts across languages and monolingual tasks.

  • Method

    The authors compare models across nine languages and five tasks, then train models on equally sized data with multilingual versus dedicated monolingual tokenizers.

  • Results

    Performance gaps occur in most but not all languages and tasks, while monolingual tokenizers outperform the mBERT tokenizer for almost every task and language.

  • Takeaways & Limitations

    Improving multilingual vocabulary balance and adding language-specific tokenizer adaptations can reduce the gap between monolingual and multilingual models.

  • Takeaways & Limitations

    The evaluation requires target-language training data, preventing use of multilingual benchmarks without training portions in languages other than English.

Abstract

from arXiv · show

In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the corresponding monolingual representation of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data, with monolingually and multilingually trained tokenizers. We find that while the pretraining data size is an important factor, a designated monolingual tokenizer plays an equally important role in the downstream performance. Our results show that languages that are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts. We further find that replacing the original multilingual tokenizer with the specialized monolingual tokenizer improves the downstream performance of the multilingual model for almost every task and language.

1 Introduction

The paper addresses whether multilingual language models underperform language-specific monolingual models and investigates whether data size or tokenizer choice explains any gap.

  • Prior claims that multilingual models underperform monolingual models lack rigorous, fair empirical support across languages and tasks.
  • The study systematically compares multilingual and monolingual pretrained models across 9 typologically diverse languages and 5 structurally different tasks.
  • The authors disentangle pretraining-data size from tokenizer vocabulary by training monolingual models on equally sized datasets with shared multilingual or dedicated language-specific tokenizers.
  • They analyze tokenizer fertility, unseen (sub)words, data size, and related factors to explain downstream performance differences.
  • Monolingually adapted tokenizers can robustly improve the monolingual performance of multilingual models.

2 Background and Related Work

Earlier research shows multilingual models can work well but may suffer from capacity, data-balance, and vocabulary limitations. This paper addresses the remaining need for controlled comparisons across diverse languages and tasks.

  • mBERT concatenates the 104 largest Wikipedias and uses a shared WordPiece vocabulary built from multilingual pretraining data.
  • The shared mBERT vocabulary contains 119,547 subword tokens.
  • Multilingual models can perform strongly on cross-lingual tasks without direct cross-lingual supervision, but their massively multilingual extensions may still face the curse of multilinguality.
  • With fixed model capacity, adding languages eventually causes performance drops, while resource-poor languages remain underrepresented in data, vocabulary, and shared representations.
  • Prior monolingual-versus-multilingual studies report task-specific gaps but often lack verified scores, controlled impartial comparisons, or broad language and task coverage.

3 Controlled Experimental Setup

The experiments compare mBERT with monolingual BERT models across diverse languages and tasks, using controlled fine-tuning and evaluation procedures. Initial results show that performance gaps vary substantially by language and task.

  • Language and task selection: Languages were chosen for typological diversity, available pretrained monolingual BERT models, and representation in benchmarks with sufficient task coverage.
  • Language and task selection: The study compares mBERT with monolingual counterparts across 9 languages from 8 language families and 5 downstream NLP tasks.
  • Language and task selection: The selected tasks are named entity recognition, sentiment analysis, question answering, universal dependency parsing, and part-of-speech tagging.
  • Task-based fine-tuning: The evaluation uses standard fine-tuning for most tasks, a transformer-based dependency parser for UDP, and task-specific metrics including F1, accuracy, UAS, LAS, exact match, and F1.
  • Initial results: Performance gaps exist to a large extent, but their size depends strongly on the language and task; monolingual gains are largest for FI, TR, KO, and AR.
  • Initial results: mBERT outperforms IndoBERT in all tasks except SA, performs competitively with JA and ZH models on most datasets, and has especially narrow gaps on POS tagging.

4 Tokenizer versus Corpus Size

The experiments separate pretraining-data size from tokenizer choice by comparing models trained on matched data with multilingual or dedicated monolingual tokenizers. Both data size and tokenizer quality align with downstream performance, with dedicated tokenizers usually improving results.

  • Pretraining corpus size: Monolingual BERTs for AR, ID, FI, KO, and TR used about twice to more than 40 times as much language data as mBERT.The largest discrepancy was reported for TR, while KO had about twice as much data.
  • Tokenizer statistics: mBERT shows much higher subword fertility for AR, FI, KO, RU, and TR, indicating over-segmentation in those languages.Fertility is the average number of subword tokens produced per tokenized word; lower scores indicate a better-suited tokenizer.
  • Tokenizer statistics: mBERT produces a proportion of continued words more than twice as high as the monolingual model for ID, alongside differences for AR, FI, KO, RU, and TR.The continued-word metric measures how often words are split across at least two sub-tokens.
  • Disentangling the factors: The results suggest that pretraining-data size and tokenizer choice are both major drivers of downstream task performance.The authors pretrain new models on Wikipedia data to disentangle these factors for AR, FI, ID, KO, and TR.
  • New pretrained models: Dedicated monolingual tokenizers outperform multilingual-tokenizer counterparts in most tasks, especially QA, UDP, and SA.For POS tagging, the performance gap is at most 0.4% accuracy; NER is more mixed.
  • Overall comparison: 38 out of 48 task, model, and language combinations favor the monolingual tokenizer, while replacing mBERT’s tokenizer improves 20 out of 24 settings.The largest reported tokenizer-related decrease occurs for SA in TR, at 0.8% accuracy.

5 Further Analysis

The analysis attributes downstream-performance differences to tokenizer capability as well as pretraining data size. Lower fertility and fewer continued words generally align with better performance, especially for UDP and QA, while effects are weaker for NER and POS.

  • Tokenizer and corpus effects: A large portion of performance gaps is attributable to tokenizer capability, not only mBERT’s pretraining-data size.The analysis separates tokenizer effects from corpus-size effects.
  • Tokenizer comparisons: For AR, FI, ID, KO, and TR, lower fertility and continued-word proportions enable consistent gains with monolingual tokenizers in both model variants.This pattern holds for monolingual and multilingual model comparisons.
  • Tokenizer and corpus parity: When tokenizer properties and pretraining corpus size differ little between monolingual models and mBERT, performance gaps are typically negligible for EN, JA, and ZH.In question answering, mBERT is favorable for these languages.
  • Correlation analysis: Lower fertility and fewer continued words correlate with higher downstream performance relative to fully fine-tuned mBERT across tasks.These tokenizer properties show effects similar to pretraining corpus size.
  • Task differences: The correlation is stronger for UDP and QA, where monolingual-tokenizer models consistently outperform counterparts using the mBERT tokenizer.The relationship is weaker for NER and POS tagging because their results are inconsistent.

6 Conclusion

The paper conducts a comprehensive comparison of monolingual and multilingual language models and disentangles pretraining-corpus size from tokenizer effects. It concludes that observed gaps depend on both factors, while monolingual tokenizers improve performance in almost every tested setting.

  • Main findings: The study finds performance gaps in most but not all languages and tasks, often smaller than previously assumed.The gaps are associated with pretraining-data size and tokenizer adaptation to the target language.
  • Controlled comparison: The authors disentangle corpus-size effects by training monolingual models on identical data with dedicated or multilingual tokenizers.The dedicated tokenizer is provided by native speakers, while the alternative is the automatically generated mBERT tokenizer.
  • Tokenizer result: For almost every task and language, monolingual tokenizers outperform the mBERT tokenizer.This comparison isolates tokenizer choice while holding the training data constant.
  • Implications: Improving language representation in multilingual vocabularies and adapting multilingual tokenizers can reduce monolingual–multilingual performance gaps.The paper also identifies tokenizer-free pretraining as a future direction.

A Reproducibility

The reproducibility materials identify the model implementations and document how language-specific mBERT pretraining data were estimated. These estimates are approximate upper limits because the original Wikipedia dumps and article passages differ from the available counts.

  • Model access: The selected pretrained models are available through the HuggingFace model hub and compatible with the HuggingFace Transformers library.Table 5 provides the model hub identifiers.
  • Corpus estimates: Language-specific mBERT corpus shares are estimated from word counts in the corresponding raw Wikipedia dumps.The study uses these counts because mBERT was pretrained on the Wikipedia dumps of its covered languages.
  • Estimation caveat: The Wikipedia-based corpus estimates are upper limits that should nevertheless be reasonably accurate, while monolingual-model data rely on author-provided information.mBERT used older dumps and only article passages, unlike the counted raw dumps.

A.3 Data for Tokenizer Analyses

The tokenizer analyses use Universal Dependencies treebanks and report the dataset examples used for the study. Training and development splits are tokenized from the listed UD v2.6 treebanks, with dataset sizes tabulated separately.

  • Dataset accounting: The datasets used for the analyses, including examples per split, are listed in Table 7.The table records dataset split sizes for reproducibility.

A.5 Training Procedure of New Models

The authors pretrain new monolingual and mBERT-based models under controlled procedures, varying tokenizer choice while largely standardizing optimization settings.

  • Training objective: The pretraining objective is masked language modeling only, omitting next sentence prediction.The procedure otherwise mostly follows the default BERT pretraining procedure.
  • Training setup: New monolingual models are pretrained from scratch for 1M steps with batch size 64.They use sequence length 128 for the first 900,000 steps and 512 for the remaining 100,000.
  • Training setup: mBERT-based models are pretrained with masked language modeling for 250,000 steps using batch size 64 and sequence length 512.Their remaining hyperparameters match those of the monolingual models.
  • Tokenizer conditions: The mBERT-based models resize their embedding layers to match the respective tokenizer vocabularies before retraining.This isolates tokenizer-vocabulary effects while preserving the rest of the model setup as much as described.
  • Tokenizer conditions: For MBERTTOK models, positional, segment, and special-token embeddings are initialized from mBERT, while remaining embeddings are randomly reinitialized.Parameters outside the embedding layer are frozen, and all pretraining runs use random seed 42.

B.1 Tokenization Analysis

The tokenization analysis compares vocabulary coverage, unknown-token rates, and sentence-length distributions to characterize differences between monolingual tokenizers and mBERT.

  • Measures: The analysis focuses on fertility and continued-word proportion because these metrics sufficiently illustrate and quantify tokenizer differences.Additional tokenization statistics are provided for completeness.
  • Vocabulary coverage: Only 5.6% of AraBERT’s vocabulary is covered by mBERT.Tokenizers with lower fertility and better performance than mBERT are often not adequately covered by mBERT’s vocabulary.
  • Unknown tokens: Unknown-token proportions are generally extremely low across the tokenized data.The tokenizers can typically split unknown words into known subwords.
  • Sentence lengths: Monolingual tokenizers with low fertility and continued-word proportions usually produce sentence-length distributions closer to human UD reference tokenizations.This indicates greater similarity to the tokenization behavior of human UD annotators than the mBERT tokenizer.

B.2 Correlation Analysis

The correlation analysis evaluates how pretraining data size and tokenizer statistics relate to downstream performance, both in controlled and noisier settings.

  • Controlled correlations: Decreases in continued-word proportion and fertility correlate with higher downstream performance relative to fully fine-tuned mBERT across all tasks.The correlations are stronger for UDP and QA, but weaker for NER and POS tagging.
  • Factor comparison: Tokenizer metrics appear more indicative of high downstream performance than pretraining corpus size in the initial analysis.The authors attribute part of this pattern to the unusually poor performance of uncased IndoBERT in comparison with a cased Indonesian monolingual model.
  • Factor comparison: Excluding Indonesian leaves tokenizer correlations mostly unchanged but strengthens the correlation between pretraining corpus size and performance.The analysis also indicates that POS tagging may correlate more with data size than tokenizer metrics after this exclusion.
  • Noisier setting: When all languages, models, and adapter-based settings are included, the three factors still correlate with downstream performance similarly.The authors interpret these correlations as indicating that the factors remain highly indicative despite other possible influences.
  • Metric relationship: Continued-word proportion and fertility have nearly identical correlation coefficients.The authors relate this similarity to the visual similarity of their plots.

C Full Results

The appendix provides full experiment results and supporting dataset, tokenizer, and figure documentation across the evaluated tasks and languages.

  • Supporting analyses: Table 5 documents selected pretrained models, their vocabulary sizes, and the proportion of tokens covered by mBERT’s vocabulary.The tokenizer analysis also includes unknown-token proportions, sentence-length distributions, and correlations with downstream performance.
  • Datasets: Table 6 lists the UD treebanks used for tokenizer analyses and their total word counts.Training and development portions are used.
  • Datasets: Table 7 lists NER, SA, QA, and UD datasets with example counts for training, development, and test portions.UD datasets support both universal dependency parsing and POS tagging experiments.
  • Full results: Table 8 reports full performance results for NER, SA, QA, UDP, and POS, using development sets only for QA.Finnish sentiment analysis and Japanese question answering lack respective datasets.
  • Full results: Table 9 reports performance for newly trained monolingual and mBERT-based models alongside prior monolingual models and fully fine-tuned mBERT.Results cover NER, SA, QA, UDP, and POS, with counterparts grouped by tokenizer choice.
  • Adapter results: Table 10 reports performance for mBERT with different adapter components across the evaluated tasks.The table corresponds to adapter-based training described in the paper.
Loading 2012.15613v2…