Source-linked AI summary
Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, Mikel Artetxe
TL;DR
Multilingual pre-trained models lose per-language performance as language coverage expands, creating a need to separate shared and language-specific capacity. The paper pre-trains X-MOD with language-specific modules, then evaluates multilingual transfer and post-hoc language addition. It reports reduced negative interference, positive transfer, and no measurable XNLI performance drop when adding unseen languages.
Problem
The curse of multilinguality causes performance drops as multilingual models support more languages, forcing a trade-off between language coverage and performance.
Method
X-MOD pre-trains language-specific modules alongside shared transformer parameters, maintaining the total number of trainable parameters per language and enabling later language extension.
Results
Across NLI, NER, and QA, X-MOD mitigates negative interference, achieves positive transfer, and improves monolingual and cross-lingual performance.
Takeaways & Limitations
Unseen languages can be added post-hoc with no measurable drop in XNLI performance, while modular pre-training mitigates interference between languages.
Takeaways & Limitations
The 75-language X-MOD model is less stable because 15 extremely low-resource added languages received only 4k additional update steps.
Abstract
from arXiv · showhide
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.
1 Introduction
Multilingual models face performance drops as they cover more languages, motivating a modular architecture that adds language-specific capacity during pre-training. X-MOD aims to reduce interference, support positive transfer, and permit later language extension.
- The curse of multilinguality creates performance drops as models support more languages, forcing a trade-off between language coverage and performance.
- X-MOD introduces language-specific modular components during pre-training while sharing most transformer parameters across languages.Each language receives individual capacity without increasing the total number of trainable parameters per language.
- The architecture is evaluated through average perplexity and transfer performance as the number of pre-trained languages increases.The evaluation compares lower perplexity and higher downstream scores across increasing language counts.
- Experiments on NLI, NER, and QA report reduced interference, positive transfer, and improved monolingual and cross-lingual performance.
- X-MOD can add unseen languages post-hoc by learning corresponding modules while leaving shared parameters frozen, with no measurable performance drop.
2 Background and related work
Multilingual transformers enable cross-lingual transfer but suffer from multilinguality and low-resource limitations. Earlier extension methods add language components after pre-training, whereas this work seeks modular pre-training from the outset.
- Transformer-based multilingual models are pre-trained on multilingual corpora, fine-tuned on a source-language task, and applied directly to target languages.
- Modular and adapter approaches add specialized components to pre-trained models and can improve parameter or training efficiency.
- Multilingual models show reduced cross-lingual and monolingual abilities for low-resource languages with smaller pre-training datasets.
- Methods for extending models to new languages include new embeddings, bilingual training, vocabulary expansion, transliteration, and subword mappings.
- Existing extension approaches can improve unseen-language performance but remain inferior to full fine-tuning on seen languages.
- The paper therefore aims to modularize the model from the start and make it extendable to new languages after pre-training.
3 Proposed approach
X-MOD combines shared transformer components with language-specific bottleneck modules trained jointly from the beginning. Its design supports multilingual pre-training, cross-lingual transfer, and post-hoc language addition.
- Architecture: X-MOD extends transformer architectures with language-specific bottleneck feed-forward modules while retaining shared attention and feed-forward components.
- Architecture: Unlike adapter methods that add components after pre-training, X-MOD trains modular components together with the transformer weights.
- Pre-training procedure: During pre-training, each example uses shared embeddings and transformer components, then routes through its language-specific bottleneck module.
- Extending to new languages: New languages are added by learning new embeddings and adapter modules through masked language modelling while freezing the remaining components.
- Fine-tuning on downstream tasks: For cross-lingual transfer, the shared weights are fine-tuned on source-language data while language modules and embeddings remain frozen.
4 Experimental design
The experiments compare X-MOD with a parameter-matched shared baseline across increasing language sets, held-out languages, and multiple downstream tasks. The design controls tokenization and update conditions while evaluating cross-lingual transfer.
- Model variants: The SHARED baseline uses one bottleneck module for all languages, matching X-MOD’s FLOPs and trainable parameters per language.
- Language sets: Models are pre-trained on 13, 30, 60, or 75 languages, with seven additional languages held out for post-hoc extension.
- Evaluation: The study evaluates XNLI, NER, and XQuAD/MLQA using designated language groups and averages results across five random seeds.
- Controlled comparisons: The experiments compare equal total updates with equal per-language examples to distinguish reduced exposure from negative interference.
- Data and tokenization: All pre-training experiments use the same XLM-R vocabulary, while post-hoc languages receive newly learned SentencePiece tokenizers.
- Evaluation: Fine-tuning uses English source data and measures zero-shot transfer in other languages across NLI, QA, and NER benchmarks.
5 Results and discussion
The results show that shared models suffer negative interference as languages increase, whereas X-MOD maintains or improves performance across pre-trained and added languages. X-MOD also matches pre-training performance when languages are added post-hoc, including languages from unique families.
- The curse of multilinguality: Shared models exhibit negative interference even when trained on the same number of examples per language.
- Lifting the curse: X-MOD maintains or improves downstream performance as the number of pre-training languages increases, unlike shared models.This holds when controlling either update steps or per-language examples.
- Pre-trained languages: The 60-language X-MOD model achieves the best cross-lingual performance across XNLI, NER, MLQA, and XQuAD.Gains are reported across almost all high- and low-resource languages relative to SHARED.
- Limitations: The 75-language X-MOD model is less stable, possibly because 15 extremely low-resource languages receive only 4k additional update steps.The authors suggest longer training could mitigate this variance.
- Added languages: X-MOD consistently outperforms SHARED for languages added post-hoc, with peak performance when pre-training on 60 languages.The results indicate that language-specific capacity benefits post-hoc language extension.
- Pre-training versus adding languages: Per-language XNLI performance is on par whether languages are pre-trained or added post-hoc, regardless of language-family similarity.The authors therefore suggest pre-training on only a subset of languages with sufficient data.
6 Further analysis
Further analysis examines training duration and compares X-MOD with post-hoc adapters. The results indicate that modular capacity must be introduced during pre-training because adapters cannot undo multilingual interference already present in shared weights.
- Update steps: With only 125k update steps on 13 languages, SHARED has a slight edge over X-MOD, motivating analysis of training duration.The authors hypothesize that update steps also matter for modular models.
- X-MOD versus adapters: Post-hoc adapters do not mitigate the curse of multilinguality already affecting shared transformer weights.Adapter performance strongly correlates with the corresponding fully shared model.
7 Conclusions
The evaluation shows that modular multilingual pre-training adds per-language capacity without increasing trainable parameters per language, mitigating negative interference while enabling positive transfer. The approach also supports post-hoc language addition, but evaluation of truly low-resource languages remains future work.
- Providing additional per-language capacity while keeping trainable parameters per language constant mitigates negative interference and achieves positive transfer.
- The optimal number of update steps for pre-training and subsequent adapter training remains insufficiently investigated and was outside the study’s scope.
- X-MOD can add unseen languages post-hoc with no measurable drop in XNLI performance.
- The study simulates language addition with held-out languages rather than evaluating truly low-resource languages.
A Additional results
Additional results report MLQA and XQuAD performance for pretrained languages, MLQA performance for added languages, and NER results across more languages. Figures also track per-language perplexity, XNLI, and NER as language coverage increases.
- MLQA and XQuAD results are reported for pretrained languages, while MLQA results are also reported for added languages.
- NER results are reported for more languages in Table 8.
- Figures 9–11 report per-language results as the number of languages increases for perplexity, XNLI, and NER.
B Intermediate checkpoints
Intermediate-checkpoint evaluation on the 60-language model shows X-MOD continuously outperforming SHARED on XNLI, indicating reduced interference throughout training.
- X-MOD continuously outperforms SHARED on XNLI across evaluated intermediate checkpoints of the 60-language model.
- The results suggest SHARED immediately suffers from negative interference, whereas X-MOD’s language-specific components mitigate the curse of multilinguality.
- X-MOD achieves considerable performance gains at all evaluated checkpoints.
C Language selection
The additional-results materials organize evaluations across intermediate checkpoints, language counts, tasks, and language sets. They include XNLI, NER, perplexity, and question-answering results for pretrained and added languages, with language assignments documented in Table 9.
- B Intermediate checkpoints: Figure 8 evaluates XNLI using intermediate checkpoints from models trained on 60 languages.
- A Additional results: Tables 5–8 report MLQA, XQuAD, and NER results for pretrained or added languages using X-MOD and SHARED variants.
- A Additional results: Figures 9–11 examine perplexity, XNLI, and NER as the number of languages increases, covering pretrained and added languages.
- C Language selection: Table 9 lists languages pretrained on or added across the 13-, 30-, 60-, and 75-language sets.