PaperlayerView exact revision on arXiv

Source-linked AI summary

MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder

arXiv:2005.00052v3cs.CL

TL;DR

Pretrained multilingual models have limited capacity, weakening transfer to low-resource and unseen languages. MAD-X uses modular language and task adapters, including invertible adapters, to enable parameter-efficient adaptation across tasks and languages, and it outperforms baselines on NER and causal commonsense reasoning while remaining competitive on QA.

  • Problem

    Limited capacity in pretrained multilingual models weakens cross-lingual transfer, especially for low-resource and unseen languages.

  • Method

    MAD-X learns modular language and task representations with adapters, including invertible adapters, and trains target-language adapters through MLM.

  • Results

    MAD-X outperforms baselines across seen and unseen languages on NER and causal commonsense reasoning, while achieving competitive QA performance with greater parameter efficiency.

  • Takeaways & Limitations

    MAD-X is particularly useful for adapting multilingual models to languages outside their training data while retaining competitive performance on high-resource languages.

  • Takeaways & Limitations

    Future work is needed for additional models, tasks, language properties, and truly low-resource languages with limited monolingual data.

Abstract

from arXiv · show

The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pre-training. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pre-trained multilingual model to a new language. MAD-X outperforms the state of the art in cross-lingual transfer across a representative set of typologically diverse languages on named entity recognition and causal commonsense reasoning, and achieves competitive results on question answering. Our code and adapters are available at AdapterHub.ml

1 Introduction

Pretrained multilingual models struggle with limited capacity, especially for low-resource and unseen languages. MAD-X addresses this through modular language and task adapters, invertible adapters, and target-language adaptation, achieving strong cross-lingual transfer results.

  • Motivation: Limited multilingual-model capacity constrains representation across languages, with performance declining further for low-resource and unseen languages.Scaling models to cover more than 7,000 languages is described as prohibitive.
  • MAD-X: MAD-X uses modular language- and task-specific adapters to adapt a pretrained multilingual model to arbitrary tasks and languages with few additional parameters.Adapters are small bottleneck layers inserted between the model’s weights.
  • MAD-X: Language adapters are trained with masked language modelling on unlabelled target-language data, while task adapters are trained on labelled source-language task data.The adapters are stacked and the target language adapter can be substituted at inference.
  • MAD-X: Invertible adapters address the mismatch between shared multilingual and target-language vocabularies during multilingual adaptation.They enable adaptation to languages unseen during pretraining without expensive language-specific token-level embeddings.
  • Evaluation: MAD-X outperforms baselines on WikiANN named entity recognition and XCOPA causal commonsense reasoning across typologically diverse seen and unseen languages.The evaluation includes high-resource and low-resource languages.
  • Evaluation: MAD-X provides competitive, more parameter-efficient performance on XQuAD question answering and includes a stronger method for adapting pretrained multilingual models to new languages.The target-language adaptation method outperforms transferring only from labelled source-language data.

2 Related Work

Cross-lingual NLP relies on multilingual representations that transfer across tasks and languages, but current models remain limited by insufficient capacity to represent all languages equally. Adapters provide a parameter-efficient route for task and domain adaptation.

  • Cross-lingual representations: Modern cross-lingual research increasingly learns general-purpose representations intended for application across multiple tasks and languages.This progression includes word-level and multilingual pretrained representations.
  • Cross-lingual representations: XLM-R is competitive with monolingual models on GLUE, yet studies report poor cross-lingual transfer across many language pairs.The cited explanation is insufficient capacity in vocabulary and representation space.
  • Adapters: Adapters were developed for multi-domain model adaptation and are used in NLP for parameter-efficient fine-tuning to new tasks and domains.Their use also avoids catastrophic forgetting.

3 Multilingual Model Adaptation for Cross-lingual Transfer

Standard cross-lingual transfer fine-tunes a multilingual model on labelled source-language data and applies it directly to a target language. Target-language MLM adaptation improves transfer but remains limited by model capacity and full-model scaling costs.

  • Standard Transfer Setup: The standard setup fine-tunes a multilingual model on labelled source-language task data before applying it directly to the target language.This balances multilingual knowledge against downstream task adaptation.
  • Target Language Adaptation: Target-language adaptation adds MLM fine-tuning on unlabelled target-language data before source-language task fine-tuning.It improves cross-lingual transfer over the standard setup and does not cause catastrophic forgetting of multilingual knowledge.
  • Target Language Adaptation: The adapted model becomes biased toward a specific target language, preventing evaluation of the same model across multiple target languages.This approach may be preferable when performance matters for one fixed target language.
  • Limitations: Target-language adaptation inherits pretrained models’ limited capacity, hindering effective adaptation to low-resource and unseen languages.Fine-tuning the full model also scales poorly across many tasks or languages.

4 Adapters for Cross-lingual Transfer

MAD-X uses modular language, task, and invertible adapters to adapt a frozen multilingual Transformer across languages and tasks. Language and invertible adapters support language adaptation, while task adapters preserve task-specific knowledge for zero-shot transfer.

  • Framework: MAD-X keeps the pretrained multilingual model fixed while learning language- and task-specific transformations through modular adapters.The framework includes language, task, and invertible adapters inserted into the Transformer.
  • Language Adapters: Language adapters are bottleneck layers with down-projection, ReLU activation, up-projection, and a residual connection at each Transformer layer.The hidden state and feed-forward residual are combined after the adapter transformation.
  • Language Adapters: Language adapters are trained with masked language modelling on unlabelled language data, then swapped from source to target language for zero-shot transfer.The source language adapter remains fixed during task training, while the target adapter replaces it at inference.
  • Task Adapters: Task adapters stack above language adapters and are the only parameters updated for downstream training, capturing task-specific knowledge intended to generalise across languages.They use the language-adapter output together with the Transformer feed-forward residual.
  • Invertible Adapters: Invertible adapters address vocabulary mismatch by transforming input embeddings and applying the inverse before tied output embeddings.They are trained with language adapters using MLM and reused through language-specific replacement during zero-shot transfer.
  • Invertible Adapters: The invertible adapter splits embeddings and applies coupled transformations F and G, enabling an explicitly defined inverse through alternating subtraction operations.The forward transformation uses o1 = F(e2) + e1 and o2 = G(o1) + e2, followed by concatenation.

5 Experiments

The experiments evaluate MAD-X and multilingual adaptation baselines across NER, QA, and CCR, using language sets that vary in resource level, pretraining coverage, and typological diversity. NER additionally measures transfer across all source–target language pairs and reports averages over 16 target languages.

  • Tasks and datasets: Experiments cover named entity recognition, question answering, and causal commonsense reasoning using WikiANN, XQuAD, and XCOPA.
  • Language selection: The NER language selection varies Wikipedia data availability, multilingual-pretraining coverage, and typological diversity.The resulting categories distinguish high- and low-resource languages covered by pretrained models from low-resource and truly low-resource languages not covered.
  • Evaluation design: NER evaluates every possible source–target language pair, including standard zero-shot transfer and monolingual in-language settings.
  • Evaluation design: QA and CCR use English as the source language across 11 XQuAD and 12 XCOPA languages, including two XCOPA languages unseen by the main model.XQuAD languages are mainly high-resource, whereas XCOPA provides a more typologically diverse set.
  • Baselines and variants: The main baseline is XLM-R, while additional comparisons adapt models with source- or target-language masked language modelling before task-specific fine-tuning.The experiments also include MAD-X variants without invertible adapters or without language and invertible adapters.
  • Training setup: NER training uses WikiAnn source-language data, with separate training schedules for high- and low-resource languages and adapter dimensionalities smaller than XLM-R Base’s 768-wide hidden layers.Models are selected using validation performance, and NER generally uses five fine-tuning runs.

6 Results and Discussion

MAD-X consistently improves cross-lingual transfer, with especially strong benefits for low-resource and unseen languages, while remaining competitive on high-resource languages and question answering.

  • Named Entity Recognition: XLM-R performance is lowest for unseen languages, while XLM-RBase MLM-TRG improves over XLM-R in 9/16 target languages but loses gains for low-resource languages.MLM-TRG also requires separately fine-tuning the full pretrained model for each target language.
  • Named Entity Recognition: Task adapters alone remain competitive on languages seen during pretraining but deteriorate significantly on unseen languages, showing that they are insufficient by themselves there.This comparison isolates the contribution of language-specific adaptation for unseen-language transfer.
  • Named Entity Recognition: More than 5 F1 points: the full MAD-X framework improves upon XLM-R on average in cross-lingual NER.Language adapters help across languages, especially low-resource ones, while invertible adapters provide further gains on 13/16 target languages.
  • Named Entity Recognition: MAD-X shows the largest relative gains when transferring from high-resource to low-resource and unseen languages, with strong gains from Arabic and competitive high-resource performance.It also performs strongly in the in-language setting for the evaluated low-resource languages.
  • Causal Commonsense Reasoning: Target language adaptation outperforms XLM-RBase, while MAD-XBase achieves the best XCOPA scores, with particularly strong gains for Haitian Creole and Quechua.Performance on the other evaluated languages is generally competitive or better.
  • Question Answering: On XQuAD, MAD-X achieves similar performance to the XLM-R baseline, while invertible adapters and target language adaptation generally improve performance.All XQuAD target languages are high-resource, with more than 100k Wikipedia articles each.

7 Further Analysis

MAD-X’s modular adapters improve transfer, especially for low-resource languages, while remaining reusable and parameter-efficient. Strong performance is reached with relatively few training iterations.

  • Impact of Invertible Adapters: Invertible adapters improve performance for many source–target NER transfer pairs, particularly when transferring to low-resource languages.Performance is consistently lower only when Maori is the source, likely due to data variation.
  • Sample Efficiency: Once trained, MAD-X language and invertible adapters are directly reusable across different tasks.This plug-and-play property reduces the repeated adaptation burden across tasks.
  • Sample Efficiency: Strong low-resource NER performance is achieved after 20k training iterations, while longer training provides only modest gains.The analysis varies the number of training iterations for language and invertible adapters.
  • Parameter Efficiency: 8.25M adapter parameters are added per language for the full MAD-X NER model, representing 3.05% of the original XLM-R Base model.The comparison uses the original model’s 270M-parameter budget.

8 Conclusion

MAD-X is a modular, model-agnostic framework for transfer across tasks and languages that uses few additional parameters. It is particularly useful for languages absent from multilingual-model training while remaining competitive on high-resource languages.

  • Conclusion: MAD-X provides a general modular framework for transfer across tasks and languages using a small number of additional parameters.The framework is designed to mitigate the capacity limitations of current multilingual models.
  • Conclusion: MAD-X is model-agnostic and can use any current pretrained multilingual model as its foundation.The conclusion presents model agnosticism as part of the framework’s generality.
  • Conclusion: MAD-X is particularly useful for adapting to languages not covered by a multilingual model’s training data while remaining competitive on high-resource languages.Future work includes testing additional models, language-specific adapters, tasks, and related-language adapters for truly low-resource settings.

A Evaluation data

The evaluation covers named entity recognition, causal commonsense reasoning, and question answering using established multilingual datasets. Several analyses report English-to-other-language transfer averaged over five runs.

  • Evaluation Tasks: Named Entity Recognition uses the WikiANN dataset.WikiANN is listed as the data source for the NER evaluation.
  • Evaluation Tasks: Causal Commonsense Reasoning uses the XCOPA dataset.XCOPA is listed as the data source for the CCR evaluation.
  • Evaluation Tasks: Question Answering uses the XQuAD dataset.XQuAD is listed as the data source for the QA evaluation.
  • Evaluation Protocol: English-to-other-language F1 scores are averaged over five runs in the reported evaluation.The passage specifies this averaging for the table of transfer results.

C NER results per language pair

The NER appendix reports F1 scores across source–target language pairs for multiple baselines and MAD-X variants. Heatmaps compare relative improvements against key baselines and ablations.

  • NER Results: NER F1 scores are reported for all combinations of source and target languages across the comparison methods.Each score is averaged over five runs.
  • NER Results: The comparisons include XLM-RBase, MLM source and target variants, MAD-XBase variants, mBERT, MAD-XmBERT, and XLM-RLarge variants.The listed figures cover both baseline models and MAD-X configurations.
  • NER Results: Table 6 reports NER F1 scores for zero-shot transfer from English.The table is identified as the English zero-shot transfer result summary.
  • Relative Improvements: MAD-XBase’s relative F1 improvements are visualized against XLM-RBase MLM-TRG and against the variant without invertible adapters.These heatmaps isolate comparisons with a baseline and an ablation.
  • Relative Improvements: Additional heatmaps compare MAD-XmBERT with mBERT and MAD-XLarge with XLM-RLarge.These comparisons extend the relative-improvement analysis to other foundations.

E XCOPA results for all settings

XCOPA results are presented in Table 7 for transfer from English. The table reports accuracy across three fine-tuning settings involving COPA and SIQA.

  • Table 7 reports accuracy scores on XCOPA test sets when transferring from English.
  • Models are evaluated after fine-tuning only on COPA training data.
  • Models are also evaluated after fine-tuning only on SIQA or sequentially on SIQA followed by COPA.
Loading 2005.00052v3…