PaperlayerView exact revision on arXiv

Source-linked AI summary

From English To Foreign Languages: Transferring Pre-trained Language Models

Ke Tran

arXiv:2002.07306v2cs.CL

TL;DR

The paper asks how English pre-trained models can be transferred to other languages without the cost of training new multilingual models from scratch. It rapidly adapts English models into bilingual LMs by initializing foreign embeddings in English space and fine-tuning the bilingual model. Across six languages, the resulting models outperform mBERT on two zero-shot tasks and also work as feature extractors for supervised dependency parsing.

  • Problem

    Recent pre-trained-model research focuses heavily on English, while training comparable models for other languages from scratch requires substantial computation.

  • Method

    The approach initializes target-language embeddings in English space and then jointly fine-tunes English and target-language components to create bilingual LMs.

  • Results

    Across six languages, the models perform better than mBERT on zero-shot natural language inference and dependency parsing, while also serving as powerful supervised parsing feature extractors.

  • Takeaways & Limitations

    English pretrained models can be rapidly transferred into effective bilingual LMs under a limited computational budget.

  • Takeaways & Limitations

    The approach currently uses parallel data only to initialize foreign embeddings, leaving its use during fine-tuning for future work.

Abstract

from arXiv · show

Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT base model within a day and a foreign BERT large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.

1 Introduction

The paper addresses the difficulty of transferring English pre-training advances to non-English languages without training multilingual models from scratch. It proposes rapidly adapting English models into bilingual LMs and evaluates them on zero-shot and supervised transfer tasks.

  • 1 Introduction: Existing multilingual options include mBERT, XLM-R, and LASER, but LASER requires parallel data and recent pretraining research remains heavily English-focused.
  • 1 Introduction: The method transfers an existing English pretrained LM to a target language under limited computation, rather than training a multilingual model from scratch.It first learns target-language parameters while keeping the English LM fixed, then fine-tunes both models.
  • 1 Introduction: A bilingual BERTBASE can be obtained within a day using one Tesla V100 16GB GPU.
  • 1 Introduction: Across six languages, the bilingual models achieve competitive or better performance than mBERT on zero-shot natural language inference and dependency parsing.
  • 1 Introduction: The bilingual models also serve as effective feature extractors for supervised dependency parsing.
  • 1 Introduction: A concurrent approach fine-tunes only target embeddings with a fixed Transformer encoder but requires 64 TPUv3 chips for good results.

2 Bilingual Pretrained LMs

The bilingual LM is built by initializing target-language embeddings in English space and then jointly fine-tuning English and target-language components. The design uses a shared encoder, separate language-specific embedding layers, and monolingual or parallel data for initialization.

  • 2 Bilingual Pretrained LMs: A pretrained LM encodes input embeddings into contextualized representations and predicts an output token at each position.For masked language modeling, corrupted tokens are replaced with [MASK] and the original tokens are predicted.
  • 2 Bilingual Pretrained LMs: The approach first initializes target-language embeddings in English space, then jointly fine-tunes the English and target-language models.The shared bilingual model is formed from English embeddings, target embeddings, and a Transformer encoder.
  • 2.1 Initializing Target Embeddings: Each target-language embedding is represented as a linear combination of English word embeddings, with translation probabilities supplying the combination weights.Weights can be estimated from parallel word alignments or aligned monolingual word vectors.
  • 2.1 Initializing Target Embeddings: sparsemax assigns zero probability to most English vocabulary items, producing sparse target-embedding initializations from a few semantically similar English words.
  • 2.2 Fine-tuning Bilingual LMs: The bilingual model uses separate English and foreign embedding and output layers with a shared encoder, which is fine-tuned on English and foreign monolingual data.Continued English tuning is intended to preserve English knowledge for zero-shot transfer while the encoder learns target-language syntax.

3 Zero-shot Experiments

The experiments evaluate transferred bilingual language models on XNLI and dependency parsing across six languages, using both parallel and monolingual-data settings.

  • 3 Zero-shot Experiments: RAMEN transfers BERT and RoBERTa autoencoding models with masked-language-model objectives, evaluating base and large variants against mBERT.BERTBASE provides the mBERT comparison, while larger BERT and RoBERTa variants test whether stronger English source models yield stronger bilingual models.
  • 3 Zero-shot Experiments: The evaluation covers XNLI and Universal Dependencies parsing across French, Russian, Arabic, Chinese, Hindi, and Vietnamese.The six languages span four language families and are evaluated using XNLI and specified Universal Dependencies treebanks.
  • 3.2 Estimating translation probabilities: Subword translation probabilities are estimated either from parallel sentence pairs or from aligned monolingual word vectors, with word vectors converted into subword vectors by weighted averaging.Parallel-data experiments tokenize 2M sentence pairs before fast-align; monolingual experiments use fastText vectors aligned into a shared English space.
  • 3 Zero-shot Experiments: Table 1 reports zero-shot XNLI classification results, distinguishing parallel-data initialization with '+' while RAMEN uses parallel data only for initialization.The table marks the best results in bold.
  • 3 Zero-shot Experiments: RAMENBASE models train within a day and RAMENLARGE models within two days on one Tesla V100 16GB GPU.The reported setup uses 120,000 updates for RAMENBASE and 300,000 for RAMENLARGE.

4 Results

RAMEN transfers pretrained English models to bilingual models that perform competitively or better than mBERT on zero-shot XNLI and dependency parsing. Larger and stronger source models improve transfer, while the approach remains computationally efficient but currently uses parallel data only for embedding initialization.

  • Experimental Setup: RAMEN evaluates zero-shot XNLI and universal dependency parsing, with dependency parsing using a graph-based parser over contextual representations without part-of-speech tags.Table 2 reports labeled attachment scores, and its caption identifies parallel-data initialization with a plus sign.
  • Cross-lingual Natural Language Inference: RAMENBASE slightly outperforms mBERT on XNLI by 2.6 points on average and by 4.4 points on Arabic.The comparison is fairest when RAMENBASE+BERT uses monolingual initialization.
  • Cross-lingual Natural Language Inference: RAMEN+RoBERTa outperforms RAMEN+BERT for most languages across comparable architectures and initialization conditions.RoBERTa-based transfer improves XNLI by 3.1 points with monolingual initialization and 4.3 points with parallel initialization.
  • Cross-lingual Natural Language Inference: RoBERTaLARGE achieves state-of-the-art XNLI results for five languages, although the models remain underfit.The authors report further potential for improving XNLI performance.
  • Universal Dependency Parsing: RAMENBASE+BERT outperforms mBERT on all dependency-parsing languages, averaging 4.5 LAS points higher and gaining 8.2 on Hindi and 6.2 on French.Arabic gains 4.2 LAS from the approach.

5 Analysis

Further RAMEN fine-tuning improves both zero-shot tasks, surpassing mBERT after 40,000 updates while adapting to foreign syntax. Initialization strongly affects transfer, and RAMEN contextual representations also support competitive supervised parsing.

  • 5.1 How does linguistic knowledge transfer happen through each training stages?: RAMEN performance generally improves on both zero-shot tasks as fine-tuning proceeds.
  • 5.1 How does linguistic knowledge transfer happen through each training stages?: After 40,000 updates, RAMENBASE+BERT surpasses mBERT on both XNLI and universal dependency parsing.This corresponds to seven training hours.
  • 5.1 How does linguistic knowledge transfer happen through each training stages?: Language similarity affects syntactic transfer: French reaches 78.0 LAS, while Arabic and Hindi reach 38.2 and 36.2 with the SVO encoder.Chinese performs poorly, possibly because it is often treated as head-final despite SVO order.
  • 5.2 Impact of initialization: Random foreign-embedding initialization reduces average zero-shot performance by 10.3% on XNLI and 11.6 LAS points on UD parsing.Arabic and Hindi parsing particularly suffer under random initialization.
  • 5.3 Are contextual representations from RAMEN also good for supervised parsing?: In supervised UD parsing, RAMENBASE+BERT is competitive with mBERT and exceeds it by 1.1 LAS points for Vietnamese.RAMENLARGE+RoBERTa obtains the best LAS results, and the findings support RAMEN contextual representations as feature extractors.

6 Conclusions

The paper presents a simple, effective method for rapidly building bilingual language models under limited computational budgets. RAMEN outperforms mBERT on two zero-shot tasks and also serves as a powerful feature extractor for supervised dependency parsing.

  • RAMEN rapidly builds bilingual language models under a limited computational budget.
  • RAMEN performs better than mBERT on zero-shot sentence classification and dependency parsing.
  • RAMEN is a powerful feature extractor for supervised dependency parsing.
Loading 2002.07306v2…