PaperlayerView exact revision on arXiv

Source-linked AI summary

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Benjamin Minixhofer, Fabian Paischer, Navid Rekabsaz

arXiv:2112.06598v2cs.CL

TL;DR

Training capable monolingual language models beyond English is computationally expensive, and massively multilingual models face performance trade-offs as language coverage grows. WECHSEL transfers English-model parameters and initializes target-language subword embeddings using multilingual static word embeddings. The paper reports improved performance over transfer baselines and prior monolingual models trained for substantially longer.

  • Problem

    Training comparable monolingual language models for non-English languages is costly, while massively multilingual models can lose performance as the number of languages grows.

    Source: §1 · p. 1

  • Method

    WECHSEL copies non-embedding parameters, replaces the tokenizer, and initializes target-language token embeddings near semantically similar English tokens using multilingual static word embeddings.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    Transferred RoBERTa and GPT-2 models outperform strong baselines and prior monolingual models trained for significantly longer.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    WECHSEL facilitates effective monolingual LMs for new medium- and low-resource languages, particularly in computationally limited settings.

    Source: §7 · p. 9

    Extrinsic evaluation covers only NLI and NER, so it may not provide a comprehensive view of language understanding tasks.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method -- called WECHSEL -- to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.

1 Introduction

Training powerful monolingual LMs for languages beyond English is costly, while massively multilingual models can lose performance as language coverage grows. WECHSEL transfers English-model parameters and initializes target-language subword embeddings semantically; the paper reports better results than transfer baselines and strong performance with fewer training steps.

1 Introduction

  • Training large LMs is costly, and training comparable non-English monolingual models from scratch would presumably require similar compute to the English models.Multilingual LMs can also lose performance beyond a certain language count, while monolingual LMs often outperform them.Source: §1 · p. 1
  • WECHSEL replaces the source tokenizer and initializes target-language token embeddings near semantically similar English tokens using multilingual static word embeddings.It copies the English model’s non-embedding parameters; token embeddings account for roughly 31% of RoBERTa and 33% of GPT-2 parameters.Source: §1 · pp. 1–2
  • Experiments transfer RoBERTa to French, German, Chinese, and Swahili, and GPT-2 to those languages plus four very low-resource languages.Evaluations cover RoBERTa NER and NLI and GPT-2 language-modeling perplexity, with models trained under the same conditions.Source: §1 · p. 2
  • WECHSEL-initialized RoBERTa and GPT-2 outperform randomly initialized and TransInner models across languages and tasks, while reaching strong performance with fewer training steps.The reported pattern covers both model types and all evaluated languages and tasks.Source: §1 · p. 2
  • The paper reports releasing more effective GPT-2 and RoBERTa models than previously published non-English models under a more efficient training setting.The code and models are publicly available.Source: §1 · p. 2

2 Related Work

Related work spans multilingual representations, bilingualizing existing models, and transferring models to create new monolingual LMs. WECHSEL follows the latter path, extending semantic subword initialization while avoiding the need to preserve source-language capabilities.

Multilingual representations

  • Multilingual static embeddings are learned jointly across languages or aligned from monolingual embeddings, while multilingual contextual models can support cross-lingual transfer.Prior work also reports limits: low-resource languages can perform worse than using no pretrained LM, and performance can decline as language coverage grows.Source: §2 · pp. 2–3

Cross-lingual transfer of monolingual LMs

  • Bilingualization replaces the tokenizer and relearns target subword embeddings while freezing other parameters; Tran (2020) instead initializes target embeddings with static word embeddings.Bilingualized models retain source-language capabilities, which can support zero-shot transfer from source-language task annotations.Source: §2 · p. 3
  • For new target-language monolingual LMs, prior transfer methods have used random-token or vocabulary-overlap initialization, while TransInner transfers non-embedding parameters and further trains the model.This setting does not require preserving source-language capabilities.Source: §2 · p. 3
  • WECHSEL extends Tran (2020) toward new monolingual LMs, generalizing semantic subword mapping to arbitrary numbers of similar subwords and temperatures.The authors report outperforming extensively scratch-trained monolingual models with substantially less compute.Source: §2 · p. 3

3 Methodology

WECHSEL transfers a source model’s non-embedding parameters and initializes target-language subword embeddings using aligned multilingual static embeddings. It represents subwords through n-grams, then transfers embeddings using semantic similarity to source subwords.

3 Methodology

  • WECHSEL copies the source model’s non-embedding parameters and uses aligned bilingual word and subword embeddings to initialize target-language embeddings.It computes source and target subword embeddings, then uses their semantic similarities to transfer embeddings into the target vocabulary.Source: §3 · p. 4

3.1 Subword Embedding Computation

  • Each tokenizer subword is represented by summing the embeddings of its occurring n-grams; subwords with no known n-grams receive zero vectors.This computation is performed separately for the source and target languages.Source: §3.1 · p. 4

3.2 Subword similarity-based Transfer

  • WECHSEL compares target and source subword vectors, then initializes each target embedding as a softmax-weighted mean of its k nearest source embeddings.Target subwords with zero vectors instead receive random-normal initialization.Source: §3.2 · p. 4

4 Experiment Design

The experiments transfer English RoBERTa and GPT-2 to medium-resource languages, and GPT-2 additionally to four low-resource languages. They evaluate language understanding and language modeling under controlled training settings and compare against transfer and scratch-trained baselines.

4 Experiment Design

  • English RoBERTa and GPT-2 are transferred to French, German, Chinese, and Swahili; GPT-2 is also transferred to Sundanese, Scottish Gaelic, Uyghur, and Malagasy.RoBERTa is evaluated on XNLI and WikiANN, while GPT-2 is evaluated by perplexity on held-out text; low-resource GPT-2 is assessed intrinsically.Source: §4 · p. 5
  • TransInner random-initializes target embeddings before training, while FullRand trains the target-language model from scratch.TransInner initially trains embeddings with other parameters frozen, then trains the full model.Source: §4 · p. 5
  • All models train for 250k steps; WECHSEL and FullRand use linear warmup then decay, while TransInner uses separate warmup phases for embeddings and the full model.Training one model takes around four days on a TPUv3-8.Source: §4 · p. 5
  • RoBERTa evaluation reports XNLI accuracy and WikiANN micro F1 before training, after 25k steps, after 250k steps, and for prior models trained longer.Reported results average three runs.Source: Table 2 · p. 6
  • The comparison includes prior monolingual models and XLM-R, which used more target-language tokens; Table 1 compares target-language token counts across models.Total tokens encountered in the target language serve as a proxy for training effort.Source: Table 1 · p. 5, §4 · p. 6

5 Results

Results cover RoBERTa downstream tasks and GPT-2 language modeling in medium- and low-resource languages, followed by an experiment on whether parameter freezing is needed.

5 Results

5.1 Transferring RoBERTa

  • On NLI, WECHSEL improves absolute accuracy over scratch by 7.15%, 6.31%, 6.94%, and 4.71% for French, German, Chinese, and Swahili, respectively.Source: §5.1 · p. 6
  • French and German models surpass CamemBERT and GBERTBase after 10% of steps, reducing effort by 64x and 39x; Chinese still trails BERTBase-Chinese on NER by 1.5% absolute micro F1.Source: §5.1 · pp. 6, 8

5.2.1 To Medium-Resource Languages

  • For medium-resource GPT-2, WECHSEL beats TransInner throughout training in French and German; in Chinese, it surpasses scratch only after a late perplexity decrease.Source: Figure 2 · p. 7, §5.2.1 · p. 8

5.2.2 To Low-Resource Languages

  • WECHSEL improves low-resource perplexity across languages, with gains tending to grow as data shrinks; these models can overfit early and need model selection.Source: §5.2.2 · p. 8

5.3 Is freezing necessary?

  • Freezing non-embedding parameters is necessary for TransInner but not for WECHSEL in the German GPT-2 experiment.Source: §5.3 · p. 8, Figure 4 · p. 9

6 Limitations and Potential Risks

WECHSEL is evaluated on up to eight languages, but many languages with diverse linguistic characteristics remain untested. Its extrinsic evaluation also covers only NLI and NER, while transferred English LMs are likely to carry societal biases and stereotypes into target-language models.

6.1 Limitations

  • WECHSEL is untested on many languages with diverse linguistic characteristics, so similar transfer improvements across all other languages cannot be ascertained.The experiments cover up to eight languages; computational constraints limited broader testing.Source: §6.1 · p. 9
  • The evaluation covers only NLI and NER, so it does not necessarily provide a comprehensive view of language understanding tasks.The authors attribute this scope to limits on available collections in various languages.Source: §6.1 · p. 9

6.2 Risks

  • Because WECHSEL transfers English LMs, societal biases and stereotypes are likely to transfer to target LMs and could contribute to unfair treatment in downstream tasks.The authors advocate conscious and responsible use of the transferred models.Source: §6.2 · p. 9

7 Conclusion

WECHSEL initializes target-language subword embeddings using multilingual static word embeddings to transfer monolingual language models. Transferred RoBERTa and GPT-2 models outperform strong baselines and prior monolingual models trained for significantly longer, supporting use in medium- to low-resource settings.

7 Conclusion

  • WECHSEL computes target-language subword embeddings from multilingual static word embeddings to transfer monolingual language models.The experiments transfer RoBERTa and GPT-2 from English to French, German, Chinese, Swahili, and four low-resource languages for GPT-2.Source: §7 · p. 9
  • Transferred RoBERTa and GPT-2 models outperform strong baselines and consistently outperform prior monolingual models trained for significantly longer.The authors also report that the results provide strong evidence toward the hypothesis that deep monolingual language models learn abstractions that generalize across languages.Source: §7 · p. 9
  • WECHSEL facilitates effective monolingual language models for new languages with medium to low resources, particularly in computationally limited settings.Source: §7 · p. 9

B Hyperparameters

The appendix points readers to tables of RoBERTa fine-tuning and transferred-model training hyperparameters, and to a qualitative sample of target-language subwords paired with their closest English tokens.

B Hyperparameters

C Qualitative subword correspondence

D Using Word Embeddings without subword information

The appendix describes a method for mapping word embeddings to subword embeddings without using subword information: it averages embeddings of words containing each subword, weighted by word frequency. The resulting WECHSELTFR variant performs on average on par with WECHSEL.

D Using Word Embeddings without subword information

  • For each subword, the method averages embeddings of words containing it after tokenization, weighting each word by its frequency.It requires word-frequency information in addition to word embeddings.Source: D Using Word Embeddings without… · p. 13
  • The appendix identifies a grid search over the temperature τ and the number of most similar tokens k parameters of WECHSEL.Source: Table 7 · p. 13
  • On average, WECHSELTFR performs on par with WECHSEL across the evaluated GPT-2 and RoBERTa models.Results are reported in Tables 9 and 10, respectively.Source: D Using Word Embeddings without… · p. 13

E Choosing a transfer baseline

The paper compares two language-agnostic transfer baselines that copy non-embedding parameters but initialize target embeddings differently. TransInner performs slightly better than TransInnerShuffleEmb and is used in subsequent experiments.

E Choosing a transfer baseline

  • TransInner copies non-embedding parameters and initializes target embeddings from a random normal distribution, while TransInnerShuffleEmb assigns each target embedding a random source-language token embedding.Both methods transfer models without language-specific information.Source: E Choosing a transfer baseline · p. 13
  • TransInner performs slightly better than TransInnerShuffleEmb and is selected for subsequent experiments.Figure 6 compares the German GPT-2 models with and without freezing non-embedding parameters at the start.Source: E Choosing a transfer baseline · p. 13, Figure 6 · p. 14

F Sensitivity Analysis w. r. t. training data size

The analysis finds that WECHSEL’s improvement increases as training data decreases, while using fastText embeddings trained on less data reduces performance. This relation is tested only for French because of computational constraints.

F Sensitivity Analysis w. r. t. training data size

  • WECHSEL’s improvement increases as the amount of training data decreases, and reduced-data fastText embeddings lower performance but retain a clear margin over TransInner and FullRand.The sensitivity analysis varies training data while evaluating on the same held-out set.Source: F Sensitivity Analysis w. r. t.… · p. 14, Table 11 · p. 15
  • The sensitivity analysis is limited to French because of computational constraints.The analysis uses random subsamples of 16MiB, 64MiB, 256MiB, and 1024MiB of the original training data.Source: F Sensitivity Analysis w. r. t.… · p. 14, Table 11 · p. 15
Loading 2112.06598v2…