PaperlayerView exact revision on arXiv

Source-linked AI summary

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei

arXiv:2106.16138v2cs.CL

TL;DR

Cross-lingual pretraining can improve transfer but often demands substantial computation. XLM-E applies ELECTRA-style replaced-token detection to multilingual text and translation pairs; it reports strong cross-lingual results with lower computation and a reasonably low transfer gap.

  • Problem

    Cross-lingual pretraining can require massive computation resources, despite masked language modeling and parallel-corpus tasks supporting cross-lingual representations and transfer.

    Source: §1 · p. 1

  • Method

    XLM-E applies multilingual and translation replaced token detection, using ELECTRA-style discrimination with multilingual corpora and parallel data.

    Source: §1 · pp. 1–2

  • Results

    XLM-E outperforms baseline models on cross-lingual understanding tasks with much less computation and achieves a reasonably low transfer gap.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors report that discriminative pretraining tends to encourage universal representations and better alignment across languages.

    Source: §1 · p. 2

Abstract

from arXiv · show

In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.

1 Introduction

The introduction frames cross-lingual pretraining as effective but computationally expensive, and presents XLM-E’s ELECTRA-style tasks as a more efficient approach. It reports competitive benchmark results with substantially less computation and better cross-lingual transferability.

1 Introduction

  • The introduction notes that masked-language-model cross-lingual pretraining can require massive computation resources.Source: §1 · p. 1
  • XLM-E uses multilingual replaced token detection and translation replaced token detection, training on multilingual corpora and parallel data.The tasks distinguish replaced tokens in multilingual sentences and translation pairs, respectively.Source: §1 · pp. 1–2
  • Across seven XTREME datasets, XLM-E achieves competitive results while using 1% of XLM-R’s computation.Figure 1 reports a 130× speedup over in-house XLM-R + TLM using the same corpora and code base.Source: Figure 1 · p. 1, §1 · p. 2
  • XLM-E shows a reasonably low transfer gap, and the authors report that discriminative pretraining encourages universal representations and better cross-language alignment.Source: §1 · p. 2

2 Background: ELECTRA

ELECTRA pretraining uses a generator to create corrupted inputs and a discriminator to identify replaced tokens. Its objective combines the generator and discriminator losses, while only the discriminator is used for downstream fine-tuning.

2 Background: ELECTRA

  • ELECTRA trains a discriminator to distinguish original tokens from tokens sampled by a generator, rather than predicting masked tokens as in MLM.Source: §2 · p. 2
  • The generator predicts masked tokens from a sentence with selected positions replaced by [MASK], and its loss is the masked-language-modeling loss.Source: §2 · p. 2
  • ELECTRA combines generator and discriminator losses as LE = LG + λLD.Source: §2 · p. 2
  • Although ELECTRA uses more parameters and training FLOPs per step, downstream fine-tuning uses only the discriminator, leaving a checkpoint similar in size to BERT-like models.Source: §2 · p. 2

3 Methods

XLM-E pretrains a generator–discriminator pair using multilingual and translation-pair replaced-token tasks, jointly trained with language-modeling objectives over multilingual and parallel data. The methods also introduce content-gated relative position bias and specify Transformer initialization.

3.1 Pre-training Tasks

  • XLM-E uses multilingual replaced token detection on multilingual sentences and translation replaced token detection on concatenated parallel pairs, with a generator proposing masked tokens and a discriminator identifying replacements.Uniform masking is used; preliminary span masking significantly weakened generator prediction accuracy and harmed pretraining.Source: §3 · p. 2, Figure 2 · p. 3, §3.1 · p. 3

3.2 Pre-training XLM-E

  • Joint pretraining combines masked language modeling, translation language modeling, multilingual replaced token detection, and translation replaced token detection over multilingual and parallel corpora.The generator and discriminator are pretrained jointly from scratch, with a smaller generator used to improve pretraining efficiency.Source: §3.2 · pp. 3–4

3.3 Gated Relative Position Bias

  • Gated relative position bias conditions learnable relative-position bias on input-token content, adaptively adjusting positional bias for different inputs.The authors note that the same distance between tokens can play different roles in different languages.Source: §3.3 · p. 4

3.4 Initialization of Transformer Parameters

  • Initialization uses uniform sampling in a small range, such as [−0.02, 0.02], and rescales each block’s attention and feed-forward output matrices by 1/(2l).The block after the embedding layer is counted as the first Transformer block.Source: §3.4 · p. 4

4 Experiments

XLM-E is evaluated across cross-lingual understanding, efficiency, alignment, and transfer analyses. The experiments report competitive XTREME results at substantially lower pre-training cost, alongside alignment and transferability findings.

4.1 Setup

  • XLM-E is pretrained from scratch with multilingual CC-100 data and parallel corpora, using language-balanced sampling and a Base-size discriminator.Training uses a 12-layer discriminator, a 4-layer generator, and 125K steps; the procedure takes about 1.7 days on 64 Nvidia A100 GPUs.Source: §4.1 · p. 4

4.2 Cross-lingual Understanding

  • On XTREME’s seven-task cross-lingual benchmark, XLM-E scores 67.6 and 69.3 in two settings; XLM-E (w/o TRTD) exceeds XLM-R by 1.2, while XLM-E exceeds XLM-ALIGN by 0.4.Models are fine-tuned on English training data and evaluated across target languages; results are averaged over target languages and five runs.Source: Table 1 · p. 5, §4.2 · p. 5

4.3 Ablation Studies

  • Removing TRTD weakens XLM-E on XNLI and MLQA, while removing gated relative position bias further reduces MLQA performance.With both components removed, XLM-E still outperforms XLM despite using only multilingual data while XLM uses parallel data.Source: Table 2 · p. 5, §4.3 · p. 6

4.4 Scaling-up Results

  • XLM-EXL achieves the best reported performance while using significantly fewer parameters than its counterparts, and scaling XLM-E consistently improves results.The comparison covers the Base, Large, and XL model sizes.Source: §4.4 · p. 6, Table 3 · p. 6

4.5 Training Efficiency

  • XLM-E uses 9.5e19 training FLOPs and outperforms XLM-R and XLM-ALIGN, each using at least 9.6e21 FLOPs, on XTREME.XLM-E without TRTD also outperforms XLM-R using 6.3e19 FLOPs; INFOXLM and XLM-ALIGN costs include their continued training from XLM-R.Source: Table 4 · p. 6, §4.5 · p. 6

4.6 Cross-lingual Alignment

  • XLM-E improves over XLM-R on Tatoeba retrieval and lowers average word-alignment AER from 21.05 to 19.32 across four language pairs.On Tatoeba-14, XLM-E scores 74.4 and 72.3 in the two retrieval directions; it performs slightly worse than INFOXLM, and TRTD improves both alignment tasks over XLM-E without TRTD.Source: Table 5 · p. 6, §4.6 · p. 7, Table 6 · p. 7

4.7 Universal Layer Across Languages

  • XLM-E performs better than XLM-R at top layers on sentence retrieval and word alignment, with best performance shifted to layer 9.For retrieval, XLM-E reaches 63.66 at layer 9 versus XLM-R's 54.42 at layer 7; the authors attribute better top-layer alignment to discriminative pre-training.Source: Figure 3 · p. 7, §4.7 · pp. 7–8, Figure 4 · p. 8

4.8 Cross-lingual Transfer Gap

  • XLM-E has the lowest transfer-gap score only on PAWS-X, while retaining reasonably low gaps on the other evaluated tasks.A lower gap denotes a smaller difference between English test performance and average performance in other languages.Source: Table 7 · p. 8, §4.8 · p. 8

5 Related Work

Related work uses multilingual masked language modeling for cross-lingual encoders and extends pretraining with parallel data. These approaches include cross-lingual understanding and generation methods.

5 Related Work

  • MLM supports cross-lingual encoders such as multilingual BERT and XLM-R, while translation language modeling predicts masked tokens from concatenated translation pairs.The passage also describes MLM as effective for learning cross-lingual representations.Source: §5 · p. 8
  • Parallel-data approaches include codeswitched inputs, contrastive learning over translation pairs, and tasks using token-level alignments.The passage attributes these approaches to ALM, InfoXLM, and several alignment-based methods.Source: §5 · p. 8
  • Cross-lingual sequence-to-sequence pretraining includes masked language modeling and autoencoding, cross-attention MLM, translation span corruption, and encoder alignment.These approaches are described for cross-lingual NLG or both NLU and NLG.Source: §5 · pp. 8–9

6 Conclusion

XLM-E applies two ELECTRA-style tasks to cross-lingual pretraining. It outperforms baseline models on cross-lingual understanding tasks with much less computation and has a reasonably low transfer gap.

6 Conclusion

  • XLM-E uses multilingual replaced token detection and translation replaced token detection for cross-lingual pretraining.The conclusion describes the model as pretrained by ELECTRA-style tasks.Source: §6 · p. 9
  • XLM-E outperforms baseline models on cross-lingual understanding tasks while using much less computation.The conclusion also reports a reasonably low cross-lingual transfer gap.Source: §6 · p. 9

7 Ethical Considerations

The paper states that its ELECTRA-style tasks require less computation than previous models and substantially reduce energy cost.

7 Ethical Considerations

  • ELECTRA-style tasks for cross-lingual pretraining require less computation than previous models and substantially reduce energy cost.Source: §7 · p. 9

A Model Hyperparameters

XLM-E uses vocabulary sizes that vary by model scale. Tables 8 and 9 present generator and discriminator hyperparameters for Base, Large, and XL models.

A Model Hyperparameters

B Hyperparameters for Pre-Training

Pre-training uses a 1M-token batch for each task, with input construction differing between multilingual and translation replaced token detection.

B Hyperparameters for Pre-Training

C Hyperparameters for Fine-Tuning

Table 11 reports XLM-E fine-tuning hyperparameters for XTREME end tasks.

C Hyperparameters for Fine-Tuning

Loading 2106.16138v2…