PaperlayerView exact revision on arXiv

Source-linked AI summary

ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

arXiv:2012.15674v4cs.CL

TL;DR

Parallel-corpus size limits cross-lingual pretraining, despite parallel data’s value for performance. ERNIE-M adds back-translation-based pretraining on monolingual corpora to learn cross-lingual semantic alignment, and reports state-of-the-art results across several downstream tasks.

  • Problem

    Parallel corpora are valuable for cross-lingual model performance, but their limited size constrains existing methods, especially for low-resource languages.

    Source: §1 · p. 1

  • Method

    ERNIE-M combines cross-attention masked language modeling on parallel data with back-translation masked language modeling that uses predicted other-language tokens and monolingual sentences as pseudo-parallel data.

    Source: §1 · p. 1, §3 · p. 5

  • Results

    ERNIE-M achieves state-of-the-art results across XNLI, MLQA, CoNLL, PAWS-X, and Tatoeba.

    Source: §5 · p. 9

  • Takeaways & Limitations

    ERNIE-M’s training objectives align cross-lingual semantic representations using both parallel and monolingual corpora.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for low-resource languages. In this paper, we propose ERNIE-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks.

1 Introduction

Cross-lingual pretraining learns shared representations, with parallel-corpus training supporting semantic alignment but limited by corpus size. ERNIE-M combines parallel- and monolingual-corpus objectives, and is reported to achieve state-of-the-art results across five cross-lingual benchmarks.

1 Introduction

  • Parallel corpora support semantic alignment and improve downstream tasks, but their limited size constrains cross-lingual models.Source: §1 · p. 1
  • ERNIE-M combines CAMLM on parallel corpora with BTMLM on monolingual corpora to learn cross-lingual semantic alignment.BTMLM predicts tokens in another language and uses them with the input sentences as pseudo-parallel training data.Source: §1 · p. 1
  • ERNIE-M is reported to outperform existing cross-lingual models and achieve state-of-the-art results across XNLI, MLQA, CoNLL, PAWS-X, and Tatoeba.The five benchmarks cover inference, question answering, named entity recognition, paraphrase identification, and cross-lingual retrieval.Source: §1 · pp. 1–2

2 Related Work

Related work covers multilingual discriminative and generative models, then connects back-translation and non-autoregressive translation to ERNIE-M’s approach of generating other-language tokens for pretraining.

2.1 Multilingual Language Models

  • Multilingual models are grouped into discriminative and generative categories.Source: §2.1 · p. 2
  • Examples span mBERT’s monolingual MMLM, XLM’s parallel-corpus TLM token alignment, XLM-R’s MMLM using 2.5T Common Crawl data across 100 languages, and semantic-representation learning in Unicoder and HICTL.ALM learns from cross-lingual code-switch sentences; IN-FOXLM adds a contrastive learning task, while VECO unifies understanding and generation tasks.Source: §2.1 · p. 2
  • Generative multilingual models use denoising, autoencoding, or autoregressive training for translation and other cross-lingual tasks.The examples discussed are MASS (Song et al. 2019), mBART (Liu et al. 2020), XNLG, and mT5 (Xue et al. 2020).Source: §2.1 · p. 2

2.2 Back Translation and Non-Autoregressive Neural Machine Translation

  • ERNIE-M draws on back-translation and non-autoregressive translation by generating another language’s tokens in batches for pretraining and sentence-alignment learning.Back-translation has been used to augment parallel training data, particularly when that data is sparse.Source: §2.2 · p. 2

3 Methodology

ERNIE-M combines parallel-corpus alignment with monolingual-corpus training through CAMLM and BTMLM. CAMLM predicts across paired languages, while BTMLM generates pseudo-parallel tokens and uses them to train masked-token recovery.

3 Methodology

  • ERNIE-M combines MMLM/TLM with CAMLM on parallel data and BTMLM on monolingual data to enhance multilingual representation.Its approach transfers alignment learned from parallel corpora to large-scale monolingual corpora.Source: §3 · pp. 2–3
  • CAMLM restores each side’s masked tokens using only the opposite sentence, aligning cross-lingual semantics while avoiding TLM’s simultaneous-pair information leakage.TLM can use both source and target sentences when predicting masked tokens; CAMLM restricts prediction to the corresponding parallel sentence.Source: §3 · p. 3
  • BTMLM generates pseudo-parallel tokens from monolingual sentences, then uses them with the original input to predict masked tokens and align semantics.The generated pseudo-tokens and monolingual sentence form a new parallel sentence pair for training.Source: §3 · pp. 4–5
  • Stage 1 appends MASK placeholders indicating target language and positions, then predicts corresponding tokens in another language from the monolingual sentence.The placeholders specify where and in which language the model should generate tokens.Source: §3 · p. 4, Figure 3 · p. 4
  • Stage 2 concatenates generated pseudo-tokens with the monolingual sentence and restores MASKs, allowing predictions to use both inputs.This lets the model explicitly learn cross-lingual semantic alignment from monolingual sentences.Source: §3 · pp. 4–5

4 Experiments

ERNIE-M is evaluated across five cross-lingual benchmarks, using 96 languages in pre-training. It reports state-of-the-art results and ablations attribute gains to cross-lingual alignment objectives, with improvements also reported under matched computational overhead.

4.1 Data and Model

  • ERNIE-M uses monolingual and parallel corpora spanning 96 languages and is initialized from XLM-R.The base model has 12 layers and the large model 24 layers.Source: §4.1 · p. 5

4.2 Experimental Evaluation

  • 77.3 average accuracy on XNLI cross-lingual transfer, 1.1 above INFOXLM; ERNIE-MLARGE reaches 82.0, 0.6 above INFOXLM-LARGE.In translate-train-all, ERNIE-M reaches 80.6 accuracy, 0.9 above INFOXLM, while ERNIE-MLARGE reaches 84.2, 0.6 above XLM-R-LARGE.Source: §4.2 · p. 5, Table 1 · p. 6
  • ERNIE-M achieves state-of-the-art CoNLL performance, beating XLM-R by 0.45 F1 when trained on English and 0.70 F1 when trained on all languages in the base model.The large model exceeds state-of-the-art by 2.21 F1 in Dutch and 1.6 F1 in German.Source: §4.2 · p. 6, Table 2 · p. 6
  • ERNIE-M achieves state-of-the-art MLQA results, outperforming INFOXLM by 0.8 in F1 and 0.5 in EM, and a new PAWS-X state of the art on most languages.The MLQA F1 and EM scores are averaged over five runs.Source: §4.2 · pp. 6–7, Table 3 · p. 7
  • 87.9 on Tatoeba gives ERNIE-M new state-of-the-art retrieval results over VECO 1.0; fine-tuning raises average accuracy across languages to 93.3.The average increases from 87.9 to 93.3 after fine-tuning.Source: §4.2 · p. 7, Table 5 · p. 7

4.3 Ablation Study

  • Compared with TLM, CAMLM and BTMLM improve XNLI by 0.3 and CoNLL by 1.3; adding BTMLM to CAMLM improves XNLI by 0.5 and CoNLL by 0.1.The ablations compare the training objectives using the XLM-R-base initialization and a 50,000-step training run.Source: §4.3 · pp. 7–8, Table 6 · p. 8
  • Trained from scratch, ERNIE-M-SMALL beats XLM-R-SMALL by 4.4 on XNLI and 6.6 on CoNLL; at matched compute, ERNIE-M scores 69.9 and 69.7 versus XLM-R’s 67.3 and 65.6.The matched-compute scores are for XNLI and CoNLL, respectively.Source: §4.3 · p. 8, Table 8 · p. 8
  • ERNIE-M has smaller English-to-other-language performance gaps on XNLI and MLQA than XLM-R and INFOXLM, while increasing pseudo-token proportion lowers perplexity during training.The paper interprets the smaller gaps as better transferability and the perplexity trend as evidence that generated pseudo-parallel tokens help convergence.Source: §4.3 · pp. 8–9, Figure 6 · p. 8

5 Conclusion

ERNIE-M combines CAMLM on parallel corpora with BTMLM on monolingual corpora to address constraints from limited parallel data. It achieves state-of-the-art results across five downstream datasets.

5 Conclusion

  • ERNIE-M achieves state-of-the-art results on XNLI, MLQA, CoNLL, PAWS-X, and Tatoeba.The paper reports results across these various downstream tasks.Source: §5 · p. 9
  • BTMLM aligns cross-lingual semantic representations from monolingual corpora.This is the second of ERNIE-M’s two training objectives.Source: §5 · p. 9
  • CAMLM enhances multilingual representations on parallel corpora as ERNIE-M’s first training objective.ERNIE-M is trained using both monolingual and parallel corpora.Source: §5 · p. 9

A.1 Pre-training Data

ERNIE-M’s pre-training data includes reconstructed CC-100 monolingual data and parallel data; the monolingual corpus covers 96 languages.

A.1 Pre-training Data

  • The monolingual training corpus contains 96 languages.Table 9 is identified as reporting CC-100 statistics used for ERNIE-M pre-training.Source: §A.1 · p. 11, Table 9 · p. 11
  • Some languages share ISO codes, including zh for Simplified and Traditional Chinese and ur for Urdu and Urdu Romanized.The passage notes these examples of code reuse.Source: §A.1 · p. 11
  • Table 10 reports statistics for the parallel data used in ERNIE-M pre-training.The supplied text does not provide the table’s values.Source: Table 10 · p. 11, §A.1 · p. 11

A.2 Hyperparameters for Pre-training

Pre-training uses XLM-R initialization for the base and large models, while the small model is trained from scratch; ERNIE-M shares XLM-R’s vocabulary.

A.2 Hyperparameters for Pre-training

  • Table 11 is identified as listing the hyperparameters used for pre-training.The supplied text does not provide the hyperparameter values.Source: §A.2 · p. 11, Table 11 · p. 11
  • The base and large models are initialized with XLM-R, while the small model is trained from scratch.Table 11 lists the pre-training hyperparameters.Source: §A.2 · p. 11, Table 11 · p. 11
  • ERNIE-M uses the same vocabulary as XLM-R.Source: §A.2 · p. 11

A.3 Hyperparameters for Fine-tuning

Fine-tuning parameters are listed for XNLI, MLQA, CoNLL, PAWS-X, and Tatoeba; test scores for the first four tasks average five runs.

A.3 Hyperparameters for Fine-tuning

  • Test-set scores for XNLI, MLQA, CoNLL, and PAWS-X are averages of five runs with different random seeds.Source: §A.3 · p. 11
  • For each task, the selected model has the best validation-set performance.Source: §A.3 · p. 11
  • Tables 12 and 13 list fine-tuning parameters for XNLI, MLQA, CoNLL, and PAWS-X, while Table 14 lists them for Tatoeba.Source: §A.3 · p. 11

A.4 Results for 15 languages model

ERNIE-M-15 outperforms the current best 15-language cross-lingual model on XNLI in both cross-lingual transfer and translate-train-all settings.

A.4 Results for 15 languages model

  • 77.5 in cross-lingual transfer and 80.7 in translate-train-all outperform HICTL by 0.2 and 0.7, respectively.These are the reported XNLI results for ERNIE-M-15.Source: §A.4 · p. 11

A.5 Results for Cross-lingual Retrieval

ERNIE-M outperforms VECO in most languages on cross-lingual retrieval and achieves state-of-the-art results. Fine-tuning raises average Tatoeba accuracy from 87.9 to 93.3.

A.5 Results for Cross-lingual Retrieval

  • Hardest negative binary cross-entropy fine-tunes ERNIE-M using the same bilingual corpora as pre-training.The method is proposed for cross-lingual retrieval.Source: §A.5 · p. 11
  • 87.9 to 93.3: fine-tuning ERNIE-M raises average Tatoeba accuracy.Table 16 reports results after fine-tuning with hardest negative binary cross-entropy loss and the same bilingual corpora used in pre-training.Source: §A.5 · p. 11, Table 16 · p. 12
  • ERNIE-M outperforms VECO in most languages and achieves state-of-the-art cross-lingual retrieval results.The comparison uses the averaged representation in the middle layer of the best XNLI model.Source: §A.5 · p. 11, Table 16 · p. 12

Related papers

Loading 2012.15674v4…