Source-linked AI summary
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou
TL;DR
The paper addresses the need for broader cross-lingual benchmark coverage across languages and task types. It introduces XGLUE and extends Unicoder for understanding and generation pre-training, finding strong cross-lingual transfer across the benchmark. The comparison between Unicoder's generation variants is not fully fair because they use different denoising tasks and generation mechanisms.
Problem
Existing GLUE-style evaluation is English-labeled and limited to natural language understanding, motivating broader cross-lingual evidence across languages and task types.
Method
The paper introduces XGLUE with 11 multilingual cross-lingual tasks and extends Unicoder for both understanding and generation, comparing it with multilingual baselines.
Results
Unicoder performs slightly better than M-BERT and XLM-Rbase on nine understanding tasks, while its generation variants perform significantly better than those baselines on two generation tasks.
Takeaways & Limitations
XGLUE provides a benchmark for evaluating cross-lingual pre-trained models across diverse understanding and generation tasks and languages.
Takeaways & Limitations
The UnicoderxFNP and UnicoderxDAE comparison is not fair because the variants use different denoising tasks and generation mechanisms; combining them is left for future work.
Abstract
from arXiv · showhide
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.
1 Introduction
XGLUE is introduced to support cross-lingual pre-training and evaluation across diverse tasks and languages. It covers both understanding and generation and provides an extended Unicoder baseline alongside multilingual model comparisons.
- XGLUE supports training cross-lingual pre-trained models with multilingual and bilingual corpora and evaluating their cross-lingual generalization.
- The benchmark provides 11 diversified tasks spanning natural language understanding and generation scenarios.
- XGLUE includes labeled data in multiple languages for each task, unlike GLUE's English-only natural language understanding setup.
- The paper extends Unicoder for understanding and generation tasks and evaluates it as a strong baseline, alongside 12-layer Multilingual BERT, XLM, and XLM-R.
2 XGLUE Benchmark1
XGLUE organizes 11 downstream tasks across understanding and generation, with English-only training data and evaluation across multiple languages. Its benchmark combines established datasets with newly constructed tasks and broad language coverage.
- The two pre-training corpora include a 101G multilingual corpus covering 100 languages and a 146G bilingual corpus covering 27 languages.
- XGLUE selects 11 cross-lingual tasks in three groups: single-input understanding, pair-input understanding, and generation.
- Every task provides training data only in English, requiring models to transfer task ability to test sets in other languages.
- The benchmark covers named entity recognition, POS tagging, news classification, machine reading comprehension, paraphrase identification, matching, ranking, and generation tasks.
- XGLUE's 11 downstream tasks cover 19 languages, with Table 3 identifying the language set and annotation sources.
- The generation tasks are Question Generation and News Title Generation, both evaluated with BLEU-4.
3 Pre-train Unicoder for Cross-lingual Understanding Tasks
For cross-lingual understanding, the paper uses a simplified Unicoder baseline with masked language modeling and bilingual translation language modeling pre-training.
- Unicoder is simplified to use two pre-training tasks, MLN and TLM, for cross-lingual understanding tasks.
- The baseline omits Unicoder's additional pre-training tasks to keep the model simple and reduce experimental cost.
- Masked Language Model (MLM): MLM samples sentences from multiple languages in each batch and masks 15% of words using masking, random-token replacement, or unchanged tokens.
- Translation Language Model (TLM): TLM concatenates bilingual sentence pairs and masks words so the model recovers them from bilingual context.
4 Pre-train Unicoder for Cross-lingual Generation Tasks
The paper extends Unicoder to cross-lingual generation with an encoder-decoder architecture and shared BPE embeddings. It introduces denoising auto-encoding and future n-gram prediction as generative pre-training tasks.
- An encoder-decoder architecture with shared BPE embeddings extends Unicoder to generation tasks.
- Multilingual Denoising Auto-Encoding (xDAE): xDAE reconstructs original multilingual text from corrupted input using a denoising auto-encoding objective.
- Multilingual Denoising Auto-Encoding (xDAE): The paper explores shuffling, word dropping, token masking, and Poisson-distributed span masking as xDAE noising strategies.
- Multilingual Future N-gram Prediction (xFNP): xFNP masks token spans, encodes the remaining text, and generates the masked spans as output.
- Multilingual Future N-gram Prediction (xFNP): Instead of predicting only the next token, xFNP generates n future tokens simultaneously to encourage explicit future planning and prevent over-fitting to local correlations.
- Multilingual Future N-gram Prediction (xFNP): The xFNP configuration sets n = 2 and weights its two loss components equally with α0 = α1 = 1.
5 Related Work
XGLUE broadens cross-lingual benchmarking beyond English-only understanding by combining multilingual annotations with understanding and generation tasks. It also extends Unicoder and situates the benchmark among existing cross-lingual pre-trained models.
- XGLUE expands beyond GLUE by providing multilingual annotations and both natural language understanding and generation tasks.GLUE contains nine English-labeled understanding tasks, whereas XGLUE includes 11 cross-lingual tasks.
- XGLUE includes five existing multilingual datasets and introduces six new tasks from Search, Ads, and News scenarios.The new tasks are intended to increase the benchmark’s practical value.
- Unlike concurrent XTREME, XGLUE covers both cross-lingual understanding and generation tasks.The paper describes this as the first such combination in cross-lingual dataset construction efforts, to the authors’ knowledge.
- The paper extends Unicoder with additional cross-lingual pre-training tasks and evaluates it as a strong baseline for both understanding and generation on XGLUE.The extension draws on ideas from BART and ProphetNet for cross-lingual generation.
- The baseline development combines Common Crawl and bilingual corpora, while XLM-R uses a much larger multilingual corpus without translation-language-model pre-training.Unicoder previously achieved better XNLI results than XLM, and XLM-R became state of the art on XNLI.
6 Experiments
Experiments evaluate cross-lingual pre-trained models across XGLUE’s understanding and generation tasks, including comparisons of pre-training scale, text-noising strategies, and fine-tuning regimes.
- 6.2 Main Result: UnicoderLC performs slightly better than M-BERT and XLM-Rbase on nine understanding tasks, while using multilingual and bilingual corpora with TLM.
- 6.2 Main Result: UnicoderLC performs better than UnicoderSC, indicating an advantage for pre-training on the larger corpus.
- 6.2 Main Result: UnicoderxDAE_SC and UnicoderxFNP_SC show good cross-lingual transfer and significantly outperform M-BERT and XLM-Rbase on two generation tasks.
- 6.2 Main Result: UnicoderxFNP_SC performs slightly better than UnicoderxDAE_SC, but the comparison is not fair because their noising tasks and generation mechanisms differ.
- 6.3.1 Pivot-language Fine-tuning: Selecting a specific pivot language can further improve downstream performance; the best XNLI pivots include Spanish, Greek, and Turkish, while French is best for NTG.
- 6.3 Fine-tuning Strategies: Multi-language fine-tuning outperforms pivot-language fine-tuning on XNLI and NTG, whereas multi-task fine-tuning benefits PAWS-X and QADSM but not XNLI, NC, or QAM.
- 6.3.4 Impacts of Text Noising Strategies: UnicoderxFNP_SC significantly outperforms XNLG, supporting the effectiveness of the fourth text-noising strategy for generation tasks.
7 Conclusion
The paper presents XGLUE as a new cross-lingual benchmark and reports comprehensive evaluations with interesting findings.
- XGLUE is presented as a new cross-lingual benchmark for comprehensive evaluations.