Source-linked AI summary
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Ming Zhou
TL;DR
Cross-lingual transfer remains difficult because existing pre-trained models cannot directly handle tasks whose training and test instances use different languages. Unicoder addresses this with multiple cross-lingual pre-training tasks and multilingual fine-tuning, improving results on XNLI and XQA relative to the stated baselines.
Problem
Existing pre-trained models transfer knowledge across tasks but cannot handle tasks when training and test instances are in different languages.
Method
Unicoder combines five pre-training tasks, including three new cross-lingual tasks, and uses a multi-language fine-tuning approach.
Results
1.8% averaged accuracy improvement is obtained on XNLI across 15 languages, and 5.5% on XQA across French and German.
Takeaways & Limitations
The experiments report large improvements on XNLI and XQA, with more languages in fine-tuning producing better results.
Abstract
from arXiv · showhide
We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM, three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.
Introduction
Unicoder addresses the inability of existing pre-trained models to transfer across languages by combining multiple cross-lingual pre-training tasks. Evaluated against Multilingual BERT and XLM on XNLI and XQA, it reports improvements and introduces a new XQA benchmark.
- Motivation: Pre-trained models transfer knowledge with limited task data but cannot handle training and test instances written in different languages.
- Prior approaches: Multilingual BERT uses a shared vocabulary across 104 languages, while XLM adds translation language modeling over bilingual sentence pairs.
- Unicoder: Unicoder extends cross-lingual pre-training with cross-lingual word recovery, paraphrase classification, and masked language modeling alongside existing tasks.
- Results: 1.8% averaged accuracy improvement is obtained on XNLI across 15 languages, using Multilingual BERT and XLM as baselines.
- Results: 5.5% averaged accuracy improvement is obtained on XQA across French and German, and the paper builds XQA as a new cross-lingual dataset.
- Contributions: The paper reports four contributions: three new pre-training tasks, the XQA dataset, multilingual fine-tuning improvements, and new state-of-the-art XNLI results.
Related work
Related work frames cross-lingual pre-training as transfer across languages requiring aligned representations. Unicoder builds on multilingual BERT and XLM while moving from word or sentence embeddings toward a pre-trained encoder.
- Cross-lingual pre-training: Cross-lingual pre-training transfers knowledge between different source and target language domains through a high-quality cross-lingual representation space.
- Cross-lingual pre-training: Earlier work aligned word representations with dictionaries or orthogonal transformations, sometimes without parallel data.
- Prior encoders: Multilingual sentence encoders have been used for XNLI, while Unicoder instead produces a pre-trained encoder rather than fixed word or sentence embeddings.
- Multilingual BERT and XLM: Multilingual BERT applies masked language modeling to monolingual documents across 104 languages, whereas XLM applies translation language modeling to concatenated parallel sentences.
Approach
Unicoder extends XLM-style pre-training with three cross-lingual tasks and a multi-language fine-tuning strategy. Its tasks learn word relations, sentence-level meaning equivalence, and language mappings from cross-lingual documents, while fine-tuning combines source and translated target-language data.
- Model setup: Unicoder follows XLM's network structure and uses a shared BPE vocabulary built from corpora across all languages.Rich-resource language corpora are downsampled to reduce excessive character-level splitting in target-language words.
- Pre-training tasks: Three new tasks augment masked and translation language modeling: cross-lingual word recovery, paraphrase classification, and masked language modeling.Their training data comes from existing large-scale machine-translation corpora.
- Pre-training tasks: Cross-lingual word recovery learns word alignments by representing source words through target-language word embeddings and predicting the original source sequence.The task is motivated by attention matrices in neural machine translation and recovers all words simultaneously without using the original words as input.
- Pre-training tasks: Cross-lingual paraphrase classification concatenates sentences from different languages and classifies whether they have the same meaning.Its dataset uses bilingual translation pairs as positives and hard negative pairs selected with a lightweight paraphrase model.
- Pre-training tasks: Cross-lingual masked language modeling applies masked language modeling to documents whose sentences alternate between languages and translations.The constructed documents are truncated to 256 sequence length before being fed to Unicoder.
- Fine-tuning strategy: Multi-language fine-tuning combines source-language training data with pseudo target-language data translated from the source, although it can harm performance in a few cases.For multiple target languages, all are used simultaneously; machine-translated data is used for languages without training data.
Experiment
Experiments evaluate Unicoder on XNLI and the newly built XQA dataset against multilingual pre-training baselines. Across fine-tuning settings, Unicoder achieves leading results, with further gains from multilingual fine-tuning.
- Evaluation setup: Experiments compare Unicoder with Multilingual BERT and XLM on XNLI, and with XLM on the newly built XQA dataset.XNLI evaluation covers 15 languages, while XQA reports results for French and German.
- XNLI results: Unicoder obtains the best XNLI result in every fine-tuning setting, reaching 76.9%, 74.9% and 75.4% average accuracy in three settings.The reported settings are TRANSLATE-TRAIN, TRANSLATE-TEST and Cross-lingual TEST, respectively.
- Fine-tuning strategy: Multi-language Fine-tuning improves XLM by 1.1% and Unicoder by 1.6% on XNLI, and outperforms TRANSLATE-TRAIN on average.The approach fine-tunes using translated training data across multiple languages.
- XNLI results: 78.5% accuracy is achieved by combining Unicoder with Multi-language Fine-tuning, a 1.8% gain over XLM fine-tuned with TRANSLATE-TRAIN.This combination is reported as a new state of the art on XNLI.
- XQA results: On XQA, Unicoder outperforms XLM in every fine-tuning setting and achieves a 5.5% gain over the XLM + TRANSLATE-TRAIN baseline.The reported XQA result is 69.7%, with the average column computed over French and German.
Analysis
The analysis examines how Unicoder’s pre-training tasks and multilingual fine-tuning affect cross-lingual performance. Results generally favor using more languages and show that transfer quality varies across language pairs.
- Ablation Study: Removing any cross-lingual pre-training task causes a performance drop, with word recovery removal hurting XNLI especially.Cross-lingual paraphrase classification produces the smallest drop among the examined tasks.
- Ablation Study: 1.6% accuracy on XNLI and 3.3% on XQA are gained with Multi-language fine-tuning.The analysis attributes these gains to fine-tuning across multiple languages.
- The relation between language number and fine-tuning performance: 15 languages always outperform 1 language across the tested languages, although Chinese and Russian do not consistently follow the increasing-language trend.The experiment compares fine-tuning settings involving 1, 2, 6, and 15 languages.
- The relation between language number and fine-tuning performance: English improves stably when fine-tuning expands from 1 to 6 and 15 languages, despite having human-labeled source-language data.The improvement is not stable in every two-language pairing.
- The relation between English and other languages: Joint fine-tuning with two languages improves average performance for most languages, but Vietnamese and Urdu cause drops, and English outcomes depend on the partner language.French and Spanish improve English, whereas Vietnamese and Thai lead to a large English performance drop.
- The relation between different languages: Unicoder transfers knowledge across languages, with Russian showing the strongest generalization and Urdu the weakest reported transfer behavior.Fine-tuning usually gives the best performance in the same language, except Greek and Urdu.
- The relation between different languages: Transfer among English, Spanish, and French is easier than transfer involving other languages.The passage relates this pattern to stronger machine translation performance among these languages.
Conclusion
The conclusion presents Unicoder as a language-insensitive encoder built with three new cross-lingual pre-training tasks and a multilingual fine-tuning strategy. Experiments report improvements on XNLI and XQA, with broader fine-tuning generally helping even rich-resource languages.
- Conclusion: Unicoder introduces cross-lingual word recovery, paraphrase classification, and masked language modeling to build a language-independent encoder.These are three of the paper’s four stated contributions.
- Conclusion: The paper proposes Multi-language Fine-tuning and builds XQA as a new cross-lingual benchmark dataset.Both are listed among the paper’s four contributions.
- Conclusion: Experiments on XNLI and XQA report large improvements, and the approach achieves new state-of-the-art results on both datasets.The conclusion also reports that using more fine-tuning languages generally improves results, including for rich-resource languages.