Source-linked AI summary
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
TL;DR
Cross-lingual pre-training needs representations that transfer across languages, and the paper addresses this through a unified mutual-information framework. It introduces XLCO, jointly trains it with MMLM and TLM on monolingual and parallel corpora, and reports improved performance across cross-lingual benchmarks. The analysis further indicates stronger cross-lingual alignment and transferability, with scope limitations for settings without word-piece overlap and English-pivot parallel data.
Problem
Cross-lingual language representations are needed to overcome language barriers and improve transfer across downstream tasks.
Method
INFOXLM jointly maximizes lower bounds of monolingual token-sequence, cross-lingual token-sequence, and cross-lingual sequence-level mutual information using MMLM, TLM, and XLCO.
Results
INFOXLM outperforms strong baselines across cross-lingual natural language inference, question answering, and sentence retrieval benchmarks.
Takeaways & Limitations
The framework and XLCO improve the cross-lingual transferability and alignment of learned representations.
Takeaways & Limitations
The analysis leaves the no-word-piece-overlap setting for future work, and the training uses English as the pivot for parallel corpora.
Abstract
from arXiv · showhide
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.
1 Introduction
The paper unifies cross-lingual pre-training through mutual-information maximization and introduces XLCO, a sequence-level contrastive task for bilingual sentence pairs. INFOXLM jointly uses monolingual and parallel data, outperforming strong baselines across cross-lingual benchmarks.
- 1 Introduction: Cross-lingual pre-training is framed as maximizing mutual information between multilingual, multi-granularity views.This framework interprets existing pretext tasks through mutual-information lower bounds.
- 1 Introduction: The framework explains MMLM and TLM as maximizing mutual-information lower bounds across monolingual and cross-lingual views.MMLM uses monolingual text, whereas TLM feeds bilingual parallel sentences into the model.
- 1 Introduction: XLCO treats translation pairs as cross-lingual views of the same meaning and distinguishes them from negative examples.Unlike TLM's token-sequence objective, XLCO maximizes sequence-level mutual information.
- 1 Introduction: INFOXLM jointly trains MMLM, TLM, and XLCO using both monolingual and parallel corpora.
- 1 Introduction: INFOXLM outperforms strong baselines on cross-lingual natural language inference, question answering, and sentence retrieval benchmarks.The analysis also reports better cross-lingual transferability.
2 Related Work
Prior work develops multilingual pre-training for understanding and generation, while mutual-information methods provide a broader framework for learning representations from paired views.
- 2 Related Work: mBERT uses multilingual masked language modeling, while XLM adds translation language modeling to learn from parallel corpora.
- 2 Related Work: Unicoder and ALM extend multilingual pre-training with additional strategies for using parallel or codeswitched data.
- 2 Related Work: MASS and mBART pretrain sequence-to-sequence models for machine translation, while XNLG targets cross-lingual language generation.
- 2 Related Work: Contrastive representation methods estimate mutual information through tractable lower bounds such as InfoNCE and variational KL divergence.
- 2 Related Work: Mutual-information methods use view pairs including image features, data augmentations, and different sequence parts.This framework has also been applied to word and contextual embeddings.
3 Information-Theoretic Framework for Cross-Lingual Pre-Training
The paper unifies cross-lingual pre-training as mutual-information maximization across multilingual, multi-granularity views, then derives XLCO for sequence-level alignment. INFOXLM jointly trains MMLM, TLM, and XLCO using monolingual and parallel data.
- Multilingual Masked Language Modeling: MMLM maximizes mutual information between masked tokens and same-language context, while shared anchor points encourage cross-lingual representation correlation.The framework identifies MMLM as maximizing a lower bound of I(c1; x1), with cross-lingual correlation emerging through shared vocabulary, positions, and special tokens.
- Translation Language Modeling: TLM maximizes a lower bound of I(c1, c2; x1) by predicting masked tokens from concatenated translation-pair contexts.Its conditional term I(c2; x1|c1) captures cross-lingual information unavailable from the monolingual context alone.
- Cross-Lingual Contrastive Learning: XLCO maximizes sequence-level mutual information between parallel sentence representations, complementing MMLM and TLM token-sequence objectives.It treats parallel sentences as views and uses contrastive learning with a positive translation and negative samples.
- Cross-Lingual Contrastive Learning: Momentum contrast constructs XLCO negatives with a query encoder, a momentum-updated key encoder, and a fixed-size queue of previously encoded sentences.This design reuses prior encodings as negatives while keeping the negative set size constant during training.
- Cross-Lingual Pre-Training: INFOXLM jointly maximizes lower bounds for monolingual token-sequence, cross-lingual token-sequence, and cross-lingual sequence-level mutual information.The combined pre-training objective uses MMLM, TLM, and XLCO, with equal weights for the loss terms.
4 Experiments
INFOXLM is evaluated across three cross-lingual understanding tasks using monolingual and parallel corpora, with ablations examining its components and design choices. It outperforms baselines across benchmarks, improves transferability and representation alignment, and benefits from jointly using XLCO and TLM.
- Experimental setup: INFOXLM is evaluated on cross-lingual natural language inference, sentence retrieval, and question answering.The experiments use XNLI, Tatoeba, and MLQA benchmarks.
- XNLI: 76.5 averaged accuracy lets INFOXLM outperform XLM-R (reimpl) by 1.5 in cross-lingual transfer on XNLI.INFOXLM also outperforms baselines in the Translate-Train-All setting, and removing XLCO hurts zero-shot transfer in most languages.
- Sentence retrieval: 77.8 and 80.6 averaged top-1 accuracies exceed XLM-R by 20.2 and 21.1 on covered language-pair retrieval in xx →en and en →xx.On 22 language pairs without parallel-data coverage, INFOXLM improves averaged accuracy by 8.1% and outperforms XLM-R on 16 of 22 pairs.
- Question answering: INFOXLM and INFOXLMLARGE obtain the best results against four baselines on MLQA, while removing XLCO reduces performance.The evaluation reports F1 and exact match scores.
- Transfer analysis: INFOXLM reduces cross-lingual transfer gaps on both MLQA and XNLI, indicating better transferability than the baselines.The gap is defined as the difference between English performance and averaged performance on other languages; lower is better.
- Representation and ablation analysis: INFOXLM outperforms XLM-R at all 12 Transformer layers, while XLCO is more effective than TLM for representation alignment although TLM helps zero-shot transfer.Removing TLM further improves representation alignment, and jointly learning XLCO and TLM improves cross-lingual transferability.
- Representation and ablation analysis: Contrast on layer 8 performs better than contrast on layer 12 when TLM is used, motivating XLCO on the universal layer.The authors attribute the alternative behavior without TLM to interference between context-sequence and sequence-level mutual information objectives.
- Representation and ablation analysis: Removing mixup contrast or momentum contrast decreases performance on XNLI and MLQA.The ablations compare INFOXLM against variants without mixup contrast or momentum contrast.
5 Conclusion
INFOXLM combines monolingual and parallel corpora with masked language modeling, translation language modeling, and cross-lingual contrastive learning. Experiments show considerably improved performance on several cross-lingual language understanding tasks.
- INFOXLM is trained on both monolingual and parallel corpora.
- The model jointly uses masked language modeling, translation language modeling, and the newly introduced cross-lingual contrastive task.Cross-lingual contrast treats bilingual pairs as views of the same meaning and makes their representations more similar than negative examples.
- INFOXLM considerably improves performance on several cross-lingual language understanding tasks.
6 Ethical Considerations
The work addresses English-centric NLP research and applications by focusing on cross-lingual pre-training. It aims to support transfer from high-resource to low-resource languages and improve access to NLP applications.
- English-centric NLP research and applications make it harder for non-English users to access NLP-related services.
- Cross-lingual pre-training transfers end-task knowledge from high-resource languages to low-resource languages.
- Incorporating parallel corpora into pre-training improves training efficiency.
A Pre-Training Data
The pre-training data comprise reconstructed CCNet monolingual texts and parallel data. The monolingual corpus covers 94 languages, with corpus and parallel-data statistics reported in the appendix tables.
- The appendix reports statistics for the monolingual CCNet corpus in Table 8.
- The appendix reports statistics for the parallel data used in pre-training in Table 9.
B Results of Training From Scratch
The from-scratch experiments use BERT-base-sized Transformer models with random initialization and specified optimization and contrastive-training schedules. INFOXLMSCRATCH significantly outperforms MMLMSCRATCH on XNLI and MLQA, and ablations support the contributions of the pre-training objectives.
- The from-scratch models follow BERT-base Transformer size and hyperparameters and use random initialization.
- Training uses Adam with batch size 256 for 1M steps, a 10K-step warmup, and peak learning rate 0.0001.
- Cross-lingual contrast uses a queue length of 16,384 and enables the key encoder after a 200K-step warmup.
- INFOXLMSCRATCH significantly outperforms MMLMSCRATCH on both XNLI and MLQA.Table 10 reports ablations of XLCO, TLM, and MMLM, with findings agreeing with Table 7.
C Hyperparameters for Pre-Training
This section presents the hyperparameters used for INFOXLM pre-training and fine-tuning, including the shared vocabulary choice and model-size-specific fine-tuning tables.
- INFOXLM pre-training uses the same vocabulary as XLM-R.
- Table 11 lists the hyperparameters used for INFOXLM pre-training.
- Table 12 lists fine-tuning hyperparameters for BASE-size models on XNLI and MLQA.
- Table 13 lists fine-tuning hyperparameters for LARGE-size models on XNLI and MLQA.
D Hyperparameters for Fine-Tuning
Fine-tuning hyperparameters are searched jointly across languages for XNLI and MLQA, with task-specific model-selection procedures and scores averaged over five random seeds.
- Fine-tuning hyperparameters are searched on the joint validation set of all languages for each task.The joint validation setting is denoted #M=1.
- For XNLI, the model is evaluated every 5,000 steps and selected by validation accuracy.
- For MLQA, the final learned model is used directly.
- Final scores are averaged over five random seeds.