Source-linked AI summary
Emerging Cross-lingual Structure in Pretrained Language Models
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, Veselin Stoyanov
TL;DR
The paper asks why multilingual masked-language models transfer across languages and tests the roles of vocabulary, domains, and parameter sharing. It combines multilingual ablations with post-hoc alignment of independently trained monolingual BERTs. The results show that shared top-layer parameters can support transfer without shared vocabulary or domain similarity, while independently trained representations can also be aligned linearly.
Problem
The central question is which pretraining factors explain effective cross-lingual transfer in multilingual masked-language models.
Method
The paper conducts multilingual ablations and aligns independently trained monolingual BERT representations with linear mappings, while probing model similarity using CKA.
Results
Shared top-layer parameters support cross-lingual transfer even without shared vocabulary or domain similarity.
Takeaways & Limitations
The findings indicate that multilingual training can automatically align universal structure, and independently trained representations may be aligned without jointly training on all unlabeled data.
Takeaways & Limitations
The experiments consider language pairs rather than multiple languages jointly, although the procedure naturally scales to multiple languages.
Abstract
from arXiv · showhide
We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of several factors that influence why these models are so effective for cross-lingual transfer. We show, contrary to what was previously hypothesized, that transfer is possible even when there is no shared vocabulary across the monolingual corpora and also when the text comes from very different domains. The only requirement is that there are some shared parameters in the top layers of the multi-lingual encoder. To better understand this result, we also show that representations from independently trained models in different languages can be aligned post-hoc quite effectively, strongly suggesting that, much like for non-contextual word embeddings, there are universal latent symmetries in the learned embedding spaces. For multilingual masked language modeling, these symmetries seem to be automatically discovered and aligned during the joint training process.
1 Introduction
The paper systematically studies why multilingual masked-language pretraining enables cross-lingual transfer. It finds that shared vocabulary and domain similarity are not necessary, while shared Transformer parameters and alignable representations are central.
- Research focus: The study provides a detailed ablation of bilingual masked-language-model pretraining regimes across zero-shot cross-lingual transfer tasks.It systematically varies vocabulary overlap, parameter sharing, and domain similarity.
- Main findings: Shared vocabulary and domain similarity are not required for language-universal representations to emerge.Transfer remains possible without shared vocabulary or similar domains.
- Representation alignment: Monolingual BERTs trained independently in different languages learn representations that align surprisingly well.A linear mapping aligns their embedding and contextual representation spaces, paralleling cross-lingual word-embedding alignment.
- Implications: The findings suggest that pretrained models may be adapted to new languages with little additional training and independently trained representations may be aligned without joint training on all unlabeled data.These are presented as implications and possible directions rather than established guarantees.
- Main findings: Parameter sharing plays the most important role in learning cross-lingual representations, while shared BPE, shared softmax, and domain similarity contribute less.The paper reports that vocabulary overlap accounts for only a few points of transfer performance.
2 Background
The background reviews multilingual language-model transfer and prior alignment work. It motivates testing whether deep contextual representations exhibit language-universal structure analogous to word-embedding spaces.
- Multilingual pretraining: mBERT and XLM established effective zero-shot cross-lingual transfer across languages without requiring additional target-language training.mBERT used multilingual Wikipedia data and no parallel data during training, while XLM also investigated parallel-sentence objectives.
- Representation alignment: Cross-lingual word-embedding alignment uses bilingual dictionaries and exploits similarities in embedding-space geometry across languages.Related work extended linear alignment to ELMo and mBERT representations for dependency parsing.
- Representation similarity: The paper hypothesizes that language-universal structures also emerge in pretrained contextual models.Because BERT representations are deep and contextual, the paper uses neural-activation similarity measures to study them.
- Representation similarity: CKA and related methods provide tools for comparing representations across layers and independently trained models.The background contrasts these approaches with CCA and SVCCA.
3 Cross-lingual Pretraining
The experiments pretrain multilingual masked-language models and evaluate them through zero-shot transfer to several NLP tasks. The setup also probes how parameter sharing and model components affect cross-lingual representation learning.
- Experimental setup: The study uses standard multilingual masked-language modeling and evaluates natural language inference, named entity recognition, and dependency parsing.Models are fine-tuned on English and directly transferred to target languages.
- Pretraining: Training masks 15% of tokens, replacing them 80% with a mask token, 10% with the original word, and 10% with a random word.The model predicts masked tokens from their context.
- Pretraining: Each model uses an 8-layer Transformer with 12 heads and GELU activations.The output softmax layer is tied to the input embeddings.
- Evaluation tasks: NER is evaluated with span-level F1 on WikiAnn or paired English and Chinese datasets.The task uses BIO tagging with a linear softmax classifier over the first subword of each word.
- Parameter-sharing analysis: Figure 1 probes cross-lingual transfer by removing sharing from embedding layers and progressively earlier Transformer layers.This isolates the effects of anchor points and shared structure.
- Evaluation tasks: Dependency parsing uses four Universal Dependencies treebanks and is measured with labeled attachment score.A graph-based parser receives representations of the first subword of each word.
4 Dissecting mBERT/XLM models
A controlled bilingual MLM study finds that parameter sharing is the main driver of cross-lingual transfer, while shared vocabulary, domain similarity, and shared softmax contribute less. Transfer remains possible without anchor points or shared bottom layers, but performance declines for more distant language pairs and when too few layers are shared.
- Experimental factors: The study tests domain similarity, shared vocabulary, shared parameters, and language similarity using bilingual MLMs across English-French, English-Russian, and English-Chinese.The experiments use Wikipedia versus Wiki-CC data and vary vocabulary anchors and which model layers are shared.
- Domain Similarity: Domain mismatch has a modest effect overall, with XNLI and parsing dropping around 2 points and NER dropping over 6 points on average.For English and Chinese NER, where neither side comes from Wikipedia, the drop is around 2 points.
- Anchor points: Effective transfer is still possible with no shared vocabulary, while adding anchor points helps especially for less closely related language pairs.Extra anchors improve English-Chinese more than English-French, with over 3 points of improvement on NER and XNLI.
- Anchor points: Removing anchor points changes XNLI and parsing by only around 1 point, while NER improves 1 point relative to default anchors when averaged over three languages.These results indicate that the contribution of anchor points was previously overestimated.
- Parameter sharing: Cross-lingual transfer worsens as fewer layers are shared, reaching random performance when embeddings and the bottom six Transformer layers are separated.Reasonably strong transfer remains possible without tying the bottom three layers, making shared upper-layer parameters central.
- Language similarity: Language similarity matters: more distant language pairs show larger performance losses, and extra anchor points partly close this gap.More complex tasks exhibit larger performance gaps, while language-specific capacity does not appear to solve the problem.
5 Similarity of BERT Models
The paper shows that independently trained monolingual BERT representations can be aligned across languages at word, contextual-word, and sentence levels using simple linear mappings. Similarity is strongest in earlier layers and for more closely related languages.
- Word and contextual-word alignment: Higher Transformer layers yield better contextual word-level alignment than lower layers, while transfer performance peaks at a middle layer.The contextual word-level evaluation uses MUSE supervision and CSLS word-translation retrieval; the transfer evaluation covers parsing and NER.
- Word and contextual-word alignment: Monolingual alignment gives relatively good zero-shot parsing and NER performance, although bilingual MLM generally performs better except for parsing on English-French.The method freezes the first i target-language layers, projects them into English space, and applies task-specific layers trained on English.
- Sentence-level alignment: Sentence retrieval performs surprisingly well after Procrustes alignment learned from only 10k parallel sentences, with the best results at lower layers.Retrieval is evaluated using precision at 1 across four language pairs without fine-tuning the monolingual models.
- Alignment method: Monolingual BERT representations are aligned at non-contextual word, contextual word, and sentence levels using orthogonal mappings.The alignment uses Procrustes mappings learned from bilingual dictionaries or parallel sentences, depending on representation level.
- Neural network similarity: CKA similarity is highest in the first few layers and correlates with sentence-level alignment performance at over 0.9 Pearson correlation across all four language pairs.CKA is invariant to orthogonal transformations, making it suitable for comparing independently trained representations.
- Neural network similarity: Representation similarity and alignment effects are stronger for closely related languages, while language distance reduces similarity in later layers.The comparison includes monolingual, bilingual, and random encoders evaluated on multi-way parallel sentences.
6 Discussion
The discussion concludes that multilingual representations can emerge with only partial Transformer-layer parameter sharing and without shared vocabulary anchors. Independently trained models also develop alignable, language-specific representation spaces, though alignment is stronger for closely related languages.
- Discussion: Multilingual representations emerge with parameter sharing in only some Transformer layers, even without shared vocabulary anchor points.The authors identify shared parameters as sufficient for representations from different languages to map into one embedding space.
- Discussion: Independently trained monolingual masked language models develop isomorphic embedding spaces whose embedding and contextual layers can be aligned linearly.The paper relates this behavior to similarities previously observed in word2vec spaces.
- Discussion: Early Transformer layers are more similar across languages than later layers, and all reported effects are stronger for closely related languages.The authors identify more distant language pairs as an opportunity for further improvement.
A Contextual word-level alignment of bilingual MLM representation
Figure 8 concerns contextual representation alignment across layers for zero-shot cross-lingual transfer.
- The figure compares contextual representation alignment across different layers in a zero-shot cross-lingual transfer setting.