How Language-Neutral is Multilingual BERT?

1 Introduction

• Earlier zero-shot studies showed cross-lingual transfer mainly for morphology and syntax, leaving mBERT’s semantic cross-lingual properties open.Pires et al. also reported promising initial sentence-retrieval results.Source: §1 · p. 1
• The authors compare representations directly on sentence retrieval, word alignment, and machine-translation quality estimation to avoid zero-shot transfer’s language-overfitting and tuning issues.They also examine language identification and whether representation similarity tracks phylogenetic language families.Source: §1 · p. 1
• mBERT remains not very language-neutral after language-agnostic fine-tuning, but language identity can be approximated as a constant shift and reduced further by a projection fitted on little parallel data.The authors also attempt to strengthen the language-neutral component through multilingual syntactic and morphological fine-tuning and adversarial language-identity removal.Source: §1 · p. 1

2 Related Work

• Zero-shot dependency parsing used a bilingual projection trained on parallel data, while zero-shot NER and POS transfer depended strongly on typological similarity.Pires et al. (2019) and Wang et al. (2019) are cited for these findings.Source: §2 · p. 1
• Wu and Dredze (2019) trained multilingual POS, NER, and XNLI models but struggled in zero-shot settings; Pires et al. (2019) found mean-difference subtraction improved retrieval for three language pairs.The present work systematically studies the subtraction idea.Source: §2 · p. 2
• Downstream models can extract task-relevant features, but their success does not directly demonstrate language-neutral representations because task models may use language knowledge.The authors argue that direct representation comparisons avoid this risk.Source: §2 · p. 2

3 Centering mBERT Representations

• The authors hypothesize that an mBERT sentence representation combines a language-specific component identifying its language and a language-neutral component capturing language-independent meaning.They assume the language-specific component is similar across sentences in the same language.Source: §3 · p. 2
• Centering estimates each language’s centroid as the mean of its sentence representations and subtracts that centroid from contextual embeddings.This places each language’s average representation at the origin.Source: §3 · p. 2
• The authors compare original and centered representations across model layers, using both [cls] and mean-pooled vectors for single-vector sentence tasks.The passage specifies that these sentence-task representations are tested in both forms.Source: §3 · p. 2

4 Probing Tasks

• Language identification trains a linear classifier on sentence representations; language similarity measures whether language centroids cluster by language family using hierarchical clustering and V-measure.The similarity analysis quantifies earlier observations that related languages tend to have similar representations on average.Source: §4 · p. 2
• Sentence retrieval selects the closest representation among parallel-side sentences using cosine distance, and also tests projections into English space fitted on small parallel sets.Plain and centered [cls] and mean-pooled representations are evaluated alongside the projections.Source: §4 · p. 2
• Word alignment finds a minimum weighted edge cover between sentence tokens, weighting edges by cosine distance and representing split tokens by averaged subword embeddings.Alignment is evaluated with F1 over sure and possible links in a manually aligned gold standard.Source: §4 · pp. 2–3
• MT quality estimation evaluates system output without a reference translation, using correlation with Human-targeted Translation Error Rate, the number of edits needed for human correction.The task is more difficult than the preceding two because it requires capturing finer-grained meaning differences.Source: §4 · p. 3
• For MT quality estimation, the authors compare source-to-output cosine distances from plain and centered representations with trained bilingual projection and supervised regression.The evaluation examines whether these distances reflect translation quality.Source: §4 · p. 3

5 Experimental Setup

• The public cased mBERT model has 12 layers, 768-dimensional representations, 12 attention heads, and a shared 120k-wordpiece vocabulary.Its hidden layer dimension is 3072.Source: §5 · p. 3
• For language identification, each language contributed 110k Wikipedia sentences for training, plus 5k each for validation and testing; training sentences also estimated centroids.Sentences had at least 20 characters; Figure 1 plots 8th-layer mean-pooled language centroids with language families highlighted.Source: §5 · p. 3, Figure 1 · p. 3
• Sentence retrieval used 3,000 WMT14 test sentences in Czech, English, French, German, Hindi, and Russian; projection experiments used WMT14 development data.Table 1 reports language-identification accuracy from the best-scoring layers.Source: §5 · p. 3, Table 1 · p. 3
• Word alignment was evaluated on manually annotated English–Czech, Swedish, German, French, and Romanian data, with FastAlign given 1M additional ParaCrawl sentence pairs.The datasets contained 2.5k Czech, 192 Swedish, 508 German, 447 French, and 248 Romanian sentences.Source: §5 · p. 3
• MT quality estimation used English–German WMT19 QE Shared Task training and test data containing source sentences, automatic translations, and manual corrections.Source: §5 · p. 3

Language Identification

• Centering considerably decreases language-identification accuracy, especially for mean-pooled embeddings, indicating that it removes much language-specific information.Source: §6 · p. 3

Language Similarity

• Language-centroid similarity tends to track language similarity, and hierarchical clustering of the centroids mostly recovers language families.Table 2 reports V-measure for clustering families represented by at least three mBERT languages.Source: §6 · pp. 3–4, Table 2 · p. 4

6 Results

• Centering dramatically improves sentence retrieval, and learned projections bring accuracy close to perfect; mBERT also surpasses FastAlign on word alignment, but projection-based sentence distances remain weak for MT quality estimation.The projection brings a much greater retrieval improvement than centering, while providing negligible word-alignment improvement.Source: §6 · p. 4, Table 3 · p. 4, Table 4 · p. 4

Parallel Sentence Retrieval

• Sentence-retrieval accuracy is higher for mean-pooled states than for [cls] embeddings and varies by mBERT layer.The comparison covers average accuracy over all 30 language pairs.Source: §6 · p. 4, Table 3 · p. 4, Figure 2 · p. 4

Word Alignment

• mBERT-based word alignment surpasses FastAlign even with its large parallel corpus, while explicit projection has a negligible effect on alignment performance.The projection-learning algorithm adds only .005 F1 points.Source: §6 · p. 4, Table 4 · p. 4

MT Quality Estimation

• For MT quality estimation, centering and projection yield only mild translation-quality correlations, far below supervised regression, because sentence complexity appears more indicative.Source: §6 · p. 4

7 Fine-tuning mBERT

• The study evaluates UDify, fine-tuned for multilingual dependency parsing, and lng-free, fine-tuned to remove language-specific information.Source: §7 · p. 4

7.1 UDify

• Source-only or MT-output-only supervised regression still shows respectable correlation with HTER, suggesting sentence structure is more useful than source–output comparison.Table 5 covers English-to-German translation on WMT19 data.Source: §7.1 · p. 4, Table 5 · p. 5
• UDify fine-tuning does not remove language identity and makes mBERT representations less semantically cross-lingual.The authors contrast this with earlier zero-shot parsing results for morphology and syntax.Source: §7.1 · p. 5
• For language-ID classifiers, adversarial training is applied at each layer to [cls] and mean-pooled representations alongside masked language modeling.Source: §7.2 · p. 5
• Adversarial training lowers language-ID accuracy most in deeper layers, while leaving other tasks unaffected.The setup jointly trains masked language modeling and adversarial language-ID classifiers.Source: §7.2 · p. 5, Figure 3 · p. 5

• mBERT embeddings do not represent similar semantic phenomena similarly across languages, so they are not directly usable for zero-shot cross-lingual tasks.Source: §8 · p. 5
• Language centroids cluster by language family, and this property persists after cross-lingual fine-tuning and adversarial language-identity removal.Source: §8 · p. 5
• Centering enables good cross-lingual sentence retrieval, while bilingual word alignment is invariant to the language-specific shift.Source: §8 · p. 5
• A supervised projection fitted on a small parallel corpus can yield a good cross-lingual representation.Source: §8 · p. 5