Source-linked AI summary
Multilingual Alignment of Contextual Word Representations
Steven Cao, Nikita Kitaev, Dan Klein
TL;DR
Multilingual BERT achieves substantial zero-shot transfer, but the reasons for its multilingualism and the limits of its alignment are unclear. The paper introduces contextual word retrieval and a fine-tuning-based alignment procedure, finding improved XNLI performance, predictive alignment scores, and corrected systematic deficiencies. These results support contextual alignment as a useful framework for analyzing and improving multilingual pre-trained models.
Problem
It is unclear why multilingual BERT supports strong zero-shot transfer and whether its cross-lingual transfer can be improved.
Method
The paper evaluates contextual alignment with word retrieval on parallel sentences and improves multilingual BERT using fine-tuning with a regularization term.
Results
2.78% average XNLI accuracy improvement over the base model accompanies alignment, which matches translate-train models for Bulgarian and Greek; retrieval accuracy correlates with zero-shot transfer.
Takeaways & Limitations
Contextual alignment is useful for understanding multilingual BERT, predicting cross-lingual transfer, and correcting systematic alignment deficiencies.
Takeaways & Limitations
The approach challenges rotation-based alignment because contextual models may violate the isometry assumption, and its usage-based explanation is framed as a hypothesis.
Abstract
from arXiv · showhide
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.
1 INTRODUCTION
The paper frames multilingual BERT’s unexplained zero-shot transfer as a question of contextual embedding alignment, then proposes procedures to measure and improve that alignment. Alignment improves XNLI transfer and reveals systematic strengths and weaknesses in BERT.
- Motivation: Multilingual BERT transfers across languages despite limited explanations for why its shared vocabulary and multilingual pre-training produce zero-shot performance.The model is pre-trained on 104 languages and reaches 74.3% Spanish accuracy after English fine-tuning, 7.1% below English accuracy.
- Approach: Contextual word retrieval evaluates alignment by asking a model to identify the translated word and sentence corresponding to a word in a parallel corpus.This task measures representations for words in context rather than isolated word vectors.
- Results: 2.78% average XNLI accuracy improvement over the base model follows alignment, with Bulgarian and Greek matching translate-train models.The alignment procedure uses 10K–250K parallel sentences per language; Bulgarian and Greek gains are almost 5% each.
- Method comparison: Comparing methods, fine-tuned BERT outperforms fastText with sentence information, while fastText outperforms rotation-aligned BERT when words have multiple occurrences.The comparison supports the paper’s focus on contextual rather than purely word-vector alignment.
- Analysis: Word retrieval shows that alignment quality predicts downstream zero-shot transfer and that BERT’s alignment varies systematically across linguistic categories.Base BERT aligns open-class parts of speech and word pairs with large usage-frequency differences less effectively.
2 RELATED WORK
Prior work aligns static word vectors with mappings, incorporates context by simplifying or aligning contextual representations, and uses parallel data during pre-training. Other studies probe multilingual BERT’s transfer patterns and limitations.
- Word vector alignment: Word-vector alignment commonly learns mappings between independently trained language-specific vectors and evaluates them with bilingual lexicon induction.Some approaches use minimal supervision by alternating between learning a linear map and inducing a dictionary.
- Contextual alignment: Context-aware alignment is challenging because different occurrences of the same word have different contextual embeddings.Prior work addressed this by removing context and aligning an average sense, including rotational alignment of contextual ELMo embeddings.
- Parallel-text pre-training: Other approaches incorporate parallel texts directly into contextual pre-training through sentence-pair similarity objectives or cross-lingual pre-training objectives.These methods add parallel data to monolingual pre-training procedures.
- Multilingual BERT analysis: Probing studies find that multilingual BERT transfers even between dissimilar languages, while transfer is stronger between typologically similar languages.The same work concludes that BERT remains weaker for certain language pairs.
3 METHODS
The paper defines contextual alignment through translation-linked word representations in parallel sentences, evaluates it with contextual word retrieval, and improves multilingual BERT by fine-tuning its weights.
- Multilingual BERT: Multilingual BERT is pretrained on 104 languages using masked word prediction, next sentence prediction, and a shared WordPiece vocabulary.Each batch contains sentences from all languages.
- Contextual alignment: Contextual alignment requires similar representations for translated words within parallel sentences, represented by word-position pairs and contextual embeddings.The paper retains the final subword vector when BERT tokenization splits aligned words.
- Contextual word retrieval: Contextual word retrieval evaluates alignment by finding the correct translated word and sentence for a queried word in a parallel corpus.Accuracy is based on exact matches, averaged across source-to-target and target-to-source retrieval directions; CSLS is used as the similarity function.
- Fine-tuning alignment: The alignment objective minimizes distances between contextual representations of word pairs while regularization preserves the target language embeddings' initialization.Gradient steps are applied directly to the pretrained model weights, with λ = 1 in the experiments.
- Fine-tuning alignment: Unlike rotation-based post-hoc methods, the procedure jointly moves non-English embeddings toward English without assuming contextual embedding spaces are approximately isometric.The multilingual procedure samples equal-sized batches from multiple language-English corpora and updates all languages simultaneously.
- Evaluation: On zero-shot XNLI, fine-tuning-based alignment improves average accuracy by 2.78% over base BERT and matches translate-train models for Bulgarian and Greek.The evaluation compares base BERT, rotation-based methods, and translate-train or fully supervised references; model selection uses average accuracy across six languages.
4 RESULTS
The alignment procedure substantially improves multilingual BERT’s zero-shot XNLI transfer, remains effective with limited parallel data, and outperforms rotation-based alternatives overall.
- ZERO-SHOT XNLI TRANSFER: Rotation-based alignment produces small gains for Bulgarian, German, and Spanish but is sub-optimal overall.The comparison supports the benefit of the proposed procedure’s greater expressivity for contextual alignment.
- ALIGNMENT WITH LESS DATA: Alignment improves transfer with as little as 10K sentences per language.The method is evaluated with varying amounts of parallel data because large datasets may not always be available.
- WORD RETRIEVAL: Fine-tuned aligned BERT is consistently effective across languages and matches fastText when context is unnecessary.Table 3 compares aligned sentence-augmented fastText with BERT before and after alignment.
5 ANALYSIS
Word retrieval is used to analyze multilingual BERT’s alignment across languages, linguistic categories, and usage-frequency differences, revealing systematic weaknesses that alignment addresses.
- WORD RETRIEVAL: Fine-tuned BERT outperforms fastText in contextual retrieval, while fastText outperforms rotation-aligned BERT.In non-contextual retrieval, fine-tuned BERT matches fastText, indicating improved type-level alignment alongside contextual alignment.
- WORD RETRIEVAL: Word retrieval accuracy is highly correlated with downstream zero-shot performance, supporting retrieval as a predictive transfer measure.Before alignment, performance varies greatly across languages; after alignment, it is consistently effective.
- WORD RETRIEVAL: Different-script languages such as Bulgarian and Greek show worse initial transfer, while post-alignment performance becomes equally aligned with English.The discrepancy is consistent with a shared-vocabulary account of multilingual BERT’s pre-training behavior.
- WORD RETRIEVAL: Table 4 groups non-contextual retrieval results into lexically overlapping, closed-class, and open-class parts of speech.Particle, Symbol, Interjection, and Other tags are omitted, and word pairs seen during training are retained for coverage.
- WORD RETRIEVAL: Multilingual BERT aligns open-class words less effectively than closed-class words and has high alignment for numerals, punctuation, and proper nouns.The latter groups benefit from high lexical overlap and the shared vocabulary.
- USAGE HYPOTHESIS FOR ALIGNMENT: Accuracy drops significantly as source-target frequency-rank differences increase, and alignment remedies this deficiency.Frequency-rank difference measures discrepancies in word usage statistics.
6 CONCLUSION
The paper concludes that contextual alignment is useful for understanding and improving multilingual pre-trained models, including with small amounts of parallel data.
- CONCLUSION: Contextual alignment improves multilingual BERT with small amounts of parallel data and corrects many systematic deficiencies.Contextual word retrieval also provides insights into the pre-training procedure and new avenues for analysis.
A.1 OPTIMIZATION HYPERPARAMETERS
The experiments use specified Adam optimization settings, one alignment epoch, three XNLI epochs, and linear warmup for the first 10% of training data.
- OPTIMIZATION HYPERPARAMETERS: Both alignment and XNLI optimization use a 5 × 10^-5 learning rate with Adam β = (0.9, 0.98) and ϵ = 10^-9.Linear learning-rate warmup covers the first 10% of training data.
- OPTIMIZATION HYPERPARAMETERS: Alignment trains for one epoch with two sentence pairs per language per batch, whereas XNLI trains for three epochs with batches of 32 examples.XNLI additionally applies 10% dropout to the BERT embeddings.
A.2 ALIGNMENT OF CHINESE, ARABIC, AND URDU
The alignment method was evaluated across eight languages using one shared BERT model and diverse parallel corpora, including distant-language settings.
- The method aligns one BERT model for eight languages before XNLI fine-tuning.The setup uses 20K parallel sentences per language.
- The experiments use MultiUN for Arabic and Chinese, Tanzil for Urdu, and Europarl for the remaining five languages.
- The results confirm that alignment works across varied languages and parallel corpora, including Quran-derived Tanzil data.
A.3 EXAMPLES OF CONTEXT-AWARE RETRIEVAL
Context-aware retrieval examples illustrate how aligned BERT handles multiple meanings of English words in parallel English-German sentences. The examples cover distinct senses of “support,” “close,” “like,” and “order” across varied contexts.
- Context-aware retrieval: Aligned BERT is used to qualitatively demonstrate disambiguation between different occurrences of a word.
- Context-aware retrieval: The English-German test set contains two meanings of “like,” with the second and third examples placing the senses in the same sentence.
- Multiple meanings of “order”: The examples also distinguish “order” in the phrases “in order to reach” and “in order to achieve.”
- Multiple meanings of “support”: The examples include multiple meanings of “support,” including providing financial assistance, aiding work, and endorsing a resolution or subsidy scheme.
- Multiple meanings of “close”: The retrieval examples include “close” in spatial, institutional, temporal, and relational contexts.