Source-linked AI summary
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Shijie Wu, Mark Dredze
TL;DR
The paper asks whether multilingual BERT supports broad zero-shot cross-lingual transfer and how its representations and transfer behavior should be understood. It evaluates mBERT across five NLP tasks and examines fine-tuning, language-specific information, and subword overlap. Across many languages and tasks, mBERT performs strongly, with further gains from freezing bottom layers and evidence that shared subwords correlate with transfer performance.
Problem
The paper investigates whether mBERT learns a cross-lingual space that supports zero-shot transfer beyond evidence from natural language inference.
Method
The paper evaluates mBERT on five NLP tasks across 39 languages and studies fine-tuning schemes, layer-wise language information, and subword overlap.
Results
mBERT performs well in zero-shot transfer across five tasks and outperforms cross-lingual embeddings, while freezing bottom layers yields further performance gains.
Takeaways & Limitations
The findings support using mBERT or similarly pretrained multilingual models for multilingual NLP without explicit cross-lingual supervision.
Takeaways & Limitations
The paper notes that mBERT under-performs models using a small amount of supervision in some settings.
Abstract
from arXiv · showhide
Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.
1 Introduction
The paper asks whether multilingual BERT learns representations that support zero-shot transfer beyond the promising XNLI result. It evaluates mBERT across five NLP tasks and examines fine-tuning, layer-wise behavior, and subword overlap.
- mBERT is pretrained on concatenated Wikipedia data for 104 languages without explicit cross-lingual alignment.
- The paper evaluates mBERT zero-shot transfer on natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing.
- mBERT achieves competitive or state-of-the-art performance with the recommended scheme of fine-tuning all parameters.
- Freezing parameters can outperform fine-tuning all mBERT parameters.
- The paper probes language-specific information across mBERT layers and measures how subword overlap influences transfer.
2 Background
The background situates zero-shot cross-lingual transfer among cross-lingual representations, contextual embeddings, and embedding-alignment methods. It also distinguishes this work from concurrent studies through broader task coverage and additional probing.
- Zero-shot cross-lingual transfer trains and selects a model in a source language, then transfers it directly to a target language.
- Cross-lingual word-embedding methods reduce supervision by jointly training representations or mapping monolingual embeddings with identical strings as a heuristic bilingual dictionary.
- Contextual embeddings from ELMo, GPT, and related models improve on standalone word embeddings through pretrained representations or parameter fine-tuning.
- Concurrent work incorporates bitext into BERT, while other studies align multilingual ELMo or report mixed benefits from multilingual pretraining.
- Compared with prior mBERT work, this paper covers more NLP tasks, including dependency parsing, and evaluates against previous state-of-the-art transfer methods.
3 Multilingual BERT
Multilingual BERT retains BERT’s Transformer architecture and self-supervised objectives while pretraining on Wikipedia data from 104 languages without explicit cross-lingual signals. Its inputs use WordPiece subwords and Transformer blocks with embeddings, normalization, feed-forward layers, and multi-head self-attention.
- BERT uses masked language modeling and next sentence classification rather than directional language modeling.
- BERT uses WordPiece to represent words as data-driven sequences of subwords.
- Fine-tuning adds task-specific classification or sequence-tagging layers while typically updating the pretrained Transformer parameters.
- mBERT follows BERT’s architecture and training procedure but uses Wikipedia data from 104 languages without explicit cross-lingual supervision.
- Sentence-pair inputs prepend [CLS] and append [SEP] around the sequences before embedding and Transformer processing.
- Each Transformer block combines multi-head self-attention with feed-forward processing, layer normalization, and dropout.
4 Tasks
The study evaluates zero-shot transfer from English to 38 languages across five NLP tasks, using task-specific layers and datasets spanning classification, tagging, and parsing.
- Task scope: Zero-shot transfer uses labeled English training data and evaluates the trained model on target languages across five NLP tasks.The tasks are document classification, natural language inference, named entity recognition, part-of-speech tagging, and dependency parsing.
- Document Classification: MLDoc provides balanced Reuters document classification for eight languages using four topic classes and classification accuracy.The model receives the first two document sentences when available, followed by a linear classification layer and softmax.
- Natural Language Inference: XNLI covers 15 languages with three-way inference over sentence pairs, evaluated by classification accuracy.The labels are entailment, neutral, and contradiction, and sentence pairs are fed directly into mBERT.
- Named Entity Recognition: NER combines CoNLL datasets and a Chinese dataset across five languages, predicting BIO entity labels with a word-level softmax and evaluating entity F1.Predictions for non-first subwords are masked when a word is split into multiple subwords.
- Tagging and Parsing: POS tagging uses Universal Dependencies treebanks covering 15 languages and evaluates predicted part-of-speech tags by accuracy.Dependency parsing uses Universal Dependencies treebanks covering 31 languages, with a graph-based parser and UAS and LAS evaluation.
5 Experiments
Experiments evaluate mBERT across five zero-shot cross-lingual tasks, comparing baselines and probing fine-tuning, layer-wise representations, and subword-sharing effects. mBERT is competitive across tasks, while selective fine-tuning and shared subwords reveal conditions affecting transfer.
- MLDoc: mBERT outperforms multilingual word embeddings on MLDoc and performs comparably with a multilingual sentence representation despite using no bitext.It performs better on distantly related Chinese and Russian but under-performs on closely related Indo-European languages.
- NER: mBERT outperforms a strong NER baseline by 6.9 absolute F1 points on average and by 11.8 points in German.The result uses a simple one-layer 0th-order CRF prediction function.
- Dependency parsing: 7.3-point UAS and 0.4-point LAS average gains over the baseline occur in dependency parsing without gold POS tags.With gold POS tags, the average improvements are 1.6 UAS points and 4.7 LAS points.
- Cross-task results: mBERT achieves strong, sometimes state-of-the-art, zero-shot transfer across all five tasks and outperforms cross-lingual embeddings in four tasks.The evaluated tasks are natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing.
- Fine-tuning: Freezing bottom mBERT layers generally improves performance, with the best depth typically n = 6 for sentence tasks and n = 3 for word-level tasks.Larger improvements are observed in under-performing languages.
- Representation analysis: The feature-based LSTM approach under-performs fine-tuning, while all tested mBERT layers achieve around 96% language-identification accuracy.The layer-wise result suggests mBERT retains language-specific information alongside cross-lingual representations.
6 Discussion
mBERT performs strongly in zero-shot cross-lingual transfer across five tasks, while its effectiveness is associated with subword sharing and improved by freezing bottom layers. The paper also identifies boundaries and open questions around supervision and low-resource transfer.
- Cross-task performance: mBERT achieves strong zero-shot transfer across five tasks covering many languages, often reaching state-of-the-art performance.The setting uses no cross-lingual signal and spans natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing.
- Cross-task performance: mBERT outperforms cross-lingual embeddings despite those methods typically using more cross-lingual supervision.
- Fine-tuning and representations: Freezing mBERT’s bottom layers during fine-tuning yields further performance gains.
- Fine-tuning and representations: Language-specific information remains preserved across all mBERT layers.
- Subword sharing: Sharing subwords helps transfer, with a strong correlation between overlapping-subword percentage and transfer performance.Subwords appearing across multiple languages with high frequency are more likely to be selected.
- Scope and future work: mBERT under-performs models using a small amount of supervision, leaving the usefulness of cross-lingual transfer for low-resource tasks as an open question.