PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Lingual Ability of Multilingual BERT: An Empirical Study

Karthikeyan K, Zihan Wang, Stephen Mayhew, Dan Roth

arXiv:1912.07840v2cs.CLcs.AIcs.LG

TL;DR

M-BERT transfers across languages despite lacking aligned data and an explicit cross-lingual objective, but the sources of this ability are not fully understood. The paper systematically studies linguistic, architectural, and learning factors across Spanish, Hindi, and Russian using textual entailment and named entity recognition. It finds that lexical overlap is not important, while structural similarity and network depth are central to cross-lingual ability.

  • Problem

    The paper asks which components explain M-BERT’s cross-lingual ability despite training without aligned data or an explicit cross-lingual objective.

  • Method

    The authors systematically analyze linguistic similarity, architecture, and input or learning objectives in B-BERT across textual entailment and named entity recognition.

  • Results

    Word-piece overlap is not significant, while structural similarity and network depth are crucial for cross-lingual ability; multi-head attention is not significant.

  • Takeaways & Limitations

    Cross-lingual ability can emerge without word-piece overlap, and model depth is an integral part of the observed transfer.

  • Takeaways & Limitations

    The study does not further dissect structural similarity and leaves interactions among more languages for future work.

Abstract

from arXiv · show

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: http://cogcomp.org/page/publication_view/900 .

1 INTRODUCTION

The paper investigates why M-BERT transfers across languages without aligned data or an explicit cross-lingual objective. Experiments isolate linguistic, architectural, and learning factors, finding that word-piece overlap matters little while depth is crucial.

  • Motivation: M-BERT is trained on raw Wikipedia from 104 languages without cross-lingual supervision or aligned data, yet generalizes across languages on downstream tasks.The study examines this unexpected cross-lingual ability using bilingual B-BERT models.
  • Study design: The study analyzes linguistic similarity, network architecture, and input or learning objectives across cross-lingual NER and textual entailment.The language pairs include English–Spanish, English–Russian, and English–Hindi.
  • Key findings: Word-piece overlap contributes little to cross-lingual performance, whereas network depth and total parameters are crucial and multi-head attention is not significant.The authors test this with a Fake-English language sharing no word-pieces with target languages.
  • Key findings: The next sentence prediction objective hurts performance, language identifiers do not affect cross-lingual performance, and character- or word-level tokenization performs worse than word-piece tokenization.

2 BACKGROUND

BERT is a Transformer-based pretrained language representation model that learns from masked language modeling and next sentence prediction. M-BERT applies this training setup across 104 languages without aligned data or a cross-lingual objective.

  • BERT: BERT predicts masked tokens and whether paired sentences are contiguous, combining masked language modeling with next sentence prediction.Inputs are sentence pairs, with randomly masked tokens and negative sentence pairings.
  • BERT: After pretraining, BERT representations are adapted to downstream tasks by adding a task-specific layer and fine-tuning the parameters.
  • Multilingual BERT: M-BERT uses the same pretraining procedure on Wikipedia text from the top 104 languages, without cross-lingual objectives or parallel corpora.Wikipedia data are resampled to account for differences in language size.

3 WHY MULTILINGUAL BERT WORKS

The paper analyzes B-BERT through linguistic properties, network architecture, and input or learning objectives to explain multilingual BERT’s cross-lingual ability.

  • The analysis covers linguistic similarities, network architecture, and input and learning objectives.

3.1 DATASETS AND EXPERIMENTAL SETUP

Experiments evaluate cross-lingual textual entailment and named entity recognition using datasets and settings designed to compare English with Hindi, Spanish, and Russian.

  • Tasks: The study evaluates cross-lingual textual entailment at the sentence level and named entity recognition at the token level.XNLI measures textual entailment, while LORELEI provides multilingual NER data.
  • XNLI: XNLI trains on English premise–hypothesis pairs and tests translated pairs in target languages using entailment, contradiction, and neutral labels.The same premises and hypotheses are used across languages for comparison.
  • NER: The NER data contain native-speaker annotations for English, Hindi, Spanish, and Russian, with English split into 80%, 10%, and 10% training, development, and test sets.The full Hindi, Spanish, and Russian datasets are used for testing.
  • Evaluation: XNLI performance is reported as accuracy, while NER uses F1 averaged over five runs with standard deviation.

3.2 LINGUISTIC PROPERTIES

B-BERT remains cross-lingual without shared word-pieces, while structural similarity—especially word ordering—supports transfer more strongly than lexical overlap or unigram frequency alone.

  • Word-piece overlap: The Fake-English intervention shifts each character’s Unicode by a large constant to eliminate character overlap while preserving other language properties.BERT cannot directly learn from the Unicode shift because word-pieces are mapped through a vocabulary embedding step before the Transformer.
  • Word-piece overlap: B-BERT remains cross-lingual even when source and target languages have no word-piece overlap.The measured performance contribution of word-piece overlap is very small, contradicting prior hypotheses emphasizing shared word-pieces.
  • Word-ordering similarity: Reducing word-order similarity significantly lowers cross-lingual performance, although performance remains significantly better than random.This residual performance indicates that additional components of structural similarity contribute to transfer.
  • Word-frequency similarity: Unigram frequencies alone do not provide enough information for cross-lingual learning.The frequency-only corpus replaces sentences with random words sampled from the original unigram distribution.
  • Structural similarity: Fake-English transfers to Spanish, Hindi, and Russian almost as well as English despite sharing no vocabulary.On XNLI, transfer from Fake-English to Spanish is better than transfer to Hindi or Russian, attributed to closer structural similarity.
  • Structural similarity: The study defines language structure as script-invariant properties including morphology, word ordering, and word frequency.The authors conclude that structural similarity is crucial for cross-lingual transfer but do not further dissect its components.

3.3 MODEL ARCHITECTURE

The study tests how B-BERT’s architecture affects monolingual and cross-lingual performance, focusing on depth, attention heads, and parameter count. Depth is consistently important, while attention-head count has little effect and parameter count is less significant except below a threshold.

  • 3.3 MODEL ARCHITECTURE: The architecture study varies depth, attention heads, and total parameters to identify components crucial for B-BERT’s cross-lingual ability.Cross-lingual experiments use Fake-English as the source and Russian as the target on XNLI, with smaller performance differences indicating better transfer.
  • 3.3.1 DEPTH: Deeper B-BERT models perform better on both English and Russian, with stronger English performance correlating with improved cross-lingual ability.Depth is varied while attention heads and total parameters remain approximately fixed.
  • 3.3.2 MULTI-HEAD ATTENTION: The number of attention heads has no significant effect on cross-lingual ability, and a single-head B-BERT can already produce satisfactory results.Depth and total parameter count are held fixed while attention-head count changes.
  • 3.3.3 TOTAL NUMBER OF PARAMETERS: Total parameter count is less significant than depth for cross-lingual performance, although it appears important below a threshold.The study changes hidden and intermediate-unit sizes while fixing depth and attention-head count.
  • 3.3 MODEL ARCHITECTURE: With sufficient depth, four-language BERT achieves comparable XNLI performance using only 15% of the parameters and 1 or 3 attention heads.This multilingual experiment supports the architectural observations derived from bilingual BERT.

3.4 INPUT AND LEARNING OBJECTIVE

The study examines how B-BERT’s input representation and learning objectives affect cross-lingual performance. Next Sentence Prediction hurts performance, language identity markers do not affect cross-lingual performance, and word-based tokenization outperforms character tokenization.

  • 3.4 INPUT AND LEARNING OBJECTIVE: The study evaluates Next Sentence Prediction, language identity markers, and character-, word-piece-, and word-level tokenization as factors in B-BERT’s cross-lingual ability.BERT is trained with Masked Language Modeling and Next Sentence Prediction, while input representations are varied across tokenization levels.
  • 3.4.1 NEXT SENTENCE PREDICTION (NSP): Next Sentence Prediction hurts cross-lingual performance more than monolingual performance.The comparison uses B-BERT trained with and without the NSP objective on XNLI and NER.
  • 3.4.2 LANGUAGE IDENTITY: Adding language identity markers does not affect B-BERT’s cross-lingual performance.Different end-of-string tokens serve as language-specific identity markers in the input.
  • 3.4.3 INPUT REPRESENTATION: Word-piece and word tokenization yield similar cross-lingual performance, and both outperform character tokenization.The comparison evaluates the performance difference between source and target languages.

4 DISCUSSION AND FUTURE WORK

The study systematically examines B-BERT’s cross-lingual ability and identifies structural similarity and network depth as crucial, while word-piece overlap and multi-head attention are not significant. It also highlights unresolved questions about multilingual interactions, structural similarity’s definition, and cross-language entailment.

  • Study scope: The study systematically analyzes B-BERT across linguistic properties, network architecture, and input and learning objectives.It uses experiments designed to isolate cross-lingual transfer between source and target languages.
  • Main findings: Structural similarity and network depth are crucial for B-BERT’s cross-lingual ability, whereas word-piece overlap and multi-head attention are not significant.Fake-English helps examine word-piece overlap while maintaining other source-language properties.
  • Future work: B-BERT was studied instead of M-BERT to control interference among languages, leaving interactions among multiple languages for future work.A specific open question is whether adding languages related to the target improves M-BERT’s cross-lingual ability.
  • Future work: Structural Similarity is significant but has an obscure definition, motivating finer experiments on its individual components.The authors identify improving its definition as a future research direction.
  • Open questions: Entailment performance drops drastically when the premise and hypothesis are in different languages, possibly because BERT matches words or phrases across them.The explanation is presented as a possible account and is left for future investigation.
Loading 1912.07840v2…