PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Cross-lingual Transferability of Monolingual Representations

Mikel Artetxe, Sebastian Ruder, Dani Yogatama

arXiv:1910.11856v3cs.CLcs.AIcs.LG

TL;DR

The paper asks whether shared vocabularies and joint multilingual training are necessary for zero-shot cross-lingual generalization. It transfers a monolingual model to new languages through lexical parameters and finds competitive performance, suggesting that monolingual models learn abstractions that generalize across languages.

  • Problem

    The paper examines whether zero-shot cross-lingual generalization requires shared vocabulary items and joint multilingual training.

  • Method

    The paper transfers a monolingual model to new languages at the lexical level by learning new language-specific embeddings while retaining the monolingually trained model.

  • Results

    MONOTRANS is competitive even in the most challenging scenarios, and monolingual models perform comparably on standard zero-shot cross-lingual transfer benchmarks.

  • Takeaways & Limitations

    Joint multilingual pre-training is not essential for cross-lingual generalization, and monolingual models learn some abstractions that generalize across languages.

  • Takeaways & Limitations

    The transfer method has stability issues when learning language-specific position embeddings for Greek, Thai, and Swahili, requiring three runs and selection by XNLI development performance.

Abstract

from arXiv · show

State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.

1 Introduction

The paper tests whether zero-shot cross-lingual transfer requires shared vocabulary and joint multilingual training by transferring a monolingual transformer through new lexical parameters. This approach is competitive with multilingual pre-training on standard benchmarks and supports a new cross-lingual question-answering dataset.

  • Motivation: Existing explanations attribute multilingual BERT’s cross-lingual generalization to shared vocabulary items, joint multilingual training, and deep cross-lingual representations.Shared subwords are hypothesized to act as anchors that spread alignment across languages.
  • Approach: The proposed alternative transfers a monolingual transformer to a new language by learning a new embedding matrix while freezing the other layers.It therefore tests the necessity of shared vocabulary and joint multilingual training.
  • Results: The lexical transfer approach is competitive with joint multilingual pre-training across XNLI, MLDoc, and PAWS-X.The comparison covers standard zero-shot cross-lingual transfer benchmarks.
  • Contribution: The paper contributes XQuAD, a cross-lingual question-answering dataset translated from SQuAD v1.1 into ten languages by professional translators.The dataset is intended as a more comprehensive cross-lingual benchmark.
  • Results: On XQuAD, the approach can be made competitive with mBERT by learning second-language-specific transformations through adapter modules.XQuAD contains 240 paragraphs and 1190 question-answer pairs translated into ten languages by professional translators.

2 Cross-lingual Transfer of Monolingual Representations

The method aligns new lexical parameters to a monolingually trained transformer and transfers it through a four-step procedure. Extensions address embedding mismatch, language-specific word order, and deeper second-language adaptation.

  • Method overview: The method transfers a model from language L1, with task supervision and a monolingual corpus, to language L2, with only a monolingual corpus.It aligns new lexical parameters to a monolingually trained deep model.
  • Four-step procedure: The four steps are monolingual pre-training in L1, learning L2 token embeddings with the same objectives, English task fine-tuning, and swapping embeddings for zero-shot transfer.The procedure uses MLM and NSP for pre-training and L2 embedding learning.
  • Vocabulary: Separate subword vocabularies are trained for each language, while special symbols such as [CLS], [SEP], and [MASK] remain shared.The model therefore has no shared language subwords, but retains shared special symbols for transfer.
  • Extensions: Language-specific position embeddings relax the restriction of reusing L1 word-order representations for L2.A separate L2 position-embedding set is learned alongside L2 token embeddings.
  • Extensions: Gaussian noise is added to word, position, and segment embeddings during fine-tuning to improve robustness to the L1-to-L2 embedding mismatch.The transformer body is trained with L1 embeddings but used with L2 embeddings at test time.
  • Extensions: Residual adapter modules allow second-language-specific deep representations while preserving alignment with L1.During fine-tuning, the original model parameters are frozen and adapter parameters are learned.

3 Experiments

The experiments compare jointly trained multilingual models, cross-lingual embedding mappings, and monolingual transfer across zero-shot classification tasks. Results show that larger or disjoint per-language vocabularies help, while monolingual transfer remains competitive despite multilinguality only at the embedding layer.

  • Experimental setting: The experiments compare JOINTMULTI, JOINTPAIR, CLWE, and MONOTRANS under English-supervised zero-shot evaluation on XNLI, MLDoc, and PAWS-X.Each model is trained or adapted for transfer to other languages without labeled target-language data.
  • XNLI: JOINTMULTI models with larger vocabularies perform better, while more languages do not improve performance.The experiments include JOINTMULTI vocabularies ranging from 32k to 200k subwords.
  • XNLI: JOINTPAIR models with disjoint vocabularies perform better, showing that shared subword vocabularies are not necessary for joint multilingual pre-training.JOINTPAIR controls the effect of multilingual joint training while varying vocabulary sharing.
  • MLDoc and PAWS-X: CLWE performs poorly outside English, although larger dimensionalities and weak supervision improve it; its performance remains below other models.In MLDoc, however, CLWE obtains the best overall results, while all four model families perform generally similarly.
  • XNLI: MONOTRANS is 3.3 points worse on average than equivalent JOINTPAIR models, but language-specific position embeddings and noised fine-tuning reduce the gap to 1.1 points.Adapters mostly improve performance, except for low-resource Urdu, Swahili, Thai, and Greek.
  • MLDoc and PAWS-X: MONOTRANS remains competitive with the best joint variant on PAWS-X, differing by only 0.6 points when language-specific position embeddings are learned.The classification results collectively show competitiveness with JOINTMULTI and JOINTPAIR despite an English-only transformer body.

4 XQuAD: Cross-lingual Question Answering Dataset

XQuAD tests cross-lingual transfer on extractive question answering using professionally translated SQuAD material. Its results show that vocabulary design remains important, while adapters can make monolingual transfer comparable to the best joint system.

  • Motivation: Extractive question answering requires identifying answer spans in longer contexts, providing a probe of structural transfer across languages.The authors characterize question answering as less susceptible to annotation artifacts than other popular tasks.
  • Dataset construction: XQuAD contains 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional human translators.The dataset uses contexts and questions from the development set and marks frequent answer spans.
  • Results: On XQuAD, larger vocabularies strongly affect JOINTMULTI, and JOINTPAIR models with disjoint vocabularies perform best.The evaluation reports F1 scores, with exact-match scores provided separately.
  • Results: Language-specific position embeddings help in most cases but completely fail for Turkish and Hindi.The same pretrained models still obtain competitive XNLI results for these languages.
  • Results: Adapters significantly improve transferred monolingual models by enabling higher-level abstractions in the new language, making MONOTRANS comparable to the best joint system.The gap between MONOTRANS and joint models is larger on XQuAD than in earlier classification experiments.

5 Discussion

The discussion finds that vocabulary allocation and monolingual transfer, rather than shared vocabulary or extensive joint training, explain much of the observed cross-lingual performance. Semantic abstractions transfer more readily than syntactic ones, while CLWE methods remain weak on challenging tasks.

  • Sharing subwords across languages is not necessary for mBERT, and scaling joint training to many languages yields no clear improvement.
  • Larger effective vocabulary sizes consistently improve multilingual models, with disjoint-vocabulary JOINTPAIR systems generally performing best.Joint vocabularies force languages to compete for shared-vocabulary allocations, whereas disjoint JOINTPAIR vocabularies allocate 32k subwords per language.
  • MONOTRANS remains competitive in challenging scenarios, indicating that joint multilingual pre-training is not essential for cross-lingual generalization.The results suggest monolingual models learn linguistic abstractions that generalize across languages.
  • Transferred non-English models are comparable to English models on WiC and perform well on SCWS, but show larger gaps on syntactic evaluation.The probing results indicate that semantic abstractions transfer more readily than syntactic abstractions.
  • CLWE models are competitive only on MLDoc and perform poorly on the more challenging XNLI and XQuAD tasks.The authors attribute this pattern to MLDoc’s superficial task nature and small training set, while noting that the model families otherwise obtain generally similar results.
  • The transfer setting supports lifelong learning by adding languages to an existing model without access to the original training data.

6 Related Work

Related work spans lexical cross-lingual embedding mappings and deep transformer-based multilingual representations. Concurrent studies similarly question the necessity of shared vocabularies and explore monolingual-to-multilingual transfer and cross-lingual question answering datasets.

  • Cross-lingual word-embedding methods learn monolingual embeddings separately and map them into a shared space through linear transformations.Unsupervised methods initialize the mapping using heuristics or adversarial training.
  • Deep multilingual methods use transformers, with mBERT extensions adding scale, parallel data, auxiliary pretraining tasks, or translation-similarity objectives.
  • Tran (2020) reports similar transfer results with a more complex monolingual-BERT approach, but finds random-initialized post-hoc embedding learning ineffective.
  • Concurrent work also finds shared vocabulary unimportant, while Lewis et al. propose a related question-answering dataset covering fewer languages without full parallel coverage.

7 Conclusions

The paper concludes that lexical transfer from monolingual models can match multilingual models on standard cross-lingual benchmarks, challenging assumptions about shared vocabularies and joint pre-training. It also finds transferable semantic abstractions and releases XQuAD as a broader benchmark.

  • Monolingual models transferred at the lexical level perform comparably to multilingual models on standard zero-shot cross-lingual benchmarks.
  • The results indicate that neither a shared vocabulary nor joint pre-training is necessary for multilingual-model performance.
  • Monolingual models learn semantic abstractions that generalize to other languages in probing experiments.
  • XQuAD is released as a more comprehensive benchmark for evaluating cross-lingual models.

A Training details

The training procedure pre-trains a single English model and then performs a separate transfer phase for each additional language. Fine-tuning uses fixed optimization settings, while adapter hyperparameters follow prior work.

  • The proposed approach pre-trains a single English model for 250k steps and transfers it to each other language for another 250k steps.
  • Fine-tuning uses Adam with a 2e-5 learning rate, batch size 32, and two epochs.
  • Adapter hyperparameters follow Houlsby et al. (2019), while noised fine-tuning uses Gaussian noise with standard deviation 0.075 and mean 0.

B XQuAD dataset details

XQuAD is a multilingual question-answering dataset built from translated SQuAD v1.1 development examples, with professionally translated contexts and questions across ten additional languages.

  • XQuAD contains 240 context paragraphs and 1190 question-answer pairs translated into ten languages.The languages are Spanish, German, Greek, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, and Hindi.
  • The dataset samples five context paragraphs from each of 48 SQuAD 1.1 development documents to promote diversity.
  • Professional human translators translated both the context paragraphs and their corresponding questions through the Gengo service.
  • Answer spans are marked with placeholder symbols, which translators were instructed to preserve in the relevant positions.An online validator checked that translated outputs retained the required format.

C Additional results

The paper reports cross-lingual classification and question-answering results for cross-lingual embedding mappings and joint multilingual training variants.

  • Table 7 reports accuracy results on the MLDoc and PAWS-X cross-lingual classification datasets.
  • Table 8 reports exact match results on the XQuAD question-answering dataset.
  • Table 9 reports F1 results for the cross-lingual embedding mappings and joint multilingual training variants.

D Probing experiments

The probing experiments evaluate transferred monolingual representations through controlled alignment settings, semantic and syntactic tasks, and additional dataset statistics and result tables.

  • Experimental setup: Monolingual models are trained in each XNLI target language and aligned to English using 3M sentences for both pretraining and alignment.
  • Semantic probing: Semantic probing uses WiC for contextual meaning classification and SCWS for contextual word-pair similarity.WiC uses a linear classifier, while SCWS compares contextual target-word representations with cosine similarity.
  • Syntactic probing: Syntactic probing selects the more probable sentence from minimally different English pairs after masking the differing word.Pairs differing in more than one subword token are discarded.
  • Experimental setup: Thai, Hindi, Swahili, and Urdu are excluded from the controlled probing setup because their corpora contain fewer than 3M sentences.
  • Reported analyses: The reported materials include XQuAD token statistics, parallel-instance examples, classification accuracy tables, question-answering results, and syntactic coverage and accuracy tables.
Loading 1910.11856v3…