Source-linked AI summary
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Mikel Artetxe, Holger Schwenk
TL;DR
The paper addresses the limited cross-lingual sharing and transfer of language-specific sentence representations. It trains a single BiLSTM encoder with shared subword vocabulary and an auxiliary decoder on parallel corpora, obtaining strong transfer results across tasks and languages, including low-resource settings.
Problem
Language-specific representations cannot leverage information across languages, while multilingual benchmarks and applications require broader language and task coverage.
Method
The system trains a single language-agnostic BiLSTM encoder with a shared BPE vocabulary and auxiliary decoder on publicly available parallel corpora for 93 languages.
Results
The approach is effective on XNLI, MLDoc, and BUCC, and achieves competitive multilingual similarity search results, including for low-resource languages.
Takeaways & Limitations
Joint multilingual training supports strong zero-shot transfer and can generalize to unseen languages to some extent when related languages were included in training.
Takeaways & Limitations
Comparisons with another BERT variant are not fully comparable because that system uses foreign-language development data and task fine-tuning, whereas this approach is fully zero-shot.
Abstract
from arXiv · showhide
We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI dataset), cross-lingual document classification (MLDoc dataset) and parallel corpus mining (BUCC dataset) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our implementation, the pre-trained encoder and the multilingual test set are available at https://github.com/facebookresearch/LASER
1 Introduction
The paper targets language- and task-general sentence embeddings that can share information across languages and support zero-shot transfer. It addresses the limitations of language-specific models by training one encoder across 93 languages for multiple applications.
- Deep-learning NLP methods are data hungry, motivating representations learned from unlabeled data for downstream systems.
- Language-specific sentence models cannot share information across languages, limiting their potential for low-resource languages.
- Universal language-agnostic sentence embeddings should generalize across both input languages and NLP tasks.
- A single encoder is trained so semantically similar sentences in different languages are close in the embedding space.
- The approach learns general-purpose sentence representations for 93 languages and evaluates them on XNLI, MLDoc, BUCC, and multilingual similarity search.
2 Related work
Prior work developed sentence and multilingual representations mainly for individual languages, small language sets, or specific applications. Sequence-to-sequence models provide a related multilingual approach by using an encoder's fixed-length representation to generate target sentences.
- Sentence embeddings are commonly produced by recurrent encoders trained unsupervised on large unlabeled corpora.
- Multilingual research has largely focused on cross-lingual word embeddings learned jointly or mapped into a shared space.
- Sequence-to-sequence encoder-decoder systems train on parallel corpora, then retain the encoder to embed sentences in the training languages.
- Existing multilingual representation work generally covers few languages, language pairs, word embeddings, or specific applications.
- Contextualized variable-length representations are another line of work, using recurrent or self-attentional encoders trained with language modeling.
3 Proposed method
The proposed system uses a shared multilingual BiLSTM encoder-decoder trained on parallel corpora, with design choices that support scaling to 93 languages. Joint encoding, separate language alignments, and shared subword units reduce language-specific modeling requirements.
- A language-agnostic BiLSTM encoder with an auxiliary decoder learns sentence embeddings from parallel corpora.
- Max-pooling over BiLSTM outputs forms sentence embeddings that initialize the decoder and enter it at every time step.
- A shared encoder and decoder use a joint 50k-operation BPE vocabulary, while language IDs specify the decoder's generation language.
- The encoder uses stacked BiLSTM layers of 512 dimensions, producing 1024-dimensional sentence representations after concatenating both directions.
- Separate alignments for each language combination avoid requiring an N-way parallel corpus and avoid quadratic training cost as languages increase.
- Training alternates over language combinations and minimizes cross-entropy loss on the parallel training corpus.
- Training uses bitexts aligned with English and Spanish, while additional languages may require only one alignment once the joint embedding is well conditioned.
- The dataset covers 93 input languages and 223 million parallel sentences collected from publicly available corpora.
4 Experimental evaluation
The evaluation measures multilingual sentence embeddings across zero-shot transfer, bitext mining, and similarity search, while extending coverage beyond existing benchmarks. Results are strong across languages and tasks, including low-resource and some unseen languages.
- XNLI: The method achieves the best XNLI zero-shot transfer results for all languages except Spanish, with strong and homogeneous transfer across languages.For 11 languages, zero-shot performance is at most 5% lower than English, including Arabic, Chinese, and Vietnamese.
- MLDoc: The system obtains the best published MLDoc results for 5 of 7 transfer languages.The authors attribute weaker Japanese performance to domain and sentence-length mismatch between MLDoc and the Japanese parallel corpus.
- Evaluation design: The evaluation combines XNLI, MLDoc, BUCC, and a new 112-language similarity-search test set because existing benchmarks cover only a subset of 93 languages.The new test set uses up to 1,000 English-aligned sentence pairs per language and evaluates nearest-neighbor error rates.
- Tatoeba: Similarity search yields error rates below 5% for 37 languages, below 10% for 48, and below 20% for 55, while 15 exceed 50%.The authors associate competitive low-resource results with joint training and report some generalization to 29 unseen languages when related languages were included in training.
5 Ablation experiments
The ablations examine encoder depth, multitask NLI training, and the number of training languages. Deeper encoders and broader multilingual training generally help, whereas adding an NLI objective improves English NLI but harms cross-lingual transfer.
- Evaluation setup: The ablation section reports average results across languages and additionally reports English accuracy for XNLI.These evaluation summaries cover the paper’s tasks, with English reported separately for XNLI.
- Encoder depth: All evaluated tasks benefit from deeper BiLSTM encoders, especially XNLI and Tatoeba.Models with 1, 3, or 5 layers were compared; deeper models were preferred, but deeper than five layers did not converge well.
- Multitask training: The NLI objective improves English NLI performance but worsens XNLI and Tatoeba transfer, with negligible BUCC effects.Different objective weightings were tested in the multitask setup.
- Number of training languages: Training on 93 languages equals or outperforms training on 18 evaluation languages for every task except MLDoc.The comparison replaces Tatoeba with the WMT 2014 test set for multilingual similarity error-rate evaluation.
6 Conclusions
The paper presents general-purpose multilingual fixed-length sentence embeddings for 93 languages, trained with one language-agnostic encoder and used across several tasks without fine-tuning. It reports effective cross-lingual performance, including competitive results for low-resource languages, while identifying future architectural and data-use directions.
- Contributions: The architecture learns fixed-length multilingual sentence embeddings for 93 languages with one language-agnostic BiLSTM encoder.The encoder is trained on publicly available parallel corpora and applied to downstream tasks without fine-tuning.
- Contributions: Experiments on XNLI, MLDoc, BUCC, and a new 112-language similarity-search test set support the approach’s effectiveness.The paper reports competitive multilingual similarity-search results even for low-resource languages.
- Future work: Future work includes self-attention encoders, monolingual-data strategies, and language-agnostic preprocessing such as SentencePiece.The proposed directions include pre-trained word embeddings, backtranslation, and other unsupervised machine-translation ideas.
- Resources: The implementation, pre-trained encoder, and multilingual test set are freely available through the LASER repository.The cited repository is provided as the public release location.
A Training data
Training combines several publicly available parallel corpora covering different languages, domains, and corpus sizes. The authors retain 93 practical languages, observe domain effects, and do not oversample low-resource languages.
- Corpora: The training data combines Europarl, United Nations, OpenSubtitles2018, Global Voices, Tanzil, and Tatoeba parallel corpora.These resources span parliamentary, institutional, subtitle, news, religious, and community-translated text.
- Corpora: OpenSubtitles2018 covers 57 languages, with corpus sizes ranging from thousands to more than 50 million sentences.The authors retain at most 2 million entries for each language pair.
- Evaluation data: Tatoeba supplies a separate test set of up to 1,000 English-aligned sentences, while remaining entries for larger languages support training.The collection contains English sentences and translations into more than 300 languages.
- Language selection: The authors retain 93 languages after discarding several constructed languages with little practical use.The discarded languages include Klingon, Kotava, Lojban, Toki Pona, and Volapük.
- Data effects: Training-data domain affects task performance, with BUCC and MLDoc tending to favor long, formal sentences.The authors did not counter this effect by oversampling low-resource languages.
B XNLI results for all language combinations
XNLI language-pair results show strong performance when premises and hypotheses use different languages. The combined accuracy appears mainly bounded by the weaker language’s standalone accuracy rather than by language distance alone.
- Cross-lingual combinations: Cross-language XNLI combinations produce strong results, without evidence that distant language pairs perform considerably worse.The table reports cases where premises and hypotheses are in different languages.
- Cross-lingual combinations: Greek-Russian performs similarly to Bulgarian-Russian despite the different language pairing.The passage uses this comparison to illustrate that combined performance is not simply determined by language distance.
- Cross-lingual combinations: French-Chinese is only 1.5 points worse than French-Spanish, despite the former pairing more distant languages.The passage compares the two XNLI combinations directly.
- Performance pattern: Combined performance seems mostly bounded by the accuracy of the language that performs worst when used alone.This is the paper’s reported interpretation of the cross-language results.
C Tatoeba: dataset
The Tatoeba13 corpus provides filtered English sentences and translations for constructing aligned multilingual similarity-search test sets. Coverage varies with the number of parallel sentences retained.
- Dataset source: Tatoeba13 is an open collection of English sentences with high-quality translations into more than 300 languages.The dataset snapshot was downloaded on November 19, 2018.
- Preprocessing: Filtering removed sentences containing “@” or “http”, sentences shorter than three words, and duplicate sentences.These steps targeted non-language-specific content, low-information examples, and repeated entries.
- Test-set coverage: Test sets contain up to 1,000 aligned sentences with English for 72 languages, 500 for 86 languages, or 100 for 112 languages.Increasing language coverage required reducing the number of aligned sentences per language.
- Comparability: English sentences differ across languages, so error rates are not directly comparable across languages.The test sets are aligned with English, but they generally do not use the same English sentence subset.
D Tatoeba: result analysis
Tatoeba similarity search performs strongly across many language families and scripts, including several low-resource languages, but performance remains uneven for isolated low-resource languages and some language groups.
- Overall results: 48 languages have similarity error rates below 10%, while 55 have error rates below 20%.The languages below 20% span 20 families and 12 scripts, including six languages with less than 400k bitext sentences.
- Low-resource languages: Low-resource Indo-Aryan and Berber languages achieve low or remarkable similarity error rates despite limited language-specific bitext.The Berber languages ber and kab have fewer than 100k training sentences.
- Joint training: Joint training benefits languages for which language-specific corpora alone would be unlikely to yield good sentence embeddings.The paper highlights Berber as an example of a language group with many speakers but limited written resources.
- Error analysis: 15 languages have similarity error rates above 50%.Four are low-resource languages with their own scripts and no close family members represented in the training data.
- Error analysis: Khmer performs much better than random with only 625 training examples, whereas several Turkic and Celtic languages have high error rates.The authors plan to investigate the causes and possible solutions for these differences.
E Tatoeba: results for unseen languages
The encoder also performs reasonably well on 29 languages absent from training. These languages share varying similarities with covered major languages while retaining distinct grammar or vocabulary.
- Unseen languages: 29 languages without any training data receive reasonably good performance from the encoder.Many are recognized minority languages spoken in specific regions.
- Unseen languages: The unseen languages share similarities with covered major languages but also differ through their own grammar or specific vocabulary.These similarities enable performance even though the encoder did not see the languages during training.