Source-linked AI summary
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Nils Reimers, Iryna Gurevych
TL;DR
The paper addresses the scarcity of multilingual sentence embedding models and training data for languages beyond English. It uses multilingual knowledge distillation to train a student on translations that mimic a monolingual teacher, producing aligned spaces and transferring source-language properties across languages. The method works across language combinations and supports extending models to multiple languages, while BUCC-based results require careful interpretation because its non-parallel portion may contain parallel sentences.
Problem
Most existing sentence embedding models are monolingual because suitable training data for other languages is scarce, while few multilingual and cross-lingual models exist.
Method
The method trains a student model on translated sentence pairs so both source and target sentences approximate the monolingual teacher’s source-language embedding.
Results
The resulting multilingual space aligns identical sentences across languages and transfers source-language vector-space properties to target languages.
Takeaways & Limitations
Models can be extended to multiple languages in the same training process, while desired source-language properties can first be developed and then transferred independently.
Takeaways & Limitations
BUCC results should be judged with care because its Wikipedia portion may contain parallel sentences, making the effect on reported scores unclear.
Abstract
from arXiv · showhide
We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training is lower. We demonstrate the effectiveness of our approach for 50+ languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.
1 Introduction
The paper introduces multilingual knowledge distillation to extend monolingual sentence embeddings to new languages using translated sentence pairs. The resulting space aligns translations while transferring the source model’s vector-space properties.
- Most existing sentence embedding models are monolingual because suitable training data for other languages is scarce.
- The method trains a student model to match the teacher’s source-language embeddings for both original and translated sentences using mean squared loss.
- The approach supports various language combinations and multilingual setups, with improvements of up to 40 accuracy points for low-resource languages compared to LASER.
- The student learns a multilingual space where identical sentences across languages are close and source-language vector-space properties transfer to other languages.
- Unlike LASER, which is less effective for non-exact sentence similarity, the proposed approach is designed to transfer the teacher space’s properties.
2 Training
Training uses parallel translated sentences to distill a teacher model’s representations into a student model. The student may share the teacher’s architecture or use a different one, with XLM-R used in the experiments.
- The teacher model maps sentences in one or more source languages to a dense vector space.
- Training requires parallel sentence pairs containing source-language sentences and their translations in target languages.
- The student is trained so its source and target representations approximate the teacher representation of the source sentence.
- The student can use the teacher’s structure and weights or a different network architecture.
- The experiments mainly use English SBERT as teacher and XLM-RoBERTa as student because the latter supports multilingual tokenization.
3 Training Data
The paper evaluates parallel corpora and bilingual dictionaries as training sources for multilingual knowledge distillation. It also balances datasets during training and examines data quantity, domain, and random-seed effects.
- OPUS provides parallel data for hundreds of language pairs, enabling evaluation across multiple training datasets.
- WikiMatrix pairs were restricted to scores above 1.05 because lower-scoring pairs were often poor quality.
- Because parallel data can be challenging to obtain for low-resource language pairs, the study also uses MUSE and Wikititles bilingual dictionaries.
- Training balances dataset sizes by sampling roughly equally from each dataset and repeating samples from smaller datasets.
- For English-German, dataset domain had minor importance and only little training data was necessary; large datasets showed roughly 0.3-point random-seed differences.
4 Experiments
The experiments evaluate multilingual knowledge distillation across semantic similarity, bitext retrieval, and similarity search, comparing multilingual encoders and distilled models. Results show strong performance for semantic similarity and low-resource alignment, while translation-retrieval benchmarks favor systems designed specifically for exact translations.
- Experimental Setup: The experiments cover multilingual and cross-lingual semantic textual similarity, bitext retrieval, and cross-lingual similarity search.STS scores sentence-pair similarity, while bitext retrieval identifies translated pairs from large monolingual corpora.
- Experimental Setup: The evaluation compares SBERT-derived distilled models with mBERT, XLM-R, LASER, mUSE, and LaBSE.The experiments include models trained only through multilingual pretraining, English NLI/STS fine-tuning, and multilingual knowledge distillation.
- Multilingual Semantic Textual Similarity: Multilingual knowledge distillation achieves state-of-the-art performance in both monolingual and cross-lingual STS, significantly outperforming LASER, mUSE, and LaBSE.SBERT-paraphrase performs best despite not being trained on an STS dataset, suggesting broad paraphrase training generalizes across topics.
- Cross-Lingual Alignment: Distillation significantly improves performance over mBERT and XLM-R trained only on English data, addressing their weak cross-language vector-space alignment.English NLI and STS fine-tuning improves some multilingual models but does not consistently align their spaces across languages.
- Bitext Retrieval: LASER and LaBSE outperform mUSE and the proposed approach on BUCC because the task rewards exact translation retrieval rather than broader semantic similarity.The proposed approach and mUSE assign high similarity to semantically similar pairs that are not exact translations.
- Evaluation Caveats: BUCC results require caution because Wikipedia sentences labeled non-parallel may contain valid translations, affecting measured scores.Among 60 analyzed high-similarity false positives, 57 were judged valid, high-quality translations.
5 Evaluation of Training Datasets
The evaluation examines how parallel-data size, domain, and language similarity affect bilingual XLM-R sentence-embedding performance. Results are favorable for English–German with limited data but less predictable for English–Arabic.
- Cross-language comparison: Bilingual models improved over the 10-language model by 2.2 points for EN-DE and 1.2 points for EN-AR.The paper relates this pattern to the curse of multilinguality, where adding languages can degrade performance under fixed model capacity.
- English–German: For EN-DE, training-data domain produced only minor differences, and little parallel training data was necessary.The evaluated domains included news, subtitles, parliamentary debates, and magazines.
- English–Arabic: 76.7 with 27,000 Tatoeba sentence pairs exceeded 66.1 with over 8 million UNPC sentences for EN-AR.These results show that more parallel data did not necessarily yield better performance for the dissimilar language pair.
- English–Arabic: EN-AR performance improved more slowly than EN-DE as reduced TED2020 training data increased.The reduced-data comparison is reported using the STS 2017 dataset.
6 Target Language Training
This section compares transferring an English sentence-embedding model to Korean with training directly on translated Korean data. Multilingual knowledge distillation slightly improves performance while retaining aligned English–Korean representations.
- Target Language Training: The study asks whether transferring an English model to Korean is better than training from scratch on suitable Korean-language data.The comparison uses the KorNLI and KorSTS datasets, translated from English and professionally post-edited for development and testing.
- Target Language Training: Multilingual knowledge distillation produced a slight improvement over direct training on translated Korean NLI and STS data.The distilled model uses XLM-R with SBERT-nli-stsb as the teacher.
- Target Language Training: Knowledge distillation yielded a model for both English and Korean with aligned vector spaces, whereas direct Korean training yielded a Korean-only model.The comparison concerns the practical scope and cross-language alignment of the resulting sentence-embedding models.
- Target Language Training: The results indicate no necessary performance drop when sentence-embedding training data are available only in English.This conclusion is stated for the Korean transfer experiment.
7 Language Bias
The section evaluates language bias in multilingual sentence pools, where language-dependent similarity can distort rankings across language combinations. Multilingual knowledge distillation shows only a minor, statistically insignificant bias in the reported comparison.
- Language Bias: A language-biased model maps same-language sentences closer because of language identity, potentially discriminating among language pairs in multilingual pools.The paper frames language bias as a problem for multilingual sentence-pool evaluation.
- Results: LaBSE embeddings separated English premises from Russian translations sharply in the first two principal components.The separation indicates that language significantly affects the resulting embedding vectors for LaBSE.
- Language Bias: Joining ten language combinations tests whether multilingual-pool performance remains near the average of the individual-language-set performances.A larger decrease in Spearman correlation indicates that some language combinations are ranked higher than others.
- Results: LaBSE showed a -1.29 difference and LASER a -0.92 difference between joined and individual-set performance, both statistically significant at p < 0.001.The negative differences indicate lower performance on multilingual pools than on monolingual pools.
- Results: mUSE and multilingual knowledge distillation showed minor, statistically insignificant language bias, with decreases of -0.19 and -0.11.The paper concludes that these approaches can be used on multilingual sentence pools without negative performance impact from language bias.
8 Related Work
Prior multilingual sentence-embedding methods include cross-lingual word-vector transformations, bag-of-words representations, encoder–decoder models, and multi-task dual encoders. The paper positions its approach against their data, training, and computational requirements.
- Related Work: Many sentence-embedding methods were trained only on English, while much multilingual-representation research focused on cross-lingual word embeddings.A common word-embedding approach learns a linear transformation between separately trained language spaces.
- Related Work: Bag-of-words representations built from cross-lingual word embeddings perform poorly in practical cross-lingual transfer settings.The paper cites prior work as evidence for this limitation.
- LASER: LASER uses an LSTM sequence-to-sequence encoder–decoder trained on parallel corpora and max-pools encoder outputs into fixed-size sentence representations.LASER was trained for 93 languages on 16 NVIDIA V100 GPUs for about five days.
- mUSE: mUSE uses a dual encoder trained with SNLI, over one billion question-answer pairs, and a translation-ranking task requiring hard negatives.The translation-ranking setup identifies the correct translation among alternatives in target languages.
9 Conclusion
The paper concludes that multilingual knowledge distillation aligns sentence-embedding spaces across languages while transferring source-language properties. Its stepwise design simplifies extension to multiple languages and reduces potential language bias, with tests spanning diverse language families.
- Multilingual knowledge distillation makes monolingual sentence embeddings multilingual with aligned vector spaces across languages.
- The approach transfers properties from the English source-language vector space to various target languages.
- A model can be extended to multiple languages within the same training process.
- Stepwise training separates creating a high-resource-language model with desired properties from extending it to further languages, simplifying training and minimizing potential language bias.
- The approach was extensively tested across languages from different language families.
- LASER and LaBSE retrieve exact translations well but assess non-exact sentence similarity less effectively and show language bias across language combinations.
A Tatoeba Similarity Search
The Tatoeba evaluation measures cross-language retrieval with cosine similarity, but comparisons are affected by differences in training-data exposure. Multilingual knowledge distillation performs strongly for languages with parallel training data, whereas performance is generally low without alignment data.
- Tatoeba evaluates bilingual retrieval by finding the most similar sentence with cosine similarity and averaging English-to-target and target-to-English accuracy.
- Multilingual knowledge distillation excludes Tatoeba training data, unlike LASER, whose train-test overlap may artificially increase reported scores.
- Accuracy scores are usually in the 90s for languages with parallel training data.
- For languages without parallel data, scores vary more and are mostly low because the model did not learn alignment for those languages.
- LaBSE and LASER had training data for most languages evaluated without parallel data for multilingual knowledge distillation.
- Figure 2 compares the first two principal components for English and Russian parallel sentences; LaBSE shows stronger language separation than mUSE and the proposed approach.