PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-agnostic BERT Sentence Embedding

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, Wei Wang

arXiv:2007.01852v2cs.CL

TL;DR

Cross-lingual sentence embeddings based on BERT were insufficiently explored, despite BERT’s effectiveness for monolingual sentence embeddings. The paper combines multilingual pretraining with dual-encoder translation ranking and related objectives, producing strong multilingual retrieval performance and a broadly released model.

  • Problem

    BERT-based cross-lingual sentence embeddings had not been systematically explored, while multilingual pretrained models did not directly produce good sentence embeddings without a sentence-level objective.

  • Method

    The paper combines MLM and TLM pretraining with dual-encoder translation ranking and additive margin softmax for multilingual sentence embeddings.

  • Results

    83.7% accuracy on Tatoeba’s 112-language retrieval task exceeds the previous state-of-the-art’s 65.5%, while performance remains competitive on monolingual transfer learning.

  • Takeaways & Limitations

    The resulting LaBSE model supports 109+ languages and achieves state-of-the-art performance across bi-text retrieval and mining tasks with increased language coverage.

  • Takeaways & Limitations

    Mining hard negatives did not provide additional gain on top of the other approaches.

Abstract

from arXiv · show

While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.

1 Introduction

The paper combines pretrained multilingual encoders with dual-encoder translation ranking to improve cross-lingual sentence embeddings, achieving strong retrieval results and releasing a model for 109+ languages.

  • The paper combines pretrained language models with dual-encoder finetuning and established cross-lingual embedding methods.The explored methods include masked language modeling, translation language modeling, dual-encoder translation ranking, and additive margin softmax.
  • The models outperform previous state-of-the-art systems on large bi-text retrieval tasks including the United Nations corpus and BUCC.These evaluations cover resource-rich languages including French, German, Spanish, Russian, and Chinese.
  • 83.7% overall accuracy on Tatoeba’s 112-language retrieval task surpasses the previous state-of-the-art’s 65.5%.The models perform especially better on low-resource languages and work well on more than 30 languages without explicit monolingual or bilingual training data.
  • The work contributes a novel pretraining and dual-encoder combination that boosts translation-ranking performance and achieves state-of-the-art bi-text mining.
  • The authors publicly release a multilingual sentence embedding model spanning 109+ languages.
  • The paper includes experiments and ablations on pretraining, negative sampling, vocabulary, data quality, and data quantity.

2 Cross-lingual Sentence Embeddings

The paper combines dual-encoder translation ranking with pretrained multilingual encoders and additive margin softmax to learn cross-lingual sentence embeddings. Its resulting model improves multilingual retrieval and produces useful embeddings across more than 109 languages.

  • Dual encoders separately encode source and target sentences, then score their sentence embeddings for translation ranking.The training objective uses in-batch negatives and directly maximizes similarity between translation pairs in a shared embedding space.
  • Bidirectional training sums source-to-target and target-to-source ranking losses for symmetric cross-lingual alignment.The one-direction loss is asymmetric because the softmax may normalize over source or target sentences.
  • Additive margin softmax subtracts a margin from positive-pair scores to separate translations from nearby non-translations.The modified scoring function is applied within the bidirectional ranking loss.
  • Pretraining combines masked language modeling and translation language modeling before dual-encoder finetuning.TLM extends MLM by training on concatenated translation pairs, while the paper contrasts pretrained and randomly initialized encoders.

3 Corpus and Training Details

The training pipeline combines filtered monolingual and mined bilingual data with shared multilingual Transformer encoders, large-scale pretraining, and distributed negative sampling. Cross-accelerator sampling expands the effective negative set while distributing encoding computation.

  • Corpus: The corpus contains 17B filtered monolingual sentences from CommonCrawl and Wikipedia, used only for customized pretraining.Filtering removes noisy text, short or excessively long lines, and low-quality sentences.
  • Corpus: Bilingual translation pairs are mined from web pages, filtered with contrastive data-selection scores, and checked by human annotators.The threshold is selected using GOOD and BAD translation judgments from a harvested subset.
  • Model configuration: The model uses either a 119,547-token multilingual BERT vocabulary or a 501,153-token customized vocabulary.The customized vocabulary is built with a cased WordPiece tokenizer and language smoothing exponent 0.3.
  • Model configuration: Sentence embeddings are L2-normalized [CLS] representations from the last Transformer block, with encoder parameters shared across languages.The architecture follows BERT Base with 12 Transformer blocks, 12 attention heads, and 768 hidden units.
  • Training: Cross-accelerator negative sampling broadcasts representations from every core so other cores’ examples become negatives.This realizes large global-batch benefits while distributing computationally intensive encoding across accelerators.
  • Training: Pretraining uses MLM and TLM on monolingual and bilingual data, respectively, with three-stage progressive stacking.The model is trained successively with 1/4, 1/2, and all Transformer layers, copying earlier-stage parameters forward.

4 Evaluation Tasks

The paper evaluates multilingual sentence embeddings on three bitext retrieval or mining tasks and English-only SentEval transfer tasks. These benchmarks test retrieval accuracy, mining F1, and downstream classification performance under different data settings.

  • Bitext retrieval: The evaluation includes United Nations, Tatoeba, and BUCC bitext retrieval or mining tasks.The retrieval tasks seek correct English translations for non-English sentences.
  • Bitext retrieval: The UN corpus contains 11.3 million aligned sentence pairs across five language pairs, creating a large translation-candidate pool.The source documents comprise 86,000 bilingual documents in en-fr, en-es, en-ru, en-ar, and en-zh.
  • Bitext retrieval: Tatoeba evaluates translation retrieval across 112 languages with up to 1,000 sentences per language and English translations.The paper reports results on both the original 112-language version and the 36-language XTREME version.
  • Bitext retrieval: BUCC extracts translation pairs from monolingual corpora across four language pairs and evaluates against gold pairs using F1.Sentence-embedding cosine similarity identifies candidate translation pairs, while the training set is used for evaluation because test ground truth is unreleased.
  • Downstream classification: English-only SentEval tasks measure transfer to sentiment, question-type, product-review, subjectivity, opinion-polarity, and paraphrase classification.The benchmark is used to compare directly with prior sentence-embedding models.
  • Evaluation scope: Reranking models can improve raw embedding retrieval performance, but the evaluation focuses on retrieval using sentence embeddings.Examples include margin-based scorers and BERT-based classifiers.

5 Results

LaBSE’s strongest configuration combines customized vocabulary, additive margin softmax, and pre-training, outperforming comparison models on major retrieval benchmarks while remaining competitive for monolingual transfer.

  • Ablations and model configurations: Additive margin softmax improves every model variation, while pre-training improves UN retrieval and helps Tatoeba only with the customized vocabulary.The public mBERT vocabulary has poor tail-language coverage, including a 71% [UNK] rate for Sinhala, which may explain the Tatoeba limitation.
  • UN and Tatoeba retrieval: LaBSE’s top configuration uses customized vocabulary, additive margin softmax, and pre-training, achieving new state-of-the-art results on three of four evaluated language directions.Its P@1 scores are 91.1 for en-es, 88.3 for en-fr, and 90.8 for en-ru.
  • UN and Tatoeba retrieval: 91.1, 88.3, and 90.8 P@1 are reported for en-es, en-fr, and en-ru, while zh-en reaches 87.7, nearly nine points above the previous best multilingual model.The zh-en score is only 0.2 below the best bilingual en-zh model.
  • BUCC retrieval: LaBSE systematically outperforms prior work on BUCC while covering all languages in one model, replacing a previous setup requiring four bilingual models.The comparison includes baselines from Artetxe and Schwenk and Yang et al.
  • Transfer learning: Despite massive language coverage, LaBSE remains competitive with monolingual English embedding models and the 16-language multilingual universal sentence encoder on SentEval transfer.This result comes from the SentEval benchmark comparison against other state-of-the-art sentence embedding models.

6 Analysis

The analysis identifies additive margin softmax as critical for cross-lingual embeddings, while pre-training improves performance and reduces the parallel data required. LaBSE also performs strongly on languages without explicit training data, but underperforms on fine-grained English semantic similarity.

  • 6.1 Additive Margin Softmax: Additive margin softmax is a critical factor in learning good cross-lingual embeddings.
  • 6.1 Additive Margin Softmax: A margin of 0 gives approximately 60 average P@1 on UN, while performance improves through margin 0.3.With margin 0.1, average P@1 rises into the 70s to 80s; the trend is consistent across model variants.
  • 6.2 Effectiveness of Pre-training: Pre-trained encoders reach their highest performance after 50K steps, and additional training does not significantly improve results.The 50K-step model sees 200M examples, compared with 1B examples for the 500K-step model.
  • 6.2 Effectiveness of Pre-training: Pre-training improves performance and dramatically reduces the amount of parallel data required.Obtaining 200M parallel examples is generally easier than obtaining 1B examples from public sources.
  • 6.3 Low Resource Languages and Languages without Explicit Training Data: LaBSE achieves around 60% average accuracy on 30+ Tatoeba languages without explicit training data.Nearly one third exceed 75% accuracy, while only seven languages fall below 25%.
  • 6.4 Semantic Textual Similarity: LaBSE performs worse than other sentence embedding models on fine-grained English semantic similarity.The authors suspect translation-pair training favors detecting meaning equivalence over distinguishing fine-grained meaning overlap.

7 Mining Parallel Text from CommonCrawl

The authors use LaBSE to mine parallel sentences from CommonCrawl for English–Chinese and English–German translation. The mined data trains NMT systems competitively with systems using WMT17 parallel data.

  • LaBSE embeddings pair non-English CommonCrawl sentences with English sentences to mine parallel data for en-zh and en-de NMT.The processed corpus contains 1.17B Chinese, 0.6B German, and 7.73B English sentences.
  • A similarity threshold of 0.6 yields 715M en-zh and 302M en-de sentence pairs, though the pairs may remain noisy.The authors apply data selection to choose higher-quality pairs for NMT training.
  • Mined data produces en-de performance only 2.8 BLEU below the best WMT17 system.
  • Mined data produces an en-zh model as good as a WMT17 NMT model trained on WMT en-zh parallel data.

8 Conclusion

The paper presents LaBSE, a 109-language sentence embedding model that achieves strong bi-text retrieval and mining performance. Its experiments highlight additive margin softmax, data quality, and diminishing benefits from more parallel data with pre-training.

  • LaBSE supports 109 languages and achieves state-of-the-art performance on bi-text retrieval and mining tasks.
  • LaBSE performs strongly on languages without explicit training data, likely because of language similarity and its massively multilingual design.
  • Experiments identify additive margin softmax as a key factor, show that parallel data quality matters, and find diminishing effects from more parallel data after pre-training.

A LaBSELarge

LaBSELarge increases model capacity but does not clearly improve retrieval performance over LaBSE. The authors attribute this to limitations in the current translation-matching objective and in-batch negative sampling.

  • LaBSELarge uses a 24-layer Transformer with 16 attention heads and hidden size 1024.It is trained for 1M pre-training steps rather than the progressive multi-stage schedule used for LaBSE.
  • LaBSELarge and LaBSE have very close average performance on UN and Tatoeba.The reported Tatoeba averages are 83.7 for LaBSE and 83.8 for LaBSELarge.
  • The authors suspect the translation-matching objective is too easy for current in-batch negative sampling to provide further learning.They leave improved negative contrast and additional hard-negative types for future work.

B Hard Negative Mining

The paper investigates hard-negative mining for cross-lingual sentence embeddings, using a weaker dual encoder to generate additional translation-pair negatives. Because mining is costly, the experiment is restricted to Spanish and adjusts the training setup for memory constraints.

  • B Hard Negative Mining: Hard negatives are challenging to apply to large datasets because the process is time consuming and computationally costly.
  • B Hard Negative Mining: Hard-negative mining uses a weaker dual encoder trained to identify translation pairs, then appends the mined negatives to each example.The pipeline follows Guo et al. (2018) and uses a deep averaging network as the weaker encoder.
  • B Hard Negative Mining: Hard negatives are tested only for Spanish, with 3 mined negatives per English source sentence and batch size reduced from 128 to 32.The restriction reflects the high cost of obtaining hard negatives for all languages and memory constraints from the expanded examples.
  • B Hard Negative Mining: Table 9 reports P@1 on UN en→xx when hard-negative examples are used in en-es.

C Supported Languages

The training set covers 109 languages, with language support and data distributions documented in Table 10 and Figure 7. English monolingual sentences are capped at 2 billion.

  • C Supported Languages: Figure 7 shows the quantity of monolingual sentences and bilingual sentence-pairs for each of the 109 training languages.
  • C Supported Languages: English monolingual sentences are capped at 2 billion in the training data.
Loading 2007.01852v2…