Source-linked AI summary
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings
Mikel Artetxe, Holger Schwenk
TL;DR
The paper addresses the need for high-quality parallel data because NMT is sensitive to training-data size and quality. It trains multilingual sentence embeddings and replaces hard cosine thresholds with neighborhood-based margins. The method improves benchmark mining and reconstruction results and yields 31.2 BLEU on filtered English-German ParaCrawl.
Problem
NMT is highly sensitive to training-data size and quality, motivating better methods for mining and filtering large parallel corpora.
Method
The method trains multilingual sentence embeddings with a sequence-to-sequence architecture and scores candidate pairs using margins relative to nearest-neighbor cosine similarities.
Results
The system achieves the best published BUCC results, improves UN reconstruction precision by more than 30 points, and reaches 31.2 BLEU on English-German ParaCrawl.
Takeaways & Limitations
Margin-based scoring improves parallel-corpus mining and filtering across BUCC, UN reconstruction, and downstream English-German machine translation.
Takeaways & Limitations
The margin approach penalizes sentences with many paraphrases, although the authors argue such cases are rare and filtering them is unlikely to cause major harm.
Abstract
from arXiv · showhide
Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collecting and filtering large parallel corpora. In this paper, we propose a new method for this task based on multilingual sentence embeddings. In contrast to previous approaches, which rely on nearest neighbor retrieval with a hard threshold over cosine similarity, our proposed method accounts for the scale inconsistencies of this measure, considering the margin between a given sentence pair and its closest candidates instead. Our experiments show large improvements over existing methods. We outperform the best published results on the BUCC mining task and the UN reconstruction task by more than 10 F1 and 30 precision points, respectively. Filtering the English-German ParaCrawl corpus with our approach, we obtain 31.2 BLEU points on newstest2014, an improvement of more than one point over the best official filtered version.
1 Introduction
Parallel corpus mining is important because neural machine translation depends strongly on training-data size and quality. The paper replaces hard cosine-similarity thresholding with a margin-based approach because cosine scales vary across sentences.
- NMT is particularly sensitive to the size and quality of its training data, making effective parallel-corpus mining and filtering crucial.
- Earlier mining systems used web-crawl metadata, classifiers over bag-of-word features, lexical translations, document retrieval, or machine translation.
- Multilingual sentence-embedding methods retrieve candidates by nearest-neighbor cosine similarity and filter them with a hard threshold.
- Cosine similarity can be poorly calibrated across sentences: an incorrect alignment may score higher than a correct one, preventing reliable fixed-threshold filtering.
- The proposed method instead evaluates the margin between a candidate pair and its k nearest neighbors.
2 Multilingual sentence embeddings
The system trains a shared multilingual encoder-decoder that maps sentences from multiple languages into fixed-length embeddings. After training, the decoder is discarded and the encoder supplies the representations for mining.
- The model uses a sequence-to-sequence architecture with a shared encoder and decoder across languages.
- The encoder is a bidirectional LSTM, and max-pooling over its outputs produces sentence embeddings.
- The decoder uses sentence embeddings to initialize its hidden and cell states and concatenates them with input embeddings at every time step.
- A joint 40k BPE vocabulary is learned from concatenated training corpora, while the encoder remains language agnostic.
- After training, the decoder is discarded and the encoder maps each sentence to a fixed-length vector.
3 Scoring and filtering parallel sentences
The method scores candidate translations by comparing their cosine similarity with neighborhood similarities rather than using cosine values alone. It generates candidates in forward, backward, intersection, or max-score modes before thresholding margin-ranked pairs.
- Nearest-neighbor cosine retrieval with a fixed threshold is limited because cosine similarity has inconsistent scales across sentences.
- A motivating example shows that a wrong pair can have higher cosine similarity than a correct pair, while a large neighbor gap indicates stronger alignment.
- The approach may penalize sentences with many paraphrases, although the authors argue such cases are rare and unlikely to cause major harm when filtered.
- Margin-based scoring: The method computes margins between a candidate cosine and the average cosine of its k nearest neighbors in both directions.
- Margin-based scoring: The explored margin variants are absolute, distance, and ratio; absolute is equivalent to cosine similarity, while distance subtracts neighborhood similarity.
- Candidate generation and filtering: Candidate generation can be forward, backward, their intersection, or max score, followed by margin sorting and thresholding.
4 Experiments and results
Experiments evaluate margin-based multilingual sentence-embedding mining on BUCC, UN reconstruction, and filtered English-German ParaCrawl. The method achieves strong gains in mining benchmarks and improves downstream translation quality.
- Evaluation setup: The evaluation covers BUCC mining, UN corpus reconstruction, and English-German machine translation using filtered ParaCrawl.Experiments use multilingual encoders trained on Europarl v7 and, for BUCC coverage, a separate model trained on the UN corpus.
- BUCC mining task: BUCC training results show margin-based scoring outperforms cosine similarity by more than 10 points, with ratio scoring beating distance by 0.3–0.5 points.Intersection gives the best retrieval results, while retrieval strategy has little effect on margin-based scoring.
- BUCC mining task: 10–15 F1 points: the proposed system improves over previous methods on the BUCC test set with consistent performance across languages, including distant ones.The reported test-set comparison uses ratio margins with maximum-score retrieval and thresholds optimized on training data.
- UN corpus reconstruction: The system outperforms Guo et al. (2018) by a large margin when aligning 11.3M UN sentences, despite using only 2M Europarl training sentences.This reconstruction task uses forward retrieval with the ratio margin function and does not require filtering.
- Filtering ParaCrawl for NMT: ParaCrawl filtering removes duplicates, language-identification mismatches, extreme sentence lengths, and high-overlap or imbalanced pairs before ratio-margin scoring.Preprocessing reduces the corpus from 4.59 billion to 64.4 million sentence pairs, after which top-scoring entries are selected up to the desired size.
- Filtering ParaCrawl for NMT: The filtered ParaCrawl system outperforms both official filtered versions and most prior systems, with gains attributed to better ParaCrawl filtering.The comparison includes systems using different training data; Edunov et al. (2018) remains ahead through large in-domain monolingual data and back-translation.
5 Conclusions and future work
The paper presents margin-based multilingual sentence-embedding mining, reports strong gains across mining and translation evaluations, and releases the code through LASER.
- The method combines a sequence-to-sequence multilingual sentence encoder with margin-based scoring to address cosine-similarity scale inconsistencies.
- More than 10 F1 points were gained over previous systems on all four BUCC language pairs, while UN reconstruction reached up to 85% precision and improved by more than 30 points.
- 31.2 BLEU points were obtained on English-German newstest2014 using filtered ParaCrawl, improving by more than one point over the best official release.
- The code is freely available in the LASER toolkit, which also includes a single encoder covering 93 languages.
- 29.4 tokenized BLEU on separately trained WMT data was comparable to Ott et al.'s 29.3, indicating the difference was not attributable to implementation details.