Source-linked AI summary
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil
TL;DR
Parallel corpus mining needs large amounts of quality data, but existing distributed approaches are computationally intensive. The paper trains bilingual sentence embeddings with semantically similar hard negatives and uses them for mining and filtering. The method reconstructs sentence-level parallel data with 48.9% P@1 for en-fr and 54.9% P@1 for en-es, while reconstructed-data NMT models come within 1-2 BLEU of models trained on original data.
Problem
Quality parallel data is critical for NMT, while existing distributed mining approaches are computationally intensive and heavily engineered.
Method
A bilingual dual-encoder separately embeds source and target sentences, scores them by dot product, and trains with semantically similar non-translation hard negatives.
Results
The method supports parallel-data mining and filtering, reconstructs the United Nations corpus at 48.9% P@1 for en-fr and 54.9% P@1 for en-es, and yields NMT models within 1-2 BLEU of original-data models.
Takeaways & Limitations
Sentence embeddings provide a lightweight alternative for mining and filtering parallel data, with performance comparable to heavier document mining and close to original-data NMT training.
Takeaways & Limitations
Translation quality from the United Nations corpus is much lower than WMT state of the art, likely because the corpus is small and domain-restricted.
Abstract
from arXiv · showhide
This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings. Our embedding models are trained to produce similar representations exclusively for bilingual sentence pairs that are translations of each other. This is achieved using a novel training method that introduces hard negatives consisting of sentences that are not translations but that have some degree of semantic similarity. The quality of the resulting embeddings are evaluated on parallel corpus reconstruction and by assessing machine translation systems trained on gold vs. mined sentence pairs. We find that the sentence embeddings can be used to reconstruct the United Nations Parallel Corpus at the sentence level with a precision of 48.9% for en-fr and 54.9% for en-es. When adapted to document level matching, we achieve a parallel document matching accuracy that is comparable to the significantly more computationally intensive approach of [Jakob 2010]. Using reconstructed parallel data, we are able to train NMT models that perform nearly as well as models trained on the original data (within 1-2 BLEU).
1 Introduction
The paper addresses computationally intensive parallel corpus mining with bilingual sentence embeddings trained to distinguish translations from semantically similar non-translations. Efficient encoders and nearest-neighbour search support sentence and document matching, while reconstructed data produces NMT models close to those trained on original parallel data.
- Method: Lightweight sentence encoders and dot-product scoring reduce mining costs, while nearest-neighbour search identifies candidate translation pairs in large unpaired collections.This avoids the heavily engineered, computationally intensive subsystems used by earlier distributed mining approaches.
- Results: 48.9% P@1 for en-fr and 54.9% P@1 for en-es were achieved when reconstructing the United Nations Parallel Corpus at sentence level.The method was also adapted to document matching and NMT training.
- Method: Hard negatives are semantically similar non-translations, and using them improves identification of true translation pairs over random negatives.The model trains separate language encoders and scores candidate pairs with embedding dot products.
- Results: NMT models trained on reconstructed parallel data perform nearly as well as models trained on the original corpus, within 1-2 BLEU.Document matching accuracy is comparable to the heavier approach of Uszkoreit et al. (2010).
2 Approach
The approach trains bilingual dual encoders to rank translations above alternatives, using dot-product sentence scores and semantically similar hard negatives. Efficient retrieval and calibrated scoring support sentence- and document-level parallel corpus mining.
- 2.1 Translation Candidates Ranking Task: The model ranks each true target translation above sampled alternatives using a log-linear probability based on a compatibility score.The normalization is approximated with the positive candidate and K −1 sampled negatives rather than all possible target sentences.
- 2.2 Bilingual Sentence Embeddings: Separate language-specific DAN encoders average word and bi-gram embeddings, transform them with feedforward networks, and score sentence pairs by the dot product of their representations.The dual-encoder architecture independently maps source x to u and target y to v, with φ(x, y) = uT · v; residual connections are included across hidden layers.
- 2.2 Bilingual Sentence Embeddings: Within-batch negatives make scoring efficient because one matrix multiplication produces diagonal scores for true pairs and off-diagonal scores for incorrect pairings.The method uses the encoding matrices U and V⊤ and supports mini-batch training with randomly shuffled negatives.
- 2.3 Semantically Similar Hard Negatives: Hard negatives are semantically similar nontranslations selected by a baseline model, making training distinguish true translations from merely related sentences.Random negatives are often obviously incorrect, whereas hard negatives receive high dot-product scores but are not the correct translation; they are used for 20% of the data because obtaining them is time-consuming.
- 2.4 Mining Parallel Data: Target sentences are pre-encoded and retrieved with approximate nearest-neighbor search, enabling efficient candidate mining from large unpaired collections.The same embeddings support independent sentence pairing and an adaptation for parallel document matching.
- 2.4.2 Calibrated Confidence Score: Dynamic score normalization addresses the lack of a globally consistent raw dot-product confidence scale across source sentences.The normalization computes source-dependent scaling and shifting values from u and u2, with weights trained on held-out supervised data.
3 Experiments
Experiments evaluate bilingual sentence embeddings on sentence retrieval, corpus reconstruction, document matching, and downstream translation, using en-fr and en-es data. Hard negatives improve retrieval over random negatives, while reconstructed data supports NMT performance close to original parallel data.
- Experimental setup: The experiments train and evaluate models for English-French and English-Spanish, using precision at N for translation-target retrieval.For each source sentence, the method ranks the true target among one positive and 10 million randomly selected negatives.
- Sentence-level matching: Hard negatives outperform random negatives across all P@N metrics, with five hard negatives improving P@1 by 8% for en-fr and 3% for en-es.Adding more hard negatives does not always further improve performance.
4 Reconstructing the United Nations Corpus
The model reconstructs UN parallel data through sentence retrieval and document matching, then tests whether mined pairs support machine translation. Hard-negative retrieval reaches 48.9% P@1 for en-fr and 54.9% for en-es, while mined-data NMT performs nearly as well as models trained on original pairs.
- 4.1 UN Sentence Level Matching: 48.9% P@1 for en-fr and 54.9% for en-es are achieved when hard negatives are used to retrieve targets from the full UN corpus.The retrieval task selects the true translation from 11.3 million aligned sentence pairs.
- 4.2 UN Document Level Matching: Document matching with Eq. (3) outperforms counting mutually aligned sentences and remains promising despite being simpler and less computationally intensive than Uszkoreit et al. (2010).
- 4.3 Evaluation Using a Translation Model: NMT models trained on mined UN pairs perform very close to Oracle models trained on original UN pairs.Sentence-level mined pairs even slightly outperform Oracle data for en-es, presumably because confidence filtering produces cleaner pairs.
- 4.3 Evaluation Using a Translation Model: The UN-trained translation results remain below WMT state of the art because the UN corpus is small and drawn from a particularly restricted domain.
- 4.4 Filtered ParaCrawl data: Filtered ParaCrawl data selected by the proposed scoring method slightly outperforms data filtered by Zipporah scores.The comparison uses 43 million en-fr pairs and 24 million en-es pairs selected under the Zipporah threshold.
5 Analysis
The analysis compares the proposed confidence scores with Zipporah through score agreement and human judgments. The methods correlate weakly, yet the proposed scores achieve higher agreement with human judgments for en-fr while sharing a partially translated and identical-sentence failure pattern.
- 5 Analysis: The Pearson correlation between Zipporah and calibrated confidence scores is only 0.4, suggesting that the methods may provide complementary information.
- 5 Analysis: The proposed score normalization model has a 4% higher GOOD translation rate than Zipporah for en-fr, while performance is close for en-es.
- 5 Analysis: A common failure pattern is pairing a partially translated sentence with its counterpart, likely because the training data lacks enough such hard negatives.
- 5 Analysis: The proposed model produces more identical sentence pairs, which are mostly labeled BAD because they are unlikely to be actual translations.
6 Related Work
Parallel corpus mining has progressed from metadata-based web document extraction to embedding-based methods that map texts into shared spaces. This paper distinguishes its approach through a more efficient dual-encoder architecture trained directly for parallel sentence extraction.
- Earlier parallel document mining commonly relied on metadata such as titles, publication dates, and document structure.
- Embedding-based approaches map source and target texts into a shared embedding space to determine whether they are parallel.
- The proposed method uses a dual-encoder rather than an encoder-decoder architecture, improving efficiency and enabling direct training for extracting parallel sentences.
7 Conclusion
The paper presents a bilingual dual-encoder approach that scores candidate translation pairs with sentence-embedding dot products and trains with semantically similar hard negatives. Hard negatives improve true-pair identification, while the method supports parallel-data mining and filtering with competitive quality and lower computational cost for mining.
- Hard negatives improve the model’s ability to identify true translation pairs.
- The bilingual dual-encoder separately embeds source and target sentences, then uses their dot product to assess translation-pair quality.
- The method is useful for both mining and filtering parallel data, comparing favorably with Zipporah for filtering.
- For mining, the approach provides a lightweight alternative to Uszkoreit et al. (2010)’s method.