Source-linked AI summary
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, Francisco Guzmán
TL;DR
The paper addresses the shortage of multilingual parallel data, especially for language pairs not involving English, by mining Wikipedia with massively multilingual sentence embeddings. It systematically extracts candidate translations across language pairs using margin-based similarity search and reports a large freely available corpus with MT-based quality evaluation.
Problem
Multilingual parallel resources are limited, particularly for language pairs that do not involve English, despite abundant multilingual monolingual text.
Method
The authors mine Wikipedia across language pairs using multilingual sentence embeddings, margin-based candidate scoring, language identification, and large-scale similarity search.
Results
135M parallel sentences were extracted in 85 languages across 1620 language pairs, with 34M aligned with English, and evaluated through BLEU scores from NMT systems trained on mined data.
Takeaways & Limitations
WikiMatrix provides a freely available, broad-topic multilingual corpus that may support machine translation and research involving many languages.
Takeaways & Limitations
The mined corpus may contain wrong or structurally similar alignments, and the approach was applied independently of LASER embedding quality for each language.
Abstract
from arXiv · showhide
We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 85 languages, including several dialects or low-resource languages. We do not limit the the extraction process to alignments with English, but systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus of parallel sentences is freely available at https://github.com/facebookresearch/LASER/tree/master/tasks/WikiMatrix. To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.
1 Introduction
The paper addresses limited multilingual parallel resources by mining Wikipedia, a diverse multilingual source, across language pairs using multilingual sentence embeddings and scalable similarity search.
- Motivation: Multilingual parallel resources are more limited when neither language is English, despite abundant monolingual text across languages and topics.Existing resources include professional translations and volunteer-translated collections, but may be formal, topic-limited, or incomplete.
- Motivation: Wikipedia contains diverse multilingual content, including translated and independently written articles that may contain mutually translated sentences.Its articles span more than 300 languages, although later edits and machine translation can reduce parallelism or introduce noise.
- Contribution: The work systematically mines the entire Wikipedia for parallel sentences across all language pairs, rather than restricting extraction to English alignments.The authors describe this as the first such large-scale, all-language-pair processing of Wikipedia’s textual content.
- Approach: The mining approach uses distances in a joint multilingual sentence-embedding space, implemented with the LASER toolkit trained on 93 languages.Fast indexing and similarity-search algorithms address the computational challenge of mining almost six hundred million sentences.
- Evaluation: The authors evaluate mined bitext quality by training neural MT systems for a subset of language pairs and testing them on TED data covering 45 languages.The evaluation is presented as an indication of the quality of the extracted parallel sentences.
2 Related work
Prior bitext-mining work uses metadata, document comparison, translation, lexical methods, and bilingual encoders, but these approaches generally do not scale to Wikipedia’s many language pairs.
- Earlier mining methods: Earlier comparable-corpus mining methods relied on engineered metadata, cross-lingual document retrieval, machine translation, language models, lexical probabilities, or named entities.These methods typically produce or score candidate document alignments before further filtering.
- This work: The paper relies on massively multilingual sentence embeddings and margin-based mining in a joint embedding space.The approach was reported to perform best in a low-resource scenario among the cited methods.
- This work: Bilingual sentence representations do not scale efficiently to thousands of language pairs, whereas one multilingual embedding can cover many languages.The paper positions this scalability difference as central when considering all possible Wikipedia language pairs.
- Wikipedia mining: Wikipedia has been mined using machine translation, structural information, named entities, time expressions, bilingual encoders, and aligned titles.Previous efforts generally targeted individual language pairs, selected languages, or non-sentence title alignments.
- Novelty: The authors report no other attempts to systematically mine parallel sentences from Wikipedia’s textual content for a large number of languages.They characterize the work as applying the same approach across many languages and scripts.
3 Distance-based mining approach
The mining pipeline represents sentences in a shared multilingual embedding space, scores candidates with a bidirectional margin, and uses compressed FAISS indexes for large-scale search.
- 3.1 Margin criterion: A multilingual sentence embedding places semantically similar sentences close together across languages, making distance an indicator of mutual translation.The approach replaces a simple absolute cosine-distance threshold with a margin criterion to improve alignment quality.
- 3.1 Margin criterion: The margin compares candidate-pair similarity with the average similarity to nearest neighbors in both languages.NN_k(x) and NN_k(y) denote the k unique nearest neighbors in the other language, with k = 4 in all experiments.
- 3.1 Margin criterion: The max strategy computes margins in both directions, unions candidates, ranks them, removes reused source or target sentences, and applies a margin threshold.This produces a one-to-one selection process after combining forward and backward candidates.
- 3.3 Fast similarity search: Brute-force distance mining has complexity O(N × M), making exhaustive search intractable for large monolingual corpora.The English and German Wikipedia corpora alone would require 6.8×10^15 distance calculations.
- 3.2 Multilingual sentence embeddings: One shared LASER embedding is used for all languages instead of training a separate bilingual embedding for each language pair.LASER uses a shared BPE vocabulary and encoder, with sentence representations obtained by max-pooling encoder output states.
- 3.3 Fast similarity search: FAISS performs compressed nearest-neighbor search over 1024-dimensional sentence representations, using product quantization and 32k search cells.The method builds one FAISS index for each language.
- 3.3 Fast similarity search: 9.2GB is sufficient for the compressed English FAISS index, reducing storage by more than fifty times relative to the original embeddings.German/English mining takes less than 3.5 hours on 8 GPUs, including bidirectional search and candidate scoring.
4 Bitext mining in Wikipedia
The paper mines Wikipedia globally by comparing sentences across languages, then evaluates preprocessing and margin thresholds through downstream NMT performance. It selects a shared threshold while examining corpus scale, noise, and comparison with Europarl.
- 4 Bitext mining in Wikipedia: Global mining compares each source sentence with all target-language sentences, enabling pairs with few shared articles but risking misalignment and lower recall.The authors chose this option because it can scale beyond corpora with readily available document alignments.
- 4.1 Corpus preparation: Language identification removes non-target-language sentences because language-agnostic embeddings can rank citations or quotes above a correct translation.Table 1 illustrates how such sentences can reduce the margin of a valid alignment.
- 4.1 Corpus preparation: 595M sentences in 182 languages remain after deduplication and language identification, with English contributing 134M and German 51M.The corpus is derived from about 300 available language editions, whose sizes vary substantially.
- 4.2 Threshold optimization: The margin threshold controls a trade-off: lower thresholds increase mined data exponentially, while excessive noisy translations eventually reduce BLEU.The optimal threshold is about 1.05 for German/English and German/French, but 1.03–1.04 for Czech/German and Czech/French; 1.04 is used as a compromise.
- 4.2 Threshold optimization: Except for Czech/French, Wikipedia bitexts outperform equally sized Europarl data in BLEU, while adding mined text to full Europarl improves BLEU by 1.1–3.1.The comparison uses NMT systems evaluated on newstest2014.
5 Result analysis
The mining process produced substantial parallel data across many language pairs, including non-English and low-resource combinations. NMT evaluation on TED provided an indication of mined-bitext quality, while corpus size and source-text quality remained important qualifications.
- Quantitative analysis: 1620 language pairs yielded at least ten thousand mined sentences, with results summarized by extracted-corpus size.Language-pair directions are counted once, and Table 3 reports counts in thousands for qualifying languages.
- Quantitative analysis: More sentences were generally mined when English was included, but Russian/Ukrainian reached 2.5M and Catalan/Spanish 1.6M without English.Other large non-English pairs included Romance-language combinations with 480k–923k sentences.
- Quantitative analysis: At least ten thousand parallel sentences were extracted for 85 languages, including pairs such as Korean/Japanese, Indonesian/Vietnamese, and Hebrew/Romance languages.Several low-resource languages yielded more parallel sentences with other languages than with English.
- Quantitative analysis: Cebuano had 17.9M filtered monolingual sentences, yet very few parallel sentences could be extracted because most Wikipedia content was bot-generated.The same issue was reported for Waray, whereas alignments for Swedish appeared better despite bot-generated articles there.
- Qualitative evaluation: NMT systems trained only on mined bitexts covered 1886 language pairs in 45 languages and were evaluated on TED test sets using BLEU.Training used pairs with more than 25,000 mined sentences and a Transformer model; no TED training data were used.
- Qualitative evaluation: Non-English evaluations included Norwegian↔Danish ≈33, Norwegian↔Swedish ≈25, Indonesian↔Vietnamese ≈16, and Japanese/Korean ≈17 BLEU.The authors describe these scores as interesting results for pairs that are usually difficult to resource.
- Qualitative evaluation: TED BLEU scores indicate mined-bitext quality but must be interpreted alongside mined-corpus sizes and possible incorrect alignments.The approach was applied to all available Wikipedia languages regardless of LASER embedding quality for each language.
6 Conclusion
The paper releases a large, systematically mined Wikipedia bitext corpus spanning many languages and language pairs. It also identifies iterative embedding improvement and application to other multilingual collections as future directions, while expecting broad topical coverage and mostly well-formed text.
- Conclusion: The released WikiMatrix corpus contains 135M parallel sentences in 85 languages, with 34M aligned with English and at least 10,000 sentences for 1620 language pairs.The same language-agnostic mining approach was used across all language pairs without language-specific optimization.
- Conclusion: The mined corpus is freely available and was evaluated by training 1886 NMT systems on 45 TED languages.The evaluation used the mined sentences as training data and TED data for evaluation.
- Future research: Future work includes retraining LASER on mined texts, repeating the mining process iteratively, and applying the methodology to ParaCrawl or CommonCrawl.These collections are identified as candidates for extending the methodology beyond Wikipedia.
- Conclusion: The authors expect WikiMatrix to contain mostly well-formed, broadly topical text with little social-media language and a low fraction of unedited machine translation.They hope the resource will support research in multilinguality, particularly machine translation.
A Appendix
The appendix documents NMT training settings and reports additional BLEU results, while noting quality concerns for alignments involving languages with small Wikipedia coverage.
- Limitations: Several small-Wikipedia alignments may be low quality because LASER was not directly trained on most of those languages.The authors retain the expectation of reasonable results when related languages are covered.
- Mined data: Table 5 lists extracted-sentence counts for languages with rather small Wikipedia editions.Alignments yielding fewer than 5k sentences are omitted.
- Training configuration: The NMT models were trained using the detailed configuration reported in Table 3 and illustrated in Figure 3.Figure 3 presents model settings for fairseq training.
- Results: Table 6 reports BLEU scores for NMT systems trained only on Wikipedia-mined bitexts.No other resources were used for training.