PaperlayerView exact revision on arXiv

Source-linked AI summary

CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB

Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin

arXiv:1911.04944v2cs.CL

TL;DR

Parallel data are scarce outside English, while monolingual web text is available at enormous scale. CCMatrix applies multilingual embedding and margin-based mining to Common Crawl without document-alignment assumptions, mining billions of sentence pairs that support strong NMT results across English-centered and non-English language pairs.

  • Problem

    Parallel multilingual resources are limited, especially for language pairs that do not involve English, despite the abundance of monolingual text.

  • Method

    CCMatrix uses margin-based mining in a shared multilingual sentence-embedding space to compare sentences across curated Common Crawl corpora without document-alignment assumptions.

  • Results

    NMT systems trained only on mined bitexts outperform the best reported single systems on WMT’19 for several English and German/French directions and achieve state-of-the-art BLEU for Russian/Japanese on WAT’19.

  • Takeaways & Limitations

    The approach yields a very large collection of high-quality mined parallel sentences, including direct alignments between many language pairs not involving English.

  • Takeaways & Limitations

    The next CCMatrix version will add more Common Crawl snapshots and focus on low-resource languages, indicating the current corpus’s coverage remains bounded.

Abstract

from arXiv · show

We show that margin-based bitext mining in a multilingual sentence space can be applied to monolingual corpora of billions of sentences. We are using ten snapshots of a curated common crawl corpus (Wenzek et al., 2019) totalling 32.7 billion unique sentences. Using one unified approach for 38 languages, we were able to mine 4.5 billions parallel sentences, out of which 661 million are aligned with English. 20 language pairs have more then 30 million parallel sentences, 112 more then 10 million, and most more than one million, including direct alignments between many European or Asian languages. To evaluate the quality of the mined bitexts, we train NMT systems for most of the language pairs and evaluate them on TED, WMT and WAT test sets. Using our mined bitexts only and no human translated parallel data, we achieve a new state-of-the-art for a single system on the WMT'19 test set for translation between English and German, Russian and Chinese, as well as German/French. In particular, our English/German system outperforms the best single one by close to 4 BLEU points and is almost on pair with best WMT'19 evaluation system which uses system combination and back-translation. We also achieve excellent results for distant languages pairs like Russian/Japanese, outperforming the best submission at the 2019 workshop on Asian Translation (WAT).

1 Introduction

Parallel sentence resources are much scarcer than monolingual text, especially outside English, motivating large-scale mining without document-structure assumptions. CCMatrix scales multilingual sentence mining to billions of web sentences and evaluates the resulting bitexts through NMT.

  • Monolingual texts are abundant across languages and topics, whereas parallel resources are more limited, particularly for language pairs not involving English.Existing parallel sources include formal institutional translations and volunteer-translated public texts, often constrained in topic or style.
  • Institutional and volunteer translation collections provide important bitexts but are limited by formal language, political-topic concentration, or uneven coverage.
  • 32.7 billion unique Common Crawl sentences replace Wikipedia’s 550 million-sentence scale, creating a substantially larger mining problem.The expansion requires comparing billions of sentence embeddings and redesigning the processing pipeline.
  • Unlike hierarchical approaches that preselect potentially parallel documents, CCMatrix compares sentences across monolingual corpora without assuming document structure.Hierarchical local mining is faster but cannot align sentences in documents that were not preselected.
  • NMT systems trained only on the mined data outperform the best reported single systems in WMT’19 and WAT’19 evaluations.The paper evaluates mined bitext quality on TED, WMT, and WAT test sets for a subset of language pairs.

2 Related work

Prior bitext mining methods use metadata, document retrieval, translation models, lexical signals, or bilingual representations. The paper builds on multilingual sentence embeddings and margin-based mining as a scalable alternative for many language pairs.

  • Early bitext mining relied heavily on engineered systems and metadata, while later methods used document retrieval, machine translation, and textual-content scoring.These approaches typically first obtain candidate document alignments and then filter or score them.
  • Shared-task systems scored candidate document pairs with language models, translation models, lexical probabilities, or Jaccard-based seed-translation expansion.
  • Multilingual sentence embeddings with margin-based mining avoid training separate bilingual representations for every language pair.The cited approach had also performed best in a low-resource scenario.
  • Wikipedia mining exploited comparable documents, structural information, entities, and translation-based comparisons, often in bilingual or high-resource settings.
  • ParaCrawl’s Bitextor uses a two-stage pipeline that identifies parallel documents before mining aligned segments within document pairs.

3 The curated Common Crawl corpus

CCMatrix mines Common Crawl snapshots after preprocessing them into cleaner, language-identified monolingual text. The corpus is intended to provide web-derived parallel data at broad scale.

  • CCMatrix uses monthly Common Crawl snapshots containing terabytes of web pages in multiple languages.The snapshots are randomly obtained by exploring URLs.
  • Figure 1 reports the number of unique sentences in ten CCNet crawls.
  • Paragraph-level deduplication removes up to 70% duplicated data and helps eliminate boilerplate, navigation menus, and cookie warnings.
  • The corpus pipeline identifies each document’s language after preprocessing the raw Common Crawl text into the CCNet dataset.

4 Distance-based mining approach

The mining system embeds sentences from many languages in one shared space, scores candidate pairs with a neighborhood-based margin, and scales indexing through parallel processing. It adapts the pipeline to corpora exceeding tens of billions of sentences.

  • 4 Distance-based mining approach: Multilingual sentence embeddings place semantically similar sentences near one another across languages, enabling distance-based candidate mining.
  • 4.1 Margin criterion: Margin scoring improves alignment quality over an absolute threshold by comparing candidate similarity with average nearest-neighbor similarities in both directions.
  • 4.1 Margin criterion: The max-strategy computes forward and backward margins, combines candidates, removes reused sentences, and applies a threshold to select mutual translations.
  • 4.1 Margin criterion: The neighborhood size k increases from 4 to 16 to reduce exclusion of valid translations when many nearest neighbors are similarly close.
  • 4.2 Multilingual sentence embeddings: One shared massively multilingual embedding is used because separate bilingual embeddings are difficult to scale to the many language pairs in CCNet.LASER uses a shared BPE vocabulary and encoder across languages, with max-pooling over encoder states for sentence representations.
  • 4.2 Multilingual sentence embeddings: The pipeline was substantially modified for a 32.7 billion-sentence corpus, whose largest language collections include 8.7 billion English and 3 billion Russian sentences.

1. text extraction

CCMatrix parallelizes text processing, indexing, and pairwise mining to make global bitext extraction feasible at web scale. The pipeline uses compressed multilingual indexes and parallel forward/backward distance calculations.

  • Pipeline overview: The pipeline has three stages: text processing, compressed index creation for each language, and pairwise parallel-data mining.Each stage is designed for parallel execution.
  • Text extraction: Text extraction splits paragraphs into sentences, performs language identification, and marks duplicates within independently processed blocks.The blockwise results are later merged into globally unique sentence sets.
  • Index creation: 32.7 · 10^9 1024-dimensional sentence embeddings would require approximately 130 TB, motivating aggressive 64-bit product-quantizer compression.FAISS indexes partition the search space into 64k cells using OPQ64,IVF65536,PQ64.
  • Index creation: A single eight-GPU index holds about three billion sentences, so English is split across three indexes because it contains 8.7 billion sentences.The indexes use GPUs for tractable exhaustive searching and are built from independently processed blocks.
  • Pairwise mining: Japanese/Russian mining entails approximately 8.7 · 10^18 distance calculations, about 1300 times more than the cited English/German example.The system addresses this scale by decoupling forward and backward distances from margin calculation and processing them in parallel.
  • Pairwise mining: English alignment uses only forward distances because searching the German index for all 8.7 billion English sentences is too expensive, whereas other pairs use both directions.For non-English pairs, the max-margin strategy calculates forward and backward distances.

5 Quantitative result analysis

The experiments examine threshold selection and the scale and distribution of mined alignments. A threshold of 1.06 yields billions of parallel sentences, including substantial direct data for non-English pairs, while alignment counts vary with language pairing.

  • 5.1 Choosing the margin threshold: A margin threshold of 1.06 gives the best Hu-Da TED performance in the reported threshold experiment.Higher thresholds improve alignment quality but reduce dataset size, creating a size–quality trade-off.
  • 5.2 Analysis: 4.5 billion parallel sentences are mined at margin threshold 1.06, including 661 million aligned with English.These counts are reported for the CCMatrix corpus.
  • 5.2 Analysis: Direct parallel data is provided for many language pairs not involving English, addressing cases where pivoting through English can be suboptimal.The paper specifically motivates direct data for morphologically rich or very different language pairs.
  • 5.2 Analysis: About three million parallel sentence pairs are obtained across the 28 considered non-English language-pair languages.The paper presents CCMatrix as the largest collection of high-quality mined parallel texts to its knowledge.
  • 5.2 Analysis: With roughly equal monolingual sizes, Polish–Czech yields 13.2M alignments versus Dutch–Czech’s 11.6M, while German alignment favors Dutch at 33.2M versus Polish at 20.4M.Both Polish and Dutch have much smaller bitexts with Indonesian despite more than 360M Indonesian sentences.
  • 5.2 Analysis: The authors suggest LASER alignments may be more reliable for languages in the same family, while shared regional interests may also increase web translation likelihood.These are presented as possible explanations for variation in extracted bitext counts.

6 Qualitative result evaluation

The authors assess CCMatrix bitext quality by training NMT systems only on mined data and evaluating them across TED, WMT’19, and WAT’19 test sets. The mined data performs strongly across language pairs, including high-resource and distant-language evaluations.

  • 6.1 TED corpus: The TED evaluation trains 702 NMT systems across 28 languages using a shared Transformer architecture and language-specific tokenization where needed.Models use six encoder and decoder layers, 60k BPE vocabularies, and selection by validation BLEU.
  • 6.1 TED corpus: 46 of 702 language pairs exceed 30 BLEU on TED, compared with 10 of 1620 for WikiMatrix.The best CCMatrix pair reaches 45.2 BLEU, versus 37.3 BLEU for WikiMatrix’s best pair.
  • 6.2 WMT’19 evaluation: NMT systems trained on CCMatrix outperform published single models trained on WMT’19 bitext data for the evaluated directions.The comparison covers En-De, En-Ru, En-Zh, and De-Fr high-resource directions.
  • 6.2 WMT’19 evaluation: Combining CCMatrix with WMT’19 bitext improves En-De performance by 0.8 BLEU on average over CCMatrix-only training.The combined system reaches 50.9 BLEU on newstest2018 and 45.1 BLEU on newstest2019.
  • 6.3 WAT’19 evaluation: CCMatrix-only Russian/Japanese training uses 9.3 million mined sentence pairs and outperforms the best WAT’19 system, especially into Japanese.The authors caution that the comparison is not direct because WAT participants were restricted to provided resources.

7 Conclusion

The paper concludes that margin-based mining in a shared multilingual sentence space scales to tens of billions of monolingual sentences without document alignment. The resulting bitexts are large and support strong NMT performance across multiple evaluations, while future work targets broader coverage and more efficient use.

  • Conclusion: Margin-based multilingual sentence mining scales to more than 36 billion unique sentences across 38 languages without document alignment.The processing pipeline parallelizes the computationally expensive steps.
  • Conclusion: The procedure yields 661 million English-aligned sentences and 4.5 billion pairwise-aligned sentences across 28 languages.The authors describe this as the largest collection of high-quality parallel sentences to their knowledge.
  • Conclusion: Training only on mined data outperforms the best reported single WMT’19 NMT systems for several English and German/French directions and reaches state-of-the-art BLEU for Russian/Japanese on WAT’19.The evaluation spans many language pairs and uses NMT systems trained on the extracted bitexts.
  • Conclusion: Future CCMatrix versions will add Common Crawl snapshots and focus on low-resource languages, while efficient training on more than fifty million bitexts remains an open question.The authors also suggest using the mined data to improve multilingual sentence embeddings.
Loading 1911.04944v2…