Source-linked AI summary
CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzman, Philipp Koehn
TL;DR
Cross-lingual document alignment is costly, especially for low-resource languages, yet high-quality aligned data is needed across multilingual NLP tasks. The paper uses URL-based weak supervision to mine Common Crawl and releases a large aligned-document dataset, alongside content-based baselines and a machine-translation case study. The resulting resource supports broad cross-lingual research, while its small manual evaluation does not establish the quality of the full dataset.
Problem
Manual cross-lingual document alignment is costly and difficult for low-resource languages, despite the need for high-quality data across multilingual NLP tasks.
Method
The paper applies high-precision URL-matching rules to Common Crawl and introduces a content-only cross-lingual document-alignment baseline.
Results
The released dataset contains over 392 million document pairs across 8144 language pairs, and the aligned documents are mined for downstream machine translation data.
Takeaways & Limitations
The dataset provides a broad benchmark and parallel-document resource for mining parallel sentences and supporting cross-lingual NLP research.
Takeaways & Limitations
The manual evaluation is small and does not establish the quality of the full dataset; the high-precision collection may require higher-recall expansion.
Abstract
from arXiv · showhide
Cross-lingual document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. In this paper, we exploit the signals embedded in URLs to label web documents at scale with an average precision of 94.5% across different language pairs. We mine sixty-eight snapshots of the Common Crawl corpus and identify web document pairs that are translations of each other. We release a new web dataset consisting of over 392 million URL pairs from Common Crawl covering documents in 8144 language pairs of which 137 pairs include English. In addition to curating this massive dataset, we introduce baseline methods that leverage cross-lingual representations to identify aligned documents based on their textual content. Finally, we demonstrate the value of this parallel documents dataset through a downstream task of mining parallel sentences and measuring the quality of machine translations from models trained on this mined data. Our objective in releasing this dataset is to foster new research in cross-lingual NLP across a variety of low, medium, and high-resource languages.
1 Introduction
The paper addresses costly cross-lingual document alignment by using URL signals for weak supervision at web scale. It releases a massive aligned-document dataset and demonstrates its use for content-based alignment and downstream machine translation.
- Manual alignment is costly and difficult for low-resource languages because document-pair search is quadratic and qualified annotators are scarce.
- URL signals provide weak supervision for identifying translated web documents across languages.
- The released dataset contains over 392 million aligned document pairs spanning 8144 language pairs, including 292 million non-English pairs and 100 million involving English.
- A simple content-only baseline aligns cross-lingual documents even when detractor documents lack parallel counterparts.
- The aligned documents can be mined for parallel sentences to train downstream sequence-to-sequence machine translation systems.
- The dataset is intended as a benchmark, parallel corpus, and source of supervision for cross-lingual NLP tasks.
2 Related Works
Prior work mined parallel data from homogeneous sources, web metadata, document structure, and neural embeddings. These approaches often relied on narrow settings or noisy signals, motivating broader web-scale alignment.
- Earlier parallel corpora were curated from specific homogeneous sources using content-based, domain-specific alignment rules.
- Metadata-based methods used publication dates and temporal heuristics, but temporal features can be sparse, noisy, and unreliable.
- Other approaches relied on document structure for bilingual document alignment.
- The WMT-2016 bilingual document-alignment work considered only English-to-French, a high-resource direction.
- Neural embedding methods have been explored for bilingual sentence- and document-level alignment, including hierarchical document embeddings.
3 Dataset Creation and Description
The dataset is built from 68 Common Crawl snapshots by deduplicating URLs, identifying document languages, and matching language-marked URLs with high-precision rules. The resulting collection supports tractable alignment benchmarking.
- Common Crawl provides monthly web snapshots with over 2 billion pages each, and this work uses 68 snapshots from 2013 to 2020.
- URL preprocessing deduplicates documents by normalizing URLs and ensuring each URL appears once in the cleaned data.
- A lightweight text classifier assigns each document its dominant language identifier.
- The matching heuristic treats URLs as translations when stripping language identifiers yields the same string, while allowing one-sided identifiers and validating predicted document language.
- The released corpus contains over 392 million aligned documents, with 100 million involving English and 292 million without English, mined across 68 snapshots.
- The benchmark subset contains 121K documents from 450 web domains, including 17.5K URL-aligned pairs.
4 Dataset Evaluation
Human evaluation measures whether URL-aligned documents are comparable or translations across languages and resource levels. Most measured directions achieve precision above 90%, while identified errors include temporal changes, partial translations, and noncomparability.
- The evaluation estimates URL-alignment precision using human judgments of representative document pairs.
- Twelve bilingual or trilingual annotators evaluated pairs side by side, with three annotators assigned to each pair.
- Table 2 reports majority-vote precision Pmaj, adjusted precision Padj, and Krippendorff alpha Kα across high-, medium-, and low-resource languages.
- 40% of identified errors involved changed pages or redirects, 20% shorter partial documents, 10% untranslated dynamic content, and 30% truly noncomparable documents.
- A majority of measured language directions achieved precision above 90% under human evaluation.
5 Document Alignment Experiments
The experiments evaluate content-based cross-lingual document alignment using LASER-derived representations, sentence weighting, cosine similarity, and greedy matching. Sentence-level importance weighting, especially SLIDF, substantially improves recall over simpler document representations across resource levels.
- Experimental Setup: The evaluation treats URL-aligned documents as a high-precision ground truth for testing content-based alignment baselines.The baselines score document pairs from source and target sets and align them with greedy bipartite matching.
- Embedding Methods: Direct Embedding applies LASER to full documents, while Sentence Average embeds individual sentences and averages their vectors.Sentence Average decomposes each document into sentences before constructing its document representation.
- Embedding Methods: Weighted Average scales sentence vectors by importance weights before averaging, using schemes inspired by tf-idf.The investigated schemes include sentence length and inverse document frequency, either separately or in combination.
- Scoring and Matching: Cosine similarity scores every source-target document pair, producing a fully connected bipartite graph within each web domain.The alignment objective assumes each page in the non-dominant language has a translated or comparable counterpart, yielding min(|Ds|, |Dt|) expected pairs.
- Scoring and Matching: Competitive Matching greedily selects the highest-scoring unused pair until min(|Ds|, |Dt|) pairs are chosen, with runtime O(|Ds||Dt| × log(|Ds||Dt|)).This avoids the intractable Hungarian-algorithm complexity reported for moderately sized web domains.
- Results: SLIDF improves alignment recall over Sentence Average by 28%, 57%, and 20% for low-, mid-, and high-resource language pairs.Sentence Length alone improves recall by 17%, 39%, and 10%, while inverse-document-frequency weighting improves it by 7%, 39%, and 15%.
- Results: Alignment performance increases with resource availability, while European languages appear consistently easier to align than non-European languages.Albanian, Serbian, Bosnian, and Belarusian each achieved over 0.50 recall despite being low-resource.
6 Case Study: MT as an Application
The case study evaluates CCAligned by mining parallel sentences from aligned documents and training multilingual translation models. The mined data produces comparable BLEU scores to other web-mined corpora, while substantially expanding language coverage beyond ParaCrawl.
- Limitations: The manual evaluation is small and does not establish the quality of the full aligned dataset.The authors distinguish evaluation of the high-precision alignment method from evaluation of the complete corpus.
- Sentence Mining: CCAligned documents are decomposed into sentences, aligned within document pairs, and aggregated into parallel-sentence training data.LASER with margin-based filtering is used to mine sentences before NMT training.
- Experimental Setup: Over 1B unique parallel sentences are mined from 100M English-including document pairs across 137 language pairs.The resulting translation models are evaluated on TED Talk test sets.
- Results: Comparable BLEU scores are obtained from CCAligned bitexts and independently mined Wikipedia and ParaCrawl corpora under constant NMT conditions.The comparison uses TED Talk test sets and includes the three web sources described in Table 4.
- Results: The results do not establish superiority among mining methods because the corpora differ in sentence counts and cleaning procedures.They nevertheless indicate that CCAligned document pairs are a valuable source of parallel data.
- Coverage: CCAligned covers 8144 language pairs, compared with ParaCrawl’s 23 European language pairs paired with English, including additional non-European pairs.This creates potential coverage for low-resource directions unavailable in ParaCrawl.
7 Conclusion & Future Works
The paper concludes that URL-aligned Common Crawl documents form a large, high-precision resource for document alignment and parallel-sentence mining. It identifies improved mining, better representations, and higher-recall alignment as directions for future work.
- Conclusion: CCAligned contains over 392 million document pairs across 8144 language pairs and 138 languages, curated from Common Crawl using URL matching.The paper also evaluates URL-aligned pairs with human annotators and introduces embedding-based content-alignment baselines.
- Conclusion: The aligned documents can be mined for parallel sentences suitable for training machine translation models, with potential value for low-resource language pairs.The stated potential follows from the dataset’s size and coverage.
- Future Work: Future work includes better parallel-sentence mining, especially for low-resource pairs, and improved cross-lingual document representations.The authors also propose using supervision from the high-precision dataset to obtain a larger, high-recall collection.