Source-linked AI summary
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, Edouard Grave
TL;DR
Large pretraining corpora require quality control, but curated resources are limited for many languages. CCNet presents a scalable pipeline that deduplicates and identifies Common Crawl documents, then filters them by similarity to high-quality corpora such as Wikipedia. The resulting corpus covers more than 100 languages, while the pipeline is publicly available for reproduction and supports multilingual representation training.
Problem
High-quality pretraining data is difficult to scale across languages because low-resource languages have much smaller curated corpora.
Method
CCNet downloads and processes Common Crawl, deduplicates paragraphs, identifies languages, and uses Wikipedia-trained language-model perplexity to filter documents.
Results
The pipeline produces curated monolingual corpora in more than 100 languages from Common Crawl.
Takeaways & Limitations
The resulting datasets support training multilingual text representations such as XLM, and the pipeline and tools are publicly available.
Abstract
from arXiv · showhide
Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia.
1. Introduction
The paper addresses the need for large, high-quality pretraining corpora across languages, especially where curated resources are small. It presents a general pipeline for extracting massive monolingual datasets from Common Crawl and makes the pipeline and tools publicly available.
- Curated datasets such as Wikipedia are difficult to replicate for low-resource languages because those languages often have much smaller resources.
- The proposed pipeline gathers massive monolingual corpora from Common Crawl while preserving data quality across many languages.The authors apply the pipeline to Common Crawl, a massive non-curated web dataset.
- The pipeline follows Common Crawl processing with corpus filtering designed to select documents close to high-quality sources such as Wikipedia.
- The pipeline and supporting tools are publicly available for reproducing the processing and tailoring the resulting corpora.
2. Related work
The work builds on earlier large-scale text preprocessing pipelines, especially fastText-based language identification for splitting Common Crawl into monolingual datasets.
- The pipeline follows the fastText approach of using a language identifier to split Common Crawl into monolingual datasets.
- Earlier preprocessing of massive datasets supported word-embedding systems including word2vec, GloVe, and fastText.
3. Methodology
The methodology processes Common Crawl snapshots by sharding and deduplicating webpages, identifying their languages, and scoring document quality with language-model perplexity. It preserves document structure and supports parallel processing and reproducible reconstruction.
- Pipeline overview: The pipeline downloads WET files, computes paragraph hashes, processes documents independently, and regroups outputs by language and perplexity score.Several indicated pipeline steps are parallelisable.
- Common Crawl corpora: Common Crawl snapshots contain 20–30TB of uncompressed text and are divided into 5GB shards for independent processing.The February 2019 snapshot contained 24TB and 1600 shards.
- Deduplication: Deduplication normalizes paragraphs, hashes them with the first 64 bits of SHA-1, and compares hashes across selected shards to remove repeated content.The process also removes boilerplate such as navigation menus and cookie warnings.
- Language identification: The fastText language classifier assigns each webpage its most probable language and a score, supporting 176 languages.It uses character n-grams and hierarchical softmax and processes 1k documents per second on one CPU core.
- LM filtering: For quality filtering, the pipeline uses language-model perplexity trained on data from a targeted domain such as Wikipedia.A tokenizer and language model are trained for each language.
- Dataset statistics: The resulting token counts by language are reported after deduplication, with a logarithmic-scale histogram for the February 2019 snapshot.
- Reproducibility: A reconstruction tool uses webpage URLs to reproduce the pipeline’s final output without rerunning the full crawl collection.
4. Ablation study
The ablations examine how deduplication scope and ordering affect data yield, resource use, and processing speed. Deduplicating before language identification improves low-resource language coverage, while broader deduplication removes more data at higher memory cost.
- 4.1. Order of LID and deduplication steps: Deduplicating before language identification increases the relative number of documents found for many low-resource languages.Removing English boilerplate from pages in other languages makes language identification more robust.
- 4.2. Impact of the amount of deduplication: 42% of characters remain after deduplication across one shard, compared with 28% across 100 shards.Broader comparison removes more duplicate content.
- 4.2. Impact of the amount of deduplication: Loading hashes from 50 shards requires 13.5GB on disk and 40GB of RAM, leading the authors to choose 50 shards as a practical tradeoff.The selected configuration runs deduplication on blocks corresponding to 3% of the corpus.
- 4.3. Benchmarking: The pipeline remains two-step because deduplication must compare billions of document paragraphs, despite extensive parallelization.The implementation uses 5GB shards as the smallest parallelization unit.
- 4.3. Benchmarking: The full pipeline takes about 9 hours using 5000 CPU cores for one Common Crawl snapshot.The language identifier uses 12.5% of CPU time, SentencePiece 33%, and the language model 13%.
5. Metrics about the resulting dataset
The resulting corpus spans many languages and provides substantial resources, including for lower-resource languages. Experiments show that perplexity-based filtering improves representation quality and that CommonCrawl pretraining outperforms Wikipedia in the reported BERT comparison.
- Corpus scale: Common Crawl provides more data than Wikipedia for Afrikaans, Gujarati, Khmer, and Burmese, supporting lower-resource languages.The reported Common Crawl sizes are 160MB, 190MB, 154MB, and 440MB, respectively, versus 103MB, 88MB, 71MB, and 153MB from Wikipedia.
- Filtering quality: Perplexity is used as a relative proxy for document quality, with language-specific thresholds dividing each corpus into three equal-sized parts.The authors retain high-perplexity content because it may be useful for specific applications.
- Representation quality: FastText embeddings show a steady performance increase from the tail to the head of datasets sorted by document perplexity.The evaluation covers semantic and syntactic analogy datasets for English and Polish CommonCrawl corpora.
- Representation quality: BERT-BASE models trained on CommonCrawl outperform identical Wikipedia-trained models by 3.3% on average on XNLI.The comparison covers English, Russian, Chinese, and Urdu under matched training-step conditions.
6. Conclusion
The paper presents a pipeline for creating curated monolingual corpora in more than 100 languages. It combines Common Crawl preprocessing with document-structure preservation and Wikipedia-distance filtering to improve dataset quality and support multilingual representations.
- Conclusion: The pipeline creates curated monolingual corpora in more than 100 languages from Common Crawl.It follows the fastText preprocessing pipeline while preserving document structure.
- Conclusion: Filtering data by distance to Wikipedia improves the quality of the resulting dataset.The curated corpora support training multilingual text-level representations such as XLM.