Source-linked AI summary
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, Benoît Sagot
TL;DR
Large language modeling has increased demand for large corpora, which are still obtained primarily through automatic web crawling despite concerns about their quality. This paper improves OSCAR and Ungoliant by producing a document-oriented corpus with line filtering and automatic annotations, while documenting hard-threshold limitations.
Problem
Large language models require large corpora, but obtaining them still relies primarily on automatically crawled web text amid concerns about corpus quality.
Method
The paper creates a document-oriented OSCAR corpus by inferring document-level language identification from line identifications, preserving contiguous lines, and adding quality-related annotations.
Results
The proposed release provides a corpus comparable in total and language-distribution size with OSCAR 21.09, together with document-preserving filtering and quality-related annotations.
Takeaways & Limitations
The annotations support filtering and querying documents by quality-related properties such as length, while the document structure is intended to remain human readable and exploitable.
Takeaways & Limitations
Hard thresholds can place documents near cutoff values into categories whose quality is similar to documents assigned to neighboring categories.
Abstract
from arXiv · showhide
The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.
1. Introduction
The paper introduces a document-oriented OSCAR corpus and pipeline that preserves document structure while adding annotations for quality-aware filtering. It releases comparable-scale multilingual data, including a 12GB corpus and deduplicated English resources, while acknowledging unresolved concerns.
- The pipeline shifts OSCAR and Ungoliant from line-level to document-oriented language classification.The change addresses existing concerns with line-level processing by treating documents as the primary unit.
- The release is comparable in total size and language distribution to OSCAR 21.09 and includes a 12GB multilingual corpus.
- A line-filtering method preserves contiguous document lines to limit integrity destruction and retain human readability.
- Automatic annotations support quality filtering and queries for documents meeting specified length criteria.
- The project also provides a deduplicated English corpus and a line-deduplication tool.
- The authors state that the improvements do not address all concerns raised about OSCAR and large web-derived corpora.
2. Related Work
Prior multilingual corpora derived from Common Crawl commonly classified text at the line level, while later pipelines introduced document-level language classification and filtering methods. These systems differ in how they identify and remove undesirable data.
- Common Crawl has been widely used to pre-train language representations and large generative language models.
- Early Common Crawl language-classification pipelines classified entries at line level using FastText.
- CCNet and mC4 use document-level language classification rather than OSCAR’s line-level classification.
- CCNet filters undesired data with small Wikipedia-trained language models, whereas mC4 uses a simple badword filter.
3. Filtering
The filtering design treats documents as whole units while selectively removing short-line sequences at document boundaries. It combines contiguous boundary filtering with a count-based safeguard, but remains vulnerable to adversarial long lines and hard thresholds.
- Document-oriented filtering must preserve document integrity and account for the document as a whole.
- The corpus targets OSCAR 21.09-like size and quality while limiting short, noisy lines and retaining data for low- and mid-resource languages.
- Boundary filtering removes contiguous sequences of short lines only at the head or tail of a document.
- Adding a long line at both document boundaries can negate the benefits of the boundary filter.
- A count-based filter discards documents when the short-line bin exceeds the long-line bin.
4. Identification
The document-oriented pipeline infers document-level language identities from line-level FastText predictions, accounting for line size and confidence. It also applies cautious criteria to identify multilingual documents and uses thresholds for selecting a dominant language when the multilingual test fails.
- Document representation: Each document is represented by lines carrying a predicted language, confidence, and byte size, with unidentified lines assigned a special language token.Document size is computed from the sizes of its constituent lines.
- Document representation: For each language, the pipeline computes its document size contribution and an overall confidence weighted across the document.These quantities support document-level language identification from line-level outputs.
- Multilingual identification: Multilingual identification is motivated by distinguishing balanced multilingual content from interface-language mismatches and translation pages.The pipeline aims to limit interface/body language mismatches while retaining documents containing the same content in multiple languages.
- Multilingual identification: The multilingual check is restricted to documents with at least 5 lines and at most 5 different languages.These restrictions form the cautious scope of the multilingual identification test.
- Fallback identification: For documents that fail the multilingual check, the largest represented language is selected using a minimum overall-confidence threshold of 0.6.The threshold is lower than the 0.8 threshold used earlier in the line-identification process.
5. Annotation
The corpus introduces non-destructive annotations so users can apply more precise downstream quality filters. These annotations flag length irregularities, noisy content, and material associated with adult-content blocklists.
- Annotation framework: Annotations preserve documents while enabling more precise downstream filtering and rapid corpus-quality assessment.A document may carry several annotations simultaneously, and annotations can support more aggressive subsequent filters.
- Length-based annotations: A length annotation flags documents in which at least 50% of lines are short.This annotation helps identify potentially tiny documents whose line structure or size may affect training tasks.
- Length-based annotations: Header and footer annotations flag short lines concentrated at the document's beginning or end, where low-quality content may occur.The annotations target possible boilerplate or other undesirable boundary content.
- Noise filtering: A noisy annotation is added when a Unicode-category-based ratio of letters to non-letters indicates potentially non-linguistic content.The filter is intended to catch material such as source code appearing in language corpora because it contains words from that language.
- Adult-content filtering: Adult-content filtering uses the UT1 blocklist, which combines thematic URL and domain lists and contains roughly 3.7M records.The blocklist covers adult-content websites and user-generated-content sites that may contain adult material.
6. Corpus
OSCAR 22.01 is a document-oriented corpus built from the November/December 2021 CommonCrawl dump and compared with OSCAR 21.09 for size, topics, and annotations. Overall corpus-size distributions are similar, but document-level identification creates substantial changes for some low-resource languages.
- OSCAR 22.01 applies the pipeline to the November/December 2021 CommonCrawl dump and evaluates size, news-related topic presence and recall, and annotation pertinence.
- The overall corpus-size distribution is similar between OSCAR 21.09 and OSCAR 22.01, although the lower end changes substantially.
- Document-level identification can substantially reduce low-resource corpus sizes when minority-language lines occur inside documents dominated by another language.
- The Alemannic German corpus decreased from 7MB to 360KB, while extracting Alemannic-identified lines from the German corpus yielded around 30MB.
- Mixed German–Alemannic documents can represent identification errors, language mixing, or documents whose majority language was misclassified.
- The new mixed-language cases could support improved language detection after identifying mismatches or verifying documents with speakers.
6.2. Annotations
The annotations expose document-quality patterns across resource levels, multilingual language pairings, adult-content labels, and clean-document distributions. Hard thresholds also create a concentration of documents at the minimum accepted size.
- Annotation distributions: Low-resource languages have a higher proportion of documents annotated with short sentences and tiny than mid- and high-resource languages.Annotation distributions are broadly similar across resource groups, but the heat map shows a much larger tiny proportion for low-resource languages.
- Multilingual documents: The multilingual corpus contains documents with two to five languages, and language tuples with fewer than 20,000 occurrences are omitted from the displayed counts.Tuple co-occurrences may reflect linguistic similarity, classification errors, or languages commonly appearing together.
- Clean documents: Clean documents in an English sample of 183,497 documents have slightly shorter mean lengths than annotated documents.Clean documents also appear to have shorter length standard deviations, although the computation yields larger values in the reported comparison.
- Adult documents: The French sample contains 32,870 adult-labeled documents out of 52,037,098, including 1,063 documents from tetu.com.The tetu.com documents represent approximately 3.2% of adult documents in the sample.
- Adult documents: Among examined adult annotations, true positives include explicit sexual content, while false positives include documents that do not meet the criterion and educational documents.Two false positives belonged to LGBTQI+ websites, and many false positives were websites that apparently should not have been blocklisted.
- Hard bounds problems: Hard thresholds label documents with fewer than 5 lines as tiny, producing many tiny documents at the ≥102B minimum accepted size.The minimum arises because the first filter removes lines shorter than 100 characters.
7. Discussion
OSCAR 22.01 preserves a corpus size comparable to OSCAR 21.09 and improves document-level filtering through annotations, but low-resource corpus sizes can decrease substantially. The discussion also identifies unresolved annotation and filtering caveats requiring future work.
- Discussion: OSCAR 22.01 is document-oriented, comparable in total size to OSCAR 21.09, and easier to filter because it includes annotations.Mid- and high-resource languages remain similar in size, while several low-resource languages shrink substantially.
- Discussion: Language-corpus merging can obscure low-resource languages, such as approximately 30MB of Alemannic data held within the German corpus.Investigating these mergers may help explain identification mismatches and improve language-identification models.
- Annotations: Length-based annotations are widespread, reaching approximately 50% in Czech, while adult blocklist filtering may perform poorly and can mislabel LGBTQI+-related websites.The authors identify these annotation caveats as targets for future OSCAR-generation pipelines.
A. Carbon Footprint
The OSCAR generation’s energy use is estimated from CPU and DRAM power, adjusted by data-center overhead, and converted into regional carbon emissions. The reported generation took 42.6 hours and emitted 16.48 g of CO2.
- 42.6 hours was the total time to generate OSCAR 22.01 on the reported infrastructure.The infrastructure used a single machine with 192 GB of RAM and two Intel Xeon Gold 5218 processors.
- 0.4266 kWh was the reported total power consumption for OSCAR generation.The estimate adds CPU and DRAM consumption and multiplies the result by a PUE coefficient of 1.58.
- 38.64 g/kWh was the regional average CO2 emission factor during the generation period.The authors use this factor to convert total power consumption into kilograms of CO2 emissions.
- 16.48 g was the reported total CO2 emission for generating one model.The paper also expresses this amount as 0.01648 kg.
B. Language Table
Table 2 reports OSCAR corpus size by language using bytes and estimated word counts. Spaced-token word counts are more reliable for Latin- and Cyrillic-alphabet languages than for Chinese or Japanese.
- Table 2 measures each language’s OSCAR corpus in bytes and number of words.The table uses standard UNIX human-readable notation for byte sizes and defines words as space-separated tokens.
- Spaced-token word counts may misrepresent corpus size for Chinese or Japanese.The paper describes this word definition as a good estimate for languages using Latin or Cyrillic alphabets.