PaperlayerView exact revision on arXiv

Source-linked AI summary

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, Thomas Wolf

arXiv:2506.20920v1cs.CL

TL;DR

Multilingual pre-training data curation remains difficult because low-resource languages lack well-tuned, language-specific filtering and deduplication pipelines. FineWeb2 introduces an automatically adaptable pipeline and produces more performant models than prior multilingual datasets for 11 of 14 evaluated languages.

  • Problem

    Low-resource languages lack well-tuned pipelines for corpus evaluation, language identification, filtering, and deduplication, limiting multilingual pre-training data curation.

    Source: §1 · pp. 1–2

  • Method

    The paper develops a data-driven FineWeb-based pipeline that adapts to language-specific statistics, validates choices across nine languages, and scales to multilingual dataset construction.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    11 out of 14 languages achieved more performant models with FineWeb2 than with prior multilingual datasets.

    Source: §5 · p. 9

  • Takeaways & Limitations

    FineWeb2 supports a consistent-but-adaptable approach to multilingual corpus curation across 1,868 language-script pairs and 20 terabytes of text.

    Source: §5 · p. 9, §6 · p. 10

    Evaluation covered only a small proportion of FineWeb2 languages and used relatively short ablation runs because of computational, task-availability, and low-resource-data constraints.

    Source: §6 · p. 10

Abstract

from arXiv · show

Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-training datasets has seen substantial recent progress, training performant multilingual LLMs remains a challenge, in large part due to the inherent difficulty of tailoring filtering and deduplication pipelines to a large number of languages. In this work, we introduce a new pre-training dataset curation pipeline based on FineWeb that can be automatically adapted to support any language. We extensively ablate our pipeline design choices on a set of nine diverse languages, guided by a set of meaningful and informative evaluation tasks that were chosen through a novel selection process based on measurable criteria. Ultimately, we show that our pipeline can be used to create non-English corpora that produce more performant models than prior datasets. We additionally introduce a straightforward and principled approach to rebalance datasets that takes into consideration both duplication count and quality, providing an additional performance uplift. Finally, we scale our pipeline to over 1000 languages using almost 100 Common Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document) multilingual dataset which we release along with our pipeline, training, and evaluation codebases.

1 Introduction

Multilingual LLM development is constrained by uneven web data availability and the difficulty of adapting curation pipelines across languages. This work introduces an automatically adaptable, data-driven pipeline and uses it to create FineWeb2, a 20-terabyte dataset spanning over 1,000 languages.

Motivation

  • Low-resource languages lack the data-processing infrastructure available for high-resource languages, while fixed multilingual pipelines risk inappropriate filtering and language-specific performance losses.Challenges include corpus-quality evaluation, language identification, filtering customization, and word segmentation; hand-designing a pipeline per language does not scale.Source: §1 · pp. 1–2

Contributions

  • The proposed FineWeb-based pipeline automatically adapts to language-specific statistics and is validated through ablations on nine diverse languages using tasks selected by measurable criteria.The study also introduces duplication- and quality-aware rebalancing and tests the resulting corpora on additional languages unseen during pipeline design.Source: §1 · p. 2

FineWeb2

  • FineWeb2 contains 20 terabytes and 5 billion documents across over 1,000 languages, processed from almost 100 Common Crawl snapshots spanning summer 2013 to April 2024.The dataset, pipeline, training code, evaluation code, and a preliminary post-deduplication dataset are released under the permissive ODC-By License.Source: §1 · p. 2

2 Preliminaries

The paper establishes multilingual preprocessing foundations by defining language/script identifiers and adapting word tokenization across languages, including those without native tokenizers.

Notation

  • Languages are identified by ISO-639-3 codes and, when needed, by language–script pairs using ISO 15924 codes.This distinguishes scripts such as Standard Arabic in Arabic versus Latin script.Source: §2 · p. 2

Separating words

  • Language- and script-specific tokenizers are required because many writing systems lack visible word boundaries or use nonstandard boundary markers.This especially affects Southeast Asian languages, Chinese, Japanese, and Korean.Source: §2 · pp. 2–3
  • The pipeline combines SpaCy, Stanza, and language-specific tokenizers, assigning proxy tokenizers from linguistically close languages when native ones are unavailable.Assignments use language-family information from Ethnologue3 and support filtering, deduplication, and evaluation across thousands of languages.Source: §2 · p. 3

3 Experimental setup

The experiments isolate data-processing effects with controlled language-specific model comparisons across nine canary languages, using a tokenizer and architecture selected for multilingual coverage. Evaluation relies on quantitatively selected early-signal tasks and a category-balanced aggregate score.

3 Experimental setup

  • Each processing step is tested with identical models differing only in whether that step is applied, isolating its downstream effect while holding model and training variables constant.The controlled variables include parameter count, architecture, tokenizer, and training token count.Source: §3 · p. 3
  • Experiments train separate models for Arabic, Chinese, French, Hindi, Russian, Swahili, Telugu, Thai, and Turkish to span varied scripts, language families, and resource levels without cross-language confounders.Every ablation or validation run therefore uses nine language-specific models.Source: §3 · p. 3

3.1 Tokenizer and model architecture

  • The Gemma tokenizer was selected because it showed better average fertility than similarly sized alternatives across the nine canary languages, while models follow a reduced-layer Llama-style architecture with 1.46 billion parameters.The models use 14 layers, 32 attention heads, length-2048 sequences, and tied embeddings.Source: §3.1 · pp. 3–4

3.2 Baseline datasets

  • Baseline comparisons use language-specific models trained on CC-100, mC4, CulturaX, HPLT, and minimally processed Common Crawl data.Data availability limits comparisons for Telugu, Swahili, and Hindi, with only CulturaX and HPLT supporting Hindi training at 30 billion tokens without excessive repetition.Source: §3.2 · p. 4

3.3 Selecting evaluation (Fine)tasks

  • The final evaluation suite contains 84 benchmarks selected from 197 tested across nine canary languages using quantitative early-signal criteria.The criteria require monotonic improvement, low noise, non-random early performance, and consistent model ordering.Source: §3.3 · p. 4
  • Aggregate scores rescale performance relative to random baselines, average within reading-comprehension, general-knowledge, language-understanding, and common-sense categories, then macro-average the categories.Scores below the random baseline are set to 0 before aggregation.Source: §3.3 · p. 4

4 The FineWeb2 pipeline

FineWeb2 adapts FineWeb’s processing pipeline to multilingual Common Crawl data through language-aware identification, deduplication, filtering, and rehydration. The resulting choices improve downstream quality across languages while scaling to many languages and data conditions.

4.1 Starting point: FineWeb

  • FineWeb2 starts from almost 100 Common Crawl snapshots, removes adult content with URL blocklisting, extracts HTML text with trafilatura, and retains the 60% of documents excluded by FineWeb’s English filter.FineWeb’s FastText threshold identifies approximately 40% of documents as English, leaving the remaining 60% as FineWeb2’s starting point.Source: §4.1 · p. 5

4.2 Language Identification (LID)

  • GlotLID outperforms FT176 on higher-resource languages but slightly trails it on lower-resource languages; FineWeb2 adopts GlotLID because its broader language and script coverage outweighs this difference.GlotLID also includes labels for unsupported scripts and common noise formats, reducing their misclassification as other languages.Source: §4.2 · p. 5

4.3 Deduplication

  • Deduplication generally improves model performance, although its impact varies substantially by language and shows no discernible relationship with resource level.The pipeline performs global MinHash deduplication before filtering so filtering experiments directly reveal final dataset performance, and it records cluster sizes for later rehydration.Source: §4.3 · p. 6

4.4 Filtering recipe

  • The filtering recipe automatically adapts language-specific thresholds using statistics from Wikipedia, the GlotLID-Corpus, and language-filtered Common Crawl data, guided by experiments on nine canary languages.This addresses the language dependence of heuristic quality rules whose suitable thresholds differ across languages.Source: §4.4 · p. 6

4.4.1 Stopwords

  • Stopword filtering derives language-specific stopwords from reference-corpus word frequencies, excludes non-alphabetic artifacts, and requires at least 2 stopwords per document.The procedure also accounts for English contamination in some reference corpora, particularly Wikipedia.Source: §4.4.1 · p. 7

4.4.2 Filtering threshold selection

  • The selected filtering thresholds use 10Tail for FineWeb Quality, Wikipedia-based Quantile for Gopher Quality, and Common-Crawl-based MeanStd for Gopher Repetition, noticeably improving performance for all languages.These choices were selected after 207 ablation models trained for 29B tokens each, comparing threshold methods across nine languages and three filter groups.Source: §4.4.2 · p. 7

4.4.3 Precision filtering lower-resource languages

  • For roughly a third of 1900 languages with contamination above 10%, high-affinity wordlists and language-related URL terms remove false positives, yielding almost 30% precision improvements for some audited low-resource languages.The additional filtering step targets low-resource languages affected by false positives from related high-resource languages.Source: §4.4.3 · p. 8

4.5 Rehydration

  • Rehydration generally provides a strong performance uplift by assigning duplication-aware weights from filtering rates, with the lowest-removal cluster size receiving weight 10 and clusters above the global removal rate receiving weight 1.Filtering-rate patterns form a broadly recurring U shape across languages, indicating that both unique and extremely duplicated data can be lower quality.Source: §4.5 · p. 9

5 Validating and Applying the FineWeb2 Pipeline

FineWeb2 scales its adaptive curation pipeline to 1,868 language-script pairs across 20 TB of multilingual text and outperforms prior multilingual datasets on most evaluated languages. However, low-resource corpora often have limited source diversity, and language-specific pipelines can still outperform the adaptive approach.

Creating the FineWeb2 dataset

  • The pipeline applied to 96 Common Crawl snapshots yields 20 TB covering 1,868 language-script pairs, with filtered and preliminary datasets released.FineWeb2 excludes English, so the authors recommend complementing it with FineWeb for full language coverage.Source: §5 · p. 9

Comparison to other datasets

  • Evaluation on German, Indonesian, Italian, Japanese, and Vietnamese tests whether the pipeline generalizes beyond the canary languages used for ablations.These unseen languages were selected using reliable evaluation tasks, while models used the same architecture, hyperparameters, and Gemma tokenizer as earlier experiments.Source: §5 · p. 9
  • FineWeb2 produces more performant models than prior multilingual datasets on 11 of 14 evaluated languages, across both canary and unseen languages.Language-specific datasets sometimes perform better, showing that expert-designed pipelines can outperform the adaptive approach; Figure 3 summarizes the comparison.Source: §5 · p. 9, Figure 3 · p. 10

Inspecting low-resource corpora

  • 70% of the 1,868 language-script pairs have more than half their documents from Bible- or Wikipedia-related domains, revealing limited diversity in low-resource corpora.Manual inspection found that many low-resource datasets contain almost exclusively Bible and/or Wikipedia content, reflecting sparse online data and narrow language-identifier training data.Source: §5 · p. 10

6 Conclusion

The paper presents an automatically adaptable multilingual data-processing pipeline, demonstrates quality-aware upsampling, and releases FineWeb2, while acknowledging limits in language evaluation, ablation duration, task criteria, and low-resource content diversity.

Contributions

  • The pipeline automatically adapts to all languages, and its design choices were extensively ablated using quantitatively identified multilingual benchmarks covering 14 languages.This contrasts with prior work using fixed pipelines for each language.Source: §6 · p. 10
  • Leveraging duplication counts and filtering results to selectively upsample higher-quality content provides a performance uplift.Source: §6 · p. 10

Limitations

  • Evaluation covered only a small proportion of FineWeb2 languages and relied on relatively short ablation runs because of computational, task-availability, and low-resource-data constraints.Source: §6 · p. 10
  • Early-signal task properties may change during training, while the selected tasks omit criteria such as cultural alignment, bias, and diversity.Source: §6 · p. 10
  • Many low-resource languages in FineWeb2 consist almost or entirely of Bible- or Wikipedia-related content, motivating further dataset improvement.Source: §6 · p. 10

A Appendix · A.1 Word tokenizers for segmentation

The pipeline combines native word tokenizers with a language-family tree to assign proxy tokenizers across languages and scripts. It completes coverage with script-based fallback tokenizers for languages lacking suitable native tools.

A.1 Word tokenizers for segmentation

  • The pipeline uses SpaCy, Stanza, and language-specific libraries to provide native word tokenizers across languages and scripts.Examples include IndicNLP for Indic languages, PyThaiNLP for Thai, Kiwipiepy for Korean, and dedicated tools for Khmer, Lao, Tibetan, and Burmese.Source: §A.1 · p. 25
  • It builds a language-family tree from Ethnologue taxonomy to organize tokenizer assignment.This tree supports proxy assignment when a language lacks a native tokenizer.Source: §A.1 · p. 25
  • Native tokenizers are first assigned to each language–script pair where the available libraries provide one.The assignment is performed before propagating tokenizers through the family tree.Source: §A.1 · p. 25
  • An upward tree pass propagates one tokenizer per script to parent nodes, choosing the tokenizer from the child language with more available data.Propagation stops before the root because distant subfamilies can differ substantially.Source: §A.1 · p. 25
  • A downward pass assigns each language the propagated parent tokenizer as a proxy when one is available.This two-pass procedure enables tokenizer assignment beyond languages with native tools.Source: §A.1 · p. 25
  • For remaining languages, the pipeline uses SpaCy’s multilingual tokenizer for Latin or Cyrillic scripts and the highest-resource native tokenizer for each other script.These fallbacks extend tokenizer coverage to many languages without native tokenizers.Source: §A.1 · p. 25

Indo-European Language Family Tree

The Indo-European language-family tree assigns native or proxy word tokenizers to languages through resource-aware propagation within subfamilies and branches.

Indo-European Language Family Tree

  • Tokenizer assignments propagate to related languages based on available native tokenizers and resource level, while stopping before the root and across top-level subfamilies.Italian is propagated within Italo-Dalmatian, Spanish upward from the Western branch as the higher-resource option, and Latin retains its native tokenizer.Source: Figure 4 · p. 26

A.2 Canary Languages

Although FineWeb2 covers more than 1,000 languages, the authors conduct in-depth evaluations on a subset of nine canary languages characterized by their families, scripts, and resource availability.

A.2 Canary Languages

  • The in-depth evaluation subset comprises 9 canary languages, documented by language family, main script, and resource availability.Source: Table 1 · p. 27, §A.2 · p. 27

A.3 Multilingual tokenizers comparison

The study compares multilingual tokenizers using subword fertility and the proportion of continued words across Wikipedia languages. Gemma is selected because it slightly outperforms Bigscience-Bloom on average while avoiding coverage and encoding problems found in alternatives.

Evaluation metrics

  • The evaluation measured subword fertility and the proportion of continued words, both of which quantify how often tokenizers split words.Subword fertility is the average number of tokens per real text word, while continued-word proportion is the fraction encoded with at least two tokens.Source: §A.3 · p. 28

Evaluation setup

  • The study computed these metrics on individual words from each language’s Wikipedia using tokenizers from nine popular multilingual models.The evaluated tokenizers came from Mistral-7B-V3, Phi3, Llama3, Qwen2.5, mT5, Bigscience-Bloom, Command-R, Gemma, and XGLM.Source: §A.3 · p. 28
  • The comparison omitted tokenizers with vocabularies exceeding 256,000 entries because their embedding layers would reduce feasible depth for the approximately 1.5-billion-parameter models.This constraint was imposed to keep the experimental model scale computationally practical.Source: §A.3 · p. 28

Tokenizer selection

  • Gemma was selected as the tokenizer because it performed slightly better than Bigscience-Bloom on average in the experimental setup.Both metrics favored lower values; the comparison also reported average and worst-case performance across languages.Source: §A.3 · p. 28, Table 2 · p. 29
  • Tokenizers were excluded when they showed poor coverage for a canary language or had practical encoding limitations.Mistral-7B-V3, Phi3, Command-R, and Llama3 provided poor Telugu coverage; XGLM and mT5 did not preserve whitespaces and could emit [UNK], especially for Chinese.Source: §A.3 · p. 28

A.4 Model architecture and training

This section specifies the shared architecture configuration and the training and optimizer settings used at 29, 100, and 350 billion tokens.

Architecture configuration

  • The paper defines a common architecture configuration for all models.Source: Table 3 · p. 30

Training settings

  • Training settings are reported for three scales: 29, 100, and 350 billion tokens, with critical batch size computed for the two larger scales following DeepSeek-AI et al. (2024).Source: Table 4 · p. 30

Optimizer settings

  • For the 100- and 350-billion-token scales, training uses a constant learning rate until the final 20% of steps, enabling continued pretraining of the resulting models.The optimizer settings for these scales are computed following DeepSeek-AI et al. (2024).Source: Table 5 · p. 30

A.5 Evaluation details … A.13 Train-Test Split

This section details the criteria, metrics, and procedures used to evaluate and adapt the multilingual filtering pipeline, including analyses of low-resource languages, pipeline stages, contamination, audits, and dataset coverage. It also describes FineWeb2’s multilingual composition and its per-language train–test splits for sufficiently large languages.

A.5.1 Task selection criteria

  • Tasks were selected only when they met quantitative criteria for monotonicity, signal-to-noise, non-random performance, and model-ordering behavior.Ordering consistency was measured observationally because no reliable selection threshold could be determined; generative tasks were allowed a lower SNR requirement to ensure multilingual coverage.Source: §A.5.1 · pp. 31–32

A.5.2 Metrics and Formulation

  • Non-generative tasks use Cloze Formulation, token-count normalization, and PMI or overlapping-word F1 where these are less noisy than standard alternatives.Cloze Formulation replaces multiple-choice letter completion because the latter can perform randomly early in training.Source: §A.5.2 · pp. 32–33

A.5.3 List of selected evaluation tasks for canary languages

  • Tables 6–11 list selected Arabic, Chinese, French, Hindi, Russian, and Swahili tasks satisfying monotonicity, signal-to-noise ratio, non-randomness, and ordering-consistency conditions.Source: Table 6 · p. 34, Table 7 · p. 34, Table 8 · p. 35

A.6.1 Classifier choice

  • The pipeline uses GlotLID V3 because its broad coverage and script-aware labels support multilingual filtering and reduce misclassification of unsupported or noisy text.GlotLID distinguishes scripts, provides script-specific labels, includes an unsupported-script label, and identifies noise documents.Source: §A.6.1 · p. 37

A.6.2 Confidence Threshold

  • A mean-and-standard-deviation threshold formula selected values within the best-performing threshold range for every language except Chinese and Hindi.Thresholds were evaluated by training models on 30 billion filtered tokens across regular removal intervals and selected values of interest.Source: §A.6.2 · p. 38, Table 15 · p. 38

A.7.1 Stopwords

  • Stopword construction required cleaning reference Wikipedias after low-resource languages inherited English content, yielding over 99% Dagbani removal with the revised list.Cleaning removed notes and references, script-mismatched articles, and articles classified as English above 70% confidence.Source: §A.7.1 · p. 40

A.7.2 Filtering thresholds

  • Filtering thresholds retain fixed FineWeb and Gopher settings where possible, tune selected quality and repetition thresholds by adaptation method, and choose the best method per filter group.Tables 25 and 26 report average ranks and removal rates used to select the pipeline configuration.Source: §A.7.2 · p. 41, Table 25 · p. 41, Table 26 · p. 41

A.7.3 Precision filtering lower resource languages

  • For low-resource languages, crawl precision depends strongly on false-positive rates, so wordlist filtering retains tokens with language affinity above γ = 0.85.This addresses n-gram LID failures when closely related high-resource languages are misclassified as low-resource text.Source: §A.7.3 · p. 42
  • Around one-third of 1,900 languages with wordlists have contamination scores above 10%, although most language data is classified as in-language.Contamination is measured as the percentage of documents removed by wordlist filtering from 10,000 randomly sampled documents per language.Source: §A.7.3 · p. 42, Figure 6 · p. 43
  • Wordlist filtering with a URL whitelist improves precision while maintaining recall for glk Arab and bar Latn, but yields limited improvement for ary Arab.The ary Arab limitation is attributed to inadequate representation in the LID training data; stricter wordlist coverage could raise precision at the risk of recall.Source: §A.7.3 · p. 43

A.8 Improvement from each pipeline step

  • The pipeline-step analysis defines progressively enhanced variants combining LID, global MinHash deduplication, heuristic filtering, and rehydration.The variants are labeled LID, LID + D, LID + D + F, and FW2 (R), respectively.Source: §A.8 · p. 44, Table 29 · p. 44, Table 30 · p. 44

A.10.1 List of selected evaluation tasks for unseen languages

  • Unseen-language evaluation selects tasks for German, Indonesian, Italian, Japanese, and Vietnamese, with full results reported for each language.The selected-task tables include approximate standard deviations, while separate tables provide the corresponding full evaluation results.Source: Table 35 · p. 48, Table 36 · p. 48, Table 37 · p. 48

A.11 FineWeb2 language composition

  • FineWeb2’s language composition is presented as percentages of total UTF-8 bytes, alongside word, document, and disk-size statistics for its largest languages.The distribution covers languages and language families, with the largest-language statistics detailed in Table 45.Source: Figure 8 · p. 53, Table 45 · p. 54, §A.11 · p. 54
  • FineWeb2 totals 3,339,271,691,958 words, 5,018,505,566 documents, and 20.78TB.Source: §A.11 · p. 56

A.12 Bible and Wikipedia content

  • 70% of the 1,868 language-script pairs have more than half of their documents from Bible- or Wikipedia-related domains, mostly because of Bible content.This corresponds to 1,320 language-script pairs.Source: §A.12 · p. 57, Figure 9 · p. 58

A.13 Train-Test Split

  • The release provides separate train and test sets per language, with test data reserved for memorization or data-attribution research rather than training.For sufficiently large languages, the test set is a hash-selected random subset containing min{1%, 100k} of pre-filtering documents, then reduced by the same filtering process as training data.Source: §A.13 · p. 59

Related papers

Loading 2506.20920v1…