Source-linked AI summary
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, Thomas Wolf
TL;DR
Multilingual LLM development is limited by the difficulty of adapting data curation to many languages and by uneven data availability. The paper develops an automatically adaptable FineWeb-based pipeline, validates it with selected early-signal tasks, and scales it into FineWeb2. FineWeb2 outperforms prior multilingual datasets for 11 of 14 evaluated languages, while low-resource corpora remain heavily concentrated in Bible- and Wikipedia-related sources.
Problem
Multilingual LLM training is challenging because low-resource languages lack the mature filtering, deduplication, language-identification, and evaluation infrastructure available for high-resource languages.
Method
The paper builds a FineWeb-based pipeline that adapts processing to language-specific statistics, selects evaluation tasks using measurable early-signal criteria, and uses duplication counts with quality signals for rebalancing.
Results
FineWeb2 produces more performant models than prior multilingual datasets on 11 out of 14 evaluated languages and covers 1,868 language-script pairs in 20 terabytes.
Takeaways & Limitations
The pipeline generalizes across languages and supports large-scale multilingual corpus creation, although language-expert hand-designed datasets can still outperform it in some cases.
Takeaways & Limitations
Computational constraints, task availability, and very small datasets limited testing to a small proportion of FineWeb2 languages, with only relatively short ablation runs.
Abstract
from arXiv · showhide
Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-training datasets has seen substantial recent progress, training performant multilingual LLMs remains a challenge, in large part due to the inherent difficulty of tailoring filtering and deduplication pipelines to a large number of languages. In this work, we introduce a new pre-training dataset curation pipeline based on FineWeb that can be automatically adapted to support any language. We extensively ablate our pipeline design choices on a set of nine diverse languages, guided by a set of meaningful and informative evaluation tasks that were chosen through a novel selection process based on measurable criteria. Ultimately, we show that our pipeline can be used to create non-English corpora that produce more performant models than prior datasets. We additionally introduce a straightforward and principled approach to rebalance datasets that takes into consideration both duplication count and quality, providing an additional performance uplift. Finally, we scale our pipeline to over 1000 languages using almost 100 Common Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document) multilingual dataset which we release along with our pipeline, training, and evaluation codebases.
1 Introduction
Multilingual LLMs remain harder to develop because data and curation pipelines are concentrated in high-resource languages. The paper introduces an automatically adaptable FineWeb-based pipeline and applies it at large scale to create FineWeb2.
- Models trained on high-resource-language data perform better because LLM capabilities largely stem from their pre-training data.
- Low-resource languages lack the mature data-processing pipelines available for high-resource languages, complicating quality evaluation, language identification, filtering, and word separation.
- The proposed pipeline automatically adapts to language-specific statistics and is validated through ablations on nine diverse languages.
- FineWeb2 processes almost 100 Common Crawl snapshots into a 20 terabyte, 5 billion document dataset covering over 1000 languages.
2 Preliminaries
Massively multilingual processing requires explicit language-script conventions and language-appropriate word segmentation. These choices support filtering, deduplication, and evaluation across writing systems with different or invisible word boundaries.
- Notation: Languages are identified with ISO-639-3 codes, optionally paired with ISO 15924 script codes for languages using multiple writing systems.
- Separating words: Word tokenization supports filtering, n-gram deduplication, and generative evaluation throughout the processing pipeline.
- Separating words: Because some writing systems lack visible word boundaries, the pipeline uses language- and script-tailored tokenizers or proxy tokenizers.
3 Experimental setup
The experiments compare processing choices across nine linguistically diverse languages using small-model early-signal evaluations. The setup also examines tokenization and multilingual dataset baselines while selecting benchmarks through measurable reliability criteria.
- Small models are compared on early-signal tasks after differing only in their training data, enabling data-quality comparisons at relatively early training stages.
- The nine canary languages span different language families, scripts, and resource levels while keeping computational requirements manageable.
- Tokenizer: Tokenizer evaluation measures subword fertility and the proportion of continued words across the nine canary languages.
- Baseline datasets: The baselines include CC-100, mC4, CulturaX, HPLT, and raw Common Crawl data.
- Selecting evaluation (Fine)tasks: Non-English evaluation tasks often lack community validation and may suffer from quality issues, motivating systematic task selection.
- Selecting evaluation (Fine)tasks: Early-signal tasks are selected using monotonicity, low noise, non-random early performance, and ordering consistency.
- Selecting evaluation (Fine)tasks: 84 benchmarks were selected from 197 tested across the nine canary languages.
- Aggregate task scores are rescaled relative to random baselines before averaging across tasks.
4 The FineWeb2 pipeline
FineWeb2 adapts language-model data curation to multilingual settings through language-specific identification, deduplication, and empirically selected filtering and rehydration procedures. The pipeline uses measurable corpus statistics and filtering outcomes to tune decisions across languages.
- 4.2 Language Identification (LID): GlotLID was selected for the pipeline because its broader language coverage outweighed slightly lower downstream performance on lower-resource languages.GlotLID outperformed FT176 on higher-resource languages but was slightly behind on lower-resource languages.
- 4.2 Language Identification (LID): Language-specific LID thresholds are chosen by evaluating models across confidence levels and using one standard deviation below each language’s median confidence, clipped to [0.3, 0.9].Arabic and Russian favor thresholds above 0.8, whereas Swahili performs best around 0.3, corresponding to almost 65% removal.
- 4.3 Deduplication: Deduplication is performed before filtering so each filtering experiment’s final dataset performance can be observed without later deduplication changing the result.MinHash clusters similar documents and retains one document per cluster using language-specific global deduplication.
- 4.3 Deduplication: Deduplication generally improves model performance, but its effect varies substantially by language and shows no discernible relationship with resource level.Languages with little or no deduplication gain still benefit from rehydration.
- 4.4 Filtering: Filtering thresholds are adapted empirically by matching language-specific metric distributions through methods including MeanStd, Quantile, 10Tail, and MedianRatio.These methods transfer or recompute threshold relationships from English and reference corpora for each filter.
- 4.5 Rehydration: Rehydration uses filtering rates as a scalable quality proxy, assigning weight 10 to the least-filtered cluster size, weight 1 above the global rate, and interpolated weights between them.For French, filtering rates form a U-shaped pattern across cluster sizes, and rehydration generally provides a strong performance uplift with little downside.
5 Validating and Applying the FineWeb2 Pipeline
The authors apply the established pipeline to create FineWeb2 and evaluate it against multilingual and language-specific datasets on both design languages and unseen languages. FineWeb2 performs better than prior multilingual datasets for most evaluated languages, while low-resource corpora remain limited in source diversity.
- Creating the FineWeb2 dataset: FineWeb2 is generated from 96 Common Crawl snapshots and contains 20 terabytes across 1,868 language-script pairs.The dataset includes 1,226 pairs with over 100 documents, 474 with more than 1,000, and 203 with at least 10,000.
- Comparison to other datasets: FineWeb2 is compared with other non-English datasets on both nine pipeline-design languages and unseen languages not used for ablations.The comparison tests whether its language-adaptive approach transfers beyond the canary languages.
- Comparison to other datasets: FineWeb2 produces more performant models than prior multilingual datasets on 11 out of 14 evaluated languages.Language-specific expert-designed pipelines still outperform FineWeb2 in some cases.
- Comparison to other datasets: The observed performance trends hold for both canary and held-out datasets, supporting the utility of the generated 1,000+ language-specific datasets.The authors characterize the overall pipeline as consistent but adaptable across languages.
- Inspecting low-resource corpora: Among 1,868 language-script pairs, 70% have more than half of their documents from Bible- or Wikipedia-related domains.Manual inspection found that many low-resource corpora contain fewer than 20 documents and are composed almost exclusively of these sources.
6 Conclusion
The paper presents a data-driven multilingual processing pipeline that automatically adapts across languages and uses duplication counts and filtering quality to improve dataset balance. Scaling this approach produced FineWeb2, while experiments and dataset coverage remain constrained in several ways.
- The pipeline automatically adapts multilingual pre-training data processing to different languages instead of using fixed language-specific pipelines.
- Ablations across 14 languages used quantitatively identified multilingual benchmarks to evaluate processing design choices.
- Duplication counts and filtering results can selectively upsample higher-quality content, providing a performance uplift.
- FineWeb2 covers 1,868 language-script pairs and 20 terabytes of text curated from 96 Common Crawl snapshots.
- Evaluation covered only a small proportion of FineWeb2 languages because of computational constraints, task availability, and very small low-resource datasets.
- The experiments used relatively short ablation runs and early-signal task properties, while selected tasks omitted criteria such as cultural alignment, bias, and diversity.
A.1 Word tokenizers for segmentation
The tokenizer assignment method scales word segmentation across languages by combining native tokenizers with proxy assignments propagated through language-family trees. Remaining languages receive script- or resource-based fallback tokenizers.
- Tokenizer coverage draws on SpaCy, Stanza, and language-specific libraries for Indic languages, Thai, Korean, Khmer, Lao, Tibetan, and Burmese.
- The process builds a language-family tree and assigns native tokenizers to language+script pairs where available.
- An upward tree pass propagates one tokenizer per script, selecting the tokenizer from the child language with more available data and excluding the root node.
- A downward pass assigns propagated parent tokenizers as proxies to languages without native tokenizers.
- For remaining languages, the method uses SpaCy multilingual tokenization for Latin or Cyrillic scripts and the highest-resource native-tokenizer language for other scripts.
Indo-European Language Family Tree
The Indo-European example illustrates tokenizer assignments represented by native or proxy markers and propagated within, but not across, top-level language subfamilies.
- Triangles mark languages with native word tokenizers, while squares mark languages assigned proxy tokenizers.
- Bracketed labels identify each assigned tokenizer, and parenthetical values show the number of languages in each subnode.
- Italian is propagated through the Italo-Dalmatian subfamily, while Spanish moves upward from the Western branch because it has more available data than Italian.
- Latin retains its own native tokenizer, and propagation stops at the first level rather than reaching the root or crossing top-level subfamilies.
A.2 Canary Languages
The paper uses a subset of languages as canaries for in-depth evaluation, with the selection organized around language families, scripts, and resource availability. Table 1 summarizes these attributes for nine languages.
- The in-depth evaluation subset contains nine canary languages selected from the broader multilingual corpus.
- Table 1 organizes the canary languages by language family, main script, and resource availability.
A.3 Multilingual tokenizers comparison
The comparison evaluates multilingual tokenizers using subword fertility and proportion of continued words, with lower values preferred across languages. After excluding poor-coverage or unknown-token cases, Gemma was selected for the experiments.
- Subword fertility measures the average tokens per real text word, quantifying how aggressively a tokenizer splits words.
- Proportion of continued words measures the share of real words encoded with at least two tokens, ranging from never splitting to always splitting.
- The study computes both metrics on Wikipedia words using tokenizers from nine popular multilingual models.
- Tokenizers with poor canary-language coverage were excluded; mT5 and XGLM also produced unknown tokens for some characters, particularly Chinese.
- Gemma was chosen because it performed slightly better than Bigscience-Bloom on average in this experimental setup.
A.4 Model architecture and training
The paper reports architecture and training configurations for its model experiments across three training scales. The supplied material identifies the configuration tables but does not provide their individual settings.
- Table 3 presents the architecture configuration used for the models.
- Table 4 reports training settings for 29, 100, and 350 billion-token training scales.
- For the 100-billion- and 350-billion-token settings, critical batch size is computed using DeepSeek-AI et al. (2024).
- Table 5 reports optimizer settings for the three training scales and uses a constant learning rate until the final 20% of steps.
A.5 Evaluation details
The evaluation framework selects early-signal tasks using measurable criteria for training behavior and compares pipeline choices across multilingual settings. It also examines robustness, contamination, and corpus-domain composition relevant to dataset evaluation.
- Task selection criteria: Early-signal tasks were required to satisfy quantitative criteria, including monotonicity, low noise, non-random early performance, and ordering consistency.The criteria were designed to identify tasks that support reliable comparisons during training.
- Signal-to-noise ratio: SNR was estimated from four models trained on unfiltered CommonCrawl data under different random seeds, using mean performance as signal and standard deviation as noise.The overall SNR averaged the signal-to-noise ratio across training steps, with a minimum required SNR of 20 except for generative tasks.
- Non-random performance: Non-randomness compared the maximum final-step improvement over a random baseline with terminal variance estimated from the last five evaluation steps.A task satisfied the criterion only when its improvement-to-variance ratio met the specified threshold.
- Ordering consistency: Ordering consistency averaged Kendall Tau-a between model rankings at consecutive steps in the second half of training, excluding the first 15 billion tokens.The exclusion addressed inconsistent early rankings that would skew the overall score.
- Filtering and corpus audits: Wordlist filtering preserved recall while improving precision for glk Arab and bar Latn, whereas ary Arab showed limited improvement.Around a third of 1,900 languages with wordlists had contamination scores above 10%, and 70% of 1,868 language-script pairs had more than half their documents from Bible- or Wikipedia-related domains.