PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

Vinko Sabolčec, Bettina Messmer, Yassine Turki, Martin Jaggi

arXiv:2610.11585v1cs.CLcs.AI

TL;DR

High-quality data selection matters for pretraining, but extending model-based quality filtering to long-tail languages without language-specific annotated data is a challenge. The paper adapts English quality classifiers using multilingual text embeddings and scores from translated text. Its classifiers maintain benchmark performance comparable to multilingual baselines and learn the English scoring criteria for languages unseen during training.

  • Problem

    How to scale high-quality data curation to long-tail languages without relying on language-specific annotated data remains an open challenge.

    Source: §1 · p. 1

  • Method

    The method trains a small MLP on multilingual encoder embeddings to predict scores assigned by English quality classifiers to machine-translated text.

    Source: §1 · p. 2

  • Results

    Across 1B, 3B, and 8B experiments, the approach maintains downstream benchmark performance comparable to multilingual filtering baselines, preserves regional and cultural knowledge, and learns English scoring criteria for languages unseen during training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support a viable path toward universal text quality classifiers by using high-resource-language research to support long-tail languages.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges due to limited availability of annotated data. This work explores extending quality filtering to over 100 languages by proposing a multilingual adaptation approach that converts an existing English quality classifier into a multilingual variant. Our approach proposes training a small multi-layer perceptron on top of Transformer encoder-only model embeddings, using multilingual text as input and scores obtained from English classifiers applied to machine-translated text as labels. Our 1B, 3B and 8B scale experiments show that our approach maintains the downstream LLM benchmark performance of existing multilingual model-based filtering baselines, without harming regional and cultural knowledge benchmarks. To further evaluate cross-lingual generalization, we compare classifier scores of high-quality synthetic data and web samples, and the correlation of classifier scores with LLM-based ones, revealing that the classifier can learn the scoring criteria of its original English variant, even for languages not included in its training data.

1 Introduction

Model-based filtering can improve pretraining data selection, but extending it to long-tail languages without language-specific annotated data remains a challenge. This work adapts English quality classifiers to multilingual text and reports comparable benchmark performance alongside cross-lingual generalization.

1 Introduction

  • High-quality subsets can match or exceed performance from larger, less refined datasets, making data selection relevant to downstream performance and training costs.Prior work reports reaching baseline performance with up to 10 times fewer tokens on selected data.Source: §1 · p. 1
  • The central challenge is scaling high-quality data curation to long-tail languages without language-specific annotated data.The proposed approach projects quality signals across languages rather than relearning them for each one.Source: §1 · pp. 1–2
  • A lightweight MLP learns from multilingual text embeddings and scores assigned by English quality classifiers to machine-translated text, reducing reliance on language-specific training data.The approach adapts existing English classifiers rather than relearning quality criteria separately for each language.Source: §1 · p. 2
  • The adapted classifiers maintain downstream benchmark performance comparable to multilingual filtering baselines while preserving regional and cultural knowledge, and learn English scoring criteria for languages unseen during training.Experiments cover 1B, 3B, and 8B models.Source: §1 · p. 2

2 Related Work

Pretraining data curation has progressed from heuristic filtering toward model-based quality selection, with multilingual methods expanding coverage but retaining data and language-coverage constraints. This work builds on English classifiers and multilingual embeddings to extend quality scoring across languages.

Pretraining Data Curation

  • Early curation pipelines combined language identification, heuristic filtering, deduplication, and perplexity filtering; later multilingual pipelines extended quality-oriented heuristics to over 1800 language-script pairs.FineWeb 2 is cited as a large-scale multilingual example.Source: §2 · p. 2

Model-based Quality Filtering

  • FineWeb-edu used LLM annotations for educational value, while DCLM used a lightweight FastText classifier for instruction-like text; model-based filtering left multilingual applications unexplored.These English approaches showed that small high-quality subsets could match or exceed performance from much larger unfiltered corpora.Source: §2 · pp. 2–3

Multilingual Model-based Quality Filtering

  • Multilingual model-based filters include per-language classifiers for 20 languages, a translated-data classifier for 17 languages, and a single classifier supporting over 100 languages.Existing methods still rely on language-specific high-quality training data or have not been evaluated beyond high-resource languages; this work does not directly compare with MuRating because its code, weights, and data are unavailable.Source: §2 · p. 3

Multilingual Embedding Models

  • Multilingual encoder-only models provide embeddings for quality filtering, with mBERT and XLM-RoBERTa supporting roughly 100 languages and mmBERT covering over 1800 language-script pairs.The proposed approach trains a lightweight MLP on top of these embeddings.Source: §2 · p. 3

3 Method

The method uses English quality classifiers to generate training scores for translated multilingual documents, then trains multilingual classifiers on embeddings of the original text. It combines sampled FineWeb 2 data with English FineWeb samples and frozen encoder embeddings.

3 Method

  • The pipeline translates multilingual documents into English, obtains quality scores from existing English classifiers, generates text embeddings, and trains classifiers using those data.The score labels come from English classifiers applied to translated text, while the multilingual text embeddings are the inputs.Source: §3 · p. 3
  • The method uses English FineWeb and multilingual FineWeb 2, which supports over 1800 language-script pairs.“Top N languages” means FineWeb 2 language-script pairs ranked by byte count in the training split.Source: §3 · p. 3
  • A random 0.1% sample from the top 100 language data was translated into English, yielding about 5 million documents at approximately 750 H100 GPU hours.The translation prompt aimed to preserve each text’s formatting, style, and organic errors.Source: §3 · p. 3
  • The training set contains about 6 million samples, including about 1 million English and 5 million multilingual examples, with MLPs trained on frozen encoder embeddings to predict English-classifier scores.The embedding models tested were XLM-RoBERTa, supporting 100 languages, and mmBERT, supporting over 1800 language-script pairs; the framework is also compatible with full fine-tuning.Source: §3 · p. 4
  • The MLP uses two 3072-dimensional hidden layers with ReLU activations and a scalar output, trained for six epochs with AdamW and a cosine schedule.Training uses 10% linear warmup, a peak learning rate of 3e-4, and a batch size of 1024.Source: §3 · p. 4
  • Unless otherwise stated, classifiers are trained on data from all processed languages, including English and the top 100 languages.The paper reports score distributions, score error distributions, and correlations on English FineWeb data in Appendix B.Source: §3 · p. 4

4 Results and Discussion

Across 1B, 3B, and 8B models, multilingual adaptations perform comparably to existing filtering baselines, while analyses find no inherent regional or cultural bias and evidence of cross-lingual scoring generalization.

4.1 Experimental Setup

  • The evaluation pretrains 1B, 3B, and 8B LLMs on the top 10% of FineWeb and FineWeb 2 data per language, preserving the original language distribution and assessing language understanding, reasoning, and knowledge benchmarks.Cross-lingual generalization analyses compare synthetic and web samples for languages 21–100 and classifier scores with LLM-judge scores beyond the top 100.Source: §4.1 · pp. 4–5

4.2 Experimental Results

  • FineWeb-edu and DCLM classifiers perform comparably with both XLM-RoBERTa and mmBERT embeddings, with only modest downstream-performance variation across these choices.The study focuses subsequent experiments on mmBERT embeddings.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • Training on English alone significantly outperforms the FW+FW2 baseline and remains competitive with model-based approaches; using all target languages brings a slight gain over top-5- or top-10-language training.Figure 1 compares 1B models trained on 100B tokens and retains the top 10% of documents for classifier-filtered data.Source: Figure 1 · p. 6, §4.2 · p. 6
  • Across 1B, 3B, and 8B scales, mFW-edu and mDCLM perform comparably to the strongest filtering baseline; mFW-edu leads at 1B, while mDCLM leads at 3B and 8B.FWHQ+FW2HQ+ is the strongest baseline, and both adapted classifiers outperform the heuristic FW+FW2 baseline.Source: Table 2 · p. 7, §4.2 · p. 7
  • mFW-edu performs comparably to the native multilingual baseline on INCLUDE and CulturalBench, while English-only mFW-edu drops at 8B on CulturalBench.The analysis reports that adaptation does not introduce bias, provided the base English classifier and embedding models are unbiased.Source: §4.2 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8
  • For out-of-distribution languages, the classifier distinguishes synthetic encyclopedia data from web data at lower score values; performance improves as more training languages are added.For in-distribution languages, web and synthetic-data scores align closely; the pattern also holds for Basque, a language isolate.Source: §4.2 · p. 8, Figure 2 · p. 9
  • Classifier scores correlate positively with LLM-judge scores both within the top 100 languages and beyond them, except for cni_Latn with gpt-oss-120b and hot_Latn with gemma-3-27b-it.The authors interpret this pattern as capturing the original scoring criteria beyond the classifier’s initial language-training distribution.Source: Figure 3 · p. 9, §4.2 · p. 9

5 Conclusion

The paper presents a scalable way to derive multilingual quality classifiers from English classifiers and reports comparable performance to native multilingual baselines without introducing English-centric biases. It also finds generalization beyond the training distribution, alongside shifted absolute scores for out-of-distribution languages.

5 Conclusion

  • The proposed approach derives scalable multilingual quality classifiers from English ones.The paper describes this as a simple approach.Source: §5 · p. 9
  • The classifiers perform on par with native multilingual baselines without introducing English-centric biases.Source: §5 · p. 9
  • The classifiers generalize quality criteria to languages outside the training distribution, although their absolute scores may shift for those languages.The paper identifies toxicity filtering as a future direction for applying the approach to other filter types.Source: §5 · p. 9

A Translation Prompt Used for Multilingual Document Translation

The appendix provides a translation prompt that asks for full English translation while preserving stylistic and formatting characteristics. It illustrates the prompt with an Italian WhatsApp guide and its English translation.

A Translation Prompt Used for Multilingual Document Translation

B Comparison Between English and Multilingually-Adapted Quality Classifiers

Figure 4 compares original classifiers and multilingual adaptations on English FineWeb data using score distributions, score errors, and score correlations.

B Comparison Between English and Multilingually-Adapted Quality Classifiers

C Configuration of the 1B, 3B, and 8B Paramter LLMs

The experiments use distinct training configurations for 1B, 3B, and 8B LLMs, including both scratch training and checkpoint cooldown runs.

C Configuration of the 1B, 3B, and 8B Paramter LLMs

  • The 1B multilingual models were trained from scratch on 100B tokens across English and the top 20 languages.The 1B monolingual models were trained from scratch on 30B tokens for several languages.Source: C Configuration of the 1B, 3B, and 8B… · p. 16
  • The 1B multilingual and monolingual configurations used batch sizes of about 2M tokens and a peak learning rate of 1.5e-4.The 3B run also used about 2M-token batches and a peak learning rate of 1.5e-4; the 8B run used about 4M-token batches and a peak learning rate of 1.1e-4.Source: C Configuration of the 1B, 3B, and 8B… · pp. 16–17
  • The 3B model was cooled down on 50B tokens across FineWeb and FineWeb 2 languages from a 200B-token pretrained checkpoint.The checkpoint used a mixture of 85% FineWeb and 15% top-33% FineWeb2-HQ data from the Apertus pretraining phase 1.Source: C Configuration of the 1B, 3B, and 8B… · p. 17
  • The 8B model was cooled down on 63B tokens across FineWeb and FineWeb 2 languages from a 9.5T-token pretrained checkpoint.Source: C Configuration of the 1B, 3B, and 8B… · p. 17

D Synthetic Encyclopedia Data Generation Process

The synthetic-encyclopedia process defines diverse article topics, generates titles and articles in multiple languages, and includes an example classifier-score distribution.

D Synthetic Encyclopedia Data Generation Process

  • Nineteen diverse categories and four seed article titles per category guide Qwen3-8B to generate 20 titles for each category.The listed categories span topics such as countries, mythology, literature, and animals.Source: D Synthetic Encyclopedia Data… · pp. 17–18
  • Synthetic encyclopedia articles are generated in eight languages, including English, Chinese, French, Arabic, Danish, Kazakh, Assamese, and Basque.The process uses Qwen3.5-35B-A3B and an encyclopedia-style prompt specifying article title, category, and language code.Source: D Synthetic Encyclopedia Data… · p. 18
  • A French-language octopus article illustrates the generated encyclopedia-style output.The example begins with the animal’s classification and physical characteristics.Source: D Synthetic Encyclopedia Data… · p. 19
  • Figure 5 presents the adapted mFW-edu classifier’s score distribution on English synthetic encyclopedia data.Source: D Synthetic Encyclopedia Data… · p. 19

E LLM-as-a-judge Prompt and Example

The multilingual FineWeb-edu judge prompt adapts the original prompt with language information and evaluates educational value using an additive five-point rubric. A Danish example response illustrates the assessment, while Figure 5 shows classifier scores on English synthetic encyclopedia data.

E LLM-as-a-judge Prompt and Example

  • The original FineWeb-edu prompt is adapted by adding language information, and reasoning outputs are disabled when applicable.Source: E LLM-as-a-judge Prompt and Example · p. 19
  • Figure 5 shows the adapted mFW-edu classifier’s score distribution on English synthetic encyclopedia data.Source: Figure 5 · p. 20
  • The rubric accumulates up to five points for educational value, progressing from basic relevant information to thorough, grade-school-suitable teaching material.Intermediate criteria distinguish superficial or disorganized material, coherent introductions, and focused textbook-like content.Source: E LLM-as-a-judge Prompt and Example · p. 20
  • For a Danish Deutzia extract, the example judge describes coherent, useful general learning material but not a comprehensive textbook chapter.Source: E LLM-as-a-judge Prompt and Example · p. 21

F Impact of Including Varying Amount of Languages in Quality Classifier Training

Table 5 compares benchmark results for 1B multilingual LLMs trained with the adapted classifier using varying amounts of training languages and several baselines.

F Impact of Including Varying Amount of Languages in Quality Classifier Training

  • Table 5 reports benchmark performance for 1B multilingual LLMs trained on 100B tokens using classifiers trained with varying amounts of languages and listed baselines.Source: Table 5 · p. 21, F Impact of Including Varying Amount… · p. 21
  • The comparison includes FW+FW2, FWHQ+FW2HQ, and FWHQ+FW2HQ+ baselines alongside the adapted mFW-edu classifier paired with mmBERT embeddings.Source: Table 5 · p. 21
  • The table retains the top 10% of documents by classifier score, except for FW+FW2, which is not model filtered.Source: Table 5 · p. 21

Related papers

Loading 2610.11585v1…