PaperlayerView exact revision on arXiv

Source-linked AI summary

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

Ansar Aynetdinov, Patrick Haller, Alan Akbik

arXiv:2604.28075v2cs.CLcs.AI

TL;DR

The paper asks whether German language modeling should favor diverse lightly filtered data or a smaller high-quality corpus repeated over epochs. It constructs hierarchical filters and compares repeated training on filtered subsets with broader training, finding that high-quality repetition consistently performs better while also releasing German models and cleaned benchmarks.

  • Problem

    For high-resource non-English languages, aggressive filtering creates a choice between preserving diverse token pools and prioritizing semantic density through a smaller high-quality core.

  • Method

    The authors build hierarchical quality filters over German web data, train models on filtered subsets across multiple epochs and regimes, and evaluate them with cleaned German benchmarks.

  • Results

    Across the tested model sizes and token budgets, multi-epoch training on small high-quality subsets consistently outperforms training on larger, less filtered mixtures, without observed early saturation.

  • Takeaways & Limitations

    Quality filtering and multi-epoch training offer a viable strategy for sample-efficient pre-training in high-resource non-English languages such as German.

  • Takeaways & Limitations

    The study focuses on German and models up to 1B parameters trained on at most 200B tokens, so broader language and scale generalization remains unclear.

Abstract

from arXiv · show

Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners prioritize diversity by training once on large amounts of lightly filtered web data, or prioritize quality by strictly filtering for a high-quality core and repeating it over multiple epochs? We investigate this trade-off for German by constructing hierarchical quality filters applied to 500M web documents, comparing multi-epoch training on the filtered subsets against single-pass training on a diverse corpus. Our experiments across multiple model scales and token budgets show that repeating high-quality data consistently outperforms single-pass training on larger, less filtered sets. Notably, the performance gap persists even after 7 epochs. Our findings suggest that for non-English LLMs, semantic concentration through quality filtering offers a more viable path to efficient language modeling than simply maximizing unique data volume. We release our German language models (called Boldt), as well as our cleaned evaluation benchmarks to the research community. Our experiments indicate that they achieve state-of-the-art results despite training on 10-360x fewer tokens than comparable models.

1 Introduction

The paper examines whether German language modeling should prioritize diverse web coverage or semantically dense, high-quality data repeated across epochs. It also evaluates downstream instruction tuning, benchmark reliability, and released German models.

  • German’s smaller high-resource web corpora create a trade-off between preserving diversity through light filtering and maximizing semantic density through strict filtering.The authors frame semantic density as expected training signal per token.
  • The study isolates semantic properties by training from scratch on filtered subsets and testing multi-epoch training and staged data curricula.The experiments examine whether cautious limits on repeated exposure remain warranted for knowledge-dense data.
  • Repeated training on high-quality German data outperforms maximizing unique web document coverage under a fixed pre-training budget.The analysis separates the effects of Coherence, Information Value, and Educational Quality.
  • Multi-epoch pre-training on filtered data improves instruction-tuning behavior, producing higher correctness in assistant tasks.
  • The authors correct significant noise in German ARC-Challenge, HellaSwag, LAMBADA, and OpenBookQA translations and release the filtered benchmarks.The release is intended to support more reliable German-language evaluation.
  • German small language models up to 1B parameters achieve state-of-the-art results for their size despite using an order of magnitude fewer tokens than competitive baselines.

2 Related Work

Prior work established that filtering and model-based selection can improve language-model training efficiency, but repeated exposure and aggressive filtering remain less explored for non-English data-constrained settings.

  • Filtered CommonCrawl datasets use deduplication, language filtering, heuristics, and classifiers to remove noisy data and improve pre-training outcomes.
  • Textbook-quality and educational-value filtering further improved sample efficiency in the Phi models, FineWeb, and FineWeb-Edu.
  • Strict filtering is less explored for non-English languages because it may critically reduce already limited token budgets.
  • Repeated exposure to the same dataset was reported to have diminishing returns after 4 epochs, while later work found up to 10 epochs on aggressively filtered data could beat one pass over a 10x larger unfiltered superset.
  • The paper extends this line of work by testing whether strict quality filtering remains practical for German, where the unfiltered pool is smaller than English-scale corpora.

3 Data and Filtering

The data pipeline starts from FW2-DE and applies hierarchical document-level filters that progress from structural coherence toward information density and educational value. Their intersection forms the DENSE CORE used for high-quality multi-epoch training.

  • The pipeline uses the German FineWeb-2 split, whose basic quality filtering is extended with a hierarchical framework for measuring semantic density.
  • Three non-mutually-exclusive document classifiers represent increasing selection strictness and score the FW2-DE pool.
  • Filtering tiers: COHERENCE removes word-salad documents, truncated HTML exports, and fragmentary snippets by targeting linguistic and structural integrity.
  • Filtering tiers: DENSE CORE is the intersection of coherence, information density, and educational filtering, producing the primary high-quality subset for multi-epoch training.
  • Dataset statistics: As filtering becomes stricter, average document length rises, with DENSE-CORE documents nearly 50% longer than RANDOM documents.
  • Tokenization: Discarding lower-quality data from tokenizer training improves tokenizer fertility on train splits, a shared holdout, and benchmark question prompts.

4 Evaluation Datasets

The paper addresses unreliable German benchmark translations by identifying language-specific task distortions, re-translating full instances, and releasing a cleaned evaluation suite spanning knowledge, reasoning, and context tracking.

  • German evaluation lacks broad-range benchmarks, while machine-translated versions often contain non-idiomatic phrasing, grammatical errors, and structural artifacts that alter task difficulty.
  • Translation artifacts: German word order can move completion targets away from sentence-final position in HellaSwag and LAMBADA, disrupting the intended prediction task.
  • Translation artifacts: The authors re-translate full benchmark instances with Tower+ 72B and discard fewer than 0.5% of instances when translation fails or breaks task logic.
  • Evaluation coverage: The modernized suite evaluates factual knowledge, grade-school science reasoning, commonsense narrative continuation, and discourse context tracking.
  • The cleaned benchmarks are released to support more rigorous and standardized assessment in German NLP.

5 Investigating the Quality-Quantity Trade-off

The experiments test whether high-quality repetition can compensate for reduced unique-token diversity under fixed training budgets. Across model scales, dense-core repetition consistently outperforms single-pass training on broader data.

  • Experimental design: The study compares fixed-budget single-pass training on diverse web data with repetition on high-density subsets.The investigation covers token allocation, model scaling, repetition limits, and instruction-tuning effects.
  • Results: 4.89 points: DENSE-CORE outperforms RANDOM on average after approximately 3.6 epochs over 28B unique tokens.The comparison uses models trained to the same 100B-token exposure.
  • Results: DENSE-CORE maintains an advantage throughout the 100B-token trajectory, while AA HIGH performs poorly despite 4.8× repetition.The reported analysis attributes AA HIGH’s weaker downstream impact to smaller annotator and classifier models.
  • Results: PHASED and SORTED curricula improve after transitioning to higher-quality data but consistently trail pure DENSE-CORE training.Their gains occur mainly in the second half of training, coinciding with the quality transition.
  • Scaling analysis: 5.14 points: the 1B DENSE-CORE model leads the 1B RANDOM baseline, widening the performance gap seen at 350M.The 350M comparison is 4.89 points on average under the same 100B-token budget.
  • Scaling analysis: The 1B DENSE-CORE model reaches parity with or exceeds multilingual GEMMA-3-1B and LLAMA-3.2-1B despite substantially fewer training tokens.It also outperforms monolingual LLÄMMLEIN-1B in the reported evaluation suite.

5.3 Experiment III: Exploring Repetition Limits (200B Budget)

The 200B-token experiment examines whether dense-core repetition eventually loses its advantage. DENSE-CORE remains ahead of RANDOM beyond seven epochs, while PHASED narrows but does not close the gap.

  • Experimental setup: The experiment extends RANDOM, PHASED, and DENSE-CORE training from 100B to 200B tokens to probe diminishing returns.DENSE-CORE reaches approximately 7.2 epochs over its 28B-token subset.
  • Results: Benefits from DENSE-CORE repetition persist beyond the four-epoch threshold previously identified in related work.The paper frames this as an extension to a non-English setting with a more constrained unfiltered data pool.
  • Results: No benchmark generalization loss is observed when the number of epochs on the same dataset is doubled.DENSE-CORE continues to lead RANDOM even while RANDOM receives entirely new unique data.
  • Results: PHASED continues scaling and narrows the gap to DENSE-CORE but never overtakes it.The reported comparison suggests that beginning with more diverse data holds back later optimization on the denser subset.
  • Scaling analysis: 2.08 points: the 1B DENSE-CORE model improves on average when trained for 200B rather than 100B tokens.This uplift is reported as more than twice the corresponding improvement for the 350M model.
  • Instruction tuning: The instruction-tuning evaluation uses 1,000 held-out German prompts judged for Likert score and binary accuracy.Final checkpoints are fine-tuned on the German SMOLTALK2 subset using an LLM-as-a-judge protocol.
  • Instruction tuning: 253/1,000: DENSE-CORE achieves the highest correct-response count among the 350M models at the 100B budget.INFORMATION VALUE reaches 249/1,000, while RANDOM reaches 178/1,000.
  • Instruction tuning: MKC does not outperform DENSE-CORE, and AA HIGH reaches 199 correct answers in the instruction-tuning evaluation.The authors connect these results with the importance of information-rich data and sufficiently high-quality repetition.

6 Model and Benchmark Release

The paper releases BOLDT German language models and updated evaluation benchmarks. The released models are positioned as competitive despite substantially smaller training corpora than comparable German-capable models.

  • Model release: The release includes BOLDT-DC-350M and BOLDT-DC-1B models trained on a 200B-token budget using the DENSE CORE subset.These models are released to support reproducibility of the paper’s results.
  • Release contents: The release also includes updated evaluation benchmarks and a preview of an instruction-tuned BOLDT-1B.The instruction-tuning data combines real and synthetic instruction-output pairs from diverse sources.
  • Model release: BOLDT-1B combines DENSE CORE with 6B tokens of German news data and is trained to an effective 230B-token count.Its context window is extended from 2048 to 4096 compared with BOLDT-DC-1B.
  • Model evaluation: BOLDT-1B improves upon BOLDT-DC-1B on all reported benchmarks except ARC-Challenge and ARC-Easy.The models remain competitive with larger, approximately 2B-parameter multilingual models despite smaller training corpora.

7 Conclusion

The paper asks whether aggressive quality filtering remains worthwhile for German despite limited data, and finds that repeated training on high-quality subsets consistently outperforms broader, less filtered alternatives. These gains extend to instruction tuning and support quality filtering with multi-epoch training as a practical strategy for efficient non-English pre-training.

  • The study directly tests whether aggressive quality filtering is worthwhile when total available German text is limited, rather than assuming that it discards too much useful diversity.
  • Repeated training on small, high-quality German subsets consistently outperforms training on larger, less strongly filtered mixtures across the tested model sizes and token budgets.The experiments do not observe an early saturation point for repeating high-quality data, while additional unfiltered data provides limited gains.
  • Curated high-quality pre-training subsets also improve instruction-tuned assistants’ correctness and helpfulness relative to more diverse, less filtered data.Introducing lower-quality documents into pre-training mixtures decreased instruction-tuned model output quality.
  • Extensive quality filtering with clearly defined rules and strong annotator models provides a viable path toward sample-efficient pre-training for high-resource non-English languages such as German.

Limitations

The evidence is limited primarily by language, model and compute scale, architecture, and safety evaluation coverage. The authors caution that findings may not generalize across languages, larger systems, or model-safety dimensions.

  • Language scope: The study focuses exclusively on German, so languages with smaller corpora or different linguistic structures may exhibit different quality–quantity trade-offs.The authors call for validation across diverse language families.
  • Model and compute scale: The experiments cover models up to 1B parameters and at most 200B training tokens, leaving substantially larger industry-scale trade-offs unresolved.
  • Architecture selection: Only dense transformer architectures are evaluated, so mixture-of-experts models and alternative attention mechanisms may behave differently with data quality and repetition.
  • Safety evaluation: The models are not evaluated for toxic generation, demographic bias, or harmful stereotypes, and aggressive filtering cannot guarantee the elimination of problematic content or biased behavior.The authors also note that repeated exposure could amplify societal biases and that their LLM-as-judge evaluation does not assess safety or harmful outputs.

A FW2-DE Annotation Details

The FW2-DE annotation pipeline uses LLM scoring to assign three semantic quality properties, trains separate regressors to scale those labels, and applies thresholds to construct nested filtered subsets.

  • Llama-3.3-70B-Instruct scores 500k sampled FW2-DE documents for Coherence, Information Value, and Educational Quality using defined annotation prompts.Educational Quality follows Lozhkov et al.’s prompt, while the paper provides its own prompts for Coherence and Information Value.
  • Three snowflake-arctic-embed-m-v2.0 regression models are fine-tuned on the annotated sample, with each model scoring one document-quality aspect.The selected training setup uses a linear learning-rate decay from 1e-4 to 0, batch size 64, and up to 10 epochs, choosing the best epoch by holdout validation accuracy.
  • The fine-tuned classifiers annotate the full 500M-document German FineWeb-2 subset.
  • COHERENCE and INFORMATION VALUE retain documents meeting maximum-score thresholds of 3 and 4, EDUCATIONAL QUALITY uses threshold 3, and DENSE-CORE applies all three thresholds jointly.

B LLM Training Details

The training setup spans 350M and 1B models, uses standardized optimization and hardware configurations, and evaluates instruction-tuned outputs with Likert-scale and binary-correctness procedures. The supplied captions identify benchmark comparisons across filtered subsets and model scales.

  • Model configurations: The 350M models use 24 layers with hidden size 1024 and FFN size 4096, while the 1B models use 16 layers with hidden size 2048 and FFN size 8192.
  • Pre-training: Pre-training uses cosine learning-rate decay with 1% warmup, AdamW, peak learning rate 5e-4, effective batch size 0.5M tokens, and gradient clipping at norm 1.0.
  • Instruction tuning: Instruction tuning changes the warmup to 5%, uses maximum learning rate 5e-5 and batch size 32, and sets AdamW β2 to 0.99.
  • Infrastructure: Pre-training runs use 8×A100 GPUs and instruction-tuning runs use 1×A100 GPU, with all models trained in bfloat16 precision.
  • Evaluation generation: Generated outputs use top-p sampling with p = 0.9 and temperature 0.6, while LLM-as-judge generation uses greedy decoding for consistency.
  • Benchmark comparisons: The benchmark tables compare 350M subset-trained models across 200B and 100B tokens, and 1B DENSE-CORE models trained on 100B and 200B tokens against similar-sized pre-trained models.
  • Evaluation: The instruction-tuned evaluations use standalone Likert-scale quality judgments and binary correctness judgments based on prompt templates.
Loading 2604.28075v2…