PaperlayerView exact revision on arXiv

Source-linked AI summary

KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report

Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, Michael Fromm, Nicolas Flores-Herr, Kristian Kersting

arXiv:2606.03773v2cs.CL

TL;DR

German pretraining resources remain less curated, documented, and validated than English counterparts. KletterMix translates a strong English mixture into German while preserving its structure and metadata, then evaluates the corpus through quality diagnostics and controlled training studies; across matched 0.6B and 7B annealing settings, the results support stronger German aggregate performance while English performance remains nearly unchanged in the larger mixture sweep.

  • Problem

    German pretraining resources are comparatively less curated, documented, and validated than English resources, limiting the available foundation for German language modeling.

  • Method

    KletterMix translates a high-quality English pretraining mixture into German while preserving document boundaries, metadata, source structure, and mixture design, then applies quality diagnostics and controlled training experiments.

  • Results

    Across matched 0.6B pretraining, corpus-choice annealing, and 7B language-mixture annealing, KletterMix has the strongest aggregate point estimates under the matched recipe, while increasing its 7B mixture share raises German aggregate estimates and leaves English aggregate performance nearly unchanged.

  • Takeaways & Limitations

    Carefully translated and documented data can transfer useful mixture structure to German and support both initialization and late-stage language adaptation.

  • Takeaways & Limitations

    Each model condition uses one training run and an approximately 12B-token budget, so larger models, additional seeds, and broader benchmark suites remain to be tested.

Abstract

from arXiv · show

High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English counterparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for language model pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source structure, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Using COMETKiwi, we show that the translated documents achieve strong quality across diverse domains, suggesting that careful translation can preserve much of the semantic and stylistic richness of the original corpus. Beyond dataset construction, we evaluate KletterMix as training data. Through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.

1 Introduction

KletterMix addresses the relative immaturity of German pretraining resources by transferring the structure of a strong English mixture through careful translation, documentation, and empirical evaluation. The resulting corpus preserves document-level structure and metadata while supporting scalable quality diagnostics and controlled training studies.

  • German pretraining resources are often smaller, less curated, weakly documented, and insufficiently validated through controlled training experiments.
  • KletterMix transfers a strong English corpus’s coverage and mixture structure into German through high-quality translation while preserving document boundaries, identifiers, metadata, source composition, and topical diversity.
  • 725B tokens: KletterMix is a German pretraining and annealing corpus translated from ClimbMix.
  • The project combines COMETKiwi and proxy-based quality diagnostics with analyses of length distributions, topic coverage, source composition, and metadata preservation.
  • KletterMix is evaluated through matched 0.6B-parameter pretraining and annealing experiments plus a 7B-parameter language-mixture annealing sweep.
  • The study tests whether careful translation, documentation, and empirical validation can strengthen German pretraining data without relying exclusively on additional native web crawling.

2 Related Work

Prior work increasingly treats pretraining corpora as documented, filtered, and evaluation-driven dataset artifacts. KletterMix extends this direction by transferring the diversity and mixture design of a strong English corpus into German through translation and controlled validation.

  • Large-scale pretraining corpora have shifted from raw web-scale collection toward documented, filtered, evaluation-driven dataset artifacts.
  • Existing corpus efforts emphasize diverse source mixtures, documentation, multilingual coverage, filtering choices, and validation through model training.
  • KletterMix differs from German-only crawling approaches by transferring a strong English mixture’s diversity and mixture decisions through high-quality translation.
  • Machine-translated pretraining data can be effective when carefully applied and filtered, including for languages with smaller or less diverse native corpora.
  • Documentation frameworks motivate explicit reporting of dataset composition, processing, intended uses, limitations, provenance, and translation artifacts.

3 KletterMix Pipeline

KletterMix uses a document-preserving, length-aware translation pipeline with scalable shard execution and post-hoc quality estimation. A target-only proxy extends expensive COMETKiwi labels to corpus-scale ranking and filtering, while fixed-threshold subsets support training ablations.

  • The five-stage pipeline comprises source normalization, length-aware routing, document-preserving translation, shard-wise execution, and post-hoc quality estimation.
  • Translated records retain document identifiers, German text, unchanged source metadata, and processing metadata such as buckets, chunking, configuration, and quality outputs.
  • Documents are routed into length-aware buckets, translated directly or through contextualized chunks, and concatenated in original order to preserve document identity and coherence.
  • The target budget is set as min(Lmax, ⌈αℓsrc + β⌉), allowing moderate target-side expansion while reducing wasted budget and truncation risk.
  • A stratified pilot subset is scored with COMETKiwi, and a gradient-boosted target-only proxy predicts quality from translated text and inexpensive metadata.
  • The proxy is validated on a disjoint 18,275-document split and deployed as a scalable ranking and filtering signal rather than a replacement for source-aware evaluation.
  • Three 12B-token filtering ablations retain documents above proxy thresholds 0.50, 0.55, or 0.60, while the primary release remains unfiltered except for basic validity failures.

4 Translation Insights

Corpus diagnostics show a diverse, heavy-tailed translated mixture and identify length and cluster-level signals for quality control. Training dynamics indicate lower KletterMix loss than established German corpora under matched conditions.

  • Corpus diagnostics: Figure 3 compares global lengths, bucket-level consistency, and cluster-level proxy-score profiles for the full release and fixed-budget 12B-token subset.
  • Corpus diagnostics: KletterMix has a heavy-tailed document-length profile, with mostly short documents and fewer documents extending into the long-context regime.
  • Corpus diagnostics: Long-context buckets contain lower tails of unexpectedly short German translations, which may indicate truncation, dropped content, or other translation failures.
  • Corpus diagnostics: The diagnostics expose document- and cluster-level quality-control signals and motivate continued corpus auditing.
  • Training dynamics: Across matched 12B-token training runs, KletterMix reaches lower training and held-out validation loss than FineWeb2-DE and GermanWeb.

5 Training Ablations

Controlled pretraining and annealing experiments test KletterMix against established German corpora under matched conditions. KletterMix shows lower validation loss and stronger downstream German benchmark performance, while 7B mixture annealing yields increasing German scores with minimal English change.

  • Matched 0.6B setup: Matched 0.6B experiments compare KletterMix, FineWeb2-DE, and GermanWeb using equal model, optimization, and token-budget conditions.Each corpus contributes a deterministic, stratified 12B-token subset; proxy-filtered KletterMix variants are also evaluated.
  • 0.6B training dynamics: KletterMix maintains lower validation loss than FineWeb2-DE and GermanWeb throughout training, indicating improved sample efficiency under a fixed token budget.The validation gap appears early and persists through the final checkpoints.
  • Benchmark evaluation: 40.2 is the best Core Avg. point estimate, while unfiltered KletterMix reaches 38.7 versus 38.3 for FineWeb2-DE.Because PIQA uses only 100 examples, aggregate standard errors are comparatively wide; KletterMix is most consistently strongest on HellaSwag and ARC-C.
  • 0.6B corpus-choice annealing: 39.4 is the Core Avg. from annealing a FineWeb2-DE checkpoint on KletterMix, improving over the source checkpoint by +1.1 points and over GermanWeb annealing by +1.8 points.The largest task-level gains are on HellaSwag, from 31.5 to 34.2, and ARC-C, from 23.0 to 25.2.
  • 7B language-mixture annealing: The 7B sweep varies only the data mixture across matched ∼12B-token annealing runs, with KletterMix supplying 0%, 5%, 10%, 15%, or 20% of tokens.All runs start from the same OLMo 3 7B Stage 1 checkpoint and use the same optimization schedule.
  • 7B language-mixture annealing: 4.5 percentage points is the German four-task mean improvement when the KletterMix share rises from 0% to 20% in 7B annealing, from 45.4% to 49.9%.Over the same comparison, the English mean changes from 66.3% to 66.6% (+0.3 points).

6 Conclusion

Across matched pretraining and annealing experiments, KletterMix supports transferring useful mixture structure from a high-quality English corpus into German data. The evidence is positive but qualified by non-uniform task results, proxy-filtering limits, and experimental scope boundaries.

  • 6 Conclusion: KletterMix achieves the strongest Core Avg. point estimates under the matched recipe across MMLU, PIQA, HellaSwag, and ARC-C.GermanWeb remains strongest on MMLU, while FineWeb2-DE remains highly competitive on PIQA; KletterMix is strongest on HellaSwag and ARC-C.
  • 6 Conclusion: At 0.6B parameters, continuing FineWeb2-DE on KletterMix yields a better Core Avg. than continuing it on GermanWeb, with gains concentrated on HellaSwag and ARC-C.
  • 6 Conclusion: At 7B parameters, increasing KletterMix from 0% to 20% in a fixed 12B-token annealing budget raises German aggregate point estimates while leaving the English aggregate nearly unchanged.
  • 6 Conclusion: Proxy scores can rank data under a fixed budget, but they are not a universal quality law because MMLU does not improve monotonically with stricter filtering.
  • 6 Conclusion: KletterMix may inherit topical, cultural, geographic, stylistic, and licensing biases from its English sources, while translation may introduce translationese, semantic drift, and unnatural German style.
  • 6 Conclusion: The model-based evidence covers 0.6B and 7B parameters, but each condition has one training run and the 7B experiment uses one starting checkpoint and a quality-ranked subset.Future work should test larger models, additional random seeds, and broader downstream benchmark suites.

A Pipeline Implementation Details

The appendix specifies the production translation backend, scalable execution procedure, prompting strategy, and target-only quality proxy used to reproduce KletterMix. It also documents corpus diagnostics and benchmark-reporting conventions.

  • Translation Backend Selection: FP8 matched FP16 within 0.001 absolute XCOMET overall and remained effectively tied on English–German, supporting the selected precision.NVFP4 was competitive in aggregate quality but showed operational instability in qualitative checks.
  • Translation Backend Selection: 7.06 requests/s, 5,633 output tokens/s, and 10,301 total tokens/s were achieved by the strongest FP8 serving configuration.The configuration used 8-way tensor parallelism, MTP-2 speculative decoding, max_num_batched_tokens 16,384, and max_num_seqs 1,024.
  • Translation Configuration and Execution: The pipeline uses length-aware routing, document-preserving chunking, dynamic target budgeting, shard-wise execution, and reproducible resumable serving.Documents fitting the source budget use one pass; longer documents are sentence-segmented into chunks of up to 20k source tokens with a 2k-token previous-translation context.
  • Prompting: Prompts request only German translation and use previous German context solely for discourse continuity in chunked documents.Explicit source and target delimiters help preserve chunk boundaries and reduce copied context or extraneous output.
  • Quality Estimation: The deployed target-only proxy captures language, text shape, length, character composition, lexical diversity, and repetition signals but cannot directly measure semantic adequacy.It is validated against COMETKiwi and intended as a scalable corpus-level signal rather than a replacement for source-aware evaluation or human inspection.
  • Benchmark and Corpus Diagnostics: Core Avg. is the unweighted mean of MMLU, PIQA, HellaSwag, and ARC-C, with task cells reported as accuracy ± evaluation-set standard error.The appendix also documents cluster-level corpus statistics, including token distribution across inherited source clusters.

B.2 Cluster Labeling Procedure

Cluster labels are assigned to fixed inherited source-cluster identifiers using the highest-scoring translated examples, then reported with descriptive summaries and quality statistics. The procedure includes qualitative checks illustrating translation failure modes and successful renderings.

  • Labeling Basis: Cluster identifiers are inherited from ClimbMix rather than recomputed from translated German text, so labels describe fixed source clusters.The procedure assigns human-readable topic names to make cluster-level corpus statistics easier to interpret.
  • Sampling and Prompting: 100 highest-scoring COMETKiwi examples per cluster are formatted into cluster-specific prompts for topic-label inference.Each example retains its document identifier, source-cluster identifier, German text, and COMETKiwi score.
  • Limitations: The inferred labels are descriptive metadata rather than ground-truth topic assignments for every document in a cluster.Selecting top-scoring examples biases annotation toward each cluster’s high-quality core and reduces noisy or truncated examples dominating the label.
  • Qualitative Validation: Qualitative inspections include refusal, wrong-language variety, translated code identifiers or keywords, and fluent scientific and academic examples.The examples pair source and German text with proxy scores to illustrate practical translation outcomes.

C.2 Extended Training Results

The extended-results appendix reports validation perplexity and next-token accuracy for matched 12B-token runs and supplies supplementary curves for KletterMix filtering ablations. These experiments use independent runs from a common checkpoint and isolate the annealing-mixture effect.

  • Validation Results: Table 11 reports final in-domain validation perplexity and next-token accuracy for matched 12B-token runs.Each model is evaluated on its own training domain’s held-out validation set, while filtered rows use KletterMix’s held-out validation set.
  • Filtering Ablations: Figure 7 adds validation perplexity and next-token accuracy curves for KletterMix filtering ablations.The supplementary curves extend the primary training and validation loss curves reported in the main text.
  • Experimental Design: The annealing experiment starts every run from the same pretrained checkpoint and varies only the fraction of ClimbMix tokens replaced by KletterMix.This isolates the effect of the annealing mixture across independent runs.
  • Experimental Design: Optimizer, learning-rate-scheduler, iteration-counter, and random-number-generator states are initialized afresh for every annealing run.Model parameters are loaded from the OLMo 3 7B Stage 1 checkpoint after conversion to a native Megatron–Bridge checkpoint.

C.3.2 Optimization and compute

The 7B annealing study holds optimization and compute settings fixed while varying KletterMix’s share of the annealing mixture. The tested mixtures range from 0% to 20% KletterMix, with German data selected by proxy-ranked quality.

  • Compute Configuration: All variants use one node with eight NVIDIA B200 GPUs, global batch 256 sequences, and 2,097,152 tokens per optimizer step.Tensor, pipeline, and context parallelism are one, yielding data parallelism of eight.
  • Training Schedule: 5,723 optimizer steps process 12,002,000,896 tokens under the 12B-token target budget.The learning rate warms up for 58 steps and then decays linearly to zero.
  • Mixture Design: x% KletterMix + (100 − x)% ClimbMix defines the annealing mixtures for x ∈ {0, 5, 10, 15, 20}.The implementation keeps an 80% ClimbMix base fixed and replaces one additional 5% slot at each mixture level.
  • Data Selection: German data is selected from the proxy-filtered pool with q̂proxy ≥ 0.60 by prioritizing the highest-scoring documents until the token budget is filled.The resulting selected material has proxy scores between 0.70 and 0.90 as an outcome of ranked selection, not a separately chosen band-pass filter.

C.3.4 Evaluation and detailed results

The evaluation compares an unannealed Stage 1 checkpoint with five matched annealing mixtures using unweighted German and English four-task means. Increasing KletterMix share improves German performance while English performance remains broadly stable, though error bars represent evaluation standard error rather than variation across training seeds.

  • Evaluation setup: The evaluation uses German and corresponding English versions of MMLU, PIQA, HellaSwag, and ARC-Challenge, reporting unweighted four-task means.The original Stage 1 checkpoint is the unannealed reference, and five annealed models are evaluated at step 5,723.
  • Caveat: The error bars quantify standard error and do not measure variation across independently trained seeds.This limits what the plotted uncertainty can establish about training-run variability.
  • Evaluation setup: The horizontal reference in Figure 8 is the unannealed Stage 1 checkpoint, while colored points represent five matched annealing mixtures with standard errors.The figure separates German results in panel (a) from English results in panel (b).
  • Results: 45.4% German and 66.3% English are achieved by the matched 0% KletterMix control after ClimbMix-only annealing.These values establish the baseline for the mixture comparison.
  • Results: 4.5 German percentage points is the gain from the 0% control to the 20% KletterMix mixture, while English changes by +0.3 points to 66.6%.German means increase from 45.4% at 0% to 49.9% at 20%, with intermediate values of 47.9%, 48.5%, and 48.9%.
  • Results: The 20% mixture has the highest displayed point estimate on all four German tasks, whereas English task scores remain comparatively stable.Figure 8 reports no broad English degradation as KletterMix share increases.

D Dataset Summary

KletterMix is a 725B-token German dataset released as sharded JSONL with a single training split. The release is intended to strengthen German-language modeling and controlled research, while retaining standard large-corpus risks and requiring careful documentation and release conditions.

  • Dataset contents: KletterMix contains 725B GPT-2 tokens of German-language text and is released under the CC-BY-NC-4.0 license.The dataset is distributed as sharded JSONL files.
  • Dataset contents: The dataset is distributed as sharded JSONL files with a single train split and structured record fields.The supplied passage introduces the record schema but does not enumerate those fields.
  • Implications: KletterMix can help reduce the gap between English and German pretraining resources, support German-language models, and enable controlled research on translated pretraining data.These are stated positive implications of the dataset.
  • Risks and release: Large-scale pretraining can amplify risks including misleading text, biased or stereotyped outputs, privacy leakage, and downstream misuse.The passage states that these risks are not unique to KletterMix.
  • Risks and release: The release therefore treats documentation, provenance preservation, quality filtering, transparent conditions, and intended-use statements as necessary safeguards.These practices are presented as necessary parts of the dataset release.
Loading 2606.03773v2…