PaperlayerView exact revision on arXiv

Source-linked AI summary

A Sovereign, Open-Source Foundation Model for German and English

Soofi-Team, :, Benedikt Droste, David Fitzek, Ruben Härle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom Röhr, Sebastian von Rohrscheidt, Jörg Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim Köhler, Alexander Löser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Björn Plüster, Patrick Putzky, Mehdi Ali, Michael Fromm, Max Lübbering

arXiv:2607.09424v3cs.CLcs.AIcs.LG

TL;DR

German remains underrepresented in general-purpose models, while dense Transformers become costly for long-context, high-concurrency serving. Soofi S addresses these gaps with a sovereign, open-source hybrid Mamba–MoE model, matching dense 14–27B models on aggregate English and German performance while delivering 8–9× their decode throughput at 40K context.

  • Problem

    German is underrepresented in general-purpose models, and dense full-attention models become increasingly costly as long-context concurrency makes their KV caches dominate serving.

    Source: §1 · p. 2

  • Method

    Soofi S combines a hybrid Mamba–MoE architecture with German-upweighted, three-phase pretraining and a near-constant per-sequence cache.

    Source: §3 · p. 9, §4.4 · p. 31

  • Results

    Soofi S is the strongest fully open German–English model, matching dense 14–27B models on aggregate performance and achieving 8–9× their decode throughput at 40K context.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Soofi S demonstrates that a fully open model can combine strong German–English capability with substantially more efficient long-context serving than dense alternatives.

    Source: §1 · p. 4, §6 · p. 35

    Under stricter open-data standards, Soofi S is not fully redistributable because the commercially licensed Genios corpus comprises 1.3% of Phase 1 effective tokens.

    Source: §3.5 · p. 13

Abstract

from arXiv · show

We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

1 Introduction

Soofi S 30B-A3B addresses gaps in substantive openness, German–English capability, and long-context serving efficiency through a hybrid Mamba–MoE architecture. It combines competitive benchmark performance with transparent, reproducible training and data artifacts.

  • The hybrid Mamba–MoE design uses Mamba-2 layers with fixed-size recurrent state, only 6 of 52 layers maintaining a KV cache, and 3.2 of 31.6 billion parameters activated per token [16, 61, 60].This provides the capacity of a 30B network at roughly the inference cost of a 3B one and targets memory-bandwidth limits in long-context, high-concurrency serving.Source: §1 · pp. 2–3
  • Soofi S is the strongest fully open model on English and German benchmarks, matches or outperforms every European sovereign baseline on German benchmarks, and matches dense 14–27B models on aggregate performance.The comparison covers models with comparable or larger active size, while Soofi S uses a fraction of their active-parameter cost.Source: §1 · p. 3
  • The release provides per-source and per-language token accounting, German–English–code mixing ratios, construction scripts, and dataset-card versus tokenizer-exact token counts.These materials address the field’s common weight-only releases that omit data and reproducibility details.Source: §1 · pp. 2–4
  • Soofi S publishes the full Warmup–Stable–Decay schedule, optimizer, hyperparameters, phase token budgets, phase boundaries, selected checkpoints, and training and evaluation code under permissive licenses.The paper also reports design and data ablations to expose the rationale behind the final configuration.Source: §1 · p. 4
  • At 40K context and batch 32, Soofi S achieves 8–9× the aggregate decode TPS/GPU of dense 14–27B models, with throughput essentially flat from 4K to 256K.Its near-constant per-sequence cache avoids the degradation seen in full-attention models; Figure 16 reports the serving comparison.Source: §1 · p. 4

2 Model Architecture and Training

Soofi S uses the Nemotron 3 Nano hybrid Mamba–Transformer MoE architecture, with sparse GQA layers limiting cache growth, and trains through staged WSD optimization culminating in a distinct 1M-token phase. The full run spans roughly 27T tokens, exhibits stable optimization, and uses sovereign German HPC infrastructure.

2 Model Architecture and Training

  • Soofi S follows Nemotron 3 Nano [61] [60] with 52 layers interleaving 23 Mamba-2, 23 granular MoE, and 6 GQA layers [16] [77] [19] [15] [41] [1].The model has ∼31.6B total parameters, ∼3.2B active per forward pass, and only the six GQA layers maintain a KV cache.Source: §2 · p. 4, Table 1 · p. 6

2.1 Optimization and Hyperparameters

  • Training uses AdamW [49] with WSD [35] [30] across ∼20T-token base pretraining, ∼5T-token main annealing, and a further ∼1.58T-token constant-learning-rate stage.The final ∼0.30T-token decay stage was run for completeness but its checkpoints were discarded because they showed no clear benchmark improvement.Source: §2.1 · pp. 6–7

2.2 Long-Context Extension

  • The long-context phase trains 2,000 iterations at 1,048,576 tokens, using context parallelism 16, global batch size 48, and 50,331,648 tokens per optimizer step.It uses a constant 1e−5 learning rate after 100 warmup iterations, enables HybridEP, and omits intra-document masking following NVIDIA’s recipe [61].Source: §2.2 · p. 7

2.3 Training Dynamics

  • Across the ∼27T-token run, WSD transitions from a warmup and stable plateau to minus_sqrt decay; loss drops during annealing, gradient norms remain stable, and throughput is mostly steady.Throughput dips are attributed to checkpointing, evaluation, and restart iterations rather than training-dynamics changes.Source: Figure 2 · p. 5, §2.3 · pp. 7–8

2.4 Compute Infrastructure

  • Training used up to 512 NVIDIA B200 GPUs and approximately 253,000 GPU-hours on Deutsche Telekom’s Industrial AI Cloud in Munich, with expert-parallel groups kept within nodes.This placement routes MoE all-to-all communication over intra-node NVLink instead of the slower inter-node fabric and supports the model’s sovereign German deployment context.Source: §2.4 · p. 8

3 Pretraining Data

Soofi S uses a three-phase, quality- and skill-based pretraining curriculum spanning approximately 27 trillion consumed tokens, with German deliberately up-weighted and Phase 3 extending context to 1M tokens. The corpus is documented at per-source granularity and is largely reconstructable, while the commercially licensed Genios component defines the stated openness boundary.

3 Pretraining Data

  • Phase 1 uses broad diversity, Phase 2 concentrates high-quality skill data, and Phase 3 extends context with length-bucketed data up to 1M tokens.The effective-token mixture is 23,051.13B in Phase 1, 6,303.0B in Phase 2, and 188B in Phase 3.Source: §3 · p. 9, Figure 3 · p. 9

3.1 Quality Tiers, Synthetic Data, and Epoching

  • High-quality and synthetic sources are up-sampled through epoch multipliers, while evaluated medium-quality pools with zero epochs are excluded from training.For Phases 1 and 2, token counts are dataset-card estimates rather than exact model-token ledgers; Phase 3 counts are exact Nemotron-3-tokenizer counts.Source: §3.1 · p. 10

3.4 Phase 3: Long-Context Extension

  • Phase 3 trains 100.66B tokens at 1M-token sequence length, about 53% of its pool, and stops after no further loss improvement is observed.The pool contains approximately 188.5B tokens from ∼21 million documents across nine length buckets, but several extreme buckets are sparse or empty.Source: §3.4 · p. 12

3.5 Data Provenance and Release

  • The release provides complete per-source mixture accounting, raw and effective token counts, epoch multipliers, weights, checkpoints, and training and evaluation code, but Genios is not redistributable.Genios comprises 1.3% of Phase 1 effective tokens; approximately 99% of the mixture can be independently reconstructed.Source: §3.5 · p. 13
  • The corpus combines code-aware web extraction, curated repository code with synthetic transformations, German and English web data, mathematics, STEM, reasoning, and pretraining-stage SFT.Nemotron-CC-Code-v1 preserves code formatting and context, while specialized synthetic data includes math, STEM, reasoning, instruction-following, science, finance, software engineering, safety, and multilingual subsets.Source: §3.5.1 · p. 13, §3.5.2 · p. 13, §3.5.3 · p. 14

3.6 Data Mixture and Ordering

  • Phase 2 shifts from Phase 1 breadth toward skill density, raising German from 7.2% to 15.3% while increasing code, reasoning, and SFT emphasis.Phase 1 is dominated by English Web at 50.3%, whereas Phase 2 assigns 36.4% to English Web, 16.4% to Code, 11.8% to Reasoning, and 8.5% to SFT.Source: §3.3 · p. 11, §3.6 · p. 15

4 Evaluations

Soofi S leads the open-source comparison on English and German aggregates, while remaining competitive with larger open-weight models and improving over its architecture-matched baseline. Its evaluation also documents a GPQA contamination incident and shows substantial long-context serving advantages from its hybrid Mamba–MoE design.

4.1 Soofi S vs Open-Source Models

  • Soofi S is strongest among open-source models, leading English aggregate by +1.5 over Olmo 3 32B and German aggregate by +5.9 over Apertus 70B.It also ranks first on four of five code benchmarks and every German benchmark in this comparison.Source: §4.1 · pp. 16, 18, Figure 4 · p. 18

4.2 Soofi S vs Open-Weight Models

  • Soofi S scores 77.3 on the English aggregate and 85.3 on the German aggregate, improving over Nemotron 3 Nano by +0.6 and +4.6, respectively.Qwen3.5 35B-A3B leads both aggregates in the open-weight subset, while Soofi S remains close to Gemma 3 27B and Ministral 3 14B.Source: §4.2 · p. 23, Table 5 · p. 24
  • Soofi S leads the open-weight comparison on HumanEval (73.8), MBPP (70.2), and MBPP-DE (84.2), but trails Nemotron on HumanEval-DE and LBPP.It ties Qwen3.5 35B-A3B on INCLUDE-DE at 61.2 and stays within a point of Ministral 3 14B on BBH, AGIEval, and ARC-Challenge.Source: §4.2 · pp. 23, 25, 26
  • The German–English recipe improves Nemotron-relative German aggregate by +4.6 and GLP-DE by +15.1 while also improving English aggregate by +0.6 and Math-EN by +7.5.The main regressions are Math-DE aggregate −1.4, NaturalQuestions −1.3, GSM8K −0.4, and HellaSwag −1.3.Source: §4.2 · p. 27, Figure 14 · p. 29

4.3 GPQA Contamination Disclosure

  • The audit found GPQA evaluation items in QA-base because split selection relied on the label train, and corrected aggregates and releases now exclude GPQA and validate splits semantically.GPQA-Diamond rose gradually from 32.3 to 43.4 without a trajectory discontinuity, so monitoring did not reveal the contamination.Source: §4.3 · pp. 29–30

4.4 Serving Efficiency

  • At 40K context and batch size 32, Soofi S sustains 4.82k TPS/GPU, 9.2× Ministral 3 14B, while its cache is about 6 KB per token per sequence.At 256K context, Soofi S remains the fastest complete prefill sweep at 372.7 seconds, and decode throughput stays nearly flat from 4K to 256K.Source: §4.4 · pp. 31–32

5 Related Work

Soofi S combines fully open, auditable pretraining with a German–English curriculum and sparse hybrid Mamba–Transformer MoE architecture. It occupies a distinct position between broad European multilingual sovereignty efforts and efficient international open-weight baselines.

Open language-model pretraining

  • Soofi S extends the fully open model tradition by releasing auditable artifacts and exact per-source accounting, rather than only weights and inference code [27] [57] [64] [5].Its scope includes a German–English base model and a sparse hybrid Mamba–Transformer MoE backbone.Source: §5 · pp. 32–33

Pretraining data acquisition and filtering

  • Soofi S reports source identifiers, raw tokens, epoch multipliers, effective counts, and excluded sources for every corpus phase, enabling finer-grained mixture auditing.This extends prior corpus-construction and filtering work beyond aggregate token counts.Source: §5 · p. 33

Training curricula and optimization recipes

  • Its three-phase curriculum progresses from diversity to high-quality and skill-focused data, then document-length buckets, with explicit bilingual token-budget reallocation.The phases emphasize German alongside web, code, mathematics, reasoning, SFT-formatted, and long-context data.Source: §5 · pp. 33–34

Efficient sparse and hybrid architectures

  • Soofi S adopts a 30B-A3B hybrid Mamba–Transformer MoE design combining sparse activation with near-constant decoding state, enabling a clean comparison against Nemotron 3 Nano [61].The related architectural lines address weight cost through MoE sparsity and sequence-state cost through Mamba-style recurrence.Source: §5 · p. 34

Multilingual and European language models

  • Unlike broad European multilingual models, Soofi S concentrates its data mixture, annealing, and evaluation on bilingual German–English depth [3] [53] [52] [74] [26] [5].This complements coverage-oriented efforts by testing capability and efficiency in a dedicated bilingual deployment setting.Source: §5 · p. 34

Positioning of Soofi S

  • Soofi S combines a fully documented European run, exact bilingual data accounting, and efficient hybrid architecture in a configuration not covered by prior work.It targets the gap between European sovereignty efforts and efficient international open-weight baselines.Source: §5 · pp. 34–35

6 Conclusion · A Author Contributions

Soofi S 30B-A3B is a sovereign, open-source German–English MoE hybrid Mamba–Transformer model that combines efficient inference with competitive aggregate capability. The conclusion emphasizes its transparent release and identifies open post-training, broader German evaluation, and higher-quality German data as priorities for future work.

6 Conclusion

  • The model combines 52-layer Mamba-2, Grouped-Query Attention, and granular MoE components with a three-phase curriculum spanning approximately 27 trillion tokens and usable context up to 1M tokens.German was up-weighted to 7.2% of the stable phase and 15.32% of the annealing mixture, more than triple the multilingual share of the reference recipe.Source: §6 · p. 35
  • Soofi S matches dense 14–27B models on English and German aggregates while achieving the best measured code aggregates and outperforming every European sovereign baseline.It activates roughly 3B of ∼30B parameters per token and matches or outperforms European sovereign baselines on every German benchmark, often by 10–30 points.Source: §6 · p. 35
  • Soofi S is released with weights, intermediate checkpoints, per-source token accounting, hyperparameters, and training and evaluation code to support auditing and rebuilding.Licensed data sources are documented through aggregate statistics and exact mixture accounting rather than redistributed raw text.Source: §6 · p. 35
  • Future work will add open supervised fine-tuning and large-scale reinforcement learning for instruct and reasoning variants, including modular reasoning approaches.The conclusion references modular reasoning in the spirit of FlexOlmo and Bar [78].Source: §6 · pp. 35–36
  • Future evaluation will expand toward broader and deeper German benchmark suites.This is identified as one of three axes for extending Soofi S.Source: §6 · p. 36
  • The principal bottleneck for further gains is continued scaling of the high-quality German data pipeline.This is the third future-work axis identified in the conclusion.Source: §6 · p. 36

A.1 Training

The training work covered the full development lifecycle, from pretraining-stack evaluation and model-methodology validation to infrastructure, scaling, execution, debugging, and experiment tracking.

  • The team developed and evaluated the pretraining stack.Source: §A.1 · p. 48
  • The team defined the model architecture and training methodology and validated framework correctness.Source: §A.1 · p. 48
  • The team benchmarked compute infrastructure and tuned the cluster interconnect.Source: §A.1 · p. 48
  • The team optimized distributed-training scaling, memory use, and throughput.Source: §A.1 · p. 48
  • The team managed training execution, stability analysis, emergency debugging, framework bug fixes, and experiment tracking.Source: §A.1 · p. 48

A.2 Data

The data section covers acquisition, mixture design, curation, preprocessing, tokenization, and context-specific experimentation, with responsibilities distributed across the listed contributors.

Data acquisition

  • Base-model data acquisition is attributed to Michael Fromm, Alex Jude, Abbas Khan, Ruben Härle, Maurice Kraus, Jan Pfister, and Daniil Gurgurov.Source: §A.2 · p. 48

Pretraining data mixture

  • Pretraining data-mixture work is attributed to Michael Fromm.Source: §A.2 · p. 48

Data curation infrastructure and experimentation

  • Data-curation infrastructure and experimentation involve Michael Fromm, Alex Jude, Abbas Khan, Ruben Härle, Maurice Kraus, Richard Rutmann, Mehdi Ali, Max Lübbering, and Maximilian Idahl.Source: §A.2 · p. 48

Data preprocessing / tokenization pipeline

  • Data preprocessing and tokenization are attributed to Richard Rutmann, Max Lübbering, and Alex Jude.Source: §A.2 · p. 48

Mid- and long-context data curation and experimentation

  • Mid- and long-context data curation and experimentation involve Michael Fromm, Alex Jude, Abbas Khan, Ruben Härle, Maurice Kraus, Sebastian Sztwiertnia, Tom Röhr, and Sebastian von Rohrscheidt.Source: §A.2 · p. 48

A.3 Evaluation · A.4 Other

The evaluation work covered methodology and infrastructure, while broader contributions included mentorship, advising, program management, strategy, technical leadership, and cross-workstream collaboration.

A.3 Evaluation

  • Evaluation methodology and infrastructure were handled by Maximilian Idahl, Benedikt Droste, Alex Jude, and Abbas Khan.Source: §A.3 · p. 48

A.4 Other

  • Mentorship, advising, program management, and broader strategy were contributed by Nicolas Flores-Herr, Simon Gottschalk, Jörg Bienert, Kristian Kersting, Andreas Hotho, Alexander Löser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Björn Plüster, and Patrick Putzky.Source: §A.4 · p. 48
  • Technical leadership and cross-workstream contributions came from Mehdi Ali, Michael Fromm, Max Lübbering, Sebastian Sztwiertnia, and Tom Röhr.Source: §A.4 · p. 48

B Detailed Pretraining Data Composition

The appendix provides full per-source token accounting for the three-stage pretraining mixture, covering diverse pretraining, high-quality annealing, and long-context extension. It distinguishes enumerated but excluded sources from consumed data and documents how source counts and phase shares are reported.

Appendix scope

  • The appendix accounts for data across Phase 1, Phase 2 annealing, and Phase 3 long-context extension, with zero-epoch rows marking enumerated sources excluded from training.Tables 6–7, 8–9, and 10–11 correspond to the three phases and underpin the mixture flow diagram in Figure 3.Source: B Detailed Pretraining Data Composition · p. 49

Phase 1: diverse pretraining

  • Phase 1 composition is reported per source using raw tokens, epochs, effective tokens, and each source’s share of Phase 1 effective tokens.Raw and effective counts are approximate because they come from source dataset cards and may use different tokenizers.Source: Table 6 · p. 49

Phase 1: category composition

  • Phase 1 is also summarized by category across the complete 23,051.13B effective-token mixture, with Nemotron 3 Nano’s reported shares provided for comparison.The category totals cover 100% of the Phase 1 mixture.Source: Table 7 · p. 50

Phase 2: high-quality annealing

C Proxy Data-Mixture Ablations

Controlled 100B-token proxy ablations identified D01 as the most stable German–English mixture, outperforming alternatives on the rank-based bilingual selection criterion. This result motivated the full-run data family while the final curriculum added separate annealing, German up-weighting, and long-context stages.

C Proxy Data-Mixture Ablations

  • The proxy study fixed the English backbone at roughly 90–93% of tokens and varied only the German or multilingual tail to isolate non-English data-mixture effects.Candidates tested German web, PDF, synthetic, commons, news, and lightly multilingual sources under a cheaper setup that was not intended to predict absolute final-model performance.Source: C Proxy Data-Mixture Ablations · pp. 52–53
  • All candidates used identical Qwen3-1.7B-based training, four Leonardo nodes, and evaluations at roughly 10B-token intervals, with the comparable endpoint at step 47,683 and ∼100B tokens.The selection criterion averaged ranks over English and German bits-per-byte plus normalized rank-choice accuracy, using only the terminal checkpoint for final averages.Source: C Proxy Data-Mixture Ablations · pp. 53–54
  • R100B = 1.75 and Rall = 3.55 make D01 the clear proxy winner, balancing English and German likelihood and accuracy across terminal and full-trace evaluations.D01 is best on English bits-per-byte and German normalized accuracy, second on English normalized accuracy, and near the strongest group on German bits-per-byte.Source: C Proxy Data-Mixture Ablations · p. 54, Table 13 · p. 55
  • D01 outperformed specialized alternatives: FinePDF-only D08 collapsed on German bits-per-byte, MultiSynt-heavy D09/D10 lacked the best bilingual aggregate, and multilingual D04 ranked second on German bits-per-byte.D12 was the strongest runner-up and slightly better on English normalized accuracy, while D06 achieved the lowest German bits-per-byte but ranked only sixth terminally and seventh across the trace.Source: C Proxy Data-Mixture Ablations · pp. 54–55
  • D01 was selected for the full run as a balanced family of HPLT-DE, German FinePDFs, MultiSynt, German-Commons, and Genios rather than a single-source or broad-multilingual optimum.The final 26.68T-token curriculum did not copy the proxy recipe mechanically: it separated pretraining and annealing, up-weighted German in both, and added long-context training.Source: C Proxy Data-Mixture Ablations · pp. 55–56

D Further Dataset Information

The Genios corpus is a commercially licensed German newspaper and trade-press collection spanning 193.1M articles from 916 publications between 2010 and 2025. Because redistribution is prohibited, only aggregate corpus statistics are reported.

Genios

  • Genios contains 193.1M articles, approximately 57.6B words, from 916 publications spanning 2010–2025.Per-year volumes range from 9.7M to 14.4M documents, and articles average approximately 298 words.Source: D Further Dataset Information · p. 56
  • The collection is dominated by regional daily newspapers, with national outlets and specialist trade and academic periodicals forming a long tail.Examples include Rheinische Post, Rhein-Zeitung, and Neue Westfälische.Source: D Further Dataset Information · p. 56
  • Genios was purchased under a commercial license and cannot be redistributed, so the paper reports aggregate corpus statistics only.Source: D Further Dataset Information · p. 56

E Further Base Model Evaluations

Long-context evaluation shows that Soofi S is near-parity with its architecture-matched reference across 1M tokens when excluding CWE, but suffers a sharp CWE degradation beyond 32K. The authors attribute this limitation to long-context data composition rather than architecture or training volume.

Long-context

  • On the full 13-subtask RULER suite, Soofi S trails Nemotron by 6.8 points on average, scoring 50 versus 60 at 1M tokens.The evaluation spans input lengths from 4K to 1M tokens and uses architecturally identical models.Source: E Further Base Model Evaluations · pp. 56–57
  • Excluding CWE, Soofi S remains within 4.4 points on average and scores 54 versus 60 at 1M, indicating effective parity across the full context window.On CWE alone, the models match through 32K, after which Soofi S falls to approximately 3% at 256K–1M while Nemotron retains 60–64%.Source: E Further Base Model Evaluations · p. 57, Figure 18 · p. 57
  • The authors attribute the CWE regression to long-context data composition: Soofi S lacks dedicated retrieval- and aggregation-oriented data, unlike Nemotron’s targeted mixture.The models share the same backbone, and their long-context token budgets are comparable: approximately 100.66B versus 121B.Source: E Further Base Model Evaluations · pp. 57–58
  • CWE degradation beyond 32K is identified as a known limitation and a target for adding retrieval- and aggregation-style synthetic data in the 32K–1M range.The proposed next iteration focuses on correcting the isolated CWE weakness while preserving near-parity on other subtasks.Source: E Further Base Model Evaluations · pp. 57–58

F Checkpoint Merging Ablations

Checkpoint merging did not produce a uniformly better model than the selected iter_1056000 checkpoint. Final-window merges slightly improved German metrics but regressed English metrics, so iter_1056000 remained the primary base checkpoint.

F Checkpoint Merging Ablations

  • The ablations averaged weights from checkpoints in the same training run or direct continuations, excluding optimizer states and adding no new training tokens.Uniform, exponential, and two-checkpoint manual blends were evaluated against iter_1056000, the checkpoint used for the base-model evaluation and long-context initialization.Source: F Checkpoint Merging Ablations · p. 58
  • German bits-per-byte improved from 0.3656 to 0.3637 and German normalized accuracy from 80.10 to 80.34 for the strongest final-window merges, whereas English bits-per-byte worsened from 0.4390 to 0.4396 and English normalized accuracy from 77.39 to 77.28.These German-side gains were accompanied by small English regressions and therefore did not clearly dominate the selected checkpoint.Source: Evaluation · p. 59
  • The uniform average over all 13 final-window checkpoints achieved the best aggregate rank, while exponential averaging with α = 0.8 was nearly tied, but neither uniformly surpassed iter_1056000.The comparison used English and German bits-per-byte plus English and German normalized accuracy, summarized by average rank; one merge was excluded for incomplete bits-per-byte coverage.Source: Evaluation · pp. 58–59, Table 16 · p. 61

Related papers

Loading 2607.09424v3…