PaperlayerView exact revision on arXiv

Source-linked AI summary

A Sovereign, Open-Source Foundation Model for German and English

Soofi-Team, :, Benedikt Droste, David Fitzek, Ruben Härle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom Röhr, Sebastian von Rohrscheidt, Jörg Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim Köhler, Alexander Löser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Björn Plüster, Patrick Putzky, Mehdi Ali, Michael Fromm, Max Lübbering

arXiv:2607.09424v3cs.CLcs.AIcs.LG

TL;DR

German-focused open models face gaps in capability and long-context serving efficiency. Soofi S addresses these with a hybrid Mamba–MoE model, matching dense 14–27B models on aggregate English and German performance while leading fully open models and improving serving throughput.

  • Problem

    German is underrepresented in general-purpose models, while full-attention models become costly as long-context KV caches grow under concurrent serving.

  • Method

    Soofi S combines a hybrid Mamba–MoE architecture with German-upweighted three-phase pretraining and a near-constant per-sequence recurrent serving state.

  • Results

    Soofi S is the strongest fully open German–English model, matches dense 14–27B aggregate performance, and delivers 8–9× dense-model decode TPS/GPU at 40K context and batch 32.

  • Takeaways & Limitations

    Soofi S demonstrates that a sovereign, fully open model can combine strong German–English capability with substantially more efficient long-context serving.

  • Takeaways & Limitations

    Under stricter open-data standards, Soofi S falls short because the commercially licensed Genios corpus cannot be redistributed, although its mixture is documented in aggregate.

Abstract

from arXiv · show

We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

1 Introduction

Soofi S 30B-A3B addresses gaps in substantive openness, German–English capability, and deployment efficiency through a hybrid Mamba–MoE architecture and comprehensive artifact release. It targets long-context serving with near-constant cache growth while matching larger dense models and leading fully open evaluations.

  • Motivation: Weight-only releases often omit the data, recipes, and decisions needed for reproduction or auditing, while German remains underrepresented in general-purpose multilingual models.Dedicated European efforts have prioritized openness and language coverage over frontier capability.
  • Approach: Soofi S combines substantive openness, German–English training, and a hybrid Mamba–MoE architecture to address all three deployment gaps.The model is designed around Mamba layers, sparse MoE activation, and extensive release artifacts.
  • Evaluation: Soofi S is the strongest fully open model on the paper’s English and German benchmarks and matches or outperforms every compared European sovereign baseline on German benchmarks.It also matches dense 14–27B international models on aggregate English and German performance at lower active-parameter cost.
  • Open release: The release includes complete corpus statistics, per-source and per-language token accounting, reproducible construction scripts, the full training recipe, and design ablations.Selected checkpoints plus training and evaluation code are also planned under permissive licenses.
  • Serving efficiency: 8–9× higher aggregate decode TPS/GPU than dense 14–27B models is measured at 40K context and batch 32, enabled by a near-constant per-sequence cache.Aggregate throughput stays essentially flat from 4K to 256K, while full-attention models degrade.

2 Model Architecture and Training

Soofi S uses the Nemotron 3 Nano hybrid Mamba–Transformer MoE architecture, combining sparse experts with limited GQA caching for efficient long-context inference. Training spans approximately 27T tokens across stable pretraining, annealing, and a distinct 1M-token long-context phase, using WSD scheduling and up to 512 NVIDIA B200 GPUs.

  • Model architecture: Soofi S uses a 52-layer hybrid architecture with 23 Mamba-2 layers, 23 granular MoE layers, and 6 GQA layers distributed through the network.The design follows the Nemotron 3 Nano reference configuration and uses shared experts in the MoE layers.
  • Model architecture: ∼31.6B total parameters include only ∼3.2B active per forward pass, or ∼3.6B including embeddings, while only the 6 GQA layers maintain a KV cache.The architecture therefore keeps KV-cache state limited to the sparse GQA components.
  • Training schedule: 19,999,984,975,872 tokens of base pretraining used the 20T EN/DE mixture, followed by 4,999,996,243,968 tokens of main annealing on the 5T high-quality EN/DE mixture.The main annealing stage applies WSD minus_sqrt decay from 1e−3 to 1e−5.
  • Training schedule: 11,920 iterations of final WSD annealing were run but discarded because their checkpoints showed no clear additional benchmark improvement over constant annealing.The discarded stage decayed the learning rate from 1e−5 to 0.
  • Long-context training: 2,000 iterations at 1,048,576-token sequences added approximately 100.66B tokens in a distinct long-context phase using context parallelism 16, micro-batch size 1, and global batch size 48.This phase used 50,331,648 tokens per optimizer step and AdamW optimization.
  • Training infrastructure: 512 NVIDIA B200 GPUs, corresponding to 64 DGX B200 nodes, were used for training on Deutsche Telekom’s Industrial AI Cloud in Munich.The system used fifth-generation NVLink/NVSwitch within nodes and NVIDIA Quantum-2 NDR InfiniBand between nodes.

3 Pretraining Data

Soofi S uses a three-phase pretraining curriculum with a large, explicitly accounted-for mixture spanning web, academic, code, mathematics, reasoning, instruction, and German data. The corpus emphasizes German and reproducibility, while its long-context stage is separately budgeted and documented under permissive openness terms.

  • Reproducibility and openness: Every source’s raw count, epoch multiplier, effective contribution, and public identifier are documented, enabling reconstruction where licenses permit; Genios is represented by aggregate statistics and exact accounting.Weights, intermediate checkpoints, training and evaluation code, and exact per-source accounting are released under permissive licenses.
  • Three-phase curriculum: 23,051.13B, 6,303.0B, and 188B effective tokens comprise Phases 1, 2, and 3, respectively, with Phase 2 increasing skill-oriented and German data.Figure 3 summarizes the effective-token mixture across the three training phases.
  • German weighting: ∼1.65T effective tokens, or 7.2% of Phase 1, come from German sources, exceeding the 5% multilingual share of the reference Nemotron 3 Nano mixture.The German component combines natural web and document text with machine-translated and synthetic sources.
  • Long-context extension: 2,000 optimizer steps at 1M-token sequences consumed approximately 100.66B tokens in the long-context stage, about 53% of its pool.Training stopped when additional long-context training produced no further loss improvement.
  • Phase 1 composition: Phase 1 allocates 50.3% to English Web, 8.0% to Academic & Wiki, 14.6% to Code, 10.4% to Reasoning, 7.2% to German, 6.0% to Mathematics, and 3.5% to SFT.High-quality and synthetic tiers run 2–6 epochs, while medium tiers run zero epochs.

4 Evaluations

Soofi S leads the open-source comparison across English, German, code, mathematics, and many knowledge and reasoning benchmarks, while remaining competitive with larger open-weight models. Its German-focused data recipe improves German performance without sacrificing English results, though German competition mathematics remains a clear gap.

  • Open-source comparison: Soofi S achieves the highest English aggregate (+1.5 over Olmo 3 32B) and German aggregate (+5.9 over Apertus 70B) among the open-source models.The comparison includes Alia 40B, EuroLLM 22B, Apertus 70B, and Olmo 3 32B.
  • Open-source comparison: Soofi S ranks first on four of five open-source code benchmarks, leading HumanEval by 10.8 points and MBPP-DE by 13.4 points.LBPP is the exception: Olmo 3 32B scores 32.1 versus 31.0 for Soofi S.
  • Open-source comparison: Soofi S ranks first on all open-source mathematics benchmarks, including Minerva-500 (+24.2 over Olmo 3 32B) and Minerva Math-EN (+27.0 over Olmo 3 32B).It also leads the subset on multiple knowledge, reasoning, and science benchmarks, including MMLU-STEM, MMLU-Pro, BBH, and ARC-Challenge.
  • Open-weight comparison: In the open-weight comparison, Soofi S scores 77.3 on the English aggregate and 85.3 on the German aggregate, trailing Qwen3.5 35B-A3B overall but improving on Nemotron by +0.6 and +4.6.It sets the best subset scores on HumanEval (73.8), MBPP (70.2), and MBPP-DE (84.2), and ties Qwen3.5 35B-A3B on INCLUDE-DE (61.2).
  • Recipe ablation: The German–English recipe raises GLP-DE by +15.1, German aggregate by +4.6, English aggregate by +0.6, code by +2.2, and Math-EN aggregate by +7.5 versus Nemotron.These pairwise results isolate the data recipe because Soofi S and Nemotron 3 Nano share the same architecture.
  • Limitations: Soofi S scores 56.0 on Minerva MATH-DE, trailing Qwen3.5 35B-A3B (76.5) and Gemma 3 27B (65.6), despite scoring 87.1 on GSM8K-Platinum-DE.This identifies German competition-style mathematics as the clearest capability gap to the frontier.

5 Related Work

Soofi S builds on open pretraining, transparent data and curriculum design, sparse and hybrid architectures, and European multilingual models. It combines these strands in a fully documented German–English model focused on bilingual depth, sovereignty, and efficient long-context serving.

  • Open language-model pretraining: Open releases increasingly provide weights, code, checkpoints, or training details, but many open-weight models still lack complete reproducibility artifacts.OPT, GPT-NeoX-20B, and Pythia exemplify progressively more transparent releases, whereas many newer models omit exact data mixtures, filtering, logs, or rejected sources.
  • Training curricula and optimization recipes: Modern pretraining separates token collection from token ordering and weighting, with curricula adapting data emphasis across training phases.Soofi S applies this philosophy to German–English training through diversity, quality-focused, and context-extension phases, with explicit bilingual token-budget reallocation and exact realized mixtures.
  • Efficient sparse and hybrid architectures: Sparse MoE and state-space approaches address complementary inference costs: MoE reduces active weight computation, while Mamba-style recurrence reduces sequence-state growth.Related hybrid systems combine Mamba-style sequence mixing, sparse attention, and MoE layers for long-context serving efficiency.
  • Multilingual and European language models: European multilingual models prioritize sovereignty, transparency, language coverage, and regulatory compatibility, whereas Soofi S concentrates on German–English bilingual depth.Its data mixture, annealing phase, and evaluation suite are designed for one bilingual deployment setting rather than broad multilingual coverage.
  • Positioning of Soofi S: Soofi S combines fully documented European pretraining, exact per-source bilingual accounting, and a sparse hybrid architecture for long-context, high-concurrency serving.The paper positions this configuration between broadly multilingual European sovereignty efforts and efficient international open-weight baselines.

6 Conclusion · A Author Contributions

Soofi S 30B-A3B is presented as a sovereign, open-source German-English MoE hybrid Mamba–Transformer model that reaches strong capability with low inference cost. The conclusion emphasizes both its competitive evaluations and unusually complete, auditable release, while identifying open post-training, German evaluation, and data scaling as future priorities.

  • 6 Conclusion: 3B of ∼30B parameters activate per token in Soofi S’s hybrid Mamba–Transformer architecture.The model combines Mamba-2, Grouped-Query Attention, and granular MoE layers.
  • 6 Conclusion: 27 trillion tokens were used for pretraining under a three-phase Warmup–Stable–Decay curriculum.The phases comprise diverse quality-tiered pretraining and high-quality annealing.
  • 6 Conclusion: 16 open base models were evaluated in a unified comparison, where Soofi S achieved the best English and German code aggregates.The code aggregates use HumanEval/MBPP averages, with LBPP reported separately.
  • 6 Conclusion: Soofi S also reached a Math-EN score essentially tied with the strongest model and second-best scores on GSM8K and English Minerva benchmarks.The English Minerva benchmarks are Minerva-500 and Minerva Math-EN.
  • 6 Conclusion: The release includes weights and artifacts needed to audit and rebuild the model, including full per-source token accounting and every hyperparameter.The accounting covers all three pretraining phases, while the hyperparameter record includes a discarded final annealing stage.
  • 6 Conclusion: Future work targets open post-training, broader German evaluation, and continued scaling of the high-quality German data pipeline.The proposed post-training includes SFT, large-scale RL, instruct and reasoning variants, and modular reasoning.

A.1 Training

The training section assigns responsibilities across pretraining-stack development, model methodology, infrastructure, distributed scaling, and execution stability. Contributions span evaluation, correctness validation, cluster optimization, memory and throughput tuning, debugging, and experiment tracking.

  • Training contributions: Pretraining-stack development and evaluation were led by Max Lübbering, Richard Rutmann, Timm Ruland, David Fitzek, and Mehdi Ali.
  • Training contributions: Model architecture, training methodology, and framework-correctness validation were handled by Timm Ruland, David Fitzek, Max Lübbering, and Richard Rutmann.
  • Training contributions: Compute infrastructure, cluster benchmarking, and interconnect tuning were handled by David Fitzek, Timm Ruland, Richard Rutmann, and Max Lübbering.
  • Training contributions: Training execution, stability analysis, emergency debugging, framework bug fixes, and experiment tracking were handled by Timm Ruland, David Fitzek, Max Lübbering, and Richard Rutmann.

A.2 Data

The data section assigns responsibility for base-model data acquisition, pretraining mixture design, curation infrastructure, preprocessing and tokenization, and mid- and long-context data experimentation.

  • Base model data acquisition: Base-model data acquisition was handled by Michael Fromm, Alex Jude, Abbas Khan, Ruben Härle, Maurice Kraus, Jan Pfister, and Daniil Gurgurov.
  • Pretraining data mixture: Michael Fromm was responsible for the pretraining data mixture.
  • Data curation infrastructure and experimentation: Data curation infrastructure and experimentation involved Michael Fromm, Alex Jude, Abbas Khan, Ruben Härle, Maurice Kraus, Richard Rutmann, Mehdi Ali, Max Lübbering, and Maximilian Idahl.
  • Data preprocessing / tokenization pipeline: Richard Rutmann, Max Lübbering, and Alex Jude handled the data preprocessing and tokenization pipeline.
  • Mid- and long-context data curation and experimentation: Mid- and long-context data curation and experimentation involved Michael Fromm, Alex Jude, Abbas Khan, Ruben Härle, Maurice Kraus, Sebastian Sztwiertnia, Tom Röhr, and Sebastian von Rohrscheidt.

A.3 Evaluation · A.4 Other

The evaluation work is credited to four contributors, while other contributions span evaluation infrastructure, mentorship and strategy, and technical leadership across workstreams.

  • A.3 Evaluation: Evaluation methodology and infrastructure were contributed by Maximilian Idahl, Benedikt Droste, Alex Jude, and Abbas Khan.
  • A.4 Other: Mentorship, advising, program management, and broader strategy were credited to Nicolas Flores-Herr, Simon Gottschalk, and Jörg Bienert.
  • A.4 Other: Mentorship, advising, program management, and broader strategy also included Kristian Kersting, Andreas Hotho, and Alexander Löser.
  • A.4 Other: The same contribution area also included Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Björn Plüster, and Patrick Putzky.
  • A.4 Other: Technical leadership and cross-workstream contributions were credited to Mehdi Ali, Michael Fromm, and Max Lübbering.
  • A.4 Other: Sebastian Sztwiertnia and Tom Röhr were also credited for technical leadership and cross-workstream contributions.

B Detailed Pretraining Data Composition

The appendix documents Soofi S’s pretraining mixture through per-source token accounting across three training phases. It distinguishes enumerated-but-excluded sources, epoch-adjusted effective tokens, and approximate versus exact token counts.

  • Scope: Per-source accounting covers Phase 1 diverse pretraining, Phase 2 high-quality annealing, and Phase 3 long-context extension.Tables 6–7 cover Phase 1, Tables 8–9 cover Phase 2, and Tables 10–11 cover Phase 3.
  • Inclusion rules: Zero-epoch rows identify sources that were enumerated but excluded from training.This convention is stated for the appendix and the phase-composition tables.
  • Counting methodology: Raw and effective token counts are approximate because source dataset cards may use different tokenizers.Exact tokenizer counts of consumed tokens are reported separately in Table 2.
  • Phase 1 composition: Phase 1 contains 23,051.13B effective tokens across categories, representing 100% of the Phase 1 mixture.The category breakdown includes a comparison with Nemotron 3 Nano’s reported shares.

C Proxy Data-Mixture Ablations

Controlled proxy ablations varied only the German or multilingual data while holding the English backbone fixed, identifying D01 as the strongest balanced bilingual mixture. D01 was selected for the full recipe, with later curriculum and long-context modifications retained.

  • Ablation design: The study varied only German or multilingual data against a fixed English backbone to choose a non-English budget without confounding English-data changes.All ablations used the same training configuration and differed only in tokenized dataset blend.
  • Proxy results: D01 achieved the best terminal average rank, R100B = 1.75, and the best full-trace average rank, Rall = 3.55.At the final checkpoint, it led English bits-per-byte and German normalized accuracy, ranked second on English normalized accuracy, and remained near the strongest group on German bits-per-byte.
  • Source-mixture patterns: D08’s almost pure German FinePDF mixture remained competitive on English but collapsed on German bits-per-byte, indicating that PDF text alone was too narrow.Heavy MultiSynt mixtures improved some German likelihood tasks without producing the best bilingual aggregate, while multilingual D04 was a strong candidate.
  • Task-group aggregates: D01 led the English likelihood, English math and QA bits-per-byte, German code bits-per-byte, and German grammar-fluency aggregates.Weaker German QA and math likelihood scores were offset by stronger German rank-choice and fluency performance for the intended German–English base-model profile.
  • Selection for the full run: The full run used D01 as its basis, then separately applied broad pretraining, high-quality annealing, German up-weighting, and a long-context extension.The proxy result justified selecting a balanced German data family but was not copied mechanically into the 26.68T-token curriculum.

D Further Dataset Information

The Genios corpus is a commercially licensed German-language archive collection covering newspaper and trade-press sources from 2010–2025. It contains 193.1M articles from 916 publications, totaling approximately 57.6B words.

  • Genios: 193.1M articles span 916 distinct German-language newspaper and trade-press publications in the Genios corpus.The collection was obtained from GBI-Genios under a commercial license.
  • Genios: ∼57.6B words cover the period from 2010–2025.Per-year volumes range from 9.7M to 14.4M documents.
  • Genios: Regional daily newspapers dominate the collection, alongside national outlets and specialist trade publications.Examples include Rheinische Post, Rhein-Zeitung, and Neue Westfälische.

E Further Base Model Evaluations

On RULER across 4K–1M-token contexts, Soofi S trails the architecturally identical Nemotron 3 Nano 30B-A3B mainly because of common-word extraction, while performing much closer when that subtask is excluded. The evidence attributes the regression to differences in long-context data mixtures and continuation recipes rather than backbone architecture.

  • Long-context: Soofi S is evaluated on RULER from 4K to 1M tokens against Nemotron 3 Nano 30B-A3B, whose identical backbone isolates long-context training effects.The comparison is conducted on the checkpoint after Soofi S’s long-context stage.
  • Long-context: 6.8 points: Soofi S trails the reference across the full 13-subtask suite, reaching 50 versus 60 at 1M tokens.This is the mean gap across input lengths.
  • Long-context: 4.4 points: excluding common-word extraction, the models track each other within this mean gap across all lengths and within ∼6 points at 1M.Common-word extraction requires aggregating and reproducing the most frequent words across the entire input; Soofi S degrades sharply past 32K on this subtask.
  • Long-context: The regression is attributed to the long-context data mixture and continuation recipe rather than the backbone architecture.Nemotron’s recipe emphasizes long-context document-QA data, synthetic retrieval-focused data up to 256K tokens, and down-weighted high-quality pretraining data.

F Checkpoint Merging Ablations

The checkpoint-merge ablation evaluated 22 post-hoc merges against iter_1056000 using four aggregate English and German suite metrics. No merge was uniformly better, although final-window averaging slightly improved German metrics.

  • Evaluation: 22 merged checkpoints were compared directly with iter_1056000 using English and German bits-per-byte plus normalized rank-choice accuracy.Bits-per-byte is lower-better, while normalized accuracy is higher-better.
  • Evaluation: One two-checkpoint final merge was excluded from the rank table because its filtered evaluation file had incomplete bits-per-byte coverage.
  • Outcome: The merge ablation found no uniformly better model than iter_1056000, with the best variants concentrated in the final annealing window.The uniform average over all 13 final-window checkpoints had the best aggregate rank, while exponential averaging with α = 0.8 was nearly tied.
  • Outcome: 0.3637 German bits-per-byte was achieved by the best merged variants, improving from 0.3656 for iter_1056000.
Loading 2607.09424v3…