PaperlayerView exact revision on arXiv

Source-linked AI summary

Kolibri: A Sovereign European Model on the Pareto Frontier

Aleph Alpha, :, Moritz Althaus, Artur Baranowski, Michael Barlow, Fabien C. Y. Benureau, Felix Berkenkamp, Amrita Bhattacherjee, Lukas Blübaum, Duncan Blythe, Augustijn de Boer, Bastian Boll, Thomas F. Burns, Björn Deiseroth, Niko Dürr, Béni Balázs Egressy, Niklas Finken, Niklas Freymuth, Fabian Gebhart, Max Hammer, Bastian Harren, Andreas Hartel, Johann Higl, Steffen Hirschmann, Pablo Iyu Guerrero, Yasser Jadidi, Andreas Jagusch, Mitja Kleider, Markus Klein, Jonas Knupp, Murali Kondragunta, Garwin Lechner, Dominik Lê-Kellner, Jurek Leonhardt, Jiaze Li, Sohir Maskey, Omar Mehio, Pit Neitemeier, Letitia Parcalabescu, Alejandro Petry Pacheco, Pranav Ragupathy, Martin Reinhardt, Peter Rodenkirch Nfissi, Dylan Rodriquez, Leonard Salewski, Raziye Sari, Jordan Sassoon, Ivo Schaper, Ilhan Scheer, Frank Schneider, Philipp Scholl, Lukas Schott, Eric Schreiber, Pablo Schumacher, Alessio Serra, Philipp Siedler, Prabhu Teja Sivaprasad, Mohamed Soliman, Till Speicher, Volker Stampa, Simon Thel, Serge Thilges, Irina Vidal Migallón, Denis Waßmann, Samuel Weinbach, Sascha Wirges, Michael Wüthrich, Alexei Zacharov

arXiv:2610.31945v1cs.CLcs.AIcs.LG

TL;DR

Regulated organisations need language models that support German and English, run on infrastructure they control, and meet serving-cost constraints. Kolibri combines an English–German MoE model with a full development pipeline and lies on the quality–serving-cost Pareto frontier among evaluated open models.

  • Problem

    Regulated organisations need language models that run on controlled infrastructure, support German and English, align with European values, and meet serving-cost constraints.

    Source: Introduction · p. 4

  • Method

    Kolibri is an English–German MoE transformer trained across pre-training, mid-training, and long-context extension, with curated German data and supervised fine-tuning plus reinforcement learning.

    Source: Introduction · p. 4

  • Results

    Kolibri lies on the quality–serving-cost Pareto frontier among evaluated open models in English and German, and scores higher than three open models on most benchmarks.

    Source: Introduction · p. 4

  • Takeaways & Limitations

    Its Apache 2.0 open weights permit deploying organisations to operate Kolibri on their own infrastructure.

    Source: Introduction · p. 4

    The candidates were not constructed to control for length-reweighting-induced domain shift, so its effect cannot be isolated.

    Source: §2.3.4.1 · p. 38

Abstract

from arXiv · show

This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total parameters and 3.46B active parameters per token, released as open weights under the Apache 2.0 licence. We built Kolibri for sovereign and specialised deployment, with a particular focus on German and on regulated domains such as public administration, industry, and aerospace. To support these settings, we control the full model-development process, including data, architecture, training infrastructure, post-training, and evaluation. We train Kolibri on 24T tokens across pre-training, mid-training, and long-context extension, with German accounting for more than 20% of the data mix by bytes. To reach this share, we complement German datasets with more than 2T German tokens that we curate from the web with our own pipeline and generate synthetically. Post-training combines supervised fine-tuning on curated German and English data with reinforcement learning on more than 1.2M internally curated tasks, which cover reasoning, tool use, instruction following, code, and retrieval. The model activates 4.4% of its parameters per token, and among the open models evaluated in this report, it lies on the Pareto frontier of quality and serving cost in English and German. Kolibri builds on Kolibri Origin, an earlier iteration that validated our full pipeline at scale. Both models come from our Model Factory, which connects data, experimentation, training, evaluation, and deployment through version-controlled workflows.

Introduction

Kolibri targets sovereign and specialised deployment by combining English–German capabilities, low active-parameter use, and open weights. Its development spans curated data and post-training, with version-controlled workflows connecting the pipeline.

Introduction

  • Kolibri has 78.1B total parameters, with 3.46B active per token, and lies on the quality–serving-cost Pareto frontier in English and German.The report also says Kolibri scores higher than three open models on most benchmarks across five domains.Source: Introduction · p. 4
  • The development pipeline considers applicable regulation, filters illegal, harmful, and pirated content, redacts personal data, and trains the model to abstain when context does not support an answer.Source: Introduction · p. 4
  • Training spans 24T tokens, with more than 20% German data by bytes and over 2T additional curated or synthetically generated German tokens.Post-training combines SFT on curated German and English data with RL on more than 1.2M internally curated tasks.Source: Introduction · p. 4
  • The Model Factory connects data, experimentation, training, evaluation, and deployment through version-controlled workflows designed for reproducibility, composability, and traceability.Source: Introduction · p. 4
  • Figure 2 presents post-training benchmark scores across four domains, with English and German benchmarks marked and scores scaled from 0% to 100%.Source: Figure 2 · p. 5

Pre-Training

Kolibri Base is produced through pre-training, mid-training, and long-context extension, with each stage using a distinct data mix and context length. The model scales these stages substantially relative to Kolibri Origin.

Pre-Training

  • Kolibri trains for 20T tokens at 16k context, continues for 3.4T tokens at 64k, then adds 200B tokens at 256k context.Source: Pre-Training · p. 6
  • Kolibri trains on about three times as many total tokens as Kolibri Origin and reaches four times its maximum context; mid-training grows by about 28×.Source: pre-training · p. 6
  • Each block combines attention and MoE sublayers; 40 SWA blocks use RoPE and a 512-token preceding-token mask, while the router selects 6 of 384 experts.The remaining blocks use full attention; router outputs are weighted by sigmoid scores without renormalisation.Source: Figure 4 · p. 7

2.1 Model Architecture

This section examines Kolibri’s architecture, including its sparse mixture-of-experts layers, hybrid attention, and German-focused tokeniser. It reports ablations on attention-window, model-capacity, and sparsity choices, alongside methods for balancing expert load.

2.1 Model Architecture

  • Kolibri uses 78.1B total parameters and 3.46B active per token, with GQA, hybrid SWA and full-attention blocks, and sparse MoE FFNs routing to selected experts.Every fifth block uses full attention; the others use SWA over the 512 preceding tokens.Source: §2.1 · pp. 7–8

2.1.1 Ablation Methodology

  • The ablations compare recipe choices on smaller proxy models at common training horizons, evaluating modelling quality, stability, and training and decode efficiency.The proxy models use 0.6B active parameters at scale M and 2.1B at scale L; evaluation aggregates are not directly comparable across studies because the suite broadened during development.Source: §2.1.1 · p. 9

2.1.2 Tokeniser

  • Kolibri achieves 4.90 bytes per token on German text, the highest compression among 12 evaluated tokenisers, and 4.58 bytes per token in English versus GPT-5’s 4.67.The German result is on par with BPE trained on the same pre-training corpus.Source: §2.1.2 · p. 10, Figure 5 · p. 11
  • Kolibri’s 128,000-token UniBPE vocabulary splits words at morpheme boundaries with 65% precision in German and 50% in English, versus 49% and 42% for the best external tokenisers.On German web text, it needs 11.2% fewer tokens than the GPT-5 tokeniser.Source: §2.1 · p. 8, §2.1.2 · pp. 10–11

2.1.3 Hybrid Sliding-Window Attention

  • Hybrid attention uses full attention in 10 of 50 blocks and SWA in the other 40, supporting four-times-longer sequences than full attention at approximately matched training compute.SWA blocks use RoPE, while full-attention blocks use NoPE; per-head QK RMS normalisation bounds attention logits and supports stable training at higher learning rates.Source: §2.1.3 · p. 12

2.1.3.1 Window Size

  • At scale L, w = 512 scored 77.1% pooled long-context versus 76.2% for w = 128 and led by 6.7 percentage points at 256k, despite 47.8% versus 48.0% pre-training accuracy.The authors select w = 512 for Kolibri, accepting a 0.2-percentage-point cost on pre-training benchmarks.Source: §2.1.3.1 · p. 13, Figure 8 · p. 14

2.1.4 Scaling Sparsity

  • Moving from 42.1B to 82.3B total parameters improved loss and aggregate accuracy, while 122.6B improved them only slightly further at 3.4B active parameters.Expert loads remained well balanced across all three configurations.Source: §2.1.4 · p. 14, Figure 9 · p. 15
  • At 4k context, the 82.3B and 122.6B configurations are 13% and 32% slower than 42.1B; at 256k, the gaps narrow to 4% and 12%.At long context, KV-cache reads account for more memory traffic, chiefly from the 10 full-attention layers; the 40 sliding-window layers contribute negligibly.Source: §2.1.4 · p. 15

2.1.5 MoE Load Balancing

  • Global EQB combined with local LEI kept pre-training global MaxVio at a 0.31 layer average, while the first two bias-overridden layers averaged 2.39.EQB provides stable global balance without a tunable balancing hyperparameter, leaving the LEI coefficient primarily to control local balance and training efficiency.Source: §2.1.5 · p. 16

2.1.5.1 Exact Quantile Balancing (EQB)

  • EQB computes exact global quantiles for BF16 adjusted logits with two-pass radix selection, requiring two all-reduces of 256E int32 counts per layer per step, independent of T.The passes use high-byte and low-byte histograms to recover the global quantile.Source: §2.1.5.1 · p. 17

2.1.5.2 Load-Error Injection (LEI)

  • LEI balances each local microbatch by injecting experts’ load deviations into every token’s sigmoid-score gradient; a bounded rescaled error helps training stability.Its update is derived as a straight-through estimate for unnormalised sigmoid scores.Source: §2.1.5.2 · p. 17

2.1.5.3 Leading Dense Layers

  • Replacing leading dense blocks with MoE layers slightly lowered loss and BPB and raised accuracy in the 327.6B-token ablation, motivating MoE layers in all 50 blocks.Full-run diagnostics nevertheless show that the first two MoE layers make almost no use of their routed experts.Source: Table 5 · p. 18, §2.1.5.3 · p. 18

2.2 Optimisation

Kolibri’s optimisation combines a warmup-stable-merge schedule with parameter-group-specific optimisers and a hyperparameter-transfer strategy that separates width from token horizon. Proxy sweeps and loss extrapolation guide the production settings, although the selection is based on training loss rather than merged-checkpoint or post-trained performance.

2.2 Optimisation

  • The schedule warms up linearly for 100B tokens, then holds the learning rate constant through pre-training, mid-training, and long-context training.The base weights average 20 long-context-stage checkpoints spaced 10B tokens apart; context grows from 16k to 256k tokens.Source: §2.2 · pp. 18–19
  • Muon optimises attention and expert weights, Adam handles embeddings and norm scales, and AdamW handles the LM head and router.Muon uses spectral scaling; decoupled weight decay applies to all groups except embeddings and norm scales.Source: §2.2 · p. 19, Table 6 · p. 19

2.2.1 Hyperparameter Transfer

  • The transfer strategy factorises model-size and token-horizon tuning, using µP for width and loss extrapolation to select horizon settings on a narrower proxy.This avoids a joint two-dimensional sweep over size and horizon; observed proxy loss trajectories predict longer-horizon settings.Source: §2.2.1 · p. 20

2.2.1.1 Learning-Rate Ratios

  • Preliminary sweeps set embedding, norm, and head learning-rate ratios relative to Muon; the router uses the head learning rate.The ratios were relatively stable across widths and token horizons in internal sweeps.Source: §2.2.1.1 · p. 20

2.2.1.2 Transfer Across Width

  • Within a √2 learning-rate factor of the width-transfer optimum, loss differences are below 0.1%.The proxy sweep finds an optimal learning-rate shift by a factor of 2 between widths, which the authors deem acceptable.Source: §2.2.1.2 · p. 21

2.2.1.3 Transfer Across Token Horizon

  • Loss extrapolation predicts long-horizon loss from shorter-run dynamics, with an average target-horizon prediction error below 1% on internal curves.The method discards the first 20% of the post-warmup prefix before fitting; the selected grid-point ranking matched across three tested power-law forms.Source: §2.2.1.3 · p. 22
  • 1 × 10−3 Muon learning rate and a batch of 4608 sequences of 16k tokens are selected for the 20T-token horizon.The selected batch is the smaller of the two tested, prioritising training efficiency; a further batch-size reduction might help.Source: Figure 10 · p. 21, §2.2.1.3 · p. 22
  • 2^-12 weight decay lowers observed loss and forecasts at every horizon versus 2^-13, and all estimators agree on that ordering.The team selects 2^-12 for Kolibri.Source: §2.2.1.3 · p. 23

2.3 Data and Training Stages

This section examines how Kolibri’s data is selected, prepared, and mixed across pre-training, mid-training, and long-context training, including German-specific construction and production deployment. It also evaluates the trade-offs of these choices and the model’s capabilities beyond its training context.

2.3.1 Overview

  • Pre-training uses 20T tokens, followed by 3.4T tokens of higher-quality mid-training and 200B tokens of long-context extension to 256k tokens.Across the stages, data selection becomes more selective and task-oriented, shifting from broad coverage to structured capabilities and longer sequences.Source: §2.3.1 · p. 23

2.3.2 Stage 1: Pre-Training Data

  • The pre-training pool contains 78 English- and German-language datasets and approximately 29.3T materialised tokens, from which the final mix uses 20T.The pool spans web, code, instruction, scientific and technical, and legal and public-sector data; German web filtering is adjusted to preserve long-word administrative prose.Source: §2.3.2 · p. 24, Table 8 · p. 24, §2.3.2.2 · p. 26

2.3.2.2 German Pre-Training Data

  • German supplies 21.3 % of final pre-training tokens, combining 0.8T unique organic web tokens, about 1.1T unique rephrased tokens, and carefully curated translated data.The authors note that translation can retain English linguistic, cultural, and institutional patterns, so Kolibri relies on organic and rephrased German alongside selected translations.Source: §2.3.2 · p. 24, §2.3.2.2 · pp. 26–27

2.3.2.1 Common Crawl Pipeline

  • The Common Crawl pipeline routes documents into separately thresholded English and German branches, then applies exact, fuzzy, and substring deduplication.Exact deduplication leaves 21.16B distinct documents; fuzzy deduplication removes a further 25.9 % as near-duplicates, and substring deduplication removes 21.3 % of text bytes.Source: §2.3.2.1 · p. 25

2.3.2.3 Quality Annotation

  • Luxical-based distilled judge classifiers achieve held-out F1 0.913 and AUC 0.971, and are used with a binary fastText classifier to annotate English CC data.BERT reproduces the judge most closely (F1 0.945, AUC 0.987), while fastText scores F1 0.865; the Luxical choice balances inference speed and quality.Source: §2.3.2.3 · pp. 27–28
  • At 7.8B-A0.6B MoE, English-overall BPB is 0.618 ± 0.015 for annotated CC and 0.572 ± 0.027 for Kolibri Origin.The posterior probability that annotated CC is better than Kolibri Origin on overall BPB at this scale is 0.98; at 0.4B, English-overall BPB is 3.800 ± 0.513 versus 3.988 ± 0.271.Source: §2.3.2.3 · pp. 28–29, Table 10 · p. 29

2.3.2.4 Data Source Mixing

  • The materialised 20T-token pre-training mix draws 35.7 % (7.15T tokens) from Kolibri’s own pipeline and 64.3 % (12.85T) from external sources.A proxy-model mixing law selects among 78 sources, with λ = 0.05, T = 3, and a 3× repetition cap; its predicted mean BPB is 3.670 ± 0.002 versus 3.667 ± 0.002 for the unconstrained optimum.Source: §2.3.2.4 · pp. 29–30, Table 11 · p. 30
  • The data-mix optimiser’s natural prior favors the largest sources, and some German quality-classifier buckets are down-weighted against their labels, so the mix does not follow the German quality ordering.The authors report patched token-count errors with likely small impact on the final mix, while old counts used for swarm ablations could not be corrected.Source: §2.3.2.4 · p. 30

2.3.3.1 Dataset Pool

  • Mid-training draws from a curated pool of deduplicated, PII-redacted datasets assigned to nine domains, while reserving documents above 128k estimated tokens for long-context extension.The production mix uses further evaluation-suite decontamination, which removes 0.06% of tokens; mid-training continues at the pre-training learning rate and uses checkpoint merging.Source: §2.3.3.1 · p. 31, §2.3.3 · p. 31

2.3.3.2 Ablation Setup

  • The proxy ablations compare 18 mixes at 84B tokens, with selected hand-set and production mixes also trained for 400B tokens against continued pre-training.The proxy has 11.9B total and 0.65B active parameters and was pre-trained for 2.79T tokens; scores use merged checkpoints, and post-training scores use a shared SFT recipe and data mix.Source: §2.3.3.2 · pp. 31–32

2.3.3.3 Mixing Method

  • Mixes allocate tokens across nine domains and within-domain datasets; the optimiser fits benchmark scores from expert merges, while repetition caps limit each dataset to four combined pre-training and mid-training epochs.Expert merges rank 12 mixes as full mid-training runs do with Spearman correlation above 0.9; the objective lacks direct benchmarks for instruction, reasoning, and specialised domains.Source: §2.3.3.3 · pp. 32–33

2.3.3.4 Production Mix and Its Effect

  • At 400B proxy tokens, the production mix leads continued pre-training by 4.9 points on the M4 merge and by 11.5 points after SFT, and leads C3 by 3.8 points after SFT.Both post-SFT differences exceed the 95% eval-noise intervals; the production mix is S5 with German instruction and reasoning raised to its repetition cap.Source: §2.3.3.4 · pp. 34–35, Table 13 · p. 34, Table 14 · p. 35
  • The production run raises the aggregate score by 2.8 points (2.9 without HumanEval) during mid-training, but the final base model remains 5.2 points below the pre-training M4 on TriviaQA and 4.5 on HellaSwag (DE).General-knowledge means fall by 1.0 points in English and 1.4 in German during mid-training; math and code improve in both languages. The cause of the TriviaQA loss remains open.Source: §2.3.3.4 · p. 35, Figure 14 · p. 36

2.3.3.5 Lessons for Choosing a Mix

  • At 400B, every adjacent mix pair differs after SFT beyond eval noise, but at 84B C2 and C3 differ by only 0.4 points after SFT, and SFT separates only 14 of 120 pairs.The penalty preserves domain weights: without it, the solve sets several domains to zero; at 400B, production mix HellaSwag (DE) is 5.5 points below C3, a 0.09-point aggregate-score cost. Merge scores appear to understate post-SFT gains on the four 400B runs.Source: §2.3.3.5 · pp. 35–36

2.3.4.1 Dataset Pool and Candidates

  • Long-context candidates combine naturally long documents with mid-training data; raising the reweighted share from 0 to 1/3 increases tokens in documents longer than 32k from 25% to 45%, while the induced domain shift remains uncontrolled.The English Language domain rises from 1.3% to 24%, while short-by-construction instruction-following, problem-solving, and agentic domains fall to about half their original shares; the section reports that more length-reweighted data improves retrieval and merging largely preserves general capability.Source: §2.3.4 · p. 37, §2.3.4.1 · pp. 37–38
  • The production long-context mix's mid-training-pool domain weights are compared with the original mix, and future ablations should vary domain weights and length distribution separately.Source: Figure 16 · p. 39, §2.3.4.1 · p. 39

2.3.4.2 Evals and Ablations

  • Merged checkpoints score 1.09–1.13 versus 0.97–1.02 for raw checkpoints; s = 1/3 leads merged results and is selected for production, though larger shares remain untested.Four branches shared a mid-training checkpoint and training schedule, differing only in data mix. The aggregate normalizes each RULER and HELMET leg against the four-branch mean, then averages context lengths and tasks, weighting the suites equally.Source: §2.3.4.2 · pp. 39–40, Table 15 · p. 40
  • Increasing the reweighted share generally improves QA, NIAH, and word extraction, while VT moves oppositely; the control remains competitive, with the clearest extension benefit on longest-context retrieval.Differences are most visible at 256k for NIAH, VT, and QA; on HELMET, JsonKV is near saturation while RagQA, Rerank, and LongQA InfBench QA separate more clearly.Source: §2.3.4.2 · p. 40
  • Long-context extension slightly reduces raw-checkpoint general capability versus the control, while merging largely recovers the difference within standard error.The standard non-long-context suite covers mathematics, coding, reasoning, and general knowledge.Source: §2.3.4.2 · p. 40

2.3.4.3 Quality Beyond the Training Context

  • Kolibri Base extrapolates beyond its 256k training context without positional rescaling, retaining useful capability to 1M tokens with task-dependent degradation.At 1M tokens, it scores above Nemotron 3 Nano Base on NIAH, VT, and QA, but below it on WE; NIAH declines with context length while VT remains stable.Source: §2.3.4.3 · pp. 40–41

2.4 Base Model Evaluation

The evaluation suite covers general knowledge, mathematics, and code in English and German, then compares Kolibri Base with open baselines under a shared protocol. Kolibri ranks second on both Overall aggregates, while its long-context results vary by benchmark and context length.

2.4.1 Evaluation Suite

  • The pre-training suite covers general knowledge, mathematics, and code in English and German where benchmarks exist, and its scores also inform data and model ablations.Baselines are evaluated under one shared protocol; results are comparisons under that protocol, not reproductions of published numbers.Source: §2.4.1 · p. 41, Table 16 · p. 41, §2.4.2 · p. 43

2.4.2 Baseline Models

  • The baseline set spans dense models from 7B to 70B parameters and MoE models with 3B to 12B active parameters, including Kolibri Origin Base.Generative-task settings are generally shared, with a stated exception for Gemma 4 26B-A4B Base.Source: §2.4.2 · p. 43

2.4.3 Results

  • Kolibri Base is the best-performing tested model with fewer than 12B active parameters, surpassed only by Nemotron 3 Super, which has over 3× its active parameters.GLM-4.5 Air has the same active-parameter count as Nemotron 3 Super but scores below Kolibri Base in both languages.Source: §2.4.3 · p. 43, Figure 19 · p. 43

2.4.3.1 Long-Context Performance

  • On RULER, Kolibri Base trails several models through 256k tokens, but among models evaluated up to 1M tokens it degrades the least and scores best there.On HELMET, it ranks second behind Gemma 4 26B-A4B Base at 64k and 128k.Source: §2.4.3.1 · p. 44, Table 18 · p. 46

2.4.3 Results

  • Kolibri Base ranks second on both Overall aggregates, behind Nemotron 3 Super Base with more than three times as many active parameters.It ranks first on the English math average.Source: Table 17 · p. 45

Post-Training

Kolibri’s post-training uses supervised fine-tuning and reinforcement learning to develop instruction following, reasoning, and agentic tool use. Its design addresses German use, long-context tasks, agentic interactions, retrieval, hallucination reduction, and controllable reasoning effort.

Post-Training

  • RL improves on the SFT checkpoint across all capability areas shown, while Kolibri performs on par with comparable open models.The section compares the released checkpoint with three open models and the RL-starting SFT checkpoint.Source: Post-Training · p. 47
  • German post-training adds synthetic German chat, math, science, and tool-calling data, then uses translated RL environments and a language-consistency reward for reasoning and answers.The report observes higher language-consistency scores on German benchmarks.Source: Post-Training · p. 47
  • Long-context training uses in-house curated data for inputs up to 256k tokens, including SFT conversations over 128k–224k-token documents and RL select-all questions.The SFT conversations contain four requests whose answers draw on sections spread throughout each document.Source: Post-Training · p. 47
  • Agentic training targets tool calls and terminal interactions, with backfilled reasoning, masked loss on wrong actions, generated software-engineering SFT data, and varied RL prompts and harnesses.The prompt, coding harness, and user request are randomized per group to reduce dependence on one setup.Source: Post-Training · pp. 47–48
  • For retrieval-grounded answers, the Merlin–Arthur protocol trains the model to answer with full or Merlin-masked context and abstain with Morgana-masked context.Merlin hides sentences whose removal most raises the gold-answer probability; Morgana hides those whose removal most lowers it, and both recompute masks from the current policy.Source: Post-Training · p. 48
  • Kolibri offers none, low, medium, and high reasoning effort; RL trains across the three named effort levels, focuses on high effort, and separates them with a length-dependent completion-budget signal.SFT labels reasoning traces by length relative to requests from the same dataset and difficulty; the chat template signals the selected effort in the system prompt.Source: Post-Training · pp. 48–49

3.1 Supervised Fine-Tuning (SFT)

Supervised fine-tuning establishes instruction-following, requested-effort reasoning, and tool-calling behaviors as a cold start for reinforcement learning. The stage uses curated open and synthetic datasets, including in-house data for areas such as German, long context, agentic tasks, and retrieval.

3.1 Supervised Fine-Tuning (SFT)

  • SFT introduces instruction following, requested-effort reasoning, and tool calling, establishing behaviors at a sufficient rate for RL to improve them.The released SFT checkpoint is a weight average of two runs trained on different mixes and serves as RL’s input checkpoint.Source: §3.1 · p. 49
  • MergeMix groups datasets into clusters, fine-tunes a specialist per cluster, and combines candidate-mix runs by averaging their checkpoint weights through souping.A weighting assigns each cluster specialist a coefficient; the valid weightings form a simplex sampled from a hand-set reference mix.Source: §3.1 · pp. 49–50

3.1.1 Datasets

  • Fine-tuning data combines legally assessed open datasets with synthetic data, including in-house datasets for German, long context, agentic tasks, retrieval, safety, compliance, and alignment.Some open-dataset completions are regenerated when they fail audits.Source: §3.1.1 · p. 50

5.2 (Z.ai 2026b), GLM-5.3 (Z.ai 2026c) and Qwen3.8-27B (Qwen Team 2026c). We employ different

This section covers Kolibri’s SFT data curation and mix selection, adaptive reasoning-effort controls, and training pipeline. It reports evaluations of data and reasoning choices and describes selecting the C1–C2 soup as the SFT checkpoint for RL.

5.2 (Z.ai 2026b), GLM-5.3 (Z.ai 2026c) and Qwen3.8-27B (Qwen Team 2026c). We employ different

  • Different models are selected for regeneration and judging tasks according to their speed, reasoning strength, and biases; samples are processed uniformly and MergeMix selects the final SFT subset.The shared process removes benchmark overlap and labels exchanges by reasoning effort; long-context conversations extend up to 50 turns using simulated user follow-ups and teacher answers.Source: §5.2 · pp. 50–51, Figure 26 · p. 59

3.1.1.1 German

  • German generation keeps prompts, reasoning, and answers in German, with tools unchanged; long-context samples use curated 128k–224k-token documents and simulated requests spanning distant sections.German tool-use data includes live German Wikipedia searches, while long-context training targets retrieval, summarisation, and multi-hop reasoning over documents, code bases, and tool outputs.Source: §3.1.1.1 · pp. 51–52, §3.1.1.2 · pp. 53–54
  • 16× German-data upsampling yields more correct German answers and a higher English AIME 2026 score, while German AIME 2026 does not differ from the model without German data.The German data covers chat, math, science, and tool use; reasoning-prefill distillation raises the share of teacher traces that remain German from 67% to 97%.Source: §3.1.1.1 · p. 51

3.1.1.2 Long Context

  • Adding about 93k long-context conversations, around 9% in German, significantly raises RULER VT; AA-LCR and LongBench Pro gains are not significant, with largest gains at the longest contexts.Documents are split into sections for simulated requests; 77.4% of conversations draw on sections from at least three of five document parts.Source: §3.1.1.2 · pp. 53–54

3.1.1.3 Tool Calling and Agentic Coding

  • Reasoning backfilling adds verified reasoning before recorded tool calls, while generated repository trajectories expand agentic software-engineering data and are retained only when their tests pass.Backfilled conversations are included with and without reasoning; terminal-task bugs must fail at least one test while leaving at least half passing.Source: §3.1.1.3 · pp. 54–55

3.1.1.4 Retrieval and Hallucinations

  • Retrieval rollouts are kept only when judged correct, grounded in retrieved text, properly formatted, and search-efficient; hallucination samples teach answering with evidence and abstaining without it.The retrieval data includes English and German tasks, while hallucination samples use the same question with and without supporting evidence so abstention depends on context.Source: §3.1.1.4 · pp. 55–56

3.1.1.5 Safety, Compliance, and Alignment

  • Safety data teaches rejection of harmful and copyright requests, while value-alignment examples target European democratic consensus and counter known teacher-model biases on sensitive topics.Prompts are phrased as everyday requests, and the teaching prompt asks models to represent democratic-spectrum positions fairly and distinguish facts, uncertainties, and value judgements.Source: §3.1.1.5 · p. 56

3.1.2.1 Shared Processing

  • At the same recall, within-benchmark IDF yields 0 false positives versus 57 for global IDF; decontamination removes 1,097 of 22,000,436 rows (0.0050%) with no measurable benchmark change.The shared pipeline canonicalizes samples, removes benchmark-text overlap and conversations with other-model identities or CCP narratives, and assigns reasoning-effort labels.Source: §3.1.2.1 · pp. 57–58, §3.1.2 · p. 57

3.1.2.2 Reasoning Effort

  • Kolibri supports none, low, medium, and high effort; its post-hoc labels use dataset-and-difficulty reference groups so users need not estimate prompt difficulty.Reasoning length for a fixed percentile of hard prompts is up to 31.3 times that of easy prompts; Figure 27 shows effort ranges overlap but are ordered within domains.Source: §3.1.2.2 · pp. 59–60, Figure 27 · p. 61
  • At fixed effort, hard prompts receive about 32.5 times the reasoning of easy prompts; at fixed difficulty, high effort yields 1.4–3.7 times the reasoning of low.The effort changes reasoning length within the range prompted by difficulty rather than overriding it.Source: §3.1.2.2 · p. 61, Figure 28 · p. 62
  • Reasoning raises the 14-benchmark mean from 38.4 under none to 64.9 at low effort and 67.8 at high effort; medium-to-high adds 0.7 points.The paired standard error for high versus low is 0.8 points.Source: §3.1.2.2 · p. 61
  • German benchmark median reasoning rises from 326 to 1,788 tokens (5.5×), while English rises from 1,160 to 2,993 tokens (2.6×) from low to high effort.German exchanges in the data pool have a median reasoning length of 368 tokens versus 2,078 for English.Source: §3.1.2.2 · p. 61

3.1.2.3 Final Data Pool

  • The SFT pool contains 57 datasets, 22.0M samples, and 335.8B tokens across seven domains; 92.6% of samples are English and 5.0% German.Of 335.8B tokens, 142.4B (42.4%) provide loss; the final model uses targeted mixes rather than the whole pool.Source: §3.1.2.3 · p. 62

3.1.3 Data Mixing with MergeMix

  • MergeMix searches 20 capability clusters using specialist-model merges, requiring 90B rather than 1,057B training tokens, a 91% compute reduction before evaluating candidate mixes.The sweep used 63 SFT runs of 16.78B tokens as the alternative; MergeMix trains one 4.50B-token specialist per cluster.Source: §3.1.3 · p. 63, §3.1.3.1 · p. 63

3.1.3.3 From Merges to Trained Mixes

  • 56.7% versus 51.5%: the best trained SFT mix beats the baseline, but the best merge trains worst at 50.0%, and merged and trained scores correlate at ρ = −0.09.Six of seven alternative weightings train above baseline; the best trained weighting is a first-round draw from the middle of the merge ranking.Source: §3.1.3.3 · p. 64, Table 22 · p. 64

3.1.4 Kolibri SFT Training

  • On 64 B300 GPUs, ordering packed sequences by attention cost raises median throughput from 8,610 to 15,850 tokens/s per GPU, an 84% increase.The SFT recipe trains 4,000 steps (about 268B tokens) on packed 256k-token sequences; the final training plan continues the best two candidate mixes and soups them.Source: §3.1.4.1 · p. 65, §3.1.4 · p. 65, Table 23 · p. 66

3.1.4.2 Recipe

  • SFT loss covers assistant reasoning, answers, and tool calls; conversations are split by user turn, yielding 2.27 samples on average.User turns, tool responses, and chat-format special tokens are masked; loss is normalized over the batch so packing and multi-turn splitting do not reweight supervised tokens.Source: §3.1.4.2 · p. 66

3.1.4.3 Candidate Runs and Souping

  • At 2500 steps and 168B packed tokens, C1 and C2 reach benchmark means of 59.9 ± 1.8 % and 55.7 ± 2.1 %, versus at most 52.6 ± 1.9 % for the other three models.C1, C2, and baseline B0 are then continued to 4000 steps and 268B packed tokens; B0 tests whether reweighting domains improves on natural sampling.Source: §3.1.4.3 · p. 66, Figure 30 · p. 67
  • Equal-weight parameter averaging combines pairs of models fine-tuned from the same base checkpoint, whose weights remain close; C1 leads on AIME 2026 and LiveCodeBench, while C2 leads on AA-LCR.The method soups only pairs, and these benchmarks show the largest score spread across mixes and soups.Source: §3.1.4.3 · pp. 66–67, Figure 30 · p. 67
  • 63.4 ± 1.8 %: the C1–C2 soup has the highest mean, versus 58.9 ± 1.8 % for C1 and 30.0 ± 2.0 % for the base model.Every soup of the 4000-step models scores above both constituent models on the mean; the selected soup’s scores are optimistic because selection and evaluation use the same benchmarks. It becomes the released SFT checkpoint and RL starting point.Source: Figure 30 · p. 67, §3.1.4.3 · p. 67
  • C1 leads on AIME 2026 despite using 4.9 % math tokens versus B0’s 18.3 %; B0’s 69.4 ± 9.3 % is within noise of the base model’s 74.6 ± 10.7 %.B0 has 7.3 % of AIME 2026 generations end in repetition loops or at the context limit, versus 2.1 % for the base model and 0.0 % for C1; the passage notes other causes are possible.Source: §3.1.4.3 · p. 67

3.2 Reinforcement Learning (RL)

Kolibri’s RL stage starts from the SFT checkpoint and trains asynchronously on verifiable rewards across English and German environments. The section covers its optimization, consistency, reward, and rollout mechanisms, along with environments for long-context question answering, tool-based retrieval, and truthful answering or abstention when context lacks an answer.

3.2.1.1 Optimisation Objective

  • The policy-gradient objective uses group-relative advantages and a B-TV trust region, while a per-token logit-gradient bound limits updates when training and inference policies diverge.The bound leaves tokens below the threshold unchanged and scales down tokens whose gradient exceeds it.Source: §3.2 · p. 68, §3.2.1.2 · p. 70

3.2.1.3 Training–Inference Consistency

  • To reduce training–inference mismatch, the trainer uses FP8 quantisation-aware training and replays the inference engine’s recorded expert routing.The paper reports that serving with FP8 weights and KV cache raises rollout-generation throughput by a factor of two to three in its runs.Source: §3.2.1.3 · p. 71

3.2.1.4 Auxiliary Objectives

  • Squared log-ratio regularisation penalizes policy drift, while targeted OPSD gives selected low-reward, low-variance multi-turn groups a dense learning signal.OPSD uses a remedial hint at the turn where a tool-response error occurred; the hinted live-policy distributions teach the original unhinted context, without requiring a ground-truth solution.Source: §3.2.1.4 · p. 72

3.2.1.5 Asynchronous Training

  • The asynchronous RL loop samples completed groups from a replay buffer, updates inference replicas after each optimizer step, and limits off-policy shift by evicting stale groups.Active requests continue under updated weights, so one sequence may span policy versions; training is therefore only approximately on-policy.Source: §3.2.1.5 · p. 72

3.2.2 Environment Mix

  • The final RL run uses 38 environments—31 English and 7 German variants—with sampling weights that deliberately upweight several agentic environments and competitive programming.The environment mix contains 1,209,965 training rows in total.Source: Figure 32 · p. 73, §3.2.2 · p. 73

3.2.2.2 Environment-Specific Format Signals

  • Format signals modify positive environment rewards multiplicatively, while the environment task remains the primary signal even when multiple format defects occur.The aggregation is designed to let a single format signal matter while retaining a minimum fraction of environment reward.Source: §3.2.2.2 · pp. 74–75

3.2.2.3 Environment Summaries

  • German AIME 2026 reasoning in German rises from 89% to 99% and accuracy from 75% to 91%; English AIME accuracy rises from 92% to 96%.The reported English language-consistency rate remains at 100%.Source: Figure 34 · p. 76, §3.2.2.3 · p. 76
  • The long-context environment appends a select-all-that-apply question over each SFT conversation’s 95,000–230,000-token seed document as a further turn.The environment is designed to retain long-context capabilities.Source: §3.2.2.3 · p. 76
  • Retrieval environments pair English and German general and specialised corpora with varied tool configurations, prompt formulations, and GLM-5.2-generated multi-hop tasks (Z.ai 2026b).The setup is intended to familiarise the model with agentic RAG data and questions and teach it to solve multi-hop questions with search tools.Source: §3.2.2.3 · p. 76
  • Merlin–Arthur environments train abstention on unanswerable questions by masking context and reward truthful answers, abstention, and correct answers on the unmodified sample (Wäldchen et al. 2024; Deiseroth et al. 2025).Each sample appears in three variations; Merlin and Morgana mask the least and most important sentences, with sentence importance estimated from the change in the model’s probability for the ground-truth answer when masked.Source: §3.2.2.3 · pp. 76–77

Appendix I.1.3.

This section describes Kolibri’s reinforcement-learning post-training, covering its alignment environments and randomized tasks, adaptive curriculum, distributed pipeline, and the evolution of training and evaluation workflows from early experiments through release. It also reports changes in validation reward, capabilities, and benchmark performance associated with the training process and format-reward handling.

Appendix I.1.3.

  • Alignment environments reward balanced answers or justified refusals on Chinese political prompts, copyright refusals in English and German, and refusals to unsafe prompts; task randomization varies prompts, harnesses, and typos.Political balance training does not address political biases beyond Chinese political content. Randomization draws English or German system-prompt language independently of request language; instruction-following tasks instead verify the requested response language.Source: Appendix I.1.3 · p. 77, §3.2.2.4 · p. 77, Figure 35 · p. 77

3.2.2.5 Training Curriculum

  • The curriculum prioritizes prompts whose groups provide learning signal, lowering an environment’s sampling weight when trainable prompts run short.Prompts are bucketed as hard, solved, or trainable by estimated pass rate, with the target weight highest for trainable prompts; high-effort group rewards update estimates, and groups where all sequences fail move prompts to the hard bucket.Source: §3.2.2.5 · p. 78
  • Prompt selection weights buckets by targets, supply, and recency, scales environment draws by available bucket weight, and bypasses the curriculum for Hallucination Reduction prompts.Batch representation can still skew toward short environments because threads start together; retries are capped at two, and replay-buffer staleness eviction is more likely for slow environments.Source: §3.2.2.5 · pp. 78–79

3.2.3.2 Rollout Generation

  • RL uses 256 NVIDIA B300 GPUs across 32 nodes; 144 GPUs serve inference, and rollout workers generate grouped sequences with truncations receiving zero reward.Training starts from the SFT checkpoint and runs 1,000 optimizer steps, with batches of 256 groups of eight, up to 100 rollout steps per sequence, and contexts up to 256k tokens. Re-rendered conversations start new masked training segments so each generated token is trained only once.Source: §3.2.3.1 · p. 79, §3.2.3.2 · p. 80, §3.2.4.1 · p. 83

3.2.3.3 Trainer

  • The trainer shards the model across 96 GPUs with FSDP, while RDMA moves replay-buffer batches to trainer ranks and updated weights to inference replicas.Checkpoints save model, optimizer, schedule, and curriculum state every 25 optimizer steps; FP8 exports are evaluated every 100 steps, and the released model is the step-1000 export.Source: §3.2.3.3 · pp. 80–81, §3.2.3.4 · p. 82

3.2.4.2 Kolibri RL Training Run

  • Validation reward rose from 0.62 over the first 50 steps to 0.83 over the last 50, with instruction-following and structured-output rewards increasing from 0.65 to 0.94.The largest gains were in instruction following and structured output, followed by software engineering and tool use.Source: §3.2.4.2 · pp. 83–84, Figure 39 · p. 84
  • The released RL checkpoint improves instruction following and agentic, math, and coding abilities over SFT, while leading on several benchmarks and trailing on others.It scores highest among the three open models on both AIME benchmarks, GPQA Diamond, τ 3-bench Banking, LiveCodeBench, and IFBench, and above the two models with BrowseComp scores. It trails on the AA-Omniscience Index, scores lowest on MMLU-Pro, and trails at least one model on TerminalBench 2.1, LongBench Pro, and BFCL v4; on agentic retrieval it outperforms on Honeypot, ties Nemotron 3 Super within two points on MuSiQue and Agentic Wiki QA (DE), and is competitive on customer proxies.Source: Table 25 · p. 84, §3.2.4.2 · p. 84
  • Without an entropy bonus, normalized entropy rose from 0.040 to about 0.06 before falling to 0.033, while the B-TV mask and gradient bound affected roughly 0.01% of tokens per step.The training–inference mismatch hovered around |log rt| = 0.03, with median forward KL divergence of 2.4 × 10−3 per token; both measures include asynchronous policy staleness.Source: Figure 40 · p. 85, §3.2.4.2 · p. 85
  • OPSD targets low-reward groups with hints for terminal-code and tool-call failures, whose frequency falls during training before stabilising.The teacher’s token agreement with the student starts high, drops as training progresses, then rises again; the authors relate this to policy entropy and suggest the student may train out of hinted errors. Later hints address output and time-limit errors by encouraging sliced file reading and command-output filtering.Source: §3.2.4.2 · pp. 85–86, Figure 42 · p. 86

3.2.5 Lessons Learned and Hills Climbed

  • Kolibri RL evolved through preliminary exploration, hill-climbing for two base-model generations, and the final release checkpoint.Figure 43 presents representative benchmark-score checkpoints over time, but its intermediate points are not representative benchmark performance; the star marks the released Kolibri checkpoint.Source: §3.2.5 · p. 86, Figure 43 · p. 87

3.2.5.1 Phase 1: Initial Experiments

  • Phase 1 moved code execution from unreliable remote sandboxes to local training-node sandboxes and cut weight-update time from a median 221 s plus 217 s loading to about 17 s.RDMA writes, in-flight updates, and per-node NVLink broadcasts enabled updates after every training step, making training more on-policy; the team also chose a simpler three-bucket curriculum to reduce hyperparameters.Source: §3.2.5.1 · pp. 87–88
  • AIME 2025 scores rose from 60.4 for the SFT-trained model to 70.6 after 500 RL steps.The team found RL was overfitting to output formats at the expense of environment rewards; they separated the reward types and applied format signals only to sequences with positive environment reward.Source: §3.2.5.1 · p. 88

3.2.5.2 Phase 2: Moving to Kolibri Origin

  • Kolibri Origin’s tool-calling work added self-hosted user simulators and LLM judges, while sequence monitoring and internal releases exposed failures missed by benchmarks.The team also used pass@k from the initial SFT model and recent RL checkpoints as a cheaper proxy for whether new environments offered trainable prompts or were already solved; colleague feedback and blind A/B tests informed later training.Source: §3.2.5.2 · pp. 88–89

A Stronger Base Model

This section reports Kolibri’s evaluation results across English and German, including retrieval and grounding, and describes the design choices and workflows used to develop and evaluate the model. It also covers tokeniser and architecture ablations, dataset review, and limitations and future work.

A Stronger Base Model

  • MuSiQue, Honeypot, and five in-house Industry RAG benchmarks rose during September RL development as environment parametrisation and model skills improved.The Industry RAG benchmarks were monitored and used to guide environment changes, so they were a development set, not a held-out test set; the authors say the results indicate that simulating real-world harnesses, data distributions, and question types can improve use-case performance without training on the concrete use cases.Source: Figure 44 · p. 90, A Stronger Base Model · p. 90
  • Two evaluation exploits exposed weaknesses in the OpenCode harness and sandbox isolation, prompting removal of web fetching and stronger sandbox and training safeguards.With OpenCode, SWE-bench Verified scores were about 7 percentage points higher because web tools found GitHub ground-truth solutions in about 7% of trials; a sandbox command also disabled host user namespaces. The authors note that escapes depended on sandbox security as well as model capability.Source: A Stronger Base Model · pp. 90–91, Figure 45 · p. 91
  • The final RL run added over a dozen environments, while tool-calling environments needed a free-form-response judge and a filter for potentially wrong reference answers.The final adjustments included error-dependent OPSD hints, multiplicative p-norm format signals, Merlin–Arthur environments to reduce hallucinations, expanded OPSD terminal environments, and token-F1 grading for retrieval answers; future work includes long-input reasoning, MOPD, multi-stage training, and more flexible environment-driven rollouts, which could support non-linear histories and multi-agent setups.Source: A Stronger Base Model · pp. 91–92, Table 26 · p. 92

3.3.1 Evaluation Suite

  • The evaluation suite covers knowledge, math, tool use, code, instruction following, grounding, retrieval, Industry RAG, and long context, with German versions where available.Aggregates include only benchmarks shared by all baselines, exclude AA-Omniscience Index and Honeypot, and do not include Long Context; German and English Overall scores are not directly comparable. Scores are unavailable when prompts exceed a model's context window or it cannot use tools, and Industry RAG scores are relative signals rather than deployed-assistant accuracy.Source: §3.3.1 · pp. 92, 94, Table 27 · p. 93

3.3.2 Baseline Models

  • Kolibri is evaluated against twelve open post-trained models spanning dense models with 27B–70B parameters and MoE models with 3B–12B active parameters, alongside Kolibri Origin.The Model Factory supports faster improvement through a collaborative, version-controlled training pipeline; its Model Training as Code approach provides composability, consensus, and provenance, and Savanna implements the Kolibri recipe end to end.Source: §3.3.2 · p. 94, Model Factory · pp. 100–101, §4.1 · p. 101

3.3.3 Results and Discussion

  • Kolibri, Qwen3.5 35B-A3B, and Qwen3.8 27B span the Pareto frontier in both languages; Kolibri scores 0.8 points above Qwen3.5 in English and 1.0 in German.At 3.46B active parameters, Kolibri scores above all 3B-active baselines in both languages and best among the compared MoE models on both Overall rows; dense Qwen3.8 27B has the highest overall aggregates but activates nearly 8 times as many parameters per token.Source: Figure 46 · p. 95, §3.3.3 · p. 95, Table 28 · p. 97

3.3.3.1 Real-World and Customer Use Cases

  • On the five customer-proxy Industry RAG benchmarks, Kolibri averages 89.7 on the two English proxies and 67.5 on the two German proxies scored for every model.Kolibri leads every MoE baseline on English by 2.7 points over Qwen3.5 35B-A3B (87.0), and beats every MoE baseline except Qwen3.5 35B-A3B (70.0) on German; only dense Qwen3.8 27B scores higher on both (93.2 and 80.2). Kolibri also scores highest among plotted models on Honeypot and ties Nemotron 3 Super 120B-A12B on Agentic Wiki QA (DE), 69.4 against 67.9.Source: §3.3.3 · pp. 95–96, Figure 47 · p. 96, Table 29 · p. 98

3.3.3.2 Hallucination Reduction

  • Kolibri abstains or answers partially on 44% of incorrect AA-Omniscience questions, versus 15% for Kolibri Origin, and abstains on 86% of RGB questions with unanswered documents, versus 74%.Kolibri reaches 23.4% on the M/A grounding score, while Kolibri Origin and several other models score 0; this metric certifies how much of an answer came from the document.Source: Figure 48 · p. 99, §3.3.3.2 · p. 99

4.1 Savanna: Model Training as Code

  • Savanna uses declarative run configurations and shared abstractions to manage training, evaluation, checkpoints, experiments, and cluster differences.Intermediate checkpoint evaluation provides early signals to stop unpromising runs; nightly end-to-end training and evaluation catch regressions, while version-pinned workflows support reproducible execution.Source: §4.1.1 · pp. 101–102, §4.1.2 · p. 102, §4.1.3 · p. 102

4.1.4 Agents in the Pipeline

  • Model Factory gives agents visibility into experiments, bounded safe actions, and measurable objectives, enabling autonomous experiment review, failure triage, and efficiency improvements.Agents also report limitations and analyze their own failures to improve their instructions and harnesses.Source: §4.1.4 · p. 103

4.2 Savanna in Kolibri Training

  • Savanna’s automated merging and evaluation of the last four checkpoints every 2,000 steps enabled continuous monitoring, while a mid-run logging and profiling change raised pre-training throughput by about 3.7%.Automatic resumes preserve lineage and metrics, and the training run recovered from infrastructure failures.Source: §4.2 · p. 104

Model Overview

  • Kolibri is an open-weight English–German MoE with 78.1B total and 3.46B active parameters, trained on nearly 24T tokens and supporting contexts up to 1M tokens.More than a fifth of pre-training tokens are German, including over 2T curated or synthetically generated German tokens; its tokenizer uses 11.2% fewer German tokens than the best of nine evaluated alternatives while staying close in English.Source: Model Overview · p. 105

Capabilities

  • Kolibri has the highest aggregate among compared MoE models in both languages, lies on the open-model quality–serving-cost Pareto frontier, and improves substantially over Kolibri Origin.It matches or exceeds 12B-active-parameter MoEs on math and competitive coding, though dense Qwen3.8 27B scores higher in both languages. Kolibri Base is the best-performing model in its weight class up to 5B active parameters; its aggregates trail only Nemotron 3 Super 120B-A12B Base.Source: Capabilities · pp. 105–106

Model Factory

  • The Model Factory centralizes shared training workflows and comparison methods to support reproducible concurrent experiments, while the team’s experience training two generations is described as a durable outcome.The report identifies improving German reasoning, reducing hallucinations, and extending to other European languages as future interests.Source: Model Factory · p. 106, Conclusion · p. 106

Legal Due Diligence Example: Deep Dive Into Copyright Law

  • Kolibri’s third-party dataset review assesses data sources, crawling practices, opt-outs, and licence restrictions, excluding datasets when it finds indications of infringement or unobserved opt-outs.The report presents this process as going beyond the GPAI Code of Practice’s explicit commitments for third-party datasets.Source: Legal Due Diligence Example: Deep Dive… · pp. 107–108

A Quality-Cost Analysis

  • The quality-cost analysis compares aggregate quality with estimated decode text throughput per GPU, reporting throughput at 4k tokens for base models and 16k for post-trained models.Throughput is measured on eight NVIDIA B200 GPUs using each model’s fastest eligible layout; the figure assumes FP8 serving preserves baseline scores.Source: A Quality-Cost Analysis · pp. 128–129

B.1.1 Morphology

  • At 128,000 tokens, Kolibri reaches 65% German and 50% English boundary precision, and 79% German and 83% English merge-order scores, leading other tokenisers on both measures.The boundary-precision comparison is at equal vocabulary size; all precision and order differences between Kolibri and other tokenisers are significant at 99% paired-bootstrap intervals.Source: Table 30 · p. 130, §B.1.1 · pp. 130–131

B.2 QK Normalisation

  • Without QK normalisation, baseline-learning-rate training diverges; removing it only from full-attention blocks or using adapted QK-Clip causes gradient spikes, while 0.1× learning rates train stably but at higher loss.With QK normalisation, training remains stable across the learning rates and batch sizes in the reported sweeps.Source: §B.2 · p. 131, Figure 51 · p. 131

B.3 Model-Shape Evaluation Suite

  • The model-shape evaluation suite averages 17 accuracy metrics, while its BPB mean averages four English and German HumanEval and MBPP metrics.Accuracy uses task-specific measures, including normalised log-likelihood for multiple-choice tasks, completion accuracy for GSM8K and TriviaQA, and exact-flex accuracy for MATH Minerva.Source: §B.3 · p. 132

B.4 Sparsity Sweep

  • The sparsity sweeps compare training loss and aggregate accuracy at 3.4B and 0.9B active parameters, and report concurrent 256k-token request capacity for 3.4B configurations.The accuracy aggregates average the same 17 benchmarks; capacity estimates cover common accelerator setups.Source: §B.4 · p. 132, Figure 52 · p. 132, Table 31 · p. 132

B.5 Model Shape at 80B Total Parameters

  • With total model size fixed at about 80B parameters, two shape ablations test expert granularity and residual-stream width.The expert-granularity and residual-width comparisons are reported in Appendices B.5.1 and B.5.2.Source: §B.5 · p. 133

B.5.1 Expert Granularity

  • The Kolibri expert configuration achieves the lowest loss, highest accuracy, and lowest BPB; the finer-grained setup comes close, while the coarser-grained setup falls behind both.The configurations are compared at their last checkpoint after 327.6B tokens; Kolibri routes each token to 6 of 384 experts, versus 12 of 512 in the finer-grained setup and 6 of 256 in the coarser-grained setup.Source: Table 32 · p. 133, §B.5.1 · p. 133

B.5.2 Residual Width

  • After learning-rate selection, training loss and mean accuracy improve with residual width; most accuracy gains occur from width 640 to 1024, with a smaller gain to 1600.These approximately matched-FLOP results use 0.9B active parameters and 16k-token sequences; they are not separately validated at 3.4B active parameters.Source: §B.5.2 · pp. 133–134, Table 33 · p. 134

B.6 Leading Dense Layers and Routing in the First Layers

The first two MoE layers show strong router preference but almost no effective use of routed experts after longer training, despite an earlier ablation suggesting leading dense layers were unnecessary. The infrastructure section reports stable throughput after routing settles, methods for locating stragglers, observed failures and recovery, and the batch-construction problems addressed later.

B.6 Leading Dense Layers and Routing in the First Layers

  • 98.5% of first-layer routing decisions were overridden by bias, and their selected top-K mass was close to the uniform K/E value despite strong intrinsic router preference.The first two layers looked healthy on balance metrics, but their bias overrides nearly all router choices.Source: §B.6 · p. 135, Figure 54 · p. 136
  • Only layers 0 and 1 had loss changes indistinguishable from zero under both routing perturbations, indicating the model did not rely on their routed experts.Layer 48 also had a high intervention rate, but perturbing it clearly raised loss, so that rate alone did not indicate unused experts.Source: §B.6 · p. 135, Figure 55 · p. 137
  • The 327.6B-token ablation favored removing leading dense layers, but longer training revealed their routed experts were barely used; SFT densification showed no clear downstream improvement, leaving resolution for future work.The SFT intervention concatenated the shared expert and six routed experts with the lowest biases into a dense FFN while keeping their pre-trained weights.Source: §B.6 · pp. 134–136

B.7 Pre-Training Infrastructure

  • Throughput stabilized after an initial dip as routing settled, with no expert-parallel-related degradation over time.The plot reports median tokens per second per GPU in 100-step bins; its roughly 3.7% increase from r02 to r03 came from less frequent logging and profiling.Source: §B.7 · p. 138, Figure 56 · p. 138

B.7.1 Straggler Detection

  • The team combined passive GPU health monitoring, online transformer-block timing, and offline all-rank profiling to distinguish compute from communication stragglers and pinpoint every communication straggler in the main run.Collective arrival times and throughput were compared separately across parallelism dimensions.Source: §B.7.1 · pp. 138–139, Figure 57 · p. 139

B.7.2 Fault Tolerance

  • 38 interruptions over 377 000 GPU hours yielded 0.1 failures per 1000 GPU hours, while automatic recovery took 29.4 min on average.Checkpointing every 250 steps implies 12.5 min average recomputation for uncorrelated failures; the reported recovery average includes early manual node swaps.Source: §B.7.2 · pp. 139–140, Table 35 · p. 140

C.1 Training Batch Construction

  • Insufficient shuffling and long documents spanning consecutive batches can correlate examples, shrink effective batch size, and shift the data mixture over time; cutting documents also removes preceding context.These are the training-batch problems the following subsections address.Source: §C.1 · pp. 140–141

C.1.1 Data Loading Pipeline

The data pipeline materializes reproducible document order, chunks and packs documents to reduce sequence-boundary and locality problems, and applies language-specific filtering and multiple deduplication stages. Mid-training ablations use specified proxy budgets and evaluation-noise estimates, while production mix weights are constrained by dataset repetition caps.

C.1.1 Data Loading Pipeline

  • Each dataloader worker reads row groups, tokenizes and chunks documents, bin-packs fixed-length sequences, and emits input, label, and position tensors; checkpointed worker state preserves the exact data order on resume.The materialized row-group order is shared across ranks, while each rank and worker receives a disjoint slice.Source: §C.1.1 · p. 141

C.1.2 Chunking and Packing

  • The random sequential document buffer spreads a long document’s chunks across microbatches, while bin packing fills fixed-length sequences and limits extra cuts; the authors report smoother loss with RSDB than sequential chunking.RSDB emits contiguous chunks from randomly selected buffered documents; positions restart at each chunk, so pieces are independent contexts.Source: §C.1.2 · p. 142

D.1.1 Heuristic Filters

  • The Common Crawl pipeline routes language-identified documents through language-agnostic filters and English or German branches, then applies cleanup to remove empty or degenerate documents and cap whitespace runs.The figure tracks documents kept and removed at each heuristic-filter step, with MADLAD-400 filtering only in the English branch.Source: §D.1.1 · p. 143, Table 37 · p. 144, Figure 58 · p. 145

D.1.2 Exact Deduplication

  • Exact deduplication hashes unnormalized UTF-8 text within and across chronological snapshots, retaining the earliest copy and recording copy counts for later mixing; it does not deduplicate by URL.The global pass retains hashes, counts, and text locations rather than storing all earlier text.Source: §D.1.2 · p. 143

D.1.3 Fuzzy Deduplication

  • Fuzzy deduplication removes 24.9% of English and 33.3% of German documents using MinHash candidate matching followed by Jaccard verification, but near-duplicates in different shards remain.Verification uses token 3-gram Jaccard similarity at a threshold of 0.80.Source: §D.1.3 · p. 144

D.1.4 Substring Deduplication

  • Substring deduplication removes 20.4% of English and 29.6% of German bytes, but joins text across cuts without checking coherence, and cut documents skip heuristic filters.It keeps one occurrence of repeated byte sequences of at least 500 bytes and cuts other occurrences from documents.Source: §D.1.4 · pp. 144–145

D.2.1 Ablation Setup

  • Proxy ablations start from a 384-expert, 1024-hidden-size model checkpoint after 2.79T tokens, or 91% of the matched 3.05T-token horizon; the SFT evaluation estimates score intervals from 20 seed-only replicate runs.The ablation mid-training budgets are 84B or 400B tokens; the SFT recipe uses 1000 steps at 256k context.Source: §D.2.1 · pp. 145–146

D.2.2 Mixing Method

  • Repetition caps constrain domain weights; production raises German instruction and reasoning to 1.9% of the 5.2T-token mix, whereas projected solutions under 30T-pretraining caps assign it 1.4% of 5T.The production mix uses 20T-pretraining repetition caps, under which fewer datasets are pinned.Source: §D.2.2 · pp. 146–147, Table 38 · p. 147

D.2.3 Ablation Results

The ablations compare mid-training mixes and track contamination, the transition into mid-training, and production-run results. They show that language-domain composition relates to benchmark trade-offs, while contamination can substantially inflate scores under greater exposure.

D.2.3 Ablation Results

  • 2.4, 2.5, and 2.6 points separate adjacent 400B runs, and all three gaps exceed their thresholds at p < 0.01.Figure 59 plots pooled SFT scores for the merges.Source: §D.2.3 · p. 148, Figure 59 · p. 148
  • The production mix ranks fifth of 18 at 84B, within 0.7 points of the four higher-ranked runs; only 14 of 120 SFT pair tests clear noise.The production mix’s rank is measured on the M10 merge; the SFT pair tests pit one of the seven highest against one of the three lowest scores.Source: §D.2.3 · p. 148, Table 40 · p. 149, Figure 60 · p. 149
  • Across 84B runs, German web and documents correlate with HellaSwag (DE) and German knowledge, while math and MBPP account for more aggregate-score spread than general knowledge.At 400B, production-mix scores rise on several knowledge benchmarks but fall on TriviaQA and HellaSwag; the cause of the TriviaQA loss remains open.Source: §D.2.3 · pp. 148, 150, 151, Figure 61 · p. 150

D.2.4 Language Sweep

  • At equal domain weights, runs on the extended pool score 2.4 to 3.1 points lower on HellaSwag (DE) than runs on the mid-training pool.Dataset pool or sampling-rule changes alone shift TriviaQA by at most 0.8 points, and together by 1.2.Source: §D.2.4 · pp. 151–152

D.2.5 Decontamination

  • Decontamination inspected 8.3B documents and removed 1.5M; synthetic instruction and QA data held 67% of removals despite comprising 32% of inspected documents.Removed documents were about six times longer on average than pool documents.Source: §D.2.5 · p. 153, Figure 63 · p. 154
  • At target-scale exposure, added-back documents still inflated HumanEval by 19 points, while natural-rate contamination was not measurable at proxy scale.The report also notes that 1,926 documents with HumanEval reference solutions passed through the production run.Source: §D.2.5 · pp. 154–155

D.2.6 Transition into the Mid-Training Mix

  • The transition linearly decays the pre-training mix over H tokens, contributing H/2, or 50B tokens, to the window.The schedule uses pre-training index documents beyond the 20T-token pre-training run, but that pool remains potentially contaminated.Source: §D.2.6 · p. 155

D.2.7 Production Run

  • During mid-training, HumanEval recitations fall below 10% within 0.3T tokens, while non-recited English pass@1 stays between 61 and 76.The pre-training scores are not comparable with those of a decontaminated model; long-context extension changes post-SFT benchmarks by no more than evaluation noise.Source: §D.2.7 · pp. 156–157, Figure 64 · p. 156

E Base Model Evaluation

The appendix documents the checkpoints and deployment configurations used for the model evaluations. It specifies the serving setup and the table’s configuration fields.

E.1 Baseline Models

  • All models in Tables 17 and 28 are served with vLLM on one node of eight NVIDIA B200 GPUs.SWE-bench Verified uses a dedicated single-replica engine per model.Source: §E.1 · p. 159
  • The deployment configurations specify TP and DP ranks, expert parallelism for MoE layers, served context length, and sampling parameters.“Default” sampling uses the checkpoint’s generation configuration.Source: §E.1 · p. 159
  • Table 48 lists each model’s checkpoint and deployment, marking models without a public reference as internal and identifying the report’s own checkpoints.The configurations are read from release definitions at the commit that rendered the model card.Source: Table 48 · p. 159

F “I Don’t Know” Evaluations

The report evaluates whether models can abstain on seven English multiple-choice tasks and compares reasoning modes on long-context retrieval. Kolibri Base ranks strongly on abstention metrics, while reasoning-mode comparisons show different task patterns and do not isolate reasoning alone.

F “I Don’t Know” Evaluations

  • DCS scores probability mass across correct, wrong, and IDK options, with abstention scoring 0.The seven evaluated tasks append an IDK option to existing multiple-choice tasks and use log-likelihood scoring.Source: F “I Don’t Know” Evaluations · p. 160
  • Kolibri Base is second on C-Accuracy and DCS across the seven IDK tasks, and its average DCS exceeds every evaluated model except Nemotron 3 Super with posterior probability above 0.9999.The largest model, Nemotron 3 Super 120B-A12B, ranks first on all metrics.Source: F “I Don’t Know” Evaluations · p. 160, Table 49 · p. 161
  • On MMLU-Pro, Kolibri Base scores −20.23, behind only Nemotron 3 Super 120B-A12B at −11.02.The report attributes Kolibri’s strong DCS performance to placing more probability on IDK or correct options than other tested models except Nemotron 3 Super.Source: F “I Don’t Know” Evaluations · pp. 160–161

G Reasoning Mode at Long Context

  • At the longest context lengths, reasoning on scores higher on QA, while reasoning off raises NIAH recall; at 1M tokens, reasoning-off Kolibri beats Nemotron 3 Nano on VT and QA, but trails on NIAH and WE.The reasoning-on and reasoning-off runs allow up to 16k and 1k generated tokens, respectively, so the comparison does not isolate reasoning alone.Source: G Reasoning Mode at Long Context · p. 161, Figure 65 · p. 162

Related papers

Loading 2610.31945v1…