PaperlayerView exact revision on arXiv

Source-linked AI summary

Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

Project Apertus, :, Yixuan Xu, Yassine Turki, Vinko Sabolčec, Valentina Pyatkin, Tommy Chu, Sven Najem-Meyer, Sean F. Carroll, Robert Smith, Raphael Kreft, Oscar Alessandro Tazza, Oleg Lavrovsky, Nicola Irmiger, Matteo Santelmo, Marian Schneider, Juan García Giraldo, Hanna Yukhymenko, Eduard Ďurech, Davit Melikidze, Dani Tamayo, Bettina Messmer, Aryan Ahadinia, Anunay Yadav, Allen Huang, Ahmad Fraij, Alessandro Bombarda, Alex Padula, Alexander Hägele, Alexander Sternfeld, Anna Hedström, Anna Kosovskaia, Arthur Wuhrmann, Arundhati Balasubramaniam, Badr Al Mahouri, Barna Pasztor, Bartosz Szostakiewicz, Bich Ngoc Doan, Bryan Schmid, Burak Sezen, Camille Challier, Clément Charmillot, Dana Shayakhmetova, Daniel Perruchoud, David Brüggemann, David Isztl, Elia Palme, Emma Anastassova, Fabian Bösch, Faruk Zahiragić, Fawzi Roberto Mohamed, Fiona Panicali, Florian Kolly, Flynn Guerra, Francesco Monti, Frederike Lübeck, Gennaro Oliva, Haofeng Yang, Haoliang Shang, Helen El Khoury, Henrique Mendonça, Jannis Vamvas, Jérome Landolt, Jessica Lam, Jingchun Xiao, Jonathan Coles, Joshua Oyebanji, Khanh Nguyen, Konstantinos Pitas, Lasse B. Strand, Li Mi, Loïc Deslarzes, Lorenzo Baggi, Lorenzo Noci, Luca Baumann, Luca Sartori, Lukas Drescher, Marco De Negri, Marco Passerini, Mark Klein, Marko Stojanovic, Martin Majsec, Matin Ansaripour, Mattea Busato, Mauro Pellonara, Max Neuwinger, Michael Graber, Michal Tešnar, Michelle Wastl, Miguel Gila, Mikolaj Boronski, Morgane Magnin, Moritz Reihs, Mukhammadali Sayfiddinov, Nathan Ranchin, Nicolas Cuveillier, Oscar Wohlfahrt, Qianqian Zou, Rada Kamysheva, Rita Sevastjanova, Roman Machacek, Sebastian Keller, Simon Matrenok, Simon Pierre, Stefano Schuppli, Suei-Wen Chen, Szabina Horváth-Mikulas, Taliesin Perez, Thilo Spinner, Thomas Kleine Buening, Tim Robinson, Tymoteusz Kwiecinski, Vilém Zouhar, Vinzenz Thoma, Xiaozhe Yao, Yannick Metz, Yuhei Fukuhara, Yusif Askari, Zixuan Chen, Alexander Ilic, Ana Klimovic, Andreas Krause, Andrei Kucharavy, Elliott Ash, Joost VandeVondele, Martin Rajman, Mennatallah El-Assady, Thomas Schulthess, Timo Kehrer, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag

arXiv:2610.15542v1cs.CL

TL;DR

Apertus 1.0 established fully open training at scale, while the Apertus family had yet to add image and audio understanding. Apertus 1.5 adds those capabilities through continued pretraining, and its released models improve over predecessors on nearly every benchmark.

  • Problem

    Apertus 1.0 established fully open training at scale, while the Apertus family had yet to add image and audio understanding.

    Source: §1 · p. 6

  • Method

    Continued pretraining of Apertus 1.0 checkpoints on text, image, and audio produces 8B and 70B models, followed by post-training to released models.

    Source: §1 · p. 6

  • Results

    Released models improve over predecessors on nearly every benchmark; Thinking lifts Apertus 1.5 8B from 51.2 to 71.4 on MATH-500.

    Source: §8 · p. 72

  • Takeaways & Limitations

    For language pairs also covered by WMT, SwissGov yields the same relative ranking, suggesting observed performance differences generalize beyond the news domain.

    Source: §5.4.1 · p. 55

    Tool calling and thinking do not combine in released models: with thinking enabled, they either omit tool calls or tend to skip reasoning.

    Source: §7.2 · p. 69

Abstract

from arXiv · show

Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.

5 Evaluations

The supplied passage identifies an appendix subsection titled “Tokenizer Reconstruction by Category.”

5 Evaluations

1 INTRODUCTION

Apertus 1.5 extends the family’s open foundation with multimodal inputs, longer context, and stronger post-training capabilities. The introduction also describes how models, code, and datasets are released for inspection and reuse.

1 INTRODUCTION

  • Apertus 1.5 adds image and audio understanding, optional thinking, a fourfold longer context window, and stronger instruction following and tool use.The models build on Apertus 1.0 through continued pretraining on text, image, and audio data.Source: §1 · p. 6
  • Multimodality enters through the tokenizer while the language-model head remains text-only and the architecture is preserved.The models also receive dedicated long-context stages in continued pretraining and post-training, extending context to 262,144 tokens.Source: §1 · p. 6
  • Post-training combines SFT, long-context SFT, RLVR, DPO, and charter alignment with SDPO to deliver the new interaction capabilities.Thinking is an optional feature of the post-training pipeline.Source: §1 · p. 7, Figure 1 · p. 7
  • The release includes weights, intermediate checkpoints, data preparation pipelines, training code, and evaluation harnesses under Apache 2.0.The introduction defines openness across weights, data, code, recipes, and values, with claims traceable to released artifacts.Source: §1 · pp. 6–7

1.1 RELEASED MODELS, CODE, AND DATASETS

  • The released models are available through gated downloads, and thinking is a request-time chat-template flag rather than a separate model.Stage checkpoints allow model lineage to be inspected checkpoint by checkpoint.Source: §1.1 · p. 7
  • The release makes training datasets auditable and includes the SFT mixture, preference data, curated datasets, and the full training and evaluation stack.The code release includes multimodal pretraining and SFT extensions, the tokenizer, data ingestion inventory, post-training framework, and evaluation harnesses.Source: §1.1 · p. 8

2 MULTIMODALITY

This section describes how Apertus 1.5 adds image and audio understanding through shared discrete-token representations and staged training, covering tokenizers, data preparation, and pipelines within the common language-model framework. It also examines ablations of modality alignment and image-token learning, including trade-offs among token cost, reconstruction detail, image understanding, and language-dependent capabilities.

2.1 UNIFIED REPRESENTATION

  • Apertus 1.5 represents images and audio as discrete tokens in a shared vocabulary, enabling one next-token objective without encoder computation in the model’s forward pass.Precomputed media tokens enter the standard data loader, keeping multimodal training close to the existing text-training infrastructure.Source: §2.1 · p. 9, §2.3 · p. 16

2.1.2 AUDIO

  • WavTokenizer emits one code per frame, creating a flat audio-token sequence; its 4,096-entry codec yields 40 tokens per second at 24 kHz.Audio is peak-normalized to −3 dBFS before offline tokenization.Source: §2.1.2 · p. 11

2.2 MULTIMODAL TRAINING DATA

  • Across six training stages, text comprises roughly 60% of the 8B mix and 55% of the 70B mix; the 8B vision pools contain 821.6 million samples.The stages span continued pretraining, long-context pretraining, supervised finetuning, and long-context supervised finetuning.Source: §2.2 · p. 11, §2.2.2 · p. 12

2.2.1 CURATION AND PROCESSING

  • Data curation uses domain-specific deduplication and caption validation, but disables perceptual hashing for later OCR- and document-focused datasets after valid examples were removed.The raw MINT-1T snapshot occupies approximately 700 TB, and processing at scale requires early compression and coordination with shared-infrastructure teams.Source: §2.2.1 · p. 12

2.2.3 VISION DATA COMPOSITION

  • General visual material dominates the vision pool, while later stages shift toward charts and diagrams; medical content is more prominent in the 70B cooldown pool.Stage 2 holds the largest vision pools: approximately 850 billion input tokens for 8B and 534 billion for 70B.Source: Figure 3 · p. 13, §2.2.3 · p. 13, Figure 4 · p. 13

2.2.4 AUDIO TRAINING DATA

  • Web and broadcast speech dominate pretraining audio pools, while dialogue and question answering dominate both supervised-finetuning pools.Stage 1 uses unpaired speech, music, and mixed audio; later stages concentrate on paired speech and text, including interleaved audio-text sequences.Source: §2.2.4 · p. 14, Figure 5 · p. 14

2.3 TOKENIZATION AND TRAINING PIPELINES

  • Offline image and audio tokenization pipelines write Megatron indexed datasets, allowing pretokenized multimodal sequences to use standard data blending and loaders.The image pipeline’s encoder and host-side changes roughly double throughput without changing emitted tokens; audio batching groups clips by duration to limit padding.Source: §2.3 · p. 16, §2.3.1 · p. 16, §2.3.2 · p. 16

2.4 ABLATIONS

  • Emu3.5 is selected over Emu3 for token efficiency, representing the same image with one-quarter as many tokens; Emu-family tokenizers offer a favorable reconstruction-quality and token-efficiency trade-off.The adopted Emu3.5 tokenizer is used to convert training images into discrete tokens offline.Source: §2.1.1 · p. 10, §2.4 · p. 17
  • Emu3.5 uses a quarter of Emu3’s tokens at an average cost of 2.7 dB PSNR, 0.07 SSIM, and 0.02 LPIPS, with the largest PSNR losses on charts and diagrams, medical images, and documents.The coarser grid covers approximately four times as many training images per fixed token budget and makes image-input inference cheaper; OCR remains a relative weakness.Source: §2.4 · p. 17, Table 1 · p. 17
  • Zero image-token weight during modality alignment was strongest for image understanding, while equal image/text loss weights scored far below other settings on nearly every benchmark group.The controlled recipe split a topic-balanced image-caption corpus between standalone-image representation learning and image-caption modality alignment, with every batch replaying 10% text-only data.Source: §2.4 · pp. 17–18
  • 0.50 to 0.54 on GQA, 0.67 to 0.71 on DocVQA, and 0.61 to 0.65 on SEED-Bench followed a roughly 57B-image-token representation-learning budget.Gains flattened at larger budgets, supporting a bounded Stage 1 token budget.Source: §2.4 · p. 18
  • At the largest representation-learning budgets, knowledge-heavy image and text benchmarks eroded and did not fully recover in later stages, motivating a larger text share in the mixtures.Source: §2.4 · p. 18

3 CONTINUED PRETRAINING RECIPE

Apertus 1.5 continued pretraining adds multimodal inputs and a 262K-token context window through a staged recipe that retains text-only prediction after initial multimodal training. The retrospective identifies learning-rate scheduling, data exposure, and model capacity as important constraints on outcomes.

3.1 PRETRAINING DATA

  • Unpaired multimodal training predicts all modalities, while paired training aligns images and audio with language using text-only prediction targets.The recipe continues Apertus 1.0 with public datasets and documented provenance, then adds higher-quality cooldown data and a long-context extension.Source: §3.1 · p. 19, §3 · p. 19

3.1.1 COMPLIANCE AND PROVENANCE

  • Data review excludes closed or restrictively licensed sources and applies machine-readable crawler opt-outs retroactively where records retain source-page URLs.Network safety filtering protects shared infrastructure but is not a comprehensive scan for sexual or violent content.Source: §3.1.1 · pp. 19–20

3.1.2 TEXT DATA SELECTION

  • New text sources must pass provenance and license review and improve a small-scale ablation; FineWeb2-HQ+ broadens quality filtering beyond the original 20 languages.Its joint multilingual classifier uses expanded positive samples and lower-resource-language upsampling, with negative examples drawn from a proxy classifier’s 50th–75th percentile.Source: §3.1.2 · p. 20

3.2 OPTIMIZER AND TRAINING RECIPE

  • Three pre-long-context stages move from all-modality prediction on unpaired data to text-only targets on paired data and cooldown; the 70B uses half its intended token budget.The 70B budget reduction is mainly in Stage 2, while cooldown length is unchanged; the table confirms the stages’ learning-rate phases and prediction targets.Source: §3.2 · pp. 20–21, Table 2 · p. 22

3.3 LONG-CONTEXT EXTENSION

  • The long-context extension increases context length in four steps from 8,192 to 262,144 tokens and adds synthetic tasks that train use of distant information.At the 32K, 65K, and 131K phases, native long-context images supply about 79%, 47%, and 34% of the fixed image budget, respectively; short-context images fill the remainder.Source: §3.3 · p. 23

3.4 PRETRAINING RUN RETROSPECTIVE

  • Stage 1 text-loss creep is driven mainly by restarting the learning-rate schedule after cooldown, then reverses during the Stage 3 cooldown.The text-only loss comparison across stages makes the creep and recovery visible; mixed-modality gradient interference is also consistent with the observed creep.Source: §3.4 · p. 24, Figure 7 · p. 25
  • Small instruction-data shares improved math, code, and instruction following; the 70B gained little multimodal performance despite ninefold size, with incomplete data exposure and a likely low base rate cited as causes.Fivefold replay regressed safety behavior through overfitting on small refusal subsets; preference alignment and reinforcement learning improved long-context evaluations without long-document training.Source: §3.4 · p. 25

4 POST-TRAINING

This section describes post-training that combines multimodal supervised finetuning and long-context recovery with verifiable-reward reinforcement learning, preference optimization, and charter alignment to shape assistant behavior. It examines how these stages and reasoning traces affect instruction following and text and multimodal capabilities, including development-benchmark gains and inference-mode-dependent reasoning-trace effects.

4.1 CHAT TEMPLATE

  • The chat template adds per-request thinking spans and explicit tool-output delimiters that mask tool outputs from loss; an empty system block prevents user content from occupying the system slot.Thinking delimiters remain parseable after detokenization, and the template supports tool calls inside thinking spans.Source: §4.1 · p. 27

4.2 SUPERVISED FINETUNING

  • Production SFT trains text, image, and speech inputs for text answers across instruction following, multilingual interaction, reasoning, code, tool use, safety, and identity.Pass@8 capability filtering leaves prompts solved at least once without reasoning traces and assigns verified traces to prompts the auxiliary 8B model fails all eight times; trace supervision adds 7–18 mathematics points but costs 12–14 instruction-following points, mostly from repetition in about a quarter of traces.Source: §4.2 · pp. 28–29

4.2.2 AUDIO

  • The audio pool contains six speech-only datasets with roughly 1.1B tokens, all training speech input to text output; it excludes music and environmental-sound supervision.These are prepared-pool counts, not sampling weights, repetition factors, or tokens actually consumed by the runs.Source: §4.2.2 · p. 33

4.2.3 VARIETY-AWARE TRANSLATION CASE STUDY

  • The Romansh case study adds 5,000 German-backtranslated SFT samples per variety, using variety identification and machine translation to target all six varieties.The supplied passages describe the method and point to translation evaluations, but do not provide their results.Source: §4.2.3 · p. 33

4.3 LONG-CONTEXT SUPERVISED FINETUNING

  • After 16,384-token SFT, Apertus 1.5 8B’s RULER score at 262,144 tokens falls from 54.8 to 8.3; dedicated full-length SFT addresses this retrieval loss.The resulting checkpoints are evaluated across context lengths, and the best is selected for reinforcement learning.Source: §4.3 · pp. 34–35

4.4 IMPROVING INSTRUCTION FOLLOWING

  • Online DPO atop RL adds 3.8 IFEval points and 1.5 Multi-IF points at 8B; at 70B, preference alignment raises AlpacaEval from 25.9 to 59.4.At 8B, online DPO alone matches verifier-driven RL on IFEval and comes within two points on Multi-IF; the section also describes a 30,000-dialogue dataset targeting vague, shifting constraints.Source: §4.4 · pp. 35–36, Table 9 · p. 36

4.5 REINFORCEMENT LEARNING

  • The mixed thinking-on/off RL configuration performs better and supplies the releases; always-on thinking produces overlong traces and rarely calls tools.Rewards are verifiable checks or exact matches, while tool-use prompts were excluded from production RL after hosted servers became a throughput bottleneck, leaving released tool behavior from SFT.Source: §4.5 · pp. 36–37

4.6.1 DIRECT PREFERENCE OPTIMIZATION

  • 8B uses online DPO alone, while 70B uses offline DPO before online DPO because online DPO alone produced qualitatively short, brittle responses.Both stages use prompts from a filtered, decontaminated Dolci-Instruct-DPO subset; offline preference pairs are formed by scoring responses across four aspects and selecting the highest- and lowest-scoring responses.Source: §4.6.1 · p. 38
  • Preference alignment raises development-suite averages from 49.3% to 54.8% for 8B and 55.3% to 61.2% for 70B, with broad gains and fewer degenerate outputs.Thinking-mode averages also rise by 4.2 points for 8B and 6.6 for 70B, although neither alignment stage trains with thinking enabled.Source: §4.6.1 · p. 39

4.6.2 CHARTER ALIGNMENT WITH SDPO

  • SDPO charter alignment ships in 8B but not 70B, whose initial capability regressions were traced to an inconsistent tokenizer configuration.With the configuration fixed, 70B training is stable, with a 2.4% charter-alignment increase, a 1.1% coding gain, and a 1.2% chat drop; SDPO distills feedback-conditioned teacher probabilities into the student.Source: §4.6.2 · pp. 39–40

4.7 SFT ABLATIONS

  • Multimodal SFT preserves text performance, while trace supervision benefits multimodal tasks when thinking is enabled but lowers vision scores by 2.2 points with thinking disabled.The paired CoT and No-CoT configurations keep prompts, multimodal inputs, and visible answers identical, differing only in whether reasoning traces are retained.Source: §4.7 · pp. 40–42, Figure 12 · p. 41
  • During subsequent text-only RL, MM CoT retains or increases its advantage over Text CoT on mathematics and reasoning, while the instruction-following runs converge.The reported comparison uses controlled Text CoT and seed-42 MM CoT checkpoints; instruction-following RL runs with thinking off.Source: §4.7 · pp. 42–43, Figure 13 · p. 43

5 EVALUATIONS

This section evaluates Apertus across text, multilingual, multimodal, long-context, tool-use, safety, and interpretability, including release-safety practices and qualitative testing while accounting for benchmark and serving pitfalls. It reports broad gains alongside task- and model-dependent limitations and persistent vulnerabilities.

5.1 EVALUATION SETUP

  • Evaluation uses public, comparable benchmark configurations and controls for pitfalls including unsuitable decoding defaults, small-sample variance, tool-call parsing, and backend differences.The text suite grows from 27 benchmarks to roughly 50; implementations are checked against published scores of openly released models, and unreliable tasks are excluded.Source: §5.1 · p. 44

5.2 BASE MODELS

  • The base models improve across a wide range of tasks; reasoning, math, and coding strengthen markedly, while multilingual performance remains strong despite small regressions.The 8B compares well with fully open peers, and the models mostly close gaps in general language understanding and factual knowledge.Source: §5.2 · p. 45

5.3 POST-TRAINED MODELS

  • Apertus 1.5 8B improves over 1.0 on all but TruthfulQA and GPQA, leads fully open peers on knowledge, instruction following, and culture, and all peers on instruction following and AGIEval.Apertus 1.5 70B improves over its predecessor on every benchmark and outperforms EuroLLM-22B on nearly all, but trails the strongest open-weight contemporaries on most axes.Source: §5.3 · p. 48, Table 23 · p. 49, Table 24 · p. 50

5.3.1 REASONING AND THINKING MODE

  • 71.4 from 51.2: thinking raises 8B MATH-500; it also improves MATH from 41.2 to 57.3 and IFBench from 35.3 to 44.3.Thinking is a request-time chat-template flag, not a separate model; it also raises 8B AIME 24 from 7.5 to 22.9 and AIME 25 from 8.3 to 23.8, while knowledge benchmarks move less and 70B GPQA declines.Source: §5.3.1 · p. 48, Table 25 · p. 50

5.3.2 POST-TRAINING LINEAGE

  • Post-training raises overall averages from 47.3% to 55.0% for 8B and from 52.2% to 61.2% for 70B, with held-out averages also increasing.Reinforcement learning contributes most to reasoning and instruction following, preference alignment to conversational and agentic tasks, and SDPO chiefly to charter alignment; alignment also increases benign-prompt refusals, while RL shortens 70B responses before offline DPO lengthens them.Source: §5.3.2 · p. 49, Table 26 · p. 51, Table 27 · p. 51

5.4 MULTILINGUAL AND CULTURAL EVALUATION

  • Apertus 1.5 8B leads compared models on multilingual ARC and Multi-IF, while thinking lifts Global-MMLU by eight points; XNLI regresses against 1.0 at both scales.Targeted post-training lets 8B surpass comparable models on German-to-Romansh translation; human evaluation confirms it avoids critical failures on Switzerland-specific pairs, and SwissGov rankings match WMT rankings for overlapping pairs. On LTBv1-eval, 8B scores 1.32%, versus 0.77% for Apertus 1.0 8B and 0.66% for 1.5 70B.Source: §5.4 · p. 52, Table 33 · p. 54, §5.4.1 · p. 55

5.5 IMAGE UNDERSTANDING

  • Within fully open models, 8B leads on document and chart understanding and general VQA, while stronger open-weight models lead overall; thinking helps reasoning-heavy but hurts extraction-style tasks.On MMMU, thinking raises the score from 39.8 to 52.2; on ChartQA, it lowers the score from 78.7 to 66.0.Source: §5.5 · p. 55, Table 34 · p. 56

5.6 AUDIO UNDERSTANDING; 5.7 LONG CONTEXT

  • Apertus is strongest on long-form and some lower-resource speech recognition but trails peers on audio understanding; long-context SFT restores performance lost after short-context SFT.Long-context extension carries the models to 256k tokens, and the 70B pipeline stays ahead of 8B through 128k on RULER and HELMET. Apertus 1.5 8B scores 36.0% on TED-LIUM long form, while vocal-sound classification is at or below chance for both sizes.Source: §5.6 · p. 57, §5.7 · p. 58, Table 35 · p. 58

5.8 TOOL USE

  • The 70B’s BFCL v3 single-turn overall accuracy rises from 58.0 to 70.1, within 1.5 points of Llama-3.3-70B, while the 8B remains weak on parallel calls.The evaluation excludes multi-turn categories, so its scores are not comparable to official leaderboard entries; the 8B nearly doubles its predecessor’s irrelevance-category score.Source: §5.8 · p. 59, Table 37 · p. 60

5.9 SECURITY, SAFETY, AND INTERPRETABILITY

  • Release safety is evaluated separately from model performance, with a three-stage approach that addresses ordinary use, standardized benchmarks, and manual review of release candidates.Safety behavior is concentrated in alignment, while over-refusal is tracked; only BBQ, ToxiGen, and WMDP inform alignment iteration, and other safety benchmarks are held out. Manual review covers dangerous-use capabilities and abliterated models are evaluated to assess intrinsic capabilities.Source: §5.9 · pp. 60–61

5.9.1 SAFETY BENCHMARK PERFORMANCE

  • Apertus 1.5 improves several safety measures, while the 8B remains vulnerable to human-written jailbreaks and multimodal jailbreak scores vary against open-weight comparators.The 8B has 1.3% attack success on direct harmful requests, 31.5% on human-written jailbreaks, and 26.5% average request success; MM-SafetyBench scores are 38.0% for 8B and 28.5% for 70B (lower is better).Source: Table 38 · p. 61, §5.9.1 · p. 61

5.9.2 OVER-REFUSAL AND HALLUCINATION

  • Alignment raises OR-Bench refusal rates, while HalluLens shows improved abstention on fabricated entities and long-form F1 but no improvement in short-form hallucination.Refusal rates rise from 3.6% after SFT to 81.5% after alignment at 8B and from 2.9% to 67.6% at 70B; HalluLens absolute scores are not comparable with its official leaderboard because the judge differs.Source: §5.9.2 · pp. 61–62, Table 39 · p. 62

5.9.3 INTERPRETABILITY

  • Apertus Claritas publishes interpretability studies, including findings on exploding activations, degeneration probes, and low-resource translation’s reliance on related high-resource representations.Layer-normalized hallucination probes reach AUC 0.90; degeneration probes detect repetitive collapse most reliably from layer 15 onward; suppressing high-resource directions degrades Galician and Afrikaans translation, whereas the reverse intervention has almost no effect.Source: §5.9.3 · pp. 62–63

5.10 QUALITATIVE SPOT-TESTING

  • Hand testing finds long-conversation degeneration and identity confusion, and informs an automated repetition metric and regression checks across post-training stages.Each release candidate also passes roughly 65 automated compatibility checks covering prompt invariants, multimodal inputs, reasoning-channel separation, tool calling, and streaming.Source: §5.10 · p. 63

6 INFRASTRUCTURE, SCALING, AND EFFICIENCY

The Apertus 1.5 campaign scaled continued pretraining on Alps and addressed performance, energy, and operational challenges across its production pipeline. At 256 nodes, tuning sustained roughly 750 tokens per second per GPU for the 70B model, while scaling to 512 nodes retained roughly 90% parallel efficiency.

6.1 INFRASTRUCTURE

  • Production used 256 nodes for 8B and 512 for 70B within a reservation that grew from approximately 150 to 600 GH200 nodes; unrelated workloads sometimes constrained availability.For about two weeks, training continued in available night and weekend reservation windows.Source: §6.1 · p. 64

6.2 FULL TRAINING RUN PERFORMANCE

  • The 8B sustains roughly 4,900 to 5,500 tokens per second per GPU and the 70B 430 to 710; doubling context lowers throughput, while delivered 70B progress is roughly 30% below sustained rates.The 70B gap reflects startup, checkpoint saving and loading, and stalls before failures.Source: §6.2 · pp. 64–65, Table 40 · p. 65
  • At 256 nodes, tuning reached roughly 750 tokens per second per GPU for 70B despite doubled sequence length; scaling to 512 nodes retained roughly 90% parallel efficiency.Table 40 aggregates production-phase performance from logged run segments.Source: §6.2 · p. 64, Table 40 · p. 65

6.3 ENERGY CONSUMPTION

  • The logged production segments total roughly 2 million GPU-hours or 1.1 GWh, versus roughly 5 GWh for Apertus 1.0 from-scratch pretraining.The estimate assumes 560 W per GH200 module and excludes missing production phases, post-training, evaluation, ablations, and idle reservation time.Source: §6.3 · p. 65

6.4 ENGINEERING CHALLENGES AND SOLUTIONS

  • To reduce sensitivity to system upgrades, the team incorporated required network components into its container environment and added pre-run node validation and per-process JIT caches.At scales beyond 256 nodes, an alternative rendezvous protocol avoided a separate collective-communication performance collapse.Source: §6.4 · pp. 65–66
  • Storage was matched to access patterns, and optimizer state—roughly 90% of checkpoint footprint—was retained only at phase boundaries.For the 70B model, removing optimizer state reduces a checkpoint from approximately 1.2 TB to 132 GB.Source: §6.4 · p. 66
  • URL-manifest ingestion risked disrupting shared egress; proxy filtering converted threat feeds into access-control lists and rejected fewer than 0.01% of candidate downloads.Filtering was developed with CSCS network and security teams after denied destinations triggered monitoring and ISP blacklisting risks.Source: §6.4 · pp. 66–67
  • The asynchronous reinforcement-learning stack required adapting weight transfer and checkpoint loading; end-to-end rollout throughput came within approximately 5% of comparable open models on the same software stack.Reward environments and code-sandbox test harnesses run on CPU-only Kubernetes services outside the GPU allocation.Source: §6.4 · p. 67

7 RESPONSIBLE RELEASE AND GOVERNANCE

Apertus 1.5 documents its release, limitations, broader impact, and community evaluation. The report describes open artifacts and governance practices while identifying capability boundaries and unresolved issues.

7.1 LICENSE AND ACCEPTABLE USE

  • Weights, training and evaluation code, public datasets, and transparency documents are released; the report does not claim the documents establish legal compliance.The models use gated downloads that record assent to the acceptable-use policy.Source: §7.1 · p. 68, §7 · p. 68

7.2 LIMITATIONS

  • On PreciseWiki, models answer more than eight of 10 short factual questions and hallucinate on roughly six to seven; alignment also increases refusals on a hard OR-Bench subset.Some OR-Bench prompts are legitimately refusable, so the subset shows a trend rather than an absolute over-refusal rate.Source: §7.2 · p. 68
  • XNLI regresses against Apertus 1.0 at both scales, long-tail language quality is unevaluated, and the models understand but do not generate images or audio.The 70B run stopped before consuming its full multimodal mixture; audio training skews toward parliamentary and political speech.Source: §7.2 · p. 68
  • Preference alignment degrades Ref-COCO grounding by about 17 points as predicted boxes become looser; models also sometimes claim other providers created them, particularly in lower-resource languages.The report accepts the grounding trade-off and identifies grounding-aware rewards as a future direction.Source: §7.2 · pp. 68–69
  • Thinking and tool calling are mutually exclusive in released models, which are not designed for agentic use and lack multi-turn tool-use evaluation.Tool-use behavior comes from supervised finetuning alone; the 8B also remains weak on parallel calls.Source: §7.2 · p. 69

7.3 BROADER IMPACT

  • Published data, recipes, code, and charter-alignment materials let institutions inspect and adapt models without vendor dependence, while open weights can also be misused or finetuned to remove safety behavior.The report describes alignment, an acceptable-use policy, a data-protection channel, and external evaluation as responses to these risks.Source: §7.3 · pp. 69–70
  • Multilingual and cultural benchmarks are the models’ strongest relative position; Apertus 1.0 pretraining data was roughly 40% non-English, and this release adds multilingual instruction data.The reported evaluations include Swiss national languages and Romansh, with benefits bounded by the language-coverage limitations.Source: §7.3 · p. 69

7.4 COMMUNITY AND EXTERNAL EVALUATION

  • Pre-release deployment testing by 15 organizations and roughly 50 experts surfaced issues missed internally; 70B serving latency ranged from 1.9 to 4.3 s, while provider quality comparison remains open.Feedback led to quantizations and community builds; the serving comparison used one run and gave the authors’ deployment a network advantage.Source: §7.4 · pp. 70–71

8 CONCLUSION

Apertus 1.5 extends the family to image and audio understanding, lengthens context, and improves benchmark performance. The conclusion highlights training lessons and keeps the release’s recipes and artifacts public.

8 CONCLUSION

  • The models improve over predecessors on nearly every benchmark; the 8B leads among fully open peers on knowledge, instruction following, and cultural benchmarks.Thinking raises 8B MATH-500 from 51.2 to 71.4; instruction following reaches 87.1 on IFEval and 83.7 on Multi-IF, above every compared model.Source: §8 · p. 72
  • Continued pretraining adds image and audio understanding and extends context fourfold to 262,144 tokens, while a second-generation post-training pipeline adds thinking and improves instruction following and tool calling.The 70B run stopped before consuming its full multimodal mixture, illustrating that multimodal capability follows exposure.Source: §8 · p. 72
  • The reported recipe lessons include tokenizer-based modality additions without architectural changes, dedicated long-context training, reasoning-data filtering, and offline preference optimization before online optimization at 70B scale.The dedicated long-context stage after supervised finetuning preserved the context extension.Source: §8 · p. 72
  • Future work targets short-form hallucination and combining tool calling with thinking; weights, intermediate checkpoints, data recipes, training and evaluation code, and the alignment charter remain public.The conclusion identifies these as remaining gaps from the release.Source: §8 · p. 72

B VISION DATASETS BY TRAINING STAGE

The appendix inventories vision data pools across training stages. It distinguishes samples and input tokens available to a run from the mixture actually consumed during training.

B VISION DATASETS BY TRAINING STAGE

  • Stagewise vision-pool tables list prepared samples and input tokens available for each training stage; the trained mixture, including repeated tokens, is reported separately.The 8B pools are reported for Stage 2 and cooldown, while the 70B cooldown also reuses Stage 2 datasets without repeating their entries.Source: B VISION DATASETS BY TRAINING STAGE · p. 93
  • The SFT tables show production pools; the capability-filtering SFT phase uses a smaller image pool, while the audio pool is unchanged.Long-context SFT’s vision pool is listed separately.Source: B VISION DATASETS BY TRAINING STAGE · p. 93, Table B.5 · p. 94, Table B.6 · p. 94
  • The tables identify vision pools for Stage 1, Stage 2, cooldown, and long-context extension, with the latter’s per-run training mix reported in Appendix F.The table captions specify each pool’s stage.Source: Table B.1 · p. 93, Table B.2 · p. 93, Table B.3 · p. 94

C AUDIO DATASETS BY TRAINING STAGE

The appendix points readers to stage-specific audio pool tables and to separate sources for per-run mixtures in long-context training.

C AUDIO DATASETS BY TRAINING STAGE

D IMAGE DATASET EXPLORATION DETAILED BREAKDOWN

The appendix maps image-dataset coverage with a 61-label taxonomy and finds substantial imbalances across image types and sources. A qualitative check supports the taxonomy’s category coherence, while task signals overlap across the training inventory.

D IMAGE DATASET EXPLORATION DETAILED BREAKDOWN

E TOKENIZER RECONSTRUCTION BY CATEGORY

  • Table E.1 compares Emu3 and Emu3.5 reconstruction quality and token cost across 350 training images grouped into six representative categories.The table reports mean tokens per image and marks the better tokenizer for each metric and category.Source: Table E.1 · p. 104

F LONG-CONTEXT DATA MIXTURE TABLES

The appendix provides per-dataset token counts for both models across the four long-context training steps, with specific short-context sources contributing to selected mixture rows.

F LONG-CONTEXT DATA MIXTURE TABLES

G TRAINING CONFIGURATIONS

The appendix specifies optimizer, data-packing, and training-stage configurations for continued pretraining and post-training. Long-context training uses a staged sequence-length curriculum, followed by distinct SFT, reinforcement-learning, and preference-alignment setups.

G TRAINING CONFIGURATIONS

  • Continued pretraining uses AdEMAMix, gradient clipping at 0.1, weight decay of 0.1, and cross-document attention masking with position resets at document boundaries.The optimizer state is fresh at stage 1 and carries over across later stage transitions.Source: G TRAINING CONFIGURATIONS · p. 107
  • The long-context curriculum progresses through 32,768, 65,536, 131,072, and 262,144 tokens, loading the previous checkpoint with a fresh optimizer state at each step.The global batch size halves with each sequence-length doubling for the 8B model.Source: G TRAINING CONFIGURATIONS · p. 107
  • Both production SFT runs use Adam without weight decay, clip gradients at 1.0, apply loss only to assistant regions, and pack samples with best-fit decreasing.The chat template is applied offline during data preparation.Source: G TRAINING CONFIGURATIONS · p. 107
  • The reinforcement-learning stage uses eight rollouts per prompt, filters groups with zero reward variance, and runs 480 steps for 8B and 180 for 70B.It uses leave-one-out advantage estimation, asymmetric clipping, no KL penalty, and AdamW at 1 × 10^-6.Source: G TRAINING CONFIGURATIONS · p. 107
  • The SDPO charter-alignment stage uses 16 rollouts per prompt and keeps the teacher fixed at the initial policy throughout training.It trains for one epoch with a 4,096-token maximum model length; the teacher template combines the prompt with feedback and instructs the model not to mention it.Source: G TRAINING CONFIGURATIONS · pp. 107–108
  • Offline DPO trains the 70B model only, while online DPO trains both models; neither stage trains a reward model or uses image inputs.Offline DPO divides β by sequence length under length normalization.Source: G TRAINING CONFIGURATIONS · p. 107, Table G.2 · p. 108

H.1 PER-BENCHMARK SFT ABLATION RESULTS

The appendix reports matched-supervision and thinking-mode comparisons behind Figure 12, alongside decontamination statistics for the SFT data. The MGSM comparison has an answer-style caveat, and the overall removal rate includes many false positives.

H.1 PER-BENCHMARK SFT ABLATION RESULTS

  • Table H.1 reports percentage-point changes for MM versus Text, MM CoT versus MM No-CoT, and MM CoT versus MM No-CoT with thinking off.Multimodal values average three seeds, while the text-only reference uses one seed.Source: §H.1 · p. 109, Table H.1 · p. 109
  • MGSM is excluded from the text average because its step-by-step prompt yields much longer text-only No-CoT answers than multimodal No-CoT answers.With reasoning traces in SFT, the multimodal arm writes out its work and scores 69.9 with thinking off, level with the text-only CoT arm at 69.2.Source: Table H.1 · p. 109

H.2 SFT DATA DECONTAMINATION DETAILS

  • 2.94% of SFT examples were removed overall, and false positives—common in multilingual data—dominate the high removal rate.The table reports original, removed, and final example counts per source before mixing.Source: §H.2 · p. 110, Table H.2 · p. 110

I PREFERENCE ALIGNMENT DATA

The appendix documents the preference data and judging process used for alignment, from prompt and response-pool construction through scoring and data release. It also describes DPO ablations and cautions that their scores are not directly comparable with the main-text results.

I PREFERENCE ALIGNMENT DATA

  • The released artifacts include preference pairs and the underlying annotated completions.The appendix documents the judge rubric, prompt-set composition, response-model pool, and resulting score distribution.Source: I PREFERENCE ALIGNMENT DATA · p. 111

I.1 JUDGE RUBRIC PROMPTS

  • The four-aspect judge scores completions on truthfulness, helpfulness, honesty, and instruction following, using deterministic expected ratings from 1 to 5.The final score is the mean of the four aspect scores; across 6,235,908 completions, aspect correlations range from 0.838 to 0.916.Source: §I.1 · pp. 111–113

I.2 PROMPT SET COMPOSITION

  • Decontamination and a length filter reduce 259,922 source prompts to 245,898 preference pairs, retaining 94.6% unevenly across sources.The two precise instruction-following tags lose 12–19%, while most other sources retain at least 95%.Source: §I.2 · p. 113, Table I.1 · p. 114

I.3 RESPONSE MODEL POOL

  • The response pool covers 24 models from 0.5B to 675B parameters, with 1,969 prompts lacking a Qwen3-Omni completion and a possible judge self-preference effect.The judge supplies 21.6% of chosen responses.Source: Table I.2 · p. 114, §I.3 · p. 115

I.4 COMPLETION GENERATION

  • Responses are sampled once per model at temperature 1.0, but models use their own default top-p and top-k settings, so decoding configurations differ.Thinking is disabled throughout.Source: §I.4 · p. 115

I.5 SCORE DISTRIBUTION

  • Max-min selection yields a mean score margin of 3.766, with 86.5% of chosen responses scoring 4.9–5.0 and 96.7% of rejected responses at 1.0.The mean gap is 3.822 before the 3,600-token filter and 3.766 after it.Source: §I.5 · p. 115, Table I.3 · p. 115, Table I.4 · p. 115

J.1 DPO ABLATIONS

  • The DPO appendix sweeps learning rate, KL penalty, and batch size and compares offline, online, and combined strategies; its intermediate-checkpoint scores are not comparable to Table 26.The ablations use earlier evaluation-suite versions and different judges for AlpacaEval and Arena-Hard.Source: §J.1 · p. 116

J.1.1 APERTUS 1.5 8B

The ablations select different DPO recipes at the two model scales and examine judge simplifications for online alignment. The final 8B comparison favors online DPO alone, while the 70B recipe uses offline DPO followed by online DPO.

J.1.1 APERTUS 1.5 8B

  • The 8B length-normalized offline DPO sweep reaches Δ=+5.5, while the best standard DPO run reaches Δ=+4.1.The selected length-normalized setting uses β=25 and learning rate 10−6; the final online learning-rate sweep favors 5·10−6 at Δ=+4.4.Source: §J.1.1 · p. 116, Table J.2 · p. 118, Table J.4 · p. 120
  • On the final Apertus 1.5 8B SFT checkpoints, online DPO matches the mixture, so the final recipe uses online DPO alone.The combined variants’ small gains on an intermediate checkpoint did not justify their extra data streams and training stages.Source: §J.1.1 · p. 117

J.1.2 APERTUS 1.5 70B

  • For Apertus 1.5 70B, the final recipe uses offline DPO followed by online DPO, restoring response length before on-policy training.Offline DPO from the RL checkpoint raises mean response length from 253 to 501 tokens; its average is 56.1, versus 55.6 for the mixture and 55.5 for online DPO.Source: §J.1.2 · p. 118, Table J.7 · p. 122

J.2.1 JUDGING ASPECTS SIMPLIFICATION

  • After removing prompt and model effects, the four judging aspects remain strongly correlated, supporting single-aspect scoring as a simplification.Helpfulness is selected over Truthfulness because Truthfulness is least correlated with the other aspects and Helpfulness performs strongly across the judging benchmarks.Source: §J.2.1 · p. 123, Table J.8 · p. 123, Table J.9 · p. 124

J.2.2 CHOICE OF JUDGE MODEL

  • Qwen3.6-27B matches or exceeds Qwen3-235B-A22B-Instruct-2507 on all three judging benchmarks and is chosen for online DPO for performance and cost.The largest reported gap is on RM-Bench’s Hard split.Source: §J.2.2 · p. 125, Table J.10 · p. 125

K CHATBOT SYSTEM PROMPT

  • When no system message is supplied, the chat template uses its default system prompt; it separately states whether thinking is enabled and declares available tools.The deployed chatbot relies on that default prompt.Source: K CHATBOT SYSTEM PROMPT · p. 126

L EVALUATION DETAILS

The evaluation appendix describes benchmark and pre-release safety procedures, then reports safety findings across distinct threat scenarios. Outcomes vary by threat and language, with some tested behaviors qualifying the release assessments.

L EVALUATION DETAILS; L.1 FULL POST-TRAINING LINEAGE

  • The evaluation appendix documents reproduction settings and benchmark suites, and provides the full 37-benchmark development-suite post-training lineage with behavioral measures.The table notes that over-refusal is measured on OR-Bench’s hard subset, where some prompts are legitimately refusable, so only trends across stages are meaningful.Source: L EVALUATION DETAILS · p. 127, §L.1 · p. 127, Table L.3 · p. 130

M.1 GENERAL INFORMATION ABOUT PRE-RELEASE EVALUATION

  • Pre-release evaluations cover English, German, French, Italian, and Serbian, test abliterated models for selected malicious-use scenarios, and include independent third-party evaluators.Manual evaluations are provided to inform release decisions.Source: §M.1 · p. 131

M.2 CHEMICAL, BIOLOGICAL, RADIOLOGICAL, NUCLEAR, AND EXPLOSIVE INDISCRIMINATE WEAPONS (CBRNE); M.3 DUAL USE AND INSURGENCY

  • In the CBRNE test cases, abliterated models produce realistic-looking recipes, but none examined contains sufficient information for household assembly or adds information beyond the web; neither release nor abliterated candidates provide useful dual-use information in analyzed cases.The CBRNE assessment also notes response variability across languages, including lower refusal in the tested low-resource language, without changing the release-safety evaluation.Source: §M.2 · p. 131, §M.3 · p. 131

M.4 LLM-ASSISTED INFORMATION OPERATIONS; M.5 (CHILD) SEXUAL ABUSE

  • The tested low-to-mid-resource language information-operation texts were not undetectable to common online free LLM detectors; child sexual-abuse assistance refusal and noncompliance improve over Apertus 1.0.Apertus 1.5 refusal and compliance rates are in line with similarly sized open-weight models, including after ablation.Source: §M.4 · p. 132, §M.5 · p. 132

M.6 OFFENSIVE CYBERSECURITY; M.7 MULTIMODAL PROMPT INJECTION

  • A third-party evaluation finds limited offensive-cyber usefulness, below common open-weight models and insufficient for automation; release candidates show no compliance with instructions embedded in images.The security team did not judge the cyber results sufficient to block release.Source: §M.6 · p. 132, §M.7 · p. 132

M.8 HARMFUL DISINFORMATION; M.9 HARMFUL HEALTH ADVICE

  • Released models show minimal disinformation recitation or disputed-territory partiality, while harmful-health-advice scenarios generally elicit warnings except in the tested low-resource language.The authors report improvement over commonly used open-weight models for harmful health advice and did not block release.Source: §M.8 · pp. 132–133, §M.9 · p. 133

M.10 PRIVATE PERSON DEFAMATION

  • Private-person defamation results vary across languages and names, but most generations are refused or disclaimed; the nonexistent-lawsuit assumption is rejected almost universally except in the tested low-resource language.The authors report improvement over commonly used open-weight models and did not block release.Source: §M.10 · p. 133

N COVERAGE OF COMMON AI SECURITY FRAMEWORK RISKS

  • Table N.1 maps NIST AI RMF and MLCommons Hazards risks to evaluation coverage, distinguishing fully, partially, and unevaluated risks and marking benchmark names in bold.The table also identifies risks marked not applicable.Source: Table N.1 · p. 134

Related papers

Loading 2610.15542v1…