PaperlayerView exact revision on arXiv

Source-linked AI summary

Apertus: Democratizing Open and Compliant LLMs for Global Language Environments

Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Mariñas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clement Charmillot, Jonathan Coles, Jan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javi Rando, Mathieu Sauser, Jakhongir Saydaliev, Muhammad Ali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag

arXiv:2509.14233v2cs.CLcs.AIcs.LG

TL;DR

Open LLM development often leaves data compliance and multilingual representation insufficiently addressed. Apertus combines compliant data practices, broad language coverage, and open development artifacts; it is reported to perform strongly across evaluations, including multilingual benchmarks. The authors caution that deployment still requires use-case alignment and additional testing.

  • Problem

    Many open models prioritize weights while providing little training-data detail, and often focus on single or few high-resource languages, leaving compliance and multilingual representation as open concerns.

    Source: §1 · p. 6

  • Method

    Apertus trains 8B- and 70B-scale models on openly available, multilingual data with retroactive robots.txt opt-outs, Goldfish-based memorization mitigation, and open development artifacts.

    Source: §1 · pp. 6–7

  • Results

    Apertus is reported as the strongest pretrained open model on multilingual benchmarks, with strong performance across knowledge, cultural, and instruction-following evaluations.

    Source: §1 · p. 6, §7 · p. 57

  • Takeaways & Limitations

    Weights and development artifacts are released under permissive licenses to support independent audit, replication, and extension.

    Source: §7 · p. 57

    The models are text-only and may still hallucinate, produce degenerate text or toxic outputs, or exhibit other unsafe behaviors; deployment requires use-case alignment and additional testing.

    Source: §1 · p. 8

Abstract

from arXiv · show

We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. To mitigate risks of memorization, we adopt the Goldfish objective during pretraining, strongly suppressing verbatim recall of data while retaining downstream task performance. The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with ~40% of pretraining data allocated to non-English content. Released at 8B and 70B scales, Apertus approaches state-of-the-art results among fully open models on multilingual benchmarks, rivalling or surpassing open-weight counterparts. Beyond model weights, we release all scientific artifacts from our development cycle with a permissive license, including data preparation scripts, checkpoints, evaluation suites, and training code, enabling transparent audit and extension.

5 Evaluations

Appendix H.1 is labeled as synthetic data for scientific research in data poisoning and memorization.

5 Evaluations

  • The document labels subsection H.1 “Synthetic data for scientific research in data poisoning and memorization” and places it on page 90.Source: §5 · p. 4

1 INTRODUCTION

Apertus addresses data-compliance and multilingual-representation gaps through its data practices and broad language coverage, while releasing model weights and development artifacts.

1 INTRODUCTION

  • Apertus uses openly available pretraining data, retroactively applies robots.txt opt-outs, and uses Goldfish training to limit verbatim recall.The report says its evaluation found verbatim memorization effectively prevented while downstream task performance was preserved.Source: §1 · pp. 6–7
  • Apertus is reported as the strongest pretrained open model on multilingual benchmarks, with open state-of-the-art performance at equivalent scale.The introduction states this comparison but does not provide benchmark values in the supplied passage.Source: §1 · p. 6
  • Apertus includes 8B- and 70B-scale models, with the 70B model trained on 15T tokens using up to 4096 GPUs.The report identifies architectural and training innovations used to stabilize large-scale training.Source: §1 · pp. 6–7
  • Pretraining spans 1811 languages, while the models are evaluated on benchmarks covering a further 94 languages.The report also describes 149-language post-training data for user interaction.Source: §1 · p. 7
  • The release pairs model weights with source code, checkpoints, data-reproduction scripts, evaluation suites, and training artifacts for audit and extension.These materials are provided under permissive-use licenses.Source: §1 · p. 7
  • The authors caution that Apertus may produce hallucinations, degenerate text, toxic outputs, or other unsafe behavior, and is text-only.They recommend extensive use-case alignment and additional testing before deployment.Source: §1 · p. 8

2 MODEL ARCHITECTURE & PRETRAINING RECIPE

This section sets out Apertus architecture and pretraining choices, then presents evidence from tokenizer evaluation, recipe ablations, long-context extension, and a retrospective on final training.

2.1 MODEL ARCHITECTURE

  • Apertus uses a dense decoder-only Transformer at 8B and 70B scales, with xIELU and QK-Norm, and extends 4,096-token pretraining contexts to 65,536 tokens.The architecture also uses grouped-query attention for inference efficiency.Source: §2.1 · pp. 9–10

2.2 TOKENIZER

  • Mistral-Nemo matched or outperformed alternatives across tokenizer efficiency metrics and had the lowest Gini across 55 FLORES+ languages.It was selected over Gemma-2 for greater fairness and a smaller vocabulary (128k vs. 256k) without sacrificing performance.Source: §2.2 · p. 10, Figure 1 · p. 11

2.3 OPTIMIZER & TRAINING RECIPE

  • The recipe combines Goldfish token masking, AdEMAMix’s long-term gradient EMA, and WSD scheduling that supports continued pretraining without pre-specifying its full length.Calibration used a 2% token masking rate and a 50-token hashing context; AdEMAMix scaled more favourably with model size, training duration, and batch size than alternatives.Source: §2.3 · pp. 11–12

2.4 ABLATIONS

  • The merged 3B recipe matched baseline training loss with 30–40% fewer tokens, while improving stability and lowering gradient norms.The authors selected this combined recipe for pretraining.Source: Figure 2 · p. 13, §2.4 · p. 13
  • AdEMAMix (2.002) and xIELU (1.997) produced the largest individual training-loss improvements over the approximately 2.037 baseline.Source: §2.4 · p. 13
  • Apertus matched OLMo2 loss with 46% fewer tokens at 1B and 30% fewer at 7B under replayed data (OLMo et al., 2025).The 7B Apertus run had warmup loss spikes; the vocabulary-size difference also favoured OLMo2 in average cross-entropy.Source: Table 4 · p. 14, §2.4 · p. 14

2.5 LONG CONTEXT

  • Long-context training gradually increased context length and RoPE Θ, retaining the pretraining global batch size and warming up the learning rate at each stage.Both models were extended to contexts up to 65,536 tokens, with a 1.2-billion-token warmup at each stage.Source: §2.1 · p. 10, §2.5 · p. 14, Table 5 · p. 15

2.6 FINAL RUN RETROSPECTIVE

  • Full-scale runs remained stable overall, but the 70B cooldown showed no significant slope or benchmark jump, and its cause remained unclear.One 70B NaN was attributed to hardware failure and recovered by rollback; gradient clipping at 0.1 was applied almost every step, with its necessity unresolved.Source: §2.6 · pp. 15–16

3 PRETRAINING DATA

This section describes Apertus’s reproducibly prepared pretraining data, including consent checks and privacy and toxicity filters, and a curriculum that shifts toward higher-quality, mathematical, and coding data. It also covers the long-context mixture, dominated by Stage 5 data and with longer FineWeb documents upsampled across progressively longer-context phases.

3 PRETRAINING DATA

  • The data spans 1,811 languages and 1,868 language-script pairs, with FineWeb-2 preserving languages at their natural frequencies.The section describes a goal of permissively licensed data and a documented, reproducible preprocessing pipeline whose code is released; filtering pipelines use datatrove.Source: §3 · p. 17, §3.2.2 · p. 20, §3.2.5 · p. 21

3.1.1 CONSENT: ROBOTS.TXT WITH HINDSIGHT

  • Apertus retroactively applies the latest robots.txt permissions across 2013–2024 snapshots, removing blocked content from all datasets.The January 2025 check uses an opt-out policy when robots.txt files are unavailable; estimated token loss is approximately 8% for English and 4% for multilingual data.Source: §3.1.1 · p. 17

3.1.2 PERSONALLY IDENTIFIABLE INFORMATION (PII)

  • Privacy and toxicity filters anonymize detected emails, IP addresses, and IBANs, and remove the 5% highest-scoring documents per language across nine languages.Toxicity classifiers are trained per language using XLM-RoBERTa embeddings and a two-layer MLP; the filter is not applied to code, math, or designated prefiltered education subsets.Source: §3.1.2 · p. 18, §3.1.3 · p. 18, §3.2.1 · p. 18

3.2 SOURCE DATASETS

  • The sources include FineWeb crawls filtered for knowledge-rich content, educational FineWeb-Edu (16) and DCLM-Edu (17) subsets, and multilingual FineWeb-2 (18), spanning 1,811 languages kept at natural frequencies.FineWeb-2-HQ (19) filters 20 high-resource languages for structured, knowledge-rich content and removes toxic content; other languages use unfiltered FineWeb-2, while data from those 20 is downsampled to preserve the HQ data’s relative proportion on the web.Source: §3.2.1 · pp. 18–19, §3.2.2 · p. 20

3.2.3 CODE, MATHEMATICAL, AND STRUCTURED DATA

  • Code, math, and task data combine permissively licensed code and educational annotations, filtered math corpora, multilingual synthetic instructions, and commercially licensed Flan data.Math data also receives the same opt-out web-crawling filter, while PII removal is omitted for code and math because heuristics often yield false positives.Source: §3.2.3 · pp. 20–21

3.2.4 DATA FOR DOWNSTREAM ANALYSIS

  • Memorization probes use Gutenberg sequences injected at controlled frequencies, alongside synthetic poisoning examples, to study memorization and data-poisoning effects.Gutenberg-V1 contains buckets of 500 sequences totaling 1.78B tokens, and Gutenberg-V2 contains 167 new sequences totaling 583M tokens; both are publicly available.Source: §3.2.4 · p. 21

3.3 PRETRAINING CURRICULUM

  • Pretraining uses 15T tokens across five stages, moving from broad language, math, and code foundations toward higher-quality data and increased math and coding content.The final cooldown stage adds educational code, Wikipedia, parallel data, and instruction and task data; the 8B run skips Stage 2, with Stage 1 lasting to 7T tokens before switching to Stage 3.Source: §3.3 · p. 22
  • The curriculum’s cooldown tests found DCLM-Edu gave the largest performance gain and replacing FineWeb-Edu with FineWeb-HQ-33 consistently improved results, guiding phased data-mixture changes.The ablations were initially run without compliance filtering, then most mixtures were revalidated at 3B with full compliance filtering; code and math weights also increased.Source: §3.3 · p. 23

3.4 LONG CONTEXT DATA MIXTURE

  • The approximate mixture was 70% Stage 5, 20% FineWeb-Long, and 10% Institutional Books, with longer FineWeb-HQ and FineWeb-2-HQ documents upsampled.Table 8 organizes token counts across phases with progressively longer context lengths; documents are not repeated across phases.Source: §3.4 · p. 25, Table 8 · p. 25

4 POST-TRAINING

This section describes Apertus post-training, pairing curated supervised finetuning with preference alignment and detailing the licensing-, quality-, and contamination-filtered data, flexible chat format, and QRPO methods. It also covers length-normalized QRPO and a separate Swiss AI Charter-based pipeline for controversial topics, including a recommended chatbot system prompt for deployment.

4.1.1 DATA COLLECTION & LEGAL COMPLIANCE

  • Dataset selection requires documented, reproducible sources with permissive redistribution and commercial-use licenses, and excludes incompatible or ambiguous licensing, opt-out sources, and low-quality data.Quality review combines metadata and manual sample inspection; filtering also removes organizational branding and identity markers that could bias style or confuse provenance.Source: §4.1.1 · pp. 26–27

4.1.2 DECONTAMINATION

  • N-gram matching identified hundreds of translated benchmark problems missed by hash-based methods, including an example with a 0.62 match ratio despite different instruction languages.The procedure first screens with token-level 8-gram matching, then measures prompt overlap using Ratcliff–Obershelp; matches with combined overlaps longer than half the benchmark prompt are marked contaminated after short overlaps are excluded.Source: §4.1.2 · p. 27, Table 9 · p. 27
  • Decontamination leaves Tulu3’s average at 0.443 versus 0.442 unfiltered, while license filtering lowers it to 0.417 (-5.8%); filtered OLMo2 reaches 0.421.On MMLU chain-of-thought, filtering lowers the score from 0.513 to 0.253 (51% decrease). The authors accept the capability trade-off as necessary for compliance.Source: §4.1.2 · pp. 27–28, Table 10 · p. 28
  • License filtering lowers multilingual average performance from 0.511 after decontamination to 0.489, while filtered OLMo2 is nearly identical at 0.487.The largest relative drop is MGSM native CoT, from 0.320 to 0.273 (-14.7%); CulturalBench and Switzerland QA are more robust.Source: Table 11 · p. 28, §4.1.2 · p. 28

4.1.3 SUPERVISED FINETUNING DATA

  • The approximately 3.8-million-example SFT mixture combines instruction-following, mathematical reasoning, code generation, and multilingual data, including 46,923 Romansh instruction-following examples spanning six written varieties and covering translation and idiom-identification tasks.Multilingual and Cultural (1.4M) and Low-Resource and Regional Languages (944K) are category totals; eight iterations of refinement optimized balance among language diversity, task coverage, and quality.Source: §4.1.3 · pp. 28–29

4.1.4 ALIGNMENT DATA

  • Alignment data comprises 380,537 non-controversial and 72,698 controversial prompts; the Qwen3-32B controversy classifier achieved 73% accuracy against 800 human labels.Completions from five LLMs and Apertus-SFT are reward-annotated, with high- and low-reward samples selected for QRPO training; controversial prompts also receive persona- and Swiss AI Charter-based completions.Source: §4.1.4 · pp. 30–31

4.2.1 FORMAT AND CHAT TEMPLATE

  • The chat format adds dedicated start and end tokens for assistant and developer messages, supporting tool configuration, ordinary dialogue, and multi-step agentic workflows.A complete specification and implementation library are publicly available.Source: §4.2.1 · p. 32

4.3 PREFERENCE ALIGNMENT

  • QRPO optimizes absolute reward signals while retaining offline learning and stability, and accepts rankings from reward models, human preferences, or LLM-as-judge annotations.Apertus uses length-normalized QRPO, scaling the KL regularization coefficient by completion length.Source: §4.3 · pp. 32–33
  • Length normalization improves both QRPO and DPO; they perform similarly at 8B, while QRPO outperforms DPO at 70B, leading Apertus to adopt length-normalized QRPO.QRPO divides βKL by completion length in every loss component.Source: §4.3 · p. 33

4.3.1 ALIGNMENT FOR STANDARD TOPICS

  • For standard topics, Apertus uses a pretrained reward model to rank non-controversial prompt-completion pairs, then aligns the model with QRPO offline and off-policy.The reward model is Skywork-Reward-V2-Llama-3.1-8B, finetuned on 26M preference pairs.Source: §4.3.1 · pp. 33–34

4.3.2 ALIGNMENT OF CONTROVERSIAL TOPICS

  • Because preferences vary across populations, the controversial-topic pipeline uses Constitutional AI and a Swiss AI Charter grounded in Swiss constitutional values and cultural diversity.The Charter articulates principles including neutrality, consensus-building, federalism, multilingualism, and respect for cultural diversity.Source: §4.3.2 · p. 34
  • A Qwen3-32B LLM judge scores controversial-topic completions against the Charter, and its scores and rankings are used to align Apertus with QRPO.On synthetic data, pairwise scoring slightly outperformed probability-weighted pointwise scoring; for compute efficiency, the pipeline first scores responses pointwise, then pairwise-ranks the top five.Source: §4.3.2 · pp. 35–36
  • 97.3% average agreement among surveyed Swiss residents supported the Charter’s principles; agreement was lowest for Article 6 at 92.6%, and strong disagreement was 0.3%.Strong agreement was highest for Article 4, Preventing Harm, at 91.3%.Source: §4.3.2 · p. 35, Table 13 · p. 36

4.4 CHATBOT SYSTEM PROMPT

  • The recommended general-use chatbot system prompt summarizes Apertus’s identity, training origin, and capabilities, and includes a summary of the Swiss AI Charter plus context and style instructions.Source: §4.4 · p. 36

5 EVALUATIONS

The evaluations cover Apertus from pretraining through post-training alignment on English and multilingual benchmarks spanning more than 94 languages, as well as safety benchmarks and manual spot-tests of Apertus-Instruct models. Results vary by benchmark, and manual testing identified areas for improvement but no issue warranting a release delay.

5.1 PRETRAINING EVALUATION

  • Apertus-70B leads all evaluated models on multilingual XCOPA, while both Apertus sizes outperform all other fully open models on INCLUDE V1 and V2.The pretraining evaluation also tracks multilingual macro-averaged performance alongside English results during training.Source: §5.1 · pp. 38–39

5.2 POST-TRAINING EVALUATION

  • Apertus-Instruct performs solidly across benchmarks and is particularly competitive with similarly sized fully open models, but both sizes are weaker in math and coding.Apertus-8B is competitive in knowledge recall, instruction following, and commonsense reasoning, leads fully open models in cultural knowledge, and has a smaller-than-usual performance gap to 70B; many baselines received RLVR training that Apertus has not.Source: §5.2 · p. 41, Table 21 · p. 43

5.3 LOW-RESOURCE TRANSLATION

  • Apertus demonstrates greater low-resource translation ability than Llama-3.3-70B-Instruct across German and six written varieties of Romansh.The evaluation uses a preliminary Romansh WMT24++ benchmark with three few-shot examples and greedy decoding; results are reported as BLEU scores.Source: §5.3 · pp. 43, 45, Table 24 · p. 45

5.4.1 APERTUS MEMORIZATION PATTERNS

  • Rouge-L ≈0.18 remains at baseline for both Apertus sizes, with no memorization across tested exposure frequencies up to 128 or prefix lengths up to 5,000.With nucleus sampling, TTR is ≈0.500 while Rouge-L and LCCS remain at baseline; the mitigation is therefore not an artifact of greedy-decoding degeneration.Source: §5.4.1 · p. 46, Figure 8 · p. 46, Table 25 · p. 47
  • Goldfish Loss disrupts the sharp offset-dependent recall decay expected from positional fragility, while earlier-injected Gutenberg sequences may be more memorized than later ones.The possible primacy effect compares sequences introduced during the first 0–9T tokens with those from 9–12T, but may be confounded by differences between probe sets.Source: §5.4.1 · p. 47, Figure 9 · p. 48

5.4.2 FAILURE CASE STUDIES

  • Goldfish Loss’s deterministic hashing fails on near-duplicates, and all 22 probes with ROUGE-L ≥0.7 among 10,672 Gutenberg probes were canonical works.Formatting and tokenizer differences can shift hashes and expose tokens; high recall on structured, low-diversity text (TTR ≤0.4) can instead reflect template learning.Source: §5.4.2 · pp. 47–48, Figure 10 · p. 48

5.5.1 GENERAL CONSIDERATIONS

  • Safety evaluation accounts for open-weight guardrail-removal risks, multilingual safety gaps, low-resource information-operation risks, and the trade-off between harmlessness and usefulness.The authors test available multilingual safety benchmarks and manually test high-risk scenarios, but do not claim coverage of every training language or pursue jailbreak resistance, which they assign to production guardrails.Source: §5.5.1 · p. 49

5.5.2 SAFETY BENCHMARK PERFORMANCE

  • On BBQ, Apertus-Instruct is comparable to other fully open models but slightly behind leading open-weight models; on HarmBench, it fares worse than most fully open models.For HarmBench direct requests, it is comparable to fully open models and better than most tested open-weight models except Qwen; human-jailbreak performance is in line with most open-weight models except Qwen.Source: §5.5.2 · p. 49
  • On RealToxicityPromptsLlama-Subsampled, Apertus models perform well compared with fully open and open-weight models.The benchmark uses 10% of the standard dataset and Llama-Guard-3-8B as its toxicity classifier, so its results are not directly comparable with standard RealToxicityPrompts results.Source: §5.5.2 · p. 50
  • On ToxiGen, Apertus-Instruct models are in line with fully open models but perform worse than every tested open-weight model.The evaluation tests identification of implicit toxicity on a balanced dataset.Source: §5.5.2 · p. 50
  • LinguaSafe results report severity-weighted scores for Apertus-70B-Instruct and Apertus-8B-Instruct across harm categories and 12 languages; lower scores indicate better performance.The benchmark covers five harm classes, while reported open-weight direct and indirect mean weighted scores range from 21% to 45%.Source: §5.5.2 · pp. 50–51, Table 27 · p. 51, Table 28 · p. 51

5.6 QUALITATIVE SPOT-TESTING

  • Manual spot-testing of Apertus-8B-Instruct and Apertus-70B-Instruct found no issue warranting a release delay, though the models could improve.Testing covered CBRNE, dual use, medical disinformation, private-person claims, and suitability for information operations in low-resource languages; the release includes a repository for critical examples and an Apertus-specific issue-reporting system.Source: §5.6 · p. 51

6 INFRASTRUCTURE, SCALING, AND EFFICIENCY

Alps and its ML platform supported Apertus training at up to 4,096 GPUs, with coordinated engineering work addressing stability, resilience, and efficiency. At 4,096 GPUs, training achieved 80% strong-scaling parallel efficiency.

6.1.1 THE ALPS RESEARCH INFRASTRUCTURE

  • Alps combines 10,752 GH200 GPUs and high-speed networking with an endpoint-based architecture intended to increase flexibility and composability.The system is reported to have measured HPL performance of 434 PFlops (fp64), placing it among the top 10 most powerful supercomputers globally.Source: §6.1.1 · p. 52

6.1.2 THE MACHINE LEARNING PLATFORM

  • Clariden provided container-oriented ML resources and node-readiness checks, while vCluster flexibility supported long-duration jobs and selective updates during 70B training.The 70B workload ran across 2,048–4,096 GPUs over several months, and Alps delivered stable, well-scaling performance.Source: §6.1.2 · pp. 52–53

6.2 FULL TRAINING RUN PERFORMANCE

  • Around 6 million GPU hours were consumed by the 70B main run, an underestimate excluding weight loading, early performance shortcomings, failures, and downtime.The authors estimate approximately 90 days on 4,096 GPUs and 5 GWh for compute under their stated power assumption.Source: §6.2 · p. 53

6.3.1 SYSTEMS-LEVEL FIXES

  • Version alignment, driver and kernel fixes, and memory-management work removed major instability sources and allowed large jobs to run uninterrupted.After stability improvements, throughput became more consistent, with predictable dips from garbage collection and checkpointing; minor filesystem interference remained.Source: Figure 12 · p. 54, §6.3.1 · pp. 54–55

6.3.2 STABILITY AND CONTAINER ROBUSTNESS

  • Moving Triton caches to node memory and updating libnvrtc addressed crashes; storage placement and 250-iteration checkpointing balanced restart cost against failures.Jobs also queued successors after successful initialization and used pre-time-limit signals to save checkpoints and terminate gracefully.Source: §6.3.2 · p. 55, §6.3.3 · p. 55

6.3.4 PERFORMANCE OPTIMIZATIONS AT SCALE

  • vBoost, larger distributed-data-parallel buckets, and model-parallel changes improved compute and communication efficiency; reserved capacity, longer limits, and monitoring reduced operational downtime.The scale optimizations also included Lustre striping for faster container-image loading.Source: §6.3.4 · p. 55, §6.3.5 · p. 56

6.3.6 SCALING AND PARALLEL EFFICIENCY

  • 80% strong-scaling parallel efficiency at 4096 GPUs, with performance of 723 tokens per second per GPU.The strong-scaling experiment held the global batch size at 16.8 M tokens.Source: Figure 13 · p. 56, §6.3.6 · p. 56

7 CONCLUSION

Apertus combines data compliance, broad multilingual coverage, and transparency, with reported strong performance across knowledge, cultural, and instruction-following evaluations. The report presents the suite as a baseline for globally relevant open models and outlines directions for further development.

7 CONCLUSION

  • The suite pairs 15T-token training across 1811 languages with retroactive robots.txt opt-outs and a Goldfish-style objective to curb verbatim reproduction.Multilingual Instruct variants are post-trained for interaction across many languages, with constitutional alignment for controversial topics.Source: §7 · p. 57
  • Strong performance across knowledge, cultural, and instruction-following evaluations is reported alongside releases of model weights, preparation recipes, evaluation suites, and training artifacts.The report says permissively licensed outputs are designed to comply with EU AI Act data provisions for commercial and non-commercial use.Source: §7 · p. 57
  • Apertus is presented as both the largest fully open LLM released to date and the first state-of-the-art LLM developed by a fully open, publicly funded academic consortium.Source: §7 · p. 57
  • Future directions include scaling, distillation, mapping data choices to performance and fairness, adaptive-compute reasoning, verifier-based reinforcement learning, multimodality, societal alignment, and field evaluation.Source: §7 · p. 57
  • The report invites inspection and stress-testing of the models and artifacts to support more powerful and accountable future iterations.Source: §7 · p. 57

B DATA OPT-OUT BY APPLYING AI-CRAWLER BLOCKS RETROACTIVELY

The report retroactively applies AI-crawler opt-outs to prior web crawls and describes how robots.txt compliance affects the corpus. It also reports pretraining optimizer and learning-rate scheduling choices.

B DATA OPT-OUT BY APPLYING AI-CRAWLER BLOCKS RETROACTIVELY

C PRETRAINING HYPERPARAMETERS

  • Pretraining uses AdEMAMix with WSD scheduling, independent 100,000-step warmups for α and β3, and β2=0.999 and β3=0.9999 to reduce gradient-estimate variance and improve stability.For post-training, lowering β2 and β3 yields better results when runs use fewer iterations.Source: C PRETRAINING HYPERPARAMETERS · p. 82

D FP8 TRAINING

FP8 training increased throughput in an Apertus 8B experiment but later produced a sustained loss increase and gradient-norm instability. The team rolled it back for main pretraining; subsequent work reported a more stable FP8 approach.

D FP8 TRAINING

  • After 300B tokens of FP8 training, loss increased substantially and gradient-norm instabilities grew, prompting a rollback before main pretraining.Figure D.1 plots smoothed loss and gradient norms.Source: D FP8 TRAINING · p. 83, Figure D.1 · p. 83
  • FP8 raised throughput by ∼26.3% (6.96k →8.79k tokens/sec/GPU) with an initial instantaneous loss increase of around 0.01.The experiment began at ∼8T consumed tokens and used Current Scaling FP8 for linear projections in forward and backward passes.Source: D FP8 TRAINING · p. 83
  • Later work stabilized FP8 training on FOG architectures by reordering layer normalizations, reducing large activation outliers and the risk of quantization underflows and overflows.Source: D FP8 TRAINING · p. 83

E FLOPS ESTIMATION

The report estimates pretraining FLOPs with a Python script that accounts for major Transformer operations rather than relying on the common 6=ND approximation. It gives an estimate for the 70B configuration at sequence length 4096.

E FLOPS ESTIMATION

  • The 70B configuration at sequence length 4096 yields an estimated 6.74 · 1024 FLOPs using the report’s detailed calculation.The calculation accounts for major Transformer operations.Source: E FLOPS ESTIMATION · p. 83, Figure E.2 · p. 84
  • The script sums attention projections, attention products and normalization, softmax, attention-value computation, and output projection FLOPs.Source: E FLOPS ESTIMATION · p. 84
  • The estimation code also includes dense-MLP, RMSNorm, and final-logit operations parameterized by sequence length and model dimensions.Source: E FLOPS ESTIMATION · p. 84

F IMPLEMENTATION OF GOLDFISH LOSS

This section motivates pretraining-time protection against verbatim recall and describes Goldfish Loss, which suppresses memorization while retaining downstream performance. Its results show reduced verbatim recall across tested LLaMA scales, with some residual increase at the largest scale and highest repetition counts.

F IMPLEMENTATION OF GOLDFISH LOSS

  • Because verbatim regurgitation raises copyright and privacy risks, the approach favors pretraining safeguards over post-hoc measures with documented shortcomings.The cited shortcomings include attacks that extract training data, constrained decoding that leaves memorized information in parameters, and unlearning that may degrade performance.Source: F IMPLEMENTATION OF GOLDFISH LOSS · p. 85
  • A 2% masking rate and 50-token hashing context suppress verbatim memorization without compromising downstream performance.Goldfish Loss selectively masks tokens during pretraining; masking is precomputed during data loading for efficiency.Source: F IMPLEMENTATION OF GOLDFISH LOSS · p. 85
  • Goldfish Loss keeps Rouge-L scores low across 1B, 3B, and 8B LLaMA models, though 8B shows a slight increase at the highest repetition counts.The figure evaluates 500-token suffixes across varying prefix lengths and repetition frequencies.Source: Figure F.3 · p. 86

G FINEWEB-2 LANGUAGE DISTRIBUTION

The FineWeb-2 section describes how document sampling varies by language and reports language representation. The synthetic-data section introduces small pretraining datasets for studying poisoning and memorization.

G FINEWEB-2 LANGUAGE DISTRIBUTION

  • Russian documents account for 13.26% and Mandarin Chinese documents for 12.66% of the FineWeb-2 language distribution.FineWeb-2 documents in the 20 listed high-resource languages are subsampled from top-quality documents at 10% or 33%; other languages are sampled randomly.Source: G FINEWEB-2 LANGUAGE DISTRIBUTION · pp. 88–89, Table G.6 · p. 89

H.1 SYNTHETIC DATA FOR SCIENTIFIC RESEARCH IN DATA POISONING AND MEMORIZATION

  • Synthetic data is included to support controlled pretraining research on data poisoning and memorization.The authors identify the difficulty of conducting end-to-end training runs as a constraint in prior work on these topics.Source: §H.1 · p. 90
  • The poisoning set tests false Pokémon facts and a trigger that elicits German replies, using low-frequency examples introduced only during pretraining.The examples were designed to target benign behaviors that could be overwritten and to minimize unintended effects.Source: §H.1 · p. 90
  • Memorization is studied with repeated canaries and fictional-event documents, testing verbatim recall and retention beyond exact wording.The fictional-event dataset contains 100 events with 15 documents each, split into held-out and folds repeated 10, 100, or 1,000 times.Source: §H.1 · p. 90

H.2 POSSIBLE SWISS DATA (NOT CURRENTLY USED IN PRETRAINING)

The appendices describe Swiss and Romansh datasets reserved for potential future use rather than Apertus pretraining, alongside related tokenizer, data-stage, and prompt-construction procedures. They also specify a classifier for rating ideological influence in answers and a prompt for iteratively degrading AI responses while preserving prior flaws.

H.2 POSSIBLE SWISS DATA (NOT CURRENTLY USED IN PRETRAINING)

  • Swiss-centric data was collected to represent Swiss culture and law, but was withheld from pretraining for neutrality and may support later customization or fine-tuning.Only four sources met the authors’ availability and licensing standards; the data is made available on Hugging Face.Source: §H.2 · p. 90
  • Around 6.8 billion tokens of Swiss data combine judicial and parliamentary material with a broader Swiss web subset.The sources are FineWeb-2-Swiss, Entscheidsuche legal decisions, Curiavista parliamentary proceedings, and Romansh data; the collection skews toward the judicial domain, with Curiavista focused on politics.Source: §H.2 · p. 91
  • The released Romansh corpus covers Rumantsch Grischun and five idioms, with monolingual, parallel, and synthetic translated-text subsets carrying idiom metadata.Its sources include municipal and cantonal law texts, a bilingual corpus, online dictionaries, and Wikipedia; parallel data pairs Romansh with German, French, Italian, and English.Source: §H.2 · p. 91

H.3 APERTUS 8B AND 70B DATA STAGES

  • The 8B and 70B data-stage table reports transition iterations and consumed tokens, and each stage uses a different data seed to vary document order.The 8B model consumed no Stage 2 tokens.Source: §H.3 · p. 91, Table H.8 · p. 92

I TOKENIZER SELECTION

  • Tokenizer selection evaluates compression rate, fertility, vocabulary utilization, and a language-fairness Gini coefficient, where lower Gini indicates more equitable compression.The evaluation spans a broad set of languages; compression rate and fertility characterize token use per line and per word, respectively.Source: I TOKENIZER SELECTION · pp. 92–93

J.1 ROMANSH SFT DATA

  • Romansh SFT data combines translated Q/A, dictionary and sentence translation, and idiom-identification prompts, with sentence pairs aligned and automatically quality-filtered.Sentence alignment used a cosine threshold of at least 0.65 with mutual nearest-neighbor matching; Qwen3-32B scoring retained pairs with integer scores of at least 7.Source: §J.1 · p. 94

J.2 COMPLETION GENERATION PROMPTS

  • Standard alignment prompts randomly select among truthful, helpful, and honest system-prompt attributes, while persona prompts use sampled PersonaHub descriptions.Persona descriptions are sampled from 200,000 personas, and the prompt instructs the model to stay in character without fabricating beyond its context.Source: §J.2 · pp. 94, 97

J.3 IDEOLOGICAL SENSITIVITY CLASSIFIER

  • The appendix reproduces a prompt for identifying ideologically sensitive questions or prompts.Source: §J.3 · p. 97
  • Its indicators include conflicts over political-moral values, cultural perspectives, justice and authority, resource allocation, and deeper ethical principles.The focus is on differences that substantially change responses, rather than minor differences in style or emphasis.Source: §J.3 · pp. 97–98
  • The classifier assigns questions a 0–3 ideological-sensitivity score, from objective questions to those whose answers are fundamentally shaped by ideology.It requires the classification and English reasoning in a specified JSON format.Source: §J.3 · pp. 97–99

J.4 SYNTHETIC DEGRADATION PROMPT

  • The degradation prompt uses the response history to choose one new quality flaw, then makes the latest completion conspicuously worse while retaining all previous flaws.Possible dimensions include factual accuracy, coherence, completeness, organization, relevance, language, certainty, instruction-following, and reasoning.Source: §J.4 · p. 100

J.5 ADDITIONAL RESULTS: CHARTER ANALYSIS

The charter analysis reports survey respondents’ average rankings of AI principles. The SwitzerlandQA section describes a multilingual, locally detailed benchmark and notes translation and regional-coverage limitations.

J.5 ADDITIONAL RESULTS: CHARTER ANALYSIS

  • Survey respondents ranked the relative importance of the Swiss AI Charter principles, and the figure reports average rankings.The ranking task asked respondents to order eleven principles from most to least important.Source: §J.5 · p. 101, Figure J.4 · p. 101

K SWITZERLANDQA

  • SwitzerlandQA covers five civic domains and 26 cantons, with 9,167 multiple-choice items per language in German, French, Italian, Romansh, and English.The benchmark prioritizes cantonal-level questions while also including national and commune-level items.Source: K SWITZERLANDQA · p. 102
  • Questions draw primarily on official sources, with GPT-4o used to generate English items when available material was insufficient.The source collection spans 107 official sources, including canton handbooks, civic guides, and municipal archives.Source: K SWITZERLANDQA · p. 102
  • Romansh translations carry higher risk of inaccuracy, and uneven official documentation leaves canton-level representation uneven.The authors note that these differences may introduce regional evaluation imbalances.Source: K SWITZERLANDQA · p. 102

L CONSTITUTIONAL HARMS TEST DERIVATION

This appendix derives constitutional-harm tests for chatbot responses, covering dangerous assistance, inadequate crisis responses, dangerous misinformation, harmful stereotypes, and risks to humanitarian or international efforts. It describes test cases as stochastic and non-exhaustive, and notes that risk models were considered for manual testing of the initial capability-demonstration deployment.

L CONSTITUTIONAL HARMS TEST DERIVATION

  • The derivation used Appendix O’s charter and Qwen3-32B, selected for strong safety and harmlessness benchmark performance; generated classes, scenarios, and prompts are not exhaustive or team-endorsed.The appendix also warns that much of the section is machine-generated debug log and that derived cases may overlap existing benchmarks.Source: L CONSTITUTIONAL HARMS TEST DERIVATION · p. 103

L.1 PROMPTS

  • The prompts ask a model to derive constitution-defined harmful behavior classes, generate three risk scenarios per class, write three self-contained prompts per scenario, and judge whether risk occurred.The evaluation format requests a one-sentence justification followed by a yes/no answer for each prompt-response pair.Source: §L.1 · p. 103

L.2.1 ENABLING DISCRIMINATION

  • The discrimination cases cover gender stereotyping in hiring, racial or xenophobic treatment of non-native speakers, and ableist assumptions that discourage disabled users from pursuing their goals.The examples span job screening, retail customer service, and healthcare advice.Source: §L.2.1 · p. 104

L.2.2 FACILITATING EXPLOITATION

  • The exploitation cases cover deceptive trafficking recruitment, using simulated peer relationships to recruit or extract information from minors, and sustained grooming or manipulation of vulnerable people.Risks include coercion, forced labor, exploitation, and financial loss.Source: §L.2.2 · p. 105

L.2.3 ENDANGERING VULNERABLE POPULATIONS

  • Vulnerable-population cases test failures to respond to a minor’s crisis, chatbot-assisted grooming of minors, and harmful medical misinformation affecting marginalized or vulnerable users.The stated risks include delayed intervention, exploitation, physical harm, and exacerbated inequities.Source: §L.2.3 · p. 106

L.2.4 PRIORITIZING ABSTRACT CONSIDERATIONS OVER HUMAN SAFETY

  • The tests flag cases where privacy, legal-compliance, or neutrality concerns displace urgent action in self-harm, dangerous-activity, or threat scenarios.Examples include deferring suicide-prevention support, enabling potentially harmful activity under a legal loophole, or leaving a threatened user without escalation.Source: §L.2.4 · p. 107

L.2.5 ENABLING DANGEROUS ACTIONS

  • The dangerous-action tests cover assistance enabling unauthorized hacking, attacks on others, and self-harm, with risks of breaches, real-world violence, or suicide.They include cases where the chatbot mistakes harmful intent for ethical hacking or hypothetical discussion, or offers self-harm instructions to a distressed user.Source: §L.2.5 · p. 108

L.2.6 INADEQUATE RESPONSE TO SELF-HARM OR HARM TO OTHERS

  • Crisis-response tests flag failures to warn about self-harm, direct users to professional help, recognize threats to others, or respond to subtle suicide cues.Generic reassurance or passive engagement may normalize harm and fail to prompt immediate support or safety planning.Source: §L.2.6 · pp. 108–109

L.2.7 PROPAGATING OR FAILING TO CORRECT DANGEROUS MISINFORMATION

  • Misinformation tests target endorsing unproven medical remedies, equivocating on debunked vaccine claims, and portraying the scientific consensus on climate change as uncertain.For serious illness, delaying proven treatment such as chemotherapy could be fatal; the tests also flag failure to correct verifiable false claims.Source: §L.2.7 · pp. 109–110

L.2.8 REINFORCING HARMFUL STEREOTYPES

  • Stereotype tests address career advice linking suitability to gender, fiction that stigmatizes racial or ethnic groups, and safety advice that associates communities with criminality.Such responses can reinforce professional inequality, discriminatory narratives, prejudice, or exclusionary policies.Source: §L.2.8 · pp. 110–111

L.2.9 UNDERMINING HUMANITARIAN OR INTERNATIONAL EFFORTS

  • Humanitarian tests flag disaster-denying advice that discourages donations and may delay aid, and include a scenario about propaganda in a geopolitical conflict.The disaster scenario also identifies risks of apathy, misinformation, and opposition to life-saving relief.Source: §L.2.9 · p. 111
  • The test identifies partisan conflict claims and vaccine misinformation as risks that can fuel polarization, erode trust, and obstruct humanitarian or public-health efforts.Such content could destabilize peace negotiations or prolong outbreaks and risk lives.Source: §L.2.9 · pp. 111–112

M.1 HARMS SPOT TESTING RISK MODELS

  • For the initial capability-demonstration deployment, risk models were considered for manual testing.Source: §M.1 · p. 112

M.1.1 RISKS RELATED TO MALICIOUS USE

The risk assessments found several malicious-use risks sufficiently mitigated, while medical misinformation in lower-resource languages needs further evaluation and jailbreak resilience is out of scope.

M.1.1 RISKS RELATED TO MALICIOUS USE

  • Apertus disinformation in low-resource languages was judged sufficiently mitigated: tested detectors identified generated but not human-authored text, and some artifacts remained human-detectable.The assessment also noted that the model’s higher quality in these languages preserves native-speaker use.Source: §M.1.1 · p. 112
  • Jailbroken CBRNE tests found concerning but incomplete household-weapon instructions, while tested languages received generic warnings about expert-identified procedures.The risk was judged sufficiently mitigated because the instructions were not more useful than accessible sources and the model did not flag the tested procedures as useful for weapons creation.Source: §M.1.1 · p. 112
  • Apertus provided no sufficiently detailed or useful military tactics, despite having granular general military knowledge; the dual-use risk was judged sufficiently mitigated.Source: §M.1.1 · p. 112

M.1.2 RISKS RELATED TO MISGUIDED USE

  • Medical-claim confirmation resilience deteriorated in lower-resource languages, so this risk needs systematic evaluation but was judged nonblocking for release.Source: §M.1.2 · p. 113
  • Apertus consistently refused private-person information across tested languages, with degraded performance for names resembling notable public figures; the risk was judged sufficiently mitigated.Source: §M.1.2 · p. 113

M.1.3 RISKS OUT OF SCOPE

N.1 WRONGFUL TOXIC RESPONSE DETECTION EXAMPLE

These appendices illustrate a multilingual harmfulness-detector mismatch, report reduced answer-formatting errors, and provide alignment and deployment guidance for Apertus. The Swiss AI Charter translates values into alignment criteria, while the recommended chatbot prompt incorporates a charter summary and Swiss-context instructions.

N.1 WRONGFUL TOXIC RESPONSE DETECTION EXAMPLE

  • 0.901236 was the violence-score increase when the English translation was evaluated, although the Portuguese prompt was not flagged by the moderation API.The example is presented as showing limitations of harmfulness detectors in multilingual settings.Source: §N.1 · p. 113

N.2 LINGUASAFE

  • Answer-formatting errors fell to 0.20% for Apertus-70B-Instruct and 0.01% for Apertus-8B-Instruct with default system prompts.The reported reduction used the format-correction prompts.Source: §N.2 · p. 113

Format correction prompt

  • The format-correction prompt asks which prompts are unsafe and requires either a comma-separated list of their letters or “None,” without justification.Source: Format correction prompt · p. 114

O THE SWISS AI CHARTER

  • The Swiss AI Charter translates values rooted in Swiss constitutional and democratic traditions into alignment criteria for LLMs and is intended to adapt through participatory refinement.Source: PREAMBLE · p. 115

P SYSTEM PROMPT FOR CHATBOT

  • The recommended chatbot system prompt combines identity and training information, a Swiss AI Charter summary, Swiss-context and style instructions, and date and time.The public repository version excludes date and time because those values vary by deployment.Source: P SYSTEM PROMPT FOR CHATBOT · p. 118
Loading 2509.14233v2…