PaperlayerView exact revision on arXiv

Source-linked AI summary

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Mingzhe Zhang, Kaiyue Wen, Kaifeng Lyu, Wenguang Chen

arXiv:2608.27370v1cs.CLcs.LG

TL;DR

Affordable, reproducible language-model pretraining remains difficult because existing open recipes can still require prohibitive compute budgets. This paper presents a hardware-accessible, cost-efficient pretraining recipe and trains Puro-2B models on consumer RTX 5090 GPUs, reaching Qwen2-1.5B performance at about $4.4K while its best checkpoint approaches Qwen2.5-1.5B at about $6.9K.

  • Problem

    Existing open pretraining recipes remain too costly for many academic and resource-constrained researchers, limiting reproducible study of complete training pipelines.

  • Method

    The paper develops a reproducibility recipe spanning system, algorithm, and data design, then trains 2B-parameter models from scratch on consumer-grade RTX 5090 GPUs.

  • Results

    A Puro-2B model reaches Qwen2-1.5B performance at about $4.4K, while the best checkpoint approaches Qwen2.5-1.5B at about $6.9K.

  • Takeaways & Limitations

    The results indicate that useful from-scratch pretraining at the billion-parameter scale is practical with a low-cost, consumer-GPU pipeline.

  • Takeaways & Limitations

    The MuonH and FP8 ablations use TPP=20, whereas production training uses TPP=700, limiting direct extrapolation of those ablation results to production.

Abstract

from arXiv · show

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \$1.5M, and reproducing SmolLM3-3B needs over \$700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \$6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about \$4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.

1 Introduction

The paper presents an affordable, reproducible, open pretraining recipe for Puro-2B that targets the high cost and incomplete transparency of existing language-model training. Its best checkpoint approaches Qwen2.5-1.5B at $6.9K, while the collection exceeds Qwen2-1.5B at about $4.4K, and the authors release the supporting artifacts under Apache 2.0.

  • Motivation: Existing pretraining costs remain prohibitive despite open-weight releases, limiting academic researchers’ ability to reproduce pipelines and study training behavior.The paper argues that reproducibility requires data, software, infrastructure, training details, and transparent cost accounting, not only model weights.
  • Recipe: The recipe trains a collection of 2B-parameter Puro-2B checkpoints from scratch on accessible infrastructure, varying training budgets and recipe variants.The collection is designed for dense billion-parameter/trillion-token pretraining and is evaluated against Qwen2-1.5B and Qwen2.5-1.5B.
  • Open release: The authors release datasets, model weights, intermediate checkpoints, configurations, and implementation to enable reproduction and inspection of the recipe.The releases include 10 checkpoint versions, data manifests and components, training code, and data-processing code; upstream data terms remain component-specific.
  • Method: The efficiency stack combines RTX 5090 hardware, blockwise FP8, MuonH optimization, Curriculum Model Averaging, and proxy-guided data selection.The components target cost per computation, throughput, token efficiency, training-data ordering, and dataset-mixture design, respectively.
  • Scaling law: The Puro Cost Scaling Law relates rental-equivalent training budget to model capability for scale-down scenarios using uniform Phase 2 data without CMA.The law is fitted from resumed Phase 1 checkpoints trained with different Phase 2 token budgets.
  • Post-training case study: Matched post-training preserves curriculum advantages, with curriculum initialization yielding higher GSM8K accuracy and improving the 15-task aggregate and most component evaluations.The comparison is reported across focused and larger mathematics settings and a broader instruction setting.
  • Results: At about $4.4K, a Puro-2B model exceeds Qwen2-1.5B, while the best checkpoint approaches Qwen2.5-1.5B at $6.9K.These results support the paper’s claim that affordable open pretraining is practical today.

2 Overview

The Puro-2B recipe combines publicly sourced data, consumer-grade RTX 5090 hardware, two-phase pretraining, curriculum-aware optimization and averaging, post-training, and evaluation. Its reported comparison surpasses Qwen2-1.5B and Gemma-2-2B under the stated budget accounting, while reproduction cost covers only finalized production pretraining compute.

  • Overview: The complete pipeline builds a corpus from publicly accessible sources, selects cost-efficient hardware, runs two pretraining phases, averages models, post-trains, evaluates, and accounts for cost.Figure 3 contrasts this open, low-cost workflow with common practice.
  • Two-phase pretraining, curriculum, and optimization: Phase 1 processes 438.8B tokens and Phase 2 processes 960.0B tokens, with the canonical run applying a curriculum over Phase 2 data.The curriculum presents the more preferred portion of each scored source later in training.
  • Two-phase pretraining, curriculum, and optimization: Hyperball optimization updates selected approximately scale-invariant matrices with MuonH, while AdamW updates remaining parameters under shared base learning-rate schedules.MuonH projects updated matrices back to their initial Frobenius radius, and its weight-learning-rate schedule is 10 times the base schedule.
  • Evaluation and post-training: PuRo-2B surpasses Qwen2-1.5B and Gemma-2-2B while using less than one sixth of the stated training cost of comparable open recipes.The comparison uses open-weight and reproducible open-recipe models of comparable scale across reasoning, mathematics, and coding.
  • Cost accounting: The headline reproduction cost covers only compute for one finalized two-phase pretraining rerun, excluding data work, experiments, post-training, evaluation, averaging, labor, and non-accelerator resources.Measured GPU-hours are converted using a normalized RTX 5090 rental-equivalent rate.

3 Training Recipe

The recipe combines consumer RTX 5090 hardware, communication and parallelism optimizations, FP8 training, Hyperball effective-learning-rate control, and data or averaging choices to enable cost-efficient pretraining. These components achieve high utilization while preserving BF16-equivalent quality and improving downstream recipe performance under stated conditions.

  • Hardware choice: RTX 5090 delivers about 2.7× the BF16 and FP8 compute efficiency of H200, motivating consumer-GPU pretraining despite lower peak throughput.The target model is small enough for the configured cluster to reach approximately 73% mixed-precision effective MFU.
  • Hardware optimization: P2P optimization raises one-way bandwidth from 31.5 to 56 GB/s, bidirectional bandwidth from 32 to 111 GB/s, and reduces latency from 14.3 to 0.4 µs.These modifications remain constrained by PCIe limits and should be enabled only when supported by the hardware topology.
  • Parallel training: The selected parallel configuration sustains median 238 TFLOP/s per GPU with global batch size 1536 on 24 GPUs, using MBS = 2, PP = 2, and DP = 12.The configuration uses layout (18|10)5 and matches the fastest strategy found by exhaustive enumeration.
  • FP8 training: Blockwise FP8 raises validation loss by 0.0031–0.0039 versus BF16 across five model sizes, corresponding to 98.0% BF16-equivalent compute retention.Model weights remain BF16, with release or deployment representations produced through separate post-training conversion.
  • Hyperball optimization: Hyperball makes effective learning rate directly schedulable; aligned effective-LR Muon reaches final validation loss 3.030, close to MuonH at 3.029.Ordinary Muon’s unaligned effective learning rate decays rapidly early, whereas MuonH follows the prescribed linear-decay schedule and overtakes it near training’s end.
  • Recipe ablations: Curriculum ordering improves scores by 1.18 points without model averaging and by 1.61 points for corresponding model averages, while longer peaks or horizons favor longer decay.The curriculum comparisons are 57.17 versus 55.99 without averaging and 57.18 versus 55.57 with averaging; the decay findings come from Figures 6 and 18.

4 Evaluation

PuRo-2B is evaluated against compact language models using a common deterministic pipeline across 15 benchmarks, outperforming Qwen2-1.5B in both mathematics/code and reasoning/knowledge while approaching Qwen2.5-1.5B. The evaluation also analyzes recipe-specific cost and efficiency factors, including RTX 5090 hardware, MuonH, FP8, and the Puro Cost Scaling Law.

  • Evaluation setup: 15 benchmarks compare pretrained/base models across mathematics and code, plus reasoning and knowledge, with all models evaluated under consistent revisions, prompts, decoding, and postprocessing.The pipeline uses generation-based evaluation for six tasks and evaluates both cloze and multiple-choice formulations when supported, selecting each model’s better formulation for aggregates; scores are percentages and averages are unweighted.
  • Mathematics and code: 43.50 average on mathematics and code, 3.21 points above Qwen2-1.5B and within 4.02 points of Qwen2.5-1.5B.PuRo-2B outperforms several open-recipe baselines, including Instella-3B, OLMoE-A1B/7B, and MiniCPM5-1B-Base, but trails Yulan-Mini-2.4B, SmolLM3-3B-Base, and MobileLLM-R1-950M-base.
  • Hardware efficiency: 2.77× higher peak BF16 compute per unit price and 2.74× higher peak FP8 compute per unit price than H200, with 73% MFU achieved in mixed-precision training.These figures use the RTX 5090 specification-and-price proxy and tuned training configuration described for the recipe.
  • Recipe efficiency: 1.19× compute-equivalent multiplier for the complete MuonH recipe, corresponding to 16.1% less theoretical compute at matched validation loss.The estimate comes from a TPP=20 matched scaling ladder and transfers the fitted horizontal multiplier to the production setting; it is a counterfactual quality-equivalent saving rather than a sequential end-to-end speedup.
  • Recipe efficiency: 98.0% BF16-equivalent compute retention for blockwise FP8, implying approximately 2.0% additional nominal compute to match BF16 quality.The estimate is based on a shared-shape fit across five ladder scales and separates precision quality retention from throughput speedup.
  • Puro Cost Scaling Law: The Puro Cost Scaling Law characterizes a recipe-specific cost–performance relationship from continued Phase 2 runs at several budgets, rather than a universal law across model families.The analysis is intended to describe scale-down behavior for communities with limited compute and time.

5 POST-TRAINING

The cost-scaling analysis finds that a uniform checkpoint at approximately $4.4K already exceeds Qwen2-1.5B without curriculum model averaging, while PuRo-2B lies above and to the left of the comparator frontier under the stated accounting protocol.

  • Cost scaling: Approximately $4.4K already exceeds Qwen2-1.5B for the uniform checkpoint without CMA.The shifted fit reduces in-sample RMSE from 0.452 to 0.209.
  • Estimation protocol: The analysis estimates compute primarily from reported GPU-hours and converts them using the reference rental rates in Table 6; otherwise it uses C = 6ND with reported accelerator and MFU information when available.When those statistics are unavailable, it assumes H100-equivalent GPU-hours at 70% MFU.
  • Pareto frontier: PuRo-2B’s rental-equivalent cost–performance point is more favorable than the comparator frontier under the stated accelerator-cost assumptions.The separation is especially pronounced among the fully-open comparators.

5 Post-Training Results and Analysis

Curriculum/CMA pretraining initialization retains an advantage after supervised adaptation, improving mathematics performance in focused and scaled settings and broad instruction-tuning results across most evaluations.

  • Experimental scope: The studies compare uniform global-reshuffle and curriculum/CMA Phase 2 endpoints under increasingly broad supervised settings, showing the recipe difference persists through adaptation.The curriculum/CMA endpoint includes late constant-LR continuation and six-checkpoint averaging.
  • Focused mathematics experiment: 68.66% versus 66.89% mean GSM8K accuracy at step 172, a 1.77 percentage-point gain for curriculum initialization across repeated runs.The focused comparison uses uniform global-reshuffle and curriculum/CMA endpoints without pretraining replay.
  • Scaled mathematics experiment: 76.12% versus 74.10% mean GSM8K accuracy at step 2,431, a 2.02 percentage-point gain for curriculum initialization, ahead in every run.This scaled setting uses a larger mathematics SFT mixture, small replay component, and longer budget; its absolute scores are not directly comparable with the focused setting.
  • Broad instruction experiment: 56.58% versus 54.99% 15-task macro-average, with curriculum improving 13 of 15 component evaluations and 15 of 18 total evaluations.The gains extend beyond mathematics-focused SFT to IFEval, BBH, and MMLU-Pro, though individual regressions occur in the appendix.

6 Related Works

Prior open-model and open-recipe efforts improve transparency and reproducibility, but practical accessibility remains limited by substantial hardware and training costs. PuRo-2B addresses this gap with a from-scratch, FP8-trained 2B model and an end-to-end, cost-accounted RTX 5090 pipeline.

  • Open recipes: Open-recipe projects release data, checkpoints, code, logs, and reproducible configurations that enable research on learning dynamics, data selection, and scaling.Examples include Pythia, OLMo, OLMoE, SmolLM3, Yulan-Mini-2.4B, and Instella [10] [35] [64] [52].
  • Accessibility gap: Despite this openness, representative open-recipe projects require large specialized clusters, leaving a gap between reproducibility and practical accessibility.Yulan-Mini-2.4B used 48 A800 GPUs, SmolLM3 384 H100 GPUs, and Instella-3B’s first stage 128 MI300X GPUs [52].
  • Cost accounting: Low-cost training requires jointly accounting for workload, realized hardware throughput, hardware price, and model quality rather than relying on token count or FLOPs alone.The paper treats cost as meaningful only under an explicit accounting boundary.
  • Efficiency methods: Related efficiency work reduces execution cost or training workload through low precision, architecture, data selection, staged training, and systems optimization [63] [26] [73] [111] [84] [55].These approaches target either the cost of a fixed workload or the workload itself, rather than necessarily providing an accessible end-to-end pretraining pipeline.
  • Consumer hardware: Consumer-GPU studies establish relevant training operations’ feasibility, but QLoRA leaves original pretraining cost unchanged and prior work focuses primarily on individual techniques [28] [83] [73].LLMQ studies full-model pretraining on RTX 4090 GPUs, while Quartet II measures NVFP4 throughput on a single RTX 5090.
  • PuRo-2B’s position: PuRo-2B combines a from-scratch 2B model, more than 1.4T training tokens, FP8 precision, measured accelerator usage, and an explicit replication-cost boundary.The end-to-end evaluation measures both resulting model quality and replication cost.

7 Conclusion and Future Direction · B COST ASSUMPTIONS

The report presents a cost-efficient, hardware-accessible pretraining recipe validated through the PuRo-2B collection, while showing that inspectable pretraining curricula affect post-training outcomes. Future work targets post-training, broader architectures, and expanded hardware recipes.

  • 7 Conclusion and Future Direction: A PuRo-2B model reaches Qwen2-1.5B performance at about $4.4K using a reproducibility recipe on consumer-grade RTX 5090 GPUs.The model collection spans different training budgets and recipe variants, exposing the pipeline’s cost–performance tradeoff.
  • 7 Conclusion and Future Direction: The PuRo-2B collection provides a direct view of the cost–performance tradeoff across training budgets and recipe variants.The collection is trained from scratch on consumer-grade RTX 5090 GPUs.
  • 7 Conclusion and Future Direction: Phase 2 pretraining recipe differences remain measurable after supervised post-training adaptation.The case study is enabled by a fully inspectable pretraining pipeline rather than model weights alone.
  • 7 Conclusion and Future Direction: Curriculum initialization improves focused-mathematics endpoint accuracy by 1.77 percentage points on average after adaptation.This result comes from following different Phase 2 pretraining recipes through supervised post-training.
  • 7 Conclusion and Future Direction: A larger mathematics recipe yields a 2.02-point mean gain after supervised post-training.The comparison is part of a case study on how Phase 2 pretraining recipes shape downstream performance.
  • 7 Conclusion and Future Direction: Future work will extend the reproducibility recipe beyond pretraining to post-training, emphasizing agentic capabilities.The report identifies post-training as the first direction for extending the foundation.
  • 7 Conclusion and Future Direction: Future directions include looped Transformers, linear-attention models, mixture-of-experts architectures, and other emerging designs.These efforts broaden the architectural space beyond standard dense Transformers.
  • 7 Conclusion and Future Direction: The authors also plan to expand hardware recipes beyond RTX 5090 GPUs.This is identified as a further direction for extending the reproducibility foundation.

A Limitations · B Reproduction-Cost Assumptions

The report limits its claims to a processed-data curriculum comparison, basic Chinese coverage, an overtrained fixed-2B regime, and a recipe-specific cost-scaling curve. It constructs cost–performance coordinates from 15-task averages and a hierarchy of reported or estimated accelerator costs.

  • A Limitations: The curriculum comparison lacks a strict corpus-wide exact- or near-duplicate audit against every proxy and final benchmark because most training data come from processed or filtered open datasets.This limitation may weaken the evidential strength of the curriculum comparison, although the supplied passage notes that its post-training advantage persists beyond the base check.
  • A Limitations: Chinese capability is only a basic coverage goal, so Chinese scores are omitted from release-facing comparison tables while the Chinese proxy axis remains for auditing data-mixture decisions.Headline evaluation instead focuses on mathematics, code, and reasoning axes as stated in the supplied passage.
  • A Limitations: Puro-2B uses approximately 1.4T tokens for 2B parameters, or about 700 tokens per parameter, making it an overtrained data-rich regime rather than a compute-optimal design.The chosen point balances RTX 5090 memory and communication limits, attainable benchmark quality, and community support for a dense base model.
  • A Limitations: Puro-2B’s fixed-2B cost-scaling law is recipe-specific and does not establish a model-size scale-up law, with RTX 5090 HBM capacity potentially limiting larger models.Scaling up model size or world size would require new communication and memory designs, potentially including a smaller vocabulary or embedding/LM-head partition.
  • B Reproduction-Cost Assumptions: Figure 1 uses Puro-2B uniform ($4.4K) and canonical curriculum ($6.9K) endpoints, while broader uniform and curriculum variants are summarized in Figure 13b.The two endpoint costs are summarized in Table 1, and curriculum model averaging is described in Figure 7.
  • B Reproduction-Cost Assumptions: The performance coordinate is the unweighted arithmetic mean across 15 benchmarks, and every plotted model must have all 15 scores present.The benchmark set includes GSM8K, MATH, sanitized-MBPP, HumanEval, MMLU, MMLU-Pro, ARC-Challenge, ARC-Easy, BoolQ, CommonsenseQA, HellaSwag, PIQA, SocialIQA, WinoGrande, and BBH.
  • B Reproduction-Cost Assumptions: Reproduction cost uses reported monetary cost first, then converts reported GPU-hours using Table 6 rates, or computes GPU-hours as H = GT from GPU count and elapsed time.Converted costs retain RMB in the audit ledger and divide by 6.8067 for the USD coordinate.
  • B Reproduction-Cost Assumptions: When only token counts are available, cost is estimated with C = 6ND using H100-equivalent Fa = 989.5 TFLOP/s and η = 0.70; entries lacking usable cost evidence are omitted.Model-specific token counts are preferred, family-level budgets are marked as estimates, and the ideal MFU assumption makes the counterpart estimate a cost lower bound.

C TRAINING DETAILS

Puro-2B’s reported training cost is based on measured active-training time and local RTX 5090 pricing, totaling $6,891. The comparison frontier retains cost-score nondominated models after excluding Puro-2B.

  • Cost accounting: $6,891 is the reported Puro-2B training cost, calculated from 22,514.41 GPU-hours at 2.0833 RMB per GPU-hour.Measured active-training time comprises 6,009.46 GPU-hours in Phase 1 and 16,504.95 GPU-hours in Phase 2.
  • Pareto frontier: The dashed comparison frontier excludes Puro-2B and retains models with no other comparison model having lower cost and higher score.At identical cost, only the highest-scoring point is retained; remaining points are kept when their score exceeds every lower-cost score.

C Production Training Details

The canonical PuRo-2B production run uses a two-phase, mixed-precision setup with MuonH/AdamW optimization and fixed 4,096-token sequences. Phase 2 restarts from a shared Phase 1 endpoint with varied decay horizons, while the pipeline assigns FP8, BF16, and FP32 to different computation components.

  • Production run setup: The canonical run uses two phases with sequence length 4,096, global batch size 1,536, micro-batch size 2, MuonH for selected matrix weights, AdamW for remaining parameters, and blockwise E4M3 FP8.MuonH uses zero weight decay, while AdamW uses weight decay 0.1; the Hyperball weight learning rate is 10 times the base rate.
  • Phase 1 learning rate schedule: Phase 1 uses a power-decay schedule with p = 1/2, τ = 1,000 steps, 1,000-step warmup, and a 5 × 10−4 asymptotic floor to support continual training.The warmup corresponds to 1,536,000 samples, and MuonH-controlled matrix groups use a 10× optimizer multiplier.
  • FP8 implementation pipeline: The FP8 pipeline uses blockwise E4M3 for linear-layer GEMMs and BF16 for hidden/residual flow, LayerNorm, and embeddings, while gradients, optimizer states, and softmax remain FP32.The listed GEMMs include forward propagation, data-gradient, and weight-gradient computations.
  • Phase 2 learning rate schedules: Phase 2 restarts from the shared Phase 1 endpoint at η0 = 1.04 × 10−3, with five decay horizons of approximately 60.1, 120.1, 240.2, 480.5, and 960.9B tokens.All schedules share ηmin = 10−5; the canonical run replaces the final 29B-token learning-rate decay with curriculum model averaging starting at step 218,000.

D MuonH Scaling-Ladder Analysis

The MuonH scaling-ladder analysis compares optimizer and precision efficiency across five models from 0.17B to 1.7B under a controlled TPP=20 setup. MuonH achieves a 1.19× compute-equivalent multiplier over updated Muon, while blockwise FP8 retains 98.0% effective compute and raises 1.7B throughput by 1.36×.

  • Experimental setup: The ladder spans five models from 0.17B to 1.7B, all trained from scratch for 20 tokens per scaling parameter with matched data, tokenizer, sequence length, batch size, validation set, and learning-rate decay.The runs use sequence length 4,096, global batch size 512, and a 1% warmup; the validation set is a subset of Nemotron-CC [89] [8].
  • Compute-equivalent fitting: The compute-equivalent comparison jointly fits baseline and variant observations with shared loss-shape parameters and variant-specific amplitude to estimate a horizontal efficiency shift.This restricted fit is not used to select a learning-rate schedule or predict the absolute loss of the 2B production run.
  • MuonH scaling result: 1.19× compute-equivalent efficiency lets MuonH match the updated Muon baseline loss with 16.1% less compute, based on a five-size ladder and a 1.68 × 10^22-FLOP 2B/1.4T-token transfer.Leave-one-model-out fits range from 1.17–1.28×, but the transfer assumes the horizontal efficiency factor persists beyond the ladder’s TPP=20 regime.
  • Precision and throughput decomposition: Blockwise FP8 retains 98.0% effective compute relative to BF16, despite a 0.0031–0.0039 higher validation loss at fixed theoretical compute.Leave-one-size-out fits range from 97.6–98.1%, corresponding to a 2.0% nominal-compute penalty at matched quality.

E Post-Training Details · F Learning Rate Schedule Diagnostics

Across focused mathematics, scaled mathematics, and broad instruction settings, curriculum initialization consistently improves post-training performance over uniform initialization, though transfer gains are non-uniform. The paper also provides additional learning-rate schedule diagnostics and fitting results.

  • E Post-Training Details: Curriculum initialization wins all six focused mathematics comparisons, including endpoints, with gains ranging from 1.36 to 2.12 points and averaging 1.77 points.The focused experiment repeats three SFT data orders and measures predefined intermediate and endpoint steps.
  • E Post-Training Details: The focused post-training configuration uses eight RTX 5090 GPUs with MuonH, a 10^-5 base learning rate, and a 10-times Hyperball matrix-group multiplier.The base learning rate ends at 10^-7 under the specified optimizer and parallelism configuration.
  • E Post-Training Details: The focused GSM8K evaluation scores all 1,319 test examples using zero-shot ChatML prompting, greedy BF16 generation, and a 512-token limit.Correctness is determined by exact numeric equivalence after normalization and structured answer extraction.
  • E Post-Training Details: Every focused GSM8K row has more curriculum-only than uniform-only correct answers, indicating gains reflect changed question coverage rather than only averaging effects.The analysis uses IoU and asymmetric C-only/U-only counts, with manual inspection separating numerical correctness from termination and parsing behavior.
  • E Post-Training Details: Curriculum leads in all three scaled mathematics endpoint runs, with a mean GSM8K improvement of 2.02 percentage points and a 1.21–3.26-point range.The scaled setting includes 2,014,933 conversations, including 99,989 replay records, and runs for 2,431 optimizer steps.
  • E Post-Training Details: Curriculum improves the broad instruction macro-average by 1.59 percentage points and is higher on 13 of 15 component evaluations.The macro-average is an unweighted average over 15 task scores, computed from underlying unrounded scores.
  • E Post-Training Details: Curriculum is higher on 15 of 18 reported evaluations, improving IFEval and MMLU-Pro while modestly decreasing on BBH.These additional tasks use separate prompts and scorers and are excluded from the primary 15-task macro-average.
  • F Learning Rate Schedule Diagnostics: The learning-rate schedule diagnostics section provides additional diagnostics and fitting results.No further diagnostic findings are supplied in the provided passage.

F.1 Multi-Power Law for Effective Learning Rate Schedules

The Multi-Power Law adapts schedule-loss modeling to scalar or effective learning-rate exposure, adding a delayed correction for learning-rate decreases. In a two-run diagnostic, effective learning rate predicts held-out validation loss better on average than ordinary learning rate, while early effective-rate decay helps explain run differences.

  • Model construction: The MPL treats cumulative learning-rate exposure as an effective optimization-time coordinate and adds a schedule-shape correction for decreases in the learning-rate signal.The baseline is Chinchilla-like, whereas the correction distinguishes schedules with similar exposure but different decay patterns.
  • Model construction: The response model delays each learning-rate decrease’s loss benefit according to subsequent exposure, with the unrealized fraction decaying as (1 + x)^-β.G(x) approaches one as exposure accumulates, representing the fraction of the eventual reduction realized by time t.
  • Analytical simplification: The G(x) = 1 simplification telescopes the response-weighted sum, retaining only cumulative exposure and the net decrease from qwarm to qt.For decaying schedules with B > 0, the final term lowers predicted loss, but the approximation discards when individual reductions occurred.
  • Empirical transfer: The diagnostic fits post-warmup validation curves for ηt and ρt on two BF16 ordinary-Muon runs, normalizing each signal by its own post-warmup peak and holding out the final 20%.The three-stage protocol fits power-only, power-plus-schedule-drop, and full-response MPL variants.
  • Empirical transfer: 0.0210 versus 0.0265 mean held-out RMSE favors effective LR over ordinary LR, with the largest gain for base Muon at 0.0270 versus 0.0422.For aligned Muon, the two representations are comparably predictive, with RMSEs of 0.0149 and 0.0108.
  • Empirical interpretation: Early effective LR decay is linked to ordinary Muon’s loss of further decay potential, while MuonH and aligned Muon use linear LR decay and eventually surpass it.This interpretation follows Equation (17) and the schedule behavior shown in Figure 5.

F.2 WSD Sweeps and Limited-Compute Schedule Estimation … H.4 Scope of the Available Comparison

The paper uses formal MPL to screen WSD schedules under limited compute, while documenting the Puro-2B curriculum, continuation, and averaging pipeline. Available comparisons support the joint configuration but do not isolate the gains of individual design choices.

  • F.2 WSD Sweeps and Limited-Compute Schedule Estimation: Across 15 unseen WSD schedules, the two-anchor estimator achieves mean curve RMSE 0.0157 and endpoint MAE 0.0067, with predicted optimal ratios rising from 0.33 to 0.85.The estimator supports screening candidate peak-rate and decay settings, not certifying an optimum, because broad plateaus, noisy minima, and one seed per configuration limit precision.
  • F.2 WSD Sweeps and Limited-Compute Schedule Estimation: The WSD diagnostic supports long-decay features but does not establish the exact power exponent of the open-ended Phase 1 schedule.The continual-training role of the Phase 1 power schedule remains a design property rather than an inferred optimum.
  • F.3 Hyperball: Maintaining a prescribed effective LR without Hyperball can produce a hill-like scalar LR as weight norms grow while orthogonalized Muon-update norms remain roughly constant.The constructed WSD example ramps the effective LR to 0.02, holds it, and begins terminal decay at step 4,000; it demonstrates feasibility, not optimality or universality.
  • G SCALING CHECKPOINT LEDGER; G Puro-2B Scaling Checkpoint Ledger: The scaling checkpoint ledger freezes the accounting coordinates for Figure 2(b), with Phase 1 shared across rows and Phase 2 tokens counted as schedule consumption rather than duplicated pool data.Endpoint comparisons are visualized in Figures 7 and 13b, while Table 16 records checkpoint lineage and normalized RTX 5090 cost accounting.
  • H Curriculum Construction and Model Averaging; H.1 Scalable Construction of Component-Local Curriculum Buckets: The curriculum assigns component-local token-percentile coordinates, aligns equal percentile ranges into 376 production buckets, and avoids imposing a single global quality ranking across sources.Each bucket contains approximately 2.5B tokens and preserves approximately the same token fraction from every component.
  • H.2 Phase Transition and Uniform-Order Control: The Phase 2 transition contains approximately 43.9B tokens—21.9B from deterministic Phase 1 replay and 21.9B from the earliest 2.34% of Phase 2—while uniform control preserves the same token multiset but reshuffles order.The Phase 2 early range is already in the component pool, so only Phase 1 replay is added to the production total; exact reproduction also requires manifests, source revisions, and checksums.
  • H.3 Constant-LR Continuation and Checkpoint Averaging: The production export uses a constant-LR continuation resumed from step 218,000 and an equal-weight average of six late checkpoints, but these configurations were not a matched ablation.The continuation fixes the group-level effective LR at 4.08 × 10−4; the six-checkpoint average spans 469 optimizer steps, or approximately 2.95B tokens, and excludes optimizer states.
  • H.4 Scope of the Available Comparison: The available evidence compares complete Phase 2 configurations rather than isolating curriculum ordering, continuation, or averaging, so the result is interpreted as a joint curriculum-and-averaging configuration.Isolated attribution would require matched factorial runs, continuation branch or rate controls, and repeated seeds; downstream analysis instead compares curriculum and uniform endpoints under matched post-training.

I Data Recipe Details … I.3 Proxy-to-Mixture Decision Audit

The paper specifies materialized data components and licensing scope, evaluates candidate sources with a controlled Qwen3-0.6B proxy, audits capability aggregation with correlation PCA, and uses the resulting evidence to compare mathematics sources and contextualize Phase 2 mixture changes.

  • I Data Recipe Details: Dataset accounting reports materialized, tokenized component exposure rather than unique upstream-token estimates, and family rows may aggregate overlapping configurations.The table covers stationary Phase 1 and Phase 2 component pools, while licensing entries distinguish verified terms from missing or unverified licenses; some NVIDIA-covered data permit training but prohibit raw-data redistribution.
  • I.1 Proxy-Measurement Protocol: Each proxy condition starts from an 86B-token Qwen3-0.6B checkpoint and receives approximately 8.4B continuation tokens, differing only in candidate source or within-source slice.The protocol uses 2,000 steps, sequence length 4,096, global batch size 1,024, and aggregates fixed-suite benchmarks into Math, Code, Chinese, and General axes.
  • I.1 Proxy-Measurement Protocol: Sources below 5B available tokens are skipped, while larger scored sources are probed at the 0th, 25th, 50th, and 75th score quantiles.Unscored sources and scored sources containing 5B–50B tokens receive one adaptive sample targeting approximately 4B tokens, with up to five retries when needed.
  • I.1 Proxy-Measurement Protocol: The candidate share ramps from 0% to 80% over the first 1,600 steps and remains at 80% for the final 400 steps to avoid abrupt distribution changes.The resulting feature vector remains conditional on proxy scale, continuation schedule, candidate share, and evaluation suite.
  • I.2 Proxy Capability Aggregation and PCA Audit: The four capability axes average standardized benchmark groups, then undergo a second standardization before correlation PCA; the first two components explain 34.5% and 28.1% of variance.PC1 primarily contrasts General with Code, while PC2 primarily contrasts the remaining capability directions described by the loadings.
  • I.2 Proxy Capability Aggregation and PCA Audit: PCA is descriptive rather than an optimization objective because component signs are arbitrary and variance-maximizing rotations do not define utility.Recipe comparisons therefore use the original aggregate axes, including the Math–General comparison in Figure 20; Figure 20 explicitly cautions that these comparisons do not establish causal gains.
  • I.3 Proxy-to-Mixture Decision Audit: Phase 2 adds 13.5B MegaMath-Web-Pro tokens, removes FineMath, and reduces Nemotron-CC-Math 4+ and SwallowMath mixture shares from 2.62% to 0.26% and from 1.64% to 0.26%, respectively.The proxy comparison reports Math and General aggregates of 2.884 and 0.667 for MegaMath-Web-Pro, versus (2.821, 0.345) for Nemotron-CC-Math 4+, (2.746, −0.387) for SwallowMath, and (1.269, −0.317) for FineMath.
  • I.3 Proxy-to-Mixture Decision Audit: The proxy audit finds DCLM q00, MegaMath-Web-Pro, MegaMath-Code, and FineWeb-Edu-CN q00 leading the General, Math, Code, and Chinese aggregates, respectively, while noting this is ex post rather than a selection rule.Production ranges are not identical to proxy slices, and the axis leaders and production mixture lack repeated-seed uncertainty estimates.
Loading 2608.27370v1…