PaperlayerView exact revision on arXiv

Source-linked AI summary

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Mingzhe Zhang, Kaiyue Wen, Kaifeng Lyu, Wenguang Chen

arXiv:2608.27370v1cs.CLcs.LG

TL;DR

Open pretraining recipes can remain prohibitively expensive for resource-constrained labs. Puro-2B combines system, algorithm, and data design to train models from scratch on consumer-grade RTX 5090 GPUs, with its best checkpoint approaching Qwen2.5-1.5B performance at about USD 6.9K in accelerator-only cost.

  • Problem

    Open pretraining recipes can remain prohibitively expensive for resource-constrained labs, leaving a gap between reproducibility and practical accessibility.

    Source: §1 · p. 4

  • Method

    The authors combine system, algorithm, and data design into a pretraining recipe, training Puro-2B models from scratch on consumer-grade RTX 5090 GPUs.

    Source: §1 · p. 4

  • Results

    About USD 6.9K in canonical accelerator-only cost produces the best Puro-2B checkpoint, which approaches Qwen2.5-1.5B performance.

    Source: §7 · p. 26

  • Takeaways & Limitations

    A fully inspectable pretraining pipeline enables studying how pretraining recipe differences persist after supervised post-training, beyond what model weights alone permit.

    Source: §7 · p. 27

    MuonH and FP8 ablations use 20 tokens per parameter rather than production training’s 700 because of limited compute.

    Source: §4.3.1 · p. 22

Abstract

from arXiv · show

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \$1.5M, and reproducing SmolLM3-3B needs over \$700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \$6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about \$4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.

1 Introduction

Puro-2B combines consumer RTX 5090 hardware with coordinated precision, optimization, curriculum, and data choices for affordable open pretraining. Its best checkpoint surpasses Qwen2-1.5B and approaches Qwen2.5-1.5B at about $6.9K under the paper’s evaluation and cost-accounting protocol.

1 Introduction

  • High reproduction costs leave a practical gap between open pretraining recipes and accessibility for resource-constrained labs.Rental-equivalent estimates exceed $1.5M for Llama3.2-3B and reach $200K for OLMoE-1B-7B and $719K for SmolLM3-3B. Open recipes expose reconstructible training artifacts beyond open weights, but reproducibility also requires infrastructure, training details, and transparent cost accounting.Source: §1 · p. 4
  • $6.9K trains the best Puro-2B checkpoint, which surpasses Qwen2-1.5B overall and approaches Qwen2.5-1.5B under the paper’s evaluation protocol.The production schedule totals 1.4T tokens, 22,514 active-training GPU-hours, and 17.6 elapsed days on consumer-grade RTX 5090 GPUs.Source: §1 · pp. 4, 6
  • Consumer RTX 5090 GPUs, blockwise FP8, and MuonH target hardware cost, execution speed, and token efficiency, with supporting ablations of the main recipe choices.Blockwise FP8 reduces per-token execution time while maintaining quality comparable to BF16. MuonH extends Muon with hyperball constraints on parameter weights and updates, paired with a carefully designed learning-rate schedule.Source: §1 · pp. 4–6
  • $4.4K is the estimated budget for crossing Qwen2-1.5B’s performance aggregate under the recipe-specific Puro Cost Scaling Law.The law fits total cost against performance using shared Phase 1 checkpoints and varying Phase 2 budgets with uniform data and no CMA. At $6.9K, curriculum achieves 1.65× cost-efficiency gains over the uniform recipe; the canonical joint design achieves 2.40× relative to the uniform scaling curve.Source: Figure 2 · p. 5, §1 · pp. 5–6
  • Curriculum Model Averaging organizes source-local data chunks and averages selected checkpoints [56], while proxy experiments guide dataset selection and mixture design.Data selection reduces required tokens, while MuonH and curriculum improve token efficiency. These components are co-designed: hardware supports FP8, MuonH and CMA require careful learning-rate schedules, and preprocessing determines curriculum ordering.Source: §1 · p. 5
  • Curriculum/CMA advantages over uniform Phase 2 training persist after matched supervised adaptation, improving mathematics accuracy and broad instruction performance.Curriculum initialization yields higher GSM8K accuracy across repeated runs in focused and larger mathematics settings. Broad instruction adaptation also improves the 15-task aggregate and most component evaluations.Source: §1 · pp. 5–6
  • The released artifacts include model weights, intermediate checkpoints, datasets, training configurations, training code, and data-processing code for reproducing and inspecting pretraining.The release includes 10 checkpoint versions. Artifacts use Apache 2.0 unless separately noted, while upstream data terms remain component-specific.Source: §1 · pp. 4, 6

2 Overview

Puro-2B combines RTX 5090 hardware, FP8 computation, two-phase pretraining, data selection, and curriculum-aware averaging in an open, low-cost recipe. Reported reproduction costs cover production pretraining accelerator compute, not total model-development costs.

2.1 Pipeline at a Glance

  • 29B final curriculum tokens use a constant learning rate, followed by averaging the last six checkpoints to obtain the final model.Following the curriculum-aware averaging principle of Luo et al. [56], this preserves late-data influence; the uniform comparison reshuffles the same Phase 2 data and uses linear learning-rate decay. The $4.4K run uses uniform ordering, while the canonical $6.9K run uses curriculum model averaging.Source: Figure 3 · p. 6, §2.1 · p. 7, Table 1 · p. 7

2 Overview

  • Approximately 2B parameters result from untying input embeddings and the output head in Qwen3-1.7B, trained from scratch on consumer-grade RTX 5090 GPUs.Megatron Core uses blockwise FP8 for main Transformer linear-layer GEMMs while sensitive operations and persistent training states remain BF16 or FP32. Phase 1 uses 24 GPUs, and Phase 2 continues on 96 GPUs with adjusted distributed parallelism.Source: Figure 3 · p. 6, §2.1 · p. 7

2.1 Pipeline at a Glance

  • 438.8B tokens in Phase 1 precede 960.0B tokens in Phase 2, where the canonical curriculum presents more preferred within-source data later.Usable source-provided quality labels determine lower-to-higher ordering and coarse chunks; sources without usable scores are randomly ordered and chunked. Aligning chunks by normalized within-source ranks approximately preserves the cross-component mixture without requiring a global quality score.Source: §2.1 · p. 7
  • MuonH projects selected approximately scale-invariant matrices back to their initial Frobenius radius after every step, while AdamW updates the remaining parameters.MuonH uses 10 times the base learning rate; the base follows a power schedule [85] in Phase 1, linear decay in main Phase 2, and a constant rate during late CMA. Both phases use 4,096-token sequences and a global batch of 1,536 sequences.Source: §2.1 · p. 7
  • Publicly accessible, use-permitted datasets supply web text, code, mathematics, and synthetic examples, with Kaiyuan-SpaRK web deduplication and proxy-guided selection and mixing.Controlled small-model experiments produce downstream benchmark profiles that guide source selection and mixture weights. Comparing profiles across within-source score ranges also determines dataset-specific filtering thresholds, rather than imposing a cross-domain sample-quality score.Source: §2.1 · pp. 7–8
  • Less than one sixth of comparable open recipes’ stated training cost is used by Puro-2B, which surpasses Qwen2-1.5B and Gemma-2-2B under common accounting.Evaluation covers general reasoning, mathematics, and coding. Paired SFT experiments give curriculum and uniform initializations identical supervised data and optimization across focused mathematics, scaled mathematics, and broad instruction tuning.Source: §2.1 · p. 8

2.2 Reproduction-Cost Boundary

  • Reported reproduction cost measures one finalized two-phase pretraining run after training shards are materialized, using GPU-hours converted at a normalized RTX 5090 rental-equivalent rate.It excludes data acquisition and preprocessing, proxy experiments, scaling and ablation studies, failed or exploratory runs, post-training, evaluation, checkpoint averaging, research labor, and non-accelerator resources. These figures are marginal accelerator-cost proxies, not total cash expenditure or development costs; taxes, depreciation, and ownership costs outside the rental-equivalent rate are also excluded.Source: §2.2 · p. 8

3 Training Recipe

This section presents a training recipe combining consumer RTX 5090 GPUs, communication-aware system tuning, FP8 mixed precision, Hyperball optimization, learning-rate scheduling, source-local data curricula, and late checkpoint averaging. It also covers effective-learning-rate diagnostics, reproducible data construction, and paired supervised adaptation to test whether pretraining differences persist.

3.1.1 RTX 5090 as a Cost-Effective GPU Choice

  • 2.7× H200’s BF16 and FP8 compute per dollar makes RTX 5090 cost-effective despite lower peak throughput, smaller memory capacity, and no NVLink.All pretraining and post-training use consumer RTX 5090 GPUs. Their lower effective price outweighs their throughput disadvantage for this workload, while unavailable Blackwell data-center accelerators make RTX 5090 the practical FP4-capable option.Source: §3.1.1 · pp. 8–9, Table 2 · p. 9

3.1.2 Hardware Setup and Tweaks

  • 56 GB/s one-way bandwidth and 0.4 µs latency follow PCIe P2P enablement, improving from 31.5 GB/s and 14.3 µs.A modified NVIDIA driver and platform changes enable P2P; separately enabling GDR increases 24-GPU AllReduce bandwidth from approximately 8.87 to 19.93 GB/s. These unsupported modifications carry user risk and topology-dependent benefits; stock drivers remain usable with lower inter-node bandwidth.Source: §3.1.2 · pp. 9–10

3.1.3 Efficient Training System

  • 238 TFLOP/s per GPU is sustained in Phase 1 on 24 GPUs; Phase 2 achieves 192 TFLOP/s per GPU on 96 GPUs.Phase 1 reaches approximately 73% equivalent MFU under a precision-weighted peak convention, not a BF16-only measurement. The system combines data and pipeline parallelism without tensor parallelism, kernel-guided micro-batch sizing, fewer layers alongside the LM head, and memory-aware optimizer placement.Source: §3.1.1 · p. 9, §3.1.3 · pp. 10–11

3.2 FP8 Mixed-Precision Training

  • 1.34× quality-adjusted net gain accompanies 1.36× median training throughput at the 1.7B scale with blockwise FP8 versus BF16.Across five model sizes at 20 tokens per parameter, FP8 increases validation loss by 0.0031–0.0039, corresponding to 98.0% BF16-equivalent compute retention. Training starts directly in FP8: Transformer linear layers use E4M3 operands, while training state and sensitive operations retain BF16/FP32, core attention remains BF16, and checkpoint weights remain BF16.Source: §3.2 · p. 11

3.3 Hyperball Optimization

  • MuonH normalizes updates and fixes selected attention and MLP matrices to their initial Frobenius radius, while other parameters use AdamW.The recipe adopts the Hyperball wrapper introduced by Wen et al. [97] for approximately scale-invariant weights. Embeddings, normalization layers, and the language-model head remain under AdamW.Source: §3.3 · pp. 11–12

3.3.1 Effective Learning Rate

  • 3.029 final validation loss for MuonH compares with 3.073 for ordinary-LR Muon and 3.030 for effective-LR-aligned Muon in matched 170M BF16 runs.MuonH directly controls effective LR, whereas ordinary Muon’s effective LR depends on evolving weight and update norms. Aligning ordinary Muon’s effective-LR trajectory largely recovers MuonH’s loss curve without fixed-radius projection, supporting schedule control as an important factor in this diagnostic.Source: §3.3.1 · pp. 12–13, Figure 5 · p. 13
  • 0.0265 to 0.0210 is the reduction in mean held-out RMSE when Multi-Power Law [57] uses effective LR rather than scalar LR.The two-run diagnostic holds out the final 20% of validation targets; improvement comes mainly from ordinary base-Muon, with comparable predictiveness for effective-LR-aligned Muon. Together with the late-stage crossover, these diagnostics motivate treating effective LR as the primary scheduling hyperparameter and retaining decay for later training.Source: §3.3.1 · p. 14

3.3.2 Learning Rate Schedule Design

  • An open-ended stable phase followed by controlled linear terminal decay accommodates continual pretraining when the total token budget is initially unknown.Warmup-Stable-Decay [43] sweeps use a Qwen3-0.6B architecture and global batch size 512: at 20 tokens per parameter, effective peaks span 0.008–0.024 in 0.004 increments and decay ratios span 0.2–1.0 in 0.2 increments. Peaks 0.008 and 0.012 are additionally tested at 50 and 100 tokens per parameter.Source: §3.3.2 · p. 14
  • Larger effective peak learning rates and longer training horizons favor longer decay, with broad competitive regions rather than precise monotone optima.Across effective peaks 0.008–0.024, the lower edge within 0.01 validation loss of the minimum moves nondecreasingly from decay ratio 0.4 to 0.8; endpoint interpolation at peak 0.024 estimates 0.85. At peak 0.012, that edge shifts from 0.4 at TPP 20 to 0.6 at TPP 50 and 100.Source: Figure 6 · p. 14, §3.3.2 · pp. 14–15
  • 438.8B tokens of open-ended power decay precede 960.0B tokens of long linear decay in the production effective-LR schedule.The first phase decreases effective LR from approximately 5 × 10^-2 to 1.04 × 10^-2, retaining a continuation path as data evolve [85]. A two-endpoint multi-power-law diagnostic [57] estimates preferred decay ratios increasing from 0.33 at peak 0.008 to 0.85 at peak 0.024, without certifying optimality.Source: Figure 6 · p. 14, §3.3.2 · p. 15

3.4 Curriculum Model Averaging

  • 376 source-aligned curriculum buckets preserve mixture weights while moving scored components from less preferred to more preferred examples.Each bucket targets approximately 2.5B tokens using cumulative-token ranks within components; unscored components retain fixed random orders. Following the CMA principle [56], the selected late continuation fixes base LR at 4.08 × 10^-5 and Hyperball effective LR at 4.08 × 10^-4, then equally averages six checkpoints.Source: §3.4 · pp. 15–16
  • 57.81 is the strongest tested endpoint, obtained through curriculum ordering, constant-LR continuation from step 218,000, and six-checkpoint averaging.Curriculum improves corresponding endpoints by 1.18 points without averaging and 1.61 points with averaging; averaging without constant-LR continuation changes uniform and curriculum endpoints by −0.42 and +0.01 points, respectively. Later-data contamination could contribute to the curriculum advantage, although the authors argue it is unlikely to explain persistent post-training improvements.Source: §3.4 · p. 16

3.5 Post-Training Recipe

  • Three paired SFT settings—focused mathematics without replay, scaled mathematics with limited replay, and broad Tulu-3 instruction tuning—test whether pretraining differences persist.GSM8K accuracy is averaged across repeated runs at matched checkpoint steps; broad evaluation uses the preselected step-300 checkpoint on the 15-task OpenCompass core suite, reporting macro-average and per-task scores. Deduplicated, packed SFT data exclude evaluation questions and answers; analysis examines solved-set IoU, asymmetric gains, termination and extraction failures, and negative transfer.Source: §3.5 · pp. 16–17

3.6 Data Recipe

  • Data construction separates recipe selection from preprocessing and shard reconstruction, treating recorded operations as essential to reproducibility.Recipe selection determines retained sources, filters, and token allocations under a limited budget; preprocessing converts that recipe into fixed, reconstructible training shards.Source: §3.6 · p. 17

3.6.1 How Do We Select a Good Data Recipe?

  • Qwen3-0.6B proxy continuations evaluate candidate sources and source-local slices on 15 benchmarks, guiding filtering, domain balance, and mathematics-focused allocation.Scores are not globally comparable across sources; large scored sources receive percentile-window probes at q00, q25, q50, and q75, and oversupplied domains retain higher-proxy-scoring slices. Phase 1 allocates 73.2% of 438.8B tokens to English, while Phase 2 allocates 18.3% of its 938.1B-token pool to mathematics and introduces 1.3% instruction-formatted data.Source: §3.6.1 · pp. 17–19

3.6.2 How Do We Preprocess Data and Reproduce the Shards?

  • Approximately 1.4T training tokens are delivered through static shards, including a 43.9B-token transition that linearly replaces Phase 1 replay with Phase 2 data.Kaiyuan-SpaRK [55] performs within-source deduplication, proxy benchmarking, heuristic filtering, and materialization with frozen revisions, retention rules, mixture weights, and seeds; training uses neither online scoring nor dynamic mixture sampling. Phase 2 consumes approximately 960.0B tokens: 21.9B replay tokens plus its 938.1B-token component pool, including pool tokens consumed during transition.Source: §3.6.2 · p. 19

4 Evaluation

PuRo-2B exceeds Qwen2-1.5B on mathematics/code and reasoning/knowledge averages under a shared deterministic evaluation pipeline. Its cost-saving estimates use distinct reference comparisons rather than measuring one end-to-end speedup.

4.1.1 Model Selection

  • Model comparisons primarily cover the 1B–3B parameter range and consistently use pretrained/base checkpoints, separating open-weight from open-recipe releases.Open-weight releases provide weights without complete training data or recipes; open-recipe releases additionally provide data and training artifacts needed to study or reproduce training.Source: §4.1.1 · pp. 19–20

4.1.2 Benchmarks

  • 15 benchmarks assess mathematics, code generation, academic knowledge, reading comprehension, commonsense understanding, and general multi-step reasoning.The suite groups four mathematics/code benchmarks and eleven reasoning/knowledge benchmarks; generated Python programs are checked with unit tests.Source: §4.1.2 · p. 20, Table 4 · p. 21, Table 5 · p. 22

4.1.3 Implementation Details

  • Evaluation uses a common deterministic OpenCompass pipeline, with greedy generation for six benchmarks and candidate-likelihood scoring for the remaining nine.Checkpoint/tokenizer revisions, datasets, prompts, shot examples, decoding, and postprocessing are fixed, including deterministic GSM8K answer extraction. Following OLMES [36], supported tasks use each model’s better cloze or multiple-choice formulation; scores are percentages and averages are unweighted arithmetic means.Source: §4.1.3 · pp. 20–21

4.2 Resulting Performance

  • 43.50% is PuRo-2B’s mathematics/code average, exceeding Qwen2-1.5B by 3.21 points and coming within 4.02 points of Qwen2.5-1.5B.PuRo-2B outperforms Instella-3B, OLMoE-A1B/7B, and MiniCPM5-1B-Base, but trails Yulan-Mini-2.4B, SmolLM3-3B-Base, and MobileLLM-R1-950M-base. MobileLLM-R1 leads PuRo-2B by 3.86 points on this four-task average.Source: Table 4 · p. 21, §4.2 · p. 21
  • 63.02% is PuRo-2B’s reasoning/knowledge average, exceeding Qwen2-1.5B by 2.48 points and coming within 2.51 points of Qwen2.5-1.5B.PuRo-2B surpasses OLMoE-A1B/7B, Yulan-Mini-2.4B, MiniCPM5-1B-Base, and MobileLLM-R1-950M-base, with a 6.29-point advantage over MobileLLM-R1. It trails the larger Instella-3B by 0.11 points and also trails SmolLM3-3B-Base.Source: §4.2 · p. 21, Table 5 · p. 22

4.3.1 Cost-Saving Factors

  • 2.77× BF16 and 2.74× FP8 peak compute per unit price favor RTX 5090 over H200 under the paper’s specification-and-price proxy.The tuned recipe achieves 73% MFU in mixed-precision training. Cost factors use different references, so their products are accounting estimates, not measured end-to-end speedups; MuonH/FP8 ablations use TPP=20 versus production TPP=700.Source: §4.3.1 · p. 22
  • 16.1% less theoretical compute for MuonH versus Muon and 1.34× matched-quality FP8 speedup versus BF16 are estimated from matched scaling ladders.MuonH’s fitted multiplier is 1.19×; this is a counterfactual saving, while equal full-token budgets instead yield lower validation loss at unchanged nominal compute. FP8 combines 98.0% fitted BF16-equivalent compute retention with 1.36× throughput at the 1.7B proxy, implying 25.2% fewer GPU-hours rather than measured full-horizon savings.Source: Figure 10 · p. 23, §4.3.1 · p. 23
  • The Puro Cost Scaling Law fits recipe-specific cost–performance scale-down using uniform Phase 2 continuations from the same Phase 1 checkpoint.The fit is P = a + b log2(C − C_P1), with total reproduction cost C and fixed Phase 1 cost C_P1 = USD 1.84K; it is not universal across model families. The separate Phase 2 cost factor uses the fitted uniform-equivalent cost of the curriculum/CMA endpoint.Source: §4.3.1 · pp. 22–23

5 POST-TRAINING

PuRo-2B achieves a more favorable rental-equivalent cost–performance position than the comparator frontier under the stated accounting assumptions. Fitted uniform-recipe cost equivalents are recipe-level estimates, not isolated curriculum gains or a universal cost law.

5 POST-TRAINING

  • $16.55K is the fitted uniform-recipe cost equivalent of the reported Puro-2B aggregate, or 2.40× the $6.89K production cost.The aggregate additionally includes constant-LR continuation and averages the last six checkpoints.Source: §5 · p. 24
  • $11.36K is the fitted uniform-recipe cost equivalent of the curriculum-decay endpoint, or 1.65× the production recipe’s $6.89K rental-equivalent estimate.The endpoint changes ordering, continuation, and checkpoint averaging together, so its fitted equivalent is a recipe-level point estimate rather than a curriculum-only gain or universal cost law.Source: §5 · p. 24
  • Approximately $4.4K already yields an observed uniform checkpoint exceeding Qwen2-1.5B without CMA.The shifted fit reduces in-sample RMSE from 0.452 for the unshifted log-cost fit to 0.209.Source: §5 · p. 24

4.3.2 Cross-Model Reproduction Cost

  • Cost estimation prioritizes reported monetary costs or GPU-hours, distinguishing reported-resource costs from compute-derived estimates.Without a GPU-hour basis, compute is estimated as C = 6ND using the reported accelerator and MFU when available, otherwise H100-equivalent GPU-hours at 70% MFU. MobileLLM-R1 requires an inferred accelerator model; models disclosing neither resource nor token-based inputs are omitted.Source: §4.3.2 · p. 24
  • 6,009.46 Phase 1 GPU-hours and 16,504.95 Phase 2 GPU-hours underpin PuRo-2B’s cost estimate, using measured active-training iteration times and the RTX 5090 reference rental rate.Comparator GPU-hours are reported directly or precisely derived from reported statistics, then converted using reference rental rates.Source: §4.3.2 · p. 24
  • PuRo-2B lies above and left of the comparator cost–performance frontier under the stated accelerator-cost assumptions, with especially pronounced separation among fully-open comparators.This comparison uses rental-equivalent costs under the paper’s accounting protocol, not uniformly reported training bills.Source: §4.3.2 · p. 24

5 Post-Training Results and Analysis

The curriculum/CMA pretraining endpoint retains an advantage over uniform global reshuffling after focused mathematics, scaled mathematics, and broad instruction tuning. These comparisons test the complete curriculum recipe, including late constant-learning-rate continuation and six-checkpoint averaging.

5 Post-Training Results and Analysis

  • The studies compare uniform Phase 2 global reshuffling with the curriculum/CMA Phase 2 endpoint, incorporating late constant-learning-rate continuation and six-checkpoint averaging.The complete curriculum recipe retains its post-training advantage as supervised adaptation expands from focused mathematics to scaled mathematics and broad instruction tuning.Source: §5 · p. 24, §5.3 · p. 25

5.3 Broad Instruction Transfer

  • 56.58% versus 54.99% on the 15-task macro-average favors curriculum initialization after broad instruction tuning.Curriculum improves 13 of 15 component evaluations; adding IFEval, BBH, and MMLU-Pro yields 15 improvements among 18 reported evaluations. Gains span several capability families, with individual regressions reported in the appendix.Source: Figure 11 · p. 25, §5.3 · p. 25

5.2 Scaled Mathematics

  • 2.02 percentage points separate curriculum’s 76.12% from uniform’s 74.10% mean GSM8K accuracy after scaled mathematics SFT.At step 2,431, curriculum leads in every repeated run under a larger mathematical SFT mixture, a small replay component, and a longer budget. Absolute scores are not directly comparable with focused mathematics because the SFT mixture and budget differ.Source: §5.2 · p. 24, Figure 11 · p. 25

5.1 Focused Mathematics

  • 1.77 percentage points separate curriculum’s 68.66% from uniform’s 66.89% mean GSM8K accuracy after focused mathematics SFT.At step 172 without pretraining replay, curriculum’s advantage holds across repeated runs, tested SFT data orders, and the predeclared intermediate checkpoint. Most correct examples are shared, but curriculum solves more examples uniform misses; explicit-answer-only scoring preserves the mean advantage, ruling out answer formatting as its sole explanation.Source: §5.1 · p. 24, Figure 11 · p. 25

6 Related Works

PuRo-2B combines reproducible pretraining with consumer-GPU training and explicit replication-cost accounting. The related work distinguishes open weights from reconstructible recipes, and execution efficiency from the workload and hardware costs of reaching target quality.

6.1 Open-Recipe Language Models

  • Open-recipe releases expose reconstructible training pipelines, whereas open-weight releases lack the exact corpora, sample order, and complete training state needed to reconstruct original runs.Open recipes include weights, data or reconstructible data recipes, training and evaluation code, configurations, and intermediate artifacts supporting scientific inspection and reproduction [35].Source: §6.1 · p. 25
  • Open-recipe projects support controlled pretraining research, and several match or surpass comparable open-weight models on parts of the evaluation, despite substantial hardware requirements.Pythia, OLMo, OLMoE, SmolLM3, Yulan-Mini-2.4B, and Instella release data, checkpoints, code, or reproducible configurations [10] [35] [64] [52]. Yulan-Mini-2.4B used 48 A800 GPUs, SmolLM3 used 384 H100 GPUs, and Instella-3B’s first stage used 128 MI300X GPUs [52].Source: §6.1 · p. 26

6.2 Low-Cost Language Model Pretraining

  • Low-cost pretraining requires accounting jointly for target model quality, required workload, realized hardware throughput, and hardware price—not token counts or FLOPs alone.FP8 accelerates Transformer matrix multiplications while retaining higher precision where needed; DeepSeek-V3 demonstrates large-scale stability [63] [26], and Quartet II extends low precision to NVFP4 [73]. These methods reduce fixed-workload execution cost, not the token or nominal FLOP budget required for target quality.Source: §6.2 · p. 26
  • Architecture, data selection, staged training, and systems optimization reduce training workloads, but prior small-model projects generally retain a specialized data-center hardware barrier.TinyLlama, JetMoE, Yulan-Mini-2.4B, and Kaiyuan-2B explore combinations of these techniques [111] [84] [55]. Kaiyuan-2B specifically improves data and training efficiency on Ascend 910A hardware [55].Source: §6.2 · p. 26, §6.1 · p. 26
  • Consumer-GPU studies establish training feasibility or throughput benefits, rather than releasing trillion-token-scale base models with explicit replication-cost boundaries.LLMQ studies full-model pretraining on RTX 4090 GPUs using FP8, recomputation, sharding, and CPU offloading [83], while Quartet II measures NVFP4 throughput on one RTX 5090 [73]. QLoRA instead reduces fine-tuning memory requirements without changing original pretraining cost [28].Source: §6.2 · p. 26
  • More than 1.4T tokens are used to pretrain PuRo-2B from scratch with FP8 on a multi-node RTX 5090 cluster.The 2B base model’s training recipe and hardware platform are evaluated end to end, measuring model quality, accelerator usage, and replication cost within an explicit accounting boundary.Source: §6.2 · p. 26

7 Conclusion and Future Direction · B COST ASSUMPTIONS

PuRo-2B demonstrates cost-efficient pretraining on consumer RTX 5090 GPUs and shows that pretraining-recipe differences remain measurable after supervised post-training. Future work targets post-training, alternative architectures, and broader hardware and budget coverage.

7 Conclusion and Future Direction

  • About $4.4K matches Qwen2-1.5B performance, while the best PuRo-2B checkpoint approaches Qwen2.5-1.5B at a canonical accelerator-only cost of about $6.9K.The collection was trained from scratch on consumer RTX 5090 GPUs across training budgets and recipe variants. These runs underpin the Puro Cost Scaling Law relating capability to training cost under this recipe.Source: §7 · p. 26
  • 1.77 percentage points of average focused-mathematics endpoint accuracy improvement from curriculum initialization remain measurable after supervised post-training.A larger mathematics recipe yields a 2.02-point mean gain, while broad instruction tuning raises the 15-task macro-average by 1.59 points. The case study follows different Phase 2 pretraining recipes through supervised post-training.Source: §7 · p. 27
  • Released datasets, weights, intermediate checkpoints, training configurations, and implementation provide an experimental foundation for studying interactions across the full model-development pipeline.The inspectable pipeline enables questions unavailable from weights alone, including how data, optimization, training schedules, and post-training interact.Source: §7 · p. 27
  • Future work will extend the reproducibility recipe beyond pretraining into post-training, emphasizing the development and study of agentic capabilities.Source: §7 · p. 27
  • Planned architectural extensions move beyond standard dense Transformers to looped Transformers, linear-attention models, mixture-of-experts architectures, and other emerging designs.Source: §7 · p. 27
  • Planned hardware recipes extend beyond RTX 5090 GPUs and span wider training budgets to study how optimal system and training choices change across hardware and scale.Source: §7 · p. 27

A Limitations · B Reproduction-Cost Assumptions

Puro-2B’s claims are qualified by unresolved contamination, limited Chinese evaluation, overtraining, and a fixed-size scaling law. Reproduction-cost comparisons follow an evidence hierarchy, with token-based estimates generally treated as lower bounds.

A Limitations

  • Curriculum gains remain potentially confounded by benchmark contamination because no strict corpus-wide exact- or near-duplicate audit was performed.Most training data comes from already processed or filtered open datasets. Curriculum advantages persist after post-training, but an audit is still needed to determine whether benchmark overlap contributes.Source: A Limitations · p. 38
  • Chinese data provides basic coverage, but Chinese capability was not a primary optimization or headline model-selection target.Release-facing comparison tables omit Chinese benchmarks while retaining the Chinese proxy axis for mixture audits; headline evaluation covers mathematics, code, reasoning, and knowledge. Future versions should specify language targets before tuning data and evaluation.Source: A Limitations · p. 38
  • Approximately 1.4T tokens train the dense 2B-parameter model at about 700 tokens per parameter, an overtrained regime not presented as compute-optimal.This regime supports recipe studies and a compact model with low inference cost. The operating point balances RTX 5090 memory and communication constraints, benchmark quality, and community support for a dense base model.Source: A Limitations · p. 38
  • The Puro Cost Scaling Law is a fixed-2B, recipe-specific scale-down curve, not a model-size scale-up law.Larger models or world sizes require redesigned communication and memory arrangements, potentially including vocabulary or embedding/LM-head partition changes. RTX 5090 HBM capacity can bottleneck model-size scaling.Source: A Limitations · p. 38

B Reproduction-Cost Assumptions

  • $6.9K is the canonical curriculum endpoint’s cost, while $4.4K is the uniform endpoint’s cost in the cost–performance figure.Table 7 rounds GPU-hours and costs to the nearest unit for presentation; Figure 1 uses unrounded coordinates.Source: B Reproduction-Cost Assumptions · p. 38, Table 7 · p. 39
  • 15 benchmark percentage scores determine performance through an unweighted arithmetic mean, with every score required for a plotted model.Benchmarks are not weighted by their numbers of examples.Source: B Reproduction-Cost Assumptions · p. 39, Table 7 · p. 39
  • Reproduction costs prioritize reported monetary costs, then reported GPU-hours, then GPU-hours calculated from accelerator count multiplied by elapsed training time.GPU-hours are multiplied by reference accelerator rental rates in RMB. The ledger retains RMB costs, while Figure 1 converts them to USD by dividing by 6.8067.Source: B Reproduction-Cost Assumptions · p. 39
  • 989.5 TFLOP/s H100-equivalent throughput and MFU 0.70 underpin token-only cost estimates, generally interpreted as reproduction-cost lower bounds.When monetary costs and accelerator-hours are unavailable, training compute is estimated using C = 6ND [44] and converted to GPU-hours and rental costs. Model-specific token counts are preferred, family-level estimates are marked, and models lacking both accelerator usage and usable token counts are omitted.Source: B Reproduction-Cost Assumptions · p. 39

C TRAINING DETAILS

Training-cost accounting uses measured active-training time for Puro-2B and explicit hardware-pricing assumptions for comparison models. The comparison frontier excludes Puro-2B and retains only nondominated cost–performance points.

C TRAINING DETAILS

  • $6,891 is Puro-2B’s training cost, calculated from measured active-training time rather than 6ND.Its 22,514.41 GPU-hours are priced at the local RTX 5090 rate of 2.0833 RMB/GPU-hour, totaling 46,905.03 RMB. The comparison frontier excludes Puro-2B, keeps the highest score at tied costs, and connects points whose scores strictly exceed every cheaper model’s score.Source: C TRAINING DETAILS · p. 40
  • 30,720–43,008 accelerator-hours are estimated for MobileLLM-R1, priced at the A100 80GB rate, with the midpoint used as its plotted coordinate.The estimate combines two 4–5-day pretraining phases and two 1–2-day mid-training phases across 128 GPUs. Because GPU models are unspecified, the hardware assumption follows the original MobileLLM training-cost report.Source: C TRAINING DETAILS · p. 40
  • 51.57% MFU and the 312-TFLOP/s A800 BF16 peak underpin Yulan-Mini-2.4B’s GPU-hour estimate using Equation (8).GPU-hours are priced using the unified A100 80GB rental rate because public A800 rental prices are rarely published.Source: C TRAINING DETAILS · p. 40

C Production Training Details

The canonical PuRo-2B production run trains to 1.4T tokens in two phases, scaling from 24 to 96 GPUs. It combines mixed-precision computation, optimizer-specific learning rates, and phase-specific schedules, ending with curriculum model averaging.

Production run setup

  • 4,096-token sequences, global batch size 1,536, and micro-batch size 2 remain fixed across both production phases.Selected matrix weights use MuonH with zero weight decay and learning rate 10 times the base rate. Remaining parameters use AdamW with weight decay 0.1.Source: C Production Training Details · p. 40

Phase 1 learning rate schedule

  • 1,000 warmup steps precede Phase 1’s power decay from base learning rate 5 × 10^-3 to 1.04 × 10^-3.The power schedule [85] uses p = 1/2 and τ = 1,000 steps, with an asymptotic floor of 5 × 10^-4 supporting continual training. Warmup covers 1,536,000 samples.Source: C Production Training Details · p. 40

Production run setup

  • 1.4T cumulative tokens produce validation loss 2.488 after Phase 2, following Phase 1’s validation loss 2.730 at 438.8B tokens.Phase 1 uses 24 GPUs with median throughput 238 TFLOP/s per GPU; Phase 2 consumes 960.0B tokens using 96 GPUs at 192 TFLOP/s per GPU. The Phase 2 materialized pool includes the early transition range, while Phase 1 replay is an additional consumed stream.Source: C Production Training Details · p. 40, Figure 12 · p. 41

Phase 2 learning rate schedules

  • 1.04 × 10^-3 initializes every Phase 2 schedule at the shared Phase 1 checkpoint, with linear decay toward 10^-5.The five decay horizons span approximately 60.1, 120.1, 240.2, 480.5, and 960.9B tokens, with the local schedule coordinate reset at the checkpoint. The canonical run uses curriculum model averaging, replacing decay with constant-LR training from step 218,000 for the final 29B tokens.Source: C Production Training Details · p. 41

FP8 implementation pipeline

  • Blockwise E4M3 FP8 handles linear-layer GEMMs, while hidden/residual flows, LayerNorm, and embeddings use BF16; gradients, optimizer-state accumulation, and softmax remain FP32.FP8 GEMMs cover forward propagation, input-gradient computation, and weight-gradient computation in both production phases.Source: C Production Training Details · pp. 40–41

D MuonH Scaling-Ladder Analysis

Across a five-model scaling ladder, MuonH shows a fitted compute-efficiency advantage over tuned Muon, while blockwise FP8 trades slightly higher validation loss for higher throughput. Estimated production-scale savings depend on extrapolating ladder results.

Experimental setup

  • 20 tokens per scaling parameter are used to train five models from 0.17B to 1.7B from scratch, following the compute-optimal ratio reported in prior work.Runs share data, tokenizer, 4,096-token sequences, batch size 512, validation set, and decay schedule. Tuned FP8 MuonH–Muon comparisons test complete recipes rather than hyperball alone, whereas FP8–BF16 MuonH comparisons use identical optimizer hyperparameters.Source: D MuonH Scaling-Ladder Analysis · p. 42

MuonH Scaling-Ladder Analysis

  • 1.19× fitted compute-equivalent efficiency for MuonH versus tuned blockwise-FP8 Muon implies an estimated 16.1% compute saving for a 2B/1.4T-token run.Leave-one-size-out estimates range from 1.17–1.28×, measuring sensitivity to ladder points rather than functional-form uncertainty. The production-scale estimate assumes the horizontal efficiency shift persists beyond 20 tokens per scaling parameter; it is not an absolute loss prediction.Source: D MuonH Scaling-Ladder Analysis · pp. 42–43

Precision and throughput decomposition

  • 1.34× quality-adjusted FP8 speedup corresponds to an estimated 25.2% reduction in GPU-hours at matched quality.This estimate combines the five-size precision-quality fit with the measured 1.36× median throughput increase at 1.7B. That largest-model throughput ratio serves as a proxy for the 2B production setting, rather than a directly measured production speedup.Source: D MuonH Scaling-Ladder Analysis · p. 44
  • 0.0031–0.0039 higher validation loss for FP8 than BF16 at fixed theoretical compute accompanies fitted effective-compute retention of 98.0% across the ladder.Leave-one-size-out retention estimates range from 97.6–98.1%. The central fit corresponds to a 2.0% nominal-compute penalty at matched quality.Source: D MuonH Scaling-Ladder Analysis · p. 44

E Post-Training Details · F Learning Rate Schedule Diagnostics

Curriculum initialization improves repeated mathematics post-training results and the broad instruction aggregate relative to uniform initialization, although transfer gains are not universal. The learning-rate schedule appendix introduces additional diagnostics and fitting results.

E Post-Training Details

  • 1.77 percentage points is the mean curriculum-over-uniform GSM8K endpoint improvement in focused mathematics, with endpoint advantages ranging from 1.36 to 2.12 points.All six matched comparisons across three repetitions and predefined steps 100 and 172 favor curriculum initialization, providing the experiment’s primary uncertainty check.Source: E Post-Training Details · p. 45, Table 12 · p. 46
  • Matched curriculum and uniform arms differ only in pretraining initialization within each setting; the three settings are complementary probes, not directly comparable training curves.Focused mathematics uses 191,767 packed records, 41.4M assistant-supervised tokens, and no replay; scaled mathematics uses 2,014,933 conversations, including 99,989 replay records, over 2,431 optimizer steps. Broad instruction uses 264,612 packed records and 82.2M supervised tokens, with its paired comparison at step 300.Source: E Post-Training Details · pp. 45–46
  • 1,319 GSM8K test examples are scored using zero-shot ChatML prompting, greedy BF16 generation, a 512-token generation limit, and normalized exact numeric equivalence.The extractor truncates continuations beginning a new user turn and prioritizes explicit final-answer phrases before extracting answers according to the established format.Source: E Post-Training Details · p. 46
  • Curriculum-only correct answers outnumber uniform-only correct answers in every focused mathematics comparison, showing that gains are not solely an artifact of averaging conflicting runs.Correct-answer overlap is measured by intersection-over-union and asymmetric success counts; manual inspection also distinguishes numerical correctness from termination and parsing behavior.Source: E Post-Training Details · p. 46, Table 12 · p. 46
  • 2.02 percentage points is the mean curriculum-over-uniform GSM8K endpoint improvement in scaled mathematics, with curriculum ahead in all three repetitions.Observed endpoint improvements range from 1.21 to 3.26 percentage points after the longer, more diverse SFT schedule.Source: Table 13 · p. 46, E Post-Training Details · p. 46
  • 1.59 percentage points separates curriculum from uniform initialization on the broad instruction macro-average, with curriculum higher on 13 of 15 component evaluations.The primary aggregate weights task scores equally, preventing larger benchmarks from dominating, and is calculated from unrounded scores.Source: Table 14 · p. 47, E Post-Training Details · p. 47
  • 15 of 18 reported evaluations favor curriculum initialization, but the additional transfer results are not uniformly positive.Curriculum improves IFEval and MMLU-Pro while BBH decreases modestly. These three diagnostics use task-specific prompts and scorers and are excluded from the primary 15-task macro-average.Source: E Post-Training Details · p. 47, Table 15 · p. 47

F Learning Rate Schedule Diagnostics

  • The learning-rate schedule appendix introduces more detailed diagnostics and fitting results.The supplied passage states this scope but provides no specific diagnostic findings or fitted values.Source: F Learning Rate Schedule Diagnostics · p. 47

F.1 Multi-Power Law for Effective Learning Rate Schedules

Adapting Multi-Power Law (MPL) [57] to effective learning rates lowers mean held-out prediction error in a two-run FC2 diagnostic. The model combines cumulative-exposure progress with delayed loss reductions from learning-rate decay.

F.1 Multi-Power Law for Effective Learning Rate Schedules

  • MPL [57] is adapted to a selected weight group by separating cumulative-exposure training progress from learning-rate schedule-shape corrections.The signal q_t can be either optimizer LR η_t or effective LR ρ_t, with trapezoidal post-warmup exposure. A Chinchilla-like power law uses cumulative exposure, including warmup, as an optimization-time coordinate but cannot alone distinguish different decay patterns.Source: §F.1 · pp. 47–48
  • Learning-rate reductions contribute delayed negative loss corrections, superposed across preceding changes as subsequent exposure realizes each reduction’s benefit.Post-change exposure is rescaled by the post-change signal level, and the unrealized response fraction follows 1 − G(x) = (1 + x)^-β.Source: §F.1 · p. 48
  • 20% of final validation targets are held out when fitting post-warmup curves from two BF16 ordinary-Muon runs.Each LR signal is normalized by its own post-warmup peak, which only rescales fitted constants. Fitting follows three stages: power-only, power plus a schedule-drop term, and the full response model.Source: §F.1 · p. 48
  • 0.0210 versus 0.0265 mean held-out RMSE favors effective LR over ordinary LR in the two-run FC2 diagnostic.Base Muon shows the largest improvement: 0.0270 for effective LR versus 0.0422 for ordinary LR. Aligned Muon is comparably predictive under both representations: 0.0149 for effective LR and 0.0108 for ordinary LR.Source: §F.1 · p. 48
  • 0.0207 mean held-out RMSE from the simplified effective-LR model approximately matches the full effective-LR MPL fit on these two curves.The simplified model yields 0.0270 for base Muon and 0.0144 for aligned Muon. Assuming G(x) = 1 collapses the correction to the net decrease since warmup, discarding reduction timing; this is an interpretive approximation, not instantaneous training behavior.Source: §F.1 · p. 48
  • The diagnostic attributes MuonH and aligned Muon’s eventual advantage over ordinary Muon to preserving effective-LR decay potential for late training.Ordinary Muon decays its effective LR aggressively early, whereas MuonH and aligned Muon use linear decay.Source: §F.1 · p. 48

F.2 WSD Sweeps and Limited-Compute Schedule Estimation … H.4 Scope of the Available Comparison

Two-anchor MPL estimates WSD decay trends as a screening diagnostic, while the appendices document component-local curricula and the released continuation-and-averaging configuration. Available comparisons support the joint configuration rather than isolated gains from ordering, continuation, or averaging.

F.2 WSD Sweeps and Limited-Compute Schedule Estimation

  • Formal MPL fits anchor curves in three stages to screen schedule choices, but broad plateaus, single seeds, and noisy minima prevent optimum certification.It first fits baseline and power-law parameters on the first 25% of each curve, then adds the instantaneous LR-drop coefficient on complete curves before jointly refining parameters. These sweeps support long-decay features, not the exact Phase 1 power exponent.Source: §F.2 · pp. 49–50
  • 0.0157 mean curve RMSE and 0.0067 endpoint MAE characterize two-anchor MPL predictions across 15 unseen WSD schedules.Each fixed-peak TPP20 fit uses decay ratios 0.2 and 1.0, withholding 0.4, 0.6, and 0.8. Estimated optimal ratios rise through 0.33, 0.57, 0.67, 0.75, and 0.85 as effective peaks increase from 0.008 to 0.024, although observed grid minima remain noisy.Source: Figure 17 · p. 49, §F.2 · pp. 49–50

F.3 Hyperball / G SCALING CHECKPOINT LEDGER

  • 3.0678 validation loss after 5,000 steps demonstrates a numerically stable ordinary-Muon run with a hill-like scalar LR maintaining the prescribed effective-LR schedule.During the stable effective-LR interval at 0.02, growing matrix weight norms and roughly constant orthogonalized-update norms require increasing scalar LR before terminal decay. This constructed example demonstrates feasibility, not optimality or a universal requirement, and shows why scalar LR needs effective-LR context.Source: Figure 19 · p. 50, §F.3 · p. 50, G SCALING CHECKPOINT LEDGER · p. 51

G Puro-2B Scaling Checkpoint Ledger

  • The checkpoint ledger fixes scaling-point lineage and cost accounting, while the score sheet defines Avg15 as the unweighted mean of 15 benchmark percentages.The ledger covers five uniform scaling points, the curriculum decay endpoint, and the canonical averaged model, reporting total active-training GPU-hours and costs at the normalized RTX 5090 rate. Phase 1 is shared, Phase 2 consumption avoids double-counting its early transition range, and the canonical score-sheet row represents the released six-checkpoint average.Source: G Puro-2B Scaling Checkpoint Ledger · p. 51, Table 16 · p. 52, Table 17 · p. 52

H Curriculum Construction and Model Averaging / H.1 Scalable Construction of Component-Local Curriculum Buckets

  • 376 production buckets align component-local token percentiles, approximately preserving the component mixture without imposing a global cross-source quality ranking.Scored components use up to 256 approximate-quantile score intervals, each subdivided into 64 deterministic hash shards; unscored components use 4096 deterministic random shards without a quality interpretation. Token-mass accounting avoids document-level global sorting and produces approximately 2.5B-token buckets, with scores used only within their own component.Source: §H.1 · pp. 53–54, H Curriculum Construction and Model… · p. 53

H.2 Phase Transition and Uniform-Order Control

  • Approximately 43.9B transition tokens combine Phase 1 replay with the earliest Phase 2 curriculum range; uniform ordering globally reshuffles the same Phase 2 pool.The transition contains approximately 21.9B replay tokens and 21.9B tokens from the earliest 2.34% of Phase 2; only replay adds to the component-pool total. Streams are materialized before training, and exact reproduction requires manifests, source revisions, and checksums—not random seeds alone.Source: §H.2 · pp. 54–55

H.3 Constant-LR Continuation and Checkpoint Averaging

  • The released model equally averages six checkpoints from a constant-LR continuation resumed at step 218,000, rather than exporting the final unaveraged iterate.The continuation fixes the base coefficient at 4.08 × 10^-5 and the MuonH matrix-group effective LR at 4.08 × 10^-4; its averaging window spans 469 steps, approximately 2.95B tokens, and excludes optimizer states. Available continuation entry points were not matched ablations, so step 218,000 is not established as optimal.Source: Table 16 · p. 52, §H.3 · p. 55

H.4 Scope of the Available Comparison

  • Available comparisons evaluate the joint curriculum-and-averaging configuration, not isolated compute-equivalent gains from ordering, continuation, or averaging.Component attribution requires crossing curriculum versus uniform ordering with averaging enabled versus disabled under the same continuation learning rate. Continuation attribution additionally requires matched rate or branch-point experiments with fixed data, transition, budget, and evaluation, plus repeated seeds; downstream endpoint comparisons use matched post-training.Source: §H.4 · p. 55

I Data Recipe Details … I.3 Proxy-to-Mixture Decision Audit

The data audit distinguishes processed token exposure from unique content and documents redistribution limits. Proxy measurements inform capability comparisons, but PCA is descriptive and proxy-to-mixture correspondence is an ex post consistency audit rather than a validated selection rule.

I Data Recipe Details

  • Component token counts measure sampled, tokenized exposure actually used by training, not upstream corpus sizes or unique content.Counts exclude Phase 2 transition and replay; overlapping configurations include original, decontaminated, short, and long Codeforces-CoTs variants and potentially thinking/non-thinking Jupyter Agent presentations. MegaMath-Code is accounted under Code despite its broader math scheduling label.Source: I Data Recipe Details · pp. 58–59
  • Dataset disclosures document verified upstream terms without treating missing licenses as redistribution permission; some components permit training but prohibit raw-data redistribution.Nemotron-CC-v2 and Nemotron-Pretraining-Code-v1’s Synthetic-Code partition use NVIDIA’s training-only data agreement, so the release provides reconstruction identifiers and accounting rather than claiming every component is redistributable. The merged Chinese web component inherits its source families’ respective terms [94] [96] [38] [39] [114] [115].Source: I Data Recipe Details · p. 59

I.1 Proxy-Measurement Protocol

  • Approximately 8.4B continuation tokens test each candidate using Qwen3-0.6B initialized from the same 86B-token checkpoint.Each continuation uses 2,000 steps, sequence length 4,096, and global batch size 1,024; candidate share ramps from 0% to 80% over 1,600 steps, then stays fixed for 400 steps. Final-checkpoint evaluation groups GSM8K/MATH as Math, MBPP/HumanEval as Code, C-Eval/CMMLU as Chinese, and remaining knowledge and commonsense tasks as General; measurements remain conditional on the proxy scale, schedule, candidate share, and suite.Source: §I.1 · p. 59
  • 5B tokens is the minimum source size; eligible sources receive adaptive samples or score-quantile probes according to size and scoring availability.Unscored sources and scored sources containing 5B–50B tokens receive one adaptive sample targeting approximately 4B tokens; larger scored sources are probed at the 0th, 25th, 50th, and 75th score quantiles. Adaptive samples below 5B tokens trigger range or sampling-fraction expansion, with at most five attempts.Source: §I.1 · p. 59

I.2 Proxy Capability Aggregation and PCA Audit

  • 34.5% and 28.1% of standardized four-axis variance are explained by PC1 and PC2, primarily contrasting General with Code and Math with Chinese, respectively.Across 43 conditions and 15 benchmarks, task-level population z-scores are averaged into four axes, reconstructed within 10^-4 of exported values, then re-standardized for correlation PCA. General–Math, Code–General, and Math–Chinese correlations are 0.136, −0.337, and −0.098, respectively.Source: §I.2 · p. 59
  • PCA describes capability conflicts, not optimization targets, causal gains, or a universal capability ordering; recipe comparisons use the original aggregate axes.Component signs are arbitrary, and maximizing variance does not define utility.Source: §I.2 · pp. 59–60, Figure 20 · p. 60

I.3 Proxy-to-Mixture Decision Audit

  • DCLM q00, MegaMath-Web-Pro, MegaMath-Code, and FineWeb-Edu-CN q00 lead General, Math, Code, and Chinese, respectively, matching Phase 2 family additions or retention.This is an ex post consistency audit, not a selection rule: production DCLM and FineWeb-Edu-CN ranges differ from proxy slices. Neither the axis leaders nor the production mixture have repeated-seed uncertainty.Source: §I.3 · p. 61
  • 2.884 Math and 0.667 General aggregates place MegaMath-Web-Pro above the three measured Phase 1 mathematics comparators under the fixed proxy protocol.Math/General aggregates are 2.821/0.345 for Nemotron-CC-Math 4+, 2.746/−0.387 for SwallowMath, and 1.269/−0.317 for FineMath. Phase 2 adds 13.5B MegaMath-Web-Pro tokens, removes FineMath, and reduces Nemotron-CC-Math 4+ and SwallowMath total-mixture shares from 2.62% to 0.26% and 2.28% to 1.42%, respectively.Source: Table 22 · p. 61, §I.3 · p. 61

Related papers

Loading 2608.27370v1…