PaperlayerView exact revision on arXiv

Source-linked AI summary

OPEN-1B: A Fully Auditable Training Run

John Donaghy, Brian Wilcox, Oğuzhan Ersoy, Shikhar Rastogi, Adam St Arnaud, Alexey Titov, Jordan Greenberg, Ben Fielding, Harry Grieve

arXiv:2609.17380v1cs.LG

TL;DR

Open-source training recipes do not guarantee that a run can be reproduced bit for bit across hardware, limiting verification of how a model was produced. Open-1B standardizes operations and data ordering so individual training steps can be replayed and checked against the published trajectory. The authors report bitwise reproducibility across the training trajectory, while noting slower quantization-aware training and weaker downstream benchmarks than OLMo 2 1B.

  • Problem

    Open-source weights, data, and recipes do not ensure exact cross-hardware reproduction, so a released model’s training trajectory cannot be fully verified.

    Source: §1 · p. 2, §7 · p. 22

  • Method

    Open-1B combines bitwise-reproducible operations, topology-independent data ordering, per-step hashes, and an audit harness for replaying individual training steps.

    Source: §1 · p. 3, §2.3 · p. 6, §7 · pp. 21–22

  • Results

    Open-1B’s training trajectory is bitwise reproducible from initialization to the final checkpoint, with per-step hashes enabling independent attestation of the final outcome.

    Source: §7 · p. 22

  • Takeaways & Limitations

    The authors frame Open-1B as an early step toward AI development where claims about how a model was built can be verified rather than taken on trust.

    Source: §7 · p. 22

    Native int8 quantization-aware training makes tensor-core GEMMs several times slower than a stock PyTorch bf16 baseline, and resource constraints led to an order of magnitude fewer training tokens than OLMo 2 1B’s 4T-token budget, with Open-1B trailing on downstream benchmarks.

    Source: §7 · p. 22

Abstract

from arXiv · show

Open-source language models have a reproducibility problem. Despite releasing weights, training data, and recipes, none of them are provably reproducible due to the non-associativity of floating-point arithmetic. Deep learning frameworks often offer a deterministic execution mode, allowing reproducible operations on the same machines. Unfortunately, this determinism does not carry across hardware such that a user can verify that a released checkpoint was actually produced using the declared training recipe. This leaves room for undisclosed data, injected biases, or backdoors that existing techniques such as proof-of-learning or proof-of-training-data cannot rule out. We introduce a new tier of model transparency, fully auditable, in which every operation on every data sample during training is independently reproducible on heterogeneous commodity hardware with bitwise certainty. By imposing a definite order on the sources of training nondeterminism, GPU kernel reductions, data batch ordering across a data-parallel cluster, and inter/intra-node collective communication, we make it possible to replay any individual step of a large, distributed training run on a single piece of commodity hardware and check it against the published trajectory. Because replaying an entire run on one machine is infeasible, we support this with a collective verification scheme in which many independent auditors each certify individual steps, together covering the whole run. We release Open-1B, a model trained under this regime, together with its full pretraining dataset, every intermediate checkpoint, the training codebase, and the audit harness needed to reproduce and verify any step of its training.

1. Introduction

Open models release weights, data, and recipes, but floating-point nonassociativity prevents exact cross-hardware reproduction, leaving their training steps unverifiable. Open-1B introduces bitwise-auditable training and a collective audit scheme, alongside released training artifacts and an audit harness.

1. Introduction

  • Open-source models release weights, data, and recipes, while open-weight models generally release only final checkpoint weights and may keep training data or methods opaque.The passage describes closed models as controlled by their creators, with API access but no access to their architecture or training trajectory.Source: §1 · pp. 2–3
  • Open-source recipes do not ensure reproducibility: floating-point nonassociativity can produce different weights, preventing exact verification of training steps.Existing heuristic verification methods provide only probabilistic verification and may fail to detect a backdoor introduced with a few data points.Source: §1 · p. 3
  • Open-1B aims to make every operation on every training sample verifiable, with an audit harness for checking individual steps bitwise against the published run.The authors state that the entire run results in a single hash proving it was trained as declared.Source: §1 · p. 3
  • Collective verification distributes audits across users, each certifying individual steps because verifying the entire run on commodity hardware is infeasible.Together, these step certifications can establish the run’s collective final state.Source: §1 · p. 3
  • Open-1B releases its pretraining dataset, intermediate checkpoints every 100 steps, training and evaluation code, and a harness supporting NVIDIA GPUs, x86 CPUs, ARM CPUs, and Apple Silicon via Metal.The contributions also include RepOps, a topologically invariant data loader, deterministic collective communications, and an int8 quantization-aware pretraining recipe adapted from Team OLMo et al. (2024).Source: §1 · pp. 3–4

2. Reproducibility Across Heterogeneous Hardware

Bitwise reproducibility requires identical results across heterogeneous hardware, not merely repeatability on one machine. The section describes how RepOps standardizes numerical operations and randomness, while a canonical data stream preserves the training sequence across cluster topologies and single-device replay.

2. Reproducibility Across Heterogeneous Hardware

  • Bitwise reproducibility requires exactly matching results across different hardware, whereas determinism means repeatability on the same machine and environment.The definition requires exact bitwise agreement, not merely agreement within one unit in the last place.Source: §2 · p. 4

2.1. Reproducible Operations

  • RepOps fixes reduction order across hardware and forces one fused multiply-add convention, making rounding sequences consistent at some speed cost.It disables processor fusion and uses explicitly unfused multiply-and-add operations on GPUs.Source: §2.1 · pp. 4–5
  • RepOps makes subnormal handling consistent by matching Apple GPUs’ flush-to-zero and denormals-are-zero behavior on processors and NVIDIA GPUs.The passage describes this as adopting the behavior of the least capable backend deliberately.Source: §2.1 · p. 5
  • Counter-based randomness computes each value from the seed and fixed logical index, avoiding platform- and workload-dependent generator state and lane assignments.This makes the random stream independent of how work is split, tensor size, or which GPU executes it.Source: §2.1 · p. 5

2.2. RepOps for Neural Network Training

  • Every operation from initialization through convergence must be bitwise reproducible for a training trajectory to be verifiable across hardware.RepOps also provides abstractions for common optimizers, loss functions, backward functions, and neural-network layers.Source: §2.2 · p. 6

2.3. The Topologically Invariant Reproducible Data Stream

  • Open-1B defines a canonical stream of packed windows independent of world size, rank, or node count, so training, resuming, and single-device replay enumerate the same data.The design supports re-sharding after failures or resource reallocation without perturbing the data sequence, and lets an auditor reconstruct a step’s ordered micro-batches.Source: §2.3 · p. 6
  • The stream deterministically interleaves sources and permutes shards and documents using seeded RNGs, then slices 4,097-token windows for training.Data-parallel ranks own windows by index modulo the number of ranks; batch size and gradient accumulation must be selected so workers do not become idle.Source: §2.3 · pp. 6–7

3. Open-1B

Open-1B pairs a reproducible, low-precision model design with a 400B-token training recipe and curated data mix. The section reports held-out loss convergence and describes gradient and weight-norm behavior during the run.

3.1. Model Architecture

  • Open-1B uses a decoder-only transformer with gain-free QK-norm, embedding normalization, and hybrid 512-token sliding-window attention, alongside int8 training and bitwise-reproducible operators.Every fifth layer and the final layer use full causal attention; the embedding and language-modeling head remain in full precision.Source: §3.1 · pp. 7–8

3.2. Training Recipe

  • The 400B-token recipe uses AdamW, global-norm clipping at 1.0, phased batch-size increases, and a token-anchored cosine learning-rate decay.Training uses a 4,096-token sequence length across 48 H100 GPUs.Source: §3.2 · pp. 8–9

3.3.1. Pretraining Data: Open-1B Mix 0626

  • Open-1B Mix 0626 combines four permissively licensed public sources into an approximately 450B-token pretraining set, with roughly 80% from web data.Sources are consumed at the same rate, completing 0.89 epochs in the 400B-token budget; the stream order is seed-determined and independent of cluster topology.Source: §3.3.1 · pp. 10–11

3.3.2. Base Evaluations

  • Base evaluation follows the OLMES protocol, reporting the best MCF or CF formulation where applicable and an OLMES-standard 10-task macro.MMLU and held-out multiple-choice evaluations use MCF only; the OLMo 2 1B macro is computed with the same pinned harness because five task results were unpublished.Source: §3.3.2 · p. 11, Table 6 · p. 11

3.4. Convergence Deep Dive

  • The run consumed 400.0B tokens over 80,957 optimizer steps and was completed in seven contiguous segments, with hash comparisons validating repeated restart steps.Each segment restarted from the latest historical checkpoint.Source: §3.4 · p. 11, Table 7 · p. 11

3.4.1. Loss convergence

  • Held-out mixture cross-entropy fell monotonically from 6.59 to 2.580, while training-loss fits showed structured residuals attributed largely to data-mix heterogeneity.The authors therefore base convergence statements on a held-out probe rather than the training curve.Source: §3.4.1 · pp. 11–12, Figure 1 · p. 12

3.4.2. Gradient behavior, Clipping, and Spike Protocol

  • Only 0.16% of steady-state steps engaged clipping, and just two batches triggered the skip protocol, accounting for ten skipped optimizer steps.The two pre-clip norms were 5.78 at step 1,782 and 6.88 at step 43,737; clipping events were isolated with immediate recovery.Source: §3.4.2 · p. 12, Figure 2 · p. 13, Table 8 · p. 13

3.4.3. Weight-norm evolution

  • Query/key projection norms stop growing near 120–180B tokens as weight decay dominates, while value projections continue growing and the input embedding saturates near 250B tokens.The unembedding peaks near 100B tokens and then declines under weight decay.Source: §3.4.3 · p. 13, Figure 3 · p. 14

4. Stability Deep Dive

The section identifies two distinct sources of training instability: an int8 attention-backward issue behind gradient creep and quantizer step-size drift during pretraining. It describes fixes and compares the int8 run with a bf16 baseline.

4.1. Gradient Creep

  • A zero query-branch gradient in int8 backward attention drove QK-norm gain upward; scale-aware gradient kernels and removing the gain addressed gradient creep.The upward gradient drift began around 116B tokens; the original drift is absent after the fixes.Source: §4.1 · p. 14, Figure 5 · p. 16

4.2. Quantizer Step-Size Drift

  • LSQ step-size drift can reinforce itself: AdamW makes update sizes primarily learning-rate-driven, while collapsed quantization levels feed rounding noise back through the straight-through estimator.The authors note LSQ is normally used for short fine-tuning on approximately stationary weights, unlike pretraining.Source: §4.2 · p. 15
  • Per-step recomputation of quantizer scales kept signal-to-noise high through the main phase, with infrequent gradient clipping.Recomputation cost approximately 2% of step time, compared with approximately 70% for disabling quantization.Source: §4.2 · p. 15
  • Recomputing each per-channel step size from current weights using the LSQ initialization formula prevents it from diverging from the weights it quantizes.The formula is 2 · mean(|w|)/√q_max, evaluated per output channel, at every step.Source: §4.2 · pp. 14–15
  • After about 2,000 steps, the first block’s gradient norm exceeded 300 while other blocks trained normally; substituting bf16 linear layers identified the weight quantizer as the cause.The integer attention kernel, sliding-window pattern, deterministic reduction order, sampler, and learning rate were ruled out by direct substitution.Source: §4.2 · p. 15, Figure 6 · p. 16
  • After 400B tokens, int8 and bf16 cross-entropy differed by approximately 0.2 nats; quantization also naturally produces a larger z-loss divergence.The authors chose int8 because reproducible-bf16 is approximately 1.6× slower than reproducible-int8.Source: §4.3 · p. 17

5. The Cost of Reproducibility

Reproducible training incurs runtime and scaling costs, particularly from state hashing and cross-node communication. The full Open-1B run reached 71% strong-scaling efficiency with hashing enabled, while optimized PyTorch remained faster.

Runtime Overhead

  • The run used six GCP nodes for 29.5 days, with MFU around 5%—an order of magnitude below optimized non-reproducible kernels.The authors characterize this as a trade-off of compute performance for reproducibility.Source: §5 · p. 17

Deterministic All-Reduce

  • The deterministic all-reduce fixes reduction order with recursive-doubling butterfly hops, but non-power-of-two node counts add fold-and-reconciliation communication.The six-node jump reflects serialized hops in the non-power-of-two fold, not bandwidth loss.Source: §5 · p. 18, Figure 8 · p. 18

Strong Scaling

  • 71% strong-scaling efficiency: with hashing every step, throughput rose from 40.0k tokens/s on one node to 169.4k on six.Without hashing, six-node throughput reached 217.4k tokens/s at 86% efficiency; forward-plus-backward operations had 99.7% efficiency.Source: §5 · p. 18, Figure 9 · p. 19, Table 11 · p. 20

Summary

  • State hashing is the largest stated optimization target: host-side hashing adds approximately 6 seconds per step, which on-device hashing could nearly eliminate.The full-stack scaling result attributes efficiency loss to state hashing and cross-node replicate reduction.Source: §5 · pp. 18–19

Runtime Overhead

  • On one node, int8 QAT ran 6.8× slower than stock PyTorch; toward six nodes, the reproducible runtime’s overhead narrowed to approximately 5.0×.The int8 QAT path ran approximately 1.8× faster than the repop bf16 path; the repop arms reached 91%/86% scaling efficiency versus 66% for PyTorch.Source: §5 · p. 19, Table 10 · p. 19, Figure 10 · p. 20

6. Audit

The audit harness replays a distributed training step on one device by imposing a fixed order on kernel reductions, data batches, and collective communications. This enables bitwise checks of individual operations and steps, while collective verification lets many auditors cover a run that one commodity machine cannot replay in full.

6. Audit

  • Bitwise-reproducible auditing requires a definite order for GPU kernel reductions, data batching, and inter- and intra-node reduce-scatter and all-reduce communications.The order matters because floating-point operations are non-associative.Source: §6 · p. 20
  • The audit harness sequentializes cluster work so users can replay each training step’s forward and backward passes, track gradients and communication reductions, and apply the optimizer on one device.The stated goal is to match the published trajectory with bitwise accuracy and thereby verify how the model was trained.Source: §6 · pp. 20–21
  • The audit replay organizes gradient reduction across microbatches, shards within a node, and replicas across nodes, with data-parallel world size equal to dp_replicate × dp_shard.A replica is a node, while a shard is the data-parallel work unit within the FSDP grouping.Source: §6 · p. 21
  • The replay uses a running sum in ascending shard order for reduce-scatter and TreeFold to reproduce the cluster’s recursive-doubling all-reduce pairing order.The fixed order is intended to match the cluster reductions bit for bit; the full reduction mechanism is deferred to Appendix A.Source: §6 · p. 21

7. Conclusion

Open-1B is presented as the first fully auditable open-source LLM, with its training infrastructure and materials released alongside it. Its training trajectory is bitwise reproducible, but the overhead and resource constraints limited its token budget and it trails on downstream benchmarks.

7. Conclusion

  • Open-1B is presented as the first fully auditable open-source LLM, supported by RepOps, a topologically invariant data loader, and deterministic collective communications.The infrastructure supports replay across CPU, NVIDIA GPU, and Apple GPU hardware, including sequential replay of communications on one device.Source: §7 · p. 21
  • Open-1B’s training trajectory, from initialization to the final checkpoint, is bitwise reproducible, with per-step hashes for independent attestation.The report releases the full pretraining dataset, intermediate checkpoints at 100-step intervals, training and evaluation code, and a harness to rerun and verify any step on commodity hardware.Source: §7 · p. 22
  • Native int8 quantization-aware training leaves tensor-core GEMMs several times slower than a stock PyTorch bf16 baseline, and Open-1B trails on downstream benchmarks after training with an order of magnitude fewer tokens than OLMo 2 1B’s 4T-token budget.The report attributes the smaller token budget to this overhead and resource constraints.Source: §7 · p. 22
  • The report frames verifiable claims about how a model was built as a step toward a fully open and transparent AI development process.It describes this as a hope for future development, not a claim that the process is already fully open.Source: §7 · p. 22

A. Deterministic Gradient Reduction

The reduction replay folds gradient accumulation and intra-node reduce-scatter into one mesh-ordered loop, then mirrors inter-node communication with TreeFold.

A. Deterministic Gradient Reduction

  • Gradient accumulation and intra-node reduce-scatter share one loop over virtual ranks in replica-major, shard-minor mesh order, while inter-node all-reduce uses TreeFold.The running gradient sum is finalized and averaged every dp_shard ranks.Source: A. Deterministic Gradient Reduction · p. 27

A.1. Inter-Node All-Reduce over Replicas

TreeFold reproduces the cluster’s recursive-doubling replica reduction order while processing partial gradients as they arrive. Its binary-carry stack limits the number of full-gradient accumulators, including for replica counts that are not powers of two.

A.1. Inter-Node All-Reduce over Replicas

  • Floating-point addition is non-associative, so matching the cluster’s recursive-doubling all-reduce requires reproducing its replica-pairing order.The reduction tree has depth ⌈log2 dp_replicate⌉.Source: §A.1 · p. 27
  • TreeFold pushes each replica partial once and merges equal-sized blocks in binary-carry order, keeping the lower-replica operand on the left to match the cluster’s grouping.It can process partials as they finish and uses at most O(log2 dp_replicate) full-gradient accumulators instead of dp_replicate; at dp_replicate = 8, the flat alternative needs 8 resident partials while the fold needs at most 4.Source: §A.1 · p. 27
  • For replica counts that are not powers of two, TreeFold combines residual blocks in ascending level order with the lower-replica block on the left, matching the cluster’s handling and preserving bit-exactness.The remaining stack entries correspond to set bits of dp_replicate in binary.Source: §A.1 · p. 28

A.2. Intra-Node Reduce-Scatter over Shards

NCCL’s topology-dependent reduction order prevents bitwise single-device replay. DeterministicReduceScatter replaces it with data routing followed by a fixed ascending-rank sum.

A.2. Intra-Node Reduce-Scatter over Shards

  • Topology-dependent NCCL reduce-scatter can differ bitwise from a single-device replay despite roughly 10−11 relative error.Source: §A.2 · p. 28
  • DeterministicReduceScatter routes shard-destined chunks by all-to-all, then sums each shard’s contributions in ascending rank order.Source: §A.2 · p. 28
  • For each replica, adding shards 0 through dp_shard−1 in that order reproduces Algorithm 1’s inner accumulator bit for bit.Source: §A.2 · p. 28

A.3. Average via Reciprocal Multiply

CPU division and CUDA’s reciprocal-multiply implementation can disagree at the bit level. The cluster and single-device replay use the same precomputed-reciprocal multiplication for every average.

A.3. Average via Reciprocal Multiply

  • CPU division and CUDA reciprocal multiplication can disagree by up to 1 ULP when the divisor is not a power of two.Source: §A.3 · p. 28
  • The microbatch, reduce-scatter, and all-reduce averages shown as divisions should be read as reciprocal-multiplies, not literal divisions.Source: §A.3 · pp. 28–29
  • Both cluster and single-device replay compute every Algorithm 1 average by multiplying by a precomputed reciprocal constant.Source: §A.3 · p. 29

B. State-Hash Overhead

A sharded canonical hash gives auditors an intermediate-step reference, but does not create bitwise reproducibility. Its absolute cost stays nearly flat across scales, so its share of wall time rises with node count.

B. State-Hash Overhead

  • The sharded canonical hash digests each rank’s batches, parameters, optimizer, and gradient shards, supplying an intermediate-step audit reference rather than reproducibility.Source: B. State-Hash Overhead · p. 29
  • At fixed global batch, the hash’s relative cost grows as node count rises and accumulation per node shrinks.Source: B. State-Hash Overhead · p. 29
  • At 1, 2, 4, and 6 nodes, hash overhead is 4.9%, 9.3%, 16.7%, and 21.4% of wall time, while costing 5.77–5.96 s/step across scales.Source: Figure 11 · p. 29, B. State-Hash Overhead · p. 29

Related papers

Loading 2609.17380v1…