PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretraining and adapting a language model on a dependency-free stack: GPT-2 124M from random weights, reproduced against llm.c, and a clinical adapter for Qwen3-0.6B

Thang Tran, Lan Dang

arXiv:2609.28568v1cs.SE

TL;DR

Language-model training lacks an independently known right answer for much of its computation, and a second end-to-end implementation is needed to check it. The paper uses numbat to pretrain GPT-2 and adapt a clinical model, reporting close reference agreement and measured throughput without sacrificing speed.

  • Problem

    Most language-model training has no independently known right answer, while an end-to-end second implementation is costly to build and most work uses one family of frameworks.

    Source: §1 · p. 2

  • Method

    The authors use numbat to reproduce llm.c’s GPT-2 pretraining and adapt a small instruction model to clinical question answering, with reference runs and a sidecar able to halt runs.

    Source: §1 · p. 2, §7 · p. 16

  • Results

    Across paired evaluations, held-out loss tracked the same-machine reference closely and one-sidedly; a second reference agreed to 0.0035, and numbat scaled 2.769× across three cards.

    Source: §1 · p. 2, §9 · p. 17

  • Takeaways & Limitations

    Reference agreement on a general model does not itself assure clinical text processing, so both stages are run under the same controls.

    Source: §7 · p. 17

    Clinical held-out cross-entropy measures corpus fit, not clinical accuracy, safety, or usefulness; the adapted model is not validated for clinical use.

    Source: §8 · p. 17

Abstract

from arXiv · show

Almost every language model in service was trained by one family of software. That concentration makes a question hard to settle: how much of what is known about training a language model describes language models, and how much describes that software? Settling it needs a second implementation able to carry a model through a whole lifecycle rather than reproduce one operator. We report such a lifecycle. Using numbat, a machine-learning stack written in Zig with no third-party runtime dependencies, we pretrain a 124.4 M-parameter GPT-2 from random initialisation over 9.91 B tokens of web text, then adapt a separate small model to clinical question answering. A reference implementation runs on identical hardware at both stages, and a sidecar with authority to halt a run supervises each. Agreement is close. Held-out cross-entropy finishes at 3.2588 against a published 3.29, and HellaSwag at 0.3053 against 0.299; across 8 paired evaluations it sits below a same-machine reference at every point, by 0.0608 on average. Re-running that reference on different hardware moves it 0.0035, which bounds how much of any gap is method rather than framework. Throughput does not pay for agreement: measured in one session at a production configuration, numbat reaches 43,374 tokens per second against PyTorch's 41,202, scaling 2.769x over three cards. Clinical adaptation ends at 2.1899 held-out loss against 2.1941. Neither model is a medical device, and neither is validated for clinical use.

1 Introduction

The paper treats language-model training as a testing problem because most of its computation lacks an independently known right answer. It presents numbat as a second implementation for pretraining and clinical adaptation, checked against a reference and supervised by run controls.

1 Introduction

  • A second implementation is needed because most of language-model training has no independently known right answer, making silent errors difficult to detect from within a run.Loss can still fall and generated text can still read fluently even when errors change the final model quality.Source: §1 · pp. 1–2
  • Earlier work tested the stack through clinical fine-tuning, vision pretraining, and endoscopy, but had not pretrained a language model from noise.Pretraining combines mixed precision, gradient accumulation, and multi-device reduction over long runs, with errors compounding across updates.Source: §1 · p. 2
  • The paper reproduces llm.c’s 124.4 M-parameter GPT-2 on numbat and adapts a separate small instruction model to clinical question answering.The reproduction matches architecture, corpus, tokenizer, precision, optimizer, and schedule, while stating one deviation explicitly.Source: §1 · p. 2
  • A same-machine reference is compared across eight paired evaluations, while a different-hardware reference agrees with the first to 0.0035 at step 2,000.The second reference run helps distinguish a framework gap from variation in how the reference was produced.Source: §1 · p. 2
  • A sidecar can halt a run that has gone wrong, and the paper reports speed at a production configuration with remaining gaps attributed to named mechanisms.The control design includes a modelled band that warns and a self-referential stall check that can stop the run.Source: §1 · p. 2
  • The authors do not claim a new or better model, and make no claim about clinical answer quality.Lower held-out loss on clinical text is not evidence that the adapted model answers clinical questions correctly.Source: §1 · pp. 2–3

2 Background

The study uses llm.c’s documented GPT-2 training run as a reproducible target. It describes the reference model, corpus, evaluation measures, and training recipe.

2 Background

  • llm.c’s smallest GPT-2 configuration has about 124 million parameters and, after 9.89 B FineWeb tokens, reported held-out loss near 3.29 and HellaSwag near 0.299.The paper takes this run as its target; GPT-2 is a decoder-only transformer [26], and llm.c includes C/CUDA and PyTorch trainers.Source: §2 · p. 3
  • llm.c documents recipe choices often left implicit and provides weights, activations, gradients, and an optimizer trace for fixed-batch checks.Those reference materials allow another implementation to be checked before training.Source: §2 · p. 3
  • The corpus is FineWeb’s ten-billion-token sample [19], tokenized with GPT-2’s byte-level encoding [22], and the inherited budget matches llm.c’s published run.The text describes this budget as roughly eighty tokens per parameter and beyond the ratio near twenty derived by Hoffmann et al. [3].Source: §2 · p. 3
  • Evaluation reports held-out cross-entropy on a fixed corpus slice and HellaSwag [28] using length-normalised likelihood for candidate endings.The fixed slice is never trained on; the HellaSwag scoring style follows llm.c.Source: §2 · p. 3
  • Training uses AdamW [14], a cosine schedule [13], mixed precision [16], and data-parallel training [12], while clinical adaptation uses low-rank adaptation [4] with rank-stabilised scaling [7].Source: §2 · p. 3

3 The stack under test

The stack under test is numbat, a self-contained machine-learning library in Zig. It exposes multiple ways to run the same core and is designed for reproducible, agent-driven checks.

3 The stack under test

  • numbat is a Zig machine-learning stack with no third-party runtime dependency across tensors, autodifferentiation, data loading, multi-device reduction, and monitoring.This keeps the chain from disk to weight update independent of the framework being compared against.Source: §3 · p. 3
  • A declarative recipe records architecture, optimization, batch geometry, evaluation cadence, and output paths, and is validated before work starts.The recipe is recorded with results, making the run itself reproducible rather than merely described.Source: §3 · p. 3
  • The trained GPT-2 has 12 blocks, 12 attention heads, 768 channels, and a context length of 1,024, with vocabulary padding to 50,304 but softmax over 50,257 real tokens.Only activations use half precision; master weights, gradients, and optimizer state remain 32-bit.Source: Figure 1 · p. 4
  • The framework trainer and SDK’s C ABI drive the same tensors, kernels, autograd, and collectives, and profiles find the same kernels in the same counts.At identical geometry, measured differences between the two routes therefore provide evidence about measurement rather than different underlying kernels.Source: §3 · pp. 3–4
  • The command-line interface provides typed diagnostics and verdict-bearing acceptance gates that an agent or person can act on.Its exit codes distinguish a failed measurement from a malformed request.Source: §3 · p. 4

4 Method

The method reproduces the GPT-2 configuration, data preparation, and optimization choices of llm.c. It checks corpus preparation independently and specifies how the held-out data and precision are handled.

4.1 Model

  • The model uses 12 transformer blocks, 12 attention heads, 768 channels, and context 1,024, totaling 124,439,808 parameters with tied input and output embeddings.Vocabulary is padded from 50,257 to 50,304 entries for matrix shapes, while the softmax covers only real tokens and padding rows receive no probability or gradient.Source: §4.1 · pp. 4–5

4.2 Corpus and tokenisation

  • FineWeb’s ten-billion-token sample is tokenized with GPT-2 byte-level encoding and cut into 100,000,000-token shards; shard zero is held out and training reads the rest in order.numbat’s preparation produced identical token payloads to llm.c’s across 200,000,000 bytes, and the run uses llm.c’s own shards.Source: §4.2 · p. 5, §4.1 · p. 5, Figure 2 · p. 5

4.3 Optimisation and precision

  • Optimization follows llm.c’s AdamW settings and cosine schedule for 18,865 steps, with bfloat16 activations and 32-bit master weights, gradients, optimizer state, normalization statistics, and loss.Gradient spike skipping is disabled because llm.c skips no steps; enabling it would make the reproduction run more robust than its reference.Source: §4.3 · p. 6

4.4 Batch geometry, and the one place we differ

The run uses a three-card batch close to llm.c’s target and checks key training components before it starts. Its reference is llm.c’s PyTorch trainer, run on the same hardware before the gated run.

4.4 Batch geometry, and the one place we differ

  • 525,312 tokens per step is 0.195% larger than llm.c’s 524,288-token batch, the sole stated recipe difference.The three-card configuration uses 19 sequences and 9 accumulation steps; step count, warmup, and schedule remain llm.c’s.Source: §4.4 · p. 6

4.5 What is checked before training starts

  • Before training, the implementation checks activations, loss, all 148 gradients, and ten AdamW steps against llm.c’s reference dump.It also checks the scorer against llm.c’s script using the same GPT-2 checkpoint and 10,042 items.Source: §4.5 · p. 6

4.6 The reference trajectory

  • The llm.c PyTorch reference runs on the same three cards before the numbat run, but stops at step 2,000 of 18,865.The reference aligns the PyTorch script with llm.c’s C trainer and reports held-out loss averaged over all cards.Source: §4.6 · pp. 6–7

4.7 Control that can act

A sidecar compares held-out loss with a reference and can halt a run; beyond measured reference points, its alerts cannot stop training. The clinical stage uses a separate same-recipe reference leg and adapts Qwen3 for clinical question answering.

4.7 Control that can act

  • After three consecutive evaluations more than 5% above the same-step reference, the sidecar writes a halt marker and terminates the run’s process group.Three evaluations guard against a noisy single reading; the disk marker distinguishes a deliberate stop from a crash.Source: §4.7 · p. 7
  • Beyond the reference’s last measured step, interpolated points can raise alerts but never stop the run.The remaining range is interpolated between the last measurement and llm.c’s published endpoint in log step count; modeled bands can be wrong in the tightening direction.Source: §4.7 · p. 7
  • Without a usable reference comparison, the gate checks for stalls against the run’s own best loss and refuses comparisons with different example counts.A supervisor relaunches from checkpoints after crashes and stops on completion or halt markers.Source: §4.7 · p. 7

4.8 The clinical stage

  • The clinical stage adapts Qwen3-0.6B with rank-16 low-rank adaptation on seven projections per layer, using three public medical QA datasets.The 194,000 input records are accounted for as converted or duplicate; the split is 168,473 training and 3,440 held-out examples.Source: §4.8 · p. 7
  • For clinical adaptation, PyTorch and numbat run the same recipe on the same three cards, one at a time, so numbat can be gated against a measured reference curve.This later full-epoch run extends the earlier study’s protocol and fault inventory rather than making a new differential-testing claim.Source: §4.8 · p. 8

5 Results

Numbat completes the GPT-2 pretraining schedule near llm.c’s published quality, while matched evaluations, a second reference run, and production-configuration speed tests probe agreement and performance. The results also cover clinical adaptation and operational supervision.

5.1 Before training: agreement with the reference dump

  • The pretraining dump checks found maximum forward-activation disagreement of 1.465 × 10−3, loss disagreement of 7.153 × 10−6, and gradient disagreement of 2.408 × 10−5.On the benchmark anchor, both llm.c’s script and numbat scored 0.2955 on the same 10,042 items; the checks establish agreement before the long run, not its correctness.Source: §5.1 · p. 8

5.2 Trajectory against the reference

  • Across 8 paired evaluations, numbat’s held-out loss was below the same-machine reference at every point, by 0.0608 on average.The comparison is close rather than evidence of improvement: numbat’s step processes 0.195% more tokens, and the gap narrows as the run proceeds.Source: §5.2 · pp. 8–9, Table 1 · p. 9

5.3 How reproducible is the reference itself?

  • A second llm.c reference run differed from the first by 0.0035 at step 2,000, bounding reference-run variation against the larger numbat-reference gap.The two reference runs were within 0.0035 at step 2,000 despite different hardware and card counts.Source: §5.3 · p. 9

5.4 Speed

  • In a one-card, one-session test, numbat’s SDK reached 43,374 tokens per second against PyTorch’s 41,202; on three cards it reached 119,215 against 43,052, a 2.769× speedup.The three-card run used one process per rank; a separate production-configuration comparison reported 115,938 against 110,589 tokens per second.Source: §5.4 · pp. 9, 11, Table 3 · p. 11, Table 4 · p. 11
  • Gradient reduction made the two-card comparison favor PyTorch, while numbat scaled to three cards at 2.673× against PyTorch’s 2.684× in a separate measurement.Numbat overlaps reduction with backward computation, but the two-card scaling rates were 92.5% of twice numbat’s single-card rate and 93.7% of twice PyTorch’s.Source: §5.4 · pp. 12–13

5.5 Endpoint

  • Held-out loss finished at 3.2588 and HellaSwag at 0.3053 after 18,865 steps and 9.91 B tokens, near llm.c’s published 3.29 and 0.299.The authors interpret the endpoint as reproducing llm.c’s result, not improving on it; the 0.195% larger token step is the most likely explanation for the small margin.Source: §5.5 · pp. 13–14

5.6 Clinical adaptation

  • After one clinical-adaptation epoch, best held-out loss was 2.1899 against the reference’s 2.1941, with both results measured on the same capped 512-example prefix at step 3,500.The runs differed by one step because their handling of the partial final step differed; their wall-clock times were not comparable because the reference shared the machine with an unrelated compile.Source: §5.6 · p. 14, Table 7 · p. 15

5.7 Operational record

  • During the clinical epoch, the sidecar issued 146 verdicts, with no alert or termination.The 146 verdicts comprised 134 normal evaluations and 12 before a first evaluation existed.Source: §5.7 · p. 15

6 Availability

The paper releases trained-model artifacts and audit records, but not numbat itself; the released components and public inputs still allow the pipeline to be reconstructed.

6 Availability

  • Both public repositories are Apache-2.0 licensed and include weights and training logs for auditing claims.Source: §6 · p. 15
  • The GPT-2 release includes 124,439,808 parameters and an evaluation curve; its vocabulary is padded to 50,304, but only the first 50,257 rows are scored.Source: §6 · p. 15
  • The Qwen3-0.6B clinical adapter has 10,092,544 trainable parameters and an evaluation curve, but neither it nor its base model is validated for clinical use.Source: §6 · p. 15
  • Clinical logs contain all 3,509 optimizer steps and gate verdicts, while pretrained-model logs are incomplete because resume truncated the metric files.The full evaluation curve was recovered from console output and published separately.Source: §6 · p. 16
  • numbat is not published while its interfaces remain under active development; the trained models and records are available, and the pipeline can be reconstructed from published components.Neither released model requires numbat to load or run.Source: §6 · p. 16

7 Discussion

The discussion frames agreement as evidence about independent implementations, not proof of correctness, and highlights controls for reference variability and run termination. It also identifies a measured attention-kernel speed gap and explains why clinical adaptation was included.

7 Discussion

  • Matching held-out-loss trajectories support that two stacks implement the same computation, but cannot rule out common-mode faults or establish that the computation is correct.A second implementation primarily covers faults that are not shared between implementations.Source: §7 · p. 16
  • A second reference run on different hardware bounded reference variability and gave a bounded alternative explanation for the 0.0608 one-sided gap.The control cost a few hours on an otherwise idle machine.Source: §7 · p. 16
  • A truncated reference makes an extrapolated band vacuous beyond its measured range, so the proposed design separates a warning-only modelled band from a self-referential check that can halt a run.Source: §7 · p. 16
  • After other measured speed-gap sources were closed, attention remained: numbat’s vendor fused kernel was about one-fifth dearer than FlashAttention-2, and its replacement was correct but slower.The slower replacement remains behind a flag; the vendor path stays the default.Source: §7 · pp. 16–17
  • Clinical adaptation tests assurance beyond general-model training because prompt rendering, masking, and scoring change across domains, and a prior study found a large-effect fault outside numerical kernels.The cited earlier study is [23].Source: §7 · p. 17

8 Limitations

The reported evidence is bounded by reference coverage, hardware and batch differences, and the scope of clinical evaluation. Speed comparisons also apply only to their stated configurations.

8 Limitations

  • The same-machine reference covers only step 2,000 of 18,865; later comparisons use a published endpoint and an interpolated band that can only warn.The separate reference reproducibility control does not extend the comparison range.Source: §8 · p. 17
  • Results cover one model size and one hardware generation on a three-card machine without a direct card-to-card link, limiting how broadly reduction costs transfer.Source: §8 · p. 17
  • The training step is 0.195% larger than llm.c’s in a direction that favors numbat at equal step count, and this batch effect was not isolated.Source: §8 · p. 17
  • Clinical held-out cross-entropy measures corpus fit, not clinical accuracy, safety, or usefulness; no clinician or patient-outcome evaluation was conducted.The adapted model is a research artifact, not a medical device, and must not inform patient care.Source: §8 · p. 17
  • Speed comparisons use torch.compile for pretraining, while the clinical reference does not; results should be interpreted only within those configurations.Source: §8 · p. 17

9 Conclusion

Numbat trained a GPT-2 from random initialisation and adapted a separate model to clinical question answering, with both stages checked against a same-machine reference and supervised by a process able to stop them. The reported result pairs close agreement with measured speed and controls for interpreting the reference comparison.

9 Conclusion

  • Numbat trained a 124.4 M-parameter GPT-2 and adapted a separate model to clinical question answering, checking both stages against a same-machine reference under supervision by a process able to stop them.Source: §9 · p. 17
  • 1.053× of compiled PyTorch on one card and 1.048× on three, with 2.769× scaling across three cards, show speed was not sacrificed for agreement.The measurements were taken with every arm in one session.Source: §9 · p. 17
  • Where a margin remained, the authors attributed it to a named mechanism and reported it unclosed, including four measured-loss optimisations and a withdrawn cross-session figure.The withdrawn figure compared sessions rather than stacks.Source: §9 · p. 17
  • The combined result is presented as evidence that an independent implementation can carry a model from random initialisation through pretraining into a specialist domain, with checks at each stage.Source: §9 · p. 17
  • The study provides a measured estimate of the cost of a second implementation and describes controls that make such a run answerable.Whether that cost is worthwhile is left to each project to judge.Source: §9 · p. 18

Availability

The paper’s numbers are generated from files under data/, while framework source is not included in those artefacts.

Availability

  • Every reported number is generated from files under data/, including evaluation curves, reference trajectories, apparatus descriptions, and measurements quoted from gates.Source: Availability · p. 18
  • No framework source is included in those artefacts.Source: Availability · p. 18

Related papers

Loading 2609.28568v1…