PaperlayerView exact revision on arXiv

Source-linked AI summary

Numbat: Building and Verifying a Self-Contained Machine-Learning Stack

Thang Tran, Lan Dang

arXiv:2609.10632v1cs.SEcs.LG

TL;DR

Large Python-orchestrated machine-learning frameworks impose costly dependencies and undocumented behavioral differences that can silently degrade training outcomes. Numbat addresses these issues with a self-contained stack and a five-level reference-based verification protocol, demonstrating end-to-end training-outcome parity on a competitive detector recipe.

  • Problem

    Large Python-orchestrated frameworks incur costly dependency, deployment, and reproducibility burdens, while undocumented recipe behaviors can silently degrade metrics across implementations.

  • Method

    Numbat combines a dependency-free single-language stack with a five-level protocol that treats an incumbent implementation as an executable specification and catalogs detected recipe divergences.

  • Results

    The stack reproduces a competitive from-scratch detector training outcome, with the exported model evaluated by the reference stack’s validator and the run reporting same-machine throughput.

  • Takeaways & Limitations

    The work provides a self-contained machine-learning stack, a transferable divergence catalog, and a reproduction protocol for holding independent implementations to reference behavior.

  • Takeaways & Limitations

    Validation is single-seed, extends the same-machine reference trajectory only to 30 epochs, and fully exercises the protocol on one detector family.

Abstract

from arXiv · show

Machine-learning systems are built almost exclusively on a few large Python-orchestrated frameworks, and they inherit those stacks' engineering costs: environments of hundreds of version-coupled packages, separate export toolchains for deployment, and the split between the language research is written in and the language products ship in. We report on the construction and verification of numbat, a machine-learning stack written in one general-purpose language (Zig) with no third-party runtime dependencies. The stack spans tensor computation, automatic differentiation, neural-network modules, mixed precision, multi-GPU training, data loading and monitoring; an SDK exposes it behind a stable, additively versioned C ABI of over 1,400 entry points, with bindings for six languages; and its clinical domain planes encode regulatory requirements as executable acceptance gates rather than documentation. Verifying such a stack is the harder half of building it: a defective training run rarely fails, it converges quietly to a slightly worse model. We treat a widely used reference implementation as an executable specification and verify against it at five levels, from operator gradient checks to an automated trajectory gate against a same-machine reference run - the arrangement our companion study formalizes as a trajectory-level differential oracle. The protocol surfaced ten silent recipe divergences, which we catalog with mechanisms and symptoms. As the acceptance test, we train a 25.9M-parameter detector of the YOLOv8m class from random initialization on COCO 2017 for the full 500-epoch schedule: the exported weights score 0.4956 mAP50-95 under the official protocol, scored by the reference stack's own validator (published endpoint 0.502), with single-GPU step time at parity on identical hardware. Weights, per-epoch metrics and the full run manifest are released.

1 Introduction

Numbat addresses the engineering and reproducibility costs of Python-orchestrated ML stacks with a self-contained Zig stack and a reference-driven verification protocol. Its end-to-end test reproduces a competitive detector recipe while exposing silent divergences and releasing the resulting artifacts.

  • Design: Numbat combines a dependency-free Zig implementation covering training through deployment with applications shipping as single static binaries.The design targets the package, deployment-toolchain and research/production-language costs of mainstream stacks.
  • Reproduction protocol: The protocol surfaced ten silent recipe divergences involving behavioral details that can degrade metrics without overt failure.Examples include initializer distributions, autocast placement, optimizer grouping, EMA semantics, clipping, loss scaling, augmentation and RNG structure.
  • Contributions: The stack’s contributions include reference-default semantics, a strictly additive C ABI, and executable acceptance gates for clinical requirements.These contributions are presented alongside the framework and its end-to-end evidence.
  • Reproduction protocol: The five-level protocol verifies operator, module, step, trajectory and outcome agreement against a same-machine reference, treating the incumbent as an executable specification.When levels disagree, numbat is changed to match the reference rather than the reverse.

2 Related work

Related work spans dominant Python-based training ecosystems, alternative self-contained or native stacks, YOLO detector development, and reproducibility research. Numbat’s distinctive combination is a full training stack with explicit behavioral compatibility and cross-framework trajectory verification.

  • Related systems: Numbat combines a single-language dependency-free training stack, PyTorch-default compatibility, and a packaged SDK, unlike the cited alternatives.The comparison includes PyTorch, TensorFlow, JAX, MLX, Burn, tinygrad, llama.cpp/ggml and ONNX Runtime.
  • The YOLO family: The YOLOv8 class combines CSP, PAN-FPN, an anchor-free decoupled head with DFL regression, task-aligned assignment and CIoU loss.These architectural components define the detector family used as background for the case study.
  • Training detectors from scratch: The paper studies whether an independent framework can reproduce a reference recipe’s outcome, not whether from-scratch detector training is possible.Prior work established that sufficient schedules can bring randomly initialized detectors to pretrained-equivalent accuracy.
  • Reproducibility and cross-stack validation: Cross-framework validation extends reproducibility work from operator agreement to convergence-level parity using independently implemented stacks as trajectory-level differential oracles.The reference implementation serves as the specification for the comparison.

3 The numbat framework

The numbat framework is a vertically integrated Zig stack spanning tensor computation, training, backends, data loading, monitoring and deployment interoperability. Its implementation emphasizes explicit memory, compile-time specialization, reference-compatible semantics and automated trajectory gating.

  • Architecture: The stack integrates tensors, kernels, autograd, modules, optimizers, distributed training, data decoding, tokenization and monitoring in one Zig codebase.Applications can ship as single static binaries, with vendor GPU libraries as the only foreign-function seams.
  • Memory and execution: Peak training memory fell from 18.5 to 13.5 GiB per GPU, a 26.8% reduction, with bitwise-identical gradients.The reduction comes from explicitly freeing saved activations as the backpropagation tape drains.
  • Memory and execution: Compile-time tensor and backend specialization removes runtime dispatch layers between model code and kernels across twelve element types.The tensor and backend dispatch are monomorphized at compilation rather than mediated by an interpreter or graph VM.
  • Training subsystem: Reference-default optimizers, mixed precision, deterministic DDP, trajectory gates and resumable checkpoints align training behavior with the reference stack.The monitoring sidecar can pause or terminate runs outside their reference bands, while checkpoints preserve optimizer, EMA, epoch and RNG state.
  • Backends and kernel generation: The single-source kernel plane compiles Zig device kernels to SPIR-V with the framework’s compiler, so targets need only a display driver at build and run time.The compiled kernel pack is embedded in the library and requires no vendor toolkit, SDK or shader compiler on the target.
  • Data pipeline: After correction and vectorization, the JPEG decoder reached 1.88× its previous throughput and the loader fully overlapped GPU computation.Parity testing exposed a quantization-table ordering bug that had perturbed every decoded pixel.
  • Interoperability and model zoo: Numbat supports ecosystem formats and a self-contained .nbq container carrying device-ready weights, tokenizer, configuration and an authenticated manifest.The layout enables zero-copy CPU mapping or a single host-to-device transfer without repacking.

4 The numbat SDK

The numbat SDK packages training and inference behind a stable, additive C ABI with thin bindings for six languages. It preserves one-stack development across platforms while enforcing numerical agreement across language surfaces.

  • Packaging and portability: Training, evaluation and inference share one self-contained library across Linux, macOS, Windows, x86-64, ARM64 and WebAssembly.The platform matrix is cross-compiled from a single development machine.
  • C ABI: The SDK exposes more than 1,400 nb_* entry points through a versioned, strictly additive C ABI using opaque handles, runtime-tagged dtypes and status codes.The boundary avoids framework headers and memory-layout assumptions to preserve interface stability.
  • Bindings: Thin Zig, C, Python, Rust, Go and TypeScript bindings expose the same ABI without third-party consumer packages.The bindings present PyTorch-shaped APIs so existing ecosystem experience transfers directly.
  • Numerical agreement: Four implementations ended a full fine-tuning epoch within 0.15% in held-out cross-entropy, with 42 paired evaluations differing by 0.134% on average.The study compared PyTorch, native numbat, and SDK-driven Python and Zig implementations.
  • Applications: A single ABI links directly into C/C++/Rust control stacks and supports static, allocator-controlled deployments for robotics and long-running systems.The same library also supports on-device adaptation without a separate training-side stack.

5 Domain planes: requirements as executable gates

numbat’s clinical domain planes treat requirements as executable acceptance gates, typed failures, and enforcing interfaces rather than documentation alone. Release testing also validates the shipped artifact, exposing defects that repository-wide checks can miss.

  • Domain scope: The planes cover medical imaging, genomics, clinical language, evidence, voice, and agent-facing interfaces, making domain machinery a substantial part of the SDK.The paper identifies this construction method—not merely the domain coverage—as the potentially generalizable engineering contribution.
  • Requirements as executable gates: Each domain plane discharges numbered requirements through executable acceptance gates with verdicts and exit codes, while explicitly listing uncovered clauses.A phase is complete only when its gate passes; pending clinical data, elapsed time, or hardware requirements remain visible.
  • Typed failures: Typed diagnostic registries map stable failure codes to severity classes and repair plans, with coverage measured from raise-site ledgers.The measured-coverage gate failed three times on registered codes that had no underlying check, unlike a hand-maintained list.
  • Interfaces that refuse: Domain interfaces enforce rules directly: licensed vocabulary indexing requires a license reference, and measurements carry run-manifest provenance or are marked untraceable.Claim gates can block releases when recorded evidence does not match the study design needed for the claim.
  • Artifact-level verification: Testing the released artifact found an ABI gate-count defect that passed repository checks because the ABI and conformance test shared the same incomplete table.The incident motivated treating the shipped archive as part of the test surface.

6 Case study: YOLO-NB-M from scratch on COCO

The case study trains a 25.9M-parameter YOLOv8m-class detector from random initialization on COCO 2017 using a recipe reproduced from the reference implementation. Verification proceeds from operator checks through trajectory supervision, with official scoring performed by the reference validator.

  • Model and data: YOLO-NB-M is a 25.9M-parameter, 80-class YOLOv8m-class detector trained from random initialization on COCO 2017.It uses an anchor-free one-stage design with CSP, PAN-FPN, DFL regression, task-aligned assignment, and a composite BCE, CIoU, and DFL loss.
  • Recipe: The released recipe reproduces reference behaviors including world-size loss scaling, optimizer grouping, global-norm clipping at 10.0, EMA, warmup, and final mosaic shutdown.These behaviors are coded in the reference trainer and are documented in the complete released recipe.
  • Reference target: The reference implementation serves as an executable specification, and numbat is independently validated against it without incorporating its source code.Disagreement at any verification level causes numbat to adopt the reference behavior rather than changing the oracle.
  • Five-level protocol: Verification spans operator gradient and backend checks, module known-answer tests, fixed-data optimization probes, and a 30-epoch same-machine trajectory gate.The step-level probes include deterministic distributed-gradient comparisons and AMP-versus-F32 agreement after fixes.
  • Outcome measurement: Official accuracy is computed by exporting numbat weights and scoring them with the reference stack’s validator under the COCO protocol, while the internal evaluator is used only for gating.The internal evaluator reads 0.011–0.014 lower than the official score at converged checkpoints.

7 Divergence catalog

The reproduction protocol exposed silent recipe divergences that can degrade or stall training without crashes, and it combines trajectory monitoring with leveled diagnosis. The catalog shows why end metrics alone are insufficient for locating these failures.

  • Observed silent failure: An ungated run plateaued at roughly one fifth of the target metric for 43 epochs, consuming about 1.5 machine-days without a crash, NaN, or increasing loss.The protocol later caught the contributing divergences, which were fixed by adopting reference behavior and re-validated.
  • Trajectory verification: Figure 2a shows numbat tracking the same-machine reference trajectory across the 30-epoch gate window, with 1,403 polls and zero trajectory violations.The figure also contrasts internal per-epoch metrics with official-protocol checkpoint scores across the full schedule.
  • General observations: Divergences compose, distinct root causes can share the symptom of starting slightly below the reference band, and effective batch size changes short-horizon kinetics.Initialization, autocast placement, and scaler collapse formed one silent freeze chain, while ignoring gradient accumulation to nominal batch 64 misled probes.

8 Results

The released epoch-500 detector reaches 0.4956 mAP50–95, within 1.3% of the published endpoint, while matching single-GPU reference performance; multi-GPU throughput remains slower and memory higher.

  • 8.1 Accuracy: 0.4956 mAP50–95 and 0.6610 mAP50 were achieved on COCO val2017 by the released epoch-500 EMA checkpoint, within 1.3% of the 0.502 published endpoint.The weights were scored under the official protocol by the reference stack’s validator.
  • 8.1 Accuracy: 97.9% to 1.24× of the reference validation trajectory was observed during the 30-epoch gate window, reaching or exceeding the corrected reference after evaluator-offset correction.The automated trajectory gate tracked the same-machine reference before the full run.
  • 8.2 Throughput, memory, energy: 245.5 ms versus 256.9 ms was the capped end-to-end single-GPU step time for numbat and the reference, respectively, with live metric streaming included for numbat.Stage timing was ahead for forward and optimizer steps and at parity for backward and loss.
  • 8.2 Throughput, memory, energy: 15.6 versus 27.8 min/epoch was the three-GPU reference versus initial numbat runtime, reflecting a serialized host-staged all-reduce.The reference overlaps bucketed NCCL reductions with backpropagation, while numbat’s initial path absorbs inter-rank arrival skew inside the step.
  • 8.2 Throughput, memory, energy: 13.5 GiB/GPU versus approximately 7 GiB was peak training memory for numbat and the reference, respectively, despite a 26.8% reduction from 18.5 GiB.The residual approximately 1.9× factor is attributed to AMP shadow parameter copies and allocator high-water behavior.
  • 8.2 Throughput, memory, energy: Zero trajectory-gate violations and zero human interventions were recorded across 1,403 polls during the 500-epoch run, which completed in 9.6 days with GPU energy bounded at ≤1.4×10^2 kWh.The run used approximately 30 automatic checkpoint-resume segments with full state continuity.

9 Reproducibility and release

The release makes the paper’s claims independently reproducible through exported weights, official-protocol scoring, per-epoch metrics, and a complete run manifest.

  • The release includes trained weights, per-epoch training and validation metrics, official score trends, training-curve exports, and a hyperparameter-and-environment manifest.
  • Released safetensors weights can be loaded by the reference stack and rescored with its validator and pycocotools, without access to numbat.
  • The reference implementation served only as a behavioral specification and independent scorer, not as a source of pretrained weights or shared framework code.

10 Limitations

The evaluation is bounded by single-seed evidence, limited reference-trajectory coverage, one detector family, higher memory use, and unresolved scaling and availability constraints.

  • The headline run uses one seed, so its equivalence claim is trajectory- and endpoint-level rather than a distributional statement.
  • The same-machine reference run covers 30 epochs, with later equivalence assessed against the published endpoint and official-protocol scoring of exported checkpoints.
  • The protocol is demonstrated on one detector family, while full five-level validation remains incomplete for the framework’s LLM, ASR, and segmentation stacks.
  • At this configuration, numbat uses approximately 1.9× the reference’s training memory, and its small-core-host DDP gap remains unresolved on larger hosts.
  • The framework’s source is proprietary, and SDK distribution terms remain outside the paper’s scope.

11 Conclusion

The paper presents numbat as a self-contained stack paired with explicit compatibility and verification disciplines. Its strongest validation reproduces a mature training recipe while using the incumbent implementation to score the result.

  • Numbat combines reference-default semantics, an additive C ABI, executable domain acceptance gates, and a five-level protocol treating the incumbent as an executable specification.
  • The validation reproduces a mature, competitive recipe from scratch and has the incumbent’s tooling score the resulting evidence.
Loading 2609.10632v1…