Source-linked AI summary
Cross-Stack Validation of Language-Model Training: A Clinical Fine-Tuning Case Study
Thang Tran, Lan Dang
TL;DR
Training can silently produce incorrect models even when runs converge, and existing differential testing rarely checks an entire training pipeline independently. This paper defines a trajectory-level cross-stack protocol and finds 0.134% average held-out cross-entropy difference across 42 paired evaluations, while exposing 17 missed faults, including data-rendering and runtime-specific faults.
Problem
Whole fine-tuning pipelines lack independent checks that can detect quiet errors across data handling, arithmetic, optimization, and evaluation.
Method
The study compares independently implemented stacks under a shared specification using cross-checks spanning pipeline stages and the learning trajectory.
Results
0.134% average held-out cross-entropy difference across 42 paired evaluations accompanied 17 surfaced faults, including data-rendering and runtime-specific faults.
Takeaways & Limitations
Independent implementations should differ across stack, orchestration, and runtime dimensions because those axes expose different classes of quiet faults.
Takeaways & Limitations
The study lacks a stochastic baseline, so interpreting residuals such as 0.134% as small remains a judgement rather than a measurement.
Abstract
from arXiv · showhide
Neural network training has an oracle problem: a run can converge normally and yield a usable model while the software beneath it computes something other than specified. Almost all such work runs on one stack, so there is rarely anything independent to check against. We study whether independently implemented training stacks can serve as differential oracles for a whole fine-tuning pipeline, rather than the operators and inference paths that prior differential testing targets. We define a trajectory-level protocol -- a shared specification, cross-check points spanning arithmetic, model loading, data rendering and the learning trajectory, and a separation of independence of the stack, the orchestration and the language runtime -- and apply it to a LoRA adaptation of Qwen3-0.6B over 168,574 clinical question-answer pairs under PyTorch and under numbat, an independent framework written in Zig, driven natively and through its C interface from six languages. Across 42 paired evaluations spanning a full epoch the two stacks' held-out cross-entropy differs by 0.134% on average, and four implementations end the epoch within 0.15% of one another. The comparison exposed 17 faults that single-implementation development had missed, two of them notable for software engineering. The fault with the largest effect on the trained model lay outside the numerical kernels: a mismatch in how clinical text was rendered moved held-out loss 0.15, some 500 times more than the arithmetic faults found beside it. And four faults were reachable only from a language whose memory model differs from the first two implementations: a scheduler migrating work across threads, a collector blind to device memory, an ownership discipline needing a primitive the interface lacked. Implementation diversity has several axes, and the runtime is one.
1 Introduction
Training can silently produce degraded models because stateful, stochastic pipelines lack independently known correct answers. This paper addresses that oracle problem by differentially validating whole fine-tuning pipelines across code-independent stacks, while showing that data rendering and runtime diversity expose faults numerical checks may miss.
- Motivation: The problem is especially difficult because errors usually avoid crashes, stateful updates propagate them, and stochastic correct runs need not agree exactly.A falling loss curve and fluent outputs therefore do not establish correctness.
- Contribution: A whole-pipeline comparison treats the learning trajectory—not an operator or forward pass—as the differential-validation signal across stacks sharing no code.The compared pipeline spans template rendering, tokenisation, forward, loss, backward, optimization, checkpointing, and evaluation.
- Contribution: The proposed protocol fixes a shared specification, checks arithmetic through trajectory levels, and distinguishes stack, orchestration, and language-runtime independence.It is presented as a trajectory-level differential validation protocol for end-to-end training.
- Results: 17 faults were exposed; the largest model effect came from data rendering rather than numerical computation, while four faults required runtime diversity.These findings support language-runtime independence as a distinct and productive validation axis, while warning that independently written versions can still fail together.
- Scope: The case study demonstrates that reproducing a full clinical adaptation across stacks is affordable because the cost of running the second implementation is measured rather than assumed.The clinical corpus provides a realistic workload, but the paper does not claim that lower held-out loss establishes clinical answer accuracy.
2 Related work
Prior work compares implementations, execution modes, APIs, generated models, or inference, whereas this study differentially validates end-to-end training through its learning trajectory. It also connects silent framework failures, design diversity, and reproducibility to a gap involving runtime semantics and pipeline behavior.
- Differential testing of deep-learning libraries: Prior differential testing compares backends, generated or mutated models, or distributed and non-distributed execution, while this study compares a real adaptation across two stacks.D3 is the closest precedent, but its oracle is a second execution mode of one library and its workload is generated rather than a real adaptation.
- What goes wrong in these frameworks: Framework studies show that silent bugs can produce neither errors nor crashes, motivating attention to failures concealed by a converging loss curve.The study’s fault inventory also reaches pipeline faults, such as text rendering before tokenisation, beyond framework-level analyses [1] [15].
- Design diversity: N-version programming proposed independently implemented versions for reliability, but correlated mistakes show that implementation independence is not a single property.This motivates separating stack, orchestration, and runtime diversity rather than treating independent implementation as one undifferentiated attribute.
- Reproducibility: Reproducibility from rerunning identical code does not establish that a separate implementation of the same specification produces the same learning behavior.The study frames cross-implementation agreement as complementary evidence rather than a replacement for conventional reproducibility.
- Gap: The study fills a gap by treating end-to-end training as a stateful stochastic computation and using its learning trajectory as the differential signal.Existing machine-learning differential testing targets APIs, generated networks, inference, or alternative execution modes within one library; the study asks whether runtime-semantic diversity exposes additional fault classes.
3 Study design
The study fixes one shared training specification and evaluates agreement across independent stacks, orchestration layers, and language runtimes. It checks equivalence from arithmetic and model loading through data rendering and the training trajectory, while treating agreement as evidence limited to the tested configuration and observables.
- Configuration: The same configuration and corpus fix the adaptation method, optimizer, schedule, batching, sequence length, precision, and evaluation protocol across implementations.Framework defaults yield to the specification; Table 1 records the identical adaptation configuration.
- Independence axes: Three independence axes separate arithmetic, orchestration, and runtime tests: PyTorch versus Zig-based numbat, fresh trainers over its C interface, and six language bindings.The runtime axis changes scheduling, memory, ownership, and foreign-function semantics without changing arithmetic, exposing four faults unavailable to the other axes.
- Validation protocol: Agreement is checked at four levels: operator arithmetic and pre-training loss, checkpoint loading, exact token rendering, and 42-point held-out-loss and gradient-norm trajectories.The ordered levels localize failures, with trajectory checks revealing faults invisible to the preceding three levels.
- Interpretation and limits: Agreement in independently computed observables across a diverse trajectory supports materially equivalent training behavior, but equal held-out loss does not establish equal functions.The inference is explicitly limited to the tested configuration and strengthened by the number and diversity of observables.
- Execution environment: Under Windows WDDM, device-memory pressure can degrade execution instead of raising an error, making Fault 17 roughly seven times slower without changing losses.The trajectory runs used separate accelerators concurrently for practicality, so they support loss comparisons at steps, not timing comparisons.
4 The workload
The workload uses a realistic clinical instruction-tuning task: a deduplicated examination corpus rendered in a single format and adapted with LoRA on Qwen3-0.6B. Its small model enables broad implementation coverage, while its exacting template makes rendering correctness consequential.
- The workload is realistic rather than synthetic, combining long sequences, a non-trivial template, a real corpus, and a current adaptation method.Clinical instruction tuning supplies all four properties, while the material is examination content rather than clinical practice and contains no patient data.
- 168,574 training and 3,440 held-out examples remain after converting and deduplicating 194,000 rows from three publicly licensed sources.Every input row is accounted for as converted, duplicate, or rejected (100.0%), and targets contain written explanations rather than answer-choice letters.
- Qwen3-0.6B [14] is adapted with low-rank adaptation [4], freezing the released weights and adding trainable matrices in a 40 MB file.The base model remains unchanged; Table 1 provides the configuration.
- The model is small enough for a full epoch on one consumer card, enabling many implementations, but Qwen3’s empty final assistant reasoning block makes the template easy to misrender.The section notes that this rendering detail matters more than the arithmetic in later analysis.
5 Results: trajectory agreement
Across a full epoch, independently implemented PyTorch and numbat training stacks closely agreed on held-out loss, while all evaluations showed genuine learning. Four additional interface-driven implementations also converged within 0.15%, but provide weaker evidence because they shared kernels.
- Learning trajectory: 31% lower held-out perplexity and 2.5314-to-2.1635 cross-entropy reduction over 10,536 updates showed the workload was a genuine learning task.All 42 evaluations improved over the preceding evaluation, though this does not establish clinical correctness.
- Initialization check: 3.672389 held-out cross-entropy matched to six decimal places for the untrained model in both stacks, checking the forward path and data pipeline before training.This agreement was established before either implementation took a step.
- Cross-stack agreement: 0.134% average held-out cross-entropy difference separated PyTorch and numbat across the epoch, with a 0.316% maximum difference and 0.138% standard deviation.The signed residual changed sign 12 times, consistent with an unbiased rather than systematic numerical difference.
- Gradient agreement: 0.1% median gradient-norm agreement provided an independent trajectory check, while larger maxima differences were attributed to individual batches with differing composition.Gradient norms are internal quantities, so matching them tests whether implementations take comparable steps beyond their reported loss.
- Interface-driven replication: 0.151% Zig and 0.060% Python mean absolute deviations led four implementations to finish within 0.15% and agree within 0.11% on gradient magnitude.These comparisons are weaker evidence because the implementations shared kernels, but they tested the interface and three independently written training loops.
6 Results: which observation exposed which fault
Cross-implementation comparison identified 17 faults that single-implementation development missed, including numerical, data-rendering, evaluation, and runtime-specific failures. The observations were complementary: data-rendering exposed the largest loss shift, while runtime diversity revealed faults unreachable by the other checks.
- Fault discovery: Seventeen faults escaped single-implementation development, including five hard failures, and every fault was found through comparing implementations.The hard failures included out-of-memory, kernel-launch, and autograd errors.
- Fault discovery: Fixing generic rather than model-taught chat rendering reduced held-out loss by 0.15, roughly 500 times the 0.0003 improvement from two numerical faults.The numerical faults involved an adapter on a shared projection and non-contiguous operands in transpose-matmul backward.
- Runtime diversity: Runtime diversity exposed four faults unreachable by the first two implementations because the additional language had a different memory model.These included thread-local state breaking under Go scheduling, garbage collection missing device-memory pressure, and Rust ownership requiring a second owning handle.
- Evaluation: A flat held-out-loss curve resulted from evaluation running without the adapter, not from failed training.Fixing the evaluation fault exposed the memory behavior described in §3.4.
- Cross-check levels: The checks were complementary: unit tests found local operator faults, forward comparisons found configuration faults, data checks found rendering faults, trajectories found accumulated-state faults, and runtime diversity found language-specific faults.No single observation level covered the full fault set.
7 Feasibility
The study finds that cross-stack validation is feasible for this workload: the independent numbat implementation trains faster and uses less device memory than the reference, while a second implementation costs about as much to run. The authors avoid treating these measurements as a performance claim because the study used one run per implementation and variable clock behavior.
- Feasibility: Eight implementations ran the Table 1 configuration for 500 updates on the same accelerator, with the first 100 discarded as warmup and 400 measured.Runs were conducted one at a time on an otherwise idle machine, with the device verified idle before and after each run.
- Feasibility: The measurements support feasibility rather than a performance claim: running a second independent implementation costs about as much as running the first.The authors caution that one run per implementation and session-to-session clock variation larger than the reported differences cannot support a performance conclusion.
8 Discussion
The discussion separates independence into stack, orchestration, and runtime dimensions, each exposing different fault classes. It also emphasizes semantic data-rendering checks and cheap cross-implementation validation before training.
- Independence spans stack, orchestration, and runtime dimensions, covering arithmetic, training-loop interfaces, and scheduling, lifetime, and ownership respectively.Runtime independence addresses non-numerical behavior, while stack independence covers arithmetic and orchestration independence covers the training loop and kernel interface.
- A 500-fold effect gap between data-rendering and arithmetic faults highlights semantic boundaries above the framework that differential framework testing does not cover.The passage cautions that this is a single observation, but identifies project-specific templates as an uncovered testing boundary.
- For practitioners: Before training, compare held-out loss on the untrained model, then compare rendered token sequences for several records to catch costly faults cheaply.The first check takes minutes and needs no second training run; the second targets the location of the study’s most expensive fault.
9 Threats to validity
The study’s validity is limited by sparse trajectory evidence, reconstructed fault-detection records, and no stochastic baseline. Its timing, scale, configuration, and clinical conclusions are also narrowly scoped.
- Fault detection: The fault matrix records the order in which development exposed faults, not controlled detection rates, so reinjecting each fault and testing every observation is the key extension.A signal that fired second might have fired first under a different discovery order.
- Trajectory evidence: Held-out cross-entropy and gradient norm provide a thin comparison vector, while one seed prevents measuring same-stack run-to-run variation against cross-stack residuals.Richer observables—prediction distributions, parameter-update summaries, and activation statistics—would strengthen the equivalent-training claim; without a stochastic envelope, interpreting 0.134% as small is judgment rather than measurement.
- Trajectory continuity: The numbat trajectory was interrupted at update 8,912 of 10,536 and resumed without optimizer moments, introducing a discontinuity that an uninterrupted rerun would remove.A supervisory rule fired on one noisy batch, and the run resumed from its last checkpoint.
- Timing: The timing study establishes feasibility, not comparative performance, because each implementation had one run on different days under an unstable clock governor.An earlier bit-identical leg was five to eight times slower; repeated interleaved runs with confidence intervals would be needed for performance claims, which the study does not make.
- Scale and configuration: The study covers one 596M-parameter model, one adaptation method, f32 precision, and one laptop-class card, leaving larger, reduced-precision, multi-device, and distributed regimes untested.Absolute memory figures also depend on the platform.
- Clinical validity: No downstream accuracy or safety evaluation was performed: lower held-out loss indicates better clinical answer-text prediction, not correctness or safety, and the artifact is not a medical device.The artifact is intended for research use.
10 Conclusion
An independently implemented training stack can differentially validate an entire fine-tuning pipeline through its learning trajectory. In a clinical Qwen3-0.6B case, cross-stack agreement exposed faults and showed that implementation diversity must span stack, orchestration, and runtime.
- Conclusion: An independently implemented training stack served as a differential oracle for the entire fine-tuning pipeline, using the learning trajectory as an affordable comparison signal.The result addresses the problem that training software can converge while computing the wrong computation.
- Conclusion: 0.134% held-out cross-entropy difference across 42 paired evaluations over a full epoch, with four implementations finishing within 0.15% of one another.The two stacks shared no training code, and the comparison surfaced 17 faults missed by single-implementation development.
- Conclusion: The most damaging fault was clinical text rendering, which changed held-out loss roughly 500 times more than the neighboring numerical faults.The fault lay outside the arithmetic and occurred before the model saw the clinical text.
- Conclusion: Four further faults were reachable only from a language with different scheduler, collector, or ownership models, showing that implementation diversity spans stack, orchestration, and runtime.A second implementation is most valuable in dimensions where it differs from the first.
Data and code availability
The authors provide measurement data and figure- and table-generation scripts, while clarifying that the adapted model is a research artifact rather than a medical device.
- Data and code availability: Measurement data for every figure and table, plus scripts generating them, are available from the authors; the adapted model is not a medical device.The model is designated a research artifact.