PaperlayerView exact revision on arXiv

Source-linked AI summary

Reproducible AI Requires Reproducible Randomness

Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo, David Hill

arXiv:2609.26461v1cs.AI

TL;DR

Whether identical PRNG states guarantee matching sequences across software libraries remains a reproducibility question. This study compares Mersenne Twister and Philox implementations with their references and finds portability ranges from direct agreement to intervention-dependent or unattainable, including a fundamental PyTorch Philox incompatibility.

  • Problem

    Whether PRNGs ported across environments reproduce identical streams from identical initialization remains uncertain, despite its relevance to reproducible scientific experiments.

    Source: §1 · p. 2, §4 · p. 7

  • Method

    The study empirically compares Mersenne Twister and Philox execution traces across implementations against their original references under identical initialization conditions.

    Source: §6 · p. 12, §10 · p. 25

  • Results

    Portability falls into three profiles: inherently matching traces, matching after user intervention, or impossible without source changes; PyTorch Philox could not produce correct reference sequences in tested configurations.

    Source: §7 · p. 17, §10 · p. 25

  • Takeaways & Limitations

    The three-profile taxonomy provides a practical framework for assessing reproducibility guarantees offered by modern software libraries.

    Source: §10 · p. 25

    PyTorch Philox portability is limited because users can manipulate only the key and first 64 counter bits, while the remaining 64 bits are reserved internally.

    Source: §8.2 · p. 22

Abstract

from arXiv · show

Pseudorandom number generators (PRNGs) constitute indispensable computational tools across multiple scientific domains, including Monte Carlo simulations, stochastic computing, and artificial intelligence (AI). The reproducibility of such applications critically depends on the ability of PRNG implementations to generate identical sequences across software environments when initialized from the same internal state. These algorithms enable the simulation of stochastic processes while providing deterministic and repeatable behaviour, thereby facilitating reproducible experiments. Modern PRNG implementations may be initialized through either a seed or, more accurately, an initial state that exceeds the capacity of a conventional integer seed. However, reliance on a simple seed alone frequently proves insufficient to ensure consistent program execution traces across different implementations. A natural assumption is that transferring the complete internal state of a generator should guarantee identical outputs regardless of the software library used. This study examines the validity of this assumption by investigating whether complete initial states can ensure cross-library fidelity and portability of PRNG streams. We focus on two widely deployed generators, Mersenne Twister and Philox, and evaluate their implementations across four major Python ecosystems-Random, NumPy, PyTorch, and TensorFlow. We compare the sequences produced by these implementations against those generated by the original reference algorithms under identical initialization conditions. Our results demonstrate that reproducibility cannot be assumed from PRNG state transfer alone, even when implementations claim to follow the same underlying algorithm. While fidelity was successfully achieved for several implementations, significant discrepancies were observed in others. Most notably, the Philox implementation in PyTorch exhibits fundamental incompatibilities with the reference algorithm, preventing exact reproduction of generator outputs across environments. These findings challenge the common expectation that access to a full internal state of a PRNG is sufficient to ensure reproducibility across software stacks. They further highlight that implementation-specific design choices can introduce hidden barriers to experimental replication, particularly in AI workflows that rely on multiple frameworks. This work shows that implementation fidelity of a PRNG is a necessary condition for scientific reproducibility and makes two primary contributions. First, it identifies practical guidelines for achieving reliable PRNG usage and reproducibility within the Python scientific and AI ecosystem. Second, it evaluates the extent to which cross-library portability and fidelity can be recovered through user-level techniques, without requiring modifications to library source code.

1 Introduction

PRNGs enable deterministic randomness in scientific and AI workflows, but matching seeds have not guaranteed matching sequences across environments. This paper tests whether initializing MT and Philox with identical states can provide cross-library consistency in Python.

1 Introduction

  • PRNGs support stochastic methods such as Monte Carlo simulations and AI tasks including data shuffling and stochastic-gradient optimization.The introduction frames reproducible sequences as important to scientific experiments and machine-learning practice.Source: §1 · p. 2
  • Prior work observed cross-platform sequence discrepancies from identically seeded generators, prompting the question of whether full states can ensure cross-library fidelity.The authors note that a seed may not uniquely determine a generator’s complete state.Source: §1 · p. 2
  • The paper aims to clarify PRNG misuse caused by limited technical knowledge, ambiguous function names or documentation, and erroneous implementations.It uses examples from Python libraries widely used in machine learning to help users avoid these pitfalls.Source: §1 · pp. 2–3
  • The study examines MT and Philox: MT remains widely used despite statistical deficiencies, while Philox is counter-based and supports parallelization.The introduction notes MT is unsuitable for cryptographic applications; CryptMT was proposed for that purpose.Source: §1 · p. 3
  • The study’s primary objective is to test whether identical PRNG states can produce consistent random sequences across Python libraries.It hypothesizes that some library implementations may deviate from their original algorithmic specifications.Source: §1 · p. 3

2 Random numbers in science with a focus on AI

PRNGs are central to scientific and AI workflows because they provide repeatable sequences for debugging and reproducible experiments, while their quality affects stochastic computation. Their use spans AI preprocessing, model training, optimization, and regularization.

Scientific uses and requirements

  • PRNGs support experimental traceability, identical digital experiments across trials, and statistically independent sequences for unbiased analysis.Unlike true random number generators based on stochastic noise, PRNGs use deterministic algorithms to produce repeatable sequences.Source: §2 · pp. 3–4
  • Quasirandom generators instead target uniform coverage of solution spaces and can improve convergence in quasi-Monte Carlo applications, but are beyond this article's scope.Their structured sequences fail conventional statistical randomness tests, are useful for limited dimensions, and can avoid statistical replication.Source: §2 · p. 4
  • Monte Carlo simulations may require hundreds of billions of random numbers for one physical experiment, and their accuracy depends directly on PRNG quality.The method's growth into a state-of-the-art technique has been enabled by computational advances and relative implementation simplicity.Source: §2 · pp. 4–5

Randomness in AI and ML

  • PRNGs appear throughout AI workflows, supporting data augmentation, shuffling, sampling, splitting, initialization, batching, stochastic optimization, and dropout.These uses span preprocessing and training, making PRNG mechanisms and implementation relevant to experimental control and reproducibility.Source: §2 · p. 5

3 Why statistically good PRNGs are important

Statistical quality matters because PRNG structure can undermine simulations, while testing provides only probabilistic confidence; in machine learning, PRNG quality and seed choice can affect results.

3.1 Statistical properties of PRNGs

  • LCGs can offer long periods and one-dimensional uniformity yet produce lattice-structured points that impair sophisticated Monte Carlo; tests detect patterns but cannot prove true randomness.Desirable PRNG qualities include extended period, efficiency, repeatability, and portability; standardized batteries such as Diehard, Dieharder, and TestU01 assess properties but cannot guarantee infallibility.Source: §3.1 · pp. 5–6

3.2 Impact of good PRNGs in ML

  • Chaos-based and ML-driven PRNGs can outperform conventional generators on NIST randomness measures but typically run 10² to 10³ times slower; GPUs enable parallel PRNG implementations [29].Source: §3.2 · p. 6
  • Studies report systematic errors from random-number quality despite statistical testing, while ML outcomes vary with seeds and depend on algorithms, datasets, and train-test splits [31] [32].In one study, seed variation produced accuracy discrepancies approaching 2% under worst-case conditions; effects of randomness quality on dropout and generalization were dataset-contingent.Source: §3.2 · pp. 6–7

4 Reproducibility

The section distinguishes reproducibility from repeatability and argues that cross-environment PRNG portability depends on fidelity to the original implementation, not simply transferring a seed or state.

4 Reproducibility

  • Reproducibility requires independent researchers to recover identical findings using the study’s specified data, protocol, code, environment, and parameters; AI studies also face documentation and missing-artifact problems [35].Source: §4 · p. 7
  • On deterministic hardware, repeatability means consistent results under identical conditions; bitwise identity aids debugging, yet stochastic elements in modern AI stacks can undermine it.Source: §4 · p. 7
  • Test accuracies between 91% and 94% in most runs contrasted with 74% in one run, illustrating how randomness can obscure model performance and impede causal diagnosis.Without controlled repetition, the researcher cannot isolate whether data shuffling, weight initialization, or dropout caused the discrepancy.Source: §4 · p. 8
  • The MT algorithm requires at least 624 integer values plus an index for its approximately 2 KB internal state, so an integer seed cannot represent every possible state.Because seeding functions can map identical seeds to different states across libraries, the section asks whether transferring complete states guarantees identical behavior.Source: §4 · p. 8
  • The section defines PRNG portability as matching the original implementation’s expected output; divergent sequences make portability “fake” and cannot support meaningful replication.It treats fidelity to expected behavior as necessary for portability.Source: §4 · pp. 8–9
  • Implementation fidelity requires adherence to the original PRNG, while modular implementations may optimize or add features only if core behavior remains interchangeable; reference sequences support verification.Source: §4 · p. 9
  • Building on Antunes [43] and Ahmed, this study adds Philox to MT portability analysis and investigates failure mechanisms and user-level mitigation strategies.Source: §4 · p. 10

5 Studied PRNGs used in ML

The section identifies MT and Philox as widely used ML-library PRNGs and compares their designs, implementation choices, and suitability for parallel computation. It emphasizes that libraries differ in their generators and that implementation details matter when studying them.

Library PRNGs

  • TensorFlow uses Threefry and Philox, while PyTorch uses MT and Philox; NumPy’s default generator is PCG, and several libraries rely on NumPy for random-number generation.TensorFlow and PyTorch implement dedicated PRNG solutions rather than using NumPy’s default generator.Source: §5 · p. 10

Study scope

  • The study focuses on MT and Philox because they are used in many well-known ML Python libraries, while Threefry is used only by TensorFlow.The authors note that MT’s statistical flaws do not affect simulation or ML experiments, whereas PCG’s parallel-computation vulnerabilities make it unsuitable for parallel stochastic experiments requiring independent sequences.Source: §5 · pp. 10–11

Mersenne Twister

  • The study uses Python’s original 32-bit MT implementation, whose state contains 624 32-bit integers plus a 16-bit integer, about 2.4 KB.MT is also the default random-number generator in Python and is included in the C++ Standard Library.Source: §5 · p. 11

Philox

  • Philox uses a key-counter mechanism rather than MT’s iteratively updated state vector; its 4 × 32-bit variant stores key and counter in 24 bytes.With a 128-bit counter, Philox can generate up to 2^128 pseudorandom 128-bit values, equivalent to four 32-bit values.Source: §5 · p. 11
  • Philox supports GPU-oriented multistream and substream parallelization, with arbitrary counter positioning enabling substream generation at minimal computational overhead.For a 128-bit counter, the described partitioning provides 2^64 independent substreams, each capable of producing 2^64 pseudorandom values; the architecture may constrain use in exascale and large-scale AI clusters.Source: §5 · pp. 11–12

6 Experiments

The experiments tested whether PRNG implementations produce identical sequences under the same initialization, comparing Python-library outputs with original Mersenne Twister and Philox references.

6 Experiments

  • The study defined identical execution traces as identical generated-number sequences and designated the original Mersenne Twister and Philox implementations as OMT and OPX references.Experiments were run on a Xeon Platinum 8470 CPU and NVIDIA H100 GPU using Python 3.11.2, TensorFlow 2.20.0, NumPy 2.3.2, and PyTorch 2.8.0+cu126.Source: §6 · p. 12

6.1 OMT experiments

  • With seed 42, only NumPy’s default generator exactly matched OMT; PyTorch’s integer outputs aligned for two successive values but skipped intermediate sequence values.The initial comparison used OMT-generated reference outputs.Source: §6.1 · pp. 13–14, Table 3 · p. 13
  • With OMT-derived full state, NumPy’s legacy and V2 generators matched the reference, Python Random matched floats but not integers, and PyTorch remained incorrect.The PyTorch mismatch persisted despite full-state initialization.Source: §6.1 · p. 14, Table 4 · p. 14

6.2 OPX experiment

  • Philox experiments initialized generators with complete states and found distinct failure modes across major Python ML libraries, compromising cross-platform reproducibility.The study compared implementations against OPX reference sequences, using 4x32-bit and 4x64-bit variants; NumPy supports only the 64-bit configuration.Source: §6.2 · pp. 15–16, Table 6 · p. 16

7 Discussion

The discussion finds that cross-library PRNG reproducibility is impeded by implementation-specific output consumption and state representations. PyTorch Philox cannot reproduce reference sequences in the tested configurations, while other portability challenges include state-transfer interfaces and documentation inconsistencies.

7 Discussion

  • For MT, complete initial states can mitigate Random’s different seeding mechanism when reproducing reference output; the same approach applies to NumPy V2.Source: §7 · p. 17
  • PyTorch integer generation aligns with reference output only when one number is skipped, implying two integers are consumed per output, while floating-point generation does not skip.This difference in integer and floating-point generation can confuse users.Source: §7 · p. 17
  • TensorFlow’s Philox implementation exhibits the same jump phenomenon as PyTorch integer generation, consuming two numbers to produce one output.Source: §7 · p. 17
  • PyTorch’s Philox state represents a 64-bit key but only 64 counter bits instead of the specified 128, and tested initialization schemes never reproduced reference sequences.The specification typically represents the 64-bit key and 128-bit counter as 2x32-bit key and 4x32-bit counter components; PyTorch stores 16 bytes, leaving only 64 bits for the counter when grouped into 64-bit values.Source: §7 · p. 17
  • Cross-library state transfer is cumbersome because implementations use structurally different state formats, and native getter/setter interfaces cannot directly exchange states across libraries.OMT uses 624 32-bit integers and a 16-bit index (about 2.4 KB), whereas PyTorch uses 5056 8-bit integers (about 5 KB) with additional undocumented metadata and padding; bridging them requires detailed knowledge of both representations.Source: §7 · p. 18
  • Documentation deficiencies, including specifications that diverge from actual implementation behavior, further obstruct reliable cross-platform reproducibility.Source: §7 · p. 18

8 Curated code

Section 8 classifies PRNG non-portability as user-induced, user-resolvable, or fundamental, then examines examples in Mersenne Twister and Philox implementations. It distinguishes interface and seeding pitfalls, user-level workarounds, and discrepancies that require rewriting library code.

8 Curated code

  • Non-portability is categorized as user-induced interface misuse, user-resolvable inconsistency, or fundamental discrepancy that user-level changes cannot fix.Fundamental cases may require source-code modifications to comply with the original generator specifications.Source: §8 · p. 18

8.1 MT curated code

  • Random’s randint(a, b) increments its upper bound internally, while NumPy V2’s default seeding differs from OMT; these pitfalls can cause user-induced non-portability.NumPy’s undocumented legacy_seeding(seed) can reproduce OMT seeding, but it is absent from the official API reference.Source: §8.1 · p. 19
  • For PyTorch MT integer generation, drawing 64-bit integers and splitting each into two 32-bit unsigned integers recovers the correct sequence without changing library source code.The curated code’s output matches the expected sequence.Source: §8.1 · pp. 19–20
  • PyTorch MT float generation masks integers to 23 bits and divides by 2^24, producing 24-bit rather than expected 32-bit floats; correction requires manually constructing floats from curated integers.This implementation discrepancy is classified as fundamental because restoring the specified behavior requires rewriting the affected function.Source: §8.1 · pp. 20–21

8.2 Philox curated code

  • For Philox, accounting for NumPy’s counter increment restores sequence consistency with the reference, while TensorFlow’s expected sequence can be recovered by splitting generated 64-bit integers.Source: §8.2 · p. 22
  • PyTorch Philox exposes only one of four generated 32-bit values and hides half the counter for parallelization; (1,1) tensor calls recover intended values but skip three subsequent numbers each time.The state exposes the key and first 64 counter bits, while the remaining 64 bits are reserved for internal parallelization management.Source: §8.2 · pp. 22–23

9 Portability summary

Corrections recover consistent pseudorandom sequences in some portability contexts, but the practicality of those corrections in scientific workflows remains uncertain.

9 Portability summary

  • Corrections restored consistent sequences for some MT and Philox implementations, but the summary leaves their practical feasibility in research workflows unresolved.Table 11 distinguishes generators that work as intended but may be misused from cases where matching sequences require bit manipulations.Source: §9 · p. 24, Table 11 · p. 24
  • PyTorch MT portability requires manually decomposing 32-bit tensors to reconstruct the reference sequence, departing from standard scientific programming practices.The paper raises concerns about sustaining this intervention in rigorous research workflows.Source: §9 · p. 24
  • Table 11 classifies portability outcomes as generators working as intended but vulnerable to user misuse, or sequences recoverable through bit manipulations.The caption defines these categories but does not provide specific implementation outcomes in the supplied passage.Source: Table 11 · p. 24

10 Conclusion

The conclusion argues that PRNG reproducibility depends on implementation fidelity as well as initialization: despite the theoretical sufficiency of a complete internal state, implementation choices can prevent identical cross-library outputs. It presents a taxonomy of portability outcomes and calls for application-specific validation of implementations and initialization strategies.

10 Conclusion

  • Complete PRNG states do not guarantee identical cross-library outputs in practice; PyTorch’s floating-point generation uses normalization that diverges from the canonical Mersenne Twister specification.The paper warns that implementation differences can mean users receive a different generator, sometimes with a different number of bits, than intended.Source: §10 · p. 25
  • The authors classify portability into implementations that match inherently, those requiring user-level intervention, and those where consistency requires source-code modification.They present this taxonomy as a practical framework for assessing reproducibility guarantees and provide computational notebooks for verification and practitioner guidance.Source: §10 · p. 25
  • When portability erodes, computational artefacts become confined to particular execution environments, impairing independent verification of research findings.The authors caution that prioritizing implementation efficiency or convenience can undermine empirical research.Source: §10 · p. 26
  • Prior research emphasized algorithm statistics [14,25,26,43,52] and seed effects [32] and, leaving full-state initialization and implementation-dependent reproducibility comparatively underexplored.Wartel [52] found statistical failures in approximately 30% of cases across many initialization states using Big Crush; Hill emphasized good initializations and parallelization methods.Source: §10 · p. 26
  • Researchers should validate that both the PRNG implementation and initialization strategy meet application-specific requirements, complementing conventional generator evaluation [22].The paper provides a repository of statistically validated initialization states and reference sequences to support repeatability checks.Source: §10 · pp. 26–27

BIOGRAPHIES

The biographies describe authors at Clermont Auvergne University ranging from bachelor’s and PhD students to full professors, with work spanning energy measurement, machine learning, and data mining.

BIOGRAPHIES

  • Anthony Bertrand is a PhD student whose thesis concerns software-based energy measurement for machine-learning programs in HPC; he contributed the original draft, software, and testing.The biography also mentions his work on reproducibility challenges, but the passage ends before specifying them.Source: BIOGRAPHIES · pp. 30–31
  • David R. C. Hill is a computer science professor who has authored or co-authored more than 300 papers; he contributed review and editing, supervision, and validation.He is a director of an international graduate track at Clermont Auvergne INP.Source: BIOGRAPHIES · pp. 30–31
  • Engelbert Mephu Nguifo is a computer science professor leading research on machine learning and data mining for complex data; he contributed review and editing.His interests also include formal concept analysis, artificial intelligence, pattern recognition, bioinformatics, big data, and knowledge representation.Source: BIOGRAPHIES · p. 31
  • Tom Schmitt is a second-year bachelor’s student interested in low-level programming and optimizations; he contributed review and editing, software, and testing.Source: BIOGRAPHIES · p. 31

Related papers

Loading 2609.26461v1…