Source-linked AI summary
Mutation Testing for Reproducibility Safeguards in Machine Learning Research Software: An Empirical Study
Ilya Shulepov
TL;DR
The paper examines whether existing ML research-repository validation workflows detect controlled changes to reproducibility-relevant experimental choices. Using MLReproMutate, it applies frozen, outcome-blind mutations and evaluates them against existing workflows, which detected 2 of 23 confirmed non-equivalent mutations. The findings support reproducibility-oriented mutation testing as a complementary assessment of safeguard sensitivity, while mutation survival does not establish repository-level irreproducibility.
Problem
Existing validation workflows may execute successfully without detecting changes to reproducibility-relevant experimental choices in ML research repositories.
Method
MLReproMutate applies four controlled mutation classes to 39 frozen repository–operator cases under an outcome-blind protocol using existing repository validation workflows.
Results
2 of 23 confirmed non-equivalent mutations were detected by the selected validation workflows, corresponding to an observed detection proportion of 8.7%.
Takeaways & Limitations
Reproducibility-oriented mutation testing provides a complementary way to assess whether existing research-software safeguards notice experimentally important changes.
Takeaways & Limitations
The selected workflow was one frozen repository validation path, so mutation survival does not show that no other safeguard would detect the change.
Abstract
from arXiv · showhide
Reproducibility in machine-learning research depends on experimental choices such as random seeds, dependency versions, data partitioning, and evaluation configuration. Existing repository validation workflows may execute successfully without detecting changes to such choices. We study this problem using MLReproMutate, research software that applies controlled, reproducibility-relevant mutations to ML research repositories and evaluates them against validation workflows already present in those repositories. We conducted an outcome-blind empirical study of 39 frozen repository-operator cases using four mutation classes: random seed, dependency pin, data split, and cross-validation fold count. Repository revisions, mutation candidates, and validation workflows were fixed before mutation outcomes were observed. Primary execution yielded outcomes for 13 of 39 cases; a bounded restoration procedure increased the combined evaluable set to 24. After excluding one confirmed-equivalent mutation, 23 confirmed non-equivalent mutations remained. The selected validation workflows detected 2 of these 23 mutations, corresponding to an observed detection proportion of 8.7%. These results do not imply that the corresponding repositories are irreproducible. Rather, they show that, in this sample, existing validation workflows often did not detect the particular controlled reproducibility-relevant changes introduced by the study. The findings motivate reproducibility-oriented mutation testing as a complementary way to assess whether research-software safeguards constrain experimentally important choices.
1 Introduction
The paper asks whether existing validation workflows detect controlled changes to reproducibility-relevant choices in ML research repositories. It introduces an outcome-blind mutation study and finds limited detection in the evaluated sample without equating survival with irreproducibility.
- Existing validation workflows may execute successfully without detecting changes to random seeds, dependencies, data partitioning, or evaluation configuration.
- MLReproMutate applies four reproducibility-relevant mutation classes—random-seed, dependency-pin, data-split, and cv-fold-count—to Python ML projects.
- The outcome-blind protocol froze repository revisions, mutation candidates, validation workflows, and prospective oracle metadata before observing mutation outcomes.
- 23 confirmed non-equivalent mutations remained after excluding one equivalent mutation, and selected workflows detected 2, yielding an observed detection proportion of 2/23 (8.7%).Primary execution produced outcomes for 13 of 39 cases; bounded restoration increased the combined evaluable set to 24.
- A surviving confirmed non-equivalent mutation shows only that the selected validation workflow did not detect that particular controlled change.
2 Background and Related Work
Prior work documents reproducibility, experimental variability, mutation testing, and dependency-update testing, while this study combines these concerns around validation sensitivity to reproducibility-relevant changes in real ML repositories.
- Prior studies show that code availability alone does not determine whether ML results can be independently reproduced.
- Random seeds, data sampling, initialization, and hyperparameter choices can materially affect measured ML performance.
- Mutation testing has established precedents in conventional software, scientific software, and machine-learning systems, including domain-specific operators.
- This study changes the measurement target from general fault or test-data adequacy to validation sensitivity for controlled reproducibility-relevant experimental changes.
- Dependency mutations may change specifications without changing the resolved environment, and prior dependency-update testing found gaps in detecting semantic problems.
- The empirical protocol freezes repository revisions, mutation candidates, and workflows, while separating infrastructure failures and verifying semantic equivalence.
- Mutation survival indicates only that the selected workflow missed one controlled change, not that the repository or scientific result is irreproducible.
3 MLReproMutate and Mutation Model
MLReproMutate applies narrow, deterministic mutations to experimental software and configuration, then evaluates each isolated mutant against an executable baseline and the repository’s selected validation workflow.
- MLReproMutate targets surrounding experimental software and configuration rather than learned-model structure, including seeds, dependencies, data partitioning, and evaluation parameters.
- The study uses four mutation classes: random-seed, dependency-pin, data-split, and cv-fold-count.
- Candidate detection and mutation application are separate operations, allowing selected candidates to be frozen and checked against their expected source locations.
- The operators use source-aware matching for supported Python constructs and apply restricted, deterministic transformations.
- The random-seed operator changes an eligible integer literal N to N + 1 while preserving explicit seeding.
- The data-split operator replaces explicit scikit-learn train_test_split stratification with None, while cv-fold-count changes eligible N ≥2 fold counts to N + 1.
- A successful unmodified baseline is required before the same validation command evaluates an isolated mutant; dependency evaluation separately identifies equivalent resolutions.
4 Research Questions
The study asks how often existing ML-repository validation workflows detect reproducibility-relevant mutations and how detection varies across workflow and oracle categories.
- RQ1 measures the overall sensitivity of selected validation workflows to confirmed non-equivalent reproducibility-relevant mutations.
- RQ2 examines mutation detection descriptively by validation-workflow type and oracle strength.
- Oracle strength is operationalized through prospectively recorded categories that are not assumed to form an ordinal scale.
- Mutation survival is not treated as evidence that a repository is irreproducible; it indicates only that the selected workflow missed that controlled change.
5 Study Design
The study froze a 39-case repository corpus, mutation candidates, revisions, and existing validation workflows before observing outcomes. MLReproMutate applied four auditable mutation classes, while baseline executability, semantic equivalence, and infrastructure failures were handled separately.
- Corpus construction: 39 frozen repository–operator cases formed the outcome-blind study corpus, with no post-outcome replacement of cases.The corpus included 10 calibration cases and 29 primary B02 cases.
- Mutation operators: Four mutation classes changed random seeds, exact dependency pins, train/test stratification, or cross-validation fold counts.The operators were restricted to mechanically identifiable, supported syntax.
- Validation workflows: Each case used one existing repository validation workflow selected before mutation execution rather than a newly constructed test suite.Workflow categories were treated as categorical, not as an assumed strength ranking.
- Outcome protocol: Baseline execution was required before mutation evaluation, and setup or workflow failures were classified as non-evaluable rather than as mutation outcomes.Only cases with evaluable baseline and mutant could receive Killed or Survived outcomes.
- Evaluated set: 24 combined evaluated cases yielded one semantically equivalent mutation and 23 confirmed non-equivalent mutations for meaningful detection analysis.The study reports counts descriptively because category sizes were small and uneven.
6 Results
The combined execution and restoration evidence produced 24 evaluable cases from the frozen 39-case frame. Across 23 confirmed non-equivalent mutations, detection was sparse and varied descriptively by operator, workflow type, and oracle category.
- Executability: 13 of 39 cases (33.3%) were evaluable during primary execution, while 26 cases (66.7%) were non-evaluable.Non-evaluable cases remained in the frozen study frame.
- Executability: 24 of 39 cases were evaluable after bounded restoration, leaving 15 cases (38.5%) non-evaluable.Within the D027 cohort, 11 cases were successfully restored and evaluated after substantive attempts.
- Mutation operators: Two dependency-pin mutations were detected among three confirmed non-equivalent dependency-pin evaluations; all meaningful random-seed, data-split, and fold-count evaluations survived.These operator-level counts are descriptive because denominators were small and unequal.
- Workflow type: By workflow type, one documented-validation and one documented-example mutation were detected, while upstream-test, CI, and documented-experiment cases had no detections.Workflow type was treated categorically rather than ordinally.
- Oracle type: In B02, 2 of 13 completion-only cases were detected and neither of 2 assertion cases was detected.Sparse cells, operator composition, and differential evaluability prevent attributing this pattern to oracle strength.
7 Discussion
The findings characterize validation sensitivity, not repository-wide reproducibility: most selected workflows accepted controlled, semantically different changes. The study presents mutation testing as a complementary probe while emphasizing narrow scope and executability constraints.
- Interpretation: 2 of 23 confirmed non-equivalent mutations were detected, while 21 changed relevant semantics without being rejected by the selected frozen workflows.This is evidence about evaluated repository–mutation–workflow combinations, not a population-wide estimate.
- Interpretation: A survived mutation means only that the selected frozen workflow did not detect that particular controlled change, not that the repository is irreproducible.A killed mutation likewise establishes rejection of only that applied mutation under study conditions.
- Workflow and oracle sensitivity: The study found no monotonic relationship between oracle explicitness and mutation detection.Both detected B02 mutations occurred in the dependency-pin × completion-only cell.
- Workflow and oracle sensitivity: Completion-only workflows can detect some dependency changes when altered environments cause imports, APIs, or runtime behavior to fail before result assertions.This mechanism does not establish that completion-only workflows are generally stronger.
- Executability: 13 of 39 cases were initially evaluable, restoration increased this to 24, and software-ecosystem barriers still left 15 cases non-evaluable.Barriers included obsolete dependencies, removed APIs, native requirements, hardware assumptions, and unavailable workflow requirements.
- Implications: Reproducibility-oriented validation can benefit from making important experimental assumptions observable and testable.Suggested checks include generated partitions, evaluation parameters, expected dependency environments, and configuration artifacts.
- Implications: MLReproMutate combines frozen revisions and workflows, explicit semantic-equivalence handling, and structured execution evidence to probe existing validation behavior.The approach complements conventional testing by targeting experimental configuration choices that may leave software operational.
8 Threats to Validity
The study’s validity is constrained by narrow mutation coverage, selected validation paths, limited and uneven cases, execution uncertainty, and sparse outcomes. These constraints narrow interpretation toward the frozen repository–operator–workflow sample rather than general claims about reproducibility.
- 8.1 Construct validity: The four operators cover only a narrow subset of reproducibility-relevant choices and exclude configuration drift, data processing, hardware effects, nondeterminism, and environment differences.The restriction improves auditable measurement precision but reduces construct coverage.
- 8.2 Internal validity: A selected validation workflow represents one repository path, so a surviving mutation does not show that other tests, inspections, or analyses would also miss it.Conversely, a killed mutation demonstrates sensitivity to one applied change, not general protection against reproducibility failures.
- 8.2 Internal validity: Outcome-dependent selection was limited by freezing repositories, revisions, candidates, workflows, and oracle categories before mutation execution.Restoration also preserved the frozen revision and workflow, prohibited compatibility patches, and used baseline-first symmetric evaluation.
- 8.3 External validity: The corpus is not a random sample, and unequal operator frames, selective repository eligibility, and non-evaluable historical software constrain generalization.The observed claim applies to selected frozen cases whose supported mutation and usable workflow could be identified under the protocol.
- 8.4 Conclusion validity: Only two of 23 confirmed non-equivalent mutations were detected, while sparse and uneven categories prevent causal or statistical conclusions about workflow or oracle type.Both detected mutations occurred in dependency-pin cases with completion-only workflows, a pattern confounded by operator composition and differential evaluability.
- 8.2 Internal validity: Present-day execution failures should not be treated as evidence that the original artifacts were irreproducible when published.Historical environments and external resources may have changed, and restoration cannot eliminate all uncertainty from contemporary execution.
9 Conclusion
The study uses MLReproMutate and an outcome-blind protocol to test whether existing ML-repository workflows detect controlled changes in reproducibility-relevant choices. Among evaluated non-equivalent mutations, detection was uncommon, while execution and workflow composition constrain interpretation.
- 9 Conclusion: The study evaluates random-seed, dependency-pin, data-split, and cross-validation mutations against existing repository workflows under frozen revisions and candidates.The protocol fixes these elements before mutation outcomes are observed.
- 9 Conclusion: 2 of 23 confirmed non-equivalent mutations were detected, while 21 were not detected by the selected validation workflows.The denominator excludes one confirmed-equivalent mutation.
- 9 Conclusion: 26 of 39 cases were non-evaluable after primary execution, reduced to 15 after bounded restoration while preserving the primary record.Executability is therefore a practical prerequisite for retrospective mutation analysis.
- 9 Conclusion: The descriptive analysis did not support attributing detection to oracle explicitness because both detected B02 mutations were dependency-pin cases with completion-only workflows.Small, uneven categories, operator composition, and differential evaluability prevent stronger inference.
- 9 Conclusion: Reproducibility-oriented mutation testing provides an auditable question about whether an existing safeguard notices a change in an experimentally important choice.MLReproMutate operationalizes this through controlled mutations, baseline-first execution, and semantic-equivalence handling.
A Supplementary Empirical Accounting
The supplementary accounting preserves the frozen 39-case corpus and distinguishes combined outcomes from their primary or bounded-restoration sources. It also records workflow, oracle, operator, and outcome fields across the listed cases.
- A.1 Full frozen corpus: The appendix reports supporting accounting without adding executions, mutation outcomes, or post hoc case selection.Table 6 joins the final accounting layer with the outcome-blind workflow frame by frozen case identifier.
- A.1 Full frozen corpus: The displayed outcome uses the canonical primary result when available, otherwise a successfully evaluated bounded-restoration result, and otherwise non-evaluable.This combined presentation does not replace separate primary and restoration accounting.
- A.1 Full frozen corpus: Table 6 identifies cases by case, repository, operator, workflow, oracle, source, and outcome, with B01 oracle entries left unclassified.Oracle classification is shown prospectively only for B02.
- A.1 Full frozen corpus: The listed corpus includes random-seed, dependency-pin, data-split, and cross-validation-related repository–operator cases with varied workflow labels and outcome annotations.The case rows include repositories such as gnina/libmolgrid, SimCSE, mindware, and sklearn-deap.
A.2 Bounded restoration accounting
The bounded-restoration appendix separates restoration accounting from primary outcomes and records which frozen cases reached evaluation. Its diagnostic oracle comparison is descriptive because sparse, uneven cells are confounded by operator composition and evaluability.
- A.2 Bounded restoration accounting: D027 was a separate bounded restoration layer that preserved the selected revision, mutation candidate, workflow, and oracle classification.The primary empirical layer remains the canonical execution record.
- A.2 Bounded restoration accounting: Within the 24-case D027 cohort, 19 had reports or assessments, 18 received substantive restoration attempts, and 11 reached mutation evaluation.Seven cases were not restored after substantive attempts, and one reported case was not attempted because repository identity could not be recovered.
- A.2 Bounded restoration accounting: Table 7 keeps report presence distinct from substantive restoration attempt and reports this accounting separately from primary mutation outcomes.The table is an accounting layer rather than an additional outcome analysis.
- A.3 Operator-by-oracle diagnostic: Both detected B02 mutations occurred in dependency-pin/completion-only cases, so sparse cells prevent attributing the pattern to oracle kind.The cross-classification is useful because observed oracle proportions cannot be interpreted independently of mutation operator.
- A.3 Operator-by-oracle diagnostic: Mutation outcomes are available only after successful baseline execution and mutation evaluation, so differential evaluability can change analyzed subset composition.The accounting tables make selected, evaluated, and meaningful counts visible rather than conditioning silently on successful executions.
B Mutation Operator Specification
MLReproMutate applies narrow, deterministic mutations to reproducibility-relevant choices and evaluates them against frozen validation workflows. The study separates execution outcomes from semantic verification and interprets surviving mutations narrowly as undetected confirmed changes, not repository-level reproducibility classifications.
- Mutation classes: The study uses four mutation classes: random-seed, dependency-pin, data-split, and cv-fold-count.The transformations are deliberately narrow and deterministic after candidate selection.
- Mutation classes: Random-seed mutations replace a selected integer-literal seed N with N + 1 in supported Python, NumPy, or PyTorch seed calls.Supported calls include Python’s random.seed, NumPy seed calls, and torch.manual_seed.
- Mutation classes: Dependency-pin mutations replace package==version with package>=version, while same-version resolutions are classified as semantically equivalent.Baseline and mutant environments are resolved independently.
- Mutation classes: Data-split mutations replace an explicit non-None stratify expression in supported train_test_split calls with None.The target is a supported scikit-learn train_test_split call.
- Mutation classes: Cv-fold-count mutations replace an explicit KFold-family n_splits value N with N + 1.Supported splitters include KFold, StratifiedKFold, RepeatedKFold, and RepeatedStratifiedKFold.
- Evaluation and study controls: Candidates retain target-location metadata and are rematched before application, while execution state, mutation outcome, and semantic verification remain distinct.Killed means a post-baseline workflow failure; Survived means workflow success after mutation, excluding infrastructure, setup, baseline, and restoration failures. The detection measure is Killed / (Killed + confirmed non-equivalent Survived).
- Evaluation and study controls: A surviving mutation means only that the selected frozen workflow did not detect that particular confirmed non-equivalent mutation.It is not a repository-level classification of reproducibility.
- Evaluation and study controls: The study freezes corpus construction, execution, restoration, and analysis boundaries across 39 cases, preserving revisions, candidates, workflows, oracles, and auditable result artifacts.D027 restoration did not rewrite the canonical primary execution record; B01 cases were not retrospectively assigned oracle categories.