Which reported inputs govern molecular docking reproducibility? A benchmark from reporting audit to independent re-execution

1 Introduction and Related Work

• Docking reproducibility concerns re-obtaining a reported score from its described inputs, distinct from testing docking accuracy against measured affinities.The study's scope is computational re-execution, not wet-lab replication.Source: §1 · p. 1, §1.1 · p. 3

1.1 Related work

• Reporting checklists specify fields but treat them equally, while this study compares fields by their score changes under controlled perturbations.Prior docking benchmarks assess scoring, ranking, docking, or screening against measured or known-active references; this study instead examines re-executability from reporting records.Source: §1.1 · p. 3
• Span verification ties extracted reporting fields to verbatim source text rather than free-text assertions.The study adopts structured extraction with a programmatic span-verification gate.Source: §1.1 · p. 3

1.2 Study overview

• Six linked analyses cover reporting completeness, input perturbations and noise, cross-scoring and cross-target checks, ligand resolvability, empirical NSE weights, and reproduction outcomes.The audit includes 50 open-access papers and 12 method fields.Source: §1.2 · p. 3

2.1 Four estimands and their separation

• NSE keeps four quantities distinct: claim-level reporting readiness and reproduction outcome, plus query-conditioned best-evidence score and verification priority.A calibrated reproduction probability is defined but not estimated at the present sample size.Source: §2.1 · p. 4, Table 6 · p. 4

2.2 The minimum executable reporting set

• NSE has 16 reporting fields, but water and ion handling, receptor preparation, and code artifacts were not directly tested.Twelve fields were included in the reporting audit; ligand identity was assessed through resolvability and protonation through controlled perturbation.Source: §2.2 · p. 4

2.3 The E0 to E4 executability ladder

• The E0–E4 ladder classifies reporting records by recoverability of target, ligand, and binding site, remaining parameters, and reported robustness or validation.E1 marks an unrecoverable critical input; E2 requires assumptions; E3 is executable as reported; E4 adds robustness or validation, without evaluating scientific correctness.Source: §2.3 · pp. 4–5

2.4 A measurement model for reporting readiness

• Readiness is measured from binary reporting items rather than asserted weights; the full 12-item set had α=0.62, with engine and random-seed items near-degenerate.The exploratory five-item scored subscale retains grid centre, grid size, exhaustiveness, redocking validation, and protein chain.Source: §2.4 · p. 5
• The five-item core readiness subscale has Cronbach’s α=0.73 and uses normalized corrected item–rest associations rather than fitted IRT discrimination.Its ranking agrees with the hand-weighted ranking at Spearman ρ=0.90 and the unweighted item sum at ρ=0.87; weights and subscale selection have unquantified sampling uncertainty.Source: §2.4 · p. 5

2.5 Best-evidence aggregation: axioms and the relevance gate

• The best-evidence score combines topical relevance and graded quality with a hard relevance threshold, an operational choice rather than a uniquely implied aggregator.Below the threshold the score is zero and discontinuous; the injected-decoy test checks implementation of that rule, not relevance validity or scientific quality.Source: §2.5 · p. 6

2.6 Graded quality

• Quality is a renormalised weighted mean of method rigour, reporting completeness, artifact availability, integrity, citation context and source, with venue entering only through a hard cap.Unassessed dimensions are excluded from the denominator rather than assigned zero.Source: §2.6 · p. 6

2.7 Calibration from score to reproduction probability

• At 37 re-executions from 7 papers, a stable calibration map was not identifiable, so the study reports a descriptive within-tolerance rate rather than a calibrated reproduction probability.The continuous quality score did not predict absolute reproduction error (Pearson r = 0.10).Source: §2.7 · p. 7

2.8 Verification priority

• Verification priority is presented only as a proposed companion, not a ranking instrument, because its objective requires a calibrated probability that is not yet estimable.No budgeted selection algorithm or approximation guarantee was tested; the interim priority uses executability, impact and cost.Source: §2.8 · p. 7

2.9 Empirical field weights

• Receptor structure received the largest empirical weight (1.02), followed by ligand identity (0.29) and protonation (0.19); grid centre, grid size, random seed and exhaustiveness received weights of 0.08, 0.05, 0.04 and 0.01.The weights are post-hoc effect-size measurements; the study did not test a scorer reweighted by them against reproduction outcomes.Source: §2.9 · pp. 7–8

2.10 Executability strata as a routing signal

• Executability class routes claims according to whether reported information permits re-execution; it is not interpreted as a probability of reproduction.The framework covers computational re-execution of box-based docking, not wet-lab replication or boxless methods such as DiffDock.Source: §2.10 · p. 8

3.1 Audit corpus and span-verified extraction

• The audit covered 50 open-access papers and 12 reporting fields, accepted extracted values only when they matched source-text spans, and used one human reviewer to assign final labels.The reviewer adjudicated a topic-stratified 30% subset; no second human coder was used.Source: §3.1 · p. 8

3.2 Controlled re-execution ablation

• One parameter at a time was changed while other inputs were held fixed, including box size, exhaustiveness, box centre, seed, protonation, ligand identity and receptor structure.Receptor effects were measured with 85 non-cognate comparisons across three targets, while box-size and exhaustiveness perturbations were also checked in a diverse 44-complex set.Source: §3.2 · pp. 8–9

3.3 Noise floors and the two-floor reproduction tolerance

• The within-machine median run-to-run range was 0.04 kcal/mol, while independent cloud comparisons yielded median absolute differences of 0.43 and 0.18 kcal/mol for two software-stack pairings; tolerance was 2.0 kcal/mol.The 0.43 kcal/mol value is a conservative reference for this study, not a universal lower bound.Source: §3.3 · p. 9

3.4 Validation with Vinardo

• The scoring-function check changed only Vina to Vinardo while retaining the prepared structures, ligands, boxes, seed and exhaustiveness, and assessed receptor-effect rank agreement with Spearman correlation.A Vina positive control reproduced the recorded cross-dock score exactly.Source: §3.4 · p. 9

3.5 Generalization: target diversity

• The cross-docking extension covered 12 targets from 7 protein families; 238 of 305 attempted runs succeeded and yielded 191 valid non-cognate cross-self comparisons.The extension added nine targets from six additional protein families; failed runs were retained in the ledger and not imputed.Source: §3.5 · p. 9

3.6 Reproduction harness (N=37)

• The reproduction harness used deterministic ligand resolution and recorded provenance for each execution; 37 quality-control-passed claims from 7 papers formed the within-tolerance analysis.Unresolved or failed entries were excluded from the within-tolerance denominator, and comparisons between the six E3 and 31 E2 claims were treated as descriptive.Source: §3.6 · p. 10

3.7 Statistics

• The analysis used equivalence tests against the 0.43 kcal/mol cross-environment margin, Wilson intervals, paper-clustered bootstrap intervals, rank and group-comparison tests, and false-discovery-rate control.Pooled comparisons were treated as exploratory because observations sharing a ligand or receptor are not independent.Source: §3.7 · p. 10

3.8 Powered design for a calibration campaign

• The proposed campaign targets about 150 quality-control-passed re-executions across at least 15 papers and three protein families, with 30 to 40 claims per executability stratum.These are planning targets; the present six E3 claims are below the approximately 40-per-stratum powered target.Source: §3.8 · pp. 10–11

4.1 Reporting completeness differed among method fields

• 100% of papers reported the docking engine and 94% the receptor structure, whereas only 2% reported a random seed; reporting-status agreement was 93.9% in the adjudicated subset.The seed fixes the stochastic component of the search, so reproducing the other 49 calculations exactly would require an assumption.Source: §4.1 · p. 11, Table 1 · p. 11

4.2 Search-space perturbations produced small score changes

• 0.054, 0.010, 0.084, and 0.04 kcal mol−1 were median changes for box size, exhaustiveness, box centre, and random seed, respectively; all were negligible against the 0.43 margin.The seed variation equalled the 0.04 kcal mol−1 within-machine self-consistency floor.Source: §4.2 · p. 12, Table 2 · p. 12

4.3 Receptor structure produced the largest effect across scoring functions and targets

• 1.02 kcal mol−1 (n=85) was the pooled receptor-structure effect; across nine additional targets it was 0.84 kcal mol−1 (n=191; P=0.48).The effect ordering was retained with Vinardo, and per-structure effects correlated across scoring functions (Spearman ρ=0.64, P=4.5 × 10^-11).Source: §4.3 · p. 13, Table 3 · p. 13

4.4 Unresolved or ambiguous ligand strings blocked automated re-execution

• 26/116 reported ligand strings (22%) remained unresolved or ambiguous after exact and deterministic name normalization, preventing their automated re-execution.This rate describes the specified resolver and does not mean manual chemical curation could never identify a structure.Source: §4.4 · p. 14

4.5 Empirical effects reordered prespecified field priorities

• 1.02 > 0.29 > 0.19 kcal mol−1 reordered priorities toward receptor structure, ligand identity with a resolvability gate, and protonation; receptor preparation, water and ions, and code artifacts were untested.The empirical weights are post-hoc effect measurements, not a reweighted scorer evaluated against reproduction outcomes.Source: §4.5 · p. 14, Table 4 · p. 14

4.6 Reproduction outcomes differed descriptively by executability stratum

• 29/37 local and 27/37 cloud re-executions were within 2.0 kcal mol−1; the E3–E2 comparison was underpowered and confounded by box reporting.E-class describes whether re-execution is directly supported by the report; these results do not establish it as a reproduction predictor.Source: §4.6 · p. 15, §4.7 · p. 21, Table 5 · p. 21

4.7 Reproduction on independent infrastructure

• Among 35 claims run on both machines, the median absolute difference was 0.18 kcal mol−1, all differences were within 1.0, and one tolerance verdict flipped.The study notes that cross-environment variation depends on the environment pair, with a separate comparison yielding 0.43 kcal mol−1.Source: §4.7 · p. 21

• The receptor-versus-search contrast persisted in the Vinardo check and the 12-target extension, while ligand-identity and protonation effects were not retested across all settings.The benchmark targets span diverse protein folds and cognate-ligand chemotypes.Source: §5 · p. 21, Figure 8 · p. 23
• The empirical hierarchy complements guidance on structural provenance and ligand-state definition, but addresses a different question from benchmarks of scoring, ranking, or screening performance.The discussion supports prioritising an exact receptor structure, a machine-resolvable ligand identifier such as a PubChem CID or InChIKey, and ligand protonation state.Source: §5 · pp. 21, 24
• 27 of 37 cloud re-executions (73%) were within the 2.0 kcal mol−1 tolerance, and local and cloud within-tolerance verdicts agreed for 34 of 35 claims.The panels show reported-versus-cloud affinities, local-versus-cloud affinities, Bland–Altman agreement, and verdict concordance.Source: Figure 7 · p. 22
• 34% and 42% of papers reported grid centre and grid size, yet each produced median absolute score changes of no more than 0.08 kcal mol−1; ligand identity shifted scores by 0.29 kcal mol−1, and 22% of ligand strings remained unresolved or ambiguous.Receptor structure produced the largest graded effect, and exact receptor structure, a resolvable ligand identifier, and protonation state are the supported reporting priorities.Source: §5 · pp. 21, 24
• The 78% within-tolerance reproduction rate is descriptive, not a calibrated probability; the evidence does not support a powered executability-stratum contrast or per-paper intervals.The empirical and psychometric weights are not prospectively validated reproduction predictors.Source: §5 · p. 24
• The conclusions apply to computational re-execution of box-based docking; other docking implementations and boxless or CNN-assisted methods were not evaluated.The reproduction analysis covered 37 claims from 7 papers, and executability-stratum comparisons were underpowered and confounded by box reporting.Source: §5 · p. 24
• Future evaluation calls for more E3 claims without box-reporting confounding, perturbations of the three untested fields, additional docking methods, and expert pairwise judgments.Expert judgments are needed before the verification-priority score can be interpreted as a validated ranking instrument.Source: §5 · p. 24

• The archive contains derived audit tables, annotation forms, agreement summaries, ligand-resolvability outputs, docking outcome tables, analysis code for retained inputs, figure images, and a checksum ledger.The README lists input paths, source versions, executable checks, and omissions.Source: §6 · p. 25
• The package audit recomputes selected counts and agreement from retained tables, but does not represent new docking runs or a fresh full-text collection.The original full build driver and some intermediate docking tables are unavailable, so the archive does not claim an end-to-end figure or docking rebuild.Source: §6 · p. 25
• Figures 1, 5, and 8 include structural vignettes of recorded benchmark complexes, which are inputs rather than additional docking results.The release provides a source-file checksum ledger; the public archive omits verbatim article spans.Source: §6 · p. 25

• Giap Duc Ha is a scientific cofounder of NewScience, which develops verification- and executability-aware tools, and the NSE method evaluated here could inform those tools.The study object was docking-claim reporting and re-execution rather than the NewScience platform; derived data, code, and provenance are supplied for independent assessment of the findings.Source: §7 · p. 25

• No human participants, animal experiments, or individual-level biomedical data were used.Source: §8 · p. 25

9 Funding

• No external funding was declared for this study.Source: §9 · p. 25

A Admissible best-evidence aggregators

• Continuity at zero requires g(0) = 0 for a separable score F(R, Q) = 100g(R)h(Q) with nonzero quality, excluding positively quality-weighted additive scores.The condition does not exclude 100√RQ; whether one missing quality artifact vetoes a score depends on how Q is constructed.Source: A Admissible best-evidence aggregators · p. 25
• The selected monotone, separable hard gate excludes scores exactly below τ_R > 0 but is discontinuous at the threshold.Other continuous zero-relevance functions meet the stated conditions; decoy exclusion checks implementation, not scientific validity.Source: A Admissible best-evidence aggregators · p. 25

B Estimability of the measurement model and the powered design

• At N ≈ 50, the audit supports Cronbach’s alpha, item–rest weights, readiness scoring and weight-robustness checks, but not stable two-parameter item-response modelling.A target of N ≈ 250 could support a stability evaluation and, with adequate fit, item parameters, per-paper confidence intervals, fit statistics and differential-item-functioning tests; a bifactor extension needs a larger corpus with at least three items per factor.Source: B Estimability of the measurement… · p. 26
• About 150 re-executions across at least 15 papers and three protein families is the planning target, because calibration requires roughly 100–150 quality-control-passed re-executions.The target combines evaluation of the E3-versus-E2 contrast and calibration; environment-dependent variation needs repeated-environment or threshold-sensitivity assessment.Source: B Estimability of the measurement… · p. 26