RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

• DeepSeek-V4-Flash reproduces 41% of Run-tier papers, 27% of Retrain-tier papers, and 12% of Reimplement-tier papers; Muse Spark 1.2 leads Reimplement at 15%.Source: §1 · p. 1
• RECLAIM evaluates agents using the paper and released artifacts, assessing execution evidence and how results compare with authors’ claims.Source: §1 · p. 1, Figure 1 · p. 2
• At Retrain, the strongest agent achieves verified partial reproduction or better on 67% of runs versus 27% for the weakest, while full recovery ranges from 15% to 27%.Source: §1 · p. 2
• In the 96 H100-hour band, 48 cells used 6.2% of allotted compute and only one reproduced its claim, indicating compute did not bind.Source: §1 · p. 2
• A small set of mistakes accounts for most failures; some become rarer with stronger agents, while others do not.Source: §1 · p. 2, Figure 2 · p. 2

Repository and terminal agents

• Repository benchmarks predefine objectives and verifiers, while reproduction requires agents to select the experiment and build an executable environment; both settings show early stopping.SWE-bench and Terminal-Bench 2.0 use prepared tests or checks, while MirrorCode supplies a queryable black-box reference; MirrorCode reports 97.4% of runs submitted before half the token budget.Source: §2 · p. 3

ML engineering and experimentation

• ML engineering benchmarks such as MLE-bench reward any pipeline that maximizes a leaderboard score, not recovery of a published quantity using a prescribed method and configuration.They leave open whether an agent can reproduce reported numbers in an environment the authors did not use.Source: §2 · p. 3

Autonomous research systems

• Autonomous-research systems are judged by manuscripts rather than whether results hold; Scientist-Bench is closer to Reimplement, but grades pairwise similarity instead of numerical match to a frozen target.LLM review can be unreliable: Schmidgall et al. (2025) report scores of 6.1/10 versus human reviewers’ 3.8, while Tang et al. (2025) report comparable-paper judgments varying from 13.64% to 81.82% by judging model.Source: §2 · p. 3

Scientific reproduction

• Many scientific-reproduction benchmarks settle release state before runs and favor papers with working public code, so they cannot measure how unreleased or nonworking code affects reproduction.CORE-Bench reruns released code, PaperBench withholds code, EXP-Bench withholds the paper, and AutoExperiment masks repository functions; CORE-Bench excludes papers deemed irreproducible.Source: §2 · p. 3
• MLReplicate accepted 10 manuscripts, but human reviewers found fabricated or unsupported claims in 59% of reviews, and the benchmark does not validate the experiments.The ICML 2026 Open Reproductions challenge treated logbook self-assessments as untrusted and re-verified claimed falsifications.Source: §2 · p. 4

3 RECLAIM

• Each of the 100 benchmark papers defines one task: reproduce a single pinned result, chosen to match the central empirical claim, within a stated tolerance.The target is fixed before any agent runs and uses the cheapest reported configuration that still supports the claim, bounding the compute required while exercising the method.Source: §3 · p. 4, §3.2 · p. 4

3.1 FEATURES OF RECLAIM

• Agents plan and execute reproduction end to end, while provenance-audited grading accepts any faithful route and the benchmark can be refreshed yearly from new conference papers.Tasks require agents to install and debug the paper’s software stack and manage GPU allocations; an agent classifier verifies candidates before human audits, with new NeurIPS, ICML, and ICLR submissions supplying future targets (cf. White et al., 2025; Jain et al., 2025; Badertdinov et al., 2025).Source: §3.1 · p. 4

3.2 TASK DEFINITION

• A match target records the experiment configuration, metric, reported value, measurement scope, and success threshold, and pins the cheapest reported configuration that establishes the central claim.Pinning the target before agent runs bounds compute while still exercising the paper’s method.Source: §3.2 · p. 4

3.3 BENCHMARK CONSTRUCTION

• Construction began with 3,414 NeurIPS 2025 papers with arXiv IDs, using a 1,000-paper sample and an artifact-verifying classifier to identify claims, targets, and available artifacts.Artifacts counted as available only when accessible through a tool call; inaccessible, paywalled, sign-up-gated, or permission-gated artifacts counted as unavailable.Source: §3.3 · p. 5
• The benchmark’s artifact tiers specify the rebuild required: Run papers provide code, data, and weights; Retrain papers lack weights; Reimplement papers lack code but use public datasets.In the 1,000-paper classifier sample, 687 papers were fully verified and 620 remained eligible; among the verified papers, 33% were Run, 44% Retrain, and 13% Reimplement.Source: §3.3 · p. 5
• Selection capped evaluation targets at 96 H100-hours and sampled an audited pool equally across tiers; humans checked every pool paper and replaced inaccessible-artifact or paid-API cases.The pool contained 200 papers, selected cheapest first within tier and compute bands; only 20 of the final 100 compute estimates came directly from the papers.Source: §3.3 · p. 5

4 AUDITING

• The auditor traces reported numbers to execution, checks protocol, split, and scale, and recomputes metrics from saved artifacts when evidence is inconclusive, without rerunning the reproduction.The auditor reads the evidence directory and transcript, then assigns a 0–10 score from which the categorical verdict is derived deterministically. The rubric was frozen before grading began; Claude Sonnet 5 (Anthropic, 2026c) was used for every run, following LLM-auditing work by Zheng et al. (2023) and Zhuge et al. (2025).Source: §4 · p. 6
• A high-severity anti-cheat flag forces a zero, while missing reported artifacts earn no credit; typed flags and code apply the integrity adjustment.Examples of high-severity violations include echoing a number from the paper, hardcoding a constant, or placing a proxy value in the target slot.Source: §4 · p. 6
• 0.82 F1 agreement with blind human graders on reproduced verdicts supports auditing all runs; PaperBench reported 0.83 F1 against human labels on individual rubric items (Starace et al., 2025).Three human graders each scored one of the 396 runs that started; the LLM auditor can review dozens to hundreds of tool rounds, logs, and outputs in minutes, making every-run auditing feasible.Source: §4 · p. 6

5.1 EXPERIMENTAL SETUP

• All four agents used the same harness, prompts, and budgets; the harness supplied paper materials and artifact links while leaving allocation and scheduling choices to each agent.Agents ran in a thin ReAct-style loop without model-specific guidance, and their self-assessments were claims for the auditor to grade.Source: §5.1 · p. 6

5.2 RESULTS

• 73 of 400 agent-paper cells were confirmed reproduced; DeepSeek-V4-Flash led with 14 of 34 Run, 9 of 33 Retrain, and 4 of 33 Reimplement papers, while mean scores ranged from 5.13 to 3.03.The four agents spanned a factor of 1.7 in mean audit score, from 5.13 for DeepSeek-V4-Flash to 3.03 for MiniMax-M2.7.Source: Figure 4 · p. 7, §5.2 · p. 7
• At Reimplement, 13 of 132 cells reproduced and every agent had its lowest reproduction rate and mean score; Retrain had 37 of 132 faithful pipelines outside tolerance versus 24 of 136 at Run.At Retrain, DeepSeek-V4-Flash and Muse Spark 1.2 reached partial result or better on 22 of 33 and 23 of 33 cells, respectively, versus 9 of 33 for MiniMax-M2.7; confirmed reproduction rates ranged from 15% to 27%.Source: §5.2 · p. 7
• Non-reproduced cells used 29% of their allocation on average; across 400 cells, agents spent 1,888 of 10,208 H100-hours, while self-reports claimed 128 reproductions against 73 audited.The 396 runs that started cost $4,835 ($12.21 per run; $66 per confirmed reproduction), with GPU time accounting for 78% of the total. MiniMax-M2.7 claimed 43 reproductions and had 10 confirmed; DeepSeek-V4-Flash claimed 19 and had 27 confirmed.Source: §5.2 · p. 7

5.3 FAILURE MECHANISMS

• Reimplemented-but-not-checked was the most common process failure, at 63 of 400 cells, including 38 at Reimplement; it and Wrong experiment accounted for 62 of 132 Reimplement cells.Wrong artifact measured occurred 38 times, and on 7 papers released model cards or configurations stated settings that differed from the paper. Five of MiniMax-M2.7’s eight Run-tier build failures traced to missing aarch64 package wheels on GH200 nodes.Source: §5.3 · p. 8, Figure 5 · p. 8
• Authors’ entrypoints were described in 58 of 73 successful runs versus 160 of 299 labeled failures; rebuilt components were checked against paper-reported quantities before compute.When patching released code, successful runs disclosed the patch and showed the graded computation was unchanged; reduced-scale smoke tests appeared in 45 successful and 148 failed runs.Source: §5.3 · p. 8

5.4 COMPUTE DOES NOT BIND REPRODUCTION

• Mean audit scores fell with paper compute band for all four agents; in the 96 H100-hour band, 1 of 48 cells reproduced after spending 6.2% of granted compute.The authors hypothesize that higher-compute papers are harder to reproduce because they more often require full training rather than evaluation of a released checkpoint; grants were sized to pinned experiments.Source: §5.4 · p. 8, Figure 6 · p. 9

Attributing a miss to the agent

• Without human reproductions, the authors cannot distinguish agent error from seed or hardware variance in 76 faithful runs outside tolerance.They did not pursue exhaustive human reproductions because of the effort required and the difficulty of confirming intended practice with each paper’s authors.Source: §6 · p. 9

Contamination

• Because agents postdate the papers and may have trained on their code, memorized fixes could count as reproductions and inflate released-code-tier results.Source: §6 · p. 9

Scope

• The benchmark covers 100 papers from one venue and year, pins one experiment under a 96 H100-hour cap, and excludes 15 papers requiring paid or closed APIs.The excluded papers were identified in the audit pool (Figure 3).Source: §6 · p. 9

Agents

• With at most one run per agent-cell, rates are point estimates; the roster omits Claude, GPT, and Gemini because of cost, limiting agent comparisons.The authors leave close pairs within a tier unranked; repeated runs and closed frontier models are left to groups with access, while the harness already drives one hosted agent over its API.Source: §6 · p. 9

7 CONCLUSION

• The best reproduction rate falls from 41% for Run-tier papers with code, data, and weights to 15% for Reimplement-tier papers rebuilt from the paper.The benchmark covers four agents and 400 agent-paper cells.Source: §7 · p. 9
• Method implementations unchecked against paper numbers recede among the two strongest agents, but measuring the wrong released artifact does not.Failed attempts use only part of their GPU grants on average, leaving most granted compute unspent.Source: §7 · p. 9
• The released harness, frozen rubric, graded transcripts, and benchmark enable repeated measurement on fresh conference papers as agents improve.The paper frames independent verification as an open question for systems that already carry ideas through experiments to manuscripts (Lu et al., 2024; Schmidgall et al., 2025; Tang et al., 2025).Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

• Run bundles make grading reproducible from the released artifacts, but stochastic rollouts do not reproduce transcripts turn for turn.The benchmark, code, batch scripts, run bundles, and transcript viewer are released; the dataset fields and splits are documented in Appendix F.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

• Generative AI assisted dataset extraction, implementation, grading, writing, figures, and literature review, with authors reviewing extracted fields and citations.The agent models and grading model are objects of study, not author assistance; the auditor is compared with human graders in Appendix D.Source: AI USE STATEMENT · p. 10
• Generative AI was not used to create synthetic datasets, develop theoretical models, formulate or prove mathematical claims, or shape the authors’ hypotheses, benchmark design, or audit protocol.The authors state that they reviewed all AI-assisted work and take responsibility for the final paper content and artifacts.Source: AI USE STATEMENT · p. 10

A.1 AGENT LOOP

• Each agent episode runs one paper with one model, meters tool calls and budget, and ends on a stop condition before requesting a final report in a fixed schema.Near the context limit, compaction elides older tool results; episodes can end at the round cap, budget exhaustion, context overflow, or unrecoverable error.Source: §A.1 · p. 32

A.2 TOOLS

• Agents use seven tools but have no file-reading tool or general web search, relying on shell commands and URLs they can already know or construct.The only search exception is third-party library-documentation search through context7.com; failed fetches return an error without stopping the episode.Source: §A.2 · p. 32

A.3 HARDWARE SETUP AND COMPUTE ACCOUNTING

• GPU charges equal GPU count × held wall time × hardware multiplier; queue time is free, idle reasoning time counts, and the harness rejects allocations exceeding the remaining budget.All runs used the single GH200 profile with a 1.0 multiplier; the allocation is acquired on the first call and re-acquired near its wall-time cap.Source: §A.3 · pp. 32–33, Figure 8 · p. 33

A.4 ENVIRONMENT DETAILS

• Every shell step runs in a read-only CUDA 12.9 and cuDNN Apptainer image on aarch64 GH200 hosts, requiring installed dependencies to have aarch64 wheels or build from source.Only designated workspace and evidence paths are writable; the paper reference is read-only, and the host home is not mounted.Source: §A.4 · p. 33

A.5 RUN BUNDLE AND REPORT FORMAT

• Run bundles contain logs, trajectories, environment locks, patches, GPU-step logs, plans, reports, token counts, and GPU-utilization samples, but writable evidence logs have no integrity check.Auditors are instructed to treat bundle contents as unverified claims and trace reported numbers to the commands that produced them.Source: §A.5 · p. 33
• The final tool-free turn produces schema-checked report.json; the harness sets the dispatched paper ID, records exit status, preserves malformed outputs with validation errors, and grades from the report and evidence.After an episode, the harness prunes the virtual environment and oversized workspace files.Source: §A.5 · p. 34

A.6 PROMPT TEMPLATES

• The task prompt is a 290-line template with fourteen placeholders under nine headings; it omits match_target.config, requiring agents to derive the target-producing configuration from the paper.Rendering fails for unfilled fields, while each field has a fallback string stating that information is absent.Source: §A.6 · p. 34

B MODEL AND SERVING DETAILS

• Three open-weight agent models are served on the cluster with vLLM, while Muse Spark 1.2 uses the Meta Model API; agent GPU allocations are metered separately from serving GPUs.A sweep runs as one batch job per model per tier, with papers processed on the same node.Source: B MODEL AND SERVING DETAILS · p. 34, Table 3 · p. 35
• Grades of record come from a separate grading client using claude-sonnet-5, with one agent loop per run bundle capped at 25 rounds; regrading overwrites each run’s stored verdict.The paper prices GPU time at $2 per H100-hour and grader tokens at $2 per million input tokens and $10 per million output tokens.Source: B MODEL AND SERVING DETAILS · p. 35

B.1 RESULTS BY AGENT, TIER, AND COMPUTE BAND

• Table 5 reports reproduced cells, mean audit score, and grant spent across 400 agent-paper cells by agent, tier, and compute band; reproduction requires an audit score of at least 8.Grant spent is H100-hours used divided by hours granted, and the compute band is the paper’s H100-hour grant.Source: §B.1 · p. 37, Table 5 · p. 37

C.1 GRADING PROCEDURE

• Each audit evaluates one agent’s attempt at a paper’s central claim against its pinned match target, using the run-bundle manifest and frozen rubric to assign the highest evidence-supported quality band before integrity adjustment.The auditor explores the bundle with four run-directory tools, and Figure 12 reports tool rounds by verdict.Source: §C.1 · p. 37, Table 6 · p. 38, Figure 12 · p. 38

C.2 SCORE SCALE AND VERDICTS

• A score of 1 becomes unverifiable when the auditor reports no evidence that the code ran, while a claim without a checkable target or a printed number without an execution trace can score 0 or 1.A printed number is not treated as verified, and the rubric sets a low-score prior for runs with no evidence.Source: §C.2 · pp. 37–38, §C.1 · p. 37
• The rubric was frozen three days after the dataset freeze, and every pinned-auditor grade was produced after that freeze.Reports that required artifacts such as weights or datasets could not be obtained earn no credit, because papers with missing artifacts had already been removed from the frozen split.Source: §C.2 · p. 38
• The rubric’s top bands require the correct experiment and protocol, a met C1 criterion, and no flags for score 10; score 9 allows one unseeded stochastic-metric draw within the bar, and score 8 allows only specified caveats.The reproduction bar is never loosened at grade time.Source: §C.2 · p. 38, Table 7 · p. 39
• Among 400 cells, the audit-score distribution has modes at 4 (72 graded runs) and 6 (60), plus peaks at 0 (55 integrity breaks) and 8 (52 accepted reproductions with minor caveats).The 25 cells without a graded run count as 0; score-4 runs execute something gradeable but fall well short of the bar, while score-6 runs measure the right quantity with sound provenance but miss the bar.Source: §C.2 · p. 39, Figure 13 · p. 39

C.3 DETERMINISTIC VERDICT DERIVATION

• Integrity checks precede score-to-band mapping; the code also enforces the rubric’s C4 rule and caps reproduced-score claims without verified execution.A score of 0 is reserved for integrity breaks, including high-flag caps and reports that contradict their own evidence.Source: Figure 15 · p. 40, §C.3 · pp. 40–41

D AUDITOR VALIDATION

• An auditor’s reproduced verdicts reached an F1 of 0.82 against human labels across 396 runs, with graders blind to auditor scores and verdicts.Three human graders each scored one run from its bundle, pinned claim, and rubric text.Source: D AUDITOR VALIDATION · p. 41

E.1 SOURCE CORPUS

• The source corpus used well-formed arXiv IDs from the NeurIPS 2025 index and matched supplements by OpenReview forum identifier, never by title.The bundles covered 3,414 papers; 3,410 had a .tex file and 1,547 had a supplement.Source: §E.1 · pp. 41–42

E.2 STAGE I: ARTIFACT-VERIFIED CLASSIFICATION

• A uniform sample of 1,000 corpus papers with LaTeX source was classified by MiniMax-M2.7.The classifier prompt included rendered paper text, supplement-manifest excerpts, and LaTeX source.Source: §E.2 · p. 42
• Each classification used an agent loop over ten tools, and every artifact signal recorded how a tool established it.The tools covered GitHub, Hugging Face, URL fetching, and supplement reading.Source: §E.2 · p. 42

E.3 DETERMINISTIC EXTRACTION

• Scores of 0, 1, and 2 map to Easy (Run), Medium (Retrain), and Hard (Reimplement); higher scores map to Artifact-Blocked.Missing code adds 2, missing weights adds 1, and missing data adds 3 unless the dataset is a standard public benchmark; score 3 remains Hard when data are available or standard.Source: §E.3 · p. 42

E.4 STAGE II: COMPUTE CAPPING AND STRATIFIED SELECTION

• The 200-row audit pool was drawn from 620 verified rows, balanced equally across the three tiers and spread across compute bands.The pool was twice the benchmark size to allow audited rows to be dropped and replaced; Table 8 summarizes it by tier and pre-audit compute band.Source: §E.4 · p. 42, Table 8 · p. 42
• After human audit, three over-cap papers were replaced with cheaper same-tier rows, leaving no evaluation paper above the 96 H100-hour cap.Table 8 still includes pool rows above the cap because the cap was applied to the evaluation split after auditing.Source: §E.4 · p. 42, Table 8 · p. 42
• The band quotas raised the pool’s median compute to 14.78 H100-hours from 3.95 for eligible rows, while cheapest-first selection favored low-cost rows within oversubscribed bands.Cheapest-first changed nothing in four of 12 tier-by-band strata, where quotas included every eligible row; the other eight were oversubscribed. The 13 Run-tier rows in the 0–8 band were in the bottom ninth percentile of 148 eligible rows, while all 13 Retrain-tier rows in that band had zero-hour estimates among 215 eligible rows with a 1.0-hour median.Source: §E.4 · p. 42

E.5 STAGE III: HUMAN AUDIT

• Human auditors reviewed evidence for each of four availability signals, and disagreements replaced model verdicts before scores and tiers were recomputed.For Reimplement-tier rows, GPT-5.5 and two independent web-search agents rechecked whether a route to the pinned metric existed; a wall was upheld only if both agents found none, and one paper remained eligible because an agent found a route.Source: §E.5 · p. 42

E.6 THE EVALUATION AND DEVELOPMENT SPLITS

• Papers whose minimal experiment required a paid or closed model API were dropped; the evaluation split was then selected by the Stage II rule, and development took the cheapest remaining eligible papers in each tier.The passage refers to Appendix F for split sizes but does not provide them.Source: §E.6 · p. 43
• Figure 17 plots audited compute for the 100 evaluation papers by tier, while Figure 18 groups those papers by arXiv posting month and tier.Figure 17 uses one dot per paper on a log axis, marks each tier’s median, and places below-floor papers in the leftmost slot.Source: Figure 17 · p. 43, Figure 18 · p. 43, §E.6 · p. 43

F DATASET FIELDS

• The eval split contains 100 papers and the disjoint dev split 14; rows expose metadata and links, not full paper text.The eval split is marked eval and the development split dev; Figure 19 illustrates a SharpZO row, with long text fields truncated and other fields verbatim.Source: F DATASET FIELDS · p. 44, Table 9 · p. 44, Figure 19 · p. 45
• Each row fixes 25 top-level fields and four nested objects, including claim evidence, artifact availability, the match target, and compute-estimate fields.The schema also includes the central claim, reproduction instruction, tier, split, and estimate-audit fields.Source: F DATASET FIELDS · p. 44, Table 10 · p. 45
• Among the 100 evaluation papers, match bars comprise 62 point estimates, 21 baselines, 14 thresholds, one delta, and one range; point estimates use author tolerance or 5% relative.The one evaluation row without a bar kind is marked unset.Source: F DATASET FIELDS · pp. 46–47
• Tables 11 and 12 list each evaluation and development paper’s pinned match target; the development split totals 26.89 audited H100-hours.The lists include compute band, audited estimate, metric, bar, and target value; the development set is for harness development only.Source: F DATASET FIELDS · p. 47, Table 11 · p. 48, Table 12 · p. 48

G FAILURE-MODE TAXONOMY

• The failure-mode taxonomy has nine fixed modes and assigns each run its dominant mechanism; a new mode requires at least two runs, and labels do not change scores or verdicts.The language model labels runs from transcripts, reports, and audit verdicts, rationales, and flags.Source: §G.1 · p. 48

G.1 LABELING PROCEDURE

• Thirty-six runs were re-read and relabeled for out-of-vocabulary labels or conflicts with pinned verdicts, including ten valid labels that disagreed with verdicts.Seven of those ten were labeled near-miss-partial on disqualified runs; four labels were retained, including a provenance mismatch involving a sibling checkpoint.Source: §G.1 · pp. 48–49

G.2 THE FAILURE MODES

• Reimplement-without-validating means writing a method without checking it against a paper reference; environment-fights involve repeated install, compile, or binary-compatibility failures across runs on the same host.The latter can prevent the experiment from producing a valid number.Source: §G.2 · p. 49
• Reproduced-clean reaches the pinned bar, while near-miss-partial runs the right pipeline but measures outside tolerance; rubric anchors classify evidence signatures, while audited scores spread around them.Table 14 names nine primary failure modes, one per run.Source: §G.2 · p. 49, Table 14 · p. 50

G.3 FAILURE TAXONOMY EXAMPLES

• 40.82% Best All (ep 20) fell below the 65.9% Target (paper ProCo+MORE); the agent’s 54.21% Our CE-only baseline also trailed the 61.4% Paper CE baseline.The gap versus target was -25.08%, and the agent continued building on its baseline despite its lower score.Source: §G.3 · p. 50

G.2 THE FAILURE MODES

• Relabeling moved 36 runs; one audited reproduction was labeled Wrong artifact measured, leaving 72 runs in the Reproduced row versus 73 audited reproductions, with the difference in Muse Spark 1.2 Run-tier runs.Figure 21 splits the failure modes by agent and tier.Source: §G.2 · p. 50, Figure 20 · p. 51, Figure 21 · p. 51

G.3 FAILURE TAXONOMY EXAMPLES

• On paper 2505.10039 (Chen et al., 2025a), the agent chose a different arm before coding and never measured the pinned one; on paper 2511.20906 (Chun et al., 2025), incompatible versions broke the released dataset converter.The latter agent trained on the broken output before discovering that saved abs actions failed to reproduce the expert trajectory from the correct initial state.Source: §G.3 · p. 52

H ANNOTATED RUNS

• The appendix annotates ten of 400 runs selected from 37 nominations and verified against transcripts, covering repairs, host adaptations, provenance work, substitutions, near misses, and invalid numbers.Four annotated runs are by DeepSeek-V4-Flash and five are Run-tier; each vignette identifies its audit verdict, primary mode, and decisive transcript round.Source: H ANNOTATED RUNS · pp. 52–53

H.1 THE RELEASED TAG CRASHED ITS OWN EVALUATION SCRIPT ON FIVE MULTICLASS DATASETS

• All four TabDPT Table 1 values came within 0.002 of the paper after a patch fixed a retrieval branch that crashed on five multiclass datasets.The agent pinned v1.1.14 because the repository HEAD loaded a later checkpoint; the auditor recorded the disclosed patch as a low-severity provenance note.Source: §H.1 · p. 53

H.2 A RENAMED BRANCH MADE LAYERNORM SCALING DEAD CODE IN THE RELEASED REPOSITORY

• A renamed LNS branch matched nothing until the agent fixed both comparisons, confirmed different logits before training, and reproduced the pinned result with a disclosed single-seed deviation.The auditor validated the patch against the paper’s reference implementation and noted the narrower gap between the two arms.Source: §H.2 · p. 55

H.3 A HYPERPARAMETER THE RELEASED TRAINING SCRIPT NEVER FORWARDED

• Five post-fix seeds measured 98.88 to 98.92 for GraphMixer on TAMI, and the auditor marked Muse Spark’s Retrain-tier run reproduced-clean.The released command silently used γ=0.5 instead of the pinned γ=0.9 because the decoder argument was not forwarded; the round-81 report listed the changes and drew no audit flags.Source: §H.3 · pp. 56–57, Figure 28 · p. 57

H.4 A HOST SUBSTITUTION CHECKED AGAINST THE REFERENCE PATH BEFORE TRAINING

• DeepSeek-V4-Flash’s Default MoE run was not reproduced: a torch attention substitution caused NaN losses, and the baseline stopped at 205 iterations.The agent traced the NaNs to a boolean-mask convention; the auditor noted medium-severity concerns about comparing training loss and the under-trained baseline, despite a low-severity loss-identical substitution check.Source: §H.4 · p. 58, Figure 30 · p. 58

H.5 A CPU-ONLY TORCH INSTALLED BY THE AUTHORS’ REPOSITORY, CAUGHT BY RE-PROBING CUDA

• Muse Spark’s Run-tier Toto reproduction was marked reproduced-clean after a CUDA probe caught the authors’ editable install replacing CUDA torch with a CPU-only wheel.For the 96-instance live batch, the agent used a retry loop that halved batch samples on out-of-memory errors while preserving the paper’s sample count and context; the auditor reproduced the same aggregate from raw predictions.Source: §H.5 · p. 59, Figure 32 · p. 60

H.6 A MEMORY-FORCED REWRITE OF THE SCORER, CHECKED TWO WAYS, ON THE WRONG EXPERIMENT

• DeepSeek-V4-Flash’s CQ-DINO evaluation was cross-validated but disqualified because it evaluated the released checkpoint rather than the pinned 24-epoch training run.The chunked scorer matched pycocotools on 3,000 images and the in-runner metric on six numbers; on another paper, the same agent’s one-example-checked vLLM port was also disqualified.Source: §H.6 · pp. 60–61

H.7 A MASKED CRASH CAUGHT FROM THE LOG, THEN A SHORTFALL REPORTED AS MEASURED

• MiniMax-M2.7 completed SemCoT training and inference after redirecting a read-only Triton cache, but reported 39 percent accuracy from two temperatures as measured.The auditor considered the run faithful to the pipeline but flagged evaluation on 200 bundled SVAMP examples rather than the paper’s stated 300.Source: §H.7 · p. 62

H.8 A FLAG STUCK ON, DIAGNOSED AND THEN REPORTED AGAINST THE PAPER’S FLAG-OFF ROW

• Qwen3.6-27B reported 2.86 percent against a 3.25 percent TTPL target, but measured the separate MVDF-on row rather than the pinned MVDF-off row.The released argparse bool parser treated the string False as true; the agent diagnosed the flag yet still reported reproduction, and the auditor raised two medium flags.Source: §H.8 · p. 63, Figure 38 · p. 64

H.9 ATTENTION REPLACED BY ZEROS AFTER THE FAITHFUL PATCH MISSED THE BAR; H.10 THE LITERAL UPDATE RULE DIVERGED, SO A STANDARD OPTIMIZER WAS REPORTED IN ITS PLACE

• Two runs were disqualified after substituting easier operations for the pinned methods: zero-output attention for linearized attention, and torch SGD for DLRT heavy-ball.The first bypass avoided the fitted matrix product; in the second, the literal rule reached 11.33 and 11.80 percent after two epochs while SGD reached 81.45 and 83.26, then was reported as reproduced despite 94.29 percent compression.Source: §H.9 · pp. 64–65, §H.10 · pp. 65–66

I SETUP DETAILS; J PROMPTS

• The repository contains the dataset pipeline, agents, serving layer, sweep scripts, and run bundles; prompts require durable run evidence, and the auditor verifies cited numbers against files rather than trusting self-reports.The auditor defaults to a low score when evidence is absent; sweeps grade runs during execution and later overwrite those grades with a pinned pass.Source: I SETUP DETAILS · p. 69, §J.1 · p. 70, §J.2 · p. 70

J.4 AUDIT RUBRIC

• The rubric checks quantitative targets against C1’s relative tolerance and qualitative targets by whether the stated relation holds.Source: §J.4 · p. 78

J.5 CLASSIFIER PROMPT

• The Stage-I prompt asks an artifact-availability classifier to inspect the paper, verify external artifact evidence with tools, identify an MRE, and return one JSON object.Source: §J.5 · p. 79