PaperlayerView exact revision on arXiv

Source-linked AI summary

Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

Jingjie Ning, Guojiang Zhao, Chen Xu, Shanshan Zhong, Xiaochuan Li, Ji Zeng, Guolin Ke

arXiv:2610.00313v1cs.AIcs.SE

TL;DR

Scientific coding agents need to assess whether revised programs satisfy written scientific requirements, not merely whether they run. Rules to Tools provides callable checks of those requirements during revision. Across two SciCode task-ID cohorts, the checks produce more complete repairs overall, while effects vary by task and cohort.

  • Problem

    Scientific coding agents must assess whether programs meet scientific requirements, since a simulation can run while violating a physical law or prescribed equation.

    Source: §1 · p. 2

  • Method

    Rules to Tools packages public scientific requirements as callable checks of the current program, evaluated alongside matched repair comparisons and independent grading.

    Source: §1 · p. 2

  • Results

    Across two task-ID cohorts, complete repair is 29/30 with checks and 26/30 with text, while outcomes and agent-side costs vary by cohort.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The reported evidence is of task-dependent repair outcomes and agent-side costs for prepared checks, rather than a uniform benefit.

    Source: §6 · p. 9

Abstract

from arXiv · show

Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group receives a callable implementation. Across two task-ID cohorts, complete repair is 26/30 with text and 29/30 with the prepared checks. Three task IDs favor tools, one favors text, and eleven tie. The eight-ID cohort scores 13/16 versus 15/16, with a task-cluster bootstrap 95% interval of [-12.5, 43.75] percentage points for the difference. The larger shared-definition SciCode cohort ties at 13/24 per group. Five development-exposed tasks with alternate starting programs score 3/10 versus 7/10. The tool group favors tasks 17, 77, and 11; initial checks flag task 17 and report no violation for tasks 77 and 11. Task 37 favors text and has no initial reported violation. A fresh source-through-Python arm also reaches 15/16, matching the dedicated command's aggregate. In a matched PDE comparison, detailed text scores 23/24 and checks score 24/24, with 31.2% lower reported model output for checks. Agent-side output savings vary by cohort, while public CPU use rises in both task-ID cohorts. These results measure task-dependent repair outcomes and agent-side costs with prepared checks.

1 INTRODUCTION

Scientific coding agents need to check whether revised programs satisfy scientific requirements, not merely whether they run. Rules to Tools (R2T) provides callable checks, and matched SciCode comparisons report differing outcomes across cohorts and tasks.

1 INTRODUCTION

  • Scientific coding agents need to assess whether programs satisfy scientific requirements, since a simulation can run while violating a physical law or prescribed equation.Benchmarks test these demands through numerical calculations, molecular simulations, and scientific software tasks (Tian et al., 2024; Hu et al., 2026; Hang et al., 2026; Anand et al., 2026; Duston et al., 2025).Source: §1 · p. 2
  • R2T packages public scientific requirements as callable checks that report measurements or errors on a program under revision.A separate benchmark evaluator grades the finished program.Source: §1 · p. 2
  • Task-level gains occur on one initially flagged task and two tasks whose initial checks report no violation.The final grader measures complete repair across the whole task.Source: §1 · p. 2
  • Matched SciCode groups share written checks, starting programs, model, and budgets, while only the tool group receives the callable implementation.Additional arms examine an initial report and checker source delivered through ordinary Python.Source: §1 · p. 2
  • Across two task-ID cohorts, repair outcomes are 13/16 versus 15/16 and 13/14 versus 14/14 for text and tools, respectively.The cohorts use task-specific written definitions and executable checks frozen before matched repair.Source: §1 · p. 2

2 RELATED WORK

Prior work studies feedback-driven revision, scientific coding and scientific feedback, executable specifications, and callable scientific knowledge. R2T examines prepared checks of public scientific requirements during revision, with independent grading and resource measurement.

2 RELATED WORK

3 EXECUTABLE SCIENTIFIC CHECKS

R2T turns public scientific requirements into measurements of candidate programs, with checks that return observations and status for use during revision. The checks are engineered representations of declared criteria and probes, not hidden final evaluators.

3 EXECUTABLE SCIENTIFIC CHECKS

  • A diagnostic maps a candidate and task specification to a relation, probe, observation, and status.The status distinguishes a discrete verdict, continuous measurement, or unavailable observation.Source: §3 · p. 3
  • Specification equivalence means the written representation describes the same declared scientific criteria and probes that the tool implements.Feedback for a candidate is generated by execution.Source: §3 · p. 3
  • Tool-generated residuals, error locations, and convergence status depend on the candidate; text-supported agents can obtain observations by implementing and executing corresponding procedures.The supplied tool makes that procedure immediately callable, while independent evaluators retain hidden solutions and final scores.Source: §3 · p. 4
  • A benchmark binding connects reusable diagnostic patterns to concrete files, entry points, numerical quantities, parameters, output paths, and physical regimes.Table 2 maps benchmark artifacts and support designs.Source: §3 · p. 4
  • Table 1 traces public requirements through probes and admitted error to measurements of candidate behavior.Its rows start with a public-task relation; the probe and admitted error define the measurement, and the returned value describes the candidate.Source: Table 1 · p. 4, §3 · p. 4
  • For scalar PDE grids, checks use physical grid spacing and an interior crop, while numerical error makes the result a continuous observation.In coupled flow, passive observers recover a vector field; the tool measures divergence and momentum-equation curl, and ambiguous captures are unavailable.Source: §3 · p. 4
  • Checks are authored or reviewed by the study team; the seven-task extension qualified checks on accepted programs and deliberately constructed faults.Across final checks, accepted-program evaluations passed 28/28, additional held-out evaluations passed 8/8, and targeted seeded faults were detected 7/7; one of seven failed starts triggered a selected check.Source: §3 · pp. 4–5
  • Agents inspect a candidate, obtain observations, save a revision, and check the saved artifact again; text-supported agents use ordinary Python and tool-supported agents call the supplied checker.At budget exhaustion, the current saved artifact is submitted.Source: §3 · p. 5

4 EVALUATION DESIGN

The evaluation compares support designs across scientific benchmarks, including matched SciCode repair cohorts and complementary studies. It defines benchmark-specific task assignments, budgets, endpoints, and cost records.

4 EVALUATION DESIGN

  • Matched SciCode groups share written checks, probes, tolerances, applicability conditions, and ordinary code execution, while the tool group receives a prebuilt implementation.Other benchmark studies use different support designs, including separate flow rule-card and implementation delivery and historical written, checker-only, and combined conditions.Source: §4 · p. 5
  • The original programming panels include twelve SciCode and twelve PDE tasks; historical panels include eighteen MDArena tasks and fifteen AInstein rule units from five repositories.Table 2 maps benchmarks to artifacts and support designs.Source: §4 · p. 5, Table 2 · p. 5
  • The first task-ID cohort uses eight archived failed starts absent from prior checker bindings; a disjoint extension uses the seven remaining eligible task IDs.Two additional access conditions on the first eight IDs provide either an initial checker report or frozen checker source for ordinary Python execution.Source: §4 · p. 5
  • Five alternate SciCode failures use different starting programs with a frozen enhanced checker and shared expanded written definitions; a seven-task flow cohort uses four trajectories per group.The flow cohort follows a frozen textual development screen.Source: §4 · p. 5
  • Controlled programming runs use DeepSeek Flash, three saved revisions, 196,608 output tokens, and 360 CPU-budget seconds per trajectory.Each public execution has a 120-second wall cap and 3 GB memory cap; the primary response cap is twenty-four.Source: §4 · p. 5
  • Complete-task success requires the benchmark-native endpoint, including passing every scored SciCode subtask and meeting PDE relative-grid-error and solve-time thresholds.Native partial scores use task-macro averaging; PDE reports numerical error and runtime separately, while output-token and CPU totals include failed attempts.Source: §4 · p. 6

5 RESULTS

Across matched SciCode comparisons, repair outcomes vary by task and support design; agent-side costs also differ across cohorts and benchmarks.

Held-out SciCode repair

  • 15/16 versus 13/16 programs were repaired by tools and text, respectively, across eight held-out SciCode IDs; the task-macro difference was 12.5 percentage points.The task-cluster bootstrap 95% interval was [−12.5, 43.75] percentage points; two tasks favored tools, one favored text, and five tied.Source: §5 · p. 6, Figure 2 · p. 7
  • The seven-ID extension adds one tool-only repair on task 11 (1/2 to 2/2), while six other tasks tie at 2/2 in both arms.Its bootstrap interval is [0, 21.43] points; exact sign tests give p = 1 for each cohort and p = 0.625 for their combined 3:1 pattern.Source: §5 · p. 7

Additional SciCode comparisons

  • The shared-definition SciCode cohort ties at 13/24, while development-exposed alternate programs score 3/10 with text and 7/10 with checks.The latter comparison uses five development-exposed tasks with different starting artifacts and matched written definitions.Source: §5 · p. 7
  • Development-exposed checks raise task-macro native subtask completion from 61.27% to 90.48%, although three task-level success contrasts favor tools and two tie.On task 80, all programs remain incomplete, but text trajectories pass 6/7 and 5/7 subtasks and both tool trajectories pass 6/7.Source: §5 · p. 7

Flow comparison

  • Under the selected flow comparison’s 24-response cap, text completes 16/28 trajectories and tools 21/28; five task signs favor tools, one favors text, and one ties.The comparison changes delivery form and capture gate; selection on text-probe failure favors the tool group.Source: §5 · p. 8

Checker access

  • On task 77, the initial-report arm scores 0/2, source through Python 1/2, and the dedicated-tool arm 2/2; these counts do not rank access forms.The source-through-Python arm produces 40 completed public checker reports and uses fresh continuations.Source: §5 · p. 8

Resource use

  • +8.7% total tokens and +38.6% public CPU accompany tools in the eight-ID cohort; across both cohorts, reported tool output is at least 4.9% lower.Across both cohorts, text and tool output totals are at least 1,327,295 and 1,262,059 tokens, respectively, while public CPU rises from 779.43 to 1,459.88 seconds (+87.3%).Source: §5 · pp. 8–9
  • Development-exposed SciCode saves 22.9% reported output, and detailed-text PDE uses 1,062,099 versus 730,723 output tokens, a 31.2% reduction with checks.Under the 24-response flow cap, text uses 1,649,905 output tokens and tools use 1,786,229.Source: §5 · p. 9

6 CONCLUSION

Rules to Tools supplies executable checks of public scientific requirements during revision. The results show task-dependent repair outcomes and costs across the evaluated designs.

Conclusion

  • Rules to Tools packages public scientific requirements as callable checks of the program under revision.The matched SciCode design holds written criteria, starting programs, model, and budgets fixed while varying access to a prepared implementation.Source: §6 · p. 9
  • Across two SciCode task-ID cohorts, complete repair is 26/30 with text and 29/30 with checks, while other SciCode comparisons vary by cohort and starting program.The shared-definition cohort ties at 13/24, and five development-exposed alternate programs score 3/10 versus 7/10.Source: §6 · p. 9
  • Matched PDE checks score 24/24 versus 23/24 with detailed text and use 31.2% fewer reported model output tokens; checker authoring time was not recorded.The conclusion also reports that complete-repair effects and agent-side costs vary by cohort.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement records model settings and dates for controlled and historical runs, and identifies materials included in the anonymous supplement.

Reproducibility statement

  • Controlled SciCode, PDE, and flow runs used DeepSeek-V4.1-Flash on 17 and 24–25 September 2026 UTC, with temperature 1, top-p 0.95, and high-effort thinking.Historical MDArena/AInstein runs used deepseek-v4-flash in August; extra checker-only attempts used deepseek-flash in September.Source: REPRODUCIBILITY STATEMENT · p. 9
  • The anonymous supplement contains per-phase settings, SciCode task inputs and prompts, public rules, and checker sources.Source: REPRODUCIBILITY STATEMENT · p. 9

AI USE STATEMENT

The paper reports its AI-assisted research workflow, defines the response cap for the primary comparison, and describes public-definition diagnostics for scientific requirements. These diagnostics include numerical and field-reconstruction checks with specified probes and tolerances.

AI USE STATEMENT

  • AI assisted literature retrieval, experiment design and execution, analysis, visualization, and drafting; authors retain final review responsibility.Source: AI USE STATEMENT · p. 9

A RESPONSE CAPS AND SENSITIVITY

  • The primary comparison cap is 24 responses; a separate 25-response sensitivity continues eligible unfinished states while carrying forward prior costs.Both conditions receive the same stopping display, which shows one response remaining at the 24-response cap.Source: A RESPONSE CAPS AND SENSITIVITY · p. 11

B PUBLIC-DEFINITION DIAGNOSTIC CONSTRUCTION

  • The Numerov diagnostic probes three energy and angular-momentum configurations on grids of 65, 129, and 97 points, with a 10^-8 residual tolerance.A separate check tests that the outer derivative remains negative after positive normalization, as required by the wavefunction convention.Source: B PUBLIC-DEFINITION DIAGNOSTIC… · p. 11
  • Rotation checks compare candidate transformation coefficients with an independently evaluated basis at degrees 1 and 2, using a deliberately transposed rotation as a negative control.The synthetic checker test preserves identity, inverse, and norm checks while violating field reconstruction; higher-degree recurrence correctness is outside its measurement.Source: B PUBLIC-DEFINITION DIAGNOSTIC… · p. 11
  • The density-response check refines radial quadrature to order 128 when estimates differ beyond 10^-8 and flags unresolved integrals.Candidate outputs are compared with the numerical reference at 10^-6 tolerance; the tested smooth complex frequencies avoid singular real-axis quadrature.Source: B PUBLIC-DEFINITION DIAGNOSTIC… · p. 11
  • Reciprocal-cell checks compare h⊤G−1h with the returned ring key and separately test sorted reflections and unique ring membership.A companion check tests the public matrix equation with the stated unit conversion.Source: B PUBLIC-DEFINITION DIAGNOSTIC… · p. 12

C TASK-ID-HELDOUT SCICODE REPAIR

In the task-ID-heldout SciCode cohort, prepared checks yield a positive task-macro repair difference, but task effects are mixed and uncertainty spans zero. The comparison holds task materials and repair conditions fixed and grades complete-module repair.

C TASK-ID-HELDOUT SCICODE REPAIR

  • The eight held-out task IDs were fixed before paired repair, and all archived starting modules failed the complete-task endpoint.Both groups received the same starting module, public task, written definitions, model, and budgets; final modules were scored on all native substeps across the stated environments.Source: C TASK-ID-HELDOUT SCICODE REPAIR · p. 12
  • 12.5 percentage points is the task-macro tool-minus-text repair difference, with a task-cluster bootstrap 95% interval of [−12.5, 43.75] points.Two tasks favor tools, one favors text, and five tie; removing task 77 leaves zero difference across the other seven tasks.Source: Table 5 · p. 12, C TASK-ID-HELDOUT SCICODE REPAIR · pp. 12–13
  • Removing Task 77 gives a zero difference across the other seven tasks, with interval [−21.43, 21.43] points; Task 77’s starting program passes all five supplied public checks.Its two successful tool trajectories later use ordinary Python to revise periodic wrapping and pressure calculations.Source: C TASK-ID-HELDOUT SCICODE REPAIR · p. 13

D INDEPENDENT SCICODE EXTENSION AND CHECKER ACCESS

The independent seven-ID SciCode extension shows a positive task-macro repair difference driven by task 11, while checks require more public CPU time. Additional access arms test an initial report and checker source run through ordinary Python.

D INDEPENDENT SCICODE EXTENSION AND CHECKER ACCESS

  • All seven extension task checkers passed selected public-program evaluations, and seven deliberately introduced faults triggered their intended checks.Only task 34's naturally failed starting module triggered a selected check; false-alarm qualification, targeted-fault detection, and starting-failure coverage are distinct measures.Source: D INDEPENDENT SCICODE EXTENSION AND… · p. 13
  • 7.14 percentage points is the task-macro repair difference, with a task-cluster bootstrap 95% interval of [0, 21.43] points.Task 11 supplies the single difference in complete repairs; the starting module passes both selected checks.Source: Table 6 · p. 13, D INDEPENDENT SCICODE EXTENSION AND… · p. 13
  • Tools use 826.73 public CPU seconds versus 322.53 for text, an additional 504.20 seconds.The reported 21.8% model-token saving is a lower bound because one interrupted text request has no returned usage.Source: D INDEPENDENT SCICODE EXTENSION AND… · pp. 13–14
  • Source-through-Python and dedicated-command arms each reach 15/16 repairs across the original eight IDs, despite different task-level outcomes.The source arm runs the checker with ordinary Python; two-repetition task rates cannot rank access forms, and the dedicated-command and source-arm mean interaction wall times are 329.9 and 299.5 seconds per trajectory.Source: D INDEPENDENT SCICODE EXTENSION AND… · p. 14

E DEVELOPMENT-EXPOSED ALTERNATE-ARTIFACT STUDY

The alternate-artifact study reports more complete repairs with frozen checks than with expanded written rules, alongside lower reported output tokens. Separate historical panels compare support forms and retain benchmark-specific partial-score meanings.

E DEVELOPMENT-EXPOSED ALTERNATE-ARTIFACT STUDY

  • Across five development-exposed tasks with alternate starting programs, expanded written rules yield 3/10 repairs and frozen checks 7/10, a 40-point task-mean difference.The exploratory task-cluster interval is [10, 70] points; the exact two-sided sign test on the three non-tied task effects gives p = 0.25.Source: E DEVELOPMENT-EXPOSED… · pp. 14–15
  • On alternate task 22, tools repair 2/2 tasks and text repairs 0/2, although the starting artifact has no violation on its four supplied rotation checks.Both successful tool trajectories revise translation coefficients beyond those checks.Source: E DEVELOPMENT-EXPOSED… · p. 15
  • Expanded written rules use 1,306,590 reported output tokens versus 1,006,870 for frozen enhanced checks, a 22.9% reduction.Charging the full 65,536-token cap for one checker-arm request with missing usage yields a reduction of at least 17.9%.Source: E DEVELOPMENT-EXPOSED… · p. 15

F HISTORICAL SUPPORT-FORM PANELS

  • Historical written-rules versus rules-plus-checks repairs are 5/18 versus 7/18 on MDArena and 4/15 versus 7/15 on AInsteinBench.Each historical task–rule unit has one execution; the study keeps benchmark strata and all three support forms separate.Source: F HISTORICAL SUPPORT-FORM PANELS · p. 15
  • Native partial scores differ by benchmark: MDArena reports 70.81%, 78.80%, and 76.90%, while AInsteinBench reports 98.73%, 92.86%, and 99.48% for rules, checker-only, and combined support.These benchmark-native quantities have separate meanings; the AInsteinBench checker-only value includes an unparsed log with a zero fallback.Source: F HISTORICAL SUPPORT-FORM PANELS · p. 15

G MUTUALLY EXCLUSIVE PDE SUPPORT DELIVERY

In the selected flow-task comparison, executable checks had higher combined task-mean success than text, while detailed PDE checks matched text completion with lower reported output. The additional cohorts and support forms qualify those results by selection, checker-gate design, and unquantified usage.

G MUTUALLY EXCLUSIVE PDE SUPPORT DELIVERY

  • The flow comparison combines a rule card with text versus callable checks without the rule card, and includes seven cases selected after textual-probe failure.The frozen screens retained 80 scalar-PDE cases and 29 flow cases; selection on text-probe failure favors the tool arm in the selected flow cohort.Source: G MUTUALLY EXCLUSIVE PDE SUPPORT… · p. 16
  • Text and tool capture gates differ—pointwise magnitude matching versus relative L2 discrepancy—so they can admit different fields and alter access to vector-equation feedback.Diagnostics retain discretization error without a binary accuracy threshold, and receive no final-score credit.Source: G MUTUALLY EXCLUSIVE PDE SUPPORT… · p. 16
  • 57.14% versus 75.00% combined task-mean success favored tools; five flow tasks favored tools, one favored text, and one tied.Across flow waves, text recovered 9/14 and 7/14 assignments, versus 11/14 and 10/14 with tools.Source: Figure 5 · p. 17, G MUTUALLY EXCLUSIVE PDE SUPPORT… · p. 17
  • Under the twenty-five-response sensitivity analysis, success was 20/28 with text and 24/28 with tools; the reported-output ordering remains unresolved.Charged execution was 1,867.72 versus 2,003.48 CPU-budget seconds, with higher tool CPU use; two text requests lacked returned usage, spanning the reported output difference.Source: G MUTUALLY EXCLUSIVE PDE SUPPORT… · p. 17

H PDE GENERATION COSTS BY SUPPORT FORM

  • 23/24 detailed-text versus 24/24 checker completions; checks used 730,723 versus 1,062,099 reported output tokens, a 31.2% reduction, while brief text also completed 24/24.Public CPU totals for detailed text and checks were 930.92 and 932.46 seconds; brief text used 875.67 seconds versus 932.46 for checks.Source: Figure 6 · p. 18, H PDE GENERATION COSTS BY SUPPORT FORM · p. 18
  • The separate 29-task PDEAgentBench ceiling screen passed 29/29 initial native endpoints; only one matched revision pair tied, with 28 pairs unstarted.Source: H PDE GENERATION COSTS BY SUPPORT FORM · p. 18

I EXTENDED TOOL-GROUP EVALUATION

  • Extended checker-only attempts covered ten MDArena and eight AInstein failed units; seventeen finished, while one stopped MDArena unit retained its grade.Cumulative reported output was 2,721,127 tokens for MDArena and 1,403,186 for AInsteinBench; usage for the interrupted request is unquantified.Source: Table 8 · p. 18, I EXTENDED TOOL-GROUP EVALUATION · p. 18
Loading 2610.00313v1…