SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback

• The authors identify costly expert effort and difficulty covering diverse domains, subdomains, and algorithmic combinations as barriers to constructing high-quality scientific coding data.Source: §1 · p. 1
• SciWalker combines scientific-library interfaces and operation modes into operators, samples operator chains as workflow cues, and uses them to guide LLM generation of problem statements, reference solutions, and tests.Execution feedback supports iterative repair, while quality review filters generated problems.Source: §1 · p. 1
• The dataset contains 8,178 scientific coding problems across five domains, with 37,820 computational substeps integrating multiple interdependent operations.The domains include mathematics, physics, chemistry, biology, and materials science.Source: §1 · p. 2
• SciCode subproblem accuracy rose 9.9 percentage points, from 29.3% to 39.2%, after reinforcement learning on Qwen3.5-9B with SciWalker-generated data.The training also improved DS-1000 by 17.3 percentage points and MATH-500 by 6.0 percentage points.Source: §1 · p. 2

• Scientific and programming data construction draws on curated research problems, scientific literature, and code resources, including approaches that augment problems or generate questions and tasks.Examples include SciCode, SciInstruct, WildSci, UniScientist, and Magicoder.Source: §2 · pp. 2–3
• Figure 2 presents the multi-turn scientific coding problem format.Source: Figure 2 · p. 3
• Scaling multistep scientific coding data requires diverse computational combinations, coherent tasks with clear scientific objectives, and reference solutions with tests.Source: §2 · p. 3
• SciWalker addresses these requirements through scientific operator composition and context design, using execution feedback to repair and select generated problems.Source: §2 · p. 3

3 Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback

• The framework combines LLMs with programmatic workflows in four stages: operator construction, graph sampling, problem generation, and validation, repair, and final refinement.Scientific coding tasks use a main problem followed by ordered subproblems whose specifications and previously generated code support stepwise implementation.Source: §3 · p. 3

3.1 Operator Construction

• Scientific library APIs are combined with operation modes such as single-case execution, parameter sweeps, and result checking to define distinct operators for graph construction and chain sampling.Treating different uses of the same API as distinct operators expands the operator space and supports more diverse workflows.Source: §3.1 · pp. 3–4

3.2 Operator Graph Sampling

• Within each subdomain, directed operator connections are scored for input/output compatibility and workflow, and random walks sample chains from retained successors as workflow cues.Source: §3.2 · p. 4

3.3 Scientific Problem Generation

• For each sampled chain, the framework assesses workflow coherence, constructs and reviews a scientific context, then generates tasks, substeps, tests, and reference and oracle implementations.The problem statement is revised for consistency with the generated code and tests.Source: §3.3 · p. 4

3.4 Validation, Repair, and Final Refinement

• Validation checks interfaces, dependencies, substep and end-to-end tests, and agreement between reference and oracle implementations; failed candidates receive feedback-driven repair or are discarded.Validated problems undergo quality refinement and renewed validation before retention.Source: §3.4 · pp. 4–5

3.5 SciWalker Implementation and Dataset Statistics

• The implementation organizes scientific Python APIs into subdomain-specific catalogs spanning five domains and 32 subdomains, with 17,920 operators and 772,178 graph edges.Operator chains of lengths 3–15 provide workflow cues for problem generation.Source: §3.5 · p. 5
• The dataset retains 8,178 high-quality problems across 32 subdomains, with 3–6 substeps per problem, 37,820 substeps total, and a mean of 4.62 substeps.The substep count does not directly correspond to operator-chain length; the model structures substeps around the scientific objective and computational dependencies.Source: Figure 3 · p. 4, §3.5 · p. 5

4 Reinforcement Learning for Scientific Coding

• The evaluation trains with reinforcement learning on execution-based rewards, using GSPO with substep execution rewards.Source: §4 · p. 5

4.1 Policy Optimization with GSPO

• GSPO optimizes groups of complete substep responses using length-normalized sequence ratios and a clipped token objective, with a shared advantage across response tokens.The token formulation is equivalent to GSPO in objective value, clipping conditions, and gradient.Source: §4.1 · pp. 5–6

4.2 Constructing Substep Training Samples

• Each trajectory yields one task-and-prior-code prompt with a reasoning/code response per step; only current response tokens are optimized, with reward 1 for passing all tests and 0 otherwise.Each substep retains its own reward; rewards from subsequent steps are not accumulated backward.Source: §4.2 · p. 6

5.1 RL Training and Evaluation Settings

• Evaluation covered 16 benchmarks across scientific computing and code generation, code repair and execution, and reasoning and knowledge.The best checkpoint was selected using SciCode evaluation scores, and each trained model used one fixed checkpoint across benchmarks.Source: §5.1 · p. 6

5.2 Results Comparison

• 9.9 points raised Qwen3.5-9B SciCode subproblem accuracy from 29.3% to 39.2%, surpassing Qwen3-32B and GPT-OSS-120B while matching GPT-5 mini.Figure 4 compares local evaluations with externally reported model scores; local evaluations cover 288 subproblems with scientific background.Source: Figure 4 · p. 7, §5.2 · p. 7
• GSPO gains extended to other coding tasks, including DS-1000 (+17.3 points), LiveCodeBench Code Generation v6 (+13.0), and APPS Introductory (+15.5).Source: §5.2 · p. 7, Table 2 · p. 8
• GSPO also improved all five reasoning and knowledge benchmarks, including MATH-500 by 6.0 points, alongside gains on code-repair tasks.Reported code-repair gains include 13.4 points on HumanEvalFix-Python and 47.8 on HumanEvalFix-JavaScript.Source: §5.2 · p. 7
• 15 of 16 benchmarks improved, and the average score rose from 72.6% to 82.0%.Table 2 reports scores averaged over three evaluations and percentage-point changes relative to the baseline.Source: §5.1 · p. 6, Table 2 · p. 8, §5.2 · p. 8
• PPO improved SciCode by 3.3 points but improved only eight benchmarks, degraded seven, and raised the overall mean by 1.2 points.The authors suggest critic-based credit-assignment errors may make PPO optimization less stable and limit generalization.Source: §5.1 · p. 6, §5.2 · p. 8

• SciWalker synthesizes scientific coding problems through operator-chain sampling and execution feedback.Source: §6 · p. 8
• The framework produced 8,178 high-quality problems spanning five scientific domains and 32 subdomains.Source: §6 · p. 8
• GSPO training on Qwen3.5-9B raised SciCode subproblem accuracy from 29.3% to 39.2%, a 9.9-percentage-point gain.The conclusion also identifies human-authored open-source code, including Python libraries, as a valuable resource for synthetic coding data.Source: §6 · p. 8

A.1 Operators and Chain Sampling

• The sampled chain has 11 operators and 10 edges, spanning adaptation, sensitivity analysis, filtering, solving, residual computation, and stability checking.It is drawn from 35 operators formed by combining scipy.optimize.fsolve with 35 operation modes.Source: §A.1 · p. 11

A.2 From Scientific Context to a Concrete Problem

• The chain receives a problem-generation potential score of 7, and its scientific context is accepted after the first selection and review round.DeepSeek-V4-Flash generates the main problem, five substeps, student and oracle implementations, and tests in one call.Source: §A.2 · p. 11
• The example’s final problem estimates chemical-reaction concentration-decay parameters, filters outliers by a residual threshold, refits, and returns parameters plus an approximate-Hessian condition number.The parameters are A, k, and B; the table summarizes the problem’s five substeps.Source: §A.2 · p. 11, Table 3 · p. 11

A.3 Execution Validation and Feedback-Driven Repair

• Full validation follows three repair rounds in which errors and the prior complete problem are fed back; four test snippets remain, with one modified in the first repair.The example illustrates joint revision of the problem statement, implementations, and tests.Source: §A.3 · p. 11

A.4 Quality Grading, Inclusion, and Revalidation

• The example earns an overall quality score of 8 and is included; its nine model calls cover assessment, context review, generation, repair, polishing, and grading.Polishing leaves the repaired content unchanged, which then passes revalidation.Source: §A.4 · p. 12

B Stage-Wise Retention Rates During Generation

• 45,346 candidates are discarded during potential assessment, leaving 21,543 chains, or 32.2% of the initial candidates, for later stages.The passage attributes this reduction to chains that do not naturally form tasks with scientific rationale.Source: B Stage-Wise Retention Rates During… · p. 12
• 21,527 chains proceed after context selection and review, a reduction of 16 from the previous stage, including context reselection.The model designs a scientific setting and reviews the solution objective and operation combination.Source: B Stage-Wise Retention Rates During… · p. 12
• 17,166 problems pass execution validation, a reduction of 4,361 from the previous stage and 25.7% of initial candidates.The framework checks generated statements, reference implementations, and tests, then feeds errors back for repair.Source: B Stage-Wise Retention Rates During… · p. 12
• 8,178 high-quality problems are retained, 12.2% of the initial 66,889 candidate operator chains, after polishing and quality grading.Review considers scientific naturalness, task difficulty, implementation complexity, and test strength.Source: B Stage-Wise Retention Rates During… · p. 12, Table 5 · p. 12

Problem statement

• The task is to implement a Python function using Mannella’s leapfrog method to solve the Langevin equation for an optically trapped microsphere in gas from a given initial condition.Source: Problem statement · p. 13

Scientific background

• The background identifies ω0 as resonant frequency, τp as momentum relaxation time, vrms as root mean square velocity, and ζ(t) as normalized white noise.It states that the stochastic differential equation can be rewritten.Source: Scientific background · p. 13
• The method description specifies a step size ∆t and states that ∆W is sampled from a Gaussian distribution with mean zero.Source: Scientific background · p. 13

Function interface and input/output specification

• The function takes initial position and velocity, start time, total simulation time, step count, momentum relaxation time, resonant frequency, and root mean square velocity.The named parameters are x0, v0, t0, steps, taup, omega0, and vrms.Source: Function interface and input/output… · pp. 13–14
• The specification names the final microsphere position as an output, while the iteration requires passing updated position and velocity to the next step.Source: Function interface and input/output… · p. 14
• Baseline evaluation uses Inspect with scientific background, enabled thinking, a 32,768-token generation limit, temperature 0.6, top_p 0.95, top_k 20, a 300-second timeout, and three independent runs.Source: Function interface and input/output… · p. 14

C.2 Results and Generation Lengths Across Three Evaluations

• None of the baseline’s three evaluations passes this SciCode subproblem.Generation length counts thinking and final-answer tokens; one response is truncated, and none encounters an execution timeout.Source: §C.2 · p. 14

C.3 Persistent Looping and State Updates

• In the third evaluation, a repeated four-line dependency-import loop spans 308 iterations and consumes 82.11% of thinking text.Generation exhausts the 32,768-token budget and ends with empty final code.Source: §C.3 · p. 14
• In the first two evaluations, the baseline updates position with v_new but leaves velocity unchanged, so the next iteration reads the old velocity.This violates the stepwise update requirement.Source: §C.3 · p. 14

D Rollout Probability Correction and Numerical Details

• Rollout-correction weights are fixed before policy updates and multiply token losses without changing rewards or group-relative advantages.For stability, Equation 3 is computed in log space with its exponent capped at 10; loss is averaged over valid tokens per response and then over substep samples.Source: D Rollout Probability Correction and… · p. 15

E Scientific Python Libraries

• Operator catalogs draw on APIs from 79 Python libraries and package namespace groups across five scientific domains to provide workflow cues for problem generation.Table 7 lists the libraries grouped by domain.Source: E Scientific Python Libraries · p. 15, Table 7 · p. 15