PaperlayerView exact revision on arXiv

Source-linked AI summary

Coupled Harness--Model Evolution for Scientific Equation Discovery

Pengze Li, Jiaquan Zhang, Encheng Su, Yunbo Long, Zihang Zeng, Xinping Liu, Han Zhang, Junchi Yu, Shixiang Tang, Aoran Wang, Xi Chen

arXiv:2610.17740v1cs.AI

TL;DR

LLM equation-discovery systems rely on both executable tools and a policy for using them, while existing approaches typically adapt only one. This paper evolves the tool harness from diagnosed failures and trains the policy on successful trajectories, reporting gains across benchmark families.

  • Problem

    Equation-discovery systems can fail because the tool library lacks a needed capability or because the policy misuses available tools, while prior approaches typically adapt only one component.

    Source: §1 · pp. 1–2

  • Method

    The framework diagnoses failures to add tools or revise harness rules, then fine-tunes a controller on successful trajectories using a frozen evolved harness and screened training formulas.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 5

  • Results

    The system's LSR-Synth aggregate of 79.1 exceeds the best external result of 63.6, and it reaches 63.8 pooled on SRSD-Feynman-240.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    The results support improving both scientific-agent instruments and search behavior from discovery evidence, with the coupling effective on shifted inputs and equation families outside LLM-SRBench.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to discover scientific equations from experimental data. Such a discovery system has two parts that must work together: a library of executable computational tools and the model's tool-use policy. The tools supply the solving capability, from diagnosing structure in the data to fitting candidate equations; the policy plans and orchestrates their use. Yet existing systems adapt only one part, keeping either the toolset or the model fixed, which leaves the corresponding failures uncorrectable. We develop a coupled evolution framework that improves both parts from the same verifiable fit-error signal. In the tool-evolution phase, an agent grows an executable tool library from a small set of generic numerical operations by diagnosing its own failures: it authors a new tool when a capability is missing and revises its selection rules when the right tool went unused. In the policy-evolution phase, a small open model (roughly 3B active parameters) is fine-tuned on successful tool-use trajectories collected with the frozen library, on training formulas screened to be disjoint from all evaluation benchmarks. Ablations show the two phases are complementary: with no tool library at all the same model solves almost nothing, the tools account for most of the gain, and trajectory training adds a further gain that is largest on the most difficult problems. The full system solves 79.1/71.3\% of LSR-Synth problems at fit tolerances $10^{-2}$/$10^{-3}$, reaches 63.8\% pooled accuracy on the distributionally different SRSD-Feynman panel, and obtains a normalized CP3-28 aggregate of $-7.27$. These results suggest a broader recipe for scientific agents: let the model and its instruments improve each other under verifiable feedback.

1 INTRODUCTION

LLM equation-discovery systems depend on both an executable tool library and a policy that uses it; failures can arise from missing capabilities or misuse. This paper couples tool evolution with policy training under verifiable fit-error feedback.

1 INTRODUCTION

  • A discovery system can fail because a needed tool is absent or because its policy misuses an available tool, requiring different fixes.The library bounds the reachable hypothesis space; the policy chooses tools, arguments, and actions on returned results.Source: §1 · pp. 1–2
  • Existing approaches typically adapt either the tools or the model, leaving the other class of failures uncorrectable.The authors identify equation discovery as a setting where coupling matters: needed equation families are hard to anticipate, and tool value depends on whether the policy invokes it.Source: §1 · p. 2
  • The framework evolves the harness from diagnosed failures, then fine-tunes the policy on successful tool-use trajectories with the evolved harness frozen.A missing capability prompts a new tool; a wrong selection prompts revisions to tool-presentation or selection rules.Source: §1 · p. 2
  • The authors report transfer across benchmarks with different equation distributions and matched ablations attributing most gains to the library and further tool-use reliability gains to policy training.They describe this as the first framework, to their knowledge, coupling failure-driven tool evolution with trajectory-based policy training for scientific equation discovery.Source: §1 · p. 2

2 RELATED WORK

Related work includes tool evolution around fixed models, training for fixed toolsets, model–asset co-evolution, and symbolic regression. This work combines failure-driven evolution of an executable SR tool library with trajectory-based model training under externally verifiable evaluation.

2 RELATED WORK

  • One line evolves agent scaffolding or tools while holding the model fixed.Examples include reusable skill libraries and methods that let models author and curate tools.Source: §2 · pp. 2–3
  • A complementary line trains models to use fixed toolsets, while symbolic-regression work includes direct expression search and LLM-driven search.The cited SR approaches span genetic programming, neural-guided search, sparse identification, and LLM agents that propose or refine equations.Source: §2 · p. 3
  • Recent model–asset co-evolution jointly changes policies and capability assets, mostly in open-ended or game-like settings scored by self-play or proxy signals.Source: §2 · p. 3
  • Within SR, nearby systems distill trajectories, adapt at test time, or configure existing solvers; this work couples failure-driven executable-tool evolution with trajectory training.The authors state they are unaware of a prior SR system combining those two components.Source: §2 · p. 3
  • The authors distinguish their evaluation setting through hard external fit criteria, out-of-distribution generalization, and memorization checks.They say these requirements motivate their diagnosis, acceptance, and data-provenance machinery.Source: §2 · p. 3

3 METHOD

The method frames equation discovery as a search for an expression that fits observations and generalizes to withheld test data. It then updates the tool harness and the model’s tool-use policy in separate phases, preserving the data boundary between fitting, selection, and testing.

3.1 PROBLEM SETUP AND OVERVIEW

  • A candidate equation is fitted on Dfit and solves an instance only when its relative prediction error stays within the benchmark tolerance on every test observation.The error is normalized by the target magnitude, making the criterion dimensionless across outcome scales.Source: §3.1 · p. 4
  • The system pairs a model with a harness of executable tools, interaction rules, and a final-candidate selection rule, updating one while holding the other fixed.During a trajectory the model and harness read only Dfit; σ may read Dsel afterward, while Dtest is reserved for external scoring.Source: §3.1 · p. 4

3.2 PHASE A: FAILURE-DRIVEN HARNESS EVOLUTION

  • Phase A holds the model fixed and evolves the harness over a development stream, beginning with a generic tool set and diagnosing failures.A missing capability prompts a new or revised tool, while a suitable but misused tool prompts changes to interaction or selection rules.Source: §3.2 · pp. 4–5, Figure 1 · p. 5
  • Harness edits are accepted only after new tools pass a promotion check and all previously solved instances remain solved.The resulting harness is frozen for Phase B and evaluation.Source: §3.2 · p. 5

3.3 PHASE B: POLICY EVOLUTION BY TRAJECTORY DISTILLATION

  • Phase B freezes the harness and fine-tunes a small open controller on teacher trajectories from synthesized formulas screened against evaluation benchmarks.The controller is Qwen3.6-35B-A3B with approximately 3B active parameters; the screening tests symbolic, affine-reparameterization, and numeric-probe equivalence.Source: §3.3 · pp. 5–6
  • Training retains teacher steps whose actions promoted a candidate, improved public-data fit error, or repaired a malformed tool call.This is cross-model distillation from GPT-5.5 trajectories to the separately initialized Qwen controller.Source: §3.3 · p. 6
  • After each trajectory, σ selects one candidate without using test data, applying Bayesian information criterion on public data for accuracy-at-tolerance benchmarks.The criterion includes a complexity term, favoring simpler equations among comparably fitting candidates.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments compare the evolved-tool, trained-controller system across four equation-discovery panels, using matched conditions to isolate the contributions of tool evolution and trajectory training. The system leads the reported LSR-Synth comparison and performs strongly on the distributionally different SRSD-Feynman and CP3-28 panels.

4.1 BENCHMARKS AND METRICS

  • The evaluation spans LSR-Synth, LSR-Transform-111, SRSD-Feynman-240, and CP3-28, with the latter two held out from tool evolution and model training.Panels use their native metrics; the LLM-SRBench accuracy criterion keeps every test point, unlike published trimmed-5% baselines.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 MAIN RESULTS

  • 79.1 on LSR-Synth exceeds the best external result of 63.6 under the all-points convention, and the system also reaches 77.5%/75.7% on LSR-Transform-111.The LSR-Synth comparison is conservative against published results using trimmed-5% scoring; Table 1 covers both benchmarks and their respective fit thresholds.Source: §4.2 · p. 7, Table 1 · p. 7
  • 63.8 pooled accuracy on SRSD-Feynman-240 exceeds uDSR's 44.6, while CP3-28 mean log10NMSE is −7.27 versus QLattice's −6.20.The dummy-variable tiers require rejecting irrelevant inputs; the CP3-28 comparison is a reference comparison using different source scoring conventions.Source: Figure 2 · p. 7, §4.2 · p. 8, Table 2 · p. 8

4.3 ABLATIONS

  • With no tool library, the backbone solves 0% of LSR-Synth; evolving tools raises raw candidate quality, and training the controller adds gains, largest on difficult problems.On the fixed evolved toolkit, training raises SRSD hard from 30.0 to 37.0 and CP3 from −6.69 to −6.99; Table 3 reports raw candidate quality before selection.Source: §4.3 · p. 8

5 DISCUSSION

The discussion traces capability growth during tool evolution and checks transfer across input-range shifts and differing formula families. The trajectory shows gains associated with broader tested function-family coverage, while the evaluation distinguishes all-points from trimmed-5% scoring.

OOD behavior and formula-distribution checks

  • The OOD all-points scores are higher than corresponding in-domain scores, but the passage links this pattern to fewer tasks with tiny minimum |y|; trimmed-5% scores are lower on OOD.The formula-level overlap firewall is separate, and SRSD-Feynman and CP3-28 are held out from tool evolution and trajectory generation; the system remains strong on those different formula families.Source: §5 · pp. 8–9

Dynamics of capability growth

  • As the library grows from 23 to 32 tools, no-BIC all-points LSR-Synth accuracy rises from 41.1% to 61.2%, and BIC reselection raises the final point to 77.5%.On CP3-28, mean log10NMSE moves from −4.76 to −6.86, then −6.91; under trimmed-5% scoring, traversal reaches 86.1% and BIC reaches 95.4%. Larger jumps follow additions that widen the tested function-family set.Source: §5 · p. 9

6 CONCLUSION

The conclusion frames equation discovery as a joint model-and-instrument process: diagnosed failures expand available tools, while successful episodes train the controller. The authors report that this coupling remains effective on shifted inputs and equation families outside LLM-SRBench.

6 CONCLUSION

  • Figure 3 tracks in-domain and shifted-input Acc@10−2 across epochs, with tool counts, no-BIC and BIC-selected points, and a fixed-toolset reference.Its horizontal line is SR-Scientist (GPT-OSS-120B), scored using a like-for-like trimmed-5% convention.Source: Figure 3 · p. 9
  • Failures from missing capabilities and failures from misusing existing tools call for different corrections: tool additions and policy improvements, respectively.The conclusion presents the two phases as complementary responses to these failure classes.Source: §6 · pp. 9–10
  • The work uses synthetic and public benchmark data, with no human subjects or personal data.The paper identifies benchmark leakage and memorization as integrity questions it addresses.Source: ETHICS STATEMENT · p. 10
  • The paper specifies its evolution loop, training-data provenance and overlap firewall, and principal training recipe; it says scripts will be released with the code.The stated reproduction materials include the shared evaluation procedure and end-to-end pipeline.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 CP3-28: ERROR-MAGNITUDE AGGREGATE AND PER-METHOD HEAD-TO-HEAD

The appendix reports a normalized mean log10NMSE aggregate for CP3-28 and a paired per-problem comparison against published methods.

A.1 CP3-28: ERROR-MAGNITUDE AGGREGATE AND PER-METHOD HEAD-TO-HEAD

  • The CP3-28 aggregate is mean log10NMSE after normalizing each problem’s MSE by test-target variance and averaging across 28 problems.The benchmark publishes per-problem MSEs but no common aggregate.Source: §A.1 · p. 13
  • After candidate selection, the trained controller with the library records −7.27 mean log10NMSE, versus −6.91 for deterministic traversal.Lower mean log10NMSE is better; the aggregate averages normalized MSE across 28 problems.Source: §A.1 · p. 13
  • Table 4 gives per-problem win/tie/loss breakdowns against 12 published methods with paired values, with ties at source order-of-magnitude resolution; after the common candidate-selection pass, trained-controller/evolved-library and deterministic-traversal aggregates are −7.27 and −6.91.Source: §A.1 · p. 13

A.2 SRSD-FEYNMAN EVALUATION UNDER THE ORIGINAL BENCHMARK METRIC

The SRSD-Feynman evaluation uses the benchmark’s R2 > 0.999 accuracy measure across standard and dummy-variable tiers. Results favor the system on harder and all dummy-variable tiers, while uDSR leads on standard easy and medium tiers.

A.2 SRSD-FEYNMAN EVALUATION UNDER THE ORIGINAL BENCHMARK METRIC

  • 58.0% on the hard tier exceeds PySR’s published best of 38.0%, and 51.0% on merged hard tiers exceeds DSR’s 25.0%.The reported accuracy criterion is R2 > 0.999.Source: §A.2 · pp. 13–14
  • The system exceeds every published system on all three dummy-variable tiers, where a solution must ignore irrelevant input columns.The evaluation pools 240 problems across standard and dummy-variable difficulty tiers.Source: §A.2 · pp. 13–14
  • uDSR remains ahead on the standard easy and medium tiers, with reported accuracies of 100.0% and 75.0%.The paper reports verified zero overlap between its training corpus and the SRSD formulas.Source: §A.2 · p. 14

A.3 THE SELF-EVOLUTION LOOP IN FULL

The Phase A loop carries the library and remaining harness rules across batches, adding tools only after a held-out promotion gate and checking edits against previously solved problems.

A.3 THE SELF-EVOLUTION LOOP IN FULL

  • Across batches, the loop carries the tool library L, remaining harness H, and set S of problems solved so far.H denotes the interaction interface and selection rule.Source: §A.3 · p. 14
  • A newly authored tool must pass a held-out promotion gate before entering the persistent library.Source: §A.3 · p. 14
  • Before committing an edit, the loop reruns the accumulated suite to preserve previously solved capability.The acceptance check is on line 21 of Algorithm 1.Source: §A.3 · p. 14

A.4 THE FROZEN TOOLKIT BY ROLE

The frozen toolkit spans the main steps of equation discovery, from diagnosing data structure to fitting and checking candidate equations. Its evolution loop diagnoses failures and either adds a missing capability or revises tool-selection rules before freezing the library and harness.

A.4 THE FROZEN TOOLKIT BY ROLE

  • The frozen library contains 33 tools: 10 diagnostics, 18 discovery fitters, 4 verifiers, and a restricted Python builder.These tools cover the empirical loop of looking at data, hypothesizing a form, fitting it, and checking it.Source: §A.4 · p. 14
  • Diagnostics characterize data without committing to a form, discovery tools propose and fit forms, and verification tools score or stress-test candidates.Examples include periodicity and dependence checks, generic and mechanism fitters, held-data fitting, component-library posteriors, and stability probes.Source: §A.4 · p. 14
  • After a failed attempt, the loop diagnoses a missing capability versus a mistaken selection, then builds or repairs a tool or edits the harness accordingly.New tools pass a promotion gate, and proposed updates are accepted only after an acceptance check; the resulting library and harness are frozen.Source: §A.4 · p. 15

A.5 LEAKAGE CHECKS

The appendices describe safeguards against benchmark leakage, unsuccessful extensions, and the procedures used to train, select, and evaluate the controller. They specify overlap checks for training formulas and information boundaries for trajectory examples, alongside fixed candidate-selection and evaluation rules.

A.5 LEAKAGE CHECKS

  • The library is developed on LLM-SRBench Phase-A; SRSD-Feynman and CP3-28 are external panels excluded from tool evolution and model training.Mechanism fitters use public-row-fitted textbook families without benchmark metadata or targets, generic fitters enumerate forms without response lookup tables or benchmark-specific constants, and only oracle tools access isolated reference pools; probes are reported separately from benchmark scores.Source: §A.5 · pp. 15–16

A.6 WHAT DID NOT WORK

  • Large-scale reinforcement learning and online tool authoring produced no improvement or measurable gain, so training uses positive-signal distillation and tool creation stays offline.The trained controller also did not replace the frontier model in Phase A; the resulting system is asymmetric, with a stronger model growing the harness and a smaller open model operating it.Source: §A.6 · p. 16

A.7 THE OVERLAP FIREWALL

  • The overlap firewall rejects training candidates that match evaluation formulas under canonical symbolic equality or affine reparameterization.Affine-equivalent formulas count as the same problem because fitting can absorb the rescaling.Source: §A.7 · p. 16
  • A numeric fingerprint adds a safety net: candidates matching evaluation formulas on shared probe points after scale normalization are rejected.Panels without public ground-truth formulas are excluded from training, and their formulas therefore do not enter the firewall's evaluation set.Source: §A.7 · p. 16

A.8 INFORMATION-BOUNDARY INVARIANTS

  • Training examples must come from complete normal episodes that avoid held-out rows, selection labels, ground-truth formulas, and private-reward feedback to the model.The scalar reward remains bookkeeping-only and is not returned to the model's context.Source: §A.8 · pp. 16–17

A.9 CANDIDATE-SELECTION PROCEDURE

  • Candidate-selection rules are fixed per metric family and applied symmetrically, using only public observations and never test observations.For trained-controller candidates, promotion over the deterministic tool anchor requires a fixed-margin improvement on both public subsets.Source: §A.9 · p. 17

A.10 TRAINING RECIPE

  • The controller uses a frozen 33-tool catalog, and its adapter is trained on positive-signal GPT-5.5 teacher trajectories from 72 generated tasks screened by the overlap firewall.The base model has approximately 3B active parameters; adapter training uses token-level cross-entropy on retained action tokens with frozen base weights.Source: §A.10 · p. 17
  • All conditions use the frozen catalog and shared promotion rule; fitting feedback, selection, and offline test/OOD scoring use distinct data roles.BIC reselection on accuracy-at-tolerance panels uses public observations only; infrastructure failures count as misses without reducing denominators, and the tool-free GPT-5.5 reference is contextual.Source: §A.10 · p. 17

Related papers

Loading 2610.17740v1…