PaperlayerView exact revision on arXiv

Source-linked AI summary

SRHarness: A Harness for Agentic Symbolic Regression

Zihan Yu, Shixuan Zhou, Hao Huang, Jingtao Ding, Yong Li

arXiv:2609.35501v1cs.AIcs.LGcs.SC

TL;DR

Agentic symbolic regression increasingly relies on LLMs to guide long-horizon discovery, making runtime support alongside models and search strategies a central concern. SRHarness provides composable scientific actions, persistent hypothesis state, and trajectory lifecycle management. On LLM-SRBench it improves numerical and symbolic discovery under matched backbones, including 93.69% symbolic accuracy on LSR-Transform versus 62.16% for SR-Scientist.

  • Problem

    Agentic symbolic regression needs runtime support for heterogeneous scientific operations and evolving hypotheses, but existing systems typically build such mechanisms around their search strategies.

    Source: §1 · p. 1

  • Method

    SRHarness provides reusable runtime support through composable scientific actions, persistent scientific state with compact model-facing views, and trajectory lifecycle management.

    Source: §1 · p. 2

  • Results

    SRHarness consistently improves numerical and symbolic results under matched backbones, reaching 93.69% symbolic accuracy on LSR-Transform versus 62.16% for SR-Scientist with DeepSeek-v4-flash-0731.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support treating structured runtime support for scientific actions, accumulated hypotheses, and long-horizon search as a distinct factor in agentic symbolic regression.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent agentic symbolic regression approaches increasingly rely on large language models to analyze data, select scientific operations, and refine hypotheses over long search trajectories. In such systems, performance depends not only on the underlying model and search strategy, but also on the runtime infrastructure that supports scientific search. We introduce SRHarness, a domain-specific harness for agentic symbolic regression built around three mechanisms: composable scientific actions that provide a common interface over raw, transformed, and candidate-derived quantities; persistent scientific state that retains evaluated hypotheses and exposes compact model-facing views; and trajectory lifecycle management that coordinates continuation, branching, restart, and termination. On LLM-SRBench, SRHarness consistently improves both numerical generalization and symbolic recovery under matched LLM backbones. With DeepSeek-v4-flash-0731, it achieves 93.69% symbolic accuracy on LSR-Transform, compared with 62.16% for SR-Scientist, and retains 72.97% accuracy on an anonymized variant that removes scientific descriptions and variable semantics, versus 39.64% for SR-Scientist. Under the same DeepSeek-v4-flash-0731 backbone, SRHarness also substantially outperforms Codex (72.97% vs. 20.72%) and reaches performance comparable to Codex with GPT-5.5, while simply providing Codex with the same scientific tools does not reproduce this advantage. These results show that effective agentic symbolic regression depends not only on models or tools, but also on structured runtime support for organizing scientific actions, accumulated hypotheses, and long-horizon search.

1 Introduction

Agentic symbolic regression makes the runtime infrastructure part of long-horizon equation discovery, but existing systems often tie harness mechanisms to their search strategies. SRHarness provides reusable runtime support and improves numerical and symbolic results under matched backbones.

1 Introduction

  • Agentic symbolic regression makes execution infrastructure increasingly consequential as LLMs autonomously select scientific operations over long-horizon equation discovery.Unlike traditional SR algorithms, recent LLM-based approaches use scientific knowledge, reasoning, and code generation; agentic methods make the LLM a controller selecting operations from intermediate evidence.Source: §1 · p. 1
  • Existing agentic SR systems implement parts of this support, but mechanisms designed around individual strategies can confound comparisons between harnesses and search strategies.Scientific operations need consistent representations and evaluation semantics so hypotheses from heterogeneous operations can be compared, revisited, revised, and reused as evidence emerges.Source: §1 · pp. 1–2
  • SRHarness combines composable scientific actions, persistent scientific state, and trajectory lifecycle management without prescribing a particular search strategy.The actions operate over raw, transformed, and candidate-derived quantities; the state retains evaluated hypotheses and evidence; lifecycle management coordinates continuation, branching, restart, and termination.Source: §1 · p. 2
  • 93.69% symbolic accuracy on LSR-Transform vs. 62.16% for SR-Scientist under matched DeepSeek-v4-flash-0731.SRHarness also reaches 72.97% on anonymized LSR-Transform vs. 39.64% for SR-Scientist, and 72.97% vs. 20.72% for Codex under the same backbone.Source: §1 · p. 2

2 Related Work

Prior LLM-guided SR methods place the LLM within a predefined search procedure, while agentic SR systems let it control analysis and search. SRHarness is presented as reusable runtime support distinct from those strategies.

2 Related Work

  • LLM-guided SR uses LLMs to propose or modify equations within predefined procedures, whereas recent agentic systems use the LLM as a controller choosing operations from intermediate evidence.Examples of the former include LLM-SR, LaSR, SGA, and SR-LLM; agentic examples include SR-Scientist, KeplerAgent, and MOT-SR.Source: §2 · p. 2
  • Agentic SR works primarily develop search strategies, with supporting mechanisms designed around those strategies; SRHarness instead offers reusable runtime support without prescribing a strategy.Its actions, persistent state, and trajectory lifecycle management separate recurring runtime concerns from scientific search logic.Source: §2 · p. 2
  • Unlike harnesses tailored to environmental interaction or evidence synthesis, SRHarness supports heterogeneous scientific analysis, fitting, and search on shared quantities with consistent candidate representations and evaluation semantics.Figure 1 provides an overview of SRHarness’s three core mechanisms.Source: §2 · p. 3, Figure 1 · p. 3

3 SRHarness: A Domain-Specific Harness for Symbolic Regression

SRHarness separates runtime support from the search strategy through shared scientific actions, persistent hypothesis state, and lifecycle management. Its reference implementation organizes long-horizon search through configurable trajectories.

3.1 Overview

  • The harness separates scientific decisions from execution context, preserves evaluated candidates and evidence outside the conversation, and coordinates trajectory transitions.Its three mechanisms are composable scientific actions, persistent scientific state, and trajectory lifecycle management.Source: §3.1 · p. 3

3.2 Composable Scientific Actions

  • Expression-based action inputs let analysis and fitting operate on raw, transformed, or candidate-derived quantities while sharing execution context and evaluation semantics.Candidate-producing actions record expressions with evaluation, complexity, diagnostic evidence, and provenance; the harness validates and evaluates them before adding them to persistent state.Source: §3.2 · p. 4

3.3 Persistent Scientific State

  • Persistent scientific state stores evaluated candidates with their evaluations, complexity, evidence, and provenance across conversational trajectories and restarts.The model receives bounded Pareto or Top-k projections; candidates omitted from a view remain stored and may appear in later views.Source: §3.3 · p. 4

3.4 Trajectory Lifecycle Management

  • Trajectory management supports continuation, branching, restart, and termination, while the reference scheduler configures restart rounds, branches, refinement depth, and local response sampling.At restart, Top-k historical candidates initialize a conversation; branches refine using the Pareto view, and execution records intermediate results while preserving the best candidate under early stopping, interruption, or failure.Source: §3.4 · p. 4

4 Experiment

Across matched backbones, SRHarness performs strongly on numerical generalization and symbolic recovery, including when semantic cues are removed. Results also show that harness design and the organization of scientific search matter alongside model choice and tool availability.

4.1 Experiment Setup

  • The benchmark covers 240 problems across LSR-Synth, LSR-Transform, and an anonymized variant; evaluation includes numerical and symbolic metrics, with test observations withheld during search.LSR-Synth evaluates ID/OOD numerical generalization; LSR-Transform emphasizes exact symbolic recovery, and the anonymized variant removes scientific descriptions and variable semantics.Source: §4.1 · p. 5

4.2.1 Performance on LSR-Synth

  • On LSR-Synth, under both matched backbones, SRHarness achieves the strongest overall results in physics, chemistry, and biology on ID and OOD splits, and reaches 6.20% SA with DeepSeek-v4-flash-0731 versus below 1% for matched LLM-based baselines.Material-science problems appear substantially easier, with most methods approaching numerical saturation.Source: §4.2.1 · p. 6
  • The LSR-Synth backbone comparison shows non-monotonic model dependence: larger or nominally stronger models do not consistently perform better.The reported exploratory comparison finds closer alignment between SR performance and long-horizon agent benchmarks than model scale or scientific question-answering scores.Source: §4.2.1 · p. 6

4.4 Harness Design Beyond Models and Tools

  • Providing Codex with SRHarness’s scientific tools does not reproduce the advantage: with GPT-5.5, SA drops from 68.47% to 64.86%, while runtime and token use increase.The results point to integration with persistent state, evaluation, and trajectory management, rather than tool availability alone.Source: §4.4 · p. 8
  • Under the same DeepSeek-v4-flash-0731 backbone, SRHarness reaches 72.97% SA versus 20.72% for Codex on LSR-Transform-Anon.SRHarness with DeepSeek-v4-flash-0731 also reaches comparable symbolic recovery to Codex with GPT-5.5, which achieves 68.47% SA.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 Ablation Study

  • Ablations show that scientific actions, composable expression views, Pareto-based state views, and trajectory refinement depth each affect symbolic accuracy.Code-executor-only access lowers SA from 74.47% to 68.47%; raw-variable-only inputs yield 67.57%, while further fitting actions yield 79.28%. A single-best state view gives 66.67%, and the tested lifecycle alternatives score 66.67% for added restarts, 72.07% for branches, and 73.87% for local response sampling, against 74.47% for the default.Source: §4.5 · pp. 8–9

5 Conclusion

SRHarness separates reusable runtime support from the model and search strategy, using composable actions, persistent state, and trajectory management. The reported results show stronger equation discovery across matched backbones and settings, including when semantic cues are removed.

5 Conclusion

  • SRHarness separates reusable runtime support from the underlying model and search strategy, organizing actions, accumulated hypotheses, and long-horizon search.The conclusion identifies composable scientific actions, persistent scientific state, and trajectory lifecycle management as its mechanisms.Source: §5 · p. 9

AI use statement

  • The authors used generative AI for ideation, methodological and experimental development, code implementation and debugging, literature work, and presentation support, and report reviewing AI-assisted work and testing generated code.They state that AI was not used to formulate or prove theorems, generate or alter benchmark datasets, or conduct qualitative or thematic data analysis.Source: AI use statement · p. 9

Ethics statement

  • The work reports no human subjects, personal or private data, or decisions affecting individuals, and foresees no ethical concerns.Source: Ethics statement · p. 9

Reproducibility statement

A.1 Benchmark

The evaluation uses complementary LLM-SRBench tracks, controlled baselines, and shared metrics to assess numerical fidelity and symbolic recovery. SRHarness searches using training-derived fitting and validation evidence, with test sets withheld until submission.

A.1 Benchmark

  • LLM-SRBench evaluation covers 128 LSR-Synth problems, 111 LSR-Transform problems, and an anonymized view of the same Transform problems with scientific semantics removed.LSR-Synth tests discovery from numerical observations across four scientific domains; LSR-Transform tests transformed equations with meaningful descriptions and variable semantics.Source: §A.1 · p. 11

A.2 Baselines

  • Controlled comparisons rerun representative methods with matched LLM backbones where implementations permit, using PySR as a non-LLM reference and SR-Scientist as the direct agentic baseline.The reproduced SR-Scientist uses a 75-call cap and token-aligned budget; its results are not directly comparable to its original paper.Source: §A.2 · pp. 11–12

A.3 Metrics

  • Acc0.1 measures the mean fraction of test points within 10% relative target error, while median NMSE captures numerical error magnitude beyond that threshold.NaN predictions are excluded; LSR-Synth Acc0.1 is reported separately for ID and OOD test sets.Source: §A.3 · p. 12
  • Symbolic accuracy (SA) tests mathematical equivalence rather than prediction similarity, using numerical diagnostics followed by a structural-equivalence judge.SA uses the full benchmark as its denominator, so missing or unevaluable submissions count as incorrect; Acc0.1 is the mean and NMSE the median over successfully archived submissions.Source: §A.3 · pp. 12–13

A.4 Implementation Details

  • SRHarness ranks candidates by validation MSE after an 80/20 split of official training observations, and accesses ID and OOD test sets only after final submission.The action set and R1-C1-L30-K1 schedule are fixed across benchmark conditions and primary backbones, with model-specific response limits as an exception.Source: §A.4 · p. 13

B.1 Detailed LSR-Synth Results

On LSR-Synth, SRHarness shows strong numerical fidelity in challenging domains and a faster search with competitive API cost. The detailed tables report domain-level NMSE and resource usage.

B.1 Detailed LSR-Synth Results

  • 1.48 × 10^-5 physics ID NMSE for SRHarness under DeepSeek-v4-flash-0731 is below SR-Scientist’s 1.85 × 10^-5 and IGSR’s 1.34 × 10^-4.The material domain is near-saturated across methods; SRHarness also remains highly competitive on chemistry and biology ID splits.Source: §B.1 · p. 13, Table 5 · p. 14
  • 11.11 minutes per problem for SRHarness under DeepSeek-v4-flash-0731 is faster than LLM-SR at 57.67 minutes and IGSR at 24.02 minutes, with $0.03 API cost.The reported runtime advantage accompanies a higher total token count and an input-dominant token profile.Source: §B.1 · p. 13, Table 6 · p. 14

B.1.1 Backbone Dependence on LSR-Synth

Across five fully evaluated backbones, long-horizon agent benchmarks align more closely with LSR-Synth performance than model scale or static scientific question answering. Domain-level polynomial results show that the material subset is unusually easy to approximate because its observed ranges cover smoother portions of nonlinear laws.

B.1.1 Backbone Dependence on LSR-Synth

  • The five-model analysis is hypothesis-generating rather than causal because the external benchmarks use different scaffolds and evaluation protocols.The reported correlations should be interpreted within that limited comparison.Source: §B.1.1 · p. 15
  • DeepSeek-v4.1-flash reaches 98.44% biology ID and 98.59% OOD Acc0.1, while DeepSeek-v4-pro-0813 leads physics ID/OOD accuracy at 93.73%/92.18%.DeepSeek-v4.1-flash outperforms the 49B-active DeepSeek-v4-pro-0813 overall despite using 16B activated parameters during decoding.Source: §B.1.1 · p. 15
  • TerminalBench 2.1 correlates with overall ID Acc0.1, SA, and −log10 median NMSE at 0.90, 0.82, and 0.90; DeepSWE’s ordering exactly matches median NMSE (ρ = 1.00).Total parameter count has lower correlations of 0.70, 0.62, and 0.30, while activated parameter count has 0.40, 0.36, and 0.10; the figure compares these backbone attributes with aggregate performance.Source: §B.1.1 · p. 14, Table 7 · p. 15, Figure 3 · p. 16

B.1.2 Domain-Specific Difficulty in LSR-Synth

  • A degree-three polynomial reaches 96.94% OOD Acc0.1 on material problems, versus 28.06% for physics, 70.32% for chemistry, and 56.35% for biology.The figure compares the same degree-three baseline across domains and shows OOD accuracy as polynomial degree increases.Source: §B.1.2 · p. 16, Figure 4 · p. 17
  • Material tasks expose mostly smoother nonlinear ranges: only 12% cross more than one periodic cycle or span more than one exponential-argument unit, making low-degree approximation unusually effective.The median exponential-argument span is 0.016 in material, versus 1.00, 4.24, and 4.95 in physics, chemistry, and biology; physics also includes many long-period and absolute-value-kink cases.Source: §B.1.2 · pp. 16–17

B.2 Detailed LSR-Transform Results

The detailed comparisons report resource use across standard and anonymized LSR-Transform, compare SRHarness with Codex, and characterize run-to-run variation. In the anonymized Codex comparison, SRHarness achieves substantially higher symbolic accuracy at comparable time and token use under the matched DeepSeek backbone.

B.2 Detailed LSR-Transform Results

  • 12.47 minutes and $0.0204 per problem: SRHarness is faster and lower-cost than LLM-SR on LSR-Transform despite comparable total token use.About 97% of SRHarness tokens are prompts, and about 93% of those prompt tokens hit the prefix cache; GLM-5.3-flash yields 7.09 minutes and $0.0242 per problem.Source: §B.2 · p. 17, Table 8 · p. 18

B.3 Detailed LSR-Transform-Anon Results

  • 12.34 minutes and $0.0252 per problem: SRHarness outperforms LLM-SR and SR-Scientist on LSR-Transform-Anon, with resource use essentially unchanged from standard LSR-Transform.The passage reports that several baselines require longer trajectories while recovering fewer target equations; GLM-5.3-flash yields 15.15 minutes and $0.0310 per problem.Source: §B.3 · p. 18, Table 9 · p. 19

B.3.1 Codex Evaluation Protocol and Additional Comparison

  • 72.97% vs. 20.72% SA: SRHarness exceeds Codex under matched DeepSeek-v4-flash-0731, with comparable time and token use.The evaluation uses isolated Codex CLI processes with a 900-second limit; adding SRHarness tools to GPT-5.5 Codex raises resource use while SA falls from 68.47% to 64.86%.Source: §B.3.1 · p. 19, Table 10 · p. 20

B.4.1 Repeated Runs

  • Across three full-configuration runs, SA ranges from 72.97% to 75.68% and median NMSE stays around 10^-14, while expression complexity ranges from 17.55 to 33.36.The average SA is 74.47%; similar numerical fidelity can accompany different discovered structures.Source: Table 11 · p. 20, §B.4.1 · p. 20

B.4.2 Ablation of Scientific Actions

The scientific-action ablations show that action structure and expression-based views are associated with symbolic recovery, while adding fitting actions improves SA alongside increased complexity and resource use.

B.4.2 Ablation of Scientific Actions

  • Restricting SRHarness to the code executor lowers SA from 74.47% to 68.47%, while mean search time changes only from 14.93 to 14.34 minutes.The authors interpret this as support for structured scientific actions beyond general computation, especially when scientific descriptions are unavailable.Source: §B.4.2 · p. 20
  • Removing expression-based views lowers SA to 67.57% even as median NMSE remains near 10^-14, and removes access to transformed variables and candidate-derived quantities for later analysis.The authors connect this result to reusing intermediate evidence to refine an evolving hypothesis.Source: §B.4.2 · p. 20
  • Adding fitting actions raises SA to 79.28%, while mean complexity increases from 23.19 to 33.16 and runtime and token use also increase.The results support extensible scientific actions while indicating that their value depends on resulting search behavior, not tool availability alone.Source: §B.4.2 · p. 20

B.4.3 Ablation of Persistent State

Persistent-state ablations indicate that model-facing views affect symbolic accuracy, while numerical fit and expression compactness do not alone establish better symbolic recovery.

B.4.3 Ablation of Persistent State

  • Without the explicit best-formula view, numerical accuracy is higher and mean complexity is lower than the full configuration, although SA is slightly lower.The passage notes that these metrics capture different aspects of discovery.Source: §B.4.3 · p. 21
  • The ablation points to a trade-off among refinement, exploration, and execution cost in trajectory lifecycle management.Neither numerical fit nor compactness alone establishes better symbolic recovery.Source: §B.4.3 · p. 21
  • Presenting only the best formula reduces SA from 74.47% to 66.67%, with little token reduction and a longer mean runtime.Removing the explicit best-formula view retains 72.97% SA.Source: Table 13 · p. 21, §B.4.3 · p. 21

B.4.4 Ablation of Trajectory Management

The ablation tests how reallocating trajectory budget affects symbolic accuracy and candidate quality. Under the tested budget, deeper refinement outperforms allocating more capacity to fresh trajectories, while local response sampling improves numerical accuracy and reduces mean complexity without establishing better symbolic recovery.

B.4.4 Ablation of Trajectory Management

  • The alternatives test lifecycle-supported budget scheduling, reallocating capacity from refinement depth to another trajectory dimension.The comparison concerns scheduling choices, not the presence or absence of lifecycle management.Source: §B.4.4 · p. 21
  • Local response sampling achieves 73.87% SA, 95.38% numerical accuracy, and mean complexity of 16.30.Its SA falls within the range of the three default runs, and the results do not establish improved symbolic recovery.Source: §B.4.4 · p. 21
  • On LSR-Transform-Anon, more restarts and more independent branches yield 66.67% and 72.07% SA, respectively, versus 74.47% for default; under the tested budget, fresh trajectories do not compensate for reduced refinement depth.Source: Table 14 · p. 21, §B.4.4 · p. 21
Loading 2609.35501v1…