PaperlayerView exact revision on arXiv

Source-linked AI summary

RISR: Residual-Informed Scientific Equation Discovery with Large Language Models

Haobo Li, Wenshuo Zhang, Wenxiao Zhao, Eunseo Jung, Rui Sheng, Yushi Sun, Peiqin Zhuang, Hao Chen, Fenghua Ling

arXiv:2610.11387v1cs.SCcs.AI

TL;DR

Aggregate fit scores do not reveal how residual errors vary across inputs, limiting the numerical guidance available for equation discovery. RISR encodes residuals to guide LLM formula proposals and predicts the utility of fitted corrections; it outperforms reported same-backbone baselines on LLM-SRBench.

  • Problem

    Aggregate fit scores rank equations but do not preserve how remaining errors vary across inputs, although residual patterns can differ between equations with comparable losses.

    Source: §1 · p. 2

  • Method

    RISR encodes aligned inputs, targets, predictions, and residuals to condition LLM proposals, then uses additive and regularized multiplicative residual views to predict fitted correction utility.

    Source: §1 · p. 2

  • Results

    Across LLM-SRBench scientific problems, RISR outperforms reported same-backbone methods, achieving 63.57% accuracy at the 1% relative-error tolerance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The case study illustrates that structural editing and coefficient calibration contribute at different points in the discovery process.

    Source: §5 · p. 10

    Some cases substantially reduce numerical error without fully recovering the underlying formula.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Symbolic regression combines structural search with numerical fitting, but aggregate fit scores do not describe how the remaining error varies across inputs. We introduce RISR, a residual-informed method that uses these error patterns to guide formula discovery and learn which corrections are worth fitting. A residual encoder compresses aligned inputs, targets, current predictions, and residuals into continuous tokens that condition a language model to propose formulas. For subsequent refinement, a dual-view relational encoder uses additive and regularized multiplicative residuals to predict the post-fit utility of candidate corrections. We evaluate RISR on scientific tasks from the LLM-SRBench. RISR achieves 63.57% and 38.50% ID accuracy at the 1% and 0.1% pointwise relative-error tolerances, respectively. The corresponding OOD accuracies are 56.07% and 38.24%. RISR outperforms the reported baselines using the same backbone. The results show that our residual-informed approach can improve numerical equation recovery.

1 INTRODUCTION

Symbolic regression must search expression structures while fitting their coefficients, but aggregate fit scores do not show how remaining errors vary across inputs. RISR uses residual patterns to guide formula search and correction selection, and improves benchmark accuracy over reported baselines using the same backbone.

Prior work

  • Prior approaches include sparse term selection, learned expression sampling, hybrid search, and generators trained on synthetic equation-observation pairs.LLM-based methods also use numerical evaluation, evolutionary search, scientific knowledge, or executable residual analysis to guide proposals.Source: §1 · p. 1

Motivation

  • Aggregate fit scores rank equations but discard where errors occur, while pointwise residuals reveal input-dependent nonlinear patterns, interactions, and localized discrepancies.Useful guidance must also account for the current equation and how fitting affects a proposed correction.Source: §1 · p. 2

Approach

  • RISR conditions LLM proposals on encoded inputs, targets, predictions, and residuals, then uses a dual-view encoder to predict candidate-correction utility after fitting.A learned structural-edit generator complements LLM proposals; corrections are fitted and accepted when training loss improves.Source: §1 · p. 2

Training

  • The encoders are trained separately on synthetic data, with the LLM frozen and correction utility supervised using outcomes on observations disjoint from coefficient fitting.The search encoder learns from residual-to-formula examples, while the refinement model learns from paired current equations, observations, and candidate actions.Source: §1 · p. 2

Results

  • 63.57% and 38.50% accuracy at the 1% and 0.1% relative-error tolerances, respectively, exceed reported same-backbone baselines.The evaluation covers scientific problems in materials science, chemistry, biology, and physics.Source: §1 · p. 2

2 RELATED WORK

Symbolic-regression systems range from explicit search procedures to neural expression generation and LLM-guided program search. RISR focuses on representing the current numerical state and learning which fitted corrections are useful.

Symbolic equation discovery

  • Existing symbolic-regression approaches include physics-inspired search, evolutionary procedures, neural expression prediction, and Transformer-guided search.Examples include AI Feynman, PySR, NeSymReS, end-to-end symbolic regression, and TPSR.Source: §2 · p. 2

Residual utility

  • A fitted-equation score alone does not determine which structural correction will help; utility also depends on residual patterns, parameterization, and composition with the equation.RISR learns correction utility from fitted outcomes and ranks corrections using additive and regularized multiplicative signals.Source: §2 · p. 3

LLM-based Symbolic Regression

  • LLM-based systems combine program generation with executable evaluation, using evolutionary search, parameter fitting, learned concept libraries, or agent-based analysis.SR-Scientist includes residual inspection through code execution.Source: §2 · p. 3

Residual representation

  • RISR learns a representation from synthetic residual-to-formula examples, compressing aligned inputs, targets, predictions, and residuals into continuous tokens for a frozen LLM.The tokens condition proposals alongside textual search context.Source: §2 · p. 3

3 RISR

RISR first uses residual-informed search to obtain an initial equation, then ranks and fits candidate corrections using two residual views. Its separately trained components use synthetic data, including held-out fitted outcomes for correction-utility supervision.

3 RISR

  • RISR has two stages: residual-informed search and structural edits produce an initial equation, then a dual-view encoder ranks corrections for fitting and acceptance.The figure summarizes the search and refinement stages.Source: §3 · p. 3, Figure 1 · p. 3

3.1 RESIDUAL-INFORMED FORMULA SEARCH

  • The search encoder processes aligned inputs, targets, predictions, and residuals through attention and spectral-field paths, projecting them into 16 tokens for a frozen LLM.Textual context includes candidate programs; a learned AST-aware edit generator proposes additions, replacements, deletions, or stopping, and the lowest-MSE equation initializes refinement.Source: §3.1 · p. 4

3.2.1 DUAL-VIEW RESIDUAL ENCODING

  • The refinement encoder compares additive residuals with a regularized multiplicative signal, using variable relations and pairwise statistics to represent correlations and product-correction alignment.The multiplicative signal avoids unstable division near zero; its regularization uses 5% of prediction RMS with a positive floor.Source: §3.2.1 · p. 4

3.2.2 CORRECTION UTILITY PREDICTION

  • Candidate actions combine an additive or multiplicative update with unary or two-variable product operators, and a shared scorer predicts their fitted utility.The operator families include polynomial, trigonometric, exponential, logarithmic, and reciprocal forms.Source: §3.2.2 · pp. 4–5

3.2.3 COEFFICIENT FITTING AND ADAPTIVE UPDATES

  • Selected corrections receive numerical coefficient fitting, and the highest-scoring untried action is accepted only when it reduces training relative squared error by more than 0.1%.The correction uses standardized inputs and variable projection; refinement allows at most four fitting attempts and refreshes residual views and rankings after acceptance.Source: §3.2.3 · p. 5

3.3 SYNTHETIC DATA AND MODEL TRAINING

  • The search encoder is trained from scratch on 6,000 synthetic residual-to-formula examples, while structural-edit training uses 20,000 examples covering additions, deletions, replacements, and stopping.All components are trained offline on synthetic data independent of benchmark equations and observations, with the pretrained LLM frozen.Source: §3.3.1 · p. 5, §3.3.2 · pp. 5–6, §3.3 · p. 5

3.3.3 REFINEMENT-ENCODER DATA AND TRAINING

  • Refinement training pairs current equations with shared observations and candidate actions, enabling supervision for how action utility changes with the current equation.The splits include 8,192 training pairs and test sets covering new observation clouds, held-out product actions, and eight-variable inputs; value, ranking, and paired-state losses are trained jointly.Source: §3.3.3 · p. 6
  • Correction utility is measured on 64 held-out observations after fitting on 192 rows, so the target evaluates fitted outcomes beyond the coefficient-fitting rows.Rejected actions receive zero utility; accepted actions receive positive or negative utility according to their held-out effect.Source: §3.3.3 · p. 6

4 EVALUATION

RISR is evaluated on scientific equation-discovery tasks using ID and OOD accuracy at two pointwise relative-error tolerances. It achieves strong accuracy across domains and backbone scales, while ablations, symbolic-accuracy assessment, and a case study examine its components and recovery behavior.

4.1 EVALUATION SETTING

  • The evaluation uses 129 LSR-Synth problems across four scientific domains, with separate training, ID, and OOD test sets and task accuracy reported at 1% and 0.1% pointwise relative-error tolerances.Published results serve as baselines; searches use a frozen Qwen3-Coder-30B-A3B-Instruct backbone and a 64-slot candidate budget.Source: §4.1 · pp. 6–8

4.2 MAIN RESULTS

  • 63.57% and 38.50% ID accuracy at the 1% and 0.1% tolerances exceed SR-Scientist+RL by 22.65 and 17.81 percentage points, respectively.RISR matches the highest reported overall accuracy at 1% across all baselines and ranks second overall at 0.1%; Table 1 reports scientific ID accuracy.Source: Table 1 · p. 7, §4.2 · p. 8
  • RISR leads materials science at 1% with 89.33% accuracy and physics at both tolerances with 57.58% and 40.91%, while ranking second in chemistry at both tolerances.In physics, its results exceed the strongest competing 120B model by 16.67 and 6.82 percentage points, respectively; in biology it ties for second at 1%.Source: §4.2 · p. 8
  • 56.07% OOD accuracy at the 1% tolerance is the highest among compared methods at the same 30B scale, exceeding SR-Scientist + RL by 16.57 percentage points.The supplied Figure 2 caption identifies method colors and distinguishes ID from OOD bars; physics reaches 40.91% OOD accuracy at 0.1%.Source: Figure 2 · p. 7, §4.2 · p. 8

4.3 ABLATION STUDY

  • 36.43% ID and 40.31% OOD accuracy at 1% for residual-informed search rise from 7.75% and 9.30% for the LLM-only method; refinement reaches 63.57% ID and 55.04% OOD.Learned-scored corrections improve 1% accuracy over random corrections by 3.10 ID and 2.33 OOD percentage points; Table 2 reports one-seed ablations.Source: §4.3 · pp. 8–9, Table 2 · p. 8

4.4 SYMBOLIC ACCURACY

  • 13.18% symbolic accuracy (17/129) exceeds existing methods, but numerical accuracy and structural recovery are complementary under the constant-assignment criterion.Table 3 reports symbolic accuracy on one seed setting.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 CASE STUDY

  • On PO30, RISR’s structural edit replaces an extraneous c5 ∗v term with a constant, and refinement calibrates coefficients without changing the selected equation’s structure.The remaining constant becomes negligible, while two independently fitted sinusoidal coefficients closely match the target’s shared coefficient; Figure 3 summarizes the case.Source: §4.5 · p. 9, Figure 3 · p. 9

5 CONCLUSION

RISR combines residual-conditioned formula search with refinement that predicts which corrections are likely to improve an equation after fitting. Its evaluation reports 63.57% and 38.50% ID accuracy at the 1% and 0.1% tolerances.

5 CONCLUSION

  • 63.57% and 38.50% ID accuracy at the 1% and 0.1% tolerances show the reported performance of RISR’s two-stage equation-discovery method.The method combines residual-conditioned LLM proposals and structural edits with a dual-view encoder that predicts correction utility after fitting.Source: §5 · pp. 9–10
  • The case study illustrates that structural editing and coefficient calibration contribute at different points in the discovery process.The examined cases include substantial numerical-error reductions despite incomplete recovery of the underlying formula.Source: §5 · p. 10
  • Generative AI assisted with writing proofs, implementing methods, translation, and formulating mathematical claims, but not with generating synthetic data or interpreting results.The authors state that they reviewed the AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The appendix details RISR’s residual-conditioned search, structural editing, and learned correction refinement, alongside synthetic training data and implementation choices. The ethics statement reports publicly available benchmark data, synthetic numerical datasets, and no identified specific ethical concerns.

ETHICS STATEMENT

  • The ethics statement says RISR uses public scientific benchmark data and synthetic numerical datasets and reports no specific ethical concerns identified for the study.The method configuration also sets limits on program size and the observations used by the structural editor.Source: ETHICS STATEMENT · p. 10, §A.3 · p. 13

A.1 FORMULA SEARCH AND NUMERICAL INPUTS

  • RISR’s search encoder receives aligned coordinates, targets, parent predictions, and residuals, retaining up to six residual-associated coordinates and encoding at most 4,000 rows.When there are more than six variables, coordinates are selected by residual correlations; the routing applies only to the encoder, while fitting retains all variables.Source: §A.1 · p. 12

A.2 POINT-ATTENTION AND SPECTRAL-FIELD PATHS

  • Point-attention and spectral-field paths represent pointwise observations and coordinate–residual interactions, then fuse them into 16 continuous tokens for the frozen LLM.The spectral path includes learned frequency features and residual-region summaries; it is not an FFT of the observations.Source: §A.2 · p. 13

A.3 AUXILIARY LEARNED STRUCTURAL-EDIT GENERATOR

  • The structural editor combines residual summaries with an abstract-syntax-tree representation to predict edit operations, target nodes, function families, variable support, and parameters.It reranks fitted edits using learned outcome scores; its weights are frozen at deployment and it does not evaluate the benchmark OOD split during search.Source: §A.3 · pp. 13–14

B.1 DUAL-VIEW CONSTRUCTION AND RELATIONAL STATISTICS

  • The dual-view refinement encoder represents additive and regularized multiplicative correction signals and uses shared relation-attention weights to produce variable representations for utility scoring.The shared utility head scores candidate actions, which specify an additive or multiplicative update and one or two variables with an operator per variable.Source: §B.1 · p. 15, §B.2 · pp. 15–16, Figure 6 · p. 16

B.3 CORRECTION COEFFICIENT FITTING

  • Stage B first recalibrates the initial equation’s coefficients, then ranks untried corrections and accepts only finite updates that reduce training loss by more than 0.1%.Refinement allows at most four attempts, stops at training NMSE ≤ 10^-12, and does not use test observations for acceptance.Source: §B.3 · p. 16, §B.4 · p. 17

C SYNTHETIC DATA AND OFFLINE TRAINING

  • Synthetic corpora train residual-to-formula generation, structural editing, and fitted-utility prediction using transformed and composed elementary functions rather than benchmark equations or observations.The search encoder and editor use separate synthetic datasets and supervision; the editor’s wider-range synthetic observations are not benchmark OOD arrays.Source: C SYNTHETIC DATA AND OFFLINE TRAINING · p. 17, §C.1 · p. 17, §C.2 · p. 18

C.3 PAIRED-STATE REFINEMENT DATA

The paired-state corpus trains correction-utility prediction on synthetic equations and shared candidate actions. Utility labels separate correction fitting from evaluation on held-out observations, and the model jointly optimizes value and ranking objectives.

C.3 PAIRED-STATE REFINEMENT DATA

  • Refinement training pairs two current equations on a shared observation cloud, with eight shared dimension-valid candidate actions per pair.Synthetic targets contain three to six components; the dimensional test uses eight variables, while training and validation use two to six.Source: §C.3 · p. 18, Table 5 · p. 18

C.4 UTILITY TARGETS, LOSSES, AND OPTIMIZATION

  • The first 192 rows provide encoder inputs, correction fits, and acceptance decisions, while the remaining 64 measure fitted utility.The latter subset is held out from correction fitting, but not from every operation used to construct the synthetic problem.Source: §C.4 · p. 19
  • Utility supervision can be negative even for an accepted action when that correction worsens predictions on the held-out rows.Rejected actions receive zero utility.Source: §C.4 · p. 19
  • The jointly trained refinement model combines utility-value, within-state ranking, and paired-state losses, weighting each of the latter two by 0.5.Training uses 8,192 synthetic pairs over 16 epochs, and the selected model remains frozen during refinement.Source: §C.4 · p. 19

D RESULT STATISTICS

The reported main-table and ID/OOD plot results average three complete runs, calculating success rates per seed before taking their mean. Domain results use each domain’s task count.

D RESULT STATISTICS

  • Main-table and ID/OOD plot results average three complete runs using seeds 0, 42, and 20260831.The success rate is computed separately for each seed before the three rates are averaged; standard deviations are sample standard deviations with two degrees of freedom.Source: D RESULT STATISTICS · p. 19, Table 6 · p. 19
  • Domain results use the task count for the corresponding domain.Table 6 gives detailed results for the three seeds.Source: D RESULT STATISTICS · p. 19, Table 6 · p. 19

E SYMBOLIC ACCURACY: PROTOCOL AND DETAILED RESULTS

The symbolic-accuracy assessment defines success by whether adjustable constants can represent the target, then checks that exported programs reproduce saved predictions. It also documents the judgment protocol and case-study reporting.

E SYMBOLIC ACCURACY: PROTOCOL AND DETAILED RESULTS

  • Symbolic accuracy asks whether some assignment of adjustable constants makes a candidate represent the target, while leaving specified normalization and operator choices unchanged.Original program parameters and accepted-correction parameters remain adjustable.Source: E SYMBOLIC ACCURACY: PROTOCOL AND… · p. 20
  • All 129 final programs, including accepted corrections, reproduced their saved predictions in all 387 training/ID/OOD replay checks.The checks used the programs’ saved coefficients.Source: E SYMBOLIC ACCURACY: PROTOCOL AND… · p. 20
  • 1,290 valid judgments comprised 17 tasks with ten Yes votes and 112 with ten No votes, with no unresolved cases.Each candidate–target pair was assessed in ten requests; unanimous outcomes were accepted and non-unanimous outcomes reserved for human review.Source: E SYMBOLIC ACCURACY: PROTOCOL AND… · p. 20

F MORE CASE STUDIES

  • The case studies report saved formulas, selected actions, acceptance decisions, and replayed errors; Stage B outputs utility scores rather than textual reasoning.The residual encoder supplies continuous numerical tokens.Source: F MORE CASE STUDIES · p. 20

F.1 CRK22: A FOUR-STEP NUMERICAL REFINEMENT

The CRK22 example reports improved numerical error after four accepted additive corrections, although the resulting formula is not judged symbolically equivalent. MatSci9 shows a different trajectory, with lower final ID error but nonmonotonic OOD error.

F.1 CRK22: A FOUR-STEP NUMERICAL REFINEMENT

  • ID NMSE fell from 1.6757×10^-7 to 4.7072×10^-8, and CRK22’s OOD point fraction rose from zero to one at both main tolerances.All four additive actions were accepted, and each changed the residual state used for the next ranking.Source: §F.1 · p. 20, Table 8 · p. 20
  • CRK22’s final formula was not counted as symbolically equivalent by the ten-vote assessment.Source: §F.1 · p. 20

F.2 MATSCI9: ADDITIVE AND MULTIPLICATIVE CORRECTIONS

  • MatSci9 ID NMSE decreased from 1.490 × 10^-9 to 2.580 × 10^-12, while final OOD NMSE remained above its third-prefix value.All four alternating additive and multiplicative attempts were accepted; the nonmonotonic test error was retained rather than replaced by the best test-scoring prefix.Source: §F.2 · p. 21, Table 9 · p. 21

G UTILITY-RANKING EXPERIMENTS

A learned scorer ranks fitted correction candidates using information about the current equation. In held-out synthetic candidate pools, it outperformed simple selection rules, while paired-state swaps performed substantially worse.

G UTILITY-RANKING EXPERIMENTS

  • The paired-state swap selected an action for one current equation using scorer outputs from its paired equation, without changing the action pool or fitting protocol.Source: G UTILITY-RANKING EXPERIMENTS · p. 21
  • The held-out evaluation used three synthetic splits, each with 1,024 paired problems, or 2,048 current-equation states, and eight dimension-valid actions per state.Source: G UTILITY-RANKING EXPERIMENTS · p. 21
  • Corrections were fitted on designated fitting rows and accepted only when fitting loss fell by more than 0.1%; utility was then measured on disjoint rows.The refinement model was frozen during evaluation, and the reported score was mean utility for the selector’s chosen action.Source: G UTILITY-RANKING EXPERIMENTS · p. 21
  • Mean utilities were 0.4627 on new observation clouds, 0.4099 on unseen product actions, and 0.4532 on eight-variable inputs, with the learned scorer outperforming residual-correlation ranking and uniform random selection on all three splits.The paired-state swap was substantially worse, consistent with the scorer using information about the current equation.Source: G UTILITY-RANKING EXPERIMENTS · p. 21

H COMPUTATIONAL ACCOUNTING

The computational accounting reports search and correction-fitting budgets alongside measured Stage-B workload for seed 20260831.

H COMPUTATIONAL ACCOUNTING

  • Stage A allowed at most 64 outer candidate submissions, including the seed; after search, the executable candidate with the lowest training MSE was selected as p0.Invalid submissions also consumed an outer slot.Source: H COMPUTATIONAL ACCOUNTING · p. 21
  • The Stage-A setup generated four continuations per LLM request, so a complete 64-slot task search used at most 64 generation requests.Source: H COMPUTATIONAL ACCOUNTING · p. 22
  • For seed 20260831, Stage B recorded 424 correction attempts, of which 400 were accepted, and 433.98 seconds of correction-fit time.The projected fitting objective was evaluated 461,860 times; p0 coefficient calibration was separate from the budget of at most four new correction-fitting attempts.Source: H COMPUTATIONAL ACCOUNTING · p. 22
Loading 2610.11387v1…