PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Every Term Adds New Structure: Sobolev Novelty for Symbolic Regression

Boxiao Wang, Kai Li, Yuheng Jing, Tianyi Liu, Chen Li, Yifan Zhang, Jian Cheng

arXiv:2609.32597v1cs.LGcs.AI

TL;DR

Symbolic regression’s equation-level objectives assess whole-expression fit but provide little information about whether individual terms contribute new structure. The paper introduces Sobolev Novelty, which measures term behavior not reconstructed by the other terms, and derives a theory-calibrated threshold. At that threshold, benchmark ground-truth equations have a higher share of structurally novel terms than expressions from existing SR methods.

  • Problem

    Equation-level objectives assess candidate fit but provide little information about whether individual terms add structure or are redundant with the rest of the expression.

    Source: §1 · pp. 1–2

  • Method

    Sobolev Novelty uses each term’s empirical function-value and derivative representation to measure the behavior not reconstructed by the remaining terms.

    Source: §1 · p. 2

  • Results

    92.6% of terms in ground-truth equations exhibit sufficient structural novelty, compared with 38.3% on average for expressions produced by 15 SR methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sobolev Novelty can complement equation-level objectives across diverse SR paradigms, where its incorporation yields consistent performance improvements.

    Source: §1 · p. 2

Abstract

from arXiv · show

Symbolic regression (SR) aims to discover compact and meaningful mathematical equations from data, but searching the vast combinatorial space of symbolic structures remains challenging. Existing methods typically guide this process using expression-level objectives, such as fitting error, which assess a candidate equation as a whole but provide little information about whether an individual term contributes genuinely new structure or is largely redundant with the rest of the expression. We introduce \textbf{Sobolev Novelty}, a term-level measure of structural independence for symbolic equations. For each term, we construct an empirical Sobolev signature from its function values and exact derivatives over the observed inputs, and quantify how much of this behavior cannot be reconstructed by the remaining terms. We further derive a theory-calibrated threshold, yielding a principled and tuning-free criterion for identifying structurally novel terms. Using this threshold, 92.6\% of terms in benchmark ground-truth equations exhibit sufficient structural novelty, compared with only 38.3\% on average for expressions produced by 15 SR methods, revealing a substantial gap between scientific equations and current SR solutions. As a lightweight plug-in, Sobolev Novelty can be incorporated into diverse SR paradigms to support term pruning, search guidance, LLM feedback, and data selection, yielding consistent performance gains and demonstrating broad applicability.

1 INTRODUCTION

Symbolic regression searches a combinatorial space, but equation-level fitting objectives say little about whether individual terms add distinguishable structure. The paper introduces Sobolev Novelty to measure each term’s behavior beyond the remaining terms, then uses a theory-calibrated threshold to expose a gap between benchmark equations and current SR outputs.

1 INTRODUCTION

  • Equation-level fitting objectives assess whole-candidate fit but provide little information about the structural role of individual terms.A term may be well reproduced by the remaining terms over the observed domain, leaving the expression’s decomposition less identifiable than its predictive performance suggests.Source: §1 · pp. 1–2
  • Sobolev Novelty represents each term through function values and exact first-order derivatives, then measures its behavior that cannot be reconstructed from the other terms.The measure is candidate-internal, requiring only the expression and observed inputs, not target derivatives or the ground-truth equation.Source: §1 · p. 2
  • 92.6% of ground-truth equation terms meet the structural novelty criterion, versus 38.3% on average across expressions from 15 SR methods.The authors describe this as a substantial gap between scientific equations and current SR solutions.Source: §1 · p. 2
  • Sobolev Novelty is a lightweight plug-in for diverse SR paradigms, including term pruning, search guidance, LLM feedback, and filtering pretraining equations.The paper reports consistent performance improvements across representative SR paradigms.Source: §1 · p. 2
  • Predictive accuracy alone does not fully characterize an equation’s quality for scientific discovery; distinguishable term-level structure is also desirable.Sobolev Novelty offers a practical way to quantify and encourage that property.Source: §1 · p. 2

2 PRELIMINARIES

Symbolic regression searches for an analytic expression that fits observed input-output data and generalizes to unseen inputs. Its methods couple expression-structure search with evaluation, commonly using predictive quality and complexity to rank or refine candidates, but these objectives do not directly characterize individual terms.

2 PRELIMINARIES

  • SR seeks an analytic expression that approximates observed targets and generalizes to unseen inputs.The dataset consists of input vectors and corresponding scalar outputs.Source: §2 · p. 2
  • SR methods couple discrete structure search over variables and operators with numerical parameter optimization against observed data.Search paradigms include genetic programming, reinforcement learning, tree search, pretrained generative models, and LLM-based generation.Source: §2 · p. 3
  • Predictive quality and other equation-level objectives, such as expression complexity, rank or refine candidates but do not directly characterize individual-term contributions.The paper introduces a term-level perspective as a complement to these global objectives.Source: §2 · p. 3

3 METHOD

Sobolev Novelty measures how much of a term’s observed values and local variation cannot be reconstructed from the other terms. The method interprets this residual geometrically and through coefficient stability, then uses those perspectives to calibrate a threshold.

3.1 FROM TERM REPRESENTABILITY TO SOBOLEV NOVELTY

  • Jointly representing each term’s function values and gradients makes redundancy require reconstructing both observed outputs and local variation.The representation places these quantities in an empirical first-order Sobolev geometry.Source: §3.1 · p. 3
  • Sobolev Novelty is the normalized leave-one-term-out residual after the remaining terms jointly reconstruct a term’s value-and-gradient representation.A common coefficient vector must reproduce both blocks; the score lies between 0 and 1, with values near zero indicating high representability by the other terms.Source: §3.1 · p. 4

3.2 A THEORY-CALIBRATED THRESHOLD

  • The projection perspective interprets higher novelty as a larger residual outside the span of the remaining term signatures.The residual is orthogonal to the projected component; a zero projection corresponds to novelty of 1.Source: §3.2 · p. 4
  • The explanation-stability perspective links lower novelty to greater coefficient sensitivity when the equation’s values and gradients change slightly.The maximum coefficient-change amplification factor squared is 1/ν_i^2, so low novelty signals a less stable decomposition.Source: §3.2 · p. 5
  • The calibrated threshold requires at least 10% of a term’s squared representation to remain unreconstructed, while bounding coefficient-change amplification.It allows some behavioral overlap but flags terms with too-small residuals and potentially excessive coefficient amplification.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments assess Sobolev Novelty as a measure of term-level structure, as guidance for symbolic search and pretraining, and in periodic-hill turbulence modeling. They span SRBench tasks and a coupled physical simulation.

4.1 A STRUCTURAL NOVELTY GAP IN EXISTING SR METHODS

  • At the theory-calibrated threshold, 92.6% of ground-truth terms qualify as structurally novel, versus 38.3% on average across 15 SR methods, a 54.3-percentage-point gap.Among candidates with R2 ≥0.8, high fitting quality and complexity comparable to ground truth do not guarantee comparable term-level novelty.Source: §4.1 · p. 6, Figure 1 · p. 6

4.2 SOBOLEV NOVELTY AS SEARCH GUIDANCE

  • Sobolev-guided GP and MCTS improve accuracy and compactness on white-box tasks, while all three guided methods improve mean R2 on black-box tasks, with backbone-dependent complexity effects.SN-IGSR improves its accuracy rank while preserving complexity under the same task set, seed, and search budget.Source: §4.2 · pp. 7–8, Figure 2 · p. 7

4.3 SOBOLEV NOVELTY AS PRETRAINING GUIDANCE

  • SN-filtered pretraining formulas reach mean R2 0.712 at 35M samples, versus 0.497 for Random and 0.350 for PhyE2E formulas.The experiment changes the formula distribution while leaving E2ESR’s architecture, training objective, and inference procedure unchanged.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 SOBOLEV NOVELTY ENABLES SCIENTIFIC DISCOVERY IN TURBULENCE MODELING

  • SN-DSRRANS reduces reattachment-position error from 1.91 to 0.53 and recirculation-velocity error from 2.52 × 10−3 to 0.38 × 10−3 m/s versus DSRRANS; SN-MCTS reduces these errors by 7.1% and 60.8%.SN-MCTS is also closer to DNS than RANS and DSRRANS on both quantities.Source: Table 1 · p. 9, §4.4 · p. 9
  • After embedding discovered constitutive relations in a RANS solver, SN-DSRRANS suppresses DSRRANS’s concentrated upper-channel high-velocity band and better recovers the broader, more continuous distribution seen in DNS.The field comparison evaluates system-level physical fidelity under coupled simulation, beyond pointwise fitting of training data.Source: §4.4 · p. 9, Figure 4 · p. 9

5 CONCLUSION

The conclusion presents Sobolev Novelty as a theory-calibrated term-level measure and reports its use across symbolic-regression methods and turbulence modeling.

5 CONCLUSION

  • Sobolev Novelty measures structural independence by jointly characterizing term function values and derivatives, and the paper proves connections to independent Sobolev energy, deletion-and-refit cost, and coefficient identifiability.The paper uses a theory-calibrated criterion for structural novelty.Source: §5 · p. 9
  • Empirical analysis finds a novelty gap between scientific equations and current SR solutions, while Sobolev guidance improves search-based and generative methods through search guidance and pretraining data selection.In turbulence modeling, discovered constitutive relations yield more accurate flow predictions in coupled physical simulations.Source: §5 · pp. 9–10
  • The paper identifies higher-order Sobolev representations and a broader range of scientific-discovery tasks as directions for future work.Source: §5 · p. 10

A THEORETICAL FOUNDATIONS AND COMPLETE PROOFS

The appendix develops the theory behind Sobolev Novelty, from an empirical value-and-gradient geometry to projection-based novelty and term-attribution stability. Its results characterize novelty through residuals and connect it to attribution sensitivity under a full-column-rank assumption.

A.1 PROBLEM SETTING AND EMPIRICAL SOBOLEV GEOMETRY

  • The empirical Sobolev geometry represents functions using sampled values and derivatives, with a fixed scale and shared input samples, coordinates, and positive weights.The resulting function-space quantity is only a seminorm because finite samples may not distinguish distinct functions; target values and derivatives are not needed.Source: §A.1 · p. 12

A.2 SOBOLEV NOVELTY AND THE PROJECTION PERSPECTIVE

  • Sobolev Novelty is the normalized leave-one-term-out residual in an empirical Sobolev space, measuring the part of a term’s representation outside the span of the remaining terms.The analysis relates squared novelty to the fraction of the term’s value-and-gradient energy that cannot be reconstructed by the other terms.Source: §A.2 · p. 13
  • Novelty ranges from 0 to 1: it is zero when a term lies in the span of the remaining terms and one when it is orthogonal to that span.A nonzero single-term signature has novelty 1.Source: §A.2 · p. 13

A.3 TERM IDENTIFIABILITY AND THE EXPLANATION STABILITY PERSPECTIVE

  • Under a full-column-rank assumption, the appendix connects novelty to the sensitivity of the behavior attributed to an individual term when coefficients change.The attribution is defined through the candidate’s value-and-gradient behavior, and the sensitivity characterization is stated in the normalized signature representation.Source: §A.3 · pp. 13–14

A.4 DERIVATION OF THE THEORY-CALIBRATED THRESHOLD

The appendix derives a novelty threshold from a bound on attribution sensitivity and characterizes the cost of deleting and refitting a term. The related-work discussion positions Sobolev Novelty as a term-level measure of independent structural information.

A.4 DERIVATION OF THE THEORY-CALIBRATED THRESHOLD

  • The theory-calibrated criterion retains terms whose squared attribution sensitivity is within one order of magnitude of the orthogonal case, equivalent to retaining more than 10% of signature energy outside the remaining-term span.The main text uses the strict rule ν_i > τ_id, with ν_i^2 > 0.1; the threshold is not optimized for a particular algorithm or dataset.Source: §A.4 · p. 15

A.5 OPTIMAL DELETION AND COEFFICIENT REFITTING

  • Deleting a term and optimally refitting the remaining fixed terms has a cost tied to its coefficient magnitude and novelty, while novelty alone measures relative reconstructability.The refit varies outer coefficients only and does not optimize target-label error or the terms’ internal parameters.Source: §A.5 · p. 15

B RELATED WORK

  • Existing accuracy, complexity, stability, influence, and coefficient-based criteria do not directly quantify each term’s independent structural information relative to the remaining terms.Sobolev Novelty is presented as a complementary measure of how much term behavior cannot be jointly reconstructed in an empirical Sobolev space.Source: B RELATED WORK · p. 16

C.1 WHITE-BOX TASKS

The appendix specifies the task sets, pretraining-corpus comparison, periodic-hill evaluation, and reporting metrics used in the experiments. It distinguishes white-box tasks with analytic references from black-box regression tasks without them.

C.1 WHITE-BOX TASKS

  • The white-box set contains 133 scientific equation tasks—119 Feynman and 14 Strogatz—with analytic ground truths, used for structural analysis and evaluations of symbolic-search and pretrained methods.Ground-truth expressions serve as structural references in the stated white-box evaluation.Source: §C.1 · p. 16

C.2 BLACK-BOX TASKS

  • The black-box set contains 122 regression tasks without reference analytic expressions and is used to evaluate GP, MCTS, and their Sobolev-guided variants, including IGSR and SN-IGSR.IGSR and SN-IGSR are evaluated only on the black-box tasks.Source: §C.2 · p. 16

C.3 PRETRAINING CORPORA

  • The E2ESR pretraining comparison uses random formulas, Sobolev-Novelty-filtered formulas, and PhyE2E formulas, holding the model architecture and training pipeline fixed.The settings use three independent runs and checkpoints from 1M to 35M accepted samples, evaluated by mean R2 over 133 white-box tasks.Source: §C.3 · pp. 16–17

C.4 PERIODIC-HILL FLOW

  • Periodic-hill flow is evaluated by embedding discovered constitutive relations in a RANS solver and comparing reattachment position, maximum recirculation-zone velocity, and streamwise velocity against DNS.Conventional RANS is the baseline; errors for the first two quantities are absolute deviations from DNS.Source: §C.4 · p. 17

C.5 EVALUATION METRICS AND REPORTING CONVENTIONS

  • Predictive accuracy is measured by test-set R2 and expression complexity by total expression-tree nodes; structural evaluation reports mean term novelty and the share exceeding the threshold.The structural analysis retains discovered expressions with test R2 ≥ 0.8 and computes novelty for ground-truth and retained expressions using the same input samples.Source: §C.5 · p. 17

D METHOD INTEGRATION AND IMPLEMENTATION DETAILS

The appendix describes four integrations of Sobolev Novelty: archival and recombination guidance for GP, reranking and deletion for MCTS, feedback for IGSR’s LLM, and formula filtering for E2ESR pretraining.

D.1 INTEGRATION WITH GENETIC PROGRAMMING

  • GP integration archives distinct terms from high-scoring expressions, prioritizes them by projection residual, and combines retained structures with fit-guided candidates.Final candidates are selected using the original GP accuracy–complexity score, without an additional novelty penalty.Source: §D.1 · p. 17

D.2 INTEGRATION WITH MONTE CARLO TREE SEARCH

  • MCTS first keeps at most 64 candidates by the original accuracy–complexity score, then reranks them using their constituent terms’ Sobolev Novelty.For the top-ranked candidate, the method may attempt one low-novelty term deletion and refit, accepting it only if the original score does not decrease.Source: §D.2 · p. 18

D.3 INTEGRATION WITH IGSR

  • IGSR adds feedback before LLM term generation, flagging low-novelty terms and providing up to three such terms and two high-novelty references.The validation objective and tree-search scoring remain unchanged, and no additional deletion or sibling-candidate reranking is performed.Source: §D.3 · p. 18

D.4 INTEGRATION WITH E2ESR

  • E2ESR uses formula-level rejection sampling before training, accepting multi-term formulas only when every term exceeds the novelty threshold.The architecture, training objective, optimizer, and inference procedures remain unchanged; the paper attributes final performance gains to pretraining data selection alone.Source: §D.4 · p. 19

D.5.1 BACKGROUND OF THE TURBULENCE MODELING TASK

The section frames turbulence closure as learning stress corrections beyond a restricted linear eddy-viscosity model. It describes the DSRRANS tensor representation and a forward-selection search that uses Sobolev Novelty to choose among similarly fitting terms.

D.5.1 BACKGROUND OF THE TURBULENCE MODELING TASK

  • The Reynolds stress represents turbulent fluctuations’ contribution to mean momentum transport, and an additional constitutive relation is needed to close the averaged flow equations.DNS provides high-fidelity reference statistics for learning and evaluating closure relations.Source: §D.5.1 · p. 19
  • Linear eddy-viscosity models constrain anisotropic stress to be proportional to mean strain rate, while DSRRANS adds a nonlinear symbolic stress correction.The restricted relation can lead to substantial errors in flows with separation, streamline curvature, and stress anisotropy.Source: §D.5.1 · p. 19
  • The regression uses transformed flow invariants as inputs and fits three coefficient functions jointly against tensor stress corrections from 9,600 local samples.The target corrects the linear eddy-viscosity contribution; prediction error is minimized over tensor components 11, 12, 22, and 33.Source: §D.5.1 · p. 20

D.5.2 CLOSURE SEARCH USING THE DSRRANS TENSOR REPRESENTATION

  • The original DSRRANS search uses an LSTM to generate coefficient expressions, tunes constants, and updates its policy using rewards from tensor-prediction errors.The nonlinear closure is a tensor-basis expansion with scalar coefficient functions of the transformed invariants.Source: §D.5.2 · pp. 20–21
  • The offline alternative searches a degree-12 bivariate monomial dictionary by forward selection, choosing the largest training-error reduction before jointly refitting selected terms.Sobolev guidance instead chooses the highest-novelty candidate among those achieving at least 94% of the largest current error reduction; novelty is measured against previously selected terms in the same coefficient channel.Source: §D.5.2 · p. 21
  • When additional compression is needed, both search settings delete terms using training error alone and jointly refit after each deletion.Source: §D.5.2 · p. 21

E.1 PER-METHOD TERM QUALIFICATION STATISTICS

The appendix reports term qualification rates under the theory-calibrated novelty threshold and identifies the scope of the accompanying white-box and black-box search results.

E.1 PER-METHOD TERM QUALIFICATION STATISTICS

  • 92.6% of ground-truth terms qualify, versus a 38.3% macro-average across 15 SR methods, a 54.3-percentage-point difference.The criterion is ν_i > τ_id; the method average is an unweighted macro-average of method-level qualification rates.Source: §E.1 · p. 21, Table 2 · p. 22, §E.2 · p. 22

E.2 SOBOLEV NOVELTY AS SEARCH GUIDANCE

  • The white-box and black-box result tables report method-level comparisons, including mean R2, expression complexity, and mean Sobolev Novelty.The appendix identifies them as the complete results corresponding to Figure 2 in the main text.Source: §E.2 · pp. 21–22, Table 3 · p. 22, Table 4 · p. 23

F ABLATION OF FIRST-ORDER DERIVATIVES

The ablation tests whether first-order derivatives improve SN-MCTS search accuracy and throughput under a fixed wall-clock budget.

F ABLATION OF FIRST-ORDER DERIVATIVES

Loading 2609.32597v1…