Source-linked AI summary
InsightSR: Refining Symbolic Regression Search Spaces via Parallel Semantic and Structural LLM Guidance
Yating Ling, Wenjing Cun, Zhitang Chen
TL;DR
Symbolic regression must search vast expression spaces while preserving physical plausibility. InsightSR uses LLM-guided semantic seeds, structural features, and closed-loop refinement around PySR, achieving strong benchmark accuracy and real-world generalization.
Problem
Symbolic regression struggles with combinatorial search spaces and with enforcing physical consistency, so data-driven methods may produce accurate expressions that violate physical principles.
Method
InsightSR guides PySR through dimensionally consistent semantic seeds, accumulating nonlinear feature transformations, and post-generation feedback that updates both pathways.
Results
Across three benchmarks, InsightSR achieves state-of-the-art accuracy while maintaining strong out-of-distribution generalization on real-world datasets.
Takeaways & Limitations
The framework shifts symbolic discovery toward shallow assembly over semantically enriched features and iterative refinement informed by search outcomes.
Takeaways & Limitations
Performance may depend on initial metadata and accurate unit synthesis when variable semantics are ambiguous.
Abstract
from arXiv · showhide
Symbolic regression (SR) seeks to discover parsimonious mathematical laws from observational data, yet conventional approaches often struggle with the vast combinatorial search space of physically meaningful expressions. We present InsightSR, a framework that embeds Large Language Models (LLMs) as a guiding layer around the PySR genetic programming engine. Rather than relying on LLMs to generate expressions directly, InsightSR uses LLMs to progressively transform the search space itself through two complementary pathways: a Semantic Seed Pathway that proposes dimensionally consistent functional skeletons, and a Structural Feature Pathway that recommends nonlinear feature transformations. These transformations accumulate over iterations, broadening the input space and shifting the symbolic search from constructing deep expression trees over raw variables to assembling shallow trees over a rich, semantically informed feature set. A post-generation feedback loop evaluates candidates, categorizes features by their empirical utility, and refines the guidance for the next iteration, transforming the discovery process from open-ended generation into iterative, self-correcting refinement. Across three benchmarks, InsightSR achieves a 95% exact recovery rate on the Feynman benchmark and 80.18% accuracy on the LLM-SRBench LSR-Transform task, substantially outperforming state-of-the-art genetic programming and neural-symbolic methods while maintaining strong out-of-distribution generalization on real-world datasets.
Introduction
InsightSR addresses symbolic regression’s combinatorial search and physical-consistency challenges by using LLMs to transform PySR’s search space through semantic seeds, structural features, and iterative feedback.
- Motivation: Symbolic regression seeks interpretable mathematical laws but faces exponentially growing expression spaces and difficulty enforcing physical constraints.Purely data-driven methods can achieve high numerical accuracy while violating basic physical principles.
- Limitations of Existing Approaches: Existing GP, MCTS, neural, and LLM-guided approaches search expression space but can have low sample efficiency, overfit, incur inference overhead, or guide only one dimension.The cited methods include GP variants, MCTS, deep reinforcement learning, transformers, and recent LLM-based systems.
- InsightSR: InsightSR embeds LLMs around PySR rather than generating final expressions directly, progressively transforming the search space through complementary guidance pathways.The framework combines semantic seeds, structural feature transformations, and a feedback loop that updates guidance across generations.
- Key Contributions: The Semantic Seed Pathway proposes dimensionally consistent functional skeletons and operator biases that prune physically inconsistent structures before numerical fitting.These domain-aware seeds provide a physics-informed warm-start for evolutionary search.
- Key Contributions: The Structural Feature Pathway accumulates nonlinear transformations across iterations, shifting search from deep trees over raw variables to shallow combinations over an enriched feature basis.This reduces the structural complexity that the symbolic engine must resolve.
- Key Contributions: Closed-loop analysis evaluates candidate accuracy, interpretability, and feature utility, then accumulates insights to refine both pathways in subsequent iterations.This converts discovery into iterative, self-correcting refinement.
Method
InsightSR guides PySR through an iterative process that resolves units, generates semantic seeds and engineered features, searches with biased complexity costs, and learns from candidate evaluations.
- Objective: The symbolic regression objective balances dataset fit against structural parsimony using MSE plus a complexity penalty.Expression complexity is measured by the number of nodes in the expression tree, with λ controlling the penalty.
- Iterative Loop: Each generation integrates LLM-derived domain knowledge before, during, and after evolutionary search, shifting PySR toward purposefully guided optimization.The process is organized as a four-phase iterative loop.
- Contextual Initialization and Unit Synthesis: Contextual initialization extracts or infers variable dimensions from metadata and problem semantics, preserving unit assignments as constraints throughout discovery.Dimensional homogeneity prunes physically inconsistent candidates.
- Parallel Guidance Strategy: Parallel guidance combines domain-aware functional skeletons with LLM-guided nonlinear feature engineering based on prior-generation performance.The semantic pathway supplies candidate topologies, while the structural pathway identifies high-utility transformations.
- Parallel Guidance Strategy: Augmented inputs concatenate raw variables with engineered transformations, which accumulate across generations and enable shallow combinations instead of deep expression trees.This progressively broadens the input space while reducing the structural depth required from the evolutionary engine.
- Search with PySR: Semantic seeds impose macro-level physical constraints while engineered features provide empirical building blocks, together confining PySR to a physically plausible but numerically flexible region.The two pathways converge within the PySR engine.
- Search with PySR: A complexity-biasing mechanism modifies baseline operator costs using LLM-provided biases, while warm-starting retains the previous generation’s best candidate.Recommended operators receive discounts, whereas physically implausible operators receive penalties.
- Strategic Analysis: After each search, the LLM evaluates Pareto-optimal candidates for numerical accuracy, physical interpretability, variable completeness, and engineered-feature utility.These assessments guide subsequent iterations.
Experimental Setup
The evaluation uses three benchmarks spanning exact symbolic recovery, cross-domain generalization, and real-world out-of-distribution generalization, alongside R2-, NMSE-, and tolerance-based accuracy metrics.
- Benchmarks: Three benchmarks evaluate InsightSR across exact recovery, cross-domain generalization, and real-world out-of-distribution performance.Feynman provides known ground-truth expressions, LLM-SRBench varies domains and data-generating mechanisms, and the Real-World benchmark includes ID and OOD splits.
- Benchmarks: The Feynman 100 benchmark contains 100 physics equations and enables exact recovery evaluation against known ground-truth expressions.Its problems cover classical mechanics, electromagnetism, quantum mechanics, and thermodynamics.
- Benchmarks: LLM-SRBench assesses cross-domain generalization across Physics, Chemistry, Biology, and Material Science through LSR-Transform and LSR-Synth categories.The benchmark contains five categories with different data-generating mechanisms and varying complexity.
- Benchmarks: The Real-World benchmark evaluates out-of-distribution generalization on four datasets using both in-distribution and out-of-distribution test splits.The datasets are Oscillator 1, Oscillator 2, E. coli, and Stress-Strain.
- Evaluation Metrics: R2 measures the proportion of variance captured, NMSE is defined as 1 − R2, and accτ marks success when maximum relative error stays within τ.Exact Recovery Rate is a special case in which the symbolic form is mathematically equivalent to the ground truth.
Results
InsightSR performs strongly across benchmark settings, recovering known Feynman equations accurately and achieving high LSR-Transform accuracy. It also maintains low error across synthetic and real-world evaluations, including out-of-distribution tests.
- Feynman Benchmark: 95% exact recovery and 0.9999 average R2 were achieved on the Feynman benchmark.The median loss declines with increasing expression complexity, while the interquartile range narrows at higher complexity levels.
- LLM-SRBench: 80.18% Acc0.1 on LSR-Transform exceeded the previous best of 50.45% achieved by LaSR.The evaluation spans five scientific domains on LLM-SRBench.
- LLM-SRBench: Accuracy reached 66.67% in Chemistry, 50.00% in Biology, 40.91% in Physics, and 92.00% in Material Science.NMSE values were several orders of magnitude lower than those of other LLM-based solvers.
- Real-World Datasets: Oscillator 1 and Oscillator 2 achieved ID NMSE values of 9.55×10^-11 and 2.45×10^-9, respectively.These values represented improvements of several orders of magnitude over LLM-SR, the previous best LLM-based method.
Discussion
InsightSR combines semantic seeding and structural feature engineering to address symbolic regression’s search-space and physical-consistency challenges. Across benchmarks, the framework reports strong recovery, accuracy, numerical precision, and real-world generalization, while remaining sensitive to the quality of initial metadata.
- Semantic Seed Pathway: 95% exact recovery on the Feynman benchmark reflects the Semantic Seed Pathway’s physics-informed warm-start.Dimensionally consistent skeletons bypass physically meaningless regions that would otherwise dominate early GP generations.
- Structural Feature Pathway: 80.18% accuracy on the LSR-Transform task reflects the Structural Feature Pathway’s reduction of the expression-depth bottleneck.Accumulated nonlinear transformations broaden the input space and shift search toward shallow combinations over enriched features.
- Limitations: The framework may be sensitive to problem descriptions because its unit-synthesis module relies on initial metadata.The learning-from-results loop queries the LLM between generations rather than at every search step.
Conclusion
InsightSR places LLMs around PySR through semantic seeding, structural feature engineering, and closed-loop feedback that progressively transforms the symbolic-regression search space. Evaluations report state-of-the-art accuracy across three benchmarks and strong out-of-distribution generalization on real-world datasets.
- Conclusion: InsightSR embeds LLMs as a guiding layer around PySR rather than generating candidate expressions directly.Its two pathways progressively transform search from deep trees over raw variables to shallow assembly over enriched features.
- Conclusion: A closed-loop feedback mechanism learns from search outcomes and refines guidance across generations.The conclusion characterizes this integration as a practical path toward automated scientific discovery.
- Conclusion: InsightSR achieves state-of-the-art accuracy across three benchmarks while maintaining strong out-of-distribution generalization on real-world datasets.These are the paper’s reported evaluation-level outcomes.