Source-linked AI summary
Interpretable Scientific Discovery with Symbolic Regression: A Review
Nour Makke, Sanjay Chawla
TL;DR
Symbolic regression addresses the need for predictive models whose relationships remain mathematically interpretable, especially in scientific and critical applications. This survey reviews SR formulations, methods, applications, strengths, and limitations, finding that method behavior and recovery depend strongly on the allowed operation library. It concludes that SR can balance prediction accuracy and interpretability, although applications to real data remain limited.
Problem
Symbolic regression is needed to infer interpretable mathematical relationships from data rather than relying only on accurate but opaque predictive models.
Method
The survey provides a structured review of SR problem definitions, methodological categories, applications, benchmarks, strengths, and limitations.
Results
3% recovery rate was achieved for Nguyen-12 with NGPPS using a pure polynomial basis, highlighting the strong dependence of recovered expressions on the allowable operation library.
Takeaways & Limitations
Symbolic regression is a promising tool for data-driven discovery of interpretable models across fundamental and applied sciences.
Takeaways & Limitations
SR methods can fail to recover ground-truth expressions involving products, multiplicative or additive factors, or mixed function libraries, and real-data applications remain limited.
Abstract
from arXiv · showhide
Symbolic regression is emerging as a promising machine learning method for learning succinct underlying interpretable mathematical expressions directly from data. Whereas it has been traditionally tackled with genetic programming, it has recently gained a growing interest in deep learning as a data-driven model discovery method, achieving significant advances in various application domains ranging from fundamental to applied sciences. This survey presents a structured and comprehensive overview of symbolic regression methods and discusses their strengths and limitations.
1 Introduction
Symbolic regression seeks mathematical expressions that are both predictive and interpretable, supporting scientific reasoning and deployment in settings where black-box models are unsuitable. This survey reviews SR methods, applications, benchmarks, strengths, and limitations.
- Motivation: Symbolic regression infers symbolic mathematical expressions from data to provide models whose input-output relationships can be logically or mathematically traced.Interpretability is especially valuable in natural sciences and critical disciplines such as healthcare.
- Interpretability and prediction: Linear models expose feature effects but may impose inaccurate structures, whereas neural networks can be predictive yet difficult to interpret mathematically.The force example contrasts a linear form with a neural-network representation whose nonlinear transformations obscure the input-output relationship.
- Scientific discovery: SR aims to connect data-driven empirical modeling with underlying physical relationships, illustrated by the contrast between Kepler’s fitted orbits and Newton’s dynamical laws.Both approaches fit observations, but Newton’s formulation described the process underlying the observed orbits.
- Historical development: Historical SR systems pursued empirical-law rediscovery, including BACON’s rediscovery of versions of Coulomb’s law and Galileo’s laws.These systems iteratively applied data-driven heuristics to formulate mathematical expressions.
- Survey scope: The survey organizes SR methods, applications, and benchmarks while discussing their strengths and limitations to characterize the field’s current status.It also accompanies a living review intended to track new SR methods and applications.
2 Problem Definition
Symbolic regression formulates model discovery as empirical-risk minimization over a discrete function class generated by composing operations, functions, variables, and constants from a library. Expressions can be represented as unary-binary trees or equivalent sequences, while library design shapes the search space.
- Problem formulation: SR minimizes a loss over candidate functions f in a function class F, with inputs x_i in R^d and scalar outputs y_i in R.The problem is expressed using classical empirical risk minimization.
- Problem formulation: Unlike conventional regression, SR uses a discrete function class, so solving the problem requires characterizing which functions belong to F.This discrete structure distinguishes SR from regression over a fixed continuous parameterization.
- Function library: The function class is generated by composing elementary operations, mathematical functions, variables, and constants drawn from a predefined library.For example, a library containing identity, arithmetic operations, and constants can generate all one-variable integer-coefficient polynomials.
- Expression representation: Unary-binary trees encode expressions with operators at internal nodes and variables or constants at terminal nodes.The expression f(x) = x1x2 −2x3 is represented as a tree and can also be traversed into a unique Polish-form sequence.
- Function library: Domain knowledge can improve library definition by restricting operations to those relevant to the studied problem, while the library must also express a broad range of numeric constants.A more focused library reduces the search space but can exclude needed structures if relevant operations are omitted.
3 Symbolic regression methods overview
The survey groups symbolic regression into regression-based, expression tree-based, physics-inspired, and mathematics-inspired methods. These approaches differ in representation and search strategy, spanning fixed parameter optimization, neural-network structures, evolutionary trees, transformers, and reinforcement learning.
- Taxonomy: The survey categorizes SR methods as regression-based, expression tree-based, physics-inspired, and mathematics-inspired.For each category, it summarizes the mathematical tool, expression form, unknowns, and search space.
- Regression-based methods: Linear regression-based methods use a fixed linear combination of library functions and optimize continuous parameters.Their deterministic formulation is offset by imposing a single model structure throughout training.
- Regression-based methods: Nonlinear regression-based methods define the model structure with a neural network and optimize its real-valued weights and biases.The neural-network formulation uses nonlinear activations and continuous parameter optimization.
- Expression tree-based methods: Expression tree-based methods represent mathematical expressions as unary-binary trees and include genetic programming, transformers, and reinforcement learning.Genetic programming applies transition rules such as mutation, crossover, and selection, while transformer and reinforcement-learning methods operate on sequences or policies.
- Expression tree-based methods: Transformer SR encodes input-output data and expressions as sequences, using attention-based encoder-decoder architectures for set-to-sequence prediction.The unknowns are the encoder and decoder weight parameters.
4 Linear symbolic regression
Linear symbolic regression restricts candidate expressions to linear combinations of library functions, converting symbolic discovery into coefficient estimation. Experiments show exact recovery for supported polynomial forms, while missing or incompatible library operations produce approximations or failures.
- Model formulation: The linear approach assumes the target expression is a linear combination of nonlinear functions selected from a predefined library.This fixes the model structure and leaves the coefficients as the unknowns.
- Model formulation: Each candidate library function receives a coefficient whose value indicates whether that function is active in the learned expression.The resulting input-output equations can be assembled into a linear system and solved for the coefficient vector.
- Library construction: The library matrix contains evaluations of candidate functions, including constants, polynomial terms, and optionally trigonometric or other elementary operations.Prior domain knowledge can narrow the library and reduce the search space, but the chosen library constrains recoverable expressions.
- Univariate experiments: For univariate synthetic data, L1 exactly recovers pure polynomial functions with R2 = 1.0, whereas mixed polynomial-trigonometric cases are not recovered exactly.For Nguyen-5, the learned function is a Taylor expansion rather than the exact sin(x2) cos(x) −1 expression.
- Univariate experiments: Linear symbolic regression cannot learn products of functions or expressions with multiplicative or additive factors inside nonlinear arguments.In such cases, it produces at best an approximation and may fail to recover the correct mathematical expression.
- Multivariate experiments: In multivariate tests, polynomial libraries exactly match polynomial targets, while trigonometric targets are approximated unless the relevant operations are included.For y1 = cos(x1) + sin(x2), a pure polynomial basis has reconstruction error of about 30%, whereas trigonometric libraries achieve error ≥10^-7; a mixed-term target remains limited when cos(x1)∗sin(x2) is absent.
5 Nonlinear symbolic regression
Nonlinear symbolic regression uses differentiable neural networks whose activation functions are replaced by mathematical operators, while sparsity encourages simpler expressions. These methods scale to high-dimensional problems but may restrict expressions involving division, logarithms, or exponentials.
- EQL replaces standard neural-network activations with elementary mathematical operations to search for symbolic expressions through differentiable training.Each hidden node can use a specific activation from a function library rather than sharing one conventional activation function.
- The EQL network learns weight parameters while selecting expression operators from a predefined library during training.The library includes functions such as identity, powers, trigonometric functions, exponential, logarithm, and sigmoid.
- L1 regularization is added to the L2 loss to enforce sparsity and favor simpler symbolic solutions.The stated motivation is to address neural-network interpretability and promote simple over complex formulas.
- Differentiability enables end-to-end training and scaling to high-dimensional problems, but back-propagation through division or logarithm requires simplifying the search space.This restriction limits the ability to produce simple expressions containing divisions, such as sin(x/y).
- EQL÷ adds division as an activation while excluding exponential and logarithm functions because of numerical issues.
6 Tree expression
Tree-based and neural sequential approaches represent symbolic expressions through structured search over operators, operands, and tokens. Genetic programming evolves expression trees, while transformers and reinforcement learning model dependencies or sequential decisions in symbolic regression.
- Tree expression: Expression-tree symbolic regression represents operators as internal nodes and variables or constants as terminal nodes drawn from a predefined function class.The example expression f(x) = x^2 − cos(x) illustrates both the tree structure and its corresponding function values.
- Genetic programming: Genetic programming evolves randomly initialized populations of expression trees using mutation, crossover, and selection.Mutation replaces a subtree, crossover exchanges subtrees between individuals, and selection retains high-fitness solutions.
- Genetic programming: Each genetic-programming generation applies transition rules, evaluates loss, and selects elite individuals until a predetermined accuracy level is reached.
- Genetic programming: GP can improve out-of-distribution performance through large population variations but does not scale well to high-dimensional datasets and is highly hyperparameter-sensitive.
- Transformers: Transformer encoders use attention and feed-forward layers to produce context-dependent representations from input embedding sequences.Self-attention derives weights from query-key relationships and combines value vectors.
- Transformers: Transformer-based symbolic regression encodes numerical inputs and symbolic expressions with different vocabularies, then predicts either expression skeletons or skeletons and constants together.In the skeleton approach, constants are fitted separately with optimization methods such as BFGS; the end-to-end approach predicts both simultaneously.
- Reinforcement learning: In reinforcement-learning symbolic regression, the neural model acts as a policy that observes states and samples symbols as actions receiving rewards.The underlying setting consists of states, actions, transitions, and rewards over time until a terminal state is reached.
7 Applications
SR applications use either direct data input or a two-step pipeline that first learns a data representation or blackbox model. The survey covers GP-, neural-, physics-, and mathematics-inspired methods, including applications that recover or interpret scientific relationships.
- Application strategies: SR algorithms use data, a learned data representation, or a learned model as input.The latter two choices define two-step approaches, while direct data input defines the one-step approach.
- Application strategies: Two-step SR either reduces the original variables through features or neural architectures, or applies SR to a learned neural or graph-based model.The survey specifically mentions principal component analysis, autoencoders, regular neural networks, and graph neural networks.
- Representation-learning applications: SINDY autoencoders identified the nonlinear pendulum dynamics as ¨z = −0.99 sin z in a reduced representation.The method is described as particularly efficient when the dynamical model may be dense in the original coordinates.
- Blackbox interpretation: Symbolic metamodels map blackbox predictions to symbolic expressions, exposing the learned functional form rather than only selected prediction explanations.For a healthcare example, the resulting expression identifies influential features and their directional relationships.
- Deep-learning methods: E2ESR uses an embedder, feedforward network, and transformer to generate expressions end-to-end without operator skeletons.It represents operators and variables symbolically while representing constants numerically, and reports accuracy, complexity, and inference time.
- Scientific discovery applications: Physics-inspired and hybrid methods include recursive solver application, neural-guided GP population seeding, and large-scale transformer pre-training.The survey also reports cosmological equation discovery better than a hand-designed equation and recovery of Newton’s gravitational law from solar-system trajectories.
8 Datasets
SR benchmarks are divided into ground-truth problems with known equations and real-world problems with unknown underlying models. Ground-truth datasets support controlled evaluation but can use physically unrealistic constants, while real-world datasets span many application domains.
- Benchmark taxonomy: SR benchmarks comprise ground-truth synthetic problems and real-world problems whose underlying model is unknown.The taxonomy is summarized in Figure 18.
- Ground-truth problems: Ground-truth problems include physics-inspired equations and real-valued symbolic functions.The survey summarizes these problems in Table 6.
- Ground-truth problems: The Feynman database contains 1194 physics-inspired equations with one to nine variables, generating one million randomly sampled entries per benchmark.Each benchmark corresponds to one equation describing a static physical system or physics process.
- Ground-truth limitations: The Feynman benchmark does not distinguish variables from constants and can sample constants far from their physical values.The survey gives unrealistic sampling ranges for the speed of light and gravitational constant as examples.
- Ground-truth problems: Other synthetic benchmarks contain mostly univariate or bivariate polynomial, trigonometric, logarithmic, exponential, and square-root functions without physical meaning.Many were proposed for GP-based methods.
- Real-world problems: Real-world benchmarks include observations and measurements from domains such as health informatics, environmental science, business, and commerce.Repositories named by the survey include PMLB, OpenML, and UCI.
9 Discussion
The survey presents SR as a response to the interpretability limits of increasingly large machine-learning models, but reports important weaknesses in expression recovery and real-world validation. Performance can depend strongly on the allowed operation library, and most applications still use synthetic data.
- Motivation: SR is motivated by the need for models whose input-output relationships can be traced mathematically, especially in scientific disciplines.The discussion contrasts this requirement with large models whose internal behavior is difficult to understand.
- Recovery limitations: NGPPS recovered Nguyen-12 at only 3% with a pure polynomial library, while the same expression was not recovered with a mixed library.The result highlights strong dependence on the set of allowable mathematical operations.
- Recovery limitations: Adding domain-relevant operations to the library is presented as a practical way to address recovery failures.The survey uses periodic astronomical data as an example where trigonometric functions should be included.
- Real-world validation: Most SR methods are evaluated on synthetic data because known ground-truth expressions enable cross-checking and performance evaluation.The survey reports only one physics application that extracted Newton’s laws from astronomical data.
10 Conclusion
The review synthesizes limitations and performance patterns across symbolic regression methods, while identifying interpretability and prediction accuracy as complementary strengths. It also points to experimental physics data as a potential avenue for further progress.
- Linear symbolic regression is limited by its predefined model structure.
- Neural network-based symbolic regression methods face numerical issues and cannot include all mathematical operations in their libraries.
- Expression tree-based methods, particularly transformer-based ones, achieve the strongest performance on synthetic data.
- Model predictions strongly depend on the allowable operations in the library basis.
- Symbolic regression balances prediction accuracy and interpretability, while its limited applications to real data remain promising.
- Applying symbolic regression to experimental physics data is proposed as a potential path for advancing the subfield.
A Datasets Benchmarks Equations
The section catalogs benchmark ground-truth expressions and collections of physics equations used in symbolic regression evaluation and scientific examples. These materials span named benchmark suites and equations from mechanics, electromagnetism, relativity, optics, and thermodynamics.
- Tables 7–9 list ground-truth expressions for the Koza, Nguyen, Jin, Keijzer, R, Korns, Livermore, and Vladislavleva benchmarks.
- Tables 10–11 present Feynman physics equations as a collection of scientific expressions.
- The listed equations include force, energy, momentum, optics, relativity, wave, quantum, and thermodynamic relationships.
- Additional listed expressions cover interference, radiation, electric fields, photon energy, and pressure-volume-temperature relationships.