Source-linked AI summary
Contemporary Symbolic Regression Methods and their Relative Performance
William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Virgolin, Ying Jin, Michael Kommenda, Jason H. Moore
TL;DR
The paper addresses the lack of uniform, robust, and transparent symbolic-regression benchmarks by building an open, reproducible platform. It evaluates contemporary symbolic-regression and machine-learning methods across diverse real-world and ground-truth problems, finding different leading approaches across task conditions. The benchmark is intended to support ongoing, controlled comparison and contribution.
Problem
Symbolic-regression research lacks standardized, transparent, reproducible benchmarks large and diverse enough to establish current state-of-the-art methods reliably.
Method
The authors create SRBench, combining curated datasets, 14 symbolic-regression methods, 7 machine-learning methods, shared evaluation code, and open contribution infrastructure.
Results
Across tasks, GP-based methods perform best on real-world and black-box regression, while AIFeynman leads exact recovery with minimal noise and DSR and GP-based methods perform best otherwise.
Takeaways & Limitations
Controlled, open benchmarking supports more certain comparisons of symbolic-regression methods across prediction, model simplicity, and exact equation recovery.
Takeaways & Limitations
The Symbolic Solution metric cannot guarantee perfect fidelity because equivalent representations are numerous and SymPy simplification is non-optimal.
Abstract
from arXiv · showhide
Many promising approaches to symbolic regression have been presented in recent years, yet progress in the field continues to suffer from a lack of uniform, robust, and transparent benchmarking standards. In this paper, we address this shortcoming by introducing an open-source, reproducible benchmarking platform for symbolic regression. We assess 14 symbolic regression methods and 7 machine learning methods on a set of 252 diverse regression problems. Our assessment includes both real-world datasets with no known model form as well as ground-truth benchmark problems, including physics equations and systems of ordinary differential equations. For the real-world datasets, we benchmark the ability of each method to learn models with low error and low complexity relative to state-of-the-art machine learning methods. For the synthetic problems, we assess each method's ability to find exact solutions in the presence of varying levels of noise. Under these controlled experiments, we conclude that the best performing methods for real-world regression combine genetic algorithms with parameter estimation and/or semantic search drivers. When tasked with recovering exact equations in the presence of noise, we find that deep learning and genetic algorithm-based approaches perform similarly. We provide a detailed guide to reproducing this experiment and contributing new methods, and encourage other researchers to collaborate with us on a common and living symbolic regression benchmark.
1 Introduction
The paper responds to weak, inconsistent symbolic-regression benchmarking by introducing an open, reproducible platform for broad method comparisons. It combines curated datasets, contemporary methods, and revision-controlled experiments to support more certain state-of-the-art assessment.
- Benchmarking gap: Symbolic regression literature lacks standardized, transparent, reproducible benchmarks spanning large and diverse problem collections.This makes it difficult to determine which methods or method families are state of the art.
- Motivation: The authors seek common, robust benchmarking to strengthen empirical guidance and cross-pollination among symbolic regression research communities.The communities include evolutionary computation, physics, engineering, statistics, and machine learning.
- Contribution: The benchmark effort combines a curated symbolic-regression dataset repository with a library that lets researchers contribute methods.The repository added 130 ground-truth datasets to PMLB with equation metadata, units, and summary statistics.
- Scope: The study compares 14 contemporary symbolic-regression methods on hundreds of regression problems.The comparison emerged from a contribution process involving researchers in the field.
- Reproducibility: The datasets, methods, benchmarking code, and results are open-source, reproducible, and revision-controlled for use as a living benchmark.The authors describe this as the largest and most comprehensive symbolic-regression benchmark effort to date.
2 Background and Motivation
Symbolic regression searches over analytical expressions and parameters, but its evaluation must balance accuracy, simplicity, and recovery of known solutions. The paper motivates controlled open benchmarking partly because commercial tools and prior studies do not support sufficiently standardized comparisons.
- SR formulation: Symbolic regression searches both analytical expression structures and parameters to estimate a noisy data-generating model.Its goal is to learn an analytical mapping from features to targets.
- Method families: Genetic programming represents candidate symbolic models as syntax trees and evolves populations through variation and selection.Koza-style GP uses functions or operations over input features and constants.
- Method families: Recent symbolic-regression methods include Bayesian optimization, recurrent neural networks, and physics-inspired divide-and-conquer strategies.These approaches emerged alongside renewed interest in learning interpretable models explicitly.
- Benchmarking constraints: Commercial platforms such as Eureqa and Wolfram cannot hold computational effort, evaluations, resource limits, and solution assessment constant in controlled comparisons.Their closed-source design also prevents researchers from uniquely identifying which software or experimental features produce performance differences.
- Evaluation criteria: Evaluation must consider the accuracy–simplicity trade-off, while recognizing that simple models may remain uninterpretable or incorrect.Ground-truth datasets help test whether recovered expressions match known analytical solutions, but synthetic benchmarks alone are insufficient.
- Prior work: Prior benchmarks were narrower, whereas this work evaluates 14 symbolic-regression methods and 7 machine-learning methods on 252 datasets.Earlier studies often focused only on prediction accuracy or solution rates.
3 SRBench
SRBench unifies diverse datasets, contemporary symbolic-regression and machine-learning methods, and shared evaluation infrastructure in an extensible open-source project. Its reproducibility and continuous-update mechanisms support repeated comparison and future method contributions.
- Platform: SRBench organizes diverse benchmark datasets, symbolic-regression methods, and machine-learning methods within a shared model-evaluation and analysis environment.The project is designed to make comparisons more certain than in previous symbolic-regression work.
- Datasets: PMLB was extended with 130 symbolic-regression datasets having known model forms and metadata about equations, units, and summary statistics.PMLB also provides utilities for data handling, metadata visualization, and new dataset contributions.
- Method integration: Contributors implement contemporary methods through a minimal Scikit-learn-compatible Python API.The benchmark includes implementations from several method families dating from 2011–2020.
- Reproducibility: Fixed package versions, complete installation and experiment code, published results, and automated incorporation of new methods support reproducibility.A continuous-integration framework updates reporting and visualization when new experiments become available.
- Method catalog: Table 1 catalogs the benchmarked symbolic-regression methods with short descriptions, references, and implementation links.The table supports method comparison and contribution tracking within the benchmark.
- Experiment settings: Table 2 specifies benchmark experiment settings, including total algorithm evaluations for each dataset, noise level, and random seed.These settings define the computational comparison conditions used by the experiments.
4 Experiment Design
The experiment evaluates symbolic regression methods across prediction and exact-equation-recovery tasks using diverse black-box and ground-truth datasets. It compares method families through accuracy, model complexity, and a symbolic-solution criterion.
- Tasks and datasets: The benchmark tests prediction accuracy with minimized model complexity and exact recovery of known ground-truth functions as separate tasks.The exact-recovery datasets originate from physics and engineering problems.
- Symbolic regression methods: Program semantics guide optimization through model outputs on training cases, including randomized-case filtering in ϵ-lexicase selection and target-matching intermediate outputs in semantic backpropagation.EPLEX rewards models performing well on difficult regions of the training data; SBP-GP adapts intermediate outputs using affine transformations.
- Symbolic regression methods: The evaluated methods span genetic programming, Pareto optimization, semantic search, gradient-based constant optimization, structured encodings, statistical recombination, Bayesian sampling, reinforcement learning, and divide-and-conquer.Examples include AIFeynman, EPLEX, SBP-GP, Operon, FEAT, MRGP, ITEA, FFX, GP-GOMEA, BSR, and DSR.
- Tasks and datasets: The benchmark includes 122 black-box problems spanning real-world and synthetic datasets across health, business, technology, environmental science, and government.These datasets range from 47 to 1 million instances and from two to 124 features.
- Metrics: Complexity counts mathematical operators, features, and constants, with both raw and sympy-simplified model forms evaluated.The operator set includes arithmetic, trigonometric, exponential, logarithmic, power, maximum, and minimum operations.
- Metrics: A symbolic solution must be nonconstant and differ from the ground-truth model by an additive constant or nonzero scalar factor.Models and solution conditions undergo sympy simplification, but the metric may miss valid solutions because representations are numerous and simplification is non-optimal.
5 Results
On black-box problems, Operon achieved the strongest test-set R2, while GP-based methods frequently ranked highly and several methods offered favorable accuracy–simplicity trade-offs. Ground-truth performance depended strongly on noise and dataset source.
- Black-box regression: Operon produced significantly more accurate models than every other method on black-box problems by test-set R2.The reported significance level was p ≤ 6.5e-05.
- Black-box regression: Four of the five highest-ranked methods and six of the ten highest-ranked methods were GP-based symbolic regression methods.The top-five group was Operon, SBP-GP, FEAT, EPLEX, and one additional method; the top-ten group also included GP-GOMEA and ITEA.
- Black-box regression: The Pareto-optimal accuracy–simplicity front consisted of Operon, GP-GOMEA, and DSR.These methods jointly provided the best trade-offs between accuracy and model simplicity across black-box problems.
- Ground-truth regression: AIFeynman recovered exact solutions on 53% of noiseless ground-truth problems, versus 27% for GP-GOMEA.At noise levels above 0.01, DSR, gplearn, AFP_FE, and AFP recovered exact solutions more often than AIFeynman.
- Ground-truth regression: AIFeynman performed significantly better than other methods on low-noise Feynman problems but was not significantly different from other methods on Strogatz datasets.The results also report that Operon struggled to recover exact solutions despite finding candidate models with near-perfect test-set scores.
6 Discussion and Conclusions
The paper presents SRBench as an open framework for objective comparison and concludes that GP-based methods perform strongly on real-world and black-box tasks, while exact recovery depends on noise. The authors identify dataset, design, analysis, coverage, and parallelism improvements for future revisions.
- Conclusions: The benchmark enables objective comparisons of contemporary symbolic regression methods across diverse regression problems.The framework is intended to support broad, controlled evaluation rather than isolated method comparisons.
- Conclusions: GP-based methods such as Operon outperform newer SR methods from other optimization fields on real-world and black-box tasks while producing simpler models than ensemble methods.On ground-truth physics and dynamical-systems problems, AIFeynman performs best with minimal noise, whereas DSR and GP-based methods perform best otherwise.
- Limitations and future work: SRBench does not yet benchmark parallel execution, include all promising methods, or include real-world data from phenomena with known first-principles models.The authors propose improving dataset curation, experiment design, and analysis in future revisions.
Checklist
The checklist reports that the study provides reproducibility materials, training details, uncertainty reporting, compute information, asset documentation, and ethics-related disclosures. It also points readers to the discussion for limitations.
- Claims and limitations: The paper reports that its specific claims are supported by statistical tests and that limitations are described in the discussion and conclusions.The checklist marks both items as satisfied.
- Ethics: The paper states that it follows ethics guidelines, discusses ethics in Section A.4, and addresses potential negative societal impacts.These checklist items are marked as satisfied.
- Reproducibility: Experimental results and a reproduction guide are available through the SRBench repository.The checklist directs readers to the project URL for the relevant materials.
- Experimental reporting: Training details, including data splits and hyperparameters, are reported in Table 2.The checklist marks this item as satisfied.
- Experimental reporting: Error bars from repeated experiments are reported in Figures 1–3, and compute resources are documented in the appendix.Both checklist items are marked as satisfied.
- Assets and data: The paper documents citations and licenses for existing assets and releases new datasets under an MIT license.The checklist also states that the datasets contain no personally identifiable information.
A Appendix
The appendix directs readers to the SRBench repository for the most up-to-date reproduction guide.
- Reproduction guide: The most up-to-date SRBench reproduction guide is available in the project repository.The appendix provides the repository URL for access.
A.1 Running the Benchmark
The benchmark can be launched through analyze.py with configurable datasets, trial counts, result paths, time limits, and execution environments. Separate commands cover black-box and ground-truth Strogatz experiments.
- Running the Benchmark: The benchmark experiments are launched from the experiments/ folder via analyze.py.The script supports parallel execution locally or on LSF and SLURM schedulers.
- Black-box experiment: The black-box experiment command specifies a PMLB dataset path, 10 trials, a results directory, and a 48-hour time limit.The command is python analyze.py /path/to/pmlb/datasets -n_trials 10 -results ../results -time_limit 48:00.
- Ground-truth experiment: The ground-truth Strogatz experiment command sets target noise to 0.0, selects Strogatz datasets, uses 10 trials, and applies tuning with a 9-minute limit.The command also writes results to ../results_sym_data and enables symbolic-data evaluation.
A.2 Contributing a Method
The benchmark supports method contributions through a living contribution guide and automated testing of submitted code and installation procedures.
- Contribution workflow: A contributor can submit a method by defining its estimator, hyperparameters, complexity function, and symbolic-model function.Contributors may add a package to the conda/pip environment or provide an install script.
- Contribution workflow: Pull requests automatically test new methods and installation procedures on a minimal benchmark before maintainer approval.Approved contributions become part of the benchmark's living method collection.
A.3 Additional Background and Motivation
The background discusses closed-source commercial SR systems, GP improvements for constant optimization, and the need to benchmark open implementations under controlled conditions.
- Commercial SR systems: Eureqa is a commercial GP-based SR system whose closed-source status prevents controlled benchmarking of key experimental variables.The constrained variables include evaluations, search space, time limits, and population size.
- Commercial SR systems: Eureqa's reported algorithmic ideas include directed acyclic graph representations, age-fitness Pareto optimization, and co-evolution of fitness predictors.The study instead benchmarks open-source AFP and AFP_FE implementations because Eureqa's current algorithms cannot be guaranteed.
- Method contributions: Figure 4 illustrates a method contribution by showing the estimator interface and functions needed to expose hyperparameters, complexity, and the symbolic model.This operational example connects the contribution process to the benchmark's required method outputs.
- Constant optimization: Local-search methods improve GP by treating constant optimization separately from evolutionary learning.The passage motivates this distinction by noting that gradient-based optimization has been relatively under-used in GP.
A.4 Additional Dataset Information
The supplementary material documents dataset access and scale, ground-truth problem construction, experimental settings, subgroup behavior, and statistical comparisons across benchmark results.
- Dataset access: PMLB datasets and metadata are distributed through Git Large File Storage under an MIT license with planned long-term maintenance.The repository also records responsibility for rights violations.
- Experimental settings: White Gaussian target noise is added as a fraction of the signal root mean square value.Ground-truth experiments receive additional simplification time, and tuned settings use updated operators and doubled evaluation budgets.
- Subgroup analysis: Friedman datasets distinguish top-ranked methods more strongly than the remaining PMLB problems, while kernel ridge regression reverses its relative performance between these groups.The authors recommend expanding PMLB and using subgroup analysis to reduce source-specific effects.
- Method and dataset groupings: Bi-clustering reveals similar behavior for AFP and AFP_FE, and for FEAT and EPLEX, while performance varies substantially across dataset groups.Operon performs strongly on some datasets but mediocrely on a subset where linear models also perform well.
- Ground-truth analysis: Operon performs well on black-box regression but finds few symbolically equivalent models, while structural-assumption methods perform worse on unconstrained symbolic recovery.Accuracy-based solution rates align more closely with black-box rankings than symbolic equivalence does.
- Statistical comparisons: Figures 9–11 use Wilcoxon signed-rank tests with Bonferroni correction to compare estimator performance across black-box and ground-truth tasks.The figures cover R2 scores, model size, and solution rates under target noise levels of 0 and 0.01.