Source-linked AI summary
Where are we now? A large benchmark study of recent symbolic regression methods
Patryk Orzechowski, William La Cava, Jason H. Moore
TL;DR
The paper examines how recent genetic-programming methods for symbolic regression compare with established machine-learning regression approaches, addressing inconsistent and limited benchmarking. It evaluates four GPSR methods and ten ML methods across 94 real-world problems, finding strong GPSR performance alongside substantially higher computational demands.
Problem
Inconsistent benchmark datasets and limited comparisons make GPSR performance relative to the broader machine-learning field difficult to assess.
Method
The study benchmarks four recent GPSR algorithms and ten established ML regression methods on 94 real-world problems using cross-validation and training error, test error, and wall-clock time.
Results
ϵ-lexicase selection in ellyn, given 1 million fitness evaluations, achieves the best test set MSE ranking across all datasets and methods, while EPLEX and MRGP produce competitive results.
Takeaways & Limitations
GPSR can perform competitively with state-of-the-art ML regression approaches, supporting its relevance in broader machine-learning comparisons.
Takeaways & Limitations
GP-based methods have much higher runtimes when run on a single thread because of their computation complexity.
Abstract
from arXiv · showhide
In this paper we provide a broad benchmarking of recent genetic programming approaches to symbolic regression in the context of state of the art machine learning approaches. We use a set of nearly 100 regression benchmark problems culled from open source repositories across the web. We conduct a rigorous benchmarking of four recent symbolic regression approaches as well as nine machine learning approaches from scikit-learn. The results suggest that symbolic regression performs strongly compared to state-of-the-art gradient boosting algorithms, although in terms of running times is among the slowest of the available methodologies. We discuss the results in detail and point to future research directions that may allow symbolic regression to gain wider adoption in the machine learning community.
1 INTRODUCTION
The paper addresses inconsistent and narrow benchmarking in genetic-programming symbolic regression by evaluating recent methods within the broader machine-learning regression context. It benchmarks four symbolic-regression algorithms and ten machine-learning approaches across 94 real-world problems using tuned models and multiple evaluation criteria.
- Motivation: Symbolic regression lacks standardized benchmark datasets and algorithms, making comparisons inconsistent and difficult to assess beyond individual experiments.Researchers often choose their own comparison methods and benchmark problems, while single hyperparameter settings increase uncertainty.
- Caveat: Benchmarking has recognized drawbacks because excessive focus on benchmark performance may stifle innovation or reduce generalization to new tasks.The paper presents this as a limitation of benchmark-centered evaluation while arguing that GP research is not yet overly benchmark-focused.
- Motivation: This benchmarking gap makes it difficult to understand how GPSR methods fit within machine learning or which GP advances meaningfully affect results.The paper links limited benchmarking to uncertainty about GPSR performance relative to broader ML and the impact of methodological advances.
- Related work: Existing broad benchmarking is more common for machine-learning classification than for GP-based regression, leaving GPSR comparatively underrepresented.Prior work included five symbolic-regression methods on nine datasets, whereas ML studies examined many classification methodologies and datasets.
- Study design: 94 real-world regression problems are used to compare four recent GPSR algorithms with ten established ML approaches.The study tunes hyperparameters through cross-validation and evaluates training error, test error, and wall-clock time.
- Study design: The study provides analysis code so researchers can benchmark additional methods within the same framework and reproduce the reported results.The code is intended to support both extension of the benchmark and reproduction of the analysis.
2 METHODS
The study benchmarks four recent genetic-programming symbolic-regression methods alongside established machine-learning regressors, using standardized datasets and method-specific implementations. The methods span multiple GP innovations and scikit-learn regression families, with experiments designed around parameter grids and dataset filtering.
- Methods compared: Four recent GPSR methods are compared with ten well-established machine-learning regression methods.
- GP methods: The GP methods encompass constant optimization, semantic search, Pareto optimization, Lasso-based weighting, and alternative selection schemes.
- GP methods: EPLEX selects parents by filtering individuals across randomized training-sample orderings rather than aggregating performance into one fitness score.
- GP methods: Ellyn combines point mutation, subtree crossover, per-generation stochastic hill-climbing weight training, and a Pareto archive balancing mean squared error and complexity.
- GP methods: GSGP uses semantic variation operators to place offspring between parent and target semantics, producing a unimodal fitness landscape but exponentially growing programs.
- Datasets and experiment design: The benchmark draws regression datasets from PMLB, excludes datasets with at least 3000 instances and two infeasible cases, and evaluates preset parameter grids.
3 RESULTS
Across 94 regression datasets, gradient boosting led training performance, while EPLEX-1M achieved the best average testing performance and competitive GP results. GP methods incurred substantially higher computation times, and several methods showed overfitting between training and testing rankings.
- Aggregate benchmark: 94 regression datasets were aggregated to compare the benchmarked algorithms.The results are presented using rankings and statistical comparisons across the dataset collection.
- Training performance: Gradient boosting achieved the best training performance, ranking in the top two for most benchmark datasets.
- Testing performance: EPLEX-1M was the best average generalizing method, slightly outperforming XGBoost, with gradient boosting third and MLP fourth.Testing results assess performance on previously unseen data.
- Generalization and overfitting: Gradient boosting moved from first to third place between training and testing, while MRGP fell from fourth to sixth, indicating overfitting.GSGP showed the largest rank decline, from eighth to thirteenth; EPLEX-1M and Lasso improved their median rankings.
- Statistical comparisons: EPLEX-1M statistically outperformed the most other methods, with 11 significant comparisons versus 9 for XGBoost and 7 for gradient boosting.No significant differences were found among EPLEX-1M, XGBoost, gradient boosting, and MLP.
- Running time: GP methods were one to three orders of magnitude slower than the nearest comparison, although EPLEX-1M completed ten times as many fitness evaluations in approximately the same time.Running-time comparisons were affected by cluster execution, single-thread runs, and different implementation languages, so they do not exclusively reflect method complexity.
4 CONCLUSIONS
The study benchmarks four recent GPSR methods against ten state-of-the-art machine-learning methods on 94 real-world regression datasets, finding competitive predictive performance but substantially higher runtimes for GP-based approaches. It also identifies scope limitations involving model complexity, dataset selection and size, and aggregate rather than dataset-specific analysis.
- Study scope and design: Four recent GPSR methods were evaluated against ten state-of-the-art ML methods on 94 real-world regression datasets, using nested cross-validation and training, testing, and runtime comparisons.The benchmark measures MSE on training and testing sets alongside runtime.
- Main findings: An ϵ-lexicase method implemented in ellyn achieved the best test-set MSE ranking across all datasets when given 1 million fitness evaluations.
- Main findings: EPLEX and MRGP produced competitive results compared with state-of-the-art ML regression approaches.
- Main findings: GP-based methods had much higher runtimes on a single thread because of their computational complexity, while parallelism may improve their running-time competitiveness.
- Limitations and future work: The analysis did not quantify symbolic-model complexity, despite explainability being a motivation for GPSR, and future work could standardize its assessment.The paper notes that the value of explainability versus predictive power depends on the application domain.
- Limitations and future work: The study used real-world datasets of up to 1000 instances and aggregated performance across datasets, motivating simulation studies, larger datasets, and analysis of dataset properties.Simulation studies could provide ground truth and scalable control of problem complexity or difficulty.