PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl

Miles Cranmer

arXiv:2305.01582v3astro-ph.IMcs.LGcs.NEcs.SCphysics.data-an

TL;DR

Scientific symbolic regression aims to recover interpretable empirical equations from data, but practical scientific settings pose requirements that many existing algorithms do not address. This paper presents PySR, an open-source symbolic-regression package using a multi-population evolutionary search with an evolve-simplify-optimize loop. The authors report that classic evolutionary and other untrained symbolic methods outperform pure deep-learning approaches on recovering historical empirical equations from original and synthetic datasets.

  • Problem

    Many symbolic-regression algorithms lack features needed for scientific equation discovery, where empirical equations must address unknown constants, noisy data, interpretability, constraints, and diverse data.

    Source: §1 · p. 2

  • Method

    PySR is an open-source symbolic-regression package built around a multi-population evolutionary algorithm and an evolve-simplify-optimize loop that optimizes equation constants.

    Source: §1 · p. 3, §2.1 · pp. 3, 6

  • Results

    Classic evolutionary and other untrained symbolic methods outperform pure deep-learning approaches in recovering historical empirical equations from original and synthetic datasets.

    Source: §4 · pp. 17–18, §3.3 · p. 17

  • Takeaways & Limitations

    PySR is designed to accelerate discovery of interpretable symbolic models from data and to make symbolic regression more accessible in science.

    Source: §4 · p. 17

    EmpiricalBench includes unstable runs in its primary score, but the authors note their greater familiarity with PySR may give it a stability advantage over less familiar packages.

    Source: §3.2.1 · p. 17

Abstract

from arXiv · show

PySR is an open-source library for practical symbolic regression, a type of machine learning which aims to discover human-interpretable symbolic models. PySR was developed to democratize and popularize symbolic regression for the sciences, and is built on a high-performance distributed back-end, a flexible search algorithm, and interfaces with several deep learning packages. PySR's internal search algorithm is a multi-population evolutionary algorithm, which consists of a unique evolve-simplify-optimize loop, designed for optimization of unknown scalar constants in newly-discovered empirical expressions. PySR's backend is the extremely optimized Julia library SymbolicRegression.jl, which can be used directly from Julia. It is capable of fusing user-defined operators into SIMD kernels at runtime, performing automatic differentiation, and distributing populations of expressions to thousands of cores across a cluster. In describing this software, we also introduce a new benchmark, "EmpiricalBench," to quantify the applicability of symbolic regression algorithms in science. This benchmark measures recovery of historical empirical equations from original and synthetic datasets.

1 Introduction

Scientific symbolic regression must discover interpretable empirical equations from real data while addressing challenges that synthetic benchmarks may not capture. The paper introduces PySR as an open-source package designed to meet those scientific requirements.

1 Introduction

  • Symbolic regression searches analytic expressions for accurate, interpretable models, automating a process scientists have historically pursued through intuition and trial and error.Empirical expressions found this way can lead to theoretical insights, as with Kepler and Planck.Source: §1 · pp. 1–2
  • Early tools found simple laws from idealized data, while later tools such as Eureqa made symbolic regression more practical on noisy, real-world scientific data.The field’s development included heuristic searches, genetic algorithms, and user-friendly tools.Source: §1 · p. 2
  • Recent symbolic-regression work spans tree-based genetic algorithms, neural-network interpretation, high-dimensional data methods, alternative expression representations, and constrained or greedy searches.Other approaches include SINDy’s linear combinations of fixed nonlinear expressions and methods combined with deep learning.Source: §1 · p. 2
  • Scientific SR must handle unknown real constants, noisy data, accuracy–simplicity trade-offs, known constraints, custom operators, and diverse data types.The authors argue existing algorithms often lack features needed for practical equation discovery in science.Source: §1 · pp. 2–3
  • PySR is an open-source Python and Julia package designed to satisfy the stated requirements for discovering empirical equations in science.The authors developed it to support equation discovery in their own field.Source: §1 · p. 3

2 Methods

This section describes PySR’s search algorithm and implementation, including its multi-population evolution and evolve-simplify-optimize loop. It also covers search constraints, the Julia backend, Python-facing exports, and configurable domain-specific operators, objectives, and losses.

2.1 Algorithm

  • PySR runs asynchronous, multi-population evolutionary search using tournament selection and mutations or crossovers to generate expressions.Evolutionary algorithms for symbolic regression were popularized in.Source: §2.1 · p. 3
  • Replacing the eldest population member, rather than the least-fit one, uses age-regularized evolution to slow premature specialization and avoid local minima.PySR also allows tournament-selection probability p to be less than 1; related regularizations in symbolic regression are noted in.Source: §2.1 · p. 4
  • Simulated annealing varies mutation acceptance and tournament-selection probability with temperature, alternating between population diversity and focus on fitter expressions.The paper reports that simulated annealing can significantly speed up search; α controls the temperature scale, with α → ∞ equivalent to regular tournament selection and α → 0 rejecting all fitness-lowering mutations.Source: §2.1 · pp. 4–5
  • PySR repeatedly evolves expressions, simplifies them occasionally, and optimizes their constants, with BFGS as the default optimizer.Occasional simplification preserves redundant intermediate expressions that may enable later mutations, while constant optimization improves discovery of equations with real constants.Source: §2.1 · pp. 5–6
  • PySR makes expression complexity user-configurable and adaptively penalizes complexity based on how frequently expressions of that size occur, encouraging exploration across simpler and more complex forms.By default, complexity is the number of expression-tree nodes; the adaptive penalty uses a moving-window count and is intended to reduce premature specialization.Source: §2.1 · p. 6
  • PySR supports user-defined unary and binary operators, arbitrary regression losses, and options for weighting data, selecting features, and constraining expressions.Custom losses can define likelihood, classification, and implicit objectives; feature selection uses gradient-boosting trees, and constraints are checked at each mutation.Source: §2.1 · pp. 7–8
  • Operator-specific argument-size and nesting constraints can restrict expression structure and reduce the complexity of discovered expressions.For example, a constraint can cap an operator’s exponent size while leaving its base unrestricted, and nested constraints can permit one operator combination while disallowing deeper or repeated nesting.Source: §2.1 · p. 9

2.2 Software implementation

  • Julia JIT compilation fuses operator combinations into SIMD-enabled kernels, speeding up expression evaluation, including for user-defined operators.PySR’s search backend is written in pure Julia, and compilation can fuse combinations up to two operators deep.Source: §2.2 · p. 9
  • PySR’s asynchronous population search uses a permanent global hall of fame for migration, and reintroducing its members can significantly speed up search.Populations evolve independently on threads or processes; the head worker records best expressions by complexity, then coordinates migration between populations and from the hall of fame.Source: §2.2 · p. 10

2.3 Interfaces

  • PySR exports expressions to SymPy, NumPy, PyTorch, and JAX, with trainable constants in PyTorch and JAX for re-optimization.Custom operators also require equivalent definitions in the target export frameworks.Source: §2.3 · p. 10, §2.4 · p. 11

2.4 Custom Operators

  • PySR permits field-specific unary or binary scalar functions as operators and compiles and vectorizes them, generating SIMD instructions when possible.The library has no fixed built-in operator set; user-defined operators must also be implemented in SymPy and, when exporting, in JAX or PyTorch.Source: §2.4 · p. 11

2.5 Custom Losses

  • PySR lets users define custom, automatically vectorized loss functions, enabling problem-specific non-Gaussian likelihoods and weighted losses.A loss can use branching logic; the passage gives absolute error and absolute error multiplied by a weight as examples.Source: §2.5 · p. 11

3 Evaluation

The evaluation compares symbolic-regression tools and introduces EmpiricalBench to test recovery of empirically discovered equations. Its discussion emphasizes that benchmark performance depends on noisy, sometimes biased datasets and that traditional heuristic algorithms recovered more expressions than the tested deep-learning approaches.

3.1 High Level Comparison

  • The evaluation compares tools using desirable features, including expressivity levels ranging from fixed expression bases to scalar, vector, or tensor expression trees.Table 1 notes that many missing features are software limitations; custom losses and operators are often not easily configurable through an API.Source: §3.1 · pp. 12–13, Table 1 · p. 13

3.2 EmpiricalBench: Empirical Science Symbolic Regression Benchmark

  • EmpiricalBench addresses gaps in synthetic benchmarks, including supplied physical constants and limited coverage of real-world tasks and noise conditions.The evaluation adapts SRBench to examine Pareto fronts or lists of candidate expressions rather than only one selected expression per method.Source: §3.2 · p. 14, §3.2.1 · p. 14
  • EmpiricalBench comprises real empirical equations first formulated from experimental data, with observables as variables and physical constants left for algorithms to discover.Original data were used where available; otherwise, data were generated from equations using realistic variable ranges and added noise.Source: §3.2 · p. 14

3.3 Discussion

  • EmpiricalBench results depend on recovering equations from noisy, sometimes biased datasets; PySR, Operon, DSR, and QLattice outperformed EQL and SR-Transformer.Figure 5 visualizes the benchmark data, which draws on original data where available and otherwise uses generated data with realistic variable ranges and noise.Source: Figure 5 · p. 15, §3.3 · p. 17

3.2.1 Results on EmpiricalBench

  • Across nine equations and six algorithms, the evaluation runs five trials per equation and algorithm, examines each Pareto front, and classifies recovery as correct, almost, failed, or incorrect.The algorithms were given the full dataset for training, and the true expression was checked for inclusion in each Pareto front.Source: §3.2.1 · pp. 15, 17
  • The benchmark includes unstable runs in its primary score, but does not measure evaluation counts or expression numerical accuracy.The authors note that PySR may have been run in a more stable environment than unfamiliar packages.Source: §3.2.1 · p. 17

4 Conclusion

PySR combines evolutionary search, a simplify-optimize loop, distributed computing, and deep-learning integrations to support practical symbolic regression, alongside the EmpiricalBench benchmark for scientific applications. On historical empirical equations, classic evolutionary and other untrained symbolic methods outperform pure deep-learning approaches; the authors also point to successful applications across fields and potential in hybrid methods.

4 Conclusion

  • PySR is an open-source symbolic regression library combining multi-population evolutionary search, an evolve-simplify-optimize loop, a distributed backend, and deep-learning integrations.The authors present it as a tool for accelerating discovery of interpretable symbolic models from data.Source: §4 · p. 17
  • EmpiricalBench provides a quantitative means to evaluate symbolic regression algorithms in scientific applications.The benchmark measures recovery of historical empirical equations from original and synthetic datasets.Source: §4 · p. 17
  • Classic evolutionary and other untrained symbolic methods, including PySR, Operon, DSR, and QLattice, outperform pure deep-learning approaches EQL and SR-Transformer on historical empirical equations from original and synthetic datasets.The authors connect deep-learning challenges to real-world data biases and heteroscedasticity, and to the nonlinearity of expression spaces.Source: §4 · pp. 17–18
  • Deep-learning methods, including generative models, reinforcement learning, and symbolic distillation, may improve symbolic regression, motivating further hybrid-method exploration.The authors present this as potential rather than a demonstrated result.Source: §4 · p. 18
  • Reported PySR applications span cloud formation, electron-transfer rules, astrophysical relations, international trade, neural differential equations, and gravitational-wave population models.The authors view these applications as validation of a science- and user-focused approach and hope PySR continues as a community tool for scientific discovery.Source: §4 · p. 18

Related papers

Loading 2305.01582v3…