PaperlayerView exact revision on arXiv

Source-linked AI summary

A Bayesian machine scientist to aid in the solution of challenging scientific problems

Roger Guimera, Ignasi Reichardt, Antoni Aguilar-Mogas, Francesco A Massucci, Manuel Miranda, Jordi Pallares, Marta Sales-Pardo

arXiv:2004.12157v1cs.LGphysics.data-anstat.ML

TL;DR

Existing methods seek interpretable closed-form models but face challenges balancing fit and complexity and exploring broad expression spaces. This paper introduces a Bayesian machine scientist that learns expression priors from a corpus, approximates model posteriors, and explores them with MCMC. It recovers generating models on synthetic data and uncovers accurate models for real systems, while posterior averaging can make predictions harder to interpret and MCMC may need greater efficiency for very large datasets.

  • Problem

    Machine scientists need to automatically identify interpretable closed-form models while balancing goodness of fit against complexity and exploring arbitrarily large expression spaces.

  • Method

    The Bayesian machine scientist combines an approximated marginal posterior with a corpus-informed prior over expressions and explores the posterior using MCMC.

  • Results

    The approach recovers true generating models from synthetic data, uncovers accurate closed-form models for real systems, and provides more accurate out-of-sample predictions than existing approaches and nonparametric methods.

  • Takeaways & Limitations

    The approach supports evaluating and comparing plausible closed-form models, including alternative explanations and conflicting theories, within the sampled expression space.

  • Takeaways & Limitations

    Posterior averaging can make predictions harder to interpret, and MCMC may require more efficient approaches for very large datasets.

Abstract

from arXiv · show

Closed-form, interpretable mathematical models have been instrumental for advancing our understanding of the world; with the data revolution, we may now be in a position to uncover new such models for many systems from physics to the social sciences. However, to deal with increasing amounts of data, we need "machine scientists" that are able to extract these models automatically from data. Here, we introduce a Bayesian machine scientist, which establishes the plausibility of models using explicit approximations to the exact marginal posterior over models and establishes its prior expectations about models by learning from a large empirical corpus of mathematical expressions. It explores the space of models using Markov chain Monte Carlo. We show that this approach uncovers accurate models for synthetic and real data and provides out-of-sample predictions that are more accurate than those of existing approaches and of other nonparametric methods.

INTRODUCTION

The paper frames machine scientists as algorithms for automatically discovering interpretable closed-form models while addressing model-selection and search challenges. It proposes a Bayesian approach that combines data fit with expression priors and uses explicit approximations to assess model plausibility.

  • Interpretable closed-form models have advanced scientific understanding and may now be discoverable for natural and socioeconomic systems using data.
  • Machine scientists are algorithms intended to identify such models automatically and assist human scientists.
  • Existing approaches include genetic programming and sparse regression, but they face difficulties generalizing complexity measures and selection criteria.
  • The search space can be arbitrarily large, while genetic programming lacks guarantees of frequent best-model exploration and sparse regression can exclude valid models.
  • The proposed Bayesian machine scientist combines goodness of fit with a prior over expressions that accounts for model complexity and samples expressions using MCMC.
  • Model plausibility uses an approximated description length, with BIC providing a first-order approximation when the likelihood is peaked near maximum-likelihood parameters and the prior is smooth there.

Sampling from the posterior distribution over expressions

The Bayesian machine scientist uses MCMC to explore the posterior over closed-form expressions and can use the resulting samples for model selection and broader predictive characterization.

  • MCMC uses three move types that can connect any closed-form expression to any other, enabling exploration of the whole expression space given enough time.
  • MCMC samples expressions from the stationary posterior distribution p(f_i|D), regardless of move frequencies and other Markov-chain parameters.
  • The most plausible expression from an MCMC run serves as the maximum a posteriori, or minimum description length, model.
  • Sampling the whole model space generates expression sequences that provide a more complete characterization and higher out-of-sample prediction accuracy.

Estimation of prior probabilities using a corpus of mathematical expressions

The Bayesian machine scientist learns prior expectations for mathematical expressions from an empirical corpus, helping control structural overfitting while exploring candidate models. Across synthetic tests, it recovers true or near-equivalent models and adapts model complexity to data quality.

  • Motivation: A uniform prior is unsuitable for a very large expression space because structurally complex models can fit finite data arbitrarily well and generalize poorly.The resulting structural overfitting is distinct from, but analogous to, traditional parameter overfitting.
  • Corpus-based prior: The prior is learned from 4080 mathematical expressions extracted from Wikipedia to reflect the empirical plausibility of operations and expression structures.The corpus is used to shape the machine scientist’s prior expectations through a maximum-entropy approach.
  • MCMC exploration: MCMC explores expression trees using three moves—root addition or removal, elementary-tree replacement, and node replacement—to sample candidate expressions.The moves are accepted or rejected with the Metropolis rule; the figure describes expression trees as operations at internal nodes and variables or parameters at leaves.
  • Synthetic validation: With 400 noisy synthetic data points, the Bayesian machine scientist identifies the correct model, whereas benchmark methods fail to recover it and tend to structurally overfit.The synthetic target expression was not included in the empirical corpus.
  • Synthetic validation: With as few as 100 points, the Bayesian machine scientist recovers the true model in the synthetic dataset, despite competing approaches failing with 400 points.The authors note that a minimum amount of data is required for correct-model identification.
  • Noise robustness: For noisy Rössler-system data, the machine scientist recovers the correct differential equations at moderate noise and selects simplified near-equivalent models at high noise.The method automatically adjusts model complexity to data quality; the selected models are almost indistinguishable from the true ones.

Use of the machine scientist on small datasets and on the Nikuradse dataset

On small and noisy datasets, the Bayesian machine scientist identifies plausible closed-form models and supports ensemble-based prediction. Applied to turbulent friction data, it finds simple scaling relationships and more accurate out-of-sample predictions than alternative approaches.

  • Small datasets: Across three small real datasets, the machine scientist identifies at least one model Pareto-better than existing models in plausibility, BIC, and cross-validation error.The identified model is better in at least one metric without being worse in either of the others.
  • Model uncertainty: The machine scientist finds a collection of similarly plausible models rather than one overwhelmingly dominant candidate, motivating posterior averaging for prediction.The posterior predictive distribution can be approximated using the sampled models.
  • Model uncertainty: Posterior averaging improves predictive coverage but makes interpretation harder because averaging interpretable models produces a noninterpretable effective model.The paper nevertheless describes model averaging as the most comprehensive predictive approach.
  • Nikuradse dataset: For the Nikuradse dataset, its predictions are significantly more accurate than those of alternative machine scientists and Gaussian processes.The comparison evaluates generalization to previously unseen data.
  • Nikuradse dataset: For turbulent friction, the median predictive model captures the expected limiting scaling despite most observations lying in a different physical regime.Many sampled models do not scale correctly, making the result notable within the paper’s analysis.
  • Nikuradse dataset: The inferred universal function of roughness Reynolds number provides an excellent fit to the original unscaled data.The proposed scaled expression is 1.73 + 0.64 − 0.96k⁺_s − 2.62 + 0.64k⁺_s.

DISCUSSION

The paper frames closed-form model discovery as requiring both a principled fit–complexity balance and systematic exploration of model space. Its Bayesian machine scientist makes assumptions explicit, performs well on observed and new data, and supports broader questions about model discoverability and predictability.

  • Closed-form model discovery repeatedly faces two difficulties: balancing goodness of fit with complexity and searching systematically through model space.
  • For the Nikuradse dataset, the Bayesian machine scientist produced significantly more accurate predictions for unseen data than the alternative approaches tested.
  • No approach can completely avoid these difficulties because model complexity cannot be made assumption-free and restricting the search space can exclude valid or true models.
  • The Bayesian approach makes its assumptions and approximations explicit, using prior expectations to represent model complexity and MCMC to explore model space.
  • The approach may require more efficient sampling for very large datasets and further investigation of alternative priors for expressions.
  • The results suggest that the approach can describe observed data and predict new data, while also supporting comparisons among phenomenological models and conflicting theories.
  • Important open questions concern whether correct models are discoverable under given conditions and how the predictability and expressiveness of interpretable models compare with deep neural networks.

MATERIALS AND METHODS

The machine scientist represents mathematical expressions as bounded expression trees and samples them with MCMC. Its moves modify nodes, replace elementary subtrees, or add roots to explore candidate models.

  • Closed-form expressions are represented as expression trees whose internal nodes are operations and whose leaves are variables or parameters.
  • Expression-tree size is limited to 50 nodes to avoid problems with improper priors.
  • An elementary tree contains at most one operation, and MCMC moves add, remove, replace, or modify these elementary structures.
  • Node replacement substitutes an operation while preserving its number of offspring and leaving the offspring branches unchanged.
  • Root addition adds an operation above the existing expression tree, using randomly selected zero-operation offspring when needed.
  • Elementary-tree replacement supplies small expression variations, whereas node replacement makes larger jumps and root replacement changes tree size or adds terms.

MCMC acceptance rules

Each proposed MCMC expression move is accepted or rejected using Metropolis’ rule so that the chain targets the posterior over expressions. The acceptance ratio accounts for description length and, when needed, asymmetric proposal probabilities.

  • At each MCMC step, one expression-tree move is proposed and accepted or rejected according to Metropolis’ rule.
  • The acceptance rule ensures that the stationary distribution is the posterior p(fi∣D).
  • The move frequencies were 5% root replacement, 45% node replacement, and 50% elementary-tree replacement; these rates affect convergence speed but not equilibrium distribution.
  • For symmetric node replacement, acceptance depends on the difference in description lengths between the proposed and current expression trees.
  • Root addition and root removal use asymmetric proposal probabilities, introducing a factor based on the number of possible roots Nroot.
  • Elementary-tree replacement accounts for the numbers of selectable initial and final elementary trees and their proposal probabilities in its acceptance ratio.
  • A controlled validation showed that the implemented moves and acceptance rules recover the expected equilibrium distribution p(fi∣D).

Avoidance of expression duplicates in MCMC

The implementation reduces expression trees to a canonical form before counting visited models, preventing equivalent mathematical expressions from being overcounted during MCMC.

  • Multiple expression trees can encode the same mathematical expression, so the implementation reduces visited trees to a canonical form using Sympy.

Parallel tempering MCMC

Parallel tempering helps the Bayesian machine scientist explore a rugged expression space by sampling across computational temperatures. The target posterior is sampled at T=1, while higher temperatures connect the posterior to the data-independent prior.

  • Parallel tempering MCMC: Parallel tempering addresses local trapping caused by large description-length differences between neighboring expressions.The method maintains replicas at logarithmically spaced, increasing temperatures.
  • Parallel tempering MCMC: The computational temperature T defines distributions ranging from the posterior at T=1 to the prior as T approaches infinity.At T=1, sampling targets p(fi|D); at infinite temperature, the distribution is p(fi), independent of data.
  • Parallel tempering MCMC: The reported results used 40 temperatures defined by T_k = 1.05^k for k = {0,1, …,39}.Expressions were taken from the T0=1 sample, with four additional restarts for the Nikuradse dataset.

Parsing of Wikipedia expressions for expression priors

The expression prior is built from a corpus of named scientific equations, parsed into symbolic form and statistically modeled to reproduce corpus properties. A maximum-entropy graph-based prior is then fitted through iterative MCMC matching of operation statistics.

  • Corpus construction: The corpus contains 4080 mathematical expressions from Wikipedia pages listing scientific equations named after people.These entries were selected as likely phenomenological models rather than mathematical identities or theorem proofs.
  • Parsing and validation: An algorithm parses ambiguous Wikipedia expressions into Sympy using heuristics, with manual checks finding fewer than 5% errors across three subsets of 200 expressions.The verification accepted mathematically correct parsings that might be contextually imperfect.
  • Expression prior: The prior represents mathematical expressions as graphs and uses exponential random graph methods to match statistical properties of the empirical corpus.The target properties include operation frequencies and their squared counts per expression.
  • Expression prior: The prior is chosen to be maximally uninformative while generating expressions with corpus-matched average properties.Its operation-specific hyperparameters are fitted so generated expressions reproduce the selected corpus statistics.
  • Hyperparameter fitting: Hyperparameter fitting iteratively generates millions of expressions, measures their operation statistics, and updates parameters until changes become sufficiently small.The procedure begins with target averages, initializes parameters, and repeatedly compares measured and target values.

Benchmark machine scientists

The Bayesian machine scientist was benchmarked against Eureqa, EPLEX, and EFS, representing genetic-algorithm, genetic-programming, and sparse-regression approaches. The comparisons used each method’s stated or study-specific operating conditions.

  • Eureqa: Eureqa searches expression space with a genetic algorithm using default operation penalties, basic operations matched to the Bayesian machine scientist, and its default fitness function.Each run lasted several weeks.
  • EPLEX: EPLEX uses genetic programming and was evaluated with a population of 1000, 1000 generations, 10 repetitions, and selection of the best-fitness realization.These conditions matched a recent systematic comparison.
  • EFS: EFS uses a genetic algorithm to generate basis functions automatically for sparse regression without requiring basis functions in advance.The method is described as fast, producing expressions within seconds.

Gaussian process models

Gaussian process models were used as nonparametric comparators for out-of-sample predictions. The evaluation tested multiple kernel families and both the raw feature D/r and its logarithm.

  • Gaussian process models: The Gaussian process used the scikit-learn implementation and was evaluated with several kernel choices.Tested options included radial-basis-function, Matérn, white, rational-quadratic, exponential, linear, and quadratic-combination kernels.
  • Gaussian process models: The comparison also tested whether Gaussian-process performance changed when using D/r or log(D/r) as the feature.The supplied passage frames this as part of the out-of-sample prediction comparison.
  • Supplementary evaluation: The supplementary materials include separate sections on BIC calculation, synthetic-data benchmarking, the Rössler system, Bessel-function recovery, small datasets, and the Nikuradse dataset.These materials document additional validation and benchmarking procedures.
Loading 2004.12157v1…