PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Feynman: a Physics-Inspired Method for Symbolic Regression

Silviu-Marian Udrescu, Max Tegmark

arXiv:1905.11481v2physics.comp-phcs.AIcs.LGhep-th

TL;DR

Symbolic regression seeks expressions that match data from unknown functions, while established regression methods cover restricted function forms. AI Feynman combines neural-network fitting with physics-inspired strategies to recursively simplify problems, discovering 100% of Feynman equations and 90% of bonus equations, compared with Eureqa’s 68% and 15%.

  • Problem

    Existing regression methods succeed for restricted cases such as linear combinations of known functions, while more general symbolic regression seeks expressions for unknown functions.

    Source: §I · p. 1

  • Method

    AI Feynman combines neural-network fitting with physics-inspired strategies to discover simplifying properties and recursively divide multidimensional problems into simpler ones.

    Source: A. Overall Algorithm · p. 3, §IV · p. 12

  • Results

    AI Feynman discovered 100% of Feynman equations and 90% of bonus equations, versus Eureqa’s 68% and 15%, respectively.

    Source: A. Key findings · p. 12

  • Takeaways & Limitations

    The paper provides a freely downloadable benchmark of 100 Feynman regression mysteries and 20 additional difficult mysteries for quantitative comparison of symbolic-regression algorithms.

    Source: §IV · p. 12

    The most complex equations requiring neural-network fitting needed more data and less noise.

    Source: A. Key findings · p. 12

Abstract

from arXiv · show

A core challenge for both physics and artificial intellicence (AI) is symbolic regression: finding a symbolic expression that matches data from an unknown function. Although this problem is likely to be NP-hard in principle, functions of practical interest often exhibit symmetries, separability, compositionality and other simplifying properties. In this spirit, we develop a recursive multidimensional symbolic regression algorithm that combines neural network fitting with a suite of physics-inspired techniques. We apply it to 100 equations from the Feynman Lectures on Physics, and it discovers all of them, while previous publicly available software cracks only 71; for a more difficult test set, we improve the state of the art success rate from 15% to 90%.

I. INTRODUCTION · II. METHODS

AI Feynman combines neural-network fitting with physics-inspired strategies to uncover simplifying structure and recursively reduce symbolic-regression problems. Its methods exploit properties common in scientific functions, although why these properties are common remains unsettled.

I. INTRODUCTION

  • Symbolic regression seeks a symbolic expression matching data from an unknown function; Kepler’s discovery that Mars’s orbit is an ellipse is an early example.Source: §I · p. 1
  • Because candidate symbol strings grow exponentially with length, general symbolic regression remains unsolved despite tractable special cases such as linear combinations of known functions.The passage describes exhaustive search by increasing string length as potentially taking longer than the age of the universe.Source: §I · p. 1
  • Prior mathematical-function approaches include sparse regression and genetic algorithms [25, 26]; the paper identifies the genetic algorithm in [27], implemented in Eureqa, as most successful.Source: §I · p. 1
  • Neural networks reveal hidden symmetry and separability, allowing harder regression problems to be recursively divided into simpler problems with fewer variables.This is the paper’s stated central contribution.Source: §I · p. 1

II. METHODS

  • The reasons these simplifying properties are common remain controversial and not fully understood, but the paper argues they can still be discovered and exploited.Source: §II · p. 3
  • Known units enable dimensional analysis, low-degree polynomial structure enables polynomial fitting, and compositionality can support parse-tree search over elementary functions.Dimensional analysis can reduce the number of independent variables; polynomial fitting solves a linear system for coefficients.Source: §II · p. 3
  • Smoothness supports neural-network approximation, while detected symmetry and separability enable transformations into simpler problems with fewer variables or disjoint variable sets.Translational, rotational, or scaling symmetry may remove one or more independent variables; separability partitions variables between two simpler problems.Source: §II · p. 3

A. Overall Algorithm · B. Dimensional Analysis

The algorithm tries property-targeting modules and recursively transforms unsolved problems into simpler pieces. Its dimensional-analysis module uses unit matching to reduce a mystery to dimensionless variables, sometimes a constant.

A. Overall Algorithm

  • The overall algorithm tries a series of modules, each designed to exploit a simplifying property.It tries different strategies in turn, like a human scientist.Source: A. Overall Algorithm · p. 3
  • When it cannot solve a problem directly, the algorithm transforms and divides it into simpler pieces, then recursively re-launches itself on each piece.Source: A. Overall Algorithm · p. 3

B. Dimensional Analysis

  • Matching the units on both sides of an equation can simplify a problem to fewer dimensionless variables, or in the best case, a constant.Source: B. Dimensional Analysis · p. 3
  • The method represents each variable’s units as a five-integer vector and forms M from input-variable vectors and b from the output vector, seeking p such that Mp = b.The five fundamental units are meter, second, kilogram, kelvin, and volt; null-space basis columns U satisfy MU = 0.Source: B. Dimensional Analysis · p. 3
  • The transformed mystery uses dimensionless variables, with their number equal to the null space’s dimensionality; when that number is positive, the method can choose a null-space basis and replace p with p+Ua.These freedoms are used to set as many elements of p and U to zero as possible.Source: B. Dimensional Analysis · p. 3
  • Choosing a basis that makes new variables depend on as few old variables as possible typically yields integer powers, making the final expression easier to find than with fractional or irrational powers.Source: B. Dimensional Analysis · p. 4

C. Polynomial Fit · D. Brute Force

The method tests low-degree polynomial fits directly and uses bounded symbolic enumeration, typically on simpler pieces after decomposition, with controls for search cost, constants, and overfitting.

C. Polynomial Fit

  • A linear-equation solver tests polynomial degrees 0–4 and succeeds when the best fit has r.m.s. error ≤ ε_p.The polynomial module tests whether a mystery can be explained by a low-order polynomial.Source: C. Polynomial Fit · p. 4

D. Brute Force

  • Brute force enumerates expressions in increasing complexity until maximum fitting error falls below ε_p or runtime exceeds t_max.Although it could solve all mysteries in principle, it is typically most useful after other modules transform or break them into simpler pieces.Source: D. Brute Force · p. 4
  • Expressions are generated as syntactically valid symbol strings by length, using reverse Polish notation to avoid parentheses.For example, x + y is represented as “xy+”.Source: D. Brute Force · p. 4
  • Three symbol subsets balance alphabet size against search length, since too many symbols increase branching while too few can make solutions impossible.The subsets used are listed in Table I.Source: Table I · p. 4, D. Brute Force · p. 4
  • Two brute-force variants automatically solve for multiplicative or additive constants, letting the search focus on symbolic form rather than numerical constants.Source: D. Brute Force · p. 4
  • To mitigate discrete-search overfitting, the winner has the smallest description length among functions with ε < ε_b, following.Description length combines string and prediction-error costs, with ε_d = 10^-15 and λ = N_d^1/2; ε_b is ten times validation error for neural-network-generated mysteries and 10^-5 otherwise.Source: D. Brute Force · pp. 4–5

E. Neural-network-based tests & transformations · 1. Neural network training

Neural networks provide the interpolation needed to test for simplifying properties when the data do not cover the required inputs. Across tested equations, the trained networks’ error values ranged from 10^-3frms to 10^-5frms.

E. Neural-network-based tests & transformations

  • Accurate high-dimensional interpolation lets the method evaluate a mystery function at chosen inputs to test for simplifying properties such as translational symmetry.The data may lack examples with the variable relationships needed for such tests.Source: E. Neural-network-based tests &… · p. 5

1. Neural network training

  • Each mystery was fit with a fully connected feed-forward network having six softplus hidden layers, with 128 neurons in the first three and 64 in the last three.Source: §1 · p. 5
  • For each mystery, the authors generated 100,000 data points, using 80% for training and the remainder for validation.Source: §1 · p. 5
  • Training ran for 100 epochs with batch size 2048 and learning rate 0.005, using r.m.s.-error loss, Adam, and weight decay of 10^-2.Source: §1 · p. 5
  • The learning-rate schedule used a 20:1 maximum-to-minimum ratio and reserved 10% of iterations for the final cycle part; momentum used β1 values 0.95 and 0.85, with β2 = 0.99.Schedules followed using FastAI.Source: §1 · p. 5
  • In realistic settings, limited network expressibility can leave a positive validation-error floor even as data grow and independent-variable noise tends to zero.Source: §1 · p. 5
  • 10^-3frms to 10^-5frms were the ϵ0NN values obtained across the tested equations.Source: §1 · p. 5

2. Translational symmetry and generalizations · 3. Separability · 4. Setting variables equal

The method uses neural-network-guided simplifications to reduce the number of variables before recursively applying symbolic regression. It tests pairwise symmetries, separability, and substitutions that set variables equal.

2. Translational symmetry and generalizations

  • Pairwise invariance under a common shift lets the method replace two variables with their difference; it also tests whether pairs can be replaced by their sum, product, or ratio.The ratio test corresponds to scaling symmetry: two variables can be rescaled simultaneously without changing the function.Source: §2 · p. 5
  • Found symmetries reduce the mystery by one input, after which a fresh instance of the full algorithm solves it recursively; the symmetry threshold is set to 7 times neural-network validation error.The paper says this threshold roughly optimized training-set performance.Source: §2 · p. 5

3. Separability

  • Separability means splitting a function into parts with no variables in common, tested in both additive and multiplicative forms; the logarithm of a multiplicatively separable function is additively separable.Source: §3 · p. 5
  • For a two-variable multiplicative test, constants are chosen as the means of the mystery-data variables, and separability is accepted when the non-separability measure’s RMS is below 10 times neural-network validation error.The passages describe the measure as vanishing for a multiplicatively separable function.Source: §3 · pp. 5–6
  • When separability is found, the method creates two reduced mysteries, solves them recursively, and combines their solutions while accounting for numerical constants; for multiple variables, it tests all possible variable subsets.For additive separability it replaces multiplication and division with addition and subtraction; for multiplicative separability it uses the corresponding product and ratio construction.Source: §3 · p. 6

4. Setting variables equal

  • The method sets each pair of input variables equal in turn, solves the resulting lower-dimensional mystery, and then tries solving the original mystery with that solution divided out.Source: §4 · p. 6
  • For Gaussian probability distribution mystery I.6.2, setting θ and σ equal and dividing out the result removes the denominator, leaving an exponential that becomes readily solvable after taking its logarithm.Source: §4 · p. 6

F. Extra Transformations · III. RESULTS · A. The Feynman Symbolic Regression Database

The paper pairs preprocessing transformations intended to simplify symbolic search with FSReD, a freely available database for quantitative testing of symbolic-regression algorithms. FSReD provides input data, optional unit information, and answer equations evaluated by symbolic equivalence.

F. Extra Transformations

  • Transforming dependent variables with functions such as square root, powers, logarithms, exponentials, and trigonometric functions reduces brute-force symbols and can make equations solvable by polynomial fitting.The same transformations are also applied to independent variables, and multiplication or division by 2 is added in this case.Source: F. Extra Transformations · p. 6
  • The algorithm tests whether a function is multiplicatively separable up to an additive constant and retains a simpler backup test, which is more robust when the constant is zero.Source: F. Extra Transformations · p. 6
  • The algorithm’s tunable hyperparameters are summarized in Table II.Source: F. Extra Transformations · p. 6, Table II · p. 6

A. The Feynman Symbolic Regression Database

B. Method comparison

Against Eureqa, AI Feynman solved more of the 100 basic mysteries; the comparison also shows how neural-network-discovered symmetries and separability help on complex cases, while dimensional analysis improves efficiency but is not required for success.

B. Method comparison

  • AI Feynman solved 100% of the 100 basic mysteries, compared with Eureqa’s 71%.The comparison allowed a maximum of two hours of CPU time per mystery.Source: B. Method comparison · p. 10
  • On the most complicated mysteries, AI Feynman’s neural network enables variable elimination by discovering symmetries and separability.Source: B. Method comparison · p. 11
  • Without dimensional analysis, AI Feynman still solved 93% of mysteries, using neural-network strategies to discover separability and translational symmetries.Dimensional analysis often saves time but is usually not necessary for successful solving; in one example, the neural-network strategy was used six times to identify symmetries, group variables, and factor out a term.Source: B. Method comparison · p. 11
  • AI Feynman progresses by reducing independent variables, whereas Eureqa’s genetic search improves fits but can become stuck near an accurate yet incorrect formula.For formulas whose parts are combined as a ratio or otherwise not summed, individual parts may be poor fits and unable to compete evolutionarily.Source: B. Method comparison · p. 11

C. Dependence on data size

The data required for discovery depends on the solving method: polynomial-fit and brute-force methods often work with few points, while neural-network-dependent equations require substantially more.

C. Dependence on data size

  • Most equations are found by polynomial-fit and brute-force methods with 10 data points, though some need 100 to avoid overfitting to an overly simple incorrect equation.With complex true equations, too little data can lead the algorithm to “discover” an incorrect, overly simple equation.Source: C. Dependence on data size · p. 11
  • Neural-network-dependent equations require 10^2–10^6 data points for r.m.s. accuracy better than 10^-3; these equations are typically more complex.Source: C. Dependence on data size · p. 11

D. Dependence on noise level

AI Feynman was tested by adding Gaussian noise to mystery outputs and increasing its relative level until it could no longer solve them. Most equations were recovered exactly at noise levels of 10^-4 or less, and almost half were solved at 10^-2.

D. Dependence on noise level

  • Almost half of the equations were still solved at relative noise ϵ = 10^-2, while most were recovered exactly with ϵ ≤ 10^-4.The results are reported for noise added to the dependent variable y.Source: D. Dependence on noise level · p. 11
  • The experiment added independent Gaussian noise to y with standard deviation ϵ yrms, starting at ϵ = 10^-6 and multiplying ϵ by 10 until the algorithm failed to solve each mystery.yrms is the mystery's root-mean-square y-value before noise was added.Source: D. Dependence on noise level · p. 11
  • For brute force and polynomial fitting, thresholds were adjusted to prefer no solution over an approximate one, but were not optimized per mystery; independent-variable noise and real-world data remain future tests.Better threshold choices might allow exact recovery at higher noise for some equations.Source: D. Dependence on noise level · p. 11

E. Bonus mysteries

AI Feynman outperformed Eureqa on the harder bonus mysteries and on the equations presented in, which were mostly arbitrary compositions unlike genuine physics formulas.

E. Bonus mysteries

  • The 20 bonus mysteries served as a test set, selected after the algorithm and hyperparameters were finalized on the 100 basic mysteries.The bonus mysteries were significantly more complex and difficult.Source: E. Bonus mysteries · pp. 11–12
  • AI Feynman solved 90% of the bonus mysteries, compared with Eureqa’s 15%.The greater gap between methods on these more complex equations reflects the need for neural-network-based strategies in more cases.Source: E. Bonus mysteries · p. 12
  • The two bonus mysteries AI Feynman failed required about two years and 100 times the age of the universe, respectively, for brute-force search after dimensional analysis.The failures were the radiated gravitational wave power mystery and the Jackson 2.11 mystery.Source: E. Bonus mysteries · p. 12
  • Adding numerically optimized parameter symbols could address both failures, provided it does not unduly slow symbolic search.The first failure arose when a large dynamic range caused the network to miss separability; suggested remedies include limiting parameter ranges or analyzing log absolute values. For the second, different units or symbols for squaring and exponentiation could shorten the brute-force search.Source: E. Bonus mysteries · p. 12
  • On the mystery functions in, AI Feynman discovered 66.7% of equations, while Eureqa discovered 48.9%.AI Feynman performed less well on this set than on genuine physics formulas because most equations were arbitrary compositions unlikely in real-world problems, lacking exploitable symmetries and separability.Source: E. Bonus mysteries · p. 12

IV. CONCLUSIONS · A. Key findings

AI Feynman combines neural-network fitting with physics-inspired techniques to simplify multidimensional symbolic regression, and outperformed Eureqa on both benchmark sets. The conclusions also highlight a public benchmark database and the method’s resilience to reduced or noisy data, with limits for the most complex equations.

IV. CONCLUSIONS

  • AI Feynman combines traditional fitting with neural-network fitting that reduces problems by discovering symmetries, separability, and other properties.The recursive simplification eliminates dependent variables in multidimensional analytic regression.Source: §IV · p. 12
  • The authors created a freely downloadable benchmark database containing 100 mysteries from the Feynman Lectures on Physics and 20 additional mysteries selected for difficulty and fame.The database was intended to facilitate quantitative benchmarking of symbolic-regression algorithms.Source: §IV · p. 12

A. Key findings

  • AI Feynman discovered 100% of the Feynman equations and 90% of the bonus equations, versus Eureqa’s 68% and 15%, respectively.The bonus set includes Kepler’s ellipse equation.Source: A. Key findings · p. 12
  • Most Feynman equations remained solvable with only 102 data points or with percent-level noise added.The most complex equations requiring neural-network fitting needed more data and less noise.Source: A. Key findings · p. 12
  • The most difficult mysteries showed the greatest improvement when AI Feynman repeatedly deployed its neural-network strategy.Its progress involved repeatedly reducing the problem to simpler ones with fewer variables.Source: A. Key findings · p. 12
  • Unlike AI Feynman’s repeated simplification, genetic algorithms must search the full problem space, risking entrapment in local optima.The authors describe this contrast with Eureqa and other genetic algorithms.Source: A. Key findings · pp. 12–13

B. Opportunities for further work

The authors outline ways to improve symbolic regression through better neural-network fitting, more targeted searches, derivative handling, and hybrid methods, and expect algorithms to continue improving.

B. Opportunities for further work

  • Automated pre-analysis could select elementary functions and hyperparameters for each mystery, adding sine and cosine when oscillations appear or excluding them otherwise.Source: B. Opportunities for further work · p. 13
  • The algorithm could be integrated into equation-discovery programs by estimating derivatives from data or neural-network interpolation and treating them as independent variables.This approach could support discovering equations involving derivatives, including partial differential equations; the authors also mention integrals as a target.Source: B. Opportunities for further work · p. 13
  • Imperfect neural-network fitting introduces de facto noise even in low-noise data; better architectures should target fitting noise near 10^-6.Poor generalization outside the observed domain is acceptable if the fit is highly accurate within it, supporting tasks such as factoring separable functions.Source: B. Opportunities for further work · p. 13
  • Brute-force searches could generalize symmetry tests beyond short arithmetic combinations and focus near neural-network-discovered exact special cases using terms that preserve them.For example, terms can be chosen to vanish or equal unity when a variable is set to a value that makes the expression exact.Source: B. Opportunities for further work · p. 13
  • A hybrid with genetic algorithms could use Pareto-frontier candidates for separable factors and retain the best full expression; brute force could also return a frontier.Genetic algorithms can perform well with significant noise, and returning a sequence of formulas trades increasing complexity for progressively better accuracy.Source: B. Opportunities for further work · p. 13

Related papers

Loading 1905.11481v2…