PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, Chandan K Reddy

arXiv:2504.10415v2cs.CLcs.AIcs.LG

TL;DR

Existing equation-discovery benchmarks often use familiar equations that LLMs can memorize, limiting evidence about genuine discovery. LLM-SRBench introduces 239 transformed and synthetic problems across four domains and finds that current systems achieve only limited symbolic accuracy, while exposing important scope boundaries in benchmark construction.

  • Problem

    Existing benchmarks rely on common textbook equations that are susceptible to LLM memorization, making rigorous evaluation of genuine equation discovery difficult.

  • Method

    LLM-SRBench evaluates equation-discovery methods on LSR-Transform and LSR-Synth problems designed to require reasoning beyond familiar mathematical forms and memorized solutions.

  • Results

    31.5% symbolic accuracy is the best reported result on LSR-Transform, while the best model solves 28.1% of LSR-Synth problems.

  • Takeaways & Limitations

    LLM-SRBench provides a challenging evaluation foundation for future LLM-based scientific equation-discovery methods.

  • Takeaways & Limitations

    Some transformations can reduce physical interpretability or obscure natural causal relationships despite remaining mathematically valid.

Abstract

from arXiv · show

Scientific equation discovery is a fundamental task in the history of scientific progress, enabling the derivation of laws governing natural phenomena. Recently, Large Language Models (LLMs) have gained interest for this task due to their potential to leverage embedded scientific knowledge for hypothesis generation. However, evaluating the true discovery capabilities of these methods remains challenging, as existing benchmarks often rely on common equations that are susceptible to memorization by LLMs, leading to inflated performance metrics that do not reflect discovery. In this paper, we introduce LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. Our benchmark comprises two main categories: LSR-Transform, which transforms common physical models into less common mathematical representations to test reasoning beyond memorized forms, and LSR-Synth, which introduces synthetic, discovery-driven problems requiring data-driven reasoning. Through extensive evaluation of several state-of-the-art methods, using both open and closed LLMs, we find that the best-performing system so far achieves only 31.5% symbolic accuracy. These findings highlight the challenges of scientific equation discovery, positioning LLM-SRBench as a valuable resource for future research.

1. Introduction

Scientific equation discovery seeks interpretable mathematical relations from observational data, but existing benchmarks often use familiar textbook equations that LLMs may memorize. LLM-SRBench addresses this evaluation gap with transformed and synthetic problems, finding that current systems achieve limited accuracy.

  • Equation discovery seeks symbolic relations that explain natural phenomena, supporting prediction, extrapolation, and scientific insight.
  • Existing equation-discovery methods face computationally prohibitive search spaces and limited integration of scientific domain knowledge.
  • Common benchmarks rely largely on textbook physics equations, making them susceptible to LLM memorization rather than genuine discovery.
  • LLM-SRBench introduces alternative mathematical representations and synthetic discovery-driven problems to evaluate reasoning beyond memorized forms.
  • 239 challenging problems span four scientific domains and two categories designed to test LLM-based scientific equation discovery.
  • 31.5% symbolic accuracy is the best reported performance on LLM-SRBench, underscoring the benchmark’s difficulty.

2. LLM-SRBench

LLM-SRBench defines equation discovery as deriving scientifically plausible hypotheses from context and data, then evaluates methods across transformed and synthetic tasks. Its protocol combines diverse hypothesis representations with data-fidelity and symbolic-equivalence assessments.

  • Equation discovery derives a hypothesis from task data and scientific context that represents underlying mathematical relations with precision and scientific plausibility.
  • LSR-Transform: LSR-Transform converts established Feynman problems into less common mathematical forms by altering symbolic representations and input-output roles.
  • LSR-Transform: LSR-Transform retains analytically solvable transformations and filters data to satisfy valid domains of the transformed equations.
  • LSR-Synth: LSR-Synth combines known scientific terms with novel synthetic terms, using solvability and novelty checks before generating benchmark data.
  • Evaluation: The benchmark evaluates both data fidelity and symbolic accuracy because numerically similar equations may differ symbolically, and symbolically similar forms may differ numerically.
  • Evaluation: Data fidelity uses Accτ and NMSE on in-domain and available out-of-domain data, while GPT-4o evaluates mathematical equivalence across hypothesis representations.

3. Experiments

Experiments evaluate LLM-based equation discovery methods across symbolic accuracy, numeric precision, NMSE, complexity, and OOD generalization. Results show substantial difficulty, dataset-dependent method strengths, and a peak symbolic accuracy of about 31%.

  • Experimental Setup: The benchmark evaluates LLM-SR, LaSR, SGA, and DataBlind across three LLM backbones using symbolic accuracy, Acc0.1, and NMSE.Each task provides a problem description, variables, dataset, and task instruction; methods generate and refine equation hypotheses.
  • Main Results: Performance remains relatively low across symbolic and numeric metrics, underscoring the fundamental difficulty of scientific equation discovery.
  • Main Results: LaSR leads LSR-Transform numerical accuracy in both Acc0.1 and NMSE, while LLM-SR with GPT-4o-mini achieves about 31% symbolic accuracy.
  • Main Results: On LSR-Synth material science problems, LaSR has better symbolic accuracy while LLM-SR achieves better numerical precision.
  • Complexity Analysis: LSR-Transform problems are substantially harder than Feynman problems at matched expression-tree complexity, including simpler problems with 0–15 nodes.
  • OOD Generalization: All methods show higher NMSE on OOD data than ID data, while LLM-SR achieves the lowest NMSE in both settings and DataBlind performs worst.
  • OOD Generalization: Symbolic accuracy positively correlates with OOD Acc0.1 and negatively correlates with OOD NMSE, supporting OOD evaluation as a measure of equation generalization.

4. Related Work

Related work spans AI-for-science applications, symbolic regression methods, and scientific benchmarks. The paper positions LLM-SRBench as a benchmark for testing LLM-based discovery beyond memorized equations.

  • AI for Scientific Discovery: AI-for-science research uses LLMs and tool-augmented agents to generate hypotheses and perform scientific data-driven analysis.
  • Symbolic Regression: Symbolic regression methods include search-based, learning-based, and hybrid approaches, but mostly lack mechanisms for incorporating scientific domain knowledge.
  • LLMs for Equation Discovery: LLM-based symbolic regression methods combine embedded scientific knowledge with evolutionary search, concept learning, or bilevel optimization.
  • Symbolic Regression Benchmarks: Existing scientific benchmarks contain well-known equations that LLMs may solve through memorization rather than scientific reasoning and discovery.
  • Symbolic Regression Benchmarks: LLM-SRBench extends prior work by assessing LLM-driven scientific equation discovery beyond memorization.

5. Conclusion

The paper introduces LLM-SRBench as a broad benchmark for testing LLM-driven scientific equation discovery under challenges designed to exceed memorization. Its reported results indicate that current systems remain far from reliable symbolic discovery.

  • LLM-SRBench contains 239 tasks: 111 LSR-Transform problems and 128 LSR-Synth problems across four scientific disciplines.
  • The benchmark supports standardized evaluation across expression strings and programs using multiple scientific equation discovery methods and LLM backbones.
  • The best reported system reaches only 31% peak performance, highlighting substantial challenges and open research opportunities.

Impact Statement

The future adoption of LLM-SRBench could significantly impact AI for science and scientific discovery, with potential societal consequences.

  • Future adoption of LLM-SRBench could significantly impact artificial intelligence for science and scientific discovery.

A. Dataset Details

LLM-SRBench constructs two complementary dataset categories to test equation discovery beyond memorized forms: transformed familiar models and synthetic cross-domain problems. The datasets preserve scientific context while introducing unfamiliar representations, novel terms, controlled complexity, and quality-filtered data.

  • LSR-Transform: LSR-Transform reformulates familiar scientific equations into less-common mathematical forms to test semantic and data-driven reasoning beyond memorized representations.Problems are obtained by analytically solving equations for different input variables while retaining the original scientific context.
  • LSR-Transform: 111 LSR-Transform equations remain after filtering for semantic difficulty, excluding overly complex expressions and problems solvable by direct sampling without data.The filtering targets reasoning and memorization rather than syntactic complexity or lengthy hypotheses.
  • LSR-Synth: LSR-Synth combines established scientific terms with synthetic, plausible variations across chemistry, physics, biology, and material science.The synthetic terms introduce nonlinearities and interactions that are uncommon in standard formulations.
  • LSR-Synth: LSR-Synth spans simple to highly complex expression trees, with randomly sampled valid parameters and domain-specific numerical or direct solution procedures.Quality control requires numerical solvability, meaningful physical behavior, and uniqueness relative to existing solutions.
  • Dataset construction: Figure 9 illustrates LSR-Synth targets composed of known and synthetic terms, while Figure 7 illustrates transformed representations derived from original Feynman problems.Together, the examples show how the benchmark challenges memorization through altered forms and novel combinations.

B.1. Data Fidelity

Data-fidelity evaluation combines a binary tolerance-based accuracy metric with scale-normalized error, while symbolic evaluation uses GPT-4o to assess mathematical equivalence across representations. The figures and accompanying methodology frame evaluation around predictive quality and semantic correctness.

  • Predictive metrics: Accτ marks an equation accurate only when its maximum point-wise relative error across test samples satisfies tolerance τ.This binary metric emphasizes consistent accuracy and exposes occasional large deviations.
  • Predictive metrics: NMSE provides a continuous, scale-invariant measure of prediction error, with 0 indicating perfect prediction.Unlike Accτ, it considers the magnitude of errors across all test points.
  • Dataset analysis: Figure 10 is described as showing LSR-Synth problem-complexity distributions across scientific domains.The supplied caption identifies the figure’s subject but does not report a specific distributional outcome.
  • Symbolic evaluation: GPT-4o evaluates symbolic equivalence between predicted and gold equations after preprocessing representations and replacing constants with placeholder parameter vectors.The procedure supports equation strings, expression trees, and executable programs while focusing on logical structure and mathematical relations.

C. Implementation Details

The implementation evaluates multiple LLM-guided discovery agents under controlled LLM-call budgets using prompts that generate, mutate, recombine, abstract, and evolve equation hypotheses. Candidate programs are evaluated on data by optimizing numeric parameters against prediction loss.

  • Baselines and backbones: Four LLM-guided equation-discovery baselines are tested with Llama-3.1-8B-Instruct, GPT-3.5-turbo, and GPT-4o-mini backbones.The setup includes one open-source and two closed-source LLM families.
  • Baselines and backbones: All baseline frameworks receive a matched budget of 1k LLM calls per problem for fair comparison.Hyperparameters are adjusted where needed to equalize access across discovery frameworks.
  • LLM-SR: LLM-SR represents candidate equations as functions, loads input-output observations, optimizes parameters with BFGS, and scores the resulting loss.The evaluation code computes mean squared error between predicted and observed outputs.
  • Prompted search operations: The prompt suite supports equation specification, initialization, mutation, crossover, concept abstraction, and concept evolution.These operations generate expressions, infer concepts from good and bad hypotheses, and create new concepts for subsequent search.
  • LaSR: LaSR prompts ask the model to propose or recombine expressions under specified variables, operators, suggestions, and reference expressions.The prompts request brief reasoning and return expression candidates in JSON-list form.

D. Additional Results and Analysis

Additional analyses show substantial variation in numeric error across methods, backbones, and benchmark problems, while symbolic accuracy strongly aligns with out-of-distribution performance. Qualitative examples further examine the hypotheses produced by different discovery methods.

  • Numeric accuracy: Figure 12 compares NMSE distributions across discovery methods with GPT-4o-mini and across LLM backbones using LLM-SR.The accompanying analysis attributes variation to transformed representations and combinations of known and synthetic terms.
  • Symbolic accuracy and generalization: The strong correlation between symbolic accuracy and OOD performance supports OOD evaluation as a metric for generalizable equation discovery.The same relationship also supports the alignment of GPT-4o symbolic evaluation with numeric generalization performance.
  • Qualitative analysis: Figure 14 compares final hypotheses for a biological population-growth problem across discovery methods using Llama-3.1-8B.The qualitative analysis reports that Direct Prompting captures basic dynamics and SGA incorporates a common population-growth component.

E. Discussion and Future Directions

LLM-SRBench results show that scientific equation discovery remains challenging for LLMs and that evaluation must address memorization, interpretability, and scientific meaningfulness.

  • Discussion: LLM-based scientific equation discovery remains challenging across model architectures, requiring domain knowledge, data-driven search, and mathematical manipulation.The discussion identifies these capabilities as a complex interplay needed for discovery.
  • Discussion: The paper questions evaluation paradigms that cannot distinguish memorized solutions from genuine scientific discovery.Existing benchmarks are described as susceptible to memorization and inadequate for assessing true discovery capabilities.
  • Benchmark design: LLM-SRBench addresses memorization by combining synthetic imaginary scenarios with transformed equations that require data-driven discovery.LSR-Synth uses scenarios not based on existing scientific knowledge, while LSR-Transform changes familiar equations into less common representations.
  • Benchmark design: Some transformed equations preserve mathematical validity but weaken physical interpretability or obscure natural causal relationships.The benchmark therefore balances mathematical rigor with physical meaningfulness and constrains transformation complexity.
  • Future directions: Future methods could use literature retrieval to ground scientific context and prioritize mathematically valid, data-consistent, novel, and meaningful discoveries.The paper also calls for evaluation beyond narrow accuracy metrics, including novelty, interpretability, and practical applicability.

F. Comparison with Standard (non-LLM) Symbolic Regression Baselines

The comparison evaluates PySR against LLM-based methods on LLM-SRBench. PySR can achieve strong numerical fit, but its symbolic performance is much weaker, especially outside physics.

  • Experimental setup: PySR operates on numerical data without scientific context, variable descriptions, or domain knowledge available to LLM-based methods.The comparison tests whether a traditional symbolic regression system can discover benchmark equations without contextual information.
  • Results: PySR achieves competitive and sometimes better Acc0.1 numerical accuracy across all datasets.Acc0.1 denotes accuracy to tolerance 0.1.
  • Results: 0% symbolic accuracy is reported for PySR on chemistry, biology, and material science datasets.The symbolic-performance gap is most pronounced on problems requiring specialized scientific knowledge.
  • Results: PySR reaches 4.54% symbolic accuracy on LSR-Synth Physics and 8.11% on LSR-Transform.Its relatively stronger physics performance suggests that some physics problems align better with PySR’s dictionary-based search.
  • Interpretation: The findings indicate that numerical data-driven optimization alone does not adequately address the scientific challenges captured by LLM-SRBench.The comparison strengthens the motivation for LLM-based scientific equation discovery.
Loading 2504.10415v2…