PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

Yiming Huang, Ziche Liu, Zhuohang Wu, Yiqian Wang, Junxia Cui, Xinkai Zou, Linjun Mao, Nan Huang, Naicheng Yu, Kaijie Zhu, Yue Ma, Kun Zhou, Letian Peng, Jingbo Shang

arXiv:2609.01552v1cs.AIcs.LG

TL;DR

It remains unclear whether LLMs genuinely discover scientific laws rather than recall published formulas or exploit simplified synthetic settings. SCILAWS-BENCH addresses this gap with paper-grounded real-data tasks and active residual-calibrated worlds, finding discovery capability but unreliable validity, novelty, and candidate selection.

  • Problem

    Existing evaluations may test memorization of published equations or reasoning in artificial settings, leaving empirical scientific-law discovery insufficiently assessed.

  • Method

    SCILAWS-BENCH evaluates nine LLMs on paper-grounded SCILAWS-REAL tasks with fixed observations and SCILAWS-PARALLEL tasks with queryable worlds containing synthesized hidden laws.

  • Results

    Models show scientific-law discovery capability, but predictive fit can diverge from validity, memorization aids published-form recovery rather than added structure, and self-selection misses stronger candidates.

  • Takeaways & Limitations

    Separating fit, validity, memorization, structural recovery, and selection provides a more complete evaluation of scientific law discovery.

Abstract

from arXiv · show

Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints. As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated. Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs. We therefore introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data. It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines. Each problem is instantiated in two complementary settings: (1) SCILAWS-REAL asks models to propose laws from fixed real observations and evaluates held-out predictive fit and scientific validity derived from the source literature, and (2) SCILAWS-PARALLEL asks models to actively query residual-calibrated worlds and recover synthesized hidden laws derived from published forms. This two-setting task design preserves each problem's scientific context while separately evaluating fixed-record law discovery and active recovery of a newly synthesized hidden law. We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck. Our work provides a paper-grounded benchmark and new empirical perspectives for evaluating AI for scientific discovery. Project page: https://yiyihum.github.io/SciLaws-Bench

1 INTRODUCTION

SCILAWS-BENCH addresses whether LLMs can discover scientific laws from evidence rather than reproduce familiar formulas. It combines paper-grounded real-data discovery with active recovery of newly synthesized laws and finds capability alongside major reliability gaps.

  • Motivation: Existing benchmarks either reuse canonical equations vulnerable to memorization or rely on artificial worlds removed from noisy scientific measurements.This leaves open how well LLMs perform open-ended empirical law discovery.
  • Benchmark: SCILAWS-BENCH contains 118 problems from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points.A cold-recall audit suggests these problems are less memorized than canonical textbook equations.
  • Two settings: SCILAWS-REAL uses fixed real observations and separately evaluates held-out predictive fit and source-grounded scientific validity.Published formulas serve as reference baselines rather than recovery targets.
  • Two settings: SCILAWS-PARALLEL lets models query residual-calibrated worlds to recover a fixed hidden law synthesized as a structural variant of a published formula.The setting tests novel structural recovery while preserving each problem’s scientific context.
  • Findings: Models struggle to balance predictive fit with scientific validity, use memory to reproduce known laws more readily than novel structure, and often fail to select their strongest generated candidates.Best-of-N search exposes stronger candidates, but self-selection captures only a small fraction of the available gains.
  • Conclusion: Current LLMs show evidence of scientific-law discovery capability, but that capability remains far from reliable.The benchmark separates fit, validity, memorization, structural recovery, and selection in evaluating this capability.

2 SCILAWS-BENCH

SCILAWS-BENCH constructs paper-linked tasks from real scientific data and evaluates both fixed-record discovery and active hidden-law recovery. Its design preserves scientific constraints while testing generalization, validity, and structural identification.

  • Construction: SCILAWS-BENCH spans 118 problems across six disciplines, 291 candidate laws, 381 scientific papers, and roughly 8M real data points.It includes both single-group and multi-group problems.
  • Task structure: The dataset contains 66 single-group and 52 multi-group problems, with multi-group tasks sharing a functional form while allowing some parameters to vary by group.This design tests whether a model can discover a law that transfers across groups.
  • Construction: The benchmark links published papers and datasets to build task metadata, executable formulas, cleaned splits, and source-grounded validity rubrics.Human reviewers verify critical variable mappings, implementations, split rationales, and rubric items against cited sources.
  • SCILAWS-REAL: SCILAWS-REAL evaluates numeric fit and scientific validity separately because held-out predictive performance can coexist with violations of paper-grounded requirements.Task-specific splits test interpolation, extrapolation, temporal transfer, or cross-condition and cross-group generalization.
  • SCILAWS-PARALLEL: SCILAWS-PARALLEL replaces fixed observations with residual-calibrated simulators whose hidden laws are newly synthesized structural variants of published formulas.Models select inputs and groups within a query budget, then submit formulas scored for hidden-structure recovery.

3 EXPERIMENTS AND ANALYSIS

The benchmark evaluates frontier LLM agents on scientific law discovery using shared agent protocols, multiple metrics, real observations, and synthesized parallel worlds. Results show that fit, validity, memorization, domain strengths, and candidate selection can diverge substantially.

  • Experimental setup: The shared ReAct-style framework gives agents Python analysis, formula submission, and, in SCILAWS-PARALLEL, fixed-budget active experimentation.Trials allow up to 30 interaction turns, while models with reasoning controls use medium reasoning effort.
  • Model performance across metrics and domains: GPT-5.5 leads all three aggregate metrics, while multi-group scores are generally lower than single-group scores, especially for scientific validity and structure recovery.The aggregate leaderboard does not establish a consistent second-place model.
  • Model performance across metrics and domains: 54.9% tie-aware fit–validity concordance rises to 69.5% only when the fit margin reaches ∆SN ≥0.4.Thus, high numeric fit does not reliably imply scientific validity or correct structure recovery.
  • Model performance across metrics and domains: GPT-5.5 ranks first in numeric fit across five of six domains but ranks seventh in Biology, where Gemini 3.5 Flash leads with 46.9% versus 41.0%.Leaders also vary across metrics and domains, including scientific validity and structure recovery.
  • How memorization shapes scientific discovery: Only 11.9% of tasks are cold-recalled by all nine models, 47.5% by none, and GPT-5.5 recalls 34.7%.This task-level pattern forms a discovery moat and is stable across the six audited vendors.
  • How memorization shapes scientific discovery: On SCILAWS-REAL, GPT-5.5 exceeds the reference on 54% of Moat tasks versus 21% of Canon tasks, while mean numeric fit remains nearly unchanged across tiers.On SCILAWS-PARALLEL, published-form recovery falls from 0.85 on Canon to 0.59 on Moat, whereas full-law recovery remains near 0.08 across tiers and ranges from 0% to 21% across models.
  • Best-of-N search reveals a selection bottleneck: From N = 5 to N = 20, oracle scores rise by 4.5 percentage points in numeric fit and 5.3 in scientific validity, while self-selected performance improves by less than 2 points on either metric.Within five-candidate groups, 60% of selected winners are Pareto-dominated; at N = 20, oracle joint balance reaches 0.93 versus 0.61 for self-selection.

4 RELATED WORK

Prior work spans symbolic regression, LLM-based scientific agents, formula-recovery benchmarks, workflow evaluations, and interactive simulated environments. SCILAWS-BENCH instead evaluates whether LLMs combine observations with scientific context to discover valid relationships.

  • Symbolic Regression: Classical and neural symbolic-regression methods search or learn equation spaces primarily from tabular observations.The paper does not position classical symbolic regression as its comparison target.
  • Symbolic Regression: SCILAWS-BENCH evaluates whether LLMs combine data with scientific context to discover scientifically valid relationships rather than only optimize formulas against observations.This framing distinguishes the benchmark from conventional symbolic-regression comparisons.
  • AI for Scientific Discovery: Recent scientific-agent systems use LLMs to propose hypotheses, run code, and inspect evidence, while other work improves agent prompts through reusable failure-mode heuristics.These efforts focus either on scientific workflows or on improving the agents themselves.
  • Discovery Benchmarks: Existing discovery benchmarks range from formula recovery to data-driven workflows and interactive simulated environments.The benchmark landscape includes AI Feynman, SRBench, DiscoveryBench, ScienceAgentBench, SciGym, PhysGym, Gravity-Bench-v1, and NewtonBench.

5 CONCLUSION

SCILAWS-BENCH evaluates scientific law discovery by separating predictive fit, scientific validity, memorization, structural recovery, and candidate selection. The results show that stronger candidates do not guarantee reliable selection or scientifically valid laws.

  • Predictive fit does not reliably establish scientific validity.
  • Memorization helps models reproduce published laws but not discover new structure.
  • Larger best-of-N candidate pools contain better laws, but models often fail to select them.
  • Separating fit, validity, memorization, structure recovery, and selection enables a more complete evaluation of scientific law discovery.

TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL

The supplementary material includes a worked construction-pipeline example connecting paper–dataset curation, fixed-data discovery, hidden-law synthesis, and residual-calibrated parallel worlds.

  • The appendices include the SCILAWS-REAL system prompt.
  • Figure 8 illustrates the pipeline from paper–dataset curation and SCILAWS-REAL to hidden-law synthesis and a residual-calibrated SCILAWS-PARALLEL world.

A.1 CONSTRUCTION DATA AND HIDDEN LAW

SCILAWS-PARALLEL worlds are constructed from real records by selecting and refitting published forms, calibrating residual variation, synthesizing admissible hidden-law variants, and exposing bounded adaptive queries without revealing the generator.

  • The simulator pools real records to calibrate a separate SCILAWS-PARALLEL world, while the solver begins without observations.
  • A published reference is selected using residual-space criteria that favor little systematic residual structure rather than merely the smallest aggregate prediction error.
  • Hidden-law candidates are screened for execution, predictive adequacy, scientific constraints, and symbolic integrity before a judge selects one using plausibility, simplicity, and source consistency.
  • Logarithmic residuals are used for log_mae and suitable positive multi-decade targets; other tasks use linear residuals.
  • Local mean residuals are removed before sampling, preserving local empirical variation and non-Gaussian shape without claiming complete measurement-process identification.
  • Residual scale may only decrease, targeting residual-space R2 ≥0.9 without forcing identical worlds.
  • Trials start empty, permit at most ten experiment calls, and enforce task-specific point and repeated-sample limits.
  • SCILAWS-PARALLEL is scored by ordinal structural recovery rather than numeric fit or scientific-validity scores.

B EXPERIMENTAL PROTOCOL

The experimental protocol fixes interfaces, budgets, and scoring across models while controlling provider settings where possible. It evaluates both standard trajectories and inference-time candidate selection against metric-specific oracles.

  • Agent interfaces, task inputs, interaction budgets, and scoring protocols are held fixed across models, with remaining provider differences reported.
  • Reasoning models use a medium reasoning tier, while decoding settings and completion budgets follow the reasoning split.
  • The agent uses Python, experiment, and final_formula tools under a one-tool-per-turn protocol with a forced-submission turn and one retry.
  • The sandbox blocks filesystem, network, introspection, and direct task-directory reads, preventing access to held-out splits or hidden generators.
  • SCILAWS-REAL provides fixed training data and specifies held-out input ranges spanning interpolation, extrapolation, temporal, cross-condition, and unseen-group evaluations.
  • SCILAWS-PARALLEL starts with no observations and limits trials to ten experiment calls, bounded distinct points per call, and three repeated samples per point.
  • GPT-5.4-mini serves as a code-executing judge for validity and structure scores, allowing formula probes to be checked by execution.
  • 20 candidate trajectories are generated for inference-time search, with self-selection producing N = 5 and N = 20 results.

C EVALUATION PROTOCOL AND VERBATIM JUDGE PROMPTS

The benchmark evaluates submitted laws through separate numeric-fit and scientific-validity channels, with executable judges, anti-hacking checks, and fit–validity concordance analysis. Judge validation uses stratified expert comparisons and reports agreement metrics against human–human ranges.

  • C EVALUATION PROTOCOL AND VERBATIM JUDGE PROMPTS: SCILAWS-REAL reports numeric_score and validity_score separately, while SCILAWS-PARALLEL reports structure_score for hidden-mechanism recovery.The separate channels distinguish predictive fit, scientific validity, and structural recovery.
  • C.1 SCILAWS-REAL REFERENCE-RELATIVE NUMERIC SCORING: Numeric scoring compares held-out submissions with the strongest published reference using each task’s primary raw metric.RMSE, SMAPE, and log-MAE are lower-is-better; R2 is higher-is-better.
  • C.1 SCILAWS-REAL REFERENCE-RELATIVE NUMERIC SCORING: The clipped numeric score assigns 0.5 to the strongest collected formula and 1.0 to a perfect predictor.The clipping operator bounds the score to [0, 1].
  • C.1 SCILAWS-REAL REFERENCE-RELATIVE NUMERIC SCORING: Single-group submissions declare used columns and a predict function, whereas multi-group submissions fit declared local parameters separately within each test group.Multi-group scores average equally across groups, and contract or runtime failures receive zero numeric score.
  • C.2 SCILAWS-REAL VALIDITY JUDGE: Validity scoring checks behavioral and structural rubrics derived from each source problem, while coefficient accuracy remains part of numeric scoring.Checks include monotonicity, finiteness, bounds, limits, separability, required dependencies, and meaningful terms.
  • C.2 SCILAWS-REAL VALIDITY JUDGE: A code-executing judge evaluates formulas on controlled probes and records rubric-level JSON verdicts, using source inspection when numeric probing is insufficient.The protocol builds deterministic input grids and supports separate handling for single- and multi-group tasks.
  • C.3 SCILAWS-REAL ANTI-HACKING RUBRIC: The anti-hacking rubric rejects hidden degrees of freedom such as lookup tables, copied aggregates, excessive parameters, and dataset-approximating code.Legitimate fitted constants are not penalized; the check targets circumvention of the declared law contract.
  • C.4 FIT–VALIDITY CONCORDANCE: Fit–validity concordance averages directional agreement across task–model pairs, using 3,616 pairs at τ = 0 and a 50% no-association reference.Confidence intervals come from 20,000 task-level bootstrap samples retaining eligible model pairs.

D.4 RESULTS

The evaluation judges agree with human experts at levels comparable to human–human agreement, while the memorization audit operationalizes cold recall as repeated structural reproduction from a data-free prompt. Structural matching permits algebraic equivalence but preserves functional-form differences.

  • Judge validation: Memorization, validity, and structure judge–human κ values are 0.77, 0.82, and 0.84, each within the corresponding human–human range.Exact agreement is 89% for memorization, 92% for validity, and 66.7% for structure.
  • Judge validation: 98.5% of structure-annotated items differ by at most one level, making quadratic-weighted κ more informative than exact match alone.Only two of 135 items differ by more than one level.
  • Memorization audit: The memorization audit separates recalling a published reference formula from discovering a useful form from task records.It measures recall without giving models any data.
  • Memorization audit: The audit covers 118 tasks and nine models, including 66 single-group and 52 multi-group tasks.These produce 1,062 task–model cells.
  • Memorization audit: Cold recall uses a restricted data-free prompt containing target and ordered input descriptions, units, symbols, and ranges, but no rows, citations, or paper text.Each task–model cell receives five completions at temperature 0.8.
  • Memorization audit: Structural matching treats algebraic rearrangements and coefficient reparameterizations as equivalent, but distinguishes exponents, term counts, variables, and functional families.Linear combinations do not match power laws, and Gompertz forms do not match logistic forms.
  • Memorization audit: A task–model cell is cold-recalled when at least three of five samples match the best-baseline reference form.All other cells are classified as not cold-recalled.

E.4 MAIN RESULTS

Cold recall is limited and uneven: most task–model cells do not reproduce the reference form, while recall concentrates in a polarized set of tasks and varies by structure and scientific domain. The target-choice robustness check leaves the overall and frontier rates essentially unchanged.

  • Overall cold recall: 30.7% of task–model cells are cold-recalled, while 69.3% are not, across 118 tasks and nine models.The strongest observed models reach the mid-30% range, but none recalls roughly two-thirds of tasks.
  • Task-level distribution: 56 tasks, or 47.5% of the panel, are recalled by no audited model, whereas 14 tasks, or 11.9%, are recalled by all nine.The latter constitute the universal-canon group.
  • Task structure: Multi-group tasks have 40.2% cold recall versus 23.2% for single-group tasks across all task–model cells.The passage associates higher recall with named cross-group or invariant-constant forms.
  • Robustness: Relaxing the recall target to any R2 ≥0.9 baseline changes the all-model cold-recall rate by only 0.85 percentage points and leaves the frontier rate unchanged.The robustness result concerns the choice of recall target.
  • Scientific domains: Mean cold recall is 20.5% in Materials & Engineering, compared with 43.4% in Ecology & Hydrology and 40.5% in Social Sciences.The frontier model reaches 59.1% in Ecology & Hydrology.

E.6 TARGET-CHOICE ROBUSTNESS

The target-choice robustness analysis finds that using the best-baseline representative does not materially inflate measured cold recall. Controlled audits also distinguish lower recall on SCILAWS-REAL from highly canonical formula recall, while integrity checks support—but do not directly observe—memorization differences.

  • Target-choice robustness: 0.85 percentage points: the relaxed R2 ≥0.9 rule increases the all-model cold-recall rate only slightly.Only 16 of 118 tasks have multiple equally good baselines, and eight task-model cells flip across three tasks.
  • Target-choice robustness: The gpt-5.5 cold-recall rate does not change under the relaxed target rule.This result further indicates that the best-baseline anchor is not materially hiding recall.
  • AI-Feynman calibration: 55.9% versus 26.8%: AI-Feynman has a higher eight-model cold-recall rate than SCILAWS-REAL under the controlled audit.The comparison uses the same prompt, judge, five-sample rule, and eight OpenAI models, differing only in corpus.
  • AI-Feynman calibration: 71.0% versus 34.7%: gpt-5.5 cold-recalls more AI-Feynman formulas than SCILAWS-REAL formulas, a 36.3-percentage-point gap.The audit treats AI-Feynman as intentionally textbook-like and highly recallable, while SCILAWS-REAL receives much lower recall.
  • Integrity checks: The capability-ladder checks support real differences in formula recall under benchmark-visible information.They address plotting, logging, completion, and judging artifacts, but do not observe training data directly.

G.3 PARALLEL × SINGLE-GROUP: THE KEELING CURVE

The Keeling-curve parallel task lets models query noisy atmospheric CO2 observations and compare an exponential-accumulation family with a quadratic alternative. In the illustrated trajectory, the model submits the quadratic, which tracks the data but misses the hidden structure.

  • Task setup: The parallel Keeling-curve task starts without observations and lets the agent choose year measurements through simulator queries.The task covers decimal years 1958–2026, uses noisy replicate observations, and allows at most 12 global constants.
  • Model comparison: The agent fits an exponential-accumulation trend with optional annual harmonics and compares it against a plain quadratic.It evaluates weighted RMSE across harmonic counts h = 0..4.
  • Submission and score: S_S = 0.25: the submitted quadratic tracks the data but is not the hidden exponential-accumulation family.The agent had fit the hidden family but ultimately submitted the quadratic formula.
  • Real-world comparison: The corresponding SCILAWS-REAL model submits a cubic trend plus three annual harmonics and scores S_N = 0.78.This score exceeds the 12-constant NOAA reference in the illustrated comparison.

G.4 PARALLEL × MULTI-GROUP: SELLMEIER DISPERSION

The Sellmeier parallel task asks models to query observations across crystal clusters and recover one shared dispersion form with cluster-specific parameters. The illustrated agent finds that adding a long-wavelength correction improves fitting and exactly recovers the hidden family.

  • Task setup: The task contains 14 crystal clusters and requires one shared form with at most four per-cluster parameters and no global fitted constants.Models may make up to 10 experiment calls with at most 20 points each.
  • Model comparison: The agent compares Cauchy, single-pole Sellmeier, and Sellmeier-plus-IR-term families across clusters.It evaluates refractive index n as a function of wavelength lambda and compares maximum fitting error.
  • Model comparison: The four-parameter Sellmeier-plus-IR-term fit reduces maximum error to approximately 1e-4, best among the tried families.The fitted form is n^2 = A + B/(lambda^2 - C) + D*lambda^2.
  • Submission and score: S_S = 1.0: the submitted model recovers the hidden n^2 = A + B/(l^2-C) + D*l^2 family exactly.No misses are reported in the illustrated parallel result.
  • Real-world comparison: On the corresponding real task, the model submits a Cauchy-type series and scores S_N = 0.21, below the strongest published baseline.The baseline is the four-term Cauchy equation.
Loading 2609.01552v1…