PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

Yiming Huang, Ziche Liu, Zhuohang Wu, Yiqian Wang, Junxia Cui, Xinkai Zou, Linjun Mao, Nan Huang, Naicheng Yu, Kaijie Zhu, Yue Ma, Kun Zhou, Letian Peng, Jingbo Shang

arXiv:2609.01552v1cs.AIcs.LG

TL;DR

Whether LLMs genuinely discover scientific laws remains open, and existing evaluations can trade scientific grounding against resistance to memorization. SCILAWS-BENCH pairs fixed-record discovery from real observations with active recovery in residual-calibrated worlds. Its results show some discovery capability, but predictive fit can diverge from validity, novel-structure recovery remains rare, and models struggle to select their strongest candidates.

  • Problem

    Whether LLMs genuinely discover scientific laws remains open, while existing evaluations face tension between scientific grounding and resistance to memorization.

    Source: §1 · p. 1

  • Method

    SCILAWS-BENCH pairs fixed-observation law discovery, evaluated for predictive fit and scientific validity, with active recovery of synthesized hidden laws in queryable worlds.

    Source: §1 · p. 2, §2.4 · p. 5

  • Results

    LLMs show evidence of discovery capability but remain unreliable: predictive fit can diverge from validity, novel-structure recovery remains rare, and models struggle to select their strongest candidates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Separating fit, validity, memorization, structure recovery, and selection makes SCILAWS-BENCH a more complete evaluation of scientific law discovery.

    Source: §5 · p. 10

Abstract

from arXiv · show

Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints. As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated. Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs. We therefore introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data. It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines. Each problem is instantiated in two complementary settings: (1) SCILAWS-REAL asks models to propose laws from fixed real observations and evaluates held-out predictive fit and scientific validity derived from the source literature, and (2) SCILAWS-PARALLEL asks models to actively query residual-calibrated worlds and recover synthesized hidden laws derived from published forms. This two-setting task design preserves each problem's scientific context while separately evaluating fixed-record law discovery and active recovery of a newly synthesized hidden law. We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck. Our work provides a paper-grounded benchmark and new empirical perspectives for evaluating AI for scientific discovery. Project page: https://yiyihum.github.io/SciLaws-Bench

1 INTRODUCTION

SCILAWS-BENCH evaluates law discovery using real scientific problems in complementary fixed-observation and active-query settings. The results suggest current LLMs can discover laws, but their performance remains unreliable across predictive fit, scientific validity, novel-structure recovery, and candidate selection.

Motivation

  • Existing evaluations risk measuring recall of canonical equations or reasoning in artificial settings rather than empirical discovery from noisy, heterogeneous scientific data.The introduction frames this as a tension between scientific grounding and resistance to memorization.Source: §1 · pp. 1–2

Benchmark construction

  • The benchmark contains 118 problems from 381 scientific papers, with 291 candidate laws and roughly 8M real data points.A cold-recall audit suggests these problems are less memorized than canonical textbook equations.Source: §1 · p. 2

Evaluation settings

  • SCILAWS-REAL tests laws proposed from fixed real observations, while SCILAWS-PARALLEL tests recovery of newly synthesized hidden laws through active queries.The parallel worlds preserve scientific context and calibrate coefficients and residual noise to corresponding real records.Source: §1 · p. 2

Evaluation setup

  • The evaluation includes nine frontier LLMs in a shared ReAct-style framework with a Python sandbox and up to 30 interaction turns per trial.SCILAWS-PARALLEL additionally provides a fixed-budget experimentation interface.Source: §1 · p. 2

Findings

  • Best-fitting formulas can fail scientific validity checks, as a nuclear-physics proposal introduced a nonexistent resonance peak.The benchmark’s findings show that predictive fit and scientific validity do not always align.Source: §1 · p. 2
  • When recall is weak, models explore beyond published formulas, but successful recovery of novel structure remains rare.Models use memory to reproduce known laws but struggle to discover novel structure.Source: §1 · p. 2
  • Best-of-N search produces stronger candidate laws, but models’ self-selection captures only a small fraction of the available gains.Models generate better laws than they can reliably select.Source: §1 · p. 2

2 SCILAWS-BENCH

SCILAWS-BENCH pairs fixed-observation law discovery with active recovery of synthesized hidden laws across 118 problems, six disciplines, and roughly 8M real data points. Its curation, task construction, and evaluation preserve scientific context while distinguishing predictive fit from scientific validity and hidden-structure recovery.

2.1 BENCHMARK OVERVIEW

  • SCILAWS-BENCH covers 118 problems, 291 candidate laws, 381 papers, and roughly 8M real data points across six disciplines, with fixed-data SCILAWS-REAL and active-query SCILAWS-PARALLEL settings.It includes single-group tasks with one global parameterization and multi-group tasks where a shared functional form must transfer across groups with varying parameters.Source: §2.1 · p. 3

2.2 SCIENTIFIC PROBLEM CURATION

  • Curation retains 66 single-group and 52 multi-group problems that meet publication, data-availability, and evaluation criteria.A matched eight-model cold-recall test found substantially lower recall of SCILAWS-REAL target forms than AI-Feynman formulas (Udrescu & Tegmark, 2020), indicating reduced accessibility through direct formula recall.Source: §2.2 · p. 4, Figure 3 · p. 4

2.3 SCILAWS-REAL

  • SCILAWS-REAL evaluates laws proposed from fixed observations against held-out numeric fit and a frozen, source-grounded scientific-validity rubric, reporting the scores separately.Task construction includes executable reference formulas, cleaned data splits, and human verification; multi-group tests hold out entire groups and use calibration observations to fit group-specific parameters before evaluating the shared law.Source: §2.3 · p. 4

2.4 SCILAWS-PARALLEL

  • SCILAWS-PARALLEL replaces fixed observations with queryable worlds whose hidden laws are synthesized structural variants of published formulas.The simulator fits parameters to real data, debiases residuals, resamples noise with k-NN bootstrap, and scales noise to preserve identifiability under each query budget.Source: §2.4 · pp. 5–6
  • SCILAWS-PARALLEL scores recovery of hidden structure on five levels, from invalid or unrelated (0) to complete hidden structure (1).Intermediate levels credit relevant variables or trends (0.25), the published base form (0.5), and the base form with most added terms (0.75).Source: §2.4 · p. 6

3 EXPERIMENTS AND ANALYSIS

The experiments evaluate a shared agent framework across fixed-record and active-query settings, finding that fit, scientific validity, and structure recovery can diverge. Memorization and model-based selection also constrain discovery: larger candidate pools contain better laws, but selectors capture little of that potential.

3.1 EXPERIMENTAL SETUP

  • The shared ReAct-style agent uses Python for analysis and submits a formula; only SCILAWS-PARALLEL permits budgeted input queries, while SCILAWS-REAL starts from fixed training records.Single-group tasks require one global predictor, while multi-group tasks use a shared form with per-group parameters; the standard protocol allows up to 30 turns. Judge agreement with expert labels falls within human–human ranges (κ = 0.77, 0.82, and 0.84).Source: §3.1 · p. 6

3.2 MODEL PERFORMANCE ACROSS METRICS AND DOMAINS

  • All models submit physically valid proton form-factor laws, but none beats the published numeric baseline and seven score SN = 0.Other examples show the metrics can diverge: GPT-5.5 beats the CO2 Keeling-curve baseline with SN = 0.78 but recovers only its trend, and fully recovers the Sellmeier law with SS = 1.0 while scoring SN = 0.21.Source: §3.2 · pp. 6–7
  • GPT-5.5 leads all three aggregate metrics, but no other model consistently ranks second, and multi-group scores are generally lower, especially for validity and structure.Aggregate scores mask domain differences: GPT-5.5 ranks first in numeric fit in five of six domains but ranks seventh in Biology, where Gemini 3.5 Flash scores 46.9% versus GPT-5.5’s 41.0%.Source: §3.2 · pp. 6–7, Table 2 · p. 7
  • Fit and validity have 54.9% tie-aware concordance across unequal-fit task pairs, rising to 69.5% only at ∆SN ≥0.4; within GPT-5.5, fit and full-law recovery correlate at r = 0.04.Across models, SCILAWS-REAL > Ref% and SCILAWS-PARALLEL full-law recovery correlate at r = 0.97, so aggregate improvements do not make fit a reliable indicator on individual tasks.Source: §3.2 · p. 6, Figure 5 · p. 7, Figure 6 · p. 8

3.3 HOW MEMORIZATION SHAPES SCIENTIFIC DISCOVERY

  • Only 11.9% of tasks are cold-recalled by all nine models, while 47.5% are recalled by none; even GPT-5.5 recalls only 34.7%.The audit gives models only the input/output variable specification and classifies a task–model pair as recalled when at least three of five samples match the published reference.Source: §3.3 · pp. 7–8
  • In SCILAWS-REAL, GPT-5.5’s rate of beating published references rises from 21% on Canon tasks to 54% on Moat tasks, while full-law recovery in SCILAWS-PARALLEL remains near 0.08 across recall tiers.Published-form recovery falls from 0.85 on Canon to 0.59 on Moat, while full-law recovery rises across models from 0% for GPT-4o-mini to 21% for GPT-5.5. On the Beer–Lambert task, all models recover the textbook form, but none recovers the added deviation term.Source: Figure 6 · p. 8, §3.3 · pp. 8–9

3.4 BEST-OF-N SEARCH REVEALS A SELECTION BOTTLENECK

  • Oracle gains from N = 5 to N = 20 reach 4.5 percentage points in numeric fit and 5.3 points in scientific validity, while self-selected gains stay below 2 points on either metric.The study draws 20 GPT-5.4-mini candidates per SCILAWS-REAL task and uses the same model to select among them.Source: Figure 7 · p. 9, §3.4 · p. 9
  • 60% of five-candidate group winners are Pareto-dominated; at N = 20, the oracle joint-balance score is 0.93 versus 0.61 for self-selection.Across full pools, 12.7% of self-selected winners lie on the Pareto front, compared with 9.6% under random selection.Source: Figure 7 · p. 9, §3.4 · p. 9

4 RELATED WORK

The related work spans symbolic regression, LLM-based scientific agents, and discovery benchmarks. SCILAWS-BENCH is framed around evaluating empirical scientific laws with real data, scientific context, and paper-grounded validity checks.

Symbolic Regression

  • Classical symbolic regression searches expression spaces with genetic programming or sparse identification, while this work does not use it as a comparison target because it assesses data-plus-context scientific discovery.Symbolic regression also includes neural approaches that learn equation priors from synthetic corpora.Source: §4 · pp. 9–10

AI for Scientific Discovery

  • Recent LLM agents propose hypotheses and inspect evidence, while equation-discovery systems refine equations and explore with code, motivating evaluation beyond synthetic formula recovery (Lu et al., 2024; Shojaee et al., 2025a).REVERE rewrites prompts using recurring cross-repository failure heuristics (Gangireddi et al., 2026), while RESTART transfers successful sub-expressions across related problems.Source: §4 · p. 10

Discovery Benchmarks

  • Existing benchmarks cover formula recovery, data-driven scientific workflows, and interactive simulated exploration; SCILAWS-BENCH instead combines multidisciplinary empirical laws, real data, domain knowledge, and paper-grounded validity checks.These benchmarks are anchors for formula recovery and active exploration, but SCILAWS-BENCH targets their combination with empirical laws and literature-based validity.Source: §4 · p. 10

5 CONCLUSION

SCILAWS-BENCH evaluates scientific law discovery through fixed-record discovery and active recovery in residual-calibrated parallel worlds. Its findings distinguish predictive fit, scientific validity, memorization, structure recovery, and candidate selection.

5 CONCLUSION

  • Predictive fit does not reliably establish scientific validity; memorization aids reproducing published laws but not discovering new structure; and models often fail to select better laws in larger candidate pools.The best-of-N study found that larger pools contain better laws, although models often fail to select them.Source: §5 · p. 10
  • SCILAWS-BENCH pairs paper-grounded fixed-record discovery with active structural identification in residual-calibrated parallel worlds.The benchmark separates evaluation of fit, validity, memorization, structure recovery, and selection.Source: §5 · p. 10
  • Future work can use the framework to develop agents that design informative experiments, reason beyond memorized laws, and reliably identify scientifically valid hypotheses.Source: §5 · p. 10

TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL · A RESIDUAL-CALIBRATED SCILAWS-PARALLEL WORLDS

Figure 8 illustrates the SCILAWS-BENCH construction pipeline, connecting paper–dataset curation and a fixed-data SCILAWS-REAL task to hidden-law synthesis in a residual-calibrated SCILAWS-PARALLEL world.

A RESIDUAL-CALIBRATED SCILAWS-PARALLEL WORLDS

  • Figure 8 presents a worked example of the SCILAWS-BENCH construction pipeline.The example spans paper–dataset curation, a fixed-data SCILAWS-REAL task, hidden-law synthesis, and a residual-calibrated SCILAWS-PARALLEL world.Source: Figure 8 · p. 16
  • The illustrated workflow begins with paper–dataset curation and a fixed-data SCILAWS-REAL task.The figure caption identifies these as stages in the benchmark construction example.Source: Figure 8 · p. 16
  • The example proceeds to hidden-law synthesis and a residual-calibrated SCILAWS-PARALLEL world.The caption names both stages but does not specify their implementation or results.Source: Figure 8 · p. 16

A.1 CONSTRUCTION DATA AND HIDDEN LAW … A.6 QUERY AND EVALUATION PROTOCOL

The appendix constructs SCILAWS-PARALLEL worlds from real records while withholding observations and the hidden law from solvers. It specifies residual fitting and calibration, group handling, query limits, and structural-recovery evaluation.

A.1 CONSTRUCTION DATA AND HIDDEN LAW

  • Real split records calibrate a separate queryable world, whose hidden law is a structurally changed, screened variant chosen for plausibility, simplicity, and source consistency.The starting reference is selected to minimize systematic residual structure; if no reference has nonnegative residual-space R2, the criterion is applied to all nondegenerate references. The solver begins without observations.Source: §A.1 · pp. 16–17

A.2 COEFFICIENT FITTING AND RESIDUAL SPACE

  • Coefficient fitting uses linear or logarithmic residual space by task, refitting shared constants and estimating declared per-group parameters separately when needed.A refit is retained only if it does not reduce residual-space R2; groups with fewer than eight usable records are omitted.Source: §A.2 · p. 17

A.3 DEBIASED LOCAL RESIDUAL BOOTSTRAP

  • The residual bootstrap subtracts a local mean trend before sampling, preserving local empirical variation and non-Gaussian residual shape without modeling the full measurement process.Inputs are standardized for neighbor distances, with positive inputs spanning more than one decade transformed logarithmically; smaller samples use the global residual pool.Source: §A.3 · p. 17

A.4 NOISE CALIBRATION AND WORLD DEFINITION

  • Noise calibration only reduces debiased residual scale, targeting residual-space R2 ≥ 0.9 without forcing every world to that exact value.For multivariate tasks, calibration inputs are resampled as observed rows to preserve empirical input correlations.Source: §A.4 · pp. 17–18

A.5 SINGLE-GROUP AND MULTI-GROUP WORLDS

  • Multi-group worlds share one hidden functional form while retaining group-specific parameters, support, residual sampling, and calibration; structural scoring evaluates the shared law and parameter pattern.Source: §A.5 · p. 18

A.6 QUERY AND EVALUATION PROTOCOL

  • Solvers adaptively choose inputs and, for multi-group tasks, groups, under a limit of ten experiment calls; out-of-support inputs are clipped and reported.Measurements accumulate in the solver’s analysis environment, while the hidden law, fitted quantities, and residual sampler remain inaccessible.Source: §A.6 · p. 18
  • SCILAWS-PARALLEL is evaluated by ordinal structural recovery against the hidden law, not by numeric fit or scientific-validity scores.Source: §A.6 · p. 18

B EXPERIMENTAL PROTOCOL … B.4 INFERENCE-TIME SEARCH AND SELECTION

The experimental protocol standardizes model interaction and decoding while distinguishing fixed-data discovery from active querying. It also evaluates inference-time candidate selection against metric-specific oracles.

B.1 BASE-MODEL DECODING SETTINGS

  • The protocol fixes the interface, inputs, interaction budget, and scoring, while standardizing reasoning effort and decoding settings where providers allow.Reasoning models use the medium tier, a 65,536-token completion budget, and provider-default temperature; GPT-4o-mini uses temperature=0.4 and an 8,192-token cap.Source: §B.1 · p. 19

B.2 AGENT HARNESS CONFIGURATION

  • Agents use one of three XML tools per turn for up to 30 turns, then receive a forced submission turn and one retry; unsubmitted tasks score zero.The tools are <python>, <experiment> for PARALLEL, and <final_formula>; multiple tool blocks in a response execute only the first.Source: §B.2 · p. 19
  • The Python sandbox blocks filesystem, network, and introspection access, limits each call to 100 seconds, and returns API failures within the loop rather than ending the task.Only whitelisted imports are available, and held-out data or hidden generators cannot be accessed except through the task tools.Source: §B.2 · p. 19

B.3 SETTING-SPECIFIC PROTOCOL

  • REAL agents receive the fixed training split and held-out input ranges, with task splits spanning interpolation, extrapolation, temporal and cross-condition holdouts, and unseen-group transfer.Source: §B.3 · p. 19
  • PARALLEL agents begin without observations and must query at least once, with caps yielding roughly ntrain distinct design points; repeated samples add noise averaging, not coverage.Returned batches are previewed as 10 rows plus statistics, while the full batch remains available in train_df.Source: §B.3 · p. 20

B.4 INFERENCE-TIME SEARCH AND SELECTION

  • Across 118 REAL tasks, GPT-5.4-mini generates 20 candidates; self-selection yields N=5 from four group winners and N=20 from a second round.N=1 is the standard single-trajectory result; selectors see the task, training data, and candidate programs, but not held-out data or official evaluator feedback.Source: §B.4 · p. 20
  • Metric-specific oracles use the expected best score from a random five-candidate subset for N=5 and the pool maximum for N=20, separately for SN, SV, and joint balance.Source: §B.4 · p. 20
  • SN and SV are normalized to each task's candidate-pool range; N=1 uses that range clipped to [0, 1], and the joint oracle maximizes J.Candidates are Pareto-compared on SN and SV; Figure 7 confidence intervals use 50,000 task-bootstrap replicates.Source: §B.4 · p. 20

C EVALUATION PROTOCOL AND VERBATIM JUDGE PROMPTS … D.3 METRICS

The appendices define separate scoring protocols for observed-data prediction, scientific validity, and hidden-law recovery, then describe how the LLM judges were validated against experts.

C.5 SCILAWS-PARALLEL STRUCTURE-RECOVERY JUDGE

  • The protocol separates REAL’s numeric and validity scores from PARALLEL’s five-level structure_score for hidden-law recovery, which is not based on prediction performance.The structure judge accepts algebraically equivalent forms, penalizes missing mechanism terms, and assigns zero for contract failures or invalid outputs.Source: C EVALUATION PROTOCOL AND VERBATIM… · p. 21, §C.5 · pp. 23–24

C.1 SCILAWS-REAL REFERENCE-RELATIVE NUMERIC SCORING

  • In SCILAWS-REAL, numeric_score compares held-out fit with the strongest collected published formula; the strongest reference scores 0.5 and a perfect predictor scores 1.0.Tasks use one primary raw metric, with lower values better for RMSE, SMAPE, or log-MAE and higher values better for R²; invalid or non-finite predictions receive zero.Source: §C.1 · p. 21

C.2 SCILAWS-REAL VALIDITY JUDGE

  • SCILAWS-REAL validity_score separately assesses scientific behavior and form, using rubric checks rather than coefficient accuracy.A code-executing judge evaluates controlled probes; task scores are the fraction of applicable rubric items satisfied, and the strict aggregate treats missing or errored scores as zero.Source: §C.2 · pp. 21–22

C.3 SCILAWS-REAL ANTI-HACKING RUBRIC

  • An anti-hacking rubric flags code that hides extra fitting capacity outside the declared law interface, and a failed verdict hard-gates that task’s validity_score to zero.Examples include lookup tables, large arrays, copied data aggregates, excessive constants, and too many locally fitted parameters.Source: §C.3 · p. 22

C.4 FIT–VALIDITY CONCORDANCE

  • Fit–validity concordance compares paired model rankings; at τ = 0 it includes 3,616 task–model pairs, with 50% representing no directional association.Confidence intervals use 20,000 task-level bootstrap samples, retaining eligible model pairs within each sampled task.Source: §C.4 · pp. 22–23

D HUMAN VALIDATION OF THE LLM JUDGES

  • Judge–human agreement on all three dimensions falls within the range of expert–expert agreement, supporting use of the judges to score the benchmark at scale.The comparison covers validity, parallel structure, and cold-recall memorization judgments.Source: D HUMAN VALIDATION OF THE LLM JUDGES · p. 24, Table 3 · p. 25

D.1 SAMPLING DESIGN; D.2 EXPERTS AND PROTOCOL

  • Validation sampled 135 items per judge dimension, equally across nine models and judge-output strata, then had five blinded domain experts independently relabel every item.Because the sample was stratified rather than prevalence-weighted, the reported agreement describes reliability on the validation sample, which deliberately covers common and borderline cases.Source: §D.1 · p. 24, §D.2 · p. 24

D.3 METRICS

  • The validation reports raw agreement and Cohen’s κ, using quadratic-weighted κ for ordinal structure scores and comparing judge agreement with pairwise expert agreement.Judge κ values include 95% bootstrap confidence intervals, interpreted against the expert–expert range.Source: §D.3 · p. 25

D.4 RESULTS … E.3 STRUCTURAL JUDGE AND VERDICT

The judge’s agreement with human annotations was comparable to human–human agreement across all three dimensions. The memorization audit tests whether models can reproduce benchmark reference formulas from a restricted, data-free task description.

D.4 RESULTS

  • Exact agreement was 89% for memorization, 92% for validity, and 66.7% for the five-level structure score; each expert’s κ with the judge was 0.66–0.88.The judge’s disagreement with the human consensus was comparable in size to disagreement among human annotators.Source: §D.4 · p. 25
  • Judge–human agreement fell within human–human ranges for memorization (κ = 0.77; 0.62–0.88), validity (κ = 0.82; 0.82–0.91), and structure (quadratic-weighted κ = 0.84; 0.74–0.90).The figure compares judge–human κ with the range of pairwise human–human κ values for each dimension.Source: §D.4 · p. 25, Figure 9 · p. 26
  • 98.5% of 135 structure-score items were within one level, and only two differed by more than one, so quadratic-weighted κ captures near-matches that exact agreement misses.Most judge–human disagreements were on adjacent structure-score levels.Source: §D.4 · p. 25, Figure 10 · p. 26

E.1 PURPOSE AND DECISION UNIT

  • The audit separates recall of a published reference formula from discovery using task records, defining each decision unit as a task-model cell evaluated against the task’s best-baseline reference.The target is the published reference that the main benchmark asks a method to beat; recalling a weaker sister formula does not explain exceeding that anchor.Source: §E.1 · p. 28
  • The latest audit panel contains 118 SCILAWS-REAL tasks and nine models from six vendors, yielding 1,062 task-model cells across 66 single-group and 52 multi-group tasks.Source: §E.1 · p. 28

E.2 COLD-RECALL PROBE

  • The cold-recall probe supplies target and ordered-input metadata but no data rows, citation, or paper text, and samples five completions per task-model cell at temperature 0.8.Models are asked to use only their own knowledge to write the standard closed-form relationship as a Python predict(X) function.Source: §E.2 · p. 28

E.3 STRUCTURAL JUDGE AND VERDICT

  • A calibrated gpt-4.1 judge at temperature 0 treats coefficients and algebraic rearrangements as equivalent, but distinguishes forms by exponents, term count, variables, and functional family.For example, a linear combination does not match a power law, and a Gompertz form does not match a logistic form.Source: §E.3 · p. 28
  • A task-model cell is cold-recalled when at least three of five samples match the reference; Figure 12 identifies a 56-task discovery moat and a 14-task universal canon.All other cells are classified as not cold-recalled under the binary protocol.Source: §E.3 · p. 28, Figure 12 · p. 29, Table 4 · p. 29

E.4 MAIN RESULTS · E.5 TYPE AND DOMAIN FINDINGS

Cold recall is limited and uneven across tasks, models, task structures, and scientific domains. Multi-group tasks and some domains are more readily recalled, while nearly half of tasks are recalled by no audited model.

E.4 MAIN RESULTS

  • 30.7% of task-model cells are cold-recalled across 118 tasks and nine models, while 69.3% are not cold-recalled.Source: §E.4 · p. 29
  • Cold recall ranges from 15.3% for gpt-4o-mini to the mid-30%s for the strongest models, but the trend is not monotone across vendors.glm-5.2 and qwen3.7-max each reach 35.6%, slightly above gpt-5.5 at 34.7%; even the highest observed rate leaves roughly two-thirds of tasks unrecalled.Source: §E.4 · p. 29
  • 56 tasks (47.5%) are cold-recalled by no audited model, while 14 (11.9%) are cold-recalled by all nine models.Source: §E.4 · p. 29
  • The 14 tasks cold-recalled by all nine models include AFM spherical indentation, Cepheid period-luminosity, and WGS84/Somigliana gravity.Source: §E.4 · p. 29

E.5 TYPE AND DOMAIN FINDINGS

  • Multi-group tasks have higher cold recall than single-group tasks: 40.2% versus 23.2% across task-model cells.Source: §E.5 · p. 29, Figure 13 · p. 30
  • Materials & Engineering has the lowest mean cold-recall rate at 20.5%, while Ecology & Hydrology and Social Sciences lead at 43.4% and 40.5%.gpt-5.5 follows the broad pattern and cold-recalls 59.1% of Ecology & Hydrology tasks.Source: Figure 14 · p. 30, §E.5 · p. 30
  • Multi-group tasks often involve named cross-group laws or invariant-constant forms, which are more likely to appear as canonical closed forms in training corpora.Source: §E.5 · p. 30

E.6 TARGET-CHOICE ROBUSTNESS … G.2 REAL × MULTI-GROUP: CO2 ADSORPTION

This section examines robustness to recall-target choices and presents case studies of real single- and multi-group law-discovery tasks. It documents solver protocols, task-specific data, submissions, scoring, and discovery traces illustrating unstable extrapolation and transfer of a shared adsorption form to unseen clusters.

E.6 TARGET-CHOICE ROBUSTNESS

  • With an R2 ≥0.9 relaxed target, only 16 of 118 tasks have multiple valid baselines; all-model cold recall rises 0.85 percentage points, gpt-5.5 is unchanged, and eight cells flip across three tasks.The audit concludes the best-baseline anchor does not materially hide recall.Source: §E.6 · p. 30

E.7 AI-FEYNMAN CALIBRATION

  • Under the controlled eight-model audit, cold recall is 55.9% for AI-Feynman versus 26.8% for SCILAWS-REAL; gpt-5.5 recalls 71.0% versus 34.7%, respectively.The audit uses the same prompt, judge, and five-sample rule; the authors describe AI-Feynman as intentionally textbook-like and the probe as neither too lax nor too strict.Source: §E.7 · p. 31

E.8 INTEGRITY CHECKS FOR THE CAPABILITY LADDER

  • The capability-ladder audit finds fixed release-order ranking, local reversals, 21,888 logged calls, no empty subject-model outputs, and raw-row reproduction of the reported results.These checks address plotting, logging, completion, and judging artifacts, not direct access to training data; the authors interpret the ladder as reflecting differences in formula recall under benchmark-visible information.Source: §E.8 · p. 31

F.1 SCILAWS-REAL SYSTEM PROMPT

  • The verbatim templates fill placeholders from task metadata and use mutually exclusive group clauses; REAL system prompts preload data for iterative Python fitting and require a final formula.For multi-group REAL tasks, one shared functional form must transfer across clusters while a few parameters are refit per cluster.Source: F VERBATIM SOLVER PROMPT TEMPLATES · p. 32, §F.1 · p. 32

F.2 SCILAWS-REAL USER PROMPT

  • REAL user prompts specify held-out evaluation by a metric against strong published references and ask for a physically reasonable formula that extrapolates sensibly.The prompt also provides training-data details and a maximum turn budget.Source: §F.2 · pp. 33–34

F.3 SCILAWS-PARALLEL SYSTEM PROMPT

  • PARALLEL system prompts start without training observations, require simulator probes before formula submission, and then allow Python analysis of collected data.For multi-group tasks, the hidden mechanism structure must be shared across clusters, with only LOCAL_FITTABLE parameters varying.Source: §F.3 · p. 34

F.4 SCILAWS-PARALLEL USER PROMPT

  • PARALLEL user prompts define structure_score as the primary objective for matching functional form up to algebraic equivalence, with zero initial observations and a bounded turn budget.The prompt explicitly says not to optimize for held-out prediction metrics.Source: §F.4 · pp. 35–36

G CASE STUDIES

  • The case-study appendix catalogs favorable and unfavorable instances of each of three evaluation scores across eight informative tasks and introduces four complete reference trajectories spanning both benchmark design axes.The supplied passage does not specify the trajectory findings.Source: G CASE STUDIES · p. 36

G.1 REAL × SINGLE-GROUP: THE PROTON FORM FACTOR

  • On the proton form factor, GPT-5.4-mini scores S_N = 0.436 (best of nine) and S_V = 1.0, yet remains below the 8-constant Arrington reference.Candidate forms had training RMSEs near 0.012 but predictions at Q^2 = 5.85 ranged from 0.088 to 1.02; the agent submitted a saturating exponential-over-quadratic form.Source: §G.1 · p. 39

G.2 REAL × MULTI-GROUP: CO2 ADSORPTION

  • S_N = 0.534 for frontier models: a shared Toth form transfers to unseen CO2-adsorption clusters and scores above the reference.In initial cluster fits, Sips and Toth achieved mean RMSEs of 0.1139 and 0.1142, respectively, versus 0.2593 for Langmuir and 0.3483 for Freundlich; the submitted Toth form fits three local parameters per cluster.Source: §G.2 · pp. 39–40

G.3 PARALLEL × SINGLE-GROUP: THE KEELING CURVE

In the parallel Keeling Curve task, the submitted quadratic tracked the observations but failed to match the hidden exponential-accumulation structure. On the Real version, a cubic trend with three annual harmonics scored higher than the NOAA reference.

G.3 PARALLEL × SINGLE-GROUP: THE KEELING CURVE

  • The parallel task began with no preloaded observations and required the agent to query atmospheric CO2 c against decimal year t from 1958 to 2026.The submission allowed at most 12 global constants and was scored on whether its form matched the hidden simulator’s structure.Source: §G.3 · p. 40
  • The agent initially designed 20 measurement times with three replicates each, yielding 60 noisy observations, then made six more experiment calls for denser grids and replicates.Source: §G.3 · p. 40
  • S_S = 0.25: the submitted quadratic tracked the data but did not match the hidden exponential-accumulation family.The agent fit an exponential-accumulation trend but ultimately submitted the quadratic.Source: §G.3 · pp. 40–41
  • On the SciLaws-Real version, a cubic trend plus three annual harmonics scored S_N = 0.78, beating the 12-constant NOAA reference.Source: §G.3 · p. 41

G.4 PARALLEL × MULTI-GROUP: SELLMEIER DISPERSION

The model actively sampled a parallel Sellmeier-dispersion task and exactly recovered its hidden law; on the real version, its score was below the strongest published baseline.

G.4 PARALLEL × MULTI-GROUP: SELLMEIER DISPERSION

  • The parallel task began with no preloaded rows and required querying 14 crystal clusters, using one shared form with at most four per-cluster parameters and no global fitted constants.The score assessed whether the submitted form matched the hidden simulator’s structure, up to algebraic equivalence.Source: §G.4 · p. 41
  • After five additional experiment rounds refining the wavelength grid and adding clusters, the agent collected 158 rows across all 14 groups.Source: §G.4 · p. 41
  • Adding an IR term to the single-pole Sellmeier form lowered max error to ~1e-4, the best result among the families tried.The agent first compared Cauchy and single-pole forms, then tried n^2 = A + B/(lambda^2-C) + D*lambda^2.Source: §G.4 · pp. 41–42
  • S_S = 1.0: the model exactly recovered the hidden n^2 = A + B/(l^2-C) + D*l^2 family, with no missed structure.The submitted form fits the parameters separately for each crystal cluster.Source: §G.4 · p. 42
  • S_N = 0.21 on the SciLaws-Real version, below the strongest published baseline, the 4-term Cauchy equation.Source: §G.4 · p. 42

Related papers

Loading 2609.01552v1…