NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents

1 INTRODUCTION

• Existing benchmark paradigms trade off scientific relevance, scalability, and resistance to memorization, while static function fitting omits interactive exploration of complex systems.OOD laws are difficult to scale, transformed laws remain vulnerable to recall, and synthetic laws lack scientific relevance.Source: §1 · p. 2
• Counterfactual law shifts alter canonical physical laws to create scalable, scientifically grounded problems that resist recall and require reasoning from first principles.Alterations can modify mathematical operators or exponents.Source: §1 · p. 2
• Agents must choose experimental inputs and interpret feedback to discover a target law embedded among confounding variables in a complex model system.This replaces fitting formulas to pre-existing tables with interactive model discovery.Source: §1 · pp. 2–3
• The benchmark independently varies target-law complexity and surrounding-system complexity, and offers code execution for tasks such as regression or hypothesis testing.The code interface is intended to reduce computational limitations as a primary bottleneck.Source: §1 · p. 3
• LLM performance degrades sharply as law or system complexity increases, and even minimal observational noise causes a sharp decline in symbolic accuracy.Code assistance boosts weaker models but can hinder stronger ones when they shift prematurely from exploration to local optimization.Source: §1 · p. 3

2 TASK FORMALIZATION

• An equation maps input variables to a numerical output and is represented structurally as an expression tree.The tree has operands at its leaves and mathematical operators at internal nodes.Source: §2 · p. 3
• A model is an ordered sequence of equations that takes system-level inputs and produces designated final outputs.An equation may use primary inputs or outputs from preceding equations.Source: §2 · p. 3
• One equation is designated as the hidden target law, while the other equations are provided as known assisting information.Agents use those assisting equations to discover the target.Source: §2 · p. 3
• Tasks range from a Vanilla Equation containing only the target law to Simple System and Complex System settings that embed it in a larger model.In the larger systems, agents must use assisting equations to disentangle confounding variables.Source: §2 · pp. 3–4

3.1 PHYSICAL LAW CURATION

• 108 shifted laws from 12 canonical laws span easy, medium, and hard mutation levels, curated by three domain experts with constants adjusted to preserve dimensional consistency.Every target law includes a constant that can compensate for dimensional changes caused by mutations.Source: §3.1 · pp. 4–5

3.2 VIRTUAL ENVIRONMENT & TASKS

• The 108 laws yield 324 tasks across three systems; agents probe models and use assisting equations to isolate the target, with optional Python computation.Code assistance is intended to offload computation such as regression or hypothesis verification.Source: §3.2 · p. 5

3.3 EVALUATION METRICS

• Symbolic Accuracy checks mathematical equivalence with an LLM judge achieving 98.3% human agreement, while RMSLE measures predictive fidelity for quantities spanning multiple orders of magnitude.RMSLE is adopted instead of NMSE used in some prior studies.Source: §3.3 · pp. 5–6

4.1 CAN LLM AGENTS PERFORM GENERALIZABLE SCIENTIFIC LAW DISCOVERY?

• All three non-thinking LLMs score below 10% overall; even their 20–40% accuracy in the simplest setting degrades in other configurations.The authors report that benchmark success depends on strong, generalizable reasoning.Source: §4.1 · p. 7
• GPT-5 and Gemini-2.5-pro score 75.9% and 65.4% overall, but only 40.3% and 16.7% in the hardest setting.Most reasoning models achieve 30–80% overall accuracy, while other reasoning models fall below 6% in the hardest setting.Source: §4.1 · p. 7

4.5 THE PARADOX OF CODE ASSISTANCE: THE EXPLORATION-EXPLOITATION TRADE-OFF

• Code improves less capable models (SA < 40%) but slightly degrades more capable models (SA ≥40%).The performance divergence is most pronounced when the code budget increases from zero to one.Source: §4.1 · p. 7, §4.5 · p. 8, Figure 5 · p. 9

4.2 SENSITIVITY TO OBSERVATIONAL NOISE

• At noise 0.0001, accuracy falls 12–16% versus noise-free; increasing noise to 0.1 further reduces symbolic accuracy while RMSLE remains relatively stable.Baseline and code-assisted agents degrade similarly as noise increases.Source: §4.2 · pp. 7–8, Figure 2 · p. 7

4.3 CROSS-DOMAIN PERFORMANCE DISPARITIES

• Average domain accuracy spans 18%–54%; in complex systems, Heat Transfer reaches 3.3% accuracy versus Acoustic Velocity’s 45.0%.Bose-Einstein Distribution has the lowest reported average accuracy, 18.1%.Source: Figure 3 · p. 7, §4.3 · p. 8

4.4 INFERENCE SCALING WITH TASK COMPLEXITY

• Figure 4 presents inference cost across task difficulty levels.The supplied passage identifies the comparison but gives no inference-cost values or trend.Source: Figure 4 · p. 8

4.5 THE PARADOX OF CODE ASSISTANCE: THE EXPLORATION-EXPLOITATION TRADE-OFF

• Stronger models favor function-fitting over basic calculations, and over-exploitation can make them settle prematurely at a local optimum.The authors describe code as computational help for weaker models and equation refinement for stronger ones.Source: §4.5 · p. 9, Figure 6 · p. 9

• Symbolic regression ranges from genetic programming to physics-inspired and deep-learning approaches [Udrescu & Tegmark, 2020; Udrescu et al., 2020; Petersen et al., 2021; Kamienny et al., 2022], with recent LLM equation discovery [Shojaee et al., 2025a;b; Xia et al., 2025; Chen et al., 2025a].Source: §5 · p. 9
• LLM-driven scientific discovery includes experimental support in chemistry and biomedicine [Gottweis et al., 2025; Yang et al., 2025; Luo et al., 2025; Chen et al., 2025b] and research workflows in machine learning [Chan et al., 2025].This line of work also includes efforts toward autonomous AI Scientists [Yamada et al., 2025].Source: §5 · p. 9
• The reported observations are correlational and should be treated as suggestive rather than causal.Source: §5 · p. 9
• Virtual environments for agents extend from general-purpose instruction-following, web-navigation, and planning sandboxes to specialized scientific-reasoning platforms.The passage cites Shridhar et al. (2021), Yao et al. (2023), Prasad et al. (2024), Wang et al. (2022), and Jansen et al. (2024).Source: §5 · p. 10

• NewtonBench uses counterfactual law shifts and interactive model discovery to benchmark LLMs, revealing fragile performance under complexity and noise and a code-assistance trade-off for stronger models.The authors position robust, generalizable discovery in complex interactive environments as the core challenge.Source: §6 · p. 10

ETHICS STATEMENT

• NewtonBench is a contamination-free virtual benchmark for evaluating LLM agents, not a training dataset for developing physics-discovery models.Its shifted laws are intended as evaluation tasks, not training examples for discovering real-world physics.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• Evaluations used public APIs, cost an estimated $10,000, and are accompanied by released prompts, implementations, code, and data.The authors also report statistical significance measures and provide model configurations for reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 LLM DETAILS

• The study evaluates 11 state-of-the-art models and distinguishes reasoning models by whether deliberate thinking is triggered before the first answer token.Experiments use temperature 0.4, four independent runs, and up to 10 rounds of 20 input sets per test case.Source: §A.1 · p. 17

A.2 MODEL SYSTEMS

• Human-curated model systems inspired by classical experiments embed target laws in interactive simulations; the acoustics example asks agents to infer sound speed through echo-time feedback.The simulation’s hidden computation sequence includes a target law and an assisting equation whose underlying principle is given to the agent.Source: §A.2 · p. 18, SIMPLE SYSTEM (LAW OF SOUND SPEED IN… · p. 18

SYSTEM COMPLEXITY ANALYSIS

• System complexity is measured by the number of assisting equations and their total operator count, and complex systems have more of both.These measures exclude the target law; Table 3 compares the counts across simple and complex systems in all 12 domains.Source: SYSTEM COMPLEXITY ANALYSIS · p. 19, Table 3 · p. 19

A.3.1 EXPRESSION TREES

• Expression trees are evaluated recursively in post-order, applying each operator after evaluating its children; each node is visited once, giving O(|n|) time.The benchmark also specifies the symbol and arity of supported operators.Source: §A.3.1 · p. 20, §A.3.2 · p. 21

A.3.3 PHYSICAL LAWS

• Expert-curated counterfactual shifts mutate canonical laws without adding variables, increase equation complexity, and avoid reversals or collapsible mutation sequences.All shifted laws are dimensionally coherent, but some may be physically implausible and are stress tests rather than claims about real-world physics.Source: §A.3.3 · p. 22

A.3.4 CATEGORIZATION ON PHYSICAL PLAUSIBILITY

• A three-level taxonomy grades shifted laws from structurally admissible variants, through minor violations, to major violations such as new singularities or loss of real, positive outputs.Level 1 retains plausible structural patterns, Level 2 breaks patterns such as symmetry, and Level 3 includes major structural distortions.Source: §A.3.4 · pp. 23–24

A.4.1 SYMBOLIC ACCURACY EVALUATION DETAILS

• Symbolic equivalence is judged with a prompted, few-shot LLM-as-a-Judge; the authors compared four models on 120 human-labeled equation pairs and selected Nemotron-ultra-253b.The selection was based on performance and open-source accessibility; Table 8 reports agreement with human evaluators.Source: §A.4.1 · p. 27, Table 8 · p. 27
• The judge’s examples treat equivalent equations as matches despite changes in variable names, constants, or algebraic form, but reject altered exponents or variable placement.The supplied examples illustrate the intended equivalence decisions rather than reporting benchmark performance.Source: §A.4.1 · p. 28

A.4.2 DATA FIDELITY EVALUATION DETAILS

• Data fidelity is assessed on 5,000 separate samples per equation, with curated nonzero outputs, nonnegative ground-truth outputs, and Modified Z-Score outlier filtering.Sampling distributions follow Matsubara et al. (2024); the physical-constant notion is only a heuristic for RMSLE sampling.Source: §A.4.2 · p. 29, Table 9 · p. 29

Style Robustness Analysis

• The LLM-as-a-Judge consistently recognizes symbolic equivalence across substantial structural and syntactic variation, supporting stylistically robust evaluation.Table 10 presents the style-robustness analysis.Source: §A.5 · p. 30, Table 10 · p. 30

Error Analysis

• GPT-4.1 achieved 98% judge accuracy, with one false positive and one false negative; errors involved log-base conflation and missed exponential–logarithmic cancellation.The false positive conflated natural and base-10 logarithms, while the false negative involved cancellation with additive terms.Source: §A.5 · p. 31

Impact of Few-shot Learning

• For GPT-4.1, judge accuracy improves monotonically as the number of in-context demonstrations increases.Table 12 reports the few-shot demonstration analysis.Source: §A.5 · p. 31, Table 12 · p. 31

B.1 RESULTS FOR INDIVIDUAL DOMAINS

• Full LLM performance figures cover gravitation (Newton’s law of universal gravitation), electrostatics (Coulomb’s law), magnetostatics (Ampère’s force law), and thermal conduction (Fourier’s law).These are Figures 8–11.Source: §B.1 · p. 32
• The figure set also covers geometrical optics (Snell’s law), nuclear physics (radioactive decay), oscillations (damped harmonic motion), and physical optics (Malus’s law).These are Figures 12–15.Source: §B.1 · p. 32
• The remaining full performance figures cover acoustics (sound speed in an ideal gas), elasticity (Hooke’s law), statistical mechanics (Bose–Einstein distribution), and calorimetry (heat transfer).These are Figures 16–19.Source: §B.1 · p. 32

B.2 RESULTS OF NON-PERFORMANCE METRICS

• Tables 13–16 report average rounds, experiments, total tokens, and tokens per round, aggregated across difficulty levels and 12 domains.The supplied passages provide table scopes but no cell values.Source: Table 13 · p. 46, Table 14 · p. 47, Table 15 · p. 48

C.1 GENERAL PROMPTS

• The general prompt asks agents to design experiments, analyze returned data, verify hypotheses, and submit a discovered law within a turn-based protocol.Some prompt variants permit Python calculations; the workflow also limits agents to 10 experiment rounds.Source: §C.1 · p. 51

C.2 DOMAIN-SPECIFIC PROMPTS

• Domain-specific prompt templates are provided in the supplementary materials.Source: §C.2 · p. 52

D.1.1 VANILLA NON-REASONING LLMS

• The model ran 20 experiments varying mass1, mass2, and distance across many orders of magnitude, yet despite collecting extensive data defaulted to inverse-square, missing the distance exponent of 1.5.The passage attributes this to an inability to perform precise computation and reliance on a known law.Source: §D.1.1 · p. 54

D.1.2 CODE-ASSISTED NON-REASONING LLMS

• With Python assistance, GPT-4.1 identifies the distance exponent as 1.5 and submits the correct gravitational law.A numerical search over candidate exponents yields the best-fit value.Source: §D.1.2 · pp. 55–56

D.2.1 VANILLA REASONING LLMS

• From the data, the model deduces a non-integer exponent of 2.5, estimating x=2.509 at c=2 and x=2.5028 at c=100, then proposes Q_total = m * delta_T * c^2.5.Across points, the fitted constant K averages 1.0030306, described as very close to 1.Source: §D.2.1 · pp. 57–58

D.2.2 CODE-ASSISTED REASONING LLMS

• With code assistance, Gemini-2.5-flash repeatedly tests simple integer-power formulas, then returns to its incorrect initial calorimetry hypothesis.The case study describes the tool use as narrowing the model’s strategy to easily programmed brute-force checks, after which it abandons data-driven analysis.Source: §D.2.2 · p. 60

E.1 MEMORIZATION IN LSR-TRANSFORM

• On LSR-Transform, newer LLMs can infer transformed laws without observed data, whereas all models achieve zero accuracy on NewtonBench DataBlind tasks even with a shift hint.The authors present the contrast as evidence that NewtonBench is effectively memorization-free.Source: §E.1 · p. 61, Table 17 · p. 61, Table 18 · p. 61

E.2 SOLVABILITY PROOF

• The solvability argument reduces system discovery to a standard black-box function-identification problem, then establishes finite identifiability under design assumptions A0–A4.Source: §E.2 · p. 63

E.2.1 PROBLEM SETTING AND ASSUMPTIONS

• The proof assumes noiseless deterministic outputs, an invertible assisting path, reachable target inputs, a finite nondegenerate candidate-law family, and a finite separating set.The candidate family is grammar-bounded, and its symbolic structures and parameters are unique under the stated conditions.Source: §E.2.1 · pp. 63–64

E.2.2 PROOF OF SOLVABILITY

• A finite set of experiments suffices to uniquely determine both the target law’s symbolic structure and its numerical constants.The proof reduces system discovery to function identification, using a separating set of target inputs.Source: §E.2.2 · p. 64

E.2.3 CONCLUSION

• Finite solvability is conditional on assumptions A0–A4; relaxing conditions such as invertible assisting paths or reachable target inputs may remove the guarantee.Source: §E.2.3 · p. 65

E.3 PHYSICS DOMAIN ANALYSIS

• Performance generally decreases as physical concepts become less tangible, while relationships with the other analyzed domain metrics are less apparent.The authors describe the inverse relationship with abstractness as potential.Source: §E.3 · p. 66, Table 19 · p. 66

E.4 EXPLORATION AND EXPLOITATION TOKENS

• The exploration rate is a token-frequency heuristic based on exploration and exploitation word sets, offered as suggestive rather than definitive behavioral evidence.The rate divides exploration-word frequency by the sum of exploration- and exploitation-word frequencies.Source: §E.4 · p. 67, Table 20 · p. 67

E.5 SYMBOLIC REGRESSION BASELINES

• PySR matches or exceeds non-reasoning LLMs in symbolic accuracy but falls below reasoning models; its RMSLE is competitive on easy equations and fifth-best on hard equations.On easy equations, it exceeds every LLM except Gemini-2.5-Pro in RMSLE.Source: §E.5 · p. 68, Figure 21 · p. 68
• After 2,500 iterations, LLM-SR marginally surpasses PySR in accuracy and RMSLE, but underperforms the code-assisted agent in accuracy despite substantially higher computational costs.LLM-SR testing was limited to the Law of Sound Speed because one inference requires 10,000 API calls.Source: §E.5 · p. 69, Table 21 · p. 69
• Symbolic regression methods achieve stronger RMSLE through iterative optimization but remain sensitive to equation complexity and underperform the agent in symbolic accuracy despite greater computational resources.LLM-SR marginally surpasses PySR on accuracy and RMSLE, yet remains below the code-assisted agent in accuracy.Source: §E.5 · p. 69

E.6 FURTHER DISCUSSIONS

• NEWTONBENCH models discovery as active variable isolation and abstraction from observed physical behavior, connecting classical experiments with modern high-dimensional, probabilistic analysis.The discussion cites Thomson’s cathode-ray work and Higgs-boson reconstruction at the LHC as contrasting scientific paradigms sharing the challenge of deriving parsimonious mathematical descriptions.Source: §E.6 · p. 70
• The benchmark evaluates strategic manipulation and law inference within a bounded system, not the creative design or construction of experimental apparatus.Real-world inquiry often requires researchers to conceive and build apparatus to isolate phenomena.Source: §E.6 · p. 70
• Future benchmarks should add first-principles model construction, information-seeking under noisy and constrained conditions, and procedural control of laboratory apparatus.The paper presents NEWTONBENCH as a controlled starting point for developing agents capable of navigating these more demanding frontiers.Source: §E.6 · p. 70

E.7 USE OF LARGE LANGUAGE MODELS

• The authors used LLMs only for grammar checking and image-generation models to enhance figure presentation.This is the manuscript’s stated use of language and image-generation models.Source: §E.7 · p. 71