PaperlayerView exact revision on arXiv

Source-linked AI summary

PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research

Qituan Zhang, Guo Chen, Linfeng Zhang, Wenlong Zhang

arXiv:2610.32575v1cs.CL

TL;DR

Existing evaluations leave gaps in testing analytical validity and reasoning after theoretical conditions change. PHYRD addresses these with Reproduce and Discover tasks built through FormulaForest. Across both tracks, models often complete individual derivations without meeting every problem requirement, while most tested models improve complete Discover performance with tools.

  • Problem

    Existing evaluations leave limited evidence about models’ use of external information and computational feedback, analytical validity, and derivation after theoretical conditions change.

    Source: §1 · p. 1

  • Method

    PHYRD pairs 63 Reproduce problems on published-result derivations with 43 Discover problems on modified conditions, using FormulaForest reference solutions and subquestion scoring.

    Source: §1 · pp. 1–2

  • Results

    Across both tracks, models often complete individual derivations without satisfying every research-problem requirement; four of five solve more Discover problems completely with tools.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    PHYRD reports subquestion and whole-problem accuracy separately, distinguishing local derivation success from completion of the full problem.

    Source: §1 · p. 2

    Uneven subfield coverage limits domain comparisons.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Large language models and agents have shown significant power in advanced physical research. In this paper, We introduce PHYRD, a living benchmark for theoretical-physics reasoning with 63 Reproduce and 43 Discover problems. Reproduce tests derivation of published results; Discover tests consequences of modified theoretical conditions. FormulaForest constructs the tasks with reference solutions and subquestion-level rubrics. Across five models, whole-problem accuracy ranges from 49.21% to 68.25% on Reproduce and from 18.60% to 74.42% on closed-book Discover. With tools, four models complete more Discover problems, but conceptual errors and missing arguments persist. Besides, our experimental results demonstrate the value of FormulaForest as synthetic data. On FrontierScience, a benchmark developed by OpenAI, it raises the accuracy of a Qwen-35B-A3B model from 61% to 72%.

1 Introduction

PHYRD addresses gaps in evaluating whether models can use information and computational feedback while preserving analytical validity, and whether they can derive consequences when theoretical conditions change. It pairs these tasks with verified reference solutions and reports that tool-assisted Discover improves complete solutions for most tested models, while errors persist.

1 Introduction

  • Existing benchmarks leave limited evidence about models’ use of external information and computational feedback, analytical validity, and derivation after theoretical conditions change.Execution or numerical agreement alone does not establish a valid analytical argument, and models must determine which arguments fail when premises change.Source: §1 · p. 1
  • PHYRD contains 63 Reproduce and 43 Discover problems, testing published-result derivations and consequences of modified theoretical conditions, respectively.Discover specifies what changes relative to the source study and which premises remain fixed.Source: §1 · pp. 1–2
  • Table 1 compares representative benchmarks’ task and design scope, including modified-condition derivation and continued expansion.Its caption defines modified-condition derivation as examining consequences of changed premises and identifies release plans for PRL-Bench and PHYRD.Source: §1 · pp. 1–2, Table 1 · p. 2
  • FormulaForest organizes source-paper formulas, definitions, and assumptions into condition-bearing mathematical dependencies that guide problem statements, reference solutions, and scoring criteria.Reference derivations undergo source verification, analytical review, and computational checks where applicable.Source: §1 · p. 2
  • Four of five models solve more Discover problems completely with tools, but conceptual errors remain similar and missing arguments still prevent complete solutions.Calculation errors decrease substantially in paired tool-assisted responses; DeepSeek and Qwen each gain 20.93 percentage points in whole-problem accuracy.Source: §1 · p. 2

2 Related Work

Related benchmarks cover competition mathematics, scientific questions, research physics, and tool-assisted workflows, but differ in whether they assess analytical derivations under changed conditions. Other work uses FormulaForest’s Reproduce pipeline for training rather than physics-task evaluation.

2 Related Work

  • MATH, SciBench, GPQA, TheoremQA, FrontierMath, HorizonMath, TPBench, and PRL-Bench span competition mathematics, scientific problems, theorem application, open mathematics, and research physics.Their task scopes include undergraduate and research physics, difficult or open mathematical problems, and tasks built from recent physics papers.Source: §2 · p. 2
  • Tool-assisted research benchmarks assess programming, data-driven workflows, hypothesis discovery, and computational reproduction, whereas PHYRD focuses on analytical derivations under source and modified theoretical conditions.ReAct combines reasoning with actions, and PAL uses generated programs for calculation.Source: §2 · p. 2
  • LiveBench and MathArena introduce tasks over time, while ProcessBench tests stepwise error identification; PHYRD also scores explicit subquestion requirements and complete solutions separately.The paper notes that model judgments can depend on response position and verbosity.Source: §2 · p. 2
  • A related study uses SciDeriv from FormulaForest’s Reproduce pipeline for cross-tokenizer distillation, while this paper contributes physics-task construction and evaluation.The related study introduces the ESCD objective.Source: §2 · p. 2
  • Figure 1 contrasts deriving a published soft-graviton theorem with analyzing spectral collisions in a modified matrix family.The figure shows selected tasks and reference results; complete statements and selected derivations appear in Appendix A.3 and A.4.Source: Figure 1 · p. 3

3 PHYRD Tasks and Dataset

PHYRD organizes theoretical-physics reasoning into Reproduce and Discover tracks, supplies public problems with references and scoring criteria, and plans timestamped updates that preserve prior versions. Its dataset contains 106 problems across five primary physics subfields.

3 PHYRD Tasks and Dataset

  • Each problem has a public statement, reference solution, per-subquestion scoring criteria, and source and verification materials, while solvers receive only the statement.The five primary subfields are astrophysics and gravitation, condensed matter, high-energy physics, quantum physics and information, and statistical physics.Source: §3 · p. 3
  • PHYRD contains 106 problems and 504 public subquestions: 279 in Reproduce and 225 in Discover.Because parts share premises and may depend on earlier results, the benchmark assesses complete solutions and paired changes at the problem level.Source: §3 · p. 3

3.1 Reproduce Tasks

  • Reproduce asks for derivations of published formulas and relationships under stated conditions, requiring conventions, normalization, and applicability to be handled rather than merely matching final expressions.Equivalent derivations are accepted; the track is closed book, with no source paper or external tools.Source: §3.1 · p. 3

3.2 Discover Tasks

  • Discover changes a source-paper object or condition and requires models to test which source relations still apply and derive new arguments when they do not.The task extends the source setting without requiring resolution of an open physics question.Source: §3.2 · p. 3
  • A Discover example merges two root collisions in a modified matrix family, requiring new Jordan and projector limits rather than reuse of an isolated-pair calculation.Discover is evaluated closed book and with specified symbolic-computation and retrieval tools on the same 43 problems.Source: §3.2 · p. 3

3.3 Dataset and Updates

  • HEP accounts for 39/63 Reproduce problems and 20/43 Discover problems; Quantum and Stat together contribute six and eleven problems, respectively.These counts describe dataset composition, not the breadth of the fields; the subfield taxonomy assigns each problem one primary field.Source: §3.3 · p. 3, Figure 2 · p. 4
  • The authors plan timestamped additions from new papers and will retain prior task and score versions to make comparisons traceable over time.The taxonomy uses primary research context to assign each problem a field.Source: §3.3 · pp. 3–4

4 FormulaForest Task Construction

FormulaForest builds tasks from source-paper mathematical relations and their conditions, then verifies task statements, derivations, and scoring. It constructs Reproduce tasks around published targets and Discover tasks around controlled changes to local mathematical structures.

4 FormulaForest Task Construction

  • FormulaForest constructs Reproduce tasks around published results and Discover tasks by modifying a local model or its conditions.Source: §4 · p. 4

4.1 Mathematical Relations

  • The source paper’s mathematical content is represented as a relation graph to retain definitions and validity conditions distributed across the paper.Source: §4.1 · p. 4
  • The graph records formulas and premises, derivation and specialization relations with conditions, and source locations.Source: §4.1 · p. 4
  • Candidate relations are identified through equation references and context, then checked against the source and, where needed, mathematically.An equation’s position alone does not establish a derivation, so explicit results remain distinct from inferred dependencies.Source: §4.1 · p. 4

4.2 Task Construction

  • For Reproduce, a published target is paired with the premises needed to derive it, and the reference solution fills gaps left by the source.Scoring accepts equivalent arguments.Source: §4.2 · p. 4
  • For Discover, a transformation is applied to a local mathematical structure and its conditions; changes may affect operators, couplings, symmetries, or boundary conditions.The construction records fixed and changed premises, then determines which source arguments survive and what new derivations are needed.Source: §4.2 · p. 4
  • Before release, derivations and conventions are verified, Discover’s modified model is checked for well-definedness, and blind solving tests agreement among the statement, solution, and scoring criteria.Source: §4.2 · p. 4
  • The same pipeline produced the SciDeriv training corpus used in a separate distillation study.Source: §4.2 · p. 4

5 Experiments

Across five models, Reproduce whole-problem accuracy ranges from 49.21% to 68.25%, while tool-assisted Discover yields more complete solutions for four models. Errors and training results show both persistent conceptual gaps and a downstream use for generated derivation tasks.

5.1 Evaluation and Metrics

  • Five models answer 63 Reproduce problems and the same 43 Discover problems in closed-book and tool-assisted settings, with subquestion and all-parts-correct accuracy reported.Both metrics use the full problem-set denominator.Source: §5.1 · p. 4

5.2 Reproduce Results

  • Of 315 Reproduce responses, 182 are complete, 130 pass some but not all subquestions, and three pass none; 78 incomplete responses miss just one part.Subfield ordering varies: GLM and Kimi each solve 8 of 10 Astro problems, while DeepSeek solves 27 of 39 HEP problems versus GPT's 25.Source: §5.2 · pp. 4–5, Figure 4 · p. 5
  • Reproduce whole-problem accuracy ranges from 49.21% to 68.25%, while subquestion accuracy ranges from 79.66% to 88.33%.The evaluation uses 63 Reproduce problems and reports both subquestion and whole-problem accuracy.Source: Figure 3 · p. 4, §5.2 · p. 4

5.3 Discover: Closed Book and Tools

  • With tools, four models gain complete Discover solutions—GPT 2, DeepSeek 9, Kimi 6, and Qwen 9—while GLM loses one.DeepSeek and Qwen each gain 20.93 percentage points in whole-problem accuracy; GLM's subquestion accuracy rises despite fewer complete solutions.Source: Figure 5 · p. 5, §5.3 · pp. 5–6

5.4 Error Analysis

  • Across 209 paired Discover answers, calculation errors fall from 123 without tools to 67 with tools, while conceptual errors change from 70 to 75.For Reproduce, 20 of GPT's 31 incorrect subquestions are incomplete derivations or conclusions, while 29 of GLM's 47 are calculation errors.Source: §5.4 · pp. 6–7
  • DeepSeek and Qwen each finish with nine more fully correct Discover problems, but the paired tool settings differ in interaction and computation as well as tool access.The paired error counts describe evaluated responses and do not isolate a tool-access effect.Source: §5.4 · p. 7

5.5 Training with Generated Tasks

  • The 17,577-prompt SciDeriv corpus contains 10,000 conclusion-assessment and 7,577 conclusion-reconstruction tasks, without stored reference completions or scalar reward labels.The tasks concern whether claims follow from, fit, or conflict with theoretical relations, or what conclusion those relations support.Source: §5.5 · p. 7
  • For the 1T Kimi teacher and 35B student, SFT raises base scores from 59.2 to 66.2 on PHYRD-40 and 61.0 to 68.0 on Olympiad-100; ESCD from that checkpoint reaches 74.3 and 72.0, respectively.Source: §5.5 · pp. 7–8

6 Conclusion

Models often complete local derivations without meeting every requirement of a research problem. On Discover, four of five models solve more problems completely in the tool-assisted setting, but failures remain when calculations leave theoretical conditions.

6 Conclusion

  • Four of five models solve more Discover problems completely in the tool-assisted setting.Source: §6 · p. 8
  • Across both tracks, models often complete individual derivations without satisfying every requirement of a research problem.Source: §6 · p. 8
  • Failures remain when calculations are applied outside theoretical conditions.Source: §6 · p. 8

Limitations

The paper identifies uneven subfield coverage as a limit on domain comparisons and notes that models may have encountered related published results during training.

Limitations

  • Uneven subfield coverage limits comparisons of model performance across domains.Source: Limitations · p. 8
  • Models may have encountered related published results during training.Source: Limitations · p. 8

Ethics Statement

The material states the authors’ ethics practices and describes PHYRD’s task organization through dataset details and worked Reproduce and Discover examples. These include a soft-graviton derivation and a root-collision problem whose simultaneous limit differs from an isolated-pair limit.

Ethics Statement

  • All evidence comes from publicly available sources; AI assistants supported language editing and coding, and the authors reviewed the content and take responsibility.Source: Ethics Statement · p. 8

A Dataset and Task Examples

  • The appendix describes subfield assignments and problem sizes, gives one Reproduce and one Discover example, and lists all 106 problems with sources.Source: A Dataset and Task Examples · p. 10

A.1 Primary physics subfields

  • Problems receive one primary field label based on research object and source context, with examples including celestial scattering amplitudes in HEP and black-hole thermodynamics in Astro.Source: §A.1 · p. 10

A.2 Problem size and shared context

  • Reproduce has 63 problems and 279 public subquestions, while Discover has 43 problems and 225; most problems have four or five subquestions.Source: §A.2 · p. 10
  • Subquestions share definitions and assumptions, may form successive derivation steps, and remain grouped when computing whole-problem accuracy.Source: §A.2 · p. 10

A.3 Soft-graviton example (A043)

  • Problem A043 asks for the subleading conformal soft graviton theorem from a supplied vertex, including its covariant and helicity forms, angular-momentum currents, and correlator insertion.Source: §A.3 · p. 10, §A.3.1 · p. 10

A.3.2 Selected derivation

  • The selected derivation separates the leading pole at ∆=1 from the requested residue, with both fermionic factors contributing at that order.Source: §A.3.2 · p. 11

A.4 Discover example: root collisions (p30)

  • The root-collision Discover task asks for spectral and Jordan structure, root displacements and permutations, projector limits, and resolvent behavior across isolated and simultaneous collisions.Source: §A.4 · p. 11, §A.4.1 · pp. 11–12, §5 · p. 12

A.4.2 Selected derivation

This section presents a selected derivation alongside catalogue information on Reproduce and Discover tasks, including Discover entries and the models and procedures used for closed-book evaluations.

A.4.2 Selected derivation

  • Equal characteristic polynomials do not fix Jordan blocks: the two example matrices have nullity ladders (2, 2, 2, 2) and (1, 2, 2, 2), while conjugate eigenvalues share partitions but reciprocal ones need not.Successive nullity differences count blocks of size at least each order; two nullities distinguish most multiplicity-four partitions, but not 2 + 2 from 3 + 1.Source: §A.4.2 · p. 12
  • Near the origin, collisions occur on δ+ = 0, δ− = 0, and v = 0; along analytic arcs, root-displacement orders depend on higher terms, and meridians interchange roots within each sector.The arcs (t^4, t^12) and (t^8, 2t^8) yield displacement orders (2, 6) and (4, 4); a finite root expansion cannot prove v vanishes identically along an arc.Source: §A.4.2 · p. 12
  • At an isolated double-root collision, the cyclic companion has one size-two Jordan block at each root, and the sum of approaching projectors tends to the rank-two generalized eigenspace projector despite individual divergence.This finite-sum conclusion does not extend to the four-root limit, where the other spectral cluster also reaches 1.Source: §A.4.2 · p. 13
  • In the four-root limit, sector and individual projectors diverge at arc-dependent orders, but these singularities cancel in the full resolvent for fixed x ≠ 1.The listed orders are t^-4 for sector projectors and t^-6 and t^-10 for individual projectors along the first arc, and t^-8 and t^-12 along the second; these are leading matrix asymptotics.Source: §A.4.2 · p. 13
  • At the three centers, rank(T − I) is 0, 2, and 3, respectively; the parameter-limit resolvent remains finite but has a fourth-order pole at x=1.The table compares resolvent and matrix-power behavior at the centers.Source: §A.4.2 · p. 13, Table 3 · p. 14

A.5 Complete task catalogue and sources

  • The catalogue covers all 63 Reproduce and 43 Discover problems, organized by dataset identifiers, subfields, and counts of public subquestions.The Reproduce catalogue links source keys to 16 papers.Source: §A.5 · p. 13, §A.5.1 · p. 15

A.5.1 Reproduce catalogue

  • Reproduce tasks ask models to derive specified results or relations, with multiple problems sometimes drawing different formulas or applicability conditions from one source paper.The catalogue’s final column records public-subquestion counts.Source: §A.5.1 · p. 13, Table 4 · p. 14

A.5.2 Discover catalogue

  • Discover tasks modify source-theory assumptions and specify new objects or quantities to derive; their requested results need not appear in the source paper.The source supplies theoretical context, while the public problem statement defines the modification.Source: §A.5.2 · p. 15
  • The catalogue lists four works on warped AdS3 black holes, positive-kinematics branch points, dynamical field range, and Janus-interface correlators.Source: §A.5.2 · p. 17
  • It also lists q03–q06 on monodromy defects and off-shell equivalence in quantum field theory and gravity.Source: §A.5.2 · p. 17

B.1 Models and Closed-Book Settings

  • Evaluated models were GPT-5.6 Sol, DeepSeek v4 Pro, GLM-5.2, Kimi K2.6, and Qwen3.7-Max; Reproduce used closed-book answers, with some truncated responses supplemented by additional calls.Supplemented responses were not treated as single-call generations.Source: §B.1 · p. 17
  • In API closed-book Discover, models received one public problem statement without retrieval, execution tools, reference solutions, or scoring criteria.Temperature was 0, high reasoning effort was requested, and token limits were 65,536 initially and 131,072 for subsequent requests without a final answer; Qwen also had a 32,768-token thinking budget.Source: §B.1 · p. 17
  • GPT-5.6 Sol’s closed-book and tool-assisted Discover collections covered the same 43 problems, but its conversational-interface collection procedure differed from the four API models.Its reasoning effort was xhigh; problems and scoring criteria were the same.Source: §B.1 · p. 17

B.2 Tool-Assisted Discover Evaluation

Tool-assisted Discover answers were collected under stated resource limits, while closed-book prompts required checkable derivations without external aids. The evaluation scores complete subquestions strictly, applies consistent rules to disputed answers, and displays per-problem scores in Figures 13–16.

B.2 Tool-Assisted Discover Evaluation

  • Four API models faced the same 43 tool-assisted Discover problems, with optional Python, public arXiv search, and paper reading under execution and time limits.Each problem allowed up to 24 assistant turns, 128 tool calls, and two hours; GPT answers came through its conversational interface rather than the described API environment.Source: §B.2 · p. 17

B.3 Closed-Book Prompt

  • Closed-book Discover required a single checkable mathematical response without retrieval, code, symbolic tools, references, or access to earlier attempts.Definitions and equations in the public statement remained available; GPT conversational collections and Reproduce answers used their respective collection settings.Source: §B.3 · pp. 17–18

B.4 Subquestion Scoring and Aggregate Metrics

  • A subquestion earns credit only when all requested conclusions, derivations, and applicability conditions are correct; each problem contributes to whole-problem accuracy only if every subquestion is correct.Subquestion accuracy averages within each problem before averaging across problems, giving each problem equal weight.Source: §B.4 · p. 18

B.5 Disputed Scoring Decisions

  • Scoring follows the public question rather than every reference-solution step, accepts valid alternative derivations, and checks disagreements against the stated conditions before assigning model error.Correcting an intermediate error can preserve credit, but an unresolved contradiction in a required argument cannot; consistent decisions are applied across models.Source: §B.5 · p. 18

C.1 Problem-Level Scores

  • Figures 13–16 show strict correct-over-applicable subquestion counts for individual problems; the Discover figures share task order and color scale, and grey NA cells mark absent closed-book final answers.The 63 Reproduce problems are split across Figures 13 and 14; Figures 15 and 16 show closed-book and tool-assisted Discover scores, respectively.Source: §C.1 · p. 18, Figure 13 · p. 19, Figure 14 · p. 20

C.2 Results by Physics Subfield

Tables 7–9 report subquestion and whole-problem accuracy by physics subfield for Reproduce and both Discover settings. Their Global column pools problems rather than averaging the subfield percentages.

C.2 Results by Physics Subfield

  • Tables 7–9 report subquestion accuracy (SAcc) and whole-problem accuracy (WAcc) by physics subfield across Reproduce, closed-book Discover, and tool-assisted Discover.The Global column pools problems rather than averaging the five subfield percentages; underlying subfield sizes appear in Appendix A.1.Source: §C.2 · p. 23, Table 7 · p. 23, Table 8 · p. 23

D.1 Error Categories and Counts

The paper categorizes model errors by their primary failure mode and illustrates how missed assumptions, invalid inferences, calculation mistakes, and omitted work undermine otherwise plausible solutions. FormulaForest’s construction uses linked source checks, derivation planning, verification, blind solving, review, and revision.

D.1 Error Categories and Counts

  • Error labels distinguish conceptual misuse, unproved derivations, calculation mistakes, and incomplete or unsupported answers; figures show classified-error composition, not rates over all subquestions.Records with multiple labels use a fixed priority order: conceptual, calculation, derivation, then incomplete.Source: §D.1 · p. 23, Table 10 · p. 23

D.2.1 Conceptual: Frequency versus Energy

  • A correct oscillator-frequency calculation does not establish the light-cone energy: the gauge relates the worldsheet-time and x+ generators by 1/(α′p+).The response identified the mode frequency but omitted this factor, which also affects the subsequent energy expansion.Source: §D.2.1 · pp. 23–24

D.2.2 Derivation: Kernel Dependence

  • Fourier inversion must retain the kernel’s dependence on both external momentum k and separation q; moving K_N(k,q) outside the k integral leaves the result unjustified.A delta function emerges only after accounting for every other k-dependent factor.Source: §D.2.2 · p. 24

D.2.3 Calculation: Rotation Sign

  • With the question’s generator definitions and Poisson-bracket convention, {M12, M23} = M13, so the response’s −M13 is inconsistent with its own definitions.A simultaneous convention change could alter the overall sign, but the response did not make one.Source: §D.2.3 · p. 24

D.2.4 Incomplete: Null-Cone Check

  • The null-cone task separately required perturbative roots for both branches, but the response stated rather than showed them.The supported roots are dx−/dx+ = f+ + O(f2) and dx+/dx− = f− + O(f2).Source: §D.2.4 · p. 24

D.3.1 Conceptual: Pairwise Root Sharing

  • Pairwise vanishing resultants do not establish a shared root: the example has three distinct pairwise intersections, gcd(Q, R, S) = 1, and three double roots rather than one triple root.The response incorrectly treated the pairwise conditions as a common root of all three factors.Source: §D.3.1 · p. 24

D.3.2 Derivation: Uniform Bound

  • A uniform product bound yields strong monotonicity by minimizing each positive, continuous d_i on the nonnegative axis and applying the assumed inequality at both minimizers.The resulting d1d2 > η2 bound makes the constant comparison matrix positive definite; integrating the Jacobian along a segment then excludes two distinct roots.Source: §D.3.2 · p. 25
Loading 2610.32575v1…