PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Research

Tingjia Miao, Wenkai Jin, Jinxin Tan, Muhua Zhang, Xianghe Pang, Zexi Liu, Yuwen Du, Tian Jin, Tu Guo, Zhengliang Zhang, Jingkun Liu, Yuelin Hu, Jiejun Zhang, Yunjie Huang, Yuhan Wang, Wenbo Li, Yinuo Gao, Shuo Chen, Rui Ye, Yuzhi Zhang, Linfeng Zhang, Kun Chen, Wei Wang, Weinan E, Siheng Chen

arXiv:2512.19799v2cs.AIhep-lat

TL;DR

Frontier theoretical and computational physics research demands extended reasoning and computation, while existing agents remain unreliable on extended workflows. The paper introduces PRL-Bench and PhysMaster, which combines adaptive exploration with hierarchical memory. PhysMaster reports a 51.08 overall score versus Codex's 48.62 on PRL-Bench, with approximately 9% lower API cost.

  • Problem

    Frontier physics research involves intensive long-horizon reasoning and numerical computation, while current agents remain insufficiently reliable on extended research workflows.

    Source: §1.1 · pp. 2–3

  • Method

    PhysMaster combines adaptive multi-trajectory exploration with hierarchical memory to support reliable long-horizon research workflows.

    Source: §3 · p. 10

  • Results

    51.08 vs. 48.62 overall score for Codex, with approximately 9% lower API cost; PhysMaster also consistently improves over corresponding ReAct baselines across eight backbone models.

    Source: §2.4 · p. 9, §3 · p. 10

  • Takeaways & Limitations

    PRL-Bench provides a controlled, research-oriented testbed for systematically evaluating autonomous scientific research agents.

    Source: §3 · p. 10

    PRL-Bench uses research-derived tasks with verifiable targets and does not directly measure scientific novelty or creativity.

    Source: §3 · p. 11

Abstract

from arXiv · show

Advances in LLM reasoning and tool use have enabled agentic science, yet frontier theoretical and computational physics remains challenging because research requires deep domain expertise, long-horizon reasoning, and reliable numerical computation. We introduce PRL-Bench, a research-reproduction benchmark adapted from 100 Physical Review Letters papers across major areas of modern physics. PRL-Bench distills realistic research workflows into traceable tasks with explicit intermediate artifacts and diverse evaluation rubrics; each task is estimated by domain experts to require more than six hours for a specialized PhD student to reproduce independently. Evaluations show that existing agents remain unreliable on extended research workflows. We therefore present PhysMaster, a scientific agent combining adaptive MCTS-based multi-trajectory exploration with hierarchical memory to improve long-horizon robustness and knowledge accumulation. PhysMaster achieves the highest overall PRL-Bench score of 51.08, outperforming Codex, OpenHands, OpenClaw, and ReAct, and yields relative improvements of 14.13 percent to 93.38 percent across backbone models. Error analysis shows that PhysMaster substantially reduces failures from incomplete long-horizon execution, while remaining bottlenecks lie in physics knowledge and analytical reasoning. Together, PRL-Bench and PhysMaster provide a rigorous benchmark and effective system for advancing autonomous AI research in frontier physics.

1 Introduction: Towards Agentic Scientific Research

The introduction presents PRL-Bench as a testbed for research workflows that existing benchmarks only partially capture, and PhysMaster as an agent combining adaptive exploration with hierarchical memory. It reports consistent performance gains and frames both contributions as a practical paradigm for agentic physics research.

1.1 Benchmarking End-to-end Physics Research

  • Existing scientific and physics benchmarks mainly assess knowledge, problem solving, or direct reproduction, with limited coverage of comprehensive exploratory research workflows.The introduction also notes that general expert benchmarks use comparatively constrained solution structures.Source: §1.1 · p. 3
  • PRL-Bench adapts 100 curated Physical Review Letters papers across six physics subfields into research-reproduction tasks validated by domain experts.Tasks preserve theoretical derivations, numerical computation, open solution paths, long-horizon workflows, and diverse rubrics; each is estimated to require a specialized PhD student over six hours to reproduce.Source: §1.1 · p. 3

1.2 Towards Autonomous AI Physicist

  • PhysMaster combines Monte Carlo Tree Search-based multi-trajectory exploration with hierarchical memory that consolidates execution experience into transferable insights.The introduction also describes LANDAU as an extensible external knowledge system integrating literature, curated priors, analytical skills, and expert workflows.Source: §1.2 · p. 4
  • 51.08 overall score, with relative gains of 14.13% to 93.38%, is reported for PhysMaster across evaluated backbone models, alongside improved workflow completeness and reliability.The introduction also reports that error analysis attributes the main improvement to greater completeness of long-horizon research workflows.Source: §1.2 · pp. 4–5

1.3 Core Contributions

  • The paper presents PRL-Bench as a rigorous evaluation benchmark and PhysMaster as a transferable architecture for autonomous research agents.Its stated contributions include traceable research artifacts, diverse evaluation rubrics, hierarchical collaboration, adaptive exploration, and integration of domain-specific scientific computing tools.Source: §1.3 · p. 5

2 Results

PRL-Bench comprises expert-curated, long-horizon research tasks spanning six physics subfields, and PhysMaster improves results over ReAct and general-purpose agent frameworks. Its largest reported gains accompany more complete workflows, while analytical reasoning and scientific expertise remain bottlenecks.

2.1 Benchmark Overview

  • PRL-Bench adapts 100 Physical Review Letters papers into expert-validated tasks spanning six physics subfields, with each task estimated to require over six hours of specialist PhD work.Tasks preserve research workflows and are adapted rather than copied directly from the source papers to mitigate potential data leakage.Source: §2.1 · pp. 5–6

2.2 Evaluation Protocol

  • Eight frontier LLM backbones are evaluated under matched prompts and tools against ReAct and PhysMaster, with both exploration loops capped at 50 iterations.Each task is scored over three runs using a GPT-5.4 judge, and scores are normalized to 0–100.Source: §2.2 · pp. 6–7

2.3 Consistent Gains of PhysMaster across Backbone Models

  • 14.13%–93.38% relative improvement over ReAct across backbones; GPT-5.4 with PhysMaster scores 51.08 overall.Improvements are reported across all 48 model–subfield combinations.Source: §2.3 · p. 7, Table 2 · p. 7

2.4 Comparison with General Agent Baselines

  • PhysMaster scores 51.08 overall, ahead of Codex (48.62), OpenClaw (46.20), and OpenHands (40.99), while costing about 9% less than Codex.PhysMaster costs more than OpenHands and OpenClaw; Table 3 reports normalized scores and average API cost per task.Source: §2.4 · pp. 8–9, Table 3 · p. 8

2.5 Failures Analysis

  • PhysMaster reduces Coverage failures, while remaining failures increasingly reflect analytical reasoning challenges and persistent gaps in fundamental scientific understanding.The paper relates fewer Coverage errors to adaptive exploration and hierarchical memory, which support recovery from intermediate failures and continuity across research stages.Source: §2.5 · pp. 9–10

3 Discussion

The discussion presents PRL-Bench as a controlled testbed for agentic physics research and PhysMaster as a system that improves long-horizon workflows across frontier backbones. It also notes that current agents retain important limitations and that the benchmark does not measure open-ended scientific novelty or creativity.

3 Discussion

  • PRL-Bench preserves open-ended reasoning, theoretical derivation, numerical computation, and long-horizon workflows in a controlled testbed for autonomous research agents.The benchmark is built from curated Physical Review Letters papers and validated by domain experts.Source: §3 · p. 10
  • Across eight frontier backbone models, PhysMaster consistently improves performance over corresponding ReAct baselines and competitive general-purpose agent frameworks.The architecture primarily reduces incomplete research workflows, while analytical reasoning remains the dominant bottleneck.Source: §3 · p. 10
  • Even the strongest systems exhibit unstable long-horizon reasoning, imperfect theoretical derivations, and insufficient domain knowledge on frontier research problems.Source: §3 · pp. 10–11
  • PRL-Bench evaluates research-derived tasks with verifiable targets, so it does not directly measure scientific novelty or creativity.Source: §3 · p. 11
  • The authors frame PhysMaster and PRL-Bench as a practical paradigm for advancing agentic science and long-horizon autonomous discovery.Source: §3 · p. 11

4 Methods: PhysMaster, the Autonomous Physics Research Agent

PhysMaster separates task clarification, iterative multi-agent execution, and final integration, while coordinating search through MCTS and consolidating knowledge across tasks. LANDAU adds a layered, extensible physics knowledge system.

4.1 Agentic Workflow and Hierarchical Agent Collaboration

  • Clarification, execution, and integration are separated across a Clarifier, Supervisor–Theoretician–Critic hierarchy, and Summarizer.The Clarifier structures the task; the execution agents develop and evaluate candidates; the Summarizer packages the selected solution and artifacts.Source: §4.1 · pp. 11–12

4.2.1 MCTS-based Exploration

  • MCTS explores complete candidate-result states through draft or Critic-guided revision, using UCB selection and propagated rewards to allocate search effort.Search ends when a candidate is accepted as complete or the configured round budget is exhausted.Source: §4.2.1 · pp. 12–13

4.2.2 Hierarchical Cognitive Memory Across Tree Exploration

  • Three memory levels preserve active execution records, distill stable node-level findings, and abstract completed tasks into transferable scientific experience.Information is progressively consolidated according to its long-term reuse value, limiting reliance on ever-growing prompt histories.Source: §4.2.2 · pp. 13–14

4.3 LANDAU: Layered Academic Data Universe

  • LANDAU combines literature, curated theoretical priors, reusable analytical and computational skills, and task workflows in a persistent, extensible knowledge base.The layered organization allows new knowledge sources to be incorporated without modifying the underlying reasoning architecture.Source: §4.3 · pp. 14–15

A Evaluation Prompt

The evaluation prompt directs an LLM judge to score candidates against task-specific rubrics, apply strict error categories, and return standardized scores and task-level judgments.

Scoring Rules

  • Each rubric item is binary and equally weighted unless explicit weights are provided, with no partial credit or discretionary intermediate scoring.The normalized 0–100 score sums only satisfied rubric items.Source: A Evaluation Prompt · p. 16

Evaluation Criteria

  • Equivalent derivations are acceptable when logically valid, numerical results must meet stated tolerances or a default 5% relative tolerance, and code must be functionally equivalent.Execution of candidate code is not required.Source: A Evaluation Prompt · p. 16

Error Types

  • Coverage applies only when a requested component is entirely absent; incorrect present content is classified as Conceptual or Analytical.Conceptual errors concern the physical framework, while Analytical errors concern derivation, calculation, or implementation after choosing the framework.Source: A Evaluation Prompt · pp. 16–17

Output Requirements

  • The judge must assess every task against its rubrics and return a score plus task-level satisfied items, unsatisfied items, dominant error type, and brief reason.The number of evaluated tasks must match the problem’s task count.Source: A Evaluation Prompt · p. 17

B Representative PRL-Bench Task

The representative task uses a graphene multilayer model to connect symmetry-enforced band degeneracy and SU(2) structure with numerical band-topology and spin-texture calculations.

Introduction

  • At ν = 2, the model’s occupied doublet is a spin-degenerate Chern band with |C| = 1 and a skyrmionic spin texture of winding number +2.The non-symmorphic symmetry enforces the band degeneracy and SU(2) Wilson-loop holonomies.Source: Introduction · p. 18

Task 1: Symmetry Analysis (Analytical)

  • The task asks students to prove that Ui commute with H, anticommute pairwise, and enforce exact twofold degeneracy, with eigenvalue swapping indicating SU(2) holonomy.It also traces the cosine potential to Hartree-Fock exchange and requires explaining why the three amplitudes are equal.Source: Task 1: Symmetry Analysis (Analytical) · p. 18

Task 2: Band Topology (Numerical)

Task 3: Spin Texture (Numerical)

Representative Prior Sources

The representative prior-source list spans physics reference areas, including quantum field theory, particle physics, condensed matter, statistical physics, and astrophysics.

Quantum Field Theory, Renormalization, and High-Energy Theory

  • The quantum-field-theory and renormalization sources include Weinberg, Das, Collins, Faddeev, Polchinski, Blumenhagen et al., and Deligne et al.The list also includes works on gravitation and cosmology, including Weinberg and Baumann.Source: Representative Prior Sources · p. 21

Particle Physics, Standard Model, and Perturbative QCD

  • Particle-physics and perturbative-QCD references include Cheng and Li, Collins, and Kovchegov and Levin.The condensed-matter and many-body references include Altland and Simons, Coleman, Bruus and Flensberg, Dupuis, and Sachdev.Source: Representative Prior Sources · p. 21

Statistical Physics and Non-equilibrium Physics

  • Statistical and non-equilibrium physics are represented by Kardar and de Groot and Mazur.The astrophysics and compact-object list includes works on galactic dynamics, accretion, stellar remnants, and compact stars.Source: Representative Prior Sources · p. 21

Related papers

Loading 2512.19799v2…