Source-linked AI summary
Deep Research in Physical Sciences: A Multi-Agent Framework and Comprehensive Benchmark
Yigeng Jiang, Tengchao Yang, Taoyong Cui, Jiaxing Wan, Yuan Wang, Weida Wang, Zhiyu Liu, Chuyi Peng, Binzhao Luo, Maoli Gao, Huaihai Huang, Yuqianer Zeng, Ziyang Zheng, Dongchen Huang, Chao Chen, Zichao Liu, Weiping Shen, Shuchen Pu, Siyu Zhou, Runmin Ma, Yusong Hu, Fei Chao, Bo Zhang, Xiawu Zheng, Zifu Wang, Lei Bai, Yunqi Cai, Shufei Zhang
TL;DR
Physical-science deep-research agents lack comprehensive evaluation on authentic, multimodal workflows. This paper introduces PhySciBench and DelveAgent, whose architectural specialization improves accuracy across scientific benchmarks while reducing inference cost.
Problem
Existing benchmarks inadequately evaluate agents on authentic physical-science workflows involving multimodal parsing, structured data extraction, external tools, and extended reasoning.
Method
The paper introduces PhySciBench, a 200-question benchmark spanning six workflow-oriented task categories, and DelveAgent, a modular framework with adaptive planning, memory, and physics-grounded reflection.
Results
DelveAgent improves accuracy by up to 7.5 percentage points across four scientific benchmarks while reducing inference cost to about one-third of Gemini Deep Research.
Takeaways & Limitations
The findings support purpose-built, domain-grounded cognitive architectures as a route to more reliable autonomous scientific reasoning than model scale alone.
Takeaways & Limitations
PhySciBench contains 200 questions and concentrates disciplinary coverage in selected contemporary subfields, so aggregate scores are not uniformly representative of all physics and chemistry.
Abstract
from arXiv · showhide
Deep research agents are Large Language Model (LLM)-based systems designed for autonomous, multi-step scientific reasoning, and they hold immense potential for accelerating research in the physical sciences. However, comprehensive and in-depth evaluations of their capabilities within this domain remain lacking. To address this gap, we introduce PhySciBench, a benchmark highly relevant to physical science research, comprising 200 expert-curated questions, balanced between physics and chemistry, across six task categories that reflect real-world scientific workflows. Evaluations of state-of-the-art models and agent systems on PhySciBench reveal limited performance; even the strongest baseline, Gemini Deep Research, achieves an accuracy of only 33.5%. Analysis of failure cases identifies three recurrent deficiencies: fragility in extended reasoning chains, limited knowledge transfer across steps, and a lack of physics-grounded self-verification. Motivated by these findings, we develop DelveAgent, a modular multi-agent framework equipped with an adaptive planning loop, dual-granularity memory, and a hierarchical physics-grounded reflection mechanism. Across four scientific benchmarks, DelveAgent improves accuracy by up to 7.5 percentage points while reducing inference costs to approximately one-third of the strongest baseline. These results establish the significance of PhySciBench as a critical benchmark for evaluating AI systems in the physical sciences and demonstrate that architectural specialization can effectively enhance the reliability of autonomous scientific research. Our data and code are publicly available at https://github.com/yigengjiang/physci-deepresearch.
1. Introduction
Physical-science research requires integrating diverse evidence, extensive knowledge, complex reasoning, and external tools, motivating deep research agents for autonomous multi-step workflows. Because existing benchmarks rarely assess authentic, tool-integrated research workflows, the paper introduces PhySciBench and DelveAgent to evaluate and improve such capabilities.
- Physical-science research integrates experimental measurements, scientific figures, structured materials data, and primary literature with broad knowledge, synthesis, reasoning, and external-tool use.
- Deep research agents use LLMs to decompose complex problems, invoke external tools, and synthesize diverse evidence across multi-step interactions.
- Existing physical-science benchmarks are scarce, isolate single tasks, and rarely evaluate authentic research scenarios or extended workflows involving external tools.
- PhySciBench evaluates end-to-end physical-science research capabilities across information extraction, scientific understanding, and application and creation.The benchmark comprises 200 expertly curated questions.
- 33.5% accuracy is achieved by Gemini Deep Research, the strongest baseline evaluated on PhySciBench.Analysis of 326 failure cases identifies brittle long-horizon reasoning, weak cross-step domain-experience transfer, and insufficient physics-grounded verification.
- DelveAgent combines adaptive planning, dual-granularity memory, and hierarchical physics-grounded reflection to replan, reuse experience and knowledge, and check scientific consistency.
2. Results
PhySciBench exposes substantial weaknesses in current physical-science deep-research systems, while DelveAgent’s specialized planning, memory, and physics-grounded reflection improve accuracy and scientific reliability. Across benchmark and user-study evaluations, DelveAgent reduces architectural failures and better anchors conclusions to supplied evidence.
- PhySciBench Results: 33.5% accuracy was achieved by Gemini Deep Research, the strongest baseline on PhySciBench, while all base models remained below 30%.Reasoning-enhanced Gemini 3 Flash and Grok-4.1-Fast scored lower still, indicating limitations beyond standard chain-of-thought processing.
- Failure Analysis: 46% of failures arose from planning and reasoning problems, as unstable long-horizon trajectories propagated early interpretation, retrieval, or synthesis errors.These failures motivated dynamic replanning through DelveAgent’s Adaptive Planning Loop.
- DelveAgent Performance: 41.0% accuracy on PhySciBench gave DelveAgent a +7.5 percentage-point gain over Gemini Deep Research’s 33.5%.The benchmark’s expert-originated questions, anti-contamination safeguards, and difficulty calibration preserve substantial evaluation headroom.
- Ablation Results: 3.0 p.p. accuracy losses followed removal of either Dual-Granularity Memory or Hierarchical Physics-Grounded Reflection, versus 2.5 p.p. after removing Adaptive Planning Loop.Removing all three caused a 6.5 p.p. degradation, below the 8.5 p.p. sum of individual drops but larger than any single ablation.
- Cross-Benchmark Results: 23.62% average accuracy across three benchmarks exceeded Gemini Deep Research’s 20.87%, with the largest improvement on SGI-DR at +6.98 percentage points.On HLE, DelveAgent scored 34.94% versus 33.67%, while FS-Research performance was comparable to the strongest baseline.
- User Studies: 8.39 versus 7.50 task completion, 8.05 versus 6.39 scientific validity, and 3.02 versus 6.05 hallucination scores favored DelveAgent across the three user studies.The largest reliability gaps occurred in open-ended studies requiring evidence-grounded conclusions, whereas the narrow numerical FMAC task produced comparable completion and validity.
3. Discussion
The discussion presents PhySciBench and DelveAgent as complementary contributions addressing undercharacterized deep-research capabilities in physical science. It attributes DelveAgent’s gains to specialized interacting mechanisms while emphasizing benchmark limitations, human oversight, and future expansion.
- Contributions: PhySciBench is introduced as a comprehensive physical-science benchmark, while DelveAgent is a modular multi-agent framework whose architectural specialization improves accuracy across diverse tasks.The benchmark addresses a gap in evaluating deep-research capabilities in physical science.
- Architectural specialization: Disabling Dual-Granularity Memory or Hierarchical Physics-Grounded Reflection each reduced accuracy by −3.0%, the largest single-component degradation.The ablation results identify knowledge grounding and physics-aware validation as dominant contributors on PhySciBench.
- Architectural specialization: Removing the adaptive planning loop reduced accuracy by 2.5%, supporting architectural specialization rather than foundation-model choice as the source of DelveAgent’s advantage.The discussion attributes reliability to the interplay of all three mechanisms rather than any single component alone.
- Benchmark design: PhySciBench uses expert-crafted questions from recent publications and adapted textbook problems, organized through three cognitive stages: Information Extraction, Scientific Understanding, and Application and Creation.This design aims to minimize training-data contamination at question creation and may provide a template for other scientific disciplines.
- Limitations: The benchmark’s 200-question size limits domain and difficulty coverage, despite revealing statistically significant performance differences; community-driven expansion is therefore needed.Coverage is balanced across physics and chemistry but depth-first within each discipline, focusing on active contemporary subfields.
- Limitations and future work: DelveAgent is intended to assist rather than replace researchers, and the authors recommend critical human oversight, especially for sensitive applications where errors have real-world consequences.Future work includes community expansion of PhySciBench and improved multimodal understanding for specialized scientific visualizations.
4. Methods
The methods establish PhySciBench as a rigorously curated physical-science benchmark and evaluate it with task-specific automatic and semantic protocols. DelveAgent uses a closed cognitive loop combining adaptive planning, dual-granularity memory, and hierarchical physics-grounded reflection, followed by comparisons across expert-level scientific benchmarks.
- PhySciBench curation: PhySciBench contains 200 expert-curated questions, balanced between physics and chemistry and divided into six mutually exclusive task categories.Candidate questions were zero-shot tested against baseline models, excluding items solvable through parametric memory or general web search.
- Evaluation protocol: Its evaluation protocol combines deterministic rule-based checks with rubric-guided semantic judgment tailored to heterogeneous task types.Structured extraction uses weighted syntactic, key-match, and value-match scoring, while code-generation outputs are executed in a sandbox before further evaluation.
- DelveAgent architecture: DelveAgent’s closed cognitive loop couples a Planner with five specialized worker agents through adaptive planning, dual-granularity memory, and hierarchical physics-grounded reflection.The Planner replans after worker failures, unexpected results, or Critic rejections rather than committing irreversibly to an initial strategy.
- Dual-Granularity Memory: Dual-granularity memory transfers knowledge through retrieved planning trajectories and domain knowledge bases built from scientific papers during problem solving.Experience Memory retrieves the top three similar historical strategies, while Domain Knowledge Bases provide reaction mechanisms and experimental protocols to workers.
- Hierarchical Physics-Grounded Reflection: Hierarchical reflection validates intermediate outputs with domain-specific verifiers and evaluates final answers with a Critic using a structured error taxonomy.Local refinement loops can detect errors such as mass imbalance or dimensional inconsistency before failures are escalated.
- Experimental evaluation: The experimental protocol compares DelveAgent with state-of-the-art baselines across multiple challenging scientific datasets using overall accuracy as the primary pass@1 metric.The evaluation uses a single sampled response per question, benchmark-native scoring for public datasets, and the composite protocol for PhySciBench.