PaperlayerView exact revision on arXiv

Source-linked AI summary

Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, Indranil Das, Qingzhi Chen, Junyi Cao, Yufeng Du, Jiabin Yu, Peixue Wu, Jinchen He, Yifan Su, Yikun Jiang, Yujie Zhang, Chang Liu, Ze-Min Huang, Weizhen Jia, Yunkai Wang, Farshid Jafarpour, Yong Zhao, Xinan Chen, Jessie Shelton, Aaron W. Young, John Bartolotta, Wenchao Xu, Yue Sun, Anjun Chu, Victor Colussi, Chris Akers, Nathan Brooks, Wenbo Fu, Jinchao Zhao, Marvin Qi, Anqi Mu, Yubo Yang, Allen Zang, Yang Lyu, Peizhi Mai, Christopher Wilson, Xuefei Guo, Juntai Zhou, Daniel Inafuku, Chi Xue, Luyu Gao, Ze Yang, Yaïr Hein, Yonatan Kahn, Kevin Zhou, Di Luo, John Drew Wilson, Jarrod T. Reilly, Dmytro Bandak, Ofir Press, Liang Yang, Xueying Wang, Hao Tong, Nicolas Chia, Eliu Huerta, Hao Peng

arXiv:2509.26574v4cs.AIcond-mat.othercs.CLhep-thquant-ph

TL;DR

Physics research demands rigorous, precise reasoning, yet evidence about LLMs on unseen, open-ended research tasks and modular workflow assistance is limited. CritPt benchmarks those tasks through expert-created challenges and checkpoints, finding early progress on narrower tasks but a substantial gap on full research-scale challenges. The results indicate that current models remain far from reliable performance in realistic physics research.

  • Problem

    Whether LLMs can solve unseen entry-level physics research problems or assist with modular research reasoning remains an open question in a field where rigorous, precise work is essential.

    Source: §1 · p. 2

  • Method

    CritPt is an expert-designed benchmark of realistic physics research reasoning, with challenges and checkpoints evaluated through standardized, physics-informed procedures.

    Source: §1 · p. 2, §2.2 · p. 5, §2.3 · p. 6

  • Results

    Models show early promise on narrowly scoped tasks but remain minimally capable of maintaining coherence and correctness through full-scale challenges; GPT-5 (high) achieves 5.7% average accuracy on challenges.

    Source: §1 · p. 4, §5 · p. 16

  • Takeaways & Limitations

    CritPt gives AI developers standardized feedback and physicists a grounded view of current AI capabilities, providing a framework to measure progress toward augmenting physics discovery.

    Source: §5 · p. 16

    The reliability check is heuristic and based on five runs under resource constraints, which the authors do not claim are statistically sufficient.

    Source: §4.3 · p. 14

Abstract

from arXiv · show

While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools.

1 Introduction

CritPt evaluates whether LLMs can reason through unseen, open-ended physics research tasks and assist with modular work. Its results show early progress on checkpoints but little reliability on full research-scale challenges.

Motivation

  • Physics solutions demand rigor, creativity, precise execution, and consistency between theory and experiment, while a single flawed inference can invalidate a solution.These demands distinguish original research reasoning from applications that mainly recombine existing information.Source: §1 · p. 2

Research questions

  • The benchmark addresses whether LLMs can solve unseen entry-level research problems, assist with modular workflow tasks, and produce trustworthy reasoning in advanced physics.The questions matter because subtle errors can mislead readers without expert judgment.Source: §1 · p. 2

Benchmark contribution

  • CritPt contains 71 composite entry-level research challenges and 190 checkpoints that expose performance on simpler subtasks.The checkpoints offer more fine-grained insight into reasoning capabilities.Source: §1 · p. 3

Benchmark design

  • CritPt was co-designed by AI researchers and physicists using experts’ own research expertise to represent frontier-physics demands.Its problems are unpublished, hand-curated, open-ended, and designed for guess-resistant answers with private solutions for 70 test challenges.Source: §1 · p. 3

Evaluation results

  • GPT-5 (high), the best-performing base model on CritPt, achieves 5.7% average accuracy on challenges; with code interpreter and web search, it improves modestly to 12.6% accuracy.Source: §1 · p. 4

2 Design choices of CritPt

CritPt turns expert physics research practice into a broad, carefully reviewed benchmark of full challenges and modular tasks. Its design aims to test substantive reasoning while supporting standardized evaluation.

2.1 Source and coverage: hand-curated research challenges from the physics community

  • CritPt’s 71 challenges and 190 checkpoints span modern physics disciplines and three research flavors; 33 challenges and 88 checkpoints cover two or more disciplines.The sampled topics range from quantum error correction to string theory, cell dynamics, black holes, nonlinear optics, and special-function asymptotics.Source: §2.1 · pp. 4–5

2.2 Benchmark criteria: leakage-resistant and reasoning-focused design

  • Problems are newly created, search-proof, well-posed, and answerable from public knowledge, with open-ended formats and physics-tailored answers designed to resist guessing.Answers may use numerical values, symbolic expressions, or Python functions; selected quantities distinguish correct from incorrect reasoning paths.Source: §2.2 · pp. 5–6

2.3 Quality control: iterative development and multi-level expert review

  • Each challenge undergoes iterative creation and multi-stage expert review, with technical derivations validated and model responses checked for artifacts.Creation averages more than 40 hours of expert effort per full challenge.Source: §2.3 · p. 6

2.4 Structure of a challenge: an example

  • A challenge pairs a mentor-like setup with a central research question, then decomposes it into 2–4 checkpoints that isolate substantive reasoning steps.Checkpoints include intermediate derivations, simplified cases, and special-case analyses rather than mere formula substitution.Source: §2.4 · pp. 6–7

Challenge: Quantum Error Detection

The quantum error detection challenge asks for logical-state fidelity in a [[4,2,2]] code under gate errors and post-selection. Its setup specifies the error model, preparation circuit, and success condition.

Setup

  • The setup encodes two logical qubits in four physical qubits and models each CNOT as followed by a two-qubit depolarizing error with probability p.The requested quantity is logical infidelity as a function of physical infidelity p.Source: Challenge: Quantum Error Detection · p. 7

Challenge question

  • The challenge prepares logical |00⟩AB with an ancilla-assisted circuit and asks for final logical-state fidelity after post-selecting detectable errors and ancilla measurement |0⟩.The ancilla detects preparation errors; measurement |0⟩ marks success and |1⟩ failure.Source: Challenge: Quantum Error Detection · p. 7

Answer

Checkpoints

The checkpoints ask for state-fidelity functions of the two-qubit gate error rate under distinct preparation and post-selection conditions. Together, they separate physical-state and logical-state fidelity tasks.

Physical-state fidelity

  • One checkpoint asks for physical fidelity of a final four-qubit state prepared with Hadamard and CNOT gates as a function of gate error rate p.The circuit is given in matrix-multiplication order, so operations occur right-to-left.Source: Checkpoints · p. 8

Logical-state fidelity

  • A logical GHZ-state checkpoint asks for final logical fidelity as a function of p after post-selecting detectable code errors.The checkpoint uses the specified state-preparation circuit and code context.Source: Checkpoints · p. 8

Ancilla-assisted fidelity

  • An ancilla-assisted preparation checkpoint asks for logical fidelity after post-selecting detectable code errors and ancilla measurement |0⟩.The corresponding answer is expressed as a polynomial in p.Source: Checkpoints · p. 8

3 Evaluation pipeline

The evaluation pipeline separates solution generation from answer formatting, then checks canonical outputs against expert answers with format-specific grading. Its public server supports comparable evaluation while keeping challenge answers private.

3.1 Two-step answer generation from models

  • Generation separates free-form solution reasoning from canonical code-block answer extraction, avoiding premature conversions and reducing parsing errors.The first round permits natural-language reasoning and derivations; the second standardizes the final answer for parsing and grading.Source: §3.1 · p. 9

3.2 Auto-grading system

  • The grader parses constrained answer code, tests Python functions on expert-selected cases, requires every composite component to match, and sandboxes execution.Sandbox defaults include a 30-second timeout and bounded memory, with higher expert-specified limits used for computationally heavy tasks.Source: §3.2 · pp. 9–10
  • Numerical answers use expert tolerance ranges or 12-significant-digit precision, while symbolic expressions receive hierarchical SymPy equivalence and simplification checks.Custom symbolic routines address cases where standard simplification is insufficient.Source: §3.2 · p. 9

3 Evaluation pipeline

  • The pipeline enables scalable, high-fidelity evaluation of diverse, complex physics outputs despite substantial upfront labor.The public evaluation pipeline and grading server are available for future testing and public access.Source: §3 · p. 8, §3.2 · p. 10

3.3 Public evaluation pipeline for CritPt challenges

  • The public server keeps 70 challenge answers private, limits submissions to complete batches, and Artificial Analysis independently validated reported results through it.The server returns accuracy over all 70 challenges and permits at most 10 submissions per account per 24 hours.Source: §3.3 · p. 10

4 Results

Across full challenges and decomposed checkpoints, current models make limited progress on average accuracy, and performance drops sharply under a stricter consistency criterion. Expert review of model responses also surfaces tool-use and output-format issues.

4.1 Challenge-level evaluation: can LLMs solve unseen research problems?

  • 5.7% average accuracy is GPT-5 (high)'s best base-model challenge result; GPT-5 (high, code) reaches 10.6%, and code plus web reaches 12.6%.All other base models score at or below 2%.Source: §4.1 · p. 10, Figure 3 · p. 12
  • Reasoning models use many more tokens, while performance gains are often disproportionate to inference cost; vendor pricing also makes DeepSeek R1 cheaper than Claude Opus 4.GPT-5 (high, code & web) uses an order of magnitude more tokens than other reasoning models, largely from web-retrieved input.Source: §4.1 · pp. 11–12, Figure 3 · p. 12

4.2 Checkpoint-level evaluation: smaller tasks that LLMs can assist today?

  • 15.3% self-carryover accuracy for GPT-5 (high) rises to 20.0% with code and 21.4% with code and web on checkpoints.Gemini 2.5 Pro and o3 (high) follow at 8.1% and 7.4%, respectively; the ordering among models below 3% warrants caution.Source: Figure 5 · p. 13, §4.2 · p. 13
  • More than 3% gains under oracle carryover for the GPT-5 (high) family and o3 (high) show that leading models can use correct intermediate answers to improve downstream reasoning.Almost all models benefit from expert-answer injection.Source: §4.2 · p. 13

4.3 Reliability metric: can we trust LLM outputs?

  • Requiring correct final answers in at least four of five runs sharply reduces measured performance across models.The authors describe this as a heuristic reliability check constrained by resources, not a statistically sufficient test or proof that 80% accuracy implies high reliability.Source: §4.3 · p. 14
  • 4.3% of challenges are consistently solved by GPT-5 (high) without tools, rising to 8.6% with code and 10.0% with code and web; other reasoning models score zero.At checkpoints, leading models consistently solve only a very limited number, and GPT-5 (high, code) slightly outperforms its code-and-web version.Source: Figure 6 · p. 14, §4.3 · p. 15
  • Current models cannot yet be trusted to reason consistently in high-stakes research contexts, so small-subtask exploration may still require expert oversight.Experts report that verifying convoluted but plausible outputs can sometimes take longer than solving the problem independently.Source: §4.2 · p. 14, §4.3 · p. 15

4.4 Detailed analysis of full model responses

  • Expert review finds that GPT-5 (high) can clutter outputs and use code or web retrieval in ways that diverge from expert expectations.The interactive platform supports systematic comparison of model responses; on the example challenge, adding web search to coding degrades GPT-5 (high)'s performance.Source: §4.4 · pp. 15–16

5 Conclusion

CritPt is an expert-designed benchmark of unpublished physics problems that probes whether LLMs can meet frontier research’s reasoning demands. The evaluation finds a substantial gap in full-challenge performance and reproducibility, while the benchmark offers a shared framework for AI developers and physicists.

5 Conclusion

  • CritPt comprises unpublished, guess-resistant problems developed by 50+ active physicists, with self-contained challenges and modular checkpoints for assessing research reasoning.Its problems are designed to reflect realistic workflows, complexity, and failure sensitivity in modern physics.Source: §5 · p. 16
  • Even advanced reasoning models show minimal ability to maintain coherence and correctness through full-scale challenges, and correct answers are often not consistently reproducible.The evaluation contrasts early promise on narrowly scoped tasks, particularly with coding tools, with a severe robustness deficit for high-stakes scientific applications.Source: §5 · p. 16
  • CritPt provides AI developers with a standardized dataset and physics-informed automatic evaluation, while physicists can inspect and critique model outputs at scale.The paper presents this cross-community process as creating a shared vocabulary for requirements of systems intended to engage with scientific discovery workflows.Source: §5 · p. 16
  • The authors present CritPt as a scientifically grounded framework for measuring AI progress and guiding development of more capable, reliable reasoning models.They also hope it will catalyze discussion between physics and AI communities about machine reasoning in scientific contexts.Source: §5 · p. 16

A.1 List of CritPt challenges

The appendix lists the 70 challenges in CritPt’s test set, spanning a broad range of physics topics. The titles include problems in quantum physics, gravity, condensed matter, and fluid dynamics, among others.

A.1 List of CritPt challenges

  • The test set contains 70 challenges, with titles ranging from holographic Weyl anomalies and stochastic cell-growth models to quantum sensing and gravity.The listed examples also include condensed-matter topics such as twisted bilayer MoTe2 and the Moore-Read state.Source: §A.1 · p. 18
  • The list also covers optics, atomic and molecular physics, astrophysics, and quantum information, including optical binding, relativistic jets, and quantum channels.Further listed topics include Yb magic wavelengths, dark-matter detection, and quantum capacity.Source: §A.1 · pp. 18–19
  • Other listed challenges address mathematical and statistical physics, nonlinear dynamics, and fluid dynamics, including tensor networks, autocatalytic cycles, and Rayleigh-Bénard convection.The titles also include lattice Gaussian sums, random walks, and Rayleigh-Darcy convection.Source: §A.1 · pp. 18–19

A.2 Prompts for two-step answer generation from model

The appendix describes a two-step answer-generation process: first prompt the model to solve the physics problem, then place its result into a code template for formatting. The prompts specify reasoning, output, and precision requirements.

A.2 Prompts for two-step answer generation from model

  • The first step prompts the model to act as a physics research assistant and provide a precise, step-by-step solution to the Markdown problem.It requests justified non-trivial steps, LaTeX mathematics, adherence to the problem’s conventions and units, and a final answer in the requested format.Source: §A.2 · p. 20
  • When code interpretation is enabled, the first-step prompt includes a SymPy template for returning the logical-state fidelity expression using the gate-error-rate input.The prompt identifies p as the two-qubit gate error rate and F_logical as the logical state fidelity.Source: §A.2 · p. 20
  • The second step fills the provided code template for display only, with no additional reasoning or derivation and no imports beyond those supplied.If no precision is specified, the prompt asks for an exact form where feasible, or at least 12 significant digits otherwise.Source: §A.2 · p. 20

A.3 Detailed API setup used in evaluation

The evaluation uses models from several providers and records their API configurations for reproducibility. The appendix distinguishes a minimal-reasoning GPT-5 configuration from other model variants.

A.3 Detailed API setup used in evaluation

  • The evaluation covers eight proprietary models from four providers and one open-source model, Meta Llama-4 Maverick, accessed via Together AI.The providers are OpenAI, Google, DeepSeek, and Anthropic; evaluation uses Inspect AI for the proprietary models.Source: §A.3 · p. 21
  • The tested models include OpenAI’s GPT-5, o3, o4-mini, and GPT-4o; Google’s Gemini 2.5 Pro and Flash; DeepSeek R1; and Anthropic’s Claude Opus 4.The appendix describes their model variants and intended capabilities, including o4-mini’s latency and cost focus and Gemini 2.5 Flash’s speed and efficiency focus.Source: §A.3 · p. 21
  • API names and configurations are summarized in Table 5, and GPT-5 (minimal) uses minimal reasoning effort with zero reasoning tokens.The table is identified as the API configurations used in the evaluation.Source: §A.3 · p. 21, Table 5 · p. 21

A.4 API usage statistics

The appendix reports API token and cost accounting, then uses a quantum error-detection example to illustrate model performance across increasingly demanding research tasks. Models handle simpler calculations more often than complex code-dependent reasoning, but incorrect answers can remain plausible or poorly presented.

A.4 API usage statistics

  • Table 6 reports average token use, pricing, and cost; tool use increases input and cached-input tokens through multi-turn interactions, while reasoning and response token counts reflect only the final round.Total cost is the sum of input and output costs, with cached input tokens charged at a separate rate.Source: Table 6 · p. 22, §A.4 · p. 22

A.5.1 Design idea from expert

  • The QEC example progresses from hand-calculating physical fidelity to logical fidelity, then to a more complex protocol requiring code to count errors and simulate their propagation.The code encodes two logical qubits in four physical qubits; the example uses a small code as an analytical test bed for concepts relevant to larger codes.Source: §A.5.1 · p. 23

A.5.2 Expert feedback on model responses

  • Many models solve subproblem 1, though some make early calculation errors and even correct responses can be difficult to verify because of poor formatting.GPT-5 (high, code) and GPT-5 (high, code & search) use code even though simple hand counting suffices.Source: §A.5.2 · pp. 23–24
  • Most models understand subproblem 2's high-level probability-counting approach but cannot execute its details correctly; GPT-5 (high, code) reliably solves it.The expert notes that code is a reasonable, though not strictly necessary, approach for this calculation.Source: §A.5.2 · p. 24
  • All models except GPT-5 (high, code) and GPT-5 (high, code & search) fail subproblem 3, which requires code to count quantum errors in a more complex setting.GPT-5 (high, code) could fairly consistently produce the correct answer, but its presentation was poorly formatted.Source: §A.5.2 · p. 24

Related papers

Loading 2509.26574v4…