PaperlayerView exact revision on arXiv

Source-linked AI summary

PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models

Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Jiaming Ji, Weike Wang, Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing-Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia, Jiawei Lin, Zheyu Shen, Jiahang Chen, Qiuhao Xiong, Binran Wang, Fengyuan Wang, Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing-Hong Cao, Ming-xing Luo, Yaodong Yang, Muhan Zhang, Hua Xing Zhu

arXiv:2504.16074v2cs.CL

TL;DR

Existing LLM reasoning benchmarks can use oversimplified tasks, contain problems models may have seen during pretraining, and include flawed questions or scoring criteria. PHYBench introduces 500 original, curated physics problems and the EED Score; evaluations find that Gemini 2.5 Pro reaches 36.9% accuracy against a 61.9% human baseline.

  • Problem

    Existing benchmarks may oversimplify reasoning tasks, risk data contamination, or contain flawed questions and scoring criteria that add noise to evaluation.

    Source: §1 · p. 2

  • Method

    PHYBench is a human-curated benchmark of original physics problems, paired with EED, a tree-edit-distance metric for comparing generated and reference expressions.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    36.9% accuracy for Gemini 2.5 Pro versus 61.9% for the human baseline; PHYBench also differentiates model reasoning abilities more strongly than widely used benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LLMs struggle with multi-step and multi-condition inference, often introducing incorrect equations and failing to identify or correct such errors.

    Source: §6 · p. 10

    The problems focus primarily on Olympiad-level difficulty with uneven topic coverage, limiting generalization to research-level reasoning; EED assesses final answers rather than the full reasoning process.

    Source: §6 · p. 10

Abstract

from arXiv · show

Current benchmarks for evaluating the reasoning capabilities of Large Language Models (LLMs) face significant limitations: task oversimplification, data contamination, and flawed evaluation items. These deficiencies necessitate more rigorous assessment methods. To address these limitations, we introduce PHYBench, a benchmark of 500 original physics problems ranging from high school to Physics Olympiad difficulty. PHYBench addresses data contamination through original content and employs a systematic curation pipeline to eliminate flawed items. Evaluations show that PHYBench activates more tokens and provides stronger differentiation between reasoning models compared to other baselines like AIME 2024, OlympiadBench and GPQA. Even the best-performing model, Gemini 2.5 Pro, achieves only 36.9% accuracy compared to human experts' 61.9%. To further enhance evaluation precision, we introduce the Expression Edit Distance (EED) Score for mathematical expression assessment, which improves sample efficiency by 204% over binary scoring. Moreover, PHYBench effectively elicits multi-step and multi-condition reasoning, providing a platform for examining models' reasoning robustness, preferences, and deficiencies. The benchmark results and dataset are publicly available at https://www.phybench.cn/.

1 Introduction

PHYBench and the EED Score address shortcomings in existing reasoning benchmarks through original, curated physics problems and fine-grained expression scoring. Evaluations find a substantial model–human accuracy gap and errors in models’ intermediate reasoning.

Benchmark contribution

  • PHYBench uses original, human-curated physics problems to assess physical perception and robust reasoning across topics and difficulty levels.The problems span high school through undergraduate and Physics Olympiad levels; the curation pipeline aims to ensure they are novel, correct, and reliably evaluable.Source: §1 · p. 2

Results

  • 36.9% accuracy for Gemini 2.5 Pro versus 61.9% for the human baseline shows a substantial performance gap on PHYBench.PHYBench also requires more output tokens and yields lower model scores than widely used benchmarks, while differentiating reasoning abilities more strongly.Source: §1 · p. 2

Metric contribution

  • The EED Score compares generated and reference expressions using tree edit distance and improves sample efficiency by 204% on PHYBench.The authors present it as an interpretable, fine-grained alternative to binary correctness scoring.Source: §1 · pp. 2–3

Reasoning analysis

  • Models often introduce incorrect conditions or equations during intermediate reasoning and have limited ability to detect or correct these mistakes.The authors contrast this with the consistent self-checking behavior seen in human reasoning.Source: §1 · pp. 2–3

2 Related Work

Related work identifies concerns about saturation, originality, and scoring fidelity in existing reasoning benchmarks. PHYBench and EED are positioned as responses to gaps in complex physical reasoning evaluation and partial-credit assessment.

Reasoning Benchmarks

  • As models approach saturation on GSM-8K, Math-500 [13], and MMLU, benchmark gains and potential overfitting have become concerns [6].HLE [8] emphasizes scientific knowledge coverage, while OlympiadBench [10] and AIME 2024 use more complex problems that may lack originality and risk pretraining contamination.Source: §2 · p. 3

Evaluation Metrics for Complex Reasoning Tasks

  • Multiple-choice and simple-answer formats, including SuperGPQA [7] and MMLU, can be solved through elimination or pattern matching rather than genuine reasoning.Human judgments are costly to scale, while model-assisted evaluation can be biased and unstable.Source: §2 · p. 3
  • Binary expression or number scoring used by OlympiadBench [10] and AIME 2024 overlooks partial correctness, motivating EED’s fine-grained comparison of symbolic expressions.EED uses SymPy expression trees and extended tree edit distance to compare generated and reference answers.Source: §2 · p. 3

3 The PHYBench Benchmark

PHYBench evaluates LLM reasoning through original physics problems designed around realistic scenarios and symbolic answers. Its curation process screens problems for clarity and validity, and its EED metric provides a fine-grained comparison of generated and reference expressions.

3.1 Overview

  • PHYBench contains 500 originally curated questions across mechanics, electromagnetism, thermodynamics, optics, modern physics, and advanced physics.Questions require symbolic expressions for target physical quantities, and most involve long-chain reasoning in realistic scenarios rather than exhaustive search spaces.Source: §3.1 · pp. 3–4

3.2 Benchmark Curation

  • Problems adapted from human physics exercises range from high school to Physics Olympiad difficulty, with 178 Peking University physics students contributing to their development and refinement.The benchmark’s questions are text-only, based on realistic scenarios, and designed to specify a target quantity symbolically.Source: §3.2 · p. 4
  • Problem construction requires symbolic-expression answers and precise statements with defined variables, unique solutions, and no unstated assumptions or external knowledge.Equivalent expression forms are allowed, while equations and floating-point approximations are rejected.Source: §3.2 · pp. 4–5
  • Expert reviewers iteratively check and refine drafted questions, and human evaluation retained 500 of 757 questions from the Reviewer’s Library.LLM outputs assist reviewers in finding ambiguity but are not used to construct questions; 81 students evaluated clarity, uniqueness, and ambiguity.Source: §3.2 · p. 5

3.3 Evaluation Metric

  • EED compares regularized expression trees from generated and reference answers using normalized node-level edit distance and an extended Zhang-Shasha algorithm.The metric’s scoring function awards partial credit for minor structural or coefficient errors and uses lower-cost subtree edits for larger subtrees.Source: §3.3 · pp. 5–6
  • A 500-problem benchmark scored with EED achieves discriminative power comparable to 1,500 problems scored with traditional accuracy-based scoring.The authors also report stability across tested scoring-function baseline and penalty-coefficient settings.Source: §3.3 · p. 6

4 Experiments

PHYBench evaluations compare LLMs with a human baseline, other reasoning benchmarks, and test-time scaling strategies. The results show a substantial human–model gap, higher token use and lower scores than other benchmarks, and distinct model behavior under scaling.

4.1 Experiment Setup

  • The evaluation covers state-of-the-art and representative LLMs, uses accuracy and EED Score, and includes a recruited student human baseline.API models use service defaults; local evaluations use temperature 0.6, top_p 0.95, and max_tokens 32,768.Source: §4 · p. 6, §4.1 · p. 6

4.2 Human Baseline

  • 81 Peking University physics students, including 50 Chinese Physics Olympiad gold medalists, averaged 61.9 ± 2.1% accuracy and 70.4 ± 1.8 EED Score.At the 99% confidence level, experts outperformed all evaluated LLMs on both metrics.Source: §4.2 · p. 6

4.3 Main Results

  • 36.9% accuracy and an EED Score of 49.5 for Gemini 2.5 Pro remain below human baselines of 61.9 ± 2.1% accuracy and 70.4 ± 1.8 EED Score.The human results are based on 559 valid answer sheets from 81 Peking University physics students.Source: §4.3 · p. 6, §4.2 · p. 6
  • 204% average sample-efficiency improvement for EED Score means 500 problems provide discriminatory power comparable to approximately 1,500 binary-accuracy problems.Accuracy and EED Score yield nearly identical model rankings, but EED has a broader score distribution and lower statistical uncertainty.Source: §4.3 · p. 7

4.4 Comparison with Other Benchmarks

  • PHYBench requires significantly more output tokens and yields lower model scores than other benchmarks, with a larger reasoning-versus-general-model performance gap.The comparison is shown in Figure 4; the authors attribute the pattern to PHYBench's greater complexity and difficulty.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5 Test Time Scaling on PHYBench

  • As sampled responses k increase, pass@k accuracy improves smoothly while preserving model ordering; majority voting also retains pronounced capability separation.The reported extrapolations show Gemini 2.5 Flash closing the gap with Gemini 2.5 Pro, while DeepSeek-R1 continues to outperform o4-mini more clearly.Source: §4.5 · p. 7, Figure 5 · p. 8

5 Error Analysis

The analysis localizes most observed mistakes to robust reasoning after the physical scenario is understood, especially semantic derivation. A perturbation study distinguishes superficial, genuine, and pseudo-genuine reasoning and finds that robustness can rely on compensatory checks rather than semantic understanding.

5.1 Stage-wise Failure Localization

  • Typically more than 90% of observed errors occur during robust reasoning, and over 90% of those errors are semantic rather than symbolic.The analysis classifies first mistakes as physical perception or robust reasoning, then divides robust-reasoning errors into semantic and symbolic categories.Source: §5.1 · p. 8
  • Models often struggle to derive new, non-entailed equations from physical context, although they are generally more reliable with given conditions and symbolic manipulation.The authors illustrate this with incorrectly assuming angular momentum conservation despite external magnetic-field torques.Source: §5.1 · p. 8

5.2 Superficial Reasoning and Robustness of Reasoning

  • The perturbation experiment injects errors into partial solution traces and identifies superficial, genuine, and pseudo-genuine reasoning from how models continue.Blindly continuing a flawed trace signals superficial reasoning; genuine reasoning detects and repairs errors, while pseudo-genuine checks offer only partial robustness.Source: §5.2 · pp. 8–9
  • Recent reasoning models show greater perturbation robustness largely through compensatory strategies, while the semantic-competence gap remains and PHYBench offers a testbed for studying it.DeepSeek-R1 uses symbolic checks, while Gemini 2.5 Pro shifts toward formal derivations; the authors describe these as not addressing the core semantic bottleneck.Source: §5.2 · p. 9

6 Conclusion and Limitations

PHYBench is presented as a 500-problem physics benchmark, and the paper reports that current LLMs struggle with multi-step, multi-condition inference. The conclusion also notes limits on research-level generalization and on EED’s ability to assess full reasoning.

6 Conclusion and Limitations

  • PHYBench contains 500 carefully curated physics problems and is reported to be challenging, robust under TTS, and effective at differentiating models.The paper also reports that state-of-the-art models fall far behind human experts.Source: §6 · p. 9
  • LLMs struggle with multi-step, multi-condition inference, often introducing incorrect equations and failing to identify or correct them.Source: §6 · p. 10
  • The benchmark’s Olympiad-level focus and uneven topic distribution limit generalization to research-level reasoning, while EED assesses final answers rather than the full reasoning process.Source: §6 · p. 10

7 Contributions and Acknowledgements

The acknowledgements describe institutional support and broad student participation in PHYBench’s development. Team members contributed across research-pipeline construction, data work, evaluation, analysis, and writing.

7 Contributions and Acknowledgements

  • PHYBench received support from Peking University’s School of Physics, the Ministry of Education Physics 101 Plan, and the National Science Foundation of China.More than a hundred School of Physics students participated.Source: §7 · p. 10
  • The listed team contributions span research-pipeline construction, data annotation, and data-quality inspection.Source: §7 · p. 10

Core Contributors

  • The core-contributor lists name Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, and Qi Liu.Source: Core Contributors · p. 10
  • Core contributors also handled model evaluation, result analysis, and paper writing.Source: Core Contributors · p. 10

List of appendices

This appendix examines shortcomings in existing reasoning benchmarks, including saturation, possible contamination, and validation problems. Its GPQA examples illustrate how questions can emphasize factual knowledge or require interpreting a physics setup and manipulating equations.

A Detailed Analysis of Limitations in Existing Reasoning Benchmarks

A.1 Oversimplified Reasoning Tasks

  • On saturated benchmarks, small score differences can be less meaningful, and formatting-sensitive answers may obscure reasoning ability; corrected formats made several models’ MATH-500 answers entirely correct.Source: §A.1 · p. 14
  • The selected GPQA questions are described as testing factual knowledge more than long or complex reasoning chains, despite their uncommon topic coverage.Source: §A.1 · p. 14

GPQA Selected Problem–Astrophysics

GPQA Selected Problem–Quantum Mechanics

A.2 Potential Data Contamination

The appendix discusses benchmark contamination and verification, then details EED’s expression-tree scoring and its rationale. It argues that partial-credit scoring can better distinguish models in low-score settings than binary evaluation.

A.2 Potential Data Contamination

  • Publicly available benchmark content may have appeared in model pretraining; Gemini 2.5 Flash’s 100% AIME 2024 accuracy and average score above 99% across 16 runs raise memorization concerns.Source: §A.2 · p. 15

Problem 1216–Wrongly extracted answer

A.3 Lack of Rigorous Verification

  • 14 of 36 sampled OlympiadBench physics problems with symbolic reference answers had questionable answer quality, including ambiguous phrasing or answer-extraction errors.Two examined problems had concrete flaws: one included an unstated quantity, and another’s incorrect ground truth caused mostly correct outputs to be judged false.Source: §A.3 · p. 15

B.1 Tree Editing Distance Algorithm

  • EED compares parsed and simplified expression trees using edit distance; the appendix describes extracting the boxed answer, normalizing LaTeX, and converting expressions for symbolic comparison.Tree nodes represent mathematical components, and the algorithm uses insertions, deletions, and updates.Source: §B.1 · pp. 17–18, Figure 6 · p. 18
  • EED gives partial credit for partly correct expressions and discounts insertion or deletion of subtrees larger than five nodes to 60% of standard operation cost.The relative edit distance determines the score, while formatting, conversion, or computation errors return zero.Source: §B.1 · p. 18, Figure 7 · p. 19

B.2 Qualitative Interpretations for Advantages of the EED Score

  • The analysis models score as a function of ability minus difficulty, with expected benchmark scores obtained by combining the scoring function and difficulty distribution.It characterizes discrimination by the derivative of score with respect to ability.Source: §B.2 · pp. 19–20
  • The paper’s model illustrates why binary scoring can have low discrimination when ability is well below average difficulty, while partial-credit EED improves score linearity and discrimination in that range.Figure 8 qualitatively presents the comparison.Source: §B.2 · pp. 19–20, Figure 8 · p. 20

B.3 Limitations and Future Work of the EED Score

EED scoring improves evaluation efficiency, while its expression-based assessment has limits: it does not verify full reasoning or physical plausibility. Pairwise confidence analysis distinguishes model performance tiers.

B.3 Limitations and Future Work of the EED Score

  • EED assesses similarity between final expressions, not correctness across the full reasoning process; process-level evaluation is costly and physics solution paths can be non-unique.Prior work’s high-quality manual process evaluation is typically limited to fewer than 10 complex problems.Source: §B.3 · p. 20
  • Equal weighting of expression-tree nodes misses physical plausibility, including dimensional correctness; unit analysis or dimensional validation are proposed future additions.These checks could improve assessment beyond structural similarity.Source: §B.3 · p. 20

C.1 Efficiency and Advantage Confidence

  • 204% average sample-efficiency enhancement (σ = 80%) lets EED-scored 500 problems provide evaluation strength equivalent to approximately 1500 accuracy-scored problems.The analysis reports higher absolute EED scores and smaller relative uncertainties than accuracy scoring, while model rankings remain consistent across both metrics.Source: §C.1 · p. 21
  • Gemini 2.5 Pro outperforms most models with 99% confidence, except o3 at 90%, and the results separate evaluated models into performance tiers.The reported tiers are top performers (Gemini 2.5 Pro, o3, o4 mini), mid-tier (DeepSeek-R1, o3 mini), non-reasoning (GPT-4.1, DeepSeek-V3), and legacy non-reasoning (GPT-4o).Source: §C.1 · p. 22

C.2 Robustness Test on EED Scoring Metric

The robustness test varies EED scoring parameters and finds that model rankings are generally stable. EED retains sample-efficiency gains over accuracy, while higher baseline scores can create an apparent gain through variance reduction.

C.2 Robustness Test on EED Scoring Metric

  • The robustness test varies baseline score s0, penalty coefficient k, and whether subtree discounting is enabled.The modified scoring function is evaluated for ranking variation and sample efficiency.Source: §C.2 · p. 22
  • Most model rankings remain stable across parameter configurations, with only a few shifts within ±1 rank; subtree discounting has no significant effect on overall rankings.Table 6 reports ranking changes against the main-text configuration and includes pairwise confidence and sampling efficiency relative to accuracy.Source: Table 6 · p. 22, §C.2 · p. 22
  • EED scoring methods significantly improve sampling efficiency over accuracy across parameter variations; higher baseline scores’ apparent gain is an artifact of variance reduction.The variance reduction shifts non-perfect scores toward the full-score direction.Source: §C.2 · p. 22

D Evaluation Experiment Setup

Models receive a common prompt asking for an expert, step-by-step physics solution. The setup extracts a single final expression from a boxed answer.

D Evaluation Experiment Setup

E TTS on Various Benchmarks

This section examines model performance across benchmarks through pass@k and majority-voting accuracy, sampling-curve estimates, EED scoring, and analyses of physical-perception and long-range-reasoning failures. It also presents a thermodynamics derivation, a perturbation prompt template, and examples of reported errors and PHYBench problems spanning mechanics, thermodynamics, radiation, and electromagnetism.

E TTS on Various Benchmarks

  • Evaluations sampled 100 PHYBench questions, all 30 AIME 2024 questions, and 72 each from OlympiadBench, MATH500, and GPQA; models ran at least 16 times per question.OlympiadBench samples split evenly between math and physics, with expression-answer physics problems; measurements included pass@k and majority-vote accuracy and EED scores.Source: E TTS on Various Benchmarks · p. 23

E.1 Pass@k

  • Exponential-decay curves align well with the sampling data, supporting credible upper-bound fits, while curves for lower-scoring models show a notably linear trend.The fitting parameters represent the upper bound, achievable improvement, and decay rate; fits for models A, B, and C appear in Table 7.Source: §E.1 · pp. 23–24, Table 7 · p. 24

E.2 Majority Voting

  • On PHYBench, majority voting improves accuracy by only a few percentage points, while pass@k gains often exceed dozens across reasoning and non-reasoning models, exposing weak self-evaluation.The authors infer that correct answers occur in model outputs but models struggle to recognize them.Source: §E.2 · p. 24, Figure 11 · p. 25

F.2 Case Study of PP

  • Physical-perception failures can undermine correct symbolic derivation: models misread the kinematics of connected balls, and even Gemini 2.5 Pro, o3, DeepSeek-R1, and 32B models failed on the problem.The authors characterize PP as decision nodes in the reasoning chain; errors can cause fundamental scenario misunderstandings or introduce unnecessary effects.Source: §F.2 · p. 26, Example Reasoning Process · p. 26, Figure 12 · p. 26

F.3 Case Study of RR

  • Models may correctly interpret a physical setup yet fail during extended symbolic derivation; PHYBench solutions average about 3,000 characters and require dozens of intermediate expressions from human solvers.The authors note that unaided models tend to produce more intermediate steps than human references, increasing opportunities for error.Source: §F.3 · p. 27

G Chain-of-Thought Poisoning Protocol

  • The chain-of-thought poisoning protocol tests robustness and genuine versus superficial reasoning by truncating reference solutions, injecting perturbations, and asking models to continue across eight conditions.The catalogue includes removing a square term, reversing an operator, combining errors, deleting an h term, and altering a physical law.Source: §G.1 · p. 27, §G.2 · p. 28

G.2 Perturbation Catalogue

  • DeepSeek-R1 accuracy fell from 97 % to 37 % when the h term was removed, while conflicting cues in a combined perturbation increased models’ aggregate error-flagging scores.Removing h was dimensionally sound but conceptually obvious; the combined perturbation joined two conflicting cues.Source: §G.2 · pp. 27–28

Genuine Reasoning: correct the answer through semantic understanding

Problem 719 (THERMODYNAMICS)

G.5 Implementation Prompt Template

  • The perturbation template requests a step-by-step solution with the final answer boxed in LaTeX, then instructs T1–T6 variants to continue rather than restart reasoning.The shown dialogue supplies the problem content to the model and appends a continuation request after the poisoned chain of thought.Source: §G.5 · p. 32

H.1 Full Question Text for Given Errors in Figure 12

  • Two examples associated with Figure 12 errors concern the middle-string tension after a ball is struck and the final velocity of a photon-irradiated relativistic mirror.The mirror problem specifies initial and final velocities β0c and β1c, with incident photons delivering energy E.Source: §H.1 · p. 32

H.2 Demonstration of Selected Problems

  • The demonstration section presents five additional problems with answers and directs readers to the PHYBench website for more detail.Source: §H.2 · p. 32

Selected Problem 1

  • Selected problems include a rod-in-bowl angular-acceleration question, a swing with impulsive torque and friction, and radiation absorbed by an eccentric-cylinder system.The swing asks for the speed difference after successive passages through the lowest point at steady state; the radiation problem asks for absorbed power density by angle.Source: Selected Problem 1 · p. 33
  • Another selected problem asks for the velocity over time of a moving square resistive loop in a magnetic field, neglecting gravity.The requested expression uses B0, v0, a, m, and R.Source: Selected Problem 1 · p. 33

Selected Problem 5

  • The electromagnetic-cannon problem asks for the Ampere force on a sliding rod when a thin-walled cylindrical rail carries uniformly distributed direct current.Its setup assumes rail radius r ≪ l and circuit self-inductance L = Ax + B.Source: Selected Problem 5 · p. 34

Related papers

Loading 2504.16074v2…