PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning

Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, Jun Liu

arXiv:2502.12054v2cs.AI

TL;DR

Physics-based reasoning remains underrepresented in LLM evaluation despite its practical relevance and need to combine physics theorems and constraints. PhysReason evaluates this capability using stratified problems and answer- and step-level scoring, finding that performance falls as difficulty and solution depth increase.

  • Problem

    Existing LLM evaluations often overlook physics-based reasoning, despite its integration of multiple theorems and constraints and relevance to practical applications.

    Source: §1 · p. 1

  • Method

    PhysReason evaluates physics reasoning with 1,200 stratified problems and a scoring framework offering answer-level and step-level evaluation.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    Model performance declines as difficulty and required solution steps increase, from 75.11% to 31.95%.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    Step-level evaluation distinguishes model capabilities more clearly than answer-level evaluation, especially on challenging problems.

    Source: §5.2 · p. 7

    PhysReason tests applying and reasoning with physics theorems under idealized conditions rather than fully reflecting real-world physics scenarios.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Large language models demonstrate remarkable capabilities across various domains, especially mathematics and logic reasoning. However, current evaluations overlook physics-based reasoning - a complex task requiring physics theorems and constraints. We present PhysReason, a 1,200-problem benchmark comprising knowledge-based (25%) and reasoning-based (75%) problems, where the latter are divided into three difficulty levels (easy, medium, hard). Notably, problems require an average of 8.1 solution steps, with hard requiring 15.6, reflecting the complexity of physics-based reasoning. We propose the Physics Solution Auto Scoring Framework, incorporating efficient answer-level and comprehensive step-level evaluations. Top-performing models like Deepseek-R1, Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on answer-level evaluation, with performance dropping from knowledge questions (75.11%) to hard problems (31.95%). Through step-level evaluation, we identified four key bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation, and Physics Condition Analysis. These findings position PhysReason as a novel and comprehensive benchmark for evaluating physics-based reasoning capabilities in large language models. Our code and data will be published at https:/dxzxy12138.github.io/PhysReason.

1 Introduction

PhysReason targets the under-evaluated challenge of physics-based reasoning, which combines theorems and constraints and is relevant to practical applications. It pairs a multi-step benchmark with answer- and step-level evaluation.

Motivation

  • Existing physics benchmarks, including ScienceQA (Lu et al., 2022), SciBench (Wang et al.), and GPQA (Rein et al., 2024), often use oversimplified reasoning and final-answer-only evaluation.They typically involve 3–4 physics formulas, motivating a benchmark with deeper reasoning and step-level evaluation.Source: §1 · p. 1

Benchmark

  • PhysReason contains 1,200 problems: 25% knowledge-based and 75% reasoning-based, with reasoning problems split evenly across easy, medium, and hard; 81% include diagrams.Solutions average 8.1 steps, rising to 15.6 for hard problems; the paper contrasts this with benchmarks that typically contain 3–4 steps.Source: §1 · p. 1

Evaluation framework

  • PSAS provides efficient answer-comparison (PSAS-A) and comprehensive step-by-step verification (PSAS-S), achieving evaluation accuracy above 98% and outperforming direct LLM-based evaluation.The framework is based on current LLMs’ information-extraction and formula-comparison capabilities.Source: §1 · p. 2

Results

  • Top models Deepseek-R1 (Guo et al., 2025), Gemini-2.0-Flash-Thinking-0121 (Deepmind), and o3-mini-high (OpenAI, 2025) average below 60%; performance falls from 75.11% to 31.95% as difficulty and required steps rise.The authors attribute this decline to models’ difficulty maintaining accuracy across consecutive solution steps.Source: §1 · p. 2
  • Step-level evaluation identifies four performance bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation Process, and Physics Condition Analysis.The authors highlight maintaining reasoning-process reliability as crucial.Source: §1 · p. 2

2 Related Work

Prior evaluations emphasize mathematical and logical reasoning, while physics reasoning requires integrating multiple principles with real-world constraints. Existing physics benchmarks cover varied knowledge levels but simplify reasoning and focus on final answers, gaps PhysReason addresses through complex reasoning and step-level evaluation.

2.1 Large Language Model Evaluation

  • LLMs perform well in math, logic, and text generation, but physics interactions expose limitations and remain underassessed despite requiring multiple principles and real-world constraints.These limitations constrain applications such as autonomous driving and robotics, while robust physics reasoning is described as a prerequisite for broader practical use.Source: §2.1 · p. 2

2.2 Physics Benchmarks

  • Physics benchmarks span K-12, college, and expert knowledge, yet typically simplify reasoning to 3-4 steps and emphasize final answers; PhysReason targets complex processes and step-level evaluation.Examples across these levels include ScienceQA and E-EVAL, MMLU and JEEBench, and OlympiadBench and GPQA.Source: §2.2 · p. 2

3 Benchmark

PhysReason is built from publicly sourced physics problems, standardized and annotated with structured solutions, then organized to evaluate physics knowledge and reasoning. Its 1,200 problems span 147 theorems and feature longer solution chains and many diagram-based questions.

3.1 Collection

  • Collection follows five stages: acquisition, standardization, translation, search prevention, and difficulty classification.Problems are drawn from public entrance exams, practice tests, and international competitions; PDFs are parsed and deduplicated, and expert candidates verify translations. Problems searchable with solutions or answers within five minutes on Google are excluded (Rein et al., 2024).Source: §3.1 · p. 3

3.2 Annotation

  • The annotation framework has eight elements: Diagram, Context, Sub-questions, Solution, Step Analysis, Answer, Theorem, and Difficulty.Step Analysis identifies the theorem used and quantities obtained at each step; Solution provides step-by-step reasoning for each sub-question.Source: §3.2 · p. 3

3.3 Characteristics

  • The 1,200 problems cover 147 physics theorems, with 25% knowledge-based and 75% reasoning-based questions; reasoning problems are split evenly across easy, medium, and hard.Topics include classic and quantum mechanics, fluid mechanics, thermodynamics, electromagnetics, optics, and relativity.Source: §3.3 · pp. 3–4
  • Solutions average 8.1 steps per problem, while hard problems require up to 15.6 steps; theorem, step, and token counts rise with difficulty.The authors report that medium and hard problems demonstrate higher complexity than existing benchmarks.Source: §3.3 · p. 4
  • Diagrams appear in 81% of problems, supporting evaluation of textual and visual reasoning in physics scenarios.The benchmark’s solution metrics also show greater complexity for medium and hard problems than existing physics-based reasoning benchmarks.Source: §3.3 · p. 4

4 Evaluation Framework

The Physics Solution Auto Scoring Framework (PSAS) uses LLMs for answer-level and step-level evaluation of open-ended physics solutions. Its step-level framework maps model reasoning to annotated steps, scores theorem application and calculations, and identifies the earliest error for analysis.

4.1 Why LLMs Can Evaluate?

  • LLMs support automated evaluation because they can extract precise content and assess formula consistency across open-ended answers and steps with diverse expressions.The framework offers answer-level and step-level evaluations to avoid labor-intensive manual assessment.Source: §4.1 · p. 4

4.2 How Answer-Level Evaluation Works?

  • PSAS-A extracts each sub-question answer, checks semantic consistency with its standard answer, and weights the score by the annotated solution length.This weighting prevents sub-questions requiring different numbers of solution steps from contributing equally.Source: §4.2 · p. 4

4.3 What is the Step?

  • Each annotated step must form a complete, independently evaluable unit that advances the solution and includes a theorem-derived formula with related calculations.Source: §4.3 · p. 4

4.4 How Step-Level Evaluation Works?

  • PSAS-S extracts model steps aligned with annotations, scores formula application and numerical results equally, then locates and analyzes the first deviation.It classifies the first error among seven types: diagram analysis, theorem application, condition analysis, process understanding, variable relationship, calculation process, and boundary condition analysis.Source: §4.4 · pp. 5–6

4.5 Whether Evaluation Trustworthy?

  • PSAS-A and PSAS-S outperform direct LLM evaluation on answer accuracy and first-error-step identification in Chain-of-Thought tests.The comparison uses Deepseek-V3 and Gemini-2.0-Flash as scoring models, with 1,000 manually annotated reasoning processes sampled from 8,400 generated processes.Source: Table 2 · p. 6, §4.5 · p. 6

5 Experiments

Experiments evaluate 15 models with zero-shot Chain-of-Thought prompting and the PSAS framework, then analyze benchmark performance, test-time scaling, error correction, and reasoning errors. Results show differences by model type and difficulty, and identify persistent challenges in sustaining accurate physics reasoning.

5.1 Setting

  • The study assessed 15 models under zero-shot Chain-of-Thought prompting, used Gemini-2.0-Flash captions for models without visual input, and selected Deepseek-V3 as the PSAS scoring model.PhysReason-mini was a balanced random sample of 200 questions, with 50 from each difficulty level.Source: §5.1 · p. 6

5.2 Main Results

  • Multiple O-like models exceeded 50% answer-level accuracy versus the non-O-like peak of 47.88%; performance declined with difficulty, while step-level evaluation better separated models on harder problems.Step-level scores also exceeded answer-level scores, showing that models can get some steps right without reaching the correct final answer.Source: Table 3 · p. 7, Table 4 · p. 7, §5.2 · p. 7
  • Physics knowledge positively correlated with reasoning; Deepseek-R1 and Gemini-2.0-Flash-Thinking-0121 excelled in both, while O-like models tended to outperform similarly knowledgeable models on reasoning.The authors conclude that effective reasoning relies on knowledge capacity and model architecture.Source: §5.2 · pp. 7–8

5.4 Performance Improving with PSAS-S

  • Direct concatenation reduced performance by 3–5%, whereas guided error localization improved it by 3–6%, supporting the value of identifying the first reasoning error before a second attempt.Guided error localization combines the question and prior reasoning with PSAS-S’s location and analysis of the first error.Source: Table 6 · p. 8, §5.4 · p. 8

5.3 Results with Test-Time Compute Scaling

5.5 Error Kind Distribution Analysis

  • Physics theorem application, process understanding, calculation, and condition analysis challenged all models; o1 and o3-mini-high had more process-understanding errors but fewer calculation errors.The authors suggest this pattern may reflect a trade-off between conceptual comprehension and computational precision.Source: Figure 4 · p. 8, §5.5 · p. 8

5.6 Hard Problem Analysis

  • Across 50 hard problems and seven models, model-specific strengths coexisted with partial solution paths that failed to remain accurate through extended reasoning.Some models earned partial scores below 1, indicating they could start a correct solution path but not sustain it.Source: §5.6 · pp. 8–9, Figure 5 · p. 9

6 Conclusion

The conclusion presents PhysReason as a physics-based reasoning benchmark paired with an auto-scoring framework, and reports that performance declines as reasoning depth increases.

6 Conclusion

  • PhysReason is introduced as a novel benchmark for physics-based reasoning.Source: §6 · p. 9
  • The Physics Solution Auto-Scoring Framework evaluates answers at both the answer and step levels.Source: §6 · p. 9
  • Experimental results show performance consistently declines as reasoning depth increases, positioning the benchmark as a standard for evaluating and improving AI models’ physics-based reasoning.Source: §6 · p. 9

Limitation

PhysReason evaluates physics-theorem reasoning mainly under idealized conditions, and its step-level evaluation adds computational time despite achieving over 98% accuracy with LLM assessment tools.

Limitation

  • PhysReason primarily tests applying and reasoning with physics theorems under idealized conditions rather than fully reflecting real-world physics scenarios.The authors note that idealized theorem application is a foundation for more complex real-world scenarios.Source: Limitation · p. 9
  • The authors argue that PhysReason remains valuable because current LLM performance on idealized conditions is unsatisfactory, and data synthesis could adapt problems to real-world scenarios.Adapting problems to real-world physics reasoning is identified as future research.Source: Limitation · p. 9
  • Over 98% accuracy with LLM assessment tools does not eliminate the PSAS-S framework’s increased evaluation time, which future work will seek to optimize while maintaining accuracy.Source: Limitation · p. 9

Ethical Statement

PhysReason’s ethical practices address data quality, bias, leakage, licensing, and environmental impact through its sourcing, curation, release, and experimental compliance.

Ethical Statement

  • The benchmark uses public official materials and undergoes cleansing, deduplication, and standardization to ensure reliability while minimizing bias and data leakage.Sources include IPhO, Gaokao, JEE, and authorized mock exams.Source: Ethical Statement · p. 9
  • PhysReason releases datasets and scripts under MIT and CC BY-NC-SA licenses to reduce unnecessary carbon footprint, while optimizing pipelines to lower computational overhead.Source: Ethical Statement · p. 9
  • All experiments strictly comply with the licenses for models and data.Source: Ethical Statement · p. 9

A Data Sources … B.2 Annotation

PhysReason draws on diverse public physics assessment materials and documents a curation pipeline spanning acquisition, standardization, translation, search prevention, classification, and detailed solution annotation. The annotation framework specifies problem conditions and a structured, formula-supported solution process.

A Data Sources

  • Sources span IPhO problems (2008–2021), Gaokao physics questions (2010–2024), Indian JEE Advanced problems (2010–2024), and Chinese provincial and municipal mock exams (2015–2024).The sources include competition and entrance-exam problems as well as mock exams from multiple administrative levels.Source: A Data Sources · pp. 12–13

B.1.1 Data Acquisition

  • The collection comprises 1,254 PDF documents totaling 27,874 pages and yields over 20,000 unique problems.Problems were systematically collected, curated, and processed from diverse sources.Source: §B.1.1 · p. 13

B.1.2 Data Standardization; 3. Accurate expressions and units

  • PDF parsing with MinerU (Wang et al., 2024a) supports conversion, deduplication, and formatting standardization, with quality checks for complete statements and consistent equations and symbols.The stated criteria include well-defined variables and conditions, accurate expressions and units, and consistent equation and symbol formatting.Source: §B.1.2 · p. 13, §3 · p. 13

B.1.3 Translation; Phase I: Translation; Phase II: Verification

  • Translation uses an initial translator phase followed by verification by Engineering Ph.D. candidates for terminology accuracy, semantic equivalence, and mathematical-expression consistency.The process standardizes physics terminology, mathematical notation, and expressions across Chinese, English, Hindi, and Russian content.Source: §B.1.3 · p. 13, Phase I: Translation · p. 13, Phase II: Verification · p. 14

B.1.4 Search Prevention

  • Problems answerable through a five-minute Google search are excluded to reduce data leakage and focus evaluation on physics problem-solving rather than information retrieval.The exclusion approach follows Rein et al. (2024).Source: §B.1.4 · p. 14

Difficulty Levels in Reasoning-based Questions

  • The difficulty criteria distinguish easy reasoning problems by 5–15-minute completion from hard problems requiring over 10 steps and 15+ minutes.The supplied passages state these thresholds but do not provide a medium-level threshold.Source: Difficulty Levels in Reasoning-based… · p. 14

B.2 Annotation

  • Annotation specifies the scenario, conditions and constraints, standard notation, connected subquestions, and a detailed solution that states applicable theorems and derives quantities.Steps should include relevant formulas and resulting values where possible, with numerical answers carrying units; annotations also record difficulty, applicable laws and formulas, and core concepts.Source: §B.2 · pp. 14–15

C Error Type Details · D Example · E Evaluation Prompt

The appendices detail the benchmark’s error taxonomy, illustrate the progression across four difficulty levels, and specify a zero-shot CoT prompt for standardized step-by-step solutions.

C Error Type Details

  • The taxonomy distinguishes diagram-reading mistakes from condition-analysis errors about system boundaries, forces, and components.Examples include misreading graph axes or trends, neglecting significant friction, and choosing the wrong system boundary.Source: C Error Type Details · pp. 15–16
  • Physics theorem application errors involve misremembering or misapplying laws, including using principles outside their valid conditions.Examples include applying Newton’s laws in a non-inertial frame without fictitious forces and using energy conservation when friction does significant work.Source: C Error Type Details · p. 15
  • Process-understanding errors concern how phenomena develop and events cause one another, while variable-relationship errors misstate dependencies among physical quantities.Examples include misunderstanding projectile motion and assuming acceleration is directly proportional to velocity.Source: C Error Type Details · p. 16
  • Calculation errors are mathematical manipulation mistakes, whereas boundary-condition errors neglect limiting cases, model ranges, or initial conditions.Examples include unit conversions and arithmetic, ignoring air resistance at high speeds, and applying a formula outside its validity range.Source: C Error Type Details · p. 16

D Example

  • Representative examples span knowledge, easy, medium, and hard questions, progressing from direct theorem use to multi-component, constrained, multi-step problem solving.The medium example requires tracking state changes across configurations; the hard example combines sequential interactions, multiple principles, constraints, and multi-step calculations.Source: D Example · p. 17

E Evaluation Prompt

  • The evaluation prompt uses zero-shot Chain-of-Thought with image information, problem context, and sequential sub-questions to elicit standardized step-by-step solutions.The number of sub-questions and solution steps can vary with problem requirements.Source: E Evaluation Prompt · p. 17, §E.2 · p. 17

E.1 Input Components

  • Input components include context and progressive sub-questions, with Gemini-2.0-flash generating image descriptions for models without direct image-processing capability.The context supplies overall background and fundamental problem information.Source: §E.1 · p. 17

E.2 Output Specifications

  • Output specifications require step-by-step reasoning for each sub-question, continuous step numbering, and one formula with its solution process per step.Mathematical formulas are enclosed in LaTeX notation.Source: §E.2 · p. 17

F Details of Experimental Result · G Details of Scientific Artifacts

The appendix provides a comprehensive answer-level evaluation of PhysReason-mini and documents the benchmark’s sources, scale, licensing, privacy safeguards, and research-use access.

F Details of Experimental Result

G Details of Scientific Artifacts

  • The benchmark draws on physics problems from Olympiads, entrance examinations, provincial and municipal mock exams, and Russian sources.Its sources include the International Physics Olympiad (2008–2021), Chinese national entrance examinations (2010–2024), Indian JEE Advanced (2010–2024), and Chinese provincial and municipal mock examinations (2015–2024).Source: G Details of Scientific Artifacts · p. 18
  • The underlying collection contains over 20,000 unique physics problems from 1,254 PDF documents spanning 27,874 pages.Source: G Details of Scientific Artifacts · p. 18
  • The problems are curated for broad coverage, with materials used under CC BY-NC-SA and MIT licenses and collected according to fair-use principles for educational and research purposes.Source: G Details of Scientific Artifacts · p. 18
  • The dataset contains no personally identifiable information because its problems come from standardized tests and competition materials.Source: G Details of Scientific Artifacts · p. 18
  • The benchmark and code are available for research use, subject to MIT terms for the implementation and the original CC BY-NC-SA licenses for educational materials.Users must provide proper attribution and follow the respective license terms.Source: G Details of Scientific Artifacts · p. 18

H Details of Computational Experiment · I Details of human annotators · J Details of Ai Assistants In Research Or Writing

The appendices document the benchmark evaluation setup, annotators, and AI assistance, alongside example problems spanning knowledge and easy, medium, and hard levels.

H Details of Computational Experiment

  • Seventeen LLMs, VLMs, and specialized models were evaluated zero-shot with Chain-of-Thought on PhysReason, except o1, using PSAS-A and PSAS-S with Deepseek-V3 as scorer.Open-source models primarily ran on NVIDIA A800 GPUs; models unable to process images received captions generated by Gemini-2.0-Flash.Source: H Details of Computational Experiment · p. 18
  • Because PSAS-S was computationally costly, some experiments used only PSAS-A; prompt templates are provided in the supplementary materials.Source: H Details of Computational Experiment · p. 19

I Details of human annotators

  • Four graduate student co-authors with high-school and undergraduate physics backgrounds annotated and evaluated physics content without collecting personal data or involving external human subjects.The paper states that formal recruitment, compensation, and ethics review were not required.Source: I Details of human annotators · p. 19

J Details of Ai Assistants In Research Or Writing

Related papers

Loading 2502.12054v2…