PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning

Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, Jun Liu

arXiv:2502.12054v2cs.AI

TL;DR

Physics-based reasoning remains under-evaluated because it requires integrating multiple theorems and constraints. This paper introduces PhysReason and PSAS for benchmark construction and automated answer- and step-level assessment, finding that model performance consistently declines as reasoning depth increases.

  • Problem

    Physics-based reasoning requires integrating multiple theorems and physical constraints, but current evaluations often overlook this capability.

  • Method

    The paper introduces PhysReason, a 1,200-problem benchmark, and PSAS with answer-level and step-level evaluation methods.

  • Results

    Leading models remain below 60% on average, while performance declines from 75.11% on knowledge questions to 31.95% on hard problems.

  • Takeaways & Limitations

    Step-level evaluation identifies four bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation, and Physics Condition Analysis.

  • Takeaways & Limitations

    PhysReason primarily tests applying and reasoning with physics theorems under idealized conditions rather than fully reflecting real-world physics scenarios.

Abstract

from arXiv · show

Large language models demonstrate remarkable capabilities across various domains, especially mathematics and logic reasoning. However, current evaluations overlook physics-based reasoning - a complex task requiring physics theorems and constraints. We present PhysReason, a 1,200-problem benchmark comprising knowledge-based (25%) and reasoning-based (75%) problems, where the latter are divided into three difficulty levels (easy, medium, hard). Notably, problems require an average of 8.1 solution steps, with hard requiring 15.6, reflecting the complexity of physics-based reasoning. We propose the Physics Solution Auto Scoring Framework, incorporating efficient answer-level and comprehensive step-level evaluations. Top-performing models like Deepseek-R1, Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on answer-level evaluation, with performance dropping from knowledge questions (75.11%) to hard problems (31.95%). Through step-level evaluation, we identified four key bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation, and Physics Condition Analysis. These findings position PhysReason as a novel and comprehensive benchmark for evaluating physics-based reasoning capabilities in large language models. Our code and data will be published at https:/dxzxy12138.github.io/PhysReason.

1 Introduction

PhysReason addresses the limited evaluation of physics-based reasoning by combining multi-step, theorem-constrained problems with automated answer- and step-level assessment. Experiments show that even leading models degrade substantially as reasoning difficulty increases.

  • Physics-based reasoning integrates multiple theorems and physics constraints, but existing evaluations often overlook it despite its relevance to robotics and autonomous driving.
  • Existing physics benchmarks typically use 3-4 formulas and focus on final answers, leaving deep reasoning processes and step-level evaluation underrepresented.
  • PhysReason contains 1,200 problems, with 25% knowledge-based and 75% reasoning-based questions distributed across easy, medium, and hard levels.
  • Solutions average 8.1 steps per problem, while hard problems reach 15.6 steps, exceeding the complexity of current physics benchmarks.
  • PSAS provides efficient answer-level and comprehensive step-level evaluation, achieving evaluation accuracy exceeding 98%.
  • Leading models remain below 60% average performance, with scores declining from 75.11% on knowledge questions to 31.95% on hard problems.

2 Related Work

Prior physics benchmarks span educational levels but generally simplify reasoning and emphasize final answers. PhysReason targets these limitations by evaluating deeper, multi-step physics reasoning.

  • Large language models perform strongly in mathematics, logic, and text generation but remain limited in physics-based interactions involving physical constraints.
  • Existing benchmarks cover K-12, college-level, and expert-level physics knowledge across diverse datasets.
  • These benchmarks typically simplify reasoning to 3-4 steps and emphasize final answers rather than detailed solution processes.
  • Table 1 compares PhysReason with other physics benchmarks using knowledge level, question type, token length, and average solution steps.

3 Benchmark

PhysReason is a curated, stratified, multimodal benchmark built from diverse physics sources and annotated for theorem-based solution analysis. Its problems cover broad physics domains and substantially varying reasoning complexity.

  • The benchmark construction process includes acquisition, standardization, translation, search prevention, and difficulty classification.
  • Sources include 1,254 PDFs containing over 20,000 unique problems from examinations and international physics competitions.
  • Problems are parsed, deduplicated, filtered, formatted, translated, and reviewed by physics-specialized Ph.D. candidates.
  • Problems whose solutions or answers could be found through a five-minute Google search are excluded to reduce potential data leakage.
  • Difficulty classification distinguishes knowledge-based problems from reasoning-based problems at easy, medium, and hard levels.
  • Annotations include diagrams, context, sub-questions, solutions, step analyses, answers, theorems, and difficulty labels.
  • PhysReason contains 1,200 problems covering 147 physics theorems across mechanics, quantum mechanics, fluids, thermodynamics, electromagnetics, optics, and relativity.
  • Solution complexity averages 8.1 steps per problem and reaches 15.6 steps for hard problems, while 81% of problems include diagrams.

4 Evaluation Framework

The Physics Solution Auto Scoring Framework combines answer-level scoring with step-level extraction, verification, error localization, and classification. Validation against manual annotations shows that PSAS outperforms direct LLM evaluation.

  • Answer-Level Evaluation: PSAS-A extracts each sub-question answer and checks its semantic consistency with the standard answer, weighting sub-questions by annotated solution length.
  • Step Definition: Each annotated step is defined as a complete, independent, and progressive reasoning unit containing a theorem-derived formula and related calculations.
  • Step-Level Evaluation: PSAS-S evaluates reasoning through Data Extraction, Scoring, First Error Step Detection, and Error Analysis phases.
  • Step-Level Evaluation: Data Extraction maps relevant model-output steps to annotated steps while preserving semantic equivalence despite redundant reasoning.
  • Step-Level Evaluation: Scoring balances theorem application and numerical calculation accuracy through two components weighted at 0.5 each.
  • Step-Level Evaluation: First Error Step Detection identifies the earliest deviation by locating model-output steps corresponding to annotated steps with scores below 1.
  • Error Analysis: PSAS-S classifies errors into seven categories, including theorem application, physics-process understanding, calculation, and condition-analysis errors.
  • Validation: Manual validation on 1,000 reasoning processes shows that PSAS achieves superior evaluation performance compared with direct LLM evaluation.

5 Experiments

Experiments evaluate 15 models with PSAS across answer- and step-level metrics, revealing declining performance with deeper reasoning and distinct error patterns. Additional tests show that guided error localization improves second attempts, while step-level scoring better distinguishes difficult reasoning abilities.

  • 5.1 Setting: 15 advanced LLMs/VLMs are evaluated under zero-shot Chain-of-Thought, with PSAS applied at answer and step levels.The workflow encourages step-by-step reasoning and uses PSAS for both evaluation levels.
  • 5.2 Main Results: O-like models exceed non-O-like models, with multiple O-like models surpassing 50% answer-level accuracy versus a 47.88% non-O-like peak.
  • 5.2 Main Results: As difficulty and required solution steps increase, model performance severely declines, exposing weakness on deep physics reasoning.
  • 5.2 Main Results: Step-level scores exceed answer-level scores and become more discriminative as difficulty increases because models can complete some correct steps without reaching the final answer.
  • 5.2 Main Results: Knowledge and reasoning scores are positively correlated, with Deepseek-R1 and Gemini-2.0-Flash-Thinking-0121 excelling in both aspects.Among similarly knowledgeable models, O-like models tend to score higher on reasoning problems.
  • 5.4 Performance Improving with PSAS-S: Guided error localization improves second-attempt performance by 3-6%, whereas direct concatenation decreases it by 3-5%.PSAS-S identifies and analyzes the first reasoning error before the second attempt.
  • 5.5 Error Kind Distribution Analysis: Four prevalent error types challenge all models: Physics Theorem Application, Physics Process Understanding, Calculation Process, and Physics Condition Analysis.o1 and o3-mini-high show more process-understanding errors but fewer calculation-process errors, suggesting a trade-off between conceptual comprehension and computational precision.
  • 5.6 Hard Problem Analysis: Across 50 hard problems, models show domain-specific strengths but often begin correct solution paths without sustaining accuracy through extended reasoning.

6 Conclusion

The paper introduces PhysReason as a physics-based reasoning benchmark with stratified difficulty and the Physics Solution Auto-Scoring Framework. Results show performance consistently declines as reasoning depth increases, establishing a basis for evaluating and improving such capabilities.

  • 6 Conclusion: PhysReason combines stratified difficulty with the Physics Solution Auto-Scoring Framework for answer- and step-level evaluation.
  • 6 Conclusion: Performance consistently declines as reasoning depth increases, positioning PhysReason as a benchmark for evaluating and improving physics-based reasoning abilities.

Limitation

PhysReason has limitations in both benchmark construction and evaluation methodology. It primarily tests theorem application under idealized conditions, while step-level scoring is more computationally expensive.

  • Benchmark construction: The benchmark primarily evaluates physics-theorem application and reasoning under idealized conditions rather than fully realistic physics scenarios.The authors note that idealized conditions provide a foundation for more complex real-world scenarios.
  • Evaluation methodology: PhysReason’s evaluation framework achieves over 98% accuracy with LLM-based assessment tools but still has methodological limitations.
  • Evaluation methodology: PSAS-S provides satisfactory evaluation performance but increases computational time.

Ethical Statement

The authors describe measures intended to support data reliability, reduce bias and leakage, protect privacy, and limit environmental and licensing risks.

  • PhysReason uses public official materials and applies data cleansing, deduplication, and standardization to support reliability and minimize bias and data leakage.
  • The authors release datasets and scripts under MIT and CC BY-NC-SA licenses while optimizing processing pipelines to reduce computational overhead.
  • The experiments comply with applicable licenses for models and data, and the benchmark contains no personally identifiable information.

A Data Sources

PhysReason is assembled from diverse physics education and assessment sources, then curated into a standardized benchmark with structured, multi-step annotations. Its problems span knowledge-based and reasoning-based categories with progressively increasing complexity.

  • Data sources: The dataset draws from IPhO, Gaokao, Chinese mock examinations, JEE Advanced, and additional non-Chinese and English sources.These sources cover international competitions, standardized examinations, mock exams, and open physics collections.
  • Data curation: The collection includes 1,254 PDF documents totaling 27,874 pages and more than 20,000 unique physics problems.
  • Data curation: The processing pipeline performs PDF parsing, format conversion, deduplication, formatting standardization, and quality assessment.
  • Data filtering: Problems whose answers could be found through a five-minute Google search are excluded to reduce data leakage.
  • Question classification: Problems are categorized as knowledge-based or reasoning-based, with reasoning problems requiring multiple-theorem integration and multi-step problem solving.
  • Annotation framework: Annotations require comprehensive step-by-step solutions, formulas and numerical results where applicable, theorem-application rationales, conditions, constraints, and hierarchical sub-questions.

C Error Type Details

The error taxonomy separates failures in interpreting diagrams, applying theorems, analyzing conditions, understanding processes, relating variables, calculating, and handling boundary conditions. Representative problems and structured prompts support analysis across four difficulty levels.

  • Physics Theorem Application Errors: Physics Theorem Application Errors arise when models misremember, misapply, or use a theorem outside its valid conditions.
  • Physics Condition Analysis Errors: Physics Condition Analysis Errors concern incorrect system boundaries, forces, components, or assumptions about whether a system is isolated.
  • Physics Process Understanding Errors: Physics Process Understanding Errors reflect misunderstandings of state changes, causal relationships, motion, energy transformation, or force effects.
  • Variable Relationship Errors: Variable Relationship Errors involve misunderstanding dependencies or functional relationships between physics quantities.
  • Calculation Process Errors: Calculation Process Errors are mathematical mistakes in equation manipulation, unit conversion, arithmetic, substitution, or calculator use.
  • Boundary Condition Analysis Errors: Boundary Condition Analysis Errors involve neglecting special cases, limiting conditions, applicable ranges, initial conditions, or model limitations.

F Details of Experimental Result

The paper provides comprehensive model-performance evaluation on PhysReason-mini, including answer-level comparisons across questions, images, and image captions.

  • Performance Evaluation: The evaluation expands earlier partial benchmarks into a comprehensive comparison across all evaluated models.The authors state that comprehensive performance is presented in Table 7.
  • Performance Evaluation: Table 7 reports answer-level model performance across different combinations of questions, images, and image captions.The table compares input configurations on the PhysReason-mini benchmark.
  • Dataset: The benchmark dataset integrates physics problems from multiple examination and olympiad sources.Sources include International Physics Olympiad, national entrance examinations, provincial and municipal examinations, and Russian physics problems.

H Details of Computational Experiment

The computational experiments evaluate physics reasoning with multiple model types and supplementary image captions, while the paper illustrates benchmark problems and stepwise theorem-based solutions.

  • Computational Setup: The experiments evaluate 17 models, including LLMs, VLMs, and specialized models, primarily using NVIDIA A800 GPUs.The evaluated models include Qwen2-VL-72B and QwQ-32B, among others.
  • Computational Setup: Models unable to process visual inputs receive image captions generated by Gemini-2.0-Flash.Some experiments use only the answer-level framework because the step-level framework is computationally expensive.
  • Benchmark Examples: The gas-cylinder example uses force equilibrium, Boyle’s law, and the ideal-gas law to obtain its requested volumes and temperature.The reported first answer is (p_0S+mg)/(p_0S-mg)V_1.
  • Benchmark Examples: A representative hard problem combines rough-track motion, elastic collision, circular motion, projectile motion, and multiple sub-questions.Its sub-questions ask for the slider velocity, the minimum starting height, and the maximum horizontal distance.
  • Benchmark Examples: The representative solution applies energy equations and centripetal-force reasoning to derive the initial velocity and motion constraints.The cited solution begins with the work-energy relation on the inclined track and the centripetal-force condition at the highest point.
Loading 2502.12054v2…