Source-linked AI summary
UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, Yang Wang
TL;DR
Undergraduate physics reasoning has received less attention than mathematical reasoning, and existing benchmarks often lack the breadth needed to evaluate it. The paper introduces UGPhysics and MARJ for broader evaluation; current models still have considerable room to improve, with OpenAI-o1-mini achieving the highest overall accuracy at 49.8%.
Problem
Physics reasoning has received less attention than mathematics, and existing benchmarks often fall short in breadth and depth for undergraduate-level evaluation.
Source: §1 · p. 2Method
The authors build bilingual UGPhysics across undergraduate physics topics and reasoning skills, and develop MARJ to combine rule-based calculation with model-based answer assessment.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors expect UGPhysics and its code to be valuable for future development of AI with strong physics reasoning abilities.
Source: §6 · p. 9UGPhysics does not yet include image-based problems or broader language coverage, which the authors identify as future directions.
Source: Impact Statement · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics, underscoring the need for a comprehensive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and comprehensive benchmark specifically designed to evaluate UnderGraduate-level Physics (UGPhysics) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-level physics problems in both English and Chinese, covering 13 subjects with seven different answer types and four distinct physics reasoning skills, all rigorously screened for data leakage. Additionally, we develop a Model-Assistant Rule-based Judgment (MARJ) pipeline specifically tailored for assessing answer correctness of physics problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAI-o1-mini), emphasizes the necessity for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements in AI for physics reasoning. Codes and data are available at https://github.com/YangLabHKUST/UGPhysics .
1. Introduction
Undergraduate physics reasoning is less extensively benchmarked than mathematical reasoning, despite its additional domain knowledge and multi-step physical-law demands. UGPhysics and MARJ address this gap, and evaluations show that current LLMs still struggle with physics problems.
1. Introduction
- Existing physics benchmarks often focus on school-level multiple-choice questions, while college-level benchmarks remain limited in scope or size for undergraduate physics.Physics problems can require domain-specific laws and repeated applications of formulas, unlike predominantly logical mathematical reasoning.Source: §1 · p. 2
- UGPhysics contains 5,520 undergraduate physics problems, translated into English for bilingual evaluation, spanning 13 subjects, 59 topics, and seven answer types.Problems are also categorized by four physics reasoning skills, and the dataset undergoes leakage detection.Source: §1 · p. 2
- MARJ combines rule-based calculation precision with model-based flexibility to assess physics answers involving constants and equivalent quantities.Human evaluation is reported to support the reliability of this judgment method.Source: §1 · p. 2
- 49.8% overall accuracy is the highest achieved on UGPhysics, by OpenAI-o1-mini, highlighting the challenge of undergraduate physics problem-solving for current LLMs.The introduction also reports that DeepSeek-R1-Distill-Llama-70B ranks second, but remains behind top-tier closed-source models.Source: §1 · p. 2
- Math-specialized LLMs yield only minor improvements over general-purpose counterparts on UGPhysics, while physics derivations require domain knowledge and practical meanings beyond abstract mathematics.The paper argues this points to a need for more high-quality physics corpora.Source: §1 · p. 2
- OpenAI-o1-mini’s primary physics errors are flawed reasoning, knowledge deficiency, and wrong application, rather than the error pattern associated with mathematical reasoning.The paper also identifies o1-like LLMs as a promising direction, while noting a performance gap between DeepSeek-R1-Distill-Llama-70B and leading closed-source models.Source: §1 · p. 2
2. Related Work
Prior physics benchmarks span school, entrance-exam, college, and competition levels, but many lack undergraduate breadth, fine-grained subjects, or diverse answer formats. Related work also motivates a hybrid approach to judging physics answers and notes the need for specialized physics training resources.
Physics Benchmarks
- Earlier physics benchmarks range from school and entrance-exam questions to college and competition problems, but often lack broad undergraduate coverage, fine-grained subjects, or diverse answer types.UGPhysics is presented as broader in undergraduate subjects, answer types, and test-example count.Source: §2 · p. 3
Answer Judgment
- Rule-based answer checks can struggle with complex responses, while model-based judges can miss numerical precision, a concern heightened by physics constants and problem-specific relative-error requirements.The passage cites a 12% judgment error rate for physics problems in OlympiadBench and presents MARJ as a hybrid response.Source: §2 · p. 3
LLMs for Reasoning
- Research on mathematical reasoning has explored prompting, fine-tuning, and continued pretraining, while specialized physics corpora for continued pretraining or fine-tuning remain lacking.UGPhysics evaluations of math-specific LLMs assess how specialized math training relates to physics reasoning.Source: §2 · p. 4
3. The UGPhysics Benchmark
UGPhysics provides a broad undergraduate-physics benchmark with labeled reasoning skills, while MARJ addresses the difficulty of judging varied physics answers. The benchmark creation and judgment process are designed to support evaluation across disciplines and answer forms.
3.1. UGPhysics and MARJ Overview
- UGPhysics contains 5,520 bilingual problems across three domains, 13 subjects, and 59 topics, with seven answer types and four labeled reasoning skills.The authors also conduct data-leakage detection on several LLMs.Source: §3.1 · p. 4
3.2. UGPhysics Creation
- Problems were collected from undergraduate exercise books, manually corrected after OCR, deduplicated, and restricted to text-only items.Progressive problems were split into independent items with relevant information included; problems without definitive answers were excluded, and Chinese problems were translated into English.Source: §3.2 · p. 4
- Each test example receives one of seven answer-type labels and one of four reasoning-skill labels, with GPT-4o used for skill annotation.The answer types comprise six atomic forms and one compound type; the skills are Knowledge Recall, Laws Application, Math Derivation, and Practical Application, with “Others” for remaining cases.Source: §3.2 · p. 4
3.3. MARJ Evaluation Framework
- MARJ combines rule-based checks for simple answers with GPT-4o review of cases the rules mark false, addressing precision and equivalent-expression challenges.The rule-based stage handles answer types separately, while the model-based stage assesses answers collectively and attends to physical constants and equivalent quantities; the authors report reliable assessment.Source: §3.3 · p. 5
4. Experiments
Across 31 LLMs, UGPhysics remains challenging: OpenAI-o1-mini achieves the highest overall accuracy, while open-source and o1-like models show strong results. Scaling helps, but math specialization brings comparatively small gains on physics problems.
4.1. Experimental Setup
- The evaluation covers 31 leading proprietary, general-purpose open-source, math-specialized, and o1-like LLMs using zero-shot prompts tailored to answer types.The experiments use greedy decoding at temperature 0, with a 4,096-token output limit except where models’ SFT configurations require a shorter limit.Source: §4.1 · pp. 5–6
4.2. Main Results
- 49.78% overall accuracy is the highest result, achieved by OpenAI-o1-mini; 15 of 31 models score below 20%, and only two exceed 40%.DeepSeek-R1-Distill-Llama-70B ranks second at 40.17%.Source: §4.2 · p. 6
- Five open-source LLMs match or exceed GPT-4o, but the best open-source model trails OpenAI-o1-mini by around 10%.GPT4o-mini also outperforms many open-source models, including some o1-like models.Source: §4.2 · p. 6
- A 7B-to-72B increase yields an approximate 17% overall-accuracy gain for Qwen2.5-Math, while DeepSeek-R1-Distill-Llama gains 27% from 8B to 70B.The Qwen2.5-Math family’s improvements are consistent across domains and languages.Source: §4.2 · p. 6
- Qwen2.5-Math-7B-Instruct exceeds Qwen2.5-7B-Instruct by 0.14% average accuracy, and the 72B versions differ by 3.3%.These gains are less pronounced than the around-or-over-10% advantages reported for math-specific LLMs on mathematical problems; the authors point to physics-specific training content as a future need.Source: §4.2 · p. 6
- QwQ-32B reaches 37.3% accuracy, close to Qwen2.5-Math-72B-Instruct at 39.5%, despite its smaller model size.QwQ-32B also outperforms DeepSeek-R1-Distill-Llama-70B on Chinese problems.Source: §4.2 · p. 6
5. Analysis
The analyses examine model performance across subjects, skills, and languages, assess MARJ against human judgments, and probe errors, data leakage, benchmark difficulty, and token use. They find strong variation in physics reasoning and identify remaining evaluation and computational challenges.
5.1. Fine-grained Analysis
- Knowledge Recall tasks were easier for the selected models than Math Derivation, while OpenAI-o1-mini outperformed peers across all skills and “Others.”Across subjects, the eight models averaged 31.0% in Semiconductor Physics and 26.7% in Atomic Physics, versus 16.5% in Theoretical Mechanics; language gaps varied by model.Source: §5.1 · pp. 6–7, Figure 2 · p. 8
5.2. Reliability of Evaluation
- MARJ matched human annotations with 98% accuracy on 100 randomly selected test examples.The evaluation also handled answers verifiable with Sympy and complex answers unsuitable for rule-based judgment.Source: §5.2 · p. 7
5.3. Error Analysis
- Among 100 incorrect OpenAI-o1-mini answers, flawed reasoning, knowledge deficiency, and incorrect application were primary errors, unlike calculation’s role in mathematics (Chen et al., 2022; Xu et al., 2025).The authors connect physics reasoning and derivation challenges to their additional knowledge requirements and real-world meanings.Source: §5.3 · p. 8, Figure 4 · p. 8
5.4. About Data Leakage
- Most tested models showed some leakage, but contaminated-and-correct counts were extremely low; Qwen2.5-MATH-72B-instruct matched 5-grams in 78 samples.The authors conclude that leakage had minimal impact on UGPhysics.Source: §5.4 · p. 8, Table 6 · p. 9
5.5. Compare to Existing Benchmarks
- 38.67% accuracy on UGPhysics was below GPT-4o’s 53.60% on GPQA, which the authors cite as evidence that UGPhysics is more challenging.DeepSeek-R1 used an average of 5,555 tokens on UGPhysics without a maximum; DeepSeek-R1-Distill-Qwen-32B used 4,081 on UGPhysics and 3,079 on MATH under an 8,192-token limit.Source: Table 7 · p. 9, §5.5 · p. 9
5.6. About the Clip Ratio
- At 8,192 tokens, only 2% of o1-mini generations exceeded the limit, versus much higher rates for open-source o1-like models; extending the limit to 16,384 brought marginal gains.A small proportion of generations still exceeded the extended limit.Source: Table 9 · p. 9, §5.6 · p. 9, Table 8 · p. 9
6. Conclusion
The paper presents UGPhysics and MARJ for evaluating LLM physics problem-solving. Its experiments find considerable room to improve physics performance, and the authors see the benchmark and code as useful for future AI development in physics reasoning.
6. Conclusion
- The paper proposes UGPhysics, an undergraduate-level physics benchmark, alongside MARJ for scoring answers to physics problems.Together, they are intended to evaluate LLM capabilities in physics problem-solving.Source: §6 · p. 9
- The experiments find that, despite strengths in mathematical reasoning, current LLMs have considerable room to improve on physics problems.Source: §6 · p. 9
- The authors expect the dataset and code to be valuable for future development of AI with strong physics reasoning abilities.Source: §6 · p. 9
Impact Statement
The impact statement and appendices describe the benchmark’s scope, filtering, answer categories, skill annotations, and problem distribution. The impact statement identifies image-containing problems and additional languages as future directions.
Impact Statement
- Future directions include adding image-containing problems for multimodal evaluation and more languages for multilingual assessment.Source: Impact Statement · p. 9
A.1. Problem Filtering
- Problems without definite answers for evaluation were excluded from automatic evaluation.Source: §A.1 · p. 14
A.2. Answer Types
- Answers are classified into seven categories: six atomic types and one compound type composed of atomic types.Source: §A.2 · p. 14
A.3. Physics Reasoning Skills Annotation
- The annotations distinguish Knowledge Recall, Laws Application, Math Derivation, and Practical Application, assigning unclassifiable skill cases to “others.”The categories cover recalling concepts, applying laws, deriving physics relations, and solving practical problems using physics knowledge.Source: §A.3 · p. 14
A.4. Distribution of Problems
- UGPhysics spans three domains, 13 core subjects, and 59 topics.Source: §A.4 · p. 14
B.1. Evaluated LLMs
The evaluation includes proprietary and open-source LLMs across general-purpose, o1-like, and specialized-math categories. Prompts are adapted to subjects and response types, with chat models using their official templates.
B.1. Evaluated LLMs
- The evaluated LLMs span proprietary commercial models, open-source general-purpose models, o1-like models, and specialized math models.Source: §B.1 · p. 15
- Examples include Qwen2.5-Math variants, QwQ-32B-Preview, and DeepSeek-R1 distilled models.The appendix also lists evaluated LLM families and variants across these categories.Source: §B.1 · pp. 15–16
B.2. Evaluation Prompts
- Prompts are tailored to different subjects and response types, and chat models use their official chat templates.The appendix shows English prompts and says Chinese prompts are similar.Source: §B.2 · p. 16
B.3. Evaluation Parameters
Evaluation uses controlled decoding settings, while MARJ combines answer-type-specific rule checks with model judgment for answers flagged false.
B.3. Evaluation Parameters
- Evaluations use greedy decoding at temperature 0 and a 4,096-token output limit, with exceptions for models requiring different settings.OpenAI-o1-mini uses an 8,192-token limit and temperature 1; some models retain their supervised-fine-tuning output limits.Source: §B.3 · p. 16
B.4. MARJ Details
- MARJ normalizes or compares answers according to type, then sends rule-flagged false answers to GPT-4o for judgment.True/false and multiple-choice answers are standardized; numeric values allow relative-error comparison, expressions remove physical constants, and intervals compare endpoints.Source: §B.4 · pp. 16–17
C. More Results
The paper reports additional results by physics subject and reasoning skill, with detailed breakdowns in four tables.
C. More Results
- Tables 17–19 report results across subjects in mechanics and thermodynamics, electromagnetism, and modern physics, while Table 20 breaks results down by reasoning skill.The supplied passage identifies the tables’ scopes but does not provide their result values.Source: C. More Results · p. 17
D. Error Analysis
The section presents example cases of physics problem-solving errors and describes answer-scoring and benchmark materials that contextualize those examples.
D. Error Analysis
- Tables 21–23 illustrate flawed reasoning in statistical mechanics, wrong application in classical electromagnetism, and knowledge deficiency in semiconductor physics.The section introduces these as examples of error types; it does not state their prevalence here.Source: D. Error Analysis · p. 17, Table 21 · p. 27, Table 22 · p. 28
- The cited tables report UGPhysics subject results for Mechanics & Thermodynamics and Electromagnetism (all figures in %), evaluation prompts, MARJ physical constants, and UGPhysics subject and physics-reasoning-skill statistics.Tables 17–18 present the subject results; Table 15 gives evaluation prompts for English problems with single or multiple answers, Table 16 lists physical constants, and Tables 13–14 describe UGPhysics topics, examples, subjects, and physics reasoning skills.Source: Table 13 · p. 19, Table 14 · p. 20, Table 15 · p. 20
- MARJ checks answer-list lengths, standardizes true/false and multiple-choice answers, normalizes expressions, and applies type-specific comparisons before model judgment.The pipeline returns a rule-based correctness result when its checks match; otherwise, it calls ModelJudge.Source: D. Error Analysis · p. 21
- The examples include a Frenkel-defect problem about interstitial atoms versus temperature and an eddy-current brake problem about induced current.The supplied material also shows worked-solution excerpts for both problems.Source: D. Error Analysis · pp. 27–28