Source-linked AI summary
How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous
TL;DR
Low reported physics-benchmark scores suggest frontier models struggle with advanced physics, despite physicists’ reports of strong practical capabilities. The paper audits six benchmarks with experts and finds that correcting benchmark and evaluation defects raises measured performance substantially.
Problem
Low reported physics-benchmark scores suggest frontier models struggle, despite physicists reporting strong model capabilities in practice.
Source: §1 · pp. 1–2Method
The authors evaluate three models on text-only physics questions and have subfield experts review questions, references, and model responses to identify errors.
Source: §2.1.2 · p. 3, §2.4 · p. 4Results
Takeaways & Limitations
Near-saturation on closed-ended physics tasks suggests a need for more demanding, expert-validated evaluations, but does not establish end-to-end physics research capability.
Source: §5 · p. 8Pre-audit and corrected scores are not always computed on the same questions because corrected evaluations use retained or repaired subsets.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Low reported scores on leading physics benchmarks, including those featured in the Artificial Analysis Intelligence Index (2026), suggest that frontier language models still struggle with advanced physics, a demanding test of their scientific reasoning and quantitative problem-solving abilities. Yet this impression does not always align with domain experts' experiences using these models in their work. We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts, focusing on text-only problems with verifiable final answers. For each subfield of physics, faculty and graduate researchers with relevant expertise carefully review problem statements, reference solutions, and model responses to distinguish genuine model errors from grader errors, incorrect reference solutions, and ambiguous or underspecified questions. Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics reasoning. We then ask experts to address these benchmarking issues by correcting erroneous reference solutions and repairing or excluding flawed questions. We find that GPT-5.6-Sol's measured mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark, while its corrected pass@4 reaches 94.4% on the 54 retained CritPt challenges. Corrected scores are computed on the retained evaluation subsets following expert review. Scores on the audited subsets of UGPhysics, PRISM-Physics, and PHYBench also rise substantially after correction. These findings suggest that current benchmarks substantially understate frontier models' ability to solve well-posed physics problems. Near-saturation on these closed-ended tasks highlights the need for more demanding, expert-validated evaluations.
1 Introduction
As physics benchmarks gained prominence, their low reported frontier-model scores raised questions about models’ scientific reasoning. Those results appeared to conflict with physicists’ reports of strong practical capabilities, motivating a closer look at whether frontier models really struggle with physics.
1 Introduction
- As mathematical benchmark performance improved, evaluation attention shifted toward physics, where benchmarks commonly reported substantially weaker frontier-model performance than expert performance.The paper frames physics as a test of scientific reasoning and problem-solving.Source: §1 · p. 1
- CritPt reported GPT-5.6-Sol mean@5 of 32.3%, while HLE-Physics reported mean@4 of 47.3% at High reasoning effort.These scores were presented as suggesting limitations in quantitative applications beyond physics.Source: §1 · p. 2
- Frontier models’ mathematical research achievements do not rule out a physics-specific gap, yet physicists report strong practical capabilities, prompting the question: do frontier models really struggle with physics?The paper notes that physics also requires modeling, abstraction, and choosing appropriate assumptions.Source: §1 · p. 2
2 Methodology: Diagnosing Errors in Physics Evaluation
The methodology evaluates six closed-ended physics benchmarks, then uses expert review to distinguish model mistakes from evaluator and benchmark defects. Corrected evaluations follow expert repairs or exclusions of flawed questions and references.
2 Methodology: Diagnosing Errors in Physics Evaluation
- The audit evaluates three frontier models on text-only physics questions, then uses expert review to identify error sources and reassesses performance after corrections.Reviewers inspect problems, reference solutions, and model responses before correcting evaluation procedures and repairing or excluding flawed questions.Source: §2 · pp. 2–3, §2.1.2 · p. 3
2.1.1 Benchmarks
- Six closed-ended benchmarks span undergraduate exercises to expert-curated challenges, with public-source and expert-authored question groups.PRISM-Physics includes final-answer and process-level evaluation, but the study uses only final-answer evaluation; equivalent final forms are accepted.Source: §2.1.1 · p. 3
2.1.2 Pre-Audit Evaluation
- Pre-audit scores use original benchmark materials and benchmark evaluators where available (CMT-Benchmark instead uses an adapted HLE pipeline), report mean@4 except CritPt’s reported mean@5, and combine model, grader, and benchmark-material errors.Fable 5’s pre-audit CritPt score uses Max reasoning effort, while its corrected score uses High.Source: §2.1.2 · p. 3, §2.4 · p. 3
2.2 Error Attribution
- A model error requires a well-posed problem and correct reference answer; grader errors reject a correct response, while benchmark errors involve defective questions or references.A missing assumption counts as a benchmark error when needed to determine the intended answer and not unambiguously inferable.Source: §2.2 · p. 3
2.3 Audit Data Collection
- To reduce review burden, four benchmarks’ audits focus on questions for which all GPT-5.6-Sol High attempts were marked incorrect.The audit runs were separate from those used for pre-audit results.Source: §2.3 · p. 3
2.4 Expert Audit
- Grader errors dominate audited PHYBench and PRISM-Physics cases, but cannot be determined for CMT-Benchmark or CritPt because original per-question judgments are unavailable.Figure 2 covers rejected answers for the first four benchmarks and all audited questions for CMT-Benchmark and CritPt.Source: Figure 2 · p. 4
- Auditors with relevant subfield expertise assess question validity and reference correctness, then classify remaining cases as model or grader errors.CritPt and CMT-Benchmark questions are repaired where possible; flawed questions on four other benchmarks are excluded.Source: §2.4 · pp. 3–4
3 Results
Expert audits attribute many apparent physics-model errors to benchmark or grading defects, and corrected evaluations show substantially higher performance. The section reports results across public-source and expert-authored benchmarks using retained or repaired question sets.
3 Results
- The analysis focuses primarily on expert-authored benchmarks and uses a common HLE-adapted evaluator for corrected evaluations; pre-audit and corrected scores may cover different questions.The common procedure assesses answer equivalence; scores are generally mean@4, with pass@4 also reported.Source: §3 · p. 4
3.1 Benchmarks Drawn From Public Sources
- GPT-5.6-Sol High’s corrected mean@4 reaches 90.23% on PHYBench, 94.59% on PRISM-Physics, and 92.07% on UGPhysics.Its pre-audit scores were 26.50%, 13.00%, and 83.00%, respectively; corrected scores use retained subsets after flawed questions are excluded.Source: §3.1 · p. 4
- PHYBench’s EED evaluator assigns zero to the model’s answer despite its matching the reference rope tension in an equivalent form.Rationalizing the denominator makes the equivalence explicit.Source: Figure 3 · p. 5
- Across the three public-source audit sets, 148 of 152 apparent errors (97.37%) are attributed to benchmark or grader errors, and four (2.63%) to models.On UGPhysics, 18 of 22 audited cases are benchmark errors, three are grader errors, and one is a model error.Source: §3.1 · p. 5
3.2.1 HLE-Physics
- With tools enabled, GPT-5.6-Sol High’s corrected mean@4 reaches 78.66% on HLE-Physics, up from 47.28%; its corrected pass@4 is 91.38%.Fable 5 High’s corrected mean@4 is 75.65%, compared with 47.03% before audit; its corrected pass@4 is 83.62%.Source: §3.2.1 · p. 5
3.2.2 CMT-Benchmark
- On CMT-Benchmark, GPT-5.6-Sol High’s mean@4 with tools rises from 61.00% to 87.24%, and its pass@4 rises from 72.00% to 97.96%.The corrected evaluation retains 49 questions, including 29 repaired questions, and excludes one; the evaluator is unchanged.Source: §3.2.2 · p. 5
3.2.3 CritPt
- On the 54 challenges retained from the 56 audited, GPT-5.6-Sol Max reaches corrected mean@4 of 87.50% and corrected pass@4 of 94.44%, using independently derived auditor reference solutions.Pre-audit pass@4 is unavailable for all models because Artificial Analysis does not report per-challenge judgments.Source: §3.2.3 · p. 5
4 Related Work
Related work documents expert-assisted physics work beyond standardized tests and broader concerns about benchmark and grader quality. Examples from other domains show that task defects can persist even after curation.
Performance of Frontier Models on Physics Beyond Standardized Benchmarks
- Frontier-model physics work outside standardized benchmarks includes expert-in-the-loop calculations, but these reports do not quantify performance on held-out questions.Examples include a reconstructed symmetry algebra, a factorization theorem, and analytic cosmic-string radiation results.Source: §4 · p. 6
Benchmark Flaws Matter More as Models Improve
- Prior benchmark research distinguishes mislabeled items from questions without clear answers and notes that fixed tests lose statistical power at high accuracy.As model capability grows, grader errors can widen the gap between measured and true performance because correct solutions become more common.Source: §4 · pp. 6–7
Rule-Based Evaluators, LLM Judges, and Reference Solution Accuracy
- Rule-based evaluators can reject equivalent answers, while LLM judges introduce their own grading inaccuracies; reference-solution accuracy is a separate issue.The paper therefore audits questions and reference solutions separately from answer evaluation.Source: §4 · p. 7
Benchmark Failures Beyond Physics
- Software benchmarks have also retained defects after curation, including reported material problems in SWE-bench Verified and SWE-bench Pro tasks.Physics benchmarks lack executable checks, so identifying defects requires re-deriving answers.Source: §4 · p. 7
5 Discussion
The discussion argues that benchmark defects explain much of the gap between reported scores and frontier-model performance on well-posed physics questions. It distinguishes near-saturation of closed-ended tasks from success on open physics research and calls for harder, verified benchmarks.
5 Discussion
- In an audit of 250 cases across four benchmark subsets, experts attribute 238 cases to question or grader defects and 12 to model errors.GPT-5.6-Sol makes no errors among audited PRISM-Physics cases, while UGPhysics has one model error; HLE-Physics and PHYBench still show model errors.Source: §5 · p. 8
- Frontier models appear near-saturated on closed-ended physics problem-set tasks, but the paper reports no fully solved open theoretical-physics problem using its agentic harnesses.The discussion calls for new, hard physics tasks with adequate verification and notes that high-quality curation may require substantial resources.Source: §5 · p. 8
- A separate expert correction of CritPt reported Fable 5.1 average pass@1 of 88.4%, broadly in line with the paper’s corrected results but not directly comparable.The evaluations differ in question sets, models, attempt budgets, and judges.Source: §5 · p. 8
A Evaluation Protocol
The evaluation uses different tool and reasoning settings across public-source and expert-authored benchmarks, then audits rejected answers to distinguish model errors from benchmark and grading problems. Expert corrections produce substantial score increases on PHYBench, PRISM-Physics, UGPhysics, and HLE-Physics.
A Evaluation Protocol
- Public-source benchmarks were evaluated without tools, while GPT-5.6-Sol and Fable 5 used coding tools on expert-authored benchmarks; Gemini 3.1 Pro used none.High reasoning effort was used unless otherwise stated; GPT-5.6-Sol used Max effort for CritPt answer generation.Source: A Evaluation Protocol · p. 11
B Benchmark Selection and Expert-Audit Details
- The pooled audit sets comprise rejected responses from HLE-Physics (98), PHYBench (56), PRISM-Physics (74), and UGPhysics (22); CritPt and CMT-Benchmark audits differ.The audit runs and selection rules differ from the pre-audit evaluations and were chosen to reduce expert-review workload.Source: B Benchmark Selection and Expert-Audit… · p. 11
B.1.1 PHYBench
- GPT-5.6-Sol High’s mean@4 rises from 26.50% to 90.23%, and its pass@4 rises from 34.00% to 95.40% on PHYBench after correction.After excluding 13 benchmark-error questions, 87 questions remain for re-evaluation.Source: §B.1.1 · p. 11
B.1.2 PRISM-Physics
- GPT-5.6-Sol High’s mean@4 rises from 13.00% to 94.59%, and its pass@4 rises from 24.00% to 95.95% on PRISM-Physics after correction.The text-only sample contains 100 questions; excluding 26 benchmark-error questions leaves 74 for re-evaluation.Source: §B.1.2 · p. 11
B.1.3 UGPhysics
- GPT-5.6-Sol High’s mean@4 rises from 83.00% to 92.07%, and its pass@4 rises from 85.00% to 93.90% on UGPhysics after correction.The randomly sampled evaluation subset contains 100 English text-only questions; 82 remain after excluding benchmark errors.Source: §B.1.3 · p. 11
B.2.1 HLE-Physics
- GPT-5.6-Sol High’s mean@4 rises from 47.28% to 78.66%, and its pass@4 rises from 55.94% to 91.38% on HLE-Physics after correction.The audit excludes 86 benchmark-error questions and retains 116 for re-evaluation.Source: §B.2.1 · p. 12
B.2.2 CMT-Benchmark
Experts reviewed and corrected benchmark materials, then reassessed model performance on retained questions. Corrections raise GPT-5.6-Sol scores on CMT-Benchmark and CritPt, while the audit data collection used different attempt and tool protocols across benchmarks.
B.2.2 CMT-Benchmark
- GPT-5.6-Sol High’s mean@4 rises from 61.00% to 87.24%, and its pass@4 rises from 72.00% to 97.96% on the corrected CMT-Benchmark.Experts identify errors in 30 of 50 questions, repair 29, and exclude one, leaving 49; two model errors occur on original valid questions.Source: §B.2.2 · p. 12
B.2.3 CritPt
- For CritPt, reviewers assess specification, repair questions where feasible, independently solve them for references, and compare model responses against those references.The official reference solutions are not publicly available, and reviewers audit all questions in the selected subset regardless of pre-audit response.Source: §B.2.3 · p. 12
- On 54 corrected CritPt challenges, GPT-5.6-Sol Max achieves a mean@4 of 87.50% and a pass@4 of 94.44%.Its pre-audit mean@5 is 32.29% on 70 challenges; the pre-audit pass@4 is unavailable because only aggregate scores were reported.Source: §B.2.3 · p. 12
B.3 Audit Data Collection
- Audit runs used GPT-5.6-Sol at High effort and stored one response plus the original evaluator’s binary decision for each question.HLE-Physics used up to five attempts, with tools enabled on the fifth, stopping once a problem was graded correct.Source: §B.3 · p. 12
- PHYBench questions received up to five tool-free attempts, stopping at an Expression Edit Distance score of 100; the highest-scoring response was retained for audit.The audit run accepted 44 of 100 answer-bearing questions and rejected 56 across all five attempts.Source: §B.3 · p. 12
- PRISM-Physics and UGPhysics audit subsets each used one attempt; corrected evaluation retained 74, 82, and 116 questions for PRISM-Physics, UGPhysics, and HLE-Physics, respectively.The corrected scores use the HLE-adapted pipeline on these retained subsets.Source: §B.3 · pp. 12–13
C Complete Counts for the Four Pooled Audits
Across four pooled audits, most rejected responses were attributed to benchmark or grader errors rather than model errors. The paper’s examples distinguish defective questions or references from correct answers rejected by evaluators.
C Complete Counts for the Four Pooled Audits
- 238 of 250 audited rejections (95.20%) were benchmark or grader errors, while 12 (4.80%) were model errors.The audits covered 502 questions, of which 252 were accepted and 250 were rejected and reviewed.Source: C Complete Counts for the Four Pooled… · p. 13
- Table 2 reports attribution counts and within-set percentages for four processed audit sets, excluding CritPt and CMT-Benchmark.The counts are not a reconstruction of Table 1 scores.Source: Table 2 · p. 13
D Representative Benchmark and Grader Errors
- Benchmark errors are defective problem statements or reference solutions, while grader errors occur when evaluators reject correct responses.The examples compare final answers while omitting derivations and normalizing mathematical typography.Source: D Representative Benchmark and Grader… · p. 13
D.1 Benchmark Errors
The audit identifies benchmark defects including underspecified questions and incorrect reference answers. Examples show how missing assumptions or information can make a question ambiguous or its expected answer wrong.
D.1 Benchmark Errors
- The PHYBench reference answer equals zero, which the reviewer identifies as incorrect.Source: §D.1 · p. 13
- PRISM-Physics asks for a time of flight without timing or distance data, and the reviewer says no option can be uniquely selected.Source: §D.1 · p. 13
- The UGPhysics damping question did not specify “without overshoot,” although critical damping gives the fastest decay without overshooting, not necessarily the fastest return to equilibrium.Source: §D.1 · p. 14
- The HLE-Physics reference answer contains a final arithmetic error despite having the correct algebraic expression.Source: §D.1 · p. 14
- The CMT Ising question omitted a normalization and mishandled the g = 0 transition; specifying assumptions changed the answer from (a; c) to (a; b; c).Source: §D.1 · pp. 14–15
D.2 Grader Errors
The audit finds grader errors on PHYBench, PRISM-Physics, HLE-Physics, and UGPhysics by checking whether model and reference answers are equivalent. The examples show equivalence despite differences in notation, algebraic form, limits, or sign conventions.
D.2 Grader Errors
- Auditors identify grader errors on PHYBench, PRISM-Physics, HLE-Physics, and UGPhysics; CritPt and CMT-Benchmark lack per-question pre-audit judgments for this check.Source: §D.2 · p. 15
- For the PHYBench bug-on-disk problem, mRω² and mω²R are the same product, so both expressions give the same maximum force.Source: §D.2 · p. 15
- For the PRISM diving problem, the model and reference give the same numerical results to stated precision and equivalent equations and initial conditions.The audit records a binary score of 0 despite the reviewer finding the responses identical.Source: §D.2 · pp. 15–16
- For the HLE polymer problem, the model's exponential factor tends to one in the stated small-x limit, yielding the reference's linear force-law magnitude.The model's minus sign denotes attraction; the agreement is limited to this small-x limit.Source: §D.2 · p. 16
E.1 CritPt: An Unjustified Real-Polarizability Assumption
In CritPt Challenge 18, all four GPT-5.6-Sol Max attempts assume real particle polarizabilities, although the problem does not require them to be real. Their answers therefore cover only a special case of the general result and omit polarizability-phase contributions.
E.1 CritPt: An Unjustified Real-Polarizability Assumption
- All four GPT-5.6-Sol Max attempts assume real polarizabilities, a condition the problem does not impose.Source: §E.1 · p. 16
- The resulting expressions match the reference only in the real-polarizability special case and omit phase contributions in the general result.Source: §E.1 · p. 16, Figure 5 · p. 17
- Figure 5 presents a representative error in which the real-polarizability assumption yields only a special case of the general result.Source: Figure 5 · p. 17
F.1 CritPt and CMT-Benchmark
The appendices describe an expert-review workflow for CritPt and CMT-Benchmark, report coverage and disagreement resolution in a separate benchmark audit, and assign project, physics-advising, and audit roles.
F.1 CritPt and CMT-Benchmark
- Reviewers matched questions to their expertise, outlined an approach before reading reference solutions, then checked statements, solutions, answers, and applicable calculations.Source: §F.1 · p. 17
- Defective questions were repaired when a defensible fix existed or excluded; retained questions received expert-verified ground truth.Figure 6 illustrates repairs to missing assumptions and an updated reference solution; reviewers could use AI assistance but remained responsible for independent verification.Source: §F.1 · p. 17, Figure 6 · p. 18
F.2 HLE-Physics, PHYBench, PRISM-Physics, and UGPhysics
- The audit produced 446 annotations: 196 questions had two reviews and 54 had one; 140 of 196 double-reviewed labels matched (71.43%), while all 56 disagreements (28.57%) received a third review.Source: §F.2 · p. 18, Table 3 · p. 18
G Author Contributions
- Project advisors designed and guided the study, the core team conducted it, physics advisors advised on audits, and data auditors performed audits and corrected reference solutions where needed.Source: G Author Contributions · p. 19