Source-linked AI summary
PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
Ruoran Xu, Wending Gao, Liyunfeng Chen, Aixin Shi, Haoyu Cheng, Zixiang Fang, Yiqiang Zou, Qiufeng Wang
TL;DR
Existing physics benchmarks provide limited evidence about multimodal models’ ability to solve authentic Olympiad-level problems because they lack sufficient difficulty, breadth, or process annotations. PhysElite constructs a large bilingual multimodal benchmark with annotated derivations and evaluates current MLLMs. Even the strongest model achieves only 33.7% answer accuracy, while process evaluation diagnoses failures in the reasoning chain.
Problem
Existing physics benchmarks lack sufficient high-difficulty problems and comprehensive visual, knowledge-point, and step-by-step process coverage.
Method
PhysElite provides 11,586 bilingual Olympiad-tier problems with diagrams, human-verified derivations, final answers, and topic and difficulty annotations.
Results
33.7% answer accuracy is achieved by the strongest evaluated model, while process-level evaluation identifies reasoning-chain failure modes across 18 MLLMs.
Takeaways & Limitations
PhysElite supports rigorous answer- and process-level evaluation of frontier models on Olympiad-level physics reasoning.
Takeaways & Limitations
Partial-credit scoring creates minor ambiguity in borderline cases, and possible overlap with commercial model training data raises data-leakage risks.
Abstract
from arXiv · showhide
Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.
1 Introduction
Existing physics benchmarks often lack Olympiad-level difficulty and step-level derivations, limiting their ability to test complex multimodal reasoning. PHYSELITE addresses these gaps with a large annotated benchmark and finds substantial remaining performance deficits.
- Prior benchmarks approach saturation, whereas PHYSELITE preserves a substantial gap between the strongest model and human reference.
- Existing benchmarks are often too easy, while Olympiad datasets remain small and infrequently updated, allowing performance saturation.
- Final-answer-only evaluation obscures where reasoning fails and limits physics problems’ use for process supervision.
- 11,586 Olympiad-tier problems pair visual diagrams with bilingual, human-verified step-by-step derivations and final answers.
- 18 open- and closed-source MLLMs are benchmarked, with process-level evaluation identifying recurring reasoning-chain failure modes.
2 Related Work
Related physics benchmarks trade off scale, modality, and difficulty. Existing resources therefore provide either broad or multimodal coverage, or Olympiad-level challenge at limited scale and with contamination concerns.
- ScienceQA, MathVista, MMMU, and SciBench broaden multimodal coverage, but their physics subsets are mostly introductory and small.
- PHYSICS and UGPhysics scale text-only physics data but exclude diagrams and do not target Olympiad difficulty.
- PhyX, PhysUniBench, PHYSICS, and PhysicsArena add visual input, yet remain capped at undergraduate or high-school difficulty.
- OlympiadBench, PhysReason, and HiPhO probe harder problems but are constrained by scale, public-source overlap, or limited statistical coverage.
3 PHYSELITE Dataset
PHYSELITE combines Olympiad-level difficulty, large scale, bilingual multimodal coverage, and process annotations. Its construction uses competition-oriented source materials, OCR and manual verification, deduplication, and human-in-the-loop labeling.
- Dataset overview: PHYSELITE combines Olympiad-level difficulty, 11,586 problems, bilingual multimodal coverage, diagrams, and step-level derivations.
- Dataset overview: Problems span mechanics, electromagnetism, thermodynamics, optics, and modern physics, with open-ended answers and difficulty ratings from 1 to 7.
- Data construction: Approximately 15,000 scanned competition-training problems were collected, converted with OCR, and manually verified for text, diagrams, and mathematical content.
- Data construction: Fuzzy matching at a 0.95 similarity threshold removed duplicates, reducing the collection to 11,586 problems.
- Annotation: Annotations use human-in-the-loop labeling, with experts resolving ambiguous difficulty cases and verifying model-assisted subject categories.
- Dataset composition: The dataset is concentrated in mechanics and electromagnetism, with medium difficulty dominant and a long tail of harder problems.
4 Experiments
Experiments evaluate 18 diverse LLMs and MLLMs on PHYSELITE using bilingual answer- and process-level scoring, with additional tests of extended thinking and visual guidance. Results show substantial difficulty, uneven benefits from reasoning modes, and stronger effects from process diagrams than textual hints.
- Evaluation Models: 18 models are evaluated, including 8 open-source and 10 closed-source systems, divided into 12 System-1 and 6 System-2 models.System-1 models use fast single-pass reasoning, whereas System-2 models use iterative long chain-of-thought reasoning.
- Evaluation Setup: 15 first-prize-winning students provide the human baseline, while multimodal models receive each problem’s schematic diagram and are evaluated separately in Chinese and English.Reported results are averaged across the two languages.
- Scoring: The Answer Score requires an exact final-answer match, whereas the Process Score grades decomposed derivation steps as fully, partially, or incorrectly correct.Three LLM judges independently score each response, and their mean is validated against 200 human-graded samples with mean absolute error 0.09.
- Overall Results: 33.7% answer accuracy is achieved by the best model, Grok-4.2, while Claude-Opus-4.6 reaches 28.1% and the strongest open-source model reaches 20.0%.All other models remain below 30%, and almost every model scores higher in English than Chinese.
- Effect of Extended Thinking: Grok-4.2’s 33.7% exceeds Claude-Opus-4.6’s 28.1% by 5.6 points, but Claude’s default mode surpasses five of six reasoning models.Extended thinking therefore provides an uneven advantage and does not substitute for strong base capability.
- Effect of Process Diagrams: All six tested models improve on both metrics when given process diagrams, whereas textual solution hints produce only marginal changes and sometimes reduce answer accuracy.Process-score gains are consistently larger than answer-score gains with added process diagrams.
5 Analysis
Step-level analyses show that models fail at different depths and severities, with persistent weaknesses in optics and non-monotonic performance across difficulty levels.
- Three analyses localize first errors, distinguish soft from hard failures, and stratify failures by physics subject and difficulty.The protocol examines where errors first appear, how severe they are, and which problem classes expose them.
- 5.1 Where the first error appears: Claude-Opus-4.6 first errs at step 2.48, whereas Qwen2.5-VL-7B fails at step 1 in 96.8% of cases.Grok-4.2 fully completes 21.8% of problems despite a similar first-error depth to Claude-Opus-4.6, showing that depth alone does not determine outcome.
- 5.2 Failure severity: Soft failures range from 7% for Qwen2.5-VL-7B to 46% for Claude-Opus-4.6, with Chinese–English gaps reaching 12 points.Closed-source frontier models have the four highest soft-failure fractions, and soft failures are consistently more common in Chinese.
- 5.2 Failure severity: Claude-Opus-4.6 has the highest last-step-only failure rate at 7.8%, usually involving numerical substitution, signs, or final simplification.These cases preserve fully correct upstream reasoning.
- 5.3 Problem-class stratification: Optics is the universal weak point, with the Optics–Thermodynamics first-step error gap exceeding 20 percentage points on average among the top eight closed-source models.The analysis associates this gap with less reliable geometric and phase-based reasoning than algebraic manipulation.
- 5.3 Problem-class stratification: Four frontier reasoning models score over 10 points higher on Hard than Medium, producing a U-shaped difficulty pattern unlike the monotone human trend.Only Grok-4.2, Kimi-K2-Thinking, and Qwen2.5-VL-7B show the expected Easy-to-Hard decrease among the listed models.
- 5.4 Error taxonomy: Physical perception, physical-law misapplication, and geometric or visual misreadings account for roughly two thirds of Claude-Opus-4.6’s sampled failures.Their reported proportions are 31%, 25%, and 8%, respectively.
6 Conclusion
PHYSELITE shows that frontier LLMs remain far from expert-level performance on both final answers and reasoning processes. Its step-level protocol identifies where reasoning chains fail and which error types dominate across topics and model families.
- Frontier LLMs remain far from expert-level performance under both answer-level and process-level metrics.The benchmark also analyzes failure locations and dominant error types across topics and model families.
- Nominal difficulty alone is insufficient for cross-benchmark interpretation because source distribution and problem-style alignment can dominate outcomes.The conclusion highlights this issue especially when comparing competition-style sets with iPhO-like evaluations.
- PHYSELITE is intended as both a rigorous evaluation benchmark and a process-supervised resource for training stronger scientific reasoners.
7 Limitations and future works
The benchmark combines broad physics coverage, multimodal examples, difficulty metadata, and open-ended answers, while acknowledging annotation ambiguity and possible training-data overlap.
- 7 Limitations and future works: Partial-credit scoring creates minor annotation ambiguity in borderline cases, and training data may slightly overlap with commercial model datasets.Future work aims to collect a fully private dataset to reduce potential data-leakage risks.
- Difficulty and answer types: PHYSELITE groups problems into Easy (1–3), Medium (4–5), and Hard (6–7) using a 1–7 difficulty scale.The dataset is concentrated around the medium range and has a long hard tail.
- Difficulty and answer types: The benchmark uses open-ended answers rather than multiple choice, requiring models to directly produce a physical quantity or relation.This makes answer-level scoring stricter and increases the diagnostic role of process scores.
- Fine-grained Physics Subjects: PHYSELITE covers Mechanics, Electromagnetism, Thermodynamics, Optics, and Modern Physics as distinct but complementary reasoning categories.The taxonomy is based on physical principles and characteristic reasoning patterns to support interpretable diagnostics.
- Fine-grained Physics Subjects: Optics requires switching between geometric and wave-based descriptions, including ray tracing and phase tracking across multiple paths.The category spans lenses, mirrors, interference, diffraction, polarization, and optical-path reasoning.
- Fine-grained Physics Subjects: Modern Physics includes relativity, quantum, atomic, and nuclear problems requiring specialized principles less common in routine high-school datasets.
- Examples: The benchmark includes visual examples from mechanics, electromagnetism, optics, thermodynamics, and modern physics problem classes.
A Sample of Thermodynamics Problem
The CDROM spectrometer example uses a concave reflection grating and a photoresistor to measure wavelength-dependent conductance, test blackbody behavior, estimate filament temperature, and correct the spectrum.
- The experiment measures photoresistor conductance across the visible spectrum using a focused first-order spectrum from a CDROM reflection grating.A 12V, 50W tungsten-filament bulb supplies the light source.
- The filament’s luminous behavior is tested against ideal-blackbody scaling before estimating its temperature at 12V.The stated experiment determines the filament temperature from its electrical and optical behavior.
- The reported filament temperature is approximately 2940 K, with conductivity correction performed using the Planck formula.The source spectrum is divided out to obtain the intrinsic spectral response of the photoresistor.
- The CDROM acts as a concave reflection grating, and the first-order diffraction angle maps photoresistor position to wavelength.Sliding the photoresistor along the focused spectrum and recording its response yields the measured spectral relationship.
- The experiment uses measured voltage and current relationships to infer temperature-dependent resistance and validate blackbody behavior.The solution cites linear resistivity and Stefan–Boltzmann scaling as part of the analysis.
C.1 Data Collection Pipeline
PHYSELITE is constructed from broad Olympiad-training materials through screening, transcription, diagram linking, deduplication, and expert-reviewed annotation. Its structured bilingual records support answer evaluation and process-level analysis across multimodal physics problems.
- Data sources: Approximately 15,000 open-ended problems were collected from contestant learning materials and private advanced-training exam sets.These image-based sources typically include statements, diagrams, step-by-step solutions, and final answers.
- Construction pipeline: The pipeline screens source images, transcribes statements and solutions into structured records, links diagrams, and removes duplicates through fuzzy matching and human review.Mathematical expressions are converted into LATEX to preserve symbolic precision.
- Data sources: PHYSELITE emphasizes daily practice and private exam materials to broaden problem styles beyond official competition datasets.The dataset includes routine elite-practice and medium-difficulty problems alongside harder Olympiad-style tasks.
- Annotation: Human annotators make final transcription, difficulty, and subject decisions, while LLMs assist only with candidate generation or label normalization.Difficulty anchors range from short easy problems to extremely hard national or international Olympiad-level items.
- Annotation: Problems are assigned to Mechanics, Electromagnetism, Thermodynamics, Optics, or Modern Physics, with cross-domain labels based on the core reasoning challenge.Human experts review proposed labels and resolve ambiguous cases.
E Annotator and Human-Level Evaluation
Human experts support dataset annotation and validate the process-scoring protocol against manually graded model responses.
- Human evaluation: Experts resolve uncertain difficulty and subject labels and manually grade a random sample of 200 model responses.Their grading is used to validate the LLM-judge process evaluation.
- Human evaluation: The averaged LLM-judge scores differ from expert scores by a mean absolute error of 0.09 on the 0–1 composite scale.The paper uses this agreement to support large-scale judge-based scoring.
F Evaluation Details
Evaluation compares 18 models under bilingual text-only or multimodal settings using answer and derivation scoring by an ensemble of three LLM judges. Results expose language-, difficulty-, and subject-specific variation.
- Evaluation setup: Each problem is evaluated separately in Chinese and English, with reported results averaged across the two languages.Multimodal models receive schematic diagrams, whereas text-only models receive only textual statements.
- Evaluation setup: 18 models are evaluated, including 10 closed-source and 8 open-source systems; 6 use extended System-2 reasoning and 12 use standard System-1 responses.Text-only models are evaluated without diagram input.
- Answer evaluation: Three LLM judges score each response using the reference answer, reference solution, and full model response, while allowing partial verdicts for multi-part problems.The final judge score is the mean of the three independent scores.
- Answer evaluation: Answer correctness accepts numerical results within 5% relative tolerance and mathematically equivalent symbolic forms, while requiring every sub-part for a fully correct multi-part verdict.A wrong final answer can still receive process partial credit but is not answer-correct.
- Process evaluation: Process scoring grades each reasoning step as 1.0 correct, 0.5 locally flawed or incomplete, or 0.0 physically wrong, mathematically invalid, or irrelevant.Models are judged on their own derivations rather than being forced to match one reference path.
- Observed patterns: Most models score higher in English, many show lower Medium than Hard scores, and Optics is the most consistent weak point across models.The paper links these patterns to language-specific expression, familiarity with problem templates, and optical geometric or phase-sensitive reasoning.
I.1 Error Taxonomy
The paper organizes model failure analysis into six recurring error types and illustrates them with example responses.
- I.1 Error Taxonomy: Six typical error types are detailed in Table 7 and illustrated through Claude-Opus-4.6 examples in Figures 14–17.The listed figure labels include physical-law, physical-perception, geometric/vision-perception, and algebraic/symbolic errors.