PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora

Hao Chen, Yumin Lin, Nadila Yushanjiang, Xin Lin, Min Zhang

arXiv:2608.25398v1cs.CL

TL;DR

Existing benchmarks provide limited support for comprehensive multimodal physics evaluation across educational stages and output generation. OmniPhys addresses this gap with a unified Chinese benchmark and evaluates both reasoning and diagram-related capabilities. Experiments show persistent challenges, with leading models below 70% strict mastery, while the Chinese-only scope limits global representativeness and complicates separation of language alignment from physics reasoning.

  • Problem

    Existing physics benchmarks rarely combine cross-stage knowledge, complex multimodal comprehension, and multimodal output generation.

    Source: §4.4 · p. 7

  • Method

    OmniPhys is a unified Chinese benchmark spanning secondary education to university, with joint reasoning evaluation and a Physics Diagram Editing task.

    Source: §1 · p. 1, §4.2 · p. 6, §4.4 · p. 7

  • Results

    Leading models remain below 70% strict mastery, and experiments reveal persistent challenges in conceptual mastery and multimodal physics reasoning.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    OmniPhys provides a benchmark for evaluating multimodal physics understanding and generation beyond surface-level answer matching.

    Source: §1 · p. 2, §6 · p. 8

    The data are predominantly from Chinese educational materials, so weaker performance may reflect limited physics reasoning, insufficient Chinese multimodal alignment, or both.

    Source: §7 · p. 9

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated strong abilities in solving diverse visual and textual reasoning tasks. However, their development in the physics domain is significantly hindered by the lack of a comprehensive benchmark. To fill this gap, we introduce OmniPhys, a large-scale benchmark for multimodal physics understanding and reasoning, covering middle school through university-level problems from Chinese Educational Corpora. OmniPhys consists of 15,246 questions and 19,850 images, accompanied by detailed annotations that support fine-grained analysis of reasoning processes and knowledge usage. Beyond conventional evaluation, OmniPhys is a benchmark that systematically evaluates multimodal outputs in the physics domain, including models' ability to generate structured physics diagrams, which constitute a fundamental component of authentic physics problem solving. Extensive evaluations reveal critical gaps in the capabilities of current MLLMs, especially in complex reasoning and visual generation. To address this, we release OmniPhys to serve as a foundational resource for advancing multimodal intelligence in physics and scientific domains. Codes and data are available at https://github.com/ECNU-RAIL/OmniPhys-EMNLP2026.

1 Introduction

OmniPhys addresses the lack of a comprehensive benchmark for multimodal physics reasoning across educational stages, inputs, and outputs. It combines broad physics coverage with diagram synthesis and editing evaluation, while experiments expose persistent reasoning challenges.

  • Physics requires integrating textual descriptions, visual diagrams, and symbolic logic for accurate reasoning.Source: §1 · p. 1
  • Existing physics datasets rarely combine middle-school-to-university knowledge fusion, complex multimodal comprehension, and multimodal output generation.Source: §1 · p. 1
  • OmniPhys covers physics mastery from secondary education to university using textual, visual, and symbolic inputs across five disciplines.Source: §1 · p. 1, Figure 1 · p. 2
  • The benchmark introduces a subset assessing models’ ability to synthesize and edit physics diagrams.Source: §1 · p. 2
  • Extensive evaluations find persistent challenges in conceptual mastery and substantial room for improvement across proprietary and open-source MLLMs.Source: §1 · p. 2

2 Related Works

Related work shows that multimodal models and physics datasets have advanced, but rigorous multimodal physics reasoning remains insufficiently evaluated. Existing benchmarks commonly emphasize text or limited image inputs and incomplete educational coverage.

  • Frontier MLLMs support complex interleaved inputs and outputs but remain vulnerable to hallucination, logical deduction failures, and numerical errors.Source: §2.1 · p. 2
  • These vulnerabilities motivate challenging benchmarks that probe the upper bounds of deep multimodal reasoning.Source: §2.1 · p. 2
  • Existing physical reasoning datasets predominantly use text-only tasks or general K-12 benchmarks with limited physics subsets.Source: §2.2 · p. 2
  • Recent image-input datasets still have incomplete educational coverage and lack multimodal output evaluation.Source: §2.2 · p. 2

3 The OmniPhys Benchmark

OmniPhys is a multimodal physics benchmark spanning junior high to university curricula and combining diagrams, formulas, and text for visually grounded reasoning. Its construction uses curated educational sources and multi-stage quality, difficulty, leakage, and human-validation procedures.

  • OmniPhys spans junior-high-to-university curricula and challenges MLLMs to synergize diagrams, formulas, and text for visually grounded reasoning.Source: §3 · pp. 2–3
  • The benchmark uses authoritative Chinese pedagogical resources and authentic examinations, releasing structured annotations rather than verbatim source PDFs.Source: §3.1 · p. 3
  • Its difficulty pyramid scales from foundational K-12 concepts to advanced university challenges.Source: §3.1 · p. 3

Data Preprocessing and Quality Filtering

  • The preprocessing pipeline checks schema completeness, removes short or incomplete samples, and deduplicates problems using a 0.95 cosine-similarity threshold.Source: §3.2 · p. 3
  • 12.1% of redundant instances were eliminated through semantic deduplication to produce the final valid dataset.Source: §3.2 · p. 4
  • Visual-dependency filtering distinguishes text-solvable, text-descriptive, and image-essential problems according to whether diagrams add necessary information.Source: §3.2 · p. 4
  • Adversarial difficulty screening discards problems correctly solved by both Qwen2.5-VL-3B and MiniCPM-V-2.6.Source: §3.2 · p. 4
  • Data-leakage prevention combines source selection, web-search filtering, and manual verification, while expert cross-validation assesses pipeline consistency.Source: §3.2 · p. 4

4 Experiments

OmniPhys evaluates proprietary and open-source MLLMs on multimodal physics reasoning across educational stages, using separate result and process measures. Experiments show persistent gaps in conceptual mastery, advanced-stage performance, and physically correct diagram generation.

Experimental Setup

  • The evaluation covers diverse proprietary and open-source MLLMs, including multiple Qwen2.5-VL and Qwen3-VL model scales.Proprietary models are accessed through official APIs, while most open-source models are evaluated locally under identical inference settings.Source: §4.1 · pp. 4–5

Dual-Track Reasoning Evaluation

  • Dual-Track Reasoning Evaluation separates objective tasks with deterministic outputs from open-ended tasks requiring step-by-step derivations.The framework is designed to assess both final-answer correctness and reasoning quality across different task types.Source: §4.2 · p. 5
  • Result Score measures objective-task answer accuracy after normalization, while Process Score measures recall of correctly recovered key reasoning steps.For multi-select and multi-slot tasks, partial credit is awarded only when predicted answers contain no incorrect options.Source: §4.2 · p. 5

Main Results Analysis

  • Accuracy peaks at Junior High and decreases monotonically toward University across both closed-source and open-source models.The consistent decline is presented as evidence that OmniPhys captures increasing cognitive complexity across educational stages.Source: Figure 2 · p. 6, §4.3 · pp. 6–7
  • Strict mastery rates Pobj remain below 70% even for leading models, indicating that perfect alignment of answers and reasoning is uncommon.Pobj requires S1 = S2 = 1.0, while Popen requires flawless derivation with S3 = 1.0.Source: §4.3 · p. 6, §4.2 · p. 6
  • Multimodal generation exposes failures that text-only metrics miss: models may produce visually faithful diagrams while violating physical constraints.Reported errors include incorrect vector directionality, broken topology, incorrect convex-lens focusing, and redundant line segments.Source: §4.4 · p. 7

5 Ablation Study

The adversarial Test-Mini subset probes difficult multimodal physics reasoning under three input settings. Visual inputs usually help, but some models treat visual information as distractor noise, revealing cross-modal alignment challenges.

5 Ablation Study

  • Test-Mini uses 10% of the data, selected by five-model consensus from samples weighted toward empirical failure rates and reasoning complexity.The selection weighting is 75% failure rate and 25% reasoning complexity.Source: §5 · p. 8
  • The ablation compares Text+Img, Text+Caption, and Text-only settings to quantify models’ dependence on visual information.Text+Img retains the original multimodal input, Text+Caption replaces diagrams with descriptions, and Text-only removes visual content.Source: §5 · p. 8
  • Test-Mini is substantially harder than the main benchmark and serves as a rigorous probe of deep reasoning capabilities.The passage explicitly characterizes the subset as significantly more difficult.Source: §5 · p. 8
  • On average, Text+Img achieves the highest performance, with MLLMs consistently outperforming LLMs across the evaluated settings.This pattern supports the role of visual constraints in the tested physics problems.Source: §5 · p. 8
  • Some models perform better with Text-only inputs, suggesting that visual inputs or captions can be misinterpreted as distractor noise by certain architectures.The authors identify persistent cross-modal alignment challenges requiring further investigation.Source: §5 · p. 8

6 Conclusion

OmniPhys is presented as a unified benchmark spanning junior-high to university physics and covering both understanding and generation. Its evaluations expose reasoning and physical-fidelity gaps, with leading models below 70% strict mastery.

6 Conclusion

  • OmniPhys evaluates physics reasoning across the full educational spectrum from junior high to university, covering both understanding and generation.The benchmark is positioned as a unified multimodal evaluation suite.Source: §6 · p. 8
  • The DTRE protocol jointly scores final answers and reasoning fidelity, exposing reasoning shortcuts that conventional answer-matching metrics overlook.This moves evaluation beyond surface-level final-answer correctness.Source: §6 · p. 8
  • OmniPhys introduces Physics Diagram Editing and reveals that frontier image-generation models systematically violate physical laws.These violations are described as a blind spot of text-only benchmarks.Source: §6 · p. 8
  • Leading models remain below 70% strict mastery, while ablations confirm the indispensable role of visual grounding.Experiments include both proprietary and open-source MLLMs.Source: §6 · p. 8
  • The paper positions OmniPhys as both an evaluation suite and a diagnostic instrument for advancing physically grounded multimodal intelligence.This conclusion follows the benchmark’s combined reasoning and generation assessments.Source: §6 · p. 8

7 Limitations

The study’s main limitations concern its predominantly Chinese educational data and the scalability of multimodal-output evaluation. These constraints complicate separating language alignment from physics reasoning and require continued human oversight.

7 Limitations

  • The data are predominantly drawn from Chinese educational materials, limiting linguistic and curricular representativeness.Future iterations are intended to incorporate international curricula such as A-Level and IPhO.Source: §7 · p. 9
  • English instructions paired with Chinese problems can confound weaker performance between limited physics reasoning and insufficient Chinese multimodal alignment.This issue arises under the paper’s cross-lingual prompting protocol.Source: §7 · p. 9
  • Methods for evaluating multimodal physical outputs remain underexplored, especially because MLLM judges may overestimate generated-content quality.Such judges can overlook subtle physical inconsistencies.Source: §7 · p. 9
  • The structured rubric mitigates judge weaknesses, but a fully scalable evaluation solution is still lacking.The authors therefore continue to target a robust framework for multimodal physics outputs.Source: §7 · p. 9

A Prompt Usage

The appendix specifies prompt and annotation procedures for visual-dependency labeling, model inference, reasoning evaluation, diagram grading, and human validation. These procedures standardize both textual reasoning assessment and multimodal-output judgments.

A Prompt Usage

  • Prompts are translated into English for readability, but experiments use Chinese prompts to align with OmniPhys and preserve cross-modal consistency.This applies to inference and evaluation.Source: A Prompt Usage · p. 12

A.1 Visual Dependency Annotation

  • Visual-dependency annotation assigns Level 1, 2, or 3 according to whether images are decorative, textually described, or essential for solving.Level 3 applies when critical quantities or relationships appear only in the image.Source: §A.1 · p. 12

A.2 Model Inference Prompt

  • The model-inference prompt requires step-by-step reasoning with each step limited to 30 words, followed by a final answer in a fixed format.The prompt also instructs models to avoid redundant explanations and extra content.Source: §A.2 · p. 13

A.3 Automated Evaluation Prompts

  • Automated reasoning evaluation decomposes standard solutions into m key steps, counts correctly included steps n, and computes process score as n/m.The evaluator receives the question, standard answer, explanation, and student response.Source: §A.3 · p. 13
  • Objective tasks use dual-track evaluation: process validity is scored separately from final-result correctness using task-specific rules.Single-choice and true-false matches receive 1.0, while multi-select and fill-in tasks use correct-slot fractions.Source: §A.3 · p. 13

A.4 Automated Evaluation Prompts for Multimodal Generation

  • A vision-language judge scores generated physics diagrams for physical fidelity using the problem, optional context image, and student answer image.Diagram grading distinguishes fully correct, partially correct, and incorrect outputs based on physical elements and laws.Source: §A.3 · p. 13, §A.4 · pp. 13–14

B Annotation Interface Details

  • Human annotation uses Label Studio to compare problem statements, input images, references, and generated images, with final scores averaged across three physics-trained graduate students.Scores reflect correctness, completeness, and visual faithfulness.Source: §A.4 · p. 14

C.1 Hardness-Based Selection Methodology

OmniPhys constructs Test-Mini by ranking questions with a hardness score combining empirical model failure and reasoning complexity, then selecting the top 10%. This produces a substantially more difficult subset whose challenge is consistent across model architectures.

Hardness scoring

  • Test-Mini ranks candidate questions using empirical model failure and reasoning complexity, combining them as H(xi) = w1 · F(xi) + w2 · C(xi).The full dataset contains N = 12,885 questions without multimodal outputs.Source: §C.1 · p. 15
  • Empirical Failure Rate F(xi) aggregates normalized scores from five baseline models, while Reasoning Complexity C(xi) uses the percentile rank of ground-truth explanation length.Source: §C.1 · p. 15
  • Weights w1 = 0.75 and w2 = 0.25 prioritize empirical difficulty while retaining logical depth in the hardness score.Source: §C.1 · p. 15

Subset construction

  • The top 10% of questions ranked by H(xi) form Test-Mini, using five state-of-the-art closed-source models as the baseline set M.Source: §C.1 · p. 15

Difficulty validation

  • All-Model Failure Rate rises from 1.6% to 15.5% in Test-Mini, alongside a substantial increase in average reasoning-chain length.Source: §C.2 · p. 15
  • Baseline-model performance drops by 56.9% to 72.5% across all models, indicating that Test-Mini difficulty is not specific to one architecture.Source: §C.2 · p. 16

Data authenticity

  • Original screenshots from Chinese examinations and authoritative textbooks illustrate the benchmark’s middle-school Mechanics, high-school Electromagnetism, and university Optics problems.Source: D Representative Source Materials · p. 16

E Human-Machine Alignment and Quality Control in Data Filtering Process

A blind review by two physics experts assessed 300 instances to examine automated visual-dependency filtering and difficulty screening. The reported alignment indicates that the adversarial filtering strategy targets non-trivial problems while preserving pedagogical integrity.

Study design

Quality-control result

F Human-Machine Alignment on Evaluation

OmniPhys validates its LLM-as-a-Judge evaluation through comparison with independent physics-expert scoring of generated responses. The judges rank performance consistently with experts, while humans score slightly more stringently and strict-correctness agreement remains high.

Study design

Alignment results

Implication

Related papers

Loading 2608.25398v1…