Source-linked AI summary
Interpretable Physics Reasoning and Performance Taxonomy in Vision-Language Models
Pranav Pawar, Kavish Shah, Akshat Bhalani, Komal Kasat, Dev Mittal, Hadi Gala, Deepali Patil, Nikita Raichada, Monali Deshmukh
TL;DR
The paper addresses the need for accessible evaluation of VLMs’ physics reasoning, since existing benchmarks can be text-focused or rely on costly simulators. It introduces a lightweight framework that procedurally generates 2D physics problems and evaluates models across domains. Results show that the models perform well on formula-based problems but struggle with spatial reasoning; Qwen2.5-VL-7B scores 0.815 overall.
Problem
Evaluating whether VLMs understand physical principles is challenging because existing benchmarks can be text-only or depend on costly simulators that limit reproducibility.
Source: §1 · p. 1Method
The framework procedurally generates mathematically rigid 2D physics problems across four domains and evaluates VLMs against computationally derived solutions.
Source: §1 · pp. 1–2, §3.2 · pp. 3–4Results
VLMs perform well on formula-based physics but struggle with spatial reasoning; Qwen2.5-VL-7B scores 0.815 overall, versus 0.70 for DeepSeek-VL-1.3B.
Source: §4.1 · p. 5, §4.7 · p. 7, §5 · p. 8Takeaways & Limitations
The findings suggest that domain-specific variations may require architectural innovations for human-level physics reasoning.
Source: §5 · p. 8
Abstract
from arXiv · showhide
As Vision-Language Models (VLMs) grow in sophistication, their ability to perform reasoning is coming under increasing supervision. While they excel at many tasks, their grasp of fundamental scientific principles, such as physics, remains an underexplored frontier. To reflect the advancements in these capabilities, we introduce a novel and accessible framework designed to rigorously evaluate VLMs on their understanding of 2D physics. Our framework features a pragmatic scenario generator that creates a diverse testbed of over 400 problems across four core domains: Projectile Motion, Collision Dynamics, Mechanics, and Fluid Dynamics. Through comprehensive evaluation of four state-of-the-art VLMs, we demonstrate a strong correlation between model scale and reasoning ability, with our top-performing model, Qwen2.5-VL-7B, achieving an overall score of 0.815. We find that while models excel at formulaic problems, they struggle significantly with domains requiring abstract spatial reasoning. By designing this framework, we aim to democratize the study of scientific reasoning in VLMs and foster deeper insights into their capabilities and limitations.
1 Introduction
The paper asks whether VLMs genuinely reason about physics, a multimodal test of language, images, and physical principles. It presents a lightweight framework for evaluating 2D physics across four procedurally generated domains.
1 Introduction
- Physics provides a multimodal test of whether VLMs grasp the relationships and principles governing the physical world.Problems combine textual descriptions, mathematical equations, and often clarifying diagrams.Source: §1 · p. 1
- Existing scientific benchmarks are often text-only, while visually grounded options may depend on computationally expensive simulators that hinder reproducibility.The paper identifies accessible evaluation tools as a gap.Source: §1 · p. 1
- The proposed lightweight 2D physics framework procedurally generates problems across Projectile Motion, Collision Dynamics, Mechanics, and Fluid Dynamics.The paper presents the framework architecture, experimental findings, and insights into leading VLMs’ scientific reasoning.Source: §1 · pp. 1–2
2 Related Works
Prior benchmarks cover general knowledge, science, multimodal reasoning, and physics-specific tasks, but the paper identifies a lack of lightweight, reproducible physics evaluation. Its framework combines structured prompts and responses, multimodal reasoning, and procedural problem generation.
2 Related Works
- Earlier knowledge and science assessments expanded reasoning evaluation, but many remained text-focused or lacked real-world visual complexity.MMLU and GSM8K focused on knowledge and grade-school mathematics; ARC, ScienceQA, PIQA, and CATER broadened the evaluated tasks.Source: §2 · p. 2
- Multimodal benchmarks advanced visual and mathematical reasoning, yet few evaluated core physics concepts.Physics-specific environments such as PHYRE and Physion offered novel tasks, but complex simulators hindered reproducibility and symbolic problem solving.Source: §2 · p. 2
- The paper’s benchmark combines structured prompts and responses, multimodal reasoning, and procedural physics generation in a lightweight, replicable package.The related-work section describes a lightweight, reproducible physics reasoning framework as absent from contemporary multimodal benchmarks.Source: §2 · p. 2
3 Methodology
The methodology pairs simulator-free procedural physics problems with computational ground truth and a multidimensional evaluation protocol. It tests four differently scaled open-source VLMs and uses prompting and rubric-based scoring to assess their performance.
3 Methodology
- A multi-phase pipeline systematically generates, evaluates, and analyzes physics problems across diverse domains while emphasizing computational productivity and reliability.The framework architecture covers scenario generation, evaluation, experimental design, and results analysis.Source: §3 · p. 2, Figure 5 · p. 3
3.1 Model Selection
- Four state-of-the-art open-source models representing different scales and architectural approaches were selected to test scaling laws in physics reasoning across different model sizes within the current VLM landscape.Source: §3.1 · p. 3
3.2 Physics Environment Design and Problem Generation
- Algorithmic scenario generation samples domain-specific physical parameters into problems at easy, medium, and hard difficulty levels.Projectile Motion, for example, samples velocities of 10–50 m/s and angles of 15°–75°.Source: §3.2 · pp. 3–4
- A computational ground-truth engine solves each generated scenario using analytical formulae and numerical methods.The framework is designed to generate mathematically rigid problems without expensive simulators.Source: §3.2 · pp. 3–4
3.3 Evaluation Protocol and Metrics
- Evaluation combines physics accuracy, reasoning quality, computational efficiency, and consistency across domains and difficulty levels.Chain-of-Thought and few-shot prompting are used, while a rubric assesses principle detection, equation use, solution steps, and result interpretation.Source: §3.3 · p. 4
4 Experimentation and Results
Across four physics environments, larger model scale generally tracked stronger overall performance, while the evaluation exposed a contrast between formulaic tasks and spatial application. Efficiency and error analyses qualify that pattern with deployment trade-offs and persistent conceptual failures.
4.1 Overall Performance Analysis
- Qwen2.5-VL-7B scored 0.815 overall, 17% above DeepSeek-VL-1.3B (0.70), and the results report a strong positive correlation between parameter count and performance.Table 1 summarizes aggregate model performance and statistical confidence measures.Source: §4.1 · pp. 4–5, Table 1 · p. 4
4.2 Domain-Specific Performance Breakdown
- Fluid and collision tasks followed clearer formulaic or conservation-law patterns, while mechanics demanded difficult spatial reasoning and projectile problems ranged from simple trajectories to harder air-resistance cases.Domain results also show that Gemma2-27B-Vision led in projectile motion, collision dynamics, and mechanics, while the smaller Qwen2.5-VL-7B outperformed LLaMA-3.2-Vision-11B overall across domains.Source: Table 2 · p. 5, §4.2 · p. 6
4.3 Qualitative Analysis of Model Responses
- Qualitative examples showed robust cross-domain reasoning from Gemma2-27B-Vision, mixed capabilities and conceptual errors in mid-sized models, and descriptive rather than mathematical responses from DeepSeek-VL-1.3B.The examples were selected to illustrate typical reasoning patterns, common errors, and model-specific capabilities.Source: §4.3 · p. 6
4.4 Error Analysis
- Conceptual errors made up 52–67% of failures, compared with 23% versus 12% computational errors in smaller models and 8–15% perception errors.The authors interpret the prevalence of conceptual errors as suggesting pattern matching rather than deep understanding.Source: §4.4 · p. 6
4.5 Model Efficiency and Practical Considerations
- Gemma2-27B-Vision led in accuracy but required 5× DeepSeek-VL-1.3B’s inference time and 15× its memory; quantization degraded performance by <3% at 8-bit and 8–12% at 4-bit.The authors report that the performance-to-inference-time ratio favors smaller models, with medium-sized models providing optimal value for many practical applications.Source: §4.5 · pp. 6–7, Table 3 · p. 6
4.6 Statistical Significance and Reliability Analysis
- Overall-score differences between every model pair were statistically significant, but some domain comparisons, including Fluid Dynamics for Qwen2.5-VL-7B and LLaMA-3.2-Vision-11B, were not.The analysis used pairwise significance testing and bootstrap confidence intervals with 1,000 iterations.Source: §4.6 · p. 7, Figure 7 · p. 7
4.7 Implications for Scientific Community
- Models handled formula-based problems better than visual tasks requiring application of physical principles, and 8-bit quantization retained most performance while reducing computational requirements.The authors identify applied spatial reasoning as a continuing challenge for current VLMs.Source: §4.7 · p. 7
5 Conclusion
The paper presents a flexible framework for evaluating VLM physics reasoning and finds that formulaic problems remain easier than spatial reasoning and applied understanding. It identifies scaling as a viable trend while pointing to domain-specific variation and future evaluation extensions.
5 Conclusion
- The framework offers a flexible, segmented approach to evaluating VLM physics reasoning, with a lightweight, replicable design intended to address a research-community gap.The paper reports experiments across four physics domains and four state-of-the-art VLMs.Source: §5 · p. 8
- The evaluation finds progress on formulaic physics problems but persistent challenges in spatial reasoning and applied understanding; model size correlates strongly with accuracy, while domain variation may call for architectural innovation.The conclusion frames the need for a domain-specific architecture as a possibility rather than a demonstrated result.Source: §5 · p. 8
- Future directions include extending assessment to 3D physics, adding thermodynamics and electromagnetism, and studying cross-domain transfer.The authors also connect physics-reasoning evaluation to potential scientific-discovery and educational applications.Source: §5 · p. 8