PaperlayerView exact revision on arXiv

Source-linked AI summary

QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models

Li Puyin, Tiange Xiang, Ella Mao, Shirley Wei, Xinye Chen, Adnan Masood, Li Fei-fei, Ehsan Adeli

arXiv:2512.19526v1cs.AI

TL;DR

Existing VLM evaluations are largely qualitative and do not finely measure numerical physical reasoning. QuantiPhy introduces a quantitative benchmark for kinematic inference and finds that current VLMs remain unreliable, often favoring memorized world knowledge over visual evidence.

  • Problem

    Existing physical-reasoning benchmarks are predominantly qualitative VQA evaluations, limiting fine-grained assessment of numerical understanding.

  • Method

    QuantiPhy evaluates VLMs on numerically grounded inference of object kinematic properties from videos using standardized prompts and a calibrated metric.

  • Results

    State-of-the-art VLMs have not established a reliable link between visual observations and quantitative physical facts, and even the best systems remain below human performance.

  • Takeaways & Limitations

    Current VLMs act more like approximate semantic guessers than precise visual measurers, limiting their reliability for embodied agents.

  • Takeaways & Limitations

    The benchmark covers only translational movement, fixed camera viewpoints, rigid objects, and simplified isolated motions.

Abstract

from arXiv · show

Understanding the physical world is essential for generalist AI agents. However, it remains unclear whether state-of-the-art vision perception models (e.g., large VLMs) can reason physical properties quantitatively. Existing evaluations are predominantly VQA-based and qualitative, offering limited insight into whether these models can infer the kinematic quantities of moving objects from video observations. To address this, we present QuantiPhy, the first benchmark designed to quantitatively measure a VLM's physical reasoning ability. Comprising more than 3.3K video-text instances with numerical ground truth, QuantiPhy evaluates a VLM's performance on estimating an object's size, velocity, and acceleration at a given timestamp, using one of these properties as an input prior. The benchmark standardizes prompts and scoring to assess numerical accuracy, enabling fair comparisons across models. Our experiments on state-of-the-art VLMs reveal a consistent gap between their qualitative plausibility and actual numerical correctness. We further provide an in-depth analysis of key factors like background noise, counterfactual priors, and strategic prompting and find that state-of-the-art VLMs lean heavily on pre-trained world knowledge rather than faithfully using the provided visual and textual inputs as references when reasoning kinematic properties quantitatively. QuantiPhy offers the first rigorous, scalable testbed to move VLMs beyond mere verbal plausibility toward a numerically grounded physical understanding.

1. Introduction

Quantitative physical reasoning remains insufficiently evaluated in VLMs because existing assessments are largely qualitative, despite the importance of numerical physical understanding for embodied applications. QuantiPhy addresses this gap with a benchmark targeting numerically grounded kinematic inference from videos.

  • Numerical physical reasoning is important for deploying VLMs in embodied AI, AR/VR, and autonomous driving.
  • Existing VLM physical-reasoning benchmarks are predominantly qualitative VQA evaluations, which cannot distinguish small from order-of-magnitude numerical errors.For example, treating 3.1 meters and 31 meters as equally incorrect obscures a tenfold difference in error.
  • QuantiPhy introduces a quantitative paradigm for evaluating VLM physical reasoning beyond qualitative VQA.
  • The benchmark asks models to transform one physical prior into numerically grounded predictions of correlated properties such as size, velocity, and acceleration.
  • QuantiPhy systematically evaluates kinematic reasoning across 2D/3D motion, static and dynamic priors, diverse scenes, standardized scoring, and 21 state-of-the-art models.

2. Related Work

Prior work has established qualitative physical-reasoning benchmarks and numerical tools for motion analysis, but existing VLM evaluations remain limited in scope and quantitative grounding. QuantiPhy targets numerical reasoning over moving-object geometry and kinematics.

  • Existing studies cover events such as collisions and falling, as well as properties including mass, friction, elasticity, and deformability.
  • Most existing VLM physical-understanding benchmarks use multiple-choice or descriptive VQA, limiting fine-grained evaluation of physical reasoning.
  • VSI-Bench and Super-VSI introduce numerical spatial evaluation, but the cited work characterizes this line as focused primarily on static objects or spatial understanding.
  • QuantiPhy benchmarks quantitative reasoning over geometric and kinematic properties of moving objects, including size, velocity, and acceleration in metric units.
  • Optical flow with detection and tracking methods demonstrates that velocity and displacement can be inferred quantitatively from video inputs.

3. Methods

QuantiPhy constructs a video benchmark for inferring world-space kinematic properties from a single textual physical prior and pixel-space motion, using four task categories and diverse simulated, laboratory, and internet data. Its design combines standardized triplets, numerical ground truth, and annotation procedures tailored to each source.

  • Task formulation: QuantiPhy provides a VLM with a video and one real-world physical prior, then asks it to estimate target-object kinematic properties in world space.The source and target objects may be the same or different.
  • Task formulation: The benchmark separates pixel-space measurements from world-space quantities through an unknown positive scale factor inferred from a provided size, velocity, or acceleration prior.
  • Scope: The benchmark focuses on translational movement and excludes rotational movement, using a fixed-camera setting that avoids entangling camera motion with object motion.
  • Benchmark setup: The four task categories combine 2D or 3D motion with Static or Dynamic physical priors.2D excludes depth change, whereas 3D includes the z-axis; Static supplies object size and Dynamic supplies velocity.
  • Dataset statistics: The dataset contains 569 unique videos and 3355 questions, organized as multiple question-video-ground-truth triplets per video.Videos typically last 2–3 seconds and occupy approximately 115MB after post-processing.
  • Data collection: Data combines controllable Blender simulations, 4D laboratory captures, and selected internet videos requiring moving objects, relatively static cameras, and reference objects with known dimensions.
  • Annotation: Annotations use automated physical-property extraction for Blender, metric depth and multi-view reconstruction for laboratory captures, and manually annotated pixel measurements mapped to world scale for internet data.

4. Evaluation on QUANTIPHY

QUANTIPHY evaluates VLMs on numerical kinematic inference across four task categories, using standardized metrics and human comparisons. Current systems remain below human performance, while scaling improves results—especially for dynamic tasks—without closing the gap.

  • Evaluation Setup: MRA evaluates numerical predictions across tolerance thresholds, providing a more calibrated measure than exact matching for continuous physical measurements.The benchmark averages accuracy over thresholds from C = {0.1, 0.2, . . . , 0.9, 0.95}.
  • Main Results: 55.6 average MRA is achieved by human annotators across categories, with scores ranging from 50.0 to 59.1.The results show that the tasks are solvable with reasonable accuracy despite humans relying on coarse visual estimation.
  • Main Results: 53.1 MRA makes ChatGPT-5.1 the strongest model overall, but no evaluated VLM surpasses the human average of 55.6.Gemini-2.5-Pro follows at 49.6 MRA, while GPT-5 and Claude Sonnet 4.5 score around 32.6 and 22.8.
  • Main Results: 46.0 overall MRA makes Qwen3-VL-Instruct-32B the strongest open-weight model, followed by InternVL-3.5-30B at 40.7 and Qwen3-VL-Instruct-8B at 38.8.These models remain below the top proprietary systems but are comparable to mid-tier closed models.
  • Scaling Effects: Qwen3-VL performance rises from 29.0 at 2B parameters to 38.8 at 8B and 46.0 at 32B, with the largest benefits on dynamic categories.InternVL shows the same pattern, but diminishing returns leave scaled models below leading proprietary systems and humans.
  • Gap to Super-Human Performance: The best systems cluster around 50% MRA despite access to precise pixel information, indicating substantial under-use of visual precision and physical priors.An ideal agent with precise frame-level pixel coordinates could theoretically recover world–pixel scale and compute target quantities exactly.

5. Dissecting Quantitative Reasoning in VLMs

The analysis probes how scene context, input faithfulness, and structured prompting affect VLM quantitative reasoning. Models benefit from relational visual context but rely heavily on learned priors, while chain-of-thought prompting rarely improves performance.

  • 5.1. Effect of Scene Context: Multiple-object scenes consistently achieve higher MRA than corresponding single-object scenes, whereas background complexity has only a mild effect.Additional objects provide comparison targets for estimating size and speed; denoised scenes slightly outperform simple-texture scenes, while complex scenes often perform best.
  • 5.2. Do VLMs Use Videos and Priors Faithfully?: Prior-only inputs often approach video-plus-prior performance, indicating that many VLMs derive quantitative estimates mainly from learned knowledge rather than video frames.This pattern appears across most models on the 161-pair analysis subset, including several strong systems.
  • 5.2. Do VLMs Use Videos and Priors Faithfully?: Counterfactual priors reduce MRA by 80% for most models and by 70% for the strongest model, showing that predictions rarely track altered numerical inputs.Outputs remain close to real-world magnitudes even when the supplied prior is multiplied by factors ranging from 0.001 to 700.
  • 5.2. Do VLMs Use Videos and Priors Faithfully?: Together, the analyses characterize current VLMs as brittle, input-unfaithful quantitative reasoners dominated by pre-trained world knowledge.Visual evidence and explicit priors function more as soft hints than hard constraints in these tasks.
  • 5.3. Do Structured Prompts Help VLMs Reason?: Structured chain-of-thought prompting improves only three of 21 models, while the remaining 19 perform worse than with direct zero-shot prompting.The authors attribute this pattern to failures in intermediate numerical subproblems, which amplify early errors.

6. Discussion and Future Work

QuantiPhy finds that VLMs often fail to connect visual observations with quantitative physical facts, relying instead on memorized world knowledge. The study also identifies important scope limitations and directions for broader evaluation and training.

  • Summary of Findings: VLMs rarely infer kinematic properties from pixels faithfully, often overriding visual evidence and explicit inputs with parametric priors.The authors characterize current systems as approximate guessers based on semantic context rather than precise visual measurers.
  • Limitations: The benchmark covers translational motion from fixed-camera videos of rigid objects, excluding rotational dynamics, deformable materials, dynamic viewpoints, and complex multi-object interactions.These design choices simplify the evaluated physical reasoning setting relative to real-world scenarios.
  • Conclusion and Future Work: Future work should expand video data to omitted physical complexities and explore physics-informed objectives or physics-rich pretraining for embodied AI.The proposed directions target both broader evaluation coverage and training methods that better support physical reasoning.

A.1. Additional Case Studies

The case studies contrast successful pixel–prior computation with fragile failures caused by counterfactual inputs, absent video, and nonstandard physics. Together with the MRA design, they show why numerical accuracy must be assessed alongside input faithfulness.

  • Case 1: Faithful pixel–prior reasoning: ChatGPT-5.1 can measure pixel trajectories, apply a physical prior, and compute accurate kinematic quantities when the intended pipeline is followed.In the faithful case, it identifies relevant frames, extracts bounding boxes, calibrates pixel-to-meter scale, and performs proportional reasoning.
  • Analysis protocol: Repeated queries show substantial response variability, so the qualitative analysis selects representative coherent traces and the benchmark records failure rates across multiple trials.Instability persists in API-based evaluation even with temperature fixed to 0.
  • Case 2: Counterfactual prior breaks faithfulness: A counterfactual car-length prior causes the model to abandon scene measurements and use a generic width-to-length heuristic, producing a seemingly accurate answer for the wrong reasons.The altered prior is 5670 m, and the final width prediction achieves relative accuracy close to 0.9 despite ignoring the video and numeric input.
  • Case 3: Video ablation reveals reliance on priors: Without video, ChatGPT-5.1 gives a speed estimate far from ground truth, illustrating that motion estimation is difficult without visual evidence.The video-ablated prompt retains the yellow car’s length prior but removes the visual trajectory.
  • Case 4: Strong gravitational prior overrides counterfactual physics: Under counterfactual basketball physics, the model ignores the video and outputs canonical gravitational acceleration instead of the simulated trajectory’s approximately 1 m/s^2 behavior.The scene uses time-varying acceleration near 1 m/s^2 rather than standard gravity.
  • Metric Design Justification: MRA scores continuous predictions by averaging binary accuracy across relative-error thresholds from 0.5 through 0.95, providing graded tolerance to ambiguity and noise.The metric compares each prediction with ground truth using relative error and grants partial credit below confidence thresholds.

A.3. Model MRA Distribution

Figure 10 shows clear differences in MRA distributions across VLMs, with frontier models performing more accurately and consistently than smaller or older systems.

  • Table 1 averages MRA across four task categories, whereas Figure 10 averages MRA across individual questions.The two means therefore summarize different aggregation levels.
  • Frontier models show higher and more concentrated MRA distributions, with means and medians generally around 0.4–0.6.ChatGPT-5.1, Gemini-2.5 Pro, Gemini-2.5 Flash, Qwen3-VL-instruct-32B, and Grok-4.1-Fast-Reasoning cluster above moderate MRA values.
  • Mid-tier models have means around 0.2–0.4, but broader distributions indicate greater variability and frequent low-scoring cases.The second tier generally centers around 0.3–0.4, while mid-tier systems center around 0.2–0.3.
  • Smaller or older models perform substantially worse, with means below 0.15 and medians near 0.0.Phi-3-Mini-128K-Instruct-3.8B, LLaVA-13B, Fuyu-8B, and MiniCPM-V 4.5-8B have distributions concentrated near zero.
  • Many distributions contain substantial mass at zero because zero denotes either a zero answer or failed numerical output.The analysis also treats API and runtime failures as MRA equal to zero.
  • Overall, frontier proprietary models show a clear performance gap over lightweight or earlier-generation models.

B. Dataset Construction Guidelines

QUANTIPHY’s dataset construction emphasizes ethical sourcing, well-defined physical tasks, balanced coverage, and standardized video–text records. The guidelines constrain scene geometry and encode task conditions systematically, while acknowledging limits in manually authored internet motion.

  • Videos use fixed cameras and translationally moving objects, with planar motion assumed for 2D tasks to make kinematic inference well-defined.The 2D assumption keeps depth approximately constant and supports a consistent pixel-to-world mapping.
  • Each record standardizes metadata, prompts, physical priors, depth information, and numerical posterior answers for quantitative evaluation.Prompts specify units and distinguish instantaneous from average velocity or acceleration.
  • A four-character code identifies the physical prior, dimensionality, object setting, and background complexity.The first character encodes size, velocity, or acceleration; the second encodes 2D or 3D; the third distinguishes single- and multiple-object reasoning.
  • Rigid objects are defined operationally by motion that can be approximated through a stable center of mass, including slightly deformable entities.This permits locally trackable birds and walking people within the benchmark’s rigid-object treatment.
  • The benchmark balances four core task categories and spans 36 fine-grained video configurations.Each category contains at least four videos, supporting nontrivial representation across conditions.
  • Internet motion is visually plausible but may not obey physical laws, so its proportion is kept moderate because its annotations are less precise.Artist-authored trajectories are not derived from force-based physical simulation.

C.4. Segmented Data

QUANTIPHY creates segmented-video variants to isolate background effects while preserving the target objects’ original annotations. The workflow combines automated segmentation, manual correction, quality review, and standardized kinematic labels.

  • Background removal changes only the scene background, so segmented videos reuse the original physical annotations unchanged.
  • Grounding DINO supplies object boxes to SAM 2, while point prompts and a custom interface handle complex multi-object scenes.The interface supports multiple annotators and provides examples across Blender, lab, and internet data.
  • Manual review removes blur, severe occlusion, difficult-to-model objects, and identifiable human subjects before inclusion.Approximately 3% of Blender data and 30% of lab data were discarded; only 72 internet clips were retained.
  • The annotation protocol distinguishes human standing, minimum walking, and maximum walking heights, while flying animals receive minimum and maximum flight widths.These measurements match the wording of the corresponding inference questions and priors.
  • The dataset aligns timestamps to rendered frames using the scene’s frame rate and render-start frame, including nonzero starting indices.The conversion uses t = 0 at the first exported frame and rounds f_start + t_sec * fps.
  • Lab annotation attempts with FoundationPose failed frequently under occlusion and did not stably localize objects in world coordinates.

D.3. Internet Data Annotation

Internet videos receive a three-stage annotation workflow that measures pixel-space motion and converts it into real-world kinematics using an available physical prior. The resulting values become QUANTIPHY’s ground-truth priors and posteriors.

  • Internet clips are annotated through metadata specification, pixel-level measurement, and conversion from pixel-space to real-world quantities.
  • Annotators verify frame rates, inspect frame timestamps, and record object size and reference-point positions at discrete times.The custom OpenCV player supports frame navigation, timestamp inspection, and frame-by-frame review.
  • Finite differences produce pixel-space size, velocity, and acceleration for prior objects and inference targets.Two-dimensional image-plane motion is handled component-wise.
  • A single positive scale factor converts pixel measurements into world units under the planar-motion assumption.The factor is estimated from a size, velocity, or acceleration prior defined at timestamp t0.
  • The converted world-space quantities serve as QUANTIPHY’s ground-truth priors and posteriors.
  • Segmented internet videos retain the same physical annotations as their original versions.

E. Vision-Language Models

The evaluation covers 21 VLMs across proprietary and open-source systems, model scales, architectures, and deployment paradigms.

  • Proprietary Models: The proprietary set includes flagship multimodal models from four major providers, including inference-time reasoning systems.
  • Open-source Models: The open-source evaluation spans 15 models with varied architectures, parameter sizes, and input modalities.
  • Open-source Models: Qwen3-VL and InternVL-3.5 are each tested across multiple parameter scales to examine scaling effects within architecture families.
  • 21 VLMs are evaluated, comprising 6 proprietary and 15 open-source models.

F. Prompt Design

The benchmark standardizes video, prompt, generation, parsing, and human-study procedures for extracting precise numerical answers from VLMs. It preserves temporal information, enforces numerical output formats, and treats non-numeric responses as failures.

  • F. Prompt Design: Models are instructed to return only a numerical answer with units, and generation uses temperature=0 when supported.
  • F. Prompt Design: The input sequence combines full video frames, system instructions, physical priors or depth information, a question, and a final output-format reminder.
  • F. Prompt Design: Videos are normalized to 480p while retaining all source frames because temporal subsampling substantially degrades velocity and acceleration tracking.
  • G. Answer Retrieval and Parsing: The parsing pipeline first accepts exact numerical responses, then searches delimiters, removes units, and extracts the last valid number from verbose outputs.
  • G. Answer Retrieval and Parsing: Qualitative descriptions, indeterminate answers, API errors, and responses without numerical conclusions are treated as failures.
  • G. Answer Retrieval and Parsing: Verbose outputs from Claude Sonnet 4.5, Fuyu-8B, and MiniCPM-V 4.5 are handled by extracting post-delimiter text and the final numerical value.
  • H. Human Study Details.: The human survey includes structured 2D and 3D interfaces with video replay, timeline scrubbing, physical priors, quantitative questions, and numeric input fields.

H.4. Results and Observations

Human performance provides an upper-bound comparison for quantitative physical reasoning, with strong consistency across 2D and 3D survey conditions. The highest-performing participants substantially outperform the evaluated VLMs.

  • MRA scores for participants completing both tasks are highly correlated across 2D and 3D surveys.
  • Top human participants achieve MRA = 0.721 in 2D and MRA = 0.724 in 3D, outperforming the evaluated VLMs.
  • The human study frames the gap between current VLM capabilities and human-level precision as a benchmark for physically grounded visual reasoning.
Loading 2512.19526v1…