PaperlayerView exact revision on arXiv

Source-linked AI summary

QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models

Li Puyin, Tiange Xiang, Ella Mao, Shirley Wei, Xinye Chen, Adnan Masood, Li Fei-fei, Ehsan Adeli

arXiv:2512.19526v1cs.AI

TL;DR

Most physical-understanding benchmarks use qualitative VQA, offering limited fine-grained evidence of numerical accuracy. QuantiPhy evaluates video-based kinematic estimates from a physical prior and finds that current VLMs rely heavily on parametric priors rather than reliably grounding quantitative facts in visual observations.

  • Problem

    Existing physical-understanding benchmarks are mostly qualitative VQA, limiting fine-grained evaluation of numerical accuracy, which is a necessary next step for VLM physical reasoning.

    Source: §1 · p. 2

  • Method

    QuantiPhy benchmarks video-based kinematic inference by asking VLMs to use a physical prior to predict related object properties in numerical, real-world units.

    Source: §1 · p. 2

  • Results

    Current state-of-the-art VLMs have not established a reliable link between visual observations and quantitative physical facts, often favoring parametric priors over visual evidence.

    Source: §6 · p. 9

  • Takeaways & Limitations

    This disconnect limits the reliability of current systems as precise visual measurers for real-world embodied agents.

    Source: §6 · p. 9

    The dataset covers only translational motion, fixed-camera views, rigid objects, and isolated movements, omitting rotation, dynamic viewpoints, deformables, and multi-object interactions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Understanding the physical world is essential for generalist AI agents. However, it remains unclear whether state-of-the-art vision perception models (e.g., large VLMs) can reason physical properties quantitatively. Existing evaluations are predominantly VQA-based and qualitative, offering limited insight into whether these models can infer the kinematic quantities of moving objects from video observations. To address this, we present QuantiPhy, the first benchmark designed to quantitatively measure a VLM's physical reasoning ability. Comprising more than 3.3K video-text instances with numerical ground truth, QuantiPhy evaluates a VLM's performance on estimating an object's size, velocity, and acceleration at a given timestamp, using one of these properties as an input prior. The benchmark standardizes prompts and scoring to assess numerical accuracy, enabling fair comparisons across models. Our experiments on state-of-the-art VLMs reveal a consistent gap between their qualitative plausibility and actual numerical correctness. We further provide an in-depth analysis of key factors like background noise, counterfactual priors, and strategic prompting and find that state-of-the-art VLMs lean heavily on pre-trained world knowledge rather than faithfully using the provided visual and textual inputs as references when reasoning kinematic properties quantitatively. QuantiPhy offers the first rigorous, scalable testbed to move VLMs beyond mere verbal plausibility toward a numerically grounded physical understanding.

1. Introduction

QuantiPhy addresses the lack of fine-grained evaluation of VLMs’ numerical physical reasoning by defining a kinematic inference task and a benchmark for video-based estimates. Its analysis finds that existing VLMs rely heavily on pre-trained world knowledge rather than the reference video and text.

Introduction

  • Measuring VLMs’ numerical accuracy in physical reasoning is important for applications such as embodied AI [12], AR/VR [18] [48], and autonomous driving [40].Source: §1 · p. 2
  • In-the-wild video often lacks the color, depth, object meshes, and camera parameters that methods such as FoundationPose require for accurate 3D localization.Evaluating VLM reasoning may also help assess and improve the physical realism of videos created by generative models.Source: §1 · p. 2
  • Existing physical-reasoning benchmarks are mostly qualitative VQA, which treats small and large numerical errors as equally wrong and cannot capture the size of the gap.For a car whose true size is 3 meters, the passage contrasts answers of 3.1 meters and 31 meters; quantitatively, the latter is 10× worse.Source: §1 · p. 2
  • QUANTIPHY evaluates video-based estimates of object size, velocity, and acceleration by formalizing how one physical prior can yield numerically grounded predictions in real-world units.The benchmark spans 2D/3D motion, static and dynamic priors, and diverse scene conditions, and provides standardized metrics, prompting, and a leaderboard over 21 state-of-the-art models.Source: §1 · p. 2
  • Analysis of scene complexity, video availability, counterfactual priors, and chain-of-thought prompting finds that VLMs rely heavily on pre-trained world knowledge and hardly infer from the reference video and text.Source: §1 · p. 2

2. Related Work

Prior work has broadened VLM physical-reasoning evaluations, but most remain qualitative or VQA-based, leaving quantitative reasoning about moving objects’ kinematics insufficiently benchmarked. Other vision methods and human reasoning illustrate ways to infer physical quantities from visual input.

2.1. Benchmarks for VLM Physical Understanding

  • VLM benchmarks broadened from synthetic events to object properties and wider physics [45], but mostly use VQA and leave moving-object metric kinematics unbenchmarked.VSI-Bench and Super-VSI [55] explore numerical spatial understanding, but VSI-Bench focuses on static objects.Source: §2.1 · pp. 2–3

2.2. Physical Reasoning Models

  • ChatGPT-4o and Gemini-1.5 Pro reach only around 60% accuracy on PhysBench versus humans’ ∼95%, while embodied reasoning remains qualitative.Generated videos often violate basic physical laws [7], with generative models more sensitive to low-level visual cues than underlying physical properties.Source: §2.2 · p. 3

2.3. Physical Understanding and Reasoning

  • Optical flow and object detection and tracking can infer motion quantities from video [33], while humans reason about scale and motion using relational cues and prior knowledge [1] [23].These human abilities support the potential for visual inputs to foster grounded quantitative understanding in large AI models.Source: §2.3 · p. 3

3. Methods

QUANTIPHY evaluates whether vision-language models can use a single physical prior and video to estimate moving objects’ kinematic properties numerically. It defines task categories and builds the benchmark from simulated, laboratory-captured, and in-the-wild videos with numerical annotations.

3.1. Overview of QUANTIPHY

  • Given a video and one real-world size, velocity, or acceleration prior for a source object, QUANTIPHY asks a VLM to estimate target-object properties in world space.The target may be the same as or different from the source; the known prior and its pixel-space counterpart determine a scale factor used to rescale other kinematic quantities.Source: §3.1 · p. 4
  • The benchmark crosses 2D or 3D movement with static size or timestamped velocity priors, yielding four task categories; rotational motion is excluded.The four categories are 2D-Static, 2D-Dynamic, 3D-Static, and 3D-Dynamic; 3D motion includes changing depth and is described as more challenging.Source: §3.1 · p. 4
  • QUANTIPHY contains 569 unique videos and 3,355 question–video instances, organized as prompt, video, and numerical-ground-truth triplets.Videos typically last 2–3 seconds and occupy approximately 115 MB, making the benchmark suitable for varied hardware settings.Source: §3.1 · p. 4

3.2. Benchmark Construction

  • Data collection combines controllable Blender simulations, 4D laboratory captures, and manually screened internet videos selected for moving-object coverage and known-size reference objects.Blender enables controlled motion and background variation, including scenes at microscopic and astronomical scales; lab captures span motions such as free fall and bouncing, while internet videos add in-the-wild scenarios.Source: §3.2 · p. 5, Figure 4 · p. 5
  • Applying SAM [32] segments moving objects against solid backgrounds, doubling the dataset without additional annotation for controlled background-complexity analysis.Source: §3.2 · p. 5
  • Annotations use automated property extraction for Blender scenes, metric depth and multi-view stereo reconstruction for lab captures, and pixel-space measurements mapped to world scale for internet videos.Lab object masks and viewpoint-specific metric depth provide the VLM’s depth prior; internet scale mapping uses reference objects with known dimensions.Source: §3.2 · pp. 5–6
  • Prompts pair the video with a textual physical prior, encourage careful analysis, and require the model to output only a numerical answer and unit.Source: §3.2 · p. 6

4. Evaluation on QUANTIPHY

The evaluation finds quantitative kinematic inference challenging for current VLMs: even the strongest systems score below the human average. It assesses 21 VLMs using MRA across four kinematic categories and compares their performance with human estimates.

4.1. Evaluation Setup

  • The evaluation tests 21 VLMs across four kinematic categories and uses MRA, which averages accuracy across relative-error tolerances rather than relying on brittle exact matches.Humans answered quantitative questions about 18 videos using the same priors and task definitions, but unlike VLMs they relied on perception, intuition, and coarse visual estimates rather than pixel-accurate video tensors.Source: §4.1 · p. 6

4.2. Main Results

  • ChatGPT-5.1 leads overall at 53.1 MRA, below the human average of 55.6; it slightly surpasses humans on 2D-Dynamic, but no VLM exceeds the human average.Human scores range from 50.0 on 2D-Static to 59.1 on 2D-Dynamic. Table 1 summarizes results across all four categories.Source: §4.2 · pp. 6–7
  • The best open-weight model, Qwen3-VL-Instruct-32B, scores 46.0 overall MRA, followed by InternVL-3.5-30B at 40.7 and Qwen3-VL-Instruct-8B at 38.8.These leading open-weight models trail top proprietary models but are comparable to mid-tier closed models; smaller open-weight models remain far behind human and large-model performance.Source: §4.2 · p. 7
  • Scaling improves MRA within Qwen3-VL and InternVL families, with stronger gains on dynamic categories, but diminishing returns leave a gap to humans and top proprietary models.Qwen3-VL rises from 29.0 at 2B to 38.8 at 8B and 46.0 at 32B; InternVL scores 25.0 at 2B, 35.4 at 8B, and 40.7 at 30B.Source: §4.2 · p. 7
  • Although pixel-level access could theoretically enable exact kinematic calculations, current VLMs cluster around 50% MRA, indicating visual precision and physical priors remain underused.The authors identify robust, accurate quantitative kinematic reasoning as an open challenge.Source: §4.2 · p. 7

5. Dissecting Quantitative Reasoning in VLMs

The analysis finds that scene references can aid quantitative inference, but VLMs rely more on learned world knowledge than on the specific video or numerical prior, and structured prompting does not systematically improve results.

5.1. Effect of Scene Context

  • Background complexity had only a mild effect on MRA, while visually complex scenes outperformed the other background conditions for most models.SAM-denoised scenes performed slightly better than simple-texture scenes; realistic backgrounds may provide scale and motion cues.Source: §5.1 · p. 7, Figure 5 · p. 9
  • Multiple-object scenes consistently yielded higher MRA than their single-object counterparts, with extra objects providing implicit comparison targets for size and speed.This trend held across plain, textured, and visually complex backgrounds.Source: §5.1 · p. 7

5.2. Do VLMs Use Videos and Priors Faithfully?

  • On a 161-pair subset, prior-only MRA was often close to and sometimes only slightly below video-plus-prior MRA, including for several strong systems.The video was removed while retaining the physical prior, object description, and question, indicating limited added value from the specific video frames in this probe.Source: §5.2 · p. 8
  • MRA fell by 80% for most models and 70% for the strongest model under counterfactual priors, whose outputs stayed near real-world magnitudes rather than scaling with the altered priors.The test multiplied each prior by α values from 0.001 to 700 while keeping the video and question unchanged; the analysis used a 161-instance 2D-only subset.Source: §5.2 · p. 8

5.3. Do Structured Prompts Help VLMs Reason?

  • Only three of 21 models improved with chain-of-thought prompting, while 19—including several strong open-weight systems—performed worse than with direct zero-shot prompting.The protocol decomposed inference into pixel-level source measurement, scale estimation, pixel-level target measurement, and real-world target estimation; intermediate numerical errors could propagate.Source: §5.3 · p. 9

6. Discussion and Future Work

The discussion concludes that current VLMs rely more on parametric priors than pixel-level evidence when estimating physical quantities. It identifies dataset limitations and calls for broader physics coverage and physics-informed training to advance embodied reasoning.

Summary of Findings

  • Current VLMs show weak input faithfulness, often overriding visual evidence and explicit inputs with memorized world knowledge when estimating kinematic properties.The authors characterize them as approximate guessers based on semantic context rather than precise visual measurers.Source: §6 · p. 9

Limitations

  • The dataset covers only translational motion, fixed-camera views, rigid objects, and isolated movements, omitting rotation, dynamic viewpoints, deformability, and multi-object interactions.Source: §6 · p. 9

Conclusion and Future Work

  • Future work should broaden video datasets to omitted physics and explore physics-informed objectives or pre-training on physics-rich data for embodied reasoning.The stated aim is to advance generalist embodied agents capable of sophisticated reasoning and interaction with the physical world.Source: §6 · p. 9

Supplementary Material … A.2. Metric Design Justification

Supplementary case studies show that VLM answers can be numerically plausible yet rely on world knowledge rather than the supplied visual and numerical evidence. The metric discussion defines MRA as a thresholded relative-accuracy measure intended to provide interpretable, robust evaluation.

A.1. Additional Case Studies

  • In a successful case, ChatGPT-5.1 measured pixel boxes, calibrated them using the supplied length prior, and computed a near-ground-truth width, though this input-faithful pipeline was fragile.The broader case analysis reports that the model often falls back to pretrained knowledge or generic heuristics when inputs are counterfactual or familiar assumptions do not apply.Source: §A.1 · pp. 14–15
  • Repeated prompts produced substantially variable responses, so the authors manually selected coherent, numerically valid traces; they characterize this single-model case study as diagnostic, not evaluative.They also report instability in API evaluation at temperature 0 and use multiple trials and failure rates in the main benchmark.Source: §A.1 · p. 14
  • With the car-length prior changed to 5670 m, ChatGPT-5.1 ignored the video and input, guessed width from a typical-car heuristic, and still achieved relative accuracy close to 0.9.This illustrates that outcome-based metrics can reward a numerically good answer even when its reasoning does not use the provided evidence.Source: §A.1 · pp. 14–15, Figure 7 · p. 15
  • Without video, ChatGPT-5.1 predicted car speed as 12 m/s, far from ground truth, but estimated width with relative accuracy ≈0.7, indicating reliance on typical-dimension priors.The case suggests that even with video available, size inference may draw on pretrained world knowledge rather than explicit pixel measurements.Source: §A.1 · p. 15
  • For a basketball scene with nonstandard acceleration near 1 m/s^2, ChatGPT-5.1 instead used gravity, predicting 9.8 m/s^2 and 14.7 m/s with relative accuracy 0 on both queries.The trace contained no pixel measurements or scale computations; the model applied 9.8 × 1.5 to estimate speed.Source: §A.1 · p. 15, Figure 9 · p. 16

A.2. Metric Design Justification

  • MRA compares prediction and ground truth by relative error |ŷ − y|/y, then averages binary accuracy across thresholds from 0.5 to 0.95 in steps of 0.05.A prediction receives partial credit when its relative error falls below a confidence threshold.Source: §A.2 · p. 16
  • MRA’s thresholded scale offers interpretable graded correctness and tolerates ambiguity and measurement noise, including temporal aliasing, motion blur, and imprecise priors.It is presented as more flexible than a regression-style loss for these natural imperfections.Source: §A.2 · p. 16
  • MRA follows Yang et al.’s VSI-Bench metric and subsequent multimodal benchmarks [9], supporting comparability and evaluation of prediction proximity across varying scales.The authors conclude that it balances informativeness, robustness, and interpretability for physically grounded numerical tasks.Source: §A.2 · p. 16

A.3. Model MRA Distribution

Figure 10 shows a marked MRA hierarchy: frontier proprietary models have higher, more consistent scores, while smaller or older models are concentrated at low scores. Many distributions also include zeros, which represent either zero answers or failed numerical outputs, including API failures treated as zero.

A.3. Model MRA Distribution

  • Figure 10 plots per-question MRA density curves with means, medians, and summary statistics; its mean is not the category-averaged Avg. MRA reported in Table 1.The displayed statistics include mean, standard deviation, median, Q25, and Q75; Table 1's Avg. MRA averages across 2D-Static, 2D-Dynamic, 3D-Static, and 3D-Dynamic task categories.Source: §A.3 · p. 16, Figure 10 · p. 17
  • Top-performing VLMs have MRA means and medians around 0.4–0.6, with smoother, more concentrated distributions than lower-performing systems.These models include ChatGPT-5.1, Gemini-2.5 Pro, Gemini-2.5 Flash, Qwen3-VL-instruct-32B, and Grok-4.1-Fast-Reasoning. Figure 10 visualizes the model distributions; its density patterns and comparative statistics show the performance gap.Source: §A.3 · pp. 16, 18, Figure 10 · p. 17
  • A second tier has means and medians around 0.3–0.4, while mid-tier models average 0.2–0.3 and are weighted toward low MRA values.The second tier includes InternVL-3.5-30B, Qwen3-VL-instruct-8B, Molmo-7B, and ChatGPT-5; mid-tier examples include Qwen-3-VL-instruct-2B, Phi-4-Multimodal-Instruct, SmolVLM-Instruct, CogVLM2 Video, Claude-4.5 Sonnet, and VILA-7B. The broader second-tier distributions indicate greater variability, while mid-tier systems rarely reach top-tier performance.Source: §A.3 · p. 18
  • The weakest models have means below 0.15 and medians around 0.0, with distributions concentrated near zero.Examples include Phi-3-Mini-128K-Instruct-3.8B, LLaVA-13B, Fuyu-8B, and MiniCPM-V 4.5-8B.Source: §A.3 · p. 18
  • An MRA of zero can indicate a zero answer or invalid numerical output, and failed API cases are treated as zero despite possible API or environment instability.The cited potential causes include API server errors, the running environment, internet traffic, and batch-size variations.Source: §A.3 · p. 18

B. Dataset Construction Guidelines … C.3. Internet Scraping

QuantiPhy’s dataset construction combines constrained video selection and a 36-category taxonomy with Blender scenes, lab-captured footage, and real-world internet videos, using varied motion-generation methods to support quantitative kinematic inference. These methods provide varied visual settings and annotated motion quantities while balancing controlled physical simulation and precise measurement against realism and annotation precision.

B.1. General Principles

  • Videos use a static camera and include a translating rigid object; 2D tasks additionally constrain motion to a plane parallel to the image plane.The dataset also requires ethically sourced assets and anonymizes personally identifiable information in raw footage.Source: §B.1 · p. 18
  • Each video–text record encodes its source and task configuration, and prompts specify units and whether velocity or acceleration is instantaneous or averaged.Records also include frame rate, which supports consistent time and kinematics calculations.Source: §B.1 · pp. 18–19
  • The dataset aims to balance four core task categories and populate all 36 fine-grained configurations while covering varied scenes and motion patterns.Examples range from astronomical and microscopic settings to everyday scenes, with uniform, accelerated, projectile, oscillatory, and centripetal motion.Source: §B.1 · p. 19

B.2.1. Video Categories Definition

  • The dataset partitions clips into 36 categories defined by physical prior, dimensionality, queried-object setting, and background complexity.The prior is size, velocity, or acceleration; object settings distinguish single-object reasoning from questions requiring multiple objects, while backgrounds are plain, simple, or complex.Source: §B.2.1 · pp. 19–20

B.2.2. Quantitative breakdown of video types.

  • 569 videos span all 36 intended configurations, with 328 2D and 241 3D clips and per-code counts from 4 to 51.Table 4 reports 300 Blender-generated, 72 internet-sourced, 112 captured, and 85 segmented videos.Source: §B.2.2 · p. 20, Table 4 · p. 23

C.1.1. Blender Toolkits and Asset Sources

  • BlenderKit supplies themed environments, while Sketchfab supplies rigged foreground models with reusable animations, reducing manual scene and motion authoring.Assets are selected under licenses permitting reuse and modification.Source: §C.1.1 · pp. 21, 23, 24

C.1.2. Two Motion Simulation Types

  • Keyframed motion is visually plausible but not necessarily physically calibrated, whereas force-based or analytical motion yields exact displacement, velocity, and acceleration ground truth.Models should use explicitly provided forces or kinematic priors rather than infer physical constants from visual appearance alone.Source: §C.1.2 · p. 24

C.1.3. Blender Videos Construction

  • The Blender subset contains 312 clips generated from approximately 125 base models and 81 scenes, with scripted control enabling accurate ground-truth quantities.Its scenes include everyday indoor and outdoor settings as well as microscopic and extraterrestrial scenarios; background variants are created by simplifying or enriching scene conditions, while balancing reuse with novel scenes to reduce unwanted correlations.Source: §C.1.3 · pp. 24–25
  • Blender scenes use keyframed or physics-driven motion, with analytic trajectories and arc-length reparameterization imposing controlled kinematic profiles on straight or curved paths.Uniform rescaling can make otherwise barely visible motion measurable while preserving physical relationships; rendering varies resolution, frame rate, camera, and appearance while retaining controlled trajectories and ground truth.Source: §C.1.3 · pp. 26, 28, 29, 30

C.2. Lab Capturing

  • Lab videos capture 3D motion with four-camera motion capture, but incomplete or ambiguous depth data prevented reliable automated object-depth extraction from segmentation masks.The team instead developed a UI for reading depth values at selected pixels, and measured physical objects before filming for use as priors or inference targets.Source: §C.2 · p. 30

C.3. Internet Scraping

  • The internet split contains 42 open-platform clips and 30 author-recorded smartphone videos, with suitable clips hand-selected for static cameras, rigid-object translation, and visible physical priors.Author recordings supplement scarce qualifying platform clips and use the same approximate-scale annotation procedure; faces, plates, and other identifying details are blurred or masked.Source: §C.3 · pp. 30–32
  • Internet videos are limited to 2D inference because calibrated geometry is unavailable; pixel-ruler trajectories and approximate real-world scale provide less precise ground truth, so their share is kept moderate.Scale references include gravity, familiar object dimensions, and known conveyor-belt speeds.Source: §C.3 · pp. 30–31

C.4. Segmented Data … D.2. Lab Data Annotation

This section describes QuantiPhy’s video segmentation, quality control, dataset ethics, and annotation procedures for simulated and lab data. It covers deriving and classifying motion and geometric quantities from scene data or reconstructed lab coordinates while distinguishing auxiliary priors from ground-truth targets.

C.4. Segmented Data

  • The benchmark compares original videos with versions whose backgrounds are removed to test how background context affects physical-property inference.SAM 2 segments targets using Grounding DINO-generated box prompts; annotators can intervene with point prompts through a custom UI for complex scenes.Source: §C.4 · pp. 32–33, Figure 29 · p. 33

C.5. Quality Control

  • Manual review removes videos with blur, severe occlusion, difficult-to-model objects, or identifiable people; retention varies across Blender, lab, and web data.Approximately 3% of Blender data and 30% of lab data were discarded, while only 72 internet-scraped clips were retained.Source: §C.5 · p. 33

C.6. Ethical Considerations

  • Dataset curation follows copyright and licensing rules, retains no personally identifiable information, and screens for harmful or biased content while prioritizing diversity.IRB documentation is available upon request.Source: §C.6 · pp. 33–34

D.1. Blender Simulation

  • Blender annotations cover five physical quantities: size, displacement over an interval, velocity, acceleration, and depth and distance.Source: §D.1 · p. 34

D.1.1. Size

  • Size labels use Blender’s internal measurements, with object-specific protocols for changing dimensions and inference questions phrased at matching precision.Humans receive standing-rest, minimum walking, and maximum walking heights; flying animals receive minimum and maximum flight widths, without a separate rest measurement.Source: §D.1.1 · p. 34

D.1.2. Displacement or Path.

  • Displacement labels are computed from Blender world-space trajectories, with timestamps aligned to the video’s render-start frame even when its Blender frame index is nonzero.Source: §D.1.2 · p. 34

1. The full 3D displacement

  • For each interval, the protocol stores 3D and planar displacement values and specifies in each question which displacement type and plane defines the ground truth.The question distinguishes full 3D displacement from planar displacement in a specified plane, such as horizontal or vertical.Source: §1 · p. 37

D.1.3. Velocity and Acceleration.

  • Velocity and acceleration are derived frame by frame from world-space positions and speeds, then queried by mapping the requested time to a frame.The protocol diagnoses clips as stationary, approximately uniform-speed, or non-uniform-speed, and as near-zero, approximately uniformly accelerated or decelerated, or irregularly accelerated.Source: §D.1.3 · pp. 37–38
  • Threshold-based acceleration diagnostics tag clips as constant-speed, uniformly accelerated or decelerated, or irregular without modifying frame-level values.These tags support questions phrased at a precision matched to the motion regime in each clip.Source: §D.1.3 · p. 39

D.1.4. Depth and Distance

  • Depth is auxiliary metadata or an optional prior, never an evaluation target, while selected aligned depth values can support inference about scene configuration.Exposed values are manually reviewed for logical sufficiency alongside visual information.Source: §D.1.4 · p. 39
  • Inter-object distances are annotated in 3D or projected planes, with 2D/3D task labels determined by the video geometry and whether the question restricts the plane.Camera distances remain depth metadata; inter-object distances provide ground-truth targets.Source: §D.1.4 · pp. 39–40

D.2. Lab Data Annotation

  • Lab annotations use calibrated-camera geometry rather than unstable off-the-shelf reconstruction, which struggled to localize objects under occlusion.Annotators click object centers across frames, pair image coordinates with metric depth, and back-project them into world coordinates.Source: §D.2 · p. 41
  • Lab object sizes are measured manually, while velocity and acceleration use center annotations from at least five adjacent frames; only task-relevant timestamps are annotated, and other cameras support occlusion estimates.Dynamic quantities are computed from world-coordinate sequences assuming a constant frame rate; for main-camera occlusion, coordinates are averaged from annotations transformed from the other three cameras.Source: §D.2 · p. 41

D.3. Internet Data Annotation · D.4. Segmented Data

QuantiPhy derives internet-video ground truth through metadata and task specification, pixel-level measurement, and conversion into real-world kinematics. For segmented videos, it reuses the original annotations because segmentation changes the background, not the objects’ physical properties.

D.3. Internet Data Annotation

  • The internet-video workflow converts raw-pixel observations into real-world kinematic ground truth through three stages: task specification, pixel measurement, and unit conversion.Source: §D.3 · p. 41
  • Annotators identify each clip’s available physical prior, feasible inference targets, inference type, and explicit quantity, unit, and time reference for each question.Source: §D.3 · p. 41
  • A Python/OpenCV tool verifies or accepts frame rate, checks frame counts, reports video properties, and supports frame navigation and pixel-coordinate readings.Source: §D.3 · p. 41
  • For each target time or interval, annotators record object pixel length and reference-point positions across frames using the verified frame rate and time step Δt = 1/f.Source: §D.3 · p. 41
  • Finite differences convert measured pixel trajectories into pixel-space kinematics, applied component-wise when motion spans two image-plane directions.Source: §D.3 · pp. 41–42
  • Assuming planar motion, annotators estimate a world-length-per-pixel scale from the available size, velocity, or acceleration prior, then rescale kinematic quantities into world units.The resulting world-space values serve as QuantiPhy’s ground-truth priors and posteriors.Source: §D.3 · p. 42

D.4. Segmented Data

  • Segmented videos reuse the original annotations without modification because isolating target objects changes the background but not their physical properties.Source: §D.4 · p. 43

E. Vision-Language Models

The evaluation covers 21 VLMs—6 proprietary and 15 open-source—across model scales, architectures, input modalities, and deployment paradigms. It includes flagship proprietary models and open-source systems selected to examine scaling and temporal processing.

E. Vision-Language Models

  • Flagship models from OpenAI, Google, Anthropic, and xAI represent proprietary VLMs, including systems with inference-time reasoning capabilities.The models include GPT-5.1, GPT-5, Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Sonnet 4.5, and Grok 4.1 (Fast Reasoning).Source: E. Vision-Language Models · p. 43
  • The 15 open-source models span architectures and input modalities, from LLaVA-13B [25], VILA-7B [24], and Qwen3-VL-8B [5] to models built for efficient reasoning.The suite also includes Fuyu-8B, Molmo-7B, SmolVLM, Phi-4 Multimodal, and Phi-3-Mini-128K-Instruct [28].Source: E. Vision-Language Models · p. 43
  • Qwen3-VL-Instruct [5] and InternVL-3.5 [10] are evaluated at three parameter scales each to compare performance across model sizes.The scales are 2B, 8B, and 32B for Qwen3-VL-Instruct and 2B, 8B, and 30B for InternVL-3.5.Source: E. Vision-Language Models · p. 43
  • MiniCPM-V 4.5 [56] and CogVLM2-Video [43] accept direct video inputs for native temporal processing, unlike frame-based approaches.Source: E. Vision-Language Models · p. 43
  • The evaluation encompasses 21 VLMs: 6 proprietary and 15 open-source models spanning different scales, architectures, and deployment paradigms.The suite includes models accessed through closed-source APIs, hosted via Replicate, and deployed by the authors.Source: E. Vision-Language Models · p. 43

F. Prompt Design … H.3. Evaluation Metric

The paper specifies constrained, reproducible prompts and a hierarchical parser for numerical VLM evaluation, then describes a human study using matched kinematic tasks and the same MRA metric.

F. Prompt Design

  • Prompts restrict responses to numerical answers with units and combine video frames, system instructions, priors or depth information, a question, and a format-reinforcing post-prompt.The sequence is standardized across models, and temperature is set to 0 where supported; hosted models without temperature control use maintainers’ defaults.Source: F. Prompt Design · pp. 43, 46
  • Videos are normalized to 480p while retaining all source frames, because preliminary exploration found temporal subsampling significantly degraded velocity and acceleration tracking.The selected expert-video-analyst persona followed a five-prompt pilot on 15 videos, with no significant reasoning-accuracy difference and relatively highest formatting adherence.Source: F. Prompt Design · p. 46

G. Answer Retrieval and Parsing

  • The hierarchical parser first accepts concise numeric responses, then searches verbose outputs for answer delimiters, sanitizes units, and extracts the last valid number or records failure.It takes the absolute value; qualitative-only responses, “Cannot be determined,” API errors, or responses without a numerical conclusion are treated as failures.Source: G. Answer Retrieval and Parsing · p. 46
  • Claude Sonnet 4.5, Fuyu-8B, and MiniCPM-V 4.5 often produced verbose explanations, but parsing post-delimiter text and the last number retrieved answers from these outputs.Representative raw responses and parsed values are shown in Figure 37.Source: G. Answer Retrieval and Parsing · p. 46

H.1. Participants

  • The human study recruited undergraduate, graduate, and PhD researchers, including technically trained participants, who could choose their own reasoning strategy.The 2D and 3D surveys involved multiple participants, and some completed both to permit within-person cross-dimensional comparisons.Source: §H.1 · pp. 46, 48

H.2. Task Construction and Experimental Design

  • Participants were randomly assigned to a 2D or 3D survey and answered 1–3 quantitative kinematic questions after each of 18 videos sampled across categories.They could replay, pause, and scrub videos; scrubbing was limited to whole seconds to match VLM temporal access, while replay was unrestricted.Source: §H.2 · p. 48

H.3. Evaluation Metric

  • Human responses use the same MRA metric as model evaluation, averaging accuracy across ten relative-error tolerance thresholds and then averaging question scores per participant.Figure 39 presents participant performance for the 2D and 3D surveys.Source: §H.3 · p. 48

H.4. Results and Observations · I. Sketchfab Model Sources

Top human participants substantially outperformed evaluated VLMs, while human performance was highly consistent across 2D and 3D tasks. The following section also identifies the Sketchfab models used to generate dataset videos.

H.4. Results and Observations

  • Among participants who completed both surveys, 2D and 3D MRA scores were highly correlated, with high performers in 2D almost universally remaining high performers in 3D.The authors interpret this as stable, transferable human physical intuition across dimensional modalities.Source: §H.4 · p. 48
  • Top human participants reached MRA = 0.721 in 2D and MRA = 0.724 in 3D, exceeding the evaluated VLMs.The authors note that this upper-bound gap is notable despite the study's modest sample size.Source: §H.4 · p. 48
  • Average human performance was not higher than model performance, even though the top human participants remained far ahead of current VLM capabilities.The comparison distinguishes average human performance from the human upper bound.Source: §H.4 · p. 48
  • The human findings suggest that strong physical reasoning competence remains challenging for contemporary VLMs, even with a modest human-study sample.This conclusion is supported by the human upper-bound scores relative to evaluated models.Source: §H.4 · p. 48
  • The authors conclude that VLMs may approximate average-level human intuition in some settings but remain short of human-like precision and expert-level reasoning.The human study is presented as a benchmark for the gap between current VLMs and physically grounded, human-level visual reasoning.Source: §H.4 · p. 48

I. Sketchfab Model Sources

  • Table 7 lists the model names, authors, and model IDs of Sketchfab models used in Blender-generated dataset videos.The table is titled “Sketchfab 3D Models” and continues from a previous page.Source: §I · pp. 48, 50, Table 7 · p. 49

Related papers

Loading 2512.19526v1…