PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding

Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, Yue Wang

arXiv:2501.16411v2cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

VLMs show promise for embodied-agent reasoning and planning, but their physical-world understanding remains limited. PhysBench evaluates this gap, and PhysAgent improves physical reasoning by 18.4% on GPT-4o.

  • Problem

    VLMs can reason and plan for embodied agents but have a documented gap in physical-world understanding, which matters for safe and effective interaction with objects and environments.

    Source: §1 · p. 1

  • Method

    The paper introduces PhysBench, a 10,002-entry benchmark spanning four physical-understanding domains, and PhysAgent, which combines vision foundation models with a physics knowledge memory.

    Source: §1 · p. 2, §4.1 · p. 9

  • Results

    Experiments on 75 models found significant physical-world understanding gaps, particularly in open-source models; PhysAgent improved physical reasoning by 18.4% on GPT-4o.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The paper demonstrates the utility of PhysBench and PhysAgent in robotic tasks, supporting physical-world understanding in machine intelligence.

    Source: §5 · p. 10

    The authors say the challenges are common across VLM datasets and commit to ongoing refinement of PhysBench.

    Source: §J.1 · p. 141

Abstract

from arXiv · show

Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world -- likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors. To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.

1 INTRODUCTION

PhysBench evaluates VLMs’ physical-world understanding, and PhysAgent combines VLM generalization with vision models and physics knowledge to improve that understanding and support embodied deployment.

Motivation

  • Understanding object properties and relationships, scene structure, and interaction dynamics enables embodied agents to interact effectively and operate safely in real-world settings (Gupta et al., 2021; Srivastava et al., 2021; Kill & Kim, 2020).These capabilities are described as intuitive physics and include anticipating interaction outcomes for planning and accident prevention (McCloskey et al., 1983; Carey, 2000).Source: §1 · p. 1
  • VLMs can reason and plan, but physical-world gaps can cause operational errors such as mishandling fragile objects or missing grasping affordances (Liu et al., 2024a; Wang et al., 2023c; Guo et al., 2024).The paper frames comprehensive physical understanding as necessary for effective VLM deployment in embodied applications (Liu et al., 2024a; Guo et al., 2024; Gao et al., 2024a).Source: §1 · p. 1

PhysBench

  • PhysBench contains 10,002 interleaved video-image-text entries and evaluates 75 VLMs across four domains and 19 subtasks.The five-step annotation process took 4,000 hours; the domains cover physical object properties and relationships, physical scene understanding, and physics-based dynamics.Source: §1 · p. 2, Figure 1 · p. 2
  • Most current VLMs perform poorly on physical-world understanding, especially scene understanding and physics-based dynamics, while closed-source models significantly outperform open-source ones.The authors identify insufficient physical knowledge in training data as a likely major factor; fine-tuning on physically grounded data improves performance.Source: §1 · p. 2

PhysAgent

  • 18.4%: PhysAgent improves GPT-4o’s zero-shot PhysBench performance using vision foundation models and selectively invoked physics-knowledge memory.The framework targets perceptual inaccuracies and insufficient knowledge, while retaining VLMs’ generalization and open-ended problem-solving abilities.Source: §1 · p. 2

Embodied deployment

  • PhysAgent and PhysBench support embodied deployment: experiments on MOKA use PhysBench fine-tuning and PhysAgent zero-shot inference across five manipulation tasks.Improvements on those tasks support the value of physical-world understanding for embodied agents.Source: §1 · p. 2

2 RELATED WORK

Related work identifies gaps in physical-comprehension benchmarks, existing reasoning models, and VLMs’ physical understanding. PhysBench and PhysAgent are presented as broader approaches to evaluating and improving these capabilities.

Physical Comprehension Datasets

  • PhysBench is presented as the first comprehensive physical-world dataset, addressing gaps in vision-only, simulated-primitive, commonsense VQA, and spatial VQA benchmarks.Table 1 compares key features of PhysBench and other physical-understanding question-answering benchmarks.Source: §2 · p. 3, Table 1 · p. 3

Physical Reasoning Models

  • Physics-specialized models predict next states, while physical oracle models rely on predefined rules and often additional modules, limiting task breadth and open-ended responses.PhysAgent is described as more flexible and adaptable across a broader range of problems without these limitations.Source: §2 · p. 3

Vision-Language Models

  • Embodied VLM agents face weak physical understanding, and spatial reasoning covers only part of physical-world understanding; PhysBench and PhysAgent aim to evaluate and improve it.The passage links stronger physical understanding to more reliable robotic control and notes spatial VLM findings of limited 3D reasoning due to insufficient data (Bonnen et al., 2024).Source: §2 · p. 3

3 PHYSBENCH

PhysBench evaluates physical-world understanding across four domains using varied visual and textual data. Tests of 75 VLMs reveal substantial performance gaps, with analyses pointing to limited physical knowledge and perceptual errors as important obstacles.

3.1 OVERVIEW OF PHYSBENCH

  • PhysBench covers physical object properties, object relationships, scene understanding, and physics-based dynamics in a test set of 10,002 entries across 19 subclasses.It is a mixed text-image-video multiple-choice benchmark; it also includes 200 validation entries and 89,998 entries for further research.Source: §3.1 · p. 4

3.2 DATASET COLLECTION PROCESS

  • Data collection combines web searches, simulations, and real-world captures, followed by human-checked captioning, question design, categorization, and quality review.Graduate students in STEM fields manually annotate questions, and repeated cleaning and validation check physical relevance, input use, answer accuracy, and categorization.Source: §3.2 · p. 5

3.3 CAN VLMS UNDERSTAND THE PHYSICAL WORLD

  • 49.49% accuracy was GPT-4o’s best-model result, while most of the 75 evaluated VLMs averaged about 40%, well below human-level physical understanding.The primary evaluation results are reported in Table 3.Source: §3.3 · p. 5, Table 3 · p. 6
  • Physical scene understanding and physics-based dynamics retain considerable room for improvement across the evaluated models.Figure 4(b) visualizes performance across PhysBench’s 19 subtasks.Source: Figure 4 · p. 6, §3.3 · p. 7
  • GPT-4 surpassed the best open-source model, LLaVA-interleave, by 20.7%; GPT-series and Gemini-1.5 models generally outperformed open-source models, while Claude showed no clear advantage.Figure 5(b) compares overall performance for eight representative open-source general VLMs.Source: Figure 5 · p. 7, §3.3 · p. 7

3.4 WHY DO VLMS STRUGGLE WITH PHYSICAL WORLD UNDERSTANDING

  • Scaling model size, training data, or frame count did not reliably improve PhysBench performance: VILA-1.5 gained 7.1% on common QA but fell 3.8% on PhysBench when scaling from 3B to 7B.Larger-data variants showed minimal improvement or declines on PhysBench, and three open-source models were insensitive to frame count or sometimes performed worse; Mantis often failed instructions with more frames, and going beyond eight frames brought no further improvement. The added training data was predominantly descriptive, though VILA-1.5’s spatial reasoning improved.Source: Figure 6 · p. 7, §3.4 · pp. 7–8
  • Perceptual errors made up 37% of GPT-4o errors, 40% of Gemini-1.5-flash errors, and 45% of Phi-3V errors; knowledge gaps accounted for 34%, 35%, and 23%, respectively.Experts classified errors in 500 sampled questions into six categories, identifying perception and knowledge gaps as the primary sources of mispredictions.Source: §3.4 · p. 8
  • Adding physical-world knowledge examples through fine-tuning or in-context learning improved VLM performance on 200 PhysBench entries, indicating some ability to transfer that knowledge.The result supports the analysis that inadequate physical-world knowledge in the original data contributed to suboptimal performance.Source: §3.4 · p. 8

4 PHYSAGENT

PhysAgent addresses VLMs’ perceptual errors and knowledge gaps with vision foundation models and physical-knowledge memory, then evaluates whether stronger physical understanding benefits embodied tasks.

4.1 HOW TO ENHANCE VLMS FOR PHYSICAL WORLD UNDERSTANDING

  • PhysAgent combines physical-knowledge memory with Depth Anything, SAM, and GroundingDINO to identify objects and locations, infer dynamics, and estimate depth or distances.It activates task-specific prompts, integrates foundation-model outputs with VLM reasoning, then uses chain-of-thought reasoning and self-verification.Source: §4.1 · p. 9, Figure 8 · p. 9
  • ContPhy underperforms its base model on three of four tasks, while CoT has minimal effect and Desp-CoT and PLR reduce performance.The paper attributes ContPhy’s declines to inflexible templates, suboptimal module invocation, and potential visual-information loss from relying on RCNN rather than directly leveraging GPT-4o.Source: §4.1 · p. 9
  • PhysAgent improves GPT-4o performance by 18.4% on PhyBench and by 49.5% in Scene, consistently outperforming prompting strategies in zero-shot evaluation.CoT has minimal impact, while Desp-CoT and PLR reduce performance; ContPhy underperforms GPT-4o on three of four tasks.Source: §4.1 · p. 9, Figure 9 · p. 10

4.2 CAN PHYSICAL WORLD UNDERSTANDING HELP IN EMBODIED APPLICATIONS

  • Fine-tuning MOKA’s VLM with PhysBench data improves performance, while PhysAgent provides stable zero-shot gains across all five tasks, especially the force task.The fine-tuning results indicate that a PhysBench subset can serve as demonstration data for open-world robotics tasks.Source: Figure 9 · p. 10, §4.2 · p. 10
  • The embodied evaluation uses GPT-4o in MOKA with a MuJoCo simulation and a 7-DoF Franka arm across five manipulation tasks requiring spatial and object-property understanding.MOKA converts VLM-generated affordance key points and waypoints into executable motions; PhysAgent assists zero-shot, or the VLM is fine-tuned with PhysBench.Source: §4.2 · p. 10, Figure 10 · p. 10

5 CONCLUSION · F More Experiments Results

PhysBench reveals gaps in VLMs’ physical-world understanding, while PhysAgent improves GPT-4o’s physical reasoning and the dataset and approach show utility in robotic tasks.

5 CONCLUSION

  • 18.4% improvement in physical reasoning on GPT-4o is achieved by PhysAgent, a framework developed to address VLMs’ physical-understanding shortfall.Source: §5 · p. 10
  • Experiments on 75 models reveal significant physical-world understanding gaps, particularly in open-source models, attributed to inadequate training data.Source: §5 · p. 10
  • PhysBench is introduced as a benchmark for assessing Vision-Language Models’ understanding of the physical world.Source: §5 · p. 10
  • The dataset and approach are demonstrated to be useful in robotic tasks.Source: §5 · p. 10
  • Appendix J discusses limitations, broader impacts, ethical considerations, and reproducibility.Source: §5 · p. 10

A DETAILED DATASET COLLECTION PROCESS … B.5 ETHICAL CONSIDERATIONS •

PhysBench’s data collection combines simulated, web-sourced, and real-world material, while its annotation protocol sets standards for accurate, diverse, deterministic questions and ongoing quality and ethical review. The process includes specialized procedures for ambiguity, visual prompts, and task-specific annotations.

A.1 SIMULATION

  • Simulation uses 679 objects and 470 HDR images, randomized layouts, varied lighting and camera conditions, and videos spanning fluid, rope, cloth, and ball dynamics.The team also saved depth, normal, and albedo outputs with configuration files, annotated object attributes and relationships, and used bounding boxes; fluid, rope, cloth, and ball videos numbered 350, 250, 200, and 200, respectively.Source: §A.1 · p. 28

A.2 WEB

  • Web collection combines topic and concept searches, about 6,000 selected Unsplash images from 57,859 downloads, roughly 500 manipulation videos, 1,356 spatial-movement QA pairs, and visual-correspondence annotations.Manipulation questions order keyframes to match instructions, with affordance points annotated in source images; nuScenes questions cover turns and straight movement, while LLM-generated physics descriptions help locate additional videos.Source: §A.2 · pp. 28–29

A.3 REAL-WORLD

  • Real-world RGBD images and videos primarily cover light, camera, and physical-dynamics tasks such as collisions, recorded with an iPhone 13 Pro Max.Source: §A.3 · p. 29

B.1 GENERAL GUIDELINES

  • Guidelines call for rigorous, diverse, challenging, deterministic, college-level English questions with images, clear categorization, and unambiguous answers among the options.Annotations are expected to meet academic standards and include all required fields; questions should reflect varied visual contexts and physics knowledge and probe model limitations.Source: §B.1 · pp. 29–30

B.2 DATA FORMAT AND STRUCTURE

  • Figure 11 presents a PhysBench question-answer pair alongside its JSON representation.Source: Figure 11 · p. 30

B.4 HANDLING AMBIGUITIES

  • Ambiguous or unclear data are flagged for detailed review and collaboratively examined in team meetings to standardize annotation.Source: §B.4 · p. 31

B.5 ETHICAL CONSIDERATIONS •

  • Ethical safeguards exclude sources that prohibit copying or redistribution, prohibit collecting private information, and require avoiding biased or harmful content while promoting fairness and inclusivity.Source: §B.5 · p. 31

B.6 DATA CONTAMINATION CONSIDERATIONS … B.9 MORE DETAILS OF THE ANNOTATION PIPELINE

PhysBench’s supplementary procedures aim to reduce contamination and improve annotation quality through input-grounded questions, structured human review, and controlled benchmark release. The pipeline combines human curation with GPT-assisted collection and question generation.

B.6 DATA CONTAMINATION CONSIDERATIONS

  • Questions were selected to require evidence from provided images or videos, rather than answers available through common knowledge, to mitigate data contamination.The goal was to test comprehension and synthesis of diverse, challenging sources.Source: §B.6 · p. 31

B.7 ANNOTATION PLATFORM

  • A GUI platform lets experts view videos and images and annotate or edit them directly, with a streamlined interface intended to improve efficiency and accuracy.The platform is shown in Figure 12.Source: §B.7 · p. 31, Figure 12 · p. 31

B.8 BENCHMARK PREPARATION AND RELEASE

  • PhysBench-test contains 10,002 entries across 19 subclasses, PhysBench-val contains 200 entries, and 89,998 additional entries are provided for research.Remaining answer labels are withheld to prevent leakage, with online evaluation planned; reported scores use the full test dataset, including the validation split.Source: §B.8 · pp. 31–32
  • The validation split is sampled to match the full dataset’s sub-task and ability-type distributions, then filled from source datasets in proportion to their representation.Source: §B.8 · p. 32

B.9 MORE DETAILS OF THE ANNOTATION PIPELINE

  • Graduate students in STEM fields completed manual annotation and review in three groups of six, using a GUI or JSON editing across collection, annotation, and quality-check stages.The workflow is depicted in Figure 13.Source: §B.9 · p. 32, Figure 13 · p. 33
  • Video and image sources include web searches, simulations, and real-world recordings; annotators trim relevant segments and describe observed physical details.GPT-guided searches and predefined scenarios helped expand collection, while simulation scripts generated content that could be simulated.Source: §B.9 · p. 32
  • GPT generated candidate questions from video annotations because direct video prompting often produced explanations instead of question-answer pairs, and human annotators curated and verified every final question.The authors also cite lower token use and caption availability for subsequent research as reasons for the approach.Source: §B.9 · p. 32
  • The five-step annotation process took 4,000 hours and included categorizing tasks in a specified JSON format and two-stage human checks for relevance, input grounding, categorization, accuracy, and consistency.Source: §B.9 · p. 32

C DETAILED TASK DESCRIPTION … D.1 GLOBAL STATICS

PhysBench extends physical-world understanding beyond 3D geometry to object properties and relationships, environmental conditions, and physical dynamics. Its task taxonomy spans four domains, and the dataset includes both image and video examples with varied question and media characteristics.

C DETAILED TASK DESCRIPTION

  • PhysBench targets physical understanding beyond 3D geometry, including object properties, physical relationships, environmental factors, and the mechanisms of physical dynamics.The authors frame this benchmark as a step toward spatial intelligence.Source: C DETAILED TASK DESCRIPTION · p. 33

C.1 ABILITY DESCRIPTION

  • The four task domains assess recognition, comparison, prediction, judgment, reasoning, and perception across properties, relationships, scenes, and physics-based dynamics.Physical relationships distinguish static from dynamic relations; scene tasks address environmental changes, and dynamics tasks concern physical phenomena.Source: §C.1 · p. 33

C.2 PHYSICAL OBJECT PROPERTY SUB-TASK

  • Physical-property tasks cover number, mass, color, and attributes such as rigidity, fluidity, hardness, malleability, elasticity, smoothness, and sharpness.Counting can be conditioned on attributes, such as how many objects of a certain color are outside a plate or are not blurred.Source: §C.2 · p. 33

C.3 PHYSICAL OBJECT RELATIONSHIPS SUB-TASK

  • Physical-object relationship tasks cover size, location, depth, distance, and movement, including position, camera-relative depth, speed, and acceleration.Tasks include object localization, distance comparison or estimation, and analysis of movement direction and changes in speed or acceleration.Source: §C.3 · pp. 33–34

C.4 PHYSICAL SCENE UNDERSTANDING SUB-TASK

  • Physical-scene tasks cover temperature changes, camera position and effects, gas conditions, and lighting color, position, intensity, and source type.Source: §C.4 · p. 35

C.5 PHYSICS-BASED DYNAMICS SUB-TASK

  • Physics-based dynamics tasks include collisions, throwing and falling, deformable-object manipulation, fluids, chemistry, and other physical phenomena.Questions have four multiple-choice options with one correct answer; the visual prompt uses a 10px red circle.Source: §C.5 · p. 35

D.1 GLOBAL STATICS

  • Questions average 16.53 words and reach 48 words; options average 4.36 words and reach 20 words.Figure 14 and Figure 15 show the respective word-count distributions; options refer to text following the choice label.Source: §D.1 · p. 36, Figure 14 · p. 36, Figure 15 · p. 36
  • Most images and videos have 1024–2048 resolution, while video frame counts cluster at 30–45 and 60+ frames.This resolution range covers 79.7% of images and 98.6% of videos; 1.3% of videos are below 1024. The two frame-count ranges account for 59.4% and 35.9%, respectively. Image resolutions range from 183 to 7201, and video resolutions from 370 to 1920.Source: §D.1 · p. 36, Figure 16 · p. 37

D.2 WORD STATICS AND WORD CLOUD … E.6 PROMPT STRATEGIES

This section documents the benchmark’s data and assets, including physical-world vocabulary, data-generation prompts, and annotation and filtering. It also describes evaluation setups, model groups, prompt formats, and prompting strategies.

D.2 WORD STATICS AND WORD CLOUD

  • Training data most often uses generic description terms, while physical terms such as “direction” are rarer; “collides” and “camera” appear in PLLaVA data without addressing mechanisms.The authors suggest this may help explain PLLaVA-13B’s lack of substantial improvement. PhysBench word statistics also highlight physical-perception terms such as “direction,” “camera,” “phenomenon,” “effects,” “relationship,” and “light.”Source: §D.2 · pp. 37, 39, Figure 21 · p. 39

E.1 PROMPT FOR LLMS

  • Case-generation prompts request physics-focused JSON video descriptions and multiple-choice questions, while captions cover phenomena, object properties, and spatial information.VQA templates target specific relationships, and generated examples undergo a five-step manual annotation process to ensure quality.Source: §E.1 · pp. 39–40

E.2 3D ASSETS

  • The authors annotated attributes for 678 selected 3D objects, adjusted their sizes and positions to avoid overlap, and used them in simulations.The asset overview reports 470 HDR environments in total.Source: §E.2 · p. 41, Figure 22 · p. 41

E.3 MODEL HYPERPARAMETERS

  • Models were grouped as image, video, or general VLMs; image and video models were tested on a single image or video, while general VLMs used the full test split.For image VLMs, videos were represented by eight concatenated frames by default; most open-source models used float16, with bfloat16 for models requiring it.Source: §E.3 · p. 41

E.3.1 IMAGE VLM

  • The image-VLM evaluation covered varied model designs, including BLIP-2, InstructBLIP, LLaVA, Qwen-VL-Chat, and LLaVA-NeXT.Images for Claude-3-sonnet and Claude-3.5-sonnet were resized to 128×128 for cost reasons; GPT-4o extracted answers when those models failed to follow answer-format instructions.Source: §E.3.1 · pp. 41, 42, 44

E.3.2 VIDEO VLM

  • The video-VLM evaluation included Chat-Univi, Video-LLaVA, and PLLaVA, but excluded VideoChatGPT, Video-LLaMA, and VideoChat2 from metric reporting because they did not answer questions reliably.PLLaVA image inputs were duplicated to match its configured number of frames.Source: §E.3.2 · p. 44

E.3.3 GENERAL VLM

  • The general-VLM group included LLaVA-NeXT-Interleave and VILA-1.5 models, which support multi-image and video inputs; LLaVA-NeXT-Interleave was trained on M4-Instruct.The passage also begins describing NVILA, but its description is incomplete in the supplied text.Source: §E.3.3 · p. 44
  • NVILA uses a scale-then-compress approach and reports 4.5× lower training costs, 3.4× lower fine-tuning memory, and up to 2.8× lower latency.The approach increases spatial and temporal resolutions before compressing visual tokens to process high-resolution images and long videos efficiently.Source: §E.3.3 · p. 46
  • The tested general VLMs included Phi-3V-128k (Abdin et al., 2024), Phi-3.5V (AzureML, 2024), mPLUG-Owl3 variants (Ye et al., 2024), and InternVL 2.0/2.5 (Wang et al., 2024e; Gao et al., 2024b).The descriptions note long-context, multi-image, and video capabilities among these models; GPT images were resized to 512×512, and testing was conducted around mid-August 2024.Source: §E.3.3 · pp. 46–47

E.4 PROMPT FOR VLM TEST

  • During testing, questions required a direct answer with one option letter (A, B, C, or D); image VLM video prompts described equally spaced frames sampled from beginning to end (Liu et al., 2024c; Zhang et al., 2024a).Source: §E.4 · pp. 47–48

E.5 REFERENCE DATASETS SUMMARY

  • The dataset was entirely human-annotated and underwent two rounds of filtering and screening.The appendix points to Table 13 for the data sources and their uses.Source: §E.5 · p. 48

E.6 PROMPT STRATEGIES

  • CoT, Desp-CoT, and PLR prompts were used to compare reasoning strategies, with GPT-4o-mini extracting final answers (Kojima et al., 2023; Wu et al., 2023d; Lu et al., 2024b).CoT asks models to think step by step, Desp-CoT provides an image or video description, and PLR replaces the visual input with a description.Source: §E.6 · p. 48

E.7 VIPERGPT IMPLEMENTATION … F.6 CORRELATION MAP

The appendices document implementation choices and evaluation protocols, visual-prompt sensitivity, expanded benchmark reporting, embodied-task validation, and cross-benchmark correlation analysis.

E.7 VIPERGPT IMPLEMENTATION; E.8 HUMAN PERFORMANCE

  • The ViperGPT-based ContPhy implementation combines perception, simulation, program parsing, and execution, while human performance was measured across 12 qualified STEM graduate students.Because task diversity makes dedicated pretrained modules impractical, GPT-4o answered remaining questions directly; each human question was assigned to all annotators and scores were averaged.Source: §E.7 · p. 49, §E.8 · p. 50

F.1 GROUNDINGDINO CONFIGURATION

  • GroundingDINO filters boxes by box confidence and phrases by text relevance, and predictions count as successful only when output and input phrase sets match exactly.The evaluation used 1,000 samples; detailed results are reported in Table 14.Source: §F.1 · p. 50, Table 14 · p. 53

F.2 EFFECT OF VISUAL PROMPTING

  • Visual prompting method (a) slightly outperformed (b), while methods (c) and (d) were near random; a 30-pixel circle radius was selected for subsequent annotations.Most models showed little sensitivity to circle size except LLaVA-1.5-7b; larger sizes generally helped LLaVA-1.5, while Phi-3V and VILA-1.5 performed best at 30 pixels.Source: Figure 23 · p. 50, §F.2 · p. 51, Figure 24 · p. 51

F.3 MORE PHYSBENCH RESULTS

  • Table 17 reports results for 39 models across eight ability categories, combining physical scene understanding and physics-based dynamics ability classifications for clarity.Tables 18–21 provide results by physical object property, object relationships, physical scene understanding, and physics-based dynamics.Source: §F.3 · p. 51, Table 17 · p. 54, Table 18 · p. 55

F.4 PHYSBENCH-VAL RESULTS

  • PhysBench-val results cover 39 VLMs on the full test dataset, including interleaved data, with sequential-frame and merged-frame video input formats distinguished.Table 22 presents the evaluation results.Source: Table 22 · p. 59

F.5 EMBODIED TASKS DETAILED DESCRIPTION

  • The embodied evaluation used a MuJoCo Franka Panda setup with one top-down RGBD camera, testing affordance, force, color, number, and tool tasks over 10 trials each.MOKA converted affordance reasoning into visual question-answering using mark-based prompts, GroundedSAM, and farthest point sampling; the tasks were fundamental and executed without hierarchical action decomposition. The force simulation modeled objects as rigid bodies, and the VLM received approximate object dimensions to choose gripping force.Source: §F.5 · pp. 60–61, Figure 25 · p. 60

F.6 CORRELATION MAP

  • The correlation map was constructed using Pearson correlation coefficients from 10 models’ performance across 15 other vision-language benchmarks.Figure 27 shows the detailed map, while Figure 4(a) illustrates relationships between PhysBench’s four major categories and other tasks.Source: §F.6 · pp. 61–62, Table 23 · p. 61, Figure 27 · p. 62

F.7 PERFORMANCE ON RELATED BENCHMARKS

PhysBench fine-tuning and zero-shot PhysAgent improve VLM performance on three related physical-world benchmarks, especially Physion++.

Setup

  • The three related benchmarks assess specific aspects of physical-world perception, while PhysBench offers broader evaluation; preprocessing adapted the existing datasets for VLMs.Source: §F.7 · p. 62
  • Evaluation used EmbSpatial, converted 200 examples per ContPhy category to multiple choice, and adapted Physion++ prompts into 250 VQA pairs.The Physion++ prompts included contextual hints, and its train and test sets were combined because the test set was small.Source: §F.7 · pp. 62–63

Results

  • 19.50% and 9.6% improvements were observed on Physion++ with PhysBench fine-tuning and zero-shot PhysAgent, respectively; fine-tuning achieved the largest gains across benchmarks.The reported results are accuracies; parameter settings and configurations matched those in the main text.Source: Table 24 · p. 63, §F.7 · p. 63

G MORE RELATED WORKS · H MORE EXAMPLES

The related-work section argues that existing vision-language, video, interleaved-input, science, and 3D benchmarks do not fully evaluate real-world physical understanding, and positions PhysBench as a broader assessment. The examples appendix clarifies that its questions use multiple choice and that example figures omit the options.

G MORE RELATED WORKS

  • Existing VLM benchmarks cover tasks such as image question answering, long-context inputs, and spatial relations, but often assess image-content responses rather than physical-world perception.The section argues that this leaves complex spatial action planning and spatial intelligence insufficiently evaluated; Table 25 compares PhysBench with other vision-language benchmarks.Source: G MORE RELATED WORKS · p. 63, Table 25 · p. 64
  • Video benchmarks span temporal perception, action understanding, reasoning, captioning, question-answering, and long-video comprehension, but do not address true physical perception or video representation mechanisms.The authors also report that video VLMs often describe videos instead of following instructions and directly answering questions.Source: G MORE RELATED WORKS · pp. 63–64
  • Interleaved benchmarks such as SEED-Bench (Li et al., 2023a; 2024a), BLINK (Fu et al., 2024), and NLVR2 (Suhr et al., 2018) focus on image content, whereas PhysBench interweaves video, image, and text.Table 25 provides a detailed comparison with other vision-language benchmarks.Source: G MORE RELATED WORKS · p. 64, Table 25 · p. 64
  • Physics exam benchmarks test physics knowledge or mathematical reasoning, but the authors characterize them as insufficient for real-world physical comprehension and describe PhysBench as the first systematic, comprehensive QA benchmark for it.The cited examples include SciQ (Welbl et al., 2017), JEEBench (Arora et al., 2023), and SciBench (Wang et al., 2024f).Source: G MORE RELATED WORKS · p. 64
  • 3D VQA works such as SQA3D (Ma et al., 2023) primarily emphasize geometric relationships, while PhysBench spans four physical-understanding categories and uses interleaved images to capture dynamics such as viewpoint rotations.Table 26 compares PhysBench with physical-understanding benchmarks that are not in question-answering format.Source: Table 26 · p. 64, G MORE RELATED WORKS · p. 65
  • Robotic manipulation approaches either directly generate actions, requiring extensive training and targeted fine-tuning, or use VLM-generated affordances, which generalize better but have weaker physical-world perception.The section gives OpenVLA’s training on 64×A100 GPUs as an example of the training demands of direct action generation.Source: G MORE RELATED WORKS · p. 65

H MORE EXAMPLES

  • The example figures mark the correct answer in red, use multiple-choice questions to reduce evaluation difficulty, and omit answer options from main-text figures to save space.The appendix reiterates that red indicates the correct answer.Source: H MORE EXAMPLES · p. 65

H.1 PHYSICAL OBJECT PROPERTY SUB-TASK … I.1 DETAILED STATICS

The appendix illustrates PhysBench examples across object properties, relationships, scene conditions, and physics-based dynamics, then analyzes common VLM error types using selected samples from three models.

H.1 PHYSICAL OBJECT PROPERTY SUB-TASK

  • Examples cover object-color identification, mass identification and comparison, number identification, and attributes including sharpness, brittleness, stiffness, elasticity, malleability, and softness.The mass examples include identification and comparison ability types; the number examples are all identification.Source: Figure 28 · p. 65, Figure 29 · p. 66, Figure 30 · p. 67

H.4 PHYSICS-BASED DYNAMICS SUB-TASK

  • Relationship examples address motion, depth, distance, size, and location, with ability types including static and dynamic.The motion, distance, size, and location figures include static and dynamic examples; the depth example is static.Source: Figure 32 · p. 69, Figure 33 · p. 69, Figure 34 · p. 70
  • Scene examples cover environmental lighting, air, temperature, and viewpoint conditions, spanning perception, reasoning, and judgment ability types.Lighting examples include perception, reasoning, and judgment; air and temperature examples are perception, while viewpoint examples include perception and reasoning.Source: Figure 37 · p. 72, Figure 38 · p. 73, Figure 39 · p. 73
  • Dynamics examples span manipulation, collision, chemistry, throwing, fluid, and other cases, with ability types including judgment, perception, reasoning, and prediction.Collision and throwing examples involve prediction; chemistry involves perception, while manipulation, fluid, and other examples cover multiple ability types.Source: Figure 42 · p. 75, Figure 43 · p. 76, Figure 44 · p. 76

I.1 DETAILED STATICS

  • The error study selected 500 samples per model for GPT-4V, Gemini-1.5-flash, and Phi-3V, presenting 60 examples in Table 27 because of space limits.It classifies errors into five categories and omits less frequent types.Source: §I.1 · p. 79
  • Perception errors are the most common, arising when VLMs fail to recognize, classify, or detect image content; other categories include reasoning, knowledge, refusal, and instruction-following errors.Reasoning errors occur despite correct recognition of text and visual content; instruction-following errors can involve missing specified conditions.Source: §I.1 · p. 79

I.2 MAIN REASON ANALYSIS · I.3 CASE STUDY · J DISCUSSION AND STATEMENT

The analysis attributes VLM errors to perception, reasoning, and missing domain knowledge, while the case-study figures sample physical tasks and illustrate varied outcomes. Other error types are uncommon but remain significant.

I.2 MAIN REASON ANALYSIS

  • Perceptual errors include basic visual misidentification and failures to interpret specialized conditions; GPT-4V can also favor text over vision, as when it treated a plate as background and misestimated depth (Cui et al., 2023).The paper argues for balancing the model’s interpretation of textual and visual inputs.Source: §I.2 · p. 79
  • Reasoning errors occur when models misinterpret causal relationships, such as overlooking another object in a friction problem or wrongly explaining wrapped paper’s lower temperature by the flame’s temperature.In both examples, the model correctly interpreted the inputs but reached an incorrect conclusion.Source: §I.2 · p. 79
  • A lack of relevant knowledge can lead to incorrect explanations, such as attributing light bending in water to refraction rather than recognizing total internal reflection.The example is presented as a failure to understand total internal reflection in water.Source: §I.2 · p. 79
  • Textual-understanding, refusal, annotation, and answer-extraction errors make up a small proportion of errors but should not be overlooked.Source: §I.2 · p. 79

I.3 CASE STUDY

  • The case-study catalog records perception errors, reasoning errors, and successes across examples spanning scene, relationship, property, and dynamics tasks.The listed cases include varying outcome patterns, including instances of success alongside errors.Source: §I.3 · p. 80
  • Case-study figures sample physical-scene light and viewpoint, collision and throwing dynamics, chemistry, and object-property attribute tasks.Source: Figure 48 · p. 81, Figure 49 · p. 82, Figure 50 · p. 83

J.1 LIMITATION … J.4 REPRODUCIBILITY STATEMENT

The authors acknowledge limits in benchmark coverage, data quality, and automated evaluation, while describing bias and misuse risks alongside commitments to ethical, licensed, open, and reproducible research.

J.2 BOARDER IMPACT

  • The authors say portions of the data come from existing datasets, pledge to address legitimate image-copyright concerns, and report compliance with model and data licenses.They also commit to publicly releasing the dataset and scripts to reduce emissions from regenerating similar datasets.Source: §J.1 · p. 141, §J.2 · p. 141

J.1 LIMITATION

  • PhysBench does not cover every aspect of the physical environment, and despite extensive collection and review, minor dataset issues may remain.The authors report spending thousands of hours on data collection, organization, and annotation.Source: §J.1 · p. 141
  • GPT-4o-mini answer extraction may hallucinate, and multiple-choice evaluation simplifies difficult open-ended assessment, though GPT-4o still scores below 50% accuracy.The authors identify evaluation of reasoning quality and exploration of more complex evaluation formats as challenges and future work.Source: §J.1 · p. 141
  • The authors state that the potential benefits outweigh risks and commit to refining PhysBench to advance physical-world understanding and embodied AI.They describe PhysBench and PhysAgent as the first effort to benchmark and enhance VLMs for physical-world understanding.Source: §J.1 · p. 141

J.2 BOARDER IMPACT

  • PhysAgent may inherit biases from its vision and language foundation models, while misuse could enable fabricated content, misinformation, or privacy infringements.The authors call for careful evaluation and mitigation of bias, ethical guidelines, and ongoing oversight.Source: §J.2 · p. 141

J.3 ETHICS STATEMENT

  • The study reports no ethical concerns, uses publicly available pre-existing models without subjective evaluations, and states compliance with the ICLR Code of Ethics.Source: §J.3 · p. 142

J.4 REPRODUCIBILITY STATEMENT

  • The authors follow standard benchmark or model testing baselines and provide implementation details in Appendices E and F.Source: §J.4 · p. 142
  • The authors commit to open-access release of data and code with reproduction instructions and report compliance with ICLR reproducibility requirements.Source: §J.4 · p. 142

K LATEST RESULTS

PhysBench shows weaker, less consistent scaling with model size than standard VQA benchmarks, while the paper adds evaluations of 36 newer VLMs. Larger models bring notable gains, but performance can stagnate or decline across some size ranges.

K LATEST RESULTS

Related papers

Loading 2501.16411v2…