PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding

Wei Chow; Jiageng Mao; Boyi Li; Daniel Seita; Vitor Guizilini; Yue Wang

Source-linked AI summary

PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding

Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, Yue Wang

arXiv:2501.16411v2Jan 27, 2025cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

VLMs have strong reasoning and planning abilities but limited physical-world understanding, creating a gap for embodied AI. The paper introduces PhysBench to evaluate this capability and PhysAgent to enhance it, reporting broad model weaknesses and an 18.4% GPT-4o improvement.

Problem
VLMs’ limited physical-world understanding remains a challenge for embodied agents that must interact with objects, navigate safely, and plan around physical dynamics.
Method
The paper introduces PhysBench, a 10,002-entry benchmark evaluated on 75 VLMs, and PhysAgent, which combines vision foundation models with physics knowledge memory.
Results
Most VLMs perform poorly on physical-world understanding, while PhysAgent improves GPT-4o’s zero-shot PhysBench performance by 18.4%.
Takeaways & Limitations
The experiments show that improving VLMs’ physical-world understanding can facilitate embodied-agent deployment, including robotic tasks.
Takeaways & Limitations
The paper states that its limitations, broader impacts, ethical considerations, and reproducibility are discussed in Appendix J.

Abstract

from arXiv · show

Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world -- likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors. To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.

1 INTRODUCTION

PhysBench targets a physical-world understanding gap in VLMs that limits embodied agents’ safe interaction, navigation, and planning. It benchmarks this gap and introduces PhysAgent to improve physical understanding through visual models and physical knowledge.

Motivation: Embodied agents need physical object properties, object relationships, scene structure, and interaction dynamics for accurate, efficient, and safe real-world operation.These capabilities support object manipulation, cluttered-environment operation, navigation, planning, and accident prevention.
Motivation: VLMs offer strong reasoning and task-planning capabilities but remain insufficient for generalist embodied agents because of weak physical-world understanding.Reported operational errors include mishandling fragile objects and failing to recognize appropriate grasping affordances.
PhysBench: PhysBench contains 10,002 interleaved video-image-text entries and evaluates 75 VLMs across four physical-world domains and 19 subtasks.The domains are physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics.
Findings: Experiments find that most current VLMs perform poorly on physical-world understanding, especially physical scene understanding and physics-based dynamics.Closed-source models significantly outperform open-source models, and training data likely contributes to the performance gap because it often lacks necessary physical knowledge.
PhysAgent: PhysAgent combines vision foundation models with a physics knowledge memory to address perceptual inaccuracies and insufficient physical knowledge.The framework supports tasks such as depth estimation and numerical distance calculation while retaining VLM generalization and open-ended problem-solving abilities.
Results: PhysAgent improves GPT-4o’s zero-shot PhysBench performance by 18.4%.The framework is designed to enhance physical understanding while supporting embodied-agent deployment.

2 RELATED WORK

Prior physical-comprehension benchmarks often isolate simple primitives, simulated scenarios, commonsense reasoning, or geometric relationships. PhysBench is positioned as a broader benchmark spanning diverse physical-world scenarios and tasks.

Physical Comprehension Datasets: Earlier physical-comprehension benchmarks primarily targeted vision-only models or simple primitives in limited simulated scenarios.Examples include spheres, cubes, and rigid-object collision events.
Physical Comprehension Datasets: Existing physical-knowledge VQA datasets mainly assess commonsense reasoning, while spatial VQA benchmarks emphasize only geometric relationships in 3D scenes.The latter represent only part of physical-world understanding.
PhysBench: PhysBench is presented as the first comprehensive dataset covering a wide variety of physical-world scenarios and tasks not covered by previous benchmarks.Its comparison with prior physical-understanding question-answering benchmarks is summarized in Table 1.
Physical Reasoning Models: Physics-specialized models are typically limited to predicting the next state and are not applicable to other tasks.They form one of two broad categories of physical reasoning models discussed in the paper.
Physical Reasoning Models: Physical oracle models support only a narrow range of tasks because they rely on predefined rules.This contrasts with approaches intended to handle broader physical-world tasks and natural-language queries.

3 PHYSBENCH

PhysBench evaluates physical-world understanding across diverse multimodal tasks and reveals substantial shortcomings in current VLMs. Its analyses indicate that performance does not reliably scale with model size, training data, or frames, while perceptual and knowledge gaps account for many errors.

3.1 OVERVIEW OF PHYSBENCH: PhysBench evaluates VLMs across four physical-world domains using 10,002 interleaved video-image-text entries.The domains cover object properties, object relationships, scene understanding, and physics-based dynamics.
3.1 OVERVIEW OF PHYSBENCH: The benchmark uses multiple images and videos to represent properties and environmental factors that single images cannot capture effectively.Examples include elasticity, mass, density, temperature, humidity, light source, and viewpoint.
3.3 CAN VLMS UNDERSTAND THE PHYSICAL WORLD: 75 evaluated VLMs achieve approximately 40% average accuracy, while GPT-4o reaches only 49.49%, remaining below human-level understanding.Performance gaps are especially notable for physical scene understanding and physics-based dynamics.
3.3 CAN VLMS UNDERSTAND THE PHYSICAL WORLD: Closed-source models generally outperform open-source models, with GPT-4 exceeding LLaVA-interleave by 20.7%.The reported comparison concerns the best open-source model in the evaluated setting.
3.4 WHY DO VLMS STRUGGLE WITH PHYSICAL WORLD UNDERSTANDING: Physical-world understanding does not clearly improve with larger models, more training data, or more frames.For example, VILA-1.5 improves 7.1% on common QA tasks from 3B to 7B parameters but decreases 3.8% on PhysBench; added data and frames likewise provide limited or negative gains.
3.4 WHY DO VLMS STRUGGLE WITH PHYSICAL WORLD UNDERSTANDING: Error analysis identifies perceptual inaccuracies and insufficient physical knowledge as primary contributors to VLM mistakes.The study also tests whether supplementary physical-world examples improve performance through fine-tuning or in-context learning.

4 PHYSAGENT

PhysAgent addresses VLM physical-understanding errors by combining task-specific physical knowledge with vision foundation models in a three-step reasoning process. It improves zero-shot physical reasoning and supports downstream embodied-agent tasks.

4 PHYSAGENT: PhysAgent integrates a physical knowledge memory with vision foundation models to address perceptual inaccuracies and insufficient physical knowledge.Its vision models enhance object, location, depth, and spatial perception, while the memory supplies physical rules and prior knowledge.
4 PHYSAGENT: PhysAgent activates task-specific prompts, integrates foundation models, and performs reasoning for each question.For example, light-related questions can retrieve knowledge about light-source movement and shadow direction.
4.1 HOW TO ENHANCE VLMS FOR PHYSICAL WORLD UNDERSTANDING: Prompting methods are unstable, while ContPhy underperforms GPT-4o in three of four tasks because of inflexible templates and information loss.CoT has minimal impact, whereas Desp-CoT and pure language reasoning decline in performance.
4.1 HOW TO ENHANCE VLMS FOR PHYSICAL WORLD UNDERSTANDING: PhysAgent consistently improves zero-shot performance, including a 49.5% improvement for GPT-4o in Scene.The method outperforms the compared prompting strategies across the reported evaluations.
4.2 PHYSICAL WORLD UNDERSTANDING IN EMBODIED AGENTS: PhysAgent yields stable zero-shot gains across five embodied tasks, with especially significant progress on the force task.MOKA uses VLM-generated key points and waypoints to convert affordance representations into executable robotic-arm motions.

5 CONCLUSION

The paper concludes that PhysBench exposes substantial physical-world-understanding gaps in VLMs and that PhysAgent improves physical reasoning while supporting robotic tasks. It identifies inadequate training data as a contributor to these gaps.

5 CONCLUSION: Experiments on 75 models identify significant physical-world-understanding gaps, particularly in open-source models, associated with inadequate training data.The conclusion frames PhysBench as a benchmark for evaluating these capabilities.
5 CONCLUSION: PhysAgent improves GPT-4o physical reasoning by 18.4%.The paper also reports utility for robotic tasks.
5 CONCLUSION: The paper demonstrates that its dataset and approach are useful in robotic tasks.This supports advancing physical-world understanding in machine intelligence.
5 CONCLUSION: The paper directs detailed limitations, broader impacts, ethical considerations, and reproducibility discussions to Appendix J.This conclusion section does not specify those details.

A DETAILED DATASET COLLECTION PROCESS

The dataset collection process combines simulation, web data, real-world recordings, and curated source datasets to build diverse physical-understanding examples. It emphasizes accurate annotation, broad coverage, varied difficulty, and rigorous review.

Data Collection: Simulation data use Blender, 679 objects, 470 HDR images, randomized configurations, and saved depth, normal, albedo, and scene-property files.These assets support rendered images and videos with corresponding physical and visual metadata.
Data Collection: The process adds lighting and camera variation, including randomized light properties, camera positions, and shooting angles.These variations target physical effects such as shadows and viewpoint changes.
Data Collection: Physics videos cover fluid, rope, cloth, and ball dynamics, while web searches retrieve physics-related demonstrations guided by predefined topics and LLM-generated descriptions.The collected web data is filtered, cleaned, and annotated.
Data Collection: Additional sources include approximately 500 manipulation videos, 1,356 nuScenes spatial-movement QA pairs, Unsplash images, visual prompts, correspondence data, and real-world RGBD recordings.These sources broaden manipulation, navigation, visual, and physical-dynamics coverage.
General Guidelines: PhysBench is designed to evaluate physical perception across physical properties, spatial relationships, and dynamic phenomena rather than descriptive VQA alone.Its guidelines require multiple tasks, diverse contexts, varied challenge levels, deterministic evaluation, and image-based questions.
Annotation and Quality Control: Annotations undergo peer review, secondary review, random audits, training, feedback, and quality checks, with examples and standardized JSON fields supporting consistency.The annotation platform and pipeline organize collection, classification, file formatting, and validation.

B.6 DATA CONTAMINATION CONSIDERATIONS

The benchmark-release and contamination procedures prioritize questions that require evidence from supplied visual data rather than easily accessible common knowledge. The annotation pipeline uses structured human collection, organization, and review stages.

B.6 DATA CONTAMINATION CONSIDERATIONS: Data-contamination risk is mitigated by selecting questions that extend beyond straightforward queries with easily accessible answers.Tasks should require models to use and synthesize the provided videos or images.
B.8 BENCHMARK PREPARATION AND RELEASE: PhysBench-test contains 10,002 entries across 19 subclasses, with 200 validation entries and 89,998 additional entries for research.Answer labels are withheld for the remaining data and an online evaluation platform is maintained to prevent leakage.
B.8 BENCHMARK PREPARATION AND RELEASE: The validation split is sampled to match the full dataset’s sub-task, ability-type, and source-dataset distributions.The procedure combines random sampling across task types with source-proportional sampling.
B.9 MORE DETAILS OF THE ANNOTATION PIPELINE: The annotation pipeline divides six-person groups across video collection, intermediate processing, and quality checks, using a GUI or direct JSON editing.Human verification checks relevance, use of all inputs, non-common-sense dependence, categorization, clarity, and answer accuracy.

C DETAILED TASK DESCRIPTION

PhysBench extends physical-world evaluation beyond ordinary visual and commonsense reasoning to object properties, relationships, scenes, and dynamics. Its tasks use varied multimodal inputs and targeted multiple-choice questions to test these capabilities.

Task taxonomy: PhysBench covers four task types: physical properties, object relationships, physical scenes, and physics-based dynamics.These categories include capabilities such as recognition, comparison, prediction, judgment, reasoning, and perception.
Physical object properties: Its physical-property tasks assess attributes including mass, size, density, tension, friction, stiffness, elasticity, and plasticity.Object-property subtasks include number, mass, color, and attributes, with questions often linking quantities to specific properties.
Physical object relationships: Object-relationship tasks evaluate size, location, depth, distance, and movement between physical objects.The tasks address both absolute and relative positions, depth relative to the camera, distance estimation, and object motion.
Physical scene understanding: Physical-scene tasks cover temperature, camera, gas, and light conditions and their effects on observed environments.These tasks extend beyond geometric relationships to environmental factors such as temperature, humidity, lighting, and viewpoint.
Physics-based dynamics: Physics-based dynamics tasks include collision, throwing, manipulation, fluid, chemistry, and other physical phenomena.The benchmark targets mechanisms such as collisions, falling, fluid flow, reactions, energy transfer, and optical or electromagnetic dynamics.
Input and question design: Questions are four-option multiple-choice items with one correct answer, using images and videos when single images cannot convey relevant physical features.The dataset reports an average question length of 16.53 words and an average option length of 4.36 words.

D.2 WORD STATICS AND WORD CLOUD

The dataset’s vocabulary emphasizes physical-world perception, whereas analyzed VLM training data is dominated by captioning and descriptive language. The appendix documents both the compared training sources and the generation pipeline for diverse cases and questions.

PhysBench word statistics: PhysBench frequently uses physical-perception terms such as direction, camera, phenomenon, effects, relationship, and light.These terms appear in the dataset’s word cloud and frequency analysis.
Training-data sources: The VILA-1.5-13B analysis uses M3IT because the repository guidelines indicate that its instruction-tuning data is included there.The listed data spans captioning, reasoning, translation, and VQA datasets.
Training-data sources: PLLaVA-13B adds 783k instructional video-to-text tuning examples to LLaVA-Next-based training.These examples include conversation, classification, captioning, reasoning, and annotated question-answering data.
Training-data comparison: LLaVA-1.5-13B, VILA-1.5-13B, and PLLaVA-13B training data most frequently contains descriptive terms such as description, phrase, summary, and region.Terms such as direction appear much less frequently, although PLLaVA-13B includes words such as collides and camera.
Case and question generation: Generated cases use video or image descriptions, physics explanations, object properties, spatial information, and multiple-choice questions with answers.The annotation process replaces examples to increase diversity, then applies a subsequent rigorous five-step process.

E.7 VIPERGPT IMPLEMENTATION

The ViperGPT-based ContPhy implementation decomposes video question answering into perception, physical simulation, program parsing, and symbolic execution. The authors find that such domain-specific modules do not scale well to PhysBench’s diverse tasks, while visual-prompt design affects model accuracy.

Oracle pipeline: The implemented ContPhy pipeline contains video perception, physical simulation, program parsing, and symbolic execution modules.GPT-4o translates questions into executable Python programs after visual and physical processing.
Oracle pipeline: Video perception detects objects and static attributes, while simulation predicts object dynamics from point clouds across scenarios.The implementation uses MASK-RCNN for detection and dynamic prediction models for simulation.
Scope limitation: Neuro-symbolic approaches are unsuitable for PhysBench’s complex and diverse tasks because they depend on pre-trained domain-specific extraction and dynamics modules.The authors note that different task types would require impractical numbers of specialized modules.
Visual prompts: Visual-prompt experiments found that methods (a) and (b) performed similarly, with method (a) slightly better for depth and attributes.Bounding-box or point-based methods (c) and (d) were close to random for attributes, likely because models could not perceive color differences.
Visual prompts: LLaVA-1.5 generally benefited from larger prompt circles, while Phi-3V and VILA-1.5 performed best with a 30-pixel radius.The authors selected a 30-pixel radius as a balance between performance and aesthetics.

F.3 MORE PHYSBENCH RESULTS

The appendix reports model evaluations across PhysBench’s ability and task dimensions, alongside embodied-agent validation using MOKA. It also describes correlation analysis with other benchmarks and the implementation details of the downstream simulation tests.

PhysBench results: Table 17 reports results for 39 VLMs across eight ability categories in PhysBench.Physical scene understanding and physics-based dynamics share ability classifications in this presentation, while their content differs.
PhysBench results: Tables 18–21 provide results for physical object properties, object relationships, physical scene understanding, and physics-based dynamics.Table 3 organizes results by task type, while the appendix tables provide category-specific evaluations.
Evaluation settings: Table 22 evaluates 39 VLMs on PhysBench-val using sequential image inputs or merged video frames after frame selection.General VLM evaluation combines video and image evaluations with interleaved data.
Embodied-agent validation: The MOKA validation uses a MuJoCo and Franka Emika Panda simulation platform to test five embodied tasks.MOKA converts affordance reasoning into visual question-answering problems using mark-based visual prompting, GroundedSAM, and farthest point sampling.
Embodied-agent validation: MOKA experiments improve VLMs either by fine-tuning with PhysBench or by using PhysAgent during inference.The evaluated tasks are described as relatively fundamental compared with complex multi-action tasks.
Benchmark correlation: The correlation analysis uses Pearson coefficients to relate PhysBench categories to 15 other vision-language benchmarks.The relationship matrix is built from results for 10 models.

F.7 PERFORMANCE ON RELATED BENCHMARKS

The paper evaluates PhysBench adaptations and PhysAgent on three related physical-world benchmarks. Fine-tuning and zero-shot PhysAgent improve performance, especially on Physion++.

PhysBench provides a more comprehensive physical-world evaluation than the three related benchmarks, which target specific perception aspects.The comparison is summarized in Table 1.
The evaluation covers EmbSpatial, ContPhy, and Physion++, with videos reformulated into natural-language prompts for VLM assessment.Physion++ prompts explicitly mark transition phases, and the combined dataset contains 250 VQA pairs.
19.50% and 9.6% improvements are observed on Physion++ with PhysBench fine-tuning and zero-shot PhysAgent, respectively.Fine-tuning achieves the most significant gains among the reported settings.

G MORE RELATED WORKS

Related benchmarks cover vision-language, video, interleaved-input, science, three-dimensional scene, and robotic-manipulation capabilities. The paper positions PhysBench as a physical-world evaluation beyond image content, commonsense reasoning, and exam-style physics.

Vision-Language and Video Benchmarks: Existing VLM and video benchmarks evaluate image question answering, chart understanding, web comprehension, temporal perception, actions, classification, reasoning, and video question answering.These benchmarks primarily assess content understanding rather than underlying physical-world mechanisms.
Interleaved Vision-Language Benchmarks: Interleaved-input benchmarks process longer multimodal inputs, but primarily assess general visual-language understanding rather than comprehensive physical-world comprehension.Examples include SEED-Bench, BLINK, and NLVR2.
Science-related Benchmarks: Science benchmarks include physics-related questions but often emphasize image analysis, mathematical reasoning, general knowledge, or text-only multistep reasoning instead of physical-world comprehension.PhysBench is described as the first systematic and comprehensive question-answering benchmark for this target.
3D Scene VQA and Robotic Manipulation: Three-dimensional scene VQA benchmarks expand multimodal scene collections and annotations, while robotic-manipulation work either directly generates actions or uses VLMs as agents.The related work connects these directions to embodied interaction without equating them with physical-world understanding.
PhysBench Positioning: PhysBench addresses existing dataset coverage gaps by evaluating object properties, object relationships, scene understanding, and physics-based dynamics.Its multiple-choice presentation reduces difficulty and facilitates evaluation.

H.1 PHYSICAL OBJECT PROPERTY SUB-TASK

The appendix illustrates PhysBench examples across physical object properties, object relationships, scene conditions, and physics-based dynamics. Examples span identification, comparison, perception, reasoning, judgment, static or dynamic relations, and prediction.

Physical Object Properties: Property examples cover color, mass, number, and material attributes including sharpness, brittleness, stiffness, elasticity, malleability, and softness.Mass examples include identification and comparison, while number examples are identification tasks.
Physical Object Relationships: Relationship examples address motion, depth, distance, size, and location using static, dynamic, or mixed ability types.Motion, distance, and location include multiple ability-type combinations, while depth is static.
Physical Scene Understanding: Scene examples cover environmental lighting, air, temperature, and viewpoint conditions with perception, reasoning, and judgment abilities.Lighting and viewpoint examples include both perception and reasoning cases.
Physics-based Dynamics: Dynamics examples cover manipulation, collision, chemistry, throwing, fluid behavior, and other phenomena requiring judgment, perception, reasoning, or prediction.Collision and throwing are prediction examples, while chemistry is perception-based.
Error-Analysis Examples: The appendix presents a case-study analysis using 500 samples per model, with 60 examples shown because of space limitations.The analysis examines GPT-4V, Gemini-1.5-flash, and Phi-3V across five error categories.

I.2 MAIN REASON ANALYSIS

The main error analysis identifies reasoning and knowledge failures alongside perceptual errors and smaller categories of residual mistakes. The examples show errors in interpreting physical mechanisms, visual structure, and task-relevant instructions.

Perceptual Errors: Perceptual errors include basic and domain-specific failures, with GPT-4V sometimes prioritizing textual information over visual inputs.One example misidentified a plate as background, producing an incorrect depth estimate.
Reasoning Errors: Reasoning errors occur when models correctly interpret text, images, and questions but fail to establish a rigorous logical chain.Examples include incorrect assumptions about friction and misattributing a wrapped paper’s temperature to the flame.
Lack of Knowledge: Knowledge errors arise when models lack relevant physical concepts, such as total internal reflection in water.The analyzed model incorrectly explained light bending in water through refraction.
Other Errors: Residual errors involving textual understanding, refusal, annotation, and answer extraction account for only a small proportion of cases.The analysis states that these categories remain significant and should not be overlooked.
Case-study Coverage: The appendix examples instantiate these categories across scene viewpoint, scene lighting, object relationships, and physics-based dynamics tasks.The figure index associates meta-tasks with perception, reasoning, and success labels.

J.1 LIMITATION

The paper identifies limits affecting PhysBench’s coverage, evaluation, dataset quality, and deployment. These include incomplete representation of the real physical environment, possible annotation issues, evaluator hallucinations, model biases, and misuse risks.

PhysBench does not encompass all aspects of the real physical environment, despite covering four major and 19 subcategories.
Minor issues may persist in a small portion of the dataset despite extensive collection, organization, annotation, repeated reviews, and quality-control processes.
GPT-4o-mini answer extraction may introduce hallucinations, while open-ended and reasoning-process evaluation remain difficult, motivating the use of multiple-choice testing.
PhysAgent may inherit biases from its vision and language foundation models, potentially producing skewed outcomes or unfair representations.
Deployment and release may enable fabricated content, misinformation, privacy infringements, and other harms, requiring ethical guidelines and ongoing oversight.

K LATEST RESULTS

The latest results extend PhysBench evaluation to additional VLMs and examine how model size relates to performance. Scaling is weaker and less consistent on PhysBench than on conventional benchmarks, especially for scene understanding, with clearer gains only at larger model sizes.

Latest model evaluation: 36 additional VLMs were evaluated in December 2024 alongside the original 39 models.The two sets are reported separately, with consolidated results available on the project page.
Model scaling: Models from 1B upward generally improve on PhysBench, but 3B-to-13B scaling yields little gain and sometimes decreases performance for VILA and PLLaVA.The reported decreases occur despite the larger model size.
Model scaling: 26B and 40B models show substantial improvements compared with previous sizes on PhysBench.These larger-scale gains contrast with the limited improvement observed between 3B and 13B.
Cross-benchmark comparison: PhysBench scales less consistently than TextVQA, MathVista, and MMMU, with performance not always positively correlated with model size.The comparison is presented across the overall benchmark and its Scene and Dynamics subcategories.
Scene scaling: Scene performance remains relatively stagnant from 5B to 20B parameters and improves notably only beyond 25B parameters.This pattern is identified as particularly evident in the PhysBench-Scene subcategory.
Evaluation breakdown: The latest evaluation reports results by task type and ability dimension, including physical object properties, object relationships, scene understanding, and physics-based dynamics.Table 28 uses task-type categories, while Table 29 uses ability dimensions; Tables 30–33 provide subtask results.