PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning

Xinrun Xu, Pi Bu, Ye Wang, Börje F. Karlsson, Ziming Wang, Tengtao Song, Qi Zhu, Jun Song, Zhiming Ding, Bo Zheng

arXiv:2508.05405v1cs.AI

TL;DR

Existing benchmarks often simplify physical interactions or assess physical reasoning through static problems, leaving interactive physical reasoning to be evaluated. DeepPHY integrates simulated environments into an interactive benchmark, and its evaluation finds that even state-of-the-art models struggle with precise, multi-step planning in dynamic environments.

  • Problem

    Existing benchmarks often simplify physical interactions or use static QA and text-based problems, rather than evaluating interactive physical reasoning in dynamic environments.

    Source: §1 · p. 2

  • Method

    DeepPHY integrates six physics-based simulation environments into an interactive benchmark for evaluating agentic VLMs’ physical reasoning.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Even state-of-the-art models struggle with precise, multi-step planning in dynamic environments.

    Source: §6 · p. 12

  • Takeaways & Limitations

    DeepPHY provides a testbed for benchmarking these limitations and facilitating development of more physically grounded AI agents.

    Source: §6 · p. 12

Abstract

from arXiv · show

Although Vision Language Models (VLMs) exhibit strong perceptual abilities and impressive visual reasoning, they struggle with attention to detail and precise action planning in complex, dynamic environments, leading to subpar performance. Real-world tasks typically require complex interactions, advanced spatial reasoning, long-term planning, and continuous strategy refinement, usually necessitating understanding the physics rules of the target scenario. However, evaluating these capabilities in real-world scenarios is often prohibitively expensive. To bridge this gap, we introduce DeepPHY, a novel benchmark framework designed to systematically evaluate VLMs' understanding and reasoning about fundamental physical principles through a series of challenging simulated environments. DeepPHY integrates multiple physical reasoning environments of varying difficulty levels and incorporates fine-grained evaluation metrics. Our evaluation finds that even state-of-the-art VLMs struggle to translate descriptive physical knowledge into precise, predictive control.

1 Introduction

Existing environments often simplify physical interactions or avoid real-world physics, leaving interactive physical reasoning difficult to evaluate. DeepPHY addresses this gap with a benchmark for agentic VLMs.

1 Introduction

  • It integrates six physics-based environments: PHYRE, I-PHYRE, Kinetix, Pooltool, Angry Birds, and Cut the Rope.Source: §1 · p. 2
  • DeepPHY is the first comprehensive benchmark suite designed to systematically evaluate interactive physical reasoning in agentic VLMs.Source: §1 · p. 2
  • A unified framework and standardized metrics make diverse simulators an accessible testbed and collect interaction data useful for training physically realistic AI agents.Source: §1 · pp. 2–3
  • The empirical study evaluates leading open- and closed-source VLMs, providing baselines and examining limitations in interaction, planning, and adaptation.Source: §1 · p. 3

2 Related Work

Prior physical-reasoning evaluations often rely on static problems or symbolic inputs, while many game-agent tasks emphasize scripted progression and discrete mechanics. DeepPHY instead evaluates physical reasoning through direct visual input and interaction in dynamic settings.

2 Related Work

  • Static QA and text-based physics problems test knowledge or deduction from fixed contexts, but omit real-time visual perception and continuous interaction.Source: §2 · p. 3
  • Simulated physical-reasoning research often uses symbolic inputs or code-generated simulator interaction, bypassing raw sensory understanding and limiting generalizability.Source: §2 · p. 3
  • Many game-agent environments rely on scripted storylines or discrete mechanics, allowing game-specific heuristics instead of requiring inference of underlying physical laws.Source: §2 · pp. 3–4
  • DeepPHY evaluates agentic VLMs directly from visual input in dynamic settings that require perception, action, and reasoning under physical principles.Source: §2 · pp. 3–4

3 DeepPHY

DeepPHY evaluates agentic VLMs’ physical understanding and sequential planning in six simulated environments. Its trial-based design tests whether agents can use failed attempts to refine their predictive models and plans.

3 DeepPHY

  • DeepPHY combines six physics-based environments to test VLMs’ understanding of physical scenes and sequential action planning.The benchmark compares environments with distinct physical challenges and observation/action-space conversions.Source: §3 · p. 4, §3.2 · p. 5, Table 2 · p. 5

3.1 Problem Formalization

  • Agents receive up to K attempts from the same initial observation, using failed-trial history to update their predictive model and improve planning.The policy evaluates candidate plans by simulating outcomes conditioned on the initial observation and past failures, then estimating success likelihood.Source: §3.1 · pp. 4–5

3.2 Environments

  • The suite spans PHYRE, I-PHYRE, Kinetix, Pooltool, Angry Birds, and Cut the Rope, covering timed obstacle removal, diverse control tasks, billiards, and puzzle interactions.PHYRE tasks require object placement to trigger physical chain reactions; I-PHYRE puzzles require timed obstacle removal.Source: §3.2 · p. 5

3.3 Observation Space

  • Annotated observations reduce object-localization demands so evaluation focuses on understanding physical dynamics and planning manipulations.The suite uses grids or labels in some environments, a top-down 2D view for Pooltool, and raw screenshots for Angry Birds.Source: §3.3 · p. 6

3.4 Action Space

  • DeepPHY converts continuous or complex controls into discrete, structured outputs to make interaction more feasible for VLMs while retaining each environment’s physical challenges.Examples include grid-and-radius choices in PHYRE, named power and spin options in Pooltool, integer controls in Kinetix, and structured commands or timed JSON actions in the games and I-PHYRE.Source: §3.4 · pp. 6–7

4 Evaluation Protocol

DeepPHY evaluates VLMs through prompted interaction with physics environments, using planning modes and metrics to assess performance. Its protocol includes 17 open- and closed-source models alongside a random-action baseline.

4.1 Evaluation Setting

  • At each timestep, agents use visual observations and interaction history to choose actions, requiring rule interpretation, physical inference, and valid action generation.The evaluation includes in-advance planning, where a complete plan is submitted per attempt, and on-the-fly planning, where agents act turn by turn and observe consequences.Source: §4.1 · p. 7
  • The protocol uses Vision-Language-Action prompts for direct action generation and World Model prompts that also require predicting environmental changes.Reported metrics include Success Rate, Pass@K, and Average Attempts, with the latter averaged over successful trials.Source: §4.1 · p. 7

4.2 Models

  • The evaluation covers 17 open- and closed-source VLMs and includes a random-action MOCK baseline.The model set includes Qwen, Claude, Gemini, and GPT variants.Source: §4.2 · p. 8

5 Experimental Results

Across environments, VLM performance remains uneven and generally weak: leading closed-source models can exceed MOCK, but models remain far behind humans and struggle to convert physical descriptions into reliable control. World-model prompting offers limited help and can hurt performance, especially in harder tasks.

5.1 Overall Performance Analysis

  • Even flagship models struggle overall: most do not surpass MOCK, and all remain far behind human success rates across the benchmark.The evaluation reports Success Rate, Pass@K, and Average Attempts over three runs.Source: §5.1 · p. 8, §5 · p. 8

5.2 Detailed Analysis by Environment

  • On PHYRE, even the most advanced models generally achieve below 4% success on the first attempt; success rises slowly with additional attempts, reaching a best reported 23.1%.This suggests models struggle to learn effectively from failed attempts and revise their strategies.Source: §5.2 · p. 10
  • In I-PHYRE, GPT-o3 reaches 81.67% success by attempt 10, while open-source models perform poorly and can fall below MOCK.The strong result contrasts with the broader struggles reported in other environments.Source: §5.2 · p. 10
  • Kinetix success declines sharply from S-level to M- and L-level tasks, and annotations help on simple tasks but may harm harder ones.The WM prompt also often lowers success rates, particularly on harder tasks.Source: Figure 3 · p. 9, §5.2 · p. 10
  • In Pooltool, GPT-4o-mini’s reported 100% success reflects a repeated brute-force shot rather than strategic cue-ball control or complex physics planning.The authors describe the result as misleading because the output is deterministic and the environment non-random.Source: Table 3 · p. 9, §5.2 · p. 10

5.3 Case Study of Prompt Format

  • World-model prompting provides limited gains in simpler tasks but often becomes a liability as task complexity rises, despite sometimes producing correct physical descriptions.The descriptive predictions fail to translate into precise, executable control signals.Source: §5.3 · pp. 11–12

6 Conclusion

DeepPHY is presented as a comprehensive benchmark for interactive physical reasoning, and the authors conclude that even state-of-the-art models struggle with precise multi-step planning and predictive control. The supplied PHYRE appendix passages describe the benchmark’s task setup and its discretized action interface.

6 Conclusion

  • DeepPHY evaluates interactive physical reasoning and reveals a gap between describing physical phenomena and using that knowledge to predict and control outcomes.The authors release it as a testbed intended to benchmark these limitations and facilitate development of more physically grounded AI agents.Source: §6 · p. 12

B Benchmark: PHYRE

  • PHYRE-B tasks require placing one dynamic ball to make the green ball contact a purple static ball or blue dynamic ball.The evaluation combines within-template and cross-template test sets into 1,000 tasks because the suite does not train or fine-tune models.Source: B Benchmark: PHYRE · p. 16

B.1 Conversion Details

  • PHYRE actions select a placement cell and discrete ball-radius level; the experiments use a 5 × 5 grid and five radius levels for 125 possible actions.The agent selects the center of a target grid cell.Source: §B.1 · p. 17

B.2 Prompts

  • The PHYRE prompt requires a single cell-and-radius action and directs agents not to repeat combinations from failed attempts.The supplied prompt also distinguishes invalid placements from failed valid attempts.Source: §B.2 · pp. 18–19

B.3 Experiment Results

PHYRE results show low success and limited improvement from feedback, while the supplied I-PHYRE passages describe a timed block-removal puzzle and its visual indexing. The prompt-comparison passage reports that WM generally underperforms VLA in PHYRE.

B.3 Experiment Results

  • GPT-o3 with VLA reaches only 23.1% final success after 10 PHYRE attempts, and open-source models often perform near or below MOCK.Most models score below 4% on their first attempt, and successful trials require four to five attempts on average.Source: §B.3 · pp. 19–20, Table 6 · p. 19
  • Most PHYRE models score below 4% on Pass@1, indicating weak advance planning from a single static observation.The passage attributes the difficulty to devising a complete solution without prior interaction history.Source: §B.3 · p. 19
  • PHYRE performance improves only marginally over attempts, with successful trials requiring four to five attempts on average.The authors say failed-trajectory feedback is insufficient for building an accurate predictive world model to guide later decisions.Source: §B.3 · p. 20
  • In PHYRE, WM consistently underperforms VLA across most models, despite accelerating early progress.The passage says gains diminish in later attempts and descriptive prediction does not translate into better procedural control.Source: §B.3 · p. 21, Table 9 · p. 21

C Benchmark: I-PHYRE

  • I-PHYRE contains 40 scenarios in which agents must time removals of designated gray blocks to direct red balls into the abyss.Scenes include static obstacles, dynamic blocks, springs, and rigid sticks; indexed images support object selection by VLMs.Source: C Benchmark: I-PHYRE · p. 21, §C.1 · p. 21

C.1 Conversion Details

  • I-PHYRE uses Planning in Advance, allowing up to 10 attempts per turn; its prompt requests a timed sequence of block-elimination actions.The action sequence is output as JSON with each action specifying a time and target-block index.Source: §C.1 · p. 22, §C.2 · p. 23

C.3 Experiment Results

I-PHYRE results show strong performance by leading closed-source models but persistent weakness among tested open-source models, and WM does not outperform VLA. The supplied Kinetix passages describe its task levels and adaptations for VLM control.

C.3 Experiment Results

  • In I-PHYRE, GPT-o3 reaches 81.67% success within 10 attempts, while GPT-o4-mini reaches 75.00% and Claude 4.0 Opus 71.67%.GPT-o3 averages 2.84 attempts for successful models; tested Qwen models perform far below MOCK, with Qwen-7B at 20% success.Source: §C.3 · pp. 25–26, Table 10 · p. 25
  • WM fails to outperform VLA for nearly all high-performing I-PHYRE models; GPT-o3 drops from 81.67% with VLA to 76.67% with WM.The authors suggest prediction demands may interfere with generating the timed action sequence.Source: Table 11 · p. 25, §C.3 · p. 26

D Benchmark: Kinetix

  • Kinetix tasks require making green objects touch blue objects while avoiding red objects, using up to four motors and two thrusters.Its levels range from simple singular-control tasks to medium coordinated mechanisms and large planning and fine-control challenges.Source: D Benchmark: Kinetix · p. 27
  • Kinetix testing uses 74 fixed handcrafted levels: 10 S-level, 24 M-level, and 40 L-level tasks.Training levels are programmatically sampled and theoretically infinite.Source: D Benchmark: Kinetix · p. 27

D.1 Conversion Details

  • Kinetix’s rendering was refactored for clearer VLM observations, with explicit labels added to interactive motors and thrusters.The figure contrasts the original ambiguous colors, clearer rendering, and labeled controls.Source: §D.1 · p. 27, Figure 10 · p. 29
  • Kinetix maps discrete VLM inputs to continuous motor and thruster controls, extends simulation steps by 16, and provides five prior steps of history.Motor inputs distinguish no action, forward torque, and reverse torque; thruster inputs select off or maximum thrust.Source: §D.1 · pp. 27–28, Table 13 · p. 27

D.3 Experiment Results

Kinetix performance falls sharply as task complexity grows, while annotations and world-model prompting do not consistently help. The case study and Pooltool setup illustrate challenges of state-aware control and how the benchmark translates billiards into VLM actions.

D.3 Experiment Results

  • Kinetix success rates drop from over 60% on S-level tasks to under 15% on L-level tasks for GPT-o3, reflecting difficulty with longer-horizon coordinated control.All model families show a significant decline as tasks move from simple to more complex levels.Source: §D.3 · p. 30
  • Annotations generally help on simple S-level tasks but can hurt on M- and L-level tasks, especially with WM prompting, where parsing labels may burden models.WM prompting also fails to improve consistently and often degrades performance relative to direct VLA.Source: §D.3 · pp. 30–31

D.4 Case Study

  • In the Kinetix case study, repeating a previously successful action without accounting for the changed position and momentum leads to an obstacle collision and failure.The example illustrates a gap between predicting an action’s outcome and controlling from the updated state.Source: §D.4 · p. 31

Pooltool

  • Pooltool evaluates billiards in a realistic-physics sandbox, with the agent tasked to pot the 9-ball by first hitting the lowest-numbered ball.The stated setup runs 100 experiments per model with up to 15 attempts each.Source: Pooltool · p. 31, §E.1 · p. 32

E.1 Conversion Details

  • Pooltool converts its 3D rendering to a 2D top-down view and exposes shot power, direction, and spin through the cue-control API.The VLM selects discrete power and strike-spot options; aiming defaults to the lowest-numbered ball, while vertical angle is ignored.Source: §E.1 · pp. 32, 34

E.2 Prompts

  • The Pooltool prompt asks agents to choose a speed and strike spot for each shot while aiming to pocket the 9-ball and avoid potting the cue ball.It also asks the agent to consider the cue ball’s position for follow-up play.Source: §E.2 · p. 35

E.3 Experiment Results

Pooltool success rates can conceal reliance on fixed-action heuristics, and world-model prompting often fails to improve performance. The accompanying Angry Birds setup gives agents screenshots and asks them to plan launches using sequential birds and their abilities.

E.3 Experiment Results

  • GPT-4o-mini (VLA) and Qwen-3B (WM) each reach 100% success, but GPT-4o-mini’s fixed Medium-speed, Top-Spin action succeeds on the eighth attempt in every trial.Other successful models dynamically adjust their shots, while WM prompting often degrades results; Gemini-2.5-Pro drops from 68.00% with VLA to 43.00% with WM.Source: §E.3 · pp. 35, 37, Table 16 · p. 36

Angry Birds

  • The Angry Birds benchmark selects 34 first-chapter levels and tests eliminating all pigs with a limited sequence of birds requiring projectile and structural reasoning.The available birds’ order constrains planning across shots.Source: Angry Birds · p. 37, §F.2 · p. 39

F.1 Conversion Details

  • For Angry Birds, agents observe a level screenshot and choose a launch angle from 0 to 90 degrees and power from 0.0 to 1.0.The action is launching the current bird; configurations of available birds are annotated.Source: §F.1 · p. 37

F.2 Prompts

  • The prompt directs agents to select shots using trajectory, structural weaknesses, bird abilities, and the required order of birds.It asks for one shoot action for the current bird, with the bird’s special ability assumed to be used optimally.Source: §F.2 · pp. 38–39

F.3 Experiment Results

Angry Birds results show low VLM success relative to the non-expert human benchmark, with failures especially involving cascading structural consequences. Cut the Rope tests timing and sequencing in dynamic physics puzzles.

F.3 Experiment Results

  • Claude 3.7 Sonnet achieves a 41.18% success rate, below the non-expert human benchmark of 64.71%.The analysis attributes model difficulty chiefly to predicting cascading structural collapse and planning shots that exploit weak points.Source: §F.3 · p. 39
  • When models complete Angry Birds tasks, some achieve mean star counts comparable to or above the human player’s 2.00, despite low overall success.Qwen-72B scores 2.30 and Claude 3.5 Sonnet 2.22 on Mean ★(Completed).Source: §F.3 · p. 40

Cut the Rope

  • Cut the Rope requires precise rope-cutting and tool use to deliver candy, testing timing, sequencing, and predictions of motion under gravity, momentum, and inertia.The task also involves swinging and falling processes.Source: Cut the Rope · pp. 40–41

G.1 Conversion Details

Cut the Rope uses annotated screenshots and real-time bubble recognition to support an action-by-action perception loop, while excluding levels with unreliable element tracking. Its prompts ask agents to plan timed, sequenced interactions toward candy delivery and star collection.

G.1 Conversion Details

  • Cut the Rope annotates static props and recognizes bubbles in real time, then pauses after each action to provide a new screenshot for the next decision.The benchmark uses 88 levels from Session 1 and excludes multi-screen transitions and levels where element positions cannot be obtained in real time.Source: §G.1 · p. 41, Figure 16 · p. 41, Figure 17 · p. 42
  • The action set includes cutting ropes, activating bubbles or air cushions, waiting for physics to unfold, and declaring success or failure.The commands structure interactions and timing within each level attempt.Source: §G.1 · p. 44

G.2 Prompts

  • The prompt asks agents to plan a sequence of cuts and tool interactions with precise timing to deliver candy to Om Nom and collect stars.It emphasizes predicting trajectories and combining tools; multi-screen levels are excluded because shifting element positions hinder reliable recognition.Source: §G.1 · p. 41, §G.2 · pp. 45–46

G.3 Experiment Results

In Cut the Rope, tested VLMs trail non-expert human players in both success and solution quality. Failures reflect difficulty with perception, physical intuition, timing, and maintaining state across viewpoint shifts.

G.3 Experiment Results

  • 26.14% success for Claude 4.0 Opus trails the human players’ 41.36% average in Cut the Rope.Humans collected 91.6 stars on average, nearly triple the best model’s 33.00.Source: §G.3 · p. 47
  • Models struggle to count ropes, infer momentum and trajectory, time actions, and retain a coherent state when screen transitions shift the viewpoint.They may act by brute force or hesitate instead of waiting for an optimal swing position; the paper concludes that recognizing physical elements does not ensure procedural control.Source: §G.3 · pp. 47–48
  • Screen-transition levels were excluded from the final benchmark because viewpoint shifts were beyond the reach of any tested VLM.Source: §G.3 · p. 48
  • 2.51 stars per completed level for humans compares with 1.88 for GPT-o3 and 1.43 for Claude 4.0 Opus, showing weaker solution quality even on solved levels.Models often achieve the primary objective without the timed maneuvers needed to collect all three stars.Source: §G.3 · p. 48

H Human performance

  • Human results are ballpark comparisons, not expert performance targets, and come from non-experts: two co-authors for Pooltool and Angry Birds and five crowdsourced workers for Cut the Rope.Source: H Human performance · p. 48

Related papers

Loading 2508.05405v1…