AgentDreamer: Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation

1 INTRODUCTION

• Video guidance may encode the wrong physical behavior—for example, sliding instead of rolling—so plausible simulations can still conflict with the instruction.The distinction is rotation, which the sliding reference omits.Source: §1 · p. 1
• AgentDreamer formulates motion planning as physical reasoning, using an off-the-shelf multimodal agent to turn text and scene geometry into explicit 3D guidance.The agent takes a direct role in constructing the motion plan.Source: §1 · p. 1, Figure 1 · p. 2
• Temporal-chunk parametric functions describe distinct motion stages and yield dense trajectories as pseudo-supervision for differentiable MPM optimization.Automatic semantic injection associates sampled points with visible object parts, supported by colored point markers and coordinate information.Source: §1 · p. 2
• Motion-IG compares masked motion-content recovery likelihoods with and without simulated motion to measure the motion's information beyond the remaining text.This isolates motion's contribution rather than scoring the complete text and motion together.Source: §1 · p. 2
• Experiments report improved text alignment over video-based guidance, with simulations more closely following the requested physical behavior.This is the paper's reported qualitative comparison in the introduction.Source: §1 · p. 2

Dynamic 3D representations

• Dynamic 3D Gaussian methods model temporal change with trajectories, deformation fields, or space-time primitives; PhysGaussian couples Gaussian primitives with MPM.Later work extends these representations to elastic deformation, material estimation, and varied material behaviors and interactions.Source: §2 · p. 2

Text-conditional physics simulation

• Text-conditioned physics methods use video diffusion guidance, while VLIPP and C-Drag (Li et al., 2025) use sparse image-space plans; AgentDreamer instead supplies dense 3D trajectories.PhysFlow uses optical flow, and MotionPhysics uses learnable motion distillation.Source: §2 · p. 3, Figure 2 · p. 3

Agent-guided physics-aware motion evaluation

• Prior agent-guided evaluation relies on semantic or physical judgments, whereas Motion-IG measures how simulated motion improves masked motion-semantic recovery beyond remaining text.The related work frames this as conditional information gain.Source: §2 · p. 3

3 METHOD

• AgentDreamer turns text and initial scene geometry into agent-generated 3D motion plans that provide pseudo supervision for physics optimization.The planned trajectories guide a differentiable MPM simulator.Source: §3 · p. 3, §3.1 · p. 3, §3.1.2 · p. 4

3.1.1 AGENT-BASED MOTION PLANNING

• Parametric functions divide the horizon into temporal chunks and produce dense, coherent trajectories while preserving spatial relationships between points.Each chunk function combines a center trajectory with local motion for point groups, and continuity is enforced between chunks.Source: Figure 3 · p. 4, §3.1.1 · p. 4
• Consistent particle-index markers across annotated views help the agent associate selected 3D points with visible object parts.The inputs also specify the coordinate origin, axis directions, and spatial units.Source: §3.1.1 · p. 4

3.1.2 MPM OPTIMIZATION

• A differentiable MPM rollout optimizes material parameters and per-particle initial velocities to reproduce planned motion under scene dynamics.The loss matches selected particles at output times; gradients update simulator parameters while the plan remains fixed.Source: §3.1.2 · pp. 4–5

3.2 MOTION-IG

• Motion-IG measures how simulated motion changes recovery likelihood for masked motion-related text beyond the remaining text and annotated views.It masks motion words and attributes while retaining object and part names; positive scores indicate motion increases recovery likelihood, while negative scores indicate it decreases likelihood.Source: §3.2 · p. 5
• Motion-IG scores masked-word recovery with and without motion using a fixed vision-language model, scoring each selected word’s first subword token.Scores are averaged over words and instructions, with instructions, masks, annotated views, and evaluation models held identical across methods.Source: §3.2 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• The evaluation uses GPT-5.6 Sol for planning, eight scenes, 20 instructions, two video-guided baselines, Motion-IG evaluators, and human rankings.The eight scenes comprise four real-world and four synthetic scenes; instructions cover five motion types.Source: §4 · p. 6, §4.1 · p. 6

4.2.1 QUALITATIVE RESULTS

• Across a bouncing ball and a rocking fox, AgentDreamer better matches the requested rebound and attached-rear rocking than PhysFlow and MotionPhysics.For the ball, the baselines deform on impact but stay near the ground, while AgentDreamer rebounds after compression.Source: §4.2.1 · p. 6, Figure 4 · p. 7

4.2.2 QUANTITATIVE RESULTS

• AgentDreamer ranks highest under all four open-weight and five API-based Motion-IG evaluators, ranks first in 62/100 API evaluator–instruction pairs, and is human-ranked first in 73.6% of trials.Human pairwise preferences favor AgentDreamer over PhysFlow in 82.6% of trials and over MotionPhysics in 85.6%.Source: §4.2.2 · pp. 6–7, Table 2 · p. 7, Figure 5 · p. 7

4.3 ANALYSIS

• Generated plans resemble classical-mechanics function families for rolling, spin decay, falling, and oscillation, while deformation uses smoothstep interpolation.The correspondence analysis checks expected forms across 20 instructions.Source: §4.3 · p. 8, Table 3 · p. 8
• AgentDreamer scores highest across five motion categories, with the largest gains for rolling, spin decay, and falling and bouncing; gains are smaller for oscillation and deformation.Both baselines have negative Motion-IG for rolling, where similar video appearances can obscure the translation-plus-rotation distinction; human evaluation shows a similar category trend, but not individual-ranking agreement.Source: §4.3 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
• Both semantic injection and parametric planning improve alignment, with parametric planning providing the larger individual gain and their combination scoring highest.Parametric motion functions provide coherent, dense target trajectories; independently predicted point positions lose temporal coherence without them.Source: Table 4 · p. 9, §4.3 · p. 9
• The agent divides the ball instruction into chunks for movement, impact compression, and partial shape recovery, capturing stages that one function may not represent.Figure 8 visualizes target positions over time and the temporal chunks.Source: §4.3 · p. 9, Figure 8 · p. 9
• On the same 60 text–motion pairs, Motion-IG assigns a distinct score to each pair, whereas the MLLM judge assigns only 11 distinct values.The comparison scales each score type to its own range; finer Motion-IG resolution reduces ties when ranking these pairs.Source: §4.3 · p. 9, Figure 9 · p. 9

5 CONCLUSION

• AgentDreamer uses agent-generated 3D motion priors to guide differentiable physics optimization, while Motion-IG measures motion’s contribution to recovering masked words.Experiments show improved text alignment over video-guided baselines.Source: §5 · p. 9
• The authors report using generative AI for research development, methods, experiments, implementation, figures, manuscript work, and literature tasks, with human review of AI-assisted work.They state that AI-generated synthetic datasets and mathematical claims were not used.Source: AI USE STATEMENT · p. 10
• The study uses existing 3D scenes, and pretrained multimodal models may introduce errors or biases; model-based scores are not treated as a sole measure of physical correctness.The authors report human judgments alongside model-based scores.Source: ETHICS STATEMENT · p. 10
• The paper identifies sections describing the motion-planning, physics-optimization, and Motion-IG procedures, and appendices covering data sources, prompts, examples, and human evaluation.The reproducibility statement points readers to those materials.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL QUALITATIVE RESULTS

• In additional examples under the same instructions, AgentDreamer’s simulated motion more closely follows the requested behavior than PhysFlow and MotionPhysics.Figures 10 and 11 show simulated-video frames on the left and corresponding sampled-point trajectories on the right.Source: A ADDITIONAL QUALITATIVE RESULTS · p. 13, Figure 10 · p. 13, Figure 11 · p. 14

B ADDITIONAL QUANTITATIVE RESULTS

• +0.46 on rolling for AgentDreamer contrasts with −0.22 and −0.24 for the baselines; AgentDreamer ranks first in every category under both evaluator groups.For oscillation, its black-box score is +0.35, close to PhysFlow at +0.33; margins are narrower for oscillation and bending and deformation.Source: Table 5 · p. 15, B ADDITIONAL QUANTITATIVE RESULTS · p. 15
• AgentDreamer is ranked first in 73.6% of human trials and last in 5.4%, while PhysFlow and MotionPhysics are ranked last in 40.2% and 54.4%, respectively.The evaluation comprises 25 participants, 20 instructions, and 500 trials.Source: Table 6 · p. 15, B ADDITIONAL QUANTITATIVE RESULTS · p. 15

• The identifiers help the agent associate sampled points with visible object parts relevant to the text instruction.The examples show the points in the initial scene images.Source: C SEMANTIC INJECTION EXAMPLES · p. 16
• Numerical identifiers mark sampled control points in two views each of a fox and a soccer ball.The annotations are used for automatic semantic injection.Source: Figure 12 · p. 16

D.1 PROMPTS FOR MOTION PLANNING

• The motion-planning prompt has the agent divide the time horizon into chunks and specify parametric functions for groups of sampled points.Source: Figure 13 · p. 17
• A separate prompt translates the motion plan into deterministic Python code that computes target point positions.Source: Figure 14 · p. 18

D.2 PROMPTS FOR EVALUATION

• Open-weight evaluators receive annotated initial views, the motion as a trajectory and rendered frames, and a description with motion words masked.Source: §D.2 · p. 19
• Black-box evaluators state a probability for each masked word because current API models expose no teacher-forced log-probabilities.These ratings replace log-probabilities in the same gain.Source: §D.2 · p. 19
• Open-weight evaluators score masked words by teacher forcing rather than sampling a completion.Source: Figure 15 · p. 19, §D.2 · p. 19
• The black-box prompt collects each masked word's stated probability in one JSON response per query.Source: Figure 16 · p. 19

• For each of 20 instructions, participants ranked videos from PhysFlow, MotionPhysics, and AgentDreamer by text–motion alignment.Each set showed three videos of the same object.Source: E DETAILS OF USER STUDY · p. 20
• Participants ranked each motion by how well it matched what should move, in which way, and how strongly.They were told not to judge image quality, lighting, or rendering artifacts.Source: E DETAILS OF USER STUDY · p. 20
• Figure 17 shows an example question with a text instruction and three simulation results for the same object to rank.Source: Figure 17 · p. 20, E DETAILS OF USER STUDY · p. 20