Source-linked AI summary
Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?
Rui Zhao, Kaiming Yang, Jifeng Zhu, Siyang Chen, Ziqi Wang, Weijia Wu, Kevin Qinghong Lin, Heng Wang, Mike Zheng Shou
TL;DR
Video-generation benchmarks typically measure visual quality without establishing whether generated motions are physically executable. Dream.exe evaluates this question by converting scene-conditioned manipulation videos into robot trajectories and executing them in simulation, finding measurable success for several models while visual quality remains a poor predictor of executability.
Problem
Standard video evaluations do not test whether generated manipulation motions can accomplish physical tasks, leaving the physical knowledge of video models directly untested.
Method
Dream.exe evaluates generated manipulation videos through visual assessment, video-to-trajectory extraction, and closed-loop execution in a physics simulator.
Results
Across 8 models and 101 tasks, several models achieve measurable execution success, yet visual quality is a poor predictor of executability.
Takeaways & Limitations
Generative priors from internet-scale data encode physically meaningful motion, but visual evaluations alone do not capture executable behavior.
Takeaways & Limitations
The benchmark requires strictly reproducible initial scene states and is constrained to scenarios suitable for its extraction pipeline.
Abstract
from arXiv · showhide
Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream$.$exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream$.$exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream$.$exe will be open-sourced at https://github.com/showlab/Dream.exe.
1. Introduction
Dream.exe tests whether visually fluent video generators encode physical knowledge by translating generated manipulation videos into robot trajectories and simulated task execution. Across eight models, several achieve measurable execution success, but visual quality poorly predicts executability.
- Motivation: Standard video benchmarks assess visual quality and temporal consistency without testing whether depicted motions can accomplish physical tasks.A visually natural but physically impossible motion can therefore appear successful under conventional metrics.
- Motivation: Dream.exe treats simulated task success as a grounding signal for evaluating whether generated manipulation motion reflects physical laws.The framework uses robotic manipulation as a concrete test of whether video-implied trajectories are executable.
- Empirical study: 8 models spanning closed-source, open-source, and robot-specific systems are evaluated, revealing systematic differences between visual quality and physical executability.The benchmark characterizes models across these categories rather than restricting evaluation to one model family.
- Findings: Several models achieve measurable execution success, while visual leaders frequently fail in execution and modestly scoring models can produce physically valid trajectories.The findings indicate that internet-scale generative priors encode meaningful physical knowledge, but visual quality is a poor executability predictor.
- Evaluation framework: The protocol combines visual assessment, video-to-trajectory extraction and evaluation, and closed-loop execution in a physics simulator.The pipeline starts from a scene image and task description, then evaluates the generated video, extracts motion, and measures execution.
2. Related Work
Prior video-generation benchmarks emphasize visual and semantic quality, while video-based robot-learning work explores behavioral priors and planning. Dream.exe addresses the missing evaluation loop by making physical executability the primary metric.
- Video Generation Models: Video-generation research progressed from diffusion-based video synthesis and image-to-video pretraining toward increasingly powerful general-purpose models.The related work positions Dream.exe within rapid advances in video generation and learned motion priors.
- Video Generation Benchmarks: EvalCrafter, VBench, and T2V-CompBench evaluate visual quality, motion quality, temporal consistency, aesthetics, alignment, and compositional reasoning.These benchmarks decompose video assessment into multiple visual and semantic dimensions.
- Video Generation Benchmarks: Existing benchmarks do not test whether generated trajectories succeed when executed by a robot controller in a physics simulator.Visual plausibility classifiers differ from direct execution-based evaluation of physical validity.
- Positioning: Dream.exe closes the evaluation gap by directly bridging generated video and physical executability.Its primary metric is execution rather than perceptual quality alone.
- Robot Learning from Video: Video-based robot-learning approaches include imitation, internet-video pretraining, video-generated planning, and synthesized visual subgoals for hierarchical control.These works provide behavioral knowledge or planning mechanisms but represent a distinct line from Dream.exe’s benchmark focus.
3. Dream.exe: Benchmark Design
Dream.exe is built as a reproducible, manually curated manipulation benchmark spanning three levels of physical complexity and diversified viewpoints. Its pipeline generates videos, evaluates them visually, extracts robot actions, and executes those actions in simulation.
- Task Suite: The task suite is based on RoboCasa365 and requires reproducible initial scene states so models can be compared under equal conditions.The benchmark excludes scenarios that cannot reliably recover the same initial state.
- Task Suite: Manual curation filters episodes for camera suitability, object visibility, trajectory clarity, and semantic unambiguity.Camera viewpoints are tuned to improve visibility of both manipulated objects and end-effectors.
- Task Suite: The benchmark stratifies tasks into three increasing difficulty levels: atomic single-object manipulation, multi-object interaction, and multi-stage composite tasks.The levels progress from single continuous interactions to coupled object states and long-horizon sequences.
- Models Evaluated: The model set spans frontier closed-source generators, open-source generators, and a robot-specific policy model.This composition supports comparisons across general-purpose, reproducible open-weight, and task-specific systems.
- Evaluation Pipeline: Each evaluation begins with a scene image and task prompt, followed by video generation and visual scoring of stability, plausibility, and task adherence.Longer videos are generated for Level 3 tasks to accommodate their extended horizons.
- Evaluation Pipeline: A five-step extraction chain converts video motion into executable actions through region initialization, point tracking, depth-based 3D lifting, end-effector calibration, and gripper-aware assembly.The resulting action stream is designed for robot-controller execution in the corresponding simulation environment.
4. Experiments
Dream.exe evaluates generated manipulation videos through visual, trajectory, and robot-execution measures under a unified task protocol. Results show measurable success across models, but visual plausibility and quality do not reliably predict executable task performance.
- Experimental setup: All models receive the same scene image and task prompt under a unified protocol, with standard and enhanced instruction variants.Wan 2.2 fine-tuned variants use RoboCasa episodes, while CosmosPolicy is evaluated with multi-view input and curated benchmark-camera variants.
- Evaluation protocol: The evaluation combines VLM-based visual scoring, 3D trajectory similarity, trajectory executability, and task-level execution success.Trajectory metrics compare generated trajectories with rollout references, while execution metrics include checkpoint success, tracking errors, and smoothness.
- Robot execution evaluation: 6.2% Level-3 SR-B is achieved only by Kling 3.0, while Level-1 and Level-2 leaders reach 20.8% and 21.4% SR-B, respectively.Non-zero sub-goal scores on difficult tasks indicate partial progress even when complete task success is absent.
- More findings: Ground-truth depth substantially improves task-level success, identifying depth estimation as a major bottleneck for general-purpose video generators.Robot-specific models lead on checkpoint executability, but general-purpose models can match or exceed them on task success and several end-effector trajectory metrics.
- More findings: r = −0.03 measures the correlation between physical plausibility and SR-B, showing that visual quality is an unreliable predictor of executability.LTX 2.3 ranks first in physical plausibility but last in SR-B, whereas SeedDance 2.0 and Kling 3.0 achieve stronger task-level outcomes despite weaker visual scores.
- More findings: Robot-video fine-tuning improves trajectory similarity and robotic appearance without significantly improving task success rates.The reported failure taxonomy includes object levitation, phantom grasp, and kinematic breakdown, with the latter two accounting for most failed trials.
5. Conclusion
Dream.exe directly tests whether video-generation models can produce manipulation motion that grounds in robotic execution. Across 8 models and 101 tasks, several achieve measurable success, but visual quality poorly predicts executability and long-horizon tasks remain challenging.
- 8 models evaluated across 101 tasks show a qualified yes: generated manipulation videos can sometimes be grounded through robotic execution.The benchmark finds measurable execution success and physically meaningful motion without robot-specific supervision.
- Several models achieve measurable execution success without robot-specific supervision, indicating physically meaningful motion in internet-scale generative priors.
- Visual quality remains a poor predictor of executability, so visually strong generations do not reliably produce successful robot behavior.
- Long-horizon tasks expose limits of current video-generation models.
A. VLM Visual Evaluation Details
The visual evaluation protocol uses VLM judgments over paired or grid-arranged video frames to assess robot stability, object consistency, physical plausibility, and task adherence.
- Two frames—the initial reference and the frame at 75% duration—are concatenated horizontally for VLM assessment.The first frame serves as reference and the later frame represents generated output.
- Robot-subject consistency is rated from A to E, distinguishing clear consistency from structural inconsistency or disappearance/substitution.Position and orientation changes are acceptable, while robotic grippers must be distinguished from robotic hands.
- Manipulated-object consistency uses the same protocol, with option E indicating that the object is absent in the later frame.
- The paired robot-subject and object ratings map through a fixed symmetric lookup table to a score from 1 to 15.The mapping assigns (A, A) to 15 and (E, E) to 1, with intermediate combinations interpolated monotonically.
- Six uniformly sampled frames are arranged in a 3 × 2 grid for evaluating common-sense physical consistency.The VLM checks floating or intersecting geometry, spontaneous object appearance, and false grasping, assigning low scores when anomalies occur.
Task Adherence
Task adherence is evaluated from a 3 × 2 grid of sequential frames, with a VLM rating how closely the generated video matches the specified manipulation task.
- A 3 × 2 sequential-frame grid is used to assess whether the robot successfully performs the described task.
- The VLM rates task adherence from 1 to 5, where 5 indicates a perfect match and 1 indicates no relevance.Scores of 4 or 5 are assigned cautiously.
B. Model Details
Dream.exe evaluates models organized by category, distinguishing open-weight models from closed-source API models and conditioning all models on an initial scene image plus task instruction.
- All evaluated models are conditioned on the initial scene image paired with a task instruction.
- Video resolution and duration are selected not to exceed each task horizon, using longer videos for Level 3 multi-stage tasks.
- Table 6 organizes evaluated models by category and marks open-weight models with ✓, distinguishing them from closed-source API models.
C. Video2Traj Implementation Details
The video-to-execution implementation tracks generated motion, estimates depth and 3D trajectories, infers orientation and gripper actions, and replays the resulting actions in simulation. Reference rollouts isolate pipeline limitations and show depth as a major execution bottleneck.
- Video2Traj Implementation Details: Depth estimation lifts tracked pixels into world-frame 3D trajectories using a robot-adapted model trained with simulator-rendered rollout data.The depth model uses low-rank adapters while the remaining DVD DiT weights remain frozen.
- Video2Traj Implementation Details: A five-step pipeline converts generated video into 2D tracks, calibrated depth, lifted 3D trajectories, orientation, gripper actions, and executable robot commands.These signals are fused into a 7D trajectory for simulator replay.
- Video2Traj Implementation Details: End-effector visual-center tracks are aligned to the initial TCP pose, while rigid 3D alignment estimates orientation for 6-DoF action generation.A temporal guard suppresses implausible rotation jumps.
- Video2Traj Implementation Details: Gripper timing is inferred from geometric cues and task-specific interaction priors, then merged with the extracted end-effector trajectory.The priors specify expected close and open event counts for each interaction mode.
- Video2Traj Implementation Details: Closed-loop execution replays delta 6-DoF actions and gripper commands through a robosuite operational-space controller after restoring each initial simulator state.Execution proceeds through controller steps that reach retained trajectory checkpoints.
- Video2Traj Implementation Details: Ground-truth rollout videos provide reference bounds, and replacing estimated depth with ground-truth depth substantially improves task-level success.The reported gap identifies depth estimation as a major bottleneck because errors can propagate through 3D lifting, TCP positioning, and contact timing.
D. Additional Quantitative Results
Additional results report visual-quality scores across instruction settings, difficulty levels, and judges, including per-judge and judge-averaged evaluations.
- D. Additional Quantitative Results: Table 8 breaks down visual-quality scores by instruction setting and difficulty level, reporting both individual-judge and judge-averaged results.The evaluated dimensions are robot stability, physical plausibility, and task adherence.
D.2. Trajectory Similarity Metrics under Different Instruction Settings
Trajectory similarity is evaluated under standard and enhanced instructions using multiple trajectory components and metrics.
- D.2. Trajectory Similarity Metrics under Different Instruction Settings: Trajectory similarity is reported for enhanced and standard instructions using end-effector visual-center, TCP, and object trajectories.HSD, DYN, and NDTW measure trajectory shape, dynamics, and temporal-alignment similarity.
E. More Results
Additional qualitative and quantitative results compare generated videos with executed rollouts and report trajectory feasibility, task success, and visual-quality context. Successful cases recover coherent interactions as executable trajectories, while artifacts such as inconsistent contacts can prevent valid action sequences.
- E. More Results: Qualitative examples compare six temporally aligned frames from each generated manipulation video with its recovered execution rollout.The comparison covers both successful and failed video-to-execution outcomes.
- E. More Results: Successful cases recover coherent object motion and stable robot-object interactions as executable trajectories.Failure cases include spurious objects and inconsistent contacts that cannot be mapped to valid robot action sequences.
- E. More Results: Trajectory-feasibility results report checkpoint reachability, TCP tracking disagreement, position and rotation errors, and executed-trajectory smoothness.The metrics are E-SR, nDTW, Pos95/Rot95, and Smth, with directionality specified in Table 10.
- E. More Results: Task-level results report binary success, partial completion, and sub-goal completion for release, placement, articulation, and core-task outcomes.The metrics are SR-B, SR-P, Rel, Place, Art, and Core.
- E. More Results: Generation-model rankings are computed separately within each instruction setting, while rollout-video reference rows are shaded and excluded from those rankings.Zero-valued entries are not highlighted even when tied.