PaperlayerView exact revision on arXiv

Source-linked AI summary

Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow

Karthik Dharmarajan, Wenlong Huang, Jiajun Wu, Li Fei-Fei, Ruohan Zhang

arXiv:2512.24766v1cs.ROcs.AIcs.CV

TL;DR

Dream2Flow addresses the challenge of translating human-oriented video predictions into low-level robotic actions for open-world manipulation. It reconstructs 3D object flow from generated videos and tracks that flow through trajectory optimization or reinforcement learning, enabling zero-shot manipulation across diverse object types and domains without task-specific demonstrations.

  • Problem

    Video models can predict plausible physical interactions, but translating human-led motions into robot actions remains difficult because of the embodiment gap and different action spaces.

  • Method

    Dream2Flow reconstructs 3D object motion from instruction-conditioned videos and uses it as an intermediate trajectory-tracking target for robot control.

  • Results

    Simulation and real-world experiments show zero-shot manipulation across rigid, articulated, deformable, and granular objects using RGB-D observations and language without task-specific demonstrations.

  • Takeaways & Limitations

    3D object flow provides a general interface for adapting video generation models to open-world robotic manipulation across embodiments and control methods.

  • Takeaways & Limitations

    The method assumes rigid grasps for real-world manipulation, requires 3–11 minutes to obtain 3D flow, and handles heavy occlusions poorly.

Abstract

from arXiv · show

Generative video modeling has emerged as a compelling tool to zero-shot reason about plausible physical interactions for open-world manipulation. Yet, it remains a challenge to translate such human-led motions into the low-level actions demanded by robotic systems. We observe that given an initial image and task instruction, these models excel at synthesizing sensible object motions. Thus, we introduce Dream2Flow, a framework that bridges video generation and robotic control through 3D object flow as an intermediate representation. Our method reconstructs 3D object motions from generated videos and formulates manipulation as object trajectory tracking. By separating the state changes from the actuators that realize those changes, Dream2Flow overcomes the embodiment gap and enables zero-shot guidance from pre-trained video models to manipulate objects of diverse categories-including rigid, articulated, deformable, and granular. Through trajectory optimization or reinforcement learning, Dream2Flow converts reconstructed 3D object flow into executable low-level commands without task-specific demonstrations. Simulation and real-world experiments highlight 3D object flow as a general and scalable interface for adapting video generation models to open-world robotic manipulation. Videos and visualizations are available at https://dream2flow.github.io/.

I. INTRODUCTION

Dream2Flow addresses the embodiment gap between human-centric video predictions and robotic action spaces by using 3D object flow as an intermediate interface. It reconstructs task-relevant object motion from generated videos and converts it into robot actions through trajectory tracking.

  • The work responds to the embodiment gap caused by video models producing human-oriented interactions rather than robot-compatible actions.Human interaction data are more extensively documented, while robots operate in different action spaces.
  • Dream2Flow uses 3D object flow to bridge high-level video simulation and low-level robot actions.The approach separates task-relevant state changes from the actuators that realize them.
  • The method formulates manipulation as tracking the object trajectory imagined by a generated video.Trajectory optimization and reinforcement learning can use the extracted 3D motion as their tracking goal.
  • The pipeline combines language-conditioned video generation, vision-based 3D flow extraction, and action synthesis with off-the-shelf models.It uses generated interaction videos, depth estimation, point tracking, and downstream control.
  • Dream2Flow is evaluated for zero-shot manipulation in simulated and real domains using RGB-D observations and language instructions.The experiments target diverse manipulation settings and examine generalization, alternative interfaces, and design choices.

C. Video Models for Robotics

Dream2Flow situates video models as sources of predicted task-relevant motion rather than direct robot policies. Its formulation maps generated object trajectories into actions while allowing different action spaces and dynamics models.

  • C. Video Models for Robotics: Dream2Flow uses generated video predictions as an intermediate object-centric interface for robot manipulation.The framework follows prior motion-field and object-centric approaches that separate desired object motion from embodiment-specific actions.
  • C. Video Models for Robotics: Given language, RGB-D input, and camera calibration, the system outputs an action sequence that follows motion inferred from generated video.The action representation is not fixed and may contain motion primitives, end-effector poses, or low-level controls.
  • C. Video Models for Robotics: The method extracts 3D object trajectories by lifting tracked image points with estimated video depth.A binary object mask and camera projection convert masked points into robot-frame trajectories.
  • C. Video Models for Robotics: The formulation represents state with task-relevant object points and robot state, then predicts dynamics under candidate actions.Time-aligned targets from the video flow define the object-motion objective at each planning step.

B. Extracting 3D Object Flows from Videos

Dream2Flow extracts 3D object flow from generated videos using depth, segmentation, and point tracking, then converts that flow into executable actions. Different domains use trajectory optimization, random shooting, or reinforcement learning with domain-specific dynamics.

  • B. Extracting 3D Object Flows from Videos: Generated RGB videos provide the visual motion source, but excluding the robot improves the physical plausibility of object trajectories.The video model is conditioned on task language and a workspace image without the robot visible.
  • B. Extracting 3D Object Flows from Videos: Calibrated video depth is obtained by aligning monocular depth estimates to the robot’s initial RGB-D observation.A global scale and shift resolve the monocular video scale ambiguity.
  • B. Extracting 3D Object Flows from Videos: The system localizes the relevant object, segments it, tracks sampled points across frames, and lifts visible points into 3D trajectories.Grounding DINO, SAM 2, and CoTracker3 provide localization, masks, and point tracks before projection into the robot frame.
  • B. Extracting 3D Object Flows from Videos: For simulated pushing, random shooting selects push parameters whose learned dynamics produce the lowest 3D object-flow cost.The push skill is parameterized by contact position, direction, and distance.
  • B. Extracting 3D Object Flows from Videos: For real-world manipulation, trajectory optimization uses absolute end-effector poses and a rigid-grasp dynamics model to follow the flow.The system selects a grasp near the video-detected interaction point and adds smoothness and reachability costs.
  • B. Extracting 3D Object Flows from Videos: For simulated door opening, reinforcement learning uses 3D object flow as an object-centric reward across multiple embodiments.Different action spaces support delta end-effector poses or delta joint angles for grippers and dexterous hands.

IV. EXPERIMENTS

Dream2Flow is evaluated across diverse manipulation tasks, robustness variations, and real-world trials. The experiments also expose failure modes associated with video generation, flow extraction, and robot execution.

  • Evaluation Tasks: Dream2Flow evaluates manipulation across simulated and real-world tasks involving rigid, articulated, deformable, and non-prehensile interactions.Tasks include Push-T, placing bread in a bowl, opening an oven, covering a bowl, and opening a door.
  • Push-T: For Push-T, six generated videos substantially morphed the T-block, which ruined tracking and downstream execution.The task uses 100 trials across ten initial configurations and different random seeds.
  • Robustness: Across six varied real-world scenarios, performance generally remains stable across object instances, backgrounds, and viewing angles, except for placing a large piece of bread.The evaluation uses five additional trials for each of six scenarios.
  • Generalization: Dream2Flow also performs different language-specified tasks in the same scene and supports in-the-wild tasks including pulling a chair, opening a drawer, sweeping pasta, and recycling a can.The same-scene behavior changes arise from adapting object-flow targets to different task instructions.
  • Failure Analysis: In the 60 real-world trials, failures arise from video morphing or hallucinated objects, invisible tracks after severe rotations or occlusions, and insufficient grasping or motion.The reported breakdown includes 12 video-generation failures, four flow-extraction failures, and four robot-execution failures.

C. How does Dream2Flow perform compared to alternative interfaces?

Dream2Flow uses 3D object flow as an intermediate representation and generally outperforms rigid-trajectory alternatives while supporting comparable policy performance across embodiments.

  • Dream2Flow outperforms AVDC and RIGVID across three real-world tasks by tracking 3D object flow rather than rigid transforms alone.
  • Policies trained with 3D object-flow rewards perform comparably to object-state-reward policies across Panda, Spot, and GR1 embodiments.
  • Spot policies move the base for reachability, while GR1 policies use the finger–palm area to pull the door more stably.

E. How does the choice of video model affect Dream2Flow

Dream2Flow’s performance depends on the video generator and downstream dynamics model: model-specific artifacts affect tasks, while particle dynamics best handles required object rotation.

  • Veo 3 excels on Open Oven, whereas Wan 2.1 performs better on simulated Push-T.
  • Kling 2.1 produces more Push-T morphing, while Wan 2.1 and Kling 2.1 often generate incorrect Open Oven articulation directions.
  • Particle dynamics substantially outperform pose and heuristic models because per-point predictions account for the rotation required in Push-T.
  • Real-robot failures commonly arise from video artifacts, tracking errors, and grasp-selection mismatches.

APPENDIX

The appendix specifies task prompts and model-input conventions for real-world manipulation, Open Door, and Push-T evaluations.

  • Real World Task Language Prompt: Real-world prompts combine an initial RGB observation with a language instruction, often specifying one-handed action and a still camera.
  • Real World Task Language Prompt: The evaluated real-world tasks include placing bread in a bowl, opening an oven, covering a bowl, pulling a chair, opening a drawer, and sweeping pasta.
  • Real World Task Language Prompt: Robustness prompts replace “bread” with “donut” or “long piece of bread,” while Kling 2.1 additionally uses relevance and negative-prompt settings.
  • Push-T Task Language Prompt: Push-T uses a goal image because success requires accurate block position, but Veo 3 was excluded because it lacked end-frame prompting.
  • Open Door Task Language Prompt: The Open Door prompt specifies opening the door rightward while the camera remains still and does not include a hand in simulation.

B. Particle Dynamics Model

The Push-T particle dynamics model predicts per-particle position changes from feature-augmented scene particles and supports replanning toward intermediate video-flow subgoals.

  • The model takes feature-augmented particles containing position, RGB, normals, and push parameters, then predicts each particle’s next-step position delta.
  • Scene particles are formed by combining four RGB-D camera point clouds, removing points outside the platform, and voxel-downsampling them.
  • The model is trained from 500 random-pushing transitions, with particle positions tracked before and after each push.
  • Dream2Flow uses nearest-neighbor matching to transfer predicted scene-particle displacements to tracked T-block particles.
  • Planning selects an intermediate video-flow subgoal after the closest matched timestep, using a fixed look-ahead of L = 20 to preserve rotational accuracy.

D. Grasp Selection

Dream2Flow selects grasps using object geometry and video-predicted hand interactions, then optimizes smooth, reachable robot trajectories for execution.

  • D. Grasp Selection: Video-predicted hand interactions identify which part of articulated objects should be grasped, while rigid single-part objects generally permit stable grasps anywhere.HaMer detects the hand position to guide grasp selection.
  • D. Grasp Selection: Dream2Flow falls back to the grasp nearest the movable object part when hand detections or proposed grasp correspondences are unavailable.
  • D. Grasp Selection: Points moving at least 1 pixel per timestep are treated as belonging to the movable flow, separating moving articulated parts from stationary structure.The threshold was chosen empirically because stationary parts exhibited lower average motion.
  • D. Grasp Selection: SAM 2 repairs intermittent tracking errors that could otherwise corrupt lifted 3D object flow and cause incorrect downstream actions.
  • D. Grasp Selection: The real-world planner minimizes task and control costs, including reachability, pose smoothness, and manipulability, before converting optimized poses into executable joint commands.The trajectory is fit with a B-spline and executed using inverse kinematics and joint impedance control.

G. In-the-Wild Tasks

In-the-wild evaluations test Dream2Flow on chair pulling, drawer opening, pasta sweeping, and can recycling, each with task-specific success criteria.

  • G. In-the-Wild Tasks: Chair pulling succeeds when the chair moves at least 5cm horizontally, although difficult configurations limit the required motion.
  • G. In-the-Wild Tasks: Drawer opening succeeds when the partially opened drawer reaches at least 90% of its full extension.
  • G. In-the-Wild Tasks: The in-the-wild suite includes pulling a chair, opening a drawer, sweeping pasta into compost, and recycling an aluminum can.Figure 11 shows rollouts for all four tasks.
  • G. In-the-Wild Tasks: Pasta sweeping requires grasping a brush and placing all four pasta pieces inside the compost bin.
  • G. In-the-Wild Tasks: Can recycling succeeds when an upright aluminum can is placed inside the recycling bin.

H. Open Door Reinforcement Learning Details

Open Door reinforcement learning compares handcrafted state rewards with 3D object-flow rewards across multiple robot embodiments and uses video-derived trajectories as guidance.

  • H. Open Door Reinforcement Learning Details: SAC policies for Franka Panda, Spot, and GR1 follow 3D object flow extracted from generated videos as a reward signal.GR1 uses 10,000 training iterations because of its larger action space.
  • H. Open Door Reinforcement Learning Details: The comparison evaluates a handcrafted object-state reward against a 3D object-flow reward.
  • H. Open Door Reinforcement Learning Details: The object-state reward combines reaching and handle-rotation terms, returning completion reward 1.0 once the hinge exceeds 0.3 radians.
  • H. Open Door Reinforcement Learning Details: The flow reward tracks a reference trajectory extracted from video and adds end-effector alignment around the object particles.Its particle term uses the closest reference timestep based on current particle positions.
  • H. Open Door Reinforcement Learning Details: For computational efficiency, door particles are transformed as the door angle changes, with the door joint angle included in the state.

I. Video Generation Failures

Generated videos can fail through object morphing or hallucinated objects, and Dream2Flow’s broader limitations include rigid-grasp assumptions, processing time, and heavy occlusion.

  • I. Video Generation Failures: Object morphing changes an existing object’s shape and can make downstream tracking fail, while hallucination introduces a new object and can cause incorrect execution.The bread-to-crackers and extra-bowl examples illustrate both failure modes.
  • I. Video Generation Failures: Dream2Flow relies on a rigid-grasp assumption for real-world manipulation, limiting the task types directly supported.Non-prehensile pushing is demonstrated through a particle dynamics model, but scaling that model to the real world is non-trivial.
  • I. Video Generation Failures: Processing 3D object flow takes 3 to 11 minutes depending on the video generation model, with video generation as the main bottleneck.
  • I. Video Generation Failures: Because Dream2Flow relies on one generated-video angle, heavy occlusions such as a hand covering most of a small object are not handled gracefully.
Loading 2512.24766v1…