Source-linked AI summary
NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
Hongyu Li, Lingfeng Sun, Yafei Hu, Duy Ta, Jennifer Barry, George Konidaris, Jiahui Fu
TL;DR
Robot manipulation methods face a data bottleneck: end-to-end VLA systems need extensive robot-specific data, while modular approaches still rely on skills that limit generalizability. NovaFlow distills generated task videos into actionable 3D object flow to guide robot actions. Its real-world experiments report performance above other zero-shot methods and imitation-learning policies trained on dozens of demonstrations.
Problem
End-to-end VLA models require extensive robot-specific data, while modular approaches still rely on predefined or learned skills that limit generalizability and scalability.
Source: §1 · p. 2Method
NovaFlow distills task-solving videos from pretrained video models into actionable 3D object flow, which its executor translates into robot actions without robot-specific data or task training.
Source: §1 · p. 2Results
Real-world experiments show NovaFlow outperforming other zero-shot methods and imitation-learning policies trained on dozens of demonstrations.
Source: §5 · p. 9Takeaways & Limitations
The object-centric, embodiment-agnostic framework handles rigid, articulated, and deformable objects across embodiments without task-specific training or demonstrations.
Source: §5 · p. 9The primary bottleneck is physical execution, particularly grasping and handling unexpected dynamics; the authors identify closed-loop feedback as a future direction.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Enabling robots to execute novel manipulation tasks zero-shot is a central goal in robotics. Most existing methods assume in-distribution tasks or rely on fine-tuning with embodiment-matched data, limiting transfer across platforms. We present NovaFlow, an autonomous manipulation framework that converts a task description into an actionable plan for a target robot without any demonstrations. Given a task description, NovaFlow synthesizes a video using a video generation model and distills it into 3D actionable object flow using off-the-shelf perception modules. From the object flow, it computes relative poses for rigid objects and realizes them as robot actions via grasp proposals and trajectory optimization. For deformable objects, this flow serves as a tracking objective for model-based planning with a particle-based dynamics model. By decoupling task understanding from low-level control, NovaFlow naturally transfers across embodiments. We validate on rigid, articulated, and deformable object manipulation tasks using a table-top Franka arm and a Spot quadrupedal mobile robot, and achieve effective zero-shot execution without demonstrations or embodiment-specific training. Project website: https://novaflow.lhy.xyz/.
1. Introduction
NovaFlow addresses the robot-data bottleneck in manipulation by separating task understanding from robot control. It distills video-generated motion into actionable 3D object flow and reports zero-shot performance across real-world tasks and object types.
1. Introduction
- End-to-end VLA models require costly robot-specific vision-language-action data, while modular methods still depend on predefined or learned skills that limit generalizability and scalability.VLA models [2] [4] contrast with the web-scale training data used for LLMs [5] [6] [7] and VLMs [9].Source: §1 · p. 2
- NovaFlow uses pretrained video generation models for task understanding, then translates generated motion into robot actions through actionable 3D object flow.The flow executor uses inverse kinematics and trajectory optimization, with correspondence-based tracking for rigid and articulated objects and particle-model planning for deformable objects.Source: §1 · p. 2
- The framework generates robot actions from one visual observation and a task description without robot-specific data or task training.Its flow generator uses pretrained perception modules for depth estimation, 3D point tracking, and object grounding.Source: §1 · p. 2
- NovaFlow achieves state-of-the-art zero-shot performance across real-world tasks, outperforming demonstration-free and data-dependent methods.The framework is described as object-centric and embodiment-agnostic, with no task-specific tuning.Source: §1 · p. 2
2. Related Work
Prior work seeks to transfer task understanding into robot actions while reducing reliance on robot-specific data. NovaFlow's related-work framing motivates video-derived, object-centric flow as an intermediate representation for that transfer.
2. Related Work
- The paper defines zero-shot or demonstration-free approaches as requiring neither robot-specific data collection nor task-specific training, and places 3D object flow between task understanding and robot control.The framing responds to the data bottleneck of end-to-end VLAs, which still rely on data collection for novel embodiments or camera views [37] [38].Source: §2 · p. 2
- The flow-generator pipeline converts a task prompt and initial image into object-centric 3D flow by lifting video to 3D, calibrating depth, tracking points, and grounding objects.The pipeline uses a video model followed by pretrained perception modules.Source: Figure 2 · p. 3
2.1. Video-based Manipulation
- Video-based manipulation methods have used inverse dynamics models, policies [28] [40] [41], or tracked poses [16], but require robot-specific data and often depend on embodiment-specific action generation.A concurrent object-centric method [16] generalizes across embodiments but assumes rigid bodies.Source: §2.1 · p. 3
2.2. Flow-based Manipulation
- Flow is model-free and does not assume object rigidity, but existing flow-manipulation methods require robot data or task-specific training for the generator or executor [42] [43] [44] [46].Chen et al. and Zhi et al. [48] train flow generators on large-scale human egocentric datasets; the authors report that generalizes less well than pretrained video models on in-the-wild object motion.Source: §2.2 · p. 3
3. NovaFlow
NovaFlow is a demonstration-free, embodiment-agnostic pipeline that turns a task description and initial observation into actionable 3D object flow, then into robot actions. Its flow generator distills motion from generated video, while its executor plans for rigid, articulated, and deformable objects.
3. NovaFlow
- NovaFlow extracts actionable 3D object flow from generated video and translates it into robot actions without robot-specific data or training before execution.The generator takes an initial RGB-D observation and task instruction; an optional goal image supports tasks requiring greater precision.Source: §3 · p. 3, §3.1 · p. 4
3.1.1. Video Generation
- The generator produces video from the initial image and task prompt, using first-last-frame-to-video generation when a goal image is supplied.The generator’s input can include an initial RGB-D image pair and known camera intrinsics.Source: §3.1.1 · p. 4, §3.1 · p. 4
3.1.2. Monocular Depth Estimation / 3.1.3. Depth Calibration
- Monocular depth lifts generated video into 3D, but systematic scale errors motivate calibrating the estimated sequence against the initial ground-truth depth map.The authors report that median scaling is more stable than affine fitting.Source: §3.1.3 · p. 4, §3.1.2 · p. 4
3.1.4. 3D Point Tracking
- A 3D point tracker uses camera intrinsics, generated video, calibrated depth, and first-frame query points to output 3D trajectories.The query points are evenly sampled on the first frame.Source: §3.1.4 · p. 4
3.1.5. Object Grounding
- Object grounding combines detection and video segmentation to retain consistently tracked surface keypoints, then uses VLM rejection sampling to select the most plausible generated flow.The VLM evaluates back-projected flow images; the authors find these more effective to reject than concatenated raw video images because they visualize motion directly.Source: §3.1.5 · p. 4
3.2. Flow Executor
- The executor converts flow over object keypoints into robot actions; it is an open-loop planner that can be extended with live tracking and handles rigid, articulated, and deformable objects.Grasp proposals determine the initial end-effector pose, after which trajectories are planned from flow with costs and constraints and tracked by the robot.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
3.2.1. Rigid Object Manipulation
- For rigid objects, keypoint flow yields model-free rigid transforms; assuming a firm, nonslipping grasp, the object pose determines end-effector poses for trajectory optimization.The rotation is found by minimizing squared error with the Kabsch algorithm, and a grasp transformation converts object poses to end-effector poses.Source: §3.2.1 · p. 5
3.2.2. Deformable Object Manipulation
- For deformable objects, flow provides point correspondences and intermediate tracking targets, while particle dynamics support model-predictive control that replans over a horizon and executes the first action.The authors describe correspondence-based costs as potentially better conditioned than correspondence-free metrics; trajectory optimization also includes smoothness, rest-pose, collision, and joint-limit terms.Source: §3.2.2 · pp. 5–6
4. Experiments
Experiments test NovaFlow across object types and embodiments, compare it with zero-shot and demonstration-trained baselines, and examine design choices, failures, and runtime.
4.1. Implementation Details
- Implementation uses replaceable pretrained modules for video generation, depth estimation, 3D tracking, object grounding, grasp proposals, and deformable-object dynamics.The paper describes all modules as drop-in replaceable.Source: §4.1 · p. 6
4.2. Real-World Experiments and Evaluation Tasks
- Evaluation spans tabletop Franka and mobile Spot setups, with rigid, articulated, and deformable tasks including precision placements, drawer opening, and rope straightening.Source: §4.2 · pp. 6–7
4.3. Comparison with Baselines
- NovaFlow achieves the highest success rates among zero-shot methods and surpasses baselines trained with 10–30 demonstrations across tasks (10 trials per task).Source: Figure 5 · p. 7, §4.3 · p. 7
- AVDC and VidBot lack actionable 3D representation, while DP and IDM rely on small task-specific datasets; the paper argues these limits hinder zero-shot task solving.DP generalizes poorly from few examples in randomized evaluations, while IDM faces domain shift from generated videos.Source: §4.3 · p. 8
4.4. Ablation Studies and Failure Analysis
- Without a goal image, Wan2.1’s block-insertion performance drops substantially despite VLM rejection sampling; Veo remains robust and achieves high task success without one.Source: Table 1 · p. 9, §4.4 · p. 9
- Failures span video plausibility, tracking, grasping, and execution; grasp and execution failures are most frequent, making physical interaction the bottleneck.Source: §4.4 · pp. 8–9
4.5. Runtime Analysis
- A complete Veo flow takes around 2 minutes end-to-end, with video generation and 3D lifting dominating runtime; closed-source video models are faster but costlier.Source: §4.5 · p. 9
5. Conclusion
NovaFlow turns natural-language task commands into robot actions by distilling generated task videos into actionable 3D object flow. Real-world experiments span object types and embodiments, while failure analysis identifies physical execution as the primary bottleneck.
5. Conclusion
- NovaFlow distills generated task-solving videos into actionable 3D object flow, an intermediate representation separating task understanding from robot control.The framework translates natural-language commands into actions and handles rigid, articulated, and deformable objects across embodiments without task-specific training or demonstrations.Source: §5 · p. 9
- Real-world experiments show NovaFlow outperforms other zero-shot methods and imitation-learning policies trained on dozens of demonstrations.The reported experiments cover tabletop and mobile manipulator tasks involving rigid, articulated, and deformable objects.Source: §5 · p. 9
- Physical execution, particularly grasping and handling unexpected dynamics, is the primary bottleneck identified by the failure analysis.The authors identify closed-loop environmental feedback to refine or replan flow as a direction for future work.Source: §5 · p. 9
Appendices
The appendix provides additional details on video generation models, prompt engineering, and experimental prompts.
Appendices
- The appendix covers video-generation models, prompt-engineering techniques, and the prompts used in experiments.Source: A. Video Generation · p. 14
A.1. Wan2.1
The appendix describes video-generation settings, prompt extension, and task prompts paired with initial robot-view observations.
A.1. Wan2.1
- Wan2.1 is the open-source generator; its I2V mode handles standard generation and FLF2V goal-image conditioning, whereas Veo lacked goal-image conditioning during experiments.Wan2.1 videos contain 41 frames at 1280×720 and 16 FPS.Source: §A.1 · p. 14
A.3. Prompt Engineering
- Prompt extension enriches simple instructions with details about style, composition, and action; Wan2.1 uses an adapted extension script and Gemini 2.5 Pro, while Veo uses native prompt enhancement.Source: §A.3 · p. 14
A.4. Generation Prompts
- The appendix provides original and extended prompts for each task alongside the corresponding initial image from the robot’s perspective.Source: §A.4 · p. 14
B. Depth Estimation
Depth estimation is calibrated against the initial ground-truth depth map, and the resulting maps support precise manipulation. The section also describes 3D point tracking in XYZ space.
B. Depth Estimation
- Estimated depth maps are scaled using the ratio between the initial ground-truth median depth and the median depth of the first estimated frame.The initial ground-truth depth map serves as the calibration reference because estimated metric depth remains ambiguous after postprocessing.Source: B. Depth Estimation · p. 18
- Calibrated depth maps are reported as sufficiently accurate for block insertion requiring millimeter-level precision.The authors report improved consistency with ground-truth depth and temporal consistency after scaling.Source: B. Depth Estimation · p. 18
C. 3D Point Tracking
- TAPIP3D tracks points in XYZ 3D space, with query points sampled on the first frame using a uniform 32 × 32 grid.Source: C. 3D Point Tracking · p. 18
D. Object Grounding
NovaFlow grounds tracked points to the target object, filters generated-flow candidates for plausibility, and refines robot trajectories under motion and safety constraints.
D. Object Grounding
- Grounded-SAM2 combines Grounding DINO and SAM2 to mask the queried object, then retains tracking points visible throughout the video.The bounding-box and text thresholds are 0.25 and 0.3, respectively.Source: D. Object Grounding · p. 18
E. Rejection Sampling
- Gemini 2.5 Pro ranks projected flow images for continuity, natural motion, object identification, and task alignment, rejecting implausible candidates.The approach can generate eight candidates in parallel and select the best using eight H100 GPUs.Source: E. Rejection Sampling · pp. 18–19
F. Trajectory Optimization
- Trajectory optimization initializes joint configurations by interpolating between IK endpoints and solves for a smooth, collision-free path with endpoint constraints.Joint-limit and collision constraints are high-weight penalties, while start and end configurations are hard constraints.Source: F. Trajectory Optimization · p. 19
Objective and Penalty Terms
The objective balances smooth motion and proximity to a default rest pose, while penalizing joint-limit violations and unsafe obstacle distances.
Objective and Penalty Terms
- Smoothness penalizes squared joint velocities, approximated with finite differences and implemented as deviation from the previous configuration.A separate regularizer keeps the trajectory near the default home configuration q_rest.Source: Objective and Penalty Terms · p. 19
- Joint-limit penalties discourage configurations outside q_min and q_max, while collision penalties enforce a safety margin from obstacles using signed distance.The collision term applies a hinge loss to the signed distance of the robot’s swept volume.Source: Objective and Penalty Terms · p. 19
- The optimization weights are w_l=100.0 for joint limits, w_s=10.0 for smoothness, w_c=15.0 for collision, and w_r=0.1 for rest pose.The joint-limit weight is described as high enough to act as a hard constraint.Source: Objective and Penalty Terms · p. 19
G. Experiments
Real-world experiments demonstrate cross-embodiment manipulation across object types, with deployment on a new platform requiring hand-eye calibration.
G. Experiments
- NovaFlow manipulates rigid, deformable, and articulated objects with tabletop and mobile manipulators, including cross-embodiment use.The experiments include a tabletop robot and the Spot mobile robot.Source: G. Experiments · p. 20
- The system is viewpoint-agnostic and can be deployed on a novel platform after hand-eye calibration.Source: G. Experiments · p. 20