Source-linked AI summary
EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation
Gehao Zhang, Zhenyang Ni, Payal Mohapatra, Han Liu, Ruohan Zhang, Qi Zhu
TL;DR
VGMs provide rich motion priors for zero-shot manipulation but can hallucinate physically implausible behavior, while geometric retargeting adds cumulative errors. EmboAlign uses VLM-derived compositional constraints to select and optimize rollouts, improving success by 43.3% points over the strongest baseline across six real-robot tasks without task-specific training data.
Problem
VGMs can produce physically implausible rollouts, and geometric retargeting can introduce cumulative depth-estimation and keypoint-tracking errors that impair manipulation.
Method
EmboAlign uses a VLM to extract compositional constraints, then applies them to VGM rollout selection and constraint-based trajectory optimization.
Results
43.3% points higher overall success than the strongest baseline was achieved across six real-robot manipulation tasks without task-specific training data.
Takeaways & Limitations
EmboAlign aligns generative motion diversity with physical and relational constraints for precise, zero-shot robotic manipulation.
Abstract
from arXiv · showhide
Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos that capture rich object dynamics, offering a compelling foundation for zero-shot robotic manipulation. However, VGMs often produce physically implausible rollouts, and converting their pixel-space motion into robot actions through geometric retargeting further introduces cumulative errors from imperfect depth estimation and keypoint tracking. To address these challenges, we present \method{}, a data-free framework that aligns VGM outputs with compositional constraints generated by vision-language models (VLMs) at inference time. The key insight is that VLMs offer a capability complementary to VGMs: structured spatial reasoning that can identify the physical constraints critical to the success and safety of manipulation execution. Given a language instruction, \method{} uses a VLM to automatically extract a set of compositional constraints capturing task-specific requirements, which are then applied at two stages: (1) constraint-guided rollout selection, which scores and filters a batch of VGM rollouts to retain the most physically plausible candidate, and (2) constraint-based trajectory optimization, which uses the selected rollout as initialization and refines the robot trajectory under the same constraint set to correct retargeting errors. We evaluate \method{} on six real-robot manipulation tasks requiring precise, constraint-sensitive execution, improving the overall success rate by 43.3\% points over the strongest baseline without any task-specific training data.
I. INTRODUCTION
EmboAlign addresses physical hallucinations in video rollouts and retargeting errors by aligning VGM outputs with VLM-derived compositional constraints at planning and execution stages. It achieves higher real-robot manipulation success without task-specific training data.
- Motivation: VGMs can generate rich, temporally coherent motion priors, but physical hallucinations and retargeting errors can prevent correct robot actions.Failures include implausible interactions, prompt-following drift, imperfect depth estimation, and keypoint-tracking errors.
- Approach: EmboAlign automatically extracts task-specific physical and relational constraints from language instructions using a VLM.The constraints capture requirements such as placement alignment, top-down approach, and obstacle avoidance.
- Results: 43.3% points higher overall success than the strongest baseline was achieved across six real-robot manipulation tasks without task-specific training data.The tasks require precise, constraint-sensitive execution, including block stacking, tool use, and safety-constrained placement.
- Approach: The framework applies constraints first to rollout selection and then to trajectory optimization.This filters physically implausible VGM samples and corrects errors introduced when video motion is retargeted into robot actions.
II. RELATED WORK
Prior work uses video models to provide motion guidance and VLMs to formulate manipulation constraints, but constraint optimization can struggle with poor local optima. EmboAlign combines VGM rollouts with constrained optimization as trajectory initialization.
- Video-based manipulation: Video-based manipulation methods extract actionable motion from generative videos through inverse dynamics, correspondences, pose tracking, depth, or simulation.These approaches generally retain the pretrained VGM and convert its outputs into action-related signals.
- Compositional constraints: VLM-generated compositional constraints have been used for relational keypoint optimization and spatio-temporal failure monitoring.ReKep generates relational keypoint costs, while Code-as-Monitor evaluates constraint satisfaction reactively and proactively.
- Motivation for EmboAlign: Constraint-based optimization can reach poor local optima or fail entirely on complex, long-horizon tasks.EmboAlign addresses this limitation by using VGM-generated rollouts as trajectory initializations.
- EmboAlign: EmboAlign combines video-model motion diversity with the physical precision of constraint-based optimization.The pipeline uses a latent world model and compositional constraints to select a rollout before optimization.
III. METHOD
EmboAlign parses RGB–D observations and language into compositional constraints, samples VGM rollouts, selects a plausible constraint-consistent candidate, and optimizes its retargeted trajectory.
- Problem Formulation: The task is to produce a feasible 6-DoF end-effector trajectory from an initial RGB–D observation and language instruction.Sparse 3D keypoints represent task-relevant objects and encode spatial relations, contacts, and placement requirements.
- Constraint Generation: A VLM parses the observation and instruction into a set of compositional constraints over the keypoint configuration.Each constraint is a scalar-valued function whose nonpositive value indicates satisfaction.
- Rollout Selection: Candidate rollout videos are sampled from a pretrained VGM and evaluated for plausibility and constraint consistency.The selected rollout is chosen from samples generated conditionally on the observation and instruction.
- Trajectory Optimization: The selected rollout is retargeted into an initial end-effector trajectory and refined under the same constraint set.The optimization penalizes constraint violations while correcting retargeting errors.
B. Compositional Constraint Generation
EmboAlign represents manipulation requirements with compositional, keypoint-based constraints generated by a VLM and uses them as trajectory-refinement objectives.
- Keypoints provide an object-agnostic representation for expressing compositional manipulation constraints.
- The generated constraints serve as optimization objectives during trajectory refinement for six real-robot tasks.
- A VLM generates constraints from the rendered initial observation and language instruction.The resulting set includes task-specific requirements for the manipulation.
- Each constraint maps a 3D keypoint configuration to a scalar cost, with c(k) ≤0 indicating satisfaction.
- The constraint set can encode both goal-state conditions and process-level requirements.Examples include centering a block on a target and approaching an object from above.
C. Constraint-Guided Video Selection
Constraint-guided selection evaluates candidate VGM rollouts for both visual plausibility and task-level spatial satisfaction, prioritizing coherent candidates while enforcing constraints.
- A batch of N candidate rollouts is sampled from a pretrained VGM for each observation-instruction pair.
- Candidates are scored using visual plausibility and spatial constraint satisfaction as complementary selection criteria.
- Visual plausibility: V-JEPA-2 measures visual plausibility through divergence between predicted and observed latent representations.Lower s_vis indicates more predictable dynamics and fewer visual artifacts.
- Spatial constraint satisfaction: Spatial evaluation transforms tracked 2D keypoints and estimated depth into 3D trajectories for constraint checking.
- Joint selection: Candidates are ranked by ascending visual-plausibility score, then checked sequentially until one satisfies the spatial-cost threshold ε.This avoids reconstructing every candidate and prioritizes visually coherent rollouts.
D. Constraint-Based Trajectory Optimization
The selected rollout is retargeted into an end-effector trajectory and optimized under the same constraints to correct errors while preserving the video-derived motion prior.
- The selected rollout’s 3D keypoint trajectories are converted into an end-effector trajectory through grasp-conditioned retargeting.
- Retargeting: Stable grasp candidates define a gripper–object transform used for motion retargeting.The highest-scoring grasp determines T_grasp.
- Retargeting: Rigid transforms fitted to keypoint correspondences recover object poses, which induce end-effector poses under a fixed gripper–object transform.
- Trajectory optimization: Retargeting accumulates errors from imperfect depth estimation, keypoint tracking noise, and rigid-body fitting.
- Trajectory optimization: The nonlinear program penalizes constraint violations while keeping the optimized trajectory close to the VGM motion prior.λ controls the tradeoff between constraint satisfaction and fidelity to the rollout.
- The optimized trajectory is executed as a 6-DoF end-effector pose sequence.
IV. EXPERIMENTS
The experiments evaluate EmboAlign on six real-robot manipulation tasks requiring precise, constraint-sensitive execution and compare baselines, components, and failure modes.
- The evaluation covers six real-robot manipulation tasks requiring precise, constraint-sensitive execution.
- The study compares EmboAlign against constraint-only and video-only baselines.
- The experiments include ablations of key components and analysis of failure modes.
A. Tasks
EmboAlign is evaluated on six real-robot tasks spanning diverse objects and interaction modes, with success defined by task-specific physical outcomes.
- Experimental Setting: Experiments use a Dobot Nova2 robot and evaluate task performance in real-world scenes.
- Task Suite: The suite contains six tasks: opening a lid, stacking blocks, pressing a stapler, hammering a block, secure placement, and pouring water.The tasks cover separation, stable placement, tool use, displacement, obstacle avoidance, and pouring into a bowl.
- Success Criteria: Trials require precise outcomes, including stable stacking, target displacement, obstacle-free placement, or a visibly successful pour.
B. Quantitative Results
EmboAlign substantially outperforms both baselines across the six tasks by combining constraint-guided rollout selection with trajectory optimization.
- Overall Performance: 68.3% overall success exceeds 21.7% for ReKep and 25.0% for NovaFlow across six real-robot tasks.These are success rates based on 10 trials per task.
- Task-Level Gains: Press the Stapler reaches 8/10 successes, compared with 0/10 for NovaFlow and 2/10 for ReKep.The reported gains are 80% over NovaFlow and 60% over ReKep.
- Task-Level Gains: Place the Block Securely reaches 8/10 successes, compared with 4/10 for NovaFlow and 1/10 for ReKep.The reported gains are 40% over NovaFlow and 70% over ReKep.
- Complementarity: VGM rollouts provide warm-start trajectories for optimization, while compositional constraints filter implausible videos and refine retargeted actions.This combination addresses initialization sensitivity in constraint-only planning and physical implausibility in video-only pipelines.
- Constraint-Guided Selection: For the stack task, constraint checking rejects invalid behaviors and selects a candidate that passes all constraints for downstream retargeting and execution.
C. Qualitative Results
Qualitative analysis illustrates how constraint filtering distinguishes diverse rollout failures, while ablations and failure analysis examine the contributions and remaining weaknesses of the pipeline.
- Qualitative Selection: On block stacking, filtering identifies deformation, object disappearance, misplacement, and incorrect-object motion among rejected rollouts.
- Qualitative Selection: Only two of six stack-task videos satisfy all constraints with cost below threshold ϵ and proceed to downstream execution.
- Ablations: Ablations separately compare video generative models and pipeline variants to assess video proposals and constraint-based validation.
- Failure Modes: Among unsuccessful trials, video generation quality accounts for 31.57% of failures, followed by VLM keypoint referring at 26.31%.Retargeting failure accounts for 15.79%, while depth estimation accounts for 15.80%.
V. CONCLUSIONS
EmboAlign aligns video generative models with compositional physical constraints for zero-shot robotic manipulation. Its VLM-derived constraints are applied during rollout selection and trajectory optimization, achieving gains over video-only and constraint-only alternatives.
- EmboAlign aligns video generative models with compositional physical constraints for zero-shot robotic manipulation.
- EmboAlign is compared with constraint-only planning without VGM proposals and video-only variants without constraint filtering or trajectory refinement.
- The video-based ablation variants use LVP as the default video generative model.
- VLM-derived constraints are applied at rollout selection and trajectory optimization without modifying pretrained model weights.
- 68.3% average success rate is achieved across six real-robot manipulation tasks.