CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization

I. INTRODUCTION

• Robot-centric data collection is expensive, while in-the-wild human videos are often unstructured and unlike the robot’s setting.Synthetic videos can depict tasks in the robot’s current environment without requiring human demonstrations.Source: §I · p. 1
• The unified formulation treats video-derived motion as an initial guess and optimizes for task-critical features while enforcing semantic consistency and physical feasibility.The formalism also casts existing approaches as instances under simplifying assumptions.Source: §I · pp. 1–2
• CASTER extracts task-relevant features from generated videos and fine-tunes their trajectories in a digital twin to avoid unintended collisions and physically infeasible behavior.Its objective weights task-critical waypoints or features more heavily than prior approaches that match every waypoint.Source: §I · p. 1
• The paper unifies prior approaches as simplified instances, derives video-generation requirements, and develops methods that more closely approximate the underlying optimization.The proposed approach weights trajectory features rather than matching every waypoint.Source: §I · pp. 1–2
• Successful transfer requires videos that can be locally modified without losing task meaning; larger basins of attraction make tasks more suitable for this approach.A larger basin means more initial guesses can converge toward the same feasible equilibrium.Source: §I · p. 2

II. RELATED WORK

• Human-video methods recover trajectories or keyframes for robot actions [18]–[22], while other approaches learn policies conditioned on generated videos or recovered motion [27]–[29].The paper motivates human-video transfer in part because zero-shot generation of consistent robot videos remains challenging when robot behavior is sparse in video-model training data [16].Source: §II · pp. 2–3
• PhysWorld [4] uses reinforcement learning in a digital twin to obtain executable actions tracking generated object motions; the paper seeks a common framework to study prior approaches and their transfer to robots.Source: §II · p. 3

III. PROBLEM STATEMENT

• The setting is zero-shot tabletop manipulation: the robot executes a language-described task from an initial observation without task-specific demonstrations or additional real-world interaction.The generator is pretrained, inference-only, and treated as a black box that cannot be fine-tuned with robot data.Source: §III · p. 3
• A successful trajectory must satisfy the task-dependent semantic predicate and remain in the robot-and-environment feasible set, including reachability, limits, collision avoidance, and required contacts.Generated videos may suggest task-consistent motions that are infeasible for the robot, or plausible motions that do not accomplish the task.Source: §III · p. 3

IV. INTRODUCING A UNIFIED FORMALISM

• The optimization formalizes video-to-robot trajectory conversion, with video-derived object motion serving as an initial guess rather than a directly executable action sequence.Generated videos offer behavioral insight but do not directly provide executable robot actions.Source: §IV · p. 3, A. Learning from Generated Videos as… · pp. 3–4

A. Learning from Generated Videos as Constrained Optimization

• Robot actions are optimized to keep resulting trajectories semantically aligned with the task and physically feasible, while staying close to video-derived object-motion estimates.The formalism constrains semantic success and physical feasibility probabilistically; the physical feasible set is determined by robot embodiment, environment, and task interactions.Source: A. Learning from Generated Videos as… · pp. 3–4
• The cost compares video and real-world object motion through task features, which can represent object poses or relative trajectories.Using identity features attempts to recreate every object motion; task-critical features can instead discard incidental variation and noise.Source: A. Learning from Generated Videos as… · p. 4
• Semantic consistency is necessary because faithfully reproducing video object trajectories alone may still fail to complete the intended task.The semantic solution manifold contains all action trajectories that achieve the task, regardless of their motions and timings.Source: A. Learning from Generated Videos as… · p. 4
• The physical feasibility set is defined by individual constraints, which can be checked to determine whether a trajectory is feasible.A digital twin approximates real constraints; added safety margins account for simulator mismatch and can improve sim-to-real robustness.Source: A. Learning from Generated Videos as… · p. 4
• A stronger constraint in the digital twin can ensure real-world feasibility when the safety margin covers simulator-to-system mismatch over the relevant trajectory set.The margin δ_j requires additional tolerance in the digital twin; when the stated mismatch condition holds, satisfying the stronger simulated constraint also satisfies the original feasibility constraint.Source: A. Learning from Generated Videos as… · p. 4

B. Video Generation Requirements

• The video generator must seed optimization within a semantic basin where physical-feasibility constraints can refine trajectories without losing task alignment.The basin is a local region of trajectory space whose physically constrained optimization converges to motions satisfying the task success predicate.Source: B. Video Generation Requirements · p. 4
• The basin radius ρ measures how far generated features may deviate from the semantic feature manifold while physical-constraint enforcement preserves their semantic mode.Its value may depend on the task, initial scene, robot and environment, and enforcement mechanism.Source: B. Video Generation Requirements · p. 5
• The analysis assumes the defined basin is sufficient for the chosen optimization procedure to converge to task-completing trajectories.The basin is defined around features extracted from the synthetic-video object trajectory.Source: B. Video Generation Requirements · p. 5
• Seeding success depends on generated-feature error relative to the basin radius; systematic video-model hallucinations are allowed but can weaken the bound by increasing feature error.Reducing feature error or increasing the radius improves the stated probability condition.Source: B. Video Generation Requirements · p. 5
• Pick-and-place tasks are favorable because their motions can vary while preserving meaning, whereas contact-rich tasks such as pushing into a crowded box are unfavorable.The distinction corresponds to a larger versus smaller region of permissible semantic-preserving trajectory modification.Source: B. Video Generation Requirements · p. 5

C. Unifying Related Approaches

• Existing methods [3], [4], and [24] can be viewed as instances of the unified optimization under simplifying assumptions.The paper compares these assumptions with a closer implementation of the formalism in its experiments.Source: C. Unifying Related Approaches · p. 5
• PhysWorld [4] tracks a single target-object trajectory in a digital twin, corresponding to a more restrictive optimization cost.Related approaches [24] map generated rigid-object motion to end-effector poses and optimize with collision and joint-limit penalties.Source: C. Unifying Related Approaches · p. 5
• RIGVid [3] replays an extracted object trajectory using a rigid-grasp assumption and kinematic mapping, without explicitly enforcing physical feasibility.In the formalism, this corresponds to a specific tracking cost and omission of the physical-feasibility constraint.Source: C. Unifying Related Approaches · p. 5

V. APPROXIMATING THE UNDERLYING SOLUTION

• CASTER optimizes task features rather than exactly following an object's video trajectory, and uses multiple sampled videos to optimize over expected behaviors.Aggregated features define a shared target, while each sampled video initializes a separate optimization run.Source: §V · pp. 5–6
• A digital twin supports constrained trajectory optimization with collision-avoidance and kinematic constraints, producing actions intended for real-robot execution.The overview depicts synthetic-video object trajectories as initial guesses that optimization can locally modify without removing their semantic meaning.Source: §V · p. 5, Figure 2 · p. 6
• The resulting robot trajectory can be executed open-loop in the real world.The digital-twin optimization makes the trajectory executable by enforcing kinematic constraints.Source: §V · p. 6, Figure 2 · p. 6

A. Extracting the Task Features

• The system validates sampled videos with a VLM and discards those that do not appear to perform the requested task.Only retained videos are used to extract task features.Source: A. Extracting the Task Features · p. 6
• Object position, depth, and orientation are tracked in the synthetic videos to produce object trajectories.Depth is estimated per frame and aligned to observed depth; dense 3D surface points support orientation tracking.Source: A. Extracting the Task Features · p. 6
• The robot matches task-critical motion features rather than exactly reproducing every object trajectory.This feature-based objective focuses optimization on the parts of motion needed for the task.Source: A. Extracting the Task Features · p. 6
• An LLM identifies task-essential target-context object relations and the trajectory dimensions to track.For pouring coffee, the mug can be the target and the pot the context; tracking may be limited to selected axes.Source: A. Extracting the Task Features · p. 6
• Across retained videos, CASTER computes median positions, mean orientations, and variance for selected object relations and dimensions.Time-dependent weights decay exponentially with cross-video variance, emphasizing features consistent across demonstrations.Source: A. Extracting the Task Features · p. 6

B. Constrained Optimization in a Digital Twin

• Trajectory splines are optimized in a digital twin using the cross-entropy method and simulated rollouts evaluated by a feature-based cost.The robot’s end-effector trajectory is parameterized by designer-selected six-dimensional pose control points.Source: B. Constrained Optimization in a… · p. 6
• The constraints reject collisions with task-irrelevant objects above the maximum force threshold and joint values beyond kinematic limits.The experiments use Isaac Sim and reconstruct objects from the initial observation.Source: B. Constrained Optimization in a… · p. 6
• Figure 3 presents four tasks through generated-video examples, digital-twin rollouts, and real-world execution.The tasks are placing a banana in a pan, stacking bowls, pouring from a kettle, and toasting bread.Source: Figure 3 · p. 7
• The approximate optimization omits a direct semantic-consistency constraint because videos are preprocessed to show the intended task.Task consistency is expected to be preserved when those videos initialize the search within a semantic basin of attraction.Source: B. Constrained Optimization in a… · p. 7

VI. EXPERIMENTS

• The real-world evaluation tests whether the robot can perform 12 tasks on its first attempt, without real-world demonstrations or additional interactions.Tasks require semantic instruction following and multiple position, orientation, and collision-avoidance constraints.Source: §VI · p. 7
• Retarget maps synthetic-video object motion directly to the robot end effector, while Mimic optimizes a physically feasible trajectory to mimic the video’s object motions.These baselines represent common strategies for transferring generated motion.Source: §VI · p. 7, 1) Retarget · p. 7, 2) Mimic · p. 7
• The table reports zero-shot success percentages for real-world manipulation tasks, with Mimic and CASTER summarized by mean and standard error across seven video-initialized CEM runs per task.Retarget is reported as a success rate only.Source: Table I · p. 7
• The evaluation samples seven valid videos per task for real-world trials; each video initializes an optimization run for Mimic and CASTER.Retarget has seven trials per task, while Mimic and CASTER have 21; trials lacking sufficient valid videos or eligible feasible trajectories count as failures.Source: 2) Mimic · p. 7
• Synthetic videos are validated semantically, and the evaluation uses a randomly sampled set of seven retained videos for real-world assessment.Videos are retained after a validator score of at least 0.75; all retained videos contribute to feature extraction.Source: 2) Mimic · p. 7

SAMPLED VIDEOS ACROSS FOUR TASKS. SUCCESS REPORTS MEAN TASK

• CASTER achieved higher average zero-shot success than the baselines; feature-based multi-video objectives give the optimizer flexibility to match consistent waypoints.Mimic often fails to satisfy physical constraints while matching each waypoint, whereas Retarget transfers infeasible video motions directly to the robot.Source: SAMPLED VIDEOS ACROSS FOUR TASKS.… · p. 8
• Simulation-to-reality physics discrepancies limited performance on pouring-kettle and covering-bucket tasks.The kettle rotates in the gripper when its handle is squeezed, and the reconstructed bucket mesh omits rotational handle joints.Source: SAMPLED VIDEOS ACROSS FOUR TASKS.… · p. 8
• The number of semantically valid videos alone does not explain downstream success; task-relevant feature error relative to the semantic basin may matter instead.The basin’s extent depends on task difficulty and robot constraints.Source: SAMPLED VIDEOS ACROSS FOUR TASKS.… · p. 8

VII. CONCLUSION

• 76.6% average zero-shot task success across 12 real-world tasks, exceeding the best-performing baseline by 34.1%.The conclusion reports this as CASTER’s overall result.Source: §VII · p. 8
• The unified formalism treats video-to-robot transfer as constrained optimization and frames existing approaches as simplified instances of that problem.CASTER removes simplifying assumptions to more closely match the formalism.Source: §VII · p. 8

A. Video Generation

• HunyuanVideo-1.5 videos use the 480p_i2v_step_distilled checkpoint, with 20 task-conditioned samples from seeds 0–19 and no generator fine-tuning.The videos are conditioned on the initial scene image and a task-specific prompt.Source: A. Video Generation · p. 9

B. Simulation Setup

• The digital twin uses Isaac Sim 6.0.1 with Isaac Lab and maps Cartesian poses to joint targets through damped least-squares differential inverse kinematics.Source: B. Simulation Setup · p. 9, Table IV · p. 9

C. Cross-Entropy Optimization

• End-effector position and relative rotation are parameterized with natural cubic splines using uniformly spaced knots; the first position and rotation knots are fixed.The original configuration uses (Kpos, Krot) = (15, 5), leaving D = 3(Kpos − 1) + 3(Krot − 1) free parameters.Source: C. Cross-Entropy Optimization · p. 9
• Both methods use 1,024 candidates for up to 10 generations, evaluate parallel physics rollouts, and select feasible low-cost trajectories for real-world evaluation.Mimic scores absolute object-trajectory agreement, while CASTER scores the shared task-feature reference; the final selection randomly samples three trajectories from the lowest-cost quartile.Source: C. Cross-Entropy Optimization · pp. 9–10
• CEM samples and clips trajectory parameters, updates its distribution from the lowest-cost 10% of candidates, and retains the best candidate found.Position offsets and rotation-vector components are bounded by ±0.40 m and ±π rad, with bounds expanded as needed to contain the fitted initialization.Source: C. Cross-Entropy Optimization · pp. 9–10

• 70.4% of generated videos pass the VLM validator on average, with lower validation rates on tasks requiring precise semantic alignment.The cited examples are “pile up blocks” and “toast bread.”Source: APPENDIX II ADDITIONAL RESULTS · p. 10, Figure 4 · p. 10
• CASTER produces a higher fraction of feasible candidates than Mimic on most tasks when compared on matched demonstrations.Feasibility means collision-free motion within workspace limits; it does not establish successful task completion.Source: Figure 4 · p. 10, APPENDIX II ADDITIONAL RESULTS · p. 10
• For three representative tasks, Fig. 5 shows feasible end-effector trajectories in X–Y and X–Z planes, with darker colors indicating lower optimization cost.The examples suggest CASTER explores a broader range of feasible motions than Mimic while preserving key task features.Source: Figure 5 · p. 10, APPENDIX II ADDITIONAL RESULTS · p. 10