Source-linked AI summary
Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers
Tianyue Wu, Boyuan An, Shuqi Zhao, Heyu Guo, Wanli Xing, Yi Ma, Kaifeng Zhang, Ruihai Wu, Masayoshi Tomizuka
TL;DR
Reliable reference motions are a bottleneck for scaling simulation-based hand–object tracking from generated videos. GALATEA reconstructs generated videos into 3-D trajectories, trains a unified multi-object, multi-trajectory tracker with simulation-based RL, and executes video plans in the real world. It grounds more than 1,500 trajectories in simulation, outperforms baselines by over 25 percentage points in success rate, and demonstrates diverse closed-loop manipulation.
Problem
Visual videos provide diverse manipulation behavior but do not directly reveal 3-D hand–object states, contacts, forces, or actions executable by a particular robot.
Method
GALATEA reconstructs 3-D hand–object trajectories from generated videos, then uses tracking-style RL and policy distillation in simulation to learn a unified controller.
Results
More than 1,500 generated-video trajectories are successfully grounded in simulation, with success rates over 25 percentage points higher than baselines.
Takeaways & Limitations
Closed-loop real-world experiments achieve diverse grasps, non-prehensile manipulation, and post-grasp object-pose adjustment on trained and unseen video plans.
Abstract
from arXiv · showhide
Generated hand-object interaction (HOI) videos provide a controllable way to propose manipulation motions. Simulation-based HOI tracking can translate such kinematic references into feasible low-level control, but its scalability is limited by the lack of reliable reference motions. We therefore combine generated videos with simulation-based HOI grounding: during training, generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, and at deployment, the video model produces motion plans that are executed by the learned tracker. In particular, we propose a method that enables scalable reference generation by HOI reconstruction with minimal manual intervention and successfully grounds more than 1,500 generated videos in simulation, achieving success rates over 25 percentage points higher than those of baselines during simulation-based training. In real-world closed-loop experiments, it achieves diverse grasps, including functional grasps, non-prehensile manipulation, and post-grasp object-pose tracking. Videos and code are available at https://boyuan-an.github.io/GALATEA/.
I. INTRODUCTION
GALATEA addresses the difficulty of turning diverse, uncontrolled visual demonstrations into executable dexterous manipulation by combining generated HOI videos with simulation-trained tracking. It reconstructs scalable motion references, trains a unified controller, and demonstrates diverse real-world behaviors.
- Motivation: In-the-wild videos offer diverse hand behaviors but obscure 3-D states, contacts, forces, and robot-executable actions amid uncontrolled variation.Extracting useful references therefore requires substantial human filtering and annotation.
- Motivation: Generated video models provide controllable manipulation references, while simulation supplies dense supervision for converting kinematic references into dynamically feasible actions.Conditioning can specify objects, behaviors, and scenes while suppressing irrelevant variation.
- Approach: GALATEA reconstructs 3-D hand–object trajectories from generated videos, then uses tracking-style RL and policy distillation to learn a unified controller.Its reconstruction combines foundation-model perception, stereo initialization, and joint hand–object optimization.
- Results: More than 1,500 reconstructed trajectories are grounded in simulation, with the RL recipe outperforming baselines by over 25 percentage points in success rate.The system reconstructs about 2,000 usable trajectories from 2,500 generated clips.
- Results: Closed-loop real-world experiments demonstrate diverse grasps, pushing and pulling, and post-grasp object-pose adjustment on trained and unseen video plans.The reported behaviors include functional grasps and non-prehensile manipulation.
II. RELATED WORK
Prior work connects generated videos to robot actions either through action-generation models or structured motion references, while dexterous-control methods vary in the quality and scale of their kinematic supervision. GALATEA reconstructs generated videos into trajectories and learns a unified simulation-trained controller from them.
- Generated-video manipulation: Some methods translate generated visual futures into actions through inverse dynamics or video-conditioned policies, while others jointly model visual futures and actions.These action components typically require embodiment-specific robot demonstrations.
- Generated-video manipulation: Methods with an explicit intermediate interface convert generated videos into structured motion references for downstream control.LVP is described as retargeting generated wrist and hand kinematics.
- Kinematic demonstrations: Kinematic-supervision methods use either curated motion-capture trajectories or noisier references reconstructed from one or a few human videos.The latter trajectories are harder to solve reliably at scale.
- GALATEA: GALATEA generates videos, reconstructs 3-D hand–object trajectories, and applies tracking-style RL with policy distillation to learn a unified controller in simulation.This pipeline uses generated-video references rather than relying solely on curated demonstrations.
- HOI reconstruction: The reconstruction pipeline uses first-frame real-image conditioning, metric depth alignment, object masks, initial motion estimates, and joint optimization to obtain HOI references.The method uses a real first-frame RGB image and language instruction rather than simulation-rendered conditioning images.
2) HOI Reconstruction:
The method reconstructs aligned hand meshes and 6-DoF object trajectories from video by combining metric depth, initial pose estimation, and joint hand–object refinement. Its objective uses silhouette, keypoint, contact, and temporal constraints to produce physically coherent motion references.
- HOI Reconstruction: Aligned MANO hand meshes and 6-DoF object trajectories are reconstructed by independently estimating both motions and then jointly refining them.This reconstruction is described as crucial for bootstrap learning.
- Metric Depth Estimation: Metric depth is corrected against first-frame stereo depth using background pixels, anchoring each frame to a metric scale and offset before object tracking.The correction fits a scaled and shifted predicted depth map under a fixed-camera, static-background assumption.
- Initial Motion Estimation: FoundationPose and WiLoR provide initial object poses and MANO hand estimates that are subsequently optimized with differentiable silhouette and reprojection constraints.Silhouette masks constrain lateral translation, rotation, and depth when mesh dimensions and camera intrinsics are known.
- Joint Hand–Object Optimization: The optimization combines projection, keypoint, contact, and temporal losses, with contact activated on detected contact frames.The contact term draws the hand toward the object surface, while temporal terms penalize velocity and acceleration.
B. Grounding Generated Videos in Simulation with HOI Tracking
The system uses a simulated Sharpa Wave Hand mounted on a Franka Research 3 arm, with PhysX simulation in Isaac Gym.
- Simulation Setup: The simulated platform uses a Sharpa Wave Hand mounted on a Franka Research 3 arm.The hardware is simulated rather than described as a physical deployment in this passage.
- Simulation Setup: PhysX provides the simulation engine within Isaac Gym.
- Simulation Setup: The setup combines the hand, arm, and simulator into a platform for simulated manipulation experiments.
1) Reference Augmentation and Preprocessing:
Reference augmentation samples variations in hand approach and post-contact object motion, then screens candidates for execution feasibility. The construction preserves hand–object relative pose after contact while constraining perturbations to 30% of source ranges.
- Reference Augmentation: Each source trajectory is augmented with 5 sampled variations in hand approach and post-contact object motion.
- Reference Augmentation: Before contact, perturbations fade from full strength to zero; after contact, they grow from zero to full strength over the remaining trajectory.The schedules are defined by α_t and β_t across first contact t_c and final frame T.
- Preprocessing: Perturbations are set to 30% of the source clearance and motion range, while sharing the post-contact transform preserves the hand–object relative pose.Augmented candidates are screened for ease of execution using inverse kinematics and trajectory slowing.
2) RL Formulation and Training Recipe:
The tracker uses asymmetric actor–critic RL with observations centered on hand–object tracking errors and actions smoothed before 30-Hz position control. Its reward combines imitation, contact, lift, and regularization terms, while SAPG distributes exploration across PPO agents.
- Observation and control: The training recipe uses asymmetric actor–critic learning to combine deployable observations with simulator-only privileged information during policy optimization.The actor and critic therefore operate with different information available during training.
- Observation and control: The actor observes wrist, MANO-keypoint, object-pose, fingertip-distance, and BPS shape features, while the critic additionally receives privileged simulator state.The actor receives no joint velocity, force, mass, center of mass, or retargeted hand-joint target.
- Observation and control: The policy outputs 29-dimensional arm and hand actions, mapping arm deltas and hand targets through joint limits before exponential smoothing.Smoothed targets are sent to 30-Hz position PD control.
- Reward: The reward tracks demonstrations through wrist, object, and hand imitation terms while adding contact, lift, action-smoothness, and effort regularization.Contact gates activate object tracking and can suppress dense hand tracking when the reference lift is not reproduced.
- Policy optimization: SAPG partitions parallel rollouts across PPO agents with different exploration settings and aggregates their experience into a shared policy update.This exploration mechanism is paired with multi-skill expert training under limited compute.
3) Policy Distillation:
Because full-reference training is difficult under limited compute and per-trajectory training scales poorly, the method trains a small number of multi-skill experts per object category.
- Expert training: Each object category uses at most two multi-skill experts, each covering about 40 source trajectories and roughly 200 trajectories after augmentation.Experts are trained from scratch on category trajectories or fine-tuned from the full multi-object set when that is more effective.
- Expert training: The strategy addresses limited compute while avoiding the poor scalability of training separately on every trajectory.Direct training on the full reference set remains difficult, whereas per-trajectory training scales poorly.
A. Ablation and Benchmark for HOI Reconstruction
The reconstruction evaluation combines blind human ranking on generated videos with quantitative metrics on annotated RGB-D clips, plus ablations and baseline comparisons. Ablations indicate that joint optimization improves relative hand–object geometry, while the full method maintains stronger overall quality than baselines.
- Evaluation setup: 480 judgments per method compare anonymized outputs from the method and three baselines across 120 generated clips.Four reviewers rank the outputs, with ties allowed and randomized method order.
- Evaluation setup: The quantitative evaluation uses 40 fixed-view RGB-D clips each from H2O and HO-Cap, comparing the full method, ablations, and baselines with three metrics.The metrics are ADD-S, MRRPEro, and CDev, with Avg. Rank aggregating their ranks.
- Metrics: ADD-S measures closest-point object alignment, MRRPEro measures wrist-to-object-center error, and CDev measures preservation of ground-truth contacts.ADD-S is reported in centimeters, while MRRPEro and CDev are reported in millimeters.
- Ablations: Without depth alignment, the method still achieves better Avg. Rank than every baseline on H2O at 2.33 and HO-Cap at 2.67.Its main degradation relative to the full method is in ADD-S, particularly on HO-Cap.
- Ablations: Joint optimization markedly improves MRRPEro and CDev on both datasets while preserving absolute object placement.The result supports better relative hand–object geometry without attributing the overall gain solely to the measured depth anchor.
B. Ablation and Benchmark for RL Formulation and Opti-
The benchmark evaluates success and tracking across objects, references, and methods, showing that the proposed formulation performs strongly and transfers beyond training conditions. Ablations indicate that SAPG, lift-aware rewards, and joint training are important to performance.
- Evaluation protocol: Success is measured over 20,000 rollouts per method–object experiment using final-frame position, rotation, and hand-keypoint thresholds.Tracking errors are averaged over intent-executing segments after contact and lift, or after first contact for pushing and pulling.
- Benchmark results: 77.4% mean and 81.0% median success make Ours (experts) the strongest method on the 10-object benchmark.Across all 42 training objects, expert policies reach 78.6% macro mean and 80.5% median success.
- Baseline comparison: 13.2 points average improvement over ManipTrans occurs across every benchmark object for the two-stage variant.The comparison uses the same benchmark objects, although ManipTrans was developed with cleaner motion-capture data.
- Ablations: Replacing SAPG with PPO, removing the lift reward, or using two-stage training reduces success on 10/10, 7/10, and 10/10 objects, respectively.The authors hypothesize that noisy reconstructed hand–object relations can make the first stage impose suboptimal trajectories that restrict later exploration.
- Tracking errors: The method achieves the lowest object-position and hand-keypoint tracking errors among arm–hand methods.DO AS I DO reports lower rotation error, but many pose-adjustment trajectories are excluded because they fail the intent-executing criteria.
C. Real-World Experiments
Real-world experiments use onboard object perception and execute generated references with both hand and object motion retained. The system demonstrates diverse manipulation but remains constrained by contact transitions, difficult grasps, and limited in-hand coverage.
- Reference design: Retaining both hand and object motion in references supports grasp choices that are part of the interaction specified by the video plan.Object-only references do not preserve the hand behavior that determines the grasp.
- Failure modes: Contact transitions are sensitive to the sim-to-real gap, causing large tracking deviations after the controller leaves the reference.The controller shows basic recovery and retry behavior but often fails after departing from the reference.
- Scope boundary: The skill set does not systematically cover in-hand manipulation because such motion appears only incidentally and reconstruction noise obscures subtle finger–object movement.The authors propose combining multiple data sources in a unified controller for everyday manipulation of single rigid objects.