Source-linked AI summary
KinoGen: Customizable Humanoid Loco-Manipulation References via Kinematically Grounded Video Generation
Qi Wu, Lulu Zhao, Sanghyun Hahn, Rishi Yennu, Emmy Yang, Dingqi Zhang, Tao Pang, Jiuguang Wang, Kuan Fang
TL;DR
Loco-manipulation references must coordinate body, hand, and object motion while fitting the target robot and task, needs that existing video pipelines do not fully meet. KinoGen conditions video generation on calibrated simulator renders and reconstructs trajectories using the same models. Across eight tasks it reports lower penetration and foot skating than video-based baselines and over 90% contact preservation.
Problem
Existing video-based humanoid pipelines do not directly recover complete whole-body, hand, and object references for the target robot and task, and open-loop generation cannot be revised to satisfy a prescribed configuration.
Source: §I · p. 1Method
KinoGen conditions a pretrained video generator on calibrated task renders, reconciles generated views with body, hand, and object priors, and fits the same robot and object models to recover and customize trajectories.
Source: §I · p. 2, B. Reconciling Video and Motion Priors · p. 4, C. Cross-View Reconstruction of… · p. 4Results
Across eight tasks, KinoGen yields the lowest penetration and foot-skating velocity, recovers 91.16% of annotated contact frames versus 69.40% for the strongest baseline, and trains policies with the highest success rate.
Source: §I · p. 2, B. Comparative Results · p. 7Takeaways & Limitations
KinoGen references support tracking-policy learning on a Unitree G1, and edited references satisfy every prescribed target with substantially less penetration than direct interpolation.
Source: §I · p. 2Reconstruction is vulnerable to severe or persistent occlusion, recovered reference quality and diversity are bounded by the video generator, and sparse keyframes leave intervening motion to the generator.
Source: §V · p. 8
Abstract
from arXiv · showhide
We present KinoGen, a framework that synthesizes customizable humanoid loco-manipulation references through kinematically grounded video generation. KinoGen conditions a pretrained video generator on calibrated renders of the target robot and object in a simulated task instance and recovers the reference by fitting the same models to the generated two-view video, so that the reference lies in the robot configuration space by construction without a human body proxy or retargeting. Body, hand, and object priors initialize the reconstruction, and direct optimization against cross-view evidence refines it. Since an edited keyframe is simply another rendered state, a user can prescribe a goal object pose, regenerate the video around the edited keyframe, and obtain a reference that satisfies the pose exactly. Across eight loco-manipulation tasks, KinoGen yields lower penetration and foot skating than video-based baselines and preserves over 90% of annotated contacts, and its customized references satisfy every prescribed target with substantially less penetration than direct interpolation. Tracking policies trained on the synthesized references are deployed on a Unitree G1 humanoid with dexterous hands. Project webpage: https://kinogen-humanoid.github.io/.
I. INTRODUCTION
KinoGen addresses the difficulty of obtaining synchronized whole-body, hand, and object references that fit a target robot and task, without costly demonstrations or post-generation retargeting. It couples video generation with simulator rendering and reconstruction, and reports improved reference quality and successful physical-robot tracking.
Motivation
- Loco-manipulation references must synchronize body, hand, and object trajectories while matching the target robot’s kinematics, task geometry, and sometimes a prescribed object pose.Human demonstrations require reconstruction and retargeting, which introduce embodiment errors, while teleoperation requires substantial effort to cover task variation.Source: §I · p. 1
- Prior video pipelines use human proxies or robot-specific recovery, may omit hands and objects, and do not revise open-loop generation to meet metric configurations.Human-proxy approaches can leave penetration or drift because robot kinematics and task-instance constraints are satisfied only after generation.Source: §I · p. 1
KinoGen
- KinoGen’s kinematic grounding conditions generation on the target task render and fits the same robot and object models to generated video, recovering complete references in their configuration spaces without retargeting.The same interface renders an edited simulated keyframe for regeneration and enforces that correction during reconstruction.Source: §I · pp. 1–2
Evaluation overview
- Across eight tasks, KinoGen has the lowest penetration and foot skating, preserves over 90% of annotated contacts, and trains policies with the highest success rate.Customized references satisfy every prescribed target with substantially less penetration than direct interpolation; policies execute four tasks on the physical robot.Source: §I · p. 2
II. RELATED WORK
Prior humanoid video pipelines recover incomplete or retargeted motion, while editing approaches do not close the loop by enforcing an edited reconstructed state. KinoGen couples a video generator and physics simulator to synthesize and customize coordinated robot–object references.
II. RELATED WORK
- Prior humanoid-video methods retarget proxy interactions or use human-motion retargeting, pose detectors, or sparse point guidance, and do not recover complete robot–object trajectories.KinoGen instead conditions on calibrated simulator renders and fits those same models to multi-view evidence, without a human proxy or separately trained robot-specific pose detector.Source: §II · p. 2
- Prior editing methods control generated motion or ground fixed videos downstream, but do not regenerate from an edited reconstructed robot–object state and enforce that state exactly.KinoGen renders the edited keyframe for keyframe-conditioned regeneration and enforces the complete keyframe as a hard reconstruction constraint.Source: §II · p. 2
III. METHOD
- KinoGen couples a generator’s interaction-behavior prior with simulator constraints on embodiment and task geometry to synthesize customizable loco-manipulation references.The pipeline conditions on calibrated task-instance renders, reconstructs with the same models, and can enforce an edited robot–object keyframe.Source: §III · p. 2, Figure 2 · p. 3
- The task is to synthesize a simulator-world robot–object trajectory, with robot configuration including floating-base pose, body joints, and finger coordinates, and object pose in SE(3).A user may additionally prescribe a goal object pose at a frame that the corrected reference must satisfy exactly.Source: §III · p. 2
- The pipeline’s four stages ground generation, reconcile video with motion priors, fit robot and object models to both views, and enforce edited keyframes during reconstruction.References serve as targets for tracking policies trained under simulated physics before deployment.Source: §III · p. 3
A. Video Generation with Kinematic Grounding
KinoGen grounds video generation by rendering the initial task state from two calibrated simulator views. This conditions synchronized generation on the target robot, scene, object, and camera viewpoints, providing shared metric evidence for reconstruction.
Conditioning on the rendered initial state
- Rendering the initial robot–object state preserves its geometry, configuration, scene layout, and camera viewpoint, placing generated behavior in the target embodiment and spatial setting.The render provides a shared metric reference, and the same camera-rendering operator is used during reconstruction.Source: A. Video Generation with Kinematic… · p. 3
Cross-view consistency
- Two calibrated initial-state views are concatenated into one conditioning image so the generated clip provides synchronized multi-view evidence for reconstruction.Clips with large cross-view inconsistencies, such as different moving limbs, are rejected and regenerated with another random seed.Source: A. Video Generation with Kinematic… · p. 3
Appearance editing
- Only hand surfaces are recolored to suit the appearance expected by pretrained video and hand models, while preserving hand geometry and leaving other robot surfaces unchanged.The simulator render remains ground truth for reconstruction masks and robot surface points.Source: A. Video Generation with Kinematic… · p. 3
Generation
- A pretrained generator produces a five-to-ten-second, 24-fps clip from the conditioning image and a prompt specifying the task, static cameras, and cross-view consistency.The synchronized two-view clip is passed to observation extraction and reconstruction initialization.Source: A. Video Generation with Kinematic… · p. 3
B. Reconciling Video and Motion Priors
KinoGen turns generated two-view video into calibrated observations and plausible initial trajectories by reconciling visual evidence with body, hand, and object priors. These initializations feed cross-view reconstruction.
Calibrated visual evidence
- Video-derived masks, robot surface tracks, and metric depth link each frame and view to known robot, object, and camera geometry, while priors address ambiguities that visual fitting alone cannot resolve.Framewise fitting can yield incoherent motion, masks weakly constrain hands, and hand–object occlusion makes direct object fitting sensitive to initialization.Source: B. Reconciling Video and Motion Priors · pp. 3–4
Whole-body motion prior
- The body trajectory is initialized with a task-conditioned G1 motion prior, with multi-view observation loss optimizing its initial noise while keeping it close to a valid sample.The procedure runs 200 iterations and samples 12 frames per iteration; its observation loss evaluates silhouette, depth, and ground terms.Source: B. Reconciling Video and Motion Priors · p. 4
Hand and object pose priors
- WiLoR initializes each hand’s wrist orientation and finger articulation, while palm position is triangulated across views; rule-based mappings initialize the robot hand’s coupled coordinates.The hand estimate is the only pipeline quantity obtained with a human body model, and it is used only for initialization before direct fitting refines the hand.Source: B. Reconciling Video and Motion Priors · p. 4
- FoundationPose runs independently on each view to initialize object pose, and the candidate with the best cross-view agreement is selected.These hand and object initializations, together with the body prior, feed cross-view reconstruction.Source: B. Reconciling Video and Motion Priors · p. 4
C. Cross-View Reconstruction of Robot-Object Interaction
KinoGen reconstructs robot, hands, and object by fitting their models to evidence from both video views. Sequential solves combine visual observations with priors and temporal regularization, producing a complete reference in robot configuration and object pose spaces.
C. Cross-View Reconstruction of Robot-Object Interaction
- The complete reference is recovered by directly fitting robot and object models to both views, with each image term comparing rendered estimates against masks, tracks, and depth.The recovered reference lies in robot configuration and object pose spaces by construction.Source: C. Cross-View Reconstruction of… · p. 4
- Hands are first reconstructed as floating bases, then their wrist targets and fixed finger coordinates guide whole-body reconstruction.Multi-view hand masks are stabilized by hand-prior and temporal terms when evidence is weak or occluded.Source: C. Cross-View Reconstruction of… · pp. 4–5
- Whole-body fitting combines silhouette, surface-point track, and depth evidence with ground, joint-limit, temporal, and body-prior regularization.The wrist loss weights hand-pose targets by multi-view observation reliability.Source: C. Cross-View Reconstruction of… · p. 5
- The object trajectory is reconstructed last from multi-view silhouette and depth evidence, with the initial object pose fixed and velocity and acceleration regularization.The robot trajectory remains fixed during this solve; the final robot-object reference is not jointly refined afterward.Source: C. Cross-View Reconstruction of… · p. 5
D. Reference Correction via Keyframe Editing
Reference customization edits a robot-object keyframe for a prescribed object pose, regenerates task-conditioned motion around it, and reconstructs the result with the edited configuration fixed. This yields the prescribed pose exactly, but does not separately guarantee contact or physical executability.
D. Reference Correction via Keyframe Editing
- A prescribed object pose at frame k is completed into a robot-object keyframe because the goal pose alone leaves the whole-body configuration unspecified.The grasping-hand wrist pose is adjusted while preserving the hand-object relative transform and finger coordinates; weighted inverse kinematics gives greater preservation weight to the free hand and both feet.Source: D. Reference Correction via Keyframe… · p. 5
- The edited keyframe is rendered back into the generator to synthesize intervening task-conditioned motion, avoiding reliance on direct state interpolation that may cross obstacles or support surfaces.Regeneration uses the original initial and last frames, the edited frame, and the task prompt to preserve task phase and timing in the experiments.Source: D. Reference Correction via Keyframe… · p. 5
- Reconstruction of the regenerated clip fixes the edited configuration at frame k, enforcing the prescribed configuration exactly while smoothing neighboring states with temporal and object-motion losses.The process can be repeated with different edits to produce multiple customized references from one grounded clip.Source: D. Reference Correction via Keyframe… · p. 5
- The hard kinematic constraint guarantees the prescribed configuration, but does not separately guarantee contact or physical executability.Source: D. Reference Correction via Keyframe… · p. 5
IV. EXPERIMENTS
The experiments ask whether grounded generation improves reference quality, which components support reconstruction, and whether keyframe customization can meet a prescribed configuration while avoiding interpolation penetrations. Figure 3 presents synthesized robot-object trajectories across several tasks, while Table I is identified as a quantitative comparison of reference quality and downstream RL performance.
IV. EXPERIMENTS
- The experiments evaluate whether grounded video generation produces higher-quality references than generated-human reconstruction and whether customization avoids penetrations from direct interpolation.Source: §IV · p. 5
- The component study asks which elements are necessary for accurate and coherent robot-object reconstruction.Source: §IV · p. 5
- Figure 3 shows temporally ordered keyframes from generated and reconstructed trajectories for box and dumbbell lifting, mopping, stool sitting, kneeling and reaching, and basket lifting; Table I compares reference quality and downstream RL performance.Source: Figure 3 · p. 6, Table I · p. 6
A. Experiment Setup
The evaluation uses eight simulated loco-manipulation tasks for a Unitree G1 with dexterous hands and trains tracking policies from each reference source under matched training conditions. Four trained policies are also qualitatively deployed on hardware.
A. Experiment Setup
- The eight-task suite spans sitting, kneeling, mopping, and lifting four object types, testing varied postures, contacts, object geometries, and manipulation goals on a Unitree G1 with dexterous hands.The robot uses two 6-DoF OYMotion ROHand hands, and scenes are rendered in Isaac Sim.Source: A. Experiment Setup · p. 6
- Each reference source trains a state-based tracking policy with the same architecture, observations, reward, simulator settings, procedure, and budget; only the reference trajectories differ.Training uses 8,192 parallel environments, 6,000 PPO updates, and 10 seeds; success requires no fall and position tracking error below 0.2 m.Source: A. Experiment Setup · p. 6
- Policies are qualitatively deployed on hardware for stool sitting, chair pickup, basketball pickup, and pickup from a kneeling posture.Deployment policies use robot proprioception and the initial object pose; that pose is reused from simulation when the object is initially invisible.Source: A. Experiment Setup · p. 6
B. Comparative Results
KinoGen reconstructs complete robot, hand, and object trajectories directly from calibrated multi-view evidence, and its references improve physical consistency and downstream policy learning over video-based baselines.
B. Comparative Results
- The comparison preserves scene and task content while contrasting human-retargeting pipelines with direct target-robot reconstruction using calibrated multi-view evidence.The reported metrics include penetration, foot skating, contact preservation, and root-acceleration smoothness.Source: B. Comparative Results · pp. 6–7
- KinoGen preserves 91.16% of annotated contact frames, versus 69.40% for the strongest baseline, while achieving the lowest total penetration and foot-skating velocity.GRAIL has the lowest root acceleration, but KinoGen is smoother than GenMimic and GenHOI while providing better contact preservation and physical feasibility.Source: B. Comparative Results · p. 7
- KinoGen preserves grasp and ground contact qualitatively, while baselines show penetration, floating, or misalignment.Source: Figure 5 · p. 7
- Policies trained on KinoGen references achieve the highest success rate, lowest reference deviation, and largest learning-curve AUC.These results indicate the references are useful for policy learning as well as geometrically coherent.Source: B. Comparative Results · p. 7
C. Ablation Study
The ablations show that calibrated views, point tracks, and motion priors contribute complementary evidence for reconstruction. In reference correction, regeneration around a metric keyframe satisfies the target while avoiding the table-crossing motion of direct interpolation.
C. Ablation Study
- Using only one view reduces worst-view IoU and increases EPE; point tracks reduce EPE while leaving silhouette overlap nearly unchanged.The second view helps resolve ambiguity, while point tracks constrain internal articulation that silhouettes cannot observe.Source: C. Ablation Study · p. 7
- Without the body prior, root acceleration rises from 0.37 to 1.01 m/s2 and penetration rises from 10.5 to 26.8 mm.Removing the hand prior also increases acceleration and penetration, as inaccurate hand targets propagate to the arm and body during refinement.Source: C. Ablation Study · p. 7
- Together, multi-view evidence locates motion, point tracks inform internal articulation, and motion priors maintain consistency with common robot motions.Source: C. Ablation Study · pp. 7–8
D. Reference Correction
- The full correction method satisfies every target exactly and significantly reduces penetration compared with direct end-effector interpolation.Interpolation also reaches every target, but its endpoint constraint does not specify a feasible approach; the full method regenerates around the metric keyframe before constrained reconstruction.Source: D. Reference Correction · p. 8, Table III · p. 8
- Without correction, placement error ranges from 1.07 to 14.55 cm across the three regions; direct interpolation reaches the target but moves through the table.The comparison samples ten targets per region and uses the same original generation for variants associated with each target.Source: D. Reference Correction · p. 8
V. CONCLUSION
KinoGen uses the same target robot and object models to condition video generation and reconstruct whole-body, dexterous-hand, and object trajectories directly in their configuration spaces. The conclusion also identifies occlusion, video-generator capability, and sparse keyframe steering as limits on the current design.
V. CONCLUSION
- Calibrated multi-view evidence and motion priors reconcile generated behavior with the target embodiment and task instance, producing references that support tracking on a Unitree G1.Users can edit robot–object keyframes, with video generation interpolating behavior and constrained reconstruction enforcing prescribed configurations with less penetration than direct interpolation.Source: §V · p. 8
- Severe or persistent occlusion limits reconstruction; reference quality and diversity are bounded by the video generator, and sparse keyframes leave intervening motion to the generator.The authors propose denser, more interactive generation–simulation feedback as future work.Source: §V · p. 8