PaperlayerView exact revision on arXiv

Source-linked AI summary

DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

Hyeonwoo Kim, Jeonghwan Kim, Kyungwon Cho, Hanbyul Joo

arXiv:2604.20841v1cs.CV

TL;DR

Synthetic videos provide rich dexterous HOI cues but are difficult to use directly for physics-based control because they are physically imperfect and purely 2D. DeVI combines 3D human reconstruction with 2D object tracking to train physically plausible control without high-quality 3D demonstrations, outperforming baselines and supporting diverse objects and scenes.

  • Problem

    Synthetic HOI videos contain rich dexterous interaction cues but their limited physical fidelity and 2D nature hinder direct use as imitation targets for physics-based control.

  • Method

    DeVI initializes scenes, generates text-conditioned HOI videos, reconstructs human motion with visual HOI alignment, and combines it with 2D object tracking as hybrid imitation targets.

  • Results

    DeVI outperforms existing 3D-demonstration approaches in dexterous HOI imitation and demonstrates diverse interactions in multi-object scenes.

  • Takeaways & Limitations

    Video can serve as an HOI-aware motion planner for dexterous physics-based control across unseen objects and diverse interaction types.

  • Takeaways & Limitations

    The method focuses on dexterous hand-object manipulation in tabletop scenarios.

Abstract

from arXiv · show

Recent advances in video generative models enable the synthesis of realistic human-object interaction videos across a wide range of scenarios and object categories, including complex dexterous manipulations that are difficult to capture with motion capture systems. While the rich interaction knowledge embedded in these synthetic videos holds strong potential for motion planning in dexterous robotic manipulation, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets in physics-based character control. We present DeVI (Dexterous Video Imitation), a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects. To overcome the imprecision of generative 2D cues, we introduce a hybrid tracking reward that integrates 3D human tracking with robust 2D object tracking. Unlike methods relying on high-quality 3D kinematic demonstrations, DeVI requires only the generated video, enabling zero-shot generalization across diverse objects and interaction types. Extensive experiments demonstrate that DeVI outperforms existing approaches that imitate 3D human-object interaction demonstrations, particularly in modeling dexterous hand-object interactions. We further validate the effectiveness of DeVI in multi-object scenes and text-driven action diversity, showcasing the advantage of using video as an HOI-aware motion planner.

1 Introduction

DeVI uses text-conditioned synthetic videos to control physically plausible agents in dexterous human-object interactions without high-quality 3D demonstrations. Its hybrid imitation targets and tracking design support diverse objects, actions, and multi-object scenes.

  • Existing physics-based motion studies largely ignore human-object interaction, limiting their applicability to robot manipulation.
  • DeVI leverages text-conditioned synthetic videos to guide physically plausible dexterous HOI control without high-quality 3D mocap demonstrations.
  • The hybrid imitation target combines a 3D human reference with a 2D object target because accurately lifting object cues into 3D is difficult.
  • DeVI is evaluated across 20 objects and outperforms 3D-demonstration baselines in imitating reference motion with dexterous manipulation.
  • The framework generalizes to multi-object scenes and diverse dexterous interactions requiring reasoning across multiple objects.

2 Related Work

Prior work addresses physics-based HOI imitation, robotic manipulation, and monocular reconstruction, but remains constrained by dexterous interaction demands and the need for accurate 3D demonstrations. DeVI targets these gaps with video-based planning, visual HOI alignment, and hybrid tracking.

  • Video-based Motion Planning for Robotic Manipulation: Video-based robotic manipulation methods commonly rely on parallel-jaw grippers, limiting multi-finger functional grasps.
  • Video-based Motion Planning for Robotic Manipulation: Open-loop retargeting of generated human hand videos is insufficient for dexterous manipulation, motivating RL-based video imitation in physics simulation.
  • Monocular HOI Reconstruction: Monocular HOI reconstruction remains difficult because human-object interaction requires spatio-temporal alignment, while prior methods often miss dexterous hand motion.
  • Monocular HOI Reconstruction: DeVI introduces visual HOI alignment and a hybrid tracking reward to reconstruct object-aligned human motion without 6D object pose estimation.
  • Physics-based HOI Motion Generation: Physics-based HOI imitation methods require high-quality 3D demonstrations for complex interactions involving object movement.

3 Preliminaries

The control problem is formulated as an MDP in which a policy maps character state and tracking goals to actions that drive a simulated humanoid toward reference motion. DeVI replaces conventional 3D goals with hybrid targets derived from synthesized video.

  • The policy πθ(at|st, gt) receives the current character state and goal vector, then samples actions specifying PD targets.
  • The goal vector represents future kinematic reference frames for human imitation and includes object pose targets for HOI imitation.
  • DeVI generates hybrid imitation targets from synthesized 2D videos, enabling references for unseen objects without pre-captured 3D mocap data.
  • The policy maximizes expected discounted cumulative reward, optimized with Proximal Policy Optimization.
  • The humanoid state includes human and object components, while the action defines PD target angles for body and hand joints.

4 DeVI: Dexterous Video Imitation

DeVI converts text-conditioned synthetic HOI videos into hybrid imitation targets and trains a physics-based humanoid policy for dexterous manipulation. It combines reconstructed 3D human motion with tracked 2D object trajectories, refining alignment and using hybrid rewards for control.

  • Video Generation and Target Extraction: DeVI synthesizes 2D HOI videos from an initialized human-object scene and text prompt, then extracts hybrid imitation targets for policy learning.The targets combine estimated 3D human motion with 2D object trajectories.
  • Video Generation and Target Extraction: The hybrid target uses 3D human pose but retains 2D object trajectories because accurate full 3D object recovery and human-object spatial alignment remain difficult.Tracked visible object projections are initialized from the rendered scene and extended through the video with a tracker.
  • Visual HOI Alignment: Visual HOI Alignment optimizes body and hand pose parameters to align the reconstructed human with both the reference video and the initial 3D object state.Its objective includes body and hand projection, temporal consistency, and HOI contact losses.
  • Visual HOI Alignment: The HOI loss encourages contact between specified human body parts and the initial object so that the interaction can produce object motion.The relevant body parts and object vertices are specified by the text prompt, such as holding a drink with the left hand.
  • Learning Humanoid Control Policy: The hybrid tracking reward trains the humanoid to track reconstructed 3D human motion while following 2D object targets.The policy uses future human target entities as goals and the hybrid reward to guide object tracking.

5 Experiments

Experiments evaluate DeVI against 3D HOI imitation baselines and through qualitative studies of object diversity, visual alignment, target awareness, and text controllability. DeVI achieves stronger reference-motion imitation while supporting diverse interactions in complex scenes.

  • Quantitative Results: DeVI outperforms PhysHOI, SkillMimic, and InterMimic across reported human and object imitation metrics on GRAB.The comparison uses 16 short GRAB HOI motions and evaluates human and object tracking separately.
  • Qualitative Results: DeVI generates both simple and affordance-specific interactions for novel objects, including picking up garbage, drinking coke, taking a photo, and wearing a hat.These interactions are generated from text prompts using the video diffusion model as an HOI-aware motion planner.
  • Ablation Study: Without visual HOI alignment, independently estimated body and hand poses misalign with the video and 3D object, whereas alignment produces feasible hand-object interactions.The aligned reconstruction enables the motion to be simulated in a physics simulator.
  • Qualitative Results: DeVI handles scenes with multiple objects and generates distinct motions for the same scene from different text prompts.The results demonstrate target awareness without explicit scene understanding and text controllability.
  • Quantitative Results: 2D object trajectory tracking achieves better HOI imitation than traditional 6D pose tracking while avoiding dense 6D reward over-constraint.The projected geometry implicitly guides object position and rotation without explicitly enforcing either.
  • Ablation Study: Visual HOI alignment reduces pixel errors, especially for hand joints, and decreases human-object distance at contact frames.The ablation evaluates 276 generated videos across 12 object categories using video-alignment and HOI metrics.

6 Discussion

DeVI uses synthetic video generation as an HOI-aware motion planner and converts its outputs into hybrid imitation targets for physics-based humanoid control. The method reconstructs human motion, tracks objects in 2D, and produces diverse interactions across objects and multi-object scenes.

  • Discussion: DeVI generates dexterous HOI in physics simulation without requiring high-quality 3D demonstrations such as motion-capture data.Its motion plans come from synthetic videos rather than kinematic demonstrations.
  • Discussion: The framework initializes a scene, generates an image-to-video motion plan, reconstructs 3D human motion, and combines it with 2D object tracking.These hybrid targets train a humanoid control policy to imitate the generated video.
  • Discussion: DeVI qualitatively plans and imitates interactions for various objects and quantitatively improves 3D human reconstruction and motion-capture imitation.It also generates diverse interactions in multi-object scenes.

A Implementation Details

The implementation constructs a tabletop scene, renders a suitable human-object view, generates a text-conditioned video, and reconstructs and aligns human motion for simulation. Contact labels are then inferred from 2D object and hand trajectories.

  • Scene Initialization: The initialized scene places an SMPL-X human at the origin, a table at (0.0, 0.4), and objects on the tabletop in physically valid states.The human faces the positive y-axis, and object poses are initialized consistently for rendering and simulation.
  • Human Representation: A textured human mesh is obtained by transferring SMPL-X offsets and skinning weights from THuman 2.0 annotations.The mesh is then deformed using linear blend skinning with K = 16 nearest vertices for approximation.
  • HOI Video Generation: Sixteen candidate cameras are placed around the scene, with a frontal view selected when both hands and the object are visible.The selected camera renders a 576 × 1024 image used as input to video generation.
  • HOI Video Generation: Wan with LightX2V LoRA generates each HOI video from a rendered image and a prompt specifying the person, action, object, hand, and details.A stationary camera is specified, and prompts may be hand-designed or automatically generated.
  • Human Motion Reconstruction: GVHMR and HaMeR reconstruct coarse body and hand motion, which is unified into a single SMPL-X model through wrist-pose transformation.The wrist rotation from the hand estimator is converted to axis-angle form for the unified model.
  • Visual HOI Alignment: Visual HOI alignment optimizes upper-body SMPL-X parameters with 2D projection and one-sided Chamfer losses to align motion with video and the 3D object.Only hands, wrists, elbows, shoulders, and related upper-body poses are optimized to avoid harmful effects on occluded joints.
  • Contact Estimation: Contact labels are estimated from object-vertex and hand-joint trajectories by identifying object motion and refining labels across neighboring frames.The procedure uses a threshold and forward-backward temporal passes.

A.8 Network Architecture

DeVI uses an actor-critic control policy and a contact reward informed by video-derived timing cues. The architecture and reward combine state encoding with force and distance objectives for dexterous interaction.

  • Network Architecture: The actor encodes human state, object state, and target future pose separately before transformer fusion, while the critic concatenates the same inputs directly.The actor then uses an MLP to output humanoid actions.
  • Contact Reward: The contact reward R_contact = R_cf · R_cd encourages simulated humanoids to establish contact with the target object.It combines contact force and hand-to-object contact distance rewards.
  • Contact Reward: The contact force reward uses video-inferred binary contact labels and force-sensor measurements from the hands.The force reward is computed separately for the left and right hands before combining them.
  • Tracking Comparison: DeVI’s 2D trajectories outperform baselines even without 6D object poses when tracking human and object motion.The comparison highlights the practical value of projected object geometry for HOI imitation.

A.10 Training Details

DeVI trains a humanoid control policy with PPO-based actor-critic learning, using hybrid imitation rewards and initialization near object contact. Training uses large-scale parallel simulation and specified optimization settings.

  • Time Sampling for Initialization: The policy initializes at the pre-contact frame with 50% probability because arbitrary-time initialization lacks the object’s 6D pose reference.The authors report improved policy learning over first-frame initialization.
  • Early Termination: Early termination uses 3D body, fingertip, and 2D object trajectory errors to stop rollouts that diverge substantially from the imitation target.The stated body thresholds include 200 mm mean error or 400 mm error for any joint.
  • Early Termination: The 2D termination threshold is derived from τ2D = α2D(W^2 + H^2)^1/2, with α2D = 0.08 yielding about 94 pixels at W = 1024 and H = 576.This threshold defines the pixel-distance scale for the 2D object trajectory.
  • Training Setup: Training collects data from 4096 Isaac Gym environments, updates after 32 rollouts, and uses batch size 1024.The actor and critic learning rates are 2 × 10^-5 and 1 × 10^-4, respectively.
  • Policy Learning: The actor-critic policy uses a transformer-based actor and MLP critic, with PPO updating the actor from policy gradients and GAE advantages.The actor outputs humanoid control actions while the critic estimates the value function.

B.1 Qualitative Results

Qualitative results show that DeVI tracks human and object interaction behavior with sparse 2D object cues, extends beyond tabletop settings, and aligns simulated motions with hybrid targets.

  • Qualitative Comparison with Baselines: DeVI follows human poses better than SkillMimic and InterMimic, while following object poses better than PhysHOI despite using only 2D object trajectories.The comparison demonstrates hybrid tracking without precise 6D object poses.
  • Non-tabletop Scenarios: DeVI and its hybrid imitation rewards extend from tabletop setups to non-tabletop pushing and pick-and-place motions.The demonstrations use the FullBodyManip dataset.
  • DeVI on GRAB Dataset: GRAB demonstrations show successful HOI imitation from relatively sparse 2D trajectory references, including scenarios not generated through video diffusion.The 3D object vertices are projected into a virtual camera view to create the 2D reference.
  • DeVI on GRAB Dataset: GRAB success requires passing all three thresholds: MPJPE (All), Tobj, and Oobj.This criterion is defined in the table caption.

B.2 Quantitative Results

On GRAB, DeVI achieves a higher imitation success ratio than baselines that use 6D poses, while relying on relatively sparse 2D object trajectories.

  • DeVI on GRAB Dataset: DeVI outperforms baselines on GRAB imitation success ratio despite using relatively sparse 2D object trajectories instead of 6D pose references.The authors attribute practical usefulness to the greater efficiency of obtaining 2D object trajectories than 6D poses.
  • DeVI on GRAB Dataset: A 6D pose reward makes DeVI’s objective more challenging, whereas the 2D object tracking reward more effectively guides policy learning through projected position and rotation.The comparison is reported under the same number of training epochs.

C Limitation and Future Work

The authors identify perspective artifacts and imperfect automatic contact labels as limitations that can reduce motion naturalness or produce unnatural interaction behaviors.

  • C.1 Perspective Artifacts in Video Diffusion: Imperfect perspective in generated videos introduces depth-direction errors in visual HOI alignment and can reduce reconstructed-motion naturalness.The error becomes significant for precise target placement, such as putting a baseball into a small cup.
  • Future Work: Multi-view video diffusion and affordance grounding are proposed as future directions for reducing depth errors and refining contact labels.These directions target the two identified limitations separately.
  • C.2 Limits of Automatic Contact Estimation: Pixel-velocity-based pseudo contact labels omit depth motion, causing estimation errors and sometimes unnatural behaviors such as quickly snatching an object.The authors suggest affordance grounding as a future refinement.
Loading 2604.20841v1…