PaperlayerView exact revision on arXiv

Source-linked AI summary

DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos

Juncheng Mu, Sizhe Yang, Yiming Bao, Hojin Bae, Tianming Wei, Linning Xu, Boyi Li, Huazhe Xu, Jiangmiao Pang

arXiv:2602.10105v1cs.RO

TL;DR

DexImit tackles the scarcity and embodiment-gap problems that make bimanual dexterous manipulation data difficult to collect and learn from. It converts monocular human videos into physically plausible robot data through a four-stage pipeline, and the resulting data supports diverse tasks and zero-shot real-world deployment. The method achieves near-perfect short-horizon performance, high success on the long-horizon Pot task, and 52% success on Stack Six Cups.

  • Problem

    Bimanual dexterous manipulation lacks large-scale data because real-world collection is expensive, while direct learning from human videos is constrained by the human–robot embodiment gap.

  • Method

    DexImit converts monocular human videos into robot data through 4D reconstruction, subtask decomposition and bimanual scheduling, structured action generation, and comprehensive data augmentation.

  • Results

    DexImit achieves near-perfect short-horizon performance, maintains a high success rate on the long-horizon Pot task, and reaches 52% success on Stack Six Cups.

  • Takeaways & Limitations

    DexImit generates diverse bimanual dexterous-manipulation data from Internet or generated human videos and supports zero-shot real-world policy deployment.

  • Takeaways & Limitations

    Sequential modules can accumulate errors, requiring manual intervention for some long videos, and the current pipeline cannot handle complex in-hand manipulation.

Abstract

from arXiv · show

Data scarcity fundamentally limits the generalization of bimanual dexterous manipulation, as real-world data collection for dexterous hands is expensive and labor-intensive. Human manipulation videos, as a direct carrier of manipulation knowledge, offer significant potential for scaling up robot learning. However, the substantial embodiment gap between human hands and robotic dexterous hands makes direct pretraining from human videos extremely challenging. To bridge this gap and unleash the potential of large-scale human manipulation video data, we propose DexImit, an automated framework that converts monocular human manipulation videos into physically plausible robot data, without any additional information. DexImit employs a four-stage generation pipeline: (1) reconstructing hand-object interactions from arbitrary viewpoints with near-metric scale; (2) performing subtask decomposition and bimanual scheduling; (3) synthesizing robot trajectories consistent with the demonstrated interactions; (4) comprehensive data augmentation for zero-shot real-world deployment. Building on these designs, DexImit can generate large-scale robot data based on human videos, either from the Internet or video generation models. DexImit is capable of handling diverse manipulation tasks, including tool use (e.g., cutting an apple), long-horizon tasks (e.g., making a beverage), and fine-grained manipulations (e.g., stacking cups).

I. INTRODUCTION

DexImit addresses scarce bimanual dexterous-manipulation data by converting monocular human videos into physically plausible robot data. Its four-stage pipeline reconstructs interactions, schedules subtasks, generates robot trajectories, and augments data for broad task coverage and zero-shot deployment.

  • Data collection for bimanual dexterous manipulation is difficult because teleoperation and hardware costs make large-scale datasets expensive.
  • Human manipulation videos provide large-scale, diverse demonstrations that encode both high-level task concepts and low-level manipulation actions.
  • DexImit targets embodiment-gap challenges by generating robot trajectories from reconstructed human hand-object interactions rather than directly pretraining on human hands.
  • DexImit uses reconstruction, action-centric scheduling, force-closure-based action generation, and comprehensive augmentation to convert videos into robot data without additional depth or camera information.
  • The framework synthesizes physically plausible data across long-horizon, tool-using, and fine-grained manipulation tasks.
  • Comprehensive augmentation of object pose and scale, camera pose, and visual observations supports zero-shot deployment without real-world data.

A. Reconstruction of 4D Hand-Object Interactions

DexImit reconstructs hand-object interactions from monocular videos by combining video understanding, segmentation, depth and pose estimation, near-metric scale alignment, and coordinate transformation. The resulting trajectories are mapped into a unified world frame for robot-data generation.

  • The reconstruction pipeline uses video understanding, semantic segmentation, object and hand reconstruction, 6D pose estimation, and camera-to-world coordinate transformation.
  • Qwen3-VL identifies the objects involved in manipulation from sampled video frames for downstream reconstruction.
  • Grounded Sam2 generates object, hand, and table masks for 3D generation, hand-trajectory estimation, and world-coordinate determination.
  • SpatialTracker v2 estimates unscaled depth, while the limited variance of human hand sizes provides a prior for recovering metric scale from RGB video.
  • An align-render-align procedure jointly addresses hand position, depth gaps, and scale alignment before continuous hand and object trajectories are reconstructed.
  • The method estimates a world frame from table geometry, hand-derived direction, and object poses, then maps arbitrary-view trajectories into that shared coordinate system.

B. Subtask Decomposition and Task Scheduling

DexImit represents manipulation tasks as ordered subtasks and schedules them across embodiments, horizons, and action combinations. Its action-centric scheduler supports unimanual, cooperative bimanual, and independently concurrent bimanual behaviors.

  • DexImit imposes no restrictions on input-video horizon, bimanual concurrency or asynchrony, or manipulation-action type.Supported actions range from unimanual grasping to cooperative bimanual grasping and fully concurrent pouring.
  • Each task is represented using an embodiment set, associated object, ordered subaction list, and current subaction index.
  • Subactions record a start frame and an action type selected from pregrasp, grasp, motion, and release.Qwen3-VL annotates these structures through video understanding, subtask decomposition, and structured labeling; manual annotations may improve long-horizon labeling.
  • The Action-Centric Scheduling Algorithm schedules subtasks across arbitrary numbers of embodiments, temporal horizons, and action combinations.

C. Source Data Generation

DexImit converts scheduled hand-object interactions into robot trajectories through force-closure-based grasp synthesis and key-frame-based motion planning. It generates feasible grasps, selects candidates consistent with demonstrated hand behavior, and propagates object transformations to end-effectors.

  • Low-level action generation combines force-closure-based grasp synthesis with key-frame-based motion planning.
  • Algorithm 1 sorts tasks by start time, maintains a priority queue, assigns trajectories to embodiment action queues, and executes nonempty actions across the horizon.
  • The scheduler advances each task's subaction index while assigning trajectories and executing the corresponding embodiment actions.
  • Grasp Synthesis: Grasp synthesis samples contact configurations, initializes hands along surface normals, and optimizes poses and contact forces for physical feasibility.Unimanual grasps use one convex-hull contact, whereas bimanual grasps use two contacts on opposite object sides.
  • Grasp Synthesis: The grasp objective penalizes contact distance, hand-object collision, and hand-hand penetration while matching target wrenches.Solving it yields physically feasible, collision-free grasp candidates.
  • Grasp Synthesis: Candidates are ranked by distance to the reconstructed human hand pose, then tested sequentially until one satisfies the stability criterion.
  • Motion Generation: Motion planning computes the relative object transformation between current and target key frames and applies it to the selected end-effector pose.The resulting terminal configuration guides motion planning toward the desired object manipulation.
  • Motion Generation: The transformed end-effector pose provides the terminal configuration for planning the selected embodiments' motion.

D. Data Augmentation

DexImit augments source trajectories across object pose, object scale, camera pose, and sensor observations to improve generalization. A 3D Diffusion Policy trained on this augmented dataset supports zero-shot real-world deployment.

  • DexImit applies four augmentation types: object pose, object scale, camera pose, and observation.
  • Object Pose: Object-pose randomization varies object position and translation to enable spatial generalization.
  • Object Scale: Object-scale augmentation applies factors of [0.8, 1.2] around the source scale of 1.0.Regenerating grasps and motions for each scale produced inconsistent supervision that destabilized training and slowed convergence in the stated policy-learning setting.
  • Camera Pose: Camera-pose augmentation randomizes both camera orientation and position for viewpoint generalization.
  • Observation: Point-cloud observations simulate depth-sensor variability by removing 30% of object points and perturbing remaining normals by 30%.
  • A 3D Diffusion Policy trained on the augmented dataset demonstrates zero-shot deployment in real-world scenarios.

IV. EXPERIMENTS

The experiments evaluate DexImit along scalability and task-difficulty dimensions, including reconstruction, usability, comparisons with existing approaches, complex manipulation, and zero-shot deployment. Reconstruction and usability analyses examine how methods, video quality, and task difficulty affect generated data.

  • The evaluation studies whether DexImit scales and the upper bound of task difficulty it can handle.
  • The experiments ask whether generated data is usable, higher quality than existing approaches, suitable for complex tool-using and long-horizon tasks, and sufficient for zero-shot deployment.
  • Object Trajectory Estimation: The reconstruction study evaluates four depth-estimation models and three pose-estimation methods for 100 short-horizon tasks.
  • Object Trajectory Estimation: Combining tracking with 6D pose estimation achieves the highest reconstruction accuracy, while ST2 provides more temporally consistent depth estimates and a higher success rate among the compared alternatives.
  • Usability: Figure 3 reports usable-data proportions across input-data quality and target-task difficulty, using two representative tasks per difficulty level.Usable data is defined as physically plausible and training-ready samples.
  • Usability: Veo3-generated videos perform strongly on low-complexity tasks, whereas casually captured videos degrade substantially as task complexity and temporal horizon increase.Informed-operator videos achieve near-complete usability for simple tasks and retain considerable success on challenging long-horizon tasks.

B. Data Quality

DexImit is evaluated for whether video-derived data supports policy learning across increasingly difficult simulated and real-world manipulation tasks. Results show strong performance on short-horizon tasks, sustained performance on long-horizon tasks, and high zero-shot success across real-world meta-tasks, while pipeline errors and unsupported in-hand manipulation remain limitations.

  • DexImit generates physically plausible data for long-horizon and fine-grained real-world manipulation tasks.
  • DexImit achieves near-perfect performance on short-horizon tasks and a 52% success rate on the fine-grained Stack Six Cups task.It also maintains a high success rate on the long-horizon Pot task, while baselines fail on challenging long-horizon tasks.
  • DexImit achieves consistently high zero-shot success rates across four real-world meta-tasks under sim-to-real transfer.The evaluated tasks include unimanual placement, independent bimanual placement, bimanual grasping, and cooperative pouring.
  • Removing scale augmentation or visual augmentation reduces real-world performance, while regenerating grasps and motions at each scale causes a drastic degradation.The results attribute the latter degradation to inconsistent motions that introduce conflicting imitation-learning supervision.
  • Limitations: DexImit’s sequential pipeline can propagate errors that render generated data unusable, and it cannot currently handle complex in-hand manipulation.

C Detailed Limitations ..................................................................... 5

The detailed grasp-synthesis procedure conditions candidate grasps on demonstrated hand-object interaction and ranks them by pose consistency before testing physical stability. Stable candidates are selected through simulation-based evaluation against the intended object motion.

  • Grasp conditioning: A vision-language model predicts the number of actively involved fingers, which defines the contact subset used for grasp synthesis.This conditions synthesized grasps on the demonstrated interaction pattern and supports physically feasible N-finger grasps.
  • Candidate ranking: Grasp candidates are ranked with a weighted metric combining hand translation and rotation errors relative to the reconstructed human pose.The translation and rotation terms are balanced by λt and λr.
  • Candidate ranking: Rotation error is represented by the angle of the relative rotation in radians.
  • Stability evaluation: Grasp stability is evaluated by simulating object motion from the grasping pose toward the intended target configuration.The stability error compares simulated and target-transformed object point clouds using mean Euclidean distance.
  • Stability evaluation: A candidate is accepted when its stability error falls below a predefined threshold, and candidates are tested sequentially until a stable grasp is found.

A.2. Visual Observation Augmentation

DexImit augments simulated 3D point-cloud observations to reproduce the sparsity, missing points, and depth noise of real sensors. The resulting training variation improves robustness and generalization during sim-to-real transfer.

  • Point-cloud construction: DexImit constructs a reference object point cloud from sampled mesh surface points and aligns it with the object pose at each simulation timestep.The transformed reference cloud is used to extract object-specific observations from the rendered scene.
  • Point-cloud construction: Object-specific observations are extracted by comparing rendered scene points with the transformed reference object cloud.
  • Noise and sparsity augmentation: The augmentation randomly retains 85% of object points to simulate occlusions and missing depth measurements.
  • Noise and sparsity augmentation: Removed observations are replaced with points perturbed along surface normals using Gaussian noise with σ = 0.015.The augmented cloud preserves the original observation cardinality.
  • Effect: The resulting point-cloud augmentation produces realistic sensor imperfections and substantially improves robustness and generalization in sim-to-real transfer.

A.3. Automatic Data Filter

DexImit uses automated filtering to remove erroneous synthesized samples caused by compounding errors across its generation pipeline. A vision-language model compares the intended video description with a rendered synthesized manipulation video to assess data usability.

  • Automated filtering is needed because compounding errors can produce erroneous samples during scalable synthetic-data generation.
  • Qwen3-VL generates a predicted video description during task scheduling, then evaluates synthesized data by comparing that description with a rendered manipulation video.

B.1. Simulation Experiments

DexImit enables zero-shot real-world deployment from generated manipulation data, while its current scope is constrained by object geometry, task duration, and manipulation setting.

  • DexImit enables zero-shot real-world deployment by combining source-data generation, augmentation, and policy learning.
  • The framework’s data-generation runtime increases with input-video length because per-frame modules scale with video duration.Depth estimation, hand-pose estimation, segmentation, and subtask decomposition scale with duration, whereas several downstream modules remain relatively stable.
  • DexImit cannot yet manipulate soft or articulated objects because its 3D generation stage assumes rigid object geometry.
  • DexImit is designed for tabletop bimanual manipulation and does not support mobile manipulation without modeling embodiment motion and environmental dynamics.
  • Manual intervention is sometimes necessary for long videos because sequential modules can accumulate errors.Intervention may involve refining VLM-based subtask decomposition or correcting reconstruction artifacts.
  • In-hand manipulation is currently unsupported because monocular video reconstruction faces severe occlusions and limited observability.
Loading 2602.10105v1…