PaperlayerView exact revision on arXiv

Source-linked AI summary

DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies

Tony Tao, Mohan Kumar Srirama, Jason Jingzhou Liu, Kenneth Shaw, Deepak Pathak

arXiv:2505.07813v2cs.ROcs.AIcs.CVcs.LGeess.SY

TL;DR

DexWild addresses the difficulty of acquiring large, diverse robot datasets for dexterous manipulation. It introduces a portable human-data collection system and co-trains human demonstrations with robot demonstrations. The resulting policies generalize to unseen environments, tasks, and embodiments, while the approach still depends on some teleoperated robot data and lacks sufficient error-recovery and tactile information.

  • Problem

    Large-scale, diverse robot datasets are needed for dexterous generalization, but teleoperation is labor-intensive and expensive to scale.

  • Method

    DexWild uses a portable human-centric system to collect demonstrations and co-trains them with a smaller set of robot demonstrations.

  • Results

    68.5% success in completely unseen environments was nearly four times higher than robot-only methods, with 5.8× better cross-embodiment transfer.

  • Takeaways & Limitations

    Human interaction data at scale can support robot policies that generalize across new environments, tasks, and embodiments with less robot-specific data.

  • Takeaways & Limitations

    DexWild still requires some teleoperated robot data, lacks error-recovery demonstrations, and uses only visual and kinematic data for contact-rich tasks.

Abstract

from arXiv · show

Large-scale, diverse robot datasets have emerged as a promising path toward enabling dexterous manipulation policies to generalize to novel environments, but acquiring such datasets presents many challenges. While teleoperation provides high-fidelity datasets, its high cost limits its scalability. Instead, what if people could use their own hands, just as they do in everyday life, to collect data? In DexWild, a diverse team of data collectors uses their hands to collect hours of interactions across a multitude of environments and objects. To record this data, we create DexWild-System, a low-cost, mobile, and easy-to-use device. The DexWild learning framework co-trains on both human and robot demonstrations, leading to improved performance compared to training on each dataset individually. This combination results in robust robot policies capable of generalizing to novel environments, tasks, and embodiments with minimal additional robot-specific data. Experimental results demonstrate that DexWild significantly improves performance, achieving a 68.5% success rate in unseen environments-nearly four times higher than policies trained with robot data only-and offering 5.8x better cross-embodiment generalization. Video results, codebases, and instructions at https://dexwild.github.io

I. INTRODUCTION

DexWild addresses the shortage of large, diverse robot datasets by collecting accurate human demonstrations in natural settings and co-training them with robot data. The framework reports stronger generalization across environments, embodiments, and tasks than robot-only approaches.

  • Large-scale, diverse robot datasets needed for dexterous generalization remain unavailable, creating a central bottleneck for robot learning.
  • Teleoperation provides high-precision data but requires specialized setups, skilled operators, and costly relocation across environments.
  • Internet videos offer visual diversity but lack reliable fine-grained hand states and structured task episodes for direct robotic use.
  • DexWild co-trains on human and robot demonstrations to learn robust dexterous manipulation policies.
  • 9,290 demonstrations across 93 environments were collected 4.6× faster than conventional robot-based methods.
  • 68.5% success in novel environments was nearly four times higher than robot-only policies, alongside 5.8× better cross-embodiment transfer.

C. Human Action Tracking Systems

DexWild-System is a portable, high-fidelity, human-centric platform for collecting dexterous demonstrations with minimal setup. It combines robust hand and wrist tracking, palm-mounted cameras, and aligned human–robot observation and action spaces for embodiment-agnostic retargeting.

  • C. Human Action Tracking Systems: DexWild-System uses a lightweight glove-based solution with electromagnetic sensing for accurate, real-time fingertip tracking that remains robust to occlusions.
  • C. Human Action Tracking Systems: DexWild-System is designed for portability, high fidelity, and embodiment-agnostic retargeting across diverse environments and robot hands.
  • C. Human Action Tracking Systems: A relative state-action representation removes the need for a global coordinate frame and allows flexible tracking-camera placement.
  • C. Human Action Tracking Systems: The system captures human demonstrations with a motion-capture glove, external-camera wrist tracking, and synchronized palm-mounted cameras.
  • C. Human Action Tracking Systems: Mirrored palm-camera placement creates visually consistent observations across human and robot embodiments for shared visual representations.
  • C. Human Action Tracking Systems: Robot hand kinematics are optimized to match human fingertip positions using fixed hyperparameters across users and varying hand sizes.

B. Training Data Modalities and Preprocessing

DexWild combines a large human dataset with a smaller teleoperated robot dataset, then preprocesses and samples both modalities for joint training. The representation uses synchronized palm-camera observations, relative state histories, and chunked robot actions.

  • B. Training Data Modalities and Preprocessing: DexWild collects a large human dataset DH and a smaller teleoperated robot dataset DR to combine diversity with embodiment grounding.
  • B. Training Data Modalities and Preprocessing: Each training batch samples human and robot transitions according to fixed co-training weights.
  • B. Training Data Modalities and Preprocessing: Observations contain synchronized palm-camera images and historical relative end-effector positions sampled across a horizon.
  • B. Training Data Modalities and Preprocessing: Actions are represented as n-step chunks whose current action includes 9-dimensional arm and 17-dimensional hand components.
  • B. Training Data Modalities and Preprocessing: For bimanual tasks, observation and action spaces are duplicated and inter-hand pose is appended to support coordination.
  • B. Training Data Modalities and Preprocessing: Human and robot actions are normalized separately, while heuristic filtering removes low-quality or invalid human trajectories without manual labeling.

C. Policy Training

DexWild trains a behavior-cloning policy on multimodal human and robot data using pretrained visual features and a diffusion-based action model. The training procedure samples weighted demonstrations, adds noise to action chunks, and learns to predict that noise.

  • C. Policy Training: DexWild uses a simple behavior-cloning objective to train on multimodal, diverse demonstrations.
  • C. Policy Training: A pretrained Vision Transformer encoder extracts visual features intended to generalize across diverse scenes, objects, and lighting.
  • C. Policy Training: The diffusion-based policy models multimodal action distributions arising from diverse human strategies and generates action chunks.
  • C. Policy Training: Training samples human and robot batches using co-training weights, encodes observations, adds Gaussian noise to action chunks, and predicts that noise.
  • C. Policy Training: The human-to-robot data weighting significantly affects real-world policy performance.

IV. EXPERIMENTS

The experiments evaluate DexWild through real-world deployment and diverse task data, with training and test objects distributed across five manipulation tasks.

  • The evaluation uses real-world deployment across diverse environments and robots, combining human demonstrations with robot teleoperation data.
  • Data collection spans five tasks, with training and test objects distributed across Spray Bottle, Toy Cleanup, Pour, Florist, and Clothes Folding.The figure reports separate train and test object counts for each task.

A. Scaling up Data Collection

DexWild-System supports broad human data collection across varied real-world settings, producing human and robot datasets for evaluating dexterous manipulation across tasks, embodiments, and environment splits.

  • Scaling up Data Collection: Ten untrained users collected data across indoor and outdoor environments, varying lighting, objects, tables, crowding, and demonstration styles.This variation exposes the system to diverse environments and interactions.
  • Scaling up Data Collection: The human dataset DH contains 9,290 demonstrations across five tasks and multiple environments.DH includes Spray Bottle, Toy Cleanup, Pour, Florist, and Clothes Folding demonstrations.
  • Scaling up Data Collection: The robot dataset DR contains 1,395 demonstrations across the same five tasks, collected with an xArm and LEAP hand V2 Advanced.
  • Scaling up Data Collection: The evaluation tasks assess functional grasping, long-horizon planning, cross-task transfer, bimanual coordination, and deformable-object manipulation.Success requires adaptation to object properties, environmental conditions, and task constraints.
  • Scaling up Data Collection: Experiments use xArm and Franka systems with LEAP hands across in-domain, in-the-wild, and in-the-wild extreme environments.The extreme setting is unseen in both human and robot datasets.
  • Scaling up Data Collection: The evaluation asks whether human data improves in-the-wild performance, enables transfer across tasks and embodiments, and scales with more DexWild-System data.

A. Zero Shot In the Wild Policies w/ DexWild

DexWild co-training combines human demonstrations for environmental diversity with robot demonstrations for precise action grounding, substantially improving generalization across novel scenes, tasks, embodiments, and dataset scales. The system also supports faster, portable data collection, enabling broader evaluation and scaling.

  • Novel-scene generalization: 79.8% in-domain, 75.1% in-the-wild, and 62.7% in-the-wild extreme success are achieved with a 1:2 robot-to-human co-training ratio.These results exceed robot-only performance of 64.7%, 28.5%, and 22.0%, respectively; increasing the human ratio to 1:5 reduces performance.
  • Novel-scene generalization: 68.1% average success on bimanual tasks compares with 13% for the robot-only baseline in in-the-wild extreme settings.DexWild policies also make meaningful task attempts when they fail, whereas robot-only policies often behave erratically or without structure.
  • Cross-task transfer: 94% success on zero-shot pouring exceeds robot-only performance of 0% and human-only performance of 11%.The policy uses human pouring demonstrations together with robot spraying demonstrations, without robot pouring data.
  • Cross-embodiment transfer: 37.5% cross-arm success versus 4.5% for the robot-only baseline represents an 8.3× improvement when transferring from an xArm to a Franka Panda arm.The transfer is evaluated in in-the-wild extreme scenes.
  • Cross-embodiment transfer: 65.3% cross-hand success versus 13.3% for the baseline demonstrates transfer from the LEAP Hand V2 Advanced to the original LEAP Hand.This result extends generalization across dexterous hands, not only across robot arms.
  • Scaling and collection: Average task performance rises from 28.7% at 20% dataset size to 67.8% with the full dataset, a 2.36× improvement.The learning curve is nonlinear, with especially steep gains between 25% and 50% data, and performance continues improving at full data usage.
  • Scaling and collection: 201 demonstrations per hour is 4.6× faster than Gello-based robot teleoperation, which collects 43 demonstrations per hour.DexWild-System nearly matches the collection rate of bare-hand demonstrations and can be set up in minutes rather than requiring heavy robot relocation and setup.

VI. CONCLUSION AND LIMITATIONS

DexWild combines large-scale human demonstrations with minimal robot data to learn policies that generalize across tasks, environments, and embodiments. The approach achieves strong unseen-environment performance but still depends on limited teleoperated data and has robustness limitations.

  • DexWild co-trains large-scale human demonstrations with minimal robot data to generalize across tasks, environments, and robot embodiments.
  • 68.5% success in completely unseen environments is nearly four times higher than methods using robot data only.
  • DexWild-System accelerates dataset creation 4.6× compared with conventional robot teleoperation.
  • Limitations: The approach still depends on a limited amount of teleoperated robot data to bridge human and robot actions.
  • Limitations: Human demonstrations rarely include error recovery, and visual-kinematic sensing limits performance in contact-rich tasks.

VII. APPENDIX

The appendix describes evaluation tasks, milestone-based scoring, and a deployment interface designed for untrained data collectors. The tasks cover grasping, long-horizon planning, precision, bimanual coordination, and deformable-object manipulation.

  • Evaluation: The evaluation covers five dexterous tasks spanning functional grasping, long-horizon planning, precision, bimanual coordination, and deformable-object manipulation.
  • Scoring: Task scores use discrete completion milestones, assigning higher values to more complex or functionally successful actions.
  • Tasks: Toy Cleanup evaluates long-horizon planning by requiring robots to collect scattered toys and deposit them in a designated bin.
  • Tasks: Pouring evaluates precise motion control and transfer from spray-bottle manipulation by requiring liquid to be poured into a container.
  • Tasks: Bimanual Florist evaluates coordinated two-hand control through bouquet pickup, handover, and vase insertion.
  • Deployment: DexWild-System provides a simple interface for deployment by untrained data collectors.

B. Data Collection Procedure

DexWild-System supports portable, human-centered data collection through a wrist camera, onboard computer, sensor pod, motion-capture glove, and palm-mounted cameras. Processing filters unreliable episodes, removes outliers, smooths trajectories, and retargets hand motions before policy training.

  • Hardware and deployment: Untrained collectors use a wrist-tracking camera, battery-powered mini-PC, motion-capture glove, and palm-mounted cameras.
  • Data organization: Each episode is stored separately, while Zed Mini SVO recordings support SLAM and wrist-pose tracking.
  • Processing: Episodes are discarded when the wrist pose cannot be reliably tracked for more than 75% of the duration.
  • Processing: The pipeline clips action outliers outside the 2nd and 97th percentiles and smooths trajectories with interpolation and Gaussian filtering.
  • Policy inputs: The behavior-cloning policy combines ViT image tokens with relative-state tokens, including interhand pose for bimanual tasks.
  • Control: A Riemannian Motion Policy generates joint-space targets from end-effector targets and incorporates real-time collision avoidance.

F. Comparing Policy Classes

DexWild co-training improves performance across policy classes, robot hands, dataset sizes, and deployment conditions. Diffusion policies benefit particularly strongly, while gains also persist for cross-embodiment transfer and larger human-data scales.

  • Policy classes: Diffusion policies achieve the highest scores across tasks and benefit more from DexWild co-training than ACT policies.
  • Cross-embodiment: DexWild co-training substantially outperforms robot-only baselines on the LEAP Hand across In-the-Wild and In-the-Wild Extreme conditions.
  • Dataset scaling: Performance increases steadily as the DexWild dataset grows from 0% to 100%, with a notable jump between 25% and 50%.
  • Cotraining ratios: Cotraining ratios are evaluated across In-Domain, In-the-Wild, and In-the-Wild Extreme conditions on xArm with LEAP Hand V2 Advanced.
  • Experimental settings: The experiments use the full training and architecture settings listed for the policy evaluations.
Loading 2505.07813v2…