Source-linked AI summary
Learning Visuotactile Skills with Two Multifingered Hands
Toru Lin, Yu Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine, Jitendra Malik
TL;DR
The paper tackles the lack of affordable teleoperation and touch-enabled multifingered hardware for learning dexterous bimanual manipulation. It introduces HATO, repurposed tactile prosthetic hands, and visuotactile policy learning, demonstrating complex task completion and benefits from vision and touch.
Problem
Affordable teleoperation for dual-arm multifingered systems and multifingered hands with touch sensing remain scarce, limiting learning from human demonstrations.
Method
The authors combine HATO, VR-based bimanual teleoperation, repurposed tactile prosthetic hands, and end-to-end policy learning from proprioceptive, visual, and touch observations.
Results
The learned policies complete four long-horizon, high-precision bimanual tasks, while vision and touch improve learning efficiency, success rate, and robustness.
Takeaways & Limitations
Visuotactile sensing and low-cost teleoperation support dexterous bimanual manipulation learned from human demonstrations, including tasks requiring coordination, bulky-object handling, and tool use.
Abstract
from arXiv · showhide
Aiming to replicate human-like dexterity, perceptual experiences, and motion patterns, we explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing. To tackle the first challenge, we develop HATO, a low-cost hands-arms teleoperation system that leverages off-the-shelf electronics, complemented with a software suite that enables efficient data collection; the comprehensive software suite also supports multimodal data processing, scalable policy learning, and smooth policy deployment. To tackle the latter challenge, we introduce a novel hardware adaptation by repurposing two prosthetic hands equipped with touch sensors for research. Using visuotactile data collected from our system, we learn skills to complete long-horizon, high-precision tasks which are difficult to achieve without multifingered dexterity and touch feedback. Furthermore, we empirically investigate the effects of dataset size, sensing modality, and visual input preprocessing on policy learning. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data. Videos, code, and datasets can be found at https://toruowo.github.io/hato/ .
I. INTRODUCTION
The paper addresses limited access to multifingered, touch-enabled bimanual manipulation by introducing an affordable teleoperation and learning system. Using visuotactile demonstrations, it learns policies for complex tasks and studies how sensing and data affect performance.
- HATO combines a low-cost VR-based teleoperation system, prosthetic hands with tactile sensing, and software for data collection, multimodal processing, policy learning, and deployment.
- Policies learned from 30 minutes to 2 hours of HATO data complete four long-horizon, high-precision bimanual tasks through end-to-end learning.The collection time includes approximately 5 to 10 minutes of practice.
- Vision and touch significantly improve learning efficiency, success rate, and robustness, while removing either modality prevents consistent success or causes failure.
- A few hundred demonstrations suffice for effective bimanual dexterous policies, and wrist-mounted cameras improve performance while depth information provides no marked benefit.
- The work releases its hardware, software, and teleoperation dataset to support evaluation and replication.
II. RELATED WORK
Prior bimanual manipulation commonly uses parallel-jaw grippers, while existing teleoperation and visuotactile systems have not combined two multifingered hands with rich sensory feedback.
- Parallel-jaw grippers dominate prior bimanual systems because advanced alternatives are costly to maintain and difficult to obtain.
- Existing teleoperation systems are largely limited to parallel-jaw grippers or single-handed scenarios, and hand retargeting introduces latency and reduces intuitiveness.
- Prior visuotactile applications cover grasping, in-hand manipulation, reconstruction, recognition, cloth manipulation, and interaction representations, but not two multifingered hands.
III. HATO: HANDS-ARMS TELE-OPERATION
HATO is a bimanual multifingered teleoperation system that maps VR controller inputs to robot arms and hands, supported by a software pipeline for collecting multimodal demonstrations.
- HATO maps Meta Quest 2 controller poses to robot-arm end-effector poses and controller inputs to multifingered hand joint positions.
- The system uses two UR5e six-degree-of-freedom arms with two repurposed Psyonic Ability Hands as end effectors.
- Each Ability Hand has five fingers with six actuated degrees of freedom per finger and custom electronics that integrate communication interfaces with power distribution.
B. Teleoperation Setup
The teleoperation setup uses Quest controllers, inverse kinematics, and continuous hand mappings to control bimanual robot arms and multifingered hands. Multifingered contacts address grasp failures that commonly affect parallel-jaw teleoperation.
- Quest controller poses are transformed into robot-coordinate end-effector poses, while inverse kinematics converts those poses into arm joint positions.
- Parallel-jaw grippers are prone to slipping, unstable holds, and unstable grasps on slippery, rigid, or oversized objects; multifingered hands add redundancy and contact area.
- Grip-button pressure continuously controls non-thumb flexion, while thumbstick position controls thumb flexion, extension, and abduction.
- HATO’s hand mapping supports power grasps and can control hands with an anthropomorphic thumb and robot arms through end-effector pose control.
C. Data Collection and Preprocessing for Policy Learning
The policy-learning pipeline collects multimodal data and uses asynchronous diffusion inference for smooth deployment, while vision is crucial for low prediction error across tasks.
- C. Data Collection and Preprocessing for Policy Learning: The pipeline records proprioception, RGB-D vision, and 60 tactile readings from both hands for policy learning.Data are collected at 10 Hz; images are resized to 240×320 before network input.
- C. Data Collection and Preprocessing for Policy Learning: Diffusion policies use multimodal observations to predict action sequences, with asynchronous inference proposed for fast and smooth deployment.The asynchronous algorithm is described as a key component of policy deployment.
- C. Data Collection and Preprocessing for Policy Learning: Across all four tasks, vision is crucial for achieving low prediction error, while touch removal does not significantly affect prediction error for Block Stacking and Steak Serving.Touch can still matter for successful completion even when its effect on prediction error is small.
A. Learning
The system formulates bimanual action prediction as conditional generation with a diffusion model, using a single multimodal observation to predict a sequence of future joint positions.
- A. Learning: The diffusion policy predicts a 16-step sequence of 24-dimensional joint-position actions from one observation of proprioception, vision, and touch.The action vector specifies desired joint positions for both arms and hands.
- A. Learning: End-effector poses are used as proprioceptive inputs instead of arm joint positions because joint configurations can vary unpredictably near singularities during teleoperation.The pose representation avoids a stated learning challenge associated with non-redundant UR5 arms.
- A. Learning: Training uses AdamW with learning rate 0.0001, weight decay 0.00001, and batch size 128, while an exponential weighted average of weights is used for evaluation and deployment.
B. Deployment
Deployment evaluates learned policies on four challenging real-world tasks, using task-specific sensing configurations and demonstrating strong performance on three tasks while steak serving remains difficult.
- B. Deployment: The four tasks test bimanual coordination, slippery-object handover, bulky-object manipulation, pouring, and high-precision steak serving.Slippery handover succeeds when the receiving hand moves more than 10 centimeters away while stably holding the object.
- B. Deployment: More demonstration trajectories consistently reduce prediction loss, with performance saturating at 75 demonstrations for Block Stacking, 200 for Wine Pouring, and 100 for Steak Serving.
- B. Deployment: Human teleoperators practice for 5 to 10 minutes before collecting data for each task.
A. Capabilities from Teleoperation
HATO-collected demonstrations produce policies that reliably complete three bimanual manipulation tasks, while steak serving remains harder because it combines a long horizon with high-precision control.
- A. Capabilities from Teleoperation: Replacing the multifingered hand with a parallel-jaw gripper keeps the rest of the system fixed for a qualitative manipulation-capability comparison.
- A. Capabilities from Teleoperation: Policies complete handover, stacking, and pouring with near-100% success, while steak serving achieves around 50% success.The system still picks up the required objects with 100% success across all tasks.
C. Learning Efficiency
The study examines how demonstration count affects policy prediction error and reports where performance approximately saturates across tasks. It also situates learning efficiency alongside task success under different sensing and camera configurations.
- C. Learning Efficiency: The policies were evaluated across default and rare Block Stacking initializations, with rare configurations designed to be harder for bimanual pickup and stable stacking.
- C. Learning Efficiency: Steak Serving evaluation separately compares success across sensing modalities and camera configurations, including pickup as an intermediate metric.
- C. Learning Efficiency: ActionMSE roughly saturates at 75 demonstrations for Block Stacking, 100 for Steak Serving, and 200 for Wine Pouring.For Slippery Handover, additional demonstrations may further improve policy robustness.
D. Importance of Vision and Touch
Vision and touch improve policy learning and robustness, especially when tasks depart from frequently demonstrated conditions. Wrist-mounted cameras provide more useful views than third-view-only input on most tasks, while depth adds little benefit.
- D. Importance of Vision and Touch: Without vision, the Steak Serving policy achieves 0/10 success at the initial object-pickup stage.Without touch, the policy also performs worse than the full sensing configuration on most tasks, while policies lacking both modalities have the highest ActionMSE.
- D. Importance of Vision and Touch: ActionMSE is informative for expected policy performance but cannot fully capture how well a diffusion policy fits the dataset distribution.The authors identify data log-likelihood estimation as a potentially better but difficult metric.
- D. Importance of Vision and Touch: On rare Block Stacking initializations, the policy without touch succeeds 4/10 times and the policy without vision does not succeed.All sensing configurations achieve 100% success on the default initialization.
- D. Importance of Vision and Touch: Wrist-only input has lower prediction error than third-view-only input on Slippery Handover, Block Stacking, and Steak Serving, with comparable errors on Wine Pouring.The authors hypothesize that wrist views are less occluded and provide spatial cues through arm movement.
- D. Importance of Vision and Touch: Adding depth produces similar prediction error to the no-depth configuration across tasks, and can hurt performance on Wine Pouring.The authors hypothesize that noisy depth readings may harm learning.
VI. DISCUSSIONS
The discussion presents visuotactile sensing and low-cost teleoperation as enabling components for consistent, robust dexterous manipulation. It also identifies haptic feedback and greater visual generalization as future directions.
- VI. DISCUSSIONS: The authors conclude that visuotactile sensing is key for completing complex, long-horizon tasks consistently and robustly.
- VI. DISCUSSIONS: Haptic feedback could improve teleoperation experience and consequently the quality of collected data.
- VI. DISCUSSIONS: Because the policy is learned from scratch without pre-training, it is susceptible to appearance changes in the scene.More robust and generalizable policies are identified as a future research direction.