Source-linked AI summary
Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, Yuke Zhu
TL;DR
Vision-based sim-to-real RL has rarely addressed contact-rich bimanual manipulation on humanoids, motivating a scalable alternative to costly real-world demonstrations. The paper introduces a practical recipe combining automated real-to-sim tuning, structured rewards, policy distillation, and hybrid object representations, and reports strong transfer across tasks and objects. The authors conclude that this approach provides a feasible path toward generalizable and robust humanoid manipulation, while remaining far from general-purpose capability.
Problem
Sim-to-real RL for dexterous manipulation has largely remained limited to single-hand or state-based setups, leaving vision-based contact-rich bimanual humanoid tasks open.
Method
The recipe combines automated real-to-sim calibration, contact- and object-goal rewards, divide-and-conquer policy distillation, and hybrid object representations.
Results
The method performs three vision-based manipulation tasks, achieving 90% success on seen objects and 60∼80% on novel objects while adapting across two robot hands.
Takeaways & Limitations
The results establish a practical and scalable recipe for high-performance vision-based dexterous manipulation via sim-to-real RL.
Takeaways & Limitations
The achieved capabilities remain far from general-purpose human manipulation, and the pipeline’s components still require substantial improvement.
Abstract
from arXiv · showhide
Learning generalizable robot manipulation policies, especially for complex multi-fingered humanoids, remains a significant challenge. Existing approaches primarily rely on extensive data collection and imitation learning, which are expensive, labor-intensive, and difficult to scale. Sim-to-real reinforcement learning (RL) offers a promising alternative, but has mostly succeeded in simpler state-based or single-hand setups. How to effectively extend this to vision-based, contact-rich bimanual manipulation tasks remains an open question. In this paper, we introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three challenging dexterous manipulation tasks: grasp-and-reach, box lift and bimanual handover. Our method features an automated real-to-sim tuning module, a generalized reward formulation based on contact and object goals, a divide-and-conquer policy distillation framework, and a hybrid object representation strategy with modality-specific augmentation. We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors -- highlighting that vision-based dexterous manipulation via sim-to-real RL is not only viable, but also scalable and broadly applicable to real-world humanoid manipulation tasks.
1 Introduction
The paper addresses the challenge of scaling sim-to-real RL from limited single-hand or state-based settings to vision-based, contact-rich bimanual humanoid manipulation. It presents a practical recipe and evaluates it across three tasks with strong performance on seen and novel objects.
- Existing sim-to-real manipulation methods are largely limited to single-hand or state-based setups, leaving vision-based bimanual humanoid tasks unresolved.
- The proposed recipe targets low-cost multi-fingered humanoids through automated real-to-sim tuning and keypoint-based reward design for coordinated manipulation.
- Three tasks—dexterous grasp-and-reach, bimanual lifting, and bimanual handover—are used to evaluate the approach.
- 90% success was achieved on seen objects, while novel objects reached 60∼80% success in zero-shot sim-to-real evaluation.
- The policies exhibited robust, adaptive, generalizable behavior and transferred across two distinct multi-fingered robot hands.
2 Background
The background identifies exploration, environment modeling, and reward specification as central obstacles when applying RL to robotics. The included recipe components frame these obstacles through autotuned modeling, approximate object representations, and contact- and object-goal rewards.
- Deep RL is brittle because of hyperparameter sensitivity and algorithmic variance that can undermine reproducibility.
- Real-world robotics makes exploration difficult through high-dimensional inputs, sparse rewards, and complex dynamics.
- Robotics requires practical solutions for incompletely modeled environments and poorly defined reward functions.
- The recipe represents manipulation rewards as a combination of contact goals and object goals.
- The background organizes modeling around approximate object geometry and tunable robot physics and URDF parameters.
C. Sample Efficient Policy Learning
The paper situates its sample-efficient policy-learning strategy within prior manipulation research, contrasting costly demonstrations and restricted RL settings with its broader sim-to-real objective.
- Vision-based sim-to-real transfer must address domain gaps in both dynamics and perception.
- The recipe combines automated modeling, structured rewards, task-aware initialization, divide-and-conquer distillation, and mixed object representations.
- The paper’s approach is positioned as extending prior task- or hardware-specific techniques toward full-hand-arm, vision-based humanoid manipulation.
- Earlier imitation-learning approaches make progress in dexterous manipulation but remain costly to scale and require large real-world datasets.
- Earlier RL manipulation studies generally use single-hand setups or intermediate object representations rather than raw pixels.
3 Our Recipe
The recipe combines calibrated simulation, structured rewards, sample-efficient policy learning, and augmented hybrid visual representations. Together, these components address modeling, coordination, exploration, and perception challenges in dexterous sim-to-real manipulation.
- 3 Our Recipe: The pipeline addresses four challenges with real-to-sim modeling, generalizable reward design, sample-efficient policy learning, and vision-based sim-to-real transfer.
- Real-to-Sim Modeling: Autotuning jointly calibrates simulator physics and URDF constants against real-robot tracking data in under four minutes.
- Real-to-Sim Modeling: The autotune module searches parameter sets in parallel simulations and selects those minimizing tracking error against real trajectories.
- Real-to-Sim Modeling: Approximate object models use simple geometric primitives with randomized physical parameters to support reliable real-world transfer.
- Generalizable Reward Design: The reward design decomposes long-horizon manipulation into hand-object contact transitions and object-state changes.
- Generalizable Reward Design: Contact markers encode fingertip contact preferences and can be procedurally generated from object geometry or human priors.
- Sample Efficient Policy Learning: Task-aware initialization samples object poses and robot joint positions from human-guided hand-object configurations.
- Sample Efficient Policy Learning: Divide-and-conquer distillation reduces exploration difficulty by decomposing multi-object manipulation into multiple single-object tasks.
4 Experiments
Experiments evaluate the recipe across three vision-based manipulation tasks, its modeling and reward components, policy-learning strategies, and robustness. Results show successful task execution, improved transfer and exploration, generalization to unseen objects, and extension to longer-horizon control.
- Experimental setup: The system evaluates grasp-and-reach, box lift, and bimanual handover on a Fourier GR1 humanoid using two cameras and robot proprioception.The experiments also include Inspire hands to assess cross-embodiment generalization.
- Object modeling: Primitive geometric object models converge faster than complex geometries while randomized primitives generalize to diverse unseen objects.The comparison uses grasp-and-reach training with different object sets.
- Reward design: Contact-marker placement produces corresponding contact behaviors, supporting contact goals as a way to specify manipulation preferences.The box-lift study varies markers across side centers and edges.
- Policy initialization: Human-guided hand-pose initialization significantly enhances exploration efficiency in challenging reinforcement-learning tasks.Table 2 compares the percentage of successful policies across tasks and ten random seeds, defining success as over 60% episodic evaluation success.
- Policy distillation: 90.0% transfer success was achieved by the mix distillation policy, compared with 63.3% for shape, 40.0% for single, and 23.3% for all.The evaluation tests each policy type on an in-distribution object over 30 trials.
- System capabilities: The best policies achieve 62.3% grasp-and-reach, 80% box-lift, and 52.5% bimanual-handover success, while remaining robust to external forces and supporting scripted longer-horizon execution.Combining depth with 3D object position improves transfer, and policies qualitatively transfer zero-shot to out-of-distribution objects.
5 Conclusion
The paper presents a comprehensive sim-to-real RL recipe for vision-based dexterous manipulation on humanoids. The learned policies generalize to unseen objects, resist force disturbances, and perform long-horizon contact-rich tasks.
- The recipe addresses environment modeling, reward design, policy learning, and sim-to-real transfer for humanoid manipulation.
- Learned policies generalize strongly to unseen objects and remain robust against force disturbances.
- The policies perform long-horizon, contact-rich manipulation tasks without extensive human demonstrations.
6 Limitations
The proposed pipeline demonstrates a feasible path toward generalizable, robust, and dexterous robot manipulation, but substantial limitations remain. The authors identify gaps in general-purpose capability, sim-to-real dynamics modeling, and hardware dexterity.
- The pipeline provides evidence of generalizability, robustness, and dexterity in robot manipulation.
- The demonstrated capabilities remain far from human-level general-purpose manipulation.
- Naive domain randomization is the only technique used to reduce the sim-to-real dynamics gap, which may contribute to low bimanual handover success.
- Reliable dexterous-manipulation hardware constrains the work because the robot hands have fewer active degrees of freedom than human hands.
Appendix
The appendix details environment modeling, structured rewards, specialist-to-generalist distillation, and implementation choices. These components combine approximate simulation, contact-and-object goals, privileged specialist training, and vision-compatible policy learning.
- 6.1 Environment Modeling Details: Underactuated joints are approximated with a tunable linear relation, qu = k · qa + b, between actuated and underactuated joint angles.
- 6.2 Reward Design Details: Grasp and lift rewards combine finger-contact goals with object-position goals for lifting.
- 6.2 Reward Design Details: Handover rewards use a stage variable to switch between first-hand and second-hand contact and object-goal terms, with bonuses for completed stages.
- 6.3 RL and Distillation Details: Specialist policies use object and robot states with PPO and asymmetric privileged critics, while generalist policies use robot states, object position, and egocentric depth.
- 6.4 Distillation Details: Successful specialist trajectories are saved as demonstrations for training task-specific generalist policies with Diffusion Policies.
- 6.4 Distillation Details: The generalist architecture encodes proprioception and object position with an ELU network and depth observations with ResNet-18 before diffusion-based action prediction.