Source-linked AI summary
Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning
Patrick Yin, Tyler Westenbroek, Zhengyu Zhang, Joshua Tran, Ignacio Dagnino, Eeshani Shilamkar, Numfor Mbiziwo-Tiapo, Simran Bagaria, Xinlei Liu, Galen Mullins, Andrey Kolobov, Abhishek Gupta
TL;DR
Robotic RL for long-horizon dexterous manipulation remains brittle because exploration repeatedly visits narrow state spaces and relies on task-specific engineering. OmniReset addresses this with diverse simulator resets and large-scale on-policy RL, then distills the learned policies for vision-based transfer. The resulting policies solve broad manipulation tasks and achieve substantially higher zero-shot real-world success than a behavior-cloning baseline, within a scope defined by user-specified rigid-body manipulation tasks.
Problem
Robotic RL manipulation requires extensive task-specific engineering and struggles when exploration repeatedly samples narrow state–action regions, limiting robust long-horizon learning.
Method
OmniReset automatically generates diverse manipulation-centric reset distributions and combines them with large-scale on-policy RL, using a shared reward without curricula or demonstrations.
Results
OmniReset solves long-horizon contact-rich tasks across broad initial-state ranges and transfers distilled RGB policies zero-shot, achieving 85.37% on Peg, 56.36% on Leg, and 15.38% on Drawer.
Takeaways & Limitations
Diverse simulator resets let large-scale RL produce complex dexterous behavior and support robust zero-shot real-world transfer without task-specific curricula or demonstrations.
Takeaways & Limitations
OmniReset assumes rigid-body manipulation in which a single user-specified object must be moved to a target configuration, requiring user-provided target, goal, and workspace information.
Abstract
from arXiv · showhide
Reinforcement learning in massively parallel physics simulations has driven major progress in sim-to-real robot learning. However, current approaches remain brittle and task-specific, relying on extensive per-task engineering to design rewards, curricula, and demonstrations. Even with this engineering, they often fail on long-horizon, contact-rich manipulation tasks and do not meaningfully scale with compute, as performance quickly saturates when training revisits the same narrow regions of state space. We introduce OmniReset, a simple and scalable framework that enables on-policy reinforcement learning to robustly solve a broad class of dexterous manipulation tasks using a single reward function, fixed algorithm hyperparameters, no curricula, and no human demonstrations. Our key insight is that long-horizon exploration can be dramatically simplified by using simulator resets to systematically expose the RL algorithm to the diverse set of robot-object interactions which underlie dexterous manipulation. OmniReset programmatically generates such resets with minimal human input, converting additional compute directly into broader behavioral coverage and continued performance gains. We show that OmniReset gracefully scales to long-horizon dexterous manipulation tasks beyond the capabilities of existing approaches and is able to learn robust policies over significantly wider ranges of initial conditions than baselines. Finally, we distill OmniReset into visuomotor policies which display robust retrying behavior and substantially higher success rates than baselines when transferred to the real world zero-shot. Project webpage: https://weirdlabuw.github.io/omnireset/
1 INTRODUCTION
OmniReset addresses the brittleness and exploration bottleneck of robotic RL by using diverse simulator resets to expose policies to recurring dexterous interactions. This enables large-scale RL to learn long-horizon manipulation without task-specific scaffolding and supports zero-shot real-world transfer.
- Robotic RL for manipulation remains less successful than for locomotion and navigation because robust policies require extensive per-task environment and reward engineering.
- Standard exploration saturates with increased parallelism by repeatedly sampling narrow state–action distributions and becoming trapped in local minima.
- Task-specific rewards, curricula, and demonstrations reduce exploration burden but limit scalability and embody the assumption that dexterity needs additional scaffolding.
- OmniReset densely covers recurring interaction modes through randomized, generic resets, allowing sparse rewards to propagate and behaviors to be stitched into long-horizon strategies.
- OmniReset automatically generates diverse initial-state distributions, enabling PPO to leverage increasing compute and learn multi-stage behaviors without shaping, curricula, or demonstrations.
- The resulting policies solve contact-rich long-horizon tasks beyond existing methods, succeed across broad initial-state ranges, and transfer zero-shot through student-teacher visuomotor distillation.
2 RELATED WORK
Prior work uses resets, exploration bonuses, curricula, demonstrations, or imitation learning to make difficult RL problems tractable. OmniReset instead combines diverse task-agnostic resets with large-scale parallel RL, without requiring curricula or human demonstrations.
- Exploiting Resets in Reinforcement Learning: Prior reset-based methods include uniform initial-state sampling, explicit curricula, reverse curricula, learned reset proposals, and demonstration-generated pathways.
- Exploiting Resets in Reinforcement Learning: OmniReset differs by showing that diverse resets can scale to long-horizon manipulation without human demonstrations, curricula, or added algorithmic structure.
- Exploration Strategies for Reinforcement Learning: Bonus-based exploration rewards novel, rare, or unpredictable states, whereas OmniReset emphasizes large-scale parallelization and resetting schemes instead of advanced exploration incentives.
- The sim-to-real pipeline collects reaching, near-object, grasped, and near-goal resets, trains a state-based policy, distills it into an RGB policy, and deploys it zero-shot.
- Leveraging Demonstrations: Demonstration-based alternatives add behavioral-cloning losses, demonstrations to replay buffers, or reward terms that encourage following demonstrations.
3 GENERATING DIVERSE RESETS FOR LEARNING DEXTEROUS MANIPULATION
OmniReset constructs manipulation RL problems from diverse, task-agnostic reset distributions that cover pathways from approach through grasping to near-goal contact. PPO then learns behavior using a shared reward and large-scale parallel simulation rather than prescribed curricula or dynamics.
- Framework and Problem Setting: OmniReset automatically constructs manipulation RL problems by generating diverse reset distributions that expose algorithms to interaction states rarely encountered under naive exploration.
- Framework and Problem Setting: The framework formalizes each task as an MDP with state, action, transition, reward, discount, and initial-state distribution components.
- Task Scope and User Inputs: Users specify a target object, goal configurations, and robot workspace, allowing OmniReset to exploit task structure while keeping required inputs limited.
- Generating Diverse Resets: Reset generation covers transport pathways by placing the target on the tabletop, randomly in the air, and at or near the goal, while sampling feasible grasps and offsets.
- Reset Regions: Reaching, near-object, stable-grasp, and near-goal resets cover approach, contact initiation, prehensile manipulation, and contact-rich goal behaviors.
- Reset Regions: The reset regions approximate pathways without ordering states or specifying connecting dynamics, leaving those paths and behaviors to reinforcement learning.
- Practical Implementation: Feasible resets are sampled offline and filtered to avoid invalid physics states while preserving efficient GPU-parallel training.
- Algorithmic Decisions for RL Training: A shared reward combines task completion, object-goal distance, gripper reach, action smoothness, and termination penalties, with fixed components and weights across tasks.
4 SIMULATION EXPERIMENTS
OmniReset is evaluated across diverse manipulation tasks, baselines, initial-condition ranges, perturbations, and reset-distribution ablations. It consistently handles broader and longer-horizon settings where baselines struggle, while performance depends on substantial parallelism and reset diversity.
- Learning Curves and Success Rates: OmniReset consistently achieves high success on Hard tasks, substantially outperforming baselines that struggle with wider initial-condition distributions.The evaluation covers Hard variants of Leg Twisting, Drawer Insertion, Peg Insertion, Cube Stacking, Wall Slide, and Cupcake Placement.
- Learning Curves and Success Rates: Baselines can solve near-goal portions of tasks but fail to scale from reaching states to the full long-horizon task.The analysis compares success from Near-Goal and Reaching reset regions.
- Emergent Curricula: OmniReset naturally learns backward from near-goal states toward the full search space without a hand-designed curriculum.This progression emerges from diverse resets and many parallel environments.
- Robustness of Policies: OmniReset remains barely affected by large perturbations, whereas baseline performance quickly degrades under small perturbations.Success is measured after perturbing demonstration-derived initial conditions with forces of different magnitudes.
- Ablations: A large number of parallel environments is essential for scaling from reaching states to the full multi-stage task.Smaller environment counts can make progress from Near-Goal states, but do not support the full task complexity.
- Ablations: Broader grasp sampling improves sample efficiency and success rate compared with narrower reset ranges.The grasp-range ablation is performed on Leg Twisting Hard.
5 DISTILLATION AND REAL-WORLD TRANSFER
OmniReset policies are distilled into visuomotor controllers and transferred zero-shot from simulation to real hardware. Across three tasks, they achieve task-dependent real-world success rates, outperform a demonstration-trained behavior-cloning baseline, and exhibit retrying behavior.
- Real-World Transfer: 85.37% Peg, 56.36% Leg, and 15.38% Drawer zero-shot real-world success are achieved by distilled RGB policies.The policies were trained on 80,000 simulation trajectories spanning a wide range of initial conditions.
- Real-World Transfer: The distilled policies substantially outperform a Diffusion Policy behavior-cloning baseline achieving ∼2% success across tasks from 100 demonstrations.The comparison uses real-world deployment performance.
- Real-World Transfer: OmniReset policies are deployed on Peg Insertion, Leg Twisting, and Drawer Insertion using RGB observations from three RealSense cameras.Control and policy inference run on a PC with an RTX 4090 GPU.
- Evaluation Metrics: First-try success and throughput reveal remaining efficiency and reliability gaps in real-world deployment.First-try success requires completing the task with one grasp and execution, without dropping and regrasping.
- Visual Randomization: Visual randomization varies lighting, backgrounds, object and robot appearance, workspace textures, camera pose, and field of view to improve robustness to visual variation.Training also uses color jitter, blur, grayscale, and noise augmentations.
- Real-World Transfer: The visuomotor policy exhibits robust retrying behavior by recovering from initial failures and completing the task.The paper presents this as evidence of broader initial-condition handling after sim-to-real transfer.
6 CONCLUSION
OmniReset uses diverse, minimally structured reset states with large-batch on-policy simulation to produce complex dexterous behaviors and robust real-world transfer. Its scope remains bounded by grasp-generation quality and uncertain extension to bimanual or dexterous-hand settings.
- OmniReset combines diverse reset states with large-batch on-policy reinforcement learning to generate complex dexterous behavior.The framework provides a general-purpose recipe for simulation data generation across manipulation tasks.
- The resulting policies demonstrate robust behavior in simulation and can transfer directly to the real world.
- Grasp-sampler quality limits OmniReset when complex, highly non-convex objects lack diverse grasps.
- Whether OmniReset scales to bimanual manipulation or dexterous hands remains an open question because stable grasp precomputation is more challenging.
7 REPRODUCIBILITY STATEMENT
The paper describes its data-generation, training, distillation, transfer, and experimental pipelines to support reproducibility, with additional ablations in the Appendix.
- The authors document the data-generation and training pipeline, distillation and transfer pipeline, and experimental results.
- Additional ablation studies are provided in the Appendix.
A.1 DETAILED REWARD SPECIFICATION
The task-agnostic reward combines shared safety, regularization, and task-related terms with fixed coefficients across environments. Its sparse success signal activates when position and orientation errors meet predefined thresholds, while performance is primarily attributed to initial-state coverage rather than reward shaping.
- The reward is a weighted sum of safety and task-related terms shared across environments with identical weights and no task-specific tuning.
- Safety regularization penalizes large actions, rapid action changes, excessive joint velocities, and invalid or unsafe robot states.
- Task-related terms encourage end-effector approach and reduce the manipulated object's relative position and orientation errors in the goal frame.
- The sparse success reward activates when both position and orientation errors fall below predefined thresholds.
- λreach = 0.1, λdist = 0.1, λsuccess = 1.0.
- Performance is largely insensitive to moderate weight changes, with initial-state coverage identified as the primary driver of successful learning.
A.2 ADDITIONAL SIMULATION RESULTS
Figure 12 compares Leg Twisting success rates from Near-Goal and Reaching regions using resets sampled from baseline demonstrations, keeping evaluation states in distribution.
- Figure 12 reports Leg Twisting success rates from Near-Goal and Reaching regions of the state space.
- Evaluation resets are sampled from baseline demonstrations to ensure policies begin from in-distribution states.
A.3.1 ROBOT KINEMATICS
Reliable sim-to-real transfer depends on matching robot kinematics, dynamics, control, contact behavior, and sensing between simulation and hardware. The paper reports several calibration and controller choices that improve transfer, while residual contact and policy-distribution mismatches remain important limitations.
- Robot kinematics: Factory-specific URDF calibration reduces systematic pose errors that otherwise degrade precision tasks such as insertion.Each physical UR7e deviates from its nominal model because of manufacturing tolerances in link lengths and joint offsets.
- Robot dynamics and system identification: Under 2 degree RMSE after system identification contrasts with approximately 7 degree RMSE using zero-default actuator parameters.The identified parameters include friction, armature, and motor delay, estimated from chirp trajectories using CMA-ES.
- Contact modeling: Contact-rich behaviors such as peg reorientation against a hole often fail to transfer because small contact-dynamics discrepancies remain despite randomization.The paper identifies more accurate contact models or multiple physics backends as directions for future work.
- Controller design: Task-space control outperforms joint-space PD and inverse-kinematics-plus-joint-PD formulations, which produce more jamming, unstable motion, and worse learned-policy performance.The paper notes that controller design affects both RL training and distilled-policy performance.
- Action space: Gradually adjusting action scale is more stable than clipping actions, which introduces discontinuities that can degrade optimization and collapse RL training.The adjustment is a parameter curriculum for sim-to-real stability rather than a curriculum over behaviors or state distributions.
- Student-teacher distillation: RGB policy performance remains approximately 50% in simulation, while larger datasets improve real-world transfer despite similar simulated performance across 10K–80K trajectories.Camera placement also matters: wrist-mounted and close third-person cameras outperform top-down viewpoints.