PaperlayerView exact revision on arXiv

Source-linked AI summary

A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

Octi Zhang, Mateo Guaman Castro, Patrick Yin, Ignacio Dagnino, Abhishek Gupta, Rosario Scalise, Byron Boots

arXiv:2610.12465v1cs.ROcs.LG

TL;DR

Robot RL often requires per-task engineering, while uniform sampling can waste experience on configurations already mastered or not yet solvable. The paper introduces Success Guided Sampling (SGS), which adapts sampling to policy success rates. Across challenging locomotion and assembly tasks, SGS outperforms prior samplers in reported comparisons and transfers distilled RGB manipulation policies zero-shot to hardware.

  • Problem

    Robot RL pipelines commonly rely on task-specific rewards, curricula, and demonstrations, while uniform sampling can spend experience on mastered or currently unsolvable configurations.

    Source: §1 · p. 2

  • Method

    SGS adapts task-configuration sampling using non-parametric estimates of policy success, weighting configurations toward moderate success rates.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across quadruped locomotion and nut-and-bolt assembly, SGS solves harder tasks than prior samplers under the same budget; at 1M parallel environments, multi-task locomotion mean success is 73% for SGS versus 54% for PLR.

    Source: §1 · p. 2, §4.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The results identify adaptive choice of practice configurations as an important component of scaling robot RL.

    Source: §6 · p. 9

    SGS maintains success estimates over discrete cells, so continuous or much larger task spaces would require a learned or hierarchical scoring representation.

    Source: §5 · p. 9

Abstract

from arXiv · show

General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to $2^{20}$ (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: https://sgs-rl.github.io/.

1 Introduction

Robot RL still depends on per-task engineering, and uniform sampling wastes experience on configurations that are already mastered or too difficult. The paper introduces Success Guided Sampling (SGS) and evaluates it across locomotion and assembly, including zero-shot transfer of manipulation policies to hardware.

1 Introduction

  • Per-task reward tuning, curricula, demonstrations, and separate single-task experts make current RL pipelines engineering-heavy and add task-specific hyperparameters.The paper describes each new task as a fresh engineering project.Source: §1 · p. 2
  • Diverse simulator resets expose policies to useful intermediate states, and prior work found that combining programmatic resets with massively parallel RL can reduce reward and demonstration engineering [12].The paper frames reset distributions as an important component of scaling robot learning.Source: §1 · p. 2
  • Uniform sampling can waste experience on configurations the policy has mastered or cannot yet solve, limiting gains from increasing parallel environments.The paper reports this limitation particularly on challenging manipulation and locomotion tasks.Source: §1 · p. 2
  • SGS uses non-parametric success-rate estimates and beta-shaped weights to prioritize configurations with moderate success, adapting sampling without a manually ordered curriculum.Its sampler is guided by current policy performance.Source: §1 · p. 2
  • The experiments use the same SGS instantiation across quadruped locomotion and NIST assembly, without demonstrations, with one shared reward per domain and up to approximately one million parallel environments.Assembly tasks include threading, insertion, connector assembly, and gear meshing.Source: §1 · p. 2
  • SGS outperforms uniform sampling and PLR on challenging multi-terrain locomotion and nut-and-bolt assembly, matches uniform sampling on simpler tasks, and transfers RGB policies zero-shot to real hardware.The transfer covers several precise assembly tasks.Source: §1 · p. 2

2 Related Work

The related work situates SGS among sim-to-real RL, reset-based exploration, large-scale simulation, and automatic curricula. The authors present it as a simple success-based sampling variant, not a fundamentally new curriculum principle.

2 Related Work

  • Prior sim-to-real RL commonly relies on task-specific reward shaping, curricula, or demonstrations [14] [31] [9] [10] [11] [32]; this work combines diverse resets, adaptive sampling, and large-scale RL without demonstrations.The authors use one shared reward function within each domain.Source: §2 · p. 3
  • Reset-based exploration spans demonstration or goal-state curricula [15] [20], geometry-based sampling [36] [22], and OmniReset's programmatic diverse resets [12]; SGS adjusts sampling using empirical success estimates.SGS maintains estimates over a fixed set of task configurations.Source: §2 · p. 3
  • SGS complements parallel-RL work on exploration and population training [38] [39], loss estimation [40], and staggered resets [41] by allocating environments according to current success rates.The paper treats task-configuration allocation as a distinct component of training.Source: §2 · p. 3
  • Among automatic curricula, SFL prioritizes levels using p(1 − p); SGS instead uses rolling success estimates and smooth beta-shaped weighting, with an adjustable target and nonzero probability for every configuration.The paper evaluates SGS against uniform sampling and PLR on scaling benchmarks, and against SFL on locomotion.Source: §2 · p. 3
  • The authors do not claim a new curriculum principle; they show that success-based sampling with diverse resets and large-scale RL can solve challenging tasks prior sampling methods struggle to learn.The paper focuses on challenging locomotion and assembly tasks.Source: §2 · p. 3

3 Method

The method frames locomotion and manipulation as goal-conditioned RL over task configurations. SGS adapts configuration sampling using policy success rates, alongside shared domain rewards and parallel simulation.

3.1 Problem Setup

  • Goal-conditioned RL starts from an MDP whose task configurations combine initial state, goal, and environment; locomotion varies terrain and poses, while manipulation varies assembly and object states.Source: §3.1 · pp. 3–4

3.2 SGS: Success Guided Sampling

  • Using recent binary outcomes, SGS scores configurations with a beta-shaped target-success kernel and samples via softmax, favoring moderate success while retaining nonzero chances for extremes.Source: §3.2 · p. 4

3.3 Algorithmic Decisions for RL Training

  • The training design combines domain-shared rewards, avoiding task-specific reward functions, with parallel environments where SGS shifts rollouts toward configurations the policy is actively learning.Source: §3.3 · p. 5

4 Experimental Results

The experiments evaluate SGS on multi-terrain quadruped locomotion and contact-rich assembly across training scales, then test zero-shot RGB-policy transfer to UR5e hardware. SGS outperforms comparison samplers on challenging tasks at large and small scales, while the hardware trials show successful trajectories on three assembly tasks.

4.1 Locomotion

  • The locomotion study trains one Anymal-D policy from scratch across 13 terrains, including procedurally generated mazes, contoured terrain, and floating islands.The terrain set spans varied difficulty and requires precise locomotion, navigation, and chained agile maneuvers.Source: §4.1 · p. 6

4.2 Manipulation

  • Manipulation evaluation covers six UR5e assembly tasks and Franka nut-and-bolt assembly, using diverse simulator resets and the same simple reward across tasks.The tasks include threading, precise insertion, connector mating, and gear alignment; nut-and-bolt assembly requires threading an M16 nut about 6.5 revolutions.Source: §4.2 · pp. 6–7

4.3 SGS benefits from increased training scale.

  • At 1M environments, SGS reaches 73% mean success on locomotion versus 54% for PLR, and 70% on Franka nut-and-bolt versus 6% for uniform sampling and 5% for PLR.The scaling experiments report means over three training seeds, with fresh evaluation resets and fixed PPO and optimizer hyperparameters.Source: Figure 5 · p. 7, §4.3 · p. 7

4.4 SGS unlocks new capabilities even at small scale.

  • For UR5e rod-in-hole insertion, SGS outperforms uniform sampling and PLR at 32K environments, while at 256K environments all methods reach approximately 98% success.At 4,096 parallel environments, SGS achieves non-zero success on many locomotion terrains, while uniform sampling and other baselines fail to learn.Source: Figure 5 · p. 7, §4.4 · p. 8, Figure 8 · p. 9

4.3 SGS benefits from increased training scale.

  • SGS sampling shifts from uniform coverage early in training to level-island jumps midway and configurations reaching the center pillar late in training.The late configurations include a substantial height difference from the other islands.Source: Figure 7 · p. 8

4.5 Zero-shot deployment on real hardware

  • State-based manipulation teachers are distilled into RGB policies with DAgger and transferred zero-shot to UR5e hardware for nut-and-bolt, rod-in-hole, and gear-mesh assembly.The reported hardware evaluation includes success, first-try success, and throughput; the figure shows successful trajectories for all three tasks.Source: Figure 4 · p. 6, §4.5 · p. 8, Table 2 · p. 8

5 Limitations

The paper identifies limits in SGS’s scoring representation, its dependence on externally provided reset distributions, and the remaining sim-to-real gap.

5 Limitations

  • SGS maintains success-rate estimates over discrete cells; continuous or much larger task spaces would require learned or hierarchical scoring.The authors report that K = 32,768 is sufficient for the tasks studied.Source: §5 · p. 9
  • SGS samples from a reset distribution but does not construct it, so broad task coverage relies on reset-diversity methods such as OmniReset.Choosing which configurations to make available remains an open problem.Source: §5 · p. 9
  • A sim-to-real transfer gap remains, and closing it to match simulated success rates is left to future work.Source: §5 · p. 9

6 Conclusion

The conclusion presents SGS as a success-based sampler for training on moderately successful task configurations. The paper reports harder-task results under large-scale RL and diverse resets, plus zero-shot transfer of distilled RGB manipulation policies to UR5e hardware.

6 Conclusion

  • SGS focuses training on task configurations with moderate success rates.Source: §6 · p. 9
  • Combined with diverse resets and large-scale RL, SGS solves harder quadruped locomotion and nut-and-bolt assembly tasks than prior samplers.Source: §6 · p. 9
  • The manipulation policies are distilled to RGB and transferred zero-shot to UR5e hardware on three NIST board assembly tasks.Source: §6 · p. 9

A Task Specification Details

This appendix specifies how locomotion and manipulation tasks are configured, reset, observed, controlled, and rewarded. The two domains use different task configurations and policy-training setups.

A.1 Locomotion

  • Locomotion configurations combine terrain type, terrain level, initial state, and goal state; procedurally generated terrain difficulty may instead represent different seeds.The setup uses 2 seeds per terrain type, 10 difficulty levels, 20 start states, and 20 goal states, yielding 104,000 configurations.Source: §A.1 · p. 14

A.2 Manipulation

  • Manipulation trains separate task policies from a buffer of 32,768 collision-checked configurations balanced across three reset strategies, with a fixed fully assembled goal.Each episode draws a reset state from that buffer.Source: §A.2 · p. 14
  • Near-Goal starts place the object along its assembly path, Stable Grasp starts place it above the target, and Reaching starts may leave it unsecured anywhere in the workspace.Near-Goal ranges vary by task; rod-in-hole starts span fractions [0, 1.1] of the insertion path.Source: §A.2 · p. 14

A.3.1 Locomotion

  • Locomotion uses terrain scans and proprioceptive observations to control 12 joint-position offsets, with shared rewards combining terminal success and failure with per-step motion and posture terms.Forward-aligned motion reward is disabled for the first 500 training steps.Source: §A.3.1 · p. 15

A.3.2 Manipulation

  • Manipulation teachers observe joint and end-effector state plus a gripper-frame point cloud, and command Cartesian translation and rotation increments with a binary gripper action.Arm commands use an operational-space controller.Source: §A.3.2 · p. 15
  • Assembly rewards combine task success with safety and motion regularization; success uses task-specific orientation, lateral-position, and insertion-depth gates.Failure termination covers abnormal joint velocities or the held object leaving workspace bounds; per-step squared-norm penalties are clamped.Source: §A.3.2 · pp. 15–16

B Additional Experimental Results

On locomotion, SGS finishes above Sampling for Learnability (SFL) at both tested scales, while SFL’s performance declines during continued training, especially at larger scales.

B.1 Comparison with Sampling for Learnability

  • SFL finishes below SGS at both scales; its best 32K checkpoint comes close but still underperforms, and continued training degrades performance most at larger scales.SGS improves monotonically during training.Source: §B.1 · p. 16

B.2 SGS Hyperparameters and Configuration Density

These experiments assess SGS sensitivity to sampler settings and reset coverage. Performance varies across tested settings, configuration diversity helps up to a point, and Near-Goal coverage is important in the tested insertion runs.

B.2 SGS Hyperparameters and Configuration Density

  • 0.39 to 0.53 mean success across tested hyperparameter settings compares with 0.49 for the nominal setting; the authors report limited sensitivity but some benefit from a small search.The locomotion settings were tested at 32K environments with two seeds per setting.Source: §B.2 · p. 16, Table 3 · p. 17
  • At 4K environments, Normal and Beta kernels reach similar mean success: 0.45 and 0.46, respectively.Each kernel result averages three training seeds.Source: §B.2 · p. 16, Table 3 · p. 17
  • 0.38 to 0.49 success accompanies increasing patch density from p = 10 to p = 20; p = 40 yields 0.48, indicating a plateau at higher diversity.The corresponding configuration counts are 52,000, 104,000, and 208,000 at 32K environments.Source: §B.2 · p. 16, Table 3 · p. 17
  • For manipulation, the small grid search selected κ = 1 and t = 0.5 with T = 2, though the winning setting performed best by only a small margin.The sweep tested κ ∈ {1, 5, 10} and t ∈ {0.33, 0.5, 0.66}.Source: §B.2 · p. 17

B.3 Sensitivity to Reset-Distribution Coverage

  • Removing Near-Goal prevented successful learning in the tested rod-in-hole runs; removing Reaching left retained-distribution success essentially unchanged, while removing Stable Grasp reduced mean success.With Stable Grasp removed, one of three seeds collapsed while the other two retained high performance.Source: §B.3 · p. 17, Table 4 · p. 17

B.4 Gravity-Curriculum Ablation

A single Franka nut-and-bolt run without a gravity curriculum achieved success close to the curriculum run on Reaching resets.

B.4 Gravity-Curriculum Ablation

  • 69.4% success without the gravity curriculum versus 70.2% with it on Franka nut-and-bolt Reaching resets suggests the curriculum is unnecessary in this setting.The no-curriculum result used one training seed; the main manipulation experiments used a gravity curriculum.Source: §B.4 · p. 17

B.5 Additional UR5e Assembly Tasks for Transfer

Additional UR5e assembly results show SGS’s largest advantage on nut-and-bolt assembly, while the transfer experiments describe an RGB-student pipeline and its hardware behaviors. Symmetry-aware resets and rewards improve peg-insertion distillation, while simulation friction and held-out evaluation expose important training conditions.

B.5 Additional UR5e Assembly Tasks for Transfer

  • UR5e policies are evaluated from the Reaching initial-state distribution; nut-and-bolt uses 256K parallel environments and the other five tasks use 64K.These results are separate from the Franka nut-and-bolt and UR5e rod-in-hole scaling experiments.Source: §B.5 · p. 17, Table 6 · p. 18
  • 90.04% success on nut-and-bolt assembly with SGS, versus 48.83% for uniform sampling and 59.77% for PLR; SGS performs comparably to uniform sampling on five simpler insertions.In a separate rod-in-hole scaling experiment, SGS succeeds at 32K environments while uniform sampling and PLR achieve zero success; all reach about 98% at 256K.Source: §B.5 · p. 18, Table 6 · p. 18

C Sim-to-Real Transfer and Evaluation

  • The UR5e transfer setup builds on OmniReset’s pipeline and uses a Robotiq 2F-85 gripper with relative Cartesian operational-space control and third-person and wrist cameras.The NIST board is taped during evaluation because its polished surface reflects the robot in the wrist camera and its low friction impairs object reorientation.Source: C Sim-to-Real Transfer and Evaluation · p. 18, §C.1 · p. 18

C.2 Simulation Modeling and Domain Randomization

  • With max friction-combine mode, only pad friction µ = 2 trained rod-in-hole insertion stably among tested values µ = 0.5, 1, 1.5, and 2.The authors report that OmniReset’s µ = 100 setting made implausible grasps succeed in simulation, contributing to the sim-to-real gap.Source: §C.2 · p. 19, Figure 10 · p. 19, Figure 9 · p. 19

C.3 Teacher–Student Distillation

  • The point-cloud teacher is distilled to an RGB student with online DAgger, while 6.67% of environments are held out because training success can overstate independent performance.The student uses side and wrist RGB images plus robot state; held-out success lags training success and indicates longer distillation is needed.Source: §C.3 · pp. 19–20, Figure 11 · p. 20
  • 98% success in Table 7 follows symmetry-aware resets and rewards for OmniReset peg insertion, with the prose reporting 98–99% and eight equivalent grasp orientations.Using the same object symmetries for resets and rewards removes preferred orientations unavailable to the RGB student, easing imitation.Source: §C.3 · p. 20, Table 7 · p. 21

C.4 Transfer Observations and Policy Behaviors

  • Hardware transfer favored clean grasps followed by direct insertion over exploiting simulator contact dynamics; the nut policy completed assembly after removing an upright-orientation constraint and tuning action scaling.For rod insertion, 1 mm per-step Gaussian point-cloud noise encouraged re-aiming after misses and wiggling when the rod was not fully seated.Source: §C.4 · p. 21

C.5 Real-World Evaluation Protocol

The real-world evaluation reports overall success, first-try success, and throughput, with trial and retry definitions tailored to insertion behavior. Trials use randomized object positions on a board fixed to the table and run for task-specific durations.

C.5 Real-World Evaluation Protocol

  • Object positions are randomized over the board, which is fixed to the table; continuous runs last about 83 minutes for nut, 42 for rod, and 15 for gear.The evaluation collects 48, 49, and 50 trajectories for nut, rod, and gear, respectively.Source: §C.5 · p. 21
  • A first-try success completes insertion on the first attempt; a failed insertion followed by another attempt counts as a retry, while pre-attempt re-grasps are not penalized.This differs from OmniReset, which counts any re-grasp as a retry.Source: §C.5 · p. 21
  • Throughput is successful trajectories divided by total evaluation time, including reset time.Source: §C.5 · p. 21
Loading 2610.12465v1…