Source-linked AI summary
Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learning
Juri Pfammatter, Kaixian Qu, Clemens Schwarke, Victor Klemm, Marco Hutter
TL;DR
Sparse-reward, long-horizon goal-conditioned reinforcement learning can leave policies without a learning signal when they rarely reach the goal. BVER grows intermediate starts from the goal and goals from the initial-state distribution, using one shared policy. It learns fastest among reference-free curricula on the compared tasks and outperforms either expansion direction alone.
Problem
With sparse rewards and long horizons, a policy that rarely reaches its goal receives no learning signal.
Source: §1 · p. 2Method
BVER grows starts outward from the goal and goals outward from the initial-state distribution, biases both toward unexplored task space, and trains one goal-conditioned policy on both.
Source: §1 · p. 2Results
BVER learns fastest among reference-free curricula on all six compared tasks and outperforms either expansion direction alone.
Source: §1 · p. 2, §5.4 · p. 9Takeaways & Limitations
In simulation, BVER yields sample-efficient and robust policies in navigation, locomotion, and manipulation.
Source: §6 · p. 9BVER relies on arbitrary-state simulator resets, bounded task spaces with meaningful distances, and approximately reversible dynamics; higher-dimensional or non-Euclidean spaces remain untested.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Long-horizon tasks with sparse rewards pose an exploration bottleneck for goal-conditioned reinforcement learning: a policy started from the initial state rarely reaches the goal and receives no learning signal. Reference motions, hand-designed curricula, and shaped rewards supply this signal but require demonstrations or task-specific engineering; automatic start-state and goal curricula avoid this but typically expand from one side only, so the full distance to the target must be covered from that side. We propose the Bidirectional Voronoi-biased Exploration curriculum for Reinforcement learning (BVER), which expands from both ends at once. Inspired by bidirectional RRT planning, BVER grows start states outward from the goal and goals outward from the initial state distribution, biases both toward unexplored task space, and steers them toward each other, training one goal-conditioned policy on both. On point-mass mazes, quadrupedal box climbing, and robot-arm ring-on-peg transfer, BVER learns faster than all compared reference-free curricula. On box climbing, it reaches 95% success on a 0.4 m box in roughly 65% fewer iterations than the best of them, is the only one of them to learn to climb a 0.7 m box, and yields a policy robust to start, goal, and yaw variation. Without a demonstration, it approaches the sample efficiency of reference-based curricula on the 0.4 m box and on ring-on-peg transfer. Ablations show that expanding from both ends outperforms either direction alone.
1 INTRODUCTION
Sparse rewards and long horizons can leave goal-conditioned policies without a learning signal, while common sources of guidance require demonstrations or task-specific design. BVER combines two-sided curriculum growth with Voronoi-biased exploration and learns faster than the compared reference-free curricula.
1 INTRODUCTION
- Reference motions, hand-designed curricula, dense rewards, and curiosity rewards can supply guidance, but require demonstrations, task-specific engineering, or extra reward terms.With sparse rewards and long horizons, a policy that rarely reaches its goal receives no learning signal.Source: §1 · p. 2
- Existing start-state and goal curricula expand from one end only, leaving a frontier to cover the full distance to the fixed end.Start-state curricula use reference states or random walks from the goal; goal curricula keep starts fixed and propose intermediate goals.Source: §1 · p. 2
- BVER grows start states outward from the goal and goals from the initial-state distribution, biases expansion toward unexplored task space, connects the two fronts, and trains one shared goal-conditioned policy.It keeps tasks of intermediate difficulty for the current policy while expanding across the task space.Source: §1 · p. 2
- BVER learns fastest among reference-free curricula on all six compared tasks, transfers to scanned terrains with the same hyperparameters, and is the only one to climb the 0.7 m box.The experiments also find that BVER outperforms either expansion direction alone.Source: §1 · p. 2
2 RELATED WORK
Prior work generates curricula through learner-progress or reward-based environment changes, intermediate starts or goals, and planning-inspired exploration. BVER relates these lines by growing starts and goals toward each other while biasing expansion toward unexplored space.
2 RELATED WORK
- Automatic curricula can adapt environment parameters, replay levels, level difficulty, or intrinsic rewards; BVER instead changes episode starts and pursued goals.The passage describes BVER as complementary to these approaches.Source: §2 · p. 2
- Start-state curricula include random walks outward from the goal, approximate-model backward reachable sets, demonstration-derived resets, and exploration from archived states.Demonstration-based resets may sample reference states uniformly, move backward on a schedule, or prioritize by success rate.Source: §2 · pp. 2–3
- Goal curricula propose intermediate goals or select achieved states by progress, achievability, novelty, relevance, or low density; some use replay-buffer states and avoid resets.BVER grows starts and goals toward each other and uses Voronoi bias rather than visit counts or density.Source: §2 · p. 3
- RRT selects vertices through random samples to bias growth toward unexplored space, while RRT-Connect grows trees from both endpoints; BVER grows a curriculum rather than a path.Florensa et al. proposed combining start-state and goal generation as future work, citing RRT-Connect as an analogue.Source: §2 · p. 3, Figure 2 · p. 3
- Robot-learning alternatives include terrain curricula with dense rewards, reference-based resets, hand-designed obstacle progressions, and curiosity rewards; BVER targets settings without that guidance.Examples include parkour rewards, jumps from selected heights, and curricula that grow obstacle dimensions or navigation distances.Source: §2 · p. 3
3 PRELIMINARIES
The preliminaries define a sparse-reward goal-conditioned task and its target objective, then formalize curricula as changing the training distribution over start–goal pairs. BVER mixes intermediate-start and intermediate-goal tasks for one shared policy.
3.1 GOAL-CONDITIONED MARKOV DECISION PROCESS
- The task is a goal-augmented MDP with a known state-to-goal mapping, binary reward for reaching within tolerance, and fixed-horizon episodes that continue after goal achievement.The target is to reach a single goal from the initial-state distribution; the objective is expected return under that target task.Source: §3.1 · p. 4
- The setup assumes arbitrary-state simulator resets and random-action rollouts, a sampleable initial distribution, a known target-goal state, and a bounded sampleable task space.Source: §3.1 · p. 4
3.2 CURRICULUM
- A curriculum replaces the target-only start–goal distribution with a distribution over auxiliary start–goal pairs while leaving the transition kernel and reward unchanged.The training tasks adapt to the learner; the objective remains the target task.Source: §3.2 · p. 4
- BVER combines intermediate starts with the target goal and initial-distribution starts with intermediate goals, mixing both task types for one shared goal-conditioned policy.Start-state curricula keep the goal fixed, whereas goal curricula keep starts drawn from the initial distribution.Source: §3.2 · p. 4, §3.1 · p. 4
4 METHOD
BVER builds a curriculum from frontier states, using task-space coverage to drive expansion outward. It trains one shared goal-conditioned policy on two expansions that grow toward each other.
4.1 VORONOI-BIASED EXPANSION
- Training on the mixed set combines intermediate-difficulty frontier states with replayed solved states, which mitigates catastrophic forgetting.This exploitation step is optional on tasks without forgetting, where the mixed set only seeds random walks.Source: §4.1 · p. 5
- Uniform task-space samples select mixed-set neighbors by Voronoi-cell volume, favoring frontier states whose cells extend into unexplored regions.Random walks from these seeds provide candidates for subsequent rollouts.Source: §4.1 · p. 5
4.2 BIDIRECTIONAL EXPANSION
- The forwards expansion exchanges start and goal roles, growing intermediate goals from reachable starts while storing rollout end states as walk seeds.The stored states are sorted by the return of their commanded goals; rollouts that reach and hold a goal end within ϵ of it.Source: §4.2 · p. 5
- BVER trains one shared policy on both initial-state/intermediate-goal and intermediate-start/target-goal pairs, splitting the training budget evenly between expansions.This covers reachable task space from both directions, teaching the policy to reach the target from varied starts and varied goals from the initial distribution.Source: §4.2 · p. 5
- The two frontiers are steered toward each other by sampling from the other direction’s solved set instead of uniformly from task space.Forwards intermediate goals grow toward the target, while intermediate starts grow toward the initial distribution.Source: §4.2 · pp. 5–6
5 EXPERIMENTS
Across point-mass mazes, box climbing, and ring transfer, experiments compare BVER with PPO and curriculum baselines, then test terrain coverage, scanned-terrain transfer, and design ablations.
5.1 SAMPLE EFFICIENCY AND ROBUSTNESS ACROSS DOMAINS
- BVER learns fastest among reference-free curricula on all tasks; on the 0.4 m box it reaches 95% success roughly 65% fewer iterations than the best reference-free baseline.Source: Figure 4 · p. 7, §5.1 · p. 7
- BVER’s box-climbing policy maintains high success under yaw perturbations and uniformly varied start and goal positions.Source: Figure 4 · p. 7, §5.1 · p. 7
5.3 SCANNED TERRAINS
- With box-climbing hyperparameters, BVER learned all four scanned terrains, and its rocky-ascent starts and goals spread across reachable terrain rather than one path.Success was 91.0±1.7% on freestanding boulder I, 84.4±2.1% on freestanding boulder II, 77.2±2.2% on rocky ascent, and 59.8±4.4% on small stone field; vanilla PPO did not succeed on any.Source: §5.3 · p. 8, Figure 6 · p. 8
5.2 TRAINING COVERAGE
- On the three-route terrain, BVER traversed stairs (64%), pillars (21%), and ramp (15%) among successful episodes; vanilla PPO used only the ramp.Source: Figure 5 · p. 8, §5.2 · p. 8
5.4 ABLATION STUDIES
- BVER reached 75% success first on every task except a tie with backwards-only expansion for 0.7 m climbing up; each one-way direction missed 75% on two of eight tasks.Source: §5.4 · p. 8, Table 1 · p. 9
- No-connect and greedy connecting each slowed maze learning on all three; they required up to 2.1 and 1.9 times more iterations, respectively, than pconnect=0.5.Source: Table 1 · p. 9, §5.4 · p. 9
6 CONCLUSION
BVER expands starts from the goal and goals from the initial distribution, biasing both toward unexplored task space and steering them toward each other; in simulation, it yields sample-efficient, robust policies across navigation, locomotion, and manipulation.
6 CONCLUSION
- BVER grows starts from the goal and goals from the initial distribution, biases both toward unexplored task space, and steers them toward each other.The random walks run in parallel with policy rollouts and add about 28% to wall-clock training time in these experiments.Source: §6 · p. 9
- The method assumes arbitrary-state simulator resets, random-action rollouts, a bounded task space with a meaningful distance metric, and approximately reversible dynamics for backward expansion.All results are in simulation; hardware deployment is left to future work.Source: §6 · p. 9
- The authors used generative AI tools to create figures from non-AI-generated data, create and edit code, and improve paper readability, and reviewed the AI-assisted work.They checked generated figures against underlying data, tested AI-assisted code, and reviewed suggested text edits.Source: AI USE STATEMENT · pp. 9–10
- Experiments can be reproduced from the project website, whose code includes hyperparameters, random seeds, and run instructions; experiments used one NVIDIA RTX 4090 GPU and five seeds.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 ALGORITHMS
BVER maintains state sets for backward and forward expansions, trains on new and mixed states, and updates its frontiers and solved sets from rollout returns.
A.1 ALGORITHMS
- Each direction maintains candidate, frontier, solved, and mixed state sets; forward states become intermediate goals, while backward states become intermediate starts.The mixed set supplies exploitation rollouts, and a fraction of solved states is replayed when it is rebuilt.Source: §A.1 · pp. 13–14
- For forward expansion, storing the achieved final state rather than the commanded goal enables full-state random-walk resets; this acts as hindsight relabeling.Exploitation is optional and was omitted on mazes and ring-on-peg transfer, where exploration-only rollouts did not lead to forgetting.Source: §A.1 · p. 13
- Random-walk expansion starts from the nearest neighbor of a sampled task-space point, then simulates action-noise trajectories to generate candidate states.With probability pconnect, the walk instead targets a state from the other expansion’s solved buffer.Source: §A.1 · p. 13
- Rollout returns sort states into the frontier when Rmin < R < Rmax and into the solved set when R ≥ Rmax.For forward expansion, the stored state is the rollout’s final state; for backward expansion, it is the initial state.Source: §A.1 · p. 13
- The overview combines backward starts-from-goal and forward goals-from-initial-distribution expansions, with the policy training on new states and the mixed set.Next seeds follow the Voronoi bias or, with probability pconnect, target the other expansion’s solved set.Source: Figure 7 · p. 14
A.2 EXPERIMENT SETUP
Experiments use PPO across point-mass mazes, quadruped locomotion and terrain tasks, and ring-on-peg manipulation, with domain-specific BVER settings and matched baselines.
A.2 EXPERIMENT SETUP
- All policies use the RSL-RL PPO implementation; the experiment tables list PPO and BVER hyperparameters, with some BVER settings shared across tasks and others domain-specific.The connect ratio and subsamples per walk are shared; all hyperparameters were selected empirically per domain in preliminary runs.Source: §A.2 · p. 14, Table 2 · p. 15, Table 3 · p. 16
- Point-Maze uses a velocity-controlled point mass in a walled grid, with a sparse reward within 1 m of the goal and horizons of 9, 32, and 15 seconds across its three mazes.The 2D action commands planar velocity; the observation includes position, velocity, and the vector to the goal.Source: §A.2 · p. 14
- Climb and scanned-terrain tasks use ANYmal D with a 50 Hz joint-position controller and sparse goal rewards, with no shaping terms.Episodes last 4 seconds on climb tasks, 12 seconds on Rocky Ascent, and 10 seconds on the other scans.Source: §A.2 · pp. 14–15
- Three-Route Terrain adds a distance-potential shaping term shared by vanilla PPO and BVER, while BVER’s reward band is evaluated on the sparse term only.Route analysis uses 128 episodes after 2,000 iterations for one seed.Source: §A.2 · p. 15
- Ring-on-peg Transfer uses a stiff-joint Franka Panda to move a hand-fixed ring between peg bottoms, with a 3 cm goal tolerance and an 8-second horizon.Its reward matches the point-mass mazes, measured in three dimensions at the ring center.Source: §A.2 · p. 15
- Baselines share PPO hyperparameters and simulation steps per iteration with BVER; RSI and Backplay use states from a successful BVER-policy demonstration, while the random curriculum draws offline collision-free states.Reverse Curriculum was implemented in Isaac Lab following the original algorithm as closely as possible.Source: §A.2 · p. 15
A.3 ABLATION LEARNING CURVES
The ablations favor bidirectional expansion and an intermediate connect ratio: direction-only variants depend on the task, while both extreme connect settings learn later than the default.
A.3 ABLATION LEARNING CURVES
- Unidirectional variants often plateau below BVER on ring-on-peg transfer, where each expands from its peg into the intervening free space but rarely reaches the other peg.Source: §A.3 · p. 16, Figure 8 · p. 16
- Combining forward and backward expansions reaches 75% success first on all tasks, tied with backward expansion on the 0.7 m climbing task.On ring-on-peg transfer neither direction alone reaches 75%; on box climbing, the expansion starting on top is stronger.Source: Figure 8 · p. 16, §A.3 · p. 16
- On all three mazes, pconnect = 0.5 learns fastest; pconnect = 0 and pconnect = 1 both rise later.Without connecting, frontiers spread over task space; greedy connecting loses the coverage bias.Source: §A.3 · p. 16, Figure 9 · p. 17