Source-linked AI summary
Stochastic Neural Networks for Hierarchical Reinforcement Learning
Carlos Florensa, Yan Duan, Pieter Abbeel
TL;DR
Sparse rewards and long horizons challenge naive deep-reinforcement-learning exploration. The paper pre-trains diverse skills with stochastic neural networks, mutual-information regularization, and a minimally specified proxy reward, then reuses them hierarchically; experiments report broad improvements across downstream tasks, with limitations around skill switching and fixed sub-policies or switch times.
Problem
Deep reinforcement learning struggles with exploration on tasks having sparse rewards or long horizons.
Method
The framework pre-trains a span of skills using Stochastic Neural Networks, a mutual-information bonus, and a single proxy reward, then trains high-level policies over those skills for downstream tasks.
Results
The experiments show that the approach learns a wide, interpretable span of skills sample-efficiently and boosts learning uniformly across a wide range of downstream tasks.
Takeaways & Limitations
Reusable skills and hierarchical policies provide a framework for improving exploration and addressing sparse-reward downstream tasks.
Takeaways & Limitations
The approach has unstable skill switching for some agents and uses fixed sub-policies and a fixed downstream switch time T.
Abstract
from arXiv · showhide
Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.
1 INTRODUCTION
Deep reinforcement learning has achieved major successes, but naive exploration performs poorly on sparse-reward tasks. The proposed framework pre-trains reusable skills with minimal task knowledge and applies them hierarchically downstream.
- Naive exploration strategies such as ϵ-greedy and uniform Gaussian noise perform poorly on tasks with sparse rewards.
- Hierarchical methods reduce search complexity by composing low-level actions into high-level primitives but require domain-specific knowledge and careful engineering.
- The framework learns a span of skills in a pre-training environment using a proxy reward requiring minimal downstream-task knowledge, then trains task-specific high-level policies over those skills.
- Stochastic Neural Networks represent multimodal policies with shared weights, supporting efficient pre-training of diverse skills.
2 RELATED WORK
Prior skill-learning approaches often depend on discrete domains, successful trajectories, demonstrations, specified goals, or restricted episodic settings. The proposed approach instead uses proxy rewards with minimal domain knowledge and supports hierarchical reuse across tasks.
- Automatic skill learning has largely focused on discrete domains, while continuous-action methods often extract skills from successful trajectories or demonstrations.
- HiREPS achieves multimodality but does not use policy gradients and was evaluated only when a single option remained active during each rollout.
- Option-critic can learn interpretable skills, but reuse across complex tasks remains an open question.
- Unlike a related pre-training approach requiring specified goals, this framework uses proxy rewards as its only pre-training signal and requires minimal domain knowledge or instrumentation.
3 PRELIMINARIES
The paper formulates reinforcement learning using finite-horizon discounted Markov decision processes and optimizes stochastic policies by their expected discounted return.
- A finite-horizon discounted MDP is defined by state and action spaces, transition probabilities, bounded rewards, an initial-state distribution, a discount factor, and a horizon.
- The stochastic policy πθ maps states to action probabilities and is parameterized by θ.
- The policy-search objective is to maximize expected discounted return over trajectories generated by the initial-state distribution, policy, and transition dynamics.
4 PROBLEM STATEMENT
The problem is to minimize total sample complexity across sparse-reward tasks that share agent dynamics and an action space while allowing task-specific state components. A minimally informed pre-training task supplies reusable skills for this setting.
- The downstream tasks must share structural regularities; without common structure, a skill set cannot be expected to accelerate learning across all tasks.
- Each task state factors into a common agent component and a task-specific component that interact only weakly.
- All tasks share the same action space, while task-specific state may include object positions or new sensory inputs.
- The algorithm aims to minimize total sample complexity across the task collection, including settings where all tasks have sparse rewards.
- The formulation uses a pre-training task constructed with minimal domain knowledge to address this challenging sparse-reward setting.
5 METHODOLOGY
The framework pre-trains diverse skills with a generic proxy reward, using stochastic neural networks and mutual-information regularization, then applies frozen skills through a high-level policy for sparse-reward tasks.
- 5.1 CONSTRUCTING THE PRE-TRAINING ENVIRONMENT: The method constructs a pre-training environment where skills are learned from a generic proxy reward requiring minimal downstream-task knowledge.The proxy reward encourages locally optimal solutions corresponding to potentially useful high-level behaviors.
- 5.2 STOCHASTIC NEURAL NETWORKS FOR SKILL LEARNING: Stochastic neural networks learn multiple skills simultaneously by integrating categorical latent variables with observations before a deterministic feed-forward network.The number of categorical classes K upper-bounds the number of skills.
- 5.2 STOCHASTIC NEURAL NETWORKS FOR SKILL LEARNING: Bilinear latent-variable integration changes the first hidden-layer weights and yields a large span of skills.The integration choice substantially affects skill quality.
- 5.2 STOCHASTIC NEURAL NETWORKS FOR SKILL LEARNING: A latent code is fixed throughout each rollout so that each code learns a temporally extended, consistent, interpretable skill.The resulting skills can later be selected for downstream tasks.
- 5.2 STOCHASTIC NEURAL NETWORKS FOR SKILL LEARNING: One SNN shares weights across skills and requires samples comparable to a single unimodal policy, making training K skills effectively K times more sample-efficient.An information-theoretic regularizer further encourages skill diversity and adds a mutual-information reward bonus based on relevant state subsets.
- 5.4 LEARNING HIGH-LEVEL POLICIES: A high-level policy selects a frozen skill for a fixed number of steps, using the resulting temporal consistency and skill quality to enhance sparse-reward exploration.The high-level policy samples among K skills, while low-level and high-level weights remain separate in the reported experiments.
6 EXPERIMENTS
The experiments evaluate the framework on hierarchical locomotion tasks with sparse rewards, including maze navigation and food collection, using a Swimmer pre-training setting and several maze variants.
- 6 EXPERIMENTS: The evaluation uses the Locomotion + Maze and Locomotion + Food Collection (Gather) tasks from the Duan et al. benchmark.The reported main results use the Swimmer robot, whose locomotion task matches the pre-training task.
- 6 EXPERIMENTS: The study constructs four mazes, including a benchmark maze, its reflection, and two variants placing the goal in opposite corners.These environments test navigation behaviors with different route requirements.
- 6 EXPERIMENTS: The sparse navigation reward is awarded only when the robot reaches the goal at the other end of the maze.The passage describes the goal placement and sparse-reward structure for the maze variants.
7 RESULTS
The experiments show that SNNs can learn diverse, interpretable skills efficiently, and that hierarchical use of pretrained skills substantially improves exploration and learning in sparse downstream tasks.
- The evaluation tests skill diversity, exploration, and sparse-task learning across the proposed hierarchical architectures.The experiments ask whether SNNs and the mutual-information bonus produce diverse skills, improve downstream exploration, and solve sparse tasks efficiently.
- Skill learning in pretrain: Independent feed-forward policies learn different advancement behaviors, while the proxy reward remains general enough to produce potentially useful downstream skills.Visitation plots measure Center-of-Mass positions over 100 rollouts of 500 time-steps, with each rollout reset to the same origin and orientation.
- Skill learning in pretrain: SNNs learn a similar or larger span of skills without training several independent policies, reducing sample complexity by the number of skills learned.Latent codes generate distinct, interpretable motions such as forward, backward, or sideways movement; bilinear integration produces at least forward and backward behaviors in 80% of trained SNNs.
- Hierarchical use of skills: Random Gaussian actions remain within the [−2, 2] × [−2, 2] region during a one-million-step rollout, whereas pretrained hierarchical policies substantially increase exploration coverage.The hierarchical rollouts use a randomly initialized Manager Network that changes one-hot skill selections every T = 500 steps.
- Hierarchical use of skills: The Multi-policy hierarchy concentrates exploration around upward and downward motion, while SNN hierarchies provide wider space coverage because their underlying policies span more skills.The Multi-policy baseline uses six independently pretrained policies and therefore six times more pretraining samples.
- Mazes and gather tasks: Hierarchical architectures learn much faster in every new sparse MDP, although the mutual-information bonus does not always improve performance.The proxy-reward baseline performs poorly because long horizons create credit-assignment difficulty, while hierarchies aggregate time-steps into useful primitives.
8 DISCUSSION AND FUTURE WORK
The framework combines diverse skill pretraining with hierarchical reuse to address sparse-reward tasks. Its limitations include unstable skill switching, fixed sub-policies and switch times, and feedforward decision-making.
- Discussion: The framework learns diverse skills with minimal supervision, then reuses them hierarchically to improve exploration in sparse-reward mazes and gathering tasks.Stochastic Neural Networks, bilinear integration, and a mutual-information bonus support a wide, interpretable skill span.
- Future work: Skill switching can be unstable for agents such as the Ant robot.The authors suggest transition policies or incorporating switching into pretraining as possible remedies.
- Future work: The current approach uses fixed sub-policies and a fixed downstream switch time T.End-to-end training or a learned termination policy could address these constraints, especially for more complex dynamic tasks.
- Future work: Feedforward architectures select the next skill using only the observation available at switching.Information gathered while the previous skill was active is not used in that decision.
A HYPERPARAMETERS
The experiments use fixed TRPO, network, SNN, and rollout settings across the downstream-task evaluations. Six skills are trained in the SNN and multi-policy comparisons.
- Hyperparameters: All policies use TRPO with step size 0.01 and discount 0.99, while each network has two layers of 32 hidden units.The stated networks include the multi-policy networks, SNN, and Manager Network.
- Hyperparameters: The SNN uses a mesh density of 10 divisions per unit for the mutual-information bonus.This grid is applied to the (x, y) space.
- Hyperparameters: The number of trained skills is six for both the SNN latent-variable dimension and the independently trained multi-policy setup.The same passage specifies a pretraining batch size of 50,000 and maximum path length of 500.
- Hyperparameters: Downstream-task algorithm parameters are measured in low-level-control time-steps.Table 1 presents the downstream-task parameterization in that unit.
B RESULTS FOR GATHER WITH BENCHMARK SETTINGS
Under the benchmark Gather settings, the SNN hierarchical approach outperforms VIME. Longer horizons expose a stronger advantage for hierarchy, while harder Snake tasks remain favorable to the proposed approach.
- Benchmark Gather: The SNN hierarchical approach outperforms the state-of-the-art intrinsic-motivation method VIME under the exact benchmark Gather settings.The comparison uses a maximum path length of 500 and batch size of 50k.
- Benchmark Gather: The Center of Mass speed-reward baseline is stronger than expected because nearby balls and the 500-step limit simplify Gather.With 5,000 steps, the hierarchy shows a much larger advantage.
- Snake: The Snake experiments use a five-link robot whose action dimension doubles and state space increases by 50% relative to Swimmer.The section also analyzes switch time and pretrained-SNN variation.
- Snake: Snake consistently learns a large span of skills across five pretraining random seeds with mutual-information bonus 0.05.The resulting visitation plots are reported in Figs. 9(a)-9(e).
- Snake: Despite larger Maze and Gather environments, the Snake hierarchical approach outperforms the Center of Mass intrinsic-motivation baseline.The robot, task sizes, and maximum path length are increased while batch size and switch time remain unchanged.
C.3 ANALYSIS OF THE SWITCH TIME T
Switch time T has limited impact on static Gather tasks and is important but not critical for Maze 0. Maze performance varies substantially with the pretrained SNN because backward skills can be essential.
- Gather: Switch time T does not critically affect static Gather performance, although lower T permits faster skill changes for more precise navigation.The analysis reports Gather results for environment sizes 10 and 15.
- Maze 0: For Maze 0, differences between switch times are important but not critical, because long skills can leave the robot bumping against a wall.Large performance variance is attributed to different pretrained SNNs.
- Pretrained SNN variation: Maze performance depends strongly on the pretrained SNN, unlike Gather where learning curves have low variance.A weak backward span from random seed 50 is identified as critical for solving Maze 0 and associated with a lack of learning.
D ANT
On the unstable Ant robot, SNN pretraining learns differentiated skills and broad directional coverage, but switching between skills remains difficult because instability can cause the robot to fall and terminate rollouts.
- Ant has a 27-dimensional state space and an 8-dimensional action space, and it may fall over without recovering.
- Skill switching is difficult because a new skill may encounter an unfamiliar state-space region, increasing the robot’s instability.
- SNN pretraining yields a large span of skills covering most directions when the mutual-information bonus is well tuned.
- Five depicted Ant rollouts terminate because the robot falls over after switching skills.