Source-linked AI summary
Accelerating Reinforcement Learning with Learned Skill Priors
Karl Pertsch, Youngwoon Lee, Joseph J. Lim
TL;DR
RL agents often learn new tasks from scratch, while large offline datasets contain transferable skills whose growing skill spaces are difficult to explore efficiently. SPiRL jointly learns a continuous skill embedding and state-conditioned skill prior offline, then integrates that prior into maximum-entropy RL. Across simulated navigation and robotic manipulation tasks, the approach accelerates downstream learning, with both skill embeddings and priors essential for effective transfer from rich datasets.
Problem
Large unstructured experience contains transferable skills, but growing skill libraries make downstream skill-space exploration difficult, while standard RL commonly learns new tasks from scratch.
Method
SPiRL jointly learns a continuous skill embedding and state-conditioned skill prior from offline data, then incorporates both into maximum-entropy RL.
Results
SPiRL accelerates learning across three simulated navigation and robotic manipulation tasks, and learned skill priors are essential for effective transfer from rich datasets.
Takeaways & Limitations
Skill priors guide exploration toward relevant parts of large skill spaces, enabling effective transfer from rich offline experience to unseen downstream tasks.
Takeaways & Limitations
The approach does not yet combine learned skill priors with semantic skills of flexible length, which is identified as future work.
Abstract
from arXiv · showhide
Intelligent agents rely heavily on prior experience when learning a new task, yet most modern reinforcement learning (RL) approaches learn every task from scratch. One approach for leveraging prior knowledge is to transfer skills learned on prior tasks to the new task. However, as the amount of prior experience increases, the number of transferable skills grows too, making it challenging to explore the full set of available skills during downstream learning. Yet, intuitively, not all skills should be explored with equal probability; for example information about the current state can hint which skills are promising to explore. In this work, we propose to implement this intuition by learning a prior over skills. We propose a deep latent variable model that jointly learns an embedding space of skills and the skill prior from offline agent experience. We then extend common maximum-entropy RL approaches to use skill priors to guide downstream learning. We validate our approach, SPiRL (Skill-Prior RL), on complex navigation and robotic manipulation tasks and show that learned skill priors are essential for effective skill transfer from rich datasets. Videos and code are available at https://clvrai.com/spirl.
1 Introduction
SPiRL addresses the difficulty of using large, unstructured experience and expanding skill libraries for downstream RL. It jointly learns skill representations and priors, then incorporates them into maximum-entropy RL to guide exploration and accelerate new-task learning.
- Motivation: Unstructured agent experience is abundant but difficult to exploit because it lacks clear task or reward definitions, while collecting new-task experience is expensive.The challenge is especially important in real-world domains such as autonomous driving, indoor navigation, and robotic manipulation.
- Motivation: Large skill libraries can reduce downstream learning efficiency because high-level policies must explore an action space whose effective size grows with the number of extracted skills.Rich datasets may therefore create a harder skill-space exploration problem despite providing more prior experience.
- Approach: SPiRL jointly learns a continuous skill-embedding space and a prior over skills from unstructured agent experience.The prior is intended to distinguish promising skills from less useful ones given the current situation.
- Approach: The learned skill prior is incorporated into maximum-entropy RL to guide exploration during downstream task learning.This extends maximum-entropy RL with both skill embeddings and skill-prior guidance.
- Results: SPiRL accelerates learning across three simulated navigation and robotic manipulation tasks, with learned skill priors essential for effectively using rich datasets.The reported validation covers complex, long-horizon navigation and robot manipulation tasks.
2 Related Work
Prior work extracts skills or behavioral priors from experience, but rich continuous skill spaces remain difficult to explore efficiently. SPiRL adds a learned prior over temporally extended skills to guide exploration, while learning skills fully offline from unstructured data.
- Prior experience: Meta-learning extracts priors from previous experience but requires defined training tasks and online data collection during pre-training.SPiRL instead learns skills fully offline from unstructured data.
- Offline learning: Offline RL can use large existing datasets in costly or safety-critical domains, but typically requires experience annotated with rewards for the downstream task.This limits direct use of unstructured experience without task-specific reward labels.
- Skill representations: Skill-transfer methods represent behaviors as options, subgoal functions, primitive libraries, or continuous latent skill spaces.Stochastic latent-variable models compactly embed many skills, but their rich spaces can make downstream exploration inefficient.
- SPiRL: SPiRL introduces a learned skill prior to guide exploration in rich skill-embedding spaces and enable efficient learning on large datasets.The approach learns skills from unstructured data and uses temporally extended actions rather than only primitive actions.
- SPiRL: Unlike action-prior methods, SPiRL learns priors over temporally extended skills, supporting complex, long-horizon downstream tasks.This distinguishes the method’s prior from approaches that regularize individual actions.
3 Approach
SPiRL learns a continuous skill space and a state-conditioned prior from unstructured offline trajectories, then uses that prior to guide hierarchical downstream RL. The approach addresses inefficient exploration in rich skill spaces by combining skill decoding with prior-regularized maximum-entropy learning.
- Skill Representation and Prior: SPiRL jointly learns a low-dimensional skill embedding space and a state-conditioned prior over skills from unstructured agent experience.The model uses an offline dataset of state-action trajectories without task or sub-skill annotations, and trains a latent skill model with an encoder, decoder, and prior.
- Skill Representation and Prior: The skill encoder maps an H-step action sequence to a posterior over latent skills, while the decoder reconstructs the action sequence from a sampled embedding.The skill model is trained with an ELBO combining reconstruction and regularization, using amortized variational inference.
- Skill Representation and Prior: The learned prior is trained to match inferred skill posteriors and is mode-covering so it represents observed skills in the current situation.The prior minimizes reverse KL divergence and can be jointly optimized with the embedding model while stopping prior-objective gradients into the encoder.
- Hierarchical Skill-Prior RL: For downstream learning, a high-level policy selects latent skills, which the decoder turns into H-step action sequences executed before the next skill is sampled.This hierarchical formulation replaces primitive actions with skills and single-step transitions and rewards with H-step transitions and rewards.
- Motivation: Large offline datasets can encode many behaviors, making direct exploration of the resulting continuous skill space inefficient for downstream learning.The effective high-level action space grows with the number of embedded skills, even when the nominal embedding dimension is small.
- Hierarchical Skill-Prior RL: SPiRL replaces the uniform-prior entropy term in maximum-entropy RL with a KL regularizer toward the learned skill prior and adapts SAC to optimize it.The approach includes automatic tuning of the regularization weight through a target divergence parameter.
4 Experiments
SPiRL is evaluated on sparse-reward, long-horizon navigation and manipulation tasks using diverse offline datasets and comparisons that isolate learned skills, priors, and temporal abstraction. Across these experiments, learned skill embeddings and skill priors support exploration and downstream transfer, while their benefits depend on suitable skill horizons and sufficient skill-space dimensionality.
- Environments & Comparisons: The experiments test transfer from diverse offline datasets to unseen, more complex tasks in maze navigation, block stacking, and kitchen manipulation.Each environment varies the task or setup between data collection and downstream learning.
- Maze Navigation: Only SPiRL learns a successful goal-reaching policy in the sparse-reward maze, while none of the baselines reaches the goal during training.The maze provides binary reward only upon reaching the goal, making exploration particularly difficult.
- Maze Navigation: SPiRL explores larger parts of the maze because its prior targets skills that navigate through doorways and hallways, whereas uniform skill sampling often causes collisions.Compared with single-step action priors, temporal abstraction produces more coherent exploration.
- Maze Navigation: A single skill prior can guide agents toward diverse maze goals, with all tested agents reaching approximately 100% success after convergence.The prior is reused across multiple downstream task agents with different goals.
- Robotic Manipulation: In block stacking and kitchen manipulation, learned skill embeddings combined with the skill prior are essential, while Flat Prior and SSP w/o Prior fail on longer-horizon tasks.Random exploration in the large extracted skill space is comparable to or worse than learning from scratch without skill transfer.
- Ablation Studies: Skill horizons that are too short provide insufficient temporal abstraction, whereas overly long horizons enlarge the skill-exploration problem and slow convergence.The study also finds that the skill space must be large enough to represent diverse skills; beyond that, dimensionality has little effect because the prior focuses exploration on relevant regions.
5 Conclusion
SPiRL combines skill embeddings with learned skill priors and incorporates those priors into maximum-entropy RL for downstream learning. The method is presented as enabling skill transfer from rich offline datasets, while the derivation replaces uniform-action entropy with divergence from a potentially state- or history-conditioned prior.
- SPiRL jointly learns a skill embedding space and skill prior from offline data, then incorporates both into maximum-entropy RL for downstream tasks.
- The extended formulation supports non-uniform action priors that may be unconditional or conditioned on the current state or previous action.
- The resulting objective replaces entropy maximization with a KL-divergence penalty measuring deviation from the action prior.
- The policy is represented as a variational distribution that induces a trajectory distribution through the environment dynamics.
- The modified Q, policy, and temperature objectives are combined into a skill-prior-regularized SAC algorithm with retained convergence guarantees from the Bellman update.
B.1 Model Architecture and Training Objective
The implementation uses recurrent skill encoding and decoding, a deep Gaussian skill-prior network, and SAC-style downstream optimization. Policy initialization from the pretrained prior is used alongside prior regularization.
- The skill encoder is a one-layer LSTM that maps action sequences into a 10-dimensional Gaussian skill embedding, while a mirrored decoder reconstructs H actions.
- The skill prior is a six-layer fully connected network that predicts a Gaussian distribution, using convolutional state features for image-based environments.
- The model uses leaky-ReLU activations, batch normalization, RAdam, batch size 16, and learning rate 1e−3; single-GPU training takes approximately 8 hours.
- Gaussian-prior KL divergence is computed analytically, while sampling-based estimates support non-Gaussian priors with slightly slower convergence.
- The downstream policy and critic mirror the prior network, and policy weights are initialized from the pretrained skill prior in addition to prior regularization.
C Environments and Data Collection
The study evaluates transfer across maze navigation, block stacking, and kitchen manipulation using image-based or vector states and increasingly difficult downstream tasks. Kitchen traces illustrate that skill-prior guidance completes a four-subtask sequence where comparison methods do not.
- Maze Navigation: Maze training uses randomly generated small layouts, while downstream learning tests a fixed start-goal problem in a maze four times larger than the training layouts.
- Maze Navigation: Maze policies receive stacked 32 × 32px local top-down observations and sparse binary rewards only near the goal.
- Block Stacking: Block-stacking data contain trajectories with up to three consecutive manipulations, whereas downstream learning uses an 11-block environment and rewards lifting and stacking.
- Kitchen environment: The skill-prior-guided kitchen agent completes all four subtasks, while a flat single-action prior completes two and unguided skill-space exploration completes one.
- Kitchen environment: The kitchen task recombines individually observed manipulations into an unseen sequence of four object manipulations.
D State-Conditioned Skill Decoder
Conditioning the skill decoder on the current state does not improve exploration without a learned prior and can reduce downstream performance. The authors attribute this to more complex high-level action dynamics and less stable learning.
- Initializing the downstream policy with prior-network weights improves training stability and convergence speed, although prior regularization alone can still learn the tasks.
- Prior initialization without continued prior regularization is insufficient for strong task learning under conventional SAC.
- State-conditioning does not improve exploration by the skill-space policy without a learned prior; efficient exploration still requires the learned skill prior.
- State-conditioning can reduce downstream learning performance by making the same high-level skill produce different action sequences across states.
E Prior Regularization Ablation
Prior regularization during downstream learning is essential: initializing with a learned skill prior alone is insufficient when conventional SAC subsequently encourages uniform skill entropy.
- Conventional SAC quickly drives the prior-initialized policy away from the learned skill prior, substantially slowing downstream learning.The ablation compares prior initialization followed by conventional SAC with prior-regularized downstream learning.
F Prior Initialization Ablation
Initializing the high-level policy with the learned skill prior improves convergence speed and training stability. Without initialization, inaccurate early divergence estimates and local optima make training more difficult, requiring additional exploration measures.
- Prior initialization improves convergence speed and training stability compared with training without initialization.The comparison initializes high-level policy weights with the learned skill prior network.
- Without initialization, sampling-based divergence estimates can be inaccurate early, while optimization may settle in locally narrow regions that limit exploration.These are identified as two major challenges for training policies from scratch.
- Analytically computing the Gaussian KL divergence provides a more stable estimate for training without initialization.Both the learned prior and high-level policy use Gaussian output distributions.
- Prior-sampled rollout steps encourage exploration during training without initialization, using ω = 1.0 initially before annealing ω to 0.The prior is used for the first 500 k steps, then its rollout-sampling fraction is annealed over the next 500 k steps.
G Training with Sub-Optimal Data
SPiRL learns useful skill embeddings and priors from substantially sub-optimal data. It remains effective with mixed-quality or mediocre-only demonstrations, outperforming baselines in downstream navigation and manipulation settings.
- The experiments test whether effective skill priors can be learned from heavily sub-optimal training data.The study considers both mixed expert and random data and data containing no expert trajectories.
- Mixed-data experiments combine random-action rollouts with portions of high-quality block-stacking trajectories.The resulting dataset varies the fraction of low-quality training data.
- SPiRL performs well when half the training data is very low quality and learns meaningful stacking skills when 75 % is low quality.The best baseline stacks only 1.5 blocks on average despite using only high-quality data.
- With mediocre-only maze data, SPiRL reaches a faraway goal at 100 % success after <1M environment steps, while the baseline fails after 3M steps.The mediocre dataset comes from a limited-capacity behavior-cloning policy whose trajectories average 2.5 wall collisions.
- A maze prior learned from sub-optimal behavior reliably reaches the goal, whereas a baseline without the learned prior fails.The figure uses data generated with a behavior-cloning policy.
- Across both data-quality scenarios, the approach learns effective skill embeddings and skill priors from substantially sub-optimal data.
H Reuse of Learned Skill Priors
A single learned skill prior can be reused across multiple downstream maze tasks. It supports efficient exploration while allowing each agent to converge to a policy reaching its task-specific goal.
- The same skill prior guides downstream agents to efficiently learn three maze tasks with different goal locations.The prior is learned once and reused across the downstream tasks.
- All agents achieve approximately 100 % success upon convergence while learning task-specific policies for their respective goals.The prior captures useful exploration behavior such as crossing doorways rather than bumping into walls, independent of goal position.