Source-linked AI summary
Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics
Krishan Rana, Ming Xu, Brendan Tidd, Michael Milford, Niko Sünderhauf
TL;DR
Skill-based RL can be difficult to explore and is limited when downstream tasks differ from those represented in the skill space. ReSkill uses state-conditioned skill sampling and a low-level residual policy; across four manipulation tasks, it learns faster and achieves higher final performance than baseline methods.
Problem
Skill-based RL explores expansive skill spaces containing irrelevant skills and has limited generality when downstream tasks differ from those represented in the skill space.
Source: §3 · p. 3, §4.2.2 · p. 5Method
ReSkill uses a state-conditioned generative skill prior to guide high-level exploration and a low-level residual policy to adapt decoded skills to task variations.
Source: §4.2.2 · p. 5Results
Takeaways & Limitations
A handful of scripted controllers can be repurposed to solve a wide range of unseen tasks.
Source: §7 · p. 8ReSkill still requires thousands of environment samples before reaching a high success rate, limiting its demonstrated use on a physical robot.
Source: §6.1 · p. 8
Abstract
from arXiv · showhide
Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. However, this skill space is expansive, and not all skills are relevant for a given robot state, making exploration difficult. Furthermore, the downstream RL agent is limited to learning structurally similar tasks to those used to construct the skill space. We firstly propose accelerating exploration in the skill space using state-conditioned generative models to directly bias the high-level agent towards only sampling skills relevant to a given state based on prior experience. Next, we propose a low-level residual policy for fine-grained skill adaptation enabling downstream RL agents to adapt to unseen task variations. Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while significantly accelerating exploration, outperforming prior works. Code and videos are available on our project website: https://krishanrana.github.io/reskill.
1 Introduction
ReSkill pairs state-conditioned skill sampling with a low-level residual policy to address inefficient exploration and limited adaptability in skill-based RL. Across four manipulation tasks, the authors report faster learning and higher final performance than baseline methods.
1 Introduction
- Skills are short sequences of single-step actions extracted from expert demonstrations and embedded as actions for a high-level RL policy [1] [2] [3].Examples include move-left, grasp-object, and lift-object.Source: §1 · p. 1
- A state-conditioned skill prior guides exploration toward skills relevant to the current robot state, including when relevant skills are multi-modal in the skill space.It is proposed as a remedy for inefficient random sampling across expansive skill spaces.Source: §1 · p. 1
- A low-level residual policy adapts available skills to task variations that cannot be solved by simply composing those skills.This relaxes the need for exhaustive expert demonstrations and allows skills extracted from classical controllers to be reused.Source: §1 · p. 2
- Across four manipulation tasks, ReSkill accelerates skill-based learning and achieves higher final performance in novel environments than baseline methods.The authors attribute exploration guidance to the state-conditioned skill prior and adaptation to the residual policy.Source: §1 · p. 2
2 Related Work
Related work frames ReSkill as extending skill-based RL with state-conditioned skill sampling, repurposing classical controllers, and combining pretrained skills with online residual adaptation.
Skill-Based RL
Combining RL with Classical Control
- Unlike residual methods tied to one handcrafted task controller [7] [8] [9] [10] [12], ReSkill decomposes controllers into task-agnostic skills for reuse across tasks.Classical control systems are described as cheap and readily available compared with expert datasets or human demonstrators.Source: §2 · p. 3
Hierarchical RL (HRL)
- ReSkill pretrains a low-level skill module from demonstrations, then learns a high-level policy and an online low-level residual policy for skill adaptation.This combines demonstration-based option learning with online low-level adaptation.Source: §2 · p. 3
3 Problem Formulation
The formulation treats temporally extended skills as high-level actions, with a state-conditioned policy selecting latent skills to maximize discounted H-step rewards.
3 Problem Formulation
- A skill is a fixed-horizon sequence of atomic actions used as an action for a high-level policy.Skills are embedded in a continuous latent space from which the policy can sample during exploration [1] [2] [3].Source: §3 · p. 3
- Skill-based RL faces a trade-off between sample efficiency and generality, alongside costly demonstration needs and difficult exploration in expansive latent spaces.The formulation notes that datasets may need to capture many task variations, while random exploration can be difficult in arbitrarily structured spaces.Source: §3 · p. 3
- The high-level policy selects latent skills by state to maximize the discounted sum of H-step rewards in the downstream MDP.The MDP is specified by states, actions, transitions, rewards, and a discount factor; each H-step reward sums single-step rewards.Source: §3 · p. 3
4 Approach
ReSkill repurposes controller demonstrations as continuous skills, then combines state-conditioned skill sampling with low-level residual actions so RL can adapt skills to task variations beyond the demonstrations.
4.1 Data Collection
- Controller trajectories for pushing and grasping are sliced into state-action skill segments, and the demonstrations need not be optimal for downstream adaptation.States include robot and object information; actions are continuous end-effector and gripper velocities.Source: §4.1 · p. 4
4.2.1 Embedding the skills
- A VAE embeds full state-action sequences and decodes atomic actions conditioned on both the current state and skill embedding.The closed-loop decoder was found to improve downstream learning, particularly in dynamic environments.Source: §4.2.1 · pp. 4–5
4.2.2 Learning the state-conditioned skill prior
- Sampling from the standard latent prior can waste exploration on irrelevant regions, while relevant skills for a state may be multimodal.The learned state-conditioned prior uses conditional real NVP and normalising flows to represent this multimodality.Source: §4.2.2 · p. 5, Figure 3 · p. 5
4.3 Reinforcement Learning in a State-Conditioned Skill Space
- The high-level policy samples a vector transformed by the skill prior into a state-relevant skill, while the decoder reconstructs actions over the skill horizon.The bijective transformation retains non-zero sampling probability for all skills in the latent space.Source: §4.3 · pp. 5–6
- A low-level residual policy adds corrective actions to decoded skills, giving the hierarchical agent executable-action control to adapt skills to task variations.The residual is conditioned on state, selected skill, and decoder-proposed action; the hierarchical policies are trained with an on-policy algorithm.Source: §4 · p. 3, §4.3 · p. 6
5 Experiments
Experiments evaluate ReSkill on four manipulation tasks with physical or dynamical variations absent from the skill-extraction environment. Across the tasks, ReSkill outperforms the baselines in sample efficiency and final-policy performance.
5 Experiments
- The simulated 7-DoF Fetch tasks introduce physical or dynamical variations absent from skill extraction and cannot be solved solely by the data-collection controllers.The tasks are adapted from the residual reinforcement learning environments proposed by Silver et al. [31].Source: §5 · p. 6, Figure 4 · p. 7
- Comparisons include scripted controllers, behavioural cloning with fine-tuning, SAC, PPO, and hierarchical methods HAC, PaRRot [4], and SPiRL [3].The compared methods span single-step action-space and skills-based RL approaches.Source: §5 · p. 6
- Across four tasks, ReSkill outperforms all baselines in sample efficiency and convergence to the highest-performing final policy.Training performance is averaged across five random seeds.Source: Figure 5 · p. 7
- Ablations remove either the low-level residual policy or the high-level skill-prior guidance.Without the skill prior, the high-level policy samples skills directly using πHL(z|s).Source: §5 · pp. 6–7
6 Results
ReSkill outperforms the baselines across four manipulation tasks, with state-conditioned skill sampling supporting efficient learning and residuals supporting adaptation. Ablations distinguish the contributions of the skill prior and residual; the method still requires thousands of samples to reach high success.
6 Results
- ReSkill outperforms all baselines across four tasks, converging with superior sample efficiency and highest performance while adapting to task variations.The results attribute sample efficiency to state-conditioned skill-prior guidance and performance to adapting skills to task variations.Source: §6 · p. 7
- Removing the skill prior impairs learning, including learning altogether in complex stacking, while removing the residual reduces final performance by losing fine-grained single-step control.The ablation figure compares the prior and residual across average training performance on four tasks.Source: §6 · p. 7, Figure 6 · p. 8
- Compared with SPiRL, ReSkill without a residual converges faster because it can sample state-relevant skills directly, whereas SPiRL requires a burn-in phase for KL regularization.SPiRL uses an indirect bias toward a unimodal skill prior.Source: §6 · p. 8
6.1 Limitations
- ReSkill still requires thousands of environment samples to reach high success, limiting the authors’ ability to demonstrate it on a physical robot.The authors identify improving sample efficiency as a future direction, while noting that off-policy learning would require care with residual-induced non-stationarity.Source: §6.1 · p. 8
- Useful skills absent from the training dataset are unlikely to be selected, especially under substantial dataset–task mismatch; thorough evaluation of residual adaptation to this issue remains future work.The authors also leave combining the VAE embedding and flow-based prior modules in one conditional VAE to future work.Source: §6.1 · p. 8
7 Conclusion
The conclusion presents ReSkill as a framework for reusing skills while adapting them to task variations. Across manipulation tasks, it reports faster learning and higher final returns than baseline methods.
7 Conclusion
- ReSkill combines a high-level agent in a state-conditioned skill space with a low-level residual agent for fine-grained adaptation to task and training-environment variations.The framework is described as supporting adaptable and efficient skill reuse.Source: §7 · p. 8
- A handful of scripted controllers can be repurposed to solve a wide range of unseen tasks.Source: §7 · p. 8
- Across the evaluated manipulation tasks and task variations, ReSkill learns faster and achieves substantially higher final returns than baseline methods.Source: §7 · p. 8
A Implementation Details
The implementation learns skill embeddings and a state-conditioned skill prior, then trains high- and low-level policies together. The appendix specifies the data, network designs, reward updates, and residual-introduction schedule.
A.0.1 Skill Embedding and Skill Prior Learning
- The skill dataset comprises state-action sequences; the state has 19 dimensions for push, cleanup, and stacking, and 43 for hook, while actions are 4-dimensional and skill horizon H is 10.Source: §A.0.1 · p. 12
- The skill embedding uses an LSTM encoder and MLP, with a 4-dimensional latent space; its decoder takes the latent skill and current state to produce bounded action sequences.The encoder yields variational-posterior parameters, and the reconstruction loss uses mean-squared error.Source: §A.0.1 · p. 12
- The state-conditioned skill prior is parameterized by a conditional Real NVP with four affine coupling layers and a standard Gaussian base distribution.Source: §A.0.1 · p. 12
- The embedding and prior are jointly trained using Lskills = Lembed + Lprior, with gradients from the prior loss to the latent skill blocked.The authors report that joint training yields more expressive priors than training the embedding and prior sequentially.Source: §A.0.1 · p. 12
A.0.2 Reinforcement Learning Setup
- The high-level policy uses reward after complete skill execution, while the residual policy uses reward after each environment step; both update from online experience.This hierarchical joint training is described as making more effective use of experience for downstream sample efficiency.Source: §A.0.2 · pp. 12–13
- Both policies use PPO, and the residual is gradually gated in over 20k steps after 20k initial high-level training steps without it.A smooth logistic schedule was used to stabilize hierarchical-agent training.Source: §A.0.2 · p. 13
B Tasks
The four downstream manipulation tasks introduce physical or dynamical variations from the skill-data collection environment. Each task uses sparse success rewards and a fixed episode horizon.
B Tasks
- Slippery Push lowers table friction, making block control harder; success requires pushing the block to its goal within 100 timesteps.The reward is 1 at the goal and 0 otherwise.Source: B Tasks · p. 13
- Table Cleanup adds a rigid tray absent from data collection, requiring the agent to place the block inside it within 50 timesteps.The tray edges act as obstacles, and reward is 1 only when the block is placed in the tray.Source: B Tasks · p. 13
- Pyramid Stack requires precise placement of a small red block on a larger blue block, above the prior controllers’ gripper height, within 50 timesteps.Reward is 1 only when the red block is successfully balanced on the blue block.Source: B Tasks · p. 13
- Complex Hook requires using a hook to move an unreachable object to its target amid unseen objects and random rigid table bumps, within 100 timesteps.Reward is 1 only when the object reaches its goal.Source: B Tasks · p. 13
C Skill Prior Ablation
The ablation measures object manipulation during early exploration and finds the NVP-based skill prior most often samples relevant behavior while retaining exploration of surrounding skills.
C Skill Prior Ablation
- >45% of the first 20k exploratory steps were meaningful with the NVP-based skill prior, versus 9.39% with Skill Space sampling.Meaningful steps involved manipulating relevant objects in the environment.Source: C Skill Prior Ablation · pp. 13–14, Table 1 · p. 14
- The skill prior directs exploratory trajectories toward the object while preserving exploration of diverse surrounding skills.Source: Figure 7 · p. 14
- Gaussian exploration produced almost no block interaction, while skill-space sampling uses the high-level policy output πHL(z|s).The authors connect Gaussian exploration’s lack of block interaction to SAC, PPO, and HAC failing to learn in sparse-reward settings.Source: C Skill Prior Ablation · pp. 14–15
D Evaluation on Long Horizon Manipulation Tasks
In Franka Kitchen, ReSkill is evaluated on unseen long-horizon sequences of subtasks and substantially outperforms SPiRL in sample efficiency and final policy performance.
D Evaluation on Long Horizon Manipulation Tasks
- The Franka Kitchen evaluation uses an unseen sequence of subtasks with sparse, binary rewards for each successfully completed subtask.The skills are trained from 400 teleoperated demonstration sequences.Source: D Evaluation on Long Horizon… · p. 15
- ReSkill substantially outperforms SPiRL in sample efficiency and convergence to higher final performance on higher-dimensional, long-horizon tasks.Source: D Evaluation on Long Horizon… · p. 15
- Introducing the residual causes high variance, attributed to the task’s long horizon and multiple strategies for adapting to it.The agent gradually converges to a stable, higher-reward solution after this transition phase.Source: D Evaluation on Long Horizon… · p. 15
- Without the residual, ReSkill reaches the same final performance as SPiRL but converges faster under the skill prior’s exploratory guidance.Source: D Evaluation on Long Horizon… · p. 15
E Data Collection
The authors extract reusable skills from noisy trajectories produced by simple handcrafted controllers, then use controllers with known weaknesses as demonstrations for downstream adaptation.
E Data Collection
- For Fetch tasks, 40k trajectories from pushing and grasping controllers train one skill module; Perlin noise diversifies rollouts, and skills use horizon H = 10.Trajectories are state-action sequences, filtered to exceed the skill horizon before random slicing.Source: E Data Collection · p. 16, Figure 9 · p. 16
E.1 Handcrafted Controllers
- The data-collection environment is an empty table with one object placed in front of the robotic arm.Source: §E.1 · p. 17
E.1.1 Reactive Push Controller
- The reactive push controller performs poorly when block sliding friction is reduced, as in SlipperyPush.Source: §E.1.1 · p. 17
E.1.2 Pick and Place Controller
- The pick-and-place controller lifts blocks no higher than 3 cm, limiting it when targets are elevated or lie beyond barriers.Both failure conditions occur in Pyramid-Stack and Table-Cleanup.Source: §E.1.2 · p. 17
E.1.3 Reactive Hook Controller
- The reactive hook controller is suboptimal amid varied objects and uneven table bumps, which can make the hook stick.It works well on an empty table with a simple block.Source: §E.1.3 · p. 18