Source-linked AI summary
Learning by Playing - Solving Sparse Reward Tasks from Scratch
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, Jost Tobias Springenberg
TL;DR
Sparse-reward reinforcement learning makes complex behaviors difficult to learn from scratch because exploration must discover long successful action sequences. SAC-X addresses this by learning auxiliary-task policies and actively scheduling them, and the paper reports success across challenging robotic manipulation tasks in simulation and on a real robot.
Problem
Sparse-reward tasks require discovering long sequences of correct actions, making success through random exploration highly unlikely.
Method
SAC-X simultaneously learns intention policies for auxiliary rewards and actively schedules their execution to explore for externally defined tasks using off-policy learning.
Results
SAC-X solves challenging sparse-reward manipulation tasks in simulation and learns from scratch on a real robot using common simple auxiliary rewards.
Takeaways & Limitations
SAC-X specifies only sparse end goals while learned auxiliary intentions provide exploration toward complicated behaviors.
Takeaways & Limitations
The construction assumes full control over auxiliary rewards and the ability to compute and evaluate them at any state-action pair.
Abstract
from arXiv · showhide
We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempts to learn simultaneously via off-policy RL. The key idea behind our method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL. Our experiments in several challenging robotic manipulation settings demonstrate the power of our approach.
1. Introduction
SAC-X addresses sparse-reward exploration by learning and scheduling auxiliary tasks without specifying the solution path. The approach is demonstrated on simulated and real-robot manipulation tasks using simple, sparse rewards.
- Motivation: Sparse rewards make long-horizon manipulation difficult because random exploration is highly unlikely to discover the rewarding configuration.The motivating example requires opening a box and placing a block inside before receiving reward.
- Motivation: Existing exploration methods rely on task-specific prior knowledge and can bias the control policy toward potentially suboptimal solutions.The paper contrasts these methods with sparse formulations that leave the solution path unspecified.
- Method: SAC-X assigns multiple reward signals to each state-action pair, trains an intention policy for each reward, and uses a scheduler to select intentions for external-task performance.Learning is off-policy and asynchronous from execution, with experience shared between intentions.
- Method: Auxiliary rewards target controllable sensory events such as touches, object motion, or image-plane proximity, enabling exploration before external rewards are observed.These rewards are designed for straightforward implementation on real robots.
- Experiments: SAC-X is evaluated on sparse-reward stacking and table-cleaning tasks in simulation and learns from scratch on a real robot.The same set of auxiliary reward functions is used across the simulated tasks.
2. Related Work
The paper builds on auxiliary-task, multitask, options, and curriculum-learning research, but distinguishes SAC-X by actively scheduling learned auxiliary policies to explore for a shared external goal.
- Auxiliary-task learning: Earlier auxiliary-task methods learn predictions or value functions for pseudo-rewards but generally do not use learned sub-policies to drive exploration toward a common external goal.They also typically assume independence between policies and value functions.
- Closest approaches: UNREAL, navigation actor-critic agents, and the Intentional Unintentional Agent provide additional learning signals or random sensory-goal exploration, whereas SAC-X switches among auxiliary tasks within episodes.This active switching is used to explore for the main task.
- Multitask and options: Unlike multitask reinforcement learning and options approaches, SAC-X learns skills grounded in auxiliary rewards rather than automatically decomposing one task into subtasks.The distinction concerns how the skills are specified and grounded.
- Curriculum perspective: SAC-X can be viewed as learning an online teaching policy over a fixed set of increasingly difficult problems, connecting its scheduler to curriculum-learning ideas.The paper relates this framing to PowerPlay and broader teaching research.
3. Preliminaries
The preliminaries formulate reinforcement learning in an MDP with states, actions, transitions, policies, and rewards. The objective is to maximize expected discounted return over trajectories.
- MDP formulation: An MDP is described using state or observation s, action vector a, transition density p(s_t+1|s_t,a_t), and policy distribution π_θ(a|s).State and observation are used interchangeably for notation.
- Rewards: After executing an action, the agent receives a scalar reward r_M(s_t,a_t).
- Objective: The reinforcement-learning objective is to maximize expected discounted cumulative reward over trajectories generated by the policy and environment dynamics.The return uses discount factor γ and an initial-state distribution.
4. Scheduled Auxiliary Control
SAC-X augments sparse-reward reinforcement learning with auxiliary tasks whose intentions are learned off-policy and actively scheduled to improve exploration. The method jointly trains task-specific intentions and a scheduler while preserving intention policies, using auxiliary rewards that can be computed from sensory information or simple object relations.
- Problem formulation: SAC-X defines auxiliary MDPs that share the main task’s state, observation, action, and transition spaces but use separate reward functions.The construction assumes auxiliary rewards can be computed and evaluated for any state-action pair; experiments also allow multiple external tasks.
- Learning the intentions: The joint intention objective trains all auxiliary policies and the main-task policy through off-policy data collected from the policies’ shared replay experience.The off-policy treatment allows each intention to learn from trajectories generated while other policies pursued different tasks.
- Learning the intentions: Each intention policy is trained to optimize its corresponding task, while learning uses states generated by other task policies so intentions remain compatible across starting states.This differs from ordinary multi-task reinforcement learning because each intention trains on state-visitation distributions induced by every possible preceding task.
- Learning the scheduler: A scheduler sequences intentions by selecting a sub-task every ξ steps and executing its policy, with the schedule optimized for the main-task return.The scheduler operates over H possible switches per episode and can use a Boltzmann policy whose temperature η controls greediness.
- Learning the scheduler: The scheduler is optimized while intentions remain fixed, because changing both would not guarantee preservation of the exploration policies.A random scheduler, SAC-U, is also evaluated; multiple intentions can provide curriculum data for one another within an episode.
5. Experiments
Experiments evaluate SAC-X with uniform and learned scheduling across simulated and real-robot manipulation tasks. SAC-Q reliably learns increasingly complex sparse-reward behaviors, including stacking, cleanup, and real-world lifting, while often outperforming baselines.
- Experimental setup: Experiments use a Kinova Jaco arm in simulation and on hardware, with general auxiliary tasks designed across available sensor modalities.Simulation uses raw joint velocities and observations from proprioception, force-torque, touch, and camera sensors; experiments use 36 actors and 36 learners.
- Stacking: Both SAC-U and SAC-Q reliably learn two-block stacking, with SAC-U reaching good performance after around 5000 episodes per actor and SAC-Q learning faster with slightly better final performance.IUA learns slower and less reliably, while DDPG fails completely on the sparse-reward task.
- Stacking: SAC-Q learns auxiliary intentions progressively, from touching or avoiding the block to moving it and finally stacking it.Figure 2 reports this progression for a subset of the 13 auxiliary intentions.
- Banana stacking: Both SAC-U and SAC-Q solve banana stacking, while the learned scheduler shows clearer advantages because it focuses on fine-tuning the stacking policy.The paper attributes this difference to the banana task requiring careful stacking-policy adjustment.
- Multiple external tasks: SAC-X learns both stacking configurations, requiring only 2500 additional episodes per actor after the initial task’s 5000 episodes per actor.The learned agent also develops a single-policy strategy for rearranging an already stacked configuration before completing the alternate stack.
- Cleanup: In cleanup, SAC-Q reliably learns all four extrinsic and 15 auxiliary tasks, whereas all baselines fail in the most complex sparse-reward setting.The four extrinsic intentions include opening the box, placing each object inside, and placing all objects inside; SAC-Q also learns to interpolate between intention policies.
- Real-robot learning: On a real robot, SAC-Q learns lifting after about 1200 episodes and achieves 100% success across about 50 trials.With additional reaching tasks, it also learns robust control strategies including dragging, pushing, lifting, carrying, and correcting imprecise placements.
6. Conclusion
SAC-X learns intention policies for auxiliary tasks and actively schedules them to explore for sparse external rewards. Across challenging robotic manipulation tasks in simulation and on a real robot, it produces reactive and robust behavior.
- SAC-X simultaneously learns intention policies for auxiliary tasks and actively schedules them to explore for sparse external target rewards.
- Simple auxiliary tasks let SAC-X learn complicated target tasks when only the sparse end goal, not the solution path, is specified.
- SAC-X was demonstrated on several challenging robotics tasks in simulation and on a real robot using common simple, sparse auxiliary tasks.
- The learned intentions are highly reactive, reliable, and exhibit rich, robust behavior.
A.1. Simulation
The simulation appendix describes Jaco-arm environments, observations, control settings, and a general auxiliary-reward suite. Auxiliary rewards capture contact, movement, proximity, and spatial relations, with definitions designed to work from image planes.
- Simulation uses a Jaco arm in MuJoCo, with a 10 ms numerical step integrated over five steps into a 50 ms control interval.
- Experiments use wooden blocks and capsule-shaped banana objects on a 60 cm x 30 cm table, with randomized object and hand initialization.
- Feature-based observations combine proprioceptive and simulated object features, totaling 56 entries by default and 58 for cleanup.
- Pixel experiments replace object observations with two 48 x 48 RGB camera images while retaining proprioceptive features.
- The general auxiliary rewards include TOUCH, NOTOUCH, MOVE, proximity, and directional relations such as ABOVE, BELOW, LEFT, and RIGHT.
A.1.2. EXTERNAL TASK REWARDS
The experiments define sparse extrinsic rewards for stacking, opening the box, and placing objects inside it, alongside auxiliary rewards for object–box relations. Reward definitions also differ slightly between simulation and the real robot.
- STACK(i) provides a sparse extrinsic reward for an object being stacked, determined in simulation from collision relationships.
- Cleanup uses INBOX(i), INBOXALL, and OPENBOX rewards for individual placement, complete placement, and lifting the box lid.
- The cleanup experiment contains 15 auxiliary reward signals and 4 extrinsic reward signals.
- Real-robot rewards omit TOUCH and NOTOUCH because the robot lacks touch sensors and add distance-based reaching rewards to reduce training time.
- Real-robot positional rewards use distance thresholds, including 10 cm, 1.5 cm, 2 cm, and 6 cm-above-target conditions.
B. Additional model details
The model uses shared neural networks with intention-specific heads and asynchronous off-policy learning. Distributed actors collect trajectories, learners compute gradients, and a parameter server averages updates for the shared models.
- Policy and Q-functions use shared network architectures, with parameters θ and φ representing the two neural networks.
- The stochastic policy has two shared 200-unit ELU layers with LayerNorm before routing outputs to intention-specific blocks.
- Intentions are one-hot encoded to select active policy and value-function heads, and this gating architecture gave the best preliminary final-task performance.
- Policy and Q-function training uses ADAM with a learning rate of 2 · 10^-4.
- A central parameter server averages gradients from learners and makes updated parameters available to actors and learners.
- Actors collect trajectories and all reward streams, while learners sample replay data and compute policy and Q-function gradients.
D.0.1. A DETAILED LOOK AT THE SAC-Q LEARNING PROCESS
SAC-Q learns auxiliary and extrinsic intentions while scheduling them to improve sparse-task exploration. In stacking and cleanup, its selective scheduling accelerates learning and succeeds on the most difficult sparse cleanup task.
- Stacking experiment: 13 auxiliary intentions and the STACK(1) extrinsic task are learned in parallel during stacking.The auxiliary set includes sensory and relational goals such as TOUCH, MOVE, CLOSE, and directional relations.
- Stacking experiment: SAC-U executes intentions uniformly, whereas SAC-Q selectively schedules auxiliaries that help collect extrinsic reward.SAC-Q ignores MOVE(2), SOUTH, and SOUTH-CLOSE in the illustrated stacking experiment.
- Stacking experiment: SAC-Q learns the stacking extrinsic task faster when it ignores MOVE(2), SOUTH, and SOUTH-CLOSE.The comparison is shown in Figure 12’s bottom plot.
- Scheduler values: The highest learned scheduler Q-value corresponds to executing STACK(1) twice, while other intention sequences also help collect STACK(1) reward.Figure 13 reports Q-values for pairs of consecutively executed intentions.
- Cleanup experiments: Both SAC-Q and SAC-U learn all cleanup tasks, but only SAC-Q learns the most difficult sparse cleanup task.The figures also report greater early learning efficiency for SAC-Q on auxiliary and extrinsic tasks.