Source-linked AI summary
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
Cédric Colas, Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, Pierre-Yves Oudeyer
TL;DR
Autonomous agents in open-ended environments must select and learn among diverse self-generated goals without external rewards. CURIOUS combines a modular UVFA-based single policy with hindsight learning and absolute-learning-progress mechanisms for goal selection and replay. In a modular robotic environment, it self-organizes learning phases and shows robustness to distracting goals, forgetting, and body changes.
Problem
Autonomous continual learning requires agents to discover diverse skills without external rewards and determine which self-generated goals and modules to practice.
Method
CURIOUS combines modular goal representations in a single UVFA-based policy with active goal selection and cross-module replay guided by absolute learning progress.
Results
CURIOUS self-organizes distinct learning phases, spends less time on impossible modules, focuses on achievable ones, and shows robustness to distracting goals, forgetting, and body changes.
Takeaways & Limitations
Absolute-learning-progress mechanisms support autonomous curriculum organization and refocusing on modules being forgotten during continual learning.
Takeaways & Limitations
The experiments assume that agents already have hand-defined modular representations of goals and modules, whereas general IMGEP settings require agents to construct them autonomously.
Abstract
from arXiv · showhide
In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to discover what is controllable in their environments, and what is not. Because some goals might prove easy and some impossible, agents must actively select which goal to practice at any moment, to maximize their overall mastery on the set of learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and 2) an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress. Agents focus sequentially on goals of increasing complexity, and focus back on goals that are being forgotten. Experiments conducted in a new modular-goal robotic environment show the resulting developmental self-organization of a learning curriculum, and demonstrate properties of robustness to distracting goals, forgetting and changes in body properties.
1. Introduction
CURIOUS addresses autonomous continual learning by combining modular multi-goal reinforcement learning with intrinsic goal selection and curriculum organization. The paper introduces a single-policy architecture, active replay mechanisms, and an environment for evaluating learning across diverse goals, including robustness to distraction, forgetting, and body changes.
- Autonomous continual learning requires agents to discover repertoires of skills without external rewards while exploring diverse forms of environmental control.
- CURIOUS uses modular goal representations and UVFA to learn continuous sets of diverse goals within a single policy, including Reach, Push, Pick and Place, and Stack.Each module has its own reward function and continuously parameterized goal space.
- Absolute learning progress guides both interaction goal selection and cross-module replay, focusing learning on goals that are learnable while disengaging from goals that are too hard or already solved.Off-policy learning allows experiences to improve learning about goals from any module.
- The paper proposes a monolithic modular multi-goal policy as an alternative to population-based and module-expert architectures, retaining flexibility associated with reinforcement learning methods.
- CURIOUS introduces a modular-goal robotic environment and evaluates comparisons with flat multi-goal RL using HER and a module-experts architecture.The environment contains multiple objects and distractors and supports Reach, Push, Pick and Place, and Stack goals.
- The studies examine automatic curriculum self-organization and robustness to distracting goals, forgetting, and changes in body properties.
2. CURIOUS
CURIOUS combines a modular UVFA architecture with learning-progress-driven module selection and cross-module replay. Its curriculum estimates competence and reallocates interaction and training toward modules showing the greatest absolute progress, including modules affected by forgetting.
- 2.1. A Modular Multi-Goal Architecture using Universal Approximators: M-UVFA encodes diverse modular goals in one policy by masking inactive goal inputs and adding a one-hot module descriptor.The policy and critic receive [s_t, g, md], while each module can have its own continuous goal dimensionality.
- 2.2. Module and Goal Selection, Cross-Module Learning, Cross-Goal Learning: CURIOUS estimates competence through self-evaluation rollouts and derives subjective learning progress from competence changes over time.These estimates feed proportional probability matching with an epsilon-greedy exploration component.
- 2.2. Module and Goal Selection, Cross-Module Learning, Cross-Goal Learning: The curriculum prioritizes modules with high absolute learning progress while retaining random exploration for solved, difficult, or plateaued modules.Absolute progress also prioritizes decreasing performance, allowing resources to return to modules being forgotten.
- 2.2. Module and Goal Selection, Cross-Module Learning, Cross-Goal Learning: Absolute learning progress guides both which module supplies the next goal and which substitute module receives replay training.The paper identifies cross-module goal replay as a novel use of learning progress beyond environment interaction.
- 2.2. Module and Goal Selection, Cross-Module Learning, Cross-Goal Learning: Cross-module learning substitutes the sampled transition’s module descriptor, then hindsight substitutes its goal with a later episode outcome before internal reward computation.The imagined module and goal determine whether the transition satisfies the relevant constraints.
3. A Modular Goal Environment
The Modular Goal Fetch Arm is a simulated robotic environment in which an agent controls a gripper to pursue diverse modular manipulation goals. These goals range from reaching and pushing to picking, placing, and stacking, with additional distracting or impossible goals possible.
- The environment adapts OpenAI Gym and places a robotic arm facing two randomly positioned cubes on a table.
- The agent controls the gripper’s position and opening in a 4D action space.
- Its modular goals include reaching a 3D target, pushing cube 1 onto a 2D target, picking and placing cube 1, and stacking cube 1 over cube 2.
- Additional modules can target out-of-reach or moving cubes, creating impossible or distracting goals.
4. Experiment and Results
Experiments compare modular policy architectures and intrinsically motivated module selection. CURIOUS learns achievable goals faster, reallocates attention to forgotten modules, recovers more quickly after sensor perturbations, and gains advantage as distracting modules increase.
- 4.1. Impact of Policy and Value Function Architecture: The flat HER architecture’s learning curve stays flat because its holistic goals require satisfying incompatible constraints simultaneously and therefore produce no reward.This motivates separating goals into modular representations.
- 4.1. Impact of Policy and Value Function Architecture: M-UVFA learns achievable goals much faster than MG-ME, reaching them in approximately 250 · 10^3 versus 450 · 10^3 episodes.The comparison concerns the multi-modular-goals approach with one policy against multi-goal module-experts.
- 4.3. Impact of the Intrinsic Motivation: Resilience to Forgetting and Sensor Perturbations: CURIOUS focuses on modules with increasing competence progress, while assigning low learning progress to already solved or currently unsolvable modules.In the visualized run, M1 is already solved after 104 episodes, whereas M3 and M4 remain too hard before 35 · 10^3 episodes.
- 4.3. Impact of the Intrinsic Motivation: Resilience to Forgetting and Sensor Perturbations: When a previously mastered module is forgotten, its learning progress increases and CURIOUS raises its selection probability, reallocating resources toward recovery.This mechanism lets the agent monitor module competence and react to forgetting.
- 4.3. Impact of the Intrinsic Motivation: Resilience to Forgetting and Sensor Perturbations: After sensor perturbation, CURIOUS recovers 95% of pre-perturbation performance in 43 · 10^3 episodes versus 78 · 10^3 for random selection, a 45% faster recovery.The perturbation affects one of four modules and causes an approximately one-quarter drop in average performance; the difference is significant at p < 10^-4.
- 4.4. Impact of the Intrinsic Motivation: Resilience to Distracting Modules: CURIOUS’s advantage over random module selection increases as distracting modules grow from 0 to 4 to 7 because it learns to target impossible modules less often.Random M-UVFA agents continue sampling distracting modules and waste time and resources on them, while CURIOUS samples them sporadically to update learning-progress estimates.
5. Discussion
The discussion presents M-UVFA as an extension enabling modular multi-goal reinforcement learning with one policy, while identifying forgetting and delayed learning-progress estimates as limitations.
- Leveraging Environment Modularity: M-UVFA extends UVFA to represent a richer diversity of modular and discrete goals within a single policy.It targets discrete goals by using a null module goal vector and encoding the goal in the module representation.
- Pros and Cons of Monolithic Policies: A monolithic policy can reuse representations across similar modules and goals, which may explain its advantage over module-expert policies.The discussion links this representation sharing to the performance comparison in Fig. 5.
- Pros and Cons of Monolithic Policies: Monolithic policies remain more prone to forgetting, especially as the number of potential modules increases.Absolute learning progress partially mitigates but does not eliminate this issue.
- A Monolithic IMGEP: CURIOUS places goal selection under intrinsic motivation rather than requiring agents to follow engineer-specified goal sequences.This perspective derives from the IMGEP framework for self-generated, parameterized exploration problems.
- Active Learning using Learning Progress: The current moving-average learning-progress estimator may react slowly to changing competence.This delay can sustain attention on mastered modules or slow responses to newly learnable ones.
6. Further Work
Further work proposes hierarchical self-generated goal selection, autonomous goal sampling, and learned modular goal representations to reduce hand-designed scaffolding.
- Hierarchical Extension: A hierarchical extension could replace CURIOUS’s high-level module-selection policy with another CURIOUS agent targeting self-generated higher-level goals.This would connect intrinsic module selection with hierarchical reinforcement learning.
- Learning a Goal Selection Policy: Goal sampling within modules is currently uniform over a predefined reachable goal space rather than learned autonomously.Future alternatives include SAGG-RIAC, which focuses on regions with higher learning progress, and GOAL-GAN, which generates intermediate-difficulty goals.
- Learning Representations for Modules and Goals: The experiments hand-define modules and goal spaces, assuming agents already possess modular goal representations.The authors describe this as scaffolding and propose combining CURIOUS with unsupervised learning of disentangled goal spaces.
7. Conclusion
The conclusion describes CURIOUS as combining M-UVFA with absolute-learning-progress-based attention, producing self-organized curricula that prioritize achievable goals and revisit forgotten abilities.
- 7. Conclusion: CURIOUS combines M-UVFA for modular goal reinforcement learning in one policy with active selection of modules maximizing absolute learning progress.The algorithm integrates representation and curriculum mechanisms within a single learning system.
- 7. Conclusion: The mechanism self-organizes distinct learning phases, with some shared across agents and others dependent on individual experience.This produces curricula that adapt to the agent’s developmental history.
- 7. Conclusion: Agents spend less time on impossible modules and refocus on modules being forgotten after model, environmental, or body changes.The conclusion gives sensory failures as an example of body-related change.
- 7. Conclusion: The environment, code, and video of the CURIOUS agent are made available online.The paper provides a GitHub repository for these resources.
Additional Background
The background introduces DDPG for continuous-action learning, UVFA for conditioning actor and critic networks on goals, and HER for hindsight relabeling in multi-goal settings.
- DDPG: DDPG is an off-policy, model-free reinforcement-learning algorithm for continuous action spaces that jointly learns an actor and critic.The actor maps states to actions, while the critic approximates the optimal action-value function.
- DDPG: Exploration collects transitions with a noisy behavioral policy, stores them in a finite replay buffer, and samples them uniformly for updates.This supports off-policy actor and critic training from previously collected experience.
- DDPG: The critic minimizes mean-squared Bellman error, while the actor is trained to maximize the critic’s output.The critic’s dataset consists of transitions containing state, action, reward, and next state.
- UVFA: UVFA conditions both actor and critic on a goal vector, yielding policies and value functions that target multiple goals.The reward function is also parameterized by the goal.
- HER: HER probabilistically replaces a failed attempted goal with an outcome later achieved in the trajectory.This hindsight relabeling increases the opportunity to learn from unsuccessful multi-goal attempts.
Additional Methods
CURIOUS selects modules and goals using competence evaluation and absolute learning progress, then trains through replay, substitution, hindsight, and internal rewards. Its implementation uses modular goal conditioning with off-policy RL and evaluates competence statistically across replicated trials.
- Module and goal selection: During rollout selection, CURIOUS samples goals uniformly within a chosen module, selecting modules by learning-progress probabilities except during random self-evaluation.Self-evaluation selects modules randomly; ordinary selection uses pLP, while exploration noise is omitted during evaluation.
- Learning-progress update: The agent updates module learning progress after self-evaluation and mixes random module exploration with exploitation proportional to absolute learning progress.The random component revisits solved, overly difficult, or plateaued modules.
- Replay and substitution: Replay samples module-relevant transitions using pLP, substitutes the stored module descriptor, and often replaces goals with later episode outcomes for cross-module and hindsight learning.Hindsight substitution occurs with probability 0.8, enabling cross-module and cross-goal updates.
- Replay and substitution: Prioritized Interest Replay selects episodes whose outcome for the trained module changed, then samples uniformly within those episodes without importance-sampling correction.This episode-level heuristic differs from PER's transition-level TD-error prioritization and is applied to all algorithms.
- Internal reward: Internal rewards are parameterized by substitute module and goal, rewarding satisfaction of module constraints such as reaching within 0.05 simulation units.Push and Pick-and-Place use cube-to-target distance, while Reach uses gripper-to-goal distance.
- Experimental setup: The experiments use a 7-DoF Fetch arm with two cubes, diverse targetable goals, and optional out-of-reach distracting modules; CURIOUS is built on parallel HER-DDPG with 19 actors.Each experiment uses 10 random seeds and one-tail Mann–Whitney U-tests at α = 0.01; evaluation uses 95 rollouts per learning-curve point.
Additional Results
CURIOUS reveals both regularities and diversity in developmental learning trajectories through internal competence and learning-progress representations. Across trials, simpler modules tend to precede harder ones, while later-module ordering varies; scalability remains limited beyond approximately ten achievable modules.
- Developmental analysis: Internal competence and learning-progress representations let the study distinguish active curriculum learning from passive exposure to environmental structure.The analysis tracks agents' beliefs about competence and progress for each module rather than relying only on behavioral curves.
- Regularities and diversity: Reach is learned first and Push second in all five trials, whereas Pick and Place and Stack vary in order or can be learned simultaneously.Early successes increase learning progress and concentrate practice, so chance influences individual trajectories after the more consistent early phases.
- Scaling properties: CURIOUS scaled in quick tests to at least 10 achievable modules but is expected to fail with substantially larger module sets.The authors suggest multiple actors and critics, each handling a subset of modules, as a possible mitigation.
- Scope of goals: The paper's IMGEP notion of a goal includes broad self-defined parameterized problems, whereas its modular-goal setting organizes specific task families such as reaching, pushing, and stacking.The broader examples include object placement, finding a key, and generating trajectories with loops.
- Computational cost: The study reports approximately 5.6 CPU-years of computation across about 130 trials, with each trial taking around 20 hours on 19 CPUs.This is a reported computational cost rather than a performance result.