PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

Sébastien Forestier, Rémy Portelas, Yoan Mollard, Pierre-Yves Oudeyer

arXiv:1708.02190v3cs.AIcs.LG

TL;DR

Autonomous learning requires agents to generate and organize their own goals rather than follow externally supplied curricula. The paper formalizes IMGEPs and instantiates AMB with population-based policies and object-centered modularity. Across environments, these systems discover diverse skills and automatically develop curricula toward more complex tasks, while relying on handcrafted perceptual and goal-space representations.

  • Problem

    Autonomous developmental learning requires machines to discover and organize diverse skills without an externally imposed target goal, curriculum, or training database.

  • Method

    IMGEPs represent self-generated goals as parameterized fitness functions, select goals using intrinsic rewards, explore with goal-parameterized policies, exploit collected data, and reuse experiments across goals; AMB adds population-based and object-centered spatio-temporal modularity.

  • Results

    IMGEP variants without hand-designed learning and AMB efficiently discover complex affordances, with AMB autonomously developing curricula from easier to more complex tasks across multiple environments.

  • Takeaways & Limitations

    Self-generated goals, intrinsic rewards, and reusable exploration data provide a unified route to autotelic agents that discover stepping-stone skills without a specified target goal.

  • Takeaways & Limitations

    The approach assumes agents already possess perceptual systems for seeing and tracking objects and handcrafted modular representations for encoding goals.

Abstract

from arXiv · show

Intrinsically motivated spontaneous exploration is a key enabler of autonomous developmental learning in human children. It enables the discovery of skill repertoires through autotelic learning, i.e. the self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present an algorithmic approach called Intrinsically Motivated Goal Exploration Processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP architecture relies on several principles: 1) self-generation of goals, generalized as parameterized fitness functions; 2) selection of goals based on intrinsic rewards; 3) exploration with incremental goal-parameterized policy search and exploitation with a batch learning algorithm; 4) systematic reuse of information acquired when targeting a goal for improving towards other goals. We present a particularly efficient form of IMGEP, called AMB, that uses a population-based policy and an object-centered spatio-temporal modularity. We provide several implementations of this architecture and demonstrate their ability to automatically generate a learning curriculum within several experimental setups. One of these experiments includes a real humanoid robot exploring multiple spaces of goals with several hundred continuous dimensions and with distractors. While no particular target goal is provided to these autotelic agents, this curriculum allows the discovery of diverse skills that act as stepping stones for learning more complex skills, e.g. nested tool use.

1. Introduction

The paper frames autonomous developmental learning as self-directed exploration and introduces IMGEPs to formalize this process for machines. It contributes a general goal formalization, the AMB architecture, and experiments across simulated, Minecraft, and humanoid-robot environments.

  • Motivation: Children autonomously generate, select, order, and practice goals instead of receiving an externally imposed curriculum.Intrinsic motivation assigns evolving interestingness to goals, producing an organized developmental trajectory.
  • Motivation: Intrinsic rewards based on learning progress can organize exploration into curricula where increasingly complex skills are practiced and reused as stepping stones.The introduction motivates this mechanism as a way to avoid beginning with rare or deceptive target rewards.
  • Contributions: IMGEP formalizes goals as abstract parameterized fitness functions over complete behavioral trajectories, including constraints and Quality-Diversity objectives.This makes multiple goal-exploration algorithms expressible within one framework.
  • Contributions: AMB is a population-based IMGEP architecture combining spatial object-centered goal spaces with temporal modularity through stepping-stone-preserving mutations.Learning progress selects among modular goal spaces associated with environmental objects.
  • Evaluation: The study evaluates IMGEP variants in a 2D simulated environment, Minecraft, and a real humanoid robotic setup, including ablations and distractors.Evaluation emphasizes sample efficiency and diversity of discovered behavioral features, with comparisons against agents exploring one target object.

2. Intrinsically Motivated Goal Exploration Processes

IMGEP is a framework for autonomous exploration of multiple goals in which experiments provide information about both targeted and other goals. It combines self-generated fitness-function goals, intrinsic-reward goal selection, exploration, exploitation, and data reuse.

  • Generalized Goals and Goal Spaces: IMGEP represents goals as parameterized fitness functions that evaluate state-action trajectories and can encode arbitrary objectives or constraints.The framework does not require all goals to be achievable or to depend only on the final state.
  • Generalized Goals and Goal Spaces: A goal space is a parameterized set of fitness functions, supporting feature maximization, target matching, constrained optimization, and Quality-Diversity formulations.Examples include maximizing speed, reaching a target position, matching a sound spectrum, or satisfying constraints while optimizing another feature.
  • Algorithmic Architecture: Each experiment computes fitness for the selected goal, intrinsic reward for the goal choice, updates exploration and goal-selection policies, and adds the result to the learning database.The same trajectory can be evaluated for other goals when the agent can compute their fitness values.
  • Algorithmic Architecture: IMGEP runs exploration and exploitation processes in parallel: exploration gathers goal-directed data, while exploitation trains a more accurate goal-parameterized policy from that database.Exploitation may use slower batch methods such as deep neural networks, support-vector machines, or Gaussian mixture models.
  • Algorithmic Architecture: Choosing goals is nontrivial because goal spaces may be infinite, continuous, and high-dimensional, while experiments targeted at one goal can still inform other goals.The framework evaluates efficiency through behavioral diversity, discovered stepping-stones, or analysis of internal models and policies.

3. Modular Population-Based IMGEP Architecture

The modular population-based IMGEP architecture uses a population of low-level policies and hierarchical object-centered goal spaces. Its exploration policy uses learning progress and temporally structured mutations to gather reusable stepping-stone data.

  • Population-Based Meta-Policies Π and Πϵ: The architecture builds a goal-parameterized policy from low-level policies and a meta-policy that selects the best policy for a goal and context.The exploration meta-policy instead selects policy mutations that gather information for the current and similar goals.
  • Population-Based Meta-Policies Π and Πϵ: Figure 1 illustrates learning-progress exploration: preference shifts from rapidly improving goals toward other improving goals, while stagnant or unreachable goals receive little preference.Competence is plotted against training time, and the exploration preference follows the competence derivative.
  • AMB Architecture: AMB hierarchically selects an object-related goal space with Γ, samples a goal with γk, chooses policy parameters with Πϵ, executes the policy, and updates its components from the outcome.The outcome encodes movements of all objects, enabling subsequent fitness evaluation across goals.
  • Spatial Modularity: Object-centered modularity constructs separate goal spaces from perceptual features of each object, with distances to target outcomes defining goal fitness.The outcome space contains features for object movements, while the overall goal space is composed of object-related subspaces.
  • Temporal Modularity: Temporal modularity preserves useful action subsequences through stepping-stone-preserving mutations rather than mutating all time-indexed policy parameters independently.This is motivated by dependencies such as reaching or grasping a tool before performing a later action on another object.
  • AMB and RMB: AMB uses learning-progress-based goal-space sampling, whereas RMB samples goal spaces randomly and can perform well when all goal spaces are relevant and distractors are absent.Both variants use stepping-stone-preserving mutations.

4. Experiments

The experiments evaluate modular population-based IMGEP architectures in tool-use environments where exploration of one task can provide information for later tasks. They compare learning architectures while examining object-centered representations, movement-aware mutations, and intrinsic motivation.

  • Experimental design: Three tool-use environments are used to study curriculum learning through tasks that can provide information for solving later tasks.The environments include a 2D simulated setup, Minecraft, and a real robotic setup.
  • Experimental design: The 2D simulated environment contains a robotic arm that can grab sticks and move toys, with randomly moving cat and dog distractors.The gripper must close near a stick handle, and only matching reachable toys can be moved.

4.1 Tool-Use Environments

The study uses three progressively different tool-use environments: a fast 2D arm task, a Minecraft multi-goal task, and a noisy high-dimensional humanoid-robot setup. Their linked tasks and distractors allow evaluation of skill reuse and modular exploration across varied sensorimotor spaces.

  • 2D Simulated Environment: The 2D environment uses a three-joint arm and gripper to grasp magnetic or Velcro sticks and move matching toys.Static objects and randomly walking cat and dog agents serve as distractors, while only one toy of each type is reachable with its stick.
  • Minecraft Mountain Cart: Minecraft Mountain Cart is a 3D multi-goal task in which navigation, tool discovery, block breaking, and cart movement form linked challenges.The agent must avoid water holes, find tools, use the pickaxe to break diamond blocks, and approach the cart.
  • Observations and outcomes: The 2D and Minecraft outcomes encode object positions and task states, while the robotic context measures the Ergo and ball rotation around the arena center.In the robotic setup, only the Ergo and ball persist between roll-outs; other distractors are reset.
  • Robotic Tool-Use Environment: The real robotic setup places a Poppy Torso arm before joysticks that control an Ergo robot and a ball producing light and sound.The setup has high-dimensional motor and sensory spaces, physical and interaction noise, and independently moving distractor robots.

4.2 Implementation of the Modular Population-Based IMGEP Architecture

The modular population-based IMGEP implementations combine parameterized motor policies, stepping-stone-preserving mutations, and intrinsic learning-progress-based selection of goal spaces. The study also compares these mechanisms with random, single-goal, and hand-designed curriculum controls.

  • Motor policies: Radial Basis Function policies control the 2D and robotic environments using five Gaussian bases and 20 total parameters.Each joint movement is a weighted sum of the five basis functions and their parameters.
  • Motor policies: Minecraft policies are closed-loop neural networks with an 18-dimensional observation vector, one 64-unit hidden layer, and three tanh outputs.Their 1411 parameters are mutated rather than trained by backpropagation.
  • Architecture: AMB implements spatial modularity by associating each object with a goal space and temporal modularity by preserving movement structure around discovered stepping-stones.A stepping-stone-preserving mutation operator modifies policy variables after the goal object has moved.
  • Intrinsic motivation: Goal-space selection uses intrinsic learning progress, combining random sampling with preferential sampling proportional to running average intrinsic rewards.The agent samples a random goal space with probability 20% and otherwise favors spaces with higher estimated progress.
  • Control conditions: Control conditions include Random Model Babbling, Single Goal Space, and a hand-engineered Fixed Curriculum.The fixed curriculum orders objects from easiest to most complex, while the single-goal condition repeatedly targets one engineer-selected object.

4.3 Results

Across three tool-use environments, IMGEP agents explored increasingly complex objects and discovered tool-mediated skills, with exploration maps showing diverse reachable outcomes. The experiments compare modular and intrinsic-motivation components using distractors and multiple control conditions.

  • Exploration outcomes: In 100k iterations, a 2D agent transported two toys across many positions inside their reachable region.The exploration map records cumulative successful object positions for one representative trial.
  • Exploration outcomes: In Minecraft, an AMB agent navigated the arena, moved the pickaxe and shovel, broke blocks, and moved the cart after 40k iterations.The learned sequence follows the environment’s tool-use dependencies.
  • Exploration outcomes: AMB agents discovered tool-use chains in all environments: sticks moved toys, pickaxes enabled mining toward carts, and joysticks controlled toys and balls.These results show sequential discovery of tools and their downstream objects within the experimental time limits.
  • Analysis: Intrinsic-reward analyses measured dependencies by relating the currently explored object to the proportion of movements that moved an object of interest.This analysis operationalizes how discovered tools act as stepping stones for later objects.
  • Stepping-stone structure: The 2D environment exhibited a Hand → Tool → Toy stepping-stone structure, with tool exploration moving the tool in about 93% of iterations but the toy in about 0.1%.Random arm movements had almost zero chances of reaching either object, while hand exploration provided the initial route to the tool.
  • Stepping-stone structure: In Minecraft, random policies found the pickaxe with 0.04% probability, broke a block with 0.00025% probability, and never moved the cart over 800k episodes.IMGEP exploration increased successive chances to about 10% for discovering the pickaxe, 1% for breaking blocks, and 8% for discovering the cart.

4.3.3 Learned Skills

The experiments evaluate learned competence and intrinsic-reward dynamics in Minecraft and the 2D and robotic environments. Learning progress prioritizes controllable objects and produces curricula in which newly discovered objects become successive exploration targets.

  • Learned competence: An AMB agent successfully learned to move the Minecraft cart close to five queried target locations.Cart competence was evaluated using uniformly generated goals and a distance-based reach criterion.
  • Intrinsic rewards: Intrinsic rewards rise when agents reduce goal–outcome distances over time, indicating progress toward controlling the selected object.Agents use this learning-progress signal to choose objects yielding higher intrinsic rewards.
  • Intrinsic rewards: In 2D, intrinsic reward rose first for the hand, then sticks, then toys, while static objects had zero progress and independently moving distractors had very low progress.The reward sequence tracks increasing access to controllable objects.
  • Intrinsic rewards: In Minecraft, agents progressed from the agent space to shovel and pickaxe, then blocks; the cart appeared after 14k episodes for one agent and 26k for another.The three distracting flowers had interest strictly equal to zero in both runs.
  • Intrinsic rewards: In the robotic environment, learning progress moved from the hand to joysticks, then to the Ergo and ball, with some agents discovering light and sound effects.Unlike the other environments, hand intrinsic reward remained high because full-trajectory goals created large, homogeneous goal spaces.
  • Curriculum dynamics: Overall, learning-progress evaluation self-organized curricula that focused on objects yielding the most progress and discovered stepping stones successively.The resulting interests changed as agents acquired control of new objects.

4.3.4 Influence of Goal Modularity

Object-centered modularity improves exploration over a flat representation by separating sensory feedback and allowing agents to monitor and target objects independently. Across the simulated, Minecraft, and robotic comparisons, modular approaches explore more diverse object spaces.

  • Goal representation: Modular goal representations associate separate object-related spaces, whereas FRGB samples goals from one combined sensory space containing all objects.The comparison tests whether disentangled object goals improve exploration.
  • 2D simulated environment: In the 2D environment, modular AMB and RMB explored the tool and toy spaces substantially better than flat FRGB and random-action controls.The figure reports that random actions never touched a toy with the stick.
  • Minecraft Mountain Cart: At 40k Minecraft iterations, AMB and RMB reached 45% and 35% median pickaxe exploration, while FRGB remained below 15%.Modular approaches significantly outperformed FRGB across all goal spaces with Welch’s t-tests at p < 0.001.
  • Robotic environment: In the robotic environment, flat FRGB agents did not explore objects beyond the hand.The modular representation independently monitors object behavior and selects disentangled goals, improving exploration efficiency.

4.3.5 Curriculum Learning

Learning-progress sampling generally outperforms random object selection and single-goal targeting, while a carefully designed fixed curriculum can achieve similar exploration in some settings. Autonomous curriculum learning therefore depends on both modular goals and the environment’s stepping-stone structure.

  • 2D simulated environment: In 2D, AMB reached 62% tool exploration and 3.3% toy exploration, compared with 37% and 0.5% for RMB.Single-goal targeting did not manage to explore either the tool or toy, whereas a predefined sequence explored both very well.
  • Minecraft Mountain Cart: In Minecraft, AMB explored significantly more than RMB across all goal spaces at 40k iterations, while single-goal cart targeting performed poorly.A hard-coded curriculum reached higher median performance in some spaces, but its effectiveness depended on knowing the task sequence.
  • Robotic environment: In the robotic environment, AMB outperformed RMB in joystick, ball, light, and sound spaces and outperformed fixed curriculum in the right-joystick and sound spaces.Welch’s tests at 100k iterations reported p < 0.05 for the comparisons against RMB.
  • Intrinsic curriculum: Across environments, learning-progress sampling improved exploration of most objects by prioritizing learnable objects, ignoring distractors, and reducing interest in already explored objects.This differs from random selection, which does not adapt object choice to progress.
  • Mutation operators: The curriculum experiments also evaluate stepping-stone-preserving mutations that retain useful movement structure around the goal object.The SSPMutation example diverges trajectories only after grasping the stick, unlike FullMutation, which can miss it from the start.

4.3.6 Influence of the Modularity of Exploration Mutations

Stepping-stone-preserving mutations substantially improve exploration because they preserve movements that reach intermediate objects while exploring subsequent actions.

  • SSPMutation preserves the movement segment that reaches a stepping-stone and explores only after the target object begins moving.This lets mutations retain successful tool-use behavior while varying later actions.
  • 95% versus 7% of tool-targeted iterations moved the magnet tool with SSPMutation versus FullMutation.
  • 53% versus 0.9% of toy-targeted iterations moved the magnet toy with SSPMutation versus FullMutation.
  • 62% versus 13% median exploration was achieved for the magnet tool with SSPMutation versus FullMutation.For the magnet toy, the medians were 3% versus 0%, with FullMutation reaching a maximum of 0.5% versus 60%.
  • The higher probability of continuing to move the target object directly improves exploration under SSPMutation.

4.3.7 Encoding of Goals

Goal encoding trades off efficient coverage of object endpoints against richer trajectory exploration. Endpoint encoding is slightly more efficient in the simulated setting, whereas trajectory encoding better represents movements whose endpoints are uninformative.

  • Endpoint encoding slightly outperformed trajectory encoding for magnet-tool exploration and had a similar magnet-toy median with greater variance.At 100k iterations, magnet-toy standard deviation was 17% with endpoint encoding versus 8.6% with trajectory encoding.
  • Trajectory encoding represents each object's whole trajectory rather than only its final point.
  • Trajectory encoding more often discovers objects that are difficult to move initially, although exploration after discovery is slower.
  • Trajectory goal spaces take longer to cover because their higher dimensionality keeps intrinsic interest elevated for longer.The interest in exploring the hand and other spaces decreases more slowly than with endpoint encoding.
  • The robotic environment uses trajectory encoding, while Minecraft Mountain Cart uses endpoint encoding.Trajectory information is useful for robotic joysticks that return to rest, making their endpoint uninformative about controlled-object movement.

4.3.8 Static and Random Distractors

Distractors challenge random exploration because they enter sensory feedback without being controllable. AMB maintains exploration by using learning progress to prioritize objects it can learn to move.

  • AMB estimates progress separately for objects, assigning zero progress to static distractors and low progress to other uncontrollable objects.
  • The evaluation compares AMB and RMB with two random and eight static distractors against environments without distractors.Exploration is measured in magnet-tool and magnet-toy spaces across 100 seeds using medians and 25th–75th percentiles.
  • RMB explores similarly to AMB without distractors but does not cope with distractors, whereas AMB shows no significant decrease.

5. Related Work

Related work connects IMGEP to intrinsic motivation, goal-conditioned learning, curriculum learning, novelty search, and robotics. The paper positions IMGEP as autonomous exploration over self-generated, potentially modular goals rather than externally specified tasks.

  • Intrinsic motivation has supported exploration for sparse or deceptive rewards and computational models of open-ended autonomous learning.
  • Prior related methods often restrict goals to states or sensory measurements, use different intrinsic rewards, or omit robotic evaluation.
  • Curriculum-learning approaches have automated learning-progress curricula but generally do not address curricula over sets of reinforcement-learning problems as IMGEP does.
  • IMGEP differs from classical goal-conditioned reinforcement learning by treating goals as autonomous rather than externally supplied and by emphasizing diverse skill exploration.
  • Novelty Search can be framed as population-based IMGEP by defining goals from behavioral-space points and rewarding proximity to them.Its objective-free novelty search corresponds to self-generated objectives within the IMGEP framing.
  • Novelty Search motivates stepping-stone exploration because unknown intermediate solutions may be necessary before approaching a final objective.

6. Discussion

The discussion presents IMGEP as a compact, general framework for autotelic exploration, with AMB organizing modular goal search through learning progress. Across simulated, Minecraft, and robotic settings, these methods discover diverse and complex skills efficiently, while relying on handcrafted perception and facing limited generalization from frozen task experts.

  • Framework and scope: IMGEP unifies algorithms in which agents self-generate goals, use intrinsic rewards, and reuse information across related goals.Goals are represented as parameterized fitness functions, allowing arbitrary trajectory objectives and constraints.
  • Framework and scope: AMB combines population-based policies with spatial and temporal object-centered modularity to sample goal spaces using learning progress.Temporal modularity preserves useful stepping-stones from discovered object movements during exploration.
  • Benefits and applications: IMGEP is especially suited to environments where unknown stepping-stones and diverse non-optimal solutions support discovery of advanced behaviors.This is most useful when expert users cannot specify the relevant targets or stepping-stones in advance.
  • Benefits and applications: Across 2D, Minecraft, and humanoid-robot experiments, IMGEP variants discovered complex affordances and tool-use skills without hand-designed learning curricula.The robotic experiment involved exploration of a high-dimensional continuous environment from scratch.
  • Efficiency: 20k, 40k, and 100k iterations were used in the real robotic, Minecraft, and 2D setups, respectively, while comparable approaches required millions of iterations.The simplest population-based implementation also completed the 20k-iteration robotic experiment on a Raspberry Pi 3.
  • Limitations: The approach assumes object perception, tracking, and handcrafted modular goal representations, and population-based frozen task experts hinder generalization relative to a single multipurpose policy.More complex initial-state distributions with only a small subset of achievable goals could impair current-agent performance.
Loading 1708.02190v3…