PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with AMIGo: Adversarially Motivated Intrinsic Goals

Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B. Tenenbaum, Tim Rocktäschel, Edward Grefenstette

arXiv:2006.12122v2cs.LGcs.AIstat.ML

TL;DR

Sparse extrinsic rewards make exploration difficult for reinforcement-learning agents. AMIGO trains a goal-conditioned student using a teacher that proposes challenging but achievable intrinsic goals. Across procedurally generated MiniGrid tasks, it reports state-of-the-art performance on harder exploration problems and succeeds where other methods fail.

  • Problem

    Many environments provide extremely sparse or no rewards, making random exploration insufficient or sample-inefficient for training.

    Source: §1 · p. 1

  • Method

    AMIGO uses a goal-generating teacher to propose progressively harder but feasible intrinsic goals for a goal-conditioned student.

    Source: §1 · pp. 1–2

  • Results

    AMIGO establishes state-of-the-art performance on harder MiniGrid exploration tasks, often reaches extrinsic goals on the hardest tasks, and is the only method to obtain reward on some of them.

    Source: §1 · p. 2, §4.4 · p. 8

  • Takeaways & Limitations

    Across 114 experiments on six procedurally generated tasks, AMIGO-trained agents solve tasks that are too difficult for state-of-the-art methods.

    Source: §1 · p. 2

    In the experiments, absolute-coordinate goals require both teacher and student to fully observe the environment.

    Source: §5 · p. 9

Abstract

from arXiv · show

A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating -- as form of meta-learning -- a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train a goal-conditioned "student" policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective "constructively adversarial" objective, the teacher learns to propose increasingly challenging -- yet achievable -- goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail.

1 INTRODUCTION

Sparse extrinsic rewards make random exploration ineffective or sample-inefficient, while intrinsic motivation offers a way to learn without immediate reward. AMIGO trains a goal-conditioned student with a teacher that proposes challenging but achievable goals, and its MiniGrid evaluation reports strong results on difficult procedurally generated tasks.

1 INTRODUCTION

  • AMIGO is a meta-learning method with a goal-generating teacher that proposes increasingly harder intrinsic goals for a goal-conditioned student.The teacher targets goals that are challenging but not impossible, creating a curriculum that encourages learning environment dynamics.Source: §1 · pp. 1–2
  • AMIGO can augment policy-gradient agents as a bolt-on, provided their input processing can accept an additional goal specification.The paper describes the approach as architecture-agnostic under that condition.Source: §1 · p. 2
  • AMIGO sets a new state of the art on some of the hardest MiniGrid environments and is the only method to obtain extrinsic reward on some of them.Across 114 experiments on six challenging exploration tasks, agents trained with AMIGO solve tasks that are too difficult for state-of-the-art methods.Source: §1 · p. 2
  • The evaluation uses six procedurally generated MiniGrid tasks, requiring policies to generalize across layouts and transition dynamics rather than memorize trajectories.MiniGrid uses symbolic, discrete observations that isolate exploration from visual perception.Source: §1 · p. 2

2 RELATED WORK

The related work situates AMIGO among intrinsic-motivation and curriculum-learning approaches. Its distinguishing idea is an automatically trained, constructively adversarial teacher that proposes challenging yet feasible goals.

2 RELATED WORK

  • AMIGO frames intrinsic motivation as a teacher proposing progressively harder goals, unlike novelty, curiosity, empowerment, and goal-diversity approaches.Its constructively adversarial objective encourages feasible but challenging goals without the discriminator used in GANs.Source: §2 · pp. 3–4
  • Unlike HER (Andrychowicz et al., 2017), AMIGO explicitly incentivizes exploration beyond the student's current reach; HER rewards visited states, including easy-to-reach ones.AMIGO's curriculum is generated automatically rather than handcrafted, and its teacher is trained directly to propose goals of appropriate difficulty.Source: §2 · p. 3
  • Sukhbaatar et al. (2017) use two independently acting modules and require resettable or reversible environments, whereas AMIGO uses one agent and allows a less constrained goal space.The earlier setup is limited to goals already reached by the policy.Source: §2 · pp. 3–4
  • GoalGAN (Florensa et al., 2018) targets continuous locomotion goals without conditioning on the current observation, limiting its direct fit to procedurally changing episodes.A concurrent continuous-control approach uses epistemic uncertainty from multiple value-function networks to select goal difficulty.Source: §2 · p. 4

3 ADVERSARIALLY MOTIVATED INTRINSIC GOALS

AMIGO trains a goal-conditioned student with a teacher that proposes achievable but challenging intrinsic goals. Their adversarial rewards encourage the student to reach goals quickly and the teacher to select goals requiring suitable effort.

3 ADVERSARIALLY MOTIVATED INTRINSIC GOALS

  • The teacher proposes intrinsic goals for a goal-conditioned student, rewarding goals the student can reach but not too quickly while the student is rewarded for reaching them fast.Both modules also receive environment reward when the agent solves the full task.Source: §3 · p. 4

3.1 TRAINING THE STUDENT

  • The student receives intrinsic reward for reaching the teacher’s goal alongside environment reward, and is trained to maximize discounted expected reward.The intrinsic goal is verified from the state; the environment supplies the extrinsic reward.Source: §3.1 · p. 4

3.2 TRAINING THE TEACHER

  • The teacher is rewarded positively when the student reaches a goal after more than t* steps, and negatively when it takes fewer steps or fails to reach it.The reward is evaluated when the goal is reached or at episode end, with separate weights for positive and negative reward.Source: §3.2 · p. 5
  • The effort threshold t* increases by one after ten consecutive successful goals that each take more than t* steps, adapting the target difficulty as the student improves.The authors adopted this heuristic because a fixed threshold varies across environments and requires careful calibration.Source: §3.2 · p. 5

3.3 TYPES OF GOALS

  • In the experiments, a goal is a tile coordinate whose state must change before the episode ends; it can be changed by reaching the tile or affecting its contents.This permits goals such as picking up keys, opening doors, or moving objects, and a coordinate may entail several subproblems to reach.Source: §3.3 · p. 5

3.4 AUXILIARY TEACHER LOSSES

  • Entropy regularization encourages teacher and student goal diversity, while rewarding novelty based on goal counts did not improve performance and was dropped.Entropy regularization together with threshold scheduling helps the teacher avoid local minima.Source: §3.4 · p. 6
  • Episode-boundary awareness rewards the teacher for choosing a goal location whose content changes between episodes, but may not suit every domain and cannot alone solve even medium-difficulty environments.The authors describe this heuristic as nonessential to AMIGO.Source: §3.4 · p. 6
  • Both teacher and student also receive environment reward when the student reaches the extrinsic goal, even if it differs from the teacher’s current intrinsic goal.This is intended to avoid the teacher encouraging the student away from an extrinsic goal it has learned to satisfy.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate AMIGO on procedurally generated MiniGrid tasks and compare it with reinforcement-learning and intrinsic-motivation baselines. They report stronger performance on harder exploration tasks and illustrate how the teacher’s proposed goals become more demanding.

4.1 ENVIRONMENTS

  • The six MiniGrid environments use full observations and change their layouts between episodes, requiring policies to handle varied configurations and transition dynamics.The tasks include key-corridor and obstacle-maze environments; actions include moving, manipulating objects, and interacting with doors.Source: §4 · p. 6, §4.1 · p. 7

4.2 AMIGO IMPLEMENTATION

  • A goal-generating teacher proposes goals at episode starts or when the student reaches a goal, while the student acts and receives reward at each step.Both policies are trained using TorchBeast’s IMPALA implementation.Source: §4.2 · p. 7

4.3 BASELINES AND EVALUATION

  • The comparison includes IMPALA, intrinsic-motivation baselines, and Asymmetric Self-Play, with each baseline evaluated using its best-performing observation configuration.Baseline comparisons consider full and partial observations, and the reported results average five seeds.Source: §4.3 · pp. 7–8

4.4 RESULTS AND DISCUSSION

  • AMIGO establishes a new state of the art on harder MiniGrid exploration problems, performs comparably to intrinsic-motivation methods on medium tasks, and often reaches the extrinsic goal on the hardest tasks.The reported comparison uses each baseline’s best-performing configuration for each environment.Source: Table 1 · p. 8, §4.4 · p. 8
  • IMPALA and Asymmetric Self-Play fail on medium and hard environments; ICM and Count struggle on medium tasks and fail on hard ones, while RND and RIDE struggle on harder environments.RND and RIDE perform competitively on medium environments.Source: §4.4 · p. 8
  • On OMhard, the teacher’s proposed goals progress from nearby targets to traversing rooms and opening doors, then to removing obstacles and interacting with objects.The figure also tracks the teacher’s target difficulty threshold across this trajectory.Source: Figure 3 · p. 9

5 CONCLUSION

AMIGO generates intrinsic goals through a teacher–student framework and outperforms competing intrinsic-motivation methods on challenging procedurally generated tasks. The conclusion describes the framework’s observation requirements and identifies directions for extending it.

5 CONCLUSION

  • AMIGO uses a goal-generating teacher to train a goal-conditioned student with intrinsic goals, supplementing extrinsic reward or replacing it when unavailable.Across 114 experiments on six tasks, it surpassed state-of-the-art intrinsic-motivation methods and obtained reward on some harder tasks where non-intrinsic RL failed.Source: §5 · p. 9
  • The experiments use absolute-coordinate goals, so both teacher and student require full environment observations in the evaluated setting.Partial observability is technically possible when absolute coordinates can still be provided and acted on.Source: §5 · p. 9
  • The authors identify linguistic goals, continuous control, visually rich domains, and more complex procedurally generated environments as future directions.They suggest language instructions could specify sequences of subgoals.Source: §5 · p. 9

A FULL RESULTS

Among intrinsic-motivation baselines, the best regime for tasks they reliably solve combines partially observed intrinsic rewards with a partially observed policy. Full-view rewards can fail to provide useful exploration signals in procedurally generated environments.

A FULL RESULTS

  • Baseline comparisons train on KCMedium and OMmedium, then use each method's best task-specific hyperparameters on the harder environment variants.Tables 2–5 report four training regimes.Source: A FULL RESULTS · p. 13
  • The harder-environment IMPALA values are presumed to be .00 because IMPALA fails to train on simpler environments.The KCmedium and OMmedium values come from Raileanu & Rockt¨aschel (2020).Source: A FULL RESULTS · p. 13
  • ASP was also checked on MiniGrid-Empty-Random-5x5-v0 and MiniGrid-KeyCorridorS3R1-v0, where it learned successfully.The check was used to validate the official PyTorch implementation.Source: A FULL RESULTS · p. 13
  • Partially observed intrinsic rewards with a partially observed policy were the best baseline regime for tasks the methods could reliably solve.The authors attribute the result to how full-view observations affect novelty and prediction-based rewards in procedurally generated environments.Source: A FULL RESULTS · p. 14
  • With full views, Count and RND treat almost all procedurally generated states as novel, so their rewards do not transfer knowledge across episodes.For ICM, the small observation change can make next-state prediction easy, causing intrinsic rewards to vanish early; for RIDE, it can make rewards nearly uniform across state-action pairs.Source: A FULL RESULTS · p. 14

B HYPERPARAMETER SWEEPS AND BEST VALUES

The appendix reports AMIGO's hyperparameter search and selected settings alongside the search procedures and best settings used for competing intrinsic-motivation methods.

B HYPERPARAMETER SWEEPS AND BEST VALUES

  • AMIGO's grid search varied batch sizes, learning rates, unroll length, entropy costs, embedding dimensions, and teacher-loss parameters α and β.The tested ranges included student batch size {8, 32, 150}, student learning rate {.001, .0001}, and α and β in {1.0, 0.7, 0.5, 0.3, 0.0}.Source: B HYPERPARAMETER SWEEPS AND BEST VALUES · p. 14
  • For RND, RIDE, COUNT, and ICM, the authors searched entropy and intrinsic-reward coefficients on KCmedium and OMmedium; ASP tuning varied three parameters.ASP's best settings were 10 sp steps, sp test rate .5, and Alice entropy .003.Source: B HYPERPARAMETER SWEEPS AND BEST VALUES · p. 14
  • AMIGO's selected settings include student and teacher batch sizes of 8 and 150, learning rates of .001 each, and α = 0.7 and β = 0.3.The chosen unroll length is 100; student and teacher entropy costs are .0005 and .01, respectively.Source: B HYPERPARAMETER SWEEPS AND BEST VALUES · p. 14
  • RND and COUNT each use partially observed intrinsic rewards and policies, with an entropy cost of 0.0005 and an intrinsic reward coefficient of 0.1.Source: B HYPERPARAMETER SWEEPS AND BEST VALUES · p. 14
  • RIDE and ICM use partially observed intrinsic rewards, with their policy-observation settings differing by method and environment.RIDE uses a partially observed policy in KCmedium and OMmedium; ICM uses a fully observed policy in KCmedium and a partially observed one in OMmedium.Source: B HYPERPARAMETER SWEEPS AND BEST VALUES · p. 14

C SAMPLE EFFICIENCY

AMIGO's training reward curves show faster convergence on the two easier environments than on the four harder ones. The reported convergence times rise from about 10 million steps to an order of 100 million steps.

C SAMPLE EFFICIENCY

  • The first two environments require less than 30 million steps, while OMmedhard and the three bottom-row environments require hundreds of millions of frames.The latter are described as substantially more challenging.Source: C SAMPLE EFFICIENCY · pp. 14–15
  • A method's invisible plot line indicates consistent failure to reach reward states during training.Figure 4 shows mean extrinsic reward over training time.Source: C SAMPLE EFFICIENCY · p. 15, Figure 4 · p. 15
  • About 10 million steps on KCmedium and OMmedium versus an order of 100 million on the four harder environments shows AMIGO's sample-efficiency contribution.Figure 4 plots mean extrinsic reward during training across two medium and four harder environments.Source: Figure 4 · p. 15, C SAMPLE EFFICIENCY · p. 15

D ABLATION STUDY

The ablations identify the combination used by the full model as the strongest configuration across environments. Removing extrinsic reward or the environment-change bonus harms performance on harder environments, while alternative gradual teacher rewards are not robust to learning dynamics.

D ABLATION STUDY

  • The study compares the full model with variants removing extrinsic reward or environment-change bonus, adding novelty, and changing the teacher reward form.The table measures steps in millions to reach final reward; zero means no extrinsic reward was obtained.Source: D ABLATION STUDY · pp. 15–16, Table 6 · p. 16
  • Removing extrinsic reward or the environment-change bonus leaves the model unable to solve harder environments; the table reports steps to convergence across environments.The full model's best results use a threshold reward with entropy regularization, extrinsic reward, and changing bonuses, but no novelty bonus.Source: Table 6 · p. 16, D ABLATION STUDY · p. 16
  • The Gaussian and linear-exponential teacher rewards are not robust to learning dynamics and can lead to goal-proposing rabbit holes with suboptimal rewards.The linear-exponential alternative rises toward the threshold and then decays exponentially as proposed goals become too hard.Source: D ABLATION STUDY · p. 16
  • Adding a novelty bonus was minimally beneficial in KChard and slightly ineffective in the other environments.The full model's best results omit the novelty bonus.Source: D ABLATION STUDY · pp. 15–16

E QUALITATIVE ANALYSIS

AMIGO’s learning trajectory moves through phases in which the teacher and student alternately cooperate, challenge one another, and explore. The teacher’s goals become progressively more complex, though OMhard remains difficult, with AMIGO solving it in only one of five runs.

E QUALITATIVE ANALYSIS

  • The five phases shift from easy nearby goals to harder challenges: teacher reward falls as the student masters nearby goals, and the two reach successive equilibria as difficulty rises.In the final phase, goals become unreachable and the student must improve.Source: E QUALITATIVE ANALYSIS · pp. 16–17
  • The components’ learning dynamics vary by phase: both benefit in the first, the fourth is adversarial, and the third and fifth require more exploration from both.The reported dynamics are associated with continual improvements to the agent’s policy.Source: E QUALITATIVE ANALYSIS · p. 17
  • In OMhard, AMIGO solved the task in only one of five runs despite a curriculum progressing from nearby goals to traversing rooms, opening doors, and interacting with objects.Figure 5 tracks student and teacher intrinsic rewards and goal difficulty, measured by t*; Figure 3 illustrates the evolving goals.Source: E QUALITATIVE ANALYSIS · pp. 17–18, Figure 5 · p. 17

F GOAL EXAMPLES

Figure 6 illustrates how teacher-proposed goals shift from nearby targets toward more complex interactions as learning progresses. This progression is often observed before the policy earns extrinsic reward.

F GOAL EXAMPLES

  • Goals typically progress from nearby targets to traversing rooms and opening doors, then to removing obstacles and interacting with objects, often before extrinsic reward appears.Figure 6 shows examples across early, mid, and late learning stages for KCmedium, OMhard, and OMmedium, respectively.Source: F GOAL EXAMPLES · p. 18, Figure 6 · p. 18
Loading 2006.12122v2…