PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Understand Goal Specifications by Modelling Reward

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, Edward Grefenstette

arXiv:1806.01946v4cs.AIcs.LG

TL;DR

Language-conditional reward functions can be difficult to implement as environments and language become more complex. AGILE jointly learns reward models from expert goal-state examples and trains instruction-conditioned policies with those rewards, achieving strong grid-world performance and supporting adaptation to changed dynamics without new expert examples.

  • Problem

    Instruction-following RL requires language-conditional reward functions whose implementation becomes difficult as environment and language complexity increases.

  • Method

    AGILE jointly trains an instruction-conditioned policy and reward model from expert instruction–goal-state pairs and policy-generated instruction–state pairs.

  • Results

    AGILE policies performed comparably with and learned as fast as policies trained against ground-truth reward and auxiliary tasks, while adapting to changed environment dynamics through reward-model reuse.

  • Takeaways & Limitations

    Separating goal-state recognition from goal-directed behavior lets learned reward models provide instruction-conditioned rewards and transfer across environment changes.

  • Takeaways & Limitations

    A substantial gap remains between the paper’s tasks and language and those likely to arise in real-world or more complex environments.

Abstract

from arXiv · show

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

1 INTRODUCTION

Instruction-following RL typically requires environment designers to implement language-conditional reward functions, a burden that grows with environmental and linguistic complexity. AGILE instead investigates learning such rewards from instruction–goal-state examples while training instruction-conditioned policies.

  • Motivation: Implementing instruction-conditional rewards requires an interpreter that evaluates instructions against environment states.This can be feasible in simple settings but becomes onerous as environments and admissible instructions grow more complex.
  • Motivation: Scaling reward implementation to natural language would require addressing ambiguity, underspecification, and fundamental language-understanding problems.
  • Research question: AGILE asks whether instruction-conditional reward functions can instead be learned and used to train instruction-conditional policies.The paper focuses on declarative commands specifying sets of possible goal states, with expert examples consisting only of goal states.
  • Experimental focus: The experiments compare AGILE-trained policies with environment-reward policies and construct instruction–goal-state data for colored, orientation-invariant block arrangements.The comparison includes learning speed and performance against environment reward and auxiliary reward-prediction objectives.

2 ADVERSARIAL GOAL-INDUCED LEARNING FROM EXAMPLES

AGILE jointly learns an instruction-conditioned policy and a discriminator-based reward model from expert goal states and policy experience. The framework separates recognizing what instruction-satisfying states look like from learning how to reach them, while addressing false negatives and enabling reward-model reuse.

  • AGILE framework: AGILE jointly trains an instruction-conditioned policy and discriminator reward model rather than relying on an environment instruction interpreter.The policy maximizes expected return using modelled stepwise rewards, while the discriminator predicts whether a state satisfies an instruction.
  • AGILE framework: The policy reward is 1 when Dφ(c, st) > 0.5 and 0 otherwise, replacing ground-truth environment reward in the training objective.
  • AGILE framework: The reward model distinguishes instruction–goal-state pairs from the expert dataset and instruction–state pairs generated by the agent.
  • False negatives: As the policy improves, treating its objective goal states as negatives can create false negatives that reduce discriminator accuracy and worsen policy learning.
  • False negatives: The anticipated negative rate ρ should be tuned per task because using all policy-buffer examples as negatives inhibits reward-model learning, while too few may remove informative negatives.
  • Reward-model reuse: AGILE separates recognizing instruction-defined goal states from learning the behavior needed to reach them, supporting reuse of the reward model across policies or environments.
  • Relation to GAIL: Unlike GAIL’s trajectory discrimination, AGILE conditions learning on instructions and distinguishes instruction–goal-state pairs from policy-generated instruction–state pairs.

3 EXPERIMENTS

Experiments evaluate AGILE in GridLU tasks involving spatial relations and abstract arrangements, using both structured and structure-agnostic instruction models. AGILE generalizes beyond memorization, reaches high performance, learns reward models from limited examples, and supports adaptation after environment changes.

  • GridLU-Relations: GridLU-Relations requires agents to induce five spatial relationships and manipulate blocks to instantiate them.The task includes relations such as NorthFrom, SouthFrom, EastFrom, WestFrom, and SameLocation.
  • GridLU-Relations: AGILE-A3C achieved 99.5% performance, while A3C with auxiliary reward prediction reached 99.9% on GridLU-Relations.The auxiliary reward-prediction objective substantially improved A3C, while AGILE-A3C achieved nearly perfect performance using an induced reward function.
  • Data efficiency: With only 8000 training examples, AGILE-A3C reached 60% performance, while optimal performance required more than 100,000 examples.The reward model therefore induced useful reward estimates from a limited set of instruction–goal-state pairs, though larger datasets improved performance.
  • Model architectures: AGILE with a structure-agnostic FiLM-LSTM model achieved a 97.5% success rate and trained almost as fast as an RL-RP agent with the same architecture.The structure-agnostic model demonstrates that AGILE is not restricted to the structured NMN architecture.
  • GridLU-Relations: The reward model peaked at 99.5% accuracy, with larger anticipated negative rates learning faster initially but deteriorating later; ρ = 25% remained stable.False-negative rates stayed below 3% of rewards, while early false positives were more common.
  • Environment adaptation: When red-square objects became immovable, success on relevant instructions fell from 98% to 52%, then recovered to 69.3% after policy fine-tuning with the frozen reward model.The result indicates that the learned reward model could be reused while adapting the policy to changed environment dynamics.
  • GridLU-Arrangements: On GridLU-Arrangements, the agent produced the correct arrangement in 58% of 200 manually evaluated episodes, with most failures involving eight blocks.The task contains 36 instructions referring to 390 million correct goal-states, and evaluation used human judgment rather than a programmed reward function.

4 RELATED WORK

The paper situates AGILE within instruction-following, semantic parsing, and apprenticeship-learning approaches. It emphasizes learning reward models from demonstrations rather than relying on fully specified execution or reward systems.

  • Instruction-following RL commonly rewards agents for reaching locations or objects, while other approaches predict formal representations or actions from demonstrations.
  • Some instruction-following methods require a predefined execution model or hard-coded trajectory comparison, which can constrain applicability across task types.
  • AGILE belongs to apprenticeship learning, which studies performing tasks from demonstrations and feedback.
  • AGILE is closely related to GAIL but learns instruction-conditioned reward models without requiring a formal reward specification language.

5 DISCUSSION

AGILE trains instruction-conditional policies with learned rewards jointly shaped by expert and agent data. The experiments indicate comparable learning and performance to strong reward-based baselines, while revealing generalization benefits and clear limits on task realism.

  • AGILE trains instruction-conditional RL agents using reward models jointly learned from expert and agent-provided data instead of environment instruction interpreters.
  • AGILE policies perform comparably with and learn as fast as policies trained using ground-truth reward plus auxiliary tasks.
  • False positives early in training help policies begin learning, while the learned reward model may provide shaped reward that mitigates sparse-reward issues.
  • Separating what should be done from how it should be done improves generalization to changed environment dynamics; fine-tuning with a frozen reward model recovers some capability.
  • The studied tasks and language remain far from real-world or more complex environments, and applying AGILE to realistic 3D settings requires handling viewpoint and body-visibility differences.

A AGILE PSEUDOCODE

AGILE alternates discriminator training with policy training. The discriminator distinguishes expert examples from selected agent states, and its classifications provide binary rewards for reinforcement-learning updates.

  • Discriminator Training: The discriminator-training procedure samples instruction-state instances, generates policy rollouts, stores them in a replay buffer, and updates the discriminator from positive and negative batches.
  • Discriminator Training: Negative examples are sampled in excess and the highest-scoring fraction is rejected according to the anticipated negative rate ρ.
  • Policy Training: The policy-training procedure uses the discriminator to compute rewards rτ = [Dφ(c, sτ) > 0.5] and applies reinforcement-learning updates to the policy.
  • Reward Choice: The paper notes that the appropriate GAIL-style reward would be −log(1−Dφ(c, st)), but AGILE did not test this alternative and only hypothesizes that it would work.

C TRAINING DETAILS

Training used concurrent A3C policy and discriminator optimization with RMSProp, alongside task-specific GridLU settings and regularization. The environment was a fully observable 5 × 5 block-based gridworld.

  • Optimization: AGILE trained the policy and discriminator concurrently using RMSProp and A3C.
  • Discriminator training: Discriminator training used a designated worker and required clipping each weight-matrix column to L2 norm at most 1.
  • Environment: GridLU is a 5 × 5 gridworld containing blocks with three shapes and three colors, plus an agent that can carry, pick up, and drop blocks.
  • Environment: The agent’s carrying state is explicitly color-coded so the world state remains fully observable.
  • Observations: GridLU observations use 56 × 56 images, with 8 × 8 block resolution needed to distinguish shapes and walls also displayed.

E EXPERIMENT DETAILS

Experiments evaluated repeated training, fixed training horizons, data efficiency, and checkpoint selection across several GridLU settings. AGILE exceeded 50% success with 8,000 examples, while best performance required up to 130,000.

  • Evaluation protocol: Every experiment was repeated five times and the average result was reported.
  • RL vs. AGILE: All agents in the RL-versus-AGILE comparison were trained for 5 · 10^8 steps.
  • Data Efficiency: Data-efficiency policies were trained for 5 · 10^8 steps, reporting each policy’s maximum success rate during training.
  • GridLU-Arrangements: For GridLU-Arrangements, training lasted 100M steps and used the checkpoint that best fooled the discriminator according to internal reward.
  • Data Efficiency: AGILE exceeded 50% success from 8,000 instruction–goal-state examples, while best performance required as many as 130,000 examples.

F.1 GRIDLU RELATIONS INSTANCE GENERATOR

The GridLU-Relations generator combines a formal instruction grammar with procedurally sampled objects, agent positions, distractors, and rejection checks. It yields 990 unique instructions and a very large space of task instances.

  • Instruction grammar: The grammar defines go-to and bring-to instructions using spatial relations, object descriptions, and the AGENT constant.
  • Instruction grammar: There are 990 unique instructions: 150 go-to and 840 bring-to possibilities, excluding the stated exception.
  • Instance sampling: Each instruction generates the objects required by its partial specification, plus one uniformly sampled distractor object, with objects and agent placed uniformly.
  • Instance sampling: Generated instances are rejected when objects overlap or all objects are identical, and sampling restarts after either sanity-check failure.
  • Sampling distribution: Go-to instructions comprise approximately 25% of generated instances despite representing approximately 15% of all possible instructions because of rejection sampling.
  • Task-space size: Each bring-to instruction has at least K = 2300 · 9 ≈ 2 · 10^4 associated initial arrangements, yielding at least 1.7 · 10^7 task instances overall.

F.2 DISCRIMINATOR EVALUATION

Discriminator evaluation compared its predictions with the ground-truth reward checker during GridLU-Relations training. The monitored error measures were false-positive and false-negative percentages.

  • Evaluation: During GridLU-Relations training, discriminator predictions were compared with the ground-truth reward checker to monitor performance indicators.
  • Error measures: Figure 8 reports discriminator false-positive percentage on the left and false-negative percentage on the right.

G ANALYSIS OF THE GRIDLU-ARRANGEMENTS TASK

The GridLU-Arrangements analysis specifies instruction syntax and evaluates FiLM-based neural architectures, including a modular discriminator and policy network. The models compose instruction-selected modules over visual states, while reward prediction and baseline components support learning and evaluation.

  • Instruction syntax: GridLU-Arrangements uses arrangement-only instructions and instructions that additionally specify block colors, such as Connected(AGENT, SCENE) and Snake(AGENT, Color(’yellow’, SCENE)).
  • FiLM-NMN architecture: The FiLM-NMN discriminator processes 56x56 RGB states with a convolutional stem before applying instruction-selected neural modules.
  • FiLM-NMN architecture: The model uses 12 modules—covering color, shape, relation, and AGENT tokens—whose connections follow the instruction and whose root output is processed further.
  • FiLM-NMN architecture: Unary modules receive one feature input and a zeroed second input, allowing the same convolutional weights to be used for unary and binary modules.
  • Discriminator and policy networks: The discriminator max-pools the FiLM-NMN output and feeds it to a 100-unit hidden-layer MLP, while the policy adds a convolutional merge and outputs softmax probabilities for five actions.
  • Alternative architecture: Structure-agnostic models use an LSTM with 100 hidden units to predict FiLM biases and multipliers for a five-layer convolutional network.
Loading 1806.01946v4…