PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli

arXiv:1706.05064v2cs.AIcs.LG

TL;DR

The paper studies zero-shot RL agents that must execute unseen and longer sequences of instructions after learning subtask skills. It combines analogy-based parameterized skills with a hierarchical meta controller that learns when to update subtasks, and reports successful generalization in a stochastic 3D domain.

  • Problem

    The paper addresses RL agents' need to execute instruction sequences while generalizing to unseen instructions and longer sequences.

  • Method

    The approach learns analogy-based parameterized skills and a hierarchical meta controller with a learned subtask-update time-scale.

  • Results

    The architecture generalizes to longer sequences of seen and unseen instructions, while the analogy-making objective generalizes to unseen parameterized tasks.

  • Takeaways & Limitations

    Learning correspondences between subtasks and when to update them supports zero-shot instruction generalization under delayed sequential decisions.

Abstract

from arXiv · show

As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.

1. Introduction

The paper introduces instruction execution as an RL problem requiring generalization to unseen subtasks and longer instruction sequences while handling delayed rewards, interruptions, and memory. It addresses these challenges with analogy-based parameterized skills and a hierarchical meta controller that learns when to update subtasks.

  • Problem: The instruction execution problem requires following a natural-language list of subtasks while responding to unexpected events.Unexpected events include bonus opportunities and emergencies, which may justify interrupting an ongoing subtask.
  • Generalization: The agent must generalize from pre-trained subtasks to unseen subtasks and longer sequences of seen and unseen instructions.Training uses only a subset of subtasks and short instruction sequences, while evaluation includes unseen subtasks and longer sequences.
  • Challenges: Delayed reward requires tracking the current instruction because success is provided only after the full instruction list is completed.The agent receives no environment signal for completing individual instructions and must decide when to advance.
  • Approach: The proposed approach learns parameterized skills with an analogy-making objective, then trains a meta controller to select subtasks from instructions.The analogy objective encourages correspondences between similar subtasks, while the hierarchical controller operates over the learned skills.
  • Approach: A learned subtask-update mechanism improves efficiency under delayed reward and permits interruption of ongoing subtasks.The mechanism lets the meta controller learn when to update the subtask rather than blindly proceeding through instructions.
  • Results: Experiments in a Minecraft-based stochastic 3D environment show successful generalization of parameterized subtasks and instruction sequences.The introduction reports successful analogy-based generalization and that the learned update mechanism outperforms several hierarchical baselines.

2. Related Work

Prior work addresses hierarchical control, zero-shot task generalization, and instruction execution, but commonly assumes fixed task sequences, non-interruptible subtasks, or supervised language grounding. This paper positions its approach around generalizing instruction sequences without language supervision.

  • Hierarchical RL: Hierarchical RL commonly combines a meta controller with lower-level subtask controllers, but much prior work assumes a fixed overall task.Earlier approaches typically use a high-level controller to organize sequential subtasks within a predetermined task structure.
  • Hierarchical Deep RL: Recent hierarchical deep RL methods improve exploration, discover skills, or modulate low-level behavior through high-level controllers.Examples include hierarchical Deep Q-Learning, option-critic, continuous meta-controller actions, and skill discovery with mutual information.
  • Hierarchical Deep RL: Most related hierarchical approaches use open-loop high-level policies that wait for a subtask to finish and therefore cannot interrupt it in principle.This limitation is directly relevant to environments containing unexpected events during instruction execution.
  • Zero-Shot Task Generalization: Zero-shot task-generalization work includes parameterized skills, dictionary learning, universal value function approximators, and shared sub-networks.These methods target generalization across task descriptions, state-goal pairs, or tasks and robots.
  • Instruction Execution: Instruction-execution research has focused on mapping natural-language instructions to actions or groundings in supervised settings.In contrast, this paper focuses on generalization across instruction sequences without language supervision.

3. Learning a Parameterized Skill

The paper defines parameterized skills that map task parameters and observations to actions and termination signals, then trains their embeddings to generalize across unseen task combinations. Experiments in a stochastic Minecraft environment evaluate analogy-based generalization across independent, object-dependent, and repetition-count tasks.

  • Skill definition: A parameterized skill maps observations and categorical task parameters to primitive actions and a binary task-completion signal.The task space is formed as a Cartesian product of parameter sets, such as action types and target objects.
  • Generalization challenge: Because training exposes only a subset of tasks, the task embedding must capture relationships among parameters to support unseen-task generalization.The paper explicitly targets generalization beyond the observed subset G′ ⊂ G.
  • Analogy-making objective: The analogy-making objective enforces consistent embedding differences between analogous tasks while additional distance constraints prevent trivial representations.For example, action–object combinations can form parallelogram-like relationships in the embedding space.
  • Optimization: Training combines actor-critic learning, termination prediction, and a separate analogy-making objective applied to task embeddings.Policy distillation pre-training was also found to give slightly better results, according to the paper.
  • Experimental setting: The Minecraft environment uses 64 × 64 RGB observations, seven object types, randomly generated worlds, and nine primitive actions.The agent interacts with objects through actions including movement, pickup, transformation, looking, and no operation.
  • Results: Analogy-based parameterized skills successfully generalized to unseen tasks in independent, object-dependent, and interpolation/extrapolation scenarios without directly experiencing them.The scenarios vary unseen action–object combinations, object-group dependencies, and repetition counts; training repetition counts were {1, 3, 5}, with evaluation on {2, 4, 6, 7}.

4. Learning to Execute Instructions using Parameterized Skill

The paper uses a hierarchical architecture in which a meta controller reads instruction sequences, selects parameters for learned skills, and manages progress, interruptions, and delayed rewards. A learned update decision lets it operate at a larger time-scale and improves execution of long and uncertain instruction sequences.

  • 4. Learning to Execute Instructions using Parameterized Skill: The architecture separates instruction execution into a meta controller that selects subtask parameters and a parameterized skill that performs subtasks and returns termination signals.The meta controller conditions decisions on observations, the instruction list, the previous subtask, and its termination signal.
  • 4.1. Meta Controller Architecture: The architecture can update subtasks at any time rather than only after termination or a fixed number of steps, enabling interruption of ongoing subtasks.This flexibility addresses unexpected events that require switching away from an ongoing instruction.
  • 4.1.1. CONTEXT: The meta controller maintains progress by summarizing observations and ongoing-subtask history in an LSTM context used by the subtask updater.The context incorporates the previous selected subtask and whether that subtask has terminated.
  • 4.1.2. SUBTASK UPDATER: The subtask updater embeds the instruction list, retrieves the instruction selected by a pointer, and computes the parameters for the corresponding subtask.Each instruction becomes a sentence embedding, while the pointer indicates which instruction the meta controller is executing.
  • 4.1.2. SUBTASK UPDATER: A location-based addressing mechanism shifts the instruction pointer among -1, 0, and +1, keeping it unchanged during execution and advancing it when an instruction finishes.The shift distribution is produced by a neural network over the three shift operations.
  • 4.2. Learning to Operate at a Large Time-Scale: The meta controller learns a binary decision c_t indicating whether to invoke the subtask updater, allowing one updater decision to produce multiple environment actions.When c_t = 0, the current subtask and internal states continue; when c_t = 1, the updater changes them.
  • 4.4. Experiments: After training on instruction sequences of length 4, Hierarchical-Dynamic finishes 18 of 20 seen instructions and 14 of 20 unseen instructions in length-20 sequences.Performance decreases as the number of instructions increases, but the architecture handles relatively long seen and unseen sequences.
  • 4.4. Experiments: The flat baseline fails even on training instructions, while fixed-time-scale hierarchical baselines struggle with loops or uncertain events that require precise updates and interruptions.Hierarchical-Short advances too early on loop instructions, whereas Hierarchical-Long can waste time pursuing bonus opportunities such as finding a randomly appearing box.

5. Conclusion

The paper introduces zero-shot task generalization in RL through analogy-based skill generalization and hierarchical execution of instruction sequences. Experiments show generalization to longer and unseen instructions in a stochastic 3D domain.

  • The paper studies RL agents that execute and generalize over sequences of instructions.
  • An analogy-making objective enables generalization over unseen parameterized tasks in various scenarios.
  • A learned meta-controller time-scale is more efficient and flexible for interrupting subtasks and handling delayed sequential decisions.
  • Empirical results show the architecture generalizes to longer sequences of instructions and unseen instructions.

A. Experimental Setting for Parameterized Tasks

Parameterized-task experiments evaluate analogy-based generalization across independent, object-dependent, and inter/extrapolation scenarios. Training varies which task parameters or object relationships are observed.

  • Evaluation protocol: Episodes terminate after 50 steps in the Independent and Object-dependent cases and after 70 steps in the Inter/Extrapolation case.
  • Independent: The Independent scenario keeps task semantics consistent across all target-object types and defines analogies across object substitutions.
  • Object-dependent: The Object-dependent scenario assigns objects to groups where interaction means pickup for Group A and transformation for Group B.
  • Inter/Extrapolation: The Inter/Extrapolation scenario defines tasks by action, object, and repetition number, training on only a subset of numbers.
  • Training sets: Tables 3 and 4 specify the training task sets for the Independent and Object-dependent cases.

B. Experimental Setting for Instruction Execution

Instruction-execution experiments train and evaluate sequences sampled from specified training and unseen instruction sets. Evaluation on longer instructions uses substantially longer episodes than meta-controller training.

  • Protocol: Meta-controller training episodes terminate after 80 steps, whereas evaluation on longer instructions terminates after 600 steps.
  • Unexpected events: A box appears with probability 0.03 and disappears after 20 steps during the instruction-execution setting.
  • Instruction sets: Training and unseen instruction sets are specified in Table 5, with sequences generated by sampling from these sets.

C. Experiment on 2D Grid-World

The 2D grid-world uses binary object-channel observations, primitive movement and interaction actions, stochastic rewards and hazards, and evaluations of parameterized skills and instruction execution.

  • Environment: The environment represents observations as an 18×10×10 binary tensor, with one channel per object type over a 10×10 grid.
  • Environment: The agent has 13 primitive actions comprising no-operation, movement, pickup, and transformation actions in four directions.
  • Rewards and hazards: Rewards include +1 for completing all instructions, −0.1 per time-step, −0.3 for visiting water, and +0.9 for transforming an enemy.
  • Parameterized-task evaluation: Parameterized skills with analogy-making successfully generalize to unseen tasks in the Independent and Object-dependent scenarios.
  • Parameterized-task evaluation: The analogy-trained skill learns high critic values around target objects for unseen tasks.
  • Instruction execution: The instruction set includes Visit, Pick up, Transform, Pick up all, and Transform all, with the last two repeating subtasks until targets disappear.
  • Instruction-execution results: The Hierarchical-Dynamic architecture outperforms Hierarchical-Short and improves with instruction length relative to the flat baseline.
  • Instruction-execution results: The learned time-scale is described as crucial for delayed reward, while enemy handling lets the architecture outperform the Shortest Path method.

D.1. Parameterized Skill

The parameterized skill is distilled from task-specific teacher policies, then fine-tuned with actor-critic learning while predicting termination and using analogy-making regularization.

  • Termination prediction: A binary classification objective trains the parameterized skill to predict whether the current state is terminal.The termination objective is a cross-entropy loss based on whether the state belongs to the task’s terminal set.
  • Policy distillation: Policy distillation trains a multi-task parameterized skill to mimic separate teacher policies for each task by minimizing their KL divergence.Training samples task mini-batches, generates episodes with the parameterized skill, and predicts teacher actions.
  • Actor-critic fine-tuning: The parameterized skill is fine-tuned with actor-critic learning using generalized advantage estimation after distillation.The advantage estimate uses γ and λ to control discounting and the bias–variance balance.
  • Analogy-making regularization: The final parameterized-skill update includes an analogy-making regularizer formed from similarity, disentanglement, and difference objectives.The regularizer is LAM = Lsim + ρ1Ldis + ρ2Ldiff, with ρ1, ρ2, and ξ as objective-specific hyperparameters.

D.2. Meta Controller

The meta controller is updated with actor-critic learning and generalized advantage estimation, while its update-frequency penalty discourages overly frequent subtask updates.

  • Meta-controller learning: Actor-critic learning with generalized advantage estimation updates the meta controller’s parameters.The update follows the stated meta-controller learning objective and its final update rule.
  • Update decision: An L1 penalty on the probability of updating penalizes frequent update decisions in the meta controller.η weights this update penalty.
  • Analogy-making objective: The final meta-controller update incorporates the analogy-making objective.The formulation identifies LAM as the analogy-making objective used in the update.

E. Architectures and Hyperparameters

The architectures use multiplicative interactions to condition neural layers on task information, with separate parameterized-skill, meta-controller, and flat-controller designs trained under specified curricula and hyperparameters.

  • Multiplicative interaction: Multiplicative interactions convert condition-variable embeddings into convolutional or fully connected layer weights instead of concatenating the variables.The approach is used in both the parameterized skill and the meta controller, with matrix factorization reducing fully connected parameters.
  • Parameterized skill: The parameterized skill combines convolutional layers and an LSTM with task-parameter embeddings, producing action, termination, and baseline outputs.Its subtask embedding is computed from two task parameters and transformed into selected network weights through multiplicative interaction.
  • Meta controller: The meta controller combines visual features, previous subtasks, retrieved instructions, and termination information to compute subtask-selection and update-related outputs.It uses a 256-unit LSTM, bag-of-words instruction embeddings, and separate networks for shifting, updating, and selecting subtask parameters.
  • Hyperparameters: The soft architecture is trained with curriculum learning and entropy regularization, then fine-tuned without curriculum learning; η = 0.001 penalizes frequent updates.The soft architecture uses learning rates of 2.5e −4 during curriculum training and 1e −4 during fine-tuning.
  • Flat controller: The flat controller uses the meta-controller layers but transforms the previously retrieved instruction to produce probabilities over primitive actions.The transformed instruction computes the weights of Conv3 and the LSTM through an MLP.
  • Curriculum learning: Training samples world sizes, wall and object densities, and instruction counts from specified ranges as curriculum-learning conditions.The meta controller is trained on instruction sequences containing 1, 2, 3, or 4 instructions, with sampling ranges determined by success rate.
Loading 1706.05064v2…