PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

Tianmin Shu, Caiming Xiong, Richard Socher

arXiv:1712.07294v1cs.AI

TL;DR

The paper addresses the difficulty of accumulating and composing multiple skills for complex reinforcement-learning tasks. It introduces a hierarchical, language-interpretable policy modulated by a stochastic temporal grammar, and evaluates it on Minecraft object-manipulation tasks, where it supports skill reuse, generalization, and efficient learning.

  • Problem

    Accumulating multiple skills in one reinforcement-learning policy is difficult, although complex tasks require combining different skills.

  • Method

    A multi-level hierarchical policy progressively acquires tasks, reuses earlier policies through human-language instructions, and uses a stochastic temporal grammar to select learned or new skills.

  • Results

    Minecraft experiments show that the full model has higher learning efficiency than a flat policy, generalizes to unseen environments, and composes interpretable hierarchical plans.

  • Takeaways & Limitations

    Human-language access to previously learned policies makes hierarchical skill reuse and composed plans interpretable within the evaluated multi-task setting.

  • Takeaways & Limitations

    The training stages currently rely on weak human supervision to define which skills should be learned.

Abstract

from arXiv · show

Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.

1 INTRODUCTION

The paper proposes a multi-level hierarchical policy that reuses learned skills, represents decisions with human instructions, and uses a stochastic temporal grammar to model task-order dependencies. Minecraft experiments show that the framework supports efficient multi-task learning, interpretable skill reuse, and temporal prediction.

  • 1 INTRODUCTION: The framework uses a hierarchical policy to reuse previously learned skills as components of new skills during progressive multi-task learning.The policy accumulates tasks from terminal policies toward higher-level policies and can unfold global plans into basic actions.
  • 1 INTRODUCTION: Tasks are encoded as human instructions, allowing the agent to communicate policy choices and generate interpretable plans in natural language.For stacking blue blocks, the hierarchy composes instructions such as “Get blue,” “Find blue,” and “Put blue.”
  • 1 INTRODUCTION: A stochastic temporal grammar models sequences of previously learned-skill or new-skill selections to capture task priorities such as obtaining an object before putting it down.The grammar is trained on positive episodes and modulates hierarchical policy selection.
  • 1 INTRODUCTION: Minecraft object-manipulation experiments demonstrate efficient hierarchical policy learning, human-instruction-based skill reuse, improved explainability, and self-supervised prediction of future actions.The evaluation targets multi-task reinforcement learning in a Minecraft world.

2 RELATED WORK

Prior work includes distillation-based knowledge transfer and modular two-layer hierarchical policies, but these approaches impose limits on task composition and policy execution. This paper instead targets multi-level composition with human-language task descriptions and explicit instruction generation.

  • 2 RELATED WORK: Existing multi-task reinforcement learning approaches mainly use knowledge transfer through distillation or modular two-layer hierarchical policy designs.The paper positions its multi-level policy closer to modular hierarchical approaches.
  • 2 RELATED WORK: Two-layer hierarchical policies limit flexibility because global policies cannot use simpler task policies as base policies and are assumed to execute through local option policies.These restrictions motivate a multi-level global policy.
  • 2 RELATED WORK: The proposed approach grounds human instructions in both visual knowledge and policies while training the agent to utter instructions as explicit explanations of decisions.The paper describes this as composing complex plans from simpler, human-described tasks.

3 MODEL

The model progressively builds hierarchical policies that reuse previously learned tasks through human-language instructions, while a stochastic temporal grammar guides task sequencing and branch selection.

  • 3.1 MULTITASK RL SETTING: The framework progressively expands task sets from G0 to GK, using each previous global policy πk−1 as the base policy for newly added tasks.Weak human supervision specifies task augmentation stages, while the model can support arbitrary augmentation orders.
  • 3.2 HIERARCHICAL POLICY: A hierarchical policy decomposes new tasks into base tasks and novel actions, allowing πk to reuse πk−1 alongside an augmented flat policy.The hierarchy includes a base policy, instruction policy, switch policy, and augmented policy.
  • 3.2 HIERARCHICAL POLICY: The instruction policy maps the current state and task to a base task, while the switch policy selects between executing that base task and taking a primitive action.The switch variable e is binary, and the instruction policy supplies tasks for the previous-stage policy to execute.
  • 3.3 STOCHASTIC TEMPORAL GRAMMAR: The stochastic temporal grammar models transitions over branch selections and instructions, providing a prior for the switch and instruction policies.Primitive action sampling is not affected by the STG.
  • 3.3 STOCHASTIC TEMPORAL GRAMMAR: Once triggered, a base policy runs to completion before the global policy chooses its next move, enabling plan composition from human-specified tasks.The resulting episode composes a plan for the task instruction at hierarchy level k.

4 LEARNING

Learning uses a two-phase curriculum with A2C optimization, alternating updates, and an STG estimated from successful episodes. The curriculum first teaches reuse of base skills, then jointly learns when to reuse them or execute novel skills.

  • 4 LEARNING: Each skill-acquisition stage first trains on base tasks, then trains on the full task set so the policy learns both skill reuse and novel-task execution.The base phase ends when average reward across base tasks exceeds a threshold.
  • 4 LEARNING: A2C trains the policies in both curriculum phases, while STG distributions are estimated from accumulated positive episodes.The STG is updated using successful trajectories rather than every episode.
  • 4.1 POLICY OPTIMIZATION BY ADVANTAGE ACTOR-CRITIC: The policy-gradient objective separates updates for the switch, instruction, and augmented policies, using importance-sampling weights and corresponding advantage estimates.The value functions include a separate switch value function for branch selection.
  • 4.1 POLICY OPTIMIZATION BY ADVANTAGE ACTOR-CRITIC: The switch value function estimates expected return conditioned on the current branch selection, complementing the global value function.The advantage estimates are defined from empirical returns and value-function estimates.
  • 4 LEARNING: Alternating updates stabilize joint optimization by training one sub-policy at a time while holding the other two fixed for M = 500 iterations.The procedure is applied within both phases of curriculum learning.
  • 4.2 LEARNING AN STG: Early episodes use ϵ-greedy exploration to reduce the risk of local minima before the learned STG begins guiding exploration.The STG starts to guide exploration as training progresses.

5 EXPERIMENTS

Experiments in Minecraft evaluate learning efficiency, curriculum learning, and policy generalization for hierarchical multi-task policies. The full model converges faster than baselines, benefits from a two-phase curriculum, and generalizes better to distractors and larger rooms.

  • Game environment and task specifications: The Minecraft environment uses two rooms, six block colors, and task groups progressing from finding to getting, putting, and stacking objects.Episodes randomly place colored blocks, and the experiments use a sparse terminal reward scheme with +1 for task completion and −0.5 for invalid instructions.
  • Learning efficiency: Within 22,000 episodes, all hierarchical variants converge, while the flat policy remains below 0.8 average reward; the full model also completes phase 1 faster.Figure 4a compares methods training π1 for G1 from the same π0.
  • Learning efficiency: The full model converges fastest and reaches the highest converged reward when adding new tasks during phase 2, with a larger advantage on more complex tasks.This comparison uses a pretrained π3 and evaluates continued learning with newly added tasks.
  • Learning efficiency: The two-phase curriculum accelerates convergence, supporting reuse of skills learned by the base policy.Figure 5 compares learning curves with and without curriculum learning.
  • Policy generalization: In distractor-rich rooms, hierarchical policy success remains 94%, whereas flat-policy success drops to 29% after training in the simpler one-item setting.Both policies perform near perfectly in the simple setting; the contrast appears when multiple items are present.
  • Policy generalization: In the larger room created by removing the wall, the full model suffers a smaller testing-success drop than the flat policy across tasks.The authors attribute this robustness mainly to repeatedly calling the base policy until the goal is achieved.
  • Policy interpretability: Hierarchical plans adapt to context: a policy may skip an already unnecessary base call or repeat a failed base-task call.The examples show π1 omitting π0 when already near the target and invoking “Find x” twice when the first call fails.

6 CONCLUSION

The paper presents a hierarchical policy modulated by a stochastic temporal grammar for efficient multi-task reinforcement learning, evaluated in Minecraft. The full model learns more efficiently than a flat policy, generalizes to unseen environments, and composes interpretable hierarchical plans, while training-stage skill definitions still require human weak supervision.

  • 6 CONCLUSION: The full model has significantly higher learning efficiency than a flat policy, generalizes well in unseen environments, and composes interpretable hierarchical plans.These results were demonstrated in Minecraft games.
  • 6 CONCLUSION: The framework uses a hierarchical policy modulated by a stochastic temporal grammar across multiple training stages for multi-task reinforcement learning.
  • 6 CONCLUSION: Human weak supervision currently defines which skills are learned at each training stage.The authors plan to automatically discover training procedures that increase the task set.

A PSEUDO CODE OF OUR ALGORITHMS

The algorithms generate trajectories by selecting either base policies or new actions under hierarchical policies, with the stochastic temporal grammar guiding policy selection. Training alternates through learning phases using replay memories, positive episodes, and policy and value-function parameters.

  • A PSEUDO CODE OF OUR ALGORITHMS: At higher policy levels, the algorithm samples a switching decision and instruction, then either executes a base policy or samples an action from the augmented policy.At the terminal level, actions are sampled directly from the policy and executed.
  • A PSEUDO CODE OF OUR ALGORITHMS: The trajectory records states, actions or switching decisions, rewards, policy distributions, and task instructions until the horizon or a nonzero reward is reached.
  • A PSEUDO CODE OF OUR ALGORITHMS: Algorithm 2 initializes replay memories, positive-episode storage, policy and value-function parameters, and uniform stochastic-temporal-grammar distributions.
  • A PSEUDO CODE OF OUR ALGORITHMS: Training specifies a discount parameter, iteration limit, update-rotation frequency, and reward threshold before repeatedly determining the current learning phase.

B ARCHITECTURES OF MODULES

The architecture combines visual and language encoders with recurrent fusion, then uses specialized policy, value, and selector modules to choose between newly generated actions and reusable base-policy actions. Visual inputs are reduced to 256 dimensions, while instructions are represented as 128-dimensional bag-of-words vectors.

  • B ARCHITECTURES OF MODULES: The visual encoder uses three convolutional layers and a fully connected layer to produce a 256-dimensional visual feature.The convolutional layers use 32, 64, and 64 filters respectively.
  • B ARCHITECTURES OF MODULES: The instruction encoder embeds words into 128-dimensional vectors and combines them with bag-of-words into one 128-dimensional output.
  • B ARCHITECTURES OF MODULES: The fusion layer concatenates visual and language representations into a 384-dimensional vector, then processes it with an LSTM having 256 hidden units.
  • B ARCHITECTURES OF MODULES: The modules separately output switching, skill-selection, item-selection, augmented-action, and value-function distributions or values.
  • B ARCHITECTURES OF MODULES: The selector chooses an action from the augmented or base policy according to the switching decision sampled by the switch policy.

C COMPOSED HIERARCHICAL PLANS

Figure 6 presents hierarchical plans composed by global policies for different tasks. Each plan starts at the top-level policy, and branches proceed from left to right as consecutive policy steps, alongside critical visual and held-item states.

  • C COMPOSED HIERARCHICAL PLANS: Figure 6 shows several hierarchical plans for different tasks composed by executing the learned global policies.
  • C COMPOSED HIERARCHICAL PLANS: All tasks begin with the top-level policy, whose branches represent consecutive steps ordered from left to right in time.
  • C COMPOSED HIERARCHICAL PLANS: The figure also displays egocentric views and held items at critical moments in a real episode example.
Loading 1712.07294v1…