Source-linked AI summary
A Deep Hierarchical Approach to Lifelong Learning in Minecraft
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, Shie Mannor
TL;DR
Lifelong-learning agents must reuse knowledge across tasks while retaining what they have learned, but high-dimensional environments such as Minecraft make this difficult. The paper introduces H-DRLN, which learns reusable Deep Skill Networks and incorporates them through a skill array or skill distillation. H-DRLN shows superior learning performance and faster convergence than DDQN in Minecraft sub-domains, while the paper identifies online skill learning and refinement as future work.
Problem
High-dimensional lifelong learning requires agents to retain knowledge, selectively reuse skills, and scale as new tasks and skills are encountered.
Method
H-DRLN learns reusable Deep Skill Networks and incorporates them through a DSN array or a distilled multi-skill network.
Results
H-DRLN provides superior learning performance and faster convergence compared to DDQN in Minecraft sub-domains by using skills.
Takeaways & Limitations
The architecture demonstrates knowledge transfer between tasks, including transfer between related tasks without additional learning, and supports multiple reusable skills.
Takeaways & Limitations
Future work includes learning skills online, refining previously learned skills online, and training agents in real-world Minecraft domains.
Abstract
from arXiv · showhide
We propose a lifelong learning system that has the ability to reuse and transfer knowledge from one task to another while efficiently retaining the previously learned knowledge-base. Knowledge is transferred by learning reusable skills to solve tasks in Minecraft, a popular video game which is an unsolved and high-dimensional lifelong learning problem. These reusable skills, which we refer to as Deep Skill Networks, are then incorporated into our novel Hierarchical Deep Reinforcement Learning Network (H-DRLN) architecture using two techniques: (1) a deep skill array and (2) skill distillation, our novel variation of policy distillation (Rusu et. al. 2015) for learning skills. Skill distillation enables the HDRLN to efficiently retain knowledge and therefore scale in lifelong learning, by accumulating knowledge and encapsulating multiple reusable skills into a single distilled network. The H-DRLN exhibits superior performance and lower learning sample complexity compared to the regular Deep Q Network (Mnih et. al. 2015) in sub-domains of Minecraft.
Introduction
Lifelong learning requires agents to retain knowledge, selectively transfer it to new tasks, and coordinate both processes efficiently. Minecraft provides a challenging high-dimensional setting where reusable skills and hierarchical reinforcement learning address task decomposition, transfer, and retention.
- Lifelong Learning: Lifelong learning systems continually learn tasks while efficiently retaining knowledge and selectively transferring it to new tasks.The system must also ensure effective interaction between retention and transfer.
- Minecraft as a Testbed: Minecraft is a high-dimensional lifelong learning domain where increasing state and action spaces make new tasks difficult to model and solve.The game combines open-ended tasks, potentially infinite time horizons, and non-trivial knowledge reuse.
- Skill Hierarchies: Minecraft tasks can be decomposed into sub-problems whose learned skills can be reused when similar sub-problems recur.Building a wooden house, for example, can be represented as a hierarchy of subtasks such as chopping trees and assembling boards.
- Research Challenge: Efficient skill learning and reuse are non-trivial but important for knowledge retention, transfer, exploration, and task solving in Minecraft.The framework must learn skills, learn when to use them, and accumulate reusable skills as their number grows.
- Reinforcement Learning Background: Deep Q-Networks provide nonlinear function approximation for learning complex policies or skills directly from raw image pixels.The paper uses this capability as background for learning skills in Minecraft.
- Proposed Approach: The H-DRLN learns reusable reinforcement-learning skills as Deep Skill Networks and retains them through a DSN array or multi-skill distillation network.Skill distillation encapsulates multiple reusable skills into a single distilled network.
Previous Research on Lifelong Learning in RL
Prior lifelong-learning work in reinforcement learning addressed individual elements of a general system rather than the full combination of retention, selective transfer, and interaction. This work combines reusable-skill transfer with scalable knowledge retention and reports reduced sample complexity in Minecraft.
- Prior Work: Previous reinforcement-learning work on lifelong learning focused on specific elements of a general lifelong-learning system.The paper positions its approach as incorporating the basic building blocks needed for a broader lifelong-learning agent.
- Retention and Transfer: Existing approaches commonly retained knowledge through shared representations, distillation, or latent bases, while transferring knowledge through learned weights.These approaches did not by themselves establish the full retention-and-transfer framework described here.
- This Work: The proposed system transfers reinforcement-learning skills from previous tasks to new target tasks and provides a mechanism for retaining multiple skills that scales with skill count.The authors report this combination in a complex Minecraft environment.
Background
Reinforcement learning models decision-making through policies and value functions, while skills extend actions over time and support hierarchical planning. Lifelong learning applies these ideas to continually retain and transfer knowledge across tasks.
- Reinforcement Learning: An RL policy maps states to probability distributions over actions, and the agent seeks to maximize expected return.
- Deep Q Networks: DQN approximates the optimal action-value function with a convolutional neural network by minimizing temporal-difference error.
- Deep Q Networks: Experience Replay stores agent experiences and samples them randomly for minibatch updates to train the DQN parameters.
- Skills and SMDPs: A skill is a temporally extended control structure defined by initiation states, an intra-skill policy, and termination probabilities.
- Skills and SMDPs: In an SMDP, a skill policy selects skills from states, and the action-value function represents their long-term value.
- Policy Distillation: Policy distillation transfers knowledge from teacher models to a student model, including multiple teachers through shared network layers and task-specific outputs.
Hierarchical Deep RL Network
The H-DRLN extends DQN with a Deep Skill Module that lets a controller choose between primitive actions and reusable skills. Its skill-aware objective and replay mechanism support hierarchical learning, including an SMDP cost function and multi-step skill transitions.
- Architecture: The H-DRLN extends DQN with a Deep Skill Module that incorporates skills through either a DSN array or a distilled multi-skill network.
- Architecture: Deep Skill Networks are pre-trained on subtasks and, given a state and skill index, output an action according to the selected skill policy.
- Architecture: The controller learns when to execute primitive actions for one timestep and when to reuse skills until their policies terminate.
- Skill Objective Function: The H-DRLN minimizes a Skill Bellman equation rather than the standard Bellman equation to learn control over primitive actions and skills.
- Skill Objective Function: This work incorporates an SMDP cost function into a deep reinforcement learning setting.
- Skill Experience Replay: Skill Experience Replay stores discounted rewards accumulated during skill execution and transitions to the state reached after k timesteps.
Experiments
The experiments evaluate reusable Deep Skill Networks and the H-DRLN on increasingly complex Minecraft domains, including two-room and multi-task settings. Results show knowledge transfer, multi-skill reuse, and skill distillation improve task solving and learning efficiency relative to DQN baselines.
- Training DSNs: The agent trained DSNs on two navigation domains plus pickup, break, and placement sub-domains of Minecraft.These domains vary in challenges including visual ambiguity, occluded exits, navigation, and terminal primitive actions.
- Single-skill transfer: The H-DRLN reused a navigation DSN to solve the two-room domain, whose first-room exit differed from the DSN’s training exit.The available action set combined primitive movement actions with the Navigate 1 DSN.
- Single-skill transfer: 50% success was achieved by vanilla DQN after 39 epochs, compared with 76% for H-DRLN using the reused DSN.The authors attribute the difference to transferred knowledge and temporally extended actions that increase exploration.
- Knowledge transfer without learning: 67.65% success was achieved by the untrained navigation DSN on the two-room domain, versus 73.08% for H-DRLN START and 76% for H-DRLN END.The DSN was evaluated without additional learning on the two-room domain and still outperformed the vanilla DQN’s 50%.
- Multi-skill distillation: Four pre-trained DSNs were incorporated into the complex domain using either a DSN array or multi-skill distillation into one student network.The distilled network used Navigate 2, Pickup, Break, and Placement DSNs as teachers.
- Complex-domain training: 94 ± 4% success was obtained by H-DRLN with DDQN and a distilled multi-skill network, compared with 91 ± 4% for the DSN array and an unsolved DDQN baseline.The results average final-10-epoch success across five trials; the distilled H-DRLN had the highest average success rate.
- Skill usage: Skills were used approximately 20% of the time by the final H-DRLN policy while still accelerating learning in the complex domain.Skill usage later declined as the agent refined its policy with primitive actions because the transferred skills were sub-optimal under changed domain conditions.
Discussion
The H-DRLN is presented as a building block for general lifelong learning, combining skill-based transfer with multi-skill knowledge retention. The paper also reports faster, stronger learning than DDQN and identifies online skill learning and refinement as future directions.
- H-DRLN combines multi-skill distillation, temporal abstractions, and a controller to support knowledge retention and selective transfer.
- The system learns DSNs in Minecraft, reuses them on new tasks, and incorporates multiple skills through a DSN array and distilled multi-skill network.
- H-DRLN provides superior learning performance and faster convergence than DDQN by making use of skills.
- The authors frame the approach as curriculum learning by training on simpler subtasks before solving a composite task.
- Future work includes learning skills online, refining previously learned skills, and training agents in real-world Minecraft domains.