JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

1 Introduction

• Minecraft tasks require long sequences of interdependent sub-tasks, and sparse rewards make long-horizon decisions difficult to learn efficiently.Learning from unstructured demonstrations without domain knowledge also remains challenging.Source: §1 · p. 1
• Agents must learn representations from varied, dynamic visual surroundings and explore under partial observability without wasting samples.Human demonstrations are diverse and may contain noise from imperfect operation.Source: §1 · pp. 1–2
• JueWu-MC automatically structures demonstrations into sub-goals, trains a high-level controller over options, and uses low-level workers to solve sub-tasks.The approach also proposes action-aware representation learning, discriminator-based self-imitation, and ensemble behavior cloning with consistency filtering.Source: §1 · p. 2

Discriminator-based Self-imitation Learning.

• Discriminator-based self-imitation uses self-generated experiences for self-correctable policies, while consistency filtering selects common human behavior for ensemble cloning.The proposed techniques address sample inefficiency in existing self-imitation and uncertainty from imperfect demonstrations.Source: Discriminator-based Self-imitation… · p. 2, §1 · p. 2

Game AI

• Minecraft approaches are grouped into end-to-end learning and hierarchical reinforcement learning with human demonstrations; JueWu-MC belongs to the latter.Prior examples include ForgER and SEIHAI in the hierarchical category.Source: §2 · p. 2

Sample-efficient Reinforcement Learning

• JueWu-MC structures highly unstructured crowd-played demonstrations by automatically defining sub-tasks and sub-goals.This contrasts with related HRL work that learns from human priors.Source: §2 · p. 2
• Unlike representation methods focused on general unlabeled data or similarity constraints, JueWu-MC measures action effects in 3D video games.The paper characterizes its approach as self-supervised representation learning.Source: §2 · p. 2
• Where prior exploration methods use curiosity or uncertainty, JueWu-MC proposes discriminator-based self-imitation to exploit past good experiences.The stated motivation is the exploration-exploitation dilemma under sample-efficiency constraints.Source: §2 · p. 2

3.1 Overview

• Unstructured long-horizon demonstrations are segmented by reward delay: long-delay atomic skills remain separate sub-tasks, while adjacent short-delay skills are merged, with each sub-goal set to the last state in its sub-task.The resulting structured demonstrations train the meta-policy by imitation and support sub-policy learning from demonstrations and environment interaction.Source: §3.1 · p. 3

3.2 Meta- and Sub-policies

• The meta-policy maps continuous states to discrete options, is trained by negative log-likelihood on demonstration states labeled by stage, and selects options by argmax at inference.Sub-policies handle resource gathering or crafting; because handcrafted action spaces are prohibited and actions are obfuscated in MineRL 2020–2021, each sub-task uses KMeans-clustered demonstration actions.Source: §3.2 · p. 3

Action-aware Representation Learning

• For resource gathering, action-aware representation learning uses an action-specific mask to focus visual features on local image regions relevant to predicting state changes.A two-stage process first trains an action-dependent linear projection to predict the next-state representation, then fixes it while optimizing the mask; the approach provides multi-view representations that can be combined with off-the-shelf RL algorithms.Source: Action-aware Representation Learning · p. 4

3.3 Learning Sub-policies to Gather Resources

• Discriminator-based self-imitation learning distinguishes states from successful and failed trajectories and supplies intrinsic rewards that guide exploration toward previously successful state distributions.Unlike advantage-based self-imitation, it avoids advantage clipping; its reward also encourages the policy to return to good states after entering a bad state distribution.Source: Action-aware Representation Learning · p. 4

3.4 Learning Sub-policies to Craft Items

• For crafting sub-tasks, consistency filtering retains demonstrations following the most common action sequence, and ensemble behavior cloning trains policies on different demonstration subsets and selects actions by majority vote.The approach targets noisy or diverse demonstrations and uncertainty on unseen states, with the ensemble intended to reduce uncertainty and mitigate compounding errors.Source: §3.4 · p. 5

4 Experiment

• Experiments in MineRL compare the approach with end-to-end methods and prior MineRL competition solutions, including SEIHAI and ForgER.The end-to-end baselines were trained with up to 8 million samples and default hyperparameters.Source: §4 · p. 5

4.1 Main Results

• 76.97 was 3.4x the second-place 22.97 and exceeded the 75.25 sum of MineRL 2020 competitors; training reached 100 after 2.5 million samples.The reported online score was lower than the training-curve performance (>100) after a MineRL version update; training-curve baselines shared version 0.3.7.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Ablation Study

• All three techniques contributed to performance; A2RL and EBC had earlier effects, while DSIL mainly boosted later sub-tasks such as stage 5.The authors note that EBC's contribution underscores the importance of robust policies for long-horizon tasks.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 Visualization

• A2RL masks focus on uncertain image regions, including objects ahead during attacks and regions changed by turning, making the learned policy more interpretable.The visualizations show current states, next states, and saliency maps for the learned mask.Source: §4.3 · p. 6, Figure 4 · p. 6
• After more training, PPO+DSIL visits a more compact region, while PPO and PPO+SIL continue exploring more widely.The authors attribute this pattern to DSIL encouraging agents to stay near good state distributions and reproduce past good behavior.Source: §4.3 · p. 6, Figure 5 · p. 6

• JueWu-MC combines a high-level controller with auto-extracted low-level workers to address sophisticated Minecraft tasks across environments.Source: §5 · p. 6
• Representation-learning and imitation-learning techniques are introduced to improve sub-policy performance and learning efficiency.Source: §5 · p. 6
• Experiments report that the pipeline outperforms baseline algorithms and solutions from past MineRL competitions; future work would apply it to other Minecraft tasks and open-world games.Source: §5 · p. 6

• MineRL is an annual research competition focused on developing sample-efficient RL algorithms for mining items in Minecraft.Source: Minecraft & MineRL · p. 8
• MineRL observations include 64 × 64 images and an inventory observation; the action space combines view adjustment, movement, and item or block interactions.Source: Minecraft & MineRL · p. 8
• Rewards are sparse: the first time an item on diamond’s prerequisite chain is obtained, it is rewarded.Source: Minecraft & MineRL · p. 8

• The approach uses the 60-million-sample MineRL Diamond dataset and assigns distinct learning procedures to seven stages, including demonstrations, RL, and random search.The meta-policy uses a three-layer MLP; all models are optimized with Adam.Source: Experiment Setting · p. 8

Stage 1

• Stage 1 uses SQIL with A2RL and explicitly shaped rewards, which increase as the agent approaches rewarding states.Source: Experiment Setting · p. 8

Stages 2–3

• Stage 2 trains ten EBC policies from demonstrations, while Stage 3 uses DQfD only to pre-train its sub-policy without environment interaction.Source: Experiment Setting · p. 8

Stages 4 and 6

• Stages 4 and 6 use random search; the authors report it is comparable to EBC for Stage 4 and achieves decent performance in Stage 6.Source: Experiment Setting · p. 8

Stages 5 and 7

• Stages 5 and 7 use PPO with A2RL and DSIL to learn individual policies.Source: Experiment Setting · p. 8

• Each evaluation averages scores across 100 episodes with different random seeds, and three seed sets are used to examine variance.Source: Experiment Setting · p. 8

Learning to Gather Resources

• For resource gathering, the algorithm stores successful and failed trajectories, trains the projection net, mask net, and discriminator, then updates the policy using an off-the-shelf RL algorithm.Source: Algorithm · pp. 8–9

Learning to Craft Items

• For crafting, EBC retains demonstrations matching the most common action pattern, then trains a population of policies from the filtered demonstrations without environment interaction.Source: Algorithm · pp. 8–9

Learning to Gather Resources

• The resource-gathering procedure computes intrinsic rewards before policy updates, while the feature-map parameters are optimized with RL objectives.Source: Algorithm · p. 9