Scaling Imitation Learning in Minecraft

1 Introduction

• Reinforcement learning needs billions of interaction steps for challenging results, while sparse rewards and large action spaces make exploration especially demanding in Minecraft.The paper notes this is particularly challenging in slow environments that cannot be run for billions of steps with available computational resources.Source: §1 · p. 1
• Imitation learning trains from expert demonstrations without environment interaction during training and is unhindered by reward sparsity.The paper evaluates it on complex Minecraft tasks with a large amount of demonstration data.Source: §1 · p. 1
• The paper introduces state- and action-space representations and demonstration processing, then investigates architecture, data augmentation, and loss-function effects.Source: §1 · p. 1
• An early approach placed second in the NeurIPS 2019 MineRL competition; this report presents a stronger form as a starting point for future competitions and related research.Source: §1 · p. 1
• ObtainIronPickaxe’s sequential subtasks make robust learning difficult because failure at one step prevents further progress.The scenario includes prerequisite resource-gathering and crafting steps.Source: Figure 1 · p. 2

2 Minecraft Environment

• Minecraft is a procedurally generated 3D open world observed from a first-person perspective, with resource information also available to the agent.The MineRL-v0 dataset contains over 500 hours of human trajectories and was presented as the first large-scale demonstration dataset for an image-based environment.Source: §2 · p. 2

2.1 Competition

• The NeurIPS 2019 competition targeted ObtainDiamond, allowed 8 million environment interaction steps, and imposed a single-GPU, four-day remote training limit.Custom, nonpublic textures were used for training and evaluation.Source: §2.1 · p. 2

2.2 Minecraft Tasks

• The study focuses on ObtainIronPickaxe and Treechop; the former has 11 sequential subtasks, while the latter rewards collecting 64 logs.ObtainIronPickaxe omits the final diamond-gathering step because tested methods and competition entries had not obtained a diamond; Treechop is used for comparison with reinforcement learning.Source: §2.2 · p. 2

3.1 Imitation Learning

• Behavior cloning trains a softmax policy to predict expert actions from observations using cross-entropy, with actions sampled or chosen by maximum probability.The section also evaluates DQfD pre-training, which incorporates rewards and selects actions greedily by action value.Source: §3.1 · p. 3
• The margin loss favors the expert action when its Q-value exceeds every alternative by at least margin b; experiments compare it with and without TD loss against cross-entropy.Without reward incorporation, the margin classification loss uses action features rather than Q-values.Source: §3.1 · p. 3

3.2 Training Setup

• The state combines a 64 × 64 RGB image with inventory and held-item information encoded as multi-hot and one-hot vectors, respectively.The held item uses a one-hot vector; inventory item amounts are represented by the number of ones in each multi-hot sub-vector.Source: §3.2 · p. 3
• Actions combine binary movement controls, camera yaw and pitch, and crafting, equipping, and placing actions; camera rotation is quantized in 22.5-degree increments.The resulting combinations are reduced to 112 movement actions by limiting simultaneous movement actions and removing redundancies.Source: §3.2 · pp. 3–4
• The network processes images with a convolutional encoder and vector state with a fully connected encoder, then concatenates them; four perceptual architectures are compared.The variants are DQN, Impala, Deep Impala, and Double Deep Impala; the last doubles channel size.Source: §3.2 · p. 4, Figure 2 · p. 4
• Training uses successful ObtainIronPickaxe and ObtainDiamond trajectories with no-op states removed, and can incorporate Treechop trajectories after completing their missing inventory observations.Treechop observations contain RGB images only, so inventory vectors are sampled from other task trajectories to complete them.Source: §3.2 · p. 4
• Augmentations tested include horizontal flipping with corresponding left-right action flips, rectangle removal, and brightness, sharpness, contrast, and posterization adjustments.Source: §3.2 · p. 4
• Training loss and cross-entropy test loss on unseen human trajectories are assessed against average return, but neither loss is highly correlated with performance change.The test loss increases as human policies differ across trajectories.Source: §3.2 · p. 4, Figure 3 · p. 5

4.1 Evaluation

• On ObtainIronPickaxe, test loss increased during training while reward improved, and neither training nor test loss tracked snapshot performance.Evaluation interaction could not be avoided even though training did not require environment interaction.Source: §4.1 · p. 5

4.2 Architecture and Augmentations

• Image flipping was the only effective augmentation, improving performance by 73%; additional augmentations had no significant impact and sometimes reduced performance.Source: §4.2 · p. 5

4.3 Margin Loss and Treechop Data Incorporation

• Additional Treechop data improved performance by a large margin, indicating that available trajectories could still bottleneck imitation learning despite the massive human dataset.Source: §4.3 · p. 5

4.2 Architecture and Augmentations

• Deep Impala improved performance by 78%, and Double Deep Impala yielded another 7% improvement.Table 1 compares architectures and augmentations; these architecture results were obtained with horizontal image flipping.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Margin Loss and Treechop Data Incorporation

• 82% of episodes reached the stone pickaxe with the best Deep Impala cross-entropy policy plus Treechop data; an iron pickaxe was built in about 1%.Typical failures included getting stuck in biomes without trees or being buried underground without sufficient resources.Source: §4.3 · p. 6
• Cross-entropy outperformed margin loss on both tasks when actions were sampled, but argmax reduced its performance below margin loss; adding TD loss to margin loss diminished performance.Source: §4.3 · p. 5, Figure 4 · p. 6

4.4 Comparison to Reinforcement Learning

• Figure 5 compares CE imitation, Rainbow reinforcement learning, and DQfD on Treechop against the task's maximal possible return.Source: Figure 5 · p. 7

• 74.5 average return for the report's best policy exceeded the earlier entry's 42.4, while the competition's top entry scored 61.6; evaluation used a different texture set.Source: §5 · p. 7

6 Discussion

• Imitation learning can work well with sufficient demonstrations, but losses correlate weakly with test-time policy performance and returns are unstable.Performance fluctuates considerably during training.Source: §6 · p. 7
• A poor prediction at an essential early state can cause trajectory collapse because later states where the network performs well are never reached.This risk arises in temporally correlated processes.Source: §6 · p. 7
• The authors hope their strong imitation-learning baseline will support future progress in Minecraft sensorimotor learning.Source: §6 · p. 7

A.1 Architectures

• The architecture embeds RGB images convolutionally, embeds an available state vector separately, then concatenates the embeddings for the final fully connected part.The state-vector embedding is one fully connected layer of size 128, or 256 for Double DeepImpala.Source: §A.1 · p. 9
• DQN uses the Atari DQN architecture, while Impala stacks three convolution, max-pooling, and residual-block stages with channels (32, 64, 64).The Impala channel sizes are double those of the original architecture.Source: §A.1 · p. 9
• DeepImpala replaces plain residual blocks with Fixup blocks and uses four convolution, max-pooling, and paired-Fixup stages.Its four stages have channel sizes (32, 64, 64, 64).Source: §A.1 · p. 9
• Double DeepImpala retains DeepImpala’s image embedding design but doubles its channel sizes to (64, 128, 128, 128).Source: §A.1 · p. 9
• The action head applies two fully connected layers with ReLU activations; its output count matches the actions, with softmax added for cross-entropy training.The first layer has size 1024, or 2048 for Double DeepImpala; margin-loss training uses a linear output.Source: §A.1 · p. 9

A.2 Action Space

• Camera motion is discretized into four 22.5-degree actions, selected when movement in that direction exceeds 11.25 degrees across the next three states.Small Gaussian noise is added to the 22.5-degree value during interaction to reduce the chance of getting stuck through discretization.Source: §A.2 · p. 9
• Pruning reduces 1280 discrete movement actions by removing sneak, conflicting pairs, multiple camera turns, and combinations exceeding three sub-actions.When trimming combinations above three sub-actions, actions are removed in the stated unimportant-to-important order.Source: §A.2 · p. 9
• The remaining combinations yield 112 movement actions, and crafting, equipping, and placing actions bring the total to 130.Jump is set to 1 unless attack is used.Source: §A.2 · p. 9

A.3 State Space

• The state includes a first-person RGB image and inventory information, except in Treechop, where it consists only of the image.Source: §A.3 · p. 9
• The held item is encoded as a four-way one-hot vector for none, wooden pickaxe, stone pickaxe, or iron pickaxe.Source: §A.3 · p. 9
• Inventory quantities use multi-hot vectors when demonstration-data averages are not too large, and single float values otherwise.Dirt, cobblestone, and stone amounts are represented as current quantity divided by the expert-data average; Table S1 gives multi-hot vector sizes by item.Source: §A.3 · p. 10, Table S1 · p. 10