PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft

Stephan Alaniz

arXiv:1803.08456v1cs.AIcs.LGstat.ML

TL;DR

Visual-input reinforcement learning often relies on model-free methods, motivating whether planning can work with a learned model under partial observability. The paper learns a DNN transition model that predicts frames and one-step-ahead rewards, then uses MCTS to plan actions in Minecraft. The resulting agent reaches performance comparable to DQN while learning faster and using fewer training samples.

  • Problem

    The paper examines whether planning can be applied to a learned environment model for a partially observable Minecraft block-placing task.

    Source: §1 · p. 2

  • Method

    A DNN transition model predicts the next frame and one-step-ahead rewards from four recent first-person frames and an action, while MCTS plans with those predictions.

    Source: §3 · p. 3, §4 · p. 3

  • Results

    The model-based agent reaches performance comparable to DQN and learns faster with greater training sample efficiency.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The approach is interesting when obtaining training samples from the environment is costly, although each action takes longer than with DQN.

    Source: §6 · p. 5

Abstract

from arXiv · show

Deep reinforcement learning has been successfully applied to several visual-input tasks using model-free methods. In this paper, we propose a model-based approach that combines learning a DNN-based transition model with Monte Carlo tree search to solve a block-placing task in Minecraft. Our learned transition model predicts the next frame and the rewards one step ahead given the last four frames of the agent's first-person-view image and the current action. Then a Monte Carlo tree search algorithm uses this model to plan the best sequence of actions for the agent to perform. On the proposed task in Minecraft, our model-based approach reaches the performance comparable to the Deep Q-Network's, but learns faster and, thus, is more training sample efficient.

1 Introduction

The paper asks whether planning can operate on a learned model under partial observability in Minecraft. It combines model learning with MCTS and targets performance competitive with DQN.

  • Model-free visual-task methods perform well, but model-based methods are generally considered more statistically efficient because they use environment information directly.Source: §1 · p. 2
  • Prior work showed model-based agents could surpass DQN with true-state modeling and proposed transition models based on video-frame prediction.Source: §1 · p. 2
  • The paper investigates whether planning is feasible with a learned model in a partially observable Minecraft building task.Source: §1 · p. 2
  • The proposed approach predicts future visual frames and possible rewards, combining deep-neural-network model learning with Monte Carlo tree search.Source: §1 · p. 2
  • On Minecraft block placement, the model-based approach achieves results competitive with DQN.Source: §1 · p. 2

2 Block-Placing Task in Minecraft

The evaluation uses a Malmo-based Minecraft block-placing task with visual observations, sparse placement rewards, and partial observability caused by the environment and placed blocks.

  • The task is implemented in Minecraft through the Malmo framework to evaluate the proposed method against DQN.Source: §2 · p. 2
  • The agent must cover colored tiles on a 5×5 field within 30 actions, using movement, turning, sideways movement, and block placement actions.Source: §2 · p. 2
  • Correct block placement gives +1, incorrect placement gives -1, and every action incurs a -0.04 penalty to encourage faster completion.Source: §2 · p. 2
  • The agent receives a grayscaled and downsampled 64×64 first-person-view image of the current state.Source: §2 · p. 2
  • Although the action and state spaces are deterministic and discrete, partial observability and obstructed pathways make systematic placement challenging.Source: §2 · p. 3

3 Model Learning

The transition model predicts future frames and rewards from recent visual history and an action, using a shared deep convolutional representation with separate prediction outputs.

  • The model takes the last four frames and the current action, predicts the next frame, and predicts rewards one step ahead for subsequent actions.Source: §3 · p. 3
  • One-step-ahead reward prediction makes search more efficient by avoiding extra simulation of rewards from neighboring transitions.Source: §3 · p. 3
  • A noop action predicts the current-state reward because the model otherwise cannot predict the reward following the first state.Source: §3 · p. 3
  • The network encodes four 64×64 frames through convolutional layers, combines the representation with a one-hot action, and uses separate deconvolutional and fully connected heads for frame and reward prediction.Source: §3 · p. 3
  • Experience replay, batch size 32, RMSProp, and MSE losses train both outputs, while reward gradients are backpropagated only to the shared embedding.Source: §3 · p. 3

4 Monte Carlo Tree Search

MCTS uses the learned transition model to roll out future states and rewards, selecting actions through UCT while carrying search information across decisions.

  • MCTS repeatedly feeds predicted frames back into the learned model to roll out trajectories and estimate future rewards, including neighboring actions.Source: §4 · p. 3
  • UCT selects actions using maximum discounted future reward, visit counts, parent visits, and an exploration parameter k.Source: §4 · p. 4
  • Unvisited states are preferred, with immediate reward breaking ties, and improved path values are propagated back to the root.Source: §4 · p. 4
  • Each action decision performs a fixed number of MCTS rollouts, then updates the search root with the environment’s new ground-truth frame.Source: §4 · p. 4
  • Search information is retained between decisions by initializing later UCT values from the previous turn’s maximum future reward.Source: §4 · p. 4

5 Experiments and Results

The model-based MCTS agent reaches performance comparable to DQN while learning substantially faster from fewer training samples. Its advantage depends on accurate short-horizon transition and reward predictions, which also reduce the impact of rollout error.

5 Experiments and Results

  • Both agents level out at roughly the same scores, while DQN overestimates Q-values during its first 1.7 million training steps.Reducing DQN’s learning rate weakens the overestimation but slows training.Source: §5 · p. 4
  • Ten-step rollouts and structured Minecraft frames limit accumulated prediction error, while a 0.95 discount rate reduces variance in future reward predictions.The discount rate was selected empirically for the best results.Source: §5 · p. 4
  • 74% versus 72% success rate: DQN ultimately beats MCTS by only a small margin, while Table 1 reports the principal comparison.The table compares smoothed scores after 1, 2.5, and 5 million training steps.Source: Table 1 · p. 5, §5 · p. 5
  • After 1 million training steps, MCTS solves many tasks before DQN learns a reasonable policy, demonstrating greater training-data efficiency.DQN requires 1.5 million additional training steps to catch up with MCTS.Source: §5 · p. 5
  • One-step-ahead reward prediction doubles MCTS scores across training and makes DQN-comparable performance achievable with only 100 trajectories.Without predicted rewards, tree search selects random actions in unexplored future states instead of greedily maximizing immediate reward.Source: §5 · p. 5

6 Conclusion

The paper combines a learned transition model with MCTS to obtain competitive model-based reinforcement learning performance on Minecraft block placement. The approach uses fewer training samples but takes longer per action and is limited by the information contained in the last four frames.

6 Conclusion

  • The model-based agent achieves comparable scores to DQN while requiring considerably less training data to learn a meaningful transition model.Source: §6 · p. 5
  • The approach combines deep-neural-network transition-model learning with Monte Carlo tree search to select actions.Source: §6 · p. 5
  • The method takes longer to perform each action than DQN because it searches a tree before acting.This makes it most relevant when obtaining environment training samples is costly.Source: §6 · p. 5
  • Using only the last four input frames can cause incorrect predictions when relevant past information is no longer visible, leading to suboptimal actions.The paper identifies recurrent neural networks as a possible direction for addressing this issue.Source: §6 · p. 5
Loading 1803.08456v1…