PaperlayerView exact revision on arXiv

Source-linked AI summary

Control of Memory, Active Perception, and Action in Minecraft

Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, Honglak Lee

arXiv:1605.09128v1cs.AIcs.CVcs.LG

TL;DR

The paper addresses RL tasks requiring memory, active perception, and control under partial observability, delayed rewards, and high-dimensional visual observations. It introduces Minecraft tasks and memory-based DRL architectures with context-dependent retrieval, then finds that the proposed architectures—particularly FRMQN—outperform baselines and generalize better to unseen environments. The paper limits its “cognitive tasks” framing to a very restricted exploration of human cognitive faculties.

  • Problem

    Existing DRL architectures have limited demonstrated generalization on RL tasks combining partial observability, delayed rewards, high-dimensional visual observations, and active perception.

  • Method

    The paper introduces Minecraft RL tasks and memory-based DRL architectures that retain recent observations and retrieve relevant memories using temporal context.

  • Results

    The proposed architectures, particularly FRMQN, outperform baseline architectures on most tasks and generalize better from training maps to unseen environments.

  • Takeaways & Limitations

    Context-dependent memory retrieval, especially with feedback from retrieved memory, can more effectively solve tasks requiring active perception and physical movement while improving generalization to unseen environments.

  • Takeaways & Limitations

    The paper’s “cognitive tasks” represent at best a very limited exploration of the range of cognitive faculties in humans.

Abstract

from arXiv · show

In this paper, we introduce a new set of reinforcement learning (RL) tasks in Minecraft (a flexible 3D world). We then use these tasks to systematically compare and contrast existing deep reinforcement learning (DRL) architectures with our new memory-based DRL architectures. These tasks are designed to emphasize, in a controllable manner, issues that pose challenges for RL methods including partial observability (due to first-person visual observations), delayed rewards, high-dimensional visual observations, and the need to use active perception in a correct manner so as to perform well in the tasks. While these tasks are conceptually simple to describe, by virtue of having all of these challenges simultaneously they are difficult for current DRL architectures. Additionally, we evaluate the generalization performance of the architectures on environments not used during training. The experimental results show that our new architectures generalize to unseen environments better than existing DRL architectures.

1. Introduction

The paper introduces Minecraft RL tasks that combine partial observability, delayed rewards, high-dimensional vision, and active perception, then evaluates architecture performance and generalization. New memory-based architectures outperform existing ones on most tasks and generalize better to unseen maps.

  • Motivation: Figure 1 requires using an earlier yellow indicator to choose the red goal instead of the visibly observed blue goal.The map and agent location visualization is unavailable to the agent.
  • Motivation: The Minecraft tasks combine first-person partial observability, delayed rewards, high-dimensional visual observations, and active perception.The agent must learn how memory interacts with active perception to collect useful observations.
  • Scope: The paper calls these tasks cognitive tasks while acknowledging that they explore only a very limited range of human cognitive faculties.This is an explicit scope limitation of the terminology and task design.
  • Evaluation: The paper evaluates neural-network architectures on training maps and tests their generalization to unseen or larger Minecraft maps.Existing architectures perform reasonably on training maps but worse on unseen or larger maps.
  • Contribution: The proposed memory-based architectures retrieve recent observations using temporal context, unlike existing RL architectures whose memory retrieval is not context-conditioned.The paper reports that these architectures outperform existing ones on most tasks and generalize better to unseen maps.

2. Related Work

Related work spans memory architectures, deep RL, and model-free RL for partially observable environments. This paper differs by combining context-dependent memory retrieval with delayed-reward RL tasks involving high-dimensional visual observations and active perception.

  • Memory Architectures: Prior memory architectures include Neural Turing Machines, RL-NTM, neural stacks, and Memory Networks, largely developed for algorithmic or supervised-learning problems.These systems use external or structured memory for tasks such as copying, reversing, pattern inference, and question answering.
  • Deep Reinforcement Learning: Deep RL work established end-to-end visual control with DQN and extended exploration and learning to Atari and 3D environments similar to Minecraft.The cited work includes convolutional control networks, Monte Carlo Tree Search data generation, and asynchronous DQN.
  • Partially Observable RL: Model-free RL for POMDPs uses recurrent or continuous memory states to summarize history for action selection under partial observability.DRQN places an LSTM on top of a DQN-based CNN and improves handling of partial observability in Atari games.
  • Departure from Related Work: The introduced architectures construct memory-retrieval queries from temporal context and use simpler writing and addressing mechanisms than Neural Turing Machines.Unlike much prior memory research, they operate in RL settings with delayed rewards rather than direct supervised error signals.
  • Task Differences: The Minecraft mazes extend earlier T-maze and MazeBase tasks with high-dimensional visual observations, deep partial observability, and learned active perception.The agent must control perception to collect useful information at the right time.

3. Background: Deep Q-Learning

The DQN framework stores transitions in replay memory and trains a neural network to approximate action values by minimizing a loss based on target Q-values. A separate target network is periodically synchronized with the learned network.

  • DQN Framework: At time t, the DQN framework represents a transition as T_t = (s_t, s_t+1, a_t, r_t) and stores it in replay memory.The state, immediate reward, and action are denoted s_t, r_t, and a_t.
  • Optimization: The network parameters θ are trained by minimizing a loss whose gradient compares the target y_i with the predicted action value Q(s, a; θ).The gradient is expressed as an expectation over states and actions sampled according to π_θ.
  • Target Value: The target Q-value is y_i = E_s′∼πθ[r + γ max_a′ Q(s′, a′; θ′)], using a target network with parameters θ′.This target bootstraps from the immediate reward and the maximum next-state action value.
  • Training Procedure: In practice, the expectation terms are approximated by sampling mini-batches of transitions from replay memory.The target network parameters θ′ are synchronized with θ after a fixed number of iterations.

4. Architectures

The paper introduces memory-based Q-network architectures that retrieve observations using context, extending visual encoding with external memory and temporal processing for RL.

  • Motivation: Context-dependent memory retrieval is motivated by cases where prior observations matter only at particular decision points.The indicator matters when the agent encounters a potential target and must choose whether to approach it.
  • Architecture overview: The proposed architectures combine CNN image features, recent-observation memory, and a context vector for retrieval and action-value estimation.The three variants are MQN, RMQN, and FRMQN.
  • Encoding: Raw pixel observations are encoded by a CNN into fixed-length feature vectors before entering the memory system.The input is an h × w image with c channels, and the encoded feature is the representation at time t.
  • Memory: The memory stores linearly transformed features from the last M observations as separate key and value blocks.Keys support retrieval, while values provide the content returned by the read operation.
  • Memory: Soft attention retrieves memory by comparing a context vector with key blocks and weighting their corresponding value blocks.The attention weights index memory locations and implicitly identify prior time steps.
  • Context: MQN uses the current observation for context, RMQN adds LSTM history, and FRMQN feeds retrieved memory back to refine context over time.FRMQN's feedback enables repeated temporal retrieval, analogous to multiple hops but operating through time.

5. Experiments

The experiments compare baseline and memory-based DRL architectures across Minecraft tasks emphasizing partial observability, delayed rewards, visual comparison, and generalization to unseen maps. Memory-based architectures generally learn more effectively and generalize better, with recurrent and feedback mechanisms especially useful for long-term dependencies and complex pattern matching.

  • Experimental setup: The experiments evaluate DQN, DRQN, MQN, RMQN, and FRMQN on Minecraft tasks using randomized episodes, visual inputs, and unseen-map generalization tests.Episodes terminate after task completion or 50 steps, with six visual and movement actions and randomized initial directions or instances where applicable.
  • 5.1. I-Maze: Description and Results: In I-Maze, the memory-based architectures converge more quickly than DQN and DRQN by retaining indicator information until the goal decision.The indicator matters mainly at the corridor’s end, creating a long delay between observation and action selection.
  • 5.1. I-Maze: Description and Results: FRMQN learns a general I-Maze strategy that retrieves the indicator at the corridor’s end, while MQN and RMQN attend less sharply and visit wrong goals more often in larger mazes.The result links feedback-based retrieval with more precise use of temporally distant observations.
  • 5.2. Pattern Matching: Description and Results: In pattern matching, MQN and FRMQN consistently learn correct training-set choices, whereas DRQN learns a sub-optimal policy and DQN and RMQN are less stable.The task requires comparing two visual patterns observed at different time steps.
  • 5.2. Pattern Matching: Description and Results: FRMQN achieves the highest unseen-map success rate in pattern matching, while MQN fails to generalize because convex-combined memories hinder direct similarity comparison.FRMQN’s recurrent connections allow retrieved patterns to be compared over time through repeatedly refined context.
  • 5.3. Random Mazes: Description and Results: Across random-maze tasks, all architectures worsen on unseen maps, but RMQN and FRMQN generalize better in most tasks, while DRQN degrades significantly.The learning curves show that improving performance on fixed training maps does not necessarily improve performance in new environments.

6. Discussion

The paper evaluates cognition-inspired Minecraft tasks on disjoint map sets and finds that memory-based architectures, especially FRQMN, generalize better to unseen environments.

  • Three classes of cognition-inspired Minecraft tasks were compared across two existing and three proposed neural architectures.
  • Training and evaluation used disjoint map sets to test whether learned value functions apply to unseen maps through interpolation and extrapolation.
  • Context-dependent memory retrieval, particularly with feedback from retrieved memory, more effectively solved tasks requiring active perception and external physical movement.
  • FRQMN showed superior ability to generalize value-function behavior from training environments to unseen environments relative to baseline architectures.

A.1. Hyperparameters

The experiments used shared visual-processing and reinforcement-learning settings, with replay-memory sizes and exploration schedules varying by task family.

  • Each architecture used convolutional layers with 32 and 64 filters, both using 4 × 4 kernels, stride 2, and padding 1.
  • Deep Q-Learning used batch size 32 and discount factor 0.99.
  • Replay memory held 10^6 entries for random mazes and 5 × 10^4 entries for I-Maze and Pattern Matching tasks.
  • Epsilon was linearly interpolated from 1 to 0.1 over the initial 10^6 steps of the epsilon-greedy policy.

A.2. Map Generation for Pattern Matching

Pattern Matching maps were generated from paired room patterns, while learning curves and trajectory figures illustrate performance across training, unseen, and larger maze environments.

  • Map Generation for Pattern Matching: The Pattern Matching generator sampled 250 of 512 possible visual patterns and created paired maps with identical or different room patterns.This produced 500 training maps: 250 with identical rooms and 250 with different rooms.
  • Map Generation for Pattern Matching: Learning curves plot average reward against training epochs, distinguishing unseen maps by size, visual pattern, or topology depending on the task.One epoch corresponds to 10K steps.
  • Map Generation for Pattern Matching: FRMQN successfully completed an unseen, larger I-Maze by visiting the red block indicated by the yellow block.
  • Map Generation for Pattern Matching: In Pattern Matching, FRMQN inspected both rooms, used backward actions along the corridor, and selected the blue block when their visual patterns matched.The figure describes this performance as near optimal.
  • Map Generation for Pattern Matching: FRMQN selected the red goal when the two rooms had different visual patterns.
  • Map Generation for Pattern Matching: On unseen larger random mazes, FRMQN sometimes found the single goal but could also fail within 100 steps when maps were large and intrinsically complex.
  • Map Generation for Pattern Matching: For sequential goals, FRMQN used memory to avoid the second goal before visiting the first and then completed the sequence in the correct order.
  • Map Generation for Pattern Matching: Indicator tasks required FRMQN to remember the indicator color and use it to avoid an incorrect goal or select the appropriate goal sequence.
Loading 1605.09128v1…