PaperlayerView exact revision on arXiv

Source-linked AI summary

Value Prediction Network

Junhyuk Oh, Satinder Singh, Honglak Lee

arXiv:1707.03497v2cs.AIcs.LG

TL;DR

Large or stochastic observation spaces make accurate future-observation models difficult, raising the question of whether planning can predict rewards and values directly. VPN combines abstract-state dynamics with value learning and, in the reported evaluations, outperforms observation-based and DQN baselines in several settings, including Atari with short lookahead.

  • Problem

    Accurate observation-prediction models are difficult in large or stochastic observation spaces, motivating planning that predicts future rewards and values without future observations.

  • Method

    VPN integrates model-based abstract-state dynamics with model-free value learning, using TD search and n-step Q-learning for planning and training.

  • Results

    VPN outperforms model-free DQN baselines in multiple domains and traditional observation-based planning in a stochastic domain, including DQN on 7 of 9 Atari games.

  • Takeaways & Limitations

    Value-prediction models can support planning without future-observation generation and may help learn better abstract-state representations.

  • Takeaways & Limitations

    The Atari evaluation uses a frame-skip setting that can make training harder in games requiring finer-grained control.

Abstract

from arXiv · show

This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.

1 Introduction

The paper asks whether planning can avoid predicting future observations, and introduces VPN to predict future rewards and values directly. VPN combines abstract-state dynamics with value learning, outperforming DQN in multiple evaluated settings.

  • Motivation: Accurate observation-prediction models are difficult to build in large or stochastic observation spaces, motivating planning without future-observation prediction.The challenge is especially pronounced for high-dimensional pixel observations and stochastic environments.
  • Motivation: VPN directly predicts future rewards and values without generating future observations.The approach targets information needed for planning while avoiding unnecessary observation details.
  • Approach: VPN unifies model-based and model-free RL by learning abstract-state dynamics while mapping abstract states to rewards and values.Training combines TD search and n-step Q-learning, with supervised learning for reward predictions and Q-learning for value predictions.
  • Results: VPN outperforms model-free baselines on a 2D navigation task and is more robust to stochasticity than an observation-prediction model.The introduction reports these empirical advantages before detailing the individual experiments.
  • Results: VPN outperforms DQN on several Atari games even with short-lookahead planning.The result suggests potential for learning better abstract-state representations and reducing sample-complexity.

2 Related Work

VPN differs from prior model-based, value-iteration, Predictron, and model-free methods by learning abstract-state dynamics for future rewards and values. Its architecture combines lookahead planning with value learning rather than relying on observation prediction or a fully enumerated state space.

  • Model-based reinforcement learning: Unlike Dyna-Q and related model-based methods, VPN uses lookahead tree search to choose actions and compute bootstrapped targets.Dyna-Q instead uses a learned observation-prediction model to generate samples for Q-learning.
  • Observation-prediction models: VPN learns abstract-state dynamics to predict future rewards and values, whereas related observation-prediction models predict future observations.This distinction applies whether observation models are used for planning or exploration.
  • Value Iteration Networks: Unlike VINs, VPN does not require a small vectorized state space with local transition dynamics.VINs perform value iteration over the entire state space, imposing those representational and topological requirements.
  • Predictron: VPN is a grounded version of Predictron in which each rollout corresponds to an environment transition, and it supports optimal-policy learning in controlled settings.Predictron rollouts are purely abstract and its setting is limited to policy evaluation.
  • Model-free deep reinforcement learning: VPN extends model-free value learning with auxiliary predictions, including reward and discount, alongside its main RL values.This places VPN among architectures that use auxiliary tasks to improve learned state representations.
  • VPN architecture: Figure 1 depicts one-step prediction of reward, discount, and next-state value followed by abstract-state unrolling for multi-step rollouts.The figure summarizes VPN’s transition from core predictions to planning over abstract states.

3 Value Prediction Network

VPN unifies abstract-state modeling, value estimation, and option-conditional planning in one architecture. It recursively simulates future options, aggregates predicted returns, and trains value, reward, and discount predictions jointly.

  • 3.1 Architecture: VPN maps observations to learned abstract states, estimates their values, predicts option outcomes, and transforms them conditionally on options.The abstract state is not required to correspond to an environment state; it supports reward, discount, and value prediction.
  • 3.1 Architecture: The core module takes an abstract state and option, then predicts reward, discount, the next abstract state, and its value, yielding Qθ(s, o) = r + γVθ(s′).The core module can be recursively applied to sequences of options for multi-step prediction.
  • 3.2 Planning: VPN planning recursively expands option rollouts to depth d and backs up estimates by averaging direct and planned values along the best option sequence.The method simulates only b-best options at each expansion step to reduce computational cost, and the resulting average represents d expected returns.
  • 3.3 Learning: Training combines value-based reinforcement learning with supervised reward and discount prediction over multi-step trajectories.The paper uses a modification of n-step Q-learning and TD search, accumulating losses from 1-step through k-step predictions.
  • 3.4 Relationship to Existing Approaches: Planning both selects options and computes bootstrapped targets, so improving future predictions can improve evaluation planning and target accuracy during training.The paper connects this interaction to potentially faster convergence than conventional Q-learning.

4 Experiments

The experiments evaluate VPN against model-free and observation-prediction baselines on navigation, generalization, planning depth, and Atari tasks. VPN benefits from lookahead over abstract states, remains robust under stochasticity, and outperforms DQN across several Atari games.

  • Experimental Questions: The experiments compare VPN with DQN, VPN(1), OPN(d), and greedy or shortest-path baselines across navigation and Atari settings.The study examines model-free performance, observation-based planning, and applicability to high-dimensional visual environments.
  • Collect Domain: VPN(5) found the optimal trajectory in the Collect domain, collecting 6 goals in 20 steps compared with DQN’s 5 goals.The task randomly places the agent, goals, and walls, and includes a time limit, goal rewards, and step penalties.
  • Collect Domain: VPN outperformed DQN and VPN(1) by a large margin, while deeper planning improved performance and VPN(5) performed best in deterministic and stochastic environments.The authors attribute the advantage over model-free baselines to simulating future abstract states for combinatorial route search.
  • Collect Domain: VPN learned faster than OPN in most deterministic cases and significantly outperformed OPN in the stochastic Collect domain.VPN predicts minimal planning-relevant information—rewards, discounts, and future abstract-state values—instead of future observations.
  • Generalization: VPN generalized more robustly than DQN and VPN(1) to unseen environments, matched OPN in deterministic settings, and significantly outperformed OPN under stochasticity.The unseen environments had either fewer goals or more walls.
  • Atari Games: VPN outperformed DQN on 7 of 9 Atari games and learned significantly faster on Seaquest, QBert, Krull, and Crazy Climber.The results used short-lookahead planning and did not require predicting future observations.

5 Conclusion

VPN integrates planning and learning by modeling abstract-state dynamics for option-conditional predictions of future rewards, discounts, and values rather than observations.

  • VPN integrates planning and learning while learning abstract-state dynamics for option-conditional predictions of future rewards, discounts, and values.
  • VPN outperforms model-free DQN baselines in multiple domains.
  • VPN outperforms traditional observation-based planning in a stochastic domain.
  • Automatically learning options that support good planning is identified as a future direction.

A Comparison between VPN and DQN in the Deterministic Collect

In deterministic Collect trajectories, VPN avoids some non-optimal choices made by DQN and uses 10 option-step planning from the initial state.

  • DQN sometimes chooses a non-optimal option and collects fewer goals than VPN.
  • VPN’s last-column visualization shows 10 option-step planning from the initial state.
  • VPN’s initial plans do not always match its actual trajectories.

B Comparison between VPN and OPN in the Stochastic Collect

On stochastic Collect trajectories from the same initial state, VPN chooses a different option than OPN and collects more goals in the illustrated case.

  • At t=6, VPN moves up to collect nearby goals, while OPN moves left toward other goals.
  • OPN collects two fewer goals than VPN in the illustrated trajectory.
  • Because goals move randomly and options have stochastic outcomes, planning should consider many possible futures.

C Examples of Planning on Atari Games

The Atari examples show VPN assigning different value estimates to action sequences, while its training algorithm combines multi-step prediction, planning, and asynchronous Q-learning updates.

  • Examples of Planning on Atari Games: VPN’s Atari value estimates vary across sequences of movement, No-operation, and Fire actions.
  • Examples of Planning on Atari Games: In Seaquest, VPN estimates higher values for moving up to refill oxygen and lower values for moving down to kill enemies.
  • Training Algorithm: Additional random-policy data slightly improves outcome-module training by reducing bias toward the agent’s behavior.
  • Training Algorithm: Algorithm 2 combines asynchronous n-step Q-learning with k-step prediction and d-step planning.
  • Training Algorithm: Training executes options, samples transitions from replay memory, performs asynchronous parameter updates, and periodically updates the target network.

E Details of Hyperparameters

The Collect-domain transition module uses an option-dependent convolution and a learned mask to model changes between consecutive abstract states.

  • Option-dependent convolutional weights and a sigmoid-generated mask transform the abstract state while preserving a residual connection from s to s′.The mask multiplies the third convolutional output, so the module learns changes between consecutive abstract states.

E.1 Collect

The Collect-domain experiments compare VPN with DQN and OPN using closely related architectures, alongside specified discounting, target-network, and exploration settings.

  • Architecture: VPN’s encoding module uses three convolutional layers, while its transition module combines option-conditioned and standard convolutions with a separate mask pathway.The mask is multiplied with the third transition-layer output.
  • Baselines: DQN shares VPN’s encoding, transition, and value modules but omits the outcome module and uses 256 hidden units for parameter matching.This DQN architecture outperforms the original DQN on Collect and several Atari games.
  • Baselines: OPN matches VPN’s model-network architecture but adds a decoding module that predicts future observations from predicted abstract states.Its value network matches the DQN baseline.
  • Training: The discount factor was 0.98, target networks synchronized every 10K steps, and epsilon decreased from 1 to 0.05 over 1M steps.

E.2 Atari Games

The Atari configuration uses convolutional encoding and option-conditioned transition layers, with separate outcome and value modules; DQN shares the first components and uses a wider value module.

  • Architecture: The Atari VPN uses two convolutional encoding layers, a masked residual transition module, and separate outcome and value modules.The outcome module ends with a scalar output, while the value module consists of two fully connected layers.
  • Baseline: The DQN baseline shares VPN’s encoding and transition modules and uses 256 hidden units in its value module to approximately match parameter counts.
Loading 1707.03497v2…