PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Variational Reinforcement Learning for POMDPs

Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, Shimon Whiteson

arXiv:1806.02426v1cs.LGstat.ML

TL;DR

Partially observable control is difficult when observations are incomplete or noisy and the environment model is unknown. DVRL learns a generative model, performs approximate inference with an ELBO-based objective, and jointly optimizes it with the policy. It consistently outperforms an RNN-based architecture across diverse tasks, while the authors identify model and representation improvements as future directions.

  • Problem

    DVRL targets reinforcement learning in POMDPs from observation streams without known latent states or transition and observation functions.

  • Method

    DVRL learns an internal generative model, performs approximate belief inference with an AESMC-based ELBO approximation, and jointly optimizes the model with an n-step policy update.

  • Results

    DVRL consistently outperforms an RNN-based architecture on a diverse set of tasks, and ablations support particle ensembles and joint ELBO-RL optimization.

  • Takeaways & Limitations

    The results support using a learned model and belief-distribution approximation to make latent representations suitable for POMDP control.

  • Takeaways & Limitations

    The authors identify more powerful model architectures, disentangled latent states, and improved generalization as directions for further improvement.

Abstract

from arXiv · show

Many real-world sequential decision making problems are partially observable by nature, and the environment model is typically unknown. Consequently, there is great need for reinforcement learning methods that can tackle such problems given only a stream of incomplete and noisy observations. In this paper, we propose deep variational reinforcement learning (DVRL), which introduces an inductive bias that allows an agent to learn a generative model of the environment and perform inference in that model to effectively aggregate the available information. We develop an n-step approximation to the evidence lower bound (ELBO), allowing the model to be trained jointly with the policy. This ensures that the latent state representation is suitable for the control task. In experiments on Mountain Hike and flickering Atari we show that our method outperforms previous approaches relying on recurrent neural networks to encode the past.

1. Introduction

DVRL addresses POMDP control when observations are incomplete or noisy and the environment model is unknown. It learns a generative model and performs approximate belief inference, jointly shaping representations for control.

  • Motivation: POMDPs arise when occlusions or noisy sensors prevent agents from observing the environment state fully.The challenge is to learn from observation streams without prior knowledge of the latent state space or transition and observation functions.
  • Motivation: Because a single observation may omit action-relevant information, effective policies generally need to aggregate the entire action-observation history.
  • Prior approaches: Model-free RNN approaches summarize history without an explicit learned environment model, placing a heavy burden on the recurrent encoder.They may remember past features or use heuristics rather than compute belief states.
  • DVRL: DVRL learns transition and observation models, performs approximate inference to update a belief state, and uses that representation for control.Its inductive bias supports a less black-box approach without assuming the latent representation or environment functions in advance.
  • DVRL: The model is trained with an AESMC-based ELBO approximation jointly with an n-step policy-gradient update, while particle ensembles capture belief-state uncertainty.This joint optimization makes the latent representation and inference suitable for the control task rather than only unsupervised reconstruction.
  • Evaluation: Experiments cover Mountain Hike and flickering Atari, where partial observability is created through noisy measurements, single-frame inputs, and blank screens.The reported evaluation spans a low-dimensional continuous environment and complex high-dimensional games.

2. Background

The background formalizes POMDPs, belief-state inference, variational autoencoding for sequential data, and n-step A2C policy learning. These components provide the modeling and optimization foundations used by DVRL.

  • Partially Observable Markov Decision Processes: A POMDP specifies state, action, observation, transition, reward, and initial-belief components, with noisy or partially occluded observations generated from latent states.
  • Partially Observable Markov Decision Processes: Policies in POMDPs condition on observation-action histories, whose exponential growth makes direct history-based methods costly for large or continuous spaces.Suffix-tree approaches reduce this burden only for small discrete observation spaces.
  • Belief states: The belief state bt = p(st|o≤t, a<t) is a sufficient history statistic for an optimal policy, but updating it requires the transition and observation model.
  • Variational Autoencoders: A VAE learns priors and observation decoders by maximizing ELBOs that lower-bound log marginal likelihood and encourage the encoder to approximate the learned posterior.
  • VAE for Time Series: For sequential data, the generative model combines an initial distribution, transition distribution, and observation distribution, with encoder distributions conditioned on prior latent states and observations.
  • VAE for Time Series: Sequential Monte Carlo improves time-series ELBO learning by propagating weighted particles, where K denotes the number of particles.Each particle contains a weight and a latent-state value, with resampling used during the sequence.
  • A2C: A2C uses n-step on-policy rewards and bootstrapped value estimates across parallel environments to update policy and value parameters.An entropy loss is added to encourage exploration.

3. Deep Variational Reinforcement Learning

DVRL extends recurrent POMDP agents by learning an environment model, inferring a particle-based belief state, and training that representation for control. Its n-step ELBO approximation enables joint optimization of the learned model, encoder, policy, and value function.

  • 3. Deep Variational Reinforcement Learning: DVRL extends RNN history encoding by learning transition and observation models and explicitly inferring a belief state for action selection.The learned model is optimized for expected return as well as the ELBO, shaping latent representations for control rather than observation prediction alone.
  • 3.1. Baseline Architecture and Training: n-step A2C supports continuous actions and avoids drawing complete trajectories from a replay buffer, while separate rollout and backpropagation lengths control distinct trade-offs.The computation graph spans ng steps independently of the ns-step parameter-update interval.
  • 3.2. Latent Belief State: The latent belief state represents uncertainty with K particles, each containing a recurrent latent state, a stochastic latent state, and an importance weight.The particles approximate the belief state in DVRL’s learned stochastic transition and observation model.
  • 3.3. Recurrent Latent State Update: The model uses stochastic transitions, an observation decoder, and a deterministic recurrent transition, while jointly optimizing the ELBO and expected return.This combines generative-model learning with the reinforcement-learning objective.
  • 3.3. Recurrent Latent State Update: At each update, DVRL resamples particle ancestors, samples new stochastic latent states from the transition model, evaluates their observation likelihoods, and aggregates them into a new belief.A second RNN summarizes the particle tuples into a vector that conditions the actor and critic and feeds the next timestep.
  • 3.4. Loss Function: The n-step ELBO approximation makes it possible to learn model and encoder parameters jointly with the reinforcement-learning loss.The approximation uses trajectory segments rather than requiring the non-additive full-trajectory ELBO at every update.

4. Related Work

Related POMDP methods often assume known models, use model-free recurrent summaries, or fail to scale to large or continuous spaces. DVRL instead enables explicit model-based belief inference while avoiding some assumptions of specialized alternatives.

  • 4. Related Work: Much POMDP literature assumes known transition and observation functions and a known latent-state representation, although realistic domains often lack them.This motivates methods that learn these components rather than treating them as given.
  • 4. Related Work: RNN-based approaches address POMDPs across recurrent, multi-agent, text-game, and Atari settings, but DVRL extends them with explicit model and belief-state reasoning.The comparison concerns the information used by the policy, not merely the recurrent architecture.
  • 4. Related Work: QMDP-Net learns a value-iteration network end-to-end, but assumes a strong transition-function structure and requires analytical belief updates.These assumptions distinguish it from DVRL’s learned stochastic model and approximate inference approach.
  • 4. Related Work: Earlier particle-filter policy learning assumed a known model and used finite differences for gradient estimation.DVRL addresses the model-learning setting with gradient-based joint optimization.
  • 4. Related Work: Spectral, Bayesian, and nonparametric dynamics-learning approaches do not scale to large or continuous state and observation spaces.This limitation is presented as a contrast to variational approaches for such settings.
  • 4. Related Work: Unlike some auxiliary-loss methods, DVRL does not require additional environmental information and uses information already obtained from the environment.The related work comparison frames the auxiliary objective as a way to improve latent representations.

5. Experiments

DVRL is evaluated on noisy Mountain Hike and flickering Atari, where it generally outperforms recurrent encoders and its performance benefits from inference-based history aggregation, particle ensembles, ELBO training, and joint optimization.

  • Experiments: DVRL deals better with noisy or partially occluded observations across Mountain Hike and flickering Atari, including high-dimensional and continuous observation spaces.
  • 5.1. Mountain Hike: In Mountain Hike, DVRL outperforms RNN-based policies especially at higher observation noise and follows the mountain ridge more effectively.The task requires estimating current position from noisy location measurements.
  • 5.1. Mountain Hike: 20.7 is the trajectory return difference at σo = 3, compared with an average difference of 11.43 for the same noise level.
  • 5.2. Atari: DVRL significantly outperforms the RNN-based policy on five of ten stochastic, flickering Atari games and narrowly underperforms significantly on only one.The results are averaged over five random seeds.
  • 5.3. Ablation Studies: Ablations show that multiple particles, ELBO loss inclusion, and joint ELBO-RL optimization are important for performance.The ablations are conducted on flickering ChopperCommand.
  • 5.3. Ablation Studies: Increasing backpropagation length helps both policies, but ng = 5 harms RNN more than DVRL because DVRL updates its belief in one step.The result supports DVRL relying more on inference computations than memory-based reasoning.

6. Conclusion

DVRL solves POMDPs from observation streams without known latent states or environment functions by combining an ELBO-based auxiliary loss with an inference-oriented policy structure. It consistently outperforms an RNN-based architecture across diverse partially observable tasks, while ablations support particle ensembles, joint optimization, and belief-like latent states.

  • DVRL addresses POMDPs using only observations, without knowledge of the latent state space or transition and observation functions.
  • Its policy incorporates an ELBO-based auxiliary loss and an inductive bias reflecting the need for inference for optimal solutions.
  • DVRL consistently outperforms an RNN-based architecture across diverse tasks, including partially observable Atari games with stochastic transitions.
  • Ablations show the necessity of particle ensembles and jointly optimizing the ELBO and reinforcement-learning objectives.
  • Results support interpreting DVRL’s latent state as a belief distribution in a learned model.
  • Access to belief distributions and learned models opens research directions involving generalization, latent representations, architectures, disentanglement, and curiosity-driven exploration.

A.1. Implementation Details

The implementation uses neural parameterizations for latent-state transitions, proposals, and observation decoders, with task-specific encoders and policy networks. Training uses RMSProp, gradient clipping, discounting, and tuned learning rates, while source code was planned for future release.

  • Transition and proposal distributions are multivariate normal, while image decoders use independent Bernoulli distributions and real-valued-vector decoders use normal distributions.
  • Neural networks determine distribution means, diagonal variances, and decoder parameters; variance outputs use softplus for positivity.
  • Inputs are concatenated, ReLUs connect layers, Atari uses batch normalization, and recurrent baselines use GRUs.
  • Separate encoding functions process observations, actions, and latent states before downstream networks.
  • The policy outputs up to 18 Atari action probabilities or two Mountain Hike parameters, while the value function has one output.
  • A2C uses 16 parallel environments and 5-step learning, with learning rates of 2 × 10−4 for Atari DVRL and 1 × 10−4 for Mountain Hike DVRL and RNN.
  • Optimization uses RMSProp with α = 0.99, gradient clipping at 0.5, discount factor γ = 0.99, and orthogonal initialization.
  • The source code was planned for future release.

A.2. Additional Experiments and Visualisations

Additional experiments report deterministic and flickering Atari results averaged over five seeds, with learning curves covering all ten games. The stochastic flickering setting is especially challenging, and DVRL significantly outperforms the RNN policy on five of ten games while significantly underperforming on one.

  • Table 2 reports deterministic and flickering Atari results averaged over 5 random seeds, with DRQN and ADRQN values taken from their respective papers.
  • DRQN and ADRQN use Q-learning rather than A2C, so their reported results are not directly comparable.
  • Figures 6 and 7 show individual learning curves for all 10 Atari games in deterministic and stochastic versions.

A.3. Computational Speed

The appendix reports Atari training speed on one GPU, showing that throughput decreases as the number of DVRL particles increases. Reported rates are 64k FPS with one particle, 48k FPS with ten, and 32k FPS with thirty.

  • Approximate Atari training speed is measured in frames per second on one GPU on a dgx1.
  • DVRL with 1 particle runs at 64k FPS, compared with 48k FPS using 10 particles and 32k FPS using 30 particles.
  • Figures 6 and 7 provide training curves for flickering deterministic and stochastic Atari environments, respectively.

A.4. Model Predictions

DVRL reconstructs current Atari observations and predicts future observations using its learned generative model, including when the current input is blank. The model preserves some state features but struggles with information-dense scenes, while training uses recurrent history aggregation with n-step A2C and an added LELBO objective.

  • Model Predictions: DVRL reconstructs the current observation and unrolls predicted future observations using its learned generative model, repeating the last action.The reconstruction is shown at dt0; predictions extend to dt values of 1, 2, 3, 10, and 30.
  • Model Predictions: The model predicts task-relevant features such as car movement, approximate ball position, and missing pins, even when the current observation is blank.It also correctly learns to randomly predict blank observations.
  • Model Predictions: DVRL remembers some current-state features, including Centipede barrier positions, but struggles with information-rich MsPacman scenes.The difficult features include previously eaten fruits and ghost locations.
  • Training: The training procedure is a detailed n-step A2C implementation with LELBO and an option to retain the computation graph for longer backpropagation.The algorithms describe recurrent belief-state computation for DVRL, recurrent-state computation for RNN, and overall training using either history encoder.
  • Training: Adding reconstruction loss to the RNN-based encoder reliably reduced performance relative to the RNN without reconstruction loss.
Loading 1806.02426v1…