Source-linked AI summary
Variational Inference for Data-Efficient Model Learning in POMDPs
Sebastian Tschiatschek, Kai Arulkumaran, Jan Stühmer, Katja Hofmann
TL;DR
Data-efficient learning in POMDPs is challenging because partial observations require reasoning over interaction histories, while RL training can demand extensive data and computation. DELIP learns POMDP dynamics and reward models with structured variational inference; experiments report parity with DRQNs when learning the full model and an advantage in the reward-only setting.
Problem
Data-efficient learning is difficult in POMDPs because agents must learn from interaction histories under partial observation, while RL approaches can require extensive data and computation.
Source: §1 · pp. 1–2Method
DELIP uses structured variational inference to learn generative models of POMDP dynamics and rewards for use with black-box planners.
Source: §1 · p. 2Results
DELIP and DRQNs perform on par when the full environment model is learned, while DELIP outperforms DRQNs for most dataset sizes when learning rewards atop a pretrained environment model.
Source: §4.3 · p. 8Takeaways & Limitations
Experiments confirm DELIP is particularly data-efficient when dynamics are consistent but rewards change or are initially unknown.
Source: §1 · p. 2Results showed large variance across random seeds.
Source: §4.3 · p. 6
Abstract
from arXiv · showhide
Partially observable Markov decision processes (POMDPs) are a powerful abstraction for tasks that require decision making under uncertainty, and capture a wide range of real world tasks. Today, effective planning approaches exist that generate effective strategies given black-box models of a POMDP task. Yet, an open question is how to acquire accurate models for complex domains. In this paper we propose DELIP, an approach to model learning for POMDPs that utilizes amortized structured variational inference. We empirically show that our model leads to effective control strategies when coupled with state-of-the-art planners. Intuitively, model-based approaches should be particularly beneficial in environments with changing reward structures, or where rewards are initially unknown. Our experiments confirm that DELIP is particularly effective in this setting.
1 Introduction
The introduction frames data-efficient learning in POMDPs as a challenge because agents must learn from interaction histories, not just individual observations. It proposes DELIP, using structured variational inference to learn dynamics and reward models, and reports particular effectiveness when rewards change or are initially unknown.
1 Introduction
- RL training can require tens to hundreds of millions of samples, with substantial computational costs even when accurate simulations are available.The cited Atari examples illustrate the scale of data use and note that training remains computationally expensive.Source: §1 · p. 1
- POMDP learning must account for how behavior depends on potentially long interaction sequences, not only individual state observations.This compounds data-efficiency challenges in settings where agents observe only local signals.Source: §1 · pp. 1–2
- DELIP uses structured variational inference to learn generative models of POMDP dynamics and rewards, and its learned models support effective strategies with black-box planners.The introduction reports particular data efficiency when dynamics are consistent but rewards change or are initially unknown.Source: §1 · p. 2
2 Notation and Background
The background presents POMDPs as controlled hidden Markov models and reviews variational inference as a way to learn latent-variable models. It then motivates a structured posterior that uses future trajectory information when inferring latent states.
2.1 Partially Observable Markov Decision Processes (POMDPs)
- POMDPs model latent states, actions, observations, rewards, transitions, and discounting; state transitions, observations, and rewards are generated conditionally on states and actions.In traditional POMDP reinforcement learning, a belief over the current state is updated from past actions, beliefs, rewards, and observations.Source: §2.1 · p. 2, Figure 1 · p. 3
2.2 Variational Inference
- Variational inference learns latent-variable models by optimizing a lower bound on the marginal observation probability, approximating the posterior with a simpler distribution and minimizing KL divergence.VAEs use generative and inference models that can be trained with stochastic gradient descent.Source: §2.2 · p. 3
2.2.1 Structured Variational Inference
- For the POMDP trajectory factorization, the posterior of a latent state depends on future observations, rewards, and actions, motivating recurrent networks to summarize them.The joint trajectory probability is conditioned on the action sequence.Source: §2.2.1 · p. 3
3 Model & Planning
DELIP parameterizes a generative POMDP model with neural networks and learns it from trajectories using structured, amortized variational inference. For planning, it supplies that learned model to POMCP for simulated rollouts and belief updates.
3.1 Model
- The prior state, transition, observation, and reward distributions are parameterized by neural networks and learned from trajectory data.Training jointly optimizes the variational posterior and generative model with the ELBO.Source: §3.1 · p. 4
- The variational posterior uses an RNN over future observations, rewards, and actions to parameterize a four-state Gaussian mixture over latent states.The posterior and generative model are jointly optimized using structured, amortized variational inference.Source: §3.1 · p. 4
3.2 Planning
- POMCP rollouts sample next states, observations, and rewards from the learned model rather than a potentially prohibitive simulator.After an action, the model also supports Bayesian filtering to update the state belief from actual observations and rewards.Source: §3.2 · p. 4
- POMCP searches from the current state belief, expanding a tree of hypothetical states and balancing exploration with exploitation.Random-policy rollouts initially estimate state values.Source: §3.2 · p. 4
4 Experiments
The experiments evaluate DELIP on a partially observed robot navigation task, including both full environment-model learning and reward learning on a pretrained model. Planning results show parity with DRQN in the full-model setting and an advantage for DELIP across most reward-only dataset sizes, with a small-data qualification.
4.1 Experimental Setup
- The task requires locating and opening the target door from boundary and door-presence signals, because the robot cannot observe its actual position or which door it faces.Episodes begin at positions sampled from a zero-mean Gaussian with standard deviation 5, clipped to world boundaries at −15 and 15.Source: §4.1 · p. 6
4.2 Learning Environment Models
- A common learned-model failure is a missing door, which can make a planning agent infer its position incorrectly and perform poorly.The successful model's predicted observations and rewards closely resemble ground truth; the failed model cannot predict all four doors and has high prediction variance.Source: §4.2 · p. 6, Figure 3 · p. 7
4.3 Planning
- The authors report large variance in results across random seeds.The statement accompanies results obtained with the same model architecture while optimization hyperparameters varied.Source: §4.3 · p. 6, §4.2 · p. 6
- The experiments compare learning the full transition, observation, and reward model with learning rewards alone on a pretrained environment model.The reward-only setting models tasks that share the same basic environment; random-policy data comprise observations and rewards for full-model learning, but rewards only for reward-only learning.Source: §4.3 · p. 7
- With a full learned model, DELIP and DRQNs perform on par; in REWARDS-ONLY, DELIP beats DRQN for most dataset sizes, except very small samples where positive door rewards are not reliably learned.Figure 4 reports performance over training-sample counts for reward learning on a pretrained environment model.Source: Figure 4 · p. 8, §4.3 · p. 8
5 Related Work
The related work contrasts model-free POMDP learning and scalable planning with the challenge of acquiring accurate models. DELIP combines variationally learned models with black-box planning, addressing a gap left by prior model-learning work that had not derived behavior policies.
Related model-free learning
- Model-free POMDP approaches can use recurrent models to maintain memory, while learning explicit memory and attention mechanisms remains a direction for data-efficiency comparisons.The paper compares with DRQN, and cites Singh et al. (1994) on the limitations of optimal deterministic memoryless policies.Source: §5 · p. 8
POMDP planning and model acquisition
- POMCP provides a scalable black-box planner, while acquiring an accurate domain model remains a separate practical challenge.The paper uses POMCP and notes that assuming an accurate model before interaction is unrealistic in many applications.Source: §5 · p. 9
- BA-POMDP approaches allow online model updates but require more domain knowledge to instantiate their parametric models than the paper's flexible learning approach.Source: §5 · p. 9
Variational inference and planning
- Prior variational-inference work learned POMDP models without deriving behavior policies; DELIP couples learned models with state-of-the-art black-box planning to demonstrate competitive policies.Source: §5 · p. 9
6 Conclusion
The conclusion presents DELIP as a flexible learner of POMDP dynamics and rewards whose models lead to effective control strategies. It points to comparisons between model-learning approaches and end-to-end learning to plan as future directions.
6 Conclusion
- DELIP flexibly learns POMDP dynamics and reward models that lead to effective control strategies.The conclusion describes this as progress toward data-efficient POMDP learning with minimal prior assumptions.Source: §6 · p. 9
- Future work will compare the advantages and trade-offs of modern model-learning approaches, including recently proposed autoregressive models.Source: §6 · p. 9
- Recent progress in learning to plan could enable principled, scalable model-based approaches learned end to end.Source: §6 · p. 9