PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks

Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, Steffen Udluft

arXiv:1605.07127v3stat.MLcs.LG

TL;DR

The paper addresses the difficulty of learning expressive stochastic transition models for model-based reinforcement learning, especially in batch settings with restricted exploration. It combines Bayesian neural networks with random input noise, α-divergence training, random roll-outs, and stochastic policy optimization. The approach solves a challenging benchmark where model-based methods usually fail and produces promising results on gas-turbine and industrial data.

  • Problem

    Common transition models are limited in the stochasticity they can express, despite real-world dynamics involving complex unobserved environmental factors.

  • Method

    The method combines BNNs with random input noise, α-divergence training, random roll-outs, and stochastic optimization for batch policy learning.

  • Results

    The approach solves a challenging benchmark where model-based methods usually fail and produces promising results on industry benchmarks, including real-world gas-turbine data.

  • Takeaways & Limitations

    A BNN trained with α = 0.5 and the proposed policy-optimization algorithm is presented as a powerful black-box tool for policy search.

  • Takeaways & Limitations

    Exact computation of the predictive objective is intractable and therefore requires approximation.

Abstract

from arXiv · show

We present an algorithm for model-based reinforcement learning that combines Bayesian neural networks (BNNs) with random roll-outs and stochastic optimization for policy learning. The BNNs are trained by minimizing $α$-divergences, allowing us to capture complicated statistical patterns in the transition dynamics, e.g. multi-modality and heteroskedasticity, which are usually missed by other common modeling approaches. We illustrate the performance of our method by solving a challenging benchmark where model-based approaches usually fail and by obtaining promising results in a real-world scenario for controlling a gas turbine.

1 INTRODUCTION

The paper targets stochastic transition dynamics that common model-based approaches cannot represent robustly. It combines Bayesian neural networks with random input noise, α-divergence training, and policy-search procedures for batch reinforcement learning.

  • 1 INTRODUCTION: Common transition models assume deterministic dynamics, limiting their ability to express complex stochasticity from unobserved environmental factors.Examples include Gaussian processes, fixed bases, and neural networks with Gaussian observation noise.
  • 1 INTRODUCTION: The proposed model uses a Bayesian neural network with a random input disturbance z to express stochastic dynamics.The disturbance is incorporated into the learned transition function alongside the current state and action.
  • 1 INTRODUCTION: α-divergence minimization with α = 0.5 trains BNN transition functions that are both scalable and expressive in their stochastic patterns.The paper identifies these two properties as characteristics not simultaneously achieved by previous work.
  • 1 INTRODUCTION: For off-policy batch reinforcement learning, the algorithm uses BNN predictions, random roll-outs, and stochastic optimization to learn an improved policy.The setting begins with data from an already-running system, making it relevant to applications where exploration is restricted.
  • 1 INTRODUCTION: The method provides a model-based solution to the Wet-Chicken benchmark and reports promising results for gas-turbine control and an industrial benchmark.These evaluations address challenging benchmark and real-world control settings.

2 BACKGROUND

The paper models stochastic dynamics with Bayesian neural networks that combine learned weight uncertainty, latent input noise, and additive observation noise. It approximates the posterior using α-divergence minimization, with α = 0.5 chosen to balance local-mode fitting and global posterior coverage while enabling scalable stochastic optimization.

  • Background: Model-based reinforcement learning learns an approximate transition distribution from state-action-next-state samples, then plans as if that approximation were the true dynamics.The target is T_approx(st+1|s,a), learned from collected transitions and used for control.
  • Bayesian neural networks with stochastic inputs: The Bayesian neural network models outputs as f(x_n, z_n; W) + ϵ_n, where latent Gaussian input noise captures complex unobserved stochasticity and additive Gaussian noise preserves a well-defined predictive density.Without z_n, randomness is restricted to additive Gaussian observation noise; without ϵ_n, finite-sample predictions yield a degenerate mixture of delta functions.
  • Bayesian neural networks with stochastic inputs: The posterior over weights W and disturbances z is approximated by a factorized Gaussian q(W,z), and predictions replace posterior integration with empirical averages over samples from q.The approximation is introduced because exact posterior computation is intractable.
  • α-divergence minimization: α controls the approximation’s behavior: α ≤ 0 favors local modes, α ≥ 1 favors global posterior coverage, and α = 0.5 balances these tendencies.As α approaches zero, the method becomes equivalent to variational Bayes.
  • α-divergence minimization: The method minimizes an energy function using stochastic gradient descent with mini-batches, sampled expectations, and the reparameterization trick, while also tuning Σ, λ, and γ.This provides a scalable approximation to direct α-divergence minimization.
  • α-divergence minimization: Using α = 0.5 rather than the α → 0 variational-Bayes limit often improves test log-likelihood and remains robust when q(z) is fixed to its prior, whereas VB fails in that setting.The reported robustness concerns cases where q(z) is not fully optimized.

3 POLICY SEARCH USING BNNS WITH STOCHASTIC INPUTS

The method learns stochastic transition models with Bayesian neural networks and optimizes policies through sampled virtual roll-outs. It combines parameter uncertainty, unobserved-process noise, and stochastic gradient optimization for finite-horizon policy search.

  • Model-based policy search: The algorithm learns a stochastic dynamics model from state transitions, then optimizes deterministic policy parameters over the learned model.The transition model uses a BNN with stochastic inputs, while policy search minimizes expected finite-horizon cost.
  • Stochastic transition modeling: The BNN represents transition uncertainty by sampling both weights W from q(W) and input disturbances z_t from a Gaussian distribution.Weight sampling captures uncertainty about model parameters, while input-noise sampling captures stochasticity from unobserved processes.
  • Policy optimization: Policy evaluation averages costs across virtual roll-outs generated with sampled models, stochastic inputs, additive noise, and the candidate policy.Each roll-out applies a_t = π(s_t; W_π) over a finite horizon T.
  • Policy optimization: The expected-cost approximation is optimized with stochastic gradient descent, using automatic differentiation on Monte Carlo samples.The algorithm samples initial states from available transitions and approximates integrations over model weights, disturbances, additive noise, and trajectories.
  • Policy visualization: Figure 3 compares VB, α = 0.5, and Gaussian-process policies by state-space action patterns and average rewards.The displayed average rewards are −2.53 for VB, −2.31 for α = 0.5, and −2.94 for the Gaussian-process policy.

4 EXPERIMENTS

The experiments evaluate stochastic BNN models and model-based policy search on Wet-Chicken and industrial benchmarks. The α-divergence BNNs capture complex predictive distributions and produce strong policy performance, including on gas-turbine data.

  • Wet-Chicken benchmark: Wet-Chicken combines bi-modal, heteroskedastic transition dynamics with a long-horizon control objective, making it difficult for model-based policy search.The benchmark uses a two-dimensional continuous river task with stochastic drift and turbulence, trained from 2,500 random transitions.
  • Wet-Chicken benchmark: BNNs with α = 0.5 produce predictive distributions close to ground truth, while Gaussian processes miss multi-modality and VB misestimates density in specific regions.The α = 0.5 model is reported as close to the ground truth in selected predictive-distribution plots.
  • Wet-Chicken benchmark: BNNs with α = 0.5 achieve the best Wet-Chicken policy reward and approach the optimal upper bound, while VB has substantially higher empirical variance.The comparison uses policies trained with Algorithm 1 over a horizon of T = 5; PSO-P indicates the optimal upper bound.
  • Industrial applications: On gas-turbine data, α = 0.5 has the highest test log-likelihood and best policy performance across five repeated experiments.The task uses 40,000 observations from a 30-dimensional real-turbine time series and simulates partial observability.
  • Industrial applications: The industrial world model predicts reward from recent action and reward histories, while environmental variables are held constant in the turbine model.The reward is defined from fatigue and consumption, and the environmental variables are assumed to satisfy E_t+1 = E_t.
  • Industrial applications: For long-term roll-outs, VB and α = 0.5 BNNs represent bi-modal trajectories and tighter confidence bands, whereas the MLP misses an upper mode.The visualization uses 75 samples over 75 steps for two starting states and compares individual samples, sample averages, and observed trajectories.

5 RELATED WORK

Prior work includes Bayesian methods for exploration, visual stochastic control, policy gradients, Gaussian-process model-based search, and recurrent-neural-network approaches. The paper positions its method as a Monte-Carlo model-based policy-gradient technique for continuous stochastic systems.

  • Prior Bayesian and control approaches: Earlier Bayesian-neural-network reinforcement learning work addressed exploration in contextual bandits, while related visual-control work combined variational autoencoders with stochastic optimal control.These approaches target different settings from the paper's policy-search problem.
  • Policy search: Policy-gradient methods are prominent policy-search algorithms, while model-based methods were historically more common in discrete spaces and model-free methods in continuous spaces.The paper contrasts these broad trends in prior reinforcement-learning research.
  • Positioning: The paper characterizes its approach as Monte-Carlo model-based policy gradient for continuous stochastic systems, related to Gaussian-process and recurrent-neural-network methods.The cited Gaussian-process approach propagates beliefs over roll-outs but is restricted to Gaussian state distributions.

6 CONCLUSION AND FUTURE WORK

The paper combines stochastic-input BNNs with random roll-outs and stochastic policy optimization for batch policy learning, achieving strong results on challenging and industrial problems while identifying safety and exploration as future work.

  • BNNs with random inputs are trained using α-divergence minimization, with α = 0.5 often outperforming variational Bayes.
  • The proposed algorithm uses random roll-outs and stochastic optimization to learn a parameterized policy from batch data, particularly targeting industry domains.
  • The method solves a challenging benchmark where model-based approaches usually fail and obtains promising results on real-world gas-turbine data.
  • Future work will address safety and exploration by using uncertainty in stochastic functions for worst-case policy optimization and efficient data collection.

A ROBUSTNESS OF α = 0.5 AND α = 1.0 WHEN q(z) IS NOT LEARNED

The experiments test whether α = 0.5 and α = 1.0 remain effective without learning q(z). Both capture bimodal and heteroskedastic predictive patterns, whereas VB converges to simpler, suboptimal distributions.

  • The evaluation uses bimodal and heteroskedastic regression problems, including noise whose magnitude changes with the input.
  • The predictive evaluation compares α = 0.5, α = 1.0, and VB while keeping q(z) fixed to the prior p(z).
  • The models use two hidden layers with 50 units per layer, Adam optimization, problem-specific learning rates, mini-batches, and 1000 training epochs.
  • Both α-divergence settings capture bimodality and heteroskedasticity, while VB produces single-mode or homoskedastic predictions.
  • BNNs trained with α = 0.5 or α = 1.0 are more robust than VB and model complex predictive distributions even when q(z) remains fixed.

B METHODS

The evaluation compares the proposed stochastic BNN approach with standard MLPs, VB, GPs, and a particle-swarm policy method that uses true dynamics as an upper-bound indicator.

  • The experiments compare the proposed method with standard MLP, VB, Gaussian Processes, and Particle Swarm Optimization Policy.
  • The standard MLP has deterministic weights and no stochastic inputs, with Gaussian output noise added during roll-outs.
  • VB is implemented through α-divergence minimization with α = 10^-6, corresponding in practice to the variational lower-bound approach.
  • Gaussian Processes are fitted separately for each target dimension using sparse FITC models with 150 inducing inputs.
  • PSO-P uses true dynamics in model-predictive control and provides an upper-bound indicator rather than a benchmark comparison method.

C MODEL PARAMETERS

The experiments use shared neural-network policy settings but vary model size, rollout horizon, training duration, batch size, and sampling across the WetChicken, Turbine, and Industrial Benchmark tasks.

  • Policies use two hidden layers with 20 rectifier units per layer, tanh outputs when action bounds are needed, and Adam optimization.
  • WetChicken: WetChicken models use two hidden layers with 20 units each and 2500 random state transitions for training.
  • Turbine: Turbine models use two hidden layers with 50 units each, horizon T = 20, mini-batches of 10, and K = 10 samples from q.
  • Industrial Benchmark: Industrial Benchmark models use two hidden layers with 75 units, horizon T = 75, 500 epochs, batches of 50, and K = 25 samples per rollout.

MODEL TRAINING

Model training uses a single GPU for all models, with Bayesian neural networks requiring substantially more time than standard neural networks. Policy training uses a single CPU, with duration varying by horizon, dataset size, and network topology.

  • Model Training: 30 minutes to 5 hours: BNN training time varied with data size and benchmark dimensionality.Standard neural-network training took 5–20 minutes under the same hardware setup.
  • Policy Training: 20 minutes to 16 hours: policy training duration varied across benchmarks and horizons.Examples include 20 minutes for Wet-Chicken with T = 5, 3–4 hours for Turbine with T = 20, and 14–16 hours for the industrial benchmark with T = 75.
Loading 1605.07127v3…