PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Recurrent State-Space Models

Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, Stefan Schaal, Marc Toussaint, Sebastian Trimpe

arXiv:1801.10395v2stat.ML

TL;DR

Probabilistic nonlinear state-space models are difficult to train robustly, despite their value for modeling uncertain dynamics. The paper proposes PR-SSM, combining GP priors with doubly stochastic variational inference and temporally correlated latent-state approximations. It evaluates the approach on real-world benchmarks and a high-dimensional problem, demonstrating robustness, scalability, and high performance against state-of-the-art methods.

  • Problem

    Probabilistic nonlinear state-space models are difficult to train robustly, particularly for higher-dimensional systems, because existing approximations may fail to represent temporal latent-state correlations.

  • Method

    PR-SSM combines GP-based transition modeling, doubly stochastic variational inference, temporally correlated latent-state distributions, and a recognition model for state initialization.

  • Results

    PR-SSM demonstrates robustness, scalability, and high performance on real-world datasets compared with state-of-the-art methods, consistently outperforming SS-GP-SSM.

  • Takeaways & Limitations

    Maintaining the true latent-state distribution and temporal correlations enables long-term gradients and makes efficient inference for complex latent spaces feasible.

  • Takeaways & Limitations

    PR-SSM depends on an a-priori fixed latent-state dimensionality.

Abstract

from arXiv · show

State-space models (SSMs) are a highly expressive model class for learning patterns in time series data and for system identification. Deterministic versions of SSMs (e.g. LSTMs) proved extremely successful in modeling complex time series data. Fully probabilistic SSMs, however, are often found hard to train, even for smaller problems. To overcome this limitation, we propose a novel model formulation and a scalable training algorithm based on doubly stochastic variational inference and Gaussian processes. In contrast to existing work, the proposed variational approximation allows one to fully capture the latent state temporal correlations. These correlations are the key to robust training. The effectiveness of the proposed PR-SSM is evaluated on a set of real-world benchmark datasets in comparison to state-of-the-art probabilistic model learning methods. Scalability and robustness are demonstrated on a high dimensional problem.

1. Introduction

Probabilistic nonlinear state-space models are valuable for safe control but difficult to train robustly. PR-SSM addresses this with temporally correlated variational inference, GP-based predictions, and scalable recognition-model initialization.

  • 1. Introduction: PR-SSM targets robust training of probabilistic, nonlinear state-space models, whose probabilistic predictions support safe learning and reduce model bias.The framework is motivated especially by higher-dimensional systems, where robust training remains only partially solved.
  • 1. Introduction: The latent transition is unrolled over time, preserving temporal correlations while enabling backpropagation through time and reducing latent-state initialization difficulties.
  • 1. Introduction: Gaussian-process priors provide probabilistic predictions, complex latent-state distributions, and principled model-complexity regularization.
  • 1. Introduction: The contributions combine gradient- and sample-based inference, a tractable variational approximation, doubly stochastic inference, and a recognition model for latent-state initialization.The approximation maintains the true latent-state posterior and temporal correlations, while doubly stochastic inference targets scalability.
  • 1. Introduction: PR-SSM is evaluated on real-world system-identification datasets against state-of-the-art methods to assess robustness and effectiveness.

2. Related Work

Related work spans latent state-space, predictive-state, autoregressive, hybrid, and GP-based probabilistic models. PR-SSM differs by efficiently incorporating true nonlinear temporal dynamics through sampling- and gradient-based learning.

  • 2. Related Work: Autoregressive methods avoid latent-state inference by mapping histories of inputs and observations to future observations, but can feed observation noise back into learned dynamics.Hybrid latent-state formulations and explicit noise modeling address this issue.
  • 2. Related Work: State-space models use compact Markovian latent representations and directly support control algorithms requiring explicit latent states.Linear models admit Kalman filtering, smoothing, and subspace-identification solutions; nonlinear models require approximate approaches.
  • 2. Related Work: Deterministic recurrent models are powerful, whereas system identification and control often require probabilistic predictions that expose model errors.PR-SSM can be interpreted as a probabilistic counterpart of deep recurrent learning, sharing explicit transition and observation-model unrolling.
  • 2. Related Work: GP-SSMs provide Bayesian modeling and automatic complexity regularization, but near-exact sample-based methods become inefficient or intractable for larger datasets and latent dimensions.Variational methods improve tractability but differ in how much joint temporal structure they retain.
  • 2. Related Work: PR-SSM efficiently incorporates true latent dynamics by combining sampling- and gradient-based learning, rather than imposing only a linear time-varying approximation.

3. Gaussian Process State-Space Model

The PR-SSM combines GP transition dynamics with a parametric observation model and sparse inducing-point approximations. Its graphical structure distinguishes observed and latent variables, with GP priors coupling jointly Gaussian quantities.

  • 3.1. Gaussian Process: A GP defines a distribution over functions through a mean function and covariance function, with finite function evaluations jointly Gaussian.Its predictive distribution at a new input is obtained by conditioning on observed function values.
  • 3.1. Gaussian Process: The model uses a squared-exponential kernel with automatic relevance determination, while any differentiable kernel could be incorporated under the sampling-based inference scheme.
  • 3.2. Sparse GP: Sparse GP approximations condition predictions on inducing inputs and outputs, making predicted function values mutually independent given the inducing points.This reduces dependence on the full training dataset, although the passage does not specify the resulting complexity here.
  • 3.3. PR-SSM Model Definition: PR-SSM places a GP prior on the transition function and uses a parametric observation model to reduce transition–observation nonidentifiability.Independent GP priors are used for each latent-state dimension with separate inducing points and outputs.
  • 3.3. PR-SSM Model Definition: The graphical model marks observed variables in gray, latent variables in white, and thick-linked variables as jointly Gaussian under a GP prior.
  • 3.3. PR-SSM Model Definition: The observation matrix selects the first D_y latent dimensions, a design suited to low-dimensional sensor observations relative to latent-state dimensionality.Those selected latent dimensions can be interpreted as noise-free sensor measurements.
  • 3.3. PR-SSM Model Definition: The transition maps the previous state and input to the next latent transition output, while the initial state distribution remains unknown and must be estimated.The notation uses x_t for latent state, u_t for input, y_t for output, and f_{t+1}=f(x_t,u_t).

4. PR-SSM Inference

PR-SSM uses a variational sparse-GP formulation that preserves latent-state temporal correlations instead of factorizing them, while combining sampling and gradients to make inference tractable. Its ELBO balances output fit against GP-prior complexity and supports stochastic optimization and sample-based prediction.

  • 4.1. Variational Sparse GP: The sparse-GP variational approximation uses explicit Gaussian inducing-output distributions, enabling stochastic gradient optimization and parameter growth independent of trajectory length.Explicit inducing outputs avoid the O(NP^2) complexity associated with eliminating them, while PR-SSM parameters grow with latent dimensions rather than time-series length.
  • 4.2. Variational Approximation: PR-SSM’s variational distribution follows the sparse-GP transition model, preserving temporal correlations that factorized approximations discard.The resulting distribution is more complex, so the method combines sampling with gradient-based optimization.
  • 4.3. Variational Evidence Lower Bound: The ELBO combines expected output log-likelihood with a regularizer penalizing deviations of the inducing-output distribution from the GP prior.This structure trades off explaining observed system behavior against model complexity.
  • 4.3. Variational Evidence Lower Bound: Because nonlinear latent-state expectations remain intractable, PR-SSM recursively samples latent states from the sparse-GP posterior and differentiates through them using reparameterization.The resulting stochastic ELBO admits analytic gradients for stochastic-gradient model optimization.
  • 4.3. Variational Evidence Lower Bound: After optimization, PR-SSM predicts by sampling the complex variational latent-state distribution rather than using moment matching approximations.The predicted observation distribution is computed from the sampled latent dynamics.

5. Extensions for Large Datasets

For large datasets and long trajectories, PR-SSM replaces full-trajectory ELBO optimization with mini-batch sub-trajectories and uses a recognition model to initialize their latent states. The recognition model also supports prediction when the initial latent state is unavailable.

  • 5. Extensions for Large Datasets: Mini-batches of sub-trajectories replace prohibitive full-gradient ELBO optimization for large datasets and long trajectories.Full-gradient training remains applicable to small datasets with short trajectories.
  • 5. Extensions for Large Datasets: A recognition model infers a Gaussian initial latent-state distribution from the first L input/output observations instead of independently optimizing each sub-trajectory’s initial state.Its mean and variance parameterize q(x1), and training errors propagate into the recognition model.
  • 5. Extensions for Large Datasets: Figure 2 contrasts untrained and trained PR-SSM predictions on Furnace data using observed data, predictions, latent/output samples, and two-standard-deviation bands.The untrained model shows random-walk latent behavior, whereas training reduces initial-state uncertainty during the first time steps; no recognition model is used in this experiment.
  • 5. Extensions for Large Datasets: The recognition model is required at prediction time because test sequences provide no initial latent-state information.It initializes predictions by inferring the unknown initial state from observed inputs and outputs.

6. Experimental Evaluation

Experiments compare PR-SSM with alternative probabilistic model-learning methods, showing robust optimization, strong benchmark performance, and scalability to a high-dimensional robotic-arm task.

  • 6.1. PR-SSM Learning: Full-gradient optimization is expensive for long trajectories, can suffer exploding or vanishing gradients, and may be fragile or prone to overfitting.These shortcomings motivate minibatch stochastic ELBO gradients and recognition-based initialization.
  • 6.2. Model Learning Benchmark: PR-SSM consistently outperforms SS-GP-SSM and remains close to the best latent autoregressive methods across five real-world benchmarks.Table 1 reports free-simulation RMSE means and standard deviations over five independently learned models.
  • 6.1. PR-SSM Learning: The recognition model initializes the latent state using initial system input/output data, reducing initial-state uncertainty, transient behavior, and deviations in predicted observations.Figure 3 compares uninformative initialization with initialization from a smoothing distribution based on the first L steps.
  • 6.1. PR-SSM Learning: Stochastic ELBO optimization with a recognition model is more robust and counteracts the overfitting tendencies observed with full-gradient learning.The stochastic scheme is also used for the subsequent model-learning benchmark because it applies to larger datasets.
  • 6.3. Large Scale Experiment: On the Sarcos task, PR-SSM robustly learns accurate forward dynamics for all seven outputs, whereas GP-NARX performs worse and fails on five joints.The task contains approximately 20,000 datapoints, with seven input and seven output dimensions.

7. Conclusion

PR-SSM combines GP priors with doubly stochastic variational inference to learn probabilistic Markovian state-space models efficiently. The framework maintains latent-state temporal structure, supports scalable and robust learning, and is evaluated against state-of-the-art methods.

  • 7. Conclusion: The framework demonstrates robustness, scalability, and high model-learning performance on real-world datasets against state-of-the-art methods.The paper also evaluates the approach on benchmark tasks involving physical-system data.
  • 7. Conclusion: PR-SSM combines GP priors and doubly stochastic variational inference to derive an optimization criterion related to deterministic RNN and LSTM training.The criterion supports explicit long-term gradient propagation through latent states.
  • 7. Conclusion: Maintaining the true latent state distribution and temporal correlations makes efficient inference in latent spaces feasible.The recognition model additionally supports learning unstable or slow dynamics and scaling to large datasets.
  • 7. Conclusion: PR-SSM requires the latent state dimensionality to be fixed a priori.The paper suggests a sparsity-enforcing latent-state prior as a potential remedy for unnecessary dimensions.
  • 7. Conclusion: Full-gradient optimization can overfit and produce unstable optimization, whereas stochastic optimization avoids those observed failure patterns.The full-gradient scheme shows overfitting in log likelihood and spikes in model loss; its stochastic counterpart uses initial-state regularization to prevent overconfident recognition predictions.

B. Model Learning Benchmark Details

The benchmark compares PR-SSM with state-of-the-art model-learning methods under a setup adapted from prior work. Configuration details and fairness-related adjustments are provided for interpreting the comparisons.

  • B. Model Learning Benchmark Details: The benchmark evaluates long-term predictive performance using a setup similar to prior work.Details of methods, configurations, and datasets are supplied, with minor adjustments introduced to enable fair comparison.

B.1. Benchmark Methods

The benchmark covers autoregressive, latent-space recurrent, and Markovian state-space methods. These baselines differ in their treatment of input noise, latent states, recurrence, and sparse GP approximations.

  • B.1. Benchmark Methods: The comparison includes one-step autoregressive models, latent-space multi-step autoregressive or recurrent models, and Markovian GP state-space models.The evaluated methods are GP-NARX and NIGP, REVARB1 and REVARB2, MSGP, and SS-GP-SSM.
  • B.1. Benchmark Methods: GP-NARX models dynamics from past inputs and outputs with a sparse GP, while NIGP models input uncertainty through state-dependent output noise.GP-NARX uses 100 inducing inputs; NIGP is trained on the full dataset where feasible.
  • B.1. Benchmark Methods: REVARB optimizes a variational lower bound using recurrent or deep GP structures, with benchmark variants containing one or two hidden layers.MSGP instead operates as a latent, noise-free GP-NARX model trained on long-term predictions.
  • B.1. Benchmark Methods: SS-GP-SSM uses a sparse-spectrum GP approximation for Markovian latent transition dynamics, with its implementation restricted to a two-dimensional latent space.Its experiments use the stated default configuration with K = 2000 and N = 40.

B.2. Benchmark Datasets

The datasets contain measured input-output trajectories from physical systems whose true states are higher dimensional than their one-dimensional inputs and outputs. On the Drives dataset, stochastic optimization is more robust than full-gradient optimization, which overfits log likelihood.

  • B.2. Benchmark Datasets: The benchmark datasets record one-dimensional inputs and outputs from physical systems including hydraulic actuators, furnaces, hair dryers, and electrical motors.Although Du = Dy = 1, the underlying system state is higher dimensional, motivating explicit latent-state modeling.
  • B.2. Benchmark Datasets: Full-gradient optimization achieves lower training RMSE and negative log likelihood than stochastic optimization on the Drives dataset.The comparison tracks long-term prediction metrics over learning iterations for training and test sets.
  • B.2. Benchmark Datasets: Test RMSE is similar across optimization schemes, but full-gradient optimization clearly overfits in test log likelihood.Its loss trajectory also has spikes and higher variance across five independent trainings, indicating less robust optimization.

C.2. Detailed Benchmark Results

The benchmark results account for sensitivity to preprocessing and prediction methods, while focusing the main comparison on uncertainty-aware long-term predictions. PR-SSM is compared against reference methods under these evaluation choices, with detailed Sarcos predictions showing stronger dynamics capture than GP-NARX.

  • Benchmark configuration: Benchmark results vary with data normalization and long-term prediction choices, so several reference-method configurations are reported.GP-NARX, NIGP, REVARB 1, and REVARB 2 are evaluated across normalized or unnormalized training data and mean or moment-matching predictions.
  • Sarcos results: PR-SSM captures the Sarcos robot-arm dynamics across all seven joint positions, whereas GP-NARX captures only a rough model for two joints.The figure compares GP-NARX predictions, PR-SSM predictions, and measured ground-truth joint positions.
  • Benchmark configuration: Moment-matching predictions are used in the main paper because uncertainty estimates are required for long-term prediction in model-based control.Mean-only predictions can sometimes achieve lower RMSE, but the main evaluation emphasizes approximate uncertainty propagation through moment matching.

C.3. Large Scale Experiment Details

The large-scale Sarcos experiment uses a seven-degree-of-freedom robot-arm dataset split into independent training and test experiments. PR-SSM trains directly on the full training data with minibatched optimization and robustly predicts all joints, improving over a GP-NARX baseline trained on a subset.

  • Task definition: The Sarcos system input consists of seven joint torques.The task uses a seven-degrees-of-freedom anthropomorphic robot-arm dataset containing positions, velocities, accelerations, and torques.
  • Large-scale Sarcos experiment: PR-SSM robustly predicts robot-arm motions for all seven joints and improves over GP-NARX, which fails to predict five joints.Long-term predictions are shown for one test experiment.
  • Dataset and split: The dataset contains 44,484 points recorded at 100 Hz, downsampled to 50 Hz, and split into 66 independent experiments.Six experiments are used for testing and 60 for training.
  • Baseline setup: Because reference methods are not directly applicable at this scale, GP-NARX is trained as a baseline on 400 inducing points and about 2,000 training points.The baseline uses only a subset of the training experiments.
  • PR-SSM setup: PR-SSM trains directly on the full training dataset using stochastic minibatched optimization and a 14-dimensional latent state.Its configuration otherwise follows the benchmark setup described in the paper.
Loading 1801.10395v2…