Source-linked AI summary
GP-VAE: Deep Probabilistic Time Series Imputation
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, Stephan Mandt
TL;DR
Missing values in multivariate time series require methods that capture temporal and cross-channel structure while providing uncertainty estimates. GP-VAE combines a VAE-based complete latent representation with Gaussian-process dynamics and structured variational inference. It reports favorable performance against classical and deep-learning baselines, with smoother imputations and interpretable uncertainty estimates.
Problem
Missing values impair multivariate time-series usefulness, while existing approaches can omit temporal or cross-channel structure and uncertainty estimation.
Method
GP-VAE maps incomplete time series into a complete latent space using a VAE, models latent dynamics with a Gaussian process, and uses structured variational inference.
Results
The model outperforms classical and modern deep-learning baselines on benchmark and medical data, while producing smoother imputations and useful uncertainty estimates.
Takeaways & Limitations
Latent Gaussian-process dynamics provide a probabilistic approach to multivariate time-series imputation with favorable performance and interpretable uncertainty estimates.
Takeaways & Limitations
Medical-data evaluation uses mortality-prediction AUROC as a proxy because ground-truth missing values are unavailable, and the data have about 80% missing features.
Abstract
from arXiv · showhide
Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years. This raises the question whether deep learning methodologies can outperform classical data imputation methods in this domain. However, naive applications of deep learning fall short in giving reliable confidence estimates and lack interpretability. We propose a new deep sequential latent variable model for dimensionality reduction and data imputation. Our modeling assumption is simple and interpretable: the high dimensional time series has a lower-dimensional representation which evolves smoothly in time according to a Gaussian process. The non-linear dimensionality reduction in the presence of missing data is achieved using a VAE approach with a novel structured variational approximation. We demonstrate that our approach outperforms several classical and deep learning-based data imputation methods on high-dimensional data from the domains of computer vision and healthcare, while additionally improving the smoothness of the imputations and providing interpretable uncertainty estimates.
1 Introduction
Missing values make multivariate time-series data difficult to use, while existing approaches often fail to combine temporal and cross-channel information with uncertainty estimation. GP-VAE addresses this by combining nonlinear latent representations, Gaussian-process dynamics, and structured efficient inference, and reports favorable benchmark performance.
- Missing values impair the usefulness and interpretability of time-series data, motivating imputation from observed values.
- Multivariate imputation should exploit both within-channel temporal correlations and correlations across channels while providing uncertainty estimates.
- Existing methods can fail with missing features, overlook complex cross-channel interactions, or ignore temporal structure when applying nonlinear dimensionality reduction.
- GP-VAE maps incomplete time series into a complete latent space with a VAE and models low-dimensional temporal dynamics using a Gaussian process.
- The model uses a Cauchy kernel for multiple time scales and structured variational inference that models posterior time correlations.
- The structured variational distribution makes sampling linear in the number of time steps rather than cubic under naïve sampling.
- Extensive comparisons with classical and state-of-the-art deep-learning methods on two domains show favorable performance in both cases.
2 Related work
Prior work includes simple heuristics, Bayesian methods, and deep generative approaches, but each has limitations involving assumptions, scalability, missingness, or temporal modeling. GP-VAE combines ideas from these traditions to model missing time series with latent temporal dynamics.
- Heuristic methods such as mean and forward imputation remain widely used because they are efficient and interpretable.
- Expectation-maximization methods can require additional modeling assumptions.
- Bayesian methods offer advantages for likelihood and uncertainty estimation, but adoption is hindered by limited scalability and kernels that are difficult to make robust to missing values.
- VAEs provide tractable likelihoods, whereas GANs generally require additional optimization to obtain latent representations.
- HI-VAE evaluates reconstruction only on observed data and uses arbitrary fills during inference, but it was not formulated for sequential data.
- Most deep-learning imputation methods do not directly model temporal dynamics; BRITS uses recurrent networks, while GRUI-GAN accounts for time-series structure.
- Removing the latent GP and structured variational distribution from GP-VAE yields HI-VAE, illustrating how the proposed model combines related approaches.
3 Model
The model embeds incomplete multivariate time series into a complete low-dimensional latent space, models temporal dynamics there with a Gaussian process, and uses structured variational inference for efficient posterior approximation.
- Model architecture: The architecture maps missing-data time series into a complete, reduced-dimensional latent space and models temporal dynamics with a Gaussian process.The latent representation captures cross-feature correlations for reconstruction while separating instantaneous feature relationships from temporal dynamics.
- Model architecture: Applying Gaussian processes in latent space avoids designing kernels directly over incomplete observations and decouples imputation from temporal modeling.Direct GP modeling in data space faces cubic kernel inversion, difficult feature-and-time kernel design, and complications from missing values.
- Temporal dynamics: A Cauchy-kernel GP prior models multiple temporal scales that conventional single-length-scale kernels may miss.The Cauchy kernel is obtained from a mixture of RBF kernels with different length scales.
- Generative model: The decoder generates observations pointwise from latent states through a potentially nonlinear function gθ, implemented as a deep neural network.In the medical interpretation, the latent state represents physiology and the decoder generates measurements such as heart rate and blood pressure.
- Inference model: Structured variational inference models temporal posterior correlations while allowing linear-time sampling in T through a tridiagonal precision structure.The approximation preserves correlations across time, permits dense covariance structure, and uses only about twice as many parameters as a fully factorized approximation.
4 Experiments
Experiments on Healing MNIST, SPRITES, and Physionet evaluate GP-VAE against classical and deep-learning baselines using reconstruction, likelihood, MSE, and downstream AUROC measures. GP-VAE generally outperforms baselines, while producing stable or smoother imputations and posterior uncertainty estimates.
- Experimental setup: Experiments span Healing MNIST, SPRITES, and Physionet, comparing GP-VAE with single-imputation, GP-based, VAE-based, and recurrent deep-learning methods.The evaluation includes both synthetic or benchmark data and real-world medical time series.
- Healing MNIST: GP-VAE outperforms baselines on Healing MNIST likelihood and MSE, with more stable reconstructions over time using neighboring-frame information.The reconstructions expose instability in related approaches, whereas GP-VAE incorporates temporal information from neighboring frames.
- Missingness mechanisms: GP-VAE outperforms all baselines on the reported Healing MNIST missingness mechanisms using MSE.The evaluation covers artificial missingness with different omission mechanisms.
- SPRITES: GP-VAE outperforms the baselines and yields the most convincing SPRITES reconstructions, while Table 1 reports NLL, MSE, and downstream AUROC.The table reports means and standard errors over the test set, with lower NLL and MSE and higher AUROC preferred.
- Real medical time series data: On Physionet, GP-VAE outperforms most baselines, performs comparably to BRITS, and provides posterior samples for uncertainty estimation.BRITS has better quantitative performance, but GP-VAE supplies a generative posterior and interpretable uncertainty across clinical variables.
- Real medical time series data: GP-VAE produces smoother clinical imputations, likely because its GP prior denoises noisy measurements similarly to a Kalman filter.The smoothing can make medical imputations more interpretable and help identify temporal trends.
5 Conclusion
The paper presents a deep probabilistic model that combines VAEs, Gaussian processes, and structured variational inference for multivariate time-series imputation. Experiments on benchmark and medical datasets found it outperformed classical and modern deep-learning baselines while performing comparably to the state of the art.
- The model maps incomplete time series into a latent space with a VAE and models temporal dynamics there using a Gaussian process.Structured variational inference approximates the latent GP posterior while retaining temporal correlations more accurately than a fully factorized approximation.
- Structured variational inference preserves temporal posterior correlations while keeping inference efficient compared with inference in the full GP posterior.
- The model was empirically validated on benchmark datasets and real-world medical data.
- The model outperformed classical baselines and modern deep-learning approaches on these tasks while performing comparably to the state of the art.
- Future work includes testing other domains, exploring additional latent-GP kernels, and using more sophisticated inference and generative networks.
A.1 Neural network architectures
The architecture uses a CNN inference network over time and an MLP generative network. Image-based experiments add 2D convolutional preprocessing before temporal processing, with experiment-specific hyperparameters reported in supplementary tables.
- The generative network is a fully connected multilayer perceptron.
- The inference network is a CNN that convolves over time, supports variable-length sequences, and integrates neighboring time steps through a fixed receptive field.
- The CNN outputs a tensor of size R^T × 3k, with each time-step row parameterizing the latent mean and covariance structure.
- Healing MNIST: Healing MNIST uses 2D convolutional preprocessing for each image state before flattening features for the 1D temporal convolution.
- Hyperparameters for Healing MNIST and SPRITES are provided in Tables S1 and S2.
A.4 Real medical time series data
For PhysioNet medical time series, the model processes non-image data directly with a 1D temporal convolution. The experiment follows the BRITS evaluation protocol and withholds 10% of observed measurements for validation.
- PhysioNet data are fed directly into the 1D convolutional layer over time without convolutional image preprocessing.
- The evaluation follows the BRITS protocol and uses the same set for training and evaluation.
- 10% of observed measurements are randomly eliminated for validation.
- The experiment’s hyperparameters are reported in Table S3.
B.1 Missingness rates on Healing MNIST
On Healing MNIST, the proposed model outperforms other deep architectures across missingness rates from 10% to 90%. The result also supports the usefulness of the Cauchy kernel for modeling temporal dynamics.
- 10%–90% missingness rates were introduced into Healing MNIST time series to evaluate sensitivity to missing-data severity.
- The comparison uses negative log likelihood, with lower values indicating better performance.
- The proposed model outperforms other deep architectures, including GP-VAE with an RBF kernel, at every tested missingness rate.
- The results indicate that the Cauchy kernel helps model the temporal dynamics.
B.2 Missingness mechanisms on Healing MNIST
The experiment evaluates Healing MNIST imputation under several structured missingness mechanisms at approximately 50% average missingness. The proposed model outperforms all baselines in likelihood and MSE across these settings, while MNAR is hardest for VAE-based models.
- Missingness mechanisms: Spatial missingness correlates missingness across neighboring pixels through a Gaussian process with an RBF kernel.This mechanism makes a pixel more likely to be missing when certain other features are missing.
- Missingness mechanisms: Positive temporal missingness increases the likelihood that a feature remains missing at the consecutive time step, using a temporal Gaussian process.The RBF kernel is defined over time separately for each feature.
- Missingness mechanisms: Negative temporal missingness decreases the likelihood of consecutive missingness and is generated with a determinantal point process over time.The process is applied separately to each feature.
- Missingness mechanisms: MNAR missingness depends on the underlying feature value, with white pixels twice as likely to be missing as black pixels.The experiment assesses all mechanisms and reports negative log likelihood in Table S5, where lower is better.
- Results: The proposed model outperforms all baselines in likelihood and MSE across every missingness mechanism.For VAE-based models, MNAR is hardest, followed by correlated features; temporal correlation is no harder than random missingness.