PaperlayerView exact revision on arXiv

Source-linked AI summary

CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

Yusuke Tashiro, Jiaming Song, Yang Song, Stefano Ermon

arXiv:2107.03502v2cs.LGstat.ML

TL;DR

Missing time-series values require probabilistic imputation methods that can use observed data effectively. CSDI directly learns a conditional distribution with score-based diffusion models, achieving stronger imputation results and competitiveness on related tasks.

  • Problem

    Missing values complicate time-series analysis, while existing imputation methods commonly rely on autoregressive models.

  • Method

    CSDI directly learns conditional distributions with score-based diffusion models conditioned on observed values and uses self-supervised training.

  • Results

    CSDI improves CRPS by 40-65% over existing probabilistic methods and reduces MAE by 5-20% versus state-of-the-art deterministic methods.

  • Takeaways & Limitations

    CSDI is also competitive for time-series interpolation and probabilistic forecasting.

  • Takeaways & Limitations

    Diffusion-model sampling is generally slower than other generative models, motivating efficiency improvements.

Abstract

from arXiv · show

The imputation of missing values in time series has many applications in healthcare and finance. While autoregressive models are natural candidates for time series imputation, score-based diffusion models have recently outperformed existing counterparts including autoregressive models in many tasks such as image generation and audio synthesis, and would be promising for time series imputation. In this paper, we propose Conditional Score-based Diffusion models for Imputation (CSDI), a novel time series imputation method that utilizes score-based diffusion models conditioned on observed data. Unlike existing score-based approaches, the conditional diffusion model is explicitly trained for imputation and can exploit correlations between observed values. On healthcare and environmental data, CSDI improves by 40-65% over existing probabilistic imputation methods on popular performance metrics. In addition, deterministic imputation by CSDI reduces the error by 5-20% compared to the state-of-the-art deterministic imputation methods. Furthermore, CSDI can also be applied to time series interpolation and probabilistic forecasting, and is competitive with existing baselines. The code is available at https://github.com/ermongroup/CSDI.

1 Introduction

CSDI applies conditional score-based diffusion models to time-series imputation, using observed values during denoising and self-supervised training. It improves probabilistic and deterministic imputation results and extends to interpolation and forecasting.

  • Contribution: CSDI directly learns conditional distributions for probabilistic imputation using diffusion models conditioned on observed values.The method is explicitly designed for imputation rather than adapting an unconditional generative model.
  • Contribution: Self-supervised training separates observed values into conditional information and imputation targets when ground-truth missing values are unavailable.This design supports training even when datasets contain no naturally missing values.
  • Results: 40-65% CRPS improvement over existing probabilistic methods is reported on healthcare and environmental data.The comparison concerns continuous ranked probability score (CRPS).
  • Results: 5-20% MAE reduction is reported for deterministic imputation versus state-of-the-art deterministic methods.The metric is mean absolute error (MAE).
  • Results: CSDI is also competitive with existing baselines for time-series interpolation and probabilistic forecasting.These are additional applications beyond missing-value imputation.

2 Related works

Prior work largely uses recurrent deep-learning models for time-series imputation, while score-based generative models have shown strong results in other domains and forecasting.

  • Time series imputations with deep learning: RNNs, including LSTMs and GRUs, are widely used for sequence modeling in deep-learning imputation methods.Later approaches combine RNNs with GANs, self-training, or attention mechanisms.
  • Score-based generative models: Score-based generative models have outperformed other deep generative methods in domains including images, audio, and graphs.The cited methods include score matching with Langevin dynamics and denoising diffusion probabilistic models.
  • Score-based generative models: TimeGrad achieves state-of-the-art probabilistic forecasting performance but cannot handle time-series imputation because it uses RNNs for past series.This highlights a task-specific limitation of an otherwise strong diffusion-based forecasting method.

3 Background

The paper formulates imputation as conditional distribution estimation and reviews diffusion models as noisy forward processes followed by learned reverse denoising. CSDI conditions this reverse process on observations.

  • 3.1 Multivariate time series imputation: A multivariate time series is represented by values X, an observation mask M, and timestamps s.The mask identifies missing and observed entries, while timestamps allow unequal intervals between consecutive entries.
  • 3.1 Multivariate time series imputation: Probabilistic imputation estimates the distribution of missing values from observed values, including interpolation and forecasting as related cases.Interpolation targets all features at selected times, whereas forecasting targets all features at future times.
  • 3.2 Denoising diffusion probabilistic models: Diffusion models use a forward Markov process that adds noise and a reverse process that denoises latent variables to recover data.The forward process uses β_t as a noise-level parameter, and the reverse process is learned.
  • 3.2 Denoising diffusion probabilistic models: In DDPMs, a trainable denoising function estimates the noise added to a noisy input and supports reverse-process sampling.The objective can also be viewed as a weighted combination of denoising score matching.
  • 3.3 Imputation with diffusion models: Imputation seeks the conditional distribution of target missing values given conditional observed values.All observed values are typically treated as conditioning information and all missing values as targets.
  • 3.3 Imputation with diffusion models: Existing diffusion-based imputation approximations add noise to both target and conditional observations, potentially harming useful observed information.CSDI instead uses a conditional reverse process that accepts observations as inputs.

4 Conditional score-based diffusion model for imputation (CSDI)

CSDI directly models the conditional distribution of imputation targets with a conditional score-based diffusion model. It trains this model self-supervised by separating observed values into conditional observations and imputation targets, with target-selection strategies adapted to missingness patterns.

  • CSDI directly learns the conditional distribution of imputation targets with a conditional score-based diffusion model.
  • During sampling, the reverse process uses all observed values as conditional observations and all missing values as imputation targets.
  • The conditional denoising function is trained by adding noise to imputation targets and minimizing a denoising objective conditioned on observed values.
  • Self-supervised training separates observed values into conditional information and imputation targets because ground-truth missing values may be unavailable.
  • Four target-selection strategies address different knowledge of test missingness: random, historical, mix, and test pattern.

5 Implementation of CSDI for time series imputation

CSDI adapts a diffusion architecture to multivariate time series by using two-dimensional attention and fixed-size padded inputs. Temporal and feature dependencies are modeled separately, with masks and side information identifying observations and sequence structure.

  • Zero padding maps variable-sized imputation targets and conditional observations into the fixed sample space R^K×L.
  • The conditional mask m_co indicates which indices are conditional observations and identifies padded positions for the denoising function.
  • For sampling, the model recovers the original target shape by extracting imputation-target indices from the padded output.
  • Two-dimensional attention uses temporal and feature Transformer layers to learn dependencies across time and across features.
  • CSDI adds 128-dimensional time embeddings and 16-dimensional categorical feature embeddings as side information.

6 Experimental results

CSDI is evaluated for probabilistic and deterministic imputation, interpolation, and probabilistic forecasting across healthcare, air quality, and broader forecasting datasets. It improves substantially over baselines for imputation and interpolation, while remaining competitive for forecasting.

  • Probabilistic imputation: CSDI reduces CRPS by 40-65% versus existing probabilistic imputation baselines on healthcare and air quality datasets.The evaluation reports normalized average CRPS using 100 generated samples and five trials.
  • Probabilistic imputation: CSDI provides accurate, high-confidence air-quality imputations and reasonable healthcare imputations, with median predictions and uncertainty quantiles illustrated in Figure 4.The air-quality examples are closer to ground-truth than GP-VAE examples.
  • Deterministic imputation: CSDI improves deterministic-imputation MAE by 5-20% compared with deterministic baselines.The deterministic prediction is obtained as the median of 100 generated samples.
  • Interpolation: CSDI outperforms the interpolation baselines in all evaluated cases on the healthcare dataset.Interpolation uses irregularly sampled healthcare time series and 100 generated samples to approximate the probability distribution.
  • Probabilistic forecasting: For probabilistic forecasting, CSDI outperforms baselines on electricity and traffic datasets and is competitive overall, with a smaller advantage than for imputation.Forecasting is evaluated with CRPS-sum across five datasets.

7 Conclusion

The paper concludes that CSDI uses conditional diffusion models for multivariate time-series imputation and outperforms existing probabilistic and deterministic methods. Future work includes improving sampling efficiency, extending the method to downstream tasks and other modalities, and developing joint training approaches.

  • Conclusion: CSDI is a conditional-diffusion approach for imputing multivariate time series that outperforms existing probabilistic and deterministic imputation methods.The paper presents this as its main conclusion.
  • Future work: Diffusion-model sampling is generally slower than sampling in other generative models, motivating future work on computational efficiency.The paper identifies ODE-based acceleration as a possible direction.
  • Future work: The authors propose extending CSDI to downstream tasks such as classification through joint training of imputation and downstream objectives.They connect this direction to conditional diffusion models' ability to learn temporal and feature dependencies with uncertainty.
  • Future work: The paper also identifies imputation on modalities beyond time series as an area for future exploration.This is stated as a possible extension of the method's application scope.

B.2 Target choice strategies for self-supervised training

CSDI uses self-supervised target-selection strategies to separate conditional observations from values that the model must impute. Strategies vary according to available knowledge about test-time missingness patterns.

  • Target choice strategies: CSDI defines four target-selection strategies for self-supervised training based on knowledge of missing patterns in the test dataset.The strategies are described as random, historical, mix, and test-pattern approaches.
  • Random strategy: The random strategy samples a target ratio uniformly from 0 to 100 and randomly selects that percentage of observed values as imputation targets.Remaining observed values become conditional information.
  • Historical strategy: The historical strategy selects targets by intersecting observed indices in a training sample with missing indices from a sampled missing-pattern dataset.All remaining observations are assigned as conditional information.
  • Mix strategy: The mix strategy chooses the random or historical strategy with equal probability for each training sample.This combines the two target-selection procedures at a 1:1 ratio.
  • Test-pattern strategy: The test-pattern strategy uses the fixed missing pattern in the test dataset to choose imputation targets.This strategy directly reflects the missingness pattern used during evaluation.

C.2 Training procedure of unconditional diffusion models for time series imputation

The unconditional diffusion imputation procedure adapts training data with missing values by inserting dummy zeros and masking them during noise estimation. Only original imputation targets contribute to the training loss.

  • Handling missing values: Training samples with missing values are padded with dummy values, using zeros at missing indices to provide fixed-shape inputs.The missing values are treated like observed values for constructing the model input.
  • Masked training: An observation mask m identifies observed indices so training excludes noise estimation at positions containing dummy values.The loss is modified to ignore missing indices because dummy values contain no information about the data distribution.

D CSDI for implementation of time series imputation

CSDI’s implementation adjusts diffusion sampling and training for time-series imputation while using conditional observations, imputation targets, temporal information, and attention-based processing. The implementation also masks conditional-observation outputs and uses padding-aware attention when needed.

  • Sampling: During sampling, the model distinguishes conditional observations from imputation targets through the conditional mask, then extracts target indices to recover the original output shape.The adjustment does not essentially change the sampling procedure because every value belongs to one of these two categories.
  • Training: During training, missing indices receive zero dummy values and masked noise, while the model estimates noise only for original imputation targets.This extends the target input so missing values can be handled without treating dummy values as informative data.
  • Architecture: The implementation uses a DiffWave-based residual architecture refined for time-series imputation, with conditional observations and imputation targets as inputs.The architecture is described in Figure 6 and uses a 1-layer TransformerEncoder for attention processing.
  • Embeddings: CSDI represents diffusion steps and temporal positions with 128-dimensional embeddings and concatenates the resulting side information.The temporal embedding uses the time series positions as additional information.
  • Output masking: The model multiplies its output by (1 − mco) to mask the indices corresponding to conditional observations.This prevents conditional-observation positions from contributing to the generated imputation output.
  • Evaluation setup: Baseline evaluations use original implementations and hyperparameters, while equal-parameter experiments did not outperform CSDI in any case.The paper also reports that matching model size failed to improve performance in more than half of the cases.

E.2.1 Experiment settings for imputation in Section 6.1

The imputation experiments use healthcare and air-quality data with task-specific splits, normalization, baselines, and diffusion settings. Ablations report that the proposed attention mechanism benefits imputation by modeling both temporal and feature dependencies.

  • Datasets and splits: The air-quality dataset contains Beijing measurements from 2014/05/01 to 2015/04/30 with artificial ground truth based on subsequent-month missing patterns.These patterns support evaluation against known values removed according to realistic missingness structures.
  • Datasets and splits: Healthcare data uses random five-part test splits and 7:1 train-validation splits, while air-quality testing uses the 3rd, 6th, 9th, and 12th months.Air-quality test data is further separated into 36-consecutive-time-step segments.
  • Preprocessing: Healthcare evaluation uses normalized data, and all task features are normalized to zero mean and unit variance during training.Normalization addresses different feature scales in the healthcare dataset.
  • Hyperparameters: The main model uses batch size 16, 200 epochs, Adam with a learning rate decayed from 0.001, four residual layers, 64 residual channels, and eight attention heads.The learning rate decays to 0.0001 and 0.00001 at 75% and 90% of training, respectively.
  • Diffusion settings: The diffusion process uses T = 50, β1 = 0.0001, βT = 0.5, and a quadratic schedule for intermediate noise levels.The quadratic schedule was selected because prior studies reported that gentle αt decay could improve sample quality.
  • Attention ablation: The ablation reports that the proposed attention mechanism outperforms alternative architectures when parameter counts are approximately matched.Comparisons indicate that both temporal and feature correlations are important for accurate imputation.
  • Evaluation metrics: The paper emphasizes CRPS rather than NLL because ELBO can diverge from NLL and become uncorrelated with generated-sample quality.For CSDI, noise-schedule choice strongly affects NLL while having little effect on sample quality and CRPS.

F.3 Experimental results for other metrics in Section 6

CSDI outperforms baselines across additional deterministic, interpolation, and probabilistic forecasting evaluations, while its sampling performance improves mainly up to about 50 samples. Qualitative examples compare its imputations with GP-VAE and an unconditional diffusion model across missingness levels and datasets.

  • Deterministic imputation: CSDI outperforms all deterministic imputation baselines on RMSE, with its advantage particularly large at low missing ratios.The results are reported in Table 9 and are consistent with earlier experiments.
  • Interpolation and forecasting: CSDI is competitive with baselines for interpolation on MAE and RMSE and for probabilistic forecasting on CRPS and MSE.The interpolation results are reported in Table 10, while forecasting results are reported in Tables 11 and 12.
  • Number of generated samples: Five or ten generated samples are sufficient to estimate good imputation distributions and outperform baselines, while gains become marginal beyond 50 samples.The experiments used 100 samples to estimate imputation distributions.
  • Target-choice strategy: For air quality data, the mix and random target-choice strategies perform almost identically, whereas the historical strategy performs slightly worse.The reported result indicates that the historical strategy is not effective for this dataset.
  • Qualitative imputation examples: Qualitative figures compare CSDI with GP-VAE or an unconditional diffusion model using medians and 5%–95% quantile bands against observed values and ground-truth targets.The comparisons cover healthcare and air quality datasets, including healthcare missing ratios of 10%, 50%, and 90%.
Loading 2107.03502v2…