PaperlayerView exact revision on arXiv

Source-linked AI summary

RDDMPI: Residual Denoising Diffusion Model for Probabilistic Multivariate Time Series Imputation

Ramiro Valdes Jara, David Chapman, Adam Meyers

arXiv:2609.11648v1cs.LGstat.ML

TL;DR

RDDMPI addresses probabilistic multivariate time series imputation by separating deterministic reconstruction from residual uncertainty modeling. It uses conditional residual diffusion with reliability-aware baseline guidance, achieving strong reconstruction and uncertainty-quantification results across five benchmark datasets, at the cost of substantial computation.

  • Problem

    Baseline imputations vary in quality across datasets, missingness patterns, and time-series regions, so conditioning should adapt to estimated reliability.

  • Method

    RDDMPI combines a pretrained deterministic baseline and latent representation with residual-space diffusion, using reliability-aware conditioning to guide denoising.

  • Results

    Across five benchmark datasets, RDDMPI achieves the best reconstruction result in 18 of 20 aggregated MSE and MAE comparisons and the lowest CRPS in eight of 10 probabilistic comparisons.

  • Takeaways & Limitations

    RDDMPI’s residual formulation and adaptive baseline fusion support strong reconstruction and uncertainty quantification across diverse missingness patterns.

  • Takeaways & Limitations

    RDDMPI requires a separately pretrained deterministic backbone and NT denoiser evaluations to generate N stochastic samples through T reverse steps.

Abstract

from arXiv · show

Multivariate time series imputation (MTSI) aims to recover missing values in temporal data composed of multiple interdependent variables. This problem is central to real-world applications such as healthcare monitoring, traffic networks, and energy systems. Recent diffusion-based approaches have shown strong potential for probabilistic imputation by learning to generate missing values through iterative denoising. However, most existing approaches perform diffusion directly in the original data space, requiring the denoising network to simultaneously capture global structure, temporal dynamics, and stochastic variability. This makes the generative task unnecessarily complex, especially when modern deterministic imputers can already provide accurate initial reconstructions. To address this limitation, we propose RDDMPI, a conditional residual diffusion framework that operates directly in residual space. Instead of modeling the full missing signal directly, we reformulate probabilistic imputation as a baseline-residual decomposition, where a pretrained model captures the dominant signal and a diffusion process models the residual uncertainty. To better exploit deterministic guidance, \model{} conditions the reverse denoising process on both the baseline-completed signal and its latent representation, while a reliability-aware conditioning mechanism adaptively controls the influence of baseline information during residual generation. This formulation simplifies the diffusion learning objective, enabling it to focus on structured correction terms rather than reconstructing the full signal. Experiments on multiple benchmark datasets demonstrate that RDDMPI consistently improves both reconstruction accuracy and uncertainty quantification.

I. INTRODUCTION

MTSI recovers missing values in interdependent temporal data, but diffusion methods that generate the full missing signal must jointly learn structure and uncertainty. RDDMPI instead decomposes imputation into deterministic baseline reconstruction and residual diffusion with adaptive conditioning.

  • RDDMPI addresses MTSI, where missing observations arise in interdependent temporal data and can degrade forecasting, anomaly detection, decision support, and monitoring.
  • Existing deterministic methods produce single estimates, while probabilistic methods model distributions because multiple plausible completions may fit the observed sequence.
  • Diffusion methods typically reconstruct the full missing signal, forcing one denoiser to learn global structure, temporal dynamics, cross-variable dependencies, and stochastic variability.
  • RDDMPI separates deterministic reconstruction from probabilistic residual modeling, using a pretrained imputer for the baseline and diffusion for correction terms over missing regions.
  • The framework conditions denoising on the baseline signal and latent representation, while reliability-aware fusion adapts baseline influence when estimates may be inaccurate.
  • RDDMPI reports state-of-the-art performance in most settings across five benchmark datasets, improving reconstruction accuracy and uncertainty quantification over deterministic and probabilistic baselines.

IV. METHODOLOGY

RDDMPI reformulates probabilistic imputation as conditional diffusion over missing-region residuals around a deterministic baseline. The method progressively corrupts residual targets, then learns their conditional distribution using baseline signals, latent representations, and masks.

  • A. Baseline-Residual Decomposition: RDDMPI decomposes imputation into deterministic baseline reconstruction followed by probabilistic modeling of residual corrections in missing regions.The pretrained baseline produces a completed signal and latent representation; diffusion models corrections capturing baseline errors and conditional variability.
  • A. Baseline-Residual Decomposition: The residual target is defined only at missing positions, so diffusion corrects the baseline without altering observed entries.The final imputed series preserves observed values while adding sampled residual corrections to missing positions.
  • B. Conditional Residual Diffusion: RDDMPI learns the conditional distribution of plausible residual corrections around the deterministic baseline using a DDPM adapted to residual space.The forward process generates noisy residual states, while the reverse process is trained to recover residual corrections conditioned on baseline information.
  • 1) Forward Diffusion Process:: The forward diffusion process progressively adds Gaussian noise to the clean missing-region residual across T steps until the residual approaches Gaussian noise.The noise schedule controls variance and cumulative signal retention during corruption.
  • 1) Forward Diffusion Process:: During training, noise is added only to missing-region residuals, and the denoiser is conditioned on the completed baseline, its latent representation, timestep, and mask.The model predicts masked noise using these conditioning inputs rather than reconstructing the full signal directly.

2) Reverse Diffusion Process:

The reverse process denoises residuals conditionally, using a learned noise predictor to generate multiple plausible corrections around the deterministic baseline. RDDMPI combines reliability-aware fusion, latent FiLM conditioning, residual blocks, and masked self-supervised training.

  • 2) Reverse Diffusion Process:: The reverse chain starts from Gaussian noise and progressively removes noise using a conditional mean parameterized by the predicted noise network.The reverse variance is fixed according to the diffusion noise schedule, following standard DDPM parameterization.
  • 2) Reverse Diffusion Process:: The denoising network receives the noisy residual, diffusion timestep, and conditioning information, while the injected noise serves as the training target rather than an input.Optimization is restricted to missing entries so observed values already fixed by conditioning are not penalized.
  • 2) Reverse Diffusion Process:: RDDMPI samples multiple reverse diffusion trajectories to approximate the conditional residual distribution, then adds each residual sample to the baseline at missing positions.The complete sample set supports probabilistic evaluation and uncertainty quantification, while its element-wise median provides the point estimate.
  • C. Conditional Denoising Network: The conditional denoising architecture combines reliability-aware input fusion, FiLM conditioning from the baseline latent representation, and residual denoising blocks.These components adaptively incorporate deterministic baseline information while estimating noise in the current residual state.
  • 2) Reverse Diffusion Process:: Algorithm 1 initializes missing-region residuals with Gaussian noise, repeatedly predicts noise and samples reverse steps, then recovers missing values by adding the final correction to the baseline.The algorithm uses a pretrained baseline, trained denoiser, observation mask, and N stochastic samples.

1) Reliability-Aware Conditioning:

RDDMPI incorporates the noisy residual as the primary denoising state while injecting baseline information through reliability-weighted signal fusion and latent conditioning. A learned gate adapts this influence across variables and timesteps.

  • 1) Reliability-Aware Conditioning:: The noisy missing-region residual defines the primary denoising state, while the baseline-completed signal is projected separately and fused through reliability weighting.The resulting hidden representation is subsequently modulated by latent conditioning before residual denoising blocks process it.
  • 1) Reliability-Aware Conditioning:: A learned reliability gate adaptively controls how strongly the baseline reconstruction influences denoising across variables and timesteps.The gate is learned from temporal convolutions and sigmoid activation rather than explicitly supervised baseline-error labels.
  • 1) Reliability-Aware Conditioning:: RDDMPI aligns the deterministic baseline’s latent representation with variable and temporal dimensions before projecting it into FiLM scale-and-shift tensors.This conditioning injects baseline structure independently across channels, variables, and timesteps.

3) Residual Denoising Blocks:

RDDMPI’s residual denoising blocks process noisy missing-region residuals with diffusion-step, temporal, variable, and side-information conditioning. The residual formulation is theoretically motivated as a simpler correction-focused learning target when the baseline approximates the conditional mean.

  • 3) Residual Denoising Blocks:: Each block forwards a residual update while accumulating skip representations for final projections that predict Gaussian noise with the residual’s variable-temporal dimensions.The output path uses two pointwise projections with an intervening ReLU activation.
  • 3) Residual Denoising Blocks:: Residual denoising blocks add diffusion-step embeddings, model temporal and cross-variable dependencies sequentially, and incorporate positional, variable, and mask side information.Temporal transformers operate across timesteps per variable, while variable transformers model interactions among variables at each timestep.
  • 3) Residual Denoising Blocks:: Residual diffusion is equivalent to standard conditional diffusion under a change of variables, with its practical advantage arising from removing much of the conditional mean.The score-based analysis states that an accurate baseline yields a lower-energy, more correction-oriented target for finite-capacity denoisers.
  • 3) Residual Denoising Blocks:: When the deterministic baseline approximates the conditional mean, residual diffusion targets remaining variation rather than relearning the missing signal’s mean structure.The residual contains intrinsic conditional uncertainty and baseline squared bias, making the target more correction-oriented when baseline bias is small.

A. Evaluation Setup

RDDMPI is evaluated on five diverse multivariate time-series benchmarks using point-wise and structured block missingness. The study compares against ten baselines with MAE, MSE, and CRPS under a reproducible multi-seed protocol.

  • A. Evaluation Setup: RDDMPI is evaluated on ETTh1, ETTh2, Weather, Exchange, and Illness, spanning energy, climate, finance, and public-health data.All datasets use an input window length of 96.
  • A. Evaluation Setup: The protocol tests both independent point missingness at ratios 0.2, 0.4, 0.6, and 0.8 and structured block missingness simulating sensor failures.Experiments use five random seeds and evaluate generalization across observability regimes.
  • A. Evaluation Setup: The comparison includes ten baselines spanning general-purpose time-series models, specialized deterministic imputers, and specialized probabilistic imputers.Examples include DLinear, ModernTCN, iTransformer, TimesNet, SAITS, ImputeFormer, T1, GP-VAE, CSDI, and FGTI.
  • A. Evaluation Setup: Reconstruction is measured with MAE and MSE, while probabilistic quality is evaluated with CRPS using 100 generated samples per masked position.MAE averages error, MSE emphasizes large deviations, and CRPS evaluates predictive accuracy and distributional quality.
  • A. Evaluation Setup: Table I reports performance across five datasets for point and block missing scenarios, averaging results over four point-missing ratios.Best and second-best results are marked with bold and underlining, respectively.

5) Implementation Details:

RDDMPI uses frozen deterministic backbones and residual diffusion to improve deterministic reconstruction and probabilistic uncertainty estimates across missingness regimes. Its gains are strongest when baseline reconstruction is difficult, while adaptive reliability-aware conditioning limits errors from unreliable baseline guidance.

  • Implementation Details: RDDMPI combines a pretrained frozen deterministic backbone with a DDPM residual model that generates 100 samples for uncertainty evaluation.The backbone supplies the baseline-completed signal and latent representation; median samples support deterministic metrics, while the full sample set supports CRPS and predictive intervals.
  • Reconstruction Accuracy: RDDMPI achieves the best reconstruction in 18 of 20 MSE and MAE comparisons and improves over T1 in 19 of 20 comparisons.Under block missingness, it outperforms T1 in both metrics on all five datasets.
  • Probabilistic Uncertainty Quantification: RDDMPI attains the lowest CRPS in eight of ten dataset and missingness settings, including all five datasets under point missingness.FGTI is best on Illness under block missingness, while CSDI is marginally better on Weather under block missingness.
  • Qualitative Comparison: Qualitative examples show RDDMPI medians following ground-truth targets more closely and producing narrower predictive intervals than CSDI on ETTh2 and Exchange.The intervals remain centered around the reconstructed trajectory and contain most target values in the examples.
  • Effect of the Deterministic Backbone: RDDMPI improves both MSE and MAE with either T1 or ImputeFormer, although improvement magnitude depends on the backbone’s quality and error structure.The gains with ImputeFormer are larger because its baseline errors leave more substantial correction terms.
  • Component Ablation: Ablations show that conditioning is not uniformly beneficial: residual diffusion suffices in simpler regimes, while reliability-aware fusion helps when baseline errors are more consequential.The complete model performs best at point-0.8 and under block missingness, where baseline information is less dependable.
  • Limitations: The framework’s main limitation is computational cost because stochastic sampling requires repeated denoiser evaluations in addition to pretraining the deterministic backbone.The paper proposes accelerated sampling and joint training as future directions.

APPENDIX A THEORETICAL ANALYSIS OF RESIDUAL DIFFUSION

The appendix analyzes residual diffusion through conditional score functions and compares practical residuals with ideally centered residuals. Under Lipschitz regularity, the discrepancy is controlled by baseline approximation error and attenuated by the diffusion coefficient.

  • Theoretical Analysis: Under Lipschitz regularity, the practical residual-score discrepancy is controlled by the baseline error ∥f(c) − m(c)∥.An accurate baseline approximation to the conditional mean keeps the induced residual score close to the optimal centered score.
  • Theoretical Analysis: The influence of baseline error on the residual score is modulated by the diffusion coefficient √¯α_t and diminishes as diffusion progresses.This links the theoretical approximation gap to the noise schedule.
  • Theoretical Analysis: Residual diffusion produces a score field closer to the ideal centered representation as noise increases, simplifying learning for finite-capacity denoisers.The appendix characterizes this as a more stable and favorable learning problem.

APPENDIX B MATHEMATICAL DETAILS OF FORWARD PROCESS DERIVATION OF RESIDUAL DIFFUSION

The appendix derives the closed-form noisy residual used by RDDMPI from the one-step forward diffusion transition. Recursive expansion and Gaussian noise aggregation yield the final forward noising expression.

  • Forward Process Derivation: The derivation recursively expands the one-step forward transition to obtain the noisy residual at diffusion step t.The appendix starts from Eq. (10) and derives the expression used in Eq. (13).
  • Forward Process Derivation: Independent Gaussian noise terms remain Gaussian after linear combination, with variance equal to the sum of their scaled individual variances.This aggregation produces the closed-form variance in the forward process.
  • Forward Process Derivation: The resulting closed-form forward noising expression is the one used by the residual diffusion model.The appendix identifies this as the final outcome of the derivation.

APPENDIX C RDDMPI TRAINING PROCEDURE

RDDMPI trains a conditional diffusion model on missing-region residuals while keeping the baseline completion and latent representation fixed as conditioning. Its sampling process trades higher inference cost for improved imputation accuracy, with reduced-step sampling offering a possible efficiency compromise.

  • Training procedure: RDDMPI constructs residual targets only in the missing region, injects Gaussian noise, and trains a conditional predictor to recover that noise.The baseline-completed signal, latent representation, and observation mask remain fixed as conditioning information; the denoising loss is computed only on missing positions.
  • Training procedure: The training algorithm samples windows and diffusion steps, masks observed entries, computes baseline completions, then noises and denoises the residual target.The deterministic baseline is pretrained separately and remains frozen during residual diffusion training.
  • Efficiency analysis: RDDMPI achieves better imputation accuracy than deterministic baselines on ETTh1 and Illness, but has substantially higher inference cost.Table V compares parameters, training speed, inference speed, and MAE averaged over point-missing ratios.
  • Efficiency analysis: Inference requires N independent T-step reverse trajectories to approximate the predictive distribution, totaling NT denoising-network evaluations.Trajectories can be batched, but denoising steps within each trajectory remain sequential.
  • Efficiency analysis: RDDMPI maintains competitive performance with substantially fewer DDIM inference steps, indicating a potential route toward narrowing its efficiency gap.DDIM selects K timesteps with K ≪ T for accelerated sampling.
  • Efficiency analysis: Future work targets adaptive timestep selection and one-step diffusion models to reduce computational cost while preserving diffusion-based imputation performance.

A. Diffusion-Step Conditioning

The residual denoising network conditions on the diffusion step through a sinusoidal embedding and models temporal and cross-variable structure with separate attention modules. Its side information combines position, variable identity, observation status, and reliability.

  • Diffusion-Step Conditioning: A fixed sinusoidal embedding represents the diffusion step and is projected into the hidden dimension before being added across variables and timesteps.The resulting representation carries both residual features and information about the current noise level.
  • Diffusion-Step Conditioning: Temporal self-attention models dependencies across timesteps independently for each variable, while variable self-attention models cross-variable interactions independently at each timestep.Both modules use multi-head self-attention with residual connections, normalization, and feed-forward layers.
  • Diffusion-Step Conditioning: Side information includes temporal-position embeddings, learned variable embeddings, the observation mask, and a learned reliability map.The side-information dimension is dside = dtime + dvar + 2.
  • Diffusion-Step Conditioning: The reliability map influences the network twice: during baseline-signal fusion and as side information in every residual denoising block.

D. Gated Activation

Each residual denoising block combines diffusion-step conditioning, temporal and variable attention, side information, gated nonlinear processing, and residual/skip projections. The skip outputs are aggregated into the final predicted noise field.

  • Gated Activation: The conditioned hidden representation is split into two parts, and a sigmoid gate modulates nonlinear candidate features before pointwise output projection.The element-wise product controls information propagation within each block.
  • Gated Activation: Each block produces a residual update and a skip connection, scales the residual path, and passes the updated representation to the next block.The skip output is retained for the final noise-prediction path.
  • Gated Activation: Skip representations from all residual blocks are summed and normalized, then passed through two pointwise projections to produce one predicted noise value per variable-timestep position.The final prediction has the same variable and temporal dimensions as the noisy residual.
  • Gated Activation: The network sequentially injects the diffusion embedding, applies temporal and cross-variable attention, fuses side information, gates features, and splits output into residual and skip paths.

APPENDIX F DATASET-SPECIFIC RDDMPI HYPERPARAMETERS

RDDMPI uses dataset-specific optimization and architectural hyperparameters, with a frozen pretrained T1 backbone. The appendix reports full point- and block-missing results, variability across five seeds, CRPS comparisons, and qualitative ETTh2 visualizations.

  • APPENDIX F DATASET-SPECIFIC RDDMPI HYPERPARAMETERS: RDDMPI hyperparameters vary by dataset and include batch size, learning rate, residual-block count, hidden channels, and attention heads.The deterministic T1 backbone is pretrained separately for each dataset and frozen during residual diffusion training.
  • Full results: Full quantitative results cover point-missing ratios 0.2, 0.4, 0.6, and 0.8, as well as block missingness across datasets.The main paper aggregates point-missing results across ratios, whereas the appendix reports them separately.
  • Full results: Results are averaged over five random seeds, with standard deviations quantifying sensitivity to initialization, training stochasticity, and generation randomness.
  • Full results: The appendix provides full CRPS comparisons under point and block missingness, with lower CRPS marked as better.
  • Qualitative results: ETTh2 visualizations show median predictions and 90% predictive intervals for 20%, 40%, 60%, 80%, and block missingness across all seven features.Observed points are red, targets blue, and predictive summaries green.
Loading 2609.11648v1…