PaperlayerView exact revision on arXiv

Source-linked AI summary

Discretizing Continuous Time Series for Imputation with Masked Diffusion Training

Dongbin Kim, Seungyun Lee, Geonwoo Shin, Jaewook Lee

arXiv:2608.19119v1cs.LGcs.AI

TL;DR

Time series imputation must recover clean signals from partial observations, but existing methods conflate missing and observed representations and often predict noise instead of original values. MDTIM uses masked diffusion with stochastic discretization and ordinal-aware modeling, outperforming deterministic and generative baselines across diverse missing scenarios while requiring less inference time than continuous diffusion baselines.

  • Problem

    Existing imputation methods lack explicit separation between missing and observed representations and often optimize noise prediction rather than direct signal recovery.

  • Method

    MDTIM combines masked diffusion with stochastic discretization, ordinal-aware soft labeling, spectral consistency, and expectation-based unmasking for continuous reconstruction.

  • Results

    MDTIM consistently outperforms deterministic and generative baselines in reconstruction accuracy and robustness across diverse missing scenarios, with substantially less inference time than continuous diffusion baselines.

  • Takeaways & Limitations

    MDTIM provides accurate and robust imputation across diverse missing scenarios while reducing inference time relative to continuous diffusion baselines.

  • Takeaways & Limitations

    Evaluation focuses on moderate-length standard benchmarks, leaving extremely long sequences and multi-scale or non-stationary domains for future work.

Abstract

from arXiv · show

Time series imputation is a crucial area for reliable time series analysis, yet it remains challenging due to the complex temporal dynamics and noise of real-world data. Existing approaches, however, exhibit two limitations: missing and observed values are embedded within the same representation space without explicit structural separation, and continuous diffusion-based methods are trained to predict added noise rather than the original signal. To address these, we propose the Masked Diffusion Time-series Imputation Model (MDTIM), which leverages the training paradigm of masked diffusion model for imputation tasks. The MASK token is structurally orthogonal to valid observations, and the model directly predicts the original values, naturally aligning both the representation and the learning objective with the imputation task. To bridge the gap between discrete masked diffusion and the continuous, ordinal nature of time series, we further introduce Stochastic Discretization, which maps continuous values to ordinal-aware tokens while preserving continuous dynamics. Our experiments on diverse benchmarks confirm that MDTIM achieves superior robustness and scalability, consistently outperforming state-of-the-art deterministic and generative baselines across various missing scenarios.

1 Introduction

The introduction frames time-series imputation as recovering clean signals from partial observations, while identifying misalignment in existing representations and objectives. MDTIM addresses this with structurally separated masking, direct signal prediction, and stochastic ordinal-aware discretization for continuous time series.

  • Motivation: Time-series imputation recovers clean signals from partial observations caused by sensor failures or transmission errors, requiring aligned representations and learning objectives.
  • Limitations: Existing Transformer and continuous diffusion methods embed missing and observed values in the same data manifold, allowing placeholders to share space with valid observations.Attention masks may block attention to missing positions, but placeholder values such as zeros still occupy the valid-observation embedding space.
  • Masked Diffusion: Masked diffusion separates missing states with a structurally orthogonal [MASK] token and directly predicts original values instead of added noise.This representation and objective more closely match the imputation goal.
  • MDTIM: MDTIM bridges discrete masked diffusion and continuous time series through Stochastic Discretization and Ordinal-Aware Soft Labeling.Stochastic Discretization injects noise during tokenization to preserve information in expectation, while soft labels capture ordinal relationships between tokens.
  • Contributions: Expectation-based Unmasking and Spectral Consistency Regularization enable precise continuous reconstruction, with MDTIM consistently outperforming state-of-the-art deterministic and generative baselines across diverse benchmarks.

2 Related Work

Time-series imputation addresses missing observations caused by practical data-collection issues by exploiting temporal, cross-variable, and missingness-pattern information. Related methods span masked RNNs, self-attention Transformers, probabilistic diffusion models, and discrete diffusion frameworks, while highlighted limitations concern reconstruction objectives and masking-ratio weighting.

  • Time-Series Imputation: Time-series data become partially observed through sensor faults, irregular sampling, and data-collection constraints, making imputation necessary for reliable downstream analysis.Naive deletion, mean imputation, and zero imputation can bias estimation and degrade downstream tasks.
  • Continuous Imputation Models: Deep imputation methods have progressed from masked RNN architectures to self-attention Transformers, with SAITS using Diagonally-Masked Self-Attention.SAITS is identified as achieving strong performance through its diagonally masked attention mechanism.
  • Continuous Imputation Models: Diffusion-based models such as CSDI and SSSD generate probabilistic imputations by conditioning on observed data and performing iterative denoising.These methods extend deterministic imputation with probabilistic reconstruction.
  • Limitations: Existing continuous diffusion training predicts “what noise was added” rather than “what was originally there” and applies uniform weighting regardless of reconstruction difficulty.The objective is also described as decoupled from the masking ratio.
  • Discrete Diffusion Models: Discrete diffusion models extend diffusion to categorical states, with D3PM progressively replacing tokens by [MASK] and MDLM adapting the framework to language modeling.MDLM uses a continuous-time weighted cross-entropy objective and provides a non-autoregressive alternative for conditional text generation.

3 Discrete Representation for Time-Series Modeling

MDTIM bridges continuous time series and discrete masked diffusion through instance-adaptive normalization, stochastic token generation, and ordinal-aware soft labeling. The discretization preserves continuous dynamics in expectation while structurally separating valid ordinal targets from the [MASK] state.

  • Instance-Adaptive Normalization: Instance-adaptive normalization computes statistics only from observed values, excluding masked positions to prevent data leakage.Each input window is normalized using local observed-value statistics, stabilizing representations under distribution shifts.
  • Stochastic Discretization: Stochastic token generation maps normalized continuous values to a vocabulary of size K by adding uniform noise before rounding.The injected noise follows ϵ ∼ U(−0.5, 0.5), producing discrete tokens while mitigating fixed-grid quantization effects.
  • Stochastic Discretization: 60% probability assigns coordinate 3.6 to token 4 and 40% to token 3, yielding E[z_t,c] ≈ c_t,c.This stochastic assignment preserves information in expectation and enables learning dynamics beyond the grid resolution.
  • Ordinal-Aware Soft Labeling: Ordinal-aware soft labeling distributes probability across nearby token classes instead of treating discretized tokens as independent categories.A truncated Gaussian kernel encodes that ordinally adjacent tokens are more semantically similar than distant bins.
  • Ordinal-Aware Soft Labeling: The target assigns zero probability to the [MASK] token and distant bins, concentrating mass on valid ordinal neighbors while preserving mask-state separation.For example, the truncation window may use w = 2, with σ controlling smoothness and Z providing normalization.

4 Masked Diffusion Framework

MDTIM models the joint distribution of multivariate time series with a factorized temporal-variate Transformer and structurally distinct channel embeddings. Its training combines ordinal-aware masked diffusion with spectral consistency, while inference averages stochastic predictions before expectation-based value recovery.

  • Architecture: MDTIM models the joint distribution of multivariate time series using a Factorized Temporal-Variate Transformer based on DiT, alternating processing of time and feature axes.The architecture processes temporal and feature dimensions through alternating factorization.
  • Architecture: Distinct channel-specific embedding matrices preserve semantic orthogonality when projecting masked input indices into the initial hidden state.Each channel c uses its own embedding matrix Ec, producing H0 ∈ R^(B×T×C×D).
  • Training objective: The dual-domain objective combines ordinal-aware masked diffusion, which aligns predictions with ordinal neighborhoods, and spectral consistency regularization for global temporal and frequency coherence.The total objective is Ltotal = Ldiff + λLFFT.
  • Inference: Inference runs M independent noise realizations, such as M = 10, averages predicted probability distributions, and recovers continuous values by expectation over bin centers.The recovered values are denormalized using instance-wise statistics to restore the original scale.

5 Experiments

Across benchmark, long-sequence, probabilistic, naturally missing, efficiency, downstream, and ablation evaluations, MDTIM consistently demonstrates strong imputation accuracy, robustness, scalability, and practical efficiency. Its gains are attributed to discrete expectation-based decoding, ordinal-aware labeling, and an appropriate vocabulary-size trade-off.

  • Main Results: MDTIM achieves the lowest MAE on Energy, ETTh, and Weather across Uniform and Geometric missing patterns, while FGTI performs best on synthetic Sine.At 30% Uniform missing on Sine, FGTI reaches 0.001 versus MDTIM’s 0.002; on ETTh, FGTI reaches 0.218 versus MDTIM’s 0.127.
  • Scalability to Long-Term Dependencies: At L = 192 with 30% Uniform missing, MDTIM achieves MAE 0.123 versus 0.192 for BRITS and 0.145 for SAITS.The performance advantage grows with sequence length, while MDTIM is comparable to SAITS at L = 48.
  • Probabilistic Imputation Performance: MDTIM consistently outperforms CSDI in CRPS across datasets and missing scenarios, including ETTh at 30% uniform missing, where it reaches 0.0863 versus 0.1120.Expectation-based decoding over discrete distributions reduces sampling ambiguity, while Ordinal-Aware Soft Labeling supports calibrated uncertainty estimates.
  • Imputation under Naturally Missing Real-World Data: On PhysioNet 2012 with approximately 80% natural missingness, MDTIM achieves the lowest MAE across all settings and outperforms SAITS.The spectral consistency loss is disabled because the data are highly sparse and irregular; FGTI degrades most and underperforms BRITS.
  • Efficiency: Small MDTIM completes Energy inference in 4.05s versus up to 894s for Large CSDI while reducing MAE from 0.074 to 0.046.The 0.65M Small MDTIM also surpasses the 88.24M Large SAITS, demonstrating parameter efficiency.
  • Downstream Forecasting: With a fixed PatchTST forecaster, MDTIM reduces Energy forecasting MAE by 19–26% versus SAITS and 16–19% versus CSDI across all horizons.Because the forecaster is identical, the forecasting differences directly reflect imputation quality.

6 Conclusion and Limitations … A.1.2 Conditional Diffusion

The paper concludes that MDTIM combines masked diffusion training with stochastic discretization and expectation-based unmasking for continuous time-series imputation, while identifying vocabulary selection, spectral regularization, and broader evaluation as future challenges. The preliminaries describe diffusion models, Gaussian corruption and learned denoising, and conditional imputation through mask-aware corruption of missing coordinates.

  • 6 Conclusion and Limitations: MDTIM adapts masked diffusion training to continuous time-series imputation using Stochastic Discretization, Ordinal-Aware Soft Labeling, and Expectation-based Unmasking.These components bridge continuous dynamics and categorical tokenization while enabling continuous reconstruction.
  • 6 Conclusion and Limitations: The optimal vocabulary size K may depend on each dataset’s signal range and granularity, motivating data-adaptive discretization.The paper proposes automatically selecting the discretization resolution to improve generalization.
  • 6 Conclusion and Limitations: Spectral consistency regularization assumes stable frequency components and is disabled on irregularly sampled benchmarks such as PhysioNet.An adaptive frequency objective that responds to local signal regularity is proposed as future work.
  • 6 Conclusion and Limitations: The evaluation uses standard moderate-length imputation benchmarks, leaving extremely long sequences and multi-scale or non-stationary domains for future study.Financial tick data is given as an example of a non-stationary domain.
  • A.1 Diffusion Models: Diffusion models learn to invert a fixed noise-injection process and have become prominent generative models for continuous data, including time-series forecasting and imputation.The preliminaries also cite applications in image, audio, and video synthesis.
  • A.1 Diffusion Models: A clean sample x_0 ∈ R^d is transformed into noisy latents x_t over T steps under a noise schedule {β_t}^T_{t=1}, with per-step and cumulative retention factors.The notation defines α_t = 1 − β_t and the cumulative retention factor ᾱ_t.
  • A.1.1 Diffusion Framework: The forward diffusion process gradually applies Gaussian corruption toward isotropic noise, while the learned reverse process incrementally denoises toward the data manifold.The reverse posterior conditioned on x_0 is Gaussian with closed-form mean and variance depending on β_t and ᾱ_t.
  • A.1.2 Conditional Diffusion: Conditional time-series imputation samples missing coordinates given observed ones by preserving observed values while applying Gaussian corruption only to missing coordinates.The network receives the partially noisy tensor, missingness mask M, and timestep t, exposing observed coordinates and values.

A.1.3 Training objective … A.2.3 Training Objective

The paper contrasts continuous diffusion’s noise-prediction objective with masked diffusion’s discrete absorbing-mask process, which directly predicts clean tokens and supports conditional imputation. Its training loss reduces to weighted cross-entropy on masked or missing coordinates while preserving observed values.

  • A.1.3 Training objective: Continuous diffusion training maximizes an ELBO whose per-step KL terms reduce to a reweighted denoising loss for predicting added noise.In the conditional setting, the noise predictor receives the partially noisy tensor and missingness mask, with loss restricted to missing coordinates; mask patterns are sampled during training.
  • A.2 Masked Diffusion Language Models: Masked Diffusion Language Models replace Gaussian corruption with an absorbing-state Markov chain that gradually masks tokens and iteratively unmasks them.The prior is the all-[MASK] sequence, and tokens are represented as one-hot vectors with 0 denoting [MASK].
  • A.2.1 MDLM Framework: The forward MDLM process independently replaces tokens with [MASK], using αt = e−σ(t) and, in experiments, the log-linear schedule αt = 1 −t.Because [MASK] is absorbing, the reverse process is organized around iterative unmasking.
  • A.2.1 MDLM Framework: The reverse kernel predicts a mixture over non-mask tokens while enforcing zero-masking and carry-over constraints, preventing remasking and preserving already-unmasked tokens.These constraints mirror the absorbing forward process.
  • A.2.2 Conditional Masking: Conditional masking treats observed tokens as anchors that remain unchanged, while only non-anchored positions undergo corruption and denoising.The anchored set corresponds to observed coordinates, and its complement corresponds to missing coordinates in imputation.
  • A.2.3 Training Objective: MDLM minimizes a negative ELBO whose per-step KL terms reduce to weighted cross-entropy over currently masked positions.The denoiser predicts the clean token directly, and the continuous-time objective samples t ∼U(0, 1) with weight w(t) = α′.
  • A.2.3 Training Objective: With anchors, training reconstructs only non-anchor masked positions, targeting the missing coordinates conditional on observed values.Observed positions never appear masked and are excluded from the training loss.

B Derivation of the Soft-Label Loss Decomposition … C.2.2 Noise Schedule and Time-Dependent Weighting

The paper decomposes ordinal-aware soft-label training into the vanilla MDLM objective, an ordinal-neighborhood KL penalty, and an entropy term, then evaluates MDTIM across diverse datasets with reproducible configurations and time-dependent diffusion weighting. Its log-linear noise schedule and weighting emphasize low-noise training while reducing emphasis on highly corrupted states.

  • B Derivation of the Soft-Label Loss Decomposition: The soft label is written as a convex combination of the one-hot ground-truth target and a distribution supported on its ordinal neighborhood.The off-target mass is ϵ := 1 − s_y, with u_y = 0 and support {i : 0 < |i − y| ≤ w}.
  • B Derivation of the Soft-Label Loss Decomposition: The resulting loss decomposes into a weighted vanilla MDLM cross-entropy, an ordinal-neighborhood KL divergence, and an entropy term.The decomposition is L = (1 − ϵ) L_NELBO + ϵ D_KL(u ∥ p_θ) + ϵ H(u).
  • B Derivation of the Soft-Label Loss Decomposition: As the soft label sharpens, the KL term vanishes and the objective recovers vanilla MDLM; otherwise, it penalizes probability outside the ordinal neighborhood.For s → e_y, equivalently ϵ → 0, L → L_NELBO; for ϵ > 0, the additional KL term is non-negative.
  • B Derivation of the Soft-Label Loss Decomposition: The same decomposition extends to the full time-weighted objective by linearity inside the expectation E_t[w(t)·].This establishes the decomposition for the objective used during diffusion training.
  • C.1 Datasets: MDTIM is evaluated on four datasets: Energy, ETTh, Weather, and synthetic Sine, covering periodic, nonlinear, stochastic, and non-stationary dynamics.The real-world benchmarks assess practical validity, while Sine provides controlled evaluation of periodic behavior.
  • C Experimental Settings: The experiments use common architectural parameters and training schemes across datasets, with reproducibility supported by detailed hyperparameter settings and specified hardware.Training used dual Intel Xeon Gold 6444Y CPUs and one NVIDIA H100 PCIe GPU with 80GB memory.
  • C.2.1 Hyperparameters: MDTIM uses the Factorized Temporal-Variate Backbone with consistent hyperparameter settings across all datasets.Detailed architecture and training settings are summarized in Table 10.
  • C.2.2 Noise Schedule and Time-Dependent Weighting: The log-linear noise schedule defines continuous-time noise σ(t) and its rate of change for t ∈ [0, 1] using ϵ = 10^-3, while w(t) balances training across noise levels.The weighting assigns higher importance near t ≈ 0 for fine-grained clean-data details and down-weights ambiguous states near t ≈ 1.

C.3 Evaluation Metrics

The evaluation measures point imputation accuracy and predictive-distribution quality exclusively on masked positions, using MAE, MSE, and CRPS. Lower CRPS reflects better probabilistic imputation through calibration and sharpness.

  • Evaluation scope: All metrics are computed exclusively on masked positions, denoted by the index set M.This evaluation applies to both point accuracy and predictive-distribution quality.
  • Point accuracy: MAE and MSE quantify point imputation accuracy by averaging errors over masked entries M.The reported metrics are Mean Absolute Error (MAE) and Mean Squared Error (MSE).
  • Probabilistic quality: CRPS assesses probabilistic imputation performance using the predictive CDF at each masked position.CRPS is reported as the average over all masked entries in M.
  • Probabilistic quality: Lower CRPS indicates better probabilistic imputation, rewarding both calibration and sharpness.In probabilistic settings, imputed values are based on the sample mean of generated trajectories.

D Additional Experimental Results · D.1 Full Imputation Performance Across All Missing Ratios · D.2 Comparison with Continuous Modeling

The supplementary experiments extend imputation comparisons across 30%, 50%, and 70% missing ratios and isolate Stochastic Discretization through a continuous-modeling ablation. MDTIM maintains the lowest MAE across most settings, while its discrete variant consistently outperforms the continuous counterpart across missing rates and types.

  • D Additional Experimental Results: Section D provides supplementary results, including the omitted 50% missing-ratio comparison and an ablation isolating Stochastic Discretization.The ablation compares the proposed discrete formulation with a continuous-modeling counterpart.
  • D.1 Full Imputation Performance Across All Missing Ratios: Table 11 evaluates MDTIM and baselines at 30%, 50%, and 70% missing ratios under Uniform and Geometric scenarios.The reported MAE values are averaged over 3 random seeds for L = 48.
  • D.1 Full Imputation Performance Across All Missing Ratios: MDTIM consistently maintains the lowest MAE across the majority of settings, with stable relative improvement over baselines across missing ratios.The results indicate that MDTIM’s advantage is not specific to one corruption level.
  • D.2 Comparison with Continuous Modeling: MDTIM (Cont) retains the Factorized Temporal-Variate backbone and masked diffusion training but directly models continuous inputs with Mean Squared Error loss.MDTIM (Disc) instead uses Stochastic Discretization and an ordinal-aware discrete diffusion objective.
  • D.2 Comparison with Continuous Modeling: Figure 4 compares MDTIM (Disc), MDTIM (Cont), SAITS, and CSDI on Energy under Uniform and Geometric missing scenarios across varying missing rates.The comparison is conducted across both missing types and multiple missing rates.
  • D.2 Comparison with Continuous Modeling: MDTIM (Cont) performs competitively with strong baselines, but MDTIM (Disc) consistently achieves lower MAE across all missing rates and missing types.The result supports the effectiveness of masked diffusion in continuous spaces while showing an additional advantage for the discrete formulation.

E Visualization of Imputation Results

MDTIM visualizations across four benchmark datasets show reconstructions under 50% uniform and geometric masking, with imputed series tracking ground truth while representing uncertainty. The results indicate that MDTIM captures complex temporal dynamics and periodicity in multivariate time series.

  • Visualization setup: Visualizations cover ETTh, Energy, Weather, and Sine datasets under 50% uniform and geometric masking.Figures show all channels for ETTh and Sine, and subsets of channels for Energy and Weather.
  • Imputation results: MDTIM’s imputed values track the ground truth while shaded intervals represent estimated uncertainty.Blue lines denote ground truth, green lines denote reconstructed values, and green shading indicates uncertainty intervals.
  • Imputation results: The visual results show that MDTIM captures complex temporal dynamics and periodicity in multivariate time series.This observation is reported across the displayed benchmark reconstructions.
Loading 2608.19119v1…