PaperlayerView exact revision on arXiv

Source-linked AI summary

SAITS: Self-Attention-based Imputation for Time Series

Wenjie Du, David Cote, Yan Liu

arXiv:2202.08516v5cs.LG

TL;DR

Missing values are pervasive in multivariate time series and hinder advanced analysis, creating a need for accurate imputation. The paper proposes SAITS, trained through joint imputation and reconstruction optimization with two diagonally-masked self-attention blocks. Experiments report state-of-the-art imputation accuracy and potential benefits for pattern recognition on incomplete real-world time series.

  • Problem

    Missing values are common in real-world multivariate time series and challenge interpretability, advanced analysis, and pattern recognition.

  • Method

    SAITS uses joint optimization and a weighted combination of two diagonally-masked self-attention blocks for multivariate time-series imputation.

  • Results

    SAITS achieves state-of-the-art imputation accuracy and shows potential to facilitate pattern-recognition models on partially observed real-world time series.

  • Takeaways & Limitations

    Self-attention provides a non-autoregressive basis for imputation that addresses recurrent models’ speed, memory, and compounding-error drawbacks.

  • Takeaways & Limitations

    The evaluation focuses on missing completely at random data, with missing values uniformly sampled independently as artificial missingness.

Abstract

from arXiv · show

Missing data in time series is a pervasive problem that puts obstacles in the way of advanced analysis. A popular solution is imputation, where the fundamental challenge is to determine what values should be filled in. This paper proposes SAITS, a novel method based on the self-attention mechanism for missing value imputation in multivariate time series. Trained by a joint-optimization approach, SAITS learns missing values from a weighted combination of two diagonally-masked self-attention (DMSA) blocks. DMSA explicitly captures both the temporal dependencies and feature correlations between time steps, which improves imputation accuracy and training speed. Meanwhile, the weighted-combination design enables SAITS to dynamically assign weights to the learned representations from two DMSA blocks according to the attention map and the missingness information. Extensive experiments quantitatively and qualitatively demonstrate that SAITS outperforms the state-of-the-art methods on the time-series imputation task efficiently and reveal SAITS' potential to improve the learning performance of pattern recognition models on incomplete time-series data from the real world. The code is open source on GitHub at https://github.com/WenjieDu/SAITS.

1 Introduction

Missing values are common in real-world multivariate time series and complicate analysis, motivating imputation methods that can handle temporal and feature relationships. The paper proposes self-attention-based training and SAITS, then evaluates them on real-world datasets.

  • Missing values arise from sensor failures, communication errors, and malfunctions, impairing interpretability and challenging classification and clustering.
  • Imputation preserves partially observed information, whereas deletion can discard data and yield biased parameter estimates.
  • Self-attention offers a non-autoregressive alternative to recurrent models that can avoid compounding errors and reduce speed and memory constraints.
  • The paper introduces joint optimization of imputation and reconstruction for self-attention models.
  • SAITS combines two diagonally-masked self-attention blocks to capture temporal dependencies and feature correlations explicitly.
  • Experiments and ablation studies on four real-world public datasets show SAITS achieves state-of-the-art imputation accuracy and potential benefits for pattern recognition on partially observed series.

2 Related Work

Prior time-series imputation work includes recurrent, generative, and self-attention-based models. The review highlights computational, dependency, training, and interpretability limitations motivating SAITS.

  • RNN-based Models: RNN-based methods model temporal information with recurrent states, while BRITS additionally incorporates feature correlations and treats missing values as graph variables.
  • RNN-based Models: RNN-based imputation models are time-consuming, memory-constrained, and poorly suited to long-term dependencies, with autoregressive methods also susceptible to compounding errors.
  • RNN-based Models: NAOMI reduces some recurrent limitations but retains a loop that can greatly slow imputation, while bidirectional RNN approaches are not deeply bidirectional.
  • VAE-based Models: VAE-based approaches use Gaussian-process priors or auxiliary covariates to construct smoother latent representations for time-series imputation.
  • Generative Models: GAN- and VAE-based generative models are difficult to train; GANs face non-convergence and mode collapse, while VAE latent variables can hinder interpretability.
  • Self-Attention-based Models: Self-attention-based methods apply attention across dimensions or use Transformer-based architectures, convolutional features, kernel regression, or time-oriented representations.

3 Methodology

SAITS combines joint optimization of masked imputation and observed reconstruction with a self-attention architecture built from two DMSA blocks and a weighted-combination module. The approach addresses limitations of training self-attention models only with reconstruction and is designed to improve imputation while retaining reconstruction performance.

  • Joint-optimization training: SAITS jointly optimizes a Masked Imputation Task and an Observed Reconstruction Task using their imputation and reconstruction losses.MIT predicts artificially masked observed values, while ORT reconstructs observed values.
  • Joint-optimization training: Training Transformer only with ORT yields much larger imputation MAE than BRITS, because the model receives no penalty for its outputs on missing values.Transformer-ORT achieves smaller reconstruction MAE than BRITS but its imputation MAE increases from the beginning.
  • Joint-optimization training: MIT artificially masks observed values and trains the model to predict them accurately, directly penalizing errors on missing-value estimates.In the described setup, 20% of observed values are randomly masked in each batch.
  • Joint-optimization training: ORT trains the model to reconstruct observed values, helping it converge toward the distribution of observed data.Its reconstruction loss is the mean absolute error between observed values and their reconstructions.
  • Joint-optimization training: Transformer-ORT+MIT steadily reduces both imputation and reconstruction MAE, while MIT primarily decreases imputation MAE and ORT further optimizes it.Transformer-MIT has slightly higher imputation MAE and higher reconstruction MAE because it is not required to converge on observed data.
  • SAITS architecture: SAITS consists of two diagonally-masked self-attention blocks followed by a weighted combination of their representations.Diagonal attention entries are set near negative infinity so their post-softmax attention weights approach zero.

Q K V

SAITS uses diagonally-masked self-attention blocks and joint optimization to impute multivariate time series. Its first block preserves observed values while learning missing values, and later representations are adaptively combined.

  • Diagonally-Masked Self-Attention: Diagonally-masked attention prevents each time step from contributing to its own estimation, forcing dependence on the other T −1 steps.This allows one attention operation to capture temporal dependencies and feature correlations between time steps.
  • First DMSA Block: The first DMSA block concatenates the input feature vector with its missing mask before projection, positional encoding, stacked attention layers, and feed-forward processing.The block produces Learned Representation 1 from the transformed sequence.
  • Second DMSA Block: The second DMSA block processes the first block’s completed output together with the missing mask to continue learning a second representation.It applies the same broad attention-and-feed-forward pattern before producing Learned Representation 2.
  • Weighted Combination Block: SAITS combines the first two learned representations using weights derived from attention weights and missingness information.The weighted combination forms Learned Representation 3, with weights dynamically assigned across time steps and features.
  • Architecture Scope: SAITS limits the architecture to two DMSA blocks because adding more provides only marginal benefit.The paper reports experiments and analysis supporting this design choice.
  • Joint Optimization: The final training objective jointly minimizes masked imputation and observed-reconstruction losses using MAE.The model combines LORT and LMIT by a weighted sum, with λ fixed as 1 in the experiments.

4 Experiments

The experiments benchmark SAITS on four public real-world datasets spanning clinical, air-quality, electricity-consumption, and transformer-temperature time series. Dataset descriptions and preprocessing details are reported alongside general dataset information.

  • Datasets: The benchmark uses four public real-world datasets from different domains: PhysioNet-2012, Air-Quality, Electricity, and Electricity Transformer Temperature.The datasets cover clinical measurements, air pollutants, electricity consumption, and transformer-related measurements.
  • Preprocessing: All datasets undergo standardization during preprocessing.The paper states that preprocessing details and general information are provided for all four datasets.
  • PhysioNet-2012: PhysioNet-2012 contains 12,000 multivariate clinical time-series samples collected from ICU patients during their first 48 hours after admission.Each sample includes up to 37 measured time-series variables.
  • Air-Quality: Air-Quality aggregates 11 variables from each of 12 Beijing monitoring sites, producing 132 features over 48 months.The dataset contains 1.6% missing values.
  • Electricity: Electricity records 15-minute electricity consumption from 370 clients over 48 months and contains no missing data.The dataset is used as a source for introducing controlled missingness in later experiments.
  • ETT: The 15-minute ETT dataset contains 69,680 sample points, each with seven features, including oil temperature and six external power-load features.The data spans two years from 2016/07/01 to 2018/06/26.

4.2 Baseline Methods

The benchmark compares SAITS with two naive imputers and five recent state-of-the-art deep-learning models. The baselines include median and last-observation filling plus GAN-, RNN-, VAE-, and BRITS-based approaches.

  • Naive Methods: The comparison includes Median and Last, two naive methods that fill missing values using training-set medians or previous observations.Last uses 0 when no previous observation exists.
  • Deep-Learning Methods: The deep-learning baselines are GRUI-GAN, E2GAN, M-RNN, GP-VAE, and BRITS.These five recent state-of-the-art models are compared with the proposed method.

4.3 Experimental Setup

Imputation performance is evaluated with MAE, RMSE, and MRE, while training uses fixed batching and early stopping. Hyper-parameters are searched separately for most models on each dataset, with SAITS-base held fixed as a baseline.

  • Evaluation Metrics: The evaluation uses MAE, RMSE, and MRE, with errors computed only on values indicated by the input mask.The metrics are Mean Absolute Error, Root Mean Square Error, and Mean Relative Error.
  • Training: Training uses batch size 128 and stops after 30 epochs without a decrease in MAE.An early-stopping strategy is applied during model training.
  • Hyper-Parameter Selection: Hyper-parameter searches are performed for every model on each dataset except SAITS-base.SAITS-base uses fixed hyper-parameters and serves as a baseline in ablation experiments.

4.4 Experimental Results

Experiments across four datasets show that SAITS delivers the strongest imputation performance while remaining efficient, robust across missing rates, and beneficial for downstream classification. Comparisons with NRTSI further show lower error with fewer parameters.

  • Imputation performance: SAITS significantly outperforms all baseline methods, Transformer, and SAITS-base across all metrics and datasets.Transformer surpasses BRITS on three datasets and is comparable on Air-Quality, while SAITS exceeds both self-attention comparators.
  • Efficiency: SAITS takes half the training time or less than BRITS per epoch, while SAITS-base uses only 15%–30% of Transformer’s parameters with comparable performance.These results support greater efficiency for SAITS-based architectures on time-series imputation.
  • Robustness to missingness: With sufficient capacity, SAITS achieves the best performance in eight of nine Electricity cases spanning missing rates from 20%–90%.SAITS-base is better than Transformer at 20%, 30%, and 40% missingness but becomes worse at higher rates because its fixed, limited capacity is insufficient.
  • Downstream classification: A classifier trained on SAITS-imputed PhysioNet-2012 data achieves the best results on every evaluation metric.SAITS improves over BRITS by 1.3% in ROC-AUC, 1.9% in PR-AUC, and 1.4% in F1-score.
  • Comparison with NRTSI: SAITS outperforms NRTSI in every tested case on Air and Gas, reducing MSE by 7%–39%.The comparison uses MSE across missing rates from 10%–80% on both datasets.
  • Comparison with NRTSI: SAITS uses only 12% and 3% of NRTSI’s parameters on Air and Gas, respectively.The paper notes that NRTSI’s shared hyperparameters may contribute to its larger parameter count.

4.5 Ablation Studies

Ablation studies validate diagonal masking and weighted representation combination in SAITS, while showing that adding a third DMSA block provides marginal benefit and wastes resources.

  • Diagonal masks: SAITS-base outperforms its version without diagonal masks on all datasets, demonstrating that DMSA improves imputation ability.
  • Model depth: The second DMSA block extends learning and is paired with weighted combination to improve performance, whereas more than two blocks provide marginal benefit.
  • Three-block variants: Three-block variants use either residual or cascade-weighted connections to combine representations from three DMSA blocks.Cascade weighting combines two representations at a time, while the residual design directly connects the block outputs.
  • Model depth: SAITS-3residual and SAITS-3cascade do not generally outperform SAITS, so the additional block adds parameters and computation without better results.

5 Conclusion

The conclusion presents SAITS as a self-attention model for multivariate time-series imputation, combining joint optimization with an efficient architecture. It reports improved accuracy and speed against established methods while identifying broader missingness patterns and datasets as future validation targets.

  • SAITS is a self-attention-based model designed to impute missing values in multivariate time series.
  • Joint optimization is designed to train self-attention-based models for the imputation task.
  • SAITS reduces MAE by 12% ∼38% and trains 2.0 ∼2.6 times faster than BRITS.
  • SAITS achieves 7% ∼39% better imputation accuracy than NRTSI and MAE 2% ∼19% smaller than Transformer with comparable training speed.
  • SAITS needs only 15% ∼30% of Transformer’s parameters to obtain comparable performance.
  • Future work will test SAITS on partially observed series with other missing patterns and on larger real-world datasets to assess generality.

A Details of Hyper-parameter Searching

The hyper-parameter search samples learning, regularization, architecture, and latent-process settings across model families, while SAITS-base uses a fixed configuration.

  • General: All models sample learning rates log-uniformly from 1 × 10−4 to 1 × 10−2 and, when applicable, dropout rates from 0.0 through 0.5.
  • RNN-based models: RNN-based models search hidden sizes from 32 to 1024, with additional GRUI-GAN and E2GAN-specific latent, pretraining, and loss-weight settings.
  • SAITS-base: SAITS-base fixes learning rate at 0.001, dropout at 0.1, N at 2, dmodel at 256, dffn at 128, and four attention heads.

B BRITS Trained by the joint-optimization approach

This section evaluates applying the joint-optimization approach to BRITS while keeping its original hyper-parameters fixed. The approach improves BRITS on three datasets but worsens performance on Electricity and remains below Transformer and SAITS.

  • BRITS is retrained with the joint-optimization approach by applying MIT, while the original training uses the w/oMIT condition.Hyper-parameters are kept exactly the same between the two conditions.
  • BRITS-wMIT outperforms BRITS-w/oMIT on PhysioNet-2012, Air-Quality, and ETT, but performs worse on Electricity.
  • Applying MIT to BRITS can improve performance on some datasets, but its benefit depends on the dataset and is not necessary universally.
  • Despite improvements over BRITS-w/oMIT on three datasets, BRITS-wMIT remains inferior to Transformer and SAITS.
Loading 2202.08516v5…