PaperlayerView exact revision on arXiv

Source-linked AI summary

Regular Time-series Generation using SGM

Haksoo Lim, Minjung Kim, Sewon Park, Noseong Park

arXiv:2301.08518v1cs.LG

TL;DR

Time-series data can be incomplete or scarce, and the paper identifies a gap in score-based methods for time-series synthesis. It proposes a conditional score network with a task-specific denoising score-matching loss, reporting state-of-the-art quality and diversity on real-world datasets. The authors note that score-based generation is slower than GAN-based generation.

Abstract

from arXiv · show

Score-based generative models (SGMs) are generative models that are in the spotlight these days. Time-series frequently occurs in our daily life, e.g., stock data, climate data, and so on. Especially, time-series forecasting and classification are popular research topics in the field of machine learning. SGMs are also known for outperforming other generative models. As a result, we apply SGMs to synthesize time-series data by learning conditional score functions. We propose a conditional score network for the time-series generation domain. Furthermore, we also derive the loss function between the score matching and the denoising score matching in the time-series generation domain. Finally, we achieve state-of-the-art results on real-world datasets in terms of sampling diversity and quality.

Introduction

The paper introduces conditional score-based generation for time series, a task the authors say prior score-based methods had not addressed. It reports strong generation quality and diversity across real-world datasets.

Introduction

  • Time-series samples may be incomplete or too few for robust machine-learning training, motivating research into synthesis models such as VAEs and GANs.The paper situates SGMs among generative models noted for sampling quality and diversity in other domains.Source: Introduction · p. 1
  • TSGM extends score-based generative models to time-series synthesis, where each observation must condition on previously generated observations.The paper distinguishes synthesis from forecasting and imputation, and says prior score-based time-series work addressed only those latter tasks.Source: Introduction · p. 1
  • The proposed method introduces a conditional score network that learns the time-gradient of the conditional log-likelihood and a task-specific denoising score-matching loss.TSGM has VP and subVP variants, determined by the stochastic differential equation type.Source: Introduction · p. 1
  • TSGM achieves the best generation quality in all but two of 10 evaluation cases, and the best sampling diversity among eight baselines.The two exceptions are discriminative scores on AI4I and Occupancy.Source: Introduction · pp. 1–2

Related Work and Preliminaries

This section reviews score-based generative models and existing time-series generation methods, then explains why forecasting and imputation score models do not directly fit time-series synthesis. It outlines the forward and reverse diffusion processes and their training objective.

Score-based Generative Models

  • SGMs add Gaussian noise to a sample in a forward process, then remove it in a reverse process to recover a new sample.The paper notes SGMs' generation quality, exact log-likelihood, and controllable generation without extra training.Source: Score-based Generative Models · p. 2

Forward Process

  • The forward SDE diffuses an original sample into a noisy sample x_s; its drift f and diffusion g can define VE, VP, or subVP variants.The score network learns the gradient of the noisy sample's log-likelihood for use in the reverse process.Source: Forward Process · p. 2

Reverse Process

  • The corresponding reverse SDE uses the score function to recover real samples from a Gaussian prior whose parameters depend on the forward SDE type.The paper attributes the corresponding reverse SDE result to Anderson (1982).Source: Reverse Process · p. 2

Training and Sampling

  • Denoising score matching substitutes for the score-matching loss because the exact log-likelihood gradient is unknown, with the transition-kernel score analytically calculable for affine drift.The loss samples s uniformly over [0, 1] and uses an appropriate weight function.Source: Training and Sampling · p. 2

Time-series Generation

  • Time-series generation models must generate each observation using its history; TimeGAN learns latent conditional dynamics with an RNN, while TimeVAE models temporal structures such as trend and seasonality.The paper also notes GAN-based models' vulnerability to mode collapse and unstable training.Source: Time-series Generation · p. 2

SGMs on Time-series

  • Existing score-based time-series methods target forecasting or imputation, including recursive forecasting with TimeGrad and whole-sequence reconstruction with CSDI.TimeGrad conditions on past observations and covariates; CSDI generates an entire sequence from a partial sequence.Source: SGMs on Time-series · pp. 2–3
  • The authors argue forecasting and imputation score functions mismatch synthesis, so they propose a conditional score network with a distinct denoising score-matching loss and prove its correctness.They contrast their task-specific loss with prior methods that use existing proofs.Source: SGMs on Time-series · p. 3

Proposed Methods : TSGM

TSGM combines a recurrent encoder-decoder with a conditional score network for sequential generation. It trains the score network with a time-series-specific denoising objective and samples latent observations recursively.

Encoder and Decoder

Conditional Score Network

  • Because observations depend on prior history, TSGM modifies U-net with 1D convolutions and conditions its score network on previous observations.Diffused data, condition, and temporal features are concatenated as inputs.Source: Conditional Score Network · p. 3

Training Objective Functions

  • The conditional score network learns gradients of conditional log likelihood on diffused prefixes, and Theorem 1 permits replacing l1(t, s) with l2(t, s).The objective is designed for time-series generation, where each observation depends on its history.Source: Training Objective Functions · p. 4
  • The practical score loss avoids an alternative requiring repeated draws from p(x_t|x_1:t−1), while the latent history is bounded by a fixed window K.The window represents a practical approximation to the full past information.Source: Training Objective Functions · p. 4

Training and Sampling Procedures

  • After pretraining, the score network learns conditional scores from forward-diffused encoder states; alternating encoder-decoder updates are optional because some datasets favor score-only training.The alternating-training choice is controlled by a Boolean hyperparameter.Source: Training and Sampling Procedures · p. 5
  • Sampling starts from Gaussian noise, recursively generates each latent vector conditioned on the previous generated vector, then decodes the completed sequence.The initial and subsequent latent vectors are generated with the predictor-corrector method.Source: Training and Sampling Procedures · p. 5

Experiments

Experiments benchmark TSGM against time-series synthesis baselines using predictive and discriminative scores and t-SNE visualizations, alongside ablation, sensitivity, and complexity analyses.

Experimental Environments

  • Five real-world datasets and eight baselines span methods from RNNs to VAEs and GANs; baseline code and selection follow their released procedures.The authors use source code from official repositories.Source: Experimental Environments · p. 5

Hyperparameters

  • Table 2 reports the best hyperparameters; input and temporal-feature dimensions are d_in and d_t, while other settings use TimeGAN and VPSDE defaults.Source: Table 2 · p. 5, Hyperparameters · p. 5

Evaluation Metrics

  • Across ten runs with different seeds, lower predictive and discriminative scores are preferred.The predictive score uses an LSTM trained on synthesized data and tested on real data; the discriminative score uses an LSTM real/fake classifier.Source: Evaluation Metrics · pp. 5–6

Experimental Results

  • TSGM has the best predictive scores and beats TimeGAN except on discriminative scores for AI4I and Occupancy.Table 3 reports the discriminative and predictive comparisons.Source: Table 3 · p. 6, Experimental Results · p. 6

Ablation study

  • Scratch-trained conditional score network, encoder, and decoder perform worse than the full model but still outperform many baselines.Table 4 compares results with and without autoencoder pretraining.Source: Ablation study · p. 6, Table 4 · p. 6

Sensitivity study

  • Reducing U-net depth from 4 to 3 improved discriminative score but slightly worsened predictive score; subVP remained good at 100 sampling steps, while VP was nearly unchanged at 500.Source: Sensitivity study · pp. 6–7

Experimental Results

  • t-SNE visualizations show TSGM samples recalling originals and exceeding TimeGAN and TimeVAE in diversity, especially on Energy and Air.Figure 2 uses red for original samples and blue for synthesized samples.Source: Experimental Results · p. 6, Figure 2 · p. 7

Empirical Complexity Analyses

  • Sampling generally takes longer than with TimeGAN and some simpler methods.Memory footprint and sampling runtime are reported in the Appendix.Source: Empirical Complexity Analyses · p. 7

Conclusion & Limitations

The paper combines an RNN-based autoencoder with a score network and task-specific denoising score matching, reporting state-of-the-art discriminative and predictive scores. It also notes sampling-speed and hyperparameter-search limitations.

Conclusion & Limitations

  • The framework combines an RNN-based autoencoder and score network with a generation-specific denoising score matching loss, and reports state-of-the-art discriminative and predictive scores.The authors also report ablation and sensitivity studies of the model design.Source: Conclusion & Limitations · p. 7
  • SGMs generate samples more slowly than GANs, a speed limitation the authors say faster-sampling methods could address without loss of sampling quality or diversity.The paper identifies this as a limitation shared by SGMs.Source: Conclusion & Limitations · p. 7
  • The experiments reused default TimeGAN and VPSDE hyperparameters for miscellaneous settings, leaving room for improvement.The authors also note that alternative autoencoders, including continuous-time methods for irregular time series, could be used.Source: Conclusion & Limitations · p. 7

TSGM: Time-series Generation using Score-based Generative Models

The passage marks the beginning of supplementary material.

TSGM: Time-series Generation using Score-based Generative Models

A Proofs

The proofs derive an alternative denoising score matching objective for time-series and establish its equivalence to the target score objective.

A Proofs

  • Theorem 2 states that l1(t, s) can be replaced by l2(t, s) for denoising score matching on time-series.Source: A Proofs · p. 10
  • The proof handles t = 1 via the naive denoising score loss and considers t > 1 by decomposing l1(t, s) and simplifying expectations.It uses expectation transformations and constants independent of θ in the derivation.Source: A Proofs · p. 10
  • The corollary defines the target objective Lscore and states that L2 = Lscore.Source: A Proofs · p. 10

B Datasets and Baselines

The experiments use five real-world datasets and compare multiple families of time-series generative baselines.

B Datasets and Baselines

  • The five datasets span stock, energy use, air quality, predictive maintenance, and room occupancy time series.The appendix table summarizes data dimensions, training-sample counts, and time-series lengths.Source: B Datasets and Baselines · p. 11
  • The stock, energy, AI4I, and occupancy datasets include correlated features; the air-quality records are hourly averages collected from sensors in an Italian city.Source: B Datasets and Baselines · p. 11
  • Baselines cover autoregressive methods, GAN-based methods, and a VAE-based method, including T-Forcing, TimeGAN, and TimeVAE.The listed families also include P-Forcing, RCGAN, C-RNN-GAN, COT-GAN, and WaveGAN.Source: B Datasets and Baselines · p. 11

C Search Space for Hyperparameters

The hyperparameter search varies the iteration count and hidden-feature dimension, while other settings follow TimeGAN and VPSDE defaults.

D Miscellaneous Experimental Environments

The experiments specify their software and hardware environment and describe model selection and score reporting for the proposed method.

E Empirical Space and Time Complexity Analyses

TSGM is slower than TimeGAN at sampling, which the authors describe as a fundamental drawback of SGMs; they note techniques that may relieve it.

Related papers

Loading 2301.08518v1…