Source-linked AI summary
TTS-GAN: A Transformer-based Time-Series Generative Adversarial Network
Xiaomin Li, Vangelis Metsis, Huangyingrui Wang, Anne Hee Hiong Ngu
TL;DR
Small medical time-series datasets limit deep learning, and RNN-based GANs have difficulty modeling long sequences with irregular temporal relations. The paper introduces TTS-GAN, a pure transformer-encoder GAN, and reports realistic synthetic sequences supported by visual, dimensionality-reduction, and quantitative fidelity evaluations.
Problem
Small medical time-series datasets limit effective deep learning, while RNN-based GANs struggle to model long sequences with irregular temporal relations.
Method
TTS-GAN uses transformer encoder architectures for both generator and discriminator to generate multidimensional time-series of various lengths.
Results
TTS-GAN generated synthetic time-series with visually similar signal patterns and distributions, and experiments demonstrated its viability as a realistic time-series generator.
Takeaways & Limitations
The results support TTS-GAN as a data-generation approach for realistic synthetic time-series when trained on real samples.
Abstract
from arXiv · showhide
Signal measurements appearing in the form of time series are one of the most common types of data used in medical machine learning applications. However, such datasets are often small, making the training of deep neural network architectures ineffective. For time-series, the suite of data augmentation tricks we can use to expand the size of the dataset is limited by the need to maintain the basic properties of the signal. Data generated by a Generative Adversarial Network (GAN) can be utilized as another data augmentation tool. RNN-based GANs suffer from the fact that they cannot effectively model long sequences of data points with irregular temporal relations. To tackle these problems, we introduce TTS-GAN, a transformer-based GAN which can successfully generate realistic synthetic time-series data sequences of arbitrary length, similar to the real ones. Both the generator and discriminator networks of the GAN model are built using a pure transformer encoder architecture. We use visualizations and dimensionality reduction techniques to demonstrate the similarity of real and generated time-series data. We also compare the quality of our generated data with the best existing alternative, which is an RNN-based time-series GAN.
1 Introduction
Medical time-series datasets are often too small for deep learning, while RNN-based GANs struggle with long sequences and irregular temporal relations. TTS-GAN addresses these challenges with a transformer-based GAN and evaluates synthetic data against real and alternative generated sequences.
- Motivation: Physiological time-series data are costly to collect and annotate, and differences in collection settings hinder dataset merging.These constraints contribute to small datasets for medical and health-related machine learning.
- Motivation: Training parameter-rich deep models on small datasets leads to over-fitting and low generalization, often forcing researchers toward shallower models.The paper identifies this as a common situation in medical and health-related machine learning.
- Motivation: Prior time-series GANs mainly use RNN architectures, whereas transformers are designed for very long sequences and avoid vanishing-gradient problems.The authors therefore motivate a transformer GAN for time-series data.
- Evaluation: PCA and t-SNE visualize synthetic and real sequence distributions in two dimensions, while signal-property measurements provide quantitative similarity comparisons.Transformer-generated and RNN-generated sequences are compared with real sequences of the same class.
- Contributions: The paper proposes heuristics for training a transformer-based GAN on time-series data and compares generated sequences qualitatively and quantitatively with real and state-of-the-art GAN-generated sequences.The comparison includes the RNN-based alternative Time-GAN.
2 Background
GANs generate data through an adversarial generator–discriminator system, while transformers process entire sequences using attention. TTS-GAN adapts image-patch processing to multidimensional time-series and applies a pure transformer design to GAN generation.
- Generative Adversarial Networks: GANs use a generator to create samples and a discriminator to distinguish generated examples from true data through minimax optimization.The generator aims to capture the distribution of true examples, while the discriminator classifies real versus generated samples.
- Transformer Architecture: Transformers process an entire sequence through layers whose token representations attend to latent representations of other tokens.This differs from recurrent networks, which consume sequences token by token.
- Time-Series Adaptation: The paper adapts pure-transformer image models to time-series GANs by processing temporal patches and positional information.It presents this as the first implementation of this image-inspired approach for a transformer GAN on time-series data.
- Time-Series Adaptation: TTS-GAN represents a time-series as a C × H × W tuple, setting H to 1 and W to the number of timesteps.The sequence is divided into patches along the W axis and receives positional encoding.
3 Methodology
TTS-GAN uses transformer encoder architectures for both generation and discrimination of multidimensional time-series data. The method converts random vectors into signal-shaped sequences and processes time-series inputs as image-like patches with positional encoding.
- Both the generator and discriminator are built from transformer encoder architectures with self-attention, feed-forward MLPs, normalization, dropout, and residual connections.
- The generator maps a random vector to a sequence with the real signals’ length, embeds it, patches it, adds positional encoding, and applies transformer blocks.A Conv2D layer reduces the generated representation to the real data dimensions.
- The discriminator divides the sequence width into patches, adds learned positional encoding, and classifies the input as real or synthetic.Its encoder input has shape (BatchSize, C, 1, (W/N) + 1).
- TTS-GAN treats each time-series as an image with height 1, using timesteps as width and channels as image channels.
- Mean Squared Error losses update both networks, with discriminator outputs compared against real and fake labels.The discriminator loss combines losses from real and synthetic data.
4 Experiments
The experiments evaluate TTS-GAN on simulated and biomedical or activity time-series datasets using visual and quantitative comparisons. Generated signals resemble real data visually and in low-dimensional distributions, while quantitative results favor TTS-GAN over Time-GAN in most cases.
- TTS-GAN is evaluated on simulated sinusoidal waves, UniMiB human activity recordings, and the PTB Diagnostic ECG dataset.The datasets include varied sequence lengths, dimensions, and signal classes.
- Synthetic samples show visually similar signal patterns to real samples in raw-data comparisons.
- PCA and t-SNE visualizations show similar distribution patterns for real and TTS-GAN-generated data.The plots use red dots for original data and blue dots for synthetic data.
- TTS-GAN wins against Time-GAN in 7 out of 10 similarity-score cases.The evaluation uses average cosine similarity and average Jensen–Shannon distance.
5 Conclusions
The paper concludes that TTS-GAN can generate realistic multidimensional time-series sequences of various lengths. Visual comparisons, dimensionality reduction, and similarity scores support the fidelity of its synthetic data.
- TTS-GAN generates multidimensional time-series data of various lengths using a transformer-based GAN.
- Raw signal and two-dimensional distribution visualizations show similarity between original and synthetic data.
- The experiments demonstrate TTS-GAN’s viability as a generator of realistic time-series when trained on real samples.
- Table 1 reports average cosine similarity and average Jensen–Shannon distance for real and synthetic data across five datasets.Higher avg_cos_sim and lower avg_jen_dis indicate better similarity.
A Appendix 1: Training Details
The experiments use fixed hardware and shared architectural and optimization settings across datasets. Synthetic data are generated from random vectors, while generator and discriminator learning rates differ.
- Experiments run on an Intel 3.40GHz CPU, 377GB RAM, and two Nvidia 1080 GPUs.
- The generator receives random vectors of size (100, 1), and both transformer stacks repeat their blocks three times.
- The generator uses learning rate 1e-4 and the discriminator uses 3e-4.
- Training follows the LSGAN setting and uses the paper’s loss function to update the model parameters.
B Appendix 2: Similarity Scores
The appendix evaluates similarity between real and synthetic signals using feature vectors derived from sequence statistics, then compares cosine similarity and Jensen–Shannon distance.
- Feature extraction: Feature vectors contain median, mean, standard deviation, variance, root mean square, maximum, and minimum values extracted from all sequence channels.
- Average Cosine Similarity: For each real–synthetic pair, cosine similarity compares feature vectors of size m.
- Average Cosine Similarity: Average cosine similarity averages pairwise similarities for real and synthetic signals from the same class.
- Average Jensen-Shannon distance: Average Jensen–Shannon distance averages distances between corresponding features from real and synthetic signals.
- Average Jensen-Shannon distance: The Jensen–Shannon calculation uses the pointwise mean of each real and synthetic feature and Kullback–Leibler divergence.