Source-linked AI summary
Tabular Transformers for Modeling Multivariate Time Series
Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, Erik Altman
TL;DR
Tabular time series are less studied than static data, although their temporal structure matters and sensitive datasets can restrict analysis. The paper proposes hierarchical transformer models for representation learning and synthetic sequence generation, reporting consistent TabBERT downstream gains and realistic TabGPT transactions.
Problem
Tabular time series have received less attention than static tabular data, although modeling them requires accounting for temporal components.
Source: §1 · p. 1Method
The paper proposes TabBERT, which uses hierarchical transformer representations, and TabGPT, which generates synthetic tabular time series.
Source: §1 · p. 1Results
Takeaways & Limitations
TabBERT features can be used for fraud detection in place of hand-engineered features, and TabGPT can generate card transactions that replace real data.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-of-the-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular sequences. We demonstrate our models on two datasets: a synthetic credit card transaction dataset, where the learned representations are used for fraud detection and synthetic data generation, and on a real pollution dataset, where the learned encodings are used to predict atmospheric pollutant concentrations. Code and data are available at https://github.com/IBM/TabFormer.
1. INTRODUCTION
The paper addresses the relative lack of methods for tabular time series, where temporal dependencies matter, by proposing transformer models for representation learning and synthetic sequence generation. It evaluates these models on credit-card transactions and pollution data.
1. INTRODUCTION
- Dynamic tabular data have received less attention than static data, despite the need to account for temporal components; prior synthetic tabular schemes also fall short of modeling temporal dependencies.The paper frames this gap against work on static tabular data and static-data generation.Source: §1 · p. 1
- TabBERT uses field-level transformers to embed rows, then treats row embeddings as tokens for BERT-based representation learning; TabGPT adapts GPT to generate synthetic tabular time series.The approach quantizes continuous fields into finite vocabularies.Source: §1 · p. 1
- Prior transformer work addressed static tables and text, whereas this paper presents its work as the first to apply transformers to tabular time series.The authors also distinguish their temporal generation goal from prior synthetic static-data approaches [7].Source: §1 · p. 1
- The models are evaluated on synthetic credit-card transactions for fraud detection and data generation, and on public pollution data for pollutant-concentration prediction.The authors open-source the synthetic transaction dataset and provide code to reproduce the experiments.Source: §1 · p. 2
2. TABBERT: UNSUPERVISED LEARNING OF MULTIVARIATE TIME SERIES REPRESENTATION
TabBERT treats multivariate tabular time series as sequences of structured rows, quantizing continuous values and modeling relationships both within and across rows. Its learned features improve reported downstream performance on fraud classification and pollution regression.
2.1. From Language Modeling to Tabular Time Series
- Each sample is a sequence of rows with fields quantized to categorical values and represented using field-specific vocabularies.This structured-row input differs from a sequence of ordinary text tokens.Source: §2.1 · p. 2, Figure 1 · p. 2
2.2. Hierarchical Tabular BERT
- A field transformer embeds each row, then a sequence transformer models relationships across rows and their temporal component.The model is trained using masked language modeling to predict masked tokens.Source: §2.2 · p. 2
2.3. TabBERT Features for Classification
- For fraud detection, training excludes the fraud label and masks 15% of sample fields for prediction.The transaction corpus is generated synthetically; fraud samples are imbalanced and the fraudulent class is upsampled during training.Source: §2.3 · pp. 2–3
2.4. TabBERT Features for Regression Tasks
- The regression evaluation uses Beijing PM2.5 data to predict PM2.5 and PM10 concentrations from sequences of time-dependent rows.Continuous features are quantized, targets are removed from inputs, and the combined RMSE is reported on a 15K-sample test set.Source: §2.4 · p. 3
2.3. TabBERT Features for Classification; 2.4. TabBERT Features for Regression Tasks
- TabBERT features improve reported performance over raw embedded features on both fraud F1 and pollution RMSE tasks.The supplied passages give no metric values for these comparisons.Source: Table 1 · p. 3, §2.4 · p. 3, §2.3 · p. 3
3. TABGPT: GENERATIVE MODELING OF MULTIVARIATE TIME SERIES TABULAR DATA
TabGPT trains user-specific GPT models on chronological transactions to predict future behavior and generate synthetic sequences, evaluating both aggregate feature fidelity and temporal similarity.
3. TABGPT: GENERATIVE MODELING OF MULTIVARIATE TIME SERIES TABULAR DATA
- Separate GPT models train on chronologically ordered ten-transaction sequences to predict future transactions from past context.The models are trained separately for selected users with relatively high transaction volumes.Source: §3 · p. 3
- Aggregate field histograms compare ground-truth and generated distributions, with χ2 distance measuring proximity.The evaluation compares histograms over all timestamps for each field.Source: §3 · p. 4
- Both user-specific TabGPT models produce feature distributions similar to ground truth, including high-entropy Amount; user 1 is generally closer.The comparison covers the columns for each selected user.Source: §3 · p. 4, Figure 4 · p. 4
- To assess temporal fidelity beyond aggregate histograms, TabBERT embeds real and generated sequences, then FID compares dataset means and covariances.Each sequence is represented by a fixed-length vector obtained by mean pooling time-wise TabBERT embeddings.Source: §3 · p. 4
- 492.95 FID separates the two real users; generated transactions match their respective users more closely while preserving cross-user distances.The authors interpret the combined FID and aggregate-histogram analyses as evidence that TabGPT learns user behavior and generates realistic transactions.Source: Table 2 · p. 4, §3 · p. 4
4. CONCLUSION
The conclusion presents TabBERT and TabGPT for multivariate time-series tabular data, reporting downstream representation gains and realistic synthetic transaction generation. It also identifies quantization as making transformer modeling of sequential tabular data possible.
4. CONCLUSION
- The paper introduces Hierarchical Tabular BERT and Tabular GPT for modeling multivariate time-series tabular data.It also open-sources a synthetic card-transaction dataset and experiment code.Source: §4 · p. 4
- TabBERT representations provide consistent downstream gains and can replace hand-engineered fraud features by capturing field and temporal dependencies.Source: §4 · p. 4
- TabGPT reliably synthesizes card transactions that can replace real data and alleviate privacy issues in off-premise or cloud-based training.The conclusion attributes transformer modeling of sequential tabular data to quantization of continuous fields.Source: §4 · p. 4