PaperlayerView exact revision on arXiv

Source-linked AI summary

REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers

Aivin V. Solatorio, Olivier Dupriez

arXiv:2302.02041v1cs.LG

TL;DR

Synthetic tabular data may help make sensitive data accessible, but few models generate relational datasets and modeling relationships across tables is challenging. REaLTabFormer uses GPT-2 for parent tables and a parent-conditioned Seq2Seq model for child tables, with target masking and statistical methods to address copying and overfitting. Experiments report better relational-statistics capture than baseline models and state-of-the-art prediction-task results on large non-relational datasets.

  • Problem

    Few synthetic-data models generate relational datasets, and modeling parent tables and their cross-table relationships is challenging.

    Source: §1 · pp. 1–2

  • Method

    REaLTabFormer generates parent tables with autoregressive GPT-2 and child tables with a parent-conditioned Seq2Seq model, using target masking and statistical methods to address copying and overfitting.

    Source: §1 · p. 2, §3.3.1 · p. 4, §3.3.2 · p. 5

  • Results

    Experiments report that REaLTabFormer captures child-table relational statistics more accurately than baselines and achieves state-of-the-art machine-learning task results for large non-relational datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that realistic synthetic tabular data can serve as a proxy for real-world data in machine-learning tasks.

    Source: §5 · p. 8

    Qδ may not give conclusive guidance when the compared distributions’ CDFs intersect at a quantile, and it works best for detecting left-tail dissimilarity.

    Source: §3.3.2 · p. 5

Abstract

from arXiv · show

Tabular data is a common form of organizing data. Multiple models are available to generate synthetic tabular datasets where observations are independent, but few have the ability to produce relational datasets. Modeling relational data is challenging as it requires modeling both a "parent" table and its relationships across tables. We introduce REaLTabFormer (Realistic Relational and Tabular Transformer), a tabular and relational synthetic data generation model. It first creates a parent table using an autoregressive GPT-2 model, then generates the relational dataset conditioned on the parent table using a sequence-to-sequence (Seq2Seq) model. We implement target masking to prevent data copying and propose the $Q_{\delta}$ statistic and statistical bootstrapping to detect overfitting. Experiments using real-world datasets show that REaLTabFormer captures the relational structure better than a baseline model. REaLTabFormer also achieves state-of-the-art results on prediction tasks, "out-of-the-box", for large non-relational datasets without needing fine-tuning.

1. Introduction

Synthetic tabular data could make sensitive data more useful, but generating relational data remains challenging because it requires modeling relationships across tables. REaLTabFormer combines parent-table and child-table generation, adds safeguards against copying, and is reported to outperform baselines on relational statistics and large-dataset prediction tasks.

1. Introduction

  • Relational synthetic-data generation has seen limited work, in part because modeling complex relationships within and across tables is challenging.Synthetic data may offer an alternative for using sensitive data in granular and segmentation analyses.Source: §1 · pp. 1–2
  • The paper frames synthetic data as useful for predictive modeling, including inferring variables that may be expensive to collect.It also describes synthetic data as a possible resource for privacy-related data access and research.Source: §1 · pp. 1–2
  • REaLTabFormer generates parent tables with an autoregressive GPT-2 model and child-table observations with a Seq2Seq model conditioned on the parent table.The Seq2Seq encoder uses the pretrained parent-table model’s weights.Source: §1 · p. 2
  • Target masking is introduced to reduce the likelihood that generated observations replicate training records, alongside statistical bootstrapping and a DCR measure to detect overfitting.The introduction identifies data copying as a concern for synthetic data intended to make sensitive information accessible.Source: §1 · pp. 1–2
  • REaLTabFormer’s experiments show more accurate child-table relational statistics than baseline models and state-of-the-art machine-learning task results for large non-relational datasets.The paper evaluates both relational and non-relational generation on real-world datasets.Source: §1 · p. 2

2. Related Work

Prior work has advanced synthetic generation for non-relational tabular data, while relational-data generation remains limited. REaLTabFormer builds on autoregressive transformers and proposes a Seq2Seq approach for generating arbitrary-length data conditioned on an input.

2. Related Work

  • Deep learning approaches including GANs, autoencoders, language models, and diffusion models have been applied to non-relational tabular generation.These studies demonstrate the capacity of deep learning models to produce more realistic data than traditional approaches such as Bayesian networks.Source: §2 · p. 2
  • Relational-data generators are limited, and existing hierarchical Gaussian Copula models do not accurately capture nuanced conditions within and across tables.The passage points to Figures 2 and 3 for these quality comparisons.Source: §2 · p. 2
  • TabGPT showed that autoregressive transformers can generate arbitrary-length transactional data, but requires separate models for each user; REaLTabFormer instead proposes Seq2Seq generation conditioned on an input.The paper describes per-user independent models as impractical for real-world applications.Source: §2 · p. 2

3. REaLTabFormer

REaLTabFormer combines autoregressive generation for non-relational parent tables with Seq2Seq generation of relational child tables. Its training and sampling strategies address copying, overfitting, and invalid outputs.

3. REaLTabFormer

  • The framework uses autoregression for non-relational tables and Seq2Seq generation for relational datasets, alongside overfitting detection and constrained sampling.The section also describes tabular encoding and target masking.Source: §3 · p. 3

3.1. The REaLTabFormer Models

  • Parent rows are generated autoregressively; related child rows form arbitrary-length sequences conditioned on representations from the frozen parent-table encoder.The encoder uses the parent-table model’s weights; only the cross-attention layer and decoder need training for the child model.Source: §3.1 · p. 3
  • Figure 2 shows REaLTabFormer closely capturing daily Sales seasonality compared with SDV’s HMA model.Source: Figure 2 · p. 3

3.2. Tabular Data Encoding

  • Column-specific vocabularies filter irrelevant tokens; numerical values are handled with a fully text-based strategy.Child-table training pairs use a fixed-length parent input and an arbitrary-length concatenation of related child observations, limited by computational resources.Source: §3.2 · pp. 3–4
  • Figure 3 shows REaLTabFormer spanning device types and imputing age values for iPodtouch users whose ages are missing in the original data.SDV learned only two of the thirteen device types.Source: Figure 3 · p. 4

3.3.1. TARGET MASKING

  • Target masking randomly replaces target tokens with a mask; sampling prohibits mask outputs and probabilistically fills them, reducing the likelihood of copying training records.The experiments use a 10% mask rate.Source: §3.3.1 · pp. 4–5

3.3.2. OVERFITTING ASSESSMENT

  • The overfitting assessment compares quantile distributions of distances to closest records and uses bootstrapped Qδ thresholds to monitor generated samples during training.The statistic is asymmetric and works best for detecting dissimilarity in the left tail, where generated samples may be unusually close to training data.Source: §3.3.2 · pp. 5–6

3.3.3. SAMPLING

  • Column-specific token domains constrain sampling, making invalid samples close to zero; the experiments generated less than 1% invalid samples.Source: §3.3.3 · p. 6

4. Experiments and Results

The evaluation compares REaLTabFormer with open-source baselines on real-world non-relational and relational datasets, using machine-learning utility and synthetic-data realism measures. REaLTabFormer performs strongly on non-relational prediction tasks without tuning and generates relational data with more accurate statistics than HMA.

4.1. Data

  • The non-relational datasets span 768–197,080 observations and cover regression, binary classification, and multiclass classification; relational tests use Rossmann and Airbnb.The non-relational datasets contain 8–50 numerical variables and 0–8 categorical variables.Source: §4.1 · p. 7

4.2. Baseline models

  • In Rossmann, Figure 4 compares average child-table Sales across parent-table StoreType categories for original data, REaLTabFormer, and SDV.The figure caption states that REaLTabFormer captures inter-table variations and relationships well.Source: Figure 4 · p. 7
  • Non-relational baselines include TVAE, CTABGAN+, Tab-DDPM, and GReaT; HMA is the relational baseline because the cited GAN implementation was unavailable.The relational comparison is limited to open-sourced models.Source: §4.2 · p. 7

4.3. Generative models training

  • REaLTabFormer uses the same parameters across datasets and relies on its overfitting metric rather than tuning against machine-learning tasks.GReaT was trained for 100 epochs per dataset, while TVAE, CTABGAN+, and Tab-DDPM used parameters tuned for the predictive task.Source: §4.3 · p. 7

4.4. Measures and Results

  • The evaluation uses tuned CatBoost models for machine-learning efficacy, generative-model validation data for early stopping, and a random forest discriminator on real and synthetic samples.For relational data, logistic detection uses ROC-AUC averaged across three cross-validation folds; reported LD is 100×(1−µRA), with higher scores indicating better quality.Source: §4.4 · pp. 7–8
  • Without task-specific tuning, REaLTabFormer achieves the best or second-best machine-learning efficacy scores against the baselines.The evaluation reports macro average F1 for classification and R2 for regression.Source: §4.4 · p. 8
  • Untuned REaLTabFormer discriminator scores are close to Tab-DDPM, while Tab-DDPM has the most indistinguishable synthetic data.Discriminator accuracy closer to 50% indicates that synthetic and real observations are harder to distinguish.Source: §4.4 · p. 8
  • Across tested relational datasets, REaLTabFormer produces significantly higher-quality synthetic data than HMA, including parent, child, and merged tables.Table 2 reports logistic detection results for those tables and states that REaLTabFormer consistently beats the SDV model.Source: Table 2 · p. 7, §4.4 · p. 8

5. Conclusion

REaLTabFormer generates non-relational and relational synthetic data, using target masking to mitigate copying and the Qδ statistic with statistical bootstrapping to detect overfitting. The paper reports realistic synthetic data for machine-learning tasks and accurate relational generation compared with open-sourced alternatives.

5. Conclusion

  • The framework generates high-quality synthetic non-relational tabular data and relational datasets.The conclusion presents REaLTabFormer as an extension of sequence-to-sequence models to relational data generation.Source: §5 · p. 8
  • Target masking mitigates data copying, while the Qδ statistic and statistical bootstrapping provide a method for detecting overfitting.The conclusion notes that the statistical method may be adapted to other generative-model training.Source: §5 · p. 8
  • The authors identify data imputation, cross-survey imputation, and upsampling for imbalanced machine-learning data as possible extensions.They also describe improved privacy strategies and additional safeguards against sensitive-data exposure as opportunities for future work.Source: §5 · p. 8

6. REaLTabFormer Python Package

The package provides workflows for fitting, saving, loading, and sampling from REaLTabFormer on tabular data. Relational generation uses a trained parent model to condition a child model.

6.1. Non-relational tabular model

  • The package supports fitting REaLTabFormer on a tabular dataset and saving the trained model and its artifacts.The example also shows generating synthetic data with the same number of observations as the real dataset.Source: §6 · p. 9, §6.1 · p. 9
  • The example workflow loads a saved model by providing its experiment directory.Source: §6.1 · p. 9

6.2. Non-relational tabular model

  • Relational training first fits and saves a parent-table model, then uses that model as the child model’s encoder.Synthetic parent samples are passed to the trained child model to generate relational observations.Source: §6.2 · p. 9

A. Raw data processing

Raw numerical columns are transformed into text-based, tokenizable representations; the section also specifies how datetime, categorical, and missing values are handled.

Numerical data

  • Numerical columns are converted to strings, aligned with zero padding, and split into fixed-length partitions for model input.The procedure includes setting a rounding resolution and moving negative signs to the start of the string; integral values use leading-zero alignment.Source: A. Raw data processing · p. 12
  • Single-character partitioning worked best in the experiments, which the authors attribute to regularization from generating numbers one digit at a time.Source: A. Raw data processing · p. 12

Datetime and categorical data

  • Datetime values are converted to Unix timestamps and processed as numeric data, while categorical values are represented by unique tokens.Source: A. Raw data processing · p. 12

Missing values

Input data aggregation

  • Tokens are assigned independently by processed column, so the same character in different columns receives different token IDs.Source: A. Raw data processing · p. 12

B.1. Non-relational tabular data

The non-relational evaluation uses six varied real-world datasets and compares REaLTabFormer with other synthetic-data models using established splits and trained baselines.

B.1. Non-relational tabular data

  • Six datasets spanning mixed numerical and categorical variables and 8 to 51 columns were used to assess synthetic tabular data generation.The collection comprises Abalone, Adult, Buddy, California Housing, Diabetes, and Facebook Comments.Source: §B.1 · p. 13
  • The experiments recreated Tab-DDPM GitHub data splits for REaLTabFormer and GReaT, whose model expects column-name context.Source: §B.1 · p. 13
  • TVAE, CTABGAN+, and Tab-DDPM were trained from scratch using the published parameters for each dataset.Source: §B.1 · p. 13

B.2. Relational tabular data

The relational evaluation uses Rossmann store sales and Airbnb new user bookings, with dataset-specific sampling and train/test restrictions.

B.2. Relational tabular data

  • The relational experiments use the Rossmann store sales and Airbnb new user bookings datasets.Source: §B.2 · p. 13
  • Rossmann training uses 80% of stores and their sales records, restricted to two months from June 2015 onward; Airbnb uses 8,000 of 10,000 sampled users for training.Airbnb test metrics and plots use the remaining 2,000 users, with users limited to at most 50 sessions.Source: §B.2 · p. 13

C. Reproducibility

The reproducibility materials specify software versions, experimental datasets and workstation hardware, and identify logistic-detection comparisons for generated tables.

C. Reproducibility

  • The experiments used be great 0.0.3, sdv 0.17.2, sdmetrics 0.8.1, transformers 4.25.1, torch 1.13.1, and Python 3.9.The HyperTransformer implementation included a bug fix; the authors state that they will open-source the package and experiments repository.Source: C. Reproducibility · p. 13
  • The experiments ran on Ubuntu 20.04 LTS with two AMD EPYC 7H12 64-Core processors, two RTX 3090 GPUs, and 1TB RAM.Source: C. Reproducibility · p. 13
  • Experiments covered Abalone, Adult, Buddy, California Housing, Diabetes, Facebook Comments, Rossmann store sales, and Airbnb new user bookings.Source: C. Reproducibility · p. 13
  • Table 4 scopes logistic-detection results for HMA-generated parent, child, and merged tables; the reported detector is logistic regression.Source: Table 4 · p. 14, C. Reproducibility · p. 14

D. Other measures and results

The appendix reports that logistic regression gives unreliable logistic-detection results, then presents joint plots and distance-to-closest-record distributions as qualitative assessments across five datasets.

D. Other measures and results

  • Logistic-regression detection scores appear unreliable and overly high despite qualitative inaccuracies, so the authors report random-forest detection results instead.They suggest logistic regression's limited capacity to learn data structure may explain the spurious scores; random forests capture nonlinearities and appear to give reasonable results.Source: D. Other measures and results · p. 14

D.1. Joint plots

  • Joint plots compare two numerical variables across model-generated samples for Abalone, Adult, Buddy, California housing, and Diabetes.The plots provide a qualitative assessment of synthetic-data quality.Source: Figure 5 · p. 14, §D.1 · p. 14, Figure 6 · p. 15

D.2. Distance to closest record (DCR) distribution

Loading 2302.02041v1…