PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models are Realistic Tabular Data Generators

Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, Gjergji Kasneci

arXiv:2210.06280v2cs.LG

TL;DR

Synthetic tabular generation must preserve heterogeneous feature types, contextual relationships, and data characteristics despite challenges including impurity, privacy, and lossy preprocessing. GReaT textually encodes records and fine-tunes an autoregressive LLM, achieving state-of-the-art performance across varied datasets while supporting arbitrary conditioning. The authors note that generated data still requires privacy and validity checks before sharing.

  • Problem

    Synthetic tabular data generation must preserve realistic characteristics in heterogeneous, noisy, contextually interconnected datasets, while existing approaches face preprocessing and conditioning limitations.

  • Method

    GReaT textually encodes feature names and values, fine-tunes a pretrained transformer-decoder LLM, and samples remaining features conditional on any specified subset.

  • Results

    GReaT maintains state-of-the-art generative performance across numerous real-world and synthetic datasets with heterogeneous feature types and varied sizes.

  • Takeaways & Limitations

    Textual encoding and autoregressive LLMs provide a realistic and flexible route for heterogeneous tabular data generation, including arbitrary conditional synthesis.

  • Takeaways & Limitations

    Generated samples can be invalid at rates consistently below 1%, and data sharing still requires verifying that reverse identification is impossible or prevented by regulation.

Abstract

from arXiv · show

Tabular data is among the oldest and most ubiquitous forms of data. However, the generation of synthetic samples with the original data's characteristics remains a significant challenge for tabular data. While many generative models from the computer vision domain, such as variational autoencoders or generative adversarial networks, have been adapted for tabular data generation, less research has been directed towards recent transformer-based large language models (LLMs), which are also generative in nature. To this end, we propose GReaT (Generation of Realistic Tabular data), which exploits an auto-regressive generative LLM to sample synthetic and yet highly realistic tabular data. Furthermore, GReaT can model tabular data distributions by conditioning on any subset of features; the remaining features are sampled without additional overhead. We demonstrate the effectiveness of the proposed approach in a series of experiments that quantify the validity and quality of the produced data samples from multiple angles. We find that GReaT maintains state-of-the-art performance across numerous real-world and synthetic data sets with heterogeneous feature types coming in various sizes.

1 INTRODUCTION

Tabular data generation is difficult because datasets are heterogeneous, noisy, privacy-sensitive, and contextually structured. GReaT addresses these challenges by encoding records as text and using an autoregressive LLM that supports arbitrary conditioning, achieving state-of-the-art performance across diverse datasets.

  • Motivation: Tabular datasets often have class imbalance, privacy constraints, and noisy or missing values that complicate data sharing and machine-learning use.Synthetic data may help alleviate these issues, although the supplied passage is truncated before stating the full consequence.
  • Core challenges: Mixed categorical and numerical features create substantial preprocessing and modeling challenges for synthetic data generation.The Adult Income dataset, for example, contains seven numerical and eight categorical variables.
  • Core challenges: Extensive preprocessing can lose information or introduce artifacts, such as artificial ordering among originally unordered categorical values.Common steps include numerical encoding, scaling, missing-value replacement, outlier removal, and smoothing.
  • Core challenges: The contextual knowledge problem concerns preserving coherent relationships among interconnected variables during generation.Examples include relationships among age, marital status, and education in Adult Income data.
  • Core challenges: Arbitrary conditioning enables synthesis given any specified feature subset, supporting arbitrary missingness patterns and subset oversampling without retraining for each feature combination.Most existing methods require retraining for each specific conditioning set.
  • Proposed approach: GReaT transforms heterogeneous tabular records into text containing feature names and values, allowing pretrained autoregressive LLMs to model the resulting representations.The encoding avoids artificial orderings and retains more information than usual numerical transformations while exposing variable names as contextual information.
  • Proposed approach: GReaT inherits arbitrary conditioning from LLMs trained with random feature-order permutations and samples unspecified features.The approach connects tabular and textual modalities through a textual encoding scheme.
  • Reported contributions: GReaT achieves state-of-the-art generative performance across diverse datasets and measures, with open-sourced results and a three-line Python implementation.The implementation is distributed through the be-great package.

2 RELATED WORK

Prior tabular generators include Bayesian networks, copulas, VAEs, GANs, and transformer-based models for related tasks. The paper positions GReaT as the first attention-based LLM approach for highly realistic non-sequential tabular generation.

  • Existing tabular methods: Classical tabular modeling methods include Chow–Liu Bayesian networks and copulas, while more recent approaches use generative adversarial networks.Synthetic tabular generation has received less attention than image and text generation.
  • Existing tabular methods: CTGAN focuses on conditional feature distributions and uses mode-specific normalization for non-Gaussian features.Its one-hot encoding of modes and categorical features can aggravate the curse of dimensionality.
  • Language-model background: Natural-language processing progressed from recurrent networks to self-attention models, including autoencoding, sequence-to-sequence, and autoregressive architectures.These developments motivate applying language-modeling approaches beyond conventional text tasks.
  • Transformer-related work: Transformer models had already been applied to tabular classification, joint tabular-text representations, and multivariate time-series generation.The cited applications differ from non-sequential tabular data generation.
  • Research gap: The paper identifies no prior work generating highly realistic non-sequential tabular data with attention-based LLMs and presents GReaT as the first rigorous exploration.It links this direction to state-of-the-art performance.
  • GReaT pipeline: The GReaT fine-tuning pipeline textually encodes tabular rows, permutes feature order, and fine-tunes a large language model on the resulting sentences.The figure uses a toy dataset inspired by Adult Income.

3 GREAT: GENERATION OF REALISTIC TABULAR DATA

GReaT fine-tunes an autoregressive LLM on textually encoded tabular rows, then samples and converts completed text back into tabular data. Random feature ordering enables arbitrary conditioning on feature subsets without retraining, while the approach preserves feature information and contextual relationships.

  • Overview: GReaT has two stages: fine-tuning a pretrained LLM on textually encoded tabular data, then sampling the fine-tuned model to generate synthetic rows.Generated textual representations are converted back into tabular format.
  • Textual encoding: The textual encoder represents each feature value with its feature name in a subject-predicate-object clause, then concatenates the clauses into a row representation.This representation uses clauses such as “Occupation is doctor” and avoids information loss from standard numerical transformations.
  • Sampling: At inference, GReaT can begin from a feature name or arbitrary feature-value combination, complete the remaining tokens autoregressively, and extract a synthetic tabular row.The method describes feature-name preconditioning as sampling from the entire joint distribution and supports applications such as missing-value imputation.
  • Feature order permutation: Randomly permuting feature clauses removes artificial order dependencies and supports arbitrary conditioning during generation.The same permutation strategy enables conditioning on any specified combination of features and values at inference time.
  • Sampling: Invalid generated samples occur at rates consistently below 1%, mainly when the language model produces category values outside the required format.The paper suggests lowering sampling temperature and applying validation to reject infrequently invalid samples.
  • Summary of advantages: GReaT combines arbitrary sampling control, contextual representations from large text corpora, and minimal preprocessing for tabular data generation.The paper presents these properties as principal strengths of the method.

4 EXPERIMENTAL EVALUATION

The evaluation compares GReaT and Distill-GReaT with established tabular generators across diverse real-world and synthetic data sets using multiple quantitative and qualitative measures. Results assess utility, realism, similarity, and the contributions of pretraining and permutation.

  • Evaluation setup: Six real-world and three synthetic data sets span diverse domains, sizes, and feature counts, with an 80%/20% train-test split preventing data leakage.The real-world sets range from fewer than 1,000 to more than 100,000 samples.
  • Evaluation setup: GReaT and Distill-GReaT use pretrained transformer-decoder language models with over 355 million and 82 million learned parameters, respectively.The models are compared against CTGAN, TVAE, and CopulaGAN baselines.
  • Evaluation measures: Machine learning efficiency tests whether models trained on synthetic data perform on real test data comparably to models trained on original data.The evaluation reports accuracy for classification and mean squared error for regression across linear/logistic regression, decision tree, and random forest models.
  • Evaluation measures: Distance to closest records evaluates whether synthetic samples resemble but do not exactly copy records from the original training set.DCR uses the minimum L1 distance to training records, with categorical matches assigned zero difference and mismatches one.
  • Evaluation measures: Discriminator accuracy measures how easily generated and original records can be distinguished, with 50% representing perfectly indistinguishable data.A tuned Random Forest discriminator is trained on mixed generated and original data.
  • Qualitative evaluation: Joint density plots compare generated and original feature distributions, including whether methods capture the strong dependency between California Housing longitude and latitude.The plots provide a qualitative comparison with baseline generators.
  • Ablation study: Pretraining improves performance on all but the very small Travel data set, while the experiments separately examine permutation and pretraining effects.Table 3 reports discrimination and Random Forest ML-efficiency results for modified Distill-GReaT models.

5 CONCLUSION

The paper presents a textual encoding strategy for heterogeneous tabular records and fine-tunes a transformer-decoder language model on that representation. It reports unprecedented generative performance and outstanding flexibility for realistic tabular data synthesis.

  • 5 CONCLUSION: GReaT represents heterogeneous tabular records as sentences that capture each record’s semantics instead of encoding them in a numerical format.The approach leverages state-of-the-art generative language models for tabular data synthesis.
  • 5 CONCLUSION: The resulting transformer-decoder network achieves unprecedented generative performance and outstanding flexibility for synthesizing realistic tabular data samples.This conclusion summarizes the reported outcome of fine-tuning the network on the textual representation.

ETHICS STATEMENT

The ethics statement emphasizes that synthetic tabular data can support sharing in critical applications, while stressing that ethics, privacy, and numerical encoding considerations remain important. It also describes transformer models as unifying heterogeneous modalities.

  • ETHICS STATEMENT: Synthetic tabular data could support data sharing in critical applications such as healthcare and finance, but ethics and privacy considerations remain necessary.The authors strongly encourage proprietary-data owners to verify relevant considerations before sharing any data, including generated data.
  • ETHICS STATEMENT: GReaT converts heterogeneous tabular data into text, representing continuous and discrete numerical values as character sequences.The passage notes that independent studies have shown transformer models can process numerical data encoded this way, including in a oneshot setting.
  • ETHICS STATEMENT: Transformer models can holistically unite numerical values, textual metadata, named categories, and open-text features into context-specific representations.This unification addresses modalities that were previously processed separately.

B ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments report ROCAUC and F1 alongside accuracy measures, and examine DCR distributions for HELOC. The DCR experiment indicates that GReaT generates new samples close to original records rather than copying training samples.

  • B ADDITIONAL EXPERIMENTAL RESULTS: ROCAUC and F1 scores supplement accuracy measures in the machine learning efficiency experiment.These additional metrics are reported for the experiment described in Section 4.
  • B ADDITIONAL EXPERIMENTAL RESULTS: The HELOC DCR distributions compare original test-to-train distances with synthetic-to-train distances to assess similarity without copying.DCR measures distance to the closest record in the original training set.
  • B ADDITIONAL EXPERIMENTAL RESULTS: The HELOC experiment indicates that GReaT generates new synthetic samples close to original samples rather than copying training records.This conclusion is drawn from the reported DCR distribution experiment.

B.2 AVERAGE NEGATIVE LOG-LIKELIHOOD METRIC FOR SYNTHETIC DATA

The evaluation measures how likely generated samples are under the training distribution and how well generated-data density models explain held-out samples. GReaT is comparable to state-of-the-art deep neural networks on high-dimensional mixture distributions.

  • Lsyn measures generated-sample likelihood under the training-data density, while Ltest evaluates original test-sample likelihood under a density fitted to generated data.The evaluation uses Bayesian networks and Gaussian mixture models, respectively, for the fitted density models.
  • LLMs are comparable with state-of-the-art deep neural networks when modeling high-dimensional mixture distributions.
  • Ltest is included because training-distribution likelihood can be prone to overfitting.

B.3 DISTANCE TO CLOSEST RECORD RESULTS

The closest-record analysis examines whether generated samples resemble training records without reproducing them exactly. Results for California Housing and HELOC indicate that generated samples are close to the originals while remaining non-identical.

  • Figures 4 and 5 show minimal-distance distributions for California Housing and HELOC, respectively.
  • Figure 6 compares original and generated Adult Income data using joint plots for Age and EducationNum.
  • Generated samples are close to the original data without copying records exactly.The comparison uses distributions of minimal distances from generated samples to the training data.

B.4 ADDITIONAL QUALITATIVE ANALYSIS RESULTS

Additional analyses examine joint feature distributions, computational efficiency, dataset coverage, and pretrained language-model configurations. GReaT and Distill-GReaT achieve top benchmark performance despite higher fine-tuning time.

  • Qualitative analysis: The qualitative analysis compares the joint distribution of Age and EducationNum in Adult Income using kernel density estimator-based joint plots.
  • Performance and runtime: The runtime comparison evaluates training or fine-tuning and sampling times using two GPUs, with 200 epochs and 1,000 generated samples for each model.
  • Performance and runtime: GReaT and Distill-GReaT demonstrate top performance across the vast majority of benchmarks and various datasets.The experiments include medical machine-learning datasets because some applications prioritize high-quality synthetic data despite computation costs.
  • Evaluation scope: The evaluation covers six real-world datasets from varied domains and three synthetic datasets with varying feature counts.
  • Evaluation scope: The study uses pretrained large language models whose structural details are reported for the evaluated configurations.

C REPRODUCIBILITY DETAILS

The paper documents implementation, hardware, dataset, model, and hyperparameter details to support reproducibility. It also releases the implementation and generated synthetic datasets.

  • The implementation uses pretrained generative language models and HuggingFace routines for fine-tuning and sampling, and the code is open-sourced.
  • Distill-GReaT is fine-tuned for 200 epochs per dataset, except California Housing and Diabetes, which use 100 epochs.
  • Synthetic data are generated using name-value pair preconditioning with temperature T set to 0.7 across experiments and datasets.
  • The proposed models use AdamW with learning rate 5 × 10^-5, while baseline models are trained for 200 epochs per dataset.
  • Experiments run on two NVIDIA 2080RTX GPUs with 12 GB RAM each, 126 GB system RAM, and a 24-core AMD Ryzen 3960X.
  • The authors share the GReaT-generated synthetic datasets alongside reproducibility information.
Loading 2210.06280v2…