PaperlayerView exact revision on arXiv

Source-linked AI summary

TabMT: Generating tabular data with masked transformers

Manbir S Gulati, Paul F Roysdon

arXiv:2312.06089v1cs.LGcs.AI

TL;DR

Synthetic tabular generation must contend with heterogeneous fields and privacy concerns. TabMT applies a Masked Transformer design to the task, and the paper reports state-of-the-art generation quality across datasets, including at scales orders of magnitude larger than prior work and with missing data. It also reports superior privacy and the ability to trade privacy against quality.

  • Problem

    Heterogeneous tabular fields present distinct generation challenges, while synthetic tabular data is important for privacy-focused applications.

    Source: §1 · p. 1

  • Method

    TabMT is a Masked Transformer design for synthetic tabular generation whose masking procedure supports missing data and privacy-quality trade-offs through temperature scaling.

    Source: §1 · p. 2

  • Results

    The paper reports state-of-the-art generation quality at scales orders of magnitude larger than prior work, with missing data present, and superior privacy with adjustable privacy-quality trade-offs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    TabMT is presented as a generator applicable to privacy-focused settings that need high-quality synthetic tabular data, including data with missing values.

    Source: §1 · p. 2, §6 · p. 9

    TabMT is slower than lightweight MLP-based methods or GANs, and optimal privacy requires additional temperature search time.

    Source: §5 · p. 8

Abstract

from arXiv · show

Autoregressive and Masked Transformers are incredibly effective as generative models and classifiers. While these models are most prevalent in NLP, they also exhibit strong performance in other domains, such as vision. This work contributes to the exploration of transformer-based models in synthetic data generation for diverse application domains. In this paper, we present TabMT, a novel Masked Transformer design for generating synthetic tabular data. TabMT effectively addresses the unique challenges posed by heterogeneous data fields and is natively able to handle missing data. Our design leverages improved masking techniques to allow for generation and demonstrates state-of-the-art performance from extremely small to extremely large tabular datasets. We evaluate TabMT for privacy-focused applications and find that it is able to generate high quality data with superior privacy tradeoffs.

1 Introduction

Tabular data’s heterogeneous fields pose distinct challenges, while synthetic tabular generators matter for privacy and other applications. The paper introduces TabMT, a Masked Transformer reported to perform strongly across datasets, handle missing data, and trade privacy against quality.

1 Introduction

  • Synthetic tabular generators serve applications including privacy preservation, data augmentation, interpretability, and anomaly detection.Prior approaches include GANs, VAEs, Autoregressive Transformers, and Diffusion models.Source: §1 · p. 1
  • TabMT is a Masked Transformer design for synthetic tabular generation intended to work across tasks and scenarios.The paper identifies heterogeneous data fields as a distinct challenge for tabular generation.Source: §1 · pp. 1–2
  • TabMT achieves state-of-the-art performance against GANs, VAEs, Autoregressive Transformers, and Diffusion models, with scalability from very small to very large datasets.The paper also reports applicability to privacy-focused generation and handling of missing data.Source: §1 · p. 2
  • Its masking procedure handles missing data and supports privacy-quality trade-offs through temperature scaling.The paper presents these capabilities as relevant to privacy-focused applications and real-world use cases.Source: §1 · p. 2

2 Related Work

Prior work applies deep generative models to language and tabular data, including several model families. TabMT explores Masked Transformers for tabular generation and is reported to outperform domain-specific netflow generators.

2 Related Work

  • Masked language models are traditionally used for embeddings, though prior work has explored masked content generation; this paper applies masked training to tabular generation.The passage situates this work among transformer language modeling and its use across tasks and domains.Source: §2 · p. 2
  • Deep tabular generators have used autoencoders, GANs, and Autoregressive Transformers, while TabMT uses Masked Transformers.The paper reports superior performance compared with prior deep tabular generators.Source: §2 · p. 2
  • TabMT is reported to handily outperform domain-specific models for netflow generation.Netflow data involve complex field rules, many possible values per field, and typically extremely large datasets.Source: §2 · p. 2

3 Method

TabMT adapts masked generation to unordered, heterogeneous tabular rows. Its masking and sampling choices address training–inference mismatches while supporting arbitrary prompts and missing values.

3 Method

  • TabMT uses bidirectional relationships between fields, which lack an inherent order in tabular data.The authors argue that bidirectional learning likely improves the model’s understanding and embeddings.Source: §3 · p. 2
  • TabMT supports arbitrary prompts and learns missing values by setting their masking probability to 1.The passage contrasts this with generators that require separate imputation before generating cleaned samples.Source: §3 · p. 3
  • Uniformly sampling each row’s masking probability trains across subset sizes, while random-order generation makes inference’s masked-set distribution match training.The design addresses mismatches caused by fixed masking probabilities and fixed-order generation.Source: §3 · p. 3
  • For categorical and continuous fields, TabMT uses separate field embeddings and constructs ordered continuous embeddings from quantized values and endpoint vectors.The ordered embedding combines each cluster’s unordered embedding with a ratio-weighted interpolation of the endpoint vectors.Source: §3 · p. 4
  • The ordered embeddings preserve value ordering while retaining unordered information useful for attention, multimodal distributions, and separating nearby values.The same structure is used for a dynamic output linear layer that can be converted to a traditional linear layer after training.Source: §3 · p. 4
  • For continuous fields, zero-initialized unordered embeddings and a learned per-field temperature help preserve ordered priors while sharpening predicted distributions.The learned temperature offers a sharpening mechanism without relying on larger endpoint-vector magnitudes.Source: §3 · p. 4

4 Evaluation

TabMT is evaluated across tabular datasets for data quality, privacy, missing-data handling, and scaling. The results report strong quality and privacy tradeoffs, native missing-data handling, and performance on a dataset with over 30 million rows.

4 Evaluation

  • The evaluation covers 15 datasets ranging from about 400 to 150,000 samples, with comparisons spanning GANs, VAEs, diffusion models, and autoregressive transformers.Data quality and privacy experiments use datasets with 6–50 columns; the evaluation uses CatBoost MLE to assess synthetic-data quality and diversity.Source: §4 · pp. 4–5, §4.1 · p. 5

4.1 Data Quality

  • TabMT matches or exceeds state-of-the-art MLE scores on 11 of 15 datasets.Table 1 reports MLE scores across techniques; F1 is used for classification and R2 for regression.Source: §4.1 · p. 5, Table 1 · p. 5
  • TabMT’s correlation errors are consistently closer to zero than those of TabDDPM or CTabGAN+.The paper treats nonzero correlation errors as evidence of erroneous or missing correlations.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 Privacy and Sample Novelty

  • Temperature scaling lets TabMT trade privacy against quality; it attains higher DCR than TabDDPM in all tested cases and higher MLE in most.The paper also reports higher privacy and MLE scores than CTabGAN+ in all tested cases.Source: §4.2 · p. 6

4.3 Missing Data

  • With 25% of values randomly missing, TabMT still trains and produces synthetic rows with no missing values.The experiment ensures nearly every row is permanently missing data; Table 3 reports MLE and its difference from training without missing values.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • TabMT can be conditioned to generate any subset of the data distribution at no additional cost, supporting augmentation of underrepresented portions.The paper contrasts this with prior art that is largely incapable of conditioning outputs.Source: §4.3 · p. 7

4.4 Scaling

  • On the CIDDS-001 netflow dataset, TabMT’s precision and recall are close to the validation set, and its diversity and quality increase with model size.The dataset contains over 30 million rows; Table 5 reports that TabMT outperforms NFGAN on precision, recall, and diversity.Source: §4 · p. 4, §4.4 · pp. 7–8, Table 5 · p. 8
  • Against NFGAN, TabMT produces more diverse data and achieves a median 20x improvement in violation probability across seven netflow invariants.The paper reports that NFGAN has reasonably high precision but poor recall and diversity because it suffers from mode collapse.Source: §4.4 · p. 8

5 Limitations and Future Work

TabMT is slower than lighter MLP- and GAN-based methods, and temperature search adds time when optimal privacy is sought. The paper also notes compute-related emissions and misuse risks.

5 Limitations and Future Work

  • TabMT’s Transformer backbone is slower than lightweight MLP methods and GANs that can produce a row in a single inference.Temperature search adds time when optimal privacy is needed, and quantizing continuous fields could pose issues in some applications.Source: §5 · p. 8
  • Netflow invariant tests cover structural rules, and per-field embeddings prevent TabMT from violating check 5.The example rule is that two public IP addresses cannot communicate with each other.Source: Table 6 · p. 9
  • The authors identify increased CO2 emissions from additional compute and the risk of synthetic data misuse, including fabricated research findings.They report roughly 410 GPU days across cloud A10 or V100 GPUs for algorithm design and experiment-result generation.Source: §5 · pp. 8–9

6 Conclusion

The paper concludes that TabMT achieves state-of-the-art synthetic tabular data quality across benchmarks and scales, including settings with missing data. It also reports strong privacy and a privacy–quality trade-off.

6 Conclusion

  • TabMT achieves state-of-the-art generation quality in comprehensive benchmarks, including at scales orders of magnitude larger than prior work and with missing data.The conclusion describes TabMT as a substantial advance in scalability, missing-data robustness, privacy-preserving generation, and data quality.Source: §6 · p. 9
  • TabMT achieves superior privacy and can trade off privacy against quality.The conclusion attributes this conclusion to the model’s privacy-preserving generation and reported privacy–quality trade-off.Source: §6 · p. 9
  • The paper presents TabMT as a Masked Transformer design and training procedure for generating synthetic tabular data.The conclusion highlights scalability, missing-data robustness, privacy-preserving generation, and data quality as its contributions.Source: §6 · p. 9

A Additional Figures

The appendix includes additional Pareto-front plots and a visualization of fake and real data manifolds for CIDDS-001 using PaCMAP.

A Additional Figures

  • Figure 5 presents additional Pareto fronts.The caption does not specify the datasets or outcomes shown.Source: Figure 5 · p. 12
  • Figure 6 visualizes fake and real data manifolds for TabMT on CIDDS-001 using PaCMAP.The caption identifies the dataset and visualization method but does not state a comparison outcome.Source: Figure 6 · p. 12

B Additional Tables

The additional tables describe training and temperature-search settings, tabular structures, and comparisons with CTabGAN+ and STaSY. The reported comparisons favor TabMT on the tested datasets.

B Additional Tables

  • Table 7 gives the training hyperparameter search space, with max bins denoting the maximum number of quantization bins for continuous attributes.The authors say many search-space hyperparameters were unimportant to final quality and that the space could likely be pruned; they used Optuna with default settings.Source: Table 7 · p. 13
  • Tables 8 and 9 show the original Netflow structure and its post-processed form, respectively, with all fields categorical after processing.The captions identify the structures but do not report comparative outcomes.Source: Table 9 · p. 13, Table 8 · p. 13
  • Temperature tuning uses a separate temperature for each field and jointly optimizes privacy and ML Efficiency; Table 10 reports search settings.The search used the full 8x data multiplier, took 0.25–5 GPU days except roughly 15 days each for MI and FB, and had no explicit trial budget.Source: Table 10 · p. 14
  • TabMT wins on both privacy and MLE against CTabGAN+ on all tested datasets in Table 11.The table caption says the higher privacy on FB versus the main privacy table demonstrates control of the MLE–DCR trade-off.Source: Table 11 · p. 14
  • TabMT scores higher than STaSY on all tested datasets in Table 12.The hyperparameter search included MLP width, depth, and learning rate.Source: Table 12 · p. 14

C Pseudocode

The pseudocode implements masked training and randomized-order generation, with embeddings and output layers designed to accommodate ordered categories and learned temperature scaling.

Ordered Embedding

  • OrderedEmbedding combines a learned residual embedding with an interpolation between learned endpoint vectors, weighted by normalized ordered cluster centers.The forward pass retrieves the resulting weight vectors by index.Source: C Pseudocode · p. 15

Dynamic Linear layer

  • DynamicLinear computes logits from the input and embedding weights, adds a learned bias, and divides by a sigmoid-transformed learned temperature.The layer accepts either an OrderedEmbedding or a standard embedding.Source: C Pseudocode · p. 15

Training step

  • Training randomly masks entries and computes cross-entropy only on masked positions, marking unmasked entries to be ignored in the loss.The mask is sampled per batch entry and applied across its fields.Source: C Pseudocode · p. 15

Batched generation

  • Generation initializes a zero batch and an all-ones mask, then samples each field in a random order from temperature-scaled model logits.Categorical sampling uses the temperature for the current field.Source: C Pseudocode · p. 15

MaskedTransformer and Embedding

  • The model uses a standard Transformer Encoder, learned positional embeddings, LayerNorm before DynamicLinear layers, and normally initialized mask and value embeddings.The stated standard deviations are 0.01 for positional embeddings and 0.05 for the mask token and standard embedding; standard embeddings are not clipped or rescaled during training.Source: C Pseudocode · p. 15

D Dataset Info

The main comparison reuses Kotelnikov et al.’s datasets and splits, while CIDDS-001 is added for scaling experiments. The listed datasets draw from UCI, OpenML, and Kaggle sources.

Dataset Info

  • The main comparison uses the same datasets and splits as Kotelnikov et al. [17].The paper lists the datasets and their sources.Source: D Dataset Info · p. 15
  • Additional listed datasets are AB Abalone and AD Adult ROC from OpenML, BU Adopt a Buddy from Kaggle, and CA California Housing from Kaggle.The entries report CC licenses for AB, AD, and CA; BU is listed with license “Other.”Source: D Dataset Info · p. 16
  • The list further includes CAR Cardiovascular Disease and CH Churn Modeling from Kaggle, plus DI Diabetes from OpenML and FB Facebook Comment Volume from UCI.The cited entries report CC licenses.Source: D Dataset Info · p. 16
  • The list includes GE Gesture Phase Prediction from UCI, HI Higgs (98K) from OpenML, HO House 16H from OpenML, and IN Medical Costs from Kaggle.The cited entries report CC licenses for GE, HI, and HO, and ODbL for IN.Source: D Dataset Info · p. 16
  • The list also includes KI King Count Housing Prices, MI MiniBOONE Particle Prediction, and WI Wilt Remote Sensing, each from OpenML.Each cited entry reports a CC license.Source: D Dataset Info · p. 16
  • CIDDS-001 is additionally used for scaling experiments and is listed with a CC license.Source: D Dataset Info · p. 16

Related papers

Loading 2312.06089v1…