Representation Learning for Tabular Data: A Comprehensive Survey

Background and challenges

• The survey focuses on supervised classification and regression, where rows are instances and columns are features, and notes heterogeneous attributes, data-quality issues, and privacy constraints.Tabular data is prevalent across domains including finance, healthcare, education, recommendation systems, and scientific research.Source: §1 · p. 1, Figure 1 · p. 1
• Although early DNNs struggled to match tree-based methods, modern architectures, training strategies, and representations have revitalized their performance, often rivaling or surpassing tree-based models.The survey examines DNN advantages and limitations and discusses dataset collection, evaluation, and analysis strategies to address performance instability across datasets.Source: §1 · pp. 1–2

Survey organization

• Specialized, transferable, and general models differ in their training and deployment scope, from same-distribution learning to fine-tuning and zero-shot use across downstream datasets.Generalization tends to increase across these categories, but specialized models can remain superior on large-scale datasets, and fine-tuning general models can improve predictive performance.Source: Figure 2 · p. 2, §1 · p. 2

Specialized models

• The survey replaces scattered architecture-based groupings with a hierarchical taxonomy organized around features, samples, and objectives, emphasizing feature- and sample-level representation strategies.The unified framework is intended to connect ideas across methods and support clearer comparisons and future model design.Source: §1 · pp. 2–3

Transferable models

• Transferable methods are grouped by pre-training source: homogeneous tabular domains, heterogeneous tabular domains, and other modalities, including vision and language.The survey further organizes language-model strategies by knowledge-extraction method and model scale, from small language models to LLMs; transfer is challenged by source–target differences in feature and label spaces.Source: §1 · p. 3

General models

• General models, or tabular foundation models, apply to downstream datasets without fine-tuning and are categorized by data-centric or model-centric adaptation strategies.The survey also covers TabPFN variants using in-context learning,, and methods that use attribute and task semantics to unify heterogeneous tasks,,.Source: §1 · p. 3

Ensembles and extensions

• The survey covers ensembles that combine strengths of multiple tabular models and extensions including open-environment learning, multimodal learning, and tabular understanding.It also overviews clustering, anomaly detection, data generation and imputation, and interpretability, while identifying open challenges to guide future research and applications.Source: §1 · p. 3

2.1 Learning with Tabular Data

• A supervised tabular task maps N rows of d numerical or categorical features to classification or regression labels, learning a predictor for unseen instances from the same distribution.The survey focuses on standard classification and regression, not ordinal regression.Source: §2.1 · p. 3

2.2 History of Tabular Data

• Tabular learning evolved from statistical and classical methods to tree ensembles and deep models; recent deep representation methods have become competitive with or superior to tree-based methods.Earlier deep approaches struggled to generalize beyond traditional tree-based methods, while newer architectures and representation strategies have shown promise.Source: §2.2 · p. 4

2.3 Challenges of Learning from Tabular Data

• Key challenges include mixed feature types and scales, permutation-invariant columns without spatial or sequential structure, missing or noisy data, and reliance on expert feature engineering.Numerical features may need scaling; categorical attributes need encoding, while missing and erroneous entries can bias predictions or reduce reliability.Source: §2.3 · p. 4
• Class imbalance, large or high-dimensional datasets, costly model and hyperparameter selection, and domain-specific privacy, fairness, and explainability constraints further complicate tabular modeling.Imbalanced labels can bias predictions toward majority classes; high dimensionality increases overfitting risk, and regulated domains may restrict model choices or require audits.Source: §2.3 · p. 5

2.4 Evaluation of a Tabular Method

• Single-task evaluation uses accuracy, AUC, F1, or ECE for classification and MSE, MAE, RMSE, or R2 for regression, with higher or lower values preferred depending on the metric.Accuracy, AUC, F1, and R2 are higher-is-better; error rate, ECE, MSE, MAE, and RMSE are lower-is-better.Source: §2.4 · p. 5
• Across datasets, average rank and significance tests are common, while PAMA and P95 capture best or near-best accuracy; normalized metrics and shifted geometric mean address scale and outlier sensitivity.Relative improvement compares against a baseline, and ELO-based pairwise competitions provide a dynamic model-to-model assessment.Source: §2.4 · pp. 5–6

2.5.1 Popular Tabular Benchmarks and Datasets

• Reliable benchmarks should span tasks, dataset sizes, domains, and feature types; semantic-rich collections support pre-training, while standard datasets are mainly used to evaluate methods.Semantic-enriched examples include UniTabE’s 7TB and 13 billion examples, OpenTabs’ approximately 46M samples, and TabLib’s 627M tables [183].Source: §2.5.1 · p. 6

2.5.2 Evaluation Protocols

• Evaluation protocols use validation-based tuning and early stopping, plus repeated random-seed splits, cross-validation, or both; holdout tuning can overfit, though reshuffling may improve its generalization efficiently.In TabZilla, 5-fold cross-validation altered earlier meta-feature findings; reshuffling can make holdout selection competitive with cross-validation at lower computational cost.Source: §2.5.2 · pp. 6–7

3.1 Advantages of deep representation learning

• DNNs learn hierarchical, high-order nonlinear feature interactions and process raw features end-to-end, reducing reliance on manual feature engineering,,,.The learned hierarchy captures lower-level interactions in initial layers and higher-order interactions in deeper layers.Source: §3.1 · p. 7
• DNNs support shared representations for related tasks and jointly model tabular features with images, audio, text, or time series,,,,,,.The survey describes these capabilities as useful for multi-task learning and multimodal pipelines.Source: §3.1 · p. 7
• Gradient-based optimization lets DNNs adapt to changing objectives and incremental data, while knowledge learned across tasks can transfer to related domains,,,,.Incremental training can integrate new data without retraining from scratch.Source: §3.1 · p. 7

3.2 Debates between Tree-Based Methods and DNNs

• Deep tabular methods can learn semantic representations and nonlinear predictors, yet often initially trail GBDT, and their advantage may shrink across more evaluation datasets,.The survey presents GBDT approaches as strong baselines.Source: §3.2 · p. 7
• Tree-based models handle high-frequency data efficiently and accept raw categorical features, whereas DNNs may need regularization, tuning, or categorical encodings,,,,,,.The cited approaches use scaling layers or periodic activations to help neural networks learn high-frequency functions,.Source: §3.2 · pp. 7–8
• Tree-based models can train and infer with fewer resources, and are generally more robust to noisy, missing, outlier, or skewed data than DNNs,,,,,,,,,.DNNs may require preprocessing, imputation, noise filtering, robust scaling, or outlier removal to address these data conditions.Source: §3.2 · p. 8
• Tree-based decision paths and feature importance are comparatively intuitive to explain, while interpretability methods and tree-behavior emulation seek to bridge the gap for DNNs,,,,,,,,,,.The survey highlights explainability as important in finance, healthcare, and regulated industries.Source: §3.2 · p. 8

4 TAXONOMY OF SPECIALIZED METHODS

• The taxonomy groups methods as specialized, transferable, and general, reflecting increasing generalization; specialized methods are further organized around features, samples, and objectives.Specialized methods learn from pure data rather than the feature semantics used by transferable methods; feature-aspect methods are deferred to Section 5.Source: §4 · p. 8

4.1 Sample-aspect Specialized Methods

• Sample-aspect methods either model sample relationships during representation training or retrieve and use neighbors at test time.Both strategies use inter-sample dependencies to improve predictions, but the latter directly retrieves neighbors when testing.Source: §4.1 · p. 9
• Sample-interaction methods learn representations from inter-sample relationships, using attention, non-parametric Transformers, Hopfield networks, prompts, or prototypes.SAINT adds inter-sample attention; NPT uses non-parametric Transformers; Hopular uses Hopfield networks with an end-to-end learned distance; Trompt treats sample information as prompts. PTaRL models global relationships through prototypes and projection to address entanglement and localization.Source: §4.1 · pp. 9–10
• Neighbor-retrieval methods build contexts from training samples to aid test predictions, with transparency, incremental learning, and robustness among their stated advantages.DNNR combines local gradient estimation and Taylor approximation with KNN; TabR encodes candidates and retrieves useful samples with an attention-like mechanism, while ModernNCA applies deep learning to neighborhood component analysis. The survey relates neighborhood methods to in-context learning, including TabPFN,, and notes LoCalPFN uses local context and local linear regression.Source: §4.1 · p. 10

4.2 Objective-aspect Specialized Methods

• Objective-aspect methods introduce inductive bias through training objectives or regularizers that modify the loss or constrain the model.PTaRL uses prototype-based projection with diversification and orthogonalization constraints; RLNs jointly optimize regularization coefficients and network weights using a Counterfactual Loss, producing sparse, interpretable networks. Dataset-specific “cocktails” combine 13 regularization techniques, and TANGOS regularizes neuron attributions to encourage specialization and orthogonality.Source: §4.2 · p. 10

5.1 Feature Encoding

• Feature tokenizers map inputs into low-dimensional embeddings, as in AutoInt, TabTransformer, SAINT, and FT-Transformer.Ordinal encoding is compact but can impose an artificial order, whereas one-hot encoding suits nominal categories but can expand dimensionality; MLP-PLR adds piecewise-linear and periodic numerical encodings, and proposes frequency reduction.Source: §5.1 · p. 11

5.2 Feature Selection

• Feature selection reduces overfitting and computation; TabNet uses sparse selection, while GRANDE learns hard axis-aligned tree ensembles with gradient descent.Tree-inspired deep methods adapt feature selection strategies to neural models.Source: §5.2 · p. 12

5.3 Feature Projection

• A well-regularized vanilla MLP can outperform specialized neural architectures and XGBoost across benchmarks.RTDL reports that properly tuned MLP and ResNet architectures can be competitive; specialized projection designs include SELU units, ExU units in NAMs, and BiSHop’s row- and column-wise Hopfield layers.Source: §5.3 · p. 12

5.4 Feature Interaction

• Feature interaction methods learn dependencies or higher-order crosses through automatic modules and approaches that mine implicit feature relationships.DCNv2 uses low-rank feature crosses and gating, while AutoInt stacks attention-based interaction layers; DANets groups correlated features, SwitchTab learns salient and shared information, ExcelFormer uses Semi-Permeable Attention, and AMFormer models additive and multiplicative interactions.Source: §5.4 · pp. 12–13

6 FROM SPECIALIZED TO TRANSFERABLE MODEL

• Transferable models pre-train on upstream tasks and adapt to downstream data, using homogeneous, heterogeneous, or cross-modal sources.They can improve target-task efficiency, reduce data and parameter requirements, and support few-shot learning; transfer is limited when source and target distributions or feature and class spaces lack sufficient overlap.Source: §6 · p. 13

6.1 Homogeneous Transferable Tabular Model

• Homogeneous transfer reuses models across related tabular tasks through supervised, masked-feature, contrastive, or hybrid pre-training objectives.Target-aware supervised pre-training can benefit downstream performance; self-supervised methods include reconstructing masked or selected features, and learning from augmented positive pairs,.Source: §6.1 · p. 14

6.2 Heterogeneous Transferable Tabular Model

• Heterogeneous transfer bridges differing feature or class spaces through shared-feature weights, feature meta-representations, optimal transport, or parameter-efficient adaptation.ReForm uses optimal transport to transform classifiers across feature and class spaces, while TabToken’s conditional contrastive loss improves feature-token embeddings and transferability.Source: §6.2 · p. 15

6.3 Reusing a Pre-trained Language Model

• Language models can support tabular transfer through feature tokenization, LLM-generated auxiliary features, or textual serialization of tables and tasks.These approaches rely on semantic context, which may be scarce; serialization can also exceed context-window limits as feature counts grow.Source: §6.3 · pp. 15–16

6.4 Reusing a Pre-trained Vision Model

• Vision-model transfer converts tabular data into images through dimensionality reduction, table reorganization, or image markers, enabling pre-trained vision models to be applied to tabular prediction.Methods include t-SNE or Bayesian metric scaling,, feature-arranged images,, and visual markers for feature values [242],.Source: §6.4 · p. 17

7 FROM TRANSFERABLE TO GENERAL MODEL

• General models target zero-shot prediction across heterogeneous downstream tabular tasks, without the additional adaptation function or fine-tuning required by transferable models.They aim to generalize better than training directly on a downstream dataset or adapting a transferable model.Source: §7 · p. 17
• Tabular heterogeneity makes joint pretraining and direct transfer difficult, prompting strategies to improve model adaptability or homogenize tabular formats.Datasets vary in dimensionality and feature meaning—even within the same application—unlike vision and text data with more shared vocabularies and relationships.Source: §7 · pp. 17–18

7.1 Raw-Feature-based General Models

• Raw-feature-based general models either standardize datasets into a shared format or tailor model adaptability to tabular tasks.TabPTM uses meta-representation for a uniform format, while HyperFast maps datasets to classifier weights and uses random projections; MotherNet accelerates weight generation with Transformer-like modules.Source: §7.1 · p. 18

7.2 TabPFN Variants

• TabPFN, uses transformer in-context learning, while TabPFN v2 adds feature tokenization and two-way attention for heterogeneous tables.Version 1 is pretrained on synthetic datasets and is suggested for N < 1000, d < 100, and C < 10; v2 increases context limits to N < 10000 and d < 500 and adds automatic feature engineering and post-hoc ensembles.Source: §7.2 · p. 18
• TabPFN variants explore new pretraining data, scalability techniques, and downstream adaptation, though some require fine-tuning and are not general models.Examples include TabForestPFN, TabDPT, APT, and TabICL; efficiency approaches compress training data, select examples, or use linear attention or state-space models, while adaptation methods include context adjustment and partial fine-tuning.Source: §7.2 · pp. 18–19

7.3 Semantics-based General Models

• Semantics-based models use column names and task semantics to project heterogeneous tasks into a shared language space for unified prediction.Examples include TabuLa-8B, GTL, GTL-S, retrieval-augmented extensions, and MediTab, which aligns out-domain data to target tasks through a learn, annotate, and refinement pipeline.Source: §7.3 · p. 19

8 TABULAR ENSEMBLE METHODS

• Classical bagging and boosting ensembles reduce bias and variance and improve robustness, while deep tabular ensembles use joint training or post-hoc prediction fusion.Training multiple deep models or sub-models can be computationally expensive and time-consuming.Source: §8 · p. 19

8.1 Joint-Training Ensembles

• Joint-training ensembles combine linear and nonlinear, tree-based and deep, or tree-node predictions,, within one process.Tree-mimic methods mix predictions from multiple tree nodes to enhance robustness.Source: §8.1 · p. 19
• TabM uses BatchEnsemble to efficiently diversify MLP base learners, while BETA tunes multiple feature projections into pre-trained TabPFN.Both approaches limit additional learnable parameters; BETA aggregates projected-set predictions and applies BatchEnsemble.Source: §8.1 · p. 19
• LLM-Boost and PFN-Boost start with LLM or TabPFN base learners, then sequentially add gradient-boosted decision-tree learners for scalability.The approach combines the prior knowledge of LLMs and TabPFN with the scalability of gradient-boosted decision trees.Source: §8.1 · p. 19

8.2 Post-Hoc Ensembles

• Post-hoc ensembles aggregate independently trained models or leverage complementary prediction styles from LLM-based methods and deep tabular models,,.Seed-based bagging improves robustness but incurs high computational overhead.Source: §8.2 · p. 19
• TabPFN feature permutations and TabPFN v2 perturbations generate diverse predictions without separately training multiple models.TabPFN v2 varies feature encoding and quantization, shuffles categorical features, and applies SVD compression, outlier removal, or power transformations such as Yeo–Johnson.Source: §8.2 · pp. 19–20
• TabPFN v2 combines fixed configurations with greedy weight selection; for large datasets, TabPFN-Bagging, averages context groups, while BoostPFN outperforms standard PFNs.BoostPFN treats TabPFN v1 as weak learners, each using a subset of training data as context; these strategies target generalization and variance reduction without excessive training costs.Source: §8.2 · p. 20

Clustering, Anomaly Detection, and Tabular Generation

• Extensions address clustering, anomaly detection, and tabular generation with interpretable assignments, contextual anomaly methods, and synthetic-data modeling.IDC predicts interpretable cluster assignments, while TableDC uses Mahalanobis distance to address overlapping clusters. Anomaly work includes mutual-information learning, the ADBench benchmark, and LLMs; diffusion models improve fidelity over GANs, while causal priors and neuro-symbolic models address synthesis challenges.Source: §9 · p. 20

Interpretability

• Interpretability methods range from tree-based explanations to deep models that expose feature effects, attention, or additive feature-wise contributions.GBDTs support feature importance, decision paths, SHAP values [310], and partial dependence plots; NeC4.5 uses a neural ensemble-generated training set. NAMs learn feature-wise networks, TabNet uses sparse sequential feature masks, and TabTransformer and FT-Transformer expose attention patterns; NODE, NODE-GAM, and DOFEN combine oblivious-tree ensembles with gradient optimization and hierarchical representations.Source: §9 · p. 20

Open-Environment Tabular Machine Learning

• Open-environment tabular learning addresses domain and temporal shifts through adaptation, robustness methods, temporal protocols, and temporal embeddings.When target-domain data is unavailable, domain generalization, domain robustness, label robustness, and ensembles aim to improve robustness. Under temporal shifts, most methods degrade, while TabM shows relative robustness; a refined protocol and temporal embedding can restore ModernNCA’s competitiveness.Source: §9 · pp. 20–21

Multi-modal Learning with Tabular Data

• Image–tabular methods use cross-modal learning to support diagnosis, transfer tabular expertise to image-only inference, and learn across incomplete heterogeneous data.MMCL uses contrastive self-supervision, CHARMS transfers tabular expert knowledge to image predictions without tabular inference inputs, and TIP combines a tabular encoder with a multimodal interaction module.Source: §9 · p. 21

Tabular Understanding

• Tabular understanding spans table detection, structure recognition, and question answering, with OCR-based and OCR-free approaches applied to these tasks.Donut uses synthetic document images for pretraining; Monkey and TextMonkey use shifted-window attention and similarity-based token filtering; mPLUG-DocOwl, TabPedia, and direct LLM prompting target document or visual table understanding.Source: §9 · p. 21

The Ability to Handle Dynamic and Open Environments

• Tabular foundation models must adapt to evolving distributions, imbalanced data, and newly emerging classes while maintaining accurate, real-time predictions,,,.Source: §10 · p. 21

The Coverage and Scope of Tabular Foundation Models

• Future work should test prediction strategies beyond in-context learning, extend models to clustering, imputation, outlier detection, and table-based QA, and study model-size/performance trade-offs.Source: §10 · p. 21

Will Foundation Models Always Help?

• A single foundation model may average predictive ability across tasks and lose task-specific expertise, motivating a model zoo that selects, adapts, and combines pretrained models,.Source: §10 · p. 21

Model Efficiency

• Foundation models need to scale to exceptionally large or sparse, high-dimensional data, accelerate real-time inference, and reduce compute costs through pruning, quantization, and efficient training,,,.Source: §10 · pp. 21–22

Bridging the Gap Between Tabular Data and Other Modalities

• Integrating tabular models with vision and language foundation models via joint embeddings or cross-modal attention could support tasks combining structured data with images or text.Source: §10 · p. 22

• Existing methods are categorized as specialized, transferable, or general models according to their generalization capabilities.Source: §11 · p. 22
• The survey also discusses ensemble techniques and extensions, including open-environment and multimodal tabular learning.Source: §11 · p. 22
• The survey aims to support understanding of the field and inspire more robust and generalizable tabular learning methods.Source: §11 · p. 22