PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Learning for Tabular Data: A Comprehensive Survey

Jun-Peng Jiang, Si-Yang Liu, Hao-Run Cai, Qile Zhou, Han-Jia Ye

arXiv:2504.16109v1cs.LG

TL;DR

Tabular learning must handle heterogeneous, imperfect datasets while achieving reliable generalization beyond individual tasks and distributions. This survey synthesizes deep tabular representation learning through taxonomies of specialized, transferable, and general models, alongside ensembles and extensions. It concludes that deep representation learning is promising, while performance can remain sensitive to strong tree-based baselines and expanding evaluation settings.

  • Problem

    Tabular datasets contain heterogeneous attributes and quality challenges, and DNN performance can be unstable relative to classical tree-based methods across datasets.

  • Method

    The survey organizes tabular methods by generalization capability, develops a feature-, sample-, and objective-based taxonomy, and reviews transfer, foundation, ensemble, and extension strategies.

  • Results

    The survey presents deep tabular representation learning as a promising direction, with recent techniques demonstrating competitive or superior performance to traditional tree-based models.

  • Takeaways & Limitations

    The resulting framework serves as a reference for understanding current tabular representation learning and guiding future research and applications.

  • Takeaways & Limitations

    Deep tabular methods often initially struggle to surpass GBDT, and their advantages may diminish as the number of evaluation datasets increases.

Abstract

from arXiv · show

Tabular data, structured as rows and columns, is among the most prevalent data types in machine learning classification and regression applications. Models for learning from tabular data have continuously evolved, with Deep Neural Networks (DNNs) recently demonstrating promising results through their capability of representation learning. In this survey, we systematically introduce the field of tabular representation learning, covering the background, challenges, and benchmarks, along with the pros and cons of using DNNs. We organize existing methods into three main categories according to their generalization capabilities: specialized, transferable, and general models. Specialized models focus on tasks where training and evaluation occur within the same data distribution. We introduce a hierarchical taxonomy for specialized models based on the key aspects of tabular data -- features, samples, and objectives -- and delve into detailed strategies for obtaining high-quality feature- and sample-level representations. Transferable models are pre-trained on one or more datasets and subsequently fine-tuned on downstream tasks, leveraging knowledge acquired from homogeneous or heterogeneous sources, or even cross-modalities such as vision and language. General models, also known as tabular foundation models, extend this concept further, allowing direct application to downstream tasks without fine-tuning. We group these general models based on the strategies used to adapt across heterogeneous datasets. Additionally, we explore ensemble methods, which integrate the strengths of multiple tabular models. Finally, we discuss representative extensions of tabular learning, including open-environment tabular machine learning, multimodal learning with tabular data, and tabular understanding. More information can be found in the following repository: https://github.com/LAMDA-Tabular/Tabular-Survey.

1 INTRODUCTION

This introduction surveys tabular representation learning, motivated by the prevalence and complexity of tabular data and recent progress in deep models. It organizes methods by generalization capability and presents unified taxonomies, benchmarks, ensembles, and extensions.

  • Background: Tabular data uses rows for instances and columns for numerical or categorical features, supporting widespread classification and regression applications across diverse domains.The survey highlights finance, healthcare, education, recommendation systems, and scientific research as application areas.
  • Challenges: Tabular datasets combine heterogeneous attributes with practical quality challenges, including noise, missing values, outliers, inaccuracies, and privacy constraints.
  • Deep Tabular Learning: Modern DNNs learn complex feature interactions and rich representations, increasingly rivaling or surpassing traditional tree-based models in tabular applications.Earlier DNNs often struggled against tree-based methods, while later architectures, training strategies, and representations improved performance.
  • Method Taxonomy: The survey classifies methods as specialized, transferable, or general according to their training scope and generalization capabilities.Specialized methods operate within one distribution, transferable methods fine-tune pre-trained knowledge, and general models apply directly without additional fine-tuning.
  • Method Taxonomy: Its specialized-method taxonomy centers on features, samples, and objectives to connect scattered architectural categories and strategies for feature- and sample-level representations.The framework is intended to support clearer comparisons and future model design.
  • Method Taxonomy: Transferable models address limited target data or computation by reusing pre-trained knowledge from homogeneous, heterogeneous, or cross-modal sources.The survey emphasizes feature- and label-space heterogeneity as a central transfer challenge and includes semantic strategies for bridging attribute spaces.
  • Extensions: General tabular models, ensembles, and broader extensions expand the survey beyond specialized and transferable methods toward zero-shot use, combined model strengths, open environments, and multimodal learning.The extensions also include clustering, anomaly detection, generation and imputation, interpretability, and tabular understanding.

2 BACKGROUND

This section defines supervised tabular learning, traces its classical foundations, and describes the challenges and evaluation practices used to compare methods across datasets.

  • Learning with Tabular Data: A supervised tabular dataset contains N examples and d features, with instances represented by numerical or categorical feature values and labels for classification or regression.The survey focuses primarily on standard classification and regression rather than ordinal regression.
  • Learning with Tabular Data: Tabular learning seeks a mapping f that predicts each label while minimizing loss and regularization under structural risk minimization.The loss measures prediction discrepancy, while regularization restricts model complexity and supports generalization to unseen instances from the same distribution.
  • Learning with Tabular Data: Classical methods implement predictions with linear or non-parametric functions, whereas deep tabular models learn embeddings ϕ(x_i) before applying a classifier.The survey gives linear models, KNN, and neural networks as representative implementations of f.
  • History of Tabular Data: Tabular machine learning evolved from database, web, and document tables toward vector-based datasets explicitly distinguished from images, text, audio, and video.Earlier work included table extraction, parsing, understanding, association-rule discovery, and classical predictive modeling.
  • Challenges of Learning from Tabular Data: Heterogeneous feature types, class imbalance, and large or high-dimensional datasets create major modeling and generalization challenges.These challenges include distinct handling for numerical and categorical attributes, biased learning toward majority classes, overfitting risk, and computational demands.
  • Evaluation of a Tabular Method: Evaluation combines task-specific metrics with aggregate comparisons across datasets because no single model consistently excels in all tabular scenarios.Classification uses metrics such as accuracy, AUC, F1, and ECE; regression uses MSE, MAE, RMSE, and R2, while multi-dataset studies use average rank and related measures.

3 FROM CLASSICAL TO DEEP METHOD

Deep neural networks extend tabular learning through representation learning, interaction modeling, multitask and multimodal integration, but tree-based methods remain strong competitors under several practical conditions.

  • Advantages of deep representation learning: DNNs can model high-order nonlinear feature interactions and learn hierarchical representations that are difficult for traditional linear or tree-based models.Their gradient-based training also supports flexible adaptation to changing objectives.
  • Advantages of deep representation learning: Deep tabular models support multitask learning and integration with images, audio, text, or time-series data through shared representations.The survey highlights multimodal pipelines and joint learning across heterogeneous data sources as important use cases.
  • Debates between Tree-Based Methods and DNNs: Tree-based ensembles often remain stronger initial baselines than deep tabular methods, with deep-model advantages diminishing as evaluation spans more datasets.Gradient Boosted Decision Trees are repeatedly treated as strong baselines in tabular learning studies.
  • Debates between Tree-Based Methods and DNNs: Tree-based methods retain practical advantages through efficient handling of high-frequency data, mixed feature types, noisy or missing values, outliers, and skewed distributions.They can also require fewer computational resources and provide more transparent decision paths and feature importance estimates.
  • Debates between Tree-Based Methods and DNNs: Scaling layers and periodic activation functions are cited as approaches for improving neural networks on high-frequency tabular functions.These methods introduce inductive bias or activation designs intended to address limitations associated with high-frequency data.

4 TAXONOMY OF SPECIALIZED METHODS

The survey organizes deep tabular methods by increasing generalization capability into specialized, transferable, and general models. Specialized methods are further structured around features, samples, and objectives, with sample interactions retrieving neighboring rows and objective methods shaping losses or regularization.

  • Specialized, transferable, and general methods form a taxonomy ordered by increasing generalization capability.
  • Specialized methods learn from pure data for tasks whose training and evaluation data share a distribution.
  • The specialized taxonomy covers feature, sample, and objective aspects of tabular data.
  • Objective Aspect: Objective-aspect methods introduce inductive bias by modifying the loss function or adding regularization to guide learning.
  • Sample Aspect: Retrieval-based sample methods aggregate information from relevant training samples before a prediction head produces the output.
  • Sample Aspect: Sample-aspect methods model inter-sample dependencies either during representation learning or through retrieval-based prediction.

5 FEATURE-ASPECT SPECIALIZED METHODS

Feature-aspect methods transform raw numerical and categorical inputs into more informative representations through encoding, selection, projection, and interaction. The survey describes tokenization and tree-inspired selection strategies, while noting that well-tuned or well-regularized simple architectures can perform competitively.

  • Feature-aspect methods transform raw tabular inputs through encoding, selection, projection, and interaction.
  • Feature Encoding: Ordinal encoding maps ordered categories to integers, whereas one-hot encoding creates a binary column for each category.
  • Feature Encoding: Target encoding assigns categories values based on target means, while leave-one-out embedding excludes the current row to reduce overfitting.
  • Feature Encoding: Feature tokenizers convert sparse, high-dimensional features into low-dimensional embeddings that support interactions among feature types.
  • Feature Selection: Feature selection reduces dimensionality by retaining valuable information, which can prevent overfitting, improve generalization, and lower computational complexity.
  • Feature Selection: Tree-inspired methods such as TabNet and GRANDE incorporate sparse selection or hard splits while retaining neural representation learning.
  • Feature Projection: With proper hyperparameter tuning, simple MLP and ResNet architectures can achieve competitive performance on tabular benchmarks.
  • Feature Projection: A well-regularized vanilla MLP can outperform specialized neural architectures and surpass XGBoost across benchmark datasets.

6 FROM SPECIALIZED TO TRANSFERABLE MODEL

Transferable tabular methods reuse knowledge from pre-trained models across source and target tasks, with adaptation strategies addressing differences in distributions, features, architectures, and modalities. The survey organizes these approaches around homogeneous and heterogeneous transfer, including supervised or self-supervised pre-training and reuse of language or vision models.

  • Transferable Models: Pre-trained models can reduce the data and resource requirements of learning target tabular tasks.Transfer typically involves pre-training on one or more upstream tasks followed by adaptation to the downstream task.
  • Transfer Types: Homogeneous transfer reuses models from related tabular tasks, while heterogeneous transfer addresses differences between source and target tasks.The survey distinguishes these categories according to the relationship between source and target task dimensions, distributions, and model families.
  • Transfer Types: Transferable methods assume sufficient similarity between source and target distributions or overlap in their feature or class spaces.Shared dimensions, features, or classes provide the basis for transferring knowledge across tabular datasets.
  • Homogeneous Transfer: Homogeneous transferable models may use supervised learning, masked language modeling, contrastive pre-training, or hybrid objectives.These objectives provide alternative ways to construct the pre-trained model before downstream adaptation.
  • Heterogeneous Transfer: Heterogeneous transfer includes feature-alignment strategies, weight regularization, pseudo-features, and selective fine-tuning of pre-trained parameters.Some approaches copy or regularize shared feature weights, while others create pseudo-values or tune only a limited subset of parameters.
  • Cross-Modal Transfer: Language and vision models extend tabular transfer by using semantic feature information or transforming tables into images for pre-trained model reuse.Vision-based methods apply dimensionality reduction, table reorganization, and image markers, while language-based methods operate through feature tokenization, engineering, or serialization.

7 FROM TRANSFERABLE TO GENERAL MODEL

General tabular models, or tabular foundation models, extend transferable models by handling heterogeneous downstream tasks directly without fine-tuning. The survey organizes their adaptability strategies around homogenizing data formats, adapting model architectures, and leveraging feature or task semantics.

  • General Models: General models apply pretrained tabular models directly to heterogeneous downstream tasks without additional fine-tuning.This zero-shot capability distinguishes them from transferable models and is intended to support diverse feature and class spaces.
  • Adaptation Strategies: Tabular heterogeneity is addressed either by improving model adaptability or by homogenizing diverse tabular formats.The survey groups general models according to data-centric and model-centric strategies for achieving generalizability.
  • Adaptation Strategies: Data-centric approaches standardize datasets into homogeneous forms, while model-centric approaches tailor models to specific tasks.TabPTM uses meta-representation for uniform formatting, whereas HyperFast learns a mapping from datasets to classifier weights and uses random projection for varying dimensions.
  • TabPFN Family: TabPFN uses transformer in-context learning to predict labels from training examples, with later versions introducing feature tokenization and two-way attention.TabPFN is pretrained on synthetically generated datasets, while TabPFN v2 projects cells into shared vectors and models row- and column-wise relationships.
  • Scope Boundary: Some TabPFN variants require additional fine-tuning and are therefore not classified as general tabular models.The survey places these variants in the subsection because of their strong relationship with TabPFN.
  • Semantic Adaptation: Semantic-based methods project heterogeneous tasks into a shared language space using information such as column names.Examples include TabuLa-8B for tabular prediction and GTL for instruction-oriented tabular modeling.

8 TABULAR ENSEMBLE METHODS

Tabular ensemble methods combine diverse learners to improve predictive robustness and accuracy. The survey distinguishes joint-training ensembles from post-hoc ensembles and highlights efficiency-oriented strategies for producing model diversity.

  • Overview: Ensemble learning improves generalization by leveraging diversity among multiple base learners, reducing bias or variance and improving robustness.Random Forest and AdaBoost exemplify bagging and boosting for tabular data.
  • Taxonomy: Deep tabular ensembles are grouped into joint-training methods that aggregate sub-networks and post-hoc methods that fuse predictions from pretrained models.Training multiple deep models or sub-models can be computationally expensive and time-consuming.
  • Joint-Training Ensembles: Joint-training ensembles combine model types such as linear, nonlinear, tree-based, and neural approaches within one training process.Tree-mimic methods mix predictions from multiple tree nodes to enhance robustness.
  • Efficiency and Hybridization: Parameter-efficient ensembles such as TabM use BatchEnsemble to create diverse MLP learners without a large increase in learnable parameters.Hybrid LLM-Boost and PFN-Boost methods combine strong prior models with the scalability of gradient-boosted decision trees.
  • Post-Hoc Ensembles: Post-hoc ensembles aggregate predictions from independently trained models, perturbations, or fixed portfolios to improve robustness and generalization.Feature permutation can generate diverse predictions from one TabPFN, while portfolio-based selection greedily learns weights over TabPFN configurations.

9 EXTENSIONS

The survey reviews extensions of deep tabular learning for synthesis, interpretability, temporal shifts, multimodal learning, and table understanding. These extensions address complex data-generation, deployment, and perception settings beyond standard supervised prediction.

  • Data Generation: Tabular diffusion models iteratively refine synthetic data to capture high-dimensional correlations and can outperform GANs in data fidelity.Synthetic-data methods still face challenges balancing realism with privacy and scaling across heterogeneous data types.
  • Interpretability: Neural additive models and TabNet introduce interpretability mechanisms that expose feature-specific effects or sequential attention patterns.NAMs combine neural expressivity with generalized additive-model intelligibility by learning jointly trained feature-specific networks.
  • Temporal Shifts: Temporal evaluation protocols and temporal embeddings address distribution shifts in real-world tabular data.Most methods degrade under temporal shifts, while TabM shows relative robustness and refined protocols can restore competitiveness for methods such as ModernNCA.
  • Multimodal Learning: Multimodal tabular learning combines tabular information with modalities such as medical images to support diagnosis under large-data and high-annotation-cost conditions.The survey identifies contrastive self-supervised learning as one approach for these interactions.
  • Tabular Understanding: Tabular understanding includes table detection, table structure recognition, and table question answering from document images.Recent systems use synthetic document generation, shifted-window attention, token filtering, or OCR-free document understanding.

10 DISCUSSIONS

Future tabular foundation models must operate in dynamic environments with imbalanced data, distribution shifts, and emerging classes. Practical deployment also requires broader coverage and lower computational demands.

  • Dynamic Environments: Dynamic environments require tabular foundation models to adapt to changing distributions, imbalanced classes, and newly emerging classes.The models must continue providing accurate predictions while updating their outputs in real time.
  • Coverage and Scope: Current tabular foundation models perform strongly on unseen classification and regression tasks, but their coverage and scope remain open questions.The survey frames broader task and environment coverage as an important future direction.
  • Efficiency: Foundation-model deployment requires efficient inference and reduced training and deployment resource usage.The survey identifies pruning, quantization, and efficient training algorithms as relevant directions for practical accessibility.

11 CONCLUSION

The survey provides a comprehensive overview of deep tabular representation learning and organizes methods by their generalization capabilities. It also covers ensembles, extensions, and future directions for developing more robust and generalizable tabular learning methods.

  • The survey covers the background, challenges, evaluation benchmarks, and trade-offs between tree-based models and DNNs for tabular representation learning.
  • Existing methods are categorized as specialized, transferable, or general models according to their generalization capabilities.
  • The survey discusses ensemble techniques and extensions including open-environment and multimodal tabular learning.
  • The authors position the survey as a reference for the field and as a basis for progress toward more robust and generalizable tabular learning methods.
Loading 2504.16109v1…