Source-linked AI summary
TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling
Yury Gorishniy, Akim Kotelnikov, Artem Babenko
TL;DR
Tabular deep learning lacks a clearly established practical baseline beyond simple MLPs, with performance consistency and efficiency still requiring broader evaluation. The paper introduces TabM, which uses parameter-efficient ensembling to imitate an MLP ensemble in one model, and finds that it provides the strongest tabular deep-learning performance while offering a practical efficiency profile. Its implicit predictions are weak individually but powerful collectively.
Problem
Tabular deep learning lacks an obvious go-to baseline beyond simple MLPs, while performance consistency, efficiency, and real-world applicability remain insufficiently characterized.
Method
TabM combines an MLP with parameter-efficient ensembling, sharing most weights while producing multiple predictions and training the implicit MLPs collectively.
Results
TabM demonstrates the best performance among tabular deep-learning models and provides an appealing performance-efficiency trade-off relative to attention- and retrieval-based models.
Takeaways & Limitations
TabM can serve as a powerful, efficient, and reliable go-to baseline for tabular deep learning.
Takeaways & Limitations
The paper leaves evaluation of TabM for uncertainty estimation and out-of-distribution detection as future work.
Abstract
from arXiv · showhide
Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval-augmented methods. This study highlights a major, yet so far overlooked opportunity for designing substantially better MLP-based tabular architectures. Namely, our new model TabM relies on efficient ensembling, where one TabM efficiently imitates an ensemble of MLPs and produces multiple predictions per object. Compared to a traditional deep ensemble, in TabM, the underlying implicit MLPs are trained simultaneously, and (by default) share most of their parameters, which results in significantly better performance and efficiency. Using TabM as a new baseline, we perform a large-scale evaluation of tabular DL architectures on public benchmarks in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light. Generally, we show that MLPs, including TabM, form a line of stronger and more practical models compared to attention- and retrieval-based architectures. In particular, we find that TabM demonstrates the best performance among tabular DL models. Then, we conduct an empirical analysis on the ensemble-like nature of TabM. We observe that the multiple predictions of TabM are weak individually, but powerful collectively. Overall, our work brings an impactful technique to tabular DL and advances the performance-efficiency trade-off with TabM -- a simple and powerful baseline for researchers and practitioners.
1 INTRODUCTION
Tabular deep learning lacks a clearly established practical baseline beyond simple MLPs, while efficiency and real-world transfer remain insufficiently characterized. The paper proposes efficient ensembling as an overlooked route to stronger MLP-based models and introduces TabM as a powerful, practical architecture.
- Motivation: Tabular deep learning lacks an obvious go-to baseline beyond simple MLP-like architectures.The literature does not consistently analyze improvements over simple baselines, efficiency properties, or transfer from academic benchmarks to real-world tasks.
- Motivation: Efficient ensembling shares most weights between ensemble members, making plain MLPs stronger while retaining simplicity and efficiency.The paper identifies this parameter-efficient approach as an overlooked path toward powerful, reliable, and reasonably efficient tabular models.
- Motivation: MLP with BatchEnsemble outperforms popular attention-based models such as FT-Transformer while being simpler and more efficient.This result motivates efficient ensembling as a practical opportunity for tabular deep learning.
- Contribution: TabM combines an MLP with an ensembling-like technique to provide high performance and simple implementation.The authors present TabM as a powerful and practical model for researchers and practitioners, informally paralleling GBDT’s ensemble structure.
- Contributions: TabM produces multiple predictions per object, competes with GBDT, and outperforms prior tabular deep-learning models while being more efficient than attention- and retrieval-based architectures.The paper also evaluates tabular deep-learning methods across performance ranks, score distributions, training time, and inference throughput.
2 RELATED WORK
Prior work spans decision-tree baselines, attention and retrieval architectures, MLP-like models, and ensemble-inspired methods for tabular learning. TabM differs through its computation flow: one MLP-based model imitates an ensemble while remaining efficient on large datasets.
- Decision-tree-based models: GBDT is a strong and efficient tabular baseline formed by an ensemble of decision trees, whereas TabM is a parameter-efficient ensemble of MLPs.
- Tabular deep learning architectures: Prior tabular deep-learning architectures include attention-based, retrieval-augmented, and MLP-like models.
- Tabular deep learning architectures: TabM imitates an ensemble of MLPs by producing multiple independently trained predictions within one model.Its computation flow distinguishes it from prior ensemble-like attempts in tabular deep learning.
- Improving tabular MLP-like models: Tabular MLPs show strong potential, but overfitting and optimization issues can limit their performance.Prior work addresses these issues through architectural modifications, regularization, and custom training techniques.
- Deep ensembles: Deep ensembles often outperform single models but require training and using multiple independently trained models.Their prediction is the mean of their members, and their power is associated with member diversity.
- Parameter-efficient deep ensembles: Parameter-efficient ensemble models produce multiple predictions while sharing much of their weights across implicit ensemble members.These approaches seek ensemble-like performance at lower computational cost than conventional deep ensembles.
3 TABM
TabM is a single MLP-based model that represents an ensemble of k submodels, trains them in parallel, and shares most weights by default. Its design combines efficient ensembling with ensemble-aware training choices to improve performance while limiting model and runtime overhead.
- BatchEnsemble shares the main linear-layer weights while using non-shared adapters R, S, and B for each implicit submodel.The packed formulation applies the shared matrix and member-specific adapters in parallel.
- TabM represents an ensemble of k MLPs that process k inputs in parallel within one model.
- TabM uses MLPs because its ensemble size is k = 32 and MLP-based parallel execution provides better efficiency than attention- or retrieval-based alternatives.
- Unlike a traditional deep ensemble, TabM trains its implicit MLPs simultaneously and shares most weights by default.Traditional MLP×k trains independently initialized MLPs and averages their predictions.
- TabM's training procedure supports ensemble-aware stopping and hyperparameter tuning, avoiding choices optimized only for individual MLPs.For TabMpacked, parallel training permits stopping when the ensemble score is optimal and tuning hyperparameters for the ensemble.
- TabMmini keeps only the first adapter R and performs slightly better than TabMnaive despite using one adapter instead of 3N.The first adapter maps the k inputs into different representation spaces before feature mixing.
- The paper attributes TabM's gains to simultaneous ensemble-member training and weight sharing, with the latter apparently acting as regularization.
4 EVALUATING TABULAR DEEP LEARNING ARCHITECTURES
The evaluation compares tabular models across 46 datasets using performance and efficiency measures. TabM ranks among the strongest tabular DL models, while many alternative DL methods are inconsistent relative to plain MLP.
- The evaluation compares MLP, attention-, retrieval-, and tree-based baselines on tabular datasets.
- Task performance: TabM is a top-tier DL model and holds leadership among DL models across the evaluated datasets.
- Task performance: Many DL methods are no better or worse than MLP on a non-negligible number of datasets, indicating less reliable performance.
- Summary: MLP-like models, including TabM, form a representative set of tabular DL baselines.
- Efficiency: The study evaluates training time and inference throughput alongside task performance.
5 ANALYSIS
The analysis examines TabM’s implicit submodels, their collective prediction, post-training selection, and dependence on ensemble size. Its performance comes from combining weak but diverse submodels, although model size and k constrain productive configurations.
- Submodel predictions: TabM predicts by averaging k implicit submodels that share most of their weights.
- Performance and training dynamics: The collective prediction has better training and test losses than individual submodels, despite the baseline MLP eventually achieving lower training loss.
- Performance and training dynamics: Individually, TabM submodels appear overfitted, whereas their collective prediction generalizes substantially better than the baseline MLP.
- Summary: TabM draws its power from the collective prediction of weak but diverse submodels.
- Selecting submodels after training: Greedy post-training selection yields TabM[G], which is slightly behind vanilla TabM and retains 8.8 ± 6.6 of the initial submodels on average.
- Dependence on k: Larger TabM configurations can accommodate more submodels effectively, but excessive k, narrow width, or shallow depth can reduce performance.
6 CONCLUSION & FUTURE WORK
The paper shows that parameter-efficient ensembling substantially strengthens tabular MLPs, yielding TabM as a high-performing and efficient baseline. It also identifies future directions involving other domains and uncertainty-related tasks.
- Conclusion: TabM is a simple MLP-based model with state-of-the-art performance and is presented as a powerful, efficient tabular DL baseline.The paper attributes this result to parameter-efficient ensembling.
- Conclusion: Parameter-efficient ensembling makes tabular MLPs substantially more effective.The conclusion highlights this as the central empirical insight.
- Conclusion: TabM’s implicit submodels are analyzed individually alongside the technical details behind the architecture.
- Future work: Future work includes applying efficient ensembles beyond tabular data and evaluating TabM for uncertainty estimation and out-of-distribution detection.
A.4 LIMITATIONS AND PRACTICAL CONSIDERATIONS
The practical discussion describes TabM’s relationship to BatchEnsemble, its simple backbone, and implementation considerations for multiple prediction branches and additional baselines.
- Limitations: TabM does not introduce new limitations beyond those already associated with BatchEnsemble.
- Backbone considerations: TabM uses a simple MLP backbone, while more advanced backbones remain possible but may require additional care.
- Backbone considerations: Normalization-based backbones may need separate trainable affine adapters for each implicit submodel, and the best strategy remains unevaluated.
- Practical considerations: Multiple prediction branches produce a set of object embeddings, so combining them into one embedding may require additional experiments.
- Additional baselines: The evaluation additionally considers MLP-PLR, ResNet, SNN, DCNv2, AutoInt, MLP-Mixer, Trompt, and TabPFN where applicable.TabPFN is unavailable for regression tasks and was not competitive on the benchmark.
B.2 TASK PERFORMANCE
The supplementary task-performance analysis extends the main comparison with additional baselines, critical-difference analysis, and efficiency measurements under smaller-dataset and large-batch settings.
- Additional comparisons: None of the additional baselines changes the paper’s main task-performance story.
- Statistical comparison: The critical-difference diagram is computed from exactly the results used for Figure 3.
- Additional comparisons: Figure 8 ranks models only over 37 randomly split datasets because several baselines were evaluated on just one domain-aware dataset.
- Efficiency: The benchmark provides a high-level efficiency intuition using transparently obtained tuned configurations rather than exhaustive hardware, budget, and batch-size comparisons.
- Efficiency: Figure 10 reports training time on datasets with fewer than 100K objects and GPU inference throughput at each model’s maximum batch size.
C DATASETS
The benchmark combines 46 public datasets with standardized preprocessing, tuning, repeated-seed evaluation, and rank calculations designed to compare tabular models across varied task settings.
- Datasets: The benchmark contains 46 publicly available datasets drawn from prior tabular-learning benchmarks.The collection includes datasets from Gorishniy et al., Grinsztajn et al., and TabReD.
- Dataset curation: Datasets with incorrect splits or label leakage were excluded, and one anonymous dataset was removed after abnormal results could not be investigated.
- Preprocessing: All deep-learning models use the same preprocessing, with quantile normalization for numerical features by default and one-hot encoding for categorical features.
- Evaluation protocol: Hyperparameters are tuned mainly with Optuna’s TPE sampler, and tuned configurations are evaluated across multiple random seeds.
- Ranking: Model ranks treat small improvements as non-wins by requiring a mean-score margin larger than the compared model’s standard deviation.
D.6 IMPLEMENTATION DETAILS OF SUBSECTION 5.2
TabM[G] is formed by greedily selecting submodels from a trained TabM according to collective validation performance. The section also specifies the ROC-AUC evaluation scope for classification datasets.
- TabM[G]: TabM[G] greedily selects submodels from a trained TabM, starting with the best one and stopping when both validation conditions are met.Selection stops when adding a submodel no longer improves collective validation performance and the collective metric exceeds that of the initial all-submodel model.
- Evaluation: Figure 11 evaluates classification datasets with ROC-AUC and excludes the benchmark’s two multiclass datasets.
- TabM[G]: A candidate submodel is ranked by the improvement it produces in the aggregated prediction, not by its individual validation metric.
D.7 IMPLEMENTATION DETAILS OF SUBSECTION 5.3
The implementation details define the benchmark protocols, model-specific embeddings, tuning spaces, and reused results across datasets. Experiments average scores over five seeds, while some baselines have explicit applicability and sampling constraints.
- Evaluation protocol: Figure 7 reports mean percentage improvements over MLP across 17 datasets, using dropout 0.1, per-k learning-rate tuning, and five seeds per dataset.
- Feature embeddings: TabM† and TabM†mini are TabM versions with non-linear feature embeddings, using the updated piecewise-linear implementation.
- Feature embeddings: MLP variants use updated piecewise-linear embeddings or periodic embeddings, with only the non-lite periodic-embedding results reported in the paper.
- Baseline protocols: Baseline results are frequently reused from prior work when training and evaluation setups are compatible, including TabR and GBDTs.The GBDT results were additionally confirmed reproducible in the authors’ setup.
- Baseline constraints: TabPFN uses different 10K-sample training subsamples across random seeds and is unavailable for regression and datasets with more than 100 features.
E PER-DATASET RESULTS WITH STANDARD DEVIATIONS
The appendix provides extended per-dataset results for three benchmarks, grouped by dataset and including five-model independent-seed ensembles.
- Extended results: Tables 18–20 report extended results grouped by datasets for the main, Grinsztajn et al. (2022), and TabReD benchmarks.
- Ensembling: Each reported ensemble consists of five models trained independently under different random seeds.