Source-linked AI summary
Xiaomi-TabLDM: A Tabular Foundation Model Technical Report
Xiaomi-TabLDM Team, :, Penghui Wang, Wei Liu, Hong Wang, Chengyue Huang, Yuxi Sun, Zirui Wang, Hongming Huang, Quan Wang, Zhenwei Xin, Ping Hou, Jie Yu, Chunxiao Liu, Erli Meng, Bin Wang
TL;DR
Tabular foundation models seek strong prediction without dataset-specific training, but broad performance and efficiency remain important goals. Xiaomi-TabLDM addresses this with in-context prediction, synthetic structural-causal pretraining, efficient architectural scaling, and test-time computation. It delivers consistently strong benchmark results, particularly in regression, alongside favorable computational efficiency.
Problem
Tabular prediction has largely relied on dataset-specific pipelines, motivating foundation models that can handle diverse tasks through pretraining and in-context learning.
Method
Xiaomi-TabLDM predicts from labeled training examples as in-context demonstrations, using synthetic structural-causal pretraining, efficient feature and expert architectures, and test-time scaling.
Results
Across four benchmark suites, Xiaomi-TabLDM is especially strong on regression, ranking first on OpenML-CTR23 and second on regression across TALENT, TabArena, and BCCO.
Takeaways & Limitations
The results indicate that architectural design and capacity allocation can improve tabular learning performance without relying solely on larger models or higher computational cost.
Takeaways & Limitations
The model is pretrained entirely on synthetic datasets generated from a structural causal model prior, whose design assumes learnable predictive signals while seeking broad structural diversity.
Abstract
from arXiv · showhide
We introduce Xiaomi-TabLDM, a tabular large data foundation model for classification and regression via in-context learning, which delivers superior prediction accuracy without requiring task-specific fine-tuning. Pretrained exclusively on synthetic data generated from structural causal models (SCMs), our model enables more flexible context utilization and more efficient capacity scaling. i) A new performance standard. Strong regression performance across benchmarks: Xiaomi-TabLDM ranks 1st on OpenML-CTR23 and 2nd on regression across TALENT, TabArena, and BCCO, demonstrating consistently strong regression performance across four complementary benchmark suites. Favorable performance--efficiency trade-off: Xiaomi-TabLDM combines strong predictive performance with substantially lower computational cost. For example, on TabArena regression, it achieves the second-highest Elo while using 82% less training time and 68% less prediction time than the top-ranked TabFM. ii) Large-scale synthetic pretraining. Xiaomi-TabLDM expands the coverage and diversity of synthetic tabular data used for pretraining. We also adopt a three-stage training strategy together with dual-stream feature grouping, lightweight Attention Residual, and sparse Mixture-of-Experts, enabling Xiaomi-TabLDM to learn richer feature interactions and expert specialization across diverse tabular tasks. iii) Test-time scaling. Xiaomi-TabLDM further extends tabular prediction through test-time compute scaling, where allocating additional computation at inference time consistently improves predictive performance over the base model.
1 Introduction
Tabular prediction has traditionally relied on dataset-specific pipelines, while tabular foundation models pursue pretrained in-context learning. Xiaomi-TabLDM extends this direction with broader synthetic pretraining, efficient scaling, and strong benchmark performance, especially for regression.
- Most tabular methods still train and tune separate models for each dataset, despite extensive architectural advances.
- Xiaomi-TabLDM combines flexible context utilization, selective information preservation, and sparse expert computation to scale capacity without proportional inference-cost increases.
- Xiaomi-TabLDM ranks first on OpenML-CTR23 regression and second on regression across TALENT, TabArena, and BCCO.
- On mixed-task evaluations, Xiaomi-TabLDM ranks second overall on TALENT, first on binary classification, and fourth overall on TabArena across 51 datasets and 816 tasks.
- 82% less training time and 68% less prediction time than TabFM accompany Xiaomi-TabLDM's second-highest TabArena regression Elo.
2 Overall Framework
Xiaomi-TabLDM uses in-context demonstrations for prediction without task-specific training. Its framework combines staged synthetic pretraining with dual-stream feature processing, adaptive residual connections, sparse experts, and a structural-causal synthetic prior.
- 2 Overall Framework: The model predicts test targets by conditioning on labeled training examples as in-context demonstrations, without task-specific training.
- 2 Overall Framework: Its transformer pipeline performs column-wise feature embedding, row-wise aggregation, and in-context learning prediction.
- Architecture: Dual-stream grouping combines complementary column offsets to produce feature representations that adapt across table widths.
- Architecture: Lightweight Attention Residual connections adaptively retrieve earlier representations, improving gradient flow and representation reuse across depth.
- Architecture: Sparse Mixture-of-Experts activates only top-K routed experts per token alongside shared experts, increasing capacity while limiting activated computation.
- Pretraining: Pretraining separately covers classification and regression through a three-stage curriculum that progressively scales dataset size.
- Synthetic Prior: Synthetic datasets come from a structural causal model prior designed to diversify dataset structures, variable types, and functional relationships while retaining predictive signals.
3 Test-Time Scaling
Test-time scaling increases inference computation while keeping the pretrained model fixed. Xiaomi-TabLDM diversifies feature views and preprocessing paths, then combines their predictions to adapt to heterogeneous datasets.
- Test-time scaling allocates additional inference-time computation while keeping the pretrained model fixed.
- Inference ensemble: Feature shuffling and subset sampling reduce sensitivity to column order and feature combinations while creating prediction diversity at negligible additional cost.
- Inference ensemble: Complementary preprocessing views include normalization schemes, quantile transformation, SVD representations, feature interactions, and regression target transformations.
- Inference ensemble: Nonnegative least squares learns estimator weights from holdout or out-of-fold predictions for classification and regression.
- The procedure scales inference by increasing diverse prediction paths rather than modifying the underlying model, exploiting complementary views while mitigating overfitting risk.
4 Evaluation
Xiaomi-TabLDM is evaluated across four complementary tabular benchmark suites, with particularly strong and consistent regression performance alongside competitive classification and overall results. It also offers a favorable performance–efficiency trade-off and learns structured row representations.
- Benchmark setup: The evaluation covers TALENT, TabArena, BCCO, and OpenML-CTR23 across diverse classification and regression tasks and dataset scales.OpenML-CTR23 focuses exclusively on regression, while TALENT, TabArena, and BCCO include mixed task types.
- TALENT: 4.03 average rank across 100 TALENT regression datasets places Xiaomi-TabLDM second, behind TabFM at 3.72.It outperforms EXAONE-Tabular, TabPFN-v3, TabICLv2, and TabPFN-v2.6.
- TALENT: 4.82 average rank places Xiaomi-TabLDM second overall on TALENT, behind TabFM at 4.23.It also ties TabFM for best binary-classification performance with a 5.72 average rank.
- TabArena: 1900 Elo places Xiaomi-TabLDM second on TabArena regression, behind TabFM at 2019.It records 1.8 dataset wins and 1.7% improvability, narrowly behind TabFM’s 1.6%.
- Efficiency: 82% less training time and 68% less prediction time than TabFM accompany Xiaomi-TabLDM’s near-leading TabArena regression performance.Its prediction time is 3.12 s per 1K samples versus TabFM’s 9.67 s, while training time is 6.99 s versus 38.85 s.
- BCCO and OpenML-CTR23: Xiaomi-TabLDM ranks second on BCCO regression, third overall on BCCO, and first on OpenML-CTR23 with a 3.03 average rank.On BCCO, it maintains competitive multiclass classification results, ranking second for Accuracy and LogLoss and fourth for AUC.
- Representations: PCA visualizations show clearer and more coherent low-dimensional structure in Xiaomi-TabLDM’s learned row embeddings than in original feature spaces.The comparison uses three representative datasets.
5 Conclusion
Xiaomi-TabLDM is designed to combine strong predictive performance with efficient model and inference profiles across heterogeneous tabular tasks. The report highlights competitive results, a favorable performance–efficiency trade-off, and architectural configurations spanning classification and regression.
- Performance: Xiaomi-TabLDM consistently ranks among the strongest methods across four benchmark suites, with particularly strong regression performance and competitive classification results.The model is evaluated across diverse classification and regression tasks.
- Efficiency: Xiaomi-TabLDM achieves stronger performance in several evaluation regimes while requiring substantially less computational cost than alternatives such as TabFM and AutoGluon.The reported trade-off emphasizes architectural design and capacity allocation rather than model size or computational cost alone.
- Conclusion: The report concludes that Xiaomi-TabLDM balances predictive performance, model capacity, and computational efficiency across heterogeneous tabular tasks.The authors position these findings as a basis for exploring scalable and efficient foundation-model architectures for structured data.
- Model configuration: The released classifier and regressor share architectural hyperparameters and sparse MoE configuration, differing only in the target encoder and final decoder linear layer.This shared configuration spans both task types while preserving task-specific output components.
B Results
The evaluation compares Xiaomi-TabLDM with tabular foundation models, deep feature-learning methods, and tree-based or classical baselines.
- Tabular foundation models: The comparisons include TabFM, EXAONE-Tabular, TabPFN-3, TabPFN-2.6, RealTabPFN, TabICLv2, TabDPT, LimiX, iLTM, TabSTAR, TabFlex, and OrionMSP.
- Deep and feature-learning methods: The evaluation also includes RealMLP, ModernNCA, TabM, and xRFM as deep or feature-learning methods.
- Baselines: Tree-based and classical baselines are included alongside the foundation and deep-learning approaches.The passage identifies XGBoost and CatBoost among the baseline methods.
B.1.1 Evaluation Metrics
The evaluation uses complementary metrics for pairwise performance, relative error reduction, dataset-level wins, tuning effects, and computational cost. Together, these measures assess both predictive quality and the expense of tuning or ensembling.
- Elo: Elo evaluates pairwise model comparisons, with rating differences determining expected win probabilities.A 400-point Elo difference corresponds to an expected win probability of approximately 91% for the higher-rated model.
- Improvability: Improvability measures the relative error reduction required for a method to match the best-performing method on each dataset, averaged across datasets.Lower values are better, and 0% indicates performance equal to the best observed method on every dataset.
- Dataset wins: Dataset wins count how often a method achieves the best performance, complementing Elo and Improvability with a task-level view of dominance.
- Tuning and ensembling: Tuning plots compare default, tuned, and tuned-and-ensembled configurations using increasingly large ensembles of sampled configurations.
- Efficiency: Computational efficiency is reported as median training and inference time per 1K samples to quantify the cost of tuning and ensembling.These measurements expose the trade-off between strong default performance and more expensive tuned pipelines.
B.1.3 TabArena Leaderboard Tables
On the full TabArena benchmark, Xiaomi-TabLDM remains highly competitive, while its strongest relative results occur on regression. The regression advantage is accompanied by substantially lower training and prediction costs than TabFM.
- Full benchmark: Across 51 datasets and 816 tasks, Xiaomi-TabLDM achieves an Elo score of 1659 and ranks fourth by Elo point estimate.Its score is nearly tied with AutoGluon 1.5 extreme at 1662 and exceeds several recent tabular foundation models.
- Regression: On 13 regression datasets, Xiaomi-TabLDM reaches an Elo score of 1900 and ranks second behind TabFM at 2019.It also achieves 1.8 dataset wins and an Improvability of 1.7%, the second-best result behind TabFM.
- Efficiency: 82% less training time and 68% less prediction time are required by Xiaomi-TabLDM than TabFM on TabArena regression.Xiaomi-TabLDM uses 6.99 s training time and 3.12 s prediction time per 1K samples, compared with TabFM’s 38.85 s and 9.67 s.
B.2 Details on TALENT benchmark
TALENT provides a standardized evaluation framework spanning diverse classification and regression datasets, metrics, and model families. The section also documents benchmark comparisons using Elo and computational-cost measures.
- Benchmark scope: The benchmark compares classical, tree-based, deep-learning, and foundation-model approaches under a unified evaluation pipeline.TALENT is presented as a comprehensive toolbox for consistent comparison across tabular datasets.
- Benchmark scope: TALENT unifies preprocessing, model interfaces, and evaluation across heterogeneous classification and regression datasets.The benchmark varies dataset size, feature dimensionality, class structure, and feature composition.
- Evaluation protocol: Classification average rank is computed from per-dataset Accuracy, measuring consistency relative to competing methods rather than average Accuracy alone.Accuracy is the primary classification metric in the reported experiments.
- TabArena comparison: TabArena reports model Elo, wins, improvability, training time, and prediction time, distinguishing default, fine-tuned, and ensembled configurations.The regression subset covers 13 datasets and reports Xiaomi-TabLDM’s second-highest Elo with substantially lower computational cost than most tuned and ensembled baselines.
C.1 Experimental Design and Data Generation
The experimental design generates tabular regression data from observable structural causal models with controlled graph density and node-level signal-to-noise ratios. Graphs, structural functions, noise, and target connectivity are sampled under explicit constraints.
- Data generation: Each dataset contains 32 observed variables, with 31 predictive features and the final topological node as the regression target.No latent variables are introduced, although evaluated models do not observe the graph, structural functions, or noise mechanisms.
- Graph design: Graphs use ordered Erdős–Rényi sampling, with sparse and dense regimes defined by separated edge-probability ranges.The sparse regime uses p ∼U[0.04, 0.10], while the dense regime uses p ∼U[0.25, 0.40].
- Target construction: The target is forced to have at least one parent when initial graph sampling leaves it disconnected from predictive features.One to three preceding nodes are connected to the target under this correction rule.
- Structural equations: Node values are generated in topological order by combining sampled structural functions with standardized exogenous noise controlled by node-level SNR.The resulting noisy values are standardized before being passed to descendants, allowing upstream perturbations to propagate.
- Experimental factors: SNR is varied through four levels, including L1, L2, L3, and Lrand, while function types and parameters remain fixed across matched experimental conditions.The design explicitly includes within-graph SNR heterogeneity for Lrand.
C.2 Noise Diversity and Experimental Settings
The noise experiments vary distribution families and graph-level SNR settings within a balanced factorial design. They include clean, single-distribution, and mixed-noise conditions, with standardization separating distributional shape from location and scale.
- Noise conditions: Fourteen conditions comprise twelve single-noise-distribution settings, one mixed-noise condition, and one clean reference condition.The clean condition removes additive noise from non-root structural equations while retaining root disturbances.
- Noise construction: Standardizing raw noise removes location and scale differences, so comparisons primarily reflect distributional shape and shape-parameter variation.The mixed condition additionally introduces node-level heterogeneity in noise distribution families.
- Repeatability control: The exponential–Weibull pair forms an A/A control because Weibull shape c = 1 is equivalent to an exponential distribution after standardization.Independent samples from the equivalent population distribution provide a repeatability reference.
- Factorial design: The balanced factorial design crosses 2 graph-density levels with 4 SNR levels and independently samples 20 base configurations per combination.Each base configuration includes graph structure, structural functions, node-level SNRs, and a feature-column permutation.
C.3 Experimental Results
Across 2,240 synthetic datasets and varied noise conditions, Xiaomi-TabLDM shows a stable relative advantage over LimiX and TabICLv2. Its first-place share remains high across both clean and noisy settings, while the A/A control quantifies sampling variation.
- Overall results: 70.3% of datasets are won by Xiaomi-TabLDM, compared with 16.0% for LimiX and 13.7% for TabICLv2.Xiaomi-TabLDM achieves the highest R2 on 1,575 of 2,240 datasets.
- Noise robustness: 63.8% is Xiaomi-TabLDM’s lowest first-place share across the thirteen non-Clean noise conditions, while its maximum reaches 78.1%.LimiX never exceeds roughly 28%, and TabICLv2’s maximum is roughly 25%.
- Repeatability: 5.0 percentage points separate the exponential and Weibull first-place shares despite their identical standardized population distribution.The difference provides an empirical reference for repeatability under independent sample realizations.
- Clean versus noisy settings: 65.6% is Xiaomi-TabLDM’s first-place share under the Clean reference condition, and most noisy conditions are no lower than this level.The reported results describe a stable relative advantage across tested noise families and perturbation intensities.