PaperlayerView exact revision on arXiv

Source-linked AI summary

TabICLv2: A better, faster, scalable, and open tabular foundation model

Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan

arXiv:2602.11139v1cs.LG

TL;DR

Tabular foundation models have surpassed traditional tabular-learning benchmarks, but open models still need strong performance and scalable long-context inference. TabICLv2 combines diverse synthetic pretraining, architectural innovations including QASSMax, and optimized training, achieving state-of-the-art results while handling large tables efficiently.

  • Problem

    Open-source tabular foundation models are needed that rival leading closed-source systems and generalize beyond shorter contexts.

  • Method

    TabICLv2 combines a diverse synthetic prior, repeated feature grouping, target-aware embeddings, QASSMax, and optimized pretraining with Muon.

  • Results

    TabICLv2 achieves state-of-the-art performance and maintains 100% accuracy with 15K negatives in a synthetic attention-fading task.

  • Takeaways & Limitations

    TabICLv2 offers an open, moderate-cost foundation for fast tabular prediction and future adaptations.

  • Takeaways & Limitations

    TabICLv2 does not natively use column names or textual features, and million-sample datasets remain challenging.

Abstract

from arXiv · show

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and commit to open research by first releasing inference code and model weights at https://github.com/soda-inria/tabicl, with synthetic data engine and pretraining code to follow.

1. Introduction

Tabular foundation models have begun matching or surpassing gradient-boosted trees, creating demand for open models that provide top-tier performance and expose their training recipes. TabICLv2 addresses this need through architectural, pretraining, synthetic-data, evaluation, and ablation contributions.

  • Tabular deep learning models recently match or surpass gradient-boosted trees on tables with up to 100K samples.
  • Tabular foundation models perform training and inference in a single Transformer forward pass through in-context learning.
  • Open-source TFMs are needed to democratize access to top-tier performance and demystify the recipes behind leading models.
  • TabICLv2 is introduced as a state-of-the-art tabular foundation model with architectural and pretraining innovations, a novel synthetic-data generator, extensive evaluations, and ablations.

2. Related Work

Related work establishes tabular foundation models as a prior-data fitted-network paradigm, with architectural, synthetic-prior, adaptation, and language-model alternatives. A central unresolved challenge is maintaining sharp attention when context length grows.

  • Tabular foundation models: PFN-style tabular foundation models predict test-target distributions directly from training data and test inputs in a single forward pass.
  • Architectural perspectives: Tabular architectures evolved from row tokens to cell-based alternating attention and then lower-complexity two-stage designs.
  • Synthetic prior datasets: Synthetic priors increasingly combine structural causal models, tree-based mechanisms, richer graphs, hierarchical structures, and controllable difficulty.
  • Adaptation: TFM adaptation strategies include fine-tuning, retrieval-based context selection, and distillation into compact models.
  • LLM-based tabular models: LLM-based tabular models use table serialization and continued pretraining but underperform TFMs when sufficient training data is available.
  • Attention struggles with long-context generalization: Standard softmax attention fades on longer contexts because its denominator grows with context length, flattening distributions and limiting length generalization.
  • Attention struggles with long-context generalization: Scalable softmax methods address fading through context-dependent or query-dependent temperature scaling, with trade-offs in implementation compatibility and design complexity.

3. Architecture

TabICLv2 combines repeated feature grouping, target-aware embeddings, staged column/row/dataset processing, and query-aware scalable softmax. These components target feature representation, many-class prediction, regression quantiles, and long-context attention.

  • Feature representation: Repeated feature grouping places each feature in multiple circularly shifted groups, preserving effective feature count while reducing representation collapse.For m columns, the groups use positions (j, j + 1, j + 3) mod m.
  • Feature representation: The (0, 1, 3) shift pattern ensures that, for at least 7 columns, no column pair appears together in more than one group.
  • Feature representation: Target-aware embeddings are added directly to all training features rather than appended as an extra column.
  • Compression then ICL: TabICLv2 processes tables through column-wise embedding, row-wise aggregation, and dataset-wise in-context learning, with QASSMax applied in TFcol and TFicl.
  • Query-aware scalable softmax: QASSMax extends scalable softmax with query-aware, element-wise rescaling intended to counteract attention fading as training-set size grows.
  • Query-aware scalable softmax: QASSMax maintains 100% accuracy with low attention entropy on a synthetic task containing 15K negative samples, while standard attention and SSMax degrade.
  • Many-class classification: Mixed-radix ensembling decomposes more than 10 classes into digit-based groupings, runs TFcol per digit, and averages outputs.
  • Quantile predictions for regression: TabICLv2 predicts 999 quantiles with pinball loss and constructs probabilistic distributions by enforcing monotonicity and extrapolating tails.

4. Pretraining and Inference

TabICLv2 improves TabICL’s pretraining with expanded dataset scales, Muon optimization, and staged training, while inference uses disk offloading and selective projections. The resulting system targets efficient processing of million-scale tables.

  • Pretraining: TabICLv2 retains three progressively larger pretraining stages while using batch size 64 and approximately 35M datasets.
  • Pretraining: Stage 1 uses 500K steps on 1,024-sample datasets, followed by 40K and 10K steps on datasets reaching 10,240 and 60K samples.
  • Pretraining: Stages 2 and 3 progressively improve performance, especially on large datasets.
  • Optimization: Muon replaces AdamW or Adam, enabling a stage-1 maximum learning rate of 8e-4 versus 1e-4 for AdamW in TabICL.
  • Compute: 24.5 GPU-days are required to pretrain one model on H100 GPUs, comprising approximately 20, 2.5, and 2 GPU-days across the three stages.
  • Inference: Disk offloading enables processing a table with 1M samples and 500 features in 450 seconds using under 24 GB CPU and 50 GB GPU memory.

5. Synthetic data prior

TabICLv2 uses an entirely synthetic, highly diverse prior built from causal graphs, varied random functions, and filtering mechanisms designed to improve pretraining quality.

  • Prior design: The new prior combines structural causal models with diverse sampling options to maximize dataset diversity while preserving computational efficiency.Its design varies dependencies, categorical cardinalities, graph connectivity, and feature-generation mechanisms.
  • Generation pipeline: Each synthetic dataset is generated by propagating random vectors through a sampled directed acyclic graph and extracting features from randomly assigned nodes.Global dataset properties, graph structure, and parent–child functions are sampled before feature extraction.
  • Sampling mechanisms: A random Cauchy graph mechanism extends graph sampling beyond the tree-structured graphs available from earlier mechanisms.It models different global and local node connectivities.
  • Generation pipeline: Eight random-function families provide varied smoothness and inductive biases, including neural networks, tree ensembles, Gaussian processes, linear and quadratic functions, plateaus, and products.The figure also shows generated two-dimensional classification datasets produced by the prior.
  • Filtering: Filtering removes datasets that lack detectable predictive structure or contain graph relationships implying independence between features and targets.In stage 1, roughly 35% of classification and 25% of regression datasets are filtered, improving pretraining convergence.

6. Experiments

Experiments on TabArena and TALENT evaluate TabICLv2 across predictive quality, runtime, class cardinality, and dataset scale, showing strong accuracy–efficiency trade-offs.

  • Evaluation setup: TabArena evaluation covers 51 datasets, while TALENT covers 300 datasets spanning binary classification, multiclass classification, and regression.The benchmarks use task-appropriate metrics including ROC AUC, log-loss, and RMSE.
  • Overall performance: TabICLv2 surpasses tuned and ensembled RealTabPFN-2.5 without tuning and substantially outperforms heavily tuned CatBoost and XGBoost with far less training time.It dominates the improvability-versus-runtime Pareto fronts on both benchmarks.
  • Runtime: 10.6× speedup is achieved over TabPFN-2.5 on an H100 GPU at 50K samples with 100 features.The CPU gap reaches 11.8× at 10K samples, and TabICLv2 is faster across all tested hardware for 100 features.
  • Many-class classification: TabICLv2 substantially outperforms all baselines on TALENT datasets with more than 10 classes using either ECOC or native mixed-radix ensembling.ECOC is slightly better, while native handling is 3× faster.
  • Scalability: TabICLv2 maintains top rankings from 10^3 to 10^5 samples and outperforms RealTabPFN-2.5 on datasets larger than 20K samples.It also performs strongly on 600K-sample datasets from the TALENT extension.

7. Ablation study

Ablations show that the synthetic prior is the largest contributor to performance, while several architectural and pretraining choices provide additional gains.

  • Architecture and prior: The TabICLv2 architecture fails to generalize when pretrained with the TabICL prior, while the TabICL architecture with the TabICLv2 prior only matches TabICL.This indicates a strong interaction between architecture and prior diversity.
  • Ablation results: The prior produces the largest ablation effect across normalized accuracy, Elo, and log-loss.The ordering of ablations is consistent across these metrics.
  • Ablation results: Approximately 100 Elo and a 64% win rate are associated with comparable gains from early target inclusion, Muon, and QASSMax.Repeated feature grouping and prior filtering yield smaller gains.

8. Limitations

TabICLv2 has unresolved limitations involving semantic features, million-scale datasets, distributional tasks, missing values, and adaptation strategies.

  • Feature semantics: TabICLv2 does not natively use semantic information from column names or textual features, although text-embedding combinations may remain reasonably fast.The paper identifies semantic features as valuable but leaves their integration unexplored.
  • Scope boundaries: Datasets with millions of samples remain challenging despite improved scalability.The paper also leaves distribution shifts and multi-output regression to future work.
  • Missing data: Missing values are currently mean-imputed, while missing indicators and missingness during pretraining remain unexplored.The authors suggest these additions may improve missing-value handling.
  • Adaptation and evaluation: Hyperparameter tuning and fine-tuning are not explored, despite potentially improving performance at the cost of increased runtime.Distributional regression is also unevaluated beyond toy datasets because established benchmarks are lacking.

9. Conclusion

TabICLv2 achieves state-of-the-art tabular foundation-model performance while redefining native scalability. Its moderate costs, open release, and emphasis on out-of-the-box performance support future adaptations and continued work on smaller, faster models.

  • TabICLv2 achieves state-of-the-art performance and redefines the native scalability of tabular foundation models.
  • The authors fully open-source TabICLv2 to democratize access to state-of-the-art tabular foundation models.
  • Moderate pretraining and inference costs make TabICLv2 an excellent basis for future adaptations.
  • The model prioritizes out-of-the-box performance and principled innovations over real-data fine-tuning or deeper and wider architectures.
  • The authors hope TabICLv2 motivates continued innovation toward smaller, faster, and better models.

Contribution Statement

The contribution statement assigns complementary responsibilities across the authors, spanning regression and multiclass extensions, scalable implementation, generative priors, architectural and pretraining enhancements, experiments, and production code.

  • JQ developed the regression framework and multiclass extensions and enabled large-n scaling through optimized implementation and QASSMax.
  • JQ conducted most experiments and benchmark evaluations and prepared production-ready code.
  • DH conceived and implemented the new generative prior and other core architectural and pretraining enhancements.
  • DH conducted small-scale experiments and implemented nanotabicl.

Appendices

The appendices formalize feature-group intersections, describe TabICLv2's column-, row-, and dataset-wise processing, explain mixed-radix handling of many classes, specify model configurations, and report staged benchmark improvements.

  • Feature grouping: For m ≥ 2k, each pair of feature columns occurs together in at most one group; the implementation uses k = 3 in practice.
  • Architecture: Column-wise embedding uses induced self-attention to reduce complexity from O(n^2) to O(nk), while row-wise interaction collapses features into row embeddings.
  • Architecture: Dataset-wise in-context learning combines training row and target embeddings, restricts test attention to training samples, and applies QASSMax for long-context generalization.
  • Many-class classification: Mixed-radix label encoding creates multiple balanced views whose combined labels uniquely identify classes, extending classification beyond the 10-class embedding capacity.
  • Model configuration: The architecture uses separate classification and regression checkpoints, with regression replacing class embeddings by continuous-target layers and outputting 999 quantile predictions.
  • Ablation results: Average rank on all TALENT datasets improves from 9.94 at Stage 1 to 5.69 at Stage 2 and 5.41 at Stage 3.For datasets with more than 10K samples, rank improves from 14.91 to 5.50 to 4.71 across the same stages.

C. Additional ablation results

Additional ablations show that TabICLv2’s conclusions are consistent across metrics, while deeper models offer at most marginal gains and added prior noise has negligible effect. The appendix also documents implementation, prior-design, filtering, and probabilistic-output choices that shaped the final system.

  • Metric consistency: Across log-loss, normalized accuracy, and Elo, ablation orderings remain consistent and support the same component-level conclusions.The appendix reports additional normalized-accuracy and Elo results alongside the main-text log-loss analysis.
  • Model scaling: A deeper model shows no clear log-loss improvement, with only slight late-pretraining gains in normalized accuracy and Elo.The larger model used 4 instead of 3 TFcol/TFrow layers and 18 instead of 12 TFicl layers; insufficient pretraining may have limited convergence.
  • Prior ablations: Adding Gaussian noise to the synthetic prior has negligible performance impact across all evaluated metrics.The experiment followed TabPFNv2’s edge-noise approach for introducing uncertainty during synthetic data generation.
  • Synthetic prior: The new prior expands diversity through correlated scalar, random graph, node-importance, converter, function, activation, and matrix-sampling mechanisms.It also uses CatBoost-style symmetric tree ensembles, extended Gaussian-process functions, and random linear, quadratic, clustering-based, and product functions.
  • Data filtering: ExtraTrees filtering rejects synthetic datasets where a shallow model fails a bootstrap improvement test over a constant baseline.Classification is converted to regression with one-hot encoding; datasets are rejected unless improvement appears in at least 95% of 200 bootstrap subsamples.
  • Regression output: Quantile-based regression supports constructing full predictive distributions, including closed-form PDF, CDF, and analytical moments.TabICLv2 predicts 999 quantiles, then addresses quantile crossing and undefined tails through monotonicity enforcement and tail extrapolation.

I.9.2. VISUALIZATION AND ANALYSIS

The visualization validates TabICLv2’s quantile-based regression predictions against known distributions and explains the aggregation metrics used for benchmark analysis.

  • Distribution visualization: Heteroscedastic predictions widen toward |x| = 3 and produce narrower peaks at x = 0 but broader peaks at x = ±2, matching ground truth.These patterns appear in both quantile bands and predicted PDFs.
  • Distribution visualization: Predicted quantile curves closely match true quantile functions across datasets, with crossing correction and exponential tail extrapolation beyond [0.001, 0.999].The figure compares predicted and ground-truth quantile functions for representative inputs.
  • Distribution visualization: Predicted CDFs follow the true smooth S-curves, while density heatmaps reproduce funnel-shaped heteroscedasticity and step-function bands.The heatmaps visualize conditional density across the input domain on a log scale.
  • Distribution visualization: Resampled points from the learned distributions closely match the original training data through inverse transform sampling.Sampling uses y = Q(U) with U ∼Uniform(0, 1).
  • Aggregation metrics: TabArena reports task-specific errors—1 −ROC AUC, Log-Loss, and RMSE—alongside Elo and average rank, with TabICLv2 consistently state-of-the-art across metrics.Elo treats datasets equally by wins, while rank-based metrics summarize dataset-wise ordering.
  • Aggregation metrics: Improvability averages each model’s relative error gap to the dataset-specific best method, capturing performance magnitude rather than only rankings.It ranges from 0% for optimal performance to 100%.

J.2. Results on all datasets

Across TabArena’s datasets, TabICLv2 delivers strong overall accuracy and efficiency, with performance varying by task type and metric. It leads the tabular foundation-model comparisons while remaining competitive with heavily tuned systems.

  • All datasets: 4.82 average rank makes TabICLv2 better than AutoGluon 1.4 at 5.24 and RealTabPFN-2.5 at 5.88 across all TabArena datasets.TabICLv2 uses a single untuned forward pass, unlike the heavily optimized AutoGluon ensemble.
  • All datasets: 57% and 59% pairwise win rates against AutoGluon 1.5 and RealTabPFN-2.5 show TabICLv2 wins on most head-to-head datasets.Average rank can differ because it penalizes occasional poor performance more strongly than win rate.
  • All datasets: TabICLv2 dominates the Pareto fronts of improvability versus runtime and Elo versus runtime among tabular foundation models.It is reported as both faster and more accurate than RealTabPFN-2.5, TabPFN-2.5, TabICL, LimiX, and Mitra.
  • Binary classification: 4.43 average rank places TabICLv2 second on 24 binary-classification datasets, behind AutoGluon 1.5 at 3.83 and ahead of RealTabPFN-2.5 at 6.90.TabICLv2 remains the strongest tabular foundation model on this subset’s Pareto fronts.
  • Multiclass classification: 6.75 average rank places TabICLv2 behind RealTabPFN-2.5 at 4.25 on 14 multiclass datasets, while still outperforming TabPFNv2 at 8.62 and LimiX at 9.31.The comparison is against a tuned and ensembled RealTabPFN-2.5 system.
  • Regression: 4.54 average rank ties TabICLv2 with RealTabPFN-2.5 on 13 regression datasets, trailing only AutoGluon 1.5 at 3.92.The strong regression result from real-data-pretrained TabDPT raises possible data-leakage questions.

K.2. Results on all datasets

Across TALENT, TabICLv2 achieves the strongest overall ranking and outperforms competing tabular foundation models across several task types and dataset sizes. Its advantages are especially clear on probabilistic binary metrics, multiclass classification, many-class classification, and large datasets.

  • TabICLv2 achieves the best average rank at 4.66, ahead of RealTabPFN-2.5 at 5.11 and TabPFN-2.5 at 5.45.
  • TabICLv2 wins 62% of pairwise comparisons against RealTabPFN-2.5 and 65% against TabPFN-2.5.
  • Binary classification: On binary classification, TabICLv2 is essentially tied with RealTabPFN-2.5 on accuracy but leads on AUC and log-loss.AUC ranks are 3.31 versus 4.62, and log-loss ranks are 2.83 versus 3.78.
  • Multiclass classification: On multiclass datasets with at most 10 classes, TabICLv2 leads RealTabPFN-2.5 on accuracy, AUC, and log-loss.The respective average ranks are 4.58 versus 5.64, 3.38 versus 5.20, and 2.67 versus 4.48.
  • Many-class classification: For datasets with more than 10 classes, both TabICLv2 variants outperform RealTabPFN-2.5-ECOC.The variants use either the ECOC wrapper or native mixed-radix ensembling.
  • Dataset size: TabICLv2 is comparable to RealTabPFN-2.5 on small datasets but has a clear advantage on large datasets over both RealTabPFN-2.5 and TabPFN-2.5.
Loading 2602.11139v1…