PaperlayerView exact revision on arXiv

Source-linked AI summary

TabICLv2: A better, faster, scalable, and open tabular foundation model

Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan

arXiv:2602.11139v2cs.LG

TL;DR

Open tabular foundation models need to rival closed or heavily adapted systems while addressing attention fading and scalability to larger datasets. TabICLv2 combines a novel diverse synthetic prior, architectural innovations including query-aware scalable softmax, and optimized pretraining with Muon. TabICLv2 achieves state-of-the-art performance, redefines the native scalability of tabular foundation models, and has moderate pretraining and inference cost.

  • Problem

    Open tabular foundation models need to rival closed or heavily adapted systems while addressing attention fading and scalability to larger datasets.

  • Method

    TabICLv2 combines a novel diverse synthetic prior, architectural innovations including query-aware scalable softmax, and optimized pretraining with Muon.

  • Results

    TabICLv2 achieves state-of-the-art performance, redefines the native scalability of tabular foundation models, and has moderate pretraining and inference cost.

  • Takeaways & Limitations

    The released prior, pretraining, and inference code provides an open basis for efficient tabular foundation-model research and downstream adaptations.

  • Takeaways & Limitations

    TabICLv2 does not natively use semantic information from column names or textual features, and datasets with millions of samples remain challenging.

Abstract

from arXiv · show

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.

1. Introduction

Tabular foundation models have recently matched or surpassed gradient-boosted trees, motivating open models that provide top-tier performance and enable downstream adaptations. TabICLv2 addresses this need as a state-of-the-art tabular foundation model through architectural, pretraining, and synthetic-data innovations.

  • Tabular foundation models recently match or surpass gradient-boosted trees on tables with up to 100K samples.
  • Their single-forward-pass in-context learning supports downstream adaptations including causal inference, generative modeling, and simulation-based inference.
  • Open-source tabular foundation models are needed to democratize top-tier performance and clarify the recipes behind strong models.
  • TabICLv2 is introduced as a state-of-the-art tabular foundation model with architectural innovations, pretraining improvements, and a novel synthetic data generator.

2. Related Work

Tabular foundation models use prior-data fitted networks to perform in-context prediction directly from training data, while recent work explores more efficient architectures, richer synthetic priors, adaptation, retrieval, and distillation. A central limitation is that standard softmax attention fades as context length grows, impairing length generalization.

  • PFN-based tabular foundation models predict test targets directly from a training set and synthetic prior over datasets, without gradient updates at inference.
  • Architectural perspectives: TabICL reduces tabular attention complexity to O(n^2+nm^2) through a two-stage design, while TabPFNv2 uses O(n^2m+nm^2) cell-based attention.
  • Synthetic prior datasets: Synthetic priors span structural causal models, tree-based mixtures, richer DAGs, hierarchical SCMs, and controllable difficulty designs.
  • Fine-tuning, retrieval, and distillation: Adaptation strategies include fine-tuning, retrieval-based context selection, and distillation into compact MLPs or trees.
  • Attention struggles with long-context generalization: Standard softmax attention flattens as context length increases because its denominator grows, preventing sharp focus on relevant tokens and limiting length generalization.

3. Architecture

TabICLv2 combines repeated feature grouping, early target-aware embedding, staged column/row/dataset processing, and query-aware scalable softmax. These designs target feature-representation collapse, many-class prediction, regression quantiles, and attention fading on long contexts.

  • Repeated feature grouping: Repeated feature grouping places each column into multiple circularly shifted groups, preserving effective feature count while breaking feature symmetries.The (0, 1, 3) shift pattern prevents column pairs from recurring together when tables have at least seven columns.
  • Target-aware embedding: Early target-aware embedding adds target representations directly to all training features, helping determine feature relevance before compression and mitigate representation collapse.EmbedTAE is linear for regression and a learnable lookup table for classification.
  • Compression then ICL: TabICLv2 processes tables through column-wise embedding, row-wise aggregation, and dataset-wise in-context learning before predicting test targets.QASSMax is applied in TFcol and TFicl to improve long-context generalization.
  • Query-aware scalable softmax: QASSMax combines log n scaling, element-wise base scaling, and bounded query-aware gating to counter attention fading.In the needle-in-a-haystack task, QASSMax maintains 100% accuracy with 15K negative samples, whereas SSMax degrades substantially.
  • Many-class classification and regression: Mixed-radix ensembling with hierarchical classification enables TabICLv2 to handle an arbitrary number of classes.For regression, it predicts 999 quantiles with pinball loss; preliminary RMSE experiments found this outperformed MSE and TabPFNv2’s bin-based approach.

4. Pretraining and Inference

TabICLv2 improves TabICL’s pretraining with three progressively larger dataset stages, Muon optimization, and efficient inference mechanisms. The resulting system improves especially on large datasets and can process million-sample tables under constrained memory.

  • Three pretraining stages: Stages 2 and 3 progressively improve performance, especially on large datasets.The three-stage schedule expands training datasets from 1,024 samples to as many as 60K samples.
  • Optimizer: Muon replaces AdamW and uses orthogonalized gradient updates, with a higher stage-1 maximum learning rate of 8e-4 than TabICL’s 1e-4.
  • Pretraining cost: Pretraining costs 24.5 H100 GPU-days per model, below TabICL’s 60 A100-days.
  • Inference scalability: Disk offloading enables processing a table with 1M samples and 500 features in 450 seconds using under 24 GB CPU and 50 GB GPU memory.Selective Q/K/V projection computation further reduces redundant work.

5. Synthetic Data Prior

TabICLv2 uses an entirely synthetic, diversity-oriented prior that generates datasets through random causal graphs, functions, sampling mechanisms, and filtering.

  • The prior samples global dataset properties, builds a random directed acyclic graph, propagates random vectors through parent–child functions, and extracts features from assigned nodes.
  • A random Cauchy graph extends prior graph generation beyond tree-structured graphs by modeling varied global and local node connectivities.
  • Eight random function families provide varied smoothness and inductive biases, including multilayer perceptrons, tree ensembles, Gaussian processes, linear and quadratic functions, plateaus, and products.
  • Filtering removes datasets that fail predictive or dependency checks; roughly 35% of classification and 25% of regression datasets are filtered in stage 1, improving convergence.
  • Correlated sampling of repeated scalars makes the prior more likely to reproduce real-dataset regularities such as categorical columns sharing cardinalities.

6. Experiments

Across TabArena and TALENT, untuned TabICLv2 achieves state-of-the-art performance while combining strong accuracy with faster inference and scaling to larger datasets.

  • TabICLv2 surpasses tuned and ensembled RealTabPFN-2.5 on both benchmarks and dominates the Pareto fronts of improvability versus runtime.
  • TabICLv2 is faster than TabPFN-2.5 across tested hardware, reaching 10.6× speedup at 50K samples on an H100 and 11.8× at 10K samples on CPU.
  • On datasets with more than 10 classes, both ECOC and native mixed-radix ensembling substantially outperform all baselines, while native handling is 3× faster.
  • TabICLv2 maintains top rankings from 10^3 to 10^5 samples and outperforms RealTabPFN-2.5 above 20K samples; it remains strong near 600K samples while TabPFN-2.5 runs out of memory.
  • With 16 training samples, TabICLv2 ranks second to RealTabPFN-2.5 while outperforming untuned classical baselines and tested LLMs.

7. Ablation Study

Ablations show that TabICLv2’s gains depend strongly on the synthetic prior and arise from several architectural and pretraining choices.

  • The reference checkpoint’s advantage over its 280K-step ablation reflects longer pretraining and eight rather than four TFcol and TFicl attention heads.
  • TabICLv2 matches RealTabPFN-2.5 after about 200K steps in log-loss and in under 100K steps for normalized accuracy.
  • Using the TabICL prior with the TabICLv2 architecture fails to match the reference, while using the TabICLv2 prior with the TabICL architecture only matches TabICL.
  • The prior produces the largest ablation effect, while early target inclusion, Muon, and QASSMax each provide comparable gains of about 100 Elo and a 64% win rate.
  • Repeated feature grouping and prior filtering yield smaller gains than the three major component changes.

8. Limitations

TabICLv2 still has important scope boundaries involving semantic features, million-scale datasets, missing values, distributional settings, and adaptation.

  • TabICLv2 does not natively use semantic information from column names or textual features, although text embeddings may remain compatible with its feature scalability.
  • Datasets with millions of samples remain challenging despite improved scalability.
  • Distributional regression is evaluated only on toy datasets, while multi-output regression and distribution-shift handling remain future work.
  • Missing values are currently mean-imputed, and missingness indicators or missingness during pretraining remain unexplored.
  • Hyperparameter tuning and fine-tuning are not explored because they would increase runtime, although they could further improve performance.

9. Conclusion

TabICLv2 achieves state-of-the-art tabular foundation-model performance while redefining native scalability and supporting open research through released code.

  • TabICLv2 achieves state-of-the-art performance and redefines the native scalability of tabular foundation models.The released prior, pretraining, and inference code supports future adaptations, while moderate pretraining and inference costs broaden access.

Contribution Statement

The contribution statement assigns complementary responsibilities across framework development, synthetic-prior and architectural work, large-scale evaluation, and manuscript refinement.

  • JQ and DH contributed equally, leading complementary work on regression and multiclass extensions, large-scale implementation, synthetic priors, architecture, and pretraining enhancements.JQ led much of the experimentation and production code, while DH implemented the generative prior and core architectural and pretraining improvements.
  • The broader team managed large-scale pretraining, systematic model-variant evaluation, technical steering, experimental design, and manuscript refinement.

Appendices

The appendices detail TabICLv2’s architecture, many-class classification mechanism, feature grouping theory, implementation choices, and staged benchmark improvements.

  • Feature grouping: Repeated feature grouping places each feature into multiple circularly shifted groups, preserving effective feature count while reducing representation collapse.The construction is supported by a theorem showing that when m ≥2k, no pair of columns occurs together in two groups; k = 3 is used in practice.
  • Implementation: The architecture uses induced self-attention for column embeddings, four [CLS] tokens for row representations, and QASSMax in dataset-wise attention for long-context generalization.
  • Many-class classification: Mixed-radix ensembling creates multiple label views with at most 10 classes, while hierarchical classification handles later ICL-stage partitioning without retraining.Together, the methods support classification tasks with an arbitrary number of classes.
  • Architecture: TabICLv2 combines column-wise embedding, row-wise interaction, and dataset-wise in-context learning through TFcol, TFrow, and TFicl.Separate checkpoints are trained for classification and regression.
  • Additional results: Stage-wise TALENT performance improves average rank from 9.94 to 5.69 to 5.41 across all datasets.For datasets exceeding 10K samples, average rank improves from 14.91 to 5.50 to 4.71 across the three stages.

C. Additional Ablation Results

Additional ablations show consistent metric ordering, limited benefit from deeper models or prior noise, and several architecture and pretraining choices that were retained or rejected.

  • Metrics: Across log-loss, normalized accuracy, and Elo, ablation ordering remains consistent, supporting the same component-level conclusions.
  • Increasing model depth: Deeper TFcol, TFrow, and TFicl models show no clear log-loss improvement, with only slight late-pretraining gains on normalized accuracy and Elo.The authors attribute the marginal gain possibly to insufficient pretraining for the larger model and did not pursue simple scaling further.
  • Adding noise to the prior: Adding Gaussian noise to the synthetic prior has negligible impact on performance across all evaluated metrics.
  • Architecture: row interaction: Full row-wise attention is important, because replacing it with induced self-attention performs considerably worse.
  • Synthetic prior: The prior expands diversity through correlated scalar sampling, random graphs, node and feature importances, precise converters, and diversified random functions.The random-function family includes CatBoost-style symmetric-tree ensembles and extended Gaussian-process functions.

I.9.2. VISUALIZATION AND ANALYSIS

QuantileDistribution is evaluated visually on four synthetic regression tasks, while TabArena’s complementary metrics quantify model performance across datasets. The appendix emphasizes improvability because it captures performance-gap magnitude, and reports that TabICLv2 is state of the art across aggregation metrics.

  • Visualization: Predicted quantile curves, PDFs, CDFs, density heatmaps, and resampled data closely match the ground-truth distributions across four synthetic regression datasets.The visualizations capture heteroscedasticity, step-function structure, tail behavior, and conditional location and spread.
  • Aggregation metrics: Elo aggregates pairwise wins, ties, and losses while treating every dataset equally and ignoring performance-gap magnitude.A 400-point Elo difference corresponds to a 10:1 expected win rate.
  • Aggregation metrics: Improvability measures the average relative error gap to the best method on each dataset, with 0% representing optimal performance.Unlike Elo and rank-based metrics, it reflects the magnitude of performance differences.
  • Aggregation metrics: Average rank is the mean per-dataset rank, where lower values indicate better overall performance.Rank-based aggregation is robust to outliers but does not express the size of performance gaps.
  • Overall evaluation: TabICLv2 consistently achieves state-of-the-art performance across improvability, Elo, and ranking metrics.The paper reports improvability primarily in the main text and provides Elo and rankings for completeness.

J.2. Results on all datasets

Across all 51 TabArena datasets, TabICLv2 performs strongly without tuning, winning most head-to-head comparisons and dominating the tabular-foundation-model runtime–performance frontier. Its results vary by task type, with especially strong binary-classification performance, competitive regression performance, and weaker multiclass ranking.

  • J.2. Results on all datasets: 4.82 average rank places default TabICLv2 ahead of AutoGluon 1.4 at 5.24 and RealTabPFN-2.5 at 5.88 across all TabArena datasets.This result comes from a single forward pass without tuning, versus a heavily optimized AutoGluon ensemble.
  • J.2. Results on all datasets: 57% and 59% pairwise win rates against AutoGluon 1.5 and RealTabPFN-2.5 show TabICLv2 wins on most head-to-head datasets.AutoGluon 1.5 nevertheless has a better average rank, 3.88 versus 4.82, because average rank penalizes occasional poor performances more strongly.
  • J.2. Results on all datasets: TabICLv2 dominates the Pareto fronts for improvability versus runtime and Elo versus runtime among tabular foundation models.It is reported as both faster and more accurate than RealTabPFN-2.5, TabPFN-2.5, TabICL, LimiX, and Mitra.
  • J.3. Results on binary classification datasets: 4.43 average rank places TabICLv2 second on 24 binary-classification datasets, behind AutoGluon 1.5 at 3.83 and ahead of RealTabPFN-2.5 at 6.90.It remains the strongest tabular foundation model on the binary-classification Pareto fronts.
  • J.4. Results on multiclass classification datasets: 6.75 average rank places TabICLv2 behind AutoGluon 1.5, RealTabPFN-2.5, and AutoGluon 1.4 on 14 multiclass datasets.It still substantially outperforms TabPFNv2 at 8.62 and LimiX at 9.31, despite using no tuning.
  • J.5. Results on regression datasets: 4.54 average rank ties TabICLv2 with RealTabPFN-2.5 on 13 regression datasets, trailing only AutoGluon 1.5 at 3.92.TabDPT’s 5.31 regression rank raises questions about possible data leakage because it is pretrained on real-world datasets.

K.2. Results on all datasets

TabICLv2 achieves the best overall TALENT ranking, outperforming RealTabPFN-2.5 and TabPFN-2.5 across the benchmark while retaining strong performance across task types and dataset sizes.

  • 4.66 average rank puts TabICLv2 ahead of RealTabPFN-2.5 (5.11) and TabPFN-2.5 (5.45), with pairwise win rates of 62% and 65%.
  • RealTabPFN-2.5 outperforms TabPFN-2.5 on TALENT, showing measurable benefits from fine-tuning on real-world data.
  • TabICLv2 substantially outperforms LimiX and TabPFNv2, whose average ranks are 8.34 and 8.82 respectively.
  • Binary classification: On binary classification, TabICLv2 nearly matches RealTabPFN-2.5 in accuracy but leads on AUC and log-loss.Its average ranks are 3.31 versus 4.62 on AUC and 2.83 versus 3.78 on log-loss.
  • Multiclass classification: TabICLv2 clearly outperforms RealTabPFN-2.5 on multiclass tasks across accuracy, AUC, and log-loss.For datasets with more than 10 classes, both its ECOC and native mixed-radix variants outperform RealTabPFN-2.5-ECOC.
  • Regression and dataset size: TabICLv2 outperforms TabPFN-2.5 on RMSE and R2, while TabPFN-2.5 has a slight edge on MAE.TabICLv2 and RealTabPFN-2.5 perform comparably on small datasets, but TabICLv2 has a clear advantage on large datasets.
Loading 2602.11139v2…