TabDPT: Scaling Tabular Foundation Models on Real Data

• Tabular foundation models must generalize across highly heterogeneous datasets, while per-dataset training requires costly model selection and hyperparameter tuning.Source: §1 · p. 1
• Increasing model size or pre-training data size, measured by number of cells, leads to consistent improvements that are predictable by power laws.Source: §1 · p. 1
• ICL retrieval combined with column-masking SSL provides TabDPT’s pre-training procedure for generalization to unseen classification and regression tasks without explicit fine-tuning.Source: §1 · p. 2
• On OpenML-CC18 and OpenML-CTR23, TabDPT consistently matches or surpasses specialized models extensively tuned per dataset, while achieving strong few-shot performance and outperforming specialized baselines on 10-shot classification tasks.Source: §1 · p. 2
• Real-data pre-training converges faster and improves downstream accuracy over training exclusively on open-source synthetic data generators.Source: §1 · p. 2
• TabDPT is released with open model weights, inference code, training code, and pre-training datasets to support reproducibility.Source: §1 · p. 2

• Many tabular foundation models require additional training on novel tasks, while LLM-based ICL struggles with numerical tokenization, memory costs, low performance, and row-order sensitivity.Source: §2 · p. 3
• Tabular-specific ICL methods handle numerical tables more naturally but depend heavily on synthetic generators whose coverage of real-world diversity is difficult to ensure.Source: §2 · p. 3
• This work presents a broad scaling-law analysis for TFMs, extending preliminary small-scale tabular evidence to systematic variation of real-data amount and model size.Source: §2 · p. 3
• Existing tabular SSL methods often require task-specific fine-tuning, whereas TabDPT claims competitive cross-task generalization without task-specific training or hyperparameter tuning.Source: §2 · p. 3

3 TabDPT Methodology

• TabDPT combines row-based in-context learning, self-supervised learning, and retrieval-based context selection to generalize across unseen classification and regression tasks without dataset-specific fine-tuning.These components form the paper’s foundation-model approach.Source: §3 · p. 3

3.1 Tabular Transformer Encoder

• Row-based encoding reduces sequence length from cell-level O(N × F × ⟨Ntok⟩) to N tokens, while omitting positional encodings for row-order invariance.Columns are approximately randomized during training to improve column-order robustness.Source: §3.1 · p. 4

3.2 Self-Supervised Learning on Tabular Data

• Self-supervised pre-training masks a randomly selected column as a regression or classification target, with column shuffling and dropping producing combinatorially varied tasks and richer inter-feature training signals.This strategy extracts more information from limited real datasets while regularizing the model.Source: §3.2 · p. 4

3.3 Retrieval-Based Pre-Training

• Retrieval-based training samples nearby rows, splits them into context and query groups, and predicts query targets using context embeddings through a transformer.The same retrieval structure is used at inference, and the authors report faster convergence and better downstream accuracy.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 Inference on New Data

• At inference, TabDPT retrieves the top K training rows for each query and performs only forward passes, while base-Cmax encoding and PCA address excess classes and features.Nearest-neighbor retrieval can add latency, but FAISS is described as providing millisecond-scale responses; excess classes require ⌈logCmax(C)⌉ forward passes.Source: §3.4 · p. 5

4.3 Results on CC18 and CTR23

• TabDPT shows the best overall performance across CC18 and CTR23, surpassing specialized deep-learning and tree-based methods and significantly outperforming TabPFN v2 on CC18 accuracy.The evaluation uses average performance across four metrics with 95% confidence intervals.Source: §4.3 · p. 6, Table 1 · p. 7
• TabDPT remains competitive without validation data, being outperformed on CC18 accuracy only by TabM using 30 hyperparameter-tuning rounds.The no-validation variant is disadvantaged relative to methods that tune on validation data.Source: §4.3 · p. 7
• Pairwise win rates rank TabDPT first overall, followed by TabPFN v2, while Tabula-8B with 32 shots is not competitive across the benchmarks.Wins are based on higher CC18 accuracy or CTR23 R2 for each dataset.Source: §4.3 · p. 7, Figure 3 · p. 8

4.4 Ablation Study

• Removing SSL causes large performance loss and overfitting after about 50 epochs, whereas SSL continues improving beyond 500 epochs on real data.The ablation identifies SSL as critical when each real dataset typically provides only one target.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
• Removing retrieval during training or inference consistently reduces classification and regression performance, while reducing context to 256 also hurts performance but less severely.The results support aligning training and inference context construction.Source: §4.4 · p. 8, Figure 4 · p. 8
• Real data with SSL consistently achieves lower test loss than synthetic training across epochs under the same compute budget.Synthetic and real-without-SSL training both saturate at 500 epochs.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 Scaling Laws

• Scaling model size and training data produces consistent improvements with fitted power laws, with scaling exponents α = 0.42 and β = 0.39.Models range from 33K to 78M parameters and training subsets from 52M to 2B cells.Source: §4.5 · p. 9

4.6 Few-Shot Learning

• With pseudo-label augmentation in the 10-shot setting, TabDPT outperforms STUNT on 5 of 7 datasets and on average accuracy without weight updates.TabDPT (semi) uses ten labeled examples per class plus pseudo-labeled unlabeled rows.Source: §4.6 · p. 9, Table 2 · p. 9

5 Conclusion and Future Work

• TabDPT generalizes across unseen tabular datasets without additional tuning, and its performance improves consistently with both model and data size.The authors characterize these improvements as scaling laws analogous to those observed in other foundation-model domains.Source: §5 · p. 10
• TabDPT is limited to rectangular datasets with i.i.d. rows and does not explicitly model temporal dependencies, distribution shifts, or hierarchical structures.The authors suggest future methodological extensions for temporal dependencies, generative modeling, and density estimation.Source: §5 · p. 10

A Bitter Lessons

• The authors conclude that efficient computation and access to high-quality data were more important for performance than architectural manipulations in their experiments.They explicitly frame this conclusion as an experience-based lesson rather than a universal guarantee.Source: A Bitter Lessons · p. 15
• Architectural changes such as robust preprocessing, class embeddings, specialized missing-value tokens, and cell-wise attention generally failed to improve performance or increased memory use.The simpler row-based architecture allowed a higher embedding dimension, while the authors caution that these lessons reflect their own real-data training experience.Source: A Bitter Lessons · p. 15

B Training Datasets

• The corpus includes 93 datasets with classification targets, 29 with regression targets, and one without a default target.The SSL procedure nevertheless generates both classification and regression targets for each dataset.Source: B Training Datasets · p. 15
• Figure B.1 summarizes the training datasets by size and domain.The full dataset inventory is provided separately in Table B.1.Source: B Training Datasets · p. 15
• Table B.1 lists each training dataset’s OpenML identifier, name, dimensions, missing-cell percentage, target type, and domain.The listed dimensions include rows, features, and cells.Source: Table B.1 · p. 15

B.1 Contamination Analysis

• Dataset contamination screening compares names, file hashes, dimensions, target statistics, feature moments, and feature–target relationships across training and evaluation datasets.Feature statistics are organized with k-d trees to enable efficient pairwise comparisons.Source: §B.1 · p. 17
• Pairs with unusual similarities were manually evaluated, and related datasets were removed from training.Source: §B.1 · p. 17

C.1 Architecture Details

• The model combines input embeddings, multiple transformer encoder layers, and task-specific output heads.The architectural parameters are summarized in Tables C.1 and C.2.Source: §C.1 · p. 17, Table C.1 · p. 18
• Preprocessing maps non-numerical values to integers, standardizes columns, clips outliers beyond 10, and standardizes retrieved context before inference.Regression context labels are also standardized to reduce distribution shifts.Source: §C.1 · p. 17
• Retrieval uses FAISS in the preprocessed raw feature space, while missing values are zeroed after normalization so they correspond to the feature mean.Adding an explicit missingness mask produced minimal improvement at nearly double the compute cost.Source: §C.1 · p. 18
• Training uses Schedule Free optimization with AdamW, label smoothing, weight decay, batch size 256, context and query lengths 1024, and bfloat16 parameters.The default learning rate is 5 × 10^-4 and weight decay is 5 × 10^-2.Source: §C.1 · p. 18

D Pseudo-Code for Training Algorithms

• Randomly sampled datasets and queries are preprocessed, indexed with FAISS, and used to retrieve approximate neighbors after masking the target column.Features and targets are then randomly subsampled and partitioned before training.Source: D Pseudo-Code for Training Algorithms · pp. 18–20
• Each training step samples an evaluation position, uses preceding points as context and following points as queries, then optimizes the task-specific loss.The training loop operates over epochs and batches produced by the dataloader.Source: D Pseudo-Code for Training Algorithms · pp. 18, 20
• The dataloader generates random classification or regression targets for training, supporting varied training tasks; targets with more than 10 unique values remain regression with 70% probability.Classification targets are created by binning values into a randomly sampled number of classes and label encoding the resulting labels.Source: D Pseudo-Code for Training Algorithms · p. 19

E Elo and Glicko2 Ratings

• TabDPT performs best on both Elo and Glicko2 ratings, followed by the leading TFM baseline TabPFN v2.Elo uncertainty is estimated by bootstrapping match-order permutations, whereas Glicko2 models uncertainty directly and is less sensitive to match order in these experiments.Source: E Elo and Glicko2 Ratings · p. 20, Figure E.1 · p. 21

F.1 Additional Results by Dataset Statistics

• TabDPT is robust across dataset sizes, feature counts, categorical fractions, and missing fractions, with only a slight relative decrease on very large CC18 datasets.The reported decrease can be mitigated by fine-tuning as suggested by Thomas et al. [58].Source: §F.1 · p. 22, Figure F.1 · p. 22, Figure F.2 · p. 22
• The additional analyses separately report AUC on CC18 and R2 on CTR across bins of feature count and categorical-feature fraction.These visualizations organize performance comparisons by dataset characteristics rather than reporting a single aggregate result.Source: §F.1 · p. 22

G Critical Difference Diagrams

• Critical-difference diagrams compare method uncertainty across CC18 and CTR23 fold results using ranks and multiple-hypothesis-testing adjustments.Their confidence intervals differ from Table 1 because the diagrams assess individual dataset realizations, whereas Table 1 uses IQM-based uncertainty over suite realizations [1].Source: G Critical Difference Diagrams · p. 23, Figure G.1 · p. 24, Figure G.2 · p. 24