TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models

1 Introduction

• Tree ensembles remain dominant because they handle heterogeneous features, variable sample sizes, and small datasets more reliably than deep models.Tabular data lack the spatial or temporal structure that often supports deep-model feature extraction.Source: §1 · p. 1
• TFM adoption is hindered by model-specific preprocessing, fragmented APIs and training protocols, and limited unified evaluation of calibration, fairness, and resource trade-offs.These inconsistencies also make model and tuning-strategy selection difficult.Source: §1 · p. 2
• TabTune standardizes the full TFM workflow through a unified, scikit-learn-compatible toolkit.It exposes a common interface, automates model-specific preprocessing, and supports zero-shot inference, meta-learning, SFT, and PEFT.Source: §1 · p. 2
• TabTune integrates performance, calibration, and fairness diagnostics with benchmarking across accuracy, resource efficiency, and adaptation strategies.The authors present it as an experimental bed for studying trade-offs among these outcomes across TFMs.Source: §1 · p. 2

2.1 Traditional Machine Learning and Deep Learning for Tabular Data

• Tree ensembles handle heterogeneous features, missing values, and modest sample sizes, while deep neural models have not consistently surpassed them on real-world tabular benchmarks.The cited challenges include limited data, complex feature–target relationships, and the lack of large-scale pretraining.Source: §2.1 · pp. 2–3

2.2 Tabular Foundation Models

• TFMs use large-scale pretraining and in-context learning or forward-pass inference; TABPFN, TABICL, and TABDPT illustrate synthetic-prior, scalable ICL, and real-table denoising approaches.The review describes TFMs as broad generalizers on structured data and notes that fine-tuning can improve their performance.Source: §2.2 · p. 3

2.3 Representation Learning and Alternative Approaches

• Alternative representation work includes graph neural networks for feature relationships and extensions of TFMs toward time-series forecasting.TabPFN-v2 is cited as a competitive temporal-task example.Source: §2.3 · pp. 3–4

2.4 Tabular ML Toolkits and Frameworks

• Existing toolkits automate model discovery or support tabular deep-learning components, whereas TabTune is designed specifically to standardize pretrained-TFM preprocessing, adaptation, and diagnostics.The passage characterizes this as bridging pretrained architectures and practical deployment workflows.Source: §2.4 · p. 4

2.5 Adaptation Techniques: Meta-Learning, Fine-Tuning, and PEFT

• Meta-learning retains support–query adaptation, SFT updates all parameters, and PEFT limits updates to low-dimensional subspaces to reduce compute and memory demands.The review describes PEFT as supporting domain adaptation under resource constraints.Source: §2.5 · p. 4

2.6 Calibration, Fairness, and Responsible Deployment

• Responsible deployment assessment involves calibration, subgroup fairness, and robustness, while TFM behavior under distribution shift and noise remains incompletely characterized.The review notes that calibration and strict fairness objectives can conflict and that in-context learning may amplify biases in demonstrations.Source: §2.6 · pp. 4–5

3 System Design

• TabTune decouples preprocessing, tuning, and orchestration into interoperable layers intended to support extensibility and transparency.Its design abstracts operational complexity while preserving flexibility for adaptation, benchmarking, and experimentation.Source: §3 · p. 5

3.1 Design Principles

• The design principles combine a shared interface and fit–predict–evaluate workflow with automatic model-aware preprocessing, hyperparameters, and adaptation procedures.The common workflow is intended to support integration into broader pipelines and reproducible experimentation.Source: §3.1 · p. 5

3.2 Modular Architecture

• TabularPipeline orchestrates model-specific DataProcessor and TuningManager components, while TabularLeaderboard compares model variants across metrics.The TuningManager supports zero-shot, supervised, meta-learning, and PEFT regimes; the processor handles preprocessing and the leaderboard enables ranked evaluation.Source: §3.2 · p. 6, Figure 2 · p. 6

3.3 Supported Models

• TabTune supports seven prominent TFMs with default tuning configurations and aligned preprocessing pipelines.The design aims to preserve model-specific fidelity while requiring minimal configuration.Source: §3.3 · p. 6

4 Implementation

• TabTune’s implementation centers on DataProcessor, TuningManager, and TabularPipeline for standardized preprocessing, adaptation, and inference.The library is described as available through PyPI and installable in existing Python environments.Source: §4 · p. 6, §4.4 · p. 8

4.1 Model-Aware Preprocessing

• Preprocessing routines adapt inputs to each model, from encoded or normalized features to distribution-aware column embeddings and leakage-preventing split-masked attention for Orion models.Orion preprocessing uses train rows for embeddings and Perceiver-style memory; the passage contrasts its distribution-aware embeddings and multi-scale sparsity with TabICL.Source: §4.1 · p. 7

4.2 Flexible Tuning Strategies

• The TuningManager supports zero-shot inference, supervised fine-tuning, episodic meta-learning, and LoRA-based PEFT under a shared training abstraction.The described LoRA configuration uses rank r = 8 and scaling factor α = 16; it can fall back to standard fine-tuning when incompatible.Source: §4.2 · p. 7

4.2.1 PEFT Implementation Details:

• PEFT selects target layers automatically, may apply adapters to eligible linear layers, and reverts to full-parameter fine-tuning when injection is incompatible.The strategy support matrix distinguishes full support, experimental support, and unsupported configurations.Source: §4.2.1 · p. 7, Table 2 · p. 8

4.2.2 Episodic Training Mechanics:

• Meta-learning forms episodes from disjoint support and query sets, remaps class labels to contiguous indices, and excludes episodes containing unseen query classes.This preserves compatibility with fixed-size classification heads.Source: §4.2.2 · p. 8

4.2.3 Data Sampling Strategies

• Optional resampling addresses label imbalance using several over- and undersampling methods, and is restricted to the training split to avoid validation or test leakage.Available methods include SMOTE, random over- and undersampling, Tomek links, k-means centroids, and neighborhood cleaning.Source: §4.2.3 · p. 8

4.3 Evaluation Utilities

• Evaluation utilities report predictive-performance metrics alongside calibration measures and subgroup fairness metrics.The listed metrics include accuracy, precision, recall, F1, AUC-ROC, ECE, MCE, Brier score, SPD, and EOD.Source: §4.3 · p. 8

4.5 Pipeline Abstraction and Workflow

• TabularPipeline coordinates ingestion, preprocessing, adaptation, prediction, and evaluation, while TabularLeaderboard runs model configurations on consistent splits and ranks results.The leaderboard constructs pipeline instances, executes them in parallel, and aggregates rankings; the workflow example requires users to specify the model and adaptation mode.Source: §4.5 · p. 9, §4.6 · p. 9

5.1 Basic Usage and Multi-Model Switching

• Switching among seven supported models requires only a parameter change, with no need to learn new APIs or preprocessing steps.The shared workflow supports rapid prototyping, comparison, deployment, and reproducibility through model persistence.Source: §5.1 · p. 10

5.2 Model Comparison and Results Analysis

• The leaderboard automates cross-validation, metric computation, and ranking across model and tuning configurations, with results exportable as DataFrames.The scenario frames these comparisons as a basis for choosing a model–tuning combination for a dataset and deployment setting.Source: §5.2 · p. 10

5.3 Fine-Tuning Demonstration

• SFT updates all model parameters and typically achieves highest accuracy with sufficient data, while meta-learning uses episodic support and query sets to adapt while preserving in-context learning.The scenarios frame fine-tuning strategies as offering distinct performance and generalization trade-offs.Source: §5.3 · p. 10, §5.3.1 · p. 11, §5.3.2 · p. 11

5.3.3 Parameter-Efficient Fine-Tuning with SFT (PEFT SFT)

• PEFT SFT combines LoRA adapters with supervised fine-tuning, achieving comparable performance to full fine-tuning while reducing memory usage by 60–80%.Source: §5.3.3 · p. 11

5.3.4 Parameter-Efficient Fine-Tuning with Meta-Learning (PEFT Meta-Learning)

• PEFT meta-learning combines LoRA adapters with episodic training to offer parameter efficiency and in-context generalization.Source: §5.3.4 · p. 11

5.4 Comprehensive Model Evaluation

• TabTune evaluates predictive performance with accuracy, precision, recall, F1-score, and AUC-ROC; calibration with ECE, MCE, and Brier score; and fairness with demographic parity and equalized odds.Source: §5.4.1 · p. 12, §5.4.2 · p. 12, §5.4.3 · p. 12

5.5 Model Persistence and Checkpointing

• TabTune supports end-to-end pipeline persistence and checkpointing for reproducible experiments and deployment.Source: §5.5 · p. 12

6.1 Experimental Design

• Evaluation compares adaptation strategies on standardized common subsets of TALENT, OpenML-CC18, and TabZilla; fairness tests use Adult Census Income, German Credit, and COMPAS with SPD, EOD, and EOpD.The evaluated subsets comprise 155/181 TALENT, 63/72 OpenML-CC18, and 27/36 TabZilla datasets; some exclusions resulted from OOM or CUDA issues.Source: §6.1 · pp. 13–14, §6.4 · p. 22

Meta-Learning Fine-Tuning :

• Meta-learning favors ORIONMSP (rank 2.26), SFT favors TABPFN (rank 1.97), and supervised PEFT favors TABDPT (rank 1.91), while TabICL and OrionBiX can degrade under full updates.PEFT retains roughly 95% of full-fine-tuning accuracy with reduced computational overhead.Source: Meta-Learning Fine-Tuning · pp. 14–15

6.2.1 Overall Leaderboards

• On the three-suite leaderboard, zero-shot TabPFN-2.5 leads pretrained TFMs at mean rank 4.13, while TFMs beat classical baselines by roughly 2–4 accuracy points.TabPFN-2.5 records ACC=0.8540/F1=0.8440 on TALENT, ACC=0.8726/F1=0.8672 on OpenML-CC18, and ACC=0.8824/F1=0.8791 on TabZilla.Source: §6.2.1 · p. 14, Table 3 · p. 15

Based on Dataset Size

• By dataset size, TABPFN leads on small and medium data, whereas XGBOOST reaches ACC=0.8911 and F1=0.8961 on large datasets, with ORIONMSP and TABDPT close behind.Fine-tuning TABICL or ORIONBIX on small datasets causes accuracy drops greater than 15%.Source: Based on Dataset Size · p. 16, Table 4 · p. 17

Based on Dataset Width

• On wide datasets, TABPFN-2.0 SFT reaches ACC=0.9346 and F1=0.9335; under severe imbalance, ORIONMSP meta-learning ranks 2.16 (ACC=0.8735, F1=0.8636).Under balanced conditions, TABPFN SFT ranks first at 1.96 (ACC=0.8336, F1=0.8267).Source: Based on Dataset Width · p. 16, Based on Class Imbalance · pp. 16–17, Table 5 · p. 18

6.2.3 Domain-Specific Analysis

• In medical datasets, TABPFN SFT ranks first (1.86), while in finance ORIONMSP meta-learning leads (rank 2.26); TABDPT PEFT remains competitive across both domains.Medical TABPFN SFT records ACC=0.8094 and F1=0.7958; finance ORIONMSP meta-learning records ACC=0.8209 and F1=0.8089.Source: §6.2.3 · pp. 18–20, Table 7 · p. 20

Zero-shot inference

• Zero-shot ORIONMSP and TABICL obtain the lowest ECE (0.0219 on TALENT), while meta-learning preserves TABPFN calibration (ECE=0.0271–0.0388, BS=0.118–0.150) but worsens TabICL on TabZilla.TabICL’s ECE on TabZilla rises from 0.0369 to 0.2033 under meta-learning.Source: Table 8 · p. 21, Zero-shot inference · p. 21, Meta-learning fine-tuning · pp. 21–22

Supervised fine-tuning (SFT)

• Under SFT, TABPFN retains ECE=0.0287–0.0469 and BS=0.124–0.158, whereas TABICL’s ECE triples to quadruples and its Brier score worsens.TABICL’s Brier Score deteriorates from approximately 0.15 to 0.23–0.41.Source: Supervised fine-tuning (SFT) · p. 22

Zero-shot inference.

• Zero-shot MITRA minimizes bias but has ACC = 0.6902; ORIONMSP and ORIONBIX balance ACC = 0.875-0.878 with moderate fairness, while CONTEXTTAB ranks second in fairness.TABPFN has ACC = 0.871 and SPD = 0.307; the results indicate transformer models remain competitively fair without task-specific adaptation.Source: Zero-shot inference · p. 22

Meta-learning fine-tuning.

• Meta-learning ORIONBIX achieves the lowest overall bias (SPD = 0.2021, EOD = 0.1624, EOpD = 0.2010) with ACC = 0.8743, while TABPFN leads accuracy at ACC = 0.8733.ORIONMSP follows closely with EOD = 0.2798 and EOpD = 0.2907; meta-learning maintains fairness similar to zero-shot while modestly improving accuracy consistency.Source: Meta-learning fine-tuning · p. 22

Supervised fine-tuning (SFT).

• Full SFT yields MITRA's lowest bias (SPD = 0.0161, EOD = 0.0170) at ACC = 0.7268; TABICL's SPD falls to 0.055 but ACC falls to 0.4584.TABPFN maintains ACC = 0.8733 with moderate, stable fairness; ORIONBIX improves fairness (EOD = 0.1761) but has ACC = 0.618, while TABDPT reaches ACC = 0.8529.Source: Supervised fine-tuning (SFT) · p. 23

Parameter-efficient fine-tuning (PEFT).

• PEFT meta-learning gives ORIONBIX and ORIONMSP balanced EOD ≈0.27–0.28 and ACC = 0.867–0.875; PEFT SFT's TABDPT achieves ACC = 0.8595 with SPD ≈0.30.PEFT preserves fairness trends while mitigating the accuracy collapse seen in full fine-tuning; MITRA minimizes bias under PEFT SFT (SPD = 0.0029) but has ACC = 0.7133.Source: Parameter-efficient fine-tuning (PEFT) · p. 23

Discussion.

• MITRA achieves best fairness (SPD = 0.0029–0.0193) but low accuracy (ACC = 0.69–0.73), while TABPFN and ORIONMSP balance ACC = 0.87–0.88 and EOD = 0.27–0.29.Meta-learning preserves fairness and accuracy most consistently; SFT can improve parity for some models but typically harms predictive fidelity, while PEFT retains high accuracy and moderate bias.Source: Discussion · pp. 23–24

7.1 Fine-Tuning Strategy Recommendations

• Fine-tuning recommendations are model-dependent: zero-shot suits calibration or resource priorities, meta-learning balances adaptation and generalization, and SFT benefits TABPFN but can degrade TABICL and ORIONBIX.PEFT approaches full-SFT accuracy with lower computational cost; the recommendations distinguish which strategies are suitable for each model.Source: §7.1 · p. 24

7.2 Mechanisms Behind Fine-Tuning Performance

• SFT can erase pretrained representations and overfit high-capacity models, whereas episodic adaptation better preserves transfer and rank stability.The discussion attributes this contrast to support–query-style episodes and reports more stable ORIONMSP ranks under meta-learning.Source: Why SFT Underperforms for TabICL and… · p. 24, Why Meta-Learning Preserves… · p. 25

When Fine-Tuning is Beneficial?

• Fine-tuning is most useful with more than 10K samples, substantial distribution shift, and architectures that retain calibration after updates; zero-shot or meta-learning is more reliable for small or balanced data.PEFT retains approximately 95% of full-SFT accuracy while reducing memory.Source: When Fine-Tuning is Beneficial? · p. 25

7.3 Calibration Insights

• TABPFN maintains low calibration error across strategies, while meta-learning stabilizes calibration for TABPFN and ORIONMSP; SFT sharply worsens TABICL calibration.The discussion recommends TABPFN in any regime or ORIONMSP with meta-learning for calibration-critical applications.Source: §7.3 · p. 25

7.4 Model Selection Guidelines

• Across dataset regimes, TABPFN is a consistent choice, while ORIONMSP is especially suited to large or imbalanced datasets.The recommendations identify ORIONMSP zero-shot or TABPFN meta-learning as defaults when dataset characteristics are unknown.Source: §7.4 · pp. 25–26

7.5 Domain-Specific Guidance

• Medical results favor TABPFN, whereas finance results highlight ORIONMSP meta-learning and efficient TABDPT adaptation for imbalanced data.In medicine, TABPFN SFT ranks 1.86 with ACC 0.8094; in finance, ORIONMSP meta-learning ranks 2.26 with ACC 0.8209 and F1 0.8089.Source: §7.5 · p. 26

7.6 Practical Value of Tabular Foundation Models

• Tabular foundation models offer modest accuracy gains over boosted trees, with reliability and robustness as their primary advantages.The discussion also notes that boosted trees may remain preferable for large, well-labeled, single-domain problems when simplicity and computational efficiency matter.Source: §7.6 · pp. 26–27

7.7 Limitations

• Current limits include classification-only task support, dataset-size and dimensionality constraints, incomplete PEFT coverage, and fairness and calibration evaluation caveats.For example, CONTEXTTAB was evaluated only on smaller benchmark subsets, while sensitive attributes are manually specified per dataset.Source: §7.7 · p. 27

8 Conclusion

• TabTune standardizes TFM workflows by abstracting model-specific preprocessing, consolidating adaptation strategies, and integrating performance, calibration, and fairness evaluation.The library focuses on pretrained TFMs rather than building models from scratch.Source: §8 · p. 27
• Adaptation outcomes depend on the model: TABPFN performs best with SFT or meta-learning, ORIONMSP generalizes robustly especially with meta-learning, and TABDPT PEFT balances accuracy and efficiency.The study emphasizes reliability and fairness alongside predictive performance.Source: §8 · p. 27
• TABPFN consistently achieves excellent calibration, while TABPFN and ORIONMSP maintain a strong fairness–accuracy balance for potential high-stakes deployment.Source: §8 · p. 27

9 Future Work

• Future work will add fine-tuning and meta-learning approaches to strengthen few-shot and domain adaptation capabilities.Source: §9 · p. 27
• Planned evaluation extensions include advanced fairness assessment, interpretability, and uncertainty quantification.Source: §9 · p. 27
• The planned task expansion covers regression, multi-label learning, and time-series modeling beyond binary and multi-class classification.Source: §9 · p. 27

10 Appendix A: Default Hyperparameters

• Default inference and tuning hyperparameters target general-purpose use and can be overridden through configuration dictionaries.The appendix provides default settings for the supported strategies.Source: §10 · p. 30, Table 10 · p. 30, Table 11 · p. 31

10.1 Zero-Shot Inference

• Zero-shot inference uses pretrained weights without parameter updates; model-specific inference settings govern prediction behavior.No training hyperparameters are required, and the model predicts through forward passes using training samples as context.Source: §10.1 · p. 30, Table 10 · p. 30

10.2 Supervised Fine-Tuning (SFT)

• SFT updates all model parameters on labeled task data; it is intended for larger datasets but carries higher memory and overfitting risks.The appendix describes SFT as optimal for datasets above 10K samples and notes that it may reduce generalization on small datasets.Source: §10.2 · p. 30, Table 11 · p. 31

10.3 Meta-Learning Fine-Tuning

• Meta-learning trains on episodic support-query splits to mimic few-shot adaptation, with guidance positioning it for datasets below 10K samples.Episodes whose query classes are absent from the support set are automatically filtered.Source: §10.3 · p. 31, Table 12 · p. 31

10.4 Parameter-Efficient Fine-Tuning (PEFT)

• PEFT applies LoRA adapters with SFT or meta-learning and reports 60-80% lower memory use than full fine-tuning, but model compatibility varies.TabPFN and ContextTab have limited or experimental PEFT support and automatically fall back to base fine-tuning when PEFT is requested.Source: §10.4 · pp. 31–32, Table 13 · p. 32

10.5 Additional Notes

• Classification strategies use CrossEntropyLoss; gradient clipping and learning-rate schedulers are applied where specified in the configuration.Meta-learning episodes with query classes absent from their support sets are filtered for consistency.Source: §10.5 · p. 32

11.1 Hardware and Software Configuration

• Experiments ran on NVIDIA L40S GPUs, with NVIDIA H200 GPUs used for some experiments that encountered out-of-memory issues.Source: §11.1 · p. 33

11.2 Data Splitting and Reproducibility

• Benchmarks use their standardized splits and fixed random seeds; cross-validation follows each suite's stated protocol.TALENT uses auto-discovered splits, OpenML-CC18 predefined splits, and TabZilla its standard evaluation protocol.Source: §11.2 · p. 33

11.3 Dataset Statistics

• Dataset statistics cover names, domains, sample and feature counts, class counts, and task types across the evaluated benchmark suites.Source: §11.3 · p. 33

• Table 14 lists the 72 datasets in the OpenML-CC18 benchmark suite.Source: OpenML-CC18 Benchmark Datasets · p. 33

TALENT Benchmark Datasets

• Table 15 lists datasets from the TALENT benchmark suite.Source: TALENT Benchmark Datasets · p. 33, Table 15 · p. 35

Fairness Evaluation Datasets

• Fairness evaluation uses datasets with demographic or other sensitive attributes to compute group-disparity metrics.The metrics are computed where sensitive features are explicitly identified.Source: Fairness Evaluation Datasets · p. 33, §11.4 · p. 39, Table 17 · p. 39

11.4 Evaluation Metrics

• Experiments report performance, calibration, and fairness metrics using standard implementations and benchmark protocols.The listed metrics include accuracy, AUC-ROC, weighted F1-score, ECE, MCE, Brier score, SPD, EOD, and EOpD.Source: §11.4 · p. 33

TabZilla Benchmark Datasets

• Table 16 lists 36 TabZilla benchmark datasets accessed via OpenML.The datasets are selected for evaluating neural-network performance on tabular data.Source: TabZilla Benchmark Datasets · p. 33, Table 16 · p. 38

12 Appendix C: Glossary of Key Terms

• The glossary provides concise definitions of technical terms used throughout the paper.Source: §12 · p. 39, Table 18 · p. 40