PaperlayerView exact revision on arXiv

Source-linked AI summary

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan, Sujay Sanghavi, Abhimanyu Das, Weihao Kong

arXiv:2609.37989v1cs.LG

TL;DR

Tabular foundation models miss semantics in column names and metadata, while searching with models trained from scratch can be noisy. TabFM-Auto uses an LLM agent to evolve pipelines around frozen TabFM using metadata and validation feedback. Across TabArena, its five configurations take the top five positions, and it also transfers to other models and ranks first on MLE-Bench-Tabular.

  • Problem

    Tabular foundation models overlook column-name and task semantics, while from-scratch MLE-agent searches jointly over features, architectures, and hyperparameters can be noisy.

    Source: §1 · p. 2

  • Method

    TabFM-Auto uses an LLM agent guided by dataset metadata and validation feedback to refine data cleaning, feature engineering, context selection, and post-processing around frozen TabFM.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across 51 TabArena datasets, five configurations take the top five positions, with the best reaching 2013.0 Elo; pipelines transfer to other frozen models by +68.8 to +143.3 Elo, and TabFM-Auto ranks first on MLE-Bench-Tabular.

    Source: §5 · pp. 10–11

  • Takeaways & Limitations

    The approach combines language-model domain reasoning with fast forward-pass inference to supply semantics missing from synthetic pretraining.

    Source: §5 · p. 10

    Gains are generally smaller on anonymized tables, and pipeline search requires repeated validation evaluations for each dataset.

    Source: §5 · p. 11

Abstract

from arXiv · show

Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM's error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. The discovered pipelines also transfer to other frozen tabular foundation models (+69 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.

1. Introduction

Tabular foundation models overlook dataset semantics in column names and metadata, while training models from scratch during pipeline search can make that search noisy. TabFM-Auto uses an LLM agent to evolve a pipeline around frozen TabFM and reports leading results across TabArena and MLE-Bench.

Motivation

  • Tabular foundation models process numerical magnitudes and category indices but cannot use column names to infer domain formulas, code hierarchies, or missing-value meanings.Generic feature operations in TabFM+ likewise ignore what columns represent.Source: §1 · pp. 1–2
  • LLMs understand column and task semantics but perform poorly as direct tabular predictors, while training-from-scratch MLE agents face noisy joint searches over features, architectures, and hyperparameters.Training variance can mask small feature gains and lead to validation overfitting.Source: §1 · p. 2

Approach

  • TabFM-Auto uses dataset metadata to guide an LLM agent that iteratively edits cleaning, feature engineering, context selection, and post-processing around frozen TabFM.The agent searches with 3-fold cross-validation on training data, then evaluates the selected pipeline once on the official test set.Source: §1 · p. 2

Results

  • Five TabFM-Auto configurations take the top five overall positions across 51 TabArena datasets; Codex with Opus 5 raises TabFM from 1785.3 to 2013.0 Elo.The configurations also outperform tuned AutoGluon ensembles and all evaluated tabular foundation models, transfer to other models with up to +143.3 Elo, and rank first on eight MLE-Bench tabular competitions.Source: §1 · pp. 2–3

2. Related Work

Prior work spans synthetic-data-pretrained tabular models, automated model selection, LLM-based feature engineering, and end-to-end MLE agents. TabFM-Auto instead uses an LLM agent to evolve the full data pipeline around a frozen tabular foundation model.

Tabular Foundation Models and In-Context Optimization

  • Synthetic-data-pretrained tabular models make predictions in one forward pass, but typically ignore column names and task descriptions at test time; TabFM-Auto keeps TabFM frozen and turns metadata into pipeline code.Prior work extends the paradigm to larger contexts, regression, real-data pretraining, or linear-time attention (Hollmann et al., 2025; Qu et al., 2025, 2026).Source: §2 · p. 3

AutoML, LLM Feature Engineering, and MLE Agents

  • End-to-end agents such as AIDE (Jiang et al., 2025), R&D-Agent (Yang et al., 2025), and MLEvolve (Du et al., 2026) retrain tree ensembles or neural networks during search, whereas TabFM-Auto keeps TabFM frozen and searches only over the data pipeline.Source: §2 · p. 4
  • LLM feature-engineering methods such as CAAFE, FeatLLM, and OCTree add derived columns, whereas TabFM-Auto evolves cleaning, context-row selection, calibration, and auxiliary-file reading too.Source: §2 · p. 3

3. Methodology

TabFM-Auto searches executable data pipelines around a frozen TabFM model using metadata and internal validation, without using held-out test labels. Its pipeline stages address table values and features, context rows, and prediction outputs.

3.1. Problem Formulation

  • The agent searches pipelines conditioned on metadata to maximize mean 3-fold validation utility around fixed TabFM weights, without touching the held-out test split.The formulation allows inductive and transductive learning by passing unlabeled validation or test inputs alongside the training split while withholding evaluation labels.Source: §3.1 · p. 4

3.2. Building Pipelines Around Frozen TabFM

  • Data cleaning handles semantically meaningful values and target transformations, while feature engineering converts metadata into domain formulas, group statistics, auxiliary-file summaries, and label-free transductive features.Writing domain formulas as explicit columns spares TabFM from inferring them in context, which helps most on real-world schemas.Source: §3.2 · p. 4
  • Context selection builds class-stratified or minority-oversampled views to address context-size limits, and post-processing restores transformed targets or calibrates classification outputs.Combining predictions across views lets TabFM use more rows than fit in one context window; calibration can account for class-prior and confidence mismatches.Source: §3.2 · pp. 4–5
  • The agent evolves an identity pipeline using cross-validation in a sandbox, then freezes the selected pipeline and scores it on official test splits.With TabFM fixed, candidates need no gradient training or retraining, and improvements over the identity pipeline come from the agent's pipeline changes.Source: §3.2 · p. 5

4. Experiments

TabFM-Auto is evaluated on TabArena and MLE-Bench-Tabular, with pipeline search run once per TabArena dataset and the resulting pipeline assessed on held-out test splits. It leads both benchmarks, while transfer tests, an ablation, and search-dynamics analyses examine the scope and behavior of its results.

4.1. TabArena

  • Search runs once per dataset using fold 0’s training data, then the frozen pipeline is tested on held-out splits across all folds.The benchmark covers 51 datasets, with up to 96 search evaluations or six hours per dataset.Source: §4.1 · p. 5
  • The five configurations take TabArena’s top five overall positions, reaching 2013.0 Elo; they raise TabFM by +197.6 Elo on classification and +467.0 Elo on regression.The regression gain is reported across all 13 datasets.Source: §4.1 · p. 6
  • Cross-validation gains carry over to official test sets: mean test error falls 4.0%–4.7% from the initial pipeline to full search budget, and Elo reaches 1979–1993.Both configurations exceed TabFM+’s 1856 Elo within about three evaluations.Source: Figure 4 · p. 7, §4.1 · pp. 7–8
  • On 17 datasets with meaningful schema descriptions, domain formulas reduce test error by 7.3%–8.2%, versus 2.3%–3.0% on datasets with anonymized or generic schemas.The passage attributes the larger reductions to agents writing formulas from dataset schemas.Source: §4.1 · p. 7
  • On runs without feature-table changes, context sampling or prior calibration improves four of five runs; hyperparameter-only tuning worsens the remaining run by 0.30%.Across the two inspected configurations, feature-table modifications occur in 97/102 runs.Source: §4.1 · p. 7
  • Under a matched six-hour budget, TabFM-Auto reaches 1979.6 Elo versus 1468.8 for the unconstrained coding agent; the ablation attributes +316.5 Elo to TabFM’s pretrained prior and +194.3 to pipeline search.Table 2 describes the controlled comparison’s shared harness, model, sandbox, and search budget.Source: Table 2 · p. 8, §4.1 · p. 8
  • Using the searched pipelines without further search improves all three tested foundation models; Claude Code with Opus 5 yields gains of +143.3 Elo for TabICLv2, +130.8 for TabPFN-3, and +88.7 for EXAONE-Tabular.Figure 5 compares each model’s default result with the same frozen model run inside the searched pipelines.Source: Figure 5 · p. 8, §4.1 · p. 9

4.2. MLE-Bench-Tabular

  • On MLE-Bench-Tabular, TabFM-Auto ranks first and second overall among evaluated submissions, at 1827 Elo and 1734 Elo, respectively.The Opus 5 configuration wins 86.7% of graded pairwise matches; Figure 7 presents overall Elo and pairwise win rates.Source: §4.2 · pp. 9–10, Figure 7 · p. 10

5. Conclusion and Discussion

TabFM-Auto evolves the data pipeline around a frozen tabular foundation model, and its configurations lead on TabArena and MLE-Bench-Tabular. The discussion identifies smaller gains on anonymized tables and repeated validation evaluations as limitations.

5. Conclusion and Discussion

  • TabFM-Auto pairs language-model domain reasoning with fast forward-pass inference by evolving the pipeline around a frozen tabular foundation model.The approach is intended to supply semantics missing from synthetic pretraining.Source: §5 · p. 10
  • Across 51 TabArena datasets, the five configurations take the top five positions, and TabFM-Auto ranks first overall on MLE-Bench-Tabular.The best TabArena result reaches 2013.0 Elo, +227.7 over TabFM; discovered pipelines transfer to other frozen models by +68.8 to +143.3 Elo.Source: §5 · pp. 10–11
  • Gains are generally smaller on anonymized tables, and pipeline search requires repeated validation evaluations for each dataset.The discussion suggests reusable operations as a possible way to warm-start future searches with fewer evaluations.Source: §5 · p. 11

A. Verification Protocol and Test Isolation

The harness checks that TabFM-Auto’s gains come from pipeline changes evaluated with frozen TabFM, without access to held-out splits. It also guards against row-order leakage and imposes implementation constraints on candidate pipelines.

A. Verification Protocol and Test Isolation

  • The harness evaluates every candidate with the same frozen 400M-parameter TabFM checkpoint and accepts edits only when they improve on the identity pipeline’s validation score.The identity pipeline passes the raw table to TabFM with default constructor settings; the harness, not the pipeline, calls the model.Source: A. Verification Protocol and Test… · p. 16, Figure 8 · p. 16
  • The harness passes predictions to postprocess() and drops the hook-provided row_id column before calling TabFM.engineer() may return at most 500 columns, so the identity pipeline fails on Bioresponse, hiva_agnostic, and QSAR-TID-11.Source: A. Verification Protocol and Test… · p. 16
  • During search, the agent sees only Fold 0’s training split, and the chosen pipeline is scored once on held-out splits after the search is frozen.The sandbox unmounts Fold 0’s official test split and the train and test splits of all other folds and datasets.Source: A. Verification Protocol and Test… · p. 16
  • Classification folds are shuffled at load time to prevent pipelines from using row positions as a proxy for labels or acquisition time.The harness restores original order only when scoring final predictions.Source: A. Verification Protocol and Test… · p. 17

B.1. Overall Leaderboard and Baseline Comparison

TabFM-Auto configurations lead the TabArena overall and task-specific leaderboard, while reducing TabFM’s oracle improvability and increasing dataset wins. Fold-0 held-out results and cross-fold comparisons provide evidence about performance beyond the search split.

B.1. Overall Leaderboard and Baseline Comparison

  • 1.26%–1.88% is TabFM-Auto’s overall oracle improvability, down from 3.20% for TabFM and 2.20% for TabFM+.Antigravity with Gemini 3.8 Flash records 26.12 dataset wins and Codex with Opus 5 records 25.00, versus 9.74 for TabFM.Source: §B.1 · p. 17
  • One pipeline search per dataset reuses the selected operations across folds, but later-fold test rows may have appeared in Fold 0’s training data; Fold 0 is the only fully held-out split.The paper states that independent searches on all 816 folds across five configurations would increase LLM token and GPU compute costs by 16×.Source: §B.1 · pp. 18–19
  • 1950.7 Elo gives Codex with Opus 5 the #1 Overall rating, while TabFM-Auto holds #1 Classification and #1–#4 Regression ratings.All five configurations outrank EXAONE-Tabular (1772.2) and 4-hour AutoGluon 1.5 (1661.0) overall.Source: §B.1 · pp. 17, 19, Figure 9 · p. 19
  • 3.74% vs. 4.53% classification and 3.23% vs. 3.28% regression G-Mean test error reductions over TabFM show similar Fold-0 and later-fold results for Codex with Opus 5.Across the five configurations, pairwise win rates against 62 TabArena methods are 91.1%–95.7% on Fold 0 and 94.4%–95.9% on later folds.Source: §B.1 · p. 19

B.2. Analysis of Discovered Pipelines

The discovered pipelines adapt TabFM’s inputs, context, and outputs using metadata and validation feedback. Their approaches differ between schemas with identifiable domain quantities and generic or anonymized columns.

Data Cleaning and Target Conditioning

  • 33.3% of datasets receive cleaning or target-conditioning changes, including masking sentinel values and applying reversible transforms to skewed regression targets.The agents inspect column names and value histograms; postprocessing inverts target transforms.Source: §B.2 · p. 21

I: Domain-Knowledge Features

  • +7.25% (Opus 5), +8.16% (Gemini 3.8 Flash), and +8.85% (Combined Best) are the multi-fold official test error reductions on 17 informative-schema datasets.These domain-feature datasets account for 6 of each configuration’s 10 largest TabArena dataset improvements.Source: §B.2 · p. 21
  • +14.6% and +17.3% test RMSE reductions on airfoil_self_noise accompany explicit engineering features built from variable names.Other examples include parsed ICD-9 codes and clinical vital-sign ratios, splice sequence motifs (+22.2% test log-loss reduction), and SDSS17 color indices (+18.8% log-loss reduction).Source: §B.2 · p. 21

II: Statistical and Structural Features

  • +2.31% (Opus 5), +2.99% (Gemini 3.8 Flash), and +3.87% (Combined Best) are the multi-fold test error reductions on generic-schema datasets.On Amazon_employee_access, label-free graph features reduce 1 − AUROC error from 0.1416 to 0.1039 for Opus 5 and 0.1063 for Gemini 3.8 Flash.Source: §B.2 · pp. 21–22

Context Selection

  • Context selection changes on 25.5% of datasets for Opus 5 and 23.5% for Gemini 3.8 Flash, often using multiple context sets to retain rare positives.The agents use the natural class distribution alongside minority-oversampled or cluster-stratified views when tables exceed 16,384 rows or have severe class imbalance.Source: §B.2 · p. 22

III: Output Calibration and Constructor Settings

  • +2.16% average test error reduction across five runs with unchanged feature matrices accompanies output calibration and/or context changes in four runs.Output calibration shifts probabilities toward the empirical training class prior in log-odds space and applies temperature scaling; the sole constructor-only run scores −0.30%.Source: §B.2 · p. 22

B.3. Transferring Final Pipelines to Other Tabular Foundation Models

TabFM-Auto’s discovered pipelines are applied unchanged around other frozen tabular foundation models and improve their ratings on both task types. The unconstrained coding agent’s pipelines also commonly include gradient-boosted trees.

B.3. Transferring Final Pipelines to Other Tabular Foundation Models

  • The transfer evaluation applies each selected pipeline’s four functions unchanged around another frozen model and scores all official test folds across 51 datasets.Context-row counts are retained, while other TABFM_KWARGS are not transferred when the target models do not support them.Source: §B.3 · pp. 22–23
  • +88.7 to +143.3 Elo are the overall gains from Claude Code with Opus 5 pipelines, the largest gains for every evaluated model.All three configurations’ pipelines improve every other model on both task types, by less than TabFM gains from the same pipelines.Source: §B.3 · p. 23, Table 7 · p. 23

B.4. Models Used by the Unconstrained Coding Agent

  • 45 of 51 unconstrained-agent pipelines use gradient-boosted trees, mostly LightGBM (35) and CatBoost (27); only 6 use a neural network.Twenty-three pipelines use one model family and 28 blend two or more; none use XGBoost.Source: §B.4 · p. 23

C. Search Dynamics and Generalization Analysis

TabFM-Auto’s search reduces official test error as validation error falls, and most validation gains transfer to held-out folds. The section also examines pipeline transfer to other foundation models and the model families used in discovered pipelines.

C. Search Dynamics and Generalization Analysis

  • Rescoring 1,486 checkpoints across official test folds tests whether repeated fold-0 cross-validation evaluations overfit validation.The checkpoints came from Claude Code with Opus 5 and Antigravity with Gemini 3.8 Flash, at fixed fractions of the search budget.Source: C. Search Dynamics and Generalization… · p. 23
  • The final pipelines help other tabular foundation models, with pipelines found by Claude Code with Opus 5 helping most.Figure 12 also reports benefits from pipelines found by the other two configurations.Source: Figure 11 · p. 24, Figure 12 · p. 24
  • Figure 13 counts model families used across 51 final pipelines; blended pipelines count once per family, so family percentages can sum to more than 100%.The right pie chart splits the GBDT family by library.Source: Figure 13 · p. 25
  • Figure 14 tracks official test G-Mean error reduction over search progress separately for 13 regression datasets and 38 classification datasets.The plotted progress is log-scaled.Source: Figure 14 · p. 25
  • At full budget, test error is 1.93% (Opus 5) and 1.97% (Gemini 3.8 Flash) lower on regression, and 4.90% and 6.07% lower on classification than P0.Both suites’ official test error decreases alongside validation error throughout search.Source: C. Search Dynamics and Generalization… · p. 25, Figure 14 · p. 25
  • In most search quarters, datasets with validation gains also improve on test; the exception is Gemini 3.8 Flash in the 50%–75% quarter, where test changes split roughly evenly.In the final quarter, validation and test gains remain positively correlated among datasets whose validation score still improves.Source: C. Search Dynamics and Generalization… · pp. 25–26, Figure 15 · p. 26

D.1. Competition-by-Competition Analysis on MLE-Bench-Tabular

Across eight MLE-Bench-Tabular competitions, TabFM-Auto’s Opus 5 and Gemini 3.8 Flash configurations rank first and second overall. Competition-specific pipelines use structured domain features, including signal processing, crystal geometry, and molecular geometry.

D.1. Competition-by-Competition Analysis on MLE-Bench-Tabular

  • TabFM-Auto ranks #1 with 1827 Elo and an 86.7% win rate, while its Gemini 3.8 Flash configuration ranks #2 with 1734 Elo and an 80.0% win rate.These pairwise Elo ratings cover graded submissions against 12 external MLE agents with complete eight-competition submissions.Source: §D.1 · p. 26
  • On volcanic-eruptions, FFT spectral energy, STA/LTA trigger ratios, signal envelopes, and inter-sensor correlations lower Opus 5’s official test MAE to 97,355, earning Gold.The error is 9.4× lower than the strongest external agent’s score on MLE-Bench’s official local split.Source: §D.1 · p. 26
  • Figure 16 plots best-so-far 3-fold cross-validation curves and final official test scores across eight competitions; two begin with zero-feature identity-pipeline failures.The zero-feature failures are on volcanic-eruptions and smartphone-decimeter.Source: Figure 16 · p. 27
  • On transparent-conductor prediction, physical crystal features reduce official test RMSLE from 0.05737 for TabFM to 0.04440 for Opus 5, earning Gold and ranking #1 among evaluated agents.The features include unit-cell volume, atomic packing density, reciprocal lattice constants, and composition-weighted properties.Source: §D.1 · p. 27
  • For scalar-coupling prediction, geometry features and stratified context windows lower official test Log-MAE to −1.54947 with Opus 5.The pipeline derives inter-atomic distances, bond-path angles, and dihedral torsion angles from molecular coordinates.Source: §D.1 · p. 27
  • Treating ventilator-pressure-prediction time steps as independent rows gives TabFM 3.78450 MAE because that representation ignores accumulated air in the lung.The competition provides 80 time steps per breath, along with valve states and lung resistance and compliance.Source: §D.1 · p. 27
  • For smartphone-decimeter, the Opus 5 GNSS pipeline reduces official test Haversine error to 4.3284 m, 18.9% below Famou-Agent 2.0 at 5.334 m.The pipeline combines elevation-weighted carrier-to-noise density, pseudorange residual dispersion, and smoothed positions across epochs.Source: §D.1 · p. 28
  • Geometric and structural features raise dec-2021 accuracy to 0.96264, lower may-2022 test 1 −AUROC error from 0.05179 to 0.00179, and reduce NYC taxi-fare test RMSE by 12.5%.Features include terrain distances and aspect, manufacturing-string character positions, and distances aligned to Manhattan’s street grid.Source: §D.1 · pp. 28–29

E. Per-Dataset Official Test Scores on TabArena

TabFM-Auto reduces overall official test error across the 51 TabArena datasets and improves results across regression, binary classification, and multiclass classification suites. Gains are not uniform: some increases occur on nearly saturated datasets and small, noisy credit tables.

E. Per-Dataset Official Test Scores on TabArena

  • On all 13 regression datasets, TabFM-Auto improves official test RMSE over TabFM, with suite G-Mean reductions of 3.11% (Opus 5) and 3.15% (Gemini 3.8 Flash), versus 1.35% for TabFM+.The largest listed reductions include airfoil_self_noise (+17.3%, Gemini 3.8 Flash), physiochemical_protein (+3.9%, Opus 5), concrete_strength (+3.8%, Opus 5), and miami_housing (+3.1%, Opus 5).Source: E. Per-Dataset Official Test Scores on… · p. 29
  • On binary classification, suite G-Mean test classification-error reductions are 3.51% (Opus 5) and 5.17% (Gemini 3.8 Flash), versus 0.81% for TabFM+.The largest listed per-dataset reductions include Amazon_employee_access (0.1416 → 0.1039, Opus 5) and churn (0.0681 → 0.0491, Gemini 3.8 Flash).Source: E. Per-Dataset Official Test Scores on… · p. 29
  • Binary-classification errors increase by at most +0.0034 in absolute 1 −AUROC; the largest increases occur on nearly saturated datasets and small, noisy credit tables.On the latter, fold-0 validation gains do not fully transfer across small evaluation folds; all other increases are at most +0.0005.Source: E. Per-Dataset Official Test Scores on… · p. 29
  • Across all 51 datasets, TabFM-Auto reduces overall G-Mean test error by 4.19% with Opus 5 and 5.05% with Gemini 3.8 Flash, roughly four to five times feature engineering’s +1.06% reduction.Source: E. Per-Dataset Official Test Scores on… · p. 29
  • On multiclass classification, test log-loss reductions are 8.39% (Opus 5) and 7.64% (Gemini 3.8 Flash), versus 1.50% for TabFM+, improving 7 of 8 datasets.The reported largest reductions combine domain feature engineering with multiclass temperature scaling and log-odds prior alignment.Source: E. Per-Dataset Official Test Scores on… · p. 29
Loading 2609.37989v1…