Agentic Search Spaces for Tabular Machine Learning

1 Introduction

• The approach addresses limited general impact from prior LLM tabular methods, which appear most useful for small datasets or semantically informative features.Practitioners often hand-tune frozen model recipes without a systematic way to explore alternatives or their interactions.Source: §1 · p. 1
• Across 45 datasets, agent-augmented spaces improve aggregate rank for nearly every family, with single-model gains of 0.5% for DNNs, 0.8% for GBDTs, and 0.5% for TabICLv2.On TabArena, Elo improves for four of five model families, by up to 125 points.Source: §1 · p. 2
• The method splits a model into modules, has an agent generate candidate implementations, and uses classical HPO to search their joint space alongside default hyperparameters.Candidate spaces are generated once per model, without access to target data, and tuned per dataset.Source: §1 · p. 2
• The released extended HPO spaces cover five families: MLP†, TabM†, LightGBM, RealMLP, and TabICLv2.The paper also makes its prompts and harnesses publicly available.Source: §1 · p. 2
• Ensembles gain 0.9% for DNNs, reaching up to 2.0% on small-to-medium regression tasks; the agentic RealMLP ensemble is outperformed only by multi-model AutoGluon.The paper also reports consistent gains on TabArena.Source: §1 · p. 2

AutoML and architecture search

• Unlike Auto-sklearn and AutoGluon’s broader model-library searches, this work starts from one model and searches its modifications, with agent-written candidates conditioned on that model.AutoPyTorch and AutoKeras are closer because they search modifications within a model family, but their components are hand-implemented.Source: §2 · p. 3

LLMs for tabular prediction

• Prior LLM tabular predictors either predict from serialized rows or generate semantic features; reported gains occur in settings such as small datasets or rich textual metadata.Source: §2 · p. 3

LLM agents for optimization and research automation

• Unlike LLM optimizers that alter how a fixed space is explored, this approach has agents generate modules along predefined pipeline axes before standard HPO.The design preserves dataset privacy, amortizes agent cost, and makes spaces reproducible and amenable to ablation.Source: §2 · p. 3

3.1 Setup and notation

• The benchmark spans regression and classification, i.i.d. and non-i.i.d. datasets, varied sample-to-feature ratios, and 768 to 1M+ objects with 5 to 1500+ features.Dataset sizes are grouped using total rows, with further dataset properties listed in Appendix B.Source: §3.1 · p. 3, Table 1 · p. 3
• Native dataset metrics are retained; aggregate comparisons use mean rank, relative improvement over tuned base-space MLP†, normalized score, and Elo at fixed budget B.Classification uses AUROC, accuracy, or log-loss, while regression uses RMSE.Source: §3.1 · p. 3
• The study covers MLP†, TabM†, RealMLP, LightGBM, and TabICLv2, using Claude Code Opus 4.8 and Codex GPT 5.5 for ablations.TabICLv2 is refit on concatenated training and validation splits for test predictions.Source: §3.1 · p. 4

3.2 From end-to-end agents to agent-generated search spaces

• The 50-iteration autoresearch loop on four datasets costs $20–50 per dataset and finishes marginally behind classical HPO, with narrow greedy search and per-dataset cost among its drawbacks.The loop also requires giving the agent access to the data.Source: §3.2 · pp. 4–5
• Agents generate code alternatives for predefined modules once per model family, and standard HPO jointly explores these categorical choices with continuous hyperparameters without seeing target datasets.The neural-family modules include preprocessing, embeddings, architecture, training, and inference; LightGBM and TabICLv2 use preprocessing and inference modules.Source: §3.2 · p. 5
• The method uses one shared space per model rather than dataset-specific spaces or conditional hyperparameters, which may overfit mostly small-to-medium public benchmarks.The authors note that tailoring a space may prove beneficial for a private large application dataset.Source: §3.2 · p. 5

4.1 Main performance

• Across four of five model families, agent-augmented spaces improve aggregate standing over base spaces on all four metrics; RealMLP is the exception.RealMLP’s base space has meta-learned defaults.Source: §4.1 · pp. 5–6, Figure 2 · p. 6
• Agentic RealMLP ensembles are the strongest model-family ensembles, behind only multi-model AutoGluon; their pool members’ predictions are less correlated than the base pool’s.The strongest single member of an agentic pool matches its base counterpart within +0.4% on average, while the ensemble boost is more pronounced.Source: Figure 2 · p. 6, §4.1 · p. 6, Table 5 · p. 8
• AutoKeras and AutoPyTorch lag behind agentic MLP† in both regimes; AutoGluon (+A) selects agentic models on 43 of 45 datasets, assigning them 54% average ensemble weight.Source: §4.1 · pp. 6–7
• Relative improvement on small and medium regression datasets is +3.7% and +1.6% for agentic MLP†, +2.1% and +1.8% for LightGBM, and +1.8% and +1.6% for TabM†.Source: §4.1 · p. 5, Table 4 · p. 7
• On TabArena, agentic spaces improve official Elo scores for MLP†, TabM, RealMLP, and TabICLv2, with agentic TabICLv2 and RealMLP above AutoGluon.The agentic RealMLP overtakes AutoGluon on TabArena but not in Figure 2.Source: §4.1 · p. 7

4.2 Agent cost and amortization

• Agent generation takes roughly 35–100 minutes per model family and is amortized across datasets, making reuse across many datasets the most attractive setting.The paper estimates a single hypothesis-generation loop at approximately 10 USD.Source: §4.2 · pp. 7–8, Table 6 · p. 8

5.1 Which modules matter?

• Numerical embeddings, architecture, training, and numerical preprocessing receive substantial importance, but no single candidate dominates within the four most important modules.Across 18 MLP regression datasets, selected combinations are pairwise distinct and all but two training-side candidates appear at least once, suggesting dataset-specific composition by HPO.Source: §5.1 · p. 8, Figure 3 · p. 9

5.2 Agent choice and stability

• Every regenerated MLP† space improves over the base on all four aggregate metrics, and the released space lies within the re-generation spread.Codex is somewhat more consistent than Claude Code (Elo 1141 ± 22 vs. 1132 ± 61), while Claude produced both the best and worst single runs (Elo 1182 and 1056).Source: §5.2 · p. 8, Table 7 · p. 10

5.3 Candidate-pool size

• The full candidate pool performs best (Elo 1143 vs 1086 for the largest subsets), with all four aggregate metrics improving monotonically as pool size grows.The comparison uses five random subsets at each pool size and full 200-trial HPO on 18 randomly chosen datasets; no downturn appears at the tested sizes.Source: §5.3 · p. 9, Table 8 · p. 10

5.4 Trial budget.

• From 50 trials onward, gaps change little; small-and-medium dataset gains are positive at every tested budget for MLP†, TabM, and LightGBM.On large datasets, gaps are smaller and mixed, except TabICLv2 (+1.8%); agentic RealMLP stays within ±0.4% of its base space on small-and-medium datasets.Source: §5.4 · p. 10

Finite-budget and fit-cost constraints

• A larger search space does not guarantee improvement under a finite HPO budget and may overfit validation; agentic fits had median cost ratios from 0.9× to 2.4× the base.The range runs from RealMLP (0.9×) to TabICLv2 (2.4×).Source: §6 · p. 10

Code review

• Agent-generated code may contain bugs and must be manually inspected before shipping.Source: §6 · p. 10

Evaluation scope

• Results cover five model families and 45 datasets, enough to show the main pattern but not to claim universal gains across tabular regimes.Source: §6 · p. 10

Ensembling scope

• The ensembling study covers four model families and one greedy-ensembling run per dataset and search space; extension to foundation models and HPO-trajectory pools remains future work.Source: §6 · p. 10

Generation and reproducibility

• Avoiding dataset-specific prompting protects against leakage and amortizes cost, but may miss modules useful for particular dataset semantics or feature types.The approach also depends on candidate-set quality and reproducibility, and agent products, prompts, and retrieval corpora should be logged as experimental artifacts.Source: §6 · p. 10

Method and role

• LLM agents generate structured, executable module implementations that expand the HPO search space, which a classical optimizer then explores.Source: §7 · p. 11

Dataset-specific composition

• No single generated candidate is universally best; the candidate basis gives HPO more ways to adapt the model pipeline to each dataset.Source: §7 · p. 11

Ensemble result and implication

• The greedy ensemble over the agentic RealMLP space is the strongest model-family ensemble in the evaluation, behind only multi-model AutoGluon.The conclusion frames agentic search-space generation as a middle ground between fixed HPO grids and fully autonomous research loops.Source: §7 · p. 11

A Additional Results on TabArena

• TabArena tests one default and 200 random configurations per family, each fitted as an 8-fold bagged ensemble; Tuned+Ensembled is greedy weighted ensembling over those configurations.The evaluation uses TabArena-Lite's first outer split; its random search and cross-validated configurations differ from the paper's Optuna and single-fit benchmark setup.Source: A Additional Results on TabArena · p. 17
• TabICLv2-A and RealMLP-A exceed AutoGluon 1.4 best_quality (1535); TabICLv2-A is behind only AutoGluon 1.5 extreme_quality (1669) and newer TFMs.Tuned-only gains are smaller or negative for several families, and the passage attributes most benefit to exploiting pool diversity through ensembling.Source: A Additional Results on TabArena · p. 17
• At Tuned+Ensembled, agentic Elo rises for MLP† (1295 to 1421), TabM (1414 to 1497), RealMLP (1502 to 1557), and TabICLv2 (1573 to 1612), and roughly matches LightGBM (1402 to 1409).Source: Figure 5 · p. 17, A Additional Results on TabArena · p. 17

B.1 Hardware

• Most experiments used one NVIDIA A100 GPU with an AMD EPYC CPU, while absolute runtimes depend on hardware, parallelism, and cluster load.Agent generation times also depend on provider load and network latency, so reported times are indicative rather than reproducible.Source: §B.1 · p. 17

B.2 Datasets

• Datasets use native benchmark metrics, with large datasets defined as TabReD plus Microsoft and the rest split at 3.0 × 10^5 cells.The remaining datasets are divided into small and medium groups at the median number of cells.Source: §B.2 · p. 18, Table 9 · p. 18

B.3 Metrics

• Four complementary metrics capture relative standing and the scale of performance changes across datasets.Mean rank and Elo measure relative standing; relative improvement and normalized score summarize change magnitude.Source: §B.3 · pp. 18–19

B.4 Hyperparameters

• MLP†, TabM†, RealMLP, and LightGBM tune model-specific choices, while TabICLv2 keeps its published checkpoint fixed and uses train-plus-validation context.TabICLv2’s context is subsampled to at most 40,000 samples or 3,840,000/nfeatures to avoid out-of-memory issues.Source: §B.4 · pp. 19–20

B.5 Hyperparameter optimization and ensembling

• Optuna’s univariate TPE uses 200 trials on small and medium datasets and 100 on large datasets, with 20 random startup trials.The default TabICLv2 model is untuned; its agentic variant receives 100 trials on every dataset. Greedy ensembles select among fitted configurations by validation error.Source: §B.5 · pp. 20–21

C Future Research Directions

• Agent-proposed model, training, and numerical-preprocessing modules are selected most often, while default numerical embeddings remain competitive.The paper connects embeddings’ relative strength to their greater prior attention in tabular deep learning and calls for further investigation of the other axes.Source: C Future Research Directions · p. 21
• The authors attribute gains on underexplored axes to a long tail of overlooked methods and agents’ broad literature retrieval, and propose systematic study of those axes.They describe this as a productive direction because agentic generation makes overlooked options cheap to enumerate.Source: C Future Research Directions · p. 22

D.1 AutoML baselines

• AutoML comparisons use unmodified systems on the same 45 datasets and test splits, with four-hour budgets; AutoGluon’s portfolio is bagged, stacked, and greedily ensembled.The agentic AutoGluon hand-off adds eight randomly sampled RealMLP and TabM† configurations to the stock portfolio under the same budget.Source: §D.1 · pp. 22–23

D.2 Hypothesis-generation workflow

• Agents inspect registered pipeline modules, propose and implement candidates, then smoke-test modules and exhaustively test combinations, dropping options with severe runtime or memory costs.Hypotheses require a clear mechanism, literature support, novelty relative to the base model, and benign default behavior; candidates exceeding 10× runtime or memory are dropped.Source: §D.2 · pp. 23–24, Figure 6 · p. 24

D.3 Autoresearch details

• At 50 trials, classical HPO matches or beats autoresearch on three of four datasets, while agentic HPO improves on classical HPO across all four.The reported comparison has no error bars because autoresearch was run once per dataset at a cost of 20–50 USD per run.Source: §D.3 · p. 25, Table 16 · p. 25, Figure 7 · p. 25
• Autoresearch greedily accepts validation-improving end-to-end edits on one task at a time, unlike module-based HPO, which captures module interactions and amortizes generation across datasets.The paper attributes autoresearch’s weaker results to its failure to amortize generation across datasets or capture the interactions exploited by joint HPO.Source: §D.3 · pp. 25–26

E Extended Results

• Each model family’s agent-generated candidate set is reused across datasets without dataset contents or feedback during generation; HPO selects by validation and evaluates on held-out test data.Reported test metrics are means and standard deviations over repeated evaluation seeds.Source: E Extended Results · p. 27

E.1 Ablation study

• Table 22 reports per-run agentic MLP† ablation results under the same split, metric, validation rule, and trial budget as the main experiments.Source: §E.1 · p. 27

E.2 Full per-dataset results

• Table 23 reports terminal-budget mean test metrics and standard deviations for Default HPO and Agentic HPO across 45 datasets; individual AutoPyTorch and AutoKeras rows are single unseeded runs.The terminal budget is 200 trials for small and medium datasets and 100 for large datasets; standard deviations are not reported for the individual-model runs.Source: §E.2 · p. 27, Table 23 · p. 28

E.3 Ensemble results

• Table 24 reports per-dataset ensemble test metrics, strongest-member metrics, and mean pairwise prediction correlations for base and agentic spaces across four model families.Greedy forward selection follows Caruana et al., using 100 random-hyperparameter configurations per family and space, or 20 for large datasets; correlation is undefined when selection yields one distinct member.Source: §E.3 · p. 32, Table 24 · p. 33
• The ensemble subtables also include AutoPyTorch and AutoGluon ensemble rows, with AutoGluon’s default portfolio under Default and its agentic RealMLP and TabM† portfolio under Agentic.Arrows indicate whether higher or lower values are better.Source: §E.3 · pp. 32–33
• The excerpt lists AutoPyTorch 7.4385 and AutoGluon 7.4264, without identifying the metric or dataset for this row.The row also includes unlabeled additional values: 7.5950 for AutoGluon.Source: §E.3 · p. 35
• Another row lists AutoPyTorch 4.3118 and AutoGluon 4.0551, but the excerpt does not identify the metric or dataset.It also gives an unlabeled AutoGluon value of 4.0598.Source: §E.3 · p. 35
• The supplied rows therefore do not establish which system performs better, because they omit the metric direction and the meaning of the additional columns.For example, one row lists AutoPyTorch 8.0876 and AutoGluon 8.1381, alongside another unlabeled AutoGluon value, 8.1415.Source: §E.3 · p. 35

• For each space, the table reports default scores, the best validation-scoring result from 200 random configurations, and a greedy ensemble built from those configurations.Source: F TabArena Per-Dataset Results · p. 37, Table 25 · p. 38
• Scores use AUROC for binary classification, log-loss for multiclass, and RMSE for regression.Source: F TabArena Per-Dataset Results · p. 37
• Base rows use official TabArena results except MLP†, which is the authors’ run under the same protocol as MLP†-A; official TabICLv2 has default scores only.TabICLv2 tuned and ensemble cells are marked missing.Source: F TabArena Per-Dataset Results · p. 37
• Table 25 presents per-dataset TabArena-Lite test scores for 51 datasets, comparing base and agentic search spaces across five model families.Source: F TabArena Per-Dataset Results · p. 37, Table 25 · p. 38