PaperlayerView exact revision on arXiv

Source-linked AI summary

Distillation of Tabular Foundation Models into Efficient Predictors

Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo

arXiv:2610.01435v1cs.LG

TL;DR

TFM predictions require labeled context at inference, making repeated serving costly, and distillation raises questions about supervision design and query coverage. The paper distills TFMs into lightweight students using full-context teacher predictions for observed and synthetic queries, reporting gains over direct training and faster repeated inference.

  • Problem

    TFMs require labeled context at inference, and it remains an open design question how to construct teacher supervision and whether expanding query coverage improves distillation.

    Source: §1 · p. 1

  • Method

    The recipe trains students solely on teacher predictions generated with the full labeled training set as context for observed and synthetic queries.

    Source: §5 · p. 9

  • Results

    Across four teacher–student pairs, distillation gains 56.7–97.9 Elo over matched tuned ensembles and the fixed recipe improves matched default students on TALENT.

    Source: §4.1 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Distilled students serve independently of labeled context, trading one-time distillation cost and some teacher accuracy for faster repeated inference.

    Source: §5 · p. 10

Abstract

from arXiv · show

Tabular foundation models (TFMs) achieve strong predictive performance through in-context learning, yet repeatedly conditioning on labeled data makes inference expensive. Knowledge distillation can reduce this cost by transferring their predictive ability to lightweight, dataset-specific students. However, the dependence of TFM predictions on both a labeled context and a query introduces two design questions: how to construct teacher supervision and whether expanding query coverage improves distillation. We examine these questions across two TFMs and both neural and tree-based students, and derive an effective distillation recipe. The recipe uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries. On TabArena, the resulting students outperform their supervised trained tuned-and-ensembled counterparts by 57-98 Elo points. Applied unchanged to TALENT, the same recipe improves matched default students on 236-258 of 300 datasets and reduces median primary error by 4.0-6.4%. The distilled students also achieve median inference speedups of 3.0-21.6 times over their teachers, offering a practical trade-off between predictive performance and repeated inference cost. Code is available at https://github.com/nums-ai/TFM_Distillation .

1 INTRODUCTION

TFM inference remains context-dependent and costly, motivating distillation into lightweight, dataset-specific students. The introduction identifies supervision construction and query coverage as key design choices and reports a recipe that performs well across benchmarks while reducing repeated-inference cost.

1 INTRODUCTION

  • TFMs process labeled context at inference, making repeated prediction costlier than trained supervised models; distillation offers a complementary route to lightweight students.The students are dataset-specific and support repeated inference without repeatedly conditioning on the teacher context.Source: §1 · p. 1
  • Training solely on teacher predictions using the full labeled training set as context performs best among tested supervision designs and can replace direct label supervision.Source: §1 · p. 2
  • The resulting recipe uses full-training-set context and trains students only on teacher predictions for observed and synthetic queries.Source: §1 · p. 2
  • All four task-agnostic query generators improve aggregate performance for every teacher–student pair, extending useful supervision beyond observed rows.The generators perturb or recombine observed features, adding teacher supervision without collecting new labels.Source: §1 · p. 2
  • 57–98 Elo points over tuned-and-ensembled students on TabArena accompany median inference speedups of 3.0–21.6× over teachers.The recipe also improves student performance across 300 TALENT datasets.Source: §1 · p. 2

2 RELATED WORK

TFMs adapt to a target dataset by conditioning on its labeled examples at inference, while supervised predictors fit separately to each dataset. Prior work reduces context costs or distills teacher information into predictors that can be served independently.

2 RELATED WORK

  • TFMs use in-context learning to predict from labeled examples in the target dataset without dataset-specific parameter updates.This supports strong out-of-the-box prediction, but inference remains dependent on the labeled context.Source: §2 · p. 2
  • Efficiency methods retrieve local contexts, route queries to experts, compress contexts, or cache representations, but retain context-dependent serving computation.Examples include Lo-CalPFN, MixturePFN, BETA, TACO, TabPFN-2.5, and TabICLv2.Source: §2 · p. 3
  • Distillation transfers TFM information into dataset-specific predictors; prior methods variously combine teacher predictions with labels, synthetic inputs, or out-of-fold supervision.FAST-DAD uses Gibbs-sampled synthetic inputs [Fakoor et al., 2020]; Pocket Foundation Models uses out-of-fold predictions [Tanna et al., 2026]; concurrent GEAR work combines synthetic samples and out-of-fold supervision [Qin et al., 2026].Source: §2 · p. 3

3 DESIGNING TFM DISTILLATION

This section tests how teacher targets should be constructed and whether synthetic queries improve distillation, then fixes a recipe using full-context teacher-only supervision and synthetic queries. Across the tested teacher–student pairs, full-context supervision and query expansion improve aggregate Elo, with CUTMIX at a 2:1 ratio selected for subsequent experiments.

3 DESIGNING TFM DISTILLATION

  • The design studies compare supervision construction and query coverage across 51 TabArena datasets, three outer splits, two teachers, and neural TabM and tree-based XGBoost students.The query generators are task-agnostic and perturb or recombine observed features to produce synthetic inputs without collecting new labels.Source: §3 · p. 3, §3.2 · p. 4

3.1 SUPERVISION CONSTRUCTION

  • 22.8–43.0 Elo: full-context teacher-only supervision beats the best tested OOF–label mixture across all four teacher–student pairs.Full-context targets perform best at α = 1; OOF targets peak at α = 0.6 when combined with direct label supervision.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 EXPANDING QUERY COVERAGE

  • 14–91 Elo: synthetic-query augmentation beats real-query-only distillation in every teacher–student pair, with CUTMIX highest in all four.The four tested generators all improve Elo over observed queries alone.Source: Figure 3 · p. 5, §3.2 · p. 5
  • Increasing CUTMIX’s synthetic-to-real query ratio from 1:1 to 2:1 adds 6.5–21.9 Elo across the four pairs while mean generation time rises 9–20%.From 2:1 to 4:1, TabICLv2–TabM and TabICLv2–XGBoost gain 2.7 and 3.8 Elo at 47% and 79% higher generation cost, while TabPFN v3–TabM gains 19.0 Elo at 46% higher cost.Source: Figure 4 · p. 5, §3.2 · p. 5
  • The fixed recipe uses the full labeled training set as context, teacher-only targets, and CUTMIX synthetic queries at a 2:1 ratio.The 2:1 ratio is chosen as a common performance–generation-cost operating point, not tuned per dataset.Source: §3.2 · p. 5

3 DESIGNING TFM DISTILLATION

  • Across 51 TabArena datasets, KD improves each student over its same-architecture tuned-and-ensembled baseline.Table 1 reports median relative primary-error reduction, dataset-level wins, ties, and losses, and Elo.Source: Table 1 · p. 6

4 EVALUATION

The fixed distillation recipe is evaluated on TabArena and TALENT against direct-training baselines, with analyses of teacher-gain transfer and inference economics.

4.1 DISTILLATION IMPROVES THE SAME STUDENT BEYOND DIRECT OPTIMIZATION

  • 56.7–97.9 Elo gains over matched tuned ensembles show that distilled students outperform tuned-and-ensembled direct training across all four teacher–student pairs.They also win on 33–37 of 51 datasets, with median relative-error reductions of 1.30–1.91%.Source: §4.1 · p. 6

4.2 TEACHER ADVANTAGE TRANSFERS THROUGH KD

  • 67.9–70.9% of the teacher–student error gap is recovered when teachers reduce raw-student error by more than 1%; teacher and KD gains are associated across datasets.The Spearman correlations are 0.67–0.82; because both axes share the raw baseline, they are descriptive.Source: §4.2 · pp. 6–7, Figure 5 · p. 7, Table 3 · p. 8

4.3 ONE-TIME DISTILLATION COST VS. INFERENCE EFFICIENCY

  • 3.0–21.6× median inference speedups come with 1.72–2.76% median relative primary-error increases and 48.0–192.4 seconds in median one-time distillation costs.With at least 10K context rows, students are faster on all 15 datasets per pair, with median speedups of 23.9–53.7× versus 1.9–13.1× below 1K rows.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 9

4.4 EVALUATION ON TALENT

  • 236–258 of 300 TALENT datasets improve over supervised students, with median primary-error reductions of 4.02–6.37%; all 12 task–pair combinations have positive median gains.The 95% confidence intervals are above zero, and the gains are not concentrated in one task.Source: §4.4 · p. 9, Table 6 · p. 9

5 CONCLUSION

The fixed distillation recipe uses full-context teacher predictions for observed and synthetic queries, outperforming matched baselines on TabArena and improving matched students on TALENT. Synthetic queries help across all four teacher–student pairs; distilled students trade some teacher accuracy for faster repeated inference.

5 CONCLUSION

  • All four task-agnostic query generators improve aggregate performance for every teacher–student pair, extending useful supervision beyond observed rows.The generators perturb or recombine observed features.Source: §5 · p. 9
  • A fixed recipe using only full-context teacher predictions for observed and synthetic queries outperforms architecture-matched tuned-and-ensembled students on TabArena and improves matched default students on TALENT with default configurations and no target-generation reselection.Source: §5 · p. 9
  • Students trained solely on full-context teacher predictions for observed and synthetic queries achieve the reported benchmark gains.The same recipe is applied unchanged across the reported benchmarks.Source: §5 · p. 9
  • Distilled students serve independently of labeled context, trading some accuracy relative to teachers for faster repeated inference.The largest speedups occur in large-context settings.Source: §5 · p. 10

A EXPERIMENTAL DETAILS

The experimental details specify the TabArena training setup, teacher-target construction, student losses, and synthetic-query operators.

A.1 SECTION 3 DETAILS

  • TabArena uses splits 0–2 on 51 datasets, with eight-fold student bagging and fitting rows as full-context targets.Validation and test labels are excluded from teacher context; OOF targets use up to eight inner folds within the fitting set.Source: §A.1 · p. 12
  • Students retain benchmark-default configurations and preprocessing, use eight teacher estimators, and receive no additional distillation temperature scaling.Classification uses soft crossentropy and regression uses squared error to teacher predictions; TabM and XGBoost balance observed and synthetic queries through group-mean losses.Source: §A.1 · p. 12
  • The operator comparison uses one synthetic query per observed row, with CutMix, Mixup, Perturb, and FullMix generating queries from fitting-set rows.CutMix replaces features with probability 0.5; Mixup uses independent feature-wise Unif(0, 1) coefficients; Perturb adds scaled numerical noise and categorical replacements; FullMix samples features independently. Synthetic targets come only from the teacher.Source: §A.1 · p. 12

A.2 SECTION 4 DETAILS

The appendix specifies benchmark partitions and aggregation procedures, followed by timing and cost measurements for serving and distillation.

A.2 SECTION 4 DETAILS

  • TabArena evaluation covers 816 dataset–split pairs, while TALENT uses fixed partitions and seeds 0–4 without outer bagging or seed ensembling.TabArena includes nine splits for 34 datasets and 30 for 17; TALENT teacher context uses training data only.Source: §A.2 · p. 12
  • Errors are averaged over matched splits or seeds before median dataset-level relative gains are computed, with each dataset equally weighted.Paired bootstrap intervals resample datasets; TALENT gains exclude reference errors at most 10^-12, while win/tie/loss counts use all 300 datasets.Source: §A.2 · p. 12

A.3 TIMING DETAILS

  • Serving latency is measured on all 51 TabArena datasets with resident models, 32-row requests, and teacher context recomputation without persistent KV caching.Measurements use one H200 host, exclude model loading, and report the median of GPU-synchronized calls after warm-up.Source: §A.3 · p. 12
  • Offline cost includes teacher-target generation, student fitting, and training overhead; speedups and break-even requests are computed per dataset before medians.Figure 4 generation time is the mean cold target-generation cost across completed splits, excluding student fitting.Source: §A.3 · p. 12

B FULL TABARENA RESULTS

Table 7 presents the full TabArena comparison across official baselines and default-configuration KD students.

B FULL TABARENA RESULTS

C SECTION 4 RESULTS EXCLUDING DESIGN SPLITS

The TabArena analyses are repeated after excluding the splits used for design studies, with the distillation recipe and design choices held fixed. The remaining analysis tests sensitivity to split reuse, not generalization to unseen datasets.

C SECTION 4 RESULTS EXCLUDING DESIGN SPLITS

  • The analysis retains 663 dataset–split pairs across the same 51 datasets: splits 3–8 for 34 datasets and splits 3–29 for 17.Outer splits 0–2 were used for the Section 3 design studies.Source: C SECTION 4 RESULTS EXCLUDING DESIGN… · p. 15
  • The fixed full-context, teacher-only CutMix recipe uses default student configurations and r = 2, without reselecting any design choice.Source: C SECTION 4 RESULTS EXCLUDING DESIGN… · p. 15
  • Elo, improvability, and win rate are recomputed over the same 72 official baseline and four KD configurations with equal dataset weights.Relative-error analyses average retained-split errors within each dataset before computing dataset-level ratios and medians; intervals use 20,000 paired dataset-bootstrap resamples.Source: C SECTION 4 RESULTS EXCLUDING DESIGN… · p. 15

C.1 COMPARISON WITH SUPERVISED STUDENTS

After excluding the design splits, all four KD configurations still outperform the matched tuned-and-ensembled student. KD configurations also exceed the highest Elo of at least 14 of 15 conventional supervised families.

C.1 COMPARISON WITH SUPERVISED STUDENTS

  • 56.4–101.6 Elo gains and 1.37–1.84% median relative-error reductions distinguish all four KD configurations from the matched tuned-and-ensembled student.Every paired interval is above zero, and KD wins on 33–36 of the 51 datasets.Source: Table 8 · p. 15, §C.1 · p. 15
  • Each KD configuration exceeds the highest Elo of at least 14 of 15 conventional supervised families; TabPFN v3–TabM exceeds all 15.RealMLP is the only family whose strongest configuration ranks above the other three KD models.Source: §C.1 · p. 15, Table 9 · p. 16
  • The broader comparison uses 76 configurations across 663 retained dataset–split pairs, with KD students at default settings and r = 2.Displayed models match Table 2; shading marks KD students.Source: Table 9 · p. 16

C.2 TEACHER ADVANTAGE AND KD GAIN

The relationship between teacher advantage and KD gain persists after excluding design splits, and KD recovers much of the teacher–student error gap on eligible datasets. These shared-baseline correlations are descriptive, and stronger teachers do not ensure KD wins on every dataset.

C.2 TEACHER ADVANTAGE AND KD GAIN

  • 0.69–0.82 Spearman correlations link teacher advantage and KD gain across all 51 datasets, with bootstrap intervals above zero.The correlations are descriptive because teacher and KD gains share the raw-student denominator.Source: Figure 6 · p. 16, §C.2 · p. 16
  • KD still loses to the raw student on one to three eligible datasets per pair, so a stronger teacher does not ensure improvement on every dataset.Source: §C.2 · p. 17
  • 67.8–73.4% median gap recovery is achieved on 42–48 eligible datasets per pair where the teacher cuts raw-student error by more than 1%.Recovery intervals resample eligible datasets selected using only the retained splits.Source: §C.2 · pp. 16–17, Table 10 · p. 17

Related papers

Loading 2610.01435v1…