Source-linked AI summary

Tabular Foundation Models Are Effectively Shallow

Irene Cannistraci, Julia E. Vogt

arXiv:2610.13253v1cs.LGcs.AI

TL;DR

In-context-learning TFMs avoid additional training but their stacked transformer blocks impose high memory and inference costs, raising the question of whether depth is redundant. The paper uses a closed-form, label-free affine translator to replace contiguous block windows and evaluates compression across models and datasets. Replacing half the blocks costs at most 0.002 median AUROC, and a controlled nanoTabPFN study finds compressed models match shallow models trained from scratch within 0.011 AUROC.

  • Problem

    In-context-learning TFMs solve new tasks without additional training, but their deep transformer stacks impose high memory and inference costs that constrain deployment on resource-limited hardware.

    Source: §1 · p. 2
  • Method

    A closed-form, label-free affine translator replaces a contiguous window of transformer blocks, with the remaining network left unchanged.

    Source: §1 · p. 2, §3 · p. 3
  • Results

    Across 8 TFMs and 171 classification datasets, replacing half the blocks costs at most 0.002 median AUROC; in nanoTabPFN, compressed models match shallow models trained from scratch within 0.011 AUROC.

    Source: §1 · p. 2
  • Takeaways & Limitations

    The results show that substantial depth can be removed from evaluated TFMs while largely preserving performance, and compression can match shallow training in the controlled nanoTabPFN setting.

    Source: §1 · p. 2

    Single-block compression can fail for continuous target rankings in regression and for LimiX-16M on very wide datasets, while shallow-training parity has only been studied in a 1.4M-parameter nanoTabPFN model.

    Source: §8 · p. 9

Abstract

from arXiv · show

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance? To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most $0.002$ median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to $6\times$ at a median cost of $0.03$ AUROC, while a single-block TabPFN-3 serves cached queries $2.4\times$ faster using $6\times$ less memory at negligible cost.

1 INTRODUCTION

In-context-learning TFMs offer zero-shot convenience but incur high memory and inference costs, motivating the question of whether architectural redundancy can be removed. A label-free affine translator enables block replacement, and broad evaluations find that substantial compression largely preserves performance.

1 INTRODUCTION

  • Keeping one TabPFN-3 block, or replacing 96% of blocks, costs −0.009 AUROC on MIMIC-III while yielding 2.4× cached-query GPU speed-up and 6.1× less memory.Figure 1 plots AUROC change, cached-query GPU speed-up, and serving-memory reduction against compression.Source: Figure 1 · p. 1
  • A closed-form, label-free affine translator bypasses a contiguous block window, with output stability indicating whether a simple linear map can absorb the skipped computation.The window is selected by minimizing forward KL divergence on a small calibration split.Source: §1 · p. 2
  • Across 8 TFMs and 171 classification datasets, replacing half the blocks costs at most 0.002 median AUROC, while up to 97% of datasets stay within one point of full-model performance.Cached-context per-query GPU latency falls 1.5×–1.9× at half depth and 2.4×–5.1× with one block kept.Source: §1 · p. 2
  • In a controlled nanoTabPFN study, a 12-layer model compressed to k layers matches a model trained from scratch at depth k within 0.011 AUROC, while a natively trained 6-layer model matches the full 12-layer baseline.Whether this parity scales to significantly deeper architectures remains open.Source: §1 · p. 2

2 RELATED WORK

Related work spans task-specific tabular transformers and newer general-purpose in-context-learning foundation models. Prior transformer-redundancy methods and concurrent tabular approaches differ from this paper in their use of additional training or labeled data.

2 RELATED WORK

  • Tabular transformers progressed from task-specific FT-Transformer, SAINT, and TabNet to general-purpose in-context-learning models including TabPFN, TabICL, TabDPT, TabFM, Mitra, Orion, and LimiX.The cited work describes differing pre-training data and attention patterns across these model families.Source: §2 · pp. 2–3
  • Vision and language research documents redundancy in transformer layers, heads, and tokens, while training-free linear maps have been used to bridge skipped block representations [Gromov et al., 2025; Men et al., 2025; Bolya et al., 2023; Venkataramanan et al., 2024; Shopkhoev et al., 2025].The cited vision work also replaces deep transformers with recurrent blocks [Jacobs et al., 2026].Source: §2 · p. 3
  • Concurrent tabular compression approaches use additional pre-training or dataset-specific gradient updates, whereas this paper uses one closed-form affine map without nonlinear adapters, gradient updates, or labeled data.The passage contrasts pre-trained looping and context compression, task-trained MLP adapters and students, and pre-trained early-exit decoders.Source: §2 · p. 3

3 METHOD

The method replaces a contiguous transformer-block window with an affine map fitted to hidden states on unlabeled calibration data. It selects the bypass window using output-distribution divergence, while retaining the remaining blocks and original classification head.

3 METHOD

  • A contiguous window of blocks s through e−1 is bypassed by an affine map fitted via ridge regression to latent representations at depths s and e.H0 is the input-embedding output, and HL is the final transformer-block output.Source: §3 · p. 3
  • Hidden states are extracted in one forward pass, and the map is fitted on at most 500 sequence positions, with regularization selected using a held-out validation split.The map is refit on the full selected subset after choosing λ.Source: §3 · p. 3
  • At inference, the map projects Hs to He while remaining blocks and the original classification head operate unchanged; dual-axis models use a separate affine map per feature.Maps at network boundaries can be absorbed into adjacent affine layers, reducing the compressed model to a standard k-block architecture without extra parameters.Source: §3 · pp. 3–4
  • For a target window width, the method selects bounds that minimize forward KL divergence between the full and compressed models’ output distributions.The KL criterion compares candidates in the same probability space and accounts for how remaining blocks and the classification head affect predictions.Source: §3 · p. 4
  • Window selection is a one-time pre-computation during context encoding; the frozen map is reused for subsequent queries, and a divergence tolerance can determine the widest acceptable window.At ϵ = 0.01, the rule removes 52–70% of single-axis model blocks at a mean cost of at most 0.004 AUROC.Source: §3 · p. 4

4 LOCATING COMPUTATIONAL REDUNDANCY

Representation similarity and attention profiles do not reliably identify replaceable blocks, so the analysis intervenes directly by replacing individual blocks. These tests find that no single block is critical in single-axis models, whereas dual-axis models each have one critical block.

4 LOCATING COMPUTATIONAL REDUNDANCY

  • The study evaluates 8 pretrained TFMs from 6 model families across 171 classification datasets, including 163 benchmark-suite datasets and eight clinical datasets.Each dataset uses train, calibration, and test splits and is run at three seeds.Source: §4 · p. 4
  • Pairwise block-representation cosine similarity varies by architecture and does not reliably predict replaceability; TabICL’s most dissimilar final block is nevertheless fully replaceable.Figure 2 shows layerwise pairwise cosine similarity across TFMs on MIMIC-III.Source: §4 · p. 4, Figure 2 · p. 5
  • Across MIMIC-III and Higgs, tabular TFMs have 81–95% of maximum attention entropy on average, while Qwen3-4B attention is sharply peaked; consecutive blocks’ five most-attended keys overlap by Jaccard index 0.10–0.19.Figure 3 compares normalized per-block attention entropy for representative tabular TFMs and Qwen3-4B.Source: §4 · p. 5, Figure 3 · p. 5
  • Single-block replacement stays within 0.007 AUROC of full performance for single-axis models, while dual-axis TabPFNv2 and LimiX-16M each have one critical block: block 0 and block 3, respectively.Every other block stays within 0.043 AUROC for TabPFNv2 and 0.009 for LimiX-16M.Source: §4 · p. 5

5 EVALUATION OF COMPRESSED MODELS

The evaluation tests contiguous block replacement from half-depth to one-block compression, using a label-free translator. Half-depth is nearly lossless in classification and regression, while one-block outcomes vary by architecture and task.

5.1 CLASSIFICATION

  • Half-depth compression costs at most 0.002 median AUROC across architectures, with mean drops of 0.001 to 0.005 across 163 classification datasets.Model rankings against one another and tuned GBDTs remain unchanged.Source: §5.1 · p. 6, Figure 5 · p. 6
  • At one block, TabPFN-3 and Orion-Bix remain nearly lossless, while TabPFNv2 loses 0.101 mean AUROC and LimiX-16M loses 0.052.TabICL, Orion-MSP, TabFM, and TabDPT have negligible mean losses of 0.015 to 0.026, with some outliers.Source: §5.1 · p. 6, Figure 5 · p. 6
  • For single-axis models, the first block is preferred, whereas dual-axis models need a critical block that a single contiguous window may not retain.TabPFNv2 keeping block 0 costs 0.281 AUROC, versus 0.071 when it keeps the last block; LimiX-16M’s critical block 3 is neither endpoint.Source: §5.1 · pp. 6–7
  • Replacing skipped blocks with an identity mapping collapses performance to near chance for most models, while the translator recovers their computation.A supervised early-exit classifier loses about 0.02 AUROC at every depth, and the translator matches or outperforms it in 81% of 2,664 configurations.Source: §5.1 · p. 7

5.2 REGRESSION

  • Single-block regression usually degrades sharply despite half-depth median losses of only 0.001 to 0.026 R2 across 41 datasets.Predictions shrink toward the mean and output rankings worsen; TabFM is an exception, losing 0.022 R2 at one block.Source: §5.2 · p. 7, Figure 6 · p. 7

6 EFFICIENCY

Compression removes much of the transformer stack’s parameter and inference cost, although encoder and translator overhead constrain some gains.

6 EFFICIENCY

  • At one block, TabFM sheds roughly 1.5B parameters, while single-axis models eliminate 91% to 96% of stack weights.On wide datasets, the dual-axis translator can moderate savings; LimiX-16M’s map adds 37k parameters per feature token when it cannot be absorbed.Source: §6 · p. 8, Table 1 · p. 8
  • Lightweight-encoder models accelerate 3.3× to 6.1× at one block on GPU, while heavy encoders limit gains to 1.1× to 1.4×.Context caching yields query speed-ups up to 5.1×; on CPUs, end-to-end speed-ups reach 12.4×.Source: §6 · p. 8, Table 1 · p. 8

7 COMPRESSION VS. PRE-TRAINING SHALLOW ARCHITECTURES

A controlled nanoTabPFN study compares compressing a pretrained model with training shallow models from scratch. Compression matches native shallow training closely, but the map must be fitted on the target rows.

7 COMPRESSION VS. PRE-TRAINING SHALLOW ARCHITECTURES

  • Compressed nanoTabPFN matches native training within 0.011 AUROC at every tested depth, while a natively trained 6-block model matches the 12-block model.The comparison covers four clinical datasets and a 1.4M-parameter model.Source: §7 · p. 9, Figure 7 · p. 9
  • Fitting the translator on 20 unseen prior tasks instead of target rows drops AUROC to 0.43–0.83.The compressed model’s parity with native shallow training therefore depends on fitting the map to the target’s unlabelled rows.Source: §7 · p. 9, Figure 7 · p. 9
  • Adding a trained affine layer to a shallow topology does not reliably improve on the plain shallow model, and deeper models collapse at the default learning rate.At k = 2, results are 0.888 versus 0.902; at k = 3, they are 0.900 versus 0.894.Source: §7 · p. 9

8 CONCLUSION

The conclusion argues that most tabular-transformer depth is replaceable with a label-free affine map, while identifying settings where extreme compression or shallow pre-training remains uncertain.

8 CONCLUSION

  • Replacing half the blocks costs at most 0.002 median AUROC, and keeping one block removes up to 96% of parameters while enabling GPU speed-ups up to 6×.A single-block TabPFN-3 serves cached queries 2.4× faster with 6× less memory at negligible cost.Source: §8 · p. 9
  • Single-block compression can fail to preserve continuous-target rankings, and LimiX-16M’s unabsorbable map can outweigh removed blocks on very wide datasets.Whether compression matches shallow pre-training at production scale remains open; the comparison is limited to a 1.4M-parameter nanoTabPFN.Source: §8 · p. 9

A.1 DATASETS

  • The dataset protocol uses disjoint train, calibration, and test splits, with patient-disjoint clinical cohorts; not every model runs on every suite dataset.All models run on eight clinical datasets, and comparisons use each model’s own full version on the same datasets.Source: §A.1 · p. 13, Table 2 · p. 13

A.2.1 GBDTS HYPERPARAMETERS

  • XGBoost and LightGBM are tuned per dataset and seed by a 30-configuration random search, selected using calibration AUROC.Default-hyperparameter variants are also reported in the suite rank tables.Source: §A.2.1 · p. 13

A.2.2 TABULAR FOUNDATION MODELS

  • The translator replaces only transformer blocks, leaving encoders and heads unchanged; withholding calibration rows changes median baseline AUROC by 0.000 for every model.Prediction-time ensembling is reduced to one member where possible, with TabICL retaining its default eight-member ensemble in a single pass.Source: Table 3 · p. 14, §A.2.2 · p. 14

A.3 TRANSLATOR IMPLEMENTATION AND ABLATIONS

The section examines how translator design and preprocessing affect compression, particularly in dual-axis models. Per-feature maps preserve accuracy better than shared maps in the reported TabPFNv2 comparison, while LimiX results depend on freezing preprocessing.

A.3 TRANSLATOR IMPLEMENTATION AND ABLATIONS

  • For dual-axis models, the translator applies a separate affine map to each feature token; with one feature token, this reduces to the shared-map case.The feature-token count is set by model preprocessing, not by the original column count.Source: §A.3 · p. 14
  • On the 153 suite datasets, frozen-preprocessing LimiX loses 0.005 AUROC on average at Half and 0.052 at One-block, versus TabPFNv2’s 0.003 and 0.098.The LimiX losses are median 0.002 and 0.029, respectively; the full model’s predictions are identical on seven of eight clinical datasets and −0.006 on Heart under frozen preprocessing.Source: §A.3 · pp. 14–15
  • At width 4, the per-feature map is within 0.02 AUROC of Full on every dataset, while the shared map loses up to 0.15; with one block kept, their median losses are 0.082 and 0.301.On wide datasets, per-feature maps can exceed the removed blocks in parameter count; savings occur when the window starts at block 0, where the map folds into the encoder.Source: §A.3 · pp. 15–16, Table 4 · p. 16
  • At width 4, 500 label-free refinement steps move ΔAUROC by at most 0.003; with one block kept, the median change is 0.000.The residual performance gap therefore remains essentially unchanged under gradient refinement of the translator.Source: §A.3 · p. 16

B.1 ATTENTION ANALYSIS

The appendix examines attention patterns, block sensitivity, label-free window and depth selection, classification results, and comparisons with TabArena and decision metrics. It reports that substantial compression can preserve model rankings and often maintain AUROC and decision performance, while extreme one-block compression affects decision metrics more.

B.1 ATTENTION ANALYSIS

  • Across datasets, surviving-block attention entropy changes by less than 0.03 nats on average and at most 0.11 on any block; attention neighborhoods nevertheless change at every block.Tabular models stay near-uniform across depth, unlike Qwen3-4B’s varying entropy profile; dual-axis models are excluded from this attention analysis because they interleave item and feature attention.Source: §B.1 · pp. 16–18

B.2 PER-BLOCK SENSITIVITY, ALL MODELS

  • Figure 11 reports each model’s per-block ΔAUROC when a single block is replaced, averaged over eight clinical datasets and three seeds.The figure presents per-block sensitivity; the supplied text does not state which blocks are most sensitive.Source: §B.2 · p. 18, Figure 11 · p. 19

B.6 PER-DATASET RESULTS AND THE IDENTITY CONTROL

  • Table 10’s per-dataset clinical results include One-block identity rows, which replace the translator with the identity to isolate its contribution.Table 7 compares Half-depth KL-selected windows with two fixed windows and a test-set oracle; TabPFNv2 uses a per-feature map and needs selection.Source: §B.6 · p. 19, Table 7 · p. 20

B.4 WINDOW SELECTION

  • With KL-selected windows and tolerance 0.01, the depth rule removes 52–70% of single-axis model blocks across 171 datasets at a mean cost of 0.001–0.004 AUROC.It removes 42% for TabPFNv2 and 22% for LimiX-16M; 85–97% of datasets remain within one point.Source: §B.4 · p. 19, Table 8 · p. 20

B.7 SUITE-SCALE RANK TABLES

  • The Half TabFM ranks above every other full model, with an average rank of 3.53, and Half TabFM, TabICL, and TabPFN-3 rank above tuned LightGBM and XGBoost.Half-depth models retain the ordering of their full-depth counterparts.Source: §B.7 · p. 20

B.5 CLASSIFICATION RESULTS

  • Figure 12 plots ΔAUROC against the share of blocks replaced on MIMIC-III for every model and swept width.The figure uses three seeds, KL-selected windows, and ridge translators; the supplied text does not state the plotted outcomes.Source: §B.5 · p. 19, Figure 12 · p. 21

B.8 CHECK AGAINST TABARENA

  • Full-model AUROC tracks the TabArena leaderboard per dataset (Pearson r 0.93–0.97), while results sit 0.4–1.2 points lower on datasets with at most 3,000 rows and 0.7–2.9 points lower on larger datasets.The paper attributes the larger-dataset gap partly to its support cap; its comparisons of each model with its own Full baseline use an identical protocol.Source: §B.8 · p. 21

B.9 DECISION METRICS BEYOND AUROC

  • At Half, decision metrics stay within about 0.01 of Full on binary tasks and within 0.02 on multi-class tasks; with one block kept, they drop more than AUROC, mostly on multi-class tasks.TabPFN-3 loses at most 0.03 balanced accuracy; the largest drops occur in the two dual-axis models.Source: §B.9 · p. 21

B.10 SUBGROUP ANALYSIS ON THE CLINICAL COHORTS

At Half depth, compression has small and broadly uniform effects across clinical subgroups, whereas One-block costs fall more heavily on patients with the rarer outcome. The affine translator also matches or beats an early-exit probe in most tested configurations, consistent with later blocks moving representations into coordinates the pre-trained head reads.

B.10 SUBGROUP ANALYSIS ON THE CLINICAL COHORTS

  • At Half depth, every completeness quartile stays within 0.007 AUROC of the overall cost, and sensitivity or specificity shifts are at most 0.07 for single-axis models and 0.05 for TabPFNv2.At One-block, the rarer-outcome patients experience larger probability shifts; the authors identify Half as the clinically deployable point.Source: §B.10 · pp. 21–22
  • At One-block, sensitivity falls as specificity rises for several models, while TabPFNv2’s large MIMIC-III threshold shifts reflect calibration change rather than ranking change.The outcome is the minority class, representing 11–14% of rows.Source: §B.10 · pp. 21–22

B.11 ABLATIONS: EARLY EXIT VS. TRANSLATOR

  • 81% of 2,664 configurations show the label-free translator matching or beating the supervised early-exit probe, whose median cost remains near −0.02 across depths.The translator’s median cost shrinks from −0.003 early to +0.000 late; later blocks move representations into coordinates read by the pre-trained head.Source: §B.11 · pp. 22–23

B.12 ONE-BLOCK ABLATIONS

One-block compression depends on which block is retained: single-axis models usually favor the first, while dual-axis models favor the last. Repeating the retained block does not replace the affine translator, and one-block costs tend to grow with dataset size.

B.12 ONE-BLOCK ABLATIONS

  • 0.281 AUROC is TabPFNv2’s cost when keeping the first block, versus 0.071 when keeping the last; for LimiX-16M the corresponding costs are 0.230 and 0.034.Single-axis models mostly keep the first block, while dual-axis models mostly keep the last; TabPFN-3 costs the same either way because its encoders do most of the work.Source: §B.12 · p. 23
  • TabPFN-3 has the same One-block cost whichever single block is kept, since its encoders do most of the work.Source: §B.12 · p. 23

B.13 CONTEXT SIZE

  • Half-depth TabFM outranks every other full model in average rank, and dataset size is negatively associated with One-block cost (Spearman ρ = −0.22).The size association covers 163 datasets and eight models; model-wise correlations range from −0.09 to −0.34.Source: Table 11 · p. 24, §B.13 · p. 24

B.12 ONE-BLOCK ABLATIONS

  • Without the translator, repeating the retained block cannot recover the model, while with it repetition brings no gain for TabICL or TabDPT and slightly increases TabFM’s cost.TabDPT stays near −0.4 without the map; with it, costs stay within 0.003 for TabICL and 0.006 for TabDPT, and rise from 0.006 to 0.016 for TabFM.Source: §B.12 · pp. 23–24

B.14 ZERO-SHOT MAP TRANSFER ACROSS DATASETS

Across the extensions, affine maps transfer most reliably between datasets with shared schemas, while regression, hardware, and shallow-training experiments reveal distinct limits and efficiencies of compression. Context extrapolation shows that compressed nanoTabPFN models with at least two blocks remain close to the full model on larger clinical support sets.

B.14 ZERO-SHOT MAP TRANSFER ACROSS DATASETS

  • At Half depth, a TabICL map transferred between MIMIC-III and eICU incurs a 0.004 AUROC penalty versus 0.003 for a local fit, but transfer weakens across different schemas.Across disparate-schema pairs, transferred maps recover 60–65% of local-map benefit for TabICL and TabPFN-3, about 30% for TabDPT, and none for TabFM; a new schema needs its own fast, label-free fit.Source: §B.14 · pp. 24–26

B.15 REGRESSION

  • With one block kept in regression, most models lose prediction ordering more than scale, while LimiX-16M’s affine recalibration recovers three quarters of its median R2 loss.For LimiX-16M, median ∆R2 changes from −0.556 to −0.139 after recalibration, while rank correlation falls from 0.92 to 0.78.Source: §B.15 · pp. 26–27, Table 21 · p. 30

B.16 EXTENDED HARDWARE MEASUREMENTS

  • Absorbing the translator into an adjacent encoder or head is exact on the eight clinical datasets, with predictions differing by at most 4 × 10^-6.Absorption is possible only at compatible linear-layer ends; selected One-block windows reach such an end in 86% of TabPFNv2 and 97% of TabDPT dataset–seed pairs.Source: §B.16 · p. 27, Table 22 · p. 30
  • On a desktop CPU, One-block end-to-end speed-up reaches 9.5–12.4× for lightweight-encoder models, compared with 1.2–1.8× for heavy-encoder models.The transformer stack takes most of the forward pass for lightweight-encoder models in this re-encode regime.Source: §B.16 · p. 27, Table 23 · p. 31

B.17 NANOTABPFN TRAINING SETUP

  • At Half depth, the compression cost stays within 0.009 across support sizes from 500 to 18,000 rows, while TabFM’s One-block cost grows from −0.005 to −0.043.The Half-depth worst dataset is TabPFNv2 on Covertype at −0.030.Source: Figure 13 · p. 28
  • On four clinical datasets, compressed models match or slightly exceed shallow models retrained at 10−3: AUROC 0.854 vs. 0.849 (k=1), 0.891 vs. 0.883 (k=2), and 0.903 vs. 0.901 (k=3).Source: §B.17 · p. 28, Table 25 · p. 31

B.18 CONTEXT EXTRAPOLATION IN NANOTABPFN

  • With 1,000 support rows, nanoTabPFN compressed to at least two blocks stays within 0.008 AUROC of the full model on both clinical cohorts, while one block loses performance.Shallow models trained from scratch do at least as well as the full model at k ≥2; at k = 2 they lead compressed models by 0.016 on eICU and 0.027 on MIMIC-III.Source: §B.18 · pp. 29–30

B.17 NANOTABPFN TRAINING SETUP

  • At the default learning rate, nanoTabPFN depths 6 and 12 collapse to uniform predictions and require an automatic restart at one-quarter the rate.The restart uses training loss alone; depths 1–3 train at the default rate.Source: §B.17 · p. 28, Figure 15 · p. 31
Loading 2610.13253v1…