Source-linked AI summary
Recurrent In-Context Learning for Tabular Foundation Models
Tejas Vaidhya, Gaurav Sahu, Irina Rish
TL;DR
Tabular foundation models adapt to new datasets through in-context learning, but their layer-specific parameters tie depth to model size. Tab-Loop replaces the ICL stage’s untied blocks with tied recurrent blocks and, at matched compute, matches the baseline with fewer parameters; its depth can also be chosen at inference.
Problem
The paper asks whether a tabular foundation model’s ICL stage needs distinct parameters for every layer or only depth, given that inference costs recur with model size and context length.
Source: §1 · pp. 1–2Method
Results
At equal compute, Tab-Loop matches the baseline with 62% fewer parameters, reaching 84.10% vs. 83.92% mean accuracy on 200 TALENT datasets and 1409 vs. 1396 Elo on TabArena.
Source: §1 · p. 2Takeaways & Limitations
The ICL stage needs depth more than distinct parameters, and tying makes depth a choice at inference that can be extended through training with deeper loops.
Source: §7 · p. 9Deeper-loop training is not compute-matched to the baseline, and the resulting test-time scaling is not parameter efficiency at equal inference cost.
Source: §5.3 · p. 8
Abstract
from arXiv · showhide
Tabular foundation models adapt to new datasets through in-context learning (ICL), and, like most transformers, they tie computational depth to the number of independently parameterized layers. We ask whether the ICL stage of such a model needs a distinct set of parameters for every layer, or only depth. We introduce Tab-Loop, which replaces the 12 untied ICL blocks of TabICLv2 with 4 tied blocks applied $k$ times and changes nothing else: the encoders, decoder, synthetic prior and training recipe are shared with an untied baseline trained for the same compute. At matched pretraining compute (18.2 EFLOP) and matched inference compute (12 block applications), Tab-Loop matches the baseline with 62\% fewer parameters (10.4M vs.\ 27.6M): held-out log loss is 0.4583 vs.\ 0.4588, mean accuracy on 200 TALENT datasets is 84.10\% vs.\ 83.92\%, and Elo on 38 TabArena datasets is 1409 vs.\ 1396. A 4-block untied control with Tab-Loop's parameter count, which sees 32\% more training tables for the same compute, trails both (83.46\%; Elo 1343). Because weights are shared, depth becomes a choice made at inference. Accuracy rises with the number of loops through the trained range and holds beyond it, and continued training with deeper loops extends the useful range, reaching the baseline's stage-2 scores at 4--5$\times$ its inference compute. At this scale, the ICL stage of a tabular foundation model needs depth more than it needs distinct parameters.
1 INTRODUCTION
Tabular foundation models adapt to new datasets at inference, making model size and context length part of the per-prediction cost. This paper tests whether the ICL stage needs distinct parameters for every layer or depth alone, introducing Tab-Loop and comparing it with an untied baseline under matched compute.
1 INTRODUCTION
- At 12 block applications, Tab-Loop matches the 12-block baseline with 62% fewer parameters: 84.10% vs. 83.92% TALENT mean accuracy and 1409 vs. 1396 TabArena Elo.The comparison uses 18.2 EFLOP training compute; the figure covers 200 TALENT datasets and 38 TabArena datasets.Source: Figure 1 · p. 2, §1 · p. 2
- The ICL stage poses a controlled-comparison question: does it need a distinct parameter set for every layer, or only depth?The looped and untied models must share everything except the ICL stage and be compared at equal training and inference compute.Source: §1 · p. 2
- Tab-Loop replaces TabICLv2’s 12 untied ICL blocks with four tied blocks while leaving the encoders, decoder, prior, and training recipe unchanged.Its controlled comparison also includes a parameter-matched four-block untied control that spends the same compute on 32% more training tables.Source: §1 · p. 2
- At the scale tested, the ICL stage relies more on depth than on distinct parameters, while the parameter-matched control with additional data trails both models.At a single loop, Tab-Loop is weaker than the control, indicating its computation is distributed across loops.Source: §1 · p. 2
- Without retraining, TabArena Elo rises from 1288 at one loop to 1424 at four, while deeper-loop training extends the useful range and reaches stage-2 baseline scores at 4–5× inference compute.TALENT accuracy peaks at 84.18% at four loops without retraining; after deeper-loop training it reaches 84.42% at sixteen loops.Source: §1 · p. 2
2 WHY RECURRENT DEPTH FOR TABULAR IN-CONTEXT LEARNING?
Tabular foundation models pay inference costs at every prediction, while their standard transformer design ties depth to parameter count. The section motivates recurrent depth as a way to retain computation while sharing parameters, and to make depth selectable at inference.
2 WHY RECURRENT DEPTH FOR TABULAR IN-CONTEXT LEARNING?
- Because adaptation happens at inference, model-size and context-length costs are paid at every prediction; recent work suggests depth need not require distinct parameters at every layer.Tabular models refine representations iteratively, and much of their depth is redundant.Source: §2 · pp. 2–3
- Tabular in-context learning iteratively refines predictions from labelled context rows, whereas parameters store knowledge and the model answers from the rows in front of it.The section compares this refinement to gradient boosting and describes successive transformer layers as steps of an iterative learner.Source: §2 · p. 3
- Removing layers reduces parameters and depth together, so depth redundancy alone cannot identify which resource the model needs; weight sharing separates them.The section notes that tabular foundation models refine representations progressively and much of their depth can be removed.Source: §2 · p. 3
- A tied model stores its blocks once, but computation still grows with depth because each shared block costs as much as an untied one.The section says Tab-Loop applies 12 blocks with 62% fewer parameters than the untied baseline.Source: §2 · p. 3
- A recurrent model with one decoder trained across depths can be read after any number of loops, allowing computation to follow dataset difficulty rather than model size.The section contrasts this with an untied stack, whose decoder reads only the final layer.Source: §2 · p. 3
- Recurrence can require one block to perform stages otherwise spread across specialized layers, risk state drift, or lose input information; Tab-Loop uses four-block loops, input reinjection, and confines recurrence to the ICL stage.The passage attributes these failure modes to prior work and describes the design choices as responses to them.Source: §2 · p. 3
- The compared models share one-pass column and row encoders; Tab-Loop reuses four tied ICL blocks, while the baseline applies 12 distinct blocks once.Tab-Loop re-adds the stage input before every block application after the first; the parameter-matched control applies four untied blocks once.Source: Figure 2 · p. 4
- The section predicts that a tied model will match an untied model at equal compute, outperform a parameter-matched model spending compute on more data, and improve with additional loops.It frames a computation-heavy, parameter-light model as a prior toward iterating on context evidence rather than recalling stored weights.Source: §2 · p. 4
3 TAB-LOOP
Tab-Loop replaces TabICLv2’s 12 untied ICL blocks with 4 blocks applied repeatedly, while retaining the other model components. Its loop count can be selected at inference, and the stage-1 comparison shows comparable loss to the baseline at equal compute.
3 TAB-LOOP
- Tab-Loop replaces TabICLv2’s 12 distinct ICL blocks with 4 shared blocks applied k times, leaving the encoders, decoder, prior, and curriculum unchanged.The shared blocks are applied in a fixed order, and encoded input X0 is reused throughout the recurrent computation.Source: §3 · p. 4, §3.1 · p. 4
3.1 LOOPED ARCHITECTURE
- Re-injecting X0 before each block application after the first keeps encoded features and context labels available without adding parameters.Each block retains the baseline’s attention, feed-forward, normalization, and residual structure.Source: §3.1 · p. 5
- After k loops, the decoder reads query-row states, and the same decoder supports predictions at any depth without depth-specific parameters.Source: §3.1 · p. 5
- With n = 4 and L = 12, ICL parameters fall from 25.7M to 8.6M and total parameters from 27.6M to 10.4M; at k = 3, per-table compute is unchanged.Tying saves memory rather than FLOPs; fewer loops are needed to save FLOPs.Source: §3.1 · p. 5
3.2 TRAINING AND INFERENCE
- Stage 1 samples k from {2, 3, 4} with final-depth supervision, matching the baseline’s expected depth and per-step compute; k = 1 is not seen in training.The restricted range keeps per-step cost and activation memory close to the baseline’s.Source: §3.2 · p. 5
- Stage 2 samples loop counts up to 32, adds a latent-consistency term, and trains Tab-Loop through stage 2 only in this version.The stage-2 objective combines cross-entropy and latent-consistency loss, with gradients tracked through at most the last eight loops.Source: §3.2 · p. 5
- At inference, k sets the ICL block-application budget and can change without retraining; this work sets it globally for each evaluation.Source: §3.2 · p. 5
- At equal compute, Tab-Loop at k = 3 tracks the baseline’s held-out loss throughout stage 1, while every reading with k ≥2 stays below the control.The comparison uses 1,024 synthetic tables; control steps are about 24% cheaper, so 660K control steps match the other runs’ 500K steps in compute.Source: Figure 3 · p. 6
4 EXPERIMENTAL DESIGN
The experiment compares three models that share the prior, encoders, decoder, optimizer, and schedule, with equal stage-1 training compute. The benchmark suite measures held-out log loss, TabArena Elo, and TALENT mean accuracy.
4 EXPERIMENTAL DESIGN
- All three models receive 18.2 EFLOP of stage-1 training; comparisons with the baseline vary tying at fixed depth and compute, while comparisons with the control vary depth against more training tables.Source: §4 · p. 6
- The held-out-prior benchmark uses 1,024 synthetic tables from a seed disjoint from training and reports query log loss with paired comparisons.Source: §4 · p. 6
- TabArena covers 38 classification datasets and 594 splits with Elo, while TALENT covers 200 datasets with mean test accuracy; models are evaluated zero-shot using default preprocessing.Intermediate checkpoints and stage-2 models use TabArena-Lite, whose Elo is not comparable with full TabArena.Source: §4 · p. 6
5 RESULTS
At matched compute, recurrent Tab-Loop matches the untied baseline; test-time loop count raises scores through the trained range, and deeper-loop training extends the useful range at higher inference cost.
5.1 AT EQUAL COMPUTE, TIED DEPTH MATCHES UNTIED DEPTH
- 84.10% vs 83.92% mean TALENT accuracy and 1409 vs 1396 full TabArena Elo show Tab-Loop matching the baseline at equal compute.At k = 3, the models use the same 12 block applications; the held-out loss ends at 0.4583 vs 0.4588, with overlapping 95% intervals on real-data suites.Source: §5.1 · pp. 6–7
- 83.46% TALENT accuracy and 1343 full TabArena Elo leave the parameter-matched control behind Tab-Loop, despite training on 32% more tables.The control also ends at 0.4681 held-out loss; Tab-Loop records 84.10% and Elo 1409, respectively.Source: §5.1 · p. 7
5.2 DEPTH IS A DIAL WITHIN THE TRAINED RANGE
- Elo rises from 1288 at one loop to 1424 at four, while TALENT accuracy reaches 84.18% at four; both remain steady one loop beyond training.Four loops are the only tested depth with a significant TALENT gain over the baseline; gains are neutral at k = 5 and k = 6.Source: §5.2 · p. 7, Figure 4 · p. 7
5.3 TRAINING WITH DEEPER LOOPS EXTENDS THE DIAL
- TabArena-Lite Elo reaches 1512 at 12 loops versus 1509 for the stage-2 baseline, while TALENT accuracy reaches 84.42% at 16 loops.These scores require 4× and about 5.3× the baseline inference compute, respectively; stage-2 training uses deeper sampled loops and its training compute is not matched to the baseline.Source: §5.3 · pp. 7–8, Figure 5 · p. 8
6 RELATED WORK
Related work spans tabular foundation models, parameter sharing and recurrent depth, while Tab-Loop distinguishes itself through a controlled comparison of recurrence in tabular ICL.
6 RELATED WORK
- Tabular foundation models evolved from TabPFN to models with improved priors, architectures, and curricula; Tab-Loop changes only TabICLv2’s untied-layer choice.The cited evaluation suites are TabArena and TALENT.Source: §6 · p. 8
- Prior work studies shared weights across depth, including looped transformers for in-context learning and longer-input generalization.The literature also examines reasoning depth, large-scale looped language-model pretraining, and the trade-off between sharing and specialization.Source: §6 · p. 8
- Language-model studies examine test-time recurrent depth, loop dynamics, and mechanisms that let one block perform different computations across iterations.These dynamics motivate asking how far beyond trained depths a looped model remains useful.Source: §6 · p. 9
- Adaptive-computation methods learn when to halt or choose recursion depth, whereas Tab-Loop makes depth an inference-time choice and leaves per-table adaptation for future work.Source: §6 · p. 9
- Unlike prior tabular recurrence work, Tab-Loop changes only an established model’s ICL stage and compares an untied baseline and parameter-matched control at equal compute.Source: §6 · p. 9
7 CONCLUSION
At modern tabular-foundation-model scale, tied depth matches distinct layers at equal compute with fewer parameters, and deeper training extends the inference-time depth range.
7 CONCLUSION
- Four tied blocks applied three times match twelve distinct blocks at equal training and inference compute, using 62% fewer parameters; the parameter-matched control falls behind.Source: §7 · p. 9
A.1 SHARED COMPONENTS
- Tab-Loop retains TabICLv2’s encoders, label embeddings, prediction head, and other shared components; their outputs are reused as injected input at each ICL block application.Table 3 identifies components shared among the baseline, control, and Tab-Loop and states that their parameter counts are identical.Source: §A.1 · p. 12, Table 3 · p. 12
A.2 ICL BLOCK SPECIFICATION
The models use identical pre-norm ICL blocks; Tab-Loop changes their number and reuse, while adding encoded-input injection before each later application.
A.2 ICL BLOCK SPECIFICATION
- Each ICL block is a standard pre-norm transformer block, and Tab-Loop differs from the baseline in block count, cyclic reuse, and additive input injection.The blocks use the same attention, feed-forward, normalization, and residual structure across models.Source: §A.2 · p. 12
A.3 INPUT INJECTION
- Before each block application after the first, the cached input X0 is added to the current state, without learned parameters or scaling.Injection is uniform, including between blocks within a loop.Source: §A.3 · p. 13
- At 2 loops, re-adding X0 lowers validation log loss by 0.0011 nats, and at 12 loops by 0.0059 nats.The comparison is against no injection.Source: §A.3 · p. 13, Figure 6 · p. 13
- At trained depths, learned linear carry is slightly worse than additive injection; at k = 32 its loss reaches 1.14 nats versus about 0.40.The unconstrained carry becomes expansive, with its state block's largest singular value growing to 2.2.Source: §A.3 · p. 13, Figure 6 · p. 13
- The implementation therefore uses parameter-free injection and leaves constrained carries to future work.Source: §A.3 · p. 13
A.4 COMPUTE ACCOUNTING
The paper accounts for compute per ICL block application and uses synthetic-prior data and loop-count schedules to train and evaluate the models at specified compute budgets.
A.4 COMPUTE ACCOUNTING
- One ICL block application costs about 44 GFLOP at the training table sizes, with identical per-application FLOPs across models.Source: §A.4 · p. 14
B.1 PRETRAINING DATA
- Pretraining uses synthetic tables from a prior mixing structural-causal-model, tree-based, and nearest-neighbour generators; held-out evaluation reuses a fixed 1,024-table slice.The held-out slice comes from the same prior, with a seed disjoint from training.Source: §B.1 · p. 14
B.2 OPTIMIZATION
- At 18.2 EFLOP, the control receives 660K stage-1 steps versus 500K for the baseline and Tab-Loop to equalize total pretraining compute.The training settings are otherwise shared.Source: §B.2 · p. 14
B.3 LOOP-COUNT SAMPLING AND GRADIENT TRUNCATION
- Stage 1 samples k uniformly from 2–4, yielding 12 expected block applications per step; stage 2 samples a mean of about 12 loops and truncates backpropagation to the final min{8, k + 1} loops.Stage 2 also runs one additional loop for the latent-consistency term.Source: §B.3 · p. 14
B.4 EVALUATION PROTOCOLS •
Evaluation uses held-out prior data and zero-shot benchmark protocols, with TabArena and TALENT scores reported across their specified classification datasets.
B.4 EVALUATION PROTOCOLS •
- At each checkpoint, Tab-Loop is evaluated at k = 1 through 6 on mean query log loss from the cached held-out slice, while the baseline and control use fixed depths.Source: §B.4 · p. 14
- TabArena evaluation covers 38 classification datasets and 594 outer-fold splits, reporting Elo with 95% bootstrap intervals; models are applied zero-shot.TabArena-Lite uses one split per dataset, and its Elo is not comparable with full TabArena.Source: §B.4 · p. 15
- TALENT evaluation uses its 200 classification datasets and official train, validation, and test splits, reporting mean test accuracy.Source: §B.4 · p. 15
C.1 EQUAL-COMPUTE COMPARISON AT INTERMEDIATE BUDGETS
At intermediate equal-compute budgets, Tab-Loop and the baseline trade leads across benchmarks, while the parameter-matched control trails Tab-Loop at both budgets.
C.1 EQUAL-COMPUTE COMPARISON AT INTERMEDIATE BUDGETS
- At 3.6 EFLOP, Tab-Loop leads on TabArena-Lite (1215 against 1197) and TALENT (82.65% against 82.58%); at 7.9 EFLOP, the baseline leads on both (1276 against 1248; 83.15% against 82.90%).The comparisons use Tab-Loop at k = 3, matching the baseline's 12 block applications.Source: §C.1 · p. 15, Figure 7 · p. 16
- The control trails Tab-Loop at k = 3 at both intermediate compute budgets.Source: §C.1 · p. 15, Figure 7 · p. 16
- By the end of training at 18.2 EFLOP, Tab-Loop and the baseline are equivalent.Source: §C.1 · p. 15