PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates

Tian Zhou, Beverly Jin, Linxiao Yang, Xue Wang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun

arXiv:2609.27679v1cs.LG

TL;DR

Tabular foundation models use labeled rows as context, but what reusable computation turns each new table into its predictor? The paper derives and implements attention-gated refinement of episode representations, transferring support corrections to queries. RefineICL improves across reported benchmark comparisons, while interventions show that support updates contribute to subsequent query prediction.

  • Problem

    Tabular foundation models predict remaining rows from labeled context, but what reusable computation converts each new table into its predictor?

    Source: §1 · p. 1

  • Method

    The paper derives a regularized support-representation correction from a leave-one-out objective and transfers it to queries, motivating attention-gated residual updates.

    Source: §1 · pp. 1–3

  • Results

    RefineICL improves all four reported metrics over TabPFN-v3 on both TabZilla views and reaches 1644.8 Elo on TabArena, 31.4 Elo above TabPFN-3 under the same evaluation.

    Source: §1 · p. 3, §6.1 · p. 7

  • Takeaways & Limitations

    In 72/72 tested episodes, skipping an intermediate support update while preserving the query output increases final query cross-entropy, supporting a role for evolving support representations in prediction.

    Source: §1 · pp. 2–3

    The L12–L24 accuracy-gap comparison applies only to those checkpoints, which differ in depth, width, and training history; parameter ratios do not establish matched-compute training advantage.

    Source: H SCOPE, LIMITATIONS, AND… · p. 27

Abstract

from arXiv · show

What reusable computation should a tabular foundation model learn when every table defines a new supervised task? We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes. Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.

1 INTRODUCTION

The paper asks what reusable computation turns a labeled table into a task-specific predictor, and proposes refining episode representations using support labels. RefineICL implements this with attention-gated updates; benchmark results and internal interventions support the approach.

1 INTRODUCTION

  • Tabular foundation models predict unlabeled rows from labeled context, but the paper asks what reusable computation converts each table into its predictor.Unlike language-model FFNs’ parameterized memories, tabular feature meanings, class identities, and decision boundaries can vary by episode.Source: §1 · p. 1
  • RefineICL uses attention-gated residual updates, selected low-rank feature interaction, and typed memory without a standalone expanded FFN.In a matched 100K-update depth grid, expansion gives no consistent validation gain and increases peak inference memory by 60.2% at L8.Source: §1 · p. 2
  • The proposed computation refines episode representations using support labels, then transfers the resulting corrections to unlabeled queries without changing model parameters.The support correction depends on label agreement and current neighborhood error.Source: §1 · pp. 1–3
  • 1644.8 Elo on TabArena, 31.4 Elo above TabPFN-3 under the same evaluation; RefineICL also improves all four reported metrics over TabPFN-v3 on both TabZilla views.On AMLB29, RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy.Source: §1 · p. 3
  • Skipping a middle-block support update while preserving its query output increases final query cross-entropy in 72/72 episodes, by 0.0510 on average.Later blocks improve unlabeled-query nearest-prototype accuracy by 2.77 points.Source: §1 · p. 3

2 RELATED WORK

Prior work frames in-context learning as optimization and develops tabular models with varied architectures and test-time computation. This paper complements those accounts by examining how support labels refine episode representations.

2 RELATED WORK

  • The optimizer view shows that Transformers can implement gradient-based or closed-form predictor updates in restricted settings, while later work questions literal gradient-descent equivalence in pretrained models.This paper instead studies updates to episode representations rather than predictor parameters.Source: §2 · p. 3
  • REFINEICL combines selected low-rank feature interaction with attention-gated contextual updates, distinguishing its episode-representation account from related interaction and gating designs.The paper tests how support labels, query state, and depth affect the learned update.Source: §2 · p. 3
  • Tabular foundation models span synthetic-task prior-data fitting, larger training distributions and test-time computation, column/row decomposition, alternating feature/row computation, and compact interleaved architectures.The paper focuses on how labeled rows form a new predictor, with controlled internal-state experiments in the medium-scale regime.Source: §2 · p. 3

3 IN-SITU REPRESENTATION REFINEMENT

The paper defines a supervised neighborhood objective over support states, derives corrections from labeled support examples, and extends those corrections to queries. The resulting attention-gated update motivates a contextual neural block.

3 IN-SITU REPRESENTATION REFINEMENT

  • Tabular ICL is framed as refining labeled episode representations and transferring the resulting correction to each query, making support states part of the computation rather than examples only to retrieve.The model inference remains a single forward pass.Source: §3 · p. 4, §3.1 · p. 4

3.1 WHAT MAKES A REPRESENTATION USEFUL FOR THIS EPISODE?

  • The leave-one-out objective predicts each support row from the others; its correct-class neighbor mass excludes self-votes and is finite with at least two support rows per class.The support update direction depends on label agreement, similarity, neighborhood error, and the current representation.Source: §3.1 · p. 4

3.2 HOW DOES THE CORRECTION REACH AN UNLABELED QUERY?

  • Kernel extension transfers support corrections to an unlabeled query: normalized similarities form an attention readout, while total similarity divided by α sets the update scale.The support labels determine each correction; query similarity determines its influence without a query label.Source: §3.2 · p. 5

3.3 HOW DOES THIS COMPUTATION GUIDE A NEURAL BLOCK?

  • The neural block separates attention-based reading from state-dependent, dimension-wise gating, then projects and adds the product as a residual update without requiring a separate expanded FFN.Depth, width, and labeled context provide resources for applying, representing, and defining task-specific updates.Source: §3.3 · pp. 5–6

4 REFINEICL: ATTENTION-GATED REPRESENTATION REFINEMENT

REFINEICL combines an attention-gated contextual block with feature processing, row compression, and typed-memory re-entry. The reported model sizes vary contextual depth and width while holding feature embedding dimension fixed.

4.1 ARCHITECTURE

  • The architecture uses attention-gated residual updates, selected low-rank feature interaction before row compression, and typed-memory re-entry to recover feature-level information.Feature states are stored after two interaction rounds and retrieved with eight-head gated reads at relative depths 1/3 and 2/3.Source: §4.1 · p. 6

4.2 MODEL SIZES

  • L12 and L24 use feature embedding dimension 256, with contextual (depth, width) settings of (12, 2048) and (24, 1024), respectively.The controlled FFN comparison holds depth and width fixed while toggling the expanded FFN.Source: §4.2 · p. 6

5 EXPERIMENTAL DESIGN

The evaluation combines public benchmark comparisons with controlled tests of representation updates and model components. Benchmarks use defined dataset and ensemble protocols, while mechanism tests use paired inputs and internal interventions.

5 EXPERIMENTAL DESIGN

  • AMLB29, CC18, Grinsztajn, and TabZilla use saved splits and four members, while TabArena uses a 38-dataset, eight-member snapshot.The TabArena and AMLB29 protocols are shown separately in Table 1; its TabArena panel uses a benchmark-informed L24 continuation.Source: §5 · p. 6, Table 1 · p. 7
  • Paired inputs and internal-state interventions test class separation, representation dimensions, nested support sets, and query-specific updates.Query permutations preserve vectors and their value distributions while changing their recipients.Source: §5 · p. 6
  • Matched d = 96 models train for 100K updates at L2, L4, and L8, with FFN-2× toggled across column, row, and contextual blocks.The comparison measures held-out validation, L8 memory, and boundary-rule conditions at 128–1,024 support rows.Source: §5 · p. 7
  • Inference-time component ablations disable one component on identical inputs, with a matched recovery study testing compensation.Paired comparisons measure each component’s effect on the trained model.Source: §5 · p. 7

6 RESULTS

REFINEICL is evaluated on public benchmarks and through controlled interventions on depth, representation width, support labels, and query-specific updates. The results combine benchmark comparisons with evidence about how support representations contribute to query prediction and how the analytic update motivates the architecture.

6.1 ACCURACY AND EFFICIENCY OF REFINEICL

  • 1644.8 Elo on TabArena is 31.4 above TabPFN-3; on AMLB29, L24 reaches 0.87173 accuracy and 0.93836 OVR-AUC, while improving AUC on 27/36 and 68/102 TabZilla datasets.The TabArena score uses a benchmark-informed continuation; the TabZilla counts compare L24 with TabPFN-v3.Source: §6.1 · p. 7, Table 2 · p. 8, §D.1 · p. 15
  • In a matched 100K-update grid, FFN expansion gives no consistent validation gain and raises L8 peak inference memory by 60.2%.Whole-model expansion worsens CE at L2 and L4; a separate 1,024-row boundary-rule probe reports a 1.11-point L8 accuracy gain.Source: §6.1 · p. 7, Table 2 · p. 8

6.2 DEPTH, WIDTH, AND EVIDENCE SCALE CONTEXTUAL REFINEMENT

  • Across 72 episodes, blocks 12–24 increase nearest-prototype accuracy by 2.77 points, while skipping block 12’s support update raises final query CE in all 72 episodes.The support-update intervention preserves block 12’s query output and increases CE by 0.0510 on average.Source: §6.2 · p. 8
  • Projecting states to rank 32 raises loss from 1.04 points at B = 4 to 8.66 at B = 64; as support grows from 128 to 1,024 rows, the L24–L12 gap widens by 4.01 points.These tests vary rule breadth and distinct labeled evidence, respectively.Source: Figure 3 · p. 8, §6.2 · p. 9

6.3 SUPPORT LABELS AND QUERY-SPECIFIC UPDATES

  • On unchanged support and query coordinates, matching support-label rules average 0.985 accuracy versus 0.515 off-diagonal; at 1,024 rows, query-permuted paired updates reduce accuracy by 56.89 points.Figure 4 tests rule selection and query specificity through alternative labels and permutations of attention, gate, or paired updates.Source: §6.3 · p. 9, Figure 4 · p. 9

A WHY THE SUPPORT UPDATE DECREASES THE OBJECTIVE

  • A regularized local optimization of the support objective yields a correction whose update decreases the episode objective under the stated smoothness condition.The local problem fixes the current gradient and kernel, and trades first-order loss change against update size and variation over support geometry.Source: A WHY THE SUPPORT UPDATE DECREASES THE… · p. 13

B EXTENDING SUPPORT UPDATES TO UNSEEN QUERIES

  • A vector-valued RKHS construction extends the support correction to unseen queries; the attention-form approximation is controlled when β∥K∥2/α ≪ 1.This bound motivates the normalized readout and state-dependent scaling.Source: B EXTENDING SUPPORT UPDATES TO UNSEEN… · pp. 13–14

C FROM THE ANALYTIC UPDATE TO THE IMPLEMENTED NETWORK

  • REFINEICL implements attention-based reading with SiLU-gated residual scaling, selected feature interaction before compression, and typed memory that retrieves stored feature states.The equations motivate computational roles but do not equate learned values or gates with analytic gradients or kernel mass.Source: C FROM THE ANALYTIC UPDATE TO THE… · p. 14, Table 3 · p. 14, Figure 6 · p. 15

D.5 AGGREGATE BENCHMARK COMPARISONS

Across separate benchmark views, REFINEICL-L24’s advantage is strongest in TabZilla, while CC18 results are mixed and the Grinsztajn Clean18 margins are small and not statistically separated.

D.5 AGGREGATE BENCHMARK COMPARISONS

  • Benchmark views retain their registered dataset lists and aggregation rules, and overlapping views are reported separately rather than pooled.The reported model-capacity reference cautions that parameter counts do not equal matched inference cost.Source: §D.5 · p. 17, Table 4 · p. 17, Table 8 · p. 18
  • In the four-member CC18 comparison, TabPFN-v3 leads on accuracy and log loss, while L24 leads on OVR-AUC.An eight-member TabICLv2 reference reaches 0.88930 accuracy, 0.93864 OVR-AUC, and 0.25615 log loss on the same 72-dataset list, but its different member budget precludes marking it as a four-member winner.Source: §D.5 · p. 17, Table 5 · p. 17
  • L24 remains ahead in both Grinsztajn sensitivity views, but its Clean18 advantage is only 0.00082 accuracy and 0.00056 AUC.The unadjusted mean differences support consistency of the sign, not statistical separation.Source: §D.5 · p. 17, Table 6 · p. 17
  • L24 improves all four reported metrics over TabPFN-v3 on both TabZilla views.Canonical36 uses the official TabZilla dataset list; Extension102 tests a separate filtering and deduplication reconstruction.Source: §D.5 · p. 18, Table 7 · p. 18

D.6 HOW BROADLY ARE THE GAINS DISTRIBUTED ACROSS DATASETS?

Paired benchmark analyses and controlled interventions characterize the breadth of L24’s gains and how depth, support updates, representation dimensions, and additional labeled evidence relate to prediction.

D.6 HOW BROADLY ARE THE GAINS DISTRIBUTED ACROSS DATASETS?

  • On TabZilla Extension, L24’s mean gains are 0.34 accuracy points [0.09, 0.62] and 0.51 AUC points [0.21, 0.93], with AUC improving on 68 datasets and declining on 17.The paired comparison resamples whole datasets and reports pointwise intervals; overlapping benchmark views are analyzed separately, not pooled.Source: §D.6 · p. 18, Table 9 · p. 18

E.1 DO LATER BLOCKS IMPROVE CLASS SEPARATION FOR THE CURRENT TASK?

  • From block 12 to block 24, query margin improves by 0.0509 [0.0403, 0.0616] and nearest-prototype accuracy by 2.77 [1.40, 4.02] percentage points.Measures are paired across 72 held-out episodes; the net changes indicate stronger class geometry after repeated contextual updates.Source: §E.1 · p. 19

E.2 DO THE LEARNED BLOCK UPDATES IMPROVE THE SUPPORT OBJECTIVE?

  • Actual support updates lower the normalized leave-one-out objective by a mean 0.00458, while row-permuted updates raise it by 0.01192; all 72 episodes favor actual assignment.The effect is a stack-level change, not a literal gradient trajectory; some blocks raise the objective, and the update-gradient mean cosine is 0.0071.Source: §E.2 · p. 19, Table 10 · p. 19
  • Skipping block 12’s support update while preserving its query output raises final query cross-entropy by 0.0510 on average across 72/72 episodes.The paired intervention also lowers query accuracy by 1.44 points, but does not establish that the leave-one-out objective mediates the contribution.Source: §E.2 · p. 19

E.3 DO BROADER RULES USE MORE REPRESENTATION DIMENSIONS?

  • The rank-32 projection increases full-minus-projected accuracy loss by 7.62 [6.03, 9.05] points for the primary comparison, indicating task-dependent use of representation dimensions in this checkpoint.The experiment does not rule out compensation by retraining a smaller-width network.Source: §E.3 · p. 20, Table 11 · p. 20

E.4 DOES ADDITIONAL DISTINCT EVIDENCE CHANGE THE ARCHITECTURE GAP?

  • The L24-minus-L12 accuracy gap grows by 4.01 [2.26, 5.79] points between 128 and 1,024 distinct labeled rows, but is not pointwise monotone.Because checkpoints differ in depth, width, and training history, this isolates how evidence changes their gap, not depth or parameter count alone.Source: §E.4 · p. 20

E.5 IS AN EXPANDED FFN NEEDED AT A FIXED TRAINING BUDGET?

At a fixed training budget, FFN effects vary with depth, while an expanded FFN can improve accuracy at greater memory cost; support labels and query-specific updates also matter to predictions.

E.5 IS AN EXPANDED FFN NEEDED AT A FIXED TRAINING BUDGET?

  • At 100K updates, L4 without an FFN is within 0.00175 CE of L8 with FFN-2×; FFN effects vary by depth, worsening CE at L2 and L4 but improving it at L8.The L2 CE and accuracy measures favor different variants; results use one training seed and held-out GraphSCM-v2 validation.Source: §E.5 · p. 20, Table 12 · p. 21

E.6 HOW MUCH ACCURACY DOES THE FFN BUY AT EACH MEMORY COST?

  • At 1,024 rows, an expanded FFN improves accuracy by 1.11 points (95% CI [0.38, 1.89]) while increasing peak memory by 60.2%.Across 128–1,024 rows, accuracy gains are 18.60 points without expansion and 19.95 points with it; intervals condition on one training seed.Source: §E.6 · p. 21, Figure 7 · p. 22

F.1 DO SUPPORT LABELS SELECT A DIFFERENT PREDICTOR?

  • With L24 parameters and support/query coordinates fixed, changing only support labels across four rule types yields matching-rule accuracy 0.98498 versus 0.51475 against other rules.The predicted class-1 probability surface follows the rule specified by the support labels on the same point cloud.Source: §F.1 · p. 22

F.2 DOES EACH QUERY NEED ITS OWN ATTENTION OUTPUT AND GATE?

  • All three query-permutation damage intervals are positive at every context size, including when the evidence–gate pair is preserved but assigned to the wrong residual state.The interventions permute query-segment vectors while retaining their sets, supporting query-specific updates.Source: §F.2 · p. 22, Table 13 · p. 22

F.3 CAN THE REMAINING COMPONENTS COMPENSATE AFTER DELETION?

Component deletions, sparse-rule tests, and layerwise readouts probe which parts of REFINEICL’s computation matter and whether deeper execution improves prediction or rule identification.

F.3 CAN THE REMAINING COMPONENTS COMPENSATE AFTER DELETION?

  • After 250 recovery updates, removing gate, memory, and row feature interaction together costs 42.54 accuracy points, versus 28.94 for gate deletion alone.Removing both memory reads continues to hurt, while most individual frontend and row-interaction changes stay near zero; these are trained-model dependence tests, not from-scratch necessity tests.Source: §F.3 · p. 23, Table 14 · p. 23
  • On 12 real-table tasks, gate deletion hurts all 12 datasets, whereas row-feature-interaction deletion has its largest effect on Madelon.The check uses four fixed splits per task and was designed after observing the RBF results, so it is not a second test specified in advance.Source: §F.3 · pp. 23–24, Table 15 · p. 24

G.1 DOES BETTER EXECUTION IMPLY BETTER SPARSE RULE IDENTIFICATION?

  • At 32 independent rows, L12 retrieves 39/48 rules and L24 32/48; at 128 rows both retrieve 48/48, after which L24 has lower own-rule query CE by 0.02329.L24’s own-rule query CE advantage is 0.00842 at 512 rows and 0.01557 at 1,024 rows; retrieval and prediction quality differ.Source: §G.1 · p. 24, Figure 9 · p. 25

G.2 DOES A RULE FOUND EARLY DISAPPEAR IN LATER LAYERS?

  • L24 never exceeds its final 32/48 rule retrieval, while L12 ends at 39/48; the layerwise criterion therefore rejects late loss of an already decodable rule.A different intermediate readout could still recover additional information, and cross-checkpoint differences may reflect width, frontend, or training history.Source: §G.2 · p. 25, Figure 10 · p. 25

G.3 CAN RESPONSE RANK BE HIGH WHEN TASK PERFORMANCE IS POOR?

True-label outputs retrieve target rules far more reliably than shuffled-label outputs, even when shuffled responses have higher rank. A separate known-target probe finds improving layerwise predictions, while solver readouts do not identify a unique internal optimizer.

G.3 CAN RESPONSE RANK BE HIGH WHEN TASK PERFORMANCE IS POOR?

  • 47/48 target rules are retrieved by both checkpoints with true labels, versus 1/48 for shuffled L12 and 0/48 for shuffled L24 despite sometimes higher effective rank.The probe uses 512 queries and 48 target rules; its counts are not pooled with the 2,048-query within-family rule set.Source: §G.3 · p. 26, Figure 11 · p. 26

G.4 DOES A LAYERWISE READOUT APPROACH A KNOWN PREDICTION TARGET?

  • 0.13047 to 0.01476 Bayes excess cross-entropy falls from block 0 to block 24, with blocks 13–24 removing another 0.01082.This known-generator probe uses 64 binary logistic episodes; the support-only logistic ERM reference is 0.01416.Source: §G.4 · p. 26
  • Affine-logit readout R2 is 0.97745 for L12 and 0.98770 for L24, but cross-checkpoint differences cannot isolate depth.The candidate optimizer directions are too similar to identify a unique internal optimizer.Source: §G.4 · p. 26

H SCOPE, LIMITATIONS, AND REPRODUCIBILITY

The theoretical update result is conditional on local smoothness, class coverage, and a small-kernel condition; empirical interventions and scale comparisons have narrower scope. Benchmark evidence and released materials also leave specific evaluation and replay boundaries.

H SCOPE, LIMITATIONS, AND REPRODUCIBILITY

  • The update proof assumes a locally smooth leave-one-out support objective with at least two examples per class, and its kernel approximation requires small β∥K∥2/α.The network does not explicitly compute the exact gradient or its scale; query risk and repeated learned-block descent are empirical questions.Source: H SCOPE, LIMITATIONS, AND… · p. 27
  • Mechanism tests emphasize synthetic classification with known rules, while inference-time disabling can create unseen states and short recovery does not replace converged ablation training.A shared final decoder may miss intermediate information; the fixed-budget FFN grid has one training seed, and paired episode intervals omit training variance.Source: H SCOPE, LIMITATIONS, AND… · p. 27
  • L12–L24 context-size accuracy-gap changes apply only to that pair because depth, width, and training history differ; added depth or context does not improve uniformly.The rank-32 projection measures sensitivity within one trained model, and parameter ratios against TabFM compare storage rather than matched-compute training advantage.Source: H SCOPE, LIMITATIONS, AND… · p. 27
  • TabArena ranking is limited to a fixed 38-dataset fold-0 snapshot with heterogeneous reference budgets; regression and complete RefineICL TALENT evaluations are outside reported scope.Overlapping OpenML views and metrics are not independent trials, and small mean differences alone do not establish statistical superiority; released source does not include trained weights, benchmark data, or continuation manifests.Source: H SCOPE, LIMITATIONS, AND… · p. 27

Related papers

Loading 2609.27679v1…