Support-Compiled Feature Folding: More Evidence at Lower Memory Across Tabular Foundation Models

1 INTRODUCTION

• On robert, encoding all 7,200 features in groups of at most 128 lowers TabICLv2 peak allocated GPU memory from 59.3 to 1.83 GiB without changing its checkpoint, context encoder, or head.This is an illustrative folding pilot; Table 1 evaluates final SCFF separately.Source: §1 · p. 1, Figure 1 · p. 2
• Full-width pairwise feature mixing costs Θ(TD^2) interactions before the context encoder reasons over support rows.This feature-side cost differs from context length and model capacity as sources of inference capability.Source: §1 · p. 1
• Full-width attention lets useful variables compete with weak and nuisance variables, whereas feature selection permanently removes signals outside its cutoff.The paper identifies processing all candidate evidence with bounded local interactions as a missing inference primitive.Source: §1 · p. 1

B MEASURED PEAK

• SCFF routes support-ranked Core and Tail features through the same bounded-width feature encoder, merges their row representations, and sends the result through the original context encoder and head once.Tail evidence is included only after a reciprocal support-subset acceptance test; folding changes feature interactions rather than ensembling predictions.Source: B MEASURED PEAK · p. 2
• The evaluation uses all eligible dataset versions in fixed AMLB-29, TabZilla, and TabArena snapshots with at least 128 raw features, subject to declared class and development-exclusion rules.The 18-dataset panel tests transfer across models and uses paired Core and Tail controls.Source: B MEASURED PEAK · p. 2
• Across six frozen backbones, SCFF improves every dataset-macro accuracy and NLL comparison on the exhaustive 18-dataset wide-table slice, with relative error reductions up to 26.1%.All four matched-width comparisons retain favorable 95% dataset-bootstrap intervals on locked folds.Source: B MEASURED PEAK · p. 2

2 RELATED WORK

• Folding applies to foundation models with a width-dependent feature encoder and compatible row representations, rather than to every model labeled a tabular foundation model.The paper identifies interface compatibility as the determinant of how folding can be applied.Source: §2 · p. 3
• Unlike test-time divide-and-conquer that combines predictions or continued pre-training for extreme widths, SCFF merges row representations before one prediction and leaves backbone weights frozen.It selects a Core with ANOVA/BH and separately encodes the remaining columns as candidate Tail evidence.Source: §2 · p. 3
• SCFF reuses one feature encoder on disjoint leaves and combines row representations without a learned set encoder, before a shared context encoder that retains all support and query rows.The label head executes once, unlike prediction averaging across predictors.Source: §2 · p. 3

3 SUPPORT-COMPILED FEATURE FOLDING

• SCFF exposes a model interface with a width-dependent feature encoder, a context encoder, and a label head, then changes the feature-encoder execution while keeping the other layers unchanged.The input episode contains support features and labels plus query features.Source: §3 · p. 3

3.1 ROUTE, FOLD, AND ENCODE

• Support-only one-way ANOVA ranks columns on a median-filled support copy, and Benjamini–Hochberg sets the Core size rather than a hand-tuned top-K.The strongest max(1, K) columns form Core; all remaining columns form candidate Tail, and query labels are unavailable to routing.Source: §3.1 · p. 3

A COMPILE ON LABELED SUPPORT ROWS

• Every column is routed to Core or Tail, partitioned into disjoint leaves of at most M = 128, and encoded by the same feature encoder into row-aligned intermediate representations.Each representation preserves the original support and query rows and has the latent width expected by the context encoder; it is not a class prediction.Source: A COMPILE ON LABELED SUPPORT ROWS · p. 4, Figure 2 · p. 4

3.2 BOUNDED CORE WITH AN OPTIONAL TAIL EXTENSION

• SCFF averages Core leaf messages into c, allowing rich feature interactions among at most M strong features at a time before combining row representations at the standard model interface.Bounded leaves increase local share, while the paper cautions that this local change does not guarantee greater final influence.Source: §3.2 · p. 5, §3.3 · p. 5
• Optional Tail messages are averaged, stripped of their Core-aligned direction, and tested as raw or support-prototype candidates with reciprocal subset checks; rejection returns the exact Core representation.Prototype candidates use all-support class centroids for query rows and leave-one-out own-class centroids for support rows; this is an acceptance test, not independent cross-fitting.Source: §3.2 · p. 5, §3.3 · p. 5
• The resulting h = c + αv enters the context encoder and label head once to produce one complete query distribution; prototype scores are not averaged model predictions.The Tail update is zero when the support checks reject it, and multiclass episodes use this fallback.Source: §3.2 · p. 5

3.3 WHAT FOLDING CHANGES

• The support-side Tail acceptance test requires support evidence to transfer in both directions, while held-out Tail controls test whether accepted evidence improves unseen queries.If accepted evidence is absent, the update is zero and the combined representation remains exactly Core.Source: §3.3 · p. 5
• For a pairwise feature mixer, bounded leaves change feature-interaction work from Θ(TD^2d) to Θ(TDMd), with an O(TM^2) sequential working set.The combined representation has shape T × d_ICL, so the context encoder and label head still run once at the original context length.Source: §3.3 · p. 5

3.4 ARCHITECTURE INTERFACE

• The architecture requires compatible row representations before a shared context encoder and must reproduce native inference; models without that interface need a distinct adapter.The passage identifies LimiX as a token-axis variant whose adapter preserves Core slots and folds remaining features into bounded Tail slots.Source: §3.4 · p. 6

4 EXPERIMENTAL DESIGN

• The benchmark covers 18 eligible datasets with 128–7,200 raw features; folds 1–4 test new query rows on the same datasets, not transfer to unseen datasets.Fold 0 is reserved for finite method selection.Source: §4 · p. 6
• Each native–SCFF pair uses the same checkpoint, preprocessing, split, and query rows; results macro-average datasets across accuracy, NLL, time, and peak GPU memory.Four backbones use matched-width comparisons; LimiX uses a token-aligned comparison and TabPFNv1 a declared capacity extension.Source: §4 · p. 6
• The Tail update is selected on fold 0 and locked before folds 1–4; held-out confirmation requires at least three of five compatible backbones to improve both metrics.The five deterministic stratified query folds are pairwise disjoint, while their complementary support sets overlap.Source: §4 · p. 6
• Paired controls compare Raw Core with Bounded Core and Tail Off with Tail On, changing one mechanism at a time while retaining one prediction over every feature.Component controls test seven TabICLv2 alternatives, including random routing, projection removal, and successive Tail variants; reported deltas compare full SCFF with each complete variant.Source: §4 · p. 6
• Width and context interventions append up to 512 split-local, label-independent columns to four real tasks and test context reinvestment under a fixed-memory budget.The fixed-memory pilot uses a separately registered mass-Huber folding variant, not a long-context validation of the final Tail update.Source: §4 · p. 7

5.1 ONE TRAINING-FREE COMPILER IMPROVES SIX FROZEN BACKBONES

• Across six backbones, SCFF improves accuracy and NLL; matched-width intervals favor it on both metrics, with relative error reductions up to 26.1% and relative NLL reductions up to 19.0%.Median paired memory savings are 2.09–2.36×, and separately observed maximum-peak ratios reach 34.3×.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 BOUNDED CORE AND TAIL RECOVERY CONTRIBUTE DISTINCT GAINS

• Across three backbones, selected Core yields +0.690 to +5.175 accuracy points, and folding that Core improves both predictive metrics; raw/bounded Core peak ratios reach 30.1–34.0×.Tail On improves both point estimates on TabPFN-3, TabICLv1, and TabICLv2 while retaining exact Core fallback.Source: Table 2 · p. 7, §5.2 · pp. 7–8

5.3 CONTROLLED ALTERNATIVES ISOLATE WHY SCFF WORKS

• Random routing falls below native, while removing Core projection costs 0.054 accuracy points and 0.00125 NLL.Raw residual nearly matches accuracy but worsens NLL by 0.00403; reciprocal support-subset checking gives the best point estimates among the fixed Tail controls.Source: Table 3 · p. 8, §5.3 · p. 8

5.4 MODERATE LEAF WIDTH PRESERVES USEFUL INTERACTION CAPACITY

• M = 32 loses quality, whereas M = 64 and M = 128 improve over native; the full evaluation uses fold-0-selected M = 128.The ablation retains all columns and covers four datasets and five folds.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 MEMORY SAVINGS BUY MORE USEFUL EVIDENCE

• At matched peak memory, folding gains +4.06 accuracy points on TabICLv2 and +3.72 points on TabPFN-3 over the widest feasible single leaf.The complete registered panel remains positive on both backbones; excluding TabICLv2’s two largest-gain datasets leaves +0.75 accuracy points and 0.0191 NLL reduction across 16 datasets.Source: §5.3 · p. 8, Table 6 · p. 9, §5.5 · p. 9

A.1 FEATURE COVERAGE AND RESOURCE SCALING

• Every preprocessed feature is encoded exactly once, but folding does not reconstruct the nonlinear full-width representation, and total device memory can still grow with retained leaf representations.Coverage, bounded leaf width, raw-residual geometry, and exact Core fallback are deterministic; better query predictions depend on the feature encoder, representation interface, and support-to-query transfer.Source: §A.1 · pp. 11–12, A FORMAL PROPERTIES AND SCOPE · p. 11

A.2 TAIL RESIDUAL AND RECIPROCAL SUPPORT-SUBSET ACCEPTANCE

• An accepted multi-leaf Tail update must improve Fisher separation and nearest-prototype NLL in both support directions without lowering either support accuracy; rejection returns the exact Core representation.The single-leaf fallback uses only the two-way Fisher condition and does not guarantee prototype-NLL improvement; multiclass tasks also fall back to Core.Source: §A.2 · p. 12

A.3 A CONDITIONAL SUPPORT-TO-QUERY STATEMENT

• Support-accepted gains transfer to queries only under the added stability condition |∆Q(v) − ∆S(v)| ≤ ε; the proposition does not guarantee gains after an arbitrary context encoder and head.Exchangeability motivates estimating query gain from support splits but does not itself imply the condition.Source: §A.3 · p. 13

B.1 SUPPORT COMPILER

• ANOVA and Benjamini–Hochberg route the highest-ranked columns to a nonempty Core, while every remaining column is retained as a candidate Tail.All leaves use the same feature encoder, and their representations are averaged with equal leaf weight before support-only validation.Source: §B.1 · p. 13

B.2 ROW-REPRESENTATION AND TOKEN-PRESERVING ADAPTERS

• Five backbones combine aligned row representations before a shared context encoder; TabPFNv1’s comparison tests expanded capacity beyond its fixed 100-feature view.LimiX instead preserves its trained token axis by encoding bounded leaves with its released tokenizer and allocating a support-RMS-rescaled token budget to Core and Tail.Source: §B.1 · p. 13, Table 8 · p. 13, §B.2 · p. 14

B.3 EXECUTION CHECKS

• Each query uses one feature partition, context-encoder call, label-head call, and complete class distribution; leaf-local prototype scores select Tail updates, not predictions.Checks require every feature to appear exactly once within the width bound, and rejected Tail updates must reproduce the exact Bounded Core representation.Source: §B.3 · p. 14, Table 9 · p. 14

C.1 BENCHMARK SELECTION

• The OpenML Wide-Table panel is the exhaustive set of fixed-registry dataset versions meeting the preregistered metadata rule; fold 0 supports finite method selection, while folds 1–4 evaluate the locked method.Confirmation tests new query rows from the same datasets, not transfer to unseen datasets.Source: §C.1 · pp. 14, 16, Table 10 · p. 15

C.2 PAIRING, AGGREGATION, AND RESOURCES

• All four matched-width comparisons have favorable locked-fold 95% dataset-bootstrap intervals for both predictive metrics.The headline five-fold point estimates include fold 0, so the complete method is confirmed separately on folds 1–4; effects average paired folds within each dataset before macro-averaging.Source: §C.2 · p. 16, Table 12 · p. 16, Table 14 · p. 17

D.1 WHY SEPARATE CORE FOLDING FROM TAIL RECOVERY?

• Bounded Core improves both point estimates on all three backbones versus the same selected Core, with separate maximum-peak ratios of 30.1–34.0×.Both folding intervals exclude zero only on TabICLv2; selection supplies most of the quality gain.Source: §D.1 · p. 18, Table 16 · p. 18

D.2 WHEN DOES BOUNDED CORE PROCESSING CHANGE PREDICTIONS?

• At selected Core width K > 256, both quality point estimates favor bounded Core on each of three backbones, while K ≤ 128 is approximately an identity control.The width-bin analysis is post-evaluation, and only five datasets contribute to the widest bin, so it describes observed effects rather than a confirmed threshold.Source: §D.2 · p. 18, Table 18 · p. 19

D.3 HOW WAS THE TAIL UPDATE SELECTED AND CONFIRMED?

• Three of five compatible backbones improve both primary metrics with the selected Tail update, satisfying the predeclared selection rule on held-out folds.TabICLv2 has favorable intervals for both Tail effects: Accuracy +0.145 pp [+0.009, +0.334] and NLL −0.00294 [−0.00673, −0.00012]; TabPFNv1 and RefineICL-24L do not improve both metrics.Source: §D.3 · pp. 19–20, Table 20 · p. 20

D.4 WHAT DO SIMPLER AND RESOURCE-MATCHED BASELINES EXPLAIN?

• The measured-memory comparison selects the widest feasible single-leaf ANOVA prefix before aggregating prediction metrics, using held-out folds and the fixed 18-dataset registry.The TabICLv2 and TabPFN-3 table captions define positive accuracy and NLL reductions as favoring folding, but the supplied passages do not give those comparison values.Source: §D.4 · pp. 20–21, Table 22 · p. 21, Table 23 · p. 21
• Folding lowers incremental peak allocation on 30/30 units, to 13.4% of the single-leaf route at the overall median, but is slower on 28/30 units.Median same-feature speed is 0.709× (about 1.41× longer); at K > 512, the median memory ratio is 0.093×.Source: §D.4 · p. 20, Table 21 · p. 20

E ROBUSTNESS AND RESOURCE ANALYSES

• For K > 256, median raw/bounded peak-memory ratios were 4.66×, 5.11×, and 4.96× for TabPFN-3, TabICLv1, and TabICLv2; K ≤128 was near parity.Source: Figure 5 · p. 22

E.1 DOES FOLDING RESIST IRRELEVANT WIDTH?

• At 512 added columns, SCFF is 1.440 accuracy points and 0.04063 NLL better than native, with smaller accuracy and NLL changes from dose zero.The intervention supports robustness to its tested nuisance generator and dose range, not arbitrary dependent or adversarial columns.Source: Figure 6 · p. 22, §E.1 · p. 22

E.2 HOW DO LEAF WIDTH AND CONTEXT INTERACT?

• M = 32 loses quality, while M = 64 and 128 improve over native; 64 is slightly better on pooled estimates, but 128 remains the default, not a uniquely optimal width.Source: §E.2 · pp. 22–23
• The context panel shows heterogeneous rather than monotone context–width effects, including mixed responses for dilbert and large gains for guillermo.Its width bands and representatives were fixed from input metadata, and context prefixes reused support order and query rows within each dataset.Source: §E.2 · p. 23, Table 25 · p. 23

E.3 CAN SAVED FEATURE MEMORY PROVIDE MORE CONTEXT?

• In the fixed-memory study, three ultra-wide tasks improved with more context, while the 2,000-feature boundary dilbert slightly worsened under an earlier averaging rule.The complete D ≥2000, n ≥10000 panel contained four datasets; the experiment predates the final residual/prototype update.Source: §E.3 · p. 23, Table 26 · p. 23

E.4 COMPOSING WITH GENERATED FEATURE SPACES

• On 10 of 11 datasets, C improved NLL relative to B while using 24.3% of B’s peak memory and 0.71× its time, but macro C did not recover raw-table A.A is the raw table, B native inference on every generated feature, and C SCFF on that same generated table.Source: §E.4 · pp. 23–24, Table 27 · p. 24

F APPLICABILITY AND FAILURE BOUNDARIES

• Tail validation currently supports binary tasks; multiclass tasks use Bounded Core, stored leaf representations grow linearly with width, and the Tail update improves only three of five compatible models.Source: F APPLICABILITY AND FAILURE BOUNDARIES · p. 24

• No development-fold observation enters the reported dataset-level values.Source: G LOCKED-FOLD DATASET EFFECTS · p. 24
• Tables 28–31 report locked-fold dataset effects for TabPFN-3, TabICLv1, TabICLv2, and RefineICL-24L, respectively; each row averages folds 1–4.Source: G LOCKED-FOLD DATASET EFFECTS · p. 24, Table 28 · p. 25, Table 29 · p. 26
• Typical locked-fold effects summarize dataset-level medians and win/tie/loss counts, with positive Accuracy and negative NLL favoring SCFF.Each entry first averages folds 1–4 within a dataset; the summaries describe datasets, not individual query rows.Source: Table 32 · p. 29

• Reproduction requires checkpoint hashes, dataset and fold records, support/query hashes, preprocessing and method revisions, and runtime fingerprints; released materials omit dataset rows and per-query predictions.Source: H REPRODUCIBILITY · p. 29
• The supplementary archive identifies a portable locked-fold metric root, indexes paired fold-level metrics, and provides a bootstrap receipt recording the same-Core estimands and intervals.Source: H REPRODUCIBILITY · p. 29
• Independent re-execution of RefineICL-24L awaits companion weights because its exact concurrent checkpoint is not redistributed.Source: H REPRODUCIBILITY · p. 29