GIST: Generator-state-Informed Supervision for Tabular Foundation Models

1 INTRODUCTION

• GIST addresses whether retained generator information can reduce query-label sampling variability during pretraining.Standard PFN pretraining uses sampled hard query labels, an unbiased Monte Carlo estimate of population loss and gradient.Source: §1 · p. 1
• GIST improves mean rank and wins on a majority of datasets in all six IID architecture–benchmark combinations at final matched checkpoints.The introduction also describes temporal and grouped OOD evaluation on BeyondArena, without stating its outcome here.Source: §1 · p. 2
• GIST couples synthetic task generation with conditional query supervision, using entropy-calibrated Gaussian encodings and posterior targets on supported label-dependent branches.Unsupported branches retain hard-label supervision, and the predictor receives standard context and query inputs.Source: §1 · p. 2
• Under a fixed generating distribution and at fixed parameters, conditional loss averaging preserves the population objective and yields no larger gradient covariance.The method uses retained generator information to construct targets, not as additional predictor inputs.Source: §1 · pp. 1–2
• 62.1% of paired dataset-level comparisons favor GIST over corresponding hard-label controls across more than 200 TabArena and TALENT datasets and three TFM architectures.Evaluations use matched pretraining schedules.Source: §1 · p. 2

2 PRELIMINARIES AND PROBLEM SETUP

• The model input S contains the labeled context, query features, and context size, while Z adds selected information retained during task generation.For each class k, pZ(k) is the conditional probability given Z, and πS(k) is its conditional expectation given S.Source: §2 · pp. 2–3
• The population objective is expected hard-label cross-entropy, estimated with an unbiased Monte Carlo average over synthetic episodes.Task, input, and query-label sampling contribute to gradient variance.Source: §2 · p. 3
• The posterior predictive πS minimizes population risk among predictors using only S, and averaging conditional targets over generator information recovers πS.The setup seeks retained states that make conditional targets tractable while comparing supervision under the same fixed generating distribution.Source: §2 · p. 3

3.1 GENERATOR-INFORMED QUERY SUPERVISION

• For terminal query-label draws, sampling probabilities directly give the conditional target when task filtering is independent of the remaining label randomness; unsupported branches retain hard labels.When sampled classes affect observed features, their evidence must update the pre-draw probabilities.Source: §3.1 · p. 3

3.2 GAUSSIAN ENCODING FOR LABEL-DEPENDENT FEATURES

• On supported label-dependent branches, GIST routes class effects through noisy category codes, whose local likelihood yields posterior query targets after task filtering.The same channel generates the category representation and defines its supervision target.Source: §3.2 · p. 4

3.3 ENTROPY-BUDGET CALIBRATION

• Entropy-budget calibration selects the largest certified noise scale, adapting to class probabilities and code distances; its bound retains at least (1 − β)H label information on average.Nearby codes can require a smaller scale, and infeasible budgets retain the original encoding and sampled hard labels.Source: §3.3 · pp. 4–5, Figure 2 · p. 4

3.4 TRAINING OBJECTIVE AND STATISTICAL GUARANTEES

• Conditional averaging preserves the population objective and yields no larger gradient covariance at fixed parameters; selective hard-label fallback retains these guarantees.The comparison is within a fixed generating distribution, and the mixed objective selects tractable branches before the residual label draw.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

• The experiments assess downstream prediction under matched schedules, clipped-gradient variability under a shared prior, and sensitivity to the entropy budget.Source: §4 · p. 6

4.1 EXPERIMENTAL SETUP

• Evaluation covers 38 TabArena and 188 TALENT IID datasets plus architecture-specific BeyondArena sets; Hard and GIST use matched schedules and inference settings.The BeyondArena sets contain 21 TabICLv2, 18 TabSwift, and 12 TabPFN-2.5 datasets; the default schedule totals 36K updates.Source: §4.1 · p. 6

4.2 PREDICTIVE PERFORMANCE

• Across all six IID architecture–benchmark combinations, GIST improves mean rank and binary ROC-AUC and wins on a majority of datasets.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8
• On BeyondArena, GIST wins 15/18 TabSwift, 11/21 TabICLv2, and 7/12 TabPFN-2.5 datasets; shared-set multiclass log loss falls for all three.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 GRADIENT VARIANCE ACROSS TRAINING AND COMPONENTS

• Below 1% at initialization, clipped-gradient covariance reductions reach 6.80%–26.04% at 36K updates, with all six pointwise intervals above zero.The final reduction is smaller at L = 8192 than at L = 1024.Source: §4.3 · pp. 7–8
• At 36K updates, non-head components account for 72.0%–91.5% of net reduction, while the head’s larger proportional reductions have limited whole-model weight.The prediction head’s within-component relative reduction ranges from 37.70% to 47.23%, while it accounts for 1.2%–13.9% of Hard gradient variance.Source: §4.3 · p. 8
• TFcol contributes more than half the net reduction in TabICLv2 and TabPFN-2.5, whereas MLPs lead in 11 of 12 post-initialization TabSwift settings.Within-column attention is also a leading contributor in TabPFN-2.5; leading contributors are largely stable across trained checkpoints, but differ by architecture.Source: §4.3 · p. 8

4.4 ENTROPY-BUDGET SENSITIVITY

• All tested entropy budgets β∈{0.05, 0.15, 0.30} improve TabSwift mean binary AUC and multiclass log loss over Hard on both IID benchmarks, though the preferred value varies.Source: §4.4 · p. 9

5 RELATED WORK

• Prior tabular prediction includes tree, neural, and AutoML approaches, while PFN-style TFMs learn across tables from synthetic tasks sampled from a prior.Source: §5 · p. 9
• PIQL uses generator representations at inference, whereas GIST uses generator information to construct targets; ALICE also replaces sampled labels with conditional probabilities.GIST jointly designs task generation and supervision to obtain tractable posterior targets through an entropy-calibrated local channel on supported label-dependent branches.Source: §5 · p. 9

6 CONCLUSION

• GIST’s results motivate considering the supervision a synthetic generator can provide alongside the tasks it produces; transfer under distribution shift remains open.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI assisted with selected manuscript drafting, revision, organization, and feedback; the authors reviewed the content and take responsibility for the final manuscript.Source: AI USE STATEMENT · p. 10

A.1 REGULARITY CONDITIONS AND OBJECTIVE PRESERVATION

• Conditional supervision preserves the posterior-predictive population objective when the stated regularity conditions hold.The conditions include integrable losses, finite gradient second moments, and interchanging differentiation with the relevant expectations.Source: §A.1 · p. 13

A.2 GRADIENT COVARIANCE AND THE SOFTMAX EXPRESSION

• The Rao-Blackwellized gradient has no larger covariance or mean-squared error than the hard-label gradient at fixed parameters.The covariance difference is positive semidefinite, and both gradients have the same mean.Source: §A.2 · p. 13

A.3 REMAINING SOURCES OF GRADIENT VARIATION

• With Z = (T, S), conditional supervision removes label variation, while task and episode variation remain.The covariance ordering is VH ⪰ VRB ⪰ Cov(GP); for more general Z, the middle term covers retained generator information conditional on S.Source: §A.3 · p. 13

A.4 SELECTIVE CONDITIONAL SUPERVISION

• Selective supervision preserves the population objective and reduces conditional gradient covariance on branches where it is enabled.The branch decision must be determined by retained state before the residual label draw.Source: §A.4 · p. 14

A.5 MULTIPLE QUERIES AND INDEPENDENT EPISODES

• For multiple queries, the result applies to the whole table without assuming conditional independence between query labels.Averaging m independent table gradients divides their covariance by m; within-table covariance accounts for cross-query contributions.Source: §A.5 · p. 14

A.6 ASYMPTOTIC PARAMETER COVARIANCE

• Under the stated M-estimation assumptions, hard-label and conditional-supervision minimizers share the m^-1/2 rate, while conditional supervision reduces limiting parameter covariance.The comparison assumes a fixed parameter dimension, a unique interior minimizer, a positive-definite population Hessian, and consistent empirical minimizers.Source: §A.6 · p. 14

B.1 OBSERVED FEATURES AND TASK ACCEPTANCE

• Pre-draw label probabilities remain valid only when the observed input and task acceptance are independent of the remaining query-label draw.When generated features depend on the sampled label, the likelihood can update the target; for X = Y with a uniform binary label, observing X = 1 makes the conditional label deterministic.Source: §B.1 · p. 15

B.2 POSTERIOR UNDER THE GAUSSIAN CHANNEL

• On supported Gaussian-channel branches, the query target is the posterior given the retained state and encoding, when downstream inputs and acceptance satisfy the stated independence condition.Under that condition, the downstream likelihood cancels and P(Yr = k | Zσ, E = 1) = P(Yr = k | H, Wr).Source: §B.2 · p. 15

B.3 OBJECTIVE PRESERVATION AFTER CALIBRATION AND FALLBACK

• Calibration with hard-label fallback on untreated branches preserves the population objective of Pcal.Exact conditional targets apply on enabled branches satisfying the downstream condition; the generator state includes the branch decision.Source: §B.3 · p. 16

B.4 INSTANTIATION IN GRAPHSCM CONVERTERS

• GraphSCM softmax-id converters can use pre-draw probabilities when the sampled class does not affect the input, while softmax-disc and softmax-int modes use posterior targets when it can.Nearest-neighbor converters retain one-hot targets, and infeasible entropy budgets use sampled hard targets.Source: §B.4 · p. 16

C.1 FROM MIXTURE ENTROPY TO CONDITIONAL LABEL ENTROPY

• The entropy analysis bounds conditional label entropy between zero and the derived upper bound, which itself does not exceed the prior label entropy.Zero-probability terms are interpreted by continuity.Source: §C.1 · pp. 16–17

C.2 MONOTONICITY, DUPLICATE CODES, AND FEASIBILITY

• Duplicate codes can leave residual uncertainty at zero noise, and an infeasible entropy budget retains the original encoding and sampled one-hot target.The positive-channel entropy guarantee applies only when calibration returns a feasible positive scale.Source: §C.2 · p. 17

C.3 AVERAGE INFORMATION GUARANTEE AND ITS SCOPE

• For feasible positive scales, the information guarantee averages over all rows and label and encoding draws before task filtering.Calibration selects the largest certified scale within this constraint; the resulting conditional supervision satisfies the stated guarantee.Source: §C.3 · p. 17

C.4 RELATIVE AND BINARY-CAP BUDGETS

• The relative and binary-cap policies coincide for binary tasks, while the binary cap limits the absolute residual-entropy allowance for multiclass tasks with H > log 2.Both policies satisfy the information guarantee; the all-row policy applies one scale to context and query encodings.Source: §C.4 · p. 18

D.1 PRETRAINING AND CHECKPOINTS

• The study pretrained TabICLv2, TabSwift, and TabPFN-2.5 under the two-stage protocol, with Stage 2 initialized from each condition’s own Stage 1 model.Checkpoint selection was fixed across datasets.Source: §D.1 · p. 18
• Hard and GIST used the same effective global batch sizes and learning-rate schedule within each architecture.The implementations had 27.55M parameters for TabICLv2, 8.20M for TabSwift, and 10.72M for TabPFN-2.5.Source: §D.1 · p. 18
• GIST used a task-level entropy fraction of 0.15, all-row averaging, selective conditional targets, and the binary-cap policy without an external Gaussian-scale cap.Architecture and inference configuration were matched within each Hard/GIST pair.Source: §D.1 · p. 18

D.2 BENCHMARK PROTOCOLS AND AGGREGATION

• TabArena evaluation used 38 classification datasets and 594 official outer splits, while TALENT used stratified five-fold cross-validation on combined dataset partitions.The main comparison covered 188 datasets shared by all methods, with 940 observed folds per method; TALENT’s transferred-hyperparameter protocol was not nested cross-validation.Source: §D.2 · p. 18
• Table 4 reports Stage 2 paired win/loss/tie counts over each architecture’s complete supported range, which cannot be used for cross-architecture rankings.The caption defines W/L/T as GIST outcomes against Hard.Source: Table 4 · p. 19, §D.2 · p. 19
• BeyondArena’s complete paired scope covered 21 datasets for TabICLv2, 18 for TabSwift, and 12 for TabPFN-2.5.Only datasets with every official split completed for both conditions entered paired comparisons; incomplete or cancelled evaluations received no imputed score.Source: §D.2 · pp. 18–19
• Mean rank and Improvability were averaged within datasets and then across datasets with equal weight, so datasets with more outer splits received no greater aggregate weight.Ranks were computed separately for each training stage’s method set.Source: §D.2 · p. 19
• Paired wins compare mean raw dataset metrics, counting higher AUC or lower log loss as a win and differences within 10^-12 as ties.Mean rank, Improvability, and win rate summarize different aspects of performance and need not agree in direction.Source: §D.2 · p. 19

• GIST lowers mean Improvability for all three architectures on the common BeyondArena set.Source: §D.3 · p. 19
• TabSwift also improves mean rank under GIST.Source: §D.3 · p. 19
• Across 11 methods, TabM has the lowest Improvability, while the Hard TabPFN-2.5 control has the lowest mean rank.Source: §D.3 · p. 19

D.4 GRADIENT DIAGNOSTICS

• The diagnostics compare Hard and GIST at fixed checkpoints and table lengths, using shared forwards and 4,000 paired batch resamples for pointwise 95% intervals.Gradients are clipped at global norm one before estimating covariance traces.Source: §D.4 · pp. 19–20, Table 7 · p. 21
• The diagnostic targets tractable calibrated-GraphSCM tasks; they differ from the full pretraining stream, and full-mixture target coverage is about 26.2% at L = 1024 and 30.7% at L = 8192.The remaining full-mixture queries are largely hard-label fallback.Source: §D.4 · p. 20

D.5 FUNCTIONAL COMPONENTS AFTER CLIPPING

• Post-clipping component contributions add over disjoint parameter groups without assuming group independence, and depend on both Hard variance share and relative reduction.Tables 8 and 9 report decomposition estimates at L = 1024 and L = 8192.Source: §D.5 · p. 21, Table 8 · p. 22, Table 9 · p. 23
• At L = 1024, TFicl’s 26.85% relative reduction contributes 2.21 percentage points, while TFcol’s 9.05% reduction contributes 7.87.TFcol accounts for 86.95% of Hard variance, compared with 8.23% for TFicl.Source: §D.5 · p. 21

D.6 BUDGET-SENSITIVITY PROTOCOL

• TabSwift sensitivity runs use fixed task-level budgets β ∈ {0.05, 0.15, 0.30} throughout both stages, and Table 10 reports full Stage 2 metrics and paired win rates.Model and data seeds are fixed across budgets; checkpoint selection is fixed across datasets.Source: §D.6 · p. 23