PaperlayerView exact revision on arXiv

Source-linked AI summary

The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models

Duong Nguyen, Nicolas Chesneau, Milan Bhan

arXiv:2610.08314v1cs.AIcs.LG

TL;DR

The paper asks whether pretrained tabular foundation models use context labels with fixed weights, while standardization makes ordinary derivatives an unreliable test. It introduces two certificates based on standardized-label predictions and finds joint processing across evaluated models, with training and attention scores associated with the measured interaction.

  • Problem

    It is unclear whether pretrained tabular foundation models use context labels only as values combined by fixed weights or whether one label changes another’s influence.

    Source: §1 · p. 1

  • Method

    Two certificates use predictions on standardized labels to test fixed-weight prediction and sums of independent nonlinear label transformations.

    Source: §1 · p. 2

  • Results

    Both certificates indicate joint processing in all five evaluated models, with the share of corrected curvature unexplained by one-label-at-a-time maps ranging from 23% to 92% across settings.

    Source: §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Explanations of these TFMs must account for context labels changing the influence of individual examples, though the certificates do not identify the computation that produces the interaction.

    Source: §5 · p. 9

    The derivative tests establish that interactions are present, not that they improve predictions; determining their predictive value requires removing them from the forward computation and measuring accuracy.

    Source: §5 · p. 9

Abstract

from arXiv · show

Linear regression and kernel smoothing offer tractable explanations of in-context learning: in both, the features determine the weight assigned to each context label. However, whether this fixed-weight account describes pretrained tabular foundation models (TFMs) remains unclear. Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map. We propose two certificates that depend only on predictions at standardized labels and can reject two distinct explanations: fixed-weight prediction and sums of independent nonlinear label transformations. Across the five public TFMs that we evaluate, our certificates show that changing one context label alters how other labels influence the prediction, a behavior we call joint processing. We further find that joint processing emerges with training and that attention scores carry most of the measured interaction. Together, these findings motivate TFM explanations that account for how context labels change the influence of individual examples.

1 INTRODUCTION

The paper asks whether pretrained tabular foundation models use context labels only as values combined by fixed weights, or whether one label changes another’s influence. It develops certificates that avoid misleading derivative evidence from label standardization and finds joint processing across the evaluated models.

1 INTRODUCTION

  • Fixed-weight regression and kernel-smoothing accounts treat features as determining example weights and labels as supplying the values combined.The paper uses this account as a starting point for asking how pretrained TFMs use context labels.Source: §1 · p. 1
  • Ordinary derivatives can suggest label interaction even when predictions on every standardized input match a fixed-weight map, because derivatives also reflect behavior off the standardized-label set.A term that vanishes on that set can change derivatives without changing valid predictions.Source: §1 · p. 2
  • Two certificates distinguish fixed-weight prediction, independent nonlinear label transformations, and joint processing, in which one label changes another’s influence.A nonzero first certificate excludes fixed-weight prediction; a nonzero second excludes both fixed weights and independent transformations.Source: §1 · p. 2
  • Both certificates indicate joint processing in all tested models, while randomly initialized copies show far less cross-example interaction.The findings associate the emergence of joint processing with training rather than architecture alone.Source: §1 · p. 2
  • Holding attention scores fixed during differentiation removes most measured interaction while leaving predictions unchanged, implicating attention-score paths in the measured interaction.The intervention identifies derivative paths in the implemented computation, not whether those paths are necessary.Source: §1 · p. 2

2 RELATED WORK

Prior explanations connect in-context learning to fixed-weight regression or kernel operations, while other accounts let labels shape task or algorithm selection. The paper situates its question among work on internal computation and asks whether one label changes how another is used.

Fixed-weight accounts

  • Simple regression and kernel accounts describe predictions as weighted sums in which features set example weights and labels only supply values [von Oswald et al., 2023; Aky¨urek et al., 2023; Ahn et al., 2023; Zhang et al., 2024; Tsai et al., 2019; Collins et al., 2024].The paper tests whether pretrained TFMs follow this fixed-weight pattern.Source: §2 · p. 2
  • A kernel-form prediction need not imply fixed label use, because embeddings supplied to the kernel can already depend jointly on features and labels (Miftachov et al., 2026; Qu et al., 2025).The full computation, including representation construction and parameter choices such as bandwidth, must be examined.Source: §2 · p. 2

Learning a task from its labels

From internal computation to prediction

  • Prior TFM studies examine internal computation, prediction quality, or label dependence in language-model demonstrations, but leave open whether one label changes how another is used.The cited work includes layerwise representations, attention-head ablations, readout strategies, and benchmark studies.Source: §2 · p. 3

3 MEASURING HOW MODELS USE LABELS

The section defines label influence and develops tests for joint processing that depend only on predictions at standardized labels. It corrects for the geometry of the standardized-label manifold and distinguishes fixed-weight maps from independent nonlinear label transformations.

3 MEASURING HOW MODELS USE LABELS

  • The proposed tests distinguish fixed-weight prediction, one-label-at-a-time transformations, and joint processing while avoiding ambiguity from behavior outside the standardized-label set.The internal prediction is evaluated on standardized labels after wrapper-level standardization and before output conversion.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 FROM A LABEL’S VALUE TO ITS INFLUENCE

  • Label sensitivity is the derivative of the internal prediction with respect to one standardized context label; fixed-weight maps keep these sensitivities constant as labels change.A label-dependent sensitivity may reflect either an independent transformation or interaction with other labels.Source: §3.1 · pp. 3–4

3.2 THE STANDARDIZATION TRAP

  • Standardization confines inputs to a manifold, so ambient derivatives can change even when a model makes exactly the same predictions as a fixed-weight map on every valid input.A term that vanishes on the manifold can alter sensitivities and Hessians without affecting any prediction made there.Source: §3.2 · p. 5
  • The first certificate removes isotropic tangent curvature, which any smooth map agreeing with a fixed-weight map on the manifold can produce; nonzero AT excludes fixed-weight prediction there.The corrected statistic depends only on predictions on the manifold, regardless of how the function extends away from it.Source: §3.2 · p. 6
  • The second certificate measures corrected curvature outside the space produced by independent transformations of individual labels; nonzero Xsep certifies that labels interact.A zero result is not proof that the map belongs to either tested class.Source: §3.2 · p. 7
  • The reported shares normalize curvature measures whose absolute magnitudes vary with prediction and sensitivity scales across models and datasets.The Xsep share is the fraction of corrected curvature no one-label-at-a-time map can produce and depends only on predictions on the manifold.Source: §3.2 · p. 7

4 EXPERIMENTS AND RESULTS

Across five public TFMs, the certificates indicate joint label processing; comparisons with random networks and attention-score derivative blocking probe where that interaction comes from.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five public TFMs on four regression datasets with 256 context rows, yielding 20 model-dataset settings and three sampled contexts per setting.The four datasets include one synthetic smooth-regression control and OpenML kin8nm, pol, and superconduct; derivatives use a single estimator and eager attention.Source: §4.1 · pp. 7–8

4.2 TO WHAT EXTENT DO TFMS PROCESS LABELS JOINTLY?

  • The study tests fixed-weight, one-label-at-a-time, and joint-processing accounts by computing exact Hessians and applying two certificates across the 20 settings.Each certificate is zero whenever its corresponding account holds.Source: §4.2 · p. 8
  • Across four datasets, all five models show joint label processing: fixed-weight and one-label-at-a-time accounts fail, with Xsep averaging 63% and ranging from 23% to 92%.Table 2 summarizes certificate shares and attention-score derivative drops averaged over the four datasets.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 WHERE DO THE INTERACTIONS COME FROM?

  • On two datasets, trained models have Xsep of at least 29% for every query, while randomly initialized copies remain at most 3.4%.AT is nonzero in both trained and random networks, whereas normalized Xsep separates them.Source: §4.3 · p. 9
  • Blocking derivatives through attention scores reduces Xsep by 85% on average and corrected curvature norm AT by 81%, while predictions remain unchanged.The reductions are in norms, not accuracy losses, and do not establish that these paths are necessary for good predictions.Source: §4.3 · p. 9

5 DISCUSSION AND CONCLUSION

The certificates detect joint processing in evaluated TFMs, while fixed-weight maps can still approximate predictions. The interaction’s source and predictive necessity remain unresolved.

5 DISCUSSION AND CONCLUSION

  • Both certificates are nonzero in all five evaluated models, and 23% to 92% of curvature departing from fixed weights comes from cross-label interaction.The interaction is weaker in randomly initialized copies, while blocking attention-score derivatives removes most measured interaction without changing predictions.Source: §5 · p. 9
  • The certificates detect interaction but cannot distinguish task inference, label-based local reweighting, or label-dependent example representations as its computation.A fitted fixed-weight map nevertheless explains a median 72% of prediction variation when labels are redrawn from the same task.Source: §5 · p. 9
  • Derivative tests establish interaction’s presence, not its necessity for prediction; determining predictive benefit requires a forward intervention and accuracy measurement.Blocking attention-score derivatives alone leaves predictions unchanged.Source: §5 · p. 9

A PROOFS AND COMPUTATION OF THE CERTIFICATES

The appendix proves the two certificate lemmas and supplies computations and calibration for the resulting statistics.

A PROOFS AND COMPUTATION OF THE CERTIFICATES

A.1 PROOF OF LEMMA 1

The proof shows that off-manifold differences can alter second derivatives without changing predictions on the standardization manifold, but their tangent curvature is restricted.

A.1 PROOF OF LEMMA 1

  • A map matching a fixed-weight map on manifold M can differ in second derivatives, but its tangent curvature is identical across tangent directions.The proof represents the difference using the two functions whose common zero set is M; after tangent projection, only an isotropic term remains.Source: §A.1 · p. 13
  • Because the two functions defining M vanish there, their smooth-multiplier representation expresses how the map may differ away from M while agreeing on it.The gradients of the defining functions are independent on M when r > 0.Source: §A.1 · p. 13
  • The tangent projector removes the additional Hessian terms along the two normal directions, leaving PT HPT = 2q PT.The normal directions are spanned by 1 and standardized labels ˜y.Source: §A.1 · p. 13
  • The traceless projected Hessian depends only on predictions on M, whereas its isotropic part cannot be identified from on-manifold behavior alone.Subtracting the tangent trace removes the isotropic correction and yields the traceless intrinsic Hessian.Source: §A.1 · p. 13

A.2 PROOF OF LEMMA 2

The proof shows that one-label-at-a-time nonlinear maps have zero Xsep after the standardization correction. It also describes a small projection solve and corrected sensitivity tests, while noting that zero certificate values do not prove class membership.

A.2 PROOF OF LEMMA 2

  • For affine-plus-separable maps, the corrected tangent curvature lies in S, so the residual Xsep is zero.The difference between maps that agree on the standardized-label manifold contributes no corrected tangent curvature, making the result independent of the off-manifold extension.Source: §A.2 · pp. 13–14
  • A zero certificate is consistent with its target map class but does not establish membership in that class.An outside-class map could also produce zero on the tested labels.Source: §A.2 · p. 14

A.3 COMPUTING THE PROJECTION ONTO S

  • The projection onto S reduces to a small linear solve using one basis operator per label.The Gram system Gc = β uses the Moore–Penrose pseudoinverse when the basis is rank-deficient; the solve remains consistent.Source: §A.3 · p. 14

A.4 THE CORRECTED PERTURBATION TEST

  • The corrected perturbation tests project out directions induced by standardization before comparing sensitivities across label permutations.One statistic averages the remaining displacement after projecting out span{1, ỹ, Pπk ỹ}; another tests whether one fixed weight vector fits the projected sensitivities across permutations.Source: §A.4 · p. 15

A.5 CALIBRATION ON MAPS WITH KNOWN BEHAVIOR

Calibration on controls with known behavior shows that the corrected statistics distinguish affine, separable, and coupled label processing. The appendix also documents the experimental scope and the model readouts and computation paths used for derivatives.

A.5 CALIBRATION ON MAPS WITH KNOWN BEHAVIOR

  • AT is zero on the affine control, while Xsep is zero on both affine and separable controls and nonzero on the coupled control.The separable control has nonzero AT, confirming that AT alone does not distinguish separable processing from coupled processing.Source: §A.5 · p. 16

B EXPERIMENTAL DETAILS

  • The appendix documents the models, datasets, repetitions, and Hessian computations behind the paper’s results.It also describes how each model first receives context labels.Source: B EXPERIMENTAL DETAILS · p. 16

B.1 MODELS AND WHERE THE LABELS ENTER

  • Derivative measurements target one internal readout per model rather than the package’s combined default prediction.The five models differ in where context labels enter; Hessians use eager softmax attention, whose first derivatives closely match native kernels.Source: §B.1 · pp. 16–17, Table 4 · p. 17

B.2 DATASETS AND PROTOCOL

The protocol specifies the dataset grids and repetitions, while noting that possible pretraining overlap limits how TABDPT results on real datasets support generalization independent of pretraining. Eager-attention Hessians are validated against native-kernel finite differences.

B.2 DATASETS AND PROTOCOL

  • The main experiment uses synthetic mi, kin8nm, pol, and superconduct, with broader Core 8 and Core 7 dataset grids for other experiments.Table 5 records the grid and replication for each experiment.Source: §B.2 · pp. 17–18, Table 5 · p. 18
  • Unknown pretraining overlap limits how TABDPT’s results on real datasets can be read as evidence of generalization independent of pretraining.The evaluated checkpoint’s pretraining dataset list is not released, so overlap with the paper’s real datasets cannot be checked.Source: §B.2 · p. 18

B.3 COMPUTING AND VALIDATING THE HESSIANS

  • Exact Hessians are assembled one column at a time, while probe-based estimates are used for analyses that do not require the full matrix.The eager-attention path is checked against finite differences of gradients from native fused kernels.Source: §B.3 · pp. 19–20
  • At n = 256, all ten tested model–dataset settings meet the Hessian-validation thresholds.Median cosine similarities round to 1.0000, the highest median relative error is 0.0034, and every matched-column fraction reaches 1.00.Source: §B.3 · p. 20, Table 6 · p. 20

C.1 CERTIFICATES IN EVERY SETTING

The appendix checks the certificates across settings and tests whether non-affinity persists in complete package predictions and trained-versus-random comparisons.

C.1 CERTIFICATES IN EVERY SETTING

  • Both first-order statistics reject the affine null on all 32 core settings, with LS residuals from 0.722 to 0.925.This shows the departure from fixed weights does not depend on the second-order instrument.Source: §C.1 · pp. 20–21
  • The exact normalized X̄sep ranges from 0.239 to 0.916 across 20 settings, with a mean of 0.63.The exact X̄AT ranges from 0.952 to 0.997 and closely matches the probe estimate on settings where both were run.Source: §C.1 · p. 21
  • On the two TABPFN models, all eight pre-head settings retain nonzero X̄sep values of 0.50 to 0.97.Their ratio to mean-readout values is 0.95±0.13, placing most measured interaction upstream of the bucket head on these settings.Source: §C.1 · p. 21
  • The full internal readout has a median setting-level relative error of 0.046, with 15 of 16 settings below 0.5.TABDPT on pol is the sole exception, at 0.567.Source: §C.1 · p. 22
  • Across tested context sizes, RT varies by at most a factor of 2.4 without a consistent trend, while X̄sep remains present at every tested size.For the two models and datasets tested with X̄sep, it ranges from 0.67 to 0.91 at n = 512.Source: §C.1 · p. 22

C.2 THE DEFAULT PREDICTIONS ARE NOT AFFINE IN THE LABELS

  • In 12 package-level context settings, measured interaction exceeds calibrated control floors by 44 to 16,900 times.The two kin8nm contexts miss the prespecified floor requirement, but still exceed their floors by 44 to 279 times.Source: §C.2 · p. 23
  • These package-level checks establish non-affinity, but do not rule out independent one-label-at-a-time transformations; Xsep remains scoped to internal single-estimator paths.Source: §C.2 · p. 23

C.3 RANDOM COPIES OF THE TRAINED MODELS

  • Every trained-model query has X̄sep ≥29.5%, with means of 44% to 98%, while random-copy means stay at most 1.7%.Matching attention entropy raises X̄sep only to 0.16 for TABICL-V2 and 0.004 for TABPFN-V2.6; the controls support a training-associated change without isolating it as training’s only change.Source: §C.3 · p. 24

C.4 PATHS THROUGH THE ATTENTION SCORES

Blocking derivatives through attention scores leaves predictions unchanged but removes most of the measured interaction. A separate intervention that replaces cached attention scores changes predictions, though its results do not establish a causal effect of the scores.

C.4 PATHS THROUGH THE ATTENTION SCORES

  • Blocking derivatives through attention scores preserves the prediction while isolating curvature carried by those paths in the evaluated computational graph.For TABICL-V2, the replay passes derivatives through both label sites; TABPFN-V3 and TABFM are differentiated through their full forward passes.Source: §C.4 · p. 24
  • Blocked attention-score derivative paths account for 0.93–1.00 of full curvature-operator norm across the five models; 0.09–0.32 survives the intervention.The two components have cosine similarities from −0.12 to 0.28, indicating a small remainder and limited cancellation.Source: §C.4 · p. 24
  • The derivative-path norm comparison is graph-dependent and does not test a forward map that freezes attention scores at base-label values.That test would require the frozen map’s own gradient and a comparison against its own prediction differences.Source: §C.4 · pp. 24–25
  • With identical labels, the cached-score hybrid reproduces the native internal pass; zero-label scores shift predictions by 0.60–0.97 standard deviations.Because later cached scores are applied to different hidden states, this agreement does not establish that varying cached scores isolates their causal effect.Source: §C.4 · p. 25
  • Cached-score replacement worsens NMSE on 13 of 16 settings, with a median patched-to-native error ratio of 5.12; interpolation increases displacement on 15 of 16.A single-call replacement can displace predictions more than replacement at every call, indicating cancellation across calls rather than additive contributions.Source: §C.4 · p. 25

C.5 CONTROLLED MODELS

Controlled transformers show that label interactions arise during training and can persist without LayerNorm. A model can also use labels effectively under contamination without allowing them to affect attention scores, although these constructed models do not identify a necessary pathway in deployed models.

A controlled transformer on a Gaussian-process task

  • The controlled transformer’s permutation response rises from about 0.35 at initialization to 0.85 ± 0.50 under Gaussian noise and 1.38 ± 0.29 under contamination.Its final off-diagonal Hessian ratio is 0.44–0.50; the Gaussian-process task has a label-linear Bayes-optimal predictor, and the small model supports no scale claim.Source: §C.5 · p. 26

LayerNorm is not necessary

  • Without LayerNorm, the model retains an off-diagonal ratio of 0.41 and loses 97% of that norm when attention-score derivatives are blocked.Its loss is 0.028 versus 0.026 with LayerNorm; blocking LayerNorm statistics alone removes 85%, reflecting overlap after label mixing.Source: §C.5 · p. 26

Useful label processing without label-dependent attention scores

  • Under symmetric contamination, the score-blocked model reaches MSE 0.0215 ± 0.0008, within 9% of token injection’s 0.0198 ± 0.0009.The label-linear control reaches 0.0388 ± 0.0003, while the score-blocked model’s non-separable curvature share is 0.085 ± 0.145 versus 0.243 ± 0.080 for token injection.Source: §C.5 · p. 26
  • Across the five-seed extension, score-blocked and label-linear losses are 5% and 91% above token injection under symmetric contamination.A slim score-blocked model with 107,993 parameters reaches MSE 0.0243 ± 0.0016, within 20% of token injection; one-sided contamination preserves the ordering, with the models 8% and 142% above token injection.Source: §C.5 · p. 26
  • The attention-score curvature share for value-only injection is below token injection under symmetric contamination (0.829 versus 0.914), but above it under one-sided contamination (0.986 versus 0.931).These constructed models establish a useful alternative to label-dependent attention scores, not a necessary pathway in deployed models.Source: §C.5 · p. 27

C.6 HOW CLOSE A FIXED-WEIGHT MAP COMES

The section compares fixed-weight surrogates with tuned smoothers, task-preserving label draws, and a constructed label-adaptive map. Task-preserving draws improve surrogate fidelity over label permutations, while the constructed map fits trained models poorly.

An observed-feature comparator

  • On 28 settings, the best tuned label-linear comparator comes within 10% of model accuracy on 9, with a median held-out NMSE gap of +0.066; a ridge surrogate loses to it on 27.The comparator selects the best of four methods tuned on a held-out fifth of context rows, favoring the comparator.Source: §C.6 · p. 27

Task-preserving approximation

  • Median held-out R2 is 0.723 under task-preserving resampling versus 0.48 under label permutations; the label-linear Nadaraya–Watson control reaches R2 = 1.0000 on all 12 settings.The comparison uses six queries per setting; task-preserving draws use fresh synthetic-task noise and bootstrapped GBDT residuals on real data.Source: §C.6 · p. 27, Table 10 · p. 28
  • Residual resampling removes feature-dependent noise structure; wild-bootstrap checks on three real datasets yield median R2 values of 0.708, 0.752, and 0.771 for i.i.d., Rademacher, and Mammen resampling.These resampling schemes proxy the task’s label distribution and differ from the fixed-moment manifold used by the certificates; permutation fidelity may be less comparable for TABDPT.Source: §C.6 · p. 27
  • A local Taylor-remainder check was numerically unstable on 2 of 12 settings and therefore supports no claim about the remainder curve.Source: §C.6 · p. 27

A constructed local model

  • Choosing h and λ reproduces the aggregate response profile on three synthetic tasks with 3, 8, and 16 indices at n = 96.The map combines a Gaussian feature kernel with an exponential gain based on each row’s deviation from the mean context label, followed by a local-linear readout.Source: §C.6 · p. 27
  • The constructed local model has median held-out R2 of 0.02 across nine settings, versus approximately one for maps already in this class.Repeating its reweighting does not improve fit: the selected fit uses a single round on 8 of 9 settings.Source: §C.6 · p. 28

Approximation fidelity and predictive cost

  • Replacing model predictions with permutation-fitted surrogates increases error in all 21 settings, but the correlation with predictive advantage is −0.03, so this is not a causal ablation.TABDPT’s permutation fidelity is protocol-sensitive: at n = 96, its R2 is −0.92 and −3.18 on synthetic mi and kin8nm, versus 0.05 and 0.03 under Table 10’s protocol.Source: Figure 3 · p. 29

Related papers

Loading 2610.08314v1…