TICDA: Tabular In-Context Data Attribution

1 INTRODUCTION

• TFMs condition predictions on labeled demonstrations, yet the influence of individual demonstrations remains poorly understood.Contexts assembled from available labeled data may include mislabeled, redundant, or low-quality examples that degrade performance or add computational overhead.Source: §1 · p. 1
• Gradient-based attribution methods lack a natural parameter-update target for TFMs, while resampling methods require a combinatorial number of full-context forward passes.TICDA is introduced as an approach to estimate every demonstration’s influence at negligible cost.Source: §1 · p. 2
• TICDA uses linear surrogates on TFM latent embeddings to measure demonstration influence in a single forward pass.The paper evaluates its use for labeling-error detection, context curation, cross-TFM transfer, and active learning.Source: §1 · p. 2

2 TICDA: A METHOD TO COMPUTE FIXED-REPRESENTATION INFLUENCES

• TICDA defines a demonstration’s leave-one-out influence as the change in loss on a labeled query when that demonstration is removed.Computing this directly for every demonstration would require one forward pass per deletion.Source: §2 · p. 2

2.1 ATTRIBUTION SURROGATE

• A ridge linear surrogate fitted on TFM row representations supplies the parameters needed to estimate demonstration influence.The surrogate uses the observed labels, and its residuals measure discrepancies between predicted class scores and one-hot labels.Source: §2.1 · p. 3

2.2 FIXED REPRESENTATIONS SIMPLIFICATION

• Holding representations fixed when a demonstration is removed avoids re-encoding the context for each deletion, so one forward pass serves all demonstrations.The fixed-representation approximation is supported by representation differences of about 0.5% of the norm at 4,096 demonstrations, decreasing as context size increases.Source: §2.2 · p. 4

2.3 DEMONSTRATION UPWEIGHTING INFLUENCE

• Closed-form upweighting influence scores indicate whether increasing a demonstration’s weight reduces or increases query loss.Self-influence uses a demonstration’s own representation and label; query-influence averages its influence over a set of queries.Source: §2.3 · p. 4
• Large self-influence identifies demonstrations whose observed labels are not supported by the rest of the context, while negative mean query-influence can guide context removal.The paper proposes these scores for labeling-error detection and curation using a labeled validation set, respectively.Source: §2.3 · p. 4

3.1 EXPERIMENTAL SETUP

• Experiments use 38 TabArena classification datasets, shared context and test splits, and contexts capped at 4,096 demonstrations; the backbones are frozen TabICLv2 and TabDPT 1.2.Attributions from those backbones are also applied unchanged to curate TabPFN-3 and TabPFN-3.5 contexts.Source: §3.1 · p. 5
• The evaluation measures LOO faithfulness, label-error AUC, runtime, curation accuracy, transfer, and active-learning AULC; the ridge coefficient is fixed at λ = 10 across datasets and backbones.For label-error detection, 20% of context labels are corrupted; curation ranks demonstrations by mean validation-set influence.Source: §3.1 · p. 5

3.2 EXPERIMENTAL RESULTS

• On TabICL, TICDA reaches the best faithfulness (0.67) and labeling-error AUC (0.89); on TabDPT, it is second-best on faithfulness (0.79) and AUC (0.88).No competitor is consistent across both backbones: DemoShapley degrades on TabICL, while IG and DETAIL perform poorly on both.Source: §3.2 · pp. 5–6, Table 1 · p. 6
• TICDA and DETAIL run in 1.88–2.74 s, two to three orders of magnitude faster than LOO, DemoShapley, and IG; only TICDA combines this low cost with top-tier results.Figure 2 reports computation time on a log scale.Source: §3.2 · p. 6, Figure 2 · p. 6
• With full TabICL representations, self-influence raises faithfulness from 0.33 to 0.67 and AUC from 0.86 to 0.89 versus query-influence; reducing dimensions leaves both metrics unchanged.The dimension reduction slightly improves computation time.Source: §3.2 · p. 6, Table 2 · p. 6
• At 40% corruption, TICDA and TICDA-IT peak around 40–45% removal, so discarding up to half the demonstrations outperforms the full context.Across both corruption settings, TICDA-IT keeps balanced accuracy at or above the full-context level; it consistently outperforms TICDA, especially at 20% corruption.Source: §3.2 · pp. 6–7, Figure 3 · p. 7
• At 50% removal, transferred TICDA scores from either TabICLv2 or TabDPT yield approximately 64% balanced accuracy on unseen targets, over four points above full-context baselines.Transfer is robust across source models; TabDPT performs comparably to the stronger TabICLv2.Source: §3.2 · p. 7, Table 3 · p. 7

3.3 TDA FOR TFM ACTIVE LEARNING

• In active learning, TICDA achieves the best average AULC across the full budget (68.35%) and the data-constrained regime with ≤512 labels (69.02%), outperforming entropy sampling and random baselines.Source: §3.3 · p. 8

• TDA groups into resampling-based, gradient-based, and training-dynamics methods, whose forms depend on whether points shape weights or influence inference directly.Source: §4 · p. 8
• Resampling- and gradient-based attribution are the two method families developed for both classical training and ICL.Source: §4 · p. 9
• DemoShapley values LLM demonstrations across resampled prompt permutations, but its combinatorial subsets and forward passes make it unaffordable for long TFM contexts (Xie et al., 2025).The passage notes that it targets few-shot LLM settings.Source: §4 · p. 9
• Influence functions lack a natural transfer to ICL demonstrations because those demonstrations update no parameters; DETAIL instead applies them to a ridge probe on hidden states (Koh & Liang, 2017; Zhou et al., 2024).The methods differ in probe-fitting regime, differentiated perturbation, and treatment of contextual representations under one forward pass.Source: §4 · p. 9

• TICDA assumes removing one demonstration leaves the remaining-row and query representations unchanged, and its exhaustive-LOO faithfulness comparison covers contexts of at most 4,096 demonstrations.Longer contexts remain untested against that reference; extensions beyond the approximation and to regression and multimodal settings are future directions.Source: §5 · p. 9
• TICDA fits a ridge surrogate on frozen row representations and holds them fixed under demonstration removal to yield closed-form influence scores at negligible cost.Source: §5 · p. 9
• Across 38 TabArena datasets and two backbones, TICDA pairs top-tier faithfulness and labeling-error detection with runtime well below LOO, DemoShapley, and IG.Its context curation preserves or improves performance while discarding up to half of demonstrations, and attribution scores transfer across TFMs.Source: §5 · p. 10

AI USE STATEMENT

• Generative AI tools helped polish the paper, generate tables and figures, review for typos and inconsistencies, and accelerate code production; the authors take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A SCIENTIFIC LIBRARIES

• The work used PyTorch, HuggingFace Transformers, and sklearn, and the authors state that code will be made public upon acceptance.Source: A SCIENTIFIC LIBRARIES · p. 13

• The appendix first proves the closed form of the influence score in Equation 8.Source: B DERIVATION OF THE INFLUENCE SCORES · p. 13
• It gives the exact effect of deleting a demonstration when representations are held fixed, clarifying what a first-order score leaves out.Source: B DERIVATION OF THE INFLUENCE SCORES · p. 13
• It states what the fixed-representation simplification neglects and measures how much representations change when a demonstration is removed.Source: B DERIVATION OF THE INFLUENCE SCORES · p. 13
• The notation defines the row-loss gradient at bB and the Frobenius inner product as tr(U^⊤V).Source: B DERIVATION OF THE INFLUENCE SCORES · p. 13

B.1 FIRST-ORDER INFLUENCE

• The first-order score follows from differentiating the ridge fit with respect to a demonstration’s weight and applying the query-loss chain rule; it requires no TFM-parameter Hessian.The derivation is closed form for the ridge surrogate.Source: §B.1 · p. 13
• Self-influence combines ridge leverage with the squared residual, and is largest for demonstrations represented in directions few others occupy.Mean leverage is below d/n, making its multiplier small on average when tabular contexts have many more demonstrations than dimensions.Source: §B.1 · pp. 13–14

B.2 EXACT DELETION AT FIXED REPRESENTATIONS

• For a query, exact fixed-representation deletion differs from the first-order score by leverage-dependent corrections that vanish as leverage shrinks.The exact effect reuses the fitted inverse Hessian, residuals, and leverages, so no refit is needed.Source: §B.2 · p. 14
• For a demonstration’s own loss, the exact deletion effect differs from self-influence by a leverage-only factor that approaches one at low leverage.The first-order score is accurate for most demonstrations when context size exceeds the surrogate dimension, but not when demonstrations are fewer than dimensions.Source: §B.2 · p. 14

B.3 WHAT THE FIXED-REPRESENTATION SIMPLIFICATION NEGLECTS

• When representations respond to a perturbation, the query representation and every row of the design can change; the simplification retains only the coefficient response.Recomputed deletion re-encodes the reduced context and refits the surrogate, requiring one TFM forward pass per deleted demonstration, a cost TICDA avoids.Source: §B.3 · pp. 14–15

C RELATION TO DETAIL: THE ROLE OF THE RIDGE PENALTY

• DETAIL’s added gradient term is common across demonstrations, since each attributed gradient includes the opposite sum of the other demonstrations’ gradients.The appendix illustrates that with 1,000 demonstrations, two such sums share 998 components.Source: C RELATION TO DETAIL: THE ROLE OF THE… · pp. 15–16
• TICDA removes a demonstration’s squared-error loss while retaining the ridge penalty; DETAIL removes its loss together with the regularizer.Thus the methods estimate different deletion effects on the surrogate.Source: C RELATION TO DETAIL: THE ROLE OF THE… · p. 15
• DETAIL’s self-score adds a demonstration-dependent cross term to TICDA’s self-influence, and that term can change the ranking.The added constant term is identical across demonstrations and does not affect ranking.Source: C RELATION TO DETAIL: THE ROLE OF THE… · p. 16

C.1 EMPIRICAL CHECK OF THE FIXED-REPRESENTATION SIMPLIFICATION

• About 3% of representation norm at 512 demonstrations and about 0.5% at 4,096, the median change roughly halves as context size doubles, close to a 1/n decrease.The pattern holds across four datasets for remaining demonstrations and queries.Source: Table 5 · p. 17, Figure 5 · p. 17, §C.1 · p. 17
• Deleting a mislabeled demonstration produces representation changes 1.6 to 1.8 times larger than deleting other demonstrations at every tested context size.The contexts contain 20% corrupted labels, and the experiment tests sizes from 512 to 8,192 demonstrations.Source: §C.1 · pp. 16–17
• The fixed-representation assumption is an approximation, but its representation-level error becomes small in the large contexts TICDA is designed for.Representations do change after deletion; the experiment uses TabICL on four TabArena datasets.Source: §C.1 · pp. 16–17

D IMPLEMENTATION DETAILS

• Integrated gradients use 32 steps to approximate gradients along the straight line between the instance being explained and the baselines.Source: D IMPLEMENTATION DETAILS · p. 17
• DETAIL uses the hyperparameters provided by its paper, with regularization set to 10.Source: D IMPLEMENTATION DETAILS · p. 17

E ADDITIONAL EXPERIMENTS

• The section examines λ's impact on TICDA performance in labeling-error detection.Source: E ADDITIONAL EXPERIMENTS · p. 18
• The section examines λ's impact on TICDA performance in context curation.Source: E ADDITIONAL EXPERIMENTS · p. 18
• Figure 6 presents TICDA's sensitivity to λ regularization.Source: Figure 6 · p. 18