Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Impact, Detection, and Mitigation

1 Introduction

• Relational joins can place post-outcome information several tables from the target, while missing historical records can prevent temporal filtering from removing it.A cited customer-prediction example involved archived websites referencing products bought only after the prediction date.Source: §1 · p. 1
• Target association alone does not establish model use, and redundant leakers can mask one another in column-removal tests.The paper motivates Integrated Gradients (IG) as a joint screening method given limitations of provenance, permutation importance, SHAP, attention, and intermediate representations.Source: §1 · p. 2
• The study compares clean and leaky support while keeping queries unchanged, then ranks influential columns with IG and tests performance after flagged-column removal.Ground-truth leaker identities construct the mismatch but are not supplied to the detector.Source: §1 · p. 2
• Support-set target leakage occurs when target-derived columns are available in labeled support contexts but absent from clean queries.The study distinguishes this setting from leakage during dataset construction, temporal splitting, or representation learning.Source: §1 · p. 2
• The study uses a 20-column stress test spanning different proxy types, modalities, semantic transparency, and relational distances, together with matched zero-, one-, and two-hop ablations.Source: §1 · p. 2

2.1 Problem Formulation

• Leaker values are generated from associated labeled targets and may be deterministic, noisy, partially observed, numerical, categorical, or text-valued.The contaminated-support comparison keeps model parameters fixed and leaves leaker values unavailable in query contexts.Source: §2.1 · p. 2

2.2 Synthetic Leaker Construction

• The stress test injects 20 target-derived columns: 14 target-table leakers and three each at one and two relational hops.Matched hop ablations use the same three proxy designs at each distance to vary placement comparably.Source: §2.2 · p. 3

2.3 Relational ICL Pipeline

• A frozen Griffin encoder [2] produces target-row embeddings for a pretrained TabPFN-based ICL head, with leakers present only in support data.The 512-dimensional target-row representation follows two relational message-passing layers.Source: §2.3 · p. 3

2.4 Attribution-Based Leaker Detection

• IG scores all relational columns through a Ridge surrogate on validation embeddings, serving as screening rather than a faithful explanation of the TabPFN head.Joint scoring avoids a separate model evaluation for every candidate column.Source: §2.4 · p. 3

2.5 Mitigation and Evaluation

• Mitigation removes flagged columns from support and query, recomputes embeddings, and measures TabPFN performance against clean and leaky-support settings.The proof-of-concept removal includes false positives and is not presented as a final automatic mitigation procedure.Source: §2.5 · p. 3

3 Experimental Setup

• Five classification tasks from rel-avito, rel-event, and rel-trial use training and validation examples as support and test examples as queries.The Griffin encoder remains frozen, and a pretrained TabPFN classifier is fitted on support embeddings for each setting.Source: §3 · p. 3
• Table 1 reports dataset-level averages across leakage ablations for clean, leaky, and post-removal ICL macro-F1 and detector F1.Per-task results are reported in Appendix E.Source: Table 1 · p. 4
• Paired runs use the same random seed, with a 13-task Bimodal encoder robustness check reported separately.The Bimodal check appears in Appendix G.Source: §3 · p. 4

4 Results and Discussion

• Matched zero-hop leakers caused the largest decreases in rel-event and rel-trial, while rel-avito was unchanged; the Bimodal check also had mean ΔF1 of −0.115, −0.002, and +0.001 at zero, one, and two hops.Rel-trial zero-hop macro-F1 fell from 0.500 to 0.435; rel-avito's low clean macro-F1 means unchanged results do not establish robustness.Source: §4 · p. 4
• Zero-hop AUROC exceeded 0.9 on all five tasks, but no rel-avito column passed the heuristic cutoff, yielding zero recall and detection F1.Leakers could rank above many non-leakers without separating enough to cross the threshold; similar signals may divide attribution across columns.Source: §4 · p. 4
• Post-detection removal recovered part of the largest rel-event full-stress-test losses, but 15 of 20 leakers remained after removal.On rel-trial zero-hop, two of three leakers were removed without false positives, while the remaining deterministic proxy ranked sixth and macro-F1 stayed near the leaky result.Source: §4 · p. 4
• For rel-event one- and two-hop settings, leakage caused no degradation, while removal included many legitimate columns and reduced performance.Missed leakers can preserve the mismatch, while false positives can remove predictive information.Source: §4 · p. 5

• Zero-hop leakers produced the clearest performance losses across five held-out tasks, indicating relational distance matters in this Griffin-TabPFN pipeline.Source: §5 · p. 5
• IG consistently ranked zero-hop leakers highly across datasets, while thresholded removal remained sensitive to calibration, false positives, and missed leakers.Source: §5 · p. 5
• The findings establish support-set-only target-derived features as a practical concern for relational ICL and motivate more reliable detection and mitigation methods.Source: §5 · p. 5

A Experimental Protocol

• Synthetic leakers are injected only into support data; they are physically removed from query schemas before query embeddings are computed.Source: A Experimental Protocol · p. 6
• After IG detection, all flagged columns, including false positives, are removed from support and query schemas, embeddings are recomputed, and TabPFN inference is repeated.Source: A Experimental Protocol · p. 6
• Clean baseline, matched three-column hop ablations, and the full 20-column stress test define the candidate settings across binary classification tasks.Source: Table 2 · p. 7

B.1 Pretraining and Held-Out Databases

• The evaluation databases rel-avito, rel-event, and rel-trial were excluded from Griffin pretraining and reserved for downstream relational ICL experiments.Source: §B.1 · p. 7, Table 3 · p. 7

B.2 Model and Inference Configuration

• The same frozen Griffin checkpoint and TabPFN configuration are used for clean, leaky, and post-removal conditions, with embeddings recomputed after columns are removed.Source: §B.2 · p. 7

C Synthetic-Leaker Construction

• The stress test injects 20 columns: 14 on target tables, three on one-hop tables, and three on two-hop tables, spanning different signal and semantic properties.Source: C Synthetic-Leaker Construction · p. 8, Table 4 · p. 8
• Proxies vary in noise, coverage, and encoding: partial coverage samples 50% of labeled rows, while opaque categorical values use deterministic class-specific hashes.Source: C Synthetic-Leaker Construction · p. 8
• Matched proxy designs isolate numerical target signal, column-name semantics, and encoded-value semantics across zero, one, and two hops.Source: C Synthetic-Leaker Construction · p. 8
• Ground-truth leaker identities support controlled ablations and detection metrics but are withheld from the detector and column-removal selection.Source: C Synthetic-Leaker Construction · p. 8

D Leaker-Type and Relational-Distance Effects

• The per-leaker Griffin-TabPFN evaluation added one leaker to support while retaining original query columns; decision-tree checks exposed leakers at both training and test time.All leakers together and each of three matched zero-hop designs achieved macro-F1 1.0 across five tasks in that sanity check.Source: D Leaker-Type and Relational-Distance… · p. 9
• Excluding rel-avito, 11 unmatched zero-hop leakers reduced macro-F1 in 27 of 33 conditions, with mean ∆F1 of −0.120, showing that exact copies and full coverage are unnecessary.Partial-coverage, 90%-fidelity, coarse categorical, rank/ordinal, and jittered proxies reduced F1 on all three remaining tasks.Source: D Leaker-Type and Relational-Distance… · p. 9
• Effects varied by task: every zero-hop leaker reduced F1 on user-ignore, while proxy effects differed on user-repeat and study-outcome, with no consistent ordering by noise, fidelity, or semantic transparency.On user-repeat, semantic and opaque text proxies caused the largest drops; on study-outcome, noisy and partial-coverage numerical proxies were among the most harmful.Source: D Leaker-Type and Relational-Distance… · p. 9
• Across 70 zero-hop task-leaker conditions, mean ∆F1 was −0.078: F1 decreased in 36, increased in six, and was unchanged in 28; matched one- and two-hop means were +0.001 and −0.002.All 28 unchanged conditions were from the two rel-avito tasks.Source: Figure 2 · p. 9, D Leaker-Type and Relational-Distance… · p. 9
• Among matched zero-hop designs, semantic-name with opaque-value had mean ∆F1 −0.138, followed by non-semantic-name with semantic-value at −0.097 and deterministic numerical at −0.070.This ordering did not hold at one or two hops, and differing encodings prevent isolating semantic transparency.Source: D Leaker-Type and Relational-Distance… · p. 10
• All three matched designs had larger effects on the target table than at higher hops, but the tested one- and two-hop results do not show that higher-hop leakage is generally harmless.Effects may depend on connectivity, aggregation, neighborhood sampling, and encoder checkpoint; only the frozen Griffin-TabPFN pipeline was tested.Source: D Leaker-Type and Relational-Distance… · p. 10

E Per-Task Prediction and Detection Results

• On rel-avito, leakers do not change predictions; rel-event shows strong zero-hop harm with limited recovery, while rel-trial’s oracle removal restores performance to the clean baseline.Higher-hop removal can reduce performance even when injected leakers had little effect.Source: E Per-Task Prediction and Detection… · p. 10
• Rel-avito’s matched zero-hop leakers rank highly (AUROC = 0.911) but none pass the cutoff; rel-event flags 16 original columns alongside its three matched leakers.For rel-trial, two text leakers are flagged without false positives, while the numerical proxy remains below threshold.Source: E Per-Task Prediction and Detection… · p. 10
• IG ranking and thresholding do not ensure recovery: false positives can remove useful features, while missed leakers can leave harmful signal behind.The section concludes that rel-avito does not justify lowering the threshold, rel-event needs better calibration, and rel-trial needs direct removal tests of high-ranked unflagged columns.Source: E Per-Task Prediction and Detection… · p. 10

F Per-Column Attribution Rankings and Thresholding

• IG aggregates absolute attribution across embedding dimensions and combines each column’s overall mean with its top-10%-occurrence mean using a geometric mean.Attributions are computed through the frozen Griffin encoder on validation batches, with the neighborhood and support examples held fixed along the path.Source: F Per-Column Attribution Rankings and… · p. 12
• Columns are group-standardized after a log transform and flagged using a normal-tail cutoff, but the resulting scores are not calibrated null p-values or formal family-wise error control.Seventeen original columns cross the cutoff in clean rel-event runs, so the rule is treated as an initial heuristic.Source: F Per-Column Attribution Rankings and… · p. 12
• The rel-avito, rel-event, and rel-trial examples show that strong rankings can diverge from thresholded detection, supporting IG as screening rather than automatic removal.On rel-trial, the high-ranked numerical proxy remains below the cutoff.Source: F Per-Column Attribution Rankings and… · p. 12

G Bimodal Encoder Robustness Check

• The check replaces Griffin’s gated relation-wise mean/max message passing with Bimodal’s dual-attention block and learned fusion, while keeping the TabPFN head and evaluation protocol unchanged.Bimodal combines relation-conditioned local attention with type-agnostic global attention.Source: G Bimodal Encoder Robustness Check · p. 12
• Table 8 reports ∆F1 relative to the corresponding clean-support baseline.Source: Table 8 · p. 13
• Across 13 tasks, mean ∆F1 is −0.115 for zero-hop, −0.002 for one-hop, and +0.001 for two-hop leakage with the Bimodal encoder.The mean effect is again larger at zero-hop than at one- or two-hops, while individual task effects remain variable.Source: G Bimodal Encoder Robustness Check · pp. 12–13, Table 8 · p. 13

H Limitations and Future Work

• The primary study uses synthetic leakers, one TabPFN-based head, and five binary tasks; a 13-task Bimodal check extends encoder and task coverage.Source: H Limitations and Future Work · p. 13
• Future evaluation will include naturally occurring leakage, additional architectures and neighborhood settings, different pretraining data, RelBench v2, and multiclass and regression tasks.Source: H Limitations and Future Work · p. 13
• Higher-hop leakage may vary with relational coverage, aggregation, and neighborhood sampling.The authors also propose testing more direct downstream attribution and improving threshold calibration.Source: H Limitations and Future Work · p. 13