PaperlayerView exact revision on arXiv

Source-linked AI summary

Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Model Dependence and Evaluation Reliability

Roshan Reddy Upendra, Alexandre Dorais, Joe Meyer, Andrew Pouret, Anastasios Lambrianos Stappas, Dinesh Katupputhur Ramprasath, Viswanath Ganapathy, Tom Palczewski, Minghua Li

arXiv:2609.36417v1cs.AI

TL;DR

Relational ICL can condition predictions on support features unavailable to clean queries, raising a support/query evaluation concern. This paper tests controlled leakage across tasks and model configurations; target-table and Full leakage show the largest aggregate deviations, and effects can reverse model comparisons.

  • Problem

    Relational ICL may expose labeled support examples to target-derived information unavailable to clean queries, creating a support/query information mismatch.

    Source: §1 · p. 2

  • Method

    The study evaluates 20 controlled leakers across 13 RelBench tasks and five ICL configurations, comparing leakage effects, detection methods, and detector-based removal.

    Source: §1 · p. 2, §3.4 · p. 5, §3.5 · p. 5

  • Results

    Median |∆F1| ranges from 0.02 to 0.11 at 0-hop and 0.06 to 0.16 under Full leakage, with higher-hop medians rounding to zero; effects vary by task and model and can reverse comparisons.

    Source: §4 · pp. 6–8

  • Takeaways & Limitations

    The support/query information boundary should be part of evaluation, with audits of prediction-time availability and mitigation validated by downstream recovery rather than detector quality alone.

    Source: §5 · p. 9

    Detector-based removal does not consistently restore clean evaluation because leakers may remain and false-positive removal can discard original columns.

    Source: §4 · p. 8

Abstract

from arXiv · show

Relational in-context learning (ICL) uses labeled support examples and their linked relational context to predict labels for new queries. This creates a failure mode when target-derived features are present in the support context but unavailable for the query. We study this setting as support-set target leakage. We construct 20 controlled target-derived features that vary in signal fidelity, representation, semantic transparency, coverage, and zero-, one-, and two-hop relational placement, and evaluate them across 13 RelBench tasks and five relational ICL configurations that vary the ICL head, message-passing depth, pretraining cohort, or relational encoder architecture. We evaluate matched 0-hop, 1-hop, and 2-hop leakage settings, together with a Full leakage condition containing all 20 leaker columns. Within the tested configurations, target-table (0-hop) and Full leakage produce the largest aggregate deviations from clean evaluation, while higher-hop effects are often weaker, consistent with differences in effective exposure associated with temporal reachability, sampling, and aggregation fidelity. Leakage effects are strongly task- and model-dependent and can reverse relative conclusions between model variants even when aggregate changes are small. For leaker detection, we compare an Integrated Gradients (IG)-based screening method with mutual information (MI) and leave-one-column-out (LOCO) on a common Baseline subset. Ranking quality is strongest in the high-impact 0-hop and Full leakage conditions, but detector-based removal does not consistently restore the clean evaluation. A four-task rel-salt case study further shows the same evaluation concern with native-schema leakage candidates from the original relational schema. These results identify the support/query information boundary as an important component of reliable relational ICL evaluation.

1 Introduction

Relational ICL can expose labeled support examples to target-derived information that is unavailable to clean queries, creating support-set target leakage. The study evaluates its effects across tasks and model configurations, including whether contamination changes model comparisons and whether detectors can support reliable mitigation.

1 Introduction

  • Support-set target leakage occurs when target-derived columns are available to labeled support examples but not to clean queries during downstream ICL inference.The mismatch is distinct from leakage during dataset construction, temporal splitting, or pretraining.Source: §1 · p. 2
  • The study tests 20 controlled leaker types across 13 RelBench tasks and five configurations that vary ICL head, message-passing depth, pretraining cohort, or encoder architecture.The leakers vary in signal fidelity, coverage, modality, semantic transparency, and relational distance.Source: §1 · pp. 2–3
  • Target-table and Full leakage produce the largest aggregate deviations in the tested configurations, while higher-hop effects are often weaker.Effects vary substantially across tasks and models, and contamination can reverse model-variant comparisons even when aggregate effects are small.Source: §1 · pp. 2–3
  • The study separately examines suspicious-column ranking, thresholded detection, and whether detector-based removal moves evaluation toward the clean result.It compares attribution-, association-, and perturbation-based ranking and tests threshold sensitivity and downstream recovery.Source: §1 · p. 3

2 Related Work

Prior relational leakage work addresses construction-time or temporal leakage, while this paper studies target-derived information available to ICL support but not clean queries. Related robustness and detection methods motivate distinct tests of model reliance and candidate-column ranking.

2 Related Work

  • Predictive Query Language identifies potentially leaking records during relational training-table construction, while Xu and Wipf restrict neighborhood labels to entities observed before prediction time; this paper instead studies support-only availability.Task Scarcity and Label Leakage studies task-specific shortcuts learned during relational transfer, whereas this setting introduces target-derived columns during downstream ICL inference without changing model parameters.Source: §2 · p. 3
  • Prior work perturbs relational context or tests robustness to feature and label perturbations, motivating comparisons across ICL heads because their readout mechanisms can differ [2026a] [2026].The cited work also reports qualitatively different in-context readout mechanisms for TabPFNv2 and TabICLv2 [2025] [2026].Source: §2 · p. 3
  • IG screens feature reliance, MI measures association without showing downstream use, and LOCO tests removal effects but requires repeated evaluations and can be affected by redundant features [2017] [2001] [2008].SHAP costs grow with evaluations and estimates depend on absent or dependent-feature handling [2017] [2021]; attention and intermediate representations alone do not establish that a feature drives the output [2019] [2019] [2021].Source: §2 · pp. 3–4

3 Methodology

The methodology defines leakage as a support/query availability mismatch, constructs controlled leaker conditions, and evaluates model comparisons, detection, and mitigation. It also includes primary RelBench tasks and a separate native-schema rel-salt case study.

3.1 Problem Formulation

  • The primary comparison contrasts clean support with leaky support while keeping the query clean, isolating target-derived information available during support inference but absent for query prediction.A leaky-support, leaky-query control distinguishes an availability mismatch from features the model does not use; signed ∆F1 and |∆F1| summarize direction and deviation magnitude.Source: §3.1 · p. 4

3.2 Synthetic Leaker Conditions

  • The 20-column stress test places 14 leakers on target rows, three one hop away, and three two hops away, with matched hop-specific and Full conditions.In support-only conditions, injected columns and their representations are removed from the query; ground-truth identities are not used by the detector or removal rule.Source: §3.2 · p. 4

3.3 Model Comparisons

  • Four variants isolate message-passing depth, pretraining cohort, ICL head, or encoder architecture relative to the Baseline.The study compares each variant with Baseline and counts strict ranking reversals while excluding tied comparisons.Source: §3.3 · pp. 4–5

3.4 Attribution-Based Detection

  • Ridge-IG uses a Ridge surrogate over relational representations, with attributions computed through the frozen encoder; its scores measure surrogate influence rather than exact downstream ICL attribution.The method fixes sampled neighborhoods and few-shot context during 12-step representation interpolation.Source: §3.4 · p. 5

3.5 Detector Baselines and Thresholding

  • Ridge-IG is compared with MI and LOCO, which respectively rank support-label association and measure downstream effects of removing each candidate column.MI does not establish downstream reliance; LOCO requires a separate evaluation per candidate and can underestimate importance under redundancy. Rankings are assessed before thresholding.Source: §3.5 · p. 5

3.6 Mitigation

  • IG-based mitigation flags columns above a within-run q90 threshold and removes flagged columns, including false positives, from both support and query before recomputing embeddings and rerunning ICL.Recovery gain measures movement toward clean evaluation; recovery can fail when leakers remain or useful original columns are removed.Source: §3.6 · pp. 5–6, §3.5 · p. 5

3.7.1 Primary Tasks

  • The primary experiments use 13 binary-classification RelBench tasks with standard splits, labeled training and validation embeddings as support, and test embeddings as queries.The tasks span rel-avito, rel-event, rel-trial, and rel-arxiv.Source: §3.7.1 · p. 6

3.7.2 Native-Schema Target-Field Leakage Case Study

  • The separate four-task multiclass rel-salt case study treats other target-designated fields as candidate leakers and compares Baseline performance before and after their removal.It uses the same detector-based removal and recovery criterion, and is excluded from cross-model aggregates.Source: §3.7.2 · p. 6

4 Results and Discussion

Across the tested configurations, 0-hop and Full leakage produce the largest aggregate deviations, but effects vary by task and model and can reverse model comparisons. Detection is strongest in the high-impact conditions, yet detector-based removal does not consistently restore clean evaluation.

4 Results and Discussion

  • Matched availability produces large gains for 0-hop and Full leakage, but support-only exposure can reduce performance; higher-hop effects depend on reachability, sampling, and aggregation fidelity.The study-outcome 2-hop aggregate agrees with the queried label about 65% of the time.Source: §4 · p. 6
  • searchstream-click shows large support-only degradation, while user-clicks and user-visits remain unchanged across all five configurations.Those latter tasks are highly imbalanced, with clean predictions already near constant-class solutions; identical aggregate scores need not mean identical predictions.Source: §4 · pp. 6–7
  • 0-hop median |∆F1| ranges from 0.02 to 0.11 across models, while Full leakage ranges from 0.06 to 0.16, the largest aggregate deviations.Dataset aggregation likewise finds rel-arxiv most sensitive, with median |∆F1| = 0.16 at 0-hop and 0.26 under Full leakage.Source: §4 · p. 6, Figure 1 · p. 7
  • No evaluated configuration eliminates leakage failure or yields a universal robustness ordering across model dimensions.PluRel has median 0-hop effect 0.02 and Full effect 0.16, whereas TabICL has 0.06 at 0-hop and 0.06 under Full leakage.Source: §4 · p. 7
  • Leakage reverses model rankings even when average effects are small: 0-hop reversals occur in 3/6 MP-8, 4/6 PluRel, 2/6 TabICL, and 5/6 Bimodal comparisons.Under Full leakage, the corresponding counts are 1/5, 3/5, 1/5, and 3/5; small task-specific shifts can exceed clean model margins.Source: §4 · pp. 7–8, Figure 2 · p. 8
  • On the common Baseline subset, IG, MI, and LOCO ranking quality is generally stronger for 0-hop and Full leakage than for higher hops.Top-five review identifies at least one leaker in 88% of 0-hop runs and 94% of Full runs.Source: §4 · p. 8
  • Detector-based removal often moves performance toward clean but does not consistently restore it, because residual leakers, removed original features, and placeholder representations alter the input distribution.TabICL user-repeat removes all active 0-hop leakers but also ten original columns, worsening performance.Source: §4 · p. 8
  • Across four multiclass Baseline tasks, retaining other target-designated SALT fields reduces mean macro-F1 from 0.362 to 0.293; removal reaches 0.294 and does not restore clean evaluation.This single-configuration case study uses native-schema leakage candidates rather than synthetic proxies.Source: §4 · pp. 8–9

5 Conclusion

Support-set target leakage is an inference-time mismatch in relational ICL, and the study finds that its effects can vary across tasks and models. The support/query information boundary therefore matters for evaluation reliability.

5 Conclusion

  • Leakage effects depend jointly on task characteristics and effective relational exposure, and can reverse model rankings even when aggregate changes are small.Higher-hop effects depend on temporal reachability, sampling, and aggregation fidelity.Source: §5 · p. 9
  • Benchmarks should audit whether support features are available under intended prediction-time semantics and examine robustness across support conditions.Mitigation should be validated by downstream recovery rather than detector quality alone.Source: §5 · p. 9

A Experimental Protocol and Effective Relational Exposure

The protocol contrasts clean evaluation with support-only leakage and a matched-availability diagnostic, using labeled training and validation embeddings as support and test embeddings as queries. Five configurations vary one component of the relational ICL pipeline at a time.

A Experimental Protocol and Effective Relational Exposure

  • Training and validation target-row embeddings form the labeled support set, while test embeddings form the query set for each RelBench task.Embeddings are computed from relational neighborhoods using the same sampling configuration within a model comparison; downstream TabPFN fits on support and predicts queries.Source: A Experimental Protocol and Effective… · p. 11
  • The primary leakage condition injects selected columns into support but not query; the matched-availability diagnostic retains them in both.After detector-based mitigation, flagged columns are removed from both schemas and relational embeddings are recomputed; the pretrained encoder remains frozen.Source: A Experimental Protocol and Effective… · p. 11

B.1 Model Details

  • The five configurations vary message-passing depth, pretraining cohort, ICL head, or relational encoder architecture while retaining shared evaluation settings within comparisons.Baseline uses four Griffin message-passing blocks, a pretrained TabPFN head, 512-dimensional representations, and two-hop neighborhoods sampled with fanout 9.Source: §B.1 · p. 12, Table 1 · p. 12

B.2 Pretraining data

The experiments use clean-pretrained CTU–RelBench checkpoints and controlled leakers spanning relational placement and representation. Detection compares attribution, association, and perturbation rankings, with performance varying by leakage condition.

B.2 Pretraining data

  • The CTU–RelBench checkpoints use clean data from 15 CTU and four RelBench databases, with the four downstream evaluation databases held out from pretraining.The PluRel configuration instead uses a Griffin checkpoint pretrained on PluRel-generated relational data.Source: §B.2 · p. 13, Table 2 · p. 13

C Synthetic Leaker Construction

  • The Full condition contains 20 target-derived columns, while matched 0-, 1-, and 2-hop conditions use three comparable proxy designs at each distance.Proxies vary in signal fidelity, coverage, modality, and semantics; temporal reachability and neighborhood sampling determine whether neighboring leakers can affect target representations.Source: C Synthetic Leaker Construction · p. 13
  • Avito user-clicks and user-visits test targets contain 1.54% and 85.06% positive examples, respectively, and their clean macro-F1 is close to constant-class solutions.The paper cautions that class imbalance does not determine leakage response, though class structure can make macro-F1 insensitive to some prediction changes.Source: C Synthetic Leaker Construction · p. 13

D.1 Ridge-IG scoring

  • IG uses a Ridge classifier as a differentiable surrogate and attributes validation-batch predictions through the frozen relational encoder using 12 midpoint integration steps.Column scores combine mean attribution and the mean among the strongest 10% of occurrences, then use robust within-group standardization; q90 is a heuristic operating point, not a calibrated test.Source: §D.1 · pp. 14–15

D.2 MI and LOCO comparison

  • IG and LOCO provide the strongest 0-hop rankings, while all three detectors are substantially weaker in tested higher-hop settings; IG remains informative under Full leakage.The comparison uses the same Baseline subset and candidate columns; raw AUPRC comparisons across leakage conditions are affected by differing leaker prevalence.Source: §D.2 · p. 15, Table 5 · p. 15

D.3 Threshold and Top-k Verification Sensitivity

Thresholded detection remains substantially stronger for 0-hop and Full leakage than for the tested higher-hop conditions, across alternative operating rules. Top-k verification can identify active leakers in the strongly exposed settings, but detector-based removal does not reliably restore clean evaluation.

D.3 Threshold and Top-k Verification Sensitivity

  • Across operating rules, thresholded identification remains stronger for 0-hop and Full leakage than for the tested higher-hop conditions.q95 gives the highest 0-hop precision and F1, while q80 gives the highest Full-condition recall and F1; no percentile rule dominates across conditions.Source: §D.3 · pp. 15–16, Table 6 · p. 16
  • At 0-hop, inspecting the top five columns identifies an active leaker in 88% of runs and recovers 67% of active leakers on average.Under Full leakage, the top-five hit rate is 94%, while small verification budgets recover little or no target-derived information in the tested higher-hop conditions.Source: §D.3 · p. 16, Table 7 · p. 17

D.4 Detector-Based Mitigation

  • Detector-based removal can leave residual leakers while also removing original columns, so post-removal performance reflects a changed feature configuration.For example, one run removed all three active leakers but also ten original columns; another removed six of twenty active leakers and eight original columns.Source: §D.4 · pp. 16–17, Table 8 · p. 17
  • A post-removal macro-F1 above the clean value does not by itself establish successful cleaning; assess distance to clean alongside detector TP/FP/FN composition.Residual leakage or false-positive feature removal can remain even when the post-removal score exceeds clean performance.Source: §D.4 · p. 17

E Individual-Leaker Effects

The individual-leaker diagnostic tests one synthetic column at a time with a clean query and the Baseline Griffin–TabPFN configuration. Target-table effects are larger and denser than higher-hop effects, but their direction and magnitude depend on the task.

E Individual-Leaker Effects

  • Across 182 target-table task–leaker conditions, mean absolute |∆F1| is 0.083, versus 0.014 across 39 one-hop conditions and 0.007 across 39 two-hop conditions.All 14 target-table constructions produce |∆F1| > 0.05 on at least one task, so the aggregate 0-hop effect is not attributable to one synthetic leaker type.Source: E Individual-Leaker Effects · p. 17, Figure 5 · p. 18
  • Effects vary by task and can be positive or negative: user-clicks is unchanged across individual constructions, while user-visits is nearly invariant.Several distinct 0-hop constructions produce substantial changes on searchstream-click, the event-interest tasks, and eligibilities-adult.Source: E Individual-Leaker Effects · p. 18
  • Only 4/39 one-hop and 2/39 two-hop conditions have |∆F1| > 0.05, compared with 65/182 at 0-hop.Higher-hop effects are sparse, not uniformly zero; their impact also depends on temporal reachability, sampled exposure, and aggregation fidelity.Source: E Individual-Leaker Effects · p. 18
  • The diagnostic does not establish a universal severity ordering by noise, semantic transparency, or representation type.Because it uses only the Baseline configuration, it also does not support model-specific leaker-family preferences.Source: E Individual-Leaker Effects · p. 18

F Complete Task-Level Effects and Ranking Reversals

Support-only leakage can change the relative performance ordering of model variants, even when aggregate effects are small. Many strict reversals persist when comparisons with small performance margins are excluded.

F Complete Task-Level Effects and Ranking Reversals

G rel-salt Native-Schema Target-Field Leakage Case Study

A four-task rel-salt case study compares reference feature configurations with configurations including other target-designated fields, then applies detector-based removal. Inclusion changes performance substantially on some tasks, and removal does not consistently recover the reference results.

G rel-salt Native-Schema Target-Field Leakage Case Study

  • Across four rel-salt tasks, target-field inclusion changes macro-F1 by −0.069 on average relative to task-specific references.The change is −0.033 for item-plant and −0.235 for sales-incoterms, while item-incoterms and sales-office show little or no change.Source: G rel-salt Native-Schema Target-Field… · p. 21, Table 13 · p. 21
  • Detector-based removal does not consistently restore reference performance: item-plant changes from −0.033 to −0.010, while sales-incoterms shifts from −0.235 to −0.252.These are paired differences relative to each task's reference configuration.Source: G rel-salt Native-Schema Target-Field… · p. 21, Table 13 · p. 21
  • The case study treats target-designated fields as candidates, not confirmed deployment-time leakers; confirmation requires field- and task-specific prediction-time availability semantics.No synthetic columns are introduced in this native-schema comparison.Source: G rel-salt Native-Schema Target-Field… · pp. 20–21

H Limitations

The study is bounded to controlled support/query availability mismatches and a limited benchmark configuration, with higher-hop and native-schema findings carrying additional interpretive constraints. Its fixed evaluation setup does not estimate population-level leakage frequency or support-sampling variance.

H Limitations

  • The synthetic tests cover one specific support/query availability mismatch, not every form of relational-database leakage.The primary benchmark uses 13 binary RelBench tasks and five ICL configurations; the four-task multiclass rel-salt analysis uses Baseline only.Source: H Limitations · p. 21
  • Fixed evaluation and subsampling mean the reported effects are not estimates of population-level leakage frequency or support-sampling variance.The experiments do not repeat support-set sampling to estimate variance.Source: H Limitations · p. 21
  • Higher-hop findings depend on the selected relational paths, temporal reachability, neighborhood sampling, and aggregation in these experiments.Per-query sampled-subgraph traces would be needed to attribute every individual prediction change causally to a specific path.Source: H Limitations · p. 21
  • Ridge-IG screens a surrogate objective over relational representations, not the downstream TabPFN or TabICLv2 prediction itself.Its q90 threshold and alternative score-based rules are heuristic rather than calibrated hypothesis tests, so it is an initial screening approach rather than a complete detector.Source: H Limitations · pp. 21–22
  • The rel-salt analysis does not independently establish prediction-time availability for every native-schema leakage candidate.It is therefore reported separately from the controlled RelBench benchmark.Source: H Limitations · p. 22

Related papers

Loading 2609.36417v1…