PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior

Shivam Dubey, Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni, Aditya Tanna, Vinay Kumar Sankarapu

arXiv:2610.03057v1cs.LG

TL;DR

Synthetic relational benchmarks do not explain which corpus properties lead models to use relations. The paper measures relational predictive signal, traces it through RT computation, and tests its role in downstream behavior. RelDiff shows the strongest parent-linked signal, and disrupting the corresponding mechanism removes its advantage on relational tasks.

  • Problem

    Synthetic-corpus benchmark comparisons do not identify which data properties induce relational computation, even when they show that one corpus produces a stronger model.

    Source: §1 · pp. 1–2

  • Method

    The study measures FK-linked predictive gain in synthetic corpora, then tests checkpoint reliance, localizes cross-table computation, and intervenes on that computation during downstream inference.

    Source: §1 · p. 2

  • Results

    RelDiff’s checkpoint shows stronger functional reliance on relational structure, and disrupting the identified computation removes its advantage on a relational task while structure-insensitive checkpoints remain nearly unchanged.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Relational reliance is an inductive bias that can help or hurt depending on whether learned dependencies match the target database.

    Source: §4 · p. 11

    The study uses one relational architecture and objective, one checkpoint per generator, and corpora that differ across multiple dimensions, limiting data-to-mechanism claims to controlled observational attribution.

    Source: §4 · pp. 11–12

Abstract

from arXiv · show

Relational foundation models are increasingly pretrained on synthetic databases, yet downstream benchmarks reveal little about why one synthetic corpus produces a better model than another. In particular, strong performance may arise from realistic row-level statistics without the model ever learning to use relational structure. We study this as a data-attribution problem: which property of synthetic pretraining data induces relational computation? Using four Relational Transformer checkpoints trained with the same architecture, initialization, objective, and compute budget on corpora produced by four relational data generators, we trace a measurable property of the data to learned computation and downstream behavior. We hypothesize that relational mechanisms emerge when cross-table information is predictively necessary for the masked-cell pretraining objective. RelDiff exhibits by far the largest predictive gain from foreign-key-linked parents, and its corresponding model is uniquely sensitive to foreign-key interventions on unseen databases. This dependence survives a random-initialization control, grows monotonically with the fraction of corrupted links, and localizes to a serial cross-table pathway. Finally, disrupting the same mechanism during downstream inference removes RelDiff's advantage on relational tasks while leaving structure-insensitive models nearly unchanged. These results connect a property of synthetic training data to a learned mechanism and, through intervention, to downstream behavior.

1 Introduction

Synthetic relational benchmarks can rank pretraining corpora but cannot reveal which data property produced a learned capability. This paper traces relational predictive necessity from synthetic data through model computation to downstream behavior.

1 Introduction

  • Benchmark scores alone cannot identify whether gains come from row statistics, schema diversity, within-table structure, or useful foreign-key signal.The paper frames this as attributing capability to a measurable property of a generated distribution rather than to generator identity.Source: §1 · p. 2
  • Relational predictive necessity arises when FK-linked information adds predictive signal unavailable from the target row, making relational context useful for masked-cell reconstruction.The hypothesis distinguishes architectural capacity from whether pretraining makes cross-table pathways functionally important.Source: §1 · p. 2
  • The study measures predictive signal in synthetic corpora, tests checkpoint reliance on FK structure, localizes internal computation, and intervenes during downstream inference.The data-side measure is a predictive gain from FK-linked information beyond target-row features.Source: §1 · p. 2
  • Structural corruption, random-initialization controls, graded interventions, and pathway-specific ablations identify stronger relational reliance in the checkpoint trained on the strongest parent-linked signal.The analysis localizes that dependence to a specific serial cross-table computation.Source: §1 · p. 2
  • Disrupting the identified computation removes RelDiff’s advantage on a relational task, while structure-insensitive checkpoints remain nearly unchanged.Relational priors can help or hurt depending on their match to the downstream database; the corpus-to-mechanism link remains observational, not a universal causal law.Source: §1 · pp. 2–3

2 Problem Setup and Hypothesis

The paper compares controlled RT checkpoints, defines the cross-table pathways they can use, and evaluates them on paired database inputs. Its hypothesis is that relational context becomes functionally useful when it supplies predictive information unavailable within a target row.

2.1 Controlled Comparison of Synthetic Pretraining Distributions

  • Four RT checkpoints share architecture, initialization, objective, and training budget, but use corpora from RelDiff, GRDM, PluRel, or RDB-PFN.The generators differ in multiple corpus properties, so generator identity is not treated as the causal variable.Source: §2.1 · p. 3

2.2 Cross-Table Computation in RT

  • Parent access exposes FK-linked parent attributes to a child-row representation, while child aggregation lets a parent representation collect information from referencing rows.Both routes are available in every checkpoint; experiments test their functional interaction rather than assuming they operate independently or serially.Source: §2.2 · p. 3

2.3 Evaluation Setting

  • Paired model-side comparisons use identical inputs and interventions on three RelBench databases, with controlled mechanistic experiments on rel-stack.GRDM’s text and datetime fields were numerically represented, so its absolute ranking is not interpreted as an isolated effect of its generation strategy.Source: §2.3 · pp. 3–4

2.4 Relational Predictive Necessity

  • Relational predictive necessity occurs when FK-linked information adds predictive signal unavailable locally, so relational context can reduce masked-cell reconstruction loss.The paper operationalizes this with out-of-sample predictive gain from adding related-table information to a target-row predictor.Source: §2.4 · p. 4, Table 1 · p. 5
  • The hypothesis predicts that stronger relational signal in a corpus corresponds to greater checkpoint dependence on correct FK structure and downstream sensitivity to the relevant computation.Inference-time interventions support causal conclusions about model behavior, but the four corpora do not isolate predictive necessity from other distributional differences.Source: §2.4 · p. 4

3 Experimental Evaluation

The experiments trace a path from parent-linked predictive signal in synthetic data to learned cross-table computation and its downstream effects. RelDiff shows the strongest relational signal and reliance, but whether that reliance helps depends on the target database.

3.1 Predictive Structure in Synthetic Pretraining Data

  • RelDiff’s linked-parent attributes raise median held-out R2 by 0.072, versus 0.009 for PluRel and approximately zero for RDB-PFN.RDB-PFN has the largest row-only score (R2 = 0.436), while RelDiff’s is 0.022; simple child aggregates do not improve prediction in any corpus.Source: §3.1 · pp. 5–6

3.2.1 Structural Interventions

  • Full FK shuffling raises RelDiff’s reconstruction loss from 0.241 to 0.443 (83.6%), while GRDM, PluRel, and RDB-PFN rise by only 5.3%, 6.1%, and 3.4%.Zeroing nbr increases RelDiff’s loss by 0.127 but has negligible effect on the other trained checkpoints.Source: §3.2.1 · p. 6

3.2.2 Architectural Floor and Graded Corruption

  • RelDiff’s response is approximately 17× the random-initialization effect, and its loss rises monotonically as FK corruption increases.The linear-fit slope is 11–30× larger than for the other trained checkpoints; the figure juxtaposes this response with the data-side parent-linked signal.Source: §3.2.2 · pp. 6–7, Figure 1 · p. 7

3.2.3 Reliance Across Real Databases

  • Across rel-stack, rel-ratebeer, and rel-arxiv, RelDiff alone shows substantial mean degradation under both FK corruption and neighbor ablation, but full FK shuffling changes its loss by +0.165, +0.067, and −0.112, respectively.Corrupting relations improves RelDiff’s rel-arxiv reconstruction, showing that learned relational reliance is not always beneficial.Source: Table 3 · p. 7, §3.2.3 · p. 7

3.3.2 A Serial Relational Pathway

  • Disabling both cross-table pathways yields the same RelDiff loss as disabling parent access alone: 0.453, supporting a serial pathway from child aggregation to parent access.The neighbor computation remains active when parent access is blocked, but its output can no longer influence the masked target.Source: Figure 2 · p. 8, §3.3.2 · p. 8

3.3.3 Fine-Grained Functional Specialization Across Heads

  • Ablating RelDiff’s head 0 accounts for approximately 39.5% of positive head-attributable loss increase, versus 12.5% under uniform contribution from eight heads.Weight-change Gini coefficients are nearly identical across checkpoints (0.016–0.020), so parameter movement does not recover this behavioral specialization.Source: Figure 3 · p. 9, §3.3.3 · p. 9

3.4.1 Downstream Evaluation and Corrected Context Scaling

  • After correcting rare-label evaluations, all four checkpoints’ mean classification AUROC decreases from context 1,024 to 30k, with RelDiff showing the smallest decline.The 30k points use n = 2048 for the two rare-label rel-stack tasks; the supported claim is relative robustness, not indefinite improvement with context.Source: §3.4.1 · p. 9, Figure 4 · p. 10

3.4.2 Intervening on the Learned Mechanism

  • At context 1,024, FK shuffling reduces RelDiff’s user-engagement AUROC from 0.847 to 0.718 (−0.129), and nbr ablation reduces it to 0.728 (−0.119); disrupting the mechanism removes its advantage.PluRel and RDB-PFN change by at most 0.007 AUROC under FK shuffling, while RelDiff falls below both untouched checkpoints.Source: Table 4 · p. 10, §3.4.2 · pp. 10–11

4 Discussion and Limitations

The discussion links predictive structure in synthetic data to relational computation and downstream behavior, while qualifying the strength and scope of that evidence. It argues that generator evaluation should consider predictive coupling, not structural variety alone.

4 Discussion and Limitations

  • Disrupting relational structure or its corresponding internal computation changes predictions, linking the learned cross-table mechanism to downstream behavior.The authors identify this as their strongest causal evidence; the same relational prior can help or hurt depending on its fit to a target database.Source: §4 · p. 11
  • Structural complexity alone may not reward cross-table computation; relational context must add predictive information unavailable from the target row.The diversity analysis does not reproduce the ordering of functional relational reliance.Source: §4 · p. 11
  • Relational reliance is not equivalent to usefulness: learned relational priors can help when dependencies align with a target database and hurt when they mismatch.Sensitivity to relational structure alone does not establish that a learned prior transfers correctly to every schema or task.Source: §4 · p. 11
  • The study uses one architecture and masked-cell objective, one checkpoint per generator, and corpora differing along multiple dimensions; the identified pathway is RT-specific.The corpus-to-reliance association is observational; a stronger test would selectively alter FK-conditioned predictive coupling and retrain. GRDM's semantic-type preprocessing also limits interpretation of its absolute ranking.Source: §4 · pp. 11–12
  • The authors trace predictive signal in synthetic data through FK dependence and a localized computation to downstream effects, motivating attribution to measurable distribution properties.Their conclusion is that generator identity alone does not explain which corpus properties produce a capability.Source: §4 · p. 12

A.1 Generator Characteristics

  • RelDiff and GRDM are reference-conditioned, whereas PluRel and RDB-PFN use synthetic priors; the generators vary multiple properties, so their categories are descriptive, not causal.Reference-conditioned generation is tied to an existing database's schema or relational structure.Source: §A.1 · p. 14, Table 5 · p. 14

A.2 GRDM Semantic-Type Conversion

The authors audit semantic types in the exported corpora because RT uses type-specific input pathways. The available GRDM export lacks preserved text and categorical columns, complicating comparisons with other checkpoints.

A.2 GRDM Semantic-Type Conversion

  • The audit checks exported semantic types because RT has separate pathways for numerical, textual, datetime, and boolean values, including a MiniLM-based text encoder.The resulting census is reported in Table 6.Source: §A.2 · p. 14
  • The audit finds no preserved text or categorical columns in the available GRDM export; string and datetime fields are represented numerically.Counts in the column-type census cover the sampled databases used in the audit.Source: Table 6 · p. 14
  • GRDM therefore received no training signal through RT's text pathway, unlike corpora retaining string-valued attributes.Reference databases whose RelDiff corpus retains string-valued attributes have no corresponding preserved string columns in the available GRDM export.Source: §A.2 · p. 14
  • GRDM comparisons combine relational-generation differences with semantic-type exposure, so its absolute ranking is not an isolated measure of relational coupling or generator design.Source: §A.2 · p. 14

B.1 Task-Level Predictive-Necessity Distributions

Task-level results show that RelDiff's parent-linked predictive gains are distributed across target-column problems rather than driven by a single extreme case. Child-aggregate gains remain close to zero across corpora.

B.1 Task-Level Predictive-Necessity Distributions

  • Table 1 summarizes relational predictive necessity using medians across target-column prediction problems, while Figure 5 displays the task-level values.Source: §B.1 · p. 14, Figure 5 · p. 15
  • Each plotted point is a held-out target-column problem and measures the change in R2 after adding relational information to row-only features.Source: §B.1 · p. 14
  • RelDiff shows more frequent and larger positive gains from linked-parent attributes, while RDB-PFN is concentrated near zero and child-aggregate gains stay close to zero.The task-level distribution supports the aggregate result rather than a result driven by one unusual target-column problem.Source: §B.1 · p. 15, Figure 5 · p. 15

B.2 Corpus Diversity as an Alternative Explanation

The diversity analysis finds that diversity rankings do not explain functional relational reliance across checkpoints. Its additional diagnostics characterize checkpoint differences but do not establish component necessity.

B.2 Corpus Diversity as an Alternative Explanation

  • Diversity is measured through mean pairwise embedding distances for features, tables, and cross-table structure, without using trained RT representations.The analysis tests whether corpus diversity offers an alternative explanation for checkpoint differences.Source: §B.2 · p. 15
  • PluRel has greater cross-table diversity than GRDM (0.85 versus 0.71) despite little FK dependence, while RDB-PFN has greater feature diversity than PluRel but remains nearly insensitive to structural corruption.The diversity ordering does not reproduce the ordering of functional relational reliance.Source: Table 7 · p. 15, §B.2 · p. 15

C Additional Checkpoint Diagnostics

  • Parameter movement, representation similarity, and attention distributions are complementary diagnostics, not evidence by themselves that a component is functionally necessary.Source: C Additional Checkpoint Diagnostics · p. 15

B.2 Corpus Diversity as an Alternative Explanation

  • RelDiff has the largest diversity on all three axes, but diversity alone does not reproduce the ordering of functional relational reliance.Source: §B.2 · p. 15, Figure 6 · p. 16

C.1 Weight-Space Specialization

Pretraining changes checkpoint weights and representations in corpus-specific ways, but neither parameter movement nor selective attention alone establishes functional relational computation.

C.1 Weight-Space Specialization

  • RelDiff’s largest attention-stream RMS deviation is in nbr, while GRDM’s is in feat; these are localization cues, not evidence of functional importance.Per-parameter movement is substantial for reference-conditioned checkpoints.Source: §C.1 · p. 16, Table 8 · p. 16

C.2 Representation Similarity

  • RDB-PFN is the largest late-layer CKA outlier despite PluRel and RDB-PFN being comparatively close in parameter space; GRDM and RelDiff have the largest late-layer CKA.Parameter-space proximity is insufficient to infer similarity of learned computation.Source: §C.2 · pp. 16–17, Table 9 · p. 17

C.3 Attention Within the Permitted FK Neighborhood

  • RelDiff and GRDM show sharper attention within permitted FK neighborhoods than PluRel and RDB-PFN, but GRDM’s selectivity does not imply comparable functional dependence under FK corruption.The nbr stream is architecturally restricted to FK-linked cells, so attention on relational edges alone is not evidence of learning.Source: §C.3 · p. 17, Table 10 · p. 17

D Negative Result: Input-Decodable Linear Probes

Input-decodable probes do not distinguish the checkpoints because their targets are already accessible without learned relational computation. High probe accuracy therefore cannot establish that such computation was learned.

D Negative Result: Input-Decodable Linear Probes

  • Probe performance does not meaningfully discriminate between checkpoints: tested properties are near ceiling in random initialization, with gains below 0.014 and negative gains for two trained checkpoints.The probes target semantic type, table identity, parent-table identity, and row degree.Source: D Negative Result: Input-Decodable… · pp. 17–18
  • In a residual architecture, original cell information can remain linearly accessible, while parent-table identity and row degree can come directly from the input FK index.Thus high probe accuracy does not demonstrate a learned relational computation.Source: D Negative Result: Input-Decodable… · p. 18
  • A more discriminative probe would target linked-parent values unavailable in the queried cell’s input and requiring cross-table computation.Source: D Negative Result: Input-Decodable… · p. 18

E Database-Specific Sign of Structural Reliance

FK shuffling changes RelDiff’s reconstruction loss in database-specific directions, while the other checkpoints show much smaller changes in this probe suite. The sign distinguishes beneficial from harmful relational reliance.

E Database-Specific Sign of Structural Reliance

  • +0.165 for rel-stack, +0.067 for rel-ratebeer, and −0.112 for rel-arxiv are RelDiff’s masked-cell reconstruction-loss changes under FK shuffling.The other checkpoints’ corresponding absolute changes are at most 0.027 in this probe suite.Source: E Database-Specific Sign of Structural… · p. 18
  • A positive change means correct FK structure helps reconstruction, whereas a negative change means destroying that structure improves it.Rel-arxiv’s sign reversal still reflects a substantial response to changed relational correspondences.Source: E Database-Specific Sign of Structural… · p. 18
  • Here, reliance means that learned computation depends on relational structure, not that the dependence necessarily improves performance.Source: E Database-Specific Sign of Structural… · p. 18

F.1 Rare-Label Re-Evaluation at 30k Context

A larger rare-label evaluation revised several AUROC estimates in previously underpowered cells, while well-powered cells remained nearly stable. FK shuffling also improved AUROC for RelDiff and GRDM on rel-ratebeer user-churn.

F.1 Rare-Label Re-Evaluation at 30k Context

  • At 30k context, the initial n = 256 evaluation had only six user-engagement and ten user-badge positives, making AUROC sensitive to individual predictions.The rerun used n = 2048, with 60 user-engagement and 64 user-badge positives.Source: §F.1 · p. 18
  • RelDiff user-engagement changed from 0.929 to 0.865 and user-badge from 0.847 to 0.812; PluRel and RDB-PFN user-engagement changed from 0.564 to 0.652 and 0.538 to 0.642.Source: §F.1 · p. 18
  • Well-powered evaluation cells changed by at most 0.007, and the paper makes no claim that RelDiff’s mean classification performance improves from context 1,024 to 30k.The pattern is consistent with instability from few positive examples rather than a systematic protocol change.Source: §F.1 · p. 18

F.2 Additional Example of a Harmful Relational Prior

  • On rel-ratebeer user-churn at context 1,024, FK shuffling raised AUROC by +0.061 for RelDiff and +0.064 for GRDM; the effect was at most 0.007 for PluRel and RDB-PFN.This mirrors reconstruction-level sign reversals: relational priors can help or hurt depending on their match to the target database and task.Source: §F.2 · p. 19

G Implementation and Reproducibility Details

The evaluation protocols use matched inputs and examples to isolate intervention effects, alongside a predictive-necessity analysis across four synthetic corpora. Mechanistic comparisons are within-checkpoint measurements, since each generator has one pretrained checkpoint.

Structural interventions.

G Implementation and Reproducibility Details

  • Predictive-necessity analysis samples 40 databases per corpus and compares held-out LightGBM predictions using row-only, row-plus-parent, and row-plus-child-aggregate features.Child aggregates are mean, standard deviation, and count; R2 scores are clipped to [−1, 1], and parent gains include 95% bootstrap confidence intervals.Source: G Implementation and Reproducibility… · p. 19

Structural interventions.

  • The rel-stack controls use 24 fixed batches of 384 masked cells, and graded corruption tests ρ ∈{0, 0.25, 0.50, 0.75, 1}, the fraction of FK assignments corrupted.The random-initialization control uses the same RT architecture without pretraining.Source: Structural interventions · p. 19
  • Pathway tests separately disable child aggregation and parent access, while head tests ablate each of eight nbr heads and measure the increase in reconstruction loss.The broader structural-reliance probe uses 27 fixed batches from rel-stack, rel-arxiv, and rel-ratebeer; its absolute losses are not directly comparable to the 24-batch protocol.Source: Structural interventions · p. 19
  • Downstream evaluation covers nine RelBench tasks at five context lengths, measures classification by AUROC, and compares original, FK-shuffled, and nbr-ablated conditions on identical examples.Corrected 30k evaluations of two rare-label rel-stack tasks use n = 2048 examples.Source: Structural interventions · p. 19
  • Each generator is represented by one pretrained checkpoint, so mechanistic interventions measure within-checkpoint effects rather than variation across independent pretraining seeds.Predictive-necessity estimates instead report bootstrap uncertainty across target-column prediction problems.Source: Structural interventions · p. 19

Related papers

Loading 2610.03057v1…