PaperlayerView exact revision on arXiv

Source-linked AI summary

PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models

Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik, Vijay Prakash Dwivedi, Johannes Hoffart, Carlos Guestrin, Jure Leskovec

arXiv:2602.04029v2cs.DBcs.AIcs.LG

TL;DR

Relational foundation models need diverse multi-table training data, but privacy and business constraints make such databases scarce. PLUREL synthesizes relational databases through graph-based schemas, bipartite key connectivity, and conditional causal feature generation; experiments reveal scaling laws, improved real-data generalization, and stronger continued-pretraining bases.

  • Problem

    Privacy and business constraints make diverse relational databases largely inaccessible, limiting the public data available for training RFMs.

  • Method

    PLUREL generates databases from scratch using directed schema graphs, bipartite primary–foreign connectivity, and conditional Structural Causal Models for features.

  • Results

    Pretraining loss exhibits power-law scaling with synthetic database count and dataset size, while larger and more diverse synthetic data improves generalization and supports continued real-data pretraining.

  • Takeaways & Limitations

    Synthetic data scaling provides a practical paradigm for expanding RFM pretraining beyond scarce public relational databases.

  • Takeaways & Limitations

    The scaling-law hypothesis is tested under a model that assumes power-law dependence and a feature implementation focused on numeric and categorical data.

Abstract

from arXiv · show

Relational Foundation Models (RFMs) facilitate data-driven decision-making by learning from complex multi-table databases. However, the diverse relational databases needed to train such models are rarely public due to privacy constraints. While there are methods to generate synthetic tabular data of arbitrary size, incorporating schema structure and primary-foreign key connectivity for multi-table generation remains challenging. Here we introduce PLUREL, a framework to synthesize multi-tabular relational databases from scratch. In a step-by-step fashion, PLUREL models (1) schemas with directed graphs, (2) inter-table primary-foreign key connectivity with bipartite graphs, and, (3) feature distributions in tables via conditional causal mechanisms. The design space across these stages supports the synthesis of a wide range of diverse databases, while being computationally lightweight. Using PLUREL, we observe for the first time that (1) RFM pretraining loss exhibits power-law scaling with the number of synthetic databases and total pretraining tokens, (2) scaling the number of synthetic databases improves generalization to real databases, and (3) synthetic pretraining yields strong base models for continued pretraining on real databases. Overall, our framework and results position synthetic data scaling as a promising paradigm for RFMs.

1. Introduction

PLUREL addresses the scarcity and structural complexity of public multi-table data for training relational foundation models by generating diverse relational databases synthetically. Its experiments show scalable pretraining behavior and benefits from synthetic data before continued real-data pretraining.

  • Privacy and business constraints leave relational databases largely inaccessible, limiting the diverse public training data available for RFMs.
  • Existing isolated-table generators omit primary–foreign key relationships that shape row-level information locality and joint distributions in real databases.
  • PLUREL synthesizes relational databases through directed schema graphs, bipartite primary–foreign connectivity, and conditional SCM-based feature generation.
  • Power-law scaling appears with both the number of synthetic databases and total pretraining tokens when neither axis bottlenecks the other.
  • +7.4% and +5.2% absolute improvements are reported for classification AUROC and regression R2, respectively, after synthetic pretraining followed by continued real-data pretraining.

2. Synthetic Relational Data Generation

PLUREL generates relational databases in three stages: schema structure, row-level primary–foreign connectivity, and feature values from conditional causal mechanisms. The framework also introduces hierarchical connectivity and temporal source patterns to produce diverse, non-i.i.d. relational data.

  • Schema Generation via Directed Graphs: Stage 1 samples a directed acyclic schema graph and independently samples table row and feature-column counts.
  • Foreign Key Generation via Bipartite Graphs: Stage 2 populates foreign keys by modeling bipartite connectivity between parent and child table rows.
  • Foreign Key Generation via Bipartite Graphs: A Hierarchical Stochastic Block Model controls whether child rows connect broadly across parent rows or concentrate on smaller subsets.
  • Feature Generation via Structural Causal Models: Stage 3 assigns each table a Structural Causal Model and generates rows in the schema graph’s topological order, conditioning child tables on synthesized parents.
  • Feature Generation via Structural Causal Models: The current implementation focuses on numeric and categorical features, while the framework can extend SCM mechanisms to richer modalities.
  • Feature Generation via Structural Causal Models: Temporal source inputs combine trend, cyclical, and fluctuation components, avoiding i.i.d. feature generation for rows sharing foreign keys.

3. Experiments

Experiments evaluate PLUREL-driven scaling for RT across synthetic data diversity and size, generalization to RelBench, and continued pretraining on real databases.

  • Experimental Setup: RT is pretrained on billions of synthetic tokens while varying PLUREL database diversity and token-count dataset size.The experiments study pretraining loss, zero-shot RelBench performance, and continued pretraining on real data.
  • Scaling Laws: Validation loss is modeled as a function of synthetic RDB diversity N and pretraining tokens S, with separate power-law hypotheses for each axis.N denotes the number of synthetic RDBs, while S denotes extracted pretraining tokens.
  • Scaling Laws: The scaling study varies synthetic RDB counts from 8 to 1024 and token budgets from 0.5B to 32B.Each grid combination is evaluated using held-out synthetic contexts from 100 RDBs.
  • Scaling Laws: Scaling-frontier points roughly follow fitted log-log power laws, but achieving the best loss requires increasing diversity and dataset size together.Holding either axis fixed produces nonmonotonic curves, limiting a single monotonic joint power law.
  • Generalization to Real Datasets: Across 18 RelBench tasks, larger synthetic datasets help most when paired with more synthetic RDBs; low diversity can make larger datasets worsen validation loss.AUROC and R2 show similar patterns, while loss saturates because RelBench is out-of-distribution for the synthetic data.
  • Continued Pretraining on Real Datasets: Synthetic plus real continued pretraining improves average zero-shot performance over real-only pretraining by +1.2% AUROC and +3.0% R2.The gains reach +7.4% AUROC and +5.2% R2 on individual tasks, while synthetic-only pretraining underperforms both baselines on most tasks.

4. Related Work

Related work spans foundation models, synthetic tabular data, relational foundation models, scaling laws, and relational database generation, while positioning PLUREL against gaps in relational pretraining data.

  • Foundation Models: Foundation models benefit from massive, diverse data, but relational databases are rarely public because they contain sensitive user or enterprise information.This limits access to the relational data needed for large-scale RFM pretraining.
  • Synthetic Data and Tabular Foundation Models: Synthetic tabular generators model column relationships, but isolated tables omit primary-foreign key connectivity across rows and tables.That connectivity shapes information locality and the joint distributions RFMs must learn.
  • Relational Foundation Models: Existing relational foundation models rely on real-world databases, with Griffin using mainly single-table pretraining and only 14 databases.The related-work discussion states that no prior work proposed a synthetic generator specifically for RFM pretraining.
  • Scaling Laws: Prior scaling-law studies characterize performance against data, model, compute, or cell-count scale in language, vision, and tabular models.The cited work motivates analyzing scaling behavior for relational foundation models.
  • Relational Database Generation: Privacy-preserving relational synthesis generally reproduces the structure and statistics of an existing database, rather than generating diverse databases from scratch for pretraining.Related methods also synthesize relational databases for testing and benchmarking analytical applications.

5. Conclusion and Future Work

The paper concludes that PLUREL enables scalable synthetic pretraining for RFMs and identifies extensions involving data design, modalities, training strategies, and joint scaling.

  • Conclusion: PLUREL pretraining loss follows a power-law trend as synthetic database count and dataset size increase.The result is reported from experiments with the Relational Transformer.
  • Conclusion: Larger and more diverse synthetic datasets improve generalization to previously unseen real data.This conclusion is part of the paper's reported RT experiments.
  • Conclusion: Synthetic pretraining produces robust base models that enhance subsequent pretraining on real data.The paper presents this as a central outcome of its experiments.
  • Future Work: Future work includes relational data curation, text extensions, semi-synthetic augmentation, synthetic-real curricula, long-context modeling, test-time scaling, and joint model-data scaling laws.These directions are proposed as ways to extend PLUREL and synthetic RFM pretraining.

Impact Statement

The impact statement frames PLUREL as a privacy-preserving way to expand access to relational pretraining data, with potential relevance across enterprise applications.

  • Impact: PLUREL can synthesize relational databases with configurable schemas, connectivity patterns, and data distributions without requiring real business or consumer data.The statement connects this capability to foundation models and relational deep learning.
  • Impact: Potential application areas include business intelligence, fraud detection, consumer analytics, healthcare, and supply chains.The statement presents these as possible enterprise and societal applications.

A. Limitations

PLUREL currently supports primary–foreign key connectivity only between a table and a different parent table, excluding self-loops.

  • PLUREL does not currently support self-loop primary–foreign key connectivity within the same table.For example, it cannot model a posts table whose ParentID foreign key refers to its own Id primary key.

B. Synthesizing Databases with PLUREL

PLUREL synthesizes relational databases through schema graphs, primary–foreign key connectivity, and table-specific causal mechanisms. Its design samples diverse graph structures and generates connected rows and features in stages.

  • Schema generation: PLUREL samples a directed schema graph whose nodes represent tables and whose edges represent inter-table relationships.The framework supports arbitrary directed-graph classes and uses graph structure to determine relational context.
  • Schema generation: Barabasi-Albert, Reverse Random-Tree, and Watts-Strogatz graph families model hub tables, hierarchies, and table clusters.Edges are sparsified or rewired for additional diversity.
  • Schema generation: For each table, PLUREL samples metadata including type, row count, feature columns, primary-key names, and foreign-key columns.Entity versus activity status is assigned from schema connectivity, and row counts are conditioned on table type.
  • Connectivity generation: PLUREL generates foreign keys by clustering child and parent rows into hierarchical blocks and sampling block-level bipartite connectivity.Each child row receives a parent primary-key index sampled from the induced connectivity structure.
  • Feature generation: Each table receives a Structural Causal Model whose graph and exogenous inputs generate feature values while incorporating parent-table values through foreign keys.Parent-table features are projected, aggregated with exogenous inputs, and reconstructed into type-specific feature values.

B.4. Computational Efficiency

PLUREL generation is computationally lightweight, while the Relational Transformer represents relational data as cell tokens and prepares bounded relation-aware contexts for prediction.

  • Computational efficiency: Generation latency increases approximately linearly from 2.5 seconds for 10 tables to 26.1 seconds for 80 tables.Across ten seeds, throughput is roughly 0.2–0.4 seconds per table and peak memory is about 1 GB at 80 tables.
  • Relational Transformer: RT represents each relational-database cell as one token containing its value, column name, and table name.Type-specific processing projects numeric, boolean, datetime, and text values into a shared embedding space.
  • Relational Attention: Relational Attention uses column, feature, and neighbor masks to model same-column patterns, row and parent-row links, and child-row relationships.A full bidirectional layer is omitted in these RT models because of limited utility on RelBench tasks.
  • Context preparation: RT constructs each seed-row context with bounded breadth-first traversal until a fixed cell budget is reached.Foreign–primary links are followed immediately, primary–foreign links are fan-out limited, and later timestamps are excluded.

D.1. Error Bars for Main Experiments

Synthetic pretraining followed by real-data pretraining generally improves zero-shot performance over real-only training, although some tasks degrade. The reported uncertainty estimates use three random seeds, and continued pretraining is robust to the base model.

  • Evaluation results: Synthetic-plus-real pretraining consistently improves over real-only RT training, with slight degradations on some tasks.The degraded tasks coincide with settings where table semantics were previously important, and PLUREL-generated values do not depend on table names or column names.
  • Evaluation results: Table 4 reports means and standard errors across three random seeds for the evaluated pretraining strategies.The table compares synthetic-only and synthetic-plus-real settings, including configurations with 1024 RDBs and 4B tokens or 512 RDBs and 16B tokens.
  • Evaluation results: Continued pretraining remains robust to the choice of base model, and a worse synthetic-only model can become better after synthetic-plus-real training.This pattern is identified as post-hoc reversal and motivates post-hoc model selection.

D.2. Architectural Improvements: Query-Key Normalization

Query-Key Normalization reduces RT’s sensitivity to initialization and improves baseline behavior during leave-one-database-out pretraining. It mitigates early overfitting while preserving higher peak performance across several RelBench tasks.

  • QK-Norm applies RMSNorm across each attention head’s dimension to reduce sensitivity to RT initialization.
  • Reducing variance across seeds: 9.4% and 10.5% maximum AUROC differences across seeds fall to 3.4% and 2.2% with QK-Norm on rel-amazon item-churn and user-churn.The comparison uses four model seeds with synthetic pretraining on 1B tokens.
  • Effects on baseline performance: Without QK-Norm, RT exhibits early overfitting on certain tasks, especially binary classification, and achieves lower peak performance.These effects were observed during leave-one-database-out pretraining on RelBench data.
  • Effects on baseline performance: 3.1% lower mean test AUROC and 3.7% lower mean test R2 occur without QK-Norm in the Real-only baseline.
Loading 2602.04029v2…