PaperlayerView exact revision on arXiv

Source-linked AI summary

RelBench v2: A Large-Scale Benchmark and Repository for Relational Data

Justin Gu, Rishabh Ranjan, Charilaos Kanatsoulis, Haiming Tang, Martin Jurkovic, Valter Hudovernik, Mark Znidar, Pranshu Chaturvedi, Parth Shroff, Fengyu Li, Jure Leskovec

arXiv:2602.12606v2cs.LG

TL;DR

Relational learning needs benchmarks that preserve the multi-table and temporal structure of real databases as models scale toward relational foundation models. RelBench v2 expands the benchmark with larger datasets, autocomplete tasks, and external integrations, and reports that RDL outperforms single-table baselines across multiple task types.

  • Problem

    Existing foundation-model approaches emphasize single-table data, leaving limited benchmark coverage for realistic multi-table relational structure and temporal prediction.

  • Method

    RelBench v2 adds four datasets, introduces temporally constrained autocomplete tasks, and integrates TGB, ReDeLEx, and 4DBInfer resources for broader relational evaluation.

  • Results

    RDL consistently outperforms LightGBM and other single-table baselines across autocomplete, forecasting, regression, classification, and recommendation evaluations.

  • Takeaways & Limitations

    RelBench v2 provides a broader unified basis for evaluating relational models across diverse datasets, temporal settings, and predictive objectives.

Abstract

from arXiv · show

Relational deep learning (RDL) has emerged as a powerful paradigm for learning directly on relational databases by modeling entities and their relationships across multiple interconnected tables. As this paradigm evolves toward larger models and relational foundation models, scalable and realistic benchmarks are essential for enabling systematic evaluation and progress. In this paper, we introduce RelBench v2, a major expansion of the RelBench benchmark for RDL. RelBench v2 adds four large-scale relational datasets spanning scholarly publications, enterprise resource planning, consumer platforms, and clinical records, increasing the benchmark to 11 datasets comprising over 22 million rows across 29 tables. We further introduce autocomplete tasks, a new class of predictive objectives that require models to infer missing attribute values directly within relational tables while respecting temporal constraints, expanding beyond traditional forecasting tasks constructed via SQL queries. In addition, RelBench v2 expands beyond its native datasets by integrating external benchmarks and evaluation frameworks: we translate event streams from the Temporal Graph Benchmark into relational schemas for unified relational-temporal evaluation, interface with ReDeLEx to provide uniform access to 70+ real-world databases suitable for pretraining, and incorporate 4DBInfer datasets and tasks to broaden multi-table prediction coverage. Experimental results demonstrate that RDL models consistently outperform single-table baselines across autocomplete, forecasting, and recommendation tasks, highlighting the importance of modeling relational structure explicitly.

1 INTRODUCTION

Relational deep learning models relational databases directly, addressing the limits of single-table approaches as foundation models expand toward richer data settings. RelBench v2 broadens evaluation with new datasets, autocomplete objectives, and external benchmark integrations.

  • Motivation: Relational deep learning models entities and relationships across interconnected tables instead of flattening schemas into single tables through manual feature engineering.This approach treats relational databases as heterogeneous graphs and applies graph-based and relational representation-learning architectures.
  • Motivation: Single-table foundation models do not capture the multi-table structure of relational databases, motivating benchmarks with realistic relational structure.Relational databases are presented as a natural next frontier for foundation models because of their ubiquity and structural richness.
  • Prior benchmark: RelBench v1 established standardized forecasting evaluation for relational learning models using real-world databases and tasks.Its evaluations demonstrated the effectiveness of relational deep learning compared with traditional approaches.
  • Contributions: RelBench v2 adds four large-scale datasets, raising the benchmark to eleven datasets with over 22 million rows across 29 tables.The new datasets span scholarly publications, enterprise resource planning, consumer platforms, and clinical records.
  • Contributions: RelBench v2 introduces autocomplete tasks that infer missing values in relational tables from relational and temporal context while preventing information leakage.It also integrates Temporal Graph Benchmark datasets, ReDeLEx access to 70+ databases, and 4DBInfer resources.

2 OVERVIEW AND DESIGN

RelBench v2 retains temporally split relational evaluation while adding autocomplete tasks that predict existing table attributes from observed values and relational context. The design also addresses leakage control, practical applications, and external benchmark integration.

  • Benchmark design: RelBench uses real-world relational databases paired with predictive tasks, with training, validation, and test data separated by fixed temporal cutoffs.Data after the test cutoff is hidden during inference to prevent test-time leakage.
  • Autocomplete tasks: Autocomplete tasks predict existing table columns rather than target labels constructed through SQL queries.At a seed time, models receive observed row values and infer a target using only available relational and temporal context.
  • Autocomplete tasks: Autocomplete tasks extend relational prediction toward applications such as recommending a payment category in enterprise sales-order interfaces.The SALT task inspired this formulation by using other fields and relational-schema context to recommend missing information.
  • Task construction: Leakage control requires removing columns that are highly correlated with the target column based on domain knowledge and training-data inspection.This design choice is necessary because correlated table columns can make autocomplete targets directly informative.
  • RDL implementation: The benchmark evaluates an RDL implementation using row embeddings, temporal-aware subgraph sampling, and heterogeneous GraphSAGE processing.The implementation passage describes the core embedding and temporal-neighborhood construction steps.
  • Paper organization: The paper organizes dataset descriptions, autocomplete and forecasting tasks, benchmark results, and external ecosystem integrations across subsequent sections.This provides the paper’s stated roadmap for the benchmark expansion.

3 RELBENCH DATASETS

RelBench v2 adds four datasets and expands coverage across scholarly, enterprise, consumer, and clinical relational data. The datasets differ substantially in scale, structure, modalities, and access conditions.

  • Dataset expansion: Four new datasets bring RelBench’s total to eleven and extend coverage into scholarly citations and enterprise operations.The expansion is intended to widen the benchmark’s breadth for relational foundation-model evaluation.
  • rel-arxiv: The arXiv-physics dataset contains over 222,000 papers, 1.5 million citation links, 143,000 authors, and 53 physics categories.It models dense many-to-many relationships among papers, authors, citations, and research areas.
  • rel-salt: The SALT dataset represents enterprise sales transactions linking document headers and line items with customer and address master data.Its relational structure captures workflows involving sales offices, shipping points, and payment terms.
  • rel-ratebeer: RateBeer contains more than two decades of interactions across beer, place, user, and brewer tables with multimodal attributes and detailed reviews.Its tables include text, categorical, temporal, and multi-aspect interaction features.
  • rel-mimic: MIMIC-IV provides deidentified electronic health records from Beth Israel Deaconess Medical Center for clinical relational learning.The implementation supports a standard 20,000-patient subset and access to the full dataset through Google BigQuery, subject to credentials.

4 AUTOCOMPLETE TASKS

RelBench v2 introduces 23 autocomplete tasks for predicting existing categorical or numerical columns, evaluated with relational context against table-based and statistical baselines. Across classification and regression, RDL generally outperforms these baselines, including under class imbalance and sparse features.

  • Task design: 23 autocomplete tasks cover binary and multiclass classification plus regression on existing dataset columns.Tasks are grouped into autocomplete classification and autocomplete regression, with task names based on target tables and columns.
  • Classification results: RDL strongly outperforms LightGBM across all reported binary and multiclass autocomplete classification tasks.Binary classification uses AUROC, while multiclass classification uses accuracy; both metrics are higher-is-better.
  • Robustness: On sales-office, RDL matches or slightly improves upon the majority baseline despite strong class imbalance, while LightGBM shows unstable and substantially worse test performance.The comparison suggests relational models can remain effective when target labels are imbalanced or feature information is sparse.
  • Regression results: RDL achieves higher R2 values than feature-based and aggregation baselines across most autocomplete regression tasks.The reported comparison indicates that relational context improves explanatory power for these regression predictions.

5 NEW FORECASTING TASKS

RELBENCH v2 expands forecasting beyond binary entity prediction to multiclass classification, regression, and recommendation, while evaluating whether relational models improve these tasks over single-table baselines.

  • Task coverage: RELBENCH v2 adds 13 forecasting tasks spanning entity classification, entity regression, and recommendation.Forecasting targets are constructed with SQL queries; recommendation predicts future temporal links between entity sets.
  • Entity classification: The benchmark introduces its first entity-level multiclass classification task through rel-arxiv’s author-category prediction.Binary and multiclass classification use the same metrics as corresponding autocomplete tasks, with binary classification evaluated by ROC-AUC and multiclass classification by accuracy.
  • Entity classification: RDL outperforms LightGBM on all new entity classification tasks, with especially large gains on the multiclass author-category task.The reported advantage is attributed to relational signals from coauthorship, citation patterns, and publication context.
  • Entity regression: RDL outperforms baselines across all new entity regression tasks, achieving higher R2 values and consistently lower MAE.The regression baseline uses raw features from the single entity table, whereas RDL incorporates relational information.
  • Recommendation: GraphSAGE or ID-GNN generally performs best on recommendation, with ID-GNN favoring entity-specific predictions and GraphSAGE favoring structural patterns.Increasing depth from two to four layers generally improved performance, although four-layer GraphSAGE exceeded available 80GB A100 memory.

6 INTEGRATING EXTERNAL BENCHMARKS INTO RELBENCH

RELBENCH v2 broadens evaluation beyond its native datasets by translating temporal event benchmarks into relational schemas and incorporating large external relational collections and task suites.

  • Temporal Graph Benchmark: TGB event streams are normalized into relational tables to compare temporal GNNs with RDL models under strict chronological evaluation.The translated tasks cover dynamic link property prediction, dynamic node property prediction, and temporal heterogeneous graph link prediction.
  • Integration scope: The external integrations extend RELBENCH’s coverage across temporal learning, real-world relational databases, and multi-dimensional benchmarking.Together, TGB, ReDeLEx, and 4DBInfer connect complementary datasets, tasks, and evaluation settings.
  • Temporal Graph Benchmark: The converted TGB datasets retain leakage-safe temporal cutoffs and span scales from small bipartite graphs to databases with tens of millions of events.Each becomes a RELBENCH database with parquet tables, schema metadata, temporal cutoffs, and tasks.
  • ReDeLEx: ReDeLEx adds more than 70 real-world relational databases spanning healthcare, government, education, sports, and business applications.Its unified pipeline connects to SQL databases, infers attribute semantics, and represents schemas as heterogeneous graphs.
  • 4DBInfer: RELBENCH v2 incorporates 7 4DBInfer datasets and 12 tasks under consistent relational, temporal, and graph-based evaluation conventions.4DBInfer varies datasets, tasks, relational-to-graph construction strategies, and predictive model families.

7 CONCLUSION

RELBENCH v2 expands the scale, task coverage, and ecosystem of relational deep learning benchmarks, supporting evaluation across diverse relational and temporal settings. Its results consistently favor explicitly modeling relational structure over single-table baselines.

  • Conclusion: RELBENCH v2 adds large-scale datasets across academic, enterprise, consumer, and clinical domains, substantially increasing benchmark diversity.The benchmark is positioned as infrastructure for evaluating relational deep learning systems and relational foundation models.
  • Conclusion: Autocomplete tasks complement forecasting and recommendation by predicting missing attributes within existing relational tables under temporal constraints.This extends relational prediction beyond targets created through SQL queries.
  • Conclusion: Integrating TGB, ReDeLEx, and 4DBInfer enables unified evaluation across relational and temporal learning settings.The integrations connect event-stream benchmarks, relational database collections, and multi-table task suites.
  • Conclusion: RDL models consistently outperform single-table baselines across autocomplete, forecasting, and recommendation tasks.The conclusion highlights relational structure as central to these benchmark results.

B DATASET SCHEMAS

The paper illustrates relational schemas for the SALT and arXiv-physics datasets and connects autocomplete to completing missing fields in an existing sales-order record.

  • Dataset schemas: Figure 1 presents the RELBENCH schema for the Sales Autocompletion Linked Business Tables dataset.SALT is the enterprise sales-order dataset introduced in RELBENCH v2.
  • Dataset schemas: Figure 2 presents the RELBENCH schema for the arXiv-physics dataset.The dataset captures scholarly publication data in the benchmark’s relational format.
  • Autocomplete example: The sales-order autocomplete example treats response fields as one dataset row and predicts terms of payment from the fields already entered.This example motivates autocomplete as filling missing attributes in an existing record.

C.2 LIST OF PREDICTIVE TASK DESCRIPTIONS

RELBENCH v2 defines predictive tasks across autocomplete, forecasting, and recommendation settings, covering entity attributes, interactions, and outcomes across multiple datasets. The task suite includes binary and multiclass classification, regression, and ranking objectives.

  • Forecasting tasks: Forecasting classification predicts future ratings, churn or dormancy, patient ICU stays, citations, and author research categories.These tasks use future outcomes such as zero activity over a specified horizon or at least one citation within six months.
  • Autocomplete tasks: Autocomplete classification covers user interactions, study properties, search behavior, sales orders, badges, and event-related attributes.Tasks include interest labels, monitoring committees, participant eligibility, click behavior, and badge class.
  • Autocomplete tasks: Autocomplete tasks predict missing values in existing relational-table columns, including user attributes, transaction prices, ratings, and sales-order fields.Examples include birth year, item price, beer ratings, and enterprise attributes such as plant, shipping point, and incoterms.
  • Autocomplete tasks: Autocomplete regression includes predicting race positions, transaction prices, beer ratings, study enrollment, and other numeric attributes.The listed tasks span rel-f1, rel-hm, rel-ratebeer, and clinical-study datasets.
  • Dataset coverage: The task descriptions span newly added datasets such as RateBeer and MIMIC-IV, alongside an SAP S/4HANA sales-order autocomplete example.The figures identify schemas for RateBeer and MIMIC-IV and illustrate payment-term prediction from filled response fields.
  • Forecasting and recommendation tasks: Forecasting regression and recommendation tasks predict future activity counts or rank users’ likely beers, places, and co-citations.Examples include user rating counts, favorite or liked items, and papers likely to be co-cited.

D.1 RESULTS WITH STANDARD DEVIATIONS

The benchmark reports mean and standard deviation over five runs for autocomplete, entity, and recommendation tasks, using task-appropriate classification, regression, and ranking metrics. Regression results are presented with R2 in the main paper and MAE in supplementary tables.

  • Evaluation protocol: 5 runs provide mean and standard-deviation results for autocomplete, entity classification and regression, and link-prediction evaluations.The reported task families include autocomplete classification and regression, entity classification and regression, and link prediction.
  • Classification metrics: AUROC measures binary classification, with higher values indicating better performance and random or majority baselines near 50.00.This convention is stated for both autocomplete and entity binary-classification tables.
  • Classification metrics: Accuracy measures multiclass classification, with higher values indicating better performance.The convention applies to autocomplete and entity multiclass-classification results.
  • Regression metrics: R2 measures regression predictive power in the main paper, while MAE is reported separately with lower values preferred.The supplementary regression tables distinguish R2 and MAE reporting.

D.3 RECOMMENDATION TASK ABLATIONS

Recommendation ablations show that node position and relational depth matter, while the usefulness of deeper message passing depends on the information available in intermediate tables. The experiments also describe temporal relational-data representations and scalable training infrastructure.

  • Node position: ID-GNN significantly outperforms plain GraphSAGE and trains faster on RateBeer recommendation tasks by capturing node-specific patterns.The paper links this advantage to multi-hop link prediction and the importance of node position and community context.
  • Multi-hop recommendation: Deeper GNNs improve user-beer-liked prediction when intermediate rating tables provide strong preference signals, but can add weaker or noisier information for sparse tables.The contrasting paths involve repeated User–Ratings–Beer traversal versus timestamped Favorites links.
  • Experimental setup: The benchmark’s recommendation results use MAP, include four-layer GNN and ID-GNN variants, and specify task-specific default hyperparameters.RateBeer training uses batch size 64 for selected models because of GPU memory constraints, while four-layer GraphSAGE remains prohibitively memory-intensive.
  • Temporal relational representation: Translated temporal datasets represent events with timestamps and foreign-key links, preserving typed relational semantics for relation-conditioned tasks.The schema uses entity tables and event tables, with separate tables by node or edge type where required.
  • Scalable infrastructure: Large event streams are stored in parquet and converted into disk-backed CSR structures so sampled GNN training avoids holding the full graph in memory.Streaming scans, memory-mappable arrays, and chunked writers keep conversion and neighbor sampling memory-bounded.
  • Baseline models: The translated TGB evaluation compares graph-native GraphSAGE, event-as-node relational GraphSAGE, and temporal attention models under controlled budgets.These model families provide complementary graph-native, schema-faithful, and temporal baselines.

E.4 RESULTS

The translated TGB experiments compare graph-native, relational, and temporal baselines across dynamic node, link, and heterogeneous link prediction. Results vary by dataset: temporal attention can lead on some heterogeneous graphs, while relational neighborhoods remain strong when typed structure is informative.

  • Translated TGB results: Tables 22–24 report validation and test results for dynamic node-property, dynamic-link, and temporal-heterogeneous link prediction.Metrics include NDCG@10, sampled-negative MRR, and sampled-negative MRR@100.
  • Temporal heterogeneous links: On temporal heterogeneous datasets, attention-based temporal models are competitive and sometimes best on thgl-software and thgl-github.The comparison is between RelEventSAGE and TGN with attention.
  • Temporal heterogeneous links: Relational GraphSAGE remains strong on thgl-forum and thgl-myket, where type-correct relational neighborhoods provide a high-signal inductive bias.The reported advantage is tied to the structure of typed relational neighborhoods.
  • Recommendation reference: On existing RELBENCH recommendation tasks, TGN with attention can substantially improve MAP on rel-f1 but does not uniformly dominate across domains.The reference comparison uses smoke runs on three recommendation tasks.
Loading 2602.12606v2…