PaperlayerView exact revision on arXiv

Source-linked AI summary

RelBench: A Benchmark for Deep Learning on Relational Databases

Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, Jure Leskovec

arXiv:2407.20060v1cs.LGcs.AIcs.DB

TL;DR

Predictive modeling on relational databases often flattens tables through manual feature engineering, which can discard relational signal and demands substantial human work. RelBench introduces diverse databases, tasks, and an RDL implementation, then evaluates RDL against an experienced data scientist’s manually engineered models. RDL matches or outperforms those models while substantially reducing the work required.

  • Problem

    Relational data is commonly flattened for tabular models, foregoing relational predictive signal, while research on RDL lacks standardized databases and tasks, implementations, and effectiveness studies.

    Source: §1 · pp. 1–2

  • Method

    RelBench provides diverse relational databases and predictive tasks, an open-source RDL implementation, and a benchmark study comparing RDL with manually engineered tabular models.

    Source: §1 · p. 2

  • Results

    RDL outperforms the data scientist in 11 of 15 tasks, reduces hours worked by 96% and lines of code by 94% on average, and takes roughly 30 minutes per task versus 12.3 hours.

    Source: §6.1 · p. 10

  • Takeaways & Limitations

    RELBENCH provides a basis for further RDL research on accurate prediction over complex multi-tabular datasets without manual feature engineering.

    Source: §8 · p. 12

    The RDL implementation is an initial demonstration, with scope for performance improvements, including improved output heads for regression tasks.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

We present RelBench, a public benchmark for solving predictive tasks over relational databases with graph neural networks. RelBench provides databases and tasks spanning diverse domains and scales, and is intended to be a foundational infrastructure for future research. We use RelBench to conduct the first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024), which combines graph neural network predictive models with (deep) tabular models that extract initial entity-level representations from raw tables. End-to-end learned RDL models fully exploit the predictive signal encoded in primary-foreign key links, marking a significant shift away from the dominant paradigm of manual feature engineering combined with tabular models. To thoroughly evaluate RDL against this prior gold-standard, we conduct an in-depth user study where an experienced data scientist manually engineers features for each task. In this study, RDL learns better models whilst reducing human work needed by more than an order of magnitude. This demonstrates the power of deep learning for solving predictive tasks over relational databases, opening up many new research opportunities enabled by RelBench.

1 Introduction

Relational data’s primary–foreign-key links can carry predictive signal, but typical workflows flatten databases through manual feature engineering, losing signal and adding software complexity. RELBENCH supplies benchmark infrastructure and open-source RDL tools, and its user study finds RDL matches or outperforms an experienced data scientist’s models while sharply reducing human effort.

1 Introduction

  • Flattening relational data for tabular models can lose predictive signal and requires extraction pipelines that introduce bugs and software complexity.The prior workflow commonly relies on manual feature engineering.Source: §1 · p. 1
  • RDL represents each database entity as a graph node, primary–foreign-key links as edges, and raw-table features as initial node representations for GNN prediction.The passage names this approach Relational Deep Learning (RDL) (Fey et al., 2024).Source: §1 · p. 1
  • RELBENCH provides databases across domains and scales, with 30 practical classification, regression, and recommendation tasks.Its databases range from 74K to 41M entities, with 3–15 tables and 15–140 columns.Source: §1 · p. 2
  • RELBENCH releases a Python package, a modifiable open-source RDL implementation, and a public leaderboard, alongside benchmarking against baselines.The benchmark is intended to provide infrastructure for future RDL research.Source: §1 · p. 2
  • RDL models match or outperform the data scientist’s models while reducing human hours by 96% and lines of code by 94% on average.The comparison uses an experienced data scientist who manually engineers features and feeds them into tabular models.Source: §1 · p. 2

2 Overview and Design

RELBENCH pairs relational databases with predictive tasks and is designed to support diverse benchmark comparisons, leakage-aware evaluation, and future research flexibility. Its implementation provides an end-to-end RDL model over a graph representation of the database.

2 Overview and Design

  • Each predictive task is defined by a training table containing entity IDs, seed times, and target labels; seed time determines when the target is predicted and future data is filtered.The database itself comprises tables connected by primary–foreign-key relationships, with some tables carrying entity-creation times.Source: §2 · p. 2
  • The benchmark curates real-world databases for diverse domains and task types, including entity classification, regression, and recommendation.Datasets vary in rows, columns, tables, and time span; tasks include practical problems such as churn prediction and post recommendation.Source: §2 · p. 3
  • Temporal splits train before VAL_TIMESTAMP, validate between VAL_TIMESTAMP and TEST_TIMESTAMP, and test afterward.The implementation hides post-TEST_TIMESTAMP data during inference and uses temporal neighbor sampling to avoid training and validation time leakage; designers remain responsible for avoiding leakage generally.Source: §2 · p. 3
  • Shared database splits support multi-task learning and pre-training, while exposed graph-conversion logic permits alternative graph constructions and uses of raw data.The implementation represents the database as a heterogeneous temporal graph and predicts over nodes or recommendation links.Source: §2 · p. 3

3 Relational Deep Learning Implementation

The RDL implementation turns relational tables and their primary–foreign-key links into a heterogeneous temporal graph, then learns task-specific predictions from table features and sampled neighborhoods. Temporal neighbor sampling ensures sampled information predates each prediction’s seed time.

3 Relational Deep Learning Implementation

  • Tables become node types, rows become nodes, and primary–foreign-key relationships become graph edges in a heterogeneous temporal graph.Nodes carry features from diverse table columns, represented with Tensor Frame, including numerical, categorical, timestamp, and text types.Source: §3 · p. 4
  • A ResNet tabular model encodes raw rows into initial node embeddings, which heterogeneous GraphSAGE updates using sum-based neighbor aggregation before end-to-end prediction.Task-specific prediction heads consume the resulting node embeddings.Source: §3 · p. 4
  • Temporal neighbor sampling builds a subgraph around each entity at its seed time and ensures every sampled node appears before that time, avoiding training-time leakage.The sampled subgraph is fed to the GNN to predict the target label.Source: §3 · p. 4
  • Entity classification uses an MLP with binary cross-entropy, while entity regression uses L1 loss.Recommendation uses either a two-tower GNN scored by embedding inner product with Bayesian Personalized Ranking loss, or an ID-GNN with an MLP head and binary cross-entropy.Source: §3 · p. 4

4 RELBENCH Datasets

RELBENCH contains seven processed real-world relational databases spanning domains and dataset sizes, with multiple predictive tasks associated with each. The examples range from historical Formula 1 records and clinical trials to e-commerce, social, and advertising data.

4 RELBENCH Datasets

  • Seven datasets with rich relational structure span diverse domains and sizes, and each is associated with multiple predictive tasks.Figure 2 illustrates the rel-trial schema, whose databases have complex relational structure and rich column features.Source: §4 · p. 4, Figure 2 · p. 4
  • rel-amazon covers book-related products, users, and reviews, while rel-hm contains customer purchase histories and product metadata.Amazon review data includes ratings, purchase status, and review text.Source: §4 · p. 5
  • rel-f1 records Formula 1 stakeholders, circuits, standings, races, and detailed season data dating back to 1950.The records include practice sessions, qualifying positions, sprints, and pit stops.Source: §4 · p. 5
  • rel-stack uses the stats-exchange site and includes user biographies, posts, comments, edits, votes, and related posts.Its source is the Stack Exchange question-and-answer network.Source: §4 · p. 5
  • rel-trial is curated from AACT and contains clinical-trial protocols, results, study designs, participant demographics, interventions, and outcomes.AACT consolidates studies registered on ClinicalTrials.gov.Source: §4 · p. 5
  • rel-event contains anonymized user actions, event metadata, demographic information, and social relations; rel-avito contains searches and ad attributes for ad-click prediction.rel-event is drawn from the Hangtime mobile app; rel-avito comes from the Avito Context Ad Clicks dataset on Kaggle.Source: §4 · pp. 5–6

5 Predictive Tasks on RELBENCH Datasets

RELBENCH organizes 30 tasks into three predictive types that vary in scale and entity overlap. Across classification, regression, and recommendation, RDL matches or outperforms baselines, with GraphSAGE or ID-GNN often performing best on recommendation tasks.

5 Predictive Tasks on RELBENCH Datasets

  • The benchmark introduces 30 tasks across classification, regression, and link prediction, varying in entity counts, repeated entities, and train-test overlap.Tasks without overlap are pure inductive tasks; others are partially transductive.Source: §5 · p. 6

5.1 Entity Classification

  • RDL matches or outperforms classification baselines, while LightGBM approaches it on study-outcome with 28 target-table columns.The rich target-table features offer LightGBM potentially useful signals without feature engineering; extracting and combining these with cross-table information remains an open research question.Source: §5.1 · pp. 6–7

5.2 Entity Regression

  • RDL matches or outperforms regression baselines, though driver-position and study-adverse show performance matching up to statistical significance.The results suggest room for improvement in regression-task performance.Source: §5.2 · p. 7

5.3 Recommendation

  • GraphSAGE or ID-GNN performs best on recommendation tasks, often by a significant margin.ID-GNN excels when predictions are entity-specific, while plain GNN excels in the reverse case.Source: §5.3 · p. 8, Table 5 · p. 8

6 Expert Data Scientist User Study

The user study compared RDL with an experienced data scientist who manually engineered features and used tabular models. RDL matched or exceeded the expert’s predictive performance on most tasks while substantially reducing reported effort.

6 Expert Data Scientist User Study

  • The expert workflow comprised exploratory analysis, feature ideation, SQL feature engineering, tabular modeling, and optional post-hoc feature-importance analysis.The study capped exploration at four hours and ideation at one hour, left SQL feature-engineering time unconstrained, and supplied a standardized LightGBM training script with hyperparameter tuning.Source: §6 · pp. 8–9

6.1 Results

  • RDL matched or outperformed the data scientist in 11 of 15 tasks, while reducing human hours by 96% and new lines of code by 94% on average.The reported average time was 12.3 hours per task for the data scientist and roughly 30 minutes for RDL.Source: Figure 3 · p. 9, Figure 4 · p. 10, §6.1 · p. 10

6 Expert Data Scientist User Study

  • The study recruited an experienced data scientist and released the work for reproducibility.The participant had five years of machine-learning experience in finance.Source: §6 · pp. 9–10

6.1 Results

  • RDL performed best against the data scientist on classification tasks, while regression remained more difficult and some expert models still performed better.The authors identify the GNN’s MLP output head as poorly suited to regression and applied boosting to improve RDL performance.Source: §6.1 · pp. 10–11
  • RDL required 96% fewer work hours, with solutions taking under an hour, because much of its GNN architecture and training-loop code could be reused across tasks.The effort comparison measures marginal work and excludes reusable infrastructure.Source: §6.1 · pp. 10–11
  • RDL needed 56 new lines of code on average, compared with 878 ± 77 for the data scientist’s workflow.The RDL addition was task-specific training-supervision code; the expert’s count included EDA, SQL, and data-formatting code.Source: §6.1 · p. 11

7 Related Work

The paper situates RelBench among graph-learning benchmarks while distinguishing its focus on reframing existing relational-database prediction tasks as graph-learning problems.

  • Established graph-learning benchmarks include Open Graph Benchmark, TUDataset, and Temporal Graph Benchmark.The paper notes that such benchmarks have sustained graph machine learning’s growth and maturation.Source: §7 · p. 11
  • Unlike benchmarks that collect tasks already recognized as graph-learning problems, RelBench recasts tasks commonly solved by other methods as graph-learning tasks.The authors argue that this expands the problem space for graph machine learning.Source: §7 · p. 11
  • Relational Deep Learning builds on graph neural networks but also calls for advances in tabular learning, temporal learning, and graph construction.Prior work explored GNN architectures on relational data, while Fey et al. (2024) proposed an end-to-end framework treating temporality as a core concept.Source: §7 · pp. 11–12

8 Conclusion

RelBench introduces a benchmark to support relational deep learning research, combining diverse databases and practical predictive tasks with an open-source implementation. The paper reports validating the approach against manual feature engineering by an experienced data scientist.

  • RelBench is introduced as a benchmark intended to facilitate research on relational deep learning.Source: §8 · p. 12
  • The benchmark provides diverse, realistic relational databases and practical tasks covering entity-level prediction and entity link prediction.Source: §8 · p. 12
  • The paper provides the first open-source RDL implementation and reports validation against manual feature engineering by an experienced data scientist.Source: §8 · p. 12

A Additional Task Information

The appendix lists predictive tasks and reports detailed results and implementation choices for RelBench experiments.

A Additional Task Information

  • The task list spans node-level prediction and link prediction, including customer churn, clinical-trial outcomes, and recommendations.Examples include predicting user visits or clicks, trial outcomes, and user-item purchases.Source: A Additional Task Information · pp. 15–16

B.1 Detailed Results

  • Tables 6–8 report means and standard deviations over five runs for entity classification, entity regression, and link prediction.Source: §B.1 · p. 16

B.2 Hyperparameter Choices

  • The default experiments share a two-layer GNN configuration and sampling setup, with learning rate and maximum epochs varying across task types.Rel-trial node tasks used modified settings after the defaults underperformed; rel-trial ID-GNN link prediction used a four-layer GNN.Source: §B.2 · p. 16

B.3 Ablations

The ablations examine graph structure, node and text features, and temporal awareness; the accompanying workflow outline lists the data scientist’s main modeling stages.

B.3 Ablations

  • Randomly permuting primary-foreign-key destinations makes GNN performance much worse, underscoring the importance of the relational graph structure.The comparison is between primary-foreign-key edges and randomly permuted destinations.Source: §B.3 · p. 18, Figure 6 · p. 19
  • Rich node features are crucial, while GloVe and BERT text embeddings consistently outperform omitting text features; BERT is usually better, especially for classification.The ablations compare features against zero-masked variants.Source: §B.3 · p. 18, Figure 7 · p. 19, Figure 8 · p. 20
  • Adding relative time embeddings significantly improves performance across a diverse range of tasks.The embedding uses the time span between an entity timestamp and the querying seed time.Source: §B.3 · pp. 18–19, Figure 9 · p. 20

C.1 Data Scientist Example Workflow

  • The data scientist workflow proceeds through exploratory data analysis, feature ideation, feature engineering, tabular machine learning, and optional feature-importance analysis.The final analysis examines trained-model performance and feature importance.Source: §C.1 · pp. 20–21

C.1.1 Exploratory Data Analysis

The workflow moves from exploring a dataset and proposing features to implementing and evaluating them. Its regression analysis finds that a LightGBM output head can improve GNN-learned features, while highlighting leakage risks and output-head limitations.

C.1.1 Exploratory Data Analysis

  • EDA examines table schemas and relationships, label statistics and class imbalance, temporal train/validation/test splits, field distributions and aggregations, relationships between columns, and outliers or unusual data patterns.In the user-churn task, roughly 82% of samples have label 1, an imbalance not enough to strictly require up-sampling. EDA takes in the order of a few hours (3-4 for most datasets in the user study).Source: §C.1.1 · p. 21

C.1.2 Feature Ideation

  • Feature ideation turns EDA and task judgment into candidate predictors, from directly available customer age to more complex purchase-history aggregates.The ideation phase takes 30 minutes to one hour in the user study.Source: §C.1.2 · p. 21, Figure 10 · p. 22

C.1.3 Feature Engineering

  • Feature engineering implements proposed predictors with joins, aggregations, and window functions, while requiring timestamp filters to prevent leakage.Validation-to-training leakage in study-outcome produced extremely high validation performance but test performance below LightGBM without feature engineering; the features were later fixed.Source: §C.1.3 · p. 22

C.1.4 Tabular Machine Learning

  • Engineered tables are converted into numerical arrays for LightGBM and hyperparameter tuning; the user-churn process took one to two hours.The study used five optimization runs with ten trials each and reported means and standard deviations over the runs.Source: §C.1.4 · p. 23

C.1.5 Post-hoc Analysis

  • SHAP analysis of user-churn identifies all-time behavior statistics and temporal information capturing seasonality as the most predictive feature groups.The analysis presents the top 30 features and their SHAP-value distributions.Source: §C.1.5 · p. 23, Figure 11 · p. 23

C.2 Regression Output Head Analysis

  • Converting regression tasks to binary classification narrows the performance gap between data scientist models and GNNs, while LightGBM on GNN features helps regression but not classification.The authors attribute the regression lift to the linear output head’s limitations, and note task formulation also affects performance.Source: §C.2 · p. 24, Table 10 · p. 24, Table 11 · p. 24

D Dataset Origins and Licenes

The section identifies the data sources and usage conditions for RELBENCH datasets, with some source-specific constraints and snapshot details.

  • rel-amazon comes from the Amazon Review Data Dump, while rel-stack was downloaded from the Internet Archive in November 2023; neither has a specified data license.Source: D Dataset Origins and Licenes · p. 25
  • rel-avito and rel-hm come from Kaggle competitions whose terms restrict use to non-commercial and academic purposes.The avito dataset is further downsampled by selecting approximately 100,000 user-table data points and connected records.Source: D Dataset Origins and Licenes · p. 25
  • rel-f1 uses Ergast API data sourced in February 2024, and rel-trial is a January 2024 ClinicalTrials.gov snapshot that will not receive newer trial results.The rel-event dataset originated in a Kaggle challenge, and its creators consented to its use in RELBENCH.Source: D Dataset Origins and Licenes · p. 25

E Additional Training Table Statistics

The appendices provide task-specific training statistics and database diagrams, and describe broader-impact scope and implementation extensions.

E Additional Training Table Statistics

F Dataset Schema

G Broader Impact

  • The current implementation must load the entire database into working memory, which is not viable for very large datasets; fully inductive link prediction with unseen tails remains out of scope.A query-based sampler for entities and their primary-foreign-key neighbors is suggested for large datasets.Source: G Broader Impact · pp. 32–33

Related papers

Loading 2407.20060v1…