PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Open and Reproducible Relational Learning: RelArena-$α$, TabPFN-Rel and RPI

Adrian Hayler, Klemens Flöge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum, Jonas Kübler, Siyuan Guo, Oscar Key, Jan Hendrik Metzen, Rylee Grace, David Salinas, Arthur Cahu, Simon Bing, Benjamin Jäger, Tuana Çelik, Mihir Manium, Vitor Monteiro, Jake Robertson, Jerry Chen, Eliott Kalfon, Tomás Pereda, Lilly Wehrhahn, Dominik Safaric, Tobias Schroeder, Georg Grab, Diana Kriuchkova, Clara Cornu, Philipp Singer, Nick Erickson, Vahid Balazadeh, Marie Salmon, Simone Alessi, Kürşat Kaya, Philipp Jund, Léo Grinsztajn, Yann LeCun, Bernhard Schölkopf, Madelon Hulsebos, Lennart Purucker, Sauraj Gambhir, Frank Hutter, Noah Hollmann

arXiv:2608.16319v1cs.LG

TL;DR

Relational learning lacks reproducible methods and standardized guidance for controlled comparisons. This paper introduces RelArena-α, TabPFN-Rel, and RPI. On RelArena-α, TabPFN-Rel ranks first among models under the same tuning regime, while RT-PluRel leads when systems are included.

  • Problem

    The community lacks reproducible methods and standardized guidance for comparing models and systems under controlled evaluation regimes.

    Source: §1 · p. 2

  • Method

    The paper develops RelArena-α to compare RelBench baselines and introduces TabPFN-Rel as a relational harness and RPI as an interface for defining prediction problems.

    Source: §2.1 · p. 6, §2.2 · p. 8, §2.3 · p. 8

  • Results

    On RelArena-α, TabPFN-Rel ranks first among models under the same tuning regime, while RT-PluRel leads when systems are included; tabular models are competitive with relational deep-learning baselines.

    Source: §3.2 · p. 9

  • Takeaways & Limitations

    The results add to evidence that flattening relational databases into tables can compete with relational deep-learning models, which retain unique modeling advantages.

    Source: §3.2 · p. 9

    Fully standardizing tuning regimes across methods remains an open problem because preprocessing requirements and runtimes vary substantially.

    Source: §4 · p. 11

Abstract

from arXiv · show

This first release of Prior Labs in relational learning shows our continued commitment to open science. We open-source three pieces of software that we expect to accelerate research in the field towards meaningful real-world impact. We aim to steer further development based on feedback from, and in collaboration with, the community. Given the early stage of development, our $α$-release targets researchers and early-adopting practitioners. Over the past years, a variety of datasets and tasks for relational learning have emerged, but the community has not converged on a reliable, reproducible way to compare different methods on these tasks. Our $α$-release, RelArena-$α$, provides a unified framework for running and comparing baselines on RelBench v1 by standardizing data loading, evaluation protocols, tuning regimes, and support for systems with custom tuning, inspired by established tabular benchmarks such as TabArena. We plan to work with the research community to further develop RelArena-$α$ into a catalyst for progress in the relational learning community. We release the initial version of TabPFN-Rel, a purpose-built relational harness for TabPFN-3. Currently ranked first among models on RelArena-$α$, TabPFN-Rel makes key improvements upon RDBLearn. Beyond its ranking, TabPFN-Rel serves as a strong baseline, adding to the growing evidence that flattening a relational database into a single table remains competitive with specialized relational architectures on real-world tasks. To facilitate adoption of relational learning methods in research and industry, we release an initial $α$-version of our Relational Predictive Interface, RPI, an open-source, model-agnostic interface that enables early adopters to easily define problems on new databases and apply any model implemented in RelArena-$α$, including TabPFN-Rel, to these problems.

1 The Current State of Relational Learning

Relational learning has attracted growing interest, but progress and real-world impact are hindered by interlocking problems in reproducibility, comparability, result aggregation, pragmatic baselines, and access to methods in practice.

1.1 Methods are not reproducible

  • Undocumented tuning spaces reach 384 combinations and over 25,000 combinations, while released code may omit training scripts or task coverage, obscuring configuration selection.The passages note that publications and codebases do not explain how these hyperparameters were obtained.Source: §1.1 · pp. 2–3

1.2 Methods are not comparable

  • On rel-f1, methods allowing later test entities’ database context gain years of signal; applying RelBench’s fixed-cutoff regime reduces ROC AUC by over ten points and increases MAE by over 40%.The impact is reported for the two classification tasks and the regression task, respectively; both evaluation regimes are valid but incomparable.Source: §1.2 · p. 3
  • A temporal-leakage fix lowered affected rel-event/user-ignore ROC AUC scores by two to six points, making self-reported results from different RelBench data states incomparable.The corrected labels were released in early 2026.Source: §1.2 · p. 3
  • Comparisons remain confounded when tuning differs: RelGNN [10] and RelGT [16] compare tuned results with untuned RDL results [1], and papers commonly copy self-reported baselines.Tuning can itself explain apparent performance differences, so standardized regimes are needed for fair comparisons.Source: §1.2 · pp. 3–4

1.3 No shared aggregation standard

  • Papers aggregate RelBench results using incompatible metrics, reference models, task subsets, and summaries, so aggregate numbers are rarely comparable and incomplete task reporting can favor selected methods.Regression summaries include normalized MAE, raw MAE, geometric means, target-standardized values, or R2; classification summaries generally average raw ROC AUC.Source: §1.3 · p. 4

1.4 Lack of pragmatic baselines

  • Flattened-table baselines reflect industry practice, but manual feature engineering with LightGBM is a standardized reference rather than a ceiling, while automated feature engineering has achieved competitive results [6] [7].A RelBench v2 autocomplete LightGBM baseline received only NaN test inputs, producing 50% ROC-AUC and negative R2.Source: §1.4 · pp. 4–5

1.5 No path from paper to practice

  • Applying relational methods to custom databases requires task-specific Python and model-integration code; proprietary PQL cannot express some multi-hop tasks, and most methods are not installable packages.The passages identify rel-avito/user-clicks and rel-trial/study-outcome as tasks whose labels require multi-hop aggregations.Source: §1.5 · p. 5

2 Towards Open, Reproducible, and Accessible Relational Learning

This early open-source release aims to advance reproducibility, comparability, and accessibility in relational learning through RelArena-α, TabPFN-Rel, and RPI, and invites community feedback and contributions.

2.1 RelArena-α

  • RelArena-α establishes a comparison framework for RelBench v1 entity-level forecasting, fetching licensed databases at runtime and treating this scope as a pragmatic, non-definitive starting point.Entity-level forecasting is widely used, while methods may not transfer across relational task types.Source: §2.1 · pp. 5–6
  • It re-runs reported methods through a unified API, repairs or reconstructs implementations as needed, and gives every method the same data state during training, tuning, and evaluation.The authors report spending hundreds of GPU hours producing baseline results.Source: §2.1 · p. 6
  • Standardized model submissions support isolated-method comparisons, while custom-tuned systems compare end-to-end pipelines; predictive performance spans categories, but efficiency and methodological effects do not.Model submissions use controlled search and validation early stopping; fully aligning tuning practices remains an open problem.Source: §2.1 · p. 6
  • Its initial baselines span graph neural networks, relational foundation models, aggregation-based approaches, and constant predictors; the set is incomplete but, to the authors’ knowledge, the most comprehensive unified evaluation to date.The benchmark will continue to grow its baseline set with community input.Source: §2.1 · p. 6

2.2 TabPFN-Rel

  • TabPFN-Rel is currently RelArena-α’s top-ranked model, showing that flattening relational databases into one table remains a strong baseline [6] [7].It builds on RDBLearn’s deep-feature-synthesis approach to convert relational tasks into flat tables.Source: §2.2 · p. 7
  • TabPFN-Rel freezes tuning data at the validation cutoff, uses TabPFN-3 with ten times more context rows at comparable runtime, and selects contexts by recency and diversity without extra runtime.It also reuses validation examples as test-time context and reattaches entity-table text columns for TabPFN-3-Plus through the API; a text-free version is available. The tuning-data fix also benefits other RelArena-α baselines, including RDBLearn.Source: §2.2 · pp. 7–8

2.3 Relational Predictive Interface: RPI

  • RPI is an open-source, model-agnostic interface integrated with RelArena-α that lets users define relational prediction tasks in YAML and run baselines, including tuning, in two lines of code.It provides documentation, an agent skill, specifications for all 21 RelBench v1 entity-level tasks, and a Kaggle example.Source: §2.3 · p. 8
  • RPI’s initial design targets researchers and early adopters but has limited safeguards against task mis-specification and supports only entity-level forecasting, so some database prediction tasks cannot yet be represented.The authors plan to work with the community on the expressivity-usability trade-off and to unify RPI’s specification standard with a similar independent effort.Source: §2.3 · p. 8

3 Results

On RelArena-α’s 21 RelBench v1 tasks, TabPFN-Rel ranks first among standardized-tuning model submissions, while RT-PluRel leads when custom-tuning systems are included. The results also show that flattened tabular approaches can compete with relational deep-learning baselines, while text features and tuning affect performance unevenly across tasks.

3.1 Experimental setup

  • The release evaluates six model submissions and a custom-tuned RT-PluRel system through RelArena-α, reporting a single-seed state across 21 entity-level RelBench v1 tasks.RelArena-α standardizes data loading, evaluation, and tuning regimes, but runtime alignment remains imperfect and may advantage more expensive methods; TabPFN-Rel does not have significantly higher runtime than other model submissions.Source: §3.1 · pp. 8–9

3.2 Elo rankings on RelArena-α

  • Tabular approaches including TabPFN-Rel and RDBLearn are competitive with relational deep-learning baselines, supporting database flattening as a strong strategy.The RDBLearn evaluation excludes LimiX from its original configuration, and RDBLearn v1.1 was released after the baseline cutoff.Source: §3.2 · p. 9
  • TabPFN-Rel ranks first among standardized-tuning models, while RT-PluRel with custom tuning achieves the highest end-to-end performance when systems are included.RT-PluRel’s result highlights the potential value of co-developing models and tuning regimes, while motivating investigation of how to transfer that benefit to other models.Source: §3.2 · p. 9, Figure 2 · p. 10
  • RT-PluRel runs on average five times slower than TabPFN-Rel (API) and 30 times slower than TabPFN-Rel (OSS), excluding preprocessing.The leaderboard required hundreds of hours; TabPFN-Rel also needs a CPU-bound feature-synthesis pass that can dominate runtime depending on dataset and hardware.Source: §3.2 · p. 10
  • Removing text features lowers TabPFN-Rel’s Elo from 1821 to 1706, with the effect concentrated in rel-event/user-ignore and rel-avito/user-clicks.Text is decisive when tasks contain informative free-text columns and close to irrelevant elsewhere.Source: §3.2 · p. 10
  • constant-per-entity beats RelGNN and RelGT on four tasks each, while only TabPFN-Rel and RT-PluRel exceed it on all 21 tasks.On rel-stack/post-votes, almost no method is significantly better than the constant predictor.Source: §3.2 · p. 10

3.3 Tunability in relational learning

  • Tuning significantly benefits RelGNN, yields smaller improvements for GraphSAGE, and does not appear to reliably improve most other methods.RT-PluRel is excluded because its system submission does not track the information needed for this isolated tuning analysis.Source: §3.3 · p. 10, Figure 3 · p. 11
  • TabPFN-Rel’s shallowest aggregation-depth default remains competitive with deeper validation-selected configurations, suggesting much predictive signal is available at depth two.Greater depth exponentially increases join paths to materialize and computational cost, making simpler featurization a potential efficiency improvement.Source: §3.3 · p. 11

4 Open Issues

RelArena-α is presented as a foundation for community collaboration, while important open issues remain in benchmark coverage, usability, task relevance, and fair evaluation.

Running models is difficult

  • Relational methods often require hours of CPU-bound preprocessing and have runtimes varying by orders of magnitude, making standardized tuning and broader adoption difficult.Preprocessing includes deep feature synthesis for flattening methods and graph materialization and tokenization for GNN methods; making methods cheaper and easier to run remains important.Source: §4 · p. 11

Data quality requires further investigation

  • The quality and day-to-day representativeness of the databases and entity-level RelBench v1 tasks used for benchmarking have not yet been established.These remain the community’s most popular tasks, but their underlying data and tasks need further investigation.Source: §4 · p. 11

Missing baselines

  • RelArena-α lacks potentially competitive baselines, including stronger tabular learners lifted through automated feature engineering; their strength and ability to close the gap remain future work.The team plans to work with authors to broaden baseline coverage and provides process documentation and skill files for agentic coding tools.Source: §4 · p. 12

Disparate task types

  • RelArena-α covers only entity-level forecasting [1]; adding recommendation [1], entity and relationship attribute prediction and foreign-key prediction [2], and autocomplete [4] requires resolving task incompatibilities and relevance.The authors plan extensions once they better understand which task types represent practitioners’ real-world predictive needs; initial investigations suggest task types differ in this respect.Source: §4 · p. 12

Timestamp boundaries are not standardized

  • Timestamp-context rules differ: GNN baselines and RT-PluRel may use rows at an aligned entity test timestamp, whereas TabPFN-Rel and RDBLearn do not, potentially disadvantaging the latter.Whether this context is permissible, task-dependent, or affects predictions is unclear; future releases plan to align boundaries across methods.Source: §4 · p. 12

5 Outlook

Prior Labs frames RelArena-α, TabPFN-Rel, and RPI as the first step in a long-term effort to advance open, reproducible relational learning. The outlook calls for continued community collaboration and non-dogmatic research on fair shared benchmarks.

5 Outlook

  • RelArena-α will add baselines, further standardize tuning regimes, and address open questions about task types and data quality with the academic and open-source community.These efforts are intended to develop the benchmark through collaboration.Source: §5 · p. 12
  • TabPFN-Rel will be co-developed with upcoming model releases and community feedback, while RPI will be iterated to improve accessibility for practitioners and bridge academia and industry.Prior Labs is also exploring ways to apply its structured-data foundation-model expertise to relational learning.Source: §5 · p. 12
  • The authors argue that lasting progress requires a non-dogmatic, welcoming, rigorous, and collaborative community in which graph, relational, and tabular researchers evaluate on the same fair benchmarks.They are optimistic that the contributions can accelerate research progress and real-world impact.Source: §5 · p. 12

A Authorship and Contributions

The section distinguishes core contributor credit and external scientific advice from the Prior Labs team roster. It also lists contributors by functional group, along with scientific advisors and the founder.

Core and external contributions

  • Adrian Hayler is listed first among core contributors, while Klemens Fl¨oge and Alan Arazi are randomized in order; Rishabh Ranjan contributed the RT-PluRel submission.Ranjan, Jure Leskovec, and the listed scientific advisors contributed scientific advice only; they did not design or implement RelArena-α or contribute intellectual property. The rest of the Prior Labs team is listed in random order per group.Source: A Authorship and Contributions · p. 17

Prior Labs team roster

  • The Research/Model Dev group includes 17 named team members.Oscar Key, Philipp Jund, Vahid Balazadeh, Felix Birkel, Mihir Manium, L´eo Grinsztajn, Arthur Cahu, Siyuan Guo, Tobias Schroeder, Jonas K¨ubler, David Salinas, Jan Hendrik Metzen, Anurag Garg, Jake Robertson, Benjamin J¨ager, Nick Erickson, and Simon Bing.Source: A Authorship and Contributions · p. 17
  • The Engineering/Platform group has four members, and Applied has two.Engineering/Platform: Dominik Safaric, Simone Alessi, Brendan Roof, and Georg Grab. Applied: Philipp Singer and Eliott Kalfon.Source: A Authorship and Contributions · p. 17
  • GTM lists five members, while Ops/People lists seven.GTM: Clara Cornu, Vitor Monteiro, Diana Kriuchkova, Tuana C¸ elik, and Lilly Wehrhahn. Ops/People: K¨urs¸at Kaya, Jerry Chen, Lydia Sidhoum, Rylee Grace, Marie Salmon, Tom´as Pereda, and Kristina Collins.Source: A Authorship and Contributions · p. 17

Advisors and founder

B Tuning Regime in RelArena-α

RelArena-α standardizes how methods are tuned, but how much tuning compute they receive remains unresolved. Its current regime combines automated validation-based selection with runtime limits and cached preprocessing, while leaving comparability challenges for future work.

B Tuning Regime in RelArena-α

  • RelArena-α largely standardizes how methods are tuned, but how much compute they receive remains unresolved and needs further iteration with broader community involvement.The two dimensions are what data, metric, and search spaces guide model selection, and how much computation is devoted to tuning.Source: B Tuning Regime in RelArena-α · p. 17

Tuning procedure

  • Methods register a standardized search space and default; RelArena-α selects configurations on inner validation data, refits the winner and default on the outer split, and reports both results.Search spaces are randomly sampled using the run seed or are fixed grids; methods may choose whether the final fit uses validation data or retains it for early stopping.Source: B Tuning Regime in RelArena-α · p. 17

Challenges in standardizing budgets

  • More than 40 hours on an RTX 6000 Pro GPU for RelGT on rel-avito/user-visits contrasts with under 30 seconds for tuned LightGBM; CPU-bound preprocessing also forced RelArena-α to abandon end-to-end execution.Preprocessing can take hours per dataset, while GPU nodes often have comparatively weak CPUs, making hundreds of hours of CPU preprocessing on those nodes difficult to justify economically.Source: B Tuning Regime in RelArena-α · pp. 17–18
  • RelArena-α permits methods to cache preprocessing artifacts if public scripts allow reconstruction and review, but cache-generation runtimes are not tracked, a limitation slated for future work.The untracked runtimes also leave unresolved how to amortize preprocessing shared across tasks from one database.Source: B Tuning Regime in RelArena-α · p. 18

Current runtime policy

  • The runtime cap is 24 hours per task; all released baselines except RelGT run under 12 hours, while untuned RT-PluRel runs a single configuration and reaches roughly 16 hours.RT-PluRel selects its training step and context internally during each fit and is listed separately from standardized tuning trials.Source: B Tuning Regime in RelArena-α · p. 18

Released configurations

  • LightGBM receives 30 trials, GraphSAGE 4, and RelGNN 10; fixed-grid methods evaluate the full grid with the default configuration first.Budgets are specified per run; sampled-space budgets are chosen primarily by computational cost.Source: B Tuning Regime in RelArena-α · p. 18
  • RelGT is the only RDL baseline with a publicly available tuning procedure, and its dataset-size-dependent grid can yield fewer trials on larger tasks; the authors intend to rerun methods under a revised regime.The initial implementation aimed to follow RelGT’s published regime as closely as possible.Source: B Tuning Regime in RelArena-α · p. 18

C Comparison against self-reported results

RelArena-α’s standardized results differ from authors’ self-reported results, but the paper treats Elo gaps as discrepancies rather than method-quality differences because evaluation and tuning regimes differ.

C Comparison against self-reported results

  • RelGNN’s Elo drops nearly 450 points and RelGT’s around 200, while TabPFN-Rel improves slightly relative to its model-report version.The comparison uses results from different evaluation and tuning regimes, so these gaps are discrepancies rather than differences in method quality.Source: Table 3 · p. 19, C Comparison against self-reported… · pp. 19–20
  • The comparison excludes RDBLearn because authors reported no results for 5 of 21 tasks, and RT-PluRel because no published results match its RelArena-α configuration.RT and PluRel papers evaluate earlier code versions without per-task fine-tuning of the PluRel-pretrained model.Source: C Comparison against self-reported… · p. 18, Table 3 · p. 19
  • GraphSAGE improves moderately under RelArena-α, likely in part because it benefits from the standardized tuning procedure.RelGNN’s decrease may reflect an insufficient standardized tuning budget for its huge configuration space, as well as implementation differences after re-implementation.Source: C Comparison against self-reported… · p. 19
  • RelGT’s discrepancy is less clear despite an implementation closely following the authors’ release and runtime substantially higher than any other RelArena-α method.The reported Elo decrease is around 200 points.Source: C Comparison against self-reported… · p. 20
  • Differences may also reflect implementation details or errors in re-implementations, and the authors aim to work with original authors toward faithful, fair representations within RelArena-α’s constraints.Table 3 places both result sets and constant baselines on a joint Elo board anchored at the global constant baseline.Source: Table 3 · p. 19, C Comparison against self-reported… · p. 20

D Results tables per Task

Tables 4 and 5 report validation-selected test performance across all 21 RelArena-α tasks, using AUROC for classification and native-unit MAE for regression. These per-task scores determine the pairwise outcomes used for Elo ratings and bootstrapped confidence intervals in Figure 2.

D Results tables per Task

  • Per-task test measurements determine the pairwise outcomes used to compute Elo ratings and bootstrapped confidence intervals in Figure 2.Tables 4 and 5 cover all 21 RelArena-α tasks.Source: D Results tables per Task · p. 20
  • Table 4 reports test AUROC for every method on the classification tasks, with higher scores better and the best per-task score bolded.Its entries are test scores for configurations selected on validation data.Source: Table 4 · p. 20, D Results tables per Task · p. 20
  • Table 5 reports test MAE in each regression task’s native units, with lower scores better and the best per-task score bolded.Its entries are test scores for configurations selected on validation data.Source: Table 5 · p. 20, D Results tables per Task · p. 20

E Effect of Tuning per Method

The section assesses per-method tuning by comparing each validation-selected configuration with that method’s default on test across 21 tasks. Unchanged performance almost exactly corresponds to validation retaining the default, indicating that the search found nothing worth taking.

E Effect of Tuning per Method

  • An unchanged test result almost exactly indicates that validation kept the default, so the search found nothing worth taking.Source: E Effect of Tuning per Method · p. 20
  • Table 6 counts, across 21 tasks, when each method’s validation-selected configuration beats, matches, or loses to its own default on test.Constant predictors have no search space and cannot change.Source: Table 6 · p. 21

F The RT-PluRel System Submission

RT-PluRel is RelArena-α’s first system submission, using a pre-trained relational transformer with internal model selection. Its sequential tuning, refit, and prediction process operates within timestamp-bounded relational contexts.

F The RT-PluRel System Submission

  • RT-PluRel is RelArena-α’s first system submission, using the 85M-parameter pre-trained relational transformer [20] [21] with an internally selected configuration.It registers an empty search space and builds each prediction context from the entity’s relational neighborhood, bounded by the split’s cutoff timestamp.Source: F The RT-PluRel System Submission · p. 21
  • Each fit delta-fine-tunes the pre-trained checkpoint with early stopping, then selects the best of 60 context configurations on validation data.Fine-tuning runs for up to 50,000 steps; context configurations are scored on up to 4,096 validation rows using four sampling seeds each.Source: F The RT-PluRel System Submission · p. 21
  • After selection, RT-PluRel refits on training plus validation data and averages predictions across eight context-sampling seeds; internal validation selection excludes it from standardized trial-budget accounting.The refit scales the selected step count by the dataset-size ratio and makes no further model selection.Source: F The RT-PluRel System Submission · p. 21

G Data Provenance

RelArena-α distributes no data: its seven RelBench v1 databases are obtained at runtime and remain subject to their original sources’ terms. The section also details provenance, attribution, and access conditions for specific databases.

G Data Provenance

  • RelArena-α distributes no data; its seven RelBench v1 databases are obtained at runtime and remain subject to their original source terms, which users should consult before relying on them.Table 7 lists the origin of each database, following RelBench’s provenance account [1].Source: G Data Provenance · pp. 21–22, Table 7 · p. 22
  • The rel-stack database contains Stack Exchange contributions under CC BY-SA 3.0; RelArena-α does not republish the content but acknowledges its communities and contributors.The dump’s license specifies attribution requirements for republishing the content.Source: G Data Provenance · p. 22
  • The Event Recommendation Engine Challenge data was shared for academic use with its creators’ consent to RelBench, and RelArena-α uses it only as RelBench distributes it [1].The section thanks Allan Carroll for sharing the data with the academic community [1].Source: G Data Provenance · p. 22
  • RelBench says users obtain rel-avito, rel-hm, and rel-event from Kaggle themselves and must accept the competition terms [1].Source: G Data Provenance · p. 22

Related papers

Loading 2608.16319v1…