Source-linked AI summary
Advancing Open and Reproducible Relational Learning: RelArena-$α$, TabPFN-Rel and RPI
Adrian Hayler, Klemens Flöge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum, Jonas Kübler, Siyuan Guo, Oscar Key, Jan Hendrik Metzen, Rylee Grace, David Salinas, Arthur Cahu, Simon Bing, Benjamin Jäger, Tuana Çelik, Mihir Manium, Vitor Monteiro, Jake Robertson, Jerry Chen, Eliott Kalfon, Tomás Pereda, Lilly Wehrhahn, Dominik Safaric, Tobias Schroeder, Georg Grab, Diana Kriuchkova, Clara Cornu, Philipp Singer, Nick Erickson, Vahid Balazadeh, Marie Salmon, Simone Alessi, Kürşat Kaya, Philipp Jund, Léo Grinsztajn, Yann LeCun, Bernhard Schölkopf, Madelon Hulsebos, Lennart Purucker, Sauraj Gambhir, Frank Hutter, Noah Hollmann
TL;DR
Relational learning lacks standardized, reproducible comparisons because methods use differing evaluation protocols and tuning budgets. This paper releases RelArena-α, TabPFN-Rel, and RPI; TabPFN-Rel ranks first among models under RelArena-α’s standardized regime, while tabular approaches remain competitive with relational deep learning baselines.
Problem
Relational learning lacks reproducible, comparable evaluation because methods use undisclosed procedures and differing evaluation regimes and tuning budgets.
Method
The paper releases RelArena-α for standardized RelBench comparisons, TabPFN-Rel as a relational harness, and RPI as a model-agnostic interface.
Results
TabPFN-Rel ranks first among models under RelArena-α’s standardized tuning regime, while tabular models remain competitive with relational deep learning baselines.
Takeaways & Limitations
Flattening relational databases into tables remains a strong strategy that can compete with relational deep learning models on benchmark tasks.
Takeaways & Limitations
Relational methods remain expensive and error-prone, with preprocessing costs and runtime differences making fully standardized tuning regimes an open problem.
Abstract
from arXiv · showhide
This first release of Prior Labs in relational learning shows our continued commitment to open science. We open-source three pieces of software that we expect to accelerate research in the field towards meaningful real-world impact. We aim to steer further development based on feedback from, and in collaboration with, the community. Given the early stage of development, our $α$-release targets researchers and early-adopting practitioners. Over the past years, a variety of datasets and tasks for relational learning have emerged, but the community has not converged on a reliable, reproducible way to compare different methods on these tasks. Our $α$-release, RelArena-$α$, provides a unified framework for running and comparing baselines on RelBench v1 by standardizing data loading, evaluation protocols, tuning regimes, and support for systems with custom tuning, inspired by established tabular benchmarks such as TabArena. We plan to work with the research community to further develop RelArena-$α$ into a catalyst for progress in the relational learning community. We release the initial version of TabPFN-Rel, a purpose-built relational harness for TabPFN-3. Currently ranked first among models on RelArena-$α$, TabPFN-Rel makes key improvements upon RDBLearn. Beyond its ranking, TabPFN-Rel serves as a strong baseline, adding to the growing evidence that flattening a relational database into a single table remains competitive with specialized relational architectures on real-world tasks. To facilitate adoption of relational learning methods in research and industry, we release an initial $α$-version of our Relational Predictive Interface, RPI, an open-source, model-agnostic interface that enables early adopters to easily define problems on new databases and apply any model implemented in RelArena-$α$, including TabPFN-Rel, to these problems.
1 The Current State of Relational Learning
Relational learning has grown rapidly, but progress and real-world impact are hindered by poor reproducibility and incomparable evaluation, tuning, data states, and aggregation practices. The field also needs pragmatic, competitive baselines and easier transfer from benchmark tasks to real-world databases.
- Reproducibility and comparability: Relational learning methods are often difficult to reproduce because training scripts, complete task coverage, or other implementation details are unavailable.Some releases provide dataset-specific checkpoints without training scripts, while others expose APIs whose scripts omit reported tasks.
- Reproducibility and comparability: Comparisons are confounded by undisclosed or mismatched tuning regimes, including search spaces ranging from 384 to over 25,000 configurations.The publications and codebases do not explain how these hyperparameters were obtained, making reported performance difficult to assess.
- Reproducibility and comparability: Subtle differences in evaluation regimes and database states can substantially change results and undermine the scientific integrity of method comparisons.Benchmark fixes may alter labels and invalidate older baseline results, which then need to be recomputed.
- Reproducibility and comparability: Even with standardized evaluation and tuning, the community lacks a shared convention for aggregating per-dataset results, especially for regression.RelBench uses ROC AUC for classification and MAE for regression, but reported summaries remain incompatible.
- Practical baselines and adoption: Researchers should compare relational methods with pragmatic tabular baselines, including manual and automated feature engineering, while making benchmark-to-real-world transfer easier.Automated feature-engineering baselines have achieved competitive performance, yet applying research methods to custom databases and prediction tasks remains unnecessarily difficult.
2 Towards Open, Reproducible, and Accessible Relational Learning
The release addresses reproducibility, comparability, pragmatic baselines, and accessibility in relational learning through RelArena-α, TabPFN-Rel, and RPI. These artifacts remain early-stage, with benchmark curation, tuning alignment, and interface safeguards still open challenges.
- RelArena-α: RelArena-α standardizes relational-learning data loading, evaluation, tuning, and model comparison through a unified framework built on RelBench v1.It retrieves databases at runtime under their respective licenses and treats the initial task set as a pragmatic starting point rather than a definitive benchmark.
- RelArena-α: RelArena-α reruns reported methods through a unified model API, aligning implementations, fixing bugs, reconstructing missing scripts, and providing reproducible baseline results.The authors report spending hundreds of GPU hours producing baseline results intended to be trustworthy and reproducible.
- RelArena-α: RelArena-α separates standardized-tuning model submissions from custom-tuning system submissions, enabling isolated methodological comparisons while accommodating novel end-to-end pipelines.Model submissions use controlled search and early stopping; system-to-system comparisons share input/output and time constraints, while efficiency comparisons across categories are not direct.
- RelArena-α: RelArena-α gives all methods the same data state during training, tuning, and evaluation, and includes GNN, foundation-model, aggregation-based, and learning-free baselines.Its baseline set includes GraphSAGE, RelGT, RelGNN, RT-PluRel, RDBLearn, TabPFN-Rel, and constant predictors.
- TabPFN-Rel: TabPFN-Rel, a TabPFN-3 relational harness built on RDBLearn, is currently the top-ranked model on RelArena-α and strengthens the case for flattening-based relational baselines.Its improvements include frozen tuning data, a larger TabPFN-3 context, text-feature support, and recency-diversity context selection.
- RPI: RPI is an open-source, model-agnostic interface integrated with RelArena-α that lets users apply baselines and tuning to their own relational prediction problems in two lines of code.Its initial design targets researchers and early adopters but has limited safeguards against task mis-specification because practitioner requirements remain incompletely understood.
3 Results
RelArena-α evaluates six model submissions and system submissions across 21 RelBench v1 entity-level tasks, finding TabPFN-Rel first among standardized-tuning models while exposing substantial runtime and tuning limitations. Tabular models remain competitive with relational baselines, but text features materially affect TabPFN-Rel performance.
- Main Results: TabPFN-Rel ranks first among models evaluated under RelArena-α’s standardized, compute-matched tuning regime.System submissions follow the data and evaluation regimes but use their own tuning protocols; RT-PluRel is the first such system.
- Main Results: TabPFN-Rel variants and RDBLearn are competitive with relational deep learning baselines, supporting database flattening as a strong relational-learning strategy.The results also recognize that relational models retain unique modeling advantages.
- Computational costs: The single-seed leaderboard required hundreds of hours of wall-clock time, and some methods are prohibitively slow on expensive datasets.TabPFN-Rel also requires a CPU-bound feature-synthesis pass before fitting, which can dominate runtime.
- Main Results: 1821 to 1706: removing text features lowers TabPFN-Rel’s Elo by 115 points, with the effect concentrated on rel-event/user-ignore and rel-avito/user-clicks.Text is decisive when tasks contain informative free-text columns and nearly irrelevant elsewhere.
- Main Results: Constant-per-entity exceeds RelGNN and RelGT on 4 tasks each, while TabPFN-Rel and RT-PluRel are the only methods that beat it on all 21 tasks.On rel-stack/post-votes, almost no method is significantly better than the constant predictor.
- Tuning: Tuning significantly benefits RelGNN, yields smaller improvements for GraphSAGE, and does not reliably improve most other RelArena-α methods.TabPFN-Rel’s shallow default remains competitive with deeper validated configurations, while greater aggregation depth raises computational costs exponentially.
4 Open Issues
RelArena-α still faces practical and methodological limitations, including costly preprocessing, uncertain data quality, missing baselines, task-type diversity, and non-standardized timestamp boundaries. These issues constrain reproducible comparison and the framework’s representativeness across relational learning applications.
- Running models is difficult: Relational learning methods remain more expensive and error-prone to run than tabular methods, often requiring hours of CPU-bound preprocessing per dataset.Examples include deep feature synthesis for RDBLearn and TabPFN-Rel, and graph materialization and tokenization for RelGT and RelGNN.
- Data quality requires further investigation: RelArena-α standardizes benchmarking on pre-existing entity-level RelBench v1 databases and tasks, leaving their quality and representativeness for practitioners’ day-to-day needs uncertain.The selected tasks are popular in the community, but their suitability for practical workloads requires further investigation.
- Missing baselines: RelArena-α may omit competitive baselines because of differing evaluation regimes, later release dates, or limited awareness.The team plans to work with authors and facilitates method additions through provided extensions.
- Disparate task types: RelArena-α currently covers only entity-level forecasting, while recommendation, attribute prediction, foreign-key prediction, and autocomplete tasks remain outside its scope.These task types can have subtle differences that make them mutually incompatible, limiting cross-task method transfer.
- Timestamp boundaries are not standardized: Timestamp boundaries are not standardized: GNN-based baselines and RT-PluRel may use rows at an entity’s test timestamp, whereas TabPFN-Rel and RDBLearn do not.This is permitted by the current API but might disadvantage flattening-based methods when test cut-off and entity timestamp align.
5 Outlook
The release is presented as the first milestone in Prior Labs’ long-term commitment to relational learning, with future work shaped by community collaboration and feedback. The authors conclude that open, reproducible progress depends on a non-dogmatic relational learning community.
- Future development: RelArena-α will evolve through community collaboration by adding baselines, standardizing tuning regimes, and addressing open questions about task types and data quality.These aims are framed as part of Prior Labs’ long-term commitment to advancing relational learning.
- Future development: TabPFN-Rel will be further co-developed alongside upcoming model releases and in response to community feedback.
- Conclusion: The authors argue that advancing open and reproducible relational learning requires the community to avoid dogmatic practices.They express optimism that the contributions can become an inflection point accelerating research progress and real-world impact.
A Authorship and Contributions
Adrian Hayler is the first-listed core contributor, while Klemens Flöge and Alan Arazi were randomized in the remaining core-contributor order. Contributions span research/model development, engineering/platform, applied work, GTM, operations/people, scientific advising, and founding.
- Core and External Contributions: Adrian Hayler is listed first among the core contributors, while Klemens Flöge and Alan Arazi were randomized in the remaining order.Rishabh Ranjan contributed the RT-PluRel system submission; external contributors Rishabh Ranjan and Jure Leskovec, along with the scientific advisors, provided scientific advice only and did not design or implement RelArena-α or contribute intellectual property.
- Research/Model Dev: Research/model development included Oscar Key, Philipp Jund, Vahid Balazadeh, Felix Birkel, Mihir Manium, Léo Grinsztajn, Arthur Cahu, Siyuan Guo, Tobias Schroeder, Jonas Kübler, David Salinas, Jan Hendrik Metzen, Anurag Garg, Jake Robertson, Benjamin Jäger, Nick Erickson, and Simon Bing.
- Engineering/Platform and Applied: Engineering/platform contributors were Dominik Safaric, Simone Alessi, Brendan Roof, and Georg Grab, while applied contributors were Philipp Singer and Eliott Kalfon.
- GTM and Ops/People: GTM contributors were Clara Cornu, Vitor Monteiro, Diana Kriuchkova, Tuana Çelik, and Lilly Wehrhahn, while operations/people contributors were Kürşat Kaya, Jerry Chen, Lydia Sidhoum, Rylee Grace, Marie Salmon, Tomás Pereda, and Kristina Collins.
- Advisors and Founders: Scientific advisors were Yann LeCun, Bernhard Schölkopf, and Madelon Hulsebos, and Sauraj Gambhir is listed as a founder.
B Tuning Regime in RelArena-α
RelArena-α standardizes how methods are tuned through common data, metrics, search-space formats, and evaluation procedures, but comparable tuning budgets remain unresolved. Its runtime policy and caching support address practical constraints while preserving inspectability of preprocessing artifacts.
- Standardization challenges: RelArena-α largely addresses how methods are tuned, but how much computation they receive remains unresolved despite substantial runtime-policy progress.The authors call for further iterations and broader academic-community involvement to achieve fully satisfactory comparability.
- Tuning procedure: RelArena-α standardizes tuning through registered search spaces, default configurations, inner-split training, and validation on each task’s primary metric.Search spaces are randomly sampled using the run seed or evaluated as fixed grids in predefined order.
- Challenges in standardizing budgets: More than 40 hours versus under 30 seconds: RelGT’s nine-configuration regime on rel-avito/user-visits contrasts with tuned LightGBM on the same task.The RelGT runtime excludes CPU-based preprocessing, which itself takes several hours.
- Challenges in standardizing budgets: CPU-bound preprocessing forced RelArena-α to abandon fully end-to-end execution, so methods may generate and publicly document cached preprocessing artifacts before runs.Examples include deep feature synthesis for RDBLearn and TabPFN-Rel, plus graph materialization or tokenization for RelGNN and RelGT; public scripts support reconstruction and label-leakage inspection.
- Current runtime policy: 24 hours is the current maximum total runtime per task, including preprocessing; all released baselines except RelGT run for less than 12 hours per task.Moderately larger search spaces are allowed on smaller tasks when their cost remains reasonable, but the constraints remain approximate.
- Released configurations: RelGT is the only RDL baseline with a publicly available tuning procedure, and its dataset-size-dependent grid can yield fewer effective trials on larger tasks than its nominal budget.The initial implementation therefore aimed to follow the authors’ published tuning regime as closely as possible.
C Comparison against self-reported results
The section compares RelArena-α results with authors’ self-reported results, excluding RDBLearn because five of 21 tasks lack reported results. Several methods show substantial Elo discrepancies, often associated with differences in tuning or evaluation regimes.
- Comparison setup: RelArena-α results are compared with authors’ self-reported results, while RDBLearn is excluded because results are missing for 5 of 21 RelBench v1 tasks.The comparison uses Table 3’s joint Elo board, which combines both result sets with constant baselines.
- Method discrepancies: GraphSAGE improves moderately under RelArena-α, likely partly because it benefits from the standardized tuning procedure.The passage characterizes GraphSAGE as having weaker self-reported performance.
- Method discrepancies: RelGNN’s Elo decreases by nearly 450 points relative to its self-reported results, likely because RelArena-α’s standardized budget insufficiently explores its huge configuration space.The authors believe tuning differences largely drive this discrepancy.
- Method discrepancies: RelGT’s Elo decreases by around 200 points, although the source is unclear despite close implementation fidelity and substantially higher runtime than other RelArena-α methods.The passage points to Appendix B for the runtime comparison.
- Method discrepancies: TabPFN-Rel improves slightly relative to the version reported in the TabPFN-3 model report.The passage contrasts this improvement with the larger discrepancies observed for RelGNN and RelGT.
- Limitations: Differences may reflect evaluation and tuning regimes, implementation details, or errors in re-implementations, motivating collaboration with original authors to improve fidelity and fairness.The authors state this as a broader qualification on comparisons between self-reported and RelArena-α results.
D Results tables per Task
Tables 4 and 5 provide per-task test results for every RelArena-α method across all 21 tasks. They use task-appropriate metrics and validation-selected configurations, supporting Elo ratings and bootstrapped confidence intervals.
- Evaluation scope: 21 RelArena-α tasks are evaluated for every method using per-task test performance.Tables 4 and 5 cover all tasks and methods in the benchmark.
- Evaluation protocol: Every reported test score comes from the configuration selected on validation data.The per-task measurements determine pairwise outcomes for Elo ratings and bootstrapped confidence intervals in Figure 2.
- Metrics: Classification tasks use AUROC, while regression tasks use MAE in their native units.Higher AUROC is better; lower MAE is better.
E Effect of Tuning per Method
Tuning is evaluated per method by comparing each validation-selected configuration with that method’s default on test across 21 tasks. The analysis also identifies when tuning leaves performance unchanged and notes that constant predictors cannot change.
- E Effect of Tuning per Method: An unchanged result occurs almost exactly when validation retains the default configuration.The unchanged count therefore indicates how often tuning finds nothing worth taking.
- E Effect of Tuning per Method: Table 6 counts, for each method, the tasks where validation-selected configurations beat, match, or lose to that method’s own default on test.The comparison covers 21 tasks.
- E Effect of Tuning per Method: Constant predictors cannot move because they have no search space.They therefore cannot produce a different selected configuration.
F The RT-PluRel System Submission
RT-PluRel is RelArena-α’s first system submission, using a pre-trained relational transformer with internal model selection and an empty external search space. Its sequential training, refit, and prediction procedures specify fine-tuning, validation-based step selection, retraining, and eight-seed ensembling.
- System submission: RT-PluRel is RelArena-α’s first system submission, registering an empty search space and running one configuration while performing model selection internally during each fit.It builds on an 85M-parameter pre-trained relational transformer representing databases with cell tokens and column, row, and foreign-key attention.
- Sequential tuning regime: Each fit uses sequential tuning: delta-fine-tuning with early stopping for up to 50,000 steps, followed by selection from context configurations sampled during training.Fine-tuning uses Muon, a constant learning rate of 5 · 10−4, batch size 256, and stochastic weight averaging.
- Refit and prediction: After validation-based selection, RT-PluRel retrains from the pre-trained checkpoint on training and validation data, then averages predictions across eight context-sampling seeds.The selected step count is scaled by the ratio of dataset sizes, with no further model selection during refitting.
G Data Provenance
RelArena-α distributes no data: all seven RelBench v1 databases are obtained at runtime through relbench and remain subject to their original sources’ terms. The section records provenance, attribution, consent, and retrieval requirements for the evaluated databases.
- Database provenance: RelArena-α distributes no data; all seven RelBench v1 databases are obtained at runtime through relbench under their original source terms.Users should consult the original sources before relying on the databases for their own purposes.
- Attribution for rel-stack: rel-stack contains Stack Exchange content contributed under CC BY-SA 3.0, whose attribution requirements apply to republishing the content.RelArena-α does not republish the dump but acknowledges the Stack Exchange communities and individual contributors.
- Acknowledgment for rel-event: rel-event was shared for academic use with its creators’ explicit consent, and RelArena-α uses it only as distributed by RelBench.The paper thanks Allan Carroll for sharing the Event Recommendation Engine Challenge data with the academic community.
- Retrieval route: RelBench requires users to obtain rel-avito, rel-hm, and rel-event from Kaggle themselves and accept the competition terms.This retrieval route is specified as part of accessing those datasets.