Source-linked AI summary
Speedrunning Relational Foundation Model Pretraining
Gabriel Wedson Mendonça de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa, Francisco Galuppo Azevedo
TL;DR
Relational pretraining lacks a shared efficiency protocol, and zero-shot evaluation requires fixing a held-out database. modded-nanoRT calibrates that choice and opens a single-GPU speedrun with an open training stack; its opening recipe reaches the target faster than the reference and wins most reported pairs outside development.
Problem
Relational pretraining lacks an open, comparable speedrun protocol, and its zero-shot evaluation requires fixing a held-out database in advance.
Source: §1 · pp. 1–2Method
modded-nanoRT fixes a single-GPU relational pretraining task, calibrates the held-out database over 45-minute multi-seed runs, and opens the training stack with typed trilinear attention, Muon, and SIGReg.
Source: §1 · p. 2Results
Takeaways & Limitations
The fixed protocol and open stack let contributors compare and accumulate relational pretraining recipe improvements across submissions.
Source: §1 · p. 1, §6 · p. 6The benchmark is scoped to one H200, prescribed artifacts and evaluation, an official metric and target, and a 45-minute budget.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception. Recipes are developed inside individual labs, reported under private protocols, and do not accumulate. Speedrun leaderboards remove that barrier by decentralizing the search rather than the training run. The problem is fixed, the stack is open, entries are ranked by wall-clock time to a quality target, and one GPU is enough to hold the record. We introduce modded-nanoRT, the first such leaderboard for Relational Transformers. Which database to hold out decides whether the benchmark can resolve improvements at all, so we fix it by calibration, ranking six candidates by how sharply a 45-minute learning curve separates improvement from seed noise. We open the leaderboard with typed trilinear attention, a relational operator that conditions attention scores and values on the foreign key traversed, together with Muon and SIGReg. The recipe reaches the target in 305.63 seconds on one NVIDIA H200, 8.11x faster than the reference implementation, and wins 13 of 15 pairs outside the development database. We will release the benchmark, its frozen artifacts, and the record runs. Our code is available at https://github.com/kunumi/nanoRT.
1 Introduction
Relational pretraining lacks a shared speedrun protocol, and its zero-shot setting makes the held-out database a consequential design choice. modded-nanoRT fixes the benchmark through calibration, opens it with three recipe changes, and reports a faster record with gains beyond the development database.
1 Introduction
- Relational foundation models learn from database schemas, whereas standard pipelines flatten links into task-specific tables that must be prepared again for each task.RelBench standardized the setting, and RT demonstrated schema-aware transfer to unseen tasks without task-specific parameters or in-context examples [Robinson et al., 2024] [Ranjan et al., 2026a].Source: §1 · p. 1
- Community speedruns decentralize recipe search: contributors use separate GPUs and shared measurements so improvements can accumulate across entries.The training run itself is not decentralized; the search over recipes is.Source: §1 · p. 1
- modded-nanoRT fixes a single-GPU, zero-shot relational pretraining task and ranks open-stack submissions by wall-clock time to a target.The held-out database must be fixed in advance because relational pretraining mixes schemas and evaluates transfer to an unseen database.Source: §1 · pp. 1–2
- Six held-out database candidates are calibrated over 45 minutes and multiple seeds, ranked by how sharply learning curves separate improvement from seed noise.The selected held-out database determines whether the benchmark can resolve improvements.Source: §1 · p. 2
- Typed trilinear attention makes the traversed foreign-key relation a multiplicative factor in attention scores and messages, alongside SIGReg and Muon.Unlike standard RT, the relation can make neighbors score differently according to the foreign key traversed.Source: §1 · p. 2
- 305.63 seconds on one NVIDIA H200 reaches the target, 8.11× faster than the reference implementation; the recipe wins 13 of 15 pairs outside development.Matched three-seed comparisons attribute the gain to Muon and to typed attention only in combination with it.Source: §1 · p. 2
2 Related Work
Prior work established relational benchmarks, schema-aware pretrained models, and time-to-quality evaluation, but did not provide an open protocol for comparing relational pretraining efficiency. modded-nanoRT adapts the speedrun format to relational data, whose mixture and held-out database impose validity constraints.
2 Related Work
- Relational benchmarks, schema-aware pretrained models, and synthetic multi-table generation address evaluation, modeling, and data scarcity, but not comparable training-efficiency protocols.As a result, recipe improvements are reported against private baselines and do not compose.Source: §2 · p. 2
- Time-to-quality benchmarks and speedruns combine algorithmic and systems improvements with open submissions; modded-nanoRT applies this format to relational pretraining.Its database mixture and held-out database introduce validity constraints specific to relational pretraining.Source: §2 · p. 2
- Typed trilinear attention conditions messages on a database relation along a query–neighbor edge, rather than on a second neighbor as in simplicial architectures.Related work includes relation-specific message transformations [Schlichtkrull et al., 2017] and trilinear scores in higher-order attention [Clift et al., 2019, Roy et al., 2025].Source: §2 · p. 2
3 The modded-nanoRT challenge
The challenge fixes the training and evaluation protocol while leaving most of the training stack open. It calibrates the held-out database and freezes the mixture, validation artifact, metric, and rules so submissions can be compared on a common target.
3 The modded-nanoRT challenge
- The training stack remains open across architecture, optimizer, regularization, context sampling, precision, and systems implementation, while a full run fits on one accelerator.Table 1 lists the protocol’s fixed quantities.Source: §3 · p. 2, Table 1 · p. 3
- The training mixture uses five of six eligible databases, excluding the held-out database; binary classification tasks share one metric, and rel-Event is excluded for confirmed temporal leakage.Database seed counts are tempered with w(d) ∝ n(d)^0.5 to balance proportional and uniform sampling [Xue et al., 2021].Source: §3 · p. 2
- rel-Avito ranked first among six candidates in 45-minute, multi-seed leave-one-out calibration, using improvement and late-curve slope relative to seed variability.The target is the best pooled AUROC reached within the calibration budget, and the validation artifact was frozen with seed, hop-radius, and text-embedding settings.Source: §3 · p. 3
- Record eligibility depends only on pooled AUROC across the two rel-Avito tasks, user-visits and user-clicks.Predictions and labels are concatenated across tasks at each scheduled evaluation.Source: §3 · p. 3
- The nano-scale reference is Ranjan et al.’s Relational Transformer, with 12 blocks, width 256, eight heads, and 19.3M trainable parameters.Its pretraining uses bfloat16, contexts of 1024 cells, batches of 256, and AdamW.Source: §3 · p. 3
- Submissions use one H200, prescribed training and validation artifacts, an official metric and target, and a 45-minute budget; runs that miss the target remain right-censored failures.Checkpoint loading, training on rel-Avito, revealing masked targets, and modifying validation are prohibited.Source: §3 · p. 3
4 The record recipe
The record recipe makes foreign-key relations explicit in attention and combines typed trilinear attention with Muon and SIGReg. These changes modify the RT backbone and its training objective.
Typed and higher-order attention
- Relation embeddings from a text encoder add no parameters for unseen schemas, and relation-type distinctness matters more than embedding content.This contrasts with relation-specific message transformations, which allocate parameters per relation type.Source: §4 · p. 4
- Typed trilinear attention makes foreign-key relation a multiplicative factor in scores and messages, so the same neighbor can rank differently depending on its route.It replaces RT’s neighbor attention in all twelve blocks; the relation state is derived from a frozen text embedding of the foreign-key description.Source: §4 · pp. 3–4, Figure 1 · p. 5
Muon
- Muon replaces matrix-parameter momentum updates with semi-orthogonal updates, aiming to advance singular directions comparably during the early training regime measured by a speedrun.It uses a five-step Newton–Schulz iteration in bfloat16; AdamW handles encoders, decoders, normalization and scale parameters, biases, and the SIGReg projector.Source: §4 · p. 4
SIGReg
- SIGReg regularizes first-block hidden states by matching random one-dimensional projection distributions to N(0, 1), with cells grouped by table and column.The stated motivation is to prevent column representations from collapsing onto a low-dimensional region; the projector trains jointly with the backbone.Source: §4 · pp. 4–5
5 Results
The final recipe wins most matched comparisons across held-out databases, while the record reaches the target in 305.63 seconds. Component results attribute gains to Muon and to typed attention only in combination with it.
5.1 The record
- 305.63 seconds on one NVIDIA H200 reaches the target, 8.11× faster than the reference implementation.The record lineage spans the reference implementation’s 41:20 to the 305.63-second record; verification accounts for 59.9% of the record duration.Source: Figure 2 · p. 5, §5.1 · p. 5
5.2 Component attribution
- Muon improved RT in all three pairs by +2.56 points on average; typed attention lost 1.21 under AdamW but gained 3.56 with Muon+SIGReg, while SIGReg had no claimed final-quality effect.The SIGReg contrasts were −0.47 and +0.26 points; the authors retain it based on time-to-target evidence, not final quality.Source: §5.2 · pp. 5–6, Table 2 · p. 6
5.3 Transfer across held-out databases
- 13 of 15 matched pairs outside development on rel-Avito favor the final recipe, which averages +1.98 AUROC points there.Across all six holdouts, it wins 15 of 18 pairs, with mean and median gains of 2.14 and 2.53 points.Source: Table 3 · p. 6, §5.3 · p. 6
6 Conclusion
modded-nanoRT holds the problem fixed while leaving the training stack open, and its opening entry reaches the target 8.11× faster than the reference. The results point to evaluation-path improvements and schedule refitting as next priorities.
6 Conclusion
- 305.63 seconds on one H200 reaches the target, 8.11× faster than the reference implementation.Matched comparisons attribute the gain to Muon and to typed attention only in combination with it; outside the development database, the recipe wins 13 of 15 pairs.Source: §6 · p. 6
- rel-Amazon’s 12.86-point standard deviation exceeds every measured effect, and reported quantities select the best scheduled checkpoint over three seeds.Source: §6 · p. 6
- Verification consumes 59.9% of the record, and the first evaluation clears the target by 5.5 points; the next entry should optimize evaluation before the update loop.The conclusion also recommends refitting learning-rate schedules to the five-minute horizon and dropping the text encoder.Source: §6 · p. 6
A What the relation vector must carry
Fixed random relation vectors descriptively outperform the semantic and shuffled variants, but no pairwise difference is significant. The ablation tests whether the foreign-key vector must carry semantic content or only distinguish relation types.
A What the relation vector must carry
- The ablation contrasts semantic foreign-key embeddings with shuffled embeddings and random vectors assigned per type/layer or per edge, which removes type identity.The third attention factor may contribute semantic content or simply distinguish relation types.Source: A What the relation vector must carry · p. 8
- Table 4 reports the rel-Avito ablation over seven seeds; its reached-target count is the number of seeds whose best pooled AUROC exceeds 0.81621.Source: Table 4 · p. 8
- Fixed random type/layer vectors have the highest means, reach the target in every seed, and have less than half the record recipe’s best-AUROC standard deviation; no pair separates at α = 0.05.Shuffling does not reduce either reported mean, so the ordering remains descriptive.Source: A What the relation vector must carry · p. 8, Figure 3 · p. 8
B Reference calibration trajectory
The reference trajectory sets the target at its best pooled AUROC, which the record recipe exceeds much earlier in its run.
B Reference calibration trajectory
- 0.81621 is the reference trajectory’s best value at 41:20 and defines the target; the record recipe exceeds it at 5:05.6, reaching 0.8714.The 45-minute calibration trajectory comprises 11,621 updates, evaluated every 500; reported times are upper bounds set by evaluation grids.Source: Figure 4 · p. 9