Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

1 INTRODUCTION

• Record age changes with the prediction cutoff, whereas the interval between two observed linked records remains fixed.This motivates representing the two temporal signals separately.Source: §1 · pp. 1–2
• Jointly encoding record age and inter-record intervals improves aggregate performance over either signal alone on both backbones.Component ablations indicate that record age contributes more.Source: §1 · p. 2
• The staged schedules yield positive aggregate transfer in all four backbone–encoding settings, and the best schedule with both encodings beats either constituent objective alone.The schedule first uses subgraph contrast, then relation recovery or future relation activity prediction.Source: §1 · p. 2
• With both encodings, staged schedules improve over supervised training with the same encodings by 3.02% on the GNN and 1.06% on the GT.The best schedules also outperform evaluated pretraining baselines, including staged variants.Source: §1 · p. 2
• Rows become typed nodes and foreign-key links become typed edges; historical subgraphs include only information available by the prediction cutoff.In the illustration, a future purchase is excluded from the sampled neighborhood.Source: Figure 1 · p. 3

2.1 RELATIONAL DEEP LEARNING

• Relational deep learning maps database rows to nodes and foreign-key links to typed edges, with reverse edges included for message passing.Node types follow source tables, and relation types follow foreign-key relations.Source: §2.1 · p. 3

2.2 GRAPH MODELS AND TEMPORAL ENCODING FOR RDL

• Existing RDL encoders represent record age through temporal features and inter-record intervals through sinusoidal projections or query–key rotations.The evaluated databases also differ in record-age distributions and time scales, motivating the paper’s two temporal encodings.Source: §2.2 · p. 3

2.3 GRAPH PRETRAINING: HETEROGENEITY, TIME, AND STAGING

• Heterogeneous pretraining makes schema structure explicit because table and relation types specify which entities and links a target describes.The framework studies a transition from historical-subgraph contrast to typed relation recovery or future activity prediction, covering node, relation, and subgraph supervision.Source: §2.3 · p. 4

3 METHOD

• Each seed representation is computed from a sampled neighborhood containing only information available by prediction cutoff τ.Static tables contribute attributes available at the cutoff even when they lack timestamp columns.Source: §3 · p. 4

3.1 TEMPORAL ENCODING

• TIMEMIX adds learned, multi-scale age features to timestamped node attributes, using type-specific projections before graph propagation.Log-scaled ages preserve variation among recent records while compressing large ages; future records are excluded.Source: §3.1 · p. 5
• TIMEROPE converts signed intervals between timestamped edge endpoints into bounded rotation phases for GNN messages and GT attention.The sign distinguishes temporal order, while logarithmic scaling and clipping limit long-gap effects; untimestamped endpoints use unrotated operations.Source: §3.1 · p. 5

3.2 TEMPORAL PRETRAINING OBJECTIVES

• REL-HIST ranks observed typed links above same-type negative destinations using endpoint representations from historical neighborhoods at the link’s observation cutoff.The target edge and its reverse are removed from encoder inputs to prevent direct access to the answer.Source: §3.2 · p. 6
• REL-FUTURE predicts whether a seed receives a specified relation in a future horizon and the number of such events, using only its cutoff-time neighborhood.Its loss combines binary occurrence and log-count prediction losses.Source: §3.2 · p. 6
• TEMP-SUB contrasts two independently corrupted views of each historical neighborhood, pooling nodes within types before combining type summaries.Views share the seed and cutoff, with masked attributes and dropped edges; type-wise averaging prevents larger tables from dominating solely by node count.Source: §3.2 · pp. 6–7

4.1 EXPERIMENTAL SETUP

• The controlled study uses original chronological splits across 11 tasks from five RelBench databases, with four random seeds and accuracy, ROC–AUC, or MAE according to task type.HETEROGNN and THGFM are the selected representative backbones; two-stage and single-stage pretraining use the same total of 100,000 steps.Source: §4.1 · p. 7

4.2 MAIN RESULTS

• +3.24% on the GNN and +2.37% on the GT: the best staged schedules with both encodings outperform the A0 supervised control without TIMEROPE or pretraining.TS→REL-HIST leads on the GNN and TS→REL-FUTURE on the GT; Figure 3 reports relative test-performance gains across tasks and highlights the best result in each panel.Source: §4.2 · p. 7, Figure 3 · p. 8
• Staged schedules rank best in all four backbone–encoding panels, and with both encodings the schedules occupy the top two positions on each backbone.Average ranks weight all 11 tasks equally; the reported ranks are 4.73, 7.14, 4.95, and 6.45.Source: §4.2 · p. 8
• With both encodings, the best staged configuration improves over A0 on 8 of 11 GNN tasks and 7 of 11 GT tasks, including all four regression tasks on both.Regression gains are +5.12% on the GNN and +8.55% on the GT, while classification results are less uniform.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

• Removing TIMEMIX lowers mean type gain by 0.59 percentage points on the GNN and 1.18 on the GT, larger drops than disabling TIMEROPE.Removing TIMEROPE lowers gains by 0.28 and 0.46 points, respectively; the combined encoding configuration has the highest mean type gain on both backbones.Source: §4.3 · p. 8, Figure 4 · p. 9
• With both encodings, staged TEMP-SUB schedules gain +3.02% on the GNN and +1.06% on the GT over matched supervised controls.At equal total pretraining steps, an initial TEMP-SUB stage improves on the corresponding single objective in seven of eight REL-HIST and REL-FUTURE comparisons, but REL-FUTURE alone is slightly higher on the GNN (+3.00% versus +2.83%).Source: §4.3 · pp. 8–9, Figure 4 · p. 9

4.4 DISCUSSION AND LIMITATIONS

• Record-age distributions range from seconds or hours to years across databases, motivating TIMEMIX’s multi-scale representation.Supported prediction horizons also vary; REL-FUTURE uses candidate horizons from one day to one year.Source: §4.4 · p. 9
• The evaluation focuses on within-database training; transfer to unseen databases remains unexplored.The authors also note that matched computational-cost comparisons and alternative stage orders would clarify objective composition and stage-order contributions.Source: §4.4 · p. 9

5 CONCLUSION

• Across five RelBench databases and 11 tasks, the two encodings together outperform either alone in aggregate on both backbones, while staged pretraining adds gains over matched supervised controls.The conclusion supports combining explicit temporal representations with staged self-supervision for within-database prediction.Source: §5 · pp. 9–10

A DATASETS AND PREDICTION TASKS

• The five RelBench databases provide one multiclass classification task, six binary classification tasks, and four regression tasks.The appendix describes task definitions as specifying prediction windows and eligibility rules.Source: A DATASETS AND PREDICTION TASKS · p. 14

• Table 1 reports stored database statistics; record counts precede temporal filtering, exclude downstream task tables and reverse edges, and count each declared foreign-key column as a directed schema relation.Separate foreign-key columns referencing the same table count as separate relations.Source: Table 1 · p. 14, §A.1 · p. 14

A.2 TASK STATISTICS AND TEMPORAL SPLITS

• A sample is an (entity, cutoff) pair, so entities may recur; Table 2 counts distinct target identifiers across all three splits.Targets cover (t, t + ∆] after cutoff t.Source: §A.2 · p. 14, Table 2 · p. 14

A.3 DATABASE DETAILS AND PREDICTION TARGETS

• Arxiv tasks predict an author’s primary category or a paper’s citation over the next 182 days; author-category includes authors publishing in that window.The database contains 222,769 papers, 143,691 authors, and 1,595,687 citation rows.Source: §A.3 · pp. 14–15
• Avito’s three tasks use four-day windows: user-clicks counts repeated clicks, user-visits counts distinct ads visited, and ad-CTR measures clicks per search-stream impression.The database links advertisements, users, searches, visits, and phone requests, with location and category tables as context.Source: §A.3 · p. 15
• F1 predicts driver DNF status within 30 days or mean finishing order over 60 days, using the benchmark’s rule that any non-Finished status is DNF.Race dates timestamp competition records.Source: §A.3 · p. 15
• H&M predicts seven-day customer churn or article sales value; Event predicts recorded attendance responses for events starting within seven days.Event targets are recorded responses, not verified attendance.Source: §A.3 · p. 15

B DETAILED RESULTS

• The section reports the full backbone comparison, encoding ablation, and pretraining results, with record-age and horizon statistics supplementing Section 4.2.Task metrics are means and standard deviations across four independent runs.Source: B DETAILED RESULTS · p. 15
• Relative gains and their aggregates are computed from task means.The runs use seeds 42, 43, 44, and 45.Source: B DETAILED RESULTS · p. 15
• Table 3 reports validation performance for backbone selection, while encoding and pretraining comparisons report test performance.Source: B DETAILED RESULTS · p. 15

B.1 BACKBONE SELECTION: GNNS AND GTS

• The controlled study uses one representative per family to reduce computational cost, so its effects are not established for RelGNN, HGT, or RelGT.Source: §B.1 · p. 15
• HETEROGNN ranks ahead of RelGNN (3.18 versus 3.36), while THGFM leads the graph Transformers at 2.27, ahead of HGT (2.55) and RelGT (3.64).The selected family representatives are HETEROGNN and THGFM; task wins are descriptive, while mean rank determines selection.Source: §B.1 · p. 15, Table 3 · p. 16
• Backbone selection uses validation performance, reserving test performance for final evaluation; downstream checkpoints also use validation metrics.Pretraining checkpoints follow objective-specific criteria.Source: §B.1 · p. 16

B.2 TEMPORAL ENCODING AND TEMPORAL INFORMATION DISTRIBUTIONS

• DAYPE is the fixed day-based, single-scale age baseline; TIMEMIX instead uses multiple learned time scales and type-specific projections of log-transformed record ages.DAYPE retains record-age information even when the paper’s controls disable TIMEROPE.Source: §B.2 · p. 16
• Combining TIMEMIX and TIMEROPE yields the highest mean type gains over DAYPE: +0.46% on HETEROGNN and +1.29% on THGFM.Table 4 reports the encoding ablation; mean type gain weights the three task types equally.Source: §B.2 · p. 16
• Record-age distributions differ across databases: Avito ages fall within 30 days, Event concentrates at 1–90 days, H&M mostly at 7–365 days, and F1 is about 78% older than a year.Fewer than 0.3% of Arxiv and F1 records are at most one day old; these distributions do not establish TIMEMIX’s predictive advantage or optimal scale count.Source: Figure 5 · p. 17, §B.2 · p. 17
• TIMEMIX alone gives slightly higher multiclass gains than the combined setup on both backbones, while the combination gives higher regression gains than either change alone.Combined gains vary by task type and individual task; some tasks decline.Source: §B.2 · pp. 16–17
• Future-horizon support varies by database: Avito supports only 1- and 7-day windows, while 365-day horizons are eligible for 67.7% of Arxiv and 87.1% of F1 events.The shared candidate horizons range from one day to one year; Figure 6 summarizes support and expected sampling distribution.Source: Figure 6 · p. 18, §B.2 · p. 18
• In Avito, expected REL-FUTURE sampling assigns 80.5% of examples to one day and 19.5% to seven days; unsupported horizons receive no samples.These shares describe supervision availability, not the predictive value of each horizon.Source: §B.2 · p. 18

Comparison with constituent objectives

• +3.02% for HETEROGNN with both encodings, versus +0.11% for TEMP-SUB and +1.14% for REL-HIST alone; the best staged schedule beats both constituents in every setting.The reported comparisons use the matched reference within each table panel.Source: Table 5 · p. 19, §B.3 · p. 19, Table 6 · p. 19
• Without our encodings, TS→REL-FUTURE leads HETEROGNN at +2.51% (vs. −0.88% TEMP-SUB and +1.34% REL-FUTURE); TS→REL-HIST leads THGFM at +1.58% (vs. −2.25% TEMP-SUB and −0.24% REL-HIST).Source: §B.3 · p. 20

Comparison with baseline pretraining

• The best proposed schedules exceed the strongest baseline gains by 0.88 and 0.31 percentage points on HETEROGNN, and 0.42 and 1.08 on THGFM.All six TEMP-SUB-initialized baselines remain below the matched supervised control on THGFM with both encodings, while both proposed schedules exceed it.Source: Table 5 · p. 19, Table 6 · p. 19, §B.3 · p. 20

Dependence on the second-stage objective

• TEMP-SUB initialization helps REL-HIST and PT-HGNN on HETEROGNN with both encodings, but lowers REL-FUTURE and ATTRMASKING gains there.On THGFM with both encodings, REL-FUTURE improves from −0.52% alone to +1.06% after TEMP-SUB; the staging effect depends on the second-stage objective and backbone.Source: §B.3 · p. 20

C ARCHITECTURE AND TRAINING CONFIGURATION

• Both backbones use three graph layers, 128-dimensional hidden states, and three-hop neighborhoods sampled with fanouts 128, 64, and 32.Primary and foreign keys define graph structure rather than attribute features; column encoders produce 128-dimensional outputs.Source: C ARCHITECTURE AND TRAINING… · p. 20, Table 7 · p. 21
• Single- and two-stage pretraining each use 100,000 steps total, as do downstream fine-tuning and direct supervised training; no hyperparameter search was performed.Two-stage pretraining allocates 50,000 steps to each stage, and its next stage initializes a fresh objective-specific module and optimizer.Source: C ARCHITECTURE AND TRAINING… · pp. 20, 22, Table 9 · p. 21

E PRETRAINING OBJECTIVE DETAILS

• For horizon δ, eligibility requires the seed at τ = tevt − δ; events at τ remain history, events at τ + δ enter targets, and negatives have no selected-relation events in the window.Candidate horizons are 1, 7, 30, 90, and 365 days; relation, event, and horizon are sampled uniformly.Source: C ARCHITECTURE AND TRAINING… · p. 22, E PRETRAINING OBJECTIVE DETAILS · p. 23

D TEMPORAL ENCODING DETAILS

• TIMEMIX encodes record age using four trainable log-scales, while TIMEROPE applies rotations using bounded, signed log-intervals.TIMEMIX scales are initialized at one second, one minute, one hour, and one day; TIMEROPE extends temporal rotation to GNN messages as well as Transformer attention.Source: D TEMPORAL ENCODING DETAILS · pp. 22–23

E PRETRAINING OBJECTIVE DETAILS

• REL-HIST recovers hidden typed links using timestamped link context, falling back to the pretraining cutoff when neither endpoint is timestamped.For each positive link, the procedure samples negative destinations under the stated constraints.Source: E PRETRAINING OBJECTIVE DETAILS · p. 23
• TEMP-SUB contrasts two corrupted views of each historical neighborhood, with separate graph- and seed-representation contrastive losses.The seed representations also receive variance and covariance regularization.Source: E PRETRAINING OBJECTIVE DETAILS · pp. 24–25

Pretraining curves

• Pretraining panels compare single-objective training with stage two after TEMP-SUB; compare metrics within a row because objective scales differ.Single-stage epochs use 10,000 steps, while each two-stage epoch uses 5,000 steps, and epoch counts restart at stage two.Source: F PRETRAINING AND FINE-TUNING CURVES · p. 25, Figure 7 · p. 26, Figure 8 · p. 26

Fine-tuning curves

• Fine-tuning panels group validation metrics by task: accuracy and ROC–AUC are higher-is-better, while MAE is lower-is-better.Fine-tuning uses 10,000 steps per epoch; the plotted curves are validation metrics, and test results are reported in Tables 5 and 6.Source: F PRETRAINING AND FINE-TUNING CURVES · p. 25, Figure 9 · p. 27, Figure 10 · p. 27

Curve settings

• The curves cover HETEROGNN and THGFM with DAYPE without TIMEROPE or TIMEMIX with TIMEROPE.These configurations correspond to the A0 and B0 controls, respectively.Source: F PRETRAINING AND FINE-TUNING CURVES · p. 25, Figure 13 · p. 29, Figure 19 · p. 32