Source-linked AI summary
Context Window Failures in Relational Foundation Models
Denis Oliveira Correa, Francisco Galuppo Azevedo
TL;DR
Current relational foundation models impose neighborhood budgets that discard records for high-cardinality entities, a limitation not stress-tested on realistic relational data. Animus introduces a synthetic financial benchmark comparing raw transactions with routine temporal aggregation, finding that aggregation substantially improves the budget-limited models. The results question whether these models are ready for high-cardinality relational data.
Problem
Current relational architectures discard records beyond fixed neighborhood budgets, but whether these limits are insufficient for realistic high-cardinality relational data has not been stress-tested.
Method
Animus is a synthetic financial benchmark that evaluates relational models on raw event data and monthly pre-aggregated representations of the same data.
Results
On raw data, RT, Griffin, and RelGT achieve R2 ≤0.18; routine aggregation recovers up to R2 = 0.65, with the conclusion tied to customers exceeding sampling budgets.
Takeaways & Limitations
The findings question whether current relational foundation models are ready for high-cardinality real-world data without practitioner-applied aggregation.
Takeaways & Limitations
The benchmark is synthetic, and its payment records add relational realism but carry no predictive signal for income.
Abstract
from arXiv · showhide
Recent Relational Deep Learning architectures have been proposed as foundation models for multi-table relational data, yet they impose constrained neighborhood budgets that force row truncation when an entity has many related records. We introduce Animus, a synthetic financial dataset in which predicting customer income requires aggregating up to tens of thousands of transactions. On the raw representation, three recently proposed models (RT, Griffin, RelGT) achieve $R^2 \le 0.18$; a single, routine, temporal pre-aggregation step recovers $R^2$ up to $0.65$. This questions whether current relational foundation models are ready for high-cardinality real-world data.
1. Introduction
Relational foundation models impose row or neighborhood budgets that discard records, but this limitation has not been stress-tested on high-cardinality data. Animus targets this gap with a synthetic financial dataset and evaluates whether routine temporal aggregation changes model performance.
- Motivation: Current relational foundation models discard rows or cells beyond fixed context, subgraph, or fanout budgets.RT is bounded by context size, RelGT by subgraph sampling, and Griffin by per-hop fanout.
- Motivation: High-frequency entities can accumulate orders of magnitude more records than these budgets allow, leaving the gap untested.
- Approach: Animus is a synthetic financial dataset designed to stress neighborhood limits in relational foundation models.
- Dataset: 60% of customers generate roughly one transaction per month, while 5% generate between 1,000 and 10,000 transactions.
- Results: Pre-aggregated input recovers up to +0.47 R2 for the three budget-limited models, whereas GraphSAGE improves by only +0.06.The models are evaluated on raw individual events and simple monthly aggregation.
2. Background
Relational Deep Learning represents relational databases as heterogeneous temporal graphs and performs entity-level prediction by aggregating neighborhood information. The evaluated architectures differ in how they sample or process neighbors, but three impose hard limits that discard records beyond their budgets.
- Paradigm: Relational Deep Learning represents each database row as a node and foreign-key relationships as directed edges in a heterogeneous temporal graph.
- Paradigm: Entity-level tasks are evaluated at fixed timestamps, requiring models to aggregate information from each entity’s graph neighborhood.
- Models: GraphSAGE uses permutation-invariant sum aggregation over sampled neighbors, while RT, Griffin, and RelGT use transformer or graph-based architectures.
- Limitation: All three budget-limited models discard records beyond their limits regardless of informativeness, while GraphSAGE has a larger but finite sampling budget.
- Research question: The central question is whether current architecture budgets are insufficient for realistic relational data.
3. Method
Animus is a 100,000-customer synthetic financial dataset for income regression under heterogeneous transaction frequencies and long relational histories. It compares raw event-level data with a cardinality-reduced monthly aggregation while preserving aggregate information.
- Dataset: Animus contains M = 100 000 customers, three tables linked by customer id, and transaction volumes reaching tens of millions of rows.Customers accrue twelve months of transaction history from registration, with a small fraction generating thousands of transactions per month.
- Prediction task: The task predicts each customer’s total credit income in the month after timestamp t using only data available up to t.Evaluation uses training, validation, and test temporal cutoffs.
- Income generation: Customer income follows assigned temporal patterns across a twelve-month tenure, including uniform, back-loaded, front-loaded, and mid-peak patterns.
- Credit transactions: Credit frequency is heterogeneous: 60% of customers have one credit per month, while high-frequency customers generate 1,000–10,000 credits per month.
- Credit transactions: A model observing only k < K(m+1) credit amounts underestimates next-month income, with the underestimate worsening as K(m+1) increases.
- Two representations: The raw representation keeps each credit and debit event as a node, with the largest customers reaching 88 717 transaction nodes.
- Two representations: Agg-Simple collapses transactions into monthly category sums and payments into monthly totals, reducing maximum neighborhoods to 12 × C rows.Both representations encode the same aggregate information; only cardinality differs, and the same hyperparameter grid is used.
4. Experiments
Experiments compare four models on raw and monthly aggregated representations, showing that neighborhood-budget constraints especially affect high-frequency customers.
- Evaluation setup: Hyperparameter search evaluates all four models on both representations, using R2 as the primary metric.MAE and RMSE are secondary metrics.
- Overall results: On raw data, RT, Griffin, and RelGT score R2 ≤0.18, while GraphSAGE reaches R2 = 0.69.RelGT fails to construct the graph because of memory constraints.
- Overall results: 0.47 R2 is recovered by RT after switching from raw data to agg-simple, improving from 0.18 to 0.65.Griffin improves from 0.11 to 0.38, a +0.27 increase.
- Per-bin analysis: 0.44 R2 separates agg-simple from raw for GraphSAGE in both high-frequency customer bins.For low-frequency customers, raw is marginally better because aggregation discards fine-grained event patterns.
- Interpretation: The results localize the failure to customers whose neighborhoods exceed the models’ sampling budgets.The authors characterize routine SQL aggregation as recovering substantial performance for budget-limited models.
5. Conclusion
Animus is introduced as a benchmark for exposing neighborhood-budget limitations in relational deep learning models. The paper also proposes a targeted-retrieval extension involving a single informative transaction among many normal ones.
- Conclusion: Animus is a synthetic financial benchmark designed to expose neighborhood-budget limitations in relational deep learning models.The conclusion reports that three models achieve R2 ≤0.18 on raw data, while routine temporal aggregation recovers up to R2 = 0.65.
- Future extension: A needle-in-a-haystack variant would test targeted retrieval of one specific transaction embedded among thousands of normal transactions.Its label would depend on a single transaction, such as a fraudulent charge.
A. Full Metrics for Best Configurations
The appendix reports full metrics for the best hyperparameter configuration of each model on each dataset representation, with several metric or configuration entries unavailable.
- Reported metrics: Table 3 reports test MAE, RMSE, and R2 for each model’s best configuration on each representation.RT did not log MAE or RMSE, and RMSE was unavailable for Griffin and RelGT grid-search configurations.
RelGT
The RelGT-related appendix material covers grid-search results and implementation constraints across representations and models. Some runs or evaluations were limited by graph-materialization memory or runtime constraints.
- RelGT: RelGT grid search on agg-simple reports the best test R2 configuration.Bold entries identify the best test R2.
- RT: RT grid-search results are reported separately for raw and agg-simple representations, with bold entries marking the best test R2.The raw and aggregated results correspond to Tables 5 and 6.
- Griffin: Griffin’s raw grid search is partial because neighborhood mapping for 31M transactions caused some runs to exceed the per-run time limit.Completed runs are reported in Table 7, while Table 8 covers agg-simple.
C. Per-Bin Results with Full Metrics
Table 9 stratifies GraphSAGE test metrics by post-cutoff credit transaction-count bin, while the overall test scores are Raw = 0.692 and Agg-Simple = 0.752.
- Per-bin stratification: 9,526 customers have no extra credit transactions after the cutoff and are excluded from the per-bin analysis.
- Overall comparison: Overall test R2 is 0.692 for Raw and 0.752 for Agg-Simple.
D. Dataset Statistics
The test statistics use a 10,000-customer split at the 2024-09-30 temporal cutoff, alongside transaction and payment node-count summaries.
- Test split: Statistics are computed on the 10,000-customer test split using a 2024-09-30 temporal cutoff.
- Transaction statistics: Table 10 reports transaction node-count statistics per customer for graph-visible nodes at the test cutoff and the full parquet file.
- Payment statistics: Table 11 reports payment node-count statistics per customer, with 69% having zero payment nodes.