PaperlayerView exact revision on arXiv

Source-linked AI summary

RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec

arXiv:2610.17867v2cs.LGcs.AIcs.DB

TL;DR

Relational predictors have generally required task-specific training or many labeled examples, despite the value of prediction on relational databases. RT-J combines real-world relational pretraining, long contexts, and random-walk retrieval; on RelBench regression, it matches strong feature-engineering pipelines with roughly 10× fewer examples. Its evaluation is limited to 21 tasks across seven databases.

  • Problem

    Existing relational predictors typically require task-specific training or substantial labeled data, while existing RFM recipes are not tailored to prediction with few in-context examples.

    Source: §1 · pp. 1–2

  • Method

    RT-J combines pretraining on synthetic and real relational data with long-context training, random-walk retrieval, and test-time context tuning and ensembling.

    Source: §1 · pp. 2–3, §4.1 · p. 5

  • Results

    RT-J matches strong feature-engineering pipelines on RelBench regression with roughly 10× fewer examples.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results move relational foundation models toward practical systems that predict on novel database tasks with few examples.

    Source: §8 · p. 10

    Evaluation is limited to RelBench, with 21 tasks on seven databases.

    Source: §8 · p. 10

Abstract

from arXiv · show

Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.

1 Introduction

Relational prediction models have typically needed task-specific training or substantial labeled data, while existing RFMs were not tailored to learn from few in-context examples. RT-J combines real-world relational pretraining, context retrieval, and test-time compute scaling to improve context efficiency.

1 Introduction

  • Existing relational predictors often require task-specific training or substantial labeled data, limiting rapid ad hoc prediction when labeled examples are scarce.Existing RFMs were not tailored for few-example prediction because training recipes, real-world pretraining data, and context retrieval were mismatched to that regime.Source: §1 · pp. 1–2
  • The paper develops long-context training, a lightweight random-walk retriever, and context tuning and ensembling to make relational transformers stronger in-context predictors.The authors’ goal is to identify effective training and inference ingredients rather than introduce a new backbone architecture.Source: §1 · p. 2
  • RT-J is produced by continuing pretraining RT-PluRel on THE JOIN, a corpus of 639 databases with 6K forecasting and 11K autocomplete tasks.Compared with original RT pretraining, the recipe used 11× more total tokens, contexts up to 8× larger, and a 4× larger model, amounting to roughly 70× more pretraining FLOPs.Source: §1 · p. 3
  • RT-J matches strong feature-engineering pipelines on RelBench regression with roughly 10× fewer examples, and with 15 examples beats the best error any pipeline reaches with 225.At matched example counts, it leads by 2.3–5.0 nMAE points; Figure 2 summarizes the context-efficiency comparison.Source: Figure 2 · p. 2, §1 · p. 3
  • The paper concludes that context-efficient relational prediction can be achieved by matching pretraining to the regime and scaling data, context, and test-time compute.Its stated direction is RFMs that can be pretrained once, reused across domains, and adapted to new tasks with few examples.Source: §1 · p. 3

2 Background: Relational Transformers for Predictive Tasks on Databases

Relational prediction uses a target row’s relational context to predict a target-column value, while in-context learning additionally conditions on labeled rows. Relational transformers encode that context as linked cells, allowing support examples to arise implicitly within a query-centered window.

Prediction tasks on relational data

  • A relational prediction task specifies a target table, target column, and prediction rows; the model predicts each target value from the row’s observed relational context.Tasks include autocompletion and forecasting, with temporal contexts restricted to rows available before prediction time.Source: §2 · p. 3

In-context learning for prediction tasks

  • Classical in-context prediction conditions on labeled support rows, each with its own relational context, alongside the masked query context.Overlapping support-row contexts can duplicate cells and consume the context budget, limiting the number of examples.Source: §2 · p. 3

Context window of relational transformers

  • Relational transformers instead build one query-centered context, where observed target-table rows provide implicit support examples and overlapping neighborhoods appear only once.The target value remains masked during context expansion.Source: §2 · p. 4

Masked token prediction with relational transformers

  • RT treats each cell as a token and predicts masked cells, combining cell values and metadata while attending within columns, rows, and linked rows.Its column, feature, and neighbor attention layers share information across the relational database while preserving its structure.Source: §2 · p. 4

3 THE JOIN: A Large-Scale Pretraining Corpus of Relational Data

THE JOIN is a corpus of real-world relational databases and generated forecasting tasks intended to expand the structural and task variety available for RFM pretraining. Its construction combines database collection and standardization with LLM task generation and model-based filtering.

3 THE JOIN: A Large-Scale Pretraining Corpus of Relational Data

  • THE JOIN comprises 639 relational databases across diverse domains and 6K forecasting tasks spanning difficulty levels, including rare events and cold-start entities.The paper motivates the corpus by noting that existing collections do not capture the structural and distributional variety of predictive tasks needed for RFM training.Source: §3 · p. 4

3.1 Collection

  • The collection draws on five source classes, requires at least two FK-joined tables and a timestamp column, and excludes databases overlapping selected benchmarks by source identity.Sources include repositories, public APIs, domain portals, community dumps, and benchmarks; source classes have qualitatively different relational structures.Source: §3.1 · p. 4

3.2 Standardization

  • Standardization applies detector-and-transformer fixes for boolean and numeric values, inferred foreign keys, temporal columns, and oversized databases.Foreign-key inference uses name matching and an ≥80% value-containment test; subsampling retains all activity rows for sampled entities.Source: §3.2 · pp. 4–5

3.3 Task generation

  • Task generation parses schemas into entity and activity tables, then uses an LLM to propose forecasting tasks from registered templates and a deterministic checker to reject invalid proposals.Eligible activity tables must have at least five unique timestamps and 100 rows; the templates cover eight reasoning regimes.Source: §3.3 · p. 5

3.4 Task filtering

  • An XGBoost model scores generated tasks, retaining tasks that are learnable against historical baselines or provide harder objectives when baseline signal is weak.The filtering process yields 6,023 forecasting tasks.Source: §3.4 · p. 5

4 Pretraining Recipe

The pretraining recipe is designed to support context-efficient relational prediction through variable long contexts and target-relevant retrieval. Random walks rank candidate examples before local BFS expands their relational neighborhoods into the context.

4.1 Long-context training

  • Training samples context size and retrieval hyperparameters stochastically, exposing RT to diverse contexts and enabling test-time tradeoffs between inference compute and quality.Because memory scales with context size and batch size, the recipe keeps batch size fixed and adaptively uses gradient accumulation at large contexts.Source: §4.1 · p. 5

4.2 Retrieval

  • Retrieval first ranks target-table rows by random-walk visit counts, then expands ranked seeds with bounded-width BFS over primary–foreign-key links until the context budget is filled.Random walks prioritize rows relevant to the target; the context size and BFS parameters can be tuned at test time after being sampled during training.Source: §4.2 · p. 6

5 Pretraining Results

RT-J is evaluated on RelBench against feature-engineering pipelines and against models with successive synthetic and real-data pretraining. It outperforms regression baselines across context sizes, while both pretraining phases contribute gains.

5 Pretraining Results

  • Pretraining samples context sizes from 512 to 8192 cells and uses target-cell masking; training proceeds from synthetic PluRel data to THE JOIN data.The model has 86M parameters, and the same selected checkpoint is used for classification and regression.Source: §5 · p. 6
  • Evaluation covers 21 RelBench forecasting tasks across 7 held-out databases, using nMAE for regression and AUROC for classification.The tasks include rare-event, cold-start, heavy-tailed, and long-horizon regimes.Source: §5 · p. 6

5.1 Baseline comparisons

  • RT-J beats every regression baseline by 2.3–5.0 nMAE points at every context size; on classification, it leads by 3.1 AUROC points when examples are scarcest.With 15 in-context examples, RT-J beats the best error reached by any baseline with 225; the strongest pipeline needs around 2300 examples—10× more—to surpass it.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2 Impact of synthetic and real pretraining phases

  • Synthetic pretraining improves fine-tuning over a randomly initialized RT by 1.7 nMAE and 0.8 AUROC points; THE JOIN adds 0.5 nMAE and 0.5 AUROC points.In-context, RT-J improves over RT-PluRel by 0.6 nMAE and 1.5 AUROC points.Source: §5.2 · p. 7

6 Test-Time Compute Scaling

RT-J uses context tuning and context ensembling as complementary test-time strategies for choosing informative cells and reducing retrieval-related prediction noise. Tuning helps regression on average, while ensembling improves both metrics.

6 Test-Time Compute Scaling

  • Context tuning selects L, ℓ, b, and ρ separately for each task on its validation split, while ensembling averages predictions from independently sampled contexts.The ensembling evaluation uses different sampler seeds; the study uses a subsampled test set for computational feasibility.Source: §6 · pp. 7–8, §6.1 · p. 8, §6.2 · p. 8

6.1 Context hyperparameter tuning

  • With one context seed, tuning lowers mean regression nMAE from 33.4 to 32.5 and improves eight of nine regression tasks, but lowers mean classification AUROC from 72.9 to 72.4.For classification, the validation-selected configuration transfers to test on only five of twelve tasks.Source: §6.1 · p. 8

6.2 Context ensembling

  • Ensembling 16 contexts instead of one yields a +1.3 AUROC-point aggregate gain for classification with tuned contexts and +1.1 points with default contexts.Regression improves by 0.2 nMAE points on average at the default context; Figure 4 reports improvements for both metrics in both configurations.Source: §6.2 · p. 8

7 Ablation Studies

The ablations examine schema semantics and retrieval in RT-J. Shuffling schema names degrades results, while random-walk retrieval is competitive with feature-similarity retrieval despite using fewer labels.

7 Ablation Studies

  • The ablations swap one component of the full RT-J pipeline at inference and evaluate on a fixed 8192-row subsample per task.Table 2 summarizes gains in nMAE and AUROC points.Source: §7 · p. 8, Table 2 · p. 10

7.1 Schema semantics

  • Shuffling table and column names degrades classification by up to 2.1 AUROC points and regression by up to 0.8 nMAE points; the gap widens with more examples.The authors suggest that schema text provides prior information not replaced by the in-context examples.Source: Figure 5 · p. 9, §7.1 · p. 9

7.2 Alternatives to random-walk retrieval

  • At L = 8192, random-walk retrieval and RDBLearn-feature similarity effectively tie at 33.4 versus 33.5 nMAE and 72.9 versus 73.0 AUROC.Random walk leads regression at L = 256 by 0.7 nMAE and again at L = 8192; feature similarity leads by 0.4–0.8 nMAE at L = 1024–4096.Source: Figure 6 · p. 9, §7.2 · pp. 9–10
  • RDBLearn-feature similarity matches random-walk performance with 394 labels versus 223, while uniform random selection retrieves 208 labels but trails by 4.1 nMAE and 6.1 AUROC.The figure’s right panels compare metrics with average labels delivered at L = 8192; non-label cells provide relational context.Source: Figure 6 · p. 9, §7.2 · p. 10

8 Discussion and Conclusion

RT-J is presented as a model for accurate predictions on new datasets with as few as a hundred examples, and its recipe combines long-context training, random-walk retrieval, and test-time scaling. The authors identify limits in context length, task coverage, and evaluation scope.

8 Discussion and Conclusion

  • RT-J matches strong feature-engineering pipelines with as much as 10× fewer examples and is intended to solve novel database prediction tasks on the fly.The discussion describes use with a hundred examples in a single forward pass, without dedicated feature-engineering pipelines or extensive labeled data.Source: §8 · p. 10
  • The pretraining recipe combines long-context training, random-walk retrieval of in-context examples, and test-time scaling through context tuning and ensembling.RT-J is pretrained first on synthetic PluRel data and then on the real-data corpus THE JOIN.Source: §8 · p. 10
  • Training is limited to 8192 context cells, RT-J cannot solve link prediction or recommendation, and evaluation covers only 21 RelBench tasks on 7 databases.The authors identify longer-context training and broader real-world relational benchmarks as future work.Source: §8 · p. 10

A Leaderboard Results

On RelBench, RT-J improves on RT-PluRel in-context and leads the RT family after task-specific fine-tuning; the separate RelArena-α release ranks RT-PluRel first, not RT-J.

RelArena-α

  • 1861 Elo: RT-PluRel ranks first on the RelArena-α combined leaderboard, ahead of TabPFN-Rel (API) at 1826; RT-J was not included.The leaderboard covers 21 RelBench v1 forecasting tasks and distinguishes standardized-tuned models from systems using their own tuning protocol.Source: A Leaderboard Results · p. 16, Figure 7 · p. 16

RelBench leaderboard

  • 32.35 vs. 32.95 nMAE and 74.45 vs. 72.98 AUROC: in-context RT-J improves over RT-PluRel on both RelBench metrics.The leaderboard table reports per-task test metrics for RT, RT-PluRel, and RT-J in both fine-tuned and in-context settings.Source: A Leaderboard Results · p. 16, Table 3 · p. 17
  • 27.11 nMAE and 79.02 AUROC: fine-tuned RT-J has the best means among RT, RT-PluRel, and RT-J.RT-PluRel records 27.57 nMAE and 78.53 AUROC; RT records 29.26 nMAE and 77.76 AUROC.Source: A Leaderboard Results · p. 16, Table 3 · p. 17

Fine-tuning pipeline

  • Fine-tuning uses four stages: selection, context search, refitting on train plus validation data, and test prediction with an eight-seed context ensemble.RT-PluRel and RT-J add a delta to frozen pretrained weights, while RT trains all weights from random initialization.Source: A Leaderboard Results · p. 16

In-context pipeline

  • In-context evaluation freezes the pretrained checkpoint, searches 120 context configurations, then averages 16 predictions per test row across four configurations and four seeds.No RelBench database is seen during pretraining.Source: A Leaderboard Results · p. 16

B Other Relational Foundation Models

The appendix situates RT-J among related relational foundation models and documents the collection, processing, validation, filtering, and licensing of its corpus and tasks, alongside its approach to source-data privacy. It also presents three forecasting-task examples across real-world domains and label families to illustrate how relational context shapes task design.

B Other Relational Foundation Models

  • No recent relational foundation model could be evaluated fairly because the protocol requires every method to use the same in-context examples.KumoRFM's API does not permit controlling examples; KumoRFM-2 had closed weights and an unavailable API; Griffin's checkpoints lack in-context learning; RDB-PFN supports only classification; and TabPFN-Rel was released after submission.Source: B Other Relational Foundation Models · p. 17

C Related Work

  • RT [23] treats database cells as tokens, includes schema semantics, and encodes primary–foreign-key structure with relational attention; RDBLearn instead aggregates neighborhoods into engineered features, changing the input representation.Source: C Related Work · p. 18

D.1 Agentic execution

  • Claude Code agents engineered and orchestrated corpus collection, while deterministic scripts handled task validation, scoring, filtering, baselines, loadability checks, and provenance.The agents inspected candidate sources, built and debugged collectors, adapted standardization routines, and iterated on pipeline stages using human-defined completion criteria.Source: §D.1 · p. 19

D.2 Phase 1: Collection

  • Source-specific collectors convert heterogeneous public database formats into a common Parquet representation, exclude exact benchmark-source overlaps, and retain key and time-column metadata.Collected sources include SQL dumps, SQLite, CSV, Parquet, JSON, archive mirrors, and paginated APIs; normalization also standardizes keys, timestamps, booleans, and mixed-type columns.Source: §D.2 · p. 19

D.3 Phase 2: Standardization

  • Standardization uses idempotent detector–transformer passes to make heterogeneous databases loadable without altering their semantic content.Repairs are conservative, with inferred keys checked for uniqueness or containment and ambiguous type cases left unchanged or flagged.Source: §D.3 · p. 22

D.4 Phase 3: Task proposal and validation

  • Task proposals are constrained to registered templates and accepted only after deterministic validation, while multiple passes target underrepresented templates and harder prediction settings.Later passes seek cases such as rare positives, cold starts, heavy-tailed regression, ranking, and derived-ratio tasks.Source: §D.4 · p. 22

D.5 Phase 4: Baselines and quality filtering

  • Tasks are filtered for usefulness using entity-mean, last-value, and fixed XGBoost baselines; retained tasks receive easy, medium, or hard labels based on XGBoost scores.The filter removes nearly trivial tasks, severe distribution drift, and baseline-fitting failures, but retains non-trivial tasks even when XGBoost finds no signal.Source: §D.5 · pp. 23, 25

D.6 Asset licenses, Citations, and Compliance

  • The corpus is intended for release under CC-BY-SA 4.0, with per-database metadata recording upstream identifiers, verbatim license strings, and inherited usage restrictions.Contributing sources are licensed bulk archives or public APIs permitting research use; redistribution must honor source-specific restrictions alongside the assembly license.Source: §D.6 · p. 25
  • Source pseudonymized identifiers are preserved verbatim, with no attempt to de-anonymize them or join them across sources.The corpus reports no personally identifying information beyond what upstream sources published under their own licenses.Source: §D.6 · p. 25

D.7 Case Studies

  • In join-berka, the account-max-balance task predicts each account’s highest balance over the following roughly 17.5 days using an eight-table, multi-hop bank schema.The Berka snapshot includes accounts, clients, loans, orders, and a ledger of over a million transactions.Source: §D.7 · p. 26
  • In join-brightkite, a cold-start task predicts whether a user will check in again within two roughly 6.7-day windows, requiring generalization from venues and friendship-graph position.The user is evaluated at their first-ever check-in, so their own label history is unavailable.Source: §D.7 · p. 26
  • In join-oulad, the task predicts whether a student’s mean assessment score in the next roughly 7.2-day window will exceed their cohort’s mean, requiring time-varying peer context.The threshold is the cohort average rather than a fixed constant, so students cannot be scored in isolation.Source: §D.7 · p. 26

D.8 Manual Audit of Generated Tasks

The audit found that generated tasks generally target meaningful quantities and practitioner decisions, while RelBench includes both future-event classification and continuous or count-valued regression forecasts.

D.8 Manual Audit of Generated Tasks

  • The audit found semantically named targets, such as max_balance and days_to_next_txn, rather than meaningless district columns a2–a16.The authors manually reviewed task descriptions and label-generation SQL because deterministic schema checks alone do not establish that tasks are meaningful.Source: §D.8 · p. 27
  • Generated labels can encode domain concepts, including comparing a student's score with the cohort mean or predicting a bike station's idle time.The station idle-time task is described as useful for operator inventory planning.Source: §D.8 · p. 27
  • Tasks mirror practitioner decisions, including overdraft exposure review, onboarding retention, and movie metadata coverage.Examples include account-max-balance, newuser-will-return, and movie-new-tag-adoption.Source: §D.8 · p. 27
  • About 100 tasks aggregated non-semantic columns, such as student-feature1-to-feature0-ratio in join-act_mooc.This was a small portion of the audited tasks.Source: §D.8 · p. 27

E RelBench Forecasting Tasks

  • RelBench classification forecasts future events or conditions, while regression predicts continuous or count-valued outcomes across varied domains.Regression examples include lifetime value, item sales, race position, and adverse-event counts.Source: E RelBench Forecasting Tasks · p. 27, Table 7 · p. 28

E.1 Hard Regimes in RelBench

RelBench includes rare-event, cold-start, and heavy-tailed tasks. The section describes RT-J’s relational architecture and pretraining, including evidence that masking only target cells outperforms masking additional cells.

E.1 Hard Regimes in RelBench

  • RelBench includes rare-event tasks at or below 5% positive, two trial tasks with 100% cold start, and heavy-tailed targets.The rare-event tasks have 3.9%, 4.8%, and 5.0% positives; the trial tasks have no test trial in training, and several targets have high skew or kurtosis.Source: §E.1 · p. 28

F Relational Transformer Architecture

  • RT-J uses 12 transformer blocks with sequential column, feature, and neighbor attention to exchange information across cells and linked rows.Cells combine value and schema information; primary and foreign keys specify connectivity, while the model uses no positional or learned row-ID embeddings.Source: F Relational Transformer Architecture · pp. 28–29

G Pretraining Details

  • Pretraining samples contexts up to 8,192 cells and uses 10,000 random walks of length 20, masking only the target cell.Training proceeds from synthetic PluRel data to THE JOIN, and the released checkpoint is selected from the THE JOIN phase’s SWA trajectory.Source: G Pretraining Details · p. 30, §H.1 · p. 30

H.1 Multi-cell masking rate

  • 74.2 validation AUROC and 33.3 nMAE: target-only masking outperformed every non-target masking rate over 10K–18K steps.At each arm’s best step, target-only masking reached 74.4/32.9 versus a best masked-arm result of 73.1/33.1.Source: §H.1 · p. 31, Figure 13 · p. 31

H.2 Pretraining task mix

The section examines task mixing, training-data scaling, context retrieval, and test-time compute. Across these settings, results vary by task and context budget, while extended-context baselines can surpass RT-J in some comparisons.

H.2 Pretraining task mix

  • The forecasting–autocompletion mixture was the only tested task mix competitive on both validation metrics.Forecasting alone was close on classification at 73.6 AUROC but 0.4 nMAE behind the mixture; autocompletion alone reached 71.2 AUROC and 34.8 nMAE.Source: §H.2 · p. 31

H.3 Compute scaling

  • Continued pretraining from RT-PluRel reached its best validation metrics within about 8K steps and stayed ahead of THE JOIN-from-scratch on regression throughout.The classification comparison also favored continued pretraining at all but a few steps around 24K; PluRel-only stopped improving after about 8K steps.Source: §H.3 · p. 31, Figure 14 · p. 32

I Random-Walk Context Construction

  • The retriever ranks same-table rows by random-walk visits along timestamp-valid primary- and foreign-key links, then launches width-bounded BFS collection from them.With recency enabled, rows are ordered by timestamp and visit count; otherwise, visit count determines order.Source: I Random-Walk Context Construction · p. 32, §H.3 · p. 32

I.1 Sensitivity to random-walk parameters

  • Reducing random-walk count or length cut sampling cost from 28.1 to 14.3–17.9 ms per example, while mean classification AUROC fell from 72.9 to 71.9.Mean regression nMAE remained between 33.3 and 33.6; the default walk setting was retained.Source: §I.1 · pp. 32–33

J.1 Baselines with extended context

  • At 131K cells, TabICLv2 pipelines exceed RT-J’s 8K-context classification AUROC by 3.6–4.2 points; LLM Agent + TabICLv2 passes its regression nMAE only from 65K cells.At 131K cells, LLM Agent + TabICLv2 reaches 32.5 versus RT-J’s 33.4 nMAE; TabICLv2 remains 0.3–0.5 nMAE behind at the two largest sizes.Source: §J.1 · p. 34, Figure 15 · p. 34

J.2 Per-task baseline curves

  • Per-task results show RT-J’s regression wins concentrated on heavy-tailed or graph-spread targets, while feature pipelines led on several other tasks.RT-J led or tied on nearly every classification task at small contexts, but results were mixed at 8K; the extended-context curves use a fixed 8,192-row-per-task test subsample.Source: §J.2 · p. 34, §J.1 · p. 34

J.3 Cost of test-time compute scaling

  • Context tuning evaluates 120 configurations with four seeds, using 144 validation passes plus one test pass; ensembling requires no extra labels beyond training.The per-task grid search selects context settings on a validation subsample, then evaluates the selected configuration on the test subsample.Source: §J.3 · pp. 34–35, §J.4.3 · p. 36

J.4 Per-task ablation curves

  • Random walk was best or tied on most regression tasks at L = 256, while RDBLearn-feature similarity led on most beyond L = 1024; schema semantics helped most tasks but not all.Shuffled names won on several named tasks, including rel-avito/user-clicks and rel-f1/driver-top3.Source: §J.4.1 · p. 36, §J.4.2 · p. 36

J.4.4 Context ensembling (per task)

Per-task context ensembling generally improved classification across most tasks, though some tasks worsened. Regression tasks were comparatively insensitive in several cases.

J.4.4 Context ensembling (per task)

  • Ensembling improved classification on most tasks, but rel-avito/user-clicks fell by 3.0 AUROC points at the default context and rel-trial/study-outcome ended 0.7 points lower at 16 seeds.Classification improved on 9 of 12 tasks at the default context and 10 of 12 at the tuned context.Source: §J.4.4 · p. 37, Figure 23 · p. 40
  • Several regression tasks, including rel-amazon/user-ltv and rel-stack/post-votes, were insensitive to increasing the number of ensembled context seeds.The per-task panels compare test nMAE for default and per-task-tuned contexts.Source: §J.4.4 · p. 37, Figure 22 · p. 39

Related papers

Loading 2610.17867v2…