PaperlayerView exact revision on arXiv

Source-linked AI summary

TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders

Wei Pang, Xiangru Jian, Hehan Li, Zhixuan Yu, Alex Xue, Jinyang Li, Zhengyuan Dong, Xinjian Zhao, Hao Xu, Chao Zhang, Reynold Cheng, M. Tamer Özsu, Tianshu Yu

arXiv:2606.09323v1cs.AIcs.DB

TL;DR

Task-specific end-to-end evaluation makes heterogeneous tabular encoders difficult to compare because wrapper and adaptation effects are entangled with representation quality. TRL-Bench standardizes representation-level probing and finds that encoder quality is capability-specific, with compositional pipelines favoring capability-matched hybrids over universal encoders.

  • Problem

    Task-specific end-to-end evaluation makes heterogeneous tabular encoders difficult to compare because wrapper, budget, and adaptation effects are entangled with encoder quality.

    Source: §1 · p. 2

  • Method

    TRL-Bench exports row-, column-, and table-level embeddings once, then evaluates them with shared lightweight modules across atomic transfer and compositional enrichment suites.

    Source: §1 · p. 2

  • Results

    Across 20 models and 16 tasks, transfer is capability-specific, while the best DLTE pipelines are capability-matched hybrids whose quality is not determined by per-stage marginal performance.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    TRL-Bench provides a common reference for comparing portable tabular representations rather than one-off task solvers under shared downstream conditions.

    Source: §5 · p. 10

    The protocol measures what common lightweight readouts extract from exported embeddings rather than fully adapted end-to-end systems, and wrapper-induced variation remains.

    Source: A Limitations · p. 17

Abstract

from arXiv · show

Tabular encoders are usually evaluated inside task-specific end-to-end pipelines, so models from different training paradigms are difficult to compare directly even when they operate on similar tabular signals. We introduce TRL-Bench, a multi-granular tabular representation learning (TRL) benchmark that standardizes cross-paradigm representation-level evaluation: each encoder exports row-, column-, or table embeddings through its supported wrapper, and shared lightweight heads probe them across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment spanning all three granularities). To support this standardized setting, we release curated benchmark assets and task reformulations, including 50 OpenML tables with 123 verified targets, 16 row-pair linkage rewrites, and a 47,772-table DLTE lake derived from 1,379 parent tables. Across 20 models and 16 tasks, TRL-Bench shows that once downstream conditions are standardized, encoder quality is capability-specific rather than captured by a single leaderboard. In TRL-CTbench, generic text encoders often lead on tasks with strong surface-text signal, while tabular specialists win where their pretraining objective aligns with the task. In TRL-Rbench, within-table prediction and cross-table linkage favor different training regimes, with atomic linkage performance correlating strongly with the row-matching stage of DLTE pipelines. In TRL-DLTE, the strongest pipelines combine capability-matched specialists rather than reuse a single encoder, and top end-to-end quality depends on non-additive compositional fit rather than per-stage marginal rank alone. TRL-Bench provides a common protocol for measuring reusable signal in exported tabular representations under shared downstream conditions. Code and data: https://github.com/LOGO-CUHKSZ/TRL-Bench

1 Introduction

TRL-Bench standardizes representation-level evaluation by comparing heterogeneous tabular encoders through shared lightweight readouts over exported row-, column-, and table-level embeddings. Across three suites, 20 models, and 16 tasks, it shows that transfer is capability-specific and that compositional fit matters for enrichment pipelines.

Benchmark scope

  • TRL-CTBENCH, TRL-RBENCH, and TRL-DLTE measure column/table transfer, row transfer, and compositional data-lake enrichment across reusable retrieval, alignment, linkage, prediction, and grounding capabilities.The suites target the granularities at which embeddings are reused in encode-once, reuse-many settings.Source: Figure 1 · p. 2, §1 · p. 2

Motivation

  • Task-specific end-to-end evaluation can confound encoder quality with wrapped predictors, training budgets, and adaptation, motivating direct comparison under one shared protocol.TRL-Bench addresses the encode-once, reuse-many setting where embeddings are indexed and reused across tasks and large multi-table corpora.Source: §1 · pp. 1–2

Standardized protocol

  • TRL-Bench runs each model once to export reusable row-, column-, or table-level embeddings, then evaluates them with shared lightweight modules without end-to-end encoder re-optimization.The benchmark operationally treats any tabular model exposing reusable embeddings as an encoder, isolating representation quality under common downstream conditions.Source: §1 · p. 2

Empirical findings

  • Across 20 models and 16 tasks, no pretraining recipe is universal: capability-specific transfer, training-scope differences, and compositional fit determine benchmark and pipeline outcomes.Generic text encoders often lead on surface-text tasks, while within-table prediction, cross-table linkage, and enrichment favor different or capability-matched models; per-stage rank alone does not determine top pipelines.Source: §1 · pp. 2–3

Assets and reformulations

  • The benchmark covers 16 tasks and 87 datasets, including 20 standardized column/table datasets, 50 OpenML row-prediction tables with 123 verified targets, and 16 rewritten linkage datasets.Its enrichment lake contains 47,772 tables derived from 1,379 parent tables, alongside representation-centric rewrites of heterogeneous source tasks.Source: §1 · p. 2

2 Related Work and Positioning

Prior tabular models and benchmarks are fragmented by granularity and task-specific evaluation, whereas TRL-BENCH standardizes cross-paradigm comparison through reusable representations across columns, rows, tables, and compositional DLTE pipelines.

Related work: model families and evaluation traditions

  • Prior tabular model families split between row-level supervised prediction and transfer versus column/table schema semantics, grounding, retrieval, and discovery, typically using task-specific evaluation.Observatory [15] compares frozen tabular embeddings through intrinsic perturbation and invariance properties rather than downstream task performance.Source: §2 · p. 3

Positioning of TRL-BENCH

  • TRL-BENCH standardizes heterogeneous tabular encoders under shared representation-level tasks and lightweight heads across columns, rows, and tables.The protocol enables direct cross-paradigm comparison under common task definitions.Source: §2 · p. 3
  • TRL-BENCH adds compositional DLTE evaluation to test whether strong atomic capabilities compose into an end-to-end table-enrichment pipeline.It evaluates models that expose reusable row-, column-, or table-level embeddings natively or through a natural architectural extraction point.Source: §2 · p. 3
  • Generative table LLMs [85] [69] and heavily task-specific fine-tuned systems [49] [70] [35] generally fall outside TRL-BENCH because they lack reusable exported representations or are end-to-end predictors.The benchmark therefore targets encoders with reusable row-, column-, or table-level embeddings.Source: §2 · p. 3

3 Benchmark Design

TRL-Bench standardizes representation-level evaluation across row, column, and table embeddings using shared lightweight downstream modules. Its three suites test transfer within granularities and composition across multi-stage table-enrichment pipelines.

3.1 Problem Setting and Standardized Representation-Level Protocol

  • TRL-Bench evaluates exported column, row, or table embeddings with fixed lightweight readouts, separating representation quality from downstream predictor choice and testing reuse across tasks.The protocol includes training-free, supervised linear/MLP, and query-conditioned modules, with standardized wrappers and supported pooling for required granularities.Source: §3.1 · p. 4

3.2 TRL-CTBENCH: Column- and Table-Level Transfer

  • TRL-CTBENCH contains 13 column- and table-level tasks spanning schema understanding, joinability, unionability, and natural-language grounding under representation-centric reformulations.The benchmark uses table-disjoint splits for four pairwise tasks and rewrites WikiTableQuestions around exported column and question embeddings with a lightweight decoder.Source: §3.2 · pp. 4–5

3.3 TRL-RBENCH: Row-Level Transfer

  • TRL-RBENCH tests whether row embeddings transfer within tables through multi-target prediction and across tables through record linkage.It includes 50 OpenML-derived tables with 123 curated targets and 16 rewritten linkage datasets, evaluated with lightweight probes on target-agnostic or paired row embeddings.Source: §3.3 · p. 5

3.4 TRL-DLTE: Multi-Stage Data Lake Table Enrichment

  • TRL-DLTE tests end-to-end composition by reconstructing removed rows and columns from a seed subtable through retrieval, alignment, decision, row matching, and merging.Its three stages can use one multi-granular model or different specialists, and UJ-H is the primary score because it harmonically combines union and join recovery.Source: §3.4 · p. 6

4 Experiments

Under a standardized representation-level protocol, TRL-Bench compares 20 heterogeneous encoders across CTBench, RBench, and DLTE. Results show capability-specific specialization, transfer-scope tradeoffs, and non-additive compositional fit rather than a universal best encoder.

4.2 Column- and Table-Level Results

  • CTBench shows specialization: generic-text rankings worsen from Schema to Grounding, while tabular specialists win tasks aligned with structural or cross-table pretraining.BERT’s family-level NR changes from 0.000 to 0.397, and GTE’s from 0.190 to 0.429; specialist wins include STARMIE on Union Search and Schema Matching, and TABERT on Grounding NR.Source: §4.2 · p. 7

4.3 Row-Level Results

  • RBench separates prediction from linkage: TABICL leads prediction, whereas BERT and GTE lead Clean and Robust Linkage, respectively.TABICL reaches AUROC 0.816, Macro-F1 0.671, and SGM 0.505; BERT reaches Clean Linkage F1 0.418, and GTE has Robust Linkage NR 0.048.Source: Table 3 · p. 8, §4.3 · p. 8
  • Row-transfer results reflect training scope: target-table SSL is competitive within-table prediction, while shared transfer encoders lead cross-table linkage.Target-table SSL has mean NR 0.48/0.47 on classification/regression prediction but 0.65/0.65 on linkage, whereas transfer encoders have 0.50/0.64 and 0.19/0.19, respectively.Source: §4.3 · p. 8

4.4 Compositional Results on TRL-DLTE

  • The best DLTE hybrid, TUTA/GTE/GTE, reaches 0.229 UJ-H, exceeding monolithic BERT/BERT/BERT by 0.090.Development and test rankings are similar across 1,120 pipelines, with Spearman ρ = 0.96 and top-50 overlap 42/50.Source: §4.4 · p. 8, Table 4 · p. 9
  • DLTE quality depends on compositional fit rather than per-stage marginal rank alone: STARMIE/TABBIE/TRANSTAB scores 0.134 UJ-H, versus 0.253 for test rank-1 STARMIE/GTE/GTE.The 0.097 swing between marginal assemblies shows that marginal leaders are a lossy selection rule, while compatibility among retrieval, column alignment, and row matching matters.Source: §4.4 · p. 9, Figure 3 · p. 9

4.5 Three Open Gaps Across the Suites

  • Across suites, the benchmark exposes specialization, transfer-scope, and composition gaps that single-paradigm or single-granularity evaluations cannot isolate.Row-linkage utility also correlates with embedding anisotropy at |¯ρ| ≈0.80, supporting a shared identity-resolution capability across RBench and DLTE.Source: §4.3 · p. 8, §4.5 · p. 9

5 Conclusion … E Appendix Task Summary

TRL-BENCH standardizes comparison of heterogeneous tabular encoders through exported embeddings and shared lightweight evaluation, positioning them as portable representation learners rather than task-specific systems. The benchmark complements existing end-to-end and task-specific resources while documenting scope, model interfaces, limitations, and supporting appendix specifications.

5 Conclusion

  • TRL-BENCH makes heterogeneous encoders comparable through a shared representation-level protocol centered on reusable exported embeddings.The protocol avoids conflating embeddings with task-specific wrappers, retraining budgets, or adaptation.Source: §5 · p. 10

Appendix

  • The appendix documents limitations, extended scope, model inventory, input policy, task summaries, and detailed DLTE operator specifications.Listed materials include the model inventory, model input policy, appendix task summary, and DLTE stages for retrieval, alignment, classification, and row matching.Source: Appendix · p. 15, M DLTE Operator Specification · p. 15

E Appendix Task Summary

  • Appendix task summaries organize column/table, row-level, and compositional tasks by downstream module and split conventions.The listed task summaries distinguish training-free, learned, query-conditioned, and stage-wise pipeline modules.Source: Appendix · p. 15, Table 7 · p. 23, Table 8 · p. 23

A Limitations

  • The benchmark standardizes task definitions and downstream evaluation, but preserves each encoder’s documented preprocessing and supported wrapper regime.This improves fidelity to original models while leaving wrapper-induced variation; dataset counts and normalized-rank summaries are convenience aggregates.Source: A Limitations · p. 17, D Model Input Policy · p. 21

B Extended Related Work and Scope

  • Existing row-level methods are treated as important references rather than direct baselines because they typically evaluate end-to-end prediction instead of reusable frozen row embeddings.The benchmark targets reuse across multiple targets and transfer to inter-table tasks such as record linkage.Source: B Extended Related Work and Scope · p. 18
  • TRL-BENCH differs from prior resources by jointly evaluating column-, row-, and table-level transfer across training paradigms and compositional data-lake enrichment.It tests whether strong atomic representations compose into useful end-to-end systems under a common protocol.Source: B Extended Related Work and Scope · pp. 18–19

C Model Inventory

  • The model inventory covers 20 frozen or trained encoders, with parameter counts sourced from loaded checkpoints, official packages, or default training configurations.Target-table SSL models are retrained per dataset and reported at the median input-feature count for the OpenML tables.Source: Table 5 · p. 20

D Model Input Policy

  • The input policy evaluates each encoder under its standard public operating regime instead of forcing a common serialization or context length.The policy includes model-specific wrapper choices, including synthesized row embeddings for TABBIE and TUTA and supported extended-input settings for TABPFN and TURL.Source: D Model Input Policy · p. 21

E.1 Benchmark Protocol Adaptations under Frozen Multi-Granular Transfer

TRL-Bench adapts reused tasks with frozen, shared readouts so multi-granular transfer remains meaningful and leakage-free. The protocol uses table-disjoint evaluation, harder retrieval and union-search settings, audited labels and identifiers, strict linkage checks, and capability-specific scoring and retrieval controls.

Cross-table generalization

  • Table-disjoint splits reduce Join Classification F1 by 0.212 on average versus pair-random splits, ensuring test tables remain unseen during transfer evaluation.The adaptation covers Join Classification, Column Overlap, Union Classification, and Union Regression.Source: §E.1 · pp. 23–24

Union search

  • The TUS-hard variant removes positives with containment ≥0.70, causing the value-overlap baseline to drop from 1.000 to 0.008 and substantially shifting rankings.This separates broader union signal from lexical overlap in frozen retrieval.Source: §E.1 · p. 24

Target curation

  • Human review, label repair, and degeneracy audits reduce 158 candidate Row Prediction tables to 50 by removing constant, near-duplicate, and mislabeled targets.The retained release supports reusable row-transfer evaluation.Source: §E.1 · p. 24

Record linkage

  • Removing label-equivalent identifiers prevents trivial near-perfect linkage F1, while strict row-disjoint auditing preserves model rankings across all 14 row models (Spearman ρ = 0.94, p = 5.6 × 10−7).For WDC and Fodors–Zagats, cluster_id, identifiers, and class are removed before encoding; the linkage protocol otherwise retains pair-disjoint source splits.Source: §E.1 · p. 24

DLTE evaluation

  • DLTE uses UJ-H as its primary metric and adds 36,740 CKAN distractor tables, making recovery of both removed blocks and table retrieval more demanding.UJ-H is the harmonic mean of union and join recall, while Cell F1 is complementary; the lake contains 11,032 targets among 47,772 tables.Source: §E.1 · p. 24

Hybrid retrieval

  • Model-only table retrieval excludes query-encoder table embeddings because hybrid retrieval compresses model differences into a narrow MRR band, better isolating table-side transfer.This adaptation applies to the Table Retrieval task.Source: §E.1 · p. 24

F Full Dataset Inventory

TRL-Bench assembles distinct inventories for column/table, row, linkage, and DLTE evaluation. These assets span 50 OpenML tables with 123 targets, 16 linkage datasets, and a 47,772-table enrichment lake derived from 1,379 parent tables.

CTBench datasets

  • CTBench covers 20 datasets across schema understanding, joinability, unionability, and grounding tasks.The inventory includes SATO, SOTAB, WikiCT, OpenData variants, unionability benchmarks, and WTQ/NQ-Tables.Source: F Full Dataset Inventory · p. 25

Row prediction

  • Row prediction uses 50 OpenML tables with 123 targets, comprising 77 classification and 46 regression targets filtered from 158 candidates.Targets come from TabArena, OpenML-CC18, and OpenML-CTR23; 36 tables support both classification and regression.Source: F Full Dataset Inventory · p. 25, Figure 4 · p. 25

Record linkage

  • Record linkage contains 16 datasets spanning eight clean and four dirty DeepMatcher datasets plus four WDC Products scales from approximately 2.5K to 30K pairs.The WDC Products variants are small, medium, large, and xlarge.Source: F Full Dataset Inventory · p. 25

DLTE

  • DLTE comprises 47,772 tables built from 1,379 parent tables, 16,548 fragments, and 36,740 CKAN distractors across clean, schema, cell, and hard noise tiers.The parent tables are split 827/207/345 across train, development, and test sets.Source: F Full Dataset Inventory · p. 25

Baseline coverage

  • Task-local baselines are evaluated only where applicable across TRL-Bench tasks, with coverage rationale provided in Appendix G.Table 10 records applicability using checkmarks and blanks.Source: Table 10 · p. 26

G Task-Local Baselines

TRL-Bench combines embedding, embedding-free, and analytical task-local baselines to separate encoder signal from probe-head, raw-data, and label-distribution effects. String-similarity baselines establish meaningful floors for row linkage, while per-dataset reporting cross-validates baseline measurements against the main results.

Baseline roles

  • Task-local baselines comprise frozen Random and TF-IDF embeddings, raw-data matchers, and Chance/Dummy analytical references serving distinct diagnostic roles.Embedding baselines share the downstream pipeline with neural encoders; embedding-free methods bypass it, while analytical baselines use dataset statistics alone.Source: G Task-Local Baselines · p. 26

Analytical and embedding baselines

  • Random embeddings match neural-encoder dimensionality at every supported granularity, so underperformance signals failure whereas parity indicates the probe head is doing the work.Chance provides classification and retrieval floors, while Dummy tests whether majority-class or mean prediction suffices; every supervised probe reports Dummy alongside linear and MLP heads.Source: G Task-Local Baselines · p. 26

TF-IDF implementations

  • TF-IDF baselines use character n-grams over serialized headers and sampled values, with 256-dimensional column vectors and 512-dimensional row vectors fitted per dataset.Column TF-IDF applies to compatible column-type and clustering tasks; row TF-IDF fits vocabulary jointly over both linkage tables and uses the same probe heads as neural row encoders.Source: G Task-Local Baselines · p. 26

Embedding-free matchers

  • The benchmark also evaluates containment, Hungarian union matching, Valentine Jaccard/Distribution matchers, and cosine-threshold linkage under task-appropriate metrics and reporting conventions.Tables 11–13 provide 5-seed per-dataset means, with intentional overlap against the main tables as a cross-validation check.Source: G Task-Local Baselines · p. 27

Row-linkage results

  • F1 = 0.380/0.495/0.227 for TF-IDF row and 0.353/0.481/0.255 for Jaccard row on DM-C/DM-D/WDC, above Random/Dummy and below strongest learned encoders.These results use the canonical avg(MLP, linear) probe protocol and define complementary sub-word and token-level linkage floors.Source: G Task-Local Baselines · p. 27, Table 12 · p. 28

H Metric Definitions and Normalized Rank … K.1 Observational Lexical-Accessibility Proxies for CTBench

TRL-Bench defines normalized ranking and task metrics, documents table-footprint coverage, and summarizes granularity-dependent performance. CTBench diagnostics associate generic-text advantages with lexical accessibility while objective-aligned specialists win selected tasks, though the evidence is observational.

H Metric Definitions and Normalized Rank

  • Normalized rank averages model ranks over the finest available units, excludes missing units from each model’s average, breaks ties by minimum rank, and lower values are better.The units differ by aggregate: CTBench uses tasks, row prediction uses target columns, and linkage uses datasets; absolute values should not be compared across suites.Source: H Metric Definitions and Normalized Rank · p. 30
  • Metrics use macro F1 for multiclass tasks, positive-class binary F1 for record linkage, nRMSE = 1 − R2 for regression, and shifted geometric aggregation across regression targets.The shift ε = 0.01 reduces outlier sensitivity and prevents collapse at zero; linkage group scores are unweighted means of per-dataset binary F1.Source: H Metric Definitions and Normalized Rank · p. 30

I Table-Footprint Coverage Across Suites

  • Table-footprint accounting counts each distinct loadable table once per role, yielding 47,772 DLTE retrieval-lake candidates and 87 dataset-source entries without weighting benchmark scores by table size.Labeled pairs and split indices are metadata rather than additional table inputs.Source: I Table-Footprint Coverage Across Suites · p. 31
  • Figures 5 and 6 characterize suite-specific row-column densities and logarithmic cell-footprint distributions using normalized intensities, medians, and 95th percentiles.The visualizations account for differing suite sizes and footprint scales rather than presenting performance results.Source: Figure 5 · p. 31, Figure 6 · p. 32, Table 14 · p. 32

J Family-Level Performance Summary Figure

  • Granularity-dependent radar profiles show no model dominates every capability axis, with strengths shifting substantially across representation granularity and task family.BERT, GTE, and TABBIE appear in both panels, while other models are evaluated only in their supported suites; values are rank-normalized within each axis.Source: Figure 7 · p. 35

K.1 Observational Lexical-Accessibility Proxies for CTBench

  • STARMIE wins schema matching (−0.067) and union search (−0.010), TURL wins Table QA (−0.016), and TAPAS wins Table Subset with 0.567 F1.Table Subset is specialist-won by the individual-best criterion, although the specialist mean is borderline at +0.006 versus generic text.Source: §K.1 · p. 36
  • Generic text encoders lead 10 of 13 CTBench tasks, especially text-dominated retrieval, search, typing, relation, and clustering tasks, while specialists win objective-aligned tasks.Table 15 operationalizes lexical accessibility through baseline gaps and generic-text advantages; the evidence is observational rather than causal.Source: Table 15 · p. 36, §K.1 · p. 36

K.2 Probe Head Complexity for Column/Table-Level Tasks … L.4 Record Linkage Split and Leakage Audit

The appendices show that representation quality, probe capacity, aggregation, and split design affect performance in task-specific ways across column/table and row-level evaluation. They also validate robust cross-table conclusions through low-overlap, hybrid-retrieval, dimension, linkage-head, and leakage audits.

K.2 Probe Head Complexity for Column/Table-Level Tasks

  • Frozen embeddings carry substantial column/table signal: linear probes beat dummy baselines by +0.65, +0.72, +0.12, +0.39, and +0.05 across five tasks, while nonlinear gains vary by task.MLPs add little on ColType and ColRel, help UnionCls and TblSubset, and slightly hurt JoinCls, where the average gap is −0.02.Source: §K.2 · pp. 36–37, Table 16 · p. 37

K.3 Aggregation Ablation for Table-Level Embeddings

  • Aggregation preferences are task- and probe-dependent rather than universal, with TOK-MEAN or COL-MEAN often best but several gaps tiny and model-specific variation modest.The main comparison therefore uses each encoder’s strongest supported aggregation averaged across MLP and linear probes.Source: §K.3 · p. 37, Table 17 · p. 38, Table 18 · p. 39

K.4 Join Search: Direct Cosine vs. Learned Projection

  • The appendix adopts learned projection for join search because directional containment better matches transfer objectives than direct cosine similarity.Direct cosine remains an informative no-training control, but learned projection is the canonical minimal probe over frozen embeddings.Source: §K.4 · p. 37

K.5 Union Search: TUS vs. TUS-hard

  • On TUS-hard, the overlap baseline falls to 0.008 and rankings reverse substantially (Spearman ρ = −0.67), with STARMIE and TABERT most robust under low overlap.TUS-hard removes the highest-overlap 36% of positives and complements rather than replaces original TUS.Source: §K.5 · pp. 37–38, Table 20 · p. 40

K.6 Table Retrieval: Model-Only vs. Hybrid Mode

  • Hybrid table retrieval improves every model’s MRR by +0.057 to +0.509, but converges to 0.509–0.555, so main comparisons use model-only representations.The convergence indicates that query-encoder table embeddings contribute most shared signal in the augmented setting.Source: §K.6 · p. 38, Table 21 · p. 40

K.7 Query Encoder Sensitivity

  • MPNet is the default query encoder: it averages +0.02 over ST5 for table retrieval, whereas ST5 averages −0.03 relative to MPNet for table QA, with differences ≤0.05.The reversal suggests longer-context pretraining helps semantic parsing, although query-encoder sensitivity remains limited.Source: §K.7 · p. 41, Table 22 · p. 41

K.8 Pair-Level Random vs. Table-Disjoint Split Ablation; L.1 Row-Prediction Probe Diagnostics

  • Table-disjoint evaluation is harder but largely preserves rankings, dropping join-classification mean F1 from 0.736 to 0.523 and increasing union-regression average nRMSE by 0.110.This protocol is therefore used by default for frozen cross-table transfer; row diagnostics likewise show TABICL broad wins and regime-specific strengths, including 57/77 classification and 38/46 regression targets.Source: §K.8 · p. 41, Table 23 · p. 42, §L.1 · p. 43

L.2 Embedding Dimension for Record Linkage; L.3 Probe Head for Record Linkage; L.4 Record Linkage Split and Leakage Audit

  • Row-linkage diagnostics favor native higher dimensions and nonlinear readouts: All-pairs F1 rises from 0.058 at d=64 to 0.139 at d=768, while MLP reaches 0.330 versus 0.166 for linear.Cosine still averages 0.365 overall and leads on WDC, where label-equivalent columns are removed and strict row-disjoint rankings remain highly correlated with canonical rankings (ρ = 0.94).Source: §L.2 · p. 45, Table 26 · p. 45, §L.3 · p. 46

L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders · L.6 Intrinsic-Geometry Diagnostics: Per-Head Breakdowns

The study evaluates eight intrinsic geometry diagnostics on exported row embeddings under a common protocol, grouping them into Spectral Spread, Spectral Shape, and Spatial Structure. Spectral Spread best predicts downstream utility overall, while diagnostic value differs between record linkage, classification, and regression and across probe heads.

L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders

  • The analysis computes all eight deterministic diagnostics from one frozen row-embedding matrix per model and dataset, using SVD and the centered covariance eigenspectrum.The taxonomy frames diagnostics as Spectral Spread, Spectral Shape, and Spatial Structure; αreq uses covariance eigenvalues on the Agrawal et al. scale.Source: §L.5 · pp. 49–50
  • The protocol aggregates within-task Spearman correlations across 13–14 row-capable models, excludes RANDOM, and tests per-task correlation distributions against zero with Wilcoxon signed-rank tests.Row Prediction combines macro-F1 for classification and −nRMSE for regression, while the detailed breakdowns appear in Tables 31–34.Source: §L.5 · p. 50
  • Spectral Spread is the strongest predictive family: RankMe (+0.485), RankMe* (+0.471), and NESum (+0.460) are the leading positive correlates overall.Higher effective rank and more uniform singular-value distributions consistently align with better performance across row-level tasks.Source: §L.5 · pp. 50–51
  • Record Linkage favors heavy-tailed spectral decay and manifold structure, with αreq ρ = −0.746 and d̂TwoNN ρ = +0.398, both p < 0.001.Intrinsic metrics correlate more strongly with Record Linkage than with Row Prediction, making them better proxies for entity matching.Source: §L.5 · pp. 51–52
  • Row Prediction shows weaker and task-dependent geometry signals: αreq is near zero overall (ρ = +0.003, p = 0.94), while d̂TwoNN is nonsignificant (ρ = −0.042, p = 0.095).Classification and regression partially cancel for αreq, indicating that spectral-shape utility is ambivalent for feature-based prediction.Source: §L.5 · pp. 51–52
  • Regression amplifies both positive Spectral Spread and negative clustering signals: RankMe reaches MLP Spearman +0.328, while Self-Cluster reaches MLP Spearman −0.327.RankMe* reaches +0.356 Spearman on the MLP head, whereas tighter intra-model clustering co-varies with worse regression.Source: §L.5 · p. 52

L.6 Intrinsic-Geometry Diagnostics: Per-Head Breakdowns

  • Per-head breakdowns place MLP and Linear results side by side, with Record Linkage additionally evaluated using Cosine-Threshold heads for direct probe-complexity comparison.Regression correlations use −nRMSE so positive values mean that higher diagnostic values correspond to better performance.Source: §L.6 · pp. 53–54, Table 32 · p. 53, Table 33 · p. 53

M DLTE Operator Specification … N.2 Pipeline Component Sensitivity

TRL-DLTE uses a fixed three-stage pipeline that retrieves candidate tables, classifies column relations, and performs profile-based row matching and merging. Its standardized calibration and diagnostics show that retrieval and column alignment strongly shape downstream recovery, while composition remains conditional on stage interactions.

M DLTE Operator Specification; M.1 Stage-1 Retrieval: Scoring and Pool

  • For each query, the pipeline retrieves the top K = 100 tables by cosine similarity from a normalized FAISS index, aligns columns, and returns an enriched table after union and join operations.Column-capable encoders use mean-pooled column embeddings for retrieval, while TAPEX and TUTA use native table embeddings.Source: M DLTE Operator Specification · p. 55, §M.1 · p. 55

M.2 Stage-2 Alignment and Classification: Thresholds and Grid

  • Stage 2 assigns UNION, JOIN, or NONE from Hungarian-matched column similarities using n⋆, r, µ, and m, with five thresholds grid-searched over 4,500 combinations per pair.Macro-F1 is optimized because NONE comprises approximately 98% of pairs, preventing majority-class accuracy from dominating calibration.Source: §M.2 · pp. 55–56, M DLTE Operator Specification · p. 55

M.3 Stage-3 Row Matching: CSLS and Profiles

  • Stage 3 selects the highest-confidence UNION and JOIN candidates, then applies CSLS-based reciprocal row matching with distinct fixed profiles for union and join operations.Union confidence uses min(r, µ), join confidence uses m, and ties follow Stage-1 rank; enriched tables are not re-encoded.Source: §M.3 · p. 56

M.4 Stage-3 Second-Pass Join on Union-Appended Rows

  • A second reciprocal pass matches union-appended rows to remaining join-side rows, enabling hard-region recall that otherwise collapses to zero for every row model.Newly matched cells use the Stage-2 key-pair alignment, and second-pass match counts are logged separately.Source: §M.4 · p. 57

M.5 What is and is not tuned

  • The pipeline fixes Stages 1 and 3, calibrates five Stage-2 thresholds per backbone–column-model pair on dev macro-F1, and holds each vector fixed across 14 row models.Calibration excludes test labels, end-to-end metrics, and row-model choice; headline pipeline selection separately uses dev UJ-H.Source: §M.2 · p. 56, §M.5 · p. 57

N DLTE Detailed Rankings

  • Dev and test pipeline rankings are highly stable, with Spearman ρ(dev, test) = 0.96 and 42 of the top 50 pipelines shared.This supports using dev-selected pipelines as descriptive of the broader test landscape across 1,120 canonical pipelines.Source: N DLTE Detailed Rankings · p. 58

N.1 Cell F1 as a Complementary Diagnostic

  • Cell F1 identifies STARMIE as the strongest retriever at 0.601, TABERT as the strongest column model at 0.628, and TABTRANSFORMER, SUBTAB, and SAINT as leading row models.The Stage 2 marginal span is 0.084, whereas the Stage 3 span is 0.026; oracle analysis attributes leading row-model Cell F1 mainly to union recovery.Source: §N.1 · p. 59

N.2 Pipeline Component Sensitivity

  • With Stage 3 fixed to TABTRANSFORMER, Starmie retrieval averaged 0.609 Cell F1, while TaBERT alignment averaged 0.650; weak retrieval created a downstream performance ceiling.The Stage 2 spread was 0.106, and alignment effects were conditional on the upstream retriever and downstream row matcher because composition was non-additive.Source: §N.2 · p. 60, Table 37 · p. 60

N.3 Per-Stage Marginal Analysis … N.8 Source Split: TabFact vs. WTQ

TRL-DLTE performance is dominated by Stage 2 and non-additive compositional fit, while Stage 3 identity-resolution differences emerge clearly when upstream stages are controlled. These findings, including hybrid-pipeline advantages, remain largely stable across parent sources.

N.3 Per-Stage Marginal Analysis

  • Stage 2 has the widest marginal spans—0.084 Cell F1 and 0.060 UJ-H—whereas Stage 3 spans only 0.026 and 0.013, so upstream effects mask row-model differences.Stage 1 retrieval recall also fails to predict downstream contribution: GTE reaches 0.801 recall@100, but STARMIE leads Cell F1 at 0.601.Source: §N.3 · pp. 60–61, §N.5 · p. 63
  • The test rank-1 pipeline STARMIE/GTE/GTE scores 0.253 UJ-H, exceeding the dev-selected TUTA/GTE/GTE at 0.229 and the marginal-leader STARMIE/TABBIE/TRANSTAB at 0.134.The gap between marginal assembly and end-to-end optima, plus split-dependent marginal leaders, indicates residual non-additive compositional fit.Source: §N.3 · p. 61

N.4 Oracle-RA Row-Model Diagnostic

  • Oracle-RA expands the row-model UJ-H spread to 0.546, revealing identity-resolving models as strong balanced matchers and union-dedup specialists as near-zero join-recall models.The taxonomy persists across noise tiers, with identity-resolving models occupying the top three and union-dedup specialists the bottom three at every tier.Source: §N.4 · p. 62, Table 39 · p. 62, Table 40 · p. 63
  • Oracle-RA UJ-H and robust-linkage rankings correlate at |ρSpearman| = 0.80 and |τKendall| = 0.63, linking DLTE row matching to atomic cross-table identity resolution.GTE and TRANSTAB lead both views, while SUBTAB and TABTRANSFORMER occupy the weakest robust-linkage positions.Source: §N.4 · pp. 62–63

N.5 Full Per-Stage Model Rankings

  • Across full marginal rankings, STARMIE leads Stage 1 downstream Cell F1 at 0.601 despite GTE’s higher 0.801 recall@100, showing candidate usability matters beyond retrieval volume.Retrieved candidates must support downstream column alignment and row matching, making Stage 1 contribution compositional rather than recall-determined.Source: §N.5 · pp. 63–64, Table 41 · p. 63

N.6 Stage 3 Row Matching: Full Pipeline Rankings

  • TaBERT dominates Stage 2 in full-pipeline ablations, while Stage 3 varies only 0.589–0.679 within the best Starmie→TaBERT configuration and averages 0.566–0.591 globally.Starmie is the strongest Stage 1 choice with average Cell F1 0.650; TAPEX is lowest at 0.558.Source: §N.6 · pp. 65–66, Table 44 · p. 65

N.7 Top-20 Pipeline Combinations

  • Cell F1 and UJ-H select disjoint top-20 pipeline lists: Cell F1’s entries all use TaBERT, whereas UJ-H is dominated by GTE, BERT, and TURL at Stage 2.The divergence reflects Cell F1’s pooled recovery emphasis versus UJ-H’s balanced recovery of both removed blocks.Source: Table 45 · p. 67, Table 46 · p. 67, §N.7 · p. 67

N.8 Source Split: TabFact vs. WTQ

  • The dev-selected hybrid TUTA/GTE/GTE beats monolithic BERT/BERT/BERT by +0.086 UJ-H on TabFact and +0.099 on WTQ, while pipeline rankings correlate at ρ = 0.871 across sources.Oracle-RA row-model rankings correlate at ρ = 0.987, with the same identity-resolving and union-dedup families at the top and bottom.Source: §N.8 · pp. 68–69, Table 47 · p. 68, Table 49 · p. 69

O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding … P.2 Task and Dataset Summary

The proprietary embedding ablation separates retrieval and structural-grounding capabilities across column/table and row tasks, while the appendix robustness diagnostics use Observatory’s protocols and remain outside the benchmark’s 16 tasks. A unified table-first aggregation reports diagnostic statistics over valid tables and applies a changed-only rule for perturbation robustness.

O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding

  • The ablation evaluates TE3-SMALL, TE3-LARGE, and ADA-002 across Table Retrieval, Table QA, Row Prediction, and Record Linkage.TE3-SMALL and TE3-LARGE use 768 dimensions for parity with open-source encoders, while ADA-002 retains its native 1536 dimensions.Source: O Proprietary Embedding Ablation:… · p. 70
  • ADA-002 leads Table Retrieval with 0.540 MRR versus GTE’s 0.476, while proprietary models rank only 5–7 on Table QA behind TURL, TABBIE, and STARMIE.The contrast indicates distinct retrieval and structural-understanding capabilities within the same task family.Source: O Proprietary Embedding Ablation:… · p. 70
  • On Record Linkage, TE3-LARGE ranks first with 0.426 F1 versus GTE’s 0.403, whereas Row Prediction is led by TABICL at 0.816 AUROC over TE3-SMALL’s 0.801.Proprietary embeddings lead matching tasks, but specialized models remain stronger on understanding tasks.Source: O Proprietary Embedding Ablation:… · p. 70
  • The OpenAI ablation is a controlled study within the embedding-only regime, not a proxy for evaluating 7B–70B generative systems.Its scope and model-selection rationale are discussed in Appendix B.Source: O Proprietary Embedding Ablation:… · p. 70

P Robustness

  • The robustness appendix recomputes Sample Fidelity, Perturbation Robustness, and Row/Column Order Insignificance from finalized embeddings, following Observatory [15] definitions and datasets.These auxiliary diagnostics are not part of the contribution and are excluded from the 16 benchmark tasks.Source: P Robustness · p. 71
  • The diagnostics use table-first aggregation, reporting dataset-level means and standard deviations over valid tables; cosine similarity is the primary cross-task metric.Sample Fidelity combines cosine similarity and MCV per table, Row/Column Order Insignificance separates them, and Perturbation Robustness reports changed-only cosine similarity.Source: P Robustness · p. 71

P.1 Unified Aggregation Protocol

  • The unified protocol defines table-level metrics by aggregating valid transformed variants and valid items, then reports their evaluation-set mean and standard deviation; MCV follows Observatory’s implementation.For Perturbation Robustness, only actually modified columns are included, and tables without changed columns are excluded.Source: §P.1 · p. 71

P.2 Task and Dataset Summary

  • Table 51 provides an overview of the three appendix diagnostics.It summarizes the diagnostic scope introduced for the robustness appendix.Source: Table 51 · p. 72

P.3 Sample Fidelity … P.7 Implementation Notes and Caveats

Across standardized robustness diagnostics, models exhibit capability-specific strengths rather than a single universal ranking: tabbie is the strongest all-round choice, while other models specialize by perturbation type. Unified table-first scoring and task-specific MCV caveats preserve the qualitative conclusions but constrain absolute-number comparisons.

P.3 Sample Fidelity

  • Sample fidelity increases for all eight models as retained rows rise from 0.25 to 0.75; tabert is most stable, tabbie the strongest balanced alternative, and tabsketchfm degrades most at 0.25.Scores compare original and subsampled column embeddings using cosine similarity and MCV, aggregated across columns within each table.Source: §P.3 · pp. 72–73, Table 53 · p. 73

P.5 Row/Column Order Insignificance

  • The order diagnostic evaluates six combinations of embedding granularity and row- or column-shuffle type, with shuffled items realigned when necessary before comparison.The dataset contains 4,451 base tables with 11 variants and 513 with 6 variants; models that canonicalize order need no extra realignment.Source: §P.5 · pp. 73–74

P.4 Perturbation Robustness

  • Perturbation robustness is hardest for content equivalence, where tabbie leads; starmie is strongest for schema changes, whereas gte and especially tabsketchfm are more sensitive.Content-equivalence comparisons use only 29 valid tables, so those comparisons are indicative rather than definitive.Source: §P.4 · p. 73, Table 55 · p. 74

P.5 Row/Column Order Insignificance

  • Row/row representations are often nearly invariant, but row/column is harder; tabbie is the most balanced shared column/table model, while tabpfn is nearly ideal for row representations.tabtransformer is least robust for rows, consistent with learned column positional embeddings, whereas tabsketchfm is highly order-stable despite weaker performance elsewhere.Source: Table 56 · p. 74, §P.5 · p. 74

P.6 Cross-Task Comparison of Shared Models

  • tabbie has the best cross-task average robustness, while tabert leads sample fidelity, starmie schema perturbations, and tabsketchfm specializes in order invariance.The comparison ranks eight models by average cosine performance across sampling, semantic perturbation, and structural reordering; lower average rank indicates better overall robustness.Source: §P.6 · pp. 74–75, Table 58 · p. 75

P.7 Implementation Notes and Caveats

  • All headline results are recomputed from finalized embeddings using table-first aggregation, while perturbation scores include only changed columns and omit MCV.Dataset-level means and standard deviations are computed after per-table aggregation, matching the unified headline protocol.Source: §P.7 · p. 75
  • MCV magnitudes are comparable only within the same task because variant counts differ and covariance estimates are rank-deficient; these caveats do not alter qualitative conclusions.Order shuffles can involve up to 11 variants, and all covariance estimates use K+1 samples in d-dimensional space with K+1 ≪d.Source: §P.7 · p. 75

Q Computational Efficiency

Embedding-generation efficiency varies substantially by granularity and training regime: generic text encoders are fastest for columns, TAPEX is far faster than TUTA for tables, and TABICL is far faster than TRANSTAB for rows. Training-based models scale super-linearly with row count, while model support limits and feature-count costs constrain deployment at larger scales.

Methodology

  • The controlled suite times unmodified production scripts end-to-end on one NVIDIA L40S GPU, covering real, semi-synthetic scaling, and bridge tables.Measurements include loading, preprocessing, per-table training, and target-agnostic row-forward extraction where the same matrix is reused across curated targets.Source: Q Computational Efficiency · pp. 76–77

Column embeddings

  • Generic text encoders lead column efficiency: BERT (5.3 s) and GTE (5.7 s) are 2.7× faster than the slowest models, TABERT (13.2 s) and TABSKETCHFM (14.1 s).Column cost is dominated by per-column tokenization overhead, and the gap widens approximately linearly with column count.Source: Q Computational Efficiency · p. 76, Table 59 · p. 76

Embedding costs

  • Row costs span 101×, from TABICL (8.7 s) to TRANSTAB (875 s), while table costs span 20×, from TAPEX (6.1 s) to TUTA (119.6 s).The row gap is primarily driven by per-table training in target-table self-supervised models; TAPEX’s table advantage reflects TUTA’s cell-level tokenization overhead.Source: Q Computational Efficiency · p. 76, Table 60 · p. 76, Table 61 · p. 77

Scaling behavior

  • Training-based row embedders scale super-linearly with row count, whereas frozen inference models scale approximately linearly; feature-count scaling is more uniform.This behavior is shown in Figure 12, while the column sweep attributes slower TABERT and TABSKETCHFM costs to tokenization.Source: Q Computational Efficiency · p. 76, Figure 12 · p. 77

Support envelope

  • Deployment support is uneven: TURL runs out of memory above 208 columns or 256 features, while TABPFN rises from 18 s at 28 features to 263 s at 208 features and times out at 1 775 features.SAINT and TABTRANSFORMER also fail on the widest 1 775-column anchor table under the one-hour budget.Source: Q Computational Efficiency · pp. 76–77

R Reproducibility Details … V Licenses and Asset Documentation

TRL-Bench specifies reproducible data access, splits, probing, seeds, statistical tests, and dataset-counting conventions, while documenting broader-impact risks and license-compliant asset release. Statistical analyses show broad model differences in RBench and DLTE stage effects, but near-ties across most CTBench models.

R Reproducibility Details

  • The benchmark fixes public-source provenance and evaluation procedures through canonical or reconstructed splits, parent-table-level DLTE manifests, unified probe heads, five seeds, and executable wrapper configurations.Supervised probes average linear and one-hidden-layer MLP heads, using hidden size 256, Adam at 10−3, up to 100 epochs, and validation early stopping; DLTE uses 827/207/345 train/dev/test parents and all 1,120 canonical pipelines completed five rounds.Source: R Reproducibility Details · p. 78

S Statistical Reporting

  • Statistical reporting uses five-seed means, per-task raw scores, macro-averaging across 123 row targets, and Friedman tests with Holm-corrected Wilcoxon comparisons.TABTRANSFORMER covers 63 row targets and is excluded from matched-sample RBench significance tests; normalized-rank aggregates receive no confidence intervals.Source: S Statistical Reporting · p. 79, Table 62 · p. 79
  • CTBench rejects equal performance overall, but only 6/28 pairwise comparisons are significant, with TABSKETCHFM weaker than top models and most models forming a near-tie.The Friedman test gives χ2 = 32.1, p < 0.0001; BERT has the best mean rank at 2.23, while family-level significance appears for Schema but not Join or Union.Source: S Statistical Reporting · p. 79, Figure 13 · p. 80
  • RBench shows strong model differences: TABICL outperforms every other model on both classification and regression, with 26/78 and 38/78 significant pairwise comparisons, respectively.Classification yields χ2 = 209.4, p = 3.7 × 10−38, while regression yields χ2 = 154.6, p = 6.5 × 10−27; all TABICL pairwise advantages have padj < 0.001.Source: S Statistical Reporting · p. 79
  • DLTE stage effects are strongest for column alignment, while Stage 3 shows a metric-dependent gap: W = 0.61 on Cell F1 versus W = 0.09 on UJ-H.Stage 2 has W = 0.79 on Cell F1 and 0.66 on UJ-H; upstream retrieval and alignment errors mask join-side Stage 3 signal in UJ-H, so average stage effects do not determine the best composition.Source: S Statistical Reporting · p. 80, Table 63 · p. 81

T Dataset Counting Protocol

  • At the finest source granularity, TRL-Bench contains 87 datasets, while alternative grouping conventions yield totals of 84, 77, or 28.The main paper uses dataset-source-level counting by default: 20 CTBench, 50 OpenML row-prediction, 16 record-linkage, and 1 DLTE source.Source: T Dataset Counting Protocol · p. 82

U Broader Impact / V Licenses and Asset Documentation

  • The benchmark can improve comparability and model selection, but sensitive records and linkage applications require attention to surveillance, re-identification, privacy, provenance, and usage restrictions.Derived assets follow upstream licenses: redistributable files are released directly, while restricted sources receive reconstruction scripts and manifests with per-asset provenance, URLs, licenses, and redistribution status.Source: U Broader Impact · p. 83, V Licenses and Asset Documentation · p. 84, Table 64 · p. 85

Related papers

Loading 2606.09323v1…