PaperlayerView exact revision on arXiv

Source-linked AI summary

TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders

Wei Pang, Xiangru Jian, Hehan Li, Zhixuan Yu, Alex Xue, Jinyang Li, Zhengyuan Dong, Xinjian Zhao, Hao Xu, Chao Zhang, Reynold Cheng, M. Tamer Özsu, Tianshu Yu

arXiv:2606.09323v1cs.AIcs.DB

TL;DR

Tabular encoders are difficult to compare because task-specific pipelines mix representation quality with wrappers, budgets, and adaptation. TRL-Bench standardizes representation-level probing across granularities and finds that transfer is capability-specific, with compositional pipelines favoring capability-matched hybrids over universal encoders.

  • Problem

    Task-specific end-to-end evaluation makes heterogeneous tabular encoders difficult to compare directly because results mix encoder quality with downstream wrappers and adaptation.

  • Method

    TRL-Bench exports row-, column-, or table-level embeddings and evaluates them with shared lightweight readouts across column/table transfer, row transfer, and compositional enrichment suites.

  • Results

    Across 20 models and 16 tasks, transfer is capability-specific: task-aligned specialists and capability-matched hybrid pipelines outperform any universal encoder strategy.

  • Takeaways & Limitations

    Reusable tabular representations should be selected by capability and composed across pipeline stages rather than ranked by a single universal leaderboard.

  • Takeaways & Limitations

    TRL-Bench measures what shared lightweight readouts extract from exported embeddings, not the best fully adapted end-to-end systems, and documented wrappers introduce some variation.

Abstract

from arXiv · show

Tabular encoders are usually evaluated inside task-specific end-to-end pipelines, so models from different training paradigms are difficult to compare directly even when they operate on similar tabular signals. We introduce TRL-Bench, a multi-granular tabular representation learning (TRL) benchmark that standardizes cross-paradigm representation-level evaluation: each encoder exports row-, column-, or table embeddings through its supported wrapper, and shared lightweight heads probe them across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment spanning all three granularities). To support this standardized setting, we release curated benchmark assets and task reformulations, including 50 OpenML tables with 123 verified targets, 16 row-pair linkage rewrites, and a 47,772-table DLTE lake derived from 1,379 parent tables. Across 20 models and 16 tasks, TRL-Bench shows that once downstream conditions are standardized, encoder quality is capability-specific rather than captured by a single leaderboard. In TRL-CTbench, generic text encoders often lead on tasks with strong surface-text signal, while tabular specialists win where their pretraining objective aligns with the task. In TRL-Rbench, within-table prediction and cross-table linkage favor different training regimes, with atomic linkage performance correlating strongly with the row-matching stage of DLTE pipelines. In TRL-DLTE, the strongest pipelines combine capability-matched specialists rather than reuse a single encoder, and top end-to-end quality depends on non-additive compositional fit rather than per-stage marginal rank alone. TRL-Bench provides a common protocol for measuring reusable signal in exported tabular representations under shared downstream conditions. Code and data: https://github.com/LOGO-CUHKSZ/TRL-Bench

1 Introduction

TRL-Bench evaluates reusable row-, column-, and table-level representations directly by exporting embeddings once and testing them with shared lightweight modules under standardized downstream conditions. Across its three suites, the benchmark shows that transfer is capability-specific, no single pretraining recipe is universally best, and effective data-lake enrichment requires compositional model choice.

  • Standardized protocol: TRL-Bench exports each encoder’s supported row-, column-, or table-level embeddings once, then evaluates them with shared lightweight downstream modules instead of end-to-end fine-tuning.This protocol isolates reusable representation quality and enables direct comparison across heterogeneous training paradigms under common task definitions.
  • Benchmark scope: The benchmark measures retrieval, schema alignment, linkage, prediction, and grounding through TRL-CTBENCH, TRL-RBENCH, and TRL-DLTE at the granularities where embeddings are reused.The suites cover column/table transfer, row transfer, and compositional data-lake table enrichment.
  • Compositional evaluation: TRL-Bench treats atomic capabilities as reusable building blocks and uses TRL-DLTE to evaluate their composition in multi-stage data-lake enrichment.This framing supports evaluating pipeline composition rather than only isolated task performance.
  • Empirical findings: Across 20 models and 16 tasks, standardized downstream conditions reveal capability-specific transfer rather than a universally dominant encoder class.In TRL-CTBENCH, generic text encoders often lead when tasks have strong surface-text signal, while other wins align better with pretraining objectives and task demands.
  • Empirical findings: Within-table prediction and noisy cross-table linkage distinguish different row-transfer behaviors, exposing structural gaps that single-paradigm or single-granularity evaluations cannot isolate.The introduction identifies row signal as multifaceted rather than reducible to one capability.

2 Related Work and Positioning

Prior tabular research is fragmented by representation granularity and typically evaluated in task-specific settings. TRL-BENCH positions itself as a shared, multi-granular representation-level protocol complementing narrower existing benchmarks.

  • Model families and evaluation traditions: Prior row-level models emphasize supervised prediction and transfer, whereas column/table models target schema semantics, grounding, retrieval, and discovery.
  • Model families and evaluation traditions: These model families are usually evaluated in task-specific settings rather than under a shared multi-granular representation-level protocol.
  • Positioning of TRL-BENCH: Existing benchmarks generally target narrow task scopes and compare models within a single task family or end-to-end pipeline.
  • Positioning of TRL-BENCH: TRL-BENCH complements existing resources by standardizing heterogeneous tabular encoders into a shared representation-level evaluation protocol across columns, rows, and tables.

3 Benchmark Design

TRL-Bench standardizes representation-level evaluation of heterogeneous tabular encoders across row, column, and table granularities under shared downstream protocols. Its three suites test atomic capabilities and multi-stage composition through curated prediction, linkage, and table-enrichment tasks.

  • Benchmark scope: TRL-Bench comprises TRL-CTBENCH, TRL-RBENCH, and TRL-DLTE to compare exported representations across columns, rows, tables, and end-to-end composition.The benchmark evaluates heterogeneous encoders without forcing a single input serialization.
  • TRL-DLTE: 47,772 tables populate the DLTE lake, comprising 11,032 targets and 36,740 CKAN distractors derived from 1,379 TabFact/WTQ parent tables.Parent tables are fragmented at four cumulative noise tiers: clean, schema, cell, and hard.
  • Shared protocol: A downstream module maps exported embeddings to task outputs using training-free geometry, lightweight learned probes, or fixed pairwise combination operators.Supervised probes average a linear head and a one-hidden-layer MLP with hidden size 256 as the canonical score.
  • TRL-CTBENCH: 13 TRL-CTBENCH tasks cover schema understanding, joinability, unionability, and grounding across 8 column-level and 5 table-level tasks.Grounding combines frozen text-query embeddings with evaluated table-side representations, while join and union tasks probe complementary or stackable tables.
  • TRL-RBENCH: TRL-RBENCH evaluates whether target-agnostic row embeddings transfer within tables through row prediction and across tables through record linkage.Its linkage suite contains 16 datasets: 8 clean DeepMatcher benchmarks, 4 dirty variants, and 4 WDC Products LSPM size variants.
  • TRL-DLTE: TRL-DLTE reconstructs parent tables by removing row and column blocks, then testing table retrieval, column alignment, union/join/none prediction, row matching, and content merging.The three stages use table, column, and row embeddings respectively, and pipelines may use one multi-granular model or different specialists.

4 Experiments

Under a standardized representation-level protocol, TRL-Bench evaluates 20 heterogeneous encoders across CTBench, RBench, and DLTE using shared probes and task-specific metrics. Results show capability-specific specialization, a transfer-scope divide between prediction and linkage, and compositional advantages for capability-matched DLTE hybrids.

  • Experimental setup: The experiments evaluate 20 models spanning generic text, table-aware, structure-aware, column-specialized, target-table self-supervised, and meta-pretrained encoders.CTBench compares the 10 models that natively expose column or table embeddings across 13 tasks.
  • TRL-CTBENCH: Generic text encoders rank best when headers and short cell strings carry most task signal, while their family-level NR worsens from Schema through Grounding.BERT’s NR changes from 0.000 to 0.397, and GTE’s from 0.190 to 0.429, across those families.
  • TRL-CTBENCH: Tabular specialists win when pretraining aligns with task structure: STARMIE reaches 0.662 MAP on Union Search and 0.764 R@GT on Schema Matching.On Table Subset, the top three are TAPAS (0.567 F1), TAPEX (0.558), and TABSKETCHFM (0.553).
  • TRL-RBENCH: Prediction and linkage favor different model families: TABICL leads prediction, whereas BERT leads Clean Linkage and GTE leads Robust Linkage.TABICL reports AUROC 0.816, Macro-F1 0.671, and SGM 0.505; BERT reaches Clean Linkage F1 0.418, and GTE has Robust Linkage NR 0.048.
  • TRL-RBENCH: Target-table self-supervised methods fit intra-table prediction but trail inter-table linkage, while transfer-based encoders provide more comparable row spaces across tables.Target-table SSL methods have mean NR 0.48/0.47 on classification/regression prediction and 0.65/0.65 on Clean/Robust linkage.
  • TRL-DLTE: Capability-matched DLTE hybrids outperform monolithic reuse: TUTA/GTE/GTE reaches 0.229 UJ-H versus 0.139 for BERT/BERT/BERT.Per-stage marginal leaders assemble to 0.134 UJ-H, below the test rank-1 STARMIE/GTE/GTE pipeline at 0.253, showing that optimal compositions matter.

5 Conclusion … E Appendix Task Summary

TRL-BENCH standardizes cross-paradigm comparison around exported embeddings, enabling heterogeneous tabular encoders to be evaluated under a shared representation-level protocol. The surrounding sections define its scope, operating policies, model inventory, task summaries, limitations, and detailed DLTE specifications.

  • 5 Conclusion: TRL-BENCH compares heterogeneous encoders through exported embeddings without conflating representations with task-specific wrappers, retraining budgets, or adaptation.It provides a common reference point for building tabular models as portable representations.
  • Appendix: The Appendix organizes supporting material on limitations, related work and scope, model inventory, input policy, and task summaries.It also includes benchmark-protocol adaptations under frozen multi-granular transfer.
  • M DLTE Operator Specification: The DLTE Operator Specification documents stage-1 retrieval, stage-2 alignment and classification, stage-3 row matching, second-pass joins, tuning scope, and detailed rankings.Additional subsections cover complementary diagnostics, component sensitivity, marginal analysis, robustness, efficiency, reproducibility, statistical reporting, dataset counting, broader impact, and asset documentation.
  • A Limitations: TRL-BENCH standardizes task definitions and downstream evaluation, but not raw model preprocessing, and is intended to measure representation-level transfer rather than replace adapted end-to-end benchmarks.Each encoder runs in its documented operating regime through its standard wrapper.
  • B Extended Related Work and Scope: The benchmark complements task-specific model papers and benchmarks by comparing transferable signal across columns, rows, and tables under one cross-paradigm interface.Its scope focuses on compatible models in the roughly ∼1M–1B parameter band rather than reproducing each task family’s strongest end-to-end stack.
  • C Model Inventory: The model inventory records parameter counts for frozen encoders, official-package models, or default-configured trained models, with some models retrained per dataset.The inventory therefore distinguishes checkpoint, package, and trained-model provenance.
  • D Model Input Policy: The input policy evaluates encoders under standard public operating regimes using source-paper or official-code preprocessing and default context settings when available.Wrappers make model-specific choices, including per-row mini-tables for TABBIE and TUTA row embeddings and serialized input for BERT and GTE.
  • E Appendix Task Summary: The Appendix task summary catalogs column- and table-level tasks, row-level and compositional tasks, downstream module types, splits, and DLTE’s stage-wise Pipeline module.The task tables distinguish Geometry, Probe, Learned proj., Dual proj., Decoder, and Pipeline modules.

E.1 Benchmark Protocol Adaptations under Frozen Multi-Granular Transfer

TRL-Bench adapts reused tasks to make frozen, multi-granular transfer evaluations meaningful and prevent leakage, overlap, degeneracy, and trivial retrieval signals. The protocol uses stricter splits, curated targets, harder retrieval variants, and metrics that reflect end-to-end recovery.

  • Protocol foundation: The protocol fixes frozen embeddings and shared lightweight readouts, while adapting reused source tasks for meaningful multi-granular representation transfer.These adaptations address task-specific validity within a common evaluation infrastructure.
  • Cross-table generalization: Table-disjoint splits keep test tables unseen and lower Join Classification F1 by 0.212 on average versus pair-random splits.This adaptation affects Join Classification, Column Overlap, Union Classification, and Union Regression.
  • Union search: For TUS-hard, removing containment ≥0.70 positives makes retrieval harder; the value-overlap baseline drops from 1.000 to 0.008.The change is intended to distinguish lexical overlap from broader union signal, with rankings shifting substantially.
  • Target curation: Human review, label repair, and degeneracy audits reduce 158 candidate Row Prediction tables to 50 released tables.The process removes constant-column targets, near-duplicates, and labeling issues before reusable row-transfer evaluation.
  • Linkage validity: Removing label-equivalent identifiers prevents trivial linkage: strict row-disjoint ablations preserve rankings across all 14 row models with Spearman ρ = 0.94.WDC removes cluster_id and identifiers, while Fodors–Zagats removes class; the ablation reports p = 5.6 × 10−7.
  • DLTE evaluation: DLTE uses UJ-H as its primary metric, adds 36,740 CKAN distractors, and evaluates retrieval over 11,032 targets among 47,772 tables.Model-only retrieval excludes query-side table embeddings to isolate table-side transfer; Cell F1 remains a complementary diagnostic.

F Full Dataset Inventory

The benchmark inventory spans CTBench datasets for schema understanding, joinability, unionability, and grounding, alongside row prediction, record linkage, and DLTE assets. These collections include 50 OpenML tables with 123 targets, 16 linkage datasets, and a 47,772-table data lake.

  • CTBench: CTBench includes 20 datasets covering schema understanding, joinability, unionability, and grounding.The inventory names datasets such as SATO, OpenData, SANTOS, and WikiTableQuestions.
  • Record linkage: Record linkage comprises 16 datasets, including DeepMatcher clean and dirty variants plus four WDC Products sizes.WDC Products ranges from approximately 2.5K pairs for small to approximately 30K pairs for xlarge.
  • DLTE: DLTE contains 1,379 parent tables and a 47,772-table lake, with 16,548 fragments, 36,740 CKAN distractors, and four noise tiers.The parent tables split into 827/207/345 train/dev/test.

G Task-Local Baselines

TRL-Bench evaluates learned tabular representations against task-local baselines that isolate pipeline, label-distribution, and raw string-similarity effects. These baselines include random and TF-IDF embeddings, embedding-free matchers, and unsupervised cosine-threshold linkage.

  • Baseline roles: Task-local baselines distinguish encoder signal from downstream pipeline effects, with Random embeddings testing whether probes alone drive performance.Matching Random indicates that the probe head, rather than the encoder, is doing the work; under-performing Random indicates a failure mode.
  • Baseline roles: Chance provides an analytical floor, while Dummy tests whether class or target distributions alone explain supervised-task performance.Dummy uses majority-class prediction for classification and mean prediction for regression, and accompanies every supervised probe task.
  • Row linkage baselines: Char-TF-IDF and Jaccard row embeddings form complementary subword- and token-level string-similarity floors for record linkage.An encoder that does not beat TF-IDF row at the same matching-head type is not capturing match signal beyond raw character overlap.
  • Embedding-free matchers: Embedding-free baselines cover value-containment joins, Hungarian union search, and Valentine schema matching under their corresponding retrieval metrics.These include Inverted-Index Containment, Hungarian Set Match, and Valentine Jaccard and Distribution matchers.
  • Row linkage baselines: Cosine-threshold record linkage supplies an unsupervised geometry-based comparison against learned linear and MLP readouts.It tunes a single cosine-similarity threshold on the validation split and is reported as a fourth linkage head.

H Metric Definitions and Normalized Rank … K.1 Observational Lexical-Accessibility Proxies for CTBench

TRL-Bench defines standardized metrics and table-footprint coverage for cross-suite evaluation, then summarizes granularity-dependent transfer and CTBench’s observational lexical-accessibility patterns. These analyses show that surface-text accessibility and capability-aligned pretraining help explain task-specific encoder performance.

  • H Metric Definitions and Normalized Rank: Normalized rank averages model ranks over the finest available evaluation units, excludes missing units without penalizing others, and treats lower values as better.CTBENCH family ranks average over tasks, while row-prediction ranks average over target columns.
  • H Metric Definitions and Normalized Rank: Macro-averaged F1 is the default for multi-class tasks, while record linkage reports binary F1 on the match-positive class.Row-prediction AUROC averages per-target ROC-AUC values across 77 classification targets, using binary or weighted one-vs-rest variants as appropriate.
  • I Table-Footprint Coverage Across Suites: TRL-Bench characterizes loadable table inputs by cell footprint Fcell(T) = nrow(T) ncol(T) across feature, entity, query, and lake-table roles.The benchmark counts concrete table objects returned by evaluation loaders and summarizes footprint distributions across its suites.
  • I Table-Footprint Coverage Across Suites: Benchmark scores are computed per task rather than weighted by table count or cell footprint across the 87 dataset-source entries.The entries span Schema, Joinability, Unionability, Grounding, Row Prediction, Record Linkage, and DLTE categories.
  • J Family-Level Performance Summary Figure: Granularity-dependent radar profiles compare column/table capability families—Schema, Join, Union, and Grounding—with row families—Classification, Regression, Clean Linkage, and Robust Linkage.BERT, GTE, and TABBIE appear in both panels because they expose both column- and row-level embeddings.
  • K.1 Observational Lexical-Accessibility Proxies for CTBench: Small neural-versus-baseline gaps observationally indicate stronger surface-text signal, using lexical proxies such as Jaccard, TF-IDF, value overlap, and Valentine matchers.The proxies compare the strongest applicable non-neural baseline with the best neural encoder on each CTBench task.
  • K.1 Observational Lexical-Accessibility Proxies for CTBench: −0.067 and −0.010 are the specialist-minus-generic-text mean gaps for schema matching and union search, where STARMIE’s column-level contrastive objective wins.TURL’s table-language modeling wins Table QA with a −0.016 gap; TAPAS wins Table Subset at 0.567 F1 over BERT and GTE.

K.2 Probe Head Complexity for Column/Table-Level Tasks … L.4 Record Linkage Split and Leakage Audit

The ablations show that exported representations contain substantial reusable signal, but probe capacity, aggregation, retrieval setup, split protocol, and encoder adaptation regime materially affect measured performance. Record-linkage diagnostics further show that nonlinear probing and strict row-disjoint testing change absolute results while preserving the main qualitative rankings.

  • K.2 Probe Head Complexity for Column/Table-Level Tasks: Linear probes already exceed dummy baselines by +0.65, +0.72, +0.12, +0.39, and +0.05 across ColType, ColRel, JoinCls, UnionCls, and TblSubset.These gains indicate that frozen embeddings encode task-relevant structure without task-specific training.
  • K.2 Probe Head Complexity for Column/Table-Level Tasks: MLP gains are negligible on ColType (+0.00) and ColRel (+0.02), larger on UnionCls (+0.19) and TblSubset (+0.24), while JoinCls favors linear probes (−0.02).Negative JoinCls gaps include BERT (−0.05) and TaBERT (−0.06), consistent with MLP overfitting on small training sets.
  • K.3 Aggregation Ablation for Table-Level Embeddings: Aggregation preferences are task-dependent: TOK-MEAN leads several MLP tasks, COL-MEAN leads others, and tiny gaps such as 0.494/0.500/0.497 limit broad conclusions.Per-model variation is modest relative to task variation, so aggregation is not treated as a major analysis axis.
  • K.4 Join Search: Direct Cosine vs. Learned Projection; K.5 Union Search: TUS vs. TUS-hard; K.6 Table Retrieval: Model-Only vs. Hybrid Mode: TUS-hard reduces the value-overlap baseline from MAP 1.000 to 0.008 and reverses rankings (Spearman ρ = −0.67), while hybrid retrieval improves every model by +0.057 to +0.509.The main comparison uses model-only retrieval to isolate each model’s own table representation; learned projection remains canonical for join search.
  • K.7 Query Encoder Sensitivity: MPNet beats ST5 on table retrieval by average ∆=+0.02, whereas ST5 beats MPNet on table QA by average ∆=−0.03.The differences are small, but the task-specific reversal suggests query-encoder sensitivity across grounding tasks.
  • K.8 Pair-Level Random vs. Table-Disjoint Split Ablation: Table-disjoint evaluation lowers performance, with JoinCls dropping 0.212 (0.736 →0.523) and UnionReg nRMSE increasing by 0.110.The protocol measures generalization to unseen tables rather than new pairs from observed tables, while relative rankings are largely preserved.
  • L.1 Row-Prediction Probe Diagnostics; L.2 Embedding Dimension for Record Linkage: TABICL remains strongest after dimensional control, scoring 0.6744 Macro-F1 and 0.4873 SGM, while winning 57/77 classification and 38/46 regression targets.Record-linkage F1 also rises with embedding size overall, from 0.058 at d=64 to 0.139 at d=768.
  • L.3 Probe Head for Record Linkage; L.4 Record Linkage Split and Leakage Audit: For linkage, MLP improves over linear by +0.164 overall (0.330 vs. 0.166), while strict row-disjoint rankings remain highly correlated with original rankings (Spearman ρ = 0.94).Pair overlap is ≤0.02% across all sources, but row overlap is common; nonlinear probing particularly rescues WDC performance.

L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders · L.6 Intrinsic-Geometry Diagnostics: Per-Head Breakdowns

The analysis evaluates eight deterministic, task-free diagnostics of exported row-embedding geometry across spectral spread, spectral shape, and spatial structure, then examines their correlations with row prediction and record linkage. Spectral spread is generally most predictive, while diagnostic utility and probe-level behavior depend strongly on task type.

  • L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders: Eight diagnostics are grouped into Spectral Spread, Spectral Shape, and Spatial Structure and computed deterministically from each frozen row-embedding matrix X.The embedding matrix is decomposed spectrally, and diagnostics capture variance allocation, spectral decay, and point-cloud organization.
  • L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders: Spectral Spread is the most predictive family: RankMe reaches mean Spearman ρ = +0.485, followed by RankMe* (+0.471) and NESum (+0.460).Higher effective-rank measures correspond to more uniform singular-value distributions and stronger downstream rankings across both row-task categories.
  • L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders: The αreq metric is the strongest overall negative correlate, with mean Spearman ρ = −0.372, concentrated on Record Linkage at ρ = −0.746.On Row Prediction, αreq has ρ = +0.003, p = 0.94 because classification and regression signals partially cancel.
  • L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders: Spatial Structure is task-specific: dTwoNN reaches ρ = +0.398 for Record Linkage but ρ = −0.042 for Row Prediction.The stronger Record Linkage correlations indicate intrinsic geometry is a better proxy for entity matching than feature-based prediction.
  • L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders: For Row Prediction regression, RankMe and RankMe* produce MLP Spearman correlations of +0.328 and +0.356, with approximately 80% directional agreement.Their corresponding MLP Pearson correlations are +0.317 and +0.341; the linear-head signal is weaker.
  • L.5 Intrinsic Embedding-Geometry Diagnostics for Row Encoders: Self-Cluster is the dominant negative regression signal, reaching MLP Pearson −0.316 and Spearman −0.327, while tighter clustering co-varies with worse regression.The MLP Spearman direction is consistent on approximately 83% of tasks.
  • L.6 Intrinsic-Geometry Diagnostics: Per-Head Breakdowns: The per-head breakdowns place MLP and Linear probes side by side, with Record Linkage additionally reporting a Cosine-Threshold head.Classification uses macro-F1, while regression uses −nRMSE so positive correlations indicate that higher metric values predict better performance.

M DLTE Operator Specification … N.2 Pipeline Component Sensitivity

The DLTE pipeline uses frozen, staged operators with FAISS table retrieval, Hungarian column alignment, calibrated relation classification, and CSLS-based row matching. Its sensitivity analysis shows retrieval and column-model choices materially affect Cell F1, with Starmie strongest at Stage 1 and TABERT dominant at Stage 2.

  • M DLTE Operator Specification: DLTE retrieves the top 100 candidates using cosine-ranked FAISS search over normalized table embeddings, then aligns columns with Hungarian assignment.Column-capable encoders use column-mean table pooling, while TAPEX and TUTA use native table variants.
  • M.2 Stage-2 Alignment and Classification: Thresholds and Grid: Stage 2 classifies candidates as UNION, JOIN, or NONE from matched-column statistics using five thresholds calibrated per Stage-1 and column-model pair.The dev objective is three-way macro-F1 because NONE comprises approximately 98% of pairs and accuracy would collapse toward the majority class.
  • M.3 Stage-3 Row Matching: CSLS and Profiles: Stage 3 selects the highest-confidence UNION and JOIN candidates with class-conditional scores before applying CSLS and reciprocal row matching.UNION confidence is min(r, µ), JOIN confidence is m, and ties use Stage-1 retrieval rank.
  • M.3 Stage-3 Row Matching: CSLS and Profiles: Fixed asymmetric profiles govern union and join row matching because false union appends are costlier than missed joins.The profiles remain constant across row encoders, so Stage-3 variation reflects row-embedding geometry rather than operator calibration.
  • M.4 Stage-3 Second-Pass Join on Union-Appended Rows: A second reciprocal join pass matches union-appended rows against remaining join-side rows and fills newly matched cells using the Stage-2 key-pair alignment.The second-pass match count is logged separately.
  • M.5 What is and is not tuned: Stages 1 and 3 are fixed, while Stage 2 performs 80 per-pair dev calibrations without using test labels, end-to-end scores, or Stage-3 row-model choices.This isolates Stage-2 classification quality conditional on the retrieval geometry.
  • N DLTE Detailed Rankings: Dev and test pipeline rankings are highly stable: Spearman ρ(dev, test) = 0.96 across 1,120 pipelines, and their top-50 sets share 42 pipelines.Kendall τ = 0.84, with p ≪10^-100.
  • N.1 Cell F1 as a Complementary Diagnostic: For complementary Cell F1 analysis, STARMIE leads Stage 1 at 0.601, TABERT leads Stage 2 at 0.628, and the best pipeline is STARMIE/TABERT/TABTRANSFORMER at 0.679.All top-20 Cell F1 pipelines use TABERT at Stage 2, while Stage 3 concentrates on TABTRANSFORMER, SUBTAB, SAINT, and TABBIE.

N.3 Per-Stage Marginal Analysis … N.8 Source Split: TabFact vs. WTQ

TRL-DLTE performance depends most strongly on capability-matched stage composition: column-model choice has the largest average downstream effect, while row-model differences emerge clearly when upstream errors are removed. Across source splits, hybrid pipelines retain their advantage and pipeline rankings remain highly consistent, indicating robust but non-monotonic compositional behavior.

  • N.3 Per-Stage Marginal Analysis: Stage 2 has the widest downstream spans, with UJ-H spanning 0.060 and Cell F1 spanning 0.084.These spans indicate that column-model choice has the largest average effect among the three stages under the current pipeline.
  • N.3 Per-Stage Marginal Analysis: Per-stage marginal leaders do not reliably assemble the best pipeline: STARMIE/TABBIE/TRANSTAB scores 0.134 UJ-H, versus 0.253 for test rank-1 STARMIE/GTE/GTE.The dev-selected TUTA/GTE/GTE pipeline scores 0.229, showing that marginal main effects carry signal but do not determine compositional performance.
  • N.4 Oracle-RA Row-Model Diagnostic: Oracle-RA expands the row-model UJ-H spread to 0.546, compared with only 0.013 for marginal end-to-end performance.Bypassing retrieval and alignment exposes row-matching quality that upstream errors otherwise mask.
  • N.4 Oracle-RA Row-Model Diagnostic: Identity-resolving row models and union-dedup specialists form distinct groups, separating balanced union/join recovery from near-perfect union recall with near-zero join recall.The taxonomy is stable across noise tiers, where identity-resolving models occupy the top three positions at every corruption level.
  • N.5 Full Per-Stage Model Rankings: Stage 1 retrieval recall is non-monotonic with downstream contribution: GTE has recall@100 of 0.801, whereas STARMIE has recall@100 of 0.740 but leads Cell F1 at 0.601.Downstream contribution also depends on candidate-set structure and compatibility with later alignment and row-matching models.
  • N.6 Stage 3 Row Matching: Full Pipeline Rankings: Across full Stage 3 pipeline rankings, TaBERT leads Stage 2 regardless of Stage 1 choice, while Stage 3 variation remains narrow within a fixed pair, spanning 0.589–0.679 for Starmie →TaBERT.The global average Cell F1 by Stage 3 model is nearly flat at 0.566–0.591.
  • N.7 Top-20 Pipeline Combinations: Cell F1 and UJ-H rank different top-20 pipeline sets: all Cell F1 leaders use TaBERT, while UJ-H leaders are dominated by GTE, BERT, and TURL at Stage 2, with zero overlap between lists.The metrics emphasize pooled cell-recovery yield versus balanced recovery of both removed blocks.
  • N.8 Source Split: TabFact vs. WTQ: +0.086 UJ-H on TabFact and +0.099 on WTQ separate the dev-selected best hybrid from the best monolith, while pipeline rankings correlate across sources at Spearman ρ = 0.871.Stage leaders are largely source-invariant: STARMIE leads Stage 1, Stage 3 is TRANSTAB-led, and Stage 2 only swaps TABBIE and TURL at rank 1.

O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding … P.2 Task and Dataset Summary

The proprietary embedding ablation separates retrieval-oriented strengths from structure- and task-specialized performance across table and row tasks. The merged robustness sections additionally define auxiliary diagnostics and a unified table-first aggregation protocol, including strict changed-only perturbation scoring.

  • O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding: ADA-002 reaches 0.540 MRR and proprietary variants occupy ranks 1–3 on Table Retrieval, outperforming GTE at 0.476.The result is attributed to large-scale semantic retrieval training transferring to table-to-query matching.
  • O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding: On Table QA, proprietary variants rank 5–7 of 11, behind TURL at 0.277, TABBIE at 0.276, and STARMIE at 0.266.The passage links this gap to specialized pretraining that encodes table structure.
  • O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding: TE3-LARGE achieves 0.426 F1 on Record Linkage, placing proprietary models in the top three and above GTE at 0.403.Record Linkage is characterized as an entity-matching task driven by text similarity.
  • O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding: TE3-SMALL reaches 0.801 AUROC on Row Prediction at rank 2, trailing TABICL at 0.816.TABICL is described as meta-pretrained, with in-context conditioning that adapts to each task.
  • O Proprietary Embedding Ablation: Retrieval vs. Structural Grounding: The proprietary OpenAI ablation is a controlled scaling study within the embedding-only regime, not a proxy for evaluating 7B–70B generative systems.This scope limitation distinguishes the ablation from the broader benchmark rationale discussed in Appendix B.
  • P Robustness: Three auxiliary Observatory diagnostics—Sample Fidelity, Perturbation Robustness, and Row/Column Order Insignificance—are applied to the model pool but are excluded from the 16 benchmark tasks.Their definitions, datasets, and headline metrics follow the Observatory originals, with numerical results recomputed from finalized embeddings.
  • P.1 Unified Aggregation Protocol: The diagnostics use table-first aggregation, reporting table_cosine_similarity and table_mcv as dataset-level means and standard deviations over valid tables.Sample Fidelity combines cosine mean ± std and MCV mean ± std per table, while Row/Column Order Insignificance reports the metrics separately.
  • P.2 Task and Dataset Summary: Perturbation Robustness uses a strict changed-only protocol: only modified columns contribute to headline cosine, and tables without changed columns are excluded.The appendix overview table summarizes the three diagnostics and their evaluation scope.

P.3 Sample Fidelity … P.7 Implementation Notes and Caveats

Across robustness diagnostics, tabbie is the strongest balanced model, while tabert, starmie, and tabsketchfm show distinct capability-specific strengths. Results use unified table-first aggregation and changed-only perturbation scoring, so absolute values require protocol-aware interpretation.

  • P.3 Sample Fidelity: Sample fidelity improves for all eight models as the retained row fraction rises from 0.25 to 0.75, with tabert most stable at every ratio.tabbie is the most balanced non-tabert model, while tabsketchfm degrades substantially when only 25% of rows are retained.
  • P.4 Perturbation Robustness: Content equivalence is the hardest perturbation, and tabbie is strongest overall while remaining most robust on that setting.starmie is nearly invariant to schema-level changes but drops more than tabbie under content-level perturbations.
  • P.5 Row/Column Order Insignificance: Many models are almost perfectly stable on row/row, whereas row/column is often harder because row semantics must survive column-order changes.Among shared column/table models, tabbie is the most balanced, while turl, tabert, and tabsketchfm also show strong order robustness.
  • P.6 Cross-Task Comparison of Shared Models: tabbie is the strongest all-round choice across sampling, semantic perturbation, and structural reordering among the eight models shared across evaluations.tabert is best for sample fidelity, starmie is especially strong for schema-level perturbations, and tabsketchfm is highly order-invariant but weaker elsewhere.
  • P.7 Implementation Notes and Caveats: The headline results are recomputed from finalized embeddings using a unified table-first protocol rather than copied from older logs.Each task aggregates to per-table scores before computing dataset-level means and standard deviations.
  • P.7 Implementation Notes and Caveats: Perturbation Robustness uses changed-only headline scoring and does not report MCV, while MCV variant counts differ between sample-fidelity and order-insignificance evaluations.These choices follow the original Observatory perturbation protocol and affect interpretation of absolute numbers and reproducibility.

Q Computational Efficiency

Embedding-generation efficiency varies substantially by granularity, model, and training regime. Frozen inference is generally faster and scales more predictably, but several models have important memory, feature-count, or table-width limits.

  • Column and table embedding cost: 20× separates table-embedding costs: TAPEX takes 6.1 s versus 119.6 s for TUTA.The difference reflects TUTA’s cell-level tokenization overhead.
  • Column and table embedding cost: Generic text encoders are fastest for column embeddings, with BERT at 5.3 s and GTE at 5.7 s, while TABERT and TABSKETCHFM reach 13.2 s and 14.1 s.The resulting spread is 2.7× and is attributed to per-column tokenization overhead.
  • Row embedding cost: 101× separates the fastest row model, TABICL at 8.7 s, from the slowest, TRANSTAB at 875 s.Per-table self-supervised training dominates the wall-clock cost of target-table models; within frozen inference, TABICL and TABPFN are fastest despite a fit step.
  • Scaling behavior: Training-based row models scale super-linearly with row count, whereas frozen inference models scale approximately linearly.Feature-count scaling is more uniform across models in the sweep.
  • Support envelope: TURL runs out of memory above 208 columns or 256 features, while TABPFN rises from 18 s at 28 features to 263 s at 208 features and times out at 1,775 features.These support-envelope limits matter when selecting models for large-scale deployment.

R Reproducibility Details … V Licenses and Asset Documentation

TRL-Bench specifies reproducible data access, splits, probing, seeds, wrappers, and statistical procedures across its benchmark suites. It also documents dataset-counting alternatives, responsible-use concerns, and license-aware transformations for released assets.

  • R Reproducibility Details: OpenML, DeepMatcher, WDC Products, CTBench, TabFact, and WTQ provide the publicly accessible source datasets used by TRL-Bench.OpenML tables use listed API dataset IDs; DLTE parent tables derive from TabFact and WTQ.
  • R Reproducibility Details: DLTE fixes an 827/207/345 parent-table train/dev/test split, corresponding to an effective 60/15/25 split created before pipeline evaluation.The split is random rather than source-stratified, with roughly 72% TabFact and 28% WTQ parents expected in each split.
  • R Reproducibility Details: Supervised probes average linear and one-hidden-layer MLP heads, trained with Adam at 10^-3 for up to 100 epochs and early stopping.All supervised probe results, including record linkage, are averaged over seeds 42, 52, 62, 72, and 82.
  • S Statistical Reporting: TRL-CTBENCH rejects equal model performance overall, but only 6 of 28 Holm-corrected pairwise comparisons are significant, all involving TABSKETCHFM underperforming top models.The Friedman statistic is χ2 = 32.1 with p < 0.0001; the other listed models are not significantly different across the full task set.
  • S Statistical Reporting: TRL-RBENCH shows stronger statistical separation: classification has 26/78 significant pairs and regression has 38/78, while TABICL beats all other models on both.The corresponding Friedman statistics are χ2 = 209.4, p = 3.7 × 10^-38, and χ2 = 154.6, p = 6.5 × 10^-27.
  • T Dataset Counting Protocol: The benchmark contains 87 datasets at finest source granularity, with alternative aggregations yielding counts of 84, 77, and 28.These totals vary according to whether regional splits, WDC size variants, and paired DeepMatcher variants are grouped.
  • U Broader Impact: TRL-BENCH’s release should document provenance, usage restrictions, privacy, and licensing because linkage and retrieval can enable surveillance, re-identification, or inappropriate dataset fusion.Derived assets include curated targets, rewritten row-pair tasks, split manifests, and DLTE fragments, with transformations documented under original licenses.
Loading 2606.09323v1…