PaperlayerView exact revision on arXiv

Source-linked AI summary

TabJoinBench: A Benchmark for Joinable Table Discovery

Sandipan De, Jin Wang, Vivek Gupta

arXiv:2610.00817v1cs.DBcs.AIcs.CLcs.IR

TL;DR

Join discovery methods are difficult to compare reproducibly because evaluations often use method-specific preprocessing, and existing benchmarks do not fully represent semantic diversity or verify shared columns as valid join keys. TABJOINBENCH combines semantic, relational, and hybrid scenarios with validated pairs and composable perturbations. Its evaluation finds that no single approach consistently dominates across settings, and the benchmark provides a reproducible foundation for future evaluation.

  • Problem

    Method-specific preprocessing and benchmarks centered on closely related equi-joins limit reproducible, fair evaluation and do not capture practical semantic diversity or verify shared columns as valid join keys.

    Source: §1 · pp. 1–2

  • Method

    TABJOINBENCH combines source-specific query–candidate pairs across semantic, relational, and data-lake scenarios, then applies composable structural, representation, and semantic perturbations while preserving ground truth.

    Source: §1 · p. 2, §3.1 · p. 2, §6 · p. 8

  • Results

    No single join discovery approach consistently dominates across benchmark settings, and perturbation impact depends on both method-exploited table properties and data source.

    Source: §6 · p. 8, §4.2 · p. 8

  • Takeaways & Limitations

    TABJOINBENCH provides a reproducible foundation for evaluating and developing join discovery methods across diverse settings.

    Source: §6 · p. 8

    Manual verification lacked inter-annotator agreement; the ground truth is not exhaustive, and the perturbation framework cannot capture the full diversity of realistic transformations.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

Join discovery aims to identify tables from large data repositories that can augment a query table with complementary information, enabling downstream tasks such as data exploration, feature engineering, and business intelligence. Although numerous join discovery methods have been proposed, existing studies rely on method-specific benchmark construction, making reproducible and fair comparison difficult. We present TabJoinBench, a benchmark for evaluating join discovery methods across semantic, relational, and hybrid data lake scenarios. TabJoinBench constructs query-candidate pairs using source-specific validation strategies, systematically introduces structural, representation, and semantic changes through composable perturbations while preserving reliable ground truth. We evaluate representative join discovery methods spanning set-based, feature-based, and learned approaches, together with general-purpose language-model embedding baselines, and publicly release the processed datasets, ground-truth annotations, and generation pipeline to facilitate reproducible evaluation and future research.

1 Introduction

Data lakes hold heterogeneous tables with relationships that can be difficult to discover, while method-specific preprocessing and limited benchmark designs impede reproducible and fair evaluation. TABJOINBENCH addresses these gaps with three join-discovery scenarios, validated query–candidate pairs, composable perturbations, and released benchmark resources.

1 Introduction

  • Data lakes contain complementary tables that can be joined, but heterogeneous schemas, value representations, and semantics make join discovery challenging.The paper focuses on finding datasets meaningfully joinable to a target through one or more shared attributes.Source: §1 · p. 1
  • Prior join discovery evaluations are fragmented: method-specific preprocessing and unreleased processed data, queries, or ground truths hinder reproduction and fair comparison.Recent benchmarks improve reproducibility but largely test equi-joins between closely related tables and assume shared columns are semantically valid keys.Source: §1 · pp. 1–2
  • TABJOINBENCH covers three scenarios using Wikidata-linked WikiTables, MMQA SQL-derived relational joins, and key-aware partitions of NYC Open Data.Together, these sources represent web tables, relational databases, and data lakes.Source: §1 · p. 2
  • Its construction starts from validated seed pairs and applies composable perturbations to create realistic data-lake variation while preserving reliable ground truth.The perturbations systematically transform the source pairs.Source: §1 · p. 2
  • The paper releases datasets, queries, ground-truth annotations, and the generation pipeline, and evaluates set-based, feature-based, learned, and embedding-based methods.The evaluation examines robustness under different join semantics and perturbations.Source: §1 · p. 2

2 Preliminary: Join Discovery

Join discovery ranks data-lake columns for a query column using a method-dependent relevance function. Ground-truth joinable columns should rank above non-joinable columns, while relevance can reflect different evidence depending on the join type.

2 Preliminary: Join Discovery

  • A join discovery method assigns relevance scores to data-lake columns with respect to a query column, ranking ground-truth joinable columns above non-joinable ones.The ground-truth set comprises the columns considered joinable to the query column.Source: §2 · p. 2
  • The relevance function may use value overlap or similarity, schema information, statistical properties, learned representations, or table context.The method determines which characteristics contribute to the score.Source: §2 · p. 2
  • Joinability evidence varies by relationship: equi-joins use exact or near-exact value correspondence, fuzzy joins allow syntactic variation, and semantic joins match underlying concepts despite limited lexical overlap.Source: §2 · p. 2

3 TABJOINBENCH

TABJOINBENCH combines validated join pairs, controlled perturbations, and a final benchmark with query tables, a heterogeneous data lake, and ground-truth annotations. Its design spans multiple join semantics and complementary data sources.

3.1 Overview

  • TABJOINBENCH supports equi-, fuzzy, and semantic joins, using meaningful query-candidate pairs and controlled, composable perturbations to model data-lake heterogeneity.The construction proceeds from validated seed pairs through perturbation to the final benchmark.Source: §3.1 · p. 2

3.2 Data Sources and Benchmark Families

  • WikiTables supplies entity-linked tables for semantic joins, MMQA supplies SQL-derived equi-joins, and NYC Open Data supports both identifier-based and textual joins in larger data-lake datasets.WikiTables hyperlinks map table cells to Wikidata entities, enabling semantic relationships beyond lexical or value overlap.Source: §3.2 · p. 3

3.3 Seed Generation and Validation

  • WikiTables pairs require at least five shared entities, exclude rare and overly common entities, and are manually classified after heuristic prioritization; 543 of 983 inspected pairs were retained as meaningful joins.The prioritization combines entity overlap, table-context and column similarity, and schema complementarity.Source: §3.3 · p. 4
  • MMQA seed pairs are derived by parsing ground-truth SQL join predicates, with duplicate pairs removed to yield 574 unique query-candidate pairs.Source: §3.3 · p. 4
  • NYC Open Data seed pairs come from splitting filtered datasets into two coherent column groups with one shared column; 70 of 82 manually checked partitions were retained.The 620 filtered datasets were partitioned using LLM-generated groups, screened by embedding-based cohesion against random partitions, and selected for verification when their z-score exceeded 1.96.Source: §3.3 · pp. 4–5

3.4 Perturbation Framework

  • Composable structural, representation, and semantic perturbations alter candidate tables while preserving join relationships, then add original and perturbed candidates to the lake.Examples include reordering or partitioning tables, changing value formats, and anonymizing or substituting column names.Source: §3.4 · pp. 5–6

4 Experiments

The experiments compare join-discovery methods across three benchmark subsets and test their robustness to perturbations. Results vary by data source and method, with no single approach dominating every setting.

4.1 Experimental Setup

  • The evaluation covers sketch-, embedding-, hybrid feature-, profile-, and learned-representation methods, plus language-model embedding baselines, using Precision@k, Recall@k, and Precision@R.Source: §4.1 · p. 6

4.2 Results and Analysis

  • On WikiTables, LSH Ensemble has the highest precision at each retrieval budget, while DeepJoin has the highest P@R.Source: §4.2 · p. 6
  • DeepJoin leads MMQA across every metric, while all methods perform substantially worse there than on the other subsets.MMQA’s numeric identifier joins offer limited semantic context and create substantial candidate-column overlap.Source: §4.2 · p. 6
  • NYC Open Data has no consistent overall leader: LSH Ensemble leads precision at budgets 5, 15, and 30, DeepJoin leads P@1, and FREYJA leads R@30 and P@R.D3L is competitive on this subset, whereas PEXESO is weakest.Source: §4.2 · pp. 6–7
  • Language-model embedding baselines perform competitively across all three subsets despite requiring no join-specific training.Source: §4.2 · p. 7
  • Perturbation effects differ by subset: representation changes degrade value-based methods on WikiTables and MMQA, while semantic changes are especially disruptive on WikiTables.On NYC, structural changes affect value-overlap, profile, and sketch methods; semantic changes degrade DeepJoin and language-model embeddings more than structural changes.Source: §4.2 · pp. 7–8
  • D3L and WarpGate are exceptions to the broader NYC pattern, retaining the Structural > Representation > Semantic ordering observed across the subsets.The proposed explanations are D3L’s additional column and distribution signals and WarpGate’s learned sensitivity to table organization.Source: §4.2 · p. 8
  • No single method or perturbation category dominates across settings; impact depends on both method properties and the underlying data source.Source: §4.2 · p. 8

5 Related Work

Prior work has concentrated more on join-discovery retrieval methods than on benchmark construction. TABJOINBENCH differs from the closest benchmark and general table corpora in its focus on retrieval evaluation across join settings.

  • Join-discovery research has focused largely on retrieval methods, with comparatively less attention to benchmark construction.LakeBench addresses scalable evaluation, while TABJOINBENCH spans semantic, relational, and hybrid join settings; MMTU treats joins as one of several table-understanding tasks rather than retrieval.Source: §5 · p. 8

6 Conclusion

TABJOINBENCH evaluates join-discovery methods across semantic, relational, and hybrid settings using controlled perturbations while preserving reliable ground truth. The evaluation finds that no single approach consistently dominates across all benchmark settings.

  • TABJOINBENCH evaluates join-discovery methods across semantic, relational, and hybrid settings.Its construction uses controlled structural, representation, and semantic perturbations while preserving reliable ground truth.Source: §6 · p. 8
  • No single evaluated approach consistently dominates across all benchmark settings.The authors present the benchmark as a reproducible foundation for developing and evaluating future join-discovery methods.Source: §6 · p. 8

7 Limitations

TABJOINBENCH’s construction has limits in manual verification, ground-truth completeness, and how fully its perturbations represent real data transformations.

  • Manual verification covered the WikiTables and NYC Open Data subsets, was performed by the primary author, and did not permit measuring inter-annotator agreement.Source: §7 · p. 8
  • The benchmark records verified positive joins but does not exhaustively enumerate valid joins, so additional relationships may be absent from its ground truth.Source: §7 · p. 9
  • The perturbation framework cannot capture the full diversity of real transformations; predefined mappings and sampled LLM-assisted augmentations may leave lexical or representation overlap.Source: §7 · p. 9

8 Ethical Considerations

The paper describes the benchmark’s source-data and AI-use practices, and explains how WikiTables pairs were labeled and selected as seed pairs.

8 Ethical Considerations

  • TABJOINBENCH uses publicly available datasets, adds no new personally identifiable information beyond its sources, and its perturbed data are not authoritative representations of the originals.Source: §8 · p. 9
  • Source-dataset biases may be reflected; LLM-assisted segmentation exposed file metadata, not data, to the LLM.Source: §8 · p. 9

A.1 WikiTables Annotation

  • WikiTables pairs received one of four labels, and only Joinable pairs were retained as seeds; ambiguous relationships were set aside as Undetermined.Source: §A.1 · p. 10
  • Annotation guidelines favored complementary joins based on meaningful entities and treated generic-key-only, incompatible, unionable, or duplicate pairs as non-Joinable or Undetermined.Source: §A.1 · p. 10

A.2 NYC Open Data Validation

NYC Open Data validation checked the quality of generated partitions, while the partitioning prompt specified entity-based normalization; Table 5 reports perturbation composition.

A.2 NYC Open Data Validation

  • Because NYC query and candidate tables come from the same source dataset, manual validation checked partition quality rather than whether a join existed.Validated partitions were retained only when their column groups represented coherent entities, shared a meaningful join attribute, and formed meaningful complementary tables.Source: §A.2 · pp. 10–11

B NYC Open Data Partitioning Prompt

  • The partitioning prompt asks for two normalized entity groups sharing exactly one column, preferring descriptive natural-key names over codes when available.It also directs the model to minimize redundancy and maximize functional dependency within each group.Source: B NYC Open Data Partitioning Prompt · p. 11

C Composition of Perturbed Segments

D Extended Perturbation Results

The extended perturbation results include retrieval-performance reporting for WikiTables and specify the retrieval cutoffs used across perturbation tables.

D Extended Perturbation Results

E Method Parameters

Methods were evaluated with their original implementations’ default hyperparameters unless otherwise specified; the section lists selected method-specific settings.

PEXESO

  • PEXESO used GloVe Wiki Gigaword 50 embeddings, with indexing parameters differing between WikiTables and the MMQA and NYC Open Data subsets.WikiTables used pivot=3, base=2, level=4; MMQA and NYC Open Data used pivot=5, base=2, level=6; both configurations used τ=0.6 and T=0.4.Source: E Method Parameters · p. 12

LSH Ensemble

D3L

  • D3L used a 0.5 similarity threshold, hash sizes of 256 for name, format, and value indices and 1024 for embedding and distribution indices, and q-gram size 3.Source: E Method Parameters · p. 12

WarpGate and Language Model Embeddings

  • WarpGate used the tabert_large_k3 table embedder; language-model column embeddings used HNSW with cosine similarity, efconstruction=100, and M=32.Embedding dimension depended on the selected embedding model.Source: E Method Parameters · p. 12

DeepJoin

  • DeepJoin was trained with MultipleNegativesRankingLoss, batch size 32, four epochs, a 10% validation split, 10,000 warm-up steps, and weight decay 0.01.Source: E Method Parameters · p. 12

Related papers

Loading 2610.00817v1…