Valentine: Evaluating Matching Techniques for Dataset Discovery

• Dataset discovery uses schema matching to identify joinable or unionable columns and tables, so matcher quality affects discovery performance.Source: §I · p. 1
• Valentine adapts existing algorithms and proposes evaluation metrics for ranked matches, which better fit discovery than traditional one-to-one matching.Source: §I · p. 1
• The paper defines four discovery-relatedness scenarios: two joinability scenarios and two unionability scenarios.Source: §I · p. 2
• Valentine extends existing methods to fabricate dataset pairs in a principled way for those scenarios.Source: §I · p. 2
• The suite implements six schema matching algorithms plus a baseline, and is designed as an extensible, open-source replacement for discovery matchers.Source: §I · p. 2
• The evaluation covers about 75K experiments: 553 dataset pairs across 135 configurations and multiple matching methods.Source: §I · p. 2

A. Dataset Discovery Methods

• Dataset discovery methods search for related tables to join or augment, and for joinable or unionable tables serving varied downstream goals.Source: A. Dataset Discovery Methods · p. 2

B. The Schema Matching Component

• Discovery methods select among six matcher types, including attribute, value, semantic, data-type, distribution, and embedding overlap.Source: B. The Schema Matching Component · p. 2
• Valentine covers the matcher types used in discovery with six state-of-the-art techniques plus a baseline; its methods can cover multiple matcher types.Source: B. The Schema Matching Component · p. 3
• Valentine offers methods suited to the information available to a discovery system and can spare researchers from building matchers or searching the schema-matching literature.Source: B. The Schema Matching Component · p. 3

• Valentine applies matching methods to denormalized dataset pairs with known schema information and values, producing ranked column-pair matches by confidence.Source: C. Evaluating Matching Techniques for… · p. 3
• Figure 1 shows the experiment process: fabricate dataset pairs with ground truth, create parameterized method runs, then execute all combinations of methods, parameters, and pairs.Source: Figure 1 · p. 3
• Recall@ground truth measures relevant matches among the top-k results and assesses how well a method ranks correct results for users examining a limited list.Source: C. Evaluating Matching Techniques for… · p. 3
• Traditional precision, recall, and F-measure are excluded because they evaluate unranked one-to-one matches rather than ranked outputs.Source: C. Evaluating Matching Techniques for… · p. 3

III. DATASET RELATEDNESS SCENARIOS

• The taxonomy refines unionability and joinability into four scenarios to cover tabular dataset discovery methods.The scenarios guide evaluation because different matching approaches may handle different problem cases better.Source: §III · p. 3

A. Unionable Relations

• Unionable relations have the same arity and a one-to-one mapping between semantically equivalent attributes, even when corresponding attributes have different but similar data types.Their instances may differ substantially, making the match difficult despite the possibility of a union.Source: A. Unionable Relations · p. 4
• View-unionable relations share a semantically corresponding subset of attributes but may also contain attributes unique to each relation.When the views cover all attributes, this reduces to the unionable case.Source: A. Unionable Relations · p. 4
• Unionable-relation discovery supports finding similar-entity tables and finding additional rows to augment a table.The paper connects these uses to dataset discovery methods, and table augmentation methods,,.Source: A. Unionable Relations · p. 4

B. Joinable Relations

• Relations are joinable when at least one semantically equivalent attribute pair has overlapping instances and joining on it does not produce a Cartesian product.The paper frames joinability as a way to connect complementary data about the same conceptual entity type.Source: B. Joinable Relations · p. 4
• Different formatting conventions can prevent instance overlap from revealing a joinable relationship across diverse data sources.In these cases, identifying semantic equivalence between attribute values is needed.Source: B. Joinable Relations · p. 4
• Semantically-joinable relations share semantically equivalent instances on at least one semantically equivalent attribute pair, while the join result is not a Cartesian product.Semantic joins are described as a superset of fuzzy joins, which exploit string-based similarities.Source: B. Joinable Relations · p. 5
• Joinability matters for dataset discovery methods that augment tables with attributes or seek extra features for machine-learning datasets.The paper links these uses to methods,, and, [11].Source: B. Joinable Relations · p. 5

IV. FABRICATING DATASET PAIRS

• Valentine fabricates dataset pairs by splitting existing tables and perturbing schema and instance information, with the original data serving as ground truth.The approach extends eTuner; the paper reports that eTuner-based automatic tuning led to better effectiveness on real-world datasets than manual tuning.Source: §IV · p. 5
• Fabrication adds keyboard-proximity typos to strings, distribution-based changes to numeric values, and schema noise through prefixes, abbreviations, or dropped vowels.Column names may be verbatim or noisy, and instances may be verbatim or noisy depending on the scenario.Source: §IV · p. 5
• Unionable pairs use horizontal splits with varying row overlap; view-unionable pairs use horizontal and vertical splits with zero row overlap and varying column overlap.The lack of row overlap in view-unionable pairs adds a challenge for naive instance-based algorithms.Source: §IV · p. 5
• Joinable pairs use vertical splits with overlapping columns and verbatim instances, while semantically-joinable pairs perturb overlapping instances so equality joins no longer recover the original table.The semantically-joinable variants include noisy instances and may have either noisy or verbatim schemata.Source: §IV · p. 6

V. DATASETS

• The evaluation dataset sources comprise 540 fabricated pairs and a second category of real-world pairs with manually created ground truth.The fabricated pairs use original tables as ground truth; the real-world pairs were curated to create ground truth manually.Source: §V · p. 6

A. Dataset Sources of Fabricated Dataset Pairs

• TPC-DI, Open Data, and ChEMBL each contribute 180 fabricated pairs, for 540 pairs across the three sources.The fabricated datasets vary in table dimensions: TPC-DI has 11–22 columns, Open Data 26–51, and ChEMBL 12–23.Source: A. Dataset Sources of Fabricated… · p. 6

B. Dataset Sources of Human-curated Dataset Pairs

• The human-curated sources include 4 WikiData pairs, 7 Magellan pairs, and 2 proprietary ING pairs.The WikiData tables vary names and values; the ING tables include difficult matching cases, including denormalized tables with nested or composite values.Source: B. Dataset Sources of Human-curated… · p. 6
• Ground truth for the human-curated datasets was created manually with an expert database administrator, but the dataset cannot be made public due to privacy constraints.The passage refers to the ground truth for both datasets and the non-public status of the dataset.Source: B. Dataset Sources of Human-curated… · p. 6

VI. MATCHING METHODS

• The matching methods are categorized as schema-based, instance-based, or hybrid, depending on whether they use schema information, data instances, or both.Schema-based methods use items such as attribute names, data types, and context; instance-based methods may compare value distributions or syntactic similarity.Source: §VI · pp. 6–7

A. Methods Description

• Cupid combines linguistic name similarity with structural context from schema trees, with WordNet used as a thesaurus in Valentine’s implementation.Because Cupid is not openly available, the implementation uses its name-similarity formula for data compatibility scores.Source: A. Methods Description · p. 7
• Similarity Flooding propagates similarity between connected schema elements over a merged graph until scores converge.Valentine implemented the original method from scratch, using Levenshtein distance because the original string-similarity function was unspecified.Source: A. Methods Description · p. 7
• COMA combines schema-based matchers and supports user feedback, while Valentine uses the Community Edition’s default schema- and instance-based strategies.COMA represents schemata as rooted directed acyclic graphs; its extension incorporates two instance-based matchers.Source: A. Methods Description · p. 7
• Distribution-based matching clusters attributes using Earth Mover’s Distance between column-value distributions.Valentine implemented the unavailable original method, substituting PuLP for IBM CPLEX in its final integer-programming step.Source: A. Methods Description · p. 7
• SemProp links names to ontology classes using pre-trained embeddings, then follows those links to relate elements and forwards unmatched pairs to a syntactic matcher.Valentine uses the Aurum implementation and its domain-specific ontology on ChEMBL datasets.Source: A. Methods Description · p. 7
• EmbDI learns relational embeddings for values and attribute names, then compares column embeddings to find relationships.Valentine imported the accompanying code; the framework can also use external knowledge such as synonym dictionaries or pre-trained embeddings.Source: A. Methods Description · p. 7

B. Method Parameterization

• For methods without available default parameters, Valentine uses grid search; other methods use parameter settings reported by their authors.Cupid, Distribution-based matching, SemProp, and the Jaccard-Levenshtein baseline are grid-searched; the parameter combinations define separate experiments in Table II.Source: B. Method Parameterization · pp. 7–8, Table II · p. 8

C. Sensitivity to Parameter Changes under Grid Search

• The sensitivity analysis varies one parameter at a time across all 180 ChEMBL dataset pairs and measures standard deviation in recall at ground truth.Table III reports the minimum, median, and maximum standard deviation for parameters with at least three tested values.Source: C. Sensitivity to Parameter Changes… · p. 8, Table III · p. 8
• Maximum effectiveness standard deviation is close to 0.5 for most parameters, although minimum and median deviations are close to zero.The largest sensitivities occur with few shared values or noisy data and low thresholds; a stricter threshold can drastically improve results.Source: C. Sensitivity to Parameter Changes… · p. 8

VII. FINDINGS

• The benchmark summarizes recall-at-ground-truth ranges across fabricated dataset pairs and average execution time across matching methods.The experiments cover TPC-DI, Open Data, and ChEMBL dataset pairs.Source: §VII · p. 8

1) Schema-based Methods:

• Under noisy attribute names, Similarity Flooding and COMA outperform Cupid, but their median recall at ground truth is close to 0.6.Figure 4 covers schema-based methods across dataset relatedness scenarios; the remaining schema structure and column neighborhoods provide no useful insights when names are poor.Source: 1) Schema-based Methods · p. 8, Figure 4 · p. 9

2) Instance-based Methods:

• View-unionable datasets are harder than unionable ones for instance-based matchers, and semantically-joinable datasets fare worse than joinable ones.Figure 5 reports instance-based effectiveness across relatedness scenarios; value-set dissimilarity and absent row overlap help explain the lower recall, while results are highly skewed outside joinable scenarios.Source: Figure 5 · p. 9, 2) Instance-based Methods · p. 9

3) Evaluation of Hybrid Methods:

• EmbDI is more effective than SemProp, but both show low or inconsistent recall; EmbDI performs worst among schema- and instance-based methods overall.The paper attributes EmbDI’s difficulty to randomness in training-data generation and reliance on overlapping values, especially when overlap is scarce or external knowledge absent.Source: 3) Evaluation of Hybrid Methods · p. 9

4) Expected Results:

• Expected results vary with the data: instance-based methods improve without noisy instances, especially for joinable pairs, while EmbDI is acceptable in joinable scenarios.Schema-based methods place all correct matches at the top with verbatim schemata.Source: 4) Expected Results · p. 9

1) WikiData:

• On WikiData, instance-based methods find all relevant matches at top ranks (recall at ground truth=1) for joinable relations with high value overlap.COMA’s instance-based approach also returns every correct match in the first ranked positions for noisy semantically-joinable relations; schema-based methods miss some matches.Source: Figure 7 · p. 10, 1) WikiData · p. 10
• For WikiData unionable relations, all four instance-based methods have better recall at ground truth than schema-based methods.For view-unionable relations, instance-based methods show similar behavior, except distribution-based matching performs poorly when fabricated columns have differing distributions.Source: 1) WikiData · pp. 9–10, Figure 7 · p. 10

2) Magellan Data:

• Magellan pairs were less informative than fabricated pairs, offering potentially misleading method-category comparisons while omitting important relatedness scenarios.The authors say the Magellan data do not cover all scenarios relevant to dataset discovery.Source: 2) Magellan Data · p. 10

3) ING Data:

• For ING#1, almost all methods found around 70% of expected matches, while Similarity Flooding placed many false positives among top ranks.Distribution-based performed best; matching columns’ almost-identical values produced similar distributions.Source: 3) ING Data · p. 10

2) Magellan Data:

• On Magellan pairs, all instance-informed methods except COMA were less effective than Valentine’s schema-based approaches, which exploit shared attribute names.The reported results were less informative than those from fabricated pairs and did not cover all important relatedness scenarios.Source: Table IV · p. 10, 2) Magellan Data · p. 10

3) ING Data:

• On ING#1, Jaccard-Levenshtein found most matches but ranked false positives highly because it compares value sets rather than distributions.The method could find most matches, but some false positives ranked high.Source: 3) ING Data · p. 11
• For ING#2, Distribution-based performed far better; COMA missed many multi-matches (authors suspect a v3.0 bug), while EmbDI’s local embeddings poorly captured matching-column relationships.The ING#2 ground truth linked each small-table column to many columns in a 60-column table.Source: 3) ING Data · p. 11

C. Efficiency Results

• Schema-based methods were by far most efficient; instance-based methods ran several orders slower, with EmbDI worst overall and its random-walk generation and embedding training costly.Distribution-based and COMA were the most efficient instance-based methods, though both could still take a long time due to value processing.Source: C. Efficiency Results · p. 11

VIII. RELATED BENCHMARKS AND FRAMEWORKS

• Unlike OAEI’s ontology focus and STBenchmark’s schema-mapping focus, XBenchMatch covers XML only; Valentine targets tabular matching for dataset discovery.The authors describe Valentine as the first study devoted to dataset-discovery literature.Source: §VIII · p. 11

IX. CONCLUSION & LESSONS LEARNED

• No method consistently won: COMA led on most fabricated pairs, Distribution-based best fit real-world ING, supporting composition of matching methods in integration pipelines.The authors recommend following COMA’s approach of composing state-of-the-art matchers, including newer embedding-based approaches.Source: §IX · p. 11
• SemProp’s pretrained embeddings had low effectiveness alone; EmbDI’s local embeddings could improve effectiveness but usually lagged other state-of-the-art methods.The authors say further research is needed to make embedding-based techniques effective.Source: §IX · pp. 11–12
• Most methods require complex, data-dependent parameterization, and grid-search results are near-optimal because tuning used ground truth; the authors expect lower performance in the wild.They argue for self-driving matchers that need less parameterization.Source: §IX · p. 12
• Ca. 70 lines of Python code produced a simple Jaccard-Levenshtein baseline that worked surprisingly well.The authors argue that simple baselines can support future comparative analysis.Source: §IX · p. 12
• The authors favor ranked candidate matches and human feedback through positive or negative examples over parameters or thresholds, framing matching as search rather than optimization.They envision matchers working alongside people who give feedback on the matching process.Source: §IX · p. 12
• Instance-based methods remain expensive when comparing large value sets, motivating approximate methods for better scaling.The authors note that finding matches can be costly when similarity metrics must be calculated over large sets.Source: §IX · p. 12