PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Entity Matching with Pre-Trained Language Models

Yuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan, Wang-Chiew Tan

arXiv:2004.00584v3cs.DBcs.CL

TL;DR

Entity matching must determine whether two data entries refer to the same real-world entity, despite costly candidate comparison and demanding language understanding. Ditto fine-tunes pre-trained Transformer language models as a simple sequence-pair classifier and adds domain knowledge, summarization, and difficult-example augmentation. Across benchmark datasets, it outperforms prior state-of-the-art systems and also performs strongly on a large-scale company-matching task.

  • Problem

    Entity matching remains challenging because large datasets make candidate comparison expensive and correct matching requires substantial language understanding and domain-specific knowledge.

  • Method

    Ditto fine-tunes pre-trained Transformer-based language models for sequence-pair classification and improves them with domain-knowledge injection, summarization, and data augmentation.

  • Results

    Ditto consistently outperforms existing entity-matching solutions on all three benchmark datasets, with significantly less training data.

  • Takeaways & Limitations

    Ditto’s performance is attributed to pre-trained language understanding, knowledge-guided text alignment, and data invariance learned from augmented data.

  • Takeaways & Limitations

    Ditto may require substantial labeled data and may not capture EM tasks dominated by numeric data or specialized domains when using general-English language models.

Abstract

from arXiv · show

We present Ditto, a novel entity matching system based on pre-trained Transformer-based language models. We fine-tune and cast EM as a sequence-pair classification problem to leverage such models with a simple architecture. Our experiments show that a straightforward application of language models such as BERT, DistilBERT, or RoBERTa pre-trained on large text corpora already significantly improves the matching quality and outperforms previous state-of-the-art (SOTA), by up to 29% of F1 score on benchmark datasets. We also developed three optimization techniques to further improve Ditto's matching capability. Ditto allows domain knowledge to be injected by highlighting important pieces of input information that may be of interest when making matching decisions. Ditto also summarizes strings that are too long so that only the essential information is retained and used for EM. Finally, Ditto adapts a SOTA technique on data augmentation for text to EM to augment the training data with (difficult) examples. This way, Ditto is forced to learn "harder" to improve the model's matching capability. The optimizations we developed further boost the performance of Ditto by up to 9.8%. Perhaps more surprisingly, we establish that Ditto can achieve the previous SOTA results with at most half the number of labeled data. Finally, we demonstrate Ditto's effectiveness on a real-world large-scale EM task. On matching two company datasets consisting of 789K and 412K records, Ditto achieves a high F1 score of 96.5%.

1 INTRODUCTION

Entity matching determines which entries from two datasets refer to the same real-world entity, but large-scale candidate generation and nuanced language understanding make it challenging. Ditto applies pre-trained language models with three optimizations to improve matching.

  • Entity matching identifies pairs of entries from two datasets that refer to the same real-world entity.
  • Blocking reduces large matching problems by pruning unlikely candidate pairs before matching.This lowers the number of pairs requiring detailed comparison.
  • Correctly matching candidates requires substantial language understanding and domain-specific knowledge.
  • Ditto fine-tunes pre-trained Transformer-based language models and casts entity matching as sequence-pair classification.Its optimizations inject domain knowledge, summarize long strings, and augment training data with difficult examples.
  • Ditto correctly distinguishes the three highly similar candidate pairs in the motivating product example, whereas some state-of-the-art systems cannot.It recognizes syntactic variation and can emphasize domain-relevant fields such as grade or edition.

2 BACKGROUND AND ARCHITECTURE

Ditto treats entity matching as binary sequence-pair classification by fine-tuning pre-trained language models on serialized data entries. Its serialization supports heterogeneous schemas, while long inputs remain constrained by language-model sequence limits.

  • An end-to-end entity matching system combines blocking, which finds high-recall candidates, with Ditto, which predicts whether each pair denotes the same entity.
  • Ditto fine-tunes pre-trained language models with a simple binary classification architecture for matching decisions.The tested models include BERT, RoBERTa, and the smaller DistilBERT.
  • Ditto serializes each entry as attribute-value text and serializes a candidate pair as two sequences separated by special tokens.The [CLS] representation is passed to fully connected classification layers.
  • Ditto’s serialization accepts heterogeneous schemas and does not require attributes to be aligned before matching.It can also serialize hierarchically structured entries.

3 OPTIMIZATIONS IN DITTO

Ditto improves matching through domain-knowledge injection, long-entry summarization, and training-data augmentation. These optimizations emphasize useful information, address sequence-length constraints, and expose the model to harder or perturbed examples.

  • 3.1 Leveraging Domain Knowledge: Domain knowledge is injected by emphasizing spans, normalizing equivalent strings, and typing tokens to help Ditto focus on matching-relevant information.Span typing adds signals that can help align spans with the same type and avoid mismatches such as street numbers versus years.
  • 3.2 Summarizing long entries: Long strings are summarized because Transformer language models have finite input lengths and important matching information may not occur at sequence beginnings.Ditto uses TF-IDF-based summarization to retain non-stopword tokens with high TF-IDF scores.
  • 3.3 Augmenting training data: Data augmentation creates harder training examples and teaches robustness to missing values, misplaced attributes, and missing tokens.Ditto uses span-level, attribute-level, and entry-level operators, with uniformly random sampling for the listed operators.
  • 3.3 Augmenting training data: MixDA interpolates original and augmented language-model outputs so augmented examples are less distorted than direct transformations.The model trains on LM(s′′) = λ · LM(s) + (1 − λ) · LM(augment(s,o)), where λ is sampled from a Beta distribution.

4 EXPERIMENTS

Experiments across ER-Magellan and WDC benchmarks show that Ditto improves matching quality, robustness, and label efficiency, with its optimizations contributing additional gains.

  • Main results: 31%: Ditto outperforms DeepMatcher+ in all 13 ER-Magellan cases, while the unoptimized baseline wins in 12/13 cases.The exception for the baseline is the Company dataset with long text.
  • Main results: 15.6%: Ditto’s average improvement on the 7 smallest datasets exceeds the 1.48% average improvement on the remaining datasets.
  • Main results: 0.57 versus 8.21: Ditto’s average performance degradation on four dirty datasets is lower than DeepMatcher+’s under data noise.
  • Main results: 4/5: Ditto reaches close to or better than DeepMatcher+ trained on full data when using less than 20% of the original training data.
  • Main results: 94.08: Ditto’s highest WDC F1 score using all 215k training examples exceeds the previous best by 3.92.With half the training data, Ditto outperforms DeepMatcher trained on the xLarge set by 2.89 in All.
  • Ablation study: 78.5%: The pre-trained language model accounts for most of Ditto’s improvement over DeepMatcher+ on ER-Magellan datasets excluding Company.Language modeling contributes 7.75 average F1 improvement versus 9.87 for full Ditto; summarization raises Company from 41% to over 93%.

5 CASE STUDY: EMPLOYER MATCHING

Ditto is applied to large-scale employer matching after blocking and targeted labeling, using domain cues and augmentation to handle difficult, incomplete records. It achieves the highest reported F1 while training faster than DeepMatcher.

  • Task and data: Ditto matches internal and public employer records across name, address, city, state, zipcode, and phone attributes.The task seeks, for each public record, an internal record representing the same employer.
  • Blocking: Blocking uses exact zipcode matches plus top-20 TF-IDF-similar internal records because missing zipcodes and multiple sites make simple blocking insufficient.The datasets remain nontrivial in size even after deduplication.
  • Labeling: 20,000 labeled pairs were sampled from blocking outputs, with high-similarity pairs sampled more often to create a difficult training set.The labels were split into training, validation, and test sets at a 3:1:1 ratio.
  • Ditto configuration: Domain tags identify street numbers and phone suffixes, while attr_del augmentation targets robustness to missing values.The recognizer tags the first number in addresses and the last four phone digits.
  • Results: 96.53 F1 is achieved by Ditto with all training data, outperforming DeepMatcher in F1 and training faster across training-set sizes.The comparison remains favorable even when Ditto uses MixDA.
  • Advanced blocking: Advanced blocking uses Sentence-BERT encoding and similarity search to reduce matching to the top-10 candidates per record.Sentence-BERT itself reaches only 92% F1 and therefore does not replace Ditto.

6 RELATED WORK AND DISCUSSION

Ditto extends prior entity-matching approaches by serializing heterogeneous records for pre-trained language models and adding configurable knowledge injection and data augmentation. Its discussion identifies fine-tuning cost, labeling needs, and domain or numeric-data limitations.

  • Related work: Earlier entity-matching systems use rules, crowdsourcing, machine learning, or deep learning architectures such as LSTM-based DeepER.These approaches address blocking or matching with different modeling strategies.
  • Comparison: Ditto serializes entries with structural tags, allowing records with different schemas, including hierarchical JSON, to be ingested uniformly.Other compared systems align same-arity attributes before matching.
  • Related pre-trained-LM work: Unlike a concurrent pre-trained-LM method, Ditto additionally supports domain knowledge, data augmentation, and summarization with broader benchmark analysis.The paper also provides a detailed comparison in an appendix.
  • Optimizations: Ditto modularizes domain knowledge through customizable preprocessing rules and introduces EM-specific augmentation operators with MixDA.The paper describes this as the first application of data augmentation to entity matching.
  • Limitations: Active learning is not straightforward for Ditto because interactive labeling requires response times compatible with its relatively long fine-tuning time.The authors leave applying active learning to Ditto for future development.
  • Limitations: General-English pre-training may not capture tasks with substantial numeric data or specialized domains such as science.The authors suggest specialized language models or hybrid numeric-text models as potential solutions.

7 CONCLUSION

The paper concludes that Ditto combines pre-trained Transformer language models with domain knowledge, summarization, and data augmentation for entity matching. It reports superior benchmark performance with less training data and outlines broader data-integration extensions.

  • Contribution: Ditto fine-tunes pre-trained Transformer-based language models using a simple entity-matching architecture.The conclusion presents this as the system's central design.
  • Optimizations: Ditto's optimizations inject domain knowledge, summarize text, and augment data to improve matching.The conclusion attributes performance to language understanding, knowledge-guided alignment, and invariance learned from augmented data.
  • Results: Ditto outperforms existing entity-matching solutions on all three benchmark datasets with significantly less training data.This is the paper's stated benchmark conclusion.
  • Future work: The authors plan to extend Ditto beyond entity matching to entity type detection and schema matching.The longer-term goal is a BERT-like model for tables.

A ARCHITECTURE OF THE PRE-TRAINED LANGUAGE MODELS

Ditto serializes two records into one sequence for a pre-trained language model, then uses task-specific layers to classify whether they match. The architecture retains structural tags while leveraging contextual token representations.

  • Input representation: Ditto serializes the two input entries as one sequence and feeds that sequence to BERT, DistilBERT, or RoBERTa.Figure 6 depicts this model architecture.
  • Architecture: The model combines token embeddings and Transformer layers from a pre-trained language model with linear and softmax task-specific layers.The pre-trained component supplies contextual representations for the matching task.
  • Classification: The [CLS] representation summarizes contextual information used by the task-specific layers for match classification.The architecture therefore reduces matching to a classifier over the encoded pair.

B TRAINING TIME AND PREDICTION TIME EXPERIMENTS

Ditto has training and prediction times comparable to DeepMatcher, while fp16 accelerates training and MixDA increases training cost.

  • 119 seconds to 1.7 hours: Ditto’s training time spans datasets from 450 to 113k examples.Ditto has similar training time to DeepMatcher despite using deeper Transformer-based models.
  • 2-3x slower: Ditto with MixDA trains more slowly than Ditto(DK) without MixDA because MixDA generates additional examples.
  • Less than 2%: the DK optimization adds only a small prediction-time overhead.DeepMatcher and Ditto have comparable prediction times per entry pair.
  • 5%: the DK optimization adds only a small training-time overhead.Ditto(DK) and Baseline omit MixDA and are therefore faster than full Ditto.
  • DeepMatcher ran out of memory on the Company dataset, so its training-time data point is not reported.

C BREAKDOWN OF THE DM+ RESULTS AND EXPERIMENTS

The DM+ comparison combines the strongest results from several baseline methods, while additional analyses examine serialized inputs and Ditto variants on ER-Magellan datasets.

  • DM+ baseline construction: DM+ takes the highest F1 scores from DeepER, Magellan, DeepMatcher, and DeepMatcher follow-up methods.
  • Ditto variant comparison: Figure 8 compares F1 scores for five ER-Magellan datasets across Ditto variants and against DeepMatcher+ on full datasets.
  • Serialized-input experiment: Up to 5.2%: replacing DeepMatcher’s input with Ditto’s serialized entries improves results in the Abt-Buy dataset.Overall results do not significantly improve.
  • Baseline sources: The reported Magellan baseline results are taken from prior work because those studies used the same evaluation settings.

D LABEL EFFICIENCY EXPERIMENTS ON THE ER-MAGELLAN BENCHMARK

The label-efficiency experiments vary training-set sizes on five ER-Magellan datasets and compare four Ditto variants with a concurrent pre-trained-LM method.

  • Experimental setup: 500 to 2,000: training-set sizes were varied on five ER-Magellan datasets, with uniform sampling from each original training set.The evaluated variants were baseline, Ditto(DA), Ditto(DK), and full Ditto.
  • Method comparison: The concurrent method is essentially identical to baseline Ditto, whereas Ditto adds domain knowledge, data augmentation, and summarization and evaluates more datasets.
  • Evaluation protocol: Best test-set epoch versus best validation-set epoch: the concurrent evaluation can produce greater-or-equal F1 by selecting the best test result directly.The paper’s validation-based method is intended to prevent test-set overfitting.
  • Comparison with concurrent work: 4/5 datasets: optimized Ditto outperforms the concurrent method, while baseline Ditto does not because of different evaluation methods.

F EXPERIMENTS ON DIFFERENT WDC PRODUCT ATTRIBUTES

WDC experiments compare Ditto and DeepMatcher under different product-attribute subsets; Ditto performs best with title alone, while longer inputs can hurt performance.

  • Dataset attributes: The four WDC input attributes are evaluated in combinations, and some attributes can be missing from entries.SpecTable appears in 7% of entries in the full training set.
  • Attribute comparison: Title alone performs significantly better than other tested attribute combinations, by 3.2% to over 30%.The range is 3.2% for computer, xlarge and over 30% for watches, small.
  • Evaluation setting: Title-only inputs are reported for Ditto, while DeepMatcher accesses all four attributes to ensure its best performance.
  • Sequence-length limitation: 75.5 to 342.7 tokens: adding description to title increases average sequence length beyond Ditto’s default 256-token limit.Some useful title information is then removed by the summarization operator.
Loading 2004.00584v3…