PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Networks for Entity Matching: A Survey

Nils Barlaug, Jon Atle Gulla

arXiv:2010.11075v2cs.DBcs.CLcs.LG

TL;DR

Entity matching asks which records refer to the same real-world entity, a longstanding problem complicated by noisy and heterogeneous data. This survey organizes neural-network methods by entity-matching process steps, compares their contributions with traditional approaches, and introduces a taxonomy. It concludes that deep learning contributes hierarchical text representations and increasingly end-to-end pipelines, while evaluation breadth and interactive use remain constrained.

  • Problem

    Entity matching remains challenging because real-world records can be noisy, heterogeneous, and difficult to align across data sources.

  • Method

    The survey maps neural methods to entity-matching process steps, compares deep learning with traditional approaches, and proposes a taxonomy of deep neural networks.

  • Results

    Deep learning contributes hierarchical representation learning that can reduce handcrafted feature engineering and coalesce multiple entity-matching steps into end-to-end neural networks.

  • Takeaways & Limitations

    The survey provides a process-based view of how neural methods address entity matching and how their network properties relate to schema matching and blocking.

  • Takeaways & Limitations

    Entity-matching evaluation is constrained by limited dataset diversity, including a lack of industrial datasets suitable for broader evaluation and transfer learning.

Abstract

from arXiv · show

Entity matching is the problem of identifying which records refer to the same real-world entity. It has been actively researched for decades, and a variety of different approaches have been developed. Even today, it remains a challenging problem, and there is still generous room for improvement. In recent years we have seen new methods based upon deep learning techniques for natural language processing emerge. In this survey, we present how neural networks have been used for entity matching. Specifically, we identify which steps of the entity matching process existing work have targeted using neural networks, and provide an overview of the different techniques used at each step. We also discuss contributions from deep learning in entity matching compared to traditional methods, and propose a taxonomy of deep neural networks for entity matching.

1 INTRODUCTION

Entity matching identifies records that refer to the same real-world entity, but remains difficult because data sources differ and records are noisy. This survey organizes neural-network approaches by the process steps they address, compares deep learning with traditional methods, and proposes a taxonomy.

  • 1 INTRODUCTION: Entity matching links records from different sources to the same real-world entities despite inconsistent identifiers and names.The motivating example involves equipment inventories and maintenance logs that use no common identifiers.
  • 1 INTRODUCTION: Poor data quality includes typos, alternative spellings, missing values, inconsistent schemas, and semi-structured attributes.Examples include names such as “John Smith” and “Smith, John,” along with inconsistent address formats.
  • 1 INTRODUCTION: With |A| records in one source and |B| ∈ Θ(|A|) in another, entity matching presents Θ(|A|^2) possible pairs but typically only O(|A|) positive matches.This imbalance contributes to the scale of the matching problem.
  • 1 INTRODUCTION: The survey asks how neural methods differ in the aspects they solve, what benefits and challenges deep learning brings, and how the methods can be categorized.These questions structure the comparison across surveyed approaches.
  • 1 INTRODUCTION: The paper uses a reference process to map neural techniques to entity-matching steps, compares deep learning with traditional approaches, and proposes a neural-network taxonomy.It also discusses challenges and potential future work.

2 BACKGROUND

The background defines entity matching over records and schemas, introduces neural-network training and representation learning, and outlines neural approaches for text. It also notes that entity matching has multiple names and several scope assumptions.

  • 2 BACKGROUND: Entity matching seeks the largest relation M ⊆ A × B whose paired records refer to the same entity, even when the sources have different schemas.The formulation matches records across two data sources rather than requiring identical attribute structures.
  • 2 BACKGROUND: The problem definition assumes cross-source matching and the same taxonomic granularity, excluding some within-source duplicates and mismatched entity granularities.Examples of differing granularity include households versus individuals and street addresses versus postal-code areas.
  • 2 BACKGROUND: Neural networks learn parameterized functions from examples by minimizing a loss with stochastic gradient descent and backpropagation.Mini-batches reduce the amount of data evaluated per update, while backpropagation differentiates through the computational graph.
  • 2 BACKGROUND: Deep learning’s central strength is hierarchical representation learning from relatively unstructured inputs such as text.Embeddings transform characters or words into vectors that can be learned jointly with the rest of the network.
  • 2 BACKGROUND: Transformer-based networks use self-attention over the whole sequence, enabling greater parallelism and shorter paths for learning long-range dependencies than recurrent networks.Pretrained language models additionally provide contextualized word embeddings and can be fine-tuned.

3 RELATED WORK

Related work spans broad entity-matching surveys, specialized treatments of techniques such as probabilistic matching and blocking, and neighboring NLP matching tasks. The paper positions itself as the first dedicated, complete, and up-to-date survey of neural networks for entity matching.

  • 3 RELATED WORK: Earlier literature includes books and tutorials covering entity matching, duplicate detection, data quality, and data integration.These sources provide broad introductions rather than focusing exclusively on neural entity matching.
  • 3 RELATED WORK: Specialized surveys cover probabilistic matching and blocking, two established components or techniques within entity matching.Probabilistic methods trace back to Newcombe et al. and received a theoretical framework from Fellegi and Sunter.
  • 3 RELATED WORK: The authors claim to provide the first dedicated, complete, and up-to-date survey focused on neural networks for entity matching.They note that earlier works covered neural networks only to varying degrees.
  • 3 RELATED WORK: The paper distinguishes entity matching from coreference resolution, entity linking, and paraphrase identification because those tasks differ in input structure, context, or matching objective.Entity matching operates on structured records whose goal is identifying the specific real-world entity they denote.

4 THE ENTITY MATCHING PROCESS

The survey models entity matching as a flexible five-step process from preprocessing through classification, while noting that methods may interleave or omit steps. Neural networks are examined across these stages against traditional comparison, blocking, and classification techniques.

  • 4 THE ENTITY MATCHING PROCESS: The reference model organizes entity matching into preprocessing, schema matching, blocking, record comparison, and classification.The model is an abstraction extracted from prior literature rather than a universally fixed procedure.
  • 4 THE ENTITY MATCHING PROCESS: The process assumes two data sources and can yield matches even when methods interleave or skip individual steps.The surveyed literature generally focuses on two sources, although the abstract process could theoretically extend to more.
  • Preprocessing: Preprocessing standardizes inputs through transformations such as punctuation removal, lowercasing, normalization, and tokenization.It may also be viewed as feature extraction into a representation suitable for downstream tasks.
  • Schema matching: Schema matching identifies semantically corresponding attributes, although it is often handled separately or manually before entity matching.The survey includes it because neural methods can potentially perform schema matching jointly with later steps.
  • Blocking: Blocking reduces the quadratic candidate space before explicit pairwise comparison, trading lower precision for efficiency while comparison provides higher precision.Candidate selection produces C ⊆ A × B with |C| much smaller than |A × B|.
  • Classification: Classification assigns each candidate pair to match or nonmatch from its similarity vector using thresholds, rules, or learned models.Traditional approaches include probabilistic methods, manually constructed rules, decision trees, and support vector machines.

4.1 Data preprocessing

Neural networks support preprocessing mainly through learned numerical representations and hierarchical feature extraction. The surveyed methods vary in embedding granularity, representation levels, and whether records are represented independently or through cross-record interactions.

  • Representation learning: Neural preprocessing replaces some manual feature extraction by mapping text into numeric embeddings and learning increasingly abstract hierarchical representations.The survey distinguishes initial embeddings from later network layers that build representations at character, word, attribute, and record levels.
  • 4.1 Data preprocessing: Embedding granularity creates a trade-off: word embeddings shorten sequences but increase vocabulary size and vulnerability to out-of-vocabulary words.Character-level embeddings avoid some vocabulary issues at the cost of longer sequences.
  • Embeddings: Fine-tuning embeddings helped on hard datasets, whereas comparisons among fastText, GloVe, and word2vec found mixed evidence for a particular embedding model.One study found fastText advantageous for domain-specific out-of-vocabulary words, while another found no significant difference among alternatives.
  • Representation levels: The surveyed methods commonly begin with embeddings and frequently use recurrent neural networks to build sequence-aware representations.The survey also reports methods using other architectural choices, including attention-based mechanisms.
  • Representation levels: Independent representations depend on one record, whereas interdependent representations incorporate the record pair into feature extraction.Interdependent representations can use cross-record attention, while independent representations are formed before comparison.

4.2 Schema matching

Schema matching determines which attributes across two sources should be compared, with complexity depending on whether schemas are aligned, reordered, or incompatible. Neural approaches range from unsupervised MLP-based matching to joint sequence and attention-based models.

  • 4.2 Schema matching: Aligned schemas require no schema matching, misaligned schemas require one-to-one correspondence, and incompatible schemas require more complex attribute connections.For entity matching, an attribute may correspond to several attributes in the other source.
  • 4.2 Schema matching: Dirty attribute values require robust comparisons because values may appear under attributes different from their intended semantic fields.This can require comparing attributes that are not normally considered semantically related.
  • 4.2 Schema matching: Schema matching can use attribute values, metadata, or both to identify semantically related attributes.Metadata commonly consists of attribute names such as title or author.
  • Neural approaches: SEMINT, SMDD, and NNSM generate training data through unsupervised attribute clustering before training multilayer perceptrons.These approaches avoid human-labeled training data by using traditional unsupervised methods to create clusters and labels.
  • Neural approaches: Other neural methods transform incompatible schemas, compare attribute vectors without supervision, or learn schema matching jointly with record-pair comparison.Seq2SeqMatcher represents records as attribute-word sequences, while other methods use attention or explicit attribute information.

4.3 Blocking

Only a small number of surveyed methods apply neural networks to blocking. These methods learn record embeddings and use approximate nearest-neighbor search to reduce the candidate space before detailed comparison.

  • 4.3 Blocking: DeepER and AutoBlock embed records in a metric space and use approximate nearest-neighbor search to produce candidate pairs for blocking.Both use cosine distance and locality-sensitive hashing to prune the Cartesian product of the two data sources.
  • 4.3 Blocking: Approximate nearest-neighbor search improves feasibility in high-dimensional spaces but provides only a high probability, rather than a guarantee, of finding nearest neighbors.Both surveyed blocking methods use locality-sensitive hashing for this approximation.
  • 4.3 Blocking: DeepER trains end-to-end with pair comparison and classification, whereas AutoBlock trains specifically for blocking with a loss applied to cosine distance.They also use different LSH variants: hyperplane LSH for DeepER and cross-polytope LSH for AutoBlock.

4.4 Record pair comparison

Neural entity-matching methods compare distributed attribute- or record-level representations, using aligned or non-aligned comparisons and distributed or scalar similarities. Cross-attention can make representations dependent on the records being compared.

  • 4.4 Record pair comparison: Methods generally learn distributed representations at the attribute or record level before comparing the two records.Comparison layers reduce representations for individual records into representations suitable for record-pair classification.
  • 4.4 Record pair comparison: Attribute-aligned comparison matches corresponding attributes, whereas record-level comparison is less dependent on aligned schemas.Attribute alignment can restrict methods from handling schema mismatch.
  • 4.4 Record pair comparison: Most surveyed methods produce distributed similarity representations, commonly using vector difference, Hadamard product, or concatenation.Scalar similarities reduce complexity and training time but are less expressive.
  • 4.4 Record pair comparison: Distributed similarities outperform nondistributed similarities, while vector difference outperforms concatenation after non-cross-attention representation layers.These findings were reported by Mudgal et al.
  • 4.4 Record pair comparison: Cross-attention lets representations depend on the comparison partner, focusing them on information relevant to the pair.DeepMatcher uses word-level cross-record attention, while Hi-EM uses attention across character and word representations.

4.5 Classification

Classification is usually binary: a network takes a similarity vector and predicts whether the record pair matches. Most classifiers are similar MLP-based networks ending in softmax, with multiclass matching as an exception.

  • 4.5 Classification: Most methods classify a similarity vector with binary output indicating whether the record pair matches.Gottapu et al. instead classify each record from A directly to a corresponding record in B using |B| classes.
  • 4.5 Classification: Similarity vectors may come from separate similarity procedures or upstream neural layers, but classification networks are generally MLP variants ending in softmax.Earlier methods more often supplied externally computed similarities, whereas newer deep learning methods produce them upstream.

5 CONTRIBUTIONS FROM DEEP LEARNING

Deep learning shifts entity matching from hand-engineered, separated stages toward learned representations and increasingly fused neural pipelines. The survey identifies reduced feature-engineering needs, transfer learning, end-to-end processing, and neural approaches to schema matching and blocking as key contributions.

  • 5.1 Learned feature extraction and comparison: Learned feature extraction reduces reliance on ad-hoc handcrafted procedures by producing semantically rich representations from less-preprocessed records.Standardized steps such as tokenization remain, and nonstandard formats may still require handcrafted preprocessing.
  • 5.1 Learned feature extraction and comparison: Neural networks can combine feature extraction with comparison, reducing dependence on handcrafted string-similarity features and syntactic similarity.Traditional pipelines often compute several string similarities before classification.
  • 5.1 Learned feature extraction and comparison: Transfer learning brings pretrained embeddings, entity-matching models, and general language models into entity matching, supporting more powerful models with fewer task-specific labels.The surveyed methods include weighted sums of pretrained word embeddings and fine-tuned pretrained language models.
  • 5 CONTRIBUTIONS FROM DEEP LEARNING: Deep learning methods fuse feature extraction, record-pair comparison, and classification into one neural-network step, while retaining separate data processing.Some methods also apply neural networks to schema matching and blocking.
  • 5.2 Coalescing the entity matching process: Nie et al. jointly train schema matching, record-pair comparison, and classification, allowing schema matching to adapt to downstream use.This is identified as the first surveyed end-to-end method to incorporate schema matching in this way.
  • 5.2 Coalescing the entity matching process: Neural blocking creates indexable record representations and retrieves candidate pairs with approximate nearest-neighbor search in subquadratic time.This reduces the core blocking mechanism to indexing rather than eliminating the blocking step.
  • 5.2 Coalescing the entity matching process: End-to-end coalescing can reduce process steps and pipeline complexity by sharing representation learning across schema matching, blocking, and record-pair comparison.The survey connects this benefit to the difficulty of tuning interactions among multiple engineered stages.

6 TAXONOMY OF DEEP NEURAL NETWORKS FOR ENTITY MATCHING

The survey’s taxonomy classifies deep entity-matching networks using two binary properties: attribute alignment and representation interdependence. These axes distinguish four architectural categories with different compatibility with schema matching and blocking.

  • 6 TAXONOMY OF DEEP NEURAL NETWORKS FOR ENTITY MATCHING: The taxonomy contains four categories defined by attribute-aligned versus non-attribute-aligned comparison and independent versus interdependent representations.It considers networks that perform feature extraction and excludes a method that does not itself represent a neural network.
  • 6 TAXONOMY OF DEEP NEURAL NETWORKS FOR ENTITY MATCHING: Attribute-aligned comparison incorporates schema knowledge and may simplify training, but it prevents schema matching.Non-attribute-aligned comparison is more compatible with schema matching, although it does not imply that a method performs it.
  • 6 TAXONOMY OF DEEP NEURAL NETWORKS FOR ENTITY MATCHING: Interdependent representations can inspect both records simultaneously, making comparison more focused but preventing subquadratic comparison over many records.Independent representations can instead support indexing-based blocking.
  • 6 TAXONOMY OF DEEP NEURAL NETWORKS FOR ENTITY MATCHING: Rigid encoders independently represent attributes before comparison, although the surveyed methods do not use them for blocking.Independent attribute representations could nevertheless be used for blocking.
  • 6 TAXONOMY OF DEEP NEURAL NETWORKS FOR ENTITY MATCHING: Flexible encoders independently represent records and can support blocking, while schema matching is possible in principle when differing schemas can be encoded.DeepER and AutoBlock demonstrate blocking with independent record representations, but no surveyed method performs schema matching this way.

7 EVALUATION

Evaluation remains difficult because surveyed methods often target different tasks, datasets, splits, and metrics, limiting direct comparison. Partial benchmark evidence suggests Transformer-based methods, especially Ditto, perform strongly, while deep learning is often most advantageous on noisy and heterogeneous data.

  • Direct comparison is limited because methods use different tasks, datasets, train-test splits, and evaluation procedures, while agreed-upon benchmarks remain uncommon.For most surveyed methods, reported experimental data do not support direct comparison.
  • Transformer-based networks, especially Ditto, appear state-of-the-art on the partially comparable Mudgal et al. benchmark.The benchmark covers only certain entity-matching aspects, and Table 8 summarizes the available comparable results.
  • Reported comparisons generally favor deep learning, but traditional methods remain competitive across several evaluations.Most comparisons use Magellan and F1 scores; DeepMatcher generally outperforms Magellan with some exceptions.
  • Deep learning’s relative strength increases as data quality decreases, with similar patterns reported for DeepMatcher, Seq2SeqMatcher, and AutoBlock.AutoBlock is especially strong against traditional blocking techniques on dirty data.
  • With few labeled examples, Magellan can significantly outperform a deep model, but transfer learning and active learning make the model substantially more competitive.With enough examples, the model performs favorably compared with traditional methods.

8 FUTURE RESEARCH

Future research should address interpretability, computational demands, training-data needs, end-to-end coverage, dataset diversity, and standardized evaluation. Promising directions include flexible pretrained encoders and benchmarks spanning more entity-matching settings.

  • 8.1.1 Explainability and ease of debugging: Deep neural networks remain difficult to debug because integrating more process steps removes inspectable checkpoints and obscures why records match.Alignment scores offer some help, but the survey reports no comprehensive debugging approach.
  • 8.1.2 Running time in interactive settings: High training and prediction costs limit deep learning’s potential in interactive entity-matching settings, especially relative to engineered traditional pipelines.Interactive users generally cannot be expected to wait very long.
  • 8.2.1 End-to-end approach with schema matching and blocking: No surveyed method yet combines schema matching, blocking, record-pair comparison, and classification in one end-to-end neural solution.Such a system would need efficient independent representations for blocking and non-attribute-aligned comparison for schema matching.
  • 8.2.2 More open datasets: More diverse open datasets are needed because domain and source variation complicates methods intended to work across datasets.The survey specifically notes a lack of industrial datasets and links broader coverage to more complete evaluation and transfer learning.
  • 8.2.2 More open datasets: Agreed-upon benchmarks could improve comparisons by standardizing matching, blocking, transfer learning, active learning, computational performance, and efficiency evaluations.Existing methods often evaluate on different data and under different procedures.
  • 8.2.3 Pretrained models: Pretrained entity-matching models could reduce training time and required examples, but broad transfer is difficult because data sources vary substantially.The survey identifies flexible encoders supporting blocking and schema matching as especially significant opportunities for transfer learning.

9 CONCLUSION

The survey maps neural-network methods onto the entity-matching process, explains deep learning’s contributions, and proposes reference models and a taxonomy. It closes by identifying challenges and future research opportunities.

  • The survey uses a reference model to show which entity-matching steps neural methods address and which techniques they employ.
  • Deep learning contributes hierarchical text representations that reduce handcrafted feature engineering and support coalescing multiple matching steps into end-to-end neural networks.
  • The authors propose a deep-learning reference model to clarify how increasingly coalesced neural approaches change the entity-matching process.
  • The taxonomy differentiates entity-matching neural networks using properties relevant to supporting schema matching and blocking.
  • Future work should address explainability, interactive running time, training-data requirements, complete end-to-end processing, and more open datasets.
Loading 2010.11075v2…