PaperlayerView exact revision on arXiv

Source-linked AI summary

Entity Matching using Decision Models

Aaron Steiner, Ralph Peeters, Christian Bizer

arXiv:2610.34626v1cs.CLcs.AI

TL;DR

Entity matching methods face a trade-off between the training-data needs of fine-tuned PLMs and the cost and latency of generative LLMs. This paper evaluates decision models against both alternatives, finding that the strongest approach their F1 performance and that calibrated probabilities can support effective cascades.

  • Problem

    Fine-tuned PLM matchers require thousands of labeled pairs and generalize poorly to unseen entities, while generative LLMs can avoid task-specific training but cost more time and money per record pair.

    Source: §1 · p. 1

  • Method

    The paper systematically evaluates six decision models across eight benchmarks and varies prompts, serializations, and model cascades without task-specific training data.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Across eight benchmarks, four decision models come within 1.23 F1 points of GPT-6 Luna; PPLX v1.2 exceeds it by 0.97 and Ditto by 7.51 points F1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    1.94 to 3.97 points of mean F1 are gained by cascades that route uncertain pairs to an ensemble of three generative LLMs.

    Source: §1 · p. 2

Abstract

from arXiv · show

Entity matching is the task of deciding whether two records describe the same real-world entity. Current entity matching methods rely on pre-trained language models (PLMs) which require task specific fine-tuning or generative large language models (LLMs) which do not require task-specific fine-tuning but are slow and expensive at matching large record sets. Decision models are a new class of LLM-based models that return a probability for each predefined answer option rather than generate text. This paper investigates decision models as an alternative to generative LLMs and fine-tuned PLMs for entity matching. We evaluate six decision models in a zero-shot setting across eight benchmarks and compare their performance with that of the generative LLM GPT-6 Luna and the PLM-based matcher Ditto. Our experiments show that, without task-specific training data, the four strongest decision models achieve mean F1 scores close to that of GPT-6 Luna, with the best model slightly surpassing it. This model also outperforms Ditto on six of the eight benchmarks. All decision models are faster than GPT-6 Luna, and most are less expensive. For three of the four strongest decision models, the predicted match probabilities are well calibrated without requiring any task-specific recalibration. A model cascade that routes the least confident record pairs to an ensemble of generative LLMs improves the mean F1 scores of the four strongest decision models by 1.9-4.0 percentage points. Using the best decision model and its best-performing prompt for each benchmark, the cascade surpasses the highest published F1 scores from literature on four of the eight benchmarks.

1 Introduction

The introduction frames decision models as a potential alternative to fine-tuned PLM matchers and generative LLMs, then previews their benchmark performance, robustness, calibration, and efficiency. The study evaluates whether they can combine strong matching results with lower cost and latency.

1 Introduction

  • Fine-tuned PLM matchers need thousands of labeled pairs and generalize poorly to unseen entities, whereas generative LLMs can perform comparably with few or no task-specific examples but cost more time and money per pair.The paper evaluates decision models against both approaches on F1, calibration, cost, and throughput.Source: §1 · p. 1
  • Decision models return match probabilities directly and need no task-specific training, while generative LLMs return text and may require confidence estimation.Calibrated probabilities can help select a precision–recall threshold or defer uncertain pairs to manual review or a stronger model.Source: §1 · pp. 1–2
  • Without task-specific data, the four strongest decision models come within 1.23 F1 points of GPT-6 Luna; PPLX v1.2 exceeds GPT-6 Luna by 0.97 and Ditto by 7.51 points F1.The study evaluates decision models on eight benchmarks.Source: §1 · p. 2
  • The strongest decision models react little to prompt wording (at most 1.21 F1 between the best and default prompts) and record serialization or perturbation (median loss of 0.78 F1 across nine input variants).Source: §1 · p. 2
  • 1.94 to 3.97 points of mean F1 are gained by cascades that route uncertain pairs to generative LLMs; the decision models’ probabilities are well calibrated without task-specific recalibration.The reported gains use an ensemble of three generative LLMs.Source: §1 · p. 2
  • Decision models process record pairs 1.4–1.8 times as fast as GPT-6 Luna and cost 6 to 85 times less than GPT-6.1 Sol, but are 14–18 times slower than Ditto on a local GPU.Their speed and cost fall between PLM-based matchers and generative LLMs.Source: §1 · p. 2

2 Experimental Setup

The shared setup tests zero-shot decision models across eight benchmarks, varying prompts, record serialization, thresholds, and cascade partners while comparing with GPT-6 Luna and Ditto.

2 Experimental Setup

  • The experiments vary decision models, prompts, serializations, thresholds, and cascade partners, with no benchmark fine-tuning or prompt examples.Astra rules are the exception: they add matching rules derived from 100 labeled development pairs per benchmark.Source: §2 · p. 2

2.1 Benchmark Datasets

  • Eight benchmarks cover product and bibliographic records, heterogeneous attributes, German offers, unseen products, and hard candidate pairs.Table 1 reports matching and non-matching pair counts in validation and test sets.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 Decision Models

  • Decision models return probabilities for fixed answer options; four open-weight models use a single forward pass and avoid task-specific fine-tuning.Five of the six models target calibrated probabilities; OpenAI discloses no calibration method.Source: §2.2 · p. 3

2.3 Record Serialization

  • The baseline serializes each record’s attributes line by line and asks a yes/no identity question, predicting a match when its probability is at least 0.5.The generative LLM receives the prompt and record pair through separate chat messages with a constrained JSON response.Source: §2.2 · p. 4, §2.3 · p. 4

2.4 Evaluation Metric and Significance

  • Matching quality is match-class F1; paired bootstrap tests compare configurations over 10,000 resampled test sets.The 95% interval holds each run’s decisions fixed and does not capture variation between runs.Source: §2.4 · p. 4

3 Matching Performance

Four decision models approach GPT-6 Luna’s mean F1, with PPLX v1.2 slightly ahead; performance varies by benchmark, and decision models outperform Ditto on six benchmarks.

Effectiveness

  • Four of six decision models finish within 1.2 F1 points of Luna Gen., and PPLX v1.2 slightly exceeds it.The reported results use the default prompt B0; Table 4 reports test-set F1 as means and standard deviations over three runs.Source: Table 4 · p. 5, §3 · p. 5

Per benchmark

  • PPLX v1.2 leads Luna Gen. by 3.7 F1 on DBLP-Scholar and 6.1 on billiger.de, while Clef-flash and Kev falter on some benchmarks.Clef-flash scores 40.4 F1 on Dn7 Walmart-Amazon and 43.8 on billiger.de; Kev scores 76.7 on DBLP-ACM.Source: Table 4 · p. 5, §3 · p. 5

Decision model versus base model

  • PPLX v1.2 exceeds its generatively prompted base model Qwen3.8-27B on all eight benchmarks, by 19 F1 on average.Averaged across benchmarks, precision is 85.8% for PPLX v1.2 and 60.0% for Qwen3.8-27B; Qwen3.8-27B returns a valid answer for every pair.Source: Table 4 · p. 5, §3 · p. 5

Comparison to a PLM baseline

  • Without task-specific training data, a decision model exceeds Ditto on six of eight benchmarks, by up to 21 F1; Ditto leads on the two bibliographic benchmarks.Ditto leads by 0.32 F1 on DBLP-ACM and 3.1 on DBLP-Scholar; the WDC Products and billiger.de test sets contain only products unseen in training.Source: Table 4 · p. 5, §3 · p. 5

4 Prompt Sensitivity

The study varies prompt wording and finds modest average sensitivity for decision models, while benchmark-specific prompt selection can improve F1, especially for Clef-flash.

Prompt design

  • The prompt comparison covers twelve prompts differing in instruction text, including rewordings, prior prompts, and handwritten or benchmark-derived matching rules.The study selects the highest-test-F1 prompt for each model and benchmark for later experiments.Source: §4 · pp. 5–6, Table 5 · p. 6

Effectiveness

  • Astra rules are the best prompt for five of seven models, but gains over B0 are at most 1.2 F1 except for Clef-flash at 4.6.Handwritten rules fall below B0 for every model and below Astra rules by up to 6.0 F1; Astra rules use benchmark-specific labeled pairs.Source: §4 · pp. 5–6, Table 5 · p. 6

Sensitivity

  • Prompt sensitivity stays at or below 3.4 F1 for all seven models, although Kev’s DBLP-ACM scores vary by 40 points across prompts.The sensitivity measure is the standard deviation of each model’s averaged F1 over prompts.Source: §4 · p. 6

Prompt-to-model fit

  • No prompt is best for every model–benchmark pair; selecting prompts per benchmark raises mean F1 by 1.0–1.8 for six models and 5.1 for Clef-flash.Astra rules are selected for 21 of 56 combinations, and gains on individual combinations reach 9.5 F1 for Luna Gen. on billiger.de.Source: §4 · p. 6

5 Robustness

This section tests whether changing record formatting or content affects matching decisions. Formatting changes have limited effects overall, while deleting an attribute is the most consequential perturbation.

5.1 Input Serialization

  • No alternative serialization improves F1 by more than 0.38 points for any model, and removing attribute names significantly lowers F1 for all but Clef-flash.Omitting missing values and JSON stay within 1.6 F1 of the default for all models except Kev with JSON.Source: §5.1 · p. 7

5.2 Perturbations

  • Attribute deletion lowers every model’s mean F1 by 2.6–3.8 points; the loss averages 7.8 points on Walmart-Amazon.Unlike the other perturbations, deletion removes information such as a brand or model number that may not appear in remaining attributes.Source: §5.2 · pp. 7–8
  • Entry swaps, reversed attribute order, lowercasing, and title typos change F1 by at most 1.6 points and flip at most 3.7% of decisions.Title typos significantly lower F1 for all models except Clef-flash, but by no more than 1.6 points.Source: §5.2 · p. 7
  • PPLX v1.2 has the lowest flip rate across all five perturbations, at most 1.3%; Kev or Clef-flash has the highest.Across nine variants, median F1 loss is 0.78 for Jev 1.13, PPLX v1.2, Clef, and Luna Dec.Source: §5.2 · p. 8

6 Match Probabilities and Model Cascades

The section evaluates raw match probabilities, validation-tuned thresholds, and cascades that defer uncertain pairs to generative LLMs. It then compares PPLX v1.2 and its cascade with published benchmark results.

6.1 Model Calibration

  • PPLX v1.2, Clef, and Jev 1.13 have calibration errors (REL ×100) of 0.10 to 0.22 without recalibration; Clef-flash and Kev have the highest calibration errors and overestimate match probabilities.Raw probabilities are assessed on full test sets using each model’s selected prompt.Source: §6.1 · p. 8, Figure 2 · p. 9

6.2 Decision Threshold

  • Validation-tuned thresholds add mean F1 of 4.5 for Clef-flash and 1.6 for Kev with per-benchmark thresholds, but bring no gain for Jev 1.13, Clef, or PPLX v1.2.For Jev 1.13, Clef, and PPLX v1.2, even test-set-optimal thresholds add at most 0.83 F1.Source: §6.2 · p. 9, Table 10 · p. 9

6.3 Cascade with an LLM Ensemble

  • At a 0.5% target risk, the other four decision-model cascades defer 10.3%–12.5% of test pairs; deferred pairs contain 88%–89% of their models’ errors, and ensemble majority correctly decides 85%–88%.Source: §6.3 · pp. 10–11
  • A cascade raises each of the four decision models’ mean F1 by two to four points, to about 91, near the ensemble’s mean F1 on all pairs.The gains are concentrated on product benchmarks, reaching 9.5 F1 on billiger.de; on two bibliographic benchmarks and Semi-HETER book records, F1 changes by at most 1.7.Source: §6.3 · p. 10, Table 11 · p. 10
  • For Jev 1.13 and PPLX v1.2, changing target risk from 0.5% to 0.25% or 1.0% changes mean F1 by at most 0.30, while deferred-pair share ranges from about one in twenty to one in six.Test selective risk stays below target in all six settings; a Sol-only cascade reaches about Sol’s level at 18%–35% of Sol’s cost.Source: §6.3 · pp. 10–11

6.4 Comparison to Published Results

  • The PPLX v1.2 cascade reaches the highest listed F1 on four English product benchmarks, but remains below the best published result on billiger.de and the three publication and book benchmarks.It exceeds the best published zero-shot LLM and trained matcher on six benchmarks each, by up to 4.3 and 27.4 F1, respectively.Source: §6.4 · p. 11, Table 12 · p. 11

7 Cost and Throughput

This section compares model costs and throughput, including decision models, generative LLMs, Ditto, and cascades.

7.1 Cost

  • With selected prompts, decision-model costs range from 8.9 USD for PPLX v1.2 to 126.5 USD for Clef per one million pairs, versus 42.9 USD for Luna Gen.Most decision models cost less than Luna Gen.; Clef and PPLX v1.2 use the same Qwen3.8-27B backbone, but Clef costs 14 times as much per pair.Source: §7.1 · p. 11, Table 13 · p. 12
  • A cascade with the ensemble costs 16%–20% of the ensemble alone; a Sol-only cascade costs 18%–28% of the ensemble cascade at nearly the same F1.The cascade’s cost is driven by the partner rather than the decision model.Source: §7.1 · p. 12, Table 13 · p. 12

7.2 Throughput

  • All decision models process record pairs 1.4–1.8 times as fast as Luna Gen., while taking 14–18 times as long as Ditto on one A40 GPU.The local-GPU and API throughput measurements are not directly comparable.Source: §7.2 · p. 12, Table 14 · p. 12

8 Error Analysis

The error analysis compares Jev 1.13 and PPLX v1.2 with three generative LLMs, examining shared errors, confidence, and error patterns.

Error Analysis

  • Jev 1.13 and PPLX v1.2 share 454 errors, about two thirds of each model’s errors, and half of the shared errors have unclear reference labels.The shared errors concentrate on billiger.de, DBLP-Scholar, WDC Products, and Dn7 Walmart-Amazon.Source: §8 · pp. 12–13
  • Most errors are low-confidence: most fall between match probabilities of 0.1 and 0.9, a range covering only 14% of Jev 1.13’s and 8% of PPLX v1.2’s test pairs.Errors made by only one of the two models lie closest to the decision threshold, while PPLX v1.2 is often confident on shared errors.Source: §8 · p. 13
  • The three LLMs make fewer errors but mostly the same ones; the errors they avoid are mostly model-specific errors near the threshold that cascades defer.All three LLMs repeat 283 of the 454 errors shared by Jev 1.13 and PPLX v1.2, mostly false positives.Source: §8 · p. 13
  • Clef misclassifies about as many pairs as Jev 1.13, while Clef-flash and Kev misclassify more than twice as many.Most additional errors are false positives on pairs that Jev 1.13, PPLX v1.2, and the three LLMs usually decide correctly.Source: §8 · p. 13
  • A cascade can mainly fix model-specific errors, while errors shared across the decision models and LLMs limit all five models.The LLMs correctly decide 49.7%–72.1% of errors made by only one of Jev 1.13 and PPLX v1.2.Source: §8 · p. 13

9 Related Work

Prior entity-matching work centers on fine-tuned PLMs and prompted or fine-tuned LLMs, while early decision-model studies leave open their performance on full benchmark test sets. Related evaluations also examine calibration, robustness, and applications beyond entity matching.

9 Related Work

  • Fine-tuned PLMs have defined entity-matching state of the art since 2020, while prior work also studies prompted and fine-tuned LLMs and cheaper alternatives.The present study extends earlier work by evaluating decision interfaces, match probabilities, thresholds, and deferral to generative LLMs.Source: §9 · p. 13
  • Decision-model research spans model audits, public-project usage, schema matching, semantic joins, and cost-accuracy optimization [40] [23] [22] [46] [19].An audit found no independent accuracy advantage over comparable language-model label probabilities, while Jev’s clearest gains were latency and cost [40].Source: §9 · p. 13
  • Earlier decision-model evaluations report strong accuracy and calibration in some settings, but weaknesses on specialist decisions and uncertainty estimation remain.JEVal found weaker performance on specialist-knowledge decisions and faithful uncertainty estimation [11]; System1-Arena reported results on non-agentic accuracy [7].Source: §9 · p. 13
  • Prior work reports sensitivity to option names, option sets, order, and input serialization, while paraphrasing effects vary by model and data [1] [39] [18] [34] [51] [17].Paraphrasing barely changed Jev’s decisions [17], but affected decision models and LLMs alike on tabular data [51].Source: §9 · p. 14
  • Before this study, published entity-matching evidence for decision models covered subsets, averages, synthetic cases, or selected test sets—not multiple models on full public benchmark test sets.System1-Arena included 500 pairs but reported a five-task domain average [7]; a 12-test-set repository reported Ditto-beating Jev F1 only on Abt-Buy and structured Walmart-Amazon [21].Source: §9 · p. 14

10 Conclusion

Across eight entity-matching benchmarks, the strongest zero-shot decision models approached or matched GPT-6 Luna’s F1 and beat Ditto on most benchmarks. Their speed, cost, calibration, and cascade performance support their use when task-specific training data is unavailable.

10 Conclusion

  • Without task-specific training data, the strongest decision models approach or match GPT-6 Luna’s F1 and outperform Ditto on most benchmarks.The conclusion characterizes decision models as suitable for entity matching across the eight evaluated benchmarks.Source: §10 · p. 14
  • Most decision models need no threshold tuning and are largely insensitive to prompt wording and input format; matching rules from a few labeled pairs add modest gains.These findings describe the tested models and evaluation setting.Source: §10 · p. 14
  • Routing the least-confident pairs to a strong generative LLM lets a cascade match that LLM’s F1 at a fraction of its cost.The conclusion attributes the cascade’s effectiveness to the decision models’ well-calibrated match probabilities.Source: §10 · p. 14
  • Decision models are faster than generative LLMs and substantially less expensive than the strongest ones, but fine-tuned PLMs can be cheaper and faster when training data is sufficient.The conclusion frames decision models as an efficient alternative specifically when task-specific training data is unavailable.Source: §10 · p. 14

Related papers

Loading 2610.34626v1…