PaperlayerView exact revision on arXiv

Source-linked AI summary

Curating Merchant-Matching Training Data with Two Confidence-Gated Local LLM Judges

Donghao Huang, Jinling Pei, Zhaoxia Wang

arXiv:2609.33878v2cs.AIcs.LG

TL;DR

Joint judge abstention can be mistaken for evidence that no acceptable candidate exists, risking false no-match labels while conservative labeling reduces coverage. The study replays two local judges’ stored outputs and retains agreed selections or joint abstentions using separate ordered thresholds that keep positive and negative label sets disjoint. Separately gated agreement retained 1,633 of 2,000 queries at 96.88% purity in this retrospective study.

  • Problem

    Joint judge abstention can be mistaken for evidence that no acceptable candidate exists, risking false no-match labels while conservative labeling reduces coverage.

    Source: §I · p. 1

  • Method

    The study replays two local judges’ stored outputs and retains agreed selections or joint abstentions using separate ordered thresholds that keep positive and negative label sets disjoint.

    Source: §I · p. 1, §VII · p. 7

  • Results

    Separately gated agreement retained 1,633 of 2,000 queries at 96.88% purity in this retrospective study.

    Source: §VII · p. 7

  • Takeaways & Limitations

    Separate thresholding and auditing for positive and negative pseudo-labels are motivated by the observed risks of noisier negatives and persistent false abstentions.

    Source: B. Curation Purity and Coverage · p. 5, C. Residual Errors and Review · p. 6, §VII · p. 7

    The retrospective study does not establish student utility, and teacher and prompt choices remain in-sample; fresh-data curation and student evaluation are needed to assess downstream value.

    Source: §VI · p. 6, §VII · p. 7

Abstract

from arXiv · show

Merchant matching resolves a noisy payment descriptor to a retrieved merchant entity or returns no match. A key challenge in curating training labels is distinguishing teacher abstention from evidence that no acceptable entity exists: false no-match labels contaminate pseudo-labeled data, while conservative labeling reduces coverage. We investigate whether agreement between two local large language model judges improves pseudo-label reliability. A label is retained only when the judges agree, with separate ordered thresholds for selections and abstentions that guarantee disjoint positive and negative label sets. Retrospective replay on 2,000 expert-annotated queries shows that higher selection thresholds can improve positive-label purity, whereas higher abstention thresholds increase false no-match labels. At thresholds (0.86, 0.80), Muse Glimmer 30B and Gemma 4 31B jointly label 1,633 queries (81.7% coverage) at 96.88% purity; positive and negative purities are 99.47% and 93.38%. This exceeds either constituent model at the same thresholds by more than two percentage points, with lower coverage. A split-half check finds only 0.14 percentage points of threshold-selection optimism. A symmetric threshold of 0.86 adds 40 erroneous no-match labels, while 46 false abstentions persist even with no confidence threshold. Across five matched within-model comparisons, higher reasoning effort yields no clear F0.5 gain and increases median latency by 1.8-5.0 times. These results motivate separate thresholding and auditing for positive and negative pseudo-labels. The study establishes label purity, not student utility; fresh-data curation and student fine-tuning remain necessary to demonstrate downstream value.

I. INTRODUCTION

The paper studies whether agreement between two local LLM judges can yield reliable merchant-matching labels while distinguishing shared abstention from evidence that no acceptable candidate exists. It evaluates separately gated labeling, threshold trade-offs, and reasoning effort using retrospective outputs.

I. INTRODUCTION

  • False no-match labels can contaminate merchant-level aggregation, while abstentions leave records unenriched.The paper frames these competing costs using a precision-weighted F0.5 objective.Source: §I · p. 1
  • The downstream goal is fresh training-data curation and fine-tuning smaller local judges, with local teachers intended to keep labeling within the operator’s environment.Expert annotation is costly, and merchant records may include sole proprietors.Source: §I · p. 1
  • Agreement on a selected merchant proposes a positive label, but joint abstention may reflect an absent candidate, insufficient evidence, or shared error.A confidence gate can turn tentative selections into abstentions, so a shared threshold may clean positive labels while making negatives noisier.Source: §I · p. 1
  • The study tests Muse Glimmer 30B and Gemma 4 31B on a frozen expert-annotated benchmark.It formalizes separately gated agreement, measures purity and coverage, compares the pair with each judge, and audits shared false abstentions and deferred rows.Source: §I · p. 1
  • The paper also evaluates within-model reasoning-effort trade-offs using paired quality comparisons and measured latency.Its replay uses existing outputs without student training or new production labeling; model and threshold selection reuse the benchmark, so estimates are retrospective.Source: §I · p. 1

II. RELATED WORK

Prior work covers entity matching, task-dependent reasoning, LLM annotation and distillation, weak supervision, and selective prediction. This paper focuses on label quality when agreement on abstention is converted into a no-match label.

Entity matching

  • The curation unit is a query and its candidate pool, labeled with a selected record or no-match.Related entity-matching work includes pairwise classification, example selection, fine-tuning, and cost-aware control –.Source: §II · pp. 1–2

Task-dependent reasoning

  • Higher reasoning effort has heterogeneous effects across tasks and models, motivating evaluation alongside task and computational cost.Prior entity-matching comparisons report favorable low-effort quality–cost trade-offs for GPT-5 but higher F1 at high effort for GPT OSS 120B and 20B.Source: §II · p. 2

LLM annotation and distillation

  • This study examines label quality in proprietary listwise matching, especially the risk of treating agreement on abstention as evidence for a negative label.Steiner and Bizer [14] demonstrate downstream distillation across five public benchmarks, whereas this paper does not study student outcomes here.Source: §II · p. 2

Weak supervision and selective prediction

  • Unlike labeling-function abstention, this policy converts joint judge abstention into a proposed no-match label that requires auditing.The paper assesses a deterministic agreement rule directly against expert labels; student utility remains unmeasured.Source: §II · p. 2

Evidence acquisition in financial reasoning

  • The judges reason once over a fixed retrieved pool and acquire no additional evidence, making their agreement a retention signal rather than an evidence-gathering step.Targeted evidence acquisition for deferred and audited rows is presented as a complementary direction, given shared false abstentions.Source: §II · p. 2

III. TASK AND EXPERIMENTAL SETUP

The evaluation replays stored judge outputs on a frozen expert-annotated benchmark, applying a gate to candidate selections and abstentions. It reports teacher quality using precision-weighted metrics because wrong selections and missed matches have asymmetric costs.

Task and gate

  • The evaluator accepts a nonnull selection only when the name-match flag, confidence threshold, and location-field checks pass; explicit abstention and gate rejection produce the same deployed action.The prompt versions state a .60 cutoff, the deployed evaluator uses τ = 0.70, and the sweep includes both.Source: §III · p. 2

Benchmark and demonstrations

  • The frozen evaluation set contains 2,000 expert-annotated queries, including 1,304 with acceptable candidates and 696 without.Purity is agreement with the expert labels, including multi-record gold sets; the original annotation procedure and inter-annotator agreement are undocumented.Source: §III · p. 2

Configurations and serving

  • The evaluated pair combines Muse Glimmer 30B at zero-shot low effort with Gemma 4 31B [20] using 20 examples; the configurations were selected on this benchmark.Their prompts and demonstrations differ, so pair gains cannot be attributed to model family alone.Source: §III · p. 2
  • Latency measures serial model-call time, not concurrent production serving or annotation cost, and each query requires both local judge calls.The local runs use an Apple M5 Max workstation with 128 GB unified memory and bf16 weights served through Ollama.Source: §III · p. 2

Judge metrics

  • F0.5 weights precision twice as much as recall, reflecting that wrong or spurious selections propagate into merchant aggregates while missed matches can be retried or reviewed.N-FMR is reported alongside F0.5 because a precision-weighted score can conceal aggressive selection on no-match rows.Source: §III · p. 3

IV. SEPARATELY GATED AGREEMENT

The policy retains agreed selections as positive labels and joint abstentions as no-match labels, leaving other rows for review. Separate ordered thresholds guarantee disjoint positive and negative sets, but an abstention gate does not certify that no acceptable candidate exists.

IV. SEPARATELY GATED AGREEMENT

  • Rows in P receive the agreed record ID, rows in N receive a no-match pseudo-label, and remaining rows are deferred for review.Different selections are deferred even if both happen to belong to the expert-accepted set; label production does not consult that set.Source: §IV · p. 3
  • The condition τabs ≤ τsel guarantees that positive and negative label sets are disjoint for all outputs.If τabs > τsel, selections in the intervening confidence range can enter both sets; the inverted pair (.80, .86) produces 19 contradictory labels in this study.Source: §IV · p. 3
  • Raising τsel shrinks P, while raising τabs grows N; these set changes are structural, but purity changes depend on the data.τabs thresholds selection confidence, not confidence in a no-match label, and a gate-induced abstention cannot certify candidate absence.Source: §IV · p. 3
  • Coverage counts retained positive and negative labels, while total purity is reported alongside positive and negative purity because label mix affects the total.The study uses pointwise retrospective intervals; its bootstrap intervals do not account for configuration search, entity clustering, or repeated inference variability.Source: §IV · p. 3

A. Reasoning Effort and the Quality–Latency Trade-off

Across five matched within-model comparisons, higher reasoning effort produced no clear F0.5 gain and increased median latency. The results apply to this frozen benchmark and do not establish that disabling reasoning is advantageous.

A. Reasoning Effort and the Quality–Latency Trade-off

  • The frozen-benchmark comparison uses 2,000 queries, τ = .70, and provider-specific effort settings; the measured latency is median serial model-call time.The table caption clarifies that latency is not production throughput or cost.Source: A. Reasoning Effort and the… · p. 3, Table I · p. 4
  • F0.5 changes were −0.42 points for Muse medium, −0.15/−0.50 for GPT OSS 120B medium/high, and −0.31/+0.04 for Sonnet medium/high.Even Sonnet’s small high-effort gain has interval [−0.52, 0.63] at 2.7× latency.Source: A. Reasoning Effort and the… · p. 3
  • All five paired intervals include zero: higher effort offers no clear F0.5 gain, while median latency increases 1.8–5.0×.Figure 1 shows the paired changes relative to low effort; comparisons of latency are within each serving backend.Source: A. Reasoning Effort and the… · p. 3, Figure 1 · p. 4
  • This effort comparison differs from Huang et al. in task, metric, candidate ranking, and thinking protocol, limiting direct transfer of effort recommendations.The present study evaluates confidence-gated F0.5 on noisy merchant queries with 19–21 candidates, unlike their eight clean-clean datasets and F1 protocol.Source: A. Reasoning Effort and the… · p. 4

B. Curation Purity and Coverage

On the 2,000-query benchmark, the two-judge policy retains 1,633 labels at 96.88% purity and 81.7% coverage. Purity responds differently to selection and abstention thresholds, and the held-out check finds small threshold-selection optimism while leaving model and prompt choices in-sample.

B. Curation Purity and Coverage

  • At the same gates, pair purity exceeds Muse’s 94.81% and Gemma’s 94.63% by 2.07 and 2.25 percentage points, respectively, with fewer retained labels.The single-judge policies retain 1,850 and 1,900 labels, respectively.Source: B. Curation Purity and Coverage · p. 4
  • Positive purity rises from 99.47% at τsel=0.86 to 99.79% at .95, while the retained positive set falls to 467 labels.At τsel=.70, the pair retains 1,100 positives at 98.91% purity; small reversals show purity is not guaranteed to rise monotonically.Source: B. Curation Purity and Coverage · p. 4, Figure 2 · p. 5
  • Negative purity falls from 93.38% at τabs=0.80 to 88.55% at .90 as false no-match counts rise from 46 to 87.At the sweep’s upper endpoint, false no-match labels reach 1,265; newly admitted matchable rows explain the decline.Source: B. Curation Purity and Coverage · p. 4, Figure 2 · p. 5
  • 1,633 labels (81.7% coverage) achieve 96.88% total purity at (τsel, τabs) = (0.86, 0.80), with positive and negative purities of 99.47% and 93.38%.The policy retains 938 positive labels and 695 negative labels; 367 rows are deferred.Source: B. Curation Purity and Coverage · p. 4, Table II · p. 5
  • With at least 1,633 retained rows, tuned Muse reaches 94.86% purity on 1,847 rows and Gemma reaches 94.93% on 1,715, below the pair’s purity.These are descriptive search maxima, not exact coverage-matched comparisons or out-of-sample guarantees.Source: B. Curation Purity and Coverage · pp. 4–5
  • At symmetric 0.86 gating, the unchanged positive set gains 64 negative labels, 40 wrong, reducing total purity to 94.64%; asymmetric gating avoids those errors.The table compares fixed judge outputs under different retention rules.Source: Table III · p. 5, B. Curation Purity and Coverage · p. 5
  • Total purity is relatively flat for τsel between .80 and .90 and τabs at or below .80, but no single operating point follows without a specified objective.Total purity also depends on the composition of the retained positive and negative labels.Source: B. Curation Purity and Coverage · pp. 5–6, Table IV · p. 6
  • Split-half threshold-selection optimism is 0.14 percentage points; the inherited point scores 96.88% purity at 81.7% coverage, while model and prompt choices remain in-sample.The objective-selected point differs from the inherited point by −0.09 purity points and +3.9 coverage points on held-out halves.Source: Table V · p. 6, B. Curation Purity and Coverage · p. 6

C. Residual Errors and Review

The paired judges retain false no-match labels that confidence thresholding cannot remove, and reviewing only deferred cases would miss them. Auditing accepted negatives is therefore important alongside adjudicating deferred rows.

C. Residual Errors and Review

  • 46 false no-match labels persist even at zero confidence; 43 already have null record IDs from both judges.The persistence reflects a limitation of thresholding these stored outputs, not proof of irreducible task ambiguity.Source: C. Residual Errors and Review · p. 6
  • The union of two additional configurations correctly selects 22 of the 46 shared false-abstention rows, leaving 24 unresolved by all four stored configurations.This union is an oracle diagnostic, not an evaluated deployable third-judge policy.Source: C. Residual Errors and Review · p. 6
  • Of 367 deferred rows, at least one constituent is correct on 358, but reviewing only disagreements would miss the 46 shared false abstentions in accepted negatives.The review-set count measures available suggestions, not human review accuracy or time saved; the authors recommend sampling accepted labels, particularly negatives, as well as adjudicating deferred cases.Source: C. Residual Errors and Review · p. 6

VI. DISCUSSION AND LIMITATIONS

The replay is retrospective, and the split-half check bounds only threshold-selection optimism, not the in-sample choice of teachers and prompts. The paper establishes label purity, not student performance or production savings, and its results are limited to one task, provider, and fixed retriever.

VI. DISCUSSION AND LIMITATIONS

  • Teacher and threshold selection reuse the evaluation benchmark, so the estimates are retrospective and the split-half check bounds only threshold-selection optimism.Model and prompt choices remain in-sample; fresh queries are needed to assess their optimism.Source: §VI · p. 6
  • The replay establishes neither student performance nor production cost savings; both require future experiments.The proposed work includes fresh training queries, adjudicated disagreements, audited negative labels, and student fine-tuning.Source: §VI · p. 6
  • Purity levels come from one English-dominant task at one provider with a fixed retriever and should not transfer to other catalogs or retrievers without re-measurement.Intervals also omit configuration selection beyond the threshold component, possible entity clustering, temporal drift, and uncertainty in expert labels.Source: §VI · p. 7
  • Corporate confidentiality restrictions prevent public release of the dataset, implementation code, and prompt templates.The analysis is a deterministic replay of stored judge outputs, with one script producing the reported tables, figures, macros, and input hash.Source: §VI · p. 7

VII. CONCLUSION

Separately gated agreement retained most queries at high label purity in retrospective replay, but the study does not establish downstream student utility. The conclusion calls for independent auditing of accepted negatives and further evaluation on fresh data.

VII. CONCLUSION

  • 1,633 of 2,000 queries were retained at 96.88% purity in the retrospective study.The conclusion identifies this as the observed result of separately gated agreement.Source: §VII · p. 7
  • Ordered thresholds prevent contradictory labels, but an ablation and 46 persistent false abstentions support independently auditing accepted negatives.Source: §VII · p. 7
  • Threshold-selection optimism was 0.14 percentage points on held-out halves, while configuration selection remained in-sample.The inherited operating point matched the full-benchmark optimum within one grid step under the stated objective.Source: §VII · p. 7
  • Higher reasoning effort yielded no clear F0.5 gain and increased median latency 1.8–5.0× in matched comparisons.Source: §VII · p. 7
  • Fresh-data curation, LoRA fine-tuning, and independent student evaluation remain necessary to establish downstream value.The study establishes label purity, not student utility.Source: §VII · p. 7

Related papers

Loading 2609.33878v2…