PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability of LLM Judges for Evaluating Entity Alignment

Vaibhava Lakshmi Ravideshik, Mayank Kejriwal

arXiv:2610.09554v1cs.AI

TL;DR

Expert annotation makes large-scale EA evaluation costly, while the reliability of LLM judges for structured prediction remains unstudied. This study benchmarks judge reliability with perturbation diagnostics, system meta-evaluation, and counterfactual label flips. Visible labels collapse discrimination, whereas label-free evaluation recovers it in supported settings.

  • Problem

    Large-scale entity-alignment evaluation requires expensive expert annotation, and the reliability of LLM judges for structured prediction tasks remains unstudied.

    Source: §1 · p. 1

  • Method

    The study benchmarks LLM judges using perturbation diagnostics, evaluation across EA systems, and counterfactual label flips, alongside a label-free predictor protocol.

    Source: §1 · pp. 1–2, §3.4 · p. 4

  • Results

    Visible labels collapse judge discrimination (J-ROC-AUC 0.12–0.87), while label-free prediction recovers strongly positive alignment discrimination on distinctive-name datasets; counterfactual label flips yield FSR 53–99%.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings identify visible system labels as a structural vulnerability in EA judging and show that stronger frontier judges can exhibit greater label sensitivity.

    Source: §5.3 · p. 8, §6 · p. 9

    The study tests only three closed-source frontier judges, and Prompt III changes both label exposure and task framing, so improvement cannot be attributed solely to label removal.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12-0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93-1.00) and significant recovery on biomedical pairs (0.93-0.95). Counterfactual experiments confirm causality (FSR 53-99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen's kappa=0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH-SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.

1 Introduction

Entity alignment is important for integrating knowledge bases, but expert annotation makes large-scale evaluation costly. This study benchmarks LLM judges for EA and examines whether visible system labels undermine their reliability.

1 Introduction

  • Expert annotation makes systematic large-scale EA benchmarking practically infeasible, despite its importance for knowledge-base integration, ontology merging, and scientific data linking.LLM-as-judge evaluation is presented as a potentially scalable alternative.Source: §1 · p. 1
  • Anchor bias occurs when a judge rationalizes a visible MATCH/NON MATCH decision instead of assessing whether the entity evidence supports it.In the Figure 1 example, the judge scores a correct NON-MATCH decision 3/10, while the label-free predictor incorrectly predicts MATCH with 8/10 confidence.Source: §1 · p. 1, Figure 1 · p. 2
  • The study introduces a systematic three-experiment framework combining perturbation diagnostics, meta-evaluation across EA systems, and counterfactual label-flip analysis.It also releases a biomedical MeSH × SNOMED-CT benchmark with 15K pairs.Source: §1 · pp. 1–2
  • Visible system labels can collapse judge discrimination (J-ROC-AUC 0.12–0.87), while an independent label-free protocol recovers strongly positive discrimination on distinctive-name datasets.Counterfactual label flips show FSR of 53–99%; a blinded two-annotator evaluation also validates the mechanism.Source: §1 · p. 2

2 Related Work

Prior work establishes both the expense of evaluating entity alignment and reliability concerns in LLM judging, but does not address discrete decision labels in structured-prediction evaluation. The paper adapts counterfactual methods and contributes a benchmark for heterogeneous biomedical graphs.

Entity Alignment

  • Existing EA evaluation remains bottlenecked by expert annotation that is expensive and slow to produce.Prior benchmarks focus on homogeneous general-domain graph pairs such as DBpedia–Wikidata and DBpedia–YAGO.Source: §2 · pp. 2–3

LLM-as-Judge Evaluation and Its Failure Modes

  • Existing LLM-judge studies identify position, verbosity, and prompt-sensitivity biases, but do not address structured prediction with a discrete decision label in the judge’s input.Prior anchoring research concerns contextual anchors in numerical estimates or opinion questions, unlike a system label embedded as a rationalization target.Source: §2 · pp. 2–3

LLM Reasoning Fragility and Shortcut Learning

  • Across the three frontier models tested, anchor bias is amplified rather than mitigated by model scale, a pattern the paper calls the frontier model paradox.The paper connects this to shortcut learning, in which models rely on memorized surface correlations rather than structural evidence (Geirhos et al., 2020).Source: §2 · p. 3

Counterfactual Evaluation

  • The study adapts counterfactual evaluation by flipping system labels while holding entity evidence fixed, isolating label exposure as the causal driver of score shifts.The approach builds on contrast-set methods (Kaushik et al., 2019; Gardner et al., 2020).Source: §2 · p. 3

Knowledge Graph Benchmarking

  • The MeSH–SNOMED CT benchmark addresses biomedical EA’s schema heterogeneity, sparse expert annotation, and domain-specific terminology.It is released with full ontological context (Ravideshik and Kejriwal, 2026a,b).Source: §2 · p. 3

3 Methodology

The methodology defines anchor bias in EA judging and evaluates it through perturbation diagnostics, judge-versus-predictor comparisons, counterfactual label flips, and blinded human annotation. These components distinguish feature reliance, correctness discrimination, and sensitivity to visible labels.

3.1 Entity Alignment Formulation

  • An EA system assigns MATCH or NON MATCH to a candidate entity pair, and the judge scores whether that decision is correct on a 0–10 scale.The evaluation question is whether the score reflects correctness or is confounded by the visible decision label.Source: §3.1 · p. 3

3.2 Anchor Bias: Definition and Formalization

  • Anchor bias is the tendency to rationalize a displayed system label rather than independently assess the evidence for entity alignment.In this case, scores track the label rather than the evidence, making the judge a validator instead of an independent arbiter.Source: §3.2 · p. 3

3.3 Perturbation Bias Diagnostics

  • Perturbation diagnostics modify entity names, evidence, and context; PBI measures average score drop, JSR measures rank consistency, and MRI captures score collapses exceeding 2 points.A positive PBI indicates reliance on the perturbed feature; MRI captures extreme failures that PBI can miss.Source: §3.3 · pp. 3–4

3.4 EA System Meta-Evaluation

  • Meta-evaluation compares evidence-locked decision validation, minimal decision validation, and label-free alignment prediction, which withholds the system decision.The first two prompts evaluate a visible decision; the third probes predictor capability instead.Source: §3.4 · p. 4, Table 1 · p. 4

3.5 Counterfactual Label Sensitivity

  • Counterfactual tests hold entity evidence fixed and flip the visible label; MAS measures average score shift, FSR the share shifting over 2 points, and LFR whether shifts follow the new label.LFR above 50% indicates following the new label, while FSR and LFR distinguish score responsiveness from its direction.Source: §3.5 · pp. 4–5

3.6 Human Evaluation Protocol

  • Two authors independently scored the same pairs across four negative-sampling strategies under all three prompts, blinded to each other’s judgments.Sampling details, inter-annotator agreement, and gold-label accuracy are reported in §4.4.Source: §3.6 · p. 5

4 Experimental Setup

The study evaluates LLM-judge reliability across EA datasets, systems, and prompting conditions using perturbation diagnostics, meta-evaluation, counterfactual label flips, and human annotation. Its human evaluation tests agreement with gold labels and whether label exposure corresponds to uneven agreement across pair types.

4 Experimental Setup

  • The three datasets include the new 15K-pair MeSH × SNOMED-CT biomedical benchmark, whose ontologies present distinct structural challenges.BIO has no shared predicate vocabulary, sparse LLM pre-training signal, and genuine many-to-many alignments.Source: §4 · p. 5
  • The study compares EasyEA and NeuSymEA with ablated variants and evaluates GPT-4o-mini, GPT-4o, and Claude Opus 4.7 as judges.The variants disable EasyEA’s LLM candidate-selection stage or NeuSymEA’s symbolic refinement component.Source: §4 · pp. 5–6
  • For Experiments 2 and 3, predicted MATCH pairs and randomly sampled NON-MATCH pairs were selected without consulting ground-truth labels.Ground-truth labels were used only to compute evaluation metrics after sampling.Source: §4 · p. 6

4.1 Experiment 1: Perturbation Bias Diagnostics

  • Experiment 1 samples MATCH pairs and applies seven perturbations, including identity anonymization, evidence suppression, identifier removal, and structured-to-prose conversion.P1 directly ablates the name-recognition shortcut; P5 is the negative control.Source: §4.1 · p. 6

4.2 Experiment 2: EA System Meta-Evaluation

  • Experiment 2 evaluates dataset, system, judge, and prompt combinations on 500 pairs per condition, measuring Score Gap, PDV, and J-ROC-AUC.For Prompts I–II, J-ROC-AUC below 0.5 operationally indicates discrimination inversion; under Prompt III it measures alignment prediction instead.Source: §4.2 · p. 6

4.3 Experiment 3: Counterfactual Label Sensitivity

  • Experiment 3 flips system decisions while holding entity evidence fixed, using 100 pairs per condition to measure label effects with MAS, FSR, and LFR.It covers EasyEA and NeuSymEA on DW and DY under Prompts I and II; BIO and Prompt III are excluded.Source: §4.3 · p. 6

4.4 Experiment 4: Human Evaluation

  • Claude Opus 4.7 had the highest human-consensus agreement (ρ = 0.920, κ = 0.871, MAE = 0.23), while label-exposed MATCH agreement was ρ = 0.228 to 0.412.Annotator agreement was Cohen’s κ = 0.902, and consensus gold-label accuracy was 91.3%.Source: §4.4 · p. 7
  • PDV ratios were 0.21–0.81 for general-domain datasets and 1.17–1.35 for BIO, indicating variance compression versus inflation under minimal prompting.The passage attributes BIO inflation to absent pre-training signal.Source: Figure 2 · p. 7

5 Results

Visible decision labels undermine EA judge discrimination, while withholding labels restores alignment-prediction capability in many settings. Counterfactual tests and biomedical results show that the bias and recovery vary with model, system, and domain.

5.1 Perturbation Bias Diagnostics

  • Identity anonymization is the singular catastrophic perturbation on BIO, producing MRI of 61–69%, while general-domain judges remain stable across conditions.No other BIO perturbation exceeds 7% MRI; order randomization, a negative control, produces MRI below 5% across conditions.Source: §5.1 · p. 7
  • Minimal prompting yields higher baseline scores across all nine conditions, with PDV ratios below 1.0 on DW and DY and above 1.0 on BIO.The reported ranges are 0.21–0.81 on DW and DY and 1.17–1.35 on BIO; the authors attribute the difference to memorized entity knowledge versus absent pre-training signal.Source: §5.1 · p. 7

5.2 EA System Meta-Evaluation

  • J-ROC-AUC falls to 0.12–0.53 under label-exposing prompts, while Prompt III recovers near-ceiling discrimination.Under Prompt III, 18 of 24 DW and DY conditions achieve statistically significant positive discrimination; recovery depends on dataset and EA system.Source: §5.2 · pp. 7–8
  • Under Prompt III, NeuSymEA variants reach J-ROC-AUC 0.981–1.000 on DW and DY, while EasyEA variants collapse on DY.Near-identical names limit label-free discrimination on DY; structure-aware alignment training is associated with robustness to predictor-mode evaluation.Source: §5.2 · p. 8

5.3 Counterfactual Label Sensitivity and the Frontier Model Paradox

  • Visible labels drive score changes in 53–99% of pairs, and Claude Opus 4.7 reaches FSR = 99% on DW under EasyEA.Prompt II amplifies label sensitivity rather than reducing it; stronger tested judges show greater sensitivity.Source: §5.3 · p. 8, Table 2 · p. 9

5.4 Biomedical Domain: A Distinct Failure Mode

  • On BIO, EasyEA achieves significant positive label-free discrimination across all three judges, with J-ROC-AUC 0.931–0.948.Recovery is weaker than on DW, reflecting sparse attribute overlap and heterogeneous relation predicates; BIO also shows prompt-invariant collapse under identity anonymization.Source: §5.4 · p. 9

6 Conclusion

The conclusion attributes judge failure to structural vulnerabilities to visible labels and entity-name cues rather than a reasoning deficit. It reports counterfactual evidence for label-following and recommends withholding system decisions from LLM judges.

6 Conclusion

  • Visible labels and entity-name cues are structural vulnerabilities, not simply a reasoning deficit.The conclusion describes Prompt III as withholding labels and reports that label-flip experiments confirm score shifts toward visible labels.Source: §6 · p. 9
  • Counterfactual label flips produce score shifts exceeding 2 points in 53–99% of pairs, with Label Following Rate at or above chance in most conditions.The conclusion says a blinded human evaluation corroborates the mechanism at the level of individual judgments.Source: §6 · p. 9
  • Practitioners should withhold system decisions from LLM judges entirely.The conclusion frames this as a structural fix rather than a prompt-based one.Source: §6 · p. 9

Ethical Considerations

The evaluation uses public entity-alignment benchmarks and public medical ontologies, and releases code to support reproducible auditing.

Ethical Considerations

  • The evaluation uses publicly available entity-alignment benchmarks and collects no new personal or sensitive data.The work focuses on system behavior and does not involve human subjects.Source: Ethical Considerations · p. 9
  • The BIO benchmark draws exclusively from public MeSH and SNOMED-CT ontologies, subject to their stated licensing terms.Source: Ethical Considerations · p. 9
  • The authors release evaluation code to promote reproducible auditing of LLM-as-judge approaches.They connect systematic reliability analysis in structured tasks with research integrity and preventing biased judgments from propagating downstream.Source: Ethical Considerations · p. 9

Limitations

This section delineates the study’s methodological and practical limits, including its judge, language, prompt-design, and human-validation choices, as well as the confounding of label exposure and task framing in Prompt III. It also describes the prompt and baseline setup, threshold-sweep metrics and scope, and the experiments’ computational scale, costs, and estimation caveats.

Limitations

  • The study evaluates only three closed-source frontier judges and English datasets, leaving open-source models, non-English data, other model families, and structured prediction tasks beyond entity alignment untested.The frontier model paradox was observed across three models accessed through one API provider.Source: Limitations · p. 10
  • Prompt III changes label exposure and task framing simultaneously, so its improvement cannot be attributed solely to label removal without a matched control.The human evaluation used two author-annotators rather than independent expert raters; blinding and strong inter-annotator agreement mitigate, but do not eliminate, this concern.Source: Limitations · p. 10

A.1 Prompt I: Evidence-Locked Evaluation

  • Prompt I exposes the system decision and frames judging as evidence-locked validation, using an eight-step chain-of-thought protocol.The judge is instructed to suppress pre-training knowledge and reason exclusively from provided evidence.Source: §A.1 · p. 13
  • Prompt I’s output format requests a score, an evidence-specific critique, an error category for scores of 5 or below, and the evidence strings used.The checklist includes missed convergent signals, over-weighted weak evidence, under-weighted distinctive evidence, reasoning beyond evidence, and mishandled data quality.Source: §A.1 · p. 15

A.2 Prompt II: Minimal Evaluation

  • Prompt II is a zero-guidance baseline that shows the system decision and asks the judge to score its justification using its default reasoning strategy.It omits chain-of-thought instructions, memory suppression, an evidence boundary, and a rubric.Source: §A.2 · p. 15

A.3 Prompt III: LLM as Predictor

  • Prompt III withholds the system decision and asks the LLM to make an independent alignment prediction, so its score directly represents match confidence rather than decision justification.Its six-step protocol reasons from supplied evidence, with high scores indicating match confidence and low scores indicating non-match confidence.Source: §A.3 · pp. 15, 17

Appendix B: Judge Meta-Evaluation Threshold Sweeps

  • Tables 3–6 report J-Precision, J-Recall, J-F1, and J-ROC-AUC across thresholds 5–9 for EasyEA on DW, DY, and BIO.The metrics compare the correctness indicator computed after sampling with the judge’s binary acceptance.Source: Appendix B · p. 17
  • Threshold sweeps report J-Prec, J-Rec, and J-F1 at t=5–9 plus J-ROC-AUC for EasyEA and EasyEA-EmbOnly on DW (n=500), and for NeuSymEA variants.Appendix B defines the thresholded judge output as indicating whether the judge correctly identifies EA-system correctness.Source: Appendix B · p. 17, Table 3 · p. 18, Table 4 · p. 18

Appendix C: Computational Cost and API Usage

  • 108,000 calls comprised Experiment 1 and 45,000 comprised Experiment 2; BIO omitted NeuSymEA because it was not evaluated on that dataset.GPT-4o-mini and GPT-4o token counts in Experiment 1 were exact; remaining counts were extrapolated from 50-row samples. Temperature 0 did not formally guarantee determinism across arbitrary model snapshots.Source: Appendix C · pp. 17–18
  • $577.17 covered 153,000 calls and 238.8M tokens across Experiments 1–2; the NeuSymEA label-flip extension cost approximately $8–12, and human evaluation used previously collected judge scores.The human evaluation required no additional API calls because it reused scores from Experiments 2–3 alongside new human annotations.Source: Appendix C · p. 18

D.0 Sample Size Justification

The appendix tests whether sample sizes and analysis choices explain the reported effects. The findings support stable estimates, threshold-robust domain asymmetry, and persistent label sensitivity under Prompt II.

D.0 Sample Size Justification

  • At n = 500, bootstrap intervals are tight and non-overlapping, while the sample size keeps API use feasible and supports stable metric estimates.Experiment 3 uses n = 100 per condition; its FSR confidence intervals are approximately ±10 percentage points, sufficient to detect the observed 53%–99% differences.Source: §D.0 · pp. 18–19

D.1 MRI Threshold Sensitivity

  • BIO MRI remains substantially higher than general-domain MRI at every tested threshold, so the t = 2 choice does not create the domain asymmetry.For BIO under Prompt II, MRI remains high across thresholds t = 1, 2, and 3 for all three judges.Source: §D.1 · p. 20, Table 9 · p. 21

D.2 Bootstrap Confidence Intervals for PBI and MRI

  • BIO estimates are non-overlapping with general-domain estimates for every judge and prompt; all BIO MRI lower bounds exceed 15%, while general-domain upper bounds stay below 9%.The only PBI confidence interval crossing zero is DY 4o Prompt I: −0.072 [−0.178, +0.036].Source: §D.2 · p. 20, Table 10 · p. 21

D.3 Label-Flip Effects Under Prompt II

  • Under Prompt II (n=100 per condition), MAS and FSR are uniformly higher than under Prompt I across datasets, judges, and systems; Claude Opus 4.7 reaches FSR 99% on DW for both systems.Low LFR, high FSR, and J-ROC-AUC < 0.5 together indicate label sensitivity without purely label-following.Source: §D.3 · p. 21

Appendix E: BIOMED Dataset Construction and Validation

Appendix E describes the construction and validation of BIO, a biomedical benchmark built from MeSH and SNOMED CT, and records its release conditions and limitations.

Appendix E: BIOMED Dataset Construction and Validation

  • SNOMED CT-derived files require a current UMLS license, and redistribution is limited to other current UMLS license holders.The benchmark does not confer any license beyond what is available directly from NLM and UMLS.Source: Appendix E · p. 22
  • 15,000 gold alignment pairs were stratified-sampled from 46,823 UMLS cross-references for BIO.The release also provides the full pre-sample set in ent_links_uri.Source: Appendix E · p. 22
  • 99.8% of MeSH and 97.2% of SNOMED CT gold entities appear in relation graphs, while 94.5% of sampled MeSH neighbors are non-gold background entities.The benchmark spans hierarchical versus description-logic schemas, different attributes and relation densities, and finer-grained SNOMED CT concepts that produce many-to-many alignments.Source: Appendix E · p. 22
  • BIO is released as six tab-separated files, including 15K gold pairs and triple files totaling 2.58GB, with no fixed train/test split.The release separates MeSH-derived and SNOMED CT-derived files so their differing license terms are clear.Source: Appendix E · p. 22
  • Ground-truth links come from UMLS CUI co-membership rather than newly adjudicated annotation, and the ontologies may change in future source releases.The source versions are MeSH December 2023 and SNOMED CT March 2026; not all biomedical concepts are one-to-one alignable.Source: Appendix E · pp. 22–23

Appendix F: Score Gap Analysis and Significance

The appendix reports score-gap patterns across prompts and perturbation diagnostics, with label-free results varying by dataset and the largest biomedical collapse tied specifically to identity anonymization.

Appendix F: Score Gap Analysis and Significance

  • 25 of 30 Prompt III conditions show a statistically significant Score Gap, while six conditions lack significant positive discrimination, concentrated in DY.Under Prompts I and II, gaps are predominantly negative or near-zero; the DY exceptions involve near-identical entity names, and Prompt III J-ROC-AUC measures prediction rather than judge correctness discrimination.Source: Table 13 · p. 23, Appendix F · p. 23

Appendix G: Full Perturbation Bias Diagnostics

  • P4 and P7 have modest effects on general-domain datasets and near-zero MRI on BIO, while P5 MRI remains below 5% across conditions.P5 is the negative control; these results distinguish BIO's identity-anonymization collapse from generic input perturbation.Source: Appendix G · p. 23

Appendix H: PDV Baseline Statistics

  • On BIO, prompt sensitivity correlates negatively with perturbation sensitivity for GPT-4o-mini (ρ = −0.470∗∗∗) and GPT-4o (ρ = −0.320∗∗∗).The passage interprets the pattern as name-inflated scores being disproportionately destabilized by identity anonymization.Source: Appendix H · p. 23

Appendix G: Full Perturbation Bias Diagnostics

  • 66.4%, 61.9%, 63.0%, and 68.8% MRI on BIO across all judges and prompts occur under identity anonymization, while no other perturbation exceeds 7% MRI.Table 14 reports P1, P2, P3, and P6; the reported BIO collapse is specific to P1 rather than generic perturbation.Source: Appendix G · p. 23, Table 14 · p. 24

Appendix I: Sampling Procedures for All Experiments

Appendix I specifies how pairs were sampled for perturbation, meta-evaluation, counterfactual, and human experiments, and notes that random negatives make the reported anchor-bias severity a conservative estimate.

Appendix I: Sampling Procedures for All Experiments

  • Experiment 1 sampled 500 true alignment pairs per dataset, with no NON-MATCH class, and applied perturbations under all three prompts.Because no system decision label was involved in sampling, ground-truth alignments served directly as the evaluation pairs.Source: Appendix I · p. 24
  • Experiment 2 sampled 250 MATCH pairs from each system's predicted alignments and 250 NON-MATCH pairs by randomly drawing entity pairs and rejecting predicted alignments.Ground-truth labels were used only after sampling to compute evaluation metrics; system outputs were fixed before judge evaluation.Source: Appendix I · p. 24
  • Random negatives are easier than hard negatives, so anchor-bias collapse below 0.5 J-ROC-AUC even on these pairs establishes a lower bound on severity in realistic settings.The comparison is to nearest-neighbor candidates and same-type pairs.Source: Appendix I · p. 24
  • Experiment 3 flipped each system decision label while holding entity evidence fixed, then compared scores under the original and flipped labels.The same sampling procedure as Experiment 2 was used; ground-truth labels were used only to compute evaluation metrics after scoring.Source: Appendix I · pp. 24–25
  • The blinded human evaluation covered 102 pairs—50 MATCH and 52 NON-MATCH—sampled across four negative strategies and scored independently by both authors under three prompts.Each annotator scored 306 instances, using the same 0–10 rubric as the LLM judges.Source: Appendix I · p. 25

Appendix J: Extended Discussion

The extended discussion reports human validation of the anchor-bias pattern, proposes a tentative explanation, and distinguishes measured EA findings from broader predictions and dataset-specific limits.

Appendix J: Extended Discussion

  • The human evaluation table reports blinded inter-annotator agreement and shows that judge agreement with human consensus collapses on MATCH pairs but remains high on NON-MATCH pairs.The table covers 102 pairs and 306 scored instances per annotator.Source: Table 17 · p. 26, Appendix J · p. 26
  • The authors offer pre-training exposure to entity names as a possible explanation for stronger judges' greater label sensitivity, but treat it as an empirical regularity across three models, not a settled explanation.Testing the explanation on open-source models is future work.Source: Appendix J · p. 26
  • Anchor bias is predicted to extend to structured-prediction tasks such as relation extraction and slot filling, but this remains unvalidated beyond EA.Testing at least one additional task is identified as the highest-priority follow-up.Source: Appendix J · p. 26
  • On near-identical-name DY pairs with weak EA systems, Prompt III is non-significant at J-ROC-AUC 0.468–0.527, a limitation attributed to absent lexical discriminative signal.The passage says prompt design cannot overcome this limitation without structural evidence.Source: Appendix J · p. 26
Loading 2610.09554v1…