PaperlayerView exact revision on arXiv

Source-linked AI summary

When Names Cross Scripts: A Source-Grounded Benchmark for Historical Entity Reconciliation in the Mongol World

Xiang Chen, Zeyu Zhang

arXiv:2608.23507v1cs.CLcs.AI

TL;DR

Historical names vary across scripts and traditions, and similar names can belong to different people, making identity reconciliation more than string matching. MHER addresses this gap with a provenance-controlled pairwise benchmark separating names from source-grounded evidence. Correctly grounded evidence substantially improves reconciliation and resolves nearly all identical-surface challenge decisions, while exposing model-dependent abstention and name-interference failures.

  • Problem

    Historical identity reconciliation must handle cross-script name variation and non-unique names without treating surface resemblance as identity.

  • Method

    MHER is a provenance-controlled benchmark for pairwise reconciliation of source-attested person-name mentions, separating Name-only from independently Source-grounded evidence.

  • Results

    Correctly grounded evidence improves paired TEST accuracy over Name-only input across five generative systems, while resolving 24/25 identical-surface different-person model–item decisions.

  • Takeaways & Limitations

    MHER provides a controlled framework for studying evidence use, abstention, and failure modes in historical NLP.

  • Takeaways & Limitations

    The benchmark begins from already identified person-name mentions and does not evaluate named-entity recognition, retrieval, or end-to-end historical evidence discovery.

Abstract

from arXiv · show

Historical people may appear under different languages, scripts, and transcription traditions, while distinct individuals may share highly similar or even identical names. This makes historical identity reconciliation more than a problem of string matching or transliteration. We introduce MHER, a provenance-controlled benchmark for pairwise reconciliation of person-name attestations from the Mongol world. MHER contains a balanced 396-pair Name-only core over 84 primary historical persons and a stricter 160-pair Source-grounded subset constructed from mention-by-source evidence, with entity-disjoint development and test splits. Across five generative systems, correctly Source-grounded evidence improves paired TEST accuracy by 12.96 to 94.44 percentage points relative to Name-only input. On five identical-surface different-person cases, all models fail under names alone (0/25 model-item decisions), whereas Source-grounded evidence yields 24/25 correct resolutions, with the remaining output an abstention. Context-only ablations show that historical descriptions often carry substantial identity information, while explicitly signaled misgrounding controls produce substantially lower performance. We also find that names are not uniformly beneficial: for Qwen3-8B, restoring surface forms converts ten otherwise correct Context-only distinctions into false identity merges. These results show that historical entity reconciliation depends not only on surface correspondence, but on whether identity judgments respond appropriately to provenance-controlled historical evidence. MHER therefore provides a controlled framework for studying evidence use, abstention, and failure modes in historical NLP.

1 Introduction

Historical entity reconciliation must distinguish name-form correspondence from person identity because historical attestations vary across scripts and traditions, while different people can share names. MHER makes this distinction experimentally testable through provenance-controlled pairwise evaluation and evidence interventions.

  • Historical attestations vary across languages, scripts, transcription systems, titles, and scholarly romanizations, while distinct people may share identical or near-identical names.
  • MHER asks whether two source-attested person-name mentions denote the same historical individual rather than mapping mentions to a predefined knowledge base.Systems may explicitly abstain when the presented evidence is insufficient.
  • The benchmark separates Name-only judgments from Source-grounded judgments using evidence associated at the mention×source level.Context-only ablations and shuffled-context controls distinguish historical-description information from correct evidence grounding.
  • MHER contains a balanced 396-pair Name-only core over 84 primary historical persons and a stricter 160-pair Source-grounded subset with entity-disjoint DEV/TEST splits.
  • The evaluation combines diverse systems with abstention, probabilistic scoring, confusable cases, repeated-entity sensitivity, and an expert-unresolved challenge.These measures target incorrect resolution, uncertainty, surface-form reliance, and failure to use historical evidence.

2 Related Work

MHER combines established strands in multilingual and historical entity processing, pairwise identity inference, record linkage, and evidence-grounded evaluation. Its narrower distinction is direct reconciliation of historically attested person mentions with independently manipulated source provenance.

  • MHER combines multilingual and cross-script entity processing, pairwise identity inference, historical NLP, record linkage, evidence grounding, benchmark falsification, and selective prediction.
  • Unlike conventional multilingual entity linking, MHER directly compares attestations without requiring resolution to a canonical knowledge-base inventory.Pairwise identity inference without a complete observed entity inventory predates MHER.
  • MHER focuses narrowly on historically attested person mentions whose identity evidence may need reconstruction across linguistic, script, and documentary traditions.
  • Transliteration can improve cross-script name correspondence, but it cannot resolve identical names belonging to different historical people or orthographically distant attestations of the same person.
  • Historical entity resources expose challenges from OCR and transcription noise, diachronic variation, heterogeneous conventions, sparse resources, ambiguity, and long-range context.
  • MHEL-LLaMo remains a mention-to-candidate or knowledge-base linking task with NIL prediction, whereas MHER manipulates evidence attached independently to two source attestations.
  • MHER treats context validity as a provenance issue, using mention×source representation and shuffled-context controls to break alignment deliberately.

3 Historical Entity Reconciliation

Historical entity reconciliation determines whether two person-name mentions refer to the same individual, without reducing identity to surface similarity or requiring a complete knowledge base. The framework separates name-intrinsic evidence from independently source-grounded historical evidence and permits abstention when evidence is insufficient.

  • Historical entity reconciliation asks whether two historically attested person-name mentions denote the same underlying individual.
  • The pairwise formulation avoids requiring either mention to map first to a complete, stable, and uncontested modern knowledge-base inventory.
  • Name correspondence is useful evidence for reconciliation but is neither necessary nor sufficient for historical identity.
  • Different languages and documentary traditions can produce distant forms for the same person, while reused names, titles, and lineage forms can make distinct people look identical.
  • The framework distinguishes evidence intrinsic to name representations from source-grounded evidence describing chronology, kinship, lineage, offices, geography, or documented events.
  • Name-only evaluation exposes mention representations, whereas Source-grounded evaluation additionally supplies evidence tied to each historical source.The intervention tests whether grounded information changes decisions otherwise made from names alone.
  • Gold identity remains binary, but systems may abstain rather than assert same or different when evidence is insufficient.This separates incorrect resolution from recognizing unresolved evidence.
  • MHER begins with already identified person-name mentions and therefore does not evaluate named-entity recognition, retrieval, or complete historical knowledge-base construction.

4 The MHER Benchmark

MHER constructs historical identity-reconciliation benchmarks from a source-linked registry, separating canonical entities from multilingual or multiscript mentions and controlling evidence provenance. Its balanced, entity-disjoint core and stricter Source-grounded subset preserve unresolved cases separately and audit contextual leakage.

  • Entities and scholarly gold: MHER derives evaluation pairs from a source-linked registry of canonical historical persons and attested multilingual or multiscript name mentions.Canonical identifiers and curator metadata support construction but are not exposed to evaluated systems.
  • Entities and scholarly gold: 84 confirmed persons define the binary benchmark, with same labels requiring one confirmed person and different labels requiring two distinct confirmed persons.Ten unresolved scholarly identifications are excluded from binary gold and retained as a separate challenge.
  • Splits and benchmark composition: Entity-disjoint DEV/TEST splitting assigns 17 persons to DEV and 67 to TEST, preventing aliases or transcriptions of one person from crossing the boundary.Exact-surface hard negatives also remain within a single split.
  • Benchmark composition: The Name-only core contains 396 balanced binary items, while Source-grounded evaluation uses mention×source evidence and yields 160 provenance-complete paired items.The 108-item Source-grounded TEST subset is compared with Name-only on the same underlying identity questions.
  • Leakage-aware context construction: Source-grounded contexts must be independently associated with each mention and cannot state equivalence, expose target aliases, or copy shared canonical descriptions.A repaired construction and audit found no source-independence failure, exact predictive duplication, explicit equivalence statement, or target-alias leakage.
  • Release boundary: The benchmark’s frozen materials and reproducibility package were not publicly released in the arXiv version, with release planned upon publication subject to source-redistribution constraints.

5 Experimental Design

The experiments isolate how reconciliation changes when names, correctly grounded historical contexts, or deliberately mismatched contexts are provided for the same identity questions. Five generative systems and non-generative baselines are evaluated with abstention-aware outputs and locked TEST procedures.

  • Evaluation logic: The primary estimand is the within-item Name-only versus Source-grounded performance difference on the same 108 TEST questions.The full 316-item Name-only TEST set is reserved for descriptive breadth rather than direct comparison.
  • Evidence conditions: Name-only provides two mention representations with language/script metadata, Context-only removes names, and Source-grounded combines mentions with correctly associated evidence.The Context-only ablation tests the signal in historical evidence independently of names.
  • Evidence conditions: Three deterministic shuffled-context controls replace original mention–evidence associations while preserving split-specific context pools and explicitly disclosing the derangement.They are informed misgrounding stress tests, not blinded estimates of evidence-alignment effects.
  • Evaluation conditions: Each model receives 316 Name-only TEST items, five 108-item contextual conditions, and ten expert-unresolved items evaluated separately from the binary core.No Source-grounded unresolved condition is constructed because uncertainty notes would leak the abstention target.
  • Baselines: Non-generative baselines range from surface matching and cross-script Unidecode similarity to contextual lexical, logistic, and multilingual embedding methods, with parameters selected on DEV only.These baselines test surface correspondence or generic similarity without generative historical reasoning.

6 Main Results

Name-only performance varies sharply across methods and model uncertainty policies, while correctly Source-grounded evidence improves all five generative systems on the same 108-item TEST questions. The strongest gains occur for first-party API systems, but the result is bounded to evidence-eligible cases.

  • Non-generative baselines: 64.87% accuracy is achieved by Gemini Embedding 2 on the full 316-item Name-only TEST set, exceeding exact normalized matching, Unicode Levenshtein, and Unidecode-Levenshtein baselines.The embedding baseline reaches macro-F1 = .625.
  • Non-generative baselines: 86.11% accuracy is reached by Gemini Embedding 2 on the paired contextual TEST subset, compared with 76.85% for token Jaccard.Generic multilingual semantics recover substantial signal but do not eliminate historical identity ambiguity.
  • Name-only behavior: Name-only generative behavior spans 4.11%–67.09% accuracy, reflecting sharply different abstention policies across the five systems.API systems often abstain heavily, whereas Qwen3-8B never abstains on the full Name-only TEST set.
  • Paired Source-grounded intervention: All five systems improve under Source-grounded evidence on the same 108 identity questions, with first-party gains ranging from 67.59 to 94.44 percentage points.The four first-party comparisons remain significant after correction across five prespecified model-level tests.
  • Paired Source-grounded intervention: 12.96 percentage points is Qwen3-8B’s paired gain, rising from 75.00% to 87.96% accuracy with a 95% paired-bootstrap CI of [2.78, 23.15].Its gain is positive but individually weaker after the prespecified Bonferroni adjustment.
  • Interpretation and scope: The near-ceiling results do not establish that historical entity reconciliation is generally solved because the paired subset requires independently sourced, provenance-traceable evidence on both sides.The intervention demonstrates resolution within this evidence-eligible subset.
  • Interpretation and scope: 87.96–100.00% is the Source-grounded accuracy range, compared with 4.63–75.00% for Name-only on the paired subset.The convergence occurs despite differences in model family, provider, access mode, and abstention propensity.

7 Falsification and Ablation Analysis

The ablations separate identity information in historical context from the correctness of its provenance. Correctly grounded evidence remains effective across systems, while disclosed misgrounding sharply reduces performance, with model-specific responses and a stable primary effect.

  • Context-only accuracy remains high across all five systems, ranging from 80.56% for GPT-5.6 Luna to 94.44% for Gemini 3.7 Flash and Qwen3-8B.
  • Source-grounded accuracy exceeds Context-only accuracy for Terra, Luna, Sol, and Gemini by 12.96, 18.52, 6.48, and 5.56 percentage points, respectively.
  • Qwen3-8B is the exception: Context-only reaches 94.44%, while Source-grounded reaches 87.96%, a nonsignificant decrease of 6.48 points.
  • Correctly grounded evidence substantially outperforms explicitly signaled shuffled-context controls across all five systems, with gaps from 28.09 to 96.30 percentage points.
  • The interpretation is bounded because shuffled conditions also change their instructions, so their gap is not a blinded causal estimate of alignment alone.
  • All five systems abstain on the ten expert-unresolved cases, confirming that the common protocol permits abstention when unresolved status is explicitly disclosed.
  • The paired effect remains stable after accounting for repeated entities, with leave-one-entity-out changes of at most 2.19 percentage points for the four first-party systems.
  • The controls show that historical context carries substantial identity information, but high Source-grounded performance is not reproduced when pair-specific alignment is invalidated and disclosed.

8 Model Behavior and Error Analysis

Model behavior reveals that names cannot resolve identical-surface historical identities without contextual evidence. It also exposes heterogeneous failure modes, including Qwen3-8B’s surface-form interference, differing abstention patterns, and model-specific prominence sensitivity.

  • 8.1 Surface-confusable Cases Expose the Limit of Name Matching: All five models score 0/5 on identical-surface different-person cases under Name-only evidence, while Source-grounded evidence yields 24/25 correct resolutions overall, with one abstention.
  • 8.1 Surface-confusable Cases Expose the Limit of Name Matching: Source-grounded decisions distinguish identical names using chronology, office, kinship, political role, and historical setting rather than string similarity alone.
  • 8.1 Surface-confusable Cases Expose the Limit of Name Matching: Source-grounded accuracy reaches 87.76–100% across 98 cross-language items and 86.67–100% across 60 cross-script items.
  • 8.2 Qwen3-8B Exhibits Surface-form Interference and Over-resolution: Qwen3-8B falls from 94.44% Context-only accuracy to 87.96% Source-grounded accuracy, although the paired difference is not statistically significant.
  • 8.2 Qwen3-8B Exhibits Surface-form Interference and Over-resolution: All 13 Qwen Source-grounded errors are false same predictions on different-person items, while all 54 same items are resolved correctly.
  • 8.2 Qwen3-8B Exhibits Surface-form Interference and Over-resolution: Restoring names changes ten Qwen decisions from correct different under Context-only evidence to erroneous same, a pattern termed surface-form interference.
  • 8.3 Abstention and Probabilistic Behavior: First-party systems abstain heavily under Name-only evidence but almost completely stop abstaining with correct grounding, whereas Qwen makes no abstentions and retains 13 false merges.
  • 8.3 Abstention and Probabilistic Behavior: For Qwen, restoring names raises mean gold probability from .732 to .792 but worsens Brier score from .164 to .202 and log loss from .352 to .385.

9 Discussion

MHER shows that historical reconciliation depends on how systems use provenance-controlled evidence, not merely on name forms. Correctly grounded evidence improves reliability, while misgrounding, abstention policies, and surface-form interference expose important evaluation distinctions.

  • Evaluation implications: Name-only scores combine orthographic signals, learned transliteration, prior familiarity, and model-specific willingness to commit.Sharp abstention differences mean accuracy alone is not a pure measure of historical knowledge.
  • Evidence-centered interpretation: Identical-surface hard negatives cannot be separated by normalization alone, but chronology, kinship, office, place, and related historical evidence can distinguish them.
  • Ablations and failure modes: Context-only results indicate that historical descriptions often carry substantial identity information even without names.Names can add evidence, but Qwen3-8B sometimes converts correct Context-only distinctions into false identity merges when surface forms are restored.
  • Implications: An evidence-centered workflow can expose sources for citation and human verification while reducing dependence on silent parametric memory.The proposed direction preserves provenance so historians can inspect, contest, or revise reconciliations.
  • Ablations and failure modes: Explicitly signaled misgrounding sharply reduces performance, showing that context presence is not equivalent to valid evidence.This comparison is an informed stress test because the prompt identifies the contexts as misgrounded.
  • Evidence-centered interpretation: Correctly source-grounded historical evidence supports reliable reconciliation across systems with dramatically different Name-only behavior.The stable finding is evidence-conditioned resolution rather than a single best-performing model.

10 Limitations

MHER’s controlled, evidence-eligible Mongol-world design strengthens causal interpretation but limits claims about generalization, archival realism, annotation independence, and model reproducibility.

  • Scope: MHER does not establish generalization beyond Mongol-world personal-name reconciliation or to other entity types and historical domains.Naming systems, transcription practices, source density, and corroborating evidence vary across domains.
  • Evidence availability: The Source-grounded TEST set includes only cases with provenance-traceable and sufficiently informative evidence assembled under the protocol.Near-ceiling performance therefore does not imply equally informative evidence will exist in real archives.
  • Evidence availability: Future evaluations should test longer, partially relevant, conflicting, and archive-retrieved evidence rather than only compact curated contexts.These settings would assess evidence discovery and combination, not merely use of already sufficient evidence.
  • Evaluation: Near-ceiling Source-grounded performance limits fine-grained ranking among the strongest systems.The benchmark is better suited to studying evidence conditions than differentiating near-ceiling models.
  • Curation and uncertainty: Gold labels come from curator-reviewed canonical and scholarly evidence records, not independent double annotation by multiple historians.Contested or ambiguous identifications are excluded from the primary binary core, while MHER-Unresolved contains only ten descriptive cases.
  • Scale: The benchmark’s 84 entities and 396 Name-only pairs are smaller than large contemporary entity-linking benchmarks because source-linked auditing is costly.Entity-disjoint splits reduce leakage, but larger registries would improve estimates of cross-entity and cross-domain generalization.
  • Interpretation: Prominence strata are visibility proxies, not measurements of training-data frequency or proof of memorization.Actual exposure would require training corpora or substantially different experimental designs.
  • Reproducibility: Commercial API results may change with deployment updates, while the Qwen3-8B result represents one fixed local deployment.Inference configuration and model provenance are therefore part of interpreting the results.

11 Conclusion

MHER frames historical identity reconciliation as a provenance-controlled comparison of source-attested person mentions rather than surface-name matching. Its experiments show that evidence can resolve cases names cannot, while provenance, abstention, and falsification controls remain central to evaluation.

  • Benchmark contribution: MHER directly asks whether two source-attested person mentions refer to the same historical individual across languages, scripts, and transcription traditions.It avoids requiring prior resolution to a modern knowledge-base entity.
  • Evidence versus names: Correctly source-grounded evidence makes identity resolution more reliable across model families despite sharply different Name-only policies.
  • Evidence versus names: Identical-surface cases become resolvable when chronology, kinship, office, place, and other historical evidence are available.
  • Evaluation implications: Context-only performance shows that the effect is not simply caused by supplying more text, while misgrounding controls produce lower performance than correct grounding.Surface forms can also interfere with otherwise sufficient evidence and induce false identity merges.
  • Broader direction: MHER illustrates why historical NLP should evaluate whether judgments change appropriately with preserved evidence, while tracking provenance, abstention, and falsification controls.
Loading 2608.23507v1…